2024-03-14T09:25:25.830 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-14T09:25:25.831 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679 --name yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679/orig.config.yaml 2024-03-14T09:25:25.891 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-14T09:25:25.981 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679 branch: wip-yuri-testing-2024-03-12-1240-reef description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} email: yweinste@redhat.com first_in_suite: false job_id: '7598679' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-yuri-testing-2024-03-12-1240-reef ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri-testing-2024-03-12-1240-reef sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8503 sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri-testing-2024-03-12-1240-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_2f465319e3c83d48f999325459bcdfd0346449c0/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIFWTHVy4dzhYc0F7qesokjZimANuuyTtAS+YDFQ+0JH/BDVHyhOXOzrddNrj9tQP+mCglI5bJ9ZMbpDbyUAyNE= smithi132.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB0jb3R9uXHmMo7Jr4xV1KZDSVG6sUsXjd19ChB3zt+yG9QrqEJ9vm8+gZ0Zh50EwHZWpspwDnDDlKLChabP6Ak= smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBECpe1ikaXqUwa5K2sajdqkpk32DfnqCajHjGJiA4oxb1AuwXb0TZ/p0fE/wUCumnkGQw5mXKypRyCfem6JEYQc= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-13_19:31:43 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-14T09:25:25.982 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_2f465319e3c83d48f999325459bcdfd0346449c0/qa; will attempt to use it 2024-03-14T09:25:25.982 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_2f465319e3c83d48f999325459bcdfd0346449c0/qa/tasks 2024-03-14T09:25:25.983 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-14T09:25:25.984 INFO:teuthology.task.internal:Checking packages... 2024-03-14T09:25:26.006 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '2f465319e3c83d48f999325459bcdfd0346449c0' 2024-03-14T09:25:26.006 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-14T09:25:26.006 INFO:teuthology.packaging:ref: None 2024-03-14T09:25:26.006 INFO:teuthology.packaging:tag: None 2024-03-14T09:25:26.006 INFO:teuthology.packaging:branch: wip-yuri-testing-2024-03-12-1240-reef 2024-03-14T09:25:26.007 INFO:teuthology.packaging:sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T09:25:26.007 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2024-03-12-1240-reef 2024-03-14T09:25:26.226 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-825.g2f465319 2024-03-14T09:25:26.227 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-14T09:25:26.234 INFO:teuthology.task.internal:no buildpackages task found 2024-03-14T09:25:26.234 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-14T09:25:26.262 INFO:teuthology.task.internal:Saving configuration 2024-03-14T09:25:26.284 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-14T09:25:26.290 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-14T09:25:26.316 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-14 09:20:12.353943', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIFWTHVy4dzhYc0F7qesokjZimANuuyTtAS+YDFQ+0JH/BDVHyhOXOzrddNrj9tQP+mCglI5bJ9ZMbpDbyUAyNE='} 2024-03-14T09:25:26.338 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-14 09:20:12.352774', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB0jb3R9uXHmMo7Jr4xV1KZDSVG6sUsXjd19ChB3zt+yG9QrqEJ9vm8+gZ0Zh50EwHZWpspwDnDDlKLChabP6Ak='} 2024-03-14T09:25:26.360 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-14 09:20:12.351403', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBECpe1ikaXqUwa5K2sajdqkpk32DfnqCajHjGJiA4oxb1AuwXb0TZ/p0fE/wUCumnkGQw5mXKypRyCfem6JEYQc='} 2024-03-14T09:25:26.360 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-14T09:25:26.366 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-14T09:25:26.366 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-14T09:25:26.366 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-14T09:25:26.366 INFO:teuthology.run_tasks:Running task console_log... 2024-03-14T09:25:26.484 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9cc08ab160>, signals=[15]) 2024-03-14T09:25:26.484 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-14T09:25:26.490 INFO:teuthology.task.internal:Opening connections... 2024-03-14T09:25:26.490 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2024-03-14T09:25:26.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T09:25:26.572 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi132.front.sepia.ceph.com 2024-03-14T09:25:26.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T09:25:26.644 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2024-03-14T09:25:26.645 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T09:25:26.722 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-14T09:25:26.729 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2024-03-14T09:25:26.750 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2024-03-14T09:25:26.751 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2024-03-14T09:25:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="Red Hat Enterprise Linux" 2024-03-14T09:25:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="8.6 (Ootpa)" 2024-03-14T09:25:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:ID="rhel" 2024-03-14T09:25:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="fedora" 2024-03-14T09:25:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="8.6" 2024-03-14T09:25:26.811 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el8" 2024-03-14T09:25:26.812 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-14T09:25:26.812 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2024-03-14T09:25:26.812 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-14T09:25:26.812 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://www.redhat.com/" 2024-03-14T09:25:26.812 INFO:teuthology.orchestra.run.smithi078.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-14T09:25:26.812 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-14T09:25:26.812 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:25:26.812 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-14T09:25:26.812 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-14T09:25:26.812 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-14T09:25:26.812 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-14T09:25:26.813 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2024-03-14T09:25:26.841 DEBUG:teuthology.orchestra.run.smithi132:> uname -m 2024-03-14T09:25:26.862 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2024-03-14T09:25:26.862 DEBUG:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2024-03-14T09:25:26.923 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="Red Hat Enterprise Linux" 2024-03-14T09:25:26.923 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="8.6 (Ootpa)" 2024-03-14T09:25:26.923 INFO:teuthology.orchestra.run.smithi132.stdout:ID="rhel" 2024-03-14T09:25:26.923 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="fedora" 2024-03-14T09:25:26.923 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="8.6" 2024-03-14T09:25:26.923 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el8" 2024-03-14T09:25:26.924 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-14T09:25:26.924 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31" 2024-03-14T09:25:26.924 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-14T09:25:26.924 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://www.redhat.com/" 2024-03-14T09:25:26.924 INFO:teuthology.orchestra.run.smithi132.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-14T09:25:26.924 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-14T09:25:26.924 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-14T09:25:26.924 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-14T09:25:26.924 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-14T09:25:26.925 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-14T09:25:26.925 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-14T09:25:26.925 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2024-03-14T09:25:26.950 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2024-03-14T09:25:26.971 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2024-03-14T09:25:26.971 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2024-03-14T09:25:27.030 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="Red Hat Enterprise Linux" 2024-03-14T09:25:27.031 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="8.6 (Ootpa)" 2024-03-14T09:25:27.031 INFO:teuthology.orchestra.run.smithi152.stdout:ID="rhel" 2024-03-14T09:25:27.031 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="fedora" 2024-03-14T09:25:27.031 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="8.6" 2024-03-14T09:25:27.031 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el8" 2024-03-14T09:25:27.031 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-14T09:25:27.031 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2024-03-14T09:25:27.031 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-14T09:25:27.031 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://www.redhat.com/" 2024-03-14T09:25:27.031 INFO:teuthology.orchestra.run.smithi152.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-14T09:25:27.031 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-14T09:25:27.031 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-14T09:25:27.031 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-14T09:25:27.031 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-14T09:25:27.032 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-14T09:25:27.032 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-14T09:25:27.032 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2024-03-14T09:25:27.058 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-14T09:25:27.068 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-14T09:25:27.108 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-14T09:25:27.108 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2024-03-14T09:25:27.111 DEBUG:teuthology.orchestra.run.smithi132:> test '!' -e /home/ubuntu/cephtest 2024-03-14T09:25:27.113 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2024-03-14T09:25:27.132 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-14T09:25:27.137 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-14T09:25:27.138 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2024-03-14T09:25:27.170 DEBUG:teuthology.orchestra.run.smithi132:> test -z $(ls -A /var/lib/ceph) 2024-03-14T09:25:27.172 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2024-03-14T09:25:27.204 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-14T09:25:27.279 INFO:teuthology.run_tasks:Running task kernel... 2024-03-14T09:25:27.299 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-14T09:25:27.299 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-14T09:25:27.299 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-14T09:25:27.300 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-14T09:25:27.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T09:25:27.319 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2024-03-14T09:25:27.632 DEBUG:teuthology.task.kernel:sha1 for client.0 is 5f16de4b3661fb3d4a041ecb2925baeffc01da4a 2024-03-14T09:25:27.633 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5f16de4b3661fb3d4a041ecb2925baeffc01da4a"... 2024-03-14T09:25:27.633 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-03-14T09:25:27.652 INFO:teuthology.orchestra.run.smithi078.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-03-14T09:25:27.652 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-513.9.1.el8_9.x86_64 vs 5f16de4b3661fb3d4a041ecb2925baeffc01da4a 2024-03-14T09:25:27.653 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-14T09:25:27.653 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum upgrade -y linux-firmware 2024-03-14T09:25:29.194 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2024-03-14T09:25:29.194 INFO:teuthology.orchestra.run.smithi078.stdout:Unable to read consumer identity 2024-03-14T09:25:29.458 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:25:29.458 INFO:teuthology.orchestra.run.smithi078.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-03-14T09:25:29.458 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:25:31.265 INFO:teuthology.orchestra.run.smithi078.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-03-14T09:25:48.014 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 989 kB/s | 16 MB 00:16 2024-03-14T09:25:51.556 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 394 kB/s | 24 kB 00:00 2024-03-14T09:25:52.397 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-14T09:25:52.398 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2024-03-14T09:25:52.399 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-14T09:25:52.520 INFO:teuthology.task.kernel:Downloading kernel 5f16de4b3661fb3d4a041ecb2925baeffc01da4a on client.0... 2024-03-14T09:25:52.520 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5f16de4b3661fb3d4a041ecb2925baeffc01da4a 2024-03-14T09:25:52.748 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/5f16de4b3661fb3d4a041ecb2925baeffc01da4a/centos/8/flavors/default/x86_64/ 2024-03-14T09:25:52.749 DEBUG:teuthology.orchestra.run.smithi078:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc7_g5f16de4b3661-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/5f16de4b3661fb3d4a041ecb2925baeffc01da4a/centos/8/flavors/default/x86_64/ --input-file=- 2024-03-14T09:25:52.751 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi078.front.sepia.ceph.com to complete... 2024-03-14T09:26:06.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14 09:26:06 URL:https://1.chacra.ceph.com/r/kernel/testing/5f16de4b3661fb3d4a041ecb2925baeffc01da4a/centos/8/flavors/default/x86_64/kernel-6.8.0_rc7_g5f16de4b3661-1.x86_64.rpm [401619248/401619248] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-14T09:26:06.231 INFO:teuthology.orchestra.run.smithi078.stderr:FINISHED --2024-03-14 09:26:06-- 2024-03-14T09:26:06.231 INFO:teuthology.orchestra.run.smithi078.stderr:Total wall clock time: 13s 2024-03-14T09:26:06.231 INFO:teuthology.orchestra.run.smithi078.stderr:Downloaded: 1 files, 383M in 13s (28.9 MB/s) 2024-03-14T09:26:06.234 INFO:teuthology.task.kernel:Installing kernel 5f16de4b3661fb3d4a041ecb2925baeffc01da4a on client.0... 2024-03-14T09:26:06.234 DEBUG:teuthology.orchestra.run.smithi078:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-14T09:26:08.058 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying... ######################################## 2024-03-14T09:26:08.146 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing... ######################################## 2024-03-14T09:26:08.295 INFO:teuthology.orchestra.run.smithi078.stdout:Updating / installing... 2024-03-14T09:26:48.799 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-6.8.0_rc7_g5f16de4b3661-1 ######################################## 2024-03-14T09:26:52.207 INFO:teuthology.orchestra.run.smithi078.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-14T09:27:12.208 INFO:teuthology.orchestra.run.smithi078.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-14T09:29:24.036 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-14T09:29:24.036 DEBUG:teuthology.orchestra.run.smithi078:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-14T09:29:24.099 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661 2024-03-14T09:29:24.099 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/System.map 2024-03-14T09:29:24.099 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/config 2024-03-14T09:29:24.099 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel 2024-03-14T09:29:24.100 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch 2024-03-14T09:29:24.100 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86 2024-03-14T09:29:24.100 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto 2024-03-14T09:29:24.100 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-14T09:29:24.100 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-14T09:29:24.100 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-14T09:29:24.100 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-14T09:29:24.100 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-14T09:29:24.100 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-14T09:29:24.100 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-14T09:29:24.100 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-14T09:29:24.100 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-14T09:29:24.100 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-14T09:29:24.100 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-14T09:29:24.101 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-14T09:29:24.101 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-14T09:29:24.101 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-14T09:29:24.101 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-14T09:29:24.101 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-14T09:29:24.101 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-14T09:29:24.101 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-14T09:29:24.101 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-14T09:29:24.101 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-14T09:29:24.101 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/kernel 2024-03-14T09:29:24.101 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/kernel/cpu 2024-03-14T09:29:24.101 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/kernel/cpu/mce 2024-03-14T09:29:24.101 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-14T09:29:24.101 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/kvm 2024-03-14T09:29:24.102 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-14T09:29:24.102 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-14T09:29:24.102 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/arch/x86/kvm/kvm.ko 2024-03-14T09:29:24.102 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto 2024-03-14T09:29:24.102 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/ansi_cprng.ko 2024-03-14T09:29:24.102 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/anubis.ko 2024-03-14T09:29:24.102 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/arc4.ko 2024-03-14T09:29:24.102 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/async_tx 2024-03-14T09:29:24.102 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/async_tx/async_memcpy.ko 2024-03-14T09:29:24.102 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/async_tx/async_pq.ko 2024-03-14T09:29:24.102 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-14T09:29:24.102 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/async_tx/async_tx.ko 2024-03-14T09:29:24.102 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/async_tx/async_xor.ko 2024-03-14T09:29:24.103 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/async_tx/raid6test.ko 2024-03-14T09:29:24.103 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/authenc.ko 2024-03-14T09:29:24.103 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/authencesn.ko 2024-03-14T09:29:24.103 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/blake2b_generic.ko 2024-03-14T09:29:24.103 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/blowfish_common.ko 2024-03-14T09:29:24.103 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/blowfish_generic.ko 2024-03-14T09:29:24.103 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/camellia_generic.ko 2024-03-14T09:29:24.103 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/cast5_generic.ko 2024-03-14T09:29:24.103 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/cast6_generic.ko 2024-03-14T09:29:24.103 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/cast_common.ko 2024-03-14T09:29:24.103 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/ccm.ko 2024-03-14T09:29:24.103 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/cmac.ko 2024-03-14T09:29:24.103 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/crc32_generic.ko 2024-03-14T09:29:24.103 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/crypto_engine.ko 2024-03-14T09:29:24.103 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/crypto_user.ko 2024-03-14T09:29:24.104 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/cts.ko 2024-03-14T09:29:24.104 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/des_generic.ko 2024-03-14T09:29:24.104 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/dh_generic.ko 2024-03-14T09:29:24.104 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/echainiv.ko 2024-03-14T09:29:24.104 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/essiv.ko 2024-03-14T09:29:24.104 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/fcrypt.ko 2024-03-14T09:29:24.104 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/khazad.ko 2024-03-14T09:29:24.104 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/lz4.ko 2024-03-14T09:29:24.104 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/lz4hc.ko 2024-03-14T09:29:24.104 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/md4.ko 2024-03-14T09:29:24.104 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/michael_mic.ko 2024-03-14T09:29:24.104 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/pcbc.ko 2024-03-14T09:29:24.104 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/pcrypt.ko 2024-03-14T09:29:24.104 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/rmd160.ko 2024-03-14T09:29:24.105 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/seed.ko 2024-03-14T09:29:24.105 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/serpent_generic.ko 2024-03-14T09:29:24.105 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/tcrypt.ko 2024-03-14T09:29:24.105 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/tea.ko 2024-03-14T09:29:24.105 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/twofish_common.ko 2024-03-14T09:29:24.105 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/twofish_generic.ko 2024-03-14T09:29:24.105 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/vmac.ko 2024-03-14T09:29:24.105 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/wp512.ko 2024-03-14T09:29:24.105 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/xcbc.ko 2024-03-14T09:29:24.105 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/xor.ko 2024-03-14T09:29:24.105 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/crypto/xxhash_generic.ko 2024-03-14T09:29:24.105 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers 2024-03-14T09:29:24.105 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/acpi 2024-03-14T09:29:24.105 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-14T09:29:24.105 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/acpi/acpi_pad.ko 2024-03-14T09:29:24.106 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/acpi/custom_method.ko 2024-03-14T09:29:24.106 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/acpi/ec_sys.ko 2024-03-14T09:29:24.106 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/acpi/platform_profile.ko 2024-03-14T09:29:24.106 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/acpi/sbs.ko 2024-03-14T09:29:24.106 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/acpi/sbshc.ko 2024-03-14T09:29:24.106 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/acpi/video.ko 2024-03-14T09:29:24.106 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata 2024-03-14T09:29:24.106 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/acard-ahci.ko 2024-03-14T09:29:24.106 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/ahci_platform.ko 2024-03-14T09:29:24.106 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/ata_generic.ko 2024-03-14T09:29:24.106 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/libahci_platform.ko 2024-03-14T09:29:24.106 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_acpi.ko 2024-03-14T09:29:24.106 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_ali.ko 2024-03-14T09:29:24.106 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_amd.ko 2024-03-14T09:29:24.107 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_artop.ko 2024-03-14T09:29:24.107 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_atiixp.ko 2024-03-14T09:29:24.107 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_atp867x.ko 2024-03-14T09:29:24.107 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_cmd640.ko 2024-03-14T09:29:24.107 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_cmd64x.ko 2024-03-14T09:29:24.107 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_cypress.ko 2024-03-14T09:29:24.107 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_efar.ko 2024-03-14T09:29:24.107 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_hpt366.ko 2024-03-14T09:29:24.107 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_hpt37x.ko 2024-03-14T09:29:24.107 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-14T09:29:24.107 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-14T09:29:24.107 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_it8213.ko 2024-03-14T09:29:24.107 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_it821x.ko 2024-03-14T09:29:24.107 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_jmicron.ko 2024-03-14T09:29:24.108 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_marvell.ko 2024-03-14T09:29:24.108 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_mpiix.ko 2024-03-14T09:29:24.108 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_netcell.ko 2024-03-14T09:29:24.108 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_ninja32.ko 2024-03-14T09:29:24.108 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_ns87410.ko 2024-03-14T09:29:24.108 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_ns87415.ko 2024-03-14T09:29:24.108 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_oldpiix.ko 2024-03-14T09:29:24.108 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_opti.ko 2024-03-14T09:29:24.108 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_optidma.ko 2024-03-14T09:29:24.108 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_pcmcia.ko 2024-03-14T09:29:24.108 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-14T09:29:24.108 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-14T09:29:24.108 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_piccolo.ko 2024-03-14T09:29:24.108 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_rdc.ko 2024-03-14T09:29:24.109 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_sch.ko 2024-03-14T09:29:24.109 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_serverworks.ko 2024-03-14T09:29:24.109 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_sil680.ko 2024-03-14T09:29:24.109 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_sis.ko 2024-03-14T09:29:24.109 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_sl82c105.ko 2024-03-14T09:29:24.109 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_triflex.ko 2024-03-14T09:29:24.109 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pata_via.ko 2024-03-14T09:29:24.109 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/pdc_adma.ko 2024-03-14T09:29:24.109 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/sata_inic162x.ko 2024-03-14T09:29:24.109 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/sata_mv.ko 2024-03-14T09:29:24.109 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/sata_nv.ko 2024-03-14T09:29:24.109 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/sata_promise.ko 2024-03-14T09:29:24.109 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/sata_qstor.ko 2024-03-14T09:29:24.109 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/sata_sil.ko 2024-03-14T09:29:24.109 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/sata_sil24.ko 2024-03-14T09:29:24.110 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/sata_sis.ko 2024-03-14T09:29:24.110 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/sata_svw.ko 2024-03-14T09:29:24.110 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/sata_sx4.ko 2024-03-14T09:29:24.110 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/sata_uli.ko 2024-03-14T09:29:24.110 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/sata_via.ko 2024-03-14T09:29:24.110 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ata/sata_vsc.ko 2024-03-14T09:29:24.110 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/auxdisplay 2024-03-14T09:29:24.110 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/auxdisplay/ks0108.ko 2024-03-14T09:29:24.110 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/base 2024-03-14T09:29:24.110 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/base/regmap 2024-03-14T09:29:24.110 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-14T09:29:24.110 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-14T09:29:24.110 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/bcma 2024-03-14T09:29:24.110 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/bcma/bcma.ko 2024-03-14T09:29:24.111 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block 2024-03-14T09:29:24.111 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/aoe 2024-03-14T09:29:24.111 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/aoe/aoe.ko 2024-03-14T09:29:24.111 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/brd.ko 2024-03-14T09:29:24.111 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/drbd 2024-03-14T09:29:24.111 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/drbd/drbd.ko 2024-03-14T09:29:24.111 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/floppy.ko 2024-03-14T09:29:24.111 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/loop.ko 2024-03-14T09:29:24.111 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/mtip32xx 2024-03-14T09:29:24.111 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-14T09:29:24.111 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/nbd.ko 2024-03-14T09:29:24.111 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/null_blk 2024-03-14T09:29:24.111 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/null_blk/null_blk.ko 2024-03-14T09:29:24.111 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/pktcdvd.ko 2024-03-14T09:29:24.111 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/rbd.ko 2024-03-14T09:29:24.112 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/virtio_blk.ko 2024-03-14T09:29:24.112 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/xen-blkback 2024-03-14T09:29:24.112 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-14T09:29:24.112 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/xen-blkfront.ko 2024-03-14T09:29:24.112 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/zram 2024-03-14T09:29:24.112 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/block/zram/zram.ko 2024-03-14T09:29:24.112 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char 2024-03-14T09:29:24.112 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/hangcheck-timer.ko 2024-03-14T09:29:24.112 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/hw_random 2024-03-14T09:29:24.112 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-14T09:29:24.112 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-14T09:29:24.112 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-14T09:29:24.112 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/hw_random/via-rng.ko 2024-03-14T09:29:24.112 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-14T09:29:24.112 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/ipmi 2024-03-14T09:29:24.113 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-14T09:29:24.113 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-14T09:29:24.113 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-14T09:29:24.113 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-14T09:29:24.113 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-14T09:29:24.113 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-14T09:29:24.113 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/lp.ko 2024-03-14T09:29:24.113 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/mwave 2024-03-14T09:29:24.113 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/mwave/mwave.ko 2024-03-14T09:29:24.113 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/ppdev.ko 2024-03-14T09:29:24.113 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/tlclk.ko 2024-03-14T09:29:24.113 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/tpm 2024-03-14T09:29:24.113 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/tpm/tpm.ko 2024-03-14T09:29:24.113 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-14T09:29:24.114 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-14T09:29:24.114 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-14T09:29:24.114 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-14T09:29:24.114 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-14T09:29:24.114 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/uv_mmtimer.ko 2024-03-14T09:29:24.114 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/char/virtio_console.ko 2024-03-14T09:29:24.114 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/cpufreq 2024-03-14T09:29:24.114 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-14T09:29:24.114 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-14T09:29:24.114 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-14T09:29:24.114 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-14T09:29:24.114 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-14T09:29:24.115 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-14T09:29:24.115 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/crypto 2024-03-14T09:29:24.115 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/crypto/ccp 2024-03-14T09:29:24.115 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-14T09:29:24.115 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/crypto/ccp/ccp.ko 2024-03-14T09:29:24.115 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/crypto/intel 2024-03-14T09:29:24.115 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/crypto/intel/qat 2024-03-14T09:29:24.115 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/crypto/intel/qat/qat_common 2024-03-14T09:29:24.115 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-14T09:29:24.115 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-14T09:29:24.116 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-14T09:29:24.116 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/crypto/padlock-aes.ko 2024-03-14T09:29:24.116 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/crypto/padlock-sha.ko 2024-03-14T09:29:24.116 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/crypto/virtio 2024-03-14T09:29:24.116 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-14T09:29:24.116 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/dca 2024-03-14T09:29:24.116 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/dca/dca.ko 2024-03-14T09:29:24.116 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/devfreq 2024-03-14T09:29:24.116 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-14T09:29:24.116 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/dma 2024-03-14T09:29:24.117 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/dma/dw 2024-03-14T09:29:24.117 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-14T09:29:24.117 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/dma/ioat 2024-03-14T09:29:24.117 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-14T09:29:24.117 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/edac 2024-03-14T09:29:24.117 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/edac/amd64_edac.ko 2024-03-14T09:29:24.117 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/edac/e752x_edac.ko 2024-03-14T09:29:24.117 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/edac/edac_mce_amd.ko 2024-03-14T09:29:24.117 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/edac/i3000_edac.ko 2024-03-14T09:29:24.117 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/edac/i3200_edac.ko 2024-03-14T09:29:24.117 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/edac/i5100_edac.ko 2024-03-14T09:29:24.117 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/edac/i5400_edac.ko 2024-03-14T09:29:24.117 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/edac/i7300_edac.ko 2024-03-14T09:29:24.117 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/edac/i7core_edac.ko 2024-03-14T09:29:24.118 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/edac/i82975x_edac.ko 2024-03-14T09:29:24.118 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/edac/ie31200_edac.ko 2024-03-14T09:29:24.118 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/edac/sb_edac.ko 2024-03-14T09:29:24.118 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/edac/x38_edac.ko 2024-03-14T09:29:24.118 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/firmware 2024-03-14T09:29:24.118 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/firmware/edd.ko 2024-03-14T09:29:24.118 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-14T09:29:24.118 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/gpio 2024-03-14T09:29:24.118 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/gpio/gpio-ich.ko 2024-03-14T09:29:24.118 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-14T09:29:24.118 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid 2024-03-14T09:29:24.118 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-appleir.ko 2024-03-14T09:29:24.118 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-aureal.ko 2024-03-14T09:29:24.119 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-axff.ko 2024-03-14T09:29:24.119 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-dr.ko 2024-03-14T09:29:24.119 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-elecom.ko 2024-03-14T09:29:24.119 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-elo.ko 2024-03-14T09:29:24.119 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-emsff.ko 2024-03-14T09:29:24.119 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-gaff.ko 2024-03-14T09:29:24.119 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-gt683r.ko 2024-03-14T09:29:24.119 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-gyration.ko 2024-03-14T09:29:24.119 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-14T09:29:24.119 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-14T09:29:24.119 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-holtekff.ko 2024-03-14T09:29:24.119 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-hyperv.ko 2024-03-14T09:29:24.120 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-icade.ko 2024-03-14T09:29:24.120 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-keytouch.ko 2024-03-14T09:29:24.120 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-kye.ko 2024-03-14T09:29:24.120 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-lcpower.ko 2024-03-14T09:29:24.120 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-led.ko 2024-03-14T09:29:24.120 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-lenovo.ko 2024-03-14T09:29:24.120 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-14T09:29:24.120 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-14T09:29:24.120 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-multitouch.ko 2024-03-14T09:29:24.121 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-ortek.ko 2024-03-14T09:29:24.121 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-penmount.ko 2024-03-14T09:29:24.121 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-petalynx.ko 2024-03-14T09:29:24.121 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-picolcd.ko 2024-03-14T09:29:24.121 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-pl.ko 2024-03-14T09:29:24.121 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-plantronics.ko 2024-03-14T09:29:24.121 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-primax.ko 2024-03-14T09:29:24.121 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-rmi.ko 2024-03-14T09:29:24.121 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-14T09:29:24.121 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-roccat-common.ko 2024-03-14T09:29:24.121 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-14T09:29:24.121 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-14T09:29:24.121 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-14T09:29:24.121 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-14T09:29:24.122 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-14T09:29:24.122 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-14T09:29:24.122 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-14T09:29:24.122 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-14T09:29:24.122 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-14T09:29:24.122 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-roccat.ko 2024-03-14T09:29:24.122 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-saitek.ko 2024-03-14T09:29:24.122 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-samsung.ko 2024-03-14T09:29:24.122 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-14T09:29:24.122 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-sjoy.ko 2024-03-14T09:29:24.122 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-sony.ko 2024-03-14T09:29:24.122 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-speedlink.ko 2024-03-14T09:29:24.122 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-steelseries.ko 2024-03-14T09:29:24.123 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-sunplus.ko 2024-03-14T09:29:24.123 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-14T09:29:24.123 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-tivo.ko 2024-03-14T09:29:24.123 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-tmff.ko 2024-03-14T09:29:24.123 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-topseed.ko 2024-03-14T09:29:24.123 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-twinhan.ko 2024-03-14T09:29:24.123 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-uclogic.ko 2024-03-14T09:29:24.123 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-waltop.ko 2024-03-14T09:29:24.123 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-wiimote.ko 2024-03-14T09:29:24.123 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-xinmo.ko 2024-03-14T09:29:24.123 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-zpff.ko 2024-03-14T09:29:24.123 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/hid-zydacron.ko 2024-03-14T09:29:24.123 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/uhid.ko 2024-03-14T09:29:24.124 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hid/wacom.ko 2024-03-14T09:29:24.124 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hv 2024-03-14T09:29:24.124 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hv/hv_balloon.ko 2024-03-14T09:29:24.124 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hv/hv_utils.ko 2024-03-14T09:29:24.124 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hv/hv_vmbus.ko 2024-03-14T09:29:24.124 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon 2024-03-14T09:29:24.124 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/abituguru.ko 2024-03-14T09:29:24.124 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/abituguru3.ko 2024-03-14T09:29:24.124 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-14T09:29:24.124 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/ad7414.ko 2024-03-14T09:29:24.124 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/ad7418.ko 2024-03-14T09:29:24.124 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/adc128d818.ko 2024-03-14T09:29:24.125 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/adm1025.ko 2024-03-14T09:29:24.125 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/adm1026.ko 2024-03-14T09:29:24.125 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/adm1029.ko 2024-03-14T09:29:24.125 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/adm1031.ko 2024-03-14T09:29:24.125 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/adm9240.ko 2024-03-14T09:29:24.125 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/ads7828.ko 2024-03-14T09:29:24.125 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/adt7410.ko 2024-03-14T09:29:24.125 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/adt7411.ko 2024-03-14T09:29:24.125 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/adt7462.ko 2024-03-14T09:29:24.125 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/adt7470.ko 2024-03-14T09:29:24.125 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/adt7475.ko 2024-03-14T09:29:24.125 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/adt7x10.ko 2024-03-14T09:29:24.125 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/amc6821.ko 2024-03-14T09:29:24.125 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/applesmc.ko 2024-03-14T09:29:24.126 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/asb100.ko 2024-03-14T09:29:24.126 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/asc7621.ko 2024-03-14T09:29:24.126 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-14T09:29:24.126 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/atxp1.ko 2024-03-14T09:29:24.126 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/coretemp.ko 2024-03-14T09:29:24.126 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-14T09:29:24.126 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/dme1737.ko 2024-03-14T09:29:24.126 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/ds1621.ko 2024-03-14T09:29:24.126 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/ds620.ko 2024-03-14T09:29:24.126 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/emc1403.ko 2024-03-14T09:29:24.126 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/emc6w201.ko 2024-03-14T09:29:24.126 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/f71805f.ko 2024-03-14T09:29:24.126 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/f71882fg.ko 2024-03-14T09:29:24.126 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/f75375s.ko 2024-03-14T09:29:24.127 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/fam15h_power.ko 2024-03-14T09:29:24.127 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/fschmd.ko 2024-03-14T09:29:24.127 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/g760a.ko 2024-03-14T09:29:24.127 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/g762.ko 2024-03-14T09:29:24.127 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/gl518sm.ko 2024-03-14T09:29:24.127 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/gl520sm.ko 2024-03-14T09:29:24.127 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-14T09:29:24.127 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/i5500_temp.ko 2024-03-14T09:29:24.127 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/i5k_amb.ko 2024-03-14T09:29:24.127 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/ibmaem.ko 2024-03-14T09:29:24.127 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/ibmpex.ko 2024-03-14T09:29:24.127 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/ina209.ko 2024-03-14T09:29:24.127 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/ina2xx.ko 2024-03-14T09:29:24.127 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/it87.ko 2024-03-14T09:29:24.127 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/k10temp.ko 2024-03-14T09:29:24.128 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/k8temp.ko 2024-03-14T09:29:24.128 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/lineage-pem.ko 2024-03-14T09:29:24.128 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/lm63.ko 2024-03-14T09:29:24.128 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/lm73.ko 2024-03-14T09:29:24.128 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/lm75.ko 2024-03-14T09:29:24.128 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/lm77.ko 2024-03-14T09:29:24.128 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/lm78.ko 2024-03-14T09:29:24.128 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/lm80.ko 2024-03-14T09:29:24.128 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/lm83.ko 2024-03-14T09:29:24.128 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/lm85.ko 2024-03-14T09:29:24.128 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/lm87.ko 2024-03-14T09:29:24.128 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/lm90.ko 2024-03-14T09:29:24.128 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/lm92.ko 2024-03-14T09:29:24.128 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/lm93.ko 2024-03-14T09:29:24.129 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/lm95234.ko 2024-03-14T09:29:24.129 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/lm95241.ko 2024-03-14T09:29:24.129 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/lm95245.ko 2024-03-14T09:29:24.129 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/ltc2945.ko 2024-03-14T09:29:24.129 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/ltc4151.ko 2024-03-14T09:29:24.129 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/ltc4215.ko 2024-03-14T09:29:24.129 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/ltc4222.ko 2024-03-14T09:29:24.129 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/ltc4245.ko 2024-03-14T09:29:24.129 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/ltc4260.ko 2024-03-14T09:29:24.129 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/ltc4261.ko 2024-03-14T09:29:24.129 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/max16065.ko 2024-03-14T09:29:24.129 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/max1619.ko 2024-03-14T09:29:24.129 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/max1668.ko 2024-03-14T09:29:24.129 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/max197.ko 2024-03-14T09:29:24.129 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/max6639.ko 2024-03-14T09:29:24.130 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/max6650.ko 2024-03-14T09:29:24.130 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/max6697.ko 2024-03-14T09:29:24.130 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/mcp3021.ko 2024-03-14T09:29:24.130 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/nct6683.ko 2024-03-14T09:29:24.130 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/nct6775-core.ko 2024-03-14T09:29:24.130 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/nct6775.ko 2024-03-14T09:29:24.130 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/nct7802.ko 2024-03-14T09:29:24.130 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/pc87360.ko 2024-03-14T09:29:24.130 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/pc87427.ko 2024-03-14T09:29:24.130 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/pcf8591.ko 2024-03-14T09:29:24.130 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/pmbus 2024-03-14T09:29:24.130 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-14T09:29:24.130 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-14T09:29:24.130 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-14T09:29:24.131 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-14T09:29:24.131 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-14T09:29:24.131 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-14T09:29:24.131 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-14T09:29:24.131 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-14T09:29:24.131 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-14T09:29:24.131 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-14T09:29:24.131 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-14T09:29:24.131 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-14T09:29:24.131 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/powr1220.ko 2024-03-14T09:29:24.131 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/sch5627.ko 2024-03-14T09:29:24.133 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/sch5636.ko 2024-03-14T09:29:24.133 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-14T09:29:24.133 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/sht15.ko 2024-03-14T09:29:24.133 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/sht21.ko 2024-03-14T09:29:24.133 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/shtc1.ko 2024-03-14T09:29:24.133 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/sis5595.ko 2024-03-14T09:29:24.134 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/smsc47b397.ko 2024-03-14T09:29:24.134 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/smsc47m1.ko 2024-03-14T09:29:24.134 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/smsc47m192.ko 2024-03-14T09:29:24.134 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/thmc50.ko 2024-03-14T09:29:24.134 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/tmp102.ko 2024-03-14T09:29:24.134 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/tmp103.ko 2024-03-14T09:29:24.134 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/tmp401.ko 2024-03-14T09:29:24.134 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/tmp421.ko 2024-03-14T09:29:24.134 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/via-cputemp.ko 2024-03-14T09:29:24.134 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/via686a.ko 2024-03-14T09:29:24.134 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/vt1211.ko 2024-03-14T09:29:24.134 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/vt8231.ko 2024-03-14T09:29:24.134 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/w83627ehf.ko 2024-03-14T09:29:24.135 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/w83627hf.ko 2024-03-14T09:29:24.135 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/w83781d.ko 2024-03-14T09:29:24.135 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/w83791d.ko 2024-03-14T09:29:24.135 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/w83792d.ko 2024-03-14T09:29:24.135 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/w83793.ko 2024-03-14T09:29:24.135 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/w83795.ko 2024-03-14T09:29:24.135 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/w83l785ts.ko 2024-03-14T09:29:24.135 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/hwmon/w83l786ng.ko 2024-03-14T09:29:24.135 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c 2024-03-14T09:29:24.135 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/algos 2024-03-14T09:29:24.135 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-14T09:29:24.135 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-14T09:29:24.135 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses 2024-03-14T09:29:24.135 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-14T09:29:24.136 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-14T09:29:24.136 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-14T09:29:24.136 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-14T09:29:24.136 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-14T09:29:24.136 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-14T09:29:24.136 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-14T09:29:24.136 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-14T09:29:24.136 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-14T09:29:24.136 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-14T09:29:24.136 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-14T09:29:24.136 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-14T09:29:24.136 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-14T09:29:24.136 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-14T09:29:24.137 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-14T09:29:24.137 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-14T09:29:24.137 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-14T09:29:24.137 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-14T09:29:24.137 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-14T09:29:24.137 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-14T09:29:24.137 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-14T09:29:24.137 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-14T09:29:24.137 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-14T09:29:24.137 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/i2c-dev.ko 2024-03-14T09:29:24.137 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/i2c-mux.ko 2024-03-14T09:29:24.137 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-14T09:29:24.137 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/i2c-smbus.ko 2024-03-14T09:29:24.137 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/i2c/i2c-stub.ko 2024-03-14T09:29:24.137 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband 2024-03-14T09:29:24.138 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/core 2024-03-14T09:29:24.138 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-14T09:29:24.138 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/core/ib_core.ko 2024-03-14T09:29:24.138 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-14T09:29:24.138 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-14T09:29:24.138 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-14T09:29:24.138 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-14T09:29:24.138 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-14T09:29:24.138 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/hw 2024-03-14T09:29:24.138 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/hw/cxgb4 2024-03-14T09:29:24.138 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-14T09:29:24.138 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/hw/mlx4 2024-03-14T09:29:24.138 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-14T09:29:24.139 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/hw/mlx5 2024-03-14T09:29:24.139 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-14T09:29:24.139 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/hw/mthca 2024-03-14T09:29:24.139 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-14T09:29:24.139 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/ulp 2024-03-14T09:29:24.139 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/ulp/ipoib 2024-03-14T09:29:24.139 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-14T09:29:24.139 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/ulp/iser 2024-03-14T09:29:24.139 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-14T09:29:24.139 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/ulp/isert 2024-03-14T09:29:24.139 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-14T09:29:24.139 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/ulp/srp 2024-03-14T09:29:24.139 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-14T09:29:24.140 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/ulp/srpt 2024-03-14T09:29:24.140 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-14T09:29:24.140 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input 2024-03-14T09:29:24.140 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/gameport 2024-03-14T09:29:24.140 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-14T09:29:24.140 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-14T09:29:24.140 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/gameport/gameport.ko 2024-03-14T09:29:24.140 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/joydev.ko 2024-03-14T09:29:24.140 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/keyboard 2024-03-14T09:29:24.140 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-14T09:29:24.140 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/mouse 2024-03-14T09:29:24.140 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/mouse/appletouch.ko 2024-03-14T09:29:24.140 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/mouse/bcm5974.ko 2024-03-14T09:29:24.140 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/mouse/cyapatp.ko 2024-03-14T09:29:24.141 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-14T09:29:24.141 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/mouse/sermouse.ko 2024-03-14T09:29:24.141 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-14T09:29:24.141 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-14T09:29:24.141 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-14T09:29:24.141 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/rmi4 2024-03-14T09:29:24.141 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-14T09:29:24.141 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/serio 2024-03-14T09:29:24.141 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/serio/altera_ps2.ko 2024-03-14T09:29:24.141 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/serio/arc_ps2.ko 2024-03-14T09:29:24.141 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-14T09:29:24.141 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/serio/serio_raw.ko 2024-03-14T09:29:24.141 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/input/sparse-keymap.ko 2024-03-14T09:29:24.141 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/leds 2024-03-14T09:29:24.141 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/leds/leds-blinkm.ko 2024-03-14T09:29:24.142 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/leds/leds-lm3530.ko 2024-03-14T09:29:24.142 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/leds/leds-lp3944.ko 2024-03-14T09:29:24.142 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/leds/leds-ss4200.ko 2024-03-14T09:29:24.142 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/leds/trigger 2024-03-14T09:29:24.142 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-14T09:29:24.142 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-14T09:29:24.142 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-14T09:29:24.142 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-14T09:29:24.142 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-14T09:29:24.142 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-14T09:29:24.142 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-14T09:29:24.142 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-14T09:29:24.142 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-14T09:29:24.142 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md 2024-03-14T09:29:24.143 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/bcache 2024-03-14T09:29:24.143 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/bcache/bcache.ko 2024-03-14T09:29:24.143 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/dm-bio-prison.ko 2024-03-14T09:29:24.143 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/dm-cache-smq.ko 2024-03-14T09:29:24.143 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/dm-cache.ko 2024-03-14T09:29:24.143 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/dm-crypt.ko 2024-03-14T09:29:24.143 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/dm-delay.ko 2024-03-14T09:29:24.143 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/dm-flakey.ko 2024-03-14T09:29:24.143 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/dm-log-userspace.ko 2024-03-14T09:29:24.143 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/dm-multipath.ko 2024-03-14T09:29:24.143 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/dm-queue-length.ko 2024-03-14T09:29:24.143 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/dm-raid.ko 2024-03-14T09:29:24.143 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/dm-round-robin.ko 2024-03-14T09:29:24.143 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/dm-service-time.ko 2024-03-14T09:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/dm-switch.ko 2024-03-14T09:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/dm-thin-pool.ko 2024-03-14T09:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/dm-verity.ko 2024-03-14T09:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/persistent-data 2024-03-14T09:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-14T09:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/raid0.ko 2024-03-14T09:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/raid1.ko 2024-03-14T09:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/raid10.ko 2024-03-14T09:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/md/raid456.ko 2024-03-14T09:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/message 2024-03-14T09:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/message/fusion 2024-03-14T09:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/message/fusion/mptbase.ko 2024-03-14T09:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/message/fusion/mptctl.ko 2024-03-14T09:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/message/fusion/mptfc.ko 2024-03-14T09:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/message/fusion/mptlan.ko 2024-03-14T09:29:24.145 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/message/fusion/mptsas.ko 2024-03-14T09:29:24.145 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/message/fusion/mptscsih.ko 2024-03-14T09:29:24.145 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/message/fusion/mptspi.ko 2024-03-14T09:29:24.145 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/mfd 2024-03-14T09:29:24.145 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/mfd/lpc_ich.ko 2024-03-14T09:29:24.145 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/mfd/lpc_sch.ko 2024-03-14T09:29:24.145 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/mfd/mfd-core.ko 2024-03-14T09:29:24.145 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/mfd/sm501.ko 2024-03-14T09:29:24.145 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/mfd/viperboard.ko 2024-03-14T09:29:24.145 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/mfd/vx855.ko 2024-03-14T09:29:24.145 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/mfd/wl1273-core.ko 2024-03-14T09:29:24.145 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc 2024-03-14T09:29:24.145 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/altera-stapl 2024-03-14T09:29:24.145 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-14T09:29:24.146 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/apds9802als.ko 2024-03-14T09:29:24.146 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/apds990x.ko 2024-03-14T09:29:24.146 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/bh1770glc.ko 2024-03-14T09:29:24.146 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/cb710 2024-03-14T09:29:24.146 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/cb710/cb710.ko 2024-03-14T09:29:24.146 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/eeprom 2024-03-14T09:29:24.146 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/eeprom/at24.ko 2024-03-14T09:29:24.146 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-14T09:29:24.146 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/eeprom/max6875.ko 2024-03-14T09:29:24.146 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/enclosure.ko 2024-03-14T09:29:24.146 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/hpilo.ko 2024-03-14T09:29:24.146 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/isl29003.ko 2024-03-14T09:29:24.146 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/isl29020.ko 2024-03-14T09:29:24.146 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/lis3lv02d 2024-03-14T09:29:24.146 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-14T09:29:24.147 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-14T09:29:24.147 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/mei 2024-03-14T09:29:24.147 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/mei/mei-me.ko 2024-03-14T09:29:24.147 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/mei/mei-txe.ko 2024-03-14T09:29:24.147 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/mei/mei.ko 2024-03-14T09:29:24.147 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/sgi-gru 2024-03-14T09:29:24.147 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-14T09:29:24.147 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/sgi-xp 2024-03-14T09:29:24.147 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-14T09:29:24.147 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-14T09:29:24.147 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-14T09:29:24.147 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/tifm_7xx1.ko 2024-03-14T09:29:24.147 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/tifm_core.ko 2024-03-14T09:29:24.147 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/tsl2550.ko 2024-03-14T09:29:24.148 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/vmw_balloon.ko 2024-03-14T09:29:24.148 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/vmw_vmci 2024-03-14T09:29:24.148 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-14T09:29:24.148 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net 2024-03-14T09:29:24.148 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/bonding 2024-03-14T09:29:24.148 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/bonding/bonding.ko 2024-03-14T09:29:24.148 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/dsa 2024-03-14T09:29:24.148 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/dsa/b53 2024-03-14T09:29:24.148 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-14T09:29:24.148 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-14T09:29:24.148 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-14T09:29:24.148 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/dsa/mv88e6xxx 2024-03-14T09:29:24.148 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-14T09:29:24.148 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/dummy.ko 2024-03-14T09:29:24.148 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/eql.ko 2024-03-14T09:29:24.149 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet 2024-03-14T09:29:24.149 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/3com 2024-03-14T09:29:24.149 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-14T09:29:24.149 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-14T09:29:24.149 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-14T09:29:24.149 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-14T09:29:24.149 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/8390 2024-03-14T09:29:24.149 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-14T09:29:24.149 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-14T09:29:24.149 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-14T09:29:24.149 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-14T09:29:24.149 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/adaptec 2024-03-14T09:29:24.149 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-14T09:29:24.149 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/agere 2024-03-14T09:29:24.150 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-14T09:29:24.150 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/alteon 2024-03-14T09:29:24.150 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-14T09:29:24.150 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/altera 2024-03-14T09:29:24.150 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-14T09:29:24.150 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/amd 2024-03-14T09:29:24.150 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-14T09:29:24.150 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-14T09:29:24.150 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-14T09:29:24.150 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/atheros 2024-03-14T09:29:24.150 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/atheros/alx 2024-03-14T09:29:24.150 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-14T09:29:24.150 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-14T09:29:24.150 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-14T09:29:24.151 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-14T09:29:24.151 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-14T09:29:24.151 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/atheros/atlx 2024-03-14T09:29:24.151 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-14T09:29:24.151 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-14T09:29:24.151 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/broadcom 2024-03-14T09:29:24.151 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-14T09:29:24.151 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-14T09:29:24.151 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-14T09:29:24.151 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-14T09:29:24.151 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-14T09:29:24.151 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/broadcom/genet 2024-03-14T09:29:24.151 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-14T09:29:24.151 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-14T09:29:24.151 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/brocade 2024-03-14T09:29:24.152 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/brocade/bna 2024-03-14T09:29:24.152 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-14T09:29:24.152 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/chelsio 2024-03-14T09:29:24.152 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-14T09:29:24.152 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-14T09:29:24.152 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-14T09:29:24.152 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-14T09:29:24.152 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-14T09:29:24.152 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-14T09:29:24.152 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-14T09:29:24.152 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-14T09:29:24.152 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-14T09:29:24.152 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-14T09:29:24.152 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/cisco 2024-03-14T09:29:24.152 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/cisco/enic 2024-03-14T09:29:24.153 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-14T09:29:24.153 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/dec 2024-03-14T09:29:24.153 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/dec/tulip 2024-03-14T09:29:24.153 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-14T09:29:24.153 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-14T09:29:24.153 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-14T09:29:24.153 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-14T09:29:24.153 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-14T09:29:24.153 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-14T09:29:24.153 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/dlink 2024-03-14T09:29:24.153 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-14T09:29:24.153 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-14T09:29:24.153 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/dnet.ko 2024-03-14T09:29:24.153 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/emulex 2024-03-14T09:29:24.153 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/emulex/benet 2024-03-14T09:29:24.154 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-14T09:29:24.154 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/ethoc.ko 2024-03-14T09:29:24.154 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/fealnx.ko 2024-03-14T09:29:24.154 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel 2024-03-14T09:29:24.154 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-14T09:29:24.154 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/e1000 2024-03-14T09:29:24.154 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-14T09:29:24.154 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/e1000e 2024-03-14T09:29:24.154 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-14T09:29:24.154 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/fm10k 2024-03-14T09:29:24.154 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-14T09:29:24.154 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/i40e 2024-03-14T09:29:24.154 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-14T09:29:24.154 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/iavf 2024-03-14T09:29:24.155 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-14T09:29:24.155 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/igb 2024-03-14T09:29:24.155 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-14T09:29:24.155 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/igbvf 2024-03-14T09:29:24.155 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-14T09:29:24.155 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-14T09:29:24.155 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-14T09:29:24.155 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-14T09:29:24.155 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-14T09:29:24.155 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/jme.ko 2024-03-14T09:29:24.155 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/marvell 2024-03-14T09:29:24.155 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-14T09:29:24.155 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-14T09:29:24.155 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-14T09:29:24.155 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/mellanox 2024-03-14T09:29:24.156 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-14T09:29:24.156 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-14T09:29:24.156 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-14T09:29:24.156 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-14T09:29:24.156 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-14T09:29:24.156 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-14T09:29:24.156 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/micrel 2024-03-14T09:29:24.156 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-14T09:29:24.156 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/myricom 2024-03-14T09:29:24.156 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-14T09:29:24.156 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-14T09:29:24.156 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/natsemi 2024-03-14T09:29:24.156 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-14T09:29:24.156 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-14T09:29:24.156 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/neterion 2024-03-14T09:29:24.157 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-14T09:29:24.157 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/nvidia 2024-03-14T09:29:24.157 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-14T09:29:24.157 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/packetengines 2024-03-14T09:29:24.157 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-14T09:29:24.157 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-14T09:29:24.157 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/qlogic 2024-03-14T09:29:24.157 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-14T09:29:24.157 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-14T09:29:24.157 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-14T09:29:24.157 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-14T09:29:24.157 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-14T09:29:24.157 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/rdc 2024-03-14T09:29:24.157 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-14T09:29:24.157 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/realtek 2024-03-14T09:29:24.158 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-14T09:29:24.158 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-14T09:29:24.158 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-14T09:29:24.158 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-14T09:29:24.158 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/rocker 2024-03-14T09:29:24.158 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-14T09:29:24.158 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/sfc 2024-03-14T09:29:24.158 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-14T09:29:24.158 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/silan 2024-03-14T09:29:24.158 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-14T09:29:24.158 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/sis 2024-03-14T09:29:24.158 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-14T09:29:24.158 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-14T09:29:24.158 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/smsc 2024-03-14T09:29:24.159 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-14T09:29:24.159 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-14T09:29:24.159 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-14T09:29:24.159 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-14T09:29:24.159 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/stmicro 2024-03-14T09:29:24.159 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-14T09:29:24.159 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-14T09:29:24.159 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-14T09:29:24.159 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/sun 2024-03-14T09:29:24.159 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-14T09:29:24.159 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-14T09:29:24.159 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-14T09:29:24.159 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-14T09:29:24.159 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/tehuti 2024-03-14T09:29:24.160 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-14T09:29:24.160 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/ti 2024-03-14T09:29:24.160 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-14T09:29:24.160 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/via 2024-03-14T09:29:24.160 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-14T09:29:24.160 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-14T09:29:24.160 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/wiznet 2024-03-14T09:29:24.160 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-14T09:29:24.160 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-14T09:29:24.160 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/xircom 2024-03-14T09:29:24.160 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-14T09:29:24.160 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/geneve.ko 2024-03-14T09:29:24.160 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/hyperv 2024-03-14T09:29:24.160 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-14T09:29:24.160 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ieee802154 2024-03-14T09:29:24.161 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-14T09:29:24.161 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ifb.ko 2024-03-14T09:29:24.161 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ipvlan 2024-03-14T09:29:24.161 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-14T09:29:24.161 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/macvlan.ko 2024-03-14T09:29:24.161 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/macvtap.ko 2024-03-14T09:29:24.161 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/mdio 2024-03-14T09:29:24.161 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/mdio.ko 2024-03-14T09:29:24.161 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-14T09:29:24.161 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-14T09:29:24.161 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-14T09:29:24.161 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/mii.ko 2024-03-14T09:29:24.161 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/net_failover.ko 2024-03-14T09:29:24.161 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/netconsole.ko 2024-03-14T09:29:24.161 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/nlmon.ko 2024-03-14T09:29:24.162 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/pcs 2024-03-14T09:29:24.162 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-14T09:29:24.162 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-14T09:29:24.162 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy 2024-03-14T09:29:24.162 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/amd.ko 2024-03-14T09:29:24.162 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-14T09:29:24.162 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-14T09:29:24.162 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-14T09:29:24.162 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/bcm87xx.ko 2024-03-14T09:29:24.162 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/broadcom.ko 2024-03-14T09:29:24.162 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/cicada.ko 2024-03-14T09:29:24.162 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/davicom.ko 2024-03-14T09:29:24.162 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/dp83640.ko 2024-03-14T09:29:24.162 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/et1011c.ko 2024-03-14T09:29:24.163 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/icplus.ko 2024-03-14T09:29:24.163 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/lxt.ko 2024-03-14T09:29:24.163 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/marvell.ko 2024-03-14T09:29:24.163 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/micrel.ko 2024-03-14T09:29:24.163 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/national.ko 2024-03-14T09:29:24.163 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/phylink.ko 2024-03-14T09:29:24.163 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/qsemi.ko 2024-03-14T09:29:24.163 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/realtek.ko 2024-03-14T09:29:24.163 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/smsc.ko 2024-03-14T09:29:24.163 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/ste10Xp.ko 2024-03-14T09:29:24.163 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/phy/vitesse.ko 2024-03-14T09:29:24.163 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ppp 2024-03-14T09:29:24.163 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-14T09:29:24.163 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ppp/ppp_async.ko 2024-03-14T09:29:24.163 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-14T09:29:24.164 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-14T09:29:24.164 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-14T09:29:24.164 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-14T09:29:24.164 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ppp/pppoe.ko 2024-03-14T09:29:24.164 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ppp/pppox.ko 2024-03-14T09:29:24.164 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/ppp/pptp.ko 2024-03-14T09:29:24.164 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/slip 2024-03-14T09:29:24.164 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/slip/slhc.ko 2024-03-14T09:29:24.164 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/slip/slip.ko 2024-03-14T09:29:24.164 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/sungem_phy.ko 2024-03-14T09:29:24.164 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/tap.ko 2024-03-14T09:29:24.164 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/team 2024-03-14T09:29:24.164 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/team/team.ko 2024-03-14T09:29:24.164 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-14T09:29:24.164 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-14T09:29:24.165 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-14T09:29:24.165 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/team/team_mode_random.ko 2024-03-14T09:29:24.165 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-14T09:29:24.165 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/tun.ko 2024-03-14T09:29:24.165 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/veth.ko 2024-03-14T09:29:24.165 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/virtio_net.ko 2024-03-14T09:29:24.165 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/vmxnet3 2024-03-14T09:29:24.165 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-14T09:29:24.165 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/vxlan 2024-03-14T09:29:24.165 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/vxlan/vxlan.ko 2024-03-14T09:29:24.165 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/xen-netback 2024-03-14T09:29:24.165 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-14T09:29:24.165 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/net/xen-netfront.ko 2024-03-14T09:29:24.165 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ntb 2024-03-14T09:29:24.165 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ntb/ntb.ko 2024-03-14T09:29:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/nvme 2024-03-14T09:29:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/nvme/host 2024-03-14T09:29:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/nvme/host/nvme-core.ko 2024-03-14T09:29:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/nvme/host/nvme.ko 2024-03-14T09:29:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/parport 2024-03-14T09:29:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/parport/parport.ko 2024-03-14T09:29:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/parport/parport_cs.ko 2024-03-14T09:29:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/parport/parport_pc.ko 2024-03-14T09:29:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/parport/parport_serial.ko 2024-03-14T09:29:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pci 2024-03-14T09:29:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pci/hotplug 2024-03-14T09:29:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-14T09:29:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pci/pcie 2024-03-14T09:29:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-14T09:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pci/xen-pcifront.ko 2024-03-14T09:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pcmcia 2024-03-14T09:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pcmcia/i82092.ko 2024-03-14T09:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pcmcia/pd6729.ko 2024-03-14T09:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-14T09:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pinctrl 2024-03-14T09:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pinctrl/intel 2024-03-14T09:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-14T09:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform 2024-03-14T09:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86 2024-03-14T09:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-14T09:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/acerhdf.ko 2024-03-14T09:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-14T09:29:24.167 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-14T09:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-14T09:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-14T09:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-14T09:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-14T09:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-14T09:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-14T09:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-14T09:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/hdaps.ko 2024-03-14T09:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/intel 2024-03-14T09:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-14T09:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/intel_ips.ko 2024-03-14T09:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-14T09:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-14T09:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-14T09:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-14T09:29:24.168 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-14T09:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-14T09:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-14T09:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-14T09:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-14T09:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/platform/x86/wmi.ko 2024-03-14T09:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/powercap 2024-03-14T09:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-14T09:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-14T09:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pps 2024-03-14T09:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pps/clients 2024-03-14T09:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-14T09:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-14T09:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pps/clients/pps_parport.ko 2024-03-14T09:29:24.169 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/pps/pps_core.ko 2024-03-14T09:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ptp 2024-03-14T09:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ptp/ptp.ko 2024-03-14T09:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ptp/ptp_kvm.ko 2024-03-14T09:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc 2024-03-14T09:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-14T09:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-14T09:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-14T09:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-14T09:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-14T09:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-14T09:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-14T09:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-14T09:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-14T09:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-14T09:29:24.170 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-em3027.ko 2024-03-14T09:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-14T09:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-14T09:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-14T09:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-14T09:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-14T09:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-14T09:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-max6900.ko 2024-03-14T09:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-14T09:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-14T09:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-14T09:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-14T09:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-14T09:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-14T09:29:24.171 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-14T09:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-14T09:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-14T09:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-14T09:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-14T09:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-14T09:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/rtc/rtc-x1205.ko 2024-03-14T09:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi 2024-03-14T09:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/3w-9xxx.ko 2024-03-14T09:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/3w-sas.ko 2024-03-14T09:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/3w-xxxx.ko 2024-03-14T09:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/BusLogic.ko 2024-03-14T09:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/a100u2w.ko 2024-03-14T09:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/aacraid 2024-03-14T09:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-14T09:29:24.172 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/advansys.ko 2024-03-14T09:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/aic7xxx 2024-03-14T09:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-14T09:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-14T09:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/am53c974.ko 2024-03-14T09:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/arcmsr 2024-03-14T09:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-14T09:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/atp870u.ko 2024-03-14T09:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/be2iscsi 2024-03-14T09:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-14T09:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/bfa 2024-03-14T09:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/bfa/bfa.ko 2024-03-14T09:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/bnx2fc 2024-03-14T09:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-14T09:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/bnx2i 2024-03-14T09:29:24.173 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-14T09:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/ch.ko 2024-03-14T09:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/csiostor 2024-03-14T09:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-14T09:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/cxgbi 2024-03-14T09:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-14T09:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-14T09:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-14T09:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-14T09:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-14T09:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/dc395x.ko 2024-03-14T09:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/device_handler 2024-03-14T09:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-14T09:29:24.174 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-14T09:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-14T09:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-14T09:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/dmx3191d.ko 2024-03-14T09:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/esas2r 2024-03-14T09:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-14T09:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/esp_scsi.ko 2024-03-14T09:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/fcoe 2024-03-14T09:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-14T09:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-14T09:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/fnic 2024-03-14T09:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/fnic/fnic.ko 2024-03-14T09:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/hpsa.ko 2024-03-14T09:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/hptiop.ko 2024-03-14T09:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/hv_storvsc.ko 2024-03-14T09:29:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/initio.ko 2024-03-14T09:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/ipr.ko 2024-03-14T09:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/ips.ko 2024-03-14T09:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/isci 2024-03-14T09:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/isci/isci.ko 2024-03-14T09:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-14T09:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-14T09:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/libfc 2024-03-14T09:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/libfc/libfc.ko 2024-03-14T09:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/libiscsi.ko 2024-03-14T09:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-14T09:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/libsas 2024-03-14T09:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/libsas/libsas.ko 2024-03-14T09:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/lpfc 2024-03-14T09:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-14T09:29:24.176 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/megaraid 2024-03-14T09:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/megaraid.ko 2024-03-14T09:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-14T09:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-14T09:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-14T09:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/mpt3sas 2024-03-14T09:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-14T09:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/mvsas 2024-03-14T09:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-14T09:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/mvumi.ko 2024-03-14T09:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/pm8001 2024-03-14T09:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-14T09:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/pmcraid.ko 2024-03-14T09:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/qla1280.ko 2024-03-14T09:29:24.177 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/qla2xxx 2024-03-14T09:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-14T09:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-14T09:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/qla4xxx 2024-03-14T09:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-14T09:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/raid_class.ko 2024-03-14T09:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/scsi_debug.ko 2024-03-14T09:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-14T09:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-14T09:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-14T09:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-14T09:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-14T09:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/ses.ko 2024-03-14T09:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/st.ko 2024-03-14T09:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/stex.ko 2024-03-14T09:29:24.178 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/sym53c8xx_2 2024-03-14T09:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-14T09:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/virtio_scsi.ko 2024-03-14T09:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-14T09:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/scsi/wd719x.ko 2024-03-14T09:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ssb 2024-03-14T09:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ssb/ssb.ko 2024-03-14T09:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/target 2024-03-14T09:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/target/iscsi 2024-03-14T09:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-14T09:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/target/loopback 2024-03-14T09:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-14T09:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/target/target_core_file.ko 2024-03-14T09:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/target/target_core_iblock.ko 2024-03-14T09:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/target/target_core_mod.ko 2024-03-14T09:29:24.179 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/target/target_core_pscsi.ko 2024-03-14T09:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/target/target_core_user.ko 2024-03-14T09:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/target/tcm_fc 2024-03-14T09:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-14T09:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal 2024-03-14T09:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel 2024-03-14T09:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/int340x_thermal 2024-03-14T09:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-14T09:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-14T09:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-14T09:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-14T09:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-14T09:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-14T09:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-14T09:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-14T09:29:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-14T09:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-14T09:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-14T09:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-14T09:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-14T09:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-14T09:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-14T09:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-14T09:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-14T09:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-14T09:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/tty 2024-03-14T09:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/tty/ipwireless 2024-03-14T09:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-14T09:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/tty/n_gsm.ko 2024-03-14T09:29:24.181 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/tty/n_hdlc.ko 2024-03-14T09:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/tty/nozomi.ko 2024-03-14T09:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/tty/serial 2024-03-14T09:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/tty/serial/8250 2024-03-14T09:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-14T09:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/tty/serial/arc_uart.ko 2024-03-14T09:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/tty/serial/jsm 2024-03-14T09:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-14T09:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ufs 2024-03-14T09:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ufs/core 2024-03-14T09:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-14T09:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ufs/host 2024-03-14T09:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-14T09:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/uio 2024-03-14T09:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/uio/uio.ko 2024-03-14T09:29:24.182 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/uio/uio_aec.ko 2024-03-14T09:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/uio/uio_cif.ko 2024-03-14T09:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/uio/uio_pci_generic.ko 2024-03-14T09:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/uio/uio_sercos3.ko 2024-03-14T09:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb 2024-03-14T09:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/class 2024-03-14T09:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/class/cdc-acm.ko 2024-03-14T09:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-14T09:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/class/usblp.ko 2024-03-14T09:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/class/usbtmc.ko 2024-03-14T09:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/host 2024-03-14T09:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-14T09:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/image 2024-03-14T09:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/image/mdc800.ko 2024-03-14T09:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/image/microtek.ko 2024-03-14T09:29:24.183 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc 2024-03-14T09:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/adutux.ko 2024-03-14T09:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/appledisplay.ko 2024-03-14T09:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/emi26.ko 2024-03-14T09:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/emi62.ko 2024-03-14T09:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/ezusb.ko 2024-03-14T09:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/idmouse.ko 2024-03-14T09:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/iowarrior.ko 2024-03-14T09:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-14T09:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/ldusb.ko 2024-03-14T09:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/legousbtower.ko 2024-03-14T09:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/sisusbvga 2024-03-14T09:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-14T09:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-14T09:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/usb3503.ko 2024-03-14T09:29:24.184 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/usblcd.ko 2024-03-14T09:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-14T09:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/uss720.ko 2024-03-14T09:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/misc/yurex.ko 2024-03-14T09:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/phy 2024-03-14T09:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/phy/phy-generic.ko 2024-03-14T09:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial 2024-03-14T09:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/aircable.ko 2024-03-14T09:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/ark3116.ko 2024-03-14T09:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-14T09:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/ch341.ko 2024-03-14T09:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/cp210x.ko 2024-03-14T09:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/cyberjack.ko 2024-03-14T09:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-14T09:29:24.185 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-14T09:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/empeg.ko 2024-03-14T09:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-14T09:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-14T09:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-14T09:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/io_ti.ko 2024-03-14T09:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/ipaq.ko 2024-03-14T09:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/ipw.ko 2024-03-14T09:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/ir-usb.ko 2024-03-14T09:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-14T09:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/keyspan.ko 2024-03-14T09:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-14T09:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-14T09:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-14T09:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/mct_u232.ko 2024-03-14T09:29:24.186 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/mos7720.ko 2024-03-14T09:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/mos7840.ko 2024-03-14T09:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/navman.ko 2024-03-14T09:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/omninet.ko 2024-03-14T09:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/opticon.ko 2024-03-14T09:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/option.ko 2024-03-14T09:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/oti6858.ko 2024-03-14T09:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/pl2303.ko 2024-03-14T09:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/qcaux.ko 2024-03-14T09:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/qcserial.ko 2024-03-14T09:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/quatech2.ko 2024-03-14T09:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/safe_serial.ko 2024-03-14T09:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/sierra.ko 2024-03-14T09:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-14T09:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/ssu100.ko 2024-03-14T09:29:24.187 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/symbolserial.ko 2024-03-14T09:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-14T09:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-14T09:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/usb_debug.ko 2024-03-14T09:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-14T09:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/visor.ko 2024-03-14T09:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/whiteheat.ko 2024-03-14T09:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-14T09:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/storage 2024-03-14T09:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/storage/uas.ko 2024-03-14T09:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-14T09:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-14T09:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-14T09:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-14T09:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-14T09:29:24.188 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-14T09:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-14T09:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/storage/ums-karma.ko 2024-03-14T09:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-14T09:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-14T09:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-14T09:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-14T09:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-14T09:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/storage/usb-storage.ko 2024-03-14T09:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/usbip 2024-03-14T09:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-14T09:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-14T09:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-14T09:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/vfio 2024-03-14T09:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/vfio/pci 2024-03-14T09:29:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-14T09:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-14T09:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/vfio/vfio.ko 2024-03-14T09:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-14T09:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/vhost 2024-03-14T09:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/vhost/vhost.ko 2024-03-14T09:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-14T09:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/vhost/vhost_net.ko 2024-03-14T09:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/vhost/vhost_scsi.ko 2024-03-14T09:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/video 2024-03-14T09:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/video/backlight 2024-03-14T09:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/video/backlight/apple_bl.ko 2024-03-14T09:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/video/backlight/lcd.ko 2024-03-14T09:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-14T09:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/virtio 2024-03-14T09:29:24.190 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/virtio/virtio.ko 2024-03-14T09:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/virtio/virtio_balloon.ko 2024-03-14T09:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/virtio/virtio_mmio.ko 2024-03-14T09:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/virtio/virtio_pci.ko 2024-03-14T09:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-14T09:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-14T09:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/virtio/virtio_ring.ko 2024-03-14T09:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1 2024-03-14T09:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1/masters 2024-03-14T09:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1/masters/ds2482.ko 2024-03-14T09:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1/masters/ds2490.ko 2024-03-14T09:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1/slaves 2024-03-14T09:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-14T09:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-14T09:29:24.191 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-14T09:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-14T09:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-14T09:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-14T09:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-14T09:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-14T09:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-14T09:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-14T09:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-14T09:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/w1/wire.ko 2024-03-14T09:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog 2024-03-14T09:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-14T09:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-14T09:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-14T09:29:24.192 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/hpwdt.ko 2024-03-14T09:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/i6300esb.ko 2024-03-14T09:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-14T09:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-14T09:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/ib700wdt.ko 2024-03-14T09:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/ibmasr.ko 2024-03-14T09:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-14T09:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-14T09:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/it87_wdt.ko 2024-03-14T09:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/machzwd.ko 2024-03-14T09:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/nv_tco.ko 2024-03-14T09:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-14T09:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-14T09:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-14T09:29:24.193 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-14T09:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/softdog.ko 2024-03-14T09:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-14T09:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/via_wdt.ko 2024-03-14T09:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-14T09:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-14T09:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-14T09:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/wdt_pci.ko 2024-03-14T09:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/watchdog/xen_wdt.ko 2024-03-14T09:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/xen 2024-03-14T09:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-14T09:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/xen/xen-evtchn.ko 2024-03-14T09:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/xen/xen-gntalloc.ko 2024-03-14T09:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/xen/xen-gntdev.ko 2024-03-14T09:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/xen/xen-pciback 2024-03-14T09:29:24.194 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-14T09:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/xen/xen-privcmd.ko 2024-03-14T09:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/xen/xenfs 2024-03-14T09:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-14T09:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs 2024-03-14T09:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/9p 2024-03-14T09:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/9p/9p.ko 2024-03-14T09:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/affs 2024-03-14T09:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/affs/affs.ko 2024-03-14T09:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/befs 2024-03-14T09:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/befs/befs.ko 2024-03-14T09:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/binfmt_misc.ko 2024-03-14T09:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/btrfs 2024-03-14T09:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/btrfs/btrfs.ko 2024-03-14T09:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/cachefiles 2024-03-14T09:29:24.195 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/cachefiles/cachefiles.ko 2024-03-14T09:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ceph 2024-03-14T09:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ceph/ceph.ko 2024-03-14T09:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/coda 2024-03-14T09:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/coda/coda.ko 2024-03-14T09:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/cramfs 2024-03-14T09:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/cramfs/cramfs.ko 2024-03-14T09:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/dlm 2024-03-14T09:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/dlm/dlm.ko 2024-03-14T09:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ecryptfs 2024-03-14T09:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-14T09:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/f2fs 2024-03-14T09:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/f2fs/f2fs.ko 2024-03-14T09:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/fat 2024-03-14T09:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/fat/fat.ko 2024-03-14T09:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/fat/msdos.ko 2024-03-14T09:29:24.196 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/fat/vfat.ko 2024-03-14T09:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/fuse 2024-03-14T09:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/fuse/cuse.ko 2024-03-14T09:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/fuse/fuse.ko 2024-03-14T09:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/gfs2 2024-03-14T09:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/gfs2/gfs2.ko 2024-03-14T09:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/hfs 2024-03-14T09:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/hfs/hfs.ko 2024-03-14T09:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/hfsplus 2024-03-14T09:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/hfsplus/hfsplus.ko 2024-03-14T09:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/isofs 2024-03-14T09:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/isofs/isofs.ko 2024-03-14T09:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/jfs 2024-03-14T09:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/jfs/jfs.ko 2024-03-14T09:29:24.197 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/lockd 2024-03-14T09:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/lockd/lockd.ko 2024-03-14T09:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/minix 2024-03-14T09:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/minix/minix.ko 2024-03-14T09:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/netfs 2024-03-14T09:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/netfs/netfs.ko 2024-03-14T09:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nfs 2024-03-14T09:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nfs/blocklayout 2024-03-14T09:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-14T09:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nfs/filelayout 2024-03-14T09:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-14T09:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nfs/flexfilelayout 2024-03-14T09:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-14T09:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nfs/nfs.ko 2024-03-14T09:29:24.198 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nfs/nfsv3.ko 2024-03-14T09:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nfs/nfsv4.ko 2024-03-14T09:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nfs_common 2024-03-14T09:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nfs_common/grace.ko 2024-03-14T09:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nfs_common/nfs_acl.ko 2024-03-14T09:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nfsd 2024-03-14T09:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nfsd/nfsd.ko 2024-03-14T09:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nilfs2 2024-03-14T09:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nilfs2/nilfs2.ko 2024-03-14T09:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls 2024-03-14T09:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/mac-celtic.ko 2024-03-14T09:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/mac-centeuro.ko 2024-03-14T09:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/mac-croatian.ko 2024-03-14T09:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/mac-cyrillic.ko 2024-03-14T09:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/mac-gaelic.ko 2024-03-14T09:29:24.199 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/mac-greek.ko 2024-03-14T09:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/mac-iceland.ko 2024-03-14T09:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/mac-inuit.ko 2024-03-14T09:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/mac-roman.ko 2024-03-14T09:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/mac-romanian.ko 2024-03-14T09:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/mac-turkish.ko 2024-03-14T09:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp1250.ko 2024-03-14T09:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp1251.ko 2024-03-14T09:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp1255.ko 2024-03-14T09:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp737.ko 2024-03-14T09:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp775.ko 2024-03-14T09:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp850.ko 2024-03-14T09:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp852.ko 2024-03-14T09:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp855.ko 2024-03-14T09:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp857.ko 2024-03-14T09:29:24.200 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp860.ko 2024-03-14T09:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp861.ko 2024-03-14T09:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp862.ko 2024-03-14T09:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp863.ko 2024-03-14T09:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp864.ko 2024-03-14T09:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp865.ko 2024-03-14T09:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp866.ko 2024-03-14T09:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp869.ko 2024-03-14T09:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp874.ko 2024-03-14T09:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp932.ko 2024-03-14T09:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp936.ko 2024-03-14T09:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp949.ko 2024-03-14T09:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_cp950.ko 2024-03-14T09:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_euc-jp.ko 2024-03-14T09:29:24.201 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_iso8859-1.ko 2024-03-14T09:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_iso8859-13.ko 2024-03-14T09:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_iso8859-14.ko 2024-03-14T09:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_iso8859-15.ko 2024-03-14T09:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_iso8859-2.ko 2024-03-14T09:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_iso8859-3.ko 2024-03-14T09:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_iso8859-4.ko 2024-03-14T09:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_iso8859-5.ko 2024-03-14T09:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_iso8859-6.ko 2024-03-14T09:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_iso8859-7.ko 2024-03-14T09:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_iso8859-9.ko 2024-03-14T09:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_koi8-r.ko 2024-03-14T09:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_koi8-ru.ko 2024-03-14T09:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_koi8-u.ko 2024-03-14T09:29:24.202 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-14T09:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/nls/nls_utf8.ko 2024-03-14T09:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ocfs2 2024-03-14T09:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ocfs2/cluster 2024-03-14T09:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-14T09:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ocfs2/dlm 2024-03-14T09:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-14T09:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ocfs2/dlmfs 2024-03-14T09:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-14T09:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ocfs2/ocfs2.ko 2024-03-14T09:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-14T09:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-14T09:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-14T09:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/overlayfs 2024-03-14T09:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/overlayfs/overlay.ko 2024-03-14T09:29:24.203 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/pstore 2024-03-14T09:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/pstore/ramoops.ko 2024-03-14T09:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/reiserfs 2024-03-14T09:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/reiserfs/reiserfs.ko 2024-03-14T09:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/romfs 2024-03-14T09:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/romfs/romfs.ko 2024-03-14T09:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/smb 2024-03-14T09:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/smb/client 2024-03-14T09:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/smb/client/cifs.ko 2024-03-14T09:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/smb/common 2024-03-14T09:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/smb/common/cifs_arc4.ko 2024-03-14T09:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/smb/common/cifs_md4.ko 2024-03-14T09:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/squashfs 2024-03-14T09:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/squashfs/squashfs.ko 2024-03-14T09:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/sysv 2024-03-14T09:29:24.204 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/sysv/sysv.ko 2024-03-14T09:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/udf 2024-03-14T09:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/udf/udf.ko 2024-03-14T09:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ufs 2024-03-14T09:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/ufs/ufs.ko 2024-03-14T09:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/xfs 2024-03-14T09:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/fs/xfs/xfs.ko 2024-03-14T09:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/kernel 2024-03-14T09:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/kernel/rcu 2024-03-14T09:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/kernel/rcu/rcutorture.ko 2024-03-14T09:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/kernel/torture.ko 2024-03-14T09:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/kernel/trace 2024-03-14T09:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-14T09:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib 2024-03-14T09:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/asn1_encoder.ko 2024-03-14T09:29:24.205 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/crc-itu-t.ko 2024-03-14T09:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/crc8.ko 2024-03-14T09:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/crypto 2024-03-14T09:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/crypto/libarc4.ko 2024-03-14T09:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/crypto/libdes.ko 2024-03-14T09:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/libcrc32c.ko 2024-03-14T09:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/lru_cache.ko 2024-03-14T09:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/lz4 2024-03-14T09:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/lz4/lz4_compress.ko 2024-03-14T09:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/lz4/lz4hc_compress.ko 2024-03-14T09:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/math 2024-03-14T09:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/math/cordic.ko 2024-03-14T09:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/raid6 2024-03-14T09:29:24.206 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/raid6/raid6_pq.ko 2024-03-14T09:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/reed_solomon 2024-03-14T09:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-14T09:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/ts_bm.ko 2024-03-14T09:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/ts_fsm.ko 2024-03-14T09:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/ts_kmp.ko 2024-03-14T09:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/zstd 2024-03-14T09:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/lib/zstd/zstd_compress.ko 2024-03-14T09:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/mm 2024-03-14T09:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/mm/hwpoison-inject.ko 2024-03-14T09:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net 2024-03-14T09:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/6lowpan 2024-03-14T09:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/6lowpan/6lowpan.ko 2024-03-14T09:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/6lowpan/nhc_dest.ko 2024-03-14T09:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/6lowpan/nhc_fragment.ko 2024-03-14T09:29:24.207 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/6lowpan/nhc_hop.ko 2024-03-14T09:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-14T09:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/6lowpan/nhc_mobility.ko 2024-03-14T09:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/6lowpan/nhc_routing.ko 2024-03-14T09:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/6lowpan/nhc_udp.ko 2024-03-14T09:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/802 2024-03-14T09:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/802/garp.ko 2024-03-14T09:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/802/mrp.ko 2024-03-14T09:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/802/p8022.ko 2024-03-14T09:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/802/psnap.ko 2024-03-14T09:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/802/stp.ko 2024-03-14T09:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/8021q 2024-03-14T09:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/8021q/8021q.ko 2024-03-14T09:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/9p 2024-03-14T09:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/9p/9pnet.ko 2024-03-14T09:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/9p/9pnet_fd.ko 2024-03-14T09:29:24.208 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/9p/9pnet_rdma.ko 2024-03-14T09:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/9p/9pnet_virtio.ko 2024-03-14T09:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/batman-adv 2024-03-14T09:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/batman-adv/batman-adv.ko 2024-03-14T09:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge 2024-03-14T09:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/br_netfilter.ko 2024-03-14T09:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/bridge.ko 2024-03-14T09:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter 2024-03-14T09:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-14T09:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-14T09:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-14T09:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-14T09:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-14T09:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-14T09:29:24.209 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-14T09:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-14T09:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-14T09:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-14T09:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-14T09:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-14T09:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-14T09:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-14T09:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-14T09:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-14T09:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-14T09:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-14T09:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-14T09:29:24.210 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-14T09:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/ebtables.ko 2024-03-14T09:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-14T09:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-14T09:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-14T09:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ceph 2024-03-14T09:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ceph/libceph.ko 2024-03-14T09:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/core 2024-03-14T09:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/core/failover.ko 2024-03-14T09:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/core/pktgen.ko 2024-03-14T09:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/dccp 2024-03-14T09:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/dccp/dccp.ko 2024-03-14T09:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/dccp/dccp_diag.ko 2024-03-14T09:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/dccp/dccp_ipv4.ko 2024-03-14T09:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/dccp/dccp_ipv6.ko 2024-03-14T09:29:24.211 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/dns_resolver 2024-03-14T09:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/dns_resolver/dns_resolver.ko 2024-03-14T09:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/dsa 2024-03-14T09:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/dsa/dsa_core.ko 2024-03-14T09:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/dsa/tag_brcm.ko 2024-03-14T09:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/dsa/tag_dsa.ko 2024-03-14T09:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/dsa/tag_none.ko 2024-03-14T09:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/dsa/tag_trailer.ko 2024-03-14T09:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ieee802154 2024-03-14T09:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ieee802154/6lowpan 2024-03-14T09:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-14T09:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ieee802154/ieee802154.ko 2024-03-14T09:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-14T09:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4 2024-03-14T09:29:24.212 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/ah4.ko 2024-03-14T09:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/esp4.ko 2024-03-14T09:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/fou.ko 2024-03-14T09:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/gre.ko 2024-03-14T09:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/inet_diag.ko 2024-03-14T09:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/ip_gre.ko 2024-03-14T09:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/ip_tunnel.ko 2024-03-14T09:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/ip_vti.ko 2024-03-14T09:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/ipcomp.ko 2024-03-14T09:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/ipip.ko 2024-03-14T09:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter 2024-03-14T09:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-14T09:29:24.213 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-14T09:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-14T09:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-14T09:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-14T09:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-14T09:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-14T09:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-14T09:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-14T09:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-14T09:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-14T09:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-14T09:29:24.214 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-14T09:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-14T09:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-14T09:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-14T09:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-14T09:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-14T09:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-14T09:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-14T09:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-14T09:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/tcp_bic.ko 2024-03-14T09:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/tcp_dctcp.ko 2024-03-14T09:29:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/tcp_diag.ko 2024-03-14T09:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/tcp_highspeed.ko 2024-03-14T09:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/tcp_htcp.ko 2024-03-14T09:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/tcp_hybla.ko 2024-03-14T09:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/tcp_illinois.ko 2024-03-14T09:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/tcp_lp.ko 2024-03-14T09:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/tcp_scalable.ko 2024-03-14T09:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/tcp_vegas.ko 2024-03-14T09:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/tcp_veno.ko 2024-03-14T09:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/tcp_westwood.ko 2024-03-14T09:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/tcp_yeah.ko 2024-03-14T09:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/tunnel4.ko 2024-03-14T09:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/udp_diag.ko 2024-03-14T09:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/udp_tunnel.ko 2024-03-14T09:29:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-14T09:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6 2024-03-14T09:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/ah6.ko 2024-03-14T09:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/esp6.ko 2024-03-14T09:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/fou6.ko 2024-03-14T09:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/ip6_tunnel.ko 2024-03-14T09:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-14T09:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/ip6_vti.ko 2024-03-14T09:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/ipcomp6.ko 2024-03-14T09:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter 2024-03-14T09:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-14T09:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-14T09:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-14T09:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-14T09:29:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-14T09:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-14T09:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-14T09:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-14T09:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-14T09:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-14T09:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-14T09:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-14T09:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-14T09:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-14T09:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-14T09:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-14T09:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-14T09:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-14T09:29:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-14T09:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-14T09:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-14T09:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-14T09:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-14T09:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-14T09:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/sit.ko 2024-03-14T09:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/tunnel6.ko 2024-03-14T09:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-14T09:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/key 2024-03-14T09:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/key/af_key.ko 2024-03-14T09:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/l2tp 2024-03-14T09:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/l2tp/l2tp_core.ko 2024-03-14T09:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-14T09:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/l2tp/l2tp_eth.ko 2024-03-14T09:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/l2tp/l2tp_ip.ko 2024-03-14T09:29:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/l2tp/l2tp_ip6.ko 2024-03-14T09:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/l2tp/l2tp_netlink.ko 2024-03-14T09:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/l2tp/l2tp_ppp.ko 2024-03-14T09:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/llc 2024-03-14T09:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/llc/llc.ko 2024-03-14T09:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/mac802154 2024-03-14T09:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/mac802154/mac802154.ko 2024-03-14T09:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/mpls 2024-03-14T09:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/mpls/mpls_gso.ko 2024-03-14T09:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter 2024-03-14T09:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset 2024-03-14T09:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset/ip_set.ko 2024-03-14T09:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-14T09:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-14T09:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-14T09:29:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-14T09:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-14T09:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-14T09:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-14T09:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-14T09:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-14T09:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-14T09:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-14T09:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-14T09:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-14T09:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-14T09:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-14T09:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipvs 2024-03-14T09:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-14T09:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-14T09:29:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-14T09:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-14T09:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-14T09:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-14T09:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-14T09:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-14T09:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-14T09:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-14T09:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-14T09:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-14T09:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-14T09:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-14T09:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_conncount.ko 2024-03-14T09:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_conntrack.ko 2024-03-14T09:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-14T09:29:24.222 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-14T09:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-14T09:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-14T09:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-14T09:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-14T09:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-14T09:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-14T09:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-14T09:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-14T09:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-14T09:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-14T09:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-14T09:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_flow_table.ko 2024-03-14T09:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-14T09:29:24.223 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_log_syslog.ko 2024-03-14T09:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_nat.ko 2024-03-14T09:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-14T09:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-14T09:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_nat_irc.ko 2024-03-14T09:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_nat_sip.ko 2024-03-14T09:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-14T09:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-14T09:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nf_tables.ko 2024-03-14T09:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nfnetlink.ko 2024-03-14T09:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-14T09:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nfnetlink_log.ko 2024-03-14T09:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-14T09:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-14T09:29:24.224 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_chain_nat.ko 2024-03-14T09:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_compat.ko 2024-03-14T09:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_ct.ko 2024-03-14T09:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-14T09:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_fib.ko 2024-03-14T09:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-14T09:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_hash.ko 2024-03-14T09:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_limit.ko 2024-03-14T09:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_log.ko 2024-03-14T09:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_masq.ko 2024-03-14T09:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_nat.ko 2024-03-14T09:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_numgen.ko 2024-03-14T09:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_queue.ko 2024-03-14T09:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_quota.ko 2024-03-14T09:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_redir.ko 2024-03-14T09:29:24.225 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_reject.ko 2024-03-14T09:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_reject_inet.ko 2024-03-14T09:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_socket.ko 2024-03-14T09:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/nft_tunnel.ko 2024-03-14T09:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_AUDIT.ko 2024-03-14T09:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-14T09:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-14T09:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-14T09:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_CT.ko 2024-03-14T09:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_DSCP.ko 2024-03-14T09:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_HL.ko 2024-03-14T09:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_HMARK.ko 2024-03-14T09:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-14T09:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_LED.ko 2024-03-14T09:29:24.226 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_LOG.ko 2024-03-14T09:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-14T09:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_NETMAP.ko 2024-03-14T09:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_NFLOG.ko 2024-03-14T09:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-14T09:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_RATEEST.ko 2024-03-14T09:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-14T09:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_SECMARK.ko 2024-03-14T09:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-14T09:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-14T09:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_TEE.ko 2024-03-14T09:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_TPROXY.ko 2024-03-14T09:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_TRACE.ko 2024-03-14T09:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_addrtype.ko 2024-03-14T09:29:24.227 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_bpf.ko 2024-03-14T09:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_cgroup.ko 2024-03-14T09:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_cluster.ko 2024-03-14T09:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_comment.ko 2024-03-14T09:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_connbytes.ko 2024-03-14T09:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_connlabel.ko 2024-03-14T09:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_connlimit.ko 2024-03-14T09:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_connmark.ko 2024-03-14T09:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_conntrack.ko 2024-03-14T09:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_cpu.ko 2024-03-14T09:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_dccp.ko 2024-03-14T09:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_devgroup.ko 2024-03-14T09:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_dscp.ko 2024-03-14T09:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_ecn.ko 2024-03-14T09:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_esp.ko 2024-03-14T09:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_hashlimit.ko 2024-03-14T09:29:24.228 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_helper.ko 2024-03-14T09:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_hl.ko 2024-03-14T09:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_ipcomp.ko 2024-03-14T09:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_iprange.ko 2024-03-14T09:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_ipvs.ko 2024-03-14T09:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_l2tp.ko 2024-03-14T09:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_length.ko 2024-03-14T09:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_limit.ko 2024-03-14T09:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_mac.ko 2024-03-14T09:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_mark.ko 2024-03-14T09:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_multiport.ko 2024-03-14T09:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_nat.ko 2024-03-14T09:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_nfacct.ko 2024-03-14T09:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_osf.ko 2024-03-14T09:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_owner.ko 2024-03-14T09:29:24.229 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_physdev.ko 2024-03-14T09:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_pkttype.ko 2024-03-14T09:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_policy.ko 2024-03-14T09:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_quota.ko 2024-03-14T09:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_rateest.ko 2024-03-14T09:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_realm.ko 2024-03-14T09:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_recent.ko 2024-03-14T09:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_sctp.ko 2024-03-14T09:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_set.ko 2024-03-14T09:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_socket.ko 2024-03-14T09:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_state.ko 2024-03-14T09:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_statistic.ko 2024-03-14T09:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_string.ko 2024-03-14T09:29:24.230 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_tcpmss.ko 2024-03-14T09:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_time.ko 2024-03-14T09:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netfilter/xt_u32.ko 2024-03-14T09:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netlink 2024-03-14T09:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/netlink/netlink_diag.ko 2024-03-14T09:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/nsh 2024-03-14T09:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/nsh/nsh.ko 2024-03-14T09:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/openvswitch 2024-03-14T09:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/openvswitch/openvswitch.ko 2024-03-14T09:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/openvswitch/vport-geneve.ko 2024-03-14T09:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/openvswitch/vport-gre.ko 2024-03-14T09:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/openvswitch/vport-vxlan.ko 2024-03-14T09:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/packet 2024-03-14T09:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/packet/af_packet_diag.ko 2024-03-14T09:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/rds 2024-03-14T09:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/rds/rds.ko 2024-03-14T09:29:24.231 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/rds/rds_rdma.ko 2024-03-14T09:29:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/rds/rds_tcp.ko 2024-03-14T09:29:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched 2024-03-14T09:29:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/act_csum.ko 2024-03-14T09:29:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/act_gact.ko 2024-03-14T09:29:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/act_mirred.ko 2024-03-14T09:29:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/act_nat.ko 2024-03-14T09:29:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/act_pedit.ko 2024-03-14T09:29:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/act_police.ko 2024-03-14T09:29:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/act_simple.ko 2024-03-14T09:29:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/act_skbedit.ko 2024-03-14T09:29:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/act_vlan.ko 2024-03-14T09:29:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/cls_basic.ko 2024-03-14T09:29:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/cls_bpf.ko 2024-03-14T09:29:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/cls_flow.ko 2024-03-14T09:29:24.232 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/cls_fw.ko 2024-03-14T09:29:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/cls_route.ko 2024-03-14T09:29:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/cls_u32.ko 2024-03-14T09:29:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/em_cmp.ko 2024-03-14T09:29:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/em_ipset.ko 2024-03-14T09:29:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/em_meta.ko 2024-03-14T09:29:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/em_nbyte.ko 2024-03-14T09:29:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/em_text.ko 2024-03-14T09:29:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/em_u32.ko 2024-03-14T09:29:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_choke.ko 2024-03-14T09:29:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_codel.ko 2024-03-14T09:29:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_drr.ko 2024-03-14T09:29:24.233 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_fq.ko 2024-03-14T09:29:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_gred.ko 2024-03-14T09:29:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_hfsc.ko 2024-03-14T09:29:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_hhf.ko 2024-03-14T09:29:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_htb.ko 2024-03-14T09:29:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_ingress.ko 2024-03-14T09:29:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_mqprio.ko 2024-03-14T09:29:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_mqprio_lib.ko 2024-03-14T09:29:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_multiq.ko 2024-03-14T09:29:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_netem.ko 2024-03-14T09:29:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_pie.ko 2024-03-14T09:29:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_plug.ko 2024-03-14T09:29:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_prio.ko 2024-03-14T09:29:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_qfq.ko 2024-03-14T09:29:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_red.ko 2024-03-14T09:29:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_sfb.ko 2024-03-14T09:29:24.235 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_sfq.ko 2024-03-14T09:29:24.235 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_tbf.ko 2024-03-14T09:29:24.235 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sched/sch_teql.ko 2024-03-14T09:29:24.235 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sctp 2024-03-14T09:29:24.235 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sctp/sctp.ko 2024-03-14T09:29:24.235 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sctp/sctp_diag.ko 2024-03-14T09:29:24.235 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sunrpc 2024-03-14T09:29:24.235 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sunrpc/auth_gss 2024-03-14T09:29:24.235 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-14T09:29:24.235 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-14T09:29:24.235 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sunrpc/sunrpc.ko 2024-03-14T09:29:24.235 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sunrpc/xprtrdma 2024-03-14T09:29:24.235 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-14T09:29:24.235 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/tipc 2024-03-14T09:29:24.235 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/tipc/diag.ko 2024-03-14T09:29:24.236 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/tipc/tipc.ko 2024-03-14T09:29:24.236 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/unix 2024-03-14T09:29:24.236 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/unix/unix_diag.ko 2024-03-14T09:29:24.236 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/vmw_vsock 2024-03-14T09:29:24.236 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-14T09:29:24.236 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-14T09:29:24.236 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/vmw_vsock/vsock.ko 2024-03-14T09:29:24.236 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-14T09:29:24.236 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-14T09:29:24.236 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/xfrm 2024-03-14T09:29:24.236 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-14T09:29:24.236 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/security 2024-03-14T09:29:24.236 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/security/keys 2024-03-14T09:29:24.236 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/security/keys/encrypted-keys 2024-03-14T09:29:24.237 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-14T09:29:24.237 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/security/keys/trusted-keys 2024-03-14T09:29:24.237 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/security/keys/trusted-keys/trusted.ko 2024-03-14T09:29:24.237 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/virt 2024-03-14T09:29:24.237 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/virt/lib 2024-03-14T09:29:24.237 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/kernel/virt/lib/irqbypass.ko 2024-03-14T09:29:24.237 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/modules.alias 2024-03-14T09:29:24.237 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/modules.alias.bin 2024-03-14T09:29:24.237 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/modules.builtin 2024-03-14T09:29:24.237 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/modules.builtin.bin 2024-03-14T09:29:24.237 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/modules.builtin.modinfo 2024-03-14T09:29:24.237 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/modules.dep 2024-03-14T09:29:24.237 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/modules.dep.bin 2024-03-14T09:29:24.237 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/modules.devname 2024-03-14T09:29:24.237 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/modules.order 2024-03-14T09:29:24.237 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/modules.softdep 2024-03-14T09:29:24.238 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/modules.symbols 2024-03-14T09:29:24.238 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/modules.symbols.bin 2024-03-14T09:29:24.238 INFO:teuthology.orchestra.run.smithi078.stdout:/lib/modules/6.8.0-rc7-g5f16de4b3661/vmlinuz 2024-03-14T09:29:24.242 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc7-g5f16de4b3661 2024-03-14T09:29:24.242 DEBUG:teuthology.orchestra.run.smithi078:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-14T09:29:24.265 INFO:teuthology.orchestra.run.smithi078.stdout:postinstall scriptlet (using /bin/sh): 2024-03-14T09:29:24.265 INFO:teuthology.orchestra.run.smithi078.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-14T09:29:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: /usr/bin/kernel-install add 6.8.0-rc7-g5f16de4b3661 /lib/modules/6.8.0-rc7-g5f16de4b3661/vmlinuz 2024-03-14T09:29:24.266 INFO:teuthology.orchestra.run.smithi078.stdout:fi 2024-03-14T09:29:24.266 INFO:teuthology.orchestra.run.smithi078.stdout:for file in vmlinuz System.map config; do 2024-03-14T09:29:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc7-g5f16de4b3661/${file}" "/boot/${file}-6.8.0-rc7-g5f16de4b3661"; then 2024-03-14T09:29:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: cp "/lib/modules/6.8.0-rc7-g5f16de4b3661/${file}" "/boot/${file}-6.8.0-rc7-g5f16de4b3661" 2024-03-14T09:29:24.266 INFO:teuthology.orchestra.run.smithi078.stdout: fi 2024-03-14T09:29:24.266 INFO:teuthology.orchestra.run.smithi078.stdout:done 2024-03-14T09:29:24.266 INFO:teuthology.orchestra.run.smithi078.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-14T09:29:24.266 INFO:teuthology.orchestra.run.smithi078.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-14T09:29:24.266 INFO:teuthology.orchestra.run.smithi078.stdout:new-kernel-pkg --remove 6.8.0-rc7-g5f16de4b3661 --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc7-g5f16de4b3661.img 2024-03-14T09:29:24.266 INFO:teuthology.orchestra.run.smithi078.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-14T09:29:24.266 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-install remove 6.8.0-rc7-g5f16de4b3661 2024-03-14T09:29:24.266 INFO:teuthology.orchestra.run.smithi078.stdout:fi 2024-03-14T09:29:24.267 INFO:teuthology.orchestra.run.smithi078.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-14T09:29:24.267 INFO:teuthology.orchestra.run.smithi078.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-14T09:29:24.267 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/update-bootloader --remove 6.8.0-rc7-g5f16de4b3661 2024-03-14T09:29:24.267 INFO:teuthology.orchestra.run.smithi078.stdout:fi 2024-03-14T09:29:24.268 DEBUG:teuthology.orchestra.run.smithi078:> sudo rpm -qi grub2-tools 2024-03-14T09:29:24.342 INFO:teuthology.orchestra.run.smithi078.stdout:Name : grub2-tools 2024-03-14T09:29:24.342 INFO:teuthology.orchestra.run.smithi078.stdout:Epoch : 1 2024-03-14T09:29:24.342 INFO:teuthology.orchestra.run.smithi078.stdout:Version : 2.02 2024-03-14T09:29:24.342 INFO:teuthology.orchestra.run.smithi078.stdout:Release : 123.el8 2024-03-14T09:29:24.342 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: x86_64 2024-03-14T09:29:24.342 INFO:teuthology.orchestra.run.smithi078.stdout:Install Date: Thu 14 Dec 2023 12:39:42 AM UTC 2024-03-14T09:29:24.342 INFO:teuthology.orchestra.run.smithi078.stdout:Group : System Environment/Base 2024-03-14T09:29:24.343 INFO:teuthology.orchestra.run.smithi078.stdout:Size : 9265450 2024-03-14T09:29:24.343 INFO:teuthology.orchestra.run.smithi078.stdout:License : GPLv3+ 2024-03-14T09:29:24.343 INFO:teuthology.orchestra.run.smithi078.stdout:Signature : RSA/SHA256, Thu 31 Mar 2022 07:24:35 PM UTC, Key ID 199e2f91fd431d51 2024-03-14T09:29:24.343 INFO:teuthology.orchestra.run.smithi078.stdout:Source RPM : grub2-2.02-123.el8.src.rpm 2024-03-14T09:29:24.343 INFO:teuthology.orchestra.run.smithi078.stdout:Build Date : Mon 28 Mar 2022 02:12:09 PM UTC 2024-03-14T09:29:24.343 INFO:teuthology.orchestra.run.smithi078.stdout:Build Host : x86-vm-09.build.eng.bos.redhat.com 2024-03-14T09:29:24.343 INFO:teuthology.orchestra.run.smithi078.stdout:Relocations : (not relocatable) 2024-03-14T09:29:24.343 INFO:teuthology.orchestra.run.smithi078.stdout:Packager : Red Hat, Inc. 2024-03-14T09:29:24.343 INFO:teuthology.orchestra.run.smithi078.stdout:Vendor : Red Hat, Inc. 2024-03-14T09:29:24.343 INFO:teuthology.orchestra.run.smithi078.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-14T09:29:24.343 INFO:teuthology.orchestra.run.smithi078.stdout:Summary : Support tools for GRUB. 2024-03-14T09:29:24.343 INFO:teuthology.orchestra.run.smithi078.stdout:Description : 2024-03-14T09:29:24.343 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:29:24.343 INFO:teuthology.orchestra.run.smithi078.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-14T09:29:24.343 INFO:teuthology.orchestra.run.smithi078.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-14T09:29:24.344 INFO:teuthology.orchestra.run.smithi078.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-14T09:29:24.344 INFO:teuthology.orchestra.run.smithi078.stdout:hardware devices. 2024-03-14T09:29:24.344 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:29:24.344 INFO:teuthology.orchestra.run.smithi078.stdout:This subpackage provides tools for support of all platforms. 2024-03-14T09:29:24.345 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-14T09:29:24.345 INFO:teuthology.task.kernel:Updating grub on smithi078 to boot 6.8.0-rc7-g5f16de4b3661 2024-03-14T09:29:24.345 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-14T09:29:25.093 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2024-03-14T09:29:27.580 INFO:teuthology.orchestra.run.smithi078.stderr:done 2024-03-14T09:29:27.582 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-03-14T09:29:27.598 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.cwcCl9YzlN 2024-03-14T09:29:27.599 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cwcCl9YzlN 2024-03-14T09:29:27.701 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.cwcCl9YzlN 2024-03-14T09:29:27.816 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.cwcCl9YzlN is 6KB 2024-03-14T09:29:27.865 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.cwcCl9YzlN 2024-03-14T09:29:27.882 DEBUG:teuthology.orchestra.run.smithi078:> sudo /bin/ls /boot/loader/entries || true 2024-03-14T09:29:27.946 INFO:teuthology.orchestra.run.smithi078.stdout:2db68f91c8f90721a6596efe657a5324-0-rescue.conf 2024-03-14T09:29:27.946 INFO:teuthology.orchestra.run.smithi078.stdout:2db68f91c8f90721a6596efe657a5324-6.8.0-rc7-g5f16de4b3661.conf 2024-03-14T09:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout:963f8b6c99184de696879b3f7da9ecce-0-rescue.conf 2024-03-14T09:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout:963f8b6c99184de696879b3f7da9ecce-4.18.0-372.9.1.el8.x86_64.conf 2024-03-14T09:29:27.947 INFO:teuthology.orchestra.run.smithi078.stdout:963f8b6c99184de696879b3f7da9ecce-4.18.0-513.9.1.el8_9.x86_64.conf 2024-03-14T09:29:27.948 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-set-default 2db68f91c8f90721a6596efe657a5324-6.8.0-rc7-g5f16de4b3661 2024-03-14T09:29:28.104 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2024-03-14T09:29:58.107 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-14T09:29:58.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-14T09:29:58.109 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T09:30:58.112 DEBUG:teuthology.orchestra.remote:timed out 2024-03-14T09:31:07.114 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-14T09:31:07.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T09:31:14.725 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-03-14T09:31:15.114 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-14T09:31:15.114 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc7-g5f16de4b3661"... 2024-03-14T09:31:15.115 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-03-14T09:31:15.155 INFO:teuthology.orchestra.run.smithi078.stdout:6.8.0-rc7-g5f16de4b3661 2024-03-14T09:31:15.156 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc7-g5f16de4b3661 vs 6.8.0-rc7-g5f16de4b3661 2024-03-14T09:31:15.156 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-14T09:31:15.156 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2024-03-14T09:31:16.160 DEBUG:teuthology.parallel:result is None 2024-03-14T09:31:16.160 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-14T09:31:16.167 INFO:teuthology.task.internal:Creating test directory... 2024-03-14T09:31:16.168 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-14T09:31:16.173 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-14T09:31:16.178 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-14T09:31:16.212 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-14T09:31:16.219 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-14T09:31:16.229 INFO:teuthology.task.internal:Creating archive directory... 2024-03-14T09:31:16.229 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-14T09:31:16.256 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-14T09:31:16.259 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-14T09:31:16.306 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-14T09:31:16.313 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-14T09:31:16.313 DEBUG:teuthology.orchestra.run.smithi132:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-14T09:31:16.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T09:31:16.346 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-14T09:31:16.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T09:31:16.365 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-14T09:31:16.369 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-14T09:31:16.389 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-14T09:31:16.441 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-14T09:31:16.460 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-14T09:31:16.479 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-14T09:31:16.482 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-14T09:31:16.590 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-14T09:31:16.627 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-14T09:31:16.630 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-14T09:31:16.639 INFO:teuthology.task.internal:Configuring sudo... 2024-03-14T09:31:16.639 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-14T09:31:16.643 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-14T09:31:16.645 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-14T09:31:16.705 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-14T09:31:16.714 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-14T09:31:16.715 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-14T09:31:16.718 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-14T09:31:16.720 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-14T09:31:16.755 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-14T09:31:16.866 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-14T09:31:16.951 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-14T09:31:17.021 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-14T09:31:17.102 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:31:17.103 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-14T09:31:17.167 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-14T09:31:17.215 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-14T09:31:17.270 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-14T09:31:17.355 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-14T09:31:17.402 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-14T09:31:17.402 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-14T09:31:17.471 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-14T09:31:17.519 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-14T09:31:17.577 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-14T09:31:17.668 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-14T09:31:17.710 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-14T09:31:17.710 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-14T09:31:17.779 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2024-03-14T09:31:17.783 DEBUG:teuthology.orchestra.run.smithi132:> sudo service rsyslog restart 2024-03-14T09:31:17.785 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2024-03-14T09:31:17.850 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-14T09:31:17.879 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-14T09:31:17.905 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-14T09:31:18.281 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-14T09:31:18.290 INFO:teuthology.task.internal:Starting timer... 2024-03-14T09:31:18.290 INFO:teuthology.run_tasks:Running task pcp... 2024-03-14T09:31:18.305 INFO:teuthology.run_tasks:Running task selinux... 2024-03-14T09:31:18.318 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-14T09:31:18.318 DEBUG:teuthology.orchestra.run.smithi078:> sudo service auditd rotate 2024-03-14T09:31:18.491 INFO:teuthology.orchestra.run.smithi078.stdout:Rotating logs: 2024-03-14T09:31:18.495 DEBUG:teuthology.orchestra.run.smithi132:> sudo service auditd rotate 2024-03-14T09:31:18.550 INFO:teuthology.orchestra.run.smithi132.stdout:Rotating logs: 2024-03-14T09:31:18.552 DEBUG:teuthology.orchestra.run.smithi152:> sudo service auditd rotate 2024-03-14T09:31:18.614 INFO:teuthology.orchestra.run.smithi152.stdout:Rotating logs: 2024-03-14T09:31:18.615 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-14T09:31:18.615 DEBUG:teuthology.orchestra.run.smithi078:> /usr/sbin/getenforce 2024-03-14T09:31:18.663 INFO:teuthology.orchestra.run.smithi078.stdout:Permissive 2024-03-14T09:31:18.663 DEBUG:teuthology.orchestra.run.smithi132:> /usr/sbin/getenforce 2024-03-14T09:31:18.691 INFO:teuthology.orchestra.run.smithi132.stdout:Permissive 2024-03-14T09:31:18.692 DEBUG:teuthology.orchestra.run.smithi152:> /usr/sbin/getenforce 2024-03-14T09:31:18.711 INFO:teuthology.orchestra.run.smithi152.stdout:Permissive 2024-03-14T09:31:18.712 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi078.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi132.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi152.front.sepia.ceph.com': 'permissive'} 2024-03-14T09:31:18.712 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-14T09:31:18.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T09:31:18.779 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-14T09:31:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T09:31:18.807 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-14T09:31:18.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T09:31:18.835 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-14T09:31:18.835 DEBUG:teuthology.orchestra.run.smithi078:> sudo /usr/sbin/setenforce permissive 2024-03-14T09:31:18.910 DEBUG:teuthology.orchestra.run.smithi132:> sudo /usr/sbin/setenforce permissive 2024-03-14T09:31:18.944 DEBUG:teuthology.orchestra.run.smithi152:> sudo /usr/sbin/setenforce permissive 2024-03-14T09:31:18.982 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-14T09:31:18.995 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-14T09:31:19.000 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-14T09:31:19.167 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-14T09:31:19.183 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-14T09:31:19.184 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi132.front.sepia.ceph.com,smithi152.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-14T09:40:54.555 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi132.front.sepia.ceph.com'), Remote(name='ubuntu@smithi152.front.sepia.ceph.com')] 2024-03-14T09:40:54.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-14T09:40:54.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T09:40:54.646 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-03-14T09:40:54.755 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-03-14T09:40:54.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-14T09:40:54.756 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T09:40:54.829 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-03-14T09:40:54.895 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-14T09:40:54.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-14T09:40:54.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T09:40:54.973 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-03-14T09:40:55.046 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-03-14T09:40:55.047 INFO:teuthology.run_tasks:Running task clock... 2024-03-14T09:40:55.057 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-14T09:40:55.057 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-14T09:40:55.058 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-14T09:40:55.062 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-14T09:40:55.062 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-14T09:40:55.064 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-14T09:40:55.064 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-14T09:40:55.094 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-14T09:40:55.108 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-14T09:40:55.123 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-14T09:40:55.130 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-14T09:40:55.139 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-14T09:40:55.163 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-14T09:40:55.186 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: ntpd: command not found 2024-03-14T09:40:55.197 INFO:teuthology.orchestra.run.smithi132.stdout:506 Cannot talk to daemon 2024-03-14T09:40:55.211 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: ntpd: command not found 2024-03-14T09:40:55.213 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-14T09:40:55.224 INFO:teuthology.orchestra.run.smithi152.stdout:506 Cannot talk to daemon 2024-03-14T09:40:55.227 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-14T09:40:55.240 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-14T09:40:55.252 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: ntpd: command not found 2024-03-14T09:40:55.255 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-14T09:40:55.280 INFO:teuthology.orchestra.run.smithi078.stdout:506 Cannot talk to daemon 2024-03-14T09:40:55.299 INFO:teuthology.orchestra.run.smithi132.stderr:bash: ntpq: command not found 2024-03-14T09:40:55.304 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-14T09:40:55.304 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-03-14T09:40:55.304 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T09:40:55.304 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T09:40:55.304 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T09:40:55.304 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T09:40:55.312 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-14T09:40:55.332 INFO:teuthology.orchestra.run.smithi152.stderr:bash: ntpq: command not found 2024-03-14T09:40:55.335 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-14T09:40:55.336 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-03-14T09:40:55.336 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T09:40:55.336 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T09:40:55.336 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T09:40:55.336 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T09:40:55.344 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-14T09:40:55.437 INFO:teuthology.orchestra.run.smithi078.stderr:bash: ntpq: command not found 2024-03-14T09:40:55.445 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-14T09:40:55.445 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-03-14T09:40:55.445 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T09:40:55.445 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T09:40:55.445 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T09:40:55.445 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T09:40:55.446 INFO:teuthology.run_tasks:Running task pexec... 2024-03-14T09:40:55.455 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-14T09:40:55.455 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi078.front.sepia.ceph.com 2024-03-14T09:40:55.456 DEBUG:teuthology.orchestra.run.smithi078:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-14T09:40:55.456 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi132.front.sepia.ceph.com 2024-03-14T09:40:55.456 DEBUG:teuthology.orchestra.run.smithi132:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-14T09:40:55.456 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi152.front.sepia.ceph.com 2024-03-14T09:40:55.457 DEBUG:teuthology.orchestra.run.smithi152:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-14T09:40:55.834 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-03-14T09:40:55.835 INFO:teuthology.orchestra.run.smithi152.stdout:Updating Subscription Management repositories. 2024-03-14T09:40:55.939 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2024-03-14T09:40:56.687 INFO:teuthology.orchestra.run.smithi152.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 150 kB/s | 4.1 kB 00:00 2024-03-14T09:40:56.714 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 152 kB/s | 4.1 kB 00:00 2024-03-14T09:40:56.925 INFO:teuthology.orchestra.run.smithi078.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 147 kB/s | 4.1 kB 00:00 2024-03-14T09:40:56.953 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 138 kB/s | 4.5 kB 00:00 2024-03-14T09:40:56.975 INFO:teuthology.orchestra.run.smithi152.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 141 kB/s | 4.5 kB 00:00 2024-03-14T09:40:57.315 INFO:teuthology.orchestra.run.smithi078.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 173 kB/s | 4.5 kB 00:00 2024-03-14T09:41:00.078 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-14T09:41:00.079 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-14T09:41:00.079 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-03-14T09:41:00.079 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-14T09:41:00.079 INFO:teuthology.orchestra.run.smithi132.stdout:Resetting modules: 2024-03-14T09:41:00.080 INFO:teuthology.orchestra.run.smithi132.stdout: container-tools 2024-03-14T09:41:00.080 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-14T09:41:00.080 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-14T09:41:00.080 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-14T09:41:00.080 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-14T09:41:00.227 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-14T09:41:00.228 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-14T09:41:00.228 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-03-14T09:41:00.228 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-14T09:41:00.228 INFO:teuthology.orchestra.run.smithi152.stdout:Resetting modules: 2024-03-14T09:41:00.228 INFO:teuthology.orchestra.run.smithi152.stdout: container-tools 2024-03-14T09:41:00.228 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-14T09:41:00.228 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-14T09:41:00.228 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-03-14T09:41:00.228 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-14T09:41:00.519 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-14T09:41:00.520 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-14T09:41:00.520 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2024-03-14T09:41:00.520 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-14T09:41:00.520 INFO:teuthology.orchestra.run.smithi078.stdout:Resetting modules: 2024-03-14T09:41:00.520 INFO:teuthology.orchestra.run.smithi078.stdout: container-tools 2024-03-14T09:41:00.520 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:41:00.520 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-14T09:41:00.520 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-03-14T09:41:00.520 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:41:05.137 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-14T09:41:05.453 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-14T09:41:05.615 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-03-14T09:41:05.966 INFO:teuthology.orchestra.run.smithi152.stdout:Updating Subscription Management repositories. 2024-03-14T09:41:06.342 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-14T09:41:06.588 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 33 kB/s | 4.1 kB 00:00 2024-03-14T09:41:06.896 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 50 kB/s | 4.5 kB 00:00 2024-03-14T09:41:06.957 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2024-03-14T09:41:07.406 INFO:teuthology.orchestra.run.smithi152.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 34 kB/s | 4.1 kB 00:00 2024-03-14T09:41:07.803 INFO:teuthology.orchestra.run.smithi152.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 kB/s | 4.5 kB 00:00 2024-03-14T09:41:08.873 INFO:teuthology.orchestra.run.smithi078.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 29 kB/s | 4.1 kB 00:00 2024-03-14T09:41:09.344 INFO:teuthology.orchestra.run.smithi078.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-03-14T09:41:09.941 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-14T09:41:10.099 INFO:teuthology.orchestra.run.smithi132.stdout:========================================================================================================== 2024-03-14T09:41:10.099 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-14T09:41:10.100 INFO:teuthology.orchestra.run.smithi132.stdout:========================================================================================================== 2024-03-14T09:41:10.100 INFO:teuthology.orchestra.run.smithi132.stdout:Installing group/module packages: 2024-03-14T09:41:10.100 INFO:teuthology.orchestra.run.smithi132.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-03-14T09:41:10.100 INFO:teuthology.orchestra.run.smithi132.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-03-14T09:41:10.100 INFO:teuthology.orchestra.run.smithi132.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-03-14T09:41:10.100 INFO:teuthology.orchestra.run.smithi132.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-03-14T09:41:10.100 INFO:teuthology.orchestra.run.smithi132.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-14T09:41:10.100 INFO:teuthology.orchestra.run.smithi132.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-03-14T09:41:10.100 INFO:teuthology.orchestra.run.smithi132.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-03-14T09:41:10.100 INFO:teuthology.orchestra.run.smithi132.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-03-14T09:41:10.100 INFO:teuthology.orchestra.run.smithi132.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-03-14T09:41:10.100 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-14T09:41:10.101 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-03-14T09:41:10.101 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-03-14T09:41:10.101 INFO:teuthology.orchestra.run.smithi132.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-03-14T09:41:10.101 INFO:teuthology.orchestra.run.smithi132.stdout:Installing weak dependencies: 2024-03-14T09:41:10.101 INFO:teuthology.orchestra.run.smithi132.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-14T09:41:10.101 INFO:teuthology.orchestra.run.smithi132.stdout:Installing module profiles: 2024-03-14T09:41:10.101 INFO:teuthology.orchestra.run.smithi132.stdout: container-tools/common 2024-03-14T09:41:10.101 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-14T09:41:10.101 INFO:teuthology.orchestra.run.smithi132.stdout:Enabling module streams: 2024-03-14T09:41:10.101 INFO:teuthology.orchestra.run.smithi132.stdout: container-tools rhel8 2024-03-14T09:41:10.101 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-14T09:41:10.101 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-14T09:41:10.101 INFO:teuthology.orchestra.run.smithi132.stdout:========================================================================================================== 2024-03-14T09:41:10.101 INFO:teuthology.orchestra.run.smithi132.stdout:Install 13 Packages 2024-03-14T09:41:10.102 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-14T09:41:10.102 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 26 M 2024-03-14T09:41:10.102 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 87 M 2024-03-14T09:41:10.102 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-14T09:41:10.405 INFO:teuthology.orchestra.run.smithi132.stdout:(1/13): python3-pytoml-0.1.14-5.git7dea353.el8. 82 kB/s | 25 kB 00:00 2024-03-14T09:41:10.430 INFO:teuthology.orchestra.run.smithi132.stdout:(2/13): criu-libs-3.18-4.module+el8.9.0+20326+3 120 kB/s | 39 kB 00:00 2024-03-14T09:41:10.455 INFO:teuthology.orchestra.run.smithi132.stdout:(3/13): python3-pyxdg-0.25-16.el8.noarch.rpm 266 kB/s | 94 kB 00:00 2024-03-14T09:41:10.480 INFO:teuthology.orchestra.run.smithi132.stdout:(4/13): yajl-2.1.0-12.el8.x86_64.rpm 543 kB/s | 41 kB 00:00 2024-03-14T09:41:10.731 INFO:teuthology.orchestra.run.smithi132.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 3.4 MB/s | 1.0 MB 00:00 2024-03-14T09:41:10.764 INFO:teuthology.orchestra.run.smithi132.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2124 2.5 MB/s | 739 kB 00:00 2024-03-14T09:41:10.819 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-14T09:41:10.956 INFO:teuthology.orchestra.run.smithi132.stdout:(7/13): buildah-1.31.3-3.module+el8.9.0+21243+a 18 MB/s | 8.8 MB 00:00 2024-03-14T09:41:10.980 INFO:teuthology.orchestra.run.smithi152.stdout:========================================================================================================== 2024-03-14T09:41:10.980 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-14T09:41:10.980 INFO:teuthology.orchestra.run.smithi152.stdout:========================================================================================================== 2024-03-14T09:41:10.980 INFO:teuthology.orchestra.run.smithi152.stdout:Installing group/module packages: 2024-03-14T09:41:10.980 INFO:teuthology.orchestra.run.smithi152.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-03-14T09:41:10.980 INFO:teuthology.orchestra.run.smithi152.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-03-14T09:41:10.980 INFO:teuthology.orchestra.run.smithi152.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-03-14T09:41:10.980 INFO:teuthology.orchestra.run.smithi152.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-03-14T09:41:10.980 INFO:teuthology.orchestra.run.smithi152.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-14T09:41:10.980 INFO:teuthology.orchestra.run.smithi152.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-03-14T09:41:10.980 INFO:teuthology.orchestra.run.smithi152.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-03-14T09:41:10.980 INFO:teuthology.orchestra.run.smithi152.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-03-14T09:41:10.980 INFO:teuthology.orchestra.run.smithi152.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-03-14T09:41:10.981 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-14T09:41:10.981 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-03-14T09:41:10.981 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-03-14T09:41:10.981 INFO:teuthology.orchestra.run.smithi152.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-03-14T09:41:10.981 INFO:teuthology.orchestra.run.smithi152.stdout:Installing weak dependencies: 2024-03-14T09:41:10.981 INFO:teuthology.orchestra.run.smithi152.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-14T09:41:10.981 INFO:teuthology.orchestra.run.smithi152.stdout:Installing module profiles: 2024-03-14T09:41:10.981 INFO:teuthology.orchestra.run.smithi152.stdout: container-tools/common 2024-03-14T09:41:10.981 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-14T09:41:10.981 INFO:teuthology.orchestra.run.smithi152.stdout:Enabling module streams: 2024-03-14T09:41:10.981 INFO:teuthology.orchestra.run.smithi152.stdout: container-tools rhel8 2024-03-14T09:41:10.981 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-14T09:41:10.981 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-14T09:41:10.981 INFO:teuthology.orchestra.run.smithi152.stdout:========================================================================================================== 2024-03-14T09:41:10.981 INFO:teuthology.orchestra.run.smithi152.stdout:Install 13 Packages 2024-03-14T09:41:10.982 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-14T09:41:10.982 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 26 M 2024-03-14T09:41:10.982 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 87 M 2024-03-14T09:41:10.982 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-14T09:41:10.998 INFO:teuthology.orchestra.run.smithi132.stdout:(8/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 896 kB/s | 239 kB 00:00 2024-03-14T09:41:11.098 INFO:teuthology.orchestra.run.smithi132.stdout:(9/13): netavark-1.7.0-2.module+el8.9.0+21243+a 11 MB/s | 3.7 MB 00:00 2024-03-14T09:41:11.161 INFO:teuthology.orchestra.run.smithi152.stdout:(1/13): criu-libs-3.18-4.module+el8.9.0+20326+3 221 kB/s | 39 kB 00:00 2024-03-14T09:41:11.186 INFO:teuthology.orchestra.run.smithi152.stdout:(2/13): python3-pytoml-0.1.14-5.git7dea353.el8. 122 kB/s | 25 kB 00:00 2024-03-14T09:41:11.240 INFO:teuthology.orchestra.run.smithi132.stdout:(10/13): python3-podman-4.6.0-2.module+el8.9.0+ 541 kB/s | 154 kB 00:00 2024-03-14T09:41:11.244 INFO:teuthology.orchestra.run.smithi152.stdout:(3/13): python3-pyxdg-0.25-16.el8.noarch.rpm 359 kB/s | 94 kB 00:00 2024-03-14T09:41:11.320 INFO:teuthology.orchestra.run.smithi152.stdout:(4/13): yajl-2.1.0-12.el8.x86_64.rpm 258 kB/s | 41 kB 00:00 2024-03-14T09:41:11.362 INFO:teuthology.orchestra.run.smithi152.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 5.7 MB/s | 1.0 MB 00:00 2024-03-14T09:41:11.374 INFO:teuthology.orchestra.run.smithi132.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 9.2 MB/s | 2.5 MB 00:00 2024-03-14T09:41:11.466 INFO:teuthology.orchestra.run.smithi132.stdout:(12/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 17 MB/s | 8.1 MB 00:00 2024-03-14T09:41:11.482 INFO:teuthology.orchestra.run.smithi132.stdout:(13/13): udica-0.2.6-20.module+el8.9.0+21243+a5 204 kB/s | 49 kB 00:00 2024-03-14T09:41:11.483 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-14T09:41:11.483 INFO:teuthology.orchestra.run.smithi132.stdout:Total 18 MB/s | 26 MB 00:01 2024-03-14T09:41:11.504 INFO:teuthology.orchestra.run.smithi152.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2124 3.9 MB/s | 739 kB 00:00 2024-03-14T09:41:11.529 INFO:teuthology.orchestra.run.smithi152.stdout:(7/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 1.4 MB/s | 239 kB 00:00 2024-03-14T09:41:11.754 INFO:teuthology.orchestra.run.smithi152.stdout:(8/13): buildah-1.31.3-3.module+el8.9.0+21243+a 17 MB/s | 8.8 MB 00:00 2024-03-14T09:41:11.766 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-14T09:41:11.771 INFO:teuthology.orchestra.run.smithi152.stdout:(9/13): python3-podman-4.6.0-2.module+el8.9.0+2 635 kB/s | 154 kB 00:00 2024-03-14T09:41:11.788 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-14T09:41:11.788 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-14T09:41:11.863 INFO:teuthology.orchestra.run.smithi152.stdout:(10/13): netavark-1.7.0-2.module+el8.9.0+21243+ 10 MB/s | 3.7 MB 00:00 2024-03-14T09:41:11.989 INFO:teuthology.orchestra.run.smithi152.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 12 MB/s | 2.5 MB 00:00 2024-03-14T09:41:12.014 INFO:teuthology.orchestra.run.smithi152.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+21243+a5 329 kB/s | 49 kB 00:00 2024-03-14T09:41:12.017 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-14T09:41:12.026 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-14T09:41:12.264 INFO:teuthology.orchestra.run.smithi152.stdout:(13/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 16 MB/s | 8.1 MB 00:00 2024-03-14T09:41:12.265 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-14T09:41:12.265 INFO:teuthology.orchestra.run.smithi152.stdout:Total 20 MB/s | 26 MB 00:01 2024-03-14T09:41:12.439 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-14T09:41:12.549 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-14T09:41:12.570 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-14T09:41:12.570 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-14T09:41:12.616 INFO:teuthology.orchestra.run.smithi078.stdout:========================================================================================================== 2024-03-14T09:41:12.617 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-03-14T09:41:12.617 INFO:teuthology.orchestra.run.smithi078.stdout:========================================================================================================== 2024-03-14T09:41:12.617 INFO:teuthology.orchestra.run.smithi078.stdout:Installing group/module packages: 2024-03-14T09:41:12.617 INFO:teuthology.orchestra.run.smithi078.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-03-14T09:41:12.617 INFO:teuthology.orchestra.run.smithi078.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-03-14T09:41:12.617 INFO:teuthology.orchestra.run.smithi078.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-03-14T09:41:12.617 INFO:teuthology.orchestra.run.smithi078.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-03-14T09:41:12.617 INFO:teuthology.orchestra.run.smithi078.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-14T09:41:12.617 INFO:teuthology.orchestra.run.smithi078.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-03-14T09:41:12.617 INFO:teuthology.orchestra.run.smithi078.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-03-14T09:41:12.617 INFO:teuthology.orchestra.run.smithi078.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-03-14T09:41:12.617 INFO:teuthology.orchestra.run.smithi078.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-03-14T09:41:12.618 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-03-14T09:41:12.618 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-03-14T09:41:12.618 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-03-14T09:41:12.618 INFO:teuthology.orchestra.run.smithi078.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-03-14T09:41:12.618 INFO:teuthology.orchestra.run.smithi078.stdout:Installing weak dependencies: 2024-03-14T09:41:12.618 INFO:teuthology.orchestra.run.smithi078.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-14T09:41:12.618 INFO:teuthology.orchestra.run.smithi078.stdout:Installing module profiles: 2024-03-14T09:41:12.618 INFO:teuthology.orchestra.run.smithi078.stdout: container-tools/common 2024-03-14T09:41:12.618 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:41:12.618 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling module streams: 2024-03-14T09:41:12.618 INFO:teuthology.orchestra.run.smithi078.stdout: container-tools rhel8 2024-03-14T09:41:12.618 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:41:12.618 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-14T09:41:12.618 INFO:teuthology.orchestra.run.smithi078.stdout:========================================================================================================== 2024-03-14T09:41:12.619 INFO:teuthology.orchestra.run.smithi078.stdout:Install 13 Packages 2024-03-14T09:41:12.619 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:41:12.619 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 26 M 2024-03-14T09:41:12.619 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 87 M 2024-03-14T09:41:12.619 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-14T09:41:12.689 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-14T09:41:12.800 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-14T09:41:12.807 INFO:teuthology.orchestra.run.smithi078.stdout:(1/13): criu-libs-3.18-4.module+el8.9.0+20326+3 211 kB/s | 39 kB 00:00 2024-03-14T09:41:12.809 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-14T09:41:12.832 INFO:teuthology.orchestra.run.smithi078.stdout:(2/13): python3-pytoml-0.1.14-5.git7dea353.el8. 117 kB/s | 25 kB 00:00 2024-03-14T09:41:12.857 INFO:teuthology.orchestra.run.smithi078.stdout:(3/13): python3-pyxdg-0.25-16.el8.noarch.rpm 396 kB/s | 94 kB 00:00 2024-03-14T09:41:12.899 INFO:teuthology.orchestra.run.smithi078.stdout:(4/13): yajl-2.1.0-12.el8.x86_64.rpm 444 kB/s | 41 kB 00:00 2024-03-14T09:41:12.958 INFO:teuthology.orchestra.run.smithi078.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 8.0 MB/s | 1.0 MB 00:00 2024-03-14T09:41:13.067 INFO:teuthology.orchestra.run.smithi078.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2124 4.3 MB/s | 739 kB 00:00 2024-03-14T09:41:13.117 INFO:teuthology.orchestra.run.smithi078.stdout:(7/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 1.5 MB/s | 239 kB 00:00 2024-03-14T09:41:13.184 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-03-14T09:41:13.276 INFO:teuthology.orchestra.run.smithi078.stdout:(8/13): buildah-1.31.3-3.module+el8.9.0+21243+a 21 MB/s | 8.8 MB 00:00 2024-03-14T09:41:13.309 INFO:teuthology.orchestra.run.smithi078.stdout:(9/13): python3-podman-4.6.0-2.module+el8.9.0+2 799 kB/s | 154 kB 00:00 2024-03-14T09:41:13.381 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-03-14T09:41:13.459 INFO:teuthology.orchestra.run.smithi078.stdout:(10/13): netavark-1.7.0-2.module+el8.9.0+21243+ 9.5 MB/s | 3.7 MB 00:00 2024-03-14T09:41:13.500 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-14T09:41:13.543 INFO:teuthology.orchestra.run.smithi078.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 11 MB/s | 2.5 MB 00:00 2024-03-14T09:41:13.549 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-03-14T09:41:13.577 INFO:teuthology.orchestra.run.smithi078.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+21243+a5 424 kB/s | 49 kB 00:00 2024-03-14T09:41:13.727 INFO:teuthology.orchestra.run.smithi078.stdout:(13/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 18 MB/s | 8.1 MB 00:00 2024-03-14T09:41:13.727 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-14T09:41:13.727 INFO:teuthology.orchestra.run.smithi078.stdout:Total 23 MB/s | 26 MB 00:01 2024-03-14T09:41:13.739 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-03-14T09:41:13.913 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-03-14T09:41:14.001 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-03-14T09:41:14.130 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-03-14T09:41:14.183 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-03-14T09:41:14.195 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-14T09:41:14.220 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-14T09:41:14.220 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-14T09:41:14.365 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-03-14T09:41:14.379 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-03-14T09:41:14.615 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-03-14T09:41:14.704 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-14T09:41:14.714 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-14T09:41:14.827 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-03-14T09:41:15.029 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-03-14T09:41:15.249 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-14T09:41:15.305 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-03-14T09:41:15.405 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-14T09:41:15.453 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-03-14T09:41:15.858 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-03-14T09:41:15.915 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-03-14T09:41:16.119 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-03-14T09:41:16.213 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-14T09:41:16.312 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-03-14T09:41:16.475 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-03-14T09:41:16.480 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-03-14T09:41:16.643 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-03-14T09:41:16.731 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-03-14T09:41:16.879 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-03-14T09:41:16.907 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-03-14T09:41:16.961 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-03-14T09:41:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-03-14T09:41:17.161 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-14T09:41:17.555 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-14T09:41:17.555 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-03-14T09:41:17.555 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-03-14T09:41:17.555 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-03-14T09:41:17.555 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-03-14T09:41:17.555 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-03-14T09:41:17.556 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-03-14T09:41:17.556 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-03-14T09:41:17.556 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-14T09:41:17.556 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-03-14T09:41:17.556 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-03-14T09:41:17.556 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-03-14T09:41:17.556 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-03-14T09:41:17.846 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-03-14T09:41:18.019 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-14T09:41:18.100 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-03-14T09:41:18.247 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-03-14T09:41:18.307 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-14T09:41:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-03-14T09:41:18.724 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-14T09:41:18.725 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-03-14T09:41:18.725 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-03-14T09:41:18.725 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-03-14T09:41:18.725 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-03-14T09:41:18.725 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-03-14T09:41:18.725 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-03-14T09:41:18.725 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-03-14T09:41:18.725 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-14T09:41:18.725 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-03-14T09:41:18.725 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-03-14T09:41:18.725 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-03-14T09:41:18.725 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-03-14T09:41:19.600 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-03-14T09:41:19.854 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-03-14T09:41:20.032 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-14T09:41:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-14T09:41:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-03-14T09:41:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-03-14T09:41:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-03-14T09:41:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-03-14T09:41:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-03-14T09:41:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-03-14T09:41:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-03-14T09:41:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-14T09:41:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-03-14T09:41:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-03-14T09:41:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-03-14T09:41:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-03-14T09:41:26.364 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-03-14T09:41:26.364 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-03-14T09:41:26.908 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-14T09:41:26.908 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-14T09:41:26.908 INFO:teuthology.orchestra.run.smithi132.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:26.908 INFO:teuthology.orchestra.run.smithi132.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:26.908 INFO:teuthology.orchestra.run.smithi132.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-03-14T09:41:26.908 INFO:teuthology.orchestra.run.smithi132.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-03-14T09:41:26.908 INFO:teuthology.orchestra.run.smithi132.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:26.908 INFO:teuthology.orchestra.run.smithi132.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:26.908 INFO:teuthology.orchestra.run.smithi132.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-03-14T09:41:26.908 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-14T09:41:26.908 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-14T09:41:26.908 INFO:teuthology.orchestra.run.smithi132.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:26.909 INFO:teuthology.orchestra.run.smithi132.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:26.909 INFO:teuthology.orchestra.run.smithi132.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-03-14T09:41:26.909 INFO:teuthology.orchestra.run.smithi132.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-14T09:41:26.909 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-14T09:41:26.909 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-14T09:41:27.052 DEBUG:teuthology.parallel:result is None 2024-03-14T09:41:27.872 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-03-14T09:41:27.872 INFO:teuthology.orchestra.run.smithi152.stdout:Installed products updated. 2024-03-14T09:41:28.458 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-14T09:41:28.458 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-14T09:41:28.458 INFO:teuthology.orchestra.run.smithi152.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:28.458 INFO:teuthology.orchestra.run.smithi152.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:28.458 INFO:teuthology.orchestra.run.smithi152.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-03-14T09:41:28.458 INFO:teuthology.orchestra.run.smithi152.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-03-14T09:41:28.458 INFO:teuthology.orchestra.run.smithi152.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:28.458 INFO:teuthology.orchestra.run.smithi152.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:28.458 INFO:teuthology.orchestra.run.smithi152.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-03-14T09:41:28.458 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-14T09:41:28.459 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-14T09:41:28.459 INFO:teuthology.orchestra.run.smithi152.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:28.459 INFO:teuthology.orchestra.run.smithi152.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:28.459 INFO:teuthology.orchestra.run.smithi152.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-03-14T09:41:28.459 INFO:teuthology.orchestra.run.smithi152.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-14T09:41:28.459 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-14T09:41:28.459 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-14T09:41:28.602 DEBUG:teuthology.parallel:result is None 2024-03-14T09:41:28.727 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-03-14T09:41:28.727 INFO:teuthology.orchestra.run.smithi078.stdout:Installed products updated. 2024-03-14T09:41:29.345 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:41:29.345 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-14T09:41:29.345 INFO:teuthology.orchestra.run.smithi078.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:29.345 INFO:teuthology.orchestra.run.smithi078.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:29.345 INFO:teuthology.orchestra.run.smithi078.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-03-14T09:41:29.345 INFO:teuthology.orchestra.run.smithi078.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-03-14T09:41:29.345 INFO:teuthology.orchestra.run.smithi078.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:29.345 INFO:teuthology.orchestra.run.smithi078.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:29.345 INFO:teuthology.orchestra.run.smithi078.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-03-14T09:41:29.345 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-14T09:41:29.345 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-14T09:41:29.345 INFO:teuthology.orchestra.run.smithi078.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:29.345 INFO:teuthology.orchestra.run.smithi078.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-03-14T09:41:29.345 INFO:teuthology.orchestra.run.smithi078.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-03-14T09:41:29.346 INFO:teuthology.orchestra.run.smithi078.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-14T09:41:29.346 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:41:29.346 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-14T09:41:29.580 DEBUG:teuthology.parallel:result is None 2024-03-14T09:41:29.580 INFO:teuthology.run_tasks:Running task install... 2024-03-14T09:41:29.591 DEBUG:teuthology.task.install:project ceph 2024-03-14T09:41:29.591 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '2f465319e3c83d48f999325459bcdfd0346449c0'}} 2024-03-14T09:41:29.592 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '2f465319e3c83d48f999325459bcdfd0346449c0'} 2024-03-14T09:41:29.592 INFO:teuthology.task.install:Using flavor: default 2024-03-14T09:41:29.598 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-14T09:41:29.598 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-03-14T09:41:29.598 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2f465319e3c83d48f999325459bcdfd0346449c0', 'tag': None, 'wait_for_package': False} 2024-03-14T09:41:29.599 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T09:41:29.601 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2f465319e3c83d48f999325459bcdfd0346449c0', 'tag': None, 'wait_for_package': False} 2024-03-14T09:41:29.601 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T09:41:29.603 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2f465319e3c83d48f999325459bcdfd0346449c0', 'tag': None, 'wait_for_package': False} 2024-03-14T09:41:29.603 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T09:41:29.827 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/flavors/default/ 2024-03-14T09:41:29.828 INFO:teuthology.task.install.rpm:Package version is 18.2.2-825.g2f465319 2024-03-14T09:41:29.831 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/flavors/default/ 2024-03-14T09:41:29.831 INFO:teuthology.task.install.rpm:Package version is 18.2.2-825.g2f465319 2024-03-14T09:41:29.844 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/flavors/default/ 2024-03-14T09:41:29.844 INFO:teuthology.task.install.rpm:Package version is 18.2.2-825.g2f465319 2024-03-14T09:41:29.936 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-14T09:41:29.936 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:41:29.936 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-14T09:41:30.003 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-14T09:41:30.003 DEBUG:teuthology.orchestra.run.smithi078:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2f465319e3c83d48f999325459bcdfd0346449c0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-14T09:41:30.009 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-14T09:41:30.009 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-14T09:41:30.009 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-14T09:41:30.016 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-14T09:41:30.016 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-14T09:41:30.016 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-14T09:41:30.042 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-14T09:41:30.042 DEBUG:teuthology.orchestra.run.smithi152:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2f465319e3c83d48f999325459bcdfd0346449c0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-14T09:41:30.050 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-14T09:41:30.050 DEBUG:teuthology.orchestra.run.smithi132:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2f465319e3c83d48f999325459bcdfd0346449c0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-14T09:41:30.109 DEBUG:teuthology.orchestra.run.smithi078:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-14T09:41:30.114 DEBUG:teuthology.orchestra.run.smithi152:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-14T09:41:30.122 DEBUG:teuthology.orchestra.run.smithi132:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-14T09:41:30.195 DEBUG:teuthology.orchestra.run.smithi152:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-14T09:41:30.200 DEBUG:teuthology.orchestra.run.smithi078:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-14T09:41:30.203 DEBUG:teuthology.orchestra.run.smithi132:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-14T09:41:30.261 INFO:teuthology.orchestra.run.smithi152.stdout:check_obsoletes = 1 2024-03-14T09:41:30.263 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum clean all 2024-03-14T09:41:30.264 INFO:teuthology.orchestra.run.smithi078.stdout:check_obsoletes = 1 2024-03-14T09:41:30.267 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum clean all 2024-03-14T09:41:30.271 INFO:teuthology.orchestra.run.smithi132.stdout:check_obsoletes = 1 2024-03-14T09:41:30.272 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum clean all 2024-03-14T09:41:30.641 INFO:teuthology.orchestra.run.smithi152.stdout:Updating Subscription Management repositories. 2024-03-14T09:41:30.650 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-03-14T09:41:30.716 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2024-03-14T09:41:31.777 INFO:teuthology.orchestra.run.smithi152.stdout:61 files removed 2024-03-14T09:41:31.830 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-14T09:41:31.885 INFO:teuthology.orchestra.run.smithi132.stdout:61 files removed 2024-03-14T09:41:31.936 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-14T09:41:32.099 INFO:teuthology.orchestra.run.smithi078.stdout:61 files removed 2024-03-14T09:41:32.155 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-14T09:41:32.166 INFO:teuthology.orchestra.run.smithi152.stdout:Updating Subscription Management repositories. 2024-03-14T09:41:32.273 INFO:teuthology.orchestra.run.smithi132.stdout:Updating Subscription Management repositories. 2024-03-14T09:41:32.601 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2024-03-14T09:41:33.865 INFO:teuthology.orchestra.run.smithi152.stdout:ceph packages for x86_64 146 kB/s | 80 kB 00:00 2024-03-14T09:41:33.989 INFO:teuthology.orchestra.run.smithi132.stdout:ceph packages for x86_64 121 kB/s | 80 kB 00:00 2024-03-14T09:41:34.260 INFO:teuthology.orchestra.run.smithi078.stdout:ceph packages for x86_64 123 kB/s | 80 kB 00:00 2024-03-14T09:41:34.481 INFO:teuthology.orchestra.run.smithi152.stdout:ceph noarch packages 17 kB/s | 9.9 kB 00:00 2024-03-14T09:41:34.615 INFO:teuthology.orchestra.run.smithi132.stdout:ceph noarch packages 17 kB/s | 9.9 kB 00:00 2024-03-14T09:41:34.877 INFO:teuthology.orchestra.run.smithi078.stdout:ceph noarch packages 17 kB/s | 9.9 kB 00:00 2024-03-14T09:41:35.115 INFO:teuthology.orchestra.run.smithi152.stdout:ceph source packages 2.5 kB/s | 1.5 kB 00:00 2024-03-14T09:41:35.290 INFO:teuthology.orchestra.run.smithi132.stdout:ceph source packages 2.3 kB/s | 1.5 kB 00:00 2024-03-14T09:41:35.432 INFO:teuthology.orchestra.run.smithi132.stdout:Copr repo for python3-asyncssh owned by ceph 30 kB/s | 3.5 kB 00:00 2024-03-14T09:41:35.484 INFO:teuthology.orchestra.run.smithi152.stdout:Copr repo for python3-asyncssh owned by ceph 10 kB/s | 3.5 kB 00:00 2024-03-14T09:41:35.486 INFO:teuthology.orchestra.run.smithi078.stdout:ceph source packages 2.6 kB/s | 1.5 kB 00:00 2024-03-14T09:41:35.637 INFO:teuthology.orchestra.run.smithi078.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2024-03-14T09:41:36.644 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-03-14T09:41:36.919 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-03-14T09:41:37.337 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 8.9 MB/s | 16 MB 00:01 2024-03-14T09:41:40.267 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 333 kB/s | 24 kB 00:00 2024-03-14T09:41:40.927 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 245 kB/s | 24 kB 00:00 2024-03-14T09:41:41.071 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 420 kB/s | 24 kB 00:00 2024-03-14T09:41:41.460 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 59 MB/s | 66 MB 00:01 2024-03-14T09:41:42.127 INFO:teuthology.orchestra.run.smithi152.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 59 MB/s | 66 MB 00:01 2024-03-14T09:41:42.456 INFO:teuthology.orchestra.run.smithi078.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 51 MB/s | 66 MB 00:01 2024-03-14T09:41:52.211 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 48 MB/s | 59 MB 00:01 2024-03-14T09:41:52.295 INFO:teuthology.orchestra.run.smithi152.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 12 MB/s | 8.8 MB 00:00 2024-03-14T09:41:52.882 INFO:teuthology.orchestra.run.smithi078.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 20 MB/s | 8.8 MB 00:00 2024-03-14T09:41:54.996 INFO:teuthology.orchestra.run.smithi152.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 60 MB/s | 59 MB 00:00 2024-03-14T09:41:55.884 INFO:teuthology.orchestra.run.smithi078.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 52 MB/s | 59 MB 00:01 2024-03-14T09:42:02.737 INFO:teuthology.orchestra.run.smithi132.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 27 MB/s | 8.8 MB 00:00 2024-03-14T09:42:09.710 INFO:teuthology.orchestra.run.smithi132.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-14T09:42:09.711 INFO:teuthology.orchestra.run.smithi132.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-14T09:42:09.730 INFO:teuthology.orchestra.run.smithi132.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-03-14T09:42:09.732 INFO:teuthology.orchestra.run.smithi132.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-03-14T09:42:09.733 INFO:teuthology.orchestra.run.smithi132.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-03-14T09:42:09.735 INFO:teuthology.orchestra.run.smithi132.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-03-14T09:42:09.737 INFO:teuthology.orchestra.run.smithi132.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-03-14T09:42:09.742 INFO:teuthology.orchestra.run.smithi132.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-03-14T09:42:09.743 INFO:teuthology.orchestra.run.smithi132.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-03-14T09:42:09.958 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-14T09:42:09.972 INFO:teuthology.orchestra.run.smithi132.stdout:=================================================================================================================================== 2024-03-14T09:42:09.973 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-14T09:42:09.973 INFO:teuthology.orchestra.run.smithi132.stdout:=================================================================================================================================== 2024-03-14T09:42:09.973 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-14T09:42:09.973 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-03-14T09:42:09.973 INFO:teuthology.orchestra.run.smithi132.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-03-14T09:42:09.973 INFO:teuthology.orchestra.run.smithi132.stdout: ceph x86_64 2:18.2.2-825.g2f465319.el8 ceph 6.4 k 2024-03-14T09:42:09.973 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base x86_64 2:18.2.2-825.g2f465319.el8 ceph 5.4 M 2024-03-14T09:42:09.973 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse x86_64 2:18.2.2-825.g2f465319.el8 ceph 857 k 2024-03-14T09:42:09.973 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-825.g2f465319.el8 ceph 165 k 2024-03-14T09:42:09.973 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr x86_64 2:18.2.2-825.g2f465319.el8 ceph 1.6 M 2024-03-14T09:42:09.973 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 137 k 2024-03-14T09:42:09.973 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 1.7 M 2024-03-14T09:42:09.973 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 7.4 M 2024-03-14T09:42:09.974 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 49 k 2024-03-14T09:42:09.974 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw x86_64 2:18.2.2-825.g2f465319.el8 ceph 13 M 2024-03-14T09:42:09.974 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test x86_64 2:18.2.2-825.g2f465319.el8 ceph 47 M 2024-03-14T09:42:09.974 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 263 k 2024-03-14T09:42:09.974 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 170 k 2024-03-14T09:42:09.974 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-mirror x86_64 2:18.2.2-825.g2f465319.el8 ceph 261 k 2024-03-14T09:42:09.974 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-top noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 18 k 2024-03-14T09:42:09.974 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-03-14T09:42:09.974 INFO:teuthology.orchestra.run.smithi132.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-03-14T09:42:09.974 INFO:teuthology.orchestra.run.smithi132.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-03-14T09:42:09.974 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel x86_64 2:18.2.2-825.g2f465319.el8 ceph 31 k 2024-03-14T09:42:09.974 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2 x86_64 2:18.2.2-825.g2f465319.el8 ceph 779 k 2024-03-14T09:42:09.974 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel x86_64 2:18.2.2-825.g2f465319.el8 ceph 139 k 2024-03-14T09:42:09.975 INFO:teuthology.orchestra.run.smithi132.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-03-14T09:42:09.975 INFO:teuthology.orchestra.run.smithi132.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-03-14T09:42:09.975 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-03-14T09:42:09.975 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-03-14T09:42:09.975 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-03-14T09:42:09.975 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-03-14T09:42:09.975 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs x86_64 2:18.2.2-825.g2f465319.el8 ceph 197 k 2024-03-14T09:42:09.975 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados x86_64 2:18.2.2-825.g2f465319.el8 ceph 372 k 2024-03-14T09:42:09.975 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd x86_64 2:18.2.2-825.g2f465319.el8 ceph 365 k 2024-03-14T09:42:09.975 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw x86_64 2:18.2.2-825.g2f465319.el8 ceph 107 k 2024-03-14T09:42:09.975 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse x86_64 2:18.2.2-825.g2f465319.el8 ceph 95 k 2024-03-14T09:42:09.975 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror x86_64 2:18.2.2-825.g2f465319.el8 ceph 3.5 M 2024-03-14T09:42:09.975 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd x86_64 2:18.2.2-825.g2f465319.el8 ceph 182 k 2024-03-14T09:42:09.975 INFO:teuthology.orchestra.run.smithi132.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-03-14T09:42:09.975 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrading: 2024-03-14T09:42:09.976 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-03-14T09:42:09.976 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-03-14T09:42:09.976 INFO:teuthology.orchestra.run.smithi132.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-03-14T09:42:09.976 INFO:teuthology.orchestra.run.smithi132.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-14T09:42:09.976 INFO:teuthology.orchestra.run.smithi132.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-03-14T09:42:09.976 INFO:teuthology.orchestra.run.smithi132.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-03-14T09:42:09.976 INFO:teuthology.orchestra.run.smithi132.stdout: librados2 x86_64 2:18.2.2-825.g2f465319.el8 ceph 3.7 M 2024-03-14T09:42:09.976 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1 x86_64 2:18.2.2-825.g2f465319.el8 ceph 3.6 M 2024-03-14T09:42:09.976 INFO:teuthology.orchestra.run.smithi132.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-03-14T09:42:09.976 INFO:teuthology.orchestra.run.smithi132.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-03-14T09:42:09.976 INFO:teuthology.orchestra.run.smithi132.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-03-14T09:42:09.976 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-03-14T09:42:09.976 INFO:teuthology.orchestra.run.smithi132.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-03-14T09:42:09.976 INFO:teuthology.orchestra.run.smithi132.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-03-14T09:42:09.976 INFO:teuthology.orchestra.run.smithi132.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-03-14T09:42:09.977 INFO:teuthology.orchestra.run.smithi132.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-03-14T09:42:09.977 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-14T09:42:09.977 INFO:teuthology.orchestra.run.smithi132.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-03-14T09:42:09.977 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common x86_64 2:18.2.2-825.g2f465319.el8 ceph 22 M 2024-03-14T09:42:09.977 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 23 k 2024-03-14T09:42:09.977 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds x86_64 2:18.2.2-825.g2f465319.el8 ceph 2.3 M 2024-03-14T09:42:09.977 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 245 k 2024-03-14T09:42:09.977 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon x86_64 2:18.2.2-825.g2f465319.el8 ceph 4.6 M 2024-03-14T09:42:09.977 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd x86_64 2:18.2.2-825.g2f465319.el8 ceph 18 M 2024-03-14T09:42:09.977 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 15 k 2024-03-14T09:42:09.977 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux x86_64 2:18.2.2-825.g2f465319.el8 ceph 24 k 2024-03-14T09:42:09.977 INFO:teuthology.orchestra.run.smithi132.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-03-14T09:42:09.977 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-14T09:42:09.978 INFO:teuthology.orchestra.run.smithi132.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-03-14T09:42:09.978 INFO:teuthology.orchestra.run.smithi132.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-03-14T09:42:09.978 INFO:teuthology.orchestra.run.smithi132.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-03-14T09:42:09.978 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite x86_64 2:18.2.2-825.g2f465319.el8 ceph 142 k 2024-03-14T09:42:09.978 INFO:teuthology.orchestra.run.smithi132.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-03-14T09:42:09.978 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-03-14T09:42:09.978 INFO:teuthology.orchestra.run.smithi132.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-14T09:42:09.978 INFO:teuthology.orchestra.run.smithi132.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-03-14T09:42:09.978 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-03-14T09:42:09.978 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-03-14T09:42:09.978 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1 x86_64 2:18.2.2-825.g2f465319.el8 ceph 474 k 2024-03-14T09:42:09.978 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-03-14T09:42:09.978 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2 x86_64 2:18.2.2-825.g2f465319.el8 ceph 6.5 M 2024-03-14T09:42:09.978 INFO:teuthology.orchestra.run.smithi132.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-03-14T09:42:09.978 INFO:teuthology.orchestra.run.smithi132.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-03-14T09:42:09.979 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-14T09:42:09.979 INFO:teuthology.orchestra.run.smithi132.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-03-14T09:42:09.979 INFO:teuthology.orchestra.run.smithi132.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-03-14T09:42:09.979 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-03-14T09:42:09.979 INFO:teuthology.orchestra.run.smithi132.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-03-14T09:42:09.979 INFO:teuthology.orchestra.run.smithi132.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-03-14T09:42:09.979 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-03-14T09:42:09.979 INFO:teuthology.orchestra.run.smithi132.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-03-14T09:42:09.979 INFO:teuthology.orchestra.run.smithi132.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-03-14T09:42:09.979 INFO:teuthology.orchestra.run.smithi132.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-14T09:42:09.979 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-14T09:42:09.979 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-03-14T09:42:09.979 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-14T09:42:09.979 INFO:teuthology.orchestra.run.smithi132.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-14T09:42:09.979 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse x86_64 2:18.2.2-825.g2f465319.el8 ceph 45 k 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common x86_64 2:18.2.2-825.g2f465319.el8 ceph 125 k 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-14T09:42:09.980 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-03-14T09:42:09.981 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-14T09:42:09.981 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-14T09:42:09.981 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-14T09:42:09.981 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-14T09:42:09.981 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-03-14T09:42:09.981 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-03-14T09:42:09.981 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-03-14T09:42:09.981 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-14T09:42:09.981 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-14T09:42:09.981 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-03-14T09:42:09.981 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-03-14T09:42:09.981 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-03-14T09:42:09.981 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-03-14T09:42:09.981 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-03-14T09:42:09.981 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-03-14T09:42:09.981 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-14T09:42:09.982 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-14T09:42:09.982 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-14T09:42:09.982 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-14T09:42:09.982 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-03-14T09:42:09.982 INFO:teuthology.orchestra.run.smithi132.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-14T09:42:09.982 INFO:teuthology.orchestra.run.smithi132.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-14T09:42:09.982 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-14T09:42:09.982 INFO:teuthology.orchestra.run.smithi132.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-14T09:42:09.982 INFO:teuthology.orchestra.run.smithi132.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-14T09:42:09.982 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-14T09:42:09.982 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-14T09:42:09.982 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-14T09:42:09.982 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-03-14T09:42:09.982 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-14T09:42:09.982 INFO:teuthology.orchestra.run.smithi132.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-03-14T09:42:09.983 INFO:teuthology.orchestra.run.smithi132.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-14T09:42:09.983 INFO:teuthology.orchestra.run.smithi132.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-03-14T09:42:09.983 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-03-14T09:42:09.983 INFO:teuthology.orchestra.run.smithi132.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-03-14T09:42:09.983 INFO:teuthology.orchestra.run.smithi132.stdout:Enabling module streams: 2024-03-14T09:42:09.983 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql 10 2024-03-14T09:42:09.983 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-14T09:42:09.983 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-14T09:42:09.983 INFO:teuthology.orchestra.run.smithi132.stdout:=================================================================================================================================== 2024-03-14T09:42:09.983 INFO:teuthology.orchestra.run.smithi132.stdout:Install 130 Packages 2024-03-14T09:42:09.983 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrade 16 Packages 2024-03-14T09:42:09.983 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-14T09:42:09.984 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 212 M 2024-03-14T09:42:09.984 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-14T09:42:10.529 INFO:teuthology.orchestra.run.smithi152.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-14T09:42:10.529 INFO:teuthology.orchestra.run.smithi152.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-14T09:42:10.547 INFO:teuthology.orchestra.run.smithi152.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-03-14T09:42:10.549 INFO:teuthology.orchestra.run.smithi152.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-03-14T09:42:10.550 INFO:teuthology.orchestra.run.smithi152.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-03-14T09:42:10.552 INFO:teuthology.orchestra.run.smithi152.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-03-14T09:42:10.554 INFO:teuthology.orchestra.run.smithi152.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-03-14T09:42:10.559 INFO:teuthology.orchestra.run.smithi152.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-03-14T09:42:10.560 INFO:teuthology.orchestra.run.smithi152.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-03-14T09:42:10.721 INFO:teuthology.orchestra.run.smithi132.stdout:(1/146): ceph-18.2.2-825.g2f465319.el8.x86_64.r 23 kB/s | 6.4 kB 00:00 2024-03-14T09:42:10.775 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-03-14T09:42:10.789 INFO:teuthology.orchestra.run.smithi152.stdout:=================================================================================================================================== 2024-03-14T09:42:10.789 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-03-14T09:42:10.790 INFO:teuthology.orchestra.run.smithi152.stdout:=================================================================================================================================== 2024-03-14T09:42:10.790 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-03-14T09:42:10.790 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-03-14T09:42:10.790 INFO:teuthology.orchestra.run.smithi152.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-03-14T09:42:10.790 INFO:teuthology.orchestra.run.smithi152.stdout: ceph x86_64 2:18.2.2-825.g2f465319.el8 ceph 6.4 k 2024-03-14T09:42:10.790 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base x86_64 2:18.2.2-825.g2f465319.el8 ceph 5.4 M 2024-03-14T09:42:10.790 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse x86_64 2:18.2.2-825.g2f465319.el8 ceph 857 k 2024-03-14T09:42:10.790 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-825.g2f465319.el8 ceph 165 k 2024-03-14T09:42:10.790 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr x86_64 2:18.2.2-825.g2f465319.el8 ceph 1.6 M 2024-03-14T09:42:10.790 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 137 k 2024-03-14T09:42:10.790 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 1.7 M 2024-03-14T09:42:10.790 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 7.4 M 2024-03-14T09:42:10.791 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 49 k 2024-03-14T09:42:10.791 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw x86_64 2:18.2.2-825.g2f465319.el8 ceph 13 M 2024-03-14T09:42:10.791 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test x86_64 2:18.2.2-825.g2f465319.el8 ceph 47 M 2024-03-14T09:42:10.791 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 263 k 2024-03-14T09:42:10.791 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 170 k 2024-03-14T09:42:10.791 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-mirror x86_64 2:18.2.2-825.g2f465319.el8 ceph 261 k 2024-03-14T09:42:10.791 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-top noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 18 k 2024-03-14T09:42:10.791 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-03-14T09:42:10.791 INFO:teuthology.orchestra.run.smithi152.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-03-14T09:42:10.791 INFO:teuthology.orchestra.run.smithi152.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-03-14T09:42:10.791 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel x86_64 2:18.2.2-825.g2f465319.el8 ceph 31 k 2024-03-14T09:42:10.792 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2 x86_64 2:18.2.2-825.g2f465319.el8 ceph 779 k 2024-03-14T09:42:10.792 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel x86_64 2:18.2.2-825.g2f465319.el8 ceph 139 k 2024-03-14T09:42:10.792 INFO:teuthology.orchestra.run.smithi152.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-03-14T09:42:10.792 INFO:teuthology.orchestra.run.smithi152.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-03-14T09:42:10.792 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-03-14T09:42:10.792 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-03-14T09:42:10.792 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-03-14T09:42:10.792 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-03-14T09:42:10.792 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs x86_64 2:18.2.2-825.g2f465319.el8 ceph 197 k 2024-03-14T09:42:10.792 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados x86_64 2:18.2.2-825.g2f465319.el8 ceph 372 k 2024-03-14T09:42:10.792 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd x86_64 2:18.2.2-825.g2f465319.el8 ceph 365 k 2024-03-14T09:42:10.792 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw x86_64 2:18.2.2-825.g2f465319.el8 ceph 107 k 2024-03-14T09:42:10.792 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse x86_64 2:18.2.2-825.g2f465319.el8 ceph 95 k 2024-03-14T09:42:10.792 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror x86_64 2:18.2.2-825.g2f465319.el8 ceph 3.5 M 2024-03-14T09:42:10.793 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd x86_64 2:18.2.2-825.g2f465319.el8 ceph 182 k 2024-03-14T09:42:10.793 INFO:teuthology.orchestra.run.smithi152.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-03-14T09:42:10.793 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrading: 2024-03-14T09:42:10.793 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-03-14T09:42:10.793 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-03-14T09:42:10.793 INFO:teuthology.orchestra.run.smithi152.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-03-14T09:42:10.793 INFO:teuthology.orchestra.run.smithi152.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-14T09:42:10.793 INFO:teuthology.orchestra.run.smithi152.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-03-14T09:42:10.793 INFO:teuthology.orchestra.run.smithi152.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-03-14T09:42:10.793 INFO:teuthology.orchestra.run.smithi152.stdout: librados2 x86_64 2:18.2.2-825.g2f465319.el8 ceph 3.7 M 2024-03-14T09:42:10.793 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1 x86_64 2:18.2.2-825.g2f465319.el8 ceph 3.6 M 2024-03-14T09:42:10.793 INFO:teuthology.orchestra.run.smithi152.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-03-14T09:42:10.794 INFO:teuthology.orchestra.run.smithi152.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-03-14T09:42:10.794 INFO:teuthology.orchestra.run.smithi152.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-03-14T09:42:10.794 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-03-14T09:42:10.794 INFO:teuthology.orchestra.run.smithi152.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-03-14T09:42:10.794 INFO:teuthology.orchestra.run.smithi152.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-03-14T09:42:10.794 INFO:teuthology.orchestra.run.smithi152.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-03-14T09:42:10.794 INFO:teuthology.orchestra.run.smithi152.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-03-14T09:42:10.794 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-03-14T09:42:10.794 INFO:teuthology.orchestra.run.smithi152.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-03-14T09:42:10.794 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common x86_64 2:18.2.2-825.g2f465319.el8 ceph 22 M 2024-03-14T09:42:10.794 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 23 k 2024-03-14T09:42:10.794 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds x86_64 2:18.2.2-825.g2f465319.el8 ceph 2.3 M 2024-03-14T09:42:10.794 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 245 k 2024-03-14T09:42:10.795 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon x86_64 2:18.2.2-825.g2f465319.el8 ceph 4.6 M 2024-03-14T09:42:10.795 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd x86_64 2:18.2.2-825.g2f465319.el8 ceph 18 M 2024-03-14T09:42:10.795 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 15 k 2024-03-14T09:42:10.795 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux x86_64 2:18.2.2-825.g2f465319.el8 ceph 24 k 2024-03-14T09:42:10.795 INFO:teuthology.orchestra.run.smithi152.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-03-14T09:42:10.795 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-14T09:42:10.795 INFO:teuthology.orchestra.run.smithi152.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-03-14T09:42:10.795 INFO:teuthology.orchestra.run.smithi152.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-03-14T09:42:10.795 INFO:teuthology.orchestra.run.smithi152.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-03-14T09:42:10.795 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite x86_64 2:18.2.2-825.g2f465319.el8 ceph 142 k 2024-03-14T09:42:10.795 INFO:teuthology.orchestra.run.smithi152.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-03-14T09:42:10.795 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-03-14T09:42:10.796 INFO:teuthology.orchestra.run.smithi152.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-14T09:42:10.796 INFO:teuthology.orchestra.run.smithi152.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-03-14T09:42:10.796 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-03-14T09:42:10.796 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-03-14T09:42:10.796 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1 x86_64 2:18.2.2-825.g2f465319.el8 ceph 474 k 2024-03-14T09:42:10.796 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-03-14T09:42:10.796 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2 x86_64 2:18.2.2-825.g2f465319.el8 ceph 6.5 M 2024-03-14T09:42:10.796 INFO:teuthology.orchestra.run.smithi152.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-03-14T09:42:10.796 INFO:teuthology.orchestra.run.smithi152.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-03-14T09:42:10.796 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-14T09:42:10.796 INFO:teuthology.orchestra.run.smithi152.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-03-14T09:42:10.796 INFO:teuthology.orchestra.run.smithi152.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-03-14T09:42:10.796 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-03-14T09:42:10.797 INFO:teuthology.orchestra.run.smithi152.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-03-14T09:42:10.797 INFO:teuthology.orchestra.run.smithi152.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-03-14T09:42:10.797 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-03-14T09:42:10.797 INFO:teuthology.orchestra.run.smithi152.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-03-14T09:42:10.797 INFO:teuthology.orchestra.run.smithi152.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-03-14T09:42:10.797 INFO:teuthology.orchestra.run.smithi152.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-14T09:42:10.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-14T09:42:10.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-03-14T09:42:10.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-14T09:42:10.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-14T09:42:10.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-14T09:42:10.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse x86_64 2:18.2.2-825.g2f465319.el8 ceph 45 k 2024-03-14T09:42:10.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common x86_64 2:18.2.2-825.g2f465319.el8 ceph 125 k 2024-03-14T09:42:10.797 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-14T09:42:10.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-03-14T09:42:10.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-14T09:42:10.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-14T09:42:10.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-03-14T09:42:10.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-14T09:42:10.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-14T09:42:10.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-14T09:42:10.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-03-14T09:42:10.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-03-14T09:42:10.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-03-14T09:42:10.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-14T09:42:10.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-14T09:42:10.798 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-14T09:42:10.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-03-14T09:42:10.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-14T09:42:10.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-14T09:42:10.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-14T09:42:10.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-14T09:42:10.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-03-14T09:42:10.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-03-14T09:42:10.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-03-14T09:42:10.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-14T09:42:10.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-14T09:42:10.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-03-14T09:42:10.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-03-14T09:42:10.800 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-03-14T09:42:10.800 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-03-14T09:42:10.800 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-03-14T09:42:10.800 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-03-14T09:42:10.800 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-14T09:42:10.800 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-14T09:42:10.800 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-14T09:42:10.800 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-14T09:42:10.800 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-03-14T09:42:10.800 INFO:teuthology.orchestra.run.smithi152.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-14T09:42:10.800 INFO:teuthology.orchestra.run.smithi152.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-14T09:42:10.800 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-14T09:42:10.800 INFO:teuthology.orchestra.run.smithi152.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-14T09:42:10.800 INFO:teuthology.orchestra.run.smithi152.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-14T09:42:10.800 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-14T09:42:10.801 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-14T09:42:10.801 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-14T09:42:10.801 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-03-14T09:42:10.801 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-14T09:42:10.801 INFO:teuthology.orchestra.run.smithi152.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-03-14T09:42:10.801 INFO:teuthology.orchestra.run.smithi152.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-14T09:42:10.801 INFO:teuthology.orchestra.run.smithi152.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-03-14T09:42:10.801 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-03-14T09:42:10.801 INFO:teuthology.orchestra.run.smithi152.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-03-14T09:42:10.801 INFO:teuthology.orchestra.run.smithi152.stdout:Enabling module streams: 2024-03-14T09:42:10.801 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql 10 2024-03-14T09:42:10.802 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-14T09:42:10.802 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-03-14T09:42:10.802 INFO:teuthology.orchestra.run.smithi152.stdout:=================================================================================================================================== 2024-03-14T09:42:10.802 INFO:teuthology.orchestra.run.smithi152.stdout:Install 130 Packages 2024-03-14T09:42:10.802 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrade 16 Packages 2024-03-14T09:42:10.802 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-14T09:42:10.802 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 212 M 2024-03-14T09:42:10.803 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-03-14T09:42:11.046 INFO:teuthology.orchestra.run.smithi152.stdout:(1/146): ceph-18.2.2-825.g2f465319.el8.x86_64.r 31 kB/s | 6.4 kB 00:00 2024-03-14T09:42:11.254 INFO:teuthology.orchestra.run.smithi132.stdout:(2/146): ceph-fuse-18.2.2-825.g2f465319.el8.x86 1.6 MB/s | 857 kB 00:00 2024-03-14T09:42:11.363 INFO:teuthology.orchestra.run.smithi132.stdout:(3/146): ceph-immutable-object-cache-18.2.2-825 1.5 MB/s | 165 kB 00:00 2024-03-14T09:42:11.854 INFO:teuthology.orchestra.run.smithi152.stdout:(2/146): ceph-fuse-18.2.2-825.g2f465319.el8.x86 1.0 MB/s | 857 kB 00:00 2024-03-14T09:42:11.888 INFO:teuthology.orchestra.run.smithi132.stdout:(4/146): ceph-common-18.2.2-825.g2f465319.el8.x 15 MB/s | 22 MB 00:01 2024-03-14T09:42:11.955 INFO:teuthology.orchestra.run.smithi152.stdout:(3/146): ceph-immutable-object-cache-18.2.2-825 1.6 MB/s | 165 kB 00:00 2024-03-14T09:42:11.963 INFO:teuthology.orchestra.run.smithi132.stdout:(5/146): ceph-base-18.2.2-825.g2f465319.el8.x86 3.5 MB/s | 5.4 MB 00:01 2024-03-14T09:42:12.160 INFO:teuthology.orchestra.run.smithi078.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-14T09:42:12.161 INFO:teuthology.orchestra.run.smithi078.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-14T09:42:12.181 INFO:teuthology.orchestra.run.smithi078.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-03-14T09:42:12.182 INFO:teuthology.orchestra.run.smithi078.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-03-14T09:42:12.184 INFO:teuthology.orchestra.run.smithi078.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-03-14T09:42:12.185 INFO:teuthology.orchestra.run.smithi078.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-03-14T09:42:12.188 INFO:teuthology.orchestra.run.smithi078.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-03-14T09:42:12.193 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-03-14T09:42:12.194 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-03-14T09:42:12.272 INFO:teuthology.orchestra.run.smithi132.stdout:(6/146): ceph-mgr-18.2.2-825.g2f465319.el8.x86_ 4.1 MB/s | 1.6 MB 00:00 2024-03-14T09:42:12.314 INFO:teuthology.orchestra.run.smithi132.stdout:(7/146): ceph-mds-18.2.2-825.g2f465319.el8.x86_ 2.4 MB/s | 2.3 MB 00:00 2024-03-14T09:42:12.430 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-03-14T09:42:12.445 INFO:teuthology.orchestra.run.smithi078.stdout:=================================================================================================================================== 2024-03-14T09:42:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-03-14T09:42:12.445 INFO:teuthology.orchestra.run.smithi078.stdout:=================================================================================================================================== 2024-03-14T09:42:12.445 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-03-14T09:42:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-03-14T09:42:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-03-14T09:42:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: ceph x86_64 2:18.2.2-825.g2f465319.el8 ceph 6.4 k 2024-03-14T09:42:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base x86_64 2:18.2.2-825.g2f465319.el8 ceph 5.4 M 2024-03-14T09:42:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse x86_64 2:18.2.2-825.g2f465319.el8 ceph 857 k 2024-03-14T09:42:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-825.g2f465319.el8 ceph 165 k 2024-03-14T09:42:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr x86_64 2:18.2.2-825.g2f465319.el8 ceph 1.6 M 2024-03-14T09:42:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 137 k 2024-03-14T09:42:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 1.7 M 2024-03-14T09:42:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 7.4 M 2024-03-14T09:42:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 49 k 2024-03-14T09:42:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw x86_64 2:18.2.2-825.g2f465319.el8 ceph 13 M 2024-03-14T09:42:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test x86_64 2:18.2.2-825.g2f465319.el8 ceph 47 M 2024-03-14T09:42:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 263 k 2024-03-14T09:42:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 170 k 2024-03-14T09:42:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror x86_64 2:18.2.2-825.g2f465319.el8 ceph 261 k 2024-03-14T09:42:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 18 k 2024-03-14T09:42:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-03-14T09:42:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-03-14T09:42:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-03-14T09:42:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel x86_64 2:18.2.2-825.g2f465319.el8 ceph 31 k 2024-03-14T09:42:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2 x86_64 2:18.2.2-825.g2f465319.el8 ceph 779 k 2024-03-14T09:42:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel x86_64 2:18.2.2-825.g2f465319.el8 ceph 139 k 2024-03-14T09:42:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-03-14T09:42:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-03-14T09:42:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-03-14T09:42:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-03-14T09:42:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-03-14T09:42:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-03-14T09:42:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs x86_64 2:18.2.2-825.g2f465319.el8 ceph 197 k 2024-03-14T09:42:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados x86_64 2:18.2.2-825.g2f465319.el8 ceph 372 k 2024-03-14T09:42:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd x86_64 2:18.2.2-825.g2f465319.el8 ceph 365 k 2024-03-14T09:42:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw x86_64 2:18.2.2-825.g2f465319.el8 ceph 107 k 2024-03-14T09:42:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse x86_64 2:18.2.2-825.g2f465319.el8 ceph 95 k 2024-03-14T09:42:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror x86_64 2:18.2.2-825.g2f465319.el8 ceph 3.5 M 2024-03-14T09:42:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd x86_64 2:18.2.2-825.g2f465319.el8 ceph 182 k 2024-03-14T09:42:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-03-14T09:42:12.448 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrading: 2024-03-14T09:42:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-03-14T09:42:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-03-14T09:42:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-03-14T09:42:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-14T09:42:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-03-14T09:42:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-03-14T09:42:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: librados2 x86_64 2:18.2.2-825.g2f465319.el8 ceph 3.7 M 2024-03-14T09:42:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1 x86_64 2:18.2.2-825.g2f465319.el8 ceph 3.6 M 2024-03-14T09:42:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-03-14T09:42:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-03-14T09:42:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-03-14T09:42:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-03-14T09:42:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-03-14T09:42:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-03-14T09:42:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-03-14T09:42:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-03-14T09:42:12.449 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-03-14T09:42:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-03-14T09:42:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common x86_64 2:18.2.2-825.g2f465319.el8 ceph 22 M 2024-03-14T09:42:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 23 k 2024-03-14T09:42:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds x86_64 2:18.2.2-825.g2f465319.el8 ceph 2.3 M 2024-03-14T09:42:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 245 k 2024-03-14T09:42:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon x86_64 2:18.2.2-825.g2f465319.el8 ceph 4.6 M 2024-03-14T09:42:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd x86_64 2:18.2.2-825.g2f465319.el8 ceph 18 M 2024-03-14T09:42:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts noarch 2:18.2.2-825.g2f465319.el8 ceph-noarch 15 k 2024-03-14T09:42:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux x86_64 2:18.2.2-825.g2f465319.el8 ceph 24 k 2024-03-14T09:42:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-03-14T09:42:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-14T09:42:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-03-14T09:42:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-03-14T09:42:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-03-14T09:42:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite x86_64 2:18.2.2-825.g2f465319.el8 ceph 142 k 2024-03-14T09:42:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-03-14T09:42:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-03-14T09:42:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-14T09:42:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-03-14T09:42:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-03-14T09:42:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-03-14T09:42:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1 x86_64 2:18.2.2-825.g2f465319.el8 ceph 474 k 2024-03-14T09:42:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-03-14T09:42:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2 x86_64 2:18.2.2-825.g2f465319.el8 ceph 6.5 M 2024-03-14T09:42:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-03-14T09:42:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-03-14T09:42:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-14T09:42:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-03-14T09:42:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-03-14T09:42:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-03-14T09:42:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-03-14T09:42:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-03-14T09:42:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-03-14T09:42:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-03-14T09:42:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-03-14T09:42:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-14T09:42:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-14T09:42:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-03-14T09:42:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-14T09:42:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-14T09:42:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-14T09:42:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse x86_64 2:18.2.2-825.g2f465319.el8 ceph 45 k 2024-03-14T09:42:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common x86_64 2:18.2.2-825.g2f465319.el8 ceph 125 k 2024-03-14T09:42:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-14T09:42:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-03-14T09:42:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-14T09:42:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-14T09:42:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-03-14T09:42:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-14T09:42:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-14T09:42:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-14T09:42:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-03-14T09:42:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-03-14T09:42:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-03-14T09:42:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-14T09:42:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-14T09:42:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-14T09:42:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-03-14T09:42:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-14T09:42:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-14T09:42:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-14T09:42:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-14T09:42:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-03-14T09:42:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-03-14T09:42:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-03-14T09:42:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-14T09:42:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-14T09:42:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-03-14T09:42:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-03-14T09:42:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-03-14T09:42:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-03-14T09:42:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-03-14T09:42:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-03-14T09:42:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-14T09:42:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-14T09:42:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-14T09:42:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-14T09:42:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-03-14T09:42:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-14T09:42:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-14T09:42:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling module streams: 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql 10 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-03-14T09:42:12.456 INFO:teuthology.orchestra.run.smithi078.stdout:=================================================================================================================================== 2024-03-14T09:42:12.457 INFO:teuthology.orchestra.run.smithi078.stdout:Install 130 Packages 2024-03-14T09:42:12.457 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrade 16 Packages 2024-03-14T09:42:12.457 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:42:12.457 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 212 M 2024-03-14T09:42:12.457 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-03-14T09:42:12.522 INFO:teuthology.orchestra.run.smithi152.stdout:(4/146): ceph-mds-18.2.2-825.g2f465319.el8.x86_ 4.0 MB/s | 2.3 MB 00:00 2024-03-14T09:42:12.881 INFO:teuthology.orchestra.run.smithi132.stdout:(8/146): ceph-mon-18.2.2-825.g2f465319.el8.x86_ 5.0 MB/s | 4.6 MB 00:00 2024-03-14T09:42:12.892 INFO:teuthology.orchestra.run.smithi078.stdout:(1/146): ceph-18.2.2-825.g2f465319.el8.x86_64.r 22 kB/s | 6.4 kB 00:00 2024-03-14T09:42:12.989 INFO:teuthology.orchestra.run.smithi132.stdout:(9/146): ceph-selinux-18.2.2-825.g2f465319.el8. 225 kB/s | 24 kB 00:00 2024-03-14T09:42:13.014 INFO:teuthology.orchestra.run.smithi152.stdout:(5/146): ceph-mgr-18.2.2-825.g2f465319.el8.x86_ 3.2 MB/s | 1.6 MB 00:00 2024-03-14T09:42:13.548 INFO:teuthology.orchestra.run.smithi152.stdout:(6/146): ceph-base-18.2.2-825.g2f465319.el8.x86 2.0 MB/s | 5.4 MB 00:02 2024-03-14T09:42:13.801 INFO:teuthology.orchestra.run.smithi078.stdout:(2/146): ceph-fuse-18.2.2-825.g2f465319.el8.x86 943 kB/s | 857 kB 00:00 2024-03-14T09:42:13.943 INFO:teuthology.orchestra.run.smithi078.stdout:(3/146): ceph-immutable-object-cache-18.2.2-825 1.1 MB/s | 165 kB 00:00 2024-03-14T09:42:14.001 INFO:teuthology.orchestra.run.smithi132.stdout:(10/146): ceph-osd-18.2.2-825.g2f465319.el8.x86 10 MB/s | 18 MB 00:01 2024-03-14T09:42:14.090 INFO:teuthology.orchestra.run.smithi132.stdout:(11/146): cephfs-mirror-18.2.2-825.g2f465319.el 2.9 MB/s | 261 kB 00:00 2024-03-14T09:42:14.157 INFO:teuthology.orchestra.run.smithi132.stdout:(12/146): libcephfs-devel-18.2.2-825.g2f465319. 463 kB/s | 31 kB 00:00 2024-03-14T09:42:14.197 INFO:teuthology.orchestra.run.smithi152.stdout:(7/146): ceph-mon-18.2.2-825.g2f465319.el8.x86_ 3.9 MB/s | 4.6 MB 00:01 2024-03-14T09:42:14.249 INFO:teuthology.orchestra.run.smithi132.stdout:(13/146): libcephfs2-18.2.2-825.g2f465319.el8.x 8.3 MB/s | 779 kB 00:00 2024-03-14T09:42:14.316 INFO:teuthology.orchestra.run.smithi132.stdout:(14/146): libcephsqlite-18.2.2-825.g2f465319.el 2.1 MB/s | 142 kB 00:00 2024-03-14T09:42:14.391 INFO:teuthology.orchestra.run.smithi132.stdout:(15/146): librados-devel-18.2.2-825.g2f465319.e 1.8 MB/s | 139 kB 00:00 2024-03-14T09:42:14.474 INFO:teuthology.orchestra.run.smithi132.stdout:(16/146): libradosstriper1-18.2.2-825.g2f465319 5.6 MB/s | 474 kB 00:00 2024-03-14T09:42:15.151 INFO:teuthology.orchestra.run.smithi078.stdout:(4/146): ceph-mds-18.2.2-825.g2f465319.el8.x86_ 1.9 MB/s | 2.3 MB 00:01 2024-03-14T09:42:15.166 INFO:teuthology.orchestra.run.smithi132.stdout:(17/146): librgw2-18.2.2-825.g2f465319.el8.x86_ 9.5 MB/s | 6.5 MB 00:00 2024-03-14T09:42:15.233 INFO:teuthology.orchestra.run.smithi132.stdout:(18/146): python3-ceph-argparse-18.2.2-825.g2f4 672 kB/s | 45 kB 00:00 2024-03-14T09:42:15.300 INFO:teuthology.orchestra.run.smithi132.stdout:(19/146): python3-ceph-common-18.2.2-825.g2f465 1.8 MB/s | 125 kB 00:00 2024-03-14T09:42:15.376 INFO:teuthology.orchestra.run.smithi132.stdout:(20/146): python3-cephfs-18.2.2-825.g2f465319.e 2.6 MB/s | 197 kB 00:00 2024-03-14T09:42:15.451 INFO:teuthology.orchestra.run.smithi132.stdout:(21/146): python3-rados-18.2.2-825.g2f465319.el 4.9 MB/s | 372 kB 00:00 2024-03-14T09:42:15.551 INFO:teuthology.orchestra.run.smithi132.stdout:(22/146): python3-rbd-18.2.2-825.g2f465319.el8. 3.6 MB/s | 365 kB 00:00 2024-03-14T09:42:15.618 INFO:teuthology.orchestra.run.smithi132.stdout:(23/146): python3-rgw-18.2.2-825.g2f465319.el8. 1.6 MB/s | 107 kB 00:00 2024-03-14T09:42:15.685 INFO:teuthology.orchestra.run.smithi132.stdout:(24/146): rbd-fuse-18.2.2-825.g2f465319.el8.x86 1.4 MB/s | 95 kB 00:00 2024-03-14T09:42:15.810 INFO:teuthology.orchestra.run.smithi078.stdout:(5/146): ceph-mgr-18.2.2-825.g2f465319.el8.x86_ 2.4 MB/s | 1.6 MB 00:00 2024-03-14T09:42:16.127 INFO:teuthology.orchestra.run.smithi132.stdout:(25/146): rbd-mirror-18.2.2-825.g2f465319.el8.x 7.8 MB/s | 3.5 MB 00:00 2024-03-14T09:42:16.202 INFO:teuthology.orchestra.run.smithi132.stdout:(26/146): rbd-nbd-18.2.2-825.g2f465319.el8.x86_ 2.4 MB/s | 182 kB 00:00 2024-03-14T09:42:16.260 INFO:teuthology.orchestra.run.smithi078.stdout:(6/146): ceph-base-18.2.2-825.g2f465319.el8.x86 1.5 MB/s | 5.4 MB 00:03 2024-03-14T09:42:16.261 INFO:teuthology.orchestra.run.smithi132.stdout:(27/146): ceph-grafana-dashboards-18.2.2-825.g2 398 kB/s | 23 kB 00:00 2024-03-14T09:42:16.320 INFO:teuthology.orchestra.run.smithi132.stdout:(28/146): ceph-mgr-cephadm-18.2.2-825.g2f465319 2.3 MB/s | 137 kB 00:00 2024-03-14T09:42:16.406 INFO:teuthology.orchestra.run.smithi152.stdout:(8/146): ceph-radosgw-18.2.2-825.g2f465319.el8. 5.9 MB/s | 13 MB 00:02 2024-03-14T09:42:16.478 INFO:teuthology.orchestra.run.smithi132.stdout:(29/146): ceph-radosgw-18.2.2-825.g2f465319.el8 3.1 MB/s | 13 MB 00:04 2024-03-14T09:42:16.506 INFO:teuthology.orchestra.run.smithi152.stdout:(9/146): ceph-selinux-18.2.2-825.g2f465319.el8. 244 kB/s | 24 kB 00:00 2024-03-14T09:42:16.528 INFO:teuthology.orchestra.run.smithi132.stdout:(30/146): ceph-mgr-dashboard-18.2.2-825.g2f4653 8.4 MB/s | 1.7 MB 00:00 2024-03-14T09:42:16.612 INFO:teuthology.orchestra.run.smithi132.stdout:(31/146): ceph-mgr-modules-core-18.2.2-825.g2f4 2.9 MB/s | 245 kB 00:00 2024-03-14T09:42:16.679 INFO:teuthology.orchestra.run.smithi132.stdout:(32/146): ceph-mgr-rook-18.2.2-825.g2f465319.el 738 kB/s | 49 kB 00:00 2024-03-14T09:42:16.746 INFO:teuthology.orchestra.run.smithi132.stdout:(33/146): ceph-prometheus-alerts-18.2.2-825.g2f 222 kB/s | 15 kB 00:00 2024-03-14T09:42:16.822 INFO:teuthology.orchestra.run.smithi132.stdout:(34/146): ceph-volume-18.2.2-825.g2f465319.el8. 3.4 MB/s | 263 kB 00:00 2024-03-14T09:42:16.921 INFO:teuthology.orchestra.run.smithi132.stdout:(35/146): cephadm-18.2.2-825.g2f465319.el8.noar 1.7 MB/s | 170 kB 00:00 2024-03-14T09:42:16.980 INFO:teuthology.orchestra.run.smithi132.stdout:(36/146): cephfs-top-18.2.2-825.g2f465319.el8.n 307 kB/s | 18 kB 00:00 2024-03-14T09:42:17.389 INFO:teuthology.orchestra.run.smithi132.stdout:(37/146): gperftools-libs-2.7-9.el8.x86_64.rpm 750 kB/s | 306 kB 00:00 2024-03-14T09:42:17.472 INFO:teuthology.orchestra.run.smithi132.stdout:(38/146): liboath-2.6.2-3.el8.x86_64.rpm 706 kB/s | 59 kB 00:00 2024-03-14T09:42:17.527 INFO:teuthology.orchestra.run.smithi078.stdout:(7/146): ceph-mon-18.2.2-825.g2f465319.el8.x86_ 2.7 MB/s | 4.6 MB 00:01 2024-03-14T09:42:17.556 INFO:teuthology.orchestra.run.smithi132.stdout:(39/146): libunwind-1.3.1-3.el8.x86_64.rpm 901 kB/s | 75 kB 00:00 2024-03-14T09:42:17.706 INFO:teuthology.orchestra.run.smithi132.stdout:(40/146): python3-asyncssh-2.7.0-2.el8.noarch.r 3.1 MB/s | 476 kB 00:00 2024-03-14T09:42:17.774 INFO:teuthology.orchestra.run.smithi132.stdout:(41/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 654 kB/s | 44 kB 00:00 2024-03-14T09:42:17.865 INFO:teuthology.orchestra.run.smithi132.stdout:(42/146): python3-beautifulsoup4-4.6.3-2.el8.1. 2.0 MB/s | 185 kB 00:00 2024-03-14T09:42:17.941 INFO:teuthology.orchestra.run.smithi132.stdout:(43/146): python3-cachetools-3.1.1-4.el8.noarch 440 kB/s | 33 kB 00:00 2024-03-14T09:42:18.016 INFO:teuthology.orchestra.run.smithi132.stdout:(44/146): python3-certifi-2018.10.15-7.el8.noar 218 kB/s | 16 kB 00:00 2024-03-14T09:42:18.100 INFO:teuthology.orchestra.run.smithi132.stdout:(45/146): python3-cheroot-8.5.2-1.el8.noarch.rp 2.0 MB/s | 173 kB 00:00 2024-03-14T09:42:18.183 INFO:teuthology.orchestra.run.smithi132.stdout:(46/146): python3-cherrypy-18.4.0-1.el8.noarch. 4.5 MB/s | 384 kB 00:00 2024-03-14T09:42:18.259 INFO:teuthology.orchestra.run.smithi132.stdout:(47/146): python3-google-auth-1.1.1-10.el8.noar 1.1 MB/s | 81 kB 00:00 2024-03-14T09:42:18.334 INFO:teuthology.orchestra.run.smithi132.stdout:(48/146): python3-jaraco-6.2-6.el8.noarch.rpm 148 kB/s | 11 kB 00:00 2024-03-14T09:42:18.418 INFO:teuthology.orchestra.run.smithi132.stdout:(49/146): python3-jaraco-functools-2.0-4.el8.no 210 kB/s | 18 kB 00:00 2024-03-14T09:42:18.543 INFO:teuthology.orchestra.run.smithi132.stdout:(50/146): python3-kubernetes-11.0.0-6.el8.noarc 9.3 MB/s | 1.2 MB 00:00 2024-03-14T09:42:18.619 INFO:teuthology.orchestra.run.smithi132.stdout:(51/146): python3-logutils-0.3.5-11.el8.noarch. 649 kB/s | 49 kB 00:00 2024-03-14T09:42:18.694 INFO:teuthology.orchestra.run.smithi132.stdout:(52/146): python3-more-itertools-7.2.0-3.el8.no 784 kB/s | 59 kB 00:00 2024-03-14T09:42:18.769 INFO:teuthology.orchestra.run.smithi132.stdout:(53/146): python3-natsort-7.1.1-2.el8.noarch.rp 805 kB/s | 60 kB 00:00 2024-03-14T09:42:18.853 INFO:teuthology.orchestra.run.smithi132.stdout:(54/146): python3-pecan-1.3.2-9.el8.noarch.rpm 3.3 MB/s | 283 kB 00:00 2024-03-14T09:42:18.928 INFO:teuthology.orchestra.run.smithi132.stdout:(55/146): python3-portend-2.6-1.el8.noarch.rpm 208 kB/s | 16 kB 00:00 2024-03-14T09:42:19.020 INFO:teuthology.orchestra.run.smithi132.stdout:(56/146): ceph-mgr-diskprediction-local-18.2.2- 2.9 MB/s | 7.4 MB 00:02 2024-03-14T09:42:19.045 INFO:teuthology.orchestra.run.smithi132.stdout:(57/146): python3-repoze-lru-0.7-6.el8.noarch.r 288 kB/s | 34 kB 00:00 2024-03-14T09:42:19.129 INFO:teuthology.orchestra.run.smithi132.stdout:(58/146): python3-rsa-4.9-2.el8.noarch.rpm 771 kB/s | 64 kB 00:00 2024-03-14T09:42:19.131 INFO:teuthology.orchestra.run.smithi152.stdout:(10/146): ceph-osd-18.2.2-825.g2f465319.el8.x86 3.2 MB/s | 18 MB 00:05 2024-03-14T09:42:19.204 INFO:teuthology.orchestra.run.smithi132.stdout:(59/146): python3-simplegeneric-0.8.1-17.el8.no 252 kB/s | 19 kB 00:00 2024-03-14T09:42:19.265 INFO:teuthology.orchestra.run.smithi152.stdout:(11/146): cephfs-mirror-18.2.2-825.g2f465319.el 1.9 MB/s | 261 kB 00:00 2024-03-14T09:42:19.279 INFO:teuthology.orchestra.run.smithi132.stdout:(60/146): python3-singledispatch-3.4.0.3-18.el8 319 kB/s | 24 kB 00:00 2024-03-14T09:42:19.348 INFO:teuthology.orchestra.run.smithi152.stdout:(12/146): libcephfs-devel-18.2.2-825.g2f465319. 372 kB/s | 31 kB 00:00 2024-03-14T09:42:19.355 INFO:teuthology.orchestra.run.smithi132.stdout:(61/146): python3-tempora-1.14.1-5.el8.noarch.r 389 kB/s | 29 kB 00:00 2024-03-14T09:42:19.388 INFO:teuthology.orchestra.run.smithi132.stdout:(62/146): python3-routes-2.4.1-12.el8.noarch.rp 534 kB/s | 196 kB 00:00 2024-03-14T09:42:19.430 INFO:teuthology.orchestra.run.smithi132.stdout:(63/146): python3-trustme-0.6.0-4.el8.noarch.rp 366 kB/s | 27 kB 00:00 2024-03-14T09:42:19.514 INFO:teuthology.orchestra.run.smithi132.stdout:(64/146): python3-webob-1.8.5-1.el8.1.noarch.rp 2.9 MB/s | 251 kB 00:00 2024-03-14T09:42:19.539 INFO:teuthology.orchestra.run.smithi132.stdout:(65/146): python3-waitress-1.4.3-1.el8.noarch.r 1.7 MB/s | 255 kB 00:00 2024-03-14T09:42:19.589 INFO:teuthology.orchestra.run.smithi132.stdout:(66/146): python3-websocket-client-0.56.0-5.el8 815 kB/s | 61 kB 00:00 2024-03-14T09:42:19.615 INFO:teuthology.orchestra.run.smithi132.stdout:(67/146): python3-webtest-2.0.33-1.el8.noarch.r 1.1 MB/s | 85 kB 00:00 2024-03-14T09:42:19.665 INFO:teuthology.orchestra.run.smithi132.stdout:(68/146): python3-zc-lockfile-2.0-2.el8.noarch. 305 kB/s | 23 kB 00:00 2024-03-14T09:42:19.665 INFO:teuthology.orchestra.run.smithi152.stdout:(13/146): libcephfs2-18.2.2-825.g2f465319.el8.x 2.4 MB/s | 779 kB 00:00 2024-03-14T09:42:19.682 INFO:teuthology.orchestra.run.smithi132.stdout:(69/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 4.1 MB/s | 70 kB 00:00 2024-03-14T09:42:19.782 INFO:teuthology.orchestra.run.smithi152.stdout:(14/146): libcephsqlite-18.2.2-825.g2f465319.el 1.2 MB/s | 142 kB 00:00 2024-03-14T09:42:19.824 INFO:teuthology.orchestra.run.smithi132.stdout:(70/146): thrift-0.13.0-2.el8.x86_64.rpm 8.3 MB/s | 1.7 MB 00:00 2024-03-14T09:42:19.849 INFO:teuthology.orchestra.run.smithi132.stdout:(71/146): libverto-devel-0.3.0-5.el8.x86_64.rpm 107 kB/s | 18 kB 00:00 2024-03-14T09:42:19.966 INFO:teuthology.orchestra.run.smithi132.stdout:(72/146): python3-jwt-1.6.1-2.el8.noarch.rpm 300 kB/s | 43 kB 00:00 2024-03-14T09:42:20.000 INFO:teuthology.orchestra.run.smithi132.stdout:(73/146): python3-oauthlib-2.1.0-1.el8.noarch.r 1.0 MB/s | 155 kB 00:00 2024-03-14T09:42:20.066 INFO:teuthology.orchestra.run.smithi152.stdout:(15/146): ceph-common-18.2.2-825.g2f465319.el8. 2.3 MB/s | 22 MB 00:09 2024-03-14T09:42:20.091 INFO:teuthology.orchestra.run.smithi152.stdout:(16/146): librados-devel-18.2.2-825.g2f465319.e 450 kB/s | 139 kB 00:00 2024-03-14T09:42:20.167 INFO:teuthology.orchestra.run.smithi132.stdout:(74/146): python3-requests-oauthlib-1.0.0-1.el8 256 kB/s | 43 kB 00:00 2024-03-14T09:42:20.192 INFO:teuthology.orchestra.run.smithi132.stdout:(75/146): python3-pycparser-2.14-14.el8.noarch. 484 kB/s | 109 kB 00:00 2024-03-14T09:42:20.269 INFO:teuthology.orchestra.run.smithi078.stdout:(8/146): ceph-radosgw-18.2.2-825.g2f465319.el8. 4.8 MB/s | 13 MB 00:02 2024-03-14T09:42:20.319 INFO:teuthology.orchestra.run.smithi078.stdout:(9/146): ceph-selinux-18.2.2-825.g2f465319.el8. 489 kB/s | 24 kB 00:00 2024-03-14T09:42:20.326 INFO:teuthology.orchestra.run.smithi132.stdout:(76/146): libaio-devel-0.3.112-1.el8.x86_64.rpm 118 kB/s | 19 kB 00:00 2024-03-14T09:42:20.351 INFO:teuthology.orchestra.run.smithi132.stdout:(77/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 276 kB/s | 44 kB 00:00 2024-03-14T09:42:20.366 INFO:teuthology.orchestra.run.smithi152.stdout:(17/146): libradosstriper1-18.2.2-825.g2f465319 1.5 MB/s | 474 kB 00:00 2024-03-14T09:42:20.414 INFO:teuthology.orchestra.run.smithi132.stdout:(78/146): pcre2-devel-10.32-2.el8.x86_64.rpm 6.8 MB/s | 605 kB 00:00 2024-03-14T09:42:20.452 INFO:teuthology.orchestra.run.smithi132.stdout:(79/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.2 MB/s | 229 kB 00:00 2024-03-14T09:42:20.467 INFO:teuthology.orchestra.run.smithi152.stdout:(18/146): python3-ceph-argparse-18.2.2-825.g2f4 447 kB/s | 45 kB 00:00 2024-03-14T09:42:20.477 INFO:teuthology.orchestra.run.smithi132.stdout:(80/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 3.4 MB/s | 220 kB 00:00 2024-03-14T09:42:20.502 INFO:teuthology.orchestra.run.smithi132.stdout:(81/146): libselinux-devel-2.9-5.el8.x86_64.rpm 3.9 MB/s | 200 kB 00:00 2024-03-14T09:42:20.536 INFO:teuthology.orchestra.run.smithi132.stdout:(82/146): keyutils-libs-devel-1.5.10-9.el8.x86_ 822 kB/s | 48 kB 00:00 2024-03-14T09:42:20.561 INFO:teuthology.orchestra.run.smithi132.stdout:(83/146): libsepol-devel-2.9-3.el8.x86_64.rpm 1.4 MB/s | 87 kB 00:00 2024-03-14T09:42:20.575 INFO:teuthology.orchestra.run.smithi152.stdout:(19/146): python3-ceph-common-18.2.2-825.g2f465 1.1 MB/s | 125 kB 00:00 2024-03-14T09:42:20.586 INFO:teuthology.orchestra.run.smithi132.stdout:(84/146): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-03-14T09:42:20.612 INFO:teuthology.orchestra.run.smithi132.stdout:(85/146): libcom_err-devel-1.45.6-4.el8.x86_64. 771 kB/s | 39 kB 00:00 2024-03-14T09:42:20.637 INFO:teuthology.orchestra.run.smithi132.stdout:(86/146): fuse-2.9.7-17.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2024-03-14T09:42:20.671 INFO:teuthology.orchestra.run.smithi132.stdout:(87/146): libgfortran-8.5.0-20.el8.x86_64.rpm 11 MB/s | 645 kB 00:00 2024-03-14T09:42:20.696 INFO:teuthology.orchestra.run.smithi132.stdout:(88/146): libquadmath-8.5.0-20.el8.x86_64.rpm 2.9 MB/s | 172 kB 00:00 2024-03-14T09:42:20.730 INFO:teuthology.orchestra.run.smithi132.stdout:(89/146): libuuid-devel-2.32.1-43.el8.x86_64.rp 1.7 MB/s | 100 kB 00:00 2024-03-14T09:42:20.734 INFO:teuthology.orchestra.run.smithi152.stdout:(20/146): python3-cephfs-18.2.2-825.g2f465319.e 1.2 MB/s | 197 kB 00:00 2024-03-14T09:42:20.755 INFO:teuthology.orchestra.run.smithi132.stdout:(90/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 4.0 MB/s | 238 kB 00:00 2024-03-14T09:42:20.780 INFO:teuthology.orchestra.run.smithi132.stdout:(91/146): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 1.4 MB/s | 73 kB 00:00 2024-03-14T09:42:20.805 INFO:teuthology.orchestra.run.smithi132.stdout:(92/146): NetworkManager-initscripts-updown-1.4 2.8 MB/s | 144 kB 00:00 2024-03-14T09:42:20.822 INFO:teuthology.orchestra.run.smithi132.stdout:(93/146): elfutils-libelf-devel-0.189-3.el8.x86 1.4 MB/s | 62 kB 00:00 2024-03-14T09:42:20.873 INFO:teuthology.orchestra.run.smithi132.stdout:(94/146): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 8.2 MB/s | 562 kB 00:00 2024-03-14T09:42:20.940 INFO:teuthology.orchestra.run.smithi132.stdout:(95/146): openssl-devel-1.1.1k-12.el8_9.x86_64. 20 MB/s | 2.3 MB 00:00 2024-03-14T09:42:20.957 INFO:teuthology.orchestra.run.smithi132.stdout:(96/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 564 kB/s | 47 kB 00:00 2024-03-14T09:42:20.999 INFO:teuthology.orchestra.run.smithi132.stdout:(97/146): python3-cryptography-3.2.1-7.el8_9.x8 9.3 MB/s | 559 kB 00:00 2024-03-14T09:42:21.034 INFO:teuthology.orchestra.run.smithi152.stdout:(21/146): python3-rados-18.2.2-825.g2f465319.el 1.2 MB/s | 372 kB 00:00 2024-03-14T09:42:21.050 INFO:teuthology.orchestra.run.smithi132.stdout:(98/146): python3-jsonpointer-1.10-11.el8.noarc 216 kB/s | 20 kB 00:00 2024-03-14T09:42:21.074 INFO:teuthology.orchestra.run.smithi132.stdout:(99/146): python3-prettytable-0.7.2-14.el8.noar 586 kB/s | 44 kB 00:00 2024-03-14T09:42:21.108 INFO:teuthology.orchestra.run.smithi132.stdout:(100/146): python3-pyasn1-modules-0.3.7-6.el8.n 1.9 MB/s | 110 kB 00:00 2024-03-14T09:42:21.141 INFO:teuthology.orchestra.run.smithi132.stdout:(101/146): python3-jsonpatch-1.21-2.el8.noarch. 396 kB/s | 27 kB 00:00 2024-03-14T09:42:21.175 INFO:teuthology.orchestra.run.smithi132.stdout:(102/146): python3-pyasn1-0.3.7-6.el8.noarch.rp 1.8 MB/s | 126 kB 00:00 2024-03-14T09:42:21.209 INFO:teuthology.orchestra.run.smithi132.stdout:(103/146): python3-werkzeug-0.12.2-4.el8.noarch 6.7 MB/s | 457 kB 00:00 2024-03-14T09:42:21.218 INFO:teuthology.orchestra.run.smithi152.stdout:(22/146): python3-rbd-18.2.2-825.g2f465319.el8. 1.9 MB/s | 365 kB 00:00 2024-03-14T09:42:21.243 INFO:teuthology.orchestra.run.smithi132.stdout:(104/146): bison-3.0.4-10.el8.x86_64.rpm 10 MB/s | 688 kB 00:00 2024-03-14T09:42:21.285 INFO:teuthology.orchestra.run.smithi132.stdout:(105/146): flex-2.6.1-9.el8.x86_64.rpm 4.1 MB/s | 320 kB 00:00 2024-03-14T09:42:21.327 INFO:teuthology.orchestra.run.smithi152.stdout:(23/146): python3-rgw-18.2.2-825.g2f465319.el8. 986 kB/s | 107 kB 00:00 2024-03-14T09:42:21.327 INFO:teuthology.orchestra.run.smithi132.stdout:(106/146): libtool-2.4.6-25.el8.x86_64.rpm 8.3 MB/s | 709 kB 00:00 2024-03-14T09:42:21.352 INFO:teuthology.orchestra.run.smithi132.stdout:(107/146): python3-markupsafe-0.23-19.el8.x86_6 579 kB/s | 39 kB 00:00 2024-03-14T09:42:21.394 INFO:teuthology.orchestra.run.smithi132.stdout:(108/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.8 MB/s | 259 kB 00:00 2024-03-14T09:42:21.419 INFO:teuthology.orchestra.run.smithi132.stdout:(109/146): uuid-1.6.2-43.el8.x86_64.rpm 960 kB/s | 64 kB 00:00 2024-03-14T09:42:21.443 INFO:teuthology.orchestra.run.smithi152.stdout:(24/146): rbd-fuse-18.2.2-825.g2f465319.el8.x86 816 kB/s | 95 kB 00:00 2024-03-14T09:42:21.461 INFO:teuthology.orchestra.run.smithi132.stdout:(110/146): python3-pyOpenSSL-19.0.0-1.el8.noarc 1.5 MB/s | 103 kB 00:00 2024-03-14T09:42:21.545 INFO:teuthology.orchestra.run.smithi132.stdout:(111/146): python3-jinja2-2.10.1-3.el8.noarch.r 6.3 MB/s | 538 kB 00:00 2024-03-14T09:42:21.686 INFO:teuthology.orchestra.run.smithi132.stdout:(112/146): python3-babel-2.5.1-7.el8.noarch.rpm 18 MB/s | 4.8 MB 00:00 2024-03-14T09:42:21.761 INFO:teuthology.orchestra.run.smithi078.stdout:(10/146): ceph-common-18.2.2-825.g2f465319.el8. 2.4 MB/s | 22 MB 00:09 2024-03-14T09:42:21.779 INFO:teuthology.orchestra.run.smithi132.stdout:(113/146): python3-numpy-1.14.3-10.el8.x86_64.r 16 MB/s | 3.7 MB 00:00 2024-03-14T09:42:21.812 INFO:teuthology.orchestra.run.smithi132.stdout:(114/146): socat-1.7.4.1-1.el8.x86_64.rpm 2.5 MB/s | 323 kB 00:00 2024-03-14T09:42:21.862 INFO:teuthology.orchestra.run.smithi132.stdout:(115/146): python3-numpy-f2py-1.14.3-10.el8.x86 2.6 MB/s | 225 kB 00:00 2024-03-14T09:42:21.896 INFO:teuthology.orchestra.run.smithi132.stdout:(116/146): librdkafka-0.11.4-3.el8.x86_64.rpm 4.1 MB/s | 354 kB 00:00 2024-03-14T09:42:21.945 INFO:teuthology.orchestra.run.smithi078.stdout:(11/146): cephfs-mirror-18.2.2-825.g2f465319.el 1.4 MB/s | 261 kB 00:00 2024-03-14T09:42:21.980 INFO:teuthology.orchestra.run.smithi132.stdout:(117/146): python3-mako-1.0.6-14.el8.noarch.rpm 1.8 MB/s | 157 kB 00:00 2024-03-14T09:42:22.036 INFO:teuthology.orchestra.run.smithi078.stdout:(12/146): libcephfs-devel-18.2.2-825.g2f465319. 338 kB/s | 31 kB 00:00 2024-03-14T09:42:22.055 INFO:teuthology.orchestra.run.smithi132.stdout:(118/146): openblas-threads-0.3.15-6.el8.x86_64 27 MB/s | 5.2 MB 00:00 2024-03-14T09:42:22.206 INFO:teuthology.orchestra.run.smithi132.stdout:(119/146): openblas-0.3.15-6.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2024-03-14T09:42:22.230 INFO:teuthology.orchestra.run.smithi132.stdout:(120/146): automake-1.16.1-8.el8.noarch.rpm 4.0 MB/s | 714 kB 00:00 2024-03-14T09:42:22.256 INFO:teuthology.orchestra.run.smithi132.stdout:(121/146): libpq-13.11-1.el8.x86_64.rpm 3.8 MB/s | 198 kB 00:00 2024-03-14T09:42:22.314 INFO:teuthology.orchestra.run.smithi132.stdout:(122/146): jq-1.6-7.el8.x86_64.rpm 2.4 MB/s | 202 kB 00:00 2024-03-14T09:42:22.337 INFO:teuthology.orchestra.run.smithi078.stdout:(13/146): libcephfs2-18.2.2-825.g2f465319.el8.x 2.5 MB/s | 779 kB 00:00 2024-03-14T09:42:22.340 INFO:teuthology.orchestra.run.smithi132.stdout:(123/146): python3-pytz-2017.2-11.el8.noarch.rp 650 kB/s | 54 kB 00:00 2024-03-14T09:42:22.398 INFO:teuthology.orchestra.run.smithi132.stdout:(124/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-03-14T09:42:22.402 INFO:teuthology.orchestra.run.smithi152.stdout:(25/146): ceph-test-18.2.2-825.g2f465319.el8.x8 7.9 MB/s | 47 MB 00:05 2024-03-14T09:42:22.446 INFO:teuthology.orchestra.run.smithi078.stdout:(14/146): libcephsqlite-18.2.2-825.g2f465319.el 1.3 MB/s | 142 kB 00:00 2024-03-14T09:42:22.491 INFO:teuthology.orchestra.run.smithi132.stdout:(125/146): postgresql-10.23-4.module+el8.9.0+21 16 MB/s | 1.5 MB 00:00 2024-03-14T09:42:22.494 INFO:teuthology.orchestra.run.smithi152.stdout:(26/146): librgw2-18.2.2-825.g2f465319.el8.x86_ 2.7 MB/s | 6.5 MB 00:02 2024-03-14T09:42:22.520 INFO:teuthology.orchestra.run.smithi152.stdout:(27/146): rbd-nbd-18.2.2-825.g2f465319.el8.x86_ 1.5 MB/s | 182 kB 00:00 2024-03-14T09:42:22.529 INFO:teuthology.orchestra.run.smithi078.stdout:(15/146): librados-devel-18.2.2-825.g2f465319.e 1.6 MB/s | 139 kB 00:00 2024-03-14T09:42:22.557 INFO:teuthology.orchestra.run.smithi132.stdout:(126/146): postgresql-contrib-10.23-4.module+el 12 MB/s | 812 kB 00:00 2024-03-14T09:42:22.570 INFO:teuthology.orchestra.run.smithi152.stdout:(28/146): ceph-mgr-cephadm-18.2.2-825.g2f465319 2.7 MB/s | 137 kB 00:00 2024-03-14T09:42:22.621 INFO:teuthology.orchestra.run.smithi152.stdout:(29/146): ceph-grafana-dashboards-18.2.2-825.g2 185 kB/s | 23 kB 00:00 2024-03-14T09:42:22.724 INFO:teuthology.orchestra.run.smithi132.stdout:(127/146): python3-scipy-1.0.0-21.module+el8.9. 33 MB/s | 14 MB 00:00 2024-03-14T09:42:22.780 INFO:teuthology.orchestra.run.smithi078.stdout:(16/146): libradosstriper1-18.2.2-825.g2f465319 1.9 MB/s | 474 kB 00:00 2024-03-14T09:42:22.805 INFO:teuthology.orchestra.run.smithi152.stdout:(30/146): ceph-mgr-dashboard-18.2.2-825.g2f4653 7.5 MB/s | 1.7 MB 00:00 2024-03-14T09:42:22.896 INFO:teuthology.orchestra.run.smithi152.stdout:(31/146): ceph-mgr-modules-core-18.2.2-825.g2f4 2.6 MB/s | 245 kB 00:00 2024-03-14T09:42:22.946 INFO:teuthology.orchestra.run.smithi152.stdout:(32/146): ceph-mgr-rook-18.2.2-825.g2f465319.el 990 kB/s | 49 kB 00:00 2024-03-14T09:42:22.988 INFO:teuthology.orchestra.run.smithi152.stdout:(33/146): ceph-prometheus-alerts-18.2.2-825.g2f 354 kB/s | 15 kB 00:00 2024-03-14T09:42:23.025 INFO:teuthology.orchestra.run.smithi132.stdout:(128/146): python3-packaging-16.8-9.el8.noarch. 161 kB/s | 48 kB 00:00 2024-03-14T09:42:23.038 INFO:teuthology.orchestra.run.smithi152.stdout:(34/146): ceph-volume-18.2.2-825.g2f465319.el8. 5.1 MB/s | 263 kB 00:00 2024-03-14T09:42:23.105 INFO:teuthology.orchestra.run.smithi078.stdout:(17/146): ceph-osd-18.2.2-825.g2f465319.el8.x86 2.6 MB/s | 18 MB 00:06 2024-03-14T09:42:23.120 INFO:teuthology.orchestra.run.smithi152.stdout:(35/146): rbd-mirror-18.2.2-825.g2f465319.el8.x 2.1 MB/s | 3.5 MB 00:01 2024-03-14T09:42:23.133 INFO:teuthology.orchestra.run.smithi132.stdout:(129/146): postgresql-server-10.23-4.module+el8 8.8 MB/s | 5.1 MB 00:00 2024-03-14T09:42:23.155 INFO:teuthology.orchestra.run.smithi152.stdout:(36/146): cephadm-18.2.2-825.g2f465319.el8.noar 1.4 MB/s | 170 kB 00:00 2024-03-14T09:42:23.205 INFO:teuthology.orchestra.run.smithi078.stdout:(18/146): python3-ceph-argparse-18.2.2-825.g2f4 448 kB/s | 45 kB 00:00 2024-03-14T09:42:23.214 INFO:teuthology.orchestra.run.smithi152.stdout:(37/146): cephfs-top-18.2.2-825.g2f465319.el8.n 192 kB/s | 18 kB 00:00 2024-03-14T09:42:23.542 INFO:teuthology.orchestra.run.smithi132.stdout:(130/146): librbd1-18.2.2-825.g2f465319.el8.x86 8.7 MB/s | 3.6 MB 00:00 2024-03-14T09:42:23.605 INFO:teuthology.orchestra.run.smithi078.stdout:(19/146): python3-ceph-common-18.2.2-825.g2f465 312 kB/s | 125 kB 00:00 2024-03-14T09:42:23.611 INFO:teuthology.orchestra.run.smithi132.stdout:(131/146): elfutils-libelf-0.189-3.el8.x86_64.r 3.3 MB/s | 232 kB 00:00 2024-03-14T09:42:23.668 INFO:teuthology.orchestra.run.smithi132.stdout:(132/146): elfutils-libs-0.189-3.el8.x86_64.rpm 5.3 MB/s | 303 kB 00:00 2024-03-14T09:42:23.718 INFO:teuthology.orchestra.run.smithi132.stdout:(133/146): libmount-2.32.1-43.el8.x86_64.rpm 4.6 MB/s | 237 kB 00:00 2024-03-14T09:42:23.760 INFO:teuthology.orchestra.run.smithi132.stdout:(134/146): libsmartcols-2.32.1-43.el8.x86_64.rp 4.2 MB/s | 179 kB 00:00 2024-03-14T09:42:23.797 INFO:teuthology.orchestra.run.smithi078.stdout:(20/146): python3-cephfs-18.2.2-825.g2f465319.e 1.0 MB/s | 197 kB 00:00 2024-03-14T09:42:23.810 INFO:teuthology.orchestra.run.smithi132.stdout:(135/146): libuuid-2.32.1-43.el8.x86_64.rpm 1.9 MB/s | 99 kB 00:00 2024-03-14T09:42:23.911 INFO:teuthology.orchestra.run.smithi132.stdout:(136/146): util-linux-2.32.1-43.el8.x86_64.rpm 25 MB/s | 2.5 MB 00:00 2024-03-14T09:42:23.952 INFO:teuthology.orchestra.run.smithi132.stdout:(137/146): util-linux-user-2.32.1-43.el8.x86_64 2.4 MB/s | 103 kB 00:00 2024-03-14T09:42:24.019 INFO:teuthology.orchestra.run.smithi132.stdout:(138/146): iproute-6.2.0-5.el8_9.x86_64.rpm 13 MB/s | 854 kB 00:00 2024-03-14T09:42:24.047 INFO:teuthology.orchestra.run.smithi078.stdout:(21/146): python3-rados-18.2.2-825.g2f465319.el 1.5 MB/s | 372 kB 00:00 2024-03-14T09:42:24.070 INFO:teuthology.orchestra.run.smithi132.stdout:(139/146): libblkid-2.32.1-43.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-14T09:42:24.112 INFO:teuthology.orchestra.run.smithi132.stdout:(140/146): libfdisk-2.32.1-43.el8.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-03-14T09:42:24.170 INFO:teuthology.orchestra.run.smithi132.stdout:(141/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 12 MB/s | 711 kB 00:00 2024-03-14T09:42:24.256 INFO:teuthology.orchestra.run.smithi078.stdout:(22/146): python3-rbd-18.2.2-825.g2f465319.el8. 1.7 MB/s | 365 kB 00:00 2024-03-14T09:42:24.262 INFO:teuthology.orchestra.run.smithi132.stdout:(142/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 16 MB/s | 1.5 MB 00:00 2024-03-14T09:42:24.313 INFO:teuthology.orchestra.run.smithi132.stdout:(143/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 6.5 MB/s | 333 kB 00:00 2024-03-14T09:42:24.371 INFO:teuthology.orchestra.run.smithi132.stdout:(144/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-03-14T09:42:24.398 INFO:teuthology.orchestra.run.smithi078.stdout:(23/146): python3-rgw-18.2.2-825.g2f465319.el8. 754 kB/s | 107 kB 00:00 2024-03-14T09:42:24.524 INFO:teuthology.orchestra.run.smithi078.stdout:(24/146): rbd-fuse-18.2.2-825.g2f465319.el8.x86 761 kB/s | 95 kB 00:00 2024-03-14T09:42:24.939 INFO:teuthology.orchestra.run.smithi152.stdout:(38/146): liboath-2.6.2-3.el8.x86_64.rpm 34 kB/s | 59 kB 00:01 2024-03-14T09:42:25.138 INFO:teuthology.orchestra.run.smithi132.stdout:(145/146): librados2-18.2.2-825.g2f465319.el8.x 1.7 MB/s | 3.7 MB 00:02 2024-03-14T09:42:25.340 INFO:teuthology.orchestra.run.smithi152.stdout:(39/146): ceph-mgr-diskprediction-local-18.2.2- 2.7 MB/s | 7.4 MB 00:02 2024-03-14T09:42:25.832 INFO:teuthology.orchestra.run.smithi152.stdout:(40/146): libunwind-1.3.1-3.el8.x86_64.rpm 84 kB/s | 75 kB 00:00 2024-03-14T09:42:26.090 INFO:teuthology.orchestra.run.smithi078.stdout:(25/146): librgw2-18.2.2-825.g2f465319.el8.x86_ 2.0 MB/s | 6.5 MB 00:03 2024-03-14T09:42:26.232 INFO:teuthology.orchestra.run.smithi078.stdout:(26/146): rbd-mirror-18.2.2-825.g2f465319.el8.x 2.0 MB/s | 3.5 MB 00:01 2024-03-14T09:42:26.257 INFO:teuthology.orchestra.run.smithi078.stdout:(27/146): rbd-nbd-18.2.2-825.g2f465319.el8.x86_ 1.1 MB/s | 182 kB 00:00 2024-03-14T09:42:26.358 INFO:teuthology.orchestra.run.smithi078.stdout:(28/146): ceph-grafana-dashboards-18.2.2-825.g2 186 kB/s | 23 kB 00:00 2024-03-14T09:42:26.383 INFO:teuthology.orchestra.run.smithi078.stdout:(29/146): ceph-mgr-cephadm-18.2.2-825.g2f465319 1.1 MB/s | 137 kB 00:00 2024-03-14T09:42:26.957 INFO:teuthology.orchestra.run.smithi152.stdout:(41/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 39 kB/s | 44 kB 00:01 2024-03-14T09:42:27.117 INFO:teuthology.orchestra.run.smithi078.stdout:(30/146): ceph-mgr-dashboard-18.2.2-825.g2f4653 2.3 MB/s | 1.7 MB 00:00 2024-03-14T09:42:27.217 INFO:teuthology.orchestra.run.smithi078.stdout:(31/146): ceph-mgr-modules-core-18.2.2-825.g2f4 2.4 MB/s | 245 kB 00:00 2024-03-14T09:42:27.309 INFO:teuthology.orchestra.run.smithi078.stdout:(32/146): ceph-mgr-rook-18.2.2-825.g2f465319.el 538 kB/s | 49 kB 00:00 2024-03-14T09:42:27.392 INFO:teuthology.orchestra.run.smithi078.stdout:(33/146): ceph-prometheus-alerts-18.2.2-825.g2f 178 kB/s | 15 kB 00:00 2024-03-14T09:42:27.493 INFO:teuthology.orchestra.run.smithi078.stdout:(34/146): ceph-volume-18.2.2-825.g2f465319.el8. 2.6 MB/s | 263 kB 00:00 2024-03-14T09:42:27.585 INFO:teuthology.orchestra.run.smithi078.stdout:(35/146): cephadm-18.2.2-825.g2f465319.el8.noar 1.8 MB/s | 170 kB 00:00 2024-03-14T09:42:28.052 INFO:teuthology.orchestra.run.smithi078.stdout:(36/146): ceph-test-18.2.2-825.g2f465319.el8.x8 6.1 MB/s | 47 MB 00:07 2024-03-14T09:42:28.077 INFO:teuthology.orchestra.run.smithi078.stdout:(37/146): cephfs-top-18.2.2-825.g2f465319.el8.n 36 kB/s | 18 kB 00:00 2024-03-14T09:42:28.463 INFO:teuthology.orchestra.run.smithi132.stdout:(146/146): ceph-test-18.2.2-825.g2f465319.el8.x 3.0 MB/s | 47 MB 00:15 2024-03-14T09:42:28.466 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-14T09:42:28.467 INFO:teuthology.orchestra.run.smithi132.stdout:Total 11 MB/s | 212 MB 00:18 2024-03-14T09:42:28.606 INFO:teuthology.orchestra.run.smithi078.stdout:(38/146): liboath-2.6.2-3.el8.x86_64.rpm 111 kB/s | 59 kB 00:00 2024-03-14T09:42:28.661 INFO:teuthology.orchestra.run.smithi078.stdout:(39/146): gperftools-libs-2.7-9.el8.x86_64.rpm 503 kB/s | 306 kB 00:00 2024-03-14T09:42:28.761 INFO:teuthology.orchestra.run.smithi078.stdout:(40/146): libunwind-1.3.1-3.el8.x86_64.rpm 484 kB/s | 75 kB 00:00 2024-03-14T09:42:28.870 INFO:teuthology.orchestra.run.smithi078.stdout:(41/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 402 kB/s | 44 kB 00:00 2024-03-14T09:42:28.895 INFO:teuthology.orchestra.run.smithi078.stdout:(42/146): python3-asyncssh-2.7.0-2.el8.noarch.r 2.0 MB/s | 476 kB 00:00 2024-03-14T09:42:29.037 INFO:teuthology.orchestra.run.smithi078.stdout:(43/146): ceph-mgr-diskprediction-local-18.2.2- 2.8 MB/s | 7.4 MB 00:02 2024-03-14T09:42:29.071 INFO:teuthology.orchestra.run.smithi078.stdout:(44/146): python3-cachetools-3.1.1-4.el8.noarch 188 kB/s | 33 kB 00:00 2024-03-14T09:42:29.096 INFO:teuthology.orchestra.run.smithi078.stdout:(45/146): python3-beautifulsoup4-4.6.3-2.el8.1. 820 kB/s | 185 kB 00:00 2024-03-14T09:42:29.196 INFO:teuthology.orchestra.run.smithi078.stdout:(46/146): python3-cheroot-8.5.2-1.el8.noarch.rp 1.4 MB/s | 173 kB 00:00 2024-03-14T09:42:29.198 INFO:teuthology.orchestra.run.smithi152.stdout:(42/146): python3-beautifulsoup4-4.6.3-2.el8.1. 83 kB/s | 185 kB 00:02 2024-03-14T09:42:29.288 INFO:teuthology.orchestra.run.smithi078.stdout:(47/146): python3-google-auth-1.1.1-10.el8.noar 886 kB/s | 81 kB 00:00 2024-03-14T09:42:29.322 INFO:teuthology.orchestra.run.smithi078.stdout:(48/146): python3-cherrypy-18.4.0-1.el8.noarch. 1.7 MB/s | 384 kB 00:00 2024-03-14T09:42:29.347 INFO:teuthology.orchestra.run.smithi078.stdout:(49/146): python3-certifi-2018.10.15-7.el8.noar 53 kB/s | 16 kB 00:00 2024-03-14T09:42:29.372 INFO:teuthology.orchestra.run.smithi078.stdout:(50/146): python3-jaraco-6.2-6.el8.noarch.rpm 133 kB/s | 11 kB 00:00 2024-03-14T09:42:29.380 INFO:teuthology.orchestra.run.smithi152.stdout:(43/146): gperftools-libs-2.7-9.el8.x86_64.rpm 49 kB/s | 306 kB 00:06 2024-03-14T09:42:29.406 INFO:teuthology.orchestra.run.smithi078.stdout:(51/146): python3-jaraco-functools-2.0-4.el8.no 210 kB/s | 18 kB 00:00 2024-03-14T09:42:29.456 INFO:teuthology.orchestra.run.smithi078.stdout:(52/146): python3-logutils-0.3.5-11.el8.noarch. 583 kB/s | 49 kB 00:00 2024-03-14T09:42:29.490 INFO:teuthology.orchestra.run.smithi078.stdout:(53/146): python3-more-itertools-7.2.0-3.el8.no 701 kB/s | 59 kB 00:00 2024-03-14T09:42:29.496 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-14T09:42:29.540 INFO:teuthology.orchestra.run.smithi078.stdout:(54/146): python3-natsort-7.1.1-2.el8.noarch.rp 721 kB/s | 60 kB 00:00 2024-03-14T09:42:29.607 INFO:teuthology.orchestra.run.smithi078.stdout:(55/146): python3-pecan-1.3.2-9.el8.noarch.rpm 2.4 MB/s | 283 kB 00:00 2024-03-14T09:42:29.632 INFO:teuthology.orchestra.run.smithi078.stdout:(56/146): python3-portend-2.6-1.el8.noarch.rpm 170 kB/s | 16 kB 00:00 2024-03-14T09:42:29.649 INFO:teuthology.orchestra.run.smithi152.stdout:(44/146): python3-cachetools-3.1.1-4.el8.noarch 73 kB/s | 33 kB 00:00 2024-03-14T09:42:29.691 INFO:teuthology.orchestra.run.smithi078.stdout:(57/146): python3-repoze-lru-0.7-6.el8.noarch.r 404 kB/s | 34 kB 00:00 2024-03-14T09:42:29.750 INFO:teuthology.orchestra.run.smithi078.stdout:(58/146): python3-routes-2.4.1-12.el8.noarch.rp 1.6 MB/s | 196 kB 00:00 2024-03-14T09:42:29.783 INFO:teuthology.orchestra.run.smithi078.stdout:(59/146): python3-kubernetes-11.0.0-6.el8.noarc 2.7 MB/s | 1.2 MB 00:00 2024-03-14T09:42:29.809 INFO:teuthology.orchestra.run.smithi078.stdout:(60/146): python3-rsa-4.9-2.el8.noarch.rpm 548 kB/s | 64 kB 00:00 2024-03-14T09:42:29.825 INFO:teuthology.orchestra.run.smithi078.stdout:(61/146): python3-simplegeneric-0.8.1-17.el8.no 249 kB/s | 19 kB 00:00 2024-03-14T09:42:29.832 INFO:teuthology.orchestra.run.smithi152.stdout:(45/146): python3-certifi-2018.10.15-7.el8.noar 36 kB/s | 16 kB 00:00 2024-03-14T09:42:29.859 INFO:teuthology.orchestra.run.smithi078.stdout:(62/146): python3-singledispatch-3.4.0.3-18.el8 316 kB/s | 24 kB 00:00 2024-03-14T09:42:29.888 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-14T09:42:29.888 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-14T09:42:29.892 INFO:teuthology.orchestra.run.smithi078.stdout:(63/146): python3-tempora-1.14.1-5.el8.noarch.r 349 kB/s | 29 kB 00:00 2024-03-14T09:42:29.918 INFO:teuthology.orchestra.run.smithi078.stdout:(64/146): python3-trustme-0.6.0-4.el8.noarch.rp 298 kB/s | 27 kB 00:00 2024-03-14T09:42:29.952 INFO:teuthology.orchestra.run.smithi078.stdout:(65/146): python3-waitress-1.4.3-1.el8.noarch.r 2.7 MB/s | 255 kB 00:00 2024-03-14T09:42:30.002 INFO:teuthology.orchestra.run.smithi078.stdout:(66/146): python3-webob-1.8.5-1.el8.1.noarch.rp 2.3 MB/s | 251 kB 00:00 2024-03-14T09:42:30.027 INFO:teuthology.orchestra.run.smithi078.stdout:(67/146): python3-websocket-client-0.56.0-5.el8 563 kB/s | 61 kB 00:00 2024-03-14T09:42:30.053 INFO:teuthology.orchestra.run.smithi078.stdout:(68/146): python3-webtest-2.0.33-1.el8.noarch.r 850 kB/s | 85 kB 00:00 2024-03-14T09:42:30.086 INFO:teuthology.orchestra.run.smithi078.stdout:(69/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.1 MB/s | 70 kB 00:00 2024-03-14T09:42:30.111 INFO:teuthology.orchestra.run.smithi078.stdout:(70/146): python3-zc-lockfile-2.0-2.el8.noarch. 211 kB/s | 23 kB 00:00 2024-03-14T09:42:30.178 INFO:teuthology.orchestra.run.smithi078.stdout:(71/146): python3-jwt-1.6.1-2.el8.noarch.rpm 639 kB/s | 43 kB 00:00 2024-03-14T09:42:30.204 INFO:teuthology.orchestra.run.smithi078.stdout:(72/146): libverto-devel-0.3.0-5.el8.x86_64.rpm 153 kB/s | 18 kB 00:00 2024-03-14T09:42:30.229 INFO:teuthology.orchestra.run.smithi078.stdout:(73/146): python3-oauthlib-2.1.0-1.el8.noarch.r 3.0 MB/s | 155 kB 00:00 2024-03-14T09:42:30.254 INFO:teuthology.orchestra.run.smithi078.stdout:(74/146): python3-pycparser-2.14-14.el8.noarch. 2.1 MB/s | 109 kB 00:00 2024-03-14T09:42:30.279 INFO:teuthology.orchestra.run.smithi078.stdout:(75/146): python3-requests-oauthlib-1.0.0-1.el8 854 kB/s | 43 kB 00:00 2024-03-14T09:42:30.305 INFO:teuthology.orchestra.run.smithi078.stdout:(76/146): libaio-devel-0.3.112-1.el8.x86_64.rpm 373 kB/s | 19 kB 00:00 2024-03-14T09:42:30.330 INFO:teuthology.orchestra.run.smithi078.stdout:(77/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 873 kB/s | 44 kB 00:00 2024-03-14T09:42:30.372 INFO:teuthology.orchestra.run.smithi078.stdout:(78/146): thrift-0.13.0-2.el8.x86_64.rpm 5.0 MB/s | 1.7 MB 00:00 2024-03-14T09:42:30.414 INFO:teuthology.orchestra.run.smithi078.stdout:(79/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.7 MB/s | 229 kB 00:00 2024-03-14T09:42:30.456 INFO:teuthology.orchestra.run.smithi078.stdout:(80/146): pcre2-devel-10.32-2.el8.x86_64.rpm 3.9 MB/s | 605 kB 00:00 2024-03-14T09:42:30.489 INFO:teuthology.orchestra.run.smithi078.stdout:(81/146): libselinux-devel-2.9-5.el8.x86_64.rpm 2.8 MB/s | 200 kB 00:00 2024-03-14T09:42:30.514 INFO:teuthology.orchestra.run.smithi078.stdout:(82/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.5 MB/s | 220 kB 00:00 2024-03-14T09:42:30.540 INFO:teuthology.orchestra.run.smithi078.stdout:(83/146): keyutils-libs-devel-1.5.10-9.el8.x86_ 574 kB/s | 48 kB 00:00 2024-03-14T09:42:30.565 INFO:teuthology.orchestra.run.smithi078.stdout:(84/146): libsepol-devel-2.9-3.el8.x86_64.rpm 1.1 MB/s | 87 kB 00:00 2024-03-14T09:42:30.590 INFO:teuthology.orchestra.run.smithi078.stdout:(85/146): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 764 kB/s | 58 kB 00:00 2024-03-14T09:42:30.616 INFO:teuthology.orchestra.run.smithi078.stdout:(86/146): libcom_err-devel-1.45.6-4.el8.x86_64. 512 kB/s | 39 kB 00:00 2024-03-14T09:42:30.632 INFO:teuthology.orchestra.run.smithi078.stdout:(87/146): fuse-2.9.7-17.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2024-03-14T09:42:30.658 INFO:teuthology.orchestra.run.smithi078.stdout:(88/146): libquadmath-8.5.0-20.el8.x86_64.rpm 4.0 MB/s | 172 kB 00:00 2024-03-14T09:42:30.683 INFO:teuthology.orchestra.run.smithi078.stdout:(89/146): libuuid-devel-2.32.1-43.el8.x86_64.rp 1.9 MB/s | 100 kB 00:00 2024-03-14T09:42:30.717 INFO:teuthology.orchestra.run.smithi078.stdout:(90/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 4.0 MB/s | 238 kB 00:00 2024-03-14T09:42:30.742 INFO:teuthology.orchestra.run.smithi078.stdout:(91/146): libgfortran-8.5.0-20.el8.x86_64.rpm 4.2 MB/s | 645 kB 00:00 2024-03-14T09:42:30.767 INFO:teuthology.orchestra.run.smithi078.stdout:(92/146): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 870 kB/s | 73 kB 00:00 2024-03-14T09:42:30.824 INFO:teuthology.orchestra.run.smithi078.stdout:(93/146): elfutils-libelf-devel-0.189-3.el8.x86 755 kB/s | 62 kB 00:00 2024-03-14T09:42:30.851 INFO:teuthology.orchestra.run.smithi078.stdout:(94/146): NetworkManager-initscripts-updown-1.4 1.1 MB/s | 144 kB 00:00 2024-03-14T09:42:30.893 INFO:teuthology.orchestra.run.smithi078.stdout:(95/146): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 4.4 MB/s | 562 kB 00:00 2024-03-14T09:42:30.918 INFO:teuthology.orchestra.run.smithi078.stdout:(96/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 706 kB/s | 47 kB 00:00 2024-03-14T09:42:30.961 INFO:teuthology.orchestra.run.smithi078.stdout:(97/146): python3-cryptography-3.2.1-7.el8_9.x8 8.2 MB/s | 559 kB 00:00 2024-03-14T09:42:30.986 INFO:teuthology.orchestra.run.smithi078.stdout:(98/146): python3-packaging-16.8-9.el8.noarch.r 722 kB/s | 48 kB 00:00 2024-03-14T09:42:31.011 INFO:teuthology.orchestra.run.smithi078.stdout:(99/146): python3-jsonpointer-1.10-11.el8.noarc 400 kB/s | 20 kB 00:00 2024-03-14T09:42:31.036 INFO:teuthology.orchestra.run.smithi078.stdout:(100/146): python3-prettytable-0.7.2-14.el8.noa 877 kB/s | 44 kB 00:00 2024-03-14T09:42:31.061 INFO:teuthology.orchestra.run.smithi078.stdout:(101/146): python3-pyasn1-modules-0.3.7-6.el8.n 2.1 MB/s | 110 kB 00:00 2024-03-14T09:42:31.081 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-14T09:42:31.091 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-14T09:42:31.095 INFO:teuthology.orchestra.run.smithi078.stdout:(102/146): python3-jsonpatch-1.21-2.el8.noarch. 451 kB/s | 27 kB 00:00 2024-03-14T09:42:31.121 INFO:teuthology.orchestra.run.smithi078.stdout:(103/146): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.1 MB/s | 126 kB 00:00 2024-03-14T09:42:31.154 INFO:teuthology.orchestra.run.smithi078.stdout:(104/146): python3-werkzeug-0.12.2-4.el8.noarch 7.6 MB/s | 457 kB 00:00 2024-03-14T09:42:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:(105/146): openssl-devel-1.1.1k-12.el8_9.x86_64 6.3 MB/s | 2.3 MB 00:00 2024-03-14T09:42:31.230 INFO:teuthology.orchestra.run.smithi078.stdout:(106/146): flex-2.6.1-9.el8.x86_64.rpm 4.2 MB/s | 320 kB 00:00 2024-03-14T09:42:31.271 INFO:teuthology.orchestra.run.smithi078.stdout:(107/146): bison-3.0.4-10.el8.x86_64.rpm 4.5 MB/s | 688 kB 00:00 2024-03-14T09:42:31.297 INFO:teuthology.orchestra.run.smithi078.stdout:(108/146): python3-markupsafe-0.23-19.el8.x86_6 582 kB/s | 39 kB 00:00 2024-03-14T09:42:31.331 INFO:teuthology.orchestra.run.smithi078.stdout:(109/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.3 MB/s | 259 kB 00:00 2024-03-14T09:42:31.356 INFO:teuthology.orchestra.run.smithi078.stdout:(110/146): uuid-1.6.2-43.el8.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-03-14T09:42:31.389 INFO:teuthology.orchestra.run.smithi078.stdout:(111/146): libtool-2.4.6-25.el8.x86_64.rpm 3.6 MB/s | 709 kB 00:00 2024-03-14T09:42:31.415 INFO:teuthology.orchestra.run.smithi078.stdout:(112/146): python3-pyOpenSSL-19.0.0-1.el8.noarc 1.2 MB/s | 103 kB 00:00 2024-03-14T09:42:31.457 INFO:teuthology.orchestra.run.smithi078.stdout:(113/146): python3-jinja2-2.10.1-3.el8.noarch.r 7.9 MB/s | 538 kB 00:00 2024-03-14T09:42:31.517 INFO:teuthology.orchestra.run.smithi078.stdout:(114/146): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2024-03-14T09:42:31.565 INFO:teuthology.orchestra.run.smithi078.stdout:(115/146): python3-numpy-f2py-1.14.3-10.el8.x86 4.4 MB/s | 225 kB 00:00 2024-03-14T09:42:31.641 INFO:teuthology.orchestra.run.smithi078.stdout:(116/146): python3-numpy-1.14.3-10.el8.x86_64.r 17 MB/s | 3.7 MB 00:00 2024-03-14T09:42:31.683 INFO:teuthology.orchestra.run.smithi078.stdout:(117/146): librdkafka-0.11.4-3.el8.x86_64.rpm 3.0 MB/s | 354 kB 00:00 2024-03-14T09:42:31.750 INFO:teuthology.orchestra.run.smithi078.stdout:(118/146): python3-babel-2.5.1-7.el8.noarch.rpm 12 MB/s | 4.8 MB 00:00 2024-03-14T09:42:31.775 INFO:teuthology.orchestra.run.smithi078.stdout:(119/146): python3-mako-1.0.6-14.el8.noarch.rpm 1.7 MB/s | 157 kB 00:00 2024-03-14T09:42:31.842 INFO:teuthology.orchestra.run.smithi078.stdout:(120/146): automake-1.16.1-8.el8.noarch.rpm 10 MB/s | 714 kB 00:00 2024-03-14T09:42:31.911 INFO:teuthology.orchestra.run.smithi078.stdout:(121/146): libpq-13.11-1.el8.x86_64.rpm 2.8 MB/s | 198 kB 00:00 2024-03-14T09:42:31.959 INFO:teuthology.orchestra.run.smithi078.stdout:(122/146): jq-1.6-7.el8.x86_64.rpm 4.1 MB/s | 202 kB 00:00 2024-03-14T09:42:32.034 INFO:teuthology.orchestra.run.smithi078.stdout:(123/146): openblas-threads-0.3.15-6.el8.x86_64 13 MB/s | 5.2 MB 00:00 2024-03-14T09:42:32.059 INFO:teuthology.orchestra.run.smithi078.stdout:(124/146): python3-pytz-2017.2-11.el8.noarch.rp 541 kB/s | 54 kB 00:00 2024-03-14T09:42:32.126 INFO:teuthology.orchestra.run.smithi078.stdout:(125/146): openblas-0.3.15-6.el8.x86_64.rpm 13 MB/s | 5.0 MB 00:00 2024-03-14T09:42:32.152 INFO:teuthology.orchestra.run.smithi078.stdout:(126/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-03-14T09:42:32.210 INFO:teuthology.orchestra.run.smithi078.stdout:(127/146): postgresql-10.23-4.module+el8.9.0+21 18 MB/s | 1.5 MB 00:00 2024-03-14T09:42:32.261 INFO:teuthology.orchestra.run.smithi078.stdout:(128/146): postgresql-contrib-10.23-4.module+el 7.3 MB/s | 812 kB 00:00 2024-03-14T09:42:32.411 INFO:teuthology.orchestra.run.smithi078.stdout:(129/146): postgresql-server-10.23-4.module+el8 25 MB/s | 5.1 MB 00:00 2024-03-14T09:42:32.620 INFO:teuthology.orchestra.run.smithi078.stdout:(130/146): python3-scipy-1.0.0-21.module+el8.9. 23 MB/s | 14 MB 00:00 2024-03-14T09:42:32.670 INFO:teuthology.orchestra.run.smithi078.stdout:(131/146): elfutils-libelf-0.189-3.el8.x86_64.r 4.6 MB/s | 232 kB 00:00 2024-03-14T09:42:32.737 INFO:teuthology.orchestra.run.smithi078.stdout:(132/146): librados2-18.2.2-825.g2f465319.el8.x 7.8 MB/s | 3.7 MB 00:00 2024-03-14T09:42:32.771 INFO:teuthology.orchestra.run.smithi078.stdout:(133/146): elfutils-libs-0.189-3.el8.x86_64.rpm 3.0 MB/s | 303 kB 00:00 2024-03-14T09:42:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:(134/146): libmount-2.32.1-43.el8.x86_64.rpm 3.5 MB/s | 237 kB 00:00 2024-03-14T09:42:32.829 INFO:teuthology.orchestra.run.smithi078.stdout:(135/146): libsmartcols-2.32.1-43.el8.x86_64.rp 3.0 MB/s | 179 kB 00:00 2024-03-14T09:42:32.846 INFO:teuthology.orchestra.run.smithi078.stdout:(136/146): libuuid-2.32.1-43.el8.x86_64.rpm 2.3 MB/s | 99 kB 00:00 2024-03-14T09:42:32.897 INFO:teuthology.orchestra.run.smithi078.stdout:(137/146): util-linux-user-2.32.1-43.el8.x86_64 2.0 MB/s | 103 kB 00:00 2024-03-14T09:42:32.964 INFO:teuthology.orchestra.run.smithi078.stdout:(138/146): util-linux-2.32.1-43.el8.x86_64.rpm 19 MB/s | 2.5 MB 00:00 2024-03-14T09:42:33.022 INFO:teuthology.orchestra.run.smithi078.stdout:(139/146): iproute-6.2.0-5.el8_9.x86_64.rpm 6.7 MB/s | 854 kB 00:00 2024-03-14T09:42:33.047 INFO:teuthology.orchestra.run.smithi078.stdout:(140/146): libblkid-2.32.1-43.el8.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2024-03-14T09:42:33.257 INFO:teuthology.orchestra.run.smithi152.stdout:(46/146): python3-asyncssh-2.7.0-2.el8.noarch.r 62 kB/s | 476 kB 00:07 2024-03-14T09:42:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:(141/146): libfdisk-2.32.1-43.el8.x86_64.rpm 5.0 MB/s | 254 kB 00:00 2024-03-14T09:42:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:(142/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 10 MB/s | 711 kB 00:00 2024-03-14T09:42:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:(143/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 15 MB/s | 1.5 MB 00:00 2024-03-14T09:42:33.257 INFO:teuthology.orchestra.run.smithi078.stdout:(144/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 3.5 MB/s | 333 kB 00:00 2024-03-14T09:42:33.265 INFO:teuthology.orchestra.run.smithi078.stdout:(145/146): librbd1-18.2.2-825.g2f465319.el8.x86 4.2 MB/s | 3.6 MB 00:00 2024-03-14T09:42:33.299 INFO:teuthology.orchestra.run.smithi078.stdout:(146/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 8.5 MB/s | 1.1 MB 00:00 2024-03-14T09:42:33.302 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-14T09:42:33.302 INFO:teuthology.orchestra.run.smithi078.stdout:Total 10 MB/s | 212 MB 00:20 2024-03-14T09:42:33.666 INFO:teuthology.orchestra.run.smithi152.stdout:(47/146): python3-cheroot-8.5.2-1.el8.noarch.rp 43 kB/s | 173 kB 00:04 2024-03-14T09:42:33.691 INFO:teuthology.orchestra.run.smithi152.stdout:(48/146): python3-google-auth-1.1.1-10.el8.noar 130 kB/s | 81 kB 00:00 2024-03-14T09:42:33.958 INFO:teuthology.orchestra.run.smithi152.stdout:(49/146): python3-jaraco-6.2-6.el8.noarch.rpm 38 kB/s | 11 kB 00:00 2024-03-14T09:42:34.050 INFO:teuthology.orchestra.run.smithi152.stdout:(50/146): python3-jaraco-functools-2.0-4.el8.no 49 kB/s | 18 kB 00:00 2024-03-14T09:42:35.184 INFO:teuthology.orchestra.run.smithi152.stdout:(51/146): python3-logutils-0.3.5-11.el8.noarch. 43 kB/s | 49 kB 00:01 2024-03-14T09:42:35.521 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-03-14T09:42:35.966 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-03-14T09:42:35.966 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-03-14T09:42:36.526 INFO:teuthology.orchestra.run.smithi152.stdout:(52/146): python3-more-itertools-7.2.0-3.el8.no 44 kB/s | 59 kB 00:01 2024-03-14T09:42:37.029 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-14T09:42:37.163 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-03-14T09:42:37.369 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-14T09:42:37.539 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-03-14T09:42:37.552 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-03-14T09:42:37.568 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-14T09:42:37.764 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-03-14T09:42:37.941 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-14T09:42:38.001 INFO:teuthology.orchestra.run.smithi152.stdout:(53/146): python3-natsort-7.1.1-2.el8.noarch.rp 41 kB/s | 60 kB 00:01 2024-03-14T09:42:38.014 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-14T09:42:38.166 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-14T09:42:38.234 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-14T09:42:38.446 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-03-14T09:42:38.648 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 6/162 2024-03-14T09:42:38.833 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-14T09:42:38.933 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-14T09:42:39.199 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-14T09:42:39.606 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-14T09:42:39.841 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librados2-2:18.2.2-825.g2f465319.el8.x86_64 9/162 2024-03-14T09:42:40.230 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:18.2.2-825.g2f465319.el8.x86_64 9/162 2024-03-14T09:42:40.433 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librbd1-2:18.2.2-825.g2f465319.el8.x86_64 10/162 2024-03-14T09:42:40.559 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:18.2.2-825.g2f465319.el8.x86_64 10/162 2024-03-14T09:42:40.655 INFO:teuthology.orchestra.run.smithi152.stdout:(54/146): python3-cherrypy-18.4.0-1.el8.noarch. 36 kB/s | 384 kB 00:10 2024-03-14T09:42:40.741 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs2-2:18.2.2-825.g2f465319.el8.x86_64 11/162 2024-03-14T09:42:40.838 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs2-2:18.2.2-825.g2f465319.el8.x86_64 11/162 2024-03-14T09:42:40.851 INFO:teuthology.orchestra.run.smithi152.stdout:(55/146): python3-portend-2.6-1.el8.noarch.rpm 80 kB/s | 16 kB 00:00 2024-03-14T09:42:41.151 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rados-2:18.2.2-825.g2f465319.el8.x86_64 12/162 2024-03-14T09:42:41.218 INFO:teuthology.orchestra.run.smithi152.stdout:(56/146): python3-repoze-lru-0.7-6.el8.noarch.r 92 kB/s | 34 kB 00:00 2024-03-14T09:42:41.382 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-14T09:42:41.470 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-14T09:42:41.668 INFO:teuthology.orchestra.run.smithi152.stdout:(57/146): python3-pecan-1.3.2-9.el8.noarch.rpm 77 kB/s | 283 kB 00:03 2024-03-14T09:42:41.689 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-03-14T09:42:41.903 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-03-14T09:42:42.085 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-03-14T09:42:42.218 INFO:teuthology.orchestra.run.smithi152.stdout:(58/146): python3-rsa-4.9-2.el8.noarch.rpm 117 kB/s | 64 kB 00:00 2024-03-14T09:42:42.478 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-03-14T09:42:42.544 INFO:teuthology.orchestra.run.smithi152.stdout:(59/146): python3-simplegeneric-0.8.1-17.el8.no 58 kB/s | 19 kB 00:00 2024-03-14T09:42:42.686 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-03-14T09:42:42.727 INFO:teuthology.orchestra.run.smithi152.stdout:(60/146): python3-singledispatch-3.4.0.3-18.el8 130 kB/s | 24 kB 00:00 2024-03-14T09:42:42.824 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephsqlite-2:18.2.2-825.g2f465319.el8.x86_64 19/162 2024-03-14T09:42:42.911 INFO:teuthology.orchestra.run.smithi152.stdout:(61/146): python3-tempora-1.14.1-5.el8.noarch.r 159 kB/s | 29 kB 00:00 2024-03-14T09:42:42.955 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephsqlite-2:18.2.2-825.g2f465319.el8.x86_64 19/162 2024-03-14T09:42:43.099 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libradosstriper1-2:18.2.2-825.g2f465319.el8.x86_ 20/162 2024-03-14T09:42:43.178 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libradosstriper1-2:18.2.2-825.g2f465319.el8.x86_ 20/162 2024-03-14T09:42:43.327 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 21/162 2024-03-14T09:42:43.343 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-03-14T09:42:43.444 INFO:teuthology.orchestra.run.smithi152.stdout:(62/146): python3-trustme-0.6.0-4.el8.noarch.rp 52 kB/s | 27 kB 00:00 2024-03-14T09:42:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-03-14T09:42:43.588 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/162 2024-03-14T09:42:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-14T09:42:43.820 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/162 2024-03-14T09:42:43.990 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-14T09:42:44.028 INFO:teuthology.orchestra.run.smithi152.stdout:(63/146): python3-routes-2.4.1-12.el8.noarch.rp 70 kB/s | 196 kB 00:02 2024-03-14T09:42:44.110 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-14T09:42:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-14T09:42:44.315 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-14T09:42:44.442 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-03-14T09:42:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-14T09:42:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-14T09:42:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-14T09:42:44.921 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-14T09:42:45.017 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-14T09:42:45.082 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-14T09:42:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-03-14T09:42:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 6/162 2024-03-14T09:42:45.671 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-14T09:42:45.686 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-14T09:42:45.695 INFO:teuthology.orchestra.run.smithi152.stdout:(64/146): python3-webob-1.8.5-1.el8.1.noarch.rp 151 kB/s | 251 kB 00:01 2024-03-14T09:42:45.806 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-14T09:42:45.848 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-0.3.15-6.el8.x86_64 27/162 2024-03-14T09:42:45.891 INFO:teuthology.orchestra.run.smithi152.stdout:(65/146): python3-waitress-1.4.3-1.el8.noarch.r 104 kB/s | 255 kB 00:02 2024-03-14T09:42:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-14T09:42:46.455 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-14T09:42:46.534 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 27/162 2024-03-14T09:42:46.646 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librados2-2:18.2.2-825.g2f465319.el8.x86_64 9/162 2024-03-14T09:42:46.813 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 28/162 2024-03-14T09:42:46.845 INFO:teuthology.orchestra.run.smithi152.stdout:(66/146): python3-websocket-client-0.56.0-5.el8 53 kB/s | 61 kB 00:01 2024-03-14T09:42:47.021 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 29/162 2024-03-14T09:42:47.054 INFO:teuthology.orchestra.run.smithi152.stdout:(67/146): python3-zc-lockfile-2.0-2.el8.noarch. 110 kB/s | 23 kB 00:00 2024-03-14T09:42:47.088 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:18.2.2-825.g2f465319.el8.x86_64 9/162 2024-03-14T09:42:47.162 INFO:teuthology.orchestra.run.smithi152.stdout:(68/146): python3-webtest-2.0.33-1.el8.noarch.r 67 kB/s | 85 kB 00:01 2024-03-14T09:42:47.188 INFO:teuthology.orchestra.run.smithi152.stdout:(69/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2024-03-14T09:42:47.246 INFO:teuthology.orchestra.run.smithi152.stdout:(70/146): libverto-devel-0.3.0-5.el8.x86_64.rpm 306 kB/s | 18 kB 00:00 2024-03-14T09:42:47.258 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-03-14T09:42:47.288 INFO:teuthology.orchestra.run.smithi152.stdout:(71/146): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2024-03-14T09:42:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librbd1-2:18.2.2-825.g2f465319.el8.x86_64 10/162 2024-03-14T09:42:47.330 INFO:teuthology.orchestra.run.smithi152.stdout:(72/146): python3-oauthlib-2.1.0-1.el8.noarch.r 3.6 MB/s | 155 kB 00:00 2024-03-14T09:42:47.409 INFO:teuthology.orchestra.run.smithi152.stdout:(73/146): python3-pycparser-2.14-14.el8.noarch. 1.4 MB/s | 109 kB 00:00 2024-03-14T09:42:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:18.2.2-825.g2f465319.el8.x86_64 10/162 2024-03-14T09:42:47.447 INFO:teuthology.orchestra.run.smithi152.stdout:(74/146): python3-requests-oauthlib-1.0.0-1.el8 1.1 MB/s | 43 kB 00:00 2024-03-14T09:42:47.481 INFO:teuthology.orchestra.run.smithi152.stdout:(75/146): libaio-devel-0.3.112-1.el8.x86_64.rpm 559 kB/s | 19 kB 00:00 2024-03-14T09:42:47.506 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-common-2:18.2.2-825.g2f465319.el8.x 31/162 2024-03-14T09:42:47.515 INFO:teuthology.orchestra.run.smithi152.stdout:(76/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-14T09:42:47.574 INFO:teuthology.orchestra.run.smithi152.stdout:(77/146): pcre2-devel-10.32-2.el8.x86_64.rpm 10 MB/s | 605 kB 00:00 2024-03-14T09:42:47.616 INFO:teuthology.orchestra.run.smithi152.stdout:(78/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.4 MB/s | 229 kB 00:00 2024-03-14T09:42:47.629 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs2-2:18.2.2-825.g2f465319.el8.x86_64 11/162 2024-03-14T09:42:47.634 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-argparse-2:18.2.2-825.g2f465319.el8 32/162 2024-03-14T09:42:47.683 INFO:teuthology.orchestra.run.smithi152.stdout:(79/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-14T09:42:47.725 INFO:teuthology.orchestra.run.smithi152.stdout:(80/146): libselinux-devel-2.9-5.el8.x86_64.rpm 4.7 MB/s | 200 kB 00:00 2024-03-14T09:42:47.730 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs2-2:18.2.2-825.g2f465319.el8.x86_64 11/162 2024-03-14T09:42:47.767 INFO:teuthology.orchestra.run.smithi152.stdout:(81/146): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.1 MB/s | 48 kB 00:00 2024-03-14T09:42:47.809 INFO:teuthology.orchestra.run.smithi152.stdout:(82/146): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2024-03-14T09:42:47.851 INFO:teuthology.orchestra.run.smithi152.stdout:(83/146): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-03-14T09:42:47.876 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cephfs-2:18.2.2-825.g2f465319.el8.x86_64 33/162 2024-03-14T09:42:47.926 INFO:teuthology.orchestra.run.smithi152.stdout:(84/146): libcom_err-devel-1.45.6-4.el8.x86_64. 515 kB/s | 39 kB 00:00 2024-03-14T09:42:47.968 INFO:teuthology.orchestra.run.smithi152.stdout:(85/146): fuse-2.9.7-17.el8.x86_64.rpm 1.9 MB/s | 83 kB 00:00 2024-03-14T09:42:48.018 INFO:teuthology.orchestra.run.smithi152.stdout:(86/146): libgfortran-8.5.0-20.el8.x86_64.rpm 13 MB/s | 645 kB 00:00 2024-03-14T09:42:48.060 INFO:teuthology.orchestra.run.smithi152.stdout:(87/146): libquadmath-8.5.0-20.el8.x86_64.rpm 4.0 MB/s | 172 kB 00:00 2024-03-14T09:42:48.072 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rados-2:18.2.2-825.g2f465319.el8.x86_64 12/162 2024-03-14T09:42:48.102 INFO:teuthology.orchestra.run.smithi152.stdout:(88/146): libuuid-devel-2.32.1-43.el8.x86_64.rp 2.3 MB/s | 100 kB 00:00 2024-03-14T09:42:48.110 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-14T09:42:48.178 INFO:teuthology.orchestra.run.smithi152.stdout:(89/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 3.1 MB/s | 238 kB 00:00 2024-03-14T09:42:48.228 INFO:teuthology.orchestra.run.smithi152.stdout:(90/146): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 1.4 MB/s | 73 kB 00:00 2024-03-14T09:42:48.270 INFO:teuthology.orchestra.run.smithi152.stdout:(91/146): NetworkManager-initscripts-updown-1.4 3.4 MB/s | 144 kB 00:00 2024-03-14T09:42:48.312 INFO:teuthology.orchestra.run.smithi152.stdout:(92/146): elfutils-libelf-devel-0.189-3.el8.x86 1.5 MB/s | 62 kB 00:00 2024-03-14T09:42:48.329 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-14T09:42:48.362 INFO:teuthology.orchestra.run.smithi152.stdout:(93/146): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 11 MB/s | 562 kB 00:00 2024-03-14T09:42:48.440 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-14T09:42:48.462 INFO:teuthology.orchestra.run.smithi152.stdout:(94/146): openssl-devel-1.1.1k-12.el8_9.x86_64. 23 MB/s | 2.3 MB 00:00 2024-03-14T09:42:48.530 INFO:teuthology.orchestra.run.smithi152.stdout:(95/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 707 kB/s | 47 kB 00:00 2024-03-14T09:42:48.588 INFO:teuthology.orchestra.run.smithi152.stdout:(96/146): python3-cryptography-3.2.1-7.el8_9.x8 9.4 MB/s | 559 kB 00:00 2024-03-14T09:42:48.672 INFO:teuthology.orchestra.run.smithi152.stdout:(97/146): python3-packaging-16.8-9.el8.noarch.r 579 kB/s | 48 kB 00:00 2024-03-14T09:42:48.674 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-03-14T09:42:48.714 INFO:teuthology.orchestra.run.smithi152.stdout:(98/146): python3-jsonpointer-1.10-11.el8.noarc 480 kB/s | 20 kB 00:00 2024-03-14T09:42:48.764 INFO:teuthology.orchestra.run.smithi152.stdout:(99/146): python3-prettytable-0.7.2-14.el8.noar 881 kB/s | 44 kB 00:00 2024-03-14T09:42:48.806 INFO:teuthology.orchestra.run.smithi152.stdout:(100/146): python3-pyasn1-modules-0.3.7-6.el8.n 2.6 MB/s | 110 kB 00:00 2024-03-14T09:42:48.856 INFO:teuthology.orchestra.run.smithi152.stdout:(101/146): python3-jsonpatch-1.21-2.el8.noarch. 532 kB/s | 27 kB 00:00 2024-03-14T09:42:48.924 INFO:teuthology.orchestra.run.smithi152.stdout:(102/146): python3-pyasn1-0.3.7-6.el8.noarch.rp 1.8 MB/s | 126 kB 00:00 2024-03-14T09:42:48.990 INFO:teuthology.orchestra.run.smithi152.stdout:(103/146): python3-werkzeug-0.12.2-4.el8.noarch 6.7 MB/s | 457 kB 00:00 2024-03-14T09:42:49.002 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-03-14T09:42:49.049 INFO:teuthology.orchestra.run.smithi152.stdout:(104/146): bison-3.0.4-10.el8.x86_64.rpm 12 MB/s | 688 kB 00:00 2024-03-14T09:42:49.108 INFO:teuthology.orchestra.run.smithi152.stdout:(105/146): flex-2.6.1-9.el8.x86_64.rpm 5.4 MB/s | 320 kB 00:00 2024-03-14T09:42:49.200 INFO:teuthology.orchestra.run.smithi152.stdout:(106/146): libtool-2.4.6-25.el8.x86_64.rpm 7.6 MB/s | 709 kB 00:00 2024-03-14T09:42:49.210 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-03-14T09:42:49.292 INFO:teuthology.orchestra.run.smithi152.stdout:(107/146): python3-markupsafe-0.23-19.el8.x86_6 424 kB/s | 39 kB 00:00 2024-03-14T09:42:49.342 INFO:teuthology.orchestra.run.smithi152.stdout:(108/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2024-03-14T09:42:49.417 INFO:teuthology.orchestra.run.smithi152.stdout:(109/146): uuid-1.6.2-43.el8.x86_64.rpm 854 kB/s | 64 kB 00:00 2024-03-14T09:42:49.476 INFO:teuthology.orchestra.run.smithi152.stdout:(110/146): python3-pyOpenSSL-19.0.0-1.el8.noarc 1.7 MB/s | 103 kB 00:00 2024-03-14T09:42:49.636 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-03-14T09:42:49.702 INFO:teuthology.orchestra.run.smithi152.stdout:(111/146): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2024-03-14T09:42:49.852 INFO:teuthology.orchestra.run.smithi152.stdout:(112/146): python3-jinja2-2.10.1-3.el8.noarch.r 3.5 MB/s | 538 kB 00:00 2024-03-14T09:42:49.853 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-14T09:42:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-03-14T09:42:50.037 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephsqlite-2:18.2.2-825.g2f465319.el8.x86_64 19/162 2024-03-14T09:42:50.069 INFO:teuthology.orchestra.run.smithi152.stdout:(113/146): python3-numpy-1.14.3-10.el8.x86_64.r 17 MB/s | 3.7 MB 00:00 2024-03-14T09:42:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephsqlite-2:18.2.2-825.g2f465319.el8.x86_64 19/162 2024-03-14T09:42:50.202 INFO:teuthology.orchestra.run.smithi152.stdout:(114/146): socat-1.7.4.1-1.el8.x86_64.rpm 2.4 MB/s | 323 kB 00:00 2024-03-14T09:42:50.242 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 35/162 2024-03-14T09:42:50.244 INFO:teuthology.orchestra.run.smithi152.stdout:(115/146): python3-kubernetes-11.0.0-6.el8.noar 73 kB/s | 1.2 MB 00:16 2024-03-14T09:42:50.278 INFO:teuthology.orchestra.run.smithi152.stdout:(116/146): python3-numpy-f2py-1.14.3-10.el8.x86 2.9 MB/s | 225 kB 00:00 2024-03-14T09:42:50.345 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libradosstriper1-2:18.2.2-825.g2f465319.el8.x86_ 20/162 2024-03-14T09:42:50.445 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libradosstriper1-2:18.2.2-825.g2f465319.el8.x86_ 20/162 2024-03-14T09:42:50.462 INFO:teuthology.orchestra.run.smithi152.stdout:(117/146): librdkafka-0.11.4-3.el8.x86_64.rpm 1.6 MB/s | 354 kB 00:00 2024-03-14T09:42:50.587 INFO:teuthology.orchestra.run.smithi152.stdout:(118/146): openblas-threads-0.3.15-6.el8.x86_64 17 MB/s | 5.2 MB 00:00 2024-03-14T09:42:50.612 INFO:teuthology.orchestra.run.smithi152.stdout:(119/146): python3-mako-1.0.6-14.el8.noarch.rpm 1.0 MB/s | 157 kB 00:00 2024-03-14T09:42:50.635 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 21/162 2024-03-14T09:42:50.787 INFO:teuthology.orchestra.run.smithi152.stdout:(120/146): openblas-0.3.15-6.el8.x86_64.rpm 25 MB/s | 5.0 MB 00:00 2024-03-14T09:42:50.821 INFO:teuthology.orchestra.run.smithi152.stdout:(121/146): automake-1.16.1-8.el8.noarch.rpm 3.3 MB/s | 714 kB 00:00 2024-03-14T09:42:50.911 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/162 2024-03-14T09:42:50.930 INFO:teuthology.orchestra.run.smithi152.stdout:(122/146): libpq-13.11-1.el8.x86_64.rpm 1.4 MB/s | 198 kB 00:00 2024-03-14T09:42:51.013 INFO:teuthology.orchestra.run.smithi152.stdout:(123/146): jq-1.6-7.el8.x86_64.rpm 1.0 MB/s | 202 kB 00:00 2024-03-14T09:42:51.038 INFO:teuthology.orchestra.run.smithi152.stdout:(124/146): python3-pytz-2017.2-11.el8.noarch.rp 499 kB/s | 54 kB 00:00 2024-03-14T09:42:51.160 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/162 2024-03-14T09:42:51.189 INFO:teuthology.orchestra.run.smithi152.stdout:(125/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-03-14T09:42:51.198 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 36/162 2024-03-14T09:42:51.345 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-14T09:42:51.448 INFO:teuthology.orchestra.run.smithi152.stdout:(126/146): python3-scipy-1.0.0-21.module+el8.9. 32 MB/s | 14 MB 00:00 2024-03-14T09:42:51.490 INFO:teuthology.orchestra.run.smithi152.stdout:(127/146): postgresql-10.23-4.module+el8.9.0+21 5.0 MB/s | 1.5 MB 00:00 2024-03-14T09:42:51.515 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-14T09:42:51.519 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 37/162 2024-03-14T09:42:51.557 INFO:teuthology.orchestra.run.smithi152.stdout:(128/146): postgresql-contrib-10.23-4.module+el 7.3 MB/s | 812 kB 00:00 2024-03-14T09:42:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-14T09:42:51.715 INFO:teuthology.orchestra.run.smithi152.stdout:(129/146): postgresql-server-10.23-4.module+el8 22 MB/s | 5.1 MB 00:00 2024-03-14T09:42:51.765 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 38/162 2024-03-14T09:42:51.991 INFO:teuthology.orchestra.run.smithi152.stdout:(130/146): librados2-18.2.2-825.g2f465319.el8.x 8.5 MB/s | 3.7 MB 00:00 2024-03-14T09:42:52.017 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-03-14T09:42:52.192 INFO:teuthology.orchestra.run.smithi152.stdout:(131/146): elfutils-libelf-0.189-3.el8.x86_64.r 1.1 MB/s | 232 kB 00:00 2024-03-14T09:42:52.233 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-03-14T09:42:52.341 INFO:teuthology.orchestra.run.smithi152.stdout:(132/146): elfutils-libs-0.189-3.el8.x86_64.rpm 2.0 MB/s | 303 kB 00:00 2024-03-14T09:42:52.395 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-14T09:42:52.411 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rbd-2:18.2.2-825.g2f465319.el8.x86_64 41/162 2024-03-14T09:42:52.475 INFO:teuthology.orchestra.run.smithi152.stdout:(133/146): librbd1-18.2.2-825.g2f465319.el8.x86 4.7 MB/s | 3.6 MB 00:00 2024-03-14T09:42:52.501 INFO:teuthology.orchestra.run.smithi152.stdout:(134/146): libmount-2.32.1-43.el8.x86_64.rpm 1.5 MB/s | 237 kB 00:00 2024-03-14T09:42:52.534 INFO:teuthology.orchestra.run.smithi152.stdout:(135/146): libsmartcols-2.32.1-43.el8.x86_64.rp 3.0 MB/s | 179 kB 00:00 2024-03-14T09:42:52.570 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-14T09:42:52.603 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados-devel-2:18.2.2-825.g2f465319.el8.x86_64 42/162 2024-03-14T09:42:52.668 INFO:teuthology.orchestra.run.smithi152.stdout:(136/146): libuuid-2.32.1-43.el8.x86_64.rpm 592 kB/s | 99 kB 00:00 2024-03-14T09:42:52.789 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-03-14T09:42:52.793 INFO:teuthology.orchestra.run.smithi152.stdout:(137/146): util-linux-2.32.1-43.el8.x86_64.rpm 9.6 MB/s | 2.5 MB 00:00 2024-03-14T09:42:52.810 INFO:teuthology.orchestra.run.smithi152.stdout:(138/146): util-linux-user-2.32.1-43.el8.x86_64 724 kB/s | 103 kB 00:00 2024-03-14T09:42:52.871 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-03-14T09:42:52.977 INFO:teuthology.orchestra.run.smithi152.stdout:(139/146): iproute-6.2.0-5.el8_9.x86_64.rpm 4.5 MB/s | 854 kB 00:00 2024-03-14T09:42:52.994 INFO:teuthology.orchestra.run.smithi152.stdout:(140/146): libblkid-2.32.1-43.el8.x86_64.rpm 1.2 MB/s | 220 kB 00:00 2024-03-14T09:42:53.072 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-14T09:42:53.077 INFO:teuthology.orchestra.run.smithi152.stdout:(141/146): libfdisk-2.32.1-43.el8.x86_64.rpm 2.5 MB/s | 254 kB 00:00 2024-03-14T09:42:53.161 INFO:teuthology.orchestra.run.smithi152.stdout:(142/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 4.2 MB/s | 711 kB 00:00 2024-03-14T09:42:53.164 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-14T09:42:53.226 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-14T09:42:53.253 INFO:teuthology.orchestra.run.smithi152.stdout:(143/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 8.4 MB/s | 1.5 MB 00:00 2024-03-14T09:42:53.312 INFO:teuthology.orchestra.run.smithi152.stdout:(144/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 2.2 MB/s | 333 kB 00:00 2024-03-14T09:42:53.367 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-03-14T09:42:53.378 INFO:teuthology.orchestra.run.smithi152.stdout:(145/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 8.5 MB/s | 1.1 MB 00:00 2024-03-14T09:42:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-0.3.15-6.el8.x86_64 27/162 2024-03-14T09:42:53.558 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-03-14T09:42:53.781 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-14T09:42:54.189 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 27/162 2024-03-14T09:42:54.217 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-14T09:42:54.469 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 28/162 2024-03-14T09:42:54.548 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-14T09:42:54.573 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-14T09:42:54.686 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 29/162 2024-03-14T09:42:54.741 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-packaging-16.8-9.el8.noarch 49/162 2024-03-14T09:42:54.889 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 50/162 2024-03-14T09:42:54.952 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 50/162 2024-03-14T09:42:54.962 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-03-14T09:42:55.195 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-common-2:18.2.2-825.g2f465319.el8.x 31/162 2024-03-14T09:42:55.224 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : jq-1.6-7.el8.x86_64 51/162 2024-03-14T09:42:55.341 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-argparse-2:18.2.2-825.g2f465319.el8 32/162 2024-03-14T09:42:55.447 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : automake-1.16.1-8.el8.noarch 52/162 2024-03-14T09:42:55.613 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : uuid-1.6.2-43.el8.x86_64 53/162 2024-03-14T09:42:55.618 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cephfs-2:18.2.2-825.g2f465319.el8.x86_64 33/162 2024-03-14T09:42:55.721 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 53/162 2024-03-14T09:42:55.848 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-14T09:42:55.895 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 54/162 2024-03-14T09:42:56.071 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 55/162 2024-03-14T09:42:56.231 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 56/162 2024-03-14T09:42:56.476 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : fuse-2.9.7-17.el8.x86_64 57/162 2024-03-14T09:42:56.670 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 58/162 2024-03-14T09:42:56.852 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 59/162 2024-03-14T09:42:57.063 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 60/162 2024-03-14T09:42:57.268 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 61/162 2024-03-14T09:42:57.569 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 62/162 2024-03-14T09:42:57.844 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 63/162 2024-03-14T09:42:57.863 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-14T09:42:58.043 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 64/162 2024-03-14T09:42:58.201 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 65/162 2024-03-14T09:42:58.250 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 35/162 2024-03-14T09:42:58.449 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 66/162 2024-03-14T09:42:58.804 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 67/162 2024-03-14T09:42:59.117 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 68/162 2024-03-14T09:42:59.311 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 36/162 2024-03-14T09:42:59.395 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 69/162 2024-03-14T09:42:59.651 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 37/162 2024-03-14T09:42:59.709 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 70/162 2024-03-14T09:42:59.893 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 38/162 2024-03-14T09:43:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-03-14T09:43:00.234 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 71/162 2024-03-14T09:43:00.383 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-03-14T09:43:00.515 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 72/162 2024-03-14T09:43:00.568 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rbd-2:18.2.2-825.g2f465319.el8.x86_64 41/162 2024-03-14T09:43:00.750 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librados-devel-2:18.2.2-825.g2f465319.el8.x86_64 42/162 2024-03-14T09:43:00.753 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 73/162 2024-03-14T09:43:00.952 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-03-14T09:43:00.981 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/162 2024-03-14T09:43:01.059 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-03-14T09:43:01.182 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 75/162 2024-03-14T09:43:01.294 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-14T09:43:01.355 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 76/162 2024-03-14T09:43:01.388 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-14T09:43:01.607 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 77/162 2024-03-14T09:43:01.611 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-03-14T09:43:01.772 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-03-14T09:43:01.782 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-03-14T09:43:01.951 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-03-14T09:43:01.960 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-14T09:43:02.252 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-03-14T09:43:02.461 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-03-14T09:43:02.551 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-14T09:43:02.642 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-03-14T09:43:02.849 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-03-14T09:43:02.878 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-14T09:43:02.922 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-14T09:43:03.048 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-03-14T09:43:03.069 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 49/162 2024-03-14T09:43:03.151 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 49/162 2024-03-14T09:43:03.259 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-03-14T09:43:03.418 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : jq-1.6-7.el8.x86_64 50/162 2024-03-14T09:43:03.527 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-03-14T09:43:03.686 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : automake-1.16.1-8.el8.noarch 51/162 2024-03-14T09:43:03.724 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-03-14T09:43:03.786 INFO:teuthology.orchestra.run.smithi152.stdout:(146/146): thrift-0.13.0-2.el8.x86_64.rpm 106 kB/s | 1.7 MB 00:16 2024-03-14T09:43:03.789 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-03-14T09:43:03.789 INFO:teuthology.orchestra.run.smithi152.stdout:Total 4.0 MB/s | 212 MB 00:52 2024-03-14T09:43:03.868 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-03-14T09:43:03.877 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : uuid-1.6.2-43.el8.x86_64 52/162 2024-03-14T09:43:03.970 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 52/162 2024-03-14T09:43:04.125 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-03-14T09:43:04.160 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 53/162 2024-03-14T09:43:04.349 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-03-14T09:43:04.361 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 54/162 2024-03-14T09:43:04.518 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-03-14T09:43:04.549 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 55/162 2024-03-14T09:43:04.775 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-packaging-16.8-9.el8.noarch 56/162 2024-03-14T09:43:04.783 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-03-14T09:43:04.795 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-03-14T09:43:04.966 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : fuse-2.9.7-17.el8.x86_64 57/162 2024-03-14T09:43:05.020 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-03-14T09:43:05.190 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-03-14T09:43:05.190 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-03-14T09:43:05.197 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 58/162 2024-03-14T09:43:05.211 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-03-14T09:43:05.394 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 59/162 2024-03-14T09:43:05.512 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-03-14T09:43:05.565 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 60/162 2024-03-14T09:43:05.758 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 61/162 2024-03-14T09:43:06.093 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 62/162 2024-03-14T09:43:06.367 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 63/162 2024-03-14T09:43:06.401 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-03-14T09:43:06.411 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-03-14T09:43:06.577 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-03-14T09:43:06.591 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 64/162 2024-03-14T09:43:06.769 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 65/162 2024-03-14T09:43:06.874 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-03-14T09:43:07.019 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 66/162 2024-03-14T09:43:07.086 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-03-14T09:43:07.411 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-03-14T09:43:07.442 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 67/162 2024-03-14T09:43:07.663 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-03-14T09:43:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 68/162 2024-03-14T09:43:07.903 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-03-14T09:43:08.005 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 69/162 2024-03-14T09:43:08.228 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 70/162 2024-03-14T09:43:08.423 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 71/162 2024-03-14T09:43:08.698 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-03-14T09:43:08.716 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 72/162 2024-03-14T09:43:08.861 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librgw2-2:18.2.2-825.g2f465319.el8.x86_64 103/162 2024-03-14T09:43:08.937 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librgw2-2:18.2.2-825.g2f465319.el8.x86_64 103/162 2024-03-14T09:43:08.969 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 73/162 2024-03-14T09:43:09.154 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rgw-2:18.2.2-825.g2f465319.el8.x86_64 104/162 2024-03-14T09:43:09.196 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/162 2024-03-14T09:43:09.387 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 75/162 2024-03-14T09:43:09.601 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 76/162 2024-03-14T09:43:09.863 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 77/162 2024-03-14T09:43:10.036 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-03-14T09:43:10.242 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-03-14T09:43:10.507 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-03-14T09:43:10.734 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-03-14T09:43:10.890 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-03-14T09:43:11.056 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-03-14T09:43:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-03-14T09:43:11.534 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-03-14T09:43:11.749 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-03-14T09:43:11.922 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-03-14T09:43:12.083 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-03-14T09:43:12.315 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-03-14T09:43:12.443 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-03-14T09:43:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-03-14T09:43:12.577 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-03-14T09:43:12.699 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-03-14T09:43:12.882 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-14T09:43:12.968 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-03-14T09:43:13.101 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-14T09:43:13.178 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 105/162 2024-03-14T09:43:13.209 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-03-14T09:43:13.308 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-03-14T09:43:13.403 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 105/162 2024-03-14T09:43:13.435 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-03-14T09:43:13.482 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-14T09:43:13.596 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-14T09:43:13.680 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-03-14T09:43:13.807 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-14T09:43:13.874 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-14T09:43:13.982 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 105/162 2024-03-14T09:43:14.111 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-03-14T09:43:14.289 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-base-2:18.2.2-825.g2f465319.el8.x86_64 106/162 2024-03-14T09:43:14.331 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 6/162 2024-03-14T09:43:14.373 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-base-2:18.2.2-825.g2f465319.el8.x86_64 106/162 2024-03-14T09:43:14.502 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-selinux-2:18.2.2-825.g2f465319.el8.x86_64 107/162 2024-03-14T09:43:14.557 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-14T09:43:14.658 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-14T09:43:14.890 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-14T09:43:14.965 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-03-14T09:43:15.286 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-03-14T09:43:15.306 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-14T09:43:15.507 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librados2-2:18.2.2-825.g2f465319.el8.x86_64 9/162 2024-03-14T09:43:15.522 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-03-14T09:43:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-03-14T09:43:15.928 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:18.2.2-825.g2f465319.el8.x86_64 9/162 2024-03-14T09:43:16.132 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librbd1-2:18.2.2-825.g2f465319.el8.x86_64 10/162 2024-03-14T09:43:16.170 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-03-14T09:43:16.257 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:18.2.2-825.g2f465319.el8.x86_64 10/162 2024-03-14T09:43:16.391 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-03-14T09:43:16.473 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs2-2:18.2.2-825.g2f465319.el8.x86_64 11/162 2024-03-14T09:43:16.603 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs2-2:18.2.2-825.g2f465319.el8.x86_64 11/162 2024-03-14T09:43:16.933 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rados-2:18.2.2-825.g2f465319.el8.x86_64 12/162 2024-03-14T09:43:17.199 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-03-14T09:43:17.223 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-14T09:43:17.328 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-14T09:43:17.367 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librgw2-2:18.2.2-825.g2f465319.el8.x86_64 103/162 2024-03-14T09:43:17.471 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librgw2-2:18.2.2-825.g2f465319.el8.x86_64 103/162 2024-03-14T09:43:17.588 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-03-14T09:43:17.651 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rgw-2:18.2.2-825.g2f465319.el8.x86_64 104/162 2024-03-14T09:43:17.862 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-03-14T09:43:18.076 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-03-14T09:43:18.477 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-03-14T09:43:18.735 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-03-14T09:43:18.923 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephsqlite-2:18.2.2-825.g2f465319.el8.x86_64 19/162 2024-03-14T09:43:19.013 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephsqlite-2:18.2.2-825.g2f465319.el8.x86_64 19/162 2024-03-14T09:43:19.173 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libradosstriper1-2:18.2.2-825.g2f465319.el8.x86_ 20/162 2024-03-14T09:43:19.236 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libradosstriper1-2:18.2.2-825.g2f465319.el8.x86_ 20/162 2024-03-14T09:43:19.443 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 21/162 2024-03-14T09:43:19.740 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/162 2024-03-14T09:43:20.036 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/162 2024-03-14T09:43:20.223 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-14T09:43:20.354 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-14T09:43:20.514 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-14T09:43:21.164 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-14T09:43:21.373 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-14T09:43:21.979 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-14T09:43:22.172 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-0.3.15-6.el8.x86_64 27/162 2024-03-14T09:43:22.443 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 105/162 2024-03-14T09:43:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 105/162 2024-03-14T09:43:22.843 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 27/162 2024-03-14T09:43:23.137 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 28/162 2024-03-14T09:43:23.344 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 29/162 2024-03-14T09:43:23.346 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 105/162 2024-03-14T09:43:23.581 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-03-14T09:43:23.633 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-base-2:18.2.2-825.g2f465319.el8.x86_64 106/162 2024-03-14T09:43:23.774 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-base-2:18.2.2-825.g2f465319.el8.x86_64 106/162 2024-03-14T09:43:23.789 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-common-2:18.2.2-825.g2f465319.el8.x 31/162 2024-03-14T09:43:23.949 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-selinux-2:18.2.2-825.g2f465319.el8.x86_64 107/162 2024-03-14T09:43:23.966 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-argparse-2:18.2.2-825.g2f465319.el8 32/162 2024-03-14T09:43:24.251 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cephfs-2:18.2.2-825.g2f465319.el8.x86_64 33/162 2024-03-14T09:43:24.493 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-14T09:43:26.237 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-14T09:43:26.625 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 35/162 2024-03-14T09:43:27.642 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 36/162 2024-03-14T09:43:27.977 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 37/162 2024-03-14T09:43:28.247 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 38/162 2024-03-14T09:43:28.507 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-03-14T09:43:28.774 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-03-14T09:43:29.010 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rbd-2:18.2.2-825.g2f465319.el8.x86_64 41/162 2024-03-14T09:43:29.211 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librados-devel-2:18.2.2-825.g2f465319.el8.x86_64 42/162 2024-03-14T09:43:29.438 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-03-14T09:43:29.514 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-03-14T09:43:29.779 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-14T09:43:29.848 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-14T09:43:30.109 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-03-14T09:43:30.315 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-03-14T09:43:30.546 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-14T09:43:30.964 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-14T09:43:31.330 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-14T09:43:31.357 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-14T09:43:31.538 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 49/162 2024-03-14T09:43:31.603 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 49/162 2024-03-14T09:43:31.888 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : jq-1.6-7.el8.x86_64 50/162 2024-03-14T09:43:32.155 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : automake-1.16.1-8.el8.noarch 51/162 2024-03-14T09:43:32.354 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : uuid-1.6.2-43.el8.x86_64 52/162 2024-03-14T09:43:32.423 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 52/162 2024-03-14T09:43:32.661 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 53/162 2024-03-14T09:43:32.870 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 54/162 2024-03-14T09:43:33.107 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 55/162 2024-03-14T09:43:33.323 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-packaging-16.8-9.el8.noarch 56/162 2024-03-14T09:43:33.542 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fuse-2.9.7-17.el8.x86_64 57/162 2024-03-14T09:43:33.778 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 58/162 2024-03-14T09:43:33.976 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 59/162 2024-03-14T09:43:34.214 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 60/162 2024-03-14T09:43:34.433 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 61/162 2024-03-14T09:43:34.730 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-selinux-2:18.2.2-825.g2f465319.el8.x86_64 107/162 2024-03-14T09:43:34.731 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /sys 2024-03-14T09:43:34.731 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /proc 2024-03-14T09:43:34.731 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /dev 2024-03-14T09:43:34.731 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /run 2024-03-14T09:43:34.731 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /mnt 2024-03-14T09:43:34.731 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /var/tmp 2024-03-14T09:43:34.731 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /home 2024-03-14T09:43:34.731 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /tmp 2024-03-14T09:43:34.731 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /dev 2024-03-14T09:43:34.731 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-14T09:43:34.756 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 62/162 2024-03-14T09:43:35.051 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 63/162 2024-03-14T09:43:35.091 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-volume-2:18.2.2-825.g2f465319.el8.noarch 108/162 2024-03-14T09:43:35.317 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 64/162 2024-03-14T09:43:35.474 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 65/162 2024-03-14T09:43:35.705 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 66/162 2024-03-14T09:43:36.087 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 67/162 2024-03-14T09:43:36.440 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 68/162 2024-03-14T09:43:36.742 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-volume-2:18.2.2-825.g2f465319.el8.noarch 108/162 2024-03-14T09:43:36.786 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 69/162 2024-03-14T09:43:36.918 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-osd-2:18.2.2-825.g2f465319.el8.x86_64 109/162 2024-03-14T09:43:37.003 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 70/162 2024-03-14T09:43:37.144 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-osd-2:18.2.2-825.g2f465319.el8.x86_64 109/162 2024-03-14T09:43:37.213 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 71/162 2024-03-14T09:43:37.282 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mds-2:18.2.2-825.g2f465319.el8.x86_64 110/162 2024-03-14T09:43:37.518 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 72/162 2024-03-14T09:43:37.759 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mds-2:18.2.2-825.g2f465319.el8.x86_64 110/162 2024-03-14T09:43:37.766 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 73/162 2024-03-14T09:43:37.907 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mon-2:18.2.2-825.g2f465319.el8.x86_64 111/162 2024-03-14T09:43:37.925 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mon-2:18.2.2-825.g2f465319.el8.x86_64 111/162 2024-03-14T09:43:37.954 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/162 2024-03-14T09:43:38.180 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 75/162 2024-03-14T09:43:38.417 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 76/162 2024-03-14T09:43:38.721 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 77/162 2024-03-14T09:43:38.928 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-03-14T09:43:39.150 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-03-14T09:43:39.501 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-03-14T09:43:39.696 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:18.2.2-825.g2f465319.el8.noarch 112/162 2024-03-14T09:43:39.793 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-03-14T09:43:39.846 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephadm-2:18.2.2-825.g2f465319.el8.noarch 112/162 2024-03-14T09:43:39.991 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-03-14T09:43:40.005 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-825.g2f465319.el 113/162 2024-03-14T09:43:40.148 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-03-14T09:43:40.175 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-825.g2f465319.e 114/162 2024-03-14T09:43:40.385 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-03-14T09:43:40.390 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-825.g2f465319.el8.noar 115/162 2024-03-14T09:43:40.615 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-825.g2f465319.el8.noar 115/162 2024-03-14T09:43:40.632 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-03-14T09:43:40.926 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-825.g2f465319.el8.no 116/162 2024-03-14T09:43:40.929 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-03-14T09:43:41.140 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-03-14T09:43:41.384 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-03-14T09:43:41.673 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-03-14T09:43:41.894 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-825.g2f465319.el8.no 116/162 2024-03-14T09:43:41.939 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-03-14T09:43:42.065 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-825.g2f46 117/162 2024-03-14T09:43:42.159 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-825.g2f46 117/162 2024-03-14T09:43:42.166 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-03-14T09:43:42.440 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-03-14T09:43:42.505 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-825.g2f465319.el8 118/162 2024-03-14T09:43:42.676 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-2:18.2.2-825.g2f465319.el8.x86_64 119/162 2024-03-14T09:43:42.710 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-03-14T09:43:42.714 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-2:18.2.2-825.g2f465319.el8.x86_64 119/162 2024-03-14T09:43:42.856 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-rook-2:18.2.2-825.g2f465319.el8.noarch 120/162 2024-03-14T09:43:42.868 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-825.g2f465319.el8.noarch 120/162 2024-03-14T09:43:42.935 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-03-14T09:43:43.201 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-03-14T09:43:44.122 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-2:18.2.2-825.g2f465319.el8.x86_64 121/162 2024-03-14T09:43:44.310 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-03-14T09:43:44.325 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-radosgw-2:18.2.2-825.g2f465319.el8.x86_64 122/162 2024-03-14T09:43:44.361 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-825.g2f465319.el8.x86_64 122/162 2024-03-14T09:43:44.565 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-825.g2f4653 123/162 2024-03-14T09:43:44.608 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-825.g2f4653 123/162 2024-03-14T09:43:44.690 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-03-14T09:43:44.756 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephfs-mirror-2:18.2.2-825.g2f465319.el8.x86_64 124/162 2024-03-14T09:43:44.968 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-03-14T09:43:45.015 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-selinux-2:18.2.2-825.g2f465319.el8.x86_64 107/162 2024-03-14T09:43:45.015 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /sys 2024-03-14T09:43:45.015 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /proc 2024-03-14T09:43:45.015 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /dev 2024-03-14T09:43:45.015 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /run 2024-03-14T09:43:45.016 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /mnt 2024-03-14T09:43:45.016 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /var/tmp 2024-03-14T09:43:45.016 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /home 2024-03-14T09:43:45.016 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /tmp 2024-03-14T09:43:45.016 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /dev 2024-03-14T09:43:45.016 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:43:45.103 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-825.g2f465319.el8.x86_64 124/162 2024-03-14T09:43:45.290 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-mirror-2:18.2.2-825.g2f465319.el8.x86_64 125/162 2024-03-14T09:43:45.295 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-03-14T09:43:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-volume-2:18.2.2-825.g2f465319.el8.noarch 108/162 2024-03-14T09:43:45.628 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-03-14T09:43:45.843 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-03-14T09:43:46.633 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-03-14T09:43:46.968 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librgw2-2:18.2.2-825.g2f465319.el8.x86_64 103/162 2024-03-14T09:43:47.044 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librgw2-2:18.2.2-825.g2f465319.el8.x86_64 103/162 2024-03-14T09:43:47.135 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-volume-2:18.2.2-825.g2f465319.el8.noarch 108/162 2024-03-14T09:43:47.252 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rgw-2:18.2.2-825.g2f465319.el8.x86_64 104/162 2024-03-14T09:43:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-osd-2:18.2.2-825.g2f465319.el8.x86_64 109/162 2024-03-14T09:43:47.610 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-osd-2:18.2.2-825.g2f465319.el8.x86_64 109/162 2024-03-14T09:43:47.781 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mds-2:18.2.2-825.g2f465319.el8.x86_64 110/162 2024-03-14T09:43:48.300 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mds-2:18.2.2-825.g2f465319.el8.x86_64 110/162 2024-03-14T09:43:48.472 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mon-2:18.2.2-825.g2f465319.el8.x86_64 111/162 2024-03-14T09:43:48.515 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mon-2:18.2.2-825.g2f465319.el8.x86_64 111/162 2024-03-14T09:43:49.713 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-mirror-2:18.2.2-825.g2f465319.el8.x86_64 125/162 2024-03-14T09:43:50.670 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-test-2:18.2.2-825.g2f465319.el8.x86_64 126/162 2024-03-14T09:43:50.976 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 127/162 2024-03-14T09:43:51.049 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 105/162 2024-03-14T09:43:51.214 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:18.2.2-825.g2f465319.el8.noarch 112/162 2024-03-14T09:43:51.229 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 128/162 2024-03-14T09:43:51.310 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 105/162 2024-03-14T09:43:51.408 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephadm-2:18.2.2-825.g2f465319.el8.noarch 112/162 2024-03-14T09:43:51.534 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-825.g2f465319.el 113/162 2024-03-14T09:43:51.624 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-fuse-2:18.2.2-825.g2f465319.el8.x86_64 129/162 2024-03-14T09:43:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-825.g2f465319.e 114/162 2024-03-14T09:43:51.891 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 105/162 2024-03-14T09:43:51.907 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 130/162 2024-03-14T09:43:51.971 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-825.g2f465319.el8.noar 115/162 2024-03-14T09:43:52.166 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libtool-2.4.6-25.el8.x86_64 131/162 2024-03-14T09:43:52.180 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 131/162 2024-03-14T09:43:52.240 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-825.g2f465319.el8.noar 115/162 2024-03-14T09:43:52.383 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-03-14T09:43:52.504 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-825.g2f465319.el8.no 116/162 2024-03-14T09:43:52.551 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-03-14T09:43:52.750 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs-devel-2:18.2.2-825.g2f465319.el8.x86_6 134/162 2024-03-14T09:43:53.245 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-base-2:18.2.2-825.g2f465319.el8.x86_64 106/162 2024-03-14T09:43:53.581 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-825.g2f465319.el8.no 116/162 2024-03-14T09:43:53.762 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-825.g2f46 117/162 2024-03-14T09:43:53.795 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-base-2:18.2.2-825.g2f465319.el8.x86_64 106/162 2024-03-14T09:43:53.894 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-825.g2f46 117/162 2024-03-14T09:43:54.117 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-selinux-2:18.2.2-825.g2f465319.el8.x86_64 107/162 2024-03-14T09:43:54.277 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-825.g2f465319.el8 118/162 2024-03-14T09:43:54.446 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-2:18.2.2-825.g2f465319.el8.x86_64 119/162 2024-03-14T09:43:54.516 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-2:18.2.2-825.g2f465319.el8.x86_64 119/162 2024-03-14T09:43:54.687 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-rook-2:18.2.2-825.g2f465319.el8.noarch 120/162 2024-03-14T09:43:54.718 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-825.g2f465319.el8.noarch 120/162 2024-03-14T09:43:55.007 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-14T09:43:55.338 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-14T09:43:55.376 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-14T09:43:55.762 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-03-14T09:43:56.055 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-2:18.2.2-825.g2f465319.el8.x86_64 121/162 2024-03-14T09:43:56.162 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-03-14T09:43:56.254 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-radosgw-2:18.2.2-825.g2f465319.el8.x86_64 122/162 2024-03-14T09:43:56.318 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-825.g2f465319.el8.x86_64 122/162 2024-03-14T09:43:56.358 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-03-14T09:43:56.516 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-825.g2f4653 123/162 2024-03-14T09:43:56.538 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephfs-top-2:18.2.2-825.g2f465319.el8.noarch 139/162 2024-03-14T09:43:56.589 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-825.g2f4653 123/162 2024-03-14T09:43:56.762 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-mirror-2:18.2.2-825.g2f465319.el8.x86_64 124/162 2024-03-14T09:43:56.763 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-fuse-2:18.2.2-825.g2f465319.el8.x86_64 140/162 2024-03-14T09:43:57.068 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-nbd-2:18.2.2-825.g2f465319.el8.x86_64 141/162 2024-03-14T09:43:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-825.g2f465319.el8.x86_64 124/162 2024-03-14T09:43:57.346 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-mirror-2:18.2.2-825.g2f465319.el8.x86_64 125/162 2024-03-14T09:43:57.365 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-03-14T09:43:57.562 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flex-2.6.1-9.el8.x86_64 143/162 2024-03-14T09:43:57.641 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 143/162 2024-03-14T09:43:57.821 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : bison-3.0.4-10.el8.x86_64 144/162 2024-03-14T09:43:57.835 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 144/162 2024-03-14T09:43:58.021 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-14T09:43:58.031 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-14T09:43:58.164 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 146/162 2024-03-14T09:43:58.311 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-03-14T09:43:58.523 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-03-14T09:43:58.691 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-03-14T09:43:58.691 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-14T09:43:58.974 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-14T09:43:58.974 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-14T09:43:59.216 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-14T09:43:59.216 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-14T09:43:59.427 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-14T09:43:59.639 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-03-14T09:43:59.830 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-03-14T09:44:00.047 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-03-14T09:44:00.047 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-14T09:44:00.266 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-14T09:44:00.266 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-03-14T09:44:00.540 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-03-14T09:44:00.540 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-14T09:44:00.789 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-14T09:44:00.789 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-14T09:44:01.014 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-14T09:44:01.014 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-14T09:44:01.264 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-14T09:44:01.409 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-03-14T09:44:01.409 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-14T09:44:02.102 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-mirror-2:18.2.2-825.g2f465319.el8.x86_64 125/162 2024-03-14T09:44:02.289 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-14T09:44:02.289 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-2:18.2.2-825.g2f465319.el8.x86_64 1/162 2024-03-14T09:44:02.289 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-base-2:18.2.2-825.g2f465319.el8.x86_64 2/162 2024-03-14T09:44:02.289 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 3/162 2024-03-14T09:44:02.289 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-fuse-2:18.2.2-825.g2f465319.el8.x86_64 4/162 2024-03-14T09:44:02.289 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-825.g2f4653 5/162 2024-03-14T09:44:02.290 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mds-2:18.2.2-825.g2f465319.el8.x86_64 6/162 2024-03-14T09:44:02.290 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-2:18.2.2-825.g2f465319.el8.x86_64 7/162 2024-03-14T09:44:02.290 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mon-2:18.2.2-825.g2f465319.el8.x86_64 8/162 2024-03-14T09:44:02.290 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-osd-2:18.2.2-825.g2f465319.el8.x86_64 9/162 2024-03-14T09:44:02.290 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-radosgw-2:18.2.2-825.g2f465319.el8.x86_64 10/162 2024-03-14T09:44:02.290 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-selinux-2:18.2.2-825.g2f465319.el8.x86_64 11/162 2024-03-14T09:44:02.290 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-test-2:18.2.2-825.g2f465319.el8.x86_64 12/162 2024-03-14T09:44:02.290 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephfs-mirror-2:18.2.2-825.g2f465319.el8.x86_64 13/162 2024-03-14T09:44:02.290 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs-devel-2:18.2.2-825.g2f465319.el8.x86_6 14/162 2024-03-14T09:44:02.290 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs2-2:18.2.2-825.g2f465319.el8.x86_64 15/162 2024-03-14T09:44:02.290 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephsqlite-2:18.2.2-825.g2f465319.el8.x86_64 16/162 2024-03-14T09:44:02.290 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados-devel-2:18.2.2-825.g2f465319.el8.x86_64 17/162 2024-03-14T09:44:02.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libradosstriper1-2:18.2.2-825.g2f465319.el8.x86_ 18/162 2024-03-14T09:44:02.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librgw2-2:18.2.2-825.g2f465319.el8.x86_64 19/162 2024-03-14T09:44:02.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-argparse-2:18.2.2-825.g2f465319.el8 20/162 2024-03-14T09:44:02.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-common-2:18.2.2-825.g2f465319.el8.x 21/162 2024-03-14T09:44:02.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cephfs-2:18.2.2-825.g2f465319.el8.x86_64 22/162 2024-03-14T09:44:02.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rados-2:18.2.2-825.g2f465319.el8.x86_64 23/162 2024-03-14T09:44:02.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rbd-2:18.2.2-825.g2f465319.el8.x86_64 24/162 2024-03-14T09:44:02.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rgw-2:18.2.2-825.g2f465319.el8.x86_64 25/162 2024-03-14T09:44:02.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-fuse-2:18.2.2-825.g2f465319.el8.x86_64 26/162 2024-03-14T09:44:02.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-mirror-2:18.2.2-825.g2f465319.el8.x86_64 27/162 2024-03-14T09:44:02.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-nbd-2:18.2.2-825.g2f465319.el8.x86_64 28/162 2024-03-14T09:44:02.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-825.g2f465319.e 29/162 2024-03-14T09:44:02.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-825.g2f465319.el8.noar 30/162 2024-03-14T09:44:02.291 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-825.g2f465319.el8.no 31/162 2024-03-14T09:44:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-825.g2f46 32/162 2024-03-14T09:44:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-825.g2f465319.el8 33/162 2024-03-14T09:44:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-rook-2:18.2.2-825.g2f465319.el8.noarch 34/162 2024-03-14T09:44:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-825.g2f465319.el 35/162 2024-03-14T09:44:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-volume-2:18.2.2-825.g2f465319.el8.noarch 36/162 2024-03-14T09:44:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephadm-2:18.2.2-825.g2f465319.el8.noarch 37/162 2024-03-14T09:44:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephfs-top-2:18.2.2-825.g2f465319.el8.noarch 38/162 2024-03-14T09:44:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-03-14T09:44:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-03-14T09:44:02.307 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-03-14T09:44:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-03-14T09:44:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-03-14T09:44:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-03-14T09:44:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-03-14T09:44:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-03-14T09:44:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-03-14T09:44:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-03-14T09:44:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-03-14T09:44:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-03-14T09:44:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-03-14T09:44:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-03-14T09:44:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-03-14T09:44:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-03-14T09:44:02.308 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-03-14T09:44:02.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-03-14T09:44:02.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-03-14T09:44:02.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-03-14T09:44:02.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-03-14T09:44:02.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-03-14T09:44:02.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-03-14T09:44:02.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-03-14T09:44:02.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-03-14T09:44:02.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-03-14T09:44:02.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-03-14T09:44:02.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-03-14T09:44:02.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-03-14T09:44:02.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-03-14T09:44:02.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-03-14T09:44:02.309 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-03-14T09:44:02.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-03-14T09:44:02.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 72/162 2024-03-14T09:44:02.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/162 2024-03-14T09:44:02.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/162 2024-03-14T09:44:02.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 75/162 2024-03-14T09:44:02.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 76/162 2024-03-14T09:44:02.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 77/162 2024-03-14T09:44:02.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 78/162 2024-03-14T09:44:02.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 79/162 2024-03-14T09:44:02.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/162 2024-03-14T09:44:02.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 81/162 2024-03-14T09:44:02.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 82/162 2024-03-14T09:44:02.310 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 83/162 2024-03-14T09:44:02.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 84/162 2024-03-14T09:44:02.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 85/162 2024-03-14T09:44:02.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 86/162 2024-03-14T09:44:02.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 87/162 2024-03-14T09:44:02.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 88/162 2024-03-14T09:44:02.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 89/162 2024-03-14T09:44:02.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 90/162 2024-03-14T09:44:02.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 91/162 2024-03-14T09:44:02.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 92/162 2024-03-14T09:44:02.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 93/162 2024-03-14T09:44:02.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 94/162 2024-03-14T09:44:02.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 95/162 2024-03-14T09:44:02.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 96/162 2024-03-14T09:44:02.311 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 97/162 2024-03-14T09:44:02.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 98/162 2024-03-14T09:44:02.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 99/162 2024-03-14T09:44:02.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 100/162 2024-03-14T09:44:02.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 101/162 2024-03-14T09:44:02.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 102/162 2024-03-14T09:44:02.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 103/162 2024-03-14T09:44:02.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 104/162 2024-03-14T09:44:02.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : bison-3.0.4-10.el8.x86_64 105/162 2024-03-14T09:44:02.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flex-2.6.1-9.el8.x86_64 106/162 2024-03-14T09:44:02.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 107/162 2024-03-14T09:44:02.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 108/162 2024-03-14T09:44:02.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 109/162 2024-03-14T09:44:02.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 110/162 2024-03-14T09:44:02.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 111/162 2024-03-14T09:44:02.312 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 112/162 2024-03-14T09:44:02.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 113/162 2024-03-14T09:44:02.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 114/162 2024-03-14T09:44:02.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 115/162 2024-03-14T09:44:02.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 116/162 2024-03-14T09:44:02.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 117/162 2024-03-14T09:44:02.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 118/162 2024-03-14T09:44:02.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 119/162 2024-03-14T09:44:02.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 120/162 2024-03-14T09:44:02.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : automake-1.16.1-8.el8.noarch 121/162 2024-03-14T09:44:02.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libpq-13.11-1.el8.x86_64 122/162 2024-03-14T09:44:02.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : jq-1.6-7.el8.x86_64 123/162 2024-03-14T09:44:02.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 124/162 2024-03-14T09:44:02.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 125/162 2024-03-14T09:44:02.313 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 126/162 2024-03-14T09:44:02.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 127/162 2024-03-14T09:44:02.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 128/162 2024-03-14T09:44:02.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 129/162 2024-03-14T09:44:02.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 130/162 2024-03-14T09:44:02.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:18.2.2-825.g2f465319.el8.x86_64 131/162 2024-03-14T09:44:02.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-03-14T09:44:02.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:18.2.2-825.g2f465319.el8.x86_64 133/162 2024-03-14T09:44:02.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-03-14T09:44:02.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-03-14T09:44:02.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-03-14T09:44:02.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-03-14T09:44:02.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-03-14T09:44:02.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-03-14T09:44:02.314 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-03-14T09:44:02.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-03-14T09:44:02.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-03-14T09:44:02.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-03-14T09:44:02.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-03-14T09:44:02.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-03-14T09:44:02.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-03-14T09:44:02.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-03-14T09:44:02.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-03-14T09:44:02.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-03-14T09:44:02.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-03-14T09:44:02.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-03-14T09:44:02.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-03-14T09:44:02.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-03-14T09:44:02.315 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-03-14T09:44:02.316 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-03-14T09:44:02.316 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-03-14T09:44:02.316 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-03-14T09:44:02.316 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-03-14T09:44:02.316 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-03-14T09:44:02.316 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-03-14T09:44:02.316 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-03-14T09:44:03.420 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-test-2:18.2.2-825.g2f465319.el8.x86_64 126/162 2024-03-14T09:44:03.725 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 127/162 2024-03-14T09:44:03.982 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 128/162 2024-03-14T09:44:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-fuse-2:18.2.2-825.g2f465319.el8.x86_64 129/162 2024-03-14T09:44:04.796 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 130/162 2024-03-14T09:44:05.020 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libtool-2.4.6-25.el8.x86_64 131/162 2024-03-14T09:44:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 131/162 2024-03-14T09:44:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-03-14T09:44:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-03-14T09:44:05.670 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs-devel-2:18.2.2-825.g2f465319.el8.x86_6 134/162 2024-03-14T09:44:08.856 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-03-14T09:44:08.856 INFO:teuthology.orchestra.run.smithi132.stdout:Installed products updated. 2024-03-14T09:44:08.973 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-14T09:44:09.236 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-14T09:44:09.309 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-14T09:44:09.462 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-14T09:44:09.462 INFO:teuthology.orchestra.run.smithi132.stdout:Upgraded: 2024-03-14T09:44:09.463 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-03-14T09:44:09.463 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-03-14T09:44:09.463 INFO:teuthology.orchestra.run.smithi132.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-03-14T09:44:09.463 INFO:teuthology.orchestra.run.smithi132.stdout: libblkid-2.32.1-43.el8.x86_64 2024-03-14T09:44:09.463 INFO:teuthology.orchestra.run.smithi132.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-03-14T09:44:09.463 INFO:teuthology.orchestra.run.smithi132.stdout: libmount-2.32.1-43.el8.x86_64 2024-03-14T09:44:09.463 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.463 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.463 INFO:teuthology.orchestra.run.smithi132.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-03-14T09:44:09.463 INFO:teuthology.orchestra.run.smithi132.stdout: libuuid-2.32.1-43.el8.x86_64 2024-03-14T09:44:09.463 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-03-14T09:44:09.463 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-03-14T09:44:09.463 INFO:teuthology.orchestra.run.smithi132.stdout: util-linux-2.32.1-43.el8.x86_64 2024-03-14T09:44:09.463 INFO:teuthology.orchestra.run.smithi132.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-03-14T09:44:09.464 INFO:teuthology.orchestra.run.smithi132.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-03-14T09:44:09.464 INFO:teuthology.orchestra.run.smithi132.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-03-14T09:44:09.464 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-14T09:44:09.464 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-03-14T09:44:09.464 INFO:teuthology.orchestra.run.smithi132.stdout: automake-1.16.1-8.el8.noarch 2024-03-14T09:44:09.464 INFO:teuthology.orchestra.run.smithi132.stdout: bison-3.0.4-10.el8.x86_64 2024-03-14T09:44:09.464 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.464 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.464 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.464 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.464 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:09.464 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.464 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.464 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.464 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:09.464 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:09.465 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:09.465 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:09.465 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:09.465 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.465 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.465 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:09.465 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.465 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.465 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.465 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:09.465 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:09.465 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-mirror-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.465 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-top-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:09.465 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-03-14T09:44:09.465 INFO:teuthology.orchestra.run.smithi132.stdout: flex-2.6.1-9.el8.x86_64 2024-03-14T09:44:09.466 INFO:teuthology.orchestra.run.smithi132.stdout: fuse-2.9.7-17.el8.x86_64 2024-03-14T09:44:09.466 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-14T09:44:09.466 INFO:teuthology.orchestra.run.smithi132.stdout: jq-1.6-7.el8.x86_64 2024-03-14T09:44:09.466 INFO:teuthology.orchestra.run.smithi132.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-03-14T09:44:09.466 INFO:teuthology.orchestra.run.smithi132.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-03-14T09:44:09.466 INFO:teuthology.orchestra.run.smithi132.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-03-14T09:44:09.466 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.466 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.466 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.466 INFO:teuthology.orchestra.run.smithi132.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-03-14T09:44:09.466 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-03-14T09:44:09.466 INFO:teuthology.orchestra.run.smithi132.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-14T09:44:09.466 INFO:teuthology.orchestra.run.smithi132.stdout: libpq-13.11-1.el8.x86_64 2024-03-14T09:44:09.467 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-03-14T09:44:09.467 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-03-14T09:44:09.467 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.467 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.469 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-03-14T09:44:09.469 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.469 INFO:teuthology.orchestra.run.smithi132.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-03-14T09:44:09.469 INFO:teuthology.orchestra.run.smithi132.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-03-14T09:44:09.469 INFO:teuthology.orchestra.run.smithi132.stdout: libtool-2.4.6-25.el8.x86_64 2024-03-14T09:44:09.469 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-14T09:44:09.470 INFO:teuthology.orchestra.run.smithi132.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-03-14T09:44:09.470 INFO:teuthology.orchestra.run.smithi132.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-03-14T09:44:09.470 INFO:teuthology.orchestra.run.smithi132.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-03-14T09:44:09.470 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-14T09:44:09.470 INFO:teuthology.orchestra.run.smithi132.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-03-14T09:44:09.470 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-0.3.15-6.el8.x86_64 2024-03-14T09:44:09.470 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-03-14T09:44:09.470 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-03-14T09:44:09.470 INFO:teuthology.orchestra.run.smithi132.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-03-14T09:44:09.470 INFO:teuthology.orchestra.run.smithi132.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-03-14T09:44:09.470 INFO:teuthology.orchestra.run.smithi132.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-03-14T09:44:09.470 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-14T09:44:09.470 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-14T09:44:09.470 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-14T09:44:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-14T09:44:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-14T09:44:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-14T09:44:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-14T09:44:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-14T09:44:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-14T09:44:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-14T09:44:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-14T09:44:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-14T09:44:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-03-14T09:44:09.471 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-14T09:44:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-14T09:44:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-14T09:44:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-03-14T09:44:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-14T09:44:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-14T09:44:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-14T09:44:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-14T09:44:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-14T09:44:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-14T09:44:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-14T09:44:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-14T09:44:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-14T09:44:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-14T09:44:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-14T09:44:09.472 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-14T09:44:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging-16.8-9.el8.noarch 2024-03-14T09:44:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-14T09:44:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend-2.6-1.el8.noarch 2024-03-14T09:44:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-14T09:44:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-14T09:44:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-14T09:44:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-14T09:44:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-14T09:44:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-14T09:44:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-14T09:44:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-14T09:44:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.473 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-14T09:44:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-14T09:44:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-03-14T09:44:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-14T09:44:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-14T09:44:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-14T09:44:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-14T09:44:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-14T09:44:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-14T09:44:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-14T09:44:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-14T09:44:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-14T09:44:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-14T09:44:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.474 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-14T09:44:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-14T09:44:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: uuid-1.6.2-43.el8.x86_64 2024-03-14T09:44:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-03-14T09:44:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-14T09:44:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-03-14T09:44:09.475 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-14T09:44:09.475 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-14T09:44:09.690 DEBUG:teuthology.parallel:result is None 2024-03-14T09:44:09.767 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-03-14T09:44:10.190 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-03-14T09:44:10.440 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-03-14T09:44:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-top-2:18.2.2-825.g2f465319.el8.noarch 139/162 2024-03-14T09:44:10.887 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-fuse-2:18.2.2-825.g2f465319.el8.x86_64 140/162 2024-03-14T09:44:11.240 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-nbd-2:18.2.2-825.g2f465319.el8.x86_64 141/162 2024-03-14T09:44:11.575 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-03-14T09:44:11.776 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flex-2.6.1-9.el8.x86_64 143/162 2024-03-14T09:44:11.895 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 143/162 2024-03-14T09:44:12.135 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : bison-3.0.4-10.el8.x86_64 144/162 2024-03-14T09:44:12.166 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 144/162 2024-03-14T09:44:12.335 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-14T09:44:12.360 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-14T09:44:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 146/162 2024-03-14T09:44:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-03-14T09:44:12.888 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-03-14T09:44:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-03-14T09:44:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-14T09:44:13.356 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-14T09:44:13.356 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-14T09:44:13.539 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-selinux-2:18.2.2-825.g2f465319.el8.x86_64 107/162 2024-03-14T09:44:13.539 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /sys 2024-03-14T09:44:13.539 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /proc 2024-03-14T09:44:13.539 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /dev 2024-03-14T09:44:13.539 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /run 2024-03-14T09:44:13.539 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /mnt 2024-03-14T09:44:13.539 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /var/tmp 2024-03-14T09:44:13.539 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /home 2024-03-14T09:44:13.539 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /tmp 2024-03-14T09:44:13.539 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /dev 2024-03-14T09:44:13.539 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-14T09:44:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-14T09:44:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-14T09:44:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-14T09:44:13.956 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-volume-2:18.2.2-825.g2f465319.el8.noarch 108/162 2024-03-14T09:44:14.096 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-03-14T09:44:14.307 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-03-14T09:44:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-03-14T09:44:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-14T09:44:14.763 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-14T09:44:14.763 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-03-14T09:44:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-03-14T09:44:15.021 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-14T09:44:15.288 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-14T09:44:15.288 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-14T09:44:15.529 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-volume-2:18.2.2-825.g2f465319.el8.noarch 108/162 2024-03-14T09:44:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-14T09:44:15.546 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-14T09:44:15.689 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-osd-2:18.2.2-825.g2f465319.el8.x86_64 109/162 2024-03-14T09:44:15.788 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-14T09:44:15.906 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-osd-2:18.2.2-825.g2f465319.el8.x86_64 109/162 2024-03-14T09:44:15.963 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-03-14T09:44:15.963 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-14T09:44:16.114 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mds-2:18.2.2-825.g2f465319.el8.x86_64 110/162 2024-03-14T09:44:16.525 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-14T09:44:16.525 INFO:teuthology.orchestra.run.smithi078.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-14T09:44:16.525 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:44:16.570 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mds-2:18.2.2-825.g2f465319.el8.x86_64 110/162 2024-03-14T09:44:16.731 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mon-2:18.2.2-825.g2f465319.el8.x86_64 111/162 2024-03-14T09:44:16.748 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mon-2:18.2.2-825.g2f465319.el8.x86_64 111/162 2024-03-14T09:44:17.204 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-2:18.2.2-825.g2f465319.el8.x86_64 1/162 2024-03-14T09:44:17.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-base-2:18.2.2-825.g2f465319.el8.x86_64 2/162 2024-03-14T09:44:17.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 3/162 2024-03-14T09:44:17.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-fuse-2:18.2.2-825.g2f465319.el8.x86_64 4/162 2024-03-14T09:44:17.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-825.g2f4653 5/162 2024-03-14T09:44:17.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mds-2:18.2.2-825.g2f465319.el8.x86_64 6/162 2024-03-14T09:44:17.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-2:18.2.2-825.g2f465319.el8.x86_64 7/162 2024-03-14T09:44:17.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mon-2:18.2.2-825.g2f465319.el8.x86_64 8/162 2024-03-14T09:44:17.205 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-osd-2:18.2.2-825.g2f465319.el8.x86_64 9/162 2024-03-14T09:44:17.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-radosgw-2:18.2.2-825.g2f465319.el8.x86_64 10/162 2024-03-14T09:44:17.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-selinux-2:18.2.2-825.g2f465319.el8.x86_64 11/162 2024-03-14T09:44:17.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-test-2:18.2.2-825.g2f465319.el8.x86_64 12/162 2024-03-14T09:44:17.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-mirror-2:18.2.2-825.g2f465319.el8.x86_64 13/162 2024-03-14T09:44:17.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs-devel-2:18.2.2-825.g2f465319.el8.x86_6 14/162 2024-03-14T09:44:17.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs2-2:18.2.2-825.g2f465319.el8.x86_64 15/162 2024-03-14T09:44:17.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephsqlite-2:18.2.2-825.g2f465319.el8.x86_64 16/162 2024-03-14T09:44:17.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados-devel-2:18.2.2-825.g2f465319.el8.x86_64 17/162 2024-03-14T09:44:17.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libradosstriper1-2:18.2.2-825.g2f465319.el8.x86_ 18/162 2024-03-14T09:44:17.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librgw2-2:18.2.2-825.g2f465319.el8.x86_64 19/162 2024-03-14T09:44:17.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-argparse-2:18.2.2-825.g2f465319.el8 20/162 2024-03-14T09:44:17.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-common-2:18.2.2-825.g2f465319.el8.x 21/162 2024-03-14T09:44:17.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cephfs-2:18.2.2-825.g2f465319.el8.x86_64 22/162 2024-03-14T09:44:17.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rados-2:18.2.2-825.g2f465319.el8.x86_64 23/162 2024-03-14T09:44:17.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rbd-2:18.2.2-825.g2f465319.el8.x86_64 24/162 2024-03-14T09:44:17.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rgw-2:18.2.2-825.g2f465319.el8.x86_64 25/162 2024-03-14T09:44:17.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-fuse-2:18.2.2-825.g2f465319.el8.x86_64 26/162 2024-03-14T09:44:17.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-mirror-2:18.2.2-825.g2f465319.el8.x86_64 27/162 2024-03-14T09:44:17.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-nbd-2:18.2.2-825.g2f465319.el8.x86_64 28/162 2024-03-14T09:44:17.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-825.g2f465319.e 29/162 2024-03-14T09:44:17.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-825.g2f465319.el8.noar 30/162 2024-03-14T09:44:17.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-825.g2f465319.el8.no 31/162 2024-03-14T09:44:17.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-825.g2f46 32/162 2024-03-14T09:44:17.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-825.g2f465319.el8 33/162 2024-03-14T09:44:17.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-rook-2:18.2.2-825.g2f465319.el8.noarch 34/162 2024-03-14T09:44:17.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-825.g2f465319.el 35/162 2024-03-14T09:44:17.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-volume-2:18.2.2-825.g2f465319.el8.noarch 36/162 2024-03-14T09:44:17.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephadm-2:18.2.2-825.g2f465319.el8.noarch 37/162 2024-03-14T09:44:17.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-top-2:18.2.2-825.g2f465319.el8.noarch 38/162 2024-03-14T09:44:17.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-03-14T09:44:17.222 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-03-14T09:44:17.222 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-03-14T09:44:17.222 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-03-14T09:44:17.222 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-03-14T09:44:17.222 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-03-14T09:44:17.222 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-03-14T09:44:17.222 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-03-14T09:44:17.222 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-03-14T09:44:17.222 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-03-14T09:44:17.222 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-03-14T09:44:17.222 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-03-14T09:44:17.222 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-03-14T09:44:17.223 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-03-14T09:44:17.223 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-03-14T09:44:17.223 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-03-14T09:44:17.223 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-03-14T09:44:17.223 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-03-14T09:44:17.223 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-03-14T09:44:17.223 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-03-14T09:44:17.223 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-03-14T09:44:17.223 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-03-14T09:44:17.223 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-03-14T09:44:17.223 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-03-14T09:44:17.224 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-03-14T09:44:17.224 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-03-14T09:44:17.224 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-03-14T09:44:17.224 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-03-14T09:44:17.224 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-03-14T09:44:17.224 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-03-14T09:44:17.224 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-03-14T09:44:17.224 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-03-14T09:44:17.224 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-03-14T09:44:17.224 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 72/162 2024-03-14T09:44:17.224 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/162 2024-03-14T09:44:17.224 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/162 2024-03-14T09:44:17.224 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 75/162 2024-03-14T09:44:17.225 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 76/162 2024-03-14T09:44:17.225 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 77/162 2024-03-14T09:44:17.225 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 78/162 2024-03-14T09:44:17.225 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 79/162 2024-03-14T09:44:17.225 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/162 2024-03-14T09:44:17.225 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 81/162 2024-03-14T09:44:17.225 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 82/162 2024-03-14T09:44:17.225 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 83/162 2024-03-14T09:44:17.225 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 84/162 2024-03-14T09:44:17.225 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 85/162 2024-03-14T09:44:17.225 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 86/162 2024-03-14T09:44:17.226 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 87/162 2024-03-14T09:44:17.226 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 88/162 2024-03-14T09:44:17.226 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 89/162 2024-03-14T09:44:17.226 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 90/162 2024-03-14T09:44:17.226 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 91/162 2024-03-14T09:44:17.226 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 92/162 2024-03-14T09:44:17.226 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 93/162 2024-03-14T09:44:17.226 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 94/162 2024-03-14T09:44:17.226 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 95/162 2024-03-14T09:44:17.226 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 96/162 2024-03-14T09:44:17.226 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 97/162 2024-03-14T09:44:17.226 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 98/162 2024-03-14T09:44:17.226 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 99/162 2024-03-14T09:44:17.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 100/162 2024-03-14T09:44:17.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 101/162 2024-03-14T09:44:17.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 102/162 2024-03-14T09:44:17.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 103/162 2024-03-14T09:44:17.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 104/162 2024-03-14T09:44:17.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 105/162 2024-03-14T09:44:17.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : bison-3.0.4-10.el8.x86_64 106/162 2024-03-14T09:44:17.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flex-2.6.1-9.el8.x86_64 107/162 2024-03-14T09:44:17.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 108/162 2024-03-14T09:44:17.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 109/162 2024-03-14T09:44:17.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 110/162 2024-03-14T09:44:17.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 111/162 2024-03-14T09:44:17.227 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 112/162 2024-03-14T09:44:17.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 113/162 2024-03-14T09:44:17.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 114/162 2024-03-14T09:44:17.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 115/162 2024-03-14T09:44:17.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 116/162 2024-03-14T09:44:17.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 117/162 2024-03-14T09:44:17.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 118/162 2024-03-14T09:44:17.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 119/162 2024-03-14T09:44:17.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 120/162 2024-03-14T09:44:17.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 121/162 2024-03-14T09:44:17.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : automake-1.16.1-8.el8.noarch 122/162 2024-03-14T09:44:17.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libpq-13.11-1.el8.x86_64 123/162 2024-03-14T09:44:17.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : jq-1.6-7.el8.x86_64 124/162 2024-03-14T09:44:17.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 125/162 2024-03-14T09:44:17.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 126/162 2024-03-14T09:44:17.229 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 127/162 2024-03-14T09:44:17.229 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 128/162 2024-03-14T09:44:17.229 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 129/162 2024-03-14T09:44:17.229 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 130/162 2024-03-14T09:44:17.229 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:18.2.2-825.g2f465319.el8.x86_64 131/162 2024-03-14T09:44:17.229 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-03-14T09:44:17.229 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:18.2.2-825.g2f465319.el8.x86_64 133/162 2024-03-14T09:44:17.229 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-03-14T09:44:17.229 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-03-14T09:44:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-03-14T09:44:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-03-14T09:44:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-03-14T09:44:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-03-14T09:44:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-03-14T09:44:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-03-14T09:44:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-03-14T09:44:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-03-14T09:44:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-03-14T09:44:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-03-14T09:44:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-03-14T09:44:17.230 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-03-14T09:44:17.231 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-03-14T09:44:17.231 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-03-14T09:44:17.231 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-03-14T09:44:17.231 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-03-14T09:44:17.231 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-03-14T09:44:17.231 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-03-14T09:44:17.231 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-03-14T09:44:17.231 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-03-14T09:44:17.231 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-03-14T09:44:17.231 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-03-14T09:44:17.231 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-03-14T09:44:17.231 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-03-14T09:44:17.231 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-03-14T09:44:17.231 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-03-14T09:44:18.520 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:18.2.2-825.g2f465319.el8.noarch 112/162 2024-03-14T09:44:18.682 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephadm-2:18.2.2-825.g2f465319.el8.noarch 112/162 2024-03-14T09:44:18.853 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-825.g2f465319.el 113/162 2024-03-14T09:44:19.015 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-825.g2f465319.e 114/162 2024-03-14T09:44:19.305 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-825.g2f465319.el8.noar 115/162 2024-03-14T09:44:19.525 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-825.g2f465319.el8.noar 115/162 2024-03-14T09:44:19.846 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-825.g2f465319.el8.no 116/162 2024-03-14T09:44:20.818 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-825.g2f465319.el8.no 116/162 2024-03-14T09:44:20.996 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-825.g2f46 117/162 2024-03-14T09:44:21.089 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-825.g2f46 117/162 2024-03-14T09:44:21.517 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-825.g2f465319.el8 118/162 2024-03-14T09:44:21.705 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-2:18.2.2-825.g2f465319.el8.x86_64 119/162 2024-03-14T09:44:21.742 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-2:18.2.2-825.g2f465319.el8.x86_64 119/162 2024-03-14T09:44:21.929 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-rook-2:18.2.2-825.g2f465319.el8.noarch 120/162 2024-03-14T09:44:21.941 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-825.g2f465319.el8.noarch 120/162 2024-03-14T09:44:23.243 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-2:18.2.2-825.g2f465319.el8.x86_64 121/162 2024-03-14T09:44:23.496 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-radosgw-2:18.2.2-825.g2f465319.el8.x86_64 122/162 2024-03-14T09:44:23.530 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-825.g2f465319.el8.x86_64 122/162 2024-03-14T09:44:23.738 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-825.g2f4653 123/162 2024-03-14T09:44:23.781 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-825.g2f4653 123/162 2024-03-14T09:44:23.955 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephfs-mirror-2:18.2.2-825.g2f465319.el8.x86_64 124/162 2024-03-14T09:44:24.303 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-825.g2f465319.el8.x86_64 124/162 2024-03-14T09:44:24.496 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-mirror-2:18.2.2-825.g2f465319.el8.x86_64 125/162 2024-03-14T09:44:25.950 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-03-14T09:44:25.951 INFO:teuthology.orchestra.run.smithi078.stdout:Installed products updated. 2024-03-14T09:44:26.627 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:44:26.628 INFO:teuthology.orchestra.run.smithi078.stdout:Upgraded: 2024-03-14T09:44:26.628 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-03-14T09:44:26.628 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-03-14T09:44:26.628 INFO:teuthology.orchestra.run.smithi078.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-03-14T09:44:26.628 INFO:teuthology.orchestra.run.smithi078.stdout: libblkid-2.32.1-43.el8.x86_64 2024-03-14T09:44:26.628 INFO:teuthology.orchestra.run.smithi078.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-03-14T09:44:26.628 INFO:teuthology.orchestra.run.smithi078.stdout: libmount-2.32.1-43.el8.x86_64 2024-03-14T09:44:26.628 INFO:teuthology.orchestra.run.smithi078.stdout: librados2-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.628 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.628 INFO:teuthology.orchestra.run.smithi078.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-03-14T09:44:26.628 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-2.32.1-43.el8.x86_64 2024-03-14T09:44:26.628 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-03-14T09:44:26.628 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-03-14T09:44:26.629 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-2.32.1-43.el8.x86_64 2024-03-14T09:44:26.629 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-03-14T09:44:26.629 INFO:teuthology.orchestra.run.smithi078.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-03-14T09:44:26.629 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-03-14T09:44:26.629 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-03-14T09:44:26.629 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-03-14T09:44:26.629 INFO:teuthology.orchestra.run.smithi078.stdout: automake-1.16.1-8.el8.noarch 2024-03-14T09:44:26.629 INFO:teuthology.orchestra.run.smithi078.stdout: bison-3.0.4-10.el8.x86_64 2024-03-14T09:44:26.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:26.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.629 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.630 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.630 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:26.630 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:26.630 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:26.630 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:26.630 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:26.630 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.630 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.630 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:26.630 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.630 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.630 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.631 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:26.631 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:26.631 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.631 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:26.631 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-03-14T09:44:26.631 INFO:teuthology.orchestra.run.smithi078.stdout: flex-2.6.1-9.el8.x86_64 2024-03-14T09:44:26.631 INFO:teuthology.orchestra.run.smithi078.stdout: fuse-2.9.7-17.el8.x86_64 2024-03-14T09:44:26.631 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-14T09:44:26.631 INFO:teuthology.orchestra.run.smithi078.stdout: jq-1.6-7.el8.x86_64 2024-03-14T09:44:26.631 INFO:teuthology.orchestra.run.smithi078.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-03-14T09:44:26.631 INFO:teuthology.orchestra.run.smithi078.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-03-14T09:44:26.631 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-03-14T09:44:26.631 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.631 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.631 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.632 INFO:teuthology.orchestra.run.smithi078.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-03-14T09:44:26.632 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-03-14T09:44:26.632 INFO:teuthology.orchestra.run.smithi078.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-14T09:44:26.632 INFO:teuthology.orchestra.run.smithi078.stdout: libpq-13.11-1.el8.x86_64 2024-03-14T09:44:26.632 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-03-14T09:44:26.632 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-03-14T09:44:26.632 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.632 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.632 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-03-14T09:44:26.632 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.632 INFO:teuthology.orchestra.run.smithi078.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-03-14T09:44:26.632 INFO:teuthology.orchestra.run.smithi078.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-03-14T09:44:26.632 INFO:teuthology.orchestra.run.smithi078.stdout: libtool-2.4.6-25.el8.x86_64 2024-03-14T09:44:26.632 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-14T09:44:26.632 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-03-14T09:44:26.632 INFO:teuthology.orchestra.run.smithi078.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-03-14T09:44:26.633 INFO:teuthology.orchestra.run.smithi078.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-03-14T09:44:26.633 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-14T09:44:26.633 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-03-14T09:44:26.633 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-0.3.15-6.el8.x86_64 2024-03-14T09:44:26.633 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-03-14T09:44:26.633 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-03-14T09:44:26.633 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-03-14T09:44:26.633 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-03-14T09:44:26.633 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-03-14T09:44:26.633 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-14T09:44:26.633 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-14T09:44:26.633 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-14T09:44:26.633 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-14T09:44:26.633 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-14T09:44:26.633 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-14T09:44:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-14T09:44:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-14T09:44:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-14T09:44:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-14T09:44:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-14T09:44:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-14T09:44:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-03-14T09:44:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-14T09:44:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-14T09:44:26.634 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-14T09:44:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-03-14T09:44:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-14T09:44:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-14T09:44:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-14T09:44:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-14T09:44:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-14T09:44:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-14T09:44:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-14T09:44:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-14T09:44:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-14T09:44:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-14T09:44:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-14T09:44:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-14T09:44:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-packaging-16.8-9.el8.noarch 2024-03-14T09:44:26.635 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-14T09:44:26.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend-2.6-1.el8.noarch 2024-03-14T09:44:26.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-14T09:44:26.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-14T09:44:26.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-14T09:44:26.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-14T09:44:26.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-14T09:44:26.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-14T09:44:26.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-14T09:44:26.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-14T09:44:26.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-14T09:44:26.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-14T09:44:26.636 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-03-14T09:44:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-14T09:44:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-14T09:44:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-14T09:44:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-14T09:44:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-14T09:44:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-14T09:44:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-14T09:44:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-14T09:44:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-14T09:44:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-14T09:44:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-14T09:44:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-14T09:44:26.637 INFO:teuthology.orchestra.run.smithi078.stdout: uuid-1.6.2-43.el8.x86_64 2024-03-14T09:44:26.638 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-03-14T09:44:26.638 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-14T09:44:26.638 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-03-14T09:44:26.638 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:44:26.638 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-03-14T09:44:27.052 DEBUG:teuthology.parallel:result is None 2024-03-14T09:44:28.927 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-mirror-2:18.2.2-825.g2f465319.el8.x86_64 125/162 2024-03-14T09:44:29.939 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-test-2:18.2.2-825.g2f465319.el8.x86_64 126/162 2024-03-14T09:44:30.266 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 127/162 2024-03-14T09:44:30.578 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 128/162 2024-03-14T09:44:31.028 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-fuse-2:18.2.2-825.g2f465319.el8.x86_64 129/162 2024-03-14T09:44:31.430 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 130/162 2024-03-14T09:44:31.677 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libtool-2.4.6-25.el8.x86_64 131/162 2024-03-14T09:44:31.691 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 131/162 2024-03-14T09:44:31.939 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-03-14T09:44:32.174 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-03-14T09:44:32.387 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs-devel-2:18.2.2-825.g2f465319.el8.x86_6 134/162 2024-03-14T09:44:34.758 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-14T09:44:35.070 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-14T09:44:35.108 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-14T09:44:35.589 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-03-14T09:44:36.033 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-03-14T09:44:36.306 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-03-14T09:44:36.536 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephfs-top-2:18.2.2-825.g2f465319.el8.noarch 139/162 2024-03-14T09:44:36.802 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-fuse-2:18.2.2-825.g2f465319.el8.x86_64 140/162 2024-03-14T09:44:37.138 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-nbd-2:18.2.2-825.g2f465319.el8.x86_64 141/162 2024-03-14T09:44:37.439 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-03-14T09:44:37.685 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flex-2.6.1-9.el8.x86_64 143/162 2024-03-14T09:44:37.766 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 143/162 2024-03-14T09:44:37.986 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : bison-3.0.4-10.el8.x86_64 144/162 2024-03-14T09:44:38.000 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 144/162 2024-03-14T09:44:38.210 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-14T09:44:38.220 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-14T09:44:38.396 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 146/162 2024-03-14T09:44:38.623 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-03-14T09:44:38.846 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-03-14T09:44:39.023 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-03-14T09:44:39.023 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-14T09:44:39.347 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-14T09:44:39.347 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-14T09:44:39.606 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-14T09:44:39.606 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-14T09:44:39.892 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-14T09:44:40.145 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-03-14T09:44:40.320 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-03-14T09:44:40.604 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-03-14T09:44:40.604 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-14T09:44:40.871 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-14T09:44:40.871 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-03-14T09:44:41.112 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-03-14T09:44:41.113 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-14T09:44:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-14T09:44:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-14T09:44:41.637 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-14T09:44:41.637 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-14T09:44:41.912 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-14T09:44:42.088 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-03-14T09:44:42.088 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-14T09:44:43.024 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-14T09:44:43.024 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-2:18.2.2-825.g2f465319.el8.x86_64 1/162 2024-03-14T09:44:43.024 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-base-2:18.2.2-825.g2f465319.el8.x86_64 2/162 2024-03-14T09:44:43.024 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 3/162 2024-03-14T09:44:43.024 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-fuse-2:18.2.2-825.g2f465319.el8.x86_64 4/162 2024-03-14T09:44:43.025 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-825.g2f4653 5/162 2024-03-14T09:44:43.025 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mds-2:18.2.2-825.g2f465319.el8.x86_64 6/162 2024-03-14T09:44:43.025 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-2:18.2.2-825.g2f465319.el8.x86_64 7/162 2024-03-14T09:44:43.025 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mon-2:18.2.2-825.g2f465319.el8.x86_64 8/162 2024-03-14T09:44:43.025 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-osd-2:18.2.2-825.g2f465319.el8.x86_64 9/162 2024-03-14T09:44:43.025 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-radosgw-2:18.2.2-825.g2f465319.el8.x86_64 10/162 2024-03-14T09:44:43.025 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-selinux-2:18.2.2-825.g2f465319.el8.x86_64 11/162 2024-03-14T09:44:43.025 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-test-2:18.2.2-825.g2f465319.el8.x86_64 12/162 2024-03-14T09:44:43.025 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephfs-mirror-2:18.2.2-825.g2f465319.el8.x86_64 13/162 2024-03-14T09:44:43.025 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs-devel-2:18.2.2-825.g2f465319.el8.x86_6 14/162 2024-03-14T09:44:43.025 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs2-2:18.2.2-825.g2f465319.el8.x86_64 15/162 2024-03-14T09:44:43.025 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephsqlite-2:18.2.2-825.g2f465319.el8.x86_64 16/162 2024-03-14T09:44:43.025 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados-devel-2:18.2.2-825.g2f465319.el8.x86_64 17/162 2024-03-14T09:44:43.025 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libradosstriper1-2:18.2.2-825.g2f465319.el8.x86_ 18/162 2024-03-14T09:44:43.026 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librgw2-2:18.2.2-825.g2f465319.el8.x86_64 19/162 2024-03-14T09:44:43.026 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-argparse-2:18.2.2-825.g2f465319.el8 20/162 2024-03-14T09:44:43.026 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-common-2:18.2.2-825.g2f465319.el8.x 21/162 2024-03-14T09:44:43.026 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cephfs-2:18.2.2-825.g2f465319.el8.x86_64 22/162 2024-03-14T09:44:43.026 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rados-2:18.2.2-825.g2f465319.el8.x86_64 23/162 2024-03-14T09:44:43.026 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rbd-2:18.2.2-825.g2f465319.el8.x86_64 24/162 2024-03-14T09:44:43.026 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rgw-2:18.2.2-825.g2f465319.el8.x86_64 25/162 2024-03-14T09:44:43.026 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-fuse-2:18.2.2-825.g2f465319.el8.x86_64 26/162 2024-03-14T09:44:43.026 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-mirror-2:18.2.2-825.g2f465319.el8.x86_64 27/162 2024-03-14T09:44:43.026 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-nbd-2:18.2.2-825.g2f465319.el8.x86_64 28/162 2024-03-14T09:44:43.026 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-825.g2f465319.e 29/162 2024-03-14T09:44:43.026 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-825.g2f465319.el8.noar 30/162 2024-03-14T09:44:43.039 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-825.g2f465319.el8.no 31/162 2024-03-14T09:44:43.039 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-825.g2f46 32/162 2024-03-14T09:44:43.039 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-825.g2f465319.el8 33/162 2024-03-14T09:44:43.039 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-rook-2:18.2.2-825.g2f465319.el8.noarch 34/162 2024-03-14T09:44:43.039 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-825.g2f465319.el 35/162 2024-03-14T09:44:43.039 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-volume-2:18.2.2-825.g2f465319.el8.noarch 36/162 2024-03-14T09:44:43.039 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephadm-2:18.2.2-825.g2f465319.el8.noarch 37/162 2024-03-14T09:44:43.039 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephfs-top-2:18.2.2-825.g2f465319.el8.noarch 38/162 2024-03-14T09:44:43.039 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-03-14T09:44:43.039 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-03-14T09:44:43.040 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-03-14T09:44:43.040 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-03-14T09:44:43.040 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-03-14T09:44:43.040 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-03-14T09:44:43.040 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-03-14T09:44:43.040 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-03-14T09:44:43.040 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-03-14T09:44:43.040 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-03-14T09:44:43.040 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-03-14T09:44:43.040 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-03-14T09:44:43.040 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-03-14T09:44:43.041 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-03-14T09:44:43.041 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-03-14T09:44:43.041 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-03-14T09:44:43.041 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-03-14T09:44:43.041 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-03-14T09:44:43.041 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-03-14T09:44:43.041 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-03-14T09:44:43.041 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-03-14T09:44:43.041 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-03-14T09:44:43.041 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-03-14T09:44:43.041 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-03-14T09:44:43.042 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-03-14T09:44:43.042 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-03-14T09:44:43.042 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-03-14T09:44:43.042 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-03-14T09:44:43.042 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-03-14T09:44:43.042 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-03-14T09:44:43.042 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-03-14T09:44:43.042 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-03-14T09:44:43.042 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-03-14T09:44:43.042 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 72/162 2024-03-14T09:44:43.042 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/162 2024-03-14T09:44:43.042 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/162 2024-03-14T09:44:43.042 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 75/162 2024-03-14T09:44:43.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 76/162 2024-03-14T09:44:43.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 77/162 2024-03-14T09:44:43.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 78/162 2024-03-14T09:44:43.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 79/162 2024-03-14T09:44:43.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/162 2024-03-14T09:44:43.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 81/162 2024-03-14T09:44:43.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 82/162 2024-03-14T09:44:43.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 83/162 2024-03-14T09:44:43.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 84/162 2024-03-14T09:44:43.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 85/162 2024-03-14T09:44:43.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 86/162 2024-03-14T09:44:43.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 87/162 2024-03-14T09:44:43.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 88/162 2024-03-14T09:44:43.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 89/162 2024-03-14T09:44:43.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 90/162 2024-03-14T09:44:43.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 91/162 2024-03-14T09:44:43.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 92/162 2024-03-14T09:44:43.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 93/162 2024-03-14T09:44:43.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 94/162 2024-03-14T09:44:43.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 95/162 2024-03-14T09:44:43.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 96/162 2024-03-14T09:44:43.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 97/162 2024-03-14T09:44:43.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 98/162 2024-03-14T09:44:43.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 99/162 2024-03-14T09:44:43.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 100/162 2024-03-14T09:44:43.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 101/162 2024-03-14T09:44:43.045 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 102/162 2024-03-14T09:44:43.045 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 103/162 2024-03-14T09:44:43.045 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 104/162 2024-03-14T09:44:43.045 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 105/162 2024-03-14T09:44:43.045 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : bison-3.0.4-10.el8.x86_64 106/162 2024-03-14T09:44:43.045 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flex-2.6.1-9.el8.x86_64 107/162 2024-03-14T09:44:43.045 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 108/162 2024-03-14T09:44:43.045 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 109/162 2024-03-14T09:44:43.045 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 110/162 2024-03-14T09:44:43.045 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 111/162 2024-03-14T09:44:43.045 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 112/162 2024-03-14T09:44:43.045 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 113/162 2024-03-14T09:44:43.045 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 114/162 2024-03-14T09:44:43.046 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 115/162 2024-03-14T09:44:43.046 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 116/162 2024-03-14T09:44:43.046 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 117/162 2024-03-14T09:44:43.046 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 118/162 2024-03-14T09:44:43.046 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 119/162 2024-03-14T09:44:43.046 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 120/162 2024-03-14T09:44:43.046 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 121/162 2024-03-14T09:44:43.046 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : automake-1.16.1-8.el8.noarch 122/162 2024-03-14T09:44:43.046 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libpq-13.11-1.el8.x86_64 123/162 2024-03-14T09:44:43.046 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : jq-1.6-7.el8.x86_64 124/162 2024-03-14T09:44:43.046 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 125/162 2024-03-14T09:44:43.046 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 126/162 2024-03-14T09:44:43.046 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 127/162 2024-03-14T09:44:43.046 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 128/162 2024-03-14T09:44:43.047 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 129/162 2024-03-14T09:44:43.047 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 130/162 2024-03-14T09:44:43.047 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:18.2.2-825.g2f465319.el8.x86_64 131/162 2024-03-14T09:44:43.047 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-03-14T09:44:43.047 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:18.2.2-825.g2f465319.el8.x86_64 133/162 2024-03-14T09:44:43.047 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-03-14T09:44:43.047 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-03-14T09:44:43.047 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-03-14T09:44:43.047 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-03-14T09:44:43.047 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-03-14T09:44:43.047 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-03-14T09:44:43.047 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-03-14T09:44:43.047 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-03-14T09:44:43.047 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-03-14T09:44:43.047 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-03-14T09:44:43.048 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-03-14T09:44:43.048 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-03-14T09:44:43.048 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-03-14T09:44:43.048 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-03-14T09:44:43.048 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-03-14T09:44:43.048 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-03-14T09:44:43.048 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-03-14T09:44:43.048 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-03-14T09:44:43.048 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-03-14T09:44:43.048 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-03-14T09:44:43.048 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-03-14T09:44:43.048 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-03-14T09:44:43.048 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-03-14T09:44:43.048 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-03-14T09:44:43.048 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-03-14T09:44:43.049 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-03-14T09:44:43.049 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-03-14T09:44:43.049 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-03-14T09:44:52.510 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-03-14T09:44:52.511 INFO:teuthology.orchestra.run.smithi152.stdout:Installed products updated. 2024-03-14T09:44:53.108 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-14T09:44:53.108 INFO:teuthology.orchestra.run.smithi152.stdout:Upgraded: 2024-03-14T09:44:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-03-14T09:44:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-03-14T09:44:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-03-14T09:44:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: libblkid-2.32.1-43.el8.x86_64 2024-03-14T09:44:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-03-14T09:44:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: libmount-2.32.1-43.el8.x86_64 2024-03-14T09:44:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: librados2-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-03-14T09:44:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: libuuid-2.32.1-43.el8.x86_64 2024-03-14T09:44:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-03-14T09:44:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-03-14T09:44:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: util-linux-2.32.1-43.el8.x86_64 2024-03-14T09:44:53.109 INFO:teuthology.orchestra.run.smithi152.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-03-14T09:44:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-03-14T09:44:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-03-14T09:44:53.110 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-03-14T09:44:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-03-14T09:44:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: automake-1.16.1-8.el8.noarch 2024-03-14T09:44:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: bison-3.0.4-10.el8.x86_64 2024-03-14T09:44:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.110 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:53.111 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-mirror-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: cephfs-top-2:18.2.2-825.g2f465319.el8.noarch 2024-03-14T09:44:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-03-14T09:44:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: flex-2.6.1-9.el8.x86_64 2024-03-14T09:44:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: fuse-2.9.7-17.el8.x86_64 2024-03-14T09:44:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-14T09:44:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: jq-1.6-7.el8.x86_64 2024-03-14T09:44:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-03-14T09:44:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-03-14T09:44:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-03-14T09:44:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-03-14T09:44:53.112 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-03-14T09:44:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-14T09:44:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: libpq-13.11-1.el8.x86_64 2024-03-14T09:44:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-03-14T09:44:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-03-14T09:44:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-03-14T09:44:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-03-14T09:44:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-03-14T09:44:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: libtool-2.4.6-25.el8.x86_64 2024-03-14T09:44:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-14T09:44:53.113 INFO:teuthology.orchestra.run.smithi152.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-03-14T09:44:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-03-14T09:44:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-03-14T09:44:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-14T09:44:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-03-14T09:44:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-0.3.15-6.el8.x86_64 2024-03-14T09:44:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-03-14T09:44:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-03-14T09:44:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-03-14T09:44:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-03-14T09:44:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-03-14T09:44:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-14T09:44:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-14T09:44:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-14T09:44:53.114 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-14T09:44:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-14T09:44:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-14T09:44:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-14T09:44:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-14T09:44:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-14T09:44:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-14T09:44:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-14T09:44:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-14T09:44:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-03-14T09:44:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-14T09:44:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-14T09:44:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-14T09:44:53.115 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-03-14T09:44:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-14T09:44:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-14T09:44:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-14T09:44:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-14T09:44:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-14T09:44:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-14T09:44:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-14T09:44:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-14T09:44:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-14T09:44:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-14T09:44:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-14T09:44:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-14T09:44:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging-16.8-9.el8.noarch 2024-03-14T09:44:53.116 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-14T09:44:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend-2.6-1.el8.noarch 2024-03-14T09:44:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-14T09:44:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-14T09:44:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-14T09:44:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-14T09:44:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-14T09:44:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-14T09:44:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-14T09:44:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-14T09:44:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-14T09:44:53.117 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-14T09:44:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-03-14T09:44:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-14T09:44:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-14T09:44:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-14T09:44:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-14T09:44:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-14T09:44:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-14T09:44:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-14T09:44:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-14T09:44:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-14T09:44:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-14T09:44:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.118 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd-2:18.2.2-825.g2f465319.el8.x86_64 2024-03-14T09:44:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-14T09:44:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-14T09:44:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: uuid-1.6.2-43.el8.x86_64 2024-03-14T09:44:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-03-14T09:44:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-14T09:44:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-03-14T09:44:53.119 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-14T09:44:53.119 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-03-14T09:44:53.321 DEBUG:teuthology.parallel:result is None 2024-03-14T09:44:53.322 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T09:44:53.474 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-14T09:44:53.534 INFO:teuthology.orchestra.run.smithi078.stdout:18.2.2-825.g2f465319.el8 2024-03-14T09:44:53.534 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-825.g2f465319.el8 2024-03-14T09:44:53.534 INFO:teuthology.task.install:The correct ceph version 18.2.2-825.g2f465319 is installed. 2024-03-14T09:44:53.536 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T09:44:53.672 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-14T09:44:53.701 INFO:teuthology.orchestra.run.smithi132.stdout:18.2.2-825.g2f465319.el8 2024-03-14T09:44:53.701 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-825.g2f465319.el8 2024-03-14T09:44:53.701 INFO:teuthology.task.install:The correct ceph version 18.2.2-825.g2f465319 is installed. 2024-03-14T09:44:53.703 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T09:44:53.919 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-14T09:44:53.944 INFO:teuthology.orchestra.run.smithi152.stdout:18.2.2-825.g2f465319.el8 2024-03-14T09:44:53.944 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-825.g2f465319.el8 2024-03-14T09:44:53.944 INFO:teuthology.task.install:The correct ceph version 18.2.2-825.g2f465319 is installed. 2024-03-14T09:44:53.946 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-14T09:44:53.946 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:44:53.946 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-14T09:44:54.041 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-14T09:44:54.041 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-14T09:44:54.090 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-14T09:44:54.090 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-14T09:44:54.135 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-14T09:44:54.136 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:44:54.136 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2024-03-14T09:44:54.202 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-14T09:44:54.306 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-14T09:44:54.307 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/daemon-helper 2024-03-14T09:44:54.339 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-14T09:44:54.409 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-14T09:44:54.409 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/daemon-helper 2024-03-14T09:44:54.440 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-14T09:44:54.509 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-14T09:44:54.510 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:44:54.510 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-14T09:44:54.574 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-14T09:44:54.679 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-14T09:44:54.679 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-14T09:44:54.709 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-14T09:44:54.778 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-14T09:44:54.779 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-14T09:44:54.811 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-14T09:44:54.881 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-14T09:44:54.882 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:44:54.882 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/stdin-killer 2024-03-14T09:44:54.945 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-14T09:44:55.049 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-14T09:44:55.049 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/stdin-killer 2024-03-14T09:44:55.081 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-14T09:44:55.153 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-14T09:44:55.153 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/stdin-killer 2024-03-14T09:44:55.185 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-14T09:44:55.256 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-14T09:44:55.326 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '2f465319e3c83d48f999325459bcdfd0346449c0', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-14T09:44:55.326 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T09:44:55.327 INFO:tasks.cephadm:Cluster fsid is 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:44:55.327 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-14T09:44:55.327 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.78', 'mon.b': '172.21.15.132', 'mon.c': '172.21.15.152'} 2024-03-14T09:44:55.327 INFO:tasks.cephadm:First mon is mon.a on smithi078 2024-03-14T09:44:55.327 INFO:tasks.cephadm:First mgr is x 2024-03-14T09:44:55.327 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-14T09:44:55.327 DEBUG:teuthology.orchestra.run.smithi078:> sudo hostname $(hostname -s) 2024-03-14T09:44:55.393 DEBUG:teuthology.orchestra.run.smithi132:> sudo hostname $(hostname -s) 2024-03-14T09:44:55.426 DEBUG:teuthology.orchestra.run.smithi152:> sudo hostname $(hostname -s) 2024-03-14T09:44:55.459 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-14T09:44:55.459 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T09:44:55.606 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '2f465319e3c83d48f999325459bcdfd0346449c0', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/77592/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.2.2-825-g2f465319', 'node_name': '172.21.2.4+braggi04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.2.2-825.g2f465319'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-03-13 03:43:12.322089', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuri-testing-2024-03-12-1240-reef', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-14T09:44:55.745 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-yuri-testing-2024-03-12-1240-reef, sha1 2f465319e3c83d48f999325459bcdfd0346449c0 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T09:44:55.746 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/x86_64/flavors/default/cephadm 2024-03-14T09:44:55.746 DEBUG:teuthology.orchestra.run.smithi078:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-14T09:44:56.202 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-rw-r--. 1 ubuntu ubuntu 701072 Mar 14 09:44 /home/ubuntu/cephtest/cephadm 2024-03-14T09:44:56.202 DEBUG:teuthology.orchestra.run.smithi132:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-14T09:44:56.604 INFO:teuthology.orchestra.run.smithi132.stdout:-rw-rw-r--. 1 ubuntu ubuntu 701072 Mar 14 09:44 /home/ubuntu/cephtest/cephadm 2024-03-14T09:44:56.604 DEBUG:teuthology.orchestra.run.smithi152:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-yuri-testing-2024-03-12-1240-reef/2f465319e3c83d48f999325459bcdfd0346449c0/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-14T09:44:57.151 INFO:teuthology.orchestra.run.smithi152.stdout:-rw-rw-r--. 1 ubuntu ubuntu 701072 Mar 14 09:44 /home/ubuntu/cephtest/cephadm 2024-03-14T09:44:57.151 DEBUG:teuthology.orchestra.run.smithi078:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-14T09:44:57.202 DEBUG:teuthology.orchestra.run.smithi132:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-14T09:44:57.223 DEBUG:teuthology.orchestra.run.smithi152:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-14T09:44:57.259 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 on all hosts... 2024-03-14T09:44:57.259 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 pull 2024-03-14T09:44:57.263 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 pull 2024-03-14T09:44:57.265 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 pull 2024-03-14T09:44:57.980 INFO:teuthology.orchestra.run.smithi132.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0... 2024-03-14T09:44:58.346 INFO:teuthology.orchestra.run.smithi078.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0... 2024-03-14T09:45:00.315 INFO:teuthology.orchestra.run.smithi152.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0... 2024-03-14T09:45:31.791 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:45:31.791 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)", 2024-03-14T09:45:31.791 INFO:teuthology.orchestra.run.smithi078.stdout: "image_id": "a9e65ad305ae06daed88c6b29b1bd5b92a5f2a05f0abcf928fb8895930b5caf2", 2024-03-14T09:45:31.791 INFO:teuthology.orchestra.run.smithi078.stdout: "repo_digests": [ 2024-03-14T09:45:31.791 INFO:teuthology.orchestra.run.smithi078.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de" 2024-03-14T09:45:31.791 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-14T09:45:31.792 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:45:33.531 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-14T09:45:33.531 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_version": "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)", 2024-03-14T09:45:33.531 INFO:teuthology.orchestra.run.smithi132.stdout: "image_id": "a9e65ad305ae06daed88c6b29b1bd5b92a5f2a05f0abcf928fb8895930b5caf2", 2024-03-14T09:45:33.531 INFO:teuthology.orchestra.run.smithi132.stdout: "repo_digests": [ 2024-03-14T09:45:33.531 INFO:teuthology.orchestra.run.smithi132.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de" 2024-03-14T09:45:33.532 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-03-14T09:45:33.532 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-14T09:45:34.441 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-03-14T09:45:34.442 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_version": "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)", 2024-03-14T09:45:34.442 INFO:teuthology.orchestra.run.smithi152.stdout: "image_id": "a9e65ad305ae06daed88c6b29b1bd5b92a5f2a05f0abcf928fb8895930b5caf2", 2024-03-14T09:45:34.442 INFO:teuthology.orchestra.run.smithi152.stdout: "repo_digests": [ 2024-03-14T09:45:34.442 INFO:teuthology.orchestra.run.smithi152.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de" 2024-03-14T09:45:34.442 INFO:teuthology.orchestra.run.smithi152.stdout: ] 2024-03-14T09:45:34.442 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-03-14T09:45:34.461 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph 2024-03-14T09:45:34.537 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /etc/ceph 2024-03-14T09:45:34.575 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph 2024-03-14T09:45:34.606 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /etc/ceph 2024-03-14T09:45:34.673 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 777 /etc/ceph 2024-03-14T09:45:34.704 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /etc/ceph 2024-03-14T09:45:34.733 INFO:tasks.cephadm:Writing seed config... 2024-03-14T09:45:34.734 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-14T09:45:34.734 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-14T09:45:34.734 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-14T09:45:34.734 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-14T09:45:34.734 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-14T09:45:34.734 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-14T09:45:34.734 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-14T09:45:34.734 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-14T09:45:34.735 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-14T09:45:34.736 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-14T09:45:34.737 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-14T09:45:34.737 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-14T09:45:34.737 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-14T09:45:34.737 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:45:34.737 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-14T09:45:34.779 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 83662c4e-e1e7-11ee-95c9-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-14T09:45:34.779 DEBUG:teuthology.orchestra.run.smithi078:mon.a> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mon.a.service 2024-03-14T09:45:34.823 DEBUG:teuthology.orchestra.run.smithi078:mgr.x> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mgr.x.service 2024-03-14T09:45:34.868 INFO:tasks.cephadm:Bootstrapping... 2024-03-14T09:45:34.868 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 -v bootstrap --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.78 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-14T09:45:34.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:-- Logs begin at Thu 2024-03-14 09:30:54 UTC. -- 2024-03-14T09:45:34.933 INFO:journalctl@ceph.mgr.x.smithi078.stdout:-- Logs begin at Thu 2024-03-14 09:30:54 UTC. -- 2024-03-14T09:45:35.082 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-03-14T09:45:35.082 INFO:teuthology.orchestra.run.smithi078.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0', '-v', 'bootstrap', '--fsid', '83662c4e-e1e7-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.78', '--skip-admin-label'] 2024-03-14T09:45:35.126 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 4.6.1 2024-03-14T09:45:35.126 INFO:teuthology.orchestra.run.smithi078.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-14T09:45:35.127 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying podman|docker is present... 2024-03-14T09:45:35.170 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 4.6.1 2024-03-14T09:45:35.171 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying lvm2 is present... 2024-03-14T09:45:35.171 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying time synchronization is in place... 2024-03-14T09:45:35.185 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-14T09:45:35.185 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-14T09:45:35.197 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-14T09:45:35.197 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-14T09:45:35.209 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout enabled 2024-03-14T09:45:35.222 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout active 2024-03-14T09:45:35.222 INFO:teuthology.orchestra.run.smithi078.stdout:Unit chronyd.service is enabled and running 2024-03-14T09:45:35.222 INFO:teuthology.orchestra.run.smithi078.stdout:Repeating the final host check... 2024-03-14T09:45:35.267 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 4.6.1 2024-03-14T09:45:35.267 INFO:teuthology.orchestra.run.smithi078.stdout:podman (/bin/podman) version 4.6.1 is present 2024-03-14T09:45:35.267 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl is present 2024-03-14T09:45:35.267 INFO:teuthology.orchestra.run.smithi078.stdout:lvcreate is present 2024-03-14T09:45:35.280 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-14T09:45:35.280 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-14T09:45:35.291 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-14T09:45:35.291 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-14T09:45:35.304 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout enabled 2024-03-14T09:45:35.316 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout active 2024-03-14T09:45:35.316 INFO:teuthology.orchestra.run.smithi078.stdout:Unit chronyd.service is enabled and running 2024-03-14T09:45:35.317 INFO:teuthology.orchestra.run.smithi078.stdout:Host looks OK 2024-03-14T09:45:35.317 INFO:teuthology.orchestra.run.smithi078.stdout:Cluster fsid: 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:45:35.317 INFO:teuthology.orchestra.run.smithi078.stdout:Acquiring lock 140593298455016 on /run/cephadm/83662c4e-e1e7-11ee-95c9-87774f69a715.lock 2024-03-14T09:45:35.317 INFO:teuthology.orchestra.run.smithi078.stdout:Lock 140593298455016 acquired on /run/cephadm/83662c4e-e1e7-11ee-95c9-87774f69a715.lock 2024-03-14T09:45:35.318 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying IP 172.21.15.78 port 3300 ... 2024-03-14T09:45:35.318 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying IP 172.21.15.78 port 6789 ... 2024-03-14T09:45:35.318 INFO:teuthology.orchestra.run.smithi078.stdout:Base mon IP(s) is [172.21.15.78:3300, 172.21.15.78:6789], mon addrv is [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2024-03-14T09:45:35.326 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2024-03-14T09:45:35.326 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.78 metric 100 2024-03-14T09:45:35.333 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-14T09:45:35.333 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-14T09:45:35.333 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1769sec hoplimit 64 pref medium 2024-03-14T09:45:35.339 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-14T09:45:35.339 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-14T09:45:35.339 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-14T09:45:35.340 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-14T09:45:35.340 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2024-03-14T09:45:35.340 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-14T09:45:35.340 INFO:teuthology.orchestra.run.smithi078.stdout:Mon IP `172.21.15.78` is in CIDR network `172.21.0.0/20` 2024-03-14T09:45:35.341 INFO:teuthology.orchestra.run.smithi078.stdout:Mon IP `172.21.15.78` is in CIDR network `172.21.0.0/20` 2024-03-14T09:45:35.341 INFO:teuthology.orchestra.run.smithi078.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-14T09:45:35.342 INFO:teuthology.orchestra.run.smithi078.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-14T09:45:35.343 INFO:teuthology.orchestra.run.smithi078.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0... 2024-03-14T09:45:36.172 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout a9e65ad305ae06daed88c6b29b1bd5b92a5f2a05f0abcf928fb8895930b5caf2 2024-03-14T09:45:36.173 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0... 2024-03-14T09:45:36.173 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Getting image source signatures 2024-03-14T09:45:36.173 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying blob sha256:b3e06e8dfae29dfb09d4827fcedc0886ec1ba2d754371dd77c1263224772a45e 2024-03-14T09:45:36.173 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-14T09:45:36.173 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying config sha256:a9e65ad305ae06daed88c6b29b1bd5b92a5f2a05f0abcf928fb8895930b5caf2 2024-03-14T09:45:36.173 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-14T09:45:36.854 INFO:teuthology.orchestra.run.smithi078.stdout:ceph: stdout ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable) 2024-03-14T09:45:36.854 INFO:teuthology.orchestra.run.smithi078.stdout:Ceph version: ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable) 2024-03-14T09:45:36.854 INFO:teuthology.orchestra.run.smithi078.stdout:Extracting ceph user uid/gid from container image... 2024-03-14T09:45:37.470 INFO:teuthology.orchestra.run.smithi078.stdout:stat: stdout 167 167 2024-03-14T09:45:37.470 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial keys... 2024-03-14T09:45:38.029 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQBBx/JlAopiKxAAS9A0XGKTLOEKZOtrM+H0wg== 2024-03-14T09:45:38.612 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQBCx/JleIgwExAAUSNAuqLTHg7BeJ7ibhCHnQ== 2024-03-14T09:45:40.591 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQBCx/JlaV5FNBAAKNPuO+IwTi+B7whzhLdQ5A== 2024-03-14T09:45:40.592 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial monmap... 2024-03-14T09:45:41.204 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-14T09:45:41.204 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-03-14T09:45:41.204 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:45:41.204 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-14T09:45:41.204 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool for a [v2:172.21.15.78:3300,v1:172.21.15.78:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-14T09:45:41.204 INFO:teuthology.orchestra.run.smithi078.stdout:setting min_mon_release = pacific 2024-03-14T09:45:41.204 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: set fsid to 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:45:41.204 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-14T09:45:41.205 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:45:41.205 INFO:teuthology.orchestra.run.smithi078.stdout:Creating mon... 2024-03-14T09:45:41.818 INFO:teuthology.orchestra.run.smithi078.stdout:create mon.a on 2024-03-14T09:45:42.063 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-03-14T09:45:42.253 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-14T09:45:42.478 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-83662c4e-e1e7-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-83662c4e-e1e7-11ee-95c9-87774f69a715.target. 2024-03-14T09:45:42.478 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-83662c4e-e1e7-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-83662c4e-e1e7-11ee-95c9-87774f69a715.target. 2024-03-14T09:45:42.811 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mon.a 2024-03-14T09:45:42.811 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to reset failed state of unit ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-14T09:45:42.992 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-83662c4e-e1e7-11ee-95c9-87774f69a715.target.wants/ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@.service. 2024-03-14T09:45:43.082 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:43 smithi078 systemd[1]: Starting Ceph mon.a for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T09:45:43.505 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-14T09:45:43.505 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-03-14T09:45:43.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:43 smithi078 bash[37614]: 2d91064e88d007db15d30cf0f0f97be94e2782495e644773f2943844c9f344f6 2024-03-14T09:45:43.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:43 smithi078 systemd[1]: Started Ceph mon.a for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:45:43.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:43 smithi078 ceph-mon[37652]: mkfs 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:45:43.521 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-14T09:45:43.521 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-14T09:45:43.521 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-03-14T09:45:43.521 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to enable service . firewalld.service is not available 2024-03-14T09:45:43.522 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mon to start... 2024-03-14T09:45:43.522 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mon... 2024-03-14T09:45:43.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:43 smithi078 ceph-mon[37652]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-14T09:45:44.360 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout cluster: 2024-03-14T09:45:44.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout id: 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:45:44.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-14T09:45:44.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-14T09:45:44.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout services: 2024-03-14T09:45:44.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.524219s) 2024-03-14T09:45:44.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-14T09:45:44.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-14T09:45:44.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-14T09:45:44.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout data: 2024-03-14T09:45:44.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-14T09:45:44.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-14T09:45:44.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-14T09:45:44.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout pgs: 2024-03-14T09:45:44.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-14T09:45:44.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.955+0000 7f9d1ee00700 1 Processor -- start 2024-03-14T09:45:44.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.955+0000 7f9d1ee00700 1 -- start start 2024-03-14T09:45:44.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.956+0000 7f9d1ee00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1807a2f0 0x7f9d1807a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:45:44.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.956+0000 7f9d1ee00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1807ace0 con 0x7f9d1807a2f0 2024-03-14T09:45:44.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.957+0000 7f9d17e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1807a2f0 0x7f9d1807a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:45:44.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.957+0000 7f9d17e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1807a2f0 0x7f9d1807a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37276/0 (socket says 172.21.15.78:37276) 2024-03-14T09:45:44.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.957+0000 7f9d17e00700 1 -- 172.21.15.78:0/4187015784 learned_addr learned my addr 172.21.15.78:0/4187015784 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:45:44.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.957+0000 7f9d17e00700 1 -- 172.21.15.78:0/4187015784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1807ae20 con 0x7f9d1807a2f0 2024-03-14T09:45:44.370 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.957+0000 7f9d17e00700 1 --2- 172.21.15.78:0/4187015784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1807a2f0 0x7f9d1807a710 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9d080098a0 tx=0x7f9d08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d479bd9db7563c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:45:44.370 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.957+0000 7f9d16a00700 1 -- 172.21.15.78:0/4187015784 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9d08004160 con 0x7f9d1807a2f0 2024-03-14T09:45:44.370 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.957+0000 7f9d16a00700 1 -- 172.21.15.78:0/4187015784 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d080042c0 con 0x7f9d1807a2f0 2024-03-14T09:45:44.370 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.957+0000 7f9d16a00700 1 -- 172.21.15.78:0/4187015784 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9d080045a0 con 0x7f9d1807a2f0 2024-03-14T09:45:44.370 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.958+0000 7f9d1ee00700 1 -- 172.21.15.78:0/4187015784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1807a2f0 msgr2=0x7f9d1807a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:45:44.370 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.958+0000 7f9d1ee00700 1 --2- 172.21.15.78:0/4187015784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1807a2f0 0x7f9d1807a710 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9d080098a0 tx=0x7f9d08009bb0 comp rx=0 tx=0).stop 2024-03-14T09:45:44.370 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.958+0000 7f9d1ee00700 1 -- 172.21.15.78:0/4187015784 shutdown_connections 2024-03-14T09:45:44.370 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.958+0000 7f9d1ee00700 1 --2- 172.21.15.78:0/4187015784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1807a2f0 0x7f9d1807a710 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:45:44.370 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.958+0000 7f9d1ee00700 1 -- 172.21.15.78:0/4187015784 >> 172.21.15.78:0/4187015784 conn(0x7f9d18101a30 msgr2=0x7f9d18103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:45:44.370 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.958+0000 7f9d1ee00700 1 -- 172.21.15.78:0/4187015784 shutdown_connections 2024-03-14T09:45:44.371 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.958+0000 7f9d1ee00700 1 -- 172.21.15.78:0/4187015784 wait complete. 2024-03-14T09:45:44.371 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.959+0000 7f9d1ee00700 1 Processor -- start 2024-03-14T09:45:44.371 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.959+0000 7f9d1ee00700 1 -- start start 2024-03-14T09:45:44.371 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.960+0000 7f9d1ee00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1807a2f0 0x7f9d1819bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:45:44.371 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.960+0000 7f9d1ee00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1819c510 con 0x7f9d1807a2f0 2024-03-14T09:45:44.371 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.960+0000 7f9d17e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1807a2f0 0x7f9d1819bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:45:44.371 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.960+0000 7f9d17e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1807a2f0 0x7f9d1819bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37284/0 (socket says 172.21.15.78:37284) 2024-03-14T09:45:44.371 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.960+0000 7f9d17e00700 1 -- 172.21.15.78:0/940051912 learned_addr learned my addr 172.21.15.78:0/940051912 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:45:44.371 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.960+0000 7f9d17e00700 1 -- 172.21.15.78:0/940051912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d08009550 con 0x7f9d1807a2f0 2024-03-14T09:45:44.371 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.961+0000 7f9d17e00700 1 --2- 172.21.15.78:0/940051912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1807a2f0 0x7f9d1819bfd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9d1807b360 tx=0x7f9d08004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:45:44.371 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.961+0000 7f9d14c00700 1 -- 172.21.15.78:0/940051912 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9d08011630 con 0x7f9d1807a2f0 2024-03-14T09:45:44.371 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.961+0000 7f9d14c00700 1 -- 172.21.15.78:0/940051912 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d08011790 con 0x7f9d1807a2f0 2024-03-14T09:45:44.372 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.961+0000 7f9d14c00700 1 -- 172.21.15.78:0/940051912 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9d08011a30 con 0x7f9d1807a2f0 2024-03-14T09:45:44.372 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.961+0000 7f9d1ee00700 1 -- 172.21.15.78:0/940051912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d1819c710 con 0x7f9d1807a2f0 2024-03-14T09:45:44.372 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.961+0000 7f9d1ee00700 1 -- 172.21.15.78:0/940051912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1819cbb0 con 0x7f9d1807a2f0 2024-03-14T09:45:44.372 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.961+0000 7f9d14c00700 1 -- 172.21.15.78:0/940051912 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9d08022070 con 0x7f9d1807a2f0 2024-03-14T09:45:44.372 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.961+0000 7f9d14c00700 1 -- 172.21.15.78:0/940051912 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9d0801c450 con 0x7f9d1807a2f0 2024-03-14T09:45:44.372 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.962+0000 7f9d1ee00700 1 -- 172.21.15.78:0/940051912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d18195df0 con 0x7f9d1807a2f0 2024-03-14T09:45:44.372 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:43.965+0000 7f9d14c00700 1 -- 172.21.15.78:0/940051912 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f9d0801b450 con 0x7f9d1807a2f0 2024-03-14T09:45:44.372 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.036+0000 7f9d1ee00700 1 -- 172.21.15.78:0/940051912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f9d1810c570 con 0x7f9d1807a2f0 2024-03-14T09:45:44.372 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.037+0000 7f9d14c00700 1 -- 172.21.15.78:0/940051912 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f9d0801b5f0 con 0x7f9d1807a2f0 2024-03-14T09:45:44.372 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.037+0000 7f9d1ee00700 1 -- 172.21.15.78:0/940051912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1807a2f0 msgr2=0x7f9d1819bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:45:44.372 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.037+0000 7f9d1ee00700 1 --2- 172.21.15.78:0/940051912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1807a2f0 0x7f9d1819bfd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9d1807b360 tx=0x7f9d08004750 comp rx=0 tx=0).stop 2024-03-14T09:45:44.372 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.038+0000 7f9d1ee00700 1 -- 172.21.15.78:0/940051912 shutdown_connections 2024-03-14T09:45:44.372 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.038+0000 7f9d1ee00700 1 --2- 172.21.15.78:0/940051912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d1807a2f0 0x7f9d1819bfd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:45:44.373 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.038+0000 7f9d1ee00700 1 -- 172.21.15.78:0/940051912 >> 172.21.15.78:0/940051912 conn(0x7f9d18101a30 msgr2=0x7f9d18103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:45:44.373 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.038+0000 7f9d1ee00700 1 -- 172.21.15.78:0/940051912 shutdown_connections 2024-03-14T09:45:44.373 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.038+0000 7f9d1ee00700 1 -- 172.21.15.78:0/940051912 wait complete. 2024-03-14T09:45:44.373 INFO:teuthology.orchestra.run.smithi078.stdout:mon is available 2024-03-14T09:45:44.373 INFO:teuthology.orchestra.run.smithi078.stdout:Assimilating anything we can from ceph.conf... 2024-03-14T09:45:44.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:44 smithi078 ceph-mon[37652]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-14T09:45:44.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:44 smithi078 ceph-mon[37652]: monmap e1: 1 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T09:45:44.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:44 smithi078 ceph-mon[37652]: fsmap 2024-03-14T09:45:44.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:44 smithi078 ceph-mon[37652]: osdmap e1: 0 total, 0 up, 0 in 2024-03-14T09:45:44.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:44 smithi078 ceph-mon[37652]: mgrmap e1: no daemons active 2024-03-14T09:45:44.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:44 smithi078 ceph-mon[37652]: from='client.? 172.21.15.78:0/940051912' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-14T09:45:45.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-14T09:45:45.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [global] 2024-03-14T09:45:45.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout fsid = 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:45:45.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2024-03-14T09:45:45.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-14T09:45:45.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-14T09:45:45.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-14T09:45:45.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-14T09:45:45.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-14T09:45:45.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-14T09:45:45.255 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-14T09:45:45.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-14T09:45:45.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [osd] 2024-03-14T09:45:45.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-14T09:45:45.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-14T09:45:45.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-14T09:45:45.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-14T09:45:45.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-14T09:45:45.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.795+0000 7fcb9c400700 1 Processor -- start 2024-03-14T09:45:45.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.795+0000 7fcb9c400700 1 -- start start 2024-03-14T09:45:45.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.796+0000 7fcb9c400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb9407a310 0x7fcb9407a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:45:45.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.796+0000 7fcb9c400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9407ad00 con 0x7fcb9407a310 2024-03-14T09:45:45.256 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.796+0000 7fcb99a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb9407a310 0x7fcb9407a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:45:45.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.796+0000 7fcb99a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb9407a310 0x7fcb9407a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37292/0 (socket says 172.21.15.78:37292) 2024-03-14T09:45:45.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.796+0000 7fcb99a00700 1 -- 172.21.15.78:0/3935862470 learned_addr learned my addr 172.21.15.78:0/3935862470 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:45:45.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.796+0000 7fcb99a00700 1 -- 172.21.15.78:0/3935862470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9407ae40 con 0x7fcb9407a310 2024-03-14T09:45:45.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.797+0000 7fcb99a00700 1 --2- 172.21.15.78:0/3935862470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb9407a310 0x7fcb9407a730 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcb880098a0 tx=0x7fcb88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b0a4a6f35110088 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:45:45.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.797+0000 7fcb93e00700 1 -- 172.21.15.78:0/3935862470 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb88004160 con 0x7fcb9407a310 2024-03-14T09:45:45.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.797+0000 7fcb93e00700 1 -- 172.21.15.78:0/3935862470 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb880042c0 con 0x7fcb9407a310 2024-03-14T09:45:45.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.797+0000 7fcb93e00700 1 -- 172.21.15.78:0/3935862470 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb880045a0 con 0x7fcb9407a310 2024-03-14T09:45:45.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.797+0000 7fcb9c400700 1 -- 172.21.15.78:0/3935862470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb9407a310 msgr2=0x7fcb9407a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:45:45.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.797+0000 7fcb9c400700 1 --2- 172.21.15.78:0/3935862470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb9407a310 0x7fcb9407a730 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcb880098a0 tx=0x7fcb88009bb0 comp rx=0 tx=0).stop 2024-03-14T09:45:45.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.798+0000 7fcb9c400700 1 -- 172.21.15.78:0/3935862470 shutdown_connections 2024-03-14T09:45:45.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.798+0000 7fcb9c400700 1 --2- 172.21.15.78:0/3935862470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb9407a310 0x7fcb9407a730 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:45:45.257 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.798+0000 7fcb9c400700 1 -- 172.21.15.78:0/3935862470 >> 172.21.15.78:0/3935862470 conn(0x7fcb94101a30 msgr2=0x7fcb94103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:45:45.258 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.798+0000 7fcb9c400700 1 -- 172.21.15.78:0/3935862470 shutdown_connections 2024-03-14T09:45:45.258 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.798+0000 7fcb9c400700 1 -- 172.21.15.78:0/3935862470 wait complete. 2024-03-14T09:45:45.258 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.799+0000 7fcb9c400700 1 Processor -- start 2024-03-14T09:45:45.258 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.799+0000 7fcb9c400700 1 -- start start 2024-03-14T09:45:45.258 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.799+0000 7fcb9c400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb9407a310 0x7fcb9419c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:45:45.258 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.799+0000 7fcb9c400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9419c540 con 0x7fcb9407a310 2024-03-14T09:45:45.258 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.799+0000 7fcb99a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb9407a310 0x7fcb9419c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:45:45.258 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.799+0000 7fcb99a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb9407a310 0x7fcb9419c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37294/0 (socket says 172.21.15.78:37294) 2024-03-14T09:45:45.258 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.799+0000 7fcb99a00700 1 -- 172.21.15.78:0/1438822946 learned_addr learned my addr 172.21.15.78:0/1438822946 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:45:45.258 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.800+0000 7fcb99a00700 1 -- 172.21.15.78:0/1438822946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb88009550 con 0x7fcb9407a310 2024-03-14T09:45:45.259 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.800+0000 7fcb99a00700 1 --2- 172.21.15.78:0/1438822946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb9407a310 0x7fcb9419c000 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fcb88004000 tx=0x7fcb88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:45:45.259 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.800+0000 7fcb92000700 1 -- 172.21.15.78:0/1438822946 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb88011630 con 0x7fcb9407a310 2024-03-14T09:45:45.259 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.800+0000 7fcb92000700 1 -- 172.21.15.78:0/1438822946 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb88011790 con 0x7fcb9407a310 2024-03-14T09:45:45.259 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.800+0000 7fcb92000700 1 -- 172.21.15.78:0/1438822946 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcb88011a30 con 0x7fcb9407a310 2024-03-14T09:45:45.259 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.800+0000 7fcb9c400700 1 -- 172.21.15.78:0/1438822946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9419c740 con 0x7fcb9407a310 2024-03-14T09:45:45.259 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.800+0000 7fcb9c400700 1 -- 172.21.15.78:0/1438822946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9419cbe0 con 0x7fcb9407a310 2024-03-14T09:45:45.259 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.801+0000 7fcb92000700 1 -- 172.21.15.78:0/1438822946 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcb88022070 con 0x7fcb9407a310 2024-03-14T09:45:45.259 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.801+0000 7fcb92000700 1 -- 172.21.15.78:0/1438822946 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcb8801c450 con 0x7fcb9407a310 2024-03-14T09:45:45.259 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.801+0000 7fcb9c400700 1 -- 172.21.15.78:0/1438822946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb94195dd0 con 0x7fcb9407a310 2024-03-14T09:45:45.259 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.804+0000 7fcb92000700 1 -- 172.21.15.78:0/1438822946 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fcb8801b450 con 0x7fcb9407a310 2024-03-14T09:45:45.259 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.872+0000 7fcb9c400700 1 -- 172.21.15.78:0/1438822946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fcb9410c580 con 0x7fcb9407a310 2024-03-14T09:45:45.260 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.877+0000 7fcb92000700 1 -- 172.21.15.78:0/1438822946 <== mon.0 v2:172.21.15.78:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb8801b5f0 con 0x7fcb9407a310 2024-03-14T09:45:45.260 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.877+0000 7fcb92000700 1 -- 172.21.15.78:0/1438822946 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fcb88037610 con 0x7fcb9407a310 2024-03-14T09:45:45.260 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.878+0000 7fcb9c400700 1 -- 172.21.15.78:0/1438822946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb9407a310 msgr2=0x7fcb9419c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:45:45.260 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.878+0000 7fcb9c400700 1 --2- 172.21.15.78:0/1438822946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb9407a310 0x7fcb9419c000 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fcb88004000 tx=0x7fcb88004750 comp rx=0 tx=0).stop 2024-03-14T09:45:45.260 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.878+0000 7fcb9c400700 1 -- 172.21.15.78:0/1438822946 shutdown_connections 2024-03-14T09:45:45.260 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.878+0000 7fcb9c400700 1 --2- 172.21.15.78:0/1438822946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb9407a310 0x7fcb9419c000 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:45:45.260 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.878+0000 7fcb9c400700 1 -- 172.21.15.78:0/1438822946 >> 172.21.15.78:0/1438822946 conn(0x7fcb94101a30 msgr2=0x7fcb94103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:45:45.260 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.878+0000 7fcb9c400700 1 -- 172.21.15.78:0/1438822946 shutdown_connections 2024-03-14T09:45:45.260 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:44.878+0000 7fcb9c400700 1 -- 172.21.15.78:0/1438822946 wait complete. 2024-03-14T09:45:45.260 INFO:teuthology.orchestra.run.smithi078.stdout:Generating new minimal ceph.conf... 2024-03-14T09:45:45.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:45 smithi078 ceph-mon[37652]: from='client.? 172.21.15.78:0/1438822946' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-14T09:45:45.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:45 smithi078 ceph-mon[37652]: from='client.? 172.21.15.78:0/1438822946' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-14T09:45:46.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.661+0000 7f7eec600700 1 Processor -- start 2024-03-14T09:45:46.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.661+0000 7f7eec600700 1 -- start start 2024-03-14T09:45:46.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.662+0000 7f7eec600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ee407a310 0x7f7ee407a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:45:46.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.662+0000 7f7eec600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee407ad00 con 0x7f7ee407a310 2024-03-14T09:45:46.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.662+0000 7f7ee9c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ee407a310 0x7f7ee407a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:45:46.077 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.662+0000 7f7ee9c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ee407a310 0x7f7ee407a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37296/0 (socket says 172.21.15.78:37296) 2024-03-14T09:45:46.077 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.662+0000 7f7ee9c00700 1 -- 172.21.15.78:0/1824336246 learned_addr learned my addr 172.21.15.78:0/1824336246 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:45:46.077 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.662+0000 7f7ee9c00700 1 -- 172.21.15.78:0/1824336246 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee407ae40 con 0x7f7ee407a310 2024-03-14T09:45:46.077 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.663+0000 7f7ee9c00700 1 --2- 172.21.15.78:0/1824336246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ee407a310 0x7f7ee407a730 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80098a0 tx=0x7f7ed8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52e11971254b565 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:45:46.077 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.663+0000 7f7ee3e00700 1 -- 172.21.15.78:0/1824336246 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ed8004160 con 0x7f7ee407a310 2024-03-14T09:45:46.077 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.663+0000 7f7ee3e00700 1 -- 172.21.15.78:0/1824336246 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ed80042c0 con 0x7f7ee407a310 2024-03-14T09:45:46.077 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.663+0000 7f7ee3e00700 1 -- 172.21.15.78:0/1824336246 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ed8011a00 con 0x7f7ee407a310 2024-03-14T09:45:46.077 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.663+0000 7f7eec600700 1 -- 172.21.15.78:0/1824336246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ee407a310 msgr2=0x7f7ee407a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:45:46.077 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.663+0000 7f7eec600700 1 --2- 172.21.15.78:0/1824336246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ee407a310 0x7f7ee407a730 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80098a0 tx=0x7f7ed8009bb0 comp rx=0 tx=0).stop 2024-03-14T09:45:46.077 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.664+0000 7f7eec600700 1 -- 172.21.15.78:0/1824336246 shutdown_connections 2024-03-14T09:45:46.077 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.664+0000 7f7eec600700 1 --2- 172.21.15.78:0/1824336246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ee407a310 0x7f7ee407a730 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:45:46.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.664+0000 7f7eec600700 1 -- 172.21.15.78:0/1824336246 >> 172.21.15.78:0/1824336246 conn(0x7f7ee4101a30 msgr2=0x7f7ee4103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:45:46.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.664+0000 7f7eec600700 1 -- 172.21.15.78:0/1824336246 shutdown_connections 2024-03-14T09:45:46.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.664+0000 7f7eec600700 1 -- 172.21.15.78:0/1824336246 wait complete. 2024-03-14T09:45:46.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.665+0000 7f7eec600700 1 Processor -- start 2024-03-14T09:45:46.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.665+0000 7f7eec600700 1 -- start start 2024-03-14T09:45:46.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.665+0000 7f7eec600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ee407a310 0x7f7ee419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:45:46.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.665+0000 7f7eec600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee419cdd0 con 0x7f7ee407a310 2024-03-14T09:45:46.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.665+0000 7f7ee9c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ee407a310 0x7f7ee419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:45:46.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.665+0000 7f7ee9c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ee407a310 0x7f7ee419c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37306/0 (socket says 172.21.15.78:37306) 2024-03-14T09:45:46.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.665+0000 7f7ee9c00700 1 -- 172.21.15.78:0/2995426535 learned_addr learned my addr 172.21.15.78:0/2995426535 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:45:46.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.666+0000 7f7ee9c00700 1 -- 172.21.15.78:0/2995426535 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed8009550 con 0x7f7ee407a310 2024-03-14T09:45:46.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.666+0000 7f7ee9c00700 1 --2- 172.21.15.78:0/2995426535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ee407a310 0x7f7ee419c890 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009130 tx=0x7f7ed8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:45:46.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.666+0000 7f7ee2000700 1 -- 172.21.15.78:0/2995426535 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ed8003ab0 con 0x7f7ee407a310 2024-03-14T09:45:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.666+0000 7f7ee2000700 1 -- 172.21.15.78:0/2995426535 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ed8003c10 con 0x7f7ee407a310 2024-03-14T09:45:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.666+0000 7f7ee2000700 1 -- 172.21.15.78:0/2995426535 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ed80247d0 con 0x7f7ee407a310 2024-03-14T09:45:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.666+0000 7f7eec600700 1 -- 172.21.15.78:0/2995426535 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee419cfd0 con 0x7f7ee407a310 2024-03-14T09:45:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.666+0000 7f7eec600700 1 -- 172.21.15.78:0/2995426535 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee419d470 con 0x7f7ee407a310 2024-03-14T09:45:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.667+0000 7f7ee2000700 1 -- 172.21.15.78:0/2995426535 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7ed8024d00 con 0x7f7ee407a310 2024-03-14T09:45:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.667+0000 7f7ee2000700 1 -- 172.21.15.78:0/2995426535 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7ed80193c0 con 0x7f7ee407a310 2024-03-14T09:45:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.668+0000 7f7eec600700 1 -- 172.21.15.78:0/2995426535 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee410cca0 con 0x7f7ee407a310 2024-03-14T09:45:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.670+0000 7f7ee2000700 1 -- 172.21.15.78:0/2995426535 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f7ed801abb0 con 0x7f7ee407a310 2024-03-14T09:45:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.741+0000 7f7eec600700 1 -- 172.21.15.78:0/2995426535 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f7ee410ce90 con 0x7f7ee407a310 2024-03-14T09:45:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.742+0000 7f7ee2000700 1 -- 172.21.15.78:0/2995426535 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f7ed801a430 con 0x7f7ee407a310 2024-03-14T09:45:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.742+0000 7f7eec600700 1 -- 172.21.15.78:0/2995426535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ee407a310 msgr2=0x7f7ee419c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:45:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.742+0000 7f7eec600700 1 --2- 172.21.15.78:0/2995426535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ee407a310 0x7f7ee419c890 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009130 tx=0x7f7ed8011ad0 comp rx=0 tx=0).stop 2024-03-14T09:45:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.743+0000 7f7eec600700 1 -- 172.21.15.78:0/2995426535 shutdown_connections 2024-03-14T09:45:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.743+0000 7f7eec600700 1 --2- 172.21.15.78:0/2995426535 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ee407a310 0x7f7ee419c890 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:45:46.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.743+0000 7f7eec600700 1 -- 172.21.15.78:0/2995426535 >> 172.21.15.78:0/2995426535 conn(0x7f7ee4101a30 msgr2=0x7f7ee41026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:45:46.080 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.743+0000 7f7eec600700 1 -- 172.21.15.78:0/2995426535 shutdown_connections 2024-03-14T09:45:46.080 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:45.743+0000 7f7eec600700 1 -- 172.21.15.78:0/2995426535 wait complete. 2024-03-14T09:45:46.080 INFO:teuthology.orchestra.run.smithi078.stdout:Restarting the monitor... 2024-03-14T09:45:46.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:46 smithi078 systemd[1]: Stopping Ceph mon.a for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T09:45:46.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:46 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mon-a[37629]: 2024-03-14T09:45:46.359+0000 7f08f8600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T09:45:46.695 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:46 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mon-a[37629]: 2024-03-14T09:45:46.359+0000 7f08f8600700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-14T09:45:47.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:46 smithi078 bash[37982]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mon-a 2024-03-14T09:45:47.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 systemd[1]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mon.a.service: Succeeded. 2024-03-14T09:45:47.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 systemd[1]: Stopped Ceph mon.a for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:45:47.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 systemd[1]: Starting Ceph mon.a for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T09:45:47.543 INFO:teuthology.orchestra.run.smithi078.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-14T09:45:47.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 bash[38105]: f2a8137bfb2d32747aa52dd15c2f9288715d4cdda07d41fcd7145a41c3b80c8c 2024-03-14T09:45:47.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 systemd[1]: Started Ceph mon.a for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:45:47.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: set uid:gid to 167:167 (ceph:ceph) 2024-03-14T09:45:47.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable), process ceph-mon, pid 7 2024-03-14T09:45:47.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: pidfile_write: ignore empty --pid-file 2024-03-14T09:45:47.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: load: jerasure load: lrc 2024-03-14T09:45:47.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: RocksDB version: 7.9.2 2024-03-14T09:45:47.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Git sha 0 2024-03-14T09:45:47.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Compile date 2024-03-13 02:27:37 2024-03-14T09:45:47.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: DB SUMMARY 2024-03-14T09:45:47.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: DB Session ID: 2W961KBCC6Z0UOWF9WNR 2024-03-14T09:45:47.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: CURRENT file: CURRENT 2024-03-14T09:45:47.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: IDENTITY file: IDENTITY 2024-03-14T09:45:47.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-14T09:45:47.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-14T09:45:47.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 98028 ; 2024-03-14T09:45:47.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.error_if_exists: 0 2024-03-14T09:45:47.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.create_if_missing: 0 2024-03-14T09:45:47.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.paranoid_checks: 1 2024-03-14T09:45:47.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-14T09:45:47.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-14T09:45:47.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-14T09:45:47.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.env: 0x55cc3ce79720 2024-03-14T09:45:47.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.fs: PosixFileSystem 2024-03-14T09:45:47.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.info_log: 0x55cc3f7bb520 2024-03-14T09:45:47.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_file_opening_threads: 16 2024-03-14T09:45:47.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.statistics: (nil) 2024-03-14T09:45:47.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.use_fsync: 0 2024-03-14T09:45:47.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_log_file_size: 0 2024-03-14T09:45:47.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-14T09:45:47.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-14T09:45:47.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.keep_log_file_num: 1000 2024-03-14T09:45:47.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.recycle_log_file_num: 0 2024-03-14T09:45:47.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.allow_fallocate: 1 2024-03-14T09:45:47.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.allow_mmap_reads: 0 2024-03-14T09:45:47.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.allow_mmap_writes: 0 2024-03-14T09:45:47.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.use_direct_reads: 0 2024-03-14T09:45:47.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-14T09:45:47.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.create_missing_column_families: 0 2024-03-14T09:45:47.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.db_log_dir: 2024-03-14T09:45:47.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.wal_dir: 2024-03-14T09:45:47.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-14T09:45:47.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-14T09:45:47.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-14T09:45:47.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-14T09:45:47.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-14T09:45:47.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-14T09:45:47.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.advise_random_on_open: 1 2024-03-14T09:45:47.808 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.db_write_buffer_size: 0 2024-03-14T09:45:47.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.write_buffer_manager: 0x55cc3ea54320 2024-03-14T09:45:47.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-14T09:45:47.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-14T09:45:47.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-14T09:45:47.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.rate_limiter: (nil) 2024-03-14T09:45:47.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-14T09:45:47.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.wal_recovery_mode: 2 2024-03-14T09:45:47.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.enable_thread_tracking: 0 2024-03-14T09:45:47.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.enable_pipelined_write: 0 2024-03-14T09:45:47.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.unordered_write: 0 2024-03-14T09:45:47.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-14T09:45:47.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-14T09:45:47.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-14T09:45:47.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-14T09:45:47.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.row_cache: None 2024-03-14T09:45:47.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.wal_filter: None 2024-03-14T09:45:47.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-14T09:45:47.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.allow_ingest_behind: 0 2024-03-14T09:45:47.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.two_write_queues: 0 2024-03-14T09:45:47.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.manual_wal_flush: 0 2024-03-14T09:45:47.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.wal_compression: 0 2024-03-14T09:45:47.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.atomic_flush: 0 2024-03-14T09:45:47.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-14T09:45:47.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-14T09:45:47.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-14T09:45:47.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.log_readahead_size: 0 2024-03-14T09:45:47.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-14T09:45:47.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.best_efforts_recovery: 0 2024-03-14T09:45:47.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-14T09:45:47.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-14T09:45:47.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.allow_data_in_errors: 0 2024-03-14T09:45:47.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.db_host_id: __hostname__ 2024-03-14T09:45:47.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-14T09:45:47.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_background_jobs: 2 2024-03-14T09:45:47.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_background_compactions: -1 2024-03-14T09:45:47.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_subcompactions: 1 2024-03-14T09:45:47.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-14T09:45:47.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-14T09:45:47.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-14T09:45:47.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_total_wal_size: 0 2024-03-14T09:45:47.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-14T09:45:47.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-14T09:45:47.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-14T09:45:47.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-14T09:45:47.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_open_files: -1 2024-03-14T09:45:47.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.bytes_per_sync: 0 2024-03-14T09:45:47.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-14T09:45:47.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-14T09:45:47.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compaction_readahead_size: 0 2024-03-14T09:45:47.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_background_flushes: -1 2024-03-14T09:45:47.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Compression algorithms supported: 2024-03-14T09:45:47.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-14T09:45:47.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: kZSTD supported: 0 2024-03-14T09:45:47.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: kXpressCompression supported: 0 2024-03-14T09:45:47.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: kLZ4HCCompression supported: 1 2024-03-14T09:45:47.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: kZlibCompression supported: 1 2024-03-14T09:45:47.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: kSnappyCompression supported: 1 2024-03-14T09:45:47.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: kLZ4Compression supported: 1 2024-03-14T09:45:47.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: kBZip2Compression supported: 0 2024-03-14T09:45:47.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-14T09:45:47.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-14T09:45:47.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-14T09:45:47.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-14T09:45:47.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-14T09:45:47.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.merge_operator: 2024-03-14T09:45:47.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compaction_filter: None 2024-03-14T09:45:47.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compaction_filter_factory: None 2024-03-14T09:45:47.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.sst_partitioner_factory: None 2024-03-14T09:45:47.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-14T09:45:47.813 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-14T09:45:47.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cc3f7bb620) 2024-03-14T09:45:47.814 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks: 1 2024-03-14T09:45:47.814 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-14T09:45:47.814 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-14T09:45:47.814 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_top_level_index_and_filter: 1 2024-03-14T09:45:47.814 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_type: 0 2024-03-14T09:45:47.814 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_index_type: 0 2024-03-14T09:45:47.814 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_shortening: 1 2024-03-14T09:45:47.814 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-14T09:45:47.814 INFO:journalctl@ceph.mon.a.smithi078.stdout: checksum: 4 2024-03-14T09:45:47.814 INFO:journalctl@ceph.mon.a.smithi078.stdout: no_block_cache: 0 2024-03-14T09:45:47.814 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache: 0x55cc3ead51f0 2024-03-14T09:45:47.814 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_name: BinnedLRUCache 2024-03-14T09:45:47.815 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_options: 2024-03-14T09:45:47.815 INFO:journalctl@ceph.mon.a.smithi078.stdout: capacity : 536870912 2024-03-14T09:45:47.815 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_shard_bits : 4 2024-03-14T09:45:47.815 INFO:journalctl@ceph.mon.a.smithi078.stdout: strict_capacity_limit : 0 2024-03-14T09:45:47.815 INFO:journalctl@ceph.mon.a.smithi078.stdout: high_pri_pool_ratio: 0.000 2024-03-14T09:45:47.815 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_compressed: (nil) 2024-03-14T09:45:47.815 INFO:journalctl@ceph.mon.a.smithi078.stdout: persistent_cache: (nil) 2024-03-14T09:45:47.815 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size: 4096 2024-03-14T09:45:47.815 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size_deviation: 10 2024-03-14T09:45:47.815 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_restart_interval: 16 2024-03-14T09:45:47.815 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_block_restart_interval: 1 2024-03-14T09:45:47.815 INFO:journalctl@ceph.mon.a.smithi078.stdout: metadata_block_size: 4096 2024-03-14T09:45:47.815 INFO:journalctl@ceph.mon.a.smithi078.stdout: partition_filters: 0 2024-03-14T09:45:47.815 INFO:journalctl@ceph.mon.a.smithi078.stdout: use_delta_encoding: 1 2024-03-14T09:45:47.815 INFO:journalctl@ceph.mon.a.smithi078.stdout: filter_policy: bloomfilter 2024-03-14T09:45:47.815 INFO:journalctl@ceph.mon.a.smithi078.stdout: whole_key_filtering: 1 2024-03-14T09:45:47.816 INFO:journalctl@ceph.mon.a.smithi078.stdout: verify_compression: 0 2024-03-14T09:45:47.816 INFO:journalctl@ceph.mon.a.smithi078.stdout: read_amp_bytes_per_bit: 0 2024-03-14T09:45:47.816 INFO:journalctl@ceph.mon.a.smithi078.stdout: format_version: 5 2024-03-14T09:45:47.816 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_index_compression: 1 2024-03-14T09:45:47.816 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_align: 0 2024-03-14T09:45:47.816 INFO:journalctl@ceph.mon.a.smithi078.stdout: max_auto_readahead_size: 262144 2024-03-14T09:45:47.816 INFO:journalctl@ceph.mon.a.smithi078.stdout: prepopulate_block_cache: 0 2024-03-14T09:45:47.816 INFO:journalctl@ceph.mon.a.smithi078.stdout: initial_auto_readahead_size: 8192 2024-03-14T09:45:47.816 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_file_reads_for_auto_readahead: 2 2024-03-14T09:45:47.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.write_buffer_size: 33554432 2024-03-14T09:45:47.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_write_buffer_number: 2 2024-03-14T09:45:47.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compression: NoCompression 2024-03-14T09:45:47.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.bottommost_compression: Disabled 2024-03-14T09:45:47.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.prefix_extractor: nullptr 2024-03-14T09:45:47.817 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-14T09:45:47.817 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.num_levels: 7 2024-03-14T09:45:47.817 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-14T09:45:47.817 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-14T09:45:47.817 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-14T09:45:47.817 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-14T09:45:47.817 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-14T09:45:47.817 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-14T09:45:47.817 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-14T09:45:47.817 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-14T09:45:47.817 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-14T09:45:47.817 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-14T09:45:47.817 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-14T09:45:47.817 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-14T09:45:47.817 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-14T09:45:47.818 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compression_opts.level: 32767 2024-03-14T09:45:47.818 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compression_opts.strategy: 0 2024-03-14T09:45:47.818 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-14T09:45:47.818 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-14T09:45:47.818 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-14T09:45:47.818 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-14T09:45:47.818 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compression_opts.enabled: false 2024-03-14T09:45:47.818 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-14T09:45:47.818 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-14T09:45:47.818 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-14T09:45:47.818 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-14T09:45:47.818 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.target_file_size_base: 67108864 2024-03-14T09:45:47.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-14T09:45:47.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-14T09:45:47.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-14T09:45:47.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-14T09:45:47.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-14T09:45:47.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-14T09:45:47.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-14T09:45:47.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-14T09:45:47.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-14T09:45:47.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-14T09:45:47.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-14T09:45:47.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-14T09:45:47.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-14T09:45:47.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-14T09:45:47.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.arena_block_size: 1048576 2024-03-14T09:45:47.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-14T09:45:47.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-14T09:45:47.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.disable_auto_compactions: 0 2024-03-14T09:45:47.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-14T09:45:47.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-14T09:45:47.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-14T09:45:47.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-14T09:45:47.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-14T09:45:47.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-14T09:45:47.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-14T09:45:47.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-14T09:45:47.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-14T09:45:47.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-14T09:45:47.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.table_properties_collectors: 2024-03-14T09:45:47.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.inplace_update_support: 0 2024-03-14T09:45:47.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-14T09:45:47.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-14T09:45:47.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-14T09:45:47.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-14T09:45:47.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.bloom_locality: 0 2024-03-14T09:45:47.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.max_successive_merges: 0 2024-03-14T09:45:47.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-14T09:45:47.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.paranoid_file_checks: 0 2024-03-14T09:45:47.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.force_consistency_checks: 1 2024-03-14T09:45:47.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.report_bg_io_stats: 0 2024-03-14T09:45:47.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.ttl: 2592000 2024-03-14T09:45:47.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-14T09:45:47.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-14T09:45:47.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-14T09:45:47.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.enable_blob_files: false 2024-03-14T09:45:47.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.min_blob_size: 0 2024-03-14T09:45:47.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.blob_file_size: 268435456 2024-03-14T09:45:47.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-14T09:45:47.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-14T09:45:47.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-14T09:45:47.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-14T09:45:47.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-14T09:45:47.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.blob_file_starting_level: 0 2024-03-14T09:45:47.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-14T09:45:47.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-14T09:45:47.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-14T09:45:47.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: be2c876a-ccfe-4df4-bab2-9c0374319cb4 2024-03-14T09:45:47.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710409547571791, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-14T09:45:47.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-14T09:45:47.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710409547574692, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93394, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 309, "table_properties": {"data_size": 91399, "index_size": 277, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13725, "raw_average_key_size": 50, "raw_value_size": 83911, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 272, "num_filter_entries": 272, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710409547, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "be2c876a-ccfe-4df4-bab2-9c0374319cb4", "db_session_id": "2W961KBCC6Z0UOWF9WNR", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710409547574863, "job": 1, "event": "recovery_finished"} 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55cc3eb72000 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: DB pointer 0x55cc3eb60000 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** DB Stats ** 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-14T09:45:47.823 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2024-03-14T09:45:47.824 INFO:journalctl@ceph.mon.a.smithi078.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-14T09:45:47.824 INFO:journalctl@ceph.mon.a.smithi078.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-14T09:45:47.824 INFO:journalctl@ceph.mon.a.smithi078.stdout: L0 2/0 93.04 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 48.5 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-14T09:45:47.824 INFO:journalctl@ceph.mon.a.smithi078.stdout: Sum 2/0 93.04 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 48.5 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-14T09:45:47.824 INFO:journalctl@ceph.mon.a.smithi078.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 48.5 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-14T09:45:47.824 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-14T09:45:47.824 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2024-03-14T09:45:47.824 INFO:journalctl@ceph.mon.a.smithi078.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-14T09:45:47.824 INFO:journalctl@ceph.mon.a.smithi078.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-14T09:45:47.824 INFO:journalctl@ceph.mon.a.smithi078.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 48.5 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-14T09:45:47.824 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-14T09:45:47.825 INFO:journalctl@ceph.mon.a.smithi078.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-14T09:45:47.825 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-14T09:45:47.825 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-14T09:45:47.825 INFO:journalctl@ceph.mon.a.smithi078.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-14T09:45:47.825 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-14T09:45:47.825 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-14T09:45:47.825 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-14T09:45:47.825 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-14T09:45:47.825 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative compaction: 0.00 GB write, 9.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-14T09:45:47.826 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval compaction: 0.00 GB write, 9.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-14T09:45:47.826 INFO:journalctl@ceph.mon.a.smithi078.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-14T09:45:47.826 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache BinnedLRUCache@0x55cc3ead51f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-14T09:45:47.826 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-14T09:45:47.826 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-14T09:45:47.826 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-14T09:45:47.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: starting mon.a rank 0 at public addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] at bind addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:45:47.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: mon.a@-1(???) e1 preinit fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:45:47.827 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: mon.a@-1(???).mds e1 new map 2024-03-14T09:45:47.827 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: mon.a@-1(???).mds e1 print_map 2024-03-14T09:45:47.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: e1 2024-03-14T09:45:47.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T09:45:47.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T09:45:47.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: legacy client fscid: -1 2024-03-14T09:45:47.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2024-03-14T09:45:47.827 INFO:journalctl@ceph.mon.a.smithi078.stdout: No filesystems configured 2024-03-14T09:45:47.828 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-14T09:45:47.828 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-14T09:45:47.828 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-14T09:45:47.828 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-14T09:45:47.828 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-14T09:45:47.828 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-14T09:45:47.828 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: monmap e1: 1 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T09:45:47.828 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: fsmap 2024-03-14T09:45:47.828 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: osdmap e1: 0 total, 0 up, 0 in 2024-03-14T09:45:47.828 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:47 smithi078 ceph-mon[38143]: mgrmap e1: no daemons active 2024-03-14T09:45:48.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.949+0000 7f91e5c00700 1 Processor -- start 2024-03-14T09:45:48.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.950+0000 7f91e5c00700 1 -- start start 2024-03-14T09:45:48.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.950+0000 7f91e5c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e01068d0 0x7f91e01076b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:45:48.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.950+0000 7f91e5c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e0106d80 con 0x7f91e01068d0 2024-03-14T09:45:48.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.951+0000 7f91dea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e01068d0 0x7f91e01076b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:45:48.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.951+0000 7f91dea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e01068d0 0x7f91e01076b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37312/0 (socket says 172.21.15.78:37312) 2024-03-14T09:45:48.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.951+0000 7f91dea00700 1 -- 172.21.15.78:0/1397427774 learned_addr learned my addr 172.21.15.78:0/1397427774 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:45:48.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.951+0000 7f91dea00700 1 -- 172.21.15.78:0/1397427774 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e0106ec0 con 0x7f91e01068d0 2024-03-14T09:45:48.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.951+0000 7f91dea00700 1 --2- 172.21.15.78:0/1397427774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e01068d0 0x7f91e01076b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f91d00098a0 tx=0x7f91d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92a15dc6f5de88b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:45:48.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.952+0000 7f91dd600700 1 -- 172.21.15.78:0/1397427774 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91d0004160 con 0x7f91e01068d0 2024-03-14T09:45:48.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.952+0000 7f91dd600700 1 -- 172.21.15.78:0/1397427774 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91d00042c0 con 0x7f91e01068d0 2024-03-14T09:45:48.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.952+0000 7f91dd600700 1 -- 172.21.15.78:0/1397427774 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91d0011a00 con 0x7f91e01068d0 2024-03-14T09:45:48.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.952+0000 7f91e5c00700 1 -- 172.21.15.78:0/1397427774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e01068d0 msgr2=0x7f91e01076b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:45:48.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.952+0000 7f91e5c00700 1 --2- 172.21.15.78:0/1397427774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e01068d0 0x7f91e01076b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f91d00098a0 tx=0x7f91d0009bb0 comp rx=0 tx=0).stop 2024-03-14T09:45:48.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.952+0000 7f91e5c00700 1 -- 172.21.15.78:0/1397427774 shutdown_connections 2024-03-14T09:45:48.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.952+0000 7f91e5c00700 1 --2- 172.21.15.78:0/1397427774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e01068d0 0x7f91e01076b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:45:48.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.952+0000 7f91e5c00700 1 -- 172.21.15.78:0/1397427774 >> 172.21.15.78:0/1397427774 conn(0x7f91e0101a70 msgr2=0x7f91e0103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:45:48.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.952+0000 7f91e5c00700 1 -- 172.21.15.78:0/1397427774 shutdown_connections 2024-03-14T09:45:48.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.952+0000 7f91e5c00700 1 -- 172.21.15.78:0/1397427774 wait complete. 2024-03-14T09:45:48.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.953+0000 7f91e5c00700 1 Processor -- start 2024-03-14T09:45:48.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.953+0000 7f91e5c00700 1 -- start start 2024-03-14T09:45:48.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.954+0000 7f91e5c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e01068d0 0x7f91e019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:45:48.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.954+0000 7f91e5c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e019cdd0 con 0x7f91e01068d0 2024-03-14T09:45:48.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.954+0000 7f91dea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e01068d0 0x7f91e019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:45:48.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.954+0000 7f91dea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e01068d0 0x7f91e019c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37328/0 (socket says 172.21.15.78:37328) 2024-03-14T09:45:48.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.954+0000 7f91dea00700 1 -- 172.21.15.78:0/2326751813 learned_addr learned my addr 172.21.15.78:0/2326751813 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:45:48.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.954+0000 7f91dea00700 1 -- 172.21.15.78:0/2326751813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d0009550 con 0x7f91e01068d0 2024-03-14T09:45:48.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.955+0000 7f91dea00700 1 --2- 172.21.15.78:0/2326751813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e01068d0 0x7f91e019c890 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009130 tx=0x7f91d0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:45:48.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.955+0000 7f91d7400700 1 -- 172.21.15.78:0/2326751813 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91d0003ab0 con 0x7f91e01068d0 2024-03-14T09:45:48.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.955+0000 7f91d7400700 1 -- 172.21.15.78:0/2326751813 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91d0003c10 con 0x7f91e01068d0 2024-03-14T09:45:48.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.955+0000 7f91e5c00700 1 -- 172.21.15.78:0/2326751813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e019cfd0 con 0x7f91e01068d0 2024-03-14T09:45:48.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.955+0000 7f91d7400700 1 -- 172.21.15.78:0/2326751813 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91d00247d0 con 0x7f91e01068d0 2024-03-14T09:45:48.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.955+0000 7f91e5c00700 1 -- 172.21.15.78:0/2326751813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e019d470 con 0x7f91e01068d0 2024-03-14T09:45:48.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.955+0000 7f91d7400700 1 -- 172.21.15.78:0/2326751813 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f91d0024d00 con 0x7f91e01068d0 2024-03-14T09:45:48.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.955+0000 7f91d7400700 1 -- 172.21.15.78:0/2326751813 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f91d00193c0 con 0x7f91e01068d0 2024-03-14T09:45:48.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.956+0000 7f91e5c00700 1 -- 172.21.15.78:0/2326751813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e010aad0 con 0x7f91e01068d0 2024-03-14T09:45:48.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:47.959+0000 7f91d7400700 1 -- 172.21.15.78:0/2326751813 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f91d001abb0 con 0x7f91e01068d0 2024-03-14T09:45:48.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:48.027+0000 7f91e5c00700 1 -- 172.21.15.78:0/2326751813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f91e010ad10 con 0x7f91e01068d0 2024-03-14T09:45:48.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:48.033+0000 7f91d7400700 1 -- 172.21.15.78:0/2326751813 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f91d001ad50 con 0x7f91e01068d0 2024-03-14T09:45:48.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:48.034+0000 7f91e5c00700 1 -- 172.21.15.78:0/2326751813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e01068d0 msgr2=0x7f91e019c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:45:48.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:48.034+0000 7f91e5c00700 1 --2- 172.21.15.78:0/2326751813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e01068d0 0x7f91e019c890 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009130 tx=0x7f91d0011ad0 comp rx=0 tx=0).stop 2024-03-14T09:45:48.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:48.034+0000 7f91e5c00700 1 -- 172.21.15.78:0/2326751813 shutdown_connections 2024-03-14T09:45:48.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:48.034+0000 7f91e5c00700 1 --2- 172.21.15.78:0/2326751813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e01068d0 0x7f91e019c890 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:45:48.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:48.034+0000 7f91e5c00700 1 -- 172.21.15.78:0/2326751813 >> 172.21.15.78:0/2326751813 conn(0x7f91e0101a70 msgr2=0x7f91e0102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:45:48.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:48.034+0000 7f91e5c00700 1 -- 172.21.15.78:0/2326751813 shutdown_connections 2024-03-14T09:45:48.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:48.034+0000 7f91e5c00700 1 -- 172.21.15.78:0/2326751813 wait complete. 2024-03-14T09:45:48.409 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-14T09:45:48.426 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-14T09:45:48.426 INFO:teuthology.orchestra.run.smithi078.stdout:Creating mgr... 2024-03-14T09:45:48.427 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-14T09:45:48.427 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-14T09:45:48.771 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mgr.x 2024-03-14T09:45:48.771 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to reset failed state of unit ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-14T09:45:48.962 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-83662c4e-e1e7-11ee-95c9-87774f69a715.target.wants/ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@.service. 2024-03-14T09:45:49.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:49 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2326751813' entity='client.admin' 2024-03-14T09:45:49.054 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:48 smithi078 systemd[1]: Starting Ceph mgr.x for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T09:45:49.537 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-14T09:45:49.537 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-03-14T09:45:49.552 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-14T09:45:49.552 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-14T09:45:49.552 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-03-14T09:45:49.552 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to enable service . firewalld.service is not available 2024-03-14T09:45:49.569 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-14T09:45:49.570 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-03-14T09:45:49.585 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-14T09:45:49.585 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-14T09:45:49.585 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-03-14T09:45:49.585 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-14T09:45:49.585 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr to start... 2024-03-14T09:45:49.585 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr... 2024-03-14T09:45:49.764 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:49 smithi078 bash[38379]: c98e9884dd1cffa89e8fe719f21da771a96950b7d5e3aafe00d7bf102ca5b387 2024-03-14T09:45:49.764 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:49 smithi078 systemd[1]: Started Ceph mgr.x for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:45:50.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:50 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/526823231' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-14T09:45:50.303 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:50 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:50.050+0000 7f01c24f8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-14T09:45:50.304 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:50 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:50.178+0000 7f01c24f8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-14T09:45:50.304 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:50 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:50.294+0000 7f01c24f8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-14T09:45:50.420 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-14T09:45:50.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-14T09:45:50.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "83662c4e-e1e7-11ee-95c9-87774f69a715", 2024-03-14T09:45:50.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-03-14T09:45:50.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-14T09:45:50.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-14T09:45:50.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-14T09:45:50.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:45:50.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-14T09:45:50.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-14T09:45:50.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-03-14T09:45:50.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-14T09:45:50.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-14T09:45:50.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-03-14T09:45:50.424 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-14T09:45:50.424 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-14T09:45:50.424 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-14T09:45:50.424 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T09:45:50.424 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-14T09:45:50.424 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-14T09:45:50.424 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:45:50.424 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-14T09:45:50.424 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T09:45:50.424 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-14T09:45:50.424 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-14T09:45:50.424 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-14T09:45:50.432 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-14T09:45:50.432 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-14T09:45:50.432 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-14T09:45:50.432 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:45:50.433 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-14T09:45:50.433 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-14T09:45:50.433 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-14T09:45:50.433 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-14T09:45:50.433 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-14T09:45:50.433 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-14T09:45:50.433 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-14T09:45:50.433 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-14T09:45:50.433 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-14T09:45:50.433 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:45:50.433 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-14T09:45:50.433 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T09:45:50.433 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-14T09:45:50.433 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-14T09:45:50.434 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:45:50.434 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-14T09:45:50.434 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-14T09:45:50.434 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-14T09:45:50.434 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-14T09:45:50.434 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-14T09:45:50.434 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-14T09:45:50.434 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-03-14T09:45:50.434 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-14T09:45:50.434 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-14T09:45:50.434 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:45:50.434 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-14T09:45:50.434 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T09:45:50.434 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-03-14T09:45:43.515620+0000", 2024-03-14T09:45:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-14T09:45:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:45:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-14T09:45:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-14T09:45:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:49.999+0000 7fcd49200700 1 Processor -- start 2024-03-14T09:45:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:49.999+0000 7fcd49200700 1 -- start start 2024-03-14T09:45:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:49.999+0000 7fcd49200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd441068d0 0x7fcd441076b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:45:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:49.999+0000 7fcd49200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd44106d80 con 0x7fcd441068d0 2024-03-14T09:45:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.000+0000 7fcd41e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd441068d0 0x7fcd441076b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:45:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.000+0000 7fcd41e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd441068d0 0x7fcd441076b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37358/0 (socket says 172.21.15.78:37358) 2024-03-14T09:45:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.000+0000 7fcd41e00700 1 -- 172.21.15.78:0/2221647597 learned_addr learned my addr 172.21.15.78:0/2221647597 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:45:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.000+0000 7fcd41e00700 1 -- 172.21.15.78:0/2221647597 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd44106ec0 con 0x7fcd441068d0 2024-03-14T09:45:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.000+0000 7fcd41e00700 1 --2- 172.21.15.78:0/2221647597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd441068d0 0x7fcd441076b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcd340098a0 tx=0x7fcd34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4182e768ec76b63 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:45:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.001+0000 7fcd40a00700 1 -- 172.21.15.78:0/2221647597 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd34004160 con 0x7fcd441068d0 2024-03-14T09:45:50.435 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.001+0000 7fcd40a00700 1 -- 172.21.15.78:0/2221647597 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd340042c0 con 0x7fcd441068d0 2024-03-14T09:45:50.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.001+0000 7fcd40a00700 1 -- 172.21.15.78:0/2221647597 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd34011a00 con 0x7fcd441068d0 2024-03-14T09:45:50.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.001+0000 7fcd49200700 1 -- 172.21.15.78:0/2221647597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd441068d0 msgr2=0x7fcd441076b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:45:50.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.001+0000 7fcd49200700 1 --2- 172.21.15.78:0/2221647597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd441068d0 0x7fcd441076b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcd340098a0 tx=0x7fcd34009bb0 comp rx=0 tx=0).stop 2024-03-14T09:45:50.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.001+0000 7fcd49200700 1 -- 172.21.15.78:0/2221647597 shutdown_connections 2024-03-14T09:45:50.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.001+0000 7fcd49200700 1 --2- 172.21.15.78:0/2221647597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd441068d0 0x7fcd441076b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:45:50.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.001+0000 7fcd49200700 1 -- 172.21.15.78:0/2221647597 >> 172.21.15.78:0/2221647597 conn(0x7fcd44101a30 msgr2=0x7fcd44103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:45:50.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.001+0000 7fcd49200700 1 -- 172.21.15.78:0/2221647597 shutdown_connections 2024-03-14T09:45:50.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.002+0000 7fcd49200700 1 -- 172.21.15.78:0/2221647597 wait complete. 2024-03-14T09:45:50.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.002+0000 7fcd49200700 1 Processor -- start 2024-03-14T09:45:50.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.003+0000 7fcd49200700 1 -- start start 2024-03-14T09:45:50.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.003+0000 7fcd49200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd441068d0 0x7fcd4419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:45:50.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.003+0000 7fcd49200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4419aba0 con 0x7fcd441068d0 2024-03-14T09:45:50.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.003+0000 7fcd41e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd441068d0 0x7fcd4419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:45:50.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.003+0000 7fcd41e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd441068d0 0x7fcd4419a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37370/0 (socket says 172.21.15.78:37370) 2024-03-14T09:45:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.003+0000 7fcd41e00700 1 -- 172.21.15.78:0/526823231 learned_addr learned my addr 172.21.15.78:0/526823231 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:45:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.003+0000 7fcd41e00700 1 -- 172.21.15.78:0/526823231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd34009550 con 0x7fcd441068d0 2024-03-14T09:45:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.004+0000 7fcd41e00700 1 --2- 172.21.15.78:0/526823231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd441068d0 0x7fcd4419a660 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcd34009130 tx=0x7fcd34011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:45:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.004+0000 7fcd3aa00700 1 -- 172.21.15.78:0/526823231 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd34003ab0 con 0x7fcd441068d0 2024-03-14T09:45:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.004+0000 7fcd3aa00700 1 -- 172.21.15.78:0/526823231 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd34003c10 con 0x7fcd441068d0 2024-03-14T09:45:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.004+0000 7fcd49200700 1 -- 172.21.15.78:0/526823231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd4419ada0 con 0x7fcd441068d0 2024-03-14T09:45:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.004+0000 7fcd3aa00700 1 -- 172.21.15.78:0/526823231 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd340247d0 con 0x7fcd441068d0 2024-03-14T09:45:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.004+0000 7fcd49200700 1 -- 172.21.15.78:0/526823231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd4419b240 con 0x7fcd441068d0 2024-03-14T09:45:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.005+0000 7fcd3aa00700 1 -- 172.21.15.78:0/526823231 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcd34024d00 con 0x7fcd441068d0 2024-03-14T09:45:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.005+0000 7fcd3aa00700 1 -- 172.21.15.78:0/526823231 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcd340193c0 con 0x7fcd441068d0 2024-03-14T09:45:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.005+0000 7fcd49200700 1 -- 172.21.15.78:0/526823231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd4410aad0 con 0x7fcd441068d0 2024-03-14T09:45:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.008+0000 7fcd3aa00700 1 -- 172.21.15.78:0/526823231 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fcd3401abb0 con 0x7fcd441068d0 2024-03-14T09:45:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.082+0000 7fcd49200700 1 -- 172.21.15.78:0/526823231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcd4410acc0 con 0x7fcd441068d0 2024-03-14T09:45:50.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.083+0000 7fcd3aa00700 1 -- 172.21.15.78:0/526823231 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fcd34035090 con 0x7fcd441068d0 2024-03-14T09:45:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.084+0000 7fcd49200700 1 -- 172.21.15.78:0/526823231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd441068d0 msgr2=0x7fcd4419a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:45:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.084+0000 7fcd49200700 1 --2- 172.21.15.78:0/526823231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd441068d0 0x7fcd4419a660 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcd34009130 tx=0x7fcd34011ad0 comp rx=0 tx=0).stop 2024-03-14T09:45:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.084+0000 7fcd49200700 1 -- 172.21.15.78:0/526823231 shutdown_connections 2024-03-14T09:45:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.084+0000 7fcd49200700 1 --2- 172.21.15.78:0/526823231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd441068d0 0x7fcd4419a660 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:45:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.084+0000 7fcd49200700 1 -- 172.21.15.78:0/526823231 >> 172.21.15.78:0/526823231 conn(0x7fcd44101a30 msgr2=0x7fcd44102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:45:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.084+0000 7fcd49200700 1 -- 172.21.15.78:0/526823231 shutdown_connections 2024-03-14T09:45:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:50.084+0000 7fcd49200700 1 -- 172.21.15.78:0/526823231 wait complete. 2024-03-14T09:45:50.438 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (1/15)... 2024-03-14T09:45:50.803 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:50 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:50.405+0000 7f01c24f8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-14T09:45:51.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:51 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:51.252+0000 7f01c24f8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-14T09:45:51.993 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:51 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:51.557+0000 7f01c24f8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-14T09:45:51.993 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:51 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:51.664+0000 7f01c24f8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-14T09:45:51.993 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:51 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:51.800+0000 7f01c24f8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-14T09:45:51.993 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:51 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:51.992+0000 7f01c24f8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-14T09:45:52.266 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:52 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:52.156+0000 7f01c24f8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-14T09:45:52.538 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:52 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:52.265+0000 7f01c24f8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-14T09:45:52.803 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:52 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:52.758+0000 7f01c24f8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-14T09:45:53.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-14T09:45:53.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-14T09:45:53.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "83662c4e-e1e7-11ee-95c9-87774f69a715", 2024-03-14T09:45:53.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-03-14T09:45:53.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-14T09:45:53.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-14T09:45:53.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-14T09:45:53.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:45:53.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-14T09:45:53.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-14T09:45:53.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-03-14T09:45:53.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-14T09:45:53.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-14T09:45:53.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-03-14T09:45:53.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-14T09:45:53.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-14T09:45:53.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-14T09:45:53.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T09:45:53.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-14T09:45:53.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-14T09:45:53.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:45:53.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-14T09:45:53.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T09:45:53.279 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-14T09:45:53.279 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-14T09:45:53.279 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-14T09:45:53.286 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-14T09:45:53.286 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-14T09:45:53.286 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-14T09:45:53.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:45:53.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-14T09:45:53.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-14T09:45:53.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-14T09:45:53.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-14T09:45:53.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-14T09:45:53.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-14T09:45:53.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-14T09:45:53.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-14T09:45:53.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-14T09:45:53.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:45:53.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-14T09:45:53.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T09:45:53.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-14T09:45:53.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-14T09:45:53.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:45:53.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-14T09:45:53.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-14T09:45:53.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-14T09:45:53.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-14T09:45:53.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-14T09:45:53.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-14T09:45:53.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-03-14T09:45:53.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-14T09:45:53.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-14T09:45:53.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:45:53.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-14T09:45:53.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T09:45:53.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-03-14T09:45:43.515620+0000", 2024-03-14T09:45:53.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-14T09:45:53.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:45:53.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-14T09:45:53.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-14T09:45:53.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.849+0000 7fba49e00700 1 Processor -- start 2024-03-14T09:45:53.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.849+0000 7fba49e00700 1 -- start start 2024-03-14T09:45:53.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.850+0000 7fba49e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4407a310 0x7fba4407a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:45:53.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.850+0000 7fba49e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4407ad00 con 0x7fba4407a310 2024-03-14T09:45:53.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.850+0000 7fba42a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4407a310 0x7fba4407a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:45:53.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.850+0000 7fba42a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4407a310 0x7fba4407a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37380/0 (socket says 172.21.15.78:37380) 2024-03-14T09:45:53.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.850+0000 7fba42a00700 1 -- 172.21.15.78:0/334208798 learned_addr learned my addr 172.21.15.78:0/334208798 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:45:53.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.851+0000 7fba42a00700 1 -- 172.21.15.78:0/334208798 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4407ae40 con 0x7fba4407a310 2024-03-14T09:45:53.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.851+0000 7fba42a00700 1 --2- 172.21.15.78:0/334208798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4407a310 0x7fba4407a730 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fba340098a0 tx=0x7fba34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9f6863227f774d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:45:53.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.851+0000 7fba41600700 1 -- 172.21.15.78:0/334208798 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba34004160 con 0x7fba4407a310 2024-03-14T09:45:53.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.851+0000 7fba41600700 1 -- 172.21.15.78:0/334208798 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba340042c0 con 0x7fba4407a310 2024-03-14T09:45:53.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.851+0000 7fba41600700 1 -- 172.21.15.78:0/334208798 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba34011a00 con 0x7fba4407a310 2024-03-14T09:45:53.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.852+0000 7fba49e00700 1 -- 172.21.15.78:0/334208798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4407a310 msgr2=0x7fba4407a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:45:53.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.852+0000 7fba49e00700 1 --2- 172.21.15.78:0/334208798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4407a310 0x7fba4407a730 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fba340098a0 tx=0x7fba34009bb0 comp rx=0 tx=0).stop 2024-03-14T09:45:53.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.852+0000 7fba49e00700 1 -- 172.21.15.78:0/334208798 shutdown_connections 2024-03-14T09:45:53.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.852+0000 7fba49e00700 1 --2- 172.21.15.78:0/334208798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4407a310 0x7fba4407a730 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:45:53.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.852+0000 7fba49e00700 1 -- 172.21.15.78:0/334208798 >> 172.21.15.78:0/334208798 conn(0x7fba44101a30 msgr2=0x7fba44103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:45:53.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.852+0000 7fba49e00700 1 -- 172.21.15.78:0/334208798 shutdown_connections 2024-03-14T09:45:53.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.852+0000 7fba49e00700 1 -- 172.21.15.78:0/334208798 wait complete. 2024-03-14T09:45:53.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.853+0000 7fba49e00700 1 Processor -- start 2024-03-14T09:45:53.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.853+0000 7fba49e00700 1 -- start start 2024-03-14T09:45:53.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.853+0000 7fba49e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4407a310 0x7fba4419c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:45:53.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.853+0000 7fba49e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4419cdd0 con 0x7fba4407a310 2024-03-14T09:45:53.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.854+0000 7fba42a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4407a310 0x7fba4419c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:45:53.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.854+0000 7fba42a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4407a310 0x7fba4419c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37382/0 (socket says 172.21.15.78:37382) 2024-03-14T09:45:53.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.854+0000 7fba42a00700 1 -- 172.21.15.78:0/1583982780 learned_addr learned my addr 172.21.15.78:0/1583982780 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:45:53.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.854+0000 7fba42a00700 1 -- 172.21.15.78:0/1583982780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba34009550 con 0x7fba4407a310 2024-03-14T09:45:53.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.854+0000 7fba42a00700 1 --2- 172.21.15.78:0/1583982780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4407a310 0x7fba4419c890 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fba34009130 tx=0x7fba34011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:45:53.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.854+0000 7fba48a00700 1 -- 172.21.15.78:0/1583982780 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba34003ab0 con 0x7fba4407a310 2024-03-14T09:45:53.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.854+0000 7fba48a00700 1 -- 172.21.15.78:0/1583982780 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba34003c10 con 0x7fba4407a310 2024-03-14T09:45:53.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.854+0000 7fba48a00700 1 -- 172.21.15.78:0/1583982780 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba340247d0 con 0x7fba4407a310 2024-03-14T09:45:53.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.854+0000 7fba49e00700 1 -- 172.21.15.78:0/1583982780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba4419cfd0 con 0x7fba4407a310 2024-03-14T09:45:53.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.855+0000 7fba49e00700 1 -- 172.21.15.78:0/1583982780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba4419d470 con 0x7fba4407a310 2024-03-14T09:45:53.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.855+0000 7fba48a00700 1 -- 172.21.15.78:0/1583982780 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fba34024d50 con 0x7fba4407a310 2024-03-14T09:45:53.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.855+0000 7fba48a00700 1 -- 172.21.15.78:0/1583982780 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fba34019490 con 0x7fba4407a310 2024-03-14T09:45:53.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.856+0000 7fba49e00700 1 -- 172.21.15.78:0/1583982780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba4410cca0 con 0x7fba4407a310 2024-03-14T09:45:53.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.858+0000 7fba48a00700 1 -- 172.21.15.78:0/1583982780 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fba3401abb0 con 0x7fba4407a310 2024-03-14T09:45:53.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.933+0000 7fba49e00700 1 -- 172.21.15.78:0/1583982780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fba4410ce90 con 0x7fba4407a310 2024-03-14T09:45:53.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.934+0000 7fba48a00700 1 -- 172.21.15.78:0/1583982780 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fba3401ad50 con 0x7fba4407a310 2024-03-14T09:45:53.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.934+0000 7fba49e00700 1 -- 172.21.15.78:0/1583982780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4407a310 msgr2=0x7fba4419c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:45:53.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.934+0000 7fba49e00700 1 --2- 172.21.15.78:0/1583982780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4407a310 0x7fba4419c890 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fba34009130 tx=0x7fba34011ad0 comp rx=0 tx=0).stop 2024-03-14T09:45:53.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.935+0000 7fba49e00700 1 -- 172.21.15.78:0/1583982780 shutdown_connections 2024-03-14T09:45:53.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.935+0000 7fba49e00700 1 --2- 172.21.15.78:0/1583982780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba4407a310 0x7fba4419c890 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:45:53.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.935+0000 7fba49e00700 1 -- 172.21.15.78:0/1583982780 >> 172.21.15.78:0/1583982780 conn(0x7fba44101a30 msgr2=0x7fba441026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:45:53.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.935+0000 7fba49e00700 1 -- 172.21.15.78:0/1583982780 shutdown_connections 2024-03-14T09:45:53.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:52.935+0000 7fba49e00700 1 -- 172.21.15.78:0/1583982780 wait complete. 2024-03-14T09:45:53.292 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (2/15)... 2024-03-14T09:45:53.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:53 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:53.264+0000 7f01c24f8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-14T09:45:53.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:53 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:53.510+0000 7f01c24f8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-14T09:45:54.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:53 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1583982780' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-14T09:45:54.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:53 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:53.632+0000 7f01c24f8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-14T09:45:54.875 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:54 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:54.565+0000 7f01c24f8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-14T09:45:54.875 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:54 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:54.687+0000 7f01c24f8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-14T09:45:55.303 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:54 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:54.874+0000 7f01c24f8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-14T09:45:56.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:55 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:55.693+0000 7f01c24f8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-14T09:45:56.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:55 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:55.933+0000 7f01c24f8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-14T09:45:56.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:56 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:56.186+0000 7f01c24f8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-14T09:45:57.589 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:57 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:57.336+0000 7f01c24f8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-14T09:45:58.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:57 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:57.715+0000 7f01c24f8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-14T09:45:58.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:57 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:57.831+0000 7f01c24f8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-14T09:45:58.554 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:45:58 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:45:58.081+0000 7f01c24f8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-14T09:45:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:58 smithi078 ceph-mon[38143]: Activating manager daemon x 2024-03-14T09:45:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:58 smithi078 ceph-mon[38143]: mgrmap e2: x(active, starting, since 0.00405182s) 2024-03-14T09:45:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:58 smithi078 ceph-mon[38143]: from='mgr.14100 172.21.15.78:0/4220403310' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-14T09:45:59.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:58 smithi078 ceph-mon[38143]: from='mgr.14100 172.21.15.78:0/4220403310' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-14T09:45:59.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:58 smithi078 ceph-mon[38143]: from='mgr.14100 172.21.15.78:0/4220403310' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-14T09:45:59.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:58 smithi078 ceph-mon[38143]: from='mgr.14100 172.21.15.78:0/4220403310' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:45:59.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:58 smithi078 ceph-mon[38143]: from='mgr.14100 172.21.15.78:0/4220403310' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-14T09:45:59.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:58 smithi078 ceph-mon[38143]: Manager daemon x is now available 2024-03-14T09:45:59.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:58 smithi078 ceph-mon[38143]: from='mgr.14100 172.21.15.78:0/4220403310' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-14T09:45:59.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:58 smithi078 ceph-mon[38143]: from='mgr.14100 172.21.15.78:0/4220403310' entity='mgr.x' 2024-03-14T09:45:59.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:58 smithi078 ceph-mon[38143]: from='mgr.14100 172.21.15.78:0/4220403310' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-14T09:45:59.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:58 smithi078 ceph-mon[38143]: from='mgr.14100 172.21.15.78:0/4220403310' entity='mgr.x' 2024-03-14T09:45:59.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:58 smithi078 ceph-mon[38143]: from='mgr.14100 172.21.15.78:0/4220403310' entity='mgr.x' 2024-03-14T09:46:00.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:59 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/510397987' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-14T09:46:00.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:45:59 smithi078 ceph-mon[38143]: mgrmap e3: x(active, since 1.00845s) 2024-03-14T09:46:02.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:01 smithi078 ceph-mon[38143]: mgrmap e4: x(active, since 2s) 2024-03-14T09:46:03.204 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-14T09:46:03.205 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-14T09:46:03.205 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "83662c4e-e1e7-11ee-95c9-87774f69a715", 2024-03-14T09:46:03.205 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-03-14T09:46:03.205 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-14T09:46:03.205 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-14T09:46:03.207 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-14T09:46:03.207 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:46:03.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-14T09:46:03.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-14T09:46:03.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-03-14T09:46:03.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-14T09:46:03.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-14T09:46:03.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-03-14T09:46:03.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-14T09:46:03.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-03-14T09:46:03.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-14T09:46:03.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T09:46:03.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-14T09:46:03.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-14T09:46:03.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:46:03.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-14T09:46:03.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T09:46:03.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-14T09:46:03.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-14T09:46:03.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-14T09:46:03.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-14T09:46:03.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-14T09:46:03.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-14T09:46:03.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:46:03.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-14T09:46:03.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-14T09:46:03.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-14T09:46:03.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-14T09:46:03.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-14T09:46:03.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-14T09:46:03.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-14T09:46:03.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-14T09:46:03.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-14T09:46:03.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:46:03.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-14T09:46:03.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T09:46:03.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-14T09:46:03.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-14T09:46:03.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:46:03.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-14T09:46:03.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-14T09:46:03.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-14T09:46:03.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-14T09:46:03.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-14T09:46:03.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-14T09:46:03.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-03-14T09:46:03.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-14T09:46:03.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-14T09:46:03.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:46:03.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-14T09:46:03.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T09:46:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-03-14T09:45:43.515620+0000", 2024-03-14T09:46:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-14T09:46:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:46:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-14T09:46:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-14T09:46:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.552+0000 7f5be0a00700 1 Processor -- start 2024-03-14T09:46:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.552+0000 7f5be0a00700 1 -- start start 2024-03-14T09:46:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.552+0000 7f5be0a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bdc106720 0x7f5bdc107500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.552+0000 7f5be0a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc106bd0 con 0x7f5bdc106720 2024-03-14T09:46:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.552+0000 7f5bda800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bdc106720 0x7f5bdc107500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.552+0000 7f5bda800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bdc106720 0x7f5bdc107500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34142/0 (socket says 172.21.15.78:34142) 2024-03-14T09:46:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.552+0000 7f5bda800700 1 -- 172.21.15.78:0/1282269835 learned_addr learned my addr 172.21.15.78:0/1282269835 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.553+0000 7f5bda800700 1 -- 172.21.15.78:0/1282269835 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc106d10 con 0x7f5bdc106720 2024-03-14T09:46:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.553+0000 7f5bda800700 1 --2- 172.21.15.78:0/1282269835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bdc106720 0x7f5bdc107500 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0098a0 tx=0x7f5bcc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a8c768eed47cbff server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.553+0000 7f5bd9400700 1 -- 172.21.15.78:0/1282269835 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bcc004160 con 0x7f5bdc106720 2024-03-14T09:46:03.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.553+0000 7f5bd9400700 1 -- 172.21.15.78:0/1282269835 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5bcc0042c0 con 0x7f5bdc106720 2024-03-14T09:46:03.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.553+0000 7f5bd9400700 1 -- 172.21.15.78:0/1282269835 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bcc011a00 con 0x7f5bdc106720 2024-03-14T09:46:03.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.554+0000 7f5be0a00700 1 -- 172.21.15.78:0/1282269835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bdc106720 msgr2=0x7f5bdc107500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:03.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.554+0000 7f5be0a00700 1 --2- 172.21.15.78:0/1282269835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bdc106720 0x7f5bdc107500 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0098a0 tx=0x7f5bcc009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:03.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.554+0000 7f5be0a00700 1 -- 172.21.15.78:0/1282269835 shutdown_connections 2024-03-14T09:46:03.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.554+0000 7f5be0a00700 1 --2- 172.21.15.78:0/1282269835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bdc106720 0x7f5bdc107500 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:03.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.554+0000 7f5be0a00700 1 -- 172.21.15.78:0/1282269835 >> 172.21.15.78:0/1282269835 conn(0x7f5bdc1018c0 msgr2=0x7f5bdc103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:03.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.554+0000 7f5be0a00700 1 -- 172.21.15.78:0/1282269835 shutdown_connections 2024-03-14T09:46:03.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.554+0000 7f5be0a00700 1 -- 172.21.15.78:0/1282269835 wait complete. 2024-03-14T09:46:03.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.555+0000 7f5be0a00700 1 Processor -- start 2024-03-14T09:46:03.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.555+0000 7f5be0a00700 1 -- start start 2024-03-14T09:46:03.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.555+0000 7f5be0a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bdc106720 0x7f5bdc19a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:03.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.555+0000 7f5be0a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc19aa20 con 0x7f5bdc106720 2024-03-14T09:46:03.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.556+0000 7f5bda800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bdc106720 0x7f5bdc19a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:03.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.556+0000 7f5bda800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bdc106720 0x7f5bdc19a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34148/0 (socket says 172.21.15.78:34148) 2024-03-14T09:46:03.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.556+0000 7f5bda800700 1 -- 172.21.15.78:0/510397987 learned_addr learned my addr 172.21.15.78:0/510397987 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:03.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.556+0000 7f5bda800700 1 -- 172.21.15.78:0/510397987 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bcc009550 con 0x7f5bdc106720 2024-03-14T09:46:03.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.556+0000 7f5bda800700 1 --2- 172.21.15.78:0/510397987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bdc106720 0x7f5bdc19a4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc011710 tx=0x7f5bcc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:03.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.556+0000 7f5bd3400700 1 -- 172.21.15.78:0/510397987 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bcc003aa0 con 0x7f5bdc106720 2024-03-14T09:46:03.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.556+0000 7f5bd3400700 1 -- 172.21.15.78:0/510397987 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5bcc003c00 con 0x7f5bdc106720 2024-03-14T09:46:03.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.556+0000 7f5be0a00700 1 -- 172.21.15.78:0/510397987 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bdc19ac20 con 0x7f5bdc106720 2024-03-14T09:46:03.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.556+0000 7f5bd3400700 1 -- 172.21.15.78:0/510397987 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bcc024800 con 0x7f5bdc106720 2024-03-14T09:46:03.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.557+0000 7f5be0a00700 1 -- 172.21.15.78:0/510397987 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bdc19b0c0 con 0x7f5bdc106720 2024-03-14T09:46:03.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.558+0000 7f5be0a00700 1 -- 172.21.15.78:0/510397987 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bdc10a920 con 0x7f5bdc106720 2024-03-14T09:46:03.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.558+0000 7f5bd3400700 1 -- 172.21.15.78:0/510397987 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 2) v1 ==== 45130+0+0 (secure 0 0 0) 0x7f5bcc01abd0 con 0x7f5bdc106720 2024-03-14T09:46:03.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.558+0000 7f5bd3400700 1 -- 172.21.15.78:0/510397987 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5bcc04bbf0 con 0x7f5bdc106720 2024-03-14T09:46:03.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.560+0000 7f5bd3400700 1 -- 172.21.15.78:0/510397987 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f5bcc018760 con 0x7f5bdc106720 2024-03-14T09:46:03.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.634+0000 7f5be0a00700 1 -- 172.21.15.78:0/510397987 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5bdc19d9b0 con 0x7f5bdc106720 2024-03-14T09:46:03.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.635+0000 7f5bd3400700 1 -- 172.21.15.78:0/510397987 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f5bcc018350 con 0x7f5bdc106720 2024-03-14T09:46:03.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.636+0000 7f5be0a00700 1 -- 172.21.15.78:0/510397987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bdc106720 msgr2=0x7f5bdc19a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:03.222 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.636+0000 7f5be0a00700 1 --2- 172.21.15.78:0/510397987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bdc106720 0x7f5bdc19a4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc011710 tx=0x7f5bcc011ad0 comp rx=0 tx=0).stop 2024-03-14T09:46:03.222 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.636+0000 7f5be0a00700 1 -- 172.21.15.78:0/510397987 shutdown_connections 2024-03-14T09:46:03.222 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.636+0000 7f5be0a00700 1 --2- 172.21.15.78:0/510397987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bdc106720 0x7f5bdc19a4e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:03.222 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.636+0000 7f5be0a00700 1 -- 172.21.15.78:0/510397987 >> 172.21.15.78:0/510397987 conn(0x7f5bdc1018c0 msgr2=0x7f5bdc102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:03.222 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.636+0000 7f5be0a00700 1 -- 172.21.15.78:0/510397987 shutdown_connections 2024-03-14T09:46:03.222 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:45:58.636+0000 7f5be0a00700 1 -- 172.21.15.78:0/510397987 wait complete. 2024-03-14T09:46:03.222 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (3/15)... 2024-03-14T09:46:06.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:06 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2738736681' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-14T09:46:06.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-14T09:46:06.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-14T09:46:06.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "83662c4e-e1e7-11ee-95c9-87774f69a715", 2024-03-14T09:46:06.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2024-03-14T09:46:06.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-14T09:46:06.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-14T09:46:06.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-14T09:46:06.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:46:06.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-14T09:46:06.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-14T09:46:06.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2024-03-14T09:46:06.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-14T09:46:06.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-14T09:46:06.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2024-03-14T09:46:06.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-14T09:46:06.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 18, 2024-03-14T09:46:06.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-14T09:46:06.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T09:46:06.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-14T09:46:06.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-14T09:46:06.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:46:06.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-14T09:46:06.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T09:46:06.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-14T09:46:06.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-14T09:46:06.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-14T09:46:06.286 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-14T09:46:06.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-14T09:46:06.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-14T09:46:06.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:46:06.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-14T09:46:06.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-14T09:46:06.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-14T09:46:06.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-14T09:46:06.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-14T09:46:06.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-14T09:46:06.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-14T09:46:06.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-14T09:46:06.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-14T09:46:06.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:46:06.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-14T09:46:06.287 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T09:46:06.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-14T09:46:06.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-14T09:46:06.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:46:06.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-14T09:46:06.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-14T09:46:06.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-14T09:46:06.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-14T09:46:06.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-14T09:46:06.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-14T09:46:06.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2024-03-14T09:46:06.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2024-03-14T09:46:06.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-14T09:46:06.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:46:06.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-14T09:46:06.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T09:46:06.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2024-03-14T09:45:43.515620+0000", 2024-03-14T09:46:06.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-14T09:46:06.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2024-03-14T09:46:06.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-14T09:46:06.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-14T09:46:06.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.666+0000 7f6ed7e00700 1 Processor -- start 2024-03-14T09:46:06.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.666+0000 7f6ed7e00700 1 -- start start 2024-03-14T09:46:06.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.666+0000 7f6ed7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ed807a2d0 0x7f6ed807a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:06.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.666+0000 7f6ed7e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed807acc0 con 0x7f6ed807a2d0 2024-03-14T09:46:06.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.667+0000 7f6ed6a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ed807a2d0 0x7f6ed807a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:06.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.667+0000 7f6ed6a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ed807a2d0 0x7f6ed807a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51768/0 (socket says 172.21.15.78:51768) 2024-03-14T09:46:06.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.667+0000 7f6ed6a00700 1 -- 172.21.15.78:0/1306307213 learned_addr learned my addr 172.21.15.78:0/1306307213 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:06.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.667+0000 7f6ed6a00700 1 -- 172.21.15.78:0/1306307213 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed807ae00 con 0x7f6ed807a2d0 2024-03-14T09:46:06.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.667+0000 7f6ed6a00700 1 --2- 172.21.15.78:0/1306307213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ed807a2d0 0x7f6ed807a6f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6ec80098a0 tx=0x7f6ec8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9554559b525e550 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:06.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.667+0000 7f6ed5600700 1 -- 172.21.15.78:0/1306307213 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ec8004160 con 0x7f6ed807a2d0 2024-03-14T09:46:06.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.667+0000 7f6ed5600700 1 -- 172.21.15.78:0/1306307213 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ec80042c0 con 0x7f6ed807a2d0 2024-03-14T09:46:06.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.667+0000 7f6ed5600700 1 -- 172.21.15.78:0/1306307213 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ec8011a00 con 0x7f6ed807a2d0 2024-03-14T09:46:06.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.668+0000 7f6ed7e00700 1 -- 172.21.15.78:0/1306307213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ed807a2d0 msgr2=0x7f6ed807a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:06.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.668+0000 7f6ed7e00700 1 --2- 172.21.15.78:0/1306307213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ed807a2d0 0x7f6ed807a6f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6ec80098a0 tx=0x7f6ec8009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:06.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.668+0000 7f6ed7e00700 1 -- 172.21.15.78:0/1306307213 shutdown_connections 2024-03-14T09:46:06.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.668+0000 7f6ed7e00700 1 --2- 172.21.15.78:0/1306307213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ed807a2d0 0x7f6ed807a6f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:06.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.668+0000 7f6ed7e00700 1 -- 172.21.15.78:0/1306307213 >> 172.21.15.78:0/1306307213 conn(0x7f6ed8101890 msgr2=0x7f6ed8103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:06.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.668+0000 7f6ed7e00700 1 -- 172.21.15.78:0/1306307213 shutdown_connections 2024-03-14T09:46:06.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.668+0000 7f6ed7e00700 1 -- 172.21.15.78:0/1306307213 wait complete. 2024-03-14T09:46:06.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.669+0000 7f6ed7e00700 1 Processor -- start 2024-03-14T09:46:06.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.669+0000 7f6ed7e00700 1 -- start start 2024-03-14T09:46:06.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.670+0000 7f6ed7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ed807a2d0 0x7f6ed819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:06.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.670+0000 7f6ed7e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed819cbe0 con 0x7f6ed807a2d0 2024-03-14T09:46:06.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.670+0000 7f6ed6a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ed807a2d0 0x7f6ed819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:06.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.670+0000 7f6ed6a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ed807a2d0 0x7f6ed819c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51772/0 (socket says 172.21.15.78:51772) 2024-03-14T09:46:06.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.670+0000 7f6ed6a00700 1 -- 172.21.15.78:0/2738736681 learned_addr learned my addr 172.21.15.78:0/2738736681 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:06.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.670+0000 7f6ed6a00700 1 -- 172.21.15.78:0/2738736681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec8009550 con 0x7f6ed807a2d0 2024-03-14T09:46:06.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.670+0000 7f6ed6a00700 1 --2- 172.21.15.78:0/2738736681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ed807a2d0 0x7f6ed819c6a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8011a20 tx=0x7f6ec8011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:06.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.670+0000 7f6ecf400700 1 -- 172.21.15.78:0/2738736681 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ec8003af0 con 0x7f6ed807a2d0 2024-03-14T09:46:06.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.671+0000 7f6ecf400700 1 -- 172.21.15.78:0/2738736681 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ec8003c50 con 0x7f6ed807a2d0 2024-03-14T09:46:06.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.671+0000 7f6ecf400700 1 -- 172.21.15.78:0/2738736681 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6ec8019760 con 0x7f6ed807a2d0 2024-03-14T09:46:06.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.671+0000 7f6ed7e00700 1 -- 172.21.15.78:0/2738736681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed819cde0 con 0x7f6ed807a2d0 2024-03-14T09:46:06.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.671+0000 7f6ed7e00700 1 -- 172.21.15.78:0/2738736681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed819d280 con 0x7f6ed807a2d0 2024-03-14T09:46:06.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.672+0000 7f6ecf400700 1 -- 172.21.15.78:0/2738736681 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 45363+0+0 (secure 0 0 0) 0x7f6ec801ab20 con 0x7f6ed807a2d0 2024-03-14T09:46:06.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.672+0000 7f6ed7e00700 1 -- 172.21.15.78:0/2738736681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed810cac0 con 0x7f6ed807a2d0 2024-03-14T09:46:06.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.672+0000 7f6ecf400700 1 --2- 172.21.15.78:0/2738736681 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f6ebc038480 0x7f6ebc03a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:06.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.672+0000 7f6ecf400700 1 -- 172.21.15.78:0/2738736681 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6ec804c5f0 con 0x7f6ed807a2d0 2024-03-14T09:46:06.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.672+0000 7f6ed6000700 1 --2- 172.21.15.78:0/2738736681 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f6ebc038480 0x7f6ebc03a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:06.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.673+0000 7f6ed6000700 1 --2- 172.21.15.78:0/2738736681 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f6ebc038480 0x7f6ebc03a940 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4006fd0 tx=0x7f6ec4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:06.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.678+0000 7f6ecf400700 1 -- 172.21.15.78:0/2738736681 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6ec801a7b0 con 0x7f6ed807a2d0 2024-03-14T09:46:06.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.953+0000 7f6ed7e00700 1 -- 172.21.15.78:0/2738736681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6ed810cd00 con 0x7f6ed807a2d0 2024-03-14T09:46:06.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.953+0000 7f6ecf400700 1 -- 172.21.15.78:0/2738736681 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f6ec8023370 con 0x7f6ed807a2d0 2024-03-14T09:46:06.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.956+0000 7f6ed7e00700 1 -- 172.21.15.78:0/2738736681 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f6ebc038480 msgr2=0x7f6ebc03a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:06.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.956+0000 7f6ed7e00700 1 --2- 172.21.15.78:0/2738736681 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f6ebc038480 0x7f6ebc03a940 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4006fd0 tx=0x7f6ec4006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:06.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.956+0000 7f6ed7e00700 1 -- 172.21.15.78:0/2738736681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ed807a2d0 msgr2=0x7f6ed819c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:06.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.956+0000 7f6ed7e00700 1 --2- 172.21.15.78:0/2738736681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ed807a2d0 0x7f6ed819c6a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8011a20 tx=0x7f6ec8011b00 comp rx=0 tx=0).stop 2024-03-14T09:46:06.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.956+0000 7f6ed7e00700 1 -- 172.21.15.78:0/2738736681 shutdown_connections 2024-03-14T09:46:06.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.956+0000 7f6ed7e00700 1 --2- 172.21.15.78:0/2738736681 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f6ebc038480 0x7f6ebc03a940 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:06.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.956+0000 7f6ed7e00700 1 --2- 172.21.15.78:0/2738736681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ed807a2d0 0x7f6ed819c6a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:06.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.956+0000 7f6ed7e00700 1 -- 172.21.15.78:0/2738736681 >> 172.21.15.78:0/2738736681 conn(0x7f6ed8101890 msgr2=0x7f6ed81024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:06.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.956+0000 7f6ed7e00700 1 -- 172.21.15.78:0/2738736681 shutdown_connections 2024-03-14T09:46:06.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:05.956+0000 7f6ed7e00700 1 -- 172.21.15.78:0/2738736681 wait complete. 2024-03-14T09:46:06.292 INFO:teuthology.orchestra.run.smithi078.stdout:mgr is available 2024-03-14T09:46:07.199 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:07 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3061173500' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-14T09:46:07.206 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-14T09:46:07.206 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [global] 2024-03-14T09:46:07.206 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout fsid = 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:46:07.206 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-14T09:46:07.206 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-14T09:46:07.206 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-14T09:46:07.206 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-14T09:46:07.207 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-14T09:46:07.207 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-14T09:46:07.207 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-14T09:46:07.207 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2024-03-14T09:46:07.207 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [osd] 2024-03-14T09:46:07.207 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-14T09:46:07.207 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-14T09:46:07.207 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-14T09:46:07.207 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-14T09:46:07.207 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-14T09:46:07.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.649+0000 7f3f63c00700 1 Processor -- start 2024-03-14T09:46:07.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.649+0000 7f3f63c00700 1 -- start start 2024-03-14T09:46:07.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.650+0000 7f3f63c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f5c07a2f0 0x7f3f5c07a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:07.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.650+0000 7f3f63c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c07ace0 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.650+0000 7f3f61200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f5c07a2f0 0x7f3f5c07a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:07.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.650+0000 7f3f61200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f5c07a2f0 0x7f3f5c07a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51778/0 (socket says 172.21.15.78:51778) 2024-03-14T09:46:07.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.650+0000 7f3f61200700 1 -- 172.21.15.78:0/3624088729 learned_addr learned my addr 172.21.15.78:0/3624088729 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:07.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.651+0000 7f3f61200700 1 -- 172.21.15.78:0/3624088729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5c07ae20 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.651+0000 7f3f61200700 1 --2- 172.21.15.78:0/3624088729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f5c07a2f0 0x7f3f5c07a710 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3f500098a0 tx=0x7f3f50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fbf5072737a3797 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:07.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.651+0000 7f3f5b400700 1 -- 172.21.15.78:0/3624088729 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f50004160 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.651+0000 7f3f5b400700 1 -- 172.21.15.78:0/3624088729 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f500042c0 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.651+0000 7f3f5b400700 1 -- 172.21.15.78:0/3624088729 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f50011a00 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.652+0000 7f3f63c00700 1 -- 172.21.15.78:0/3624088729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f5c07a2f0 msgr2=0x7f3f5c07a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:07.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.652+0000 7f3f63c00700 1 --2- 172.21.15.78:0/3624088729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f5c07a2f0 0x7f3f5c07a710 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3f500098a0 tx=0x7f3f50009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:07.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.652+0000 7f3f63c00700 1 -- 172.21.15.78:0/3624088729 shutdown_connections 2024-03-14T09:46:07.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.652+0000 7f3f63c00700 1 --2- 172.21.15.78:0/3624088729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f5c07a2f0 0x7f3f5c07a710 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:07.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.652+0000 7f3f63c00700 1 -- 172.21.15.78:0/3624088729 >> 172.21.15.78:0/3624088729 conn(0x7f3f5c101a20 msgr2=0x7f3f5c103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:07.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.652+0000 7f3f63c00700 1 -- 172.21.15.78:0/3624088729 shutdown_connections 2024-03-14T09:46:07.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.652+0000 7f3f63c00700 1 -- 172.21.15.78:0/3624088729 wait complete. 2024-03-14T09:46:07.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.653+0000 7f3f63c00700 1 Processor -- start 2024-03-14T09:46:07.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.653+0000 7f3f63c00700 1 -- start start 2024-03-14T09:46:07.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.653+0000 7f3f63c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f5c07a2f0 0x7f3f5c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:07.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.653+0000 7f3f63c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c19cd70 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.210 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.654+0000 7f3f61200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f5c07a2f0 0x7f3f5c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:07.210 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.654+0000 7f3f61200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f5c07a2f0 0x7f3f5c19c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51786/0 (socket says 172.21.15.78:51786) 2024-03-14T09:46:07.210 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.654+0000 7f3f61200700 1 -- 172.21.15.78:0/3061173500 learned_addr learned my addr 172.21.15.78:0/3061173500 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:07.210 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.654+0000 7f3f61200700 1 -- 172.21.15.78:0/3061173500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f50009550 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.210 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.654+0000 7f3f61200700 1 --2- 172.21.15.78:0/3061173500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f5c07a2f0 0x7f3f5c19c830 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3f50009fb0 tx=0x7f3f50011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:07.210 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.654+0000 7f3f59600700 1 -- 172.21.15.78:0/3061173500 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f50003a40 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.210 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.654+0000 7f3f59600700 1 -- 172.21.15.78:0/3061173500 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f50003ba0 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.210 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.654+0000 7f3f59600700 1 -- 172.21.15.78:0/3061173500 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f5001a750 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.210 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.654+0000 7f3f63c00700 1 -- 172.21.15.78:0/3061173500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f5c19cf70 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.210 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.655+0000 7f3f63c00700 1 -- 172.21.15.78:0/3061173500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5c19d410 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.210 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.655+0000 7f3f59600700 1 -- 172.21.15.78:0/3061173500 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 45363+0+0 (secure 0 0 0) 0x7f3f50024b60 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.211 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.656+0000 7f3f63c00700 1 -- 172.21.15.78:0/3061173500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f5c10ccb0 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.211 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.656+0000 7f3f59600700 1 --2- 172.21.15.78:0/3061173500 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f3f44038430 0x7f3f4403a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:07.211 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.656+0000 7f3f59600700 1 -- 172.21.15.78:0/3061173500 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3f5004c090 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.211 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.656+0000 7f3f5be00700 1 --2- 172.21.15.78:0/3061173500 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f3f44038430 0x7f3f4403a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:07.211 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.657+0000 7f3f5be00700 1 --2- 172.21.15.78:0/3061173500 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f3f44038430 0x7f3f4403a8f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3f48006fd0 tx=0x7f3f48006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:07.211 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.661+0000 7f3f59600700 1 -- 172.21.15.78:0/3061173500 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3f50018350 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.211 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.862+0000 7f3f63c00700 1 -- 172.21.15.78:0/3061173500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f3f5c10cef0 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.211 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.863+0000 7f3f59600700 1 -- 172.21.15.78:0/3061173500 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f3f50018350 con 0x7f3f5c07a2f0 2024-03-14T09:46:07.211 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.865+0000 7f3f63c00700 1 -- 172.21.15.78:0/3061173500 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f3f44038430 msgr2=0x7f3f4403a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:07.211 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.865+0000 7f3f63c00700 1 --2- 172.21.15.78:0/3061173500 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f3f44038430 0x7f3f4403a8f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3f48006fd0 tx=0x7f3f48006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:07.212 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.865+0000 7f3f63c00700 1 -- 172.21.15.78:0/3061173500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f5c07a2f0 msgr2=0x7f3f5c19c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:07.212 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.865+0000 7f3f63c00700 1 --2- 172.21.15.78:0/3061173500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f5c07a2f0 0x7f3f5c19c830 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3f50009fb0 tx=0x7f3f50011ad0 comp rx=0 tx=0).stop 2024-03-14T09:46:07.212 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.866+0000 7f3f63c00700 1 -- 172.21.15.78:0/3061173500 shutdown_connections 2024-03-14T09:46:07.212 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.866+0000 7f3f63c00700 1 --2- 172.21.15.78:0/3061173500 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f3f44038430 0x7f3f4403a8f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:07.212 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.866+0000 7f3f63c00700 1 --2- 172.21.15.78:0/3061173500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f5c07a2f0 0x7f3f5c19c830 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:07.212 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.866+0000 7f3f63c00700 1 -- 172.21.15.78:0/3061173500 >> 172.21.15.78:0/3061173500 conn(0x7f3f5c101a20 msgr2=0x7f3f5c102640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:07.212 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.866+0000 7f3f63c00700 1 -- 172.21.15.78:0/3061173500 shutdown_connections 2024-03-14T09:46:07.212 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:06.866+0000 7f3f63c00700 1 -- 172.21.15.78:0/3061173500 wait complete. 2024-03-14T09:46:07.212 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling cephadm module... 2024-03-14T09:46:08.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:08 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3243415092' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-14T09:46:08.951 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:08 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: ignoring --setuser ceph since I am not root 2024-03-14T09:46:08.951 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:08 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: ignoring --setgroup ceph since I am not root 2024-03-14T09:46:08.957 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.576+0000 7f7197e00700 1 Processor -- start 2024-03-14T09:46:08.957 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.576+0000 7f7197e00700 1 -- start start 2024-03-14T09:46:08.957 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.576+0000 7f7197e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719007a2d0 0x7f719007a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:08.957 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.576+0000 7f7197e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719007acc0 con 0x7f719007a2d0 2024-03-14T09:46:08.957 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.576+0000 7f7195400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719007a2d0 0x7f719007a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:08.957 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.576+0000 7f7195400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719007a2d0 0x7f719007a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51802/0 (socket says 172.21.15.78:51802) 2024-03-14T09:46:08.958 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.576+0000 7f7195400700 1 -- 172.21.15.78:0/1128868665 learned_addr learned my addr 172.21.15.78:0/1128868665 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:08.958 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.577+0000 7f7195400700 1 -- 172.21.15.78:0/1128868665 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719007ae00 con 0x7f719007a2d0 2024-03-14T09:46:08.958 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.577+0000 7f7195400700 1 --2- 172.21.15.78:0/1128868665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719007a2d0 0x7f719007a6f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f71840098a0 tx=0x7f7184009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56be9d741dad3f1b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:08.958 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.577+0000 7f718fe00700 1 -- 172.21.15.78:0/1128868665 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7184004160 con 0x7f719007a2d0 2024-03-14T09:46:08.958 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.577+0000 7f718fe00700 1 -- 172.21.15.78:0/1128868665 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f71840042c0 con 0x7f719007a2d0 2024-03-14T09:46:08.958 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.577+0000 7f718fe00700 1 -- 172.21.15.78:0/1128868665 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7184011a00 con 0x7f719007a2d0 2024-03-14T09:46:08.958 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.578+0000 7f7197e00700 1 -- 172.21.15.78:0/1128868665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719007a2d0 msgr2=0x7f719007a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:08.958 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.578+0000 7f7197e00700 1 --2- 172.21.15.78:0/1128868665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719007a2d0 0x7f719007a6f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f71840098a0 tx=0x7f7184009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:08.958 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.578+0000 7f7197e00700 1 -- 172.21.15.78:0/1128868665 shutdown_connections 2024-03-14T09:46:08.958 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.578+0000 7f7197e00700 1 --2- 172.21.15.78:0/1128868665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719007a2d0 0x7f719007a6f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:08.958 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.578+0000 7f7197e00700 1 -- 172.21.15.78:0/1128868665 >> 172.21.15.78:0/1128868665 conn(0x7f7190101a70 msgr2=0x7f7190103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:08.958 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.578+0000 7f7197e00700 1 -- 172.21.15.78:0/1128868665 shutdown_connections 2024-03-14T09:46:08.958 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.578+0000 7f7197e00700 1 -- 172.21.15.78:0/1128868665 wait complete. 2024-03-14T09:46:08.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.579+0000 7f7197e00700 1 Processor -- start 2024-03-14T09:46:08.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.579+0000 7f7197e00700 1 -- start start 2024-03-14T09:46:08.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.579+0000 7f7197e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719007a2d0 0x7f719019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:08.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.579+0000 7f7197e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719019cd70 con 0x7f719007a2d0 2024-03-14T09:46:08.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.580+0000 7f7195400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719007a2d0 0x7f719019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:08.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.580+0000 7f7195400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719007a2d0 0x7f719019c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51808/0 (socket says 172.21.15.78:51808) 2024-03-14T09:46:08.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.580+0000 7f7195400700 1 -- 172.21.15.78:0/3243415092 learned_addr learned my addr 172.21.15.78:0/3243415092 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:08.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.580+0000 7f7195400700 1 -- 172.21.15.78:0/3243415092 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7184009550 con 0x7f719007a2d0 2024-03-14T09:46:08.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.580+0000 7f7195400700 1 --2- 172.21.15.78:0/3243415092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719007a2d0 0x7f719019c830 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7184009130 tx=0x7f7184011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:08.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.580+0000 7f718e000700 1 -- 172.21.15.78:0/3243415092 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7184003b70 con 0x7f719007a2d0 2024-03-14T09:46:08.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.580+0000 7f718e000700 1 -- 172.21.15.78:0/3243415092 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7184003cd0 con 0x7f719007a2d0 2024-03-14T09:46:08.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.580+0000 7f7197e00700 1 -- 172.21.15.78:0/3243415092 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719019cf70 con 0x7f719007a2d0 2024-03-14T09:46:08.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.580+0000 7f718e000700 1 -- 172.21.15.78:0/3243415092 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71840197d0 con 0x7f719007a2d0 2024-03-14T09:46:08.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.581+0000 7f7197e00700 1 -- 172.21.15.78:0/3243415092 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719019d410 con 0x7f719007a2d0 2024-03-14T09:46:08.959 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.581+0000 7f7197e00700 1 -- 172.21.15.78:0/3243415092 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f719010cc80 con 0x7f719007a2d0 2024-03-14T09:46:08.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.582+0000 7f718e000700 1 -- 172.21.15.78:0/3243415092 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 45363+0+0 (secure 0 0 0) 0x7f718401abb0 con 0x7f719007a2d0 2024-03-14T09:46:08.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.582+0000 7f718e000700 1 --2- 172.21.15.78:0/3243415092 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f71780384d0 0x7f717803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:08.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.582+0000 7f718e000700 1 -- 172.21.15.78:0/3243415092 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f718404c5c0 con 0x7f719007a2d0 2024-03-14T09:46:08.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.582+0000 7f7194a00700 1 --2- 172.21.15.78:0/3243415092 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f71780384d0 0x7f717803a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:08.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.583+0000 7f7194a00700 1 --2- 172.21.15.78:0/3243415092 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f71780384d0 0x7f717803a990 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f717c006fd0 tx=0x7f717c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:08.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.587+0000 7f718e000700 1 -- 172.21.15.78:0/3243415092 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f718401a430 con 0x7f719007a2d0 2024-03-14T09:46:08.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:07.838+0000 7f7197e00700 1 -- 172.21.15.78:0/3243415092 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f719019fd00 con 0x7f719007a2d0 2024-03-14T09:46:08.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:08.596+0000 7f718e000700 1 -- 172.21.15.78:0/3243415092 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 5) v1 ==== 45374+0+0 (secure 0 0 0) 0x7f718401afe0 con 0x7f719007a2d0 2024-03-14T09:46:08.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:08.598+0000 7f718e000700 1 -- 172.21.15.78:0/3243415092 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f718404bfe0 con 0x7f719007a2d0 2024-03-14T09:46:08.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:08.602+0000 7f7197e00700 1 -- 172.21.15.78:0/3243415092 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f71780384d0 msgr2=0x7f717803a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:08.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:08.602+0000 7f7197e00700 1 --2- 172.21.15.78:0/3243415092 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f71780384d0 0x7f717803a990 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f717c006fd0 tx=0x7f717c006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:08.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:08.602+0000 7f7197e00700 1 -- 172.21.15.78:0/3243415092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719007a2d0 msgr2=0x7f719019c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:08.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:08.602+0000 7f7197e00700 1 --2- 172.21.15.78:0/3243415092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719007a2d0 0x7f719019c830 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7184009130 tx=0x7f7184011ad0 comp rx=0 tx=0).stop 2024-03-14T09:46:08.960 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:08.602+0000 7f7197e00700 1 -- 172.21.15.78:0/3243415092 shutdown_connections 2024-03-14T09:46:08.961 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:08.602+0000 7f7197e00700 1 --2- 172.21.15.78:0/3243415092 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7f71780384d0 0x7f717803a990 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:08.961 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:08.602+0000 7f7197e00700 1 --2- 172.21.15.78:0/3243415092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f719007a2d0 0x7f719019c830 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:08.961 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:08.602+0000 7f7197e00700 1 -- 172.21.15.78:0/3243415092 >> 172.21.15.78:0/3243415092 conn(0x7f7190101a70 msgr2=0x7f7190102640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:08.961 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:08.602+0000 7f7197e00700 1 -- 172.21.15.78:0/3243415092 shutdown_connections 2024-03-14T09:46:08.961 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:08.602+0000 7f7197e00700 1 -- 172.21.15.78:0/3243415092 wait complete. 2024-03-14T09:46:09.303 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:09 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:09.196+0000 7f3eb0fc9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-14T09:46:09.596 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:09 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:09.321+0000 7f3eb0fc9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-14T09:46:09.596 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:09 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:09.440+0000 7f3eb0fc9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-14T09:46:09.596 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:09 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:09.550+0000 7f3eb0fc9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-14T09:46:09.856 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:09 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3243415092' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-14T09:46:09.856 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:09 smithi078 ceph-mon[38143]: mgrmap e5: x(active, since 10s) 2024-03-14T09:46:10.001 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-14T09:46:10.001 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-14T09:46:10.001 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-14T09:46:10.001 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-14T09:46:10.001 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-14T09:46:10.002 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-14T09:46:10.002 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.352+0000 7fb7b0e00700 1 Processor -- start 2024-03-14T09:46:10.002 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.352+0000 7fb7b0e00700 1 -- start start 2024-03-14T09:46:10.002 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.353+0000 7fb7b0e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac07a2d0 0x7fb7ac07a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:10.002 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.353+0000 7fb7b0e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac07acc0 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.002 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.353+0000 7fb7a9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac07a2d0 0x7fb7ac07a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:10.002 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.353+0000 7fb7a9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac07a2d0 0x7fb7ac07a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51848/0 (socket says 172.21.15.78:51848) 2024-03-14T09:46:10.002 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.353+0000 7fb7a9e00700 1 -- 172.21.15.78:0/1740111504 learned_addr learned my addr 172.21.15.78:0/1740111504 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:10.002 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.353+0000 7fb7a9e00700 1 -- 172.21.15.78:0/1740111504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ac07ae00 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.002 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.354+0000 7fb7a9e00700 1 --2- 172.21.15.78:0/1740111504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac07a2d0 0x7fb7ac07a6f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb79c0098a0 tx=0x7fb79c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a416fad5310cfb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:10.002 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.354+0000 7fb7a8a00700 1 -- 172.21.15.78:0/1740111504 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb79c004160 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.002 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.354+0000 7fb7a8a00700 1 -- 172.21.15.78:0/1740111504 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb79c0042c0 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.002 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.354+0000 7fb7a8a00700 1 -- 172.21.15.78:0/1740111504 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb79c011a00 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.003 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.355+0000 7fb7b0e00700 1 -- 172.21.15.78:0/1740111504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac07a2d0 msgr2=0x7fb7ac07a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:10.003 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.355+0000 7fb7b0e00700 1 --2- 172.21.15.78:0/1740111504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac07a2d0 0x7fb7ac07a6f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb79c0098a0 tx=0x7fb79c009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:10.003 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.355+0000 7fb7b0e00700 1 -- 172.21.15.78:0/1740111504 shutdown_connections 2024-03-14T09:46:10.003 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.355+0000 7fb7b0e00700 1 --2- 172.21.15.78:0/1740111504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac07a2d0 0x7fb7ac07a6f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:10.003 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.355+0000 7fb7b0e00700 1 -- 172.21.15.78:0/1740111504 >> 172.21.15.78:0/1740111504 conn(0x7fb7ac101a30 msgr2=0x7fb7ac103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:10.003 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.355+0000 7fb7b0e00700 1 -- 172.21.15.78:0/1740111504 shutdown_connections 2024-03-14T09:46:10.003 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.355+0000 7fb7b0e00700 1 -- 172.21.15.78:0/1740111504 wait complete. 2024-03-14T09:46:10.003 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.356+0000 7fb7b0e00700 1 Processor -- start 2024-03-14T09:46:10.003 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.356+0000 7fb7b0e00700 1 -- start start 2024-03-14T09:46:10.003 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.356+0000 7fb7b0e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac07a2d0 0x7fb7ac19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:10.003 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.356+0000 7fb7b0e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac19cd90 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.003 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.357+0000 7fb7a9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac07a2d0 0x7fb7ac19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:10.003 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.357+0000 7fb7a9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac07a2d0 0x7fb7ac19c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51854/0 (socket says 172.21.15.78:51854) 2024-03-14T09:46:10.003 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.357+0000 7fb7a9e00700 1 -- 172.21.15.78:0/266273391 learned_addr learned my addr 172.21.15.78:0/266273391 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:10.004 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.357+0000 7fb7a9e00700 1 -- 172.21.15.78:0/266273391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c009550 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.004 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.357+0000 7fb7a9e00700 1 --2- 172.21.15.78:0/266273391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac07a2d0 0x7fb7ac19c850 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009130 tx=0x7fb79c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:10.004 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.357+0000 7fb7a2a00700 1 -- 172.21.15.78:0/266273391 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb79c003b70 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.004 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.357+0000 7fb7a2a00700 1 -- 172.21.15.78:0/266273391 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb79c003cd0 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.004 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.357+0000 7fb7b0e00700 1 -- 172.21.15.78:0/266273391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ac19cf90 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.004 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.358+0000 7fb7a2a00700 1 -- 172.21.15.78:0/266273391 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb79c0197d0 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.004 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.358+0000 7fb7b0e00700 1 -- 172.21.15.78:0/266273391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ac19d430 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.004 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.359+0000 7fb7b0e00700 1 -- 172.21.15.78:0/266273391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ac10cc80 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.004 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.359+0000 7fb7a2a00700 1 -- 172.21.15.78:0/266273391 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 45374+0+0 (secure 0 0 0) 0x7fb79c01abb0 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.004 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.360+0000 7fb7a2a00700 1 --2- 172.21.15.78:0/266273391 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7fb790038570 0x7fb79003aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:10.004 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.360+0000 7fb7a2a00700 1 -- 172.21.15.78:0/266273391 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb79c04c5e0 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.004 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.360+0000 7fb7a9400700 1 -- 172.21.15.78:0/266273391 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7fb790038570 msgr2=0x7fb79003aa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1325295736 2024-03-14T09:46:10.004 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.360+0000 7fb7a9400700 1 --2- 172.21.15.78:0/266273391 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7fb790038570 0x7fb79003aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T09:46:10.004 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.364+0000 7fb7a2a00700 1 -- 172.21.15.78:0/266273391 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb79c01a430 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.004 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.561+0000 7fb7a9400700 1 -- 172.21.15.78:0/266273391 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7fb790038570 msgr2=0x7fb79003aa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1325295736 2024-03-14T09:46:10.005 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.561+0000 7fb7a9400700 1 --2- 172.21.15.78:0/266273391 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7fb790038570 0x7fb79003aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-14T09:46:10.005 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.630+0000 7fb7b0e00700 1 -- 172.21.15.78:0/266273391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fb7ac19fcc0 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.005 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.630+0000 7fb7a2a00700 1 -- 172.21.15.78:0/266273391 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fb79c02db40 con 0x7fb7ac07a2d0 2024-03-14T09:46:10.005 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.633+0000 7fb7b0e00700 1 -- 172.21.15.78:0/266273391 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7fb790038570 msgr2=0x7fb79003aa30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T09:46:10.005 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.633+0000 7fb7b0e00700 1 --2- 172.21.15.78:0/266273391 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7fb790038570 0x7fb79003aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:10.005 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.633+0000 7fb7b0e00700 1 -- 172.21.15.78:0/266273391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac07a2d0 msgr2=0x7fb7ac19c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:10.005 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.633+0000 7fb7b0e00700 1 --2- 172.21.15.78:0/266273391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac07a2d0 0x7fb7ac19c850 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009130 tx=0x7fb79c011ad0 comp rx=0 tx=0).stop 2024-03-14T09:46:10.005 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.633+0000 7fb7b0e00700 1 -- 172.21.15.78:0/266273391 shutdown_connections 2024-03-14T09:46:10.005 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.633+0000 7fb7b0e00700 1 --2- 172.21.15.78:0/266273391 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7fb790038570 0x7fb79003aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:10.005 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.633+0000 7fb7b0e00700 1 --2- 172.21.15.78:0/266273391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7ac07a2d0 0x7fb7ac19c850 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:10.005 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.633+0000 7fb7b0e00700 1 -- 172.21.15.78:0/266273391 >> 172.21.15.78:0/266273391 conn(0x7fb7ac101a30 msgr2=0x7fb7ac102630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:10.005 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.633+0000 7fb7b0e00700 1 -- 172.21.15.78:0/266273391 shutdown_connections 2024-03-14T09:46:10.005 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:09.633+0000 7fb7b0e00700 1 -- 172.21.15.78:0/266273391 wait complete. 2024-03-14T09:46:10.005 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for the mgr to restart... 2024-03-14T09:46:10.006 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr epoch 5... 2024-03-14T09:46:10.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:10 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/266273391' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-14T09:46:10.737 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:10.422+0000 7f3eb0fc9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-14T09:46:11.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:10.736+0000 7f3eb0fc9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-14T09:46:11.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:10.845+0000 7f3eb0fc9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-14T09:46:11.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:10.981+0000 7f3eb0fc9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-14T09:46:11.447 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:11 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:11.173+0000 7f3eb0fc9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-14T09:46:11.448 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:11 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:11.337+0000 7f3eb0fc9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-14T09:46:11.803 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:11 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:11.447+0000 7f3eb0fc9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-14T09:46:12.303 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:11 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:11.903+0000 7f3eb0fc9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-14T09:46:12.743 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:12 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:12.380+0000 7f3eb0fc9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-14T09:46:12.743 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:12 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:12.622+0000 7f3eb0fc9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-14T09:46:12.743 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:12 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:12.742+0000 7f3eb0fc9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-14T09:46:13.984 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:13 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:13.672+0000 7f3eb0fc9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-14T09:46:13.984 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:13 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:13.795+0000 7f3eb0fc9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-14T09:46:13.984 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:13 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:13.983+0000 7f3eb0fc9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-14T09:46:15.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:14 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:14.771+0000 7f3eb0fc9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-14T09:46:15.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:15 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:15.007+0000 7f3eb0fc9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-14T09:46:15.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:15 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:15.255+0000 7f3eb0fc9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-14T09:46:16.737 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:16 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:16.372+0000 7f3eb0fc9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-14T09:46:16.737 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:16 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:16.736+0000 7f3eb0fc9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-14T09:46:17.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:16 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:16.851+0000 7f3eb0fc9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-14T09:46:17.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:17 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:17.098+0000 7f3eb0fc9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-14T09:46:17.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:17 smithi078 ceph-mon[38143]: Active manager daemon x restarted 2024-03-14T09:46:17.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:17 smithi078 ceph-mon[38143]: Activating manager daemon x 2024-03-14T09:46:17.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:17 smithi078 ceph-mon[38143]: osdmap e2: 0 total, 0 up, 0 in 2024-03-14T09:46:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:17 smithi078 ceph-mon[38143]: mgrmap e6: x(active, starting, since 0.00429656s) 2024-03-14T09:46:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:17 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:46:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:17 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-14T09:46:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:17 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-14T09:46:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:17 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-14T09:46:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:17 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-14T09:46:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:17 smithi078 ceph-mon[38143]: Manager daemon x is now available 2024-03-14T09:46:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:17 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' 2024-03-14T09:46:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:17 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' 2024-03-14T09:46:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:17 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:46:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:17 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:46:18.493 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-14T09:46:18.493 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-14T09:46:18.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-14T09:46:18.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-14T09:46:18.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.437+0000 7ff4f9e00700 1 Processor -- start 2024-03-14T09:46:18.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.437+0000 7ff4f9e00700 1 -- start start 2024-03-14T09:46:18.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.437+0000 7ff4f9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f41068c0 0x7ff4f41076a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:18.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.437+0000 7ff4f9e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f4106d70 con 0x7ff4f41068c0 2024-03-14T09:46:18.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.437+0000 7ff4f2a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f41068c0 0x7ff4f41076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:18.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.437+0000 7ff4f2a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f41068c0 0x7ff4f41076a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51860/0 (socket says 172.21.15.78:51860) 2024-03-14T09:46:18.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.437+0000 7ff4f2a00700 1 -- 172.21.15.78:0/3680461341 learned_addr learned my addr 172.21.15.78:0/3680461341 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:18.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.438+0000 7ff4f2a00700 1 -- 172.21.15.78:0/3680461341 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f4106eb0 con 0x7ff4f41068c0 2024-03-14T09:46:18.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.438+0000 7ff4f2a00700 1 --2- 172.21.15.78:0/3680461341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f41068c0 0x7ff4f41076a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff4e40098a0 tx=0x7ff4e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72bbd2c4c72779e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:18.497 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.438+0000 7ff4f1600700 1 -- 172.21.15.78:0/3680461341 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4e4004160 con 0x7ff4f41068c0 2024-03-14T09:46:18.497 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.438+0000 7ff4f1600700 1 -- 172.21.15.78:0/3680461341 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4e40042c0 con 0x7ff4f41068c0 2024-03-14T09:46:18.497 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.438+0000 7ff4f1600700 1 -- 172.21.15.78:0/3680461341 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4e4011a00 con 0x7ff4f41068c0 2024-03-14T09:46:18.497 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.439+0000 7ff4f9e00700 1 -- 172.21.15.78:0/3680461341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f41068c0 msgr2=0x7ff4f41076a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:18.497 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.439+0000 7ff4f9e00700 1 --2- 172.21.15.78:0/3680461341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f41068c0 0x7ff4f41076a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff4e40098a0 tx=0x7ff4e4009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:18.497 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.439+0000 7ff4f9e00700 1 -- 172.21.15.78:0/3680461341 shutdown_connections 2024-03-14T09:46:18.497 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.439+0000 7ff4f9e00700 1 --2- 172.21.15.78:0/3680461341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f41068c0 0x7ff4f41076a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:18.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.439+0000 7ff4f9e00700 1 -- 172.21.15.78:0/3680461341 >> 172.21.15.78:0/3680461341 conn(0x7ff4f4101a20 msgr2=0x7ff4f4103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:18.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.439+0000 7ff4f9e00700 1 -- 172.21.15.78:0/3680461341 shutdown_connections 2024-03-14T09:46:18.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.439+0000 7ff4f9e00700 1 -- 172.21.15.78:0/3680461341 wait complete. 2024-03-14T09:46:18.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.440+0000 7ff4f9e00700 1 Processor -- start 2024-03-14T09:46:18.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.440+0000 7ff4f9e00700 1 -- start start 2024-03-14T09:46:18.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.440+0000 7ff4f9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f41068c0 0x7ff4f407bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:18.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.441+0000 7ff4f9e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f407c070 con 0x7ff4f41068c0 2024-03-14T09:46:18.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.441+0000 7ff4f2a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f41068c0 0x7ff4f407bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:18.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.441+0000 7ff4f2a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f41068c0 0x7ff4f407bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51866/0 (socket says 172.21.15.78:51866) 2024-03-14T09:46:18.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.441+0000 7ff4f2a00700 1 -- 172.21.15.78:0/222118112 learned_addr learned my addr 172.21.15.78:0/222118112 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:18.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.441+0000 7ff4f2a00700 1 -- 172.21.15.78:0/222118112 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e4009550 con 0x7ff4f41068c0 2024-03-14T09:46:18.498 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.441+0000 7ff4f2a00700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f41068c0 0x7ff4f407bb30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009130 tx=0x7ff4e4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:18.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.441+0000 7ff4f8a00700 1 -- 172.21.15.78:0/222118112 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4e4003ab0 con 0x7ff4f41068c0 2024-03-14T09:46:18.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.442+0000 7ff4f8a00700 1 -- 172.21.15.78:0/222118112 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4e4003c10 con 0x7ff4f41068c0 2024-03-14T09:46:18.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.442+0000 7ff4f8a00700 1 -- 172.21.15.78:0/222118112 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4e40247d0 con 0x7ff4f41068c0 2024-03-14T09:46:18.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.442+0000 7ff4f9e00700 1 -- 172.21.15.78:0/222118112 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f407a2b0 con 0x7ff4f41068c0 2024-03-14T09:46:18.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.442+0000 7ff4f9e00700 1 -- 172.21.15.78:0/222118112 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f407a750 con 0x7ff4f41068c0 2024-03-14T09:46:18.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.443+0000 7ff4f8a00700 1 -- 172.21.15.78:0/222118112 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 45374+0+0 (secure 0 0 0) 0x7ff4e402d3f0 con 0x7ff4f41068c0 2024-03-14T09:46:18.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.443+0000 7ff4f8a00700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7ff4d80384d0 0x7ff4d803a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:18.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.443+0000 7ff4f8a00700 1 -- 172.21.15.78:0/222118112 --> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4d803b0a0 con 0x7ff4d80384d0 2024-03-14T09:46:18.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.443+0000 7ff4f8a00700 1 -- 172.21.15.78:0/222118112 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff4e404c330 con 0x7ff4f41068c0 2024-03-14T09:46:18.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.443+0000 7ff4f2000700 1 -- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7ff4d80384d0 msgr2=0x7ff4d803a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1325295736 2024-03-14T09:46:18.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.443+0000 7ff4f2000700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7ff4d80384d0 0x7ff4d803a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T09:46:18.499 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.644+0000 7ff4f2000700 1 -- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7ff4d80384d0 msgr2=0x7ff4d803a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1325295736 2024-03-14T09:46:18.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:10.644+0000 7ff4f2000700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7ff4d80384d0 0x7ff4d803a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-14T09:46:18.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:11.045+0000 7ff4f2000700 1 -- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7ff4d80384d0 msgr2=0x7ff4d803a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1325295736 2024-03-14T09:46:18.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:11.045+0000 7ff4f2000700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7ff4d80384d0 0x7ff4d803a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-14T09:46:18.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:11.846+0000 7ff4f2000700 1 -- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7ff4d80384d0 msgr2=0x7ff4d803a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1325295736 2024-03-14T09:46:18.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:11.846+0000 7ff4f2000700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7ff4d80384d0 0x7ff4d803a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-14T09:46:18.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:13.448+0000 7ff4f2000700 1 -- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7ff4d80384d0 msgr2=0x7ff4d803a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1325295736 2024-03-14T09:46:18.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:13.448+0000 7ff4f2000700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7ff4d80384d0 0x7ff4d803a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-14T09:46:18.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:16.651+0000 7ff4f2000700 1 -- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7ff4d80384d0 msgr2=0x7ff4d803a990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1325295736 2024-03-14T09:46:18.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:16.651+0000 7ff4f2000700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7ff4d80384d0 0x7ff4d803a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-14T09:46:18.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:17.102+0000 7ff4f8a00700 1 -- 172.21.15.78:0/222118112 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mgrmap(e 6) v1 ==== 45141+0+0 (secure 0 0 0) 0x7ff4e401a770 con 0x7ff4f41068c0 2024-03-14T09:46:18.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:17.103+0000 7ff4f8a00700 1 -- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7ff4d80384d0 msgr2=0x7ff4d803a990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T09:46:18.500 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:17.103+0000 7ff4f8a00700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7ff4d80384d0 0x7ff4d803a990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:18.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.104+0000 7ff4f8a00700 1 -- 172.21.15.78:0/222118112 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 7) v1 ==== 45268+0+0 (secure 0 0 0) 0x7ff4e404dce0 con 0x7ff4f41068c0 2024-03-14T09:46:18.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.105+0000 7ff4f8a00700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7ff4d803bdb0 0x7ff4d803e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:18.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.105+0000 7ff4f8a00700 1 -- 172.21.15.78:0/222118112 --> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4d803b0a0 con 0x7ff4d803bdb0 2024-03-14T09:46:18.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.105+0000 7ff4f2000700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7ff4d803bdb0 0x7ff4d803e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:18.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.106+0000 7ff4f2000700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7ff4d803bdb0 0x7ff4d803e1a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0003a10 tx=0x7ff4e0009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:18.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.106+0000 7ff4f8a00700 1 -- 172.21.15.78:0/222118112 <== mgr.14120 v2:172.21.15.78:6800/3890232539 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7ff4d803b0a0 con 0x7ff4d803bdb0 2024-03-14T09:46:18.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.113+0000 7ff4e9600700 1 -- 172.21.15.78:0/222118112 --> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff4f40623e0 con 0x7ff4d803bdb0 2024-03-14T09:46:18.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.113+0000 7ff4f8a00700 1 -- 172.21.15.78:0/222118112 <== mgr.14120 v2:172.21.15.78:6800/3890232539 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ff4f40623e0 con 0x7ff4d803bdb0 2024-03-14T09:46:18.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.114+0000 7ff4e9600700 1 -- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7ff4d803bdb0 msgr2=0x7ff4d803e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:18.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.114+0000 7ff4e9600700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7ff4d803bdb0 0x7ff4d803e1a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0003a10 tx=0x7ff4e0009290 comp rx=0 tx=0).stop 2024-03-14T09:46:18.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.114+0000 7ff4e9600700 1 -- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f41068c0 msgr2=0x7ff4f407bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:18.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.114+0000 7ff4e9600700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f41068c0 0x7ff4f407bb30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009130 tx=0x7ff4e4011ad0 comp rx=0 tx=0).stop 2024-03-14T09:46:18.501 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.114+0000 7ff4e9600700 1 -- 172.21.15.78:0/222118112 shutdown_connections 2024-03-14T09:46:18.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.114+0000 7ff4e9600700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7ff4d803bdb0 0x7ff4d803e1a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:18.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.114+0000 7ff4e9600700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4f41068c0 0x7ff4f407bb30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:18.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.114+0000 7ff4e9600700 1 --2- 172.21.15.78:0/222118112 >> [v2:172.21.15.78:6800/1325295736,v1:172.21.15.78:6801/1325295736] conn(0x7ff4d80384d0 0x7ff4d803a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:18.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.114+0000 7ff4e9600700 1 -- 172.21.15.78:0/222118112 >> 172.21.15.78:0/222118112 conn(0x7ff4f4101a20 msgr2=0x7ff4f4102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:18.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.114+0000 7ff4e9600700 1 -- 172.21.15.78:0/222118112 shutdown_connections 2024-03-14T09:46:18.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.114+0000 7ff4e9600700 1 -- 172.21.15.78:0/222118112 wait complete. 2024-03-14T09:46:18.502 INFO:teuthology.orchestra.run.smithi078.stdout:mgr epoch 5 is available 2024-03-14T09:46:18.502 INFO:teuthology.orchestra.run.smithi078.stdout:Setting orchestrator backend to cephadm... 2024-03-14T09:46:18.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:18 smithi078 ceph-mon[38143]: Found migration_current of "None". Setting to last migration. 2024-03-14T09:46:18.502 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:18 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:46:18.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:18 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-14T09:46:18.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:18 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-14T09:46:18.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:18 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' 2024-03-14T09:46:18.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:18 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' 2024-03-14T09:46:18.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:18 smithi078 ceph-mon[38143]: mgrmap e7: x(active, since 1.00729s) 2024-03-14T09:46:19.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:19 smithi078 ceph-mon[38143]: [14/Mar/2024:09:46:17] ENGINE Bus STARTING 2024-03-14T09:46:19.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:19 smithi078 ceph-mon[38143]: [14/Mar/2024:09:46:18] ENGINE Serving on http://172.21.15.78:8765 2024-03-14T09:46:19.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:19 smithi078 ceph-mon[38143]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-14T09:46:19.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:19 smithi078 ceph-mon[38143]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-14T09:46:19.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:19 smithi078 ceph-mon[38143]: [14/Mar/2024:09:46:18] ENGINE Serving on https://172.21.15.78:7150 2024-03-14T09:46:19.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:19 smithi078 ceph-mon[38143]: [14/Mar/2024:09:46:18] ENGINE Bus STARTED 2024-03-14T09:46:19.457 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:19 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:46:19.457 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:19 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' 2024-03-14T09:46:19.457 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:19 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:46:19.491 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.884+0000 7f424b400700 1 Processor -- start 2024-03-14T09:46:19.491 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.884+0000 7f424b400700 1 -- start start 2024-03-14T09:46:19.491 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.884+0000 7f424b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f424407a2f0 0x7f424407a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:19.491 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.884+0000 7f424b400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424407ace0 con 0x7f424407a2f0 2024-03-14T09:46:19.491 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.884+0000 7f4248a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f424407a2f0 0x7f424407a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:19.491 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.884+0000 7f4248a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f424407a2f0 0x7f424407a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42762/0 (socket says 172.21.15.78:42762) 2024-03-14T09:46:19.491 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.884+0000 7f4248a00700 1 -- 172.21.15.78:0/1218409145 learned_addr learned my addr 172.21.15.78:0/1218409145 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:19.491 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.885+0000 7f4248a00700 1 -- 172.21.15.78:0/1218409145 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424407ae20 con 0x7f424407a2f0 2024-03-14T09:46:19.491 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.885+0000 7f4248a00700 1 --2- 172.21.15.78:0/1218409145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f424407a2f0 0x7f424407a710 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f42380098a0 tx=0x7f4238009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c4de41f4a41d75b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:19.492 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.885+0000 7f4243400700 1 -- 172.21.15.78:0/1218409145 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4238004160 con 0x7f424407a2f0 2024-03-14T09:46:19.492 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.885+0000 7f4243400700 1 -- 172.21.15.78:0/1218409145 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42380042c0 con 0x7f424407a2f0 2024-03-14T09:46:19.492 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.885+0000 7f4243400700 1 -- 172.21.15.78:0/1218409145 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4238011a00 con 0x7f424407a2f0 2024-03-14T09:46:19.492 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.886+0000 7f424b400700 1 -- 172.21.15.78:0/1218409145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f424407a2f0 msgr2=0x7f424407a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:19.492 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.886+0000 7f424b400700 1 --2- 172.21.15.78:0/1218409145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f424407a2f0 0x7f424407a710 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f42380098a0 tx=0x7f4238009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:19.492 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.886+0000 7f424b400700 1 -- 172.21.15.78:0/1218409145 shutdown_connections 2024-03-14T09:46:19.492 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.886+0000 7f424b400700 1 --2- 172.21.15.78:0/1218409145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f424407a2f0 0x7f424407a710 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:19.492 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.886+0000 7f424b400700 1 -- 172.21.15.78:0/1218409145 >> 172.21.15.78:0/1218409145 conn(0x7f4244101a70 msgr2=0x7f4244103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:19.492 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.886+0000 7f424b400700 1 -- 172.21.15.78:0/1218409145 shutdown_connections 2024-03-14T09:46:19.492 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.886+0000 7f424b400700 1 -- 172.21.15.78:0/1218409145 wait complete. 2024-03-14T09:46:19.492 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.887+0000 7f424b400700 1 Processor -- start 2024-03-14T09:46:19.492 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.887+0000 7f424b400700 1 -- start start 2024-03-14T09:46:19.493 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.888+0000 7f424b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f424407a2f0 0x7f424419c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:19.493 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.888+0000 7f424b400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424419cd30 con 0x7f424407a2f0 2024-03-14T09:46:19.493 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.888+0000 7f4248a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f424407a2f0 0x7f424419c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:19.493 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.888+0000 7f4248a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f424407a2f0 0x7f424419c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42768/0 (socket says 172.21.15.78:42768) 2024-03-14T09:46:19.493 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.888+0000 7f4248a00700 1 -- 172.21.15.78:0/4283425532 learned_addr learned my addr 172.21.15.78:0/4283425532 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:19.493 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.888+0000 7f4248a00700 1 -- 172.21.15.78:0/4283425532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4238009550 con 0x7f424407a2f0 2024-03-14T09:46:19.493 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.888+0000 7f4248a00700 1 --2- 172.21.15.78:0/4283425532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f424407a2f0 0x7f424419c7f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4238009130 tx=0x7f4238011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:19.493 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.889+0000 7f4241600700 1 -- 172.21.15.78:0/4283425532 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4238003b70 con 0x7f424407a2f0 2024-03-14T09:46:19.493 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.889+0000 7f4241600700 1 -- 172.21.15.78:0/4283425532 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4238003cd0 con 0x7f424407a2f0 2024-03-14T09:46:19.493 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.889+0000 7f424b400700 1 -- 172.21.15.78:0/4283425532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f424419cf30 con 0x7f424407a2f0 2024-03-14T09:46:19.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.889+0000 7f4241600700 1 -- 172.21.15.78:0/4283425532 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42380197d0 con 0x7f424407a2f0 2024-03-14T09:46:19.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.889+0000 7f424b400700 1 -- 172.21.15.78:0/4283425532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f424419d3d0 con 0x7f424407a2f0 2024-03-14T09:46:19.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.890+0000 7f4241600700 1 -- 172.21.15.78:0/4283425532 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 7) v1 ==== 45268+0+0 (secure 0 0 0) 0x7f423801a430 con 0x7f424407a2f0 2024-03-14T09:46:19.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.890+0000 7f424b400700 1 -- 172.21.15.78:0/4283425532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424410ccc0 con 0x7f424407a2f0 2024-03-14T09:46:19.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.890+0000 7f4241600700 1 --2- 172.21.15.78:0/4283425532 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f422c0383f0 0x7f422c03a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:19.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.890+0000 7f4241600700 1 -- 172.21.15.78:0/4283425532 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f423804c470 con 0x7f424407a2f0 2024-03-14T09:46:19.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.891+0000 7f4243e00700 1 --2- 172.21.15.78:0/4283425532 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f422c0383f0 0x7f422c03a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:19.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.891+0000 7f4243e00700 1 --2- 172.21.15.78:0/4283425532 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f422c0383f0 0x7f422c03a8b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4230006fd0 tx=0x7f4230006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:19.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:18.896+0000 7f4241600700 1 -- 172.21.15.78:0/4283425532 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f423802d960 con 0x7f424407a2f0 2024-03-14T09:46:19.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.113+0000 7f424b400700 1 -- 172.21.15.78:0/4283425532 --> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f4244102cf0 con 0x7f422c0383f0 2024-03-14T09:46:19.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.123+0000 7f4241600700 1 -- 172.21.15.78:0/4283425532 <== mgr.14120 v2:172.21.15.78:6800/3890232539 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4244102cf0 con 0x7f422c0383f0 2024-03-14T09:46:19.494 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.126+0000 7f424b400700 1 -- 172.21.15.78:0/4283425532 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f422c0383f0 msgr2=0x7f422c03a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:19.495 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.126+0000 7f424b400700 1 --2- 172.21.15.78:0/4283425532 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f422c0383f0 0x7f422c03a8b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4230006fd0 tx=0x7f4230006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:19.495 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.126+0000 7f424b400700 1 -- 172.21.15.78:0/4283425532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f424407a2f0 msgr2=0x7f424419c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:19.495 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.126+0000 7f424b400700 1 --2- 172.21.15.78:0/4283425532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f424407a2f0 0x7f424419c7f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4238009130 tx=0x7f4238011ad0 comp rx=0 tx=0).stop 2024-03-14T09:46:19.495 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.126+0000 7f424b400700 1 -- 172.21.15.78:0/4283425532 shutdown_connections 2024-03-14T09:46:19.495 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.126+0000 7f424b400700 1 --2- 172.21.15.78:0/4283425532 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f422c0383f0 0x7f422c03a8b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:19.495 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.126+0000 7f424b400700 1 --2- 172.21.15.78:0/4283425532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f424407a2f0 0x7f424419c7f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:19.495 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.126+0000 7f424b400700 1 -- 172.21.15.78:0/4283425532 >> 172.21.15.78:0/4283425532 conn(0x7f4244101a70 msgr2=0x7f42441025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:19.495 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.126+0000 7f424b400700 1 -- 172.21.15.78:0/4283425532 shutdown_connections 2024-03-14T09:46:19.495 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.126+0000 7f424b400700 1 -- 172.21.15.78:0/4283425532 wait complete. 2024-03-14T09:46:20.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:20 smithi078 ceph-mon[38143]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:46:20.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:20 smithi078 ceph-mon[38143]: mgrmap e8: x(active, since 2s) 2024-03-14T09:46:20.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-14T09:46:20.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.889+0000 7f8673000700 1 Processor -- start 2024-03-14T09:46:20.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.889+0000 7f8673000700 1 -- start start 2024-03-14T09:46:20.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.890+0000 7f8673000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f866c107330 0x7f866c109720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:20.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.890+0000 7f8673000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866c079ec0 con 0x7f866c107330 2024-03-14T09:46:20.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.890+0000 7f866be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f866c107330 0x7f866c109720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:20.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.890+0000 7f866be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f866c107330 0x7f866c109720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42772/0 (socket says 172.21.15.78:42772) 2024-03-14T09:46:20.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.890+0000 7f866be00700 1 -- 172.21.15.78:0/3806031154 learned_addr learned my addr 172.21.15.78:0/3806031154 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:20.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.890+0000 7f866be00700 1 -- 172.21.15.78:0/3806031154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866c109c60 con 0x7f866c107330 2024-03-14T09:46:20.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.891+0000 7f866be00700 1 --2- 172.21.15.78:0/3806031154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f866c107330 0x7f866c109720 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f865c0098a0 tx=0x7f865c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e20c96d9c8df4c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:20.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.891+0000 7f866aa00700 1 -- 172.21.15.78:0/3806031154 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f865c004160 con 0x7f866c107330 2024-03-14T09:46:20.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.891+0000 7f866aa00700 1 -- 172.21.15.78:0/3806031154 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f865c0042c0 con 0x7f866c107330 2024-03-14T09:46:20.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.891+0000 7f866aa00700 1 -- 172.21.15.78:0/3806031154 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f865c011a00 con 0x7f866c107330 2024-03-14T09:46:20.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.891+0000 7f8673000700 1 -- 172.21.15.78:0/3806031154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f866c107330 msgr2=0x7f866c109720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:20.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.891+0000 7f8673000700 1 --2- 172.21.15.78:0/3806031154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f866c107330 0x7f866c109720 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f865c0098a0 tx=0x7f865c009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:20.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.892+0000 7f8673000700 1 -- 172.21.15.78:0/3806031154 shutdown_connections 2024-03-14T09:46:20.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.892+0000 7f8673000700 1 --2- 172.21.15.78:0/3806031154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f866c107330 0x7f866c109720 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:20.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.892+0000 7f8673000700 1 -- 172.21.15.78:0/3806031154 >> 172.21.15.78:0/3806031154 conn(0x7f866c100980 msgr2=0x7f866c102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:20.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.892+0000 7f8673000700 1 -- 172.21.15.78:0/3806031154 shutdown_connections 2024-03-14T09:46:20.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.892+0000 7f8673000700 1 -- 172.21.15.78:0/3806031154 wait complete. 2024-03-14T09:46:20.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.893+0000 7f8673000700 1 Processor -- start 2024-03-14T09:46:20.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.893+0000 7f8673000700 1 -- start start 2024-03-14T09:46:20.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.893+0000 7f8673000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f866c107330 0x7f866c19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:20.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.893+0000 7f8673000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866c19cda0 con 0x7f866c107330 2024-03-14T09:46:20.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.893+0000 7f866be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f866c107330 0x7f866c19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:20.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.893+0000 7f866be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f866c107330 0x7f866c19c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42778/0 (socket says 172.21.15.78:42778) 2024-03-14T09:46:20.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.893+0000 7f866be00700 1 -- 172.21.15.78:0/146940057 learned_addr learned my addr 172.21.15.78:0/146940057 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:20.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.894+0000 7f866be00700 1 -- 172.21.15.78:0/146940057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865c009550 con 0x7f866c107330 2024-03-14T09:46:20.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.894+0000 7f866be00700 1 --2- 172.21.15.78:0/146940057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f866c107330 0x7f866c19c860 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f865c009130 tx=0x7f865c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:20.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.894+0000 7f8668c00700 1 -- 172.21.15.78:0/146940057 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f865c003b70 con 0x7f866c107330 2024-03-14T09:46:20.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.894+0000 7f8668c00700 1 -- 172.21.15.78:0/146940057 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f865c003cd0 con 0x7f866c107330 2024-03-14T09:46:20.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.894+0000 7f8673000700 1 -- 172.21.15.78:0/146940057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866c19cfa0 con 0x7f866c107330 2024-03-14T09:46:20.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.894+0000 7f8668c00700 1 -- 172.21.15.78:0/146940057 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f865c0197d0 con 0x7f866c107330 2024-03-14T09:46:20.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.894+0000 7f8673000700 1 -- 172.21.15.78:0/146940057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f866c19d440 con 0x7f866c107330 2024-03-14T09:46:20.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.895+0000 7f8668c00700 1 -- 172.21.15.78:0/146940057 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45374+0+0 (secure 0 0 0) 0x7f865c01a430 con 0x7f866c107330 2024-03-14T09:46:20.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.895+0000 7f8673000700 1 -- 172.21.15.78:0/146940057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f866c10aa60 con 0x7f866c107330 2024-03-14T09:46:20.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.896+0000 7f8668c00700 1 --2- 172.21.15.78:0/146940057 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f8650038520 0x7f865003a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:20.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.896+0000 7f8668c00700 1 -- 172.21.15.78:0/146940057 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f865c04c6c0 con 0x7f866c107330 2024-03-14T09:46:20.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.896+0000 7f866b400700 1 --2- 172.21.15.78:0/146940057 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f8650038520 0x7f865003a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:20.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.897+0000 7f866b400700 1 --2- 172.21.15.78:0/146940057 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f8650038520 0x7f865003a9e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8658006fd0 tx=0x7f8658006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:20.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:19.901+0000 7f8668c00700 1 -- 172.21.15.78:0/146940057 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f865c02d9a0 con 0x7f866c107330 2024-03-14T09:46:20.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.103+0000 7f8673000700 1 -- 172.21.15.78:0/146940057 --> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f866c101bb0 con 0x7f8650038520 2024-03-14T09:46:20.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.104+0000 7f8668c00700 1 -- 172.21.15.78:0/146940057 <== mgr.14120 v2:172.21.15.78:6800/3890232539 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f866c101bb0 con 0x7f8650038520 2024-03-14T09:46:20.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.106+0000 7f8673000700 1 -- 172.21.15.78:0/146940057 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f8650038520 msgr2=0x7f865003a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:20.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.106+0000 7f8673000700 1 --2- 172.21.15.78:0/146940057 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f8650038520 0x7f865003a9e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8658006fd0 tx=0x7f8658006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:20.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.106+0000 7f8673000700 1 -- 172.21.15.78:0/146940057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f866c107330 msgr2=0x7f866c19c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:20.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.106+0000 7f8673000700 1 --2- 172.21.15.78:0/146940057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f866c107330 0x7f866c19c860 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f865c009130 tx=0x7f865c011ad0 comp rx=0 tx=0).stop 2024-03-14T09:46:20.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.107+0000 7f8673000700 1 -- 172.21.15.78:0/146940057 shutdown_connections 2024-03-14T09:46:20.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.107+0000 7f8673000700 1 --2- 172.21.15.78:0/146940057 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f8650038520 0x7f865003a9e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:20.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.107+0000 7f8673000700 1 --2- 172.21.15.78:0/146940057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f866c107330 0x7f866c19c860 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:20.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.107+0000 7f8673000700 1 -- 172.21.15.78:0/146940057 >> 172.21.15.78:0/146940057 conn(0x7f866c100980 msgr2=0x7f866c1014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:20.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.107+0000 7f8673000700 1 -- 172.21.15.78:0/146940057 shutdown_connections 2024-03-14T09:46:20.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.107+0000 7f8673000700 1 -- 172.21.15.78:0/146940057 wait complete. 2024-03-14T09:46:20.458 INFO:teuthology.orchestra.run.smithi078.stdout:Generating ssh key... 2024-03-14T09:46:21.481 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-mon[38143]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:46:21.481 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: Generating public/private rsa key pair. 2024-03-14T09:46:21.481 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: Your identification has been saved in /tmp/tmp43ro2alh/key. 2024-03-14T09:46:21.481 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: Your public key has been saved in /tmp/tmp43ro2alh/key.pub. 2024-03-14T09:46:21.481 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: The key fingerprint is: 2024-03-14T09:46:21.481 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: SHA256:sMH2EUhVC1gdIB8mQ6hPkIn5h+AbwcGDjMHKhgofnbQ ceph-83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:46:21.481 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: The key's randomart image is: 2024-03-14T09:46:21.481 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: +---[RSA 3072]----+ 2024-03-14T09:46:21.481 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: |O=.o +*=O+o. | 2024-03-14T09:46:21.481 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: |*B+ o.o* +.. | 2024-03-14T09:46:21.481 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: |=oo* o= o . | 2024-03-14T09:46:21.482 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: |+== E. = . | 2024-03-14T09:46:21.482 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: |+.o= . S | 2024-03-14T09:46:21.482 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: |... . | 2024-03-14T09:46:21.482 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: | | 2024-03-14T09:46:21.482 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: | | 2024-03-14T09:46:21.482 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: | | 2024-03-14T09:46:21.482 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: +----[SHA256]-----+ 2024-03-14T09:46:21.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.814+0000 7f6900a00700 1 Processor -- start 2024-03-14T09:46:21.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.814+0000 7f6900a00700 1 -- start start 2024-03-14T09:46:21.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.814+0000 7f6900a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68fc07a2d0 0x7f68fc07a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:21.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.814+0000 7f6900a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68fc07acc0 con 0x7f68fc07a2d0 2024-03-14T09:46:21.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.815+0000 7f68f9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68fc07a2d0 0x7f68fc07a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:21.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.815+0000 7f68f9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68fc07a2d0 0x7f68fc07a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42786/0 (socket says 172.21.15.78:42786) 2024-03-14T09:46:21.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.815+0000 7f68f9e00700 1 -- 172.21.15.78:0/279915696 learned_addr learned my addr 172.21.15.78:0/279915696 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:21.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.815+0000 7f68f9e00700 1 -- 172.21.15.78:0/279915696 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68fc07ae00 con 0x7f68fc07a2d0 2024-03-14T09:46:21.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.815+0000 7f68f9e00700 1 --2- 172.21.15.78:0/279915696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68fc07a2d0 0x7f68fc07a6f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0098a0 tx=0x7f68ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807708e68bf8d572 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:21.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.815+0000 7f68f8a00700 1 -- 172.21.15.78:0/279915696 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68ec004160 con 0x7f68fc07a2d0 2024-03-14T09:46:21.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.815+0000 7f68f8a00700 1 -- 172.21.15.78:0/279915696 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68ec0042c0 con 0x7f68fc07a2d0 2024-03-14T09:46:21.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.815+0000 7f68f8a00700 1 -- 172.21.15.78:0/279915696 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68ec011a00 con 0x7f68fc07a2d0 2024-03-14T09:46:21.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.816+0000 7f6900a00700 1 -- 172.21.15.78:0/279915696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68fc07a2d0 msgr2=0x7f68fc07a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:21.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.816+0000 7f6900a00700 1 --2- 172.21.15.78:0/279915696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68fc07a2d0 0x7f68fc07a6f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0098a0 tx=0x7f68ec009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:21.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.816+0000 7f6900a00700 1 -- 172.21.15.78:0/279915696 shutdown_connections 2024-03-14T09:46:21.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.816+0000 7f6900a00700 1 --2- 172.21.15.78:0/279915696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68fc07a2d0 0x7f68fc07a6f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:21.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.816+0000 7f6900a00700 1 -- 172.21.15.78:0/279915696 >> 172.21.15.78:0/279915696 conn(0x7f68fc101a20 msgr2=0x7f68fc103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:21.627 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.816+0000 7f6900a00700 1 -- 172.21.15.78:0/279915696 shutdown_connections 2024-03-14T09:46:21.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.816+0000 7f6900a00700 1 -- 172.21.15.78:0/279915696 wait complete. 2024-03-14T09:46:21.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.817+0000 7f6900a00700 1 Processor -- start 2024-03-14T09:46:21.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.817+0000 7f6900a00700 1 -- start start 2024-03-14T09:46:21.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.818+0000 7f6900a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68fc07a2d0 0x7f68fc19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:21.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.818+0000 7f6900a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68fc19cd70 con 0x7f68fc07a2d0 2024-03-14T09:46:21.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.818+0000 7f68f9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68fc07a2d0 0x7f68fc19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:21.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.818+0000 7f68f9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68fc07a2d0 0x7f68fc19c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42798/0 (socket says 172.21.15.78:42798) 2024-03-14T09:46:21.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.818+0000 7f68f9e00700 1 -- 172.21.15.78:0/2589403521 learned_addr learned my addr 172.21.15.78:0/2589403521 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:21.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.818+0000 7f68f9e00700 1 -- 172.21.15.78:0/2589403521 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec009550 con 0x7f68fc07a2d0 2024-03-14T09:46:21.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.818+0000 7f68f9e00700 1 --2- 172.21.15.78:0/2589403521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68fc07a2d0 0x7f68fc19c830 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f68ec009130 tx=0x7f68ec011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:21.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.819+0000 7f68f2a00700 1 -- 172.21.15.78:0/2589403521 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68ec003b70 con 0x7f68fc07a2d0 2024-03-14T09:46:21.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.819+0000 7f68f2a00700 1 -- 172.21.15.78:0/2589403521 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68ec003cd0 con 0x7f68fc07a2d0 2024-03-14T09:46:21.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.819+0000 7f6900a00700 1 -- 172.21.15.78:0/2589403521 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68fc19cf70 con 0x7f68fc07a2d0 2024-03-14T09:46:21.628 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.819+0000 7f68f2a00700 1 -- 172.21.15.78:0/2589403521 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68ec0197d0 con 0x7f68fc07a2d0 2024-03-14T09:46:21.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.819+0000 7f6900a00700 1 -- 172.21.15.78:0/2589403521 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68fc19d410 con 0x7f68fc07a2d0 2024-03-14T09:46:21.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.820+0000 7f68f2a00700 1 -- 172.21.15.78:0/2589403521 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45374+0+0 (secure 0 0 0) 0x7f68ec01a430 con 0x7f68fc07a2d0 2024-03-14T09:46:21.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.820+0000 7f6900a00700 1 -- 172.21.15.78:0/2589403521 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68fc10cc70 con 0x7f68fc07a2d0 2024-03-14T09:46:21.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.820+0000 7f68f2a00700 1 --2- 172.21.15.78:0/2589403521 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f68e0038520 0x7f68e003a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:21.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.820+0000 7f68f2a00700 1 -- 172.21.15.78:0/2589403521 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f68ec04c670 con 0x7f68fc07a2d0 2024-03-14T09:46:21.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.821+0000 7f68f9400700 1 --2- 172.21.15.78:0/2589403521 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f68e0038520 0x7f68e003a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:21.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.821+0000 7f68f9400700 1 --2- 172.21.15.78:0/2589403521 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f68e0038520 0x7f68e003a9e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f68e8006fd0 tx=0x7f68e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:21.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:20.825+0000 7f68f2a00700 1 -- 172.21.15.78:0/2589403521 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f68ec02d960 con 0x7f68fc07a2d0 2024-03-14T09:46:21.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:21.026+0000 7f6900a00700 1 -- 172.21.15.78:0/2589403521 --> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f68fc102d50 con 0x7f68e0038520 2024-03-14T09:46:21.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:21.296+0000 7f68f2a00700 1 -- 172.21.15.78:0/2589403521 <== mgr.14120 v2:172.21.15.78:6800/3890232539 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f68fc102d50 con 0x7f68e0038520 2024-03-14T09:46:21.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:21.299+0000 7f6900a00700 1 -- 172.21.15.78:0/2589403521 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f68e0038520 msgr2=0x7f68e003a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:21.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:21.299+0000 7f6900a00700 1 --2- 172.21.15.78:0/2589403521 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f68e0038520 0x7f68e003a9e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f68e8006fd0 tx=0x7f68e8006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:21.629 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:21.299+0000 7f6900a00700 1 -- 172.21.15.78:0/2589403521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68fc07a2d0 msgr2=0x7f68fc19c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:21.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:21.299+0000 7f6900a00700 1 --2- 172.21.15.78:0/2589403521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68fc07a2d0 0x7f68fc19c830 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f68ec009130 tx=0x7f68ec011ad0 comp rx=0 tx=0).stop 2024-03-14T09:46:21.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:21.299+0000 7f6900a00700 1 -- 172.21.15.78:0/2589403521 shutdown_connections 2024-03-14T09:46:21.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:21.299+0000 7f6900a00700 1 --2- 172.21.15.78:0/2589403521 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f68e0038520 0x7f68e003a9e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:21.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:21.299+0000 7f6900a00700 1 --2- 172.21.15.78:0/2589403521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68fc07a2d0 0x7f68fc19c830 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:21.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:21.299+0000 7f6900a00700 1 -- 172.21.15.78:0/2589403521 >> 172.21.15.78:0/2589403521 conn(0x7f68fc101a20 msgr2=0x7f68fc102640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:21.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:21.299+0000 7f6900a00700 1 -- 172.21.15.78:0/2589403521 shutdown_connections 2024-03-14T09:46:21.630 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:21.299+0000 7f6900a00700 1 -- 172.21.15.78:0/2589403521 wait complete. 2024-03-14T09:46:22.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:22 smithi078 ceph-mon[38143]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:46:22.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:22 smithi078 ceph-mon[38143]: Generating ssh key... 2024-03-14T09:46:22.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:22 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' 2024-03-14T09:46:22.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:22 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' 2024-03-14T09:46:22.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCRdPpCviAowYx6L50GvU8eS1Qhy1Uel/UV4vugBaXb6iVDW/dkIu4epdMWTKbcaqvErLlV/nYeSGSQj2KGaX+vvakqz3fztCSszZt9KipIJ5WPqCIpubA7RoaSSepMc0DqaJn5x579+po+H1SZsCdpyNMRNgKpWNHnEeWAdWSuwk70q5tFZPZVMLVq1w0pJBOCCUQLkJoSWCw0Xe4zWQMfTOtqIHachyuZQkdxI9trp8KtHVEnOyrlmTjqrNO+9WIR92qtBVdlKOZllkGDcqtyrI/a1PweLvdt9wRQ0Qne9simBTOO3uQva3nPFWhynsR5eHiWTRHanPtpeepk+2CXMgQpjtJQ+OTL30QZLl8FixoiPLkj0IX9IE2UbtxEQUEPELSsGIAmrQe8B91OxvCYR1D570EcwyNGWfefI0oP8Mfme+gBCIFpaes7uNo8JXmZmhNMKGPrYGcragT2sQSDLD0o58tP+7Ox1k9ka9i+fRbDJlSJ9xFRbpS/0/SQJfM= ceph-83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:46:22.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.015+0000 7fb28c000700 1 Processor -- start 2024-03-14T09:46:22.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.016+0000 7fb28c000700 1 -- start start 2024-03-14T09:46:22.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.016+0000 7fb28c000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb28407a310 0x7fb28407a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:22.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.016+0000 7fb28c000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28407ad00 con 0x7fb28407a310 2024-03-14T09:46:22.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.016+0000 7fb289600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb28407a310 0x7fb28407a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:22.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.016+0000 7fb289600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb28407a310 0x7fb28407a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42800/0 (socket says 172.21.15.78:42800) 2024-03-14T09:46:22.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.016+0000 7fb289600700 1 -- 172.21.15.78:0/2615608079 learned_addr learned my addr 172.21.15.78:0/2615608079 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:22.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.017+0000 7fb289600700 1 -- 172.21.15.78:0/2615608079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28407ae40 con 0x7fb28407a310 2024-03-14T09:46:22.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.017+0000 7fb289600700 1 --2- 172.21.15.78:0/2615608079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb28407a310 0x7fb28407a730 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb2780098a0 tx=0x7fb278009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902e47c45c1cabbd server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:22.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.017+0000 7fb283e00700 1 -- 172.21.15.78:0/2615608079 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb278004160 con 0x7fb28407a310 2024-03-14T09:46:22.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.017+0000 7fb283e00700 1 -- 172.21.15.78:0/2615608079 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2780042c0 con 0x7fb28407a310 2024-03-14T09:46:22.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.017+0000 7fb283e00700 1 -- 172.21.15.78:0/2615608079 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb278011a00 con 0x7fb28407a310 2024-03-14T09:46:22.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.018+0000 7fb28c000700 1 -- 172.21.15.78:0/2615608079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb28407a310 msgr2=0x7fb28407a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:22.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.018+0000 7fb28c000700 1 --2- 172.21.15.78:0/2615608079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb28407a310 0x7fb28407a730 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb2780098a0 tx=0x7fb278009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:22.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.018+0000 7fb28c000700 1 -- 172.21.15.78:0/2615608079 shutdown_connections 2024-03-14T09:46:22.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.018+0000 7fb28c000700 1 --2- 172.21.15.78:0/2615608079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb28407a310 0x7fb28407a730 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:22.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.018+0000 7fb28c000700 1 -- 172.21.15.78:0/2615608079 >> 172.21.15.78:0/2615608079 conn(0x7fb284101a20 msgr2=0x7fb284103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:22.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.018+0000 7fb28c000700 1 -- 172.21.15.78:0/2615608079 shutdown_connections 2024-03-14T09:46:22.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.018+0000 7fb28c000700 1 -- 172.21.15.78:0/2615608079 wait complete. 2024-03-14T09:46:22.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.019+0000 7fb28c000700 1 Processor -- start 2024-03-14T09:46:22.581 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.019+0000 7fb28c000700 1 -- start start 2024-03-14T09:46:22.582 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.019+0000 7fb28c000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb28407a310 0x7fb28419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:22.582 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.019+0000 7fb28c000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28419cdc0 con 0x7fb28407a310 2024-03-14T09:46:22.582 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.020+0000 7fb289600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb28407a310 0x7fb28419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:22.582 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.020+0000 7fb289600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb28407a310 0x7fb28419c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42804/0 (socket says 172.21.15.78:42804) 2024-03-14T09:46:22.582 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.020+0000 7fb289600700 1 -- 172.21.15.78:0/2151587408 learned_addr learned my addr 172.21.15.78:0/2151587408 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:22.582 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.020+0000 7fb289600700 1 -- 172.21.15.78:0/2151587408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb278009550 con 0x7fb28407a310 2024-03-14T09:46:22.582 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.020+0000 7fb289600700 1 --2- 172.21.15.78:0/2151587408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb28407a310 0x7fb28419c880 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb278009130 tx=0x7fb278011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:22.582 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.020+0000 7fb282000700 1 -- 172.21.15.78:0/2151587408 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb278003ab0 con 0x7fb28407a310 2024-03-14T09:46:22.583 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.020+0000 7fb282000700 1 -- 172.21.15.78:0/2151587408 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb278003c10 con 0x7fb28407a310 2024-03-14T09:46:22.583 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.020+0000 7fb282000700 1 -- 172.21.15.78:0/2151587408 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2780247d0 con 0x7fb28407a310 2024-03-14T09:46:22.583 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.021+0000 7fb28c000700 1 -- 172.21.15.78:0/2151587408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb28419cfc0 con 0x7fb28407a310 2024-03-14T09:46:22.583 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.021+0000 7fb28c000700 1 -- 172.21.15.78:0/2151587408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb28419d460 con 0x7fb28407a310 2024-03-14T09:46:22.583 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.022+0000 7fb28c000700 1 -- 172.21.15.78:0/2151587408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb28410cc90 con 0x7fb28407a310 2024-03-14T09:46:22.583 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.022+0000 7fb282000700 1 -- 172.21.15.78:0/2151587408 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45374+0+0 (secure 0 0 0) 0x7fb27801abb0 con 0x7fb28407a310 2024-03-14T09:46:22.583 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.022+0000 7fb282000700 1 --2- 172.21.15.78:0/2151587408 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7fb26c038520 0x7fb26c03a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:22.583 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.022+0000 7fb282000700 1 -- 172.21.15.78:0/2151587408 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb27804c560 con 0x7fb28407a310 2024-03-14T09:46:22.583 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.022+0000 7fb288c00700 1 --2- 172.21.15.78:0/2151587408 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7fb26c038520 0x7fb26c03a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:22.583 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.023+0000 7fb288c00700 1 --2- 172.21.15.78:0/2151587408 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7fb26c038520 0x7fb26c03a9e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb270006fd0 tx=0x7fb270006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:22.583 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.027+0000 7fb282000700 1 -- 172.21.15.78:0/2151587408 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb278018530 con 0x7fb28407a310 2024-03-14T09:46:22.583 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.228+0000 7fb28c000700 1 -- 172.21.15.78:0/2151587408 --> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb284102dc0 con 0x7fb26c038520 2024-03-14T09:46:22.583 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.229+0000 7fb282000700 1 -- 172.21.15.78:0/2151587408 <== mgr.14120 v2:172.21.15.78:6800/3890232539 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fb284102dc0 con 0x7fb26c038520 2024-03-14T09:46:22.583 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.231+0000 7fb28c000700 1 -- 172.21.15.78:0/2151587408 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7fb26c038520 msgr2=0x7fb26c03a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:22.584 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.231+0000 7fb28c000700 1 --2- 172.21.15.78:0/2151587408 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7fb26c038520 0x7fb26c03a9e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb270006fd0 tx=0x7fb270006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:22.584 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.231+0000 7fb28c000700 1 -- 172.21.15.78:0/2151587408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb28407a310 msgr2=0x7fb28419c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:22.584 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.231+0000 7fb28c000700 1 --2- 172.21.15.78:0/2151587408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb28407a310 0x7fb28419c880 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb278009130 tx=0x7fb278011ad0 comp rx=0 tx=0).stop 2024-03-14T09:46:22.584 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.232+0000 7fb28c000700 1 -- 172.21.15.78:0/2151587408 shutdown_connections 2024-03-14T09:46:22.584 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.232+0000 7fb28c000700 1 --2- 172.21.15.78:0/2151587408 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7fb26c038520 0x7fb26c03a9e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:22.584 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.232+0000 7fb28c000700 1 --2- 172.21.15.78:0/2151587408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb28407a310 0x7fb28419c880 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:22.584 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.232+0000 7fb28c000700 1 -- 172.21.15.78:0/2151587408 >> 172.21.15.78:0/2151587408 conn(0x7fb284101a20 msgr2=0x7fb284102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:22.584 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.232+0000 7fb28c000700 1 -- 172.21.15.78:0/2151587408 shutdown_connections 2024-03-14T09:46:22.584 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.232+0000 7fb28c000700 1 -- 172.21.15.78:0/2151587408 wait complete. 2024-03-14T09:46:22.584 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-14T09:46:22.584 INFO:teuthology.orchestra.run.smithi078.stdout:Adding key to root@localhost authorized_keys... 2024-03-14T09:46:22.584 INFO:teuthology.orchestra.run.smithi078.stdout:Adding host smithi078... 2024-03-14T09:46:23.830 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:23 smithi078 ceph-mon[38143]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:46:25.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:24 smithi078 ceph-mon[38143]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi078", "addr": "172.21.15.78", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:46:25.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:25 smithi078 ceph-mon[38143]: Deploying cephadm binary to smithi078 2024-03-14T09:46:26.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Added host 'smithi078' with addr '172.21.15.78' 2024-03-14T09:46:26.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.976+0000 7f2c56000700 1 Processor -- start 2024-03-14T09:46:26.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.977+0000 7f2c56000700 1 -- start start 2024-03-14T09:46:26.481 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.977+0000 7f2c56000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c501068d0 0x7f2c501076b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:26.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.977+0000 7f2c56000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c50106d80 con 0x7f2c501068d0 2024-03-14T09:46:26.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.977+0000 7f2c4f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c501068d0 0x7f2c501076b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:26.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.977+0000 7f2c4f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c501068d0 0x7f2c501076b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42820/0 (socket says 172.21.15.78:42820) 2024-03-14T09:46:26.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.977+0000 7f2c4f400700 1 -- 172.21.15.78:0/2861879267 learned_addr learned my addr 172.21.15.78:0/2861879267 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:26.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.978+0000 7f2c4f400700 1 -- 172.21.15.78:0/2861879267 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c50106ec0 con 0x7f2c501068d0 2024-03-14T09:46:26.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.978+0000 7f2c4f400700 1 --2- 172.21.15.78:0/2861879267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c501068d0 0x7f2c501076b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2c400098a0 tx=0x7f2c40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6df88a39f8ce8b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:26.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.978+0000 7f2c4e000700 1 -- 172.21.15.78:0/2861879267 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c40004160 con 0x7f2c501068d0 2024-03-14T09:46:26.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.978+0000 7f2c4e000700 1 -- 172.21.15.78:0/2861879267 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c400042c0 con 0x7f2c501068d0 2024-03-14T09:46:26.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.978+0000 7f2c4e000700 1 -- 172.21.15.78:0/2861879267 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c40011a00 con 0x7f2c501068d0 2024-03-14T09:46:26.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.979+0000 7f2c56000700 1 -- 172.21.15.78:0/2861879267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c501068d0 msgr2=0x7f2c501076b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:26.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.979+0000 7f2c56000700 1 --2- 172.21.15.78:0/2861879267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c501068d0 0x7f2c501076b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2c400098a0 tx=0x7f2c40009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:26.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.979+0000 7f2c56000700 1 -- 172.21.15.78:0/2861879267 shutdown_connections 2024-03-14T09:46:26.482 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.979+0000 7f2c56000700 1 --2- 172.21.15.78:0/2861879267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c501068d0 0x7f2c501076b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:26.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.979+0000 7f2c56000700 1 -- 172.21.15.78:0/2861879267 >> 172.21.15.78:0/2861879267 conn(0x7f2c50101a70 msgr2=0x7f2c50103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:26.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.979+0000 7f2c56000700 1 -- 172.21.15.78:0/2861879267 shutdown_connections 2024-03-14T09:46:26.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.979+0000 7f2c56000700 1 -- 172.21.15.78:0/2861879267 wait complete. 2024-03-14T09:46:26.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.980+0000 7f2c56000700 1 Processor -- start 2024-03-14T09:46:26.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.980+0000 7f2c56000700 1 -- start start 2024-03-14T09:46:26.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.980+0000 7f2c56000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c501068d0 0x7f2c5007bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:26.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.980+0000 7f2c56000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5007c070 con 0x7f2c501068d0 2024-03-14T09:46:26.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.981+0000 7f2c4f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c501068d0 0x7f2c5007bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:26.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.981+0000 7f2c4f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c501068d0 0x7f2c5007bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42836/0 (socket says 172.21.15.78:42836) 2024-03-14T09:46:26.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.981+0000 7f2c4f400700 1 -- 172.21.15.78:0/3344827465 learned_addr learned my addr 172.21.15.78:0/3344827465 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:26.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.981+0000 7f2c4f400700 1 -- 172.21.15.78:0/3344827465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c40009550 con 0x7f2c501068d0 2024-03-14T09:46:26.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.981+0000 7f2c4f400700 1 --2- 172.21.15.78:0/3344827465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c501068d0 0x7f2c5007bb30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009130 tx=0x7f2c40011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:26.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.981+0000 7f2c47e00700 1 -- 172.21.15.78:0/3344827465 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c40003ab0 con 0x7f2c501068d0 2024-03-14T09:46:26.483 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.981+0000 7f2c47e00700 1 -- 172.21.15.78:0/3344827465 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c40003c10 con 0x7f2c501068d0 2024-03-14T09:46:26.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.981+0000 7f2c47e00700 1 -- 172.21.15.78:0/3344827465 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c400247d0 con 0x7f2c501068d0 2024-03-14T09:46:26.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.982+0000 7f2c56000700 1 -- 172.21.15.78:0/3344827465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c5007a2b0 con 0x7f2c501068d0 2024-03-14T09:46:26.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.982+0000 7f2c56000700 1 -- 172.21.15.78:0/3344827465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c5007a750 con 0x7f2c501068d0 2024-03-14T09:46:26.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.983+0000 7f2c47e00700 1 -- 172.21.15.78:0/3344827465 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45374+0+0 (secure 0 0 0) 0x7f2c4001a430 con 0x7f2c501068d0 2024-03-14T09:46:26.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.983+0000 7f2c45600700 1 -- 172.21.15.78:0/3344827465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c500623e0 con 0x7f2c501068d0 2024-03-14T09:46:26.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.983+0000 7f2c47e00700 1 --2- 172.21.15.78:0/3344827465 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f2c340384d0 0x7f2c3403a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:26.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.983+0000 7f2c47e00700 1 -- 172.21.15.78:0/3344827465 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2c4004c5e0 con 0x7f2c501068d0 2024-03-14T09:46:26.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.983+0000 7f2c4ea00700 1 --2- 172.21.15.78:0/3344827465 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f2c340384d0 0x7f2c3403a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:26.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.984+0000 7f2c4ea00700 1 --2- 172.21.15.78:0/3344827465 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f2c340384d0 0x7f2c3403a990 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c006fd0 tx=0x7f2c3c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:26.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:22.989+0000 7f2c47e00700 1 -- 172.21.15.78:0/3344827465 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2c40018530 con 0x7f2c501068d0 2024-03-14T09:46:26.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:23.198+0000 7f2c45600700 1 -- 172.21.15.78:0/3344827465 --> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi078", "addr": "172.21.15.78", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c5007a990 con 0x7f2c340384d0 2024-03-14T09:46:26.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.110+0000 7f2c47e00700 1 -- 172.21.15.78:0/3344827465 <== mgr.14120 v2:172.21.15.78:6800/3890232539 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f2c5007a990 con 0x7f2c340384d0 2024-03-14T09:46:26.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.113+0000 7f2c45600700 1 -- 172.21.15.78:0/3344827465 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f2c340384d0 msgr2=0x7f2c3403a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:26.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.113+0000 7f2c45600700 1 --2- 172.21.15.78:0/3344827465 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f2c340384d0 0x7f2c3403a990 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c006fd0 tx=0x7f2c3c006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:26.484 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.113+0000 7f2c45600700 1 -- 172.21.15.78:0/3344827465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c501068d0 msgr2=0x7f2c5007bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:26.485 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.113+0000 7f2c45600700 1 --2- 172.21.15.78:0/3344827465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c501068d0 0x7f2c5007bb30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009130 tx=0x7f2c40011ad0 comp rx=0 tx=0).stop 2024-03-14T09:46:26.485 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.113+0000 7f2c45600700 1 -- 172.21.15.78:0/3344827465 shutdown_connections 2024-03-14T09:46:26.485 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.113+0000 7f2c45600700 1 --2- 172.21.15.78:0/3344827465 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f2c340384d0 0x7f2c3403a990 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:26.485 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.113+0000 7f2c45600700 1 --2- 172.21.15.78:0/3344827465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c501068d0 0x7f2c5007bb30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:26.485 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.113+0000 7f2c45600700 1 -- 172.21.15.78:0/3344827465 >> 172.21.15.78:0/3344827465 conn(0x7f2c50101a70 msgr2=0x7f2c50102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:26.485 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.114+0000 7f2c45600700 1 -- 172.21.15.78:0/3344827465 shutdown_connections 2024-03-14T09:46:26.485 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.114+0000 7f2c45600700 1 -- 172.21.15.78:0/3344827465 wait complete. 2024-03-14T09:46:26.485 INFO:teuthology.orchestra.run.smithi078.stdout:Deploying unmanaged mon service... 2024-03-14T09:46:27.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:27 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' 2024-03-14T09:46:27.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:27 smithi078 ceph-mon[38143]: Added host smithi078 2024-03-14T09:46:27.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:27 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:46:27.682 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-14T09:46:27.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.937+0000 7f1d0c600700 1 Processor -- start 2024-03-14T09:46:27.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.937+0000 7f1d0c600700 1 -- start start 2024-03-14T09:46:27.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.937+0000 7f1d0c600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d04107350 0x7f1d04109740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:27.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.937+0000 7f1d0c600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d04079ec0 con 0x7f1d04107350 2024-03-14T09:46:27.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.938+0000 7f1d09c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d04107350 0x7f1d04109740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:27.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.938+0000 7f1d09c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d04107350 0x7f1d04109740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39360/0 (socket says 172.21.15.78:39360) 2024-03-14T09:46:27.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.938+0000 7f1d09c00700 1 -- 172.21.15.78:0/2433181119 learned_addr learned my addr 172.21.15.78:0/2433181119 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:27.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.938+0000 7f1d09c00700 1 -- 172.21.15.78:0/2433181119 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d04109c80 con 0x7f1d04107350 2024-03-14T09:46:27.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.938+0000 7f1d09c00700 1 --2- 172.21.15.78:0/2433181119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d04107350 0x7f1d04109740 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80098a0 tx=0x7f1cf8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=625dc0f0e8d68429 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:27.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.939+0000 7f1d03e00700 1 -- 172.21.15.78:0/2433181119 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cf8004160 con 0x7f1d04107350 2024-03-14T09:46:27.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.939+0000 7f1d03e00700 1 -- 172.21.15.78:0/2433181119 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1cf80042c0 con 0x7f1d04107350 2024-03-14T09:46:27.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.939+0000 7f1d03e00700 1 -- 172.21.15.78:0/2433181119 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cf8011a00 con 0x7f1d04107350 2024-03-14T09:46:27.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.939+0000 7f1d0c600700 1 -- 172.21.15.78:0/2433181119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d04107350 msgr2=0x7f1d04109740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:27.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.939+0000 7f1d0c600700 1 --2- 172.21.15.78:0/2433181119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d04107350 0x7f1d04109740 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80098a0 tx=0x7f1cf8009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:27.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.939+0000 7f1d0c600700 1 -- 172.21.15.78:0/2433181119 shutdown_connections 2024-03-14T09:46:27.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.939+0000 7f1d0c600700 1 --2- 172.21.15.78:0/2433181119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d04107350 0x7f1d04109740 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:27.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.939+0000 7f1d0c600700 1 -- 172.21.15.78:0/2433181119 >> 172.21.15.78:0/2433181119 conn(0x7f1d041009c0 msgr2=0x7f1d04102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:27.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.940+0000 7f1d0c600700 1 -- 172.21.15.78:0/2433181119 shutdown_connections 2024-03-14T09:46:27.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.940+0000 7f1d0c600700 1 -- 172.21.15.78:0/2433181119 wait complete. 2024-03-14T09:46:27.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.941+0000 7f1d0c600700 1 Processor -- start 2024-03-14T09:46:27.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.941+0000 7f1d0c600700 1 -- start start 2024-03-14T09:46:27.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.942+0000 7f1d0c600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d04107350 0x7f1d0419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:27.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.942+0000 7f1d0c600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0419abe0 con 0x7f1d04107350 2024-03-14T09:46:27.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.942+0000 7f1d09c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d04107350 0x7f1d0419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:27.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.942+0000 7f1d09c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d04107350 0x7f1d0419a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39372/0 (socket says 172.21.15.78:39372) 2024-03-14T09:46:27.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.942+0000 7f1d09c00700 1 -- 172.21.15.78:0/1936111144 learned_addr learned my addr 172.21.15.78:0/1936111144 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:27.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.942+0000 7f1d09c00700 1 -- 172.21.15.78:0/1936111144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf8009550 con 0x7f1d04107350 2024-03-14T09:46:27.685 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.943+0000 7f1d09c00700 1 --2- 172.21.15.78:0/1936111144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d04107350 0x7f1d0419a6a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8009130 tx=0x7f1cf8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:27.685 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.943+0000 7f1d02000700 1 -- 172.21.15.78:0/1936111144 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cf80038b0 con 0x7f1d04107350 2024-03-14T09:46:27.685 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.943+0000 7f1d02000700 1 -- 172.21.15.78:0/1936111144 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1cf8003a10 con 0x7f1d04107350 2024-03-14T09:46:27.685 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.943+0000 7f1d0c600700 1 -- 172.21.15.78:0/1936111144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d0419ade0 con 0x7f1d04107350 2024-03-14T09:46:27.685 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.943+0000 7f1d02000700 1 -- 172.21.15.78:0/1936111144 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1cf8019550 con 0x7f1d04107350 2024-03-14T09:46:27.685 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.943+0000 7f1d0c600700 1 -- 172.21.15.78:0/1936111144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0419b280 con 0x7f1d04107350 2024-03-14T09:46:27.685 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.944+0000 7f1d02000700 1 -- 172.21.15.78:0/1936111144 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45374+0+0 (secure 0 0 0) 0x7f1cf80197c0 con 0x7f1d04107350 2024-03-14T09:46:27.685 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.945+0000 7f1d02000700 1 --2- 172.21.15.78:0/1936111144 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f1cec0384b0 0x7f1cec03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:27.685 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.945+0000 7f1d02000700 1 -- 172.21.15.78:0/1936111144 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1cf804c660 con 0x7f1d04107350 2024-03-14T09:46:27.685 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.945+0000 7f1cf7400700 1 -- 172.21.15.78:0/1936111144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce00052f0 con 0x7f1d04107350 2024-03-14T09:46:27.685 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.945+0000 7f1d09200700 1 --2- 172.21.15.78:0/1936111144 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f1cec0384b0 0x7f1cec03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:27.685 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.945+0000 7f1d09200700 1 --2- 172.21.15.78:0/1936111144 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f1cec0384b0 0x7f1cec03a970 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0006fd0 tx=0x7f1cf0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:27.685 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:26.950+0000 7f1d02000700 1 -- 172.21.15.78:0/1936111144 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1cf8003b80 con 0x7f1d04107350 2024-03-14T09:46:27.685 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:27.176+0000 7f1cf7400700 1 -- 172.21.15.78:0/1936111144 --> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1ce0000bc0 con 0x7f1cec0384b0 2024-03-14T09:46:27.686 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:27.182+0000 7f1d02000700 1 -- 172.21.15.78:0/1936111144 <== mgr.14120 v2:172.21.15.78:6800/3890232539 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1ce0000bc0 con 0x7f1cec0384b0 2024-03-14T09:46:27.686 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:27.185+0000 7f1cf7400700 1 -- 172.21.15.78:0/1936111144 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f1cec0384b0 msgr2=0x7f1cec03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:27.686 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:27.185+0000 7f1cf7400700 1 --2- 172.21.15.78:0/1936111144 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f1cec0384b0 0x7f1cec03a970 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0006fd0 tx=0x7f1cf0006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:27.686 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:27.185+0000 7f1cf7400700 1 -- 172.21.15.78:0/1936111144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d04107350 msgr2=0x7f1d0419a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:27.686 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:27.185+0000 7f1cf7400700 1 --2- 172.21.15.78:0/1936111144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d04107350 0x7f1d0419a6a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8009130 tx=0x7f1cf8011ad0 comp rx=0 tx=0).stop 2024-03-14T09:46:27.686 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:27.185+0000 7f1cf7400700 1 -- 172.21.15.78:0/1936111144 shutdown_connections 2024-03-14T09:46:27.686 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:27.185+0000 7f1cf7400700 1 --2- 172.21.15.78:0/1936111144 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f1cec0384b0 0x7f1cec03a970 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:27.686 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:27.185+0000 7f1cf7400700 1 --2- 172.21.15.78:0/1936111144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d04107350 0x7f1d0419a6a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:27.686 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:27.185+0000 7f1cf7400700 1 -- 172.21.15.78:0/1936111144 >> 172.21.15.78:0/1936111144 conn(0x7f1d041009c0 msgr2=0x7f1d04101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:27.686 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:27.185+0000 7f1cf7400700 1 -- 172.21.15.78:0/1936111144 shutdown_connections 2024-03-14T09:46:27.686 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:27.185+0000 7f1cf7400700 1 -- 172.21.15.78:0/1936111144 wait complete. 2024-03-14T09:46:27.686 INFO:teuthology.orchestra.run.smithi078.stdout:Deploying unmanaged mgr service... 2024-03-14T09:46:28.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:28 smithi078 ceph-mon[38143]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:46:28.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:28 smithi078 ceph-mon[38143]: Saving service mon spec with placement count:5 2024-03-14T09:46:28.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:28 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' 2024-03-14T09:46:28.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:28 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' 2024-03-14T09:46:28.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:28 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' 2024-03-14T09:46:28.732 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-14T09:46:28.732 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.102+0000 7f915f800700 1 Processor -- start 2024-03-14T09:46:28.732 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.102+0000 7f915f800700 1 -- start start 2024-03-14T09:46:28.732 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.102+0000 7f915f800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f915807a330 0x7f915807a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:28.732 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.102+0000 7f915f800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915807ad20 con 0x7f915807a330 2024-03-14T09:46:28.732 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.103+0000 7f915ce00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f915807a330 0x7f915807a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:28.732 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.103+0000 7f915ce00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f915807a330 0x7f915807a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39374/0 (socket says 172.21.15.78:39374) 2024-03-14T09:46:28.733 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.103+0000 7f915ce00700 1 -- 172.21.15.78:0/1646059404 learned_addr learned my addr 172.21.15.78:0/1646059404 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:28.733 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.103+0000 7f915ce00700 1 -- 172.21.15.78:0/1646059404 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915807ae60 con 0x7f915807a330 2024-03-14T09:46:28.733 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.103+0000 7f915ce00700 1 --2- 172.21.15.78:0/1646059404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f915807a330 0x7f915807a750 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f914c0098a0 tx=0x7f914c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332d1a605f89b930 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:28.733 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.103+0000 7f9157400700 1 -- 172.21.15.78:0/1646059404 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f914c004160 con 0x7f915807a330 2024-03-14T09:46:28.733 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.104+0000 7f9157400700 1 -- 172.21.15.78:0/1646059404 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f914c0042c0 con 0x7f915807a330 2024-03-14T09:46:28.733 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.104+0000 7f9157400700 1 -- 172.21.15.78:0/1646059404 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f914c0119b0 con 0x7f915807a330 2024-03-14T09:46:28.733 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.104+0000 7f915f800700 1 -- 172.21.15.78:0/1646059404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f915807a330 msgr2=0x7f915807a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:28.733 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.104+0000 7f915f800700 1 --2- 172.21.15.78:0/1646059404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f915807a330 0x7f915807a750 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f914c0098a0 tx=0x7f914c009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:28.733 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.104+0000 7f915f800700 1 -- 172.21.15.78:0/1646059404 shutdown_connections 2024-03-14T09:46:28.733 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.104+0000 7f915f800700 1 --2- 172.21.15.78:0/1646059404 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f915807a330 0x7f915807a750 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:28.733 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.104+0000 7f915f800700 1 -- 172.21.15.78:0/1646059404 >> 172.21.15.78:0/1646059404 conn(0x7f9158101a70 msgr2=0x7f9158103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:28.733 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.104+0000 7f915f800700 1 -- 172.21.15.78:0/1646059404 shutdown_connections 2024-03-14T09:46:28.733 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.104+0000 7f915f800700 1 -- 172.21.15.78:0/1646059404 wait complete. 2024-03-14T09:46:28.734 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.106+0000 7f915f800700 1 Processor -- start 2024-03-14T09:46:28.734 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.106+0000 7f915f800700 1 -- start start 2024-03-14T09:46:28.734 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.106+0000 7f915f800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f915807a330 0x7f915819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:28.734 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.106+0000 7f915f800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915819cd80 con 0x7f915807a330 2024-03-14T09:46:28.734 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.106+0000 7f915ce00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f915807a330 0x7f915819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:28.734 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.106+0000 7f915ce00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f915807a330 0x7f915819c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39382/0 (socket says 172.21.15.78:39382) 2024-03-14T09:46:28.734 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.106+0000 7f915ce00700 1 -- 172.21.15.78:0/96139418 learned_addr learned my addr 172.21.15.78:0/96139418 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:28.734 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.107+0000 7f915ce00700 1 -- 172.21.15.78:0/96139418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c009550 con 0x7f915807a330 2024-03-14T09:46:28.734 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.107+0000 7f915ce00700 1 --2- 172.21.15.78:0/96139418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f915807a330 0x7f915819c840 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f914c006af0 tx=0x7f914c011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:28.734 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.107+0000 7f9155600700 1 -- 172.21.15.78:0/96139418 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f914c003a40 con 0x7f915807a330 2024-03-14T09:46:28.734 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.107+0000 7f9155600700 1 -- 172.21.15.78:0/96139418 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f914c003ba0 con 0x7f915807a330 2024-03-14T09:46:28.734 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.107+0000 7f9155600700 1 -- 172.21.15.78:0/96139418 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f914c0196c0 con 0x7f915807a330 2024-03-14T09:46:28.734 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.107+0000 7f915f800700 1 -- 172.21.15.78:0/96139418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f915819cf80 con 0x7f915807a330 2024-03-14T09:46:28.734 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.107+0000 7f915f800700 1 -- 172.21.15.78:0/96139418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f915819d420 con 0x7f915807a330 2024-03-14T09:46:28.735 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.108+0000 7f9155600700 1 -- 172.21.15.78:0/96139418 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45374+0+0 (secure 0 0 0) 0x7f914c01aa80 con 0x7f915807a330 2024-03-14T09:46:28.735 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.108+0000 7f915f800700 1 -- 172.21.15.78:0/96139418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915810ccd0 con 0x7f915807a330 2024-03-14T09:46:28.735 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.109+0000 7f9155600700 1 --2- 172.21.15.78:0/96139418 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f91400384d0 0x7f914003a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:28.735 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.109+0000 7f9155600700 1 -- 172.21.15.78:0/96139418 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f914c04c330 con 0x7f915807a330 2024-03-14T09:46:28.735 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.109+0000 7f9157e00700 1 --2- 172.21.15.78:0/96139418 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f91400384d0 0x7f914003a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:28.735 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.110+0000 7f9157e00700 1 --2- 172.21.15.78:0/96139418 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f91400384d0 0x7f914003a990 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9144006fd0 tx=0x7f9144006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:28.735 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.114+0000 7f9155600700 1 -- 172.21.15.78:0/96139418 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f914c01a430 con 0x7f915807a330 2024-03-14T09:46:28.735 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.340+0000 7f915f800700 1 -- 172.21.15.78:0/96139418 --> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f915819fcb0 con 0x7f91400384d0 2024-03-14T09:46:28.735 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.345+0000 7f9155600700 1 -- 172.21.15.78:0/96139418 <== mgr.14120 v2:172.21.15.78:6800/3890232539 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f915819fcb0 con 0x7f91400384d0 2024-03-14T09:46:28.735 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.348+0000 7f915f800700 1 -- 172.21.15.78:0/96139418 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f91400384d0 msgr2=0x7f914003a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:28.735 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.348+0000 7f915f800700 1 --2- 172.21.15.78:0/96139418 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f91400384d0 0x7f914003a990 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9144006fd0 tx=0x7f9144006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:28.735 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.348+0000 7f915f800700 1 -- 172.21.15.78:0/96139418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f915807a330 msgr2=0x7f915819c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:28.735 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.348+0000 7f915f800700 1 --2- 172.21.15.78:0/96139418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f915807a330 0x7f915819c840 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f914c006af0 tx=0x7f914c011a10 comp rx=0 tx=0).stop 2024-03-14T09:46:28.736 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.348+0000 7f915f800700 1 -- 172.21.15.78:0/96139418 shutdown_connections 2024-03-14T09:46:28.736 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.348+0000 7f915f800700 1 --2- 172.21.15.78:0/96139418 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f91400384d0 0x7f914003a990 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:28.736 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.348+0000 7f915f800700 1 --2- 172.21.15.78:0/96139418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f915807a330 0x7f915819c840 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:28.736 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.348+0000 7f915f800700 1 -- 172.21.15.78:0/96139418 >> 172.21.15.78:0/96139418 conn(0x7f9158101a70 msgr2=0x7f9158102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:28.736 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.348+0000 7f915f800700 1 -- 172.21.15.78:0/96139418 shutdown_connections 2024-03-14T09:46:28.736 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:28.348+0000 7f915f800700 1 -- 172.21.15.78:0/96139418 wait complete. 2024-03-14T09:46:29.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:29 smithi078 ceph-mon[38143]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:46:29.621 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:29 smithi078 ceph-mon[38143]: Saving service mgr spec with placement count:2 2024-03-14T09:46:29.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:29 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' 2024-03-14T09:46:29.622 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:29 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' 2024-03-14T09:46:29.778 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.162+0000 7fecfcc00700 1 Processor -- start 2024-03-14T09:46:29.778 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.162+0000 7fecfcc00700 1 -- start start 2024-03-14T09:46:29.778 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.162+0000 7fecfcc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecf8108b20 0x7fecf8109900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:29.778 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.162+0000 7fecfcc00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf8108fd0 con 0x7fecf8108b20 2024-03-14T09:46:29.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.163+0000 7fecf5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecf8108b20 0x7fecf8109900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:29.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.163+0000 7fecf5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecf8108b20 0x7fecf8109900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39398/0 (socket says 172.21.15.78:39398) 2024-03-14T09:46:29.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.163+0000 7fecf5e00700 1 -- 172.21.15.78:0/859661870 learned_addr learned my addr 172.21.15.78:0/859661870 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:29.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.163+0000 7fecf5e00700 1 -- 172.21.15.78:0/859661870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf8109110 con 0x7fecf8108b20 2024-03-14T09:46:29.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.163+0000 7fecf5e00700 1 --2- 172.21.15.78:0/859661870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecf8108b20 0x7fecf8109900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fece80098a0 tx=0x7fece8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ac4946b94d8de4 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:29.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.164+0000 7fecf4a00700 1 -- 172.21.15.78:0/859661870 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fece8004160 con 0x7fecf8108b20 2024-03-14T09:46:29.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.164+0000 7fecf4a00700 1 -- 172.21.15.78:0/859661870 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fece80042c0 con 0x7fecf8108b20 2024-03-14T09:46:29.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.164+0000 7fecf4a00700 1 -- 172.21.15.78:0/859661870 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fece80119b0 con 0x7fecf8108b20 2024-03-14T09:46:29.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.164+0000 7fecfcc00700 1 -- 172.21.15.78:0/859661870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecf8108b20 msgr2=0x7fecf8109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:29.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.164+0000 7fecfcc00700 1 --2- 172.21.15.78:0/859661870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecf8108b20 0x7fecf8109900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fece80098a0 tx=0x7fece8009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:29.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.165+0000 7fecfcc00700 1 -- 172.21.15.78:0/859661870 shutdown_connections 2024-03-14T09:46:29.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.165+0000 7fecfcc00700 1 --2- 172.21.15.78:0/859661870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecf8108b20 0x7fecf8109900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:29.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.165+0000 7fecfcc00700 1 -- 172.21.15.78:0/859661870 >> 172.21.15.78:0/859661870 conn(0x7fecf807bab0 msgr2=0x7fecf81064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:29.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.165+0000 7fecfcc00700 1 -- 172.21.15.78:0/859661870 shutdown_connections 2024-03-14T09:46:29.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.165+0000 7fecfcc00700 1 -- 172.21.15.78:0/859661870 wait complete. 2024-03-14T09:46:29.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.166+0000 7fecfcc00700 1 Processor -- start 2024-03-14T09:46:29.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.166+0000 7fecfcc00700 1 -- start start 2024-03-14T09:46:29.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.166+0000 7fecfcc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecf8108b20 0x7fecf819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:29.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.166+0000 7fecfcc00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf819cdd0 con 0x7fecf8108b20 2024-03-14T09:46:29.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.167+0000 7fecf5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecf8108b20 0x7fecf819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:29.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.167+0000 7fecf5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecf8108b20 0x7fecf819c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39404/0 (socket says 172.21.15.78:39404) 2024-03-14T09:46:29.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.167+0000 7fecf5e00700 1 -- 172.21.15.78:0/987170045 learned_addr learned my addr 172.21.15.78:0/987170045 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:29.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.167+0000 7fecf5e00700 1 -- 172.21.15.78:0/987170045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece8009550 con 0x7fecf8108b20 2024-03-14T09:46:29.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.167+0000 7fecf5e00700 1 --2- 172.21.15.78:0/987170045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecf8108b20 0x7fecf819c890 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fece8003e60 tx=0x7fece8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:29.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.167+0000 7feceea00700 1 -- 172.21.15.78:0/987170045 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fece8003a60 con 0x7fecf8108b20 2024-03-14T09:46:29.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.167+0000 7feceea00700 1 -- 172.21.15.78:0/987170045 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fece8003bc0 con 0x7fecf8108b20 2024-03-14T09:46:29.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.167+0000 7fecfcc00700 1 -- 172.21.15.78:0/987170045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf819cfd0 con 0x7fecf8108b20 2024-03-14T09:46:29.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.167+0000 7feceea00700 1 -- 172.21.15.78:0/987170045 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fece80196c0 con 0x7fecf8108b20 2024-03-14T09:46:29.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.168+0000 7fecfcc00700 1 -- 172.21.15.78:0/987170045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf819d470 con 0x7fecf8108b20 2024-03-14T09:46:29.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.168+0000 7feceea00700 1 -- 172.21.15.78:0/987170045 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45374+0+0 (secure 0 0 0) 0x7fece801a430 con 0x7fecf8108b20 2024-03-14T09:46:29.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.169+0000 7feceea00700 1 --2- 172.21.15.78:0/987170045 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7fecdc0384b0 0x7fecdc03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:29.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.169+0000 7feceea00700 1 -- 172.21.15.78:0/987170045 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fece804c6f0 con 0x7fecf8108b20 2024-03-14T09:46:29.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.169+0000 7fecfcc00700 1 -- 172.21.15.78:0/987170045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf80623e0 con 0x7fecf8108b20 2024-03-14T09:46:29.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.170+0000 7fecf5400700 1 --2- 172.21.15.78:0/987170045 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7fecdc0384b0 0x7fecdc03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:29.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.170+0000 7fecf5400700 1 --2- 172.21.15.78:0/987170045 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7fecdc0384b0 0x7fecdc03a970 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fece4006fd0 tx=0x7fece4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:29.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.175+0000 7feceea00700 1 -- 172.21.15.78:0/987170045 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fece801a730 con 0x7fecf8108b20 2024-03-14T09:46:29.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.411+0000 7fecfcc00700 1 -- 172.21.15.78:0/987170045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fecf819fe90 con 0x7fecf8108b20 2024-03-14T09:46:29.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.415+0000 7feceea00700 1 -- 172.21.15.78:0/987170045 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fece8010380 con 0x7fecf8108b20 2024-03-14T09:46:29.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.418+0000 7fecfcc00700 1 -- 172.21.15.78:0/987170045 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7fecdc0384b0 msgr2=0x7fecdc03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:29.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.418+0000 7fecfcc00700 1 --2- 172.21.15.78:0/987170045 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7fecdc0384b0 0x7fecdc03a970 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fece4006fd0 tx=0x7fece4006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:29.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.418+0000 7fecfcc00700 1 -- 172.21.15.78:0/987170045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecf8108b20 msgr2=0x7fecf819c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:29.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.418+0000 7fecfcc00700 1 --2- 172.21.15.78:0/987170045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecf8108b20 0x7fecf819c890 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fece8003e60 tx=0x7fece8011a30 comp rx=0 tx=0).stop 2024-03-14T09:46:29.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.418+0000 7fecfcc00700 1 -- 172.21.15.78:0/987170045 shutdown_connections 2024-03-14T09:46:29.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.418+0000 7fecfcc00700 1 --2- 172.21.15.78:0/987170045 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7fecdc0384b0 0x7fecdc03a970 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:29.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.418+0000 7fecfcc00700 1 --2- 172.21.15.78:0/987170045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fecf8108b20 0x7fecf819c890 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:29.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.418+0000 7fecfcc00700 1 -- 172.21.15.78:0/987170045 >> 172.21.15.78:0/987170045 conn(0x7fecf807bab0 msgr2=0x7fecf8105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:29.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.418+0000 7fecfcc00700 1 -- 172.21.15.78:0/987170045 shutdown_connections 2024-03-14T09:46:29.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:29.418+0000 7fecfcc00700 1 -- 172.21.15.78:0/987170045 wait complete. 2024-03-14T09:46:30.772 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:30 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/987170045' entity='client.admin' 2024-03-14T09:46:30.778 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.205+0000 7eff07200700 1 Processor -- start 2024-03-14T09:46:30.778 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.205+0000 7eff07200700 1 -- start start 2024-03-14T09:46:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.206+0000 7eff07200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff0007a310 0x7eff0007a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.206+0000 7eff07200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0007ad00 con 0x7eff0007a310 2024-03-14T09:46:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.206+0000 7efeffe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff0007a310 0x7eff0007a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.206+0000 7efeffe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff0007a310 0x7eff0007a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39410/0 (socket says 172.21.15.78:39410) 2024-03-14T09:46:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.206+0000 7efeffe00700 1 -- 172.21.15.78:0/124740525 learned_addr learned my addr 172.21.15.78:0/124740525 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.207+0000 7efeffe00700 1 -- 172.21.15.78:0/124740525 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0007ae40 con 0x7eff0007a310 2024-03-14T09:46:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.207+0000 7efeffe00700 1 --2- 172.21.15.78:0/124740525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff0007a310 0x7eff0007a730 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efef00098a0 tx=0x7efef0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac33e2e5d41e0650 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.207+0000 7efefea00700 1 -- 172.21.15.78:0/124740525 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efef0004160 con 0x7eff0007a310 2024-03-14T09:46:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.207+0000 7efefea00700 1 -- 172.21.15.78:0/124740525 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efef00042c0 con 0x7eff0007a310 2024-03-14T09:46:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.207+0000 7efefea00700 1 -- 172.21.15.78:0/124740525 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efef00119b0 con 0x7eff0007a310 2024-03-14T09:46:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.208+0000 7eff07200700 1 -- 172.21.15.78:0/124740525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff0007a310 msgr2=0x7eff0007a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.208+0000 7eff07200700 1 --2- 172.21.15.78:0/124740525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff0007a310 0x7eff0007a730 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efef00098a0 tx=0x7efef0009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.208+0000 7eff07200700 1 -- 172.21.15.78:0/124740525 shutdown_connections 2024-03-14T09:46:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.208+0000 7eff07200700 1 --2- 172.21.15.78:0/124740525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff0007a310 0x7eff0007a730 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.208+0000 7eff07200700 1 -- 172.21.15.78:0/124740525 >> 172.21.15.78:0/124740525 conn(0x7eff00101a70 msgr2=0x7eff00103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.208+0000 7eff07200700 1 -- 172.21.15.78:0/124740525 shutdown_connections 2024-03-14T09:46:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.208+0000 7eff07200700 1 -- 172.21.15.78:0/124740525 wait complete. 2024-03-14T09:46:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.209+0000 7eff07200700 1 Processor -- start 2024-03-14T09:46:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.209+0000 7eff07200700 1 -- start start 2024-03-14T09:46:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.210+0000 7eff07200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff0007a310 0x7eff0019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.210+0000 7eff07200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0019cdd0 con 0x7eff0007a310 2024-03-14T09:46:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.210+0000 7efeffe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff0007a310 0x7eff0019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.210+0000 7efeffe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff0007a310 0x7eff0019c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39412/0 (socket says 172.21.15.78:39412) 2024-03-14T09:46:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.210+0000 7efeffe00700 1 -- 172.21.15.78:0/3925164762 learned_addr learned my addr 172.21.15.78:0/3925164762 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.210+0000 7efeffe00700 1 -- 172.21.15.78:0/3925164762 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef0009550 con 0x7eff0007a310 2024-03-14T09:46:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.211+0000 7efeffe00700 1 --2- 172.21.15.78:0/3925164762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff0007a310 0x7eff0019c890 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efef0003e60 tx=0x7efef0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.211+0000 7efefcc00700 1 -- 172.21.15.78:0/3925164762 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efef0003a60 con 0x7eff0007a310 2024-03-14T09:46:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.211+0000 7efefcc00700 1 -- 172.21.15.78:0/3925164762 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efef0003bc0 con 0x7eff0007a310 2024-03-14T09:46:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.211+0000 7efefcc00700 1 -- 172.21.15.78:0/3925164762 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efef00196c0 con 0x7eff0007a310 2024-03-14T09:46:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.211+0000 7eff07200700 1 -- 172.21.15.78:0/3925164762 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0019cfd0 con 0x7eff0007a310 2024-03-14T09:46:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.211+0000 7eff07200700 1 -- 172.21.15.78:0/3925164762 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0019d470 con 0x7eff0007a310 2024-03-14T09:46:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.212+0000 7eff07200700 1 -- 172.21.15.78:0/3925164762 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0010ccf0 con 0x7eff0007a310 2024-03-14T09:46:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.212+0000 7efefcc00700 1 -- 172.21.15.78:0/3925164762 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45374+0+0 (secure 0 0 0) 0x7efef001a430 con 0x7eff0007a310 2024-03-14T09:46:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.213+0000 7efefcc00700 1 --2- 172.21.15.78:0/3925164762 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7efee4038500 0x7efee403a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.213+0000 7efefcc00700 1 -- 172.21.15.78:0/3925164762 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efef004c600 con 0x7eff0007a310 2024-03-14T09:46:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.213+0000 7efeff400700 1 --2- 172.21.15.78:0/3925164762 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7efee4038500 0x7efee403a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.214+0000 7efeff400700 1 --2- 172.21.15.78:0/3925164762 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7efee4038500 0x7efee403a9c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efeec006fd0 tx=0x7efeec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.218+0000 7efefcc00700 1 -- 172.21.15.78:0/3925164762 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7efef0010380 con 0x7eff0007a310 2024-03-14T09:46:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.427+0000 7eff07200700 1 -- 172.21.15.78:0/3925164762 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7eff0010cf60 con 0x7eff0007a310 2024-03-14T09:46:30.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.431+0000 7efefcc00700 1 -- 172.21.15.78:0/3925164762 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7efef0010380 con 0x7eff0007a310 2024-03-14T09:46:30.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.433+0000 7eff07200700 1 -- 172.21.15.78:0/3925164762 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7efee4038500 msgr2=0x7efee403a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:30.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.433+0000 7eff07200700 1 --2- 172.21.15.78:0/3925164762 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7efee4038500 0x7efee403a9c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efeec006fd0 tx=0x7efeec006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:30.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.433+0000 7eff07200700 1 -- 172.21.15.78:0/3925164762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff0007a310 msgr2=0x7eff0019c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:30.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.433+0000 7eff07200700 1 --2- 172.21.15.78:0/3925164762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff0007a310 0x7eff0019c890 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efef0003e60 tx=0x7efef0011a30 comp rx=0 tx=0).stop 2024-03-14T09:46:30.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.434+0000 7eff07200700 1 -- 172.21.15.78:0/3925164762 shutdown_connections 2024-03-14T09:46:30.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.434+0000 7eff07200700 1 --2- 172.21.15.78:0/3925164762 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7efee4038500 0x7efee403a9c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:30.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.434+0000 7eff07200700 1 --2- 172.21.15.78:0/3925164762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff0007a310 0x7eff0019c890 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:30.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.434+0000 7eff07200700 1 -- 172.21.15.78:0/3925164762 >> 172.21.15.78:0/3925164762 conn(0x7eff00101a70 msgr2=0x7eff00102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:30.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.434+0000 7eff07200700 1 -- 172.21.15.78:0/3925164762 shutdown_connections 2024-03-14T09:46:30.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:30.434+0000 7eff07200700 1 -- 172.21.15.78:0/3925164762 wait complete. 2024-03-14T09:46:30.782 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling the dashboard module... 2024-03-14T09:46:31.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:31 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3925164762' entity='client.admin' 2024-03-14T09:46:31.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:31 smithi078 ceph-mon[38143]: from='mgr.14120 172.21.15.78:0/3820345366' entity='mgr.x' 2024-03-14T09:46:32.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:32 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3253837603' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-14T09:46:32.797 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:32 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: ignoring --setuser ceph since I am not root 2024-03-14T09:46:32.797 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:32 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: ignoring --setgroup ceph since I am not root 2024-03-14T09:46:32.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.194+0000 7f631a000700 1 Processor -- start 2024-03-14T09:46:32.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.194+0000 7f631a000700 1 -- start start 2024-03-14T09:46:32.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.194+0000 7f631a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631407a2f0 0x7f631407a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:32.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.194+0000 7f631a000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631407ace0 con 0x7f631407a2f0 2024-03-14T09:46:32.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.195+0000 7f6313400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631407a2f0 0x7f631407a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:32.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.195+0000 7f6313400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631407a2f0 0x7f631407a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39428/0 (socket says 172.21.15.78:39428) 2024-03-14T09:46:32.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.195+0000 7f6313400700 1 -- 172.21.15.78:0/3235511130 learned_addr learned my addr 172.21.15.78:0/3235511130 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:32.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.195+0000 7f6313400700 1 -- 172.21.15.78:0/3235511130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631407ae20 con 0x7f631407a2f0 2024-03-14T09:46:32.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.195+0000 7f6313400700 1 --2- 172.21.15.78:0/3235511130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631407a2f0 0x7f631407a710 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f63040098a0 tx=0x7f6304009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=668e1e345e8b0878 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:32.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.196+0000 7f6312000700 1 -- 172.21.15.78:0/3235511130 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6304004160 con 0x7f631407a2f0 2024-03-14T09:46:32.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.196+0000 7f6312000700 1 -- 172.21.15.78:0/3235511130 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63040042c0 con 0x7f631407a2f0 2024-03-14T09:46:32.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.196+0000 7f6312000700 1 -- 172.21.15.78:0/3235511130 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f63040119b0 con 0x7f631407a2f0 2024-03-14T09:46:32.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.196+0000 7f631a000700 1 -- 172.21.15.78:0/3235511130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631407a2f0 msgr2=0x7f631407a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:32.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.196+0000 7f631a000700 1 --2- 172.21.15.78:0/3235511130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631407a2f0 0x7f631407a710 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f63040098a0 tx=0x7f6304009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:32.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.197+0000 7f631a000700 1 -- 172.21.15.78:0/3235511130 shutdown_connections 2024-03-14T09:46:32.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.197+0000 7f631a000700 1 --2- 172.21.15.78:0/3235511130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631407a2f0 0x7f631407a710 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:32.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.197+0000 7f631a000700 1 -- 172.21.15.78:0/3235511130 >> 172.21.15.78:0/3235511130 conn(0x7f6314101a70 msgr2=0x7f6314103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:32.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.197+0000 7f631a000700 1 -- 172.21.15.78:0/3235511130 shutdown_connections 2024-03-14T09:46:32.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.197+0000 7f631a000700 1 -- 172.21.15.78:0/3235511130 wait complete. 2024-03-14T09:46:32.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.198+0000 7f631a000700 1 Processor -- start 2024-03-14T09:46:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.199+0000 7f631a000700 1 -- start start 2024-03-14T09:46:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.199+0000 7f631a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631407a2f0 0x7f631419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.199+0000 7f631a000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631419cdc0 con 0x7f631407a2f0 2024-03-14T09:46:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.199+0000 7f6313400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631407a2f0 0x7f631419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.199+0000 7f6313400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631407a2f0 0x7f631419c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39442/0 (socket says 172.21.15.78:39442) 2024-03-14T09:46:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.199+0000 7f6313400700 1 -- 172.21.15.78:0/3253837603 learned_addr learned my addr 172.21.15.78:0/3253837603 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.199+0000 7f6313400700 1 -- 172.21.15.78:0/3253837603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6304009550 con 0x7f631407a2f0 2024-03-14T09:46:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.200+0000 7f6313400700 1 --2- 172.21.15.78:0/3253837603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631407a2f0 0x7f631419c880 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6304011670 tx=0x7f6304011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.200+0000 7f630be00700 1 -- 172.21.15.78:0/3253837603 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6304003b20 con 0x7f631407a2f0 2024-03-14T09:46:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.200+0000 7f630be00700 1 -- 172.21.15.78:0/3253837603 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6304003c80 con 0x7f631407a2f0 2024-03-14T09:46:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.200+0000 7f630be00700 1 -- 172.21.15.78:0/3253837603 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6304019770 con 0x7f631407a2f0 2024-03-14T09:46:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.200+0000 7f631a000700 1 -- 172.21.15.78:0/3253837603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631419cfc0 con 0x7f631407a2f0 2024-03-14T09:46:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.200+0000 7f631a000700 1 -- 172.21.15.78:0/3253837603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f631419d460 con 0x7f631407a2f0 2024-03-14T09:46:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.202+0000 7f630be00700 1 -- 172.21.15.78:0/3253837603 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45374+0+0 (secure 0 0 0) 0x7f630401a430 con 0x7f631407a2f0 2024-03-14T09:46:32.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.202+0000 7f631a000700 1 -- 172.21.15.78:0/3253837603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f631410cd10 con 0x7f631407a2f0 2024-03-14T09:46:32.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.202+0000 7f630be00700 1 --2- 172.21.15.78:0/3253837603 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f62f8038520 0x7f62f803a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:32.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.202+0000 7f630be00700 1 -- 172.21.15.78:0/3253837603 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f630404c630 con 0x7f631407a2f0 2024-03-14T09:46:32.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.202+0000 7f6312a00700 1 --2- 172.21.15.78:0/3253837603 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f62f8038520 0x7f62f803a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:32.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.203+0000 7f6312a00700 1 --2- 172.21.15.78:0/3253837603 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f62f8038520 0x7f62f803a9e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6300006fd0 tx=0x7f6300006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:32.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.208+0000 7f630be00700 1 -- 172.21.15.78:0/3253837603 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f630402d9a0 con 0x7f631407a2f0 2024-03-14T09:46:32.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:31.490+0000 7f631a000700 1 -- 172.21.15.78:0/3253837603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f631410cf50 con 0x7f631407a2f0 2024-03-14T09:46:32.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:32.437+0000 7f630be00700 1 -- 172.21.15.78:0/3253837603 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 9) v1 ==== 45387+0+0 (secure 0 0 0) 0x7f6304022d90 con 0x7f631407a2f0 2024-03-14T09:46:32.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:32.439+0000 7f630be00700 1 -- 172.21.15.78:0/3253837603 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f630400bdf0 con 0x7f631407a2f0 2024-03-14T09:46:32.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:32.442+0000 7f631a000700 1 -- 172.21.15.78:0/3253837603 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f62f8038520 msgr2=0x7f62f803a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:32.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:32.442+0000 7f631a000700 1 --2- 172.21.15.78:0/3253837603 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f62f8038520 0x7f62f803a9e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6300006fd0 tx=0x7f6300006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:32.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:32.442+0000 7f631a000700 1 -- 172.21.15.78:0/3253837603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631407a2f0 msgr2=0x7f631419c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:32.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:32.442+0000 7f631a000700 1 --2- 172.21.15.78:0/3253837603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631407a2f0 0x7f631419c880 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6304011670 tx=0x7f6304011a30 comp rx=0 tx=0).stop 2024-03-14T09:46:32.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:32.443+0000 7f631a000700 1 -- 172.21.15.78:0/3253837603 shutdown_connections 2024-03-14T09:46:32.806 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:32.443+0000 7f631a000700 1 --2- 172.21.15.78:0/3253837603 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f62f8038520 0x7f62f803a9e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:32.806 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:32.443+0000 7f631a000700 1 --2- 172.21.15.78:0/3253837603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f631407a2f0 0x7f631419c880 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:32.806 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:32.443+0000 7f631a000700 1 -- 172.21.15.78:0/3253837603 >> 172.21.15.78:0/3253837603 conn(0x7f6314101a70 msgr2=0x7f6314102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:32.806 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:32.443+0000 7f631a000700 1 -- 172.21.15.78:0/3253837603 shutdown_connections 2024-03-14T09:46:32.806 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:32.443+0000 7f631a000700 1 -- 172.21.15.78:0/3253837603 wait complete. 2024-03-14T09:46:33.430 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:33 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:33.073+0000 7fcb142bc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-14T09:46:33.430 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:33 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:33.197+0000 7fcb142bc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-14T09:46:33.430 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:33 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:33.316+0000 7fcb142bc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-14T09:46:33.721 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:33 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:33.429+0000 7fcb142bc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-14T09:46:33.721 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:33 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3253837603' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-14T09:46:33.722 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:33 smithi078 ceph-mon[38143]: mgrmap e9: x(active, since 15s) 2024-03-14T09:46:33.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-14T09:46:33.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-14T09:46:33.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-14T09:46:33.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-14T09:46:33.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-14T09:46:33.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-14T09:46:33.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.228+0000 7f2e2b600700 1 Processor -- start 2024-03-14T09:46:33.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.228+0000 7f2e2b600700 1 -- start start 2024-03-14T09:46:33.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.229+0000 7f2e2b600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e2407a2d0 0x7f2e2407a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:33.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.229+0000 7f2e2b600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2407acc0 con 0x7f2e2407a2d0 2024-03-14T09:46:33.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.229+0000 7f2e28c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e2407a2d0 0x7f2e2407a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:33.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.229+0000 7f2e28c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e2407a2d0 0x7f2e2407a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39468/0 (socket says 172.21.15.78:39468) 2024-03-14T09:46:33.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.229+0000 7f2e28c00700 1 -- 172.21.15.78:0/2890782291 learned_addr learned my addr 172.21.15.78:0/2890782291 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:33.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.230+0000 7f2e28c00700 1 -- 172.21.15.78:0/2890782291 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2407ae00 con 0x7f2e2407a2d0 2024-03-14T09:46:33.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.230+0000 7f2e28c00700 1 --2- 172.21.15.78:0/2890782291 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e2407a2d0 0x7f2e2407a6f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2e180098a0 tx=0x7f2e18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d8048c68c5890b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:33.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.230+0000 7f2e23400700 1 -- 172.21.15.78:0/2890782291 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e18004160 con 0x7f2e2407a2d0 2024-03-14T09:46:33.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.230+0000 7f2e23400700 1 -- 172.21.15.78:0/2890782291 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e180042c0 con 0x7f2e2407a2d0 2024-03-14T09:46:33.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.230+0000 7f2e23400700 1 -- 172.21.15.78:0/2890782291 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e180119b0 con 0x7f2e2407a2d0 2024-03-14T09:46:33.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.231+0000 7f2e2b600700 1 -- 172.21.15.78:0/2890782291 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e2407a2d0 msgr2=0x7f2e2407a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:33.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.231+0000 7f2e2b600700 1 --2- 172.21.15.78:0/2890782291 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e2407a2d0 0x7f2e2407a6f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2e180098a0 tx=0x7f2e18009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:33.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.231+0000 7f2e2b600700 1 -- 172.21.15.78:0/2890782291 shutdown_connections 2024-03-14T09:46:33.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.231+0000 7f2e2b600700 1 --2- 172.21.15.78:0/2890782291 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e2407a2d0 0x7f2e2407a6f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:33.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.231+0000 7f2e2b600700 1 -- 172.21.15.78:0/2890782291 >> 172.21.15.78:0/2890782291 conn(0x7f2e24101a30 msgr2=0x7f2e24103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:33.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.231+0000 7f2e2b600700 1 -- 172.21.15.78:0/2890782291 shutdown_connections 2024-03-14T09:46:33.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.231+0000 7f2e2b600700 1 -- 172.21.15.78:0/2890782291 wait complete. 2024-03-14T09:46:33.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.232+0000 7f2e2b600700 1 Processor -- start 2024-03-14T09:46:33.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.232+0000 7f2e2b600700 1 -- start start 2024-03-14T09:46:33.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.233+0000 7f2e2b600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e2407a2d0 0x7f2e2419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:33.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.233+0000 7f2e2b600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2419cdc0 con 0x7f2e2407a2d0 2024-03-14T09:46:33.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.233+0000 7f2e28c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e2407a2d0 0x7f2e2419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:33.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.233+0000 7f2e28c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e2407a2d0 0x7f2e2419c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39480/0 (socket says 172.21.15.78:39480) 2024-03-14T09:46:33.876 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.233+0000 7f2e28c00700 1 -- 172.21.15.78:0/3931394192 learned_addr learned my addr 172.21.15.78:0/3931394192 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:33.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.233+0000 7f2e28c00700 1 -- 172.21.15.78:0/3931394192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e18009550 con 0x7f2e2407a2d0 2024-03-14T09:46:33.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.233+0000 7f2e28c00700 1 --2- 172.21.15.78:0/3931394192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e2407a2d0 0x7f2e2419c880 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e18011670 tx=0x7f2e18011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:33.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.234+0000 7f2e21600700 1 -- 172.21.15.78:0/3931394192 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e18003b20 con 0x7f2e2407a2d0 2024-03-14T09:46:33.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.234+0000 7f2e21600700 1 -- 172.21.15.78:0/3931394192 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e18003c80 con 0x7f2e2407a2d0 2024-03-14T09:46:33.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.234+0000 7f2e2b600700 1 -- 172.21.15.78:0/3931394192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2419cfc0 con 0x7f2e2407a2d0 2024-03-14T09:46:33.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.234+0000 7f2e21600700 1 -- 172.21.15.78:0/3931394192 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e18019770 con 0x7f2e2407a2d0 2024-03-14T09:46:33.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.234+0000 7f2e2b600700 1 -- 172.21.15.78:0/3931394192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2419d460 con 0x7f2e2407a2d0 2024-03-14T09:46:33.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.235+0000 7f2e2b600700 1 -- 172.21.15.78:0/3931394192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2410ccd0 con 0x7f2e2407a2d0 2024-03-14T09:46:33.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.235+0000 7f2e21600700 1 -- 172.21.15.78:0/3931394192 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 9) v1 ==== 45387+0+0 (secure 0 0 0) 0x7f2e1801a430 con 0x7f2e2407a2d0 2024-03-14T09:46:33.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.236+0000 7f2e21600700 1 --2- 172.21.15.78:0/3931394192 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f2e0c038570 0x7f2e0c03aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:33.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.236+0000 7f2e21600700 1 -- 172.21.15.78:0/3931394192 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2e1804c750 con 0x7f2e2407a2d0 2024-03-14T09:46:33.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.236+0000 7f2e23e00700 1 -- 172.21.15.78:0/3931394192 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f2e0c038570 msgr2=0x7f2e0c03aa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3890232539 2024-03-14T09:46:33.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.236+0000 7f2e23e00700 1 --2- 172.21.15.78:0/3931394192 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f2e0c038570 0x7f2e0c03aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T09:46:33.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.241+0000 7f2e21600700 1 -- 172.21.15.78:0/3931394192 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2e1802d9a0 con 0x7f2e2407a2d0 2024-03-14T09:46:33.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.437+0000 7f2e23e00700 1 -- 172.21.15.78:0/3931394192 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f2e0c038570 msgr2=0x7f2e0c03aa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3890232539 2024-03-14T09:46:33.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.437+0000 7f2e23e00700 1 --2- 172.21.15.78:0/3931394192 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f2e0c038570 0x7f2e0c03aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-14T09:46:33.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.506+0000 7f2e2b600700 1 -- 172.21.15.78:0/3931394192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f2e2419fd50 con 0x7f2e2407a2d0 2024-03-14T09:46:33.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.506+0000 7f2e21600700 1 -- 172.21.15.78:0/3931394192 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f2e1802d350 con 0x7f2e2407a2d0 2024-03-14T09:46:33.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.509+0000 7f2e2b600700 1 -- 172.21.15.78:0/3931394192 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f2e0c038570 msgr2=0x7f2e0c03aa30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T09:46:33.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.509+0000 7f2e2b600700 1 --2- 172.21.15.78:0/3931394192 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f2e0c038570 0x7f2e0c03aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:33.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.509+0000 7f2e2b600700 1 -- 172.21.15.78:0/3931394192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e2407a2d0 msgr2=0x7f2e2419c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:33.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.509+0000 7f2e2b600700 1 --2- 172.21.15.78:0/3931394192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e2407a2d0 0x7f2e2419c880 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e18011670 tx=0x7f2e18011a30 comp rx=0 tx=0).stop 2024-03-14T09:46:33.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.509+0000 7f2e2b600700 1 -- 172.21.15.78:0/3931394192 shutdown_connections 2024-03-14T09:46:33.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.509+0000 7f2e2b600700 1 --2- 172.21.15.78:0/3931394192 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f2e0c038570 0x7f2e0c03aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:33.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.509+0000 7f2e2b600700 1 --2- 172.21.15.78:0/3931394192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e2407a2d0 0x7f2e2419c880 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:33.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.509+0000 7f2e2b600700 1 -- 172.21.15.78:0/3931394192 >> 172.21.15.78:0/3931394192 conn(0x7f2e24101a30 msgr2=0x7f2e24102630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:33.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.509+0000 7f2e2b600700 1 -- 172.21.15.78:0/3931394192 shutdown_connections 2024-03-14T09:46:33.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:33.509+0000 7f2e2b600700 1 -- 172.21.15.78:0/3931394192 wait complete. 2024-03-14T09:46:33.879 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for the mgr to restart... 2024-03-14T09:46:33.879 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr epoch 9... 2024-03-14T09:46:34.303 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:34 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:34.286+0000 7fcb142bc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-14T09:46:34.710 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:34 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3931394192' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-14T09:46:34.711 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:34 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:34.600+0000 7fcb142bc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-14T09:46:35.040 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:34 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:34.709+0000 7fcb142bc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-14T09:46:35.041 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:34 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:34.846+0000 7fcb142bc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-14T09:46:35.303 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:35 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:35.040+0000 7fcb142bc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-14T09:46:35.303 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:35 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:35.203+0000 7fcb142bc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-14T09:46:35.772 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:35 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:35.313+0000 7fcb142bc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-14T09:46:36.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:35 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:35.771+0000 7fcb142bc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-14T09:46:36.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:36 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:36.252+0000 7fcb142bc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-14T09:46:36.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:36 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:36.496+0000 7fcb142bc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-14T09:46:37.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:36 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:36.616+0000 7fcb142bc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-14T09:46:37.803 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:37 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:37.547+0000 7fcb142bc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-14T09:46:37.803 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:37 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:37.671+0000 7fcb142bc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-14T09:46:38.303 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:37 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:37.859+0000 7fcb142bc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-14T09:46:39.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:38 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:38.650+0000 7fcb142bc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-14T09:46:39.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:38 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:38.886+0000 7fcb142bc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-14T09:46:39.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:39 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:39.136+0000 7fcb142bc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-14T09:46:40.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:40 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:40.251+0000 7fcb142bc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-14T09:46:40.985 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:40 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:40.617+0000 7fcb142bc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-14T09:46:40.985 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:40 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:40.733+0000 7fcb142bc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-14T09:46:41.303 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:46:40 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:46:40.984+0000 7fcb142bc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-14T09:46:41.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:41 smithi078 ceph-mon[38143]: Active manager daemon x restarted 2024-03-14T09:46:41.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:41 smithi078 ceph-mon[38143]: Activating manager daemon x 2024-03-14T09:46:41.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:41 smithi078 ceph-mon[38143]: osdmap e3: 0 total, 0 up, 0 in 2024-03-14T09:46:41.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:41 smithi078 ceph-mon[38143]: mgrmap e10: x(active, starting, since 0.00383081s) 2024-03-14T09:46:41.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:46:41.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-14T09:46:41.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-14T09:46:41.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-14T09:46:41.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-14T09:46:41.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:41 smithi078 ceph-mon[38143]: Manager daemon x is now available 2024-03-14T09:46:41.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:46:41.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:46:42.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-14T09:46:42.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-14T09:46:42.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:42 smithi078 ceph-mon[38143]: [14/Mar/2024:09:46:41] ENGINE Bus STARTING 2024-03-14T09:46:42.281 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:42 smithi078 ceph-mon[38143]: mgrmap e11: x(active, since 1.00731s) 2024-03-14T09:46:42.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2024-03-14T09:46:42.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-14T09:46:42.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-14T09:46:42.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2024-03-14T09:46:42.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.292+0000 7f47e6c00700 1 Processor -- start 2024-03-14T09:46:42.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.292+0000 7f47e6c00700 1 -- start start 2024-03-14T09:46:42.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.292+0000 7f47e6c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e007a2f0 0x7f47e007a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:42.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.292+0000 7f47e6c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e007ace0 con 0x7f47e007a2f0 2024-03-14T09:46:42.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.293+0000 7f47dfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e007a2f0 0x7f47e007a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:42.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.293+0000 7f47dfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e007a2f0 0x7f47e007a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39164/0 (socket says 172.21.15.78:39164) 2024-03-14T09:46:42.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.293+0000 7f47dfe00700 1 -- 172.21.15.78:0/405136245 learned_addr learned my addr 172.21.15.78:0/405136245 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:42.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.293+0000 7f47dfe00700 1 -- 172.21.15.78:0/405136245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e007ae20 con 0x7f47e007a2f0 2024-03-14T09:46:42.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.293+0000 7f47dfe00700 1 --2- 172.21.15.78:0/405136245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e007a2f0 0x7f47e007a710 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f47d00098a0 tx=0x7f47d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be80bf40f3185c63 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:42.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.294+0000 7f47dea00700 1 -- 172.21.15.78:0/405136245 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47d0004160 con 0x7f47e007a2f0 2024-03-14T09:46:42.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.294+0000 7f47dea00700 1 -- 172.21.15.78:0/405136245 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f47d00042c0 con 0x7f47e007a2f0 2024-03-14T09:46:42.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.294+0000 7f47dea00700 1 -- 172.21.15.78:0/405136245 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47d00119b0 con 0x7f47e007a2f0 2024-03-14T09:46:42.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.294+0000 7f47e6c00700 1 -- 172.21.15.78:0/405136245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e007a2f0 msgr2=0x7f47e007a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:42.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.294+0000 7f47e6c00700 1 --2- 172.21.15.78:0/405136245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e007a2f0 0x7f47e007a710 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f47d00098a0 tx=0x7f47d0009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:42.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.294+0000 7f47e6c00700 1 -- 172.21.15.78:0/405136245 shutdown_connections 2024-03-14T09:46:42.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.294+0000 7f47e6c00700 1 --2- 172.21.15.78:0/405136245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e007a2f0 0x7f47e007a710 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:42.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.294+0000 7f47e6c00700 1 -- 172.21.15.78:0/405136245 >> 172.21.15.78:0/405136245 conn(0x7f47e0101a30 msgr2=0x7f47e0103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:42.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.295+0000 7f47e6c00700 1 -- 172.21.15.78:0/405136245 shutdown_connections 2024-03-14T09:46:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.295+0000 7f47e6c00700 1 -- 172.21.15.78:0/405136245 wait complete. 2024-03-14T09:46:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.296+0000 7f47e6c00700 1 Processor -- start 2024-03-14T09:46:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.296+0000 7f47e6c00700 1 -- start start 2024-03-14T09:46:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.297+0000 7f47e6c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e007a2f0 0x7f47e019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.297+0000 7f47e6c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e019cda0 con 0x7f47e007a2f0 2024-03-14T09:46:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.297+0000 7f47dfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e007a2f0 0x7f47e019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.297+0000 7f47dfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e007a2f0 0x7f47e019c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39176/0 (socket says 172.21.15.78:39176) 2024-03-14T09:46:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.297+0000 7f47dfe00700 1 -- 172.21.15.78:0/339483468 learned_addr learned my addr 172.21.15.78:0/339483468 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.297+0000 7f47dfe00700 1 -- 172.21.15.78:0/339483468 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d0009550 con 0x7f47e007a2f0 2024-03-14T09:46:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.297+0000 7f47dfe00700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e007a2f0 0x7f47e019c860 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f47d0011670 tx=0x7f47d0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.298+0000 7f47dcc00700 1 -- 172.21.15.78:0/339483468 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47d0003b20 con 0x7f47e007a2f0 2024-03-14T09:46:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.298+0000 7f47dcc00700 1 -- 172.21.15.78:0/339483468 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f47d0003c80 con 0x7f47e007a2f0 2024-03-14T09:46:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.298+0000 7f47e6c00700 1 -- 172.21.15.78:0/339483468 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e019cfa0 con 0x7f47e007a2f0 2024-03-14T09:46:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.298+0000 7f47dcc00700 1 -- 172.21.15.78:0/339483468 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47d0019770 con 0x7f47e007a2f0 2024-03-14T09:46:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.298+0000 7f47e6c00700 1 -- 172.21.15.78:0/339483468 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e019d440 con 0x7f47e007a2f0 2024-03-14T09:46:42.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.299+0000 7f47dcc00700 1 -- 172.21.15.78:0/339483468 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 9) v1 ==== 45387+0+0 (secure 0 0 0) 0x7f47d001ab60 con 0x7f47e007a2f0 2024-03-14T09:46:42.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.300+0000 7f47dcc00700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f47c4038570 0x7f47c403aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:42.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.300+0000 7f47dcc00700 1 -- 172.21.15.78:0/339483468 --> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47c403b140 con 0x7f47c4038570 2024-03-14T09:46:42.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.300+0000 7f47dcc00700 1 -- 172.21.15.78:0/339483468 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f47d004c6e0 con 0x7f47e007a2f0 2024-03-14T09:46:42.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.300+0000 7f47df400700 1 -- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f47c4038570 msgr2=0x7f47c403aa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3890232539 2024-03-14T09:46:42.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.300+0000 7f47df400700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f47c4038570 0x7f47c403aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T09:46:42.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.501+0000 7f47df400700 1 -- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f47c4038570 msgr2=0x7f47c403aa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3890232539 2024-03-14T09:46:42.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.501+0000 7f47df400700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f47c4038570 0x7f47c403aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-14T09:46:42.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.902+0000 7f47df400700 1 -- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f47c4038570 msgr2=0x7f47c403aa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3890232539 2024-03-14T09:46:42.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:34.902+0000 7f47df400700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f47c4038570 0x7f47c403aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-14T09:46:42.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:35.703+0000 7f47df400700 1 -- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f47c4038570 msgr2=0x7f47c403aa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3890232539 2024-03-14T09:46:42.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:35.703+0000 7f47df400700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f47c4038570 0x7f47c403aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-14T09:46:42.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:37.304+0000 7f47df400700 1 -- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f47c4038570 msgr2=0x7f47c403aa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3890232539 2024-03-14T09:46:42.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:37.304+0000 7f47df400700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f47c4038570 0x7f47c403aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-14T09:46:42.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:40.508+0000 7f47df400700 1 -- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f47c4038570 msgr2=0x7f47c403aa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3890232539 2024-03-14T09:46:42.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:40.508+0000 7f47df400700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f47c4038570 0x7f47c403aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-14T09:46:42.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:40.989+0000 7f47dcc00700 1 -- 172.21.15.78:0/339483468 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mgrmap(e 10) v1 ==== 45154+0+0 (secure 0 0 0) 0x7f47d002d350 con 0x7f47e007a2f0 2024-03-14T09:46:42.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:40.989+0000 7f47dcc00700 1 -- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f47c4038570 msgr2=0x7f47c403aa30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T09:46:42.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:40.989+0000 7f47dcc00700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f47c4038570 0x7f47c403aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:42.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:41.992+0000 7f47dcc00700 1 -- 172.21.15.78:0/339483468 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 11) v1 ==== 45281+0+0 (secure 0 0 0) 0x7f47d004e270 con 0x7f47e007a2f0 2024-03-14T09:46:42.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:41.992+0000 7f47dcc00700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f47c403bfe0 0x7f47c403e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:42.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:41.992+0000 7f47dcc00700 1 -- 172.21.15.78:0/339483468 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47c403b140 con 0x7f47c403bfe0 2024-03-14T09:46:42.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:41.993+0000 7f47df400700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f47c403bfe0 0x7f47c403e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:42.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:41.994+0000 7f47df400700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f47c403bfe0 0x7f47c403e3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f47cc003a10 tx=0x7f47cc009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:42.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:41.995+0000 7f47dcc00700 1 -- 172.21.15.78:0/339483468 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f47c403b140 con 0x7f47c403bfe0 2024-03-14T09:46:42.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.001+0000 7f47e6c00700 1 -- 172.21.15.78:0/339483468 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f47e010cd10 con 0x7f47c403bfe0 2024-03-14T09:46:42.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.001+0000 7f47dcc00700 1 -- 172.21.15.78:0/339483468 <== mgr.14152 v2:172.21.15.78:6800/2580616009 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f47e010cd10 con 0x7f47c403bfe0 2024-03-14T09:46:42.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.002+0000 7f47e6c00700 1 -- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f47c403bfe0 msgr2=0x7f47c403e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:42.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.002+0000 7f47e6c00700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f47c403bfe0 0x7f47c403e3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f47cc003a10 tx=0x7f47cc009270 comp rx=0 tx=0).stop 2024-03-14T09:46:42.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.002+0000 7f47e6c00700 1 -- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e007a2f0 msgr2=0x7f47e019c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:42.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.002+0000 7f47e6c00700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e007a2f0 0x7f47e019c860 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f47d0011670 tx=0x7f47d0011a30 comp rx=0 tx=0).stop 2024-03-14T09:46:42.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.002+0000 7f47e6c00700 1 -- 172.21.15.78:0/339483468 shutdown_connections 2024-03-14T09:46:42.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.002+0000 7f47e6c00700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f47c403bfe0 0x7f47c403e3d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:42.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.002+0000 7f47e6c00700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47e007a2f0 0x7f47e019c860 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:42.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.002+0000 7f47e6c00700 1 --2- 172.21.15.78:0/339483468 >> [v2:172.21.15.78:6800/3890232539,v1:172.21.15.78:6801/3890232539] conn(0x7f47c4038570 0x7f47c403aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:42.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.002+0000 7f47e6c00700 1 -- 172.21.15.78:0/339483468 >> 172.21.15.78:0/339483468 conn(0x7f47e0101a30 msgr2=0x7f47e01025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:42.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.002+0000 7f47e6c00700 1 -- 172.21.15.78:0/339483468 shutdown_connections 2024-03-14T09:46:42.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.002+0000 7f47e6c00700 1 -- 172.21.15.78:0/339483468 wait complete. 2024-03-14T09:46:42.455 INFO:teuthology.orchestra.run.smithi078.stdout:mgr epoch 9 is available 2024-03-14T09:46:42.455 INFO:teuthology.orchestra.run.smithi078.stdout:Generating a dashboard self-signed certificate... 2024-03-14T09:46:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-14T09:46:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.869+0000 7f568d600700 1 Processor -- start 2024-03-14T09:46:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.869+0000 7f568d600700 1 -- start start 2024-03-14T09:46:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.869+0000 7f568d600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f568807a310 0x7f568807a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.869+0000 7f568d600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568807ad00 con 0x7f568807a310 2024-03-14T09:46:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.870+0000 7f5686a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f568807a310 0x7f568807a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.870+0000 7f5686a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f568807a310 0x7f568807a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39254/0 (socket says 172.21.15.78:39254) 2024-03-14T09:46:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.870+0000 7f5686a00700 1 -- 172.21.15.78:0/1980389465 learned_addr learned my addr 172.21.15.78:0/1980389465 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.870+0000 7f5686a00700 1 -- 172.21.15.78:0/1980389465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568807ae40 con 0x7f568807a310 2024-03-14T09:46:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.871+0000 7f5686a00700 1 --2- 172.21.15.78:0/1980389465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f568807a310 0x7f568807a730 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f56780098a0 tx=0x7f5678009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70567f4bbc2629cc server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.871+0000 7f5685600700 1 -- 172.21.15.78:0/1980389465 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5678004160 con 0x7f568807a310 2024-03-14T09:46:43.547 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.871+0000 7f5685600700 1 -- 172.21.15.78:0/1980389465 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56780042c0 con 0x7f568807a310 2024-03-14T09:46:43.547 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.871+0000 7f5685600700 1 -- 172.21.15.78:0/1980389465 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56780119b0 con 0x7f568807a310 2024-03-14T09:46:43.547 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.872+0000 7f568d600700 1 -- 172.21.15.78:0/1980389465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f568807a310 msgr2=0x7f568807a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:43.547 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.872+0000 7f568d600700 1 --2- 172.21.15.78:0/1980389465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f568807a310 0x7f568807a730 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f56780098a0 tx=0x7f5678009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:43.547 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.872+0000 7f568d600700 1 -- 172.21.15.78:0/1980389465 shutdown_connections 2024-03-14T09:46:43.547 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.872+0000 7f568d600700 1 --2- 172.21.15.78:0/1980389465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f568807a310 0x7f568807a730 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:43.547 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.872+0000 7f568d600700 1 -- 172.21.15.78:0/1980389465 >> 172.21.15.78:0/1980389465 conn(0x7f5688101a20 msgr2=0x7f5688103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:43.547 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.872+0000 7f568d600700 1 -- 172.21.15.78:0/1980389465 shutdown_connections 2024-03-14T09:46:43.547 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.872+0000 7f568d600700 1 -- 172.21.15.78:0/1980389465 wait complete. 2024-03-14T09:46:43.547 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.873+0000 7f568d600700 1 Processor -- start 2024-03-14T09:46:43.547 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.873+0000 7f568d600700 1 -- start start 2024-03-14T09:46:43.547 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.874+0000 7f568d600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f568807a310 0x7f568819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:43.547 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.874+0000 7f568d600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568819cde0 con 0x7f568807a310 2024-03-14T09:46:43.547 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.874+0000 7f5686a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f568807a310 0x7f568819c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:43.548 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.874+0000 7f5686a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f568807a310 0x7f568819c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39266/0 (socket says 172.21.15.78:39266) 2024-03-14T09:46:43.548 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.874+0000 7f5686a00700 1 -- 172.21.15.78:0/2709617211 learned_addr learned my addr 172.21.15.78:0/2709617211 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:43.548 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.874+0000 7f5686a00700 1 -- 172.21.15.78:0/2709617211 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5678009550 con 0x7f568807a310 2024-03-14T09:46:43.548 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.874+0000 7f5686a00700 1 --2- 172.21.15.78:0/2709617211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f568807a310 0x7f568819c8a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5678003e60 tx=0x7f5678011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:43.548 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.875+0000 7f567f400700 1 -- 172.21.15.78:0/2709617211 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5678003a60 con 0x7f568807a310 2024-03-14T09:46:43.548 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.875+0000 7f567f400700 1 -- 172.21.15.78:0/2709617211 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5678003bc0 con 0x7f568807a310 2024-03-14T09:46:43.548 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.875+0000 7f568d600700 1 -- 172.21.15.78:0/2709617211 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568819cfe0 con 0x7f568807a310 2024-03-14T09:46:43.548 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.875+0000 7f567f400700 1 -- 172.21.15.78:0/2709617211 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56780196c0 con 0x7f568807a310 2024-03-14T09:46:43.548 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.875+0000 7f568d600700 1 -- 172.21.15.78:0/2709617211 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568819d480 con 0x7f568807a310 2024-03-14T09:46:43.548 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.876+0000 7f568d600700 1 -- 172.21.15.78:0/2709617211 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f568810cce0 con 0x7f568807a310 2024-03-14T09:46:43.548 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.876+0000 7f567f400700 1 -- 172.21.15.78:0/2709617211 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 11) v1 ==== 45281+0+0 (secure 0 0 0) 0x7f567801ab60 con 0x7f568807a310 2024-03-14T09:46:43.548 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.876+0000 7f567f400700 1 --2- 172.21.15.78:0/2709617211 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f566c038440 0x7f566c03a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:43.548 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.876+0000 7f567f400700 1 -- 172.21.15.78:0/2709617211 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f567804c5f0 con 0x7f568807a310 2024-03-14T09:46:43.549 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.877+0000 7f5686000700 1 --2- 172.21.15.78:0/2709617211 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f566c038440 0x7f566c03a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:43.549 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.877+0000 7f5686000700 1 --2- 172.21.15.78:0/2709617211 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f566c038440 0x7f566c03a900 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5674006fd0 tx=0x7f5674006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:43.549 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:42.882+0000 7f567f400700 1 -- 172.21.15.78:0/2709617211 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5678010790 con 0x7f568807a310 2024-03-14T09:46:43.549 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:43.099+0000 7f568d600700 1 -- 172.21.15.78:0/2709617211 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f568819fd10 con 0x7f566c038440 2024-03-14T09:46:43.549 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:43.202+0000 7f567f400700 1 -- 172.21.15.78:0/2709617211 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f568819fd10 con 0x7f566c038440 2024-03-14T09:46:43.549 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:43.204+0000 7f568d600700 1 -- 172.21.15.78:0/2709617211 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f566c038440 msgr2=0x7f566c03a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:43.549 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:43.204+0000 7f568d600700 1 --2- 172.21.15.78:0/2709617211 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f566c038440 0x7f566c03a900 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5674006fd0 tx=0x7f5674006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:43.549 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:43.204+0000 7f568d600700 1 -- 172.21.15.78:0/2709617211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f568807a310 msgr2=0x7f568819c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:43.549 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:43.204+0000 7f568d600700 1 --2- 172.21.15.78:0/2709617211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f568807a310 0x7f568819c8a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5678003e60 tx=0x7f5678011a30 comp rx=0 tx=0).stop 2024-03-14T09:46:43.549 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:43.204+0000 7f568d600700 1 -- 172.21.15.78:0/2709617211 shutdown_connections 2024-03-14T09:46:43.549 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:43.204+0000 7f568d600700 1 --2- 172.21.15.78:0/2709617211 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f566c038440 0x7f566c03a900 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:43.549 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:43.204+0000 7f568d600700 1 --2- 172.21.15.78:0/2709617211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f568807a310 0x7f568819c8a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:43.549 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:43.204+0000 7f568d600700 1 -- 172.21.15.78:0/2709617211 >> 172.21.15.78:0/2709617211 conn(0x7f5688101a20 msgr2=0x7f5688102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:43.549 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:43.205+0000 7f568d600700 1 -- 172.21.15.78:0/2709617211 shutdown_connections 2024-03-14T09:46:43.549 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:43.205+0000 7f568d600700 1 -- 172.21.15.78:0/2709617211 wait complete. 2024-03-14T09:46:43.550 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial admin user... 2024-03-14T09:46:43.769 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:43 smithi078 ceph-mon[38143]: [14/Mar/2024:09:46:41] ENGINE Serving on https://172.21.15.78:7150 2024-03-14T09:46:43.769 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:43 smithi078 ceph-mon[38143]: [14/Mar/2024:09:46:41] ENGINE Serving on http://172.21.15.78:8765 2024-03-14T09:46:43.769 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:43 smithi078 ceph-mon[38143]: [14/Mar/2024:09:46:41] ENGINE Bus STARTED 2024-03-14T09:46:43.769 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:43 smithi078 ceph-mon[38143]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-14T09:46:43.769 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:43 smithi078 ceph-mon[38143]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-14T09:46:43.769 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:46:43.769 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:46:43.769 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:46:44.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:44 smithi078 ceph-mon[38143]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:46:44.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:44 smithi078 ceph-mon[38143]: mgrmap e12: x(active, since 2s) 2024-03-14T09:46:44.795 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:44 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:46:44.945 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$AJr6ZjHti/wd0VENVLNjvOHgs7Zw6si7Q85f/B5zbIcjsFgADotaG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710409604, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-14T09:46:44.945 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.009+0000 7f156fe00700 1 Processor -- start 2024-03-14T09:46:44.945 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.009+0000 7f156fe00700 1 -- start start 2024-03-14T09:46:44.945 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.009+0000 7f156fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570106720 0x7f1570107500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:44.945 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.010+0000 7f156fe00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1570106bd0 con 0x7f1570106720 2024-03-14T09:46:44.945 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.010+0000 7f156ea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570106720 0x7f1570107500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:44.945 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.010+0000 7f156ea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570106720 0x7f1570107500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40566/0 (socket says 172.21.15.78:40566) 2024-03-14T09:46:44.945 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.010+0000 7f156ea00700 1 -- 172.21.15.78:0/4280469368 learned_addr learned my addr 172.21.15.78:0/4280469368 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:44.945 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.010+0000 7f156ea00700 1 -- 172.21.15.78:0/4280469368 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1570106d10 con 0x7f1570106720 2024-03-14T09:46:44.945 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.011+0000 7f156ea00700 1 --2- 172.21.15.78:0/4280469368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570106720 0x7f1570107500 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f15600098a0 tx=0x7f1560009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c46a724d42c85c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:44.945 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.011+0000 7f156d600700 1 -- 172.21.15.78:0/4280469368 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1560004160 con 0x7f1570106720 2024-03-14T09:46:44.946 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.011+0000 7f156d600700 1 -- 172.21.15.78:0/4280469368 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f15600042c0 con 0x7f1570106720 2024-03-14T09:46:44.946 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.011+0000 7f156d600700 1 -- 172.21.15.78:0/4280469368 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15600119b0 con 0x7f1570106720 2024-03-14T09:46:44.946 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.011+0000 7f156fe00700 1 -- 172.21.15.78:0/4280469368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570106720 msgr2=0x7f1570107500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:44.946 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.011+0000 7f156fe00700 1 --2- 172.21.15.78:0/4280469368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570106720 0x7f1570107500 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f15600098a0 tx=0x7f1560009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:44.946 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.012+0000 7f156fe00700 1 -- 172.21.15.78:0/4280469368 shutdown_connections 2024-03-14T09:46:44.946 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.012+0000 7f156fe00700 1 --2- 172.21.15.78:0/4280469368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570106720 0x7f1570107500 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:44.946 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.012+0000 7f156fe00700 1 -- 172.21.15.78:0/4280469368 >> 172.21.15.78:0/4280469368 conn(0x7f15701018c0 msgr2=0x7f1570103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:44.946 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.012+0000 7f156fe00700 1 -- 172.21.15.78:0/4280469368 shutdown_connections 2024-03-14T09:46:44.946 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.012+0000 7f156fe00700 1 -- 172.21.15.78:0/4280469368 wait complete. 2024-03-14T09:46:44.946 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.013+0000 7f156fe00700 1 Processor -- start 2024-03-14T09:46:44.946 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.013+0000 7f156fe00700 1 -- start start 2024-03-14T09:46:44.946 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.013+0000 7f156fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570106720 0x7f157019c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:44.947 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.013+0000 7f156fe00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157019cc70 con 0x7f1570106720 2024-03-14T09:46:44.947 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.014+0000 7f156ea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570106720 0x7f157019c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:44.947 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.014+0000 7f156ea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570106720 0x7f157019c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40574/0 (socket says 172.21.15.78:40574) 2024-03-14T09:46:44.947 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.014+0000 7f156ea00700 1 -- 172.21.15.78:0/1378580807 learned_addr learned my addr 172.21.15.78:0/1378580807 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:44.947 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.014+0000 7f156ea00700 1 -- 172.21.15.78:0/1378580807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1560009550 con 0x7f1570106720 2024-03-14T09:46:44.947 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.014+0000 7f156ea00700 1 --2- 172.21.15.78:0/1378580807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570106720 0x7f157019c730 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1560011670 tx=0x7f1560011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:44.947 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.015+0000 7f1567400700 1 -- 172.21.15.78:0/1378580807 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1560003af0 con 0x7f1570106720 2024-03-14T09:46:44.947 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.015+0000 7f156fe00700 1 -- 172.21.15.78:0/1378580807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157019ce70 con 0x7f1570106720 2024-03-14T09:46:44.947 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.015+0000 7f1567400700 1 -- 172.21.15.78:0/1378580807 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1560003c50 con 0x7f1570106720 2024-03-14T09:46:44.947 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.015+0000 7f1567400700 1 -- 172.21.15.78:0/1378580807 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1560024820 con 0x7f1570106720 2024-03-14T09:46:44.947 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.015+0000 7f156fe00700 1 -- 172.21.15.78:0/1378580807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f157019d310 con 0x7f1570106720 2024-03-14T09:46:44.947 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.016+0000 7f1567400700 1 -- 172.21.15.78:0/1378580807 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 45387+0+0 (secure 0 0 0) 0x7f1560024aa0 con 0x7f1570106720 2024-03-14T09:46:44.947 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.016+0000 7f156fe00700 1 -- 172.21.15.78:0/1378580807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f157010a970 con 0x7f1570106720 2024-03-14T09:46:44.947 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.016+0000 7f1567400700 1 --2- 172.21.15.78:0/1378580807 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1554038520 0x7f155403a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:44.948 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.016+0000 7f1567400700 1 -- 172.21.15.78:0/1378580807 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f156004c670 con 0x7f1570106720 2024-03-14T09:46:44.948 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.017+0000 7f156e000700 1 --2- 172.21.15.78:0/1378580807 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1554038520 0x7f155403a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:44.948 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.017+0000 7f156e000700 1 --2- 172.21.15.78:0/1378580807 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1554038520 0x7f155403a9e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f155c006fd0 tx=0x7f155c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:44.948 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.024+0000 7f1567400700 1 -- 172.21.15.78:0/1378580807 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1560010380 con 0x7f1570106720 2024-03-14T09:46:44.948 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.284+0000 7f156fe00700 1 -- 172.21.15.78:0/1378580807 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1570062200 con 0x7f1554038520 2024-03-14T09:46:44.948 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.515+0000 7f1567400700 1 -- 172.21.15.78:0/1378580807 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f1570062200 con 0x7f1554038520 2024-03-14T09:46:44.948 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.517+0000 7f156fe00700 1 -- 172.21.15.78:0/1378580807 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1554038520 msgr2=0x7f155403a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:44.948 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.517+0000 7f156fe00700 1 --2- 172.21.15.78:0/1378580807 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1554038520 0x7f155403a9e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f155c006fd0 tx=0x7f155c006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:44.948 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.518+0000 7f156fe00700 1 -- 172.21.15.78:0/1378580807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570106720 msgr2=0x7f157019c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:44.948 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.518+0000 7f156fe00700 1 --2- 172.21.15.78:0/1378580807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570106720 0x7f157019c730 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1560011670 tx=0x7f1560011a30 comp rx=0 tx=0).stop 2024-03-14T09:46:44.948 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.518+0000 7f156fe00700 1 -- 172.21.15.78:0/1378580807 shutdown_connections 2024-03-14T09:46:44.948 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.518+0000 7f156fe00700 1 --2- 172.21.15.78:0/1378580807 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1554038520 0x7f155403a9e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:44.948 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.518+0000 7f156fe00700 1 --2- 172.21.15.78:0/1378580807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1570106720 0x7f157019c730 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:44.949 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.518+0000 7f156fe00700 1 -- 172.21.15.78:0/1378580807 >> 172.21.15.78:0/1378580807 conn(0x7f15701018c0 msgr2=0x7f1570102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:44.949 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.518+0000 7f156fe00700 1 -- 172.21.15.78:0/1378580807 shutdown_connections 2024-03-14T09:46:44.949 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:44.518+0000 7f156fe00700 1 -- 172.21.15.78:0/1378580807 wait complete. 2024-03-14T09:46:44.949 INFO:teuthology.orchestra.run.smithi078.stdout:Fetching dashboard port number... 2024-03-14T09:46:45.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:45 smithi078 ceph-mon[38143]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:46:45.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:45 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:46:45.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 8443 2024-03-14T09:46:45.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.315+0000 7ff44f800700 1 Processor -- start 2024-03-14T09:46:45.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.315+0000 7ff44f800700 1 -- start start 2024-03-14T09:46:45.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.315+0000 7ff44f800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff44807a310 0x7ff44807a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:45.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.315+0000 7ff44f800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44807ad00 con 0x7ff44807a310 2024-03-14T09:46:45.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.315+0000 7ff44ce00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff44807a310 0x7ff44807a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:45.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.315+0000 7ff44ce00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff44807a310 0x7ff44807a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40580/0 (socket says 172.21.15.78:40580) 2024-03-14T09:46:45.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.315+0000 7ff44ce00700 1 -- 172.21.15.78:0/3396907377 learned_addr learned my addr 172.21.15.78:0/3396907377 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:45.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.316+0000 7ff44ce00700 1 -- 172.21.15.78:0/3396907377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44807ae40 con 0x7ff44807a310 2024-03-14T09:46:45.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.316+0000 7ff44ce00700 1 --2- 172.21.15.78:0/3396907377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff44807a310 0x7ff44807a730 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0098a0 tx=0x7ff43c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd5bfee6100b15ae server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:45.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.316+0000 7ff447400700 1 -- 172.21.15.78:0/3396907377 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff43c004160 con 0x7ff44807a310 2024-03-14T09:46:45.877 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.316+0000 7ff447400700 1 -- 172.21.15.78:0/3396907377 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff43c0042c0 con 0x7ff44807a310 2024-03-14T09:46:45.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.316+0000 7ff447400700 1 -- 172.21.15.78:0/3396907377 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff43c0119b0 con 0x7ff44807a310 2024-03-14T09:46:45.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.317+0000 7ff44f800700 1 -- 172.21.15.78:0/3396907377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff44807a310 msgr2=0x7ff44807a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:45.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.317+0000 7ff44f800700 1 --2- 172.21.15.78:0/3396907377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff44807a310 0x7ff44807a730 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0098a0 tx=0x7ff43c009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:45.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.317+0000 7ff44f800700 1 -- 172.21.15.78:0/3396907377 shutdown_connections 2024-03-14T09:46:45.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.317+0000 7ff44f800700 1 --2- 172.21.15.78:0/3396907377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff44807a310 0x7ff44807a730 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:45.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.317+0000 7ff44f800700 1 -- 172.21.15.78:0/3396907377 >> 172.21.15.78:0/3396907377 conn(0x7ff448101a70 msgr2=0x7ff448103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:45.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.317+0000 7ff44f800700 1 -- 172.21.15.78:0/3396907377 shutdown_connections 2024-03-14T09:46:45.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.317+0000 7ff44f800700 1 -- 172.21.15.78:0/3396907377 wait complete. 2024-03-14T09:46:45.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.319+0000 7ff44f800700 1 Processor -- start 2024-03-14T09:46:45.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.319+0000 7ff44f800700 1 -- start start 2024-03-14T09:46:45.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.319+0000 7ff44f800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff44807a310 0x7ff44819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:45.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.319+0000 7ff44f800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44819cdd0 con 0x7ff44807a310 2024-03-14T09:46:45.878 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.319+0000 7ff44ce00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff44807a310 0x7ff44819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:45.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.319+0000 7ff44ce00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff44807a310 0x7ff44819c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40590/0 (socket says 172.21.15.78:40590) 2024-03-14T09:46:45.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.319+0000 7ff44ce00700 1 -- 172.21.15.78:0/3716629269 learned_addr learned my addr 172.21.15.78:0/3716629269 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:45.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.320+0000 7ff44ce00700 1 -- 172.21.15.78:0/3716629269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c009550 con 0x7ff44807a310 2024-03-14T09:46:45.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.320+0000 7ff44ce00700 1 --2- 172.21.15.78:0/3716629269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff44807a310 0x7ff44819c890 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff43c003e60 tx=0x7ff43c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:45.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.320+0000 7ff445600700 1 -- 172.21.15.78:0/3716629269 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff43c003a60 con 0x7ff44807a310 2024-03-14T09:46:45.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.320+0000 7ff445600700 1 -- 172.21.15.78:0/3716629269 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff43c003bc0 con 0x7ff44807a310 2024-03-14T09:46:45.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.320+0000 7ff44f800700 1 -- 172.21.15.78:0/3716629269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44819cfd0 con 0x7ff44807a310 2024-03-14T09:46:45.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.320+0000 7ff445600700 1 -- 172.21.15.78:0/3716629269 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff43c0196c0 con 0x7ff44807a310 2024-03-14T09:46:45.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.320+0000 7ff44f800700 1 -- 172.21.15.78:0/3716629269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff44819d470 con 0x7ff44807a310 2024-03-14T09:46:45.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.321+0000 7ff445600700 1 -- 172.21.15.78:0/3716629269 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 45387+0+0 (secure 0 0 0) 0x7ff43c01ab60 con 0x7ff44807a310 2024-03-14T09:46:45.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.321+0000 7ff44f800700 1 -- 172.21.15.78:0/3716629269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff44810ccf0 con 0x7ff44807a310 2024-03-14T09:46:45.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.322+0000 7ff445600700 1 --2- 172.21.15.78:0/3716629269 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff430038550 0x7ff43003aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:45.879 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.322+0000 7ff445600700 1 -- 172.21.15.78:0/3716629269 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff43c04c7f0 con 0x7ff44807a310 2024-03-14T09:46:45.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.322+0000 7ff447e00700 1 --2- 172.21.15.78:0/3716629269 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff430038550 0x7ff43003aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:45.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.322+0000 7ff447e00700 1 --2- 172.21.15.78:0/3716629269 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff430038550 0x7ff43003aa10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff434006fd0 tx=0x7ff434006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:45.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.327+0000 7ff445600700 1 -- 172.21.15.78:0/3716629269 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff43c010790 con 0x7ff44807a310 2024-03-14T09:46:45.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.530+0000 7ff44f800700 1 -- 172.21.15.78:0/3716629269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7ff44819fd00 con 0x7ff44807a310 2024-03-14T09:46:45.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.530+0000 7ff445600700 1 -- 172.21.15.78:0/3716629269 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7ff43c0470b0 con 0x7ff44807a310 2024-03-14T09:46:45.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.532+0000 7ff44f800700 1 -- 172.21.15.78:0/3716629269 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff430038550 msgr2=0x7ff43003aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:45.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.532+0000 7ff44f800700 1 --2- 172.21.15.78:0/3716629269 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff430038550 0x7ff43003aa10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff434006fd0 tx=0x7ff434006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:45.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.532+0000 7ff44f800700 1 -- 172.21.15.78:0/3716629269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff44807a310 msgr2=0x7ff44819c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:45.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.532+0000 7ff44f800700 1 --2- 172.21.15.78:0/3716629269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff44807a310 0x7ff44819c890 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff43c003e60 tx=0x7ff43c011a30 comp rx=0 tx=0).stop 2024-03-14T09:46:45.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.533+0000 7ff44f800700 1 -- 172.21.15.78:0/3716629269 shutdown_connections 2024-03-14T09:46:45.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.533+0000 7ff44f800700 1 --2- 172.21.15.78:0/3716629269 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff430038550 0x7ff43003aa10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:45.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.533+0000 7ff44f800700 1 --2- 172.21.15.78:0/3716629269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff44807a310 0x7ff44819c890 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:45.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.533+0000 7ff44f800700 1 -- 172.21.15.78:0/3716629269 >> 172.21.15.78:0/3716629269 conn(0x7ff448101a70 msgr2=0x7ff448102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:45.880 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.533+0000 7ff44f800700 1 -- 172.21.15.78:0/3716629269 shutdown_connections 2024-03-14T09:46:45.881 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:45.533+0000 7ff44f800700 1 -- 172.21.15.78:0/3716629269 wait complete. 2024-03-14T09:46:45.893 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-14T09:46:45.893 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2024-03-14T09:46:45.908 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-14T09:46:45.908 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2024-03-14T09:46:45.908 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2024-03-14T09:46:45.908 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-14T09:46:45.910 INFO:teuthology.orchestra.run.smithi078.stdout:Ceph Dashboard is now available at: 2024-03-14T09:46:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:46:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: URL: https://smithi078.front.sepia.ceph.com:8443/ 2024-03-14T09:46:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: User: admin 2024-03-14T09:46:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: Password: dh5m6a554s 2024-03-14T09:46:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:46:45.911 INFO:teuthology.orchestra.run.smithi078.stdout:Saving cluster configuration to /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config directory 2024-03-14T09:46:45.912 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling autotune for osd_memory_target 2024-03-14T09:46:46.766 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:46 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3716629269' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-14T09:46:46.904 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.327+0000 7f76b1800700 1 Processor -- start 2024-03-14T09:46:46.904 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.327+0000 7f76b1800700 1 -- start start 2024-03-14T09:46:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.328+0000 7f76b1800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76ac07a2f0 0x7f76ac07a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.328+0000 7f76b1800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac07ace0 con 0x7f76ac07a2f0 2024-03-14T09:46:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.328+0000 7f76aaa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76ac07a2f0 0x7f76ac07a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.328+0000 7f76aaa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76ac07a2f0 0x7f76ac07a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40598/0 (socket says 172.21.15.78:40598) 2024-03-14T09:46:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.328+0000 7f76aaa00700 1 -- 172.21.15.78:0/2494297311 learned_addr learned my addr 172.21.15.78:0/2494297311 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.329+0000 7f76aaa00700 1 -- 172.21.15.78:0/2494297311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac07ae20 con 0x7f76ac07a2f0 2024-03-14T09:46:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.329+0000 7f76aaa00700 1 --2- 172.21.15.78:0/2494297311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76ac07a2f0 0x7f76ac07a710 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f769c0098a0 tx=0x7f769c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f02a377d085bc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.329+0000 7f76aa000700 1 -- 172.21.15.78:0/2494297311 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f769c004160 con 0x7f76ac07a2f0 2024-03-14T09:46:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.329+0000 7f76aa000700 1 -- 172.21.15.78:0/2494297311 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f769c0042c0 con 0x7f76ac07a2f0 2024-03-14T09:46:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.329+0000 7f76aa000700 1 -- 172.21.15.78:0/2494297311 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f769c0119b0 con 0x7f76ac07a2f0 2024-03-14T09:46:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.330+0000 7f76b1800700 1 -- 172.21.15.78:0/2494297311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76ac07a2f0 msgr2=0x7f76ac07a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.330+0000 7f76b1800700 1 --2- 172.21.15.78:0/2494297311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76ac07a2f0 0x7f76ac07a710 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f769c0098a0 tx=0x7f769c009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.330+0000 7f76b1800700 1 -- 172.21.15.78:0/2494297311 shutdown_connections 2024-03-14T09:46:46.905 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.330+0000 7f76b1800700 1 --2- 172.21.15.78:0/2494297311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76ac07a2f0 0x7f76ac07a710 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:46.906 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.330+0000 7f76b1800700 1 -- 172.21.15.78:0/2494297311 >> 172.21.15.78:0/2494297311 conn(0x7f76ac101a70 msgr2=0x7f76ac103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:46.906 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.330+0000 7f76b1800700 1 -- 172.21.15.78:0/2494297311 shutdown_connections 2024-03-14T09:46:46.906 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.330+0000 7f76b1800700 1 -- 172.21.15.78:0/2494297311 wait complete. 2024-03-14T09:46:46.906 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.331+0000 7f76b1800700 1 Processor -- start 2024-03-14T09:46:46.906 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.331+0000 7f76b1800700 1 -- start start 2024-03-14T09:46:46.906 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.331+0000 7f76b1800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76ac07a2f0 0x7f76ac19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:46.906 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.331+0000 7f76b1800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac19cd80 con 0x7f76ac07a2f0 2024-03-14T09:46:46.906 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.331+0000 7f76aaa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76ac07a2f0 0x7f76ac19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:46.906 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.331+0000 7f76aaa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76ac07a2f0 0x7f76ac19c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40604/0 (socket says 172.21.15.78:40604) 2024-03-14T09:46:46.906 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.331+0000 7f76aaa00700 1 -- 172.21.15.78:0/84015448 learned_addr learned my addr 172.21.15.78:0/84015448 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:46.906 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.332+0000 7f76aaa00700 1 -- 172.21.15.78:0/84015448 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769c009550 con 0x7f76ac07a2f0 2024-03-14T09:46:46.906 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.332+0000 7f76aaa00700 1 --2- 172.21.15.78:0/84015448 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76ac07a2f0 0x7f76ac19c840 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f769c011670 tx=0x7f769c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:46.906 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.332+0000 7f76a3400700 1 -- 172.21.15.78:0/84015448 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f769c003b20 con 0x7f76ac07a2f0 2024-03-14T09:46:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.332+0000 7f76a3400700 1 -- 172.21.15.78:0/84015448 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f769c003c80 con 0x7f76ac07a2f0 2024-03-14T09:46:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.332+0000 7f76a3400700 1 -- 172.21.15.78:0/84015448 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f769c019770 con 0x7f76ac07a2f0 2024-03-14T09:46:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.332+0000 7f76b1800700 1 -- 172.21.15.78:0/84015448 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ac19cf80 con 0x7f76ac07a2f0 2024-03-14T09:46:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.332+0000 7f76b1800700 1 -- 172.21.15.78:0/84015448 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ac19d420 con 0x7f76ac07a2f0 2024-03-14T09:46:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.333+0000 7f76a3400700 1 -- 172.21.15.78:0/84015448 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 45387+0+0 (secure 0 0 0) 0x7f769c01a430 con 0x7f76ac07a2f0 2024-03-14T09:46:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.333+0000 7f76b1800700 1 -- 172.21.15.78:0/84015448 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76ac10cd10 con 0x7f76ac07a2f0 2024-03-14T09:46:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.334+0000 7f76a3400700 1 --2- 172.21.15.78:0/84015448 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7690038520 0x7f769003a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.334+0000 7f76a3400700 1 -- 172.21.15.78:0/84015448 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f769c04c890 con 0x7f76ac07a2f0 2024-03-14T09:46:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.334+0000 7f76a3e00700 1 --2- 172.21.15.78:0/84015448 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7690038520 0x7f769003a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.334+0000 7f76a3e00700 1 --2- 172.21.15.78:0/84015448 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7690038520 0x7f769003a9e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7694006fd0 tx=0x7f7694006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.338+0000 7f76a3400700 1 -- 172.21.15.78:0/84015448 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f769c02d9a0 con 0x7f76ac07a2f0 2024-03-14T09:46:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.541+0000 7f76b1800700 1 -- 172.21.15.78:0/84015448 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f76ac10cf50 con 0x7f76ac07a2f0 2024-03-14T09:46:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.542+0000 7f76a3400700 1 -- 172.21.15.78:0/84015448 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f769c02d350 con 0x7f76ac07a2f0 2024-03-14T09:46:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.544+0000 7f76b1800700 1 -- 172.21.15.78:0/84015448 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7690038520 msgr2=0x7f769003a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:46.908 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.544+0000 7f76b1800700 1 --2- 172.21.15.78:0/84015448 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7690038520 0x7f769003a9e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7694006fd0 tx=0x7f7694006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:46.908 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.544+0000 7f76b1800700 1 -- 172.21.15.78:0/84015448 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76ac07a2f0 msgr2=0x7f76ac19c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:46.908 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.544+0000 7f76b1800700 1 --2- 172.21.15.78:0/84015448 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76ac07a2f0 0x7f76ac19c840 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f769c011670 tx=0x7f769c011a30 comp rx=0 tx=0).stop 2024-03-14T09:46:46.908 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.544+0000 7f76b1800700 1 -- 172.21.15.78:0/84015448 shutdown_connections 2024-03-14T09:46:46.908 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.544+0000 7f76b1800700 1 --2- 172.21.15.78:0/84015448 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7690038520 0x7f769003a9e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:46.908 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.544+0000 7f76b1800700 1 --2- 172.21.15.78:0/84015448 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f76ac07a2f0 0x7f76ac19c840 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:46.908 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.544+0000 7f76b1800700 1 -- 172.21.15.78:0/84015448 >> 172.21.15.78:0/84015448 conn(0x7f76ac101a70 msgr2=0x7f76ac1025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:46.908 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.544+0000 7f76b1800700 1 -- 172.21.15.78:0/84015448 shutdown_connections 2024-03-14T09:46:46.908 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:46.544+0000 7f76b1800700 1 -- 172.21.15.78:0/84015448 wait complete. 2024-03-14T09:46:47.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.287+0000 7f2789800700 1 Processor -- start 2024-03-14T09:46:47.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.287+0000 7f2789800700 1 -- start start 2024-03-14T09:46:47.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.287+0000 7f2789800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f278407a330 0x7f278407a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:47.972 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.287+0000 7f2789800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278407ad20 con 0x7f278407a330 2024-03-14T09:46:47.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.288+0000 7f2782a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f278407a330 0x7f278407a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:47.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.288+0000 7f2782a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f278407a330 0x7f278407a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40608/0 (socket says 172.21.15.78:40608) 2024-03-14T09:46:47.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.288+0000 7f2782a00700 1 -- 172.21.15.78:0/3161234612 learned_addr learned my addr 172.21.15.78:0/3161234612 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:47.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.288+0000 7f2782a00700 1 -- 172.21.15.78:0/3161234612 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278407ae60 con 0x7f278407a330 2024-03-14T09:46:47.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.288+0000 7f2782a00700 1 --2- 172.21.15.78:0/3161234612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f278407a330 0x7f278407a750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f27740098a0 tx=0x7f2774009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e128125a970277c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:47.973 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.288+0000 7f2781600700 1 -- 172.21.15.78:0/3161234612 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2774004160 con 0x7f278407a330 2024-03-14T09:46:47.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.288+0000 7f2781600700 1 -- 172.21.15.78:0/3161234612 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27740042c0 con 0x7f278407a330 2024-03-14T09:46:47.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.288+0000 7f2781600700 1 -- 172.21.15.78:0/3161234612 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27740119b0 con 0x7f278407a330 2024-03-14T09:46:47.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.289+0000 7f2789800700 1 -- 172.21.15.78:0/3161234612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f278407a330 msgr2=0x7f278407a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:47.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.289+0000 7f2789800700 1 --2- 172.21.15.78:0/3161234612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f278407a330 0x7f278407a750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f27740098a0 tx=0x7f2774009bb0 comp rx=0 tx=0).stop 2024-03-14T09:46:47.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.289+0000 7f2789800700 1 -- 172.21.15.78:0/3161234612 shutdown_connections 2024-03-14T09:46:47.974 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.289+0000 7f2789800700 1 --2- 172.21.15.78:0/3161234612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f278407a330 0x7f278407a750 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:47.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.289+0000 7f2789800700 1 -- 172.21.15.78:0/3161234612 >> 172.21.15.78:0/3161234612 conn(0x7f2784101a70 msgr2=0x7f2784103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:47.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.289+0000 7f2789800700 1 -- 172.21.15.78:0/3161234612 shutdown_connections 2024-03-14T09:46:47.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.289+0000 7f2789800700 1 -- 172.21.15.78:0/3161234612 wait complete. 2024-03-14T09:46:47.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.290+0000 7f2789800700 1 Processor -- start 2024-03-14T09:46:47.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.290+0000 7f2789800700 1 -- start start 2024-03-14T09:46:47.975 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.291+0000 7f2789800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f278407a330 0x7f278419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:47.976 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.291+0000 7f2789800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278419cd80 con 0x7f278407a330 2024-03-14T09:46:47.976 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.291+0000 7f2782a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f278407a330 0x7f278419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:47.976 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.291+0000 7f2782a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f278407a330 0x7f278419c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40612/0 (socket says 172.21.15.78:40612) 2024-03-14T09:46:47.976 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.291+0000 7f2782a00700 1 -- 172.21.15.78:0/1099360375 learned_addr learned my addr 172.21.15.78:0/1099360375 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:47.976 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.291+0000 7f2782a00700 1 -- 172.21.15.78:0/1099360375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2774009550 con 0x7f278407a330 2024-03-14T09:46:47.976 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.291+0000 7f2782a00700 1 --2- 172.21.15.78:0/1099360375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f278407a330 0x7f278419c840 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2774011670 tx=0x7f2774011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:47.976 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.292+0000 7f277b400700 1 -- 172.21.15.78:0/1099360375 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2774003b20 con 0x7f278407a330 2024-03-14T09:46:47.977 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.292+0000 7f277b400700 1 -- 172.21.15.78:0/1099360375 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2774003c80 con 0x7f278407a330 2024-03-14T09:46:47.977 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.292+0000 7f2789800700 1 -- 172.21.15.78:0/1099360375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278419cf80 con 0x7f278407a330 2024-03-14T09:46:47.977 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.292+0000 7f277b400700 1 -- 172.21.15.78:0/1099360375 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2774019770 con 0x7f278407a330 2024-03-14T09:46:47.977 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.292+0000 7f2789800700 1 -- 172.21.15.78:0/1099360375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278419d420 con 0x7f278407a330 2024-03-14T09:46:47.977 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.293+0000 7f277b400700 1 -- 172.21.15.78:0/1099360375 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 45387+0+0 (secure 0 0 0) 0x7f277401a430 con 0x7f278407a330 2024-03-14T09:46:47.977 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.293+0000 7f2789800700 1 -- 172.21.15.78:0/1099360375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278410ccd0 con 0x7f278407a330 2024-03-14T09:46:47.978 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.293+0000 7f277b400700 1 --2- 172.21.15.78:0/1099360375 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2768038520 0x7f276803a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:47.978 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.293+0000 7f277b400700 1 -- 172.21.15.78:0/1099360375 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f277404c840 con 0x7f278407a330 2024-03-14T09:46:47.978 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.293+0000 7f2782000700 1 --2- 172.21.15.78:0/1099360375 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2768038520 0x7f276803a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:47.978 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.294+0000 7f2782000700 1 --2- 172.21.15.78:0/1099360375 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2768038520 0x7f276803a9e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2770006fd0 tx=0x7f2770006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:47.978 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.298+0000 7f277b400700 1 -- 172.21.15.78:0/1099360375 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f277401ab60 con 0x7f278407a330 2024-03-14T09:46:47.978 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.518+0000 7f277b400700 1 -- 172.21.15.78:0/1099360375 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 13) v1 ==== 45430+0+0 (secure 0 0 0) 0x7f2774019a90 con 0x7f278407a330 2024-03-14T09:46:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.607+0000 7f2789800700 1 -- 172.21.15.78:0/1099360375 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f278410cee0 con 0x7f278407a330 2024-03-14T09:46:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.609+0000 7f277b400700 1 -- 172.21.15.78:0/1099360375 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f277402db80 con 0x7f278407a330 2024-03-14T09:46:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.611+0000 7f2789800700 1 -- 172.21.15.78:0/1099360375 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2768038520 msgr2=0x7f276803a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.611+0000 7f2789800700 1 --2- 172.21.15.78:0/1099360375 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2768038520 0x7f276803a9e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2770006fd0 tx=0x7f2770006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.611+0000 7f2789800700 1 -- 172.21.15.78:0/1099360375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f278407a330 msgr2=0x7f278419c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:47.979 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.611+0000 7f2789800700 1 --2- 172.21.15.78:0/1099360375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f278407a330 0x7f278419c840 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2774011670 tx=0x7f2774011a30 comp rx=0 tx=0).stop 2024-03-14T09:46:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.611+0000 7f2789800700 1 -- 172.21.15.78:0/1099360375 shutdown_connections 2024-03-14T09:46:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.611+0000 7f2789800700 1 --2- 172.21.15.78:0/1099360375 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2768038520 0x7f276803a9e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.611+0000 7f2789800700 1 --2- 172.21.15.78:0/1099360375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f278407a330 0x7f278419c840 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.611+0000 7f2789800700 1 -- 172.21.15.78:0/1099360375 >> 172.21.15.78:0/1099360375 conn(0x7f2784101a70 msgr2=0x7f2784102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.611+0000 7f2789800700 1 -- 172.21.15.78:0/1099360375 shutdown_connections 2024-03-14T09:46:47.980 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2024-03-14T09:46:47.611+0000 7f2789800700 1 -- 172.21.15.78:0/1099360375 wait complete. 2024-03-14T09:46:47.981 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-14T09:46:47.981 INFO:teuthology.orchestra.run.smithi078.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-14T09:46:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:46:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-14T09:46:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:46:47.981 INFO:teuthology.orchestra.run.smithi078.stdout:Or, if you are only running a single cluster on this host: 2024-03-14T09:46:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:46:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-14T09:46:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:46:47.982 INFO:teuthology.orchestra.run.smithi078.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-14T09:46:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:46:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: ceph telemetry on 2024-03-14T09:46:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:46:47.983 INFO:teuthology.orchestra.run.smithi078.stdout:For more information see: 2024-03-14T09:46:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:46:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-14T09:46:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:46:47.983 INFO:teuthology.orchestra.run.smithi078.stdout:Bootstrap complete. 2024-03-14T09:46:48.024 INFO:tasks.cephadm:Fetching config... 2024-03-14T09:46:48.024 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:46:48.024 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-14T09:46:48.067 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-14T09:46:48.067 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:46:48.068 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-14T09:46:48.150 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-14T09:46:48.150 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:46:48.150 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-14T09:46:48.259 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-14T09:46:48.259 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:46:48.259 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-14T09:46:48.342 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-14T09:46:48.343 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCRdPpCviAowYx6L50GvU8eS1Qhy1Uel/UV4vugBaXb6iVDW/dkIu4epdMWTKbcaqvErLlV/nYeSGSQj2KGaX+vvakqz3fztCSszZt9KipIJ5WPqCIpubA7RoaSSepMc0DqaJn5x579+po+H1SZsCdpyNMRNgKpWNHnEeWAdWSuwk70q5tFZPZVMLVq1w0pJBOCCUQLkJoSWCw0Xe4zWQMfTOtqIHachyuZQkdxI9trp8KtHVEnOyrlmTjqrNO+9WIR92qtBVdlKOZllkGDcqtyrI/a1PweLvdt9wRQ0Qne9simBTOO3uQva3nPFWhynsR5eHiWTRHanPtpeepk+2CXMgQpjtJQ+OTL30QZLl8FixoiPLkj0IX9IE2UbtxEQUEPELSsGIAmrQe8B91OxvCYR1D570EcwyNGWfefI0oP8Mfme+gBCIFpaes7uNo8JXmZmhNMKGPrYGcragT2sQSDLD0o58tP+7Ox1k9ka9i+fRbDJlSJ9xFRbpS/0/SQJfM= ceph-83662c4e-e1e7-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-14T09:46:48.474 INFO:teuthology.orchestra.run.smithi078.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCRdPpCviAowYx6L50GvU8eS1Qhy1Uel/UV4vugBaXb6iVDW/dkIu4epdMWTKbcaqvErLlV/nYeSGSQj2KGaX+vvakqz3fztCSszZt9KipIJ5WPqCIpubA7RoaSSepMc0DqaJn5x579+po+H1SZsCdpyNMRNgKpWNHnEeWAdWSuwk70q5tFZPZVMLVq1w0pJBOCCUQLkJoSWCw0Xe4zWQMfTOtqIHachyuZQkdxI9trp8KtHVEnOyrlmTjqrNO+9WIR92qtBVdlKOZllkGDcqtyrI/a1PweLvdt9wRQ0Qne9simBTOO3uQva3nPFWhynsR5eHiWTRHanPtpeepk+2CXMgQpjtJQ+OTL30QZLl8FixoiPLkj0IX9IE2UbtxEQUEPELSsGIAmrQe8B91OxvCYR1D570EcwyNGWfefI0oP8Mfme+gBCIFpaes7uNo8JXmZmhNMKGPrYGcragT2sQSDLD0o58tP+7Ox1k9ka9i+fRbDJlSJ9xFRbpS/0/SQJfM= ceph-83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:46:48.502 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCRdPpCviAowYx6L50GvU8eS1Qhy1Uel/UV4vugBaXb6iVDW/dkIu4epdMWTKbcaqvErLlV/nYeSGSQj2KGaX+vvakqz3fztCSszZt9KipIJ5WPqCIpubA7RoaSSepMc0DqaJn5x579+po+H1SZsCdpyNMRNgKpWNHnEeWAdWSuwk70q5tFZPZVMLVq1w0pJBOCCUQLkJoSWCw0Xe4zWQMfTOtqIHachyuZQkdxI9trp8KtHVEnOyrlmTjqrNO+9WIR92qtBVdlKOZllkGDcqtyrI/a1PweLvdt9wRQ0Qne9simBTOO3uQva3nPFWhynsR5eHiWTRHanPtpeepk+2CXMgQpjtJQ+OTL30QZLl8FixoiPLkj0IX9IE2UbtxEQUEPELSsGIAmrQe8B91OxvCYR1D570EcwyNGWfefI0oP8Mfme+gBCIFpaes7uNo8JXmZmhNMKGPrYGcragT2sQSDLD0o58tP+7Ox1k9ka9i+fRbDJlSJ9xFRbpS/0/SQJfM= ceph-83662c4e-e1e7-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-14T09:46:48.547 INFO:teuthology.orchestra.run.smithi132.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCRdPpCviAowYx6L50GvU8eS1Qhy1Uel/UV4vugBaXb6iVDW/dkIu4epdMWTKbcaqvErLlV/nYeSGSQj2KGaX+vvakqz3fztCSszZt9KipIJ5WPqCIpubA7RoaSSepMc0DqaJn5x579+po+H1SZsCdpyNMRNgKpWNHnEeWAdWSuwk70q5tFZPZVMLVq1w0pJBOCCUQLkJoSWCw0Xe4zWQMfTOtqIHachyuZQkdxI9trp8KtHVEnOyrlmTjqrNO+9WIR92qtBVdlKOZllkGDcqtyrI/a1PweLvdt9wRQ0Qne9simBTOO3uQva3nPFWhynsR5eHiWTRHanPtpeepk+2CXMgQpjtJQ+OTL30QZLl8FixoiPLkj0IX9IE2UbtxEQUEPELSsGIAmrQe8B91OxvCYR1D570EcwyNGWfefI0oP8Mfme+gBCIFpaes7uNo8JXmZmhNMKGPrYGcragT2sQSDLD0o58tP+7Ox1k9ka9i+fRbDJlSJ9xFRbpS/0/SQJfM= ceph-83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:46:48.560 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCRdPpCviAowYx6L50GvU8eS1Qhy1Uel/UV4vugBaXb6iVDW/dkIu4epdMWTKbcaqvErLlV/nYeSGSQj2KGaX+vvakqz3fztCSszZt9KipIJ5WPqCIpubA7RoaSSepMc0DqaJn5x579+po+H1SZsCdpyNMRNgKpWNHnEeWAdWSuwk70q5tFZPZVMLVq1w0pJBOCCUQLkJoSWCw0Xe4zWQMfTOtqIHachyuZQkdxI9trp8KtHVEnOyrlmTjqrNO+9WIR92qtBVdlKOZllkGDcqtyrI/a1PweLvdt9wRQ0Qne9simBTOO3uQva3nPFWhynsR5eHiWTRHanPtpeepk+2CXMgQpjtJQ+OTL30QZLl8FixoiPLkj0IX9IE2UbtxEQUEPELSsGIAmrQe8B91OxvCYR1D570EcwyNGWfefI0oP8Mfme+gBCIFpaes7uNo8JXmZmhNMKGPrYGcragT2sQSDLD0o58tP+7Ox1k9ka9i+fRbDJlSJ9xFRbpS/0/SQJfM= ceph-83662c4e-e1e7-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-14T09:46:48.609 INFO:teuthology.orchestra.run.smithi152.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCRdPpCviAowYx6L50GvU8eS1Qhy1Uel/UV4vugBaXb6iVDW/dkIu4epdMWTKbcaqvErLlV/nYeSGSQj2KGaX+vvakqz3fztCSszZt9KipIJ5WPqCIpubA7RoaSSepMc0DqaJn5x579+po+H1SZsCdpyNMRNgKpWNHnEeWAdWSuwk70q5tFZPZVMLVq1w0pJBOCCUQLkJoSWCw0Xe4zWQMfTOtqIHachyuZQkdxI9trp8KtHVEnOyrlmTjqrNO+9WIR92qtBVdlKOZllkGDcqtyrI/a1PweLvdt9wRQ0Qne9simBTOO3uQva3nPFWhynsR5eHiWTRHanPtpeepk+2CXMgQpjtJQ+OTL30QZLl8FixoiPLkj0IX9IE2UbtxEQUEPELSsGIAmrQe8B91OxvCYR1D570EcwyNGWfefI0oP8Mfme+gBCIFpaes7uNo8JXmZmhNMKGPrYGcragT2sQSDLD0o58tP+7Ox1k9ka9i+fRbDJlSJ9xFRbpS/0/SQJfM= ceph-83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:46:48.622 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-14T09:46:48.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:48 smithi078 ceph-mon[38143]: mgrmap e13: x(active, since 6s) 2024-03-14T09:46:48.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:48 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1099360375' entity='client.admin' 2024-03-14T09:46:48.985 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:46:50.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.245+0000 7f119ce00700 1 -- 172.21.15.78:0/573443584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1198074d80 msgr2=0x7f11980731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:50.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.245+0000 7f119ce00700 1 --2- 172.21.15.78:0/573443584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1198074d80 0x7f11980731e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1188009910 tx=0x7f1188009c20 comp rx=0 tx=0).stop 2024-03-14T09:46:50.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.245+0000 7f119ce00700 1 -- 172.21.15.78:0/573443584 shutdown_connections 2024-03-14T09:46:50.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.245+0000 7f119ce00700 1 --2- 172.21.15.78:0/573443584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1198074d80 0x7f11980731e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:50.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.245+0000 7f119ce00700 1 -- 172.21.15.78:0/573443584 >> 172.21.15.78:0/573443584 conn(0x7f11980fbcb0 msgr2=0x7f11980fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:50.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.245+0000 7f119ce00700 1 -- 172.21.15.78:0/573443584 shutdown_connections 2024-03-14T09:46:50.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.246+0000 7f119ce00700 1 -- 172.21.15.78:0/573443584 wait complete. 2024-03-14T09:46:50.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.247+0000 7f119ce00700 1 Processor -- start 2024-03-14T09:46:50.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.247+0000 7f119ce00700 1 -- start start 2024-03-14T09:46:50.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.247+0000 7f119ce00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1198074d80 0x7f1198198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:50.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.247+0000 7f119ce00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11981986b0 con 0x7f1198074d80 2024-03-14T09:46:50.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.247+0000 7f1195e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1198074d80 0x7f1198198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:50.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.248+0000 7f1195e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1198074d80 0x7f1198198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40624/0 (socket says 172.21.15.78:40624) 2024-03-14T09:46:50.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.248+0000 7f1195e00700 1 -- 172.21.15.78:0/2729382977 learned_addr learned my addr 172.21.15.78:0/2729382977 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:50.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.248+0000 7f1195e00700 1 -- 172.21.15.78:0/2729382977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11880095f0 con 0x7f1198074d80 2024-03-14T09:46:50.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.248+0000 7f1195e00700 1 --2- 172.21.15.78:0/2729382977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1198074d80 0x7f1198198170 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1188005950 tx=0x7f11880057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:50.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.248+0000 7f118ea00700 1 -- 172.21.15.78:0/2729382977 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f118801d070 con 0x7f1198074d80 2024-03-14T09:46:50.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.248+0000 7f118ea00700 1 -- 172.21.15.78:0/2729382977 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f118800bc10 con 0x7f1198074d80 2024-03-14T09:46:50.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.248+0000 7f118ea00700 1 -- 172.21.15.78:0/2729382977 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f11880039d0 con 0x7f1198074d80 2024-03-14T09:46:50.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.248+0000 7f119ce00700 1 -- 172.21.15.78:0/2729382977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11981988b0 con 0x7f1198074d80 2024-03-14T09:46:50.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.248+0000 7f119ce00700 1 -- 172.21.15.78:0/2729382977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1198198d50 con 0x7f1198074d80 2024-03-14T09:46:50.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.249+0000 7f118ea00700 1 -- 172.21.15.78:0/2729382977 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45430+0+0 (secure 0 0 0) 0x7f1188004390 con 0x7f1198074d80 2024-03-14T09:46:50.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.250+0000 7f119ce00700 1 -- 172.21.15.78:0/2729382977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1198191a50 con 0x7f1198074d80 2024-03-14T09:46:50.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.250+0000 7f118ea00700 1 --2- 172.21.15.78:0/2729382977 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f117c0385c0 0x7f117c03aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:50.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.250+0000 7f118ea00700 1 -- 172.21.15.78:0/2729382977 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f118804c280 con 0x7f1198074d80 2024-03-14T09:46:50.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.250+0000 7f1195400700 1 --2- 172.21.15.78:0/2729382977 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f117c0385c0 0x7f117c03aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:50.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.251+0000 7f1195400700 1 --2- 172.21.15.78:0/2729382977 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f117c0385c0 0x7f117c03aa80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1184006fd0 tx=0x7f1184006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:50.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.255+0000 7f118ea00700 1 -- 172.21.15.78:0/2729382977 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1188029360 con 0x7f1198074d80 2024-03-14T09:46:50.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.473+0000 7f119ce00700 1 -- 172.21.15.78:0/2729382977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f11980623e0 con 0x7f1198074d80 2024-03-14T09:46:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.477+0000 7f118ea00700 1 -- 172.21.15.78:0/2729382977 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f11980623e0 con 0x7f1198074d80 2024-03-14T09:46:50.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.479+0000 7f119ce00700 1 -- 172.21.15.78:0/2729382977 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f117c0385c0 msgr2=0x7f117c03aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:50.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.479+0000 7f119ce00700 1 --2- 172.21.15.78:0/2729382977 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f117c0385c0 0x7f117c03aa80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1184006fd0 tx=0x7f1184006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:50.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.479+0000 7f119ce00700 1 -- 172.21.15.78:0/2729382977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1198074d80 msgr2=0x7f1198198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:50.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.479+0000 7f119ce00700 1 --2- 172.21.15.78:0/2729382977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1198074d80 0x7f1198198170 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1188005950 tx=0x7f11880057c0 comp rx=0 tx=0).stop 2024-03-14T09:46:50.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.479+0000 7f119ce00700 1 -- 172.21.15.78:0/2729382977 shutdown_connections 2024-03-14T09:46:50.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.479+0000 7f119ce00700 1 --2- 172.21.15.78:0/2729382977 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f117c0385c0 0x7f117c03aa80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:50.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.479+0000 7f119ce00700 1 --2- 172.21.15.78:0/2729382977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1198074d80 0x7f1198198170 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:50.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.479+0000 7f119ce00700 1 -- 172.21.15.78:0/2729382977 >> 172.21.15.78:0/2729382977 conn(0x7f11980fbcb0 msgr2=0x7f11980fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:50.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.480+0000 7f119ce00700 1 -- 172.21.15.78:0/2729382977 shutdown_connections 2024-03-14T09:46:50.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:50.480+0000 7f119ce00700 1 -- 172.21.15.78:0/2729382977 wait complete. 2024-03-14T09:46:50.896 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-14T09:46:50.896 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-14T09:46:51.258 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:46:51.752 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:51 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2729382977' entity='client.admin' 2024-03-14T09:46:52.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.699+0000 7fcabd600700 1 -- 172.21.15.78:0/2703747554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcab8074d60 msgr2=0x7fcab80731c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:52.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.699+0000 7fcabd600700 1 --2- 172.21.15.78:0/2703747554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcab8074d60 0x7fcab80731c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009910 tx=0x7fcaa8009c20 comp rx=0 tx=0).stop 2024-03-14T09:46:52.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.699+0000 7fcabd600700 1 -- 172.21.15.78:0/2703747554 shutdown_connections 2024-03-14T09:46:52.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.699+0000 7fcabd600700 1 --2- 172.21.15.78:0/2703747554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcab8074d60 0x7fcab80731c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:52.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.699+0000 7fcabd600700 1 -- 172.21.15.78:0/2703747554 >> 172.21.15.78:0/2703747554 conn(0x7fcab80fbcb0 msgr2=0x7fcab80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:52.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.699+0000 7fcabd600700 1 -- 172.21.15.78:0/2703747554 shutdown_connections 2024-03-14T09:46:52.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.699+0000 7fcabd600700 1 -- 172.21.15.78:0/2703747554 wait complete. 2024-03-14T09:46:52.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.700+0000 7fcabd600700 1 Processor -- start 2024-03-14T09:46:52.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.701+0000 7fcabd600700 1 -- start start 2024-03-14T09:46:52.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.701+0000 7fcabd600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcab8074d60 0x7fcab8198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:52.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.701+0000 7fcabd600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab81986b0 con 0x7fcab8074d60 2024-03-14T09:46:52.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.701+0000 7fcab6a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcab8074d60 0x7fcab8198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:52.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.701+0000 7fcab6a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcab8074d60 0x7fcab8198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40648/0 (socket says 172.21.15.78:40648) 2024-03-14T09:46:52.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.701+0000 7fcab6a00700 1 -- 172.21.15.78:0/590813904 learned_addr learned my addr 172.21.15.78:0/590813904 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:52.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.702+0000 7fcab6a00700 1 -- 172.21.15.78:0/590813904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa80095f0 con 0x7fcab8074d60 2024-03-14T09:46:52.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.702+0000 7fcab6a00700 1 --2- 172.21.15.78:0/590813904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcab8074d60 0x7fcab8198170 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8005950 tx=0x7fcaa80057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:52.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.702+0000 7fcaaf400700 1 -- 172.21.15.78:0/590813904 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcaa801d070 con 0x7fcab8074d60 2024-03-14T09:46:52.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.702+0000 7fcaaf400700 1 -- 172.21.15.78:0/590813904 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcaa800bc10 con 0x7fcab8074d60 2024-03-14T09:46:52.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.702+0000 7fcabd600700 1 -- 172.21.15.78:0/590813904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab81988b0 con 0x7fcab8074d60 2024-03-14T09:46:52.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.702+0000 7fcaaf400700 1 -- 172.21.15.78:0/590813904 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcaa80039d0 con 0x7fcab8074d60 2024-03-14T09:46:52.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.702+0000 7fcabd600700 1 -- 172.21.15.78:0/590813904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab8198d50 con 0x7fcab8074d60 2024-03-14T09:46:52.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.703+0000 7fcaaf400700 1 -- 172.21.15.78:0/590813904 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45430+0+0 (secure 0 0 0) 0x7fcaa8004390 con 0x7fcab8074d60 2024-03-14T09:46:52.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.703+0000 7fcabd600700 1 -- 172.21.15.78:0/590813904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab8191a50 con 0x7fcab8074d60 2024-03-14T09:46:52.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.704+0000 7fcaaf400700 1 --2- 172.21.15.78:0/590813904 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fca9c038570 0x7fca9c03aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:52.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.704+0000 7fcaaf400700 1 -- 172.21.15.78:0/590813904 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcaa804c1f0 con 0x7fcab8074d60 2024-03-14T09:46:52.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.704+0000 7fcaafe00700 1 --2- 172.21.15.78:0/590813904 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fca9c038570 0x7fca9c03aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:52.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.705+0000 7fcaafe00700 1 --2- 172.21.15.78:0/590813904 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fca9c038570 0x7fca9c03aa30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0006fd0 tx=0x7fcaa0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:52.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.709+0000 7fcaaf400700 1 -- 172.21.15.78:0/590813904 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcaa8029360 con 0x7fcab8074d60 2024-03-14T09:46:52.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.917+0000 7fcabd600700 1 -- 172.21.15.78:0/590813904 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fcab80611f0 con 0x7fca9c038570 2024-03-14T09:46:52.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.921+0000 7fcaaf400700 1 -- 172.21.15.78:0/590813904 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcab80611f0 con 0x7fca9c038570 2024-03-14T09:46:52.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.923+0000 7fcabd600700 1 -- 172.21.15.78:0/590813904 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fca9c038570 msgr2=0x7fca9c03aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:52.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.923+0000 7fcabd600700 1 --2- 172.21.15.78:0/590813904 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fca9c038570 0x7fca9c03aa30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0006fd0 tx=0x7fcaa0006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:52.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.923+0000 7fcabd600700 1 -- 172.21.15.78:0/590813904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcab8074d60 msgr2=0x7fcab8198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:52.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.924+0000 7fcabd600700 1 --2- 172.21.15.78:0/590813904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcab8074d60 0x7fcab8198170 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8005950 tx=0x7fcaa80057c0 comp rx=0 tx=0).stop 2024-03-14T09:46:52.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.924+0000 7fcabd600700 1 -- 172.21.15.78:0/590813904 shutdown_connections 2024-03-14T09:46:52.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.924+0000 7fcabd600700 1 --2- 172.21.15.78:0/590813904 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fca9c038570 0x7fca9c03aa30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:52.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.924+0000 7fcabd600700 1 --2- 172.21.15.78:0/590813904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcab8074d60 0x7fcab8198170 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:52.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.924+0000 7fcabd600700 1 -- 172.21.15.78:0/590813904 >> 172.21.15.78:0/590813904 conn(0x7fcab80fbcb0 msgr2=0x7fcab80fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:52.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.924+0000 7fcabd600700 1 -- 172.21.15.78:0/590813904 shutdown_connections 2024-03-14T09:46:52.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:52.924+0000 7fcabd600700 1 -- 172.21.15.78:0/590813904 wait complete. 2024-03-14T09:46:53.339 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi132 2024-03-14T09:46:53.340 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-14T09:46:53.340 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/etc/ceph/ceph.conf 2024-03-14T09:46:53.360 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-14T09:46:53.361 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:46:53.420 INFO:tasks.cephadm:Adding host smithi132 to orchestrator... 2024-03-14T09:46:53.420 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch host add smithi132 2024-03-14T09:46:53.791 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:46:53.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:46:53.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:46:53.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:46:53.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:46:53.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:46:53.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:46:53.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:46:53.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:46:53.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:46:53.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:46:54.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:54 smithi078 ceph-mon[38143]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:46:54.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:54 smithi078 ceph-mon[38143]: Updating smithi078:/etc/ceph/ceph.conf 2024-03-14T09:46:54.778 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:54 smithi078 ceph-mon[38143]: Updating smithi078:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:46:55.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.022+0000 7f5b75c00700 1 -- 172.21.15.78:0/3915316413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b70106590 msgr2=0x7f5b70106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:55.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.022+0000 7f5b75c00700 1 --2- 172.21.15.78:0/3915316413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b70106590 0x7f5b70106970 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009910 tx=0x7f5b60009c20 comp rx=0 tx=0).stop 2024-03-14T09:46:55.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.022+0000 7f5b75c00700 1 -- 172.21.15.78:0/3915316413 shutdown_connections 2024-03-14T09:46:55.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.022+0000 7f5b75c00700 1 --2- 172.21.15.78:0/3915316413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b70106590 0x7f5b70106970 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:55.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.022+0000 7f5b75c00700 1 -- 172.21.15.78:0/3915316413 >> 172.21.15.78:0/3915316413 conn(0x7f5b700fbf90 msgr2=0x7f5b700fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:55.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.022+0000 7f5b75c00700 1 -- 172.21.15.78:0/3915316413 shutdown_connections 2024-03-14T09:46:55.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.022+0000 7f5b75c00700 1 -- 172.21.15.78:0/3915316413 wait complete. 2024-03-14T09:46:55.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.023+0000 7f5b75c00700 1 Processor -- start 2024-03-14T09:46:55.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.023+0000 7f5b75c00700 1 -- start start 2024-03-14T09:46:55.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.024+0000 7f5b75c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b70106590 0x7f5b700730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:55.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.024+0000 7f5b75c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b70074f70 con 0x7f5b70106590 2024-03-14T09:46:55.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.024+0000 7f5b6fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b70106590 0x7f5b700730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:55.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.024+0000 7f5b6fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b70106590 0x7f5b700730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51228/0 (socket says 172.21.15.78:51228) 2024-03-14T09:46:55.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.024+0000 7f5b6fe00700 1 -- 172.21.15.78:0/1445477505 learned_addr learned my addr 172.21.15.78:0/1445477505 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:55.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.024+0000 7f5b6fe00700 1 -- 172.21.15.78:0/1445477505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b600095f0 con 0x7f5b70106590 2024-03-14T09:46:55.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.025+0000 7f5b6fe00700 1 --2- 172.21.15.78:0/1445477505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b70106590 0x7f5b700730b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5b60006010 tx=0x7f5b60004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:55.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.025+0000 7f5b6cc00700 1 -- 172.21.15.78:0/1445477505 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b6001d070 con 0x7f5b70106590 2024-03-14T09:46:55.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.025+0000 7f5b6cc00700 1 -- 172.21.15.78:0/1445477505 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b6000bc10 con 0x7f5b70106590 2024-03-14T09:46:55.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.025+0000 7f5b75c00700 1 -- 172.21.15.78:0/1445477505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b700735f0 con 0x7f5b70106590 2024-03-14T09:46:55.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.025+0000 7f5b6cc00700 1 -- 172.21.15.78:0/1445477505 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5b600038a0 con 0x7f5b70106590 2024-03-14T09:46:55.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.025+0000 7f5b75c00700 1 -- 172.21.15.78:0/1445477505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b70073a70 con 0x7f5b70106590 2024-03-14T09:46:55.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.026+0000 7f5b6cc00700 1 -- 172.21.15.78:0/1445477505 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45430+0+0 (secure 0 0 0) 0x7f5b6000b6f0 con 0x7f5b70106590 2024-03-14T09:46:55.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.026+0000 7f5b75c00700 1 -- 172.21.15.78:0/1445477505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b70101c00 con 0x7f5b70106590 2024-03-14T09:46:55.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.027+0000 7f5b6cc00700 1 --2- 172.21.15.78:0/1445477505 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5b54038570 0x7f5b5403aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:55.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.027+0000 7f5b6cc00700 1 -- 172.21.15.78:0/1445477505 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5b6004c100 con 0x7f5b70106590 2024-03-14T09:46:55.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.027+0000 7f5b6f400700 1 --2- 172.21.15.78:0/1445477505 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5b54038570 0x7f5b5403aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:55.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.027+0000 7f5b6f400700 1 --2- 172.21.15.78:0/1445477505 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5b54038570 0x7f5b5403aa30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c006fd0 tx=0x7f5b5c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:55.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.031+0000 7f5b6cc00700 1 -- 172.21.15.78:0/1445477505 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5b60029540 con 0x7f5b70106590 2024-03-14T09:46:55.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:55.245+0000 7f5b75c00700 1 -- 172.21.15.78:0/1445477505 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi132", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b70073d10 con 0x7f5b54038570 2024-03-14T09:46:55.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:55 smithi078 ceph-mon[38143]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:46:55.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:55 smithi078 ceph-mon[38143]: Updating smithi078:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T09:46:55.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:55 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:46:55.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:55 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:46:55.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:55 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:46:56.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:56 smithi078 ceph-mon[38143]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi132", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:46:57.418 INFO:teuthology.orchestra.run.smithi078.stdout:Added host 'smithi132' with addr '172.21.15.132' 2024-03-14T09:46:57.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:57.417+0000 7f5b6cc00700 1 -- 172.21.15.78:0/1445477505 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5b70073d10 con 0x7f5b54038570 2024-03-14T09:46:57.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:57.420+0000 7f5b75c00700 1 -- 172.21.15.78:0/1445477505 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5b54038570 msgr2=0x7f5b5403aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:57.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:57.420+0000 7f5b75c00700 1 --2- 172.21.15.78:0/1445477505 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5b54038570 0x7f5b5403aa30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c006fd0 tx=0x7f5b5c006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:57.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:57.420+0000 7f5b75c00700 1 -- 172.21.15.78:0/1445477505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b70106590 msgr2=0x7f5b700730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:57.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:57.420+0000 7f5b75c00700 1 --2- 172.21.15.78:0/1445477505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b70106590 0x7f5b700730b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5b60006010 tx=0x7f5b60004ff0 comp rx=0 tx=0).stop 2024-03-14T09:46:57.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:57.420+0000 7f5b75c00700 1 -- 172.21.15.78:0/1445477505 shutdown_connections 2024-03-14T09:46:57.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:57.420+0000 7f5b75c00700 1 --2- 172.21.15.78:0/1445477505 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5b54038570 0x7f5b5403aa30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:57.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:57.420+0000 7f5b75c00700 1 --2- 172.21.15.78:0/1445477505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b70106590 0x7f5b700730b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:57.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:57.420+0000 7f5b75c00700 1 -- 172.21.15.78:0/1445477505 >> 172.21.15.78:0/1445477505 conn(0x7f5b700fbf90 msgr2=0x7f5b700fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:57.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:57.420+0000 7f5b75c00700 1 -- 172.21.15.78:0/1445477505 shutdown_connections 2024-03-14T09:46:57.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:57.420+0000 7f5b75c00700 1 -- 172.21.15.78:0/1445477505 wait complete. 2024-03-14T09:46:57.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:57 smithi078 ceph-mon[38143]: Deploying cephadm binary to smithi132 2024-03-14T09:46:57.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:57 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:46:57.751 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:57 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:46:57.781 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-14T09:46:58.127 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:46:58.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:58 smithi078 ceph-mon[38143]: Added host smithi132 2024-03-14T09:46:58.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:58 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:46:58.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:58 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:46:59.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.213+0000 7f2f20400700 1 -- 172.21.15.78:0/2415910110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f18106680 msgr2=0x7f2f18106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:59.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.213+0000 7f2f20400700 1 --2- 172.21.15.78:0/2415910110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f18106680 0x7f2f18106a60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c009910 tx=0x7f2f0c009c20 comp rx=0 tx=0).stop 2024-03-14T09:46:59.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.213+0000 7f2f20400700 1 -- 172.21.15.78:0/2415910110 shutdown_connections 2024-03-14T09:46:59.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.213+0000 7f2f20400700 1 --2- 172.21.15.78:0/2415910110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f18106680 0x7f2f18106a60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:59.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.213+0000 7f2f20400700 1 -- 172.21.15.78:0/2415910110 >> 172.21.15.78:0/2415910110 conn(0x7f2f180fc190 msgr2=0x7f2f180fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:59.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.213+0000 7f2f20400700 1 -- 172.21.15.78:0/2415910110 shutdown_connections 2024-03-14T09:46:59.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.213+0000 7f2f20400700 1 -- 172.21.15.78:0/2415910110 wait complete. 2024-03-14T09:46:59.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.214+0000 7f2f20400700 1 Processor -- start 2024-03-14T09:46:59.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.214+0000 7f2f20400700 1 -- start start 2024-03-14T09:46:59.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.214+0000 7f2f20400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f18106680 0x7f2f18198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:59.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.214+0000 7f2f20400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f18198ab0 con 0x7f2f18106680 2024-03-14T09:46:59.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.215+0000 7f2f1da00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f18106680 0x7f2f18198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:59.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.215+0000 7f2f1da00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f18106680 0x7f2f18198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51248/0 (socket says 172.21.15.78:51248) 2024-03-14T09:46:59.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.215+0000 7f2f1da00700 1 -- 172.21.15.78:0/1608743624 learned_addr learned my addr 172.21.15.78:0/1608743624 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:46:59.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.215+0000 7f2f1da00700 1 -- 172.21.15.78:0/1608743624 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c0095f0 con 0x7f2f18106680 2024-03-14T09:46:59.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.215+0000 7f2f1da00700 1 --2- 172.21.15.78:0/1608743624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f18106680 0x7f2f18198570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c006010 tx=0x7f2f0c004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:59.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.215+0000 7f2f16000700 1 -- 172.21.15.78:0/1608743624 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f0c01d070 con 0x7f2f18106680 2024-03-14T09:46:59.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.215+0000 7f2f16000700 1 -- 172.21.15.78:0/1608743624 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2f0c00bbb0 con 0x7f2f18106680 2024-03-14T09:46:59.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.216+0000 7f2f20400700 1 -- 172.21.15.78:0/1608743624 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f18198d10 con 0x7f2f18106680 2024-03-14T09:46:59.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.216+0000 7f2f16000700 1 -- 172.21.15.78:0/1608743624 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f0c003810 con 0x7f2f18106680 2024-03-14T09:46:59.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.216+0000 7f2f20400700 1 -- 172.21.15.78:0/1608743624 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f18199190 con 0x7f2f18106680 2024-03-14T09:46:59.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.217+0000 7f2f16000700 1 -- 172.21.15.78:0/1608743624 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45430+0+0 (secure 0 0 0) 0x7f2f0c00b6f0 con 0x7f2f18106680 2024-03-14T09:46:59.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.217+0000 7f2f20400700 1 -- 172.21.15.78:0/1608743624 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f18101dc0 con 0x7f2f18106680 2024-03-14T09:46:59.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.217+0000 7f2f16000700 1 --2- 172.21.15.78:0/1608743624 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2f000385c0 0x7f2f0003aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:46:59.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.217+0000 7f2f16000700 1 -- 172.21.15.78:0/1608743624 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2f0c04c130 con 0x7f2f18106680 2024-03-14T09:46:59.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.218+0000 7f2f1d000700 1 --2- 172.21.15.78:0/1608743624 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2f000385c0 0x7f2f0003aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:46:59.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.218+0000 7f2f1d000700 1 --2- 172.21.15.78:0/1608743624 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2f000385c0 0x7f2f0003aa80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2f04006fd0 tx=0x7f2f04006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:46:59.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.222+0000 7f2f16000700 1 -- 172.21.15.78:0/1608743624 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2f0c029540 con 0x7f2f18106680 2024-03-14T09:46:59.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.430+0000 7f2f20400700 1 -- 172.21.15.78:0/1608743624 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f1810adb0 con 0x7f2f000385c0 2024-03-14T09:46:59.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.432+0000 7f2f16000700 1 -- 172.21.15.78:0/1608743624 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f2f1810adb0 con 0x7f2f000385c0 2024-03-14T09:46:59.432 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:46:59.432 INFO:teuthology.orchestra.run.smithi078.stdout:[{"addr": "172.21.15.78", "hostname": "smithi078", "labels": [], "status": ""}, {"addr": "172.21.15.132", "hostname": "smithi132", "labels": [], "status": ""}] 2024-03-14T09:46:59.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.434+0000 7f2f20400700 1 -- 172.21.15.78:0/1608743624 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2f000385c0 msgr2=0x7f2f0003aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:59.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.434+0000 7f2f20400700 1 --2- 172.21.15.78:0/1608743624 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2f000385c0 0x7f2f0003aa80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2f04006fd0 tx=0x7f2f04006e40 comp rx=0 tx=0).stop 2024-03-14T09:46:59.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.434+0000 7f2f20400700 1 -- 172.21.15.78:0/1608743624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f18106680 msgr2=0x7f2f18198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:46:59.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.434+0000 7f2f20400700 1 --2- 172.21.15.78:0/1608743624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f18106680 0x7f2f18198570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c006010 tx=0x7f2f0c004ff0 comp rx=0 tx=0).stop 2024-03-14T09:46:59.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.434+0000 7f2f20400700 1 -- 172.21.15.78:0/1608743624 shutdown_connections 2024-03-14T09:46:59.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.434+0000 7f2f20400700 1 --2- 172.21.15.78:0/1608743624 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2f000385c0 0x7f2f0003aa80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:59.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.434+0000 7f2f20400700 1 --2- 172.21.15.78:0/1608743624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f18106680 0x7f2f18198570 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:46:59.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.434+0000 7f2f20400700 1 -- 172.21.15.78:0/1608743624 >> 172.21.15.78:0/1608743624 conn(0x7f2f180fc190 msgr2=0x7f2f18108260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:46:59.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.435+0000 7f2f20400700 1 -- 172.21.15.78:0/1608743624 shutdown_connections 2024-03-14T09:46:59.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:46:59.435+0000 7f2f20400700 1 -- 172.21.15.78:0/1608743624 wait complete. 2024-03-14T09:46:59.842 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi152 2024-03-14T09:46:59.842 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-14T09:46:59.842 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.conf 2024-03-14T09:46:59.864 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-14T09:46:59.864 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:46:59.924 INFO:tasks.cephadm:Adding host smithi152 to orchestrator... 2024-03-14T09:46:59.924 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch host add smithi152 2024-03-14T09:47:00.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:59 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:00.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:46:59 smithi078 ceph-mon[38143]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:47:00.270 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:47:01.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.417+0000 7fe2e6000700 1 -- 172.21.15.78:0/123812844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e0106680 msgr2=0x7fe2e0106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:01.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.417+0000 7fe2e6000700 1 --2- 172.21.15.78:0/123812844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e0106680 0x7fe2e0106a60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009910 tx=0x7fe2d0009c20 comp rx=0 tx=0).stop 2024-03-14T09:47:01.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.417+0000 7fe2e6000700 1 -- 172.21.15.78:0/123812844 shutdown_connections 2024-03-14T09:47:01.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.417+0000 7fe2e6000700 1 --2- 172.21.15.78:0/123812844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e0106680 0x7fe2e0106a60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:01.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.417+0000 7fe2e6000700 1 -- 172.21.15.78:0/123812844 >> 172.21.15.78:0/123812844 conn(0x7fe2e00fc190 msgr2=0x7fe2e00fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:01.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.417+0000 7fe2e6000700 1 -- 172.21.15.78:0/123812844 shutdown_connections 2024-03-14T09:47:01.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.417+0000 7fe2e6000700 1 -- 172.21.15.78:0/123812844 wait complete. 2024-03-14T09:47:01.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.418+0000 7fe2e6000700 1 Processor -- start 2024-03-14T09:47:01.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.418+0000 7fe2e6000700 1 -- start start 2024-03-14T09:47:01.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.418+0000 7fe2e6000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e0106680 0x7fe2e0198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:01.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.418+0000 7fe2e6000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e0198ab0 con 0x7fe2e0106680 2024-03-14T09:47:01.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.419+0000 7fe2df400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e0106680 0x7fe2e0198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:01.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.419+0000 7fe2df400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e0106680 0x7fe2e0198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51270/0 (socket says 172.21.15.78:51270) 2024-03-14T09:47:01.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.419+0000 7fe2df400700 1 -- 172.21.15.78:0/2323696470 learned_addr learned my addr 172.21.15.78:0/2323696470 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:47:01.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.419+0000 7fe2df400700 1 -- 172.21.15.78:0/2323696470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d00095f0 con 0x7fe2e0106680 2024-03-14T09:47:01.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.419+0000 7fe2df400700 1 --2- 172.21.15.78:0/2323696470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e0106680 0x7fe2e0198570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0006010 tx=0x7fe2d0004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:01.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.419+0000 7fe2d7e00700 1 -- 172.21.15.78:0/2323696470 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe2d001d070 con 0x7fe2e0106680 2024-03-14T09:47:01.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.419+0000 7fe2d7e00700 1 -- 172.21.15.78:0/2323696470 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2d000bbb0 con 0x7fe2e0106680 2024-03-14T09:47:01.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.419+0000 7fe2e6000700 1 -- 172.21.15.78:0/2323696470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e0198d10 con 0x7fe2e0106680 2024-03-14T09:47:01.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.419+0000 7fe2d7e00700 1 -- 172.21.15.78:0/2323696470 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe2d0003810 con 0x7fe2e0106680 2024-03-14T09:47:01.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.420+0000 7fe2e6000700 1 -- 172.21.15.78:0/2323696470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e0199190 con 0x7fe2e0106680 2024-03-14T09:47:01.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.420+0000 7fe2d7e00700 1 -- 172.21.15.78:0/2323696470 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45430+0+0 (secure 0 0 0) 0x7fe2d0003a10 con 0x7fe2e0106680 2024-03-14T09:47:01.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.421+0000 7fe2e6000700 1 -- 172.21.15.78:0/2323696470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e0101dc0 con 0x7fe2e0106680 2024-03-14T09:47:01.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.421+0000 7fe2d7e00700 1 --2- 172.21.15.78:0/2323696470 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe2c4038570 0x7fe2c403aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:01.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.421+0000 7fe2d7e00700 1 -- 172.21.15.78:0/2323696470 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe2d004c360 con 0x7fe2e0106680 2024-03-14T09:47:01.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.421+0000 7fe2dea00700 1 --2- 172.21.15.78:0/2323696470 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe2c4038570 0x7fe2c403aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:01.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.422+0000 7fe2dea00700 1 --2- 172.21.15.78:0/2323696470 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe2c4038570 0x7fe2c403aa30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc006fd0 tx=0x7fe2cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:01.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.426+0000 7fe2d7e00700 1 -- 172.21.15.78:0/2323696470 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe2d0029770 con 0x7fe2e0106680 2024-03-14T09:47:01.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:01.635+0000 7fe2e6000700 1 -- 172.21.15.78:0/2323696470 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2e010adb0 con 0x7fe2c4038570 2024-03-14T09:47:02.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:02 smithi078 ceph-mon[38143]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:02.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:02.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:02.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:02.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:02.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:47:02.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:02.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:02.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:02 smithi078 ceph-mon[38143]: Updating smithi132:/etc/ceph/ceph.conf 2024-03-14T09:47:02.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:02 smithi078 ceph-mon[38143]: Updating smithi132:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:47:02.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:02.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:02.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:03.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:03 smithi078 ceph-mon[38143]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:47:03.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:03 smithi078 ceph-mon[38143]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:47:03.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:03 smithi078 ceph-mon[38143]: Updating smithi132:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T09:47:03.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:03 smithi078 ceph-mon[38143]: Deploying cephadm binary to smithi152 2024-03-14T09:47:03.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:03.797+0000 7fe2d7e00700 1 -- 172.21.15.78:0/2323696470 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fe2e010adb0 con 0x7fe2c4038570 2024-03-14T09:47:03.798 INFO:teuthology.orchestra.run.smithi078.stdout:Added host 'smithi152' with addr '172.21.15.152' 2024-03-14T09:47:03.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:03.800+0000 7fe2e6000700 1 -- 172.21.15.78:0/2323696470 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe2c4038570 msgr2=0x7fe2c403aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:03.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:03.800+0000 7fe2e6000700 1 --2- 172.21.15.78:0/2323696470 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe2c4038570 0x7fe2c403aa30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc006fd0 tx=0x7fe2cc006e40 comp rx=0 tx=0).stop 2024-03-14T09:47:03.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:03.800+0000 7fe2e6000700 1 -- 172.21.15.78:0/2323696470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e0106680 msgr2=0x7fe2e0198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:03.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:03.800+0000 7fe2e6000700 1 --2- 172.21.15.78:0/2323696470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e0106680 0x7fe2e0198570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0006010 tx=0x7fe2d0004ff0 comp rx=0 tx=0).stop 2024-03-14T09:47:03.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:03.800+0000 7fe2e6000700 1 -- 172.21.15.78:0/2323696470 shutdown_connections 2024-03-14T09:47:03.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:03.800+0000 7fe2e6000700 1 --2- 172.21.15.78:0/2323696470 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe2c4038570 0x7fe2c403aa30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:03.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:03.800+0000 7fe2e6000700 1 --2- 172.21.15.78:0/2323696470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e0106680 0x7fe2e0198570 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:03.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:03.800+0000 7fe2e6000700 1 -- 172.21.15.78:0/2323696470 >> 172.21.15.78:0/2323696470 conn(0x7fe2e00fc190 msgr2=0x7fe2e0108280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:03.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:03.800+0000 7fe2e6000700 1 -- 172.21.15.78:0/2323696470 shutdown_connections 2024-03-14T09:47:03.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:03.800+0000 7fe2e6000700 1 -- 172.21.15.78:0/2323696470 wait complete. 2024-03-14T09:47:04.186 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-14T09:47:04.532 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:47:04.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:04 smithi078 ceph-mon[38143]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:04.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:04 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:04.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:04 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:04.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:04 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:05.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.631+0000 7f3fb7a00700 1 -- 172.21.15.78:0/1123037744 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0106680 msgr2=0x7f3fb0106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:05.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.631+0000 7f3fb7a00700 1 --2- 172.21.15.78:0/1123037744 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0106680 0x7f3fb0106a60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4009910 tx=0x7f3fa4009c20 comp rx=0 tx=0).stop 2024-03-14T09:47:05.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.631+0000 7f3fb7a00700 1 -- 172.21.15.78:0/1123037744 shutdown_connections 2024-03-14T09:47:05.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.631+0000 7f3fb7a00700 1 --2- 172.21.15.78:0/1123037744 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0106680 0x7f3fb0106a60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:05.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.631+0000 7f3fb7a00700 1 -- 172.21.15.78:0/1123037744 >> 172.21.15.78:0/1123037744 conn(0x7f3fb00fc190 msgr2=0x7f3fb00fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:05.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.631+0000 7f3fb7a00700 1 -- 172.21.15.78:0/1123037744 shutdown_connections 2024-03-14T09:47:05.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.631+0000 7f3fb7a00700 1 -- 172.21.15.78:0/1123037744 wait complete. 2024-03-14T09:47:05.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.633+0000 7f3fb7a00700 1 Processor -- start 2024-03-14T09:47:05.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.633+0000 7f3fb7a00700 1 -- start start 2024-03-14T09:47:05.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.633+0000 7f3fb7a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0106680 0x7f3fb0196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:05.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.633+0000 7f3fb7a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb01968b0 con 0x7f3fb0106680 2024-03-14T09:47:05.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.634+0000 7f3fb5000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0106680 0x7f3fb0196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:05.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.634+0000 7f3fb5000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0106680 0x7f3fb0196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45244/0 (socket says 172.21.15.78:45244) 2024-03-14T09:47:05.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.634+0000 7f3fb5000700 1 -- 172.21.15.78:0/2368816182 learned_addr learned my addr 172.21.15.78:0/2368816182 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:47:05.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.634+0000 7f3fb5000700 1 -- 172.21.15.78:0/2368816182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa40095f0 con 0x7f3fb0106680 2024-03-14T09:47:05.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.634+0000 7f3fb5000700 1 --2- 172.21.15.78:0/2368816182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0106680 0x7f3fb0196370 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4006010 tx=0x7f3fa4004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:05.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.634+0000 7f3fad600700 1 -- 172.21.15.78:0/2368816182 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fa401d070 con 0x7f3fb0106680 2024-03-14T09:47:05.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.634+0000 7f3fad600700 1 -- 172.21.15.78:0/2368816182 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3fa400bbb0 con 0x7f3fb0106680 2024-03-14T09:47:05.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.634+0000 7f3fad600700 1 -- 172.21.15.78:0/2368816182 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fa4003810 con 0x7f3fb0106680 2024-03-14T09:47:05.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.634+0000 7f3fb7a00700 1 -- 172.21.15.78:0/2368816182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb0196b10 con 0x7f3fb0106680 2024-03-14T09:47:05.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.635+0000 7f3fb7a00700 1 -- 172.21.15.78:0/2368816182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb0196f90 con 0x7f3fb0106680 2024-03-14T09:47:05.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.635+0000 7f3fad600700 1 -- 172.21.15.78:0/2368816182 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45430+0+0 (secure 0 0 0) 0x7f3fa400b6f0 con 0x7f3fb0106680 2024-03-14T09:47:05.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.636+0000 7f3fb7a00700 1 -- 172.21.15.78:0/2368816182 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb0101dc0 con 0x7f3fb0106680 2024-03-14T09:47:05.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.636+0000 7f3fad600700 1 --2- 172.21.15.78:0/2368816182 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3f980385c0 0x7f3f9803aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:05.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.636+0000 7f3fad600700 1 -- 172.21.15.78:0/2368816182 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3fa404c170 con 0x7f3fb0106680 2024-03-14T09:47:05.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.636+0000 7f3fafe00700 1 --2- 172.21.15.78:0/2368816182 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3f980385c0 0x7f3f9803aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:05.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.637+0000 7f3fafe00700 1 --2- 172.21.15.78:0/2368816182 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3f980385c0 0x7f3f9803aa80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c006fd0 tx=0x7f3f9c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:05.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.641+0000 7f3fad600700 1 -- 172.21.15.78:0/2368816182 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3fa4029540 con 0x7f3fb0106680 2024-03-14T09:47:05.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:05 smithi078 ceph-mon[38143]: Added host smithi152 2024-03-14T09:47:05.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:05 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:05.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:05 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:05.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.847+0000 7f3fb7a00700 1 -- 172.21.15.78:0/2368816182 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3fb010adb0 con 0x7f3f980385c0 2024-03-14T09:47:05.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.848+0000 7f3fad600700 1 -- 172.21.15.78:0/2368816182 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f3fb010adb0 con 0x7f3f980385c0 2024-03-14T09:47:05.849 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:47:05.849 INFO:teuthology.orchestra.run.smithi078.stdout:[{"addr": "172.21.15.78", "hostname": "smithi078", "labels": [], "status": ""}, {"addr": "172.21.15.132", "hostname": "smithi132", "labels": [], "status": ""}, {"addr": "172.21.15.152", "hostname": "smithi152", "labels": [], "status": ""}] 2024-03-14T09:47:05.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.851+0000 7f3fb7a00700 1 -- 172.21.15.78:0/2368816182 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3f980385c0 msgr2=0x7f3f9803aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:05.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.851+0000 7f3fb7a00700 1 --2- 172.21.15.78:0/2368816182 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3f980385c0 0x7f3f9803aa80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c006fd0 tx=0x7f3f9c006e40 comp rx=0 tx=0).stop 2024-03-14T09:47:05.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.851+0000 7f3fb7a00700 1 -- 172.21.15.78:0/2368816182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0106680 msgr2=0x7f3fb0196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:05.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.851+0000 7f3fb7a00700 1 --2- 172.21.15.78:0/2368816182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0106680 0x7f3fb0196370 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4006010 tx=0x7f3fa4004ff0 comp rx=0 tx=0).stop 2024-03-14T09:47:05.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.851+0000 7f3fb7a00700 1 -- 172.21.15.78:0/2368816182 shutdown_connections 2024-03-14T09:47:05.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.851+0000 7f3fb7a00700 1 --2- 172.21.15.78:0/2368816182 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3f980385c0 0x7f3f9803aa80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:05.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.851+0000 7f3fb7a00700 1 --2- 172.21.15.78:0/2368816182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fb0106680 0x7f3fb0196370 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:05.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.851+0000 7f3fb7a00700 1 -- 172.21.15.78:0/2368816182 >> 172.21.15.78:0/2368816182 conn(0x7f3fb00fc190 msgr2=0x7f3fb00fd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:05.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.851+0000 7f3fb7a00700 1 -- 172.21.15.78:0/2368816182 shutdown_connections 2024-03-14T09:47:05.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:05.851+0000 7f3fb7a00700 1 -- 172.21.15.78:0/2368816182 wait complete. 2024-03-14T09:47:06.186 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-14T09:47:06.186 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-14T09:47:06.532 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:47:06.689 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:06 smithi078 ceph-mon[38143]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:07.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.658+0000 7f5561400700 1 -- 172.21.15.78:0/578579265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f555c0731e0 msgr2=0x7f555c0735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:07.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.658+0000 7f5561400700 1 --2- 172.21.15.78:0/578579265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f555c0731e0 0x7f555c0735c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f554c009910 tx=0x7f554c009c20 comp rx=0 tx=0).stop 2024-03-14T09:47:07.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.658+0000 7f5561400700 1 -- 172.21.15.78:0/578579265 shutdown_connections 2024-03-14T09:47:07.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.658+0000 7f5561400700 1 --2- 172.21.15.78:0/578579265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f555c0731e0 0x7f555c0735c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:07.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.658+0000 7f5561400700 1 -- 172.21.15.78:0/578579265 >> 172.21.15.78:0/578579265 conn(0x7f555c0fc190 msgr2=0x7f555c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:07.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.658+0000 7f5561400700 1 -- 172.21.15.78:0/578579265 shutdown_connections 2024-03-14T09:47:07.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.659+0000 7f5561400700 1 -- 172.21.15.78:0/578579265 wait complete. 2024-03-14T09:47:07.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.660+0000 7f5561400700 1 Processor -- start 2024-03-14T09:47:07.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.660+0000 7f5561400700 1 -- start start 2024-03-14T09:47:07.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.660+0000 7f5561400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f555c0731e0 0x7f555c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:07.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.660+0000 7f5561400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555c198b50 con 0x7f555c0731e0 2024-03-14T09:47:07.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.660+0000 7f555a800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f555c0731e0 0x7f555c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:07.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.660+0000 7f555a800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f555c0731e0 0x7f555c198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45272/0 (socket says 172.21.15.78:45272) 2024-03-14T09:47:07.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.660+0000 7f555a800700 1 -- 172.21.15.78:0/2459389783 learned_addr learned my addr 172.21.15.78:0/2459389783 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:47:07.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.661+0000 7f555a800700 1 -- 172.21.15.78:0/2459389783 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c0095f0 con 0x7f555c0731e0 2024-03-14T09:47:07.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.661+0000 7f555a800700 1 --2- 172.21.15.78:0/2459389783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f555c0731e0 0x7f555c198610 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f554c004f40 tx=0x7f554c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:07.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.661+0000 7f5553400700 1 -- 172.21.15.78:0/2459389783 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f554c01d070 con 0x7f555c0731e0 2024-03-14T09:47:07.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.661+0000 7f5553400700 1 -- 172.21.15.78:0/2459389783 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f554c004370 con 0x7f555c0731e0 2024-03-14T09:47:07.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.661+0000 7f5561400700 1 -- 172.21.15.78:0/2459389783 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f555c198d50 con 0x7f555c0731e0 2024-03-14T09:47:07.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.661+0000 7f5553400700 1 -- 172.21.15.78:0/2459389783 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f554c003970 con 0x7f555c0731e0 2024-03-14T09:47:07.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.661+0000 7f5561400700 1 -- 172.21.15.78:0/2459389783 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f555c199170 con 0x7f555c0731e0 2024-03-14T09:47:07.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.662+0000 7f5553400700 1 -- 172.21.15.78:0/2459389783 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45430+0+0 (secure 0 0 0) 0x7f554c003c30 con 0x7f555c0731e0 2024-03-14T09:47:07.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.663+0000 7f5561400700 1 -- 172.21.15.78:0/2459389783 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f555c109cd0 con 0x7f555c0731e0 2024-03-14T09:47:07.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.663+0000 7f5553400700 1 --2- 172.21.15.78:0/2459389783 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5540038570 0x7f554003aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:07.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.663+0000 7f5553400700 1 -- 172.21.15.78:0/2459389783 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f554c04c270 con 0x7f555c0731e0 2024-03-14T09:47:07.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.663+0000 7f5559e00700 1 --2- 172.21.15.78:0/2459389783 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5540038570 0x7f554003aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:07.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.664+0000 7f5559e00700 1 --2- 172.21.15.78:0/2459389783 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5540038570 0x7f554003aa30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5548006fd0 tx=0x7f5548006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:07.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.668+0000 7f5553400700 1 -- 172.21.15.78:0/2459389783 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f554c00fd00 con 0x7f555c0731e0 2024-03-14T09:47:07.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:07 smithi078 ceph-mon[38143]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:47:07.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:07.881+0000 7f5561400700 1 -- 172.21.15.78:0/2459389783 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f555c074890 con 0x7f555c0731e0 2024-03-14T09:47:08.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:08.613+0000 7f5553400700 1 -- 172.21.15.78:0/2459389783 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f554c026070 con 0x7f555c0731e0 2024-03-14T09:47:08.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:08.615+0000 7f5561400700 1 -- 172.21.15.78:0/2459389783 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5540038570 msgr2=0x7f554003aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:08.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:08.616+0000 7f5561400700 1 --2- 172.21.15.78:0/2459389783 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5540038570 0x7f554003aa30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5548006fd0 tx=0x7f5548006e40 comp rx=0 tx=0).stop 2024-03-14T09:47:08.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:08.616+0000 7f5561400700 1 -- 172.21.15.78:0/2459389783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f555c0731e0 msgr2=0x7f555c198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:08.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:08.616+0000 7f5561400700 1 --2- 172.21.15.78:0/2459389783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f555c0731e0 0x7f555c198610 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f554c004f40 tx=0x7f554c005020 comp rx=0 tx=0).stop 2024-03-14T09:47:08.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:08.616+0000 7f5561400700 1 -- 172.21.15.78:0/2459389783 shutdown_connections 2024-03-14T09:47:08.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:08.616+0000 7f5561400700 1 --2- 172.21.15.78:0/2459389783 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5540038570 0x7f554003aa30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:08.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:08.616+0000 7f5561400700 1 --2- 172.21.15.78:0/2459389783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f555c0731e0 0x7f555c198610 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:08.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:08.616+0000 7f5561400700 1 -- 172.21.15.78:0/2459389783 >> 172.21.15.78:0/2459389783 conn(0x7f555c0fc190 msgr2=0x7f555c106930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:08.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:08.616+0000 7f5561400700 1 -- 172.21.15.78:0/2459389783 shutdown_connections 2024-03-14T09:47:08.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:08.616+0000 7f5561400700 1 -- 172.21.15.78:0/2459389783 wait complete. 2024-03-14T09:47:08.617 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2024-03-14T09:47:08.792 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:08 smithi078 ceph-mon[38143]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:08 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:08 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:08 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:08 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:08 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:47:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:08 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:08 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:08 smithi078 ceph-mon[38143]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-14T09:47:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:08 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2459389783' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-14T09:47:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:08 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:08 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:08.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:08 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:08.961 INFO:tasks.cephadm:Adding mon.a on smithi078 2024-03-14T09:47:08.961 INFO:tasks.cephadm:Adding mon.b on smithi132 2024-03-14T09:47:08.961 INFO:tasks.cephadm:Adding mon.c on smithi152 2024-03-14T09:47:08.961 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi078:172.21.15.78=a;smithi132:172.21.15.132=b;smithi152:172.21.15.152=c' 2024-03-14T09:47:09.201 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:47:09.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:09 smithi078 ceph-mon[38143]: Updating smithi152:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:47:09.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:09 smithi078 ceph-mon[38143]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:47:09.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:09 smithi078 ceph-mon[38143]: Updating smithi152:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T09:47:09.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:09 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2459389783' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-14T09:47:09.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:09 smithi078 ceph-mon[38143]: osdmap e4: 0 total, 0 up, 0 in 2024-03-14T09:47:10.168 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.166+0000 7f6b13fff700 1 -- 172.21.15.152:0/118792932 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b0c005470 con 0x7f6b1c1066c0 2024-03-14T09:47:10.168 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.167+0000 7f6b232c9700 1 -- 172.21.15.152:0/118792932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b1c1066c0 msgr2=0x7f6b1c106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:10.168 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.167+0000 7f6b232c9700 1 --2- 172.21.15.152:0/118792932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b1c1066c0 0x7f6b1c106aa0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c009910 tx=0x7f6b0c009c20 comp rx=0 tx=0).stop 2024-03-14T09:47:10.169 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.167+0000 7f6b232c9700 1 -- 172.21.15.152:0/118792932 shutdown_connections 2024-03-14T09:47:10.169 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.167+0000 7f6b232c9700 1 --2- 172.21.15.152:0/118792932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b1c1066c0 0x7f6b1c106aa0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:10.169 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.167+0000 7f6b232c9700 1 -- 172.21.15.152:0/118792932 >> 172.21.15.152:0/118792932 conn(0x7f6b1c0fc1f0 msgr2=0x7f6b1c0fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:10.169 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.167+0000 7f6b232c9700 1 -- 172.21.15.152:0/118792932 shutdown_connections 2024-03-14T09:47:10.169 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.167+0000 7f6b232c9700 1 -- 172.21.15.152:0/118792932 wait complete. 2024-03-14T09:47:10.169 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.168+0000 7f6b232c9700 1 Processor -- start 2024-03-14T09:47:10.169 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.168+0000 7f6b232c9700 1 -- start start 2024-03-14T09:47:10.169 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.168+0000 7f6b232c9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b1c1066c0 0x7f6b1c196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:10.169 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.168+0000 7f6b232c9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1c1968a0 con 0x7f6b1c1066c0 2024-03-14T09:47:10.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.168+0000 7f6b21065700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b1c1066c0 0x7f6b1c196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:10.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.168+0000 7f6b21065700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b1c1066c0 0x7f6b1c196360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.152:34982/0 (socket says 172.21.15.152:34982) 2024-03-14T09:47:10.170 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.168+0000 7f6b21065700 1 -- 172.21.15.152:0/1842336478 learned_addr learned my addr 172.21.15.152:0/1842336478 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-14T09:47:10.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.169+0000 7f6b21065700 1 -- 172.21.15.152:0/1842336478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0c0095f0 con 0x7f6b1c1066c0 2024-03-14T09:47:10.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.169+0000 7f6b21065700 1 --2- 172.21.15.152:0/1842336478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b1c1066c0 0x7f6b1c196360 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c004f40 tx=0x7f6b0c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:10.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.169+0000 7f6b127fc700 1 -- 172.21.15.152:0/1842336478 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b0c01d070 con 0x7f6b1c1066c0 2024-03-14T09:47:10.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.169+0000 7f6b232c9700 1 -- 172.21.15.152:0/1842336478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b1c196b00 con 0x7f6b1c1066c0 2024-03-14T09:47:10.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.169+0000 7f6b127fc700 1 -- 172.21.15.152:0/1842336478 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b0c00bbb0 con 0x7f6b1c1066c0 2024-03-14T09:47:10.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.169+0000 7f6b232c9700 1 -- 172.21.15.152:0/1842336478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b1c196f80 con 0x7f6b1c1066c0 2024-03-14T09:47:10.171 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.169+0000 7f6b127fc700 1 -- 172.21.15.152:0/1842336478 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b0c003810 con 0x7f6b1c1066c0 2024-03-14T09:47:10.172 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.170+0000 7f6b232c9700 1 -- 172.21.15.152:0/1842336478 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b1c101e00 con 0x7f6b1c1066c0 2024-03-14T09:47:10.172 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.170+0000 7f6b127fc700 1 -- 172.21.15.152:0/1842336478 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45430+0+0 (secure 0 0 0) 0x7f6b0c00bd20 con 0x7f6b1c1066c0 2024-03-14T09:47:10.172 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.170+0000 7f6b127fc700 1 --2- 172.21.15.152:0/1842336478 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6b08038570 0x7f6b0803aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:10.172 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.170+0000 7f6b127fc700 1 -- 172.21.15.152:0/1842336478 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6b0c04c160 con 0x7f6b1c1066c0 2024-03-14T09:47:10.172 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.171+0000 7f6b20864700 1 --2- 172.21.15.152:0/1842336478 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6b08038570 0x7f6b0803aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:10.172 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.171+0000 7f6b20864700 1 --2- 172.21.15.152:0/1842336478 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6b08038570 0x7f6b0803aa30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6b18006fd0 tx=0x7f6b18006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:10.176 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.176+0000 7f6b127fc700 1 -- 172.21.15.152:0/1842336478 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6b0c02aba0 con 0x7f6b1c1066c0 2024-03-14T09:47:10.379 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.378+0000 7f6b232c9700 1 -- 172.21.15.152:0/1842336478 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi132:172.21.15.132=b;smithi152:172.21.15.152=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b1c10aeb0 con 0x7f6b08038570 2024-03-14T09:47:10.386 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.385+0000 7f6b127fc700 1 -- 172.21.15.152:0/1842336478 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6b1c10aeb0 con 0x7f6b08038570 2024-03-14T09:47:10.386 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled mon update... 2024-03-14T09:47:10.388 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.387+0000 7f6b232c9700 1 -- 172.21.15.152:0/1842336478 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6b08038570 msgr2=0x7f6b0803aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:10.388 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.387+0000 7f6b232c9700 1 --2- 172.21.15.152:0/1842336478 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6b08038570 0x7f6b0803aa30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6b18006fd0 tx=0x7f6b18006e40 comp rx=0 tx=0).stop 2024-03-14T09:47:10.388 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.387+0000 7f6b232c9700 1 -- 172.21.15.152:0/1842336478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b1c1066c0 msgr2=0x7f6b1c196360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:10.388 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.387+0000 7f6b232c9700 1 --2- 172.21.15.152:0/1842336478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b1c1066c0 0x7f6b1c196360 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c004f40 tx=0x7f6b0c005020 comp rx=0 tx=0).stop 2024-03-14T09:47:10.388 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.387+0000 7f6b232c9700 1 -- 172.21.15.152:0/1842336478 shutdown_connections 2024-03-14T09:47:10.388 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.387+0000 7f6b232c9700 1 --2- 172.21.15.152:0/1842336478 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6b08038570 0x7f6b0803aa30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:10.388 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.387+0000 7f6b232c9700 1 --2- 172.21.15.152:0/1842336478 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6b1c1066c0 0x7f6b1c196360 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:10.388 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.387+0000 7f6b232c9700 1 -- 172.21.15.152:0/1842336478 >> 172.21.15.152:0/1842336478 conn(0x7f6b1c0fc1f0 msgr2=0x7f6b1c108310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:10.389 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.387+0000 7f6b232c9700 1 -- 172.21.15.152:0/1842336478 shutdown_connections 2024-03-14T09:47:10.389 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:10.387+0000 7f6b232c9700 1 -- 172.21.15.152:0/1842336478 wait complete. 2024-03-14T09:47:10.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:10 smithi078 ceph-mon[38143]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:10.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:10.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:10.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:10.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:10.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:10.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T09:47:10.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:10.811 DEBUG:teuthology.orchestra.run.smithi132:mon.b> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mon.b.service 2024-03-14T09:47:10.814 DEBUG:teuthology.orchestra.run.smithi152:mon.c> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mon.c.service 2024-03-14T09:47:10.816 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-14T09:47:10.816 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T09:47:10.846 INFO:journalctl@ceph.mon.c.smithi152.stdout:-- Logs begin at Thu 2024-03-14 09:24:45 UTC. -- 2024-03-14T09:47:10.850 INFO:journalctl@ceph.mon.b.smithi132.stdout:-- Logs begin at Thu 2024-03-14 09:24:09 UTC. -- 2024-03-14T09:47:11.205 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:47:11.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:11 smithi078 ceph-mon[38143]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi132:172.21.15.132=b;smithi152:172.21.15.152=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:47:11.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:11 smithi078 ceph-mon[38143]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi132:172.21.15.132=b;smithi152:172.21.15.152=c;count:3 2024-03-14T09:47:11.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:11 smithi078 ceph-mon[38143]: Deploying daemon mon.c on smithi152 2024-03-14T09:47:12.759 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.757+0000 7f2ea7183700 1 -- 172.21.15.152:0/2623370945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ea0106690 msgr2=0x7f2ea0106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:12.759 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.757+0000 7f2ea7183700 1 --2- 172.21.15.152:0/2623370945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ea0106690 0x7f2ea0106a70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2e94009910 tx=0x7f2e94009c20 comp rx=0 tx=0).stop 2024-03-14T09:47:12.759 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.757+0000 7f2ea7183700 1 -- 172.21.15.152:0/2623370945 shutdown_connections 2024-03-14T09:47:12.759 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.757+0000 7f2ea7183700 1 --2- 172.21.15.152:0/2623370945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ea0106690 0x7f2ea0106a70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:12.759 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.757+0000 7f2ea7183700 1 -- 172.21.15.152:0/2623370945 >> 172.21.15.152:0/2623370945 conn(0x7f2ea00fc1c0 msgr2=0x7f2ea00fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:12.759 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.757+0000 7f2ea7183700 1 -- 172.21.15.152:0/2623370945 shutdown_connections 2024-03-14T09:47:12.760 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.757+0000 7f2ea7183700 1 -- 172.21.15.152:0/2623370945 wait complete. 2024-03-14T09:47:12.760 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.758+0000 7f2ea7183700 1 Processor -- start 2024-03-14T09:47:12.760 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.758+0000 7f2ea7183700 1 -- start start 2024-03-14T09:47:12.760 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.758+0000 7f2ea7183700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ea0106690 0x7f2ea0196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:12.760 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.758+0000 7f2ea7183700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea0196880 con 0x7f2ea0106690 2024-03-14T09:47:12.760 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.758+0000 7f2ea4f1f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ea0106690 0x7f2ea0196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:12.760 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.758+0000 7f2ea4f1f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ea0106690 0x7f2ea0196340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.152:34998/0 (socket says 172.21.15.152:34998) 2024-03-14T09:47:12.760 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.758+0000 7f2ea4f1f700 1 -- 172.21.15.152:0/3388215025 learned_addr learned my addr 172.21.15.152:0/3388215025 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-14T09:47:12.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.759+0000 7f2ea4f1f700 1 -- 172.21.15.152:0/3388215025 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e940095f0 con 0x7f2ea0106690 2024-03-14T09:47:12.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.759+0000 7f2ea4f1f700 1 --2- 172.21.15.152:0/3388215025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ea0106690 0x7f2ea0196340 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2e94004f40 tx=0x7f2e94005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:12.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.759+0000 7f2e9dffb700 1 -- 172.21.15.152:0/3388215025 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e9401d070 con 0x7f2ea0106690 2024-03-14T09:47:12.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.759+0000 7f2ea7183700 1 -- 172.21.15.152:0/3388215025 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea0196ae0 con 0x7f2ea0106690 2024-03-14T09:47:12.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.759+0000 7f2e9dffb700 1 -- 172.21.15.152:0/3388215025 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e9400bbb0 con 0x7f2ea0106690 2024-03-14T09:47:12.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.759+0000 7f2e9dffb700 1 -- 172.21.15.152:0/3388215025 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e94003810 con 0x7f2ea0106690 2024-03-14T09:47:12.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.759+0000 7f2ea7183700 1 -- 172.21.15.152:0/3388215025 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea0196f60 con 0x7f2ea0106690 2024-03-14T09:47:12.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.760+0000 7f2ea7183700 1 -- 172.21.15.152:0/3388215025 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea0101dd0 con 0x7f2ea0106690 2024-03-14T09:47:12.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.760+0000 7f2e9dffb700 1 -- 172.21.15.152:0/3388215025 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45430+0+0 (secure 0 0 0) 0x7f2e9400b6f0 con 0x7f2ea0106690 2024-03-14T09:47:12.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.761+0000 7f2e9dffb700 1 --2- 172.21.15.152:0/3388215025 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2e90038570 0x7f2e9003aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:12.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.761+0000 7f2e9dffb700 1 -- 172.21.15.152:0/3388215025 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2e9404c100 con 0x7f2ea0106690 2024-03-14T09:47:12.762 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.761+0000 7f2e9ffff700 1 --2- 172.21.15.152:0/3388215025 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2e90038570 0x7f2e9003aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:12.763 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.761+0000 7f2e9ffff700 1 --2- 172.21.15.152:0/3388215025 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2e90038570 0x7f2e9003aa30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c006fd0 tx=0x7f2e8c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:12.767 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:12.766+0000 7f2e9dffb700 1 -- 172.21.15.152:0/3388215025 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2e94029540 con 0x7f2ea0106690 2024-03-14T09:47:12.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:12 smithi078 ceph-mon[38143]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:13.082 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:13.081+0000 7f2ea7183700 1 -- 172.21.15.152:0/3388215025 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ea010adc0 con 0x7f2ea0106690 2024-03-14T09:47:13.083 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:13.082+0000 7f2e9dffb700 1 -- 172.21.15.152:0/3388215025 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7f2e94026070 con 0x7f2ea0106690 2024-03-14T09:47:13.083 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-14T09:47:13.083 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","modified":"2024-03-14T09:45:40.908003Z","created":"2024-03-14T09:45:40.908003Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T09:47:13.086 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:13.086+0000 7f2ea7183700 1 -- 172.21.15.152:0/3388215025 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2e90038570 msgr2=0x7f2e9003aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:13.087 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:13.086+0000 7f2ea7183700 1 --2- 172.21.15.152:0/3388215025 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2e90038570 0x7f2e9003aa30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c006fd0 tx=0x7f2e8c006e40 comp rx=0 tx=0).stop 2024-03-14T09:47:13.087 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:13.086+0000 7f2ea7183700 1 -- 172.21.15.152:0/3388215025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ea0106690 msgr2=0x7f2ea0196340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:13.087 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:13.086+0000 7f2ea7183700 1 --2- 172.21.15.152:0/3388215025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ea0106690 0x7f2ea0196340 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2e94004f40 tx=0x7f2e94005020 comp rx=0 tx=0).stop 2024-03-14T09:47:13.087 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:13.086+0000 7f2ea7183700 1 -- 172.21.15.152:0/3388215025 shutdown_connections 2024-03-14T09:47:13.087 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:13.086+0000 7f2ea7183700 1 --2- 172.21.15.152:0/3388215025 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2e90038570 0x7f2e9003aa30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:13.087 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:13.086+0000 7f2ea7183700 1 --2- 172.21.15.152:0/3388215025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ea0106690 0x7f2ea0196340 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:13.087 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:13.086+0000 7f2ea7183700 1 -- 172.21.15.152:0/3388215025 >> 172.21.15.152:0/3388215025 conn(0x7f2ea00fc1c0 msgr2=0x7f2ea01082e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:13.087 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:13.086+0000 7f2ea7183700 1 -- 172.21.15.152:0/3388215025 shutdown_connections 2024-03-14T09:47:13.087 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:13.086+0000 7f2ea7183700 1 -- 172.21.15.152:0/3388215025 wait complete. 2024-03-14T09:47:13.088 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2024-03-14T09:47:13.287 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 systemd[1]: Starting Ceph mon.c for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T09:47:14.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: set uid:gid to 167:167 (ceph:ceph) 2024-03-14T09:47:14.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable), process ceph-mon, pid 7 2024-03-14T09:47:14.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: pidfile_write: ignore empty --pid-file 2024-03-14T09:47:14.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: load: jerasure load: lrc 2024-03-14T09:47:14.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: RocksDB version: 7.9.2 2024-03-14T09:47:14.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Git sha 0 2024-03-14T09:47:14.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Compile date 2024-03-13 02:27:37 2024-03-14T09:47:14.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: DB SUMMARY 2024-03-14T09:47:14.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: DB Session ID: KM237S4H3QOX69N6P7DO 2024-03-14T09:47:14.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: CURRENT file: CURRENT 2024-03-14T09:47:14.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: IDENTITY file: IDENTITY 2024-03-14T09:47:14.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-14T09:47:14.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-14T09:47:14.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-14T09:47:14.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.error_if_exists: 0 2024-03-14T09:47:14.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.create_if_missing: 0 2024-03-14T09:47:14.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.paranoid_checks: 1 2024-03-14T09:47:14.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-14T09:47:14.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-14T09:47:14.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-14T09:47:14.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.env: 0x55fa1ce6a720 2024-03-14T09:47:14.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.fs: PosixFileSystem 2024-03-14T09:47:14.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.info_log: 0x55fa1f4854e0 2024-03-14T09:47:14.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_file_opening_threads: 16 2024-03-14T09:47:14.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.statistics: (nil) 2024-03-14T09:47:14.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.use_fsync: 0 2024-03-14T09:47:14.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_log_file_size: 0 2024-03-14T09:47:14.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-14T09:47:14.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-14T09:47:14.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.keep_log_file_num: 1000 2024-03-14T09:47:14.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.recycle_log_file_num: 0 2024-03-14T09:47:14.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.allow_fallocate: 1 2024-03-14T09:47:14.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.allow_mmap_reads: 0 2024-03-14T09:47:14.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.allow_mmap_writes: 0 2024-03-14T09:47:14.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.use_direct_reads: 0 2024-03-14T09:47:14.003 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-14T09:47:14.003 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.create_missing_column_families: 0 2024-03-14T09:47:14.003 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.db_log_dir: 2024-03-14T09:47:14.003 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.wal_dir: 2024-03-14T09:47:14.003 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-14T09:47:14.003 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-14T09:47:14.003 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-14T09:47:14.003 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-14T09:47:14.003 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-14T09:47:14.003 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-14T09:47:14.003 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.advise_random_on_open: 1 2024-03-14T09:47:14.003 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.db_write_buffer_size: 0 2024-03-14T09:47:14.003 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.write_buffer_manager: 0x55fa1e71e320 2024-03-14T09:47:14.003 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-14T09:47:14.004 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-14T09:47:14.004 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-14T09:47:14.004 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.rate_limiter: (nil) 2024-03-14T09:47:14.004 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-14T09:47:14.004 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.wal_recovery_mode: 2 2024-03-14T09:47:14.004 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.enable_thread_tracking: 0 2024-03-14T09:47:14.004 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.enable_pipelined_write: 0 2024-03-14T09:47:14.004 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.unordered_write: 0 2024-03-14T09:47:14.004 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-14T09:47:14.004 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-14T09:47:14.004 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-14T09:47:14.004 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-14T09:47:14.004 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.row_cache: None 2024-03-14T09:47:14.004 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.wal_filter: None 2024-03-14T09:47:14.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-14T09:47:14.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.allow_ingest_behind: 0 2024-03-14T09:47:14.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.two_write_queues: 0 2024-03-14T09:47:14.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.manual_wal_flush: 0 2024-03-14T09:47:14.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.wal_compression: 0 2024-03-14T09:47:14.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.atomic_flush: 0 2024-03-14T09:47:14.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-14T09:47:14.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-14T09:47:14.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-14T09:47:14.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.log_readahead_size: 0 2024-03-14T09:47:14.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-14T09:47:14.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.best_efforts_recovery: 0 2024-03-14T09:47:14.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-14T09:47:14.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-14T09:47:14.005 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.allow_data_in_errors: 0 2024-03-14T09:47:14.006 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.db_host_id: __hostname__ 2024-03-14T09:47:14.006 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-14T09:47:14.006 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_background_jobs: 2 2024-03-14T09:47:14.006 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_background_compactions: -1 2024-03-14T09:47:14.006 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_subcompactions: 1 2024-03-14T09:47:14.006 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-14T09:47:14.006 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-14T09:47:14.006 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-14T09:47:14.006 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_total_wal_size: 0 2024-03-14T09:47:14.006 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-14T09:47:14.006 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-14T09:47:14.006 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-14T09:47:14.006 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-14T09:47:14.006 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_open_files: -1 2024-03-14T09:47:14.007 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.bytes_per_sync: 0 2024-03-14T09:47:14.007 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-14T09:47:14.007 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-14T09:47:14.007 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compaction_readahead_size: 0 2024-03-14T09:47:14.007 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_background_flushes: -1 2024-03-14T09:47:14.007 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Compression algorithms supported: 2024-03-14T09:47:14.007 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-14T09:47:14.007 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: kZSTD supported: 0 2024-03-14T09:47:14.007 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: kXpressCompression supported: 0 2024-03-14T09:47:14.007 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: kLZ4HCCompression supported: 1 2024-03-14T09:47:14.007 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: kZlibCompression supported: 1 2024-03-14T09:47:14.007 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: kSnappyCompression supported: 1 2024-03-14T09:47:14.007 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: kLZ4Compression supported: 1 2024-03-14T09:47:14.007 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: kBZip2Compression supported: 0 2024-03-14T09:47:14.008 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-14T09:47:14.008 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-14T09:47:14.008 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-14T09:47:14.008 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-14T09:47:14.008 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-14T09:47:14.008 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.merge_operator: 2024-03-14T09:47:14.008 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compaction_filter: None 2024-03-14T09:47:14.008 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compaction_filter_factory: None 2024-03-14T09:47:14.008 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.sst_partitioner_factory: None 2024-03-14T09:47:14.008 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-14T09:47:14.008 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-14T09:47:14.008 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fa1f4855e0) 2024-03-14T09:47:14.008 INFO:journalctl@ceph.mon.c.smithi152.stdout: cache_index_and_filter_blocks: 1 2024-03-14T09:47:14.008 INFO:journalctl@ceph.mon.c.smithi152.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-14T09:47:14.009 INFO:journalctl@ceph.mon.c.smithi152.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-14T09:47:14.009 INFO:journalctl@ceph.mon.c.smithi152.stdout: pin_top_level_index_and_filter: 1 2024-03-14T09:47:14.009 INFO:journalctl@ceph.mon.c.smithi152.stdout: index_type: 0 2024-03-14T09:47:14.009 INFO:journalctl@ceph.mon.c.smithi152.stdout: data_block_index_type: 0 2024-03-14T09:47:14.009 INFO:journalctl@ceph.mon.c.smithi152.stdout: index_shortening: 1 2024-03-14T09:47:14.009 INFO:journalctl@ceph.mon.c.smithi152.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-14T09:47:14.009 INFO:journalctl@ceph.mon.c.smithi152.stdout: checksum: 4 2024-03-14T09:47:14.009 INFO:journalctl@ceph.mon.c.smithi152.stdout: no_block_cache: 0 2024-03-14T09:47:14.009 INFO:journalctl@ceph.mon.c.smithi152.stdout: block_cache: 0x55fa1e79f350 2024-03-14T09:47:14.009 INFO:journalctl@ceph.mon.c.smithi152.stdout: block_cache_name: BinnedLRUCache 2024-03-14T09:47:14.009 INFO:journalctl@ceph.mon.c.smithi152.stdout: block_cache_options: 2024-03-14T09:47:14.009 INFO:journalctl@ceph.mon.c.smithi152.stdout: capacity : 536870912 2024-03-14T09:47:14.009 INFO:journalctl@ceph.mon.c.smithi152.stdout: num_shard_bits : 4 2024-03-14T09:47:14.009 INFO:journalctl@ceph.mon.c.smithi152.stdout: strict_capacity_limit : 0 2024-03-14T09:47:14.010 INFO:journalctl@ceph.mon.c.smithi152.stdout: high_pri_pool_ratio: 0.000 2024-03-14T09:47:14.010 INFO:journalctl@ceph.mon.c.smithi152.stdout: block_cache_compressed: (nil) 2024-03-14T09:47:14.010 INFO:journalctl@ceph.mon.c.smithi152.stdout: persistent_cache: (nil) 2024-03-14T09:47:14.010 INFO:journalctl@ceph.mon.c.smithi152.stdout: block_size: 4096 2024-03-14T09:47:14.010 INFO:journalctl@ceph.mon.c.smithi152.stdout: block_size_deviation: 10 2024-03-14T09:47:14.010 INFO:journalctl@ceph.mon.c.smithi152.stdout: block_restart_interval: 16 2024-03-14T09:47:14.010 INFO:journalctl@ceph.mon.c.smithi152.stdout: index_block_restart_interval: 1 2024-03-14T09:47:14.010 INFO:journalctl@ceph.mon.c.smithi152.stdout: metadata_block_size: 4096 2024-03-14T09:47:14.010 INFO:journalctl@ceph.mon.c.smithi152.stdout: partition_filters: 0 2024-03-14T09:47:14.010 INFO:journalctl@ceph.mon.c.smithi152.stdout: use_delta_encoding: 1 2024-03-14T09:47:14.010 INFO:journalctl@ceph.mon.c.smithi152.stdout: filter_policy: bloomfilter 2024-03-14T09:47:14.010 INFO:journalctl@ceph.mon.c.smithi152.stdout: whole_key_filtering: 1 2024-03-14T09:47:14.010 INFO:journalctl@ceph.mon.c.smithi152.stdout: verify_compression: 0 2024-03-14T09:47:14.011 INFO:journalctl@ceph.mon.c.smithi152.stdout: read_amp_bytes_per_bit: 0 2024-03-14T09:47:14.011 INFO:journalctl@ceph.mon.c.smithi152.stdout: format_version: 5 2024-03-14T09:47:14.011 INFO:journalctl@ceph.mon.c.smithi152.stdout: enable_index_compression: 1 2024-03-14T09:47:14.011 INFO:journalctl@ceph.mon.c.smithi152.stdout: block_align: 0 2024-03-14T09:47:14.011 INFO:journalctl@ceph.mon.c.smithi152.stdout: max_auto_readahead_size: 262144 2024-03-14T09:47:14.011 INFO:journalctl@ceph.mon.c.smithi152.stdout: prepopulate_block_cache: 0 2024-03-14T09:47:14.011 INFO:journalctl@ceph.mon.c.smithi152.stdout: initial_auto_readahead_size: 8192 2024-03-14T09:47:14.011 INFO:journalctl@ceph.mon.c.smithi152.stdout: num_file_reads_for_auto_readahead: 2 2024-03-14T09:47:14.011 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.write_buffer_size: 33554432 2024-03-14T09:47:14.011 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_write_buffer_number: 2 2024-03-14T09:47:14.011 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compression: NoCompression 2024-03-14T09:47:14.011 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.bottommost_compression: Disabled 2024-03-14T09:47:14.011 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.prefix_extractor: nullptr 2024-03-14T09:47:14.011 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-14T09:47:14.012 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.num_levels: 7 2024-03-14T09:47:14.012 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-14T09:47:14.012 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-14T09:47:14.012 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-14T09:47:14.012 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-14T09:47:14.012 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-14T09:47:14.012 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-14T09:47:14.012 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-14T09:47:14.012 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-14T09:47:14.013 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-14T09:47:14.013 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-14T09:47:14.013 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-14T09:47:14.013 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-14T09:47:14.013 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-14T09:47:14.014 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compression_opts.level: 32767 2024-03-14T09:47:14.014 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compression_opts.strategy: 0 2024-03-14T09:47:14.014 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-14T09:47:14.014 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-14T09:47:14.014 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-14T09:47:14.014 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-14T09:47:14.014 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compression_opts.enabled: false 2024-03-14T09:47:14.014 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-14T09:47:14.014 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-14T09:47:14.014 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-14T09:47:14.014 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-14T09:47:14.014 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.target_file_size_base: 67108864 2024-03-14T09:47:14.015 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-14T09:47:14.015 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-14T09:47:14.015 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-14T09:47:14.015 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-14T09:47:14.015 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-14T09:47:14.015 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-14T09:47:14.015 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-14T09:47:14.015 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-14T09:47:14.015 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-14T09:47:14.015 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-14T09:47:14.015 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-14T09:47:14.015 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-14T09:47:14.016 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-14T09:47:14.016 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-14T09:47:14.016 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.arena_block_size: 1048576 2024-03-14T09:47:14.016 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-14T09:47:14.016 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-14T09:47:14.016 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.disable_auto_compactions: 0 2024-03-14T09:47:14.016 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-14T09:47:14.016 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-14T09:47:14.016 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-14T09:47:14.016 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-14T09:47:14.016 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-14T09:47:14.016 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-14T09:47:14.016 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-14T09:47:14.017 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-14T09:47:14.017 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-14T09:47:14.017 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-14T09:47:14.017 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.table_properties_collectors: 2024-03-14T09:47:14.017 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.inplace_update_support: 0 2024-03-14T09:47:14.017 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-14T09:47:14.017 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-14T09:47:14.017 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-14T09:47:14.017 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-14T09:47:14.017 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.bloom_locality: 0 2024-03-14T09:47:14.017 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.max_successive_merges: 0 2024-03-14T09:47:14.018 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-14T09:47:14.018 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.paranoid_file_checks: 0 2024-03-14T09:47:14.018 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.force_consistency_checks: 1 2024-03-14T09:47:14.018 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.report_bg_io_stats: 0 2024-03-14T09:47:14.018 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.ttl: 2592000 2024-03-14T09:47:14.018 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-14T09:47:14.018 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-14T09:47:14.018 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-14T09:47:14.018 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.enable_blob_files: false 2024-03-14T09:47:14.018 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.min_blob_size: 0 2024-03-14T09:47:14.018 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.blob_file_size: 268435456 2024-03-14T09:47:14.018 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-14T09:47:14.019 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-14T09:47:14.019 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-14T09:47:14.019 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-14T09:47:14.019 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-14T09:47:14.019 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.blob_file_starting_level: 0 2024-03-14T09:47:14.019 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-14T09:47:14.019 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-14T09:47:14.019 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-14T09:47:14.019 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6857b8fc-cfdd-4162-b98f-cdfe985a3a64 2024-03-14T09:47:14.019 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710409633736597, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-14T09:47:14.019 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-14T09:47:14.019 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710409633737004, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710409633, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6857b8fc-cfdd-4162-b98f-cdfe985a3a64", "db_session_id": "KM237S4H3QOX69N6P7DO", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-14T09:47:14.019 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710409633737071, "job": 1, "event": "recovery_finished"} 2024-03-14T09:47:14.020 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-14T09:47:14.020 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-14T09:47:14.020 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fa1e83c000 2024-03-14T09:47:14.020 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: DB pointer 0x55fa1e82a000 2024-03-14T09:47:14.020 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-14T09:47:14.020 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-14T09:47:14.020 INFO:journalctl@ceph.mon.c.smithi152.stdout: ** DB Stats ** 2024-03-14T09:47:14.020 INFO:journalctl@ceph.mon.c.smithi152.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-14T09:47:14.020 INFO:journalctl@ceph.mon.c.smithi152.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-14T09:47:14.020 INFO:journalctl@ceph.mon.c.smithi152.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-14T09:47:14.020 INFO:journalctl@ceph.mon.c.smithi152.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-14T09:47:14.020 INFO:journalctl@ceph.mon.c.smithi152.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-14T09:47:14.020 INFO:journalctl@ceph.mon.c.smithi152.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-14T09:47:14.021 INFO:journalctl@ceph.mon.c.smithi152.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-14T09:47:14.021 INFO:journalctl@ceph.mon.c.smithi152.stdout: 2024-03-14T09:47:14.021 INFO:journalctl@ceph.mon.c.smithi152.stdout: ** Compaction Stats [default] ** 2024-03-14T09:47:14.021 INFO:journalctl@ceph.mon.c.smithi152.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-14T09:47:14.021 INFO:journalctl@ceph.mon.c.smithi152.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-14T09:47:14.021 INFO:journalctl@ceph.mon.c.smithi152.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-14T09:47:14.021 INFO:journalctl@ceph.mon.c.smithi152.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-14T09:47:14.021 INFO:journalctl@ceph.mon.c.smithi152.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-14T09:47:14.021 INFO:journalctl@ceph.mon.c.smithi152.stdout: 2024-03-14T09:47:14.021 INFO:journalctl@ceph.mon.c.smithi152.stdout: ** Compaction Stats [default] ** 2024-03-14T09:47:14.021 INFO:journalctl@ceph.mon.c.smithi152.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-14T09:47:14.021 INFO:journalctl@ceph.mon.c.smithi152.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-14T09:47:14.021 INFO:journalctl@ceph.mon.c.smithi152.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-14T09:47:14.022 INFO:journalctl@ceph.mon.c.smithi152.stdout: 2024-03-14T09:47:14.022 INFO:journalctl@ceph.mon.c.smithi152.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-14T09:47:14.022 INFO:journalctl@ceph.mon.c.smithi152.stdout: 2024-03-14T09:47:14.022 INFO:journalctl@ceph.mon.c.smithi152.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-14T09:47:14.022 INFO:journalctl@ceph.mon.c.smithi152.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-14T09:47:14.022 INFO:journalctl@ceph.mon.c.smithi152.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-14T09:47:14.022 INFO:journalctl@ceph.mon.c.smithi152.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-14T09:47:14.022 INFO:journalctl@ceph.mon.c.smithi152.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-14T09:47:14.022 INFO:journalctl@ceph.mon.c.smithi152.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-14T09:47:14.022 INFO:journalctl@ceph.mon.c.smithi152.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-14T09:47:14.022 INFO:journalctl@ceph.mon.c.smithi152.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-14T09:47:14.022 INFO:journalctl@ceph.mon.c.smithi152.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-14T09:47:14.022 INFO:journalctl@ceph.mon.c.smithi152.stdout: Block cache BinnedLRUCache@0x55fa1e79f350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-14T09:47:14.023 INFO:journalctl@ceph.mon.c.smithi152.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-14T09:47:14.023 INFO:journalctl@ceph.mon.c.smithi152.stdout: 2024-03-14T09:47:14.023 INFO:journalctl@ceph.mon.c.smithi152.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-14T09:47:14.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-14T09:47:14.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: using public_addr v2:172.21.15.152:0/0 -> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2024-03-14T09:47:14.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: starting mon.c rank -1 at public addrs [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] at bind addrs [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:47:14.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: mon.c@-1(???) e0 preinit fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:47:14.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 bash[48241]: a31b518db22e20e663fd29b24bcff12c89ed3722f1870745a96a8a3fe183820d 2024-03-14T09:47:14.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 systemd[1]: Started Ceph mon.c for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:47:14.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: mon.c@-1(synchronizing).mds e1 new map 2024-03-14T09:47:14.023 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-14T09:47:14.023 INFO:journalctl@ceph.mon.c.smithi152.stdout: e1 2024-03-14T09:47:14.023 INFO:journalctl@ceph.mon.c.smithi152.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T09:47:14.023 INFO:journalctl@ceph.mon.c.smithi152.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T09:47:14.024 INFO:journalctl@ceph.mon.c.smithi152.stdout: legacy client fscid: -1 2024-03-14T09:47:14.024 INFO:journalctl@ceph.mon.c.smithi152.stdout: 2024-03-14T09:47:14.024 INFO:journalctl@ceph.mon.c.smithi152.stdout: No filesystems configured 2024-03-14T09:47:14.024 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-14T09:47:14.024 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-14T09:47:14.024 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-14T09:47:14.024 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-14T09:47:14.024 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-14T09:47:14.024 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-14T09:47:14.024 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-14T09:47:14.024 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-14T09:47:14.024 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-14T09:47:14.024 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-14T09:47:14.024 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: Added host smithi152 2024-03-14T09:47:14.025 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:14.025 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:14.025 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:14.025 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:47:14.025 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:14.025 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:14.025 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:14.025 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:14.025 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:14.025 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:47:14.025 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:14.025 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:14.026 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-14T09:47:14.026 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2459389783' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-14T09:47:14.026 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:14.026 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:14.026 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:14.026 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: Updating smithi152:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:47:14.026 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:47:14.026 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: Updating smithi152:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T09:47:14.026 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2459389783' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-14T09:47:14.026 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: osdmap e4: 0 total, 0 up, 0 in 2024-03-14T09:47:14.026 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:14.027 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:14.027 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:14.027 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:14.027 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:14.027 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:14.027 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T09:47:14.027 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:14.027 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi132:172.21.15.132=b;smithi152:172.21.15.152=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:47:14.027 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi132:172.21.15.132=b;smithi152:172.21.15.152=c;count:3 2024-03-14T09:47:14.027 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: Deploying daemon mon.c on smithi152 2024-03-14T09:47:14.027 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:14.027 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:14.027 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: from='client.? 172.21.15.152:0/3388215025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T09:47:14.028 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:13 smithi152 ceph-mon[48279]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-14T09:47:14.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:13 smithi078 ceph-mon[38143]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:14.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:13 smithi078 ceph-mon[38143]: from='client.? 172.21.15.152:0/3388215025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T09:47:14.533 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-14T09:47:14.533 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T09:47:14.759 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.c/config 2024-03-14T09:47:18.832 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.831+0000 7f100530e700 1 -- 172.21.15.152:0/607544800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe80057d0 msgr2=0x7f0fe8007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:18.832 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.831+0000 7f100530e700 1 --2- 172.21.15.152:0/607544800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe80057d0 0x7f0fe8007c60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f10000ff5f0 tx=0x7f0ff000feb0 comp rx=0 tx=0).stop 2024-03-14T09:47:18.832 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.831+0000 7f100530e700 1 -- 172.21.15.152:0/607544800 shutdown_connections 2024-03-14T09:47:18.832 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.831+0000 7f100530e700 1 --2- 172.21.15.152:0/607544800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe80057d0 0x7f0fe8007c60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:18.832 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.831+0000 7f100530e700 1 -- 172.21.15.152:0/607544800 >> 172.21.15.152:0/607544800 conn(0x7f10000fb250 msgr2=0x7f10000fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:18.832 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.831+0000 7f100530e700 1 -- 172.21.15.152:0/607544800 shutdown_connections 2024-03-14T09:47:18.832 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.831+0000 7f100530e700 1 -- 172.21.15.152:0/607544800 wait complete. 2024-03-14T09:47:18.833 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.832+0000 7f100530e700 1 Processor -- start 2024-03-14T09:47:18.833 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.832+0000 7f100530e700 1 -- start start 2024-03-14T09:47:18.833 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.832+0000 7f100530e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe80057d0 0x7f1000198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:18.833 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.832+0000 7f100530e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1000198c90 0x7f100019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:18.833 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.832+0000 7f100530e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10001991d0 con 0x7f0fe80057d0 2024-03-14T09:47:18.833 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.832+0000 7f100530e700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1000199340 con 0x7f1000198c90 2024-03-14T09:47:18.834 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.833+0000 7f0ffe7fc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1000198c90 0x7f100019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:18.834 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.833+0000 7f0ffe7fc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1000198c90 0x7f100019cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:56834/0 (socket says 172.21.15.152:56834) 2024-03-14T09:47:18.835 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.833+0000 7f0ffeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe80057d0 0x7f1000198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:18.835 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.833+0000 7f0ffe7fc700 1 -- 172.21.15.152:0/1136121575 learned_addr learned my addr 172.21.15.152:0/1136121575 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-14T09:47:18.835 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.833+0000 7f0ffe7fc700 1 -- 172.21.15.152:0/1136121575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1000198c90 msgr2=0x7f100019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-14T09:47:18.835 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.833+0000 7f0ffe7fc700 1 -- 172.21.15.152:0/1136121575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1000198c90 msgr2=0x7f100019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-14T09:47:18.835 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.833+0000 7f0ffe7fc700 1 --2- 172.21.15.152:0/1136121575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1000198c90 0x7f100019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-14T09:47:18.835 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.833+0000 7f0ffe7fc700 1 --2- 172.21.15.152:0/1136121575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1000198c90 0x7f100019cf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T09:47:18.835 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.833+0000 7f0ffeffd700 1 -- 172.21.15.152:0/1136121575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1000198c90 msgr2=0x7f100019cf80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T09:47:18.835 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.833+0000 7f0ffeffd700 1 --2- 172.21.15.152:0/1136121575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1000198c90 0x7f100019cf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:18.835 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.833+0000 7f0ffeffd700 1 -- 172.21.15.152:0/1136121575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff000f900 con 0x7f0fe80057d0 2024-03-14T09:47:18.835 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.833+0000 7f0ffeffd700 1 --2- 172.21.15.152:0/1136121575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe80057d0 0x7f1000198750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0010010 tx=0x7f0ff0004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:18.835 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.834+0000 7f0fe7fff700 1 -- 172.21.15.152:0/1136121575 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f0ff0013b00 con 0x7f0fe80057d0 2024-03-14T09:47:18.835 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.834+0000 7f0fe7fff700 1 -- 172.21.15.152:0/1136121575 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ff0004b70 con 0x7f0fe80057d0 2024-03-14T09:47:18.835 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.834+0000 7f100530e700 1 -- 172.21.15.152:0/1136121575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100019d4c0 con 0x7f0fe80057d0 2024-03-14T09:47:18.835 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.834+0000 7f0fe7fff700 1 -- 172.21.15.152:0/1136121575 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f0ff0003b60 con 0x7f0fe80057d0 2024-03-14T09:47:18.835 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.834+0000 7f100530e700 1 -- 172.21.15.152:0/1136121575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f100019da10 con 0x7f0fe80057d0 2024-03-14T09:47:18.836 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.835+0000 7f0fe7fff700 1 -- 172.21.15.152:0/1136121575 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45430+0+0 (secure 0 0 0) 0x7f0ff0011470 con 0x7f0fe80057d0 2024-03-14T09:47:18.836 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.835+0000 7f100530e700 1 -- 172.21.15.152:0/1136121575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10001924d0 con 0x7f0fe80057d0 2024-03-14T09:47:18.836 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.835+0000 7f0fe7fff700 1 --2- 172.21.15.152:0/1136121575 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0fec0383b0 0x7f0fec03a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:18.836 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.835+0000 7f0fe7fff700 1 -- 172.21.15.152:0/1136121575 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0ff004d870 con 0x7f0fe80057d0 2024-03-14T09:47:18.836 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.835+0000 7f0ffe7fc700 1 --2- 172.21.15.152:0/1136121575 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0fec0383b0 0x7f0fec03a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:18.837 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.836+0000 7f0ffe7fc700 1 --2- 172.21.15.152:0/1136121575 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0fec0383b0 0x7f0fec03a870 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009180 tx=0x7f0ff4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:18.842 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:18.841+0000 7f0fe7fff700 1 -- 172.21.15.152:0/1136121575 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0ff00107e0 con 0x7f0fe80057d0 2024-03-14T09:47:19.112 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:19.111+0000 7f100530e700 1 -- 172.21.15.152:0/1136121575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f100019dc80 con 0x7f0fe80057d0 2024-03-14T09:47:19.113 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:19.112+0000 7f0fe7fff700 1 -- 172.21.15.152:0/1136121575 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7f0ff00103d0 con 0x7f0fe80057d0 2024-03-14T09:47:19.113 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-14T09:47:19.113 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":2,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","modified":"2024-03-14T09:47:13.766662Z","created":"2024-03-14T09:45:40.908003Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:3300","nonce":0},{"type":"v1","addr":"172.21.15.152:6789","nonce":0}]},"addr":"172.21.15.152:6789/0","public_addr":"172.21.15.152:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-14T09:47:19.114 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:19.114+0000 7f100530e700 1 -- 172.21.15.152:0/1136121575 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0fec0383b0 msgr2=0x7f0fec03a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:19.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:19.114+0000 7f100530e700 1 --2- 172.21.15.152:0/1136121575 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0fec0383b0 0x7f0fec03a870 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009180 tx=0x7f0ff4008040 comp rx=0 tx=0).stop 2024-03-14T09:47:19.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:19.114+0000 7f100530e700 1 -- 172.21.15.152:0/1136121575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe80057d0 msgr2=0x7f1000198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:19.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:19.114+0000 7f100530e700 1 --2- 172.21.15.152:0/1136121575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe80057d0 0x7f1000198750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0010010 tx=0x7f0ff0004f80 comp rx=0 tx=0).stop 2024-03-14T09:47:19.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:19.114+0000 7f100530e700 1 -- 172.21.15.152:0/1136121575 shutdown_connections 2024-03-14T09:47:19.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:19.114+0000 7f100530e700 1 --2- 172.21.15.152:0/1136121575 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0fec0383b0 0x7f0fec03a870 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:19.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:19.114+0000 7f100530e700 1 --2- 172.21.15.152:0/1136121575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe80057d0 0x7f1000198750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:19.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:19.114+0000 7f100530e700 1 --2- 172.21.15.152:0/1136121575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1000198c90 0x7f100019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:19.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:19.114+0000 7f100530e700 1 -- 172.21.15.152:0/1136121575 >> 172.21.15.152:0/1136121575 conn(0x7f10000fb250 msgr2=0x7f10000fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:19.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:19.114+0000 7f100530e700 1 -- 172.21.15.152:0/1136121575 shutdown_connections 2024-03-14T09:47:19.115 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:19.114+0000 7f100530e700 1 -- 172.21.15.152:0/1136121575 wait complete. 2024-03-14T09:47:19.115 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 2 2024-03-14T09:47:19.134 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:47:19.134 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:19.135 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: mon.a calling monitor election 2024-03-14T09:47:19.135 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:19.135 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:19.135 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:19.135 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: mon.c calling monitor election 2024-03-14T09:47:19.135 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:19.135 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:19.135 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:19.135 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:19.135 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-14T09:47:19.135 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: monmap e2: 2 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T09:47:19.135 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: fsmap 2024-03-14T09:47:19.135 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: osdmap e4: 0 total, 0 up, 0 in 2024-03-14T09:47:19.135 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: mgrmap e13: x(active, since 37s) 2024-03-14T09:47:19.136 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: overall HEALTH_OK 2024-03-14T09:47:19.136 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:19.136 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:19.136 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:19.136 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T09:47:19.136 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:19.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:47:19.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:19.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: mon.a calling monitor election 2024-03-14T09:47:19.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:19.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:19.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:19.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: mon.c calling monitor election 2024-03-14T09:47:19.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:19.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:19.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:19.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:19.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-14T09:47:19.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: monmap e2: 2 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T09:47:19.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: fsmap 2024-03-14T09:47:19.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: osdmap e4: 0 total, 0 up, 0 in 2024-03-14T09:47:19.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: mgrmap e13: x(active, since 37s) 2024-03-14T09:47:19.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: overall HEALTH_OK 2024-03-14T09:47:19.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:19.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:19.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:19.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T09:47:19.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:19.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:19 smithi152 ceph-mon[48279]: Deploying daemon mon.b on smithi132 2024-03-14T09:47:19.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:19 smithi152 ceph-mon[48279]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:19.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:19 smithi152 ceph-mon[48279]: from='client.? 172.21.15.152:0/1136121575' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T09:47:19.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:19 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:20.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:19 smithi078 ceph-mon[38143]: Deploying daemon mon.b on smithi132 2024-03-14T09:47:20.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:19 smithi078 ceph-mon[38143]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:20.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:19 smithi078 ceph-mon[38143]: from='client.? 172.21.15.152:0/1136121575' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T09:47:20.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:19 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:20.718 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-14T09:47:20.718 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T09:47:20.946 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.c/config 2024-03-14T09:47:21.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 systemd[1]: Started Ceph mon.b for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:47:21.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.manual_wal_flush: 0 2024-03-14T09:47:21.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.wal_compression: 0 2024-03-14T09:47:21.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.atomic_flush: 0 2024-03-14T09:47:21.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-14T09:47:21.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-14T09:47:21.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-14T09:47:21.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.log_readahead_size: 0 2024-03-14T09:47:21.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-14T09:47:21.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.best_efforts_recovery: 0 2024-03-14T09:47:21.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-14T09:47:21.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-14T09:47:21.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.allow_data_in_errors: 0 2024-03-14T09:47:21.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.db_host_id: __hostname__ 2024-03-14T09:47:21.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-14T09:47:21.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_background_jobs: 2 2024-03-14T09:47:21.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_background_compactions: -1 2024-03-14T09:47:21.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_subcompactions: 1 2024-03-14T09:47:21.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-14T09:47:21.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-14T09:47:21.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-14T09:47:21.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_total_wal_size: 0 2024-03-14T09:47:21.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-14T09:47:21.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-14T09:47:21.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-14T09:47:21.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-14T09:47:21.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_open_files: -1 2024-03-14T09:47:21.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.bytes_per_sync: 0 2024-03-14T09:47:21.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-14T09:47:21.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-14T09:47:21.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compaction_readahead_size: 0 2024-03-14T09:47:21.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_background_flushes: -1 2024-03-14T09:47:21.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Compression algorithms supported: 2024-03-14T09:47:21.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-14T09:47:21.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: kZSTD supported: 0 2024-03-14T09:47:21.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: kXpressCompression supported: 0 2024-03-14T09:47:21.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: kLZ4HCCompression supported: 1 2024-03-14T09:47:21.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: kZlibCompression supported: 1 2024-03-14T09:47:21.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: kSnappyCompression supported: 1 2024-03-14T09:47:21.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: kLZ4Compression supported: 1 2024-03-14T09:47:21.774 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: kBZip2Compression supported: 0 2024-03-14T09:47:21.774 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-14T09:47:21.774 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-14T09:47:21.774 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-14T09:47:21.774 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-14T09:47:21.774 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-14T09:47:21.774 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.merge_operator: 2024-03-14T09:47:21.774 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compaction_filter: None 2024-03-14T09:47:21.774 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compaction_filter_factory: None 2024-03-14T09:47:21.774 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.sst_partitioner_factory: None 2024-03-14T09:47:21.774 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-14T09:47:21.774 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-14T09:47:21.774 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5640345535e0) 2024-03-14T09:47:21.774 INFO:journalctl@ceph.mon.b.smithi132.stdout: cache_index_and_filter_blocks: 1 2024-03-14T09:47:21.775 INFO:journalctl@ceph.mon.b.smithi132.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-14T09:47:21.775 INFO:journalctl@ceph.mon.b.smithi132.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-14T09:47:21.775 INFO:journalctl@ceph.mon.b.smithi132.stdout: pin_top_level_index_and_filter: 1 2024-03-14T09:47:21.775 INFO:journalctl@ceph.mon.b.smithi132.stdout: index_type: 0 2024-03-14T09:47:21.775 INFO:journalctl@ceph.mon.b.smithi132.stdout: data_block_index_type: 0 2024-03-14T09:47:21.775 INFO:journalctl@ceph.mon.b.smithi132.stdout: index_shortening: 1 2024-03-14T09:47:21.775 INFO:journalctl@ceph.mon.b.smithi132.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-14T09:47:21.775 INFO:journalctl@ceph.mon.b.smithi132.stdout: checksum: 4 2024-03-14T09:47:21.775 INFO:journalctl@ceph.mon.b.smithi132.stdout: no_block_cache: 0 2024-03-14T09:47:21.775 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_cache: 0x56403386d1f0 2024-03-14T09:47:21.775 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_cache_name: BinnedLRUCache 2024-03-14T09:47:21.775 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_cache_options: 2024-03-14T09:47:21.775 INFO:journalctl@ceph.mon.b.smithi132.stdout: capacity : 536870912 2024-03-14T09:47:21.775 INFO:journalctl@ceph.mon.b.smithi132.stdout: num_shard_bits : 4 2024-03-14T09:47:21.775 INFO:journalctl@ceph.mon.b.smithi132.stdout: strict_capacity_limit : 0 2024-03-14T09:47:21.776 INFO:journalctl@ceph.mon.b.smithi132.stdout: high_pri_pool_ratio: 0.000 2024-03-14T09:47:21.776 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_cache_compressed: (nil) 2024-03-14T09:47:21.776 INFO:journalctl@ceph.mon.b.smithi132.stdout: persistent_cache: (nil) 2024-03-14T09:47:21.776 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_size: 4096 2024-03-14T09:47:21.776 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_size_deviation: 10 2024-03-14T09:47:21.776 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_restart_interval: 16 2024-03-14T09:47:21.776 INFO:journalctl@ceph.mon.b.smithi132.stdout: index_block_restart_interval: 1 2024-03-14T09:47:21.776 INFO:journalctl@ceph.mon.b.smithi132.stdout: metadata_block_size: 4096 2024-03-14T09:47:21.776 INFO:journalctl@ceph.mon.b.smithi132.stdout: partition_filters: 0 2024-03-14T09:47:21.776 INFO:journalctl@ceph.mon.b.smithi132.stdout: use_delta_encoding: 1 2024-03-14T09:47:21.776 INFO:journalctl@ceph.mon.b.smithi132.stdout: filter_policy: bloomfilter 2024-03-14T09:47:21.776 INFO:journalctl@ceph.mon.b.smithi132.stdout: whole_key_filtering: 1 2024-03-14T09:47:21.776 INFO:journalctl@ceph.mon.b.smithi132.stdout: verify_compression: 0 2024-03-14T09:47:21.776 INFO:journalctl@ceph.mon.b.smithi132.stdout: read_amp_bytes_per_bit: 0 2024-03-14T09:47:21.776 INFO:journalctl@ceph.mon.b.smithi132.stdout: format_version: 5 2024-03-14T09:47:21.777 INFO:journalctl@ceph.mon.b.smithi132.stdout: enable_index_compression: 1 2024-03-14T09:47:21.777 INFO:journalctl@ceph.mon.b.smithi132.stdout: block_align: 0 2024-03-14T09:47:21.777 INFO:journalctl@ceph.mon.b.smithi132.stdout: max_auto_readahead_size: 262144 2024-03-14T09:47:21.777 INFO:journalctl@ceph.mon.b.smithi132.stdout: prepopulate_block_cache: 0 2024-03-14T09:47:21.777 INFO:journalctl@ceph.mon.b.smithi132.stdout: initial_auto_readahead_size: 8192 2024-03-14T09:47:21.777 INFO:journalctl@ceph.mon.b.smithi132.stdout: num_file_reads_for_auto_readahead: 2 2024-03-14T09:47:21.777 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.write_buffer_size: 33554432 2024-03-14T09:47:21.778 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_write_buffer_number: 2 2024-03-14T09:47:21.778 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compression: NoCompression 2024-03-14T09:47:21.778 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.bottommost_compression: Disabled 2024-03-14T09:47:21.778 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.prefix_extractor: nullptr 2024-03-14T09:47:21.778 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-14T09:47:21.779 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.num_levels: 7 2024-03-14T09:47:21.779 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-14T09:47:21.779 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-14T09:47:21.779 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-14T09:47:21.779 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-14T09:47:21.779 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-14T09:47:21.779 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-14T09:47:21.779 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-14T09:47:21.779 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-14T09:47:21.779 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-14T09:47:21.779 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-14T09:47:21.779 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-14T09:47:21.779 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-14T09:47:21.779 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-14T09:47:21.779 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compression_opts.level: 32767 2024-03-14T09:47:21.780 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compression_opts.strategy: 0 2024-03-14T09:47:21.780 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-14T09:47:21.780 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-14T09:47:21.780 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-14T09:47:21.780 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-14T09:47:21.780 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compression_opts.enabled: false 2024-03-14T09:47:21.780 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-14T09:47:21.780 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-14T09:47:21.780 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-14T09:47:21.780 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-14T09:47:21.780 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.target_file_size_base: 67108864 2024-03-14T09:47:21.780 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-14T09:47:21.780 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-14T09:47:21.780 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-14T09:47:21.780 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-14T09:47:21.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-14T09:47:21.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-14T09:47:21.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-14T09:47:21.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-14T09:47:21.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-14T09:47:21.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-14T09:47:21.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-14T09:47:21.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-14T09:47:21.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-14T09:47:21.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-14T09:47:21.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.arena_block_size: 1048576 2024-03-14T09:47:21.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-14T09:47:21.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-14T09:47:21.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.disable_auto_compactions: 0 2024-03-14T09:47:21.781 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-14T09:47:21.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-14T09:47:21.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-14T09:47:21.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-14T09:47:21.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-14T09:47:21.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-14T09:47:21.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-14T09:47:21.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-14T09:47:21.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-14T09:47:21.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-14T09:47:21.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.table_properties_collectors: 2024-03-14T09:47:21.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.inplace_update_support: 0 2024-03-14T09:47:21.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-14T09:47:21.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-14T09:47:21.782 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-14T09:47:21.783 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-14T09:47:21.783 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.bloom_locality: 0 2024-03-14T09:47:21.783 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.max_successive_merges: 0 2024-03-14T09:47:21.783 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-14T09:47:21.783 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.paranoid_file_checks: 0 2024-03-14T09:47:21.783 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.force_consistency_checks: 1 2024-03-14T09:47:21.783 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.report_bg_io_stats: 0 2024-03-14T09:47:21.783 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.ttl: 2592000 2024-03-14T09:47:21.783 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-14T09:47:21.783 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-14T09:47:21.783 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-14T09:47:21.783 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.enable_blob_files: false 2024-03-14T09:47:21.783 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.min_blob_size: 0 2024-03-14T09:47:21.783 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.blob_file_size: 268435456 2024-03-14T09:47:21.783 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-14T09:47:21.783 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-14T09:47:21.784 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-14T09:47:21.784 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-14T09:47:21.784 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-14T09:47:21.784 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.blob_file_starting_level: 0 2024-03-14T09:47:21.784 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-14T09:47:21.784 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-14T09:47:21.784 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-14T09:47:21.784 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8a0789b6-f1d3-478c-8e9e-6923f8dd2c0a 2024-03-14T09:47:21.784 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710409641367644, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-14T09:47:21.784 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-14T09:47:21.784 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710409641368070, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710409641, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8a0789b6-f1d3-478c-8e9e-6923f8dd2c0a", "db_session_id": "9AJYRJTU6D9AUW4THB4A", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-14T09:47:21.784 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710409641368140, "job": 1, "event": "recovery_finished"} 2024-03-14T09:47:21.784 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-14T09:47:21.784 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-14T09:47:21.784 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56403390a000 2024-03-14T09:47:21.785 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: DB pointer 0x5640338f8000 2024-03-14T09:47:21.785 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-14T09:47:21.785 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-14T09:47:21.785 INFO:journalctl@ceph.mon.b.smithi132.stdout: ** DB Stats ** 2024-03-14T09:47:21.785 INFO:journalctl@ceph.mon.b.smithi132.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-14T09:47:21.785 INFO:journalctl@ceph.mon.b.smithi132.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-14T09:47:21.785 INFO:journalctl@ceph.mon.b.smithi132.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-14T09:47:21.785 INFO:journalctl@ceph.mon.b.smithi132.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-14T09:47:21.785 INFO:journalctl@ceph.mon.b.smithi132.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-14T09:47:21.785 INFO:journalctl@ceph.mon.b.smithi132.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-14T09:47:21.785 INFO:journalctl@ceph.mon.b.smithi132.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-14T09:47:21.785 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-03-14T09:47:21.785 INFO:journalctl@ceph.mon.b.smithi132.stdout: ** Compaction Stats [default] ** 2024-03-14T09:47:21.785 INFO:journalctl@ceph.mon.b.smithi132.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-14T09:47:21.785 INFO:journalctl@ceph.mon.b.smithi132.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-14T09:47:21.786 INFO:journalctl@ceph.mon.b.smithi132.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-14T09:47:21.786 INFO:journalctl@ceph.mon.b.smithi132.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-14T09:47:21.786 INFO:journalctl@ceph.mon.b.smithi132.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-14T09:47:21.786 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-03-14T09:47:21.786 INFO:journalctl@ceph.mon.b.smithi132.stdout: ** Compaction Stats [default] ** 2024-03-14T09:47:21.786 INFO:journalctl@ceph.mon.b.smithi132.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-14T09:47:21.786 INFO:journalctl@ceph.mon.b.smithi132.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-14T09:47:21.786 INFO:journalctl@ceph.mon.b.smithi132.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-14T09:47:21.786 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-03-14T09:47:21.786 INFO:journalctl@ceph.mon.b.smithi132.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-14T09:47:21.786 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-03-14T09:47:21.786 INFO:journalctl@ceph.mon.b.smithi132.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-14T09:47:21.786 INFO:journalctl@ceph.mon.b.smithi132.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-14T09:47:21.786 INFO:journalctl@ceph.mon.b.smithi132.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-14T09:47:21.787 INFO:journalctl@ceph.mon.b.smithi132.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-14T09:47:21.787 INFO:journalctl@ceph.mon.b.smithi132.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-14T09:47:21.787 INFO:journalctl@ceph.mon.b.smithi132.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-14T09:47:21.787 INFO:journalctl@ceph.mon.b.smithi132.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-14T09:47:21.787 INFO:journalctl@ceph.mon.b.smithi132.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-14T09:47:21.787 INFO:journalctl@ceph.mon.b.smithi132.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-14T09:47:21.787 INFO:journalctl@ceph.mon.b.smithi132.stdout: Block cache BinnedLRUCache@0x56403386d1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-14T09:47:21.787 INFO:journalctl@ceph.mon.b.smithi132.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-14T09:47:21.787 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-03-14T09:47:21.787 INFO:journalctl@ceph.mon.b.smithi132.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-14T09:47:21.787 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-14T09:47:21.787 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: using public_addr v2:172.21.15.132:0/0 -> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] 2024-03-14T09:47:21.787 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: starting mon.b rank -1 at public addrs [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] at bind addrs [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:47:21.787 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.b@-1(???) e0 preinit fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:47:21.787 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.b@-1(synchronizing).mds e1 new map 2024-03-14T09:47:21.788 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-14T09:47:21.788 INFO:journalctl@ceph.mon.b.smithi132.stdout: e1 2024-03-14T09:47:21.788 INFO:journalctl@ceph.mon.b.smithi132.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T09:47:21.788 INFO:journalctl@ceph.mon.b.smithi132.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T09:47:21.788 INFO:journalctl@ceph.mon.b.smithi132.stdout: legacy client fscid: -1 2024-03-14T09:47:21.788 INFO:journalctl@ceph.mon.b.smithi132.stdout: 2024-03-14T09:47:21.788 INFO:journalctl@ceph.mon.b.smithi132.stdout: No filesystems configured 2024-03-14T09:47:21.788 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-14T09:47:21.788 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-14T09:47:21.788 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-14T09:47:21.788 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-14T09:47:21.788 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-14T09:47:21.788 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-14T09:47:21.788 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-14T09:47:21.788 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-14T09:47:21.788 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-14T09:47:21.789 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-14T09:47:21.789 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: Added host smithi152 2024-03-14T09:47:21.789 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:21.789 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:21.789 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:21.789 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:47:21.789 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:21.789 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:21.789 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:21.789 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:21.789 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:21.789 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi152", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:47:21.789 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:21.789 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:21.789 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-14T09:47:21.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2459389783' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-14T09:47:21.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:21.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:21.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:21.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: Updating smithi152:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:47:21.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:47:21.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: Updating smithi152:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T09:47:21.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2459389783' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-14T09:47:21.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: osdmap e4: 0 total, 0 up, 0 in 2024-03-14T09:47:21.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:21.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:21.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:21.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:21.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:21.790 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T09:47:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi132:172.21.15.132=b;smithi152:172.21.15.152=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:47:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi132:172.21.15.132=b;smithi152:172.21.15.152=c;count:3 2024-03-14T09:47:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: Deploying daemon mon.c on smithi152 2024-03-14T09:47:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='client.? 172.21.15.152:0/3388215025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T09:47:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:47:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.a calling monitor election 2024-03-14T09:47:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:21.791 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.c calling monitor election 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: monmap e2: 2 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: fsmap 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: osdmap e4: 0 total, 0 up, 0 in 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mgrmap e13: x(active, since 37s) 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: overall HEALTH_OK 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: Deploying daemon mon.b on smithi132 2024-03-14T09:47:21.792 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:21.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='client.? 172.21.15.152:0/1136121575' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T09:47:21.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:21.793 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:21 smithi132 ceph-mon[47959]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-14T09:47:26.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:26.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:47:26.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:26.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: mon.a calling monitor election 2024-03-14T09:47:26.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: mon.c calling monitor election 2024-03-14T09:47:26.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:26.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:26.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:26.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:26.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:26.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:26.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:26.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:26.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-14T09:47:26.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T09:47:26.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: fsmap 2024-03-14T09:47:26.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: osdmap e4: 0 total, 0 up, 0 in 2024-03-14T09:47:26.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: mgrmap e13: x(active, since 45s) 2024-03-14T09:47:26.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-14T09:47:26.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-14T09:47:26.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-14T09:47:26.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: mon.b (rank 2) addr [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] is down (out of quorum) 2024-03-14T09:47:26.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:26.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:26.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:26.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:26.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:26.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:47:26.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:26.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: mon.a calling monitor election 2024-03-14T09:47:26.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: mon.c calling monitor election 2024-03-14T09:47:26.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:26.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:26.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:26.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:26.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:26.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:26.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:26.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:26.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-14T09:47:26.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T09:47:26.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: fsmap 2024-03-14T09:47:26.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: osdmap e4: 0 total, 0 up, 0 in 2024-03-14T09:47:26.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: mgrmap e13: x(active, since 45s) 2024-03-14T09:47:26.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-14T09:47:26.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-14T09:47:26.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-14T09:47:26.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: mon.b (rank 2) addr [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] is down (out of quorum) 2024-03-14T09:47:26.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:26.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:26.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:26.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:27.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:27 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:27.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:27 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:27.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:27 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:27.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:27 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:27.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:27 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:27.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:27 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:27.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:27 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:27.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:27 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:27.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:27 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:27.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:27 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:27.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:27 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:27.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:27 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:27.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:27 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:27.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:27 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:27.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:27 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:27.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:27 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:27.971 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.969+0000 7f0c3d2bf700 1 -- 172.21.15.152:0/4182457110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c20006230 msgr2=0x7f0c200086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:27.971 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.969+0000 7f0c3d2bf700 1 --2- 172.21.15.152:0/4182457110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c20006230 0x7f0c200086a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0c38068e30 tx=0x7f0c2800fe90 comp rx=0 tx=0).stop 2024-03-14T09:47:27.971 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.969+0000 7f0c3d2bf700 1 -- 172.21.15.152:0/4182457110 shutdown_connections 2024-03-14T09:47:27.971 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.969+0000 7f0c3d2bf700 1 --2- 172.21.15.152:0/4182457110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c20006230 0x7f0c200086a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:27.971 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.969+0000 7f0c3d2bf700 1 -- 172.21.15.152:0/4182457110 >> 172.21.15.152:0/4182457110 conn(0x7f0c380fb1a0 msgr2=0x7f0c380fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:27.971 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.969+0000 7f0c3d2bf700 1 -- 172.21.15.152:0/4182457110 shutdown_connections 2024-03-14T09:47:27.971 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.969+0000 7f0c3d2bf700 1 -- 172.21.15.152:0/4182457110 wait complete. 2024-03-14T09:47:27.971 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.970+0000 7f0c3d2bf700 1 Processor -- start 2024-03-14T09:47:27.971 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.970+0000 7f0c3d2bf700 1 -- start start 2024-03-14T09:47:27.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.970+0000 7f0c3d2bf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c20006230 0x7f0c380ff650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:27.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.970+0000 7f0c3d2bf700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c380ffb90 0x7f0c38100010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:27.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.970+0000 7f0c3d2bf700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c381a2870 0x7f0c381a4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:27.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.970+0000 7f0c3d2bf700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c381a5260 con 0x7f0c20006230 2024-03-14T09:47:27.974 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.970+0000 7f0c3d2bf700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c381a53d0 con 0x7f0c380ffb90 2024-03-14T09:47:27.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.970+0000 7f0c3d2bf700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c381a5590 con 0x7f0c381a2870 2024-03-14T09:47:27.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c377fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c381a2870 0x7f0c381a4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:27.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c377fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c381a2870 0x7f0c381a4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:60822/0 (socket says 172.21.15.152:60822) 2024-03-14T09:47:27.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c377fe700 1 -- 172.21.15.152:0/265461093 learned_addr learned my addr 172.21.15.152:0/265461093 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-14T09:47:27.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c36ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c20006230 0x7f0c380ff650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:27.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c367fc700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c380ffb90 0x7f0c38100010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:27.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c377fe700 1 -- 172.21.15.152:0/265461093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c381a2870 msgr2=0x7f0c381a4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-14T09:47:27.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c377fe700 1 -- 172.21.15.152:0/265461093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c381a2870 msgr2=0x7f0c381a4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-14T09:47:27.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c377fe700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c381a2870 0x7f0c381a4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-14T09:47:27.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c377fe700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c381a2870 0x7f0c381a4c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T09:47:27.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c367fc700 1 -- 172.21.15.152:0/265461093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c380ffb90 msgr2=0x7f0c38100010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-14T09:47:27.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c367fc700 1 -- 172.21.15.152:0/265461093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c380ffb90 msgr2=0x7f0c38100010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-14T09:47:27.975 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c367fc700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c380ffb90 0x7f0c38100010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-14T09:47:27.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c367fc700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c380ffb90 0x7f0c38100010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T09:47:27.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c36ffd700 1 -- 172.21.15.152:0/265461093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c381a2870 msgr2=0x7f0c381a4c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T09:47:27.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c36ffd700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c381a2870 0x7f0c381a4c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:27.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c36ffd700 1 -- 172.21.15.152:0/265461093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c380ffb90 msgr2=0x7f0c38100010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T09:47:27.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c36ffd700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c380ffb90 0x7f0c38100010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:27.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c36ffd700 1 -- 172.21.15.152:0/265461093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2800f8e0 con 0x7f0c20006230 2024-03-14T09:47:27.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c36ffd700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c20006230 0x7f0c380ff650 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0c28010010 tx=0x7f0c28005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:27.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c1ffff700 1 -- 172.21.15.152:0/265461093 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c28011d60 con 0x7f0c20006230 2024-03-14T09:47:27.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c1ffff700 1 -- 172.21.15.152:0/265461093 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c28011ec0 con 0x7f0c20006230 2024-03-14T09:47:27.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c3d2bf700 1 -- 172.21.15.152:0/265461093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c381a5810 con 0x7f0c20006230 2024-03-14T09:47:27.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c1ffff700 1 -- 172.21.15.152:0/265461093 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c280038e0 con 0x7f0c20006230 2024-03-14T09:47:27.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.971+0000 7f0c3d2bf700 1 -- 172.21.15.152:0/265461093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c381a5cb0 con 0x7f0c20006230 2024-03-14T09:47:27.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.972+0000 7f0c3d2bf700 1 -- 172.21.15.152:0/265461093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c380694b0 con 0x7f0c20006230 2024-03-14T09:47:27.976 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.972+0000 7f0c1ffff700 1 -- 172.21.15.152:0/265461093 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45430+0+0 (secure 0 0 0) 0x7f0c28003a40 con 0x7f0c20006230 2024-03-14T09:47:27.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.972+0000 7f0c1ffff700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0c240389f0 0x7f0c2403aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:27.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.973+0000 7f0c1ffff700 1 -- 172.21.15.152:0/265461093 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0c2804e5e0 con 0x7f0c20006230 2024-03-14T09:47:27.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.973+0000 7f0c367fc700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0c240389f0 0x7f0c2403aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:27.977 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.973+0000 7f0c367fc700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0c240389f0 0x7f0c2403aeb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c009850 tx=0x7f0c2c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:27.978 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:27.978+0000 7f0c1ffff700 1 -- 172.21.15.152:0/265461093 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0c28039760 con 0x7f0c20006230 2024-03-14T09:47:28.250 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:28.249+0000 7f0c3d2bf700 1 -- 172.21.15.152:0/265461093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c381088d0 con 0x7f0c20006230 2024-03-14T09:47:28.250 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:28.250+0000 7f0c1ffff700 1 -- 172.21.15.152:0/265461093 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1274 (secure 0 0 0) 0x7f0c28025380 con 0x7f0c20006230 2024-03-14T09:47:28.251 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-03-14T09:47:28.251 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":3,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","modified":"2024-03-14T09:47:21.395664Z","created":"2024-03-14T09:45:40.908003Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:3300","nonce":0},{"type":"v1","addr":"172.21.15.152:6789","nonce":0}]},"addr":"172.21.15.152:6789/0","public_addr":"172.21.15.152:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-14T09:47:28.253 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:28.252+0000 7f0c3d2bf700 1 -- 172.21.15.152:0/265461093 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0c240389f0 msgr2=0x7f0c2403aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:28.253 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:28.252+0000 7f0c3d2bf700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0c240389f0 0x7f0c2403aeb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c009850 tx=0x7f0c2c008040 comp rx=0 tx=0).stop 2024-03-14T09:47:28.253 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:28.252+0000 7f0c3d2bf700 1 -- 172.21.15.152:0/265461093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c20006230 msgr2=0x7f0c380ff650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:28.253 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:28.252+0000 7f0c3d2bf700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c20006230 0x7f0c380ff650 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0c28010010 tx=0x7f0c28005e50 comp rx=0 tx=0).stop 2024-03-14T09:47:28.253 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:28.252+0000 7f0c3d2bf700 1 -- 172.21.15.152:0/265461093 shutdown_connections 2024-03-14T09:47:28.253 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:28.252+0000 7f0c3d2bf700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c381a2870 0x7f0c381a4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:28.253 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:28.252+0000 7f0c3d2bf700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c20006230 0x7f0c380ff650 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:28.253 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:28.252+0000 7f0c3d2bf700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0c240389f0 0x7f0c2403aeb0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:28.253 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:28.252+0000 7f0c3d2bf700 1 --2- 172.21.15.152:0/265461093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c380ffb90 0x7f0c38100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:28.253 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:28.252+0000 7f0c3d2bf700 1 -- 172.21.15.152:0/265461093 >> 172.21.15.152:0/265461093 conn(0x7f0c380fb1a0 msgr2=0x7f0c381027a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:28.253 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:28.252+0000 7f0c3d2bf700 1 -- 172.21.15.152:0/265461093 shutdown_connections 2024-03-14T09:47:28.253 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:28.252+0000 7f0c3d2bf700 1 -- 172.21.15.152:0/265461093 wait complete. 2024-03-14T09:47:28.254 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 3 2024-03-14T09:47:28.638 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-14T09:47:28.638 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-14T09:47:28.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:28.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:47:28.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:28.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: mon.a calling monitor election 2024-03-14T09:47:28.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: mon.c calling monitor election 2024-03-14T09:47:28.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:47:28.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:28.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:28.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:28.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:28.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:28.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:28.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:28.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-14T09:47:28.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T09:47:28.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: fsmap 2024-03-14T09:47:28.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: osdmap e4: 0 total, 0 up, 0 in 2024-03-14T09:47:28.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: mgrmap e13: x(active, since 45s) 2024-03-14T09:47:28.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-14T09:47:28.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-14T09:47:28.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-14T09:47:28.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: mon.b (rank 2) addr [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] is down (out of quorum) 2024-03-14T09:47:28.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:28.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:28.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:28.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:28.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:28.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:28.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:28.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:28.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:28.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:28.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:28.773 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:47:29.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: mon.b calling monitor election 2024-03-14T09:47:29.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: Updating smithi132:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:47:29.236 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: Updating smithi152:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:47:29.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: Updating smithi078:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:47:29.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: mon.b calling monitor election 2024-03-14T09:47:29.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: mon.c calling monitor election 2024-03-14T09:47:29.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: mon.a calling monitor election 2024-03-14T09:47:29.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-14T09:47:29.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T09:47:29.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: fsmap 2024-03-14T09:47:29.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: osdmap e4: 0 total, 0 up, 0 in 2024-03-14T09:47:29.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: mgrmap e13: x(active, since 47s) 2024-03-14T09:47:29.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-14T09:47:29.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: Cluster is now healthy 2024-03-14T09:47:29.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: overall HEALTH_OK 2024-03-14T09:47:29.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.237 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T09:47:29.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T09:47:29.238 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:28 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: mon.b calling monitor election 2024-03-14T09:47:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: Updating smithi132:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:47:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: Updating smithi152:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:47:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: Updating smithi078:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:47:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: mon.b calling monitor election 2024-03-14T09:47:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: mon.c calling monitor election 2024-03-14T09:47:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: mon.a calling monitor election 2024-03-14T09:47:29.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-14T09:47:29.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T09:47:29.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: fsmap 2024-03-14T09:47:29.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: osdmap e4: 0 total, 0 up, 0 in 2024-03-14T09:47:29.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: mgrmap e13: x(active, since 47s) 2024-03-14T09:47:29.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-14T09:47:29.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: Cluster is now healthy 2024-03-14T09:47:29.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: overall HEALTH_OK 2024-03-14T09:47:29.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.251 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.251 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.251 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T09:47:29.251 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T09:47:29.251 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:28 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: mon.b calling monitor election 2024-03-14T09:47:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: Updating smithi132:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:47:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: Updating smithi152:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:47:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: Updating smithi078:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:47:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: mon.b calling monitor election 2024-03-14T09:47:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: mon.c calling monitor election 2024-03-14T09:47:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: mon.a calling monitor election 2024-03-14T09:47:29.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-14T09:47:29.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0],c=[v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T09:47:29.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: fsmap 2024-03-14T09:47:29.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: osdmap e4: 0 total, 0 up, 0 in 2024-03-14T09:47:29.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: mgrmap e13: x(active, since 47s) 2024-03-14T09:47:29.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-14T09:47:29.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: Cluster is now healthy 2024-03-14T09:47:29.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: overall HEALTH_OK 2024-03-14T09:47:29.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:29.272 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T09:47:29.272 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T09:47:29.272 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:30.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:29 smithi078 ceph-mon[38143]: Reconfiguring mon.a (unknown last config time)... 2024-03-14T09:47:30.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:29 smithi078 ceph-mon[38143]: Reconfiguring daemon mon.a on smithi078 2024-03-14T09:47:30.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:29 smithi078 ceph-mon[38143]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:30.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:29 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:30.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:29 smithi152 ceph-mon[48279]: Reconfiguring mon.a (unknown last config time)... 2024-03-14T09:47:30.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:29 smithi152 ceph-mon[48279]: Reconfiguring daemon mon.a on smithi078 2024-03-14T09:47:30.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:29 smithi152 ceph-mon[48279]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:30.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:29 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:30.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:29 smithi132 ceph-mon[47959]: Reconfiguring mon.a (unknown last config time)... 2024-03-14T09:47:30.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:29 smithi132 ceph-mon[47959]: Reconfiguring daemon mon.a on smithi078 2024-03-14T09:47:30.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:29 smithi132 ceph-mon[47959]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:30.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:29 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:47:30.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.568+0000 7fdd4c600700 1 -- 172.21.15.78:0/2768366553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd44106090 msgr2=0x7fdd4410dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:30.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.568+0000 7fdd4c600700 1 --2- 172.21.15.78:0/2768366553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd44106090 0x7fdd4410dc60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00b0a0 tx=0x7fdd3c00b3b0 comp rx=0 tx=0).stop 2024-03-14T09:47:30.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.568+0000 7fdd4c600700 1 -- 172.21.15.78:0/2768366553 shutdown_connections 2024-03-14T09:47:30.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.568+0000 7fdd4c600700 1 --2- 172.21.15.78:0/2768366553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd44100fb0 0x7fdd44101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:30.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.568+0000 7fdd4c600700 1 --2- 172.21.15.78:0/2768366553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd44106090 0x7fdd4410dc60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:30.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.568+0000 7fdd4c600700 1 --2- 172.21.15.78:0/2768366553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd44101960 0x7fdd441059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:30.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.568+0000 7fdd4c600700 1 -- 172.21.15.78:0/2768366553 >> 172.21.15.78:0/2768366553 conn(0x7fdd440fc800 msgr2=0x7fdd440fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:30.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.568+0000 7fdd4c600700 1 -- 172.21.15.78:0/2768366553 shutdown_connections 2024-03-14T09:47:30.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.568+0000 7fdd4c600700 1 -- 172.21.15.78:0/2768366553 wait complete. 2024-03-14T09:47:30.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.569+0000 7fdd4c600700 1 Processor -- start 2024-03-14T09:47:30.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.569+0000 7fdd4c600700 1 -- start start 2024-03-14T09:47:30.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd4c600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd44100fb0 0x7fdd4419f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:30.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd4c600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd44101960 0x7fdd4419fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:30.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd4c600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd44106090 0x7fdd44199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:30.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd4c600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd44199db0 con 0x7fdd44100fb0 2024-03-14T09:47:30.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd4c600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd44199f20 con 0x7fdd44106090 2024-03-14T09:47:30.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd4c600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4419a0e0 con 0x7fdd44101960 2024-03-14T09:47:30.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd49c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd44100fb0 0x7fdd4419f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:30.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd49c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd44100fb0 0x7fdd4419f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48678/0 (socket says 172.21.15.78:48678) 2024-03-14T09:47:30.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd49c00700 1 -- 172.21.15.78:0/3602329322 learned_addr learned my addr 172.21.15.78:0/3602329322 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:47:30.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd49200700 1 --2- 172.21.15.78:0/3602329322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd44101960 0x7fdd4419fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:30.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd4a600700 1 --2- 172.21.15.78:0/3602329322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd44106090 0x7fdd44199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:30.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd4a600700 1 -- 172.21.15.78:0/3602329322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd44101960 msgr2=0x7fdd4419fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:30.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd4a600700 1 --2- 172.21.15.78:0/3602329322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd44101960 0x7fdd4419fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:30.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd4a600700 1 -- 172.21.15.78:0/3602329322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd44100fb0 msgr2=0x7fdd4419f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:30.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd4a600700 1 --2- 172.21.15.78:0/3602329322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd44100fb0 0x7fdd4419f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:30.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd4a600700 1 -- 172.21.15.78:0/3602329322 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c009d00 con 0x7fdd44106090 2024-03-14T09:47:30.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.570+0000 7fdd49c00700 1 --2- 172.21.15.78:0/3602329322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd44100fb0 0x7fdd4419f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:47:30.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.571+0000 7fdd4a600700 1 --2- 172.21.15.78:0/3602329322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd44106090 0x7fdd44199750 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00b070 tx=0x7fdd3c007930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:30.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.571+0000 7fdd42000700 1 -- 172.21.15.78:0/3602329322 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c00e040 con 0x7fdd44106090 2024-03-14T09:47:30.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.571+0000 7fdd42000700 1 -- 172.21.15.78:0/3602329322 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd3c01d470 con 0x7fdd44106090 2024-03-14T09:47:30.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.571+0000 7fdd42000700 1 -- 172.21.15.78:0/3602329322 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c0052a0 con 0x7fdd44106090 2024-03-14T09:47:30.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.571+0000 7fdd4c600700 1 -- 172.21.15.78:0/3602329322 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4419a360 con 0x7fdd44106090 2024-03-14T09:47:30.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.571+0000 7fdd4c600700 1 -- 172.21.15.78:0/3602329322 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd441a6600 con 0x7fdd44106090 2024-03-14T09:47:30.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.572+0000 7fdd42000700 1 -- 172.21.15.78:0/3602329322 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 13) v1 ==== 45430+0+0 (secure 0 0 0) 0x7fdd3c01d5e0 con 0x7fdd44106090 2024-03-14T09:47:30.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.572+0000 7fdd4c600700 1 -- 172.21.15.78:0/3602329322 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4419a580 con 0x7fdd44106090 2024-03-14T09:47:30.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.573+0000 7fdd42000700 1 --2- 172.21.15.78:0/3602329322 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdd2c0387a0 0x7fdd2c03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:30.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.573+0000 7fdd42000700 1 -- 172.21.15.78:0/3602329322 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdd3c04c4c0 con 0x7fdd44106090 2024-03-14T09:47:30.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.573+0000 7fdd49c00700 1 --2- 172.21.15.78:0/3602329322 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdd2c0387a0 0x7fdd2c03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:30.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.573+0000 7fdd49c00700 1 --2- 172.21.15.78:0/3602329322 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdd2c0387a0 0x7fdd2c03ac60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdd38011550 tx=0x7fdd38009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:30.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.578+0000 7fdd42000700 1 -- 172.21.15.78:0/3602329322 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fdd3c025ba0 con 0x7fdd44106090 2024-03-14T09:47:30.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.778+0000 7fdd4c600700 1 -- 172.21.15.78:0/3602329322 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fdd4419ac10 con 0x7fdd44106090 2024-03-14T09:47:30.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.779+0000 7fdd42000700 1 -- 172.21.15.78:0/3602329322 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7fdd3c021030 con 0x7fdd44106090 2024-03-14T09:47:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:# minimal ceph.conf for 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:47:30.780 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2024-03-14T09:47:30.780 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:47:30.780 INFO:teuthology.orchestra.run.smithi078.stdout: mon_host = [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] 2024-03-14T09:47:30.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.781+0000 7fdd4c600700 1 -- 172.21.15.78:0/3602329322 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdd2c0387a0 msgr2=0x7fdd2c03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:30.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.781+0000 7fdd4c600700 1 --2- 172.21.15.78:0/3602329322 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdd2c0387a0 0x7fdd2c03ac60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdd38011550 tx=0x7fdd38009260 comp rx=0 tx=0).stop 2024-03-14T09:47:30.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.781+0000 7fdd4c600700 1 -- 172.21.15.78:0/3602329322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd44106090 msgr2=0x7fdd44199750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:30.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.781+0000 7fdd4c600700 1 --2- 172.21.15.78:0/3602329322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd44106090 0x7fdd44199750 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00b070 tx=0x7fdd3c007930 comp rx=0 tx=0).stop 2024-03-14T09:47:30.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.781+0000 7fdd4c600700 1 -- 172.21.15.78:0/3602329322 shutdown_connections 2024-03-14T09:47:30.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.781+0000 7fdd4c600700 1 --2- 172.21.15.78:0/3602329322 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdd2c0387a0 0x7fdd2c03ac60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:30.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.781+0000 7fdd4c600700 1 --2- 172.21.15.78:0/3602329322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd44106090 0x7fdd44199750 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:30.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.781+0000 7fdd4c600700 1 --2- 172.21.15.78:0/3602329322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd44100fb0 0x7fdd4419f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:30.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.781+0000 7fdd4c600700 1 --2- 172.21.15.78:0/3602329322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd44101960 0x7fdd4419fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:30.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.781+0000 7fdd4c600700 1 -- 172.21.15.78:0/3602329322 >> 172.21.15.78:0/3602329322 conn(0x7fdd440fc800 msgr2=0x7fdd440feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:30.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.781+0000 7fdd4c600700 1 -- 172.21.15.78:0/3602329322 shutdown_connections 2024-03-14T09:47:30.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:30.781+0000 7fdd4c600700 1 -- 172.21.15.78:0/3602329322 wait complete. 2024-03-14T09:47:31.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:31.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:31.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:31 smithi078 ceph-mon[38143]: Reconfiguring mon.b (monmap changed)... 2024-03-14T09:47:31.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T09:47:31.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T09:47:31.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:31.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:31 smithi078 ceph-mon[38143]: Reconfiguring daemon mon.b on smithi132 2024-03-14T09:47:31.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:31 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3602329322' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:31.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:31.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:31.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T09:47:31.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T09:47:31.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:31.159 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-14T09:47:31.159 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:47:31.159 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.conf 2024-03-14T09:47:31.228 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:47:31.228 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:47:31.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:31.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:31.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:31 smithi132 ceph-mon[47959]: Reconfiguring mon.b (monmap changed)... 2024-03-14T09:47:31.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T09:47:31.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T09:47:31.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:31.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:31 smithi132 ceph-mon[47959]: Reconfiguring daemon mon.b on smithi132 2024-03-14T09:47:31.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:31 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3602329322' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:31.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:31.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:31.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T09:47:31.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T09:47:31.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:31.332 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:31.332 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:31.332 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:31 smithi152 ceph-mon[48279]: Reconfiguring mon.b (monmap changed)... 2024-03-14T09:47:31.332 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T09:47:31.332 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T09:47:31.332 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:31.332 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:31 smithi152 ceph-mon[48279]: Reconfiguring daemon mon.b on smithi132 2024-03-14T09:47:31.332 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:31 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3602329322' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:31.332 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:31.332 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:31.333 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T09:47:31.333 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T09:47:31.333 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:31.334 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-14T09:47:31.334 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.conf 2024-03-14T09:47:31.364 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-14T09:47:31.365 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:47:31.431 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-14T09:47:31.431 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.conf 2024-03-14T09:47:31.461 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-14T09:47:31.461 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:47:31.534 INFO:tasks.cephadm:Adding mgr.x on smithi078 2024-03-14T09:47:31.535 INFO:tasks.cephadm:Adding mgr.y on smithi132 2024-03-14T09:47:31.535 INFO:tasks.cephadm:Adding mgr.z on smithi152 2024-03-14T09:47:31.535 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi078=x;smithi132=y;smithi152=z' 2024-03-14T09:47:32.003 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.c/config 2024-03-14T09:47:32.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:32 smithi152 ceph-mon[48279]: Reconfiguring mon.c (monmap changed)... 2024-03-14T09:47:32.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:32 smithi152 ceph-mon[48279]: Reconfiguring daemon mon.c on smithi152 2024-03-14T09:47:32.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:32 smithi152 ceph-mon[48279]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:32.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:32 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:32.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:32 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:32.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:32 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:32.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:32 smithi078 ceph-mon[38143]: Reconfiguring mon.c (monmap changed)... 2024-03-14T09:47:32.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:32 smithi078 ceph-mon[38143]: Reconfiguring daemon mon.c on smithi152 2024-03-14T09:47:32.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:32 smithi078 ceph-mon[38143]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:32.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:32 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:32.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:32 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:32.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:32 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:32 smithi132 ceph-mon[47959]: Reconfiguring mon.c (monmap changed)... 2024-03-14T09:47:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:32 smithi132 ceph-mon[47959]: Reconfiguring daemon mon.c on smithi152 2024-03-14T09:47:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:32 smithi132 ceph-mon[47959]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:32 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:32 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:32 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:33.030 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.027+0000 7f0feb5c9700 1 -- 172.21.15.152:0/2563399688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fe41001f0 msgr2=0x7f0fe4100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:33.030 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.027+0000 7f0feb5c9700 1 --2- 172.21.15.152:0/2563399688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fe41001f0 0x7f0fe4100610 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009990 tx=0x7f0fd4009ca0 comp rx=0 tx=0).stop 2024-03-14T09:47:33.030 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.027+0000 7f0feb5c9700 1 -- 172.21.15.152:0/2563399688 shutdown_connections 2024-03-14T09:47:33.030 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.027+0000 7f0feb5c9700 1 --2- 172.21.15.152:0/2563399688 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe41018d0 0x7f0fe4108160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:33.030 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.027+0000 7f0feb5c9700 1 --2- 172.21.15.152:0/2563399688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fe41001f0 0x7f0fe4100610 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:33.030 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.027+0000 7f0feb5c9700 1 --2- 172.21.15.152:0/2563399688 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fe4100f10 0x7f0fe4101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:33.030 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.027+0000 7f0feb5c9700 1 -- 172.21.15.152:0/2563399688 >> 172.21.15.152:0/2563399688 conn(0x7f0fe40fa9f0 msgr2=0x7f0fe40fce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:33.031 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.027+0000 7f0feb5c9700 1 -- 172.21.15.152:0/2563399688 shutdown_connections 2024-03-14T09:47:33.031 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.027+0000 7f0feb5c9700 1 -- 172.21.15.152:0/2563399688 wait complete. 2024-03-14T09:47:33.031 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0feb5c9700 1 Processor -- start 2024-03-14T09:47:33.031 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0feb5c9700 1 -- start start 2024-03-14T09:47:33.031 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0feb5c9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe41001f0 0x7f0fe40ff860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:33.031 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0feb5c9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fe4100f10 0x7f0fe419be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:33.031 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0feb5c9700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fe41018d0 0x7f0fe419e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:33.031 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0feb5c9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe419ebd0 con 0x7f0fe41001f0 2024-03-14T09:47:33.031 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0feb5c9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe419ed40 con 0x7f0fe4100f10 2024-03-14T09:47:33.031 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0feb5c9700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe419ef00 con 0x7f0fe41018d0 2024-03-14T09:47:33.031 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0fe9b66700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fe41018d0 0x7f0fe419e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:33.031 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0fe9b66700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fe41018d0 0x7f0fe419e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:60838/0 (socket says 172.21.15.152:60838) 2024-03-14T09:47:33.032 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0fe9b66700 1 -- 172.21.15.152:0/2974607656 learned_addr learned my addr 172.21.15.152:0/2974607656 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-14T09:47:33.032 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0fe8b64700 1 --2- 172.21.15.152:0/2974607656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fe4100f10 0x7f0fe419be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:33.032 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0fe9b66700 1 -- 172.21.15.152:0/2974607656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fe4100f10 msgr2=0x7f0fe419be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:33.032 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0fe9b66700 1 --2- 172.21.15.152:0/2974607656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fe4100f10 0x7f0fe419be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:33.032 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0fe9b66700 1 -- 172.21.15.152:0/2974607656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe41001f0 msgr2=0x7f0fe40ff860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:33.032 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0fe9b66700 1 --2- 172.21.15.152:0/2974607656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe41001f0 0x7f0fe40ff860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:33.032 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.028+0000 7f0fe9b66700 1 -- 172.21.15.152:0/2974607656 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd40095f0 con 0x7f0fe41018d0 2024-03-14T09:47:33.032 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.029+0000 7f0fe9b66700 1 --2- 172.21.15.152:0/2974607656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fe41018d0 0x7f0fe419e430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc007ee0 tx=0x7f0fdc00d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:33.032 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.029+0000 7f0fda7fc700 1 -- 172.21.15.152:0/2974607656 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc017070 con 0x7f0fe41018d0 2024-03-14T09:47:33.032 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.029+0000 7f0fda7fc700 1 -- 172.21.15.152:0/2974607656 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fdc00f040 con 0x7f0fe41018d0 2024-03-14T09:47:33.032 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.029+0000 7f0fda7fc700 1 -- 172.21.15.152:0/2974607656 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc005740 con 0x7f0fe41018d0 2024-03-14T09:47:33.032 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.029+0000 7f0feb5c9700 1 -- 172.21.15.152:0/2974607656 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe419f1e0 con 0x7f0fe41018d0 2024-03-14T09:47:33.032 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.029+0000 7f0feb5c9700 1 -- 172.21.15.152:0/2974607656 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe419f690 con 0x7f0fe41018d0 2024-03-14T09:47:33.033 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.030+0000 7f0fda7fc700 1 -- 172.21.15.152:0/2974607656 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 13) v1 ==== 45430+0+0 (secure 0 0 0) 0x7f0fdc0058a0 con 0x7f0fe41018d0 2024-03-14T09:47:33.033 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.030+0000 7f0feb5c9700 1 -- 172.21.15.152:0/2974607656 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe4107210 con 0x7f0fe41018d0 2024-03-14T09:47:33.033 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.030+0000 7f0fda7fc700 1 --2- 172.21.15.152:0/2974607656 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0fd00387a0 0x7f0fd003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:33.033 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.030+0000 7f0fda7fc700 1 -- 172.21.15.152:0/2974607656 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0fdc04a920 con 0x7f0fe41018d0 2024-03-14T09:47:33.034 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.030+0000 7f0fe9365700 1 --2- 172.21.15.152:0/2974607656 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0fd00387a0 0x7f0fd003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:33.034 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.031+0000 7f0fe9365700 1 --2- 172.21.15.152:0/2974607656 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0fd00387a0 0x7f0fd003ac60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0fd400b7b0 tx=0x7f0fd4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:33.035 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.035+0000 7f0fda7fc700 1 -- 172.21.15.152:0/2974607656 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0fdc01c410 con 0x7f0fe41018d0 2024-03-14T09:47:33.237 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.237+0000 7f0feb5c9700 1 -- 172.21.15.152:0/2974607656 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi132=y;smithi152=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fe402cdc0 con 0x7f0fd00387a0 2024-03-14T09:47:33.244 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.244+0000 7f0fda7fc700 1 -- 172.21.15.152:0/2974607656 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0fe402cdc0 con 0x7f0fd00387a0 2024-03-14T09:47:33.245 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled mgr update... 2024-03-14T09:47:33.247 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.246+0000 7f0feb5c9700 1 -- 172.21.15.152:0/2974607656 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0fd00387a0 msgr2=0x7f0fd003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:33.247 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.246+0000 7f0feb5c9700 1 --2- 172.21.15.152:0/2974607656 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0fd00387a0 0x7f0fd003ac60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0fd400b7b0 tx=0x7f0fd4005fb0 comp rx=0 tx=0).stop 2024-03-14T09:47:33.247 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.246+0000 7f0feb5c9700 1 -- 172.21.15.152:0/2974607656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fe41018d0 msgr2=0x7f0fe419e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:33.247 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.246+0000 7f0feb5c9700 1 --2- 172.21.15.152:0/2974607656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fe41018d0 0x7f0fe419e430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc007ee0 tx=0x7f0fdc00d3b0 comp rx=0 tx=0).stop 2024-03-14T09:47:33.247 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.246+0000 7f0feb5c9700 1 -- 172.21.15.152:0/2974607656 shutdown_connections 2024-03-14T09:47:33.247 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.246+0000 7f0feb5c9700 1 --2- 172.21.15.152:0/2974607656 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0fe41018d0 0x7f0fe419e430 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:33.247 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.246+0000 7f0feb5c9700 1 --2- 172.21.15.152:0/2974607656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe41001f0 0x7f0fe40ff860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:33.247 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.246+0000 7f0feb5c9700 1 --2- 172.21.15.152:0/2974607656 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0fd00387a0 0x7f0fd003ac60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:33.247 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.246+0000 7f0feb5c9700 1 --2- 172.21.15.152:0/2974607656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fe4100f10 0x7f0fe419be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:33.247 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.246+0000 7f0feb5c9700 1 -- 172.21.15.152:0/2974607656 >> 172.21.15.152:0/2974607656 conn(0x7f0fe40fa9f0 msgr2=0x7f0fe4104350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:33.248 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.246+0000 7f0feb5c9700 1 -- 172.21.15.152:0/2974607656 shutdown_connections 2024-03-14T09:47:33.248 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:47:33.246+0000 7f0feb5c9700 1 -- 172.21.15.152:0/2974607656 wait complete. 2024-03-14T09:47:33.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:33 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:33.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:33 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:33.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:33 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:33.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:33 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:33.408 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:33 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:33.409 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:33 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:33.409 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:33 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:33.409 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:33 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:33 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:33 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:33 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:33 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:33.627 DEBUG:teuthology.orchestra.run.smithi132:mgr.y> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mgr.y.service 2024-03-14T09:47:33.630 DEBUG:teuthology.orchestra.run.smithi152:mgr.z> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mgr.z.service 2024-03-14T09:47:33.632 INFO:tasks.cephadm:Deploying OSDs... 2024-03-14T09:47:33.632 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:47:33.632 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2024-03-14T09:47:33.662 INFO:journalctl@ceph.mgr.y.smithi132.stdout:-- Logs begin at Thu 2024-03-14 09:24:09 UTC. -- 2024-03-14T09:47:33.663 INFO:journalctl@ceph.mgr.z.smithi152.stdout:-- Logs begin at Thu 2024-03-14 09:24:45 UTC. -- 2024-03-14T09:47:33.680 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-14T09:47:33.680 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2024-03-14T09:47:33.769 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-14T09:47:33.769 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T09:47:33.769 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 452 Links: 1 2024-03-14T09:47:33.769 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T09:47:33.770 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T09:47:33.770 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-14 09:46:51.595679082 +0000 2024-03-14T09:47:33.770 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-14 09:40:10.988118837 +0000 2024-03-14T09:47:33.770 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-14 09:40:10.988118837 +0000 2024-03-14T09:47:33.770 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-14 09:40:10.959119335 +0000 2024-03-14T09:47:33.770 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-14T09:47:33.842 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-03-14T09:47:33.842 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-03-14T09:47:33.843 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000329395 s, 1.6 MB/s 2024-03-14T09:47:33.845 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-14T09:47:33.894 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2024-03-14T09:47:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-14T09:47:33.940 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T09:47:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 458 Links: 1 2024-03-14T09:47:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T09:47:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T09:47:33.940 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-14 09:46:51.658677987 +0000 2024-03-14T09:47:33.941 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-14 09:40:11.450110916 +0000 2024-03-14T09:47:33.941 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-14 09:40:11.450110916 +0000 2024-03-14T09:47:33.941 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-14 09:40:11.424111361 +0000 2024-03-14T09:47:33.941 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-14T09:47:34.046 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-03-14T09:47:34.046 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-03-14T09:47:34.046 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000291939 s, 1.8 MB/s 2024-03-14T09:47:34.048 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-14T09:47:34.092 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2024-03-14T09:47:34.174 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-14T09:47:34.175 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T09:47:34.175 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 464 Links: 1 2024-03-14T09:47:34.175 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T09:47:34.175 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T09:47:34.175 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-14 09:46:51.711677065 +0000 2024-03-14T09:47:34.175 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-14 09:40:11.911103011 +0000 2024-03-14T09:47:34.175 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-14 09:40:11.911103011 +0000 2024-03-14T09:47:34.175 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-14 09:40:11.888103405 +0000 2024-03-14T09:47:34.175 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-14T09:47:34.278 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-03-14T09:47:34.278 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-03-14T09:47:34.278 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000290378 s, 1.8 MB/s 2024-03-14T09:47:34.280 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-14T09:47:34.324 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2024-03-14T09:47:34.407 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-14T09:47:34.407 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T09:47:34.407 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 470 Links: 1 2024-03-14T09:47:34.407 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T09:47:34.407 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T09:47:34.407 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-14 09:46:51.768676074 +0000 2024-03-14T09:47:34.407 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-14 09:40:12.376095038 +0000 2024-03-14T09:47:34.407 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-14 09:40:12.376095038 +0000 2024-03-14T09:47:34.407 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-14 09:40:12.352095449 +0000 2024-03-14T09:47:34.408 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-14T09:47:34.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:34 smithi152 ceph-mon[48279]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:34.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:34 smithi152 ceph-mon[48279]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi132=y;smithi152=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:47:34.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:34 smithi152 ceph-mon[48279]: Saving service mgr spec with placement smithi078=x;smithi132=y;smithi152=z;count:3 2024-03-14T09:47:34.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:34.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:34.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:34.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:34.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:34.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T09:47:34.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-14T09:47:34.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T09:47:34.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:34.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:34 smithi152 ceph-mon[48279]: Deploying daemon mgr.z on smithi152 2024-03-14T09:47:34.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:34 smithi078 ceph-mon[38143]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:34.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:34 smithi078 ceph-mon[38143]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi132=y;smithi152=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:47:34.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:34 smithi078 ceph-mon[38143]: Saving service mgr spec with placement smithi078=x;smithi132=y;smithi152=z;count:3 2024-03-14T09:47:34.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:34.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:34.503 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:34.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:34.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:34.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T09:47:34.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-14T09:47:34.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T09:47:34.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:34.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:34 smithi078 ceph-mon[38143]: Deploying daemon mgr.z on smithi152 2024-03-14T09:47:34.509 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-03-14T09:47:34.509 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-03-14T09:47:34.509 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000325628 s, 1.6 MB/s 2024-03-14T09:47:34.512 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-14T09:47:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:34 smithi132 ceph-mon[47959]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:34 smithi132 ceph-mon[47959]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi132=y;smithi152=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:47:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:34 smithi132 ceph-mon[47959]: Saving service mgr spec with placement smithi078=x;smithi132=y;smithi152=z;count:3 2024-03-14T09:47:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T09:47:34.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-14T09:47:34.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T09:47:34.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:34.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:34 smithi132 ceph-mon[47959]: Deploying daemon mgr.z on smithi152 2024-03-14T09:47:34.555 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-14T09:47:34.555 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/scratch_devs of=/dev/stdout 2024-03-14T09:47:34.576 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-14T09:47:34.576 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_1 2024-03-14T09:47:34.638 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-14T09:47:34.638 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T09:47:34.638 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 6h/6d Inode: 177606 Links: 1 2024-03-14T09:47:34.638 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T09:47:34.638 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T09:47:34.638 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-14 09:47:20.970124357 +0000 2024-03-14T09:47:34.638 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-14 09:39:07.702984119 +0000 2024-03-14T09:47:34.638 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-14 09:39:07.702984119 +0000 2024-03-14T09:47:34.638 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: - 2024-03-14T09:47:34.639 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-14T09:47:34.709 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-14T09:47:34.710 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-14T09:47:34.710 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000280097 s, 1.8 MB/s 2024-03-14T09:47:34.711 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-14T09:47:34.772 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_2 2024-03-14T09:47:34.833 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-14T09:47:34.833 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T09:47:34.833 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 6h/6d Inode: 177080 Links: 1 2024-03-14T09:47:34.833 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T09:47:34.833 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T09:47:34.833 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-14 09:47:20.970124357 +0000 2024-03-14T09:47:34.833 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-14 09:39:08.050974855 +0000 2024-03-14T09:47:34.833 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-14 09:39:08.050974855 +0000 2024-03-14T09:47:34.833 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: - 2024-03-14T09:47:34.833 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-14T09:47:34.903 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-14T09:47:34.903 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-14T09:47:34.904 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000233252 s, 2.2 MB/s 2024-03-14T09:47:34.905 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-14T09:47:34.945 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:34 smithi152 systemd[1]: Starting Ceph mgr.z for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T09:47:34.965 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_3 2024-03-14T09:47:35.024 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-14T09:47:35.024 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T09:47:35.024 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 6h/6d Inode: 176075 Links: 1 2024-03-14T09:47:35.024 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T09:47:35.024 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T09:47:35.024 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-14 09:47:20.970124357 +0000 2024-03-14T09:47:35.024 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-14 09:39:08.404965433 +0000 2024-03-14T09:47:35.025 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-14 09:39:08.404965433 +0000 2024-03-14T09:47:35.025 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: - 2024-03-14T09:47:35.025 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-14T09:47:35.092 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-14T09:47:35.092 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-14T09:47:35.092 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000198045 s, 2.6 MB/s 2024-03-14T09:47:35.093 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-14T09:47:35.152 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_4 2024-03-14T09:47:35.211 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-14T09:47:35.211 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T09:47:35.211 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 6h/6d Inode: 178651 Links: 1 2024-03-14T09:47:35.211 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T09:47:35.211 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T09:47:35.211 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-14 09:47:20.970124357 +0000 2024-03-14T09:47:35.211 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-14 09:39:08.751956196 +0000 2024-03-14T09:47:35.211 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-14 09:39:08.751956196 +0000 2024-03-14T09:47:35.211 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: - 2024-03-14T09:47:35.211 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-14T09:47:35.279 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-14T09:47:35.279 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-14T09:47:35.279 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000231101 s, 2.2 MB/s 2024-03-14T09:47:35.280 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-14T09:47:35.337 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-03-14T09:47:35.337 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-03-14T09:47:35.357 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-14T09:47:35.357 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_1 2024-03-14T09:47:35.417 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-14T09:47:35.417 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T09:47:35.417 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 6h/6d Inode: 180557 Links: 1 2024-03-14T09:47:35.417 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T09:47:35.417 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T09:47:35.417 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-14 09:47:34.636378001 +0000 2024-03-14T09:47:35.417 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-14 09:39:18.328979289 +0000 2024-03-14T09:47:35.417 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-14 09:39:18.328979289 +0000 2024-03-14T09:47:35.417 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: - 2024-03-14T09:47:35.418 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-14T09:47:35.487 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:35 smithi152 ceph-mgr[50168]: -- 172.21.15.152:0/4065581003 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d75e17e1a0 con 0x55d75e154c00 2024-03-14T09:47:35.487 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:35 smithi152 bash[50130]: a7ed9640f3c20b0d88772498d6431f36315db9ad760ebdeb81877b8f10a2e347 2024-03-14T09:47:35.487 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:35 smithi152 systemd[1]: Started Ceph mgr.z for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:47:35.490 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-14T09:47:35.491 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-14T09:47:35.491 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000223781 s, 2.3 MB/s 2024-03-14T09:47:35.492 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-14T09:47:35.550 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_2 2024-03-14T09:47:35.609 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-14T09:47:35.609 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T09:47:35.609 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 6h/6d Inode: 180632 Links: 1 2024-03-14T09:47:35.610 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T09:47:35.610 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T09:47:35.610 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-14 09:47:34.636378001 +0000 2024-03-14T09:47:35.610 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-14 09:39:18.675974913 +0000 2024-03-14T09:47:35.610 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-14 09:39:18.675974913 +0000 2024-03-14T09:47:35.610 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: - 2024-03-14T09:47:35.610 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-14T09:47:35.613 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:35.513+0000 7f437ef53200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-14T09:47:35.641 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-14T09:47:35.641 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-14T09:47:35.641 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000230128 s, 2.2 MB/s 2024-03-14T09:47:35.642 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-14T09:47:35.702 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_3 2024-03-14T09:47:35.748 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:35.612+0000 7f437ef53200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-14T09:47:35.749 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:35.709+0000 7f437ef53200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-14T09:47:35.761 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-14T09:47:35.761 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T09:47:35.762 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 6h/6d Inode: 180685 Links: 1 2024-03-14T09:47:35.762 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T09:47:35.762 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T09:47:35.762 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-14 09:47:34.636378001 +0000 2024-03-14T09:47:35.762 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-14 09:39:19.021970549 +0000 2024-03-14T09:47:35.762 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-14 09:39:19.021970549 +0000 2024-03-14T09:47:35.762 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: - 2024-03-14T09:47:35.762 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-14T09:47:35.830 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-14T09:47:35.830 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-14T09:47:35.831 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000192712 s, 2.7 MB/s 2024-03-14T09:47:35.832 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-14T09:47:35.890 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_4 2024-03-14T09:47:35.947 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-14T09:47:35.947 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T09:47:35.947 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 6h/6d Inode: 175667 Links: 1 2024-03-14T09:47:35.947 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T09:47:35.948 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T09:47:35.948 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-03-14 09:47:34.636378001 +0000 2024-03-14T09:47:35.948 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-03-14 09:39:19.366966198 +0000 2024-03-14T09:47:35.948 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-03-14 09:39:19.366966198 +0000 2024-03-14T09:47:35.948 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: - 2024-03-14T09:47:35.948 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-14T09:47:35.998 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:35.798+0000 7f437ef53200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-14T09:47:36.016 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-03-14T09:47:36.016 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-03-14T09:47:36.016 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000230132 s, 2.2 MB/s 2024-03-14T09:47:36.018 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-14T09:47:36.076 INFO:tasks.cephadm:Deploying osd.0 on smithi078 with /dev/vg_nvme/lv_4... 2024-03-14T09:47:36.076 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-14T09:47:36.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:36 smithi132 ceph-mon[47959]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:36.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:36 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:36.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:36 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:36.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:36 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:36.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:36 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T09:47:36.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:36 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-14T09:47:36.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:36 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T09:47:36.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:36 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:36.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:36 smithi132 ceph-mon[47959]: Deploying daemon mgr.y on smithi132 2024-03-14T09:47:36.432 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:47:36.456 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:36 smithi152 ceph-mon[48279]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:36.457 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:36 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:36.457 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:36 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:36.457 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:36 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:36.457 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:36 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T09:47:36.457 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:36 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-14T09:47:36.457 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:36 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T09:47:36.457 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:36 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:36.457 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:36 smithi152 ceph-mon[48279]: Deploying daemon mgr.y on smithi132 2024-03-14T09:47:36.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:36 smithi078 ceph-mon[38143]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:36.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:36 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:36.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:36 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:36.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:36 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:36.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:36 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T09:47:36.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:36 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-14T09:47:36.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:36 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T09:47:36.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:36 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:36.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:36 smithi078 ceph-mon[38143]: Deploying daemon mgr.y on smithi132 2024-03-14T09:47:36.707 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:36 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:36.456+0000 7f437ef53200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-14T09:47:36.770 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:36 smithi132 systemd[1]: Starting Ceph mgr.y for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T09:47:36.999 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:36 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:36.706+0000 7f437ef53200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-14T09:47:36.999 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:36 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:36.791+0000 7f437ef53200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-14T09:47:36.999 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:36 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:36.899+0000 7f437ef53200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-14T09:47:37.270 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:36 smithi132 ceph-mgr[49465]: -- 172.21.15.132:0/1190320026 <== mon.2 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b0c95ce1a0 con 0x55b0c95a5000 2024-03-14T09:47:37.270 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:36 smithi132 bash[49428]: 3e8637b9cc83d337ab3528d52ea23e87cb6c15c9f8ced12789ea14460bfa51eb 2024-03-14T09:47:37.270 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:36 smithi132 systemd[1]: Started Ceph mgr.y for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:47:37.450 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:37 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:37.351+0000 7f39b70fc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-14T09:47:37.499 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:37 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:37.059+0000 7f437ef53200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-14T09:47:37.499 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:37 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:37.186+0000 7f437ef53200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-14T09:47:37.499 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:37 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:37.271+0000 7f437ef53200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-14T09:47:37.648 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:37 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:37.448+0000 7f39b70fc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-14T09:47:37.648 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:37 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:37.547+0000 7f39b70fc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-14T09:47:37.770 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:37 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:37.647+0000 7f39b70fc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-14T09:47:37.999 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:37 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:37.678+0000 7f437ef53200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-14T09:47:38.160 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-14T09:47:38.160 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-14T09:47:38.160 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2024-03-14T09:47:38.160 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2024-03-14T09:47:38.160 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0558203 s, 188 MB/s 2024-03-14T09:47:38.160 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2024-03-14T09:47:38.288 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:38.093+0000 7f437ef53200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-14T09:47:38.288 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:38 smithi152 ceph-mon[48279]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:38.289 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:38 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:38.289 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:38 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:38.289 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:38 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:38.289 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:38 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:38.289 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:38 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:38.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:38 smithi078 ceph-mon[38143]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:38.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:38 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:38.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:38 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:38.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:38 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:38.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:38 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:38.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:38 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:38.316 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:38 smithi132 ceph-mon[47959]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:38.316 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:38 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:38.316 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:38 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:38.316 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:38 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:38.316 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:38 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:38.316 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:38 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:38.567 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:38 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:38.315+0000 7f39b70fc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-14T09:47:38.578 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_4 2024-03-14T09:47:38.749 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:38.287+0000 7f437ef53200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-14T09:47:38.749 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:38.383+0000 7f437ef53200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-14T09:47:38.915 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:38 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:38.566+0000 7f39b70fc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-14T09:47:38.915 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:38 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:38.650+0000 7f39b70fc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-14T09:47:38.915 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:38 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:38.757+0000 7f39b70fc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-14T09:47:38.916 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:38 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:38.914+0000 7f39b70fc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-14T09:47:39.142 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:47:39.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:39 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:39.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:39 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:39.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:39 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:39.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:39 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:39.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:39 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:39.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:39 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:39.213 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:39 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:39.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:39 smithi078 ceph-mon[38143]: Reconfiguring mgr.x (unknown last config time)... 2024-03-14T09:47:39.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:39 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T09:47:39.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:39 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T09:47:39.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:39 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:39.214 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:39 smithi078 ceph-mon[38143]: Reconfiguring daemon mgr.x on smithi078 2024-03-14T09:47:39.270 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:39 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:39.041+0000 7f39b70fc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-14T09:47:39.270 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:39 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:39.124+0000 7f39b70fc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-14T09:47:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:39 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:39 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:39 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:39 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:39.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:39 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:39.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:39 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:39.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:39 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:39.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:39 smithi132 ceph-mon[47959]: Reconfiguring mgr.x (unknown last config time)... 2024-03-14T09:47:39.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:39 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T09:47:39.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:39 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T09:47:39.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:39 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:39.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:39 smithi132 ceph-mon[47959]: Reconfiguring daemon mgr.x on smithi078 2024-03-14T09:47:39.385 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:39 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:39.142+0000 7f437ef53200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-14T09:47:39.385 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:39 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:39.238+0000 7f437ef53200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-14T09:47:39.385 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:39 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:39.385 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:39 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:39.385 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:39 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:39.385 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:39 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:39.385 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:39 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:39.385 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:39 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:39.385 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:39 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:39.385 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:39 smithi152 ceph-mon[48279]: Reconfiguring mgr.x (unknown last config time)... 2024-03-14T09:47:39.385 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:39 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T09:47:39.386 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:39 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T09:47:39.386 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:39 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:39.386 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:39 smithi152 ceph-mon[48279]: Reconfiguring daemon mgr.x on smithi078 2024-03-14T09:47:39.749 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:39 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:39.384+0000 7f437ef53200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-14T09:47:39.770 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:39 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:39.497+0000 7f39b70fc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-14T09:47:40.171 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:39 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:39.887+0000 7f39b70fc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-14T09:47:40.171 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:40 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:40.078+0000 7f39b70fc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-14T09:47:40.171 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:40 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:40.170+0000 7f39b70fc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-14T09:47:40.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.379+0000 7f64a7a00700 1 -- 172.21.15.78:0/1949747203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64a0102980 msgr2=0x7f64a0109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:40.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.379+0000 7f64a7a00700 1 --2- 172.21.15.78:0/1949747203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64a0102980 0x7f64a0109230 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f649800b0d0 tx=0x7f649800b3e0 comp rx=0 tx=0).stop 2024-03-14T09:47:40.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.379+0000 7f64a7a00700 1 -- 172.21.15.78:0/1949747203 shutdown_connections 2024-03-14T09:47:40.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.379+0000 7f64a7a00700 1 --2- 172.21.15.78:0/1949747203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64a0102980 0x7f64a0109230 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:40.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.379+0000 7f64a7a00700 1 --2- 172.21.15.78:0/1949747203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64a0100de0 0x7f64a0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:40.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.379+0000 7f64a7a00700 1 --2- 172.21.15.78:0/1949747203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64a0101fe0 0x7f64a0102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:40.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.379+0000 7f64a7a00700 1 -- 172.21.15.78:0/1949747203 >> 172.21.15.78:0/1949747203 conn(0x7f64a00fc380 msgr2=0x7f64a00fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:40.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.379+0000 7f64a7a00700 1 -- 172.21.15.78:0/1949747203 shutdown_connections 2024-03-14T09:47:40.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.379+0000 7f64a7a00700 1 -- 172.21.15.78:0/1949747203 wait complete. 2024-03-14T09:47:40.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.380+0000 7f64a7a00700 1 Processor -- start 2024-03-14T09:47:40.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f64a7a00700 1 -- start start 2024-03-14T09:47:40.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f64a7a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64a0100de0 0x7f64a0074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:40.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f64a7a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64a0101fe0 0x7f64a00731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:40.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f64a7a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64a0102980 0x7f64a00736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:40.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f64a7a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a019b0f0 con 0x7f64a0101fe0 2024-03-14T09:47:40.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f64a7a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a0073c20 con 0x7f64a0100de0 2024-03-14T09:47:40.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f64a7a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a0073de0 con 0x7f64a0102980 2024-03-14T09:47:40.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f649fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64a0101fe0 0x7f64a00731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:40.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f649fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64a0101fe0 0x7f64a00731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36526/0 (socket says 172.21.15.78:36526) 2024-03-14T09:47:40.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f649fe00700 1 -- 172.21.15.78:0/2646650335 learned_addr learned my addr 172.21.15.78:0/2646650335 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:47:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f64a5000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64a0100de0 0x7f64a0074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f64a5a00700 1 --2- 172.21.15.78:0/2646650335 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64a0102980 0x7f64a00736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f649fe00700 1 -- 172.21.15.78:0/2646650335 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64a0102980 msgr2=0x7f64a00736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f649fe00700 1 --2- 172.21.15.78:0/2646650335 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64a0102980 0x7f64a00736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f649fe00700 1 -- 172.21.15.78:0/2646650335 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64a0100de0 msgr2=0x7f64a0074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f649fe00700 1 --2- 172.21.15.78:0/2646650335 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64a0100de0 0x7f64a0074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.381+0000 7f649fe00700 1 -- 172.21.15.78:0/2646650335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6498009d00 con 0x7f64a0101fe0 2024-03-14T09:47:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.382+0000 7f649fe00700 1 --2- 172.21.15.78:0/2646650335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64a0101fe0 0x7f64a00731a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f648c00e9a0 tx=0x7f648c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.382+0000 7f649d600700 1 -- 172.21.15.78:0/2646650335 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648c009790 con 0x7f64a0101fe0 2024-03-14T09:47:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.382+0000 7f64a7a00700 1 -- 172.21.15.78:0/2646650335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a01a5aa0 con 0x7f64a0101fe0 2024-03-14T09:47:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.382+0000 7f649d600700 1 -- 172.21.15.78:0/2646650335 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f648c010430 con 0x7f64a0101fe0 2024-03-14T09:47:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.382+0000 7f649d600700 1 -- 172.21.15.78:0/2646650335 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648c018680 con 0x7f64a0101fe0 2024-03-14T09:47:40.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.382+0000 7f64a7a00700 1 -- 172.21.15.78:0/2646650335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a01a5e50 con 0x7f64a0101fe0 2024-03-14T09:47:40.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.383+0000 7f649d600700 1 -- 172.21.15.78:0/2646650335 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45430+0+0 (secure 0 0 0) 0x7f648c00ca10 con 0x7f64a0101fe0 2024-03-14T09:47:40.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.383+0000 7f64a7a00700 1 -- 172.21.15.78:0/2646650335 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a0108220 con 0x7f64a0101fe0 2024-03-14T09:47:40.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.384+0000 7f649d600700 1 --2- 172.21.15.78:0/2646650335 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f64880387a0 0x7f648803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:47:40.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.384+0000 7f649d600700 1 -- 172.21.15.78:0/2646650335 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f648c014070 con 0x7f64a0101fe0 2024-03-14T09:47:40.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.384+0000 7f64a5000700 1 --2- 172.21.15.78:0/2646650335 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f64880387a0 0x7f648803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:47:40.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.384+0000 7f64a5000700 1 --2- 172.21.15.78:0/2646650335 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f64880387a0 0x7f648803ac60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f64a0101e40 tx=0x7f6494009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:47:40.386 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:40 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:40.013+0000 7f437ef53200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-14T09:47:40.386 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:40 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:40.193+0000 7f437ef53200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-14T09:47:40.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.389+0000 7f649d600700 1 -- 172.21.15.78:0/2646650335 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f648c00cdb0 con 0x7f64a0101fe0 2024-03-14T09:47:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:40.603+0000 7f64a7a00700 1 -- 172.21.15.78:0/2646650335 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f64a00611f0 con 0x7f64880387a0 2024-03-14T09:47:40.749 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:40 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:40.386+0000 7f437ef53200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-14T09:47:40.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:40 smithi152 ceph-mon[48279]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:40.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:40 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:40.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:40 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:40.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:40 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:40.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:40 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:40.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:40 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:40.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:40 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:40.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:40 smithi132 ceph-mon[47959]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:40.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:40 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:40.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:40 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:40.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:40 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:40.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:40 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:40.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:40 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:40.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:40 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:40.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:40 smithi078 ceph-mon[38143]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:40.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:40 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:40.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:40 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:40.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:40 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:40.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:40 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:40.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:40 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:40.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:40 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:41.270 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:40 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:40.925+0000 7f39b70fc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-14T09:47:41.270 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:41 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:41.020+0000 7f39b70fc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-14T09:47:41.270 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:41 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:41.166+0000 7f39b70fc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-14T09:47:41.595 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:41 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:41.312+0000 7f437ef53200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-14T09:47:41.595 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:47:41.595 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:47:41.595 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:41.595 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:41.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:47:41.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:47:41.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:41.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:41.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:47:41.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:47:41.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:41.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:41.873 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:41 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:41.595+0000 7f437ef53200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-14T09:47:41.873 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:41 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:41.682+0000 7f437ef53200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-14T09:47:42.156 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:41 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:41.787+0000 7f39b70fc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-14T09:47:42.156 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:41 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:41.965+0000 7f39b70fc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-14T09:47:42.156 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:42 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:42.155+0000 7f39b70fc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-14T09:47:42.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:42.489+0000 7f649d600700 1 -- 172.21.15.78:0/2646650335 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 14) v1 ==== 90381+0+0 (secure 0 0 0) 0x7f648c018830 con 0x7f64a0101fe0 2024-03-14T09:47:42.749 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:47:41 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:47:41.873+0000 7f437ef53200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-14T09:47:42.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:42 smithi152 ceph-mon[48279]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:47:42.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:42 smithi152 ceph-mon[48279]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:42.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:42 smithi152 ceph-mon[48279]: Standby manager daemon z started 2024-03-14T09:47:42.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:42 smithi152 ceph-mon[48279]: from='mgr.? 172.21.15.152:0/3563534501' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-14T09:47:42.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:42 smithi152 ceph-mon[48279]: from='mgr.? 172.21.15.152:0/3563534501' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T09:47:42.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:42 smithi152 ceph-mon[48279]: from='mgr.? 172.21.15.152:0/3563534501' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-14T09:47:42.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:42 smithi152 ceph-mon[48279]: from='mgr.? 172.21.15.152:0/3563534501' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T09:47:42.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:42 smithi132 ceph-mon[47959]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:47:42.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:42 smithi132 ceph-mon[47959]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:42.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:42 smithi132 ceph-mon[47959]: Standby manager daemon z started 2024-03-14T09:47:42.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:42 smithi132 ceph-mon[47959]: from='mgr.? 172.21.15.152:0/3563534501' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-14T09:47:42.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:42 smithi132 ceph-mon[47959]: from='mgr.? 172.21.15.152:0/3563534501' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T09:47:42.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:42 smithi132 ceph-mon[47959]: from='mgr.? 172.21.15.152:0/3563534501' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-14T09:47:42.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:42 smithi132 ceph-mon[47959]: from='mgr.? 172.21.15.152:0/3563534501' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T09:47:42.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:42 smithi078 ceph-mon[38143]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:47:42.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:42 smithi078 ceph-mon[38143]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:42.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:42 smithi078 ceph-mon[38143]: Standby manager daemon z started 2024-03-14T09:47:42.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:42 smithi078 ceph-mon[38143]: from='mgr.? 172.21.15.152:0/3563534501' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-14T09:47:42.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:42 smithi078 ceph-mon[38143]: from='mgr.? 172.21.15.152:0/3563534501' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T09:47:42.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:42 smithi078 ceph-mon[38143]: from='mgr.? 172.21.15.152:0/3563534501' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-14T09:47:42.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:42 smithi078 ceph-mon[38143]: from='mgr.? 172.21.15.152:0/3563534501' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T09:47:43.361 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:43 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:43.079+0000 7f39b70fc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-14T09:47:43.361 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:43 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:43.360+0000 7f39b70fc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-14T09:47:43.635 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:43 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:43.447+0000 7f39b70fc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-14T09:47:43.636 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:43 smithi132 ceph-mon[47959]: mgrmap e14: x(active, since 61s), standbys: z 2024-03-14T09:47:43.636 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-14T09:47:43.636 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:43 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1551358112' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9918c01-e344-4789-83f9-6f9425f98542"}]: dispatch 2024-03-14T09:47:43.636 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:43 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9918c01-e344-4789-83f9-6f9425f98542"}]: dispatch 2024-03-14T09:47:43.636 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:43 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9918c01-e344-4789-83f9-6f9425f98542"}]': finished 2024-03-14T09:47:43.636 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:43 smithi132 ceph-mon[47959]: osdmap e5: 1 total, 0 up, 1 in 2024-03-14T09:47:43.636 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:47:43.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:43 smithi152 ceph-mon[48279]: mgrmap e14: x(active, since 61s), standbys: z 2024-03-14T09:47:43.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-14T09:47:43.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:43 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1551358112' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9918c01-e344-4789-83f9-6f9425f98542"}]: dispatch 2024-03-14T09:47:43.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:43 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9918c01-e344-4789-83f9-6f9425f98542"}]: dispatch 2024-03-14T09:47:43.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:43 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9918c01-e344-4789-83f9-6f9425f98542"}]': finished 2024-03-14T09:47:43.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:43 smithi152 ceph-mon[48279]: osdmap e5: 1 total, 0 up, 1 in 2024-03-14T09:47:43.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:47:43.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:43 smithi078 ceph-mon[38143]: mgrmap e14: x(active, since 61s), standbys: z 2024-03-14T09:47:43.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-14T09:47:43.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:43 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1551358112' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9918c01-e344-4789-83f9-6f9425f98542"}]: dispatch 2024-03-14T09:47:43.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:43 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9918c01-e344-4789-83f9-6f9425f98542"}]: dispatch 2024-03-14T09:47:43.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:43 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9918c01-e344-4789-83f9-6f9425f98542"}]': finished 2024-03-14T09:47:43.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:43 smithi078 ceph-mon[38143]: osdmap e5: 1 total, 0 up, 1 in 2024-03-14T09:47:43.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:47:44.020 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:47:43 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:47:43.635+0000 7f39b70fc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-14T09:47:44.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:44.502+0000 7f649d600700 1 -- 172.21.15.78:0/2646650335 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mgrmap(e 15) v1 ==== 135332+0+0 (secure 0 0 0) 0x7f648c01dcb0 con 0x7f64a0101fe0 2024-03-14T09:47:44.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:44 smithi152 ceph-mon[48279]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:44.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:44 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/981707888' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:47:44.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:44 smithi152 ceph-mon[48279]: Standby manager daemon y started 2024-03-14T09:47:44.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:44 smithi152 ceph-mon[48279]: from='mgr.? 172.21.15.132:0/536268938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-14T09:47:44.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:44 smithi152 ceph-mon[48279]: from='mgr.? 172.21.15.132:0/536268938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T09:47:44.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:44 smithi152 ceph-mon[48279]: from='mgr.? 172.21.15.132:0/536268938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-14T09:47:44.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:44 smithi152 ceph-mon[48279]: from='mgr.? 172.21.15.132:0/536268938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T09:47:44.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:44 smithi132 ceph-mon[47959]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:44.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:44 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/981707888' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:47:44.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:44 smithi132 ceph-mon[47959]: Standby manager daemon y started 2024-03-14T09:47:44.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:44 smithi132 ceph-mon[47959]: from='mgr.? 172.21.15.132:0/536268938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-14T09:47:44.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:44 smithi132 ceph-mon[47959]: from='mgr.? 172.21.15.132:0/536268938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T09:47:44.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:44 smithi132 ceph-mon[47959]: from='mgr.? 172.21.15.132:0/536268938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-14T09:47:44.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:44 smithi132 ceph-mon[47959]: from='mgr.? 172.21.15.132:0/536268938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T09:47:44.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:44 smithi078 ceph-mon[38143]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:44.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:44 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/981707888' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:47:44.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:44 smithi078 ceph-mon[38143]: Standby manager daemon y started 2024-03-14T09:47:44.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:44 smithi078 ceph-mon[38143]: from='mgr.? 172.21.15.132:0/536268938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-14T09:47:44.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:44 smithi078 ceph-mon[38143]: from='mgr.? 172.21.15.132:0/536268938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T09:47:44.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:44 smithi078 ceph-mon[38143]: from='mgr.? 172.21.15.132:0/536268938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-14T09:47:44.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:44 smithi078 ceph-mon[38143]: from='mgr.? 172.21.15.132:0/536268938' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T09:47:45.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:45 smithi132 ceph-mon[47959]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-03-14T09:47:45.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:45 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-14T09:47:45.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:45 smithi078 ceph-mon[38143]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-03-14T09:47:45.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:45 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-14T09:47:45.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:45 smithi152 ceph-mon[48279]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-03-14T09:47:45.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:45 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-14T09:47:46.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:46 smithi132 ceph-mon[47959]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:46.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:46 smithi078 ceph-mon[38143]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:46.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:46 smithi152 ceph-mon[48279]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:48.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:48 smithi132 ceph-mon[47959]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:48.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:48 smithi078 ceph-mon[38143]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:48.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:48 smithi152 ceph-mon[48279]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:49.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:49 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-14T09:47:49.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:49 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:49.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:49 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-14T09:47:49.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:49 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:49.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:49 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-14T09:47:49.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:49 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:50.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:50 smithi132 ceph-mon[47959]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:50.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:50 smithi132 ceph-mon[47959]: Deploying daemon osd.0 on smithi078 2024-03-14T09:47:50.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:50 smithi078 ceph-mon[38143]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:50.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:50 smithi078 ceph-mon[38143]: Deploying daemon osd.0 on smithi078 2024-03-14T09:47:50.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:50 smithi152 ceph-mon[48279]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:50.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:50 smithi152 ceph-mon[48279]: Deploying daemon osd.0 on smithi078 2024-03-14T09:47:52.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:52 smithi132 ceph-mon[47959]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:52.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:52 smithi078 ceph-mon[38143]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:52.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:52 smithi152 ceph-mon[48279]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:53.917 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:53 smithi078 ceph-mon[38143]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:53.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:53 smithi152 ceph-mon[48279]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:54.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:53 smithi132 ceph-mon[47959]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:55.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:55 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:55.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:55 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:55.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:55 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:55.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:55 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:55.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:55 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:55.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:55 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:55 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:55 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:55 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:56.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:56 smithi078 ceph-mon[38143]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:56.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:56 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:56.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:56 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:56.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:56 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:56.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:56 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:56.702 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:56 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:56.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:56 smithi152 ceph-mon[48279]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:56.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:56 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:56.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:56 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:56.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:56 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:56.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:56 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:56.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:56 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:56.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:56.754+0000 7f649d600700 1 -- 172.21.15.78:0/2646650335 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f64a00611f0 con 0x7f64880387a0 2024-03-14T09:47:56.755 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 0 on host 'smithi078' 2024-03-14T09:47:56.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:56.756+0000 7f64a7a00700 1 -- 172.21.15.78:0/2646650335 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f64880387a0 msgr2=0x7f648803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:56.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:56.756+0000 7f64a7a00700 1 --2- 172.21.15.78:0/2646650335 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f64880387a0 0x7f648803ac60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f64a0101e40 tx=0x7f6494009260 comp rx=0 tx=0).stop 2024-03-14T09:47:56.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:56.756+0000 7f64a7a00700 1 -- 172.21.15.78:0/2646650335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64a0101fe0 msgr2=0x7f64a00731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:47:56.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:56.756+0000 7f64a7a00700 1 --2- 172.21.15.78:0/2646650335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64a0101fe0 0x7f64a00731a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f648c00e9a0 tx=0x7f648c00ecb0 comp rx=0 tx=0).stop 2024-03-14T09:47:56.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:56.756+0000 7f64a7a00700 1 -- 172.21.15.78:0/2646650335 shutdown_connections 2024-03-14T09:47:56.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:56.756+0000 7f64a7a00700 1 --2- 172.21.15.78:0/2646650335 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f64880387a0 0x7f648803ac60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:56.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:56.756+0000 7f64a7a00700 1 --2- 172.21.15.78:0/2646650335 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f64a0100de0 0x7f64a0074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:56.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:56.756+0000 7f64a7a00700 1 --2- 172.21.15.78:0/2646650335 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f64a0102980 0x7f64a00736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:56.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:56.756+0000 7f64a7a00700 1 --2- 172.21.15.78:0/2646650335 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64a0101fe0 0x7f64a00731a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:47:56.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:56.756+0000 7f64a7a00700 1 -- 172.21.15.78:0/2646650335 >> 172.21.15.78:0/2646650335 conn(0x7f64a00fc380 msgr2=0x7f64a0105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:47:56.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:56.756+0000 7f64a7a00700 1 -- 172.21.15.78:0/2646650335 shutdown_connections 2024-03-14T09:47:56.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:47:56.756+0000 7f64a7a00700 1 -- 172.21.15.78:0/2646650335 wait complete. 2024-03-14T09:47:56.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:56 smithi132 ceph-mon[47959]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:56.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:56 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:56.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:56 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:56.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:56 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:47:56.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:56 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:47:56.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:56 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:57.127 DEBUG:teuthology.orchestra.run.smithi078:osd.0> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.0.service 2024-03-14T09:47:57.131 INFO:tasks.cephadm:Deploying osd.1 on smithi132 with /dev/vg_nvme/lv_4... 2024-03-14T09:47:57.131 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-14T09:47:57.207 INFO:journalctl@ceph.osd.0.smithi078.stdout:-- Logs begin at Thu 2024-03-14 09:30:54 UTC. -- 2024-03-14T09:47:57.357 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.b/config 2024-03-14T09:47:57.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:57 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:57.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:57 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:57.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:57 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:57.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:57 smithi152 ceph-mon[48279]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:58.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:57 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:58.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:57 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:58.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:57 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:58.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:57 smithi132 ceph-mon[47959]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:58.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:57 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:58.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:57 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:47:58.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:57 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:47:58.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:57 smithi078 ceph-mon[38143]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:47:58.554 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 14 09:47:58 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-0[56324]: 2024-03-14T09:47:58.120+0000 7f18f1a01640 -1 osd.0 0 log_to_monitors true 2024-03-14T09:47:58.663 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-14T09:47:58.663 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-14T09:47:58.663 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-03-14T09:47:58.663 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-03-14T09:47:58.664 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113281 s, 926 MB/s 2024-03-14T09:47:58.664 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-03-14T09:47:58.962 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:58 smithi132 ceph-mon[47959]: from='osd.0 [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-14T09:47:58.962 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:58 smithi132 ceph-mon[47959]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-14T09:47:58.994 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi132:vg_nvme/lv_4 2024-03-14T09:47:58.998 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:58 smithi152 ceph-mon[48279]: from='osd.0 [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-14T09:47:58.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:58 smithi152 ceph-mon[48279]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-14T09:47:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:58 smithi078 ceph-mon[38143]: from='osd.0 [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-14T09:47:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:58 smithi078 ceph-mon[38143]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-14T09:47:59.209 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.b/config 2024-03-14T09:48:00.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:59 smithi132 ceph-mon[47959]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-14T09:48:00.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:59 smithi132 ceph-mon[47959]: osdmap e6: 1 total, 0 up, 1 in 2024-03-14T09:48:00.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:59 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:48:00.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:59 smithi132 ceph-mon[47959]: from='osd.0 [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:48:00.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:59 smithi132 ceph-mon[47959]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:48:00.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:47:59 smithi132 ceph-mon[47959]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:48:00.053 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 14 09:47:59 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-0[56324]: 2024-03-14T09:47:59.766+0000 7f18e6000700 -1 osd.0 0 waiting for initial osdmap 2024-03-14T09:48:00.053 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 14 09:47:59 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-0[56324]: 2024-03-14T09:47:59.769+0000 7f18ddc00700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T09:48:00.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:59 smithi078 ceph-mon[38143]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-14T09:48:00.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:59 smithi078 ceph-mon[38143]: osdmap e6: 1 total, 0 up, 1 in 2024-03-14T09:48:00.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:59 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:48:00.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:59 smithi078 ceph-mon[38143]: from='osd.0 [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:48:00.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:59 smithi078 ceph-mon[38143]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:48:00.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:47:59 smithi078 ceph-mon[38143]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:48:00.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.195+0000 7f28e623e700 1 -- 172.21.15.132:0/84362658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28e0101870 msgr2=0x7f28e0108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:00.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.195+0000 7f28e623e700 1 --2- 172.21.15.132:0/84362658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28e0101870 0x7f28e0108120 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f28d800b0d0 tx=0x7f28d800b3e0 comp rx=0 tx=0).stop 2024-03-14T09:48:00.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.195+0000 7f28e623e700 1 -- 172.21.15.132:0/84362658 shutdown_connections 2024-03-14T09:48:00.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.195+0000 7f28e623e700 1 --2- 172.21.15.132:0/84362658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28e0101870 0x7f28e0108120 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:00.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.195+0000 7f28e623e700 1 --2- 172.21.15.132:0/84362658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28e00ffcd0 0x7f28e01000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:00.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.195+0000 7f28e623e700 1 --2- 172.21.15.132:0/84362658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28e0100ed0 0x7f28e0101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:00.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.195+0000 7f28e623e700 1 -- 172.21.15.132:0/84362658 >> 172.21.15.132:0/84362658 conn(0x7f28e00fb270 msgr2=0x7f28e00fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:48:00.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.195+0000 7f28e623e700 1 -- 172.21.15.132:0/84362658 shutdown_connections 2024-03-14T09:48:00.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.195+0000 7f28e623e700 1 -- 172.21.15.132:0/84362658 wait complete. 2024-03-14T09:48:00.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.196+0000 7f28e623e700 1 Processor -- start 2024-03-14T09:48:00.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.196+0000 7f28e623e700 1 -- start start 2024-03-14T09:48:00.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.196+0000 7f28e623e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28e00ffcd0 0x7f28e0197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:48:00.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.196+0000 7f28e623e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28e0100ed0 0x7f28e0197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:48:00.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.196+0000 7f28e623e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28e0101870 0x7f28e019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:48:00.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.196+0000 7f28e623e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e019ec90 con 0x7f28e00ffcd0 2024-03-14T09:48:00.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.196+0000 7f28e623e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e019edd0 con 0x7f28e0101870 2024-03-14T09:48:00.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.196+0000 7f28e623e700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e019ef60 con 0x7f28e0100ed0 2024-03-14T09:48:00.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.196+0000 7f28dffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28e0101870 0x7f28e019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:48:00.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.196+0000 7f28dffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28e0101870 0x7f28e019e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:45242/0 (socket says 172.21.15.132:45242) 2024-03-14T09:48:00.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.196+0000 7f28dffff700 1 -- 172.21.15.132:0/1816100014 learned_addr learned my addr 172.21.15.132:0/1816100014 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-14T09:48:00.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.197+0000 7f28dffff700 1 -- 172.21.15.132:0/1816100014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28e0100ed0 msgr2=0x7f28e0197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:00.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.197+0000 7f28deffd700 1 --2- 172.21.15.132:0/1816100014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28e0100ed0 0x7f28e0197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:48:00.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.197+0000 7f28df7fe700 1 --2- 172.21.15.132:0/1816100014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28e00ffcd0 0x7f28e0197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:48:00.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.197+0000 7f28dffff700 1 --2- 172.21.15.132:0/1816100014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28e0100ed0 0x7f28e0197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:00.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.197+0000 7f28dffff700 1 -- 172.21.15.132:0/1816100014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28e00ffcd0 msgr2=0x7f28e0197150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:00.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.197+0000 7f28dffff700 1 --2- 172.21.15.132:0/1816100014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28e00ffcd0 0x7f28e0197150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:00.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.197+0000 7f28dffff700 1 -- 172.21.15.132:0/1816100014 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d8009d00 con 0x7f28e0101870 2024-03-14T09:48:00.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.197+0000 7f28deffd700 1 --2- 172.21.15.132:0/1816100014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28e0100ed0 0x7f28e0197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:48:00.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.197+0000 7f28df7fe700 1 --2- 172.21.15.132:0/1816100014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28e00ffcd0 0x7f28e0197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:48:00.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.197+0000 7f28dffff700 1 --2- 172.21.15.132:0/1816100014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28e0101870 0x7f28e019e6c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f28d8007bd0 tx=0x7f28d80094f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:48:00.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.197+0000 7f28dcff9700 1 -- 172.21.15.132:0/1816100014 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d800e040 con 0x7f28e0101870 2024-03-14T09:48:00.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.197+0000 7f28dcff9700 1 -- 172.21.15.132:0/1816100014 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28d8012dc0 con 0x7f28e0101870 2024-03-14T09:48:00.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.197+0000 7f28e623e700 1 -- 172.21.15.132:0/1816100014 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e019f1e0 con 0x7f28e0101870 2024-03-14T09:48:00.200 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.197+0000 7f28dcff9700 1 -- 172.21.15.132:0/1816100014 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d801b450 con 0x7f28e0101870 2024-03-14T09:48:00.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.197+0000 7f28e623e700 1 -- 172.21.15.132:0/1816100014 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e019f6a0 con 0x7f28e0101870 2024-03-14T09:48:00.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.198+0000 7f28e623e700 1 -- 172.21.15.132:0/1816100014 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e004ea80 con 0x7f28e0101870 2024-03-14T09:48:00.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.200+0000 7f28dcff9700 1 -- 172.21.15.132:0/1816100014 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 15) v1 ==== 135332+0+0 (secure 0 0 0) 0x7f28d8012900 con 0x7f28e0101870 2024-03-14T09:48:00.201 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.200+0000 7f28dcff9700 1 --2- 172.21.15.132:0/1816100014 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f28c80a0a80 0x7f28c80a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:48:00.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.200+0000 7f28dcff9700 1 -- 172.21.15.132:0/1816100014 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1487+0+0 (secure 0 0 0) 0x7f28d80cc9b0 con 0x7f28e0101870 2024-03-14T09:48:00.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.201+0000 7f28df7fe700 1 --2- 172.21.15.132:0/1816100014 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f28c80a0a80 0x7f28c80a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:48:00.202 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.201+0000 7f28df7fe700 1 --2- 172.21.15.132:0/1816100014 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f28c80a0a80 0x7f28c80a2f40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f28d4005fd0 tx=0x7f28d4005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:48:00.203 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.204+0000 7f28dcff9700 1 -- 172.21.15.132:0/1816100014 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f28d8097fa0 con 0x7f28e0101870 2024-03-14T09:48:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:59 smithi152 ceph-mon[48279]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-14T09:48:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:59 smithi152 ceph-mon[48279]: osdmap e6: 1 total, 0 up, 1 in 2024-03-14T09:48:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:59 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:48:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:59 smithi152 ceph-mon[48279]: from='osd.0 [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:48:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:59 smithi152 ceph-mon[48279]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:48:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:47:59 smithi152 ceph-mon[48279]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:48:00.404 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:00.405+0000 7f28e623e700 1 -- 172.21.15.132:0/1816100014 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f28e00fd3c0 con 0x7f28c80a0a80 2024-03-14T09:48:01.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-14T09:48:01.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: osdmap e7: 1 total, 0 up, 1 in 2024-03-14T09:48:01.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:48:01.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:48:01.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: Detected new or changed devices on smithi078 2024-03-14T09:48:01.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:01.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:01.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:48:01.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi078 to 17270M 2024-03-14T09:48:01.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:01.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:01.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:01.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:01.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:48:01.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:48:01.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:48:01.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:00 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:01.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-14T09:48:01.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: osdmap e7: 1 total, 0 up, 1 in 2024-03-14T09:48:01.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:48:01.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:48:01.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: Detected new or changed devices on smithi078 2024-03-14T09:48:01.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:01.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:01.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:48:01.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi078 to 17270M 2024-03-14T09:48:01.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:01.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:01.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:01.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:01.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:48:01.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:48:01.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:48:01.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:00 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:01.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-14T09:48:01.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: osdmap e7: 1 total, 0 up, 1 in 2024-03-14T09:48:01.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:48:01.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:48:01.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: Detected new or changed devices on smithi078 2024-03-14T09:48:01.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:01.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:01.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:48:01.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi078 to 17270M 2024-03-14T09:48:01.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:01.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:01.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:01.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:01.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:48:01.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:48:01.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:48:01.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:00 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:02.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:01 smithi078 ceph-mon[38143]: purged_snaps scrub starts 2024-03-14T09:48:02.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:01 smithi078 ceph-mon[38143]: purged_snaps scrub ok 2024-03-14T09:48:02.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:01 smithi078 ceph-mon[38143]: osd.0 [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001] boot 2024-03-14T09:48:02.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:01 smithi078 ceph-mon[38143]: osdmap e8: 1 total, 1 up, 1 in 2024-03-14T09:48:02.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:01 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:48:02.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:01 smithi078 ceph-mon[38143]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:48:02.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:01 smithi152 ceph-mon[48279]: purged_snaps scrub starts 2024-03-14T09:48:02.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:01 smithi152 ceph-mon[48279]: purged_snaps scrub ok 2024-03-14T09:48:02.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:01 smithi152 ceph-mon[48279]: osd.0 [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001] boot 2024-03-14T09:48:02.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:01 smithi152 ceph-mon[48279]: osdmap e8: 1 total, 1 up, 1 in 2024-03-14T09:48:02.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:01 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:48:02.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:01 smithi152 ceph-mon[48279]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:48:02.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:01 smithi132 ceph-mon[47959]: purged_snaps scrub starts 2024-03-14T09:48:02.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:01 smithi132 ceph-mon[47959]: purged_snaps scrub ok 2024-03-14T09:48:02.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:01 smithi132 ceph-mon[47959]: osd.0 [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001] boot 2024-03-14T09:48:02.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:01 smithi132 ceph-mon[47959]: osdmap e8: 1 total, 1 up, 1 in 2024-03-14T09:48:02.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:01 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:48:02.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:01 smithi132 ceph-mon[47959]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T09:48:03.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:02 smithi078 ceph-mon[38143]: osdmap e9: 1 total, 1 up, 1 in 2024-03-14T09:48:03.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:02 smithi078 ceph-mon[38143]: from='client.? 172.21.15.132:0/1724409367' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71"}]: dispatch 2024-03-14T09:48:03.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:02 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71"}]: dispatch 2024-03-14T09:48:03.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:02 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71"}]': finished 2024-03-14T09:48:03.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:02 smithi078 ceph-mon[38143]: osdmap e10: 2 total, 1 up, 2 in 2024-03-14T09:48:03.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:03.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:02 smithi152 ceph-mon[48279]: osdmap e9: 1 total, 1 up, 1 in 2024-03-14T09:48:03.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:02 smithi152 ceph-mon[48279]: from='client.? 172.21.15.132:0/1724409367' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71"}]: dispatch 2024-03-14T09:48:03.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:02 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71"}]: dispatch 2024-03-14T09:48:03.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:02 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71"}]': finished 2024-03-14T09:48:03.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:02 smithi152 ceph-mon[48279]: osdmap e10: 2 total, 1 up, 2 in 2024-03-14T09:48:03.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:02 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:03.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:02 smithi132 ceph-mon[47959]: osdmap e9: 1 total, 1 up, 1 in 2024-03-14T09:48:03.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:02 smithi132 ceph-mon[47959]: from='client.? 172.21.15.132:0/1724409367' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71"}]: dispatch 2024-03-14T09:48:03.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:02 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71"}]: dispatch 2024-03-14T09:48:03.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:02 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71"}]': finished 2024-03-14T09:48:03.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:02 smithi132 ceph-mon[47959]: osdmap e10: 2 total, 1 up, 2 in 2024-03-14T09:48:03.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:02 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:04.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:03 smithi078 ceph-mon[38143]: from='client.? 172.21.15.132:0/2385264820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:48:04.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:03 smithi078 ceph-mon[38143]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:04.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:03 smithi152 ceph-mon[48279]: from='client.? 172.21.15.132:0/2385264820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:48:04.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:03 smithi152 ceph-mon[48279]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:04.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:03 smithi132 ceph-mon[47959]: from='client.? 172.21.15.132:0/2385264820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:48:04.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:03 smithi132 ceph-mon[47959]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:06.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:06 smithi132 ceph-mon[47959]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:06.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:06 smithi152 ceph-mon[48279]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:06.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:06 smithi078 ceph-mon[38143]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:08.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:08 smithi132 ceph-mon[47959]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:08.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:08 smithi152 ceph-mon[48279]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:08.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:08 smithi078 ceph-mon[38143]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:09.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:09 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-14T09:48:09.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:09 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:09.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:09 smithi132 ceph-mon[47959]: Deploying daemon osd.1 on smithi132 2024-03-14T09:48:09.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:09 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-14T09:48:09.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:09 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:09.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:09 smithi152 ceph-mon[48279]: Deploying daemon osd.1 on smithi132 2024-03-14T09:48:09.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:09 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-14T09:48:09.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:09 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:09.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:09 smithi078 ceph-mon[38143]: Deploying daemon osd.1 on smithi132 2024-03-14T09:48:10.133 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:10 smithi132 ceph-mon[47959]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:10.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:10 smithi152 ceph-mon[48279]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:10.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:10 smithi078 ceph-mon[38143]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:12.423 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:12 smithi132 ceph-mon[47959]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:12.423 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:12 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:12.423 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:12 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:12.423 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:12 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:12.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:12 smithi152 ceph-mon[48279]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:12.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:12 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:12.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:12 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:12.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:12 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:12.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:12 smithi078 ceph-mon[38143]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:12.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:12 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:12.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:12 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:12.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:12 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:14.113 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 1 on host 'smithi132' 2024-03-14T09:48:14.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:14.112+0000 7f28dcff9700 1 -- 172.21.15.132:0/1816100014 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f28e00fd3c0 con 0x7f28c80a0a80 2024-03-14T09:48:14.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:14.114+0000 7f28e623e700 1 -- 172.21.15.132:0/1816100014 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f28c80a0a80 msgr2=0x7f28c80a2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:14.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:14.114+0000 7f28e623e700 1 --2- 172.21.15.132:0/1816100014 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f28c80a0a80 0x7f28c80a2f40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f28d4005fd0 tx=0x7f28d4005f00 comp rx=0 tx=0).stop 2024-03-14T09:48:14.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:14.114+0000 7f28e623e700 1 -- 172.21.15.132:0/1816100014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28e0101870 msgr2=0x7f28e019e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:14.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:14.114+0000 7f28e623e700 1 --2- 172.21.15.132:0/1816100014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28e0101870 0x7f28e019e6c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f28d8007bd0 tx=0x7f28d80094f0 comp rx=0 tx=0).stop 2024-03-14T09:48:14.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:14.114+0000 7f28e623e700 1 -- 172.21.15.132:0/1816100014 shutdown_connections 2024-03-14T09:48:14.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:14.114+0000 7f28e623e700 1 --2- 172.21.15.132:0/1816100014 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f28c80a0a80 0x7f28c80a2f40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:14.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:14.114+0000 7f28e623e700 1 --2- 172.21.15.132:0/1816100014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28e0101870 0x7f28e019e6c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:14.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:14.114+0000 7f28e623e700 1 --2- 172.21.15.132:0/1816100014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28e00ffcd0 0x7f28e0197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:14.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:14.114+0000 7f28e623e700 1 --2- 172.21.15.132:0/1816100014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28e0100ed0 0x7f28e0197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:14.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:14.114+0000 7f28e623e700 1 -- 172.21.15.132:0/1816100014 >> 172.21.15.132:0/1816100014 conn(0x7f28e00fb270 msgr2=0x7f28e00fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:48:14.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:14.114+0000 7f28e623e700 1 -- 172.21.15.132:0/1816100014 shutdown_connections 2024-03-14T09:48:14.115 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:14.114+0000 7f28e623e700 1 -- 172.21.15.132:0/1816100014 wait complete. 2024-03-14T09:48:14.289 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:14 smithi132 ceph-mon[47959]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:14.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:14 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:14.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:14 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:14.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:14 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:14.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:14 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:14.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:14 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:14.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:14 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:14.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:14 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:14.290 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:14 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:14.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:14 smithi152 ceph-mon[48279]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:14.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:14 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:14.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:14 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:14.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:14 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:14.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:14 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:14.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:14 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:14.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:14 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:14.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:14 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:14.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:14 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:14.507 DEBUG:teuthology.orchestra.run.smithi132:osd.1> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.1.service 2024-03-14T09:48:14.509 INFO:tasks.cephadm:Deploying osd.2 on smithi152 with /dev/vg_nvme/lv_4... 2024-03-14T09:48:14.509 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-14T09:48:14.538 INFO:journalctl@ceph.osd.1.smithi132.stdout:-- Logs begin at Thu 2024-03-14 09:24:09 UTC. -- 2024-03-14T09:48:14.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:14 smithi078 ceph-mon[38143]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:14.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:14 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:14.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:14 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:14.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:14 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:14.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:14 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:14.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:14 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:14.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:14 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:14.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:14 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:14.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:14 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:14.733 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.c/config 2024-03-14T09:48:16.053 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-14T09:48:16.053 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-14T09:48:16.053 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-14T09:48:16.053 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-14T09:48:16.053 INFO:teuthology.orchestra.run.smithi152.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0156501 s, 670 MB/s 2024-03-14T09:48:16.053 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-14T09:48:16.109 INFO:journalctl@ceph.osd.1.smithi132.stdout:Mar 14 09:48:15 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-1[52934]: 2024-03-14T09:48:15.816+0000 7fed60cf1640 -1 osd.1 0 log_to_monitors true 2024-03-14T09:48:16.406 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:16 smithi132 ceph-mon[47959]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:16.406 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:16 smithi132 ceph-mon[47959]: from='osd.1 [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-14T09:48:16.406 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:16 smithi132 ceph-mon[47959]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-14T09:48:16.463 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:16 smithi152 ceph-mon[48279]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:16.463 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:16 smithi152 ceph-mon[48279]: from='osd.1 [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-14T09:48:16.463 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:16 smithi152 ceph-mon[48279]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-14T09:48:16.482 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_4 2024-03-14T09:48:16.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:16 smithi078 ceph-mon[38143]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:16.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:16 smithi078 ceph-mon[38143]: from='osd.1 [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-14T09:48:16.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:16 smithi078 ceph-mon[38143]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-14T09:48:16.701 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.c/config 2024-03-14T09:48:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:17 smithi152 ceph-mon[48279]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-14T09:48:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:17 smithi152 ceph-mon[48279]: osdmap e11: 2 total, 1 up, 2 in 2024-03-14T09:48:17.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:17 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:17.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:17 smithi152 ceph-mon[48279]: from='osd.1 [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:48:17.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:17 smithi152 ceph-mon[48279]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:48:17.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:17 smithi152 ceph-mon[48279]: Detected new or changed devices on smithi132 2024-03-14T09:48:17.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:17 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:17.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:17 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:17.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:17 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:48:17.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:17 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi132 to 17114M 2024-03-14T09:48:17.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:17 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:17.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:17 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:17.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:17 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:17.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:17 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:17.520 INFO:journalctl@ceph.osd.1.smithi132.stdout:Mar 14 09:48:17 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-1[52934]: 2024-03-14T09:48:17.121+0000 7fed56355700 -1 osd.1 0 waiting for initial osdmap 2024-03-14T09:48:17.520 INFO:journalctl@ceph.osd.1.smithi132.stdout:Mar 14 09:48:17 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-1[52934]: 2024-03-14T09:48:17.124+0000 7fed50947700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T09:48:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:17 smithi132 ceph-mon[47959]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-14T09:48:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:17 smithi132 ceph-mon[47959]: osdmap e11: 2 total, 1 up, 2 in 2024-03-14T09:48:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:17 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:17 smithi132 ceph-mon[47959]: from='osd.1 [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:48:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:17 smithi132 ceph-mon[47959]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:48:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:17 smithi132 ceph-mon[47959]: Detected new or changed devices on smithi132 2024-03-14T09:48:17.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:17 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:17.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:17 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:17.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:17 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:48:17.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:17 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi132 to 17114M 2024-03-14T09:48:17.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:17 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:17.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:17 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:17.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:17 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:17.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:17 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:17.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:17 smithi078 ceph-mon[38143]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-14T09:48:17.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:17 smithi078 ceph-mon[38143]: osdmap e11: 2 total, 1 up, 2 in 2024-03-14T09:48:17.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:17 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:17 smithi078 ceph-mon[38143]: from='osd.1 [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:48:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:17 smithi078 ceph-mon[38143]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:48:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:17 smithi078 ceph-mon[38143]: Detected new or changed devices on smithi132 2024-03-14T09:48:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:17 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:17 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:17 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:48:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:17 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi132 to 17114M 2024-03-14T09:48:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:17 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:17 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:17 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:17.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:17 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:17.704 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.704+0000 7fc41d863700 1 -- 172.21.15.152:0/1141028688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc418101840 msgr2=0x7fc4181080f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:17.705 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.704+0000 7fc41d863700 1 --2- 172.21.15.152:0/1141028688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc418101840 0x7fc4181080f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc41000b0a0 tx=0x7fc41000b3b0 comp rx=0 tx=0).stop 2024-03-14T09:48:17.705 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.704+0000 7fc41d863700 1 -- 172.21.15.152:0/1141028688 shutdown_connections 2024-03-14T09:48:17.705 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.704+0000 7fc41d863700 1 --2- 172.21.15.152:0/1141028688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc418101840 0x7fc4181080f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:17.705 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.704+0000 7fc41d863700 1 --2- 172.21.15.152:0/1141028688 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4180ffca0 0x7fc4181000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:17.705 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.704+0000 7fc41d863700 1 --2- 172.21.15.152:0/1141028688 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc418100ea0 0x7fc418101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:17.705 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.704+0000 7fc41d863700 1 -- 172.21.15.152:0/1141028688 >> 172.21.15.152:0/1141028688 conn(0x7fc4180fb240 msgr2=0x7fc4180fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:48:17.705 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.704+0000 7fc41d863700 1 -- 172.21.15.152:0/1141028688 shutdown_connections 2024-03-14T09:48:17.705 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.704+0000 7fc41d863700 1 -- 172.21.15.152:0/1141028688 wait complete. 2024-03-14T09:48:17.707 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.704+0000 7fc41d863700 1 Processor -- start 2024-03-14T09:48:17.707 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc41d863700 1 -- start start 2024-03-14T09:48:17.707 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc41d863700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4180ffca0 0x7fc418069f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:48:17.707 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc41d863700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc418100ea0 0x7fc41819bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:48:17.707 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc41d863700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc418101840 0x7fc41819e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:48:17.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc41d863700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41819ed00 con 0x7fc4180ffca0 2024-03-14T09:48:17.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc41d863700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41819ee70 con 0x7fc418100ea0 2024-03-14T09:48:17.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc41d863700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41819f030 con 0x7fc418101840 2024-03-14T09:48:17.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc4177fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc418101840 0x7fc41819e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:48:17.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc4177fe700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc418101840 0x7fc41819e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:38946/0 (socket says 172.21.15.152:38946) 2024-03-14T09:48:17.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc4177fe700 1 -- 172.21.15.152:0/350469511 learned_addr learned my addr 172.21.15.152:0/350469511 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-14T09:48:17.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc4167fc700 1 --2- 172.21.15.152:0/350469511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc418100ea0 0x7fc41819bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:48:17.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc416ffd700 1 --2- 172.21.15.152:0/350469511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4180ffca0 0x7fc418069f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:48:17.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc4177fe700 1 -- 172.21.15.152:0/350469511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc418100ea0 msgr2=0x7fc41819bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:17.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc4177fe700 1 --2- 172.21.15.152:0/350469511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc418100ea0 0x7fc41819bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:17.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc4177fe700 1 -- 172.21.15.152:0/350469511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4180ffca0 msgr2=0x7fc418069f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:17.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc4177fe700 1 --2- 172.21.15.152:0/350469511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4180ffca0 0x7fc418069f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:17.708 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc4177fe700 1 -- 172.21.15.152:0/350469511 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc410009d00 con 0x7fc418101840 2024-03-14T09:48:17.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc416ffd700 1 --2- 172.21.15.152:0/350469511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4180ffca0 0x7fc418069f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:48:17.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc4177fe700 1 --2- 172.21.15.152:0/350469511 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc418101840 0x7fc41819e560 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc410004d00 tx=0x7fc410004d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:48:17.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc41c861700 1 -- 172.21.15.152:0/350469511 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41000e040 con 0x7fc418101840 2024-03-14T09:48:17.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc41c861700 1 -- 172.21.15.152:0/350469511 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc41000ce50 con 0x7fc418101840 2024-03-14T09:48:17.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc41d863700 1 -- 172.21.15.152:0/350469511 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41819f2b0 con 0x7fc418101840 2024-03-14T09:48:17.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc41c861700 1 -- 172.21.15.152:0/350469511 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41001bb20 con 0x7fc418101840 2024-03-14T09:48:17.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.705+0000 7fc41d863700 1 -- 172.21.15.152:0/350469511 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41819f720 con 0x7fc418101840 2024-03-14T09:48:17.709 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.707+0000 7fc41d863700 1 -- 172.21.15.152:0/350469511 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc418107060 con 0x7fc418101840 2024-03-14T09:48:17.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.707+0000 7fc41c861700 1 -- 172.21.15.152:0/350469511 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 15) v1 ==== 135332+0+0 (secure 0 0 0) 0x7fc41001bc80 con 0x7fc418101840 2024-03-14T09:48:17.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.708+0000 7fc41c861700 1 --2- 172.21.15.152:0/350469511 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc4000a0a00 0x7fc4000a2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:48:17.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.708+0000 7fc41c861700 1 -- 172.21.15.152:0/350469511 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2029+0+0 (secure 0 0 0) 0x7fc4100cc420 con 0x7fc418101840 2024-03-14T09:48:17.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.709+0000 7fc416ffd700 1 --2- 172.21.15.152:0/350469511 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc4000a0a00 0x7fc4000a2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:48:17.710 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.709+0000 7fc416ffd700 1 --2- 172.21.15.152:0/350469511 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc4000a0a00 0x7fc4000a2ec0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc418100d00 tx=0x7fc40c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:48:17.713 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.713+0000 7fc41c861700 1 -- 172.21.15.152:0/350469511 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc4100977f0 con 0x7fc418101840 2024-03-14T09:48:17.911 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:17.910+0000 7fc41d863700 1 -- 172.21.15.152:0/350469511 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4180611f0 con 0x7fc4000a0a00 2024-03-14T09:48:18.384 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:18 smithi152 ceph-mon[48279]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:18.384 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:18 smithi152 ceph-mon[48279]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-14T09:48:18.384 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:18 smithi152 ceph-mon[48279]: osdmap e12: 2 total, 1 up, 2 in 2024-03-14T09:48:18.384 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:18.384 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:18.384 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:48:18.384 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:48:18.384 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:18 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:18.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:18 smithi132 ceph-mon[47959]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:18.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:18 smithi132 ceph-mon[47959]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-14T09:48:18.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:18 smithi132 ceph-mon[47959]: osdmap e12: 2 total, 1 up, 2 in 2024-03-14T09:48:18.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:18 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:18.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:18 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:18.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:18 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:48:18.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:18 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:48:18.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:18 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:18.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:18 smithi078 ceph-mon[38143]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:18.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:18 smithi078 ceph-mon[38143]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-14T09:48:18.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:18 smithi078 ceph-mon[38143]: osdmap e12: 2 total, 1 up, 2 in 2024-03-14T09:48:18.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:18.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:18.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:48:18.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:48:18.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:18 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:19.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:19 smithi152 ceph-mon[48279]: purged_snaps scrub starts 2024-03-14T09:48:19.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:19 smithi152 ceph-mon[48279]: purged_snaps scrub ok 2024-03-14T09:48:19.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:19 smithi152 ceph-mon[48279]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:48:19.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:19 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:19.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:19 smithi152 ceph-mon[48279]: osd.1 [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960] boot 2024-03-14T09:48:19.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:19 smithi152 ceph-mon[48279]: osdmap e13: 2 total, 2 up, 2 in 2024-03-14T09:48:19.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:19 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:19.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:19 smithi132 ceph-mon[47959]: purged_snaps scrub starts 2024-03-14T09:48:19.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:19 smithi132 ceph-mon[47959]: purged_snaps scrub ok 2024-03-14T09:48:19.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:19 smithi132 ceph-mon[47959]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:48:19.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:19 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:19.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:19 smithi132 ceph-mon[47959]: osd.1 [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960] boot 2024-03-14T09:48:19.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:19 smithi132 ceph-mon[47959]: osdmap e13: 2 total, 2 up, 2 in 2024-03-14T09:48:19.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:19 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:19.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:19 smithi078 ceph-mon[38143]: purged_snaps scrub starts 2024-03-14T09:48:19.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:19 smithi078 ceph-mon[38143]: purged_snaps scrub ok 2024-03-14T09:48:19.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:19 smithi078 ceph-mon[38143]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:48:19.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:19 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:19.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:19 smithi078 ceph-mon[38143]: osd.1 [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960] boot 2024-03-14T09:48:19.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:19 smithi078 ceph-mon[38143]: osdmap e13: 2 total, 2 up, 2 in 2024-03-14T09:48:19.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:19 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:48:20.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:20 smithi152 ceph-mon[48279]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:20.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:20 smithi152 ceph-mon[48279]: osdmap e14: 2 total, 2 up, 2 in 2024-03-14T09:48:20.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:20 smithi152 ceph-mon[48279]: from='client.? 172.21.15.152:0/1374399423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "410edac5-366f-4f37-aaf3-75383d995b28"}]: dispatch 2024-03-14T09:48:20.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:20 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "410edac5-366f-4f37-aaf3-75383d995b28"}]: dispatch 2024-03-14T09:48:20.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:20 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "410edac5-366f-4f37-aaf3-75383d995b28"}]': finished 2024-03-14T09:48:20.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:20 smithi152 ceph-mon[48279]: osdmap e15: 3 total, 2 up, 3 in 2024-03-14T09:48:20.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:20 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:48:20.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:20 smithi132 ceph-mon[47959]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:20.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:20 smithi132 ceph-mon[47959]: osdmap e14: 2 total, 2 up, 2 in 2024-03-14T09:48:20.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:20 smithi132 ceph-mon[47959]: from='client.? 172.21.15.152:0/1374399423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "410edac5-366f-4f37-aaf3-75383d995b28"}]: dispatch 2024-03-14T09:48:20.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:20 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "410edac5-366f-4f37-aaf3-75383d995b28"}]: dispatch 2024-03-14T09:48:20.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:20 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "410edac5-366f-4f37-aaf3-75383d995b28"}]': finished 2024-03-14T09:48:20.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:20 smithi132 ceph-mon[47959]: osdmap e15: 3 total, 2 up, 3 in 2024-03-14T09:48:20.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:20 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:48:20.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:20 smithi078 ceph-mon[38143]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-14T09:48:20.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:20 smithi078 ceph-mon[38143]: osdmap e14: 2 total, 2 up, 2 in 2024-03-14T09:48:20.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:20 smithi078 ceph-mon[38143]: from='client.? 172.21.15.152:0/1374399423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "410edac5-366f-4f37-aaf3-75383d995b28"}]: dispatch 2024-03-14T09:48:20.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:20 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "410edac5-366f-4f37-aaf3-75383d995b28"}]: dispatch 2024-03-14T09:48:20.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:20 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "410edac5-366f-4f37-aaf3-75383d995b28"}]': finished 2024-03-14T09:48:20.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:20 smithi078 ceph-mon[38143]: osdmap e15: 3 total, 2 up, 3 in 2024-03-14T09:48:20.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:20 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:48:21.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:21 smithi152 ceph-mon[48279]: from='client.? 172.21.15.152:0/3520960446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:48:21.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:21 smithi132 ceph-mon[47959]: from='client.? 172.21.15.152:0/3520960446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:48:21.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:21 smithi078 ceph-mon[38143]: from='client.? 172.21.15.152:0/3520960446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:48:22.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:22 smithi152 ceph-mon[48279]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:22.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:22 smithi132 ceph-mon[47959]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:22.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:22 smithi078 ceph-mon[38143]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:23.804 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:23 smithi152 ceph-mon[48279]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:24.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:23 smithi132 ceph-mon[47959]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:24.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:23 smithi078 ceph-mon[38143]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:26.444 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:26 smithi152 ceph-mon[48279]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:26.444 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-14T09:48:26.444 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:26.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:26 smithi132 ceph-mon[47959]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:26.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:26 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-14T09:48:26.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:26 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:26.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:26 smithi078 ceph-mon[38143]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:26.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-14T09:48:26.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:27.429 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:27 smithi152 ceph-mon[48279]: Deploying daemon osd.2 on smithi152 2024-03-14T09:48:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:27 smithi132 ceph-mon[47959]: Deploying daemon osd.2 on smithi152 2024-03-14T09:48:27.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:27 smithi078 ceph-mon[38143]: Deploying daemon osd.2 on smithi152 2024-03-14T09:48:28.428 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:28 smithi152 ceph-mon[48279]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:28.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:28 smithi132 ceph-mon[47959]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:28.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:28 smithi078 ceph-mon[38143]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:30.347 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:30 smithi152 ceph-mon[48279]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:30.348 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:30 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:30.348 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:30 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:30.348 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:30 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:30.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:30 smithi132 ceph-mon[47959]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:30.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:30 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:30.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:30 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:30.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:30 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:30.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:30 smithi078 ceph-mon[38143]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:30.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:30 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:30.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:30 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:30.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:30 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:32.068 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:32.067+0000 7fc41c861700 1 -- 172.21.15.152:0/350469511 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc4180611f0 con 0x7fc4000a0a00 2024-03-14T09:48:32.068 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 2 on host 'smithi152' 2024-03-14T09:48:32.070 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:32.069+0000 7fc41d863700 1 -- 172.21.15.152:0/350469511 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc4000a0a00 msgr2=0x7fc4000a2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:32.070 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:32.069+0000 7fc41d863700 1 --2- 172.21.15.152:0/350469511 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc4000a0a00 0x7fc4000a2ec0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc418100d00 tx=0x7fc40c008040 comp rx=0 tx=0).stop 2024-03-14T09:48:32.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:32.069+0000 7fc41d863700 1 -- 172.21.15.152:0/350469511 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc418101840 msgr2=0x7fc41819e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:32.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:32.069+0000 7fc41d863700 1 --2- 172.21.15.152:0/350469511 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc418101840 0x7fc41819e560 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc410004d00 tx=0x7fc410004d30 comp rx=0 tx=0).stop 2024-03-14T09:48:32.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:32.069+0000 7fc41d863700 1 -- 172.21.15.152:0/350469511 shutdown_connections 2024-03-14T09:48:32.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:32.069+0000 7fc41d863700 1 --2- 172.21.15.152:0/350469511 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc418101840 0x7fc41819e560 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:32.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:32.069+0000 7fc41d863700 1 --2- 172.21.15.152:0/350469511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4180ffca0 0x7fc418069f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:32.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:32.069+0000 7fc41d863700 1 --2- 172.21.15.152:0/350469511 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc4000a0a00 0x7fc4000a2ec0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:32.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:32.069+0000 7fc41d863700 1 --2- 172.21.15.152:0/350469511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc418100ea0 0x7fc41819bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:32.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:32.069+0000 7fc41d863700 1 -- 172.21.15.152:0/350469511 >> 172.21.15.152:0/350469511 conn(0x7fc4180fb240 msgr2=0x7fc4181041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:48:32.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:32.070+0000 7fc41d863700 1 -- 172.21.15.152:0/350469511 shutdown_connections 2024-03-14T09:48:32.071 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:48:32.070+0000 7fc41d863700 1 -- 172.21.15.152:0/350469511 wait complete. 2024-03-14T09:48:32.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:32 smithi152 ceph-mon[48279]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:32.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:32 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:32.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:32 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:32.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:32 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:32.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:32 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:32.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:32 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:32.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:32 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:32.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:32 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:32.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:32 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:32.486 DEBUG:teuthology.orchestra.run.smithi152:osd.2> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.2.service 2024-03-14T09:48:32.488 INFO:tasks.cephadm:Deploying osd.3 on smithi078 with /dev/vg_nvme/lv_3... 2024-03-14T09:48:32.488 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-14T09:48:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:32 smithi132 ceph-mon[47959]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:32 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:32 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:32 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:32 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:32 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:32 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:32 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:32 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:32.521 INFO:journalctl@ceph.osd.2.smithi152.stdout:-- Logs begin at Thu 2024-03-14 09:24:45 UTC. -- 2024-03-14T09:48:32.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:32 smithi078 ceph-mon[38143]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:32.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:32 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:32.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:32 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:32.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:32 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:32.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:32 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:32.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:32 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:32.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:32 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:32.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:32 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:32.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:32 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:32.856 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:48:33.777 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:33 smithi078 ceph-mon[38143]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:34.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:33 smithi152 ceph-mon[48279]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:34.000 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 14 09:48:33 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-2[54057]: 2024-03-14T09:48:33.669+0000 7f74a874a640 -1 osd.2 0 log_to_monitors true 2024-03-14T09:48:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:33 smithi132 ceph-mon[47959]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:34.570 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-14T09:48:34.570 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-14T09:48:34.570 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2024-03-14T09:48:34.570 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2024-03-14T09:48:34.570 INFO:teuthology.orchestra.run.smithi078.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0296868 s, 353 MB/s 2024-03-14T09:48:34.571 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2024-03-14T09:48:34.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:34 smithi152 ceph-mon[48279]: from='osd.2 [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-14T09:48:34.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:34 smithi152 ceph-mon[48279]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-14T09:48:34.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:34 smithi152 ceph-mon[48279]: Detected new or changed devices on smithi152 2024-03-14T09:48:34.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:34.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:34.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:48:34.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:34 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi152 to 17114M 2024-03-14T09:48:34.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:34.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:34.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:34.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:34.913 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:34 smithi078 ceph-mon[38143]: from='osd.2 [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-14T09:48:34.913 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:34 smithi078 ceph-mon[38143]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-14T09:48:34.913 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:34 smithi078 ceph-mon[38143]: Detected new or changed devices on smithi152 2024-03-14T09:48:34.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:34.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:34.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:48:34.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:34 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi152 to 17114M 2024-03-14T09:48:34.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:34.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:34.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:34.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:34.941 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_3 2024-03-14T09:48:35.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:34 smithi132 ceph-mon[47959]: from='osd.2 [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-14T09:48:35.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:34 smithi132 ceph-mon[47959]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-14T09:48:35.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:34 smithi132 ceph-mon[47959]: Detected new or changed devices on smithi152 2024-03-14T09:48:35.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:35.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:35.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:48:35.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:34 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi152 to 17114M 2024-03-14T09:48:35.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:35.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:35.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:35.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:35.289 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:48:35.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:35 smithi078 ceph-mon[38143]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-14T09:48:35.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:35 smithi078 ceph-mon[38143]: osdmap e16: 3 total, 2 up, 3 in 2024-03-14T09:48:35.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:35 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:48:35.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:35 smithi078 ceph-mon[38143]: from='osd.2 [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:48:35.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:35 smithi078 ceph-mon[38143]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:48:35.788 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:35 smithi078 ceph-mon[38143]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:35.999 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 14 09:48:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-2[54057]: 2024-03-14T09:48:35.651+0000 7f749ddae700 -1 osd.2 0 waiting for initial osdmap 2024-03-14T09:48:35.999 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 14 09:48:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-2[54057]: 2024-03-14T09:48:35.655+0000 7f7496b9d700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T09:48:35.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:35 smithi152 ceph-mon[48279]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-14T09:48:35.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:35 smithi152 ceph-mon[48279]: osdmap e16: 3 total, 2 up, 3 in 2024-03-14T09:48:35.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:35 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:48:36.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:35 smithi152 ceph-mon[48279]: from='osd.2 [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:48:36.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:35 smithi152 ceph-mon[48279]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:48:36.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:35 smithi152 ceph-mon[48279]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:36.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:35 smithi132 ceph-mon[47959]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-14T09:48:36.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:35 smithi132 ceph-mon[47959]: osdmap e16: 3 total, 2 up, 3 in 2024-03-14T09:48:36.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:35 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:48:36.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:35 smithi132 ceph-mon[47959]: from='osd.2 [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:48:36.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:35 smithi132 ceph-mon[47959]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:48:36.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:35 smithi132 ceph-mon[47959]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:36.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.412+0000 7fc27d400700 1 -- 172.21.15.78:0/2379559438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc278100bf0 msgr2=0x7fc278101010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:36.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.412+0000 7fc27d400700 1 --2- 172.21.15.78:0/2379559438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc278100bf0 0x7fc278101010 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc2680099c0 tx=0x7fc268009cd0 comp rx=0 tx=0).stop 2024-03-14T09:48:36.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.412+0000 7fc27d400700 1 -- 172.21.15.78:0/2379559438 shutdown_connections 2024-03-14T09:48:36.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.412+0000 7fc27d400700 1 --2- 172.21.15.78:0/2379559438 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc278102790 0x7fc278109040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:36.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.412+0000 7fc27d400700 1 --2- 172.21.15.78:0/2379559438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc278100bf0 0x7fc278101010 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:36.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.412+0000 7fc27d400700 1 --2- 172.21.15.78:0/2379559438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc278101df0 0x7fc278102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:36.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.412+0000 7fc27d400700 1 -- 172.21.15.78:0/2379559438 >> 172.21.15.78:0/2379559438 conn(0x7fc2780fc190 msgr2=0x7fc2780fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:48:36.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.412+0000 7fc27d400700 1 -- 172.21.15.78:0/2379559438 shutdown_connections 2024-03-14T09:48:36.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.412+0000 7fc27d400700 1 -- 172.21.15.78:0/2379559438 wait complete. 2024-03-14T09:48:36.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.413+0000 7fc27d400700 1 Processor -- start 2024-03-14T09:48:36.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.413+0000 7fc27d400700 1 -- start start 2024-03-14T09:48:36.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.413+0000 7fc27d400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc278100bf0 0x7fc278199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:48:36.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.413+0000 7fc27d400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc278101df0 0x7fc278199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:48:36.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.413+0000 7fc27d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc278102790 0x7fc2781a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:48:36.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.413+0000 7fc27d400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2781a0c90 con 0x7fc278101df0 2024-03-14T09:48:36.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.413+0000 7fc27d400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2781a0dd0 con 0x7fc278102790 2024-03-14T09:48:36.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.413+0000 7fc27d400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2781a0f60 con 0x7fc278100bf0 2024-03-14T09:48:36.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.413+0000 7fc277400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc278101df0 0x7fc278199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:48:36.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.413+0000 7fc277e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc278100bf0 0x7fc278199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:48:36.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.413+0000 7fc277400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc278101df0 0x7fc278199690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41108/0 (socket says 172.21.15.78:41108) 2024-03-14T09:48:36.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.414+0000 7fc277400700 1 -- 172.21.15.78:0/2740551997 learned_addr learned my addr 172.21.15.78:0/2740551997 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:48:36.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.414+0000 7fc27ca00700 1 --2- 172.21.15.78:0/2740551997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc278102790 0x7fc2781a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:48:36.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.414+0000 7fc277400700 1 -- 172.21.15.78:0/2740551997 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc278100bf0 msgr2=0x7fc278199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:36.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.414+0000 7fc277400700 1 --2- 172.21.15.78:0/2740551997 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc278100bf0 0x7fc278199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:36.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.414+0000 7fc277400700 1 -- 172.21.15.78:0/2740551997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc278102790 msgr2=0x7fc2781a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:36.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.414+0000 7fc277400700 1 --2- 172.21.15.78:0/2740551997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc278102790 0x7fc2781a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:36.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.414+0000 7fc277400700 1 -- 172.21.15.78:0/2740551997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2680095f0 con 0x7fc278101df0 2024-03-14T09:48:36.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.414+0000 7fc277400700 1 --2- 172.21.15.78:0/2740551997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc278101df0 0x7fc278199690 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc26400e9d0 tx=0x7fc26400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:48:36.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.414+0000 7fc274c00700 1 -- 172.21.15.78:0/2740551997 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc264009790 con 0x7fc278101df0 2024-03-14T09:48:36.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.414+0000 7fc274c00700 1 -- 172.21.15.78:0/2740551997 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc26400ee70 con 0x7fc278101df0 2024-03-14T09:48:36.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.414+0000 7fc274c00700 1 -- 172.21.15.78:0/2740551997 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc264010ea0 con 0x7fc278101df0 2024-03-14T09:48:36.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.414+0000 7fc27d400700 1 -- 172.21.15.78:0/2740551997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2781a1240 con 0x7fc278101df0 2024-03-14T09:48:36.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.415+0000 7fc27d400700 1 -- 172.21.15.78:0/2740551997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2781a16f0 con 0x7fc278101df0 2024-03-14T09:48:36.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.416+0000 7fc27d400700 1 -- 172.21.15.78:0/2740551997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc27804ea80 con 0x7fc278101df0 2024-03-14T09:48:36.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.417+0000 7fc274c00700 1 -- 172.21.15.78:0/2740551997 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 15) v1 ==== 135332+0+0 (secure 0 0 0) 0x7fc26400cbd0 con 0x7fc278101df0 2024-03-14T09:48:36.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.419+0000 7fc274c00700 1 --2- 172.21.15.78:0/2740551997 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc25c0a0a80 0x7fc25c0a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:48:36.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.420+0000 7fc274c00700 1 -- 172.21.15.78:0/2740551997 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 2571+0+0 (secure 0 0 0) 0x7fc264014070 con 0x7fc278101df0 2024-03-14T09:48:36.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.420+0000 7fc277e00700 1 --2- 172.21.15.78:0/2740551997 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc25c0a0a80 0x7fc25c0a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:48:36.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.420+0000 7fc277e00700 1 --2- 172.21.15.78:0/2740551997 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc25c0a0a80 0x7fc25c0a2f40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc268009990 tx=0x7fc268000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:48:36.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.424+0000 7fc274c00700 1 -- 172.21.15.78:0/2740551997 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc264052960 con 0x7fc278101df0 2024-03-14T09:48:36.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:36.635+0000 7fc27d400700 1 -- 172.21.15.78:0/2740551997 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc278108ce0 con 0x7fc25c0a0a80 2024-03-14T09:48:36.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:36 smithi152 ceph-mon[48279]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-14T09:48:36.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:36 smithi152 ceph-mon[48279]: osdmap e17: 3 total, 2 up, 3 in 2024-03-14T09:48:36.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:36 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:48:36.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:36 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:48:36.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:36 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:48:36.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:36 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:48:36.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:36 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:37.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:36 smithi132 ceph-mon[47959]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-14T09:48:37.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:36 smithi132 ceph-mon[47959]: osdmap e17: 3 total, 2 up, 3 in 2024-03-14T09:48:37.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:36 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:48:37.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:36 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:48:37.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:36 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:48:37.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:36 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:48:37.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:36 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:37.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:36 smithi078 ceph-mon[38143]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-14T09:48:37.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:36 smithi078 ceph-mon[38143]: osdmap e17: 3 total, 2 up, 3 in 2024-03-14T09:48:37.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:36 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:48:37.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:36 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:48:37.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:36 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:48:37.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:36 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:48:37.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:36 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:37.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:37 smithi078 ceph-mon[38143]: purged_snaps scrub starts 2024-03-14T09:48:37.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:37 smithi078 ceph-mon[38143]: purged_snaps scrub ok 2024-03-14T09:48:37.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:37 smithi078 ceph-mon[38143]: from='client.14259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:48:37.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:37 smithi078 ceph-mon[38143]: osd.2 [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535] boot 2024-03-14T09:48:37.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:37 smithi078 ceph-mon[38143]: osdmap e18: 3 total, 3 up, 3 in 2024-03-14T09:48:37.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:37 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:48:37.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:37 smithi078 ceph-mon[38143]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:37.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:37 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-14T09:48:37.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:37 smithi152 ceph-mon[48279]: purged_snaps scrub starts 2024-03-14T09:48:37.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:37 smithi152 ceph-mon[48279]: purged_snaps scrub ok 2024-03-14T09:48:37.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:37 smithi152 ceph-mon[48279]: from='client.14259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:48:37.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:37 smithi152 ceph-mon[48279]: osd.2 [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535] boot 2024-03-14T09:48:37.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:37 smithi152 ceph-mon[48279]: osdmap e18: 3 total, 3 up, 3 in 2024-03-14T09:48:37.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:37 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:48:37.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:37 smithi152 ceph-mon[48279]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:37.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:37 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-14T09:48:38.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:37 smithi132 ceph-mon[47959]: purged_snaps scrub starts 2024-03-14T09:48:38.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:37 smithi132 ceph-mon[47959]: purged_snaps scrub ok 2024-03-14T09:48:38.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:37 smithi132 ceph-mon[47959]: from='client.14259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:48:38.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:37 smithi132 ceph-mon[47959]: osd.2 [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535] boot 2024-03-14T09:48:38.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:37 smithi132 ceph-mon[47959]: osdmap e18: 3 total, 3 up, 3 in 2024-03-14T09:48:38.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:37 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:48:38.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:37 smithi132 ceph-mon[47959]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-14T09:48:38.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:37 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-14T09:48:38.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:38 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-14T09:48:38.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:38 smithi152 ceph-mon[48279]: osdmap e19: 3 total, 3 up, 3 in 2024-03-14T09:48:38.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:38 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-14T09:48:39.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:38 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-14T09:48:39.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:38 smithi132 ceph-mon[47959]: osdmap e19: 3 total, 3 up, 3 in 2024-03-14T09:48:39.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:38 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-14T09:48:39.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:38 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-14T09:48:39.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:38 smithi078 ceph-mon[38143]: osdmap e19: 3 total, 3 up, 3 in 2024-03-14T09:48:39.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:38 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-14T09:48:39.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:39 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-14T09:48:39.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:39 smithi152 ceph-mon[48279]: osdmap e20: 3 total, 3 up, 3 in 2024-03-14T09:48:39.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:39 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1655108017' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f480434-66c7-4aba-bea5-e74d94e480d8"}]: dispatch 2024-03-14T09:48:39.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:39 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1655108017' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f480434-66c7-4aba-bea5-e74d94e480d8"}]': finished 2024-03-14T09:48:39.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:39 smithi152 ceph-mon[48279]: osdmap e21: 4 total, 3 up, 4 in 2024-03-14T09:48:39.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:39 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:48:39.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:39 smithi152 ceph-mon[48279]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-14T09:48:40.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:39 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-14T09:48:40.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:39 smithi132 ceph-mon[47959]: osdmap e20: 3 total, 3 up, 3 in 2024-03-14T09:48:40.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:39 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1655108017' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f480434-66c7-4aba-bea5-e74d94e480d8"}]: dispatch 2024-03-14T09:48:40.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:39 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1655108017' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f480434-66c7-4aba-bea5-e74d94e480d8"}]': finished 2024-03-14T09:48:40.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:39 smithi132 ceph-mon[47959]: osdmap e21: 4 total, 3 up, 4 in 2024-03-14T09:48:40.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:39 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:48:40.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:39 smithi132 ceph-mon[47959]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-14T09:48:40.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:39 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-14T09:48:40.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:39 smithi078 ceph-mon[38143]: osdmap e20: 3 total, 3 up, 3 in 2024-03-14T09:48:40.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:39 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1655108017' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f480434-66c7-4aba-bea5-e74d94e480d8"}]: dispatch 2024-03-14T09:48:40.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:39 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1655108017' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f480434-66c7-4aba-bea5-e74d94e480d8"}]': finished 2024-03-14T09:48:40.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:39 smithi078 ceph-mon[38143]: osdmap e21: 4 total, 3 up, 4 in 2024-03-14T09:48:40.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:39 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:48:40.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:39 smithi078 ceph-mon[38143]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-14T09:48:40.055 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 14 09:48:39 smithi078 sudo[61979]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-14T09:48:40.055 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 14 09:48:39 smithi078 sudo[61979]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T09:48:40.055 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 14 09:48:39 smithi078 sudo[61979]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T09:48:40.055 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 14 09:48:39 smithi078 sudo[61979]: pam_unix(sudo:session): session closed for user root 2024-03-14T09:48:40.553 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 14 09:48:40 smithi078 sudo[62031]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-14T09:48:40.553 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 14 09:48:40 smithi078 sudo[62031]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T09:48:40.553 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 14 09:48:40 smithi078 sudo[62031]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T09:48:40.553 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 14 09:48:40 smithi078 sudo[62031]: pam_unix(sudo:session): session closed for user root 2024-03-14T09:48:40.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:40 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2824528264' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:48:40.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:40 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:40.771 INFO:journalctl@ceph.osd.1.smithi132.stdout:Mar 14 09:48:40 smithi132 sudo[58532]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-14T09:48:40.771 INFO:journalctl@ceph.osd.1.smithi132.stdout:Mar 14 09:48:40 smithi132 sudo[58532]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T09:48:40.771 INFO:journalctl@ceph.osd.1.smithi132.stdout:Mar 14 09:48:40 smithi132 sudo[58532]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T09:48:40.771 INFO:journalctl@ceph.osd.1.smithi132.stdout:Mar 14 09:48:40 smithi132 sudo[58532]: pam_unix(sudo:session): session closed for user root 2024-03-14T09:48:40.771 INFO:journalctl@ceph.osd.1.smithi132.stdout:Mar 14 09:48:40 smithi132 sudo[58535]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-14T09:48:40.771 INFO:journalctl@ceph.osd.1.smithi132.stdout:Mar 14 09:48:40 smithi132 sudo[58535]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T09:48:40.771 INFO:journalctl@ceph.osd.1.smithi132.stdout:Mar 14 09:48:40 smithi132 sudo[58535]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T09:48:40.771 INFO:journalctl@ceph.osd.1.smithi132.stdout:Mar 14 09:48:40 smithi132 sudo[58535]: pam_unix(sudo:session): session closed for user root 2024-03-14T09:48:40.936 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:40 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2824528264' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:48:40.936 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:40 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:41.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:40 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2824528264' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:48:41.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:40 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:41.249 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 14 09:48:40 smithi152 sudo[58715]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-14T09:48:41.249 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 14 09:48:40 smithi152 sudo[58715]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T09:48:41.249 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 14 09:48:40 smithi152 sudo[58715]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T09:48:41.249 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 14 09:48:40 smithi152 sudo[58715]: pam_unix(sudo:session): session closed for user root 2024-03-14T09:48:41.250 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 14 09:48:41 smithi152 sudo[58718]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-14T09:48:41.250 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 14 09:48:41 smithi152 sudo[58718]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T09:48:41.250 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 14 09:48:41 smithi152 sudo[58718]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T09:48:41.250 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 14 09:48:41 smithi152 sudo[58718]: pam_unix(sudo:session): session closed for user root 2024-03-14T09:48:42.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:41 smithi078 sudo[62782]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-14T09:48:42.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:41 smithi078 sudo[62782]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T09:48:42.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:41 smithi078 sudo[62782]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T09:48:42.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:41 smithi078 sudo[62782]: pam_unix(sudo:session): session closed for user root 2024-03-14T09:48:42.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:42.301+0000 7fc274c00700 1 -- 172.21.15.78:0/2740551997 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fc2640ccd00 con 0x7fc278101df0 2024-03-14T09:48:42.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:42 smithi078 sudo[63162]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-14T09:48:42.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:42 smithi078 sudo[63162]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T09:48:42.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:42 smithi078 sudo[63162]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T09:48:42.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:42 smithi078 sudo[63162]: pam_unix(sudo:session): session closed for user root 2024-03-14T09:48:42.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:42.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:42.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:42 smithi078 ceph-mon[38143]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-14T09:48:42.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:42.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:42.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:42 smithi132 ceph-mon[47959]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-14T09:48:42.774 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:42 smithi152 sudo[58721]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-14T09:48:42.774 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:42 smithi152 sudo[58721]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T09:48:42.774 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:42 smithi152 sudo[58721]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T09:48:42.774 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:42 smithi152 sudo[58721]: pam_unix(sudo:session): session closed for user root 2024-03-14T09:48:42.774 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:42 smithi152 sudo[58724]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-14T09:48:42.775 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:42 smithi152 sudo[58724]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T09:48:42.775 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:42 smithi152 sudo[58724]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T09:48:42.775 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:42 smithi152 sudo[58724]: pam_unix(sudo:session): session closed for user root 2024-03-14T09:48:43.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:43.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:43.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:42 smithi152 ceph-mon[48279]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-14T09:48:43.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 sudo[58538]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-14T09:48:43.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 sudo[58538]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T09:48:43.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 sudo[58538]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T09:48:43.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 sudo[58538]: pam_unix(sudo:session): session closed for user root 2024-03-14T09:48:43.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 sudo[58541]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-14T09:48:43.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 sudo[58541]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T09:48:43.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 sudo[58541]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T09:48:43.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 sudo[58541]: pam_unix(sudo:session): session closed for user root 2024-03-14T09:48:43.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:43 smithi078 ceph-mon[38143]: pgmap v71: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-14T09:48:43.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:43 smithi078 ceph-mon[38143]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-14T09:48:43.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:48:43.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:48:43.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:48:43.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:43 smithi078 ceph-mon[38143]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-14T09:48:43.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:43 smithi078 ceph-mon[38143]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-14T09:48:43.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:48:43.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:48:43.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:48:43.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:43 smithi078 ceph-mon[38143]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-14T09:48:43.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:43 smithi078 ceph-mon[38143]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-14T09:48:43.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:48:43.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:48:43.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:48:43.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:43 smithi078 ceph-mon[38143]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-14T09:48:43.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:43 smithi152 ceph-mon[48279]: pgmap v71: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-14T09:48:43.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:43 smithi152 ceph-mon[48279]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-14T09:48:43.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:48:43.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:48:43.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:48:43.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:43 smithi152 ceph-mon[48279]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-14T09:48:43.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:43 smithi152 ceph-mon[48279]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-14T09:48:43.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:48:43.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:48:43.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:48:43.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:43 smithi152 ceph-mon[48279]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-14T09:48:43.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:43 smithi152 ceph-mon[48279]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-14T09:48:43.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:48:43.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:48:43.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:48:43.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:43 smithi152 ceph-mon[48279]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-14T09:48:43.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 ceph-mon[47959]: pgmap v71: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-14T09:48:43.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 ceph-mon[47959]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-14T09:48:43.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:48:43.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:48:43.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:48:43.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 ceph-mon[47959]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-14T09:48:43.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 ceph-mon[47959]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-14T09:48:43.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:48:43.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:48:43.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:48:43.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 ceph-mon[47959]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-14T09:48:43.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 ceph-mon[47959]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-14T09:48:43.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:48:43.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:48:43.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:48:43.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:43 smithi132 ceph-mon[47959]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-14T09:48:44.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:44 smithi078 ceph-mon[38143]: pgmap v72: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-14T09:48:44.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:44 smithi152 ceph-mon[48279]: pgmap v72: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-14T09:48:44.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:44 smithi132 ceph-mon[47959]: pgmap v72: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-14T09:48:46.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:46 smithi078 ceph-mon[38143]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 457 KiB/s wr, 31 op/s 2024-03-14T09:48:46.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:46 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-14T09:48:46.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:46 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:46.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:46 smithi078 ceph-mon[38143]: Deploying daemon osd.3 on smithi078 2024-03-14T09:48:46.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:46 smithi152 ceph-mon[48279]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 457 KiB/s wr, 31 op/s 2024-03-14T09:48:46.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:46 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-14T09:48:46.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:46 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:46.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:46 smithi152 ceph-mon[48279]: Deploying daemon osd.3 on smithi078 2024-03-14T09:48:46.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:46 smithi132 ceph-mon[47959]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 457 KiB/s wr, 31 op/s 2024-03-14T09:48:46.785 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:46 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-14T09:48:46.785 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:46 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:46.785 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:46 smithi132 ceph-mon[47959]: Deploying daemon osd.3 on smithi078 2024-03-14T09:48:48.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:48 smithi078 ceph-mon[38143]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-03-14T09:48:48.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:48 smithi152 ceph-mon[48279]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-03-14T09:48:48.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:48 smithi132 ceph-mon[47959]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2024-03-14T09:48:50.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:50 smithi152 ceph-mon[48279]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-03-14T09:48:50.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:50 smithi132 ceph-mon[47959]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-03-14T09:48:50.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:50 smithi078 ceph-mon[38143]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2024-03-14T09:48:51.470 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:51 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:51.470 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:51 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:51.470 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:51 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:51.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:51 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:51.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:51 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:51.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:51 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:51.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:51 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:51.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:51 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:51.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:51 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:52.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:52 smithi078 ceph-mon[38143]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-14T09:48:52.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:52 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:52.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:52 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:52.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:52 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:52.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:52 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:52.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:52 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:52.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:52 smithi152 ceph-mon[48279]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-14T09:48:52.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:52 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:52.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:52 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:52.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:52 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:52.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:52 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:52.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:52 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:52.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:52 smithi132 ceph-mon[47959]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-14T09:48:52.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:52 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:52.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:52 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:52.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:52 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:52.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:52 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:52.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:52 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:53.084 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 3 on host 'smithi078' 2024-03-14T09:48:53.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:53.082+0000 7fc274c00700 1 -- 172.21.15.78:0/2740551997 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc278108ce0 con 0x7fc25c0a0a80 2024-03-14T09:48:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:53.085+0000 7fc27d400700 1 -- 172.21.15.78:0/2740551997 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc25c0a0a80 msgr2=0x7fc25c0a2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:53.085+0000 7fc27d400700 1 --2- 172.21.15.78:0/2740551997 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc25c0a0a80 0x7fc25c0a2f40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc268009990 tx=0x7fc268000bc0 comp rx=0 tx=0).stop 2024-03-14T09:48:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:53.085+0000 7fc27d400700 1 -- 172.21.15.78:0/2740551997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc278101df0 msgr2=0x7fc278199690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:53.085+0000 7fc27d400700 1 --2- 172.21.15.78:0/2740551997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc278101df0 0x7fc278199690 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc26400e9d0 tx=0x7fc26400ece0 comp rx=0 tx=0).stop 2024-03-14T09:48:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:53.085+0000 7fc27d400700 1 -- 172.21.15.78:0/2740551997 shutdown_connections 2024-03-14T09:48:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:53.086+0000 7fc27d400700 1 --2- 172.21.15.78:0/2740551997 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc25c0a0a80 0x7fc25c0a2f40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:53.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:53.086+0000 7fc27d400700 1 --2- 172.21.15.78:0/2740551997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc278102790 0x7fc2781a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:53.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:53.086+0000 7fc27d400700 1 --2- 172.21.15.78:0/2740551997 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc278100bf0 0x7fc278199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:53.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:53.086+0000 7fc27d400700 1 --2- 172.21.15.78:0/2740551997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc278101df0 0x7fc278199690 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc26400e9d0 tx=0x7fc26400ece0 comp rx=0 tx=0).stop 2024-03-14T09:48:53.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:53.086+0000 7fc27d400700 1 -- 172.21.15.78:0/2740551997 >> 172.21.15.78:0/2740551997 conn(0x7fc2780fc190 msgr2=0x7fc2781075c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:48:53.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:53.086+0000 7fc27d400700 1 -- 172.21.15.78:0/2740551997 shutdown_connections 2024-03-14T09:48:53.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:48:53.086+0000 7fc27d400700 1 -- 172.21.15.78:0/2740551997 wait complete. 2024-03-14T09:48:53.447 DEBUG:teuthology.orchestra.run.smithi078:osd.3> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.3.service 2024-03-14T09:48:53.451 INFO:tasks.cephadm:Deploying osd.4 on smithi132 with /dev/vg_nvme/lv_3... 2024-03-14T09:48:53.451 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-14T09:48:53.527 INFO:journalctl@ceph.osd.3.smithi078.stdout:-- Logs begin at Thu 2024-03-14 09:30:54 UTC. -- 2024-03-14T09:48:53.684 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.b/config 2024-03-14T09:48:53.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:53 smithi132 ceph-mon[47959]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-14T09:48:53.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:53.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:53.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:53.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:53 smithi152 ceph-mon[48279]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-14T09:48:53.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:53.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:53.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:54.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:53 smithi078 ceph-mon[38143]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-14T09:48:54.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:48:54.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:54.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:54.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:54 smithi132 ceph-mon[47959]: from='osd.3 [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-14T09:48:54.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:54 smithi078 ceph-mon[38143]: from='osd.3 [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-14T09:48:54.804 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 14 09:48:54 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-3[64184]: 2024-03-14T09:48:54.412+0000 7fc410069640 -1 osd.3 0 log_to_monitors true 2024-03-14T09:48:54.906 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-14T09:48:54.906 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-14T09:48:54.907 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-03-14T09:48:54.907 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-03-14T09:48:54.907 INFO:teuthology.orchestra.run.smithi132.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114546 s, 915 MB/s 2024-03-14T09:48:54.907 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 2024-03-14T09:48:54.907 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-03-14T09:48:54.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:54 smithi152 ceph-mon[48279]: from='osd.3 [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-14T09:48:55.304 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi132:vg_nvme/lv_3 2024-03-14T09:48:55.519 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.b/config 2024-03-14T09:48:55.928 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 14 09:48:55 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-3[64184]: 2024-03-14T09:48:55.664+0000 7fc404600700 -1 osd.3 0 waiting for initial osdmap 2024-03-14T09:48:55.928 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 14 09:48:55 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-3[64184]: 2024-03-14T09:48:55.670+0000 7fc3fcc00700 -1 osd.3 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T09:48:55.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:55 smithi078 ceph-mon[38143]: from='osd.3 [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-14T09:48:55.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:55 smithi078 ceph-mon[38143]: osdmap e22: 4 total, 3 up, 4 in 2024-03-14T09:48:55.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:55 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:48:55.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:55 smithi078 ceph-mon[38143]: from='osd.3 [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:48:55.928 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:55 smithi078 ceph-mon[38143]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-14T09:48:55.983 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:55 smithi132 ceph-mon[47959]: from='osd.3 [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-14T09:48:55.983 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:55 smithi132 ceph-mon[47959]: osdmap e22: 4 total, 3 up, 4 in 2024-03-14T09:48:55.983 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:55 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:48:55.983 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:55 smithi132 ceph-mon[47959]: from='osd.3 [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:48:55.983 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:55 smithi132 ceph-mon[47959]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-14T09:48:55.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:55 smithi152 ceph-mon[48279]: from='osd.3 [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-14T09:48:55.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:55 smithi152 ceph-mon[48279]: osdmap e22: 4 total, 3 up, 4 in 2024-03-14T09:48:55.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:55 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:48:55.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:55 smithi152 ceph-mon[48279]: from='osd.3 [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:48:55.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:55 smithi152 ceph-mon[48279]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-14T09:48:56.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.527+0000 7f1fe4c65700 1 -- 172.21.15.132:0/524329360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fe0100ed0 msgr2=0x7f1fe0101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:56.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.527+0000 7f1fe4c65700 1 --2- 172.21.15.132:0/524329360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fe0100ed0 0x7f1fe0101330 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1fd00099c0 tx=0x7f1fd0009cd0 comp rx=0 tx=0).stop 2024-03-14T09:48:56.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.527+0000 7f1fe4c65700 1 -- 172.21.15.132:0/524329360 shutdown_connections 2024-03-14T09:48:56.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.527+0000 7f1fe4c65700 1 --2- 172.21.15.132:0/524329360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fe0101870 0x7f1fe0108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:56.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.527+0000 7f1fe4c65700 1 --2- 172.21.15.132:0/524329360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fe00ffcd0 0x7f1fe01000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:56.531 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.527+0000 7f1fe4c65700 1 --2- 172.21.15.132:0/524329360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fe0100ed0 0x7f1fe0101330 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:56.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.527+0000 7f1fe4c65700 1 -- 172.21.15.132:0/524329360 >> 172.21.15.132:0/524329360 conn(0x7f1fe00fb270 msgr2=0x7f1fe00fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:48:56.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.527+0000 7f1fe4c65700 1 -- 172.21.15.132:0/524329360 shutdown_connections 2024-03-14T09:48:56.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.527+0000 7f1fe4c65700 1 -- 172.21.15.132:0/524329360 wait complete. 2024-03-14T09:48:56.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.528+0000 7f1fe4c65700 1 Processor -- start 2024-03-14T09:48:56.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.528+0000 7f1fe4c65700 1 -- start start 2024-03-14T09:48:56.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.528+0000 7f1fe4c65700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fe00ffcd0 0x7f1fe0195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:48:56.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.528+0000 7f1fe4c65700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fe0100ed0 0x7f1fe01964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:48:56.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.528+0000 7f1fe4c65700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fe0101870 0x7f1fe019d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:48:56.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.528+0000 7f1fe4c65700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe019dac0 con 0x7f1fe0100ed0 2024-03-14T09:48:56.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.528+0000 7f1fe4c65700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe019dc00 con 0x7f1fe00ffcd0 2024-03-14T09:48:56.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.528+0000 7f1fe4c65700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe019dd90 con 0x7f1fe0101870 2024-03-14T09:48:56.532 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.528+0000 7f1fde59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fe00ffcd0 0x7f1fe0195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:48:56.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.528+0000 7f1fde59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fe00ffcd0 0x7f1fe0195f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:55746/0 (socket says 172.21.15.132:55746) 2024-03-14T09:48:56.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.528+0000 7f1fde59c700 1 -- 172.21.15.132:0/3050803448 learned_addr learned my addr 172.21.15.132:0/3050803448 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-14T09:48:56.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.529+0000 7f1fddd9b700 1 --2- 172.21.15.132:0/3050803448 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fe0100ed0 0x7f1fe01964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:48:56.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.529+0000 7f1fded9d700 1 --2- 172.21.15.132:0/3050803448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fe0101870 0x7f1fe019d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:48:56.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.529+0000 7f1fde59c700 1 -- 172.21.15.132:0/3050803448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fe0101870 msgr2=0x7f1fe019d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:56.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.529+0000 7f1fde59c700 1 --2- 172.21.15.132:0/3050803448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fe0101870 0x7f1fe019d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:56.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.529+0000 7f1fde59c700 1 -- 172.21.15.132:0/3050803448 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fe0100ed0 msgr2=0x7f1fe01964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:48:56.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.529+0000 7f1fde59c700 1 --2- 172.21.15.132:0/3050803448 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fe0100ed0 0x7f1fe01964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:48:56.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.529+0000 7f1fde59c700 1 -- 172.21.15.132:0/3050803448 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd00095f0 con 0x7f1fe00ffcd0 2024-03-14T09:48:56.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.529+0000 7f1fddd9b700 1 --2- 172.21.15.132:0/3050803448 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fe0100ed0 0x7f1fe01964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:48:56.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.529+0000 7f1fde59c700 1 --2- 172.21.15.132:0/3050803448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fe00ffcd0 0x7f1fe0195f80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1fd400d980 tx=0x7f1fd400dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:48:56.533 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.529+0000 7f1fcf7fe700 1 -- 172.21.15.132:0/3050803448 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd400de60 con 0x7f1fe00ffcd0 2024-03-14T09:48:56.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.529+0000 7f1fe4c65700 1 -- 172.21.15.132:0/3050803448 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe019e070 con 0x7f1fe00ffcd0 2024-03-14T09:48:56.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.529+0000 7f1fcf7fe700 1 -- 172.21.15.132:0/3050803448 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1fd40044e0 con 0x7f1fe00ffcd0 2024-03-14T09:48:56.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.529+0000 7f1fcf7fe700 1 -- 172.21.15.132:0/3050803448 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4003bc0 con 0x7f1fe00ffcd0 2024-03-14T09:48:56.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.529+0000 7f1fe4c65700 1 -- 172.21.15.132:0/3050803448 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe019e570 con 0x7f1fe00ffcd0 2024-03-14T09:48:56.534 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.530+0000 7f1fe4c65700 1 -- 172.21.15.132:0/3050803448 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe0107100 con 0x7f1fe00ffcd0 2024-03-14T09:48:56.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.531+0000 7f1fcf7fe700 1 -- 172.21.15.132:0/3050803448 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f1fd4009bb0 con 0x7f1fe00ffcd0 2024-03-14T09:48:56.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.532+0000 7f1fcf7fe700 1 --2- 172.21.15.132:0/3050803448 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1fc80a0b10 0x7f1fc80a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:48:56.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.532+0000 7f1fcf7fe700 1 -- 172.21.15.132:0/3050803448 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 3389+0+0 (secure 0 0 0) 0x7f1fd40cae60 con 0x7f1fe00ffcd0 2024-03-14T09:48:56.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.532+0000 7f1fddd9b700 1 --2- 172.21.15.132:0/3050803448 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1fc80a0b10 0x7f1fc80a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:48:56.535 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.532+0000 7f1fddd9b700 1 --2- 172.21.15.132:0/3050803448 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1fc80a0b10 0x7f1fc80a2fd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0000c00 tx=0x7f1fd00037c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:48:56.536 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.535+0000 7f1fcf7fe700 1 -- 172.21.15.132:0/3050803448 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1fd4095ad0 con 0x7f1fe00ffcd0 2024-03-14T09:48:56.742 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:48:56.741+0000 7f1fe4c65700 1 -- 172.21.15.132:0/3050803448 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1fe00611f0 con 0x7f1fc80a0b10 2024-03-14T09:48:56.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:56 smithi152 ceph-mon[48279]: from='osd.3 [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-14T09:48:56.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:56 smithi152 ceph-mon[48279]: osdmap e23: 4 total, 3 up, 4 in 2024-03-14T09:48:56.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:56 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:48:56.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:56 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:48:56.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:56 smithi152 ceph-mon[48279]: Detected new or changed devices on smithi078 2024-03-14T09:48:56.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:56 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:57.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:56 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:57.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:56 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:48:57.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:56 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:48:57.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:56 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi078 to 8635M 2024-03-14T09:48:57.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:56 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:57.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:56 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:57.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:56 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:57.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:56 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:57.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:56 smithi132 ceph-mon[47959]: from='osd.3 [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-14T09:48:57.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:56 smithi132 ceph-mon[47959]: osdmap e23: 4 total, 3 up, 4 in 2024-03-14T09:48:57.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:56 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:48:57.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:56 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:48:57.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:56 smithi132 ceph-mon[47959]: Detected new or changed devices on smithi078 2024-03-14T09:48:57.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:56 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:57.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:56 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:57.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:56 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:48:57.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:56 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:48:57.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:56 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi078 to 8635M 2024-03-14T09:48:57.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:56 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:57.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:56 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:57.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:56 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:57.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:56 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:57.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:56 smithi078 ceph-mon[38143]: from='osd.3 [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-14T09:48:57.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:56 smithi078 ceph-mon[38143]: osdmap e23: 4 total, 3 up, 4 in 2024-03-14T09:48:57.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:56 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:48:57.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:56 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:48:57.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:56 smithi078 ceph-mon[38143]: Detected new or changed devices on smithi078 2024-03-14T09:48:57.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:56 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:57.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:56 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:57.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:56 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:48:57.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:56 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:48:57.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:56 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi078 to 8635M 2024-03-14T09:48:57.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:56 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:57.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:56 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:57.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:56 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:48:57.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:56 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:48:57.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:57 smithi152 ceph-mon[48279]: purged_snaps scrub starts 2024-03-14T09:48:57.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:57 smithi152 ceph-mon[48279]: purged_snaps scrub ok 2024-03-14T09:48:57.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:57 smithi152 ceph-mon[48279]: osd.3 [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058] boot 2024-03-14T09:48:57.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:57 smithi152 ceph-mon[48279]: osdmap e24: 4 total, 4 up, 4 in 2024-03-14T09:48:58.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:57 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:48:58.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:57 smithi152 ceph-mon[48279]: from='client.24239 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:48:58.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:57 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:48:58.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:57 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:48:58.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:57 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:58.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:57 smithi152 ceph-mon[48279]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-14T09:48:58.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:57 smithi152 ceph-mon[48279]: osdmap e25: 4 total, 4 up, 4 in 2024-03-14T09:48:58.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:57 smithi132 ceph-mon[47959]: purged_snaps scrub starts 2024-03-14T09:48:58.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:57 smithi132 ceph-mon[47959]: purged_snaps scrub ok 2024-03-14T09:48:58.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:57 smithi132 ceph-mon[47959]: osd.3 [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058] boot 2024-03-14T09:48:58.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:57 smithi132 ceph-mon[47959]: osdmap e24: 4 total, 4 up, 4 in 2024-03-14T09:48:58.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:57 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:48:58.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:57 smithi132 ceph-mon[47959]: from='client.24239 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:48:58.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:57 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:48:58.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:57 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:48:58.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:57 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:58.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:57 smithi132 ceph-mon[47959]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-14T09:48:58.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:57 smithi132 ceph-mon[47959]: osdmap e25: 4 total, 4 up, 4 in 2024-03-14T09:48:58.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:57 smithi078 ceph-mon[38143]: purged_snaps scrub starts 2024-03-14T09:48:58.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:57 smithi078 ceph-mon[38143]: purged_snaps scrub ok 2024-03-14T09:48:58.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:57 smithi078 ceph-mon[38143]: osd.3 [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058] boot 2024-03-14T09:48:58.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:57 smithi078 ceph-mon[38143]: osdmap e24: 4 total, 4 up, 4 in 2024-03-14T09:48:58.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:57 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:48:58.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:57 smithi078 ceph-mon[38143]: from='client.24239 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:48:58.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:57 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:48:58.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:57 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:48:58.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:57 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:48:58.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:57 smithi078 ceph-mon[38143]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-14T09:48:58.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:57 smithi078 ceph-mon[38143]: osdmap e25: 4 total, 4 up, 4 in 2024-03-14T09:48:58.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:58 smithi152 ceph-mon[48279]: from='client.? 172.21.15.132:0/3254310098' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc8670cb-b4c1-42fc-9134-4fce4b31e03c"}]: dispatch 2024-03-14T09:48:58.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:58 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc8670cb-b4c1-42fc-9134-4fce4b31e03c"}]: dispatch 2024-03-14T09:48:58.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:58 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc8670cb-b4c1-42fc-9134-4fce4b31e03c"}]': finished 2024-03-14T09:48:58.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:58 smithi152 ceph-mon[48279]: osdmap e26: 5 total, 4 up, 5 in 2024-03-14T09:48:58.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:58 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:48:59.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:58 smithi132 ceph-mon[47959]: from='client.? 172.21.15.132:0/3254310098' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc8670cb-b4c1-42fc-9134-4fce4b31e03c"}]: dispatch 2024-03-14T09:48:59.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:58 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc8670cb-b4c1-42fc-9134-4fce4b31e03c"}]: dispatch 2024-03-14T09:48:59.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:58 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc8670cb-b4c1-42fc-9134-4fce4b31e03c"}]': finished 2024-03-14T09:48:59.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:58 smithi132 ceph-mon[47959]: osdmap e26: 5 total, 4 up, 5 in 2024-03-14T09:48:59.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:58 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:48:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:58 smithi078 ceph-mon[38143]: from='client.? 172.21.15.132:0/3254310098' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc8670cb-b4c1-42fc-9134-4fce4b31e03c"}]: dispatch 2024-03-14T09:48:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:58 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc8670cb-b4c1-42fc-9134-4fce4b31e03c"}]: dispatch 2024-03-14T09:48:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:58 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc8670cb-b4c1-42fc-9134-4fce4b31e03c"}]': finished 2024-03-14T09:48:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:58 smithi078 ceph-mon[38143]: osdmap e26: 5 total, 4 up, 5 in 2024-03-14T09:48:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:58 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:48:59.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:59 smithi152 ceph-mon[48279]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-03-14T09:48:59.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:59 smithi152 ceph-mon[48279]: from='client.? 172.21.15.132:0/4145300087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:48:59.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:59 smithi152 ceph-mon[48279]: osdmap e27: 5 total, 4 up, 5 in 2024-03-14T09:48:59.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:48:59 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:00.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:59 smithi132 ceph-mon[47959]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-03-14T09:49:00.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:59 smithi132 ceph-mon[47959]: from='client.? 172.21.15.132:0/4145300087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:49:00.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:59 smithi132 ceph-mon[47959]: osdmap e27: 5 total, 4 up, 5 in 2024-03-14T09:49:00.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:48:59 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:00.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:59 smithi078 ceph-mon[38143]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-03-14T09:49:00.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:59 smithi078 ceph-mon[38143]: from='client.? 172.21.15.132:0/4145300087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:49:00.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:59 smithi078 ceph-mon[38143]: osdmap e27: 5 total, 4 up, 5 in 2024-03-14T09:49:00.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:48:59 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:01.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:01 smithi078 ceph-mon[38143]: osdmap e28: 5 total, 4 up, 5 in 2024-03-14T09:49:01.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:01 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:01.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:01 smithi152 ceph-mon[48279]: osdmap e28: 5 total, 4 up, 5 in 2024-03-14T09:49:01.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:01 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:02.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:01 smithi132 ceph-mon[47959]: osdmap e28: 5 total, 4 up, 5 in 2024-03-14T09:49:02.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:01 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:02.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:02 smithi132 ceph-mon[47959]: pgmap v88: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-14T09:49:02.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:02 smithi152 ceph-mon[48279]: pgmap v88: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-14T09:49:03.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:02 smithi078 ceph-mon[38143]: pgmap v88: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-14T09:49:03.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:03 smithi152 ceph-mon[48279]: pgmap v89: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-14T09:49:04.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:03 smithi132 ceph-mon[47959]: pgmap v89: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-14T09:49:04.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:03 smithi078 ceph-mon[38143]: pgmap v89: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-14T09:49:04.932 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:04 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-14T09:49:04.932 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:04 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:04.932 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:04 smithi132 ceph-mon[47959]: Deploying daemon osd.4 on smithi132 2024-03-14T09:49:04.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:04 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-14T09:49:04.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:04 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:04.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:04 smithi152 ceph-mon[48279]: Deploying daemon osd.4 on smithi132 2024-03-14T09:49:05.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:04 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-14T09:49:05.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:04 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:05.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:04 smithi078 ceph-mon[38143]: Deploying daemon osd.4 on smithi132 2024-03-14T09:49:05.981 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:05 smithi132 ceph-mon[47959]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-14T09:49:05.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:05 smithi152 ceph-mon[48279]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-14T09:49:06.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:05 smithi078 ceph-mon[38143]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-14T09:49:08.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:08 smithi152 ceph-mon[48279]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-14T09:49:08.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:08 smithi132 ceph-mon[47959]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-14T09:49:08.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:08 smithi078 ceph-mon[38143]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-14T09:49:09.215 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:09 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:09.215 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:09 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:09.215 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:09 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:09.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:09 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:09.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:09 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:09.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:09 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:09.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:09 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:09.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:09 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:09.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:09 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:10.397 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:10.397+0000 7f1fcf7fe700 1 -- 172.21.15.132:0/3050803448 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1fe00611f0 con 0x7f1fc80a0b10 2024-03-14T09:49:10.397 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 4 on host 'smithi132' 2024-03-14T09:49:10.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:10.398+0000 7f1fe4c65700 1 -- 172.21.15.132:0/3050803448 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1fc80a0b10 msgr2=0x7f1fc80a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:49:10.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:10.398+0000 7f1fe4c65700 1 --2- 172.21.15.132:0/3050803448 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1fc80a0b10 0x7f1fc80a2fd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0000c00 tx=0x7f1fd00037c0 comp rx=0 tx=0).stop 2024-03-14T09:49:10.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:10.398+0000 7f1fe4c65700 1 -- 172.21.15.132:0/3050803448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fe00ffcd0 msgr2=0x7f1fe0195f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:49:10.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:10.398+0000 7f1fe4c65700 1 --2- 172.21.15.132:0/3050803448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fe00ffcd0 0x7f1fe0195f80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1fd400d980 tx=0x7f1fd400dd40 comp rx=0 tx=0).stop 2024-03-14T09:49:10.399 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:10.399+0000 7f1fe4c65700 1 -- 172.21.15.132:0/3050803448 shutdown_connections 2024-03-14T09:49:10.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:10.399+0000 7f1fe4c65700 1 --2- 172.21.15.132:0/3050803448 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1fc80a0b10 0x7f1fc80a2fd0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:10.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:10.399+0000 7f1fe4c65700 1 --2- 172.21.15.132:0/3050803448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fe00ffcd0 0x7f1fe0195f80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:10.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:10.399+0000 7f1fe4c65700 1 --2- 172.21.15.132:0/3050803448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fe0101870 0x7f1fe019d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:10.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:10.399+0000 7f1fe4c65700 1 --2- 172.21.15.132:0/3050803448 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fe0100ed0 0x7f1fe01964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:10.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:10.399+0000 7f1fe4c65700 1 -- 172.21.15.132:0/3050803448 >> 172.21.15.132:0/3050803448 conn(0x7f1fe00fb270 msgr2=0x7f1fe0104240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:49:10.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:10.399+0000 7f1fe4c65700 1 -- 172.21.15.132:0/3050803448 shutdown_connections 2024-03-14T09:49:10.400 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:10.399+0000 7f1fe4c65700 1 -- 172.21.15.132:0/3050803448 wait complete. 2024-03-14T09:49:10.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:10 smithi132 ceph-mon[47959]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-03-14T09:49:10.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:10.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:10.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:10.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:10.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:10.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:10.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:10.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:10.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:10 smithi152 ceph-mon[48279]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-03-14T09:49:10.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:10.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:10.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:10.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:10.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:10.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:10.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:10.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:10.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:10 smithi078 ceph-mon[38143]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-03-14T09:49:10.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:10.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:10.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:10.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:10.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:10.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:10.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:10.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:10.837 DEBUG:teuthology.orchestra.run.smithi132:osd.4> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.4.service 2024-03-14T09:49:10.840 INFO:tasks.cephadm:Deploying osd.5 on smithi152 with /dev/vg_nvme/lv_3... 2024-03-14T09:49:10.840 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-14T09:49:10.869 INFO:journalctl@ceph.osd.4.smithi132.stdout:-- Logs begin at Thu 2024-03-14 09:24:09 UTC. -- 2024-03-14T09:49:11.071 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.c/config 2024-03-14T09:49:12.437 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-14T09:49:12.437 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-14T09:49:12.438 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-14T09:49:12.438 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-14T09:49:12.438 INFO:teuthology.orchestra.run.smithi152.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0156118 s, 672 MB/s 2024-03-14T09:49:12.438 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-14T09:49:12.455 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 14 09:49:12 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-4[61611]: 2024-03-14T09:49:12.105+0000 7f7a622cd640 -1 osd.4 0 log_to_monitors true 2024-03-14T09:49:12.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:12 smithi152 ceph-mon[48279]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-03-14T09:49:12.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:12 smithi152 ceph-mon[48279]: from='osd.4 [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-14T09:49:12.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:12 smithi152 ceph-mon[48279]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-14T09:49:12.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:12 smithi132 ceph-mon[47959]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-03-14T09:49:12.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:12 smithi132 ceph-mon[47959]: from='osd.4 [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-14T09:49:12.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:12 smithi132 ceph-mon[47959]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-14T09:49:12.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:12 smithi078 ceph-mon[38143]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-03-14T09:49:12.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:12 smithi078 ceph-mon[38143]: from='osd.4 [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-14T09:49:12.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:12 smithi078 ceph-mon[38143]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-14T09:49:12.813 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_3 2024-03-14T09:49:13.030 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.c/config 2024-03-14T09:49:13.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:13 smithi152 ceph-mon[48279]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-14T09:49:13.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:13 smithi152 ceph-mon[48279]: osdmap e29: 5 total, 4 up, 5 in 2024-03-14T09:49:13.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:13.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:13 smithi152 ceph-mon[48279]: from='osd.4 [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:49:13.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:13 smithi152 ceph-mon[48279]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:49:13.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:13.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:13.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:13.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:13.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:13.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:13.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:13.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:13.770 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 14 09:49:13 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-4[61611]: 2024-03-14T09:49:13.471+0000 7f7a57931700 -1 osd.4 0 waiting for initial osdmap 2024-03-14T09:49:13.770 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 14 09:49:13 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-4[61611]: 2024-03-14T09:49:13.479+0000 7f7a53726700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T09:49:13.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:13 smithi132 ceph-mon[47959]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-14T09:49:13.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:13 smithi132 ceph-mon[47959]: osdmap e29: 5 total, 4 up, 5 in 2024-03-14T09:49:13.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:13 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:13.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:13 smithi132 ceph-mon[47959]: from='osd.4 [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:49:13.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:13 smithi132 ceph-mon[47959]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:49:13.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:13 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:13.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:13 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:13.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:13 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:13.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:13 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:13.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:13 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:13.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:13 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:13.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:13 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:13.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:13 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:13.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:13 smithi078 ceph-mon[38143]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-14T09:49:13.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:13 smithi078 ceph-mon[38143]: osdmap e29: 5 total, 4 up, 5 in 2024-03-14T09:49:13.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:13 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:13.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:13 smithi078 ceph-mon[38143]: from='osd.4 [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:49:13.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:13 smithi078 ceph-mon[38143]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:49:13.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:13 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:13.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:13 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:13.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:13 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:13.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:13 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:13.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:13 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:13.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:13 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:13.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:13 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:13.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:13 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:14.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.045+0000 7fa480341700 1 -- 172.21.15.152:0/3551589466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa478101880 msgr2=0x7fa478108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:49:14.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.045+0000 7fa480341700 1 --2- 172.21.15.152:0/3551589466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa478101880 0x7fa478108130 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa47000b0a0 tx=0x7fa47000b3b0 comp rx=0 tx=0).stop 2024-03-14T09:49:14.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.046+0000 7fa480341700 1 -- 172.21.15.152:0/3551589466 shutdown_connections 2024-03-14T09:49:14.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.046+0000 7fa480341700 1 --2- 172.21.15.152:0/3551589466 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa478101880 0x7fa478108130 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:14.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.046+0000 7fa480341700 1 --2- 172.21.15.152:0/3551589466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4780ffce0 0x7fa478100100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:14.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.046+0000 7fa480341700 1 --2- 172.21.15.152:0/3551589466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa478100ee0 0x7fa478101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:14.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.046+0000 7fa480341700 1 -- 172.21.15.152:0/3551589466 >> 172.21.15.152:0/3551589466 conn(0x7fa4780fb260 msgr2=0x7fa4780fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:49:14.049 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.046+0000 7fa480341700 1 -- 172.21.15.152:0/3551589466 shutdown_connections 2024-03-14T09:49:14.049 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.046+0000 7fa480341700 1 -- 172.21.15.152:0/3551589466 wait complete. 2024-03-14T09:49:14.049 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.046+0000 7fa480341700 1 Processor -- start 2024-03-14T09:49:14.049 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.046+0000 7fa480341700 1 -- start start 2024-03-14T09:49:14.049 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.046+0000 7fa480341700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4780ffce0 0x7fa4781972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:49:14.049 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.046+0000 7fa480341700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa478100ee0 0x7fa4781977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:49:14.049 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.046+0000 7fa480341700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa478101880 0x7fa47819e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:49:14.049 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.046+0000 7fa480341700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47819edf0 con 0x7fa478100ee0 2024-03-14T09:49:14.049 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.046+0000 7fa480341700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47819ef30 con 0x7fa478101880 2024-03-14T09:49:14.049 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.046+0000 7fa480341700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47819f0c0 con 0x7fa4780ffce0 2024-03-14T09:49:14.049 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.046+0000 7fa47e0dd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4780ffce0 0x7fa4781972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:49:14.049 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa47e0dd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4780ffce0 0x7fa4781972b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:42726/0 (socket says 172.21.15.152:42726) 2024-03-14T09:49:14.049 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa47e0dd700 1 -- 172.21.15.152:0/1062145509 learned_addr learned my addr 172.21.15.152:0/1062145509 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-14T09:49:14.050 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa47e8de700 1 --2- 172.21.15.152:0/1062145509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa478101880 0x7fa47819e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:49:14.050 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa47d8dc700 1 --2- 172.21.15.152:0/1062145509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa478100ee0 0x7fa4781977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:49:14.050 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa47e0dd700 1 -- 172.21.15.152:0/1062145509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa478101880 msgr2=0x7fa47819e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:49:14.050 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa47e0dd700 1 --2- 172.21.15.152:0/1062145509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa478101880 0x7fa47819e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:14.050 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa47e0dd700 1 -- 172.21.15.152:0/1062145509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa478100ee0 msgr2=0x7fa4781977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:49:14.050 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa47e0dd700 1 --2- 172.21.15.152:0/1062145509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa478100ee0 0x7fa4781977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:14.050 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa47e0dd700 1 -- 172.21.15.152:0/1062145509 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa470009d00 con 0x7fa4780ffce0 2024-03-14T09:49:14.050 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa47e8de700 1 --2- 172.21.15.152:0/1062145509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa478101880 0x7fa47819e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:49:14.050 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa47d8dc700 1 --2- 172.21.15.152:0/1062145509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa478100ee0 0x7fa4781977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:49:14.050 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa47e0dd700 1 --2- 172.21.15.152:0/1062145509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4780ffce0 0x7fa4781972b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa47400b530 tx=0x7fa47400b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:49:14.050 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa46f7fe700 1 -- 172.21.15.152:0/1062145509 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa474011830 con 0x7fa4780ffce0 2024-03-14T09:49:14.050 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa46f7fe700 1 -- 172.21.15.152:0/1062145509 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa474011e50 con 0x7fa4780ffce0 2024-03-14T09:49:14.050 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa480341700 1 -- 172.21.15.152:0/1062145509 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47819f3a0 con 0x7fa4780ffce0 2024-03-14T09:49:14.050 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa46f7fe700 1 -- 172.21.15.152:0/1062145509 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47400f2e0 con 0x7fa4780ffce0 2024-03-14T09:49:14.051 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.047+0000 7fa480341700 1 -- 172.21.15.152:0/1062145509 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47819f970 con 0x7fa4780ffce0 2024-03-14T09:49:14.051 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.048+0000 7fa480341700 1 -- 172.21.15.152:0/1062145509 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47804ea80 con 0x7fa4780ffce0 2024-03-14T09:49:14.051 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.049+0000 7fa46f7fe700 1 -- 172.21.15.152:0/1062145509 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fa474010580 con 0x7fa4780ffce0 2024-03-14T09:49:14.051 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.050+0000 7fa46f7fe700 1 --2- 172.21.15.152:0/1062145509 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa4640a0b10 0x7fa4640a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:49:14.052 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.050+0000 7fa46f7fe700 1 -- 172.21.15.152:0/1062145509 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3821+0+0 (secure 0 0 0) 0x7fa4740cae80 con 0x7fa4780ffce0 2024-03-14T09:49:14.052 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.051+0000 7fa47d8dc700 1 --2- 172.21.15.152:0/1062145509 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa4640a0b10 0x7fa4640a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:49:14.052 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.051+0000 7fa47d8dc700 1 --2- 172.21.15.152:0/1062145509 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa4640a0b10 0x7fa4640a2fd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa468007900 tx=0x7fa468008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:49:14.053 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.054+0000 7fa46f7fe700 1 -- 172.21.15.152:0/1062145509 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa4740959e0 con 0x7fa4780ffce0 2024-03-14T09:49:14.251 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:14.251+0000 7fa480341700 1 -- 172.21.15.152:0/1062145509 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4780fd430 con 0x7fa4640a0b10 2024-03-14T09:49:14.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:14 smithi152 ceph-mon[48279]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-03-14T09:49:14.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:14 smithi152 ceph-mon[48279]: Detected new or changed devices on smithi132 2024-03-14T09:49:14.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:14 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi132 to 8557M 2024-03-14T09:49:14.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:14 smithi152 ceph-mon[48279]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-14T09:49:14.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:14 smithi152 ceph-mon[48279]: osdmap e30: 5 total, 4 up, 5 in 2024-03-14T09:49:14.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:14 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:14.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:14 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:14.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:14 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:49:14.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:14 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:49:14.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:14 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:14.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:14 smithi132 ceph-mon[47959]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-03-14T09:49:14.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:14 smithi132 ceph-mon[47959]: Detected new or changed devices on smithi132 2024-03-14T09:49:14.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:14 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi132 to 8557M 2024-03-14T09:49:14.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:14 smithi132 ceph-mon[47959]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-14T09:49:14.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:14 smithi132 ceph-mon[47959]: osdmap e30: 5 total, 4 up, 5 in 2024-03-14T09:49:14.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:14 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:14.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:14 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:14.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:14 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:49:14.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:14 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:49:14.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:14 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:14.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:14 smithi078 ceph-mon[38143]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-03-14T09:49:14.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:14 smithi078 ceph-mon[38143]: Detected new or changed devices on smithi132 2024-03-14T09:49:14.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:14 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi132 to 8557M 2024-03-14T09:49:14.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:14 smithi078 ceph-mon[38143]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-14T09:49:14.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:14 smithi078 ceph-mon[38143]: osdmap e30: 5 total, 4 up, 5 in 2024-03-14T09:49:14.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:14 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:14.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:14 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:14.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:14 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:49:14.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:14 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:49:14.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:14 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:15.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:15 smithi152 ceph-mon[48279]: purged_snaps scrub starts 2024-03-14T09:49:15.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:15 smithi152 ceph-mon[48279]: purged_snaps scrub ok 2024-03-14T09:49:15.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:15 smithi152 ceph-mon[48279]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:49:15.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:15 smithi152 ceph-mon[48279]: osd.4 [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797] boot 2024-03-14T09:49:15.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:15 smithi152 ceph-mon[48279]: osdmap e31: 5 total, 5 up, 5 in 2024-03-14T09:49:15.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:15 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:15.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:15 smithi132 ceph-mon[47959]: purged_snaps scrub starts 2024-03-14T09:49:15.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:15 smithi132 ceph-mon[47959]: purged_snaps scrub ok 2024-03-14T09:49:15.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:15 smithi132 ceph-mon[47959]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:49:15.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:15 smithi132 ceph-mon[47959]: osd.4 [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797] boot 2024-03-14T09:49:15.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:15 smithi132 ceph-mon[47959]: osdmap e31: 5 total, 5 up, 5 in 2024-03-14T09:49:15.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:15 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:15.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:15 smithi078 ceph-mon[38143]: purged_snaps scrub starts 2024-03-14T09:49:15.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:15 smithi078 ceph-mon[38143]: purged_snaps scrub ok 2024-03-14T09:49:15.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:15 smithi078 ceph-mon[38143]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:49:15.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:15 smithi078 ceph-mon[38143]: osd.4 [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797] boot 2024-03-14T09:49:15.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:15 smithi078 ceph-mon[38143]: osdmap e31: 5 total, 5 up, 5 in 2024-03-14T09:49:15.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:15 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:49:16.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:16 smithi152 ceph-mon[48279]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-03-14T09:49:16.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:16 smithi152 ceph-mon[48279]: osdmap e32: 5 total, 5 up, 5 in 2024-03-14T09:49:16.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:16 smithi152 ceph-mon[48279]: from='client.? 172.21.15.152:0/2499369751' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a315a4be-a807-4aef-b476-21ca966b20fa"}]: dispatch 2024-03-14T09:49:16.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:16 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a315a4be-a807-4aef-b476-21ca966b20fa"}]: dispatch 2024-03-14T09:49:16.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:16 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a315a4be-a807-4aef-b476-21ca966b20fa"}]': finished 2024-03-14T09:49:16.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:16 smithi152 ceph-mon[48279]: osdmap e33: 6 total, 5 up, 6 in 2024-03-14T09:49:16.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:16 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:49:16.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:16 smithi132 ceph-mon[47959]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-03-14T09:49:16.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:16 smithi132 ceph-mon[47959]: osdmap e32: 5 total, 5 up, 5 in 2024-03-14T09:49:16.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:16 smithi132 ceph-mon[47959]: from='client.? 172.21.15.152:0/2499369751' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a315a4be-a807-4aef-b476-21ca966b20fa"}]: dispatch 2024-03-14T09:49:16.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:16 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a315a4be-a807-4aef-b476-21ca966b20fa"}]: dispatch 2024-03-14T09:49:16.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:16 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a315a4be-a807-4aef-b476-21ca966b20fa"}]': finished 2024-03-14T09:49:16.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:16 smithi132 ceph-mon[47959]: osdmap e33: 6 total, 5 up, 6 in 2024-03-14T09:49:16.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:16 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:49:16.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:16 smithi078 ceph-mon[38143]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-03-14T09:49:16.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:16 smithi078 ceph-mon[38143]: osdmap e32: 5 total, 5 up, 5 in 2024-03-14T09:49:16.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:16 smithi078 ceph-mon[38143]: from='client.? 172.21.15.152:0/2499369751' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a315a4be-a807-4aef-b476-21ca966b20fa"}]: dispatch 2024-03-14T09:49:16.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:16 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a315a4be-a807-4aef-b476-21ca966b20fa"}]: dispatch 2024-03-14T09:49:16.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:16 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a315a4be-a807-4aef-b476-21ca966b20fa"}]': finished 2024-03-14T09:49:16.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:16 smithi078 ceph-mon[38143]: osdmap e33: 6 total, 5 up, 6 in 2024-03-14T09:49:16.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:16 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:49:17.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:17 smithi152 ceph-mon[48279]: from='client.? 172.21.15.152:0/2841682177' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:49:17.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:17 smithi132 ceph-mon[47959]: from='client.? 172.21.15.152:0/2841682177' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:49:17.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:17 smithi078 ceph-mon[38143]: from='client.? 172.21.15.152:0/2841682177' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:49:18.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:18 smithi152 ceph-mon[48279]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:18.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:18 smithi132 ceph-mon[47959]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:18.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:18 smithi078 ceph-mon[38143]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:20.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:20 smithi152 ceph-mon[48279]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:20.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:20 smithi132 ceph-mon[47959]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:20.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:20 smithi078 ceph-mon[38143]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:22.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:22 smithi132 ceph-mon[47959]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:22.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:22 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-14T09:49:22.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:22 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:22.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:22 smithi078 ceph-mon[38143]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:22.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:22 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-14T09:49:22.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:22 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:22.866 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:22 smithi152 ceph-mon[48279]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:22.866 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:22 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-14T09:49:22.866 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:22 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:23.693 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:23 smithi152 ceph-mon[48279]: Deploying daemon osd.5 on smithi152 2024-03-14T09:49:23.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:23 smithi132 ceph-mon[47959]: Deploying daemon osd.5 on smithi152 2024-03-14T09:49:23.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:23 smithi078 ceph-mon[38143]: Deploying daemon osd.5 on smithi152 2024-03-14T09:49:24.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:24 smithi152 ceph-mon[48279]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:24.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:24 smithi132 ceph-mon[47959]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:24.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:24 smithi078 ceph-mon[38143]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:26.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:26 smithi132 ceph-mon[47959]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:26.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:26 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:26.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:26 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:26.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:26 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:26.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:26 smithi078 ceph-mon[38143]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:26.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:26.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:26.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:26.998 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:26 smithi152 ceph-mon[48279]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:26.998 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:26.998 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:26.998 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:28.662 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:28.661+0000 7fa46f7fe700 1 -- 172.21.15.152:0/1062145509 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa4780fd430 con 0x7fa4640a0b10 2024-03-14T09:49:28.663 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 5 on host 'smithi152' 2024-03-14T09:49:28.664 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:28.664+0000 7fa480341700 1 -- 172.21.15.152:0/1062145509 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa4640a0b10 msgr2=0x7fa4640a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:49:28.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:28.664+0000 7fa480341700 1 --2- 172.21.15.152:0/1062145509 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa4640a0b10 0x7fa4640a2fd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa468007900 tx=0x7fa468008040 comp rx=0 tx=0).stop 2024-03-14T09:49:28.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:28.664+0000 7fa480341700 1 -- 172.21.15.152:0/1062145509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4780ffce0 msgr2=0x7fa4781972b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:49:28.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:28.664+0000 7fa480341700 1 --2- 172.21.15.152:0/1062145509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4780ffce0 0x7fa4781972b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa47400b530 tx=0x7fa47400b8f0 comp rx=0 tx=0).stop 2024-03-14T09:49:28.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:28.664+0000 7fa480341700 1 -- 172.21.15.152:0/1062145509 shutdown_connections 2024-03-14T09:49:28.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:28.664+0000 7fa480341700 1 --2- 172.21.15.152:0/1062145509 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa4640a0b10 0x7fa4640a2fd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:28.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:28.664+0000 7fa480341700 1 --2- 172.21.15.152:0/1062145509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa478101880 0x7fa47819e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:28.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:28.664+0000 7fa480341700 1 --2- 172.21.15.152:0/1062145509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa4780ffce0 0x7fa4781972b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:28.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:28.664+0000 7fa480341700 1 --2- 172.21.15.152:0/1062145509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa478100ee0 0x7fa4781977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:28.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:28.664+0000 7fa480341700 1 -- 172.21.15.152:0/1062145509 >> 172.21.15.152:0/1062145509 conn(0x7fa4780fb260 msgr2=0x7fa4780fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:49:28.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:28.664+0000 7fa480341700 1 -- 172.21.15.152:0/1062145509 shutdown_connections 2024-03-14T09:49:28.665 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:49:28.664+0000 7fa480341700 1 -- 172.21.15.152:0/1062145509 wait complete. 2024-03-14T09:49:28.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:28 smithi132 ceph-mon[47959]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:28.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:28.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:28.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:28.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:28.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:28 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:28.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:28 smithi078 ceph-mon[38143]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:28.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:28 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:28.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:28 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:28.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:28 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:28.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:28 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:28.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:28 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:28.907 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:28 smithi152 ceph-mon[48279]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:28.907 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:28 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:28.907 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:28 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:28.907 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:28 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:28.907 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:28 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:28.907 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:28 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:29.075 DEBUG:teuthology.orchestra.run.smithi152:osd.5> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.5.service 2024-03-14T09:49:29.077 INFO:tasks.cephadm:Deploying osd.6 on smithi078 with /dev/vg_nvme/lv_2... 2024-03-14T09:49:29.077 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-14T09:49:29.106 INFO:journalctl@ceph.osd.5.smithi152.stdout:-- Logs begin at Thu 2024-03-14 09:24:45 UTC. -- 2024-03-14T09:49:29.433 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:49:29.923 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:29 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:29.923 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:29 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:29.923 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:29 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:29.949 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:29 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:29.949 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:29 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:29.949 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:29 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:30.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:29 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:30.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:29 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:30.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:29 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:30.249 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 09:49:30 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T09:49:30.060+0000 7f5faaaab640 -1 osd.5 0 log_to_monitors true 2024-03-14T09:49:30.928 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:30 smithi152 ceph-mon[48279]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:30.928 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:30 smithi152 ceph-mon[48279]: from='osd.5 [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-14T09:49:30.928 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:30 smithi152 ceph-mon[48279]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-14T09:49:31.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:30 smithi132 ceph-mon[47959]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:31.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:30 smithi132 ceph-mon[47959]: from='osd.5 [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-14T09:49:31.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:30 smithi132 ceph-mon[47959]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-14T09:49:31.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:30 smithi078 ceph-mon[38143]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:31.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:30 smithi078 ceph-mon[38143]: from='osd.5 [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-14T09:49:31.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:30 smithi078 ceph-mon[38143]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-14T09:49:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-14T09:49:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-14T09:49:31.096 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2024-03-14T09:49:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2024-03-14T09:49:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0297532 s, 352 MB/s 2024-03-14T09:49:31.096 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 2024-03-14T09:49:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2024-03-14T09:49:31.469 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_2 2024-03-14T09:49:31.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:31 smithi078 ceph-mon[38143]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-14T09:49:31.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:31 smithi078 ceph-mon[38143]: osdmap e34: 6 total, 5 up, 6 in 2024-03-14T09:49:31.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:49:31.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:31 smithi078 ceph-mon[38143]: from='osd.5 [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:49:31.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:31 smithi078 ceph-mon[38143]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:49:31.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:31 smithi078 ceph-mon[38143]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:31.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:31 smithi078 ceph-mon[38143]: Detected new or changed devices on smithi152 2024-03-14T09:49:31.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:31.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:31.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:31.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:31.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:31 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi152 to 8557M 2024-03-14T09:49:31.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:31.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:31.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:31.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:31.822 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:49:31.999 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T09:49:31.632+0000 7f5fa010f700 -1 osd.5 0 waiting for initial osdmap 2024-03-14T09:49:31.999 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T09:49:31.642+0000 7f5f99f00700 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T09:49:31.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-mon[48279]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-14T09:49:31.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-mon[48279]: osdmap e34: 6 total, 5 up, 6 in 2024-03-14T09:49:31.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:49:31.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-mon[48279]: from='osd.5 [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:49:31.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-mon[48279]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:49:32.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-mon[48279]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:32.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-mon[48279]: Detected new or changed devices on smithi152 2024-03-14T09:49:32.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:32.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:32.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:32.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:32.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi152 to 8557M 2024-03-14T09:49:32.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:32.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:32.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:32.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:31 smithi132 ceph-mon[47959]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-14T09:49:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:31 smithi132 ceph-mon[47959]: osdmap e34: 6 total, 5 up, 6 in 2024-03-14T09:49:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:49:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:31 smithi132 ceph-mon[47959]: from='osd.5 [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:49:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:31 smithi132 ceph-mon[47959]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:49:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:31 smithi132 ceph-mon[47959]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-03-14T09:49:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:31 smithi132 ceph-mon[47959]: Detected new or changed devices on smithi152 2024-03-14T09:49:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:32.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:32.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:31 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi152 to 8557M 2024-03-14T09:49:32.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:32.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:32.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:32.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:32.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.940+0000 7f413ae00700 1 -- 172.21.15.78:0/1310540744 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4134101fe0 msgr2=0x7f4134102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:49:32.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.940+0000 7f413ae00700 1 --2- 172.21.15.78:0/1310540744 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4134101fe0 0x7f4134102440 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f41200099c0 tx=0x7f4120009cd0 comp rx=0 tx=0).stop 2024-03-14T09:49:32.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.940+0000 7f413ae00700 1 -- 172.21.15.78:0/1310540744 shutdown_connections 2024-03-14T09:49:32.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.940+0000 7f413ae00700 1 --2- 172.21.15.78:0/1310540744 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4134100de0 0x7f4134101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:32.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.940+0000 7f413ae00700 1 --2- 172.21.15.78:0/1310540744 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4134102980 0x7f4134109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:32.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.940+0000 7f413ae00700 1 --2- 172.21.15.78:0/1310540744 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4134101fe0 0x7f4134102440 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:32.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.940+0000 7f413ae00700 1 -- 172.21.15.78:0/1310540744 >> 172.21.15.78:0/1310540744 conn(0x7f41340fc380 msgr2=0x7f41340fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:49:32.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.940+0000 7f413ae00700 1 -- 172.21.15.78:0/1310540744 shutdown_connections 2024-03-14T09:49:32.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.940+0000 7f413ae00700 1 -- 172.21.15.78:0/1310540744 wait complete. 2024-03-14T09:49:32.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.941+0000 7f413ae00700 1 Processor -- start 2024-03-14T09:49:32.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.941+0000 7f413ae00700 1 -- start start 2024-03-14T09:49:32.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.941+0000 7f413ae00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4134100de0 0x7f4134197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:49:32.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.941+0000 7f413ae00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4134101fe0 0x7f41341976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:49:32.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.941+0000 7f413ae00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4134102980 0x7f413419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:49:32.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.941+0000 7f413ae00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413419eca0 con 0x7f4134100de0 2024-03-14T09:49:32.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.941+0000 7f413ae00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413419ede0 con 0x7f4134102980 2024-03-14T09:49:32.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.941+0000 7f413ae00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413419ef70 con 0x7f4134101fe0 2024-03-14T09:49:32.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.942+0000 7f4133e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4134100de0 0x7f4134197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:49:32.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.942+0000 7f4133e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4134100de0 0x7f4134197160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35866/0 (socket says 172.21.15.78:35866) 2024-03-14T09:49:32.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.942+0000 7f4133400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4134101fe0 0x7f41341976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:49:32.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.942+0000 7f4133e00700 1 -- 172.21.15.78:0/3834887676 learned_addr learned my addr 172.21.15.78:0/3834887676 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:49:32.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.942+0000 7f4138e00700 1 --2- 172.21.15.78:0/3834887676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4134102980 0x7f413419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:49:32.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.942+0000 7f4133e00700 1 -- 172.21.15.78:0/3834887676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4134101fe0 msgr2=0x7f41341976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:49:32.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.942+0000 7f4133e00700 1 --2- 172.21.15.78:0/3834887676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4134101fe0 0x7f41341976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:32.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.942+0000 7f4133e00700 1 -- 172.21.15.78:0/3834887676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4134102980 msgr2=0x7f413419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:49:32.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.942+0000 7f4133e00700 1 --2- 172.21.15.78:0/3834887676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4134102980 0x7f413419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:32.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.942+0000 7f4133e00700 1 -- 172.21.15.78:0/3834887676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41200095f0 con 0x7f4134100de0 2024-03-14T09:49:32.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.942+0000 7f4133400700 1 --2- 172.21.15.78:0/3834887676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4134101fe0 0x7f41341976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:49:32.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.942+0000 7f4138e00700 1 --2- 172.21.15.78:0/3834887676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4134102980 0x7f413419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:49:32.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.942+0000 7f4133e00700 1 --2- 172.21.15.78:0/3834887676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4134100de0 0x7f4134197160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f412400e9a0 tx=0x7f412400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:49:32.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.943+0000 7f4130c00700 1 -- 172.21.15.78:0/3834887676 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4124009790 con 0x7f4134100de0 2024-03-14T09:49:32.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.943+0000 7f413ae00700 1 -- 172.21.15.78:0/3834887676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413419f250 con 0x7f4134100de0 2024-03-14T09:49:32.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.943+0000 7f4130c00700 1 -- 172.21.15.78:0/3834887676 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f412400c810 con 0x7f4134100de0 2024-03-14T09:49:32.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.943+0000 7f4130c00700 1 -- 172.21.15.78:0/3834887676 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4124003c70 con 0x7f4134100de0 2024-03-14T09:49:32.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.943+0000 7f413ae00700 1 -- 172.21.15.78:0/3834887676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f413419f750 con 0x7f4134100de0 2024-03-14T09:49:32.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.944+0000 7f413ae00700 1 -- 172.21.15.78:0/3834887676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4134108260 con 0x7f4134100de0 2024-03-14T09:49:32.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.946+0000 7f4130c00700 1 -- 172.21.15.78:0/3834887676 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f412400cc80 con 0x7f4134100de0 2024-03-14T09:49:32.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.947+0000 7f4130c00700 1 --2- 172.21.15.78:0/3834887676 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f41180a0a90 0x7f41180a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:49:32.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.947+0000 7f4130c00700 1 -- 172.21.15.78:0/3834887676 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f4124014070 con 0x7f4134100de0 2024-03-14T09:49:32.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.947+0000 7f4133400700 1 --2- 172.21.15.78:0/3834887676 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f41180a0a90 0x7f41180a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:49:32.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.948+0000 7f4133400700 1 --2- 172.21.15.78:0/3834887676 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f41180a0a90 0x7f41180a2f50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4120003730 tx=0x7f412001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:49:32.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:32.951+0000 7f4130c00700 1 -- 172.21.15.78:0/3834887676 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4124095ba0 con 0x7f4134100de0 2024-03-14T09:49:32.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:32 smithi152 ceph-mon[48279]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-14T09:49:32.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:32 smithi152 ceph-mon[48279]: osdmap e35: 6 total, 5 up, 6 in 2024-03-14T09:49:32.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:32 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:49:32.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:32 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:49:33.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:32 smithi132 ceph-mon[47959]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-14T09:49:33.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:32 smithi132 ceph-mon[47959]: osdmap e35: 6 total, 5 up, 6 in 2024-03-14T09:49:33.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:32 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:49:33.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:32 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:49:33.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:32 smithi078 ceph-mon[38143]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-14T09:49:33.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:32 smithi078 ceph-mon[38143]: osdmap e35: 6 total, 5 up, 6 in 2024-03-14T09:49:33.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:32 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:49:33.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:32 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:49:33.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:33.163+0000 7f413ae00700 1 -- 172.21.15.78:0/3834887676 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f41340611f0 con 0x7f41180a0a90 2024-03-14T09:49:33.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:33 smithi152 ceph-mon[48279]: purged_snaps scrub starts 2024-03-14T09:49:33.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:33 smithi152 ceph-mon[48279]: purged_snaps scrub ok 2024-03-14T09:49:33.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:33 smithi152 ceph-mon[48279]: osd.5 [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776] boot 2024-03-14T09:49:33.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:33 smithi152 ceph-mon[48279]: osdmap e36: 6 total, 6 up, 6 in 2024-03-14T09:49:33.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:33 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:49:33.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:33 smithi152 ceph-mon[48279]: osdmap e37: 6 total, 6 up, 6 in 2024-03-14T09:49:34.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:33 smithi152 ceph-mon[48279]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-03-14T09:49:34.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:33 smithi152 ceph-mon[48279]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:49:34.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:33 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:49:34.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:33 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:49:34.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:33 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:34.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:33 smithi078 ceph-mon[38143]: purged_snaps scrub starts 2024-03-14T09:49:34.013 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:33 smithi078 ceph-mon[38143]: purged_snaps scrub ok 2024-03-14T09:49:34.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:33 smithi078 ceph-mon[38143]: osd.5 [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776] boot 2024-03-14T09:49:34.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:33 smithi078 ceph-mon[38143]: osdmap e36: 6 total, 6 up, 6 in 2024-03-14T09:49:34.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:33 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:49:34.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:33 smithi078 ceph-mon[38143]: osdmap e37: 6 total, 6 up, 6 in 2024-03-14T09:49:34.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:33 smithi078 ceph-mon[38143]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-03-14T09:49:34.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:33 smithi078 ceph-mon[38143]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:49:34.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:33 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:49:34.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:33 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:49:34.014 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:33 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:33 smithi132 ceph-mon[47959]: purged_snaps scrub starts 2024-03-14T09:49:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:33 smithi132 ceph-mon[47959]: purged_snaps scrub ok 2024-03-14T09:49:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:33 smithi132 ceph-mon[47959]: osd.5 [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776] boot 2024-03-14T09:49:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:33 smithi132 ceph-mon[47959]: osdmap e36: 6 total, 6 up, 6 in 2024-03-14T09:49:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:33 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:49:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:33 smithi132 ceph-mon[47959]: osdmap e37: 6 total, 6 up, 6 in 2024-03-14T09:49:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:33 smithi132 ceph-mon[47959]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-03-14T09:49:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:33 smithi132 ceph-mon[47959]: from='client.14313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:49:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:33 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:49:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:33 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:49:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:33 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:34.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:34 smithi152 ceph-mon[48279]: osdmap e38: 6 total, 6 up, 6 in 2024-03-14T09:49:35.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:34 smithi132 ceph-mon[47959]: osdmap e38: 6 total, 6 up, 6 in 2024-03-14T09:49:35.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:34 smithi078 ceph-mon[38143]: osdmap e38: 6 total, 6 up, 6 in 2024-03-14T09:49:35.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:35 smithi152 ceph-mon[48279]: osdmap e39: 6 total, 6 up, 6 in 2024-03-14T09:49:35.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:35 smithi152 ceph-mon[48279]: pgmap v116: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-14T09:49:35.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:35 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/735897261' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b9f16ed-3677-433d-976b-99b3b21f3479"}]: dispatch 2024-03-14T09:49:35.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:35 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/735897261' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b9f16ed-3677-433d-976b-99b3b21f3479"}]': finished 2024-03-14T09:49:35.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:35 smithi152 ceph-mon[48279]: osdmap e40: 7 total, 6 up, 7 in 2024-03-14T09:49:35.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:35 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:36.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:35 smithi132 ceph-mon[47959]: osdmap e39: 6 total, 6 up, 6 in 2024-03-14T09:49:36.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:35 smithi132 ceph-mon[47959]: pgmap v116: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-14T09:49:36.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:35 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/735897261' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b9f16ed-3677-433d-976b-99b3b21f3479"}]: dispatch 2024-03-14T09:49:36.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:35 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/735897261' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b9f16ed-3677-433d-976b-99b3b21f3479"}]': finished 2024-03-14T09:49:36.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:35 smithi132 ceph-mon[47959]: osdmap e40: 7 total, 6 up, 7 in 2024-03-14T09:49:36.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:35 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:36.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:35 smithi078 ceph-mon[38143]: osdmap e39: 6 total, 6 up, 6 in 2024-03-14T09:49:36.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:35 smithi078 ceph-mon[38143]: pgmap v116: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-14T09:49:36.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:35 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/735897261' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b9f16ed-3677-433d-976b-99b3b21f3479"}]: dispatch 2024-03-14T09:49:36.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:35 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/735897261' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b9f16ed-3677-433d-976b-99b3b21f3479"}]': finished 2024-03-14T09:49:36.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:35 smithi078 ceph-mon[38143]: osdmap e40: 7 total, 6 up, 7 in 2024-03-14T09:49:36.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:35 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:36.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:36 smithi152 ceph-mon[48279]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-14T09:49:36.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:36 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/4077716246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:49:37.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:36 smithi132 ceph-mon[47959]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-14T09:49:37.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:36 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/4077716246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:49:37.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:36 smithi078 ceph-mon[38143]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-14T09:49:37.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:36 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/4077716246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:49:37.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:37 smithi152 ceph-mon[48279]: pgmap v118: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-14T09:49:38.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:37 smithi132 ceph-mon[47959]: pgmap v118: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-14T09:49:38.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:37 smithi078 ceph-mon[38143]: pgmap v118: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-14T09:49:40.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:40 smithi078 ceph-mon[38143]: pgmap v119: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-14T09:49:40.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:40 smithi152 ceph-mon[48279]: pgmap v119: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-14T09:49:40.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:40 smithi132 ceph-mon[47959]: pgmap v119: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-14T09:49:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:41 smithi152 ceph-mon[48279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-14T09:49:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:41 smithi152 ceph-mon[48279]: Cluster is now healthy 2024-03-14T09:49:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:41 smithi132 ceph-mon[47959]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-14T09:49:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:41 smithi132 ceph-mon[47959]: Cluster is now healthy 2024-03-14T09:49:41.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:41 smithi078 ceph-mon[38143]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-14T09:49:41.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:41 smithi078 ceph-mon[38143]: Cluster is now healthy 2024-03-14T09:49:42.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:42 smithi078 ceph-mon[38143]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-14T09:49:42.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-14T09:49:42.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:42.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:42 smithi152 ceph-mon[48279]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-14T09:49:42.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-14T09:49:42.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:42.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:42 smithi132 ceph-mon[47959]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-14T09:49:42.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-14T09:49:42.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:43.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:43 smithi078 ceph-mon[38143]: Deploying daemon osd.6 on smithi078 2024-03-14T09:49:43.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:43 smithi152 ceph-mon[48279]: Deploying daemon osd.6 on smithi078 2024-03-14T09:49:43.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:43 smithi132 ceph-mon[47959]: Deploying daemon osd.6 on smithi078 2024-03-14T09:49:44.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:44 smithi078 ceph-mon[38143]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-14T09:49:44.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:44 smithi152 ceph-mon[48279]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-14T09:49:44.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:44 smithi132 ceph-mon[47959]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-14T09:49:46.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:46 smithi078 ceph-mon[38143]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-14T09:49:46.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:46 smithi152 ceph-mon[48279]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-14T09:49:46.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:46 smithi132 ceph-mon[47959]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-14T09:49:48.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:48.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:48.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:48.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:47 smithi152 ceph-mon[48279]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-14T09:49:48.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:48.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:48.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:48.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:47 smithi132 ceph-mon[47959]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-14T09:49:48.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:48.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:48.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:48.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:47 smithi078 ceph-mon[38143]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-14T09:49:49.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:49.911+0000 7f4130c00700 1 -- 172.21.15.78:0/3834887676 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f41340611f0 con 0x7f41180a0a90 2024-03-14T09:49:49.913 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 6 on host 'smithi078' 2024-03-14T09:49:49.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:49.914+0000 7f413ae00700 1 -- 172.21.15.78:0/3834887676 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f41180a0a90 msgr2=0x7f41180a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:49:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:49.914+0000 7f413ae00700 1 --2- 172.21.15.78:0/3834887676 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f41180a0a90 0x7f41180a2f50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4120003730 tx=0x7f412001a040 comp rx=0 tx=0).stop 2024-03-14T09:49:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:49.914+0000 7f413ae00700 1 -- 172.21.15.78:0/3834887676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4134100de0 msgr2=0x7f4134197160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:49:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:49.914+0000 7f413ae00700 1 --2- 172.21.15.78:0/3834887676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4134100de0 0x7f4134197160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f412400e9a0 tx=0x7f412400ecb0 comp rx=0 tx=0).stop 2024-03-14T09:49:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:49.914+0000 7f413ae00700 1 -- 172.21.15.78:0/3834887676 shutdown_connections 2024-03-14T09:49:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:49.914+0000 7f413ae00700 1 --2- 172.21.15.78:0/3834887676 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f41180a0a90 0x7f41180a2f50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:49.914+0000 7f413ae00700 1 --2- 172.21.15.78:0/3834887676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4134102980 0x7f413419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:49.914+0000 7f413ae00700 1 --2- 172.21.15.78:0/3834887676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4134100de0 0x7f4134197160 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:49.914+0000 7f413ae00700 1 --2- 172.21.15.78:0/3834887676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4134101fe0 0x7f41341976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:49.914+0000 7f413ae00700 1 -- 172.21.15.78:0/3834887676 >> 172.21.15.78:0/3834887676 conn(0x7f41340fc380 msgr2=0x7f41341053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:49:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:49.914+0000 7f413ae00700 1 -- 172.21.15.78:0/3834887676 shutdown_connections 2024-03-14T09:49:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:49:49.914+0000 7f413ae00700 1 -- 172.21.15.78:0/3834887676 wait complete. 2024-03-14T09:49:50.113 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:49 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:50.113 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:49 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:50.113 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:49 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:50.113 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:49 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:50.113 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:49 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:50.114 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:49 smithi078 ceph-mon[38143]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-14T09:49:50.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:49 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:50.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:49 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:50.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:49 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:50.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:49 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:50.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:49 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:50.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:49 smithi152 ceph-mon[48279]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-14T09:49:50.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:49 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:50.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:49 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:50.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:49 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:50.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:49 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:50.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:49 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:50.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:49 smithi132 ceph-mon[47959]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-14T09:49:50.304 DEBUG:teuthology.orchestra.run.smithi078:osd.6> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.6.service 2024-03-14T09:49:50.309 INFO:tasks.cephadm:Deploying osd.7 on smithi132 with /dev/vg_nvme/lv_2... 2024-03-14T09:49:50.309 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-14T09:49:50.381 INFO:journalctl@ceph.osd.6.smithi078.stdout:-- Logs begin at Thu 2024-03-14 09:30:54 UTC. -- 2024-03-14T09:49:50.539 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.b/config 2024-03-14T09:49:51.011 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:50 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:51.011 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:50 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:51.011 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:50 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:51.011 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 09:49:50 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T09:49:50.932+0000 7fc38f642640 -1 osd.6 0 log_to_monitors true 2024-03-14T09:49:51.016 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:50 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:51.016 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:50 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:51.016 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:50 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:51.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:50 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:51.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:50 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:51.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:50 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:49:51.873 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-14T09:49:51.873 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-14T09:49:51.873 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-03-14T09:49:51.873 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-03-14T09:49:51.873 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113463 s, 924 MB/s 2024-03-14T09:49:51.873 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-03-14T09:49:52.177 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:51 smithi132 ceph-mon[47959]: from='osd.6 [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-14T09:49:52.177 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:51 smithi132 ceph-mon[47959]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-14T09:49:52.218 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi132:vg_nvme/lv_2 2024-03-14T09:49:52.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:51 smithi152 ceph-mon[48279]: from='osd.6 [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-14T09:49:52.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:51 smithi152 ceph-mon[48279]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-14T09:49:52.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:51 smithi078 ceph-mon[38143]: from='osd.6 [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-14T09:49:52.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:51 smithi078 ceph-mon[38143]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-14T09:49:52.436 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.b/config 2024-03-14T09:49:53.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:52 smithi152 ceph-mon[48279]: from='osd.6 [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-14T09:49:53.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:52 smithi152 ceph-mon[48279]: osdmap e41: 7 total, 6 up, 7 in 2024-03-14T09:49:53.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:52 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:53.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:52 smithi152 ceph-mon[48279]: from='osd.6 [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:49:53.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:52 smithi132 ceph-mon[47959]: from='osd.6 [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-14T09:49:53.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:52 smithi132 ceph-mon[47959]: osdmap e41: 7 total, 6 up, 7 in 2024-03-14T09:49:53.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:52 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:53.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:52 smithi132 ceph-mon[47959]: from='osd.6 [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:49:53.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 09:49:52 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T09:49:52.905+0000 7fc383c00700 -1 osd.6 0 waiting for initial osdmap 2024-03-14T09:49:53.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 09:49:52 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T09:49:52.916+0000 7fc37a400700 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T09:49:53.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:52 smithi078 ceph-mon[38143]: from='osd.6 [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-14T09:49:53.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:52 smithi078 ceph-mon[38143]: osdmap e41: 7 total, 6 up, 7 in 2024-03-14T09:49:53.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:52 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:53.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:52 smithi078 ceph-mon[38143]: from='osd.6 [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:49:53.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.452+0000 7f6871566700 1 -- 172.21.15.132:0/520996518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f686c106aa0 msgr2=0x7f686c108f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:49:53.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.452+0000 7f6871566700 1 --2- 172.21.15.132:0/520996518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f686c106aa0 0x7f686c108f10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f686400b0a0 tx=0x7f686400b3b0 comp rx=0 tx=0).stop 2024-03-14T09:49:53.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.452+0000 7f6871566700 1 -- 172.21.15.132:0/520996518 shutdown_connections 2024-03-14T09:49:53.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.452+0000 7f6871566700 1 --2- 172.21.15.132:0/520996518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f686c106aa0 0x7f686c108f10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:53.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.452+0000 7f6871566700 1 --2- 172.21.15.132:0/520996518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f686c101840 0x7f686c103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:53.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.452+0000 7f6871566700 1 --2- 172.21.15.132:0/520996518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f686c104170 0x7f686c106560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:53.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.452+0000 7f6871566700 1 -- 172.21.15.132:0/520996518 >> 172.21.15.132:0/520996518 conn(0x7f686c0fb270 msgr2=0x7f686c0fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:49:53.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.452+0000 7f6871566700 1 -- 172.21.15.132:0/520996518 shutdown_connections 2024-03-14T09:49:53.452 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.452+0000 7f6871566700 1 -- 172.21.15.132:0/520996518 wait complete. 2024-03-14T09:49:53.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.453+0000 7f6871566700 1 Processor -- start 2024-03-14T09:49:53.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.453+0000 7f6871566700 1 -- start start 2024-03-14T09:49:53.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f6871566700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f686c101840 0x7f686c1971d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:49:53.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f6871566700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f686c104170 0x7f686c197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:49:53.454 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f6871566700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f686c106aa0 0x7f686c19e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:49:53.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f6871566700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686c19ed10 con 0x7f686c104170 2024-03-14T09:49:53.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f6871566700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686c19ee50 con 0x7f686c106aa0 2024-03-14T09:49:53.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f6871566700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686c19efe0 con 0x7f686c101840 2024-03-14T09:49:53.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f686b7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f686c106aa0 0x7f686c19e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:49:53.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f686b7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f686c106aa0 0x7f686c19e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:48984/0 (socket says 172.21.15.132:48984) 2024-03-14T09:49:53.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f686b7fe700 1 -- 172.21.15.132:0/2245049266 learned_addr learned my addr 172.21.15.132:0/2245049266 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-14T09:49:53.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f686affd700 1 --2- 172.21.15.132:0/2245049266 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f686c101840 0x7f686c1971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:49:53.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f686a7fc700 1 --2- 172.21.15.132:0/2245049266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f686c104170 0x7f686c197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:49:53.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f686b7fe700 1 -- 172.21.15.132:0/2245049266 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f686c101840 msgr2=0x7f686c1971d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:49:53.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f686b7fe700 1 --2- 172.21.15.132:0/2245049266 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f686c101840 0x7f686c1971d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:53.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f686b7fe700 1 -- 172.21.15.132:0/2245049266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f686c104170 msgr2=0x7f686c197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:49:53.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f686b7fe700 1 --2- 172.21.15.132:0/2245049266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f686c104170 0x7f686c197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:49:53.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f686b7fe700 1 -- 172.21.15.132:0/2245049266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6864009d00 con 0x7f686c106aa0 2024-03-14T09:49:53.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f686b7fe700 1 --2- 172.21.15.132:0/2245049266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f686c106aa0 0x7f686c19e740 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f686400b070 tx=0x7f68640094c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:49:53.455 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f6863fff700 1 -- 172.21.15.132:0/2245049266 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686400e040 con 0x7f686c106aa0 2024-03-14T09:49:53.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f6863fff700 1 -- 172.21.15.132:0/2245049266 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f686401d940 con 0x7f686c106aa0 2024-03-14T09:49:53.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f6871566700 1 -- 172.21.15.132:0/2245049266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f686c19f260 con 0x7f686c106aa0 2024-03-14T09:49:53.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f6863fff700 1 -- 172.21.15.132:0/2245049266 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686401b9b0 con 0x7f686c106aa0 2024-03-14T09:49:53.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.454+0000 7f6871566700 1 -- 172.21.15.132:0/2245049266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f686c19f680 con 0x7f686c106aa0 2024-03-14T09:49:53.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.455+0000 7f6871566700 1 -- 172.21.15.132:0/2245049266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f686c0fcdf0 con 0x7f686c106aa0 2024-03-14T09:49:53.456 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.457+0000 7f6863fff700 1 -- 172.21.15.132:0/2245049266 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f686401dab0 con 0x7f686c106aa0 2024-03-14T09:49:53.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.458+0000 7f6863fff700 1 --2- 172.21.15.132:0/2245049266 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f68580a0a90 0x7f68580a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:49:53.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.458+0000 7f6863fff700 1 -- 172.21.15.132:0/2245049266 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4685+0+0 (secure 0 0 0) 0x7f68640cd590 con 0x7f686c106aa0 2024-03-14T09:49:53.457 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.458+0000 7f686affd700 1 --2- 172.21.15.132:0/2245049266 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f68580a0a90 0x7f68580a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:49:53.458 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.458+0000 7f686affd700 1 --2- 172.21.15.132:0/2245049266 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f68580a0a90 0x7f68580a2f50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f685c005950 tx=0x7f685c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:49:53.460 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.461+0000 7f6863fff700 1 -- 172.21.15.132:0/2245049266 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6864060960 con 0x7f686c106aa0 2024-03-14T09:49:53.658 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:49:53.658+0000 7f6871566700 1 -- 172.21.15.132:0/2245049266 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f686c02cd20 con 0x7f68580a0a90 2024-03-14T09:49:54.177 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: from='osd.6 [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-14T09:49:54.178 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: osdmap e42: 7 total, 6 up, 7 in 2024-03-14T09:49:54.178 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:54.178 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:54.178 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: Detected new or changed devices on smithi078 2024-03-14T09:49:54.178 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:54.178 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:54.178 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:54.178 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:54.178 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:54.179 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi078 to 5756M 2024-03-14T09:49:54.179 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:54.179 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:54.179 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:54.179 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-03-14T09:49:54.179 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:54.179 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:49:54.179 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:49:54.179 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:54.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: from='osd.6 [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-14T09:49:54.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: osdmap e42: 7 total, 6 up, 7 in 2024-03-14T09:49:54.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:54.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:54.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: Detected new or changed devices on smithi078 2024-03-14T09:49:54.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:54.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:54.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:54.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:54.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:54.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi078 to 5756M 2024-03-14T09:49:54.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:54.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:54.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:54.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-03-14T09:49:54.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:54.251 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:49:54.251 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:49:54.251 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:54.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: from='osd.6 [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-14T09:49:54.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: osdmap e42: 7 total, 6 up, 7 in 2024-03-14T09:49:54.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:54.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:54.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: Detected new or changed devices on smithi078 2024-03-14T09:49:54.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:54.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:54.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:54.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:54.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:49:54.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi078 to 5756M 2024-03-14T09:49:54.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:54.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:54.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:49:54.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-03-14T09:49:54.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:49:54.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:49:54.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:49:54.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:49:55.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:54 smithi152 ceph-mon[48279]: purged_snaps scrub starts 2024-03-14T09:49:55.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:54 smithi152 ceph-mon[48279]: purged_snaps scrub ok 2024-03-14T09:49:55.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:54 smithi152 ceph-mon[48279]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:49:55.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:54 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:55.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:54 smithi152 ceph-mon[48279]: osd.6 [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467] boot 2024-03-14T09:49:55.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:54 smithi152 ceph-mon[48279]: osdmap e43: 7 total, 7 up, 7 in 2024-03-14T09:49:55.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:54 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:55.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:54 smithi132 ceph-mon[47959]: purged_snaps scrub starts 2024-03-14T09:49:55.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:54 smithi132 ceph-mon[47959]: purged_snaps scrub ok 2024-03-14T09:49:55.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:54 smithi132 ceph-mon[47959]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:49:55.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:54 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:55.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:54 smithi132 ceph-mon[47959]: osd.6 [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467] boot 2024-03-14T09:49:55.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:54 smithi132 ceph-mon[47959]: osdmap e43: 7 total, 7 up, 7 in 2024-03-14T09:49:55.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:54 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:55.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:54 smithi078 ceph-mon[38143]: purged_snaps scrub starts 2024-03-14T09:49:55.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:54 smithi078 ceph-mon[38143]: purged_snaps scrub ok 2024-03-14T09:49:55.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:54 smithi078 ceph-mon[38143]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:49:55.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:54 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:55.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:54 smithi078 ceph-mon[38143]: osd.6 [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467] boot 2024-03-14T09:49:55.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:54 smithi078 ceph-mon[38143]: osdmap e43: 7 total, 7 up, 7 in 2024-03-14T09:49:55.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:54 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:49:56.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:56 smithi078 ceph-mon[38143]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:49:56.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:56 smithi078 ceph-mon[38143]: osdmap e44: 7 total, 7 up, 7 in 2024-03-14T09:49:56.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:56 smithi078 ceph-mon[38143]: from='client.? 172.21.15.132:0/1525757363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f097c83a-d182-409b-bfe8-5bf6a6627fef"}]: dispatch 2024-03-14T09:49:56.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:56 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f097c83a-d182-409b-bfe8-5bf6a6627fef"}]: dispatch 2024-03-14T09:49:56.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:56 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f097c83a-d182-409b-bfe8-5bf6a6627fef"}]': finished 2024-03-14T09:49:56.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:56 smithi078 ceph-mon[38143]: osdmap e45: 8 total, 7 up, 8 in 2024-03-14T09:49:56.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:56 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:49:56.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:56 smithi152 ceph-mon[48279]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:49:56.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:56 smithi152 ceph-mon[48279]: osdmap e44: 7 total, 7 up, 7 in 2024-03-14T09:49:56.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:56 smithi152 ceph-mon[48279]: from='client.? 172.21.15.132:0/1525757363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f097c83a-d182-409b-bfe8-5bf6a6627fef"}]: dispatch 2024-03-14T09:49:56.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:56 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f097c83a-d182-409b-bfe8-5bf6a6627fef"}]: dispatch 2024-03-14T09:49:56.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:56 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f097c83a-d182-409b-bfe8-5bf6a6627fef"}]': finished 2024-03-14T09:49:56.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:56 smithi152 ceph-mon[48279]: osdmap e45: 8 total, 7 up, 8 in 2024-03-14T09:49:56.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:56 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:49:56.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:56 smithi132 ceph-mon[47959]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:49:56.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:56 smithi132 ceph-mon[47959]: osdmap e44: 7 total, 7 up, 7 in 2024-03-14T09:49:56.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:56 smithi132 ceph-mon[47959]: from='client.? 172.21.15.132:0/1525757363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f097c83a-d182-409b-bfe8-5bf6a6627fef"}]: dispatch 2024-03-14T09:49:56.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:56 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f097c83a-d182-409b-bfe8-5bf6a6627fef"}]: dispatch 2024-03-14T09:49:56.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:56 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f097c83a-d182-409b-bfe8-5bf6a6627fef"}]': finished 2024-03-14T09:49:56.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:56 smithi132 ceph-mon[47959]: osdmap e45: 8 total, 7 up, 8 in 2024-03-14T09:49:56.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:56 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:49:57.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:57 smithi078 ceph-mon[38143]: from='client.? 172.21.15.132:0/3618457815' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:49:57.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:57 smithi152 ceph-mon[48279]: from='client.? 172.21.15.132:0/3618457815' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:49:57.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:57 smithi132 ceph-mon[47959]: from='client.? 172.21.15.132:0/3618457815' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:49:58.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:49:58 smithi078 ceph-mon[38143]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:49:58.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:49:58 smithi152 ceph-mon[48279]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:49:58.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:49:58 smithi132 ceph-mon[47959]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:00.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:00 smithi132 ceph-mon[47959]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:00.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:00 smithi132 ceph-mon[47959]: overall HEALTH_OK 2024-03-14T09:50:00.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:00 smithi078 ceph-mon[38143]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:00.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:00 smithi078 ceph-mon[38143]: overall HEALTH_OK 2024-03-14T09:50:00.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:00 smithi152 ceph-mon[48279]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:00.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:00 smithi152 ceph-mon[48279]: overall HEALTH_OK 2024-03-14T09:50:02.482 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:02 smithi132 ceph-mon[47959]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:02.482 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:02 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-14T09:50:02.482 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:02 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:02.482 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:02 smithi132 ceph-mon[47959]: Deploying daemon osd.7 on smithi132 2024-03-14T09:50:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:02 smithi152 ceph-mon[48279]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:02 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-14T09:50:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:02 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:02 smithi152 ceph-mon[48279]: Deploying daemon osd.7 on smithi132 2024-03-14T09:50:02.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:02 smithi078 ceph-mon[38143]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:02.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-14T09:50:02.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:02.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:02 smithi078 ceph-mon[38143]: Deploying daemon osd.7 on smithi132 2024-03-14T09:50:03.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:03 smithi132 ceph-mon[47959]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:03.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:03 smithi152 ceph-mon[48279]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:04.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:03 smithi078 ceph-mon[38143]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:06.142 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:06 smithi132 ceph-mon[47959]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:06.143 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:06 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:06.143 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:06 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:06.143 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:06 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:06.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:06 smithi078 ceph-mon[38143]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:06.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:06 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:06.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:06 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:06.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:06 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:06.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:06 smithi152 ceph-mon[48279]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:06.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:06 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:06.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:06 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:06.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:06 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:07.262 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:07.261+0000 7f6863fff700 1 -- 172.21.15.132:0/2245049266 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f686c02cd20 con 0x7f68580a0a90 2024-03-14T09:50:07.263 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 7 on host 'smithi132' 2024-03-14T09:50:07.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:07.264+0000 7f6871566700 1 -- 172.21.15.132:0/2245049266 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f68580a0a90 msgr2=0x7f68580a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:50:07.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:07.264+0000 7f6871566700 1 --2- 172.21.15.132:0/2245049266 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f68580a0a90 0x7f68580a2f50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f685c005950 tx=0x7f685c00b410 comp rx=0 tx=0).stop 2024-03-14T09:50:07.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:07.264+0000 7f6871566700 1 -- 172.21.15.132:0/2245049266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f686c106aa0 msgr2=0x7f686c19e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:50:07.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:07.264+0000 7f6871566700 1 --2- 172.21.15.132:0/2245049266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f686c106aa0 0x7f686c19e740 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f686400b070 tx=0x7f68640094c0 comp rx=0 tx=0).stop 2024-03-14T09:50:07.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:07.264+0000 7f6871566700 1 -- 172.21.15.132:0/2245049266 shutdown_connections 2024-03-14T09:50:07.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:07.264+0000 7f6871566700 1 --2- 172.21.15.132:0/2245049266 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f68580a0a90 0x7f68580a2f50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:07.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:07.264+0000 7f6871566700 1 --2- 172.21.15.132:0/2245049266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f686c106aa0 0x7f686c19e740 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:07.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:07.264+0000 7f6871566700 1 --2- 172.21.15.132:0/2245049266 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f686c101840 0x7f686c1971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:07.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:07.264+0000 7f6871566700 1 --2- 172.21.15.132:0/2245049266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f686c104170 0x7f686c197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:07.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:07.264+0000 7f6871566700 1 -- 172.21.15.132:0/2245049266 >> 172.21.15.132:0/2245049266 conn(0x7f686c0fb270 msgr2=0x7f686c102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:50:07.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:07.264+0000 7f6871566700 1 -- 172.21.15.132:0/2245049266 shutdown_connections 2024-03-14T09:50:07.265 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:07.264+0000 7f6871566700 1 -- 172.21.15.132:0/2245049266 wait complete. 2024-03-14T09:50:07.472 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:07.472 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:07.472 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:07.472 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:07.472 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:07.472 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:07.472 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:07.472 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:07.664 DEBUG:teuthology.orchestra.run.smithi132:osd.7> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.7.service 2024-03-14T09:50:07.666 INFO:tasks.cephadm:Deploying osd.8 on smithi152 with /dev/vg_nvme/lv_2... 2024-03-14T09:50:07.666 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-14T09:50:07.715 INFO:journalctl@ceph.osd.7.smithi132.stdout:-- Logs begin at Thu 2024-03-14 09:24:09 UTC. -- 2024-03-14T09:50:07.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:07.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:07.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:07.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:07.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:07.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:07.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:07.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:07.914 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.c/config 2024-03-14T09:50:08.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:08 smithi132 ceph-mon[47959]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:08.735 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:08 smithi152 ceph-mon[48279]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:08.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:08 smithi078 ceph-mon[38143]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:09.296 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-14T09:50:09.297 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-14T09:50:09.297 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-14T09:50:09.297 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-14T09:50:09.297 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011295 s, 928 MB/s 2024-03-14T09:50:09.297 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-14T09:50:09.338 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 09:50:09 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T09:50:09.059+0000 7fc27ae2c640 -1 osd.7 0 log_to_monitors true 2024-03-14T09:50:09.637 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:09 smithi152 ceph-mon[48279]: from='osd.7 [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-14T09:50:09.637 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:09 smithi152 ceph-mon[48279]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-14T09:50:09.646 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:09 smithi132 ceph-mon[47959]: from='osd.7 [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-14T09:50:09.647 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:09 smithi132 ceph-mon[47959]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-14T09:50:09.697 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_2 2024-03-14T09:50:09.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:09 smithi078 ceph-mon[38143]: from='osd.7 [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-14T09:50:09.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:09 smithi078 ceph-mon[38143]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-14T09:50:09.916 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.c/config 2024-03-14T09:50:10.636 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:10 smithi152 ceph-mon[48279]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:10.636 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:10 smithi152 ceph-mon[48279]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-14T09:50:10.636 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:10 smithi152 ceph-mon[48279]: osdmap e46: 8 total, 7 up, 8 in 2024-03-14T09:50:10.636 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:50:10.637 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:10 smithi152 ceph-mon[48279]: from='osd.7 [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:50:10.637 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:10 smithi152 ceph-mon[48279]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:50:10.637 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:10.637 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:10.637 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:10.637 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:10.637 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:10.637 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:10.637 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:10.637 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:10.637 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:10.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T09:50:10.356+0000 7fc270490700 -1 osd.7 0 waiting for initial osdmap 2024-03-14T09:50:10.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T09:50:10.368+0000 7fc26c285700 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T09:50:10.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-mon[47959]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:10.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-mon[47959]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-14T09:50:10.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-mon[47959]: osdmap e46: 8 total, 7 up, 8 in 2024-03-14T09:50:10.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:50:10.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-mon[47959]: from='osd.7 [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:50:10.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-mon[47959]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:50:10.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:10.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:10.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:10.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:10.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:10.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:10.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:10.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:10.772 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:10.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:10 smithi078 ceph-mon[38143]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:10.805 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:10 smithi078 ceph-mon[38143]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-14T09:50:10.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:10 smithi078 ceph-mon[38143]: osdmap e46: 8 total, 7 up, 8 in 2024-03-14T09:50:10.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:50:10.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:10 smithi078 ceph-mon[38143]: from='osd.7 [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:50:10.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:10 smithi078 ceph-mon[38143]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:50:10.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:10.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:10.806 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:10.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:10.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:10.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:10.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:10.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:10.807 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:11.015 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.012+0000 7ff4e0edc700 1 -- 172.21.15.152:0/2208137246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff4dc100dc0 msgr2=0x7ff4dc1011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:50:11.015 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.012+0000 7ff4e0edc700 1 --2- 172.21.15.152:0/2208137246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff4dc100dc0 0x7ff4dc1011e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc009990 tx=0x7ff4cc009ca0 comp rx=0 tx=0).stop 2024-03-14T09:50:11.015 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.012+0000 7ff4e0edc700 1 -- 172.21.15.152:0/2208137246 shutdown_connections 2024-03-14T09:50:11.015 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.012+0000 7ff4e0edc700 1 --2- 172.21.15.152:0/2208137246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4dc102960 0x7ff4dc109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:11.015 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.012+0000 7ff4e0edc700 1 --2- 172.21.15.152:0/2208137246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff4dc100dc0 0x7ff4dc1011e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:11.015 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.012+0000 7ff4e0edc700 1 --2- 172.21.15.152:0/2208137246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4dc101fc0 0x7ff4dc102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:11.015 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.012+0000 7ff4e0edc700 1 -- 172.21.15.152:0/2208137246 >> 172.21.15.152:0/2208137246 conn(0x7ff4dc0fc360 msgr2=0x7ff4dc0fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:50:11.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.012+0000 7ff4e0edc700 1 -- 172.21.15.152:0/2208137246 shutdown_connections 2024-03-14T09:50:11.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.012+0000 7ff4e0edc700 1 -- 172.21.15.152:0/2208137246 wait complete. 2024-03-14T09:50:11.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.012+0000 7ff4e0edc700 1 Processor -- start 2024-03-14T09:50:11.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4e0edc700 1 -- start start 2024-03-14T09:50:11.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4e0edc700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4dc100dc0 0x7ff4dc106b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:50:11.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4e0edc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4dc101fc0 0x7ff4dc107040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:50:11.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4e0edc700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff4dc102960 0x7ff4dc103680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:50:11.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4e0edc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc103e20 con 0x7ff4dc101fc0 2024-03-14T09:50:11.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4e0edc700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc103f90 con 0x7ff4dc100dc0 2024-03-14T09:50:11.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4e0edc700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4dc104150 con 0x7ff4dc102960 2024-03-14T09:50:11.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4dad9d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff4dc102960 0x7ff4dc103680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:50:11.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4dad9d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff4dc102960 0x7ff4dc103680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:50204/0 (socket says 172.21.15.152:50204) 2024-03-14T09:50:11.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4dad9d700 1 -- 172.21.15.152:0/4183013357 learned_addr learned my addr 172.21.15.152:0/4183013357 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-14T09:50:11.016 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4da59c700 1 --2- 172.21.15.152:0/4183013357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4dc100dc0 0x7ff4dc106b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:50:11.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4d9d9b700 1 --2- 172.21.15.152:0/4183013357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4dc101fc0 0x7ff4dc107040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:50:11.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4dad9d700 1 -- 172.21.15.152:0/4183013357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4dc100dc0 msgr2=0x7ff4dc106b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:50:11.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4dad9d700 1 --2- 172.21.15.152:0/4183013357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4dc100dc0 0x7ff4dc106b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:11.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4dad9d700 1 -- 172.21.15.152:0/4183013357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4dc101fc0 msgr2=0x7ff4dc107040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:50:11.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4dad9d700 1 --2- 172.21.15.152:0/4183013357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4dc101fc0 0x7ff4dc107040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:11.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4dad9d700 1 -- 172.21.15.152:0/4183013357 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc0095f0 con 0x7ff4dc102960 2024-03-14T09:50:11.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4dad9d700 1 --2- 172.21.15.152:0/4183013357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff4dc102960 0x7ff4dc103680 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4007d70 tx=0x7ff4d400a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:50:11.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4cb7fe700 1 -- 172.21.15.152:0/4183013357 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d400ac90 con 0x7ff4dc102960 2024-03-14T09:50:11.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.014+0000 7ff4cb7fe700 1 -- 172.21.15.152:0/4183013357 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4d400adf0 con 0x7ff4dc102960 2024-03-14T09:50:11.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.013+0000 7ff4e0edc700 1 -- 172.21.15.152:0/4183013357 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4dc1a5850 con 0x7ff4dc102960 2024-03-14T09:50:11.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.014+0000 7ff4cb7fe700 1 -- 172.21.15.152:0/4183013357 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d40115e0 con 0x7ff4dc102960 2024-03-14T09:50:11.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.014+0000 7ff4e0edc700 1 -- 172.21.15.152:0/4183013357 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4dc1a5cd0 con 0x7ff4dc102960 2024-03-14T09:50:11.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.014+0000 7ff4e0edc700 1 -- 172.21.15.152:0/4183013357 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4dc04ea80 con 0x7ff4dc102960 2024-03-14T09:50:11.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.016+0000 7ff4cb7fe700 1 -- 172.21.15.152:0/4183013357 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7ff4d4004ad0 con 0x7ff4dc102960 2024-03-14T09:50:11.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.016+0000 7ff4cb7fe700 1 --2- 172.21.15.152:0/4183013357 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff4c40a0a90 0x7ff4c40a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:50:11.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.016+0000 7ff4cb7fe700 1 -- 172.21.15.152:0/4183013357 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5117+0+0 (secure 0 0 0) 0x7ff4d40cb970 con 0x7ff4dc102960 2024-03-14T09:50:11.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.017+0000 7ff4da59c700 1 --2- 172.21.15.152:0/4183013357 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff4c40a0a90 0x7ff4c40a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:50:11.019 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.017+0000 7ff4da59c700 1 --2- 172.21.15.152:0/4183013357 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff4c40a0a90 0x7ff4c40a2f50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00b5c0 tx=0x7ff4cc003860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:50:11.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.020+0000 7ff4cb7fe700 1 -- 172.21.15.152:0/4183013357 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff4d4095fc0 con 0x7ff4dc102960 2024-03-14T09:50:11.219 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:11.218+0000 7ff4e0edc700 1 -- 172.21.15.152:0/4183013357 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4dc108eb0 con 0x7ff4c40a0a90 2024-03-14T09:50:11.622 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:11 smithi152 ceph-mon[48279]: Detected new or changed devices on smithi132 2024-03-14T09:50:11.623 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:11 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi132 to 5704M 2024-03-14T09:50:11.623 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:11 smithi152 ceph-mon[48279]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-14T09:50:11.623 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:11 smithi152 ceph-mon[48279]: osdmap e47: 8 total, 7 up, 8 in 2024-03-14T09:50:11.623 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:11 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:50:11.623 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:11 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:50:11.623 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:11 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:50:11.623 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:11 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:50:11.623 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:11 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:11.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:11 smithi132 ceph-mon[47959]: Detected new or changed devices on smithi132 2024-03-14T09:50:11.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:11 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi132 to 5704M 2024-03-14T09:50:11.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:11 smithi132 ceph-mon[47959]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-14T09:50:11.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:11 smithi132 ceph-mon[47959]: osdmap e47: 8 total, 7 up, 8 in 2024-03-14T09:50:11.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:11 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:50:11.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:11 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:50:11.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:11 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:50:11.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:11 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:50:11.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:11 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:11.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:11 smithi078 ceph-mon[38143]: Detected new or changed devices on smithi132 2024-03-14T09:50:11.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:11 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi132 to 5704M 2024-03-14T09:50:11.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:11 smithi078 ceph-mon[38143]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-14T09:50:11.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:11 smithi078 ceph-mon[38143]: osdmap e47: 8 total, 7 up, 8 in 2024-03-14T09:50:11.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:11 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:50:11.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:11 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:50:11.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:11 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:50:11.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:11 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:50:11.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:11 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:12.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:12 smithi152 ceph-mon[48279]: purged_snaps scrub starts 2024-03-14T09:50:12.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:12 smithi152 ceph-mon[48279]: purged_snaps scrub ok 2024-03-14T09:50:12.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:12 smithi152 ceph-mon[48279]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:12.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:12 smithi152 ceph-mon[48279]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:50:12.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:12 smithi152 ceph-mon[48279]: osd.7 [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712] boot 2024-03-14T09:50:12.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:12 smithi152 ceph-mon[48279]: osdmap e48: 8 total, 8 up, 8 in 2024-03-14T09:50:12.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:12 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:50:12.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:12 smithi152 ceph-mon[48279]: osdmap e49: 8 total, 8 up, 8 in 2024-03-14T09:50:12.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:12 smithi132 ceph-mon[47959]: purged_snaps scrub starts 2024-03-14T09:50:12.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:12 smithi132 ceph-mon[47959]: purged_snaps scrub ok 2024-03-14T09:50:12.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:12 smithi132 ceph-mon[47959]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:12.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:12 smithi132 ceph-mon[47959]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:50:12.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:12 smithi132 ceph-mon[47959]: osd.7 [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712] boot 2024-03-14T09:50:12.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:12 smithi132 ceph-mon[47959]: osdmap e48: 8 total, 8 up, 8 in 2024-03-14T09:50:12.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:12 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:50:12.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:12 smithi132 ceph-mon[47959]: osdmap e49: 8 total, 8 up, 8 in 2024-03-14T09:50:12.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:12 smithi078 ceph-mon[38143]: purged_snaps scrub starts 2024-03-14T09:50:12.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:12 smithi078 ceph-mon[38143]: purged_snaps scrub ok 2024-03-14T09:50:12.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:12 smithi078 ceph-mon[38143]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-03-14T09:50:12.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:12 smithi078 ceph-mon[38143]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:50:12.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:12 smithi078 ceph-mon[38143]: osd.7 [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712] boot 2024-03-14T09:50:12.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:12 smithi078 ceph-mon[38143]: osdmap e48: 8 total, 8 up, 8 in 2024-03-14T09:50:12.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:12 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:50:12.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:12 smithi078 ceph-mon[38143]: osdmap e49: 8 total, 8 up, 8 in 2024-03-14T09:50:13.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:13 smithi152 ceph-mon[48279]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:13.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:13 smithi152 ceph-mon[48279]: from='client.? 172.21.15.152:0/274377908' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "238d2afc-7d81-4089-98be-2852f0504b6c"}]: dispatch 2024-03-14T09:50:13.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:13 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "238d2afc-7d81-4089-98be-2852f0504b6c"}]: dispatch 2024-03-14T09:50:13.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:13 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "238d2afc-7d81-4089-98be-2852f0504b6c"}]': finished 2024-03-14T09:50:14.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:13 smithi152 ceph-mon[48279]: osdmap e50: 9 total, 8 up, 9 in 2024-03-14T09:50:14.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:13 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:14.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:13 smithi132 ceph-mon[47959]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:14.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:13 smithi132 ceph-mon[47959]: from='client.? 172.21.15.152:0/274377908' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "238d2afc-7d81-4089-98be-2852f0504b6c"}]: dispatch 2024-03-14T09:50:14.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:13 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "238d2afc-7d81-4089-98be-2852f0504b6c"}]: dispatch 2024-03-14T09:50:14.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:13 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "238d2afc-7d81-4089-98be-2852f0504b6c"}]': finished 2024-03-14T09:50:14.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:13 smithi132 ceph-mon[47959]: osdmap e50: 9 total, 8 up, 9 in 2024-03-14T09:50:14.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:13 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:14.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:13 smithi078 ceph-mon[38143]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:14.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:13 smithi078 ceph-mon[38143]: from='client.? 172.21.15.152:0/274377908' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "238d2afc-7d81-4089-98be-2852f0504b6c"}]: dispatch 2024-03-14T09:50:14.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:13 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "238d2afc-7d81-4089-98be-2852f0504b6c"}]: dispatch 2024-03-14T09:50:14.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:13 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "238d2afc-7d81-4089-98be-2852f0504b6c"}]': finished 2024-03-14T09:50:14.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:13 smithi078 ceph-mon[38143]: osdmap e50: 9 total, 8 up, 9 in 2024-03-14T09:50:14.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:13 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:14.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:14 smithi152 ceph-mon[48279]: from='client.? 172.21.15.152:0/3886495030' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:50:14.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:14 smithi152 ceph-mon[48279]: osdmap e51: 9 total, 8 up, 9 in 2024-03-14T09:50:14.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:14 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:15.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:14 smithi132 ceph-mon[47959]: from='client.? 172.21.15.152:0/3886495030' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:50:15.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:14 smithi132 ceph-mon[47959]: osdmap e51: 9 total, 8 up, 9 in 2024-03-14T09:50:15.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:14 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:15.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:14 smithi078 ceph-mon[38143]: from='client.? 172.21.15.152:0/3886495030' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:50:15.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:14 smithi078 ceph-mon[38143]: osdmap e51: 9 total, 8 up, 9 in 2024-03-14T09:50:15.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:14 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:16.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:16 smithi152 ceph-mon[48279]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:16.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:16 smithi152 ceph-mon[48279]: osdmap e52: 9 total, 8 up, 9 in 2024-03-14T09:50:16.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:16 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:16.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:16 smithi132 ceph-mon[47959]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:16.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:16 smithi132 ceph-mon[47959]: osdmap e52: 9 total, 8 up, 9 in 2024-03-14T09:50:16.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:16 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:16.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:16 smithi078 ceph-mon[38143]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:16.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:16 smithi078 ceph-mon[38143]: osdmap e52: 9 total, 8 up, 9 in 2024-03-14T09:50:16.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:16 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:18.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:18 smithi152 ceph-mon[48279]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:18.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:18 smithi132 ceph-mon[47959]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:18.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:18 smithi078 ceph-mon[38143]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:20.491 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:20 smithi152 ceph-mon[48279]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:20.491 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:20 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-14T09:50:20.491 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:20 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:20.491 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:20 smithi152 ceph-mon[48279]: Deploying daemon osd.8 on smithi152 2024-03-14T09:50:20.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:20 smithi132 ceph-mon[47959]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:20.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:20 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-14T09:50:20.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:20 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:20.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:20 smithi132 ceph-mon[47959]: Deploying daemon osd.8 on smithi152 2024-03-14T09:50:20.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:20 smithi078 ceph-mon[38143]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:20.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:20 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-14T09:50:20.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:20 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:20.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:20 smithi078 ceph-mon[38143]: Deploying daemon osd.8 on smithi152 2024-03-14T09:50:22.411 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:22 smithi152 ceph-mon[48279]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:22.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:22 smithi132 ceph-mon[47959]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:22.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:22 smithi078 ceph-mon[38143]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:24.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:24 smithi152 ceph-mon[48279]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:24.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:24 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:24.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:24 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:24.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:24 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:24.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:24 smithi132 ceph-mon[47959]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:24.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:24 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:24.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:24 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:24.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:24 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:24.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:24 smithi078 ceph-mon[38143]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:24.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:24 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:24.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:24 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:24.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:24 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:25.475 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:25.476+0000 7ff4cb7fe700 1 -- 172.21.15.152:0/4183013357 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff4dc108eb0 con 0x7ff4c40a0a90 2024-03-14T09:50:25.476 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 8 on host 'smithi152' 2024-03-14T09:50:25.478 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:25.478+0000 7ff4e0edc700 1 -- 172.21.15.152:0/4183013357 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff4c40a0a90 msgr2=0x7ff4c40a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:50:25.478 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:25.478+0000 7ff4e0edc700 1 --2- 172.21.15.152:0/4183013357 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff4c40a0a90 0x7ff4c40a2f50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00b5c0 tx=0x7ff4cc003860 comp rx=0 tx=0).stop 2024-03-14T09:50:25.479 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:25.478+0000 7ff4e0edc700 1 -- 172.21.15.152:0/4183013357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff4dc102960 msgr2=0x7ff4dc103680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:50:25.479 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:25.478+0000 7ff4e0edc700 1 --2- 172.21.15.152:0/4183013357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff4dc102960 0x7ff4dc103680 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4007d70 tx=0x7ff4d400a340 comp rx=0 tx=0).stop 2024-03-14T09:50:25.479 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:25.478+0000 7ff4e0edc700 1 -- 172.21.15.152:0/4183013357 shutdown_connections 2024-03-14T09:50:25.479 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:25.478+0000 7ff4e0edc700 1 --2- 172.21.15.152:0/4183013357 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff4c40a0a90 0x7ff4c40a2f50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:25.479 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:25.478+0000 7ff4e0edc700 1 --2- 172.21.15.152:0/4183013357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff4dc100dc0 0x7ff4dc106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:25.479 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:25.478+0000 7ff4e0edc700 1 --2- 172.21.15.152:0/4183013357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff4dc102960 0x7ff4dc103680 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:25.479 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:25.478+0000 7ff4e0edc700 1 --2- 172.21.15.152:0/4183013357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4dc101fc0 0x7ff4dc107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:25.479 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:25.478+0000 7ff4e0edc700 1 -- 172.21.15.152:0/4183013357 >> 172.21.15.152:0/4183013357 conn(0x7ff4dc0fc360 msgr2=0x7ff4dc107790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:50:25.479 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:25.478+0000 7ff4e0edc700 1 -- 172.21.15.152:0/4183013357 shutdown_connections 2024-03-14T09:50:25.479 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:50:25.479+0000 7ff4e0edc700 1 -- 172.21.15.152:0/4183013357 wait complete. 2024-03-14T09:50:25.881 DEBUG:teuthology.orchestra.run.smithi152:osd.8> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.8.service 2024-03-14T09:50:25.883 INFO:tasks.cephadm:Deploying osd.9 on smithi078 with /dev/vg_nvme/lv_1... 2024-03-14T09:50:25.883 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-14T09:50:25.909 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:25.909 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:25.909 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:25.909 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:25.909 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:25.909 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:25 smithi152 ceph-mon[48279]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:25.909 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:25.909 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:25.909 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:25.915 INFO:journalctl@ceph.osd.8.smithi152.stdout:-- Logs begin at Thu 2024-03-14 09:24:45 UTC. -- 2024-03-14T09:50:26.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:26.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:26.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:26.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:26.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:26.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:25 smithi132 ceph-mon[47959]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:26.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:26.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:26.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:26.283 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:50:26.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:26.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:26.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:26.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:26.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:26.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:25 smithi078 ceph-mon[38143]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:26.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:26.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:26.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:27.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 09:50:27 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T09:50:27.162+0000 7f5c35078640 -1 osd.8 0 log_to_monitors true 2024-03-14T09:50:27.988 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-14T09:50:27.988 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-14T09:50:27.988 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2024-03-14T09:50:27.988 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2024-03-14T09:50:27.988 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0310148 s, 338 MB/s 2024-03-14T09:50:27.988 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2024-03-14T09:50:28.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:28 smithi078 ceph-mon[38143]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:28.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:28 smithi078 ceph-mon[38143]: from='osd.8 [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-14T09:50:28.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:28 smithi078 ceph-mon[38143]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-14T09:50:28.371 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi078:vg_nvme/lv_1 2024-03-14T09:50:28.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:28 smithi152 ceph-mon[48279]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:28.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:28 smithi152 ceph-mon[48279]: from='osd.8 [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-14T09:50:28.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:28 smithi152 ceph-mon[48279]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-14T09:50:28.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:28 smithi132 ceph-mon[47959]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:28.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:28 smithi132 ceph-mon[47959]: from='osd.8 [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-14T09:50:28.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:28 smithi132 ceph-mon[47959]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-14T09:50:28.726 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:50:29.215 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:29 smithi078 ceph-mon[38143]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-14T09:50:29.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:29 smithi078 ceph-mon[38143]: osdmap e53: 9 total, 8 up, 9 in 2024-03-14T09:50:29.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:29 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:29.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:29 smithi078 ceph-mon[38143]: from='osd.8 [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:50:29.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:29 smithi078 ceph-mon[38143]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:50:29.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:29 smithi078 ceph-mon[38143]: Detected new or changed devices on smithi152 2024-03-14T09:50:29.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:29 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:29.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:29 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:29.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:29 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:29.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:29 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:29.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:29 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:29.216 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:29 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi152 to 5704M 2024-03-14T09:50:29.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:29 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:29.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:29 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:29.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:29 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:29.217 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:29 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:29.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T09:50:29.115+0000 7f5c2a6dc700 -1 osd.8 0 waiting for initial osdmap 2024-03-14T09:50:29.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T09:50:29.126+0000 7f5c25cd0700 -1 osd.8 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T09:50:29.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-mon[48279]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-14T09:50:29.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-mon[48279]: osdmap e53: 9 total, 8 up, 9 in 2024-03-14T09:50:29.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:29.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-mon[48279]: from='osd.8 [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:50:29.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-mon[48279]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:50:29.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-mon[48279]: Detected new or changed devices on smithi152 2024-03-14T09:50:29.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:29.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:29.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:29.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:29.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:29.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi152 to 5704M 2024-03-14T09:50:29.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:29.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:29.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:29.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:29 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:29.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:29 smithi132 ceph-mon[47959]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-14T09:50:29.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:29 smithi132 ceph-mon[47959]: osdmap e53: 9 total, 8 up, 9 in 2024-03-14T09:50:29.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:29 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:29.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:29 smithi132 ceph-mon[47959]: from='osd.8 [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:50:29.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:29 smithi132 ceph-mon[47959]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:50:29.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:29 smithi132 ceph-mon[47959]: Detected new or changed devices on smithi152 2024-03-14T09:50:29.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:29 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:29.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:29 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:29.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:29 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:29.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:29 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:29.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:29 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:29.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:29 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi152 to 5704M 2024-03-14T09:50:29.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:29 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:29.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:29 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:29.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:29 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:29.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:29 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.898+0000 7f4f35c00700 1 -- 172.21.15.78:0/480325641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f30102980 msgr2=0x7f4f30109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:50:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.898+0000 7f4f35c00700 1 --2- 172.21.15.78:0/480325641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f30102980 0x7f4f30109230 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800b0d0 tx=0x7f4f2800b3e0 comp rx=0 tx=0).stop 2024-03-14T09:50:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.899+0000 7f4f35c00700 1 -- 172.21.15.78:0/480325641 shutdown_connections 2024-03-14T09:50:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.899+0000 7f4f35c00700 1 --2- 172.21.15.78:0/480325641 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f30100de0 0x7f4f30101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.899+0000 7f4f35c00700 1 --2- 172.21.15.78:0/480325641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f30102980 0x7f4f30109230 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.899+0000 7f4f35c00700 1 --2- 172.21.15.78:0/480325641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f30101fe0 0x7f4f30102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.899+0000 7f4f35c00700 1 -- 172.21.15.78:0/480325641 >> 172.21.15.78:0/480325641 conn(0x7f4f300fc380 msgr2=0x7f4f300fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:50:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.899+0000 7f4f35c00700 1 -- 172.21.15.78:0/480325641 shutdown_connections 2024-03-14T09:50:29.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.899+0000 7f4f35c00700 1 -- 172.21.15.78:0/480325641 wait complete. 2024-03-14T09:50:29.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.900+0000 7f4f35c00700 1 Processor -- start 2024-03-14T09:50:29.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.900+0000 7f4f35c00700 1 -- start start 2024-03-14T09:50:29.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.900+0000 7f4f35c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f30100de0 0x7f4f30199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:50:29.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.900+0000 7f4f35c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f30101fe0 0x7f4f30199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:50:29.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.900+0000 7f4f35c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f30102980 0x7f4f301a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:50:29.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.900+0000 7f4f35c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f301a0f70 con 0x7f4f30101fe0 2024-03-14T09:50:29.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.900+0000 7f4f35c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f301a10b0 con 0x7f4f30102980 2024-03-14T09:50:29.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.900+0000 7f4f35c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f301a1240 con 0x7f4f30100de0 2024-03-14T09:50:29.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.900+0000 7f4f2e000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f30101fe0 0x7f4f30199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:50:29.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.900+0000 7f4f2e000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f30101fe0 0x7f4f30199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33352/0 (socket says 172.21.15.78:33352) 2024-03-14T09:50:29.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.900+0000 7f4f2e000700 1 -- 172.21.15.78:0/317540581 learned_addr learned my addr 172.21.15.78:0/317540581 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:50:29.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.900+0000 7f4f2ea00700 1 --2- 172.21.15.78:0/317540581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f30100de0 0x7f4f30199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:50:29.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.901+0000 7f4f2f400700 1 --2- 172.21.15.78:0/317540581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f30102980 0x7f4f301a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:50:29.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.901+0000 7f4f2e000700 1 -- 172.21.15.78:0/317540581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f30100de0 msgr2=0x7f4f30199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:50:29.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.901+0000 7f4f2e000700 1 --2- 172.21.15.78:0/317540581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f30100de0 0x7f4f30199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:29.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.901+0000 7f4f2e000700 1 -- 172.21.15.78:0/317540581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f30102980 msgr2=0x7f4f301a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:50:29.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.901+0000 7f4f2e000700 1 --2- 172.21.15.78:0/317540581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f30102980 0x7f4f301a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:29.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.901+0000 7f4f2e000700 1 -- 172.21.15.78:0/317540581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c009520 con 0x7f4f30101fe0 2024-03-14T09:50:29.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.901+0000 7f4f2e000700 1 --2- 172.21.15.78:0/317540581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f30101fe0 0x7f4f30199970 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00eb60 tx=0x7f4f1c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:50:29.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.901+0000 7f4f27400700 1 -- 172.21.15.78:0/317540581 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c00cbd0 con 0x7f4f30101fe0 2024-03-14T09:50:29.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.901+0000 7f4f35c00700 1 -- 172.21.15.78:0/317540581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f28009d00 con 0x7f4f30101fe0 2024-03-14T09:50:29.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.901+0000 7f4f27400700 1 -- 172.21.15.78:0/317540581 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f1c0044e0 con 0x7f4f30101fe0 2024-03-14T09:50:29.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.901+0000 7f4f27400700 1 -- 172.21.15.78:0/317540581 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c003df0 con 0x7f4f30101fe0 2024-03-14T09:50:29.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.901+0000 7f4f35c00700 1 -- 172.21.15.78:0/317540581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f301a1890 con 0x7f4f30101fe0 2024-03-14T09:50:29.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.902+0000 7f4f35c00700 1 -- 172.21.15.78:0/317540581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f3004ea80 con 0x7f4f30101fe0 2024-03-14T09:50:29.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.904+0000 7f4f27400700 1 -- 172.21.15.78:0/317540581 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f4f1c004000 con 0x7f4f30101fe0 2024-03-14T09:50:29.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.905+0000 7f4f27400700 1 --2- 172.21.15.78:0/317540581 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f140a0a90 0x7f4f140a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:50:29.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.906+0000 7f4f27400700 1 -- 172.21.15.78:0/317540581 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5549+0+0 (secure 0 0 0) 0x7f4f1c014070 con 0x7f4f30101fe0 2024-03-14T09:50:29.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.906+0000 7f4f2ea00700 1 --2- 172.21.15.78:0/317540581 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f140a0a90 0x7f4f140a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:50:29.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.906+0000 7f4f2ea00700 1 --2- 172.21.15.78:0/317540581 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f140a0a90 0x7f4f140a2f50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4f30101e40 tx=0x7f4f20009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:50:29.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:29.910+0000 7f4f27400700 1 -- 172.21.15.78:0/317540581 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4f1c095d80 con 0x7f4f30101fe0 2024-03-14T09:50:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:30.126+0000 7f4f35c00700 1 -- 172.21.15.78:0/317540581 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f30074700 con 0x7f4f140a0a90 2024-03-14T09:50:30.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:30 smithi152 ceph-mon[48279]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:30.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:30 smithi152 ceph-mon[48279]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-14T09:50:30.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:30 smithi152 ceph-mon[48279]: osdmap e54: 9 total, 8 up, 9 in 2024-03-14T09:50:30.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:30 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:30.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:30 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:30.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:30 smithi132 ceph-mon[47959]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:30.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:30 smithi132 ceph-mon[47959]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-14T09:50:30.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:30 smithi132 ceph-mon[47959]: osdmap e54: 9 total, 8 up, 9 in 2024-03-14T09:50:30.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:30 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:30.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:30 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:30.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:30 smithi078 ceph-mon[38143]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:30.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:30 smithi078 ceph-mon[38143]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-14T09:50:30.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:30 smithi078 ceph-mon[38143]: osdmap e54: 9 total, 8 up, 9 in 2024-03-14T09:50:30.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:30 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:30.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:30 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:31.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:31 smithi078 ceph-mon[38143]: purged_snaps scrub starts 2024-03-14T09:50:31.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:31 smithi078 ceph-mon[38143]: purged_snaps scrub ok 2024-03-14T09:50:31.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:31 smithi078 ceph-mon[38143]: osd.8 [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397] boot 2024-03-14T09:50:31.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:31 smithi078 ceph-mon[38143]: osdmap e55: 9 total, 9 up, 9 in 2024-03-14T09:50:31.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:31.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:31 smithi078 ceph-mon[38143]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:50:31.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:50:31.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:50:31.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:31 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:31.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:31 smithi078 ceph-mon[38143]: osdmap e56: 9 total, 9 up, 9 in 2024-03-14T09:50:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:31 smithi152 ceph-mon[48279]: purged_snaps scrub starts 2024-03-14T09:50:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:31 smithi152 ceph-mon[48279]: purged_snaps scrub ok 2024-03-14T09:50:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:31 smithi152 ceph-mon[48279]: osd.8 [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397] boot 2024-03-14T09:50:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:31 smithi152 ceph-mon[48279]: osdmap e55: 9 total, 9 up, 9 in 2024-03-14T09:50:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:31 smithi152 ceph-mon[48279]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:50:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:50:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:50:31.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:31 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:31.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:31 smithi152 ceph-mon[48279]: osdmap e56: 9 total, 9 up, 9 in 2024-03-14T09:50:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:31 smithi132 ceph-mon[47959]: purged_snaps scrub starts 2024-03-14T09:50:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:31 smithi132 ceph-mon[47959]: purged_snaps scrub ok 2024-03-14T09:50:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:31 smithi132 ceph-mon[47959]: osd.8 [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397] boot 2024-03-14T09:50:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:31 smithi132 ceph-mon[47959]: osdmap e55: 9 total, 9 up, 9 in 2024-03-14T09:50:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:50:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:31 smithi132 ceph-mon[47959]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:50:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:50:31.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:50:31.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:31 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:31.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:31 smithi132 ceph-mon[47959]: osdmap e56: 9 total, 9 up, 9 in 2024-03-14T09:50:32.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:32 smithi152 ceph-mon[48279]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:32.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:32 smithi152 ceph-mon[48279]: osdmap e57: 9 total, 9 up, 9 in 2024-03-14T09:50:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:32 smithi132 ceph-mon[47959]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:32 smithi132 ceph-mon[47959]: osdmap e57: 9 total, 9 up, 9 in 2024-03-14T09:50:32.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:32 smithi078 ceph-mon[38143]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-03-14T09:50:32.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:32 smithi078 ceph-mon[38143]: osdmap e57: 9 total, 9 up, 9 in 2024-03-14T09:50:33.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:33 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1765586272' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15f5ddad-e37a-4291-8d62-05fa69894552"}]: dispatch 2024-03-14T09:50:33.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:33 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1765586272' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15f5ddad-e37a-4291-8d62-05fa69894552"}]': finished 2024-03-14T09:50:33.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:33 smithi152 ceph-mon[48279]: osdmap e58: 10 total, 9 up, 10 in 2024-03-14T09:50:33.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:33 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:33 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1765586272' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15f5ddad-e37a-4291-8d62-05fa69894552"}]: dispatch 2024-03-14T09:50:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:33 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1765586272' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15f5ddad-e37a-4291-8d62-05fa69894552"}]': finished 2024-03-14T09:50:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:33 smithi132 ceph-mon[47959]: osdmap e58: 10 total, 9 up, 10 in 2024-03-14T09:50:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:33 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:33.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:33 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1765586272' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15f5ddad-e37a-4291-8d62-05fa69894552"}]: dispatch 2024-03-14T09:50:33.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:33 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1765586272' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15f5ddad-e37a-4291-8d62-05fa69894552"}]': finished 2024-03-14T09:50:33.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:33 smithi078 ceph-mon[38143]: osdmap e58: 10 total, 9 up, 10 in 2024-03-14T09:50:33.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:33 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:34.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:34 smithi152 ceph-mon[48279]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2024-03-14T09:50:34.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:34 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2059934036' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:50:34.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:34 smithi152 ceph-mon[48279]: osdmap e59: 10 total, 9 up, 10 in 2024-03-14T09:50:34.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:34 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:34 smithi132 ceph-mon[47959]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2024-03-14T09:50:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:34 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2059934036' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:50:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:34 smithi132 ceph-mon[47959]: osdmap e59: 10 total, 9 up, 10 in 2024-03-14T09:50:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:34 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:34.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:34 smithi078 ceph-mon[38143]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2024-03-14T09:50:34.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:34 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2059934036' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:50:34.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:34 smithi078 ceph-mon[38143]: osdmap e59: 10 total, 9 up, 10 in 2024-03-14T09:50:34.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:34 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:36.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:36 smithi078 ceph-mon[38143]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-03-14T09:50:36.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:36 smithi152 ceph-mon[48279]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-03-14T09:50:36.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:36 smithi132 ceph-mon[47959]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-03-14T09:50:38.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:38 smithi152 ceph-mon[48279]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-14T09:50:38.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:38 smithi132 ceph-mon[47959]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-14T09:50:38.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:38 smithi078 ceph-mon[38143]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-14T09:50:39.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:39 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-14T09:50:39.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:39 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:39.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:39 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-14T09:50:39.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:39 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:39.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:39 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-14T09:50:39.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:39 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:40.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:40 smithi078 ceph-mon[38143]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-14T09:50:40.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:40 smithi078 ceph-mon[38143]: Deploying daemon osd.9 on smithi078 2024-03-14T09:50:40.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:40 smithi152 ceph-mon[48279]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-14T09:50:40.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:40 smithi152 ceph-mon[48279]: Deploying daemon osd.9 on smithi078 2024-03-14T09:50:40.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:40 smithi132 ceph-mon[47959]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-14T09:50:40.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:40 smithi132 ceph-mon[47959]: Deploying daemon osd.9 on smithi078 2024-03-14T09:50:42.421 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:42 smithi078 ceph-mon[38143]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-14T09:50:42.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:42 smithi152 ceph-mon[48279]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-14T09:50:42.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:42 smithi132 ceph-mon[47959]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-14T09:50:43.986 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:43 smithi078 ceph-mon[38143]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-14T09:50:43.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:43 smithi152 ceph-mon[48279]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-14T09:50:44.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:43 smithi132 ceph-mon[47959]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-14T09:50:45.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:45 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:45.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:45 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:45.221 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:45 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:45.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:45 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:45.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:45 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:45.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:45 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:45.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:45 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:45.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:45 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:45.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:45 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:46.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:46 smithi152 ceph-mon[48279]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-14T09:50:46.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:46 smithi132 ceph-mon[47959]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-14T09:50:46.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:46 smithi078 ceph-mon[38143]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-14T09:50:46.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:46.979+0000 7f4f27400700 1 -- 172.21.15.78:0/317540581 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4f30074700 con 0x7f4f140a0a90 2024-03-14T09:50:46.980 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 9 on host 'smithi078' 2024-03-14T09:50:46.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:46.983+0000 7f4f35c00700 1 -- 172.21.15.78:0/317540581 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f140a0a90 msgr2=0x7f4f140a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:50:46.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:46.983+0000 7f4f35c00700 1 --2- 172.21.15.78:0/317540581 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f140a0a90 0x7f4f140a2f50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4f30101e40 tx=0x7f4f20009260 comp rx=0 tx=0).stop 2024-03-14T09:50:46.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:46.983+0000 7f4f35c00700 1 -- 172.21.15.78:0/317540581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f30101fe0 msgr2=0x7f4f30199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:50:46.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:46.983+0000 7f4f35c00700 1 --2- 172.21.15.78:0/317540581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f30101fe0 0x7f4f30199970 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00eb60 tx=0x7f4f1c00ee70 comp rx=0 tx=0).stop 2024-03-14T09:50:46.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:46.983+0000 7f4f35c00700 1 -- 172.21.15.78:0/317540581 shutdown_connections 2024-03-14T09:50:46.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:46.983+0000 7f4f35c00700 1 --2- 172.21.15.78:0/317540581 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f140a0a90 0x7f4f140a2f50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:46.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:46.983+0000 7f4f35c00700 1 --2- 172.21.15.78:0/317540581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f30102980 0x7f4f301a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:46.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:46.983+0000 7f4f35c00700 1 --2- 172.21.15.78:0/317540581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f30100de0 0x7f4f30199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:46.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:46.983+0000 7f4f35c00700 1 --2- 172.21.15.78:0/317540581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f30101fe0 0x7f4f30199970 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:46.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:46.983+0000 7f4f35c00700 1 -- 172.21.15.78:0/317540581 >> 172.21.15.78:0/317540581 conn(0x7f4f300fc380 msgr2=0x7f4f30072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:50:46.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:46.984+0000 7f4f35c00700 1 -- 172.21.15.78:0/317540581 shutdown_connections 2024-03-14T09:50:46.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:50:46.984+0000 7f4f35c00700 1 -- 172.21.15.78:0/317540581 wait complete. 2024-03-14T09:50:47.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:47.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:47.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:47.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:47.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:47.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:47.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:47.186 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:47.361 DEBUG:teuthology.orchestra.run.smithi078:osd.9> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.9.service 2024-03-14T09:50:47.366 INFO:tasks.cephadm:Deploying osd.10 on smithi132 with /dev/vg_nvme/lv_1... 2024-03-14T09:50:47.366 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-14T09:50:47.399 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:47.399 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:47.399 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:47.399 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:47.399 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:47.399 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:47.400 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:47.400 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:47.442 INFO:journalctl@ceph.osd.9.smithi078.stdout:-- Logs begin at Thu 2024-03-14 09:30:54 UTC. -- 2024-03-14T09:50:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:50:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:47.596 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.b/config 2024-03-14T09:50:48.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:48 smithi132 ceph-mon[47959]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-14T09:50:48.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:48 smithi078 ceph-mon[38143]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-14T09:50:48.304 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 09:50:48 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T09:50:48.074+0000 7fa7b7c14640 -1 osd.9 0 log_to_monitors true 2024-03-14T09:50:48.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:48 smithi152 ceph-mon[48279]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-14T09:50:48.931 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-14T09:50:48.931 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-14T09:50:48.931 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-03-14T09:50:48.931 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-03-14T09:50:48.931 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111809 s, 938 MB/s 2024-03-14T09:50:48.931 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-03-14T09:50:49.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:49 smithi132 ceph-mon[47959]: from='osd.9 [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-14T09:50:49.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:49 smithi132 ceph-mon[47959]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-14T09:50:49.336 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi132:vg_nvme/lv_1 2024-03-14T09:50:49.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:49 smithi152 ceph-mon[48279]: from='osd.9 [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-14T09:50:49.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:49 smithi152 ceph-mon[48279]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-14T09:50:49.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:49 smithi078 ceph-mon[38143]: from='osd.9 [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-14T09:50:49.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:49 smithi078 ceph-mon[38143]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-14T09:50:49.596 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.b/config 2024-03-14T09:50:50.303 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 09:50:50 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T09:50:50.097+0000 7fa7ac200700 -1 osd.9 0 waiting for initial osdmap 2024-03-14T09:50:50.303 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 09:50:50 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T09:50:50.127+0000 7fa7a2000700 -1 osd.9 61 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T09:50:50.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:50 smithi078 ceph-mon[38143]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-03-14T09:50:50.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:50 smithi078 ceph-mon[38143]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-14T09:50:50.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:50 smithi078 ceph-mon[38143]: osdmap e60: 10 total, 9 up, 10 in 2024-03-14T09:50:50.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:50 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:50.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:50 smithi078 ceph-mon[38143]: from='osd.9 [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:50:50.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:50 smithi078 ceph-mon[38143]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:50:50.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:50 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:50.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:50 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:50.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:50 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:50.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:50 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:50.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:50 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:50.382 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:50 smithi132 ceph-mon[47959]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-03-14T09:50:50.383 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:50 smithi132 ceph-mon[47959]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-14T09:50:50.383 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:50 smithi132 ceph-mon[47959]: osdmap e60: 10 total, 9 up, 10 in 2024-03-14T09:50:50.383 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:50 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:50.383 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:50 smithi132 ceph-mon[47959]: from='osd.9 [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:50:50.383 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:50 smithi132 ceph-mon[47959]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:50:50.383 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:50 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:50.383 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:50 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:50.383 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:50 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:50.383 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:50 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:50.383 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:50 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:50.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:50 smithi152 ceph-mon[48279]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-03-14T09:50:50.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:50 smithi152 ceph-mon[48279]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-14T09:50:50.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:50 smithi152 ceph-mon[48279]: osdmap e60: 10 total, 9 up, 10 in 2024-03-14T09:50:50.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:50 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:50.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:50 smithi152 ceph-mon[48279]: from='osd.9 [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:50:50.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:50 smithi152 ceph-mon[48279]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2024-03-14T09:50:50.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:50 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:50.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:50 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:50.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:50 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:50.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:50 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:50.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:50 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:50.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.560+0000 7f07f193e700 1 -- 172.21.15.132:0/2089482099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ec0ffcd0 msgr2=0x7f07ec1000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:50:50.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.560+0000 7f07f193e700 1 --2- 172.21.15.132:0/2089482099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ec0ffcd0 0x7f07ec1000f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f07e00099c0 tx=0x7f07e0009cd0 comp rx=0 tx=0).stop 2024-03-14T09:50:50.562 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.560+0000 7f07f193e700 1 -- 172.21.15.132:0/2089482099 shutdown_connections 2024-03-14T09:50:50.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.560+0000 7f07f193e700 1 --2- 172.21.15.132:0/2089482099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ec0ffcd0 0x7f07ec1000f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:50.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.560+0000 7f07f193e700 1 --2- 172.21.15.132:0/2089482099 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07ec101870 0x7f07ec108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:50.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.560+0000 7f07f193e700 1 --2- 172.21.15.132:0/2089482099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07ec100ed0 0x7f07ec101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:50.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.560+0000 7f07f193e700 1 -- 172.21.15.132:0/2089482099 >> 172.21.15.132:0/2089482099 conn(0x7f07ec0fb270 msgr2=0x7f07ec0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:50:50.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.560+0000 7f07f193e700 1 -- 172.21.15.132:0/2089482099 shutdown_connections 2024-03-14T09:50:50.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.560+0000 7f07f193e700 1 -- 172.21.15.132:0/2089482099 wait complete. 2024-03-14T09:50:50.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.560+0000 7f07f193e700 1 Processor -- start 2024-03-14T09:50:50.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07f193e700 1 -- start start 2024-03-14T09:50:50.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07f193e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ec0ffcd0 0x7f07ec195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:50:50.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07f193e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07ec100ed0 0x7f07ec196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:50:50.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07f193e700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07ec101870 0x7f07ec19d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:50:50.563 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07f193e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec19db10 con 0x7f07ec100ed0 2024-03-14T09:50:50.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07f193e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec19dc50 con 0x7f07ec0ffcd0 2024-03-14T09:50:50.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07f193e700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec19dde0 con 0x7f07ec101870 2024-03-14T09:50:50.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07eaffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ec0ffcd0 0x7f07ec195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:50:50.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07eaffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ec0ffcd0 0x7f07ec195fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:52774/0 (socket says 172.21.15.132:52774) 2024-03-14T09:50:50.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07eaffd700 1 -- 172.21.15.132:0/219080665 learned_addr learned my addr 172.21.15.132:0/219080665 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-03-14T09:50:50.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07eb7fe700 1 --2- 172.21.15.132:0/219080665 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07ec101870 0x7f07ec19d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:50:50.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07eaffd700 1 -- 172.21.15.132:0/219080665 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07ec101870 msgr2=0x7f07ec19d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:50:50.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07eaffd700 1 --2- 172.21.15.132:0/219080665 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07ec101870 0x7f07ec19d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:50.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07eaffd700 1 -- 172.21.15.132:0/219080665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07ec100ed0 msgr2=0x7f07ec196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:50:50.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07eaffd700 1 --2- 172.21.15.132:0/219080665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07ec100ed0 0x7f07ec196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:50:50.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07eaffd700 1 -- 172.21.15.132:0/219080665 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e00095f0 con 0x7f07ec0ffcd0 2024-03-14T09:50:50.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.561+0000 7f07eaffd700 1 --2- 172.21.15.132:0/219080665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ec0ffcd0 0x7f07ec195fd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f07e0009f50 tx=0x7f07e00057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:50:50.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.562+0000 7f07f093c700 1 -- 172.21.15.132:0/219080665 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e001d070 con 0x7f07ec0ffcd0 2024-03-14T09:50:50.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.562+0000 7f07f093c700 1 -- 172.21.15.132:0/219080665 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f07e0004200 con 0x7f07ec0ffcd0 2024-03-14T09:50:50.564 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.562+0000 7f07f093c700 1 -- 172.21.15.132:0/219080665 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e0022e40 con 0x7f07ec0ffcd0 2024-03-14T09:50:50.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.562+0000 7f07f193e700 1 -- 172.21.15.132:0/219080665 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07ec19e060 con 0x7f07ec0ffcd0 2024-03-14T09:50:50.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.562+0000 7f07f193e700 1 -- 172.21.15.132:0/219080665 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ec19e4b0 con 0x7f07ec0ffcd0 2024-03-14T09:50:50.565 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.563+0000 7f07f193e700 1 -- 172.21.15.132:0/219080665 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07ec107100 con 0x7f07ec0ffcd0 2024-03-14T09:50:50.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.564+0000 7f07f093c700 1 -- 172.21.15.132:0/219080665 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f07e0022570 con 0x7f07ec0ffcd0 2024-03-14T09:50:50.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.564+0000 7f07f093c700 1 --2- 172.21.15.132:0/219080665 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f07d40a0a90 0x7f07d40a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:50:50.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.564+0000 7f07f093c700 1 -- 172.21.15.132:0/219080665 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5981+0+0 (secure 0 0 0) 0x7f07e00cca80 con 0x7f07ec0ffcd0 2024-03-14T09:50:50.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.565+0000 7f07ea7fc700 1 --2- 172.21.15.132:0/219080665 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f07d40a0a90 0x7f07d40a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:50:50.566 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.565+0000 7f07ea7fc700 1 --2- 172.21.15.132:0/219080665 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f07d40a0a90 0x7f07d40a2f50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f07dc00a990 tx=0x7f07dc005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:50:50.569 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.569+0000 7f07f093c700 1 -- 172.21.15.132:0/219080665 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f07e0096f20 con 0x7f07ec0ffcd0 2024-03-14T09:50:50.771 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:50:50.770+0000 7f07f193e700 1 -- 172.21.15.132:0/219080665 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f07ec0611f0 con 0x7f07d40a0a90 2024-03-14T09:50:51.490 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:51 smithi132 ceph-mon[47959]: purged_snaps scrub starts 2024-03-14T09:50:51.490 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:51 smithi132 ceph-mon[47959]: purged_snaps scrub ok 2024-03-14T09:50:51.491 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:51 smithi132 ceph-mon[47959]: Detected new or changed devices on smithi078 2024-03-14T09:50:51.491 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:51 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:51.491 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:51 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi078 to 4317M 2024-03-14T09:50:51.491 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:51 smithi132 ceph-mon[47959]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-14T09:50:51.491 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:51 smithi132 ceph-mon[47959]: osdmap e61: 10 total, 9 up, 10 in 2024-03-14T09:50:51.491 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:51 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:51.491 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:51 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:51.491 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:51 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:51.491 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:51 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:51.491 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:51 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:51.491 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:51 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:51.491 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:51 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:50:51.491 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:51 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:50:51.491 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:51 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:51 smithi152 ceph-mon[48279]: purged_snaps scrub starts 2024-03-14T09:50:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:51 smithi152 ceph-mon[48279]: purged_snaps scrub ok 2024-03-14T09:50:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:51 smithi152 ceph-mon[48279]: Detected new or changed devices on smithi078 2024-03-14T09:50:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:51 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:51 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi078 to 4317M 2024-03-14T09:50:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:51 smithi152 ceph-mon[48279]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-14T09:50:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:51 smithi152 ceph-mon[48279]: osdmap e61: 10 total, 9 up, 10 in 2024-03-14T09:50:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:51 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:51 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:51 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:51 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:51 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:51 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:51 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:50:51.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:51 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:50:51.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:51 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:51.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:51 smithi078 ceph-mon[38143]: purged_snaps scrub starts 2024-03-14T09:50:51.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:51 smithi078 ceph-mon[38143]: purged_snaps scrub ok 2024-03-14T09:50:51.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:51 smithi078 ceph-mon[38143]: Detected new or changed devices on smithi078 2024-03-14T09:50:51.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:51 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:50:51.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:51 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi078 to 4317M 2024-03-14T09:50:51.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:51 smithi078 ceph-mon[38143]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2024-03-14T09:50:51.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:51 smithi078 ceph-mon[38143]: osdmap e61: 10 total, 9 up, 10 in 2024-03-14T09:50:51.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:51 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:51.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:51 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:51.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:51 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:51.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:51 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:51.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:51 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:50:51.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:51 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:50:51.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:51 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:50:51.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:51 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:50:51.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:51 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:52.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:52 smithi152 ceph-mon[48279]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:50:52.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:52 smithi152 ceph-mon[48279]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-03-14T09:50:52.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:52 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:52.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:52 smithi152 ceph-mon[48279]: osd.9 [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096] boot 2024-03-14T09:50:52.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:52 smithi152 ceph-mon[48279]: osdmap e62: 10 total, 10 up, 10 in 2024-03-14T09:50:52.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:52 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:52.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:52 smithi132 ceph-mon[47959]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:50:52.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:52 smithi132 ceph-mon[47959]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-03-14T09:50:52.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:52 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:52.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:52 smithi132 ceph-mon[47959]: osd.9 [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096] boot 2024-03-14T09:50:52.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:52 smithi132 ceph-mon[47959]: osdmap e62: 10 total, 10 up, 10 in 2024-03-14T09:50:52.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:52 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:52.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:52 smithi078 ceph-mon[38143]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:50:52.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:52 smithi078 ceph-mon[38143]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-03-14T09:50:52.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:52 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:52.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:52 smithi078 ceph-mon[38143]: osd.9 [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096] boot 2024-03-14T09:50:52.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:52 smithi078 ceph-mon[38143]: osdmap e62: 10 total, 10 up, 10 in 2024-03-14T09:50:52.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:52 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:50:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:53 smithi152 ceph-mon[48279]: osdmap e63: 10 total, 10 up, 10 in 2024-03-14T09:50:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:53 smithi152 ceph-mon[48279]: from='client.? 172.21.15.132:0/1900022063' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a325f26-e31f-4f0e-b09f-8bda5b27fbcd"}]: dispatch 2024-03-14T09:50:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:53 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a325f26-e31f-4f0e-b09f-8bda5b27fbcd"}]: dispatch 2024-03-14T09:50:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:53 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a325f26-e31f-4f0e-b09f-8bda5b27fbcd"}]': finished 2024-03-14T09:50:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:53 smithi152 ceph-mon[48279]: osdmap e64: 11 total, 10 up, 11 in 2024-03-14T09:50:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:50:53.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:53 smithi132 ceph-mon[47959]: osdmap e63: 10 total, 10 up, 10 in 2024-03-14T09:50:53.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:53 smithi132 ceph-mon[47959]: from='client.? 172.21.15.132:0/1900022063' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a325f26-e31f-4f0e-b09f-8bda5b27fbcd"}]: dispatch 2024-03-14T09:50:53.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:53 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a325f26-e31f-4f0e-b09f-8bda5b27fbcd"}]: dispatch 2024-03-14T09:50:53.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:53 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a325f26-e31f-4f0e-b09f-8bda5b27fbcd"}]': finished 2024-03-14T09:50:53.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:53 smithi132 ceph-mon[47959]: osdmap e64: 11 total, 10 up, 11 in 2024-03-14T09:50:53.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:50:53.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:53 smithi078 ceph-mon[38143]: osdmap e63: 10 total, 10 up, 10 in 2024-03-14T09:50:53.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:53 smithi078 ceph-mon[38143]: from='client.? 172.21.15.132:0/1900022063' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a325f26-e31f-4f0e-b09f-8bda5b27fbcd"}]: dispatch 2024-03-14T09:50:53.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:53 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a325f26-e31f-4f0e-b09f-8bda5b27fbcd"}]: dispatch 2024-03-14T09:50:53.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:53 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a325f26-e31f-4f0e-b09f-8bda5b27fbcd"}]': finished 2024-03-14T09:50:53.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:53 smithi078 ceph-mon[38143]: osdmap e64: 11 total, 10 up, 11 in 2024-03-14T09:50:53.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:50:54.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:54 smithi152 ceph-mon[48279]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:50:54.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:54 smithi152 ceph-mon[48279]: from='client.? 172.21.15.132:0/2898009835' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:50:54.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:54 smithi132 ceph-mon[47959]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:50:54.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:54 smithi132 ceph-mon[47959]: from='client.? 172.21.15.132:0/2898009835' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:50:54.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:54 smithi078 ceph-mon[38143]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:50:54.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:54 smithi078 ceph-mon[38143]: from='client.? 172.21.15.132:0/2898009835' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:50:56.473 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:56 smithi132 ceph-mon[47959]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:50:56.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:56 smithi152 ceph-mon[48279]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:50:56.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:56 smithi078 ceph-mon[38143]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:50:58.351 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:58 smithi132 ceph-mon[47959]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:50:58.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:58 smithi152 ceph-mon[48279]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:50:58.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:58 smithi078 ceph-mon[38143]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:50:59.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:59 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-14T09:50:59.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:59 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:59.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:50:59 smithi152 ceph-mon[48279]: Deploying daemon osd.10 on smithi132 2024-03-14T09:50:59.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:59 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-14T09:50:59.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:59 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:59.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:50:59 smithi132 ceph-mon[47959]: Deploying daemon osd.10 on smithi132 2024-03-14T09:50:59.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:59 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-14T09:50:59.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:59 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:50:59.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:50:59 smithi078 ceph-mon[38143]: Deploying daemon osd.10 on smithi132 2024-03-14T09:51:00.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:00 smithi132 ceph-mon[47959]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:51:00.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:00 smithi152 ceph-mon[48279]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:51:00.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:00 smithi078 ceph-mon[38143]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:51:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:02 smithi152 ceph-mon[48279]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:51:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:02 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:02 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:02.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:02 smithi132 ceph-mon[47959]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:51:02.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:02 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:02.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:02 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:02.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:02 smithi078 ceph-mon[38143]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:51:02.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:02.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:03.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:03 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:51:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:03 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:51:03.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:03 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:51:04.627 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:51:04.626+0000 7f07f093c700 1 -- 172.21.15.132:0/219080665 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f07ec0611f0 con 0x7f07d40a0a90 2024-03-14T09:51:04.627 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 10 on host 'smithi132' 2024-03-14T09:51:04.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:51:04.629+0000 7f07f193e700 1 -- 172.21.15.132:0/219080665 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f07d40a0a90 msgr2=0x7f07d40a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:04.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:51:04.629+0000 7f07f193e700 1 --2- 172.21.15.132:0/219080665 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f07d40a0a90 0x7f07d40a2f50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f07dc00a990 tx=0x7f07dc005c80 comp rx=0 tx=0).stop 2024-03-14T09:51:04.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:51:04.629+0000 7f07f193e700 1 -- 172.21.15.132:0/219080665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ec0ffcd0 msgr2=0x7f07ec195fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:04.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:51:04.629+0000 7f07f193e700 1 --2- 172.21.15.132:0/219080665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ec0ffcd0 0x7f07ec195fd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f07e0009f50 tx=0x7f07e00057f0 comp rx=0 tx=0).stop 2024-03-14T09:51:04.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:51:04.629+0000 7f07f193e700 1 -- 172.21.15.132:0/219080665 shutdown_connections 2024-03-14T09:51:04.629 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:51:04.629+0000 7f07f193e700 1 --2- 172.21.15.132:0/219080665 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f07d40a0a90 0x7f07d40a2f50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:04.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:51:04.629+0000 7f07f193e700 1 --2- 172.21.15.132:0/219080665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07ec0ffcd0 0x7f07ec195fd0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:04.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:51:04.629+0000 7f07f193e700 1 --2- 172.21.15.132:0/219080665 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f07ec101870 0x7f07ec19d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:04.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:51:04.629+0000 7f07f193e700 1 --2- 172.21.15.132:0/219080665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07ec100ed0 0x7f07ec196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:04.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:51:04.629+0000 7f07f193e700 1 -- 172.21.15.132:0/219080665 >> 172.21.15.132:0/219080665 conn(0x7f07ec0fb270 msgr2=0x7f07ec104240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:04.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:51:04.629+0000 7f07f193e700 1 -- 172.21.15.132:0/219080665 shutdown_connections 2024-03-14T09:51:04.630 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-14T09:51:04.629+0000 7f07f193e700 1 -- 172.21.15.132:0/219080665 wait complete. 2024-03-14T09:51:04.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:04 smithi132 ceph-mon[47959]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:51:04.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:04 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:04.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:04 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:04.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:04 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:04.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:04 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:51:04.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:04 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:04.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:04 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:51:04.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:04 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:04.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:04 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:04.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:04 smithi152 ceph-mon[48279]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:51:04.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:04 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:04.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:04 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:04.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:04 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:04.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:04 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:51:04.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:04 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:04.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:04 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:51:04.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:04 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:05.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:04 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:05.007 DEBUG:teuthology.orchestra.run.smithi132:osd.10> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.10.service 2024-03-14T09:51:05.008 INFO:tasks.cephadm:Deploying osd.11 on smithi152 with /dev/vg_nvme/lv_1... 2024-03-14T09:51:05.009 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-14T09:51:05.042 INFO:journalctl@ceph.osd.10.smithi132.stdout:-- Logs begin at Thu 2024-03-14 09:24:09 UTC. -- 2024-03-14T09:51:05.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:04 smithi078 ceph-mon[38143]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:51:05.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:04 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:05.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:04 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:05.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:04 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:05.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:04 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:51:05.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:04 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:05.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:04 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:51:05.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:04 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:05.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:04 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:05.241 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.c/config 2024-03-14T09:51:06.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 09:51:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T09:51:06.085+0000 7f5cd38d7640 -1 osd.10 0 log_to_monitors true 2024-03-14T09:51:06.681 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-14T09:51:06.681 INFO:teuthology.orchestra.run.smithi152.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-14T09:51:06.681 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10+0 records in 2024-03-14T09:51:06.681 INFO:teuthology.orchestra.run.smithi152.stderr:10+0 records out 2024-03-14T09:51:06.681 INFO:teuthology.orchestra.run.smithi152.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112429 s, 933 MB/s 2024-03-14T09:51:06.681 INFO:teuthology.orchestra.run.smithi152.stderr:--> Zapping successful for: 2024-03-14T09:51:06.929 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:06 smithi132 ceph-mon[47959]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:51:06.929 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:06 smithi132 ceph-mon[47959]: from='osd.10 [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-14T09:51:06.929 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:06 smithi132 ceph-mon[47959]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-14T09:51:06.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:06 smithi152 ceph-mon[48279]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:51:06.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:06 smithi152 ceph-mon[48279]: from='osd.10 [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-14T09:51:06.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:06 smithi152 ceph-mon[48279]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-14T09:51:07.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:06 smithi078 ceph-mon[38143]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:51:07.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:06 smithi078 ceph-mon[38143]: from='osd.10 [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-14T09:51:07.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:06 smithi078 ceph-mon[38143]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-14T09:51:07.072 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi152:vg_nvme/lv_1 2024-03-14T09:51:07.295 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.c/config 2024-03-14T09:51:07.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:07 smithi152 ceph-mon[48279]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-14T09:51:07.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:07 smithi152 ceph-mon[48279]: osdmap e65: 11 total, 10 up, 11 in 2024-03-14T09:51:07.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:51:07.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:07 smithi152 ceph-mon[48279]: from='osd.10 [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:51:07.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:07 smithi152 ceph-mon[48279]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:51:07.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:07.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:07.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:07.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:07.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:07.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:07.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:07.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:07.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:51:07.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:07 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:08.020 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T09:51:07.605+0000 7f5cc8f3b700 -1 osd.10 0 waiting for initial osdmap 2024-03-14T09:51:08.020 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T09:51:07.618+0000 7f5cc2d2c700 -1 osd.10 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T09:51:08.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-mon[47959]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-14T09:51:08.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-mon[47959]: osdmap e65: 11 total, 10 up, 11 in 2024-03-14T09:51:08.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:51:08.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-mon[47959]: from='osd.10 [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:51:08.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-mon[47959]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:51:08.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:08.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:08.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:08.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:08.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:08.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:08.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:08.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:08.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:51:08.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:07 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:08.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:07 smithi078 ceph-mon[38143]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-14T09:51:08.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:07 smithi078 ceph-mon[38143]: osdmap e65: 11 total, 10 up, 11 in 2024-03-14T09:51:08.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:51:08.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:07 smithi078 ceph-mon[38143]: from='osd.10 [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:51:08.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:07 smithi078 ceph-mon[38143]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-03-14T09:51:08.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:08.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:08.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:08.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:08.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:08.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:08.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:08.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:08.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:51:08.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:07 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:08.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.428+0000 7f140535b700 1 -- 172.21.15.152:0/115214574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1400101870 msgr2=0x7f1400108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:08.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.428+0000 7f140535b700 1 --2- 172.21.15.152:0/115214574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1400101870 0x7f1400108120 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f13f800b0a0 tx=0x7f13f800b3b0 comp rx=0 tx=0).stop 2024-03-14T09:51:08.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.428+0000 7f140535b700 1 -- 172.21.15.152:0/115214574 shutdown_connections 2024-03-14T09:51:08.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.428+0000 7f140535b700 1 --2- 172.21.15.152:0/115214574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f14000ffcd0 0x7f14001000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:08.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.428+0000 7f140535b700 1 --2- 172.21.15.152:0/115214574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1400101870 0x7f1400108120 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:08.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.428+0000 7f140535b700 1 --2- 172.21.15.152:0/115214574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1400100ed0 0x7f1400101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:08.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.428+0000 7f140535b700 1 -- 172.21.15.152:0/115214574 >> 172.21.15.152:0/115214574 conn(0x7f14000fb270 msgr2=0x7f14000fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:08.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.428+0000 7f140535b700 1 -- 172.21.15.152:0/115214574 shutdown_connections 2024-03-14T09:51:08.432 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.428+0000 7f140535b700 1 -- 172.21.15.152:0/115214574 wait complete. 2024-03-14T09:51:08.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.429+0000 7f140535b700 1 Processor -- start 2024-03-14T09:51:08.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.429+0000 7f140535b700 1 -- start start 2024-03-14T09:51:08.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f140535b700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14000ffcd0 0x7f1400197210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:08.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f140535b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1400100ed0 0x7f1400197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:08.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f140535b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1400101870 0x7f140019e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:08.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f140535b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140019ed50 con 0x7f1400100ed0 2024-03-14T09:51:08.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f140535b700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140019ee90 con 0x7f1400101870 2024-03-14T09:51:08.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f140535b700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140019f020 con 0x7f14000ffcd0 2024-03-14T09:51:08.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f13feffd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14000ffcd0 0x7f1400197210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:08.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f13feffd700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14000ffcd0 0x7f1400197210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:48390/0 (socket says 172.21.15.152:48390) 2024-03-14T09:51:08.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f13feffd700 1 -- 172.21.15.152:0/2087198562 learned_addr learned my addr 172.21.15.152:0/2087198562 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-14T09:51:08.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f13ff7fe700 1 --2- 172.21.15.152:0/2087198562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1400101870 0x7f140019e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:08.433 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f13feffd700 1 -- 172.21.15.152:0/2087198562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1400101870 msgr2=0x7f140019e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:08.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f13feffd700 1 --2- 172.21.15.152:0/2087198562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1400101870 0x7f140019e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:08.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f13feffd700 1 -- 172.21.15.152:0/2087198562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1400100ed0 msgr2=0x7f1400197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:08.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f13feffd700 1 --2- 172.21.15.152:0/2087198562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1400100ed0 0x7f1400197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:08.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f13feffd700 1 -- 172.21.15.152:0/2087198562 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f8009d00 con 0x7f14000ffcd0 2024-03-14T09:51:08.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f13feffd700 1 --2- 172.21.15.152:0/2087198562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14000ffcd0 0x7f1400197210 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f13f400d9b0 tx=0x7f13f400dd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:08.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f13e7fff700 1 -- 172.21.15.152:0/2087198562 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f400be80 con 0x7f14000ffcd0 2024-03-14T09:51:08.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f140535b700 1 -- 172.21.15.152:0/2087198562 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f140019f300 con 0x7f14000ffcd0 2024-03-14T09:51:08.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f13e7fff700 1 -- 172.21.15.152:0/2087198562 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13f40044e0 con 0x7f14000ffcd0 2024-03-14T09:51:08.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f13e7fff700 1 -- 172.21.15.152:0/2087198562 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f4003b40 con 0x7f14000ffcd0 2024-03-14T09:51:08.434 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.430+0000 7f140535b700 1 -- 172.21.15.152:0/2087198562 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140019f800 con 0x7f14000ffcd0 2024-03-14T09:51:08.435 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.431+0000 7f140535b700 1 -- 172.21.15.152:0/2087198562 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f140004ea80 con 0x7f14000ffcd0 2024-03-14T09:51:08.435 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.432+0000 7f13e7fff700 1 -- 172.21.15.152:0/2087198562 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f13f4009940 con 0x7f14000ffcd0 2024-03-14T09:51:08.435 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.433+0000 7f13e7fff700 1 --2- 172.21.15.152:0/2087198562 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f13e80a0a90 0x7f13e80a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:08.435 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.433+0000 7f13e7fff700 1 -- 172.21.15.152:0/2087198562 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6727+0+0 (secure 0 0 0) 0x7f13f40cb760 con 0x7f14000ffcd0 2024-03-14T09:51:08.435 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.433+0000 7f13fe7fc700 1 --2- 172.21.15.152:0/2087198562 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f13e80a0a90 0x7f13e80a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:08.435 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.434+0000 7f13fe7fc700 1 --2- 172.21.15.152:0/2087198562 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f13e80a0a90 0x7f13e80a2f50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f13f000a990 tx=0x7f13f0005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:08.438 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.437+0000 7f13e7fff700 1 -- 172.21.15.152:0/2087198562 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f13f4095760 con 0x7f14000ffcd0 2024-03-14T09:51:08.638 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:08.637+0000 7f140535b700 1 -- 172.21.15.152:0/2087198562 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f14000fc050 con 0x7f13e80a0a90 2024-03-14T09:51:08.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:08 smithi152 ceph-mon[48279]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:51:08.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:08 smithi152 ceph-mon[48279]: Detected new or changed devices on smithi132 2024-03-14T09:51:08.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:08 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi132 to 4278M 2024-03-14T09:51:08.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:08 smithi152 ceph-mon[48279]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-14T09:51:08.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:08 smithi152 ceph-mon[48279]: osdmap e66: 11 total, 10 up, 11 in 2024-03-14T09:51:08.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:08 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:51:08.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:08 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:51:08.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:08 smithi152 ceph-mon[48279]: osd.10 [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089] boot 2024-03-14T09:51:09.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:08 smithi152 ceph-mon[48279]: osdmap e67: 11 total, 11 up, 11 in 2024-03-14T09:51:09.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:08 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:51:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:08 smithi132 ceph-mon[47959]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:51:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:08 smithi132 ceph-mon[47959]: Detected new or changed devices on smithi132 2024-03-14T09:51:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:08 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi132 to 4278M 2024-03-14T09:51:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:08 smithi132 ceph-mon[47959]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-14T09:51:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:08 smithi132 ceph-mon[47959]: osdmap e66: 11 total, 10 up, 11 in 2024-03-14T09:51:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:08 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:51:09.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:08 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:51:09.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:08 smithi132 ceph-mon[47959]: osd.10 [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089] boot 2024-03-14T09:51:09.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:08 smithi132 ceph-mon[47959]: osdmap e67: 11 total, 11 up, 11 in 2024-03-14T09:51:09.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:08 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:51:09.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:08 smithi078 ceph-mon[38143]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-03-14T09:51:09.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:08 smithi078 ceph-mon[38143]: Detected new or changed devices on smithi132 2024-03-14T09:51:09.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:08 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi132 to 4278M 2024-03-14T09:51:09.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:08 smithi078 ceph-mon[38143]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-03-14T09:51:09.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:08 smithi078 ceph-mon[38143]: osdmap e66: 11 total, 10 up, 11 in 2024-03-14T09:51:09.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:08 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:51:09.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:08 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:51:09.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:08 smithi078 ceph-mon[38143]: osd.10 [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089] boot 2024-03-14T09:51:09.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:08 smithi078 ceph-mon[38143]: osdmap e67: 11 total, 11 up, 11 in 2024-03-14T09:51:09.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:08 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:51:09.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:09 smithi152 ceph-mon[48279]: purged_snaps scrub starts 2024-03-14T09:51:09.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:09 smithi152 ceph-mon[48279]: purged_snaps scrub ok 2024-03-14T09:51:09.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:09 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:51:09.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:09 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:51:09.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:09 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:09.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:09 smithi152 ceph-mon[48279]: osdmap e68: 11 total, 11 up, 11 in 2024-03-14T09:51:10.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:09 smithi132 ceph-mon[47959]: purged_snaps scrub starts 2024-03-14T09:51:10.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:09 smithi132 ceph-mon[47959]: purged_snaps scrub ok 2024-03-14T09:51:10.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:09 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:51:10.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:09 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:51:10.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:09 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:10.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:09 smithi132 ceph-mon[47959]: osdmap e68: 11 total, 11 up, 11 in 2024-03-14T09:51:10.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:09 smithi078 ceph-mon[38143]: purged_snaps scrub starts 2024-03-14T09:51:10.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:09 smithi078 ceph-mon[38143]: purged_snaps scrub ok 2024-03-14T09:51:10.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:09 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T09:51:10.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:09 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T09:51:10.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:09 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:10.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:09 smithi078 ceph-mon[38143]: osdmap e68: 11 total, 11 up, 11 in 2024-03-14T09:51:10.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:10 smithi152 ceph-mon[48279]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:51:10.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:10 smithi152 ceph-mon[48279]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-03-14T09:51:10.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:10 smithi152 ceph-mon[48279]: osdmap e69: 11 total, 11 up, 11 in 2024-03-14T09:51:10.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:10 smithi152 ceph-mon[48279]: from='client.? 172.21.15.152:0/4192937465' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37c97c4c-a16b-4b02-9bd0-585aea0baafc"}]: dispatch 2024-03-14T09:51:10.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:10 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37c97c4c-a16b-4b02-9bd0-585aea0baafc"}]: dispatch 2024-03-14T09:51:10.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:10 smithi152 ceph-mon[48279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37c97c4c-a16b-4b02-9bd0-585aea0baafc"}]': finished 2024-03-14T09:51:10.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:10 smithi152 ceph-mon[48279]: osdmap e70: 12 total, 11 up, 12 in 2024-03-14T09:51:10.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:10 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:11.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:10 smithi132 ceph-mon[47959]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:51:11.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:10 smithi132 ceph-mon[47959]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-03-14T09:51:11.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:10 smithi132 ceph-mon[47959]: osdmap e69: 11 total, 11 up, 11 in 2024-03-14T09:51:11.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:10 smithi132 ceph-mon[47959]: from='client.? 172.21.15.152:0/4192937465' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37c97c4c-a16b-4b02-9bd0-585aea0baafc"}]: dispatch 2024-03-14T09:51:11.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:10 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37c97c4c-a16b-4b02-9bd0-585aea0baafc"}]: dispatch 2024-03-14T09:51:11.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:10 smithi132 ceph-mon[47959]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37c97c4c-a16b-4b02-9bd0-585aea0baafc"}]': finished 2024-03-14T09:51:11.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:10 smithi132 ceph-mon[47959]: osdmap e70: 12 total, 11 up, 12 in 2024-03-14T09:51:11.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:10 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:11.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:10 smithi078 ceph-mon[38143]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi152:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:51:11.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:10 smithi078 ceph-mon[38143]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-03-14T09:51:11.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:10 smithi078 ceph-mon[38143]: osdmap e69: 11 total, 11 up, 11 in 2024-03-14T09:51:11.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:10 smithi078 ceph-mon[38143]: from='client.? 172.21.15.152:0/4192937465' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37c97c4c-a16b-4b02-9bd0-585aea0baafc"}]: dispatch 2024-03-14T09:51:11.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:10 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37c97c4c-a16b-4b02-9bd0-585aea0baafc"}]: dispatch 2024-03-14T09:51:11.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:10 smithi078 ceph-mon[38143]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37c97c4c-a16b-4b02-9bd0-585aea0baafc"}]': finished 2024-03-14T09:51:11.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:10 smithi078 ceph-mon[38143]: osdmap e70: 12 total, 11 up, 12 in 2024-03-14T09:51:11.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:10 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:11.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:11 smithi152 ceph-mon[48279]: from='client.? 172.21.15.152:0/2676311793' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:51:11.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:11 smithi152 ceph-mon[48279]: osdmap e71: 12 total, 11 up, 12 in 2024-03-14T09:51:11.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:11 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:12.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:11 smithi132 ceph-mon[47959]: from='client.? 172.21.15.152:0/2676311793' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:51:12.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:11 smithi132 ceph-mon[47959]: osdmap e71: 12 total, 11 up, 12 in 2024-03-14T09:51:12.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:11 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:12.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:11 smithi078 ceph-mon[38143]: from='client.? 172.21.15.152:0/2676311793' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T09:51:12.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:11 smithi078 ceph-mon[38143]: osdmap e71: 12 total, 11 up, 12 in 2024-03-14T09:51:12.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:11 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:12.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:12 smithi152 ceph-mon[48279]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-03-14T09:51:13.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:12 smithi132 ceph-mon[47959]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-03-14T09:51:13.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:12 smithi078 ceph-mon[38143]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-03-14T09:51:13.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:13 smithi152 ceph-mon[48279]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-14T09:51:14.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:13 smithi132 ceph-mon[47959]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-14T09:51:14.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:13 smithi078 ceph-mon[38143]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-14T09:51:16.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:16 smithi152 ceph-mon[48279]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-14T09:51:16.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:16 smithi132 ceph-mon[47959]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-14T09:51:16.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:16 smithi078 ceph-mon[38143]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-14T09:51:17.430 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:17 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-14T09:51:17.431 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:17 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:17 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-14T09:51:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:17 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:17.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:17 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-14T09:51:17.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:17 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:18.475 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:18 smithi152 ceph-mon[48279]: Deploying daemon osd.11 on smithi152 2024-03-14T09:51:18.476 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:18 smithi152 ceph-mon[48279]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-14T09:51:18.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:18 smithi132 ceph-mon[47959]: Deploying daemon osd.11 on smithi152 2024-03-14T09:51:18.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:18 smithi132 ceph-mon[47959]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-14T09:51:18.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:18 smithi078 ceph-mon[38143]: Deploying daemon osd.11 on smithi152 2024-03-14T09:51:18.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:18 smithi078 ceph-mon[38143]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-14T09:51:20.371 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:20 smithi152 ceph-mon[48279]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-14T09:51:20.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:20 smithi132 ceph-mon[47959]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-14T09:51:20.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:20 smithi078 ceph-mon[38143]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-14T09:51:21.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:21 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:21.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:21 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:51:21.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:21 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:21.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:21.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:51:21.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:21 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:21 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:21 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:51:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:21 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:22.633 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:22 smithi152 ceph-mon[48279]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-14T09:51:22.633 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:22 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:22.633 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:22 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:22.633 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:22 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:22.634 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:22 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:51:22.634 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:22 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:22.744 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:22.743+0000 7f13e7fff700 1 -- 172.21.15.152:0/2087198562 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f14000fc050 con 0x7f13e80a0a90 2024-03-14T09:51:22.744 INFO:teuthology.orchestra.run.smithi152.stdout:Created osd(s) 11 on host 'smithi152' 2024-03-14T09:51:22.746 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:22.745+0000 7f140535b700 1 -- 172.21.15.152:0/2087198562 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f13e80a0a90 msgr2=0x7f13e80a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:22.746 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:22.745+0000 7f140535b700 1 --2- 172.21.15.152:0/2087198562 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f13e80a0a90 0x7f13e80a2f50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f13f000a990 tx=0x7f13f0005c80 comp rx=0 tx=0).stop 2024-03-14T09:51:22.746 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:22.745+0000 7f140535b700 1 -- 172.21.15.152:0/2087198562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14000ffcd0 msgr2=0x7f1400197210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:22.747 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:22.745+0000 7f140535b700 1 --2- 172.21.15.152:0/2087198562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14000ffcd0 0x7f1400197210 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f13f400d9b0 tx=0x7f13f400dd70 comp rx=0 tx=0).stop 2024-03-14T09:51:22.747 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:22.745+0000 7f140535b700 1 -- 172.21.15.152:0/2087198562 shutdown_connections 2024-03-14T09:51:22.747 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:22.745+0000 7f140535b700 1 --2- 172.21.15.152:0/2087198562 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f13e80a0a90 0x7f13e80a2f50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:22.747 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:22.745+0000 7f140535b700 1 --2- 172.21.15.152:0/2087198562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1400101870 0x7f140019e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:22.747 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:22.745+0000 7f140535b700 1 --2- 172.21.15.152:0/2087198562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f14000ffcd0 0x7f1400197210 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:22.747 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:22.745+0000 7f140535b700 1 --2- 172.21.15.152:0/2087198562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1400100ed0 0x7f1400197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:22.747 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:22.746+0000 7f140535b700 1 -- 172.21.15.152:0/2087198562 >> 172.21.15.152:0/2087198562 conn(0x7f14000fb270 msgr2=0x7f14000fb940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:22.747 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:22.746+0000 7f140535b700 1 -- 172.21.15.152:0/2087198562 shutdown_connections 2024-03-14T09:51:22.747 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:22.746+0000 7f140535b700 1 -- 172.21.15.152:0/2087198562 wait complete. 2024-03-14T09:51:22.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:22 smithi132 ceph-mon[47959]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-14T09:51:22.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:22 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:22.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:22 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:22.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:22 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:22.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:22 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:51:22.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:22 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:22.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:22 smithi078 ceph-mon[38143]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-14T09:51:22.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:22 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:22.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:22 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:22.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:22 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:22.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:22 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:51:22.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:22 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:23.162 DEBUG:teuthology.orchestra.run.smithi152:osd.11> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.11.service 2024-03-14T09:51:23.164 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-14T09:51:23.164 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-14T09:51:23.193 INFO:journalctl@ceph.osd.11.smithi152.stdout:-- Logs begin at Thu 2024-03-14 09:24:45 UTC. -- 2024-03-14T09:51:23.533 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:51:23.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:23 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:23.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:23 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:23.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:23 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:51:23.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:23 smithi152 ceph-mon[48279]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-03-14T09:51:24.016 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:23 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:24.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:23 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:24.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:23 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:51:24.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:23 smithi078 ceph-mon[38143]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-03-14T09:51:24.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:23 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:24.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:23 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:24.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:23 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:51:24.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:23 smithi132 ceph-mon[47959]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-03-14T09:51:24.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 09:51:24 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T09:51:24.390+0000 7f7cea37e640 -1 osd.11 0 log_to_monitors true 2024-03-14T09:51:24.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.681+0000 7fda14600700 1 -- 172.21.15.78:0/269998827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c101fe0 msgr2=0x7fda0c102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:24.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.681+0000 7fda14600700 1 --2- 172.21.15.78:0/269998827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c101fe0 0x7fda0c102440 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd9f80099c0 tx=0x7fd9f8009cd0 comp rx=0 tx=0).stop 2024-03-14T09:51:24.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.681+0000 7fda14600700 1 -- 172.21.15.78:0/269998827 shutdown_connections 2024-03-14T09:51:24.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.682+0000 7fda14600700 1 --2- 172.21.15.78:0/269998827 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda0c102980 0x7fda0c109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:24.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.682+0000 7fda14600700 1 --2- 172.21.15.78:0/269998827 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda0c100de0 0x7fda0c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:24.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.682+0000 7fda14600700 1 --2- 172.21.15.78:0/269998827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c101fe0 0x7fda0c102440 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:24.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.682+0000 7fda14600700 1 -- 172.21.15.78:0/269998827 >> 172.21.15.78:0/269998827 conn(0x7fda0c0fc380 msgr2=0x7fda0c0fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:24.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.682+0000 7fda14600700 1 -- 172.21.15.78:0/269998827 shutdown_connections 2024-03-14T09:51:24.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.682+0000 7fda14600700 1 -- 172.21.15.78:0/269998827 wait complete. 2024-03-14T09:51:24.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.683+0000 7fda14600700 1 Processor -- start 2024-03-14T09:51:24.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.683+0000 7fda14600700 1 -- start start 2024-03-14T09:51:24.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.683+0000 7fda14600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda0c100de0 0x7fda0c074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:24.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.683+0000 7fda14600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda0c101fe0 0x7fda0c0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:24.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.683+0000 7fda14600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c102980 0x7fda0c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:24.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.683+0000 7fda14600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c073e80 con 0x7fda0c102980 2024-03-14T09:51:24.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.683+0000 7fda14600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c073ff0 con 0x7fda0c101fe0 2024-03-14T09:51:24.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.683+0000 7fda14600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c1a11a0 con 0x7fda0c100de0 2024-03-14T09:51:24.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.683+0000 7fda11c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda0c100de0 0x7fda0c074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:24.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.683+0000 7fda11c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda0c100de0 0x7fda0c074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:39626/0 (socket says 172.21.15.78:39626) 2024-03-14T09:51:24.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.683+0000 7fda11c00700 1 -- 172.21.15.78:0/2483361679 learned_addr learned my addr 172.21.15.78:0/2483361679 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:51:24.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.683+0000 7fda12600700 1 --2- 172.21.15.78:0/2483361679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c102980 0x7fda0c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:24.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.684+0000 7fda11200700 1 --2- 172.21.15.78:0/2483361679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda0c101fe0 0x7fda0c0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:24.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.684+0000 7fda11c00700 1 -- 172.21.15.78:0/2483361679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda0c101fe0 msgr2=0x7fda0c0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:24.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.684+0000 7fda11c00700 1 --2- 172.21.15.78:0/2483361679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda0c101fe0 0x7fda0c0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:24.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.684+0000 7fda11c00700 1 -- 172.21.15.78:0/2483361679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c102980 msgr2=0x7fda0c0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:24.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.684+0000 7fda11c00700 1 --2- 172.21.15.78:0/2483361679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c102980 0x7fda0c0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.684+0000 7fda11c00700 1 -- 172.21.15.78:0/2483361679 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f80095f0 con 0x7fda0c100de0 2024-03-14T09:51:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.684+0000 7fda11c00700 1 --2- 172.21.15.78:0/2483361679 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda0c100de0 0x7fda0c074b00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fda0000e970 tx=0x7fda0000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.684+0000 7fda0a000700 1 -- 172.21.15.78:0/2483361679 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda00009790 con 0x7fda0c100de0 2024-03-14T09:51:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.684+0000 7fda0a000700 1 -- 172.21.15.78:0/2483361679 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda0000c7e0 con 0x7fda0c100de0 2024-03-14T09:51:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.684+0000 7fda14600700 1 -- 172.21.15.78:0/2483361679 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0c1a1430 con 0x7fda0c100de0 2024-03-14T09:51:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.684+0000 7fda0a000700 1 -- 172.21.15.78:0/2483361679 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda00003c70 con 0x7fda0c100de0 2024-03-14T09:51:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.684+0000 7fda14600700 1 -- 172.21.15.78:0/2483361679 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0c1a1930 con 0x7fda0c100de0 2024-03-14T09:51:24.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.685+0000 7fda14600700 1 -- 172.21.15.78:0/2483361679 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda0c108260 con 0x7fda0c100de0 2024-03-14T09:51:24.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.687+0000 7fda0a000700 1 -- 172.21.15.78:0/2483361679 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fda0000cc50 con 0x7fda0c100de0 2024-03-14T09:51:24.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.688+0000 7fda0a000700 1 --2- 172.21.15.78:0/2483361679 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd9f40a0a90 0x7fd9f40a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:24.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.689+0000 7fda0a000700 1 -- 172.21.15.78:0/2483361679 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fda00014070 con 0x7fda0c100de0 2024-03-14T09:51:24.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.689+0000 7fda11200700 1 --2- 172.21.15.78:0/2483361679 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd9f40a0a90 0x7fd9f40a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:24.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.689+0000 7fda11200700 1 --2- 172.21.15.78:0/2483361679 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd9f40a0a90 0x7fd9f40a2f50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8003730 tx=0x7fd9f800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:24.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.691+0000 7fda0a000700 1 -- 172.21.15.78:0/2483361679 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fda00095f00 con 0x7fda0c100de0 2024-03-14T09:51:24.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.896+0000 7fda14600700 1 -- 172.21.15.78:0/2483361679 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fda0c04ea80 con 0x7fda0c100de0 2024-03-14T09:51:24.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.897+0000 7fda0a000700 1 -- 172.21.15.78:0/2483361679 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) v1 ==== 74+0+133 (secure 0 0 0) 0x7fda00003dd0 con 0x7fda0c100de0 2024-03-14T09:51:24.897 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:51:24.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.899+0000 7fda14600700 1 -- 172.21.15.78:0/2483361679 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd9f40a0a90 msgr2=0x7fd9f40a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:24.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.899+0000 7fda14600700 1 --2- 172.21.15.78:0/2483361679 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd9f40a0a90 0x7fd9f40a2f50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8003730 tx=0x7fd9f800b540 comp rx=0 tx=0).stop 2024-03-14T09:51:24.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.899+0000 7fda14600700 1 -- 172.21.15.78:0/2483361679 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda0c100de0 msgr2=0x7fda0c074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:24.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.899+0000 7fda14600700 1 --2- 172.21.15.78:0/2483361679 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda0c100de0 0x7fda0c074b00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fda0000e970 tx=0x7fda0000ec80 comp rx=0 tx=0).stop 2024-03-14T09:51:24.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.899+0000 7fda14600700 1 -- 172.21.15.78:0/2483361679 shutdown_connections 2024-03-14T09:51:24.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.899+0000 7fda14600700 1 --2- 172.21.15.78:0/2483361679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda0c102980 0x7fda0c0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:24.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.899+0000 7fda14600700 1 --2- 172.21.15.78:0/2483361679 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fda0c100de0 0x7fda0c074b00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:24.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.899+0000 7fda14600700 1 --2- 172.21.15.78:0/2483361679 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd9f40a0a90 0x7fd9f40a2f50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:24.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.899+0000 7fda14600700 1 --2- 172.21.15.78:0/2483361679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda0c101fe0 0x7fda0c0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:24.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.899+0000 7fda14600700 1 -- 172.21.15.78:0/2483361679 >> 172.21.15.78:0/2483361679 conn(0x7fda0c0fc380 msgr2=0x7fda0c1053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:24.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.899+0000 7fda14600700 1 -- 172.21.15.78:0/2483361679 shutdown_connections 2024-03-14T09:51:24.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:24.899+0000 7fda14600700 1 -- 172.21.15.78:0/2483361679 wait complete. 2024-03-14T09:51:24.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:24 smithi152 ceph-mon[48279]: from='osd.11 [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-14T09:51:24.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:24 smithi152 ceph-mon[48279]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-14T09:51:25.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:24 smithi132 ceph-mon[47959]: from='osd.11 [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-14T09:51:25.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:24 smithi132 ceph-mon[47959]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-14T09:51:25.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:24 smithi078 ceph-mon[38143]: from='osd.11 [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-14T09:51:25.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:24 smithi078 ceph-mon[38143]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-14T09:51:25.290 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":72,"num_osds":12,"num_up_osds":11,"osd_up_since":1710409867,"num_in_osds":12,"osd_in_since":1710409870,"num_remapped_pgs":0} 2024-03-14T09:51:25.762 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T09:51:25.746+0000 7f7cdf9e2700 -1 osd.11 0 waiting for initial osdmap 2024-03-14T09:51:25.763 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-14T09:51:25.763 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: osdmap e72: 12 total, 11 up, 12 in 2024-03-14T09:51:25.763 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:25.763 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: from='osd.11 [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:51:25.763 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:51:25.763 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2483361679' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-14T09:51:25.763 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-03-14T09:51:25.763 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: Detected new or changed devices on smithi152 2024-03-14T09:51:25.763 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:25.763 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:25.763 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:25.763 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:25.763 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:25.763 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:25.763 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi152 to 4278M 2024-03-14T09:51:25.764 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:25.764 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:25.764 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:51:25.764 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:26.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-14T09:51:26.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: osdmap e72: 12 total, 11 up, 12 in 2024-03-14T09:51:26.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:26.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: from='osd.11 [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:51:26.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:51:26.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2483361679' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-14T09:51:26.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-03-14T09:51:26.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: Detected new or changed devices on smithi152 2024-03-14T09:51:26.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:26.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:26.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:26.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:26.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:26.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:26.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi152 to 4278M 2024-03-14T09:51:26.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:26.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:26.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:51:26.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:25 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:26.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-14T09:51:26.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: osdmap e72: 12 total, 11 up, 12 in 2024-03-14T09:51:26.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:26.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: from='osd.11 [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:51:26.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]: dispatch 2024-03-14T09:51:26.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2483361679' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-14T09:51:26.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-03-14T09:51:26.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: Detected new or changed devices on smithi152 2024-03-14T09:51:26.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:26.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:26.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:26.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:26.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:26.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:51:26.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi152 to 4278M 2024-03-14T09:51:26.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:26.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:26.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:51:26.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:25 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:26.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 09:51:25 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T09:51:25.761+0000 7f7cd9fd4700 -1 osd.11 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T09:51:26.290 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-14T09:51:26.643 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:51:27.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:26 smithi132 ceph-mon[47959]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-14T09:51:27.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:26 smithi132 ceph-mon[47959]: osdmap e73: 12 total, 11 up, 12 in 2024-03-14T09:51:27.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:26 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:27.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:26 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:27.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:26 smithi078 ceph-mon[38143]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-14T09:51:27.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:26 smithi078 ceph-mon[38143]: osdmap e73: 12 total, 11 up, 12 in 2024-03-14T09:51:27.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:27.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:26 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:27.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:26 smithi152 ceph-mon[48279]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi152", "root=default"]}]': finished 2024-03-14T09:51:27.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:26 smithi152 ceph-mon[48279]: osdmap e73: 12 total, 11 up, 12 in 2024-03-14T09:51:27.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:27.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:26 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:27.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.760+0000 7f37eb200700 1 -- 172.21.15.78:0/3838534982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e4102790 msgr2=0x7f37e4109040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:27.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.760+0000 7f37eb200700 1 --2- 172.21.15.78:0/3838534982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e4102790 0x7f37e4109040 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f37e0009fc0 tx=0x7f37e000b330 comp rx=0 tx=0).stop 2024-03-14T09:51:27.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.760+0000 7f37eb200700 1 -- 172.21.15.78:0/3838534982 shutdown_connections 2024-03-14T09:51:27.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.760+0000 7f37eb200700 1 --2- 172.21.15.78:0/3838534982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e4102790 0x7f37e4109040 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:27.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.760+0000 7f37eb200700 1 --2- 172.21.15.78:0/3838534982 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37e4100bf0 0x7f37e4101010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:27.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.760+0000 7f37eb200700 1 --2- 172.21.15.78:0/3838534982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37e4101df0 0x7f37e4102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:27.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.760+0000 7f37eb200700 1 -- 172.21.15.78:0/3838534982 >> 172.21.15.78:0/3838534982 conn(0x7f37e40fc190 msgr2=0x7f37e40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:27.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.760+0000 7f37eb200700 1 -- 172.21.15.78:0/3838534982 shutdown_connections 2024-03-14T09:51:27.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.760+0000 7f37eb200700 1 -- 172.21.15.78:0/3838534982 wait complete. 2024-03-14T09:51:27.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.762+0000 7f37eb200700 1 Processor -- start 2024-03-14T09:51:27.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.762+0000 7f37eb200700 1 -- start start 2024-03-14T09:51:27.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.762+0000 7f37eb200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37e4100bf0 0x7f37e41992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:27.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.762+0000 7f37eb200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e4101df0 0x7f37e41997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:27.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.762+0000 7f37eb200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37e4102790 0x7f37e41a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:27.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.762+0000 7f37eb200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e41a0de0 con 0x7f37e4101df0 2024-03-14T09:51:27.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.762+0000 7f37eb200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e41a0f20 con 0x7f37e4102790 2024-03-14T09:51:27.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.762+0000 7f37eb200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e41a10b0 con 0x7f37e4100bf0 2024-03-14T09:51:27.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.762+0000 7f37e9400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e4101df0 0x7f37e41997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:27.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.762+0000 7f37e9400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e4101df0 0x7f37e41997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51164/0 (socket says 172.21.15.78:51164) 2024-03-14T09:51:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.762+0000 7f37e9400700 1 -- 172.21.15.78:0/3883927729 learned_addr learned my addr 172.21.15.78:0/3883927729 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:51:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.762+0000 7f37e9e00700 1 --2- 172.21.15.78:0/3883927729 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37e4100bf0 0x7f37e41992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.763+0000 7f37ea800700 1 --2- 172.21.15.78:0/3883927729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37e4102790 0x7f37e41a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.763+0000 7f37e9400700 1 -- 172.21.15.78:0/3883927729 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37e4100bf0 msgr2=0x7f37e41992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.763+0000 7f37e9400700 1 --2- 172.21.15.78:0/3883927729 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37e4100bf0 0x7f37e41992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.763+0000 7f37e9400700 1 -- 172.21.15.78:0/3883927729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37e4102790 msgr2=0x7f37e41a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.763+0000 7f37e9400700 1 --2- 172.21.15.78:0/3883927729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37e4102790 0x7f37e41a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.763+0000 7f37e9400700 1 -- 172.21.15.78:0/3883927729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d80095f0 con 0x7f37e4101df0 2024-03-14T09:51:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.763+0000 7f37e9e00700 1 --2- 172.21.15.78:0/3883927729 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37e4100bf0 0x7f37e41992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:51:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.763+0000 7f37e9400700 1 --2- 172.21.15.78:0/3883927729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e4101df0 0x7f37e41997e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f37d400e9d0 tx=0x7f37d400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.763+0000 7f37dea00700 1 -- 172.21.15.78:0/3883927729 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d4009790 con 0x7f37e4101df0 2024-03-14T09:51:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.763+0000 7f37dea00700 1 -- 172.21.15.78:0/3883927729 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37d400ee70 con 0x7f37e4101df0 2024-03-14T09:51:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.763+0000 7f37eb200700 1 -- 172.21.15.78:0/3883927729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e0009c40 con 0x7f37e4101df0 2024-03-14T09:51:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.763+0000 7f37dea00700 1 -- 172.21.15.78:0/3883927729 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d4010ea0 con 0x7f37e4101df0 2024-03-14T09:51:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.763+0000 7f37eb200700 1 -- 172.21.15.78:0/3883927729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e41a1750 con 0x7f37e4101df0 2024-03-14T09:51:27.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.764+0000 7f37eb200700 1 -- 172.21.15.78:0/3883927729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e404ea80 con 0x7f37e4101df0 2024-03-14T09:51:27.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.766+0000 7f37dea00700 1 -- 172.21.15.78:0/3883927729 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f37d400cbd0 con 0x7f37e4101df0 2024-03-14T09:51:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.769+0000 7f37dea00700 1 --2- 172.21.15.78:0/3883927729 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f37cc0a0b10 0x7f37cc0a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.769+0000 7f37dea00700 1 -- 172.21.15.78:0/3883927729 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f37d4014070 con 0x7f37e4101df0 2024-03-14T09:51:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.769+0000 7f37e9e00700 1 --2- 172.21.15.78:0/3883927729 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f37cc0a0b10 0x7f37cc0a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:27.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.769+0000 7f37e9e00700 1 --2- 172.21.15.78:0/3883927729 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f37cc0a0b10 0x7f37cc0a2fd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f37e4101c50 tx=0x7f37d8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:27.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.771+0000 7f37dea00700 1 -- 172.21.15.78:0/3883927729 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f37d4095fb0 con 0x7f37e4101df0 2024-03-14T09:51:27.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:27 smithi078 ceph-mon[38143]: purged_snaps scrub starts 2024-03-14T09:51:27.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:27 smithi078 ceph-mon[38143]: purged_snaps scrub ok 2024-03-14T09:51:27.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:27 smithi078 ceph-mon[38143]: osd.11 [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042] boot 2024-03-14T09:51:27.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:27 smithi078 ceph-mon[38143]: osdmap e74: 12 total, 12 up, 12 in 2024-03-14T09:51:27.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:27 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:27.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:27 smithi078 ceph-mon[38143]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:27.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.977+0000 7f37eb200700 1 -- 172.21.15.78:0/3883927729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f37e410ca70 con 0x7f37e4101df0 2024-03-14T09:51:27.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.977+0000 7f37dea00700 1 -- 172.21.15.78:0/3883927729 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v75) v1 ==== 74+0+133 (secure 0 0 0) 0x7f37d40995d0 con 0x7f37e4101df0 2024-03-14T09:51:27.978 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:51:27.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.980+0000 7f37eb200700 1 -- 172.21.15.78:0/3883927729 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f37cc0a0b10 msgr2=0x7f37cc0a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.980+0000 7f37eb200700 1 --2- 172.21.15.78:0/3883927729 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f37cc0a0b10 0x7f37cc0a2fd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f37e4101c50 tx=0x7f37d8009310 comp rx=0 tx=0).stop 2024-03-14T09:51:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.980+0000 7f37eb200700 1 -- 172.21.15.78:0/3883927729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e4101df0 msgr2=0x7f37e41997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.980+0000 7f37eb200700 1 --2- 172.21.15.78:0/3883927729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e4101df0 0x7f37e41997e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f37d400e9d0 tx=0x7f37d400ece0 comp rx=0 tx=0).stop 2024-03-14T09:51:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.980+0000 7f37eb200700 1 -- 172.21.15.78:0/3883927729 shutdown_connections 2024-03-14T09:51:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.980+0000 7f37eb200700 1 --2- 172.21.15.78:0/3883927729 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f37cc0a0b10 0x7f37cc0a2fd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.980+0000 7f37eb200700 1 --2- 172.21.15.78:0/3883927729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37e4102790 0x7f37e41a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.980+0000 7f37eb200700 1 --2- 172.21.15.78:0/3883927729 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37e4100bf0 0x7f37e41992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.980+0000 7f37eb200700 1 --2- 172.21.15.78:0/3883927729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37e4101df0 0x7f37e41997e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.980+0000 7f37eb200700 1 -- 172.21.15.78:0/3883927729 >> 172.21.15.78:0/3883927729 conn(0x7f37e40fc190 msgr2=0x7f37e4072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.980+0000 7f37eb200700 1 -- 172.21.15.78:0/3883927729 shutdown_connections 2024-03-14T09:51:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:27.980+0000 7f37eb200700 1 -- 172.21.15.78:0/3883927729 wait complete. 2024-03-14T09:51:28.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:27 smithi132 ceph-mon[47959]: purged_snaps scrub starts 2024-03-14T09:51:28.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:27 smithi132 ceph-mon[47959]: purged_snaps scrub ok 2024-03-14T09:51:28.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:27 smithi132 ceph-mon[47959]: osd.11 [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042] boot 2024-03-14T09:51:28.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:27 smithi132 ceph-mon[47959]: osdmap e74: 12 total, 12 up, 12 in 2024-03-14T09:51:28.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:27 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:28.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:27 smithi132 ceph-mon[47959]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:28.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:27 smithi152 ceph-mon[48279]: purged_snaps scrub starts 2024-03-14T09:51:28.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:27 smithi152 ceph-mon[48279]: purged_snaps scrub ok 2024-03-14T09:51:28.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:27 smithi152 ceph-mon[48279]: osd.11 [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042] boot 2024-03-14T09:51:28.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:27 smithi152 ceph-mon[48279]: osdmap e74: 12 total, 12 up, 12 in 2024-03-14T09:51:28.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:27 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:51:28.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:27 smithi152 ceph-mon[48279]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:28.350 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":75,"num_osds":12,"num_up_osds":12,"osd_up_since":1710409886,"num_in_osds":12,"osd_in_since":1710409870,"num_remapped_pgs":0} 2024-03-14T09:51:28.351 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-14T09:51:28.704 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:51:29.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:28 smithi132 ceph-mon[47959]: osdmap e75: 12 total, 12 up, 12 in 2024-03-14T09:51:29.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:28 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3883927729' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-14T09:51:29.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:28 smithi078 ceph-mon[38143]: osdmap e75: 12 total, 12 up, 12 in 2024-03-14T09:51:29.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:28 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3883927729' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-14T09:51:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:28 smithi152 ceph-mon[48279]: osdmap e75: 12 total, 12 up, 12 in 2024-03-14T09:51:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:28 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3883927729' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-14T09:51:29.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.822+0000 7f1514a00700 1 -- 172.21.15.78:0/3213487534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15100731e0 msgr2=0x7f15100735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:29.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.822+0000 7f1514a00700 1 --2- 172.21.15.78:0/3213487534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15100731e0 0x7f15100735c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1500009990 tx=0x7f1500009ca0 comp rx=0 tx=0).stop 2024-03-14T09:51:29.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.822+0000 7f1514a00700 1 -- 172.21.15.78:0/3213487534 shutdown_connections 2024-03-14T09:51:29.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.822+0000 7f1514a00700 1 --2- 172.21.15.78:0/3213487534 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f151010d590 0x7f151010fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:29.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.822+0000 7f1514a00700 1 --2- 172.21.15.78:0/3213487534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15100731e0 0x7f15100735c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:29.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.822+0000 7f1514a00700 1 --2- 172.21.15.78:0/3213487534 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1510073b00 0x7f151010d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:29.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.822+0000 7f1514a00700 1 -- 172.21.15.78:0/3213487534 >> 172.21.15.78:0/3213487534 conn(0x7f15100fc800 msgr2=0x7f15100fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:29.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.823+0000 7f1514a00700 1 -- 172.21.15.78:0/3213487534 shutdown_connections 2024-03-14T09:51:29.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.823+0000 7f1514a00700 1 -- 172.21.15.78:0/3213487534 wait complete. 2024-03-14T09:51:29.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.823+0000 7f1514a00700 1 Processor -- start 2024-03-14T09:51:29.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.824+0000 7f1514a00700 1 -- start start 2024-03-14T09:51:29.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.824+0000 7f1514a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15100731e0 0x7f15101997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:29.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.824+0000 7f1514a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1510073b00 0x7f1510199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:29.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.824+0000 7f1514a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f151010d590 0x7f151019d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:29.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.824+0000 7f1514a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151019d600 con 0x7f1510073b00 2024-03-14T09:51:29.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.824+0000 7f1514a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151019d770 con 0x7f15100731e0 2024-03-14T09:51:29.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.824+0000 7f1514a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151019d930 con 0x7f151010d590 2024-03-14T09:51:29.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.824+0000 7f150d400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1510073b00 0x7f1510199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:29.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.824+0000 7f150d400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1510073b00 0x7f1510199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51174/0 (socket says 172.21.15.78:51174) 2024-03-14T09:51:29.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.824+0000 7f150de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15100731e0 0x7f15101997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:29.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.824+0000 7f150d400700 1 -- 172.21.15.78:0/447869607 learned_addr learned my addr 172.21.15.78:0/447869607 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:51:29.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.824+0000 7f150e800700 1 --2- 172.21.15.78:0/447869607 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f151010d590 0x7f151019d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:29.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.825+0000 7f150d400700 1 -- 172.21.15.78:0/447869607 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f151010d590 msgr2=0x7f151019d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:29.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.825+0000 7f150d400700 1 --2- 172.21.15.78:0/447869607 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f151010d590 0x7f151019d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:29.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.825+0000 7f150d400700 1 -- 172.21.15.78:0/447869607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15100731e0 msgr2=0x7f15101997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:29.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.825+0000 7f150d400700 1 --2- 172.21.15.78:0/447869607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15100731e0 0x7f15101997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:29.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.825+0000 7f150d400700 1 -- 172.21.15.78:0/447869607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15000095f0 con 0x7f1510073b00 2024-03-14T09:51:29.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.825+0000 7f150d400700 1 --2- 172.21.15.78:0/447869607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1510073b00 0x7f1510199d00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00d950 tx=0x7f14fc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:29.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.825+0000 7f1506a00700 1 -- 172.21.15.78:0/447869607 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc00f890 con 0x7f1510073b00 2024-03-14T09:51:29.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.825+0000 7f1506a00700 1 -- 172.21.15.78:0/447869607 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14fc010460 con 0x7f1510073b00 2024-03-14T09:51:29.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.825+0000 7f1514a00700 1 -- 172.21.15.78:0/447869607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151019dc10 con 0x7f1510073b00 2024-03-14T09:51:29.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.825+0000 7f1506a00700 1 -- 172.21.15.78:0/447869607 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc00e610 con 0x7f1510073b00 2024-03-14T09:51:29.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.825+0000 7f1514a00700 1 -- 172.21.15.78:0/447869607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15101a6560 con 0x7f1510073b00 2024-03-14T09:51:29.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.827+0000 7f1514a00700 1 -- 172.21.15.78:0/447869607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15101087a0 con 0x7f1510073b00 2024-03-14T09:51:29.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.828+0000 7f1506a00700 1 -- 172.21.15.78:0/447869607 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f14fc0108d0 con 0x7f1510073b00 2024-03-14T09:51:29.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.830+0000 7f1506a00700 1 --2- 172.21.15.78:0/447869607 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f14f40a0a90 0x7f14f40a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:29.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.830+0000 7f1506a00700 1 -- 172.21.15.78:0/447869607 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f14fc0cb9f0 con 0x7f1510073b00 2024-03-14T09:51:29.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.830+0000 7f150de00700 1 --2- 172.21.15.78:0/447869607 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f14f40a0a90 0x7f14f40a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:29.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.830+0000 7f150de00700 1 --2- 172.21.15.78:0/447869607 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f14f40a0a90 0x7f14f40a2f50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1500006010 tx=0x7f150000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:29.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:29.833+0000 7f1506a00700 1 -- 172.21.15.78:0/447869607 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f14fc0957c0 con 0x7f1510073b00 2024-03-14T09:51:30.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:29 smithi132 ceph-mon[47959]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:30.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:30.037+0000 7f1514a00700 1 -- 172.21.15.78:0/447869607 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f151019aad0 con 0x7f1510073b00 2024-03-14T09:51:30.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:30.038+0000 7f1506a00700 1 -- 172.21.15.78:0/447869607 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+18387 (secure 0 0 0) 0x7f14fc098de0 con 0x7f1510073b00 2024-03-14T09:51:30.039 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:51:30.039 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":75,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:51:27.749782+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:51:30.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:30.040+0000 7f1514a00700 1 -- 172.21.15.78:0/447869607 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f14f40a0a90 msgr2=0x7f14f40a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:30.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:30.040+0000 7f1514a00700 1 --2- 172.21.15.78:0/447869607 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f14f40a0a90 0x7f14f40a2f50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1500006010 tx=0x7f150000b540 comp rx=0 tx=0).stop 2024-03-14T09:51:30.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:30.040+0000 7f1514a00700 1 -- 172.21.15.78:0/447869607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1510073b00 msgr2=0x7f1510199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:30.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:30.040+0000 7f1514a00700 1 --2- 172.21.15.78:0/447869607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1510073b00 0x7f1510199d00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00d950 tx=0x7f14fc00dc60 comp rx=0 tx=0).stop 2024-03-14T09:51:30.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:30.040+0000 7f1514a00700 1 -- 172.21.15.78:0/447869607 shutdown_connections 2024-03-14T09:51:30.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:30.040+0000 7f1514a00700 1 --2- 172.21.15.78:0/447869607 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f14f40a0a90 0x7f14f40a2f50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:30.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:30.040+0000 7f1514a00700 1 --2- 172.21.15.78:0/447869607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15100731e0 0x7f15101997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:30.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:30.040+0000 7f1514a00700 1 --2- 172.21.15.78:0/447869607 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f151010d590 0x7f151019d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:30.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:30.040+0000 7f1514a00700 1 --2- 172.21.15.78:0/447869607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1510073b00 0x7f1510199d00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:30.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:30.040+0000 7f1514a00700 1 -- 172.21.15.78:0/447869607 >> 172.21.15.78:0/447869607 conn(0x7f15100fc800 msgr2=0x7f15100fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:30.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:30.041+0000 7f1514a00700 1 -- 172.21.15.78:0/447869607 shutdown_connections 2024-03-14T09:51:30.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:30.041+0000 7f1514a00700 1 -- 172.21.15.78:0/447869607 wait complete. 2024-03-14T09:51:30.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:29 smithi078 ceph-mon[38143]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:30.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:29 smithi152 ceph-mon[48279]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:30.423 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T09:48:37.618450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-14T09:51:30.423 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-14T09:51:30.778 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:51:31.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:30 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/447869607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:51:31.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:30 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/447869607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:51:31.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:30 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/447869607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:51:31.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.924+0000 7f1dd2600700 1 -- 172.21.15.78:0/1571843027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1dcc073900 msgr2=0x7f1dcc10d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.924+0000 7f1dd2600700 1 --2- 172.21.15.78:0/1571843027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1dcc073900 0x7f1dcc10d2d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009990 tx=0x7f1db4009ca0 comp rx=0 tx=0).stop 2024-03-14T09:51:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.924+0000 7f1dd2600700 1 -- 172.21.15.78:0/1571843027 shutdown_connections 2024-03-14T09:51:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.924+0000 7f1dd2600700 1 --2- 172.21.15.78:0/1571843027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dcc072fe0 0x7f1dcc0733c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.924+0000 7f1dd2600700 1 --2- 172.21.15.78:0/1571843027 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1dcc10d810 0x7f1dcc10fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.924+0000 7f1dd2600700 1 --2- 172.21.15.78:0/1571843027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1dcc073900 0x7f1dcc10d2d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.924+0000 7f1dd2600700 1 -- 172.21.15.78:0/1571843027 >> 172.21.15.78:0/1571843027 conn(0x7f1dcc077ed0 msgr2=0x7f1dcc0782e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.924+0000 7f1dd2600700 1 -- 172.21.15.78:0/1571843027 shutdown_connections 2024-03-14T09:51:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.924+0000 7f1dd2600700 1 -- 172.21.15.78:0/1571843027 wait complete. 2024-03-14T09:51:31.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.925+0000 7f1dd2600700 1 Processor -- start 2024-03-14T09:51:31.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.926+0000 7f1dd2600700 1 -- start start 2024-03-14T09:51:31.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.926+0000 7f1dd2600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1dcc072fe0 0x7f1dcc19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:31.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.926+0000 7f1dd2600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dcc073900 0x7f1dcc19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:31.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.926+0000 7f1dd2600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1dcc10d810 0x7f1dcc199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:31.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.926+0000 7f1dd2600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dcc199cc0 con 0x7f1dcc10d810 2024-03-14T09:51:31.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.926+0000 7f1dd2600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dcc199e30 con 0x7f1dcc073900 2024-03-14T09:51:31.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.926+0000 7f1dd2600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dcc199ff0 con 0x7f1dcc072fe0 2024-03-14T09:51:31.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.926+0000 7f1dcbe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1dcc10d810 0x7f1dcc199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:31.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.926+0000 7f1dcbe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1dcc10d810 0x7f1dcc199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51192/0 (socket says 172.21.15.78:51192) 2024-03-14T09:51:31.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.926+0000 7f1dcbe00700 1 -- 172.21.15.78:0/693034599 learned_addr learned my addr 172.21.15.78:0/693034599 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:51:31.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.926+0000 7f1dc3e00700 1 --2- 172.21.15.78:0/693034599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dcc073900 0x7f1dcc19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:31.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.926+0000 7f1dcb400700 1 --2- 172.21.15.78:0/693034599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1dcc072fe0 0x7f1dcc19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:31.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.927+0000 7f1dcbe00700 1 -- 172.21.15.78:0/693034599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1dcc072fe0 msgr2=0x7f1dcc19f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:31.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.927+0000 7f1dcbe00700 1 --2- 172.21.15.78:0/693034599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1dcc072fe0 0x7f1dcc19f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:31.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.927+0000 7f1dcbe00700 1 -- 172.21.15.78:0/693034599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dcc073900 msgr2=0x7f1dcc19fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:31.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.927+0000 7f1dcbe00700 1 --2- 172.21.15.78:0/693034599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dcc073900 0x7f1dcc19fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:31.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.927+0000 7f1dcbe00700 1 -- 172.21.15.78:0/693034599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db40095f0 con 0x7f1dcc10d810 2024-03-14T09:51:31.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.927+0000 7f1dcbe00700 1 --2- 172.21.15.78:0/693034599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1dcc10d810 0x7f1dcc199660 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40094f0 tx=0x7f1dc400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:31.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.927+0000 7f1dc8c00700 1 -- 172.21.15.78:0/693034599 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc4004db0 con 0x7f1dcc10d810 2024-03-14T09:51:31.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.927+0000 7f1dc8c00700 1 -- 172.21.15.78:0/693034599 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1dc400f040 con 0x7f1dcc10d810 2024-03-14T09:51:31.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.927+0000 7f1dd2600700 1 -- 172.21.15.78:0/693034599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dcc19a2d0 con 0x7f1dcc10d810 2024-03-14T09:51:31.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.927+0000 7f1dc8c00700 1 -- 172.21.15.78:0/693034599 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc4005980 con 0x7f1dcc10d810 2024-03-14T09:51:31.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.927+0000 7f1dd2600700 1 -- 172.21.15.78:0/693034599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dcc1a6510 con 0x7f1dcc10d810 2024-03-14T09:51:31.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.928+0000 7f1dd2600700 1 -- 172.21.15.78:0/693034599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dcc108a20 con 0x7f1dcc10d810 2024-03-14T09:51:31.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.930+0000 7f1dc8c00700 1 -- 172.21.15.78:0/693034599 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f1dc4005020 con 0x7f1dcc10d810 2024-03-14T09:51:31.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.931+0000 7f1dc8c00700 1 --2- 172.21.15.78:0/693034599 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1db00a0b10 0x7f1db00a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:31.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.932+0000 7f1dc8c00700 1 -- 172.21.15.78:0/693034599 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1dc40cb420 con 0x7f1dcc10d810 2024-03-14T09:51:31.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.932+0000 7f1dcb400700 1 --2- 172.21.15.78:0/693034599 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1db00a0b10 0x7f1db00a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:31.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.932+0000 7f1dcb400700 1 --2- 172.21.15.78:0/693034599 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1db00a0b10 0x7f1db00a2fd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc007be0 tx=0x7f1dbc0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:31.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:31.936+0000 7f1dc8c00700 1 -- 172.21.15.78:0/693034599 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1dc40951f0 con 0x7f1dcc10d810 2024-03-14T09:51:32.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:31 smithi078 ceph-mon[38143]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:32.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:32.135+0000 7f1dd2600700 1 -- 172.21.15.78:0/693034599 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1dcc19ab20 con 0x7f1dcc10d810 2024-03-14T09:51:32.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:32.135+0000 7f1dc8c00700 1 -- 172.21.15.78:0/693034599 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1dc4098810 con 0x7f1dcc10d810 2024-03-14T09:51:32.136 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-14T09:51:32.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:32.138+0000 7f1dd2600700 1 -- 172.21.15.78:0/693034599 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1db00a0b10 msgr2=0x7f1db00a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:32.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:32.138+0000 7f1dd2600700 1 --2- 172.21.15.78:0/693034599 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1db00a0b10 0x7f1db00a2fd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc007be0 tx=0x7f1dbc0073d0 comp rx=0 tx=0).stop 2024-03-14T09:51:32.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:32.138+0000 7f1dd2600700 1 -- 172.21.15.78:0/693034599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1dcc10d810 msgr2=0x7f1dcc199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:32.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:32.138+0000 7f1dd2600700 1 --2- 172.21.15.78:0/693034599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1dcc10d810 0x7f1dcc199660 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40094f0 tx=0x7f1dc400d3b0 comp rx=0 tx=0).stop 2024-03-14T09:51:32.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:32.138+0000 7f1dd2600700 1 -- 172.21.15.78:0/693034599 shutdown_connections 2024-03-14T09:51:32.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:32.138+0000 7f1dd2600700 1 --2- 172.21.15.78:0/693034599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1dcc10d810 0x7f1dcc199660 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:32.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:32.138+0000 7f1dd2600700 1 --2- 172.21.15.78:0/693034599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1dcc072fe0 0x7f1dcc19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:32.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:32.138+0000 7f1dd2600700 1 --2- 172.21.15.78:0/693034599 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1db00a0b10 0x7f1db00a2fd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:32.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:32.138+0000 7f1dd2600700 1 --2- 172.21.15.78:0/693034599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1dcc073900 0x7f1dcc19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:32.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:32.138+0000 7f1dd2600700 1 -- 172.21.15.78:0/693034599 >> 172.21.15.78:0/693034599 conn(0x7f1dcc077ed0 msgr2=0x7f1dcc0ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:32.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:32.138+0000 7f1dd2600700 1 -- 172.21.15.78:0/693034599 shutdown_connections 2024-03-14T09:51:32.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:32.138+0000 7f1dd2600700 1 -- 172.21.15.78:0/693034599 wait complete. 2024-03-14T09:51:32.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:31 smithi152 ceph-mon[48279]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:32.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:31 smithi132 ceph-mon[47959]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:32.518 INFO:tasks.cephadm:Adding mds.a on smithi078 2024-03-14T09:51:32.518 INFO:tasks.cephadm:Adding mds.d on smithi078 2024-03-14T09:51:32.518 INFO:tasks.cephadm:Adding mds.g on smithi078 2024-03-14T09:51:32.518 INFO:tasks.cephadm:Adding mds.j on smithi078 2024-03-14T09:51:32.518 INFO:tasks.cephadm:Adding mds.b on smithi132 2024-03-14T09:51:32.518 INFO:tasks.cephadm:Adding mds.e on smithi132 2024-03-14T09:51:32.518 INFO:tasks.cephadm:Adding mds.h on smithi132 2024-03-14T09:51:32.518 INFO:tasks.cephadm:Adding mds.k on smithi132 2024-03-14T09:51:32.518 INFO:tasks.cephadm:Adding mds.c on smithi152 2024-03-14T09:51:32.518 INFO:tasks.cephadm:Adding mds.f on smithi152 2024-03-14T09:51:32.518 INFO:tasks.cephadm:Adding mds.i on smithi152 2024-03-14T09:51:32.518 INFO:tasks.cephadm:Adding mds.l on smithi152 2024-03-14T09:51:32.519 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l' 2024-03-14T09:51:32.752 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.c/config 2024-03-14T09:51:32.905 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:32 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/693034599' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:51:33.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:32 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/693034599' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:51:33.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:32 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/693034599' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:51:33.876 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.874+0000 7fe2b1049700 1 -- 172.21.15.152:0/432111808 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2ac102070 msgr2=0x7fe2ac1024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:34.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.874+0000 7fe2b1049700 1 --2- 172.21.15.152:0/432111808 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2ac102070 0x7fe2ac1024d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0009990 tx=0x7fe2a0009ca0 comp rx=0 tx=0).stop 2024-03-14T09:51:34.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.874+0000 7fe2b1049700 1 -- 172.21.15.152:0/432111808 shutdown_connections 2024-03-14T09:51:34.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.874+0000 7fe2b1049700 1 --2- 172.21.15.152:0/432111808 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2ac102a10 0x7fe2ac1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:34.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.874+0000 7fe2b1049700 1 --2- 172.21.15.152:0/432111808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2ac0ff090 0x7fe2ac0ff4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:34.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.874+0000 7fe2b1049700 1 --2- 172.21.15.152:0/432111808 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2ac102070 0x7fe2ac1024d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:34.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.874+0000 7fe2b1049700 1 -- 172.21.15.152:0/432111808 >> 172.21.15.152:0/432111808 conn(0x7fe2ac0fa9f0 msgr2=0x7fe2ac0fce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:34.004 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.874+0000 7fe2b1049700 1 -- 172.21.15.152:0/432111808 shutdown_connections 2024-03-14T09:51:34.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.874+0000 7fe2b1049700 1 -- 172.21.15.152:0/432111808 wait complete. 2024-03-14T09:51:34.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.875+0000 7fe2b1049700 1 Processor -- start 2024-03-14T09:51:34.005 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.875+0000 7fe2b1049700 1 -- start start 2024-03-14T09:51:34.007 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe2b1049700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2ac0ff090 0x7fe2ac101980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:34.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe2b1049700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2ac102070 0x7fe2ac100020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:34.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe2b1049700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2ac102a10 0x7fe2ac100560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:34.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe2b1049700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ac100d00 con 0x7fe2ac102a10 2024-03-14T09:51:34.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe2b1049700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ac100e70 con 0x7fe2ac102070 2024-03-14T09:51:34.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe2b1049700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ac1a1210 con 0x7fe2ac0ff090 2024-03-14T09:51:34.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe2aad9d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2ac0ff090 0x7fe2ac101980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:34.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe2aad9d700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2ac0ff090 0x7fe2ac101980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.152:46780/0 (socket says 172.21.15.152:46780) 2024-03-14T09:51:34.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe2aad9d700 1 -- 172.21.15.152:0/852352792 learned_addr learned my addr 172.21.15.152:0/852352792 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-03-14T09:51:34.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe2aa59c700 1 --2- 172.21.15.152:0/852352792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2ac102070 0x7fe2ac100020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:34.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe2aad9d700 1 -- 172.21.15.152:0/852352792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2ac102070 msgr2=0x7fe2ac100020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:34.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe2aad9d700 1 --2- 172.21.15.152:0/852352792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2ac102070 0x7fe2ac100020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:34.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe2aad9d700 1 -- 172.21.15.152:0/852352792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2ac102a10 msgr2=0x7fe2ac100560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:34.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe2aad9d700 1 --2- 172.21.15.152:0/852352792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2ac102a10 0x7fe2ac100560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:34.008 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe2aad9d700 1 -- 172.21.15.152:0/852352792 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a00095f0 con 0x7fe2ac0ff090 2024-03-14T09:51:34.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe2aad9d700 1 --2- 172.21.15.152:0/852352792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2ac0ff090 0x7fe2ac101980 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe29c00b500 tx=0x7fe29c00b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:34.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe293fff700 1 -- 172.21.15.152:0/852352792 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29c011830 con 0x7fe2ac0ff090 2024-03-14T09:51:34.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.876+0000 7fe293fff700 1 -- 172.21.15.152:0/852352792 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe29c011e50 con 0x7fe2ac0ff090 2024-03-14T09:51:34.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.877+0000 7fe293fff700 1 -- 172.21.15.152:0/852352792 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29c00f2e0 con 0x7fe2ac0ff090 2024-03-14T09:51:34.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.877+0000 7fe2b1049700 1 -- 172.21.15.152:0/852352792 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2ac1a14a0 con 0x7fe2ac0ff090 2024-03-14T09:51:34.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.877+0000 7fe2b1049700 1 -- 172.21.15.152:0/852352792 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2ac1a1950 con 0x7fe2ac0ff090 2024-03-14T09:51:34.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.878+0000 7fe2b1049700 1 -- 172.21.15.152:0/852352792 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2ac108350 con 0x7fe2ac0ff090 2024-03-14T09:51:34.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.879+0000 7fe293fff700 1 -- 172.21.15.152:0/852352792 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fe29c010580 con 0x7fe2ac0ff090 2024-03-14T09:51:34.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.879+0000 7fe293fff700 1 --2- 172.21.15.152:0/852352792 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe2940a0a90 0x7fe2940a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:34.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.879+0000 7fe293fff700 1 -- 172.21.15.152:0/852352792 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe29c0cb610 con 0x7fe2ac0ff090 2024-03-14T09:51:34.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.880+0000 7fe2aa59c700 1 --2- 172.21.15.152:0/852352792 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe2940a0a90 0x7fe2940a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:34.010 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.880+0000 7fe2aa59c700 1 --2- 172.21.15.152:0/852352792 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe2940a0a90 0x7fe2940a2f50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe2a000b5c0 tx=0x7fe2a0003790 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:34.010 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:33.884+0000 7fe293fff700 1 -- 172.21.15.152:0/852352792 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe29c095480 con 0x7fe2ac0ff090 2024-03-14T09:51:34.010 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:33 smithi152 ceph-mon[48279]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:34.056 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:33 smithi078 ceph-mon[38143]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:34.086 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:34.085+0000 7fe2b1049700 1 -- 172.21.15.152:0/852352792 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2ac02ce20 con 0x7fe2940a0a90 2024-03-14T09:51:34.099 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:34.094+0000 7fe293fff700 1 -- 172.21.15.152:0/852352792 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fe2ac02ce20 con 0x7fe2940a0a90 2024-03-14T09:51:34.100 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled mds.all update... 2024-03-14T09:51:34.101 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:34.096+0000 7fe2b1049700 1 -- 172.21.15.152:0/852352792 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe2940a0a90 msgr2=0x7fe2940a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:34.101 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:34.096+0000 7fe2b1049700 1 --2- 172.21.15.152:0/852352792 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe2940a0a90 0x7fe2940a2f50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe2a000b5c0 tx=0x7fe2a0003790 comp rx=0 tx=0).stop 2024-03-14T09:51:34.101 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:34.096+0000 7fe2b1049700 1 -- 172.21.15.152:0/852352792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2ac0ff090 msgr2=0x7fe2ac101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:34.101 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:34.096+0000 7fe2b1049700 1 --2- 172.21.15.152:0/852352792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2ac0ff090 0x7fe2ac101980 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe29c00b500 tx=0x7fe29c00b810 comp rx=0 tx=0).stop 2024-03-14T09:51:34.101 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:34.096+0000 7fe2b1049700 1 -- 172.21.15.152:0/852352792 shutdown_connections 2024-03-14T09:51:34.101 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:34.096+0000 7fe2b1049700 1 --2- 172.21.15.152:0/852352792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2ac102a10 0x7fe2ac100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:34.101 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:34.096+0000 7fe2b1049700 1 --2- 172.21.15.152:0/852352792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe2ac0ff090 0x7fe2ac101980 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:34.101 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:34.096+0000 7fe2b1049700 1 --2- 172.21.15.152:0/852352792 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe2940a0a90 0x7fe2940a2f50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:34.102 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:34.096+0000 7fe2b1049700 1 --2- 172.21.15.152:0/852352792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe2ac102070 0x7fe2ac100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:34.102 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:34.096+0000 7fe2b1049700 1 -- 172.21.15.152:0/852352792 >> 172.21.15.152:0/852352792 conn(0x7fe2ac0fa9f0 msgr2=0x7fe2ac105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:34.102 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:34.096+0000 7fe2b1049700 1 -- 172.21.15.152:0/852352792 shutdown_connections 2024-03-14T09:51:34.102 INFO:teuthology.orchestra.run.smithi152.stderr:2024-03-14T09:51:34.096+0000 7fe2b1049700 1 -- 172.21.15.152:0/852352792 wait complete. 2024-03-14T09:51:34.272 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:33 smithi132 ceph-mon[47959]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:34.509 DEBUG:teuthology.orchestra.run.smithi078:mds.a> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.a.service 2024-03-14T09:51:34.515 DEBUG:teuthology.orchestra.run.smithi078:mds.d> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.d.service 2024-03-14T09:51:34.562 DEBUG:teuthology.orchestra.run.smithi078:mds.g> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.g.service 2024-03-14T09:51:34.585 INFO:journalctl@ceph.mds.a.smithi078.stdout:-- Logs begin at Thu 2024-03-14 09:30:54 UTC. -- 2024-03-14T09:51:34.592 DEBUG:teuthology.orchestra.run.smithi078:mds.j> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.j.service 2024-03-14T09:51:34.629 INFO:journalctl@ceph.mds.d.smithi078.stdout:-- Logs begin at Thu 2024-03-14 09:30:54 UTC. -- 2024-03-14T09:51:34.631 DEBUG:teuthology.orchestra.run.smithi132:mds.b> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.b.service 2024-03-14T09:51:34.635 DEBUG:teuthology.orchestra.run.smithi132:mds.e> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.e.service 2024-03-14T09:51:34.655 INFO:journalctl@ceph.mds.g.smithi078.stdout:-- Logs begin at Thu 2024-03-14 09:30:54 UTC. -- 2024-03-14T09:51:34.673 INFO:journalctl@ceph.mds.b.smithi132.stdout:-- Logs begin at Thu 2024-03-14 09:24:09 UTC. -- 2024-03-14T09:51:34.675 DEBUG:teuthology.orchestra.run.smithi132:mds.h> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.h.service 2024-03-14T09:51:34.696 INFO:journalctl@ceph.mds.j.smithi078.stdout:-- Logs begin at Thu 2024-03-14 09:30:54 UTC. -- 2024-03-14T09:51:34.705 INFO:journalctl@ceph.mds.e.smithi132.stdout:-- Logs begin at Thu 2024-03-14 09:24:09 UTC. -- 2024-03-14T09:51:34.707 DEBUG:teuthology.orchestra.run.smithi132:mds.k> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.k.service 2024-03-14T09:51:34.734 INFO:journalctl@ceph.mds.h.smithi132.stdout:-- Logs begin at Thu 2024-03-14 09:24:09 UTC. -- 2024-03-14T09:51:34.736 DEBUG:teuthology.orchestra.run.smithi152:mds.c> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.c.service 2024-03-14T09:51:34.739 DEBUG:teuthology.orchestra.run.smithi152:mds.f> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.f.service 2024-03-14T09:51:34.763 INFO:journalctl@ceph.mds.k.smithi132.stdout:-- Logs begin at Thu 2024-03-14 09:24:09 UTC. -- 2024-03-14T09:51:34.771 INFO:journalctl@ceph.mds.c.smithi152.stdout:-- Logs begin at Thu 2024-03-14 09:24:45 UTC. -- 2024-03-14T09:51:34.772 DEBUG:teuthology.orchestra.run.smithi152:mds.i> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.i.service 2024-03-14T09:51:34.798 INFO:journalctl@ceph.mds.f.smithi152.stdout:-- Logs begin at Thu 2024-03-14 09:24:45 UTC. -- 2024-03-14T09:51:34.800 DEBUG:teuthology.orchestra.run.smithi152:mds.l> sudo journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.l.service 2024-03-14T09:51:34.826 INFO:journalctl@ceph.mds.i.smithi152.stdout:-- Logs begin at Thu 2024-03-14 09:24:45 UTC. -- 2024-03-14T09:51:34.828 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-14T09:51:34.828 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:51:34.854 INFO:journalctl@ceph.mds.l.smithi152.stdout:-- Logs begin at Thu 2024-03-14 09:24:45 UTC. -- 2024-03-14T09:51:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.032+0000 7fcc78800700 1 -- 172.21.15.78:0/3983908793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc701027e0 msgr2=0x7fcc70109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.032+0000 7fcc78800700 1 --2- 172.21.15.78:0/3983908793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc701027e0 0x7fcc70109090 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c00b0d0 tx=0x7fcc6c00b3e0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.032+0000 7fcc78800700 1 -- 172.21.15.78:0/3983908793 shutdown_connections 2024-03-14T09:51:35.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.032+0000 7fcc78800700 1 --2- 172.21.15.78:0/3983908793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc70100c50 0x7fcc70101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.032+0000 7fcc78800700 1 --2- 172.21.15.78:0/3983908793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc701027e0 0x7fcc70109090 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.032+0000 7fcc78800700 1 --2- 172.21.15.78:0/3983908793 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc70101e40 0x7fcc701022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.032+0000 7fcc78800700 1 -- 172.21.15.78:0/3983908793 >> 172.21.15.78:0/3983908793 conn(0x7fcc700fc1c0 msgr2=0x7fcc700fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.032+0000 7fcc78800700 1 -- 172.21.15.78:0/3983908793 shutdown_connections 2024-03-14T09:51:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.032+0000 7fcc78800700 1 -- 172.21.15.78:0/3983908793 wait complete. 2024-03-14T09:51:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.033+0000 7fcc78800700 1 Processor -- start 2024-03-14T09:51:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.033+0000 7fcc78800700 1 -- start start 2024-03-14T09:51:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.033+0000 7fcc78800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc70100c50 0x7fcc70199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.033+0000 7fcc78800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc70101e40 0x7fcc70199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.033+0000 7fcc78800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc701027e0 0x7fcc701a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.034+0000 7fcc78800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc701a0e70 con 0x7fcc70100c50 2024-03-14T09:51:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.034+0000 7fcc78800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc701a0fb0 con 0x7fcc701027e0 2024-03-14T09:51:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.034+0000 7fcc78800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc701a1140 con 0x7fcc70101e40 2024-03-14T09:51:35.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.034+0000 7fcc75e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc70100c50 0x7fcc70199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.034+0000 7fcc75e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc70100c50 0x7fcc70199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42078/0 (socket says 172.21.15.78:42078) 2024-03-14T09:51:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.034+0000 7fcc75e00700 1 -- 172.21.15.78:0/1044797241 learned_addr learned my addr 172.21.15.78:0/1044797241 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:51:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.034+0000 7fcc75400700 1 --2- 172.21.15.78:0/1044797241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc70101e40 0x7fcc70199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.034+0000 7fcc76800700 1 --2- 172.21.15.78:0/1044797241 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc701027e0 0x7fcc701a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.034+0000 7fcc76800700 1 -- 172.21.15.78:0/1044797241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc70101e40 msgr2=0x7fcc70199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.034+0000 7fcc76800700 1 --2- 172.21.15.78:0/1044797241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc70101e40 0x7fcc70199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.034+0000 7fcc76800700 1 -- 172.21.15.78:0/1044797241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc70100c50 msgr2=0x7fcc70199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.034+0000 7fcc76800700 1 --2- 172.21.15.78:0/1044797241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc70100c50 0x7fcc70199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.034+0000 7fcc76800700 1 -- 172.21.15.78:0/1044797241 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6c009d00 con 0x7fcc701027e0 2024-03-14T09:51:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.034+0000 7fcc75e00700 1 --2- 172.21.15.78:0/1044797241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc70100c50 0x7fcc70199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:51:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.034+0000 7fcc76800700 1 --2- 172.21.15.78:0/1044797241 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc701027e0 0x7fcc701a08a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c007b60 tx=0x7fcc6c008e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.035+0000 7fcc6aa00700 1 -- 172.21.15.78:0/1044797241 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c00e040 con 0x7fcc701027e0 2024-03-14T09:51:35.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.035+0000 7fcc78800700 1 -- 172.21.15.78:0/1044797241 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc701a13c0 con 0x7fcc701027e0 2024-03-14T09:51:35.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.035+0000 7fcc6aa00700 1 -- 172.21.15.78:0/1044797241 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc6c005070 con 0x7fcc701027e0 2024-03-14T09:51:35.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.035+0000 7fcc6aa00700 1 -- 172.21.15.78:0/1044797241 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c01d6d0 con 0x7fcc701027e0 2024-03-14T09:51:35.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.035+0000 7fcc78800700 1 -- 172.21.15.78:0/1044797241 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc701a17e0 con 0x7fcc701027e0 2024-03-14T09:51:35.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.036+0000 7fcc78800700 1 -- 172.21.15.78:0/1044797241 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc700679e0 con 0x7fcc701027e0 2024-03-14T09:51:35.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.037+0000 7fcc6aa00700 1 -- 172.21.15.78:0/1044797241 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fcc6c0051e0 con 0x7fcc701027e0 2024-03-14T09:51:35.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.039+0000 7fcc6aa00700 1 --2- 172.21.15.78:0/1044797241 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcc580a0a90 0x7fcc580a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:35.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.039+0000 7fcc6aa00700 1 -- 172.21.15.78:0/1044797241 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcc6c0ccba0 con 0x7fcc701027e0 2024-03-14T09:51:35.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.039+0000 7fcc75e00700 1 --2- 172.21.15.78:0/1044797241 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcc580a0a90 0x7fcc580a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:35.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.039+0000 7fcc75e00700 1 --2- 172.21.15.78:0/1044797241 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcc580a0a90 0x7fcc580a2f50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcc64005950 tx=0x7fcc64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:35.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.043+0000 7fcc6aa00700 1 -- 172.21.15.78:0/1044797241 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcc6c096a00 con 0x7fcc701027e0 2024-03-14T09:51:35.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.266+0000 7fcc78800700 1 -- 172.21.15.78:0/1044797241 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc7010c8e0 con 0x7fcc701027e0 2024-03-14T09:51:35.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.267+0000 7fcc6aa00700 1 -- 172.21.15.78:0/1044797241 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+18387 (secure 0 0 0) 0x7fcc6c09a020 con 0x7fcc701027e0 2024-03-14T09:51:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:51:35.268 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":75,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:51:27.749782+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:51:35.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.269+0000 7fcc78800700 1 -- 172.21.15.78:0/1044797241 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcc580a0a90 msgr2=0x7fcc580a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:35.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.269+0000 7fcc78800700 1 --2- 172.21.15.78:0/1044797241 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcc580a0a90 0x7fcc580a2f50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcc64005950 tx=0x7fcc64009260 comp rx=0 tx=0).stop 2024-03-14T09:51:35.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.270+0000 7fcc78800700 1 -- 172.21.15.78:0/1044797241 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc701027e0 msgr2=0x7fcc701a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:35.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.270+0000 7fcc78800700 1 --2- 172.21.15.78:0/1044797241 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc701027e0 0x7fcc701a08a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c007b60 tx=0x7fcc6c008e90 comp rx=0 tx=0).stop 2024-03-14T09:51:35.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.270+0000 7fcc78800700 1 -- 172.21.15.78:0/1044797241 shutdown_connections 2024-03-14T09:51:35.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.270+0000 7fcc78800700 1 --2- 172.21.15.78:0/1044797241 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcc580a0a90 0x7fcc580a2f50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.270+0000 7fcc78800700 1 --2- 172.21.15.78:0/1044797241 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc701027e0 0x7fcc701a08a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.270+0000 7fcc78800700 1 --2- 172.21.15.78:0/1044797241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc70100c50 0x7fcc70199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.270+0000 7fcc78800700 1 --2- 172.21.15.78:0/1044797241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcc70101e40 0x7fcc70199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.270+0000 7fcc78800700 1 -- 172.21.15.78:0/1044797241 >> 172.21.15.78:0/1044797241 conn(0x7fcc700fc1c0 msgr2=0x7fcc70072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:35.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.270+0000 7fcc78800700 1 -- 172.21.15.78:0/1044797241 shutdown_connections 2024-03-14T09:51:35.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.270+0000 7fcc78800700 1 -- 172.21.15.78:0/1044797241 wait complete. 2024-03-14T09:51:35.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:35 smithi078 ceph-mon[38143]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:51:35.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:35 smithi078 ceph-mon[38143]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l;count:12 2024-03-14T09:51:35.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:35 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:35.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:35 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:51:35.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:35 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:35.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:35 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:51:35.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:35 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:35.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:35 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:35.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:35 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:35.288 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:35 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:35.291 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T09:48:37.618450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-14T09:51:35.291 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T09:51:35.400 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:35 smithi152 ceph-mon[48279]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:51:35.400 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:35 smithi152 ceph-mon[48279]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l;count:12 2024-03-14T09:51:35.400 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:35 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:35.400 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:35 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:51:35.400 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:35 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:35.400 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:35 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:51:35.400 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:35 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:35.400 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:35 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:35.401 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:35 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:35.401 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:35 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:35.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.485+0000 7fd39bc00700 1 -- 172.21.15.78:0/414894721 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd394101f40 msgr2=0x7fd3941023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:35.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.485+0000 7fd39bc00700 1 --2- 172.21.15.78:0/414894721 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd394101f40 0x7fd3941023a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd3800099c0 tx=0x7fd380009cd0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.486+0000 7fd39bc00700 1 -- 172.21.15.78:0/414894721 shutdown_connections 2024-03-14T09:51:35.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.486+0000 7fd39bc00700 1 --2- 172.21.15.78:0/414894721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd394100d50 0x7fd394101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.486+0000 7fd39bc00700 1 --2- 172.21.15.78:0/414894721 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3941028e0 0x7fd394109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.486+0000 7fd39bc00700 1 --2- 172.21.15.78:0/414894721 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd394101f40 0x7fd3941023a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.486+0000 7fd39bc00700 1 -- 172.21.15.78:0/414894721 >> 172.21.15.78:0/414894721 conn(0x7fd3940fc2c0 msgr2=0x7fd3940fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:35.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.486+0000 7fd39bc00700 1 -- 172.21.15.78:0/414894721 shutdown_connections 2024-03-14T09:51:35.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.486+0000 7fd39bc00700 1 -- 172.21.15.78:0/414894721 wait complete. 2024-03-14T09:51:35.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.487+0000 7fd39bc00700 1 Processor -- start 2024-03-14T09:51:35.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.487+0000 7fd39bc00700 1 -- start start 2024-03-14T09:51:35.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.487+0000 7fd39bc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd394100d50 0x7fd394074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:35.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.487+0000 7fd39bc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd394101f40 0x7fd3940731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:35.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.487+0000 7fd39bc00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3941028e0 0x7fd3940736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:35.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.487+0000 7fd39bc00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd394073e80 con 0x7fd394100d50 2024-03-14T09:51:35.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.487+0000 7fd39bc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd394073ff0 con 0x7fd394101f40 2024-03-14T09:51:35.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.487+0000 7fd39bc00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3941a1170 con 0x7fd3941028e0 2024-03-14T09:51:35.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.487+0000 7fd399200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd394100d50 0x7fd394074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:35.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.487+0000 7fd399200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd394100d50 0x7fd394074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42106/0 (socket says 172.21.15.78:42106) 2024-03-14T09:51:35.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.487+0000 7fd399200700 1 -- 172.21.15.78:0/1251162840 learned_addr learned my addr 172.21.15.78:0/1251162840 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:51:35.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.487+0000 7fd399c00700 1 --2- 172.21.15.78:0/1251162840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3941028e0 0x7fd3940736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:35.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.488+0000 7fd393e00700 1 --2- 172.21.15.78:0/1251162840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd394101f40 0x7fd3940731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:35.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.488+0000 7fd399c00700 1 -- 172.21.15.78:0/1251162840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd394101f40 msgr2=0x7fd3940731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:35.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.488+0000 7fd399c00700 1 --2- 172.21.15.78:0/1251162840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd394101f40 0x7fd3940731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.488+0000 7fd399c00700 1 -- 172.21.15.78:0/1251162840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd394100d50 msgr2=0x7fd394074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:35.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.488+0000 7fd399c00700 1 --2- 172.21.15.78:0/1251162840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd394100d50 0x7fd394074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.488+0000 7fd399c00700 1 -- 172.21.15.78:0/1251162840 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3800095f0 con 0x7fd3941028e0 2024-03-14T09:51:35.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.488+0000 7fd399200700 1 --2- 172.21.15.78:0/1251162840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd394100d50 0x7fd394074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:51:35.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.488+0000 7fd399c00700 1 --2- 172.21.15.78:0/1251162840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3941028e0 0x7fd3940736e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd38c00efc0 tx=0x7fd38c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:35.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.488+0000 7fd391600700 1 -- 172.21.15.78:0/1251162840 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c018070 con 0x7fd3941028e0 2024-03-14T09:51:35.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.488+0000 7fd391600700 1 -- 172.21.15.78:0/1251162840 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd38c00f040 con 0x7fd3941028e0 2024-03-14T09:51:35.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.488+0000 7fd39bc00700 1 -- 172.21.15.78:0/1251162840 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3941a13a0 con 0x7fd3941028e0 2024-03-14T09:51:35.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.488+0000 7fd391600700 1 -- 172.21.15.78:0/1251162840 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c00cd40 con 0x7fd3941028e0 2024-03-14T09:51:35.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.488+0000 7fd39bc00700 1 -- 172.21.15.78:0/1251162840 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3941a1850 con 0x7fd3941028e0 2024-03-14T09:51:35.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.489+0000 7fd39bc00700 1 -- 172.21.15.78:0/1251162840 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3941081d0 con 0x7fd3941028e0 2024-03-14T09:51:35.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.491+0000 7fd391600700 1 -- 172.21.15.78:0/1251162840 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fd38c003a00 con 0x7fd3941028e0 2024-03-14T09:51:35.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.492+0000 7fd391600700 1 --2- 172.21.15.78:0/1251162840 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd37c0a0b10 0x7fd37c0a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:35.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.493+0000 7fd391600700 1 -- 172.21.15.78:0/1251162840 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd38c0cb810 con 0x7fd3941028e0 2024-03-14T09:51:35.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.493+0000 7fd399200700 1 --2- 172.21.15.78:0/1251162840 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd37c0a0b10 0x7fd37c0a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:35.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.493+0000 7fd399200700 1 --2- 172.21.15.78:0/1251162840 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd37c0a0b10 0x7fd37c0a2fd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd3880105c0 tx=0x7fd38800b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:35.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.496+0000 7fd391600700 1 -- 172.21.15.78:0/1251162840 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd38c095670 con 0x7fd3941028e0 2024-03-14T09:51:35.523 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:35 smithi132 ceph-mon[47959]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:51:35.524 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:35 smithi132 ceph-mon[47959]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l;count:12 2024-03-14T09:51:35.524 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:35 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:35.524 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:35 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:51:35.524 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:35 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:35.524 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:35 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:51:35.524 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:35 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:35.524 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:35 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:35.524 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:35 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:35.524 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:35 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:35.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.700+0000 7fd39bc00700 1 -- 172.21.15.78:0/1251162840 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd39410cc00 con 0x7fd3941028e0 2024-03-14T09:51:35.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.701+0000 7fd391600700 1 -- 172.21.15.78:0/1251162840 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd38c098c90 con 0x7fd3941028e0 2024-03-14T09:51:35.702 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-14T09:51:35.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.704+0000 7fd39bc00700 1 -- 172.21.15.78:0/1251162840 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd37c0a0b10 msgr2=0x7fd37c0a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:35.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.704+0000 7fd39bc00700 1 --2- 172.21.15.78:0/1251162840 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd37c0a0b10 0x7fd37c0a2fd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd3880105c0 tx=0x7fd38800b4a0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.704+0000 7fd39bc00700 1 -- 172.21.15.78:0/1251162840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3941028e0 msgr2=0x7fd3940736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:35.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.704+0000 7fd39bc00700 1 --2- 172.21.15.78:0/1251162840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3941028e0 0x7fd3940736e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd38c00efc0 tx=0x7fd38c00c3a0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.704+0000 7fd39bc00700 1 -- 172.21.15.78:0/1251162840 shutdown_connections 2024-03-14T09:51:35.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.704+0000 7fd39bc00700 1 --2- 172.21.15.78:0/1251162840 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3941028e0 0x7fd3940736e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.704+0000 7fd39bc00700 1 --2- 172.21.15.78:0/1251162840 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd394100d50 0x7fd394074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.704+0000 7fd39bc00700 1 --2- 172.21.15.78:0/1251162840 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd37c0a0b10 0x7fd37c0a2fd0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.704+0000 7fd39bc00700 1 --2- 172.21.15.78:0/1251162840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd394101f40 0x7fd3940731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.704+0000 7fd39bc00700 1 -- 172.21.15.78:0/1251162840 >> 172.21.15.78:0/1251162840 conn(0x7fd3940fc2c0 msgr2=0x7fd394105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:35.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.704+0000 7fd39bc00700 1 -- 172.21.15.78:0/1251162840 shutdown_connections 2024-03-14T09:51:35.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.704+0000 7fd39bc00700 1 -- 172.21.15.78:0/1251162840 wait complete. 2024-03-14T09:51:35.724 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:51:35.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.917+0000 7f285f600700 1 -- 172.21.15.78:0/2892699533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28581027e0 msgr2=0x7f2858109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:35.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.917+0000 7f285f600700 1 --2- 172.21.15.78:0/2892699533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28581027e0 0x7f2858109090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f285000b0d0 tx=0x7f285000b3e0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.917+0000 7f285f600700 1 -- 172.21.15.78:0/2892699533 shutdown_connections 2024-03-14T09:51:35.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.917+0000 7f285f600700 1 --2- 172.21.15.78:0/2892699533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2858100c50 0x7f2858101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.917+0000 7f285f600700 1 --2- 172.21.15.78:0/2892699533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28581027e0 0x7f2858109090 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.917+0000 7f285f600700 1 --2- 172.21.15.78:0/2892699533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2858101e40 0x7f28581022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.917+0000 7f285f600700 1 -- 172.21.15.78:0/2892699533 >> 172.21.15.78:0/2892699533 conn(0x7f28580fc1c0 msgr2=0x7f28580fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:35.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.917+0000 7f285f600700 1 -- 172.21.15.78:0/2892699533 shutdown_connections 2024-03-14T09:51:35.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.917+0000 7f285f600700 1 -- 172.21.15.78:0/2892699533 wait complete. 2024-03-14T09:51:35.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.918+0000 7f285f600700 1 Processor -- start 2024-03-14T09:51:35.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.918+0000 7f285f600700 1 -- start start 2024-03-14T09:51:35.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.918+0000 7f285f600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2858100c50 0x7f2858074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:35.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.918+0000 7f285f600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2858101e40 0x7f28580731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:35.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.918+0000 7f285f600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28581027e0 0x7f28580736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:35.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.918+0000 7f285f600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2858073e80 con 0x7f2858101e40 2024-03-14T09:51:35.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.918+0000 7f285f600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2858073ff0 con 0x7f2858100c50 2024-03-14T09:51:35.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.918+0000 7f285f600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28581a1140 con 0x7f28581027e0 2024-03-14T09:51:35.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.919+0000 7f2857e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2858101e40 0x7f28580731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:35.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.919+0000 7f285cc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2858100c50 0x7f2858074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:35.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.919+0000 7f2857e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2858101e40 0x7f28580731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42114/0 (socket says 172.21.15.78:42114) 2024-03-14T09:51:35.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.919+0000 7f2857e00700 1 -- 172.21.15.78:0/241066247 learned_addr learned my addr 172.21.15.78:0/241066247 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:51:35.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.919+0000 7f285d600700 1 --2- 172.21.15.78:0/241066247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28581027e0 0x7f28580736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:35.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.919+0000 7f2857e00700 1 -- 172.21.15.78:0/241066247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28581027e0 msgr2=0x7f28580736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:35.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.919+0000 7f2857e00700 1 --2- 172.21.15.78:0/241066247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28581027e0 0x7f28580736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.919+0000 7f2857e00700 1 -- 172.21.15.78:0/241066247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2858100c50 msgr2=0x7f2858074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:35.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.919+0000 7f2857e00700 1 --2- 172.21.15.78:0/241066247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2858100c50 0x7f2858074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:35.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.919+0000 7f2857e00700 1 -- 172.21.15.78:0/241066247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2850009d00 con 0x7f2858101e40 2024-03-14T09:51:35.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.919+0000 7f285cc00700 1 --2- 172.21.15.78:0/241066247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2858100c50 0x7f2858074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:51:35.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.919+0000 7f2857e00700 1 --2- 172.21.15.78:0/241066247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2858101e40 0x7f28580731a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f284400e9a0 tx=0x7f284400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:35.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.919+0000 7f2855600700 1 -- 172.21.15.78:0/241066247 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844009790 con 0x7f2858101e40 2024-03-14T09:51:35.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.920+0000 7f2855600700 1 -- 172.21.15.78:0/241066247 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f284400ce70 con 0x7f2858101e40 2024-03-14T09:51:35.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.920+0000 7f285f600700 1 -- 172.21.15.78:0/241066247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28581a13d0 con 0x7f2858101e40 2024-03-14T09:51:35.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.920+0000 7f2855600700 1 -- 172.21.15.78:0/241066247 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844018420 con 0x7f2858101e40 2024-03-14T09:51:35.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.920+0000 7f285f600700 1 -- 172.21.15.78:0/241066247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28581a18e0 con 0x7f2858101e40 2024-03-14T09:51:35.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.921+0000 7f285f600700 1 -- 172.21.15.78:0/241066247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2858108080 con 0x7f2858101e40 2024-03-14T09:51:35.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.922+0000 7f2855600700 1 -- 172.21.15.78:0/241066247 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f28440106f0 con 0x7f2858101e40 2024-03-14T09:51:35.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.924+0000 7f2855600700 1 --2- 172.21.15.78:0/241066247 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f28400a0a90 0x7f28400a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:35.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.924+0000 7f2855600700 1 -- 172.21.15.78:0/241066247 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2844014070 con 0x7f2858101e40 2024-03-14T09:51:35.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.924+0000 7f285cc00700 1 --2- 172.21.15.78:0/241066247 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f28400a0a90 0x7f28400a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:35.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.924+0000 7f285cc00700 1 --2- 172.21.15.78:0/241066247 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f28400a0a90 0x7f28400a2f50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2858101ca0 tx=0x7f284c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:35.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:35.928+0000 7f2855600700 1 -- 172.21.15.78:0/241066247 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2844052960 con 0x7f2858101e40 2024-03-14T09:51:36.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:36.182+0000 7f285f600700 1 -- 172.21.15.78:0/241066247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28580679e0 con 0x7f2858101e40 2024-03-14T09:51:36.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:36.182+0000 7f2855600700 1 -- 172.21.15.78:0/241066247 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f2844095b70 con 0x7f2858101e40 2024-03-14T09:51:36.183 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:51:36.183 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-14T09:51:36.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:36.184+0000 7f285f600700 1 -- 172.21.15.78:0/241066247 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f28400a0a90 msgr2=0x7f28400a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:36.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:36.185+0000 7f285f600700 1 --2- 172.21.15.78:0/241066247 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f28400a0a90 0x7f28400a2f50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2858101ca0 tx=0x7f284c009260 comp rx=0 tx=0).stop 2024-03-14T09:51:36.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:36.185+0000 7f285f600700 1 -- 172.21.15.78:0/241066247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2858101e40 msgr2=0x7f28580731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:36.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:36.185+0000 7f285f600700 1 --2- 172.21.15.78:0/241066247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2858101e40 0x7f28580731a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f284400e9a0 tx=0x7f284400ecb0 comp rx=0 tx=0).stop 2024-03-14T09:51:36.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:36.185+0000 7f285f600700 1 -- 172.21.15.78:0/241066247 shutdown_connections 2024-03-14T09:51:36.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:36.185+0000 7f285f600700 1 --2- 172.21.15.78:0/241066247 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f28400a0a90 0x7f28400a2f50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:36.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:36.185+0000 7f285f600700 1 --2- 172.21.15.78:0/241066247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2858100c50 0x7f2858074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:36.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:36.185+0000 7f285f600700 1 --2- 172.21.15.78:0/241066247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28581027e0 0x7f28580736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:36.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:36.185+0000 7f285f600700 1 --2- 172.21.15.78:0/241066247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2858101e40 0x7f28580731a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:36.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:36.185+0000 7f285f600700 1 -- 172.21.15.78:0/241066247 >> 172.21.15.78:0/241066247 conn(0x7f28580fc1c0 msgr2=0x7f28581051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:36.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:36.185+0000 7f285f600700 1 -- 172.21.15.78:0/241066247 shutdown_connections 2024-03-14T09:51:36.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:36.185+0000 7f285f600700 1 -- 172.21.15.78:0/241066247 wait complete. 2024-03-14T09:51:36.187 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 1 2024-03-14T09:51:36.338 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:36 smithi152 ceph-mon[48279]: Deploying daemon mds.l on smithi152 2024-03-14T09:51:36.338 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:36 smithi152 ceph-mon[48279]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:36.338 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:36 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1044797241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:51:36.338 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:36 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1251162840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:51:36.338 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 14 09:51:36 smithi152 systemd[1]: Starting Ceph mds.l for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T09:51:36.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:36 smithi132 ceph-mon[47959]: Deploying daemon mds.l on smithi152 2024-03-14T09:51:36.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:36 smithi132 ceph-mon[47959]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:36.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:36 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1044797241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:51:36.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:36 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1251162840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:51:36.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:36 smithi078 ceph-mon[38143]: Deploying daemon mds.l on smithi152 2024-03-14T09:51:36.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:36 smithi078 ceph-mon[38143]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:36.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:36 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1044797241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:51:36.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:36 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1251162840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:51:36.749 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 14 09:51:36 smithi152 ceph-mds[84796]: -- 172.21.15.152:0/3357256038 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5589279081a0 con 0x5589278df000 2024-03-14T09:51:36.749 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 14 09:51:36 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-l[84773]: starting mds.l at 2024-03-14T09:51:36.749 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 14 09:51:36 smithi152 bash[84758]: e452c2e7aa9eb9c80bea7fac6d5636cf347e6ae390d445b463d4389657b2678b 2024-03-14T09:51:36.749 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 14 09:51:36 smithi152 systemd[1]: Started Ceph mds.l for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:51:37.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:37 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/241066247' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:37.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:37 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.152:6832/4187874184,v1:172.21.15.152:6833/4187874184] up:boot 2024-03-14T09:51:37.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:37 smithi132 ceph-mon[47959]: fsmap 1 up:standby 2024-03-14T09:51:37.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:37 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-14T09:51:37.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:37 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:37.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:37 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:37.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:37 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:37.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:37 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:37.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:37 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:37.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:37 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:37.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:37 smithi132 ceph-mon[47959]: Deploying daemon mds.k on smithi132 2024-03-14T09:51:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:37 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/241066247' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:37 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.152:6832/4187874184,v1:172.21.15.152:6833/4187874184] up:boot 2024-03-14T09:51:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:37 smithi152 ceph-mon[48279]: fsmap 1 up:standby 2024-03-14T09:51:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:37 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-14T09:51:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:37 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:37 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:37.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:37 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:37.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:37 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:37.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:37 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:37.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:37 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:37.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:37 smithi152 ceph-mon[48279]: Deploying daemon mds.k on smithi132 2024-03-14T09:51:37.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:37 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/241066247' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:37.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:37 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.152:6832/4187874184,v1:172.21.15.152:6833/4187874184] up:boot 2024-03-14T09:51:37.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:37 smithi078 ceph-mon[38143]: fsmap 1 up:standby 2024-03-14T09:51:37.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:37 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-14T09:51:37.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:37 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:37.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:37 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:37.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:37 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:37.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:37 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:37.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:37 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:37.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:37 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:37.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:37 smithi078 ceph-mon[38143]: Deploying daemon mds.k on smithi132 2024-03-14T09:51:38.020 INFO:journalctl@ceph.mds.k.smithi132.stdout:Mar 14 09:51:37 smithi132 systemd[1]: Starting Ceph mds.k for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T09:51:38.205 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:51:38.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.414+0000 7f758e400700 1 -- 172.21.15.78:0/3376342891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7588074df0 msgr2=0x7f7588073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:38.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.414+0000 7f758e400700 1 --2- 172.21.15.78:0/3376342891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7588074df0 0x7f7588073240 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7578009990 tx=0x7f7578009ca0 comp rx=0 tx=0).stop 2024-03-14T09:51:38.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.414+0000 7f758e400700 1 -- 172.21.15.78:0/3376342891 shutdown_connections 2024-03-14T09:51:38.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.414+0000 7f758e400700 1 --2- 172.21.15.78:0/3376342891 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7588108b10 0x7f758810afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:38.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.414+0000 7f758e400700 1 --2- 172.21.15.78:0/3376342891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7588074df0 0x7f7588073240 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:38.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.414+0000 7f758e400700 1 --2- 172.21.15.78:0/3376342891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7588073810 0x7f7588073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:38.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.414+0000 7f758e400700 1 -- 172.21.15.78:0/3376342891 >> 172.21.15.78:0/3376342891 conn(0x7f75880fc1c0 msgr2=0x7f75880fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:38.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.414+0000 7f758e400700 1 -- 172.21.15.78:0/3376342891 shutdown_connections 2024-03-14T09:51:38.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.414+0000 7f758e400700 1 -- 172.21.15.78:0/3376342891 wait complete. 2024-03-14T09:51:38.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.415+0000 7f758e400700 1 Processor -- start 2024-03-14T09:51:38.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.415+0000 7f758e400700 1 -- start start 2024-03-14T09:51:38.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f758e400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7588073810 0x7f7588199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f758e400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7588074df0 0x7f7588199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f758e400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7588108b10 0x7f75881a0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f758e400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75881a0d00 con 0x7f7588108b10 2024-03-14T09:51:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f758e400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75881a0e40 con 0x7f7588074df0 2024-03-14T09:51:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f758e400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75881a0fd0 con 0x7f7588073810 2024-03-14T09:51:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f7587e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7588108b10 0x7f75881a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f7586a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7588074df0 0x7f7588199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f7587400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7588073810 0x7f7588199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f7586a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7588074df0 0x7f7588199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:54920/0 (socket says 172.21.15.78:54920) 2024-03-14T09:51:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f7586a00700 1 -- 172.21.15.78:0/990705116 learned_addr learned my addr 172.21.15.78:0/990705116 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:51:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f7587400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7588073810 0x7f7588199220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:42060/0 (socket says 172.21.15.78:42060) 2024-03-14T09:51:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f7586a00700 1 -- 172.21.15.78:0/990705116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7588073810 msgr2=0x7f7588199220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f7586a00700 1 --2- 172.21.15.78:0/990705116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7588073810 0x7f7588199220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f7586a00700 1 -- 172.21.15.78:0/990705116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7588108b10 msgr2=0x7f75881a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:38.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f7586a00700 1 --2- 172.21.15.78:0/990705116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7588108b10 0x7f75881a0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:38.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f7586a00700 1 -- 172.21.15.78:0/990705116 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75780095f0 con 0x7f7588074df0 2024-03-14T09:51:38.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.416+0000 7f7587e00700 1 --2- 172.21.15.78:0/990705116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7588108b10 0x7f75881a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:51:38.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.417+0000 7f7586a00700 1 --2- 172.21.15.78:0/990705116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7588074df0 0x7f7588199760 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f757400e970 tx=0x7f757400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:38.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.417+0000 7f757fe00700 1 -- 172.21.15.78:0/990705116 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7574009790 con 0x7f7588074df0 2024-03-14T09:51:38.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.417+0000 7f757fe00700 1 -- 172.21.15.78:0/990705116 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f757400c7e0 con 0x7f7588074df0 2024-03-14T09:51:38.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.417+0000 7f758e400700 1 -- 172.21.15.78:0/990705116 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75881a12b0 con 0x7f7588074df0 2024-03-14T09:51:38.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.417+0000 7f757fe00700 1 -- 172.21.15.78:0/990705116 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7574003c70 con 0x7f7588074df0 2024-03-14T09:51:38.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.417+0000 7f758e400700 1 -- 172.21.15.78:0/990705116 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75881a17c0 con 0x7f7588074df0 2024-03-14T09:51:38.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.418+0000 7f758e400700 1 -- 172.21.15.78:0/990705116 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75881078d0 con 0x7f7588074df0 2024-03-14T09:51:38.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.420+0000 7f757fe00700 1 -- 172.21.15.78:0/990705116 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f757400cc50 con 0x7f7588074df0 2024-03-14T09:51:38.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.421+0000 7f757fe00700 1 --2- 172.21.15.78:0/990705116 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f756c0a0a90 0x7f756c0a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:38.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.422+0000 7f757fe00700 1 -- 172.21.15.78:0/990705116 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7574014070 con 0x7f7588074df0 2024-03-14T09:51:38.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.422+0000 7f7587400700 1 --2- 172.21.15.78:0/990705116 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f756c0a0a90 0x7f756c0a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:38.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.422+0000 7f7587400700 1 --2- 172.21.15.78:0/990705116 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f756c0a0a90 0x7f756c0a2f50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f757800b2e0 tx=0x7f7578005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:38.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.425+0000 7f757fe00700 1 -- 172.21.15.78:0/990705116 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7574095bf0 con 0x7f7588074df0 2024-03-14T09:51:38.520 INFO:journalctl@ceph.mds.k.smithi132.stdout:Mar 14 09:51:38 smithi132 ceph-mds[85022]: -- 172.21.15.132:0/3773549394 <== mon.2 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eaa3d0c1a0 con 0x55eaa3ce2c00 2024-03-14T09:51:38.520 INFO:journalctl@ceph.mds.k.smithi132.stdout:Mar 14 09:51:38 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-k[84999]: starting mds.k at 2024-03-14T09:51:38.520 INFO:journalctl@ceph.mds.k.smithi132.stdout:Mar 14 09:51:38 smithi132 bash[84984]: db03ab2e61b8debdea0e04383fd76717f3b8716ff5778501dfd6535cea44f032 2024-03-14T09:51:38.520 INFO:journalctl@ceph.mds.k.smithi132.stdout:Mar 14 09:51:38 smithi132 systemd[1]: Started Ceph mds.k for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:51:38.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.697+0000 7f758e400700 1 -- 172.21.15.78:0/990705116 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75880679e0 con 0x7f7588074df0 2024-03-14T09:51:38.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.698+0000 7f757fe00700 1 -- 172.21.15.78:0/990705116 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2017 (secure 0 0 0) 0x7f7574003dd0 con 0x7f7588074df0 2024-03-14T09:51:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:51:38.699 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-03-14T09:51:38.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.701+0000 7f758e400700 1 -- 172.21.15.78:0/990705116 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f756c0a0a90 msgr2=0x7f756c0a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:38.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.701+0000 7f758e400700 1 --2- 172.21.15.78:0/990705116 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f756c0a0a90 0x7f756c0a2f50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f757800b2e0 tx=0x7f7578005fd0 comp rx=0 tx=0).stop 2024-03-14T09:51:38.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.701+0000 7f758e400700 1 -- 172.21.15.78:0/990705116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7588074df0 msgr2=0x7f7588199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:38.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.701+0000 7f758e400700 1 --2- 172.21.15.78:0/990705116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7588074df0 0x7f7588199760 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f757400e970 tx=0x7f757400ec80 comp rx=0 tx=0).stop 2024-03-14T09:51:38.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.701+0000 7f758e400700 1 -- 172.21.15.78:0/990705116 shutdown_connections 2024-03-14T09:51:38.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.701+0000 7f758e400700 1 --2- 172.21.15.78:0/990705116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7588108b10 0x7f75881a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:38.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.701+0000 7f758e400700 1 --2- 172.21.15.78:0/990705116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7588073810 0x7f7588199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:38.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.701+0000 7f758e400700 1 --2- 172.21.15.78:0/990705116 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f756c0a0a90 0x7f756c0a2f50 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:38.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.701+0000 7f758e400700 1 --2- 172.21.15.78:0/990705116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7588074df0 0x7f7588199760 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:38.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.701+0000 7f758e400700 1 -- 172.21.15.78:0/990705116 >> 172.21.15.78:0/990705116 conn(0x7f75880fc1c0 msgr2=0x7f7588104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:38.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.701+0000 7f758e400700 1 -- 172.21.15.78:0/990705116 shutdown_connections 2024-03-14T09:51:38.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:38.701+0000 7f758e400700 1 -- 172.21.15.78:0/990705116 wait complete. 2024-03-14T09:51:38.703 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 3 2024-03-14T09:51:38.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:38 smithi078 ceph-mon[38143]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:38.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:38 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:38.900 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:38 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:38.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:38 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:38.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:38 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:38.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:38 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:38.901 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:38 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:38.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:38 smithi152 ceph-mon[48279]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:38.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:38 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:38.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:38 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:38.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:38 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:38.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:38 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:38.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:38 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:38.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:38 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:39.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:38 smithi132 ceph-mon[47959]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:39.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:38 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:39.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:38 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:39.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:38 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:39.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:38 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:39.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:38 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:39.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:38 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:39.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:39 smithi078 ceph-mon[38143]: Deploying daemon mds.j on smithi078 2024-03-14T09:51:39.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:39 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.132:6832/726198552,v1:172.21.15.132:6833/726198552] up:boot 2024-03-14T09:51:39.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:39 smithi078 ceph-mon[38143]: fsmap 2 up:standby 2024-03-14T09:51:39.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:39 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-14T09:51:39.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:39 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/990705116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:39.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:39 smithi152 ceph-mon[48279]: Deploying daemon mds.j on smithi078 2024-03-14T09:51:39.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:39 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.132:6832/726198552,v1:172.21.15.132:6833/726198552] up:boot 2024-03-14T09:51:39.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:39 smithi152 ceph-mon[48279]: fsmap 2 up:standby 2024-03-14T09:51:39.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:39 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-14T09:51:39.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:39 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/990705116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:40.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:39 smithi132 ceph-mon[47959]: Deploying daemon mds.j on smithi078 2024-03-14T09:51:40.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:39 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.132:6832/726198552,v1:172.21.15.132:6833/726198552] up:boot 2024-03-14T09:51:40.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:39 smithi132 ceph-mon[47959]: fsmap 2 up:standby 2024-03-14T09:51:40.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:39 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-14T09:51:40.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:39 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/990705116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:40.053 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 14 09:51:40 smithi078 systemd[1]: Starting Ceph mds.j for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T09:51:40.722 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:51:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:40 smithi078 ceph-mon[38143]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:40.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:40 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:40.786 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 14 09:51:40 smithi078 ceph-mds[86724]: -- 172.21.15.78:0/2482229043 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5564ba5561a0 con 0x5564ba52e800 2024-03-14T09:51:40.786 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 14 09:51:40 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-j[86701]: starting mds.j at 2024-03-14T09:51:40.786 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 14 09:51:40 smithi078 bash[86686]: 484e71a0bc4407124552f0f4f0c5611f2d37904bf76d069b2ad3beb9f27538c4 2024-03-14T09:51:40.786 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 14 09:51:40 smithi078 systemd[1]: Started Ceph mds.j for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:51:40.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.927+0000 7f3890a00700 1 -- 172.21.15.78:0/601252983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c10d220 msgr2=0x7f388c10f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:40.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.927+0000 7f3890a00700 1 --2- 172.21.15.78:0/601252983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c10d220 0x7f388c10f610 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f388400b0a0 tx=0x7f388400b3b0 comp rx=0 tx=0).stop 2024-03-14T09:51:40.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.927+0000 7f3890a00700 1 -- 172.21.15.78:0/601252983 shutdown_connections 2024-03-14T09:51:40.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.927+0000 7f3890a00700 1 --2- 172.21.15.78:0/601252983 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f388c074df0 0x7f388c073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:40.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.927+0000 7f3890a00700 1 --2- 172.21.15.78:0/601252983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c10d220 0x7f388c10f610 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:40.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.927+0000 7f3890a00700 1 --2- 172.21.15.78:0/601252983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f388c073810 0x7f388c073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:40.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.927+0000 7f3890a00700 1 -- 172.21.15.78:0/601252983 >> 172.21.15.78:0/601252983 conn(0x7f388c0fc1c0 msgr2=0x7f388c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:40.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.927+0000 7f3890a00700 1 -- 172.21.15.78:0/601252983 shutdown_connections 2024-03-14T09:51:40.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.927+0000 7f3890a00700 1 -- 172.21.15.78:0/601252983 wait complete. 2024-03-14T09:51:40.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.928+0000 7f3890a00700 1 Processor -- start 2024-03-14T09:51:40.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.928+0000 7f3890a00700 1 -- start start 2024-03-14T09:51:40.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.928+0000 7f3890a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f388c073810 0x7f388c19d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:40.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.928+0000 7f3890a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f388c074df0 0x7f388c19dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:40.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.928+0000 7f3890a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c10d220 0x7f388c1a4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:40.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.928+0000 7f3890a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c1a5230 con 0x7f388c10d220 2024-03-14T09:51:40.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.928+0000 7f3890a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c1a5370 con 0x7f388c073810 2024-03-14T09:51:40.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.928+0000 7f3890a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c1a5500 con 0x7f388c074df0 2024-03-14T09:51:40.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.929+0000 7f388a800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c10d220 0x7f388c1a4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:40.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.929+0000 7f388a800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c10d220 0x7f388c1a4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42178/0 (socket says 172.21.15.78:42178) 2024-03-14T09:51:40.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.929+0000 7f388a800700 1 -- 172.21.15.78:0/1446596244 learned_addr learned my addr 172.21.15.78:0/1446596244 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:51:40.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.929+0000 7f3889400700 1 --2- 172.21.15.78:0/1446596244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f388c074df0 0x7f388c19dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:40.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.929+0000 7f3889e00700 1 --2- 172.21.15.78:0/1446596244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f388c073810 0x7f388c19d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:40.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.929+0000 7f388a800700 1 -- 172.21.15.78:0/1446596244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f388c074df0 msgr2=0x7f388c19dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:40.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.929+0000 7f388a800700 1 --2- 172.21.15.78:0/1446596244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f388c074df0 0x7f388c19dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:40.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.929+0000 7f388a800700 1 -- 172.21.15.78:0/1446596244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f388c073810 msgr2=0x7f388c19d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.929+0000 7f388a800700 1 --2- 172.21.15.78:0/1446596244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f388c073810 0x7f388c19d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.929+0000 7f388a800700 1 -- 172.21.15.78:0/1446596244 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3884009d00 con 0x7f388c10d220 2024-03-14T09:51:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.929+0000 7f388a800700 1 --2- 172.21.15.78:0/1446596244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c10d220 0x7f388c1a4c60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3884007d00 tx=0x7f3884005ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.929+0000 7f3882a00700 1 -- 172.21.15.78:0/1446596244 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388400e040 con 0x7f388c10d220 2024-03-14T09:51:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.930+0000 7f3890a00700 1 -- 172.21.15.78:0/1446596244 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388c1a5780 con 0x7f388c10d220 2024-03-14T09:51:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.930+0000 7f3882a00700 1 -- 172.21.15.78:0/1446596244 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3884003e10 con 0x7f388c10d220 2024-03-14T09:51:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.930+0000 7f3882a00700 1 -- 172.21.15.78:0/1446596244 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3884012a90 con 0x7f388c10d220 2024-03-14T09:51:40.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.930+0000 7f3890a00700 1 -- 172.21.15.78:0/1446596244 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f388c1a5c20 con 0x7f388c10d220 2024-03-14T09:51:40.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.931+0000 7f3890a00700 1 -- 172.21.15.78:0/1446596244 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388c103480 con 0x7f388c10d220 2024-03-14T09:51:40.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.933+0000 7f3882a00700 1 -- 172.21.15.78:0/1446596244 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f38840056c0 con 0x7f388c10d220 2024-03-14T09:51:40.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.934+0000 7f3882a00700 1 --2- 172.21.15.78:0/1446596244 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f38700a0a90 0x7f38700a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:40.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.934+0000 7f3882a00700 1 -- 172.21.15.78:0/1446596244 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f38840cd160 con 0x7f388c10d220 2024-03-14T09:51:40.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.934+0000 7f3889e00700 1 --2- 172.21.15.78:0/1446596244 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f38700a0a90 0x7f38700a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:40.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.935+0000 7f3889e00700 1 --2- 172.21.15.78:0/1446596244 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f38700a0a90 0x7f38700a2f50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f387c006fd0 tx=0x7f387c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:40.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:40.938+0000 7f3882a00700 1 -- 172.21.15.78:0/1446596244 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3884096fc0 con 0x7f388c10d220 2024-03-14T09:51:40.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:40 smithi152 ceph-mon[48279]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:40.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:40 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:41.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:40 smithi132 ceph-mon[47959]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:41.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:40 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:41.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:41.227+0000 7f3890a00700 1 -- 172.21.15.78:0/1446596244 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f388c0679e0 con 0x7f388c10d220 2024-03-14T09:51:41.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:41.227+0000 7f3882a00700 1 -- 172.21.15.78:0/1446596244 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2771 (secure 0 0 0) 0x7f388409a5e0 con 0x7f388c10d220 2024-03-14T09:51:41.228 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:51:41.228 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-03-14T09:51:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:41.230+0000 7f3890a00700 1 -- 172.21.15.78:0/1446596244 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f38700a0a90 msgr2=0x7f38700a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:41.230+0000 7f3890a00700 1 --2- 172.21.15.78:0/1446596244 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f38700a0a90 0x7f38700a2f50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f387c006fd0 tx=0x7f387c009190 comp rx=0 tx=0).stop 2024-03-14T09:51:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:41.230+0000 7f3890a00700 1 -- 172.21.15.78:0/1446596244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c10d220 msgr2=0x7f388c1a4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:41.230+0000 7f3890a00700 1 --2- 172.21.15.78:0/1446596244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c10d220 0x7f388c1a4c60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3884007d00 tx=0x7f3884005ad0 comp rx=0 tx=0).stop 2024-03-14T09:51:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:41.230+0000 7f3890a00700 1 -- 172.21.15.78:0/1446596244 shutdown_connections 2024-03-14T09:51:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:41.230+0000 7f3890a00700 1 --2- 172.21.15.78:0/1446596244 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f38700a0a90 0x7f38700a2f50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:41.230+0000 7f3890a00700 1 --2- 172.21.15.78:0/1446596244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f388c073810 0x7f388c19d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:41.230+0000 7f3890a00700 1 --2- 172.21.15.78:0/1446596244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f388c10d220 0x7f388c1a4c60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:41.230+0000 7f3890a00700 1 --2- 172.21.15.78:0/1446596244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f388c074df0 0x7f388c19dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:41.230+0000 7f3890a00700 1 -- 172.21.15.78:0/1446596244 >> 172.21.15.78:0/1446596244 conn(0x7f388c0fc1c0 msgr2=0x7f388c1005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:41.230+0000 7f3890a00700 1 -- 172.21.15.78:0/1446596244 shutdown_connections 2024-03-14T09:51:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:41.230+0000 7f3890a00700 1 -- 172.21.15.78:0/1446596244 wait complete. 2024-03-14T09:51:41.232 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 4 2024-03-14T09:51:41.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:41.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:41.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:41.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:41.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:41.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:41 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.78:6834/996011838,v1:172.21.15.78:6835/996011838] up:boot 2024-03-14T09:51:41.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:41 smithi152 ceph-mon[48279]: fsmap 3 up:standby 2024-03-14T09:51:41.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-14T09:51:42.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:41 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1446596244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:42.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:42.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:42.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:42.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:42.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:42.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:41 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.78:6834/996011838,v1:172.21.15.78:6835/996011838] up:boot 2024-03-14T09:51:42.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:41 smithi132 ceph-mon[47959]: fsmap 3 up:standby 2024-03-14T09:51:42.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-14T09:51:42.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:41 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1446596244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:42.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:42.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:42.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:42.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:42.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:42.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:41 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.78:6834/996011838,v1:172.21.15.78:6835/996011838] up:boot 2024-03-14T09:51:42.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:41 smithi078 ceph-mon[38143]: fsmap 3 up:standby 2024-03-14T09:51:42.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-14T09:51:42.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:41 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1446596244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:42.553 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 14 09:51:42 smithi078 systemd[1]: Starting Ceph mds.d for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T09:51:42.887 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 14 09:51:42 smithi078 bash[87074]: b53b9544ba2b370ebca231d0a44ae0dc8d640749fe0d05acdec6c00df6cffeb8 2024-03-14T09:51:42.887 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 14 09:51:42 smithi078 systemd[1]: Started Ceph mds.d for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:51:42.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:42 smithi078 ceph-mon[38143]: Deploying daemon mds.d on smithi078 2024-03-14T09:51:42.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:42 smithi078 ceph-mon[38143]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:42.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:42 smithi152 ceph-mon[48279]: Deploying daemon mds.d on smithi078 2024-03-14T09:51:42.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:42 smithi152 ceph-mon[48279]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:43.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:42 smithi132 ceph-mon[47959]: Deploying daemon mds.d on smithi078 2024-03-14T09:51:43.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:42 smithi132 ceph-mon[47959]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:43.252 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:51:43.303 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 14 09:51:42 smithi078 ceph-mds[87112]: -- 172.21.15.78:0/2196427061 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556dace541a0 con 0x556dace2d000 2024-03-14T09:51:43.303 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 14 09:51:42 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-d[87089]: starting mds.d at 2024-03-14T09:51:43.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.448+0000 7f24ae800700 1 -- 172.21.15.78:0/286074063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24a8100c50 msgr2=0x7f24a8101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:43.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.448+0000 7f24ae800700 1 --2- 172.21.15.78:0/286074063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24a8100c50 0x7f24a8101070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f24980099c0 tx=0x7f2498009cd0 comp rx=0 tx=0).stop 2024-03-14T09:51:43.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.448+0000 7f24ae800700 1 -- 172.21.15.78:0/286074063 shutdown_connections 2024-03-14T09:51:43.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.448+0000 7f24ae800700 1 --2- 172.21.15.78:0/286074063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f24a81027e0 0x7f24a8109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:43.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.448+0000 7f24ae800700 1 --2- 172.21.15.78:0/286074063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24a8100c50 0x7f24a8101070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:43.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.448+0000 7f24ae800700 1 --2- 172.21.15.78:0/286074063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24a8101e40 0x7f24a81022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:43.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.448+0000 7f24ae800700 1 -- 172.21.15.78:0/286074063 >> 172.21.15.78:0/286074063 conn(0x7f24a80fc1c0 msgr2=0x7f24a80fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:43.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.448+0000 7f24ae800700 1 -- 172.21.15.78:0/286074063 shutdown_connections 2024-03-14T09:51:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.448+0000 7f24ae800700 1 -- 172.21.15.78:0/286074063 wait complete. 2024-03-14T09:51:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.449+0000 7f24ae800700 1 Processor -- start 2024-03-14T09:51:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.449+0000 7f24ae800700 1 -- start start 2024-03-14T09:51:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.450+0000 7f24ae800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f24a8100c50 0x7f24a81991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.450+0000 7f24ae800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24a8101e40 0x7f24a8199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.450+0000 7f24ae800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24a81027e0 0x7f24a81a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.450+0000 7f24ae800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a81a0d20 con 0x7f24a81027e0 2024-03-14T09:51:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.450+0000 7f24ae800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a81a0e60 con 0x7f24a8101e40 2024-03-14T09:51:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.450+0000 7f24ae800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a81a0ff0 con 0x7f24a8100c50 2024-03-14T09:51:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.450+0000 7f24a7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24a81027e0 0x7f24a81a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.450+0000 7f24a7400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f24a8100c50 0x7f24a81991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.450+0000 7f24a7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24a81027e0 0x7f24a81a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42212/0 (socket says 172.21.15.78:42212) 2024-03-14T09:51:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.450+0000 7f24a7e00700 1 -- 172.21.15.78:0/1550242778 learned_addr learned my addr 172.21.15.78:0/1550242778 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:51:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.450+0000 7f249fe00700 1 --2- 172.21.15.78:0/1550242778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24a8101e40 0x7f24a8199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.450+0000 7f24a7e00700 1 -- 172.21.15.78:0/1550242778 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f24a8100c50 msgr2=0x7f24a81991f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.450+0000 7f24a7e00700 1 --2- 172.21.15.78:0/1550242778 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f24a8100c50 0x7f24a81991f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.450+0000 7f24a7e00700 1 -- 172.21.15.78:0/1550242778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24a8101e40 msgr2=0x7f24a8199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.450+0000 7f24a7e00700 1 --2- 172.21.15.78:0/1550242778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24a8101e40 0x7f24a8199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.451+0000 7f24a7e00700 1 -- 172.21.15.78:0/1550242778 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24980095f0 con 0x7f24a81027e0 2024-03-14T09:51:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.451+0000 7f24a7400700 1 --2- 172.21.15.78:0/1550242778 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f24a8100c50 0x7f24a81991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:51:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.451+0000 7f249fe00700 1 --2- 172.21.15.78:0/1550242778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24a8101e40 0x7f24a8199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:51:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.451+0000 7f24a7e00700 1 --2- 172.21.15.78:0/1550242778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24a81027e0 0x7f24a81a0750 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f24a000ef30 tx=0x7f24a000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.451+0000 7f24a4c00700 1 -- 172.21.15.78:0/1550242778 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a0018070 con 0x7f24a81027e0 2024-03-14T09:51:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.451+0000 7f24ae800700 1 -- 172.21.15.78:0/1550242778 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a81a1270 con 0x7f24a81027e0 2024-03-14T09:51:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.451+0000 7f24a4c00700 1 -- 172.21.15.78:0/1550242778 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24a000f040 con 0x7f24a81027e0 2024-03-14T09:51:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.451+0000 7f24a4c00700 1 -- 172.21.15.78:0/1550242778 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a000cd00 con 0x7f24a81027e0 2024-03-14T09:51:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.451+0000 7f24ae800700 1 -- 172.21.15.78:0/1550242778 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a81a1780 con 0x7f24a81027e0 2024-03-14T09:51:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.452+0000 7f24ae800700 1 -- 172.21.15.78:0/1550242778 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a80679e0 con 0x7f24a81027e0 2024-03-14T09:51:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.454+0000 7f24a4c00700 1 -- 172.21.15.78:0/1550242778 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f24a0003a00 con 0x7f24a81027e0 2024-03-14T09:51:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.455+0000 7f24a4c00700 1 --2- 172.21.15.78:0/1550242778 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f248c0a0a90 0x7f248c0a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.455+0000 7f24a4c00700 1 -- 172.21.15.78:0/1550242778 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f24a00cc0f0 con 0x7f24a81027e0 2024-03-14T09:51:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.455+0000 7f24a7400700 1 --2- 172.21.15.78:0/1550242778 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f248c0a0a90 0x7f248c0a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.456+0000 7f24a7400700 1 --2- 172.21.15.78:0/1550242778 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f248c0a0a90 0x7f248c0a2f50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2498009990 tx=0x7f249801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.458+0000 7f24a4c00700 1 -- 172.21.15.78:0/1550242778 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f24a0095f50 con 0x7f24a81027e0 2024-03-14T09:51:43.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.709+0000 7f24ae800700 1 -- 172.21.15.78:0/1550242778 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24a810c8d0 con 0x7f24a81027e0 2024-03-14T09:51:43.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.709+0000 7f24a4c00700 1 -- 172.21.15.78:0/1550242778 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3525 (secure 0 0 0) 0x7f24a00b9020 con 0x7f24a81027e0 2024-03-14T09:51:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:51:43.710 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-03-14T09:51:43.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.711+0000 7f24ae800700 1 -- 172.21.15.78:0/1550242778 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f248c0a0a90 msgr2=0x7f248c0a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:43.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.711+0000 7f24ae800700 1 --2- 172.21.15.78:0/1550242778 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f248c0a0a90 0x7f248c0a2f50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2498009990 tx=0x7f249801a040 comp rx=0 tx=0).stop 2024-03-14T09:51:43.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.711+0000 7f24ae800700 1 -- 172.21.15.78:0/1550242778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24a81027e0 msgr2=0x7f24a81a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:43.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.711+0000 7f24ae800700 1 --2- 172.21.15.78:0/1550242778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24a81027e0 0x7f24a81a0750 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f24a000ef30 tx=0x7f24a000c3a0 comp rx=0 tx=0).stop 2024-03-14T09:51:43.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.712+0000 7f24ae800700 1 -- 172.21.15.78:0/1550242778 shutdown_connections 2024-03-14T09:51:43.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.712+0000 7f24ae800700 1 --2- 172.21.15.78:0/1550242778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24a81027e0 0x7f24a81a0750 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:43.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.712+0000 7f24ae800700 1 --2- 172.21.15.78:0/1550242778 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f24a8100c50 0x7f24a81991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:43.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.712+0000 7f24ae800700 1 --2- 172.21.15.78:0/1550242778 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f248c0a0a90 0x7f248c0a2f50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:43.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.712+0000 7f24ae800700 1 --2- 172.21.15.78:0/1550242778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24a8101e40 0x7f24a8199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:43.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.712+0000 7f24ae800700 1 -- 172.21.15.78:0/1550242778 >> 172.21.15.78:0/1550242778 conn(0x7f24a80fc1c0 msgr2=0x7f24a8107610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:43.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.712+0000 7f24ae800700 1 -- 172.21.15.78:0/1550242778 shutdown_connections 2024-03-14T09:51:43.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:43.712+0000 7f24ae800700 1 -- 172.21.15.78:0/1550242778 wait complete. 2024-03-14T09:51:43.713 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 5 2024-03-14T09:51:44.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:44.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:44.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:44.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:44.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:44.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:44.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:43 smithi078 ceph-mon[38143]: Deploying daemon mds.e on smithi132 2024-03-14T09:51:44.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:43 smithi078 ceph-mon[38143]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:44.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:43 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.78:6836/176895986,v1:172.21.15.78:6837/176895986] up:boot 2024-03-14T09:51:44.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:43 smithi078 ceph-mon[38143]: fsmap 4 up:standby 2024-03-14T09:51:44.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:43 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-14T09:51:44.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:43 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1550242778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:44.237 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:44.237 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:44.237 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:44.237 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:44.237 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:44.237 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:44.238 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:43 smithi132 ceph-mon[47959]: Deploying daemon mds.e on smithi132 2024-03-14T09:51:44.238 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:43 smithi132 ceph-mon[47959]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:44.238 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:43 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.78:6836/176895986,v1:172.21.15.78:6837/176895986] up:boot 2024-03-14T09:51:44.238 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:43 smithi132 ceph-mon[47959]: fsmap 4 up:standby 2024-03-14T09:51:44.238 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:43 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-14T09:51:44.238 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:43 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1550242778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:44.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:44.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:44.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:44.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:44.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:44.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:44.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:43 smithi152 ceph-mon[48279]: Deploying daemon mds.e on smithi132 2024-03-14T09:51:44.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:43 smithi152 ceph-mon[48279]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:44.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:43 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.78:6836/176895986,v1:172.21.15.78:6837/176895986] up:boot 2024-03-14T09:51:44.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:43 smithi152 ceph-mon[48279]: fsmap 4 up:standby 2024-03-14T09:51:44.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:43 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-14T09:51:44.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:43 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1550242778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:45.020 INFO:journalctl@ceph.mds.e.smithi132.stdout:Mar 14 09:51:44 smithi132 bash[85325]: df6e50ca2a6d69b3502eac6cb0b1a168510e410576a0b6639fd6ce4a9647192e 2024-03-14T09:51:45.020 INFO:journalctl@ceph.mds.e.smithi132.stdout:Mar 14 09:51:44 smithi132 systemd[1]: Started Ceph mds.e for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:51:45.732 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:51:45.865 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:45 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:45.865 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:45 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:45.865 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:45 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:45.865 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:45 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:45.865 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:45 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:45.865 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:45 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:45.865 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:45 smithi152 ceph-mon[48279]: Deploying daemon mds.f on smithi152 2024-03-14T09:51:45.866 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:45 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.132:6834/1709886688,v1:172.21.15.132:6835/1709886688] up:boot 2024-03-14T09:51:45.866 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:45 smithi152 ceph-mon[48279]: fsmap 5 up:standby 2024-03-14T09:51:45.866 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:45 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-14T09:51:45.866 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:45 smithi152 ceph-mon[48279]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:45.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.951+0000 7f99f9000700 1 -- 172.21.15.78:0/2598247707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99f4101e40 msgr2=0x7f99f41022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:45.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.951+0000 7f99f9000700 1 --2- 172.21.15.78:0/2598247707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99f4101e40 0x7f99f41022a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f99e00099c0 tx=0x7f99e0009cd0 comp rx=0 tx=0).stop 2024-03-14T09:51:45.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.951+0000 7f99f9000700 1 -- 172.21.15.78:0/2598247707 shutdown_connections 2024-03-14T09:51:45.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.951+0000 7f99f9000700 1 --2- 172.21.15.78:0/2598247707 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99f4100c50 0x7f99f4101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:45.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.951+0000 7f99f9000700 1 --2- 172.21.15.78:0/2598247707 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99f41027e0 0x7f99f4109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:45.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.951+0000 7f99f9000700 1 --2- 172.21.15.78:0/2598247707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99f4101e40 0x7f99f41022a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:45.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.951+0000 7f99f9000700 1 -- 172.21.15.78:0/2598247707 >> 172.21.15.78:0/2598247707 conn(0x7f99f40fc1c0 msgr2=0x7f99f40fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:45.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.951+0000 7f99f9000700 1 -- 172.21.15.78:0/2598247707 shutdown_connections 2024-03-14T09:51:45.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.951+0000 7f99f9000700 1 -- 172.21.15.78:0/2598247707 wait complete. 2024-03-14T09:51:45.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.952+0000 7f99f9000700 1 Processor -- start 2024-03-14T09:51:45.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.953+0000 7f99f9000700 1 -- start start 2024-03-14T09:51:45.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.953+0000 7f99f9000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99f4100c50 0x7f99f4074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:45.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.953+0000 7f99f9000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99f4101e40 0x7f99f40731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:45.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.953+0000 7f99f9000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99f41027e0 0x7f99f40736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:45.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.953+0000 7f99f9000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f4073e80 con 0x7f99f4101e40 2024-03-14T09:51:45.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.953+0000 7f99f9000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f4073ff0 con 0x7f99f4100c50 2024-03-14T09:51:45.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.953+0000 7f99f9000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f41a0fd0 con 0x7f99f41027e0 2024-03-14T09:51:45.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.953+0000 7f99f1400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99f4101e40 0x7f99f40731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:45.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.953+0000 7f99f1400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99f4101e40 0x7f99f40731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39566/0 (socket says 172.21.15.78:39566) 2024-03-14T09:51:45.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.953+0000 7f99f1400700 1 -- 172.21.15.78:0/3223381620 learned_addr learned my addr 172.21.15.78:0/3223381620 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:51:45.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.953+0000 7f99f2800700 1 --2- 172.21.15.78:0/3223381620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99f41027e0 0x7f99f40736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:45.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.953+0000 7f99f1400700 1 -- 172.21.15.78:0/3223381620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99f41027e0 msgr2=0x7f99f40736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:45.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.953+0000 7f99f1400700 1 --2- 172.21.15.78:0/3223381620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99f41027e0 0x7f99f40736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:45.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.953+0000 7f99f1400700 1 -- 172.21.15.78:0/3223381620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99f4100c50 msgr2=0x7f99f4074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:45.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.953+0000 7f99f1e00700 1 --2- 172.21.15.78:0/3223381620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99f4100c50 0x7f99f4074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:45.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.954+0000 7f99f1400700 1 --2- 172.21.15.78:0/3223381620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99f4100c50 0x7f99f4074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:45.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.954+0000 7f99f1400700 1 -- 172.21.15.78:0/3223381620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e00095f0 con 0x7f99f4101e40 2024-03-14T09:51:45.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.954+0000 7f99f1e00700 1 --2- 172.21.15.78:0/3223381620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99f4100c50 0x7f99f4074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:51:45.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.954+0000 7f99f1400700 1 --2- 172.21.15.78:0/3223381620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99f4101e40 0x7f99f40731a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f99e0003730 tx=0x7f99e000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:45.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.954+0000 7f99eaa00700 1 -- 172.21.15.78:0/3223381620 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e001d070 con 0x7f99f4101e40 2024-03-14T09:51:45.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.954+0000 7f99eaa00700 1 -- 172.21.15.78:0/3223381620 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99e000fe70 con 0x7f99f4101e40 2024-03-14T09:51:45.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.954+0000 7f99f9000700 1 -- 172.21.15.78:0/3223381620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f41a1200 con 0x7f99f4101e40 2024-03-14T09:51:45.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.954+0000 7f99eaa00700 1 -- 172.21.15.78:0/3223381620 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0017650 con 0x7f99f4101e40 2024-03-14T09:51:45.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.954+0000 7f99f9000700 1 -- 172.21.15.78:0/3223381620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f41a1620 con 0x7f99f4101e40 2024-03-14T09:51:45.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.955+0000 7f99f9000700 1 -- 172.21.15.78:0/3223381620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f4108000 con 0x7f99f4101e40 2024-03-14T09:51:45.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.957+0000 7f99eaa00700 1 -- 172.21.15.78:0/3223381620 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f99e0022470 con 0x7f99f4101e40 2024-03-14T09:51:45.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.959+0000 7f99eaa00700 1 --2- 172.21.15.78:0/3223381620 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99d80a0b10 0x7f99d80a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:45.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.959+0000 7f99eaa00700 1 -- 172.21.15.78:0/3223381620 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f99e00cced0 con 0x7f99f4101e40 2024-03-14T09:51:45.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.960+0000 7f99f1e00700 1 --2- 172.21.15.78:0/3223381620 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99d80a0b10 0x7f99d80a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:45.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.960+0000 7f99f1e00700 1 --2- 172.21.15.78:0/3223381620 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99d80a0b10 0x7f99d80a2fd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f99e4005fd0 tx=0x7f99e4005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:45.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:45.963+0000 7f99eaa00700 1 -- 172.21.15.78:0/3223381620 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f99e0096dd0 con 0x7f99f4101e40 2024-03-14T09:51:46.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:45 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:46.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:45 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:46.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:45 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:46.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:45 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:46.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:45 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:46.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:45 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:46.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:45 smithi132 ceph-mon[47959]: Deploying daemon mds.f on smithi152 2024-03-14T09:51:46.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:45 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.132:6834/1709886688,v1:172.21.15.132:6835/1709886688] up:boot 2024-03-14T09:51:46.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:45 smithi132 ceph-mon[47959]: fsmap 5 up:standby 2024-03-14T09:51:46.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:45 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-14T09:51:46.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:45 smithi132 ceph-mon[47959]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:46.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:45 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:46.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:45 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:46.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:45 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:46.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:45 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:46.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:45 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:46.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:45 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:46.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:45 smithi078 ceph-mon[38143]: Deploying daemon mds.f on smithi152 2024-03-14T09:51:46.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:45 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.132:6834/1709886688,v1:172.21.15.132:6835/1709886688] up:boot 2024-03-14T09:51:46.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:45 smithi078 ceph-mon[38143]: fsmap 5 up:standby 2024-03-14T09:51:46.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:45 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-14T09:51:46.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:45 smithi078 ceph-mon[38143]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:46.115 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 14 09:51:46 smithi152 systemd[1]: Starting Ceph mds.f for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T09:51:46.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:46.215+0000 7f99f9000700 1 -- 172.21.15.78:0/3223381620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99f410c9e0 con 0x7f99f4101e40 2024-03-14T09:51:46.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:46.216+0000 7f99eaa00700 1 -- 172.21.15.78:0/3223381620 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4285 (secure 0 0 0) 0x7f99e009a3f0 con 0x7f99f4101e40 2024-03-14T09:51:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:51:46.217 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2024-03-14T09:51:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:46.218+0000 7f99f9000700 1 -- 172.21.15.78:0/3223381620 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99d80a0b10 msgr2=0x7f99d80a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:46.218+0000 7f99f9000700 1 --2- 172.21.15.78:0/3223381620 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99d80a0b10 0x7f99d80a2fd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f99e4005fd0 tx=0x7f99e4005e40 comp rx=0 tx=0).stop 2024-03-14T09:51:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:46.218+0000 7f99f9000700 1 -- 172.21.15.78:0/3223381620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99f4101e40 msgr2=0x7f99f40731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:46.218+0000 7f99f9000700 1 --2- 172.21.15.78:0/3223381620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99f4101e40 0x7f99f40731a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f99e0003730 tx=0x7f99e000fa00 comp rx=0 tx=0).stop 2024-03-14T09:51:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:46.218+0000 7f99f9000700 1 -- 172.21.15.78:0/3223381620 shutdown_connections 2024-03-14T09:51:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:46.218+0000 7f99f9000700 1 --2- 172.21.15.78:0/3223381620 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99d80a0b10 0x7f99d80a2fd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:46.218+0000 7f99f9000700 1 --2- 172.21.15.78:0/3223381620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99f4100c50 0x7f99f4074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:46.218+0000 7f99f9000700 1 --2- 172.21.15.78:0/3223381620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99f41027e0 0x7f99f40736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:46.218+0000 7f99f9000700 1 --2- 172.21.15.78:0/3223381620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99f4101e40 0x7f99f40731a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:46.218+0000 7f99f9000700 1 -- 172.21.15.78:0/3223381620 >> 172.21.15.78:0/3223381620 conn(0x7f99f40fc1c0 msgr2=0x7f99f4105140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:46.219+0000 7f99f9000700 1 -- 172.21.15.78:0/3223381620 shutdown_connections 2024-03-14T09:51:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:46.219+0000 7f99f9000700 1 -- 172.21.15.78:0/3223381620 wait complete. 2024-03-14T09:51:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 6 2024-03-14T09:51:46.992 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 14 09:51:46 smithi152 ceph-mds[85139]: -- 172.21.15.152:0/15349680 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d3420dc1a0 con 0x55d3420b3000 2024-03-14T09:51:46.992 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 14 09:51:46 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-f[85116]: starting mds.f at 2024-03-14T09:51:46.992 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 14 09:51:46 smithi152 bash[85101]: bad61adda90bc2b48f4a47e7b70622160919e435de985c97d9ebe9c651c3b02b 2024-03-14T09:51:46.992 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 14 09:51:46 smithi152 systemd[1]: Started Ceph mds.f for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:51:47.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:47 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3223381620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:47.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:47.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:47.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:47.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:47.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:47.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:47 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:47.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:47 smithi152 ceph-mon[48279]: Deploying daemon mds.c on smithi152 2024-03-14T09:51:47.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:47 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3223381620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:47.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:47.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:47.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:47.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:47.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:47.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:47 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:47.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:47 smithi132 ceph-mon[47959]: Deploying daemon mds.c on smithi152 2024-03-14T09:51:47.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:47 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3223381620' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:47.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:47.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:47.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:47.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:47.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:47.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:47 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:47.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:47 smithi078 ceph-mon[38143]: Deploying daemon mds.c on smithi152 2024-03-14T09:51:48.239 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:51:48.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:48 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.152:6834/2986677339,v1:172.21.15.152:6835/2986677339] up:boot 2024-03-14T09:51:48.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:48 smithi152 ceph-mon[48279]: fsmap 6 up:standby 2024-03-14T09:51:48.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:48 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-14T09:51:48.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:48 smithi152 ceph-mon[48279]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:48.249 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 14 09:51:47 smithi152 systemd[1]: Starting Ceph mds.c for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T09:51:48.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:48 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.152:6834/2986677339,v1:172.21.15.152:6835/2986677339] up:boot 2024-03-14T09:51:48.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:48 smithi132 ceph-mon[47959]: fsmap 6 up:standby 2024-03-14T09:51:48.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:48 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-14T09:51:48.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:48 smithi132 ceph-mon[47959]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:48.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:48 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.152:6834/2986677339,v1:172.21.15.152:6835/2986677339] up:boot 2024-03-14T09:51:48.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:48 smithi078 ceph-mon[38143]: fsmap 6 up:standby 2024-03-14T09:51:48.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:48 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-14T09:51:48.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:48 smithi078 ceph-mon[38143]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:48.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.440+0000 7f5145200700 1 -- 172.21.15.78:0/1639373700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5140101e20 msgr2=0x7f5140102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:48.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.440+0000 7f5145200700 1 --2- 172.21.15.78:0/1639373700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5140101e20 0x7f5140102280 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f512c0099c0 tx=0x7f512c009cd0 comp rx=0 tx=0).stop 2024-03-14T09:51:48.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.440+0000 7f5145200700 1 -- 172.21.15.78:0/1639373700 shutdown_connections 2024-03-14T09:51:48.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.440+0000 7f5145200700 1 --2- 172.21.15.78:0/1639373700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51401027c0 0x7f5140109070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:48.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.440+0000 7f5145200700 1 --2- 172.21.15.78:0/1639373700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5140100c30 0x7f5140101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:48.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.440+0000 7f5145200700 1 --2- 172.21.15.78:0/1639373700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5140101e20 0x7f5140102280 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:48.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.440+0000 7f5145200700 1 -- 172.21.15.78:0/1639373700 >> 172.21.15.78:0/1639373700 conn(0x7f51400fc1c0 msgr2=0x7f51400fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:48.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.440+0000 7f5145200700 1 -- 172.21.15.78:0/1639373700 shutdown_connections 2024-03-14T09:51:48.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.440+0000 7f5145200700 1 -- 172.21.15.78:0/1639373700 wait complete. 2024-03-14T09:51:48.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.441+0000 7f5145200700 1 Processor -- start 2024-03-14T09:51:48.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.442+0000 7f5145200700 1 -- start start 2024-03-14T09:51:48.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.442+0000 7f5145200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5140100c30 0x7f5140199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:48.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.442+0000 7f5145200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5140101e20 0x7f5140199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:48.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.442+0000 7f5145200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51401027c0 0x7f51401a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:48.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.442+0000 7f5145200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51401a0d50 con 0x7f5140100c30 2024-03-14T09:51:48.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.442+0000 7f5145200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51401a0e90 con 0x7f51401027c0 2024-03-14T09:51:48.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.442+0000 7f5145200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51401a1020 con 0x7f5140101e20 2024-03-14T09:51:48.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.442+0000 7f513f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5140100c30 0x7f5140199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:48.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.442+0000 7f513ea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5140101e20 0x7f5140199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:48.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.442+0000 7f513fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51401027c0 0x7f51401a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:48.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.442+0000 7f513f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5140100c30 0x7f5140199220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39590/0 (socket says 172.21.15.78:39590) 2024-03-14T09:51:48.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.442+0000 7f513f400700 1 -- 172.21.15.78:0/2012282607 learned_addr learned my addr 172.21.15.78:0/2012282607 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:51:48.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.443+0000 7f513ea00700 1 -- 172.21.15.78:0/2012282607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51401027c0 msgr2=0x7f51401a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:48.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.443+0000 7f513ea00700 1 --2- 172.21.15.78:0/2012282607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51401027c0 0x7f51401a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:48.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.443+0000 7f513ea00700 1 -- 172.21.15.78:0/2012282607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5140100c30 msgr2=0x7f5140199220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:48.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.443+0000 7f513ea00700 1 --2- 172.21.15.78:0/2012282607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5140100c30 0x7f5140199220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:48.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.443+0000 7f513ea00700 1 -- 172.21.15.78:0/2012282607 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512c0095f0 con 0x7f5140101e20 2024-03-14T09:51:48.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.443+0000 7f513f400700 1 --2- 172.21.15.78:0/2012282607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5140100c30 0x7f5140199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:51:48.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.443+0000 7f513ea00700 1 --2- 172.21.15.78:0/2012282607 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5140101e20 0x7f5140199760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f512c009fd0 tx=0x7f512c00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:48.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.443+0000 7f5137e00700 1 -- 172.21.15.78:0/2012282607 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c01d070 con 0x7f5140101e20 2024-03-14T09:51:48.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.443+0000 7f5145200700 1 -- 172.21.15.78:0/2012282607 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51401a12a0 con 0x7f5140101e20 2024-03-14T09:51:48.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.443+0000 7f5137e00700 1 -- 172.21.15.78:0/2012282607 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f512c005050 con 0x7f5140101e20 2024-03-14T09:51:48.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.443+0000 7f5137e00700 1 -- 172.21.15.78:0/2012282607 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c004a70 con 0x7f5140101e20 2024-03-14T09:51:48.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.443+0000 7f5145200700 1 -- 172.21.15.78:0/2012282607 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51401a16d0 con 0x7f5140101e20 2024-03-14T09:51:48.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.444+0000 7f5145200700 1 -- 172.21.15.78:0/2012282607 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5140067970 con 0x7f5140101e20 2024-03-14T09:51:48.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.446+0000 7f5137e00700 1 -- 172.21.15.78:0/2012282607 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f512c005430 con 0x7f5140101e20 2024-03-14T09:51:48.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.447+0000 7f5137e00700 1 --2- 172.21.15.78:0/2012282607 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f51240a0a90 0x7f51240a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:48.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.448+0000 7f5137e00700 1 -- 172.21.15.78:0/2012282607 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f512c0cca80 con 0x7f5140101e20 2024-03-14T09:51:48.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.448+0000 7f513f400700 1 --2- 172.21.15.78:0/2012282607 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f51240a0a90 0x7f51240a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:48.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.448+0000 7f513f400700 1 --2- 172.21.15.78:0/2012282607 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f51240a0a90 0x7f51240a2f50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5130005950 tx=0x7f5130009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:48.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.450+0000 7f5137e00700 1 -- 172.21.15.78:0/2012282607 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f512c0968e0 con 0x7f5140101e20 2024-03-14T09:51:48.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.721+0000 7f5145200700 1 -- 172.21.15.78:0/2012282607 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f514010c8b0 con 0x7f5140101e20 2024-03-14T09:51:48.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.722+0000 7f5137e00700 1 -- 172.21.15.78:0/2012282607 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5045 (secure 0 0 0) 0x7f512c099f00 con 0x7f5140101e20 2024-03-14T09:51:48.723 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:51:48.723 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2024-03-14T09:51:48.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.724+0000 7f5145200700 1 -- 172.21.15.78:0/2012282607 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f51240a0a90 msgr2=0x7f51240a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:48.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.724+0000 7f5145200700 1 --2- 172.21.15.78:0/2012282607 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f51240a0a90 0x7f51240a2f50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5130005950 tx=0x7f5130009260 comp rx=0 tx=0).stop 2024-03-14T09:51:48.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.724+0000 7f5145200700 1 -- 172.21.15.78:0/2012282607 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5140101e20 msgr2=0x7f5140199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:48.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.724+0000 7f5145200700 1 --2- 172.21.15.78:0/2012282607 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5140101e20 0x7f5140199760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f512c009fd0 tx=0x7f512c00f690 comp rx=0 tx=0).stop 2024-03-14T09:51:48.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.725+0000 7f5145200700 1 -- 172.21.15.78:0/2012282607 shutdown_connections 2024-03-14T09:51:48.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.725+0000 7f5145200700 1 --2- 172.21.15.78:0/2012282607 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f51240a0a90 0x7f51240a2f50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:48.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.725+0000 7f5145200700 1 --2- 172.21.15.78:0/2012282607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51401027c0 0x7f51401a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:48.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.725+0000 7f5145200700 1 --2- 172.21.15.78:0/2012282607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5140100c30 0x7f5140199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:48.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.725+0000 7f5145200700 1 --2- 172.21.15.78:0/2012282607 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5140101e20 0x7f5140199760 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:48.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.725+0000 7f5145200700 1 -- 172.21.15.78:0/2012282607 >> 172.21.15.78:0/2012282607 conn(0x7f51400fc1c0 msgr2=0x7f5140072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:48.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.725+0000 7f5145200700 1 -- 172.21.15.78:0/2012282607 shutdown_connections 2024-03-14T09:51:48.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:48.725+0000 7f5145200700 1 -- 172.21.15.78:0/2012282607 wait complete. 2024-03-14T09:51:48.727 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 7 2024-03-14T09:51:48.748 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 14 09:51:48 smithi152 ceph-mds[85481]: -- 172.21.15.152:0/3985928280 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556c4821e1a0 con 0x556c481f4c00 2024-03-14T09:51:48.749 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 14 09:51:48 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-c[85458]: starting mds.c at 2024-03-14T09:51:48.749 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 14 09:51:48 smithi152 bash[85443]: 7c11fe5f60163d3d67c0d8899377a18a35a31aab83cedc0a05efc8b4914d37c0 2024-03-14T09:51:48.749 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 14 09:51:48 smithi152 systemd[1]: Started Ceph mds.c for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:51:49.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:49 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:49.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:49 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:49.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:49 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:49.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:49 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:49.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:49 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:49.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:49 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:49.750 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:49 smithi152 ceph-mon[48279]: Deploying daemon mds.h on smithi132 2024-03-14T09:51:49.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:49 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2012282607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:49.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:49 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.152:6836/2123600730,v1:172.21.15.152:6837/2123600730] up:boot 2024-03-14T09:51:49.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:49 smithi152 ceph-mon[48279]: fsmap 7 up:standby 2024-03-14T09:51:49.751 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:49 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-14T09:51:49.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:49 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:49.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:49 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:49.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:49 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:49.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:49 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:49.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:49 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:49.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:49 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:49.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:49 smithi132 ceph-mon[47959]: Deploying daemon mds.h on smithi132 2024-03-14T09:51:49.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:49 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2012282607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:49.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:49 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.152:6836/2123600730,v1:172.21.15.152:6837/2123600730] up:boot 2024-03-14T09:51:49.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:49 smithi132 ceph-mon[47959]: fsmap 7 up:standby 2024-03-14T09:51:49.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:49 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-14T09:51:49.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:49 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:49.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:49 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:49.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:49 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:49.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:49 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:49.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:49 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:49.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:49 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:49.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:49 smithi078 ceph-mon[38143]: Deploying daemon mds.h on smithi132 2024-03-14T09:51:49.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:49 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2012282607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:49.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:49 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.152:6836/2123600730,v1:172.21.15.152:6837/2123600730] up:boot 2024-03-14T09:51:49.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:49 smithi078 ceph-mon[38143]: fsmap 7 up:standby 2024-03-14T09:51:49.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:49 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-14T09:51:50.492 INFO:journalctl@ceph.mds.h.smithi132.stdout:Mar 14 09:51:50 smithi132 bash[85669]: ecd951122b1d757e468784c8e25d14a02f2c888f19bd6015ee1b4ce1f16dfd14 2024-03-14T09:51:50.492 INFO:journalctl@ceph.mds.h.smithi132.stdout:Mar 14 09:51:50 smithi132 systemd[1]: Started Ceph mds.h for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:51:50.746 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:51:50.748 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:50 smithi152 ceph-mon[48279]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:50.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:50 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:50.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:50 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:50.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:50 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:50.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:50 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:50.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:50 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:50.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:50 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:50.751 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:50 smithi132 ceph-mon[47959]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:50.751 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:50 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:50.751 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:50 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:50.752 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:50 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:50.752 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:50 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:50.752 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:50 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:50.752 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:50 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:50.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:50 smithi078 ceph-mon[38143]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:50.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:50 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:50.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:50 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:50.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:50 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:50.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:50 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:50.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:50 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:50.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:50 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:50.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.947+0000 7f4aa1a00700 1 -- 172.21.15.78:0/1467636393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a9c101e40 msgr2=0x7f4a9c1022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:50.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.947+0000 7f4aa1a00700 1 --2- 172.21.15.78:0/1467636393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a9c101e40 0x7f4a9c1022a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4a880099c0 tx=0x7f4a88009cd0 comp rx=0 tx=0).stop 2024-03-14T09:51:50.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.947+0000 7f4aa1a00700 1 -- 172.21.15.78:0/1467636393 shutdown_connections 2024-03-14T09:51:50.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.947+0000 7f4aa1a00700 1 --2- 172.21.15.78:0/1467636393 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c1027e0 0x7f4a9c109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:50.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.947+0000 7f4aa1a00700 1 --2- 172.21.15.78:0/1467636393 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a9c100c50 0x7f4a9c101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:50.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.947+0000 7f4aa1a00700 1 --2- 172.21.15.78:0/1467636393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a9c101e40 0x7f4a9c1022a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:50.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.947+0000 7f4aa1a00700 1 -- 172.21.15.78:0/1467636393 >> 172.21.15.78:0/1467636393 conn(0x7f4a9c0fc1c0 msgr2=0x7f4a9c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:50.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.947+0000 7f4aa1a00700 1 -- 172.21.15.78:0/1467636393 shutdown_connections 2024-03-14T09:51:50.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.947+0000 7f4aa1a00700 1 -- 172.21.15.78:0/1467636393 wait complete. 2024-03-14T09:51:50.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.948+0000 7f4aa1a00700 1 Processor -- start 2024-03-14T09:51:50.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.948+0000 7f4aa1a00700 1 -- start start 2024-03-14T09:51:50.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4aa1a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a9c100c50 0x7f4a9c197050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:50.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4aa1a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c101e40 0x7f4a9c197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:50.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4aa1a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a9c1027e0 0x7f4a9c19e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:50.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4aa1a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c19eb30 con 0x7f4a9c1027e0 2024-03-14T09:51:50.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4aa1a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c19ec70 con 0x7f4a9c101e40 2024-03-14T09:51:50.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4aa1a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c19ee00 con 0x7f4a9c100c50 2024-03-14T09:51:50.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4a9b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a9c1027e0 0x7f4a9c19e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:50.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4a9aa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a9c100c50 0x7f4a9c197050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:50.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4a9aa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a9c100c50 0x7f4a9c197050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:56572/0 (socket says 172.21.15.78:56572) 2024-03-14T09:51:50.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4a9aa00700 1 -- 172.21.15.78:0/2237394903 learned_addr learned my addr 172.21.15.78:0/2237394903 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:51:50.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4a9b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a9c1027e0 0x7f4a9c19e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39598/0 (socket says 172.21.15.78:39598) 2024-03-14T09:51:50.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4a9a000700 1 --2- 172.21.15.78:0/2237394903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c101e40 0x7f4a9c197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:50.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4a9aa00700 1 -- 172.21.15.78:0/2237394903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c101e40 msgr2=0x7f4a9c197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:50.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4a9aa00700 1 --2- 172.21.15.78:0/2237394903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c101e40 0x7f4a9c197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:50.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4a9aa00700 1 -- 172.21.15.78:0/2237394903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a9c1027e0 msgr2=0x7f4a9c19e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:50.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4a9aa00700 1 --2- 172.21.15.78:0/2237394903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a9c1027e0 0x7f4a9c19e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:50.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4a9aa00700 1 -- 172.21.15.78:0/2237394903 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a880095f0 con 0x7f4a9c100c50 2024-03-14T09:51:50.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4a9b400700 1 --2- 172.21.15.78:0/2237394903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a9c1027e0 0x7f4a9c19e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:51:50.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.949+0000 7f4a9aa00700 1 --2- 172.21.15.78:0/2237394903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a9c100c50 0x7f4a9c197050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00d980 tx=0x7f4a8c00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:50.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.950+0000 7f4a93400700 1 -- 172.21.15.78:0/2237394903 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c004020 con 0x7f4a9c100c50 2024-03-14T09:51:50.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.950+0000 7f4a93400700 1 -- 172.21.15.78:0/2237394903 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a8c00f690 con 0x7f4a9c100c50 2024-03-14T09:51:50.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.950+0000 7f4a93400700 1 -- 172.21.15.78:0/2237394903 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c003a60 con 0x7f4a9c100c50 2024-03-14T09:51:50.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.950+0000 7f4aa1a00700 1 -- 172.21.15.78:0/2237394903 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9c19f0e0 con 0x7f4a9c100c50 2024-03-14T09:51:50.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.950+0000 7f4aa1a00700 1 -- 172.21.15.78:0/2237394903 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9c19f5a0 con 0x7f4a9c100c50 2024-03-14T09:51:50.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.951+0000 7f4aa1a00700 1 -- 172.21.15.78:0/2237394903 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9c108120 con 0x7f4a9c100c50 2024-03-14T09:51:50.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.953+0000 7f4a93400700 1 -- 172.21.15.78:0/2237394903 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f4a8c00fb00 con 0x7f4a9c100c50 2024-03-14T09:51:50.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.954+0000 7f4a93400700 1 --2- 172.21.15.78:0/2237394903 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4a800a0b10 0x7f4a800a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:50.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.954+0000 7f4a93400700 1 -- 172.21.15.78:0/2237394903 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4a8c0cb760 con 0x7f4a9c100c50 2024-03-14T09:51:50.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.954+0000 7f4a9a000700 1 --2- 172.21.15.78:0/2237394903 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4a800a0b10 0x7f4a800a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:50.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.955+0000 7f4a9a000700 1 --2- 172.21.15.78:0/2237394903 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4a800a0b10 0x7f4a800a2fd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4a88009990 tx=0x7f4a8800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:50.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:50.958+0000 7f4a93400700 1 -- 172.21.15.78:0/2237394903 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4a8c095530 con 0x7f4a9c100c50 2024-03-14T09:51:51.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:51.215+0000 7f4aa1a00700 1 -- 172.21.15.78:0/2237394903 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a9c0679e0 con 0x7f4a9c100c50 2024-03-14T09:51:51.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:51.216+0000 7f4a93400700 1 -- 172.21.15.78:0/2237394903 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6562 (secure 0 0 0) 0x7f4a8c098b50 con 0x7f4a9c100c50 2024-03-14T09:51:51.217 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:51:51.217 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24527,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2024-03-14T09:51:51.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:51.219+0000 7f4aa1a00700 1 -- 172.21.15.78:0/2237394903 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4a800a0b10 msgr2=0x7f4a800a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:51.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:51.219+0000 7f4aa1a00700 1 --2- 172.21.15.78:0/2237394903 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4a800a0b10 0x7f4a800a2fd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4a88009990 tx=0x7f4a8800b540 comp rx=0 tx=0).stop 2024-03-14T09:51:51.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:51.219+0000 7f4aa1a00700 1 -- 172.21.15.78:0/2237394903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a9c100c50 msgr2=0x7f4a9c197050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:51.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:51.219+0000 7f4aa1a00700 1 --2- 172.21.15.78:0/2237394903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a9c100c50 0x7f4a9c197050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00d980 tx=0x7f4a8c00dc90 comp rx=0 tx=0).stop 2024-03-14T09:51:51.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:51.219+0000 7f4aa1a00700 1 -- 172.21.15.78:0/2237394903 shutdown_connections 2024-03-14T09:51:51.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:51.219+0000 7f4aa1a00700 1 --2- 172.21.15.78:0/2237394903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a9c1027e0 0x7f4a9c19e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:51.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:51.219+0000 7f4aa1a00700 1 --2- 172.21.15.78:0/2237394903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a9c100c50 0x7f4a9c197050 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:51.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:51.219+0000 7f4aa1a00700 1 --2- 172.21.15.78:0/2237394903 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4a800a0b10 0x7f4a800a2fd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:51.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:51.219+0000 7f4aa1a00700 1 --2- 172.21.15.78:0/2237394903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9c101e40 0x7f4a9c197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:51.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:51.219+0000 7f4aa1a00700 1 -- 172.21.15.78:0/2237394903 >> 172.21.15.78:0/2237394903 conn(0x7f4a9c0fc1c0 msgr2=0x7f4a9c105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:51.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:51.219+0000 7f4aa1a00700 1 -- 172.21.15.78:0/2237394903 shutdown_connections 2024-03-14T09:51:51.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:51.219+0000 7f4aa1a00700 1 -- 172.21.15.78:0/2237394903 wait complete. 2024-03-14T09:51:51.221 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 9 2024-03-14T09:51:51.664 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:51 smithi152 ceph-mon[48279]: Deploying daemon mds.i on smithi152 2024-03-14T09:51:51.664 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:51 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] up:boot 2024-03-14T09:51:51.664 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:51 smithi152 ceph-mon[48279]: fsmap 8 up:standby 2024-03-14T09:51:51.664 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:51 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-14T09:51:51.665 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:51 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2237394903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:51.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:51 smithi132 ceph-mon[47959]: Deploying daemon mds.i on smithi152 2024-03-14T09:51:51.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:51 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] up:boot 2024-03-14T09:51:51.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:51 smithi132 ceph-mon[47959]: fsmap 8 up:standby 2024-03-14T09:51:51.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:51 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-14T09:51:51.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:51 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2237394903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:51.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:51 smithi078 ceph-mon[38143]: Deploying daemon mds.i on smithi152 2024-03-14T09:51:51.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:51 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] up:boot 2024-03-14T09:51:51.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:51 smithi078 ceph-mon[38143]: fsmap 8 up:standby 2024-03-14T09:51:51.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:51 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-14T09:51:51.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:51 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2237394903' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:52.002 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 14 09:51:51 smithi152 systemd[1]: Starting Ceph mds.i for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T09:51:52.495 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 14 09:51:52 smithi152 ceph-mds[85824]: -- 172.21.15.152:0/4160285462 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558ed24121a0 con 0x558ed23e8c00 2024-03-14T09:51:52.495 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 14 09:51:52 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-i[85801]: starting mds.i at 2024-03-14T09:51:52.495 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 14 09:51:52 smithi152 bash[85786]: cfd7c7cd8790e1748e966745d7ef403766cca50fc35a3130cb905c581dfba050 2024-03-14T09:51:52.495 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 14 09:51:52 smithi152 systemd[1]: Started Ceph mds.i for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:51:52.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:52 smithi152 ceph-mon[48279]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:52.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:52 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:52.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:52 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:52.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:52 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:52.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:52 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:52.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:52 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:52.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:52 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:52.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:52 smithi132 ceph-mon[47959]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:52.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:52 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:52.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:52 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:52.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:52 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:52.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:52 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:52.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:52 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:52.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:52 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:52.782 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:52 smithi078 ceph-mon[38143]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:52.782 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:52 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:52.782 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:52 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:52.782 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:52 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:52.782 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:52 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:52.782 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:52 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:52.782 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:52 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:53.239 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:51:53.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.457+0000 7fa917000700 1 -- 172.21.15.78:0/608673275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa910073810 msgr2=0x7fa910073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:53.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.457+0000 7fa917000700 1 --2- 172.21.15.78:0/608673275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa910073810 0x7fa910073c90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc009990 tx=0x7fa8fc009ca0 comp rx=0 tx=0).stop 2024-03-14T09:51:53.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.457+0000 7fa917000700 1 -- 172.21.15.78:0/608673275 shutdown_connections 2024-03-14T09:51:53.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.457+0000 7fa917000700 1 --2- 172.21.15.78:0/608673275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa910108b10 0x7fa91010afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:53.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.457+0000 7fa917000700 1 --2- 172.21.15.78:0/608673275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa910074df0 0x7fa910073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:53.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.457+0000 7fa917000700 1 --2- 172.21.15.78:0/608673275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa910073810 0x7fa910073c90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:53.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.457+0000 7fa917000700 1 -- 172.21.15.78:0/608673275 >> 172.21.15.78:0/608673275 conn(0x7fa9100fc1c0 msgr2=0x7fa9100fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:53.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.457+0000 7fa917000700 1 -- 172.21.15.78:0/608673275 shutdown_connections 2024-03-14T09:51:53.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.457+0000 7fa917000700 1 -- 172.21.15.78:0/608673275 wait complete. 2024-03-14T09:51:53.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.458+0000 7fa917000700 1 Processor -- start 2024-03-14T09:51:53.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.458+0000 7fa917000700 1 -- start start 2024-03-14T09:51:53.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa917000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa910073810 0x7fa910199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:53.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa917000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa910074df0 0x7fa9101997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:53.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa917000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa910108b10 0x7fa9101a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:53.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa917000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9101a0dc0 con 0x7fa910108b10 2024-03-14T09:51:53.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa917000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9101a0f00 con 0x7fa910073810 2024-03-14T09:51:53.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa917000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9101a1090 con 0x7fa910074df0 2024-03-14T09:51:53.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa90f400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa910074df0 0x7fa9101997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:53.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa915000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa910108b10 0x7fa9101a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:53.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa90fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa910073810 0x7fa910199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:53.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa90f400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa910074df0 0x7fa9101997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:56586/0 (socket says 172.21.15.78:56586) 2024-03-14T09:51:53.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa90fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa910073810 0x7fa910199290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:32842/0 (socket says 172.21.15.78:32842) 2024-03-14T09:51:53.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa90f400700 1 -- 172.21.15.78:0/3198476218 learned_addr learned my addr 172.21.15.78:0/3198476218 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:51:53.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa90fe00700 1 -- 172.21.15.78:0/3198476218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa910074df0 msgr2=0x7fa9101997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:53.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa90fe00700 1 --2- 172.21.15.78:0/3198476218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa910074df0 0x7fa9101997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:53.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa90fe00700 1 -- 172.21.15.78:0/3198476218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa910108b10 msgr2=0x7fa9101a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:53.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa90fe00700 1 --2- 172.21.15.78:0/3198476218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa910108b10 0x7fa9101a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:53.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa90fe00700 1 -- 172.21.15.78:0/3198476218 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa908009c40 con 0x7fa910073810 2024-03-14T09:51:53.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.459+0000 7fa90f400700 1 --2- 172.21.15.78:0/3198476218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa910074df0 0x7fa9101997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:51:53.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.460+0000 7fa90fe00700 1 --2- 172.21.15.78:0/3198476218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa910073810 0x7fa910199290 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa90000e9f0 tx=0x7fa90000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:53.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.460+0000 7fa90cc00700 1 -- 172.21.15.78:0/3198476218 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90000c9c0 con 0x7fa910073810 2024-03-14T09:51:53.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.460+0000 7fa917000700 1 -- 172.21.15.78:0/3198476218 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8fc0095f0 con 0x7fa910073810 2024-03-14T09:51:53.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.460+0000 7fa90cc00700 1 -- 172.21.15.78:0/3198476218 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa900010430 con 0x7fa910073810 2024-03-14T09:51:53.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.460+0000 7fa90cc00700 1 -- 172.21.15.78:0/3198476218 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa900018680 con 0x7fa910073810 2024-03-14T09:51:53.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.460+0000 7fa917000700 1 -- 172.21.15.78:0/3198476218 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9101a1740 con 0x7fa910073810 2024-03-14T09:51:53.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.461+0000 7fa917000700 1 -- 172.21.15.78:0/3198476218 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9101078d0 con 0x7fa910073810 2024-03-14T09:51:53.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.463+0000 7fa90cc00700 1 -- 172.21.15.78:0/3198476218 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fa900010860 con 0x7fa910073810 2024-03-14T09:51:53.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.464+0000 7fa90cc00700 1 --2- 172.21.15.78:0/3198476218 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa8f40a0b90 0x7fa8f40a3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:53.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.465+0000 7fa90cc00700 1 -- 172.21.15.78:0/3198476218 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa900014070 con 0x7fa910073810 2024-03-14T09:51:53.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.465+0000 7fa90f400700 1 --2- 172.21.15.78:0/3198476218 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa8f40a0b90 0x7fa8f40a3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:53.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.465+0000 7fa90f400700 1 --2- 172.21.15.78:0/3198476218 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa8f40a0b90 0x7fa8f40a3050 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00b2e0 tx=0x7fa8fc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.467+0000 7fa90cc00700 1 -- 172.21.15.78:0/3198476218 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa900095fc0 con 0x7fa910073810 2024-03-14T09:51:53.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:53 smithi078 ceph-mon[38143]: Deploying daemon mds.g on smithi078 2024-03-14T09:51:53.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:53 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.152:6838/1965809497,v1:172.21.15.152:6839/1965809497] up:boot 2024-03-14T09:51:53.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:53 smithi078 ceph-mon[38143]: fsmap 9 up:standby 2024-03-14T09:51:53.693 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:53 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-14T09:51:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.743+0000 7fa917000700 1 -- 172.21.15.78:0/3198476218 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa91010ecf0 con 0x7fa910073810 2024-03-14T09:51:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.744+0000 7fa90cc00700 1 -- 172.21.15.78:0/3198476218 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7324 (secure 0 0 0) 0x7fa9000995e0 con 0x7fa910073810 2024-03-14T09:51:53.745 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:51:53.745 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24527,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2024-03-14T09:51:53.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:53 smithi152 ceph-mon[48279]: Deploying daemon mds.g on smithi078 2024-03-14T09:51:53.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:53 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.152:6838/1965809497,v1:172.21.15.152:6839/1965809497] up:boot 2024-03-14T09:51:53.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:53 smithi152 ceph-mon[48279]: fsmap 9 up:standby 2024-03-14T09:51:53.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:53 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-14T09:51:53.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.748+0000 7fa917000700 1 -- 172.21.15.78:0/3198476218 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa8f40a0b90 msgr2=0x7fa8f40a3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:53.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.748+0000 7fa917000700 1 --2- 172.21.15.78:0/3198476218 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa8f40a0b90 0x7fa8f40a3050 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00b2e0 tx=0x7fa8fc005fb0 comp rx=0 tx=0).stop 2024-03-14T09:51:53.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.748+0000 7fa917000700 1 -- 172.21.15.78:0/3198476218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa910073810 msgr2=0x7fa910199290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:53.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.748+0000 7fa917000700 1 --2- 172.21.15.78:0/3198476218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa910073810 0x7fa910199290 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa90000e9f0 tx=0x7fa90000ed00 comp rx=0 tx=0).stop 2024-03-14T09:51:53.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.749+0000 7fa917000700 1 -- 172.21.15.78:0/3198476218 shutdown_connections 2024-03-14T09:51:53.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.749+0000 7fa917000700 1 --2- 172.21.15.78:0/3198476218 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa8f40a0b90 0x7fa8f40a3050 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:53.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.749+0000 7fa917000700 1 --2- 172.21.15.78:0/3198476218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa910073810 0x7fa910199290 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa90000e9f0 tx=0x7fa90000ed00 comp rx=0 tx=0).stop 2024-03-14T09:51:53.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.749+0000 7fa917000700 1 --2- 172.21.15.78:0/3198476218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa910108b10 0x7fa9101a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:53.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.749+0000 7fa917000700 1 --2- 172.21.15.78:0/3198476218 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa910074df0 0x7fa9101997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:53.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.749+0000 7fa917000700 1 -- 172.21.15.78:0/3198476218 >> 172.21.15.78:0/3198476218 conn(0x7fa9100fc1c0 msgr2=0x7fa910104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:53.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.749+0000 7fa917000700 1 -- 172.21.15.78:0/3198476218 shutdown_connections 2024-03-14T09:51:53.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:53.749+0000 7fa917000700 1 -- 172.21.15.78:0/3198476218 wait complete. 2024-03-14T09:51:53.752 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 10 2024-03-14T09:51:53.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:53 smithi132 ceph-mon[47959]: Deploying daemon mds.g on smithi078 2024-03-14T09:51:53.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:53 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.152:6838/1965809497,v1:172.21.15.152:6839/1965809497] up:boot 2024-03-14T09:51:53.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:53 smithi132 ceph-mon[47959]: fsmap 9 up:standby 2024-03-14T09:51:53.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:53 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-14T09:51:54.054 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 14 09:51:53 smithi078 systemd[1]: Starting Ceph mds.g for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T09:51:54.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:54 smithi152 ceph-mon[48279]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:54.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:54 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3198476218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:54.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:54 smithi132 ceph-mon[47959]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:54.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:54 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3198476218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:54.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:54 smithi078 ceph-mon[38143]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:54.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:54 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3198476218' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:54.803 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 14 09:51:54 smithi078 bash[87641]: 90b8142a99a131135b8f17a8dadd2da1aa23c1bb3435a20232feca9ac53e9250 2024-03-14T09:51:54.804 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 14 09:51:54 smithi078 systemd[1]: Started Ceph mds.g for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:51:54.804 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 14 09:51:54 smithi078 ceph-mds[87679]: -- 172.21.15.78:0/2323344732 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5578150ac1a0 con 0x557815084800 2024-03-14T09:51:54.804 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 14 09:51:54 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-g[87656]: starting mds.g at 2024-03-14T09:51:55.756 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:55 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:55.756 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:55 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:55.756 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:55 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:55.756 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:55 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:55.756 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:55 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:55.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:55 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:55.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:55 smithi078 ceph-mon[38143]: Deploying daemon mds.a on smithi078 2024-03-14T09:51:55.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:55 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] up:boot 2024-03-14T09:51:55.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:55 smithi078 ceph-mon[38143]: fsmap 10 up:standby 2024-03-14T09:51:55.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:55 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-14T09:51:55.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:55 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:55.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:55 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:55.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:55 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:55.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:55 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:55.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:55 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:55.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:55 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:55.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:55 smithi132 ceph-mon[47959]: Deploying daemon mds.a on smithi078 2024-03-14T09:51:55.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:55 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] up:boot 2024-03-14T09:51:55.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:55 smithi132 ceph-mon[47959]: fsmap 10 up:standby 2024-03-14T09:51:55.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:55 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-14T09:51:55.772 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:51:55.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:55 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:55.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:55 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:55.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:55 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:55.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:55 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:55.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:55 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:55.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:55 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:55.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:55 smithi152 ceph-mon[48279]: Deploying daemon mds.a on smithi078 2024-03-14T09:51:55.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:55 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] up:boot 2024-03-14T09:51:55.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:55 smithi152 ceph-mon[48279]: fsmap 10 up:standby 2024-03-14T09:51:56.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:55 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-14T09:51:56.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.145+0000 7f4f49600700 1 -- 172.21.15.78:0/783770740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f44073810 msgr2=0x7f4f44073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:56.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.145+0000 7f4f49600700 1 --2- 172.21.15.78:0/783770740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f44073810 0x7f4f44073c90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4f300098f0 tx=0x7f4f30009c00 comp rx=0 tx=0).stop 2024-03-14T09:51:56.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.146+0000 7f4f49600700 1 -- 172.21.15.78:0/783770740 shutdown_connections 2024-03-14T09:51:56.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.146+0000 7f4f49600700 1 --2- 172.21.15.78:0/783770740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f44074df0 0x7f4f44073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:56.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.146+0000 7f4f49600700 1 --2- 172.21.15.78:0/783770740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f44108b10 0x7f4f4410afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:56.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.146+0000 7f4f49600700 1 --2- 172.21.15.78:0/783770740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f44073810 0x7f4f44073c90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:56.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.146+0000 7f4f49600700 1 -- 172.21.15.78:0/783770740 >> 172.21.15.78:0/783770740 conn(0x7f4f440fc1c0 msgr2=0x7f4f440fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:56.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.146+0000 7f4f49600700 1 -- 172.21.15.78:0/783770740 shutdown_connections 2024-03-14T09:51:56.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.146+0000 7f4f49600700 1 -- 172.21.15.78:0/783770740 wait complete. 2024-03-14T09:51:56.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.147+0000 7f4f49600700 1 Processor -- start 2024-03-14T09:51:56.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.147+0000 7f4f49600700 1 -- start start 2024-03-14T09:51:56.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.147+0000 7f4f49600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f44073810 0x7f4f441992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:56.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.147+0000 7f4f49600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f44074df0 0x7f4f441997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:56.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.147+0000 7f4f49600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f44108b10 0x7f4f441a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:56.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.147+0000 7f4f49600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f441a0d80 con 0x7f4f44074df0 2024-03-14T09:51:56.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.147+0000 7f4f49600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f441a0ec0 con 0x7f4f44108b10 2024-03-14T09:51:56.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.147+0000 7f4f49600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f441a1050 con 0x7f4f44073810 2024-03-14T09:51:56.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.147+0000 7f4f42000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f44074df0 0x7f4f441997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:56.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.148+0000 7f4f42000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f44074df0 0x7f4f441997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38978/0 (socket says 172.21.15.78:38978) 2024-03-14T09:51:56.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.148+0000 7f4f42000700 1 -- 172.21.15.78:0/1507255278 learned_addr learned my addr 172.21.15.78:0/1507255278 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:51:56.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.148+0000 7f4f42a00700 1 --2- 172.21.15.78:0/1507255278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f44073810 0x7f4f441992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:56.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.148+0000 7f4f43400700 1 --2- 172.21.15.78:0/1507255278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f44108b10 0x7f4f441a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:56.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.148+0000 7f4f42000700 1 -- 172.21.15.78:0/1507255278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f44073810 msgr2=0x7f4f441992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:56.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.148+0000 7f4f42000700 1 --2- 172.21.15.78:0/1507255278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f44073810 0x7f4f441992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:56.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.148+0000 7f4f42000700 1 -- 172.21.15.78:0/1507255278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f44108b10 msgr2=0x7f4f441a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:56.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.148+0000 7f4f42000700 1 --2- 172.21.15.78:0/1507255278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f44108b10 0x7f4f441a07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:56.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.148+0000 7f4f42000700 1 -- 172.21.15.78:0/1507255278 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f30009520 con 0x7f4f44074df0 2024-03-14T09:51:56.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.148+0000 7f4f43400700 1 --2- 172.21.15.78:0/1507255278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f44108b10 0x7f4f441a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:51:56.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.148+0000 7f4f42000700 1 --2- 172.21.15.78:0/1507255278 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f44074df0 0x7f4f441997e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000fb70 tx=0x7f4f3000b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:56.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.148+0000 7f4f3b400700 1 -- 172.21.15.78:0/1507255278 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3001c070 con 0x7f4f44074df0 2024-03-14T09:51:56.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.149+0000 7f4f49600700 1 -- 172.21.15.78:0/1507255278 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f441a12d0 con 0x7f4f44074df0 2024-03-14T09:51:56.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.148+0000 7f4f3b400700 1 -- 172.21.15.78:0/1507255278 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f3000f7d0 con 0x7f4f44074df0 2024-03-14T09:51:56.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.149+0000 7f4f3b400700 1 -- 172.21.15.78:0/1507255278 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f300039b0 con 0x7f4f44074df0 2024-03-14T09:51:56.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.149+0000 7f4f49600700 1 -- 172.21.15.78:0/1507255278 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f441a1750 con 0x7f4f44074df0 2024-03-14T09:51:56.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.150+0000 7f4f49600700 1 -- 172.21.15.78:0/1507255278 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f441078d0 con 0x7f4f44074df0 2024-03-14T09:51:56.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.151+0000 7f4f3b400700 1 -- 172.21.15.78:0/1507255278 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f4f30020030 con 0x7f4f44074df0 2024-03-14T09:51:56.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.153+0000 7f4f3b400700 1 --2- 172.21.15.78:0/1507255278 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f280a0b10 0x7f4f280a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:56.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.153+0000 7f4f3b400700 1 -- 172.21.15.78:0/1507255278 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4f300ccc30 con 0x7f4f44074df0 2024-03-14T09:51:56.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.153+0000 7f4f42a00700 1 --2- 172.21.15.78:0/1507255278 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f280a0b10 0x7f4f280a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:56.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.154+0000 7f4f42a00700 1 --2- 172.21.15.78:0/1507255278 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f280a0b10 0x7f4f280a2fd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009c60 tx=0x7f4f34009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:56.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.157+0000 7f4f3b400700 1 -- 172.21.15.78:0/1507255278 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4f30096a90 con 0x7f4f44074df0 2024-03-14T09:51:56.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.457+0000 7f4f49600700 1 -- 172.21.15.78:0/1507255278 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f4410ed60 con 0x7f4f44074df0 2024-03-14T09:51:56.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.458+0000 7f4f3b400700 1 -- 172.21.15.78:0/1507255278 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8082 (secure 0 0 0) 0x7f4f3009a0b0 con 0x7f4f44074df0 2024-03-14T09:51:56.459 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:51:56.459 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24527,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[]} 2024-03-14T09:51:56.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.460+0000 7f4f49600700 1 -- 172.21.15.78:0/1507255278 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f280a0b10 msgr2=0x7f4f280a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:56.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.460+0000 7f4f49600700 1 --2- 172.21.15.78:0/1507255278 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f280a0b10 0x7f4f280a2fd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4f34009c60 tx=0x7f4f34009260 comp rx=0 tx=0).stop 2024-03-14T09:51:56.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.460+0000 7f4f49600700 1 -- 172.21.15.78:0/1507255278 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f44074df0 msgr2=0x7f4f441997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:56.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.460+0000 7f4f49600700 1 --2- 172.21.15.78:0/1507255278 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f44074df0 0x7f4f441997e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000fb70 tx=0x7f4f3000b790 comp rx=0 tx=0).stop 2024-03-14T09:51:56.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.460+0000 7f4f49600700 1 -- 172.21.15.78:0/1507255278 shutdown_connections 2024-03-14T09:51:56.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.460+0000 7f4f49600700 1 --2- 172.21.15.78:0/1507255278 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f280a0b10 0x7f4f280a2fd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:56.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.460+0000 7f4f49600700 1 --2- 172.21.15.78:0/1507255278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f44108b10 0x7f4f441a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:56.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.460+0000 7f4f49600700 1 --2- 172.21.15.78:0/1507255278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f44073810 0x7f4f441992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:56.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.460+0000 7f4f49600700 1 --2- 172.21.15.78:0/1507255278 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f44074df0 0x7f4f441997e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:56.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.460+0000 7f4f49600700 1 -- 172.21.15.78:0/1507255278 >> 172.21.15.78:0/1507255278 conn(0x7f4f440fc1c0 msgr2=0x7f4f44104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:56.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.461+0000 7f4f49600700 1 -- 172.21.15.78:0/1507255278 shutdown_connections 2024-03-14T09:51:56.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:56.461+0000 7f4f49600700 1 -- 172.21.15.78:0/1507255278 wait complete. 2024-03-14T09:51:56.463 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 11 2024-03-14T09:51:56.756 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:56 smithi078 ceph-mon[38143]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:56.757 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:56 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1507255278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:56.757 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 14 09:51:56 smithi078 systemd[1]: Starting Ceph mds.a for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T09:51:56.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:56 smithi132 ceph-mon[47959]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:56.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:56 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1507255278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:56.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:56 smithi152 ceph-mon[48279]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:56.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:56 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1507255278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:51:57.303 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 14 09:51:56 smithi078 ceph-mds[88067]: -- 172.21.15.78:0/3905121702 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5646143721a0 con 0x56461434a800 2024-03-14T09:51:57.303 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 14 09:51:56 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-a[88044]: starting mds.a at 2024-03-14T09:51:57.303 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 14 09:51:56 smithi078 bash[88029]: 7fb9b0e19bcb23520d41b4e5d66db868fc168fc9907aaa915631c8b41b6dda87 2024-03-14T09:51:57.304 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 14 09:51:56 smithi078 systemd[1]: Started Ceph mds.a for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:51:58.082 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:57 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:58.083 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:57 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:58.083 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:57 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:58.083 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:57 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:58.083 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:57 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:58.083 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:57 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:58.083 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:57 smithi132 ceph-mon[47959]: Deploying daemon mds.b on smithi132 2024-03-14T09:51:58.083 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:57 smithi132 ceph-mon[47959]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:58.083 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:57 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.78:6840/1109060843,v1:172.21.15.78:6841/1109060843] up:boot 2024-03-14T09:51:58.083 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:57 smithi132 ceph-mon[47959]: fsmap 11 up:standby 2024-03-14T09:51:58.083 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:57 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-14T09:51:58.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:57 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:58.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:57 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:58.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:57 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:58.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:57 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:58.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:57 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:58.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:57 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:58.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:57 smithi152 ceph-mon[48279]: Deploying daemon mds.b on smithi132 2024-03-14T09:51:58.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:57 smithi152 ceph-mon[48279]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:58.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:57 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.78:6840/1109060843,v1:172.21.15.78:6841/1109060843] up:boot 2024-03-14T09:51:58.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:57 smithi152 ceph-mon[48279]: fsmap 11 up:standby 2024-03-14T09:51:58.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:57 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-14T09:51:58.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:57 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:58.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:57 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:58.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:57 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:51:58.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:57 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T09:51:58.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:57 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T09:51:58.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:57 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:51:58.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:57 smithi078 ceph-mon[38143]: Deploying daemon mds.b on smithi132 2024-03-14T09:51:58.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:57 smithi078 ceph-mon[38143]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:51:58.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:57 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.78:6840/1109060843,v1:172.21.15.78:6841/1109060843] up:boot 2024-03-14T09:51:58.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:57 smithi078 ceph-mon[38143]: fsmap 11 up:standby 2024-03-14T09:51:58.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:57 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-14T09:51:58.484 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:51:58.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.687+0000 7fdd2ee00700 1 -- 172.21.15.78:0/2741097437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd281027e0 msgr2=0x7fdd28109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:58.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.687+0000 7fdd2ee00700 1 --2- 172.21.15.78:0/2741097437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd281027e0 0x7fdd28109090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdd2000b0d0 tx=0x7fdd2000b3e0 comp rx=0 tx=0).stop 2024-03-14T09:51:58.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.687+0000 7fdd2ee00700 1 -- 172.21.15.78:0/2741097437 shutdown_connections 2024-03-14T09:51:58.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.688+0000 7fdd2ee00700 1 --2- 172.21.15.78:0/2741097437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd281027e0 0x7fdd28109090 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:58.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.688+0000 7fdd2ee00700 1 --2- 172.21.15.78:0/2741097437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd28100c50 0x7fdd28101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:58.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.688+0000 7fdd2ee00700 1 --2- 172.21.15.78:0/2741097437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd28101e40 0x7fdd281022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:58.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.688+0000 7fdd2ee00700 1 -- 172.21.15.78:0/2741097437 >> 172.21.15.78:0/2741097437 conn(0x7fdd280fc1c0 msgr2=0x7fdd280fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:58.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.688+0000 7fdd2ee00700 1 -- 172.21.15.78:0/2741097437 shutdown_connections 2024-03-14T09:51:58.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.688+0000 7fdd2ee00700 1 -- 172.21.15.78:0/2741097437 wait complete. 2024-03-14T09:51:58.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.689+0000 7fdd2ee00700 1 Processor -- start 2024-03-14T09:51:58.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.689+0000 7fdd2ee00700 1 -- start start 2024-03-14T09:51:58.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.689+0000 7fdd2ee00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd28100c50 0x7fdd28074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:58.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.689+0000 7fdd2ee00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd28101e40 0x7fdd280731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.689+0000 7fdd2ee00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd281027e0 0x7fdd280736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.689+0000 7fdd2ee00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd28073e80 con 0x7fdd28100c50 2024-03-14T09:51:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.689+0000 7fdd2ee00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd28073ff0 con 0x7fdd28101e40 2024-03-14T09:51:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.689+0000 7fdd2ee00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd281a1170 con 0x7fdd281027e0 2024-03-14T09:51:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.689+0000 7fdd27e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd28100c50 0x7fdd28074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.689+0000 7fdd27e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd28100c50 0x7fdd28074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39000/0 (socket says 172.21.15.78:39000) 2024-03-14T09:51:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.689+0000 7fdd27e00700 1 -- 172.21.15.78:0/3786287562 learned_addr learned my addr 172.21.15.78:0/3786287562 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:51:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.689+0000 7fdd27400700 1 --2- 172.21.15.78:0/3786287562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd28101e40 0x7fdd280731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.689+0000 7fdd2ce00700 1 --2- 172.21.15.78:0/3786287562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd281027e0 0x7fdd280736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.690+0000 7fdd27e00700 1 -- 172.21.15.78:0/3786287562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd281027e0 msgr2=0x7fdd280736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.690+0000 7fdd27e00700 1 --2- 172.21.15.78:0/3786287562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd281027e0 0x7fdd280736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.690+0000 7fdd27e00700 1 -- 172.21.15.78:0/3786287562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd28101e40 msgr2=0x7fdd280731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.690+0000 7fdd27e00700 1 --2- 172.21.15.78:0/3786287562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd28101e40 0x7fdd280731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.690+0000 7fdd27e00700 1 -- 172.21.15.78:0/3786287562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd20009d00 con 0x7fdd28100c50 2024-03-14T09:51:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.690+0000 7fdd2ce00700 1 --2- 172.21.15.78:0/3786287562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd281027e0 0x7fdd280736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:51:58.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.690+0000 7fdd27e00700 1 --2- 172.21.15.78:0/3786287562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd28100c50 0x7fdd28074b10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800e970 tx=0x7fdd1800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:58.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.690+0000 7fdd24c00700 1 -- 172.21.15.78:0/3786287562 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18009790 con 0x7fdd28100c50 2024-03-14T09:51:58.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.690+0000 7fdd24c00700 1 -- 172.21.15.78:0/3786287562 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd1800eec0 con 0x7fdd28100c50 2024-03-14T09:51:58.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.690+0000 7fdd24c00700 1 -- 172.21.15.78:0/3786287562 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18003c90 con 0x7fdd28100c50 2024-03-14T09:51:58.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.690+0000 7fdd2ee00700 1 -- 172.21.15.78:0/3786287562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd281a1400 con 0x7fdd28100c50 2024-03-14T09:51:58.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.690+0000 7fdd2ee00700 1 -- 172.21.15.78:0/3786287562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd281a1910 con 0x7fdd28100c50 2024-03-14T09:51:58.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.691+0000 7fdd2ee00700 1 -- 172.21.15.78:0/3786287562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd28108080 con 0x7fdd28100c50 2024-03-14T09:51:58.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.693+0000 7fdd24c00700 1 -- 172.21.15.78:0/3786287562 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fdd1800cc70 con 0x7fdd28100c50 2024-03-14T09:51:58.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.694+0000 7fdd24c00700 1 --2- 172.21.15.78:0/3786287562 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdd0c0a0b10 0x7fdd0c0a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:51:58.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.694+0000 7fdd24c00700 1 -- 172.21.15.78:0/3786287562 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdd18014070 con 0x7fdd28100c50 2024-03-14T09:51:58.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.695+0000 7fdd27400700 1 --2- 172.21.15.78:0/3786287562 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdd0c0a0b10 0x7fdd0c0a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:51:58.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.695+0000 7fdd27400700 1 --2- 172.21.15.78:0/3786287562 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdd0c0a0b10 0x7fdd0c0a2fd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdd14005950 tx=0x7fdd1400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:51:58.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.698+0000 7fdd24c00700 1 -- 172.21.15.78:0/3786287562 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fdd18095b90 con 0x7fdd28100c50 2024-03-14T09:51:58.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.967+0000 7fdd2ee00700 1 -- 172.21.15.78:0/3786287562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd280679e0 con 0x7fdd28100c50 2024-03-14T09:51:58.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.968+0000 7fdd24c00700 1 -- 172.21.15.78:0/3786287562 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8840 (secure 0 0 0) 0x7fdd180991b0 con 0x7fdd28100c50 2024-03-14T09:51:58.970 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:51:58.970 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24527,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[]} 2024-03-14T09:51:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.970+0000 7fdd2ee00700 1 -- 172.21.15.78:0/3786287562 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdd0c0a0b10 msgr2=0x7fdd0c0a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.970+0000 7fdd2ee00700 1 --2- 172.21.15.78:0/3786287562 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdd0c0a0b10 0x7fdd0c0a2fd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdd14005950 tx=0x7fdd1400b410 comp rx=0 tx=0).stop 2024-03-14T09:51:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.970+0000 7fdd2ee00700 1 -- 172.21.15.78:0/3786287562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd28100c50 msgr2=0x7fdd28074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:51:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.970+0000 7fdd2ee00700 1 --2- 172.21.15.78:0/3786287562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd28100c50 0x7fdd28074b10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800e970 tx=0x7fdd1800ec80 comp rx=0 tx=0).stop 2024-03-14T09:51:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.970+0000 7fdd2ee00700 1 -- 172.21.15.78:0/3786287562 shutdown_connections 2024-03-14T09:51:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.970+0000 7fdd2ee00700 1 --2- 172.21.15.78:0/3786287562 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdd281027e0 0x7fdd280736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.970+0000 7fdd2ee00700 1 --2- 172.21.15.78:0/3786287562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd28100c50 0x7fdd28074b10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.970+0000 7fdd2ee00700 1 --2- 172.21.15.78:0/3786287562 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdd0c0a0b10 0x7fdd0c0a2fd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.970+0000 7fdd2ee00700 1 --2- 172.21.15.78:0/3786287562 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd28101e40 0x7fdd280731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:51:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.970+0000 7fdd2ee00700 1 -- 172.21.15.78:0/3786287562 >> 172.21.15.78:0/3786287562 conn(0x7fdd280fc1c0 msgr2=0x7fdd281051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:51:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.971+0000 7fdd2ee00700 1 -- 172.21.15.78:0/3786287562 shutdown_connections 2024-03-14T09:51:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:51:58.971+0000 7fdd2ee00700 1 -- 172.21.15.78:0/3786287562 wait complete. 2024-03-14T09:51:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 12 2024-03-14T09:51:59.020 INFO:journalctl@ceph.mds.b.smithi132.stdout:Mar 14 09:51:58 smithi132 bash[86012]: 3dcddc4b55226c2ab7ab3b0e1e92f46b5b7b58866dc6715412887664721f630b 2024-03-14T09:51:59.020 INFO:journalctl@ceph.mds.b.smithi132.stdout:Mar 14 09:51:58 smithi132 systemd[1]: Started Ceph mds.b for 83662c4e-e1e7-11ee-95c9-87774f69a715. 2024-03-14T09:52:00.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:59 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:00.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:59 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:00.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:59 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:00.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:59 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:00.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:59 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:00.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:59 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3786287562' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:52:00.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:59 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] up:boot 2024-03-14T09:52:00.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:59 smithi078 ceph-mon[38143]: fsmap 12 up:standby 2024-03-14T09:52:00.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:59 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-14T09:52:00.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:59 smithi078 ceph-mon[38143]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:52:00.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:51:59 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:52:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:59 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:59 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:59 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:59 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:59 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:59 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3786287562' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:52:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:59 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] up:boot 2024-03-14T09:52:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:59 smithi152 ceph-mon[48279]: fsmap 12 up:standby 2024-03-14T09:52:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:59 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-14T09:52:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:59 smithi152 ceph-mon[48279]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:52:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:51:59 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:52:00.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:59 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:00.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:59 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:00.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:59 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:00.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:59 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:00.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:59 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:00.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:59 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3786287562' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:52:00.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:59 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] up:boot 2024-03-14T09:52:00.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:59 smithi132 ceph-mon[47959]: fsmap 12 up:standby 2024-03-14T09:52:00.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:59 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-14T09:52:00.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:59 smithi132 ceph-mon[47959]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:52:00.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:51:59 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:52:00.990 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:52:01.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.205+0000 7f3fc6600700 1 -- 172.21.15.78:0/2286254856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3fc01016b0 msgr2=0x7f3fc0103aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:01.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.205+0000 7f3fc6600700 1 --2- 172.21.15.78:0/2286254856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3fc01016b0 0x7f3fc0103aa0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0009990 tx=0x7f3fb0009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.205+0000 7f3fc6600700 1 -- 172.21.15.78:0/2286254856 shutdown_connections 2024-03-14T09:52:01.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.205+0000 7f3fc6600700 1 --2- 172.21.15.78:0/2286254856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3fc01016b0 0x7f3fc0103aa0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.205+0000 7f3fc6600700 1 --2- 172.21.15.78:0/2286254856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fc0106910 0x7f3fc0108d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.205+0000 7f3fc6600700 1 --2- 172.21.15.78:0/2286254856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3fc0103fe0 0x7f3fc01063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.205+0000 7f3fc6600700 1 -- 172.21.15.78:0/2286254856 >> 172.21.15.78:0/2286254856 conn(0x7f3fc00fb0a0 msgr2=0x7f3fc00fd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:01.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.205+0000 7f3fc6600700 1 -- 172.21.15.78:0/2286254856 shutdown_connections 2024-03-14T09:52:01.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.205+0000 7f3fc6600700 1 -- 172.21.15.78:0/2286254856 wait complete. 2024-03-14T09:52:01.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.206+0000 7f3fc6600700 1 Processor -- start 2024-03-14T09:52:01.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.206+0000 7f3fc6600700 1 -- start start 2024-03-14T09:52:01.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.207+0000 7f3fc6600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fc01016b0 0x7f3fc01991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:01.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.207+0000 7f3fc6600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3fc0103fe0 0x7f3fc0199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:01.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.207+0000 7f3fc6600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3fc0106910 0x7f3fc01a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:01.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.207+0000 7f3fc6600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc01a0d20 con 0x7f3fc01016b0 2024-03-14T09:52:01.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.207+0000 7f3fc6600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc01a0e60 con 0x7f3fc0106910 2024-03-14T09:52:01.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.207+0000 7f3fc6600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc01a0ff0 con 0x7f3fc0103fe0 2024-03-14T09:52:01.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.207+0000 7f3fbea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3fc0103fe0 0x7f3fc0199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:01.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.207+0000 7f3fbea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3fc0103fe0 0x7f3fc0199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:38762/0 (socket says 172.21.15.78:38762) 2024-03-14T09:52:01.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.207+0000 7f3fbfe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3fc0106910 0x7f3fc01a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.207+0000 7f3fbea00700 1 -- 172.21.15.78:0/1209797800 learned_addr learned my addr 172.21.15.78:0/1209797800 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.207+0000 7f3fbf400700 1 --2- 172.21.15.78:0/1209797800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fc01016b0 0x7f3fc01991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.207+0000 7f3fbea00700 1 -- 172.21.15.78:0/1209797800 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3fc0106910 msgr2=0x7f3fc01a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.207+0000 7f3fbea00700 1 --2- 172.21.15.78:0/1209797800 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3fc0106910 0x7f3fc01a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.207+0000 7f3fbea00700 1 -- 172.21.15.78:0/1209797800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fc01016b0 msgr2=0x7f3fc01991f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.207+0000 7f3fbea00700 1 --2- 172.21.15.78:0/1209797800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fc01016b0 0x7f3fc01991f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.207+0000 7f3fbea00700 1 -- 172.21.15.78:0/1209797800 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac009520 con 0x7f3fc0103fe0 2024-03-14T09:52:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.208+0000 7f3fbea00700 1 --2- 172.21.15.78:0/1209797800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3fc0103fe0 0x7f3fc0199730 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3fac00eb30 tx=0x7f3fac00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.208+0000 7f3fb7e00700 1 -- 172.21.15.78:0/1209797800 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac009880 con 0x7f3fc0103fe0 2024-03-14T09:52:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.208+0000 7f3fb7e00700 1 -- 172.21.15.78:0/1209797800 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3fac0044e0 con 0x7f3fc0103fe0 2024-03-14T09:52:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.208+0000 7f3fb7e00700 1 -- 172.21.15.78:0/1209797800 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac003d80 con 0x7f3fc0103fe0 2024-03-14T09:52:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.208+0000 7f3fc6600700 1 -- 172.21.15.78:0/1209797800 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb00095f0 con 0x7f3fc0103fe0 2024-03-14T09:52:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.208+0000 7f3fc6600700 1 -- 172.21.15.78:0/1209797800 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc01a1600 con 0x7f3fc0103fe0 2024-03-14T09:52:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.209+0000 7f3fc6600700 1 -- 172.21.15.78:0/1209797800 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc01084e0 con 0x7f3fc0103fe0 2024-03-14T09:52:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.211+0000 7f3fb7e00700 1 -- 172.21.15.78:0/1209797800 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f3fac003f20 con 0x7f3fc0103fe0 2024-03-14T09:52:01.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.212+0000 7f3fb7e00700 1 --2- 172.21.15.78:0/1209797800 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3fa40a0a90 0x7f3fa40a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:01.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.212+0000 7f3fb7e00700 1 -- 172.21.15.78:0/1209797800 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3fac014070 con 0x7f3fc0103fe0 2024-03-14T09:52:01.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.212+0000 7f3fbf400700 1 --2- 172.21.15.78:0/1209797800 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3fa40a0a90 0x7f3fa40a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:01.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.213+0000 7f3fbf400700 1 --2- 172.21.15.78:0/1209797800 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3fa40a0a90 0x7f3fa40a2f50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3fb000b2e0 tx=0x7f3fb0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.216+0000 7f3fb7e00700 1 -- 172.21.15.78:0/1209797800 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3fac095f80 con 0x7f3fc0103fe0 2024-03-14T09:52:01.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:01 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:01 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:01 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:01 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:01 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:01 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:01 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:52:01.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:01 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:52:01.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:01 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:01 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:01 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:52:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:01 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:01 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:01 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:01 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:01 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:01 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:01 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:52:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:01 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:52:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:01 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:01 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:01 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:52:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:01 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:01 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:01 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:01 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:01 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:01 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:01 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:52:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:01 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:52:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:01 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:01 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:01.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:01 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:52:01.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.522+0000 7f3fc6600700 1 -- 172.21.15.78:0/1209797800 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fc00679e0 con 0x7f3fc0103fe0 2024-03-14T09:52:01.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.523+0000 7f3fb7e00700 1 -- 172.21.15.78:0/1209797800 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9601 (secure 0 0 0) 0x7f3fac0995a0 con 0x7f3fc0103fe0 2024-03-14T09:52:01.524 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:01.524 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24527,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24572,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2024-03-14T09:52:01.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.526+0000 7f3fc6600700 1 -- 172.21.15.78:0/1209797800 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3fa40a0a90 msgr2=0x7f3fa40a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:01.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.526+0000 7f3fc6600700 1 --2- 172.21.15.78:0/1209797800 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3fa40a0a90 0x7f3fa40a2f50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3fb000b2e0 tx=0x7f3fb0005fb0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.526+0000 7f3fc6600700 1 -- 172.21.15.78:0/1209797800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3fc0103fe0 msgr2=0x7f3fc0199730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:01.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.526+0000 7f3fc6600700 1 --2- 172.21.15.78:0/1209797800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3fc0103fe0 0x7f3fc0199730 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3fac00eb30 tx=0x7f3fac00ee40 comp rx=0 tx=0).stop 2024-03-14T09:52:01.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.527+0000 7f3fc6600700 1 -- 172.21.15.78:0/1209797800 shutdown_connections 2024-03-14T09:52:01.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.527+0000 7f3fc6600700 1 --2- 172.21.15.78:0/1209797800 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3fa40a0a90 0x7f3fa40a2f50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.527+0000 7f3fc6600700 1 --2- 172.21.15.78:0/1209797800 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3fc0106910 0x7f3fc01a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.527+0000 7f3fc6600700 1 --2- 172.21.15.78:0/1209797800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3fc01016b0 0x7f3fc01991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.527+0000 7f3fc6600700 1 --2- 172.21.15.78:0/1209797800 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3fc0103fe0 0x7f3fc0199730 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.527+0000 7f3fc6600700 1 -- 172.21.15.78:0/1209797800 >> 172.21.15.78:0/1209797800 conn(0x7f3fc00fb0a0 msgr2=0x7f3fc0100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:01.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.527+0000 7f3fc6600700 1 -- 172.21.15.78:0/1209797800 shutdown_connections 2024-03-14T09:52:01.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.527+0000 7f3fc6600700 1 -- 172.21.15.78:0/1209797800 wait complete. 2024-03-14T09:52:01.529 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 13 2024-03-14T09:52:01.549 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:52:01.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.758+0000 7f99e2600700 1 -- 172.21.15.78:0/619652568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99dc074d70 msgr2=0x7f99dc0731c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:01.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.758+0000 7f99e2600700 1 --2- 172.21.15.78:0/619652568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99dc074d70 0x7f99dc0731c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009990 tx=0x7f99d0009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.758+0000 7f99e2600700 1 -- 172.21.15.78:0/619652568 shutdown_connections 2024-03-14T09:52:01.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.758+0000 7f99e2600700 1 --2- 172.21.15.78:0/619652568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99dc074d70 0x7f99dc0731c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.758+0000 7f99e2600700 1 --2- 172.21.15.78:0/619652568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99dc108b00 0x7f99dc10af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.758+0000 7f99e2600700 1 --2- 172.21.15.78:0/619652568 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99dc073790 0x7f99dc073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.758+0000 7f99e2600700 1 -- 172.21.15.78:0/619652568 >> 172.21.15.78:0/619652568 conn(0x7f99dc0fc180 msgr2=0x7f99dc0fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:01.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.758+0000 7f99e2600700 1 -- 172.21.15.78:0/619652568 shutdown_connections 2024-03-14T09:52:01.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.758+0000 7f99e2600700 1 -- 172.21.15.78:0/619652568 wait complete. 2024-03-14T09:52:01.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.759+0000 7f99e2600700 1 Processor -- start 2024-03-14T09:52:01.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.759+0000 7f99e2600700 1 -- start start 2024-03-14T09:52:01.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99e2600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99dc073790 0x7f99dc1991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:01.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99e2600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99dc074d70 0x7f99dc199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:01.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99e2600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99dc108b00 0x7f99dc1a0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:01.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99e2600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc1a0d00 con 0x7f99dc073790 2024-03-14T09:52:01.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99e2600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc1a0e40 con 0x7f99dc108b00 2024-03-14T09:52:01.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99e2600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc1a0fd0 con 0x7f99dc074d70 2024-03-14T09:52:01.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99e1200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99dc073790 0x7f99dc1991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99e1200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99dc073790 0x7f99dc1991d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39030/0 (socket says 172.21.15.78:39030) 2024-03-14T09:52:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99e1200700 1 -- 172.21.15.78:0/3156265282 learned_addr learned my addr 172.21.15.78:0/3156265282 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99e1c00700 1 --2- 172.21.15.78:0/3156265282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99dc108b00 0x7f99dc1a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99dbe00700 1 --2- 172.21.15.78:0/3156265282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99dc074d70 0x7f99dc199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99e1200700 1 -- 172.21.15.78:0/3156265282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99dc074d70 msgr2=0x7f99dc199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99e1200700 1 --2- 172.21.15.78:0/3156265282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99dc074d70 0x7f99dc199710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99e1200700 1 -- 172.21.15.78:0/3156265282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99dc108b00 msgr2=0x7f99dc1a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99e1200700 1 --2- 172.21.15.78:0/3156265282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99dc108b00 0x7f99dc1a0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99e1200700 1 -- 172.21.15.78:0/3156265282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d4009c40 con 0x7f99dc073790 2024-03-14T09:52:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.760+0000 7f99e1c00700 1 --2- 172.21.15.78:0/3156265282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99dc108b00 0x7f99dc1a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.761+0000 7f99e1200700 1 --2- 172.21.15.78:0/3156265282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99dc073790 0x7f99dc1991d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f99d0005910 tx=0x7f99d0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.761+0000 7f99d9600700 1 -- 172.21.15.78:0/3156265282 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d001d070 con 0x7f99dc073790 2024-03-14T09:52:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.761+0000 7f99d9600700 1 -- 172.21.15.78:0/3156265282 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99d0021410 con 0x7f99dc073790 2024-03-14T09:52:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.761+0000 7f99d9600700 1 -- 172.21.15.78:0/3156265282 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d0017650 con 0x7f99dc073790 2024-03-14T09:52:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.761+0000 7f99e2600700 1 -- 172.21.15.78:0/3156265282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d00095f0 con 0x7f99dc073790 2024-03-14T09:52:01.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.761+0000 7f99e2600700 1 -- 172.21.15.78:0/3156265282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99dc1a1490 con 0x7f99dc073790 2024-03-14T09:52:01.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.762+0000 7f99e2600700 1 -- 172.21.15.78:0/3156265282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99dc107910 con 0x7f99dc073790 2024-03-14T09:52:01.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.764+0000 7f99d9600700 1 -- 172.21.15.78:0/3156265282 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f99d0021580 con 0x7f99dc073790 2024-03-14T09:52:01.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.766+0000 7f99d9600700 1 --2- 172.21.15.78:0/3156265282 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99c40a0a90 0x7f99c40a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:01.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.766+0000 7f99d9600700 1 -- 172.21.15.78:0/3156265282 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f99d00ccef0 con 0x7f99dc073790 2024-03-14T09:52:01.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.766+0000 7f99dbe00700 1 --2- 172.21.15.78:0/3156265282 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99c40a0a90 0x7f99c40a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:01.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.767+0000 7f99dbe00700 1 --2- 172.21.15.78:0/3156265282 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99c40a0a90 0x7f99c40a2f50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f99c8009c60 tx=0x7f99c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:01.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:01.770+0000 7f99d9600700 1 -- 172.21.15.78:0/3156265282 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f99d0096d50 con 0x7f99dc073790 2024-03-14T09:52:02.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.008+0000 7f99e2600700 1 -- 172.21.15.78:0/3156265282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99dc067dc0 con 0x7f99dc073790 2024-03-14T09:52:02.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.009+0000 7f99d9600700 1 -- 172.21.15.78:0/3156265282 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+18387 (secure 0 0 0) 0x7f99d009a370 con 0x7f99dc073790 2024-03-14T09:52:02.010 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:02.010 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":75,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:51:27.749782+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:52:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.011+0000 7f99e2600700 1 -- 172.21.15.78:0/3156265282 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99c40a0a90 msgr2=0x7f99c40a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.011+0000 7f99e2600700 1 --2- 172.21.15.78:0/3156265282 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99c40a0a90 0x7f99c40a2f50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f99c8009c60 tx=0x7f99c8009260 comp rx=0 tx=0).stop 2024-03-14T09:52:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.011+0000 7f99e2600700 1 -- 172.21.15.78:0/3156265282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99dc073790 msgr2=0x7f99dc1991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:02.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.011+0000 7f99e2600700 1 --2- 172.21.15.78:0/3156265282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99dc073790 0x7f99dc1991d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f99d0005910 tx=0x7f99d0009580 comp rx=0 tx=0).stop 2024-03-14T09:52:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.011+0000 7f99e2600700 1 -- 172.21.15.78:0/3156265282 shutdown_connections 2024-03-14T09:52:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.011+0000 7f99e2600700 1 --2- 172.21.15.78:0/3156265282 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99c40a0a90 0x7f99c40a2f50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.011+0000 7f99e2600700 1 --2- 172.21.15.78:0/3156265282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f99dc108b00 0x7f99dc1a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.011+0000 7f99e2600700 1 --2- 172.21.15.78:0/3156265282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99dc073790 0x7f99dc1991d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.011+0000 7f99e2600700 1 --2- 172.21.15.78:0/3156265282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f99dc074d70 0x7f99dc199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.011+0000 7f99e2600700 1 -- 172.21.15.78:0/3156265282 >> 172.21.15.78:0/3156265282 conn(0x7f99dc0fc180 msgr2=0x7f99dc104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.012+0000 7f99e2600700 1 -- 172.21.15.78:0/3156265282 shutdown_connections 2024-03-14T09:52:02.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.012+0000 7f99e2600700 1 -- 172.21.15.78:0/3156265282 wait complete. 2024-03-14T09:52:02.031 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T09:48:37.618450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-14T09:52:02.032 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T09:52:02.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.251+0000 7fc8b4400700 1 -- 172.21.15.78:0/3102901743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8ac108c10 msgr2=0x7fc8ac10b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:02.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.251+0000 7fc8b4400700 1 --2- 172.21.15.78:0/3102901743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8ac108c10 0x7fc8ac10b0a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc8a400b0a0 tx=0x7fc8a400b3b0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.251+0000 7fc8b4400700 1 -- 172.21.15.78:0/3102901743 shutdown_connections 2024-03-14T09:52:02.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.251+0000 7fc8b4400700 1 --2- 172.21.15.78:0/3102901743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8ac108c10 0x7fc8ac10b0a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.251+0000 7fc8b4400700 1 --2- 172.21.15.78:0/3102901743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8ac074df0 0x7fc8ac073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.251+0000 7fc8b4400700 1 --2- 172.21.15.78:0/3102901743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8ac073810 0x7fc8ac073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.251+0000 7fc8b4400700 1 -- 172.21.15.78:0/3102901743 >> 172.21.15.78:0/3102901743 conn(0x7fc8ac0fc2c0 msgr2=0x7fc8ac0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:02.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.251+0000 7fc8b4400700 1 -- 172.21.15.78:0/3102901743 shutdown_connections 2024-03-14T09:52:02.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.251+0000 7fc8b4400700 1 -- 172.21.15.78:0/3102901743 wait complete. 2024-03-14T09:52:02.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.252+0000 7fc8b4400700 1 Processor -- start 2024-03-14T09:52:02.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.252+0000 7fc8b4400700 1 -- start start 2024-03-14T09:52:02.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.252+0000 7fc8b4400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8ac073810 0x7fc8ac1993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:02.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.252+0000 7fc8b4400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8ac074df0 0x7fc8ac199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:02.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.252+0000 7fc8b4400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8ac108c10 0x7fc8ac1a0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:02.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.252+0000 7fc8b4400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac1a0f20 con 0x7fc8ac073810 2024-03-14T09:52:02.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.252+0000 7fc8b4400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac1a1060 con 0x7fc8ac108c10 2024-03-14T09:52:02.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.252+0000 7fc8b4400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac1a11f0 con 0x7fc8ac074df0 2024-03-14T09:52:02.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.253+0000 7fc8b1a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8ac073810 0x7fc8ac1993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:02.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.253+0000 7fc8b2400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8ac108c10 0x7fc8ac1a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:02.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.253+0000 7fc8b2400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8ac108c10 0x7fc8ac1a0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48432/0 (socket says 172.21.15.78:48432) 2024-03-14T09:52:02.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.253+0000 7fc8b2400700 1 -- 172.21.15.78:0/2082646680 learned_addr learned my addr 172.21.15.78:0/2082646680 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:02.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.253+0000 7fc8b1000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8ac074df0 0x7fc8ac199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:02.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.253+0000 7fc8b2400700 1 -- 172.21.15.78:0/2082646680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8ac074df0 msgr2=0x7fc8ac199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:02.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.253+0000 7fc8b2400700 1 --2- 172.21.15.78:0/2082646680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8ac074df0 0x7fc8ac199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.253+0000 7fc8b2400700 1 -- 172.21.15.78:0/2082646680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8ac073810 msgr2=0x7fc8ac1993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:02.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.253+0000 7fc8b2400700 1 --2- 172.21.15.78:0/2082646680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8ac073810 0x7fc8ac1993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.253+0000 7fc8b2400700 1 -- 172.21.15.78:0/2082646680 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a4009d00 con 0x7fc8ac108c10 2024-03-14T09:52:02.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.253+0000 7fc8b2400700 1 --2- 172.21.15.78:0/2082646680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8ac108c10 0x7fc8ac1a0950 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4007a60 tx=0x7fc8a4007930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:02.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.254+0000 7fc8aa000700 1 -- 172.21.15.78:0/2082646680 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a400e040 con 0x7fc8ac108c10 2024-03-14T09:52:02.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.254+0000 7fc8aa000700 1 -- 172.21.15.78:0/2082646680 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8a4004c50 con 0x7fc8ac108c10 2024-03-14T09:52:02.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.254+0000 7fc8b4400700 1 -- 172.21.15.78:0/2082646680 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ac1a1470 con 0x7fc8ac108c10 2024-03-14T09:52:02.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.254+0000 7fc8aa000700 1 -- 172.21.15.78:0/2082646680 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a401b6c0 con 0x7fc8ac108c10 2024-03-14T09:52:02.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.254+0000 7fc8b4400700 1 -- 172.21.15.78:0/2082646680 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ac1a1890 con 0x7fc8ac108c10 2024-03-14T09:52:02.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.255+0000 7fc8b4400700 1 -- 172.21.15.78:0/2082646680 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ac1079d0 con 0x7fc8ac108c10 2024-03-14T09:52:02.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.257+0000 7fc8aa000700 1 -- 172.21.15.78:0/2082646680 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fc8a401d470 con 0x7fc8ac108c10 2024-03-14T09:52:02.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.258+0000 7fc8aa000700 1 --2- 172.21.15.78:0/2082646680 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8940a0b10 0x7fc8940a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:02.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.258+0000 7fc8aa000700 1 -- 172.21.15.78:0/2082646680 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc8a40ccda0 con 0x7fc8ac108c10 2024-03-14T09:52:02.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.258+0000 7fc8b1a00700 1 --2- 172.21.15.78:0/2082646680 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8940a0b10 0x7fc8940a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:02.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.259+0000 7fc8b1a00700 1 --2- 172.21.15.78:0/2082646680 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8940a0b10 0x7fc8940a2fd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009c60 tx=0x7fc8a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:02.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.262+0000 7fc8aa000700 1 -- 172.21.15.78:0/2082646680 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc8a4096ca0 con 0x7fc8ac108c10 2024-03-14T09:52:02.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:02 smithi078 ceph-mon[38143]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:52:02.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:02 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1209797800' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:52:02.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:02 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3156265282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:02.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:02.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:02 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:02 smithi152 ceph-mon[48279]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:52:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:02 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1209797800' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:52:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:02 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3156265282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:02 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:02 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:02.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.517+0000 7fc8b4400700 1 -- 172.21.15.78:0/2082646680 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc8ac10ede0 con 0x7fc8ac108c10 2024-03-14T09:52:02.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.517+0000 7fc8aa000700 1 -- 172.21.15.78:0/2082646680 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc8a409a2c0 con 0x7fc8ac108c10 2024-03-14T09:52:02.518 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-14T09:52:02.519 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:02 smithi132 ceph-mon[47959]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:52:02.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:02 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1209797800' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:52:02.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:02 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3156265282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:02.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:02 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:02.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:02 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.520+0000 7fc8b4400700 1 -- 172.21.15.78:0/2082646680 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8940a0b10 msgr2=0x7fc8940a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.520+0000 7fc8b4400700 1 --2- 172.21.15.78:0/2082646680 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8940a0b10 0x7fc8940a2fd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009c60 tx=0x7fc8a0009260 comp rx=0 tx=0).stop 2024-03-14T09:52:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.520+0000 7fc8b4400700 1 -- 172.21.15.78:0/2082646680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8ac108c10 msgr2=0x7fc8ac1a0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.520+0000 7fc8b4400700 1 --2- 172.21.15.78:0/2082646680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8ac108c10 0x7fc8ac1a0950 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4007a60 tx=0x7fc8a4007930 comp rx=0 tx=0).stop 2024-03-14T09:52:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.520+0000 7fc8b4400700 1 -- 172.21.15.78:0/2082646680 shutdown_connections 2024-03-14T09:52:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.520+0000 7fc8b4400700 1 --2- 172.21.15.78:0/2082646680 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8940a0b10 0x7fc8940a2fd0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.520+0000 7fc8b4400700 1 --2- 172.21.15.78:0/2082646680 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8ac108c10 0x7fc8ac1a0950 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.520+0000 7fc8b4400700 1 --2- 172.21.15.78:0/2082646680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8ac073810 0x7fc8ac1993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.520+0000 7fc8b4400700 1 --2- 172.21.15.78:0/2082646680 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8ac074df0 0x7fc8ac199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.520+0000 7fc8b4400700 1 -- 172.21.15.78:0/2082646680 >> 172.21.15.78:0/2082646680 conn(0x7fc8ac0fc2c0 msgr2=0x7fc8ac104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.520+0000 7fc8b4400700 1 -- 172.21.15.78:0/2082646680 shutdown_connections 2024-03-14T09:52:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.520+0000 7fc8b4400700 1 -- 172.21.15.78:0/2082646680 wait complete. 2024-03-14T09:52:02.542 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-14T09:52:02.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.743+0000 7ff772a00700 1 -- 172.21.15.78:0/2725636135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff76c074d70 msgr2=0x7ff76c0731c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:02.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.743+0000 7ff772a00700 1 --2- 172.21.15.78:0/2725636135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff76c074d70 0x7ff76c0731c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff75c0098f0 tx=0x7ff75c009c00 comp rx=0 tx=0).stop 2024-03-14T09:52:02.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.743+0000 7ff772a00700 1 -- 172.21.15.78:0/2725636135 shutdown_connections 2024-03-14T09:52:02.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.743+0000 7ff772a00700 1 --2- 172.21.15.78:0/2725636135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff76c074d70 0x7ff76c0731c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.743+0000 7ff772a00700 1 --2- 172.21.15.78:0/2725636135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff76c108b60 0x7ff76c10aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.743+0000 7ff772a00700 1 --2- 172.21.15.78:0/2725636135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff76c073790 0x7ff76c073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.743+0000 7ff772a00700 1 -- 172.21.15.78:0/2725636135 >> 172.21.15.78:0/2725636135 conn(0x7ff76c0fc1e0 msgr2=0x7ff76c0fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:02.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.744+0000 7ff772a00700 1 -- 172.21.15.78:0/2725636135 shutdown_connections 2024-03-14T09:52:02.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.744+0000 7ff772a00700 1 -- 172.21.15.78:0/2725636135 wait complete. 2024-03-14T09:52:02.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.745+0000 7ff772a00700 1 Processor -- start 2024-03-14T09:52:02.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.745+0000 7ff772a00700 1 -- start start 2024-03-14T09:52:02.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.745+0000 7ff772a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff76c073790 0x7ff76c199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:02.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.745+0000 7ff772a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff76c074d70 0x7ff76c1997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:02.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.745+0000 7ff772a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff76c108b60 0x7ff76c1a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:02.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.745+0000 7ff772a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c1a0da0 con 0x7ff76c074d70 2024-03-14T09:52:02.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.745+0000 7ff772a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c1a0ee0 con 0x7ff76c108b60 2024-03-14T09:52:02.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.745+0000 7ff772a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c1a1070 con 0x7ff76c073790 2024-03-14T09:52:02.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.745+0000 7ff76b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff76c074d70 0x7ff76c1997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:02.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.746+0000 7ff76b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff76c074d70 0x7ff76c1997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39056/0 (socket says 172.21.15.78:39056) 2024-03-14T09:52:02.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.746+0000 7ff76b400700 1 -- 172.21.15.78:0/1420434886 learned_addr learned my addr 172.21.15.78:0/1420434886 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:02.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.746+0000 7ff76be00700 1 --2- 172.21.15.78:0/1420434886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff76c073790 0x7ff76c199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:02.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.746+0000 7ff770a00700 1 --2- 172.21.15.78:0/1420434886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff76c108b60 0x7ff76c1a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:02.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.746+0000 7ff76be00700 1 -- 172.21.15.78:0/1420434886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff76c108b60 msgr2=0x7ff76c1a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:02.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.746+0000 7ff76be00700 1 --2- 172.21.15.78:0/1420434886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff76c108b60 0x7ff76c1a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.746+0000 7ff76be00700 1 -- 172.21.15.78:0/1420434886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff76c074d70 msgr2=0x7ff76c1997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:02.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.746+0000 7ff76be00700 1 --2- 172.21.15.78:0/1420434886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff76c074d70 0x7ff76c1997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:02.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.746+0000 7ff76be00700 1 -- 172.21.15.78:0/1420434886 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7580095f0 con 0x7ff76c073790 2024-03-14T09:52:02.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.746+0000 7ff770a00700 1 --2- 172.21.15.78:0/1420434886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff76c108b60 0x7ff76c1a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:52:02.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.746+0000 7ff76be00700 1 --2- 172.21.15.78:0/1420434886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff76c073790 0x7ff76c199270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009230 tx=0x7ff75c00b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:02.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.746+0000 7ff768c00700 1 -- 172.21.15.78:0/1420434886 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c01c070 con 0x7ff76c073790 2024-03-14T09:52:02.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.746+0000 7ff768c00700 1 -- 172.21.15.78:0/1420434886 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff75c00f7d0 con 0x7ff76c073790 2024-03-14T09:52:02.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.746+0000 7ff772a00700 1 -- 172.21.15.78:0/1420434886 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff75c009520 con 0x7ff76c073790 2024-03-14T09:52:02.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.746+0000 7ff768c00700 1 -- 172.21.15.78:0/1420434886 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c003990 con 0x7ff76c073790 2024-03-14T09:52:02.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.747+0000 7ff772a00700 1 -- 172.21.15.78:0/1420434886 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff76c1a1580 con 0x7ff76c073790 2024-03-14T09:52:02.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.748+0000 7ff772a00700 1 -- 172.21.15.78:0/1420434886 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff76c107970 con 0x7ff76c073790 2024-03-14T09:52:02.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.749+0000 7ff768c00700 1 -- 172.21.15.78:0/1420434886 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7ff75c020030 con 0x7ff76c073790 2024-03-14T09:52:02.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.750+0000 7ff768c00700 1 --2- 172.21.15.78:0/1420434886 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff7500a0a90 0x7ff7500a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:02.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.751+0000 7ff768c00700 1 -- 172.21.15.78:0/1420434886 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff75c0ccac0 con 0x7ff76c073790 2024-03-14T09:52:02.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.751+0000 7ff76b400700 1 --2- 172.21.15.78:0/1420434886 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff7500a0a90 0x7ff7500a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:02.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.751+0000 7ff76b400700 1 --2- 172.21.15.78:0/1420434886 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff7500a0a90 0x7ff7500a2f50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff758005d10 tx=0x7ff758009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:02.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.755+0000 7ff768c00700 1 -- 172.21.15.78:0/1420434886 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff75c096920 con 0x7ff76c073790 2024-03-14T09:52:02.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.999+0000 7ff772a00700 1 -- 172.21.15.78:0/1420434886 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff76c0679e0 con 0x7ff76c073790 2024-03-14T09:52:03.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:02.999+0000 7ff768c00700 1 -- 172.21.15.78:0/1420434886 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v75) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff75c099f40 con 0x7ff76c073790 2024-03-14T09:52:03.000 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:03.000 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2024-03-14T09:52:03.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T09:52:03.001 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2024-03-14T09:52:03.001 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2024-03-14T09:52:03.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:52:03.001 INFO:teuthology.orchestra.run.smithi078.stdout:] 2024-03-14T09:52:03.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.002+0000 7ff772a00700 1 -- 172.21.15.78:0/1420434886 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff7500a0a90 msgr2=0x7ff7500a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:03.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.002+0000 7ff772a00700 1 --2- 172.21.15.78:0/1420434886 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff7500a0a90 0x7ff7500a2f50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff758005d10 tx=0x7ff758009310 comp rx=0 tx=0).stop 2024-03-14T09:52:03.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.002+0000 7ff772a00700 1 -- 172.21.15.78:0/1420434886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff76c073790 msgr2=0x7ff76c199270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:03.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.002+0000 7ff772a00700 1 --2- 172.21.15.78:0/1420434886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff76c073790 0x7ff76c199270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009230 tx=0x7ff75c00b790 comp rx=0 tx=0).stop 2024-03-14T09:52:03.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.002+0000 7ff772a00700 1 -- 172.21.15.78:0/1420434886 shutdown_connections 2024-03-14T09:52:03.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.002+0000 7ff772a00700 1 --2- 172.21.15.78:0/1420434886 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff7500a0a90 0x7ff7500a2f50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:03.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.002+0000 7ff772a00700 1 --2- 172.21.15.78:0/1420434886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff76c108b60 0x7ff76c1a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:03.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.002+0000 7ff772a00700 1 --2- 172.21.15.78:0/1420434886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff76c073790 0x7ff76c199270 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:03.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.002+0000 7ff772a00700 1 --2- 172.21.15.78:0/1420434886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff76c074d70 0x7ff76c1997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:03.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.002+0000 7ff772a00700 1 -- 172.21.15.78:0/1420434886 >> 172.21.15.78:0/1420434886 conn(0x7ff76c0fc1e0 msgr2=0x7ff76c104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:03.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.002+0000 7ff772a00700 1 -- 172.21.15.78:0/1420434886 shutdown_connections 2024-03-14T09:52:03.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.002+0000 7ff772a00700 1 -- 172.21.15.78:0/1420434886 wait complete. 2024-03-14T09:52:03.023 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-14T09:52:03.023 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-14T09:52:03.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.220+0000 7f9c5a600700 1 -- 172.21.15.78:0/2555166208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c54073810 msgr2=0x7f9c54073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:03.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.220+0000 7f9c5a600700 1 --2- 172.21.15.78:0/2555166208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c54073810 0x7f9c54073c90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9c40009990 tx=0x7f9c40009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:03.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.221+0000 7f9c5a600700 1 -- 172.21.15.78:0/2555166208 shutdown_connections 2024-03-14T09:52:03.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.221+0000 7f9c5a600700 1 --2- 172.21.15.78:0/2555166208 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c54074df0 0x7f9c54073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:03.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.221+0000 7f9c5a600700 1 --2- 172.21.15.78:0/2555166208 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c54108c10 0x7f9c5410b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:03.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.221+0000 7f9c5a600700 1 --2- 172.21.15.78:0/2555166208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c54073810 0x7f9c54073c90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:03.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.221+0000 7f9c5a600700 1 -- 172.21.15.78:0/2555166208 >> 172.21.15.78:0/2555166208 conn(0x7f9c540fc2c0 msgr2=0x7f9c540fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:03.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.221+0000 7f9c5a600700 1 -- 172.21.15.78:0/2555166208 shutdown_connections 2024-03-14T09:52:03.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.221+0000 7f9c5a600700 1 -- 172.21.15.78:0/2555166208 wait complete. 2024-03-14T09:52:03.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.222+0000 7f9c5a600700 1 Processor -- start 2024-03-14T09:52:03.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.222+0000 7f9c5a600700 1 -- start start 2024-03-14T09:52:03.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.222+0000 7f9c5a600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c54073810 0x7f9c54199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:03.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.222+0000 7f9c5a600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c54074df0 0x7f9c54199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:03.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.222+0000 7f9c5a600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c54108c10 0x7f9c541a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:03.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.222+0000 7f9c5a600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c541a0e80 con 0x7f9c54108c10 2024-03-14T09:52:03.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.222+0000 7f9c5a600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c541a0fc0 con 0x7f9c54074df0 2024-03-14T09:52:03.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.222+0000 7f9c5a600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c541a1150 con 0x7f9c54073810 2024-03-14T09:52:03.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.222+0000 7f9c53e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c54108c10 0x7f9c541a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:03.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.222+0000 7f9c53e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c54108c10 0x7f9c541a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39074/0 (socket says 172.21.15.78:39074) 2024-03-14T09:52:03.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.222+0000 7f9c52a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c54074df0 0x7f9c54199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:03.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.222+0000 7f9c53e00700 1 -- 172.21.15.78:0/3368250863 learned_addr learned my addr 172.21.15.78:0/3368250863 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:03.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.222+0000 7f9c53400700 1 --2- 172.21.15.78:0/3368250863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c54073810 0x7f9c54199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:03.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.223+0000 7f9c53e00700 1 -- 172.21.15.78:0/3368250863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c54073810 msgr2=0x7f9c54199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:03.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.223+0000 7f9c53e00700 1 --2- 172.21.15.78:0/3368250863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c54073810 0x7f9c54199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:03.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.223+0000 7f9c53e00700 1 -- 172.21.15.78:0/3368250863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c54074df0 msgr2=0x7f9c54199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:03.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.223+0000 7f9c53e00700 1 --2- 172.21.15.78:0/3368250863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c54074df0 0x7f9c54199890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:03.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.223+0000 7f9c53e00700 1 -- 172.21.15.78:0/3368250863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c400095f0 con 0x7f9c54108c10 2024-03-14T09:52:03.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.223+0000 7f9c53e00700 1 --2- 172.21.15.78:0/3368250863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c54108c10 0x7f9c541a08b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00ef30 tx=0x7f9c4c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:03.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.223+0000 7f9c4be00700 1 -- 172.21.15.78:0/3368250863 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c018070 con 0x7f9c54108c10 2024-03-14T09:52:03.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.223+0000 7f9c5a600700 1 -- 172.21.15.78:0/3368250863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c541a13d0 con 0x7f9c54108c10 2024-03-14T09:52:03.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.223+0000 7f9c4be00700 1 -- 172.21.15.78:0/3368250863 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9c4c00f040 con 0x7f9c54108c10 2024-03-14T09:52:03.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.223+0000 7f9c4be00700 1 -- 172.21.15.78:0/3368250863 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c00cd00 con 0x7f9c54108c10 2024-03-14T09:52:03.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.223+0000 7f9c5a600700 1 -- 172.21.15.78:0/3368250863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c541a1930 con 0x7f9c54108c10 2024-03-14T09:52:03.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.224+0000 7f9c5a600700 1 -- 172.21.15.78:0/3368250863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c541079d0 con 0x7f9c54108c10 2024-03-14T09:52:03.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.226+0000 7f9c4be00700 1 -- 172.21.15.78:0/3368250863 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f9c4c003a00 con 0x7f9c54108c10 2024-03-14T09:52:03.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.227+0000 7f9c4be00700 1 --2- 172.21.15.78:0/3368250863 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c380a0a90 0x7f9c380a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:03.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.228+0000 7f9c4be00700 1 -- 172.21.15.78:0/3368250863 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9c4c0cb6c0 con 0x7f9c54108c10 2024-03-14T09:52:03.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.228+0000 7f9c53400700 1 --2- 172.21.15.78:0/3368250863 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c380a0a90 0x7f9c380a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:03.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.228+0000 7f9c53400700 1 --2- 172.21.15.78:0/3368250863 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c380a0a90 0x7f9c380a2f50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009c60 tx=0x7f9c44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:03.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.232+0000 7f9c4be00700 1 -- 172.21.15.78:0/3368250863 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9c4c0955c0 con 0x7f9c54108c10 2024-03-14T09:52:03.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:03 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:03.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:03 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:03.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:03 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2082646680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:52:03.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:03 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:03.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:03 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:03.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:03 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:52:03.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:03 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:52:03.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:03 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:03.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:03 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1420434886' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-14T09:52:03.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:03.440+0000 7f9c5a600700 1 -- 172.21.15.78:0/3368250863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9c5410ebb0 con 0x7f9c54108c10 2024-03-14T09:52:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:03 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:03 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:03 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2082646680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:52:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:03 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:03 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:03 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:52:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:03 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:52:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:03 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:03 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1420434886' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-14T09:52:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:03 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:03 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:03 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2082646680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:52:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:03 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:03 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:03 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:52:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:03 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:52:03.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:03 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:52:03.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:03 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1420434886' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-14T09:52:04.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.203+0000 7f9c4be00700 1 -- 172.21.15.78:0/3368250863 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v76) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9c4c098be0 con 0x7f9c54108c10 2024-03-14T09:52:04.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.209+0000 7f9c5a600700 1 -- 172.21.15.78:0/3368250863 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c380a0a90 msgr2=0x7f9c380a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:04.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.209+0000 7f9c5a600700 1 --2- 172.21.15.78:0/3368250863 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c380a0a90 0x7f9c380a2f50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009c60 tx=0x7f9c44009260 comp rx=0 tx=0).stop 2024-03-14T09:52:04.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.209+0000 7f9c5a600700 1 -- 172.21.15.78:0/3368250863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c54108c10 msgr2=0x7f9c541a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:04.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.209+0000 7f9c5a600700 1 --2- 172.21.15.78:0/3368250863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c54108c10 0x7f9c541a08b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00ef30 tx=0x7f9c4c00c3a0 comp rx=0 tx=0).stop 2024-03-14T09:52:04.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.209+0000 7f9c5a600700 1 -- 172.21.15.78:0/3368250863 shutdown_connections 2024-03-14T09:52:04.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.209+0000 7f9c5a600700 1 --2- 172.21.15.78:0/3368250863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c54108c10 0x7f9c541a08b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:04.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.209+0000 7f9c5a600700 1 --2- 172.21.15.78:0/3368250863 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c54073810 0x7f9c54199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:04.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.209+0000 7f9c5a600700 1 --2- 172.21.15.78:0/3368250863 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c380a0a90 0x7f9c380a2f50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:04.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.209+0000 7f9c5a600700 1 --2- 172.21.15.78:0/3368250863 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c54074df0 0x7f9c54199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:04.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.209+0000 7f9c5a600700 1 -- 172.21.15.78:0/3368250863 >> 172.21.15.78:0/3368250863 conn(0x7f9c540fc2c0 msgr2=0x7f9c54104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:04.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.210+0000 7f9c5a600700 1 -- 172.21.15.78:0/3368250863 shutdown_connections 2024-03-14T09:52:04.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.210+0000 7f9c5a600700 1 -- 172.21.15.78:0/3368250863 wait complete. 2024-03-14T09:52:04.212 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2024-03-14T09:52:04.230 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-14T09:52:04.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.429+0000 7f5294e00700 1 -- 172.21.15.78:0/1420603011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5290073810 msgr2=0x7f5290073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:04.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.429+0000 7f5294e00700 1 --2- 172.21.15.78:0/1420603011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5290073810 0x7f5290073c90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f527c009990 tx=0x7f527c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:04.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.429+0000 7f5294e00700 1 -- 172.21.15.78:0/1420603011 shutdown_connections 2024-03-14T09:52:04.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.429+0000 7f5294e00700 1 --2- 172.21.15.78:0/1420603011 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5290108c00 0x7f529010b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:04.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.429+0000 7f5294e00700 1 --2- 172.21.15.78:0/1420603011 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5290074df0 0x7f5290073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:04.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.429+0000 7f5294e00700 1 --2- 172.21.15.78:0/1420603011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5290073810 0x7f5290073c90 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:04.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.429+0000 7f5294e00700 1 -- 172.21.15.78:0/1420603011 >> 172.21.15.78:0/1420603011 conn(0x7f52900fc2b0 msgr2=0x7f52900fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:04.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.429+0000 7f5294e00700 1 -- 172.21.15.78:0/1420603011 shutdown_connections 2024-03-14T09:52:04.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.429+0000 7f5294e00700 1 -- 172.21.15.78:0/1420603011 wait complete. 2024-03-14T09:52:04.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.430+0000 7f5294e00700 1 Processor -- start 2024-03-14T09:52:04.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.430+0000 7f5294e00700 1 -- start start 2024-03-14T09:52:04.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.431+0000 7f5294e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5290073810 0x7f5290199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:04.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.431+0000 7f5294e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5290074df0 0x7f52901998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:04.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.431+0000 7f5294e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5290108c00 0x7f52901a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:04.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.431+0000 7f5294e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52901a0e70 con 0x7f5290073810 2024-03-14T09:52:04.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.431+0000 7f5294e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52901a0fb0 con 0x7f5290108c00 2024-03-14T09:52:04.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.431+0000 7f5294e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52901a1140 con 0x7f5290074df0 2024-03-14T09:52:04.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.431+0000 7f528de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5290073810 0x7f5290199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:04.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.431+0000 7f528de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5290073810 0x7f5290199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36912/0 (socket says 172.21.15.78:36912) 2024-03-14T09:52:04.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.431+0000 7f528de00700 1 -- 172.21.15.78:0/701537141 learned_addr learned my addr 172.21.15.78:0/701537141 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:04.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.431+0000 7f528d400700 1 --2- 172.21.15.78:0/701537141 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5290074df0 0x7f52901998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:04.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.431+0000 7f528e800700 1 --2- 172.21.15.78:0/701537141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5290108c00 0x7f52901a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:04.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.431+0000 7f528de00700 1 -- 172.21.15.78:0/701537141 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5290074df0 msgr2=0x7f52901998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:04.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.431+0000 7f528de00700 1 --2- 172.21.15.78:0/701537141 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5290074df0 0x7f52901998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:04.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.431+0000 7f528de00700 1 -- 172.21.15.78:0/701537141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5290108c00 msgr2=0x7f52901a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:04.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.431+0000 7f528de00700 1 --2- 172.21.15.78:0/701537141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5290108c00 0x7f52901a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:04.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.432+0000 7f528de00700 1 -- 172.21.15.78:0/701537141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527c0095f0 con 0x7f5290073810 2024-03-14T09:52:04.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.432+0000 7f528de00700 1 --2- 172.21.15.78:0/701537141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5290073810 0x7f5290199390 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f528000d950 tx=0x7f528000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:04.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.432+0000 7f5286a00700 1 -- 172.21.15.78:0/701537141 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5280009940 con 0x7f5290073810 2024-03-14T09:52:04.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.432+0000 7f5286a00700 1 -- 172.21.15.78:0/701537141 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5280010460 con 0x7f5290073810 2024-03-14T09:52:04.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.432+0000 7f5294e00700 1 -- 172.21.15.78:0/701537141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52901a1420 con 0x7f5290073810 2024-03-14T09:52:04.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.432+0000 7f5286a00700 1 -- 172.21.15.78:0/701537141 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528000f620 con 0x7f5290073810 2024-03-14T09:52:04.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.432+0000 7f5294e00700 1 -- 172.21.15.78:0/701537141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52901a1930 con 0x7f5290073810 2024-03-14T09:52:04.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.433+0000 7f5294e00700 1 -- 172.21.15.78:0/701537141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52901079c0 con 0x7f5290073810 2024-03-14T09:52:04.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.435+0000 7f5286a00700 1 -- 172.21.15.78:0/701537141 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f52800108d0 con 0x7f5290073810 2024-03-14T09:52:04.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.436+0000 7f5286a00700 1 --2- 172.21.15.78:0/701537141 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f52740a0a90 0x7f52740a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:04.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.437+0000 7f5286a00700 1 -- 172.21.15.78:0/701537141 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f52800cbc40 con 0x7f5290073810 2024-03-14T09:52:04.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.437+0000 7f528d400700 1 --2- 172.21.15.78:0/701537141 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f52740a0a90 0x7f52740a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:04.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.437+0000 7f528d400700 1 --2- 172.21.15.78:0/701537141 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f52740a0a90 0x7f52740a2f50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f527c00b5c0 tx=0x7f527c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:04.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.439+0000 7f5286a00700 1 -- 172.21.15.78:0/701537141 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f52800958a0 con 0x7f5290073810 2024-03-14T09:52:04.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:04 smithi152 ceph-mon[48279]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:52:04.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:04 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3368250863' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-14T09:52:04.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:04 smithi132 ceph-mon[47959]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:52:04.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:04 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3368250863' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-14T09:52:04.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:04 smithi078 ceph-mon[38143]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:52:04.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:04 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3368250863' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-14T09:52:04.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:04.650+0000 7f5294e00700 1 -- 172.21.15.78:0/701537141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f5290067ac0 con 0x7f5290073810 2024-03-14T09:52:05.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.213+0000 7f5286a00700 1 -- 172.21.15.78:0/701537141 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v77) v1 ==== 171+0+0 (secure 0 0 0) 0x7f5280095030 con 0x7f5290073810 2024-03-14T09:52:05.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.220+0000 7f5294e00700 1 -- 172.21.15.78:0/701537141 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f52740a0a90 msgr2=0x7f52740a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:05.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.220+0000 7f5294e00700 1 --2- 172.21.15.78:0/701537141 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f52740a0a90 0x7f52740a2f50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f527c00b5c0 tx=0x7f527c005fd0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.220+0000 7f5294e00700 1 -- 172.21.15.78:0/701537141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5290073810 msgr2=0x7f5290199390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:05.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.220+0000 7f5294e00700 1 --2- 172.21.15.78:0/701537141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5290073810 0x7f5290199390 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f528000d950 tx=0x7f528000dc60 comp rx=0 tx=0).stop 2024-03-14T09:52:05.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.222+0000 7f5294e00700 1 -- 172.21.15.78:0/701537141 shutdown_connections 2024-03-14T09:52:05.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.222+0000 7f5294e00700 1 --2- 172.21.15.78:0/701537141 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f52740a0a90 0x7f52740a2f50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.222+0000 7f5294e00700 1 --2- 172.21.15.78:0/701537141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5290108c00 0x7f52901a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.222+0000 7f5294e00700 1 --2- 172.21.15.78:0/701537141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5290073810 0x7f5290199390 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.222+0000 7f5294e00700 1 --2- 172.21.15.78:0/701537141 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5290074df0 0x7f52901998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.222+0000 7f5294e00700 1 -- 172.21.15.78:0/701537141 >> 172.21.15.78:0/701537141 conn(0x7f52900fc2b0 msgr2=0x7f5290104b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:05.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.223+0000 7f5294e00700 1 -- 172.21.15.78:0/701537141 shutdown_connections 2024-03-14T09:52:05.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.223+0000 7f5294e00700 1 -- 172.21.15.78:0/701537141 wait complete. 2024-03-14T09:52:05.227 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2024-03-14T09:52:05.250 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-14T09:52:05.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.445+0000 7f7e07200700 1 -- 172.21.15.78:0/628860762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e00100d40 msgr2=0x7f7e00101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:05.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.445+0000 7f7e07200700 1 --2- 172.21.15.78:0/628860762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e00100d40 0x7f7e00101160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7df00099c0 tx=0x7f7df0009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.445+0000 7f7e07200700 1 -- 172.21.15.78:0/628860762 shutdown_connections 2024-03-14T09:52:05.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.445+0000 7f7e07200700 1 --2- 172.21.15.78:0/628860762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e00100d40 0x7f7e00101160 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.445+0000 7f7e07200700 1 --2- 172.21.15.78:0/628860762 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e001028d0 0x7f7e00109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.445+0000 7f7e07200700 1 --2- 172.21.15.78:0/628860762 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7e00101f30 0x7f7e00102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.445+0000 7f7e07200700 1 -- 172.21.15.78:0/628860762 >> 172.21.15.78:0/628860762 conn(0x7f7e000fc2b0 msgr2=0x7f7e000fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:05.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.445+0000 7f7e07200700 1 -- 172.21.15.78:0/628860762 shutdown_connections 2024-03-14T09:52:05.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.445+0000 7f7e07200700 1 -- 172.21.15.78:0/628860762 wait complete. 2024-03-14T09:52:05.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.446+0000 7f7e07200700 1 Processor -- start 2024-03-14T09:52:05.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.446+0000 7f7e07200700 1 -- start start 2024-03-14T09:52:05.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.446+0000 7f7e07200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e00100d40 0x7f7e00074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:05.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.446+0000 7f7e07200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7e00101f30 0x7f7e000731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:05.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.446+0000 7f7e07200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e001028d0 0x7f7e000736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:05.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.446+0000 7f7e07200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e00073e80 con 0x7f7e00100d40 2024-03-14T09:52:05.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.446+0000 7f7e07200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e00073ff0 con 0x7f7e001028d0 2024-03-14T09:52:05.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.446+0000 7f7e07200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e001a11a0 con 0x7f7e00101f30 2024-03-14T09:52:05.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.447+0000 7f7dffe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e00100d40 0x7f7e00074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:05.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.447+0000 7f7dffe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e00100d40 0x7f7e00074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36920/0 (socket says 172.21.15.78:36920) 2024-03-14T09:52:05.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.447+0000 7f7dffe00700 1 -- 172.21.15.78:0/1484014752 learned_addr learned my addr 172.21.15.78:0/1484014752 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:05.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.447+0000 7f7e05200700 1 --2- 172.21.15.78:0/1484014752 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e001028d0 0x7f7e000736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:05.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.447+0000 7f7dff400700 1 --2- 172.21.15.78:0/1484014752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7e00101f30 0x7f7e000731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:05.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.447+0000 7f7dffe00700 1 -- 172.21.15.78:0/1484014752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7e00101f30 msgr2=0x7f7e000731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:05.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.447+0000 7f7dffe00700 1 --2- 172.21.15.78:0/1484014752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7e00101f30 0x7f7e000731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.447+0000 7f7dffe00700 1 -- 172.21.15.78:0/1484014752 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e001028d0 msgr2=0x7f7e000736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:05.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.447+0000 7f7dffe00700 1 --2- 172.21.15.78:0/1484014752 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e001028d0 0x7f7e000736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.447+0000 7f7dffe00700 1 -- 172.21.15.78:0/1484014752 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df00095f0 con 0x7f7e00100d40 2024-03-14T09:52:05.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.447+0000 7f7dffe00700 1 --2- 172.21.15.78:0/1484014752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e00100d40 0x7f7e00074b10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7df0003730 tx=0x7f7df000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:05.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.447+0000 7f7dfcc00700 1 -- 172.21.15.78:0/1484014752 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df001d070 con 0x7f7e00100d40 2024-03-14T09:52:05.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.447+0000 7f7dfcc00700 1 -- 172.21.15.78:0/1484014752 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7df0005050 con 0x7f7e00100d40 2024-03-14T09:52:05.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.447+0000 7f7dfcc00700 1 -- 172.21.15.78:0/1484014752 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df0004af0 con 0x7f7e00100d40 2024-03-14T09:52:05.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.447+0000 7f7e07200700 1 -- 172.21.15.78:0/1484014752 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e001a13d0 con 0x7f7e00100d40 2024-03-14T09:52:05.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.448+0000 7f7e07200700 1 -- 172.21.15.78:0/1484014752 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e001a1740 con 0x7f7e00100d40 2024-03-14T09:52:05.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.449+0000 7f7e07200700 1 -- 172.21.15.78:0/1484014752 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e00108210 con 0x7f7e00100d40 2024-03-14T09:52:05.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.450+0000 7f7dfcc00700 1 -- 172.21.15.78:0/1484014752 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f7df00051c0 con 0x7f7e00100d40 2024-03-14T09:52:05.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.453+0000 7f7dfcc00700 1 --2- 172.21.15.78:0/1484014752 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7de40a0a90 0x7f7de40a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:05.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.453+0000 7f7dfcc00700 1 -- 172.21.15.78:0/1484014752 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f7df00cd0e0 con 0x7f7e00100d40 2024-03-14T09:52:05.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.453+0000 7f7dff400700 1 --2- 172.21.15.78:0/1484014752 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7de40a0a90 0x7f7de40a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:05.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.453+0000 7f7dff400700 1 --2- 172.21.15.78:0/1484014752 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7de40a0a90 0x7f7de40a2f50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009bf0 tx=0x7f7dec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:05.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.456+0000 7f7dfcc00700 1 -- 172.21.15.78:0/1484014752 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7df0096c50 con 0x7f7e00100d40 2024-03-14T09:52:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:05 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3368250863' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-14T09:52:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:05 smithi152 ceph-mon[48279]: osdmap e76: 12 total, 12 up, 12 in 2024-03-14T09:52:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:05 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/701537141' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-14T09:52:05.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:05 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3368250863' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-14T09:52:05.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:05 smithi132 ceph-mon[47959]: osdmap e76: 12 total, 12 up, 12 in 2024-03-14T09:52:05.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:05 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/701537141' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-14T09:52:05.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:05 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3368250863' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-14T09:52:05.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:05 smithi078 ceph-mon[38143]: osdmap e76: 12 total, 12 up, 12 in 2024-03-14T09:52:05.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:05 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/701537141' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-14T09:52:05.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.706+0000 7f7e07200700 1 -- 172.21.15.78:0/1484014752 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7e0010cb40 con 0x7f7e00100d40 2024-03-14T09:52:05.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.714+0000 7f7dfcc00700 1 -- 172.21.15.78:0/1484014752 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7df009a270 con 0x7f7e00100d40 2024-03-14T09:52:05.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.718+0000 7f7e07200700 1 -- 172.21.15.78:0/1484014752 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7de40a0a90 msgr2=0x7f7de40a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:05.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.718+0000 7f7e07200700 1 --2- 172.21.15.78:0/1484014752 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7de40a0a90 0x7f7de40a2f50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009bf0 tx=0x7f7dec009260 comp rx=0 tx=0).stop 2024-03-14T09:52:05.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.718+0000 7f7e07200700 1 -- 172.21.15.78:0/1484014752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e00100d40 msgr2=0x7f7e00074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:05.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.718+0000 7f7e07200700 1 --2- 172.21.15.78:0/1484014752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e00100d40 0x7f7e00074b10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7df0003730 tx=0x7f7df000f690 comp rx=0 tx=0).stop 2024-03-14T09:52:05.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.718+0000 7f7e07200700 1 -- 172.21.15.78:0/1484014752 shutdown_connections 2024-03-14T09:52:05.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.718+0000 7f7e07200700 1 --2- 172.21.15.78:0/1484014752 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7de40a0a90 0x7f7de40a2f50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.718+0000 7f7e07200700 1 --2- 172.21.15.78:0/1484014752 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e001028d0 0x7f7e000736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.718+0000 7f7e07200700 1 --2- 172.21.15.78:0/1484014752 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e00100d40 0x7f7e00074b10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.718+0000 7f7e07200700 1 --2- 172.21.15.78:0/1484014752 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7e00101f30 0x7f7e000731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.718+0000 7f7e07200700 1 -- 172.21.15.78:0/1484014752 >> 172.21.15.78:0/1484014752 conn(0x7f7e000fc2b0 msgr2=0x7f7e00105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:05.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.719+0000 7f7e07200700 1 -- 172.21.15.78:0/1484014752 shutdown_connections 2024-03-14T09:52:05.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.719+0000 7f7e07200700 1 -- 172.21.15.78:0/1484014752 wait complete. 2024-03-14T09:52:05.725 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 2 and data pool 3 2024-03-14T09:52:05.754 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-14T09:52:05.754 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-14T09:52:05.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.950+0000 7fc7a6800700 1 -- 172.21.15.78:0/1178911987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a0101f30 msgr2=0x7fc7a0102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.950+0000 7fc7a6800700 1 --2- 172.21.15.78:0/1178911987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a0101f30 0x7fc7a0102390 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc78c0099c0 tx=0x7fc78c009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.950+0000 7fc7a6800700 1 -- 172.21.15.78:0/1178911987 shutdown_connections 2024-03-14T09:52:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.950+0000 7fc7a6800700 1 --2- 172.21.15.78:0/1178911987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7a01028d0 0x7fc7a0109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.950+0000 7fc7a6800700 1 --2- 172.21.15.78:0/1178911987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7a0100d40 0x7fc7a0101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.950+0000 7fc7a6800700 1 --2- 172.21.15.78:0/1178911987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a0101f30 0x7fc7a0102390 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.950+0000 7fc7a6800700 1 -- 172.21.15.78:0/1178911987 >> 172.21.15.78:0/1178911987 conn(0x7fc7a00fc2b0 msgr2=0x7fc7a00fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.951+0000 7fc7a6800700 1 -- 172.21.15.78:0/1178911987 shutdown_connections 2024-03-14T09:52:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.951+0000 7fc7a6800700 1 -- 172.21.15.78:0/1178911987 wait complete. 2024-03-14T09:52:05.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.952+0000 7fc7a6800700 1 Processor -- start 2024-03-14T09:52:05.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.952+0000 7fc7a6800700 1 -- start start 2024-03-14T09:52:05.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.952+0000 7fc7a6800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7a0100d40 0x7fc7a0074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:05.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.952+0000 7fc7a6800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7a0101f30 0x7fc7a00731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:05.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.952+0000 7fc7a6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a01028d0 0x7fc7a00736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:05.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.952+0000 7fc7a6800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a0073e80 con 0x7fc7a01028d0 2024-03-14T09:52:05.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.952+0000 7fc7a6800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a0073ff0 con 0x7fc7a0100d40 2024-03-14T09:52:05.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.952+0000 7fc7a6800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a01a1190 con 0x7fc7a0101f30 2024-03-14T09:52:05.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.952+0000 7fc79fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a01028d0 0x7fc7a00736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:05.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.952+0000 7fc79fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a01028d0 0x7fc7a00736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36928/0 (socket says 172.21.15.78:36928) 2024-03-14T09:52:05.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.952+0000 7fc79fe00700 1 -- 172.21.15.78:0/2048382945 learned_addr learned my addr 172.21.15.78:0/2048382945 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:05.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.952+0000 7fc79f400700 1 --2- 172.21.15.78:0/2048382945 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7a0100d40 0x7fc7a0074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:05.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.952+0000 7fc79ea00700 1 --2- 172.21.15.78:0/2048382945 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7a0101f30 0x7fc7a00731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:05.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.953+0000 7fc79fe00700 1 -- 172.21.15.78:0/2048382945 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7a0101f30 msgr2=0x7fc7a00731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:05.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.953+0000 7fc79fe00700 1 --2- 172.21.15.78:0/2048382945 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7a0101f30 0x7fc7a00731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.953+0000 7fc79fe00700 1 -- 172.21.15.78:0/2048382945 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7a0100d40 msgr2=0x7fc7a0074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:05.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.953+0000 7fc79fe00700 1 --2- 172.21.15.78:0/2048382945 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7a0100d40 0x7fc7a0074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:05.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.953+0000 7fc79fe00700 1 -- 172.21.15.78:0/2048382945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c0095f0 con 0x7fc7a01028d0 2024-03-14T09:52:05.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.953+0000 7fc79fe00700 1 --2- 172.21.15.78:0/2048382945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a01028d0 0x7fc7a00736e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc798007ee0 tx=0x7fc79800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:05.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.953+0000 7fc797e00700 1 -- 172.21.15.78:0/2048382945 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc798017070 con 0x7fc7a01028d0 2024-03-14T09:52:05.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.953+0000 7fc797e00700 1 -- 172.21.15.78:0/2048382945 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc79800f040 con 0x7fc7a01028d0 2024-03-14T09:52:05.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.953+0000 7fc7a6800700 1 -- 172.21.15.78:0/2048382945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a01a1420 con 0x7fc7a01028d0 2024-03-14T09:52:05.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.953+0000 7fc797e00700 1 -- 172.21.15.78:0/2048382945 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7980054b0 con 0x7fc7a01028d0 2024-03-14T09:52:05.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.953+0000 7fc7a6800700 1 -- 172.21.15.78:0/2048382945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a01a1930 con 0x7fc7a01028d0 2024-03-14T09:52:05.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.954+0000 7fc7a6800700 1 -- 172.21.15.78:0/2048382945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a0108210 con 0x7fc7a01028d0 2024-03-14T09:52:05.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.956+0000 7fc797e00700 1 -- 172.21.15.78:0/2048382945 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fc798004ad0 con 0x7fc7a01028d0 2024-03-14T09:52:05.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.957+0000 7fc797e00700 1 --2- 172.21.15.78:0/2048382945 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc7840a0a90 0x7fc7840a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:05.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.957+0000 7fc797e00700 1 -- 172.21.15.78:0/2048382945 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc7980cb730 con 0x7fc7a01028d0 2024-03-14T09:52:05.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.957+0000 7fc79f400700 1 --2- 172.21.15.78:0/2048382945 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc7840a0a90 0x7fc7840a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:05.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.958+0000 7fc79f400700 1 --2- 172.21.15.78:0/2048382945 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc7840a0a90 0x7fc7840a2f50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0101d90 tx=0x7fc790009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:05.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:05.961+0000 7fc797e00700 1 -- 172.21.15.78:0/2048382945 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc7980951b0 con 0x7fc7a01028d0 2024-03-14T09:52:06.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.235+0000 7fc7a6800700 1 -- 172.21.15.78:0/2048382945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fc7a0067ac0 con 0x7fc7a01028d0 2024-03-14T09:52:06.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:06 smithi152 ceph-mon[48279]: pgmap v228: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:52:06.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:06 smithi152 ceph-mon[48279]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-14T09:52:06.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:06 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/701537141' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-14T09:52:06.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:06 smithi152 ceph-mon[48279]: osdmap e77: 12 total, 12 up, 12 in 2024-03-14T09:52:06.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:06 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1484014752' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-14T09:52:06.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:06 smithi152 ceph-mon[48279]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-14T09:52:06.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:06 smithi152 ceph-mon[48279]: osdmap e78: 12 total, 12 up, 12 in 2024-03-14T09:52:06.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:06 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1484014752' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-14T09:52:06.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:06 smithi152 ceph-mon[48279]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-14T09:52:06.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:06 smithi152 ceph-mon[48279]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-14T09:52:06.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:06 smithi132 ceph-mon[47959]: pgmap v228: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:52:06.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:06 smithi132 ceph-mon[47959]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-14T09:52:06.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:06 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/701537141' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-14T09:52:06.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:06 smithi132 ceph-mon[47959]: osdmap e77: 12 total, 12 up, 12 in 2024-03-14T09:52:06.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:06 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1484014752' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-14T09:52:06.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:06 smithi132 ceph-mon[47959]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-14T09:52:06.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:06 smithi132 ceph-mon[47959]: osdmap e78: 12 total, 12 up, 12 in 2024-03-14T09:52:06.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:06 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1484014752' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-14T09:52:06.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:06 smithi132 ceph-mon[47959]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-14T09:52:06.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:06 smithi132 ceph-mon[47959]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-14T09:52:06.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:06 smithi078 ceph-mon[38143]: pgmap v228: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:52:06.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:06 smithi078 ceph-mon[38143]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-14T09:52:06.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:06 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/701537141' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-14T09:52:06.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:06 smithi078 ceph-mon[38143]: osdmap e77: 12 total, 12 up, 12 in 2024-03-14T09:52:06.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:06 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1484014752' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-14T09:52:06.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:06 smithi078 ceph-mon[38143]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-14T09:52:06.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:06 smithi078 ceph-mon[38143]: osdmap e78: 12 total, 12 up, 12 in 2024-03-14T09:52:06.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:06 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1484014752' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-14T09:52:06.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:06 smithi078 ceph-mon[38143]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-14T09:52:06.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:06 smithi078 ceph-mon[38143]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-14T09:52:06.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.721+0000 7fc797e00700 1 -- 172.21.15.78:0/2048382945 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v79) v1 ==== 157+0+0 (secure 0 0 0) 0x7fc7980987d0 con 0x7fc7a01028d0 2024-03-14T09:52:06.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.724+0000 7fc7a6800700 1 -- 172.21.15.78:0/2048382945 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc7840a0a90 msgr2=0x7fc7840a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:06.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.724+0000 7fc7a6800700 1 --2- 172.21.15.78:0/2048382945 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc7840a0a90 0x7fc7840a2f50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0101d90 tx=0x7fc790009260 comp rx=0 tx=0).stop 2024-03-14T09:52:06.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.724+0000 7fc7a6800700 1 -- 172.21.15.78:0/2048382945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a01028d0 msgr2=0x7fc7a00736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:06.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.724+0000 7fc7a6800700 1 --2- 172.21.15.78:0/2048382945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a01028d0 0x7fc7a00736e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc798007ee0 tx=0x7fc79800d3b0 comp rx=0 tx=0).stop 2024-03-14T09:52:06.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.724+0000 7fc7a6800700 1 -- 172.21.15.78:0/2048382945 shutdown_connections 2024-03-14T09:52:06.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.724+0000 7fc7a6800700 1 --2- 172.21.15.78:0/2048382945 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc7840a0a90 0x7fc7840a2f50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:06.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.724+0000 7fc7a6800700 1 --2- 172.21.15.78:0/2048382945 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7a0100d40 0x7fc7a0074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:06.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.724+0000 7fc7a6800700 1 --2- 172.21.15.78:0/2048382945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a01028d0 0x7fc7a00736e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:06.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.724+0000 7fc7a6800700 1 --2- 172.21.15.78:0/2048382945 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7a0101f30 0x7fc7a00731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:06.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.724+0000 7fc7a6800700 1 -- 172.21.15.78:0/2048382945 >> 172.21.15.78:0/2048382945 conn(0x7fc7a00fc2b0 msgr2=0x7fc7a0105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:06.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.725+0000 7fc7a6800700 1 -- 172.21.15.78:0/2048382945 shutdown_connections 2024-03-14T09:52:06.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.725+0000 7fc7a6800700 1 -- 172.21.15.78:0/2048382945 wait complete. 2024-03-14T09:52:06.744 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-14T09:52:06.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.944+0000 7f9d40c00700 1 -- 172.21.15.78:0/3990285023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d3c102850 msgr2=0x7f9d3c109100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:06.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.944+0000 7f9d40c00700 1 --2- 172.21.15.78:0/3990285023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d3c102850 0x7f9d3c109100 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400b0d0 tx=0x7f9d3400b3e0 comp rx=0 tx=0).stop 2024-03-14T09:52:06.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.944+0000 7f9d40c00700 1 -- 172.21.15.78:0/3990285023 shutdown_connections 2024-03-14T09:52:06.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.944+0000 7f9d40c00700 1 --2- 172.21.15.78:0/3990285023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d3c102850 0x7f9d3c109100 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:06.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.944+0000 7f9d40c00700 1 --2- 172.21.15.78:0/3990285023 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d3c100cc0 0x7f9d3c1010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:06.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.944+0000 7f9d40c00700 1 --2- 172.21.15.78:0/3990285023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d3c101eb0 0x7f9d3c102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:06.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.944+0000 7f9d40c00700 1 -- 172.21.15.78:0/3990285023 >> 172.21.15.78:0/3990285023 conn(0x7f9d3c0fc250 msgr2=0x7f9d3c0fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:06.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.944+0000 7f9d40c00700 1 -- 172.21.15.78:0/3990285023 shutdown_connections 2024-03-14T09:52:06.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.944+0000 7f9d40c00700 1 -- 172.21.15.78:0/3990285023 wait complete. 2024-03-14T09:52:06.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.945+0000 7f9d40c00700 1 Processor -- start 2024-03-14T09:52:06.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.945+0000 7f9d40c00700 1 -- start start 2024-03-14T09:52:06.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.945+0000 7f9d40c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d3c100cc0 0x7f9d3c199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:06.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.945+0000 7f9d40c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d3c101eb0 0x7f9d3c1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:06.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.945+0000 7f9d40c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d3c102850 0x7f9d3c1a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:06.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.945+0000 7f9d40c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c1a0e90 con 0x7f9d3c101eb0 2024-03-14T09:52:06.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.945+0000 7f9d40c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c1a0fd0 con 0x7f9d3c100cc0 2024-03-14T09:52:06.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.945+0000 7f9d40c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c1a1160 con 0x7f9d3c102850 2024-03-14T09:52:06.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.945+0000 7f9d39400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d3c101eb0 0x7f9d3c1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:06.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.946+0000 7f9d39400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d3c101eb0 0x7f9d3c1998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36944/0 (socket says 172.21.15.78:36944) 2024-03-14T09:52:06.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.946+0000 7f9d39e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d3c100cc0 0x7f9d3c199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:06.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.946+0000 7f9d39400700 1 -- 172.21.15.78:0/2549309204 learned_addr learned my addr 172.21.15.78:0/2549309204 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:06.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.946+0000 7f9d3a800700 1 --2- 172.21.15.78:0/2549309204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d3c102850 0x7f9d3c1a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:06.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.946+0000 7f9d39e00700 1 -- 172.21.15.78:0/2549309204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d3c102850 msgr2=0x7f9d3c1a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:06.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.946+0000 7f9d39e00700 1 --2- 172.21.15.78:0/2549309204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d3c102850 0x7f9d3c1a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:06.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.946+0000 7f9d39e00700 1 -- 172.21.15.78:0/2549309204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d3c101eb0 msgr2=0x7f9d3c1998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:06.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.946+0000 7f9d39e00700 1 --2- 172.21.15.78:0/2549309204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d3c101eb0 0x7f9d3c1998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:06.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.946+0000 7f9d39e00700 1 -- 172.21.15.78:0/2549309204 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d34009d00 con 0x7f9d3c100cc0 2024-03-14T09:52:06.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.946+0000 7f9d3a800700 1 --2- 172.21.15.78:0/2549309204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d3c102850 0x7f9d3c1a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:52:06.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.946+0000 7f9d39e00700 1 --2- 172.21.15.78:0/2549309204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d3c100cc0 0x7f9d3c199360 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c00e9c0 tx=0x7f9d2c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:06.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.946+0000 7f9d32a00700 1 -- 172.21.15.78:0/2549309204 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c00cbd0 con 0x7f9d3c100cc0 2024-03-14T09:52:06.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.946+0000 7f9d40c00700 1 -- 172.21.15.78:0/2549309204 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3c1a1440 con 0x7f9d3c100cc0 2024-03-14T09:52:06.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.946+0000 7f9d32a00700 1 -- 172.21.15.78:0/2549309204 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d2c00eeb0 con 0x7f9d3c100cc0 2024-03-14T09:52:06.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.946+0000 7f9d32a00700 1 -- 172.21.15.78:0/2549309204 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d2c003c70 con 0x7f9d3c100cc0 2024-03-14T09:52:06.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.947+0000 7f9d40c00700 1 -- 172.21.15.78:0/2549309204 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3c1a1950 con 0x7f9d3c100cc0 2024-03-14T09:52:06.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.947+0000 7f9d40c00700 1 -- 172.21.15.78:0/2549309204 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3c0679e0 con 0x7f9d3c100cc0 2024-03-14T09:52:06.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.950+0000 7f9d32a00700 1 -- 172.21.15.78:0/2549309204 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f9d2c003dd0 con 0x7f9d3c100cc0 2024-03-14T09:52:06.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.951+0000 7f9d32a00700 1 --2- 172.21.15.78:0/2549309204 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9d200a0a80 0x7f9d200a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:06.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.951+0000 7f9d32a00700 1 -- 172.21.15.78:0/2549309204 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8205+0+0 (secure 0 0 0) 0x7f9d2c014070 con 0x7f9d3c100cc0 2024-03-14T09:52:06.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.951+0000 7f9d39400700 1 --2- 172.21.15.78:0/2549309204 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9d200a0a80 0x7f9d200a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:06.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.952+0000 7f9d39400700 1 --2- 172.21.15.78:0/2549309204 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9d200a0a80 0x7f9d200a2f40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9d28009d20 tx=0x7f9d28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:06.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:06.953+0000 7f9d32a00700 1 -- 172.21.15.78:0/2549309204 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9d2c095e30 con 0x7f9d3c100cc0 2024-03-14T09:52:07.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:07.161+0000 7f9d40c00700 1 -- 172.21.15.78:0/2549309204 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9d3c10c8e0 con 0x7f9d3c100cc0 2024-03-14T09:52:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:07 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2048382945' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-14T09:52:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:07 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] up:active 2024-03-14T09:52:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:07 smithi152 ceph-mon[48279]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-14T09:52:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:07 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2048382945' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-14T09:52:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:07 smithi152 ceph-mon[48279]: osdmap e79: 12 total, 12 up, 12 in 2024-03-14T09:52:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:07 smithi152 ceph-mon[48279]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-14T09:52:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:07 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2549309204' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-14T09:52:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:07 smithi152 ceph-mon[48279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-14T09:52:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:07 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2048382945' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-14T09:52:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:07 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] up:active 2024-03-14T09:52:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:07 smithi132 ceph-mon[47959]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-14T09:52:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:07 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2048382945' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-14T09:52:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:07 smithi132 ceph-mon[47959]: osdmap e79: 12 total, 12 up, 12 in 2024-03-14T09:52:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:07 smithi132 ceph-mon[47959]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-14T09:52:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:07 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2549309204' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-14T09:52:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:07 smithi132 ceph-mon[47959]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-14T09:52:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:07 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2048382945' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-14T09:52:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:07 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] up:active 2024-03-14T09:52:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:07 smithi078 ceph-mon[38143]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-14T09:52:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:07 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2048382945' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-14T09:52:07.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:07 smithi078 ceph-mon[38143]: osdmap e79: 12 total, 12 up, 12 in 2024-03-14T09:52:07.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:07 smithi078 ceph-mon[38143]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-14T09:52:07.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:07 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2549309204' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-14T09:52:07.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:07 smithi078 ceph-mon[38143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-14T09:52:08.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:08 smithi152 ceph-mon[48279]: pgmap v232: 97 pgs: 5 creating+peering, 40 active+clean, 52 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-03-14T09:52:08.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:08 smithi152 ceph-mon[48279]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-14T09:52:08.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:08 smithi152 ceph-mon[48279]: Cluster is now healthy 2024-03-14T09:52:08.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:08 smithi132 ceph-mon[47959]: pgmap v232: 97 pgs: 5 creating+peering, 40 active+clean, 52 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-03-14T09:52:08.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:08 smithi132 ceph-mon[47959]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-14T09:52:08.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:08 smithi132 ceph-mon[47959]: Cluster is now healthy 2024-03-14T09:52:08.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:08 smithi078 ceph-mon[38143]: pgmap v232: 97 pgs: 5 creating+peering, 40 active+clean, 52 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-03-14T09:52:08.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:08 smithi078 ceph-mon[38143]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-14T09:52:08.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:08 smithi078 ceph-mon[38143]: Cluster is now healthy 2024-03-14T09:52:08.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.737+0000 7f9d32a00700 1 -- 172.21.15.78:0/2549309204 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v81) v1 ==== 228+0+0 (secure 0 0 0) 0x7f9d2c099450 con 0x7f9d3c100cc0 2024-03-14T09:52:08.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.745+0000 7f9d40c00700 1 -- 172.21.15.78:0/2549309204 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9d200a0a80 msgr2=0x7f9d200a2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:08.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.745+0000 7f9d40c00700 1 --2- 172.21.15.78:0/2549309204 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9d200a0a80 0x7f9d200a2f40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9d28009d20 tx=0x7f9d28009260 comp rx=0 tx=0).stop 2024-03-14T09:52:08.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.745+0000 7f9d40c00700 1 -- 172.21.15.78:0/2549309204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d3c100cc0 msgr2=0x7f9d3c199360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:08.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.745+0000 7f9d40c00700 1 --2- 172.21.15.78:0/2549309204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d3c100cc0 0x7f9d3c199360 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c00e9c0 tx=0x7f9d2c00ecd0 comp rx=0 tx=0).stop 2024-03-14T09:52:08.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.745+0000 7f9d40c00700 1 -- 172.21.15.78:0/2549309204 shutdown_connections 2024-03-14T09:52:08.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.745+0000 7f9d40c00700 1 --2- 172.21.15.78:0/2549309204 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9d200a0a80 0x7f9d200a2f40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:08.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.745+0000 7f9d40c00700 1 --2- 172.21.15.78:0/2549309204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d3c100cc0 0x7f9d3c199360 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:08.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.745+0000 7f9d40c00700 1 --2- 172.21.15.78:0/2549309204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d3c102850 0x7f9d3c1a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:08.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.745+0000 7f9d40c00700 1 --2- 172.21.15.78:0/2549309204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d3c101eb0 0x7f9d3c1998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:08.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.745+0000 7f9d40c00700 1 -- 172.21.15.78:0/2549309204 >> 172.21.15.78:0/2549309204 conn(0x7f9d3c0fc250 msgr2=0x7f9d3c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:08.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.746+0000 7f9d40c00700 1 -- 172.21.15.78:0/2549309204 shutdown_connections 2024-03-14T09:52:08.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.746+0000 7f9d40c00700 1 -- 172.21.15.78:0/2549309204 wait complete. 2024-03-14T09:52:08.747 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data_ec' created 2024-03-14T09:52:08.766 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-14T09:52:08.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.962+0000 7f2dd2800700 1 -- 172.21.15.78:0/2734579989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dcc1028e0 msgr2=0x7f2dcc109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:08.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.962+0000 7f2dd2800700 1 --2- 172.21.15.78:0/2734579989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dcc1028e0 0x7f2dcc109190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2dc400b0d0 tx=0x7f2dc400b3e0 comp rx=0 tx=0).stop 2024-03-14T09:52:08.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.963+0000 7f2dd2800700 1 -- 172.21.15.78:0/2734579989 shutdown_connections 2024-03-14T09:52:08.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.963+0000 7f2dd2800700 1 --2- 172.21.15.78:0/2734579989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dcc100d50 0x7f2dcc101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:08.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.963+0000 7f2dd2800700 1 --2- 172.21.15.78:0/2734579989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dcc1028e0 0x7f2dcc109190 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:08.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.963+0000 7f2dd2800700 1 --2- 172.21.15.78:0/2734579989 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dcc101f40 0x7f2dcc1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:08.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.963+0000 7f2dd2800700 1 -- 172.21.15.78:0/2734579989 >> 172.21.15.78:0/2734579989 conn(0x7f2dcc0fc2c0 msgr2=0x7f2dcc0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:08.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.963+0000 7f2dd2800700 1 -- 172.21.15.78:0/2734579989 shutdown_connections 2024-03-14T09:52:08.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.963+0000 7f2dd2800700 1 -- 172.21.15.78:0/2734579989 wait complete. 2024-03-14T09:52:08.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.964+0000 7f2dd2800700 1 Processor -- start 2024-03-14T09:52:08.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.964+0000 7f2dd2800700 1 -- start start 2024-03-14T09:52:08.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.964+0000 7f2dd2800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dcc100d50 0x7f2dcc199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:08.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.964+0000 7f2dd2800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dcc101f40 0x7f2dcc199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:08.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.964+0000 7f2dd2800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dcc1028e0 0x7f2dcc1a09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:08.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.964+0000 7f2dd2800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc1a0fc0 con 0x7f2dcc100d50 2024-03-14T09:52:08.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.964+0000 7f2dd2800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc1a1100 con 0x7f2dcc1028e0 2024-03-14T09:52:08.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.964+0000 7f2dd2800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc1a1290 con 0x7f2dcc101f40 2024-03-14T09:52:08.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.965+0000 7f2dcb400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dcc100d50 0x7f2dcc199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:08.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.965+0000 7f2dcaa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dcc101f40 0x7f2dcc199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:08.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.965+0000 7f2dcaa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dcc101f40 0x7f2dcc199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:52216/0 (socket says 172.21.15.78:52216) 2024-03-14T09:52:08.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.965+0000 7f2dcaa00700 1 -- 172.21.15.78:0/740402709 learned_addr learned my addr 172.21.15.78:0/740402709 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:08.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.965+0000 7f2dcbe00700 1 --2- 172.21.15.78:0/740402709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dcc1028e0 0x7f2dcc1a09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:08.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.965+0000 7f2dcb400700 1 -- 172.21.15.78:0/740402709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dcc101f40 msgr2=0x7f2dcc199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:08.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.965+0000 7f2dcb400700 1 --2- 172.21.15.78:0/740402709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dcc101f40 0x7f2dcc199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:08.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.965+0000 7f2dcb400700 1 -- 172.21.15.78:0/740402709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dcc1028e0 msgr2=0x7f2dcc1a09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:08.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.965+0000 7f2dcb400700 1 --2- 172.21.15.78:0/740402709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dcc1028e0 0x7f2dcc1a09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.965+0000 7f2dcb400700 1 -- 172.21.15.78:0/740402709 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc4009d00 con 0x7f2dcc100d50 2024-03-14T09:52:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.965+0000 7f2dcb400700 1 --2- 172.21.15.78:0/740402709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dcc100d50 0x7f2dcc199530 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc00e9c0 tx=0x7f2dbc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.965+0000 7f2dc3e00700 1 -- 172.21.15.78:0/740402709 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc004220 con 0x7f2dcc100d50 2024-03-14T09:52:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.965+0000 7f2dc3e00700 1 -- 172.21.15.78:0/740402709 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2dbc004380 con 0x7f2dcc100d50 2024-03-14T09:52:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.965+0000 7f2dd2800700 1 -- 172.21.15.78:0/740402709 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dcc1a1570 con 0x7f2dcc100d50 2024-03-14T09:52:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.965+0000 7f2dc3e00700 1 -- 172.21.15.78:0/740402709 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc003be0 con 0x7f2dcc100d50 2024-03-14T09:52:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.965+0000 7f2dd2800700 1 -- 172.21.15.78:0/740402709 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dcc1a1ad0 con 0x7f2dcc100d50 2024-03-14T09:52:08.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.966+0000 7f2dd2800700 1 -- 172.21.15.78:0/740402709 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dcc067ac0 con 0x7f2dcc100d50 2024-03-14T09:52:08.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.969+0000 7f2dc3e00700 1 -- 172.21.15.78:0/740402709 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f2dbc003d40 con 0x7f2dcc100d50 2024-03-14T09:52:08.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.970+0000 7f2dc3e00700 1 --2- 172.21.15.78:0/740402709 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2db00a0b10 0x7f2db00a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:08.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.970+0000 7f2dc3e00700 1 -- 172.21.15.78:0/740402709 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f2dbc014070 con 0x7f2dcc100d50 2024-03-14T09:52:08.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.970+0000 7f2dcaa00700 1 --2- 172.21.15.78:0/740402709 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2db00a0b10 0x7f2db00a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:08.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.971+0000 7f2dcaa00700 1 --2- 172.21.15.78:0/740402709 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2db00a0b10 0x7f2db00a2fd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009d20 tx=0x7f2db8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:08.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:08.973+0000 7f2dc3e00700 1 -- 172.21.15.78:0/740402709 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2dbc095d90 con 0x7f2dcc100d50 2024-03-14T09:52:09.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.179+0000 7f2dd2800700 1 -- 172.21.15.78:0/740402709 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f2dcc10c9e0 con 0x7f2dcc100d50 2024-03-14T09:52:09.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:09 smithi132 ceph-mon[47959]: osdmap e80: 12 total, 12 up, 12 in 2024-03-14T09:52:09.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:09 smithi132 ceph-mon[47959]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-14T09:52:09.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:09 smithi132 ceph-mon[47959]: osdmap e81: 12 total, 12 up, 12 in 2024-03-14T09:52:09.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:09 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/740402709' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-14T09:52:09.271 INFO:journalctl@ceph.mds.b.smithi132.stdout:Mar 14 09:52:09 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-b[86027]: 2024-03-14T09:52:09.014+0000 7f2f722b4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-14T09:52:09.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:09 smithi152 ceph-mon[48279]: osdmap e80: 12 total, 12 up, 12 in 2024-03-14T09:52:09.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:09 smithi152 ceph-mon[48279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-14T09:52:09.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:09 smithi152 ceph-mon[48279]: osdmap e81: 12 total, 12 up, 12 in 2024-03-14T09:52:09.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:09 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/740402709' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-14T09:52:09.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:09 smithi078 ceph-mon[38143]: osdmap e80: 12 total, 12 up, 12 in 2024-03-14T09:52:09.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:09 smithi078 ceph-mon[38143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-14T09:52:09.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:09 smithi078 ceph-mon[38143]: osdmap e81: 12 total, 12 up, 12 in 2024-03-14T09:52:09.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:09 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/740402709' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-14T09:52:09.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.743+0000 7f2dc3e00700 1 -- 172.21.15.78:0/740402709 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v82) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2dbc0993b0 con 0x7f2dcc100d50 2024-03-14T09:52:09.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.749+0000 7f2dd2800700 1 -- 172.21.15.78:0/740402709 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2db00a0b10 msgr2=0x7f2db00a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:09.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.749+0000 7f2dd2800700 1 --2- 172.21.15.78:0/740402709 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2db00a0b10 0x7f2db00a2fd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009d20 tx=0x7f2db8009260 comp rx=0 tx=0).stop 2024-03-14T09:52:09.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.749+0000 7f2dd2800700 1 -- 172.21.15.78:0/740402709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dcc100d50 msgr2=0x7f2dcc199530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:09.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.749+0000 7f2dd2800700 1 --2- 172.21.15.78:0/740402709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dcc100d50 0x7f2dcc199530 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc00e9c0 tx=0x7f2dbc00ecd0 comp rx=0 tx=0).stop 2024-03-14T09:52:09.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.750+0000 7f2dd2800700 1 -- 172.21.15.78:0/740402709 shutdown_connections 2024-03-14T09:52:09.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.750+0000 7f2dd2800700 1 --2- 172.21.15.78:0/740402709 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2db00a0b10 0x7f2db00a2fd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:09.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.750+0000 7f2dd2800700 1 --2- 172.21.15.78:0/740402709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dcc1028e0 0x7f2dcc1a09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:09.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.750+0000 7f2dd2800700 1 --2- 172.21.15.78:0/740402709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dcc100d50 0x7f2dcc199530 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:09.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.750+0000 7f2dd2800700 1 --2- 172.21.15.78:0/740402709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dcc101f40 0x7f2dcc199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:09.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.751+0000 7f2dd2800700 1 -- 172.21.15.78:0/740402709 >> 172.21.15.78:0/740402709 conn(0x7f2dcc0fc2c0 msgr2=0x7f2dcc072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:09.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.751+0000 7f2dd2800700 1 -- 172.21.15.78:0/740402709 shutdown_connections 2024-03-14T09:52:09.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.751+0000 7f2dd2800700 1 -- 172.21.15.78:0/740402709 wait complete. 2024-03-14T09:52:09.758 INFO:teuthology.orchestra.run.smithi078.stderr:set pool 4 allow_ec_overwrites to true 2024-03-14T09:52:09.781 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-14T09:52:09.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.975+0000 7fb5b0a00700 1 -- 172.21.15.78:0/2168769437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5ac101f40 msgr2=0x7fb5ac1023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:09.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.975+0000 7fb5b0a00700 1 --2- 172.21.15.78:0/2168769437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5ac101f40 0x7fb5ac1023a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb5980099c0 tx=0x7fb598009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:09.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.975+0000 7fb5b0a00700 1 -- 172.21.15.78:0/2168769437 shutdown_connections 2024-03-14T09:52:09.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.975+0000 7fb5b0a00700 1 --2- 172.21.15.78:0/2168769437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5ac1028e0 0x7fb5ac109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:09.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.975+0000 7fb5b0a00700 1 --2- 172.21.15.78:0/2168769437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb5ac100d50 0x7fb5ac101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:09.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.975+0000 7fb5b0a00700 1 --2- 172.21.15.78:0/2168769437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5ac101f40 0x7fb5ac1023a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:09.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.975+0000 7fb5b0a00700 1 -- 172.21.15.78:0/2168769437 >> 172.21.15.78:0/2168769437 conn(0x7fb5ac0fc2c0 msgr2=0x7fb5ac0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:09.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.975+0000 7fb5b0a00700 1 -- 172.21.15.78:0/2168769437 shutdown_connections 2024-03-14T09:52:09.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.975+0000 7fb5b0a00700 1 -- 172.21.15.78:0/2168769437 wait complete. 2024-03-14T09:52:09.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.976+0000 7fb5b0a00700 1 Processor -- start 2024-03-14T09:52:09.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.977+0000 7fb5b0a00700 1 -- start start 2024-03-14T09:52:09.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.977+0000 7fb5b0a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb5ac100d50 0x7fb5ac199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:09.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.977+0000 7fb5b0a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5ac101f40 0x7fb5ac199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:09.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.977+0000 7fb5b0a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5ac1028e0 0x7fb5ac1a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:09.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.977+0000 7fb5b0a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac1a0e40 con 0x7fb5ac1028e0 2024-03-14T09:52:09.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.977+0000 7fb5b0a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac1a0f80 con 0x7fb5ac101f40 2024-03-14T09:52:09.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.977+0000 7fb5b0a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac1a1110 con 0x7fb5ac100d50 2024-03-14T09:52:09.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.977+0000 7fb5aa800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5ac1028e0 0x7fb5ac1a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:09.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.977+0000 7fb5aa800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5ac1028e0 0x7fb5ac1a0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37008/0 (socket says 172.21.15.78:37008) 2024-03-14T09:52:09.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.977+0000 7fb5aa800700 1 -- 172.21.15.78:0/4127224268 learned_addr learned my addr 172.21.15.78:0/4127224268 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:09.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.977+0000 7fb5a9e00700 1 --2- 172.21.15.78:0/4127224268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb5ac100d50 0x7fb5ac199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:09.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.977+0000 7fb5a9400700 1 --2- 172.21.15.78:0/4127224268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5ac101f40 0x7fb5ac199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:09.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.978+0000 7fb5aa800700 1 -- 172.21.15.78:0/4127224268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb5ac100d50 msgr2=0x7fb5ac199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:09.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.978+0000 7fb5aa800700 1 --2- 172.21.15.78:0/4127224268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb5ac100d50 0x7fb5ac199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:09.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.978+0000 7fb5aa800700 1 -- 172.21.15.78:0/4127224268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5ac101f40 msgr2=0x7fb5ac199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:09.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.978+0000 7fb5aa800700 1 --2- 172.21.15.78:0/4127224268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5ac101f40 0x7fb5ac199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:09.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.978+0000 7fb5aa800700 1 -- 172.21.15.78:0/4127224268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5980095f0 con 0x7fb5ac1028e0 2024-03-14T09:52:09.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.978+0000 7fb5aa800700 1 --2- 172.21.15.78:0/4127224268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5ac1028e0 0x7fb5ac1a0870 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400ef30 tx=0x7fb5a400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:09.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.978+0000 7fb5a2a00700 1 -- 172.21.15.78:0/4127224268 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4018070 con 0x7fb5ac1028e0 2024-03-14T09:52:09.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.978+0000 7fb5b0a00700 1 -- 172.21.15.78:0/4127224268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5ac1a1390 con 0x7fb5ac1028e0 2024-03-14T09:52:09.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.978+0000 7fb5a2a00700 1 -- 172.21.15.78:0/4127224268 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5a400f040 con 0x7fb5ac1028e0 2024-03-14T09:52:09.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.978+0000 7fb5a2a00700 1 -- 172.21.15.78:0/4127224268 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a400cd00 con 0x7fb5ac1028e0 2024-03-14T09:52:09.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.978+0000 7fb5b0a00700 1 -- 172.21.15.78:0/4127224268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ac1a18a0 con 0x7fb5ac1028e0 2024-03-14T09:52:09.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.979+0000 7fb5b0a00700 1 -- 172.21.15.78:0/4127224268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ac067ac0 con 0x7fb5ac1028e0 2024-03-14T09:52:09.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.981+0000 7fb5a2a00700 1 -- 172.21.15.78:0/4127224268 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fb5a4003a00 con 0x7fb5ac1028e0 2024-03-14T09:52:09.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.982+0000 7fb5a2a00700 1 --2- 172.21.15.78:0/4127224268 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb5900a0a90 0x7fb5900a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:09.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.982+0000 7fb5a2a00700 1 -- 172.21.15.78:0/4127224268 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8698+0+0 (secure 0 0 0) 0x7fb5a40cc720 con 0x7fb5ac1028e0 2024-03-14T09:52:09.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.983+0000 7fb5a9e00700 1 --2- 172.21.15.78:0/4127224268 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb5900a0a90 0x7fb5900a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:09.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.983+0000 7fb5a9e00700 1 --2- 172.21.15.78:0/4127224268 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb5900a0a90 0x7fb5900a2f50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac101da0 tx=0x7fb59c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:09.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:09.984+0000 7fb5a2a00700 1 -- 172.21.15.78:0/4127224268 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb5a4095f50 con 0x7fb5ac1028e0 2024-03-14T09:52:10.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.243+0000 7fb5b0a00700 1 -- 172.21.15.78:0/4127224268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fb5ac10c9d0 con 0x7fb5ac1028e0 2024-03-14T09:52:10.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.249+0000 7fb5a2a00700 1 -- 172.21.15.78:0/4127224268 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7fb5a40b9020 con 0x7fb5ac1028e0 2024-03-14T09:52:10.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.252+0000 7fb5b0a00700 1 -- 172.21.15.78:0/4127224268 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb5900a0a90 msgr2=0x7fb5900a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:10.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.252+0000 7fb5b0a00700 1 --2- 172.21.15.78:0/4127224268 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb5900a0a90 0x7fb5900a2f50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac101da0 tx=0x7fb59c009260 comp rx=0 tx=0).stop 2024-03-14T09:52:10.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.252+0000 7fb5b0a00700 1 -- 172.21.15.78:0/4127224268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5ac1028e0 msgr2=0x7fb5ac1a0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:10.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.252+0000 7fb5b0a00700 1 --2- 172.21.15.78:0/4127224268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5ac1028e0 0x7fb5ac1a0870 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400ef30 tx=0x7fb5a400c3a0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.252+0000 7fb5b0a00700 1 -- 172.21.15.78:0/4127224268 shutdown_connections 2024-03-14T09:52:10.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.252+0000 7fb5b0a00700 1 --2- 172.21.15.78:0/4127224268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5ac1028e0 0x7fb5ac1a0870 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.252+0000 7fb5b0a00700 1 --2- 172.21.15.78:0/4127224268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb5ac100d50 0x7fb5ac199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.252+0000 7fb5b0a00700 1 --2- 172.21.15.78:0/4127224268 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb5900a0a90 0x7fb5900a2f50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.252+0000 7fb5b0a00700 1 --2- 172.21.15.78:0/4127224268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5ac101f40 0x7fb5ac199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.252+0000 7fb5b0a00700 1 -- 172.21.15.78:0/4127224268 >> 172.21.15.78:0/4127224268 conn(0x7fb5ac0fc2c0 msgr2=0x7fb5ac107760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:10.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.252+0000 7fb5b0a00700 1 -- 172.21.15.78:0/4127224268 shutdown_connections 2024-03-14T09:52:10.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.252+0000 7fb5b0a00700 1 -- 172.21.15.78:0/4127224268 wait complete. 2024-03-14T09:52:10.254 INFO:teuthology.orchestra.run.smithi078.stderr:added data pool 4 to fsmap 2024-03-14T09:52:10.271 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:52:10.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.463+0000 7ff336000700 1 -- 172.21.15.78:0/2013959865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff330101e40 msgr2=0x7ff3301022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:10.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.463+0000 7ff336000700 1 --2- 172.21.15.78:0/2013959865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff330101e40 0x7ff3301022a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff31c0099c0 tx=0x7ff31c009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.463+0000 7ff336000700 1 -- 172.21.15.78:0/2013959865 shutdown_connections 2024-03-14T09:52:10.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.463+0000 7ff336000700 1 --2- 172.21.15.78:0/2013959865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff330100c50 0x7ff330101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.463+0000 7ff336000700 1 --2- 172.21.15.78:0/2013959865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3301027e0 0x7ff330109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.463+0000 7ff336000700 1 --2- 172.21.15.78:0/2013959865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff330101e40 0x7ff3301022a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.463+0000 7ff336000700 1 -- 172.21.15.78:0/2013959865 >> 172.21.15.78:0/2013959865 conn(0x7ff3300fc1c0 msgr2=0x7ff3300fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:10.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.463+0000 7ff336000700 1 -- 172.21.15.78:0/2013959865 shutdown_connections 2024-03-14T09:52:10.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.463+0000 7ff336000700 1 -- 172.21.15.78:0/2013959865 wait complete. 2024-03-14T09:52:10.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.464+0000 7ff336000700 1 Processor -- start 2024-03-14T09:52:10.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.464+0000 7ff336000700 1 -- start start 2024-03-14T09:52:10.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.464+0000 7ff336000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff330100c50 0x7ff3301991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:10.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.464+0000 7ff336000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff330101e40 0x7ff330199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:10.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.465+0000 7ff336000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3301027e0 0x7ff3301a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:10.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.465+0000 7ff336000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3301a0d10 con 0x7ff330101e40 2024-03-14T09:52:10.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.465+0000 7ff336000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3301a0e50 con 0x7ff3301027e0 2024-03-14T09:52:10.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.465+0000 7ff336000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3301a0fe0 con 0x7ff330100c50 2024-03-14T09:52:10.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.465+0000 7ff32ea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff330101e40 0x7ff330199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:10.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.465+0000 7ff32ea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff330101e40 0x7ff330199720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37026/0 (socket says 172.21.15.78:37026) 2024-03-14T09:52:10.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.465+0000 7ff32ea00700 1 -- 172.21.15.78:0/889862802 learned_addr learned my addr 172.21.15.78:0/889862802 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:10.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.465+0000 7ff32f400700 1 --2- 172.21.15.78:0/889862802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff330100c50 0x7ff3301991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:10.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.465+0000 7ff32fe00700 1 --2- 172.21.15.78:0/889862802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3301027e0 0x7ff3301a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:10.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.465+0000 7ff32ea00700 1 -- 172.21.15.78:0/889862802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff330100c50 msgr2=0x7ff3301991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:10.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.465+0000 7ff32ea00700 1 --2- 172.21.15.78:0/889862802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff330100c50 0x7ff3301991e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.465+0000 7ff32ea00700 1 -- 172.21.15.78:0/889862802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3301027e0 msgr2=0x7ff3301a0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:10.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.465+0000 7ff32ea00700 1 --2- 172.21.15.78:0/889862802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3301027e0 0x7ff3301a0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.465+0000 7ff32ea00700 1 -- 172.21.15.78:0/889862802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c0095f0 con 0x7ff330101e40 2024-03-14T09:52:10.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.465+0000 7ff32ea00700 1 --2- 172.21.15.78:0/889862802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff330101e40 0x7ff330199720 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff31c00f690 tx=0x7ff31c00f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:10.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.466+0000 7ff327e00700 1 -- 172.21.15.78:0/889862802 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c01c070 con 0x7ff330101e40 2024-03-14T09:52:10.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.466+0000 7ff327e00700 1 -- 172.21.15.78:0/889862802 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff31c00fe90 con 0x7ff330101e40 2024-03-14T09:52:10.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.466+0000 7ff336000700 1 -- 172.21.15.78:0/889862802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3301a1260 con 0x7ff330101e40 2024-03-14T09:52:10.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.466+0000 7ff327e00700 1 -- 172.21.15.78:0/889862802 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31c022e00 con 0x7ff330101e40 2024-03-14T09:52:10.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.466+0000 7ff336000700 1 -- 172.21.15.78:0/889862802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3301a16e0 con 0x7ff330101e40 2024-03-14T09:52:10.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.467+0000 7ff336000700 1 -- 172.21.15.78:0/889862802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3300679e0 con 0x7ff330101e40 2024-03-14T09:52:10.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.469+0000 7ff327e00700 1 -- 172.21.15.78:0/889862802 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7ff31c020070 con 0x7ff330101e40 2024-03-14T09:52:10.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.470+0000 7ff327e00700 1 --2- 172.21.15.78:0/889862802 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff3140a0b10 0x7ff3140a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:10.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.470+0000 7ff327e00700 1 -- 172.21.15.78:0/889862802 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff31c0cd010 con 0x7ff330101e40 2024-03-14T09:52:10.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.470+0000 7ff32f400700 1 --2- 172.21.15.78:0/889862802 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff3140a0b10 0x7ff3140a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:10.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.471+0000 7ff32f400700 1 --2- 172.21.15.78:0/889862802 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff3140a0b10 0x7ff3140a2fd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff330101ca0 tx=0x7ff320009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:10.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.474+0000 7ff327e00700 1 -- 172.21.15.78:0/889862802 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff31c096880 con 0x7ff330101e40 2024-03-14T09:52:10.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:10 smithi152 ceph-mon[48279]: pgmap v235: 129 pgs: 5 creating+peering, 79 active+clean, 45 unknown; 579 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s wr, 9 op/s 2024-03-14T09:52:10.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:10 smithi152 ceph-mon[48279]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-14T09:52:10.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:10 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/740402709' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-14T09:52:10.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:10 smithi152 ceph-mon[48279]: osdmap e82: 12 total, 12 up, 12 in 2024-03-14T09:52:10.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:10 smithi132 ceph-mon[47959]: pgmap v235: 129 pgs: 5 creating+peering, 79 active+clean, 45 unknown; 579 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s wr, 9 op/s 2024-03-14T09:52:10.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:10 smithi132 ceph-mon[47959]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-14T09:52:10.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:10 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/740402709' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-14T09:52:10.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:10 smithi132 ceph-mon[47959]: osdmap e82: 12 total, 12 up, 12 in 2024-03-14T09:52:10.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:10 smithi078 ceph-mon[38143]: pgmap v235: 129 pgs: 5 creating+peering, 79 active+clean, 45 unknown; 579 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s wr, 9 op/s 2024-03-14T09:52:10.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:10 smithi078 ceph-mon[38143]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-14T09:52:10.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:10 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/740402709' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-14T09:52:10.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:10 smithi078 ceph-mon[38143]: osdmap e82: 12 total, 12 up, 12 in 2024-03-14T09:52:10.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.725+0000 7ff336000700 1 -- 172.21.15.78:0/889862802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff33010caa0 con 0x7ff330101e40 2024-03-14T09:52:10.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.726+0000 7ff327e00700 1 -- 172.21.15.78:0/889862802 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10810 (secure 0 0 0) 0x7ff31c099ea0 con 0x7ff330101e40 2024-03-14T09:52:10.727 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:10.727 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24527,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:52:10.245795+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T09:52:10.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.728+0000 7ff336000700 1 -- 172.21.15.78:0/889862802 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff3140a0b10 msgr2=0x7ff3140a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:10.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.728+0000 7ff336000700 1 --2- 172.21.15.78:0/889862802 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff3140a0b10 0x7ff3140a2fd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff330101ca0 tx=0x7ff320009260 comp rx=0 tx=0).stop 2024-03-14T09:52:10.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.729+0000 7ff336000700 1 -- 172.21.15.78:0/889862802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff330101e40 msgr2=0x7ff330199720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:10.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.729+0000 7ff336000700 1 --2- 172.21.15.78:0/889862802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff330101e40 0x7ff330199720 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff31c00f690 tx=0x7ff31c00f6c0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.729+0000 7ff336000700 1 -- 172.21.15.78:0/889862802 shutdown_connections 2024-03-14T09:52:10.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.729+0000 7ff336000700 1 --2- 172.21.15.78:0/889862802 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff3140a0b10 0x7ff3140a2fd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.729+0000 7ff336000700 1 --2- 172.21.15.78:0/889862802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3301027e0 0x7ff3301a0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.729+0000 7ff336000700 1 --2- 172.21.15.78:0/889862802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff330100c50 0x7ff3301991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.729+0000 7ff336000700 1 --2- 172.21.15.78:0/889862802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff330101e40 0x7ff330199720 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.729+0000 7ff336000700 1 -- 172.21.15.78:0/889862802 >> 172.21.15.78:0/889862802 conn(0x7ff3300fc1c0 msgr2=0x7ff330072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:10.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.729+0000 7ff336000700 1 -- 172.21.15.78:0/889862802 shutdown_connections 2024-03-14T09:52:10.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.729+0000 7ff336000700 1 -- 172.21.15.78:0/889862802 wait complete. 2024-03-14T09:52:10.731 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2024-03-14T09:52:10.748 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:52:10.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.938+0000 7f362c400700 1 -- 172.21.15.78:0/2464157242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3624101e40 msgr2=0x7f36241022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:10.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.938+0000 7f362c400700 1 --2- 172.21.15.78:0/2464157242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3624101e40 0x7f36241022a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f36100099c0 tx=0x7f3610009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.938+0000 7f362c400700 1 -- 172.21.15.78:0/2464157242 shutdown_connections 2024-03-14T09:52:10.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.938+0000 7f362c400700 1 --2- 172.21.15.78:0/2464157242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3624100c50 0x7f3624101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.938+0000 7f362c400700 1 --2- 172.21.15.78:0/2464157242 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f36241027e0 0x7f3624109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.938+0000 7f362c400700 1 --2- 172.21.15.78:0/2464157242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3624101e40 0x7f36241022a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.938+0000 7f362c400700 1 -- 172.21.15.78:0/2464157242 >> 172.21.15.78:0/2464157242 conn(0x7f36240fc1c0 msgr2=0x7f36240fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:10.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.938+0000 7f362c400700 1 -- 172.21.15.78:0/2464157242 shutdown_connections 2024-03-14T09:52:10.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.938+0000 7f362c400700 1 -- 172.21.15.78:0/2464157242 wait complete. 2024-03-14T09:52:10.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.939+0000 7f362c400700 1 Processor -- start 2024-03-14T09:52:10.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.939+0000 7f362c400700 1 -- start start 2024-03-14T09:52:10.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f362c400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3624100c50 0x7f3624199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:10.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f362c400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3624101e40 0x7f3624199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:10.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f362c400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36241027e0 0x7f36241a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:10.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f362c400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36241a0d20 con 0x7f36241027e0 2024-03-14T09:52:10.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f362c400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36241a0e60 con 0x7f3624101e40 2024-03-14T09:52:10.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f362c400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36241a0ff0 con 0x7f3624100c50 2024-03-14T09:52:10.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f362a400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36241027e0 0x7f36241a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:10.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f362a400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36241027e0 0x7f36241a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37046/0 (socket says 172.21.15.78:37046) 2024-03-14T09:52:10.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f362a400700 1 -- 172.21.15.78:0/179125888 learned_addr learned my addr 172.21.15.78:0/179125888 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:10.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f3629a00700 1 --2- 172.21.15.78:0/179125888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3624100c50 0x7f3624199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:10.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f3629000700 1 --2- 172.21.15.78:0/179125888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3624101e40 0x7f3624199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:10.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f362a400700 1 -- 172.21.15.78:0/179125888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3624100c50 msgr2=0x7f3624199240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:10.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f362a400700 1 --2- 172.21.15.78:0/179125888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3624100c50 0x7f3624199240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f362a400700 1 -- 172.21.15.78:0/179125888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3624101e40 msgr2=0x7f3624199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:10.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f362a400700 1 --2- 172.21.15.78:0/179125888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3624101e40 0x7f3624199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:10.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f362a400700 1 -- 172.21.15.78:0/179125888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36100095f0 con 0x7f36241027e0 2024-03-14T09:52:10.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.940+0000 7f3629000700 1 --2- 172.21.15.78:0/179125888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3624101e40 0x7f3624199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:10.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.941+0000 7f362a400700 1 --2- 172.21.15.78:0/179125888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36241027e0 0x7f36241a0750 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f361c007ee0 tx=0x7f361c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:10.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.941+0000 7f3622000700 1 -- 172.21.15.78:0/179125888 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c017070 con 0x7f36241027e0 2024-03-14T09:52:10.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.941+0000 7f362c400700 1 -- 172.21.15.78:0/179125888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36241a12d0 con 0x7f36241027e0 2024-03-14T09:52:10.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.941+0000 7f3622000700 1 -- 172.21.15.78:0/179125888 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f361c00f040 con 0x7f36241027e0 2024-03-14T09:52:10.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.941+0000 7f3622000700 1 -- 172.21.15.78:0/179125888 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361c005550 con 0x7f36241027e0 2024-03-14T09:52:10.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.941+0000 7f362c400700 1 -- 172.21.15.78:0/179125888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36241a1830 con 0x7f36241027e0 2024-03-14T09:52:10.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.942+0000 7f362c400700 1 -- 172.21.15.78:0/179125888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36240679e0 con 0x7f36241027e0 2024-03-14T09:52:10.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.944+0000 7f3622000700 1 -- 172.21.15.78:0/179125888 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f361c00db20 con 0x7f36241027e0 2024-03-14T09:52:10.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.945+0000 7f3622000700 1 --2- 172.21.15.78:0/179125888 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f360c0a0a90 0x7f360c0a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:10.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.945+0000 7f3622000700 1 -- 172.21.15.78:0/179125888 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f361c0cb9a0 con 0x7f36241027e0 2024-03-14T09:52:10.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.945+0000 7f3629a00700 1 --2- 172.21.15.78:0/179125888 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f360c0a0a90 0x7f360c0a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:10.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.946+0000 7f3629a00700 1 --2- 172.21.15.78:0/179125888 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f360c0a0a90 0x7f360c0a2f50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3624101ca0 tx=0x7f361800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:10.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:10.949+0000 7f3622000700 1 -- 172.21.15.78:0/179125888 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f361c095120 con 0x7f36241027e0 2024-03-14T09:52:11.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.175+0000 7f362c400700 1 -- 172.21.15.78:0/179125888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f362410ca50 con 0x7f36241027e0 2024-03-14T09:52:11.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.176+0000 7f3622000700 1 -- 172.21.15.78:0/179125888 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+23925 (secure 0 0 0) 0x7f361c098740 con 0x7f36241027e0 2024-03-14T09:52:11.177 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:11.178 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":83,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:52:10.245498+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:52:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.179+0000 7f362c400700 1 -- 172.21.15.78:0/179125888 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f360c0a0a90 msgr2=0x7f360c0a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.179+0000 7f362c400700 1 --2- 172.21.15.78:0/179125888 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f360c0a0a90 0x7f360c0a2f50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3624101ca0 tx=0x7f361800b410 comp rx=0 tx=0).stop 2024-03-14T09:52:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.179+0000 7f362c400700 1 -- 172.21.15.78:0/179125888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36241027e0 msgr2=0x7f36241a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.179+0000 7f362c400700 1 --2- 172.21.15.78:0/179125888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36241027e0 0x7f36241a0750 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f361c007ee0 tx=0x7f361c00d3b0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.179+0000 7f362c400700 1 -- 172.21.15.78:0/179125888 shutdown_connections 2024-03-14T09:52:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.179+0000 7f362c400700 1 --2- 172.21.15.78:0/179125888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36241027e0 0x7f36241a0750 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.179+0000 7f362c400700 1 --2- 172.21.15.78:0/179125888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3624100c50 0x7f3624199240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.179+0000 7f362c400700 1 --2- 172.21.15.78:0/179125888 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f360c0a0a90 0x7f360c0a2f50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.179+0000 7f362c400700 1 --2- 172.21.15.78:0/179125888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3624101e40 0x7f3624199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.179+0000 7f362c400700 1 -- 172.21.15.78:0/179125888 >> 172.21.15.78:0/179125888 conn(0x7f36240fc1c0 msgr2=0x7f3624072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.179+0000 7f362c400700 1 -- 172.21.15.78:0/179125888 shutdown_connections 2024-03-14T09:52:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.179+0000 7f362c400700 1 -- 172.21.15.78:0/179125888 wait complete. 2024-03-14T09:52:11.200 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:52:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.393+0000 7fa30de00700 1 -- 172.21.15.78:0/3269250201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa308073810 msgr2=0x7fa308073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.393+0000 7fa30de00700 1 --2- 172.21.15.78:0/3269250201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa308073810 0x7fa308073c90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4009990 tx=0x7fa2f4009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.393+0000 7fa30de00700 1 -- 172.21.15.78:0/3269250201 shutdown_connections 2024-03-14T09:52:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.393+0000 7fa30de00700 1 --2- 172.21.15.78:0/3269250201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa308108b10 0x7fa30810afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.393+0000 7fa30de00700 1 --2- 172.21.15.78:0/3269250201 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa308074df0 0x7fa308073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.393+0000 7fa30de00700 1 --2- 172.21.15.78:0/3269250201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa308073810 0x7fa308073c90 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.393+0000 7fa30de00700 1 -- 172.21.15.78:0/3269250201 >> 172.21.15.78:0/3269250201 conn(0x7fa3080fc1c0 msgr2=0x7fa3080fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.393+0000 7fa30de00700 1 -- 172.21.15.78:0/3269250201 shutdown_connections 2024-03-14T09:52:11.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.393+0000 7fa30de00700 1 -- 172.21.15.78:0/3269250201 wait complete. 2024-03-14T09:52:11.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.394+0000 7fa30de00700 1 Processor -- start 2024-03-14T09:52:11.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.394+0000 7fa30de00700 1 -- start start 2024-03-14T09:52:11.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.394+0000 7fa30de00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa308073810 0x7fa308199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:11.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.394+0000 7fa30de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa308074df0 0x7fa308199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:11.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.394+0000 7fa30de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa308108b10 0x7fa3081a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:11.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.394+0000 7fa30de00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3081a0d40 con 0x7fa308074df0 2024-03-14T09:52:11.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.394+0000 7fa30de00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3081a0e80 con 0x7fa308108b10 2024-03-14T09:52:11.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.394+0000 7fa30de00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3081a1010 con 0x7fa308073810 2024-03-14T09:52:11.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.395+0000 7fa306000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa308074df0 0x7fa308199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:11.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.395+0000 7fa306000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa308074df0 0x7fa308199750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37072/0 (socket says 172.21.15.78:37072) 2024-03-14T09:52:11.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.395+0000 7fa306000700 1 -- 172.21.15.78:0/324535450 learned_addr learned my addr 172.21.15.78:0/324535450 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:11.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.395+0000 7fa306a00700 1 --2- 172.21.15.78:0/324535450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa308073810 0x7fa308199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:11.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.395+0000 7fa307400700 1 --2- 172.21.15.78:0/324535450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa308108b10 0x7fa3081a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:11.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.395+0000 7fa306000700 1 -- 172.21.15.78:0/324535450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa308073810 msgr2=0x7fa308199210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:11.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.395+0000 7fa306000700 1 --2- 172.21.15.78:0/324535450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa308073810 0x7fa308199210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.395+0000 7fa306000700 1 -- 172.21.15.78:0/324535450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa308108b10 msgr2=0x7fa3081a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:11.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.395+0000 7fa306000700 1 --2- 172.21.15.78:0/324535450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa308108b10 0x7fa3081a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.395+0000 7fa306000700 1 -- 172.21.15.78:0/324535450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f40095f0 con 0x7fa308074df0 2024-03-14T09:52:11.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.395+0000 7fa307400700 1 --2- 172.21.15.78:0/324535450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa308108b10 0x7fa3081a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:52:11.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.395+0000 7fa306000700 1 --2- 172.21.15.78:0/324535450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa308074df0 0x7fa308199750 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4000c00 tx=0x7fa2f4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:11.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.395+0000 7fa30ca00700 1 -- 172.21.15.78:0/324535450 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f401d070 con 0x7fa308074df0 2024-03-14T09:52:11.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.396+0000 7fa30ca00700 1 -- 172.21.15.78:0/324535450 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa2f4021410 con 0x7fa308074df0 2024-03-14T09:52:11.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.396+0000 7fa30de00700 1 -- 172.21.15.78:0/324535450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3081a1290 con 0x7fa308074df0 2024-03-14T09:52:11.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.396+0000 7fa30ca00700 1 -- 172.21.15.78:0/324535450 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4017650 con 0x7fa308074df0 2024-03-14T09:52:11.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.396+0000 7fa30de00700 1 -- 172.21.15.78:0/324535450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3081a16b0 con 0x7fa308074df0 2024-03-14T09:52:11.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.397+0000 7fa30de00700 1 -- 172.21.15.78:0/324535450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3081078d0 con 0x7fa308074df0 2024-03-14T09:52:11.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.398+0000 7fa30ca00700 1 -- 172.21.15.78:0/324535450 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fa2f4021580 con 0x7fa308074df0 2024-03-14T09:52:11.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.400+0000 7fa30ca00700 1 --2- 172.21.15.78:0/324535450 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa2ec0a0a90 0x7fa2ec0a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:11.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.400+0000 7fa30ca00700 1 -- 172.21.15.78:0/324535450 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa2f40cdcd0 con 0x7fa308074df0 2024-03-14T09:52:11.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.400+0000 7fa306a00700 1 --2- 172.21.15.78:0/324535450 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa2ec0a0a90 0x7fa2ec0a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:11.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.400+0000 7fa306a00700 1 --2- 172.21.15.78:0/324535450 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa2ec0a0a90 0x7fa2ec0a2f50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8009c60 tx=0x7fa2f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:11.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.402+0000 7fa30ca00700 1 -- 172.21.15.78:0/324535450 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa2f40974e0 con 0x7fa308074df0 2024-03-14T09:52:11.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:11 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/4127224268' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-14T09:52:11.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:11 smithi152 ceph-mon[48279]: osdmap e83: 12 total, 12 up, 12 in 2024-03-14T09:52:11.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:11 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/4127224268' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-14T09:52:11.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:11 smithi152 ceph-mon[48279]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-14T09:52:11.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:11 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/889862802' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:52:11.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:11 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/179125888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:11.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:11 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/4127224268' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-14T09:52:11.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:11 smithi132 ceph-mon[47959]: osdmap e83: 12 total, 12 up, 12 in 2024-03-14T09:52:11.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:11 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/4127224268' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-14T09:52:11.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:11 smithi132 ceph-mon[47959]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-14T09:52:11.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:11 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/889862802' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:52:11.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:11 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/179125888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:11.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:11 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/4127224268' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-14T09:52:11.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:11 smithi078 ceph-mon[38143]: osdmap e83: 12 total, 12 up, 12 in 2024-03-14T09:52:11.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:11 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/4127224268' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-14T09:52:11.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:11 smithi078 ceph-mon[38143]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-14T09:52:11.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:11 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/889862802' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:52:11.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:11 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/179125888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:11.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.615+0000 7fa30de00700 1 -- 172.21.15.78:0/324535450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa30810eab0 con 0x7fa308074df0 2024-03-14T09:52:11.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.617+0000 7fa30ca00700 1 -- 172.21.15.78:0/324535450 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+23925 (secure 0 0 0) 0x7fa2f409ab00 con 0x7fa308074df0 2024-03-14T09:52:11.618 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:11.618 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":83,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:52:10.245498+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:52:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.619+0000 7fa30de00700 1 -- 172.21.15.78:0/324535450 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa2ec0a0a90 msgr2=0x7fa2ec0a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.619+0000 7fa30de00700 1 --2- 172.21.15.78:0/324535450 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa2ec0a0a90 0x7fa2ec0a2f50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8009c60 tx=0x7fa2f8009260 comp rx=0 tx=0).stop 2024-03-14T09:52:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.619+0000 7fa30de00700 1 -- 172.21.15.78:0/324535450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa308074df0 msgr2=0x7fa308199750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.619+0000 7fa30de00700 1 --2- 172.21.15.78:0/324535450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa308074df0 0x7fa308199750 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4000c00 tx=0x7fa2f4009580 comp rx=0 tx=0).stop 2024-03-14T09:52:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.619+0000 7fa30de00700 1 -- 172.21.15.78:0/324535450 shutdown_connections 2024-03-14T09:52:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.619+0000 7fa30de00700 1 --2- 172.21.15.78:0/324535450 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa2ec0a0a90 0x7fa2ec0a2f50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.619+0000 7fa30de00700 1 --2- 172.21.15.78:0/324535450 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa308108b10 0x7fa3081a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.619+0000 7fa30de00700 1 --2- 172.21.15.78:0/324535450 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa308073810 0x7fa308199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.619+0000 7fa30de00700 1 --2- 172.21.15.78:0/324535450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa308074df0 0x7fa308199750 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.619+0000 7fa30de00700 1 -- 172.21.15.78:0/324535450 >> 172.21.15.78:0/324535450 conn(0x7fa3080fc1c0 msgr2=0x7fa308104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:11.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.620+0000 7fa30de00700 1 -- 172.21.15.78:0/324535450 shutdown_connections 2024-03-14T09:52:11.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.620+0000 7fa30de00700 1 -- 172.21.15.78:0/324535450 wait complete. 2024-03-14T09:52:11.654 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:52:11.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.846+0000 7ff983c00700 1 -- 172.21.15.78:0/3760135259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101e40 msgr2=0x7ff97c1022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:11.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.846+0000 7ff983c00700 1 --2- 172.21.15.78:0/3760135259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101e40 0x7ff97c1022a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff9680099c0 tx=0x7ff968009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.847+0000 7ff983c00700 1 -- 172.21.15.78:0/3760135259 shutdown_connections 2024-03-14T09:52:11.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.847+0000 7ff983c00700 1 --2- 172.21.15.78:0/3760135259 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff97c1027e0 0x7ff97c109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.847+0000 7ff983c00700 1 --2- 172.21.15.78:0/3760135259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff97c100c50 0x7ff97c101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.847+0000 7ff983c00700 1 --2- 172.21.15.78:0/3760135259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101e40 0x7ff97c1022a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.847+0000 7ff983c00700 1 -- 172.21.15.78:0/3760135259 >> 172.21.15.78:0/3760135259 conn(0x7ff97c0fc1c0 msgr2=0x7ff97c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:11.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.847+0000 7ff983c00700 1 -- 172.21.15.78:0/3760135259 shutdown_connections 2024-03-14T09:52:11.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.847+0000 7ff983c00700 1 -- 172.21.15.78:0/3760135259 wait complete. 2024-03-14T09:52:11.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.848+0000 7ff983c00700 1 Processor -- start 2024-03-14T09:52:11.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.848+0000 7ff983c00700 1 -- start start 2024-03-14T09:52:11.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.848+0000 7ff983c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff97c100c50 0x7ff97c1992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:11.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.848+0000 7ff983c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101e40 0x7ff97c1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:11.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.848+0000 7ff983c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff97c1027e0 0x7ff97c1a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:11.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.848+0000 7ff983c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c1a0d90 con 0x7ff97c101e40 2024-03-14T09:52:11.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.848+0000 7ff983c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c1a0ed0 con 0x7ff97c100c50 2024-03-14T09:52:11.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.848+0000 7ff983c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97c1a1060 con 0x7ff97c1027e0 2024-03-14T09:52:11.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.848+0000 7ff97be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101e40 0x7ff97c1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:11.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.848+0000 7ff97be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101e40 0x7ff97c1997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37084/0 (socket says 172.21.15.78:37084) 2024-03-14T09:52:11.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.848+0000 7ff97be00700 1 -- 172.21.15.78:0/1303058185 learned_addr learned my addr 172.21.15.78:0/1303058185 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:11.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.848+0000 7ff981200700 1 --2- 172.21.15.78:0/1303058185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff97c100c50 0x7ff97c1992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:11.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.849+0000 7ff981c00700 1 --2- 172.21.15.78:0/1303058185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff97c1027e0 0x7ff97c1a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:11.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.849+0000 7ff97be00700 1 -- 172.21.15.78:0/1303058185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff97c1027e0 msgr2=0x7ff97c1a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:11.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.849+0000 7ff97be00700 1 --2- 172.21.15.78:0/1303058185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff97c1027e0 0x7ff97c1a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.849+0000 7ff97be00700 1 -- 172.21.15.78:0/1303058185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff97c100c50 msgr2=0x7ff97c1992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:11.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.849+0000 7ff97be00700 1 --2- 172.21.15.78:0/1303058185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff97c100c50 0x7ff97c1992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:11.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.849+0000 7ff97be00700 1 -- 172.21.15.78:0/1303058185 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9680095f0 con 0x7ff97c101e40 2024-03-14T09:52:11.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.849+0000 7ff981c00700 1 --2- 172.21.15.78:0/1303058185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff97c1027e0 0x7ff97c1a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:52:11.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.849+0000 7ff981200700 1 --2- 172.21.15.78:0/1303058185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff97c100c50 0x7ff97c1992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:52:11.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.849+0000 7ff97be00700 1 --2- 172.21.15.78:0/1303058185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101e40 0x7ff97c1997f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff968003730 tx=0x7ff96800fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:11.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.849+0000 7ff979600700 1 -- 172.21.15.78:0/1303058185 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96801d070 con 0x7ff97c101e40 2024-03-14T09:52:11.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.849+0000 7ff979600700 1 -- 172.21.15.78:0/1303058185 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff96800bda0 con 0x7ff97c101e40 2024-03-14T09:52:11.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.849+0000 7ff979600700 1 -- 172.21.15.78:0/1303058185 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff968017650 con 0x7ff97c101e40 2024-03-14T09:52:11.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.849+0000 7ff983c00700 1 -- 172.21.15.78:0/1303058185 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff97c1a12e0 con 0x7ff97c101e40 2024-03-14T09:52:11.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.849+0000 7ff983c00700 1 -- 172.21.15.78:0/1303058185 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff97c1a16b0 con 0x7ff97c101e40 2024-03-14T09:52:11.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.850+0000 7ff983c00700 1 -- 172.21.15.78:0/1303058185 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff97c067150 con 0x7ff97c101e40 2024-03-14T09:52:11.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.852+0000 7ff979600700 1 -- 172.21.15.78:0/1303058185 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7ff968022470 con 0x7ff97c101e40 2024-03-14T09:52:11.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.853+0000 7ff979600700 1 --2- 172.21.15.78:0/1303058185 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff9640a0b10 0x7ff9640a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:11.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.854+0000 7ff979600700 1 -- 172.21.15.78:0/1303058185 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff9680cd470 con 0x7ff97c101e40 2024-03-14T09:52:11.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.854+0000 7ff981200700 1 --2- 172.21.15.78:0/1303058185 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff9640a0b10 0x7ff9640a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:11.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.854+0000 7ff981200700 1 --2- 172.21.15.78:0/1303058185 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff9640a0b10 0x7ff9640a2fd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff97c101ca0 tx=0x7ff970009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:11.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:11.857+0000 7ff979600700 1 -- 172.21.15.78:0/1303058185 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff968096ce0 con 0x7ff97c101e40 2024-03-14T09:52:12.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.111+0000 7ff983c00700 1 -- 172.21.15.78:0/1303058185 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff97c10c8d0 con 0x7ff97c101e40 2024-03-14T09:52:12.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.112+0000 7ff979600700 1 -- 172.21.15.78:0/1303058185 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10810 (secure 0 0 0) 0x7ff96809a300 con 0x7ff97c101e40 2024-03-14T09:52:12.113 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:12.113 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24527,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24538,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:52:10.245795+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T09:52:12.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.114+0000 7ff983c00700 1 -- 172.21.15.78:0/1303058185 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff9640a0b10 msgr2=0x7ff9640a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:12.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.114+0000 7ff983c00700 1 --2- 172.21.15.78:0/1303058185 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff9640a0b10 0x7ff9640a2fd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff97c101ca0 tx=0x7ff970009260 comp rx=0 tx=0).stop 2024-03-14T09:52:12.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.114+0000 7ff983c00700 1 -- 172.21.15.78:0/1303058185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101e40 msgr2=0x7ff97c1997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:12.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.114+0000 7ff983c00700 1 --2- 172.21.15.78:0/1303058185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101e40 0x7ff97c1997f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff968003730 tx=0x7ff96800fab0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.114+0000 7ff983c00700 1 -- 172.21.15.78:0/1303058185 shutdown_connections 2024-03-14T09:52:12.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.114+0000 7ff983c00700 1 --2- 172.21.15.78:0/1303058185 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff9640a0b10 0x7ff9640a2fd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.114+0000 7ff983c00700 1 --2- 172.21.15.78:0/1303058185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff97c100c50 0x7ff97c1992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.114+0000 7ff983c00700 1 --2- 172.21.15.78:0/1303058185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff97c1027e0 0x7ff97c1a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.114+0000 7ff983c00700 1 --2- 172.21.15.78:0/1303058185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff97c101e40 0x7ff97c1997f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.114+0000 7ff983c00700 1 -- 172.21.15.78:0/1303058185 >> 172.21.15.78:0/1303058185 conn(0x7ff97c0fc1c0 msgr2=0x7ff97c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:12.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.115+0000 7ff983c00700 1 -- 172.21.15.78:0/1303058185 shutdown_connections 2024-03-14T09:52:12.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.115+0000 7ff983c00700 1 -- 172.21.15.78:0/1303058185 wait complete. 2024-03-14T09:52:12.116 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2024-03-14T09:52:12.134 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:52:12.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.327+0000 7f1b87600700 1 -- 172.21.15.78:0/228130167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b801016f0 msgr2=0x7f1b80103ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:12.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.327+0000 7f1b87600700 1 --2- 172.21.15.78:0/228130167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b801016f0 0x7f1b80103ae0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009990 tx=0x7f1b74009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.327+0000 7f1b87600700 1 -- 172.21.15.78:0/228130167 shutdown_connections 2024-03-14T09:52:12.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.327+0000 7f1b87600700 1 --2- 172.21.15.78:0/228130167 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b80106950 0x7f1b80108d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.327+0000 7f1b87600700 1 --2- 172.21.15.78:0/228130167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b801016f0 0x7f1b80103ae0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.327+0000 7f1b87600700 1 --2- 172.21.15.78:0/228130167 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b80104020 0x7f1b80106410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.327+0000 7f1b87600700 1 -- 172.21.15.78:0/228130167 >> 172.21.15.78:0/228130167 conn(0x7f1b800fb0a0 msgr2=0x7f1b800fd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:12.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.327+0000 7f1b87600700 1 -- 172.21.15.78:0/228130167 shutdown_connections 2024-03-14T09:52:12.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.327+0000 7f1b87600700 1 -- 172.21.15.78:0/228130167 wait complete. 2024-03-14T09:52:12.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.328+0000 7f1b87600700 1 Processor -- start 2024-03-14T09:52:12.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.328+0000 7f1b87600700 1 -- start start 2024-03-14T09:52:12.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.328+0000 7f1b87600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b801016f0 0x7f1b80101010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.328+0000 7f1b87600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b80104020 0x7f1b800ff6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.328+0000 7f1b87600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b80106950 0x7f1b800ffbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.328+0000 7f1b87600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b80100380 con 0x7f1b80104020 2024-03-14T09:52:12.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.328+0000 7f1b87600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b801004f0 con 0x7f1b80106950 2024-03-14T09:52:12.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.328+0000 7f1b87600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b801a1030 con 0x7f1b801016f0 2024-03-14T09:52:12.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.329+0000 7f1b7fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b80104020 0x7f1b800ff6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.329+0000 7f1b7fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b80104020 0x7f1b800ff6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37104/0 (socket says 172.21.15.78:37104) 2024-03-14T09:52:12.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.329+0000 7f1b7fe00700 1 -- 172.21.15.78:0/1632017191 learned_addr learned my addr 172.21.15.78:0/1632017191 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:12.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.329+0000 7f1b85600700 1 --2- 172.21.15.78:0/1632017191 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b80106950 0x7f1b800ffbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.329+0000 7f1b84c00700 1 --2- 172.21.15.78:0/1632017191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b801016f0 0x7f1b80101010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.329+0000 7f1b7fe00700 1 -- 172.21.15.78:0/1632017191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b801016f0 msgr2=0x7f1b80101010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:12.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.329+0000 7f1b7fe00700 1 --2- 172.21.15.78:0/1632017191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b801016f0 0x7f1b80101010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.329+0000 7f1b7fe00700 1 -- 172.21.15.78:0/1632017191 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b80106950 msgr2=0x7f1b800ffbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:12.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.329+0000 7f1b7fe00700 1 --2- 172.21.15.78:0/1632017191 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b80106950 0x7f1b800ffbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.329+0000 7f1b7fe00700 1 -- 172.21.15.78:0/1632017191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6c009520 con 0x7f1b80104020 2024-03-14T09:52:12.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.329+0000 7f1b7fe00700 1 --2- 172.21.15.78:0/1632017191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b80104020 0x7f1b800ff6a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c00eb30 tx=0x7f1b6c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:12.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.329+0000 7f1b7d600700 1 -- 172.21.15.78:0/1632017191 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c009880 con 0x7f1b80104020 2024-03-14T09:52:12.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.330+0000 7f1b7d600700 1 -- 172.21.15.78:0/1632017191 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b6c00ce30 con 0x7f1b80104020 2024-03-14T09:52:12.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.330+0000 7f1b87600700 1 -- 172.21.15.78:0/1632017191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b740095f0 con 0x7f1b80104020 2024-03-14T09:52:12.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.330+0000 7f1b7d600700 1 -- 172.21.15.78:0/1632017191 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c003d10 con 0x7f1b80104020 2024-03-14T09:52:12.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.330+0000 7f1b87600700 1 -- 172.21.15.78:0/1632017191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b801a1690 con 0x7f1b80104020 2024-03-14T09:52:12.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.331+0000 7f1b87600700 1 -- 172.21.15.78:0/1632017191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b800fcc20 con 0x7f1b80104020 2024-03-14T09:52:12.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.333+0000 7f1b7d600700 1 -- 172.21.15.78:0/1632017191 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f1b6c00c970 con 0x7f1b80104020 2024-03-14T09:52:12.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.334+0000 7f1b7d600700 1 --2- 172.21.15.78:0/1632017191 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1b680a0b10 0x7f1b680a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.334+0000 7f1b7d600700 1 -- 172.21.15.78:0/1632017191 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1b6c014070 con 0x7f1b80104020 2024-03-14T09:52:12.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.334+0000 7f1b84c00700 1 --2- 172.21.15.78:0/1632017191 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1b680a0b10 0x7f1b680a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.335+0000 7f1b84c00700 1 --2- 172.21.15.78:0/1632017191 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1b680a0b10 0x7f1b680a2fd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1b7400b2e0 tx=0x7f1b74005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:12.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.338+0000 7f1b7d600700 1 -- 172.21.15.78:0/1632017191 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1b6c095dc0 con 0x7f1b80104020 2024-03-14T09:52:12.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:12 smithi152 ceph-mon[48279]: pgmap v238: 129 pgs: 1 creating+peering, 107 active+clean, 21 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2024-03-14T09:52:12.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:12 smithi152 ceph-mon[48279]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-14T09:52:12.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:12 smithi152 ceph-mon[48279]: Cluster is now healthy 2024-03-14T09:52:12.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:12 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/324535450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:12.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:12 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1303058185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:52:12.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:12 smithi132 ceph-mon[47959]: pgmap v238: 129 pgs: 1 creating+peering, 107 active+clean, 21 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2024-03-14T09:52:12.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:12 smithi132 ceph-mon[47959]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-14T09:52:12.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:12 smithi132 ceph-mon[47959]: Cluster is now healthy 2024-03-14T09:52:12.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:12 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/324535450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:12.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:12 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1303058185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:52:12.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:12 smithi078 ceph-mon[38143]: pgmap v238: 129 pgs: 1 creating+peering, 107 active+clean, 21 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2024-03-14T09:52:12.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:12 smithi078 ceph-mon[38143]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-14T09:52:12.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:12 smithi078 ceph-mon[38143]: Cluster is now healthy 2024-03-14T09:52:12.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:12 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/324535450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:12.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:12 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1303058185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:52:12.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.556+0000 7f1b87600700 1 -- 172.21.15.78:0/1632017191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b800679e0 con 0x7f1b80104020 2024-03-14T09:52:12.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.557+0000 7f1b7d600700 1 -- 172.21.15.78:0/1632017191 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+23925 (secure 0 0 0) 0x7f1b6c0993e0 con 0x7f1b80104020 2024-03-14T09:52:12.558 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:12.559 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":83,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:52:10.245498+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:52:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.560+0000 7f1b87600700 1 -- 172.21.15.78:0/1632017191 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1b680a0b10 msgr2=0x7f1b680a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.560+0000 7f1b87600700 1 --2- 172.21.15.78:0/1632017191 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1b680a0b10 0x7f1b680a2fd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1b7400b2e0 tx=0x7f1b74005fd0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.560+0000 7f1b87600700 1 -- 172.21.15.78:0/1632017191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b80104020 msgr2=0x7f1b800ff6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.560+0000 7f1b87600700 1 --2- 172.21.15.78:0/1632017191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b80104020 0x7f1b800ff6a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c00eb30 tx=0x7f1b6c00ee40 comp rx=0 tx=0).stop 2024-03-14T09:52:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.560+0000 7f1b87600700 1 -- 172.21.15.78:0/1632017191 shutdown_connections 2024-03-14T09:52:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.560+0000 7f1b87600700 1 --2- 172.21.15.78:0/1632017191 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1b680a0b10 0x7f1b680a2fd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.560+0000 7f1b87600700 1 --2- 172.21.15.78:0/1632017191 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b80106950 0x7f1b800ffbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.560+0000 7f1b87600700 1 --2- 172.21.15.78:0/1632017191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b801016f0 0x7f1b80101010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.560+0000 7f1b87600700 1 --2- 172.21.15.78:0/1632017191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b80104020 0x7f1b800ff6a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.560+0000 7f1b87600700 1 -- 172.21.15.78:0/1632017191 >> 172.21.15.78:0/1632017191 conn(0x7f1b800fb0a0 msgr2=0x7f1b80102270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.560+0000 7f1b87600700 1 -- 172.21.15.78:0/1632017191 shutdown_connections 2024-03-14T09:52:12.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.560+0000 7f1b87600700 1 -- 172.21.15.78:0/1632017191 wait complete. 2024-03-14T09:52:12.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-14T09:52:05.708711+0000', 'modified': '2024-03-14T09:52:10.245795+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24572}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.132:6839/2192113559', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6838', 'nonce': 2192113559}, {'type': 'v1', 'addr': '172.21.15.132:6839', 'nonce': 2192113559}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-03-14T09:52:12.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-14T09:52:12.580 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-14T09:52:12.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.773+0000 7ffb10c00700 1 -- 172.21.15.78:0/1194431891 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb0c100d40 msgr2=0x7ffb0c101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:12.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.773+0000 7ffb10c00700 1 --2- 172.21.15.78:0/1194431891 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb0c100d40 0x7ffb0c101160 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ffafc0098f0 tx=0x7ffafc009c00 comp rx=0 tx=0).stop 2024-03-14T09:52:12.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.773+0000 7ffb10c00700 1 -- 172.21.15.78:0/1194431891 shutdown_connections 2024-03-14T09:52:12.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.773+0000 7ffb10c00700 1 --2- 172.21.15.78:0/1194431891 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb0c1028d0 0x7ffb0c109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.773+0000 7ffb10c00700 1 --2- 172.21.15.78:0/1194431891 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb0c100d40 0x7ffb0c101160 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.773+0000 7ffb10c00700 1 --2- 172.21.15.78:0/1194431891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb0c101f30 0x7ffb0c102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.773+0000 7ffb10c00700 1 -- 172.21.15.78:0/1194431891 >> 172.21.15.78:0/1194431891 conn(0x7ffb0c0fc2b0 msgr2=0x7ffb0c0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:12.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.773+0000 7ffb10c00700 1 -- 172.21.15.78:0/1194431891 shutdown_connections 2024-03-14T09:52:12.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.773+0000 7ffb10c00700 1 -- 172.21.15.78:0/1194431891 wait complete. 2024-03-14T09:52:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.774+0000 7ffb10c00700 1 Processor -- start 2024-03-14T09:52:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.774+0000 7ffb10c00700 1 -- start start 2024-03-14T09:52:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb10c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb0c100d40 0x7ffb0c197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb10c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb0c101f30 0x7ffb0c1976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb10c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb0c1028d0 0x7ffb0c19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb10c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c19ecb0 con 0x7ffb0c1028d0 2024-03-14T09:52:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb10c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c19edf0 con 0x7ffb0c101f30 2024-03-14T09:52:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb10c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c19ef80 con 0x7ffb0c100d40 2024-03-14T09:52:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb0a800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb0c1028d0 0x7ffb0c19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb09e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb0c100d40 0x7ffb0c197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb09400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb0c101f30 0x7ffb0c1976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb0a800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb0c1028d0 0x7ffb0c19e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37120/0 (socket says 172.21.15.78:37120) 2024-03-14T09:52:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb0a800700 1 -- 172.21.15.78:0/607790676 learned_addr learned my addr 172.21.15.78:0/607790676 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb09e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb0c100d40 0x7ffb0c197180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:52368/0 (socket says 172.21.15.78:52368) 2024-03-14T09:52:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb09400700 1 -- 172.21.15.78:0/607790676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb0c100d40 msgr2=0x7ffb0c197180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb09400700 1 --2- 172.21.15.78:0/607790676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb0c100d40 0x7ffb0c197180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb09400700 1 -- 172.21.15.78:0/607790676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb0c1028d0 msgr2=0x7ffb0c19e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb09400700 1 --2- 172.21.15.78:0/607790676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb0c1028d0 0x7ffb0c19e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb09400700 1 -- 172.21.15.78:0/607790676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffafc009520 con 0x7ffb0c101f30 2024-03-14T09:52:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.775+0000 7ffb0a800700 1 --2- 172.21.15.78:0/607790676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb0c1028d0 0x7ffb0c19e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.776+0000 7ffb09400700 1 --2- 172.21.15.78:0/607790676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb0c101f30 0x7ffb0c1976c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800e9f0 tx=0x7ffaf800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.776+0000 7ffb02a00700 1 -- 172.21.15.78:0/607790676 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf800c9c0 con 0x7ffb0c101f30 2024-03-14T09:52:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.776+0000 7ffb10c00700 1 -- 172.21.15.78:0/607790676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb0c19f260 con 0x7ffb0c101f30 2024-03-14T09:52:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.776+0000 7ffb02a00700 1 -- 172.21.15.78:0/607790676 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffaf8010430 con 0x7ffb0c101f30 2024-03-14T09:52:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.776+0000 7ffb02a00700 1 -- 172.21.15.78:0/607790676 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8018680 con 0x7ffb0c101f30 2024-03-14T09:52:12.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.776+0000 7ffb10c00700 1 -- 172.21.15.78:0/607790676 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb0c19f7c0 con 0x7ffb0c101f30 2024-03-14T09:52:12.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.779+0000 7ffb02a00700 1 -- 172.21.15.78:0/607790676 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7ffaf8010860 con 0x7ffb0c101f30 2024-03-14T09:52:12.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.781+0000 7ffb2471fb80 1 Processor -- start 2024-03-14T09:52:12.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.781+0000 7ffb2471fb80 1 -- start start 2024-03-14T09:52:12.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.781+0000 7ffb02a00700 1 --2- 172.21.15.78:0/607790676 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffaf00a0a90 0x7ffaf00a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.781+0000 7ffb2471fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa32fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.781+0000 7ffb2471fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa33510 0x5647afa33e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.781+0000 7ffb2471fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa343a0 0x5647afa34820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.781+0000 7ffb2471fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647af987940 con 0x5647afa32bb0 2024-03-14T09:52:12.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.781+0000 7ffb02a00700 1 -- 172.21.15.78:0/607790676 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ffaf8014070 con 0x7ffb0c101f30 2024-03-14T09:52:12.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.781+0000 7ffb2471fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647af985300 con 0x5647afa343a0 2024-03-14T09:52:12.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.781+0000 7ffb2471fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647af980bf0 con 0x5647afa33510 2024-03-14T09:52:12.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.781+0000 7ffb0a800700 1 --2- 172.21.15.78:0/607790676 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffaf00a0a90 0x7ffaf00a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.782+0000 7ffb09e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa32fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.782+0000 7ffb09400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa33510 0x5647afa33e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.782+0000 7ffb09e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa32fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37128/0 (socket says 172.21.15.78:37128) 2024-03-14T09:52:12.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.782+0000 7ffb09e00700 1 -- 172.21.15.78:0/3612247273 learned_addr learned my addr 172.21.15.78:0/3612247273 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:12.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.782+0000 7ffb0a800700 1 --2- 172.21.15.78:0/3612247273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa343a0 0x5647afa34820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.782+0000 7ffb09400700 1 -- 172.21.15.78:0/3612247273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa343a0 msgr2=0x5647afa34820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:52:12.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.782+0000 7ffb09400700 1 --2- 172.21.15.78:0/3612247273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa343a0 0x5647afa34820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.782+0000 7ffb09400700 1 -- 172.21.15.78:0/3612247273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 msgr2=0x5647afa32fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:52:12.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.782+0000 7ffb09400700 1 --2- 172.21.15.78:0/3612247273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa32fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.782+0000 7ffb09400700 1 -- 172.21.15.78:0/3612247273 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647af802360 con 0x5647afa33510 2024-03-14T09:52:12.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.782+0000 7ffb0a800700 1 --2- 172.21.15.78:0/3612247273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa343a0 0x5647afa34820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:52:12.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.782+0000 7ffb0a800700 1 --2- 172.21.15.78:0/607790676 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffaf00a0a90 0x7ffaf00a2f50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009e10 tx=0x7ffb04007be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:12.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.782+0000 7ffb09400700 1 --2- 172.21.15.78:0/3612247273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa33510 0x5647afa33e10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800c5b0 tx=0x7ffaf80996f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ff91f75b91ee9bf server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:12.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.782+0000 7ffaf7000700 1 -- 172.21.15.78:0/3612247273 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf80997f0 con 0x5647afa33510 2024-03-14T09:52:12.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.783+0000 7ffaf7000700 1 -- 172.21.15.78:0/3612247273 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffaf809d620 con 0x5647afa33510 2024-03-14T09:52:12.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.783+0000 7ffaf7000700 1 -- 172.21.15.78:0/3612247273 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf80b92b0 con 0x5647afa33510 2024-03-14T09:52:12.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.783+0000 7ffb2471fb80 1 -- 172.21.15.78:0/3612247273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa33510 msgr2=0x5647afa33e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:12.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.783+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/3612247273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa33510 0x5647afa33e10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800c5b0 tx=0x7ffaf80996f0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.783+0000 7ffb2471fb80 1 -- 172.21.15.78:0/3612247273 shutdown_connections 2024-03-14T09:52:12.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.783+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/3612247273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa343a0 0x5647afa34820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.783+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/3612247273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa32fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.783+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/3612247273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa33510 0x5647afa33e10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.783+0000 7ffb2471fb80 1 -- 172.21.15.78:0/3612247273 >> 172.21.15.78:0/3612247273 conn(0x5647af9a41d0 msgr2=0x5647afa29580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:12.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.783+0000 7ffb2471fb80 1 -- 172.21.15.78:0/3612247273 shutdown_connections 2024-03-14T09:52:12.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.783+0000 7ffb2471fb80 1 -- 172.21.15.78:0/3612247273 wait complete. 2024-03-14T09:52:12.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.784+0000 7ffb2471fb80 1 -- start start 2024-03-14T09:52:12.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.784+0000 7ffb2471fb80 1 Processor -- start 2024-03-14T09:52:12.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.784+0000 7ffb2471fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa3cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.784+0000 7ffb2471fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa33510 0x5647afa3cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.784+0000 7ffb2471fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa343a0 0x5647afa44030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.784+0000 7ffb2471fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647af91cd30 con 0x5647afa32bb0 2024-03-14T09:52:12.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.784+0000 7ffb2471fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647af91c9d0 con 0x5647afa33510 2024-03-14T09:52:12.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.784+0000 7ffb2471fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647af804650 con 0x5647afa343a0 2024-03-14T09:52:12.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.784+0000 7ffb2471fb80 10 client.0 _ll_register_callbacks cb 0x5647af788350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:52:12.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.785+0000 7ffb09e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa3cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.785+0000 7ffb09e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa3cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37132/0 (socket says 172.21.15.78:37132) 2024-03-14T09:52:12.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.785+0000 7ffb09e00700 1 -- 172.21.15.78:0/636684202 learned_addr learned my addr 172.21.15.78:0/636684202 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:12.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.785+0000 7ffb0a800700 1 --2- 172.21.15.78:0/636684202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa343a0 0x5647afa44030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.785+0000 7ffb09400700 1 --2- 172.21.15.78:0/636684202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa33510 0x5647afa3cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.785+0000 7ffb0a800700 1 -- 172.21.15.78:0/636684202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa33510 msgr2=0x5647afa3cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:52:12.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.785+0000 7ffb0a800700 1 --2- 172.21.15.78:0/636684202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa33510 0x5647afa3cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.785+0000 7ffb0a800700 1 -- 172.21.15.78:0/636684202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 msgr2=0x5647afa3cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:52:12.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.785+0000 7ffb0a800700 1 --2- 172.21.15.78:0/636684202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa3cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.785+0000 7ffb0a800700 1 -- 172.21.15.78:0/636684202 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb0c19f7c0 con 0x5647afa343a0 2024-03-14T09:52:12.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.785+0000 7ffb09e00700 1 --2- 172.21.15.78:0/636684202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa3cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:12.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.785+0000 7ffb0a800700 1 --2- 172.21.15.78:0/636684202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa343a0 0x5647afa44030 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ffb040062a0 tx=0x7ffb04004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78ad394606632443 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:12.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.785+0000 7ffaeb400700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:52:12.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.786+0000 7ffaeb400700 1 -- 172.21.15.78:0/636684202 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04025470 con 0x5647afa343a0 2024-03-14T09:52:12.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.786+0000 7ffaeb400700 1 -- 172.21.15.78:0/636684202 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ffb040255d0 con 0x5647afa343a0 2024-03-14T09:52:12.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.786+0000 7ffb2471fb80 1 -- 172.21.15.78:0/636684202 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5647afa216d0 con 0x5647afa343a0 2024-03-14T09:52:12.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.786+0000 7ffaeb400700 1 -- 172.21.15.78:0/636684202 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb04025790 con 0x5647afa343a0 2024-03-14T09:52:12.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.786+0000 7ffb2471fb80 10 client.24670 fetch_fsmap learned FSMap version 17 2024-03-14T09:52:12.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.786+0000 7ffb2471fb80 1 -- 172.21.15.78:0/636684202 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x5647af7aeb10 con 0x5647afa343a0 2024-03-14T09:52:12.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.786+0000 7ffaeb400700 1 -- 172.21.15.78:0/636684202 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5647afa216d0 con 0x5647afa343a0 2024-03-14T09:52:12.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.786+0000 7ffaeb400700 1 -- 172.21.15.78:0/636684202 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7ffb04018800 con 0x5647afa343a0 2024-03-14T09:52:12.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.786+0000 7ffb2471fb80 10 client.24670 fetch_fsmap finished waiting for FSMap version 17 2024-03-14T09:52:12.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.786+0000 7ffb2471fb80 10 client.24670 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-14T09:52:12.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.786+0000 7ffb2471fb80 20 client.24670 populate_metadata read hostname 'smithi078' 2024-03-14T09:52:12.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.787+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/636684202 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5647afa471d0 0x5647afa4cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.787+0000 7ffb2471fb80 4 client.24670 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:52:12.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.787+0000 7ffb2471fb80 1 -- 172.21.15.78:0/636684202 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5647af9796b0 con 0x5647afa471d0 2024-03-14T09:52:12.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.787+0000 7ffb09e00700 1 --2- 172.21.15.78:0/636684202 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5647afa471d0 0x5647afa4cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.787+0000 7ffb09e00700 1 --2- 172.21.15.78:0/636684202 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5647afa471d0 0x5647afa4cff0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d80287fabc542d9e server_cookie=84cad79eaefac312 in_seq=0 out_seq=0 2024-03-14T09:52:12.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.788+0000 7ffaeb400700 10 client.24670 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:52:12.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.788+0000 7ffaeb400700 1 -- 172.21.15.78:0/636684202 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5647af9796b0 con 0x5647afa471d0 2024-03-14T09:52:12.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.788+0000 7ffaeb400700 10 client.24670 handle_command_reply: tid=0 2024-03-14T09:52:12.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.788+0000 7ffb2471fb80 1 client.24670 shutdown 2024-03-14T09:52:12.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.789+0000 7ffb2471fb80 1 -- 172.21.15.78:0/636684202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa343a0 msgr2=0x5647afa44030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:12.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.789+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/636684202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa343a0 0x5647afa44030 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ffb040062a0 tx=0x7ffb04004b40 comp rx=0 tx=0).stop 2024-03-14T09:52:12.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.789+0000 7ffb2471fb80 1 -- 172.21.15.78:0/636684202 shutdown_connections 2024-03-14T09:52:12.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.789+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/636684202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa343a0 0x5647afa44030 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.789+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/636684202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa3cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.789+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/636684202 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5647afa471d0 0x5647afa4cff0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.789+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/636684202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa33510 0x5647afa3cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.789+0000 7ffaeb400700 0 client.24670 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:52:12.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.789+0000 7ffb2471fb80 1 -- 172.21.15.78:0/636684202 >> 172.21.15.78:0/636684202 conn(0x5647af9a41d0 msgr2=0x5647afa28bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:12.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.789+0000 7ffb2471fb80 1 -- 172.21.15.78:0/636684202 shutdown_connections 2024-03-14T09:52:12.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.789+0000 7ffb2471fb80 1 -- 172.21.15.78:0/636684202 wait complete. 2024-03-14T09:52:12.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.789+0000 7ffb2471fb80 20 client.24670 trim_cache size 0 max 16384 2024-03-14T09:52:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.806+0000 7ffb2471fb80 1 Processor -- start 2024-03-14T09:52:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.806+0000 7ffb2471fb80 1 -- start start 2024-03-14T09:52:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.806+0000 7ffb2471fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa32bb0 0x5647afa41fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.806+0000 7ffb2471fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa33510 0x5647afa3a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.806+0000 7ffb2471fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa343a0 0x5647afa37330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.806+0000 7ffb2471fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647af9799c0 con 0x5647afa33510 2024-03-14T09:52:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.806+0000 7ffb2471fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647af97d880 con 0x5647afa32bb0 2024-03-14T09:52:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.806+0000 7ffb2471fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647af93ead0 con 0x5647afa343a0 2024-03-14T09:52:12.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.807+0000 7ffb09400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa33510 0x5647afa3a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.807+0000 7ffb09400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa33510 0x5647afa3a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37136/0 (socket says 172.21.15.78:37136) 2024-03-14T09:52:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.807+0000 7ffb09400700 1 -- 172.21.15.78:0/2265736320 learned_addr learned my addr 172.21.15.78:0/2265736320 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.807+0000 7ffb0a800700 1 --2- 172.21.15.78:0/2265736320 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa343a0 0x5647afa37330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.807+0000 7ffb09e00700 1 --2- 172.21.15.78:0/2265736320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa32bb0 0x5647afa41fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.807+0000 7ffb09400700 1 -- 172.21.15.78:0/2265736320 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa343a0 msgr2=0x5647afa37330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:52:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.807+0000 7ffb09400700 1 --2- 172.21.15.78:0/2265736320 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa343a0 0x5647afa37330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.807+0000 7ffb09400700 1 -- 172.21.15.78:0/2265736320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa32bb0 msgr2=0x5647afa41fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:52:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.807+0000 7ffb09400700 1 --2- 172.21.15.78:0/2265736320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa32bb0 0x5647afa41fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.807+0000 7ffb09400700 1 -- 172.21.15.78:0/2265736320 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647af946e50 con 0x5647afa33510 2024-03-14T09:52:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.807+0000 7ffb09400700 1 --2- 172.21.15.78:0/2265736320 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa33510 0x5647afa3a8e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8095cf0 tx=0x7ffaf80995c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8385e1ae073198c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.807+0000 7ffaf7a00700 1 -- 172.21.15.78:0/2265736320 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf80997f0 con 0x5647afa33510 2024-03-14T09:52:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.807+0000 7ffaf7a00700 1 -- 172.21.15.78:0/2265736320 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffaf80a1c70 con 0x5647afa33510 2024-03-14T09:52:12.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.807+0000 7ffaf7a00700 1 -- 172.21.15.78:0/2265736320 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8010680 con 0x5647afa33510 2024-03-14T09:52:12.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.808+0000 7ffb2471fb80 1 -- 172.21.15.78:0/2265736320 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa33510 msgr2=0x5647afa3a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:12.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.808+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/2265736320 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa33510 0x5647afa3a8e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8095cf0 tx=0x7ffaf80995c0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.808+0000 7ffb2471fb80 1 -- 172.21.15.78:0/2265736320 shutdown_connections 2024-03-14T09:52:12.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.808+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/2265736320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa32bb0 0x5647afa41fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.808+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/2265736320 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa343a0 0x5647afa37330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.808+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/2265736320 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa33510 0x5647afa3a8e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.808+0000 7ffb2471fb80 1 -- 172.21.15.78:0/2265736320 >> 172.21.15.78:0/2265736320 conn(0x5647af9a41d0 msgr2=0x5647afa25630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:12.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.808+0000 7ffb2471fb80 1 -- 172.21.15.78:0/2265736320 shutdown_connections 2024-03-14T09:52:12.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.808+0000 7ffb2471fb80 1 -- 172.21.15.78:0/2265736320 wait complete. 2024-03-14T09:52:12.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.808+0000 7ffb2471fb80 1 -- start start 2024-03-14T09:52:12.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.809+0000 7ffb2471fb80 1 Processor -- start 2024-03-14T09:52:12.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.809+0000 7ffb2471fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa41f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.809+0000 7ffb2471fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa33510 0x5647afa35460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.809+0000 7ffb2471fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa343a0 0x5647afa359a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.809+0000 7ffb2471fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647afa36eb0 con 0x5647afa32bb0 2024-03-14T09:52:12.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.809+0000 7ffb2471fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647afa35ee0 con 0x5647afa33510 2024-03-14T09:52:12.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.809+0000 7ffb2471fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647afa360a0 con 0x5647afa343a0 2024-03-14T09:52:12.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.809+0000 7ffb2471fb80 10 client.0 _ll_register_callbacks cb 0x7ffb0401e050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:52:12.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.810+0000 7ffb09e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa41f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.810+0000 7ffb09e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa41f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37144/0 (socket says 172.21.15.78:37144) 2024-03-14T09:52:12.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.810+0000 7ffb09e00700 1 -- 172.21.15.78:0/1082577430 learned_addr learned my addr 172.21.15.78:0/1082577430 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:12.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.810+0000 7ffb09400700 1 --2- 172.21.15.78:0/1082577430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa33510 0x5647afa35460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.810+0000 7ffb0a800700 1 --2- 172.21.15.78:0/1082577430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa343a0 0x5647afa359a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.810+0000 7ffb09e00700 1 -- 172.21.15.78:0/1082577430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa343a0 msgr2=0x5647afa359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:52:12.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.810+0000 7ffb09e00700 1 --2- 172.21.15.78:0/1082577430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa343a0 0x5647afa359a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.810+0000 7ffb09e00700 1 -- 172.21.15.78:0/1082577430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa33510 msgr2=0x5647afa35460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:52:12.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.810+0000 7ffb09e00700 1 --2- 172.21.15.78:0/1082577430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa33510 0x5647afa35460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.810+0000 7ffb09e00700 1 -- 172.21.15.78:0/1082577430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf80b1040 con 0x5647afa32bb0 2024-03-14T09:52:12.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.810+0000 7ffb09e00700 1 --2- 172.21.15.78:0/1082577430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa41f40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ffafc009340 tx=0x7ffafc0271a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c517d9fd2d74d4ca server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:12.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.810+0000 7ffaeb400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:52:12.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.811+0000 7ffaeb400700 1 -- 172.21.15.78:0/1082577430 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc027b00 con 0x5647afa32bb0 2024-03-14T09:52:12.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.811+0000 7ffaeb400700 1 -- 172.21.15.78:0/1082577430 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ffafc027c60 con 0x5647afa32bb0 2024-03-14T09:52:12.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.811+0000 7ffb2471fb80 1 -- 172.21.15.78:0/1082577430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffaf809d620 con 0x5647afa32bb0 2024-03-14T09:52:12.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.811+0000 7ffaeb400700 1 -- 172.21.15.78:0/1082577430 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffafc027e20 con 0x5647afa32bb0 2024-03-14T09:52:12.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.811+0000 7ffb2471fb80 10 client.14658 fetch_fsmap learned FSMap version 17 2024-03-14T09:52:12.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.811+0000 7ffb2471fb80 1 -- 172.21.15.78:0/1082577430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x5647af806c80 con 0x5647afa32bb0 2024-03-14T09:52:12.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.811+0000 7ffaeb400700 1 -- 172.21.15.78:0/1082577430 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffaf809d620 con 0x5647afa32bb0 2024-03-14T09:52:12.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.811+0000 7ffaeb400700 1 -- 172.21.15.78:0/1082577430 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7ffafc009920 con 0x5647afa32bb0 2024-03-14T09:52:12.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.811+0000 7ffb2471fb80 10 client.14658 fetch_fsmap finished waiting for FSMap version 17 2024-03-14T09:52:12.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.811+0000 7ffb2471fb80 10 client.14658 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-14T09:52:12.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.811+0000 7ffb2471fb80 20 client.14658 populate_metadata read hostname 'smithi078' 2024-03-14T09:52:12.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.811+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/1082577430 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5647afa9ede0 0x5647afa9b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:12.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.811+0000 7ffb2471fb80 4 client.14658 mds_command: new command op to 24572 tid=0[{"prefix": "status"}] 2024-03-14T09:52:12.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.811+0000 7ffb2471fb80 1 -- 172.21.15.78:0/1082577430 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5647af7cfca0 con 0x5647afa9ede0 2024-03-14T09:52:12.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.812+0000 7ffb09400700 1 --2- 172.21.15.78:0/1082577430 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5647afa9ede0 0x5647afa9b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:12.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.812+0000 7ffaeb400700 10 client.14658 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:52:12.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.812+0000 7ffb09400700 1 --2- 172.21.15.78:0/1082577430 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5647afa9ede0 0x5647afa9b730 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51d5f708a5441c7f server_cookie=e1f64f575bd02e10 in_seq=0 out_seq=0 2024-03-14T09:52:12.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.812+0000 7ffaeb400700 1 -- 172.21.15.78:0/1082577430 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x5647af7cfca0 con 0x5647afa9ede0 2024-03-14T09:52:12.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.812+0000 7ffaeb400700 10 client.14658 handle_command_reply: tid=0 2024-03-14T09:52:12.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.812+0000 7ffb2471fb80 1 client.14658 shutdown 2024-03-14T09:52:12.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.813+0000 7ffb2471fb80 1 -- 172.21.15.78:0/1082577430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 msgr2=0x5647afa41f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:12.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.813+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/1082577430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa41f40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ffafc009340 tx=0x7ffafc0271a0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.813+0000 7ffb2471fb80 1 -- 172.21.15.78:0/1082577430 shutdown_connections 2024-03-14T09:52:12.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.813+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/1082577430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5647afa343a0 0x5647afa359a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.813+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/1082577430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5647afa32bb0 0x5647afa41f40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.813+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/1082577430 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5647afa9ede0 0x5647afa9b730 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.813+0000 7ffb2471fb80 1 --2- 172.21.15.78:0/1082577430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5647afa33510 0x5647afa35460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.813+0000 7ffb2471fb80 1 -- 172.21.15.78:0/1082577430 >> 172.21.15.78:0/1082577430 conn(0x5647af9a41d0 msgr2=0x5647afa3a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:12.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.813+0000 7ffaeb400700 0 client.14658 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:52:12.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.813+0000 7ffb2471fb80 1 -- 172.21.15.78:0/1082577430 shutdown_connections 2024-03-14T09:52:12.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.813+0000 7ffb2471fb80 1 -- 172.21.15.78:0/1082577430 wait complete. 2024-03-14T09:52:12.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.813+0000 7ffb2471fb80 20 client.14658 trim_cache size 0 max 16384 2024-03-14T09:52:12.820 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:52:12.820 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "83662c4e-e1e7-11ee-95c9-87774f69a715", 2024-03-14T09:52:12.820 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-03-14T09:52:12.820 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24572, 2024-03-14T09:52:12.820 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-14T09:52:12.820 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-14T09:52:12.821 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-14T09:52:12.821 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 7.1015008469999996, 2024-03-14T09:52:12.821 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 17, 2024-03-14T09:52:12.821 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 83, 2024-03-14T09:52:12.821 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 82, 2024-03-14T09:52:12.821 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 14.078610971 2024-03-14T09:52:12.821 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:52:12.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.813+0000 7ffb10c00700 1 -- 172.21.15.78:0/607790676 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffaf00a0a90 msgr2=0x7ffaf00a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:12.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.813+0000 7ffb10c00700 1 --2- 172.21.15.78:0/607790676 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffaf00a0a90 0x7ffaf00a2f50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009e10 tx=0x7ffb04007be0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.813+0000 7ffb10c00700 1 -- 172.21.15.78:0/607790676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb0c101f30 msgr2=0x7ffb0c1976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:12.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.813+0000 7ffb10c00700 1 --2- 172.21.15.78:0/607790676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb0c101f30 0x7ffb0c1976c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800e9f0 tx=0x7ffaf800ed00 comp rx=0 tx=0).stop 2024-03-14T09:52:12.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.814+0000 7ffb10c00700 1 -- 172.21.15.78:0/607790676 shutdown_connections 2024-03-14T09:52:12.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.814+0000 7ffb10c00700 1 --2- 172.21.15.78:0/607790676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb0c1028d0 0x7ffb0c19e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.814+0000 7ffb10c00700 1 --2- 172.21.15.78:0/607790676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb0c100d40 0x7ffb0c197180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.814+0000 7ffb10c00700 1 --2- 172.21.15.78:0/607790676 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffaf00a0a90 0x7ffaf00a2f50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.814+0000 7ffb10c00700 1 --2- 172.21.15.78:0/607790676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb0c101f30 0x7ffb0c1976c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:12.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.814+0000 7ffb10c00700 1 -- 172.21.15.78:0/607790676 >> 172.21.15.78:0/607790676 conn(0x7ffb0c0fc2b0 msgr2=0x7ffb0c1053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:12.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.814+0000 7ffb10c00700 1 -- 172.21.15.78:0/607790676 shutdown_connections 2024-03-14T09:52:12.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:12.814+0000 7ffb10c00700 1 -- 172.21.15.78:0/607790676 wait complete. 2024-03-14T09:52:12.833 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-14T09:52:12.833 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-14T09:52:12.833 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-14T09:52:13.186 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:13.498 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:13 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1632017191' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:13.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:13 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1632017191' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:13.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:13 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1632017191' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:14.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.390+0000 7f2da9200700 1 -- 172.21.15.78:0/3230203941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2da4073b00 msgr2=0x7f2da410d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:14.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.390+0000 7f2da9200700 1 --2- 172.21.15.78:0/3230203941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2da4073b00 0x7f2da410d050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2d90009990 tx=0x7f2d90009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:14.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.390+0000 7f2da9200700 1 -- 172.21.15.78:0/3230203941 shutdown_connections 2024-03-14T09:52:14.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.390+0000 7f2da9200700 1 --2- 172.21.15.78:0/3230203941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2da40731e0 0x7f2da40735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:14.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.390+0000 7f2da9200700 1 --2- 172.21.15.78:0/3230203941 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2da410d590 0x7f2da410fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:14.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.390+0000 7f2da9200700 1 --2- 172.21.15.78:0/3230203941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2da4073b00 0x7f2da410d050 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:14.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.390+0000 7f2da9200700 1 -- 172.21.15.78:0/3230203941 >> 172.21.15.78:0/3230203941 conn(0x7f2da40fc800 msgr2=0x7f2da40fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:14.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.390+0000 7f2da9200700 1 -- 172.21.15.78:0/3230203941 shutdown_connections 2024-03-14T09:52:14.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.390+0000 7f2da9200700 1 -- 172.21.15.78:0/3230203941 wait complete. 2024-03-14T09:52:14.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.391+0000 7f2da9200700 1 Processor -- start 2024-03-14T09:52:14.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.391+0000 7f2da9200700 1 -- start start 2024-03-14T09:52:14.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.392+0000 7f2da9200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2da40731e0 0x7f2da419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:14.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.392+0000 7f2da9200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2da4073b00 0x7f2da419faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:14.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.392+0000 7f2da9200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2da410d590 0x7f2da41996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:14.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.392+0000 7f2da9200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da4199e60 con 0x7f2da4073b00 2024-03-14T09:52:14.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.392+0000 7f2da9200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da4199fd0 con 0x7f2da40731e0 2024-03-14T09:52:14.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.392+0000 7f2da9200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da419a190 con 0x7f2da410d590 2024-03-14T09:52:14.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.392+0000 7f2da1400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2da4073b00 0x7f2da419faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:14.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.392+0000 7f2da1400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2da4073b00 0x7f2da419faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44460/0 (socket says 172.21.15.78:44460) 2024-03-14T09:52:14.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.392+0000 7f2da1400700 1 -- 172.21.15.78:0/2231059159 learned_addr learned my addr 172.21.15.78:0/2231059159 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:14.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.392+0000 7f2da1e00700 1 --2- 172.21.15.78:0/2231059159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2da40731e0 0x7f2da419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:14.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.392+0000 7f2da2800700 1 --2- 172.21.15.78:0/2231059159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2da410d590 0x7f2da41996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:14.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.392+0000 7f2da1400700 1 -- 172.21.15.78:0/2231059159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2da410d590 msgr2=0x7f2da41996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:14.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.392+0000 7f2da1400700 1 --2- 172.21.15.78:0/2231059159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2da410d590 0x7f2da41996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:14.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.392+0000 7f2da1400700 1 -- 172.21.15.78:0/2231059159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2da40731e0 msgr2=0x7f2da419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:14.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.392+0000 7f2da1400700 1 --2- 172.21.15.78:0/2231059159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2da40731e0 0x7f2da419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:14.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.392+0000 7f2da1400700 1 -- 172.21.15.78:0/2231059159 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d900095f0 con 0x7f2da4073b00 2024-03-14T09:52:14.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.393+0000 7f2da1400700 1 --2- 172.21.15.78:0/2231059159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2da4073b00 0x7f2da419faf0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2d90005910 tx=0x7f2d90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:14.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.393+0000 7f2d9aa00700 1 -- 172.21.15.78:0/2231059159 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9001d070 con 0x7f2da4073b00 2024-03-14T09:52:14.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.393+0000 7f2d9aa00700 1 -- 172.21.15.78:0/2231059159 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d90022470 con 0x7f2da4073b00 2024-03-14T09:52:14.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.393+0000 7f2da9200700 1 -- 172.21.15.78:0/2231059159 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da419a410 con 0x7f2da4073b00 2024-03-14T09:52:14.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.393+0000 7f2d9aa00700 1 -- 172.21.15.78:0/2231059159 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d90017650 con 0x7f2da4073b00 2024-03-14T09:52:14.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.393+0000 7f2da9200700 1 -- 172.21.15.78:0/2231059159 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da41a6530 con 0x7f2da4073b00 2024-03-14T09:52:14.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.394+0000 7f2da9200700 1 -- 172.21.15.78:0/2231059159 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da41087a0 con 0x7f2da4073b00 2024-03-14T09:52:14.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.396+0000 7f2d9aa00700 1 -- 172.21.15.78:0/2231059159 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f2d900225e0 con 0x7f2da4073b00 2024-03-14T09:52:14.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.397+0000 7f2d9aa00700 1 --2- 172.21.15.78:0/2231059159 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2d880a0a90 0x7f2d880a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:14.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.397+0000 7f2d9aa00700 1 -- 172.21.15.78:0/2231059159 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2d900cd570 con 0x7f2da4073b00 2024-03-14T09:52:14.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.397+0000 7f2da1e00700 1 --2- 172.21.15.78:0/2231059159 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2d880a0a90 0x7f2d880a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:14.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.398+0000 7f2da1e00700 1 --2- 172.21.15.78:0/2231059159 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2d880a0a90 0x7f2d880a2f50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2d94009c60 tx=0x7f2d94009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:14.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.401+0000 7f2d9aa00700 1 -- 172.21.15.78:0/2231059159 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2d90096d80 con 0x7f2da4073b00 2024-03-14T09:52:14.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:14 smithi152 ceph-mon[48279]: pgmap v239: 129 pgs: 1 creating+peering, 128 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 575 B/s wr, 2 op/s 2024-03-14T09:52:14.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:14 smithi132 ceph-mon[47959]: pgmap v239: 129 pgs: 1 creating+peering, 128 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 575 B/s wr, 2 op/s 2024-03-14T09:52:14.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:14 smithi078 ceph-mon[38143]: pgmap v239: 129 pgs: 1 creating+peering, 128 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 575 B/s wr, 2 op/s 2024-03-14T09:52:14.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.647+0000 7f2da9200700 1 -- 172.21.15.78:0/2231059159 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2da419ab40 con 0x7f2da4073b00 2024-03-14T09:52:14.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.648+0000 7f2d9aa00700 1 -- 172.21.15.78:0/2231059159 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7f2d9009a3a0 con 0x7f2da4073b00 2024-03-14T09:52:14.648 INFO:teuthology.orchestra.run.smithi078.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-14T09:52:14.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.650+0000 7f2da9200700 1 -- 172.21.15.78:0/2231059159 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2d880a0a90 msgr2=0x7f2d880a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:14.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.650+0000 7f2da9200700 1 --2- 172.21.15.78:0/2231059159 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2d880a0a90 0x7f2d880a2f50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2d94009c60 tx=0x7f2d94009260 comp rx=0 tx=0).stop 2024-03-14T09:52:14.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.650+0000 7f2da9200700 1 -- 172.21.15.78:0/2231059159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2da4073b00 msgr2=0x7f2da419faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:14.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.650+0000 7f2da9200700 1 --2- 172.21.15.78:0/2231059159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2da4073b00 0x7f2da419faf0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2d90005910 tx=0x7f2d90004730 comp rx=0 tx=0).stop 2024-03-14T09:52:14.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.650+0000 7f2da9200700 1 -- 172.21.15.78:0/2231059159 shutdown_connections 2024-03-14T09:52:14.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.650+0000 7f2da9200700 1 --2- 172.21.15.78:0/2231059159 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2d880a0a90 0x7f2d880a2f50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:14.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.650+0000 7f2da9200700 1 --2- 172.21.15.78:0/2231059159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2da40731e0 0x7f2da419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:14.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.650+0000 7f2da9200700 1 --2- 172.21.15.78:0/2231059159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2da410d590 0x7f2da41996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:14.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.650+0000 7f2da9200700 1 --2- 172.21.15.78:0/2231059159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2da4073b00 0x7f2da419faf0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:14.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.650+0000 7f2da9200700 1 -- 172.21.15.78:0/2231059159 >> 172.21.15.78:0/2231059159 conn(0x7f2da40fc800 msgr2=0x7f2da40fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:14.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.650+0000 7f2da9200700 1 -- 172.21.15.78:0/2231059159 shutdown_connections 2024-03-14T09:52:14.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:14.650+0000 7f2da9200700 1 -- 172.21.15.78:0/2231059159 wait complete. 2024-03-14T09:52:15.012 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-14T09:52:15.012 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-14T09:52:15.012 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2024-03-14T09:52:15.012 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-14T09:52:15.012 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-14T09:52:15.012 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-14T09:52:15.012 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-14T09:52:15.012 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-14T09:52:15.012 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-14T09:52:15.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:15.013 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2024-03-14T09:52:15.056 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-14T09:52:15.056 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-14T09:52:15.056 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2024-03-14T09:52:15.056 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-03-14T09:52:15.057 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2024-03-14T09:52:15.057 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-03-14T09:52:15.057 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-14T09:52:15.057 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2024-03-14T09:52:15.057 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f0 2024-03-14T09:52:15.057 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-14T09:52:15.057 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2024-03-14T09:52:15.057 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f1 2024-03-14T09:52:15.057 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-14T09:52:15.057 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2024-03-14T09:52:15.057 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-14T09:52:15.057 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2024-03-14T09:52:15.057 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-14T09:52:15.057 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 41940sec preferred_lft 41940sec 2024-03-14T09:52:15.057 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2024-03-14T09:52:15.058 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-03-14T09:52:15.059 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-14T09:52:15.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:15.059 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-14T09:52:15.059 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-14T09:52:15.059 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add name ceph-brx type bridge 2024-03-14T09:52:15.060 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr flush dev ceph-brx 2024-03-14T09:52:15.060 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set ceph-brx up 2024-03-14T09:52:15.060 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-14T09:52:15.060 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-14T09:52:15.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:15 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2231059159' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-14T09:52:15.499 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:15 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2231059159' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-14T09:52:15.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:15.511 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-14T09:52:15.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:15 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2231059159' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-14T09:52:15.577 INFO:teuthology.orchestra.run.smithi078.stdout:1 2024-03-14T09:52:15.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:15.581 DEBUG:teuthology.orchestra.run.smithi078:> route 2024-03-14T09:52:15.642 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2024-03-14T09:52:15.642 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-14T09:52:15.642 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-14T09:52:15.642 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-14T09:52:15.642 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-14T09:52:15.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:15.643 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-14T09:52:15.643 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-14T09:52:15.643 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-14T09:52:15.643 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-14T09:52:15.643 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-14T09:52:15.644 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-14T09:52:16.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:16 smithi078 ceph-mon[38143]: pgmap v240: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1 op/s 2024-03-14T09:52:16.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:16.300 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-03-14T09:52:16.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:16.344 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list-id 2024-03-14T09:52:16.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:16.427 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-14T09:52:16.427 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-14T09:52:16.427 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-14T09:52:16.427 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-14T09:52:16.427 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-14T09:52:16.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:16 smithi152 ceph-mon[48279]: pgmap v240: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1 op/s 2024-03-14T09:52:16.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:16 smithi132 ceph-mon[47959]: pgmap v240: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1 op/s 2024-03-14T09:52:16.604 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-14T09:52:16.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:16.604 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-14T09:52:16.604 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-14T09:52:16.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-14T09:52:16.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-14T09:52:16.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-14T09:52:16.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-14T09:52:16.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-14T09:52:16.605 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-14T09:52:16.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:16.868 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-14T09:52:16.868 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-14T09:52:16.868 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2024-03-14T09:52:16.868 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set dev brx.0 master ceph-brx 2024-03-14T09:52:16.868 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-14T09:52:17.034 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-14T09:52:17.034 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-14T09:52:17.034 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-14T09:52:17.075 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-14T09:52:17.075 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-14T09:52:17.075 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-14T09:52:17.156 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-03-14T09:52:17.263 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-14T09:52:17.263 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-14T09:52:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-14T09:52:17.329 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-14T09:52:17.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:17.330 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-03-14T09:52:17.372 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-14T09:52:17.372 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-14T09:52:17.416 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-03-14T09:52:17.482 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-14T09:52:17.482 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-14T09:52:17.547 INFO:teuthology.orchestra.run.smithi078.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-14T09:52:17.550 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-14T09:52:17.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:17.550 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-03-14T09:52:17.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:2024-03-14T09:52:17.604+0000 7fda161d45c0 -1 init, newargv = 0x55768a2fdcd0 newargc=15 2024-03-14T09:52:17.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[90482]: starting ceph client 2024-03-14T09:52:17.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[90482]: starting fuse 2024-03-14T09:52:18.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:18 smithi152 ceph-mon[48279]: pgmap v241: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-14T09:52:18.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:18 smithi132 ceph-mon[47959]: pgmap v241: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-14T09:52:18.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:18 smithi078 ceph-mon[38143]: pgmap v241: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-14T09:52:18.693 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-03-14T09:52:18.767 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-14T09:52:18.767 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-14T09:52:18.832 INFO:teuthology.orchestra.run.smithi078.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-14T09:52:18.834 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-14T09:52:18.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:18.835 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-03-14T09:52:18.876 INFO:teuthology.orchestra.run.smithi078.stdout:122 2024-03-14T09:52:18.877 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [122] 2024-03-14T09:52:18.877 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-14T09:52:18.877 DEBUG:teuthology.orchestra.run.smithi078:> import glob 2024-03-14T09:52:18.877 DEBUG:teuthology.orchestra.run.smithi078:> import re 2024-03-14T09:52:18.877 DEBUG:teuthology.orchestra.run.smithi078:> import os 2024-03-14T09:52:18.877 DEBUG:teuthology.orchestra.run.smithi078:> import subprocess 2024-03-14T09:52:18.877 DEBUG:teuthology.orchestra.run.smithi078:> 2024-03-14T09:52:18.877 DEBUG:teuthology.orchestra.run.smithi078:> def _find_admin_socket(client_name): 2024-03-14T09:52:18.877 DEBUG:teuthology.orchestra.run.smithi078:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-14T09:52:18.877 DEBUG:teuthology.orchestra.run.smithi078:> files = glob.glob(asok_path) 2024-03-14T09:52:18.877 DEBUG:teuthology.orchestra.run.smithi078:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> # Given a non-glob path, it better be there 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> if "*" not in asok_path: 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> assert(len(files) == 1) 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> return files[0] 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> for f in files: 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> contents = proc_f.read() 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> if mountpoint in contents: 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> return f 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> print(_find_admin_socket("client.admin")) 2024-03-14T09:52:18.878 DEBUG:teuthology.orchestra.run.smithi078:> ' 2024-03-14T09:52:19.057 INFO:teuthology.orchestra.run.smithi078.stdout:/var/run/ceph/ceph-client.admin.90482.asok 2024-03-14T09:52:19.234 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.90482.asok 2024-03-14T09:52:19.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:19.235 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.90482.asok status 2024-03-14T09:52:19.457 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:52:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: "metadata": { 2024-03-14T09:52:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_sha1": "2f465319e3c83d48f999325459bcdfd0346449c0", 2024-03-14T09:52:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)", 2024-03-14T09:52:19.457 INFO:teuthology.orchestra.run.smithi078.stdout: "entity_id": "admin", 2024-03-14T09:52:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": "smithi078", 2024-03-14T09:52:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-14T09:52:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "pid": "90482", 2024-03-14T09:52:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "root": "/" 2024-03-14T09:52:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T09:52:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_count": 0, 2024-03-14T09:52:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_pinned_count": 0, 2024-03-14T09:52:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24698, 2024-03-14T09:52:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "inst": { 2024-03-14T09:52:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "name": { 2024-03-14T09:52:19.458 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "client", 2024-03-14T09:52:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 24698 2024-03-14T09:52:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T09:52:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-03-14T09:52:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-03-14T09:52:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-14T09:52:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2270427262 2024-03-14T09:52:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:52:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T09:52:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-03-14T09:52:19.459 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-03-14T09:52:19.460 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-14T09:52:19.460 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2270427262 2024-03-14T09:52:19.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T09:52:19.460 INFO:teuthology.orchestra.run.smithi078.stdout: "inst_str": "client.24698 172.21.15.78:0/2270427262", 2024-03-14T09:52:19.460 INFO:teuthology.orchestra.run.smithi078.stdout: "addr_str": "172.21.15.78:0/2270427262", 2024-03-14T09:52:19.460 INFO:teuthology.orchestra.run.smithi078.stdout: "inode_count": 1, 2024-03-14T09:52:19.460 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_epoch": 17, 2024-03-14T09:52:19.460 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch": 83, 2024-03-14T09:52:19.460 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch_barrier": 0, 2024-03-14T09:52:19.460 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklisted": false, 2024-03-14T09:52:19.460 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs" 2024-03-14T09:52:19.460 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:52:19.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:19.470 DEBUG:teuthology.orchestra.run.smithi078:> stat --file-system '--printf=%T 2024-03-14T09:52:19.470 DEBUG:teuthology.orchestra.run.smithi078:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-14T09:52:19.513 INFO:teuthology.orchestra.run.smithi078.stdout:fuseblk 2024-03-14T09:52:19.513 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-14T09:52:19.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:19.513 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-14T09:52:19.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:19.621 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-14T09:52:19.747 INFO:teuthology.orchestra.run.smithi078.stdout:# file: . 2024-03-14T09:52:19.747 INFO:teuthology.orchestra.run.smithi078.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-14T09:52:19.748 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:19.749 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:52:19.749 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-14T09:52:19.791 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:52:20.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.022+0000 7f1fc3800700 1 -- 172.21.15.78:0/466579357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fbc1027e0 msgr2=0x7f1fbc109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:20.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.022+0000 7f1fc3800700 1 --2- 172.21.15.78:0/466579357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fbc1027e0 0x7f1fbc109090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4009fc0 tx=0x7f1fb400b330 comp rx=0 tx=0).stop 2024-03-14T09:52:20.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.022+0000 7f1fc3800700 1 -- 172.21.15.78:0/466579357 shutdown_connections 2024-03-14T09:52:20.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.022+0000 7f1fc3800700 1 --2- 172.21.15.78:0/466579357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fbc100c50 0x7f1fbc101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.022+0000 7f1fc3800700 1 --2- 172.21.15.78:0/466579357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fbc1027e0 0x7f1fbc109090 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.022+0000 7f1fc3800700 1 --2- 172.21.15.78:0/466579357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fbc101e40 0x7f1fbc1022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.022+0000 7f1fc3800700 1 -- 172.21.15.78:0/466579357 >> 172.21.15.78:0/466579357 conn(0x7f1fbc0fc1c0 msgr2=0x7f1fbc0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:20.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.022+0000 7f1fc3800700 1 -- 172.21.15.78:0/466579357 shutdown_connections 2024-03-14T09:52:20.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.022+0000 7f1fc3800700 1 -- 172.21.15.78:0/466579357 wait complete. 2024-03-14T09:52:20.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.023+0000 7f1fc3800700 1 Processor -- start 2024-03-14T09:52:20.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.023+0000 7f1fc3800700 1 -- start start 2024-03-14T09:52:20.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.023+0000 7f1fc3800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fbc100c50 0x7f1fbc1970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:20.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.023+0000 7f1fc3800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fbc101e40 0x7f1fbc197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:20.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.023+0000 7f1fc3800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fbc1027e0 0x7f1fbc19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:20.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.023+0000 7f1fc3800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc19ec10 con 0x7f1fbc100c50 2024-03-14T09:52:20.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.023+0000 7f1fc3800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc19ed50 con 0x7f1fbc101e40 2024-03-14T09:52:20.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.023+0000 7f1fc3800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fbc19eee0 con 0x7f1fbc1027e0 2024-03-14T09:52:20.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.023+0000 7f1fc0e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fbc100c50 0x7f1fbc1970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:20.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.024+0000 7f1fc0e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fbc100c50 0x7f1fbc1970e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44488/0 (socket says 172.21.15.78:44488) 2024-03-14T09:52:20.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.024+0000 7f1fc0e00700 1 -- 172.21.15.78:0/1331291925 learned_addr learned my addr 172.21.15.78:0/1331291925 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:20.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.024+0000 7f1fc1800700 1 --2- 172.21.15.78:0/1331291925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fbc1027e0 0x7f1fbc19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:20.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.024+0000 7f1fbbe00700 1 --2- 172.21.15.78:0/1331291925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fbc101e40 0x7f1fbc197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:20.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.024+0000 7f1fc0e00700 1 -- 172.21.15.78:0/1331291925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fbc1027e0 msgr2=0x7f1fbc19e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:20.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.024+0000 7f1fc0e00700 1 --2- 172.21.15.78:0/1331291925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fbc1027e0 0x7f1fbc19e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.024+0000 7f1fc0e00700 1 -- 172.21.15.78:0/1331291925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fbc101e40 msgr2=0x7f1fbc197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:20.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.024+0000 7f1fc0e00700 1 --2- 172.21.15.78:0/1331291925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fbc101e40 0x7f1fbc197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.024+0000 7f1fc0e00700 1 -- 172.21.15.78:0/1331291925 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb4009c40 con 0x7f1fbc100c50 2024-03-14T09:52:20.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.024+0000 7f1fbbe00700 1 --2- 172.21.15.78:0/1331291925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fbc101e40 0x7f1fbc197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:52:20.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.024+0000 7f1fc0e00700 1 --2- 172.21.15.78:0/1331291925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fbc100c50 0x7f1fbc1970e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1fb000e9f0 tx=0x7f1fb000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:20.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.024+0000 7f1fb9600700 1 -- 172.21.15.78:0/1331291925 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0004220 con 0x7f1fbc100c50 2024-03-14T09:52:20.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.024+0000 7f1fb9600700 1 -- 172.21.15.78:0/1331291925 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1fb0004380 con 0x7f1fbc100c50 2024-03-14T09:52:20.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.024+0000 7f1fc3800700 1 -- 172.21.15.78:0/1331291925 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fbc19f1c0 con 0x7f1fbc100c50 2024-03-14T09:52:20.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.024+0000 7f1fb9600700 1 -- 172.21.15.78:0/1331291925 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0003c00 con 0x7f1fbc100c50 2024-03-14T09:52:20.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.025+0000 7f1fc3800700 1 -- 172.21.15.78:0/1331291925 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fbc19f720 con 0x7f1fbc100c50 2024-03-14T09:52:20.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.026+0000 7f1fc3800700 1 -- 172.21.15.78:0/1331291925 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fbc108000 con 0x7f1fbc100c50 2024-03-14T09:52:20.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.027+0000 7f1fb9600700 1 -- 172.21.15.78:0/1331291925 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f1fb000c7c0 con 0x7f1fbc100c50 2024-03-14T09:52:20.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.029+0000 7f1fb9600700 1 --2- 172.21.15.78:0/1331291925 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1fa40a0b10 0x7f1fa40a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:20.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.029+0000 7f1fb9600700 1 -- 172.21.15.78:0/1331291925 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1fb0014070 con 0x7f1fbc100c50 2024-03-14T09:52:20.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.029+0000 7f1fbbe00700 1 --2- 172.21.15.78:0/1331291925 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1fa40a0b10 0x7f1fa40a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:20.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.029+0000 7f1fbbe00700 1 --2- 172.21.15.78:0/1331291925 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1fa40a0b10 0x7f1fa40a2fd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009cb0 tx=0x7f1fa8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:20.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.033+0000 7f1fb9600700 1 -- 172.21.15.78:0/1331291925 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1fb0095d70 con 0x7f1fbc100c50 2024-03-14T09:52:20.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.253+0000 7f1fc3800700 1 -- 172.21.15.78:0/1331291925 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fbc0679e0 con 0x7f1fbc100c50 2024-03-14T09:52:20.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.255+0000 7f1fb9600700 1 -- 172.21.15.78:0/1331291925 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+23925 (secure 0 0 0) 0x7f1fb0099390 con 0x7f1fbc100c50 2024-03-14T09:52:20.256 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:20.256 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":83,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:52:10.245498+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:52:20.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.258+0000 7f1fc3800700 1 -- 172.21.15.78:0/1331291925 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1fa40a0b10 msgr2=0x7f1fa40a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:20.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.258+0000 7f1fc3800700 1 --2- 172.21.15.78:0/1331291925 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1fa40a0b10 0x7f1fa40a2fd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009cb0 tx=0x7f1fa8009260 comp rx=0 tx=0).stop 2024-03-14T09:52:20.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.258+0000 7f1fc3800700 1 -- 172.21.15.78:0/1331291925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fbc100c50 msgr2=0x7f1fbc1970e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:20.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.258+0000 7f1fc3800700 1 --2- 172.21.15.78:0/1331291925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fbc100c50 0x7f1fbc1970e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1fb000e9f0 tx=0x7f1fb000ed00 comp rx=0 tx=0).stop 2024-03-14T09:52:20.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.258+0000 7f1fc3800700 1 -- 172.21.15.78:0/1331291925 shutdown_connections 2024-03-14T09:52:20.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.258+0000 7f1fc3800700 1 --2- 172.21.15.78:0/1331291925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1fbc1027e0 0x7f1fbc19e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.258+0000 7f1fc3800700 1 --2- 172.21.15.78:0/1331291925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fbc100c50 0x7f1fbc1970e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.258+0000 7f1fc3800700 1 --2- 172.21.15.78:0/1331291925 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f1fa40a0b10 0x7f1fa40a2fd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.258+0000 7f1fc3800700 1 --2- 172.21.15.78:0/1331291925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1fbc101e40 0x7f1fbc197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.258+0000 7f1fc3800700 1 -- 172.21.15.78:0/1331291925 >> 172.21.15.78:0/1331291925 conn(0x7f1fbc0fc1c0 msgr2=0x7f1fbc105140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:20.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.258+0000 7f1fc3800700 1 -- 172.21.15.78:0/1331291925 shutdown_connections 2024-03-14T09:52:20.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.258+0000 7f1fc3800700 1 -- 172.21.15.78:0/1331291925 wait complete. 2024-03-14T09:52:20.276 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T09:48:37.618450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-14T09:52:03.938666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-14T09:52:05.149985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-14T09:52:08.227698+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-14T09:52:20.277 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T09:52:20.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.472+0000 7f7b1da00700 1 -- 172.21.15.78:0/1726704330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b181028e0 msgr2=0x7f7b18109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.472+0000 7f7b1da00700 1 --2- 172.21.15.78:0/1726704330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b181028e0 0x7f7b18109190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7b1000b0d0 tx=0x7f7b1000b3e0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.472+0000 7f7b1da00700 1 -- 172.21.15.78:0/1726704330 shutdown_connections 2024-03-14T09:52:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.472+0000 7f7b1da00700 1 --2- 172.21.15.78:0/1726704330 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b18100d50 0x7f7b18101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.472+0000 7f7b1da00700 1 --2- 172.21.15.78:0/1726704330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b181028e0 0x7f7b18109190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.472+0000 7f7b1da00700 1 --2- 172.21.15.78:0/1726704330 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7b18101f40 0x7f7b181023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.472+0000 7f7b1da00700 1 -- 172.21.15.78:0/1726704330 >> 172.21.15.78:0/1726704330 conn(0x7f7b180fc2c0 msgr2=0x7f7b180fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.472+0000 7f7b1da00700 1 -- 172.21.15.78:0/1726704330 shutdown_connections 2024-03-14T09:52:20.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.472+0000 7f7b1da00700 1 -- 172.21.15.78:0/1726704330 wait complete. 2024-03-14T09:52:20.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.473+0000 7f7b1da00700 1 Processor -- start 2024-03-14T09:52:20.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.473+0000 7f7b1da00700 1 -- start start 2024-03-14T09:52:20.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.473+0000 7f7b1da00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b18100d50 0x7f7b18199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:20.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.473+0000 7f7b1da00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b18101f40 0x7f7b18199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:20.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.473+0000 7f7b1da00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7b181028e0 0x7f7b181a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:20.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.473+0000 7f7b1da00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b181a0f40 con 0x7f7b18101f40 2024-03-14T09:52:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.473+0000 7f7b1da00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b181a1080 con 0x7f7b18100d50 2024-03-14T09:52:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.473+0000 7f7b1da00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b181a1210 con 0x7f7b181028e0 2024-03-14T09:52:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.473+0000 7f7b16000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b18101f40 0x7f7b18199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.473+0000 7f7b16000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b18101f40 0x7f7b18199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44498/0 (socket says 172.21.15.78:44498) 2024-03-14T09:52:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.473+0000 7f7b16000700 1 -- 172.21.15.78:0/1127702005 learned_addr learned my addr 172.21.15.78:0/1127702005 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.473+0000 7f7b16a00700 1 --2- 172.21.15.78:0/1127702005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b18100d50 0x7f7b18199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.474+0000 7f7b17400700 1 --2- 172.21.15.78:0/1127702005 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7b181028e0 0x7f7b181a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.474+0000 7f7b16000700 1 -- 172.21.15.78:0/1127702005 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7b181028e0 msgr2=0x7f7b181a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.474+0000 7f7b16000700 1 --2- 172.21.15.78:0/1127702005 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7b181028e0 0x7f7b181a0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.474+0000 7f7b16000700 1 -- 172.21.15.78:0/1127702005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b18100d50 msgr2=0x7f7b18199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.474+0000 7f7b16000700 1 --2- 172.21.15.78:0/1127702005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b18100d50 0x7f7b18199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.474+0000 7f7b16000700 1 -- 172.21.15.78:0/1127702005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b10009d00 con 0x7f7b18101f40 2024-03-14T09:52:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.474+0000 7f7b16000700 1 --2- 172.21.15.78:0/1127702005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b18101f40 0x7f7b18199950 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400e9a0 tx=0x7f7b0400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.474+0000 7f7b0f400700 1 -- 172.21.15.78:0/1127702005 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b04009790 con 0x7f7b18101f40 2024-03-14T09:52:20.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.474+0000 7f7b1da00700 1 -- 172.21.15.78:0/1127702005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b181a14f0 con 0x7f7b18101f40 2024-03-14T09:52:20.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.474+0000 7f7b0f400700 1 -- 172.21.15.78:0/1127702005 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b0400cbd0 con 0x7f7b18101f40 2024-03-14T09:52:20.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.474+0000 7f7b0f400700 1 -- 172.21.15.78:0/1127702005 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b04018680 con 0x7f7b18101f40 2024-03-14T09:52:20.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.474+0000 7f7b1da00700 1 -- 172.21.15.78:0/1127702005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b181a1a00 con 0x7f7b18101f40 2024-03-14T09:52:20.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.475+0000 7f7b1da00700 1 -- 172.21.15.78:0/1127702005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b18067ac0 con 0x7f7b18101f40 2024-03-14T09:52:20.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.477+0000 7f7b0f400700 1 -- 172.21.15.78:0/1127702005 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f7b0400cd40 con 0x7f7b18101f40 2024-03-14T09:52:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.478+0000 7f7b0f400700 1 --2- 172.21.15.78:0/1127702005 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7afc0a0a90 0x7f7afc0a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.478+0000 7f7b0f400700 1 -- 172.21.15.78:0/1127702005 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7b04014070 con 0x7f7b18101f40 2024-03-14T09:52:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.479+0000 7f7b16a00700 1 --2- 172.21.15.78:0/1127702005 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7afc0a0a90 0x7f7afc0a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:20.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.479+0000 7f7b16a00700 1 --2- 172.21.15.78:0/1127702005 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7afc0a0a90 0x7f7afc0a2f50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7b18101da0 tx=0x7f7b08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:20.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.482+0000 7f7b0f400700 1 -- 172.21.15.78:0/1127702005 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7b04052960 con 0x7f7b18101f40 2024-03-14T09:52:20.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:20 smithi132 ceph-mon[47959]: pgmap v242: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 109 B/s wr, 0 op/s 2024-03-14T09:52:20.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:20 smithi078 ceph-mon[38143]: pgmap v242: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 109 B/s wr, 0 op/s 2024-03-14T09:52:20.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.689+0000 7f7b1da00700 1 -- 172.21.15.78:0/1127702005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7b181a1d20 con 0x7f7b18101f40 2024-03-14T09:52:20.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.689+0000 7f7b0f400700 1 -- 172.21.15.78:0/1127702005 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7b04095c70 con 0x7f7b18101f40 2024-03-14T09:52:20.690 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-14T09:52:20.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.691+0000 7f7b1da00700 1 -- 172.21.15.78:0/1127702005 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7afc0a0a90 msgr2=0x7f7afc0a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:20.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.691+0000 7f7b1da00700 1 --2- 172.21.15.78:0/1127702005 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7afc0a0a90 0x7f7afc0a2f50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7b18101da0 tx=0x7f7b08009260 comp rx=0 tx=0).stop 2024-03-14T09:52:20.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.691+0000 7f7b1da00700 1 -- 172.21.15.78:0/1127702005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b18101f40 msgr2=0x7f7b18199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:20.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.691+0000 7f7b1da00700 1 --2- 172.21.15.78:0/1127702005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b18101f40 0x7f7b18199950 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400e9a0 tx=0x7f7b0400ecb0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.691+0000 7f7b1da00700 1 -- 172.21.15.78:0/1127702005 shutdown_connections 2024-03-14T09:52:20.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.691+0000 7f7b1da00700 1 --2- 172.21.15.78:0/1127702005 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7afc0a0a90 0x7f7afc0a2f50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.691+0000 7f7b1da00700 1 --2- 172.21.15.78:0/1127702005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b18100d50 0x7f7b18199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.691+0000 7f7b1da00700 1 --2- 172.21.15.78:0/1127702005 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7b181028e0 0x7f7b181a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.691+0000 7f7b1da00700 1 --2- 172.21.15.78:0/1127702005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b18101f40 0x7f7b18199950 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.691+0000 7f7b1da00700 1 -- 172.21.15.78:0/1127702005 >> 172.21.15.78:0/1127702005 conn(0x7f7b180fc2c0 msgr2=0x7f7b18072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:20.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.692+0000 7f7b1da00700 1 -- 172.21.15.78:0/1127702005 shutdown_connections 2024-03-14T09:52:20.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.692+0000 7f7b1da00700 1 -- 172.21.15.78:0/1127702005 wait complete. 2024-03-14T09:52:20.710 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-14T09:52:20.748 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:20 smithi152 ceph-mon[48279]: pgmap v242: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 109 B/s wr, 0 op/s 2024-03-14T09:52:20.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.904+0000 7f9c53400700 1 -- 172.21.15.78:0/837809655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c101f40 msgr2=0x7f9c4c1023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.904+0000 7f9c53400700 1 --2- 172.21.15.78:0/837809655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c101f40 0x7f9c4c1023a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9c380099c0 tx=0x7f9c38009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.904+0000 7f9c53400700 1 -- 172.21.15.78:0/837809655 shutdown_connections 2024-03-14T09:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.904+0000 7f9c53400700 1 --2- 172.21.15.78:0/837809655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c4c100d50 0x7f9c4c101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.904+0000 7f9c53400700 1 --2- 172.21.15.78:0/837809655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c4c1028e0 0x7f9c4c109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.904+0000 7f9c53400700 1 --2- 172.21.15.78:0/837809655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c101f40 0x7f9c4c1023a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.904+0000 7f9c53400700 1 -- 172.21.15.78:0/837809655 >> 172.21.15.78:0/837809655 conn(0x7f9c4c0fc2c0 msgr2=0x7f9c4c0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.904+0000 7f9c53400700 1 -- 172.21.15.78:0/837809655 shutdown_connections 2024-03-14T09:52:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.904+0000 7f9c53400700 1 -- 172.21.15.78:0/837809655 wait complete. 2024-03-14T09:52:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.905+0000 7f9c53400700 1 Processor -- start 2024-03-14T09:52:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.905+0000 7f9c53400700 1 -- start start 2024-03-14T09:52:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.905+0000 7f9c53400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c4c100d50 0x7f9c4c197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.905+0000 7f9c53400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c101f40 0x7f9c4c197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.905+0000 7f9c53400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c4c1028e0 0x7f9c4c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.905+0000 7f9c53400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c19ec40 con 0x7f9c4c101f40 2024-03-14T09:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.905+0000 7f9c53400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c19ed80 con 0x7f9c4c1028e0 2024-03-14T09:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.905+0000 7f9c53400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c19ef10 con 0x7f9c4c100d50 2024-03-14T09:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.906+0000 7f9c4be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c101f40 0x7f9c4c197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.906+0000 7f9c4be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c101f40 0x7f9c4c197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44524/0 (socket says 172.21.15.78:44524) 2024-03-14T09:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.906+0000 7f9c4be00700 1 -- 172.21.15.78:0/3706363774 learned_addr learned my addr 172.21.15.78:0/3706363774 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.906+0000 7f9c50a00700 1 --2- 172.21.15.78:0/3706363774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c4c100d50 0x7f9c4c197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.906+0000 7f9c51400700 1 --2- 172.21.15.78:0/3706363774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c4c1028e0 0x7f9c4c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.906+0000 7f9c4be00700 1 -- 172.21.15.78:0/3706363774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c4c100d50 msgr2=0x7f9c4c197110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.906+0000 7f9c4be00700 1 --2- 172.21.15.78:0/3706363774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c4c100d50 0x7f9c4c197110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.906+0000 7f9c4be00700 1 -- 172.21.15.78:0/3706363774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c4c1028e0 msgr2=0x7f9c4c19e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.906+0000 7f9c4be00700 1 --2- 172.21.15.78:0/3706363774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c4c1028e0 0x7f9c4c19e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.906+0000 7f9c4be00700 1 -- 172.21.15.78:0/3706363774 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c380095f0 con 0x7f9c4c101f40 2024-03-14T09:52:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.906+0000 7f9c4be00700 1 --2- 172.21.15.78:0/3706363774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c101f40 0x7f9c4c197650 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9c38003730 tx=0x7f9c3800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.906+0000 7f9c49600700 1 -- 172.21.15.78:0/3706363774 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3801d070 con 0x7f9c4c101f40 2024-03-14T09:52:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.906+0000 7f9c49600700 1 -- 172.21.15.78:0/3706363774 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9c38005050 con 0x7f9c4c101f40 2024-03-14T09:52:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.906+0000 7f9c53400700 1 -- 172.21.15.78:0/3706363774 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4c19f190 con 0x7f9c4c101f40 2024-03-14T09:52:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.906+0000 7f9c49600700 1 -- 172.21.15.78:0/3706363774 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38004b10 con 0x7f9c4c101f40 2024-03-14T09:52:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.907+0000 7f9c53400700 1 -- 172.21.15.78:0/3706363774 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4c19f5b0 con 0x7f9c4c101f40 2024-03-14T09:52:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.908+0000 7f9c53400700 1 -- 172.21.15.78:0/3706363774 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c4c1081d0 con 0x7f9c4c101f40 2024-03-14T09:52:20.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.909+0000 7f9c49600700 1 -- 172.21.15.78:0/3706363774 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f9c380051c0 con 0x7f9c4c101f40 2024-03-14T09:52:20.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.911+0000 7f9c49600700 1 --2- 172.21.15.78:0/3706363774 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c340a0a90 0x7f9c340a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:20.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.911+0000 7f9c49600700 1 -- 172.21.15.78:0/3706363774 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9c380cd370 con 0x7f9c4c101f40 2024-03-14T09:52:20.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.911+0000 7f9c50a00700 1 --2- 172.21.15.78:0/3706363774 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c340a0a90 0x7f9c340a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:20.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.912+0000 7f9c50a00700 1 --2- 172.21.15.78:0/3706363774 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c340a0a90 0x7f9c340a2f50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c101da0 tx=0x7f9c40009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:20.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:20.913+0000 7f9c49600700 1 -- 172.21.15.78:0/3706363774 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9c38096b80 con 0x7f9c4c101f40 2024-03-14T09:52:21.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.116+0000 7f9c53400700 1 -- 172.21.15.78:0/3706363774 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9c4c10cbd0 con 0x7f9c4c101f40 2024-03-14T09:52:21.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.117+0000 7f9c49600700 1 -- 172.21.15.78:0/3706363774 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9c3809a1a0 con 0x7f9c4c101f40 2024-03-14T09:52:21.118 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:52:21.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.119+0000 7f9c53400700 1 -- 172.21.15.78:0/3706363774 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c340a0a90 msgr2=0x7f9c340a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:21.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.119+0000 7f9c53400700 1 --2- 172.21.15.78:0/3706363774 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c340a0a90 0x7f9c340a2f50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c101da0 tx=0x7f9c40009260 comp rx=0 tx=0).stop 2024-03-14T09:52:21.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.119+0000 7f9c53400700 1 -- 172.21.15.78:0/3706363774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c101f40 msgr2=0x7f9c4c197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:21.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.119+0000 7f9c53400700 1 --2- 172.21.15.78:0/3706363774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c101f40 0x7f9c4c197650 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9c38003730 tx=0x7f9c3800f690 comp rx=0 tx=0).stop 2024-03-14T09:52:21.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.119+0000 7f9c53400700 1 -- 172.21.15.78:0/3706363774 shutdown_connections 2024-03-14T09:52:21.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.119+0000 7f9c53400700 1 --2- 172.21.15.78:0/3706363774 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c340a0a90 0x7f9c340a2f50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.119+0000 7f9c53400700 1 --2- 172.21.15.78:0/3706363774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c4c1028e0 0x7f9c4c19e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.119+0000 7f9c53400700 1 --2- 172.21.15.78:0/3706363774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c4c100d50 0x7f9c4c197110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.119+0000 7f9c53400700 1 --2- 172.21.15.78:0/3706363774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c4c101f40 0x7f9c4c197650 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.119+0000 7f9c53400700 1 -- 172.21.15.78:0/3706363774 >> 172.21.15.78:0/3706363774 conn(0x7f9c4c0fc2c0 msgr2=0x7f9c4c105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:21.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.119+0000 7f9c53400700 1 -- 172.21.15.78:0/3706363774 shutdown_connections 2024-03-14T09:52:21.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.120+0000 7f9c53400700 1 -- 172.21.15.78:0/3706363774 wait complete. 2024-03-14T09:52:21.138 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-14T09:52:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.335+0000 7f4358e00700 1 -- 172.21.15.78:0/3909953648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4354101f30 msgr2=0x7f4354102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.335+0000 7f4358e00700 1 --2- 172.21.15.78:0/3909953648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4354101f30 0x7f4354102390 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f43400099c0 tx=0x7f4340009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.335+0000 7f4358e00700 1 -- 172.21.15.78:0/3909953648 shutdown_connections 2024-03-14T09:52:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.335+0000 7f4358e00700 1 --2- 172.21.15.78:0/3909953648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f43541028d0 0x7f4354109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.335+0000 7f4358e00700 1 --2- 172.21.15.78:0/3909953648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4354100d40 0x7f4354101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.335+0000 7f4358e00700 1 --2- 172.21.15.78:0/3909953648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4354101f30 0x7f4354102390 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.335+0000 7f4358e00700 1 -- 172.21.15.78:0/3909953648 >> 172.21.15.78:0/3909953648 conn(0x7f43540fc2b0 msgr2=0x7f43540fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:21.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.335+0000 7f4358e00700 1 -- 172.21.15.78:0/3909953648 shutdown_connections 2024-03-14T09:52:21.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.335+0000 7f4358e00700 1 -- 172.21.15.78:0/3909953648 wait complete. 2024-03-14T09:52:21.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.337+0000 7f4358e00700 1 Processor -- start 2024-03-14T09:52:21.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.337+0000 7f4358e00700 1 -- start start 2024-03-14T09:52:21.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.337+0000 7f4358e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4354100d40 0x7f4354197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:21.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.337+0000 7f4358e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4354101f30 0x7f4354197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:21.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.337+0000 7f4358e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43541028d0 0x7f435419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:21.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.337+0000 7f4358e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435419ec30 con 0x7f4354101f30 2024-03-14T09:52:21.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.337+0000 7f4358e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435419ed70 con 0x7f4354100d40 2024-03-14T09:52:21.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.337+0000 7f4358e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435419ef00 con 0x7f43541028d0 2024-03-14T09:52:21.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.338+0000 7f4351400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4354101f30 0x7f4354197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:21.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.338+0000 7f4351400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4354101f30 0x7f4354197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44548/0 (socket says 172.21.15.78:44548) 2024-03-14T09:52:21.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.338+0000 7f4351400700 1 -- 172.21.15.78:0/1275934961 learned_addr learned my addr 172.21.15.78:0/1275934961 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:21.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.338+0000 7f4351e00700 1 --2- 172.21.15.78:0/1275934961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4354100d40 0x7f4354197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:21.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.338+0000 7f4352800700 1 --2- 172.21.15.78:0/1275934961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43541028d0 0x7f435419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:21.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.338+0000 7f4351400700 1 -- 172.21.15.78:0/1275934961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43541028d0 msgr2=0x7f435419e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:21.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.338+0000 7f4351400700 1 --2- 172.21.15.78:0/1275934961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43541028d0 0x7f435419e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.338+0000 7f4351400700 1 -- 172.21.15.78:0/1275934961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4354100d40 msgr2=0x7f4354197100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:21.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.338+0000 7f4351400700 1 --2- 172.21.15.78:0/1275934961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4354100d40 0x7f4354197100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.338+0000 7f4351400700 1 -- 172.21.15.78:0/1275934961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43400095f0 con 0x7f4354101f30 2024-03-14T09:52:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.338+0000 7f4351e00700 1 --2- 172.21.15.78:0/1275934961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4354100d40 0x7f4354197100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.338+0000 7f4352800700 1 --2- 172.21.15.78:0/1275934961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43541028d0 0x7f435419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.338+0000 7f4351400700 1 --2- 172.21.15.78:0/1275934961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4354101f30 0x7f4354197640 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4340003730 tx=0x7f434000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.338+0000 7f434aa00700 1 -- 172.21.15.78:0/1275934961 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434001d070 con 0x7f4354101f30 2024-03-14T09:52:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.338+0000 7f434aa00700 1 -- 172.21.15.78:0/1275934961 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f434000bda0 con 0x7f4354101f30 2024-03-14T09:52:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.339+0000 7f434aa00700 1 -- 172.21.15.78:0/1275934961 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43400173f0 con 0x7f4354101f30 2024-03-14T09:52:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.339+0000 7f4358e00700 1 -- 172.21.15.78:0/1275934961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435419f180 con 0x7f4354101f30 2024-03-14T09:52:21.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.339+0000 7f4358e00700 1 -- 172.21.15.78:0/1275934961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f435419f5a0 con 0x7f4354101f30 2024-03-14T09:52:21.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.340+0000 7f4358e00700 1 -- 172.21.15.78:0/1275934961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4354108160 con 0x7f4354101f30 2024-03-14T09:52:21.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.341+0000 7f434aa00700 1 -- 172.21.15.78:0/1275934961 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f4340022470 con 0x7f4354101f30 2024-03-14T09:52:21.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.343+0000 7f434aa00700 1 --2- 172.21.15.78:0/1275934961 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f43380a0b10 0x7f43380a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:21.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.343+0000 7f434aa00700 1 -- 172.21.15.78:0/1275934961 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f43400cd350 con 0x7f4354101f30 2024-03-14T09:52:21.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.343+0000 7f4351e00700 1 --2- 172.21.15.78:0/1275934961 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f43380a0b10 0x7f43380a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:21.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.343+0000 7f4351e00700 1 --2- 172.21.15.78:0/1275934961 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f43380a0b10 0x7f43380a2fd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4354101d90 tx=0x7f4344009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:21.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.347+0000 7f434aa00700 1 -- 172.21.15.78:0/1275934961 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4340096bc0 con 0x7f4354101f30 2024-03-14T09:52:21.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:21 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1331291925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:21.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:21 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1127702005' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:52:21.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:21 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3706363774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:52:21.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.551+0000 7f4358e00700 1 -- 172.21.15.78:0/1275934961 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4354107b40 con 0x7f4354101f30 2024-03-14T09:52:21.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.552+0000 7f434aa00700 1 -- 172.21.15.78:0/1275934961 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f434009a1e0 con 0x7f4354101f30 2024-03-14T09:52:21.552 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-14T09:52:21.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:21 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1331291925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:21.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:21 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1127702005' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:52:21.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:21 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3706363774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:52:21.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.554+0000 7f4358e00700 1 -- 172.21.15.78:0/1275934961 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f43380a0b10 msgr2=0x7f43380a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:21.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.554+0000 7f4358e00700 1 --2- 172.21.15.78:0/1275934961 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f43380a0b10 0x7f43380a2fd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4354101d90 tx=0x7f4344009260 comp rx=0 tx=0).stop 2024-03-14T09:52:21.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.554+0000 7f4358e00700 1 -- 172.21.15.78:0/1275934961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4354101f30 msgr2=0x7f4354197640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:21.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.554+0000 7f4358e00700 1 --2- 172.21.15.78:0/1275934961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4354101f30 0x7f4354197640 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4340003730 tx=0x7f434000fa00 comp rx=0 tx=0).stop 2024-03-14T09:52:21.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.554+0000 7f4358e00700 1 -- 172.21.15.78:0/1275934961 shutdown_connections 2024-03-14T09:52:21.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.554+0000 7f4358e00700 1 --2- 172.21.15.78:0/1275934961 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f43380a0b10 0x7f43380a2fd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.554+0000 7f4358e00700 1 --2- 172.21.15.78:0/1275934961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4354100d40 0x7f4354197100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.554+0000 7f4358e00700 1 --2- 172.21.15.78:0/1275934961 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f43541028d0 0x7f435419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.554+0000 7f4358e00700 1 --2- 172.21.15.78:0/1275934961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4354101f30 0x7f4354197640 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.554+0000 7f4358e00700 1 -- 172.21.15.78:0/1275934961 >> 172.21.15.78:0/1275934961 conn(0x7f43540fc2b0 msgr2=0x7f43541052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:21.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.554+0000 7f4358e00700 1 -- 172.21.15.78:0/1275934961 shutdown_connections 2024-03-14T09:52:21.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.554+0000 7f4358e00700 1 -- 172.21.15.78:0/1275934961 wait complete. 2024-03-14T09:52:21.573 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-14T09:52:21.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:21 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1331291925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:21.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:21 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1127702005' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:52:21.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:21 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3706363774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:52:21.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.770+0000 7fc47f800700 1 -- 172.21.15.78:0/926411198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4781028e0 msgr2=0x7fc478109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:21.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.770+0000 7fc47f800700 1 --2- 172.21.15.78:0/926411198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4781028e0 0x7fc478109190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc47000b0d0 tx=0x7fc47000b3e0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.770+0000 7fc47f800700 1 -- 172.21.15.78:0/926411198 shutdown_connections 2024-03-14T09:52:21.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.770+0000 7fc47f800700 1 --2- 172.21.15.78:0/926411198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc478100d50 0x7fc478101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.770+0000 7fc47f800700 1 --2- 172.21.15.78:0/926411198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4781028e0 0x7fc478109190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.770+0000 7fc47f800700 1 --2- 172.21.15.78:0/926411198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc478101f40 0x7fc4781023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.770+0000 7fc47f800700 1 -- 172.21.15.78:0/926411198 >> 172.21.15.78:0/926411198 conn(0x7fc4780fc2c0 msgr2=0x7fc4780fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:21.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.770+0000 7fc47f800700 1 -- 172.21.15.78:0/926411198 shutdown_connections 2024-03-14T09:52:21.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.770+0000 7fc47f800700 1 -- 172.21.15.78:0/926411198 wait complete. 2024-03-14T09:52:21.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.771+0000 7fc47f800700 1 Processor -- start 2024-03-14T09:52:21.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.771+0000 7fc47f800700 1 -- start start 2024-03-14T09:52:21.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.771+0000 7fc47f800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc478100d50 0x7fc478199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:21.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.771+0000 7fc47f800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc478101f40 0x7fc478199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:21.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.771+0000 7fc47f800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4781028e0 0x7fc4781a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:21.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.772+0000 7fc47f800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4781a0f40 con 0x7fc4781028e0 2024-03-14T09:52:21.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.772+0000 7fc47f800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4781a1080 con 0x7fc478101f40 2024-03-14T09:52:21.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.772+0000 7fc47f800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4781a1210 con 0x7fc478100d50 2024-03-14T09:52:21.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.772+0000 7fc47d800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4781028e0 0x7fc4781a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:21.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.772+0000 7fc47ce00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc478100d50 0x7fc478199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:21.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.772+0000 7fc477e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc478101f40 0x7fc478199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:21.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.772+0000 7fc47d800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4781028e0 0x7fc4781a0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44578/0 (socket says 172.21.15.78:44578) 2024-03-14T09:52:21.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.772+0000 7fc47ce00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc478100d50 0x7fc478199410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60362/0 (socket says 172.21.15.78:60362) 2024-03-14T09:52:21.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.772+0000 7fc47d800700 1 -- 172.21.15.78:0/368083454 learned_addr learned my addr 172.21.15.78:0/368083454 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:21.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.772+0000 7fc477e00700 1 -- 172.21.15.78:0/368083454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc478100d50 msgr2=0x7fc478199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:21.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.772+0000 7fc477e00700 1 --2- 172.21.15.78:0/368083454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc478100d50 0x7fc478199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.772+0000 7fc477e00700 1 -- 172.21.15.78:0/368083454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4781028e0 msgr2=0x7fc4781a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.772+0000 7fc477e00700 1 --2- 172.21.15.78:0/368083454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4781028e0 0x7fc4781a0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.772+0000 7fc477e00700 1 -- 172.21.15.78:0/368083454 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc470009d00 con 0x7fc478101f40 2024-03-14T09:52:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.772+0000 7fc47d800700 1 --2- 172.21.15.78:0/368083454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4781028e0 0x7fc4781a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.772+0000 7fc477e00700 1 --2- 172.21.15.78:0/368083454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc478101f40 0x7fc478199950 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc46400e9f0 tx=0x7fc46400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.773+0000 7fc475600700 1 -- 172.21.15.78:0/368083454 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46400c9c0 con 0x7fc478101f40 2024-03-14T09:52:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.773+0000 7fc475600700 1 -- 172.21.15.78:0/368083454 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc464010430 con 0x7fc478101f40 2024-03-14T09:52:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.773+0000 7fc47f800700 1 -- 172.21.15.78:0/368083454 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4781a14f0 con 0x7fc478101f40 2024-03-14T09:52:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.773+0000 7fc475600700 1 -- 172.21.15.78:0/368083454 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc464018680 con 0x7fc478101f40 2024-03-14T09:52:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.773+0000 7fc47f800700 1 -- 172.21.15.78:0/368083454 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4781a1a00 con 0x7fc478101f40 2024-03-14T09:52:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.774+0000 7fc47f800700 1 -- 172.21.15.78:0/368083454 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc478067ac0 con 0x7fc478101f40 2024-03-14T09:52:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.776+0000 7fc475600700 1 -- 172.21.15.78:0/368083454 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fc464010860 con 0x7fc478101f40 2024-03-14T09:52:21.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.777+0000 7fc475600700 1 --2- 172.21.15.78:0/368083454 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc4600a0a90 0x7fc4600a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:21.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.777+0000 7fc475600700 1 -- 172.21.15.78:0/368083454 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc464014070 con 0x7fc478101f40 2024-03-14T09:52:21.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.777+0000 7fc47ce00700 1 --2- 172.21.15.78:0/368083454 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc4600a0a90 0x7fc4600a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:21.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.778+0000 7fc47ce00700 1 --2- 172.21.15.78:0/368083454 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc4600a0a90 0x7fc4600a2f50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc478101da0 tx=0x7fc46c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:21.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.780+0000 7fc475600700 1 -- 172.21.15.78:0/368083454 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc464095cc0 con 0x7fc478101f40 2024-03-14T09:52:21.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.984+0000 7fc47f800700 1 -- 172.21.15.78:0/368083454 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc47810c9e0 con 0x7fc478101f40 2024-03-14T09:52:21.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.984+0000 7fc475600700 1 -- 172.21.15.78:0/368083454 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v83) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc4640992e0 con 0x7fc478101f40 2024-03-14T09:52:21.985 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:52:21.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.987+0000 7fc47f800700 1 -- 172.21.15.78:0/368083454 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc4600a0a90 msgr2=0x7fc4600a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:21.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.987+0000 7fc47f800700 1 --2- 172.21.15.78:0/368083454 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc4600a0a90 0x7fc4600a2f50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc478101da0 tx=0x7fc46c009260 comp rx=0 tx=0).stop 2024-03-14T09:52:21.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.987+0000 7fc47f800700 1 -- 172.21.15.78:0/368083454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc478101f40 msgr2=0x7fc478199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:21.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.987+0000 7fc47f800700 1 --2- 172.21.15.78:0/368083454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc478101f40 0x7fc478199950 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc46400e9f0 tx=0x7fc46400ed00 comp rx=0 tx=0).stop 2024-03-14T09:52:21.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.987+0000 7fc47f800700 1 -- 172.21.15.78:0/368083454 shutdown_connections 2024-03-14T09:52:21.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.987+0000 7fc47f800700 1 --2- 172.21.15.78:0/368083454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc4781028e0 0x7fc4781a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.987+0000 7fc47f800700 1 --2- 172.21.15.78:0/368083454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc478100d50 0x7fc478199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.987+0000 7fc47f800700 1 --2- 172.21.15.78:0/368083454 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc4600a0a90 0x7fc4600a2f50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.987+0000 7fc47f800700 1 --2- 172.21.15.78:0/368083454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc478101f40 0x7fc478199950 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:21.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.987+0000 7fc47f800700 1 -- 172.21.15.78:0/368083454 >> 172.21.15.78:0/368083454 conn(0x7fc4780fc2c0 msgr2=0x7fc478072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:21.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.987+0000 7fc47f800700 1 -- 172.21.15.78:0/368083454 shutdown_connections 2024-03-14T09:52:21.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:21.987+0000 7fc47f800700 1 -- 172.21.15.78:0/368083454 wait complete. 2024-03-14T09:52:22.006 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-14T09:52:22.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.202+0000 7f199a000700 1 -- 172.21.15.78:0/1996249806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19941017b0 msgr2=0x7f1994103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:22.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.202+0000 7f199a000700 1 --2- 172.21.15.78:0/1996249806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19941017b0 0x7f1994103ba0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1984009990 tx=0x7f1984009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.202+0000 7f199a000700 1 -- 172.21.15.78:0/1996249806 shutdown_connections 2024-03-14T09:52:22.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.202+0000 7f199a000700 1 --2- 172.21.15.78:0/1996249806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19941017b0 0x7f1994103ba0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.202+0000 7f199a000700 1 --2- 172.21.15.78:0/1996249806 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1994106a10 0x7f1994108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.202+0000 7f199a000700 1 --2- 172.21.15.78:0/1996249806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19941040e0 0x7f19941064d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.202+0000 7f199a000700 1 -- 172.21.15.78:0/1996249806 >> 172.21.15.78:0/1996249806 conn(0x7f19940fb1a0 msgr2=0x7f19940fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:22.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.202+0000 7f199a000700 1 -- 172.21.15.78:0/1996249806 shutdown_connections 2024-03-14T09:52:22.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.202+0000 7f199a000700 1 -- 172.21.15.78:0/1996249806 wait complete. 2024-03-14T09:52:22.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.203+0000 7f199a000700 1 Processor -- start 2024-03-14T09:52:22.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.203+0000 7f199a000700 1 -- start start 2024-03-14T09:52:22.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.203+0000 7f199a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19941017b0 0x7f1994199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:22.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.203+0000 7f199a000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19941040e0 0x7f19941997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:22.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.203+0000 7f199a000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1994106a10 0x7f19941a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:22.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.203+0000 7f199a000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19941a0dc0 con 0x7f19941017b0 2024-03-14T09:52:22.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.203+0000 7f199a000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19941a0f00 con 0x7f1994106a10 2024-03-14T09:52:22.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.203+0000 7f199a000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19941a1090 con 0x7f19941040e0 2024-03-14T09:52:22.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.204+0000 7f1993400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19941017b0 0x7f1994199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:22.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.204+0000 7f1993400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19941017b0 0x7f1994199290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44600/0 (socket says 172.21.15.78:44600) 2024-03-14T09:52:22.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.204+0000 7f1993400700 1 -- 172.21.15.78:0/2146466559 learned_addr learned my addr 172.21.15.78:0/2146466559 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:22.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.204+0000 7f198aa00700 1 --2- 172.21.15.78:0/2146466559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19941040e0 0x7f19941997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:22.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.204+0000 7f1993e00700 1 --2- 172.21.15.78:0/2146466559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1994106a10 0x7f19941a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:22.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.204+0000 7f1993400700 1 -- 172.21.15.78:0/2146466559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19941040e0 msgr2=0x7f19941997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:22.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.204+0000 7f1993400700 1 --2- 172.21.15.78:0/2146466559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19941040e0 0x7f19941997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.204+0000 7f1993400700 1 -- 172.21.15.78:0/2146466559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1994106a10 msgr2=0x7f19941a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:22.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.204+0000 7f1993400700 1 --2- 172.21.15.78:0/2146466559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1994106a10 0x7f19941a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.204+0000 7f1993400700 1 -- 172.21.15.78:0/2146466559 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19840095f0 con 0x7f19941017b0 2024-03-14T09:52:22.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.204+0000 7f1993400700 1 --2- 172.21.15.78:0/2146466559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19941017b0 0x7f1994199290 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f198400b680 tx=0x7f1984009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:22.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.205+0000 7f1990c00700 1 -- 172.21.15.78:0/2146466559 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198401d070 con 0x7f19941017b0 2024-03-14T09:52:22.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.205+0000 7f199a000700 1 -- 172.21.15.78:0/2146466559 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19941a1310 con 0x7f19941017b0 2024-03-14T09:52:22.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.205+0000 7f1990c00700 1 -- 172.21.15.78:0/2146466559 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1984022470 con 0x7f19941017b0 2024-03-14T09:52:22.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.205+0000 7f1990c00700 1 -- 172.21.15.78:0/2146466559 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1984017650 con 0x7f19941017b0 2024-03-14T09:52:22.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.205+0000 7f199a000700 1 -- 172.21.15.78:0/2146466559 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19941a1790 con 0x7f19941017b0 2024-03-14T09:52:22.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.206+0000 7f199a000700 1 -- 172.21.15.78:0/2146466559 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19941085e0 con 0x7f19941017b0 2024-03-14T09:52:22.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.208+0000 7f1990c00700 1 -- 172.21.15.78:0/2146466559 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f19840225e0 con 0x7f19941017b0 2024-03-14T09:52:22.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.210+0000 7f1990c00700 1 --2- 172.21.15.78:0/2146466559 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19780a0a90 0x7f19780a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:22.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.210+0000 7f1990c00700 1 -- 172.21.15.78:0/2146466559 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f19840cd550 con 0x7f19941017b0 2024-03-14T09:52:22.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.210+0000 7f198aa00700 1 --2- 172.21.15.78:0/2146466559 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19780a0a90 0x7f19780a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:22.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.210+0000 7f198aa00700 1 --2- 172.21.15.78:0/2146466559 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19780a0a90 0x7f19780a2f50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f197c00f540 tx=0x7f197c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:22.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.214+0000 7f1990c00700 1 -- 172.21.15.78:0/2146466559 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1984096d60 con 0x7f19941017b0 2024-03-14T09:52:22.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.439+0000 7f199a000700 1 -- 172.21.15.78:0/2146466559 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1994067a90 con 0x7f19941017b0 2024-03-14T09:52:22.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.439+0000 7f1990c00700 1 -- 172.21.15.78:0/2146466559 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v83) v1 ==== 81+0+591 (secure 0 0 0) 0x7f198409a380 con 0x7f19941017b0 2024-03-14T09:52:22.441 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/3890232539 2024-03-15T09:46:40.988022+0000 2024-03-14T09:52:22.441 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/3890232539 2024-03-15T09:46:40.988022+0000 2024-03-14T09:52:22.441 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2260754576 2024-03-15T09:46:40.988022+0000 2024-03-14T09:52:22.441 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/1325295736 2024-03-15T09:46:17.100896+0000 2024-03-14T09:52:22.441 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/119757382 2024-03-15T09:46:17.100896+0000 2024-03-14T09:52:22.441 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/1325295736 2024-03-15T09:46:17.100896+0000 2024-03-14T09:52:22.442 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1127810454 2024-03-15T09:46:17.100896+0000 2024-03-14T09:52:22.442 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2380275837 2024-03-15T09:46:40.988022+0000 2024-03-14T09:52:22.442 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2472038902 2024-03-15T09:46:17.100896+0000 2024-03-14T09:52:22.442 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/4237457492 2024-03-15T09:46:40.988022+0000 2024-03-14T09:52:22.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.442+0000 7f199a000700 1 -- 172.21.15.78:0/2146466559 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19780a0a90 msgr2=0x7f19780a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:22.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.442+0000 7f199a000700 1 --2- 172.21.15.78:0/2146466559 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19780a0a90 0x7f19780a2f50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f197c00f540 tx=0x7f197c00b040 comp rx=0 tx=0).stop 2024-03-14T09:52:22.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.442+0000 7f199a000700 1 -- 172.21.15.78:0/2146466559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19941017b0 msgr2=0x7f1994199290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:22.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.442+0000 7f199a000700 1 --2- 172.21.15.78:0/2146466559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19941017b0 0x7f1994199290 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f198400b680 tx=0x7f1984009580 comp rx=0 tx=0).stop 2024-03-14T09:52:22.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.442+0000 7f199a000700 1 -- 172.21.15.78:0/2146466559 shutdown_connections 2024-03-14T09:52:22.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.442+0000 7f199a000700 1 --2- 172.21.15.78:0/2146466559 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19780a0a90 0x7f19780a2f50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.442+0000 7f199a000700 1 --2- 172.21.15.78:0/2146466559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1994106a10 0x7f19941a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.442+0000 7f199a000700 1 --2- 172.21.15.78:0/2146466559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19941017b0 0x7f1994199290 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.442+0000 7f199a000700 1 --2- 172.21.15.78:0/2146466559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19941040e0 0x7f19941997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.442+0000 7f199a000700 1 -- 172.21.15.78:0/2146466559 >> 172.21.15.78:0/2146466559 conn(0x7f19940fb1a0 msgr2=0x7f1994100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:22.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.442+0000 7f199a000700 1 -- 172.21.15.78:0/2146466559 shutdown_connections 2024-03-14T09:52:22.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.442+0000 7f199a000700 1 -- 172.21.15.78:0/2146466559 wait complete. 2024-03-14T09:52:22.446 INFO:teuthology.orchestra.run.smithi078.stderr:listed 10 entries 2024-03-14T09:52:22.460 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:52:22.460 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-14T09:52:22.503 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:52:22.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:22 smithi132 ceph-mon[47959]: pgmap v243: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 94 B/s wr, 0 op/s 2024-03-14T09:52:22.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:22 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1275934961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:52:22.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:22 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/368083454' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:52:22.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:22 smithi078 ceph-mon[38143]: pgmap v243: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 94 B/s wr, 0 op/s 2024-03-14T09:52:22.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:22 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1275934961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:52:22.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:22 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/368083454' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:52:22.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.735+0000 7fcd9e200700 1 -- 172.21.15.78:0/936396825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd98101e40 msgr2=0x7fcd981022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:22.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.735+0000 7fcd9e200700 1 --2- 172.21.15.78:0/936396825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd98101e40 0x7fcd981022a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcd840099c0 tx=0x7fcd84009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.735+0000 7fcd9e200700 1 -- 172.21.15.78:0/936396825 shutdown_connections 2024-03-14T09:52:22.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.735+0000 7fcd9e200700 1 --2- 172.21.15.78:0/936396825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd981027e0 0x7fcd98109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.735+0000 7fcd9e200700 1 --2- 172.21.15.78:0/936396825 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd98100c50 0x7fcd98101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.735+0000 7fcd9e200700 1 --2- 172.21.15.78:0/936396825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd98101e40 0x7fcd981022a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.735+0000 7fcd9e200700 1 -- 172.21.15.78:0/936396825 >> 172.21.15.78:0/936396825 conn(0x7fcd980fc1c0 msgr2=0x7fcd980fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:22.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.735+0000 7fcd9e200700 1 -- 172.21.15.78:0/936396825 shutdown_connections 2024-03-14T09:52:22.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.736+0000 7fcd9e200700 1 -- 172.21.15.78:0/936396825 wait complete. 2024-03-14T09:52:22.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.736+0000 7fcd9e200700 1 Processor -- start 2024-03-14T09:52:22.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.737+0000 7fcd9e200700 1 -- start start 2024-03-14T09:52:22.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.737+0000 7fcd9e200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd98100c50 0x7fcd98074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:22.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.737+0000 7fcd9e200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd98101e40 0x7fcd980731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:22.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.737+0000 7fcd9e200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd981027e0 0x7fcd980736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:22.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.737+0000 7fcd9e200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd98073de0 con 0x7fcd98100c50 2024-03-14T09:52:22.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.737+0000 7fcd9e200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd98073f50 con 0x7fcd98101e40 2024-03-14T09:52:22.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.737+0000 7fcd9e200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd981a10b0 con 0x7fcd981027e0 2024-03-14T09:52:22.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.737+0000 7fcd97400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd98100c50 0x7fcd98074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:22.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.737+0000 7fcd97400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd98100c50 0x7fcd98074ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44626/0 (socket says 172.21.15.78:44626) 2024-03-14T09:52:22.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.737+0000 7fcd97400700 1 -- 172.21.15.78:0/312073411 learned_addr learned my addr 172.21.15.78:0/312073411 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:22.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.737+0000 7fcd96a00700 1 --2- 172.21.15.78:0/312073411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd98101e40 0x7fcd980731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:22.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.737+0000 7fcd97e00700 1 --2- 172.21.15.78:0/312073411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd981027e0 0x7fcd980736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:22.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.738+0000 7fcd97400700 1 -- 172.21.15.78:0/312073411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd981027e0 msgr2=0x7fcd980736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:22.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.738+0000 7fcd97400700 1 --2- 172.21.15.78:0/312073411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd981027e0 0x7fcd980736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.738+0000 7fcd97400700 1 -- 172.21.15.78:0/312073411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd98101e40 msgr2=0x7fcd980731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:22.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.738+0000 7fcd97400700 1 --2- 172.21.15.78:0/312073411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd98101e40 0x7fcd980731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.738+0000 7fcd97400700 1 -- 172.21.15.78:0/312073411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd840095f0 con 0x7fcd98100c50 2024-03-14T09:52:22.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.738+0000 7fcd97e00700 1 --2- 172.21.15.78:0/312073411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd981027e0 0x7fcd980736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:52:22.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.738+0000 7fcd96a00700 1 --2- 172.21.15.78:0/312073411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd98101e40 0x7fcd980731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:22.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.738+0000 7fcd97400700 1 --2- 172.21.15.78:0/312073411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd98100c50 0x7fcd98074ac0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcd8800d980 tx=0x7fcd8800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:22.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.738+0000 7fcd8fe00700 1 -- 172.21.15.78:0/312073411 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8800b950 con 0x7fcd98100c50 2024-03-14T09:52:22.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.738+0000 7fcd9e200700 1 -- 172.21.15.78:0/312073411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd981a1310 con 0x7fcd98100c50 2024-03-14T09:52:22.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.738+0000 7fcd8fe00700 1 -- 172.21.15.78:0/312073411 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd88007620 con 0x7fcd98100c50 2024-03-14T09:52:22.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.738+0000 7fcd8fe00700 1 -- 172.21.15.78:0/312073411 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd88004830 con 0x7fcd98100c50 2024-03-14T09:52:22.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.738+0000 7fcd9e200700 1 -- 172.21.15.78:0/312073411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd981a1870 con 0x7fcd98100c50 2024-03-14T09:52:22.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.739+0000 7fcd9e200700 1 -- 172.21.15.78:0/312073411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd98108120 con 0x7fcd98100c50 2024-03-14T09:52:22.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.741+0000 7fcd8fe00700 1 -- 172.21.15.78:0/312073411 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fcd88010640 con 0x7fcd98100c50 2024-03-14T09:52:22.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.742+0000 7fcd8fe00700 1 --2- 172.21.15.78:0/312073411 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcd7c0a0a90 0x7fcd7c0a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:22.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.742+0000 7fcd8fe00700 1 -- 172.21.15.78:0/312073411 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fcd880cc420 con 0x7fcd98100c50 2024-03-14T09:52:22.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.743+0000 7fcd96a00700 1 --2- 172.21.15.78:0/312073411 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcd7c0a0a90 0x7fcd7c0a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:22.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.743+0000 7fcd96a00700 1 --2- 172.21.15.78:0/312073411 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcd7c0a0a90 0x7fcd7c0a2f50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcd84006010 tx=0x7fcd8400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:22.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.747+0000 7fcd8fe00700 1 -- 172.21.15.78:0/312073411 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcd88095ba0 con 0x7fcd98100c50 2024-03-14T09:52:22.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:22 smithi152 ceph-mon[48279]: pgmap v243: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 94 B/s wr, 0 op/s 2024-03-14T09:52:22.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:22 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1275934961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:52:22.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:22 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/368083454' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:52:22.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.972+0000 7fcd9e200700 1 -- 172.21.15.78:0/312073411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd9802ccb0 con 0x7fcd98100c50 2024-03-14T09:52:22.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.974+0000 7fcd8fe00700 1 -- 172.21.15.78:0/312073411 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+23925 (secure 0 0 0) 0x7fcd880991c0 con 0x7fcd98100c50 2024-03-14T09:52:22.974 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:22.975 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":83,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:52:10.245498+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:52:22.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.976+0000 7fcd9e200700 1 -- 172.21.15.78:0/312073411 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcd7c0a0a90 msgr2=0x7fcd7c0a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:22.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.976+0000 7fcd9e200700 1 --2- 172.21.15.78:0/312073411 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcd7c0a0a90 0x7fcd7c0a2f50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcd84006010 tx=0x7fcd8400b540 comp rx=0 tx=0).stop 2024-03-14T09:52:22.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.976+0000 7fcd9e200700 1 -- 172.21.15.78:0/312073411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd98100c50 msgr2=0x7fcd98074ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:22.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.976+0000 7fcd9e200700 1 --2- 172.21.15.78:0/312073411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd98100c50 0x7fcd98074ac0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcd8800d980 tx=0x7fcd8800dc90 comp rx=0 tx=0).stop 2024-03-14T09:52:22.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.976+0000 7fcd9e200700 1 -- 172.21.15.78:0/312073411 shutdown_connections 2024-03-14T09:52:22.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.976+0000 7fcd9e200700 1 --2- 172.21.15.78:0/312073411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcd981027e0 0x7fcd980736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.976+0000 7fcd9e200700 1 --2- 172.21.15.78:0/312073411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd98100c50 0x7fcd98074ac0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.976+0000 7fcd9e200700 1 --2- 172.21.15.78:0/312073411 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcd7c0a0a90 0x7fcd7c0a2f50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.976+0000 7fcd9e200700 1 --2- 172.21.15.78:0/312073411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd98101e40 0x7fcd980731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:22.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.976+0000 7fcd9e200700 1 -- 172.21.15.78:0/312073411 >> 172.21.15.78:0/312073411 conn(0x7fcd980fc1c0 msgr2=0x7fcd98105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:22.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.977+0000 7fcd9e200700 1 -- 172.21.15.78:0/312073411 shutdown_connections 2024-03-14T09:52:22.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:22.977+0000 7fcd9e200700 1 -- 172.21.15.78:0/312073411 wait complete. 2024-03-14T09:52:22.996 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T09:48:37.618450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-14T09:52:03.938666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-14T09:52:05.149985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-14T09:52:08.227698+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-14T09:52:22.996 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T09:52:23.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.230+0000 7f8788c00700 1 -- 172.21.15.78:0/2198919934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8784100d50 msgr2=0x7f8784101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:23.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.230+0000 7f8788c00700 1 --2- 172.21.15.78:0/2198919934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8784100d50 0x7f8784101170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f87740099c0 tx=0x7f8774009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.230+0000 7f8788c00700 1 -- 172.21.15.78:0/2198919934 shutdown_connections 2024-03-14T09:52:23.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.230+0000 7f8788c00700 1 --2- 172.21.15.78:0/2198919934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f87841028e0 0x7f8784109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.230+0000 7f8788c00700 1 --2- 172.21.15.78:0/2198919934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8784100d50 0x7f8784101170 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.230+0000 7f8788c00700 1 --2- 172.21.15.78:0/2198919934 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8784101f40 0x7f87841023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.230+0000 7f8788c00700 1 -- 172.21.15.78:0/2198919934 >> 172.21.15.78:0/2198919934 conn(0x7f87840fc2c0 msgr2=0x7f87840fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:23.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.230+0000 7f8788c00700 1 -- 172.21.15.78:0/2198919934 shutdown_connections 2024-03-14T09:52:23.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.230+0000 7f8788c00700 1 -- 172.21.15.78:0/2198919934 wait complete. 2024-03-14T09:52:23.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.231+0000 7f8788c00700 1 Processor -- start 2024-03-14T09:52:23.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.231+0000 7f8788c00700 1 -- start start 2024-03-14T09:52:23.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.231+0000 7f8788c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8784100d50 0x7f8784199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.231+0000 7f8788c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8784101f40 0x7f8784199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.231+0000 7f8788c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87841028e0 0x7f87841a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.231+0000 7f8788c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87841a0e50 con 0x7f87841028e0 2024-03-14T09:52:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.231+0000 7f8788c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87841a0f90 con 0x7f8784101f40 2024-03-14T09:52:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.231+0000 7f8788c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87841a1120 con 0x7f8784100d50 2024-03-14T09:52:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.231+0000 7f8782800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87841028e0 0x7f87841a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.232+0000 7f8782800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87841028e0 0x7f87841a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44642/0 (socket says 172.21.15.78:44642) 2024-03-14T09:52:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.232+0000 7f8782800700 1 -- 172.21.15.78:0/1503110897 learned_addr learned my addr 172.21.15.78:0/1503110897 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:23.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.232+0000 7f8781400700 1 --2- 172.21.15.78:0/1503110897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8784101f40 0x7f8784199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:23.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.232+0000 7f8781e00700 1 --2- 172.21.15.78:0/1503110897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8784100d50 0x7f8784199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:23.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.232+0000 7f8782800700 1 -- 172.21.15.78:0/1503110897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8784100d50 msgr2=0x7f8784199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:23.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.232+0000 7f8782800700 1 --2- 172.21.15.78:0/1503110897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8784100d50 0x7f8784199320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.232+0000 7f8782800700 1 -- 172.21.15.78:0/1503110897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8784101f40 msgr2=0x7f8784199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:23.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.232+0000 7f8782800700 1 --2- 172.21.15.78:0/1503110897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8784101f40 0x7f8784199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.232+0000 7f8782800700 1 -- 172.21.15.78:0/1503110897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87740095f0 con 0x7f87841028e0 2024-03-14T09:52:23.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.232+0000 7f8782800700 1 --2- 172.21.15.78:0/1503110897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87841028e0 0x7f87841a0880 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f877c00ef30 tx=0x7f877c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:23.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.232+0000 7f877aa00700 1 -- 172.21.15.78:0/1503110897 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c018070 con 0x7f87841028e0 2024-03-14T09:52:23.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.232+0000 7f877aa00700 1 -- 172.21.15.78:0/1503110897 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f877c00f040 con 0x7f87841028e0 2024-03-14T09:52:23.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.232+0000 7f8788c00700 1 -- 172.21.15.78:0/1503110897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87841a13a0 con 0x7f87841028e0 2024-03-14T09:52:23.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.232+0000 7f877aa00700 1 -- 172.21.15.78:0/1503110897 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c00cd00 con 0x7f87841028e0 2024-03-14T09:52:23.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.233+0000 7f8788c00700 1 -- 172.21.15.78:0/1503110897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87841a1900 con 0x7f87841028e0 2024-03-14T09:52:23.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.233+0000 7f8788c00700 1 -- 172.21.15.78:0/1503110897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8784067ac0 con 0x7f87841028e0 2024-03-14T09:52:23.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.235+0000 7f877aa00700 1 -- 172.21.15.78:0/1503110897 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f877c003a00 con 0x7f87841028e0 2024-03-14T09:52:23.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.237+0000 7f877aa00700 1 --2- 172.21.15.78:0/1503110897 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87680a0b90 0x7f87680a3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:23.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.238+0000 7f877aa00700 1 -- 172.21.15.78:0/1503110897 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f877c0cbd30 con 0x7f87841028e0 2024-03-14T09:52:23.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.238+0000 7f8781e00700 1 --2- 172.21.15.78:0/1503110897 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87680a0b90 0x7f87680a3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:23.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.238+0000 7f8781e00700 1 --2- 172.21.15.78:0/1503110897 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87680a0b90 0x7f87680a3050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8774009990 tx=0x7f8774000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:23.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.241+0000 7f877aa00700 1 -- 172.21.15.78:0/1503110897 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f877c0d1020 con 0x7f87841028e0 2024-03-14T09:52:23.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.450+0000 7f8788c00700 1 -- 172.21.15.78:0/1503110897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f87841a1bc0 con 0x7f87841028e0 2024-03-14T09:52:23.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.450+0000 7f877aa00700 1 -- 172.21.15.78:0/1503110897 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f877c095540 con 0x7f87841028e0 2024-03-14T09:52:23.451 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-14T09:52:23.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.453+0000 7f8788c00700 1 -- 172.21.15.78:0/1503110897 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87680a0b90 msgr2=0x7f87680a3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:23.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.453+0000 7f8788c00700 1 --2- 172.21.15.78:0/1503110897 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87680a0b90 0x7f87680a3050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8774009990 tx=0x7f8774000bc0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.453+0000 7f8788c00700 1 -- 172.21.15.78:0/1503110897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87841028e0 msgr2=0x7f87841a0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:23.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.453+0000 7f8788c00700 1 --2- 172.21.15.78:0/1503110897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87841028e0 0x7f87841a0880 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f877c00ef30 tx=0x7f877c00c3a0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.453+0000 7f8788c00700 1 -- 172.21.15.78:0/1503110897 shutdown_connections 2024-03-14T09:52:23.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.453+0000 7f8788c00700 1 --2- 172.21.15.78:0/1503110897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f87841028e0 0x7f87841a0880 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.453+0000 7f8788c00700 1 --2- 172.21.15.78:0/1503110897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8784100d50 0x7f8784199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.453+0000 7f8788c00700 1 --2- 172.21.15.78:0/1503110897 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87680a0b90 0x7f87680a3050 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.453+0000 7f8788c00700 1 --2- 172.21.15.78:0/1503110897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8784101f40 0x7f8784199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.453+0000 7f8788c00700 1 -- 172.21.15.78:0/1503110897 >> 172.21.15.78:0/1503110897 conn(0x7f87840fc2c0 msgr2=0x7f8784107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:23.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.453+0000 7f8788c00700 1 -- 172.21.15.78:0/1503110897 shutdown_connections 2024-03-14T09:52:23.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.453+0000 7f8788c00700 1 -- 172.21.15.78:0/1503110897 wait complete. 2024-03-14T09:52:23.471 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-14T09:52:23.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:23 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2146466559' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-14T09:52:23.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:23 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/312073411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:23.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:23 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2146466559' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-14T09:52:23.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:23 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/312073411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:23.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.665+0000 7fb129000700 1 -- 172.21.15.78:0/223654232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb124100d50 msgr2=0x7fb124101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:23.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.665+0000 7fb129000700 1 --2- 172.21.15.78:0/223654232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb124100d50 0x7fb124101170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb1140099c0 tx=0x7fb114009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.666+0000 7fb129000700 1 -- 172.21.15.78:0/223654232 shutdown_connections 2024-03-14T09:52:23.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.666+0000 7fb129000700 1 --2- 172.21.15.78:0/223654232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1241028e0 0x7fb124109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.666+0000 7fb129000700 1 --2- 172.21.15.78:0/223654232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb124100d50 0x7fb124101170 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.666+0000 7fb129000700 1 --2- 172.21.15.78:0/223654232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb124101f40 0x7fb1241023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.666+0000 7fb129000700 1 -- 172.21.15.78:0/223654232 >> 172.21.15.78:0/223654232 conn(0x7fb1240fc2c0 msgr2=0x7fb1240fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:23.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.666+0000 7fb129000700 1 -- 172.21.15.78:0/223654232 shutdown_connections 2024-03-14T09:52:23.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.666+0000 7fb129000700 1 -- 172.21.15.78:0/223654232 wait complete. 2024-03-14T09:52:23.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.667+0000 7fb129000700 1 Processor -- start 2024-03-14T09:52:23.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.667+0000 7fb129000700 1 -- start start 2024-03-14T09:52:23.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.667+0000 7fb129000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb124100d50 0x7fb124199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:23.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.667+0000 7fb129000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb124101f40 0x7fb1241998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:23.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.667+0000 7fb129000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1241028e0 0x7fb1241a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:23.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.667+0000 7fb129000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1241a0ea0 con 0x7fb1241028e0 2024-03-14T09:52:23.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.667+0000 7fb129000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1241a0fe0 con 0x7fb124100d50 2024-03-14T09:52:23.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.667+0000 7fb129000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1241a1170 con 0x7fb124101f40 2024-03-14T09:52:23.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.667+0000 7fb122800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1241028e0 0x7fb1241a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:23.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.667+0000 7fb122800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1241028e0 0x7fb1241a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44660/0 (socket says 172.21.15.78:44660) 2024-03-14T09:52:23.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.667+0000 7fb121e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb124100d50 0x7fb124199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:23.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.667+0000 7fb122800700 1 -- 172.21.15.78:0/2641823253 learned_addr learned my addr 172.21.15.78:0/2641823253 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:23.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.667+0000 7fb121400700 1 --2- 172.21.15.78:0/2641823253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb124101f40 0x7fb1241998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:23.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.668+0000 7fb121e00700 1 -- 172.21.15.78:0/2641823253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb124101f40 msgr2=0x7fb1241998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:23.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.668+0000 7fb121e00700 1 --2- 172.21.15.78:0/2641823253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb124101f40 0x7fb1241998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.668+0000 7fb121e00700 1 -- 172.21.15.78:0/2641823253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1241028e0 msgr2=0x7fb1241a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:23.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.668+0000 7fb121e00700 1 --2- 172.21.15.78:0/2641823253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1241028e0 0x7fb1241a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.668+0000 7fb121e00700 1 -- 172.21.15.78:0/2641823253 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1140095f0 con 0x7fb124100d50 2024-03-14T09:52:23.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.668+0000 7fb121e00700 1 --2- 172.21.15.78:0/2641823253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb124100d50 0x7fb124199370 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb1140036f0 tx=0x7fb114005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:23.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.668+0000 7fb11aa00700 1 -- 172.21.15.78:0/2641823253 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11401d070 con 0x7fb124100d50 2024-03-14T09:52:23.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.668+0000 7fb11aa00700 1 -- 172.21.15.78:0/2641823253 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb114003f90 con 0x7fb124100d50 2024-03-14T09:52:23.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.668+0000 7fb129000700 1 -- 172.21.15.78:0/2641823253 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1241a13f0 con 0x7fb124100d50 2024-03-14T09:52:23.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.668+0000 7fb11aa00700 1 -- 172.21.15.78:0/2641823253 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb114022bb0 con 0x7fb124100d50 2024-03-14T09:52:23.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.669+0000 7fb129000700 1 -- 172.21.15.78:0/2641823253 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1241a1810 con 0x7fb124100d50 2024-03-14T09:52:23.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.669+0000 7fb129000700 1 -- 172.21.15.78:0/2641823253 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb124067ac0 con 0x7fb124100d50 2024-03-14T09:52:23.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.671+0000 7fb11aa00700 1 -- 172.21.15.78:0/2641823253 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fb114003ad0 con 0x7fb124100d50 2024-03-14T09:52:23.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.673+0000 7fb11aa00700 1 --2- 172.21.15.78:0/2641823253 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb1080a0a90 0x7fb1080a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:23.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.673+0000 7fb11aa00700 1 -- 172.21.15.78:0/2641823253 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb1140cde00 con 0x7fb124100d50 2024-03-14T09:52:23.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.673+0000 7fb121400700 1 --2- 172.21.15.78:0/2641823253 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb1080a0a90 0x7fb1080a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:23.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.673+0000 7fb121400700 1 --2- 172.21.15.78:0/2641823253 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb1080a0a90 0x7fb1080a2f50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb110005950 tx=0x7fb11000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:23.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.675+0000 7fb11aa00700 1 -- 172.21.15.78:0/2641823253 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb114097610 con 0x7fb124100d50 2024-03-14T09:52:23.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:23 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2146466559' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-14T09:52:23.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:23 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/312073411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:23.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.882+0000 7fb129000700 1 -- 172.21.15.78:0/2641823253 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb12410cb60 con 0x7fb124100d50 2024-03-14T09:52:23.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.883+0000 7fb11aa00700 1 -- 172.21.15.78:0/2641823253 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb11409ac30 con 0x7fb124100d50 2024-03-14T09:52:23.883 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:52:23.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.885+0000 7fb129000700 1 -- 172.21.15.78:0/2641823253 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb1080a0a90 msgr2=0x7fb1080a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:23.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.885+0000 7fb129000700 1 --2- 172.21.15.78:0/2641823253 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb1080a0a90 0x7fb1080a2f50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb110005950 tx=0x7fb11000b410 comp rx=0 tx=0).stop 2024-03-14T09:52:23.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.885+0000 7fb129000700 1 -- 172.21.15.78:0/2641823253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb124100d50 msgr2=0x7fb124199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:23.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.885+0000 7fb129000700 1 --2- 172.21.15.78:0/2641823253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb124100d50 0x7fb124199370 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb1140036f0 tx=0x7fb114005740 comp rx=0 tx=0).stop 2024-03-14T09:52:23.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.885+0000 7fb129000700 1 -- 172.21.15.78:0/2641823253 shutdown_connections 2024-03-14T09:52:23.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.885+0000 7fb129000700 1 --2- 172.21.15.78:0/2641823253 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb1080a0a90 0x7fb1080a2f50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.885+0000 7fb129000700 1 --2- 172.21.15.78:0/2641823253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb124100d50 0x7fb124199370 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.885+0000 7fb129000700 1 --2- 172.21.15.78:0/2641823253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1241028e0 0x7fb1241a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.885+0000 7fb129000700 1 --2- 172.21.15.78:0/2641823253 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb124101f40 0x7fb1241998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:23.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.885+0000 7fb129000700 1 -- 172.21.15.78:0/2641823253 >> 172.21.15.78:0/2641823253 conn(0x7fb1240fc2c0 msgr2=0x7fb124072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:23.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.885+0000 7fb129000700 1 -- 172.21.15.78:0/2641823253 shutdown_connections 2024-03-14T09:52:23.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:23.885+0000 7fb129000700 1 -- 172.21.15.78:0/2641823253 wait complete. 2024-03-14T09:52:23.904 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-14T09:52:24.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.098+0000 7ff1b5c00700 1 -- 172.21.15.78:0/1142416339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1b01040e0 msgr2=0x7ff1b01064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:24.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.098+0000 7ff1b5c00700 1 --2- 172.21.15.78:0/1142416339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1b01040e0 0x7ff1b01064d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff19c009990 tx=0x7ff19c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.098+0000 7ff1b5c00700 1 -- 172.21.15.78:0/1142416339 shutdown_connections 2024-03-14T09:52:24.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.098+0000 7ff1b5c00700 1 --2- 172.21.15.78:0/1142416339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1b0106a10 0x7ff1b0108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.098+0000 7ff1b5c00700 1 --2- 172.21.15.78:0/1142416339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01017b0 0x7ff1b0103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.098+0000 7ff1b5c00700 1 --2- 172.21.15.78:0/1142416339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1b01040e0 0x7ff1b01064d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.098+0000 7ff1b5c00700 1 -- 172.21.15.78:0/1142416339 >> 172.21.15.78:0/1142416339 conn(0x7ff1b00fb1a0 msgr2=0x7ff1b00fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:24.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.098+0000 7ff1b5c00700 1 -- 172.21.15.78:0/1142416339 shutdown_connections 2024-03-14T09:52:24.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.098+0000 7ff1b5c00700 1 -- 172.21.15.78:0/1142416339 wait complete. 2024-03-14T09:52:24.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.099+0000 7ff1b5c00700 1 Processor -- start 2024-03-14T09:52:24.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.099+0000 7ff1b5c00700 1 -- start start 2024-03-14T09:52:24.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.099+0000 7ff1b5c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1b01017b0 0x7ff1b0199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:24.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.099+0000 7ff1b5c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01040e0 0x7ff1b0199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:24.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.099+0000 7ff1b5c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1b0106a10 0x7ff1b01a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:24.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.099+0000 7ff1b5c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b01a0e30 con 0x7ff1b0106a10 2024-03-14T09:52:24.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.099+0000 7ff1b5c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b01a0f70 con 0x7ff1b01017b0 2024-03-14T09:52:24.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.099+0000 7ff1b5c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b01a1100 con 0x7ff1b01040e0 2024-03-14T09:52:24.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.100+0000 7ff1af400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1b0106a10 0x7ff1b01a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:24.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.100+0000 7ff1af400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1b0106a10 0x7ff1b01a0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41340/0 (socket says 172.21.15.78:41340) 2024-03-14T09:52:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.100+0000 7ff1af400700 1 -- 172.21.15.78:0/1799633648 learned_addr learned my addr 172.21.15.78:0/1799633648 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.100+0000 7ff1aea00700 1 --2- 172.21.15.78:0/1799633648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1b01017b0 0x7ff1b0199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.100+0000 7ff1ae000700 1 --2- 172.21.15.78:0/1799633648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01040e0 0x7ff1b0199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.100+0000 7ff1af400700 1 -- 172.21.15.78:0/1799633648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01040e0 msgr2=0x7ff1b0199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.100+0000 7ff1af400700 1 --2- 172.21.15.78:0/1799633648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01040e0 0x7ff1b0199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.100+0000 7ff1af400700 1 -- 172.21.15.78:0/1799633648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1b01017b0 msgr2=0x7ff1b0199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.100+0000 7ff1af400700 1 --2- 172.21.15.78:0/1799633648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1b01017b0 0x7ff1b0199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.100+0000 7ff1af400700 1 -- 172.21.15.78:0/1799633648 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c0095f0 con 0x7ff1b0106a10 2024-03-14T09:52:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.100+0000 7ff1ae000700 1 --2- 172.21.15.78:0/1799633648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01040e0 0x7ff1b0199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:52:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.100+0000 7ff1af400700 1 --2- 172.21.15.78:0/1799633648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1b0106a10 0x7ff1b01a0860 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff1a800ef30 tx=0x7ff1a800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.100+0000 7ff1a7400700 1 -- 172.21.15.78:0/1799633648 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8018070 con 0x7ff1b0106a10 2024-03-14T09:52:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.101+0000 7ff1a7400700 1 -- 172.21.15.78:0/1799633648 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1a800f040 con 0x7ff1b0106a10 2024-03-14T09:52:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.101+0000 7ff1b5c00700 1 -- 172.21.15.78:0/1799633648 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b01a1380 con 0x7ff1b0106a10 2024-03-14T09:52:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.101+0000 7ff1a7400700 1 -- 172.21.15.78:0/1799633648 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a800cdb0 con 0x7ff1b0106a10 2024-03-14T09:52:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.101+0000 7ff1b5c00700 1 -- 172.21.15.78:0/1799633648 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b01a18e0 con 0x7ff1b0106a10 2024-03-14T09:52:24.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.102+0000 7ff1b5c00700 1 -- 172.21.15.78:0/1799633648 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b01085e0 con 0x7ff1b0106a10 2024-03-14T09:52:24.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.104+0000 7ff1a7400700 1 -- 172.21.15.78:0/1799633648 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7ff1a8003a00 con 0x7ff1b0106a10 2024-03-14T09:52:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.106+0000 7ff1a7400700 1 --2- 172.21.15.78:0/1799633648 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff1940a0b10 0x7ff1940a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.106+0000 7ff1a7400700 1 -- 172.21.15.78:0/1799633648 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff1a80cbdc0 con 0x7ff1b0106a10 2024-03-14T09:52:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.106+0000 7ff1aea00700 1 --2- 172.21.15.78:0/1799633648 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff1940a0b10 0x7ff1940a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:24.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.106+0000 7ff1aea00700 1 --2- 172.21.15.78:0/1799633648 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff1940a0b10 0x7ff1940a2fd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009c60 tx=0x7ff1a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.110+0000 7ff1a7400700 1 -- 172.21.15.78:0/1799633648 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff1a80955d0 con 0x7ff1b0106a10 2024-03-14T09:52:24.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.315+0000 7ff1b5c00700 1 -- 172.21.15.78:0/1799633648 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff1b0107fc0 con 0x7ff1b0106a10 2024-03-14T09:52:24.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.316+0000 7ff1a7400700 1 -- 172.21.15.78:0/1799633648 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff1a8098bf0 con 0x7ff1b0106a10 2024-03-14T09:52:24.316 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-14T09:52:24.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.318+0000 7ff1b5c00700 1 -- 172.21.15.78:0/1799633648 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff1940a0b10 msgr2=0x7ff1940a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:24.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.318+0000 7ff1b5c00700 1 --2- 172.21.15.78:0/1799633648 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff1940a0b10 0x7ff1940a2fd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009c60 tx=0x7ff1a0009260 comp rx=0 tx=0).stop 2024-03-14T09:52:24.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.318+0000 7ff1b5c00700 1 -- 172.21.15.78:0/1799633648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1b0106a10 msgr2=0x7ff1b01a0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:24.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.318+0000 7ff1b5c00700 1 --2- 172.21.15.78:0/1799633648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1b0106a10 0x7ff1b01a0860 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff1a800ef30 tx=0x7ff1a800c3a0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.318+0000 7ff1b5c00700 1 -- 172.21.15.78:0/1799633648 shutdown_connections 2024-03-14T09:52:24.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.318+0000 7ff1b5c00700 1 --2- 172.21.15.78:0/1799633648 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff1940a0b10 0x7ff1940a2fd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.318+0000 7ff1b5c00700 1 --2- 172.21.15.78:0/1799633648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1b01017b0 0x7ff1b0199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.318+0000 7ff1b5c00700 1 --2- 172.21.15.78:0/1799633648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1b0106a10 0x7ff1b01a0860 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.318+0000 7ff1b5c00700 1 --2- 172.21.15.78:0/1799633648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1b01040e0 0x7ff1b0199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.318+0000 7ff1b5c00700 1 -- 172.21.15.78:0/1799633648 >> 172.21.15.78:0/1799633648 conn(0x7ff1b00fb1a0 msgr2=0x7ff1b0101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:24.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.318+0000 7ff1b5c00700 1 -- 172.21.15.78:0/1799633648 shutdown_connections 2024-03-14T09:52:24.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.318+0000 7ff1b5c00700 1 -- 172.21.15.78:0/1799633648 wait complete. 2024-03-14T09:52:24.337 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-14T09:52:24.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:24 smithi132 ceph-mon[47959]: pgmap v244: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-14T09:52:24.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:24 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1503110897' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:52:24.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:24 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2641823253' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:52:24.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.532+0000 7f8b20a00700 1 -- 172.21.15.78:0/3021926914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b1c1028d0 msgr2=0x7f8b1c109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:24.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.532+0000 7f8b20a00700 1 --2- 172.21.15.78:0/3021926914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b1c1028d0 0x7f8b1c109180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009f90 tx=0x7f8b1400b330 comp rx=0 tx=0).stop 2024-03-14T09:52:24.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.532+0000 7f8b20a00700 1 -- 172.21.15.78:0/3021926914 shutdown_connections 2024-03-14T09:52:24.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.532+0000 7f8b20a00700 1 --2- 172.21.15.78:0/3021926914 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b1c1028d0 0x7f8b1c109180 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.532+0000 7f8b20a00700 1 --2- 172.21.15.78:0/3021926914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b1c100d40 0x7f8b1c101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.532+0000 7f8b20a00700 1 --2- 172.21.15.78:0/3021926914 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b1c101f30 0x7f8b1c102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.532+0000 7f8b20a00700 1 -- 172.21.15.78:0/3021926914 >> 172.21.15.78:0/3021926914 conn(0x7f8b1c0fc2b0 msgr2=0x7f8b1c0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:24.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.532+0000 7f8b20a00700 1 -- 172.21.15.78:0/3021926914 shutdown_connections 2024-03-14T09:52:24.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.532+0000 7f8b20a00700 1 -- 172.21.15.78:0/3021926914 wait complete. 2024-03-14T09:52:24.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.533+0000 7f8b20a00700 1 Processor -- start 2024-03-14T09:52:24.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b20a00700 1 -- start start 2024-03-14T09:52:24.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b20a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b1c100d40 0x7f8b1c199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:24.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b20a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b1c101f30 0x7f8b1c199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:24.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b20a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b1c1028d0 0x7f8b1c1a0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:24.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b20a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c1a0f60 con 0x7f8b1c1028d0 2024-03-14T09:52:24.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b20a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c1a10a0 con 0x7f8b1c100d40 2024-03-14T09:52:24.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b20a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1c1a1230 con 0x7f8b1c101f30 2024-03-14T09:52:24.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b19e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b1c100d40 0x7f8b1c199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:24.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b19400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b1c101f30 0x7f8b1c199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:24.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b1a800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b1c1028d0 0x7f8b1c1a0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:24.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b19e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b1c100d40 0x7f8b1c199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:49804/0 (socket says 172.21.15.78:49804) 2024-03-14T09:52:24.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b19400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b1c101f30 0x7f8b1c199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:40868/0 (socket says 172.21.15.78:40868) 2024-03-14T09:52:24.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b19e00700 1 -- 172.21.15.78:0/4180633649 learned_addr learned my addr 172.21.15.78:0/4180633649 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:24.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b19e00700 1 -- 172.21.15.78:0/4180633649 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b1c101f30 msgr2=0x7f8b1c199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:24.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b19e00700 1 --2- 172.21.15.78:0/4180633649 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b1c101f30 0x7f8b1c199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b19e00700 1 -- 172.21.15.78:0/4180633649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b1c1028d0 msgr2=0x7f8b1c1a0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:24.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b19e00700 1 --2- 172.21.15.78:0/4180633649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b1c1028d0 0x7f8b1c1a0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b19e00700 1 -- 172.21.15.78:0/4180633649 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0c0095f0 con 0x7f8b1c100d40 2024-03-14T09:52:24.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.534+0000 7f8b1a800700 1 --2- 172.21.15.78:0/4180633649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b1c1028d0 0x7f8b1c1a0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:24.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.535+0000 7f8b19e00700 1 --2- 172.21.15.78:0/4180633649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b1c100d40 0x7f8b1c199430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c00ebf0 tx=0x7f8b0c00ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:24.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.535+0000 7f8b12a00700 1 -- 172.21.15.78:0/4180633649 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c009a10 con 0x7f8b1c100d40 2024-03-14T09:52:24.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.535+0000 7f8b20a00700 1 -- 172.21.15.78:0/4180633649 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b14009c40 con 0x7f8b1c100d40 2024-03-14T09:52:24.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.535+0000 7f8b12a00700 1 -- 172.21.15.78:0/4180633649 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b0c00cc90 con 0x7f8b1c100d40 2024-03-14T09:52:24.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.535+0000 7f8b12a00700 1 -- 172.21.15.78:0/4180633649 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c018750 con 0x7f8b1c100d40 2024-03-14T09:52:24.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.535+0000 7f8b20a00700 1 -- 172.21.15.78:0/4180633649 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b1c1a18e0 con 0x7f8b1c100d40 2024-03-14T09:52:24.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.536+0000 7f8b20a00700 1 -- 172.21.15.78:0/4180633649 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b1c067ac0 con 0x7f8b1c100d40 2024-03-14T09:52:24.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.538+0000 7f8b12a00700 1 -- 172.21.15.78:0/4180633649 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f8b0c0188b0 con 0x7f8b1c100d40 2024-03-14T09:52:24.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.539+0000 7f8b12a00700 1 --2- 172.21.15.78:0/4180633649 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f8b000a0a90 0x7f8b000a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:24.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.539+0000 7f8b12a00700 1 -- 172.21.15.78:0/4180633649 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8b0c014070 con 0x7f8b1c100d40 2024-03-14T09:52:24.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.540+0000 7f8b19400700 1 --2- 172.21.15.78:0/4180633649 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f8b000a0a90 0x7f8b000a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:24.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.540+0000 7f8b19400700 1 --2- 172.21.15.78:0/4180633649 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f8b000a0a90 0x7f8b000a2f50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8b08009c60 tx=0x7f8b08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:24.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.541+0000 7f8b12a00700 1 -- 172.21.15.78:0/4180633649 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f8b0c096720 con 0x7f8b1c100d40 2024-03-14T09:52:24.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:24 smithi078 ceph-mon[38143]: pgmap v244: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-14T09:52:24.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:24 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1503110897' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:52:24.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:24 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2641823253' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:52:24.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.748+0000 7f8b20a00700 1 -- 172.21.15.78:0/4180633649 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8b1c1a1ba0 con 0x7f8b1c100d40 2024-03-14T09:52:24.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.748+0000 7f8b12a00700 1 -- 172.21.15.78:0/4180633649 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v83) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8b0c099d40 con 0x7f8b1c100d40 2024-03-14T09:52:24.748 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:52:24.748 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:24 smithi152 ceph-mon[48279]: pgmap v244: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-14T09:52:24.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:24 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1503110897' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:52:24.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:24 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2641823253' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:52:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.751+0000 7f8b20a00700 1 -- 172.21.15.78:0/4180633649 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f8b000a0a90 msgr2=0x7f8b000a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.751+0000 7f8b20a00700 1 --2- 172.21.15.78:0/4180633649 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f8b000a0a90 0x7f8b000a2f50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8b08009c60 tx=0x7f8b08009260 comp rx=0 tx=0).stop 2024-03-14T09:52:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.751+0000 7f8b20a00700 1 -- 172.21.15.78:0/4180633649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b1c100d40 msgr2=0x7f8b1c199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.751+0000 7f8b20a00700 1 --2- 172.21.15.78:0/4180633649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b1c100d40 0x7f8b1c199430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c00ebf0 tx=0x7f8b0c00ef00 comp rx=0 tx=0).stop 2024-03-14T09:52:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.751+0000 7f8b20a00700 1 -- 172.21.15.78:0/4180633649 shutdown_connections 2024-03-14T09:52:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.751+0000 7f8b20a00700 1 --2- 172.21.15.78:0/4180633649 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f8b000a0a90 0x7f8b000a2f50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.751+0000 7f8b20a00700 1 --2- 172.21.15.78:0/4180633649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b1c100d40 0x7f8b1c199430 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.751+0000 7f8b20a00700 1 --2- 172.21.15.78:0/4180633649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b1c1028d0 0x7f8b1c1a0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.751+0000 7f8b20a00700 1 --2- 172.21.15.78:0/4180633649 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b1c101f30 0x7f8b1c199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.751+0000 7f8b20a00700 1 -- 172.21.15.78:0/4180633649 >> 172.21.15.78:0/4180633649 conn(0x7f8b1c0fc2b0 msgr2=0x7f8b1c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.751+0000 7f8b20a00700 1 -- 172.21.15.78:0/4180633649 shutdown_connections 2024-03-14T09:52:24.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.751+0000 7f8b20a00700 1 -- 172.21.15.78:0/4180633649 wait complete. 2024-03-14T09:52:24.769 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-14T09:52:24.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.962+0000 7ff31de00700 1 -- 172.21.15.78:0/3088574035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff318074d90 msgr2=0x7ff3180731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:24.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.962+0000 7ff31de00700 1 --2- 172.21.15.78:0/3088574035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff318074d90 0x7ff3180731e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff308009990 tx=0x7ff308009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.962+0000 7ff31de00700 1 -- 172.21.15.78:0/3088574035 shutdown_connections 2024-03-14T09:52:24.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.962+0000 7ff31de00700 1 --2- 172.21.15.78:0/3088574035 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff318108c10 0x7ff31810b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.962+0000 7ff31de00700 1 --2- 172.21.15.78:0/3088574035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff318074d90 0x7ff3180731e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.962+0000 7ff31de00700 1 --2- 172.21.15.78:0/3088574035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3180737b0 0x7ff318073c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.962+0000 7ff31de00700 1 -- 172.21.15.78:0/3088574035 >> 172.21.15.78:0/3088574035 conn(0x7ff3180fc2b0 msgr2=0x7ff3180fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:24.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.962+0000 7ff31de00700 1 -- 172.21.15.78:0/3088574035 shutdown_connections 2024-03-14T09:52:24.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.962+0000 7ff31de00700 1 -- 172.21.15.78:0/3088574035 wait complete. 2024-03-14T09:52:24.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.963+0000 7ff31de00700 1 Processor -- start 2024-03-14T09:52:24.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.963+0000 7ff31de00700 1 -- start start 2024-03-14T09:52:24.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff31de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3180737b0 0x7ff3181992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:24.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff31de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff318074d90 0x7ff318199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:24.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff31de00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff318108c10 0x7ff3181a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:24.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff31de00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3181a0dc0 con 0x7ff318074d90 2024-03-14T09:52:24.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff31de00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3181a0f00 con 0x7ff3180737b0 2024-03-14T09:52:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff31de00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3181a1090 con 0x7ff318108c10 2024-03-14T09:52:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff316000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff318074d90 0x7ff318199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff316a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3180737b0 0x7ff3181992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff316a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3180737b0 0x7ff3181992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:49818/0 (socket says 172.21.15.78:49818) 2024-03-14T09:52:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff316000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff318074d90 0x7ff318199820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41358/0 (socket says 172.21.15.78:41358) 2024-03-14T09:52:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff316a00700 1 -- 172.21.15.78:0/3328925175 learned_addr learned my addr 172.21.15.78:0/3328925175 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff317400700 1 --2- 172.21.15.78:0/3328925175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff318108c10 0x7ff3181a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff316a00700 1 -- 172.21.15.78:0/3328925175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff318108c10 msgr2=0x7ff3181a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff316a00700 1 --2- 172.21.15.78:0/3328925175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff318108c10 0x7ff3181a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff316a00700 1 -- 172.21.15.78:0/3328925175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff318074d90 msgr2=0x7ff318199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff316a00700 1 --2- 172.21.15.78:0/3328925175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff318074d90 0x7ff318199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff316a00700 1 -- 172.21.15.78:0/3328925175 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3080095f0 con 0x7ff3180737b0 2024-03-14T09:52:24.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.964+0000 7ff316000700 1 --2- 172.21.15.78:0/3328925175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff318074d90 0x7ff318199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:24.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.965+0000 7ff316a00700 1 --2- 172.21.15.78:0/3328925175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3180737b0 0x7ff3181992e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff308005910 tx=0x7ff308009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:24.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.965+0000 7ff31ca00700 1 -- 172.21.15.78:0/3328925175 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30801d070 con 0x7ff3180737b0 2024-03-14T09:52:24.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.965+0000 7ff31ca00700 1 -- 172.21.15.78:0/3328925175 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff308021410 con 0x7ff3180737b0 2024-03-14T09:52:24.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.965+0000 7ff31de00700 1 -- 172.21.15.78:0/3328925175 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3181a1310 con 0x7ff3180737b0 2024-03-14T09:52:24.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.965+0000 7ff31ca00700 1 -- 172.21.15.78:0/3328925175 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff308017650 con 0x7ff3180737b0 2024-03-14T09:52:24.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.965+0000 7ff31de00700 1 -- 172.21.15.78:0/3328925175 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3181a1730 con 0x7ff3180737b0 2024-03-14T09:52:24.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.966+0000 7ff31de00700 1 -- 172.21.15.78:0/3328925175 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff318107a20 con 0x7ff3180737b0 2024-03-14T09:52:24.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.968+0000 7ff31ca00700 1 -- 172.21.15.78:0/3328925175 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7ff308021580 con 0x7ff3180737b0 2024-03-14T09:52:24.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.969+0000 7ff31ca00700 1 --2- 172.21.15.78:0/3328925175 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff2fc0a0a90 0x7ff2fc0a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:24.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.969+0000 7ff31ca00700 1 -- 172.21.15.78:0/3328925175 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff3080cdb40 con 0x7ff3180737b0 2024-03-14T09:52:24.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.969+0000 7ff316000700 1 --2- 172.21.15.78:0/3328925175 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff2fc0a0a90 0x7ff2fc0a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:24.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.970+0000 7ff316000700 1 --2- 172.21.15.78:0/3328925175 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff2fc0a0a90 0x7ff2fc0a2f50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff304011490 tx=0x7ff304009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:24.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:24.971+0000 7ff31ca00700 1 -- 172.21.15.78:0/3328925175 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff308097350 con 0x7ff3180737b0 2024-03-14T09:52:25.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.206+0000 7ff31de00700 1 -- 172.21.15.78:0/3328925175 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff31810ec50 con 0x7ff3180737b0 2024-03-14T09:52:25.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.207+0000 7ff31ca00700 1 -- 172.21.15.78:0/3328925175 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v83) v1 ==== 81+0+591 (secure 0 0 0) 0x7ff30809a970 con 0x7ff3180737b0 2024-03-14T09:52:25.206 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/3890232539 2024-03-15T09:46:40.988022+0000 2024-03-14T09:52:25.206 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/3890232539 2024-03-15T09:46:40.988022+0000 2024-03-14T09:52:25.206 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2260754576 2024-03-15T09:46:40.988022+0000 2024-03-14T09:52:25.206 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/1325295736 2024-03-15T09:46:17.100896+0000 2024-03-14T09:52:25.207 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/119757382 2024-03-15T09:46:17.100896+0000 2024-03-14T09:52:25.207 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/1325295736 2024-03-15T09:46:17.100896+0000 2024-03-14T09:52:25.207 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1127810454 2024-03-15T09:46:17.100896+0000 2024-03-14T09:52:25.207 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2380275837 2024-03-15T09:46:40.988022+0000 2024-03-14T09:52:25.207 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2472038902 2024-03-15T09:46:17.100896+0000 2024-03-14T09:52:25.207 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/4237457492 2024-03-15T09:46:40.988022+0000 2024-03-14T09:52:25.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.209+0000 7ff31de00700 1 -- 172.21.15.78:0/3328925175 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff2fc0a0a90 msgr2=0x7ff2fc0a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:25.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.209+0000 7ff31de00700 1 --2- 172.21.15.78:0/3328925175 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff2fc0a0a90 0x7ff2fc0a2f50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff304011490 tx=0x7ff304009260 comp rx=0 tx=0).stop 2024-03-14T09:52:25.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.209+0000 7ff31de00700 1 -- 172.21.15.78:0/3328925175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3180737b0 msgr2=0x7ff3181992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:25.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.209+0000 7ff31de00700 1 --2- 172.21.15.78:0/3328925175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3180737b0 0x7ff3181992e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff308005910 tx=0x7ff308009580 comp rx=0 tx=0).stop 2024-03-14T09:52:25.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.209+0000 7ff31de00700 1 -- 172.21.15.78:0/3328925175 shutdown_connections 2024-03-14T09:52:25.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.209+0000 7ff31de00700 1 --2- 172.21.15.78:0/3328925175 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff2fc0a0a90 0x7ff2fc0a2f50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:25.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.209+0000 7ff31de00700 1 --2- 172.21.15.78:0/3328925175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3180737b0 0x7ff3181992e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:25.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.209+0000 7ff31de00700 1 --2- 172.21.15.78:0/3328925175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff318108c10 0x7ff3181a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:25.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.209+0000 7ff31de00700 1 --2- 172.21.15.78:0/3328925175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff318074d90 0x7ff318199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:25.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.209+0000 7ff31de00700 1 -- 172.21.15.78:0/3328925175 >> 172.21.15.78:0/3328925175 conn(0x7ff3180fc2b0 msgr2=0x7ff318104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:25.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.209+0000 7ff31de00700 1 -- 172.21.15.78:0/3328925175 shutdown_connections 2024-03-14T09:52:25.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.210+0000 7ff31de00700 1 -- 172.21.15.78:0/3328925175 wait complete. 2024-03-14T09:52:25.210 INFO:teuthology.orchestra.run.smithi078.stderr:listed 10 entries 2024-03-14T09:52:25.227 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi078.front.sepia.ceph.com... 2024-03-14T09:52:25.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:25.228 DEBUG:teuthology.orchestra.run.smithi078:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-14T09:52:25.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[90482]: fuse finished with error 0 and tester_r 0 2024-03-14T09:52:25.294 INFO:teuthology.orchestra.run:waiting for 300 2024-03-14T09:52:25.374 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2024-03-14T09:52:25.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:52:25.374 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-14T09:52:25.422 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:52:25.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:25 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1799633648' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:52:25.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:25 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/4180633649' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:52:25.479 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:25 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3328925175' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-14T09:52:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:25 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1799633648' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:52:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:25 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/4180633649' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:52:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:25 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3328925175' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-14T09:52:25.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.616+0000 7f6163200700 1 -- 172.21.15.78:0/1180135598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f615c100c50 msgr2=0x7f615c101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:25.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.616+0000 7f6163200700 1 --2- 172.21.15.78:0/1180135598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f615c100c50 0x7f615c101070 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f614c009990 tx=0x7f614c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:25.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.616+0000 7f6163200700 1 -- 172.21.15.78:0/1180135598 shutdown_connections 2024-03-14T09:52:25.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.616+0000 7f6163200700 1 --2- 172.21.15.78:0/1180135598 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f615c1027e0 0x7f615c109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:25.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.616+0000 7f6163200700 1 --2- 172.21.15.78:0/1180135598 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f615c100c50 0x7f615c101070 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:25.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.616+0000 7f6163200700 1 --2- 172.21.15.78:0/1180135598 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f615c101e40 0x7f615c1022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:25.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.616+0000 7f6163200700 1 -- 172.21.15.78:0/1180135598 >> 172.21.15.78:0/1180135598 conn(0x7f615c0fc1c0 msgr2=0x7f615c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:25.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.616+0000 7f6163200700 1 -- 172.21.15.78:0/1180135598 shutdown_connections 2024-03-14T09:52:25.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.616+0000 7f6163200700 1 -- 172.21.15.78:0/1180135598 wait complete. 2024-03-14T09:52:25.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.617+0000 7f6163200700 1 Processor -- start 2024-03-14T09:52:25.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.617+0000 7f6163200700 1 -- start start 2024-03-14T09:52:25.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.618+0000 7f6163200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f615c100c50 0x7f615c199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:25.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.618+0000 7f6163200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f615c101e40 0x7f615c199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:25.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.618+0000 7f6163200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f615c1027e0 0x7f615c1a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:25.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.618+0000 7f6163200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615c1a0d50 con 0x7f615c1027e0 2024-03-14T09:52:25.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.618+0000 7f6163200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615c1a0e90 con 0x7f615c101e40 2024-03-14T09:52:25.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.618+0000 7f6163200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615c1a1020 con 0x7f615c100c50 2024-03-14T09:52:25.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.618+0000 7f6161200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f615c1027e0 0x7f615c1a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:25.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.618+0000 7f6161200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f615c1027e0 0x7f615c1a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41370/0 (socket says 172.21.15.78:41370) 2024-03-14T09:52:25.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.618+0000 7f6161200700 1 -- 172.21.15.78:0/1123806461 learned_addr learned my addr 172.21.15.78:0/1123806461 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:25.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.618+0000 7f615b400700 1 --2- 172.21.15.78:0/1123806461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f615c101e40 0x7f615c199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:25.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.618+0000 7f615be00700 1 --2- 172.21.15.78:0/1123806461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f615c100c50 0x7f615c199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:25.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.618+0000 7f6161200700 1 -- 172.21.15.78:0/1123806461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f615c100c50 msgr2=0x7f615c199220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:25.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.618+0000 7f6161200700 1 --2- 172.21.15.78:0/1123806461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f615c100c50 0x7f615c199220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:25.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.618+0000 7f6161200700 1 -- 172.21.15.78:0/1123806461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f615c101e40 msgr2=0x7f615c199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:25.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.618+0000 7f6161200700 1 --2- 172.21.15.78:0/1123806461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f615c101e40 0x7f615c199760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:25.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.618+0000 7f6161200700 1 -- 172.21.15.78:0/1123806461 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c0095f0 con 0x7f615c1027e0 2024-03-14T09:52:25.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.619+0000 7f6161200700 1 --2- 172.21.15.78:0/1123806461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f615c1027e0 0x7f615c1a0780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f615400ef30 tx=0x7f615400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:25.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.619+0000 7f6158c00700 1 -- 172.21.15.78:0/1123806461 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6154018070 con 0x7f615c1027e0 2024-03-14T09:52:25.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.619+0000 7f6163200700 1 -- 172.21.15.78:0/1123806461 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f615c1a12a0 con 0x7f615c1027e0 2024-03-14T09:52:25.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.619+0000 7f6158c00700 1 -- 172.21.15.78:0/1123806461 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f615400f040 con 0x7f615c1027e0 2024-03-14T09:52:25.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.619+0000 7f6158c00700 1 -- 172.21.15.78:0/1123806461 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615400cd00 con 0x7f615c1027e0 2024-03-14T09:52:25.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.619+0000 7f6163200700 1 -- 172.21.15.78:0/1123806461 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f615c1a1800 con 0x7f615c1027e0 2024-03-14T09:52:25.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.620+0000 7f6163200700 1 -- 172.21.15.78:0/1123806461 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615c0679e0 con 0x7f615c1027e0 2024-03-14T09:52:25.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.622+0000 7f6158c00700 1 -- 172.21.15.78:0/1123806461 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f6154003a00 con 0x7f615c1027e0 2024-03-14T09:52:25.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.624+0000 7f6158c00700 1 --2- 172.21.15.78:0/1123806461 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f61400a0a90 0x7f61400a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:25.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.624+0000 7f6158c00700 1 -- 172.21.15.78:0/1123806461 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f61540cbdb0 con 0x7f615c1027e0 2024-03-14T09:52:25.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.624+0000 7f615be00700 1 --2- 172.21.15.78:0/1123806461 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f61400a0a90 0x7f61400a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:25.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.625+0000 7f615be00700 1 --2- 172.21.15.78:0/1123806461 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f61400a0a90 0x7f61400a2f50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f614c000c00 tx=0x7f614c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:25.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.628+0000 7f6158c00700 1 -- 172.21.15.78:0/1123806461 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f61540d0070 con 0x7f615c1027e0 2024-03-14T09:52:25.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:25 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1799633648' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:52:25.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:25 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/4180633649' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:52:25.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:25 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3328925175' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-14T09:52:25.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.859+0000 7f6163200700 1 -- 172.21.15.78:0/1123806461 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f615c10ca50 con 0x7f615c1027e0 2024-03-14T09:52:25.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.860+0000 7f6158c00700 1 -- 172.21.15.78:0/1123806461 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+23925 (secure 0 0 0) 0x7f61540955c0 con 0x7f615c1027e0 2024-03-14T09:52:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:25.860 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":83,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:52:10.245498+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:52:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.863+0000 7f6163200700 1 -- 172.21.15.78:0/1123806461 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f61400a0a90 msgr2=0x7f61400a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.863+0000 7f6163200700 1 --2- 172.21.15.78:0/1123806461 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f61400a0a90 0x7f61400a2f50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f614c000c00 tx=0x7f614c005fb0 comp rx=0 tx=0).stop 2024-03-14T09:52:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.863+0000 7f6163200700 1 -- 172.21.15.78:0/1123806461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f615c1027e0 msgr2=0x7f615c1a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.863+0000 7f6163200700 1 --2- 172.21.15.78:0/1123806461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f615c1027e0 0x7f615c1a0780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f615400ef30 tx=0x7f615400c3a0 comp rx=0 tx=0).stop 2024-03-14T09:52:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.863+0000 7f6163200700 1 -- 172.21.15.78:0/1123806461 shutdown_connections 2024-03-14T09:52:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.863+0000 7f6163200700 1 --2- 172.21.15.78:0/1123806461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f615c1027e0 0x7f615c1a0780 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.863+0000 7f6163200700 1 --2- 172.21.15.78:0/1123806461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f615c100c50 0x7f615c199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.863+0000 7f6163200700 1 --2- 172.21.15.78:0/1123806461 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f61400a0a90 0x7f61400a2f50 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.863+0000 7f6163200700 1 --2- 172.21.15.78:0/1123806461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f615c101e40 0x7f615c199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.863+0000 7f6163200700 1 -- 172.21.15.78:0/1123806461 >> 172.21.15.78:0/1123806461 conn(0x7f615c0fc1c0 msgr2=0x7f615c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.863+0000 7f6163200700 1 -- 172.21.15.78:0/1123806461 shutdown_connections 2024-03-14T09:52:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:25.863+0000 7f6163200700 1 -- 172.21.15.78:0/1123806461 wait complete. 2024-03-14T09:52:25.882 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.075+0000 7f9a8b800700 1 -- 172.21.15.78:0/1160736885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a84073810 msgr2=0x7f9a84073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.075+0000 7f9a8b800700 1 --2- 172.21.15.78:0/1160736885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a84073810 0x7f9a84073c90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9a70009990 tx=0x7f9a70009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.075+0000 7f9a8b800700 1 -- 172.21.15.78:0/1160736885 shutdown_connections 2024-03-14T09:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.075+0000 7f9a8b800700 1 --2- 172.21.15.78:0/1160736885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a84108b10 0x7f9a8410afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.075+0000 7f9a8b800700 1 --2- 172.21.15.78:0/1160736885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a84074df0 0x7f9a84073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.075+0000 7f9a8b800700 1 --2- 172.21.15.78:0/1160736885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a84073810 0x7f9a84073c90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.075+0000 7f9a8b800700 1 -- 172.21.15.78:0/1160736885 >> 172.21.15.78:0/1160736885 conn(0x7f9a840fc1c0 msgr2=0x7f9a840fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:26.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.075+0000 7f9a8b800700 1 -- 172.21.15.78:0/1160736885 shutdown_connections 2024-03-14T09:52:26.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.075+0000 7f9a8b800700 1 -- 172.21.15.78:0/1160736885 wait complete. 2024-03-14T09:52:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.076+0000 7f9a8b800700 1 Processor -- start 2024-03-14T09:52:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.076+0000 7f9a8b800700 1 -- start start 2024-03-14T09:52:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a8b800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a84073810 0x7f9a84199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a8b800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a84074df0 0x7f9a841997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a8b800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a84108b10 0x7f9a841a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a8b800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a841a0d50 con 0x7f9a84074df0 2024-03-14T09:52:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a8b800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a841a0e90 con 0x7f9a84108b10 2024-03-14T09:52:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a8b800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a841a1020 con 0x7f9a84073810 2024-03-14T09:52:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a83e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a84074df0 0x7f9a841997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a83e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a84074df0 0x7f9a841997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41396/0 (socket says 172.21.15.78:41396) 2024-03-14T09:52:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a83e00700 1 -- 172.21.15.78:0/2997049057 learned_addr learned my addr 172.21.15.78:0/2997049057 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a89800700 1 --2- 172.21.15.78:0/2997049057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a84108b10 0x7f9a841a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a88e00700 1 --2- 172.21.15.78:0/2997049057 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a84073810 0x7f9a84199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a83e00700 1 -- 172.21.15.78:0/2997049057 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a84073810 msgr2=0x7f9a84199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a83e00700 1 --2- 172.21.15.78:0/2997049057 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a84073810 0x7f9a84199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a83e00700 1 -- 172.21.15.78:0/2997049057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a84108b10 msgr2=0x7f9a841a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a83e00700 1 --2- 172.21.15.78:0/2997049057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a84108b10 0x7f9a841a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a83e00700 1 -- 172.21.15.78:0/2997049057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a700095f0 con 0x7f9a84074df0 2024-03-14T09:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.077+0000 7f9a89800700 1 --2- 172.21.15.78:0/2997049057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a84108b10 0x7f9a841a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.078+0000 7f9a83e00700 1 --2- 172.21.15.78:0/2997049057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a84074df0 0x7f9a841997b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000b680 tx=0x7f9a70004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.078+0000 7f9a81600700 1 -- 172.21.15.78:0/2997049057 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7001d070 con 0x7f9a84074df0 2024-03-14T09:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.078+0000 7f9a81600700 1 -- 172.21.15.78:0/2997049057 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a70021410 con 0x7f9a84074df0 2024-03-14T09:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.078+0000 7f9a81600700 1 -- 172.21.15.78:0/2997049057 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a70017650 con 0x7f9a84074df0 2024-03-14T09:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.078+0000 7f9a8b800700 1 -- 172.21.15.78:0/2997049057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a841a12a0 con 0x7f9a84074df0 2024-03-14T09:52:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.078+0000 7f9a8b800700 1 -- 172.21.15.78:0/2997049057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a841a1670 con 0x7f9a84074df0 2024-03-14T09:52:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.079+0000 7f9a8b800700 1 -- 172.21.15.78:0/2997049057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a841078d0 con 0x7f9a84074df0 2024-03-14T09:52:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.081+0000 7f9a81600700 1 -- 172.21.15.78:0/2997049057 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f9a70021580 con 0x7f9a84074df0 2024-03-14T09:52:26.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.082+0000 7f9a81600700 1 --2- 172.21.15.78:0/2997049057 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9a6c0a0a90 0x7f9a6c0a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:26.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.082+0000 7f9a81600700 1 -- 172.21.15.78:0/2997049057 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9a700cd610 con 0x7f9a84074df0 2024-03-14T09:52:26.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.083+0000 7f9a88e00700 1 --2- 172.21.15.78:0/2997049057 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9a6c0a0a90 0x7f9a6c0a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:26.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.083+0000 7f9a88e00700 1 --2- 172.21.15.78:0/2997049057 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9a6c0a0a90 0x7f9a6c0a2f50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009c60 tx=0x7f9a78009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:26.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.086+0000 7f9a81600700 1 -- 172.21.15.78:0/2997049057 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9a70096e20 con 0x7f9a84074df0 2024-03-14T09:52:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.315+0000 7f9a8b800700 1 -- 172.21.15.78:0/2997049057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a8410eab0 con 0x7f9a84074df0 2024-03-14T09:52:26.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.317+0000 7f9a81600700 1 -- 172.21.15.78:0/2997049057 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+23925 (secure 0 0 0) 0x7f9a7009a440 con 0x7f9a84074df0 2024-03-14T09:52:26.317 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:26.317 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":83,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:52:10.245498+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:52:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.319+0000 7f9a8b800700 1 -- 172.21.15.78:0/2997049057 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9a6c0a0a90 msgr2=0x7f9a6c0a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.319+0000 7f9a8b800700 1 --2- 172.21.15.78:0/2997049057 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9a6c0a0a90 0x7f9a6c0a2f50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009c60 tx=0x7f9a78009260 comp rx=0 tx=0).stop 2024-03-14T09:52:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.319+0000 7f9a8b800700 1 -- 172.21.15.78:0/2997049057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a84074df0 msgr2=0x7f9a841997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.319+0000 7f9a8b800700 1 --2- 172.21.15.78:0/2997049057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a84074df0 0x7f9a841997b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000b680 tx=0x7f9a70004730 comp rx=0 tx=0).stop 2024-03-14T09:52:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.319+0000 7f9a8b800700 1 -- 172.21.15.78:0/2997049057 shutdown_connections 2024-03-14T09:52:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.319+0000 7f9a8b800700 1 --2- 172.21.15.78:0/2997049057 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9a6c0a0a90 0x7f9a6c0a2f50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.319+0000 7f9a8b800700 1 --2- 172.21.15.78:0/2997049057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a84108b10 0x7f9a841a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.319+0000 7f9a8b800700 1 --2- 172.21.15.78:0/2997049057 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a84073810 0x7f9a84199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.319+0000 7f9a8b800700 1 --2- 172.21.15.78:0/2997049057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a84074df0 0x7f9a841997b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.319+0000 7f9a8b800700 1 -- 172.21.15.78:0/2997049057 >> 172.21.15.78:0/2997049057 conn(0x7f9a840fc1c0 msgr2=0x7f9a84104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.319+0000 7f9a8b800700 1 -- 172.21.15.78:0/2997049057 shutdown_connections 2024-03-14T09:52:26.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.320+0000 7f9a8b800700 1 -- 172.21.15.78:0/2997049057 wait complete. 2024-03-14T09:52:26.339 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-14T09:52:26.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:26 smithi132 ceph-mon[47959]: pgmap v245: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-14T09:52:26.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:26 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1123806461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:26.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.535+0000 7fb727a00700 1 -- 172.21.15.78:0/1541156972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb720101f30 msgr2=0x7fb720102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:26.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.535+0000 7fb727a00700 1 --2- 172.21.15.78:0/1541156972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb720101f30 0x7fb720102390 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb70c0099c0 tx=0x7fb70c009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:26.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.536+0000 7fb727a00700 1 -- 172.21.15.78:0/1541156972 shutdown_connections 2024-03-14T09:52:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.536+0000 7fb727a00700 1 --2- 172.21.15.78:0/1541156972 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7201028d0 0x7fb720109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.536+0000 7fb727a00700 1 --2- 172.21.15.78:0/1541156972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb720100d40 0x7fb720101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.536+0000 7fb727a00700 1 --2- 172.21.15.78:0/1541156972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb720101f30 0x7fb720102390 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.536+0000 7fb727a00700 1 -- 172.21.15.78:0/1541156972 >> 172.21.15.78:0/1541156972 conn(0x7fb7200fc2b0 msgr2=0x7fb7200fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.536+0000 7fb727a00700 1 -- 172.21.15.78:0/1541156972 shutdown_connections 2024-03-14T09:52:26.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.536+0000 7fb727a00700 1 -- 172.21.15.78:0/1541156972 wait complete. 2024-03-14T09:52:26.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.537+0000 7fb727a00700 1 Processor -- start 2024-03-14T09:52:26.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.537+0000 7fb727a00700 1 -- start start 2024-03-14T09:52:26.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.537+0000 7fb727a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb720100d40 0x7fb720074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:26.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.537+0000 7fb727a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb720101f30 0x7fb7200731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:26.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.537+0000 7fb727a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7201028d0 0x7fb7200736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:26.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.537+0000 7fb727a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb720073e80 con 0x7fb720100d40 2024-03-14T09:52:26.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.537+0000 7fb727a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb720073ff0 con 0x7fb720101f30 2024-03-14T09:52:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.537+0000 7fb727a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7201a1170 con 0x7fb7201028d0 2024-03-14T09:52:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.537+0000 7fb725000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb720100d40 0x7fb720074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.537+0000 7fb725000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb720100d40 0x7fb720074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41416/0 (socket says 172.21.15.78:41416) 2024-03-14T09:52:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.537+0000 7fb725000700 1 -- 172.21.15.78:0/135874427 learned_addr learned my addr 172.21.15.78:0/135874427 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.537+0000 7fb71fe00700 1 --2- 172.21.15.78:0/135874427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb720101f30 0x7fb7200731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.537+0000 7fb725a00700 1 --2- 172.21.15.78:0/135874427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7201028d0 0x7fb7200736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.538+0000 7fb725000700 1 -- 172.21.15.78:0/135874427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7201028d0 msgr2=0x7fb7200736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.538+0000 7fb725000700 1 --2- 172.21.15.78:0/135874427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7201028d0 0x7fb7200736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.538+0000 7fb725000700 1 -- 172.21.15.78:0/135874427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb720101f30 msgr2=0x7fb7200731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.538+0000 7fb725000700 1 --2- 172.21.15.78:0/135874427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb720101f30 0x7fb7200731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.538+0000 7fb725000700 1 -- 172.21.15.78:0/135874427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70c0095f0 con 0x7fb720100d40 2024-03-14T09:52:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.538+0000 7fb725000700 1 --2- 172.21.15.78:0/135874427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb720100d40 0x7fb720074b10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb71400d980 tx=0x7fb71400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:26.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.538+0000 7fb71d600700 1 -- 172.21.15.78:0/135874427 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71400f890 con 0x7fb720100d40 2024-03-14T09:52:26.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.538+0000 7fb71d600700 1 -- 172.21.15.78:0/135874427 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb714010460 con 0x7fb720100d40 2024-03-14T09:52:26.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.538+0000 7fb727a00700 1 -- 172.21.15.78:0/135874427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7201a1400 con 0x7fb720100d40 2024-03-14T09:52:26.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.538+0000 7fb71d600700 1 -- 172.21.15.78:0/135874427 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7140195c0 con 0x7fb720100d40 2024-03-14T09:52:26.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.538+0000 7fb727a00700 1 -- 172.21.15.78:0/135874427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7201a1960 con 0x7fb720100d40 2024-03-14T09:52:26.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.540+0000 7fb727a00700 1 -- 172.21.15.78:0/135874427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7201081e0 con 0x7fb720100d40 2024-03-14T09:52:26.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.541+0000 7fb71d600700 1 -- 172.21.15.78:0/135874427 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fb7140105d0 con 0x7fb720100d40 2024-03-14T09:52:26.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.542+0000 7fb71d600700 1 --2- 172.21.15.78:0/135874427 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb7080a0b10 0x7fb7080a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:26.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.543+0000 7fb71d600700 1 -- 172.21.15.78:0/135874427 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb7140cbf30 con 0x7fb720100d40 2024-03-14T09:52:26.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.543+0000 7fb71fe00700 1 --2- 172.21.15.78:0/135874427 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb7080a0b10 0x7fb7080a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:26.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.543+0000 7fb71fe00700 1 --2- 172.21.15.78:0/135874427 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb7080a0b10 0x7fb7080a2fd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb70c006010 tx=0x7fb70c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:26.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.546+0000 7fb71d600700 1 -- 172.21.15.78:0/135874427 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb7140956b0 con 0x7fb720100d40 2024-03-14T09:52:26.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:26 smithi078 ceph-mon[38143]: pgmap v245: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-14T09:52:26.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:26 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1123806461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:26.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:26 smithi152 ceph-mon[48279]: pgmap v245: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-14T09:52:26.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:26 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1123806461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:26.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:26.798+0000 7fb727a00700 1 -- 172.21.15.78:0/135874427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb72010cc70 con 0x7fb720100d40 2024-03-14T09:52:27.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.276+0000 7fb71d600700 1 -- 172.21.15.78:0/135874427 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb714098cd0 con 0x7fb720100d40 2024-03-14T09:52:27.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.279+0000 7fb727a00700 1 -- 172.21.15.78:0/135874427 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb7080a0b10 msgr2=0x7fb7080a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:27.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.279+0000 7fb727a00700 1 --2- 172.21.15.78:0/135874427 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb7080a0b10 0x7fb7080a2fd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb70c006010 tx=0x7fb70c00b540 comp rx=0 tx=0).stop 2024-03-14T09:52:27.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.279+0000 7fb727a00700 1 -- 172.21.15.78:0/135874427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb720100d40 msgr2=0x7fb720074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:27.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.279+0000 7fb727a00700 1 --2- 172.21.15.78:0/135874427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb720100d40 0x7fb720074b10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb71400d980 tx=0x7fb71400dc90 comp rx=0 tx=0).stop 2024-03-14T09:52:27.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.279+0000 7fb727a00700 1 -- 172.21.15.78:0/135874427 shutdown_connections 2024-03-14T09:52:27.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.279+0000 7fb727a00700 1 --2- 172.21.15.78:0/135874427 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb7201028d0 0x7fb7200736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:27.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.279+0000 7fb727a00700 1 --2- 172.21.15.78:0/135874427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb720100d40 0x7fb720074b10 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:27.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.279+0000 7fb727a00700 1 --2- 172.21.15.78:0/135874427 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb7080a0b10 0x7fb7080a2fd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:27.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.279+0000 7fb727a00700 1 --2- 172.21.15.78:0/135874427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb720101f30 0x7fb7200731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:27.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.279+0000 7fb727a00700 1 -- 172.21.15.78:0/135874427 >> 172.21.15.78:0/135874427 conn(0x7fb7200fc2b0 msgr2=0x7fb720105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:27.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.279+0000 7fb727a00700 1 -- 172.21.15.78:0/135874427 shutdown_connections 2024-03-14T09:52:27.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.279+0000 7fb727a00700 1 -- 172.21.15.78:0/135874427 wait complete. 2024-03-14T09:52:27.299 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-14T09:52:27.299 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-14T09:52:27.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.496+0000 7f7259e00700 1 -- 172.21.15.78:0/1228898281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72541027f0 msgr2=0x7f72541090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:27.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.496+0000 7f7259e00700 1 --2- 172.21.15.78:0/1228898281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72541027f0 0x7f72541090a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f724c00b0d0 tx=0x7f724c00b3e0 comp rx=0 tx=0).stop 2024-03-14T09:52:27.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.496+0000 7f7259e00700 1 -- 172.21.15.78:0/1228898281 shutdown_connections 2024-03-14T09:52:27.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.496+0000 7f7259e00700 1 --2- 172.21.15.78:0/1228898281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72541027f0 0x7f72541090a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.496+0000 7f7259e00700 1 --2- 172.21.15.78:0/1228898281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7254100c60 0x7f7254101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.496+0000 7f7259e00700 1 --2- 172.21.15.78:0/1228898281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7254101e50 0x7f72541022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.496+0000 7f7259e00700 1 -- 172.21.15.78:0/1228898281 >> 172.21.15.78:0/1228898281 conn(0x7f72540fc1f0 msgr2=0x7f72540fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.496+0000 7f7259e00700 1 -- 172.21.15.78:0/1228898281 shutdown_connections 2024-03-14T09:52:27.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.497+0000 7f7259e00700 1 -- 172.21.15.78:0/1228898281 wait complete. 2024-03-14T09:52:27.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.497+0000 7f7259e00700 1 Processor -- start 2024-03-14T09:52:27.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.498+0000 7f7259e00700 1 -- start start 2024-03-14T09:52:27.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.498+0000 7f7259e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7254100c60 0x7f72541993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:27.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.498+0000 7f7259e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7254101e50 0x7f72541998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:27.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.498+0000 7f7259e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72541027f0 0x7f72541a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:27.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.498+0000 7f7259e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72541a0e40 con 0x7f72541027f0 2024-03-14T09:52:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.498+0000 7f7259e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72541a0f80 con 0x7f7254100c60 2024-03-14T09:52:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.498+0000 7f7259e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72541a1110 con 0x7f7254101e50 2024-03-14T09:52:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.498+0000 7f7253400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72541027f0 0x7f72541a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.498+0000 7f7253400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72541027f0 0x7f72541a0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41438/0 (socket says 172.21.15.78:41438) 2024-03-14T09:52:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.498+0000 7f7253400700 1 -- 172.21.15.78:0/2728786491 learned_addr learned my addr 172.21.15.78:0/2728786491 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.498+0000 7f7252000700 1 --2- 172.21.15.78:0/2728786491 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7254101e50 0x7f72541998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.498+0000 7f7252a00700 1 --2- 172.21.15.78:0/2728786491 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7254100c60 0x7f72541993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.499+0000 7f7252000700 1 -- 172.21.15.78:0/2728786491 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7254100c60 msgr2=0x7f72541993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.499+0000 7f7252000700 1 --2- 172.21.15.78:0/2728786491 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7254100c60 0x7f72541993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.499+0000 7f7252000700 1 -- 172.21.15.78:0/2728786491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72541027f0 msgr2=0x7f72541a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.499+0000 7f7252000700 1 --2- 172.21.15.78:0/2728786491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72541027f0 0x7f72541a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.499+0000 7f7252000700 1 -- 172.21.15.78:0/2728786491 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724c009d00 con 0x7f7254101e50 2024-03-14T09:52:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.499+0000 7f7253400700 1 --2- 172.21.15.78:0/2728786491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72541027f0 0x7f72541a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:27.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.499+0000 7f7252000700 1 --2- 172.21.15.78:0/2728786491 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7254101e50 0x7f72541998f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f724000e9f0 tx=0x7f724000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:27.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.499+0000 7f7258a00700 1 -- 172.21.15.78:0/2728786491 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724000c9c0 con 0x7f7254101e50 2024-03-14T09:52:27.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.499+0000 7f7258a00700 1 -- 172.21.15.78:0/2728786491 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7240010430 con 0x7f7254101e50 2024-03-14T09:52:27.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.499+0000 7f7259e00700 1 -- 172.21.15.78:0/2728786491 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72541a13f0 con 0x7f7254101e50 2024-03-14T09:52:27.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.499+0000 7f7258a00700 1 -- 172.21.15.78:0/2728786491 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240018680 con 0x7f7254101e50 2024-03-14T09:52:27.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.499+0000 7f7259e00700 1 -- 172.21.15.78:0/2728786491 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72541a1900 con 0x7f7254101e50 2024-03-14T09:52:27.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.500+0000 7f7259e00700 1 -- 172.21.15.78:0/2728786491 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72540679e0 con 0x7f7254101e50 2024-03-14T09:52:27.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.502+0000 7f7258a00700 1 -- 172.21.15.78:0/2728786491 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f7240010860 con 0x7f7254101e50 2024-03-14T09:52:27.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.503+0000 7f7258a00700 1 --2- 172.21.15.78:0/2728786491 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f72380a0a80 0x7f72380a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:27.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.503+0000 7f7258a00700 1 -- 172.21.15.78:0/2728786491 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7240014070 con 0x7f7254101e50 2024-03-14T09:52:27.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.504+0000 7f7252a00700 1 --2- 172.21.15.78:0/2728786491 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f72380a0a80 0x7f72380a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:27.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.504+0000 7f7252a00700 1 --2- 172.21.15.78:0/2728786491 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f72380a0a80 0x7f72380a2f40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7254101cb0 tx=0x7f7244009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:27.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.508+0000 7f7258a00700 1 -- 172.21.15.78:0/2728786491 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7240052960 con 0x7f7254101e50 2024-03-14T09:52:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:27 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2997049057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:27 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/135874427' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-14T09:52:27.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:27 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2997049057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:27.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:27 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/135874427' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-14T09:52:27.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:27 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2997049057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:27.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:27 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/135874427' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-14T09:52:27.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:27.768+0000 7f7259e00700 1 -- 172.21.15.78:0/2728786491 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f725410caf0 con 0x7f7254101e50 2024-03-14T09:52:28.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:28 smithi078 ceph-mon[38143]: pgmap v246: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-14T09:52:28.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:28 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/135874427' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-14T09:52:28.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:28 smithi078 ceph-mon[38143]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-14T09:52:28.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:28 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2728786491' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-14T09:52:28.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:28 smithi078 ceph-mon[38143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-14T09:52:28.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.701+0000 7f7258a00700 1 -- 172.21.15.78:0/2728786491 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7f7240095b10 con 0x7f7254101e50 2024-03-14T09:52:28.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.704+0000 7f7259e00700 1 -- 172.21.15.78:0/2728786491 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f72380a0a80 msgr2=0x7f72380a2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:28.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.704+0000 7f7259e00700 1 --2- 172.21.15.78:0/2728786491 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f72380a0a80 0x7f72380a2f40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7254101cb0 tx=0x7f7244009260 comp rx=0 tx=0).stop 2024-03-14T09:52:28.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.704+0000 7f7259e00700 1 -- 172.21.15.78:0/2728786491 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7254101e50 msgr2=0x7f72541998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:28.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.704+0000 7f7259e00700 1 --2- 172.21.15.78:0/2728786491 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7254101e50 0x7f72541998f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f724000e9f0 tx=0x7f724000ed00 comp rx=0 tx=0).stop 2024-03-14T09:52:28.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.704+0000 7f7259e00700 1 -- 172.21.15.78:0/2728786491 shutdown_connections 2024-03-14T09:52:28.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.704+0000 7f7259e00700 1 --2- 172.21.15.78:0/2728786491 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f72380a0a80 0x7f72380a2f40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:28.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.704+0000 7f7259e00700 1 --2- 172.21.15.78:0/2728786491 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7254100c60 0x7f72541993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:28.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.704+0000 7f7259e00700 1 --2- 172.21.15.78:0/2728786491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72541027f0 0x7f72541a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:28.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.704+0000 7f7259e00700 1 --2- 172.21.15.78:0/2728786491 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7254101e50 0x7f72541998f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:28.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.704+0000 7f7259e00700 1 -- 172.21.15.78:0/2728786491 >> 172.21.15.78:0/2728786491 conn(0x7f72540fc1f0 msgr2=0x7f7254072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:28.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.705+0000 7f7259e00700 1 -- 172.21.15.78:0/2728786491 shutdown_connections 2024-03-14T09:52:28.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.705+0000 7f7259e00700 1 -- 172.21.15.78:0/2728786491 wait complete. 2024-03-14T09:52:28.731 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-14T09:52:28.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:28 smithi152 ceph-mon[48279]: pgmap v246: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-14T09:52:28.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:28 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/135874427' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-14T09:52:28.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:28 smithi152 ceph-mon[48279]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-14T09:52:28.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:28 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2728786491' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-14T09:52:28.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:28 smithi152 ceph-mon[48279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-14T09:52:28.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:28 smithi132 ceph-mon[47959]: pgmap v246: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-14T09:52:28.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:28 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/135874427' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-14T09:52:28.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:28 smithi132 ceph-mon[47959]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-14T09:52:28.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:28 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2728786491' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-14T09:52:28.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:28 smithi132 ceph-mon[47959]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-14T09:52:28.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.932+0000 7fd8b9400700 1 -- 172.21.15.78:0/3110601888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b4101f30 msgr2=0x7fd8b4102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:28.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.932+0000 7fd8b9400700 1 --2- 172.21.15.78:0/3110601888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b4101f30 0x7fd8b4102390 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00099c0 tx=0x7fd8a0009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:28.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.932+0000 7fd8b9400700 1 -- 172.21.15.78:0/3110601888 shutdown_connections 2024-03-14T09:52:28.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.932+0000 7fd8b9400700 1 --2- 172.21.15.78:0/3110601888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8b4100d40 0x7fd8b4101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:28.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.932+0000 7fd8b9400700 1 --2- 172.21.15.78:0/3110601888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd8b41028d0 0x7fd8b4109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:28.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.932+0000 7fd8b9400700 1 --2- 172.21.15.78:0/3110601888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b4101f30 0x7fd8b4102390 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:28.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.932+0000 7fd8b9400700 1 -- 172.21.15.78:0/3110601888 >> 172.21.15.78:0/3110601888 conn(0x7fd8b40fc2b0 msgr2=0x7fd8b40fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:28.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.932+0000 7fd8b9400700 1 -- 172.21.15.78:0/3110601888 shutdown_connections 2024-03-14T09:52:28.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.932+0000 7fd8b9400700 1 -- 172.21.15.78:0/3110601888 wait complete. 2024-03-14T09:52:28.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.933+0000 7fd8b9400700 1 Processor -- start 2024-03-14T09:52:28.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.933+0000 7fd8b9400700 1 -- start start 2024-03-14T09:52:28.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.934+0000 7fd8b9400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b4100d40 0x7fd8b4074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:28.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.934+0000 7fd8b9400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8b4101f30 0x7fd8b40731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:28.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.934+0000 7fd8b9400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd8b41028d0 0x7fd8b40736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:28.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.934+0000 7fd8b9400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b4073e80 con 0x7fd8b4100d40 2024-03-14T09:52:28.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.934+0000 7fd8b9400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b4073ff0 con 0x7fd8b4101f30 2024-03-14T09:52:28.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.934+0000 7fd8b9400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b41a1120 con 0x7fd8b41028d0 2024-03-14T09:52:28.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.934+0000 7fd8b2800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b4100d40 0x7fd8b4074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:28.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.934+0000 7fd8b2800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b4100d40 0x7fd8b4074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41448/0 (socket says 172.21.15.78:41448) 2024-03-14T09:52:28.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.934+0000 7fd8b2800700 1 -- 172.21.15.78:0/2221705016 learned_addr learned my addr 172.21.15.78:0/2221705016 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:28.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.934+0000 7fd8b1e00700 1 --2- 172.21.15.78:0/2221705016 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8b4101f30 0x7fd8b40731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:28.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.934+0000 7fd8b3200700 1 --2- 172.21.15.78:0/2221705016 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd8b41028d0 0x7fd8b40736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:28.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.934+0000 7fd8b2800700 1 -- 172.21.15.78:0/2221705016 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd8b41028d0 msgr2=0x7fd8b40736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:28.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.934+0000 7fd8b2800700 1 --2- 172.21.15.78:0/2221705016 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd8b41028d0 0x7fd8b40736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:28.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.934+0000 7fd8b2800700 1 -- 172.21.15.78:0/2221705016 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8b4101f30 msgr2=0x7fd8b40731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:28.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.934+0000 7fd8b2800700 1 --2- 172.21.15.78:0/2221705016 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8b4101f30 0x7fd8b40731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:28.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.934+0000 7fd8b2800700 1 -- 172.21.15.78:0/2221705016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a00095f0 con 0x7fd8b4100d40 2024-03-14T09:52:28.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.935+0000 7fd8b2800700 1 --2- 172.21.15.78:0/2221705016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b4100d40 0x7fd8b4074b10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400e970 tx=0x7fd8a400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:28.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.935+0000 7fd8ab400700 1 -- 172.21.15.78:0/2221705016 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4009790 con 0x7fd8b4100d40 2024-03-14T09:52:28.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.935+0000 7fd8ab400700 1 -- 172.21.15.78:0/2221705016 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8a400c780 con 0x7fd8b4100d40 2024-03-14T09:52:28.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.935+0000 7fd8ab400700 1 -- 172.21.15.78:0/2221705016 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4003be0 con 0x7fd8b4100d40 2024-03-14T09:52:28.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.935+0000 7fd8b9400700 1 -- 172.21.15.78:0/2221705016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b41a13b0 con 0x7fd8b4100d40 2024-03-14T09:52:28.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.935+0000 7fd8b9400700 1 -- 172.21.15.78:0/2221705016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b41a18c0 con 0x7fd8b4100d40 2024-03-14T09:52:28.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.936+0000 7fd8b9400700 1 -- 172.21.15.78:0/2221705016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b41081c0 con 0x7fd8b4100d40 2024-03-14T09:52:28.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.938+0000 7fd8ab400700 1 -- 172.21.15.78:0/2221705016 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fd8a400cbf0 con 0x7fd8b4100d40 2024-03-14T09:52:28.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.939+0000 7fd8ab400700 1 --2- 172.21.15.78:0/2221705016 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd8980a0b10 0x7fd8980a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:28.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.939+0000 7fd8ab400700 1 -- 172.21.15.78:0/2221705016 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd8a4014070 con 0x7fd8b4100d40 2024-03-14T09:52:28.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.939+0000 7fd8b1e00700 1 --2- 172.21.15.78:0/2221705016 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd8980a0b10 0x7fd8980a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:28.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.940+0000 7fd8b1e00700 1 --2- 172.21.15.78:0/2221705016 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd8980a0b10 0x7fd8980a2fd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0009990 tx=0x7fd8a0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:28.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:28.943+0000 7fd8ab400700 1 -- 172.21.15.78:0/2221705016 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd8a4095b20 con 0x7fd8b4100d40 2024-03-14T09:52:29.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.195+0000 7fd8b9400700 1 -- 172.21.15.78:0/2221705016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fd8b4067a90 con 0x7fd8b4100d40 2024-03-14T09:52:29.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:29 smithi078 ceph-mon[38143]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-14T09:52:29.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:29 smithi078 ceph-mon[38143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-14T09:52:29.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:29 smithi078 ceph-mon[38143]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-14T09:52:29.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:29 smithi078 ceph-mon[38143]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-14T09:52:29.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:29 smithi078 ceph-mon[38143]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-03-14T09:52:29.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:29 smithi078 ceph-mon[38143]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-14T09:52:29.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:29 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2221705016' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-14T09:52:29.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.707+0000 7fd8ab400700 1 -- 172.21.15.78:0/2221705016 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7fd8a4099140 con 0x7fd8b4100d40 2024-03-14T09:52:29.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.709+0000 7fd8b9400700 1 -- 172.21.15.78:0/2221705016 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd8980a0b10 msgr2=0x7fd8980a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:29.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.709+0000 7fd8b9400700 1 --2- 172.21.15.78:0/2221705016 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd8980a0b10 0x7fd8980a2fd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0009990 tx=0x7fd8a0000bc0 comp rx=0 tx=0).stop 2024-03-14T09:52:29.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.709+0000 7fd8b9400700 1 -- 172.21.15.78:0/2221705016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b4100d40 msgr2=0x7fd8b4074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:29.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.709+0000 7fd8b9400700 1 --2- 172.21.15.78:0/2221705016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b4100d40 0x7fd8b4074b10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400e970 tx=0x7fd8a400ec80 comp rx=0 tx=0).stop 2024-03-14T09:52:29.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.710+0000 7fd8b9400700 1 -- 172.21.15.78:0/2221705016 shutdown_connections 2024-03-14T09:52:29.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.710+0000 7fd8b9400700 1 --2- 172.21.15.78:0/2221705016 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd8b41028d0 0x7fd8b40736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:29.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.710+0000 7fd8b9400700 1 --2- 172.21.15.78:0/2221705016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b4100d40 0x7fd8b4074b10 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:29.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.710+0000 7fd8b9400700 1 --2- 172.21.15.78:0/2221705016 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd8980a0b10 0x7fd8980a2fd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:29.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.710+0000 7fd8b9400700 1 --2- 172.21.15.78:0/2221705016 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8b4101f30 0x7fd8b40731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:29.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.710+0000 7fd8b9400700 1 -- 172.21.15.78:0/2221705016 >> 172.21.15.78:0/2221705016 conn(0x7fd8b40fc2b0 msgr2=0x7fd8b4105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:29.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.710+0000 7fd8b9400700 1 -- 172.21.15.78:0/2221705016 shutdown_connections 2024-03-14T09:52:29.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.710+0000 7fd8b9400700 1 -- 172.21.15.78:0/2221705016 wait complete. 2024-03-14T09:52:29.733 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-14T09:52:29.748 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:29 smithi152 ceph-mon[48279]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-14T09:52:29.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:29 smithi152 ceph-mon[48279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-14T09:52:29.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:29 smithi152 ceph-mon[48279]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-14T09:52:29.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:29 smithi152 ceph-mon[48279]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-14T09:52:29.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:29 smithi152 ceph-mon[48279]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-03-14T09:52:29.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:29 smithi152 ceph-mon[48279]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-14T09:52:29.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:29 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2221705016' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-14T09:52:29.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:29 smithi132 ceph-mon[47959]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-14T09:52:29.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:29 smithi132 ceph-mon[47959]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-14T09:52:29.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:29 smithi132 ceph-mon[47959]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-14T09:52:29.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:29 smithi132 ceph-mon[47959]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-14T09:52:29.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:29 smithi132 ceph-mon[47959]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-03-14T09:52:29.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:29 smithi132 ceph-mon[47959]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-14T09:52:29.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:29 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2221705016' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-14T09:52:29.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.933+0000 7fab9c400700 1 -- 172.21.15.78:0/2680581015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab94101f40 msgr2=0x7fab941023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:29.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.933+0000 7fab9c400700 1 --2- 172.21.15.78:0/2680581015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab94101f40 0x7fab941023a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fab800099c0 tx=0x7fab80009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:29.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.933+0000 7fab9c400700 1 -- 172.21.15.78:0/2680581015 shutdown_connections 2024-03-14T09:52:29.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.933+0000 7fab9c400700 1 --2- 172.21.15.78:0/2680581015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab941028e0 0x7fab94109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:29.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.933+0000 7fab9c400700 1 --2- 172.21.15.78:0/2680581015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab94100d50 0x7fab94101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:29.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.933+0000 7fab9c400700 1 --2- 172.21.15.78:0/2680581015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab94101f40 0x7fab941023a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:29.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.933+0000 7fab9c400700 1 -- 172.21.15.78:0/2680581015 >> 172.21.15.78:0/2680581015 conn(0x7fab940fc2c0 msgr2=0x7fab940fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:29.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.933+0000 7fab9c400700 1 -- 172.21.15.78:0/2680581015 shutdown_connections 2024-03-14T09:52:29.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.933+0000 7fab9c400700 1 -- 172.21.15.78:0/2680581015 wait complete. 2024-03-14T09:52:29.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.934+0000 7fab9c400700 1 Processor -- start 2024-03-14T09:52:29.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.934+0000 7fab9c400700 1 -- start start 2024-03-14T09:52:29.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.935+0000 7fab9c400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab94100d50 0x7fab94199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:29.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.935+0000 7fab9c400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab94101f40 0x7fab94199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:29.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.935+0000 7fab9c400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab941028e0 0x7fab941a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:29.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.935+0000 7fab9c400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab941a0e40 con 0x7fab941028e0 2024-03-14T09:52:29.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.935+0000 7fab9c400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab941a0f80 con 0x7fab94101f40 2024-03-14T09:52:29.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.935+0000 7fab9c400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab941a1110 con 0x7fab94100d50 2024-03-14T09:52:29.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.935+0000 7fab99a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab94100d50 0x7fab94199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:29.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.935+0000 7fab99000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab94101f40 0x7fab94199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:29.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.935+0000 7fab99a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab94100d50 0x7fab94199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:40986/0 (socket says 172.21.15.78:40986) 2024-03-14T09:52:29.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.935+0000 7fab99a00700 1 -- 172.21.15.78:0/2667493017 learned_addr learned my addr 172.21.15.78:0/2667493017 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:29.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.935+0000 7fab9a400700 1 --2- 172.21.15.78:0/2667493017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab941028e0 0x7fab941a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:29.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.935+0000 7fab99000700 1 -- 172.21.15.78:0/2667493017 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab94100d50 msgr2=0x7fab94199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:29.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.935+0000 7fab99000700 1 --2- 172.21.15.78:0/2667493017 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab94100d50 0x7fab94199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:29.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.935+0000 7fab99000700 1 -- 172.21.15.78:0/2667493017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab941028e0 msgr2=0x7fab941a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:29.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.935+0000 7fab99000700 1 --2- 172.21.15.78:0/2667493017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab941028e0 0x7fab941a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:29.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.935+0000 7fab99000700 1 -- 172.21.15.78:0/2667493017 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab800095f0 con 0x7fab94101f40 2024-03-14T09:52:29.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.936+0000 7fab99000700 1 --2- 172.21.15.78:0/2667493017 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab94101f40 0x7fab94199850 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fab800098e0 tx=0x7fab8000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:29.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.936+0000 7fab92000700 1 -- 172.21.15.78:0/2667493017 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8001d070 con 0x7fab94101f40 2024-03-14T09:52:29.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.936+0000 7fab92000700 1 -- 172.21.15.78:0/2667493017 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab80005020 con 0x7fab94101f40 2024-03-14T09:52:29.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.936+0000 7fab92000700 1 -- 172.21.15.78:0/2667493017 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab800049e0 con 0x7fab94101f40 2024-03-14T09:52:29.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.936+0000 7fab9c400700 1 -- 172.21.15.78:0/2667493017 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab941a1390 con 0x7fab94101f40 2024-03-14T09:52:29.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.936+0000 7fab9c400700 1 -- 172.21.15.78:0/2667493017 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab941a1750 con 0x7fab94101f40 2024-03-14T09:52:29.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.937+0000 7fab9c400700 1 -- 172.21.15.78:0/2667493017 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab94067ac0 con 0x7fab94101f40 2024-03-14T09:52:29.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.939+0000 7fab92000700 1 -- 172.21.15.78:0/2667493017 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fab80005190 con 0x7fab94101f40 2024-03-14T09:52:29.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.940+0000 7fab92000700 1 --2- 172.21.15.78:0/2667493017 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fab7c0a0b10 0x7fab7c0a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:29.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.940+0000 7fab92000700 1 -- 172.21.15.78:0/2667493017 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fab800ccd70 con 0x7fab94101f40 2024-03-14T09:52:29.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.941+0000 7fab99a00700 1 --2- 172.21.15.78:0/2667493017 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fab7c0a0b10 0x7fab7c0a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:29.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.941+0000 7fab99a00700 1 --2- 172.21.15.78:0/2667493017 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fab7c0a0b10 0x7fab7c0a2fd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fab88005950 tx=0x7fab88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:29.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:29.943+0000 7fab92000700 1 -- 172.21.15.78:0/2667493017 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fab80096580 con 0x7fab94101f40 2024-03-14T09:52:30.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.200+0000 7fab9c400700 1 -- 172.21.15.78:0/2667493017 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fab9410c9d0 con 0x7fab94101f40 2024-03-14T09:52:30.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:30 smithi078 ceph-mon[38143]: pgmap v247: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-14T09:52:30.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:30 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] up:active 2024-03-14T09:52:30.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:30 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2221705016' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-14T09:52:30.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:30 smithi078 ceph-mon[38143]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-14T09:52:30.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:30 smithi078 ceph-mon[38143]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-14T09:52:30.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:30 smithi078 ceph-mon[38143]: Cluster is now healthy 2024-03-14T09:52:30.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:30 smithi078 ceph-mon[38143]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-03-14T09:52:30.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:30 smithi078 ceph-mon[38143]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:creating} 9 up:standby 2024-03-14T09:52:30.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:30 smithi078 ceph-mon[38143]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-14T09:52:30.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:30 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2667493017' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-14T09:52:30.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:30 smithi078 ceph-mon[38143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-14T09:52:30.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.716+0000 7fab92000700 1 -- 172.21.15.78:0/2667493017 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7fab80099ba0 con 0x7fab94101f40 2024-03-14T09:52:30.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.719+0000 7fab9c400700 1 -- 172.21.15.78:0/2667493017 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fab7c0a0b10 msgr2=0x7fab7c0a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:30.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.719+0000 7fab9c400700 1 --2- 172.21.15.78:0/2667493017 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fab7c0a0b10 0x7fab7c0a2fd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fab88005950 tx=0x7fab88009260 comp rx=0 tx=0).stop 2024-03-14T09:52:30.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.719+0000 7fab9c400700 1 -- 172.21.15.78:0/2667493017 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab94101f40 msgr2=0x7fab94199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:30.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.719+0000 7fab9c400700 1 --2- 172.21.15.78:0/2667493017 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab94101f40 0x7fab94199850 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fab800098e0 tx=0x7fab8000f690 comp rx=0 tx=0).stop 2024-03-14T09:52:30.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.719+0000 7fab9c400700 1 -- 172.21.15.78:0/2667493017 shutdown_connections 2024-03-14T09:52:30.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.719+0000 7fab9c400700 1 --2- 172.21.15.78:0/2667493017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab941028e0 0x7fab941a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:30.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.719+0000 7fab9c400700 1 --2- 172.21.15.78:0/2667493017 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab94100d50 0x7fab94199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:30.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.719+0000 7fab9c400700 1 --2- 172.21.15.78:0/2667493017 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fab7c0a0b10 0x7fab7c0a2fd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:30.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.719+0000 7fab9c400700 1 --2- 172.21.15.78:0/2667493017 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab94101f40 0x7fab94199850 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:30.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.719+0000 7fab9c400700 1 -- 172.21.15.78:0/2667493017 >> 172.21.15.78:0/2667493017 conn(0x7fab940fc2c0 msgr2=0x7fab94107760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:30.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.719+0000 7fab9c400700 1 -- 172.21.15.78:0/2667493017 shutdown_connections 2024-03-14T09:52:30.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.719+0000 7fab9c400700 1 -- 172.21.15.78:0/2667493017 wait complete. 2024-03-14T09:52:30.739 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-03-14T09:52:30.739 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-14T09:52:30.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:30 smithi152 ceph-mon[48279]: pgmap v247: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-14T09:52:30.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:30 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] up:active 2024-03-14T09:52:30.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:30 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2221705016' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-14T09:52:30.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:30 smithi152 ceph-mon[48279]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-14T09:52:30.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:30 smithi152 ceph-mon[48279]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-14T09:52:30.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:30 smithi152 ceph-mon[48279]: Cluster is now healthy 2024-03-14T09:52:30.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:30 smithi152 ceph-mon[48279]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-03-14T09:52:30.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:30 smithi152 ceph-mon[48279]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:creating} 9 up:standby 2024-03-14T09:52:30.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:30 smithi152 ceph-mon[48279]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-14T09:52:30.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:30 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2667493017' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-14T09:52:30.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:30 smithi152 ceph-mon[48279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-14T09:52:30.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:30 smithi132 ceph-mon[47959]: pgmap v247: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-14T09:52:30.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:30 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] up:active 2024-03-14T09:52:30.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:30 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2221705016' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-14T09:52:30.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:30 smithi132 ceph-mon[47959]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-14T09:52:30.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:30 smithi132 ceph-mon[47959]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-14T09:52:30.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:30 smithi132 ceph-mon[47959]: Cluster is now healthy 2024-03-14T09:52:30.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:30 smithi132 ceph-mon[47959]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-03-14T09:52:30.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:30 smithi132 ceph-mon[47959]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:creating} 9 up:standby 2024-03-14T09:52:30.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:30 smithi132 ceph-mon[47959]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-14T09:52:30.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:30 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2667493017' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-14T09:52:30.771 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:30 smithi132 ceph-mon[47959]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-14T09:52:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.935+0000 7f42cd400700 1 -- 172.21.15.78:0/2663804289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42c8073810 msgr2=0x7f42c8073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.935+0000 7f42cd400700 1 --2- 172.21.15.78:0/2663804289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42c8073810 0x7f42c8073c90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f42b4009990 tx=0x7f42b4009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.935+0000 7f42cd400700 1 -- 172.21.15.78:0/2663804289 shutdown_connections 2024-03-14T09:52:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.935+0000 7f42cd400700 1 --2- 172.21.15.78:0/2663804289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42c8074df0 0x7f42c8073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.935+0000 7f42cd400700 1 --2- 172.21.15.78:0/2663804289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42c8108c00 0x7f42c810b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.935+0000 7f42cd400700 1 --2- 172.21.15.78:0/2663804289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42c8073810 0x7f42c8073c90 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.935+0000 7f42cd400700 1 -- 172.21.15.78:0/2663804289 >> 172.21.15.78:0/2663804289 conn(0x7f42c80fc2b0 msgr2=0x7f42c80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.936+0000 7f42cd400700 1 -- 172.21.15.78:0/2663804289 shutdown_connections 2024-03-14T09:52:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.936+0000 7f42cd400700 1 -- 172.21.15.78:0/2663804289 wait complete. 2024-03-14T09:52:30.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.937+0000 7f42cd400700 1 Processor -- start 2024-03-14T09:52:30.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.937+0000 7f42cd400700 1 -- start start 2024-03-14T09:52:30.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.937+0000 7f42cd400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42c8073810 0x7f42c8199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:30.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.937+0000 7f42cd400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42c8074df0 0x7f42c81998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:30.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.937+0000 7f42cd400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42c8108c00 0x7f42c81a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:30.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.937+0000 7f42cd400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c81a0e70 con 0x7f42c8074df0 2024-03-14T09:52:30.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.937+0000 7f42cd400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c81a0fb0 con 0x7f42c8108c00 2024-03-14T09:52:30.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.937+0000 7f42cd400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c81a1140 con 0x7f42c8073810 2024-03-14T09:52:30.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.937+0000 7f42c5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42c8074df0 0x7f42c81998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:30.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.937+0000 7f42c6800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42c8073810 0x7f42c8199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:30.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.937+0000 7f42c5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42c8074df0 0x7f42c81998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41462/0 (socket says 172.21.15.78:41462) 2024-03-14T09:52:30.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.937+0000 7f42c5e00700 1 -- 172.21.15.78:0/1594931525 learned_addr learned my addr 172.21.15.78:0/1594931525 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:30.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.937+0000 7f42c7200700 1 --2- 172.21.15.78:0/1594931525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42c8108c00 0x7f42c81a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:30.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.938+0000 7f42c5e00700 1 -- 172.21.15.78:0/1594931525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42c8073810 msgr2=0x7f42c8199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:30.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.938+0000 7f42c5e00700 1 --2- 172.21.15.78:0/1594931525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42c8073810 0x7f42c8199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:30.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.938+0000 7f42c5e00700 1 -- 172.21.15.78:0/1594931525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42c8108c00 msgr2=0x7f42c81a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:30.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.938+0000 7f42c5e00700 1 --2- 172.21.15.78:0/1594931525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42c8108c00 0x7f42c81a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:30.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.938+0000 7f42c5e00700 1 -- 172.21.15.78:0/1594931525 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b40095f0 con 0x7f42c8074df0 2024-03-14T09:52:30.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.938+0000 7f42c5e00700 1 --2- 172.21.15.78:0/1594931525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42c8074df0 0x7f42c81998d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f42b400b680 tx=0x7f42b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:30.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.938+0000 7f42bf400700 1 -- 172.21.15.78:0/1594931525 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b401d070 con 0x7f42c8074df0 2024-03-14T09:52:30.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.938+0000 7f42cd400700 1 -- 172.21.15.78:0/1594931525 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c81a13c0 con 0x7f42c8074df0 2024-03-14T09:52:30.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.938+0000 7f42bf400700 1 -- 172.21.15.78:0/1594931525 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f42b4021410 con 0x7f42c8074df0 2024-03-14T09:52:30.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.938+0000 7f42bf400700 1 -- 172.21.15.78:0/1594931525 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b4017650 con 0x7f42c8074df0 2024-03-14T09:52:30.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.938+0000 7f42cd400700 1 -- 172.21.15.78:0/1594931525 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c81a17f0 con 0x7f42c8074df0 2024-03-14T09:52:30.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.939+0000 7f42cd400700 1 -- 172.21.15.78:0/1594931525 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c81079c0 con 0x7f42c8074df0 2024-03-14T09:52:30.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.941+0000 7f42bf400700 1 -- 172.21.15.78:0/1594931525 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f42b4021580 con 0x7f42c8074df0 2024-03-14T09:52:30.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.942+0000 7f42bf400700 1 --2- 172.21.15.78:0/1594931525 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f42ac0a0a90 0x7f42ac0a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:30.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.942+0000 7f42bf400700 1 -- 172.21.15.78:0/1594931525 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f42b40cdcb0 con 0x7f42c8074df0 2024-03-14T09:52:30.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.943+0000 7f42c6800700 1 --2- 172.21.15.78:0/1594931525 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f42ac0a0a90 0x7f42ac0a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:30.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.943+0000 7f42c6800700 1 --2- 172.21.15.78:0/1594931525 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f42ac0a0a90 0x7f42ac0a2f50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f42b8009c60 tx=0x7f42b8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:30.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:30.945+0000 7f42bf400700 1 -- 172.21.15.78:0/1594931525 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f42b40974c0 con 0x7f42c8074df0 2024-03-14T09:52:31.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.186+0000 7f42cd400700 1 -- 172.21.15.78:0/1594931525 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f42c80611f0 con 0x7f42ac0a0a90 2024-03-14T09:52:31.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.210+0000 7f42bf400700 1 -- 172.21.15.78:0/1594931525 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f42c80611f0 con 0x7f42ac0a0a90 2024-03-14T09:52:31.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.213+0000 7f42cd400700 1 -- 172.21.15.78:0/1594931525 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f42ac0a0a90 msgr2=0x7f42ac0a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:31.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.213+0000 7f42cd400700 1 --2- 172.21.15.78:0/1594931525 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f42ac0a0a90 0x7f42ac0a2f50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f42b8009c60 tx=0x7f42b8009260 comp rx=0 tx=0).stop 2024-03-14T09:52:31.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.213+0000 7f42cd400700 1 -- 172.21.15.78:0/1594931525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42c8074df0 msgr2=0x7f42c81998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:31.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.213+0000 7f42cd400700 1 --2- 172.21.15.78:0/1594931525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42c8074df0 0x7f42c81998d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f42b400b680 tx=0x7f42b4004730 comp rx=0 tx=0).stop 2024-03-14T09:52:31.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.213+0000 7f42cd400700 1 -- 172.21.15.78:0/1594931525 shutdown_connections 2024-03-14T09:52:31.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.213+0000 7f42cd400700 1 --2- 172.21.15.78:0/1594931525 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f42ac0a0a90 0x7f42ac0a2f50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.213+0000 7f42cd400700 1 --2- 172.21.15.78:0/1594931525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f42c8108c00 0x7f42c81a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.213+0000 7f42cd400700 1 --2- 172.21.15.78:0/1594931525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f42c8073810 0x7f42c8199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.213+0000 7f42cd400700 1 --2- 172.21.15.78:0/1594931525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42c8074df0 0x7f42c81998d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.213+0000 7f42cd400700 1 -- 172.21.15.78:0/1594931525 >> 172.21.15.78:0/1594931525 conn(0x7f42c80fc2b0 msgr2=0x7f42c8104b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:31.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.213+0000 7f42cd400700 1 -- 172.21.15.78:0/1594931525 shutdown_connections 2024-03-14T09:52:31.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.213+0000 7f42cd400700 1 -- 172.21.15.78:0/1594931525 wait complete. 2024-03-14T09:52:31.231 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-14T09:52:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.430+0000 7f2b18800700 1 -- 172.21.15.78:0/2242474328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b10100d40 msgr2=0x7f2b10101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.430+0000 7f2b18800700 1 --2- 172.21.15.78:0/2242474328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b10100d40 0x7f2b10101160 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2b040099c0 tx=0x7f2b04009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.430+0000 7f2b18800700 1 -- 172.21.15.78:0/2242474328 shutdown_connections 2024-03-14T09:52:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.430+0000 7f2b18800700 1 --2- 172.21.15.78:0/2242474328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2b101028d0 0x7f2b10109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.430+0000 7f2b18800700 1 --2- 172.21.15.78:0/2242474328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b10100d40 0x7f2b10101160 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.430+0000 7f2b18800700 1 --2- 172.21.15.78:0/2242474328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b10101f30 0x7f2b10102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.430+0000 7f2b18800700 1 -- 172.21.15.78:0/2242474328 >> 172.21.15.78:0/2242474328 conn(0x7f2b100fc2b0 msgr2=0x7f2b100fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.430+0000 7f2b18800700 1 -- 172.21.15.78:0/2242474328 shutdown_connections 2024-03-14T09:52:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.430+0000 7f2b18800700 1 -- 172.21.15.78:0/2242474328 wait complete. 2024-03-14T09:52:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.431+0000 7f2b18800700 1 Processor -- start 2024-03-14T09:52:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.431+0000 7f2b18800700 1 -- start start 2024-03-14T09:52:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b18800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2b10100d40 0x7f2b10074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b18800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b10101f30 0x7f2b100731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b18800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b101028d0 0x7f2b100736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b18800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b10073e80 con 0x7f2b101028d0 2024-03-14T09:52:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b18800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b10073ff0 con 0x7f2b10100d40 2024-03-14T09:52:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b18800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b101a1130 con 0x7f2b10101f30 2024-03-14T09:52:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b16800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b101028d0 0x7f2b100736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b15400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b10101f30 0x7f2b100731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b16800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b101028d0 0x7f2b100736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41494/0 (socket says 172.21.15.78:41494) 2024-03-14T09:52:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b16800700 1 -- 172.21.15.78:0/3660948685 learned_addr learned my addr 172.21.15.78:0/3660948685 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b15e00700 1 --2- 172.21.15.78:0/3660948685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2b10100d40 0x7f2b10074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b15400700 1 -- 172.21.15.78:0/3660948685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2b10100d40 msgr2=0x7f2b10074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b15400700 1 --2- 172.21.15.78:0/3660948685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2b10100d40 0x7f2b10074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b15400700 1 -- 172.21.15.78:0/3660948685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b101028d0 msgr2=0x7f2b100736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b15400700 1 --2- 172.21.15.78:0/3660948685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b101028d0 0x7f2b100736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b15400700 1 -- 172.21.15.78:0/3660948685 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b040095f0 con 0x7f2b10101f30 2024-03-14T09:52:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.432+0000 7f2b16800700 1 --2- 172.21.15.78:0/3660948685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b101028d0 0x7f2b100736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.433+0000 7f2b15400700 1 --2- 172.21.15.78:0/3660948685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b10101f30 0x7f2b100731a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2b0000e9f0 tx=0x7f2b0000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.433+0000 7f2b0aa00700 1 -- 172.21.15.78:0/3660948685 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0000c9c0 con 0x7f2b10101f30 2024-03-14T09:52:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.433+0000 7f2b0aa00700 1 -- 172.21.15.78:0/3660948685 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b00010430 con 0x7f2b10101f30 2024-03-14T09:52:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.433+0000 7f2b18800700 1 -- 172.21.15.78:0/3660948685 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b101a1360 con 0x7f2b10101f30 2024-03-14T09:52:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.433+0000 7f2b0aa00700 1 -- 172.21.15.78:0/3660948685 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b00018680 con 0x7f2b10101f30 2024-03-14T09:52:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.433+0000 7f2b18800700 1 -- 172.21.15.78:0/3660948685 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b101a1810 con 0x7f2b10101f30 2024-03-14T09:52:31.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.434+0000 7f2b18800700 1 -- 172.21.15.78:0/3660948685 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b101081e0 con 0x7f2b10101f30 2024-03-14T09:52:31.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.437+0000 7f2b0aa00700 1 -- 172.21.15.78:0/3660948685 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f2b00010860 con 0x7f2b10101f30 2024-03-14T09:52:31.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.438+0000 7f2b0aa00700 1 --2- 172.21.15.78:0/3660948685 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2af80a0a90 0x7f2af80a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:31.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.438+0000 7f2b0aa00700 1 -- 172.21.15.78:0/3660948685 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2b00014070 con 0x7f2b10101f30 2024-03-14T09:52:31.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.439+0000 7f2b15e00700 1 --2- 172.21.15.78:0/3660948685 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2af80a0a90 0x7f2af80a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:31.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.439+0000 7f2b15e00700 1 --2- 172.21.15.78:0/3660948685 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2af80a0a90 0x7f2af80a2f50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009990 tx=0x7f2b04000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:31.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.439+0000 7f2b0aa00700 1 -- 172.21.15.78:0/3660948685 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2b00052960 con 0x7f2b10101f30 2024-03-14T09:52:31.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:52:31 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:52:31.192+0000 7fca6a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:52:31.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:52:31 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:52:31.192+0000 7fca6a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:52:31.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:52:31 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:52:31.192+0000 7fca6a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:52:31.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:52:31 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:52:31.192+0000 7fca6a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:52:31.554 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:52:31 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:52:31.192+0000 7fca6a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:52:31.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.680+0000 7f2b18800700 1 -- 172.21.15.78:0/3660948685 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b100611f0 con 0x7f2af80a0a90 2024-03-14T09:52:31.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.703+0000 7f2b0aa00700 1 -- 172.21.15.78:0/3660948685 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2b100611f0 con 0x7f2af80a0a90 2024-03-14T09:52:31.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.705+0000 7f2b18800700 1 -- 172.21.15.78:0/3660948685 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2af80a0a90 msgr2=0x7f2af80a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:31.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.705+0000 7f2b18800700 1 --2- 172.21.15.78:0/3660948685 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2af80a0a90 0x7f2af80a2f50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009990 tx=0x7f2b04000bc0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.705+0000 7f2b18800700 1 -- 172.21.15.78:0/3660948685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b10101f30 msgr2=0x7f2b100731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:31.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.705+0000 7f2b18800700 1 --2- 172.21.15.78:0/3660948685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b10101f30 0x7f2b100731a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2b0000e9f0 tx=0x7f2b0000ed00 comp rx=0 tx=0).stop 2024-03-14T09:52:31.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.705+0000 7f2b18800700 1 -- 172.21.15.78:0/3660948685 shutdown_connections 2024-03-14T09:52:31.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.706+0000 7f2b18800700 1 --2- 172.21.15.78:0/3660948685 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2af80a0a90 0x7f2af80a2f50 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.706+0000 7f2b18800700 1 --2- 172.21.15.78:0/3660948685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2b10100d40 0x7f2b10074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.706+0000 7f2b18800700 1 --2- 172.21.15.78:0/3660948685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b101028d0 0x7f2b100736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.706+0000 7f2b18800700 1 --2- 172.21.15.78:0/3660948685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b10101f30 0x7f2b100731a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.706+0000 7f2b18800700 1 -- 172.21.15.78:0/3660948685 >> 172.21.15.78:0/3660948685 conn(0x7f2b100fc2b0 msgr2=0x7f2b10105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:31.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.706+0000 7f2b18800700 1 -- 172.21.15.78:0/3660948685 shutdown_connections 2024-03-14T09:52:31.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.706+0000 7f2b18800700 1 -- 172.21.15.78:0/3660948685 wait complete. 2024-03-14T09:52:31.724 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-14T09:52:31.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.922+0000 7f5d82200700 1 -- 172.21.15.78:0/1366737001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d7c101f30 msgr2=0x7f5d7c102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.922+0000 7f5d82200700 1 --2- 172.21.15.78:0/1366737001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d7c101f30 0x7f5d7c102390 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5d680099c0 tx=0x7f5d68009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.922+0000 7f5d82200700 1 -- 172.21.15.78:0/1366737001 shutdown_connections 2024-03-14T09:52:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.922+0000 7f5d82200700 1 --2- 172.21.15.78:0/1366737001 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d7c100d40 0x7f5d7c101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.922+0000 7f5d82200700 1 --2- 172.21.15.78:0/1366737001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d7c1028d0 0x7f5d7c109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.922+0000 7f5d82200700 1 --2- 172.21.15.78:0/1366737001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d7c101f30 0x7f5d7c102390 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.922+0000 7f5d82200700 1 -- 172.21.15.78:0/1366737001 >> 172.21.15.78:0/1366737001 conn(0x7f5d7c0fc2b0 msgr2=0x7f5d7c0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.922+0000 7f5d82200700 1 -- 172.21.15.78:0/1366737001 shutdown_connections 2024-03-14T09:52:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.922+0000 7f5d82200700 1 -- 172.21.15.78:0/1366737001 wait complete. 2024-03-14T09:52:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.923+0000 7f5d82200700 1 Processor -- start 2024-03-14T09:52:31.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.923+0000 7f5d82200700 1 -- start start 2024-03-14T09:52:31.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d82200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d7c100d40 0x7f5d7c199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:31.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d82200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d7c101f30 0x7f5d7c1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:31.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d82200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d7c1028d0 0x7f5d7c1a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d82200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c1a0e40 con 0x7f5d7c1028d0 2024-03-14T09:52:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d82200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c1a0f80 con 0x7f5d7c100d40 2024-03-14T09:52:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d82200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c1a1110 con 0x7f5d7c101f30 2024-03-14T09:52:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d7be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d7c1028d0 0x7f5d7c1a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d7b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d7c100d40 0x7f5d7c199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d7b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d7c100d40 0x7f5d7c199360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:49988/0 (socket says 172.21.15.78:49988) 2024-03-14T09:52:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d7be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d7c1028d0 0x7f5d7c1a0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41518/0 (socket says 172.21.15.78:41518) 2024-03-14T09:52:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d7b400700 1 -- 172.21.15.78:0/3714421913 learned_addr learned my addr 172.21.15.78:0/3714421913 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d7aa00700 1 --2- 172.21.15.78:0/3714421913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d7c101f30 0x7f5d7c1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d7be00700 1 -- 172.21.15.78:0/3714421913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d7c101f30 msgr2=0x7f5d7c1998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d7be00700 1 --2- 172.21.15.78:0/3714421913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d7c101f30 0x7f5d7c1998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d7be00700 1 -- 172.21.15.78:0/3714421913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d7c100d40 msgr2=0x7f5d7c199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d7be00700 1 --2- 172.21.15.78:0/3714421913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d7c100d40 0x7f5d7c199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:31.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.924+0000 7f5d7be00700 1 -- 172.21.15.78:0/3714421913 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d680095f0 con 0x7f5d7c1028d0 2024-03-14T09:52:31.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.925+0000 7f5d7be00700 1 --2- 172.21.15.78:0/3714421913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d7c1028d0 0x7f5d7c1a0870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5d74007f10 tx=0x7f5d7400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:31.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.925+0000 7f5d73e00700 1 -- 172.21.15.78:0/3714421913 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d74004db0 con 0x7f5d7c1028d0 2024-03-14T09:52:31.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.925+0000 7f5d73e00700 1 -- 172.21.15.78:0/3714421913 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d7400f040 con 0x7f5d7c1028d0 2024-03-14T09:52:31.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.925+0000 7f5d73e00700 1 -- 172.21.15.78:0/3714421913 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d74005990 con 0x7f5d7c1028d0 2024-03-14T09:52:31.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.925+0000 7f5d82200700 1 -- 172.21.15.78:0/3714421913 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c1a13f0 con 0x7f5d7c1028d0 2024-03-14T09:52:31.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.925+0000 7f5d82200700 1 -- 172.21.15.78:0/3714421913 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c1a18b0 con 0x7f5d7c1028d0 2024-03-14T09:52:31.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.926+0000 7f5d82200700 1 -- 172.21.15.78:0/3714421913 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7c067ac0 con 0x7f5d7c1028d0 2024-03-14T09:52:31.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.928+0000 7f5d73e00700 1 -- 172.21.15.78:0/3714421913 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f5d7400db20 con 0x7f5d7c1028d0 2024-03-14T09:52:31.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.929+0000 7f5d73e00700 1 --2- 172.21.15.78:0/3714421913 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5d600a0b10 0x7f5d600a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:31.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.929+0000 7f5d73e00700 1 -- 172.21.15.78:0/3714421913 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5d740cba30 con 0x7f5d7c1028d0 2024-03-14T09:52:31.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.930+0000 7f5d7b400700 1 --2- 172.21.15.78:0/3714421913 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5d600a0b10 0x7f5d600a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:31.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.930+0000 7f5d7b400700 1 --2- 172.21.15.78:0/3714421913 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5d600a0b10 0x7f5d600a2fd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c005950 tx=0x7f5d6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:31.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:31.934+0000 7f5d73e00700 1 -- 172.21.15.78:0/3714421913 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5d740951b0 con 0x7f5d7c1028d0 2024-03-14T09:52:31.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:31 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] up:active 2024-03-14T09:52:31.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:31 smithi152 ceph-mon[48279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-14T09:52:31.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:31 smithi152 ceph-mon[48279]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 9 up:standby 2024-03-14T09:52:31.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:31 smithi152 ceph-mon[48279]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-14T09:52:31.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:31 smithi152 ceph-mon[48279]: pgmap v248: 129 pgs: 129 active+clean; 582 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-03-14T09:52:31.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:31 smithi152 ceph-mon[48279]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:52:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:31 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] up:active 2024-03-14T09:52:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:31 smithi132 ceph-mon[47959]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-14T09:52:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:31 smithi132 ceph-mon[47959]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 9 up:standby 2024-03-14T09:52:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:31 smithi132 ceph-mon[47959]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-14T09:52:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:31 smithi132 ceph-mon[47959]: pgmap v248: 129 pgs: 129 active+clean; 582 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-03-14T09:52:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:31 smithi132 ceph-mon[47959]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:52:32.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:31 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] up:active 2024-03-14T09:52:32.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:31 smithi078 ceph-mon[38143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-14T09:52:32.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:31 smithi078 ceph-mon[38143]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 9 up:standby 2024-03-14T09:52:32.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:31 smithi078 ceph-mon[38143]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-14T09:52:32.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:31 smithi078 ceph-mon[38143]: pgmap v248: 129 pgs: 129 active+clean; 582 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-03-14T09:52:32.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:31 smithi078 ceph-mon[38143]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:52:32.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.172+0000 7f5d82200700 1 -- 172.21.15.78:0/3714421913 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d7c074700 con 0x7f5d600a0b10 2024-03-14T09:52:32.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.177+0000 7f5d73e00700 1 -- 172.21.15.78:0/3714421913 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f5d7c074700 con 0x7f5d600a0b10 2024-03-14T09:52:32.177 INFO:teuthology.orchestra.run.smithi078.stdout:/volumes/qa/sv_0/5c13563f-740b-44cc-bd2a-0b01fca90c0b 2024-03-14T09:52:32.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.179+0000 7f5d82200700 1 -- 172.21.15.78:0/3714421913 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5d600a0b10 msgr2=0x7f5d600a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:32.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.179+0000 7f5d82200700 1 --2- 172.21.15.78:0/3714421913 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5d600a0b10 0x7f5d600a2fd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c005950 tx=0x7f5d6c009260 comp rx=0 tx=0).stop 2024-03-14T09:52:32.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.179+0000 7f5d82200700 1 -- 172.21.15.78:0/3714421913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d7c1028d0 msgr2=0x7f5d7c1a0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:32.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.179+0000 7f5d82200700 1 --2- 172.21.15.78:0/3714421913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d7c1028d0 0x7f5d7c1a0870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5d74007f10 tx=0x7f5d7400d3b0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.179+0000 7f5d82200700 1 -- 172.21.15.78:0/3714421913 shutdown_connections 2024-03-14T09:52:32.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.179+0000 7f5d82200700 1 --2- 172.21.15.78:0/3714421913 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5d600a0b10 0x7f5d600a2fd0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.179+0000 7f5d82200700 1 --2- 172.21.15.78:0/3714421913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d7c100d40 0x7f5d7c199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.179+0000 7f5d82200700 1 --2- 172.21.15.78:0/3714421913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d7c1028d0 0x7f5d7c1a0870 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.179+0000 7f5d82200700 1 --2- 172.21.15.78:0/3714421913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d7c101f30 0x7f5d7c1998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.179+0000 7f5d82200700 1 -- 172.21.15.78:0/3714421913 >> 172.21.15.78:0/3714421913 conn(0x7f5d7c0fc2b0 msgr2=0x7f5d7c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:32.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.180+0000 7f5d82200700 1 -- 172.21.15.78:0/3714421913 shutdown_connections 2024-03-14T09:52:32.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.180+0000 7f5d82200700 1 -- 172.21.15.78:0/3714421913 wait complete. 2024-03-14T09:52:32.198 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-14T09:52:32.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.397+0000 7f5e87e00700 1 -- 172.21.15.78:0/1631387129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e80074d90 msgr2=0x7f5e800731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.397+0000 7f5e87e00700 1 --2- 172.21.15.78:0/1631387129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e80074d90 0x7f5e800731e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009990 tx=0x7f5e74009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.397+0000 7f5e87e00700 1 -- 172.21.15.78:0/1631387129 shutdown_connections 2024-03-14T09:52:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.397+0000 7f5e87e00700 1 --2- 172.21.15.78:0/1631387129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e80108c20 0x7f5e8010b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.397+0000 7f5e87e00700 1 --2- 172.21.15.78:0/1631387129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e80074d90 0x7f5e800731e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.397+0000 7f5e87e00700 1 --2- 172.21.15.78:0/1631387129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e800737b0 0x7f5e80073c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.397+0000 7f5e87e00700 1 -- 172.21.15.78:0/1631387129 >> 172.21.15.78:0/1631387129 conn(0x7f5e800fc2c0 msgr2=0x7f5e800fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.397+0000 7f5e87e00700 1 -- 172.21.15.78:0/1631387129 shutdown_connections 2024-03-14T09:52:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.397+0000 7f5e87e00700 1 -- 172.21.15.78:0/1631387129 wait complete. 2024-03-14T09:52:32.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.398+0000 7f5e87e00700 1 Processor -- start 2024-03-14T09:52:32.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.398+0000 7f5e87e00700 1 -- start start 2024-03-14T09:52:32.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.399+0000 7f5e87e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e800737b0 0x7f5e80199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:32.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.399+0000 7f5e87e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e80074d90 0x7f5e80199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:32.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.399+0000 7f5e87e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e80108c20 0x7f5e801a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:32.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.399+0000 7f5e87e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e801a0e60 con 0x7f5e80074d90 2024-03-14T09:52:32.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.399+0000 7f5e87e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e801a0fa0 con 0x7f5e80108c20 2024-03-14T09:52:32.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.399+0000 7f5e87e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e801a1130 con 0x7f5e800737b0 2024-03-14T09:52:32.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.399+0000 7f5e84a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e80074d90 0x7f5e80199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:32.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.399+0000 7f5e85400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e800737b0 0x7f5e80199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:32.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.399+0000 7f5e84a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e80074d90 0x7f5e80199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41546/0 (socket says 172.21.15.78:41546) 2024-03-14T09:52:32.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.399+0000 7f5e84a00700 1 -- 172.21.15.78:0/1721349160 learned_addr learned my addr 172.21.15.78:0/1721349160 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:32.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.399+0000 7f5e85e00700 1 --2- 172.21.15.78:0/1721349160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e80108c20 0x7f5e801a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:32.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.400+0000 7f5e84a00700 1 -- 172.21.15.78:0/1721349160 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e800737b0 msgr2=0x7f5e80199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:32.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.400+0000 7f5e84a00700 1 --2- 172.21.15.78:0/1721349160 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e800737b0 0x7f5e80199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.400+0000 7f5e84a00700 1 -- 172.21.15.78:0/1721349160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e80108c20 msgr2=0x7f5e801a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:32.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.400+0000 7f5e84a00700 1 --2- 172.21.15.78:0/1721349160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e80108c20 0x7f5e801a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.400+0000 7f5e84a00700 1 -- 172.21.15.78:0/1721349160 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e740095f0 con 0x7f5e80074d90 2024-03-14T09:52:32.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.400+0000 7f5e85400700 1 --2- 172.21.15.78:0/1721349160 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e800737b0 0x7f5e80199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:32.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.400+0000 7f5e84a00700 1 --2- 172.21.15.78:0/1721349160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e80074d90 0x7f5e80199870 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00e9a0 tx=0x7f5e6c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:32.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.400+0000 7f5e7e000700 1 -- 172.21.15.78:0/1721349160 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c009790 con 0x7f5e80074d90 2024-03-14T09:52:32.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.400+0000 7f5e7e000700 1 -- 172.21.15.78:0/1721349160 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e6c010430 con 0x7f5e80074d90 2024-03-14T09:52:32.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.400+0000 7f5e87e00700 1 -- 172.21.15.78:0/1721349160 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e801a1410 con 0x7f5e80074d90 2024-03-14T09:52:32.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.400+0000 7f5e7e000700 1 -- 172.21.15.78:0/1721349160 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c018680 con 0x7f5e80074d90 2024-03-14T09:52:32.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.400+0000 7f5e87e00700 1 -- 172.21.15.78:0/1721349160 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e801a1970 con 0x7f5e80074d90 2024-03-14T09:52:32.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.401+0000 7f5e87e00700 1 -- 172.21.15.78:0/1721349160 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e80107a30 con 0x7f5e80074d90 2024-03-14T09:52:32.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.403+0000 7f5e7e000700 1 -- 172.21.15.78:0/1721349160 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f5e6c0105a0 con 0x7f5e80074d90 2024-03-14T09:52:32.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.405+0000 7f5e7e000700 1 --2- 172.21.15.78:0/1721349160 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5e680a0b10 0x7f5e680a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:32.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.405+0000 7f5e7e000700 1 -- 172.21.15.78:0/1721349160 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5e6c014070 con 0x7f5e80074d90 2024-03-14T09:52:32.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.405+0000 7f5e85400700 1 --2- 172.21.15.78:0/1721349160 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5e680a0b10 0x7f5e680a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:32.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.406+0000 7f5e85400700 1 --2- 172.21.15.78:0/1721349160 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5e680a0b10 0x7f5e680a2fd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400b2e0 tx=0x7f5e74005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:32.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.407+0000 7f5e7e000700 1 -- 172.21.15.78:0/1721349160 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5e6c095dc0 con 0x7f5e80074d90 2024-03-14T09:52:32.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.642+0000 7f5e87e00700 1 -- 172.21.15.78:0/1721349160 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e800611f0 con 0x7f5e680a0b10 2024-03-14T09:52:32.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.665+0000 7f5e7e000700 1 -- 172.21.15.78:0/1721349160 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5e800611f0 con 0x7f5e680a0b10 2024-03-14T09:52:32.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.667+0000 7f5e87e00700 1 -- 172.21.15.78:0/1721349160 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5e680a0b10 msgr2=0x7f5e680a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:32.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.667+0000 7f5e87e00700 1 --2- 172.21.15.78:0/1721349160 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5e680a0b10 0x7f5e680a2fd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400b2e0 tx=0x7f5e74005fd0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.667+0000 7f5e87e00700 1 -- 172.21.15.78:0/1721349160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e80074d90 msgr2=0x7f5e80199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:32.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.667+0000 7f5e87e00700 1 --2- 172.21.15.78:0/1721349160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e80074d90 0x7f5e80199870 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00e9a0 tx=0x7f5e6c00ecb0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.667+0000 7f5e87e00700 1 -- 172.21.15.78:0/1721349160 shutdown_connections 2024-03-14T09:52:32.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.667+0000 7f5e87e00700 1 --2- 172.21.15.78:0/1721349160 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5e680a0b10 0x7f5e680a2fd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.667+0000 7f5e87e00700 1 --2- 172.21.15.78:0/1721349160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e80108c20 0x7f5e801a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.667+0000 7f5e87e00700 1 --2- 172.21.15.78:0/1721349160 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5e800737b0 0x7f5e80199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.667+0000 7f5e87e00700 1 --2- 172.21.15.78:0/1721349160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5e80074d90 0x7f5e80199870 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.667+0000 7f5e87e00700 1 -- 172.21.15.78:0/1721349160 >> 172.21.15.78:0/1721349160 conn(0x7f5e800fc2c0 msgr2=0x7f5e80104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.668+0000 7f5e87e00700 1 -- 172.21.15.78:0/1721349160 shutdown_connections 2024-03-14T09:52:32.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.668+0000 7f5e87e00700 1 -- 172.21.15.78:0/1721349160 wait complete. 2024-03-14T09:52:32.686 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-14T09:52:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.884+0000 7f3a64200700 1 -- 172.21.15.78:0/4017665758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a5c106a30 msgr2=0x7f3a5c108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.884+0000 7f3a64200700 1 --2- 172.21.15.78:0/4017665758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a5c106a30 0x7f3a5c108ea0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400b0a0 tx=0x7f3a5400b3b0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.884+0000 7f3a64200700 1 -- 172.21.15.78:0/4017665758 shutdown_connections 2024-03-14T09:52:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.884+0000 7f3a64200700 1 --2- 172.21.15.78:0/4017665758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a5c106a30 0x7f3a5c108ea0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.884+0000 7f3a64200700 1 --2- 172.21.15.78:0/4017665758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a5c1017d0 0x7f3a5c103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.884+0000 7f3a64200700 1 --2- 172.21.15.78:0/4017665758 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a5c104100 0x7f3a5c1064f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.884+0000 7f3a64200700 1 -- 172.21.15.78:0/4017665758 >> 172.21.15.78:0/4017665758 conn(0x7f3a5c0fb1c0 msgr2=0x7f3a5c0fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:32.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.884+0000 7f3a64200700 1 -- 172.21.15.78:0/4017665758 shutdown_connections 2024-03-14T09:52:32.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.884+0000 7f3a64200700 1 -- 172.21.15.78:0/4017665758 wait complete. 2024-03-14T09:52:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.885+0000 7f3a64200700 1 Processor -- start 2024-03-14T09:52:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.885+0000 7f3a64200700 1 -- start start 2024-03-14T09:52:32.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.886+0000 7f3a64200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a5c1017d0 0x7f3a5c1971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.886+0000 7f3a64200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a5c104100 0x7f3a5c197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.886+0000 7f3a64200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a5c106a30 0x7f3a5c19e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.886+0000 7f3a64200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c19ed10 con 0x7f3a5c104100 2024-03-14T09:52:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.886+0000 7f3a64200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c19ee50 con 0x7f3a5c1017d0 2024-03-14T09:52:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.886+0000 7f3a64200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c19efe0 con 0x7f3a5c106a30 2024-03-14T09:52:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.886+0000 7f3a60e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a5c104100 0x7f3a5c197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.886+0000 7f3a60e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a5c104100 0x7f3a5c197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41562/0 (socket says 172.21.15.78:41562) 2024-03-14T09:52:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.886+0000 7f3a62200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a5c106a30 0x7f3a5c19e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.886+0000 7f3a60e00700 1 -- 172.21.15.78:0/2601596055 learned_addr learned my addr 172.21.15.78:0/2601596055 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.886+0000 7f3a61800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a5c1017d0 0x7f3a5c1971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:32.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.886+0000 7f3a60e00700 1 -- 172.21.15.78:0/2601596055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a5c106a30 msgr2=0x7f3a5c19e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.886+0000 7f3a60e00700 1 --2- 172.21.15.78:0/2601596055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a5c106a30 0x7f3a5c19e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.886+0000 7f3a60e00700 1 -- 172.21.15.78:0/2601596055 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a5c1017d0 msgr2=0x7f3a5c1971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.886+0000 7f3a60e00700 1 --2- 172.21.15.78:0/2601596055 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a5c1017d0 0x7f3a5c1971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.886+0000 7f3a60e00700 1 -- 172.21.15.78:0/2601596055 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a54009d00 con 0x7f3a5c104100 2024-03-14T09:52:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.887+0000 7f3a60e00700 1 --2- 172.21.15.78:0/2601596055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a5c104100 0x7f3a5c197720 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800e9a0 tx=0x7f3a4800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.887+0000 7f3a5a000700 1 -- 172.21.15.78:0/2601596055 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48009790 con 0x7f3a5c104100 2024-03-14T09:52:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.887+0000 7f3a5a000700 1 -- 172.21.15.78:0/2601596055 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a48010430 con 0x7f3a5c104100 2024-03-14T09:52:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.887+0000 7f3a5a000700 1 -- 172.21.15.78:0/2601596055 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48018680 con 0x7f3a5c104100 2024-03-14T09:52:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.887+0000 7f3a64200700 1 -- 172.21.15.78:0/2601596055 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a5c19f2c0 con 0x7f3a5c104100 2024-03-14T09:52:32.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.887+0000 7f3a64200700 1 -- 172.21.15.78:0/2601596055 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a5c19f7d0 con 0x7f3a5c104100 2024-03-14T09:52:32.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.888+0000 7f3a64200700 1 -- 172.21.15.78:0/2601596055 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a5c0fcd40 con 0x7f3a5c104100 2024-03-14T09:52:32.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.890+0000 7f3a5a000700 1 -- 172.21.15.78:0/2601596055 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f3a480105a0 con 0x7f3a5c104100 2024-03-14T09:52:32.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.891+0000 7f3a5a000700 1 --2- 172.21.15.78:0/2601596055 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3a440a0b10 0x7f3a440a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:32.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.891+0000 7f3a5a000700 1 -- 172.21.15.78:0/2601596055 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3a48014070 con 0x7f3a5c104100 2024-03-14T09:52:32.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.891+0000 7f3a61800700 1 --2- 172.21.15.78:0/2601596055 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3a440a0b10 0x7f3a440a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:32.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.892+0000 7f3a61800700 1 --2- 172.21.15.78:0/2601596055 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3a440a0b10 0x7f3a440a2fd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009c60 tx=0x7f3a50009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:32.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:32.895+0000 7f3a5a000700 1 -- 172.21.15.78:0/2601596055 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3a48052960 con 0x7f3a5c104100 2024-03-14T09:52:32.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:32 smithi152 ceph-mon[48279]: from='client.24814 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:52:32.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:32 smithi152 ceph-mon[48279]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:52:33.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:32 smithi132 ceph-mon[47959]: from='client.24814 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:52:33.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:32 smithi132 ceph-mon[47959]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:52:33.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:32 smithi078 ceph-mon[38143]: from='client.24814 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:52:33.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:32 smithi078 ceph-mon[38143]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:52:33.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.135+0000 7f3a64200700 1 -- 172.21.15.78:0/2601596055 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a5c0611f0 con 0x7f3a440a0b10 2024-03-14T09:52:33.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.139+0000 7f3a5a000700 1 -- 172.21.15.78:0/2601596055 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f3a5c0611f0 con 0x7f3a440a0b10 2024-03-14T09:52:33.139 INFO:teuthology.orchestra.run.smithi078.stdout:/volumes/qa/sv_1/ee9e45a3-d5e7-49ff-9604-f8158c25a08c 2024-03-14T09:52:33.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.141+0000 7f3a64200700 1 -- 172.21.15.78:0/2601596055 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3a440a0b10 msgr2=0x7f3a440a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:33.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.141+0000 7f3a64200700 1 --2- 172.21.15.78:0/2601596055 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3a440a0b10 0x7f3a440a2fd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009c60 tx=0x7f3a50009260 comp rx=0 tx=0).stop 2024-03-14T09:52:33.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.141+0000 7f3a64200700 1 -- 172.21.15.78:0/2601596055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a5c104100 msgr2=0x7f3a5c197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:33.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.141+0000 7f3a64200700 1 --2- 172.21.15.78:0/2601596055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a5c104100 0x7f3a5c197720 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800e9a0 tx=0x7f3a4800ecb0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.141+0000 7f3a64200700 1 -- 172.21.15.78:0/2601596055 shutdown_connections 2024-03-14T09:52:33.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.141+0000 7f3a64200700 1 --2- 172.21.15.78:0/2601596055 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3a440a0b10 0x7f3a440a2fd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.141+0000 7f3a64200700 1 --2- 172.21.15.78:0/2601596055 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a5c1017d0 0x7f3a5c1971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.141+0000 7f3a64200700 1 --2- 172.21.15.78:0/2601596055 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a5c106a30 0x7f3a5c19e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.141+0000 7f3a64200700 1 --2- 172.21.15.78:0/2601596055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a5c104100 0x7f3a5c197720 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.141+0000 7f3a64200700 1 -- 172.21.15.78:0/2601596055 >> 172.21.15.78:0/2601596055 conn(0x7f3a5c0fb1c0 msgr2=0x7f3a5c102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:33.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.141+0000 7f3a64200700 1 -- 172.21.15.78:0/2601596055 shutdown_connections 2024-03-14T09:52:33.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.141+0000 7f3a64200700 1 -- 172.21.15.78:0/2601596055 wait complete. 2024-03-14T09:52:33.160 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.352+0000 7f5419e00700 1 -- 172.21.15.78:0/316544862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5414073810 msgr2=0x7f5414073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.352+0000 7f5419e00700 1 --2- 172.21.15.78:0/316544862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5414073810 0x7f5414073c90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5400009990 tx=0x7f5400009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.352+0000 7f5419e00700 1 -- 172.21.15.78:0/316544862 shutdown_connections 2024-03-14T09:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.352+0000 7f5419e00700 1 --2- 172.21.15.78:0/316544862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5414108b10 0x7f541410afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.352+0000 7f5419e00700 1 --2- 172.21.15.78:0/316544862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5414074df0 0x7f5414073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.352+0000 7f5419e00700 1 --2- 172.21.15.78:0/316544862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5414073810 0x7f5414073c90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.352+0000 7f5419e00700 1 -- 172.21.15.78:0/316544862 >> 172.21.15.78:0/316544862 conn(0x7f54140fc1c0 msgr2=0x7f54140fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.352+0000 7f5419e00700 1 -- 172.21.15.78:0/316544862 shutdown_connections 2024-03-14T09:52:33.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.352+0000 7f5419e00700 1 -- 172.21.15.78:0/316544862 wait complete. 2024-03-14T09:52:33.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.353+0000 7f5419e00700 1 Processor -- start 2024-03-14T09:52:33.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.353+0000 7f5419e00700 1 -- start start 2024-03-14T09:52:33.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.353+0000 7f5419e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5414073810 0x7f5414199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:33.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.353+0000 7f5419e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5414074df0 0x7f54141997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:33.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.353+0000 7f5419e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5414108b10 0x7f54141a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:33.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.353+0000 7f5419e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54141a0d90 con 0x7f5414074df0 2024-03-14T09:52:33.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.354+0000 7f5419e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54141a0ed0 con 0x7f5414108b10 2024-03-14T09:52:33.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.354+0000 7f5419e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54141a1060 con 0x7f5414073810 2024-03-14T09:52:33.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.354+0000 7f5412000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5414074df0 0x7f54141997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:33.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.354+0000 7f5413400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5414108b10 0x7f54141a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:33.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.354+0000 7f5412a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5414073810 0x7f5414199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:33.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.354+0000 7f5413400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5414108b10 0x7f54141a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:50038/0 (socket says 172.21.15.78:50038) 2024-03-14T09:52:33.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.354+0000 7f5412000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5414074df0 0x7f54141997a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41576/0 (socket says 172.21.15.78:41576) 2024-03-14T09:52:33.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.354+0000 7f5413400700 1 -- 172.21.15.78:0/1996046731 learned_addr learned my addr 172.21.15.78:0/1996046731 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:33.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.354+0000 7f5412000700 1 -- 172.21.15.78:0/1996046731 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5414073810 msgr2=0x7f5414199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:33.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.354+0000 7f5412000700 1 --2- 172.21.15.78:0/1996046731 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5414073810 0x7f5414199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.354+0000 7f5412000700 1 -- 172.21.15.78:0/1996046731 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5414108b10 msgr2=0x7f54141a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:33.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.354+0000 7f5412000700 1 --2- 172.21.15.78:0/1996046731 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5414108b10 0x7f54141a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.354+0000 7f5412000700 1 -- 172.21.15.78:0/1996046731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54000095f0 con 0x7f5414074df0 2024-03-14T09:52:33.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.354+0000 7f5412000700 1 --2- 172.21.15.78:0/1996046731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5414074df0 0x7f54141997a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5400000c00 tx=0x7f5400009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:33.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.355+0000 7f5418a00700 1 -- 172.21.15.78:0/1996046731 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540001d070 con 0x7f5414074df0 2024-03-14T09:52:33.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.355+0000 7f5418a00700 1 -- 172.21.15.78:0/1996046731 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5400021410 con 0x7f5414074df0 2024-03-14T09:52:33.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.355+0000 7f5419e00700 1 -- 172.21.15.78:0/1996046731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54141a12e0 con 0x7f5414074df0 2024-03-14T09:52:33.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.355+0000 7f5418a00700 1 -- 172.21.15.78:0/1996046731 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5400017650 con 0x7f5414074df0 2024-03-14T09:52:33.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.355+0000 7f5419e00700 1 -- 172.21.15.78:0/1996046731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54141a1760 con 0x7f5414074df0 2024-03-14T09:52:33.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.356+0000 7f5419e00700 1 -- 172.21.15.78:0/1996046731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54141078d0 con 0x7f5414074df0 2024-03-14T09:52:33.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.358+0000 7f5418a00700 1 -- 172.21.15.78:0/1996046731 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f5400021580 con 0x7f5414074df0 2024-03-14T09:52:33.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.359+0000 7f5418a00700 1 --2- 172.21.15.78:0/1996046731 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f53f80a0b10 0x7f53f80a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:33.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.359+0000 7f5418a00700 1 -- 172.21.15.78:0/1996046731 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f54000cdba0 con 0x7f5414074df0 2024-03-14T09:52:33.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.359+0000 7f5412a00700 1 --2- 172.21.15.78:0/1996046731 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f53f80a0b10 0x7f53f80a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:33.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.360+0000 7f5412a00700 1 --2- 172.21.15.78:0/1996046731 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f53f80a0b10 0x7f53f80a2fd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5404009c60 tx=0x7f5404009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:33.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.361+0000 7f5418a00700 1 -- 172.21.15.78:0/1996046731 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f54000973b0 con 0x7f5414074df0 2024-03-14T09:52:33.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.611+0000 7f5419e00700 1 -- 172.21.15.78:0/1996046731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54140679e0 con 0x7f5414074df0 2024-03-14T09:52:33.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.612+0000 7f5418a00700 1 -- 172.21.15.78:0/1996046731 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f5400005740 con 0x7f5414074df0 2024-03-14T09:52:33.613 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:33.613 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:52:30.715484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:52:33.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.615+0000 7f5419e00700 1 -- 172.21.15.78:0/1996046731 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f53f80a0b10 msgr2=0x7f53f80a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:33.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.615+0000 7f5419e00700 1 --2- 172.21.15.78:0/1996046731 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f53f80a0b10 0x7f53f80a2fd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5404009c60 tx=0x7f5404009260 comp rx=0 tx=0).stop 2024-03-14T09:52:33.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.615+0000 7f5419e00700 1 -- 172.21.15.78:0/1996046731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5414074df0 msgr2=0x7f54141997a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:33.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.615+0000 7f5419e00700 1 --2- 172.21.15.78:0/1996046731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5414074df0 0x7f54141997a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5400000c00 tx=0x7f5400009580 comp rx=0 tx=0).stop 2024-03-14T09:52:33.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.615+0000 7f5419e00700 1 -- 172.21.15.78:0/1996046731 shutdown_connections 2024-03-14T09:52:33.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.615+0000 7f5419e00700 1 --2- 172.21.15.78:0/1996046731 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f53f80a0b10 0x7f53f80a2fd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.615+0000 7f5419e00700 1 --2- 172.21.15.78:0/1996046731 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5414108b10 0x7f54141a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.615+0000 7f5419e00700 1 --2- 172.21.15.78:0/1996046731 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5414073810 0x7f5414199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.615+0000 7f5419e00700 1 --2- 172.21.15.78:0/1996046731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5414074df0 0x7f54141997a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.615+0000 7f5419e00700 1 -- 172.21.15.78:0/1996046731 >> 172.21.15.78:0/1996046731 conn(0x7f54140fc1c0 msgr2=0x7f5414104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:33.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.615+0000 7f5419e00700 1 -- 172.21.15.78:0/1996046731 shutdown_connections 2024-03-14T09:52:33.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.615+0000 7f5419e00700 1 -- 172.21.15.78:0/1996046731 wait complete. 2024-03-14T09:52:33.616 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2024-03-14T09:52:33.635 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:52:33.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.837+0000 7fade6200700 1 -- 172.21.15.78:0/965502476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fade0074df0 msgr2=0x7fade0073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:33.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.837+0000 7fade6200700 1 --2- 172.21.15.78:0/965502476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fade0074df0 0x7fade0073240 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009990 tx=0x7fadd0009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.837+0000 7fade6200700 1 -- 172.21.15.78:0/965502476 shutdown_connections 2024-03-14T09:52:33.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.837+0000 7fade6200700 1 --2- 172.21.15.78:0/965502476 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fade0108b10 0x7fade010afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.837+0000 7fade6200700 1 --2- 172.21.15.78:0/965502476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fade0074df0 0x7fade0073240 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.837+0000 7fade6200700 1 --2- 172.21.15.78:0/965502476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fade0073810 0x7fade0073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.837+0000 7fade6200700 1 -- 172.21.15.78:0/965502476 >> 172.21.15.78:0/965502476 conn(0x7fade00fc1c0 msgr2=0x7fade00fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:33.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.837+0000 7fade6200700 1 -- 172.21.15.78:0/965502476 shutdown_connections 2024-03-14T09:52:33.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.837+0000 7fade6200700 1 -- 172.21.15.78:0/965502476 wait complete. 2024-03-14T09:52:33.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.838+0000 7fade6200700 1 Processor -- start 2024-03-14T09:52:33.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.839+0000 7fade6200700 1 -- start start 2024-03-14T09:52:33.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.839+0000 7fade6200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fade0073810 0x7fade01991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:33.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.839+0000 7fade6200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fade0074df0 0x7fade0199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:33.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.839+0000 7fade6200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fade0108b10 0x7fade01a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:33.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.839+0000 7fade6200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade01a0d20 con 0x7fade0108b10 2024-03-14T09:52:33.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.839+0000 7fade6200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade01a0e60 con 0x7fade0074df0 2024-03-14T09:52:33.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.839+0000 7fade6200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade01a0ff0 con 0x7fade0073810 2024-03-14T09:52:33.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.839+0000 7faddf400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fade0073810 0x7fade01991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:33.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.839+0000 7faddfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fade0108b10 0x7fade01a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:33.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.839+0000 7faddf400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fade0073810 0x7fade01991f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:56540/0 (socket says 172.21.15.78:56540) 2024-03-14T09:52:33.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.839+0000 7faddf400700 1 -- 172.21.15.78:0/3656041553 learned_addr learned my addr 172.21.15.78:0/3656041553 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:33.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.839+0000 7faddea00700 1 --2- 172.21.15.78:0/3656041553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fade0074df0 0x7fade0199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:33.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.840+0000 7faddf400700 1 -- 172.21.15.78:0/3656041553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fade0074df0 msgr2=0x7fade0199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:33.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.840+0000 7faddf400700 1 --2- 172.21.15.78:0/3656041553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fade0074df0 0x7fade0199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.840+0000 7faddf400700 1 -- 172.21.15.78:0/3656041553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fade0108b10 msgr2=0x7fade01a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:33.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.840+0000 7faddf400700 1 --2- 172.21.15.78:0/3656041553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fade0108b10 0x7fade01a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:33.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.840+0000 7faddf400700 1 -- 172.21.15.78:0/3656041553 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd00095f0 con 0x7fade0073810 2024-03-14T09:52:33.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.840+0000 7faddea00700 1 --2- 172.21.15.78:0/3656041553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fade0074df0 0x7fade0199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:52:33.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.840+0000 7faddf400700 1 --2- 172.21.15.78:0/3656041553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fade0073810 0x7fade01991f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fadd000b680 tx=0x7fadd0009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:33.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.840+0000 7fadd7e00700 1 -- 172.21.15.78:0/3656041553 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd001d070 con 0x7fade0073810 2024-03-14T09:52:33.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.840+0000 7fade6200700 1 -- 172.21.15.78:0/3656041553 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade01a1270 con 0x7fade0073810 2024-03-14T09:52:33.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.840+0000 7fadd7e00700 1 -- 172.21.15.78:0/3656041553 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fadd0022470 con 0x7fade0073810 2024-03-14T09:52:33.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.840+0000 7fadd7e00700 1 -- 172.21.15.78:0/3656041553 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0017650 con 0x7fade0073810 2024-03-14T09:52:33.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.840+0000 7fade6200700 1 -- 172.21.15.78:0/3656041553 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade01a16f0 con 0x7fade0073810 2024-03-14T09:52:33.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.841+0000 7fade6200700 1 -- 172.21.15.78:0/3656041553 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade01078d0 con 0x7fade0073810 2024-03-14T09:52:33.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.843+0000 7fadd7e00700 1 -- 172.21.15.78:0/3656041553 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fadd0022870 con 0x7fade0073810 2024-03-14T09:52:33.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.844+0000 7fadd7e00700 1 --2- 172.21.15.78:0/3656041553 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fadc40a0b20 0x7fadc40a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:33.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.844+0000 7fadd7e00700 1 -- 172.21.15.78:0/3656041553 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fadd00cdc90 con 0x7fade0073810 2024-03-14T09:52:33.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.845+0000 7faddea00700 1 --2- 172.21.15.78:0/3656041553 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fadc40a0b20 0x7fadc40a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:33.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.845+0000 7faddea00700 1 --2- 172.21.15.78:0/3656041553 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fadc40a0b20 0x7fadc40a2fe0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fade0074b10 tx=0x7fadcc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:33.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:33.847+0000 7fadd7e00700 1 -- 172.21.15.78:0/3656041553 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fadd0097340 con 0x7fade0073810 2024-03-14T09:52:33.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:33 smithi152 ceph-mon[48279]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:52:33.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:33 smithi152 ceph-mon[48279]: pgmap v249: 129 pgs: 129 active+clean; 584 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-14T09:52:33.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:33 smithi152 ceph-mon[48279]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:52:33.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:33 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1996046731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:52:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:33 smithi132 ceph-mon[47959]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:52:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:33 smithi132 ceph-mon[47959]: pgmap v249: 129 pgs: 129 active+clean; 584 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-14T09:52:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:33 smithi132 ceph-mon[47959]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:52:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:33 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1996046731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:52:34.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:33 smithi078 ceph-mon[38143]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:52:34.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:33 smithi078 ceph-mon[38143]: pgmap v249: 129 pgs: 129 active+clean; 584 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-14T09:52:34.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:33 smithi078 ceph-mon[38143]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:52:34.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:33 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1996046731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:52:34.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.054+0000 7fade6200700 1 -- 172.21.15.78:0/3656041553 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fade00679e0 con 0x7fade0073810 2024-03-14T09:52:34.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.056+0000 7fadd7e00700 1 -- 172.21.15.78:0/3656041553 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+23925 (secure 0 0 0) 0x7fadd009a960 con 0x7fade0073810 2024-03-14T09:52:34.057 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:34.057 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":83,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:52:10.245498+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:52:34.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.059+0000 7fade6200700 1 -- 172.21.15.78:0/3656041553 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fadc40a0b20 msgr2=0x7fadc40a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:34.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.059+0000 7fade6200700 1 --2- 172.21.15.78:0/3656041553 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fadc40a0b20 0x7fadc40a2fe0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fade0074b10 tx=0x7fadcc009260 comp rx=0 tx=0).stop 2024-03-14T09:52:34.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.059+0000 7fade6200700 1 -- 172.21.15.78:0/3656041553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fade0073810 msgr2=0x7fade01991f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:34.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.059+0000 7fade6200700 1 --2- 172.21.15.78:0/3656041553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fade0073810 0x7fade01991f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fadd000b680 tx=0x7fadd0009580 comp rx=0 tx=0).stop 2024-03-14T09:52:34.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.059+0000 7fade6200700 1 -- 172.21.15.78:0/3656041553 shutdown_connections 2024-03-14T09:52:34.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.059+0000 7fade6200700 1 --2- 172.21.15.78:0/3656041553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fade0108b10 0x7fade01a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.059+0000 7fade6200700 1 --2- 172.21.15.78:0/3656041553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fade0073810 0x7fade01991f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.059+0000 7fade6200700 1 --2- 172.21.15.78:0/3656041553 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fadc40a0b20 0x7fadc40a2fe0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.059+0000 7fade6200700 1 --2- 172.21.15.78:0/3656041553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fade0074df0 0x7fade0199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.059+0000 7fade6200700 1 -- 172.21.15.78:0/3656041553 >> 172.21.15.78:0/3656041553 conn(0x7fade00fc1c0 msgr2=0x7fade0104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:34.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.059+0000 7fade6200700 1 -- 172.21.15.78:0/3656041553 shutdown_connections 2024-03-14T09:52:34.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.059+0000 7fade6200700 1 -- 172.21.15.78:0/3656041553 wait complete. 2024-03-14T09:52:34.079 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-14T09:52:34.079 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-14T09:52:34.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.271+0000 7fe33c400700 1 -- 172.21.15.78:0/2720290805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3341027e0 msgr2=0x7fe334109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:34.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.271+0000 7fe33c400700 1 --2- 172.21.15.78:0/2720290805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3341027e0 0x7fe334109090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00b0d0 tx=0x7fe32c00b3e0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.271+0000 7fe33c400700 1 -- 172.21.15.78:0/2720290805 shutdown_connections 2024-03-14T09:52:34.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.271+0000 7fe33c400700 1 --2- 172.21.15.78:0/2720290805 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe334100c50 0x7fe334101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.271+0000 7fe33c400700 1 --2- 172.21.15.78:0/2720290805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3341027e0 0x7fe334109090 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.271+0000 7fe33c400700 1 --2- 172.21.15.78:0/2720290805 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe334101e40 0x7fe3341022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.271+0000 7fe33c400700 1 -- 172.21.15.78:0/2720290805 >> 172.21.15.78:0/2720290805 conn(0x7fe3340fc1c0 msgr2=0x7fe3340fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:34.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.271+0000 7fe33c400700 1 -- 172.21.15.78:0/2720290805 shutdown_connections 2024-03-14T09:52:34.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.271+0000 7fe33c400700 1 -- 172.21.15.78:0/2720290805 wait complete. 2024-03-14T09:52:34.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.272+0000 7fe33c400700 1 Processor -- start 2024-03-14T09:52:34.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.272+0000 7fe33c400700 1 -- start start 2024-03-14T09:52:34.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.273+0000 7fe33c400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe334100c50 0x7fe3341993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:34.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.273+0000 7fe33c400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe334101e40 0x7fe3341998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:34.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.273+0000 7fe33c400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3341027e0 0x7fe3341a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:34.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.273+0000 7fe33c400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3341a0e90 con 0x7fe334100c50 2024-03-14T09:52:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.273+0000 7fe33c400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3341a0fd0 con 0x7fe334101e40 2024-03-14T09:52:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.273+0000 7fe33c400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3341a1160 con 0x7fe3341027e0 2024-03-14T09:52:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.273+0000 7fe339a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe334100c50 0x7fe3341993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.273+0000 7fe339a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe334100c50 0x7fe3341993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50452/0 (socket says 172.21.15.78:50452) 2024-03-14T09:52:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.273+0000 7fe339a00700 1 -- 172.21.15.78:0/1808463574 learned_addr learned my addr 172.21.15.78:0/1808463574 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.273+0000 7fe339000700 1 --2- 172.21.15.78:0/1808463574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe334101e40 0x7fe3341998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.273+0000 7fe33a400700 1 --2- 172.21.15.78:0/1808463574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3341027e0 0x7fe3341a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.273+0000 7fe339a00700 1 -- 172.21.15.78:0/1808463574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3341027e0 msgr2=0x7fe3341a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.273+0000 7fe339a00700 1 --2- 172.21.15.78:0/1808463574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3341027e0 0x7fe3341a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.273+0000 7fe339a00700 1 -- 172.21.15.78:0/1808463574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe334101e40 msgr2=0x7fe3341998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.273+0000 7fe339a00700 1 --2- 172.21.15.78:0/1808463574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe334101e40 0x7fe3341998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.273+0000 7fe339a00700 1 -- 172.21.15.78:0/1808463574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c009d00 con 0x7fe334100c50 2024-03-14T09:52:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.274+0000 7fe339a00700 1 --2- 172.21.15.78:0/1808463574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe334100c50 0x7fe3341993b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe32800e970 tx=0x7fe32800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:34.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.274+0000 7fe332000700 1 -- 172.21.15.78:0/1808463574 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe328009790 con 0x7fe334100c50 2024-03-14T09:52:34.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.274+0000 7fe332000700 1 -- 172.21.15.78:0/1808463574 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe32800c7e0 con 0x7fe334100c50 2024-03-14T09:52:34.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.274+0000 7fe332000700 1 -- 172.21.15.78:0/1808463574 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe328003c70 con 0x7fe334100c50 2024-03-14T09:52:34.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.274+0000 7fe33c400700 1 -- 172.21.15.78:0/1808463574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3341a1440 con 0x7fe334100c50 2024-03-14T09:52:34.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.274+0000 7fe33c400700 1 -- 172.21.15.78:0/1808463574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3341a1900 con 0x7fe334100c50 2024-03-14T09:52:34.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.275+0000 7fe33c400700 1 -- 172.21.15.78:0/1808463574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3340679e0 con 0x7fe334100c50 2024-03-14T09:52:34.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.277+0000 7fe332000700 1 -- 172.21.15.78:0/1808463574 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fe32800cc50 con 0x7fe334100c50 2024-03-14T09:52:34.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.278+0000 7fe332000700 1 --2- 172.21.15.78:0/1808463574 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe31c0a0b20 0x7fe31c0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:34.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.278+0000 7fe332000700 1 -- 172.21.15.78:0/1808463574 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe328014070 con 0x7fe334100c50 2024-03-14T09:52:34.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.278+0000 7fe339000700 1 --2- 172.21.15.78:0/1808463574 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe31c0a0b20 0x7fe31c0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:34.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.279+0000 7fe339000700 1 --2- 172.21.15.78:0/1808463574 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe31c0a0b20 0x7fe31c0a2fe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe320009c60 tx=0x7fe320009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:34.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.282+0000 7fe332000700 1 -- 172.21.15.78:0/1808463574 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe328095c20 con 0x7fe334100c50 2024-03-14T09:52:34.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.496+0000 7fe33c400700 1 -- 172.21.15.78:0/1808463574 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe334074700 con 0x7fe31c0a0b20 2024-03-14T09:52:34.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.511+0000 7fe332000700 1 -- 172.21.15.78:0/1808463574 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422671 (secure 0 0 0) 0x7fe334074700 con 0x7fe31c0a0b20 2024-03-14T09:52:34.519 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:34.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.517+0000 7fe33c400700 1 -- 172.21.15.78:0/1808463574 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe31c0a0b20 msgr2=0x7fe31c0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:34.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.517+0000 7fe33c400700 1 --2- 172.21.15.78:0/1808463574 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe31c0a0b20 0x7fe31c0a2fe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe320009c60 tx=0x7fe320009260 comp rx=0 tx=0).stop 2024-03-14T09:52:34.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.517+0000 7fe33c400700 1 -- 172.21.15.78:0/1808463574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe334100c50 msgr2=0x7fe3341993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:34.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.517+0000 7fe33c400700 1 --2- 172.21.15.78:0/1808463574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe334100c50 0x7fe3341993b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe32800e970 tx=0x7fe32800ec80 comp rx=0 tx=0).stop 2024-03-14T09:52:34.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.517+0000 7fe33c400700 1 -- 172.21.15.78:0/1808463574 shutdown_connections 2024-03-14T09:52:34.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.517+0000 7fe33c400700 1 --2- 172.21.15.78:0/1808463574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3341027e0 0x7fe3341a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.517+0000 7fe33c400700 1 --2- 172.21.15.78:0/1808463574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe334100c50 0x7fe3341993b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.517+0000 7fe33c400700 1 --2- 172.21.15.78:0/1808463574 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe31c0a0b20 0x7fe31c0a2fe0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.517+0000 7fe33c400700 1 --2- 172.21.15.78:0/1808463574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe334101e40 0x7fe3341998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.517+0000 7fe33c400700 1 -- 172.21.15.78:0/1808463574 >> 172.21.15.78:0/1808463574 conn(0x7fe3340fc1c0 msgr2=0x7fe334072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:34.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.517+0000 7fe33c400700 1 -- 172.21.15.78:0/1808463574 shutdown_connections 2024-03-14T09:52:34.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.517+0000 7fe33c400700 1 -- 172.21.15.78:0/1808463574 wait complete. 2024-03-14T09:52:34.524 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-14T09:52:34.537 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-03-14T09:52:33.072198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597587,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":150,"num_read_kb":131,"num_write":298,"num_write_kb":2980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":241,"ondisk_log_size":241,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":335088,"kb_used_data":10896,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124606736,"statfs":{"total":1151940427776,"available":1151597297664,"internally_reserved":0,"allocated":11157504,"data_stored":7179486,"data_compressed":247893,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":19131,"internal_metadata":331855173},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4407,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":14,"num_read_kb":12,"num_write":71,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003427"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.749056+0000","last_change":"2024-03-14T09:52:05.737617+0000","last_active":"2024-03-14T09:52:09.749056+0000","last_peered":"2024-03-14T09:52:09.749056+0000","last_clean":"2024-03-14T09:52:09.749056+0000","last_became_active":"2024-03-14T09:52:05.736796+0000","last_became_peered":"2024-03-14T09:52:05.736796+0000","last_unstale":"2024-03-14T09:52:09.749056+0000","last_undegraded":"2024-03-14T09:52:09.749056+0000","last_fullsized":"2024-03-14T09:52:09.749056+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:47:52.437748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.756225+0000","last_change":"2024-03-14T09:52:05.749474+0000","last_active":"2024-03-14T09:52:09.756225+0000","last_peered":"2024-03-14T09:52:09.756225+0000","last_clean":"2024-03-14T09:52:09.756225+0000","last_became_active":"2024-03-14T09:52:05.748927+0000","last_became_peered":"2024-03-14T09:52:05.748927+0000","last_unstale":"2024-03-14T09:52:09.756225+0000","last_undegraded":"2024-03-14T09:52:09.756225+0000","last_fullsized":"2024-03-14T09:52:09.756225+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:28:15.852021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.753505+0000","last_change":"2024-03-14T09:52:05.745098+0000","last_active":"2024-03-14T09:52:09.753505+0000","last_peered":"2024-03-14T09:52:09.753505+0000","last_clean":"2024-03-14T09:52:09.753505+0000","last_became_active":"2024-03-14T09:52:05.744755+0000","last_became_peered":"2024-03-14T09:52:05.744755+0000","last_unstale":"2024-03-14T09:52:09.753505+0000","last_undegraded":"2024-03-14T09:52:09.753505+0000","last_fullsized":"2024-03-14T09:52:09.753505+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:35:41.932386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748570+0000","last_change":"2024-03-14T09:52:05.737429+0000","last_active":"2024-03-14T09:52:09.748570+0000","last_peered":"2024-03-14T09:52:09.748570+0000","last_clean":"2024-03-14T09:52:09.748570+0000","last_became_active":"2024-03-14T09:52:05.736516+0000","last_became_peered":"2024-03-14T09:52:05.736516+0000","last_unstale":"2024-03-14T09:52:09.748570+0000","last_undegraded":"2024-03-14T09:52:09.748570+0000","last_fullsized":"2024-03-14T09:52:09.748570+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:48:38.158783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.752188+0000","last_change":"2024-03-14T09:52:05.729634+0000","last_active":"2024-03-14T09:52:09.752188+0000","last_peered":"2024-03-14T09:52:09.752188+0000","last_clean":"2024-03-14T09:52:09.752188+0000","last_became_active":"2024-03-14T09:52:05.727889+0000","last_became_peered":"2024-03-14T09:52:05.727889+0000","last_unstale":"2024-03-14T09:52:09.752188+0000","last_undegraded":"2024-03-14T09:52:09.752188+0000","last_fullsized":"2024-03-14T09:52:09.752188+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:01:13.216763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.755590+0000","last_change":"2024-03-14T09:52:05.728270+0000","last_active":"2024-03-14T09:52:09.755590+0000","last_peered":"2024-03-14T09:52:09.755590+0000","last_clean":"2024-03-14T09:52:09.755590+0000","last_became_active":"2024-03-14T09:52:05.727915+0000","last_became_peered":"2024-03-14T09:52:05.727915+0000","last_unstale":"2024-03-14T09:52:09.755590+0000","last_undegraded":"2024-03-14T09:52:09.755590+0000","last_fullsized":"2024-03-14T09:52:09.755590+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:22:22.616901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744072+0000","last_change":"2024-03-14T09:52:05.727365+0000","last_active":"2024-03-14T09:52:09.744072+0000","last_peered":"2024-03-14T09:52:09.744072+0000","last_clean":"2024-03-14T09:52:09.744072+0000","last_became_active":"2024-03-14T09:52:05.726984+0000","last_became_peered":"2024-03-14T09:52:05.726984+0000","last_unstale":"2024-03-14T09:52:09.744072+0000","last_undegraded":"2024-03-14T09:52:09.744072+0000","last_fullsized":"2024-03-14T09:52:09.744072+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:42:57.129718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.743534+0000","last_change":"2024-03-14T09:52:05.729622+0000","last_active":"2024-03-14T09:52:09.743534+0000","last_peered":"2024-03-14T09:52:09.743534+0000","last_clean":"2024-03-14T09:52:09.743534+0000","last_became_active":"2024-03-14T09:52:05.729293+0000","last_became_peered":"2024-03-14T09:52:05.729293+0000","last_unstale":"2024-03-14T09:52:09.743534+0000","last_undegraded":"2024-03-14T09:52:09.743534+0000","last_fullsized":"2024-03-14T09:52:09.743534+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:13:20.875857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.747564+0000","last_change":"2024-03-14T09:52:05.741620+0000","last_active":"2024-03-14T09:52:09.747564+0000","last_peered":"2024-03-14T09:52:09.747564+0000","last_clean":"2024-03-14T09:52:09.747564+0000","last_became_active":"2024-03-14T09:52:05.734079+0000","last_became_peered":"2024-03-14T09:52:05.734079+0000","last_unstale":"2024-03-14T09:52:09.747564+0000","last_undegraded":"2024-03-14T09:52:09.747564+0000","last_fullsized":"2024-03-14T09:52:09.747564+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:02:35.529671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.755870+0000","last_change":"2024-03-14T09:52:05.743424+0000","last_active":"2024-03-14T09:52:09.755870+0000","last_peered":"2024-03-14T09:52:09.755870+0000","last_clean":"2024-03-14T09:52:09.755870+0000","last_became_active":"2024-03-14T09:52:05.743100+0000","last_became_peered":"2024-03-14T09:52:05.743100+0000","last_unstale":"2024-03-14T09:52:09.755870+0000","last_undegraded":"2024-03-14T09:52:09.755870+0000","last_fullsized":"2024-03-14T09:52:09.755870+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:32:33.492163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744618+0000","last_change":"2024-03-14T09:52:05.735654+0000","last_active":"2024-03-14T09:52:09.744618+0000","last_peered":"2024-03-14T09:52:09.744618+0000","last_clean":"2024-03-14T09:52:09.744618+0000","last_became_active":"2024-03-14T09:52:05.735371+0000","last_became_peered":"2024-03-14T09:52:05.735371+0000","last_unstale":"2024-03-14T09:52:09.744618+0000","last_undegraded":"2024-03-14T09:52:09.744618+0000","last_fullsized":"2024-03-14T09:52:09.744618+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:19:50.011554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748484+0000","last_change":"2024-03-14T09:52:05.737322+0000","last_active":"2024-03-14T09:52:09.748484+0000","last_peered":"2024-03-14T09:52:09.748484+0000","last_clean":"2024-03-14T09:52:09.748484+0000","last_became_active":"2024-03-14T09:52:05.736834+0000","last_became_peered":"2024-03-14T09:52:05.736834+0000","last_unstale":"2024-03-14T09:52:09.748484+0000","last_undegraded":"2024-03-14T09:52:09.748484+0000","last_fullsized":"2024-03-14T09:52:09.748484+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:11:04.601826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.906071+0000","last_change":"2024-03-14T09:52:05.740209+0000","last_active":"2024-03-14T09:52:09.906071+0000","last_peered":"2024-03-14T09:52:09.906071+0000","last_clean":"2024-03-14T09:52:09.906071+0000","last_became_active":"2024-03-14T09:52:05.732949+0000","last_became_peered":"2024-03-14T09:52:05.732949+0000","last_unstale":"2024-03-14T09:52:09.906071+0000","last_undegraded":"2024-03-14T09:52:09.906071+0000","last_fullsized":"2024-03-14T09:52:09.906071+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:15:56.377569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.754711+0000","last_change":"2024-03-14T09:52:05.748795+0000","last_active":"2024-03-14T09:52:09.754711+0000","last_peered":"2024-03-14T09:52:09.754711+0000","last_clean":"2024-03-14T09:52:09.754711+0000","last_became_active":"2024-03-14T09:52:05.746107+0000","last_became_peered":"2024-03-14T09:52:05.746107+0000","last_unstale":"2024-03-14T09:52:09.754711+0000","last_undegraded":"2024-03-14T09:52:09.754711+0000","last_fullsized":"2024-03-14T09:52:09.754711+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:49:07.932854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.756958+0000","last_change":"2024-03-14T09:52:05.744174+0000","last_active":"2024-03-14T09:52:09.756958+0000","last_peered":"2024-03-14T09:52:09.756958+0000","last_clean":"2024-03-14T09:52:09.756958+0000","last_became_active":"2024-03-14T09:52:05.743848+0000","last_became_peered":"2024-03-14T09:52:05.743848+0000","last_unstale":"2024-03-14T09:52:09.756958+0000","last_undegraded":"2024-03-14T09:52:09.756958+0000","last_fullsized":"2024-03-14T09:52:09.756958+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:07:23.260734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748597+0000","last_change":"2024-03-14T09:52:05.733677+0000","last_active":"2024-03-14T09:52:09.748597+0000","last_peered":"2024-03-14T09:52:09.748597+0000","last_clean":"2024-03-14T09:52:09.748597+0000","last_became_active":"2024-03-14T09:52:05.730967+0000","last_became_peered":"2024-03-14T09:52:05.730967+0000","last_unstale":"2024-03-14T09:52:09.748597+0000","last_undegraded":"2024-03-14T09:52:09.748597+0000","last_fullsized":"2024-03-14T09:52:09.748597+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:33:53.484741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748820+0000","last_change":"2024-03-14T09:52:05.737836+0000","last_active":"2024-03-14T09:52:09.748820+0000","last_peered":"2024-03-14T09:52:09.748820+0000","last_clean":"2024-03-14T09:52:09.748820+0000","last_became_active":"2024-03-14T09:52:05.737257+0000","last_became_peered":"2024-03-14T09:52:05.737257+0000","last_unstale":"2024-03-14T09:52:09.748820+0000","last_undegraded":"2024-03-14T09:52:09.748820+0000","last_fullsized":"2024-03-14T09:52:09.748820+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:53.723415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.766338+0000","last_change":"2024-03-14T09:52:05.741360+0000","last_active":"2024-03-14T09:52:09.766338+0000","last_peered":"2024-03-14T09:52:09.766338+0000","last_clean":"2024-03-14T09:52:09.766338+0000","last_became_active":"2024-03-14T09:52:05.740673+0000","last_became_peered":"2024-03-14T09:52:05.740673+0000","last_unstale":"2024-03-14T09:52:09.766338+0000","last_undegraded":"2024-03-14T09:52:09.766338+0000","last_fullsized":"2024-03-14T09:52:09.766338+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:13:41.561338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.758671+0000","last_change":"2024-03-14T09:52:05.739134+0000","last_active":"2024-03-14T09:52:09.758671+0000","last_peered":"2024-03-14T09:52:09.758671+0000","last_clean":"2024-03-14T09:52:09.758671+0000","last_became_active":"2024-03-14T09:52:05.738439+0000","last_became_peered":"2024-03-14T09:52:05.738439+0000","last_unstale":"2024-03-14T09:52:09.758671+0000","last_undegraded":"2024-03-14T09:52:09.758671+0000","last_fullsized":"2024-03-14T09:52:09.758671+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:05:36.997462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.250705+0000","last_change":"2024-03-14T09:52:09.769284+0000","last_active":"2024-03-14T09:52:10.250705+0000","last_peered":"2024-03-14T09:52:10.250705+0000","last_clean":"2024-03-14T09:52:10.250705+0000","last_became_active":"2024-03-14T09:52:09.768808+0000","last_became_peered":"2024-03-14T09:52:09.768808+0000","last_unstale":"2024-03-14T09:52:10.250705+0000","last_undegraded":"2024-03-14T09:52:10.250705+0000","last_fullsized":"2024-03-14T09:52:10.250705+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:48:56.015410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.720923+0000","last_change":"2024-03-14T09:52:05.227962+0000","last_active":"2024-03-14T09:52:29.720923+0000","last_peered":"2024-03-14T09:52:29.720923+0000","last_clean":"2024-03-14T09:52:29.720923+0000","last_became_active":"2024-03-14T09:52:05.226313+0000","last_became_peered":"2024-03-14T09:52:05.226313+0000","last_unstale":"2024-03-14T09:52:29.720923+0000","last_undegraded":"2024-03-14T09:52:29.720923+0000","last_fullsized":"2024-03-14T09:52:29.720923+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:54:46.291940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744051+0000","last_change":"2024-03-14T09:52:05.739004+0000","last_active":"2024-03-14T09:52:09.744051+0000","last_peered":"2024-03-14T09:52:09.744051+0000","last_clean":"2024-03-14T09:52:09.744051+0000","last_became_active":"2024-03-14T09:52:05.732217+0000","last_became_peered":"2024-03-14T09:52:05.732217+0000","last_unstale":"2024-03-14T09:52:09.744051+0000","last_undegraded":"2024-03-14T09:52:09.744051+0000","last_fullsized":"2024-03-14T09:52:09.744051+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:02:21.926623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.486348+0000","last_change":"2024-03-14T09:52:09.776128+0000","last_active":"2024-03-14T09:52:10.486348+0000","last_peered":"2024-03-14T09:52:10.486348+0000","last_clean":"2024-03-14T09:52:10.486348+0000","last_became_active":"2024-03-14T09:52:09.775701+0000","last_became_peered":"2024-03-14T09:52:09.775701+0000","last_unstale":"2024-03-14T09:52:10.486348+0000","last_undegraded":"2024-03-14T09:52:10.486348+0000","last_fullsized":"2024-03-14T09:52:10.486348+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:41:51.790627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"83'2","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967420+0000","last_change":"2024-03-14T09:52:05.234269+0000","last_active":"2024-03-14T09:52:10.967420+0000","last_peered":"2024-03-14T09:52:10.967420+0000","last_clean":"2024-03-14T09:52:10.967420+0000","last_became_active":"2024-03-14T09:52:05.233418+0000","last_became_peered":"2024-03-14T09:52:05.233418+0000","last_unstale":"2024-03-14T09:52:10.967420+0000","last_undegraded":"2024-03-14T09:52:10.967420+0000","last_fullsized":"2024-03-14T09:52:10.967420+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:22:46.894461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.755348+0000","last_change":"2024-03-14T09:52:05.741786+0000","last_active":"2024-03-14T09:52:09.755348+0000","last_peered":"2024-03-14T09:52:09.755348+0000","last_clean":"2024-03-14T09:52:09.755348+0000","last_became_active":"2024-03-14T09:52:05.741461+0000","last_became_peered":"2024-03-14T09:52:05.741461+0000","last_unstale":"2024-03-14T09:52:09.755348+0000","last_undegraded":"2024-03-14T09:52:09.755348+0000","last_fullsized":"2024-03-14T09:52:09.755348+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:21:17.409530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.486387+0000","last_change":"2024-03-14T09:52:09.768851+0000","last_active":"2024-03-14T09:52:10.486387+0000","last_peered":"2024-03-14T09:52:10.486387+0000","last_clean":"2024-03-14T09:52:10.486387+0000","last_became_active":"2024-03-14T09:52:09.768485+0000","last_became_peered":"2024-03-14T09:52:09.768485+0000","last_unstale":"2024-03-14T09:52:10.486387+0000","last_undegraded":"2024-03-14T09:52:10.486387+0000","last_fullsized":"2024-03-14T09:52:10.486387+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:00:51.189790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"83'4","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.718482+0000","last_change":"2024-03-14T09:52:05.232864+0000","last_active":"2024-03-14T09:52:29.718482+0000","last_peered":"2024-03-14T09:52:29.718482+0000","last_clean":"2024-03-14T09:52:29.718482+0000","last_became_active":"2024-03-14T09:52:05.232546+0000","last_became_peered":"2024-03-14T09:52:05.232546+0000","last_unstale":"2024-03-14T09:52:29.718482+0000","last_undegraded":"2024-03-14T09:52:29.718482+0000","last_fullsized":"2024-03-14T09:52:29.718482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:05:00.418174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.747368+0000","last_change":"2024-03-14T09:52:05.742994+0000","last_active":"2024-03-14T09:52:09.747368+0000","last_peered":"2024-03-14T09:52:09.747368+0000","last_clean":"2024-03-14T09:52:09.747368+0000","last_became_active":"2024-03-14T09:52:05.742652+0000","last_became_peered":"2024-03-14T09:52:05.742652+0000","last_unstale":"2024-03-14T09:52:09.747368+0000","last_undegraded":"2024-03-14T09:52:09.747368+0000","last_fullsized":"2024-03-14T09:52:09.747368+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:46:47.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.486376+0000","last_change":"2024-03-14T09:52:09.763009+0000","last_active":"2024-03-14T09:52:10.486376+0000","last_peered":"2024-03-14T09:52:10.486376+0000","last_clean":"2024-03-14T09:52:10.486376+0000","last_became_active":"2024-03-14T09:52:09.762621+0000","last_became_peered":"2024-03-14T09:52:09.762621+0000","last_unstale":"2024-03-14T09:52:10.486376+0000","last_undegraded":"2024-03-14T09:52:10.486376+0000","last_fullsized":"2024-03-14T09:52:10.486376+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:56:22.162409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.752237+0000","last_change":"2024-03-14T09:52:05.234039+0000","last_active":"2024-03-14T09:52:09.752237+0000","last_peered":"2024-03-14T09:52:09.752237+0000","last_clean":"2024-03-14T09:52:09.752237+0000","last_became_active":"2024-03-14T09:52:05.233612+0000","last_became_peered":"2024-03-14T09:52:05.233612+0000","last_unstale":"2024-03-14T09:52:09.752237+0000","last_undegraded":"2024-03-14T09:52:09.752237+0000","last_fullsized":"2024-03-14T09:52:09.752237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:29:52.968178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744407+0000","last_change":"2024-03-14T09:52:05.731932+0000","last_active":"2024-03-14T09:52:09.744407+0000","last_peered":"2024-03-14T09:52:09.744407+0000","last_clean":"2024-03-14T09:52:09.744407+0000","last_became_active":"2024-03-14T09:52:05.731618+0000","last_became_peered":"2024-03-14T09:52:05.731618+0000","last_unstale":"2024-03-14T09:52:09.744407+0000","last_undegraded":"2024-03-14T09:52:09.744407+0000","last_fullsized":"2024-03-14T09:52:09.744407+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:50:26.151271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.251006+0000","last_change":"2024-03-14T09:52:09.757856+0000","last_active":"2024-03-14T09:52:10.251006+0000","last_peered":"2024-03-14T09:52:10.251006+0000","last_clean":"2024-03-14T09:52:10.251006+0000","last_became_active":"2024-03-14T09:52:09.757395+0000","last_became_peered":"2024-03-14T09:52:09.757395+0000","last_unstale":"2024-03-14T09:52:10.251006+0000","last_undegraded":"2024-03-14T09:52:10.251006+0000","last_fullsized":"2024-03-14T09:52:10.251006+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:34:06.600727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"83'5","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:30.728822+0000","last_change":"2024-03-14T09:52:05.228527+0000","last_active":"2024-03-14T09:52:30.728822+0000","last_peered":"2024-03-14T09:52:30.728822+0000","last_clean":"2024-03-14T09:52:30.728822+0000","last_became_active":"2024-03-14T09:52:05.228177+0000","last_became_peered":"2024-03-14T09:52:05.228177+0000","last_unstale":"2024-03-14T09:52:30.728822+0000","last_undegraded":"2024-03-14T09:52:30.728822+0000","last_fullsized":"2024-03-14T09:52:30.728822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:17:57.529308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.752152+0000","last_change":"2024-03-14T09:52:05.734041+0000","last_active":"2024-03-14T09:52:09.752152+0000","last_peered":"2024-03-14T09:52:09.752152+0000","last_clean":"2024-03-14T09:52:09.752152+0000","last_became_active":"2024-03-14T09:52:05.732491+0000","last_became_peered":"2024-03-14T09:52:05.732491+0000","last_unstale":"2024-03-14T09:52:09.752152+0000","last_undegraded":"2024-03-14T09:52:09.752152+0000","last_fullsized":"2024-03-14T09:52:09.752152+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:07:31.909382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.250818+0000","last_change":"2024-03-14T09:52:09.769584+0000","last_active":"2024-03-14T09:52:10.250818+0000","last_peered":"2024-03-14T09:52:10.250818+0000","last_clean":"2024-03-14T09:52:10.250818+0000","last_became_active":"2024-03-14T09:52:09.768847+0000","last_became_peered":"2024-03-14T09:52:09.768847+0000","last_unstale":"2024-03-14T09:52:10.250818+0000","last_undegraded":"2024-03-14T09:52:10.250818+0000","last_fullsized":"2024-03-14T09:52:10.250818+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:12:18.005122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"78'1","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.755585+0000","last_change":"2024-03-14T09:52:05.228893+0000","last_active":"2024-03-14T09:52:09.755585+0000","last_peered":"2024-03-14T09:52:09.755585+0000","last_clean":"2024-03-14T09:52:09.755585+0000","last_became_active":"2024-03-14T09:52:05.228569+0000","last_became_peered":"2024-03-14T09:52:05.228569+0000","last_unstale":"2024-03-14T09:52:09.755585+0000","last_undegraded":"2024-03-14T09:52:09.755585+0000","last_fullsized":"2024-03-14T09:52:09.755585+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:28:03.200437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744492+0000","last_change":"2024-03-14T09:52:05.736388+0000","last_active":"2024-03-14T09:52:09.744492+0000","last_peered":"2024-03-14T09:52:09.744492+0000","last_clean":"2024-03-14T09:52:09.744492+0000","last_became_active":"2024-03-14T09:52:05.735131+0000","last_became_peered":"2024-03-14T09:52:05.735131+0000","last_unstale":"2024-03-14T09:52:09.744492+0000","last_undegraded":"2024-03-14T09:52:09.744492+0000","last_fullsized":"2024-03-14T09:52:09.744492+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:00:54.120087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967733+0000","last_change":"2024-03-14T09:52:09.770321+0000","last_active":"2024-03-14T09:52:10.967733+0000","last_peered":"2024-03-14T09:52:10.967733+0000","last_clean":"2024-03-14T09:52:10.967733+0000","last_became_active":"2024-03-14T09:52:09.769710+0000","last_became_peered":"2024-03-14T09:52:09.769710+0000","last_unstale":"2024-03-14T09:52:10.967733+0000","last_undegraded":"2024-03-14T09:52:10.967733+0000","last_fullsized":"2024-03-14T09:52:10.967733+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:12:12.635248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"83'2","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.721097+0000","last_change":"2024-03-14T09:52:05.223912+0000","last_active":"2024-03-14T09:52:29.721097+0000","last_peered":"2024-03-14T09:52:29.721097+0000","last_clean":"2024-03-14T09:52:29.721097+0000","last_became_active":"2024-03-14T09:52:05.223380+0000","last_became_peered":"2024-03-14T09:52:05.223380+0000","last_unstale":"2024-03-14T09:52:29.721097+0000","last_undegraded":"2024-03-14T09:52:29.721097+0000","last_fullsized":"2024-03-14T09:52:29.721097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:08:33.098215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.743438+0000","last_change":"2024-03-14T09:52:05.743277+0000","last_active":"2024-03-14T09:52:09.743438+0000","last_peered":"2024-03-14T09:52:09.743438+0000","last_clean":"2024-03-14T09:52:09.743438+0000","last_became_active":"2024-03-14T09:52:05.741386+0000","last_became_peered":"2024-03-14T09:52:05.741386+0000","last_unstale":"2024-03-14T09:52:09.743438+0000","last_undegraded":"2024-03-14T09:52:09.743438+0000","last_fullsized":"2024-03-14T09:52:09.743438+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:08:08.320279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967655+0000","last_change":"2024-03-14T09:52:09.766485+0000","last_active":"2024-03-14T09:52:10.967655+0000","last_peered":"2024-03-14T09:52:10.967655+0000","last_clean":"2024-03-14T09:52:10.967655+0000","last_became_active":"2024-03-14T09:52:09.766115+0000","last_became_peered":"2024-03-14T09:52:09.766115+0000","last_unstale":"2024-03-14T09:52:10.967655+0000","last_undegraded":"2024-03-14T09:52:10.967655+0000","last_fullsized":"2024-03-14T09:52:10.967655+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:34:45.067757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"83'1","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.968581+0000","last_change":"2024-03-14T09:52:05.233852+0000","last_active":"2024-03-14T09:52:10.968581+0000","last_peered":"2024-03-14T09:52:10.968581+0000","last_clean":"2024-03-14T09:52:10.968581+0000","last_became_active":"2024-03-14T09:52:05.233548+0000","last_became_peered":"2024-03-14T09:52:05.233548+0000","last_unstale":"2024-03-14T09:52:10.968581+0000","last_undegraded":"2024-03-14T09:52:10.968581+0000","last_fullsized":"2024-03-14T09:52:10.968581+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:08:51.821465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.760078+0000","last_change":"2024-03-14T09:52:05.731215+0000","last_active":"2024-03-14T09:52:09.760078+0000","last_peered":"2024-03-14T09:52:09.760078+0000","last_clean":"2024-03-14T09:52:09.760078+0000","last_became_active":"2024-03-14T09:52:05.730924+0000","last_became_peered":"2024-03-14T09:52:05.730924+0000","last_unstale":"2024-03-14T09:52:09.760078+0000","last_undegraded":"2024-03-14T09:52:09.760078+0000","last_fullsized":"2024-03-14T09:52:09.760078+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:12:17.560297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967932+0000","last_change":"2024-03-14T09:52:09.753775+0000","last_active":"2024-03-14T09:52:10.967932+0000","last_peered":"2024-03-14T09:52:10.967932+0000","last_clean":"2024-03-14T09:52:10.967932+0000","last_became_active":"2024-03-14T09:52:09.753422+0000","last_became_peered":"2024-03-14T09:52:09.753422+0000","last_unstale":"2024-03-14T09:52:10.967932+0000","last_undegraded":"2024-03-14T09:52:10.967932+0000","last_fullsized":"2024-03-14T09:52:10.967932+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:59:25.625909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.755861+0000","last_change":"2024-03-14T09:52:05.229064+0000","last_active":"2024-03-14T09:52:09.755861+0000","last_peered":"2024-03-14T09:52:09.755861+0000","last_clean":"2024-03-14T09:52:09.755861+0000","last_became_active":"2024-03-14T09:52:05.228751+0000","last_became_peered":"2024-03-14T09:52:05.228751+0000","last_unstale":"2024-03-14T09:52:09.755861+0000","last_undegraded":"2024-03-14T09:52:09.755861+0000","last_fullsized":"2024-03-14T09:52:09.755861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:13:29.832866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748082+0000","last_change":"2024-03-14T09:52:05.737798+0000","last_active":"2024-03-14T09:52:09.748082+0000","last_peered":"2024-03-14T09:52:09.748082+0000","last_clean":"2024-03-14T09:52:09.748082+0000","last_became_active":"2024-03-14T09:52:05.737449+0000","last_became_peered":"2024-03-14T09:52:05.737449+0000","last_unstale":"2024-03-14T09:52:09.748082+0000","last_undegraded":"2024-03-14T09:52:09.748082+0000","last_fullsized":"2024-03-14T09:52:09.748082+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:40:10.079770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.966435+0000","last_change":"2024-03-14T09:52:09.768368+0000","last_active":"2024-03-14T09:52:10.966435+0000","last_peered":"2024-03-14T09:52:10.966435+0000","last_clean":"2024-03-14T09:52:10.966435+0000","last_became_active":"2024-03-14T09:52:09.767622+0000","last_became_peered":"2024-03-14T09:52:09.767622+0000","last_unstale":"2024-03-14T09:52:10.966435+0000","last_undegraded":"2024-03-14T09:52:10.966435+0000","last_fullsized":"2024-03-14T09:52:10.966435+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:19:28.836317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"83'2","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.715610+0000","last_change":"2024-03-14T09:52:05.229778+0000","last_active":"2024-03-14T09:52:28.715610+0000","last_peered":"2024-03-14T09:52:28.715610+0000","last_clean":"2024-03-14T09:52:28.715610+0000","last_became_active":"2024-03-14T09:52:05.229468+0000","last_became_peered":"2024-03-14T09:52:05.229468+0000","last_unstale":"2024-03-14T09:52:28.715610+0000","last_undegraded":"2024-03-14T09:52:28.715610+0000","last_fullsized":"2024-03-14T09:52:28.715610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:49:58.433646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.755436+0000","last_change":"2024-03-14T09:52:05.751027+0000","last_active":"2024-03-14T09:52:09.755436+0000","last_peered":"2024-03-14T09:52:09.755436+0000","last_clean":"2024-03-14T09:52:09.755436+0000","last_became_active":"2024-03-14T09:52:05.750650+0000","last_became_peered":"2024-03-14T09:52:05.750650+0000","last_unstale":"2024-03-14T09:52:09.755436+0000","last_undegraded":"2024-03-14T09:52:09.755436+0000","last_fullsized":"2024-03-14T09:52:09.755436+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:05:37.264396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.486422+0000","last_change":"2024-03-14T09:52:09.768207+0000","last_active":"2024-03-14T09:52:10.486422+0000","last_peered":"2024-03-14T09:52:10.486422+0000","last_clean":"2024-03-14T09:52:10.486422+0000","last_became_active":"2024-03-14T09:52:09.767756+0000","last_became_peered":"2024-03-14T09:52:09.767756+0000","last_unstale":"2024-03-14T09:52:10.486422+0000","last_undegraded":"2024-03-14T09:52:10.486422+0000","last_fullsized":"2024-03-14T09:52:10.486422+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:13:12.544995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"83'4","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.717910+0000","last_change":"2024-03-14T09:52:05.235472+0000","last_active":"2024-03-14T09:52:28.717910+0000","last_peered":"2024-03-14T09:52:28.717910+0000","last_clean":"2024-03-14T09:52:28.717910+0000","last_became_active":"2024-03-14T09:52:05.235106+0000","last_became_peered":"2024-03-14T09:52:05.235106+0000","last_unstale":"2024-03-14T09:52:28.717910+0000","last_undegraded":"2024-03-14T09:52:28.717910+0000","last_fullsized":"2024-03-14T09:52:28.717910+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:30:36.446130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.747531+0000","last_change":"2024-03-14T09:52:05.747615+0000","last_active":"2024-03-14T09:52:09.747531+0000","last_peered":"2024-03-14T09:52:09.747531+0000","last_clean":"2024-03-14T09:52:09.747531+0000","last_became_active":"2024-03-14T09:52:05.747344+0000","last_became_peered":"2024-03-14T09:52:05.747344+0000","last_unstale":"2024-03-14T09:52:09.747531+0000","last_undegraded":"2024-03-14T09:52:09.747531+0000","last_fullsized":"2024-03-14T09:52:09.747531+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:52:23.096124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967414+0000","last_change":"2024-03-14T09:52:09.768295+0000","last_active":"2024-03-14T09:52:10.967414+0000","last_peered":"2024-03-14T09:52:10.967414+0000","last_clean":"2024-03-14T09:52:10.967414+0000","last_became_active":"2024-03-14T09:52:09.760997+0000","last_became_peered":"2024-03-14T09:52:09.760997+0000","last_unstale":"2024-03-14T09:52:10.967414+0000","last_undegraded":"2024-03-14T09:52:10.967414+0000","last_fullsized":"2024-03-14T09:52:10.967414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:22:09.909302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"83'10","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:25.294154+0000","last_change":"2024-03-14T09:52:05.227876+0000","last_active":"2024-03-14T09:52:25.294154+0000","last_peered":"2024-03-14T09:52:25.294154+0000","last_clean":"2024-03-14T09:52:25.294154+0000","last_became_active":"2024-03-14T09:52:05.227519+0000","last_became_peered":"2024-03-14T09:52:05.227519+0000","last_unstale":"2024-03-14T09:52:25.294154+0000","last_undegraded":"2024-03-14T09:52:25.294154+0000","last_fullsized":"2024-03-14T09:52:25.294154+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:12.616609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3795,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.752533+0000","last_change":"2024-03-14T09:52:05.737432+0000","last_active":"2024-03-14T09:52:09.752533+0000","last_peered":"2024-03-14T09:52:09.752533+0000","last_clean":"2024-03-14T09:52:09.752533+0000","last_became_active":"2024-03-14T09:52:05.735050+0000","last_became_peered":"2024-03-14T09:52:05.735050+0000","last_unstale":"2024-03-14T09:52:09.752533+0000","last_undegraded":"2024-03-14T09:52:09.752533+0000","last_fullsized":"2024-03-14T09:52:09.752533+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T09:57:45.383155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.968799+0000","last_change":"2024-03-14T09:52:09.754909+0000","last_active":"2024-03-14T09:52:10.968799+0000","last_peered":"2024-03-14T09:52:10.968799+0000","last_clean":"2024-03-14T09:52:10.968799+0000","last_became_active":"2024-03-14T09:52:09.754367+0000","last_became_peered":"2024-03-14T09:52:09.754367+0000","last_unstale":"2024-03-14T09:52:10.968799+0000","last_undegraded":"2024-03-14T09:52:10.968799+0000","last_fullsized":"2024-03-14T09:52:10.968799+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:52:34.251555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"83'6","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.720249+0000","last_change":"2024-03-14T09:52:05.224869+0000","last_active":"2024-03-14T09:52:29.720249+0000","last_peered":"2024-03-14T09:52:29.720249+0000","last_clean":"2024-03-14T09:52:29.720249+0000","last_became_active":"2024-03-14T09:52:05.224499+0000","last_became_peered":"2024-03-14T09:52:05.224499+0000","last_unstale":"2024-03-14T09:52:29.720249+0000","last_undegraded":"2024-03-14T09:52:29.720249+0000","last_fullsized":"2024-03-14T09:52:29.720249+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:49:01.729417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.906059+0000","last_change":"2024-03-14T09:52:05.729168+0000","last_active":"2024-03-14T09:52:09.906059+0000","last_peered":"2024-03-14T09:52:09.906059+0000","last_clean":"2024-03-14T09:52:09.906059+0000","last_became_active":"2024-03-14T09:52:05.728806+0000","last_became_peered":"2024-03-14T09:52:05.728806+0000","last_unstale":"2024-03-14T09:52:09.906059+0000","last_undegraded":"2024-03-14T09:52:09.906059+0000","last_fullsized":"2024-03-14T09:52:09.906059+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:54:36.758853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.966838+0000","last_change":"2024-03-14T09:52:09.770126+0000","last_active":"2024-03-14T09:52:10.966838+0000","last_peered":"2024-03-14T09:52:10.966838+0000","last_clean":"2024-03-14T09:52:10.966838+0000","last_became_active":"2024-03-14T09:52:09.769734+0000","last_became_peered":"2024-03-14T09:52:09.769734+0000","last_unstale":"2024-03-14T09:52:10.966838+0000","last_undegraded":"2024-03-14T09:52:10.966838+0000","last_fullsized":"2024-03-14T09:52:10.966838+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:50:37.839858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"83'3","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.716453+0000","last_change":"2024-03-14T09:52:05.235201+0000","last_active":"2024-03-14T09:52:28.716453+0000","last_peered":"2024-03-14T09:52:28.716453+0000","last_clean":"2024-03-14T09:52:28.716453+0000","last_became_active":"2024-03-14T09:52:05.234816+0000","last_became_peered":"2024-03-14T09:52:05.234816+0000","last_unstale":"2024-03-14T09:52:28.716453+0000","last_undegraded":"2024-03-14T09:52:28.716453+0000","last_fullsized":"2024-03-14T09:52:28.716453+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:01:50.070913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744271+0000","last_change":"2024-03-14T09:52:05.735304+0000","last_active":"2024-03-14T09:52:09.744271+0000","last_peered":"2024-03-14T09:52:09.744271+0000","last_clean":"2024-03-14T09:52:09.744271+0000","last_became_active":"2024-03-14T09:52:05.734977+0000","last_became_peered":"2024-03-14T09:52:05.734977+0000","last_unstale":"2024-03-14T09:52:09.744271+0000","last_undegraded":"2024-03-14T09:52:09.744271+0000","last_fullsized":"2024-03-14T09:52:09.744271+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:08:28.071962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.968580+0000","last_change":"2024-03-14T09:52:09.770721+0000","last_active":"2024-03-14T09:52:10.968580+0000","last_peered":"2024-03-14T09:52:10.968580+0000","last_clean":"2024-03-14T09:52:10.968580+0000","last_became_active":"2024-03-14T09:52:09.770270+0000","last_became_peered":"2024-03-14T09:52:09.770270+0000","last_unstale":"2024-03-14T09:52:10.968580+0000","last_undegraded":"2024-03-14T09:52:10.968580+0000","last_fullsized":"2024-03-14T09:52:10.968580+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:09:33.480235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"83'3","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.718282+0000","last_change":"2024-03-14T09:52:05.227260+0000","last_active":"2024-03-14T09:52:29.718282+0000","last_peered":"2024-03-14T09:52:29.718282+0000","last_clean":"2024-03-14T09:52:29.718282+0000","last_became_active":"2024-03-14T09:52:05.226922+0000","last_became_peered":"2024-03-14T09:52:05.226922+0000","last_unstale":"2024-03-14T09:52:29.718282+0000","last_undegraded":"2024-03-14T09:52:29.718282+0000","last_fullsized":"2024-03-14T09:52:29.718282+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:36:09.770478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.743907+0000","last_change":"2024-03-14T09:52:05.734278+0000","last_active":"2024-03-14T09:52:09.743907+0000","last_peered":"2024-03-14T09:52:09.743907+0000","last_clean":"2024-03-14T09:52:09.743907+0000","last_became_active":"2024-03-14T09:52:05.732051+0000","last_became_peered":"2024-03-14T09:52:05.732051+0000","last_unstale":"2024-03-14T09:52:09.743907+0000","last_undegraded":"2024-03-14T09:52:09.743907+0000","last_fullsized":"2024-03-14T09:52:09.743907+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:33:07.137983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.906154+0000","last_change":"2024-03-14T09:52:09.764068+0000","last_active":"2024-03-14T09:52:09.906154+0000","last_peered":"2024-03-14T09:52:09.906154+0000","last_clean":"2024-03-14T09:52:09.906154+0000","last_became_active":"2024-03-14T09:52:09.763613+0000","last_became_peered":"2024-03-14T09:52:09.763613+0000","last_unstale":"2024-03-14T09:52:09.906154+0000","last_undegraded":"2024-03-14T09:52:09.906154+0000","last_fullsized":"2024-03-14T09:52:09.906154+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:32:15.524602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"83'15","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.721151+0000","last_change":"2024-03-14T09:52:05.233845+0000","last_active":"2024-03-14T09:52:29.721151+0000","last_peered":"2024-03-14T09:52:29.721151+0000","last_clean":"2024-03-14T09:52:29.721151+0000","last_became_active":"2024-03-14T09:52:05.233527+0000","last_became_peered":"2024-03-14T09:52:05.233527+0000","last_unstale":"2024-03-14T09:52:29.721151+0000","last_undegraded":"2024-03-14T09:52:29.721151+0000","last_fullsized":"2024-03-14T09:52:29.721151+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T15:20:57.496761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.745334+0000","last_change":"2024-03-14T09:52:05.734366+0000","last_active":"2024-03-14T09:52:09.745334+0000","last_peered":"2024-03-14T09:52:09.745334+0000","last_clean":"2024-03-14T09:52:09.745334+0000","last_became_active":"2024-03-14T09:52:05.734058+0000","last_became_peered":"2024-03-14T09:52:05.734058+0000","last_unstale":"2024-03-14T09:52:09.745334+0000","last_undegraded":"2024-03-14T09:52:09.745334+0000","last_fullsized":"2024-03-14T09:52:09.745334+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:49:49.233683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.968347+0000","last_change":"2024-03-14T09:52:09.760241+0000","last_active":"2024-03-14T09:52:10.968347+0000","last_peered":"2024-03-14T09:52:10.968347+0000","last_clean":"2024-03-14T09:52:10.968347+0000","last_became_active":"2024-03-14T09:52:09.759797+0000","last_became_peered":"2024-03-14T09:52:09.759797+0000","last_unstale":"2024-03-14T09:52:10.968347+0000","last_undegraded":"2024-03-14T09:52:10.968347+0000","last_fullsized":"2024-03-14T09:52:10.968347+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:50:33.035775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'1","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.966996+0000","last_change":"2024-03-14T09:52:05.232932+0000","last_active":"2024-03-14T09:52:10.966996+0000","last_peered":"2024-03-14T09:52:10.966996+0000","last_clean":"2024-03-14T09:52:10.966996+0000","last_became_active":"2024-03-14T09:52:05.232608+0000","last_became_peered":"2024-03-14T09:52:05.232608+0000","last_unstale":"2024-03-14T09:52:10.966996+0000","last_undegraded":"2024-03-14T09:52:10.966996+0000","last_fullsized":"2024-03-14T09:52:10.966996+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:43:59.613531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748431+0000","last_change":"2024-03-14T09:52:05.733762+0000","last_active":"2024-03-14T09:52:09.748431+0000","last_peered":"2024-03-14T09:52:09.748431+0000","last_clean":"2024-03-14T09:52:09.748431+0000","last_became_active":"2024-03-14T09:52:05.733417+0000","last_became_peered":"2024-03-14T09:52:05.733417+0000","last_unstale":"2024-03-14T09:52:09.748431+0000","last_undegraded":"2024-03-14T09:52:09.748431+0000","last_fullsized":"2024-03-14T09:52:09.748431+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:11:12.593952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967586+0000","last_change":"2024-03-14T09:52:09.761405+0000","last_active":"2024-03-14T09:52:10.967586+0000","last_peered":"2024-03-14T09:52:10.967586+0000","last_clean":"2024-03-14T09:52:10.967586+0000","last_became_active":"2024-03-14T09:52:09.760935+0000","last_became_peered":"2024-03-14T09:52:09.760935+0000","last_unstale":"2024-03-14T09:52:10.967586+0000","last_undegraded":"2024-03-14T09:52:10.967586+0000","last_fullsized":"2024-03-14T09:52:10.967586+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:12:30.663121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.766214+0000","last_change":"2024-03-14T09:52:05.227636+0000","last_active":"2024-03-14T09:52:09.766214+0000","last_peered":"2024-03-14T09:52:09.766214+0000","last_clean":"2024-03-14T09:52:09.766214+0000","last_became_active":"2024-03-14T09:52:05.227274+0000","last_became_peered":"2024-03-14T09:52:05.227274+0000","last_unstale":"2024-03-14T09:52:09.766214+0000","last_undegraded":"2024-03-14T09:52:09.766214+0000","last_fullsized":"2024-03-14T09:52:09.766214+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:13:25.380090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.756002+0000","last_change":"2024-03-14T09:52:05.733158+0000","last_active":"2024-03-14T09:52:09.756002+0000","last_peered":"2024-03-14T09:52:09.756002+0000","last_clean":"2024-03-14T09:52:09.756002+0000","last_became_active":"2024-03-14T09:52:05.732812+0000","last_became_peered":"2024-03-14T09:52:05.732812+0000","last_unstale":"2024-03-14T09:52:09.756002+0000","last_undegraded":"2024-03-14T09:52:09.756002+0000","last_fullsized":"2024-03-14T09:52:09.756002+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:01:49.154860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.906092+0000","last_change":"2024-03-14T09:52:09.761596+0000","last_active":"2024-03-14T09:52:09.906092+0000","last_peered":"2024-03-14T09:52:09.906092+0000","last_clean":"2024-03-14T09:52:09.906092+0000","last_became_active":"2024-03-14T09:52:09.761184+0000","last_became_peered":"2024-03-14T09:52:09.761184+0000","last_unstale":"2024-03-14T09:52:09.906092+0000","last_undegraded":"2024-03-14T09:52:09.906092+0000","last_fullsized":"2024-03-14T09:52:09.906092+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:49:02.567472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"83'1","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.250602+0000","last_change":"2024-03-14T09:52:05.231828+0000","last_active":"2024-03-14T09:52:10.250602+0000","last_peered":"2024-03-14T09:52:10.250602+0000","last_clean":"2024-03-14T09:52:10.250602+0000","last_became_active":"2024-03-14T09:52:05.231490+0000","last_became_peered":"2024-03-14T09:52:05.231490+0000","last_unstale":"2024-03-14T09:52:10.250602+0000","last_undegraded":"2024-03-14T09:52:10.250602+0000","last_fullsized":"2024-03-14T09:52:10.250602+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:24:53.899728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.756180+0000","last_change":"2024-03-14T09:52:05.744646+0000","last_active":"2024-03-14T09:52:09.756180+0000","last_peered":"2024-03-14T09:52:09.756180+0000","last_clean":"2024-03-14T09:52:09.756180+0000","last_became_active":"2024-03-14T09:52:05.744327+0000","last_became_peered":"2024-03-14T09:52:05.744327+0000","last_unstale":"2024-03-14T09:52:09.756180+0000","last_undegraded":"2024-03-14T09:52:09.756180+0000","last_fullsized":"2024-03-14T09:52:09.756180+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:46:50.643350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.906188+0000","last_change":"2024-03-14T09:52:09.765374+0000","last_active":"2024-03-14T09:52:09.906188+0000","last_peered":"2024-03-14T09:52:09.906188+0000","last_clean":"2024-03-14T09:52:09.906188+0000","last_became_active":"2024-03-14T09:52:09.764939+0000","last_became_peered":"2024-03-14T09:52:09.764939+0000","last_unstale":"2024-03-14T09:52:09.906188+0000","last_undegraded":"2024-03-14T09:52:09.906188+0000","last_fullsized":"2024-03-14T09:52:09.906188+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:08:10.688746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.745417+0000","last_change":"2024-03-14T09:52:05.729076+0000","last_active":"2024-03-14T09:52:09.745417+0000","last_peered":"2024-03-14T09:52:09.745417+0000","last_clean":"2024-03-14T09:52:09.745417+0000","last_became_active":"2024-03-14T09:52:05.728711+0000","last_became_peered":"2024-03-14T09:52:05.728711+0000","last_unstale":"2024-03-14T09:52:09.745417+0000","last_undegraded":"2024-03-14T09:52:09.745417+0000","last_fullsized":"2024-03-14T09:52:09.745417+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:11:05.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"83'4","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.721389+0000","last_change":"2024-03-14T09:52:05.236209+0000","last_active":"2024-03-14T09:52:29.721389+0000","last_peered":"2024-03-14T09:52:29.721389+0000","last_clean":"2024-03-14T09:52:29.721389+0000","last_became_active":"2024-03-14T09:52:05.235517+0000","last_became_peered":"2024-03-14T09:52:05.235517+0000","last_unstale":"2024-03-14T09:52:29.721389+0000","last_undegraded":"2024-03-14T09:52:29.721389+0000","last_fullsized":"2024-03-14T09:52:29.721389+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:09:54.275728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.250741+0000","last_change":"2024-03-14T09:52:09.761095+0000","last_active":"2024-03-14T09:52:10.250741+0000","last_peered":"2024-03-14T09:52:10.250741+0000","last_clean":"2024-03-14T09:52:10.250741+0000","last_became_active":"2024-03-14T09:52:09.760662+0000","last_became_peered":"2024-03-14T09:52:09.760662+0000","last_unstale":"2024-03-14T09:52:10.250741+0000","last_undegraded":"2024-03-14T09:52:10.250741+0000","last_fullsized":"2024-03-14T09:52:10.250741+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:07:17.670605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.743531+0000","last_change":"2024-03-14T09:52:05.736078+0000","last_active":"2024-03-14T09:52:09.743531+0000","last_peered":"2024-03-14T09:52:09.743531+0000","last_clean":"2024-03-14T09:52:09.743531+0000","last_became_active":"2024-03-14T09:52:05.735715+0000","last_became_peered":"2024-03-14T09:52:05.735715+0000","last_unstale":"2024-03-14T09:52:09.743531+0000","last_undegraded":"2024-03-14T09:52:09.743531+0000","last_fullsized":"2024-03-14T09:52:09.743531+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:41:32.863680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"83'3","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.718447+0000","last_change":"2024-03-14T09:52:05.225756+0000","last_active":"2024-03-14T09:52:28.718447+0000","last_peered":"2024-03-14T09:52:28.718447+0000","last_clean":"2024-03-14T09:52:28.718447+0000","last_became_active":"2024-03-14T09:52:05.225378+0000","last_became_peered":"2024-03-14T09:52:05.225378+0000","last_unstale":"2024-03-14T09:52:28.718447+0000","last_undegraded":"2024-03-14T09:52:28.718447+0000","last_fullsized":"2024-03-14T09:52:28.718447+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:14:49.594636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":335,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.250396+0000","last_change":"2024-03-14T09:51:11.588790+0000","last_active":"2024-03-14T09:52:10.250396+0000","last_peered":"2024-03-14T09:52:10.250396+0000","last_clean":"2024-03-14T09:52:10.250396+0000","last_became_active":"2024-03-14T09:51:11.588287+0000","last_became_peered":"2024-03-14T09:51:11.588287+0000","last_unstale":"2024-03-14T09:52:10.250396+0000","last_undegraded":"2024-03-14T09:52:10.250396+0000","last_fullsized":"2024-03-14T09:52:10.250396+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:48:37.651174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:48:37.651174+0000","last_clean_scrub_stamp":"2024-03-14T09:48:37.651174+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:06:24.252565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967223+0000","last_change":"2024-03-14T09:52:09.761344+0000","last_active":"2024-03-14T09:52:10.967223+0000","last_peered":"2024-03-14T09:52:10.967223+0000","last_clean":"2024-03-14T09:52:10.967223+0000","last_became_active":"2024-03-14T09:52:09.760899+0000","last_became_peered":"2024-03-14T09:52:09.760899+0000","last_unstale":"2024-03-14T09:52:10.967223+0000","last_undegraded":"2024-03-14T09:52:10.967223+0000","last_fullsized":"2024-03-14T09:52:10.967223+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:24:26.212157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.905984+0000","last_change":"2024-03-14T09:52:05.738735+0000","last_active":"2024-03-14T09:52:09.905984+0000","last_peered":"2024-03-14T09:52:09.905984+0000","last_clean":"2024-03-14T09:52:09.905984+0000","last_became_active":"2024-03-14T09:52:05.738447+0000","last_became_peered":"2024-03-14T09:52:05.738447+0000","last_unstale":"2024-03-14T09:52:09.905984+0000","last_undegraded":"2024-03-14T09:52:09.905984+0000","last_fullsized":"2024-03-14T09:52:09.905984+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:03:24.598291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'3","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.719407+0000","last_change":"2024-03-14T09:52:05.230107+0000","last_active":"2024-03-14T09:52:29.719407+0000","last_peered":"2024-03-14T09:52:29.719407+0000","last_clean":"2024-03-14T09:52:29.719407+0000","last_became_active":"2024-03-14T09:52:05.229812+0000","last_became_peered":"2024-03-14T09:52:05.229812+0000","last_unstale":"2024-03-14T09:52:29.719407+0000","last_undegraded":"2024-03-14T09:52:29.719407+0000","last_fullsized":"2024-03-14T09:52:29.719407+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:09:44.884867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967721+0000","last_change":"2024-03-14T09:52:09.763424+0000","last_active":"2024-03-14T09:52:10.967721+0000","last_peered":"2024-03-14T09:52:10.967721+0000","last_clean":"2024-03-14T09:52:10.967721+0000","last_became_active":"2024-03-14T09:52:09.763028+0000","last_became_peered":"2024-03-14T09:52:09.763028+0000","last_unstale":"2024-03-14T09:52:10.967721+0000","last_undegraded":"2024-03-14T09:52:10.967721+0000","last_fullsized":"2024-03-14T09:52:10.967721+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:47:31.415788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"83'1","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.714897+0000","last_change":"2024-03-14T09:52:05.235099+0000","last_active":"2024-03-14T09:52:28.714897+0000","last_peered":"2024-03-14T09:52:28.714897+0000","last_clean":"2024-03-14T09:52:28.714897+0000","last_became_active":"2024-03-14T09:52:05.234798+0000","last_became_peered":"2024-03-14T09:52:05.234798+0000","last_unstale":"2024-03-14T09:52:28.714897+0000","last_undegraded":"2024-03-14T09:52:28.714897+0000","last_fullsized":"2024-03-14T09:52:28.714897+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:00:07.461931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.906103+0000","last_change":"2024-03-14T09:52:05.742736+0000","last_active":"2024-03-14T09:52:09.906103+0000","last_peered":"2024-03-14T09:52:09.906103+0000","last_clean":"2024-03-14T09:52:09.906103+0000","last_became_active":"2024-03-14T09:52:05.740927+0000","last_became_peered":"2024-03-14T09:52:05.740927+0000","last_unstale":"2024-03-14T09:52:09.906103+0000","last_undegraded":"2024-03-14T09:52:09.906103+0000","last_fullsized":"2024-03-14T09:52:09.906103+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:50:17.738763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967299+0000","last_change":"2024-03-14T09:52:09.762722+0000","last_active":"2024-03-14T09:52:10.967299+0000","last_peered":"2024-03-14T09:52:10.967299+0000","last_clean":"2024-03-14T09:52:10.967299+0000","last_became_active":"2024-03-14T09:52:09.759053+0000","last_became_peered":"2024-03-14T09:52:09.759053+0000","last_unstale":"2024-03-14T09:52:10.967299+0000","last_undegraded":"2024-03-14T09:52:10.967299+0000","last_fullsized":"2024-03-14T09:52:10.967299+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:40:29.680002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"83'3","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.719859+0000","last_change":"2024-03-14T09:52:05.224778+0000","last_active":"2024-03-14T09:52:29.719859+0000","last_peered":"2024-03-14T09:52:29.719859+0000","last_clean":"2024-03-14T09:52:29.719859+0000","last_became_active":"2024-03-14T09:52:05.223158+0000","last_became_peered":"2024-03-14T09:52:05.223158+0000","last_unstale":"2024-03-14T09:52:29.719859+0000","last_undegraded":"2024-03-14T09:52:29.719859+0000","last_fullsized":"2024-03-14T09:52:29.719859+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:01:29.910099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744495+0000","last_change":"2024-03-14T09:52:05.741753+0000","last_active":"2024-03-14T09:52:09.744495+0000","last_peered":"2024-03-14T09:52:09.744495+0000","last_clean":"2024-03-14T09:52:09.744495+0000","last_became_active":"2024-03-14T09:52:05.741451+0000","last_became_peered":"2024-03-14T09:52:05.741451+0000","last_unstale":"2024-03-14T09:52:09.744495+0000","last_undegraded":"2024-03-14T09:52:09.744495+0000","last_fullsized":"2024-03-14T09:52:09.744495+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:42:59.920909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.250575+0000","last_change":"2024-03-14T09:52:09.755884+0000","last_active":"2024-03-14T09:52:10.250575+0000","last_peered":"2024-03-14T09:52:10.250575+0000","last_clean":"2024-03-14T09:52:10.250575+0000","last_became_active":"2024-03-14T09:52:09.755474+0000","last_became_peered":"2024-03-14T09:52:09.755474+0000","last_unstale":"2024-03-14T09:52:10.250575+0000","last_undegraded":"2024-03-14T09:52:10.250575+0000","last_fullsized":"2024-03-14T09:52:10.250575+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:59:23.383388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"83'4","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.717963+0000","last_change":"2024-03-14T09:52:05.232673+0000","last_active":"2024-03-14T09:52:28.717963+0000","last_peered":"2024-03-14T09:52:28.717963+0000","last_clean":"2024-03-14T09:52:28.717963+0000","last_became_active":"2024-03-14T09:52:05.232296+0000","last_became_peered":"2024-03-14T09:52:05.232296+0000","last_unstale":"2024-03-14T09:52:28.717963+0000","last_undegraded":"2024-03-14T09:52:28.717963+0000","last_fullsized":"2024-03-14T09:52:28.717963+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:24:56.638451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.756306+0000","last_change":"2024-03-14T09:52:05.744906+0000","last_active":"2024-03-14T09:52:09.756306+0000","last_peered":"2024-03-14T09:52:09.756306+0000","last_clean":"2024-03-14T09:52:09.756306+0000","last_became_active":"2024-03-14T09:52:05.743753+0000","last_became_peered":"2024-03-14T09:52:05.743753+0000","last_unstale":"2024-03-14T09:52:09.756306+0000","last_undegraded":"2024-03-14T09:52:09.756306+0000","last_fullsized":"2024-03-14T09:52:09.756306+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T09:59:40.358352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.486476+0000","last_change":"2024-03-14T09:52:09.768355+0000","last_active":"2024-03-14T09:52:10.486476+0000","last_peered":"2024-03-14T09:52:10.486476+0000","last_clean":"2024-03-14T09:52:10.486476+0000","last_became_active":"2024-03-14T09:52:09.767919+0000","last_became_peered":"2024-03-14T09:52:09.767919+0000","last_unstale":"2024-03-14T09:52:10.486476+0000","last_undegraded":"2024-03-14T09:52:10.486476+0000","last_fullsized":"2024-03-14T09:52:10.486476+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:45:08.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"83'2","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.714262+0000","last_change":"2024-03-14T09:52:05.230343+0000","last_active":"2024-03-14T09:52:28.714262+0000","last_peered":"2024-03-14T09:52:28.714262+0000","last_clean":"2024-03-14T09:52:28.714262+0000","last_became_active":"2024-03-14T09:52:05.229974+0000","last_became_peered":"2024-03-14T09:52:05.229974+0000","last_unstale":"2024-03-14T09:52:28.714262+0000","last_undegraded":"2024-03-14T09:52:28.714262+0000","last_fullsized":"2024-03-14T09:52:28.714262+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:21:02.458108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.751986+0000","last_change":"2024-03-14T09:52:05.738347+0000","last_active":"2024-03-14T09:52:09.751986+0000","last_peered":"2024-03-14T09:52:09.751986+0000","last_clean":"2024-03-14T09:52:09.751986+0000","last_became_active":"2024-03-14T09:52:05.737961+0000","last_became_peered":"2024-03-14T09:52:05.737961+0000","last_unstale":"2024-03-14T09:52:09.751986+0000","last_undegraded":"2024-03-14T09:52:09.751986+0000","last_fullsized":"2024-03-14T09:52:09.751986+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:02:57.252069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.968677+0000","last_change":"2024-03-14T09:52:09.761930+0000","last_active":"2024-03-14T09:52:10.968677+0000","last_peered":"2024-03-14T09:52:10.968677+0000","last_clean":"2024-03-14T09:52:10.968677+0000","last_became_active":"2024-03-14T09:52:09.761515+0000","last_became_peered":"2024-03-14T09:52:09.761515+0000","last_unstale":"2024-03-14T09:52:10.968677+0000","last_undegraded":"2024-03-14T09:52:10.968677+0000","last_fullsized":"2024-03-14T09:52:10.968677+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T15:49:39.262277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"83'3","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.715856+0000","last_change":"2024-03-14T09:52:05.233579+0000","last_active":"2024-03-14T09:52:28.715856+0000","last_peered":"2024-03-14T09:52:28.715856+0000","last_clean":"2024-03-14T09:52:28.715856+0000","last_became_active":"2024-03-14T09:52:05.231605+0000","last_became_peered":"2024-03-14T09:52:05.231605+0000","last_unstale":"2024-03-14T09:52:28.715856+0000","last_undegraded":"2024-03-14T09:52:28.715856+0000","last_fullsized":"2024-03-14T09:52:28.715856+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:55:38.282203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.747784+0000","last_change":"2024-03-14T09:52:05.736586+0000","last_active":"2024-03-14T09:52:09.747784+0000","last_peered":"2024-03-14T09:52:09.747784+0000","last_clean":"2024-03-14T09:52:09.747784+0000","last_became_active":"2024-03-14T09:52:05.736235+0000","last_became_peered":"2024-03-14T09:52:05.736235+0000","last_unstale":"2024-03-14T09:52:09.747784+0000","last_undegraded":"2024-03-14T09:52:09.747784+0000","last_fullsized":"2024-03-14T09:52:09.747784+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:46:53.293698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.486435+0000","last_change":"2024-03-14T09:52:09.768203+0000","last_active":"2024-03-14T09:52:10.486435+0000","last_peered":"2024-03-14T09:52:10.486435+0000","last_clean":"2024-03-14T09:52:10.486435+0000","last_became_active":"2024-03-14T09:52:09.767777+0000","last_became_peered":"2024-03-14T09:52:09.767777+0000","last_unstale":"2024-03-14T09:52:10.486435+0000","last_undegraded":"2024-03-14T09:52:10.486435+0000","last_fullsized":"2024-03-14T09:52:10.486435+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:20:47.598905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.757962+0000","last_change":"2024-03-14T09:52:05.232359+0000","last_active":"2024-03-14T09:52:09.757962+0000","last_peered":"2024-03-14T09:52:09.757962+0000","last_clean":"2024-03-14T09:52:09.757962+0000","last_became_active":"2024-03-14T09:52:05.231977+0000","last_became_peered":"2024-03-14T09:52:05.231977+0000","last_unstale":"2024-03-14T09:52:09.757962+0000","last_undegraded":"2024-03-14T09:52:09.757962+0000","last_fullsized":"2024-03-14T09:52:09.757962+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:32.678157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.743968+0000","last_change":"2024-03-14T09:52:05.735822+0000","last_active":"2024-03-14T09:52:09.743968+0000","last_peered":"2024-03-14T09:52:09.743968+0000","last_clean":"2024-03-14T09:52:09.743968+0000","last_became_active":"2024-03-14T09:52:05.735059+0000","last_became_peered":"2024-03-14T09:52:05.735059+0000","last_unstale":"2024-03-14T09:52:09.743968+0000","last_undegraded":"2024-03-14T09:52:09.743968+0000","last_fullsized":"2024-03-14T09:52:09.743968+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T15:22:33.033689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.486463+0000","last_change":"2024-03-14T09:52:09.759794+0000","last_active":"2024-03-14T09:52:10.486463+0000","last_peered":"2024-03-14T09:52:10.486463+0000","last_clean":"2024-03-14T09:52:10.486463+0000","last_became_active":"2024-03-14T09:52:09.759366+0000","last_became_peered":"2024-03-14T09:52:09.759366+0000","last_unstale":"2024-03-14T09:52:10.486463+0000","last_undegraded":"2024-03-14T09:52:10.486463+0000","last_fullsized":"2024-03-14T09:52:10.486463+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:52:40.641264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"83'4","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.716964+0000","last_change":"2024-03-14T09:52:05.232061+0000","last_active":"2024-03-14T09:52:28.716964+0000","last_peered":"2024-03-14T09:52:28.716964+0000","last_clean":"2024-03-14T09:52:28.716964+0000","last_became_active":"2024-03-14T09:52:05.231720+0000","last_became_peered":"2024-03-14T09:52:05.231720+0000","last_unstale":"2024-03-14T09:52:28.716964+0000","last_undegraded":"2024-03-14T09:52:28.716964+0000","last_fullsized":"2024-03-14T09:52:28.716964+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:37:59.947955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744188+0000","last_change":"2024-03-14T09:52:05.734073+0000","last_active":"2024-03-14T09:52:09.744188+0000","last_peered":"2024-03-14T09:52:09.744188+0000","last_clean":"2024-03-14T09:52:09.744188+0000","last_became_active":"2024-03-14T09:52:05.731748+0000","last_became_peered":"2024-03-14T09:52:05.731748+0000","last_unstale":"2024-03-14T09:52:09.744188+0000","last_undegraded":"2024-03-14T09:52:09.744188+0000","last_fullsized":"2024-03-14T09:52:09.744188+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:01:11.789162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.763969+0000","last_change":"2024-03-14T09:52:09.763309+0000","last_active":"2024-03-14T09:52:09.763969+0000","last_peered":"2024-03-14T09:52:09.763969+0000","last_clean":"2024-03-14T09:52:09.763969+0000","last_became_active":"2024-03-14T09:52:09.762868+0000","last_became_peered":"2024-03-14T09:52:09.762868+0000","last_unstale":"2024-03-14T09:52:09.763969+0000","last_undegraded":"2024-03-14T09:52:09.763969+0000","last_fullsized":"2024-03-14T09:52:09.763969+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:30:03.403911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"83'3","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.721134+0000","last_change":"2024-03-14T09:52:05.232310+0000","last_active":"2024-03-14T09:52:29.721134+0000","last_peered":"2024-03-14T09:52:29.721134+0000","last_clean":"2024-03-14T09:52:29.721134+0000","last_became_active":"2024-03-14T09:52:05.231931+0000","last_became_peered":"2024-03-14T09:52:05.231931+0000","last_unstale":"2024-03-14T09:52:29.721134+0000","last_undegraded":"2024-03-14T09:52:29.721134+0000","last_fullsized":"2024-03-14T09:52:29.721134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:35:44.274800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744698+0000","last_change":"2024-03-14T09:52:05.731797+0000","last_active":"2024-03-14T09:52:09.744698+0000","last_peered":"2024-03-14T09:52:09.744698+0000","last_clean":"2024-03-14T09:52:09.744698+0000","last_became_active":"2024-03-14T09:52:05.730478+0000","last_became_peered":"2024-03-14T09:52:05.730478+0000","last_unstale":"2024-03-14T09:52:09.744698+0000","last_undegraded":"2024-03-14T09:52:09.744698+0000","last_fullsized":"2024-03-14T09:52:09.744698+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:52:50.684430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.966910+0000","last_change":"2024-03-14T09:52:09.772786+0000","last_active":"2024-03-14T09:52:10.966910+0000","last_peered":"2024-03-14T09:52:10.966910+0000","last_clean":"2024-03-14T09:52:10.966910+0000","last_became_active":"2024-03-14T09:52:09.772198+0000","last_became_peered":"2024-03-14T09:52:09.772198+0000","last_unstale":"2024-03-14T09:52:10.966910+0000","last_undegraded":"2024-03-14T09:52:10.966910+0000","last_fullsized":"2024-03-14T09:52:10.966910+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:28:56.620266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"78'1","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.968653+0000","last_change":"2024-03-14T09:52:05.237657+0000","last_active":"2024-03-14T09:52:10.968653+0000","last_peered":"2024-03-14T09:52:10.968653+0000","last_clean":"2024-03-14T09:52:10.968653+0000","last_became_active":"2024-03-14T09:52:05.237381+0000","last_became_peered":"2024-03-14T09:52:05.237381+0000","last_unstale":"2024-03-14T09:52:10.968653+0000","last_undegraded":"2024-03-14T09:52:10.968653+0000","last_fullsized":"2024-03-14T09:52:10.968653+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:34:44.372128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744062+0000","last_change":"2024-03-14T09:52:05.730454+0000","last_active":"2024-03-14T09:52:09.744062+0000","last_peered":"2024-03-14T09:52:09.744062+0000","last_clean":"2024-03-14T09:52:09.744062+0000","last_became_active":"2024-03-14T09:52:05.730046+0000","last_became_peered":"2024-03-14T09:52:05.730046+0000","last_unstale":"2024-03-14T09:52:09.744062+0000","last_undegraded":"2024-03-14T09:52:09.744062+0000","last_fullsized":"2024-03-14T09:52:09.744062+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:33:19.296993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.968408+0000","last_change":"2024-03-14T09:52:09.755969+0000","last_active":"2024-03-14T09:52:10.968408+0000","last_peered":"2024-03-14T09:52:10.968408+0000","last_clean":"2024-03-14T09:52:10.968408+0000","last_became_active":"2024-03-14T09:52:09.755512+0000","last_became_peered":"2024-03-14T09:52:09.755512+0000","last_unstale":"2024-03-14T09:52:10.968408+0000","last_undegraded":"2024-03-14T09:52:10.968408+0000","last_fullsized":"2024-03-14T09:52:10.968408+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:31:03.388090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748616+0000","last_change":"2024-03-14T09:52:05.232676+0000","last_active":"2024-03-14T09:52:09.748616+0000","last_peered":"2024-03-14T09:52:09.748616+0000","last_clean":"2024-03-14T09:52:09.748616+0000","last_became_active":"2024-03-14T09:52:05.232332+0000","last_became_peered":"2024-03-14T09:52:05.232332+0000","last_unstale":"2024-03-14T09:52:09.748616+0000","last_undegraded":"2024-03-14T09:52:09.748616+0000","last_fullsized":"2024-03-14T09:52:09.748616+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:06:11.020555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744295+0000","last_change":"2024-03-14T09:52:05.733338+0000","last_active":"2024-03-14T09:52:09.744295+0000","last_peered":"2024-03-14T09:52:09.744295+0000","last_clean":"2024-03-14T09:52:09.744295+0000","last_became_active":"2024-03-14T09:52:05.732994+0000","last_became_peered":"2024-03-14T09:52:05.732994+0000","last_unstale":"2024-03-14T09:52:09.744295+0000","last_undegraded":"2024-03-14T09:52:09.744295+0000","last_fullsized":"2024-03-14T09:52:09.744295+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:31:25.085916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.747878+0000","last_change":"2024-03-14T09:52:05.733121+0000","last_active":"2024-03-14T09:52:09.747878+0000","last_peered":"2024-03-14T09:52:09.747878+0000","last_clean":"2024-03-14T09:52:09.747878+0000","last_became_active":"2024-03-14T09:52:05.732671+0000","last_became_peered":"2024-03-14T09:52:05.732671+0000","last_unstale":"2024-03-14T09:52:09.747878+0000","last_undegraded":"2024-03-14T09:52:09.747878+0000","last_fullsized":"2024-03-14T09:52:09.747878+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:18:54.180131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.751482+0000","last_change":"2024-03-14T09:52:05.736534+0000","last_active":"2024-03-14T09:52:09.751482+0000","last_peered":"2024-03-14T09:52:09.751482+0000","last_clean":"2024-03-14T09:52:09.751482+0000","last_became_active":"2024-03-14T09:52:05.736202+0000","last_became_peered":"2024-03-14T09:52:05.736202+0000","last_unstale":"2024-03-14T09:52:09.751482+0000","last_undegraded":"2024-03-14T09:52:09.751482+0000","last_fullsized":"2024-03-14T09:52:09.751482+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:22:46.161561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748730+0000","last_change":"2024-03-14T09:52:05.739434+0000","last_active":"2024-03-14T09:52:09.748730+0000","last_peered":"2024-03-14T09:52:09.748730+0000","last_clean":"2024-03-14T09:52:09.748730+0000","last_became_active":"2024-03-14T09:52:05.738023+0000","last_became_peered":"2024-03-14T09:52:05.738023+0000","last_unstale":"2024-03-14T09:52:09.748730+0000","last_undegraded":"2024-03-14T09:52:09.748730+0000","last_fullsized":"2024-03-14T09:52:09.748730+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:51:44.173645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.751960+0000","last_change":"2024-03-14T09:52:05.751367+0000","last_active":"2024-03-14T09:52:09.751960+0000","last_peered":"2024-03-14T09:52:09.751960+0000","last_clean":"2024-03-14T09:52:09.751960+0000","last_became_active":"2024-03-14T09:52:05.750816+0000","last_became_peered":"2024-03-14T09:52:05.750816+0000","last_unstale":"2024-03-14T09:52:09.751960+0000","last_undegraded":"2024-03-14T09:52:09.751960+0000","last_fullsized":"2024-03-14T09:52:09.751960+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:57:58.403356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.906111+0000","last_change":"2024-03-14T09:52:05.746652+0000","last_active":"2024-03-14T09:52:09.906111+0000","last_peered":"2024-03-14T09:52:09.906111+0000","last_clean":"2024-03-14T09:52:09.906111+0000","last_became_active":"2024-03-14T09:52:05.746214+0000","last_became_peered":"2024-03-14T09:52:05.746214+0000","last_unstale":"2024-03-14T09:52:09.906111+0000","last_undegraded":"2024-03-14T09:52:09.906111+0000","last_fullsized":"2024-03-14T09:52:09.906111+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:21:39.244617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748036+0000","last_change":"2024-03-14T09:52:05.735078+0000","last_active":"2024-03-14T09:52:09.748036+0000","last_peered":"2024-03-14T09:52:09.748036+0000","last_clean":"2024-03-14T09:52:09.748036+0000","last_became_active":"2024-03-14T09:52:05.734632+0000","last_became_peered":"2024-03-14T09:52:05.734632+0000","last_unstale":"2024-03-14T09:52:09.748036+0000","last_undegraded":"2024-03-14T09:52:09.748036+0000","last_fullsized":"2024-03-14T09:52:09.748036+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:08:45.687765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.758608+0000","last_change":"2024-03-14T09:52:05.739758+0000","last_active":"2024-03-14T09:52:09.758608+0000","last_peered":"2024-03-14T09:52:09.758608+0000","last_clean":"2024-03-14T09:52:09.758608+0000","last_became_active":"2024-03-14T09:52:05.739423+0000","last_became_peered":"2024-03-14T09:52:05.739423+0000","last_unstale":"2024-03-14T09:52:09.758608+0000","last_undegraded":"2024-03-14T09:52:09.758608+0000","last_fullsized":"2024-03-14T09:52:09.758608+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:47:22.594910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744583+0000","last_change":"2024-03-14T09:52:05.733281+0000","last_active":"2024-03-14T09:52:09.744583+0000","last_peered":"2024-03-14T09:52:09.744583+0000","last_clean":"2024-03-14T09:52:09.744583+0000","last_became_active":"2024-03-14T09:52:05.730081+0000","last_became_peered":"2024-03-14T09:52:05.730081+0000","last_unstale":"2024-03-14T09:52:09.744583+0000","last_undegraded":"2024-03-14T09:52:09.744583+0000","last_fullsized":"2024-03-14T09:52:09.744583+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:40:26.354063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.745308+0000","last_change":"2024-03-14T09:52:05.726938+0000","last_active":"2024-03-14T09:52:09.745308+0000","last_peered":"2024-03-14T09:52:09.745308+0000","last_clean":"2024-03-14T09:52:09.745308+0000","last_became_active":"2024-03-14T09:52:05.726602+0000","last_became_peered":"2024-03-14T09:52:05.726602+0000","last_unstale":"2024-03-14T09:52:09.745308+0000","last_undegraded":"2024-03-14T09:52:09.745308+0000","last_fullsized":"2024-03-14T09:52:09.745308+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:26:20.354368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.743577+0000","last_change":"2024-03-14T09:52:05.740348+0000","last_active":"2024-03-14T09:52:09.743577+0000","last_peered":"2024-03-14T09:52:09.743577+0000","last_clean":"2024-03-14T09:52:09.743577+0000","last_became_active":"2024-03-14T09:52:05.740055+0000","last_became_peered":"2024-03-14T09:52:05.740055+0000","last_unstale":"2024-03-14T09:52:09.743577+0000","last_undegraded":"2024-03-14T09:52:09.743577+0000","last_fullsized":"2024-03-14T09:52:09.743577+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:07:41.498862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.749884+0000","last_change":"2024-03-14T09:52:05.742945+0000","last_active":"2024-03-14T09:52:09.749884+0000","last_peered":"2024-03-14T09:52:09.749884+0000","last_clean":"2024-03-14T09:52:09.749884+0000","last_became_active":"2024-03-14T09:52:05.742483+0000","last_became_peered":"2024-03-14T09:52:05.742483+0000","last_unstale":"2024-03-14T09:52:09.749884+0000","last_undegraded":"2024-03-14T09:52:09.749884+0000","last_fullsized":"2024-03-14T09:52:09.749884+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:05:31.053608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.906081+0000","last_change":"2024-03-14T09:52:05.737822+0000","last_active":"2024-03-14T09:52:09.906081+0000","last_peered":"2024-03-14T09:52:09.906081+0000","last_clean":"2024-03-14T09:52:09.906081+0000","last_became_active":"2024-03-14T09:52:05.735294+0000","last_became_peered":"2024-03-14T09:52:05.735294+0000","last_unstale":"2024-03-14T09:52:09.906081+0000","last_undegraded":"2024-03-14T09:52:09.906081+0000","last_fullsized":"2024-03-14T09:52:09.906081+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:06:28.437259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748615+0000","last_change":"2024-03-14T09:52:05.746991+0000","last_active":"2024-03-14T09:52:09.748615+0000","last_peered":"2024-03-14T09:52:09.748615+0000","last_clean":"2024-03-14T09:52:09.748615+0000","last_became_active":"2024-03-14T09:52:05.746558+0000","last_became_peered":"2024-03-14T09:52:05.746558+0000","last_unstale":"2024-03-14T09:52:09.748615+0000","last_undegraded":"2024-03-14T09:52:09.748615+0000","last_fullsized":"2024-03-14T09:52:09.748615+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:46:11.540134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247929,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7219,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":14,"num_read_kb":12,"num_write":117,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":57613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579919,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":447276,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.379}]},{"osd":1,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.57999999999999996}]},{"osd":2,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.35899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.497}]},{"osd":6,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.61799999999999999}]},{"osd":7,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.71599999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":9,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.66400000000000003}]},{"osd":10,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.70499999999999996}]}]},{"osd":10,"up_from":67,"seq":287762808849,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":1041051,"data_compressed":47579,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":1,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.443}]},{"osd":2,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.51300000000000001}]},{"osd":3,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.73999999999999999}]},{"osd":4,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.54200000000000004}]},{"osd":5,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.755}]},{"osd":6,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.876},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.81699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":8,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.78200000000000003}]},{"osd":9,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.85699999999999998}]},{"osd":11,"last update":"Thu Mar 14 09:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.77300000000000002}]}]},{"osd":9,"up_from":62,"seq":266287972375,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":921600,"data_stored":451608,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.624}]},{"osd":2,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.48199999999999998}]},{"osd":3,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.40699999999999997}]},{"osd":4,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.60699999999999998}]},{"osd":5,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.66800000000000004}]},{"osd":6,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.63300000000000001}]},{"osd":8,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.72899999999999998}]},{"osd":10,"last update":"Thu Mar 14 09:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.752},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.75900000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]}]},{"osd":8,"up_from":55,"seq":236223201305,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":447101,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":9.7769999999999992,"5min":9.7769999999999992,"15min":9.7769999999999992},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.64600000000000002}]},{"osd":2,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.53300000000000003}]},{"osd":3,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.435},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.68400000000000005}]},{"osd":4,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.56499999999999995}]},{"osd":5,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.40999999999999998}]},{"osd":6,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.69799999999999995}]},{"osd":7,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.628},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.61699999999999999}]},{"osd":9,"last update":"Thu Mar 14 09:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.58199999999999996}]},{"osd":10,"last update":"Thu Mar 14 09:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.67200000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]}]},{"osd":7,"up_from":48,"seq":206158430238,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":1039150,"data_compressed":47579,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.40999999999999998}]},{"osd":1,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.42699999999999999,"15min":0.41999999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.41699999999999998,"15min":0.40899999999999997},"min":{"1min":0.29199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.57699999999999996,"15min":0.56799999999999995},"min":{"1min":0.441,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.69499999999999995}]},{"osd":3,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.42199999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.47499999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.55700000000000005}]},{"osd":4,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.48799999999999999,"15min":0.47799999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.503,"15min":0.497},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.44500000000000001}]},{"osd":5,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.58799999999999997,"15min":0.57599999999999996},"min":{"1min":0.47799999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.71699999999999997}]},{"osd":6,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.64300000000000002,"15min":0.63500000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66500000000000004,"15min":0.65900000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.56999999999999995}]},{"osd":8,"last update":"Thu Mar 14 09:51:33 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.65500000000000003}]},{"osd":9,"last update":"Thu Mar 14 09:51:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.68100000000000005}]},{"osd":10,"last update":"Thu Mar 14 09:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.73899999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]}]},{"osd":6,"up_from":43,"seq":184683593760,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":446694,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.47399999999999998,"15min":0.46999999999999997},"min":{"1min":0.40000000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.76400000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.48299999999999998},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.66100000000000003,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.39300000000000002}]},{"osd":1,"last update":"Thu Mar 14 09:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.65300000000000002,"15min":0.64100000000000001},"min":{"1min":0.53400000000000003,"5min":0.433,"15min":0.433},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.60999999999999999,"15min":0.59999999999999998},"min":{"1min":0.46500000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81799999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.627}]},{"osd":2,"last update":"Thu Mar 14 09:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63200000000000001,"15min":0.63},"min":{"1min":0.49399999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.76900000000000002,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.67300000000000004,"15min":0.66900000000000004},"min":{"1min":0.48599999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.89400000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.80400000000000005}]},{"osd":3,"last update":"Thu Mar 14 09:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.46500000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.83699999999999997,"5min":0.874,"15min":0.874},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.41599999999999998}]},{"osd":4,"last update":"Thu Mar 14 09:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.64600000000000002,"15min":0.63900000000000001},"min":{"1min":0.54300000000000004,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.877,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.72199999999999998,"15min":0.71399999999999997},"min":{"1min":0.56699999999999995,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.78100000000000003}]},{"osd":5,"last update":"Thu Mar 14 09:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.76300000000000001,"15min":0.77500000000000002},"min":{"1min":0.54100000000000004,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.84499999999999997,"5min":3.1059999999999999,"15min":3.1059999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.70599999999999996,"15min":0.70799999999999996},"min":{"1min":0.503,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.85399999999999998,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.54500000000000004}]},{"osd":7,"last update":"Thu Mar 14 09:52:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.69299999999999995,"15min":0.68700000000000006},"min":{"1min":0.51100000000000001,"5min":0.495,"15min":0.495},"max":{"1min":0.90200000000000002,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.77300000000000002,"15min":0.77100000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.82399999999999995}]},{"osd":8,"last update":"Thu Mar 14 09:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.58899999999999997}]},{"osd":9,"last update":"Thu Mar 14 09:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.56299999999999994}]},{"osd":10,"last update":"Thu Mar 14 09:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.66400000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":909312,"data_stored":450489,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54700000000000004,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.315,"15min":0.315},"max":{"1min":0.74099999999999999,"5min":1.917,"15min":1.917},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.40500000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.746,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.63700000000000001}]},{"osd":2,"last update":"Thu Mar 14 09:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.42199999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69899999999999995,"5min":1.885,"15min":1.885},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.52100000000000002,"15min":0.51900000000000002},"min":{"1min":0.39200000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76500000000000001,"5min":1.167,"15min":1.167},"last":0.748}]},{"osd":3,"last update":"Thu Mar 14 09:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.39300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76800000000000002,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.75800000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.60999999999999999}]},{"osd":4,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.378,"15min":0.36399999999999999},"min":{"1min":0.31,"5min":0.249,"15min":0.249},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.437},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.42999999999999999,"15min":0.40000000000000002},"min":{"1min":0.316,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":2.1339999999999999,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":0.45800000000000002}]},{"osd":5,"last update":"Thu Mar 14 09:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.48199999999999998,"15min":0.47099999999999997},"min":{"1min":0.40500000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.42799999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.79300000000000004}]},{"osd":6,"last update":"Thu Mar 14 09:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63300000000000001,"15min":0.627},"min":{"1min":0.50900000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.83099999999999996,"15min":0.86899999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77800000000000002,"5min":5.5229999999999997,"15min":5.5229999999999997},"last":0.48599999999999999}]},{"osd":7,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.51100000000000001,"15min":0.48899999999999999},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.49199999999999999,"15min":0.48499999999999999},"min":{"1min":0.33300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.53200000000000003}]},{"osd":8,"last update":"Thu Mar 14 09:51:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.81599999999999995}]},{"osd":9,"last update":"Thu Mar 14 09:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.72399999999999998}]},{"osd":10,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.871}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":448172,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 14 09:52:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.56699999999999995,"15min":0.52900000000000003},"min":{"1min":0.34999999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.58899999999999997,"15min":0.54900000000000004},"min":{"1min":0.28199999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":0.71599999999999997}]},{"osd":2,"last update":"Thu Mar 14 09:51:43 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.58699999999999997,"15min":0.54300000000000004},"min":{"1min":0.45300000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":3.7989999999999999,"5min":3.7989999999999999,"15min":3.7989999999999999},"last":0.63},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.59399999999999997,"15min":0.55800000000000005},"min":{"1min":0.45000000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93899999999999995,"5min":1.04,"15min":1.04},"last":0.753}]},{"osd":3,"last update":"Thu Mar 14 09:51:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.31,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.87,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.495},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.48499999999999999,"15min":0.47799999999999998},"min":{"1min":0.36799999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71099999999999997,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.58599999999999997}]},{"osd":4,"last update":"Thu Mar 14 09:52:22 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.71399999999999997,"15min":0.66600000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.8220000000000001,"5min":3.8220000000000001,"15min":3.8220000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.61499999999999999,"15min":0.58299999999999996},"min":{"1min":0.38900000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.573,"5min":3.573,"15min":3.573},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Mar 14 09:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.70099999999999996,"15min":0.66600000000000004},"min":{"1min":0.54800000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":3.863,"5min":3.863,"15min":3.863},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.67300000000000004,"15min":0.66900000000000004},"min":{"1min":0.48799999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.73199999999999998}]},{"osd":6,"last update":"Thu Mar 14 09:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59599999999999997,"15min":0.58999999999999997},"min":{"1min":0.39900000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.107,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.748,"5min":0.71299999999999997,"15min":0.70699999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.0940000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.81499999999999995}]},{"osd":7,"last update":"Thu Mar 14 09:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.7509999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.67600000000000005}]},{"osd":8,"last update":"Thu Mar 14 09:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.872},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.76100000000000001}]},{"osd":9,"last update":"Thu Mar 14 09:51:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.745},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.93000000000000005}]},{"osd":10,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.76900000000000002}]},{"osd":11,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":2.5590000000000002,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":3.71,"5min":3.71,"15min":3.71},"last":0.82999999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":471091,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.52000000000000002,"15min":0.51400000000000001},"min":{"1min":0.39100000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.76500000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.35199999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.74299999999999999,"5min":0.879,"15min":0.879},"last":0.57599999999999996}]},{"osd":1,"last update":"Thu Mar 14 09:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.52800000000000002,"15min":0.51600000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.74199999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80600000000000005,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.64700000000000002}]},{"osd":3,"last update":"Thu Mar 14 09:52:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53100000000000003,"15min":0.51900000000000002},"min":{"1min":0.442,"5min":0.37,"15min":0.37},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.627},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55500000000000005,"15min":0.54400000000000004},"min":{"1min":0.41999999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78900000000000003,"5min":0.871,"15min":0.871},"last":0.77500000000000002}]},{"osd":4,"last update":"Thu Mar 14 09:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.73199999999999998,"5min":0.87,"15min":0.87},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.68300000000000005}]},{"osd":5,"last update":"Thu Mar 14 09:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.44500000000000001,"15min":0.437},"min":{"1min":0.32600000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.47399999999999998,"15min":0.47199999999999998},"min":{"1min":0.32900000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.72699999999999998,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.72599999999999998}]},{"osd":6,"last update":"Thu Mar 14 09:52:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.52900000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.83899999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65900000000000003,"15min":0.65700000000000003},"min":{"1min":0.46700000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.86599999999999999}]},{"osd":7,"last update":"Thu Mar 14 09:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.755},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.96599999999999997}]},{"osd":8,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.84299999999999997}]},{"osd":9,"last update":"Thu Mar 14 09:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.995}]},{"osd":10,"last update":"Thu Mar 14 09:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.88300000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]}]},{"osd":3,"up_from":24,"seq":103079215150,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":446978,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.40600000000000003,"15min":0.39600000000000002},"min":{"1min":0.34000000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.372},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42799999999999999,"15min":0.42399999999999999},"min":{"1min":0.33400000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.64500000000000002,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Mar 14 09:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.60599999999999998,"15min":0.58999999999999997},"min":{"1min":0.46100000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.60199999999999998,"15min":0.58299999999999996},"min":{"1min":0.46500000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Mar 14 09:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.61799999999999999,"15min":0.60199999999999998},"min":{"1min":0.46400000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.63500000000000001,"15min":0.621},"min":{"1min":0.47899999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.71099999999999997}]},{"osd":4,"last update":"Thu Mar 14 09:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.61799999999999999,"15min":0.59599999999999997},"min":{"1min":0.42099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.61399999999999999,"15min":0.59899999999999998},"min":{"1min":0.48199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.68799999999999994}]},{"osd":5,"last update":"Thu Mar 14 09:51:36 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.68600000000000005,"15min":0.68200000000000005},"min":{"1min":0.48099999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69199999999999995,"15min":0.69099999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.04,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.76500000000000001}]},{"osd":6,"last update":"Thu Mar 14 09:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.65900000000000003,"15min":0.65400000000000003},"min":{"1min":0.42999999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96599999999999997,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.65500000000000003,"15min":0.64200000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.754}]},{"osd":7,"last update":"Thu Mar 14 09:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.53500000000000003,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.109,"5min":1.123,"15min":1.123},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77000000000000002,"15min":0.76900000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.64500000000000002}]},{"osd":8,"last update":"Thu Mar 14 09:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.871}]},{"osd":9,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.80600000000000005}]},{"osd":10,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.77200000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986217,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":909312,"data_stored":447290,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.40100000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71599999999999997,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.52300000000000002,"15min":0.51600000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.35099999999999998},"min":{"1min":0.26600000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.29199999999999998}]},{"osd":2,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.55900000000000005,"15min":0.53600000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53100000000000003,"15min":0.51500000000000001},"min":{"1min":0.433,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.40699999999999997,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76500000000000001,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55400000000000005,"15min":0.55700000000000005},"min":{"1min":0.39800000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.46100000000000002}]},{"osd":5,"last update":"Thu Mar 14 09:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.51700000000000002,"15min":0.50700000000000001},"min":{"1min":0.34799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.52500000000000002,"15min":0.52300000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.66600000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Mar 14 09:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.57799999999999996,"15min":0.57099999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.438,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.65000000000000002}]},{"osd":7,"last update":"Thu Mar 14 09:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.379,"5min":0.318,"15min":0.318},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.45600000000000002,"15min":0.45000000000000001},"min":{"1min":0.33200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.36599999999999999}]},{"osd":8,"last update":"Thu Mar 14 09:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.629},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.61699999999999999}]},{"osd":9,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.505},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.67200000000000004}]},{"osd":10,"last update":"Thu Mar 14 09:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.58699999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]},{"osd":5,"up_from":36,"seq":154618822694,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":1042586,"data_compressed":47579,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52600000000000002,"15min":0.52400000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73999999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54900000000000004,"15min":0.54500000000000004},"min":{"1min":0.39200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72799999999999998,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.52200000000000002,"15min":0.51000000000000001},"min":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Mar 14 09:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.38500000000000001,"15min":0.375},"min":{"1min":0.27200000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38400000000000001,"15min":0.376},"min":{"1min":0.28999999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.41599999999999998}]},{"osd":3,"last update":"Thu Mar 14 09:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42399999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58399999999999996,"15min":0.58099999999999996},"min":{"1min":0.44700000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.79900000000000004,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.51300000000000001}]},{"osd":4,"last update":"Thu Mar 14 09:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.55600000000000005,"15min":0.54800000000000004},"min":{"1min":0.41999999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.54100000000000004,"15min":0.53100000000000003},"min":{"1min":0.36299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67700000000000005}]},{"osd":6,"last update":"Thu Mar 14 09:51:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.626,"15min":0.625},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82199999999999995,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.629,"15min":0.623},"min":{"1min":0.47799999999999998,"5min":0.432,"15min":0.432},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.47499999999999998}]},{"osd":7,"last update":"Thu Mar 14 09:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.60799999999999998,"15min":0.59299999999999997},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.59899999999999998,"15min":0.58499999999999996},"min":{"1min":0.46400000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.66800000000000004}]},{"osd":8,"last update":"Thu Mar 14 09:51:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.38100000000000001}]},{"osd":9,"last update":"Thu Mar 14 09:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.63900000000000001}]},{"osd":10,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.71299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34227,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34227,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":41727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-14T09:52:34.545 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-14T09:52:34.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.770+0000 7fd7a7000700 1 -- 172.21.15.78:0/2987420436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a01027e0 msgr2=0x7fd7a0109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:34.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.770+0000 7fd7a7000700 1 --2- 172.21.15.78:0/2987420436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a01027e0 0x7fd7a0109090 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd798009fc0 tx=0x7fd79800b330 comp rx=0 tx=0).stop 2024-03-14T09:52:34.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.770+0000 7fd7a7000700 1 -- 172.21.15.78:0/2987420436 shutdown_connections 2024-03-14T09:52:34.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.770+0000 7fd7a7000700 1 --2- 172.21.15.78:0/2987420436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7a0100c50 0x7fd7a0101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.770+0000 7fd7a7000700 1 --2- 172.21.15.78:0/2987420436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a01027e0 0x7fd7a0109090 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.770+0000 7fd7a7000700 1 --2- 172.21.15.78:0/2987420436 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7a0101e40 0x7fd7a01022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.770+0000 7fd7a7000700 1 -- 172.21.15.78:0/2987420436 >> 172.21.15.78:0/2987420436 conn(0x7fd7a00fc1c0 msgr2=0x7fd7a00fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:34.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.770+0000 7fd7a7000700 1 -- 172.21.15.78:0/2987420436 shutdown_connections 2024-03-14T09:52:34.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.770+0000 7fd7a7000700 1 -- 172.21.15.78:0/2987420436 wait complete. 2024-03-14T09:52:34.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.771+0000 7fd7a7000700 1 Processor -- start 2024-03-14T09:52:34.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.771+0000 7fd7a7000700 1 -- start start 2024-03-14T09:52:34.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.772+0000 7fd7a7000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a0100c50 0x7fd7a01992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:34.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.772+0000 7fd7a7000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7a0101e40 0x7fd7a0199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:34.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.772+0000 7fd7a7000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7a01027e0 0x7fd7a01a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:34.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.772+0000 7fd7a7000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a01a0e10 con 0x7fd7a0100c50 2024-03-14T09:52:34.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.772+0000 7fd7a7000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a01a0f50 con 0x7fd7a01027e0 2024-03-14T09:52:34.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.772+0000 7fd7a7000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a01a10e0 con 0x7fd7a0101e40 2024-03-14T09:52:34.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.772+0000 7fd79fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a0100c50 0x7fd7a01992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:34.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.772+0000 7fd79fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a0100c50 0x7fd7a01992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50472/0 (socket says 172.21.15.78:50472) 2024-03-14T09:52:34.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.772+0000 7fd79fe00700 1 -- 172.21.15.78:0/1068012631 learned_addr learned my addr 172.21.15.78:0/1068012631 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:34.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.772+0000 7fd79f400700 1 --2- 172.21.15.78:0/1068012631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7a0101e40 0x7fd7a0199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:34.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.772+0000 7fd7a5000700 1 --2- 172.21.15.78:0/1068012631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7a01027e0 0x7fd7a01a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:34.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.772+0000 7fd79fe00700 1 -- 172.21.15.78:0/1068012631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7a0101e40 msgr2=0x7fd7a0199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:34.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.772+0000 7fd79fe00700 1 --2- 172.21.15.78:0/1068012631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7a0101e40 0x7fd7a0199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.772+0000 7fd79fe00700 1 -- 172.21.15.78:0/1068012631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7a01027e0 msgr2=0x7fd7a01a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:34.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.772+0000 7fd79fe00700 1 --2- 172.21.15.78:0/1068012631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7a01027e0 0x7fd7a01a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:34.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.772+0000 7fd79fe00700 1 -- 172.21.15.78:0/1068012631 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd798009c40 con 0x7fd7a0100c50 2024-03-14T09:52:34.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.773+0000 7fd79fe00700 1 --2- 172.21.15.78:0/1068012631 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a0100c50 0x7fd7a01992e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd79000e970 tx=0x7fd79000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:34.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.773+0000 7fd79cc00700 1 -- 172.21.15.78:0/1068012631 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790009790 con 0x7fd7a0100c50 2024-03-14T09:52:34.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.773+0000 7fd79cc00700 1 -- 172.21.15.78:0/1068012631 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd79000eeb0 con 0x7fd7a0100c50 2024-03-14T09:52:34.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.773+0000 7fd79cc00700 1 -- 172.21.15.78:0/1068012631 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790003d20 con 0x7fd7a0100c50 2024-03-14T09:52:34.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.773+0000 7fd7a7000700 1 -- 172.21.15.78:0/1068012631 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a01a13c0 con 0x7fd7a0100c50 2024-03-14T09:52:34.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.773+0000 7fd7a7000700 1 -- 172.21.15.78:0/1068012631 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a01a18d0 con 0x7fd7a0100c50 2024-03-14T09:52:34.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.774+0000 7fd7a7000700 1 -- 172.21.15.78:0/1068012631 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a00679e0 con 0x7fd7a0100c50 2024-03-14T09:52:34.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.776+0000 7fd79cc00700 1 -- 172.21.15.78:0/1068012631 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fd790003e80 con 0x7fd7a0100c50 2024-03-14T09:52:34.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.777+0000 7fd79cc00700 1 --2- 172.21.15.78:0/1068012631 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd7840a0ba0 0x7fd7840a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:34.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.777+0000 7fd79cc00700 1 -- 172.21.15.78:0/1068012631 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd790014070 con 0x7fd7a0100c50 2024-03-14T09:52:34.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.777+0000 7fd79f400700 1 --2- 172.21.15.78:0/1068012631 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd7840a0ba0 0x7fd7840a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:34.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.778+0000 7fd79f400700 1 --2- 172.21.15.78:0/1068012631 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd7840a0ba0 0x7fd7840a3060 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd78c005fd0 tx=0x7fd78c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:34.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.781+0000 7fd79cc00700 1 -- 172.21.15.78:0/1068012631 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd790095c00 con 0x7fd7a0100c50 2024-03-14T09:52:34.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:34 smithi078 ceph-mon[38143]: mgrmap e17: x(active, since 5m), standbys: z, y 2024-03-14T09:52:34.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:34 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3656041553' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:34.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:34 smithi078 ceph-mon[38143]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:52:34.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:34.995+0000 7fd7a7000700 1 -- 172.21.15.78:0/1068012631 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd7a0074700 con 0x7fd7840a0ba0 2024-03-14T09:52:34.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:34 smithi152 ceph-mon[48279]: mgrmap e17: x(active, since 5m), standbys: z, y 2024-03-14T09:52:34.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:34 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3656041553' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:34.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:34 smithi152 ceph-mon[48279]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:52:35.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:35.008+0000 7fd79cc00700 1 -- 172.21.15.78:0/1068012631 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422671 (secure 0 0 0) 0x7fd7a0074700 con 0x7fd7840a0ba0 2024-03-14T09:52:35.015 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:35.013+0000 7fd7a7000700 1 -- 172.21.15.78:0/1068012631 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd7840a0ba0 msgr2=0x7fd7840a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:35.013+0000 7fd7a7000700 1 --2- 172.21.15.78:0/1068012631 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd7840a0ba0 0x7fd7840a3060 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd78c005fd0 tx=0x7fd78c009310 comp rx=0 tx=0).stop 2024-03-14T09:52:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:35.013+0000 7fd7a7000700 1 -- 172.21.15.78:0/1068012631 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a0100c50 msgr2=0x7fd7a01992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:35.013+0000 7fd7a7000700 1 --2- 172.21.15.78:0/1068012631 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a0100c50 0x7fd7a01992e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd79000e970 tx=0x7fd79000ed30 comp rx=0 tx=0).stop 2024-03-14T09:52:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:35.013+0000 7fd7a7000700 1 -- 172.21.15.78:0/1068012631 shutdown_connections 2024-03-14T09:52:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:35.013+0000 7fd7a7000700 1 --2- 172.21.15.78:0/1068012631 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd7840a0ba0 0x7fd7840a3060 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:35.013+0000 7fd7a7000700 1 --2- 172.21.15.78:0/1068012631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd7a01027e0 0x7fd7a01a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:35.013+0000 7fd7a7000700 1 --2- 172.21.15.78:0/1068012631 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7a0100c50 0x7fd7a01992e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:35.013+0000 7fd7a7000700 1 --2- 172.21.15.78:0/1068012631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7a0101e40 0x7fd7a0199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:35.013+0000 7fd7a7000700 1 -- 172.21.15.78:0/1068012631 >> 172.21.15.78:0/1068012631 conn(0x7fd7a00fc1c0 msgr2=0x7fd7a0072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:35.014+0000 7fd7a7000700 1 -- 172.21.15.78:0/1068012631 shutdown_connections 2024-03-14T09:52:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:35.014+0000 7fd7a7000700 1 -- 172.21.15.78:0/1068012631 wait complete. 2024-03-14T09:52:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-14T09:52:35.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:34 smithi132 ceph-mon[47959]: mgrmap e17: x(active, since 5m), standbys: z, y 2024-03-14T09:52:35.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:34 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3656041553' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:35.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:34 smithi132 ceph-mon[47959]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:52:35.032 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-03-14T09:52:33.072198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597587,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":150,"num_read_kb":131,"num_write":298,"num_write_kb":2980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":241,"ondisk_log_size":241,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":335088,"kb_used_data":10896,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124606736,"statfs":{"total":1151940427776,"available":1151597297664,"internally_reserved":0,"allocated":11157504,"data_stored":7179486,"data_compressed":247893,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":19131,"internal_metadata":331855173},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4407,"num_objects":32,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":14,"num_read_kb":12,"num_write":71,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003427"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.749056+0000","last_change":"2024-03-14T09:52:05.737617+0000","last_active":"2024-03-14T09:52:09.749056+0000","last_peered":"2024-03-14T09:52:09.749056+0000","last_clean":"2024-03-14T09:52:09.749056+0000","last_became_active":"2024-03-14T09:52:05.736796+0000","last_became_peered":"2024-03-14T09:52:05.736796+0000","last_unstale":"2024-03-14T09:52:09.749056+0000","last_undegraded":"2024-03-14T09:52:09.749056+0000","last_fullsized":"2024-03-14T09:52:09.749056+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:47:52.437748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.756225+0000","last_change":"2024-03-14T09:52:05.749474+0000","last_active":"2024-03-14T09:52:09.756225+0000","last_peered":"2024-03-14T09:52:09.756225+0000","last_clean":"2024-03-14T09:52:09.756225+0000","last_became_active":"2024-03-14T09:52:05.748927+0000","last_became_peered":"2024-03-14T09:52:05.748927+0000","last_unstale":"2024-03-14T09:52:09.756225+0000","last_undegraded":"2024-03-14T09:52:09.756225+0000","last_fullsized":"2024-03-14T09:52:09.756225+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:28:15.852021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.753505+0000","last_change":"2024-03-14T09:52:05.745098+0000","last_active":"2024-03-14T09:52:09.753505+0000","last_peered":"2024-03-14T09:52:09.753505+0000","last_clean":"2024-03-14T09:52:09.753505+0000","last_became_active":"2024-03-14T09:52:05.744755+0000","last_became_peered":"2024-03-14T09:52:05.744755+0000","last_unstale":"2024-03-14T09:52:09.753505+0000","last_undegraded":"2024-03-14T09:52:09.753505+0000","last_fullsized":"2024-03-14T09:52:09.753505+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:35:41.932386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748570+0000","last_change":"2024-03-14T09:52:05.737429+0000","last_active":"2024-03-14T09:52:09.748570+0000","last_peered":"2024-03-14T09:52:09.748570+0000","last_clean":"2024-03-14T09:52:09.748570+0000","last_became_active":"2024-03-14T09:52:05.736516+0000","last_became_peered":"2024-03-14T09:52:05.736516+0000","last_unstale":"2024-03-14T09:52:09.748570+0000","last_undegraded":"2024-03-14T09:52:09.748570+0000","last_fullsized":"2024-03-14T09:52:09.748570+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:48:38.158783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.752188+0000","last_change":"2024-03-14T09:52:05.729634+0000","last_active":"2024-03-14T09:52:09.752188+0000","last_peered":"2024-03-14T09:52:09.752188+0000","last_clean":"2024-03-14T09:52:09.752188+0000","last_became_active":"2024-03-14T09:52:05.727889+0000","last_became_peered":"2024-03-14T09:52:05.727889+0000","last_unstale":"2024-03-14T09:52:09.752188+0000","last_undegraded":"2024-03-14T09:52:09.752188+0000","last_fullsized":"2024-03-14T09:52:09.752188+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:01:13.216763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.755590+0000","last_change":"2024-03-14T09:52:05.728270+0000","last_active":"2024-03-14T09:52:09.755590+0000","last_peered":"2024-03-14T09:52:09.755590+0000","last_clean":"2024-03-14T09:52:09.755590+0000","last_became_active":"2024-03-14T09:52:05.727915+0000","last_became_peered":"2024-03-14T09:52:05.727915+0000","last_unstale":"2024-03-14T09:52:09.755590+0000","last_undegraded":"2024-03-14T09:52:09.755590+0000","last_fullsized":"2024-03-14T09:52:09.755590+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:22:22.616901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744072+0000","last_change":"2024-03-14T09:52:05.727365+0000","last_active":"2024-03-14T09:52:09.744072+0000","last_peered":"2024-03-14T09:52:09.744072+0000","last_clean":"2024-03-14T09:52:09.744072+0000","last_became_active":"2024-03-14T09:52:05.726984+0000","last_became_peered":"2024-03-14T09:52:05.726984+0000","last_unstale":"2024-03-14T09:52:09.744072+0000","last_undegraded":"2024-03-14T09:52:09.744072+0000","last_fullsized":"2024-03-14T09:52:09.744072+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:42:57.129718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.743534+0000","last_change":"2024-03-14T09:52:05.729622+0000","last_active":"2024-03-14T09:52:09.743534+0000","last_peered":"2024-03-14T09:52:09.743534+0000","last_clean":"2024-03-14T09:52:09.743534+0000","last_became_active":"2024-03-14T09:52:05.729293+0000","last_became_peered":"2024-03-14T09:52:05.729293+0000","last_unstale":"2024-03-14T09:52:09.743534+0000","last_undegraded":"2024-03-14T09:52:09.743534+0000","last_fullsized":"2024-03-14T09:52:09.743534+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:13:20.875857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.747564+0000","last_change":"2024-03-14T09:52:05.741620+0000","last_active":"2024-03-14T09:52:09.747564+0000","last_peered":"2024-03-14T09:52:09.747564+0000","last_clean":"2024-03-14T09:52:09.747564+0000","last_became_active":"2024-03-14T09:52:05.734079+0000","last_became_peered":"2024-03-14T09:52:05.734079+0000","last_unstale":"2024-03-14T09:52:09.747564+0000","last_undegraded":"2024-03-14T09:52:09.747564+0000","last_fullsized":"2024-03-14T09:52:09.747564+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:02:35.529671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.755870+0000","last_change":"2024-03-14T09:52:05.743424+0000","last_active":"2024-03-14T09:52:09.755870+0000","last_peered":"2024-03-14T09:52:09.755870+0000","last_clean":"2024-03-14T09:52:09.755870+0000","last_became_active":"2024-03-14T09:52:05.743100+0000","last_became_peered":"2024-03-14T09:52:05.743100+0000","last_unstale":"2024-03-14T09:52:09.755870+0000","last_undegraded":"2024-03-14T09:52:09.755870+0000","last_fullsized":"2024-03-14T09:52:09.755870+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:32:33.492163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744618+0000","last_change":"2024-03-14T09:52:05.735654+0000","last_active":"2024-03-14T09:52:09.744618+0000","last_peered":"2024-03-14T09:52:09.744618+0000","last_clean":"2024-03-14T09:52:09.744618+0000","last_became_active":"2024-03-14T09:52:05.735371+0000","last_became_peered":"2024-03-14T09:52:05.735371+0000","last_unstale":"2024-03-14T09:52:09.744618+0000","last_undegraded":"2024-03-14T09:52:09.744618+0000","last_fullsized":"2024-03-14T09:52:09.744618+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:19:50.011554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748484+0000","last_change":"2024-03-14T09:52:05.737322+0000","last_active":"2024-03-14T09:52:09.748484+0000","last_peered":"2024-03-14T09:52:09.748484+0000","last_clean":"2024-03-14T09:52:09.748484+0000","last_became_active":"2024-03-14T09:52:05.736834+0000","last_became_peered":"2024-03-14T09:52:05.736834+0000","last_unstale":"2024-03-14T09:52:09.748484+0000","last_undegraded":"2024-03-14T09:52:09.748484+0000","last_fullsized":"2024-03-14T09:52:09.748484+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:11:04.601826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.906071+0000","last_change":"2024-03-14T09:52:05.740209+0000","last_active":"2024-03-14T09:52:09.906071+0000","last_peered":"2024-03-14T09:52:09.906071+0000","last_clean":"2024-03-14T09:52:09.906071+0000","last_became_active":"2024-03-14T09:52:05.732949+0000","last_became_peered":"2024-03-14T09:52:05.732949+0000","last_unstale":"2024-03-14T09:52:09.906071+0000","last_undegraded":"2024-03-14T09:52:09.906071+0000","last_fullsized":"2024-03-14T09:52:09.906071+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:15:56.377569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.754711+0000","last_change":"2024-03-14T09:52:05.748795+0000","last_active":"2024-03-14T09:52:09.754711+0000","last_peered":"2024-03-14T09:52:09.754711+0000","last_clean":"2024-03-14T09:52:09.754711+0000","last_became_active":"2024-03-14T09:52:05.746107+0000","last_became_peered":"2024-03-14T09:52:05.746107+0000","last_unstale":"2024-03-14T09:52:09.754711+0000","last_undegraded":"2024-03-14T09:52:09.754711+0000","last_fullsized":"2024-03-14T09:52:09.754711+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:49:07.932854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.756958+0000","last_change":"2024-03-14T09:52:05.744174+0000","last_active":"2024-03-14T09:52:09.756958+0000","last_peered":"2024-03-14T09:52:09.756958+0000","last_clean":"2024-03-14T09:52:09.756958+0000","last_became_active":"2024-03-14T09:52:05.743848+0000","last_became_peered":"2024-03-14T09:52:05.743848+0000","last_unstale":"2024-03-14T09:52:09.756958+0000","last_undegraded":"2024-03-14T09:52:09.756958+0000","last_fullsized":"2024-03-14T09:52:09.756958+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:07:23.260734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748597+0000","last_change":"2024-03-14T09:52:05.733677+0000","last_active":"2024-03-14T09:52:09.748597+0000","last_peered":"2024-03-14T09:52:09.748597+0000","last_clean":"2024-03-14T09:52:09.748597+0000","last_became_active":"2024-03-14T09:52:05.730967+0000","last_became_peered":"2024-03-14T09:52:05.730967+0000","last_unstale":"2024-03-14T09:52:09.748597+0000","last_undegraded":"2024-03-14T09:52:09.748597+0000","last_fullsized":"2024-03-14T09:52:09.748597+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:33:53.484741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748820+0000","last_change":"2024-03-14T09:52:05.737836+0000","last_active":"2024-03-14T09:52:09.748820+0000","last_peered":"2024-03-14T09:52:09.748820+0000","last_clean":"2024-03-14T09:52:09.748820+0000","last_became_active":"2024-03-14T09:52:05.737257+0000","last_became_peered":"2024-03-14T09:52:05.737257+0000","last_unstale":"2024-03-14T09:52:09.748820+0000","last_undegraded":"2024-03-14T09:52:09.748820+0000","last_fullsized":"2024-03-14T09:52:09.748820+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:53.723415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.766338+0000","last_change":"2024-03-14T09:52:05.741360+0000","last_active":"2024-03-14T09:52:09.766338+0000","last_peered":"2024-03-14T09:52:09.766338+0000","last_clean":"2024-03-14T09:52:09.766338+0000","last_became_active":"2024-03-14T09:52:05.740673+0000","last_became_peered":"2024-03-14T09:52:05.740673+0000","last_unstale":"2024-03-14T09:52:09.766338+0000","last_undegraded":"2024-03-14T09:52:09.766338+0000","last_fullsized":"2024-03-14T09:52:09.766338+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:13:41.561338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.758671+0000","last_change":"2024-03-14T09:52:05.739134+0000","last_active":"2024-03-14T09:52:09.758671+0000","last_peered":"2024-03-14T09:52:09.758671+0000","last_clean":"2024-03-14T09:52:09.758671+0000","last_became_active":"2024-03-14T09:52:05.738439+0000","last_became_peered":"2024-03-14T09:52:05.738439+0000","last_unstale":"2024-03-14T09:52:09.758671+0000","last_undegraded":"2024-03-14T09:52:09.758671+0000","last_fullsized":"2024-03-14T09:52:09.758671+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:05:36.997462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.250705+0000","last_change":"2024-03-14T09:52:09.769284+0000","last_active":"2024-03-14T09:52:10.250705+0000","last_peered":"2024-03-14T09:52:10.250705+0000","last_clean":"2024-03-14T09:52:10.250705+0000","last_became_active":"2024-03-14T09:52:09.768808+0000","last_became_peered":"2024-03-14T09:52:09.768808+0000","last_unstale":"2024-03-14T09:52:10.250705+0000","last_undegraded":"2024-03-14T09:52:10.250705+0000","last_fullsized":"2024-03-14T09:52:10.250705+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:48:56.015410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"83'9","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.720923+0000","last_change":"2024-03-14T09:52:05.227962+0000","last_active":"2024-03-14T09:52:29.720923+0000","last_peered":"2024-03-14T09:52:29.720923+0000","last_clean":"2024-03-14T09:52:29.720923+0000","last_became_active":"2024-03-14T09:52:05.226313+0000","last_became_peered":"2024-03-14T09:52:05.226313+0000","last_unstale":"2024-03-14T09:52:29.720923+0000","last_undegraded":"2024-03-14T09:52:29.720923+0000","last_fullsized":"2024-03-14T09:52:29.720923+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:54:46.291940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744051+0000","last_change":"2024-03-14T09:52:05.739004+0000","last_active":"2024-03-14T09:52:09.744051+0000","last_peered":"2024-03-14T09:52:09.744051+0000","last_clean":"2024-03-14T09:52:09.744051+0000","last_became_active":"2024-03-14T09:52:05.732217+0000","last_became_peered":"2024-03-14T09:52:05.732217+0000","last_unstale":"2024-03-14T09:52:09.744051+0000","last_undegraded":"2024-03-14T09:52:09.744051+0000","last_fullsized":"2024-03-14T09:52:09.744051+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:02:21.926623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.486348+0000","last_change":"2024-03-14T09:52:09.776128+0000","last_active":"2024-03-14T09:52:10.486348+0000","last_peered":"2024-03-14T09:52:10.486348+0000","last_clean":"2024-03-14T09:52:10.486348+0000","last_became_active":"2024-03-14T09:52:09.775701+0000","last_became_peered":"2024-03-14T09:52:09.775701+0000","last_unstale":"2024-03-14T09:52:10.486348+0000","last_undegraded":"2024-03-14T09:52:10.486348+0000","last_fullsized":"2024-03-14T09:52:10.486348+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:41:51.790627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"83'2","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967420+0000","last_change":"2024-03-14T09:52:05.234269+0000","last_active":"2024-03-14T09:52:10.967420+0000","last_peered":"2024-03-14T09:52:10.967420+0000","last_clean":"2024-03-14T09:52:10.967420+0000","last_became_active":"2024-03-14T09:52:05.233418+0000","last_became_peered":"2024-03-14T09:52:05.233418+0000","last_unstale":"2024-03-14T09:52:10.967420+0000","last_undegraded":"2024-03-14T09:52:10.967420+0000","last_fullsized":"2024-03-14T09:52:10.967420+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:22:46.894461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.755348+0000","last_change":"2024-03-14T09:52:05.741786+0000","last_active":"2024-03-14T09:52:09.755348+0000","last_peered":"2024-03-14T09:52:09.755348+0000","last_clean":"2024-03-14T09:52:09.755348+0000","last_became_active":"2024-03-14T09:52:05.741461+0000","last_became_peered":"2024-03-14T09:52:05.741461+0000","last_unstale":"2024-03-14T09:52:09.755348+0000","last_undegraded":"2024-03-14T09:52:09.755348+0000","last_fullsized":"2024-03-14T09:52:09.755348+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:21:17.409530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.486387+0000","last_change":"2024-03-14T09:52:09.768851+0000","last_active":"2024-03-14T09:52:10.486387+0000","last_peered":"2024-03-14T09:52:10.486387+0000","last_clean":"2024-03-14T09:52:10.486387+0000","last_became_active":"2024-03-14T09:52:09.768485+0000","last_became_peered":"2024-03-14T09:52:09.768485+0000","last_unstale":"2024-03-14T09:52:10.486387+0000","last_undegraded":"2024-03-14T09:52:10.486387+0000","last_fullsized":"2024-03-14T09:52:10.486387+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:00:51.189790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"83'4","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.718482+0000","last_change":"2024-03-14T09:52:05.232864+0000","last_active":"2024-03-14T09:52:29.718482+0000","last_peered":"2024-03-14T09:52:29.718482+0000","last_clean":"2024-03-14T09:52:29.718482+0000","last_became_active":"2024-03-14T09:52:05.232546+0000","last_became_peered":"2024-03-14T09:52:05.232546+0000","last_unstale":"2024-03-14T09:52:29.718482+0000","last_undegraded":"2024-03-14T09:52:29.718482+0000","last_fullsized":"2024-03-14T09:52:29.718482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:05:00.418174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.747368+0000","last_change":"2024-03-14T09:52:05.742994+0000","last_active":"2024-03-14T09:52:09.747368+0000","last_peered":"2024-03-14T09:52:09.747368+0000","last_clean":"2024-03-14T09:52:09.747368+0000","last_became_active":"2024-03-14T09:52:05.742652+0000","last_became_peered":"2024-03-14T09:52:05.742652+0000","last_unstale":"2024-03-14T09:52:09.747368+0000","last_undegraded":"2024-03-14T09:52:09.747368+0000","last_fullsized":"2024-03-14T09:52:09.747368+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:46:47.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.486376+0000","last_change":"2024-03-14T09:52:09.763009+0000","last_active":"2024-03-14T09:52:10.486376+0000","last_peered":"2024-03-14T09:52:10.486376+0000","last_clean":"2024-03-14T09:52:10.486376+0000","last_became_active":"2024-03-14T09:52:09.762621+0000","last_became_peered":"2024-03-14T09:52:09.762621+0000","last_unstale":"2024-03-14T09:52:10.486376+0000","last_undegraded":"2024-03-14T09:52:10.486376+0000","last_fullsized":"2024-03-14T09:52:10.486376+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:56:22.162409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.752237+0000","last_change":"2024-03-14T09:52:05.234039+0000","last_active":"2024-03-14T09:52:09.752237+0000","last_peered":"2024-03-14T09:52:09.752237+0000","last_clean":"2024-03-14T09:52:09.752237+0000","last_became_active":"2024-03-14T09:52:05.233612+0000","last_became_peered":"2024-03-14T09:52:05.233612+0000","last_unstale":"2024-03-14T09:52:09.752237+0000","last_undegraded":"2024-03-14T09:52:09.752237+0000","last_fullsized":"2024-03-14T09:52:09.752237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:29:52.968178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744407+0000","last_change":"2024-03-14T09:52:05.731932+0000","last_active":"2024-03-14T09:52:09.744407+0000","last_peered":"2024-03-14T09:52:09.744407+0000","last_clean":"2024-03-14T09:52:09.744407+0000","last_became_active":"2024-03-14T09:52:05.731618+0000","last_became_peered":"2024-03-14T09:52:05.731618+0000","last_unstale":"2024-03-14T09:52:09.744407+0000","last_undegraded":"2024-03-14T09:52:09.744407+0000","last_fullsized":"2024-03-14T09:52:09.744407+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:50:26.151271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.251006+0000","last_change":"2024-03-14T09:52:09.757856+0000","last_active":"2024-03-14T09:52:10.251006+0000","last_peered":"2024-03-14T09:52:10.251006+0000","last_clean":"2024-03-14T09:52:10.251006+0000","last_became_active":"2024-03-14T09:52:09.757395+0000","last_became_peered":"2024-03-14T09:52:09.757395+0000","last_unstale":"2024-03-14T09:52:10.251006+0000","last_undegraded":"2024-03-14T09:52:10.251006+0000","last_fullsized":"2024-03-14T09:52:10.251006+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:34:06.600727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"83'5","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:30.728822+0000","last_change":"2024-03-14T09:52:05.228527+0000","last_active":"2024-03-14T09:52:30.728822+0000","last_peered":"2024-03-14T09:52:30.728822+0000","last_clean":"2024-03-14T09:52:30.728822+0000","last_became_active":"2024-03-14T09:52:05.228177+0000","last_became_peered":"2024-03-14T09:52:05.228177+0000","last_unstale":"2024-03-14T09:52:30.728822+0000","last_undegraded":"2024-03-14T09:52:30.728822+0000","last_fullsized":"2024-03-14T09:52:30.728822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:17:57.529308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.752152+0000","last_change":"2024-03-14T09:52:05.734041+0000","last_active":"2024-03-14T09:52:09.752152+0000","last_peered":"2024-03-14T09:52:09.752152+0000","last_clean":"2024-03-14T09:52:09.752152+0000","last_became_active":"2024-03-14T09:52:05.732491+0000","last_became_peered":"2024-03-14T09:52:05.732491+0000","last_unstale":"2024-03-14T09:52:09.752152+0000","last_undegraded":"2024-03-14T09:52:09.752152+0000","last_fullsized":"2024-03-14T09:52:09.752152+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:07:31.909382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.250818+0000","last_change":"2024-03-14T09:52:09.769584+0000","last_active":"2024-03-14T09:52:10.250818+0000","last_peered":"2024-03-14T09:52:10.250818+0000","last_clean":"2024-03-14T09:52:10.250818+0000","last_became_active":"2024-03-14T09:52:09.768847+0000","last_became_peered":"2024-03-14T09:52:09.768847+0000","last_unstale":"2024-03-14T09:52:10.250818+0000","last_undegraded":"2024-03-14T09:52:10.250818+0000","last_fullsized":"2024-03-14T09:52:10.250818+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:12:18.005122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"78'1","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.755585+0000","last_change":"2024-03-14T09:52:05.228893+0000","last_active":"2024-03-14T09:52:09.755585+0000","last_peered":"2024-03-14T09:52:09.755585+0000","last_clean":"2024-03-14T09:52:09.755585+0000","last_became_active":"2024-03-14T09:52:05.228569+0000","last_became_peered":"2024-03-14T09:52:05.228569+0000","last_unstale":"2024-03-14T09:52:09.755585+0000","last_undegraded":"2024-03-14T09:52:09.755585+0000","last_fullsized":"2024-03-14T09:52:09.755585+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:28:03.200437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744492+0000","last_change":"2024-03-14T09:52:05.736388+0000","last_active":"2024-03-14T09:52:09.744492+0000","last_peered":"2024-03-14T09:52:09.744492+0000","last_clean":"2024-03-14T09:52:09.744492+0000","last_became_active":"2024-03-14T09:52:05.735131+0000","last_became_peered":"2024-03-14T09:52:05.735131+0000","last_unstale":"2024-03-14T09:52:09.744492+0000","last_undegraded":"2024-03-14T09:52:09.744492+0000","last_fullsized":"2024-03-14T09:52:09.744492+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:00:54.120087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967733+0000","last_change":"2024-03-14T09:52:09.770321+0000","last_active":"2024-03-14T09:52:10.967733+0000","last_peered":"2024-03-14T09:52:10.967733+0000","last_clean":"2024-03-14T09:52:10.967733+0000","last_became_active":"2024-03-14T09:52:09.769710+0000","last_became_peered":"2024-03-14T09:52:09.769710+0000","last_unstale":"2024-03-14T09:52:10.967733+0000","last_undegraded":"2024-03-14T09:52:10.967733+0000","last_fullsized":"2024-03-14T09:52:10.967733+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:12:12.635248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"83'2","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.721097+0000","last_change":"2024-03-14T09:52:05.223912+0000","last_active":"2024-03-14T09:52:29.721097+0000","last_peered":"2024-03-14T09:52:29.721097+0000","last_clean":"2024-03-14T09:52:29.721097+0000","last_became_active":"2024-03-14T09:52:05.223380+0000","last_became_peered":"2024-03-14T09:52:05.223380+0000","last_unstale":"2024-03-14T09:52:29.721097+0000","last_undegraded":"2024-03-14T09:52:29.721097+0000","last_fullsized":"2024-03-14T09:52:29.721097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:08:33.098215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.743438+0000","last_change":"2024-03-14T09:52:05.743277+0000","last_active":"2024-03-14T09:52:09.743438+0000","last_peered":"2024-03-14T09:52:09.743438+0000","last_clean":"2024-03-14T09:52:09.743438+0000","last_became_active":"2024-03-14T09:52:05.741386+0000","last_became_peered":"2024-03-14T09:52:05.741386+0000","last_unstale":"2024-03-14T09:52:09.743438+0000","last_undegraded":"2024-03-14T09:52:09.743438+0000","last_fullsized":"2024-03-14T09:52:09.743438+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:08:08.320279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967655+0000","last_change":"2024-03-14T09:52:09.766485+0000","last_active":"2024-03-14T09:52:10.967655+0000","last_peered":"2024-03-14T09:52:10.967655+0000","last_clean":"2024-03-14T09:52:10.967655+0000","last_became_active":"2024-03-14T09:52:09.766115+0000","last_became_peered":"2024-03-14T09:52:09.766115+0000","last_unstale":"2024-03-14T09:52:10.967655+0000","last_undegraded":"2024-03-14T09:52:10.967655+0000","last_fullsized":"2024-03-14T09:52:10.967655+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:34:45.067757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"83'1","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.968581+0000","last_change":"2024-03-14T09:52:05.233852+0000","last_active":"2024-03-14T09:52:10.968581+0000","last_peered":"2024-03-14T09:52:10.968581+0000","last_clean":"2024-03-14T09:52:10.968581+0000","last_became_active":"2024-03-14T09:52:05.233548+0000","last_became_peered":"2024-03-14T09:52:05.233548+0000","last_unstale":"2024-03-14T09:52:10.968581+0000","last_undegraded":"2024-03-14T09:52:10.968581+0000","last_fullsized":"2024-03-14T09:52:10.968581+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:08:51.821465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.760078+0000","last_change":"2024-03-14T09:52:05.731215+0000","last_active":"2024-03-14T09:52:09.760078+0000","last_peered":"2024-03-14T09:52:09.760078+0000","last_clean":"2024-03-14T09:52:09.760078+0000","last_became_active":"2024-03-14T09:52:05.730924+0000","last_became_peered":"2024-03-14T09:52:05.730924+0000","last_unstale":"2024-03-14T09:52:09.760078+0000","last_undegraded":"2024-03-14T09:52:09.760078+0000","last_fullsized":"2024-03-14T09:52:09.760078+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:12:17.560297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967932+0000","last_change":"2024-03-14T09:52:09.753775+0000","last_active":"2024-03-14T09:52:10.967932+0000","last_peered":"2024-03-14T09:52:10.967932+0000","last_clean":"2024-03-14T09:52:10.967932+0000","last_became_active":"2024-03-14T09:52:09.753422+0000","last_became_peered":"2024-03-14T09:52:09.753422+0000","last_unstale":"2024-03-14T09:52:10.967932+0000","last_undegraded":"2024-03-14T09:52:10.967932+0000","last_fullsized":"2024-03-14T09:52:10.967932+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:59:25.625909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.755861+0000","last_change":"2024-03-14T09:52:05.229064+0000","last_active":"2024-03-14T09:52:09.755861+0000","last_peered":"2024-03-14T09:52:09.755861+0000","last_clean":"2024-03-14T09:52:09.755861+0000","last_became_active":"2024-03-14T09:52:05.228751+0000","last_became_peered":"2024-03-14T09:52:05.228751+0000","last_unstale":"2024-03-14T09:52:09.755861+0000","last_undegraded":"2024-03-14T09:52:09.755861+0000","last_fullsized":"2024-03-14T09:52:09.755861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:13:29.832866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748082+0000","last_change":"2024-03-14T09:52:05.737798+0000","last_active":"2024-03-14T09:52:09.748082+0000","last_peered":"2024-03-14T09:52:09.748082+0000","last_clean":"2024-03-14T09:52:09.748082+0000","last_became_active":"2024-03-14T09:52:05.737449+0000","last_became_peered":"2024-03-14T09:52:05.737449+0000","last_unstale":"2024-03-14T09:52:09.748082+0000","last_undegraded":"2024-03-14T09:52:09.748082+0000","last_fullsized":"2024-03-14T09:52:09.748082+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:40:10.079770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.966435+0000","last_change":"2024-03-14T09:52:09.768368+0000","last_active":"2024-03-14T09:52:10.966435+0000","last_peered":"2024-03-14T09:52:10.966435+0000","last_clean":"2024-03-14T09:52:10.966435+0000","last_became_active":"2024-03-14T09:52:09.767622+0000","last_became_peered":"2024-03-14T09:52:09.767622+0000","last_unstale":"2024-03-14T09:52:10.966435+0000","last_undegraded":"2024-03-14T09:52:10.966435+0000","last_fullsized":"2024-03-14T09:52:10.966435+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:19:28.836317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"83'2","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.715610+0000","last_change":"2024-03-14T09:52:05.229778+0000","last_active":"2024-03-14T09:52:28.715610+0000","last_peered":"2024-03-14T09:52:28.715610+0000","last_clean":"2024-03-14T09:52:28.715610+0000","last_became_active":"2024-03-14T09:52:05.229468+0000","last_became_peered":"2024-03-14T09:52:05.229468+0000","last_unstale":"2024-03-14T09:52:28.715610+0000","last_undegraded":"2024-03-14T09:52:28.715610+0000","last_fullsized":"2024-03-14T09:52:28.715610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:49:58.433646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.755436+0000","last_change":"2024-03-14T09:52:05.751027+0000","last_active":"2024-03-14T09:52:09.755436+0000","last_peered":"2024-03-14T09:52:09.755436+0000","last_clean":"2024-03-14T09:52:09.755436+0000","last_became_active":"2024-03-14T09:52:05.750650+0000","last_became_peered":"2024-03-14T09:52:05.750650+0000","last_unstale":"2024-03-14T09:52:09.755436+0000","last_undegraded":"2024-03-14T09:52:09.755436+0000","last_fullsized":"2024-03-14T09:52:09.755436+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:05:37.264396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.486422+0000","last_change":"2024-03-14T09:52:09.768207+0000","last_active":"2024-03-14T09:52:10.486422+0000","last_peered":"2024-03-14T09:52:10.486422+0000","last_clean":"2024-03-14T09:52:10.486422+0000","last_became_active":"2024-03-14T09:52:09.767756+0000","last_became_peered":"2024-03-14T09:52:09.767756+0000","last_unstale":"2024-03-14T09:52:10.486422+0000","last_undegraded":"2024-03-14T09:52:10.486422+0000","last_fullsized":"2024-03-14T09:52:10.486422+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:13:12.544995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"83'4","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.717910+0000","last_change":"2024-03-14T09:52:05.235472+0000","last_active":"2024-03-14T09:52:28.717910+0000","last_peered":"2024-03-14T09:52:28.717910+0000","last_clean":"2024-03-14T09:52:28.717910+0000","last_became_active":"2024-03-14T09:52:05.235106+0000","last_became_peered":"2024-03-14T09:52:05.235106+0000","last_unstale":"2024-03-14T09:52:28.717910+0000","last_undegraded":"2024-03-14T09:52:28.717910+0000","last_fullsized":"2024-03-14T09:52:28.717910+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:30:36.446130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.747531+0000","last_change":"2024-03-14T09:52:05.747615+0000","last_active":"2024-03-14T09:52:09.747531+0000","last_peered":"2024-03-14T09:52:09.747531+0000","last_clean":"2024-03-14T09:52:09.747531+0000","last_became_active":"2024-03-14T09:52:05.747344+0000","last_became_peered":"2024-03-14T09:52:05.747344+0000","last_unstale":"2024-03-14T09:52:09.747531+0000","last_undegraded":"2024-03-14T09:52:09.747531+0000","last_fullsized":"2024-03-14T09:52:09.747531+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:52:23.096124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967414+0000","last_change":"2024-03-14T09:52:09.768295+0000","last_active":"2024-03-14T09:52:10.967414+0000","last_peered":"2024-03-14T09:52:10.967414+0000","last_clean":"2024-03-14T09:52:10.967414+0000","last_became_active":"2024-03-14T09:52:09.760997+0000","last_became_peered":"2024-03-14T09:52:09.760997+0000","last_unstale":"2024-03-14T09:52:10.967414+0000","last_undegraded":"2024-03-14T09:52:10.967414+0000","last_fullsized":"2024-03-14T09:52:10.967414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:22:09.909302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"83'10","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:25.294154+0000","last_change":"2024-03-14T09:52:05.227876+0000","last_active":"2024-03-14T09:52:25.294154+0000","last_peered":"2024-03-14T09:52:25.294154+0000","last_clean":"2024-03-14T09:52:25.294154+0000","last_became_active":"2024-03-14T09:52:05.227519+0000","last_became_peered":"2024-03-14T09:52:05.227519+0000","last_unstale":"2024-03-14T09:52:25.294154+0000","last_undegraded":"2024-03-14T09:52:25.294154+0000","last_fullsized":"2024-03-14T09:52:25.294154+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:12.616609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3795,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.752533+0000","last_change":"2024-03-14T09:52:05.737432+0000","last_active":"2024-03-14T09:52:09.752533+0000","last_peered":"2024-03-14T09:52:09.752533+0000","last_clean":"2024-03-14T09:52:09.752533+0000","last_became_active":"2024-03-14T09:52:05.735050+0000","last_became_peered":"2024-03-14T09:52:05.735050+0000","last_unstale":"2024-03-14T09:52:09.752533+0000","last_undegraded":"2024-03-14T09:52:09.752533+0000","last_fullsized":"2024-03-14T09:52:09.752533+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T09:57:45.383155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.968799+0000","last_change":"2024-03-14T09:52:09.754909+0000","last_active":"2024-03-14T09:52:10.968799+0000","last_peered":"2024-03-14T09:52:10.968799+0000","last_clean":"2024-03-14T09:52:10.968799+0000","last_became_active":"2024-03-14T09:52:09.754367+0000","last_became_peered":"2024-03-14T09:52:09.754367+0000","last_unstale":"2024-03-14T09:52:10.968799+0000","last_undegraded":"2024-03-14T09:52:10.968799+0000","last_fullsized":"2024-03-14T09:52:10.968799+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:52:34.251555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"83'6","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.720249+0000","last_change":"2024-03-14T09:52:05.224869+0000","last_active":"2024-03-14T09:52:29.720249+0000","last_peered":"2024-03-14T09:52:29.720249+0000","last_clean":"2024-03-14T09:52:29.720249+0000","last_became_active":"2024-03-14T09:52:05.224499+0000","last_became_peered":"2024-03-14T09:52:05.224499+0000","last_unstale":"2024-03-14T09:52:29.720249+0000","last_undegraded":"2024-03-14T09:52:29.720249+0000","last_fullsized":"2024-03-14T09:52:29.720249+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:49:01.729417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.906059+0000","last_change":"2024-03-14T09:52:05.729168+0000","last_active":"2024-03-14T09:52:09.906059+0000","last_peered":"2024-03-14T09:52:09.906059+0000","last_clean":"2024-03-14T09:52:09.906059+0000","last_became_active":"2024-03-14T09:52:05.728806+0000","last_became_peered":"2024-03-14T09:52:05.728806+0000","last_unstale":"2024-03-14T09:52:09.906059+0000","last_undegraded":"2024-03-14T09:52:09.906059+0000","last_fullsized":"2024-03-14T09:52:09.906059+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:54:36.758853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.966838+0000","last_change":"2024-03-14T09:52:09.770126+0000","last_active":"2024-03-14T09:52:10.966838+0000","last_peered":"2024-03-14T09:52:10.966838+0000","last_clean":"2024-03-14T09:52:10.966838+0000","last_became_active":"2024-03-14T09:52:09.769734+0000","last_became_peered":"2024-03-14T09:52:09.769734+0000","last_unstale":"2024-03-14T09:52:10.966838+0000","last_undegraded":"2024-03-14T09:52:10.966838+0000","last_fullsized":"2024-03-14T09:52:10.966838+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:50:37.839858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"83'3","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.716453+0000","last_change":"2024-03-14T09:52:05.235201+0000","last_active":"2024-03-14T09:52:28.716453+0000","last_peered":"2024-03-14T09:52:28.716453+0000","last_clean":"2024-03-14T09:52:28.716453+0000","last_became_active":"2024-03-14T09:52:05.234816+0000","last_became_peered":"2024-03-14T09:52:05.234816+0000","last_unstale":"2024-03-14T09:52:28.716453+0000","last_undegraded":"2024-03-14T09:52:28.716453+0000","last_fullsized":"2024-03-14T09:52:28.716453+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:01:50.070913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744271+0000","last_change":"2024-03-14T09:52:05.735304+0000","last_active":"2024-03-14T09:52:09.744271+0000","last_peered":"2024-03-14T09:52:09.744271+0000","last_clean":"2024-03-14T09:52:09.744271+0000","last_became_active":"2024-03-14T09:52:05.734977+0000","last_became_peered":"2024-03-14T09:52:05.734977+0000","last_unstale":"2024-03-14T09:52:09.744271+0000","last_undegraded":"2024-03-14T09:52:09.744271+0000","last_fullsized":"2024-03-14T09:52:09.744271+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:08:28.071962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.968580+0000","last_change":"2024-03-14T09:52:09.770721+0000","last_active":"2024-03-14T09:52:10.968580+0000","last_peered":"2024-03-14T09:52:10.968580+0000","last_clean":"2024-03-14T09:52:10.968580+0000","last_became_active":"2024-03-14T09:52:09.770270+0000","last_became_peered":"2024-03-14T09:52:09.770270+0000","last_unstale":"2024-03-14T09:52:10.968580+0000","last_undegraded":"2024-03-14T09:52:10.968580+0000","last_fullsized":"2024-03-14T09:52:10.968580+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:09:33.480235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"83'3","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.718282+0000","last_change":"2024-03-14T09:52:05.227260+0000","last_active":"2024-03-14T09:52:29.718282+0000","last_peered":"2024-03-14T09:52:29.718282+0000","last_clean":"2024-03-14T09:52:29.718282+0000","last_became_active":"2024-03-14T09:52:05.226922+0000","last_became_peered":"2024-03-14T09:52:05.226922+0000","last_unstale":"2024-03-14T09:52:29.718282+0000","last_undegraded":"2024-03-14T09:52:29.718282+0000","last_fullsized":"2024-03-14T09:52:29.718282+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:36:09.770478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.743907+0000","last_change":"2024-03-14T09:52:05.734278+0000","last_active":"2024-03-14T09:52:09.743907+0000","last_peered":"2024-03-14T09:52:09.743907+0000","last_clean":"2024-03-14T09:52:09.743907+0000","last_became_active":"2024-03-14T09:52:05.732051+0000","last_became_peered":"2024-03-14T09:52:05.732051+0000","last_unstale":"2024-03-14T09:52:09.743907+0000","last_undegraded":"2024-03-14T09:52:09.743907+0000","last_fullsized":"2024-03-14T09:52:09.743907+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:33:07.137983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.906154+0000","last_change":"2024-03-14T09:52:09.764068+0000","last_active":"2024-03-14T09:52:09.906154+0000","last_peered":"2024-03-14T09:52:09.906154+0000","last_clean":"2024-03-14T09:52:09.906154+0000","last_became_active":"2024-03-14T09:52:09.763613+0000","last_became_peered":"2024-03-14T09:52:09.763613+0000","last_unstale":"2024-03-14T09:52:09.906154+0000","last_undegraded":"2024-03-14T09:52:09.906154+0000","last_fullsized":"2024-03-14T09:52:09.906154+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:32:15.524602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"83'15","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.721151+0000","last_change":"2024-03-14T09:52:05.233845+0000","last_active":"2024-03-14T09:52:29.721151+0000","last_peered":"2024-03-14T09:52:29.721151+0000","last_clean":"2024-03-14T09:52:29.721151+0000","last_became_active":"2024-03-14T09:52:05.233527+0000","last_became_peered":"2024-03-14T09:52:05.233527+0000","last_unstale":"2024-03-14T09:52:29.721151+0000","last_undegraded":"2024-03-14T09:52:29.721151+0000","last_fullsized":"2024-03-14T09:52:29.721151+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T15:20:57.496761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.745334+0000","last_change":"2024-03-14T09:52:05.734366+0000","last_active":"2024-03-14T09:52:09.745334+0000","last_peered":"2024-03-14T09:52:09.745334+0000","last_clean":"2024-03-14T09:52:09.745334+0000","last_became_active":"2024-03-14T09:52:05.734058+0000","last_became_peered":"2024-03-14T09:52:05.734058+0000","last_unstale":"2024-03-14T09:52:09.745334+0000","last_undegraded":"2024-03-14T09:52:09.745334+0000","last_fullsized":"2024-03-14T09:52:09.745334+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:49:49.233683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.968347+0000","last_change":"2024-03-14T09:52:09.760241+0000","last_active":"2024-03-14T09:52:10.968347+0000","last_peered":"2024-03-14T09:52:10.968347+0000","last_clean":"2024-03-14T09:52:10.968347+0000","last_became_active":"2024-03-14T09:52:09.759797+0000","last_became_peered":"2024-03-14T09:52:09.759797+0000","last_unstale":"2024-03-14T09:52:10.968347+0000","last_undegraded":"2024-03-14T09:52:10.968347+0000","last_fullsized":"2024-03-14T09:52:10.968347+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:50:33.035775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"78'1","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.966996+0000","last_change":"2024-03-14T09:52:05.232932+0000","last_active":"2024-03-14T09:52:10.966996+0000","last_peered":"2024-03-14T09:52:10.966996+0000","last_clean":"2024-03-14T09:52:10.966996+0000","last_became_active":"2024-03-14T09:52:05.232608+0000","last_became_peered":"2024-03-14T09:52:05.232608+0000","last_unstale":"2024-03-14T09:52:10.966996+0000","last_undegraded":"2024-03-14T09:52:10.966996+0000","last_fullsized":"2024-03-14T09:52:10.966996+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:43:59.613531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748431+0000","last_change":"2024-03-14T09:52:05.733762+0000","last_active":"2024-03-14T09:52:09.748431+0000","last_peered":"2024-03-14T09:52:09.748431+0000","last_clean":"2024-03-14T09:52:09.748431+0000","last_became_active":"2024-03-14T09:52:05.733417+0000","last_became_peered":"2024-03-14T09:52:05.733417+0000","last_unstale":"2024-03-14T09:52:09.748431+0000","last_undegraded":"2024-03-14T09:52:09.748431+0000","last_fullsized":"2024-03-14T09:52:09.748431+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:11:12.593952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967586+0000","last_change":"2024-03-14T09:52:09.761405+0000","last_active":"2024-03-14T09:52:10.967586+0000","last_peered":"2024-03-14T09:52:10.967586+0000","last_clean":"2024-03-14T09:52:10.967586+0000","last_became_active":"2024-03-14T09:52:09.760935+0000","last_became_peered":"2024-03-14T09:52:09.760935+0000","last_unstale":"2024-03-14T09:52:10.967586+0000","last_undegraded":"2024-03-14T09:52:10.967586+0000","last_fullsized":"2024-03-14T09:52:10.967586+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:12:30.663121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.766214+0000","last_change":"2024-03-14T09:52:05.227636+0000","last_active":"2024-03-14T09:52:09.766214+0000","last_peered":"2024-03-14T09:52:09.766214+0000","last_clean":"2024-03-14T09:52:09.766214+0000","last_became_active":"2024-03-14T09:52:05.227274+0000","last_became_peered":"2024-03-14T09:52:05.227274+0000","last_unstale":"2024-03-14T09:52:09.766214+0000","last_undegraded":"2024-03-14T09:52:09.766214+0000","last_fullsized":"2024-03-14T09:52:09.766214+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:13:25.380090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.756002+0000","last_change":"2024-03-14T09:52:05.733158+0000","last_active":"2024-03-14T09:52:09.756002+0000","last_peered":"2024-03-14T09:52:09.756002+0000","last_clean":"2024-03-14T09:52:09.756002+0000","last_became_active":"2024-03-14T09:52:05.732812+0000","last_became_peered":"2024-03-14T09:52:05.732812+0000","last_unstale":"2024-03-14T09:52:09.756002+0000","last_undegraded":"2024-03-14T09:52:09.756002+0000","last_fullsized":"2024-03-14T09:52:09.756002+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:01:49.154860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.906092+0000","last_change":"2024-03-14T09:52:09.761596+0000","last_active":"2024-03-14T09:52:09.906092+0000","last_peered":"2024-03-14T09:52:09.906092+0000","last_clean":"2024-03-14T09:52:09.906092+0000","last_became_active":"2024-03-14T09:52:09.761184+0000","last_became_peered":"2024-03-14T09:52:09.761184+0000","last_unstale":"2024-03-14T09:52:09.906092+0000","last_undegraded":"2024-03-14T09:52:09.906092+0000","last_fullsized":"2024-03-14T09:52:09.906092+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:49:02.567472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"83'1","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.250602+0000","last_change":"2024-03-14T09:52:05.231828+0000","last_active":"2024-03-14T09:52:10.250602+0000","last_peered":"2024-03-14T09:52:10.250602+0000","last_clean":"2024-03-14T09:52:10.250602+0000","last_became_active":"2024-03-14T09:52:05.231490+0000","last_became_peered":"2024-03-14T09:52:05.231490+0000","last_unstale":"2024-03-14T09:52:10.250602+0000","last_undegraded":"2024-03-14T09:52:10.250602+0000","last_fullsized":"2024-03-14T09:52:10.250602+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:24:53.899728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.756180+0000","last_change":"2024-03-14T09:52:05.744646+0000","last_active":"2024-03-14T09:52:09.756180+0000","last_peered":"2024-03-14T09:52:09.756180+0000","last_clean":"2024-03-14T09:52:09.756180+0000","last_became_active":"2024-03-14T09:52:05.744327+0000","last_became_peered":"2024-03-14T09:52:05.744327+0000","last_unstale":"2024-03-14T09:52:09.756180+0000","last_undegraded":"2024-03-14T09:52:09.756180+0000","last_fullsized":"2024-03-14T09:52:09.756180+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:46:50.643350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.906188+0000","last_change":"2024-03-14T09:52:09.765374+0000","last_active":"2024-03-14T09:52:09.906188+0000","last_peered":"2024-03-14T09:52:09.906188+0000","last_clean":"2024-03-14T09:52:09.906188+0000","last_became_active":"2024-03-14T09:52:09.764939+0000","last_became_peered":"2024-03-14T09:52:09.764939+0000","last_unstale":"2024-03-14T09:52:09.906188+0000","last_undegraded":"2024-03-14T09:52:09.906188+0000","last_fullsized":"2024-03-14T09:52:09.906188+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:08:10.688746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.745417+0000","last_change":"2024-03-14T09:52:05.729076+0000","last_active":"2024-03-14T09:52:09.745417+0000","last_peered":"2024-03-14T09:52:09.745417+0000","last_clean":"2024-03-14T09:52:09.745417+0000","last_became_active":"2024-03-14T09:52:05.728711+0000","last_became_peered":"2024-03-14T09:52:05.728711+0000","last_unstale":"2024-03-14T09:52:09.745417+0000","last_undegraded":"2024-03-14T09:52:09.745417+0000","last_fullsized":"2024-03-14T09:52:09.745417+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:11:05.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"83'4","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.721389+0000","last_change":"2024-03-14T09:52:05.236209+0000","last_active":"2024-03-14T09:52:29.721389+0000","last_peered":"2024-03-14T09:52:29.721389+0000","last_clean":"2024-03-14T09:52:29.721389+0000","last_became_active":"2024-03-14T09:52:05.235517+0000","last_became_peered":"2024-03-14T09:52:05.235517+0000","last_unstale":"2024-03-14T09:52:29.721389+0000","last_undegraded":"2024-03-14T09:52:29.721389+0000","last_fullsized":"2024-03-14T09:52:29.721389+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:09:54.275728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.250741+0000","last_change":"2024-03-14T09:52:09.761095+0000","last_active":"2024-03-14T09:52:10.250741+0000","last_peered":"2024-03-14T09:52:10.250741+0000","last_clean":"2024-03-14T09:52:10.250741+0000","last_became_active":"2024-03-14T09:52:09.760662+0000","last_became_peered":"2024-03-14T09:52:09.760662+0000","last_unstale":"2024-03-14T09:52:10.250741+0000","last_undegraded":"2024-03-14T09:52:10.250741+0000","last_fullsized":"2024-03-14T09:52:10.250741+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:07:17.670605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.743531+0000","last_change":"2024-03-14T09:52:05.736078+0000","last_active":"2024-03-14T09:52:09.743531+0000","last_peered":"2024-03-14T09:52:09.743531+0000","last_clean":"2024-03-14T09:52:09.743531+0000","last_became_active":"2024-03-14T09:52:05.735715+0000","last_became_peered":"2024-03-14T09:52:05.735715+0000","last_unstale":"2024-03-14T09:52:09.743531+0000","last_undegraded":"2024-03-14T09:52:09.743531+0000","last_fullsized":"2024-03-14T09:52:09.743531+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:41:32.863680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"83'3","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.718447+0000","last_change":"2024-03-14T09:52:05.225756+0000","last_active":"2024-03-14T09:52:28.718447+0000","last_peered":"2024-03-14T09:52:28.718447+0000","last_clean":"2024-03-14T09:52:28.718447+0000","last_became_active":"2024-03-14T09:52:05.225378+0000","last_became_peered":"2024-03-14T09:52:05.225378+0000","last_unstale":"2024-03-14T09:52:28.718447+0000","last_undegraded":"2024-03-14T09:52:28.718447+0000","last_fullsized":"2024-03-14T09:52:28.718447+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:14:49.594636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":335,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.250396+0000","last_change":"2024-03-14T09:51:11.588790+0000","last_active":"2024-03-14T09:52:10.250396+0000","last_peered":"2024-03-14T09:52:10.250396+0000","last_clean":"2024-03-14T09:52:10.250396+0000","last_became_active":"2024-03-14T09:51:11.588287+0000","last_became_peered":"2024-03-14T09:51:11.588287+0000","last_unstale":"2024-03-14T09:52:10.250396+0000","last_undegraded":"2024-03-14T09:52:10.250396+0000","last_fullsized":"2024-03-14T09:52:10.250396+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:48:37.651174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:48:37.651174+0000","last_clean_scrub_stamp":"2024-03-14T09:48:37.651174+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:06:24.252565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967223+0000","last_change":"2024-03-14T09:52:09.761344+0000","last_active":"2024-03-14T09:52:10.967223+0000","last_peered":"2024-03-14T09:52:10.967223+0000","last_clean":"2024-03-14T09:52:10.967223+0000","last_became_active":"2024-03-14T09:52:09.760899+0000","last_became_peered":"2024-03-14T09:52:09.760899+0000","last_unstale":"2024-03-14T09:52:10.967223+0000","last_undegraded":"2024-03-14T09:52:10.967223+0000","last_fullsized":"2024-03-14T09:52:10.967223+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:24:26.212157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.905984+0000","last_change":"2024-03-14T09:52:05.738735+0000","last_active":"2024-03-14T09:52:09.905984+0000","last_peered":"2024-03-14T09:52:09.905984+0000","last_clean":"2024-03-14T09:52:09.905984+0000","last_became_active":"2024-03-14T09:52:05.738447+0000","last_became_peered":"2024-03-14T09:52:05.738447+0000","last_unstale":"2024-03-14T09:52:09.905984+0000","last_undegraded":"2024-03-14T09:52:09.905984+0000","last_fullsized":"2024-03-14T09:52:09.905984+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:03:24.598291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'3","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.719407+0000","last_change":"2024-03-14T09:52:05.230107+0000","last_active":"2024-03-14T09:52:29.719407+0000","last_peered":"2024-03-14T09:52:29.719407+0000","last_clean":"2024-03-14T09:52:29.719407+0000","last_became_active":"2024-03-14T09:52:05.229812+0000","last_became_peered":"2024-03-14T09:52:05.229812+0000","last_unstale":"2024-03-14T09:52:29.719407+0000","last_undegraded":"2024-03-14T09:52:29.719407+0000","last_fullsized":"2024-03-14T09:52:29.719407+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:09:44.884867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967721+0000","last_change":"2024-03-14T09:52:09.763424+0000","last_active":"2024-03-14T09:52:10.967721+0000","last_peered":"2024-03-14T09:52:10.967721+0000","last_clean":"2024-03-14T09:52:10.967721+0000","last_became_active":"2024-03-14T09:52:09.763028+0000","last_became_peered":"2024-03-14T09:52:09.763028+0000","last_unstale":"2024-03-14T09:52:10.967721+0000","last_undegraded":"2024-03-14T09:52:10.967721+0000","last_fullsized":"2024-03-14T09:52:10.967721+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:47:31.415788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"83'1","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.714897+0000","last_change":"2024-03-14T09:52:05.235099+0000","last_active":"2024-03-14T09:52:28.714897+0000","last_peered":"2024-03-14T09:52:28.714897+0000","last_clean":"2024-03-14T09:52:28.714897+0000","last_became_active":"2024-03-14T09:52:05.234798+0000","last_became_peered":"2024-03-14T09:52:05.234798+0000","last_unstale":"2024-03-14T09:52:28.714897+0000","last_undegraded":"2024-03-14T09:52:28.714897+0000","last_fullsized":"2024-03-14T09:52:28.714897+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:00:07.461931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.906103+0000","last_change":"2024-03-14T09:52:05.742736+0000","last_active":"2024-03-14T09:52:09.906103+0000","last_peered":"2024-03-14T09:52:09.906103+0000","last_clean":"2024-03-14T09:52:09.906103+0000","last_became_active":"2024-03-14T09:52:05.740927+0000","last_became_peered":"2024-03-14T09:52:05.740927+0000","last_unstale":"2024-03-14T09:52:09.906103+0000","last_undegraded":"2024-03-14T09:52:09.906103+0000","last_fullsized":"2024-03-14T09:52:09.906103+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:50:17.738763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.967299+0000","last_change":"2024-03-14T09:52:09.762722+0000","last_active":"2024-03-14T09:52:10.967299+0000","last_peered":"2024-03-14T09:52:10.967299+0000","last_clean":"2024-03-14T09:52:10.967299+0000","last_became_active":"2024-03-14T09:52:09.759053+0000","last_became_peered":"2024-03-14T09:52:09.759053+0000","last_unstale":"2024-03-14T09:52:10.967299+0000","last_undegraded":"2024-03-14T09:52:10.967299+0000","last_fullsized":"2024-03-14T09:52:10.967299+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:40:29.680002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"83'3","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.719859+0000","last_change":"2024-03-14T09:52:05.224778+0000","last_active":"2024-03-14T09:52:29.719859+0000","last_peered":"2024-03-14T09:52:29.719859+0000","last_clean":"2024-03-14T09:52:29.719859+0000","last_became_active":"2024-03-14T09:52:05.223158+0000","last_became_peered":"2024-03-14T09:52:05.223158+0000","last_unstale":"2024-03-14T09:52:29.719859+0000","last_undegraded":"2024-03-14T09:52:29.719859+0000","last_fullsized":"2024-03-14T09:52:29.719859+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:01:29.910099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744495+0000","last_change":"2024-03-14T09:52:05.741753+0000","last_active":"2024-03-14T09:52:09.744495+0000","last_peered":"2024-03-14T09:52:09.744495+0000","last_clean":"2024-03-14T09:52:09.744495+0000","last_became_active":"2024-03-14T09:52:05.741451+0000","last_became_peered":"2024-03-14T09:52:05.741451+0000","last_unstale":"2024-03-14T09:52:09.744495+0000","last_undegraded":"2024-03-14T09:52:09.744495+0000","last_fullsized":"2024-03-14T09:52:09.744495+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:42:59.920909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.250575+0000","last_change":"2024-03-14T09:52:09.755884+0000","last_active":"2024-03-14T09:52:10.250575+0000","last_peered":"2024-03-14T09:52:10.250575+0000","last_clean":"2024-03-14T09:52:10.250575+0000","last_became_active":"2024-03-14T09:52:09.755474+0000","last_became_peered":"2024-03-14T09:52:09.755474+0000","last_unstale":"2024-03-14T09:52:10.250575+0000","last_undegraded":"2024-03-14T09:52:10.250575+0000","last_fullsized":"2024-03-14T09:52:10.250575+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:59:23.383388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"83'4","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.717963+0000","last_change":"2024-03-14T09:52:05.232673+0000","last_active":"2024-03-14T09:52:28.717963+0000","last_peered":"2024-03-14T09:52:28.717963+0000","last_clean":"2024-03-14T09:52:28.717963+0000","last_became_active":"2024-03-14T09:52:05.232296+0000","last_became_peered":"2024-03-14T09:52:05.232296+0000","last_unstale":"2024-03-14T09:52:28.717963+0000","last_undegraded":"2024-03-14T09:52:28.717963+0000","last_fullsized":"2024-03-14T09:52:28.717963+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:24:56.638451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.756306+0000","last_change":"2024-03-14T09:52:05.744906+0000","last_active":"2024-03-14T09:52:09.756306+0000","last_peered":"2024-03-14T09:52:09.756306+0000","last_clean":"2024-03-14T09:52:09.756306+0000","last_became_active":"2024-03-14T09:52:05.743753+0000","last_became_peered":"2024-03-14T09:52:05.743753+0000","last_unstale":"2024-03-14T09:52:09.756306+0000","last_undegraded":"2024-03-14T09:52:09.756306+0000","last_fullsized":"2024-03-14T09:52:09.756306+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T09:59:40.358352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.486476+0000","last_change":"2024-03-14T09:52:09.768355+0000","last_active":"2024-03-14T09:52:10.486476+0000","last_peered":"2024-03-14T09:52:10.486476+0000","last_clean":"2024-03-14T09:52:10.486476+0000","last_became_active":"2024-03-14T09:52:09.767919+0000","last_became_peered":"2024-03-14T09:52:09.767919+0000","last_unstale":"2024-03-14T09:52:10.486476+0000","last_undegraded":"2024-03-14T09:52:10.486476+0000","last_fullsized":"2024-03-14T09:52:10.486476+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:45:08.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"83'2","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.714262+0000","last_change":"2024-03-14T09:52:05.230343+0000","last_active":"2024-03-14T09:52:28.714262+0000","last_peered":"2024-03-14T09:52:28.714262+0000","last_clean":"2024-03-14T09:52:28.714262+0000","last_became_active":"2024-03-14T09:52:05.229974+0000","last_became_peered":"2024-03-14T09:52:05.229974+0000","last_unstale":"2024-03-14T09:52:28.714262+0000","last_undegraded":"2024-03-14T09:52:28.714262+0000","last_fullsized":"2024-03-14T09:52:28.714262+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:21:02.458108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.751986+0000","last_change":"2024-03-14T09:52:05.738347+0000","last_active":"2024-03-14T09:52:09.751986+0000","last_peered":"2024-03-14T09:52:09.751986+0000","last_clean":"2024-03-14T09:52:09.751986+0000","last_became_active":"2024-03-14T09:52:05.737961+0000","last_became_peered":"2024-03-14T09:52:05.737961+0000","last_unstale":"2024-03-14T09:52:09.751986+0000","last_undegraded":"2024-03-14T09:52:09.751986+0000","last_fullsized":"2024-03-14T09:52:09.751986+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:02:57.252069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.968677+0000","last_change":"2024-03-14T09:52:09.761930+0000","last_active":"2024-03-14T09:52:10.968677+0000","last_peered":"2024-03-14T09:52:10.968677+0000","last_clean":"2024-03-14T09:52:10.968677+0000","last_became_active":"2024-03-14T09:52:09.761515+0000","last_became_peered":"2024-03-14T09:52:09.761515+0000","last_unstale":"2024-03-14T09:52:10.968677+0000","last_undegraded":"2024-03-14T09:52:10.968677+0000","last_fullsized":"2024-03-14T09:52:10.968677+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T15:49:39.262277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"83'3","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.715856+0000","last_change":"2024-03-14T09:52:05.233579+0000","last_active":"2024-03-14T09:52:28.715856+0000","last_peered":"2024-03-14T09:52:28.715856+0000","last_clean":"2024-03-14T09:52:28.715856+0000","last_became_active":"2024-03-14T09:52:05.231605+0000","last_became_peered":"2024-03-14T09:52:05.231605+0000","last_unstale":"2024-03-14T09:52:28.715856+0000","last_undegraded":"2024-03-14T09:52:28.715856+0000","last_fullsized":"2024-03-14T09:52:28.715856+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:55:38.282203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.747784+0000","last_change":"2024-03-14T09:52:05.736586+0000","last_active":"2024-03-14T09:52:09.747784+0000","last_peered":"2024-03-14T09:52:09.747784+0000","last_clean":"2024-03-14T09:52:09.747784+0000","last_became_active":"2024-03-14T09:52:05.736235+0000","last_became_peered":"2024-03-14T09:52:05.736235+0000","last_unstale":"2024-03-14T09:52:09.747784+0000","last_undegraded":"2024-03-14T09:52:09.747784+0000","last_fullsized":"2024-03-14T09:52:09.747784+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:46:53.293698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.486435+0000","last_change":"2024-03-14T09:52:09.768203+0000","last_active":"2024-03-14T09:52:10.486435+0000","last_peered":"2024-03-14T09:52:10.486435+0000","last_clean":"2024-03-14T09:52:10.486435+0000","last_became_active":"2024-03-14T09:52:09.767777+0000","last_became_peered":"2024-03-14T09:52:09.767777+0000","last_unstale":"2024-03-14T09:52:10.486435+0000","last_undegraded":"2024-03-14T09:52:10.486435+0000","last_fullsized":"2024-03-14T09:52:10.486435+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:20:47.598905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.757962+0000","last_change":"2024-03-14T09:52:05.232359+0000","last_active":"2024-03-14T09:52:09.757962+0000","last_peered":"2024-03-14T09:52:09.757962+0000","last_clean":"2024-03-14T09:52:09.757962+0000","last_became_active":"2024-03-14T09:52:05.231977+0000","last_became_peered":"2024-03-14T09:52:05.231977+0000","last_unstale":"2024-03-14T09:52:09.757962+0000","last_undegraded":"2024-03-14T09:52:09.757962+0000","last_fullsized":"2024-03-14T09:52:09.757962+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:32.678157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.743968+0000","last_change":"2024-03-14T09:52:05.735822+0000","last_active":"2024-03-14T09:52:09.743968+0000","last_peered":"2024-03-14T09:52:09.743968+0000","last_clean":"2024-03-14T09:52:09.743968+0000","last_became_active":"2024-03-14T09:52:05.735059+0000","last_became_peered":"2024-03-14T09:52:05.735059+0000","last_unstale":"2024-03-14T09:52:09.743968+0000","last_undegraded":"2024-03-14T09:52:09.743968+0000","last_fullsized":"2024-03-14T09:52:09.743968+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T15:22:33.033689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.486463+0000","last_change":"2024-03-14T09:52:09.759794+0000","last_active":"2024-03-14T09:52:10.486463+0000","last_peered":"2024-03-14T09:52:10.486463+0000","last_clean":"2024-03-14T09:52:10.486463+0000","last_became_active":"2024-03-14T09:52:09.759366+0000","last_became_peered":"2024-03-14T09:52:09.759366+0000","last_unstale":"2024-03-14T09:52:10.486463+0000","last_undegraded":"2024-03-14T09:52:10.486463+0000","last_fullsized":"2024-03-14T09:52:10.486463+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:52:40.641264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"83'4","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:28.716964+0000","last_change":"2024-03-14T09:52:05.232061+0000","last_active":"2024-03-14T09:52:28.716964+0000","last_peered":"2024-03-14T09:52:28.716964+0000","last_clean":"2024-03-14T09:52:28.716964+0000","last_became_active":"2024-03-14T09:52:05.231720+0000","last_became_peered":"2024-03-14T09:52:05.231720+0000","last_unstale":"2024-03-14T09:52:28.716964+0000","last_undegraded":"2024-03-14T09:52:28.716964+0000","last_fullsized":"2024-03-14T09:52:28.716964+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:37:59.947955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744188+0000","last_change":"2024-03-14T09:52:05.734073+0000","last_active":"2024-03-14T09:52:09.744188+0000","last_peered":"2024-03-14T09:52:09.744188+0000","last_clean":"2024-03-14T09:52:09.744188+0000","last_became_active":"2024-03-14T09:52:05.731748+0000","last_became_peered":"2024-03-14T09:52:05.731748+0000","last_unstale":"2024-03-14T09:52:09.744188+0000","last_undegraded":"2024-03-14T09:52:09.744188+0000","last_fullsized":"2024-03-14T09:52:09.744188+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:01:11.789162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.763969+0000","last_change":"2024-03-14T09:52:09.763309+0000","last_active":"2024-03-14T09:52:09.763969+0000","last_peered":"2024-03-14T09:52:09.763969+0000","last_clean":"2024-03-14T09:52:09.763969+0000","last_became_active":"2024-03-14T09:52:09.762868+0000","last_became_peered":"2024-03-14T09:52:09.762868+0000","last_unstale":"2024-03-14T09:52:09.763969+0000","last_undegraded":"2024-03-14T09:52:09.763969+0000","last_fullsized":"2024-03-14T09:52:09.763969+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:30:03.403911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"83'3","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:29.721134+0000","last_change":"2024-03-14T09:52:05.232310+0000","last_active":"2024-03-14T09:52:29.721134+0000","last_peered":"2024-03-14T09:52:29.721134+0000","last_clean":"2024-03-14T09:52:29.721134+0000","last_became_active":"2024-03-14T09:52:05.231931+0000","last_became_peered":"2024-03-14T09:52:05.231931+0000","last_unstale":"2024-03-14T09:52:29.721134+0000","last_undegraded":"2024-03-14T09:52:29.721134+0000","last_fullsized":"2024-03-14T09:52:29.721134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:35:44.274800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744698+0000","last_change":"2024-03-14T09:52:05.731797+0000","last_active":"2024-03-14T09:52:09.744698+0000","last_peered":"2024-03-14T09:52:09.744698+0000","last_clean":"2024-03-14T09:52:09.744698+0000","last_became_active":"2024-03-14T09:52:05.730478+0000","last_became_peered":"2024-03-14T09:52:05.730478+0000","last_unstale":"2024-03-14T09:52:09.744698+0000","last_undegraded":"2024-03-14T09:52:09.744698+0000","last_fullsized":"2024-03-14T09:52:09.744698+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:52:50.684430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.966910+0000","last_change":"2024-03-14T09:52:09.772786+0000","last_active":"2024-03-14T09:52:10.966910+0000","last_peered":"2024-03-14T09:52:10.966910+0000","last_clean":"2024-03-14T09:52:10.966910+0000","last_became_active":"2024-03-14T09:52:09.772198+0000","last_became_peered":"2024-03-14T09:52:09.772198+0000","last_unstale":"2024-03-14T09:52:10.966910+0000","last_undegraded":"2024-03-14T09:52:10.966910+0000","last_fullsized":"2024-03-14T09:52:10.966910+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:28:56.620266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"78'1","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.968653+0000","last_change":"2024-03-14T09:52:05.237657+0000","last_active":"2024-03-14T09:52:10.968653+0000","last_peered":"2024-03-14T09:52:10.968653+0000","last_clean":"2024-03-14T09:52:10.968653+0000","last_became_active":"2024-03-14T09:52:05.237381+0000","last_became_peered":"2024-03-14T09:52:05.237381+0000","last_unstale":"2024-03-14T09:52:10.968653+0000","last_undegraded":"2024-03-14T09:52:10.968653+0000","last_fullsized":"2024-03-14T09:52:10.968653+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:34:44.372128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744062+0000","last_change":"2024-03-14T09:52:05.730454+0000","last_active":"2024-03-14T09:52:09.744062+0000","last_peered":"2024-03-14T09:52:09.744062+0000","last_clean":"2024-03-14T09:52:09.744062+0000","last_became_active":"2024-03-14T09:52:05.730046+0000","last_became_peered":"2024-03-14T09:52:05.730046+0000","last_unstale":"2024-03-14T09:52:09.744062+0000","last_undegraded":"2024-03-14T09:52:09.744062+0000","last_fullsized":"2024-03-14T09:52:09.744062+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:33:19.296993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-14T09:52:10.968408+0000","last_change":"2024-03-14T09:52:09.755969+0000","last_active":"2024-03-14T09:52:10.968408+0000","last_peered":"2024-03-14T09:52:10.968408+0000","last_clean":"2024-03-14T09:52:10.968408+0000","last_became_active":"2024-03-14T09:52:09.755512+0000","last_became_peered":"2024-03-14T09:52:09.755512+0000","last_unstale":"2024-03-14T09:52:10.968408+0000","last_undegraded":"2024-03-14T09:52:10.968408+0000","last_fullsized":"2024-03-14T09:52:10.968408+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:31:03.388090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748616+0000","last_change":"2024-03-14T09:52:05.232676+0000","last_active":"2024-03-14T09:52:09.748616+0000","last_peered":"2024-03-14T09:52:09.748616+0000","last_clean":"2024-03-14T09:52:09.748616+0000","last_became_active":"2024-03-14T09:52:05.232332+0000","last_became_peered":"2024-03-14T09:52:05.232332+0000","last_unstale":"2024-03-14T09:52:09.748616+0000","last_undegraded":"2024-03-14T09:52:09.748616+0000","last_fullsized":"2024-03-14T09:52:09.748616+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:06:11.020555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744295+0000","last_change":"2024-03-14T09:52:05.733338+0000","last_active":"2024-03-14T09:52:09.744295+0000","last_peered":"2024-03-14T09:52:09.744295+0000","last_clean":"2024-03-14T09:52:09.744295+0000","last_became_active":"2024-03-14T09:52:05.732994+0000","last_became_peered":"2024-03-14T09:52:05.732994+0000","last_unstale":"2024-03-14T09:52:09.744295+0000","last_undegraded":"2024-03-14T09:52:09.744295+0000","last_fullsized":"2024-03-14T09:52:09.744295+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:31:25.085916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.747878+0000","last_change":"2024-03-14T09:52:05.733121+0000","last_active":"2024-03-14T09:52:09.747878+0000","last_peered":"2024-03-14T09:52:09.747878+0000","last_clean":"2024-03-14T09:52:09.747878+0000","last_became_active":"2024-03-14T09:52:05.732671+0000","last_became_peered":"2024-03-14T09:52:05.732671+0000","last_unstale":"2024-03-14T09:52:09.747878+0000","last_undegraded":"2024-03-14T09:52:09.747878+0000","last_fullsized":"2024-03-14T09:52:09.747878+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:18:54.180131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.751482+0000","last_change":"2024-03-14T09:52:05.736534+0000","last_active":"2024-03-14T09:52:09.751482+0000","last_peered":"2024-03-14T09:52:09.751482+0000","last_clean":"2024-03-14T09:52:09.751482+0000","last_became_active":"2024-03-14T09:52:05.736202+0000","last_became_peered":"2024-03-14T09:52:05.736202+0000","last_unstale":"2024-03-14T09:52:09.751482+0000","last_undegraded":"2024-03-14T09:52:09.751482+0000","last_fullsized":"2024-03-14T09:52:09.751482+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:22:46.161561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748730+0000","last_change":"2024-03-14T09:52:05.739434+0000","last_active":"2024-03-14T09:52:09.748730+0000","last_peered":"2024-03-14T09:52:09.748730+0000","last_clean":"2024-03-14T09:52:09.748730+0000","last_became_active":"2024-03-14T09:52:05.738023+0000","last_became_peered":"2024-03-14T09:52:05.738023+0000","last_unstale":"2024-03-14T09:52:09.748730+0000","last_undegraded":"2024-03-14T09:52:09.748730+0000","last_fullsized":"2024-03-14T09:52:09.748730+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:51:44.173645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.751960+0000","last_change":"2024-03-14T09:52:05.751367+0000","last_active":"2024-03-14T09:52:09.751960+0000","last_peered":"2024-03-14T09:52:09.751960+0000","last_clean":"2024-03-14T09:52:09.751960+0000","last_became_active":"2024-03-14T09:52:05.750816+0000","last_became_peered":"2024-03-14T09:52:05.750816+0000","last_unstale":"2024-03-14T09:52:09.751960+0000","last_undegraded":"2024-03-14T09:52:09.751960+0000","last_fullsized":"2024-03-14T09:52:09.751960+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:57:58.403356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.906111+0000","last_change":"2024-03-14T09:52:05.746652+0000","last_active":"2024-03-14T09:52:09.906111+0000","last_peered":"2024-03-14T09:52:09.906111+0000","last_clean":"2024-03-14T09:52:09.906111+0000","last_became_active":"2024-03-14T09:52:05.746214+0000","last_became_peered":"2024-03-14T09:52:05.746214+0000","last_unstale":"2024-03-14T09:52:09.906111+0000","last_undegraded":"2024-03-14T09:52:09.906111+0000","last_fullsized":"2024-03-14T09:52:09.906111+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:21:39.244617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748036+0000","last_change":"2024-03-14T09:52:05.735078+0000","last_active":"2024-03-14T09:52:09.748036+0000","last_peered":"2024-03-14T09:52:09.748036+0000","last_clean":"2024-03-14T09:52:09.748036+0000","last_became_active":"2024-03-14T09:52:05.734632+0000","last_became_peered":"2024-03-14T09:52:05.734632+0000","last_unstale":"2024-03-14T09:52:09.748036+0000","last_undegraded":"2024-03-14T09:52:09.748036+0000","last_fullsized":"2024-03-14T09:52:09.748036+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:08:45.687765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.758608+0000","last_change":"2024-03-14T09:52:05.739758+0000","last_active":"2024-03-14T09:52:09.758608+0000","last_peered":"2024-03-14T09:52:09.758608+0000","last_clean":"2024-03-14T09:52:09.758608+0000","last_became_active":"2024-03-14T09:52:05.739423+0000","last_became_peered":"2024-03-14T09:52:05.739423+0000","last_unstale":"2024-03-14T09:52:09.758608+0000","last_undegraded":"2024-03-14T09:52:09.758608+0000","last_fullsized":"2024-03-14T09:52:09.758608+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:47:22.594910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.744583+0000","last_change":"2024-03-14T09:52:05.733281+0000","last_active":"2024-03-14T09:52:09.744583+0000","last_peered":"2024-03-14T09:52:09.744583+0000","last_clean":"2024-03-14T09:52:09.744583+0000","last_became_active":"2024-03-14T09:52:05.730081+0000","last_became_peered":"2024-03-14T09:52:05.730081+0000","last_unstale":"2024-03-14T09:52:09.744583+0000","last_undegraded":"2024-03-14T09:52:09.744583+0000","last_fullsized":"2024-03-14T09:52:09.744583+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:40:26.354063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.745308+0000","last_change":"2024-03-14T09:52:05.726938+0000","last_active":"2024-03-14T09:52:09.745308+0000","last_peered":"2024-03-14T09:52:09.745308+0000","last_clean":"2024-03-14T09:52:09.745308+0000","last_became_active":"2024-03-14T09:52:05.726602+0000","last_became_peered":"2024-03-14T09:52:05.726602+0000","last_unstale":"2024-03-14T09:52:09.745308+0000","last_undegraded":"2024-03-14T09:52:09.745308+0000","last_fullsized":"2024-03-14T09:52:09.745308+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:26:20.354368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.743577+0000","last_change":"2024-03-14T09:52:05.740348+0000","last_active":"2024-03-14T09:52:09.743577+0000","last_peered":"2024-03-14T09:52:09.743577+0000","last_clean":"2024-03-14T09:52:09.743577+0000","last_became_active":"2024-03-14T09:52:05.740055+0000","last_became_peered":"2024-03-14T09:52:05.740055+0000","last_unstale":"2024-03-14T09:52:09.743577+0000","last_undegraded":"2024-03-14T09:52:09.743577+0000","last_fullsized":"2024-03-14T09:52:09.743577+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:07:41.498862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.749884+0000","last_change":"2024-03-14T09:52:05.742945+0000","last_active":"2024-03-14T09:52:09.749884+0000","last_peered":"2024-03-14T09:52:09.749884+0000","last_clean":"2024-03-14T09:52:09.749884+0000","last_became_active":"2024-03-14T09:52:05.742483+0000","last_became_peered":"2024-03-14T09:52:05.742483+0000","last_unstale":"2024-03-14T09:52:09.749884+0000","last_undegraded":"2024-03-14T09:52:09.749884+0000","last_fullsized":"2024-03-14T09:52:09.749884+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:05:31.053608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.906081+0000","last_change":"2024-03-14T09:52:05.737822+0000","last_active":"2024-03-14T09:52:09.906081+0000","last_peered":"2024-03-14T09:52:09.906081+0000","last_clean":"2024-03-14T09:52:09.906081+0000","last_became_active":"2024-03-14T09:52:05.735294+0000","last_became_peered":"2024-03-14T09:52:05.735294+0000","last_unstale":"2024-03-14T09:52:09.906081+0000","last_undegraded":"2024-03-14T09:52:09.906081+0000","last_fullsized":"2024-03-14T09:52:09.906081+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:06:28.437259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-14T09:52:09.748615+0000","last_change":"2024-03-14T09:52:05.746991+0000","last_active":"2024-03-14T09:52:09.748615+0000","last_peered":"2024-03-14T09:52:09.748615+0000","last_clean":"2024-03-14T09:52:09.748615+0000","last_became_active":"2024-03-14T09:52:05.746558+0000","last_became_peered":"2024-03-14T09:52:05.746558+0000","last_unstale":"2024-03-14T09:52:09.748615+0000","last_undegraded":"2024-03-14T09:52:09.748615+0000","last_fullsized":"2024-03-14T09:52:09.748615+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:46:11.540134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247929,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7219,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":14,"num_read_kb":12,"num_write":117,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":57613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579919,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":447276,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.379}]},{"osd":1,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.57999999999999996}]},{"osd":2,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.35899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.497}]},{"osd":6,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.61799999999999999}]},{"osd":7,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.71599999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":9,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.66400000000000003}]},{"osd":10,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.70499999999999996}]}]},{"osd":10,"up_from":67,"seq":287762808849,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":1041051,"data_compressed":47579,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":1,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.443}]},{"osd":2,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.51300000000000001}]},{"osd":3,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.73999999999999999}]},{"osd":4,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.54200000000000004}]},{"osd":5,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.755}]},{"osd":6,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.876},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.81699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":8,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.78200000000000003}]},{"osd":9,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.85699999999999998}]},{"osd":11,"last update":"Thu Mar 14 09:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.77300000000000002}]}]},{"osd":9,"up_from":62,"seq":266287972375,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":921600,"data_stored":451608,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.38100000000000001}]},{"osd":1,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.624}]},{"osd":2,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.48199999999999998}]},{"osd":3,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.40699999999999997}]},{"osd":4,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.60699999999999998}]},{"osd":5,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.66800000000000004}]},{"osd":6,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.63300000000000001}]},{"osd":8,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.72899999999999998}]},{"osd":10,"last update":"Thu Mar 14 09:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.752},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.75900000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]}]},{"osd":8,"up_from":55,"seq":236223201305,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":447101,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.159,"5min":1.159,"15min":1.159},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":9.7769999999999992,"5min":9.7769999999999992,"15min":9.7769999999999992},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.64600000000000002}]},{"osd":2,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.53300000000000003}]},{"osd":3,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.435},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.68400000000000005}]},{"osd":4,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.56499999999999995}]},{"osd":5,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.40999999999999998}]},{"osd":6,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.69799999999999995}]},{"osd":7,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.628},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.61699999999999999}]},{"osd":9,"last update":"Thu Mar 14 09:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.58199999999999996}]},{"osd":10,"last update":"Thu Mar 14 09:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.67200000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]}]},{"osd":7,"up_from":48,"seq":206158430238,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":1039150,"data_compressed":47579,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.40999999999999998}]},{"osd":1,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.42699999999999999,"15min":0.41999999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.41699999999999998,"15min":0.40899999999999997},"min":{"1min":0.29199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.57699999999999996,"15min":0.56799999999999995},"min":{"1min":0.441,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.69499999999999995}]},{"osd":3,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.42199999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.47499999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.55700000000000005}]},{"osd":4,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.48799999999999999,"15min":0.47799999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.503,"15min":0.497},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.44500000000000001}]},{"osd":5,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.58799999999999997,"15min":0.57599999999999996},"min":{"1min":0.47799999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.71699999999999997}]},{"osd":6,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.64300000000000002,"15min":0.63500000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66500000000000004,"15min":0.65900000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.56999999999999995}]},{"osd":8,"last update":"Thu Mar 14 09:51:33 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.65500000000000003}]},{"osd":9,"last update":"Thu Mar 14 09:51:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.68100000000000005}]},{"osd":10,"last update":"Thu Mar 14 09:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.73899999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]}]},{"osd":6,"up_from":43,"seq":184683593760,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":446694,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.47399999999999998,"15min":0.46999999999999997},"min":{"1min":0.40000000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.76400000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.48099999999999998,"15min":0.48299999999999998},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.66100000000000003,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.39300000000000002}]},{"osd":1,"last update":"Thu Mar 14 09:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.65300000000000002,"15min":0.64100000000000001},"min":{"1min":0.53400000000000003,"5min":0.433,"15min":0.433},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.60999999999999999,"15min":0.59999999999999998},"min":{"1min":0.46500000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81799999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.627}]},{"osd":2,"last update":"Thu Mar 14 09:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63200000000000001,"15min":0.63},"min":{"1min":0.49399999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.76900000000000002,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.67300000000000004,"15min":0.66900000000000004},"min":{"1min":0.48599999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.89400000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.80400000000000005}]},{"osd":3,"last update":"Thu Mar 14 09:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.46500000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.83699999999999997,"5min":0.874,"15min":0.874},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.39300000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.41599999999999998}]},{"osd":4,"last update":"Thu Mar 14 09:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.64600000000000002,"15min":0.63900000000000001},"min":{"1min":0.54300000000000004,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.877,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.72199999999999998,"15min":0.71399999999999997},"min":{"1min":0.56699999999999995,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.78100000000000003}]},{"osd":5,"last update":"Thu Mar 14 09:51:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.76300000000000001,"15min":0.77500000000000002},"min":{"1min":0.54100000000000004,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.84499999999999997,"5min":3.1059999999999999,"15min":3.1059999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.70599999999999996,"15min":0.70799999999999996},"min":{"1min":0.503,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.85399999999999998,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.54500000000000004}]},{"osd":7,"last update":"Thu Mar 14 09:52:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.69299999999999995,"15min":0.68700000000000006},"min":{"1min":0.51100000000000001,"5min":0.495,"15min":0.495},"max":{"1min":0.90200000000000002,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.77300000000000002,"15min":0.77100000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.82399999999999995}]},{"osd":8,"last update":"Thu Mar 14 09:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.58899999999999997}]},{"osd":9,"last update":"Thu Mar 14 09:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.56299999999999994}]},{"osd":10,"last update":"Thu Mar 14 09:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.66400000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":909312,"data_stored":450489,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54700000000000004,"15min":0.56100000000000005},"min":{"1min":0.378,"5min":0.315,"15min":0.315},"max":{"1min":0.74099999999999999,"5min":1.917,"15min":1.917},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53900000000000003,"15min":0.53300000000000003},"min":{"1min":0.40500000000000003,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.746,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.63700000000000001}]},{"osd":2,"last update":"Thu Mar 14 09:51:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.42199999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69899999999999995,"5min":1.885,"15min":1.885},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.52100000000000002,"15min":0.51900000000000002},"min":{"1min":0.39200000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76500000000000001,"5min":1.167,"15min":1.167},"last":0.748}]},{"osd":3,"last update":"Thu Mar 14 09:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.39300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76800000000000002,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.75800000000000001,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.60999999999999999}]},{"osd":4,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.378,"15min":0.36399999999999999},"min":{"1min":0.31,"5min":0.249,"15min":0.249},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.437},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.42999999999999999,"15min":0.40000000000000002},"min":{"1min":0.316,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":2.1339999999999999,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":0.45800000000000002}]},{"osd":5,"last update":"Thu Mar 14 09:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.48199999999999998,"15min":0.47099999999999997},"min":{"1min":0.40500000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.42799999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.79300000000000004}]},{"osd":6,"last update":"Thu Mar 14 09:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63300000000000001,"15min":0.627},"min":{"1min":0.50900000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.83099999999999996,"15min":0.86899999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77800000000000002,"5min":5.5229999999999997,"15min":5.5229999999999997},"last":0.48599999999999999}]},{"osd":7,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.51100000000000001,"15min":0.48899999999999999},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.49199999999999999,"15min":0.48499999999999999},"min":{"1min":0.33300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.53200000000000003}]},{"osd":8,"last update":"Thu Mar 14 09:51:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.81599999999999995}]},{"osd":9,"last update":"Thu Mar 14 09:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.72399999999999998}]},{"osd":10,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.871}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":448172,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 14 09:52:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.56699999999999995,"15min":0.52900000000000003},"min":{"1min":0.34999999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.58899999999999997,"15min":0.54900000000000004},"min":{"1min":0.28199999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":0.71599999999999997}]},{"osd":2,"last update":"Thu Mar 14 09:51:43 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.58699999999999997,"15min":0.54300000000000004},"min":{"1min":0.45300000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":3.7989999999999999,"5min":3.7989999999999999,"15min":3.7989999999999999},"last":0.63},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.59399999999999997,"15min":0.55800000000000005},"min":{"1min":0.45000000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.93899999999999995,"5min":1.04,"15min":1.04},"last":0.753}]},{"osd":3,"last update":"Thu Mar 14 09:51:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.47599999999999998,"15min":0.46400000000000002},"min":{"1min":0.31,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.87,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.495},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.48499999999999999,"15min":0.47799999999999998},"min":{"1min":0.36799999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71099999999999997,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.58599999999999997}]},{"osd":4,"last update":"Thu Mar 14 09:52:22 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.71399999999999997,"15min":0.66600000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.8220000000000001,"5min":3.8220000000000001,"15min":3.8220000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.61499999999999999,"15min":0.58299999999999996},"min":{"1min":0.38900000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.573,"5min":3.573,"15min":3.573},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Mar 14 09:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.70099999999999996,"15min":0.66600000000000004},"min":{"1min":0.54800000000000004,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":3.863,"5min":3.863,"15min":3.863},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.67300000000000004,"15min":0.66900000000000004},"min":{"1min":0.48799999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.73199999999999998}]},{"osd":6,"last update":"Thu Mar 14 09:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59599999999999997,"15min":0.58999999999999997},"min":{"1min":0.39900000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.107,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.748,"5min":0.71299999999999997,"15min":0.70699999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.0940000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.81499999999999995}]},{"osd":7,"last update":"Thu Mar 14 09:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.7509999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.67600000000000005}]},{"osd":8,"last update":"Thu Mar 14 09:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.872},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.76100000000000001}]},{"osd":9,"last update":"Thu Mar 14 09:51:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.745},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.93000000000000005}]},{"osd":10,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.76900000000000002}]},{"osd":11,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":2.5590000000000002,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":3.71,"5min":3.71,"15min":3.71},"last":0.82999999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":471091,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.52000000000000002,"15min":0.51400000000000001},"min":{"1min":0.39100000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.76500000000000001,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.35199999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.74299999999999999,"5min":0.879,"15min":0.879},"last":0.57599999999999996}]},{"osd":1,"last update":"Thu Mar 14 09:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.52800000000000002,"15min":0.51600000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.74199999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80600000000000005,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.64700000000000002}]},{"osd":3,"last update":"Thu Mar 14 09:52:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53100000000000003,"15min":0.51900000000000002},"min":{"1min":0.442,"5min":0.37,"15min":0.37},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.627},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55500000000000005,"15min":0.54400000000000004},"min":{"1min":0.41999999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78900000000000003,"5min":0.871,"15min":0.871},"last":0.77500000000000002}]},{"osd":4,"last update":"Thu Mar 14 09:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.73199999999999998,"5min":0.87,"15min":0.87},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.68300000000000005}]},{"osd":5,"last update":"Thu Mar 14 09:51:38 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.44500000000000001,"15min":0.437},"min":{"1min":0.32600000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.47399999999999998,"15min":0.47199999999999998},"min":{"1min":0.32900000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.72699999999999998,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.72599999999999998}]},{"osd":6,"last update":"Thu Mar 14 09:52:02 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.52900000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.83899999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65900000000000003,"15min":0.65700000000000003},"min":{"1min":0.46700000000000003,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.86599999999999999}]},{"osd":7,"last update":"Thu Mar 14 09:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.755},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.96599999999999997}]},{"osd":8,"last update":"Thu Mar 14 09:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.84299999999999997}]},{"osd":9,"last update":"Thu Mar 14 09:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.995}]},{"osd":10,"last update":"Thu Mar 14 09:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.88300000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0169999999999999}]}]},{"osd":3,"up_from":24,"seq":103079215150,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":446978,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.40600000000000003,"15min":0.39600000000000002},"min":{"1min":0.34000000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.372},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.42799999999999999,"15min":0.42399999999999999},"min":{"1min":0.33400000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.64500000000000002,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Mar 14 09:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.60599999999999998,"15min":0.58999999999999997},"min":{"1min":0.46100000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.60199999999999998,"15min":0.58299999999999996},"min":{"1min":0.46500000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.47799999999999998}]},{"osd":2,"last update":"Thu Mar 14 09:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.61799999999999999,"15min":0.60199999999999998},"min":{"1min":0.46400000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.63500000000000001,"15min":0.621},"min":{"1min":0.47899999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.71099999999999997}]},{"osd":4,"last update":"Thu Mar 14 09:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.61799999999999999,"15min":0.59599999999999997},"min":{"1min":0.42099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.61399999999999999,"15min":0.59899999999999998},"min":{"1min":0.48199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.68799999999999994}]},{"osd":5,"last update":"Thu Mar 14 09:51:36 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.68600000000000005,"15min":0.68200000000000005},"min":{"1min":0.48099999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69199999999999995,"15min":0.69099999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.04,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.76500000000000001}]},{"osd":6,"last update":"Thu Mar 14 09:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.65900000000000003,"15min":0.65400000000000003},"min":{"1min":0.42999999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96599999999999997,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.65500000000000003,"15min":0.64200000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.754}]},{"osd":7,"last update":"Thu Mar 14 09:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.53500000000000003,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.109,"5min":1.123,"15min":1.123},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77000000000000002,"15min":0.76900000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.64500000000000002}]},{"osd":8,"last update":"Thu Mar 14 09:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.871}]},{"osd":9,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.80600000000000005}]},{"osd":10,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.77200000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986217,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":909312,"data_stored":447290,"data_compressed":11684,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.40100000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71599999999999997,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.52300000000000002,"15min":0.51600000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.35099999999999998},"min":{"1min":0.26600000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.29199999999999998}]},{"osd":2,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.55900000000000005,"15min":0.53600000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53100000000000003,"15min":0.51500000000000001},"min":{"1min":0.433,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.40699999999999997,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76500000000000001,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55400000000000005,"15min":0.55700000000000005},"min":{"1min":0.39800000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.46100000000000002}]},{"osd":5,"last update":"Thu Mar 14 09:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.51700000000000002,"15min":0.50700000000000001},"min":{"1min":0.34799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.52500000000000002,"15min":0.52300000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.66600000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Mar 14 09:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.57799999999999996,"15min":0.57099999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.438,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.65000000000000002}]},{"osd":7,"last update":"Thu Mar 14 09:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.379,"5min":0.318,"15min":0.318},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.45600000000000002,"15min":0.45000000000000001},"min":{"1min":0.33200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.36599999999999999}]},{"osd":8,"last update":"Thu Mar 14 09:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.629},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.61699999999999999}]},{"osd":9,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.505},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.67200000000000004}]},{"osd":10,"last update":"Thu Mar 14 09:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.58699999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]},{"osd":5,"up_from":36,"seq":154618822694,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":1042586,"data_compressed":47579,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52600000000000002,"15min":0.52400000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73999999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54900000000000004,"15min":0.54500000000000004},"min":{"1min":0.39200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72799999999999998,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.51600000000000001,"15min":0.50900000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.52200000000000002,"15min":0.51000000000000001},"min":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.69599999999999995}]},{"osd":2,"last update":"Thu Mar 14 09:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.38500000000000001,"15min":0.375},"min":{"1min":0.27200000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38400000000000001,"15min":0.376},"min":{"1min":0.28999999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.41599999999999998}]},{"osd":3,"last update":"Thu Mar 14 09:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.42399999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58399999999999996,"15min":0.58099999999999996},"min":{"1min":0.44700000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.79900000000000004,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.51300000000000001}]},{"osd":4,"last update":"Thu Mar 14 09:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.55600000000000005,"15min":0.54800000000000004},"min":{"1min":0.41999999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.54100000000000004,"15min":0.53100000000000003},"min":{"1min":0.36299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67700000000000005}]},{"osd":6,"last update":"Thu Mar 14 09:51:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.626,"15min":0.625},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82199999999999995,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.629,"15min":0.623},"min":{"1min":0.47799999999999998,"5min":0.432,"15min":0.432},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.47499999999999998}]},{"osd":7,"last update":"Thu Mar 14 09:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.60799999999999998,"15min":0.59299999999999997},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.59899999999999998,"15min":0.58499999999999996},"min":{"1min":0.46400000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.66800000000000004}]},{"osd":8,"last update":"Thu Mar 14 09:51:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.38100000000000001}]},{"osd":9,"last update":"Thu Mar 14 09:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.63900000000000001}]},{"osd":10,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.71299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34227,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34227,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":41727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-14T09:52:35.039 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-14T09:52:35.040 INFO:tasks.cephadm:Setting up client nodes... 2024-03-14T09:52:35.040 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-14T09:52:35.431 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:35.960 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:35 smithi078 ceph-mon[38143]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:52:35.960 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:35 smithi078 ceph-mon[38143]: pgmap v250: 129 pgs: 129 active+clean; 619 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2024-03-14T09:52:35.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:35 smithi152 ceph-mon[48279]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:52:35.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:35 smithi152 ceph-mon[48279]: pgmap v250: 129 pgs: 129 active+clean; 619 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2024-03-14T09:52:36.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:35 smithi132 ceph-mon[47959]: from='client.14838 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:52:36.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:35 smithi132 ceph-mon[47959]: pgmap v250: 129 pgs: 129 active+clean; 619 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.7 KiB/s wr, 10 op/s 2024-03-14T09:52:36.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.662+0000 7f2795200700 1 -- 172.21.15.78:0/3929362641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2790100d90 msgr2=0x7f27901011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:36.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.662+0000 7f2795200700 1 --2- 172.21.15.78:0/3929362641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2790100d90 0x7f27901011b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f27800099c0 tx=0x7f2780009cd0 comp rx=0 tx=0).stop 2024-03-14T09:52:36.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.662+0000 7f2795200700 1 -- 172.21.15.78:0/3929362641 shutdown_connections 2024-03-14T09:52:36.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.662+0000 7f2795200700 1 --2- 172.21.15.78:0/3929362641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2790102930 0x7f27901091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:36.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.662+0000 7f2795200700 1 --2- 172.21.15.78:0/3929362641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2790100d90 0x7f27901011b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.662+0000 7f2795200700 1 --2- 172.21.15.78:0/3929362641 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2790101f90 0x7f27901023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.662+0000 7f2795200700 1 -- 172.21.15.78:0/3929362641 >> 172.21.15.78:0/3929362641 conn(0x7f27900fc330 msgr2=0x7f27900fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.663+0000 7f2795200700 1 -- 172.21.15.78:0/3929362641 shutdown_connections 2024-03-14T09:52:36.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.663+0000 7f2795200700 1 -- 172.21.15.78:0/3929362641 wait complete. 2024-03-14T09:52:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.664+0000 7f2795200700 1 Processor -- start 2024-03-14T09:52:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.664+0000 7f2795200700 1 -- start start 2024-03-14T09:52:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.664+0000 7f2795200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2790100d90 0x7f2790199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.664+0000 7f2795200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2790101f90 0x7f2790199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:36.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.664+0000 7f2795200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2790102930 0x7f27901a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.664+0000 7f2795200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27901a0e80 con 0x7f2790102930 2024-03-14T09:52:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.664+0000 7f2795200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27901a0fc0 con 0x7f2790100d90 2024-03-14T09:52:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.664+0000 7f2795200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27901a1150 con 0x7f2790101f90 2024-03-14T09:52:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.664+0000 7f278e800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2790102930 0x7f27901a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.664+0000 7f278de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2790100d90 0x7f2790199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.664+0000 7f278e800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2790102930 0x7f27901a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50482/0 (socket says 172.21.15.78:50482) 2024-03-14T09:52:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.664+0000 7f278e800700 1 -- 172.21.15.78:0/1418779679 learned_addr learned my addr 172.21.15.78:0/1418779679 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.664+0000 7f278d400700 1 --2- 172.21.15.78:0/1418779679 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2790101f90 0x7f2790199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.665+0000 7f278e800700 1 -- 172.21.15.78:0/1418779679 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2790101f90 msgr2=0x7f2790199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.665+0000 7f278e800700 1 --2- 172.21.15.78:0/1418779679 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2790101f90 0x7f2790199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.665+0000 7f278e800700 1 -- 172.21.15.78:0/1418779679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2790100d90 msgr2=0x7f2790199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.665+0000 7f278e800700 1 --2- 172.21.15.78:0/1418779679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2790100d90 0x7f2790199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.665+0000 7f278e800700 1 -- 172.21.15.78:0/1418779679 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27800095f0 con 0x7f2790102930 2024-03-14T09:52:36.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.665+0000 7f278e800700 1 --2- 172.21.15.78:0/1418779679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2790102930 0x7f27901a08b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f278800ef60 tx=0x7f278800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.665+0000 7f2786a00700 1 -- 172.21.15.78:0/1418779679 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2788018070 con 0x7f2790102930 2024-03-14T09:52:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.665+0000 7f2786a00700 1 -- 172.21.15.78:0/1418779679 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f278800f040 con 0x7f2790102930 2024-03-14T09:52:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.665+0000 7f2795200700 1 -- 172.21.15.78:0/1418779679 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27901a13d0 con 0x7f2790102930 2024-03-14T09:52:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.665+0000 7f2786a00700 1 -- 172.21.15.78:0/1418779679 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278800ccc0 con 0x7f2790102930 2024-03-14T09:52:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.665+0000 7f2795200700 1 -- 172.21.15.78:0/1418779679 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27901a18d0 con 0x7f2790102930 2024-03-14T09:52:36.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.666+0000 7f2795200700 1 -- 172.21.15.78:0/1418779679 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f279004ea80 con 0x7f2790102930 2024-03-14T09:52:36.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.668+0000 7f2786a00700 1 -- 172.21.15.78:0/1418779679 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f2788003a00 con 0x7f2790102930 2024-03-14T09:52:36.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.669+0000 7f2786a00700 1 --2- 172.21.15.78:0/1418779679 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27740a0ba0 0x7f27740a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:36.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.670+0000 7f2786a00700 1 -- 172.21.15.78:0/1418779679 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f27880cbe20 con 0x7f2790102930 2024-03-14T09:52:36.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.670+0000 7f278de00700 1 --2- 172.21.15.78:0/1418779679 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27740a0ba0 0x7f27740a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:36.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.670+0000 7f278de00700 1 --2- 172.21.15.78:0/1418779679 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27740a0ba0 0x7f27740a3060 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2780003730 tx=0x7f2780000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:36.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.674+0000 7f2786a00700 1 -- 172.21.15.78:0/1418779679 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f27880954d0 con 0x7f2790102930 2024-03-14T09:52:36.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.956+0000 7f2795200700 1 -- 172.21.15.78:0/1418779679 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f279010cc50 con 0x7f2790102930 2024-03-14T09:52:36.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.960+0000 7f2786a00700 1 -- 172.21.15.78:0/1418779679 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7f2788098af0 con 0x7f2790102930 2024-03-14T09:52:36.961 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-03-14T09:52:36.961 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDkyPJl3+MUORAAz0ajtCKryvxI85mNN4S3Xw== 2024-03-14T09:52:36.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.963+0000 7f2795200700 1 -- 172.21.15.78:0/1418779679 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27740a0ba0 msgr2=0x7f27740a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:36.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.963+0000 7f2795200700 1 --2- 172.21.15.78:0/1418779679 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27740a0ba0 0x7f27740a3060 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2780003730 tx=0x7f2780000bc0 comp rx=0 tx=0).stop 2024-03-14T09:52:36.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.963+0000 7f2795200700 1 -- 172.21.15.78:0/1418779679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2790102930 msgr2=0x7f27901a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:36.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.963+0000 7f2795200700 1 --2- 172.21.15.78:0/1418779679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2790102930 0x7f27901a08b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f278800ef60 tx=0x7f278800ef90 comp rx=0 tx=0).stop 2024-03-14T09:52:36.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.963+0000 7f2795200700 1 -- 172.21.15.78:0/1418779679 shutdown_connections 2024-03-14T09:52:36.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.963+0000 7f2795200700 1 --2- 172.21.15.78:0/1418779679 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27740a0ba0 0x7f27740a3060 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:36.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.963+0000 7f2795200700 1 --2- 172.21.15.78:0/1418779679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2790100d90 0x7f2790199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:36.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.963+0000 7f2795200700 1 --2- 172.21.15.78:0/1418779679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2790102930 0x7f27901a08b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:36.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.963+0000 7f2795200700 1 --2- 172.21.15.78:0/1418779679 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2790101f90 0x7f2790199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:36.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.963+0000 7f2795200700 1 -- 172.21.15.78:0/1418779679 >> 172.21.15.78:0/1418779679 conn(0x7f27900fc330 msgr2=0x7f2790107760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:36.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.963+0000 7f2795200700 1 -- 172.21.15.78:0/1418779679 shutdown_connections 2024-03-14T09:52:36.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:36.963+0000 7f2795200700 1 -- 172.21.15.78:0/1418779679 wait complete. 2024-03-14T09:52:37.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:37 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1418779679' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-14T09:52:37.235 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:37 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1418779679' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-14T09:52:37.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:37 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1418779679' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-14T09:52:37.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:37 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1418779679' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-14T09:52:37.423 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:52:37.423 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-14T09:52:37.424 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-14T09:52:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:37 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1418779679' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-14T09:52:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:37 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1418779679' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-14T09:52:37.518 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-14T09:52:37.518 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-14T09:52:37.518 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-14T09:52:37.919 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:38.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:38 smithi132 ceph-mon[47959]: pgmap v251: 129 pgs: 129 active+clean; 620 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-14T09:52:38.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:38 smithi078 ceph-mon[38143]: pgmap v251: 129 pgs: 129 active+clean; 620 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-14T09:52:38.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:38 smithi152 ceph-mon[48279]: pgmap v251: 129 pgs: 129 active+clean; 620 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-14T09:52:39.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.102+0000 7f47d1e00700 1 -- 172.21.15.78:0/3885301460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47cc100e30 msgr2=0x7f47cc1012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.102+0000 7f47d1e00700 1 --2- 172.21.15.78:0/3885301460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47cc100e30 0x7f47cc1012b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f47b8009990 tx=0x7f47b8009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.102+0000 7f47d1e00700 1 -- 172.21.15.78:0/3885301460 shutdown_connections 2024-03-14T09:52:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.102+0000 7f47d1e00700 1 --2- 172.21.15.78:0/3885301460 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc1017f0 0x7f47cc10dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.102+0000 7f47d1e00700 1 --2- 172.21.15.78:0/3885301460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47cc106ec0 0x7f47cc1072a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.102+0000 7f47d1e00700 1 --2- 172.21.15.78:0/3885301460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47cc100e30 0x7f47cc1012b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.103+0000 7f47d1e00700 1 -- 172.21.15.78:0/3885301460 >> 172.21.15.78:0/3885301460 conn(0x7f47cc077fa0 msgr2=0x7f47cc0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.103+0000 7f47d1e00700 1 -- 172.21.15.78:0/3885301460 shutdown_connections 2024-03-14T09:52:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.103+0000 7f47d1e00700 1 -- 172.21.15.78:0/3885301460 wait complete. 2024-03-14T09:52:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.103+0000 7f47d1e00700 1 Processor -- start 2024-03-14T09:52:39.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.104+0000 7f47d1e00700 1 -- start start 2024-03-14T09:52:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.104+0000 7f47d1e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc100e30 0x7f47cc0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.104+0000 7f47d1e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47cc1017f0 0x7f47cc0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.104+0000 7f47d1e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47cc106ec0 0x7f47cc073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.104+0000 7f47d1e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc1a5940 con 0x7f47cc1017f0 2024-03-14T09:52:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.104+0000 7f47d1e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc1a5ab0 con 0x7f47cc100e30 2024-03-14T09:52:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.104+0000 7f47d1e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47cc1a5c70 con 0x7f47cc106ec0 2024-03-14T09:52:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.104+0000 7f47ca000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47cc1017f0 0x7f47cc0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.104+0000 7f47ca000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47cc1017f0 0x7f47cc0735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50514/0 (socket says 172.21.15.78:50514) 2024-03-14T09:52:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.104+0000 7f47caa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc100e30 0x7f47cc0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.104+0000 7f47ca000700 1 -- 172.21.15.78:0/1789695474 learned_addr learned my addr 172.21.15.78:0/1789695474 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.105+0000 7f47cb400700 1 --2- 172.21.15.78:0/1789695474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47cc106ec0 0x7f47cc073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.105+0000 7f47caa00700 1 -- 172.21.15.78:0/1789695474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47cc106ec0 msgr2=0x7f47cc073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:39.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.105+0000 7f47caa00700 1 --2- 172.21.15.78:0/1789695474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47cc106ec0 0x7f47cc073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.105+0000 7f47caa00700 1 -- 172.21.15.78:0/1789695474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47cc1017f0 msgr2=0x7f47cc0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.105+0000 7f47caa00700 1 --2- 172.21.15.78:0/1789695474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47cc1017f0 0x7f47cc0735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.105+0000 7f47caa00700 1 -- 172.21.15.78:0/1789695474 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47bc009520 con 0x7f47cc100e30 2024-03-14T09:52:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.105+0000 7f47caa00700 1 --2- 172.21.15.78:0/1789695474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc100e30 0x7f47cc0730b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f47bc00e9f0 tx=0x7f47bc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.105+0000 7f47d0a00700 1 -- 172.21.15.78:0/1789695474 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc00cb70 con 0x7f47cc100e30 2024-03-14T09:52:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.105+0000 7f47d0a00700 1 -- 172.21.15.78:0/1789695474 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f47bc00ee90 con 0x7f47cc100e30 2024-03-14T09:52:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.105+0000 7f47d1e00700 1 -- 172.21.15.78:0/1789695474 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b80095f0 con 0x7f47cc100e30 2024-03-14T09:52:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.106+0000 7f47d0a00700 1 -- 172.21.15.78:0/1789695474 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc003cf0 con 0x7f47cc100e30 2024-03-14T09:52:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.106+0000 7f47d1e00700 1 -- 172.21.15.78:0/1789695474 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47cc1a6260 con 0x7f47cc100e30 2024-03-14T09:52:39.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.106+0000 7f47d1e00700 1 -- 172.21.15.78:0/1789695474 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47cc1092c0 con 0x7f47cc100e30 2024-03-14T09:52:39.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.109+0000 7f47d0a00700 1 -- 172.21.15.78:0/1789695474 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f47bc00ccd0 con 0x7f47cc100e30 2024-03-14T09:52:39.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.110+0000 7f47d0a00700 1 --2- 172.21.15.78:0/1789695474 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f47b00a0b20 0x7f47b00a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:39.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.110+0000 7f47d0a00700 1 -- 172.21.15.78:0/1789695474 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f47bc014070 con 0x7f47cc100e30 2024-03-14T09:52:39.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.110+0000 7f47ca000700 1 --2- 172.21.15.78:0/1789695474 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f47b00a0b20 0x7f47b00a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:39.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.111+0000 7f47ca000700 1 --2- 172.21.15.78:0/1789695474 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f47b00a0b20 0x7f47b00a2fe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f47b800b6f0 tx=0x7f47b8005bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:39.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.112+0000 7f47d0a00700 1 -- 172.21.15.78:0/1789695474 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f47bc095e30 con 0x7f47cc100e30 2024-03-14T09:52:39.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.369+0000 7f47d1e00700 1 -- 172.21.15.78:0/1789695474 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f47cc074540 con 0x7f47cc100e30 2024-03-14T09:52:39.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.374+0000 7f47d0a00700 1 -- 172.21.15.78:0/1789695474 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+259472 (secure 0 0 0) 0x7f47bc099450 con 0x7f47cc100e30 2024-03-14T09:52:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.378+0000 7f47d1e00700 1 -- 172.21.15.78:0/1789695474 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f47b00a0b20 msgr2=0x7f47b00a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:39.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.378+0000 7f47d1e00700 1 --2- 172.21.15.78:0/1789695474 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f47b00a0b20 0x7f47b00a2fe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f47b800b6f0 tx=0x7f47b8005bc0 comp rx=0 tx=0).stop 2024-03-14T09:52:39.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.378+0000 7f47d1e00700 1 -- 172.21.15.78:0/1789695474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc100e30 msgr2=0x7f47cc0730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:39.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.378+0000 7f47d1e00700 1 --2- 172.21.15.78:0/1789695474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc100e30 0x7f47cc0730b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f47bc00e9f0 tx=0x7f47bc00ed00 comp rx=0 tx=0).stop 2024-03-14T09:52:39.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.378+0000 7f47d1e00700 1 -- 172.21.15.78:0/1789695474 shutdown_connections 2024-03-14T09:52:39.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.378+0000 7f47d1e00700 1 --2- 172.21.15.78:0/1789695474 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f47b00a0b20 0x7f47b00a2fe0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:39.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.378+0000 7f47d1e00700 1 --2- 172.21.15.78:0/1789695474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc100e30 0x7f47cc0730b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:39.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.378+0000 7f47d1e00700 1 --2- 172.21.15.78:0/1789695474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47cc106ec0 0x7f47cc073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:39.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.378+0000 7f47d1e00700 1 --2- 172.21.15.78:0/1789695474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47cc1017f0 0x7f47cc0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:39.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.378+0000 7f47d1e00700 1 -- 172.21.15.78:0/1789695474 >> 172.21.15.78:0/1789695474 conn(0x7f47cc077fa0 msgr2=0x7f47cc0ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:39.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.378+0000 7f47d1e00700 1 -- 172.21.15.78:0/1789695474 shutdown_connections 2024-03-14T09:52:39.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:39.378+0000 7f47d1e00700 1 -- 172.21.15.78:0/1789695474 wait complete. 2024-03-14T09:52:39.772 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":2580616009},{"type":"v1","addr":"172.21.15.78:6801","nonce":2580616009}]},"active_addr":"172.21.15.78:6801/2580616009","active_change":"2024-03-14T09:46:40.988130+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24115,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.78:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":3308792589}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":2934089773}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":1935098096}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":3220727810}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":3057489088}]}]} 2024-03-14T09:52:39.777 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-14T09:52:39.777 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-14T09:52:39.778 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-14T09:52:40.138 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:40.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:40 smithi078 ceph-mon[38143]: pgmap v252: 129 pgs: 129 active+clean; 625 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 13 op/s 2024-03-14T09:52:40.427 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:40 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1789695474' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-14T09:52:40.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:40 smithi152 ceph-mon[48279]: pgmap v252: 129 pgs: 129 active+clean; 625 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 13 op/s 2024-03-14T09:52:40.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:40 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1789695474' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-14T09:52:40.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:40 smithi132 ceph-mon[47959]: pgmap v252: 129 pgs: 129 active+clean; 625 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 13 op/s 2024-03-14T09:52:40.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:40 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1789695474' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-14T09:52:41.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.287+0000 7f2140400700 1 -- 172.21.15.78:0/821065385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2138106090 msgr2=0x7f213810dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:41.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.287+0000 7f2140400700 1 --2- 172.21.15.78:0/821065385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2138106090 0x7f213810dc60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f213000b0a0 tx=0x7f213000b3b0 comp rx=0 tx=0).stop 2024-03-14T09:52:41.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.287+0000 7f2140400700 1 -- 172.21.15.78:0/821065385 shutdown_connections 2024-03-14T09:52:41.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.287+0000 7f2140400700 1 --2- 172.21.15.78:0/821065385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2138106090 0x7f213810dc60 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:41.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.287+0000 7f2140400700 1 --2- 172.21.15.78:0/821065385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2138100fb0 0x7f2138101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:41.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.287+0000 7f2140400700 1 --2- 172.21.15.78:0/821065385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2138101960 0x7f21381059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:41.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.287+0000 7f2140400700 1 -- 172.21.15.78:0/821065385 >> 172.21.15.78:0/821065385 conn(0x7f21380fc800 msgr2=0x7f21380fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:41.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.287+0000 7f2140400700 1 -- 172.21.15.78:0/821065385 shutdown_connections 2024-03-14T09:52:41.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.287+0000 7f2140400700 1 -- 172.21.15.78:0/821065385 wait complete. 2024-03-14T09:52:41.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.288+0000 7f2140400700 1 Processor -- start 2024-03-14T09:52:41.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.288+0000 7f2140400700 1 -- start start 2024-03-14T09:52:41.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.289+0000 7f2140400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2138100fb0 0x7f213819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:41.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.289+0000 7f2140400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2138101960 0x7f213819fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:41.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.289+0000 7f2140400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2138106090 0x7f2138199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:41.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.289+0000 7f2140400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2138199e00 con 0x7f2138101960 2024-03-14T09:52:41.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.289+0000 7f2140400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2138199f70 con 0x7f2138100fb0 2024-03-14T09:52:41.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.289+0000 7f2140400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213819a130 con 0x7f2138106090 2024-03-14T09:52:41.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.289+0000 7f213d000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2138101960 0x7f213819fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:41.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.289+0000 7f213d000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2138101960 0x7f213819fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50534/0 (socket says 172.21.15.78:50534) 2024-03-14T09:52:41.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.289+0000 7f213d000700 1 -- 172.21.15.78:0/3241192864 learned_addr learned my addr 172.21.15.78:0/3241192864 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:41.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.289+0000 7f213da00700 1 --2- 172.21.15.78:0/3241192864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2138100fb0 0x7f213819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:41.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.289+0000 7f213e400700 1 --2- 172.21.15.78:0/3241192864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2138106090 0x7f2138199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:41.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.289+0000 7f213d000700 1 -- 172.21.15.78:0/3241192864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2138106090 msgr2=0x7f2138199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:41.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.289+0000 7f213d000700 1 --2- 172.21.15.78:0/3241192864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2138106090 0x7f2138199790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:41.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.289+0000 7f213d000700 1 -- 172.21.15.78:0/3241192864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2138100fb0 msgr2=0x7f213819f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:41.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.289+0000 7f213d000700 1 --2- 172.21.15.78:0/3241192864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2138100fb0 0x7f213819f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:41.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.289+0000 7f213d000700 1 -- 172.21.15.78:0/3241192864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2130009d00 con 0x7f2138101960 2024-03-14T09:52:41.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.290+0000 7f213d000700 1 --2- 172.21.15.78:0/3241192864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2138101960 0x7f213819fb90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f212400e970 tx=0x7f212400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:41.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.290+0000 7f2136000700 1 -- 172.21.15.78:0/3241192864 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2124009790 con 0x7f2138101960 2024-03-14T09:52:41.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.290+0000 7f2136000700 1 -- 172.21.15.78:0/3241192864 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f212400c7e0 con 0x7f2138101960 2024-03-14T09:52:41.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.290+0000 7f2136000700 1 -- 172.21.15.78:0/3241192864 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21240054e0 con 0x7f2138101960 2024-03-14T09:52:41.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.290+0000 7f2140400700 1 -- 172.21.15.78:0/3241192864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f213819a410 con 0x7f2138101960 2024-03-14T09:52:41.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.290+0000 7f2140400700 1 -- 172.21.15.78:0/3241192864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21381a6640 con 0x7f2138101960 2024-03-14T09:52:41.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.291+0000 7f2140400700 1 -- 172.21.15.78:0/3241192864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2138102490 con 0x7f2138101960 2024-03-14T09:52:41.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.293+0000 7f2136000700 1 -- 172.21.15.78:0/3241192864 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f212400c950 con 0x7f2138101960 2024-03-14T09:52:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.295+0000 7f2136000700 1 --2- 172.21.15.78:0/3241192864 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f21200a0ba0 0x7f21200a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.295+0000 7f2136000700 1 -- 172.21.15.78:0/3241192864 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2124014070 con 0x7f2138101960 2024-03-14T09:52:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.295+0000 7f213da00700 1 --2- 172.21.15.78:0/3241192864 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f21200a0ba0 0x7f21200a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.296+0000 7f213da00700 1 --2- 172.21.15.78:0/3241192864 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f21200a0ba0 0x7f21200a3060 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f212c009c60 tx=0x7f212c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.299+0000 7f2136000700 1 -- 172.21.15.78:0/3241192864 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2124095bc0 con 0x7f2138101960 2024-03-14T09:52:41.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-03-14T09:52:41.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-03-14T09:52:41.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-14T09:52:41.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-03-14T09:52:41.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]: dispatch 2024-03-14T09:52:41.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-03-14T09:52:41.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2024-03-14T09:52:41.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-03-14T09:52:41.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-03-14T09:52:41.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-03-14T09:52:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-03-14T09:52:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-03-14T09:52:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-14T09:52:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-03-14T09:52:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]: dispatch 2024-03-14T09:52:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-03-14T09:52:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2024-03-14T09:52:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-03-14T09:52:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-03-14T09:52:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-03-14T09:52:41.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.503+0000 7f2140400700 1 -- 172.21.15.78:0/3241192864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2138111130 con 0x7f2138101960 2024-03-14T09:52:41.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.505+0000 7f2136000700 1 -- 172.21.15.78:0/3241192864 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+24453 (secure 0 0 0) 0x7f21240991e0 con 0x7f2138101960 2024-03-14T09:52:41.506 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:41.506 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":84,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:52:41.128742+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:52:41.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.507+0000 7f2140400700 1 -- 172.21.15.78:0/3241192864 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f21200a0ba0 msgr2=0x7f21200a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:41.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.507+0000 7f2140400700 1 --2- 172.21.15.78:0/3241192864 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f21200a0ba0 0x7f21200a3060 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f212c009c60 tx=0x7f212c009260 comp rx=0 tx=0).stop 2024-03-14T09:52:41.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.507+0000 7f2140400700 1 -- 172.21.15.78:0/3241192864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2138101960 msgr2=0x7f213819fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:41.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.507+0000 7f2140400700 1 --2- 172.21.15.78:0/3241192864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2138101960 0x7f213819fb90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f212400e970 tx=0x7f212400ec80 comp rx=0 tx=0).stop 2024-03-14T09:52:41.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.507+0000 7f2140400700 1 -- 172.21.15.78:0/3241192864 shutdown_connections 2024-03-14T09:52:41.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.507+0000 7f2140400700 1 --2- 172.21.15.78:0/3241192864 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f21200a0ba0 0x7f21200a3060 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:41.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.507+0000 7f2140400700 1 --2- 172.21.15.78:0/3241192864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2138100fb0 0x7f213819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:41.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.507+0000 7f2140400700 1 --2- 172.21.15.78:0/3241192864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2138106090 0x7f2138199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:41.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.507+0000 7f2140400700 1 --2- 172.21.15.78:0/3241192864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2138101960 0x7f213819fb90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:41.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.507+0000 7f2140400700 1 -- 172.21.15.78:0/3241192864 >> 172.21.15.78:0/3241192864 conn(0x7f21380fc800 msgr2=0x7f21380fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:41.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.508+0000 7f2140400700 1 -- 172.21.15.78:0/3241192864 shutdown_connections 2024-03-14T09:52:41.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:41.508+0000 7f2140400700 1 -- 172.21.15.78:0/3241192864 wait complete. 2024-03-14T09:52:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-03-14T09:52:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-03-14T09:52:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-14T09:52:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-03-14T09:52:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]: dispatch 2024-03-14T09:52:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2024-03-14T09:52:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2024-03-14T09:52:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-03-14T09:52:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-03-14T09:52:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-03-14T09:52:41.899 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-14T09:52:41.899 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-14T09:52:42.267 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:42.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:42 smithi078 ceph-mon[38143]: pgmap v253: 129 pgs: 129 active+clean; 625 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.2 KiB/s wr, 14 op/s 2024-03-14T09:52:42.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-03-14T09:52:42.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-03-14T09:52:42.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-14T09:52:42.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-03-14T09:52:42.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]': finished 2024-03-14T09:52:42.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-03-14T09:52:42.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-03-14T09:52:42.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-03-14T09:52:42.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-03-14T09:52:42.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-03-14T09:52:42.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:42 smithi078 ceph-mon[38143]: osdmap e84: 12 total, 12 up, 12 in 2024-03-14T09:52:42.413 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:42 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3241192864' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:42.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:42 smithi152 ceph-mon[48279]: pgmap v253: 129 pgs: 129 active+clean; 625 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.2 KiB/s wr, 14 op/s 2024-03-14T09:52:42.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-03-14T09:52:42.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-03-14T09:52:42.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-14T09:52:42.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-03-14T09:52:42.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]': finished 2024-03-14T09:52:42.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-03-14T09:52:42.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-03-14T09:52:42.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-03-14T09:52:42.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-03-14T09:52:42.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-03-14T09:52:42.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:42 smithi152 ceph-mon[48279]: osdmap e84: 12 total, 12 up, 12 in 2024-03-14T09:52:42.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:42 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3241192864' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:42.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:42 smithi132 ceph-mon[47959]: pgmap v253: 129 pgs: 129 active+clean; 625 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.2 KiB/s wr, 14 op/s 2024-03-14T09:52:42.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-03-14T09:52:42.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-03-14T09:52:42.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-14T09:52:42.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-03-14T09:52:42.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 6]}]': finished 2024-03-14T09:52:42.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2024-03-14T09:52:42.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-03-14T09:52:42.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-03-14T09:52:42.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-03-14T09:52:42.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-03-14T09:52:42.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:42 smithi132 ceph-mon[47959]: osdmap e84: 12 total, 12 up, 12 in 2024-03-14T09:52:42.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:42 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3241192864' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:43.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.344+0000 7ff6eb600700 1 -- 172.21.15.78:0/727398527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6e4100fb0 msgr2=0x7ff6e4101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:43.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.344+0000 7ff6eb600700 1 --2- 172.21.15.78:0/727398527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6e4100fb0 0x7ff6e4101390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80098f0 tx=0x7ff6d8009c00 comp rx=0 tx=0).stop 2024-03-14T09:52:43.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.344+0000 7ff6eb600700 1 -- 172.21.15.78:0/727398527 shutdown_connections 2024-03-14T09:52:43.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.344+0000 7ff6eb600700 1 --2- 172.21.15.78:0/727398527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6e4100fb0 0x7ff6e4101390 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:43.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.344+0000 7ff6eb600700 1 --2- 172.21.15.78:0/727398527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6e4106090 0x7ff6e410dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:43.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.344+0000 7ff6eb600700 1 --2- 172.21.15.78:0/727398527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6e4101960 0x7ff6e41059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:43.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.344+0000 7ff6eb600700 1 -- 172.21.15.78:0/727398527 >> 172.21.15.78:0/727398527 conn(0x7ff6e40fc800 msgr2=0x7ff6e40fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:43.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.344+0000 7ff6eb600700 1 -- 172.21.15.78:0/727398527 shutdown_connections 2024-03-14T09:52:43.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.344+0000 7ff6eb600700 1 -- 172.21.15.78:0/727398527 wait complete. 2024-03-14T09:52:43.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.345+0000 7ff6eb600700 1 Processor -- start 2024-03-14T09:52:43.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.345+0000 7ff6eb600700 1 -- start start 2024-03-14T09:52:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.345+0000 7ff6eb600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6e4100fb0 0x7ff6e40730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.346+0000 7ff6eb600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6e4101960 0x7ff6e40735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.346+0000 7ff6eb600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6e4106090 0x7ff6e4073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.346+0000 7ff6eb600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e41a5a50 con 0x7ff6e4100fb0 2024-03-14T09:52:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.346+0000 7ff6eb600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e41a5bc0 con 0x7ff6e4101960 2024-03-14T09:52:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.346+0000 7ff6eb600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e41a5d80 con 0x7ff6e4106090 2024-03-14T09:52:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.346+0000 7ff6e8c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6e4100fb0 0x7ff6e40730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.346+0000 7ff6e8c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6e4100fb0 0x7ff6e40730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50552/0 (socket says 172.21.15.78:50552) 2024-03-14T09:52:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.346+0000 7ff6e9600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6e4106090 0x7ff6e4073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.346+0000 7ff6e8c00700 1 -- 172.21.15.78:0/1149243889 learned_addr learned my addr 172.21.15.78:0/1149243889 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.346+0000 7ff6e3e00700 1 --2- 172.21.15.78:0/1149243889 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6e4101960 0x7ff6e40735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.346+0000 7ff6e8c00700 1 -- 172.21.15.78:0/1149243889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6e4106090 msgr2=0x7ff6e4073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.346+0000 7ff6e8c00700 1 --2- 172.21.15.78:0/1149243889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6e4106090 0x7ff6e4073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.346+0000 7ff6e8c00700 1 -- 172.21.15.78:0/1149243889 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6e4101960 msgr2=0x7ff6e40735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.346+0000 7ff6e8c00700 1 --2- 172.21.15.78:0/1149243889 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6e4101960 0x7ff6e40735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.346+0000 7ff6e8c00700 1 -- 172.21.15.78:0/1149243889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d8009520 con 0x7ff6e4100fb0 2024-03-14T09:52:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.346+0000 7ff6e9600700 1 --2- 172.21.15.78:0/1149243889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6e4106090 0x7ff6e4073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.347+0000 7ff6e8c00700 1 --2- 172.21.15.78:0/1149243889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6e4100fb0 0x7ff6e40730b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8005950 tx=0x7ff6d800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.347+0000 7ff6e1600700 1 -- 172.21.15.78:0/1149243889 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d801c070 con 0x7ff6e4100fb0 2024-03-14T09:52:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.347+0000 7ff6e1600700 1 -- 172.21.15.78:0/1149243889 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6d800bbd0 con 0x7ff6e4100fb0 2024-03-14T09:52:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.347+0000 7ff6eb600700 1 -- 172.21.15.78:0/1149243889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e41a6000 con 0x7ff6e4100fb0 2024-03-14T09:52:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.347+0000 7ff6e1600700 1 -- 172.21.15.78:0/1149243889 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8003970 con 0x7ff6e4100fb0 2024-03-14T09:52:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.347+0000 7ff6eb600700 1 -- 172.21.15.78:0/1149243889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e41a6410 con 0x7ff6e4100fb0 2024-03-14T09:52:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.348+0000 7ff6eb600700 1 -- 172.21.15.78:0/1149243889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e4109300 con 0x7ff6e4100fb0 2024-03-14T09:52:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.350+0000 7ff6e1600700 1 -- 172.21.15.78:0/1149243889 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7ff6d8020030 con 0x7ff6e4100fb0 2024-03-14T09:52:43.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.351+0000 7ff6e1600700 1 --2- 172.21.15.78:0/1149243889 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff6cc0a0b20 0x7ff6cc0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:43.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.351+0000 7ff6e1600700 1 -- 172.21.15.78:0/1149243889 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff6d80cd350 con 0x7ff6e4100fb0 2024-03-14T09:52:43.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.352+0000 7ff6e3e00700 1 --2- 172.21.15.78:0/1149243889 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff6cc0a0b20 0x7ff6cc0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:43.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.352+0000 7ff6e3e00700 1 --2- 172.21.15.78:0/1149243889 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff6cc0a0b20 0x7ff6cc0a2fe0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40747e0 tx=0x7ff6d0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:43.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.356+0000 7ff6e1600700 1 -- 172.21.15.78:0/1149243889 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff6d80968e0 con 0x7ff6e4100fb0 2024-03-14T09:52:43.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:43 smithi152 ceph-mon[48279]: osdmap e85: 12 total, 12 up, 12 in 2024-03-14T09:52:43.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:43 smithi132 ceph-mon[47959]: osdmap e85: 12 total, 12 up, 12 in 2024-03-14T09:52:43.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:43 smithi078 ceph-mon[38143]: osdmap e85: 12 total, 12 up, 12 in 2024-03-14T09:52:43.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.559+0000 7ff6eb600700 1 -- 172.21.15.78:0/1149243889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6e4102490 con 0x7ff6e4100fb0 2024-03-14T09:52:43.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.561+0000 7ff6e1600700 1 -- 172.21.15.78:0/1149243889 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+24453 (secure 0 0 0) 0x7ff6d8099f00 con 0x7ff6e4100fb0 2024-03-14T09:52:43.562 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:52:43.562 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":85,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:52:42.134030+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:52:43.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.563+0000 7ff6eb600700 1 -- 172.21.15.78:0/1149243889 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff6cc0a0b20 msgr2=0x7ff6cc0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:43.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.563+0000 7ff6eb600700 1 --2- 172.21.15.78:0/1149243889 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff6cc0a0b20 0x7ff6cc0a2fe0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40747e0 tx=0x7ff6d0009310 comp rx=0 tx=0).stop 2024-03-14T09:52:43.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.563+0000 7ff6eb600700 1 -- 172.21.15.78:0/1149243889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6e4100fb0 msgr2=0x7ff6e40730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:43.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.563+0000 7ff6eb600700 1 --2- 172.21.15.78:0/1149243889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6e4100fb0 0x7ff6e40730b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8005950 tx=0x7ff6d800fa00 comp rx=0 tx=0).stop 2024-03-14T09:52:43.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.563+0000 7ff6eb600700 1 -- 172.21.15.78:0/1149243889 shutdown_connections 2024-03-14T09:52:43.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.563+0000 7ff6eb600700 1 --2- 172.21.15.78:0/1149243889 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6e4106090 0x7ff6e4073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:43.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.563+0000 7ff6eb600700 1 --2- 172.21.15.78:0/1149243889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6e4100fb0 0x7ff6e40730b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:43.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.563+0000 7ff6eb600700 1 --2- 172.21.15.78:0/1149243889 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff6cc0a0b20 0x7ff6cc0a2fe0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:43.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.563+0000 7ff6eb600700 1 --2- 172.21.15.78:0/1149243889 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6e4101960 0x7ff6e40735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:43.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.563+0000 7ff6eb600700 1 -- 172.21.15.78:0/1149243889 >> 172.21.15.78:0/1149243889 conn(0x7ff6e40fc800 msgr2=0x7ff6e40fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:43.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.563+0000 7ff6eb600700 1 -- 172.21.15.78:0/1149243889 shutdown_connections 2024-03-14T09:52:43.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:43.563+0000 7ff6eb600700 1 -- 172.21.15.78:0/1149243889 wait complete. 2024-03-14T09:52:43.912 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-14T09:52:43.913 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-14T09:52:43.913 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-14T09:52:43.913 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-14T09:52:43.914 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-14T09:52:43.914 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-14T09:52:43.915 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-14T09:52:43.915 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-14T09:52:43.916 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-14T09:52:43.916 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-14T09:52:43.916 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-14T09:52:43.917 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-14T09:52:44.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:44 smithi152 ceph-mon[48279]: pgmap v256: 129 pgs: 5 peering, 124 active+clean; 625 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-14T09:52:44.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:44 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1149243889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:44.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:44 smithi132 ceph-mon[47959]: pgmap v256: 129 pgs: 5 peering, 124 active+clean; 625 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-14T09:52:44.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:44 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1149243889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:44.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:44 smithi078 ceph-mon[38143]: pgmap v256: 129 pgs: 5 peering, 124 active+clean; 625 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-14T09:52:44.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:44 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1149243889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:52:45.477 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:46.002 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:46.262 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:46 smithi078 ceph-mon[38143]: pgmap v257: 129 pgs: 5 peering, 124 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.1 KiB/s wr, 8 op/s 2024-03-14T09:52:46.289 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:46.390 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:46.443 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:46.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:46 smithi152 ceph-mon[48279]: pgmap v257: 129 pgs: 5 peering, 124 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.1 KiB/s wr, 8 op/s 2024-03-14T09:52:46.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:46 smithi132 ceph-mon[47959]: pgmap v257: 129 pgs: 5 peering, 124 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.1 KiB/s wr, 8 op/s 2024-03-14T09:52:46.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.748+0000 7f32de400700 1 -- 172.21.15.78:0/1676972312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32d810c760 msgr2=0x7f32d81142c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:46.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.748+0000 7f32de400700 1 --2- 172.21.15.78:0/1676972312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32d810c760 0x7f32d81142c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f32d000b0a0 tx=0x7f32d000b3b0 comp rx=0 tx=0).stop 2024-03-14T09:52:46.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.748+0000 7f32de400700 1 -- 172.21.15.78:0/1676972312 shutdown_connections 2024-03-14T09:52:46.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.748+0000 7f32de400700 1 --2- 172.21.15.78:0/1676972312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32d8107e50 0x7f32d8108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:46.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.748+0000 7f32de400700 1 --2- 172.21.15.78:0/1676972312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32d810c760 0x7f32d81142c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:46.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.748+0000 7f32de400700 1 --2- 172.21.15.78:0/1676972312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32d8108800 0x7f32d810c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:46.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.748+0000 7f32de400700 1 -- 172.21.15.78:0/1676972312 >> 172.21.15.78:0/1676972312 conn(0x7f32d8076b90 msgr2=0x7f32d8078fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:46.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.748+0000 7f32de400700 1 -- 172.21.15.78:0/1676972312 shutdown_connections 2024-03-14T09:52:46.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.748+0000 7f32de400700 1 -- 172.21.15.78:0/1676972312 wait complete. 2024-03-14T09:52:46.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.749+0000 7f32de400700 1 Processor -- start 2024-03-14T09:52:46.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.750+0000 7f32de400700 1 -- start start 2024-03-14T09:52:46.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.750+0000 7f32de400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32d8107e50 0x7f32d81a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:46.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.750+0000 7f32de400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32d8108800 0x7f32d81a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:46.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.750+0000 7f32de400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32d810c760 0x7f32d819db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:46.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.750+0000 7f32de400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d819e190 con 0x7f32d8108800 2024-03-14T09:52:46.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.750+0000 7f32de400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d819e300 con 0x7f32d8107e50 2024-03-14T09:52:46.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.750+0000 7f32de400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d819e4c0 con 0x7f32d810c760 2024-03-14T09:52:46.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.750+0000 7f32d7e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32d810c760 0x7f32d819db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:46.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.750+0000 7f32d7e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32d810c760 0x7f32d819db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:46846/0 (socket says 172.21.15.78:46846) 2024-03-14T09:52:46.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.751+0000 7f32d7e00700 1 -- 172.21.15.78:0/3054144796 learned_addr learned my addr 172.21.15.78:0/3054144796 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:46.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.750+0000 7f32d7400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32d8107e50 0x7f32d81a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:46.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.751+0000 7f32d6a00700 1 --2- 172.21.15.78:0/3054144796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32d8108800 0x7f32d81a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:46.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.751+0000 7f32d7e00700 1 -- 172.21.15.78:0/3054144796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32d8107e50 msgr2=0x7f32d81a3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:46.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.751+0000 7f32d7e00700 1 --2- 172.21.15.78:0/3054144796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32d8107e50 0x7f32d81a3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:46.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.751+0000 7f32d7e00700 1 -- 172.21.15.78:0/3054144796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32d8108800 msgr2=0x7f32d81a3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:46.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.751+0000 7f32d7e00700 1 --2- 172.21.15.78:0/3054144796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32d8108800 0x7f32d81a3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:46.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.751+0000 7f32d7e00700 1 -- 172.21.15.78:0/3054144796 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c4009520 con 0x7f32d810c760 2024-03-14T09:52:46.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.751+0000 7f32d6a00700 1 --2- 172.21.15.78:0/3054144796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32d8108800 0x7f32d81a3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:46.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.752+0000 7f32d7e00700 1 --2- 172.21.15.78:0/3054144796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32d810c760 0x7f32d819db20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f32d000b070 tx=0x7f32d000bda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:46.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.752+0000 7f32cfe00700 1 -- 172.21.15.78:0/3054144796 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d000e040 con 0x7f32d810c760 2024-03-14T09:52:46.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.752+0000 7f32de400700 1 -- 172.21.15.78:0/3054144796 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d0009d00 con 0x7f32d810c760 2024-03-14T09:52:46.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.752+0000 7f32cfe00700 1 -- 172.21.15.78:0/3054144796 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f32d001d480 con 0x7f32d810c760 2024-03-14T09:52:46.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.752+0000 7f32de400700 1 -- 172.21.15.78:0/3054144796 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d81aa7b0 con 0x7f32d810c760 2024-03-14T09:52:46.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.752+0000 7f32cfe00700 1 -- 172.21.15.78:0/3054144796 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d00052a0 con 0x7f32d810c760 2024-03-14T09:52:46.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.753+0000 7f32de400700 1 -- 172.21.15.78:0/3054144796 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f32b8000fc0 con 0x7f32d810c760 2024-03-14T09:52:46.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.755+0000 7f32cfe00700 1 -- 172.21.15.78:0/3054144796 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f32d0003a00 con 0x7f32d810c760 2024-03-14T09:52:46.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.756+0000 7f32cfe00700 1 --2- 172.21.15.78:0/3054144796 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f32bc0a0ba0 0x7f32bc0a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:46.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.756+0000 7f32cfe00700 1 -- 172.21.15.78:0/3054144796 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f32d00cd690 con 0x7f32d810c760 2024-03-14T09:52:46.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.757+0000 7f32d7400700 1 --2- 172.21.15.78:0/3054144796 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f32bc0a0ba0 0x7f32bc0a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:46.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.757+0000 7f32cfe00700 1 --2- 172.21.15.78:0/3054144796 >> [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776] conn(0x7f32bc0abbc0 0x7f32bc0ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:46.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.757+0000 7f32cfe00700 1 -- 172.21.15.78:0/3054144796 --> [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32bc0ae730 con 0x7f32bc0abbc0 2024-03-14T09:52:46.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.757+0000 7f32cfe00700 1 -- 172.21.15.78:0/3054144796 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32d00cda00 con 0x7f32d810c760 2024-03-14T09:52:46.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.757+0000 7f32d6a00700 1 --2- 172.21.15.78:0/3054144796 >> [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776] conn(0x7f32bc0abbc0 0x7f32bc0ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:46.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.757+0000 7f32d7400700 1 --2- 172.21.15.78:0/3054144796 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f32bc0a0ba0 0x7f32bc0a3060 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f32c8005950 tx=0x7f32c800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:46.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.757+0000 7f32d6a00700 1 --2- 172.21.15.78:0/3054144796 >> [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776] conn(0x7f32bc0abbc0 0x7f32bc0ae060 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:46.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.758+0000 7f32cfe00700 1 -- 172.21.15.78:0/3054144796 <== osd.5 v2:172.21.15.152:6808/1175069776 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f32bc0ae730 con 0x7f32bc0abbc0 2024-03-14T09:52:46.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.775+0000 7f32de400700 1 -- 172.21.15.78:0/3054144796 --> [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f32b8002d40 con 0x7f32bc0abbc0 2024-03-14T09:52:46.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.776+0000 7f32cfe00700 1 -- 172.21.15.78:0/3054144796 <== osd.5 v2:172.21.15.152:6808/1175069776 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f32b8002d40 con 0x7f32bc0abbc0 2024-03-14T09:52:46.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.777+0000 7f32de400700 1 -- 172.21.15.78:0/3054144796 >> [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776] conn(0x7f32bc0abbc0 msgr2=0x7f32bc0ae060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:46.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.777+0000 7f32de400700 1 --2- 172.21.15.78:0/3054144796 >> [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776] conn(0x7f32bc0abbc0 0x7f32bc0ae060 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:46.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.777+0000 7f32de400700 1 -- 172.21.15.78:0/3054144796 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f32bc0a0ba0 msgr2=0x7f32bc0a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:46.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.777+0000 7f32de400700 1 --2- 172.21.15.78:0/3054144796 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f32bc0a0ba0 0x7f32bc0a3060 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f32c8005950 tx=0x7f32c800a300 comp rx=0 tx=0).stop 2024-03-14T09:52:46.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.777+0000 7f32de400700 1 -- 172.21.15.78:0/3054144796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32d810c760 msgr2=0x7f32d819db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:46.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.777+0000 7f32de400700 1 --2- 172.21.15.78:0/3054144796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32d810c760 0x7f32d819db20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f32d000b070 tx=0x7f32d000bda0 comp rx=0 tx=0).stop 2024-03-14T09:52:46.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.777+0000 7f32d7e00700 1 -- 172.21.15.78:0/3054144796 reap_dead start 2024-03-14T09:52:46.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.777+0000 7f32de400700 1 -- 172.21.15.78:0/3054144796 shutdown_connections 2024-03-14T09:52:46.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.777+0000 7f32de400700 1 -- 172.21.15.78:0/3054144796 >> 172.21.15.78:0/3054144796 conn(0x7f32d8076b90 msgr2=0x7f32d8078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:46.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.777+0000 7f32de400700 1 -- 172.21.15.78:0/3054144796 shutdown_connections 2024-03-14T09:52:46.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:46.777+0000 7f32de400700 1 -- 172.21.15.78:0/3054144796 wait complete. 2024-03-14T09:52:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:47.100 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:47.278 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:47.280 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:47.552 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:47.553 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:48.245 INFO:teuthology.orchestra.run.smithi078.stdout:154618822697 2024-03-14T09:52:48.245 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-14T09:52:48.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:48 smithi078 ceph-mon[38143]: pgmap v258: 129 pgs: 3 peering, 126 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.2 KiB/s wr, 6 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-03-14T09:52:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:48.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:48 smithi152 ceph-mon[48279]: pgmap v258: 129 pgs: 3 peering, 126 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.2 KiB/s wr, 6 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-03-14T09:52:48.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:48 smithi132 ceph-mon[47959]: pgmap v258: 129 pgs: 3 peering, 126 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.2 KiB/s wr, 6 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-03-14T09:52:50.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.100+0000 7f5952a00700 1 -- 172.21.15.78:0/559898665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f594c100fb0 msgr2=0x7f594c101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:50.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.100+0000 7f5952a00700 1 --2- 172.21.15.78:0/559898665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f594c100fb0 0x7f594c101390 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f593c009990 tx=0x7f593c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:50.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.101+0000 7f5952a00700 1 -- 172.21.15.78:0/559898665 shutdown_connections 2024-03-14T09:52:50.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.101+0000 7f5952a00700 1 --2- 172.21.15.78:0/559898665 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f594c106090 0x7f594c10dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:50.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.101+0000 7f5952a00700 1 --2- 172.21.15.78:0/559898665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f594c100fb0 0x7f594c101390 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:50.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.101+0000 7f5952a00700 1 --2- 172.21.15.78:0/559898665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f594c101960 0x7f594c1059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:50.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.101+0000 7f5952a00700 1 -- 172.21.15.78:0/559898665 >> 172.21.15.78:0/559898665 conn(0x7f594c0fc800 msgr2=0x7f594c0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:50.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.101+0000 7f5952a00700 1 -- 172.21.15.78:0/559898665 shutdown_connections 2024-03-14T09:52:50.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.101+0000 7f5952a00700 1 -- 172.21.15.78:0/559898665 wait complete. 2024-03-14T09:52:50.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.102+0000 7f5952a00700 1 Processor -- start 2024-03-14T09:52:50.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.102+0000 7f5952a00700 1 -- start start 2024-03-14T09:52:50.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.102+0000 7f5952a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f594c100fb0 0x7f594c19f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:50.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.102+0000 7f5952a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f594c101960 0x7f594c19faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:50.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.102+0000 7f5952a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f594c106090 0x7f594c1996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:50.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.102+0000 7f5952a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c199d50 con 0x7f594c100fb0 2024-03-14T09:52:50.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.102+0000 7f5952a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c199ec0 con 0x7f594c101960 2024-03-14T09:52:50.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.102+0000 7f5952a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c19a080 con 0x7f594c106090 2024-03-14T09:52:50.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.103+0000 7f594be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f594c100fb0 0x7f594c19f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:50.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.103+0000 7f594be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f594c100fb0 0x7f594c19f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34216/0 (socket says 172.21.15.78:34216) 2024-03-14T09:52:50.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.103+0000 7f594be00700 1 -- 172.21.15.78:0/2131970263 learned_addr learned my addr 172.21.15.78:0/2131970263 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:50.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.103+0000 7f5950a00700 1 --2- 172.21.15.78:0/2131970263 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f594c106090 0x7f594c1996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:50.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.103+0000 7f594b400700 1 --2- 172.21.15.78:0/2131970263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f594c101960 0x7f594c19faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:50.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.103+0000 7f594be00700 1 -- 172.21.15.78:0/2131970263 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f594c106090 msgr2=0x7f594c1996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:50.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.103+0000 7f594be00700 1 --2- 172.21.15.78:0/2131970263 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f594c106090 0x7f594c1996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:50.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.103+0000 7f594be00700 1 -- 172.21.15.78:0/2131970263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f594c101960 msgr2=0x7f594c19faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:50.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.103+0000 7f594be00700 1 --2- 172.21.15.78:0/2131970263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f594c101960 0x7f594c19faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:50.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.103+0000 7f594be00700 1 -- 172.21.15.78:0/2131970263 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c0095f0 con 0x7f594c100fb0 2024-03-14T09:52:50.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.103+0000 7f594be00700 1 --2- 172.21.15.78:0/2131970263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f594c100fb0 0x7f594c19f560 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f593c005910 tx=0x7f593c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:50.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.104+0000 7f5948c00700 1 -- 172.21.15.78:0/2131970263 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c01d070 con 0x7f594c100fb0 2024-03-14T09:52:50.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.104+0000 7f5948c00700 1 -- 172.21.15.78:0/2131970263 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f593c022470 con 0x7f594c100fb0 2024-03-14T09:52:50.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.104+0000 7f5952a00700 1 -- 172.21.15.78:0/2131970263 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594c19a300 con 0x7f594c100fb0 2024-03-14T09:52:50.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.104+0000 7f5948c00700 1 -- 172.21.15.78:0/2131970263 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c017650 con 0x7f594c100fb0 2024-03-14T09:52:50.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.104+0000 7f5952a00700 1 -- 172.21.15.78:0/2131970263 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594c1a6550 con 0x7f594c100fb0 2024-03-14T09:52:50.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.105+0000 7f5952a00700 1 -- 172.21.15.78:0/2131970263 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f594c1113e0 con 0x7f594c100fb0 2024-03-14T09:52:50.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.107+0000 7f5948c00700 1 -- 172.21.15.78:0/2131970263 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f593c0225e0 con 0x7f594c100fb0 2024-03-14T09:52:50.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.108+0000 7f5948c00700 1 --2- 172.21.15.78:0/2131970263 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f59300a0b20 0x7f59300a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:50.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.108+0000 7f5948c00700 1 -- 172.21.15.78:0/2131970263 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f593c0cdea0 con 0x7f594c100fb0 2024-03-14T09:52:50.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.108+0000 7f594b400700 1 --2- 172.21.15.78:0/2131970263 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f59300a0b20 0x7f59300a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:50.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.108+0000 7f5948c00700 1 --2- 172.21.15.78:0/2131970263 >> [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960] conn(0x7f59300abb40 0x7f59300adfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:50.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.108+0000 7f5948c00700 1 -- 172.21.15.78:0/2131970263 --> [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f59300ae6b0 con 0x7f59300abb40 2024-03-14T09:52:50.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.108+0000 7f5948c00700 1 -- 172.21.15.78:0/2131970263 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f593c0ce210 con 0x7f594c100fb0 2024-03-14T09:52:50.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.109+0000 7f5950a00700 1 --2- 172.21.15.78:0/2131970263 >> [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960] conn(0x7f59300abb40 0x7f59300adfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:50.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.109+0000 7f594b400700 1 --2- 172.21.15.78:0/2131970263 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f59300a0b20 0x7f59300a2fe0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f594c19adb0 tx=0x7f5938009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:50.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.109+0000 7f5950a00700 1 --2- 172.21.15.78:0/2131970263 >> [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960] conn(0x7f59300abb40 0x7f59300adfe0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:50.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.110+0000 7f5948c00700 1 -- 172.21.15.78:0/2131970263 <== osd.1 v2:172.21.15.132:6800/162295960 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f59300ae6b0 con 0x7f59300abb40 2024-03-14T09:52:50.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.126+0000 7f5952a00700 1 -- 172.21.15.78:0/2131970263 --> [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f594c102490 con 0x7f59300abb40 2024-03-14T09:52:50.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.127+0000 7f5948c00700 1 -- 172.21.15.78:0/2131970263 <== osd.1 v2:172.21.15.132:6800/162295960 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f594c102490 con 0x7f59300abb40 2024-03-14T09:52:50.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.127+0000 7f5952a00700 1 -- 172.21.15.78:0/2131970263 >> [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960] conn(0x7f59300abb40 msgr2=0x7f59300adfe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:50.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.127+0000 7f5952a00700 1 --2- 172.21.15.78:0/2131970263 >> [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960] conn(0x7f59300abb40 0x7f59300adfe0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:50.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.128+0000 7f5952a00700 1 -- 172.21.15.78:0/2131970263 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f59300a0b20 msgr2=0x7f59300a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:50.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.128+0000 7f5952a00700 1 --2- 172.21.15.78:0/2131970263 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f59300a0b20 0x7f59300a2fe0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f594c19adb0 tx=0x7f5938009260 comp rx=0 tx=0).stop 2024-03-14T09:52:50.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.128+0000 7f5952a00700 1 -- 172.21.15.78:0/2131970263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f594c100fb0 msgr2=0x7f594c19f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:50.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.128+0000 7f5952a00700 1 --2- 172.21.15.78:0/2131970263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f594c100fb0 0x7f594c19f560 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f593c005910 tx=0x7f593c004730 comp rx=0 tx=0).stop 2024-03-14T09:52:50.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.128+0000 7f5950a00700 1 -- 172.21.15.78:0/2131970263 reap_dead start 2024-03-14T09:52:50.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.128+0000 7f5952a00700 1 -- 172.21.15.78:0/2131970263 shutdown_connections 2024-03-14T09:52:50.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.128+0000 7f5952a00700 1 -- 172.21.15.78:0/2131970263 >> 172.21.15.78:0/2131970263 conn(0x7f594c0fc800 msgr2=0x7f594c0fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:50.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.128+0000 7f5952a00700 1 -- 172.21.15.78:0/2131970263 shutdown_connections 2024-03-14T09:52:50.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:50.128+0000 7f5952a00700 1 -- 172.21.15.78:0/2131970263 wait complete. 2024-03-14T09:52:50.411 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:50 smithi078 ceph-mon[38143]: pgmap v259: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.2 KiB/s wr, 7 op/s; 86 B/s, 1 keys/s, 1 objects/s recovering 2024-03-14T09:52:50.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:50 smithi152 ceph-mon[48279]: pgmap v259: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.2 KiB/s wr, 7 op/s; 86 B/s, 1 keys/s, 1 objects/s recovering 2024-03-14T09:52:50.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:50 smithi132 ceph-mon[47959]: pgmap v259: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.2 KiB/s wr, 7 op/s; 86 B/s, 1 keys/s, 1 objects/s recovering 2024-03-14T09:52:51.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.081+0000 7fcf73800700 1 -- 172.21.15.78:0/1802885444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf6c10d5e0 msgr2=0x7fcf6c10f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:51.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.081+0000 7fcf73800700 1 --2- 172.21.15.78:0/1802885444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf6c10d5e0 0x7fcf6c10f9f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcf6400b0a0 tx=0x7fcf6400b3b0 comp rx=0 tx=0).stop 2024-03-14T09:52:51.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.081+0000 7fcf73800700 1 -- 172.21.15.78:0/1802885444 shutdown_connections 2024-03-14T09:52:51.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.081+0000 7fcf73800700 1 --2- 172.21.15.78:0/1802885444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf6c10d5e0 0x7fcf6c10f9f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:51.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.081+0000 7fcf73800700 1 --2- 172.21.15.78:0/1802885444 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf6c073180 0x7fcf6c073560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:51.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.081+0000 7fcf73800700 1 --2- 172.21.15.78:0/1802885444 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf6c073aa0 0x7fcf6c10d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:51.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.081+0000 7fcf73800700 1 -- 172.21.15.78:0/1802885444 >> 172.21.15.78:0/1802885444 conn(0x7fcf6c0fc850 msgr2=0x7fcf6c0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:51.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.081+0000 7fcf73800700 1 -- 172.21.15.78:0/1802885444 shutdown_connections 2024-03-14T09:52:51.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.081+0000 7fcf73800700 1 -- 172.21.15.78:0/1802885444 wait complete. 2024-03-14T09:52:51.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.082+0000 7fcf73800700 1 Processor -- start 2024-03-14T09:52:51.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.082+0000 7fcf73800700 1 -- start start 2024-03-14T09:52:51.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.082+0000 7fcf73800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf6c073180 0x7fcf6c19f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:51.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.082+0000 7fcf73800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf6c073aa0 0x7fcf6c19fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:51.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.082+0000 7fcf73800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf6c10d5e0 0x7fcf6c199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:51.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.082+0000 7fcf73800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c199e30 con 0x7fcf6c10d5e0 2024-03-14T09:52:51.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.082+0000 7fcf73800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c199fa0 con 0x7fcf6c073aa0 2024-03-14T09:52:51.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.082+0000 7fcf73800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c19a160 con 0x7fcf6c073180 2024-03-14T09:52:51.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.083+0000 7fcf71800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf6c10d5e0 0x7fcf6c199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:51.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.083+0000 7fcf71800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf6c10d5e0 0x7fcf6c199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34228/0 (socket says 172.21.15.78:34228) 2024-03-14T09:52:51.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.083+0000 7fcf71800700 1 -- 172.21.15.78:0/3184067547 learned_addr learned my addr 172.21.15.78:0/3184067547 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:51.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.083+0000 7fcf70e00700 1 --2- 172.21.15.78:0/3184067547 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf6c073180 0x7fcf6c19f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:51.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.083+0000 7fcf6be00700 1 --2- 172.21.15.78:0/3184067547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf6c073aa0 0x7fcf6c19fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:51.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.083+0000 7fcf71800700 1 -- 172.21.15.78:0/3184067547 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf6c073180 msgr2=0x7fcf6c19f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:51.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.083+0000 7fcf71800700 1 --2- 172.21.15.78:0/3184067547 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcf6c073180 0x7fcf6c19f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:51.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.083+0000 7fcf71800700 1 -- 172.21.15.78:0/3184067547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf6c073aa0 msgr2=0x7fcf6c19fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:51.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.083+0000 7fcf71800700 1 --2- 172.21.15.78:0/3184067547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcf6c073aa0 0x7fcf6c19fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:51.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.083+0000 7fcf71800700 1 -- 172.21.15.78:0/3184067547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf64009d00 con 0x7fcf6c10d5e0 2024-03-14T09:52:51.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.083+0000 7fcf71800700 1 --2- 172.21.15.78:0/3184067547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf6c10d5e0 0x7fcf6c199780 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcf64006070 tx=0x7fcf640078c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.084+0000 7fcf69600700 1 -- 172.21.15.78:0/3184067547 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6400e040 con 0x7fcf6c10d5e0 2024-03-14T09:52:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.084+0000 7fcf69600700 1 -- 172.21.15.78:0/3184067547 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf6401d470 con 0x7fcf6c10d5e0 2024-03-14T09:52:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.084+0000 7fcf73800700 1 -- 172.21.15.78:0/3184067547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf6c19a3e0 con 0x7fcf6c10d5e0 2024-03-14T09:52:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.084+0000 7fcf69600700 1 -- 172.21.15.78:0/3184067547 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6401b6c0 con 0x7fcf6c10d5e0 2024-03-14T09:52:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.084+0000 7fcf73800700 1 -- 172.21.15.78:0/3184067547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf6c1a6650 con 0x7fcf6c10d5e0 2024-03-14T09:52:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.085+0000 7fcf73800700 1 -- 172.21.15.78:0/3184067547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcf6c1135c0 con 0x7fcf6c10d5e0 2024-03-14T09:52:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.087+0000 7fcf69600700 1 -- 172.21.15.78:0/3184067547 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fcf6401d5e0 con 0x7fcf6c10d5e0 2024-03-14T09:52:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.088+0000 7fcf69600700 1 --2- 172.21.15.78:0/3184067547 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcf540a0b20 0x7fcf540a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.088+0000 7fcf69600700 1 -- 172.21.15.78:0/3184067547 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcf640cdbe0 con 0x7fcf6c10d5e0 2024-03-14T09:52:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.088+0000 7fcf70e00700 1 --2- 172.21.15.78:0/3184067547 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcf540a0b20 0x7fcf540a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.088+0000 7fcf69600700 1 --2- 172.21.15.78:0/3184067547 >> [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042] conn(0x7fcf540aba10 0x7fcf540adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.088+0000 7fcf69600700 1 -- 172.21.15.78:0/3184067547 --> [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf540ae580 con 0x7fcf540aba10 2024-03-14T09:52:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.088+0000 7fcf69600700 1 -- 172.21.15.78:0/3184067547 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf640cdf50 con 0x7fcf6c10d5e0 2024-03-14T09:52:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.089+0000 7fcf6be00700 1 --2- 172.21.15.78:0/3184067547 >> [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042] conn(0x7fcf540aba10 0x7fcf540adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:51.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.089+0000 7fcf70e00700 1 --2- 172.21.15.78:0/3184067547 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcf540a0b20 0x7fcf540a2fe0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009c60 tx=0x7fcf60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:51.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.089+0000 7fcf6be00700 1 --2- 172.21.15.78:0/3184067547 >> [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042] conn(0x7fcf540aba10 0x7fcf540adeb0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:51.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.090+0000 7fcf69600700 1 -- 172.21.15.78:0/3184067547 <== osd.11 v2:172.21.15.152:6824/2318800042 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fcf540ae580 con 0x7fcf540aba10 2024-03-14T09:52:51.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.107+0000 7fcf73800700 1 -- 172.21.15.78:0/3184067547 --> [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcf6c074830 con 0x7fcf540aba10 2024-03-14T09:52:51.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.107+0000 7fcf69600700 1 -- 172.21.15.78:0/3184067547 <== osd.11 v2:172.21.15.152:6824/2318800042 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcf6c074830 con 0x7fcf540aba10 2024-03-14T09:52:51.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.108+0000 7fcf73800700 1 -- 172.21.15.78:0/3184067547 >> [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042] conn(0x7fcf540aba10 msgr2=0x7fcf540adeb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:51.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.108+0000 7fcf73800700 1 --2- 172.21.15.78:0/3184067547 >> [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042] conn(0x7fcf540aba10 0x7fcf540adeb0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:51.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.108+0000 7fcf73800700 1 -- 172.21.15.78:0/3184067547 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcf540a0b20 msgr2=0x7fcf540a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.108+0000 7fcf73800700 1 --2- 172.21.15.78:0/3184067547 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcf540a0b20 0x7fcf540a2fe0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009c60 tx=0x7fcf60009260 comp rx=0 tx=0).stop 2024-03-14T09:52:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.108+0000 7fcf73800700 1 -- 172.21.15.78:0/3184067547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf6c10d5e0 msgr2=0x7fcf6c199780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.108+0000 7fcf73800700 1 --2- 172.21.15.78:0/3184067547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf6c10d5e0 0x7fcf6c199780 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcf64006070 tx=0x7fcf640078c0 comp rx=0 tx=0).stop 2024-03-14T09:52:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.108+0000 7fcf71800700 1 -- 172.21.15.78:0/3184067547 reap_dead start 2024-03-14T09:52:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.108+0000 7fcf73800700 1 -- 172.21.15.78:0/3184067547 shutdown_connections 2024-03-14T09:52:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.108+0000 7fcf73800700 1 -- 172.21.15.78:0/3184067547 >> 172.21.15.78:0/3184067547 conn(0x7fcf6c0fc850 msgr2=0x7fcf6c0fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.108+0000 7fcf73800700 1 -- 172.21.15.78:0/3184067547 shutdown_connections 2024-03-14T09:52:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:51.108+0000 7fcf73800700 1 -- 172.21.15.78:0/3184067547 wait complete. 2024-03-14T09:52:52.410 INFO:teuthology.orchestra.run.smithi078.stdout:55834574906 2024-03-14T09:52:52.411 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-14T09:52:52.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:52 smithi152 ceph-mon[48279]: pgmap v260: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.8 KiB/s wr, 6 op/s; 69 B/s, 1 keys/s, 1 objects/s recovering 2024-03-14T09:52:52.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:52 smithi078 ceph-mon[38143]: pgmap v260: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.8 KiB/s wr, 6 op/s; 69 B/s, 1 keys/s, 1 objects/s recovering 2024-03-14T09:52:52.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:52 smithi132 ceph-mon[47959]: pgmap v260: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.8 KiB/s wr, 6 op/s; 69 B/s, 1 keys/s, 1 objects/s recovering 2024-03-14T09:52:53.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.055+0000 7f3ea2c00700 1 -- 172.21.15.78:0/2487568408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c101070 msgr2=0x7f3e9c10d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.055+0000 7f3ea2c00700 1 --2- 172.21.15.78:0/2487568408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c101070 0x7f3e9c10d280 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3e88009990 tx=0x7f3e88009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.055+0000 7f3ea2c00700 1 -- 172.21.15.78:0/2487568408 shutdown_connections 2024-03-14T09:52:53.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.055+0000 7f3ea2c00700 1 --2- 172.21.15.78:0/2487568408 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e9c10d7c0 0x7f3e9c10fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.055+0000 7f3ea2c00700 1 --2- 172.21.15.78:0/2487568408 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e9c100750 0x7f3e9c100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.055+0000 7f3ea2c00700 1 --2- 172.21.15.78:0/2487568408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c101070 0x7f3e9c10d280 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.055+0000 7f3ea2c00700 1 -- 172.21.15.78:0/2487568408 >> 172.21.15.78:0/2487568408 conn(0x7f3e9c076db0 msgr2=0x7f3e9c0771c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:53.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.056+0000 7f3ea2c00700 1 -- 172.21.15.78:0/2487568408 shutdown_connections 2024-03-14T09:52:53.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.056+0000 7f3ea2c00700 1 -- 172.21.15.78:0/2487568408 wait complete. 2024-03-14T09:52:53.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.057+0000 7f3ea2c00700 1 Processor -- start 2024-03-14T09:52:53.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.057+0000 7f3ea2c00700 1 -- start start 2024-03-14T09:52:53.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.058+0000 7f3ea2c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e9c100750 0x7f3e9c10bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:53.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.058+0000 7f3ea2c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c101070 0x7f3e9c10c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:53.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.058+0000 7f3ea2c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e9c10d7c0 0x7f3e9c1a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:53.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.058+0000 7f3ea2c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c1a9ea0 con 0x7f3e9c101070 2024-03-14T09:52:53.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.058+0000 7f3ea2c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c1a9fe0 con 0x7f3e9c10d7c0 2024-03-14T09:52:53.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.058+0000 7f3ea2c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c1aa1a0 con 0x7f3e9c100750 2024-03-14T09:52:53.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.058+0000 7f3e9be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e9c100750 0x7f3e9c10bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:53.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.058+0000 7f3e9b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c101070 0x7f3e9c10c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:53.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.058+0000 7f3e9be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e9c100750 0x7f3e9c10bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:46960/0 (socket says 172.21.15.78:46960) 2024-03-14T09:52:53.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.058+0000 7f3e9be00700 1 -- 172.21.15.78:0/2452877094 learned_addr learned my addr 172.21.15.78:0/2452877094 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:53.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.059+0000 7f3ea0c00700 1 --2- 172.21.15.78:0/2452877094 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e9c10d7c0 0x7f3e9c1a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:53.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.059+0000 7f3e9be00700 1 -- 172.21.15.78:0/2452877094 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e9c10d7c0 msgr2=0x7f3e9c1a98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.059+0000 7f3e9be00700 1 --2- 172.21.15.78:0/2452877094 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e9c10d7c0 0x7f3e9c1a98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.059+0000 7f3e9be00700 1 -- 172.21.15.78:0/2452877094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c101070 msgr2=0x7f3e9c10c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.059+0000 7f3e9be00700 1 --2- 172.21.15.78:0/2452877094 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c101070 0x7f3e9c10c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.059+0000 7f3e9be00700 1 -- 172.21.15.78:0/2452877094 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e880095f0 con 0x7f3e9c100750 2024-03-14T09:52:53.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.059+0000 7f3ea0c00700 1 --2- 172.21.15.78:0/2452877094 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e9c10d7c0 0x7f3e9c1a98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:52:53.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.059+0000 7f3e9be00700 1 --2- 172.21.15.78:0/2452877094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e9c100750 0x7f3e9c10bc10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00e9a0 tx=0x7f3e8c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:53.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.059+0000 7f3e98c00700 1 -- 172.21.15.78:0/2452877094 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c009790 con 0x7f3e9c100750 2024-03-14T09:52:53.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.060+0000 7f3ea2c00700 1 -- 172.21.15.78:0/2452877094 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9c1aa480 con 0x7f3e9c100750 2024-03-14T09:52:53.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.060+0000 7f3e98c00700 1 -- 172.21.15.78:0/2452877094 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e8c00cbd0 con 0x7f3e9c100750 2024-03-14T09:52:53.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.060+0000 7f3e98c00700 1 -- 172.21.15.78:0/2452877094 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c018680 con 0x7f3e9c100750 2024-03-14T09:52:53.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.060+0000 7f3ea2c00700 1 -- 172.21.15.78:0/2452877094 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9c1aa9d0 con 0x7f3e9c100750 2024-03-14T09:52:53.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.061+0000 7f3ea2c00700 1 -- 172.21.15.78:0/2452877094 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3e7c000fc0 con 0x7f3e9c100750 2024-03-14T09:52:53.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.063+0000 7f3e98c00700 1 -- 172.21.15.78:0/2452877094 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f3e8c010430 con 0x7f3e9c100750 2024-03-14T09:52:53.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.064+0000 7f3e98c00700 1 --2- 172.21.15.78:0/2452877094 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3e800a0ba0 0x7f3e800a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:53.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.064+0000 7f3e98c00700 1 -- 172.21.15.78:0/2452877094 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3e8c014070 con 0x7f3e9c100750 2024-03-14T09:52:53.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.065+0000 7f3e98c00700 1 --2- 172.21.15.78:0/2452877094 >> [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467] conn(0x7f3e800abbc0 0x7f3e800ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:53.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.065+0000 7f3e98c00700 1 -- 172.21.15.78:0/2452877094 --> [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e800ae730 con 0x7f3e800abbc0 2024-03-14T09:52:53.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.065+0000 7f3e98c00700 1 -- 172.21.15.78:0/2452877094 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e8c0cd380 con 0x7f3e9c100750 2024-03-14T09:52:53.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.065+0000 7f3e9b400700 1 --2- 172.21.15.78:0/2452877094 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3e800a0ba0 0x7f3e800a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:53.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.065+0000 7f3ea0c00700 1 --2- 172.21.15.78:0/2452877094 >> [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467] conn(0x7f3e800abbc0 0x7f3e800ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:53.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.066+0000 7f3e9b400700 1 --2- 172.21.15.78:0/2452877094 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3e800a0ba0 0x7f3e800a3060 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c10d170 tx=0x7f3e8800b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:53.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.066+0000 7f3ea0c00700 1 --2- 172.21.15.78:0/2452877094 >> [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467] conn(0x7f3e800abbc0 0x7f3e800ae060 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:53.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.067+0000 7f3e98c00700 1 -- 172.21.15.78:0/2452877094 <== osd.6 v2:172.21.15.78:6818/996423467 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f3e800ae730 con 0x7f3e800abbc0 2024-03-14T09:52:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.085+0000 7f3ea2c00700 1 -- 172.21.15.78:0/2452877094 --> [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3e7c002d40 con 0x7f3e800abbc0 2024-03-14T09:52:53.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.086+0000 7f3e98c00700 1 -- 172.21.15.78:0/2452877094 <== osd.6 v2:172.21.15.78:6818/996423467 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f3e7c002d40 con 0x7f3e800abbc0 2024-03-14T09:52:53.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.087+0000 7f3ea2c00700 1 -- 172.21.15.78:0/2452877094 >> [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467] conn(0x7f3e800abbc0 msgr2=0x7f3e800ae060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.087+0000 7f3ea2c00700 1 --2- 172.21.15.78:0/2452877094 >> [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467] conn(0x7f3e800abbc0 0x7f3e800ae060 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.087+0000 7f3ea2c00700 1 -- 172.21.15.78:0/2452877094 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3e800a0ba0 msgr2=0x7f3e800a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.087+0000 7f3ea2c00700 1 --2- 172.21.15.78:0/2452877094 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3e800a0ba0 0x7f3e800a3060 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c10d170 tx=0x7f3e8800b280 comp rx=0 tx=0).stop 2024-03-14T09:52:53.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.087+0000 7f3ea2c00700 1 -- 172.21.15.78:0/2452877094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e9c100750 msgr2=0x7f3e9c10bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.087+0000 7f3ea2c00700 1 --2- 172.21.15.78:0/2452877094 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3e9c100750 0x7f3e9c10bc10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00e9a0 tx=0x7f3e8c00ecb0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.087+0000 7f3ea0c00700 1 -- 172.21.15.78:0/2452877094 reap_dead start 2024-03-14T09:52:53.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.087+0000 7f3ea2c00700 1 -- 172.21.15.78:0/2452877094 shutdown_connections 2024-03-14T09:52:53.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.087+0000 7f3ea2c00700 1 -- 172.21.15.78:0/2452877094 >> 172.21.15.78:0/2452877094 conn(0x7f3e9c076db0 msgr2=0x7f3e9c10ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:53.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.087+0000 7f3ea2c00700 1 -- 172.21.15.78:0/2452877094 shutdown_connections 2024-03-14T09:52:53.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.087+0000 7f3ea2c00700 1 -- 172.21.15.78:0/2452877094 wait complete. 2024-03-14T09:52:53.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.129+0000 7fbeabc00700 1 -- 172.21.15.78:0/1424255200 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbea4100fb0 msgr2=0x7fbea4101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.129+0000 7fbeabc00700 1 --2- 172.21.15.78:0/1424255200 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbea4100fb0 0x7fbea4101390 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbe980098f0 tx=0x7fbe98009c00 comp rx=0 tx=0).stop 2024-03-14T09:52:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.129+0000 7fbeabc00700 1 -- 172.21.15.78:0/1424255200 shutdown_connections 2024-03-14T09:52:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.129+0000 7fbeabc00700 1 --2- 172.21.15.78:0/1424255200 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbea4106090 0x7fbea410dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.129+0000 7fbeabc00700 1 --2- 172.21.15.78:0/1424255200 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbea4100fb0 0x7fbea4101390 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.129+0000 7fbeabc00700 1 --2- 172.21.15.78:0/1424255200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbea4101960 0x7fbea41059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.129+0000 7fbeabc00700 1 -- 172.21.15.78:0/1424255200 >> 172.21.15.78:0/1424255200 conn(0x7fbea40fc800 msgr2=0x7fbea40fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.129+0000 7fbeabc00700 1 -- 172.21.15.78:0/1424255200 shutdown_connections 2024-03-14T09:52:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.129+0000 7fbeabc00700 1 -- 172.21.15.78:0/1424255200 wait complete. 2024-03-14T09:52:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.130+0000 7fbeabc00700 1 Processor -- start 2024-03-14T09:52:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.130+0000 7fbeabc00700 1 -- start start 2024-03-14T09:52:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.130+0000 7fbeabc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbea4100fb0 0x7fbea41976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.130+0000 7fbeabc00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbea4101960 0x7fbea4197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:53.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.130+0000 7fbeabc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbea4106090 0x7fbea419bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.130+0000 7fbeabc00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea419c560 con 0x7fbea4106090 2024-03-14T09:52:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.130+0000 7fbeabc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea419c6d0 con 0x7fbea4100fb0 2024-03-14T09:52:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.130+0000 7fbeabc00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea419c890 con 0x7fbea4101960 2024-03-14T09:52:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.131+0000 7fbea9200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbea4100fb0 0x7fbea41976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.131+0000 7fbea9c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbea4106090 0x7fbea419bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.131+0000 7fbea9200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbea4100fb0 0x7fbea41976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:47676/0 (socket says 172.21.15.78:47676) 2024-03-14T09:52:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.131+0000 7fbea9200700 1 -- 172.21.15.78:0/3898059134 learned_addr learned my addr 172.21.15.78:0/3898059134 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.131+0000 7fbea3e00700 1 --2- 172.21.15.78:0/3898059134 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbea4101960 0x7fbea4197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.131+0000 7fbea9200700 1 -- 172.21.15.78:0/3898059134 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbea4101960 msgr2=0x7fbea4197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.131+0000 7fbea9200700 1 --2- 172.21.15.78:0/3898059134 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbea4101960 0x7fbea4197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.131+0000 7fbea9200700 1 -- 172.21.15.78:0/3898059134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbea4106090 msgr2=0x7fbea419bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.131+0000 7fbea9200700 1 --2- 172.21.15.78:0/3898059134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbea4106090 0x7fbea419bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.131+0000 7fbea9200700 1 -- 172.21.15.78:0/3898059134 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe900095f0 con 0x7fbea4100fb0 2024-03-14T09:52:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.131+0000 7fbea9c00700 1 --2- 172.21.15.78:0/3898059134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbea4106090 0x7fbea419bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.131+0000 7fbea9200700 1 --2- 172.21.15.78:0/3898059134 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbea4100fb0 0x7fbea41976b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbe98005950 tx=0x7fbe9800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.132+0000 7fbea1600700 1 -- 172.21.15.78:0/3898059134 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9801c070 con 0x7fbea4100fb0 2024-03-14T09:52:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.132+0000 7fbea1600700 1 -- 172.21.15.78:0/3898059134 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe9800baa0 con 0x7fbea4100fb0 2024-03-14T09:52:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.132+0000 7fbea1600700 1 -- 172.21.15.78:0/3898059134 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe980038e0 con 0x7fbea4100fb0 2024-03-14T09:52:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.132+0000 7fbeabc00700 1 -- 172.21.15.78:0/3898059134 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe98009520 con 0x7fbea4100fb0 2024-03-14T09:52:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.132+0000 7fbeabc00700 1 -- 172.21.15.78:0/3898059134 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea419cd40 con 0x7fbea4100fb0 2024-03-14T09:52:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.133+0000 7fbeabc00700 1 -- 172.21.15.78:0/3898059134 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbea41113e0 con 0x7fbea4100fb0 2024-03-14T09:52:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.136+0000 7fbea1600700 1 -- 172.21.15.78:0/3898059134 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fbe98020030 con 0x7fbea4100fb0 2024-03-14T09:52:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.137+0000 7fbea1600700 1 --2- 172.21.15.78:0/3898059134 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fbe8c0a0ba0 0x7fbe8c0a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.137+0000 7fbea1600700 1 -- 172.21.15.78:0/3898059134 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbe980cd7f0 con 0x7fbea4100fb0 2024-03-14T09:52:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.137+0000 7fbea3e00700 1 --2- 172.21.15.78:0/3898059134 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fbe8c0a0ba0 0x7fbe8c0a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.137+0000 7fbea1600700 1 --2- 172.21.15.78:0/3898059134 >> [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712] conn(0x7fbe8c0abbc0 0x7fbe8c0ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.137+0000 7fbea1600700 1 -- 172.21.15.78:0/3898059134 --> [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbe8c0ae730 con 0x7fbe8c0abbc0 2024-03-14T09:52:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.137+0000 7fbea1600700 1 -- 172.21.15.78:0/3898059134 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe980cdb60 con 0x7fbea4100fb0 2024-03-14T09:52:53.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.138+0000 7fbea9c00700 1 --2- 172.21.15.78:0/3898059134 >> [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712] conn(0x7fbe8c0abbc0 0x7fbe8c0ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:53.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.138+0000 7fbea3e00700 1 --2- 172.21.15.78:0/3898059134 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fbe8c0a0ba0 0x7fbe8c0a3060 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbea4198bc0 tx=0x7fbe90009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:53.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.138+0000 7fbea9c00700 1 --2- 172.21.15.78:0/3898059134 >> [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712] conn(0x7fbe8c0abbc0 0x7fbe8c0ae060 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:53.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.139+0000 7fbea1600700 1 -- 172.21.15.78:0/3898059134 <== osd.7 v2:172.21.15.132:6816/1879810712 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fbe8c0ae730 con 0x7fbe8c0abbc0 2024-03-14T09:52:53.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.167+0000 7fbeabc00700 1 -- 172.21.15.78:0/3898059134 --> [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbea4102490 con 0x7fbe8c0abbc0 2024-03-14T09:52:53.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.168+0000 7fbea1600700 1 -- 172.21.15.78:0/3898059134 <== osd.7 v2:172.21.15.132:6816/1879810712 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fbea4102490 con 0x7fbe8c0abbc0 2024-03-14T09:52:53.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.168+0000 7fbeabc00700 1 -- 172.21.15.78:0/3898059134 >> [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712] conn(0x7fbe8c0abbc0 msgr2=0x7fbe8c0ae060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.168+0000 7fbeabc00700 1 --2- 172.21.15.78:0/3898059134 >> [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712] conn(0x7fbe8c0abbc0 0x7fbe8c0ae060 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.168+0000 7fbeabc00700 1 -- 172.21.15.78:0/3898059134 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fbe8c0a0ba0 msgr2=0x7fbe8c0a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.168+0000 7fbeabc00700 1 --2- 172.21.15.78:0/3898059134 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fbe8c0a0ba0 0x7fbe8c0a3060 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbea4198bc0 tx=0x7fbe90009310 comp rx=0 tx=0).stop 2024-03-14T09:52:53.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.168+0000 7fbeabc00700 1 -- 172.21.15.78:0/3898059134 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbea4100fb0 msgr2=0x7fbea41976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.168+0000 7fbeabc00700 1 --2- 172.21.15.78:0/3898059134 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbea4100fb0 0x7fbea41976b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbe98005950 tx=0x7fbe9800fa00 comp rx=0 tx=0).stop 2024-03-14T09:52:53.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.169+0000 7fbea9c00700 1 -- 172.21.15.78:0/3898059134 reap_dead start 2024-03-14T09:52:53.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.169+0000 7fbeabc00700 1 -- 172.21.15.78:0/3898059134 shutdown_connections 2024-03-14T09:52:53.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.169+0000 7fbeabc00700 1 -- 172.21.15.78:0/3898059134 >> 172.21.15.78:0/3898059134 conn(0x7fbea40fc800 msgr2=0x7fbea40fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:53.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.169+0000 7fbeabc00700 1 -- 172.21.15.78:0/3898059134 shutdown_connections 2024-03-14T09:52:53.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.169+0000 7fbeabc00700 1 -- 172.21.15.78:0/3898059134 wait complete. 2024-03-14T09:52:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.880+0000 7f5459a00700 1 -- 172.21.15.78:0/1983719308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5454107f80 msgr2=0x7f545410bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.880+0000 7f5459a00700 1 --2- 172.21.15.78:0/1983719308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5454107f80 0x7f545410bfd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f54400098f0 tx=0x7f5440009c00 comp rx=0 tx=0).stop 2024-03-14T09:52:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.880+0000 7f5459a00700 1 -- 172.21.15.78:0/1983719308 shutdown_connections 2024-03-14T09:52:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.880+0000 7f5459a00700 1 --2- 172.21.15.78:0/1983719308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f54541075d0 0x7f54541079b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.880+0000 7f5459a00700 1 --2- 172.21.15.78:0/1983719308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f545410c6b0 0x7f5454114280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.880+0000 7f5459a00700 1 --2- 172.21.15.78:0/1983719308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5454107f80 0x7f545410bfd0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.880+0000 7f5459a00700 1 -- 172.21.15.78:0/1983719308 >> 172.21.15.78:0/1983719308 conn(0x7f5454074580 msgr2=0x7f54540769a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.880+0000 7f5459a00700 1 -- 172.21.15.78:0/1983719308 shutdown_connections 2024-03-14T09:52:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.880+0000 7f5459a00700 1 -- 172.21.15.78:0/1983719308 wait complete. 2024-03-14T09:52:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.881+0000 7f5459a00700 1 Processor -- start 2024-03-14T09:52:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.882+0000 7f5459a00700 1 -- start start 2024-03-14T09:52:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.882+0000 7f5459a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54541075d0 0x7f54541a39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.882+0000 7f5459a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5454107f80 0x7f54541a3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.882+0000 7f5459a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f545410c6b0 0x7f545419db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.882+0000 7f5459a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545419e170 con 0x7f54541075d0 2024-03-14T09:52:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.882+0000 7f5459a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545419e2e0 con 0x7f545410c6b0 2024-03-14T09:52:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.882+0000 7f5459a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545419e4a0 con 0x7f5454107f80 2024-03-14T09:52:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.883+0000 7f5452000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5454107f80 0x7f54541a3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.883+0000 7f5452a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54541075d0 0x7f54541a39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.883+0000 7f5452000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5454107f80 0x7f54541a3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:46914/0 (socket says 172.21.15.78:46914) 2024-03-14T09:52:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.883+0000 7f5452000700 1 -- 172.21.15.78:0/3385746564 learned_addr learned my addr 172.21.15.78:0/3385746564 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.884+0000 7f5452000700 1 -- 172.21.15.78:0/3385746564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f545410c6b0 msgr2=0x7f545419db10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T09:52:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.884+0000 7f5453400700 1 --2- 172.21.15.78:0/3385746564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f545410c6b0 0x7f545419db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.884+0000 7f5452000700 1 --2- 172.21.15.78:0/3385746564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f545410c6b0 0x7f545419db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.884+0000 7f5452000700 1 -- 172.21.15.78:0/3385746564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54541075d0 msgr2=0x7f54541a39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.884+0000 7f5452000700 1 --2- 172.21.15.78:0/3385746564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54541075d0 0x7f54541a39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.884+0000 7f5452000700 1 -- 172.21.15.78:0/3385746564 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54440095f0 con 0x7f5454107f80 2024-03-14T09:52:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.885+0000 7f5452a00700 1 --2- 172.21.15.78:0/3385746564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54541075d0 0x7f54541a39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.885+0000 7f5453400700 1 --2- 172.21.15.78:0/3385746564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f545410c6b0 0x7f545419db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:52:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.885+0000 7f5452000700 1 --2- 172.21.15.78:0/3385746564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5454107f80 0x7f54541a3f10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5440009230 tx=0x7f544000fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.886+0000 7f544b400700 1 -- 172.21.15.78:0/3385746564 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544001c070 con 0x7f5454107f80 2024-03-14T09:52:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.886+0000 7f544b400700 1 -- 172.21.15.78:0/3385746564 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f544000bb20 con 0x7f5454107f80 2024-03-14T09:52:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.886+0000 7f544b400700 1 -- 172.21.15.78:0/3385746564 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54400038e0 con 0x7f5454107f80 2024-03-14T09:52:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.886+0000 7f5459a00700 1 -- 172.21.15.78:0/3385746564 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5440009520 con 0x7f5454107f80 2024-03-14T09:52:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.886+0000 7f5459a00700 1 -- 172.21.15.78:0/3385746564 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f545419e8f0 con 0x7f5454107f80 2024-03-14T09:52:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.887+0000 7f5459a00700 1 -- 172.21.15.78:0/3385746564 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f54541179d0 con 0x7f5454107f80 2024-03-14T09:52:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.889+0000 7f544b400700 1 -- 172.21.15.78:0/3385746564 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f5440020070 con 0x7f5454107f80 2024-03-14T09:52:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.891+0000 7f544b400700 1 --2- 172.21.15.78:0/3385746564 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f54380a0c20 0x7f54380a30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.892+0000 7f544b400700 1 -- 172.21.15.78:0/3385746564 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f54400cce50 con 0x7f5454107f80 2024-03-14T09:52:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.892+0000 7f5452a00700 1 --2- 172.21.15.78:0/3385746564 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f54380a0c20 0x7f54380a30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.892+0000 7f544b400700 1 --2- 172.21.15.78:0/3385746564 >> [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797] conn(0x7f54380abc40 0x7f54380ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.892+0000 7f544b400700 1 -- 172.21.15.78:0/3385746564 --> [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54380ae7b0 con 0x7f54380abc40 2024-03-14T09:52:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.892+0000 7f544b400700 1 -- 172.21.15.78:0/3385746564 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54400cf470 con 0x7f5454107f80 2024-03-14T09:52:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.893+0000 7f5453400700 1 --2- 172.21.15.78:0/3385746564 >> [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797] conn(0x7f54380abc40 0x7f54380ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.893+0000 7f5452a00700 1 --2- 172.21.15.78:0/3385746564 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f54380a0c20 0x7f54380a30e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5444005fd0 tx=0x7f5444009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.893+0000 7f5453400700 1 --2- 172.21.15.78:0/3385746564 >> [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797] conn(0x7f54380abc40 0x7f54380ae0e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.894+0000 7f544b400700 1 -- 172.21.15.78:0/3385746564 <== osd.4 v2:172.21.15.132:6808/830577797 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f54380ae7b0 con 0x7f54380abc40 2024-03-14T09:52:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.912+0000 7f5459a00700 1 -- 172.21.15.78:0/3385746564 --> [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f54540614b0 con 0x7f54380abc40 2024-03-14T09:52:53.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.912+0000 7f544b400700 1 -- 172.21.15.78:0/3385746564 <== osd.4 v2:172.21.15.132:6808/830577797 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f54540614b0 con 0x7f54380abc40 2024-03-14T09:52:53.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.913+0000 7f5459a00700 1 -- 172.21.15.78:0/3385746564 >> [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797] conn(0x7f54380abc40 msgr2=0x7f54380ae0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.913+0000 7f5459a00700 1 --2- 172.21.15.78:0/3385746564 >> [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797] conn(0x7f54380abc40 0x7f54380ae0e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:53.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.913+0000 7f5459a00700 1 -- 172.21.15.78:0/3385746564 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f54380a0c20 msgr2=0x7f54380a30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.913+0000 7f5459a00700 1 --2- 172.21.15.78:0/3385746564 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f54380a0c20 0x7f54380a30e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5444005fd0 tx=0x7f5444009310 comp rx=0 tx=0).stop 2024-03-14T09:52:53.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.913+0000 7f5459a00700 1 -- 172.21.15.78:0/3385746564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5454107f80 msgr2=0x7f54541a3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:53.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.913+0000 7f5459a00700 1 --2- 172.21.15.78:0/3385746564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5454107f80 0x7f54541a3f10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5440009230 tx=0x7f544000fa00 comp rx=0 tx=0).stop 2024-03-14T09:52:53.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.913+0000 7f5453400700 1 -- 172.21.15.78:0/3385746564 reap_dead start 2024-03-14T09:52:53.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.913+0000 7f5459a00700 1 -- 172.21.15.78:0/3385746564 shutdown_connections 2024-03-14T09:52:53.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.913+0000 7f5459a00700 1 -- 172.21.15.78:0/3385746564 >> 172.21.15.78:0/3385746564 conn(0x7f5454074580 msgr2=0x7f54540759c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:53.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.913+0000 7f5459a00700 1 -- 172.21.15.78:0/3385746564 shutdown_connections 2024-03-14T09:52:53.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.913+0000 7f5459a00700 1 -- 172.21.15.78:0/3385746564 wait complete. 2024-03-14T09:52:53.812 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:53 smithi078 ceph-mon[38143]: pgmap v261: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.6 KiB/s wr, 6 op/s; 63 B/s, 0 keys/s, 1 objects/s recovering 2024-03-14T09:52:53.895 INFO:teuthology.orchestra.run.smithi078.stdout:184683593765 2024-03-14T09:52:53.895 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-14T09:52:53.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:53 smithi152 ceph-mon[48279]: pgmap v261: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.6 KiB/s wr, 6 op/s; 63 B/s, 0 keys/s, 1 objects/s recovering 2024-03-14T09:52:54.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:53 smithi132 ceph-mon[47959]: pgmap v261: 129 pgs: 129 active+clean; 640 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.6 KiB/s wr, 6 op/s; 63 B/s, 0 keys/s, 1 objects/s recovering 2024-03-14T09:52:54.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.835+0000 7ff304600700 1 -- 172.21.15.78:0/2188909321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2fc101960 msgr2=0x7ff2fc1059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.835+0000 7ff304600700 1 --2- 172.21.15.78:0/2188909321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2fc101960 0x7ff2fc1059b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8009990 tx=0x7ff2e8009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.835+0000 7ff304600700 1 -- 172.21.15.78:0/2188909321 shutdown_connections 2024-03-14T09:52:54.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.835+0000 7ff304600700 1 --2- 172.21.15.78:0/2188909321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2fc100fb0 0x7ff2fc101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.835+0000 7ff304600700 1 --2- 172.21.15.78:0/2188909321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff2fc106090 0x7ff2fc10dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.835+0000 7ff304600700 1 --2- 172.21.15.78:0/2188909321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2fc101960 0x7ff2fc1059b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.835+0000 7ff304600700 1 -- 172.21.15.78:0/2188909321 >> 172.21.15.78:0/2188909321 conn(0x7ff2fc0fc800 msgr2=0x7ff2fc0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:54.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.835+0000 7ff304600700 1 -- 172.21.15.78:0/2188909321 shutdown_connections 2024-03-14T09:52:54.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.835+0000 7ff304600700 1 -- 172.21.15.78:0/2188909321 wait complete. 2024-03-14T09:52:54.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.836+0000 7ff304600700 1 Processor -- start 2024-03-14T09:52:54.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.836+0000 7ff304600700 1 -- start start 2024-03-14T09:52:54.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff304600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2fc100fb0 0x7ff2fc0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff304600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2fc101960 0x7ff2fc0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff304600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff2fc106090 0x7ff2fc073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff304600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc1a5ab0 con 0x7ff2fc101960 2024-03-14T09:52:54.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff304600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc1a5c20 con 0x7ff2fc100fb0 2024-03-14T09:52:54.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff304600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc1a5de0 con 0x7ff2fc106090 2024-03-14T09:52:54.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff301200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2fc101960 0x7ff2fc0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff301c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2fc100fb0 0x7ff2fc0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff301200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2fc101960 0x7ff2fc0735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36216/0 (socket says 172.21.15.78:36216) 2024-03-14T09:52:54.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff301200700 1 -- 172.21.15.78:0/3088635259 learned_addr learned my addr 172.21.15.78:0/3088635259 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:54.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff302600700 1 --2- 172.21.15.78:0/3088635259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff2fc106090 0x7ff2fc073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff301200700 1 -- 172.21.15.78:0/3088635259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff2fc106090 msgr2=0x7ff2fc073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff301200700 1 --2- 172.21.15.78:0/3088635259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff2fc106090 0x7ff2fc073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff301200700 1 -- 172.21.15.78:0/3088635259 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2fc100fb0 msgr2=0x7ff2fc0730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff301200700 1 --2- 172.21.15.78:0/3088635259 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2fc100fb0 0x7ff2fc0730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff301200700 1 -- 172.21.15.78:0/3088635259 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e80095f0 con 0x7ff2fc101960 2024-03-14T09:52:54.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff302600700 1 --2- 172.21.15.78:0/3088635259 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff2fc106090 0x7ff2fc073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:52:54.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.837+0000 7ff301c00700 1 --2- 172.21.15.78:0/3088635259 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2fc100fb0 0x7ff2fc0730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:54.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.838+0000 7ff301200700 1 --2- 172.21.15.78:0/3088635259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2fc101960 0x7ff2fc0735f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8000c00 tx=0x7ff2e8005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:54.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.838+0000 7ff2fa000700 1 -- 172.21.15.78:0/3088635259 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e801d070 con 0x7ff2fc101960 2024-03-14T09:52:54.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.838+0000 7ff2fa000700 1 -- 172.21.15.78:0/3088635259 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2e8022470 con 0x7ff2fc101960 2024-03-14T09:52:54.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.838+0000 7ff2fa000700 1 -- 172.21.15.78:0/3088635259 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8017650 con 0x7ff2fc101960 2024-03-14T09:52:54.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.838+0000 7ff304600700 1 -- 172.21.15.78:0/3088635259 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2fc1a6060 con 0x7ff2fc101960 2024-03-14T09:52:54.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.838+0000 7ff304600700 1 -- 172.21.15.78:0/3088635259 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2fc1a6420 con 0x7ff2fc101960 2024-03-14T09:52:54.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.839+0000 7ff304600700 1 -- 172.21.15.78:0/3088635259 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff2fc1113b0 con 0x7ff2fc101960 2024-03-14T09:52:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.841+0000 7ff2fa000700 1 -- 172.21.15.78:0/3088635259 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7ff2e800fad0 con 0x7ff2fc101960 2024-03-14T09:52:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.842+0000 7ff2fa000700 1 --2- 172.21.15.78:0/3088635259 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff2e40a0b20 0x7ff2e40a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.843+0000 7ff2fa000700 1 -- 172.21.15.78:0/3088635259 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff2e80ce200 con 0x7ff2fc101960 2024-03-14T09:52:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.843+0000 7ff301c00700 1 --2- 172.21.15.78:0/3088635259 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff2e40a0b20 0x7ff2e40a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.843+0000 7ff2fa000700 1 --2- 172.21.15.78:0/3088635259 >> [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001] conn(0x7ff2e40abb40 0x7ff2e40adfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.843+0000 7ff2fa000700 1 -- 172.21.15.78:0/3088635259 --> [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2e40ae6b0 con 0x7ff2e40abb40 2024-03-14T09:52:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.843+0000 7ff2fa000700 1 -- 172.21.15.78:0/3088635259 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2e80ce600 con 0x7ff2fc101960 2024-03-14T09:52:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.843+0000 7ff302600700 1 --2- 172.21.15.78:0/3088635259 >> [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001] conn(0x7ff2e40abb40 0x7ff2e40adfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.843+0000 7ff301c00700 1 --2- 172.21.15.78:0/3088635259 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff2e40a0b20 0x7ff2e40a2fe0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0011490 tx=0x7ff2f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.844+0000 7ff302600700 1 --2- 172.21.15.78:0/3088635259 >> [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001] conn(0x7ff2e40abb40 0x7ff2e40adfe0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:54.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.845+0000 7ff2fa000700 1 -- 172.21.15.78:0/3088635259 <== osd.0 v2:172.21.15.78:6802/1911630001 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff2e40ae6b0 con 0x7ff2e40abb40 2024-03-14T09:52:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.863+0000 7ff304600700 1 -- 172.21.15.78:0/3088635259 --> [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff2fc061430 con 0x7ff2e40abb40 2024-03-14T09:52:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.863+0000 7ff2fa000700 1 -- 172.21.15.78:0/3088635259 <== osd.0 v2:172.21.15.78:6802/1911630001 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff2fc061430 con 0x7ff2e40abb40 2024-03-14T09:52:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.864+0000 7ff304600700 1 -- 172.21.15.78:0/3088635259 >> [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001] conn(0x7ff2e40abb40 msgr2=0x7ff2e40adfe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.864+0000 7ff304600700 1 --2- 172.21.15.78:0/3088635259 >> [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001] conn(0x7ff2e40abb40 0x7ff2e40adfe0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.864+0000 7ff304600700 1 -- 172.21.15.78:0/3088635259 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff2e40a0b20 msgr2=0x7ff2e40a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.864+0000 7ff304600700 1 --2- 172.21.15.78:0/3088635259 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff2e40a0b20 0x7ff2e40a2fe0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0011490 tx=0x7ff2f0009260 comp rx=0 tx=0).stop 2024-03-14T09:52:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.864+0000 7ff304600700 1 -- 172.21.15.78:0/3088635259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2fc101960 msgr2=0x7ff2fc0735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.864+0000 7ff304600700 1 --2- 172.21.15.78:0/3088635259 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2fc101960 0x7ff2fc0735f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8000c00 tx=0x7ff2e8005740 comp rx=0 tx=0).stop 2024-03-14T09:52:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.864+0000 7ff302600700 1 -- 172.21.15.78:0/3088635259 reap_dead start 2024-03-14T09:52:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.864+0000 7ff304600700 1 -- 172.21.15.78:0/3088635259 shutdown_connections 2024-03-14T09:52:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.864+0000 7ff304600700 1 -- 172.21.15.78:0/3088635259 >> 172.21.15.78:0/3088635259 conn(0x7ff2fc0fc800 msgr2=0x7ff2fc0fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.864+0000 7ff304600700 1 -- 172.21.15.78:0/3088635259 shutdown_connections 2024-03-14T09:52:54.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.864+0000 7ff304600700 1 -- 172.21.15.78:0/3088635259 wait complete. 2024-03-14T09:52:54.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.626+0000 7f7a9d000700 1 -- 172.21.15.78:0/2233634287 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a98106ec0 msgr2=0x7f7a981072a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.626+0000 7f7a9d000700 1 --2- 172.21.15.78:0/2233634287 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a98106ec0 0x7f7a981072a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7a880098f0 tx=0x7f7a88009c00 comp rx=0 tx=0).stop 2024-03-14T09:52:54.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.626+0000 7f7a9d000700 1 -- 172.21.15.78:0/2233634287 shutdown_connections 2024-03-14T09:52:54.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.626+0000 7f7a9d000700 1 --2- 172.21.15.78:0/2233634287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a98069140 0x7f7a9810db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.626+0000 7f7a9d000700 1 --2- 172.21.15.78:0/2233634287 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a98106ec0 0x7f7a981072a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.626+0000 7f7a9d000700 1 --2- 172.21.15.78:0/2233634287 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a98068780 0x7f7a98068c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.626+0000 7f7a9d000700 1 -- 172.21.15.78:0/2233634287 >> 172.21.15.78:0/2233634287 conn(0x7f7a98076db0 msgr2=0x7f7a980771c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.627+0000 7f7a9d000700 1 -- 172.21.15.78:0/2233634287 shutdown_connections 2024-03-14T09:52:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.627+0000 7f7a9d000700 1 -- 172.21.15.78:0/2233634287 wait complete. 2024-03-14T09:52:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.628+0000 7f7a9d000700 1 Processor -- start 2024-03-14T09:52:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.628+0000 7f7a9d000700 1 -- start start 2024-03-14T09:52:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.629+0000 7f7a9d000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a98068780 0x7f7a9819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.629+0000 7f7a9d000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a98069140 0x7f7a9819fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.629+0000 7f7a9d000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a98199920 0x7f7a98199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.629+0000 7f7a9d000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9819a3a0 con 0x7f7a98068780 2024-03-14T09:52:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.629+0000 7f7a9d000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9819a510 con 0x7f7a98199920 2024-03-14T09:52:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.629+0000 7f7a9d000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9819a680 con 0x7f7a98069140 2024-03-14T09:52:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.629+0000 7f7a95e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a98068780 0x7f7a9819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.629+0000 7f7a95e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a98068780 0x7f7a9819f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34324/0 (socket says 172.21.15.78:34324) 2024-03-14T09:52:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.629+0000 7f7a95e00700 1 -- 172.21.15.78:0/1777455802 learned_addr learned my addr 172.21.15.78:0/1777455802 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:54.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.629+0000 7f7a96800700 1 --2- 172.21.15.78:0/1777455802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a98199920 0x7f7a98199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.630+0000 7f7a95400700 1 --2- 172.21.15.78:0/1777455802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a98069140 0x7f7a9819fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.630+0000 7f7a95e00700 1 -- 172.21.15.78:0/1777455802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a98069140 msgr2=0x7f7a9819fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.630+0000 7f7a95e00700 1 --2- 172.21.15.78:0/1777455802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7a98069140 0x7f7a9819fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.630+0000 7f7a95e00700 1 -- 172.21.15.78:0/1777455802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a98199920 msgr2=0x7f7a98199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.630+0000 7f7a95e00700 1 --2- 172.21.15.78:0/1777455802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a98199920 0x7f7a98199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.630+0000 7f7a95e00700 1 -- 172.21.15.78:0/1777455802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a88009520 con 0x7f7a98068780 2024-03-14T09:52:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.630+0000 7f7a95e00700 1 --2- 172.21.15.78:0/1777455802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a98068780 0x7f7a9819f7e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009fd0 tx=0x7f7a8800f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.631+0000 7f7a8ea00700 1 -- 172.21.15.78:0/1777455802 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8801c070 con 0x7f7a98068780 2024-03-14T09:52:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.631+0000 7f7a9d000700 1 -- 172.21.15.78:0/1777455802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a981a6600 con 0x7f7a98068780 2024-03-14T09:52:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.631+0000 7f7a8ea00700 1 -- 172.21.15.78:0/1777455802 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a8800bb20 con 0x7f7a98068780 2024-03-14T09:52:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.631+0000 7f7a8ea00700 1 -- 172.21.15.78:0/1777455802 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a88003900 con 0x7f7a98068780 2024-03-14T09:52:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.631+0000 7f7a9d000700 1 -- 172.21.15.78:0/1777455802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a981a69c0 con 0x7f7a98068780 2024-03-14T09:52:54.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.632+0000 7f7a9d000700 1 -- 172.21.15.78:0/1777455802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7a98111380 con 0x7f7a98068780 2024-03-14T09:52:54.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.634+0000 7f7a8ea00700 1 -- 172.21.15.78:0/1777455802 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f7a88020030 con 0x7f7a98068780 2024-03-14T09:52:54.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.636+0000 7f7a8ea00700 1 --2- 172.21.15.78:0/1777455802 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7a7c0a0b20 0x7f7a7c0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.637+0000 7f7a8ea00700 1 -- 172.21.15.78:0/1777455802 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7a880cd440 con 0x7f7a98068780 2024-03-14T09:52:54.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.637+0000 7f7a95400700 1 --2- 172.21.15.78:0/1777455802 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7a7c0a0b20 0x7f7a7c0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.637+0000 7f7a8ea00700 1 --2- 172.21.15.78:0/1777455802 >> [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058] conn(0x7f7a7c0abb40 0x7f7a7c0adfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.637+0000 7f7a8ea00700 1 -- 172.21.15.78:0/1777455802 --> [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a7c0ae6b0 con 0x7f7a7c0abb40 2024-03-14T09:52:54.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.637+0000 7f7a8ea00700 1 -- 172.21.15.78:0/1777455802 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a880cfa20 con 0x7f7a98068780 2024-03-14T09:52:54.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.637+0000 7f7a96800700 1 --2- 172.21.15.78:0/1777455802 >> [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058] conn(0x7f7a7c0abb40 0x7f7a7c0adfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.638+0000 7f7a95400700 1 --2- 172.21.15.78:0/1777455802 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7a7c0a0b20 0x7f7a7c0a2fe0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7a9819aae0 tx=0x7f7a84009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:54.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.638+0000 7f7a96800700 1 --2- 172.21.15.78:0/1777455802 >> [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058] conn(0x7f7a7c0abb40 0x7f7a7c0adfe0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:54.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.639+0000 7f7a8ea00700 1 -- 172.21.15.78:0/1777455802 <== osd.3 v2:172.21.15.78:6810/3422121058 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7a7c0ae6b0 con 0x7f7a7c0abb40 2024-03-14T09:52:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.656+0000 7f7a9d000700 1 -- 172.21.15.78:0/1777455802 --> [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7a9810c8d0 con 0x7f7a7c0abb40 2024-03-14T09:52:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.657+0000 7f7a8ea00700 1 -- 172.21.15.78:0/1777455802 <== osd.3 v2:172.21.15.78:6810/3422121058 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7a9810c8d0 con 0x7f7a7c0abb40 2024-03-14T09:52:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.657+0000 7f7a9d000700 1 -- 172.21.15.78:0/1777455802 >> [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058] conn(0x7f7a7c0abb40 msgr2=0x7f7a7c0adfe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.657+0000 7f7a9d000700 1 --2- 172.21.15.78:0/1777455802 >> [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058] conn(0x7f7a7c0abb40 0x7f7a7c0adfe0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.657+0000 7f7a9d000700 1 -- 172.21.15.78:0/1777455802 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7a7c0a0b20 msgr2=0x7f7a7c0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.657+0000 7f7a9d000700 1 --2- 172.21.15.78:0/1777455802 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7a7c0a0b20 0x7f7a7c0a2fe0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7a9819aae0 tx=0x7f7a84009310 comp rx=0 tx=0).stop 2024-03-14T09:52:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.657+0000 7f7a9d000700 1 -- 172.21.15.78:0/1777455802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a98068780 msgr2=0x7f7a9819f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.657+0000 7f7a9d000700 1 --2- 172.21.15.78:0/1777455802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a98068780 0x7f7a9819f7e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009fd0 tx=0x7f7a8800f9e0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.657+0000 7f7a96800700 1 -- 172.21.15.78:0/1777455802 reap_dead start 2024-03-14T09:52:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.657+0000 7f7a9d000700 1 -- 172.21.15.78:0/1777455802 shutdown_connections 2024-03-14T09:52:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.657+0000 7f7a9d000700 1 -- 172.21.15.78:0/1777455802 >> 172.21.15.78:0/1777455802 conn(0x7f7a98076db0 msgr2=0x7f7a9810bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:54.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.658+0000 7f7a9d000700 1 -- 172.21.15.78:0/1777455802 shutdown_connections 2024-03-14T09:52:54.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.658+0000 7f7a9d000700 1 -- 172.21.15.78:0/1777455802 wait complete. 2024-03-14T09:52:54.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.762+0000 7f8784a00700 1 -- 172.21.15.78:0/3054217363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8780100fb0 msgr2=0x7f8780101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.762+0000 7f8784a00700 1 --2- 172.21.15.78:0/3054217363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8780100fb0 0x7f8780101390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8770009990 tx=0x7f8770009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.762+0000 7f8784a00700 1 -- 172.21.15.78:0/3054217363 shutdown_connections 2024-03-14T09:52:54.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.762+0000 7f8784a00700 1 --2- 172.21.15.78:0/3054217363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8780100fb0 0x7f8780101390 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.762+0000 7f8784a00700 1 --2- 172.21.15.78:0/3054217363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8780106090 0x7f878010dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.762+0000 7f8784a00700 1 --2- 172.21.15.78:0/3054217363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8780101960 0x7f87801059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.762+0000 7f8784a00700 1 -- 172.21.15.78:0/3054217363 >> 172.21.15.78:0/3054217363 conn(0x7f87800fc800 msgr2=0x7f87800fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:54.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.763+0000 7f8784a00700 1 -- 172.21.15.78:0/3054217363 shutdown_connections 2024-03-14T09:52:54.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.763+0000 7f8784a00700 1 -- 172.21.15.78:0/3054217363 wait complete. 2024-03-14T09:52:54.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.764+0000 7f8784a00700 1 Processor -- start 2024-03-14T09:52:54.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.764+0000 7f8784a00700 1 -- start start 2024-03-14T09:52:54.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.764+0000 7f8784a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8780100fb0 0x7f878019f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.764+0000 7f8784a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8780101960 0x7f878019fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.764+0000 7f8784a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8780106090 0x7f8780199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.764+0000 7f8784a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8780199de0 con 0x7f8780100fb0 2024-03-14T09:52:54.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.764+0000 7f8784a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8780199f50 con 0x7f8780106090 2024-03-14T09:52:54.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.764+0000 7f8784a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878019a110 con 0x7f8780101960 2024-03-14T09:52:54.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.765+0000 7f877e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8780106090 0x7f8780199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.765+0000 7f877de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8780100fb0 0x7f878019f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.765+0000 7f877e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8780106090 0x7f8780199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:47716/0 (socket says 172.21.15.78:47716) 2024-03-14T09:52:54.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.765+0000 7f877e800700 1 -- 172.21.15.78:0/765500162 learned_addr learned my addr 172.21.15.78:0/765500162 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:54.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.765+0000 7f877d400700 1 --2- 172.21.15.78:0/765500162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8780101960 0x7f878019fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.765+0000 7f877e800700 1 -- 172.21.15.78:0/765500162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8780101960 msgr2=0x7f878019fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.765+0000 7f877e800700 1 --2- 172.21.15.78:0/765500162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8780101960 0x7f878019fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.765+0000 7f877e800700 1 -- 172.21.15.78:0/765500162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8780100fb0 msgr2=0x7f878019f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.765+0000 7f877e800700 1 --2- 172.21.15.78:0/765500162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8780100fb0 0x7f878019f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.765+0000 7f877e800700 1 -- 172.21.15.78:0/765500162 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876c009520 con 0x7f8780106090 2024-03-14T09:52:54.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.765+0000 7f877de00700 1 --2- 172.21.15.78:0/765500162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8780100fb0 0x7f878019f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:52:54.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.765+0000 7f877e800700 1 --2- 172.21.15.78:0/765500162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8780106090 0x7f8780199780 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8778007ee0 tx=0x7f877800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:54.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.765+0000 7f8776a00700 1 -- 172.21.15.78:0/765500162 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8778017070 con 0x7f8780106090 2024-03-14T09:52:54.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.766+0000 7f8784a00700 1 -- 172.21.15.78:0/765500162 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87700095f0 con 0x7f8780106090 2024-03-14T09:52:54.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.766+0000 7f8784a00700 1 -- 172.21.15.78:0/765500162 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87801a6410 con 0x7f8780106090 2024-03-14T09:52:54.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.766+0000 7f8776a00700 1 -- 172.21.15.78:0/765500162 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f877800f040 con 0x7f8780106090 2024-03-14T09:52:54.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.766+0000 7f8776a00700 1 -- 172.21.15.78:0/765500162 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8778005740 con 0x7f8780106090 2024-03-14T09:52:54.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.767+0000 7f8784a00700 1 -- 172.21.15.78:0/765500162 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8760000fc0 con 0x7f8780106090 2024-03-14T09:52:54.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.769+0000 7f8776a00700 1 -- 172.21.15.78:0/765500162 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f87780058a0 con 0x7f8780106090 2024-03-14T09:52:54.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.770+0000 7f8776a00700 1 --2- 172.21.15.78:0/765500162 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87640a0ba0 0x7f87640a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.771+0000 7f8776a00700 1 -- 172.21.15.78:0/765500162 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f87780cbd60 con 0x7f8780106090 2024-03-14T09:52:54.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.771+0000 7f877de00700 1 --2- 172.21.15.78:0/765500162 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87640a0ba0 0x7f87640a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.771+0000 7f8776a00700 1 --2- 172.21.15.78:0/765500162 >> [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535] conn(0x7f87640abbc0 0x7f87640ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.771+0000 7f8776a00700 1 -- 172.21.15.78:0/765500162 --> [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f87640ae730 con 0x7f87640abbc0 2024-03-14T09:52:54.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.771+0000 7f8776a00700 1 -- 172.21.15.78:0/765500162 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87780cc0d0 con 0x7f8780106090 2024-03-14T09:52:54.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.772+0000 7f877d400700 1 --2- 172.21.15.78:0/765500162 >> [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535] conn(0x7f87640abbc0 0x7f87640ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.772+0000 7f877de00700 1 --2- 172.21.15.78:0/765500162 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87640a0ba0 0x7f87640a3060 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8770006010 tx=0x7f877000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:54.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.772+0000 7f877d400700 1 --2- 172.21.15.78:0/765500162 >> [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535] conn(0x7f87640abbc0 0x7f87640ae060 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:54.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.773+0000 7f8776a00700 1 -- 172.21.15.78:0/765500162 <== osd.2 v2:172.21.15.152:6800/3822041535 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f87640ae730 con 0x7f87640abbc0 2024-03-14T09:52:54.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.790+0000 7f8784a00700 1 -- 172.21.15.78:0/765500162 --> [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8760002d40 con 0x7f87640abbc0 2024-03-14T09:52:54.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.790+0000 7f8776a00700 1 -- 172.21.15.78:0/765500162 <== osd.2 v2:172.21.15.152:6800/3822041535 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8760002d40 con 0x7f87640abbc0 2024-03-14T09:52:54.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.791+0000 7f8784a00700 1 -- 172.21.15.78:0/765500162 >> [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535] conn(0x7f87640abbc0 msgr2=0x7f87640ae060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.791+0000 7f8784a00700 1 --2- 172.21.15.78:0/765500162 >> [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535] conn(0x7f87640abbc0 0x7f87640ae060 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.791+0000 7f8784a00700 1 -- 172.21.15.78:0/765500162 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87640a0ba0 msgr2=0x7f87640a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.791+0000 7f8784a00700 1 --2- 172.21.15.78:0/765500162 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87640a0ba0 0x7f87640a3060 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8770006010 tx=0x7f877000b260 comp rx=0 tx=0).stop 2024-03-14T09:52:54.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.791+0000 7f8784a00700 1 -- 172.21.15.78:0/765500162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8780106090 msgr2=0x7f8780199780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.791+0000 7f8784a00700 1 --2- 172.21.15.78:0/765500162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8780106090 0x7f8780199780 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8778007ee0 tx=0x7f877800d3b0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.791+0000 7f877e800700 1 -- 172.21.15.78:0/765500162 reap_dead start 2024-03-14T09:52:54.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.791+0000 7f8784a00700 1 -- 172.21.15.78:0/765500162 shutdown_connections 2024-03-14T09:52:54.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.791+0000 7f8784a00700 1 -- 172.21.15.78:0/765500162 >> 172.21.15.78:0/765500162 conn(0x7f87800fc800 msgr2=0x7f87800fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:54.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.791+0000 7f8784a00700 1 -- 172.21.15.78:0/765500162 shutdown_connections 2024-03-14T09:52:54.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:53.791+0000 7f8784a00700 1 -- 172.21.15.78:0/765500162 wait complete. 2024-03-14T09:52:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:133143986222 2024-03-14T09:52:54.651 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-14T09:52:54.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.921+0000 7fed1fa00700 1 -- 172.21.15.78:0/2760615211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed18100fb0 msgr2=0x7fed18101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.921+0000 7fed1fa00700 1 --2- 172.21.15.78:0/2760615211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed18100fb0 0x7fed18101390 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009990 tx=0x7fed0c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.921+0000 7fed1fa00700 1 -- 172.21.15.78:0/2760615211 shutdown_connections 2024-03-14T09:52:54.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.921+0000 7fed1fa00700 1 --2- 172.21.15.78:0/2760615211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed18106090 0x7fed1810dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.921+0000 7fed1fa00700 1 --2- 172.21.15.78:0/2760615211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed18100fb0 0x7fed18101390 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.921+0000 7fed1fa00700 1 --2- 172.21.15.78:0/2760615211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed18101960 0x7fed181059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.921+0000 7fed1fa00700 1 -- 172.21.15.78:0/2760615211 >> 172.21.15.78:0/2760615211 conn(0x7fed180fc800 msgr2=0x7fed180fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:54.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.922+0000 7fed1fa00700 1 -- 172.21.15.78:0/2760615211 shutdown_connections 2024-03-14T09:52:54.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.922+0000 7fed1fa00700 1 -- 172.21.15.78:0/2760615211 wait complete. 2024-03-14T09:52:54.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.923+0000 7fed1fa00700 1 Processor -- start 2024-03-14T09:52:54.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.923+0000 7fed1fa00700 1 -- start start 2024-03-14T09:52:54.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.924+0000 7fed1fa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed18100fb0 0x7fed1810d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.924+0000 7fed1fa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed18101960 0x7fed1810dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.924+0000 7fed1fa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed18106090 0x7fed1810a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.924+0000 7fed1fa00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1810b000 con 0x7fed18101960 2024-03-14T09:52:54.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.924+0000 7fed1fa00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1810b170 con 0x7fed18106090 2024-03-14T09:52:54.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.924+0000 7fed1fa00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1810b330 con 0x7fed18100fb0 2024-03-14T09:52:54.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.924+0000 7fed17e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed18101960 0x7fed1810dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.924+0000 7fed1d000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed18100fb0 0x7fed1810d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.925+0000 7fed17e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed18101960 0x7fed1810dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34254/0 (socket says 172.21.15.78:34254) 2024-03-14T09:52:54.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.925+0000 7fed17e00700 1 -- 172.21.15.78:0/2975746895 learned_addr learned my addr 172.21.15.78:0/2975746895 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:54.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.925+0000 7fed1d000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed18100fb0 0x7fed1810d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:46936/0 (socket says 172.21.15.78:46936) 2024-03-14T09:52:54.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.925+0000 7fed1da00700 1 --2- 172.21.15.78:0/2975746895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed18106090 0x7fed1810a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.925+0000 7fed1da00700 1 -- 172.21.15.78:0/2975746895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed18100fb0 msgr2=0x7fed1810d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.925+0000 7fed1da00700 1 --2- 172.21.15.78:0/2975746895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed18100fb0 0x7fed1810d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.925+0000 7fed1da00700 1 -- 172.21.15.78:0/2975746895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed18101960 msgr2=0x7fed1810dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.925+0000 7fed1da00700 1 --2- 172.21.15.78:0/2975746895 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed18101960 0x7fed1810dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.925+0000 7fed1da00700 1 -- 172.21.15.78:0/2975746895 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed04009520 con 0x7fed18106090 2024-03-14T09:52:54.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.925+0000 7fed1d000700 1 --2- 172.21.15.78:0/2975746895 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fed18100fb0 0x7fed1810d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:54.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.925+0000 7fed1da00700 1 --2- 172.21.15.78:0/2975746895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed18106090 0x7fed1810a830 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fed1000efc0 tx=0x7fed1000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:54.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.926+0000 7fed15600700 1 -- 172.21.15.78:0/2975746895 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed10018070 con 0x7fed18106090 2024-03-14T09:52:54.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.926+0000 7fed15600700 1 -- 172.21.15.78:0/2975746895 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed1000f040 con 0x7fed18106090 2024-03-14T09:52:54.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.926+0000 7fed15600700 1 -- 172.21.15.78:0/2975746895 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1000cd40 con 0x7fed18106090 2024-03-14T09:52:54.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.926+0000 7fed1fa00700 1 -- 172.21.15.78:0/2975746895 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0c0095f0 con 0x7fed18106090 2024-03-14T09:52:54.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.926+0000 7fed1fa00700 1 -- 172.21.15.78:0/2975746895 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed181aa720 con 0x7fed18106090 2024-03-14T09:52:54.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.927+0000 7fed1fa00700 1 -- 172.21.15.78:0/2975746895 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fed18074800 con 0x7fed18106090 2024-03-14T09:52:54.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.930+0000 7fed15600700 1 -- 172.21.15.78:0/2975746895 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fed10003a00 con 0x7fed18106090 2024-03-14T09:52:54.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.932+0000 7fed15600700 1 --2- 172.21.15.78:0/2975746895 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fed000a0ba0 0x7fed000a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.932+0000 7fed15600700 1 -- 172.21.15.78:0/2975746895 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fed100cc450 con 0x7fed18106090 2024-03-14T09:52:54.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.932+0000 7fed1d000700 1 --2- 172.21.15.78:0/2975746895 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fed000a0ba0 0x7fed000a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.933+0000 7fed15600700 1 --2- 172.21.15.78:0/2975746895 >> [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096] conn(0x7fed000abbc0 0x7fed000ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:54.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.933+0000 7fed15600700 1 -- 172.21.15.78:0/2975746895 --> [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fed000ae730 con 0x7fed000abbc0 2024-03-14T09:52:54.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.933+0000 7fed15600700 1 -- 172.21.15.78:0/2975746895 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed100cc7c0 con 0x7fed18106090 2024-03-14T09:52:54.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.933+0000 7fed1d000700 1 --2- 172.21.15.78:0/2975746895 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fed000a0ba0 0x7fed000a3060 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00b2e0 tx=0x7fed0c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:54.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.933+0000 7fed17e00700 1 --2- 172.21.15.78:0/2975746895 >> [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096] conn(0x7fed000abbc0 0x7fed000ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:54.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.934+0000 7fed17e00700 1 --2- 172.21.15.78:0/2975746895 >> [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096] conn(0x7fed000abbc0 0x7fed000ae060 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:54.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.935+0000 7fed15600700 1 -- 172.21.15.78:0/2975746895 <== osd.9 v2:172.21.15.78:6826/803422096 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fed000ae730 con 0x7fed000abbc0 2024-03-14T09:52:54.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.964+0000 7fed1fa00700 1 -- 172.21.15.78:0/2975746895 --> [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fed18102490 con 0x7fed000abbc0 2024-03-14T09:52:54.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.965+0000 7fed15600700 1 -- 172.21.15.78:0/2975746895 <== osd.9 v2:172.21.15.78:6826/803422096 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fed18102490 con 0x7fed000abbc0 2024-03-14T09:52:54.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.965+0000 7fed1fa00700 1 -- 172.21.15.78:0/2975746895 >> [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096] conn(0x7fed000abbc0 msgr2=0x7fed000ae060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.965+0000 7fed1fa00700 1 --2- 172.21.15.78:0/2975746895 >> [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096] conn(0x7fed000abbc0 0x7fed000ae060 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.966+0000 7fed1fa00700 1 -- 172.21.15.78:0/2975746895 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fed000a0ba0 msgr2=0x7fed000a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.966+0000 7fed1fa00700 1 --2- 172.21.15.78:0/2975746895 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fed000a0ba0 0x7fed000a3060 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00b2e0 tx=0x7fed0c005fd0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.966+0000 7fed1fa00700 1 -- 172.21.15.78:0/2975746895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed18106090 msgr2=0x7fed1810a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:54.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.966+0000 7fed1fa00700 1 --2- 172.21.15.78:0/2975746895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed18106090 0x7fed1810a830 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fed1000efc0 tx=0x7fed1000c3a0 comp rx=0 tx=0).stop 2024-03-14T09:52:54.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.966+0000 7fed1da00700 1 -- 172.21.15.78:0/2975746895 reap_dead start 2024-03-14T09:52:54.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.966+0000 7fed1fa00700 1 -- 172.21.15.78:0/2975746895 shutdown_connections 2024-03-14T09:52:54.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.966+0000 7fed1fa00700 1 -- 172.21.15.78:0/2975746895 >> 172.21.15.78:0/2975746895 conn(0x7fed180fc800 msgr2=0x7fed180fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:54.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.966+0000 7fed1fa00700 1 -- 172.21.15.78:0/2975746895 shutdown_connections 2024-03-14T09:52:54.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:52.966+0000 7fed1fa00700 1 -- 172.21.15.78:0/2975746895 wait complete. 2024-03-14T09:52:54.730 INFO:teuthology.orchestra.run.smithi078.stdout:317827579923 2024-03-14T09:52:54.730 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-14T09:52:55.169 INFO:teuthology.orchestra.run.smithi078.stdout:266287972380 2024-03-14T09:52:55.169 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-14T09:52:55.348 INFO:teuthology.orchestra.run.smithi078.stdout:103079215155 2024-03-14T09:52:55.348 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-14T09:52:55.629 INFO:teuthology.orchestra.run.smithi078.stdout:34359738430 2024-03-14T09:52:55.630 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-14T09:52:55.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.786+0000 7f5200600700 1 -- 172.21.15.78:0/1176580599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51f8100fb0 msgr2=0x7f51f8101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:55.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.786+0000 7f5200600700 1 --2- 172.21.15.78:0/1176580599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51f8100fb0 0x7f51f8101390 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f51ec009990 tx=0x7f51ec009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:55.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.786+0000 7f5200600700 1 -- 172.21.15.78:0/1176580599 shutdown_connections 2024-03-14T09:52:55.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.786+0000 7f5200600700 1 --2- 172.21.15.78:0/1176580599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51f8106090 0x7f51f810dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:55.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.786+0000 7f5200600700 1 --2- 172.21.15.78:0/1176580599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51f8100fb0 0x7f51f8101390 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:55.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.786+0000 7f5200600700 1 --2- 172.21.15.78:0/1176580599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51f8101960 0x7f51f81059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:55.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.786+0000 7f5200600700 1 -- 172.21.15.78:0/1176580599 >> 172.21.15.78:0/1176580599 conn(0x7f51f80fc800 msgr2=0x7f51f80fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:55.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.786+0000 7f5200600700 1 -- 172.21.15.78:0/1176580599 shutdown_connections 2024-03-14T09:52:55.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.786+0000 7f5200600700 1 -- 172.21.15.78:0/1176580599 wait complete. 2024-03-14T09:52:55.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.787+0000 7f5200600700 1 Processor -- start 2024-03-14T09:52:55.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.788+0000 7f5200600700 1 -- start start 2024-03-14T09:52:55.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.788+0000 7f5200600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51f8100fb0 0x7f51f819f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:55.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.788+0000 7f5200600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51f8101960 0x7f51f819fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:55.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.788+0000 7f5200600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51f8106090 0x7f51f8199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:55.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.788+0000 7f5200600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f8199cf0 con 0x7f51f8101960 2024-03-14T09:52:55.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.788+0000 7f5200600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f8199e60 con 0x7f51f8106090 2024-03-14T09:52:55.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.788+0000 7f5200600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f819a020 con 0x7f51f8100fb0 2024-03-14T09:52:55.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.788+0000 7f51fd200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51f8101960 0x7f51f819fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:55.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.788+0000 7f51fd200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51f8101960 0x7f51f819fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36238/0 (socket says 172.21.15.78:36238) 2024-03-14T09:52:55.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.788+0000 7f51fd200700 1 -- 172.21.15.78:0/1804347690 learned_addr learned my addr 172.21.15.78:0/1804347690 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:55.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.788+0000 7f51fdc00700 1 --2- 172.21.15.78:0/1804347690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51f8100fb0 0x7f51f819f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:55.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.788+0000 7f51fe600700 1 --2- 172.21.15.78:0/1804347690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51f8106090 0x7f51f8199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:55.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.789+0000 7f51fd200700 1 -- 172.21.15.78:0/1804347690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51f8100fb0 msgr2=0x7f51f819f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:55.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.789+0000 7f51fd200700 1 --2- 172.21.15.78:0/1804347690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51f8100fb0 0x7f51f819f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:55.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.789+0000 7f51fd200700 1 -- 172.21.15.78:0/1804347690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51f8106090 msgr2=0x7f51f8199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:55.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.789+0000 7f51fd200700 1 --2- 172.21.15.78:0/1804347690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51f8106090 0x7f51f8199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:55.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.789+0000 7f51fd200700 1 -- 172.21.15.78:0/1804347690 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ec0095f0 con 0x7f51f8101960 2024-03-14T09:52:55.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.789+0000 7f51fe600700 1 --2- 172.21.15.78:0/1804347690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51f8106090 0x7f51f8199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:55.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.789+0000 7f51fd200700 1 --2- 172.21.15.78:0/1804347690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51f8101960 0x7f51f819fa90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f51e400e970 tx=0x7f51e400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:55.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.789+0000 7f51f6000700 1 -- 172.21.15.78:0/1804347690 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e4009790 con 0x7f51f8101960 2024-03-14T09:52:55.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.789+0000 7f5200600700 1 -- 172.21.15.78:0/1804347690 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f819a300 con 0x7f51f8101960 2024-03-14T09:52:55.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.789+0000 7f51f6000700 1 -- 172.21.15.78:0/1804347690 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51e400c7e0 con 0x7f51f8101960 2024-03-14T09:52:55.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.790+0000 7f51f6000700 1 -- 172.21.15.78:0/1804347690 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e40054e0 con 0x7f51f8101960 2024-03-14T09:52:55.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.790+0000 7f5200600700 1 -- 172.21.15.78:0/1804347690 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f81a6540 con 0x7f51f8101960 2024-03-14T09:52:55.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.791+0000 7f5200600700 1 -- 172.21.15.78:0/1804347690 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f51f81113a0 con 0x7f51f8101960 2024-03-14T09:52:55.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.794+0000 7f51f6000700 1 -- 172.21.15.78:0/1804347690 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f51e400c950 con 0x7f51f8101960 2024-03-14T09:52:55.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.796+0000 7f51f6000700 1 --2- 172.21.15.78:0/1804347690 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f51e00a0b20 0x7f51e00a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:55.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.796+0000 7f51f6000700 1 -- 172.21.15.78:0/1804347690 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f51e4014070 con 0x7f51f8101960 2024-03-14T09:52:55.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.796+0000 7f51fdc00700 1 --2- 172.21.15.78:0/1804347690 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f51e00a0b20 0x7f51e00a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:55.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.796+0000 7f51f6000700 1 --2- 172.21.15.78:0/1804347690 >> [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397] conn(0x7f51e00abb40 0x7f51e00adfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:55.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.796+0000 7f51f6000700 1 -- 172.21.15.78:0/1804347690 --> [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51e00ae6b0 con 0x7f51e00abb40 2024-03-14T09:52:55.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.796+0000 7f51f6000700 1 -- 172.21.15.78:0/1804347690 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51e40ccd80 con 0x7f51f8101960 2024-03-14T09:52:55.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.797+0000 7f51fe600700 1 --2- 172.21.15.78:0/1804347690 >> [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397] conn(0x7f51e00abb40 0x7f51e00adfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:55.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.797+0000 7f51fdc00700 1 --2- 172.21.15.78:0/1804347690 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f51e00a0b20 0x7f51e00a2fe0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00b2e0 tx=0x7f51ec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:55.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.797+0000 7f51fe600700 1 --2- 172.21.15.78:0/1804347690 >> [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397] conn(0x7f51e00abb40 0x7f51e00adfe0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:55.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.798+0000 7f51f6000700 1 -- 172.21.15.78:0/1804347690 <== osd.8 v2:172.21.15.152:6816/2604540397 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f51e00ae6b0 con 0x7f51e00abb40 2024-03-14T09:52:55.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.815+0000 7f5200600700 1 -- 172.21.15.78:0/1804347690 --> [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f51f810d290 con 0x7f51e00abb40 2024-03-14T09:52:55.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.815+0000 7f51f6000700 1 -- 172.21.15.78:0/1804347690 <== osd.8 v2:172.21.15.152:6816/2604540397 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f51f810d290 con 0x7f51e00abb40 2024-03-14T09:52:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.816+0000 7f5200600700 1 -- 172.21.15.78:0/1804347690 >> [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397] conn(0x7f51e00abb40 msgr2=0x7f51e00adfe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.816+0000 7f5200600700 1 --2- 172.21.15.78:0/1804347690 >> [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397] conn(0x7f51e00abb40 0x7f51e00adfe0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.816+0000 7f5200600700 1 -- 172.21.15.78:0/1804347690 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f51e00a0b20 msgr2=0x7f51e00a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.816+0000 7f5200600700 1 --2- 172.21.15.78:0/1804347690 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f51e00a0b20 0x7f51e00a2fe0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00b2e0 tx=0x7f51ec005fd0 comp rx=0 tx=0).stop 2024-03-14T09:52:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.816+0000 7f5200600700 1 -- 172.21.15.78:0/1804347690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51f8101960 msgr2=0x7f51f819fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.816+0000 7f5200600700 1 --2- 172.21.15.78:0/1804347690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51f8101960 0x7f51f819fa90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f51e400e970 tx=0x7f51e400ec80 comp rx=0 tx=0).stop 2024-03-14T09:52:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.816+0000 7f51fe600700 1 -- 172.21.15.78:0/1804347690 reap_dead start 2024-03-14T09:52:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.816+0000 7f5200600700 1 -- 172.21.15.78:0/1804347690 shutdown_connections 2024-03-14T09:52:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.816+0000 7f5200600700 1 -- 172.21.15.78:0/1804347690 >> 172.21.15.78:0/1804347690 conn(0x7f51f80fc800 msgr2=0x7f51f80feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.816+0000 7f5200600700 1 -- 172.21.15.78:0/1804347690 shutdown_connections 2024-03-14T09:52:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:55.816+0000 7f5200600700 1 -- 172.21.15.78:0/1804347690 wait complete. 2024-03-14T09:52:56.060 INFO:teuthology.orchestra.run.smithi078.stdout:77309411381 2024-03-14T09:52:56.060 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-14T09:52:56.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.089+0000 7f28bcc00700 1 -- 172.21.15.78:0/286646053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28b8106ec0 msgr2=0x7f28b81072a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:56.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.089+0000 7f28bcc00700 1 --2- 172.21.15.78:0/286646053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28b8106ec0 0x7f28b81072a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f28a8009990 tx=0x7f28a8009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:56.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.089+0000 7f28bcc00700 1 -- 172.21.15.78:0/286646053 shutdown_connections 2024-03-14T09:52:56.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.089+0000 7f28bcc00700 1 --2- 172.21.15.78:0/286646053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28b8069140 0x7f28b810db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:56.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.089+0000 7f28bcc00700 1 --2- 172.21.15.78:0/286646053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28b8106ec0 0x7f28b81072a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:56.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.089+0000 7f28bcc00700 1 --2- 172.21.15.78:0/286646053 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28b8068780 0x7f28b8068c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:56.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.089+0000 7f28bcc00700 1 -- 172.21.15.78:0/286646053 >> 172.21.15.78:0/286646053 conn(0x7f28b8076db0 msgr2=0x7f28b80771c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:56.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.090+0000 7f28bcc00700 1 -- 172.21.15.78:0/286646053 shutdown_connections 2024-03-14T09:52:56.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.090+0000 7f28bcc00700 1 -- 172.21.15.78:0/286646053 wait complete. 2024-03-14T09:52:56.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.091+0000 7f28bcc00700 1 Processor -- start 2024-03-14T09:52:56.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.091+0000 7f28bcc00700 1 -- start start 2024-03-14T09:52:56.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.091+0000 7f28bcc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28b8068780 0x7f28b8199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:56.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.091+0000 7f28bcc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28b8069140 0x7f28b8199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:56.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.091+0000 7f28bcc00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28b8106ec0 0x7f28b819cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:56.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.091+0000 7f28bcc00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b819d5b0 con 0x7f28b8069140 2024-03-14T09:52:56.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.091+0000 7f28bcc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b819d720 con 0x7f28b8068780 2024-03-14T09:52:56.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.091+0000 7f28bcc00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b819d8e0 con 0x7f28b8106ec0 2024-03-14T09:52:56.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.091+0000 7f28b5400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28b8069140 0x7f28b8199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:56.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.092+0000 7f28b5400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28b8069140 0x7f28b8199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36258/0 (socket says 172.21.15.78:36258) 2024-03-14T09:52:56.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.092+0000 7f28b5400700 1 -- 172.21.15.78:0/232091572 learned_addr learned my addr 172.21.15.78:0/232091572 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:56.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.092+0000 7f28b5e00700 1 --2- 172.21.15.78:0/232091572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28b8068780 0x7f28b8199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:56.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.092+0000 7f28b6800700 1 --2- 172.21.15.78:0/232091572 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28b8106ec0 0x7f28b819cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:56.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.092+0000 7f28b5e00700 1 -- 172.21.15.78:0/232091572 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28b8106ec0 msgr2=0x7f28b819cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:56.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.092+0000 7f28b5e00700 1 --2- 172.21.15.78:0/232091572 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28b8106ec0 0x7f28b819cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:56.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.092+0000 7f28b5e00700 1 -- 172.21.15.78:0/232091572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28b8069140 msgr2=0x7f28b8199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:56.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.092+0000 7f28b5e00700 1 --2- 172.21.15.78:0/232091572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28b8069140 0x7f28b8199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:56.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.092+0000 7f28b5e00700 1 -- 172.21.15.78:0/232091572 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a80095f0 con 0x7f28b8068780 2024-03-14T09:52:56.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.092+0000 7f28b5400700 1 --2- 172.21.15.78:0/232091572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28b8069140 0x7f28b8199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:52:56.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.092+0000 7f28b5e00700 1 --2- 172.21.15.78:0/232091572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28b8068780 0x7f28b8199770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f28a80098e0 tx=0x7f28a8005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:56.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.093+0000 7f28aea00700 1 -- 172.21.15.78:0/232091572 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a801d070 con 0x7f28b8068780 2024-03-14T09:52:56.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.093+0000 7f28aea00700 1 -- 172.21.15.78:0/232091572 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28a8022470 con 0x7f28b8068780 2024-03-14T09:52:56.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.093+0000 7f28aea00700 1 -- 172.21.15.78:0/232091572 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a8017650 con 0x7f28b8068780 2024-03-14T09:52:56.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.093+0000 7f28bcc00700 1 -- 172.21.15.78:0/232091572 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b819db60 con 0x7f28b8068780 2024-03-14T09:52:56.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.093+0000 7f28bcc00700 1 -- 172.21.15.78:0/232091572 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b81a6340 con 0x7f28b8068780 2024-03-14T09:52:56.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.094+0000 7f28a3e00700 1 -- 172.21.15.78:0/232091572 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2890000fc0 con 0x7f28b8068780 2024-03-14T09:52:56.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.096+0000 7f28aea00700 1 -- 172.21.15.78:0/232091572 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f28a80225e0 con 0x7f28b8068780 2024-03-14T09:52:56.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.097+0000 7f28aea00700 1 --2- 172.21.15.78:0/232091572 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f289c0a0ba0 0x7f289c0a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:56.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.098+0000 7f28aea00700 1 -- 172.21.15.78:0/232091572 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f28a80ce120 con 0x7f28b8068780 2024-03-14T09:52:56.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.098+0000 7f28b5400700 1 --2- 172.21.15.78:0/232091572 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f289c0a0ba0 0x7f289c0a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:56.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.098+0000 7f28aea00700 1 --2- 172.21.15.78:0/232091572 >> [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089] conn(0x7f289c0abbc0 0x7f289c0ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:56.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.098+0000 7f28aea00700 1 -- 172.21.15.78:0/232091572 --> [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f289c0ae730 con 0x7f289c0abbc0 2024-03-14T09:52:56.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.098+0000 7f28aea00700 1 -- 172.21.15.78:0/232091572 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28a80ce490 con 0x7f28b8068780 2024-03-14T09:52:56.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.099+0000 7f28b6800700 1 --2- 172.21.15.78:0/232091572 >> [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089] conn(0x7f289c0abbc0 0x7f289c0ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:56.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.099+0000 7f28b5400700 1 --2- 172.21.15.78:0/232091572 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f289c0a0ba0 0x7f289c0a3060 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f28b819acd0 tx=0x7f28a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:56.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.099+0000 7f28b6800700 1 --2- 172.21.15.78:0/232091572 >> [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089] conn(0x7f289c0abbc0 0x7f289c0ae060 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:56.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.100+0000 7f28aea00700 1 -- 172.21.15.78:0/232091572 <== osd.10 v2:172.21.15.132:6824/81920089 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f289c0ae730 con 0x7f289c0abbc0 2024-03-14T09:52:56.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.118+0000 7f28a3e00700 1 -- 172.21.15.78:0/232091572 --> [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2890002d40 con 0x7f289c0abbc0 2024-03-14T09:52:56.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.119+0000 7f28aea00700 1 -- 172.21.15.78:0/232091572 <== osd.10 v2:172.21.15.132:6824/81920089 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2890002d40 con 0x7f289c0abbc0 2024-03-14T09:52:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.119+0000 7f28a3e00700 1 -- 172.21.15.78:0/232091572 >> [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089] conn(0x7f289c0abbc0 msgr2=0x7f289c0ae060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.119+0000 7f28a3e00700 1 --2- 172.21.15.78:0/232091572 >> [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089] conn(0x7f289c0abbc0 0x7f289c0ae060 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.119+0000 7f28a3e00700 1 -- 172.21.15.78:0/232091572 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f289c0a0ba0 msgr2=0x7f289c0a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.119+0000 7f28a3e00700 1 --2- 172.21.15.78:0/232091572 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f289c0a0ba0 0x7f289c0a3060 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f28b819acd0 tx=0x7f28a4009260 comp rx=0 tx=0).stop 2024-03-14T09:52:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.120+0000 7f28a3e00700 1 -- 172.21.15.78:0/232091572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28b8068780 msgr2=0x7f28b8199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.120+0000 7f28a3e00700 1 --2- 172.21.15.78:0/232091572 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f28b8068780 0x7f28b8199770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f28a80098e0 tx=0x7f28a8005740 comp rx=0 tx=0).stop 2024-03-14T09:52:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.120+0000 7f28b6800700 1 -- 172.21.15.78:0/232091572 reap_dead start 2024-03-14T09:52:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.120+0000 7f28a3e00700 1 -- 172.21.15.78:0/232091572 shutdown_connections 2024-03-14T09:52:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.120+0000 7f28a3e00700 1 -- 172.21.15.78:0/232091572 >> 172.21.15.78:0/232091572 conn(0x7f28b8076db0 msgr2=0x7f28b810bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.120+0000 7f28a3e00700 1 -- 172.21.15.78:0/232091572 shutdown_connections 2024-03-14T09:52:56.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:56.120+0000 7f28a3e00700 1 -- 172.21.15.78:0/232091572 wait complete. 2024-03-14T09:52:56.145 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:56 smithi078 ceph-mon[38143]: pgmap v262: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s; 57 B/s, 0 keys/s, 1 objects/s recovering 2024-03-14T09:52:56.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:56 smithi152 ceph-mon[48279]: pgmap v262: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s; 57 B/s, 0 keys/s, 1 objects/s recovering 2024-03-14T09:52:56.518 INFO:teuthology.orchestra.run.smithi078.stdout:206158430243 2024-03-14T09:52:56.518 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-14T09:52:56.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:56 smithi132 ceph-mon[47959]: pgmap v262: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s; 57 B/s, 0 keys/s, 1 objects/s recovering 2024-03-14T09:52:56.631 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:56.752 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:56.876 INFO:teuthology.orchestra.run.smithi078.stdout:287762808855 2024-03-14T09:52:56.876 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-14T09:52:57.060 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:57.065 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:57.068 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:57.460 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:57.534 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:58.167 INFO:teuthology.orchestra.run.smithi078.stdout:236223201310 2024-03-14T09:52:58.167 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-14T09:52:58.336 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:52:58 smithi078 ceph-mon[38143]: pgmap v263: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s; 57 B/s, 0 keys/s, 1 objects/s recovering 2024-03-14T09:52:58.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:52:58 smithi152 ceph-mon[48279]: pgmap v263: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s; 57 B/s, 0 keys/s, 1 objects/s recovering 2024-03-14T09:52:58.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:52:58 smithi132 ceph-mon[47959]: pgmap v263: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s; 57 B/s, 0 keys/s, 1 objects/s recovering 2024-03-14T09:52:58.545 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:52:59.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.773+0000 7fcb95e00700 1 -- 172.21.15.78:0/644782091 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb90106ec0 msgr2=0x7fcb901072a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:59.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.773+0000 7fcb95e00700 1 --2- 172.21.15.78:0/644782091 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb90106ec0 0x7fcb901072a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcb80009990 tx=0x7fcb80009ca0 comp rx=0 tx=0).stop 2024-03-14T09:52:59.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.774+0000 7fcb95e00700 1 -- 172.21.15.78:0/644782091 shutdown_connections 2024-03-14T09:52:59.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.774+0000 7fcb95e00700 1 --2- 172.21.15.78:0/644782091 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb90106ec0 0x7fcb901072a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:59.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.774+0000 7fcb95e00700 1 --2- 172.21.15.78:0/644782091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb90069140 0x7fcb9010db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:59.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.774+0000 7fcb95e00700 1 --2- 172.21.15.78:0/644782091 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb90068780 0x7fcb90068c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:59.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.774+0000 7fcb95e00700 1 -- 172.21.15.78:0/644782091 >> 172.21.15.78:0/644782091 conn(0x7fcb90076db0 msgr2=0x7fcb900771c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:52:59.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.774+0000 7fcb95e00700 1 -- 172.21.15.78:0/644782091 shutdown_connections 2024-03-14T09:52:59.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.774+0000 7fcb95e00700 1 -- 172.21.15.78:0/644782091 wait complete. 2024-03-14T09:52:59.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.775+0000 7fcb95e00700 1 Processor -- start 2024-03-14T09:52:59.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.775+0000 7fcb95e00700 1 -- start start 2024-03-14T09:52:59.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.775+0000 7fcb95e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb90068780 0x7fcb900ffce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:59.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.775+0000 7fcb95e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb90069140 0x7fcb90100220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:59.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.775+0000 7fcb95e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb90106ec0 0x7fcb90100760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:59.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.775+0000 7fcb95e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb901a59d0 con 0x7fcb90068780 2024-03-14T09:52:59.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.775+0000 7fcb95e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb901a5b40 con 0x7fcb90069140 2024-03-14T09:52:59.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.775+0000 7fcb95e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb901a5d00 con 0x7fcb90106ec0 2024-03-14T09:52:59.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.776+0000 7fcb8ea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb90068780 0x7fcb900ffce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:59.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.776+0000 7fcb8ea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb90068780 0x7fcb900ffce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36272/0 (socket says 172.21.15.78:36272) 2024-03-14T09:52:59.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.776+0000 7fcb8ea00700 1 -- 172.21.15.78:0/1798765856 learned_addr learned my addr 172.21.15.78:0/1798765856 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:52:59.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.776+0000 7fcb8f400700 1 --2- 172.21.15.78:0/1798765856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb90106ec0 0x7fcb90100760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:59.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.776+0000 7fcb8ea00700 1 -- 172.21.15.78:0/1798765856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb90106ec0 msgr2=0x7fcb90100760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:52:59.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.776+0000 7fcb8ea00700 1 --2- 172.21.15.78:0/1798765856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb90106ec0 0x7fcb90100760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:59.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.776+0000 7fcb8ea00700 1 -- 172.21.15.78:0/1798765856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb90069140 msgr2=0x7fcb90100220 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-14T09:52:59.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.776+0000 7fcb8ea00700 1 --2- 172.21.15.78:0/1798765856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb90069140 0x7fcb90100220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:52:59.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.776+0000 7fcb8ea00700 1 -- 172.21.15.78:0/1798765856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb800095f0 con 0x7fcb90068780 2024-03-14T09:52:59.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.776+0000 7fcb8ea00700 1 --2- 172.21.15.78:0/1798765856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb90068780 0x7fcb900ffce0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcb80000c00 tx=0x7fcb80005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:59.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.777+0000 7fcb94a00700 1 -- 172.21.15.78:0/1798765856 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8001d070 con 0x7fcb90068780 2024-03-14T09:52:59.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.777+0000 7fcb95e00700 1 -- 172.21.15.78:0/1798765856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb901a5f80 con 0x7fcb90068780 2024-03-14T09:52:59.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.777+0000 7fcb95e00700 1 -- 172.21.15.78:0/1798765856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb901a6390 con 0x7fcb90068780 2024-03-14T09:52:59.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.777+0000 7fcb94a00700 1 -- 172.21.15.78:0/1798765856 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb80022470 con 0x7fcb90068780 2024-03-14T09:52:59.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.777+0000 7fcb94a00700 1 -- 172.21.15.78:0/1798765856 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb80017650 con 0x7fcb90068780 2024-03-14T09:52:59.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.778+0000 7fcb95e00700 1 -- 172.21.15.78:0/1798765856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb700052f0 con 0x7fcb90068780 2024-03-14T09:52:59.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.780+0000 7fcb94a00700 1 -- 172.21.15.78:0/1798765856 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fcb8000fb00 con 0x7fcb90068780 2024-03-14T09:52:59.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.782+0000 7fcb94a00700 1 --2- 172.21.15.78:0/1798765856 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcb740a0ba0 0x7fcb740a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:52:59.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.782+0000 7fcb8e000700 1 --2- 172.21.15.78:0/1798765856 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcb740a0ba0 0x7fcb740a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:52:59.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.783+0000 7fcb94a00700 1 -- 172.21.15.78:0/1798765856 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcb800cd840 con 0x7fcb90068780 2024-03-14T09:52:59.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.783+0000 7fcb8e000700 1 --2- 172.21.15.78:0/1798765856 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcb740a0ba0 0x7fcb740a3060 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcb90101410 tx=0x7fcb7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:52:59.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.787+0000 7fcb94a00700 1 -- 172.21.15.78:0/1798765856 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcb80096f50 con 0x7fcb90068780 2024-03-14T09:53:00.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:52:59.999+0000 7fcb95e00700 1 -- 172.21.15.78:0/1798765856 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fcb70005160 con 0x7fcb90068780 2024-03-14T09:53:00.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:00.000+0000 7fcb94a00700 1 -- 172.21.15.78:0/1798765856 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fcb80027070 con 0x7fcb90068780 2024-03-14T09:53:00.001 INFO:teuthology.orchestra.run.smithi078.stdout:317827579925 2024-03-14T09:53:00.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:00.002+0000 7fcb95e00700 1 -- 172.21.15.78:0/1798765856 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcb740a0ba0 msgr2=0x7fcb740a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:00.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:00.002+0000 7fcb95e00700 1 --2- 172.21.15.78:0/1798765856 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcb740a0ba0 0x7fcb740a3060 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcb90101410 tx=0x7fcb7c009260 comp rx=0 tx=0).stop 2024-03-14T09:53:00.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:00.002+0000 7fcb95e00700 1 -- 172.21.15.78:0/1798765856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb90068780 msgr2=0x7fcb900ffce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:00.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:00.002+0000 7fcb95e00700 1 --2- 172.21.15.78:0/1798765856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb90068780 0x7fcb900ffce0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcb80000c00 tx=0x7fcb80005740 comp rx=0 tx=0).stop 2024-03-14T09:53:00.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:00.002+0000 7fcb95e00700 1 -- 172.21.15.78:0/1798765856 shutdown_connections 2024-03-14T09:53:00.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:00.002+0000 7fcb95e00700 1 --2- 172.21.15.78:0/1798765856 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcb90106ec0 0x7fcb90100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:00.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:00.002+0000 7fcb95e00700 1 --2- 172.21.15.78:0/1798765856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb90068780 0x7fcb900ffce0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:00.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:00.002+0000 7fcb95e00700 1 --2- 172.21.15.78:0/1798765856 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fcb740a0ba0 0x7fcb740a3060 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:00.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:00.002+0000 7fcb95e00700 1 --2- 172.21.15.78:0/1798765856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcb90069140 0x7fcb90100220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:00.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:00.002+0000 7fcb95e00700 1 -- 172.21.15.78:0/1798765856 >> 172.21.15.78:0/1798765856 conn(0x7fcb90076db0 msgr2=0x7fcb900feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:00.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:00.003+0000 7fcb95e00700 1 -- 172.21.15.78:0/1798765856 shutdown_connections 2024-03-14T09:53:00.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:00.003+0000 7fcb95e00700 1 -- 172.21.15.78:0/1798765856 wait complete. 2024-03-14T09:53:00.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:00 smithi078 ceph-mon[38143]: pgmap v264: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s; 48 B/s, 0 objects/s recovering 2024-03-14T09:53:00.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:00 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1798765856' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-14T09:53:00.335 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:53:00.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:00 smithi152 ceph-mon[48279]: pgmap v264: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s; 48 B/s, 0 objects/s recovering 2024-03-14T09:53:00.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:00 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1798765856' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-14T09:53:00.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:00 smithi132 ceph-mon[47959]: pgmap v264: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s; 48 B/s, 0 objects/s recovering 2024-03-14T09:53:00.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:00 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1798765856' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-14T09:53:01.062 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579923 got 317827579925 for osd.11 2024-03-14T09:53:01.165 DEBUG:teuthology.parallel:result is None 2024-03-14T09:53:01.176 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:53:01.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.500+0000 7f9c48e00700 1 -- 172.21.15.78:0/2426535996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c44073ac0 msgr2=0x7f9c4410d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:01.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.500+0000 7f9c48e00700 1 --2- 172.21.15.78:0/2426535996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c44073ac0 0x7f9c4410d090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009990 tx=0x7f9c30009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:01.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.500+0000 7f9c48e00700 1 -- 172.21.15.78:0/2426535996 shutdown_connections 2024-03-14T09:53:01.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.500+0000 7f9c48e00700 1 --2- 172.21.15.78:0/2426535996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c4410d5d0 0x7f9c4410fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:01.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.500+0000 7f9c48e00700 1 --2- 172.21.15.78:0/2426535996 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c440731a0 0x7f9c44073580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:01.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.500+0000 7f9c48e00700 1 --2- 172.21.15.78:0/2426535996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c44073ac0 0x7f9c4410d090 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:01.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.500+0000 7f9c48e00700 1 -- 172.21.15.78:0/2426535996 >> 172.21.15.78:0/2426535996 conn(0x7f9c440fc800 msgr2=0x7f9c440fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:01.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.500+0000 7f9c48e00700 1 -- 172.21.15.78:0/2426535996 shutdown_connections 2024-03-14T09:53:01.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.500+0000 7f9c48e00700 1 -- 172.21.15.78:0/2426535996 wait complete. 2024-03-14T09:53:01.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.501+0000 7f9c48e00700 1 Processor -- start 2024-03-14T09:53:01.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.501+0000 7f9c48e00700 1 -- start start 2024-03-14T09:53:01.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.502+0000 7f9c48e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c440731a0 0x7f9c4419f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:01.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.502+0000 7f9c48e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c44073ac0 0x7f9c4419fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:01.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.502+0000 7f9c48e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c4410d5d0 0x7f9c441996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:01.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.502+0000 7f9c48e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c44199d10 con 0x7f9c44073ac0 2024-03-14T09:53:01.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.502+0000 7f9c48e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c44199e80 con 0x7f9c440731a0 2024-03-14T09:53:01.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.502+0000 7f9c48e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4419a040 con 0x7f9c4410d5d0 2024-03-14T09:53:01.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.502+0000 7f9c41400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c44073ac0 0x7f9c4419fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:01.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.502+0000 7f9c41e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c440731a0 0x7f9c4419f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:01.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.502+0000 7f9c41400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c44073ac0 0x7f9c4419fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36286/0 (socket says 172.21.15.78:36286) 2024-03-14T09:53:01.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.502+0000 7f9c41e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c440731a0 0x7f9c4419f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48080/0 (socket says 172.21.15.78:48080) 2024-03-14T09:53:01.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.502+0000 7f9c41400700 1 -- 172.21.15.78:0/2879208308 learned_addr learned my addr 172.21.15.78:0/2879208308 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:01.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.502+0000 7f9c42800700 1 --2- 172.21.15.78:0/2879208308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c4410d5d0 0x7f9c441996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:01.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.503+0000 7f9c41e00700 1 -- 172.21.15.78:0/2879208308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c4410d5d0 msgr2=0x7f9c441996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:01.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.503+0000 7f9c41e00700 1 --2- 172.21.15.78:0/2879208308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c4410d5d0 0x7f9c441996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:01.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.503+0000 7f9c41e00700 1 -- 172.21.15.78:0/2879208308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c44073ac0 msgr2=0x7f9c4419fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:01.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.503+0000 7f9c41e00700 1 --2- 172.21.15.78:0/2879208308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c44073ac0 0x7f9c4419fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:01.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.503+0000 7f9c41e00700 1 -- 172.21.15.78:0/2879208308 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c300095f0 con 0x7f9c440731a0 2024-03-14T09:53:01.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.503+0000 7f9c41e00700 1 --2- 172.21.15.78:0/2879208308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c440731a0 0x7f9c4419f570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400e9f0 tx=0x7f9c3400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:01.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.503+0000 7f9c3aa00700 1 -- 172.21.15.78:0/2879208308 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3400c9c0 con 0x7f9c440731a0 2024-03-14T09:53:01.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.503+0000 7f9c48e00700 1 -- 172.21.15.78:0/2879208308 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4419a320 con 0x7f9c440731a0 2024-03-14T09:53:01.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.503+0000 7f9c3aa00700 1 -- 172.21.15.78:0/2879208308 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9c34010430 con 0x7f9c440731a0 2024-03-14T09:53:01.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.503+0000 7f9c3aa00700 1 -- 172.21.15.78:0/2879208308 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c34018680 con 0x7f9c440731a0 2024-03-14T09:53:01.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.503+0000 7f9c48e00700 1 -- 172.21.15.78:0/2879208308 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c441a6560 con 0x7f9c440731a0 2024-03-14T09:53:01.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.505+0000 7f9c48e00700 1 -- 172.21.15.78:0/2879208308 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c441087e0 con 0x7f9c440731a0 2024-03-14T09:53:01.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.506+0000 7f9c3aa00700 1 -- 172.21.15.78:0/2879208308 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f9c34010810 con 0x7f9c440731a0 2024-03-14T09:53:01.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.508+0000 7f9c3aa00700 1 --2- 172.21.15.78:0/2879208308 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c280a0b20 0x7f9c280a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:01.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.508+0000 7f9c3aa00700 1 -- 172.21.15.78:0/2879208308 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9c34014070 con 0x7f9c440731a0 2024-03-14T09:53:01.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.508+0000 7f9c41400700 1 --2- 172.21.15.78:0/2879208308 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c280a0b20 0x7f9c280a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:01.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.508+0000 7f9c41400700 1 --2- 172.21.15.78:0/2879208308 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c280a0b20 0x7f9c280a2fe0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9c4419adc0 tx=0x7f9c3000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:01.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.512+0000 7f9c3aa00700 1 -- 172.21.15.78:0/2879208308 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9c34052960 con 0x7f9c440731a0 2024-03-14T09:53:01.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.773+0000 7f9c48e00700 1 -- 172.21.15.78:0/2879208308 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9c4419ab70 con 0x7f9c440731a0 2024-03-14T09:53:01.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.773+0000 7f9c3aa00700 1 -- 172.21.15.78:0/2879208308 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9c34095b60 con 0x7f9c440731a0 2024-03-14T09:53:01.774 INFO:teuthology.orchestra.run.smithi078.stdout:55834574908 2024-03-14T09:53:01.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.776+0000 7f9c48e00700 1 -- 172.21.15.78:0/2879208308 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c280a0b20 msgr2=0x7f9c280a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:01.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.776+0000 7f9c48e00700 1 --2- 172.21.15.78:0/2879208308 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c280a0b20 0x7f9c280a2fe0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9c4419adc0 tx=0x7f9c3000b260 comp rx=0 tx=0).stop 2024-03-14T09:53:01.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.776+0000 7f9c48e00700 1 -- 172.21.15.78:0/2879208308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c440731a0 msgr2=0x7f9c4419f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:01.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.776+0000 7f9c48e00700 1 --2- 172.21.15.78:0/2879208308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c440731a0 0x7f9c4419f570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400e9f0 tx=0x7f9c3400ed00 comp rx=0 tx=0).stop 2024-03-14T09:53:01.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.776+0000 7f9c48e00700 1 -- 172.21.15.78:0/2879208308 shutdown_connections 2024-03-14T09:53:01.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.776+0000 7f9c48e00700 1 --2- 172.21.15.78:0/2879208308 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9c280a0b20 0x7f9c280a2fe0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:01.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.776+0000 7f9c48e00700 1 --2- 172.21.15.78:0/2879208308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c440731a0 0x7f9c4419f570 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:01.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.776+0000 7f9c48e00700 1 --2- 172.21.15.78:0/2879208308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9c4410d5d0 0x7f9c441996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:01.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.776+0000 7f9c48e00700 1 --2- 172.21.15.78:0/2879208308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c44073ac0 0x7f9c4419fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:01.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.776+0000 7f9c48e00700 1 -- 172.21.15.78:0/2879208308 >> 172.21.15.78:0/2879208308 conn(0x7f9c440fc800 msgr2=0x7f9c440fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:01.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.776+0000 7f9c48e00700 1 -- 172.21.15.78:0/2879208308 shutdown_connections 2024-03-14T09:53:01.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:01.776+0000 7f9c48e00700 1 -- 172.21.15.78:0/2879208308 wait complete. 2024-03-14T09:53:01.931 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:53:02.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:02 smithi078 ceph-mon[38143]: pgmap v265: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-14T09:53:02.395 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:02 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2879208308' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-14T09:53:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:02 smithi152 ceph-mon[48279]: pgmap v265: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-14T09:53:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:02 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2879208308' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-14T09:53:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.501+0000 7f010bc00700 1 -- 172.21.15.78:0/1276577930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01041031d0 msgr2=0x7f01041035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.501+0000 7f010bc00700 1 --2- 172.21.15.78:0/1276577930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01041031d0 0x7f01041035b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f00f8009990 tx=0x7f00f8009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.501+0000 7f010bc00700 1 -- 172.21.15.78:0/1276577930 shutdown_connections 2024-03-14T09:53:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.501+0000 7f010bc00700 1 --2- 172.21.15.78:0/1276577930 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f01041082b0 0x7f010410fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.501+0000 7f010bc00700 1 --2- 172.21.15.78:0/1276577930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01041031d0 0x7f01041035b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.501+0000 7f010bc00700 1 --2- 172.21.15.78:0/1276577930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0104103b80 0x7f0104107bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.501+0000 7f010bc00700 1 -- 172.21.15.78:0/1276577930 >> 172.21.15.78:0/1276577930 conn(0x7f01040fea40 msgr2=0x7f0104100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.502+0000 7f010bc00700 1 -- 172.21.15.78:0/1276577930 shutdown_connections 2024-03-14T09:53:02.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.502+0000 7f010bc00700 1 -- 172.21.15.78:0/1276577930 wait complete. 2024-03-14T09:53:02.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.504+0000 7f010bc00700 1 Processor -- start 2024-03-14T09:53:02.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.504+0000 7f010bc00700 1 -- start start 2024-03-14T09:53:02.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.505+0000 7f010bc00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f01041031d0 0x7f010419f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:02.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.505+0000 7f010bc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0104103b80 0x7f010419fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:02.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.505+0000 7f010bc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01041082b0 0x7f01041996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:02.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.505+0000 7f0109200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f01041031d0 0x7f010419f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:02.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.506+0000 7f0109200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f01041031d0 0x7f010419f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:44512/0 (socket says 172.21.15.78:44512) 2024-03-14T09:53:02.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.506+0000 7f010bc00700 1 -- 172.21.15.78:0/1654934095 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0104199d30 con 0x7f0104103b80 2024-03-14T09:53:02.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.506+0000 7f0109200700 1 -- 172.21.15.78:0/1654934095 learned_addr learned my addr 172.21.15.78:0/1654934095 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:02.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.506+0000 7f010bc00700 1 -- 172.21.15.78:0/1654934095 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0104199ea0 con 0x7f01041082b0 2024-03-14T09:53:02.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.506+0000 7f010bc00700 1 -- 172.21.15.78:0/1654934095 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010419a060 con 0x7f01041031d0 2024-03-14T09:53:02.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.506+0000 7f0109c00700 1 --2- 172.21.15.78:0/1654934095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01041082b0 0x7f01041996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:02.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.506+0000 7f0103e00700 1 --2- 172.21.15.78:0/1654934095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0104103b80 0x7f010419fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:02.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.506+0000 7f0109200700 1 -- 172.21.15.78:0/1654934095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01041082b0 msgr2=0x7f01041996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.506+0000 7f0109200700 1 --2- 172.21.15.78:0/1654934095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01041082b0 0x7f01041996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.506+0000 7f0109200700 1 -- 172.21.15.78:0/1654934095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0104103b80 msgr2=0x7f010419fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.506+0000 7f0109200700 1 --2- 172.21.15.78:0/1654934095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0104103b80 0x7f010419fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.506+0000 7f0109200700 1 -- 172.21.15.78:0/1654934095 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f80095f0 con 0x7f01041031d0 2024-03-14T09:53:02.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.506+0000 7f0103e00700 1 --2- 172.21.15.78:0/1654934095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0104103b80 0x7f010419fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:53:02.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.507+0000 7f0109200700 1 --2- 172.21.15.78:0/1654934095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f01041031d0 0x7f010419f590 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f00f8005910 tx=0x7f00f8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:02.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.507+0000 7f0101600700 1 -- 172.21.15.78:0/1654934095 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f801d070 con 0x7f01041031d0 2024-03-14T09:53:02.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.508+0000 7f0101600700 1 -- 172.21.15.78:0/1654934095 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00f8022470 con 0x7f01041031d0 2024-03-14T09:53:02.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.508+0000 7f0101600700 1 -- 172.21.15.78:0/1654934095 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f8017650 con 0x7f01041031d0 2024-03-14T09:53:02.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.508+0000 7f010bc00700 1 -- 172.21.15.78:0/1654934095 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f010419a2e0 con 0x7f01041031d0 2024-03-14T09:53:02.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.508+0000 7f010bc00700 1 -- 172.21.15.78:0/1654934095 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01041a6580 con 0x7f01041031d0 2024-03-14T09:53:02.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.509+0000 7f010bc00700 1 -- 172.21.15.78:0/1654934095 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f010410b570 con 0x7f01041031d0 2024-03-14T09:53:02.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.511+0000 7f0101600700 1 -- 172.21.15.78:0/1654934095 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f00f80225e0 con 0x7f01041031d0 2024-03-14T09:53:02.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.513+0000 7f0101600700 1 --2- 172.21.15.78:0/1654934095 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f00ec0a0ba0 0x7f00ec0a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:02.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.514+0000 7f0101600700 1 -- 172.21.15.78:0/1654934095 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f00f80cdef0 con 0x7f01041031d0 2024-03-14T09:53:02.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.514+0000 7f0103e00700 1 --2- 172.21.15.78:0/1654934095 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f00ec0a0ba0 0x7f00ec0a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:02.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.515+0000 7f0101600700 1 -- 172.21.15.78:0/1654934095 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f00f8097480 con 0x7f01041031d0 2024-03-14T09:53:02.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.515+0000 7f0103e00700 1 --2- 172.21.15.78:0/1654934095 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f00ec0a0ba0 0x7f00ec0a3060 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f010419ade0 tx=0x7f00f00091e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:02.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:02 smithi132 ceph-mon[47959]: pgmap v265: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-14T09:53:02.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:02 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2879208308' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-14T09:53:02.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.635+0000 7fe74ae00700 1 -- 172.21.15.78:0/4028719311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe744106e30 msgr2=0x7fe744107210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.635+0000 7fe74ae00700 1 --2- 172.21.15.78:0/4028719311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe744106e30 0x7fe744107210 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe734009990 tx=0x7fe734009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.636+0000 7fe74ae00700 1 -- 172.21.15.78:0/4028719311 shutdown_connections 2024-03-14T09:53:02.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.636+0000 7fe74ae00700 1 --2- 172.21.15.78:0/4028719311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe744106e30 0x7fe744107210 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.636+0000 7fe74ae00700 1 --2- 172.21.15.78:0/4028719311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe744101820 0x7fe74410dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.636+0000 7fe74ae00700 1 --2- 172.21.15.78:0/4028719311 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe744100e60 0x7fe7441012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.636+0000 7fe74ae00700 1 -- 172.21.15.78:0/4028719311 >> 172.21.15.78:0/4028719311 conn(0x7fe744077ed0 msgr2=0x7fe7440782e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:02.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.636+0000 7fe74ae00700 1 -- 172.21.15.78:0/4028719311 shutdown_connections 2024-03-14T09:53:02.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.636+0000 7fe74ae00700 1 -- 172.21.15.78:0/4028719311 wait complete. 2024-03-14T09:53:02.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.637+0000 7fe74ae00700 1 Processor -- start 2024-03-14T09:53:02.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.637+0000 7fe74ae00700 1 -- start start 2024-03-14T09:53:02.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.637+0000 7fe74ae00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe744100e60 0x7fe74419f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:02.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.637+0000 7fe74ae00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe744101820 0x7fe74419fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:02.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.637+0000 7fe74ae00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe744106e30 0x7fe744199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:02.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.637+0000 7fe74ae00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe744199c90 con 0x7fe744106e30 2024-03-14T09:53:02.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.637+0000 7fe74ae00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe744199e00 con 0x7fe744101820 2024-03-14T09:53:02.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.637+0000 7fe74ae00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe744199fc0 con 0x7fe744100e60 2024-03-14T09:53:02.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.638+0000 7fe743400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe744101820 0x7fe74419fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:02.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.638+0000 7fe743e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe744100e60 0x7fe74419f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:02.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.638+0000 7fe743400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe744101820 0x7fe74419fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48122/0 (socket says 172.21.15.78:48122) 2024-03-14T09:53:02.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.638+0000 7fe743400700 1 -- 172.21.15.78:0/3768170888 learned_addr learned my addr 172.21.15.78:0/3768170888 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:02.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.638+0000 7fe748e00700 1 --2- 172.21.15.78:0/3768170888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe744106e30 0x7fe744199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:02.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.638+0000 7fe743e00700 1 -- 172.21.15.78:0/3768170888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe744101820 msgr2=0x7fe74419fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.638+0000 7fe743e00700 1 --2- 172.21.15.78:0/3768170888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe744101820 0x7fe74419fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.638+0000 7fe743e00700 1 -- 172.21.15.78:0/3768170888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe744106e30 msgr2=0x7fe744199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.638+0000 7fe743e00700 1 --2- 172.21.15.78:0/3768170888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe744106e30 0x7fe744199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.638+0000 7fe743e00700 1 -- 172.21.15.78:0/3768170888 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7340095f0 con 0x7fe744100e60 2024-03-14T09:53:02.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.638+0000 7fe743400700 1 --2- 172.21.15.78:0/3768170888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe744101820 0x7fe74419fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:02.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.639+0000 7fe743e00700 1 --2- 172.21.15.78:0/3768170888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe744100e60 0x7fe74419f4f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe734006010 tx=0x7fe734005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.639+0000 7fe740c00700 1 -- 172.21.15.78:0/3768170888 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73401d070 con 0x7fe744100e60 2024-03-14T09:53:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.639+0000 7fe740c00700 1 -- 172.21.15.78:0/3768170888 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe734022470 con 0x7fe744100e60 2024-03-14T09:53:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.639+0000 7fe74ae00700 1 -- 172.21.15.78:0/3768170888 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe74419a240 con 0x7fe744100e60 2024-03-14T09:53:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.639+0000 7fe740c00700 1 -- 172.21.15.78:0/3768170888 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734017650 con 0x7fe744100e60 2024-03-14T09:53:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.639+0000 7fe74ae00700 1 -- 172.21.15.78:0/3768170888 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7441a64e0 con 0x7fe744100e60 2024-03-14T09:53:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.640+0000 7fe74ae00700 1 -- 172.21.15.78:0/3768170888 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe744104110 con 0x7fe744100e60 2024-03-14T09:53:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.642+0000 7fe740c00700 1 -- 172.21.15.78:0/3768170888 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fe734022890 con 0x7fe744100e60 2024-03-14T09:53:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.643+0000 7fe740c00700 1 --2- 172.21.15.78:0/3768170888 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe7280a0b20 0x7fe7280a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.643+0000 7fe740c00700 1 -- 172.21.15.78:0/3768170888 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe7340cd7e0 con 0x7fe744100e60 2024-03-14T09:53:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.643+0000 7fe743400700 1 --2- 172.21.15.78:0/3768170888 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe7280a0b20 0x7fe7280a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.644+0000 7fe743400700 1 --2- 172.21.15.78:0/3768170888 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe7280a0b20 0x7fe7280a2fe0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe74419ad40 tx=0x7fe730009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.647+0000 7fe740c00700 1 -- 172.21.15.78:0/3768170888 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe734096d70 con 0x7fe744100e60 2024-03-14T09:53:02.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.752+0000 7f010bc00700 1 -- 172.21.15.78:0/1654934095 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f01041046b0 con 0x7f01041031d0 2024-03-14T09:53:02.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.753+0000 7f0101600700 1 -- 172.21.15.78:0/1654934095 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f00f8027020 con 0x7f01041031d0 2024-03-14T09:53:02.754 INFO:teuthology.orchestra.run.smithi078.stdout:266287972381 2024-03-14T09:53:02.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.756+0000 7f010bc00700 1 -- 172.21.15.78:0/1654934095 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f00ec0a0ba0 msgr2=0x7f00ec0a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.756+0000 7f010bc00700 1 --2- 172.21.15.78:0/1654934095 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f00ec0a0ba0 0x7f00ec0a3060 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f010419ade0 tx=0x7f00f00091e0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.756+0000 7f010bc00700 1 -- 172.21.15.78:0/1654934095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f01041031d0 msgr2=0x7f010419f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.756+0000 7f010bc00700 1 --2- 172.21.15.78:0/1654934095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f01041031d0 0x7f010419f590 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f00f8005910 tx=0x7f00f8004730 comp rx=0 tx=0).stop 2024-03-14T09:53:02.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.756+0000 7f010bc00700 1 -- 172.21.15.78:0/1654934095 shutdown_connections 2024-03-14T09:53:02.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.756+0000 7f010bc00700 1 --2- 172.21.15.78:0/1654934095 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f00ec0a0ba0 0x7f00ec0a3060 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.756+0000 7f010bc00700 1 --2- 172.21.15.78:0/1654934095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01041082b0 0x7f01041996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.756+0000 7f010bc00700 1 --2- 172.21.15.78:0/1654934095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f01041031d0 0x7f010419f590 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.756+0000 7f010bc00700 1 --2- 172.21.15.78:0/1654934095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0104103b80 0x7f010419fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.756+0000 7f010bc00700 1 -- 172.21.15.78:0/1654934095 >> 172.21.15.78:0/1654934095 conn(0x7f01040fea40 msgr2=0x7f01040ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:02.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.756+0000 7f010bc00700 1 -- 172.21.15.78:0/1654934095 shutdown_connections 2024-03-14T09:53:02.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.757+0000 7f010bc00700 1 -- 172.21.15.78:0/1654934095 wait complete. 2024-03-14T09:53:02.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.764+0000 7fdfba000700 1 -- 172.21.15.78:0/1798057223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb4068780 msgr2=0x7fdfb4068c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.764+0000 7fdfba000700 1 --2- 172.21.15.78:0/1798057223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb4068780 0x7fdfb4068c00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0009990 tx=0x7fdfa0009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.764+0000 7fdfba000700 1 -- 172.21.15.78:0/1798057223 shutdown_connections 2024-03-14T09:53:02.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.764+0000 7fdfba000700 1 --2- 172.21.15.78:0/1798057223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfb4069140 0x7fdfb410db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.764+0000 7fdfba000700 1 --2- 172.21.15.78:0/1798057223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfb4106ec0 0x7fdfb41072a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.764+0000 7fdfba000700 1 --2- 172.21.15.78:0/1798057223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb4068780 0x7fdfb4068c00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.764+0000 7fdfba000700 1 -- 172.21.15.78:0/1798057223 >> 172.21.15.78:0/1798057223 conn(0x7fdfb4076db0 msgr2=0x7fdfb40771c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:02.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.764+0000 7fdfba000700 1 -- 172.21.15.78:0/1798057223 shutdown_connections 2024-03-14T09:53:02.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.765+0000 7fdfba000700 1 -- 172.21.15.78:0/1798057223 wait complete. 2024-03-14T09:53:02.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.766+0000 7fdfba000700 1 Processor -- start 2024-03-14T09:53:02.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.766+0000 7fdfba000700 1 -- start start 2024-03-14T09:53:02.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.766+0000 7fdfba000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfb4068780 0x7fdfb4197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:02.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.767+0000 7fdfba000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfb4069140 0x7fdfb4197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:02.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.767+0000 7fdfba000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb4106ec0 0x7fdfb419be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.767+0000 7fdfba000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb419c410 con 0x7fdfb4068780 2024-03-14T09:53:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.767+0000 7fdfba000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb419c580 con 0x7fdfb4106ec0 2024-03-14T09:53:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.767+0000 7fdfba000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb419c740 con 0x7fdfb4069140 2024-03-14T09:53:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.767+0000 7fdfb3400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfb4068780 0x7fdfb4197510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.767+0000 7fdfb3400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfb4068780 0x7fdfb4197510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36350/0 (socket says 172.21.15.78:36350) 2024-03-14T09:53:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.767+0000 7fdfb3400700 1 -- 172.21.15.78:0/3320583292 learned_addr learned my addr 172.21.15.78:0/3320583292 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.767+0000 7fdfb3e00700 1 --2- 172.21.15.78:0/3320583292 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb4106ec0 0x7fdfb419be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.768+0000 7fdfb3400700 1 -- 172.21.15.78:0/3320583292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfb4069140 msgr2=0x7fdfb4197a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-14T09:53:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.768+0000 7fdfb3400700 1 --2- 172.21.15.78:0/3320583292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfb4069140 0x7fdfb4197a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.768+0000 7fdfb3400700 1 -- 172.21.15.78:0/3320583292 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb4106ec0 msgr2=0x7fdfb419be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.768+0000 7fdfb3400700 1 --2- 172.21.15.78:0/3320583292 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb4106ec0 0x7fdfb419be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.768+0000 7fdfb3400700 1 -- 172.21.15.78:0/3320583292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa00095f0 con 0x7fdfb4068780 2024-03-14T09:53:02.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.768+0000 7fdfb3400700 1 --2- 172.21.15.78:0/3320583292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfb4068780 0x7fdfb4197510 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400d980 tx=0x7fdfa400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:02.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.768+0000 7fdfabe00700 1 -- 172.21.15.78:0/3320583292 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4004220 con 0x7fdfb4068780 2024-03-14T09:53:02.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.768+0000 7fdfabe00700 1 -- 172.21.15.78:0/3320583292 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdfa4004cf0 con 0x7fdfb4068780 2024-03-14T09:53:02.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.768+0000 7fdfabe00700 1 -- 172.21.15.78:0/3320583292 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4005720 con 0x7fdfb4068780 2024-03-14T09:53:02.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.769+0000 7fdfba000700 1 -- 172.21.15.78:0/3320583292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb419ca20 con 0x7fdfb4068780 2024-03-14T09:53:02.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.769+0000 7fdfba000700 1 -- 172.21.15.78:0/3320583292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb40ffde0 con 0x7fdfb4068780 2024-03-14T09:53:02.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.771+0000 7fdfba000700 1 -- 172.21.15.78:0/3320583292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb4109250 con 0x7fdfb4068780 2024-03-14T09:53:02.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.773+0000 7fdfabe00700 1 -- 172.21.15.78:0/3320583292 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fdfa4004410 con 0x7fdfb4068780 2024-03-14T09:53:02.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.774+0000 7fdfabe00700 1 --2- 172.21.15.78:0/3320583292 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdf980a0b20 0x7fdf980a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:02.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.774+0000 7fdfabe00700 1 -- 172.21.15.78:0/3320583292 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdfa40cc160 con 0x7fdfb4068780 2024-03-14T09:53:02.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.775+0000 7fdfb2a00700 1 --2- 172.21.15.78:0/3320583292 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdf980a0b20 0x7fdf980a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:02.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.776+0000 7fdfb2a00700 1 --2- 172.21.15.78:0/3320583292 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdf980a0b20 0x7fdf980a2fe0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00098e0 tx=0x7fdfa0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:02.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.784+0000 7fdfabe00700 1 -- 172.21.15.78:0/3320583292 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fdfa40956c0 con 0x7fdfb4068780 2024-03-14T09:53:02.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.925+0000 7fe74ae00700 1 -- 172.21.15.78:0/3768170888 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe74404eab0 con 0x7fe744100e60 2024-03-14T09:53:02.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.926+0000 7fe740c00700 1 -- 172.21.15.78:0/3768170888 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe73409a390 con 0x7fe744100e60 2024-03-14T09:53:02.927 INFO:teuthology.orchestra.run.smithi078.stdout:103079215156 2024-03-14T09:53:02.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.928+0000 7fe74ae00700 1 -- 172.21.15.78:0/3768170888 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe7280a0b20 msgr2=0x7fe7280a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.928+0000 7fe74ae00700 1 --2- 172.21.15.78:0/3768170888 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe7280a0b20 0x7fe7280a2fe0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe74419ad40 tx=0x7fe730009260 comp rx=0 tx=0).stop 2024-03-14T09:53:02.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.928+0000 7fe74ae00700 1 -- 172.21.15.78:0/3768170888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe744100e60 msgr2=0x7fe74419f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.928+0000 7fe74ae00700 1 --2- 172.21.15.78:0/3768170888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe744100e60 0x7fe74419f4f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe734006010 tx=0x7fe734005740 comp rx=0 tx=0).stop 2024-03-14T09:53:02.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.929+0000 7fe74ae00700 1 -- 172.21.15.78:0/3768170888 shutdown_connections 2024-03-14T09:53:02.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.929+0000 7fe74ae00700 1 --2- 172.21.15.78:0/3768170888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe744106e30 0x7fe744199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.929+0000 7fe74ae00700 1 --2- 172.21.15.78:0/3768170888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe744100e60 0x7fe74419f4f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.929+0000 7fe74ae00700 1 --2- 172.21.15.78:0/3768170888 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe7280a0b20 0x7fe7280a2fe0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.929+0000 7fe74ae00700 1 --2- 172.21.15.78:0/3768170888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe744101820 0x7fe74419fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.929+0000 7fe74ae00700 1 -- 172.21.15.78:0/3768170888 >> 172.21.15.78:0/3768170888 conn(0x7fe744077ed0 msgr2=0x7fe74410c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:02.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.929+0000 7fe74ae00700 1 -- 172.21.15.78:0/3768170888 shutdown_connections 2024-03-14T09:53:02.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.929+0000 7fe74ae00700 1 -- 172.21.15.78:0/3768170888 wait complete. 2024-03-14T09:53:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.126+0000 7f5c39a00700 1 -- 172.21.15.78:0/1383651557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c34106ec0 msgr2=0x7f5c341072a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.126+0000 7f5c39a00700 1 --2- 172.21.15.78:0/1383651557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c34106ec0 0x7f5c341072a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5c240099c0 tx=0x7f5c24009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.127+0000 7f5c39a00700 1 -- 172.21.15.78:0/1383651557 shutdown_connections 2024-03-14T09:53:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.127+0000 7f5c39a00700 1 --2- 172.21.15.78:0/1383651557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c341017f0 0x7f5c3410dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.127+0000 7f5c39a00700 1 --2- 172.21.15.78:0/1383651557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c34106ec0 0x7f5c341072a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.127+0000 7f5c39a00700 1 --2- 172.21.15.78:0/1383651557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c34100e30 0x7f5c341012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.127+0000 7f5c39a00700 1 -- 172.21.15.78:0/1383651557 >> 172.21.15.78:0/1383651557 conn(0x7f5c34077fa0 msgr2=0x7f5c340783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.127+0000 7f5c39a00700 1 -- 172.21.15.78:0/1383651557 shutdown_connections 2024-03-14T09:53:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.127+0000 7f5c39a00700 1 -- 172.21.15.78:0/1383651557 wait complete. 2024-03-14T09:53:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.128+0000 7f5c39a00700 1 Processor -- start 2024-03-14T09:53:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.128+0000 7f5c39a00700 1 -- start start 2024-03-14T09:53:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.128+0000 7f5c39a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c34100e30 0x7f5c340730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.128+0000 7f5c39a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c341017f0 0x7f5c340735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.128+0000 7f5c39a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c34106ec0 0x7f5c34073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.128+0000 7f5c39a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c341a5a10 con 0x7f5c341017f0 2024-03-14T09:53:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.128+0000 7f5c39a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c341a5b80 con 0x7f5c34106ec0 2024-03-14T09:53:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.128+0000 7f5c39a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c341a5d40 con 0x7f5c34100e30 2024-03-14T09:53:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.128+0000 7f5c32000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c341017f0 0x7f5c340735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.128+0000 7f5c32000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c341017f0 0x7f5c340735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36296/0 (socket says 172.21.15.78:36296) 2024-03-14T09:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.128+0000 7f5c32000700 1 -- 172.21.15.78:0/3671714281 learned_addr learned my addr 172.21.15.78:0/3671714281 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.128+0000 7f5c32a00700 1 --2- 172.21.15.78:0/3671714281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c34100e30 0x7f5c340730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.129+0000 7f5c33400700 1 --2- 172.21.15.78:0/3671714281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c34106ec0 0x7f5c34073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.129+0000 7f5c32000700 1 -- 172.21.15.78:0/3671714281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c34100e30 msgr2=0x7f5c340730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.129+0000 7f5c32000700 1 --2- 172.21.15.78:0/3671714281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c34100e30 0x7f5c340730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.129+0000 7f5c32000700 1 -- 172.21.15.78:0/3671714281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c34106ec0 msgr2=0x7f5c34073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.129+0000 7f5c32000700 1 --2- 172.21.15.78:0/3671714281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c34106ec0 0x7f5c34073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.129+0000 7f5c32000700 1 -- 172.21.15.78:0/3671714281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c240095f0 con 0x7f5c341017f0 2024-03-14T09:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.129+0000 7f5c32000700 1 --2- 172.21.15.78:0/3671714281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c341017f0 0x7f5c340735f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5c2000e9a0 tx=0x7f5c2000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.129+0000 7f5c2b400700 1 -- 172.21.15.78:0/3671714281 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c20009790 con 0x7f5c341017f0 2024-03-14T09:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.129+0000 7f5c2b400700 1 -- 172.21.15.78:0/3671714281 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5c20010430 con 0x7f5c341017f0 2024-03-14T09:53:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.129+0000 7f5c39a00700 1 -- 172.21.15.78:0/3671714281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c341a5fc0 con 0x7f5c341017f0 2024-03-14T09:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.129+0000 7f5c2b400700 1 -- 172.21.15.78:0/3671714281 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c20018680 con 0x7f5c341017f0 2024-03-14T09:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.130+0000 7f5c39a00700 1 -- 172.21.15.78:0/3671714281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c341a6480 con 0x7f5c341017f0 2024-03-14T09:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.131+0000 7f5c39a00700 1 -- 172.21.15.78:0/3671714281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c341092a0 con 0x7f5c341017f0 2024-03-14T09:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.132+0000 7f5c2b400700 1 -- 172.21.15.78:0/3671714281 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f5c200105a0 con 0x7f5c341017f0 2024-03-14T09:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.134+0000 7f5c2b400700 1 --2- 172.21.15.78:0/3671714281 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5c180a0ba0 0x7f5c180a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.135+0000 7f5c2b400700 1 -- 172.21.15.78:0/3671714281 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5c20014070 con 0x7f5c341017f0 2024-03-14T09:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.135+0000 7f5c32a00700 1 --2- 172.21.15.78:0/3671714281 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5c180a0ba0 0x7f5c180a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.135+0000 7f5c32a00700 1 --2- 172.21.15.78:0/3671714281 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5c180a0ba0 0x7f5c180a3060 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009fd0 tx=0x7f5c24005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.138+0000 7f5c2b400700 1 -- 172.21.15.78:0/3671714281 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5c20052960 con 0x7f5c341017f0 2024-03-14T09:53:02.980 INFO:teuthology.orchestra.run.smithi078.stdout:184683593767 2024-03-14T09:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.368+0000 7f5c39a00700 1 -- 172.21.15.78:0/3671714281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f5c341024c0 con 0x7f5c341017f0 2024-03-14T09:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.369+0000 7f5c2b400700 1 -- 172.21.15.78:0/3671714281 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5c20095b70 con 0x7f5c341017f0 2024-03-14T09:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.371+0000 7f5c39a00700 1 -- 172.21.15.78:0/3671714281 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5c180a0ba0 msgr2=0x7f5c180a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.371+0000 7f5c39a00700 1 --2- 172.21.15.78:0/3671714281 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5c180a0ba0 0x7f5c180a3060 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009fd0 tx=0x7f5c24005fb0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.371+0000 7f5c39a00700 1 -- 172.21.15.78:0/3671714281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c341017f0 msgr2=0x7f5c340735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.371+0000 7f5c39a00700 1 --2- 172.21.15.78:0/3671714281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c341017f0 0x7f5c340735f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5c2000e9a0 tx=0x7f5c2000ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.372+0000 7f5c39a00700 1 -- 172.21.15.78:0/3671714281 shutdown_connections 2024-03-14T09:53:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.372+0000 7f5c39a00700 1 --2- 172.21.15.78:0/3671714281 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5c180a0ba0 0x7f5c180a3060 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.372+0000 7f5c39a00700 1 --2- 172.21.15.78:0/3671714281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c34106ec0 0x7f5c34073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.372+0000 7f5c39a00700 1 --2- 172.21.15.78:0/3671714281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5c34100e30 0x7f5c340730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.372+0000 7f5c39a00700 1 --2- 172.21.15.78:0/3671714281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c341017f0 0x7f5c340735f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.372+0000 7f5c39a00700 1 -- 172.21.15.78:0/3671714281 >> 172.21.15.78:0/3671714281 conn(0x7f5c34077fa0 msgr2=0x7f5c340ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.372+0000 7f5c39a00700 1 -- 172.21.15.78:0/3671714281 shutdown_connections 2024-03-14T09:53:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:02.372+0000 7f5c39a00700 1 -- 172.21.15.78:0/3671714281 wait complete. 2024-03-14T09:53:03.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.049+0000 7fdfba000700 1 -- 172.21.15.78:0/3320583292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fdfb404eab0 con 0x7fdfb4068780 2024-03-14T09:53:03.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.049+0000 7fdfabe00700 1 -- 172.21.15.78:0/3320583292 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdfa4098ce0 con 0x7fdfb4068780 2024-03-14T09:53:03.050 INFO:teuthology.orchestra.run.smithi078.stdout:77309411382 2024-03-14T09:53:03.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.052+0000 7fdfba000700 1 -- 172.21.15.78:0/3320583292 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdf980a0b20 msgr2=0x7fdf980a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:03.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.052+0000 7fdfba000700 1 --2- 172.21.15.78:0/3320583292 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdf980a0b20 0x7fdf980a2fe0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00098e0 tx=0x7fdfa0000bc0 comp rx=0 tx=0).stop 2024-03-14T09:53:03.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.052+0000 7fdfba000700 1 -- 172.21.15.78:0/3320583292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfb4068780 msgr2=0x7fdfb4197510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:03.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.052+0000 7fdfba000700 1 --2- 172.21.15.78:0/3320583292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfb4068780 0x7fdfb4197510 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400d980 tx=0x7fdfa400dc90 comp rx=0 tx=0).stop 2024-03-14T09:53:03.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.052+0000 7fdfba000700 1 -- 172.21.15.78:0/3320583292 shutdown_connections 2024-03-14T09:53:03.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.052+0000 7fdfba000700 1 --2- 172.21.15.78:0/3320583292 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdf980a0b20 0x7fdf980a2fe0 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00098e0 tx=0x7fdfa0000bc0 comp rx=0 tx=0).stop 2024-03-14T09:53:03.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.052+0000 7fdfba000700 1 --2- 172.21.15.78:0/3320583292 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb4106ec0 0x7fdfb419be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:03.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.052+0000 7fdfba000700 1 --2- 172.21.15.78:0/3320583292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdfb4068780 0x7fdfb4197510 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:03.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.052+0000 7fdfba000700 1 --2- 172.21.15.78:0/3320583292 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdfb4069140 0x7fdfb4197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:03.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.052+0000 7fdfba000700 1 -- 172.21.15.78:0/3320583292 >> 172.21.15.78:0/3320583292 conn(0x7fdfb4076db0 msgr2=0x7fdfb40fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:03.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.052+0000 7fdfba000700 1 -- 172.21.15.78:0/3320583292 shutdown_connections 2024-03-14T09:53:03.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.052+0000 7fdfba000700 1 -- 172.21.15.78:0/3320583292 wait complete. 2024-03-14T09:53:03.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:03 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3671714281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-14T09:53:03.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:03 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1654934095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-14T09:53:03.444 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:03 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3768170888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-14T09:53:03.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:03 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3320583292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-14T09:53:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:03 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3671714281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-14T09:53:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:03 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1654934095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-14T09:53:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:03 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3768170888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-14T09:53:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:03 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3320583292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-14T09:53:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:03 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3671714281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-14T09:53:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:03 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1654934095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-14T09:53:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:03 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3768170888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-14T09:53:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:03 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3320583292' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-14T09:53:03.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.753+0000 7f9967800700 1 -- 172.21.15.78:0/374647489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9960100fb0 msgr2=0x7f9960101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:03.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.753+0000 7f9967800700 1 --2- 172.21.15.78:0/374647489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9960100fb0 0x7f9960101390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9954009990 tx=0x7f9954009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:03.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.753+0000 7f9967800700 1 -- 172.21.15.78:0/374647489 shutdown_connections 2024-03-14T09:53:03.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.753+0000 7f9967800700 1 --2- 172.21.15.78:0/374647489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9960100fb0 0x7f9960101390 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:03.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.753+0000 7f9967800700 1 --2- 172.21.15.78:0/374647489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9960106090 0x7f996010dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:03.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.753+0000 7f9967800700 1 --2- 172.21.15.78:0/374647489 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9960101960 0x7f99601059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:03.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.753+0000 7f9967800700 1 -- 172.21.15.78:0/374647489 >> 172.21.15.78:0/374647489 conn(0x7f99600fc800 msgr2=0x7f99600fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:03.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.753+0000 7f9967800700 1 -- 172.21.15.78:0/374647489 shutdown_connections 2024-03-14T09:53:03.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.753+0000 7f9967800700 1 -- 172.21.15.78:0/374647489 wait complete. 2024-03-14T09:53:03.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.754+0000 7f9967800700 1 Processor -- start 2024-03-14T09:53:03.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.754+0000 7f9967800700 1 -- start start 2024-03-14T09:53:03.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.755+0000 7f9967800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9960100fb0 0x7f99600730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:03.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.755+0000 7f9967800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9960101960 0x7f99600735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:03.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.755+0000 7f9967800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9960106090 0x7f9960073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:03.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.755+0000 7f9967800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99601a5a40 con 0x7f9960101960 2024-03-14T09:53:03.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.755+0000 7f9967800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99601a5bb0 con 0x7f9960100fb0 2024-03-14T09:53:03.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.755+0000 7f9967800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99601a5d70 con 0x7f9960106090 2024-03-14T09:53:03.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.755+0000 7f9964e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9960100fb0 0x7f99600730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:03.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.755+0000 7f9964e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9960100fb0 0x7f99600730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48154/0 (socket says 172.21.15.78:48154) 2024-03-14T09:53:03.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.755+0000 7f9964e00700 1 -- 172.21.15.78:0/4036839438 learned_addr learned my addr 172.21.15.78:0/4036839438 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:03.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.755+0000 7f995fe00700 1 --2- 172.21.15.78:0/4036839438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9960101960 0x7f99600735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:03.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.755+0000 7f9965800700 1 --2- 172.21.15.78:0/4036839438 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9960106090 0x7f9960073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:03.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.755+0000 7f9964e00700 1 -- 172.21.15.78:0/4036839438 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9960106090 msgr2=0x7f9960073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:03.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.755+0000 7f9964e00700 1 --2- 172.21.15.78:0/4036839438 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9960106090 0x7f9960073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:03.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.755+0000 7f9964e00700 1 -- 172.21.15.78:0/4036839438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9960101960 msgr2=0x7f99600735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:03.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.755+0000 7f9964e00700 1 --2- 172.21.15.78:0/4036839438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9960101960 0x7f99600735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:03.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.756+0000 7f9964e00700 1 -- 172.21.15.78:0/4036839438 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99540095f0 con 0x7f9960100fb0 2024-03-14T09:53:03.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.756+0000 7f9964e00700 1 --2- 172.21.15.78:0/4036839438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9960100fb0 0x7f99600730b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9954005770 tx=0x7f99540057a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:03.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.756+0000 7f995d600700 1 -- 172.21.15.78:0/4036839438 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995401d070 con 0x7f9960100fb0 2024-03-14T09:53:03.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.756+0000 7f995d600700 1 -- 172.21.15.78:0/4036839438 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9954021410 con 0x7f9960100fb0 2024-03-14T09:53:03.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.756+0000 7f9967800700 1 -- 172.21.15.78:0/4036839438 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99601a5ff0 con 0x7f9960100fb0 2024-03-14T09:53:03.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.756+0000 7f995d600700 1 -- 172.21.15.78:0/4036839438 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954017650 con 0x7f9960100fb0 2024-03-14T09:53:03.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.756+0000 7f9967800700 1 -- 172.21.15.78:0/4036839438 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99601a6400 con 0x7f9960100fb0 2024-03-14T09:53:03.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.757+0000 7f9967800700 1 -- 172.21.15.78:0/4036839438 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9960109350 con 0x7f9960100fb0 2024-03-14T09:53:03.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.759+0000 7f995d600700 1 -- 172.21.15.78:0/4036839438 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f99540217f0 con 0x7f9960100fb0 2024-03-14T09:53:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.760+0000 7f995d600700 1 --2- 172.21.15.78:0/4036839438 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99480a0b20 0x7f99480a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.760+0000 7f995d600700 1 -- 172.21.15.78:0/4036839438 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f99540cd720 con 0x7f9960100fb0 2024-03-14T09:53:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.761+0000 7f995fe00700 1 --2- 172.21.15.78:0/4036839438 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99480a0b20 0x7f99480a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:03.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.761+0000 7f995fe00700 1 --2- 172.21.15.78:0/4036839438 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99480a0b20 0x7f99480a2fe0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9960074790 tx=0x7f994c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:03.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.764+0000 7f995d600700 1 -- 172.21.15.78:0/4036839438 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9954096cb0 con 0x7f9960100fb0 2024-03-14T09:53:03.933 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593765 got 184683593767 for osd.6 2024-03-14T09:53:03.933 DEBUG:teuthology.parallel:result is None 2024-03-14T09:53:03.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.986+0000 7f9967800700 1 -- 172.21.15.78:0/4036839438 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f9960074540 con 0x7f9960100fb0 2024-03-14T09:53:03.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.987+0000 7f995d600700 1 -- 172.21.15.78:0/4036839438 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f995409a2d0 con 0x7f9960100fb0 2024-03-14T09:53:03.988 INFO:teuthology.orchestra.run.smithi078.stdout:154618822701 2024-03-14T09:53:03.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.989+0000 7f9967800700 1 -- 172.21.15.78:0/4036839438 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99480a0b20 msgr2=0x7f99480a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:03.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.989+0000 7f9967800700 1 --2- 172.21.15.78:0/4036839438 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99480a0b20 0x7f99480a2fe0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9960074790 tx=0x7f994c009260 comp rx=0 tx=0).stop 2024-03-14T09:53:03.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.989+0000 7f9967800700 1 -- 172.21.15.78:0/4036839438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9960100fb0 msgr2=0x7f99600730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:03.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.989+0000 7f9967800700 1 --2- 172.21.15.78:0/4036839438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9960100fb0 0x7f99600730b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9954005770 tx=0x7f99540057a0 comp rx=0 tx=0).stop 2024-03-14T09:53:03.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.990+0000 7f9967800700 1 -- 172.21.15.78:0/4036839438 shutdown_connections 2024-03-14T09:53:03.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.990+0000 7f9967800700 1 --2- 172.21.15.78:0/4036839438 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f99480a0b20 0x7f99480a2fe0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:03.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.990+0000 7f9967800700 1 --2- 172.21.15.78:0/4036839438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9960100fb0 0x7f99600730b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:03.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.990+0000 7f9967800700 1 --2- 172.21.15.78:0/4036839438 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9960106090 0x7f9960073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.990+0000 7f9967800700 1 --2- 172.21.15.78:0/4036839438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9960101960 0x7f99600735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.990+0000 7f9967800700 1 -- 172.21.15.78:0/4036839438 >> 172.21.15.78:0/4036839438 conn(0x7f99600fc800 msgr2=0x7f99600fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.990+0000 7f9967800700 1 -- 172.21.15.78:0/4036839438 shutdown_connections 2024-03-14T09:53:03.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:03.990+0000 7f9967800700 1 -- 172.21.15.78:0/4036839438 wait complete. 2024-03-14T09:53:04.374 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:04 smithi078 ceph-mon[38143]: pgmap v266: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-14T09:53:04.374 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:04 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:53:04.374 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:04 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/4036839438' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-14T09:53:04.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.420+0000 7f7ac1000700 1 -- 172.21.15.78:0/3771000142 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7abc101960 msgr2=0x7f7abc1059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:04.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.420+0000 7f7ac1000700 1 --2- 172.21.15.78:0/3771000142 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7abc101960 0x7f7abc1059b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009990 tx=0x7f7aa8009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.420+0000 7f7ac1000700 1 -- 172.21.15.78:0/3771000142 shutdown_connections 2024-03-14T09:53:04.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.420+0000 7f7ac1000700 1 --2- 172.21.15.78:0/3771000142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7abc106090 0x7f7abc10dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.420+0000 7f7ac1000700 1 --2- 172.21.15.78:0/3771000142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc100fb0 0x7f7abc101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.420+0000 7f7ac1000700 1 --2- 172.21.15.78:0/3771000142 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7abc101960 0x7f7abc1059b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.420+0000 7f7ac1000700 1 -- 172.21.15.78:0/3771000142 >> 172.21.15.78:0/3771000142 conn(0x7f7abc0fc800 msgr2=0x7f7abc0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:04.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.420+0000 7f7ac1000700 1 -- 172.21.15.78:0/3771000142 shutdown_connections 2024-03-14T09:53:04.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.420+0000 7f7ac1000700 1 -- 172.21.15.78:0/3771000142 wait complete. 2024-03-14T09:53:04.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.421+0000 7f7ac1000700 1 Processor -- start 2024-03-14T09:53:04.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.421+0000 7f7ac1000700 1 -- start start 2024-03-14T09:53:04.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.421+0000 7f7ac1000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7abc100fb0 0x7f7abc19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:04.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.421+0000 7f7ac1000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7abc101960 0x7f7abc19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.421+0000 7f7ac1000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc106090 0x7f7abc199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.421+0000 7f7ac1000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc199cc0 con 0x7f7abc106090 2024-03-14T09:53:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.421+0000 7f7ac1000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc199e30 con 0x7f7abc100fb0 2024-03-14T09:53:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.421+0000 7f7ac1000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc199ff0 con 0x7f7abc101960 2024-03-14T09:53:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.422+0000 7f7ab9400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7abc101960 0x7f7abc19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.422+0000 7f7aba800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc106090 0x7f7abc199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.422+0000 7f7ab9400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7abc101960 0x7f7abc19fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:48776/0 (socket says 172.21.15.78:48776) 2024-03-14T09:53:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.422+0000 7f7ab9400700 1 -- 172.21.15.78:0/8995384 learned_addr learned my addr 172.21.15.78:0/8995384 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.422+0000 7f7ab9e00700 1 --2- 172.21.15.78:0/8995384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7abc100fb0 0x7f7abc19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.422+0000 7f7ab9400700 1 -- 172.21.15.78:0/8995384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7abc100fb0 msgr2=0x7f7abc19f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.422+0000 7f7ab9400700 1 --2- 172.21.15.78:0/8995384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7abc100fb0 0x7f7abc19f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.422+0000 7f7ab9400700 1 -- 172.21.15.78:0/8995384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc106090 msgr2=0x7f7abc199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.422+0000 7f7ab9400700 1 --2- 172.21.15.78:0/8995384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc106090 0x7f7abc199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.422+0000 7f7ab9400700 1 -- 172.21.15.78:0/8995384 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa80095f0 con 0x7f7abc101960 2024-03-14T09:53:04.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.422+0000 7f7ab9400700 1 --2- 172.21.15.78:0/8995384 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7abc101960 0x7f7abc19fa60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8000c00 tx=0x7f7aa8009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:04.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.423+0000 7f7ab2a00700 1 -- 172.21.15.78:0/8995384 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa801d070 con 0x7f7abc101960 2024-03-14T09:53:04.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.423+0000 7f7ab2a00700 1 -- 172.21.15.78:0/8995384 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7aa8021410 con 0x7f7abc101960 2024-03-14T09:53:04.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.423+0000 7f7ab2a00700 1 -- 172.21.15.78:0/8995384 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8017650 con 0x7f7abc101960 2024-03-14T09:53:04.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.423+0000 7f7ac1000700 1 -- 172.21.15.78:0/8995384 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7abc19a270 con 0x7f7abc101960 2024-03-14T09:53:04.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.423+0000 7f7ac1000700 1 -- 172.21.15.78:0/8995384 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7abc1a6510 con 0x7f7abc101960 2024-03-14T09:53:04.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.424+0000 7f7ac1000700 1 -- 172.21.15.78:0/8995384 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7abc102490 con 0x7f7abc101960 2024-03-14T09:53:04.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.426+0000 7f7ab2a00700 1 -- 172.21.15.78:0/8995384 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f7aa8021580 con 0x7f7abc101960 2024-03-14T09:53:04.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.428+0000 7f7ab2a00700 1 --2- 172.21.15.78:0/8995384 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7aa00a0ba0 0x7f7aa00a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:04.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.428+0000 7f7ab2a00700 1 -- 172.21.15.78:0/8995384 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7aa80cd740 con 0x7f7abc101960 2024-03-14T09:53:04.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.428+0000 7f7ab9e00700 1 --2- 172.21.15.78:0/8995384 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7aa00a0ba0 0x7f7aa00a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:04.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.429+0000 7f7ab9e00700 1 --2- 172.21.15.78:0/8995384 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7aa00a0ba0 0x7f7aa00a3060 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7aac009c60 tx=0x7f7aac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:04.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.431+0000 7f7ab2a00700 1 -- 172.21.15.78:0/8995384 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7aa8096cd0 con 0x7f7abc101960 2024-03-14T09:53:04.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:04 smithi152 ceph-mon[48279]: pgmap v266: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-14T09:53:04.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:04 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:53:04.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:04 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/4036839438' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-14T09:53:04.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:04 smithi132 ceph-mon[47959]: pgmap v266: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-14T09:53:04.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:04 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:53:04.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:04 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/4036839438' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-14T09:53:04.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.651+0000 7f7ac1000700 1 -- 172.21.15.78:0/8995384 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f7abc19a4c0 con 0x7f7abc101960 2024-03-14T09:53:04.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.651+0000 7f7ab2a00700 1 -- 172.21.15.78:0/8995384 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7aa809a2f0 con 0x7f7abc101960 2024-03-14T09:53:04.652 INFO:teuthology.orchestra.run.smithi078.stdout:206158430245 2024-03-14T09:53:04.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.653+0000 7f7ac1000700 1 -- 172.21.15.78:0/8995384 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7aa00a0ba0 msgr2=0x7f7aa00a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:04.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.653+0000 7f7ac1000700 1 --2- 172.21.15.78:0/8995384 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7aa00a0ba0 0x7f7aa00a3060 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7aac009c60 tx=0x7f7aac009260 comp rx=0 tx=0).stop 2024-03-14T09:53:04.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.654+0000 7f7ac1000700 1 -- 172.21.15.78:0/8995384 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7abc101960 msgr2=0x7f7abc19fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:04.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.654+0000 7f7ac1000700 1 --2- 172.21.15.78:0/8995384 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7abc101960 0x7f7abc19fa60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8000c00 tx=0x7f7aa8009580 comp rx=0 tx=0).stop 2024-03-14T09:53:04.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.654+0000 7f7ac1000700 1 -- 172.21.15.78:0/8995384 shutdown_connections 2024-03-14T09:53:04.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.654+0000 7f7ac1000700 1 --2- 172.21.15.78:0/8995384 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7aa00a0ba0 0x7f7aa00a3060 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.654+0000 7f7ac1000700 1 --2- 172.21.15.78:0/8995384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7abc100fb0 0x7f7abc19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.654+0000 7f7ac1000700 1 --2- 172.21.15.78:0/8995384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7abc106090 0x7f7abc199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.654+0000 7f7ac1000700 1 --2- 172.21.15.78:0/8995384 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7abc101960 0x7f7abc19fa60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.654+0000 7f7ac1000700 1 -- 172.21.15.78:0/8995384 >> 172.21.15.78:0/8995384 conn(0x7f7abc0fc800 msgr2=0x7f7abc0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:04.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.654+0000 7f7ac1000700 1 -- 172.21.15.78:0/8995384 shutdown_connections 2024-03-14T09:53:04.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.654+0000 7f7ac1000700 1 -- 172.21.15.78:0/8995384 wait complete. 2024-03-14T09:53:04.716 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574906 got 55834574908 for osd.1 2024-03-14T09:53:04.717 DEBUG:teuthology.parallel:result is None 2024-03-14T09:53:04.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.727+0000 7f71f5800700 1 -- 172.21.15.78:0/3676912644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0100fb0 msgr2=0x7f71f0101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:04.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.727+0000 7f71f5800700 1 --2- 172.21.15.78:0/3676912644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0100fb0 0x7f71f0101390 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f71e0009990 tx=0x7f71e0009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.727+0000 7f71f5800700 1 -- 172.21.15.78:0/3676912644 shutdown_connections 2024-03-14T09:53:04.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.727+0000 7f71f5800700 1 --2- 172.21.15.78:0/3676912644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71f0106090 0x7f71f010dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.727+0000 7f71f5800700 1 --2- 172.21.15.78:0/3676912644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0100fb0 0x7f71f0101390 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.727+0000 7f71f5800700 1 --2- 172.21.15.78:0/3676912644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71f0101960 0x7f71f01059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.727+0000 7f71f5800700 1 -- 172.21.15.78:0/3676912644 >> 172.21.15.78:0/3676912644 conn(0x7f71f00fc800 msgr2=0x7f71f00fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:04.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.727+0000 7f71f5800700 1 -- 172.21.15.78:0/3676912644 shutdown_connections 2024-03-14T09:53:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.727+0000 7f71f5800700 1 -- 172.21.15.78:0/3676912644 wait complete. 2024-03-14T09:53:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.728+0000 7f71f5800700 1 Processor -- start 2024-03-14T09:53:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.728+0000 7f71f5800700 1 -- start start 2024-03-14T09:53:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.729+0000 7f71f5800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71f0100fb0 0x7f71f019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.729+0000 7f71f5800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71f0101960 0x7f71f019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.729+0000 7f71f5800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0106090 0x7f71f010cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.729+0000 7f71f5800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f010d430 con 0x7f71f0106090 2024-03-14T09:53:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.729+0000 7f71f5800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f010d5a0 con 0x7f71f0100fb0 2024-03-14T09:53:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.729+0000 7f71f5800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f010d760 con 0x7f71f0101960 2024-03-14T09:53:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.729+0000 7f71ee000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71f0101960 0x7f71f019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.729+0000 7f71ef400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0106090 0x7f71f010cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.729+0000 7f71ee000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71f0101960 0x7f71f019a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:48800/0 (socket says 172.21.15.78:48800) 2024-03-14T09:53:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.729+0000 7f71ee000700 1 -- 172.21.15.78:0/3190091705 learned_addr learned my addr 172.21.15.78:0/3190091705 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.729+0000 7f71eea00700 1 --2- 172.21.15.78:0/3190091705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71f0100fb0 0x7f71f019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.730+0000 7f71eea00700 1 -- 172.21.15.78:0/3190091705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71f0101960 msgr2=0x7f71f019a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.730+0000 7f71eea00700 1 --2- 172.21.15.78:0/3190091705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71f0101960 0x7f71f019a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.730+0000 7f71eea00700 1 -- 172.21.15.78:0/3190091705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0106090 msgr2=0x7f71f010cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.730+0000 7f71eea00700 1 --2- 172.21.15.78:0/3190091705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0106090 0x7f71f010cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.730+0000 7f71eea00700 1 -- 172.21.15.78:0/3190091705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e00095f0 con 0x7f71f0100fb0 2024-03-14T09:53:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.730+0000 7f71ef400700 1 --2- 172.21.15.78:0/3190091705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0106090 0x7f71f010cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.730+0000 7f71eea00700 1 --2- 172.21.15.78:0/3190091705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71f0100fb0 0x7f71f019a100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f71e000b680 tx=0x7f71e0005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:04.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.730+0000 7f71e7400700 1 -- 172.21.15.78:0/3190091705 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e001d070 con 0x7f71f0100fb0 2024-03-14T09:53:04.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.730+0000 7f71e7400700 1 -- 172.21.15.78:0/3190091705 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71e000be70 con 0x7f71f0100fb0 2024-03-14T09:53:04.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.730+0000 7f71f5800700 1 -- 172.21.15.78:0/3190091705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f010d9e0 con 0x7f71f0100fb0 2024-03-14T09:53:04.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.730+0000 7f71e7400700 1 -- 172.21.15.78:0/3190091705 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e0021410 con 0x7f71f0100fb0 2024-03-14T09:53:04.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.731+0000 7f71f5800700 1 -- 172.21.15.78:0/3190091705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f019e0a0 con 0x7f71f0100fb0 2024-03-14T09:53:04.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.732+0000 7f71f5800700 1 -- 172.21.15.78:0/3190091705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f0102490 con 0x7f71f0100fb0 2024-03-14T09:53:04.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.735+0000 7f71e7400700 1 -- 172.21.15.78:0/3190091705 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f71e000f660 con 0x7f71f0100fb0 2024-03-14T09:53:04.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.737+0000 7f71e7400700 1 --2- 172.21.15.78:0/3190091705 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f71d40a0be0 0x7f71d40a30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:04.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.737+0000 7f71e7400700 1 -- 172.21.15.78:0/3190091705 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f71e00cd4e0 con 0x7f71f0100fb0 2024-03-14T09:53:04.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.737+0000 7f71ee000700 1 --2- 172.21.15.78:0/3190091705 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f71d40a0be0 0x7f71d40a30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:04.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.738+0000 7f71ee000700 1 --2- 172.21.15.78:0/3190091705 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f71d40a0be0 0x7f71d40a30a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f71f0196a00 tx=0x7f71dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:04.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.740+0000 7f71e7400700 1 -- 172.21.15.78:0/3190091705 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f71e0096ad0 con 0x7f71f0100fb0 2024-03-14T09:53:04.769 INFO:tasks.cephadm.ceph_manager.ceph:need seq 103079215155 got 103079215156 for osd.3 2024-03-14T09:53:04.769 DEBUG:teuthology.parallel:result is None 2024-03-14T09:53:04.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.966+0000 7f71f5800700 1 -- 172.21.15.78:0/3190091705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f71f01111a0 con 0x7f71f0100fb0 2024-03-14T09:53:04.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.967+0000 7f71e7400700 1 -- 172.21.15.78:0/3190091705 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f71e009a0f0 con 0x7f71f0100fb0 2024-03-14T09:53:04.968 INFO:teuthology.orchestra.run.smithi078.stdout:133143986224 2024-03-14T09:53:04.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.970+0000 7f71f5800700 1 -- 172.21.15.78:0/3190091705 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f71d40a0be0 msgr2=0x7f71d40a30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.970+0000 7f71f5800700 1 --2- 172.21.15.78:0/3190091705 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f71d40a0be0 0x7f71d40a30a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f71f0196a00 tx=0x7f71dc009260 comp rx=0 tx=0).stop 2024-03-14T09:53:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.970+0000 7f71f5800700 1 -- 172.21.15.78:0/3190091705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71f0100fb0 msgr2=0x7f71f019a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.970+0000 7f71f5800700 1 --2- 172.21.15.78:0/3190091705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71f0100fb0 0x7f71f019a100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f71e000b680 tx=0x7f71e0005740 comp rx=0 tx=0).stop 2024-03-14T09:53:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.970+0000 7f71f5800700 1 -- 172.21.15.78:0/3190091705 shutdown_connections 2024-03-14T09:53:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.970+0000 7f71f5800700 1 --2- 172.21.15.78:0/3190091705 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f71d40a0be0 0x7f71d40a30a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.970+0000 7f71f5800700 1 --2- 172.21.15.78:0/3190091705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71f0100fb0 0x7f71f019a100 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.970+0000 7f71f5800700 1 --2- 172.21.15.78:0/3190091705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71f0106090 0x7f71f010cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.970+0000 7f71f5800700 1 --2- 172.21.15.78:0/3190091705 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71f0101960 0x7f71f019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.970+0000 7f71f5800700 1 -- 172.21.15.78:0/3190091705 >> 172.21.15.78:0/3190091705 conn(0x7f71f00fc800 msgr2=0x7f71f00feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.970+0000 7f71f5800700 1 -- 172.21.15.78:0/3190091705 shutdown_connections 2024-03-14T09:53:04.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:04.970+0000 7f71f5800700 1 -- 172.21.15.78:0/3190091705 wait complete. 2024-03-14T09:53:05.132 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411381 got 77309411382 for osd.2 2024-03-14T09:53:05.132 DEBUG:teuthology.parallel:result is None 2024-03-14T09:53:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:53:05.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:05 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:05.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:05 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:05.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:05 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:05.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:05 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:05.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:05 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/8995384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-14T09:53:05.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:05 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3190091705' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-14T09:53:05.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:05 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/850818978' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-14T09:53:05.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.074+0000 7f29fb600700 1 -- 172.21.15.78:0/3638996338 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29f4101960 msgr2=0x7f29f41059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:05.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.074+0000 7f29fb600700 1 --2- 172.21.15.78:0/3638996338 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29f4101960 0x7f29f41059b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009990 tx=0x7f29e0009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:05.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.074+0000 7f29fb600700 1 -- 172.21.15.78:0/3638996338 shutdown_connections 2024-03-14T09:53:05.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.074+0000 7f29fb600700 1 --2- 172.21.15.78:0/3638996338 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29f4100fb0 0x7f29f4101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:05.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.074+0000 7f29fb600700 1 --2- 172.21.15.78:0/3638996338 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29f4106090 0x7f29f410dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:05.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.074+0000 7f29fb600700 1 --2- 172.21.15.78:0/3638996338 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29f4101960 0x7f29f41059b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:05.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.074+0000 7f29fb600700 1 -- 172.21.15.78:0/3638996338 >> 172.21.15.78:0/3638996338 conn(0x7f29f40fc800 msgr2=0x7f29f40fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.074+0000 7f29fb600700 1 -- 172.21.15.78:0/3638996338 shutdown_connections 2024-03-14T09:53:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.074+0000 7f29fb600700 1 -- 172.21.15.78:0/3638996338 wait complete. 2024-03-14T09:53:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.076+0000 7f29fb600700 1 Processor -- start 2024-03-14T09:53:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.076+0000 7f29fb600700 1 -- start start 2024-03-14T09:53:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.076+0000 7f29fb600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29f4100fb0 0x7f29f41976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.076+0000 7f29fb600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29f4101960 0x7f29f4197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.076+0000 7f29fb600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29f4106090 0x7f29f419bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.076+0000 7f29fb600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f419c590 con 0x7f29f4101960 2024-03-14T09:53:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.076+0000 7f29fb600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f419c700 con 0x7f29f4100fb0 2024-03-14T09:53:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.076+0000 7f29fb600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f419c8c0 con 0x7f29f4106090 2024-03-14T09:53:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.077+0000 7f29f3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29f4101960 0x7f29f4197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.077+0000 7f29f8c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29f4100fb0 0x7f29f41976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:05.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.077+0000 7f29f8c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29f4100fb0 0x7f29f41976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:38268/0 (socket says 172.21.15.78:38268) 2024-03-14T09:53:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.077+0000 7f29f8c00700 1 -- 172.21.15.78:0/850818978 learned_addr learned my addr 172.21.15.78:0/850818978 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.077+0000 7f29f9600700 1 --2- 172.21.15.78:0/850818978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29f4106090 0x7f29f419bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.077+0000 7f29f8c00700 1 -- 172.21.15.78:0/850818978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29f4106090 msgr2=0x7f29f419bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.077+0000 7f29f8c00700 1 --2- 172.21.15.78:0/850818978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29f4106090 0x7f29f419bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.077+0000 7f29f8c00700 1 -- 172.21.15.78:0/850818978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29f4101960 msgr2=0x7f29f4197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.077+0000 7f29f8c00700 1 --2- 172.21.15.78:0/850818978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29f4101960 0x7f29f4197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.077+0000 7f29f8c00700 1 -- 172.21.15.78:0/850818978 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e00095f0 con 0x7f29f4100fb0 2024-03-14T09:53:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.077+0000 7f29f3e00700 1 --2- 172.21.15.78:0/850818978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29f4101960 0x7f29f4197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.077+0000 7f29f8c00700 1 --2- 172.21.15.78:0/850818978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29f4100fb0 0x7f29f41976e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f29e800e9a0 tx=0x7f29e800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.078+0000 7f29f1600700 1 -- 172.21.15.78:0/850818978 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e8009790 con 0x7f29f4100fb0 2024-03-14T09:53:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.078+0000 7f29f1600700 1 -- 172.21.15.78:0/850818978 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f29e8010430 con 0x7f29f4100fb0 2024-03-14T09:53:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.078+0000 7f29fb600700 1 -- 172.21.15.78:0/850818978 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f419cba0 con 0x7f29f4100fb0 2024-03-14T09:53:05.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.078+0000 7f29f1600700 1 -- 172.21.15.78:0/850818978 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e8018680 con 0x7f29f4100fb0 2024-03-14T09:53:05.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.078+0000 7f29fb600700 1 -- 172.21.15.78:0/850818978 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f41a4400 con 0x7f29f4100fb0 2024-03-14T09:53:05.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.079+0000 7f29fb600700 1 -- 172.21.15.78:0/850818978 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f41093b0 con 0x7f29f4100fb0 2024-03-14T09:53:05.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.081+0000 7f29f1600700 1 -- 172.21.15.78:0/850818978 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f29e80105a0 con 0x7f29f4100fb0 2024-03-14T09:53:05.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.082+0000 7f29f1600700 1 --2- 172.21.15.78:0/850818978 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f29dc0a0b20 0x7f29dc0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:05.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.082+0000 7f29f1600700 1 -- 172.21.15.78:0/850818978 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f29e8014070 con 0x7f29f4100fb0 2024-03-14T09:53:05.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.082+0000 7f29f3e00700 1 --2- 172.21.15.78:0/850818978 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f29dc0a0b20 0x7f29dc0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:05.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.083+0000 7f29f3e00700 1 --2- 172.21.15.78:0/850818978 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f29dc0a0b20 0x7f29dc0a2fe0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f29f4198bf0 tx=0x7f29e000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:05.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.084+0000 7f29f1600700 1 -- 172.21.15.78:0/850818978 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f29e8095ee0 con 0x7f29f4100fb0 2024-03-14T09:53:05.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.299+0000 7f29fb600700 1 -- 172.21.15.78:0/850818978 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f29f4102490 con 0x7f29f4100fb0 2024-03-14T09:53:05.620 INFO:teuthology.orchestra.run.smithi078.stdout:34359738432 2024-03-14T09:53:05.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.300+0000 7f29f1600700 1 -- 172.21.15.78:0/850818978 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f29e8099500 con 0x7f29f4100fb0 2024-03-14T09:53:05.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.302+0000 7f29fb600700 1 -- 172.21.15.78:0/850818978 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f29dc0a0b20 msgr2=0x7f29dc0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:05.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.302+0000 7f29fb600700 1 --2- 172.21.15.78:0/850818978 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f29dc0a0b20 0x7f29dc0a2fe0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f29f4198bf0 tx=0x7f29e000b260 comp rx=0 tx=0).stop 2024-03-14T09:53:05.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.302+0000 7f29fb600700 1 -- 172.21.15.78:0/850818978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29f4100fb0 msgr2=0x7f29f41976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:05.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.302+0000 7f29fb600700 1 --2- 172.21.15.78:0/850818978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29f4100fb0 0x7f29f41976e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f29e800e9a0 tx=0x7f29e800ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:05.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.302+0000 7f29fb600700 1 -- 172.21.15.78:0/850818978 shutdown_connections 2024-03-14T09:53:05.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.302+0000 7f29fb600700 1 --2- 172.21.15.78:0/850818978 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f29dc0a0b20 0x7f29dc0a2fe0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:05.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.302+0000 7f29fb600700 1 --2- 172.21.15.78:0/850818978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29f4100fb0 0x7f29f41976e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:05.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.302+0000 7f29fb600700 1 --2- 172.21.15.78:0/850818978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f29f4106090 0x7f29f419bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:05.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.302+0000 7f29fb600700 1 --2- 172.21.15.78:0/850818978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29f4101960 0x7f29f4197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:05.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.302+0000 7f29fb600700 1 -- 172.21.15.78:0/850818978 >> 172.21.15.78:0/850818978 conn(0x7f29f40fc800 msgr2=0x7f29f40fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:05.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.303+0000 7f29fb600700 1 -- 172.21.15.78:0/850818978 shutdown_connections 2024-03-14T09:53:05.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:05.303+0000 7f29fb600700 1 -- 172.21.15.78:0/850818978 wait complete. 2024-03-14T09:53:05.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:05 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:05.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:05 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:05.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:05 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:05.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:05 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:05.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:05 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/8995384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-14T09:53:05.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:05 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3190091705' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-14T09:53:05.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:05 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/850818978' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-14T09:53:05.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:05 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:05.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:05 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:05.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:05 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:05.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:05 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:05.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:05 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/8995384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-14T09:53:05.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:05 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3190091705' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-14T09:53:05.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:05 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/850818978' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-14T09:53:05.790 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972380 got 266287972381 for osd.9 2024-03-14T09:53:05.790 DEBUG:teuthology.parallel:result is None 2024-03-14T09:53:05.935 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430243 got 206158430245 for osd.7 2024-03-14T09:53:05.935 DEBUG:teuthology.parallel:result is None 2024-03-14T09:53:06.032 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822697 got 154618822701 for osd.5 2024-03-14T09:53:06.032 DEBUG:teuthology.parallel:result is None 2024-03-14T09:53:06.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.379+0000 7f5d05400700 1 -- 172.21.15.78:0/233145935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d00106090 msgr2=0x7f5d0010dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:06.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.379+0000 7f5d05400700 1 --2- 172.21.15.78:0/233145935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d00106090 0x7f5d0010dc60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800b0a0 tx=0x7f5cf800b3b0 comp rx=0 tx=0).stop 2024-03-14T09:53:06.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.379+0000 7f5d05400700 1 -- 172.21.15.78:0/233145935 shutdown_connections 2024-03-14T09:53:06.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.379+0000 7f5d05400700 1 --2- 172.21.15.78:0/233145935 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d00100fb0 0x7f5d00101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:06.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.379+0000 7f5d05400700 1 --2- 172.21.15.78:0/233145935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d00106090 0x7f5d0010dc60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:06.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.379+0000 7f5d05400700 1 --2- 172.21.15.78:0/233145935 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d00101960 0x7f5d001059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:06.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.379+0000 7f5d05400700 1 -- 172.21.15.78:0/233145935 >> 172.21.15.78:0/233145935 conn(0x7f5d000fc800 msgr2=0x7f5d000fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:06.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.379+0000 7f5d05400700 1 -- 172.21.15.78:0/233145935 shutdown_connections 2024-03-14T09:53:06.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.379+0000 7f5d05400700 1 -- 172.21.15.78:0/233145935 wait complete. 2024-03-14T09:53:06.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.380+0000 7f5d05400700 1 Processor -- start 2024-03-14T09:53:06.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.380+0000 7f5d05400700 1 -- start start 2024-03-14T09:53:06.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.380+0000 7f5d05400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d00100fb0 0x7f5d000730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:06.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.380+0000 7f5d05400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d00101960 0x7f5d000735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:06.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.380+0000 7f5d05400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d00106090 0x7f5d00073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:06.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.380+0000 7f5d05400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d001a5aa0 con 0x7f5d00100fb0 2024-03-14T09:53:06.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.380+0000 7f5d05400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d001a5c10 con 0x7f5d00101960 2024-03-14T09:53:06.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.381+0000 7f5d05400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d001a5dd0 con 0x7f5d00106090 2024-03-14T09:53:06.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.381+0000 7f5cfe800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d00100fb0 0x7f5d000730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:06.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.381+0000 7f5cfde00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d00101960 0x7f5d000735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:06.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.381+0000 7f5cfe800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d00100fb0 0x7f5d000730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47532/0 (socket says 172.21.15.78:47532) 2024-03-14T09:53:06.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.381+0000 7f5cfe800700 1 -- 172.21.15.78:0/1242387921 learned_addr learned my addr 172.21.15.78:0/1242387921 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:06.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.381+0000 7f5cff200700 1 --2- 172.21.15.78:0/1242387921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d00106090 0x7f5d00073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.381+0000 7f5cfe800700 1 -- 172.21.15.78:0/1242387921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d00106090 msgr2=0x7f5d00073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.381+0000 7f5cfe800700 1 --2- 172.21.15.78:0/1242387921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d00106090 0x7f5d00073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.381+0000 7f5cfe800700 1 -- 172.21.15.78:0/1242387921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d00101960 msgr2=0x7f5d000735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:06.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.381+0000 7f5cfe800700 1 --2- 172.21.15.78:0/1242387921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d00101960 0x7f5d000735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:06.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.381+0000 7f5cfe800700 1 -- 172.21.15.78:0/1242387921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf8009d00 con 0x7f5d00100fb0 2024-03-14T09:53:06.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.381+0000 7f5cfe800700 1 --2- 172.21.15.78:0/1242387921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d00100fb0 0x7f5d000730b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000e9a0 tx=0x7f5cf000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:06.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.382+0000 7f5cf7400700 1 -- 172.21.15.78:0/1242387921 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0009790 con 0x7f5d00100fb0 2024-03-14T09:53:06.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.382+0000 7f5cf7400700 1 -- 172.21.15.78:0/1242387921 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5cf0010430 con 0x7f5d00100fb0 2024-03-14T09:53:06.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.382+0000 7f5d05400700 1 -- 172.21.15.78:0/1242387921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d001a60b0 con 0x7f5d00100fb0 2024-03-14T09:53:06.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.382+0000 7f5cf7400700 1 -- 172.21.15.78:0/1242387921 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0018680 con 0x7f5d00100fb0 2024-03-14T09:53:06.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.382+0000 7f5d05400700 1 -- 172.21.15.78:0/1242387921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d001a6560 con 0x7f5d00100fb0 2024-03-14T09:53:06.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.383+0000 7f5d05400700 1 -- 172.21.15.78:0/1242387921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d001093b0 con 0x7f5d00100fb0 2024-03-14T09:53:06.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.385+0000 7f5cf7400700 1 -- 172.21.15.78:0/1242387921 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f5cf00105a0 con 0x7f5d00100fb0 2024-03-14T09:53:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.386+0000 7f5cf7400700 1 --2- 172.21.15.78:0/1242387921 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5ce40a0b20 0x7f5ce40a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.386+0000 7f5cf7400700 1 -- 172.21.15.78:0/1242387921 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5cf0014070 con 0x7f5d00100fb0 2024-03-14T09:53:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.386+0000 7f5cfde00700 1 --2- 172.21.15.78:0/1242387921 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5ce40a0b20 0x7f5ce40a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:06.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.387+0000 7f5cfde00700 1 --2- 172.21.15.78:0/1242387921 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5ce40a0b20 0x7f5ce40a2fe0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5d000747e0 tx=0x7f5cec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:06.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.390+0000 7f5cf7400700 1 -- 172.21.15.78:0/1242387921 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5cf0052960 con 0x7f5d00100fb0 2024-03-14T09:53:06.467 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:06 smithi078 ceph-mon[38143]: pgmap v267: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-14T09:53:06.486 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986222 got 133143986224 for osd.4 2024-03-14T09:53:06.487 DEBUG:teuthology.parallel:result is None 2024-03-14T09:53:06.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.634+0000 7f5d05400700 1 -- 172.21.15.78:0/1242387921 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f5d00074590 con 0x7f5d00100fb0 2024-03-14T09:53:06.641 INFO:teuthology.orchestra.run.smithi078.stdout:287762808857 2024-03-14T09:53:06.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.634+0000 7f5cf7400700 1 -- 172.21.15.78:0/1242387921 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f5cf0095bf0 con 0x7f5d00100fb0 2024-03-14T09:53:06.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.637+0000 7f5d05400700 1 -- 172.21.15.78:0/1242387921 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5ce40a0b20 msgr2=0x7f5ce40a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:06.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.637+0000 7f5d05400700 1 --2- 172.21.15.78:0/1242387921 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5ce40a0b20 0x7f5ce40a2fe0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5d000747e0 tx=0x7f5cec009260 comp rx=0 tx=0).stop 2024-03-14T09:53:06.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.637+0000 7f5d05400700 1 -- 172.21.15.78:0/1242387921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d00100fb0 msgr2=0x7f5d000730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:06.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.637+0000 7f5d05400700 1 --2- 172.21.15.78:0/1242387921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d00100fb0 0x7f5d000730b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000e9a0 tx=0x7f5cf000ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:06.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.637+0000 7f5d05400700 1 -- 172.21.15.78:0/1242387921 shutdown_connections 2024-03-14T09:53:06.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.637+0000 7f5d05400700 1 --2- 172.21.15.78:0/1242387921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5d00106090 0x7f5d00073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:06.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.637+0000 7f5d05400700 1 --2- 172.21.15.78:0/1242387921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d00100fb0 0x7f5d000730b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:06.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.637+0000 7f5d05400700 1 --2- 172.21.15.78:0/1242387921 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5ce40a0b20 0x7f5ce40a2fe0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:06.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.637+0000 7f5d05400700 1 --2- 172.21.15.78:0/1242387921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d00101960 0x7f5d000735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:06.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.637+0000 7f5d05400700 1 -- 172.21.15.78:0/1242387921 >> 172.21.15.78:0/1242387921 conn(0x7f5d000fc800 msgr2=0x7f5d000feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:06.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.637+0000 7f5d05400700 1 -- 172.21.15.78:0/1242387921 shutdown_connections 2024-03-14T09:53:06.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:06.637+0000 7f5d05400700 1 -- 172.21.15.78:0/1242387921 wait complete. 2024-03-14T09:53:06.643 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738430 got 34359738432 for osd.0 2024-03-14T09:53:06.643 DEBUG:teuthology.parallel:result is None 2024-03-14T09:53:06.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:06 smithi152 ceph-mon[48279]: pgmap v267: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-14T09:53:06.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:06 smithi132 ceph-mon[47959]: pgmap v267: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-14T09:53:07.189 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808855 got 287762808857 for osd.10 2024-03-14T09:53:07.189 DEBUG:teuthology.parallel:result is None 2024-03-14T09:53:07.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:07 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1242387921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-14T09:53:07.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:07 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1242387921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-14T09:53:07.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:07 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1242387921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-14T09:53:07.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.824+0000 7f7ab7800700 1 -- 172.21.15.78:0/4153584637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ab01031a0 msgr2=0x7f7ab0103580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:07.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.824+0000 7f7ab7800700 1 --2- 172.21.15.78:0/4153584637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ab01031a0 0x7f7ab0103580 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4009990 tx=0x7f7aa4009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:07.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.824+0000 7f7ab7800700 1 -- 172.21.15.78:0/4153584637 shutdown_connections 2024-03-14T09:53:07.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.824+0000 7f7ab7800700 1 --2- 172.21.15.78:0/4153584637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ab0108280 0x7f7ab010fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.824+0000 7f7ab7800700 1 --2- 172.21.15.78:0/4153584637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ab01031a0 0x7f7ab0103580 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.824+0000 7f7ab7800700 1 --2- 172.21.15.78:0/4153584637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ab0103b50 0x7f7ab0107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.824+0000 7f7ab7800700 1 -- 172.21.15.78:0/4153584637 >> 172.21.15.78:0/4153584637 conn(0x7f7ab00fea50 msgr2=0x7f7ab0100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.824+0000 7f7ab7800700 1 -- 172.21.15.78:0/4153584637 shutdown_connections 2024-03-14T09:53:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.825+0000 7f7ab7800700 1 -- 172.21.15.78:0/4153584637 wait complete. 2024-03-14T09:53:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.826+0000 7f7ab7800700 1 Processor -- start 2024-03-14T09:53:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.826+0000 7f7ab7800700 1 -- start start 2024-03-14T09:53:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.827+0000 7f7ab7800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ab01031a0 0x7f7ab019f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.827+0000 7f7ab7800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ab0103b50 0x7f7ab019faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.827+0000 7f7ab7800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ab0108280 0x7f7ab0199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.827+0000 7f7ab7800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab0199da0 con 0x7f7ab0103b50 2024-03-14T09:53:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.827+0000 7f7ab7800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab0199f10 con 0x7f7ab01031a0 2024-03-14T09:53:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.827+0000 7f7ab7800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab019a0d0 con 0x7f7ab0108280 2024-03-14T09:53:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.827+0000 7f7aafe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ab0103b50 0x7f7ab019faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.827+0000 7f7ab4e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ab01031a0 0x7f7ab019f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.827+0000 7f7ab4e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ab01031a0 0x7f7ab019f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:38304/0 (socket says 172.21.15.78:38304) 2024-03-14T09:53:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.827+0000 7f7ab4e00700 1 -- 172.21.15.78:0/3421084446 learned_addr learned my addr 172.21.15.78:0/3421084446 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.827+0000 7f7aafe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ab0103b50 0x7f7ab019faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47564/0 (socket says 172.21.15.78:47564) 2024-03-14T09:53:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.827+0000 7f7ab5800700 1 --2- 172.21.15.78:0/3421084446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ab0108280 0x7f7ab0199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.827+0000 7f7aafe00700 1 -- 172.21.15.78:0/3421084446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ab0108280 msgr2=0x7f7ab0199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.828+0000 7f7aafe00700 1 --2- 172.21.15.78:0/3421084446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ab0108280 0x7f7ab0199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.828+0000 7f7aafe00700 1 -- 172.21.15.78:0/3421084446 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ab01031a0 msgr2=0x7f7ab019f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.828+0000 7f7aafe00700 1 --2- 172.21.15.78:0/3421084446 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ab01031a0 0x7f7ab019f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.828+0000 7f7aafe00700 1 -- 172.21.15.78:0/3421084446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa40095f0 con 0x7f7ab0103b50 2024-03-14T09:53:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.828+0000 7f7ab4e00700 1 --2- 172.21.15.78:0/3421084446 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ab01031a0 0x7f7ab019f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.828+0000 7f7aafe00700 1 --2- 172.21.15.78:0/3421084446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ab0103b50 0x7f7ab019faf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c00e9c0 tx=0x7f7a9c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:07.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.828+0000 7f7aad600700 1 -- 172.21.15.78:0/3421084446 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c00c9c0 con 0x7f7ab0103b50 2024-03-14T09:53:07.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.828+0000 7f7aad600700 1 -- 172.21.15.78:0/3421084446 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a9c010430 con 0x7f7ab0103b50 2024-03-14T09:53:07.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.828+0000 7f7ab7800700 1 -- 172.21.15.78:0/3421084446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab019a3b0 con 0x7f7ab0103b50 2024-03-14T09:53:07.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.828+0000 7f7aad600700 1 -- 172.21.15.78:0/3421084446 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c018680 con 0x7f7ab0103b50 2024-03-14T09:53:07.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.828+0000 7f7ab7800700 1 -- 172.21.15.78:0/3421084446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab01a65a0 con 0x7f7ab0103b50 2024-03-14T09:53:07.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.830+0000 7f7ab7800700 1 -- 172.21.15.78:0/3421084446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab010b540 con 0x7f7ab0103b50 2024-03-14T09:53:07.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.832+0000 7f7aad600700 1 -- 172.21.15.78:0/3421084446 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f7a9c00cb20 con 0x7f7ab0103b50 2024-03-14T09:53:07.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.833+0000 7f7aad600700 1 --2- 172.21.15.78:0/3421084446 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7a980a0ba0 0x7f7a980a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:07.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.833+0000 7f7aad600700 1 -- 172.21.15.78:0/3421084446 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7a9c014070 con 0x7f7ab0103b50 2024-03-14T09:53:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.833+0000 7f7ab4e00700 1 --2- 172.21.15.78:0/3421084446 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7a980a0ba0 0x7f7a980a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:07.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.834+0000 7f7ab4e00700 1 --2- 172.21.15.78:0/3421084446 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7a980a0ba0 0x7f7a980a3060 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4006010 tx=0x7f7aa400b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:07.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:07.837+0000 7f7aad600700 1 -- 172.21.15.78:0/3421084446 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7a9c052960 con 0x7f7ab0103b50 2024-03-14T09:53:08.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:08.062+0000 7f7ab7800700 1 -- 172.21.15.78:0/3421084446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f7ab004ea80 con 0x7f7ab0103b50 2024-03-14T09:53:08.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:08.063+0000 7f7aad600700 1 -- 172.21.15.78:0/3421084446 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7a9c095a90 con 0x7f7ab0103b50 2024-03-14T09:53:08.063 INFO:teuthology.orchestra.run.smithi078.stdout:236223201313 2024-03-14T09:53:08.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:08.065+0000 7f7ab7800700 1 -- 172.21.15.78:0/3421084446 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7a980a0ba0 msgr2=0x7f7a980a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:08.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:08.065+0000 7f7ab7800700 1 --2- 172.21.15.78:0/3421084446 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7a980a0ba0 0x7f7a980a3060 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4006010 tx=0x7f7aa400b280 comp rx=0 tx=0).stop 2024-03-14T09:53:08.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:08.065+0000 7f7ab7800700 1 -- 172.21.15.78:0/3421084446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ab0103b50 msgr2=0x7f7ab019faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:08.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:08.065+0000 7f7ab7800700 1 --2- 172.21.15.78:0/3421084446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ab0103b50 0x7f7ab019faf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c00e9c0 tx=0x7f7a9c00ecd0 comp rx=0 tx=0).stop 2024-03-14T09:53:08.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:08.065+0000 7f7ab7800700 1 -- 172.21.15.78:0/3421084446 shutdown_connections 2024-03-14T09:53:08.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:08.065+0000 7f7ab7800700 1 --2- 172.21.15.78:0/3421084446 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7a980a0ba0 0x7f7a980a3060 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:08.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:08.065+0000 7f7ab7800700 1 --2- 172.21.15.78:0/3421084446 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ab01031a0 0x7f7ab019f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:08.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:08.065+0000 7f7ab7800700 1 --2- 172.21.15.78:0/3421084446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ab0108280 0x7f7ab0199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:08.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:08.065+0000 7f7ab7800700 1 --2- 172.21.15.78:0/3421084446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ab0103b50 0x7f7ab019faf0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:08.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:08.065+0000 7f7ab7800700 1 -- 172.21.15.78:0/3421084446 >> 172.21.15.78:0/3421084446 conn(0x7f7ab00fea50 msgr2=0x7f7ab00fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:08.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:08.065+0000 7f7ab7800700 1 -- 172.21.15.78:0/3421084446 shutdown_connections 2024-03-14T09:53:08.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:08.065+0000 7f7ab7800700 1 -- 172.21.15.78:0/3421084446 wait complete. 2024-03-14T09:53:08.451 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201310 got 236223201313 for osd.8 2024-03-14T09:53:08.451 DEBUG:teuthology.parallel:result is None 2024-03-14T09:53:08.451 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-14T09:53:08.452 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-14T09:53:08.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:08 smithi152 ceph-mon[48279]: pgmap v268: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:08.749 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:08 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3421084446' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-14T09:53:08.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:08 smithi132 ceph-mon[47959]: pgmap v268: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:08.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:08 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3421084446' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-14T09:53:08.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:08 smithi078 ceph-mon[38143]: pgmap v268: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:08.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:08 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3421084446' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-14T09:53:08.829 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:53:09.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:09 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:09.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:09 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:09.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:09 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:53:09.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:09 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:53:09.753 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:09 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:09.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:09 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:09.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:09 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:09.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:09 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:53:09.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:09 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:53:09.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:09 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:09.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:09 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:09.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:09 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:09.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:09 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:53:09.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:09 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:53:09.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:09 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' 2024-03-14T09:53:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:09.999+0000 7fc832400700 1 -- 172.21.15.78:0/240652652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc82c073180 msgr2=0x7fc82c073560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:09.999+0000 7fc832400700 1 --2- 172.21.15.78:0/240652652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc82c073180 0x7fc82c073560 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009990 tx=0x7fc81c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:09.999+0000 7fc832400700 1 -- 172.21.15.78:0/240652652 shutdown_connections 2024-03-14T09:53:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:09.999+0000 7fc832400700 1 --2- 172.21.15.78:0/240652652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc82c10d5e0 0x7fc82c10f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:09.999+0000 7fc832400700 1 --2- 172.21.15.78:0/240652652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc82c073180 0x7fc82c073560 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:09.999+0000 7fc832400700 1 --2- 172.21.15.78:0/240652652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc82c073aa0 0x7fc82c10d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:09.999+0000 7fc832400700 1 -- 172.21.15.78:0/240652652 >> 172.21.15.78:0/240652652 conn(0x7fc82c0fc850 msgr2=0x7fc82c0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:09.999+0000 7fc832400700 1 -- 172.21.15.78:0/240652652 shutdown_connections 2024-03-14T09:53:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:09.999+0000 7fc832400700 1 -- 172.21.15.78:0/240652652 wait complete. 2024-03-14T09:53:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.000+0000 7fc832400700 1 Processor -- start 2024-03-14T09:53:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.000+0000 7fc832400700 1 -- start start 2024-03-14T09:53:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.000+0000 7fc832400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc82c073180 0x7fc82c19f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.000+0000 7fc832400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc82c073aa0 0x7fc82c19fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.001+0000 7fc832400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc82c10d5e0 0x7fc82c199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.001+0000 7fc832400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c199ce0 con 0x7fc82c073aa0 2024-03-14T09:53:10.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.001+0000 7fc832400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c199e50 con 0x7fc82c10d5e0 2024-03-14T09:53:10.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.001+0000 7fc832400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82c19a010 con 0x7fc82c073180 2024-03-14T09:53:10.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.001+0000 7fc823e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc82c073aa0 0x7fc82c19fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:10.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.001+0000 7fc823e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc82c073aa0 0x7fc82c19fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47576/0 (socket says 172.21.15.78:47576) 2024-03-14T09:53:10.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.001+0000 7fc823e00700 1 -- 172.21.15.78:0/2448858419 learned_addr learned my addr 172.21.15.78:0/2448858419 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:10.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.001+0000 7fc82b400700 1 --2- 172.21.15.78:0/2448858419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc82c073180 0x7fc82c19f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:10.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.001+0000 7fc82be00700 1 --2- 172.21.15.78:0/2448858419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc82c10d5e0 0x7fc82c199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:10.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.001+0000 7fc823e00700 1 -- 172.21.15.78:0/2448858419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc82c073180 msgr2=0x7fc82c19f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:10.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.001+0000 7fc823e00700 1 --2- 172.21.15.78:0/2448858419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc82c073180 0x7fc82c19f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:10.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.001+0000 7fc823e00700 1 -- 172.21.15.78:0/2448858419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc82c10d5e0 msgr2=0x7fc82c199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:10.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.001+0000 7fc823e00700 1 --2- 172.21.15.78:0/2448858419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc82c10d5e0 0x7fc82c199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:10.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.001+0000 7fc823e00700 1 -- 172.21.15.78:0/2448858419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc824009c40 con 0x7fc82c073aa0 2024-03-14T09:53:10.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.001+0000 7fc823e00700 1 --2- 172.21.15.78:0/2448858419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc82c073aa0 0x7fc82c19fa80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc81400e9a0 tx=0x7fc81400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:10.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.002+0000 7fc828c00700 1 -- 172.21.15.78:0/2448858419 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc814009790 con 0x7fc82c073aa0 2024-03-14T09:53:10.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.002+0000 7fc828c00700 1 -- 172.21.15.78:0/2448858419 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc81400ce80 con 0x7fc82c073aa0 2024-03-14T09:53:10.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.002+0000 7fc832400700 1 -- 172.21.15.78:0/2448858419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81c0095f0 con 0x7fc82c073aa0 2024-03-14T09:53:10.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.002+0000 7fc828c00700 1 -- 172.21.15.78:0/2448858419 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc814018680 con 0x7fc82c073aa0 2024-03-14T09:53:10.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.002+0000 7fc832400700 1 -- 172.21.15.78:0/2448858419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82c19a4e0 con 0x7fc82c073aa0 2024-03-14T09:53:10.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.003+0000 7fc832400700 1 -- 172.21.15.78:0/2448858419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc82c1087f0 con 0x7fc82c073aa0 2024-03-14T09:53:10.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.005+0000 7fc828c00700 1 -- 172.21.15.78:0/2448858419 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fc814010430 con 0x7fc82c073aa0 2024-03-14T09:53:10.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.006+0000 7fc828c00700 1 --2- 172.21.15.78:0/2448858419 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8100a0ba0 0x7fc8100a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:10.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.006+0000 7fc828c00700 1 -- 172.21.15.78:0/2448858419 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc814014070 con 0x7fc82c073aa0 2024-03-14T09:53:10.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.007+0000 7fc82b400700 1 --2- 172.21.15.78:0/2448858419 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8100a0ba0 0x7fc8100a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:10.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.007+0000 7fc82b400700 1 --2- 172.21.15.78:0/2448858419 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8100a0ba0 0x7fc8100a3060 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc81c006010 tx=0x7fc81c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:10.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.009+0000 7fc828c00700 1 -- 172.21.15.78:0/2448858419 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc814095f30 con 0x7fc82c073aa0 2024-03-14T09:53:10.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.206+0000 7fc832400700 1 -- 172.21.15.78:0/2448858419 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc82c19ab40 con 0x7fc8100a0ba0 2024-03-14T09:53:10.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.220+0000 7fc828c00700 1 -- 172.21.15.78:0/2448858419 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+424964 (secure 0 0 0) 0x7fc82c19ab40 con 0x7fc8100a0ba0 2024-03-14T09:53:10.228 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:10.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.225+0000 7fc832400700 1 -- 172.21.15.78:0/2448858419 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8100a0ba0 msgr2=0x7fc8100a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:10.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.225+0000 7fc832400700 1 --2- 172.21.15.78:0/2448858419 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8100a0ba0 0x7fc8100a3060 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc81c006010 tx=0x7fc81c00b260 comp rx=0 tx=0).stop 2024-03-14T09:53:10.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.225+0000 7fc832400700 1 -- 172.21.15.78:0/2448858419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc82c073aa0 msgr2=0x7fc82c19fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:10.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.225+0000 7fc832400700 1 --2- 172.21.15.78:0/2448858419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc82c073aa0 0x7fc82c19fa80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc81400e9a0 tx=0x7fc81400ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:10.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.226+0000 7fc832400700 1 -- 172.21.15.78:0/2448858419 shutdown_connections 2024-03-14T09:53:10.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.226+0000 7fc832400700 1 --2- 172.21.15.78:0/2448858419 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8100a0ba0 0x7fc8100a3060 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:10.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.226+0000 7fc832400700 1 --2- 172.21.15.78:0/2448858419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc82c10d5e0 0x7fc82c199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:10.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.226+0000 7fc832400700 1 --2- 172.21.15.78:0/2448858419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc82c073180 0x7fc82c19f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:10.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.226+0000 7fc832400700 1 --2- 172.21.15.78:0/2448858419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc82c073aa0 0x7fc82c19fa80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:10.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.226+0000 7fc832400700 1 -- 172.21.15.78:0/2448858419 >> 172.21.15.78:0/2448858419 conn(0x7fc82c0fc850 msgr2=0x7fc82c0fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:10.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.226+0000 7fc832400700 1 -- 172.21.15.78:0/2448858419 shutdown_connections 2024-03-14T09:53:10.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:10.226+0000 7fc832400700 1 -- 172.21.15.78:0/2448858419 wait complete. 2024-03-14T09:53:10.232 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-14T09:53:10.601 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-03-14T09:53:09.082586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656237,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":383,"num_read_kb":313,"num_write":336,"num_write_kb":3052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1181430,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":335612,"kb_used_data":11368,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124606212,"statfs":{"total":1151940427776,"available":1151596761088,"internally_reserved":0,"allocated":11640832,"data_stored":7586014,"data_compressed":323961,"data_compressed_allocated":442368,"data_compressed_original":2383872,"omap_allocated":19131,"internal_metadata":331855173},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":71,"num_read_kb":34,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003432"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141879+0000","last_change":"2024-03-14T09:52:05.737617+0000","last_active":"2024-03-14T09:52:41.141879+0000","last_peered":"2024-03-14T09:52:41.141879+0000","last_clean":"2024-03-14T09:52:41.141879+0000","last_became_active":"2024-03-14T09:52:05.736796+0000","last_became_peered":"2024-03-14T09:52:05.736796+0000","last_unstale":"2024-03-14T09:52:41.141879+0000","last_undegraded":"2024-03-14T09:52:41.141879+0000","last_fullsized":"2024-03-14T09:52:41.141879+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:47:52.437748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142968+0000","last_change":"2024-03-14T09:52:05.749474+0000","last_active":"2024-03-14T09:52:41.142968+0000","last_peered":"2024-03-14T09:52:41.142968+0000","last_clean":"2024-03-14T09:52:41.142968+0000","last_became_active":"2024-03-14T09:52:05.748927+0000","last_became_peered":"2024-03-14T09:52:05.748927+0000","last_unstale":"2024-03-14T09:52:41.142968+0000","last_undegraded":"2024-03-14T09:52:41.142968+0000","last_fullsized":"2024-03-14T09:52:41.142968+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:28:15.852021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142862+0000","last_change":"2024-03-14T09:52:05.745098+0000","last_active":"2024-03-14T09:52:41.142862+0000","last_peered":"2024-03-14T09:52:41.142862+0000","last_clean":"2024-03-14T09:52:41.142862+0000","last_became_active":"2024-03-14T09:52:05.744755+0000","last_became_peered":"2024-03-14T09:52:05.744755+0000","last_unstale":"2024-03-14T09:52:41.142862+0000","last_undegraded":"2024-03-14T09:52:41.142862+0000","last_fullsized":"2024-03-14T09:52:41.142862+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:35:41.932386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142028+0000","last_change":"2024-03-14T09:52:05.737429+0000","last_active":"2024-03-14T09:52:41.142028+0000","last_peered":"2024-03-14T09:52:41.142028+0000","last_clean":"2024-03-14T09:52:41.142028+0000","last_became_active":"2024-03-14T09:52:05.736516+0000","last_became_peered":"2024-03-14T09:52:05.736516+0000","last_unstale":"2024-03-14T09:52:41.142028+0000","last_undegraded":"2024-03-14T09:52:41.142028+0000","last_fullsized":"2024-03-14T09:52:41.142028+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:48:38.158783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137401+0000","last_change":"2024-03-14T09:52:05.729634+0000","last_active":"2024-03-14T09:52:41.137401+0000","last_peered":"2024-03-14T09:52:41.137401+0000","last_clean":"2024-03-14T09:52:41.137401+0000","last_became_active":"2024-03-14T09:52:05.727889+0000","last_became_peered":"2024-03-14T09:52:05.727889+0000","last_unstale":"2024-03-14T09:52:41.137401+0000","last_undegraded":"2024-03-14T09:52:41.137401+0000","last_fullsized":"2024-03-14T09:52:41.137401+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:01:13.216763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.147610+0000","last_change":"2024-03-14T09:52:42.147610+0000","last_active":"2024-03-14T09:52:42.147610+0000","last_peered":"2024-03-14T09:52:42.147610+0000","last_clean":"2024-03-14T09:52:42.147610+0000","last_became_active":"2024-03-14T09:52:42.147262+0000","last_became_peered":"2024-03-14T09:52:42.147262+0000","last_unstale":"2024-03-14T09:52:42.147610+0000","last_undegraded":"2024-03-14T09:52:42.147610+0000","last_fullsized":"2024-03-14T09:52:42.147610+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:20:49.855564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136737+0000","last_change":"2024-03-14T09:52:05.727365+0000","last_active":"2024-03-14T09:52:41.136737+0000","last_peered":"2024-03-14T09:52:41.136737+0000","last_clean":"2024-03-14T09:52:41.136737+0000","last_became_active":"2024-03-14T09:52:05.726984+0000","last_became_peered":"2024-03-14T09:52:05.726984+0000","last_unstale":"2024-03-14T09:52:41.136737+0000","last_undegraded":"2024-03-14T09:52:41.136737+0000","last_fullsized":"2024-03-14T09:52:41.136737+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:42:57.129718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145478+0000","last_change":"2024-03-14T09:52:05.729622+0000","last_active":"2024-03-14T09:52:41.145478+0000","last_peered":"2024-03-14T09:52:41.145478+0000","last_clean":"2024-03-14T09:52:41.145478+0000","last_became_active":"2024-03-14T09:52:05.729293+0000","last_became_peered":"2024-03-14T09:52:05.729293+0000","last_unstale":"2024-03-14T09:52:41.145478+0000","last_undegraded":"2024-03-14T09:52:41.145478+0000","last_fullsized":"2024-03-14T09:52:41.145478+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:13:20.875857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.144495+0000","last_change":"2024-03-14T09:52:05.741620+0000","last_active":"2024-03-14T09:52:41.144495+0000","last_peered":"2024-03-14T09:52:41.144495+0000","last_clean":"2024-03-14T09:52:41.144495+0000","last_became_active":"2024-03-14T09:52:05.734079+0000","last_became_peered":"2024-03-14T09:52:05.734079+0000","last_unstale":"2024-03-14T09:52:41.144495+0000","last_undegraded":"2024-03-14T09:52:41.144495+0000","last_fullsized":"2024-03-14T09:52:41.144495+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:02:35.529671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143730+0000","last_change":"2024-03-14T09:52:05.743424+0000","last_active":"2024-03-14T09:52:41.143730+0000","last_peered":"2024-03-14T09:52:41.143730+0000","last_clean":"2024-03-14T09:52:41.143730+0000","last_became_active":"2024-03-14T09:52:05.743100+0000","last_became_peered":"2024-03-14T09:52:05.743100+0000","last_unstale":"2024-03-14T09:52:41.143730+0000","last_undegraded":"2024-03-14T09:52:41.143730+0000","last_fullsized":"2024-03-14T09:52:41.143730+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:32:33.492163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137363+0000","last_change":"2024-03-14T09:52:05.735654+0000","last_active":"2024-03-14T09:52:41.137363+0000","last_peered":"2024-03-14T09:52:41.137363+0000","last_clean":"2024-03-14T09:52:41.137363+0000","last_became_active":"2024-03-14T09:52:05.735371+0000","last_became_peered":"2024-03-14T09:52:05.735371+0000","last_unstale":"2024-03-14T09:52:41.137363+0000","last_undegraded":"2024-03-14T09:52:41.137363+0000","last_fullsized":"2024-03-14T09:52:41.137363+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:19:50.011554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.146700+0000","last_change":"2024-03-14T09:52:42.146700+0000","last_active":"2024-03-14T09:52:42.146700+0000","last_peered":"2024-03-14T09:52:42.146700+0000","last_clean":"2024-03-14T09:52:42.146700+0000","last_became_active":"2024-03-14T09:52:42.145377+0000","last_became_peered":"2024-03-14T09:52:42.145377+0000","last_unstale":"2024-03-14T09:52:42.146700+0000","last_undegraded":"2024-03-14T09:52:42.146700+0000","last_fullsized":"2024-03-14T09:52:42.146700+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:35:52.735091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136767+0000","last_change":"2024-03-14T09:52:05.740209+0000","last_active":"2024-03-14T09:52:41.136767+0000","last_peered":"2024-03-14T09:52:41.136767+0000","last_clean":"2024-03-14T09:52:41.136767+0000","last_became_active":"2024-03-14T09:52:05.732949+0000","last_became_peered":"2024-03-14T09:52:05.732949+0000","last_unstale":"2024-03-14T09:52:41.136767+0000","last_undegraded":"2024-03-14T09:52:41.136767+0000","last_fullsized":"2024-03-14T09:52:41.136767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:15:56.377569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141029+0000","last_change":"2024-03-14T09:52:05.748795+0000","last_active":"2024-03-14T09:52:41.141029+0000","last_peered":"2024-03-14T09:52:41.141029+0000","last_clean":"2024-03-14T09:52:41.141029+0000","last_became_active":"2024-03-14T09:52:05.746107+0000","last_became_peered":"2024-03-14T09:52:05.746107+0000","last_unstale":"2024-03-14T09:52:41.141029+0000","last_undegraded":"2024-03-14T09:52:41.141029+0000","last_fullsized":"2024-03-14T09:52:41.141029+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:49:07.932854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140932+0000","last_change":"2024-03-14T09:52:05.744174+0000","last_active":"2024-03-14T09:52:41.140932+0000","last_peered":"2024-03-14T09:52:41.140932+0000","last_clean":"2024-03-14T09:52:41.140932+0000","last_became_active":"2024-03-14T09:52:05.743848+0000","last_became_peered":"2024-03-14T09:52:05.743848+0000","last_unstale":"2024-03-14T09:52:41.140932+0000","last_undegraded":"2024-03-14T09:52:41.140932+0000","last_fullsized":"2024-03-14T09:52:41.140932+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:07:23.260734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143348+0000","last_change":"2024-03-14T09:52:05.733677+0000","last_active":"2024-03-14T09:52:41.143348+0000","last_peered":"2024-03-14T09:52:41.143348+0000","last_clean":"2024-03-14T09:52:41.143348+0000","last_became_active":"2024-03-14T09:52:05.730967+0000","last_became_peered":"2024-03-14T09:52:05.730967+0000","last_unstale":"2024-03-14T09:52:41.143348+0000","last_undegraded":"2024-03-14T09:52:41.143348+0000","last_fullsized":"2024-03-14T09:52:41.143348+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:33:53.484741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141583+0000","last_change":"2024-03-14T09:52:05.737836+0000","last_active":"2024-03-14T09:52:41.141583+0000","last_peered":"2024-03-14T09:52:41.141583+0000","last_clean":"2024-03-14T09:52:41.141583+0000","last_became_active":"2024-03-14T09:52:05.737257+0000","last_became_peered":"2024-03-14T09:52:05.737257+0000","last_unstale":"2024-03-14T09:52:41.141583+0000","last_undegraded":"2024-03-14T09:52:41.141583+0000","last_fullsized":"2024-03-14T09:52:41.141583+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:53.723415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142940+0000","last_change":"2024-03-14T09:52:05.741360+0000","last_active":"2024-03-14T09:52:41.142940+0000","last_peered":"2024-03-14T09:52:41.142940+0000","last_clean":"2024-03-14T09:52:41.142940+0000","last_became_active":"2024-03-14T09:52:05.740673+0000","last_became_peered":"2024-03-14T09:52:05.740673+0000","last_unstale":"2024-03-14T09:52:41.142940+0000","last_undegraded":"2024-03-14T09:52:41.142940+0000","last_fullsized":"2024-03-14T09:52:41.142940+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:13:41.561338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143917+0000","last_change":"2024-03-14T09:52:05.739134+0000","last_active":"2024-03-14T09:52:41.143917+0000","last_peered":"2024-03-14T09:52:41.143917+0000","last_clean":"2024-03-14T09:52:41.143917+0000","last_became_active":"2024-03-14T09:52:05.738439+0000","last_became_peered":"2024-03-14T09:52:05.738439+0000","last_unstale":"2024-03-14T09:52:41.143917+0000","last_undegraded":"2024-03-14T09:52:41.143917+0000","last_fullsized":"2024-03-14T09:52:41.143917+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:05:36.997462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142821+0000","last_change":"2024-03-14T09:52:09.769284+0000","last_active":"2024-03-14T09:52:41.142821+0000","last_peered":"2024-03-14T09:52:41.142821+0000","last_clean":"2024-03-14T09:52:41.142821+0000","last_became_active":"2024-03-14T09:52:09.768808+0000","last_became_peered":"2024-03-14T09:52:09.768808+0000","last_unstale":"2024-03-14T09:52:41.142821+0000","last_undegraded":"2024-03-14T09:52:41.142821+0000","last_fullsized":"2024-03-14T09:52:41.142821+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:48:56.015410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"83'9","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.154303+0000","last_change":"2024-03-14T09:52:42.154303+0000","last_active":"2024-03-14T09:52:42.154303+0000","last_peered":"2024-03-14T09:52:42.154303+0000","last_clean":"2024-03-14T09:52:42.154303+0000","last_became_active":"2024-03-14T09:52:42.147756+0000","last_became_peered":"2024-03-14T09:52:42.147756+0000","last_unstale":"2024-03-14T09:52:42.154303+0000","last_undegraded":"2024-03-14T09:52:42.154303+0000","last_fullsized":"2024-03-14T09:52:42.154303+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:01:27.630337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145550+0000","last_change":"2024-03-14T09:52:05.739004+0000","last_active":"2024-03-14T09:52:41.145550+0000","last_peered":"2024-03-14T09:52:41.145550+0000","last_clean":"2024-03-14T09:52:41.145550+0000","last_became_active":"2024-03-14T09:52:05.732217+0000","last_became_peered":"2024-03-14T09:52:05.732217+0000","last_unstale":"2024-03-14T09:52:41.145550+0000","last_undegraded":"2024-03-14T09:52:41.145550+0000","last_fullsized":"2024-03-14T09:52:41.145550+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:02:21.926623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140568+0000","last_change":"2024-03-14T09:52:09.776128+0000","last_active":"2024-03-14T09:52:41.140568+0000","last_peered":"2024-03-14T09:52:41.140568+0000","last_clean":"2024-03-14T09:52:41.140568+0000","last_became_active":"2024-03-14T09:52:09.775701+0000","last_became_peered":"2024-03-14T09:52:09.775701+0000","last_unstale":"2024-03-14T09:52:41.140568+0000","last_undegraded":"2024-03-14T09:52:41.140568+0000","last_fullsized":"2024-03-14T09:52:41.140568+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:41:51.790627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"83'2","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146080+0000","last_change":"2024-03-14T09:52:05.234269+0000","last_active":"2024-03-14T09:52:41.146080+0000","last_peered":"2024-03-14T09:52:41.146080+0000","last_clean":"2024-03-14T09:52:41.146080+0000","last_became_active":"2024-03-14T09:52:05.233418+0000","last_became_peered":"2024-03-14T09:52:05.233418+0000","last_unstale":"2024-03-14T09:52:41.146080+0000","last_undegraded":"2024-03-14T09:52:41.146080+0000","last_fullsized":"2024-03-14T09:52:41.146080+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:22:46.894461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143057+0000","last_change":"2024-03-14T09:52:05.741786+0000","last_active":"2024-03-14T09:52:41.143057+0000","last_peered":"2024-03-14T09:52:41.143057+0000","last_clean":"2024-03-14T09:52:41.143057+0000","last_became_active":"2024-03-14T09:52:05.741461+0000","last_became_peered":"2024-03-14T09:52:05.741461+0000","last_unstale":"2024-03-14T09:52:41.143057+0000","last_undegraded":"2024-03-14T09:52:41.143057+0000","last_fullsized":"2024-03-14T09:52:41.143057+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:21:17.409530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141735+0000","last_change":"2024-03-14T09:52:09.768851+0000","last_active":"2024-03-14T09:52:41.141735+0000","last_peered":"2024-03-14T09:52:41.141735+0000","last_clean":"2024-03-14T09:52:41.141735+0000","last_became_active":"2024-03-14T09:52:09.768485+0000","last_became_peered":"2024-03-14T09:52:09.768485+0000","last_unstale":"2024-03-14T09:52:41.141735+0000","last_undegraded":"2024-03-14T09:52:41.141735+0000","last_fullsized":"2024-03-14T09:52:41.141735+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:00:51.189790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"83'4","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.154227+0000","last_change":"2024-03-14T09:52:42.154227+0000","last_active":"2024-03-14T09:52:42.154227+0000","last_peered":"2024-03-14T09:52:42.154227+0000","last_clean":"2024-03-14T09:52:42.154227+0000","last_became_active":"2024-03-14T09:52:42.149841+0000","last_became_peered":"2024-03-14T09:52:42.149841+0000","last_unstale":"2024-03-14T09:52:42.154227+0000","last_undegraded":"2024-03-14T09:52:42.154227+0000","last_fullsized":"2024-03-14T09:52:42.154227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:04:46.911921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.144336+0000","last_change":"2024-03-14T09:52:05.742994+0000","last_active":"2024-03-14T09:52:41.144336+0000","last_peered":"2024-03-14T09:52:41.144336+0000","last_clean":"2024-03-14T09:52:41.144336+0000","last_became_active":"2024-03-14T09:52:05.742652+0000","last_became_peered":"2024-03-14T09:52:05.742652+0000","last_unstale":"2024-03-14T09:52:41.144336+0000","last_undegraded":"2024-03-14T09:52:41.144336+0000","last_fullsized":"2024-03-14T09:52:41.144336+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:46:47.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140467+0000","last_change":"2024-03-14T09:52:09.763009+0000","last_active":"2024-03-14T09:52:41.140467+0000","last_peered":"2024-03-14T09:52:41.140467+0000","last_clean":"2024-03-14T09:52:41.140467+0000","last_became_active":"2024-03-14T09:52:09.762621+0000","last_became_peered":"2024-03-14T09:52:09.762621+0000","last_unstale":"2024-03-14T09:52:41.140467+0000","last_undegraded":"2024-03-14T09:52:41.140467+0000","last_fullsized":"2024-03-14T09:52:41.140467+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:56:22.162409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137429+0000","last_change":"2024-03-14T09:52:05.234039+0000","last_active":"2024-03-14T09:52:41.137429+0000","last_peered":"2024-03-14T09:52:41.137429+0000","last_clean":"2024-03-14T09:52:41.137429+0000","last_became_active":"2024-03-14T09:52:05.233612+0000","last_became_peered":"2024-03-14T09:52:05.233612+0000","last_unstale":"2024-03-14T09:52:41.137429+0000","last_undegraded":"2024-03-14T09:52:41.137429+0000","last_fullsized":"2024-03-14T09:52:41.137429+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:29:52.968178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.426506+0000","last_change":"2024-03-14T09:52:42.426506+0000","last_active":"2024-03-14T09:52:42.426506+0000","last_peered":"2024-03-14T09:52:42.426506+0000","last_clean":"2024-03-14T09:52:42.426506+0000","last_became_active":"2024-03-14T09:52:42.426062+0000","last_became_peered":"2024-03-14T09:52:42.426062+0000","last_unstale":"2024-03-14T09:52:42.426506+0000","last_undegraded":"2024-03-14T09:52:42.426506+0000","last_fullsized":"2024-03-14T09:52:42.426506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:23:42.943381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143755+0000","last_change":"2024-03-14T09:52:09.757856+0000","last_active":"2024-03-14T09:52:41.143755+0000","last_peered":"2024-03-14T09:52:41.143755+0000","last_clean":"2024-03-14T09:52:41.143755+0000","last_became_active":"2024-03-14T09:52:09.757395+0000","last_became_peered":"2024-03-14T09:52:09.757395+0000","last_unstale":"2024-03-14T09:52:41.143755+0000","last_undegraded":"2024-03-14T09:52:41.143755+0000","last_fullsized":"2024-03-14T09:52:41.143755+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:34:06.600727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"83'5","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:53:05.776205+0000","last_change":"2024-03-14T09:52:05.228527+0000","last_active":"2024-03-14T09:53:05.776205+0000","last_peered":"2024-03-14T09:53:05.776205+0000","last_clean":"2024-03-14T09:53:05.776205+0000","last_became_active":"2024-03-14T09:52:05.228177+0000","last_became_peered":"2024-03-14T09:52:05.228177+0000","last_unstale":"2024-03-14T09:53:05.776205+0000","last_undegraded":"2024-03-14T09:53:05.776205+0000","last_fullsized":"2024-03-14T09:53:05.776205+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:17:57.529308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.138384+0000","last_change":"2024-03-14T09:52:05.734041+0000","last_active":"2024-03-14T09:52:41.138384+0000","last_peered":"2024-03-14T09:52:41.138384+0000","last_clean":"2024-03-14T09:52:41.138384+0000","last_became_active":"2024-03-14T09:52:05.732491+0000","last_became_peered":"2024-03-14T09:52:05.732491+0000","last_unstale":"2024-03-14T09:52:41.138384+0000","last_undegraded":"2024-03-14T09:52:41.138384+0000","last_fullsized":"2024-03-14T09:52:41.138384+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:07:31.909382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143044+0000","last_change":"2024-03-14T09:52:09.769584+0000","last_active":"2024-03-14T09:52:41.143044+0000","last_peered":"2024-03-14T09:52:41.143044+0000","last_clean":"2024-03-14T09:52:41.143044+0000","last_became_active":"2024-03-14T09:52:09.768847+0000","last_became_peered":"2024-03-14T09:52:09.768847+0000","last_unstale":"2024-03-14T09:52:41.143044+0000","last_undegraded":"2024-03-14T09:52:41.143044+0000","last_fullsized":"2024-03-14T09:52:41.143044+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:12:18.005122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"83'3","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142733+0000","last_change":"2024-03-14T09:52:05.228893+0000","last_active":"2024-03-14T09:52:41.142733+0000","last_peered":"2024-03-14T09:52:41.142733+0000","last_clean":"2024-03-14T09:52:41.142733+0000","last_became_active":"2024-03-14T09:52:05.228569+0000","last_became_peered":"2024-03-14T09:52:05.228569+0000","last_unstale":"2024-03-14T09:52:41.142733+0000","last_undegraded":"2024-03-14T09:52:41.142733+0000","last_fullsized":"2024-03-14T09:52:41.142733+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:28:03.200437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146405+0000","last_change":"2024-03-14T09:52:05.736388+0000","last_active":"2024-03-14T09:52:41.146405+0000","last_peered":"2024-03-14T09:52:41.146405+0000","last_clean":"2024-03-14T09:52:41.146405+0000","last_became_active":"2024-03-14T09:52:05.735131+0000","last_became_peered":"2024-03-14T09:52:05.735131+0000","last_unstale":"2024-03-14T09:52:41.146405+0000","last_undegraded":"2024-03-14T09:52:41.146405+0000","last_fullsized":"2024-03-14T09:52:41.146405+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:00:54.120087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141484+0000","last_change":"2024-03-14T09:52:09.770321+0000","last_active":"2024-03-14T09:52:41.141484+0000","last_peered":"2024-03-14T09:52:41.141484+0000","last_clean":"2024-03-14T09:52:41.141484+0000","last_became_active":"2024-03-14T09:52:09.769710+0000","last_became_peered":"2024-03-14T09:52:09.769710+0000","last_unstale":"2024-03-14T09:52:41.141484+0000","last_undegraded":"2024-03-14T09:52:41.141484+0000","last_fullsized":"2024-03-14T09:52:41.141484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:12:12.635248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"83'2","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136475+0000","last_change":"2024-03-14T09:52:05.223912+0000","last_active":"2024-03-14T09:52:41.136475+0000","last_peered":"2024-03-14T09:52:41.136475+0000","last_clean":"2024-03-14T09:52:41.136475+0000","last_became_active":"2024-03-14T09:52:05.223380+0000","last_became_peered":"2024-03-14T09:52:05.223380+0000","last_unstale":"2024-03-14T09:52:41.136475+0000","last_undegraded":"2024-03-14T09:52:41.136475+0000","last_fullsized":"2024-03-14T09:52:41.136475+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:08:33.098215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145370+0000","last_change":"2024-03-14T09:52:05.743277+0000","last_active":"2024-03-14T09:52:41.145370+0000","last_peered":"2024-03-14T09:52:41.145370+0000","last_clean":"2024-03-14T09:52:41.145370+0000","last_became_active":"2024-03-14T09:52:05.741386+0000","last_became_peered":"2024-03-14T09:52:05.741386+0000","last_unstale":"2024-03-14T09:52:41.145370+0000","last_undegraded":"2024-03-14T09:52:41.145370+0000","last_fullsized":"2024-03-14T09:52:41.145370+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:08:08.320279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141726+0000","last_change":"2024-03-14T09:52:09.766485+0000","last_active":"2024-03-14T09:52:41.141726+0000","last_peered":"2024-03-14T09:52:41.141726+0000","last_clean":"2024-03-14T09:52:41.141726+0000","last_became_active":"2024-03-14T09:52:09.766115+0000","last_became_peered":"2024-03-14T09:52:09.766115+0000","last_unstale":"2024-03-14T09:52:41.141726+0000","last_undegraded":"2024-03-14T09:52:41.141726+0000","last_fullsized":"2024-03-14T09:52:41.141726+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:34:45.067757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"83'1","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145976+0000","last_change":"2024-03-14T09:52:05.233852+0000","last_active":"2024-03-14T09:52:41.145976+0000","last_peered":"2024-03-14T09:52:41.145976+0000","last_clean":"2024-03-14T09:52:41.145976+0000","last_became_active":"2024-03-14T09:52:05.233548+0000","last_became_peered":"2024-03-14T09:52:05.233548+0000","last_unstale":"2024-03-14T09:52:41.145976+0000","last_undegraded":"2024-03-14T09:52:41.145976+0000","last_fullsized":"2024-03-14T09:52:41.145976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:08:51.821465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141954+0000","last_change":"2024-03-14T09:52:05.731215+0000","last_active":"2024-03-14T09:52:41.141954+0000","last_peered":"2024-03-14T09:52:41.141954+0000","last_clean":"2024-03-14T09:52:41.141954+0000","last_became_active":"2024-03-14T09:52:05.730924+0000","last_became_peered":"2024-03-14T09:52:05.730924+0000","last_unstale":"2024-03-14T09:52:41.141954+0000","last_undegraded":"2024-03-14T09:52:41.141954+0000","last_fullsized":"2024-03-14T09:52:41.141954+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:12:17.560297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145920+0000","last_change":"2024-03-14T09:52:09.753775+0000","last_active":"2024-03-14T09:52:41.145920+0000","last_peered":"2024-03-14T09:52:41.145920+0000","last_clean":"2024-03-14T09:52:41.145920+0000","last_became_active":"2024-03-14T09:52:09.753422+0000","last_became_peered":"2024-03-14T09:52:09.753422+0000","last_unstale":"2024-03-14T09:52:41.145920+0000","last_undegraded":"2024-03-14T09:52:41.145920+0000","last_fullsized":"2024-03-14T09:52:41.145920+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:59:25.625909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"83'3","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:53:06.776870+0000","last_change":"2024-03-14T09:52:05.229064+0000","last_active":"2024-03-14T09:53:06.776870+0000","last_peered":"2024-03-14T09:53:06.776870+0000","last_clean":"2024-03-14T09:53:06.776870+0000","last_became_active":"2024-03-14T09:52:05.228751+0000","last_became_peered":"2024-03-14T09:52:05.228751+0000","last_unstale":"2024-03-14T09:53:06.776870+0000","last_undegraded":"2024-03-14T09:53:06.776870+0000","last_fullsized":"2024-03-14T09:53:06.776870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:13:29.832866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145697+0000","last_change":"2024-03-14T09:52:05.737798+0000","last_active":"2024-03-14T09:52:41.145697+0000","last_peered":"2024-03-14T09:52:41.145697+0000","last_clean":"2024-03-14T09:52:41.145697+0000","last_became_active":"2024-03-14T09:52:05.737449+0000","last_became_peered":"2024-03-14T09:52:05.737449+0000","last_unstale":"2024-03-14T09:52:41.145697+0000","last_undegraded":"2024-03-14T09:52:41.145697+0000","last_fullsized":"2024-03-14T09:52:41.145697+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:40:10.079770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141907+0000","last_change":"2024-03-14T09:52:09.768368+0000","last_active":"2024-03-14T09:52:41.141907+0000","last_peered":"2024-03-14T09:52:41.141907+0000","last_clean":"2024-03-14T09:52:41.141907+0000","last_became_active":"2024-03-14T09:52:09.767622+0000","last_became_peered":"2024-03-14T09:52:09.767622+0000","last_unstale":"2024-03-14T09:52:41.141907+0000","last_undegraded":"2024-03-14T09:52:41.141907+0000","last_fullsized":"2024-03-14T09:52:41.141907+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:19:28.836317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"83'2","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137100+0000","last_change":"2024-03-14T09:52:05.229778+0000","last_active":"2024-03-14T09:52:41.137100+0000","last_peered":"2024-03-14T09:52:41.137100+0000","last_clean":"2024-03-14T09:52:41.137100+0000","last_became_active":"2024-03-14T09:52:05.229468+0000","last_became_peered":"2024-03-14T09:52:05.229468+0000","last_unstale":"2024-03-14T09:52:41.137100+0000","last_undegraded":"2024-03-14T09:52:41.137100+0000","last_fullsized":"2024-03-14T09:52:41.137100+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:49:58.433646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143199+0000","last_change":"2024-03-14T09:52:05.751027+0000","last_active":"2024-03-14T09:52:41.143199+0000","last_peered":"2024-03-14T09:52:41.143199+0000","last_clean":"2024-03-14T09:52:41.143199+0000","last_became_active":"2024-03-14T09:52:05.750650+0000","last_became_peered":"2024-03-14T09:52:05.750650+0000","last_unstale":"2024-03-14T09:52:41.143199+0000","last_undegraded":"2024-03-14T09:52:41.143199+0000","last_fullsized":"2024-03-14T09:52:41.143199+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:05:37.264396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141928+0000","last_change":"2024-03-14T09:52:09.768207+0000","last_active":"2024-03-14T09:52:41.141928+0000","last_peered":"2024-03-14T09:52:41.141928+0000","last_clean":"2024-03-14T09:52:41.141928+0000","last_became_active":"2024-03-14T09:52:09.767756+0000","last_became_peered":"2024-03-14T09:52:09.767756+0000","last_unstale":"2024-03-14T09:52:41.141928+0000","last_undegraded":"2024-03-14T09:52:41.141928+0000","last_fullsized":"2024-03-14T09:52:41.141928+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:13:12.544995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"83'4","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146023+0000","last_change":"2024-03-14T09:52:05.235472+0000","last_active":"2024-03-14T09:52:41.146023+0000","last_peered":"2024-03-14T09:52:41.146023+0000","last_clean":"2024-03-14T09:52:41.146023+0000","last_became_active":"2024-03-14T09:52:05.235106+0000","last_became_peered":"2024-03-14T09:52:05.235106+0000","last_unstale":"2024-03-14T09:52:41.146023+0000","last_undegraded":"2024-03-14T09:52:41.146023+0000","last_fullsized":"2024-03-14T09:52:41.146023+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:30:36.446130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.144505+0000","last_change":"2024-03-14T09:52:05.747615+0000","last_active":"2024-03-14T09:52:41.144505+0000","last_peered":"2024-03-14T09:52:41.144505+0000","last_clean":"2024-03-14T09:52:41.144505+0000","last_became_active":"2024-03-14T09:52:05.747344+0000","last_became_peered":"2024-03-14T09:52:05.747344+0000","last_unstale":"2024-03-14T09:52:41.144505+0000","last_undegraded":"2024-03-14T09:52:41.144505+0000","last_fullsized":"2024-03-14T09:52:41.144505+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:52:23.096124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141847+0000","last_change":"2024-03-14T09:52:09.768295+0000","last_active":"2024-03-14T09:52:41.141847+0000","last_peered":"2024-03-14T09:52:41.141847+0000","last_clean":"2024-03-14T09:52:41.141847+0000","last_became_active":"2024-03-14T09:52:09.760997+0000","last_became_peered":"2024-03-14T09:52:09.760997+0000","last_unstale":"2024-03-14T09:52:41.141847+0000","last_undegraded":"2024-03-14T09:52:41.141847+0000","last_fullsized":"2024-03-14T09:52:41.141847+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:22:09.909302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'31","reported_seq":132,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:53:07.788981+0000","last_change":"2024-03-14T09:52:05.227876+0000","last_active":"2024-03-14T09:53:07.788981+0000","last_peered":"2024-03-14T09:53:07.788981+0000","last_clean":"2024-03-14T09:53:07.788981+0000","last_became_active":"2024-03-14T09:52:05.227519+0000","last_became_peered":"2024-03-14T09:52:05.227519+0000","last_unstale":"2024-03-14T09:53:07.788981+0000","last_undegraded":"2024-03-14T09:53:07.788981+0000","last_fullsized":"2024-03-14T09:53:07.788981+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:12.616609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61729,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":83,"num_read_kb":102,"num_write":32,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137730+0000","last_change":"2024-03-14T09:52:05.737432+0000","last_active":"2024-03-14T09:52:41.137730+0000","last_peered":"2024-03-14T09:52:41.137730+0000","last_clean":"2024-03-14T09:52:41.137730+0000","last_became_active":"2024-03-14T09:52:05.735050+0000","last_became_peered":"2024-03-14T09:52:05.735050+0000","last_unstale":"2024-03-14T09:52:41.137730+0000","last_undegraded":"2024-03-14T09:52:41.137730+0000","last_fullsized":"2024-03-14T09:52:41.137730+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T09:57:45.383155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146516+0000","last_change":"2024-03-14T09:52:09.754909+0000","last_active":"2024-03-14T09:52:41.146516+0000","last_peered":"2024-03-14T09:52:41.146516+0000","last_clean":"2024-03-14T09:52:41.146516+0000","last_became_active":"2024-03-14T09:52:09.754367+0000","last_became_peered":"2024-03-14T09:52:09.754367+0000","last_unstale":"2024-03-14T09:52:41.146516+0000","last_undegraded":"2024-03-14T09:52:41.146516+0000","last_fullsized":"2024-03-14T09:52:41.146516+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:52:34.251555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"83'6","reported_seq":25,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.144607+0000","last_change":"2024-03-14T09:52:05.224869+0000","last_active":"2024-03-14T09:52:41.144607+0000","last_peered":"2024-03-14T09:52:41.144607+0000","last_clean":"2024-03-14T09:52:41.144607+0000","last_became_active":"2024-03-14T09:52:05.224499+0000","last_became_peered":"2024-03-14T09:52:05.224499+0000","last_unstale":"2024-03-14T09:52:41.144607+0000","last_undegraded":"2024-03-14T09:52:41.144607+0000","last_fullsized":"2024-03-14T09:52:41.144607+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:49:01.729417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136754+0000","last_change":"2024-03-14T09:52:05.729168+0000","last_active":"2024-03-14T09:52:41.136754+0000","last_peered":"2024-03-14T09:52:41.136754+0000","last_clean":"2024-03-14T09:52:41.136754+0000","last_became_active":"2024-03-14T09:52:05.728806+0000","last_became_peered":"2024-03-14T09:52:05.728806+0000","last_unstale":"2024-03-14T09:52:41.136754+0000","last_undegraded":"2024-03-14T09:52:41.136754+0000","last_fullsized":"2024-03-14T09:52:41.136754+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:54:36.758853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140504+0000","last_change":"2024-03-14T09:52:09.770126+0000","last_active":"2024-03-14T09:52:41.140504+0000","last_peered":"2024-03-14T09:52:41.140504+0000","last_clean":"2024-03-14T09:52:41.140504+0000","last_became_active":"2024-03-14T09:52:09.769734+0000","last_became_peered":"2024-03-14T09:52:09.769734+0000","last_unstale":"2024-03-14T09:52:41.140504+0000","last_undegraded":"2024-03-14T09:52:41.140504+0000","last_fullsized":"2024-03-14T09:52:41.140504+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:50:37.839858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"83'3","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146334+0000","last_change":"2024-03-14T09:52:05.235201+0000","last_active":"2024-03-14T09:52:41.146334+0000","last_peered":"2024-03-14T09:52:41.146334+0000","last_clean":"2024-03-14T09:52:41.146334+0000","last_became_active":"2024-03-14T09:52:05.234816+0000","last_became_peered":"2024-03-14T09:52:05.234816+0000","last_unstale":"2024-03-14T09:52:41.146334+0000","last_undegraded":"2024-03-14T09:52:41.146334+0000","last_fullsized":"2024-03-14T09:52:41.146334+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:01:50.070913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137236+0000","last_change":"2024-03-14T09:52:05.735304+0000","last_active":"2024-03-14T09:52:41.137236+0000","last_peered":"2024-03-14T09:52:41.137236+0000","last_clean":"2024-03-14T09:52:41.137236+0000","last_became_active":"2024-03-14T09:52:05.734977+0000","last_became_peered":"2024-03-14T09:52:05.734977+0000","last_unstale":"2024-03-14T09:52:41.137236+0000","last_undegraded":"2024-03-14T09:52:41.137236+0000","last_fullsized":"2024-03-14T09:52:41.137236+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:08:28.071962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142729+0000","last_change":"2024-03-14T09:52:09.770721+0000","last_active":"2024-03-14T09:52:41.142729+0000","last_peered":"2024-03-14T09:52:41.142729+0000","last_clean":"2024-03-14T09:52:41.142729+0000","last_became_active":"2024-03-14T09:52:09.770270+0000","last_became_peered":"2024-03-14T09:52:09.770270+0000","last_unstale":"2024-03-14T09:52:41.142729+0000","last_undegraded":"2024-03-14T09:52:41.142729+0000","last_fullsized":"2024-03-14T09:52:41.142729+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:09:33.480235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"83'4","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:53:03.776074+0000","last_change":"2024-03-14T09:52:05.227260+0000","last_active":"2024-03-14T09:53:03.776074+0000","last_peered":"2024-03-14T09:53:03.776074+0000","last_clean":"2024-03-14T09:53:03.776074+0000","last_became_active":"2024-03-14T09:52:05.226922+0000","last_became_peered":"2024-03-14T09:52:05.226922+0000","last_unstale":"2024-03-14T09:53:03.776074+0000","last_undegraded":"2024-03-14T09:53:03.776074+0000","last_fullsized":"2024-03-14T09:53:03.776074+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:36:09.770478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":932,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137037+0000","last_change":"2024-03-14T09:52:05.734278+0000","last_active":"2024-03-14T09:52:41.137037+0000","last_peered":"2024-03-14T09:52:41.137037+0000","last_clean":"2024-03-14T09:52:41.137037+0000","last_became_active":"2024-03-14T09:52:05.732051+0000","last_became_peered":"2024-03-14T09:52:05.732051+0000","last_unstale":"2024-03-14T09:52:41.137037+0000","last_undegraded":"2024-03-14T09:52:41.137037+0000","last_fullsized":"2024-03-14T09:52:41.137037+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:33:07.137983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136993+0000","last_change":"2024-03-14T09:52:09.764068+0000","last_active":"2024-03-14T09:52:41.136993+0000","last_peered":"2024-03-14T09:52:41.136993+0000","last_clean":"2024-03-14T09:52:41.136993+0000","last_became_active":"2024-03-14T09:52:09.763613+0000","last_became_peered":"2024-03-14T09:52:09.763613+0000","last_unstale":"2024-03-14T09:52:41.136993+0000","last_undegraded":"2024-03-14T09:52:41.136993+0000","last_fullsized":"2024-03-14T09:52:41.136993+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:32:15.524602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"83'15","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.438205+0000","last_change":"2024-03-14T09:52:42.438205+0000","last_active":"2024-03-14T09:52:42.438205+0000","last_peered":"2024-03-14T09:52:42.438205+0000","last_clean":"2024-03-14T09:52:42.438205+0000","last_became_active":"2024-03-14T09:52:42.426268+0000","last_became_peered":"2024-03-14T09:52:42.426268+0000","last_unstale":"2024-03-14T09:52:42.438205+0000","last_undegraded":"2024-03-14T09:52:42.438205+0000","last_fullsized":"2024-03-14T09:52:42.438205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:10:09.651131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145909+0000","last_change":"2024-03-14T09:52:05.734366+0000","last_active":"2024-03-14T09:52:41.145909+0000","last_peered":"2024-03-14T09:52:41.145909+0000","last_clean":"2024-03-14T09:52:41.145909+0000","last_became_active":"2024-03-14T09:52:05.734058+0000","last_became_peered":"2024-03-14T09:52:05.734058+0000","last_unstale":"2024-03-14T09:52:41.145909+0000","last_undegraded":"2024-03-14T09:52:41.145909+0000","last_fullsized":"2024-03-14T09:52:41.145909+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:49:49.233683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145893+0000","last_change":"2024-03-14T09:52:09.760241+0000","last_active":"2024-03-14T09:52:41.145893+0000","last_peered":"2024-03-14T09:52:41.145893+0000","last_clean":"2024-03-14T09:52:41.145893+0000","last_became_active":"2024-03-14T09:52:09.759797+0000","last_became_peered":"2024-03-14T09:52:09.759797+0000","last_unstale":"2024-03-14T09:52:41.145893+0000","last_undegraded":"2024-03-14T09:52:41.145893+0000","last_fullsized":"2024-03-14T09:52:41.145893+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:50:33.035775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"83'2","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.138494+0000","last_change":"2024-03-14T09:52:05.232932+0000","last_active":"2024-03-14T09:52:41.138494+0000","last_peered":"2024-03-14T09:52:41.138494+0000","last_clean":"2024-03-14T09:52:41.138494+0000","last_became_active":"2024-03-14T09:52:05.232608+0000","last_became_peered":"2024-03-14T09:52:05.232608+0000","last_unstale":"2024-03-14T09:52:41.138494+0000","last_undegraded":"2024-03-14T09:52:41.138494+0000","last_fullsized":"2024-03-14T09:52:41.138494+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:43:59.613531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141511+0000","last_change":"2024-03-14T09:52:05.733762+0000","last_active":"2024-03-14T09:52:41.141511+0000","last_peered":"2024-03-14T09:52:41.141511+0000","last_clean":"2024-03-14T09:52:41.141511+0000","last_became_active":"2024-03-14T09:52:05.733417+0000","last_became_peered":"2024-03-14T09:52:05.733417+0000","last_unstale":"2024-03-14T09:52:41.141511+0000","last_undegraded":"2024-03-14T09:52:41.141511+0000","last_fullsized":"2024-03-14T09:52:41.141511+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:11:12.593952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.426896+0000","last_change":"2024-03-14T09:52:42.426896+0000","last_active":"2024-03-14T09:52:42.426896+0000","last_peered":"2024-03-14T09:52:42.426896+0000","last_clean":"2024-03-14T09:52:42.426896+0000","last_became_active":"2024-03-14T09:52:42.426287+0000","last_became_peered":"2024-03-14T09:52:42.426287+0000","last_unstale":"2024-03-14T09:52:42.426896+0000","last_undegraded":"2024-03-14T09:52:42.426896+0000","last_fullsized":"2024-03-14T09:52:42.426896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:53:16.523707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142321+0000","last_change":"2024-03-14T09:52:05.227636+0000","last_active":"2024-03-14T09:52:41.142321+0000","last_peered":"2024-03-14T09:52:41.142321+0000","last_clean":"2024-03-14T09:52:41.142321+0000","last_became_active":"2024-03-14T09:52:05.227274+0000","last_became_peered":"2024-03-14T09:52:05.227274+0000","last_unstale":"2024-03-14T09:52:41.142321+0000","last_undegraded":"2024-03-14T09:52:41.142321+0000","last_fullsized":"2024-03-14T09:52:41.142321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:13:25.380090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142838+0000","last_change":"2024-03-14T09:52:05.733158+0000","last_active":"2024-03-14T09:52:41.142838+0000","last_peered":"2024-03-14T09:52:41.142838+0000","last_clean":"2024-03-14T09:52:41.142838+0000","last_became_active":"2024-03-14T09:52:05.732812+0000","last_became_peered":"2024-03-14T09:52:05.732812+0000","last_unstale":"2024-03-14T09:52:41.142838+0000","last_undegraded":"2024-03-14T09:52:41.142838+0000","last_fullsized":"2024-03-14T09:52:41.142838+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:01:49.154860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136641+0000","last_change":"2024-03-14T09:52:09.761596+0000","last_active":"2024-03-14T09:52:41.136641+0000","last_peered":"2024-03-14T09:52:41.136641+0000","last_clean":"2024-03-14T09:52:41.136641+0000","last_became_active":"2024-03-14T09:52:09.761184+0000","last_became_peered":"2024-03-14T09:52:09.761184+0000","last_unstale":"2024-03-14T09:52:41.136641+0000","last_undegraded":"2024-03-14T09:52:41.136641+0000","last_fullsized":"2024-03-14T09:52:41.136641+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:49:02.567472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"83'1","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136845+0000","last_change":"2024-03-14T09:52:05.231828+0000","last_active":"2024-03-14T09:52:41.136845+0000","last_peered":"2024-03-14T09:52:41.136845+0000","last_clean":"2024-03-14T09:52:41.136845+0000","last_became_active":"2024-03-14T09:52:05.231490+0000","last_became_peered":"2024-03-14T09:52:05.231490+0000","last_unstale":"2024-03-14T09:52:41.136845+0000","last_undegraded":"2024-03-14T09:52:41.136845+0000","last_fullsized":"2024-03-14T09:52:41.136845+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:24:53.899728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142678+0000","last_change":"2024-03-14T09:52:05.744646+0000","last_active":"2024-03-14T09:52:41.142678+0000","last_peered":"2024-03-14T09:52:41.142678+0000","last_clean":"2024-03-14T09:52:41.142678+0000","last_became_active":"2024-03-14T09:52:05.744327+0000","last_became_peered":"2024-03-14T09:52:05.744327+0000","last_unstale":"2024-03-14T09:52:41.142678+0000","last_undegraded":"2024-03-14T09:52:41.142678+0000","last_fullsized":"2024-03-14T09:52:41.142678+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:46:50.643350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137005+0000","last_change":"2024-03-14T09:52:09.765374+0000","last_active":"2024-03-14T09:52:41.137005+0000","last_peered":"2024-03-14T09:52:41.137005+0000","last_clean":"2024-03-14T09:52:41.137005+0000","last_became_active":"2024-03-14T09:52:09.764939+0000","last_became_peered":"2024-03-14T09:52:09.764939+0000","last_unstale":"2024-03-14T09:52:41.137005+0000","last_undegraded":"2024-03-14T09:52:41.137005+0000","last_fullsized":"2024-03-14T09:52:41.137005+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:08:10.688746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146095+0000","last_change":"2024-03-14T09:52:05.729076+0000","last_active":"2024-03-14T09:52:41.146095+0000","last_peered":"2024-03-14T09:52:41.146095+0000","last_clean":"2024-03-14T09:52:41.146095+0000","last_became_active":"2024-03-14T09:52:05.728711+0000","last_became_peered":"2024-03-14T09:52:05.728711+0000","last_unstale":"2024-03-14T09:52:41.146095+0000","last_undegraded":"2024-03-14T09:52:41.146095+0000","last_fullsized":"2024-03-14T09:52:41.146095+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:11:05.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"83'4","reported_seq":57,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:53:03.775639+0000","last_change":"2024-03-14T09:52:05.236209+0000","last_active":"2024-03-14T09:53:03.775639+0000","last_peered":"2024-03-14T09:53:03.775639+0000","last_clean":"2024-03-14T09:53:03.775639+0000","last_became_active":"2024-03-14T09:52:05.235517+0000","last_became_peered":"2024-03-14T09:52:05.235517+0000","last_unstale":"2024-03-14T09:53:03.775639+0000","last_undegraded":"2024-03-14T09:53:03.775639+0000","last_fullsized":"2024-03-14T09:53:03.775639+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:09:54.275728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137171+0000","last_change":"2024-03-14T09:52:09.761095+0000","last_active":"2024-03-14T09:52:41.137171+0000","last_peered":"2024-03-14T09:52:41.137171+0000","last_clean":"2024-03-14T09:52:41.137171+0000","last_became_active":"2024-03-14T09:52:09.760662+0000","last_became_peered":"2024-03-14T09:52:09.760662+0000","last_unstale":"2024-03-14T09:52:41.137171+0000","last_undegraded":"2024-03-14T09:52:41.137171+0000","last_fullsized":"2024-03-14T09:52:41.137171+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:07:17.670605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145420+0000","last_change":"2024-03-14T09:52:05.736078+0000","last_active":"2024-03-14T09:52:41.145420+0000","last_peered":"2024-03-14T09:52:41.145420+0000","last_clean":"2024-03-14T09:52:41.145420+0000","last_became_active":"2024-03-14T09:52:05.735715+0000","last_became_peered":"2024-03-14T09:52:05.735715+0000","last_unstale":"2024-03-14T09:52:41.145420+0000","last_undegraded":"2024-03-14T09:52:41.145420+0000","last_fullsized":"2024-03-14T09:52:41.145420+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:41:32.863680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"83'3","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141835+0000","last_change":"2024-03-14T09:52:05.225756+0000","last_active":"2024-03-14T09:52:41.141835+0000","last_peered":"2024-03-14T09:52:41.141835+0000","last_clean":"2024-03-14T09:52:41.141835+0000","last_became_active":"2024-03-14T09:52:05.225378+0000","last_became_peered":"2024-03-14T09:52:05.225378+0000","last_unstale":"2024-03-14T09:52:41.141835+0000","last_undegraded":"2024-03-14T09:52:41.141835+0000","last_fullsized":"2024-03-14T09:52:41.141835+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:14:49.594636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":337,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136821+0000","last_change":"2024-03-14T09:51:11.588790+0000","last_active":"2024-03-14T09:52:41.136821+0000","last_peered":"2024-03-14T09:52:41.136821+0000","last_clean":"2024-03-14T09:52:41.136821+0000","last_became_active":"2024-03-14T09:51:11.588287+0000","last_became_peered":"2024-03-14T09:51:11.588287+0000","last_unstale":"2024-03-14T09:52:41.136821+0000","last_undegraded":"2024-03-14T09:52:41.136821+0000","last_fullsized":"2024-03-14T09:52:41.136821+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:48:37.651174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:48:37.651174+0000","last_clean_scrub_stamp":"2024-03-14T09:48:37.651174+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:06:24.252565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146527+0000","last_change":"2024-03-14T09:52:09.761344+0000","last_active":"2024-03-14T09:52:41.146527+0000","last_peered":"2024-03-14T09:52:41.146527+0000","last_clean":"2024-03-14T09:52:41.146527+0000","last_became_active":"2024-03-14T09:52:09.760899+0000","last_became_peered":"2024-03-14T09:52:09.760899+0000","last_unstale":"2024-03-14T09:52:41.146527+0000","last_undegraded":"2024-03-14T09:52:41.146527+0000","last_fullsized":"2024-03-14T09:52:41.146527+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:24:26.212157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136377+0000","last_change":"2024-03-14T09:52:05.738735+0000","last_active":"2024-03-14T09:52:41.136377+0000","last_peered":"2024-03-14T09:52:41.136377+0000","last_clean":"2024-03-14T09:52:41.136377+0000","last_became_active":"2024-03-14T09:52:05.738447+0000","last_became_peered":"2024-03-14T09:52:05.738447+0000","last_unstale":"2024-03-14T09:52:41.136377+0000","last_undegraded":"2024-03-14T09:52:41.136377+0000","last_fullsized":"2024-03-14T09:52:41.136377+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:03:24.598291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'3","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137056+0000","last_change":"2024-03-14T09:52:05.230107+0000","last_active":"2024-03-14T09:52:41.137056+0000","last_peered":"2024-03-14T09:52:41.137056+0000","last_clean":"2024-03-14T09:52:41.137056+0000","last_became_active":"2024-03-14T09:52:05.229812+0000","last_became_peered":"2024-03-14T09:52:05.229812+0000","last_unstale":"2024-03-14T09:52:41.137056+0000","last_undegraded":"2024-03-14T09:52:41.137056+0000","last_fullsized":"2024-03-14T09:52:41.137056+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:09:44.884867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141204+0000","last_change":"2024-03-14T09:52:09.763424+0000","last_active":"2024-03-14T09:52:41.141204+0000","last_peered":"2024-03-14T09:52:41.141204+0000","last_clean":"2024-03-14T09:52:41.141204+0000","last_became_active":"2024-03-14T09:52:09.763028+0000","last_became_peered":"2024-03-14T09:52:09.763028+0000","last_unstale":"2024-03-14T09:52:41.141204+0000","last_undegraded":"2024-03-14T09:52:41.141204+0000","last_fullsized":"2024-03-14T09:52:41.141204+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:47:31.415788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"83'1","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137766+0000","last_change":"2024-03-14T09:52:05.235099+0000","last_active":"2024-03-14T09:52:41.137766+0000","last_peered":"2024-03-14T09:52:41.137766+0000","last_clean":"2024-03-14T09:52:41.137766+0000","last_became_active":"2024-03-14T09:52:05.234798+0000","last_became_peered":"2024-03-14T09:52:05.234798+0000","last_unstale":"2024-03-14T09:52:41.137766+0000","last_undegraded":"2024-03-14T09:52:41.137766+0000","last_fullsized":"2024-03-14T09:52:41.137766+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:00:07.461931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136781+0000","last_change":"2024-03-14T09:52:05.742736+0000","last_active":"2024-03-14T09:52:41.136781+0000","last_peered":"2024-03-14T09:52:41.136781+0000","last_clean":"2024-03-14T09:52:41.136781+0000","last_became_active":"2024-03-14T09:52:05.740927+0000","last_became_peered":"2024-03-14T09:52:05.740927+0000","last_unstale":"2024-03-14T09:52:41.136781+0000","last_undegraded":"2024-03-14T09:52:41.136781+0000","last_fullsized":"2024-03-14T09:52:41.136781+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:50:17.738763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145386+0000","last_change":"2024-03-14T09:52:09.762722+0000","last_active":"2024-03-14T09:52:41.145386+0000","last_peered":"2024-03-14T09:52:41.145386+0000","last_clean":"2024-03-14T09:52:41.145386+0000","last_became_active":"2024-03-14T09:52:09.759053+0000","last_became_peered":"2024-03-14T09:52:09.759053+0000","last_unstale":"2024-03-14T09:52:41.145386+0000","last_undegraded":"2024-03-14T09:52:41.145386+0000","last_fullsized":"2024-03-14T09:52:41.145386+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:40:29.680002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"83'5","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142908+0000","last_change":"2024-03-14T09:52:05.224778+0000","last_active":"2024-03-14T09:52:41.142908+0000","last_peered":"2024-03-14T09:52:41.142908+0000","last_clean":"2024-03-14T09:52:41.142908+0000","last_became_active":"2024-03-14T09:52:05.223158+0000","last_became_peered":"2024-03-14T09:52:05.223158+0000","last_unstale":"2024-03-14T09:52:41.142908+0000","last_undegraded":"2024-03-14T09:52:41.142908+0000","last_fullsized":"2024-03-14T09:52:41.142908+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:01:29.910099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146027+0000","last_change":"2024-03-14T09:52:05.741753+0000","last_active":"2024-03-14T09:52:41.146027+0000","last_peered":"2024-03-14T09:52:41.146027+0000","last_clean":"2024-03-14T09:52:41.146027+0000","last_became_active":"2024-03-14T09:52:05.741451+0000","last_became_peered":"2024-03-14T09:52:05.741451+0000","last_unstale":"2024-03-14T09:52:41.146027+0000","last_undegraded":"2024-03-14T09:52:41.146027+0000","last_fullsized":"2024-03-14T09:52:41.146027+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:42:59.920909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142537+0000","last_change":"2024-03-14T09:52:09.755884+0000","last_active":"2024-03-14T09:52:41.142537+0000","last_peered":"2024-03-14T09:52:41.142537+0000","last_clean":"2024-03-14T09:52:41.142537+0000","last_became_active":"2024-03-14T09:52:09.755474+0000","last_became_peered":"2024-03-14T09:52:09.755474+0000","last_unstale":"2024-03-14T09:52:41.142537+0000","last_undegraded":"2024-03-14T09:52:41.142537+0000","last_fullsized":"2024-03-14T09:52:41.142537+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:59:23.383388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"83'4","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141058+0000","last_change":"2024-03-14T09:52:05.232673+0000","last_active":"2024-03-14T09:52:41.141058+0000","last_peered":"2024-03-14T09:52:41.141058+0000","last_clean":"2024-03-14T09:52:41.141058+0000","last_became_active":"2024-03-14T09:52:05.232296+0000","last_became_peered":"2024-03-14T09:52:05.232296+0000","last_unstale":"2024-03-14T09:52:41.141058+0000","last_undegraded":"2024-03-14T09:52:41.141058+0000","last_fullsized":"2024-03-14T09:52:41.141058+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:24:56.638451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143026+0000","last_change":"2024-03-14T09:52:05.744906+0000","last_active":"2024-03-14T09:52:41.143026+0000","last_peered":"2024-03-14T09:52:41.143026+0000","last_clean":"2024-03-14T09:52:41.143026+0000","last_became_active":"2024-03-14T09:52:05.743753+0000","last_became_peered":"2024-03-14T09:52:05.743753+0000","last_unstale":"2024-03-14T09:52:41.143026+0000","last_undegraded":"2024-03-14T09:52:41.143026+0000","last_fullsized":"2024-03-14T09:52:41.143026+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T09:59:40.358352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140739+0000","last_change":"2024-03-14T09:52:09.768355+0000","last_active":"2024-03-14T09:52:41.140739+0000","last_peered":"2024-03-14T09:52:41.140739+0000","last_clean":"2024-03-14T09:52:41.140739+0000","last_became_active":"2024-03-14T09:52:09.767919+0000","last_became_peered":"2024-03-14T09:52:09.767919+0000","last_unstale":"2024-03-14T09:52:41.140739+0000","last_undegraded":"2024-03-14T09:52:41.140739+0000","last_fullsized":"2024-03-14T09:52:41.140739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:45:08.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"83'2","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143444+0000","last_change":"2024-03-14T09:52:05.230343+0000","last_active":"2024-03-14T09:52:41.143444+0000","last_peered":"2024-03-14T09:52:41.143444+0000","last_clean":"2024-03-14T09:52:41.143444+0000","last_became_active":"2024-03-14T09:52:05.229974+0000","last_became_peered":"2024-03-14T09:52:05.229974+0000","last_unstale":"2024-03-14T09:52:41.143444+0000","last_undegraded":"2024-03-14T09:52:41.143444+0000","last_fullsized":"2024-03-14T09:52:41.143444+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:21:02.458108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137283+0000","last_change":"2024-03-14T09:52:05.738347+0000","last_active":"2024-03-14T09:52:41.137283+0000","last_peered":"2024-03-14T09:52:41.137283+0000","last_clean":"2024-03-14T09:52:41.137283+0000","last_became_active":"2024-03-14T09:52:05.737961+0000","last_became_peered":"2024-03-14T09:52:05.737961+0000","last_unstale":"2024-03-14T09:52:41.137283+0000","last_undegraded":"2024-03-14T09:52:41.137283+0000","last_fullsized":"2024-03-14T09:52:41.137283+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:02:57.252069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142859+0000","last_change":"2024-03-14T09:52:09.761930+0000","last_active":"2024-03-14T09:52:41.142859+0000","last_peered":"2024-03-14T09:52:41.142859+0000","last_clean":"2024-03-14T09:52:41.142859+0000","last_became_active":"2024-03-14T09:52:09.761515+0000","last_became_peered":"2024-03-14T09:52:09.761515+0000","last_unstale":"2024-03-14T09:52:41.142859+0000","last_undegraded":"2024-03-14T09:52:41.142859+0000","last_fullsized":"2024-03-14T09:52:41.142859+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T15:49:39.262277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"83'3","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145432+0000","last_change":"2024-03-14T09:52:05.233579+0000","last_active":"2024-03-14T09:52:41.145432+0000","last_peered":"2024-03-14T09:52:41.145432+0000","last_clean":"2024-03-14T09:52:41.145432+0000","last_became_active":"2024-03-14T09:52:05.231605+0000","last_became_peered":"2024-03-14T09:52:05.231605+0000","last_unstale":"2024-03-14T09:52:41.145432+0000","last_undegraded":"2024-03-14T09:52:41.145432+0000","last_fullsized":"2024-03-14T09:52:41.145432+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:55:38.282203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140457+0000","last_change":"2024-03-14T09:52:05.736586+0000","last_active":"2024-03-14T09:52:41.140457+0000","last_peered":"2024-03-14T09:52:41.140457+0000","last_clean":"2024-03-14T09:52:41.140457+0000","last_became_active":"2024-03-14T09:52:05.736235+0000","last_became_peered":"2024-03-14T09:52:05.736235+0000","last_unstale":"2024-03-14T09:52:41.140457+0000","last_undegraded":"2024-03-14T09:52:41.140457+0000","last_fullsized":"2024-03-14T09:52:41.140457+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:46:53.293698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140743+0000","last_change":"2024-03-14T09:52:09.768203+0000","last_active":"2024-03-14T09:52:41.140743+0000","last_peered":"2024-03-14T09:52:41.140743+0000","last_clean":"2024-03-14T09:52:41.140743+0000","last_became_active":"2024-03-14T09:52:09.767777+0000","last_became_peered":"2024-03-14T09:52:09.767777+0000","last_unstale":"2024-03-14T09:52:41.140743+0000","last_undegraded":"2024-03-14T09:52:41.140743+0000","last_fullsized":"2024-03-14T09:52:41.140743+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:20:47.598905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"83'1","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142704+0000","last_change":"2024-03-14T09:52:05.232359+0000","last_active":"2024-03-14T09:52:41.142704+0000","last_peered":"2024-03-14T09:52:41.142704+0000","last_clean":"2024-03-14T09:52:41.142704+0000","last_became_active":"2024-03-14T09:52:05.231977+0000","last_became_peered":"2024-03-14T09:52:05.231977+0000","last_unstale":"2024-03-14T09:52:41.142704+0000","last_undegraded":"2024-03-14T09:52:41.142704+0000","last_fullsized":"2024-03-14T09:52:41.142704+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:32.678157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145654+0000","last_change":"2024-03-14T09:52:05.735822+0000","last_active":"2024-03-14T09:52:41.145654+0000","last_peered":"2024-03-14T09:52:41.145654+0000","last_clean":"2024-03-14T09:52:41.145654+0000","last_became_active":"2024-03-14T09:52:05.735059+0000","last_became_peered":"2024-03-14T09:52:05.735059+0000","last_unstale":"2024-03-14T09:52:41.145654+0000","last_undegraded":"2024-03-14T09:52:41.145654+0000","last_fullsized":"2024-03-14T09:52:41.145654+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T15:22:33.033689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140934+0000","last_change":"2024-03-14T09:52:09.759794+0000","last_active":"2024-03-14T09:52:41.140934+0000","last_peered":"2024-03-14T09:52:41.140934+0000","last_clean":"2024-03-14T09:52:41.140934+0000","last_became_active":"2024-03-14T09:52:09.759366+0000","last_became_peered":"2024-03-14T09:52:09.759366+0000","last_unstale":"2024-03-14T09:52:41.140934+0000","last_undegraded":"2024-03-14T09:52:41.140934+0000","last_fullsized":"2024-03-14T09:52:41.140934+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:52:40.641264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"83'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.325733+0000","last_change":"2024-03-14T09:52:42.325733+0000","last_active":"2024-03-14T09:52:42.325733+0000","last_peered":"2024-03-14T09:52:42.325733+0000","last_clean":"2024-03-14T09:52:42.325733+0000","last_became_active":"2024-03-14T09:52:42.322002+0000","last_became_peered":"2024-03-14T09:52:42.322002+0000","last_unstale":"2024-03-14T09:52:42.325733+0000","last_undegraded":"2024-03-14T09:52:42.325733+0000","last_fullsized":"2024-03-14T09:52:42.325733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:22:10.439271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137057+0000","last_change":"2024-03-14T09:52:05.734073+0000","last_active":"2024-03-14T09:52:41.137057+0000","last_peered":"2024-03-14T09:52:41.137057+0000","last_clean":"2024-03-14T09:52:41.137057+0000","last_became_active":"2024-03-14T09:52:05.731748+0000","last_became_peered":"2024-03-14T09:52:05.731748+0000","last_unstale":"2024-03-14T09:52:41.137057+0000","last_undegraded":"2024-03-14T09:52:41.137057+0000","last_fullsized":"2024-03-14T09:52:41.137057+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:01:11.789162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143980+0000","last_change":"2024-03-14T09:52:09.763309+0000","last_active":"2024-03-14T09:52:41.143980+0000","last_peered":"2024-03-14T09:52:41.143980+0000","last_clean":"2024-03-14T09:52:41.143980+0000","last_became_active":"2024-03-14T09:52:09.762868+0000","last_became_peered":"2024-03-14T09:52:09.762868+0000","last_unstale":"2024-03-14T09:52:41.143980+0000","last_undegraded":"2024-03-14T09:52:41.143980+0000","last_fullsized":"2024-03-14T09:52:41.143980+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:30:03.403911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"83'3","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143238+0000","last_change":"2024-03-14T09:52:05.232310+0000","last_active":"2024-03-14T09:52:41.143238+0000","last_peered":"2024-03-14T09:52:41.143238+0000","last_clean":"2024-03-14T09:52:41.143238+0000","last_became_active":"2024-03-14T09:52:05.231931+0000","last_became_peered":"2024-03-14T09:52:05.231931+0000","last_unstale":"2024-03-14T09:52:41.143238+0000","last_undegraded":"2024-03-14T09:52:41.143238+0000","last_fullsized":"2024-03-14T09:52:41.143238+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:35:44.274800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137430+0000","last_change":"2024-03-14T09:52:05.731797+0000","last_active":"2024-03-14T09:52:41.137430+0000","last_peered":"2024-03-14T09:52:41.137430+0000","last_clean":"2024-03-14T09:52:41.137430+0000","last_became_active":"2024-03-14T09:52:05.730478+0000","last_became_peered":"2024-03-14T09:52:05.730478+0000","last_unstale":"2024-03-14T09:52:41.137430+0000","last_undegraded":"2024-03-14T09:52:41.137430+0000","last_fullsized":"2024-03-14T09:52:41.137430+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:52:50.684430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140879+0000","last_change":"2024-03-14T09:52:09.772786+0000","last_active":"2024-03-14T09:52:41.140879+0000","last_peered":"2024-03-14T09:52:41.140879+0000","last_clean":"2024-03-14T09:52:41.140879+0000","last_became_active":"2024-03-14T09:52:09.772198+0000","last_became_peered":"2024-03-14T09:52:09.772198+0000","last_unstale":"2024-03-14T09:52:41.140879+0000","last_undegraded":"2024-03-14T09:52:41.140879+0000","last_fullsized":"2024-03-14T09:52:41.140879+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:28:56.620266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"78'1","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146209+0000","last_change":"2024-03-14T09:52:05.237657+0000","last_active":"2024-03-14T09:52:41.146209+0000","last_peered":"2024-03-14T09:52:41.146209+0000","last_clean":"2024-03-14T09:52:41.146209+0000","last_became_active":"2024-03-14T09:52:05.237381+0000","last_became_peered":"2024-03-14T09:52:05.237381+0000","last_unstale":"2024-03-14T09:52:41.146209+0000","last_undegraded":"2024-03-14T09:52:41.146209+0000","last_fullsized":"2024-03-14T09:52:41.146209+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:34:44.372128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137056+0000","last_change":"2024-03-14T09:52:05.730454+0000","last_active":"2024-03-14T09:52:41.137056+0000","last_peered":"2024-03-14T09:52:41.137056+0000","last_clean":"2024-03-14T09:52:41.137056+0000","last_became_active":"2024-03-14T09:52:05.730046+0000","last_became_peered":"2024-03-14T09:52:05.730046+0000","last_unstale":"2024-03-14T09:52:41.137056+0000","last_undegraded":"2024-03-14T09:52:41.137056+0000","last_fullsized":"2024-03-14T09:52:41.137056+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:33:19.296993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145692+0000","last_change":"2024-03-14T09:52:09.755969+0000","last_active":"2024-03-14T09:52:41.145692+0000","last_peered":"2024-03-14T09:52:41.145692+0000","last_clean":"2024-03-14T09:52:41.145692+0000","last_became_active":"2024-03-14T09:52:09.755512+0000","last_became_peered":"2024-03-14T09:52:09.755512+0000","last_unstale":"2024-03-14T09:52:41.145692+0000","last_undegraded":"2024-03-14T09:52:41.145692+0000","last_fullsized":"2024-03-14T09:52:41.145692+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:31:03.388090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137026+0000","last_change":"2024-03-14T09:52:05.232676+0000","last_active":"2024-03-14T09:52:41.137026+0000","last_peered":"2024-03-14T09:52:41.137026+0000","last_clean":"2024-03-14T09:52:41.137026+0000","last_became_active":"2024-03-14T09:52:05.232332+0000","last_became_peered":"2024-03-14T09:52:05.232332+0000","last_unstale":"2024-03-14T09:52:41.137026+0000","last_undegraded":"2024-03-14T09:52:41.137026+0000","last_fullsized":"2024-03-14T09:52:41.137026+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:06:11.020555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137349+0000","last_change":"2024-03-14T09:52:05.733338+0000","last_active":"2024-03-14T09:52:41.137349+0000","last_peered":"2024-03-14T09:52:41.137349+0000","last_clean":"2024-03-14T09:52:41.137349+0000","last_became_active":"2024-03-14T09:52:05.732994+0000","last_became_peered":"2024-03-14T09:52:05.732994+0000","last_unstale":"2024-03-14T09:52:41.137349+0000","last_undegraded":"2024-03-14T09:52:41.137349+0000","last_fullsized":"2024-03-14T09:52:41.137349+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:31:25.085916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140575+0000","last_change":"2024-03-14T09:52:05.733121+0000","last_active":"2024-03-14T09:52:41.140575+0000","last_peered":"2024-03-14T09:52:41.140575+0000","last_clean":"2024-03-14T09:52:41.140575+0000","last_became_active":"2024-03-14T09:52:05.732671+0000","last_became_peered":"2024-03-14T09:52:05.732671+0000","last_unstale":"2024-03-14T09:52:41.140575+0000","last_undegraded":"2024-03-14T09:52:41.140575+0000","last_fullsized":"2024-03-14T09:52:41.140575+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:18:54.180131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143025+0000","last_change":"2024-03-14T09:52:05.736534+0000","last_active":"2024-03-14T09:52:41.143025+0000","last_peered":"2024-03-14T09:52:41.143025+0000","last_clean":"2024-03-14T09:52:41.143025+0000","last_became_active":"2024-03-14T09:52:05.736202+0000","last_became_peered":"2024-03-14T09:52:05.736202+0000","last_unstale":"2024-03-14T09:52:41.143025+0000","last_undegraded":"2024-03-14T09:52:41.143025+0000","last_fullsized":"2024-03-14T09:52:41.143025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:22:46.161561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142597+0000","last_change":"2024-03-14T09:52:05.739434+0000","last_active":"2024-03-14T09:52:41.142597+0000","last_peered":"2024-03-14T09:52:41.142597+0000","last_clean":"2024-03-14T09:52:41.142597+0000","last_became_active":"2024-03-14T09:52:05.738023+0000","last_became_peered":"2024-03-14T09:52:05.738023+0000","last_unstale":"2024-03-14T09:52:41.142597+0000","last_undegraded":"2024-03-14T09:52:41.142597+0000","last_fullsized":"2024-03-14T09:52:41.142597+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:51:44.173645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141083+0000","last_change":"2024-03-14T09:52:05.751367+0000","last_active":"2024-03-14T09:52:41.141083+0000","last_peered":"2024-03-14T09:52:41.141083+0000","last_clean":"2024-03-14T09:52:41.141083+0000","last_became_active":"2024-03-14T09:52:05.750816+0000","last_became_peered":"2024-03-14T09:52:05.750816+0000","last_unstale":"2024-03-14T09:52:41.141083+0000","last_undegraded":"2024-03-14T09:52:41.141083+0000","last_fullsized":"2024-03-14T09:52:41.141083+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:57:58.403356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136880+0000","last_change":"2024-03-14T09:52:05.746652+0000","last_active":"2024-03-14T09:52:41.136880+0000","last_peered":"2024-03-14T09:52:41.136880+0000","last_clean":"2024-03-14T09:52:41.136880+0000","last_became_active":"2024-03-14T09:52:05.746214+0000","last_became_peered":"2024-03-14T09:52:05.746214+0000","last_unstale":"2024-03-14T09:52:41.136880+0000","last_undegraded":"2024-03-14T09:52:41.136880+0000","last_fullsized":"2024-03-14T09:52:41.136880+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:21:39.244617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.147977+0000","last_change":"2024-03-14T09:52:42.147977+0000","last_active":"2024-03-14T09:52:42.147977+0000","last_peered":"2024-03-14T09:52:42.147977+0000","last_clean":"2024-03-14T09:52:42.147977+0000","last_became_active":"2024-03-14T09:52:42.147623+0000","last_became_peered":"2024-03-14T09:52:42.147623+0000","last_unstale":"2024-03-14T09:52:42.147977+0000","last_undegraded":"2024-03-14T09:52:42.147977+0000","last_fullsized":"2024-03-14T09:52:42.147977+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:38:35.351709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143440+0000","last_change":"2024-03-14T09:52:05.739758+0000","last_active":"2024-03-14T09:52:41.143440+0000","last_peered":"2024-03-14T09:52:41.143440+0000","last_clean":"2024-03-14T09:52:41.143440+0000","last_became_active":"2024-03-14T09:52:05.739423+0000","last_became_peered":"2024-03-14T09:52:05.739423+0000","last_unstale":"2024-03-14T09:52:41.143440+0000","last_undegraded":"2024-03-14T09:52:41.143440+0000","last_fullsized":"2024-03-14T09:52:41.143440+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:47:22.594910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146100+0000","last_change":"2024-03-14T09:52:05.733281+0000","last_active":"2024-03-14T09:52:41.146100+0000","last_peered":"2024-03-14T09:52:41.146100+0000","last_clean":"2024-03-14T09:52:41.146100+0000","last_became_active":"2024-03-14T09:52:05.730081+0000","last_became_peered":"2024-03-14T09:52:05.730081+0000","last_unstale":"2024-03-14T09:52:41.146100+0000","last_undegraded":"2024-03-14T09:52:41.146100+0000","last_fullsized":"2024-03-14T09:52:41.146100+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:40:26.354063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146032+0000","last_change":"2024-03-14T09:52:05.726938+0000","last_active":"2024-03-14T09:52:41.146032+0000","last_peered":"2024-03-14T09:52:41.146032+0000","last_clean":"2024-03-14T09:52:41.146032+0000","last_became_active":"2024-03-14T09:52:05.726602+0000","last_became_peered":"2024-03-14T09:52:05.726602+0000","last_unstale":"2024-03-14T09:52:41.146032+0000","last_undegraded":"2024-03-14T09:52:41.146032+0000","last_fullsized":"2024-03-14T09:52:41.146032+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:26:20.354368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.149962+0000","last_change":"2024-03-14T09:52:42.149962+0000","last_active":"2024-03-14T09:52:42.149962+0000","last_peered":"2024-03-14T09:52:42.149962+0000","last_clean":"2024-03-14T09:52:42.149962+0000","last_became_active":"2024-03-14T09:52:42.149726+0000","last_became_peered":"2024-03-14T09:52:42.149726+0000","last_unstale":"2024-03-14T09:52:42.149962+0000","last_undegraded":"2024-03-14T09:52:42.149962+0000","last_fullsized":"2024-03-14T09:52:42.149962+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:20:18.294456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.144160+0000","last_change":"2024-03-14T09:52:05.742945+0000","last_active":"2024-03-14T09:52:41.144160+0000","last_peered":"2024-03-14T09:52:41.144160+0000","last_clean":"2024-03-14T09:52:41.144160+0000","last_became_active":"2024-03-14T09:52:05.742483+0000","last_became_peered":"2024-03-14T09:52:05.742483+0000","last_unstale":"2024-03-14T09:52:41.144160+0000","last_undegraded":"2024-03-14T09:52:41.144160+0000","last_fullsized":"2024-03-14T09:52:41.144160+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:05:31.053608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136786+0000","last_change":"2024-03-14T09:52:05.737822+0000","last_active":"2024-03-14T09:52:41.136786+0000","last_peered":"2024-03-14T09:52:41.136786+0000","last_clean":"2024-03-14T09:52:41.136786+0000","last_became_active":"2024-03-14T09:52:05.735294+0000","last_became_peered":"2024-03-14T09:52:05.735294+0000","last_unstale":"2024-03-14T09:52:41.136786+0000","last_undegraded":"2024-03-14T09:52:41.136786+0000","last_fullsized":"2024-03-14T09:52:41.136786+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:06:28.437259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140721+0000","last_change":"2024-03-14T09:52:05.746991+0000","last_active":"2024-03-14T09:52:41.140721+0000","last_peered":"2024-03-14T09:52:41.140721+0000","last_clean":"2024-03-14T09:52:41.140721+0000","last_became_active":"2024-03-14T09:52:05.746558+0000","last_became_peered":"2024-03-14T09:52:05.746558+0000","last_unstale":"2024-03-14T09:52:41.140721+0000","last_undegraded":"2024-03-14T09:52:41.140721+0000","last_fullsized":"2024-03-14T09:52:41.140721+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:46:11.540134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247929,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65635,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":247,"num_read_kb":194,"num_write":153,"num_write_kb":112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":694,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":196905,"data_compressed":4812,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579927,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":933888,"data_stored":470724,"data_compressed":17622,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.52900000000000003}]},{"osd":1,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.48299999999999998}]},{"osd":2,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.434}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.55500000000000005}]},{"osd":5,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.623}]},{"osd":6,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.64400000000000002}]},{"osd":7,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.63600000000000001}]},{"osd":8,"last update":"Thu Mar 14 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.65700000000000003}]},{"osd":9,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.71499999999999997}]},{"osd":10,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.60999999999999999}]}]},{"osd":10,"up_from":67,"seq":287762808858,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1119483,"data_compressed":55121,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.65100000000000002}]},{"osd":1,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.80800000000000005}]},{"osd":2,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.84699999999999998}]},{"osd":3,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.67500000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.77600000000000002}]},{"osd":5,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.89300000000000002}]},{"osd":6,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.83899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":8,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.98199999999999998}]},{"osd":9,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":1.014}]},{"osd":11,"last update":"Thu Mar 14 09:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.88300000000000001}]}]},{"osd":9,"up_from":62,"seq":266287972383,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":474474,"data_compressed":17622,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.56899999999999995,"15min":0.57699999999999996},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68400000000000005,"15min":0.68000000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.504,"5min":2.504,"15min":2.504},"last":0.505},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.73499999999999999,"15min":0.74299999999999999},"min":{"1min":0.50800000000000001,"5min":0.432,"15min":0.432},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":0.58899999999999997}]},{"osd":2,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.54200000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":2.3380000000000001,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":0.624},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.78000000000000003,"15min":0.79100000000000004},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.2360000000000002,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":0.81799999999999995}]},{"osd":3,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.71199999999999997,"15min":0.72499999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.0649999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.67600000000000005,"15min":0.68400000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.86699999999999999}]},{"osd":4,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.81000000000000005,"15min":0.82399999999999995},"min":{"1min":0.52500000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.79700000000000004,"15min":0.81000000000000005},"min":{"1min":0.53000000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.755}]},{"osd":5,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.85099999999999998,"15min":0.86699999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.83599999999999997,"15min":0.84199999999999997},"min":{"1min":0.57799999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":0.69999999999999996}]},{"osd":6,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.88600000000000001,"15min":0.90600000000000003},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.90500000000000003,"15min":0.92100000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.77300000000000002}]},{"osd":8,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.92100000000000004,"15min":0.94099999999999995},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.93600000000000005,"15min":0.95399999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":0.94899999999999995}]},{"osd":10,"last update":"Thu Mar 14 09:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.93600000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]}]},{"osd":8,"up_from":55,"seq":236223201313,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":469967,"data_compressed":17622,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.042,"15min":1.1200000000000001},"min":{"1min":0.36899999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80100000000000005,"5min":9.7769999999999992,"15min":9.7769999999999992},"last":0.58199999999999996}]},{"osd":1,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.39300000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.433,"5min":0.432,"15min":0.432},"max":{"1min":0.747,"5min":0.877,"15min":0.877},"last":0.68999999999999995}]},{"osd":2,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.435,"15min":0.42699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.45600000000000002,"15min":0.44900000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.435,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.72999999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.62,"15min":0.61599999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.79000000000000004,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.748}]},{"osd":4,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.57699999999999996,"15min":0.56899999999999995},"min":{"1min":0.48599999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.84999999999999998,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.59399999999999997,"15min":0.58399999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.63300000000000001}]},{"osd":5,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.499},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.52500000000000002,"15min":0.51700000000000002},"min":{"1min":0.38700000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.67900000000000005}]},{"osd":6,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.64800000000000002,"15min":0.64600000000000002},"min":{"1min":0.42799999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.78100000000000003}]},{"osd":7,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.629,"15min":0.62},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.56100000000000005,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.76700000000000002}]},{"osd":9,"last update":"Thu Mar 14 09:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.73299999999999998,"15min":0.72999999999999998},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.53800000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.83899999999999997}]},{"osd":10,"last update":"Thu Mar 14 09:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.80300000000000005}]},{"osd":11,"last update":"Thu Mar 14 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.81200000000000006}]}]},{"osd":7,"up_from":48,"seq":206158430246,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":1065530,"data_compressed":53517,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.40999999999999998}]},{"osd":1,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.42699999999999999,"15min":0.41999999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.41699999999999998,"15min":0.40899999999999997},"min":{"1min":0.29199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.57699999999999996,"15min":0.56799999999999995},"min":{"1min":0.441,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.42199999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.47499999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.41099999999999998}]},{"osd":4,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.48799999999999999,"15min":0.47799999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.503,"15min":0.497},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.42699999999999999}]},{"osd":5,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.58799999999999997,"15min":0.57599999999999996},"min":{"1min":0.47799999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.49099999999999999}]},{"osd":6,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.64300000000000002,"15min":0.63500000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66500000000000004,"15min":0.65900000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.54700000000000004}]},{"osd":8,"last update":"Thu Mar 14 09:52:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.56499999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.92300000000000004,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.89500000000000002,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.505}]},{"osd":9,"last update":"Thu Mar 14 09:53:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.75700000000000001,"15min":0.76300000000000001},"min":{"1min":0.55500000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.98299999999999998,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.75,"5min":0.77000000000000002,"15min":0.77400000000000002},"min":{"1min":0.61399999999999999,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.92300000000000004,"5min":1.008,"15min":1.008},"last":0.53600000000000003}]},{"osd":10,"last update":"Thu Mar 14 09:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.59199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":6,"up_from":43,"seq":184683593768,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":465554,"data_compressed":17622,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.51200000000000001,"15min":0.48199999999999998},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.54,"5min":3.54,"15min":3.54},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.52300000000000002,"15min":0.497},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.9860000000000002,"5min":3.9860000000000002,"15min":3.9860000000000002},"last":0.89800000000000002}]},{"osd":1,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.70699999999999996,"15min":0.65900000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":3.7589999999999999,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.75,"5min":0.64100000000000001,"15min":0.61099999999999999},"min":{"1min":0.52900000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.875}]},{"osd":2,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.68400000000000005,"15min":0.64700000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":4.0190000000000001,"5min":4.0190000000000001,"15min":4.0190000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.72399999999999998,"15min":0.68600000000000005},"min":{"1min":0.46100000000000002,"5min":0.38,"15min":0.38},"max":{"1min":3.5790000000000002,"5min":3.5790000000000002,"15min":3.5790000000000002},"last":1.0069999999999999}]},{"osd":3,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.63800000000000001,"15min":0.60699999999999998},"min":{"1min":0.41999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.7959999999999998,"5min":3.7959999999999998,"15min":3.7959999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.55400000000000005,"15min":0.54600000000000004},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78300000000000003,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.73999999999999999}]},{"osd":4,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.69599999999999995,"15min":0.65600000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.76700000000000002,"15min":0.72899999999999998},"min":{"1min":0.53000000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":3.8559999999999999,"5min":3.8559999999999999,"15min":3.8559999999999999},"last":1.1180000000000001}]},{"osd":5,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.76500000000000001,"15min":0.77500000000000002},"min":{"1min":0.58799999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1120000000000001,"5min":3.1059999999999999,"15min":3.1059999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.746,"15min":0.72099999999999997},"min":{"1min":0.53500000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":4.0469999999999997,"5min":4.0469999999999997,"15min":4.0469999999999997},"last":0.82099999999999995}]},{"osd":7,"last update":"Thu Mar 14 09:52:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.69299999999999995,"15min":0.68700000000000006},"min":{"1min":0.51100000000000001,"5min":0.495,"15min":0.495},"max":{"1min":0.90200000000000002,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.77300000000000002,"15min":0.77100000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":1.155}]},{"osd":8,"last update":"Thu Mar 14 09:52:37 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.89300000000000002,"15min":0.91100000000000003},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":0.94699999999999995,"5min":3.117,"15min":3.117},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.75,"15min":0.753},"min":{"1min":0.55800000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.0029999999999999,"5min":1.143,"15min":1.143},"last":0.86099999999999999}]},{"osd":9,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.84999999999999998,"15min":0.86199999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.95299999999999996,"5min":1.119,"15min":1.119},"last":1.2150000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.79900000000000004,"15min":0.80500000000000005},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":1.0649999999999999}]},{"osd":10,"last update":"Thu Mar 14 09:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":1.034},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.97599999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]}]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":527305,"data_compressed":19226,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.56399999999999995,"15min":0.56699999999999995},"min":{"1min":0.34599999999999997,"5min":0.315,"15min":0.315},"max":{"1min":1.3879999999999999,"5min":1.917,"15min":1.917},"last":0.52600000000000002},{"interface":"front","average":{"1min":1.05,"5min":0.64300000000000002,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":7.6399999999999997,"5min":7.6399999999999997,"15min":7.6399999999999997},"last":0.498}]},{"osd":2,"last update":"Thu Mar 14 09:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.55100000000000005,"15min":0.54400000000000004},"min":{"1min":0.38800000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.4419999999999999,"5min":1.885,"15min":1.885},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.58699999999999997,"15min":0.54100000000000004},"min":{"1min":0.39900000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"last":0.45100000000000001}]},{"osd":3,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.59899999999999998,"15min":0.57699999999999996},"min":{"1min":0.39300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.1829999999999998,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.58699999999999997,"15min":0.57699999999999996},"min":{"1min":0.434,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.6299999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":0.47199999999999998}]},{"osd":4,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.378,"15min":0.36399999999999999},"min":{"1min":0.31,"5min":0.249,"15min":0.249},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.42999999999999999,"15min":0.40000000000000002},"min":{"1min":0.316,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":2.1339999999999999,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Mar 14 09:52:37 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.53600000000000003,"15min":0.48899999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":2.9849999999999999,"5min":2.9849999999999999,"15min":2.9849999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.55400000000000005,"15min":0.51700000000000002},"min":{"1min":0.44800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.57699999999999996}]},{"osd":6,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.69899999999999995,"15min":0.64900000000000002},"min":{"1min":0.60199999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":3.7879999999999998,"5min":3.7879999999999998,"15min":3.7879999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.80400000000000005,"15min":0.85999999999999999},"min":{"1min":0.46500000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.5910000000000002,"5min":5.5229999999999997,"15min":5.5229999999999997},"last":0.58599999999999997}]},{"osd":7,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.51100000000000001,"15min":0.48899999999999999},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.49199999999999999,"15min":0.48499999999999999},"min":{"1min":0.33300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.50900000000000001}]},{"osd":8,"last update":"Thu Mar 14 09:52:33 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.68000000000000005,"15min":0.623},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":6.4500000000000002,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.66000000000000003,"15min":0.64000000000000001},"min":{"1min":0.50600000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.59599999999999997}]},{"osd":9,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":0.76300000000000001,"15min":0.70599999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":6.5149999999999997,"5min":6.5149999999999997,"15min":6.5149999999999997},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.359,"5min":0.83799999999999997,"15min":0.751},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":9.8900000000000006,"5min":9.8900000000000006,"15min":9.8900000000000006},"last":0.64600000000000002}]},{"osd":10,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.621},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.68999999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738433,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":950272,"data_stored":466942,"data_compressed":17622,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 14 09:52:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.56699999999999995,"15min":0.52900000000000003},"min":{"1min":0.34999999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.58899999999999997,"15min":0.54900000000000004},"min":{"1min":0.28199999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":0.64000000000000001}]},{"osd":2,"last update":"Thu Mar 14 09:52:44 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.67000000000000004,"15min":0.57099999999999995},"min":{"1min":0.503,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":5.1349999999999998,"5min":5.1349999999999998,"15min":5.1349999999999998},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.64900000000000002,"15min":0.57599999999999996},"min":{"1min":0.53500000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":0.80900000000000005}]},{"osd":3,"last update":"Thu Mar 14 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":0.75600000000000001}]},{"osd":4,"last update":"Thu Mar 14 09:52:22 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.71399999999999997,"15min":0.66600000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.8220000000000001,"5min":3.8220000000000001,"15min":3.8220000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.61499999999999999,"15min":0.58299999999999996},"min":{"1min":0.38900000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.573,"5min":3.573,"15min":3.573},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Mar 14 09:52:39 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.73399999999999999,"15min":0.67700000000000005},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.415,"5min":3.863,"15min":3.863},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.69699999999999995,"15min":0.67700000000000005},"min":{"1min":0.48499999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.621}]},{"osd":6,"last update":"Thu Mar 14 09:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.65600000000000003,"15min":0.60999999999999999},"min":{"1min":0.41299999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2370000000000001,"5min":5.2370000000000001,"15min":5.2370000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.76500000000000001,"15min":0.72499999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.5890000000000004,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":0.84099999999999997}]},{"osd":7,"last update":"Thu Mar 14 09:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.7509999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.76700000000000002}]},{"osd":8,"last update":"Thu Mar 14 09:52:34 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.83699999999999997,"15min":0.83199999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.80100000000000005,"15min":0.79300000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":3.6709999999999998,"5min":3.6709999999999998,"15min":3.6709999999999998},"last":0.73799999999999999}]},{"osd":9,"last update":"Thu Mar 14 09:52:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.88600000000000001,"15min":0.83999999999999997},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":6.7699999999999996,"5min":6.7699999999999996,"15min":6.7699999999999996},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.2110000000000001,"5min":0.93100000000000005,"15min":0.88400000000000001},"min":{"1min":0.45400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":6.0199999999999996,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":0.872}]},{"osd":10,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.68700000000000006}]},{"osd":11,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":2.5590000000000002,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":3.71,"5min":3.71,"15min":3.71},"last":0.82999999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411384,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":528383,"data_compressed":19226,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52600000000000002,"15min":0.51600000000000001},"min":{"1min":0.42799999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.52600000000000002},"min":{"1min":0.439,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85199999999999998,"5min":0.879,"15min":0.879},"last":0.52000000000000002}]},{"osd":1,"last update":"Thu Mar 14 09:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.55900000000000005,"15min":0.52600000000000002},"min":{"1min":0.41799999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.55300000000000005,"15min":0.54100000000000004},"min":{"1min":0.438,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.81699999999999995,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.44400000000000001}]},{"osd":3,"last update":"Thu Mar 14 09:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.55000000000000004,"15min":0.52600000000000002},"min":{"1min":0.42599999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.624},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56999999999999995,"15min":0.54900000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.57599999999999996}]},{"osd":4,"last update":"Thu Mar 14 09:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.73199999999999998,"5min":0.87,"15min":0.87},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.39500000000000002}]},{"osd":5,"last update":"Thu Mar 14 09:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.47099999999999997,"15min":0.44500000000000001},"min":{"1min":0.315,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.48999999999999999,"15min":0.47699999999999998},"min":{"1min":0.34399999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.72599999999999998,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.32800000000000001}]},{"osd":6,"last update":"Thu Mar 14 09:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.66100000000000003,"15min":0.65000000000000002},"min":{"1min":0.55100000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.67600000000000005,"15min":0.66300000000000003},"min":{"1min":0.56699999999999995,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Mar 14 09:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.54400000000000004}]},{"osd":8,"last update":"Thu Mar 14 09:52:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63,"15min":0.623},"min":{"1min":0.38800000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.505},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.64200000000000002,"15min":0.63700000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.41499999999999998}]},{"osd":9,"last update":"Thu Mar 14 09:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.63900000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.75700000000000001,"15min":0.746},"min":{"1min":0.59299999999999997,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.749}]},{"osd":10,"last update":"Thu Mar 14 09:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.59799999999999998}]},{"osd":11,"last update":"Thu Mar 14 09:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.55600000000000005}]}]},{"osd":3,"up_from":24,"seq":103079215158,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":465748,"data_compressed":17622,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.42199999999999999,"15min":0.40100000000000002},"min":{"1min":0.32300000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.44600000000000001,"15min":0.42999999999999999},"min":{"1min":0.373,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.502}]},{"osd":1,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61099999999999999,"15min":0.59099999999999997},"min":{"1min":0.39400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.82799999999999996,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60699999999999998,"15min":0.58499999999999996},"min":{"1min":0.437,"5min":0.374,"15min":0.374},"max":{"1min":0.76300000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.61799999999999999}]},{"osd":2,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63400000000000001,"15min":0.60699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84099999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.751},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64000000000000001,"15min":0.623},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.80400000000000005,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.67500000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.61799999999999999,"15min":0.59599999999999997},"min":{"1min":0.42099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.61399999999999999,"15min":0.59899999999999998},"min":{"1min":0.48199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Mar 14 09:52:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.46700000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.82499999999999996,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70099999999999996,"15min":0.69399999999999995},"min":{"1min":0.51500000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.96199999999999997,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.79200000000000004}]},{"osd":6,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64800000000000002,"15min":0.65000000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86599999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65800000000000003,"15min":0.64300000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.98799999999999999,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.52700000000000002}]},{"osd":7,"last update":"Thu Mar 14 09:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.53500000000000003,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.109,"5min":1.123,"15min":1.123},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77000000000000002,"15min":0.76900000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.69399999999999995}]},{"osd":8,"last update":"Thu Mar 14 09:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.871}]},{"osd":9,"last update":"Thu Mar 14 09:52:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.80000000000000004,"15min":0.81499999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.98099999999999998,"5min":1.097,"15min":1.097},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.83999999999999997,"5min":1.121,"15min":1.121},"last":0.63500000000000001}]},{"osd":10,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.78000000000000003}]},{"osd":11,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.81299999999999994}]}]},{"osd":4,"up_from":31,"seq":133143986225,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":470548,"data_compressed":17622,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.40100000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71599999999999997,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.52300000000000002,"15min":0.51600000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.35099999999999998},"min":{"1min":0.26600000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.378},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.40200000000000002}]},{"osd":2,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.55900000000000005,"15min":0.53600000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53100000000000003,"15min":0.51500000000000001},"min":{"1min":0.433,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63700000000000001}]},{"osd":3,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.40699999999999997,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76500000000000001,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55400000000000005,"15min":0.55700000000000005},"min":{"1min":0.39800000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.57399999999999995}]},{"osd":5,"last update":"Thu Mar 14 09:52:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.57099999999999995,"15min":0.52500000000000002},"min":{"1min":0.39100000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.3479999999999999,"5min":4.3479999999999999,"15min":4.3479999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.53800000000000003,"15min":0.52700000000000002},"min":{"1min":0.42099999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.75600000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.49099999999999999}]},{"osd":6,"last update":"Thu Mar 14 09:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.48099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73499999999999999,"5min":0.876,"15min":0.876},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.60899999999999999,"15min":0.59999999999999998},"min":{"1min":0.42799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.80500000000000005,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.66200000000000003}]},{"osd":7,"last update":"Thu Mar 14 09:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.379,"5min":0.318,"15min":0.318},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.45600000000000002,"15min":0.45000000000000001},"min":{"1min":0.33200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.61799999999999999}]},{"osd":8,"last update":"Thu Mar 14 09:52:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.63800000000000001,"15min":0.63100000000000001},"min":{"1min":0.41299999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.60699999999999998,"15min":0.59899999999999998},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.79300000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.59299999999999997}]},{"osd":9,"last update":"Thu Mar 14 09:52:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.47599999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.75,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71299999999999997,"15min":0.70899999999999996},"min":{"1min":0.61899999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.84099999999999997,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.73399999999999999}]},{"osd":10,"last update":"Thu Mar 14 09:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.71999999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]}]},{"osd":5,"up_from":36,"seq":154618822702,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":1061356,"data_compressed":53517,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.40200000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56000000000000005,"15min":0.54900000000000004},"min":{"1min":0.46899999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81000000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.372}]},{"osd":1,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.53600000000000003,"15min":0.51600000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79100000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.628,"5min":0.54700000000000004,"15min":0.51900000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.40100000000000002,"15min":0.38100000000000001},"min":{"1min":0.30299999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.68899999999999995,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.40400000000000003,"15min":0.38300000000000001},"min":{"1min":0.28100000000000003,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.44,"5min":0.372,"15min":0.372},"max":{"1min":0.77800000000000002,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.59799999999999998,"15min":0.58599999999999997},"min":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.57599999999999996,"15min":0.55400000000000005},"min":{"1min":0.40999999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55000000000000004,"15min":0.53400000000000003},"min":{"1min":0.39300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78400000000000003,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.63}]},{"osd":6,"last update":"Thu Mar 14 09:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.628,"15min":0.626},"min":{"1min":0.51800000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.79100000000000004,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.64000000000000001,"15min":0.627},"min":{"1min":0.44600000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.82799999999999996,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.64500000000000002}]},{"osd":7,"last update":"Thu Mar 14 09:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.60799999999999998,"15min":0.59299999999999997},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.59899999999999998,"15min":0.58499999999999996},"min":{"1min":0.46400000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.70999999999999996}]},{"osd":8,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59699999999999998,"15min":0.59199999999999997},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.443},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.56399999999999995},"min":{"1min":0.374,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.433}]},{"osd":9,"last update":"Thu Mar 14 09:52:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.72099999999999997,"15min":0.71199999999999997},"min":{"1min":0.56799999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96299999999999997,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.74299999999999999,"15min":0.73699999999999999},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.54400000000000004}]},{"osd":10,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34227,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34227,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":61841,"data_compressed":1604,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62919,"data_compressed":1604,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63651,"data_compressed":1604,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-14T09:53:10.609 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-14T09:53:10.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:10 smithi132 ceph-mon[47959]: pgmap v269: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-14T09:53:10.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:10 smithi078 ceph-mon[38143]: pgmap v269: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-14T09:53:10.971 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:53:10.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:10 smithi152 ceph-mon[48279]: pgmap v269: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-14T09:53:11.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:11 smithi132 ceph-mon[47959]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:53:11.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:11 smithi078 ceph-mon[38143]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:53:11.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:11 smithi152 ceph-mon[48279]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:53:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.134+0000 7fd75b800700 1 -- 172.21.15.78:0/85155026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd754101960 msgr2=0x7fd7541059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.134+0000 7fd75b800700 1 --2- 172.21.15.78:0/85155026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd754101960 0x7fd7541059b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd740009990 tx=0x7fd740009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.134+0000 7fd75b800700 1 -- 172.21.15.78:0/85155026 shutdown_connections 2024-03-14T09:53:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.134+0000 7fd75b800700 1 --2- 172.21.15.78:0/85155026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd754100fb0 0x7fd754101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.134+0000 7fd75b800700 1 --2- 172.21.15.78:0/85155026 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd754106090 0x7fd75410dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.134+0000 7fd75b800700 1 --2- 172.21.15.78:0/85155026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd754101960 0x7fd7541059b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:12.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.134+0000 7fd75b800700 1 -- 172.21.15.78:0/85155026 >> 172.21.15.78:0/85155026 conn(0x7fd7540fc800 msgr2=0x7fd7540fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:12.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.134+0000 7fd75b800700 1 -- 172.21.15.78:0/85155026 shutdown_connections 2024-03-14T09:53:12.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.134+0000 7fd75b800700 1 -- 172.21.15.78:0/85155026 wait complete. 2024-03-14T09:53:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.135+0000 7fd75b800700 1 Processor -- start 2024-03-14T09:53:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.135+0000 7fd75b800700 1 -- start start 2024-03-14T09:53:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.136+0000 7fd75b800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd754100fb0 0x7fd75419e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.136+0000 7fd75b800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd754101960 0x7fd75419e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.136+0000 7fd75b800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd754106090 0x7fd7541996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.136+0000 7fd75b800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd754199cf0 con 0x7fd754100fb0 2024-03-14T09:53:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.136+0000 7fd75b800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd754199e60 con 0x7fd754101960 2024-03-14T09:53:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.136+0000 7fd75b800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75419a020 con 0x7fd754106090 2024-03-14T09:53:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.136+0000 7fd758e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd754100fb0 0x7fd75419e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.136+0000 7fd753e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd754101960 0x7fd75419e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.136+0000 7fd753e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd754101960 0x7fd75419e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:38352/0 (socket says 172.21.15.78:38352) 2024-03-14T09:53:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.136+0000 7fd753e00700 1 -- 172.21.15.78:0/2770136345 learned_addr learned my addr 172.21.15.78:0/2770136345 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.136+0000 7fd759800700 1 --2- 172.21.15.78:0/2770136345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd754106090 0x7fd7541996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.137+0000 7fd758e00700 1 -- 172.21.15.78:0/2770136345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd754106090 msgr2=0x7fd7541996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.137+0000 7fd758e00700 1 --2- 172.21.15.78:0/2770136345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd754106090 0x7fd7541996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.137+0000 7fd758e00700 1 -- 172.21.15.78:0/2770136345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd754101960 msgr2=0x7fd75419e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.137+0000 7fd758e00700 1 --2- 172.21.15.78:0/2770136345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd754101960 0x7fd75419e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.137+0000 7fd758e00700 1 -- 172.21.15.78:0/2770136345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7400095f0 con 0x7fd754100fb0 2024-03-14T09:53:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.137+0000 7fd753e00700 1 --2- 172.21.15.78:0/2770136345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd754101960 0x7fd75419e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:12.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.137+0000 7fd758e00700 1 --2- 172.21.15.78:0/2770136345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd754100fb0 0x7fd75419e3b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd74800d750 tx=0x7fd74800da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:12.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.137+0000 7fd751600700 1 -- 172.21.15.78:0/2770136345 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd748004020 con 0x7fd754100fb0 2024-03-14T09:53:12.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.137+0000 7fd751600700 1 -- 172.21.15.78:0/2770136345 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd748004830 con 0x7fd754100fb0 2024-03-14T09:53:12.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.137+0000 7fd75b800700 1 -- 172.21.15.78:0/2770136345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd75419a300 con 0x7fd754100fb0 2024-03-14T09:53:12.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.137+0000 7fd751600700 1 -- 172.21.15.78:0/2770136345 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7480054c0 con 0x7fd754100fb0 2024-03-14T09:53:12.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.137+0000 7fd75b800700 1 -- 172.21.15.78:0/2770136345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7541a6500 con 0x7fd754100fb0 2024-03-14T09:53:12.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.138+0000 7fd75b800700 1 -- 172.21.15.78:0/2770136345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd754102490 con 0x7fd754100fb0 2024-03-14T09:53:12.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.140+0000 7fd751600700 1 -- 172.21.15.78:0/2770136345 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fd7480049a0 con 0x7fd754100fb0 2024-03-14T09:53:12.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.141+0000 7fd751600700 1 --2- 172.21.15.78:0/2770136345 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd73c0a0b20 0x7fd73c0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:12.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.142+0000 7fd751600700 1 -- 172.21.15.78:0/2770136345 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd7480cbe40 con 0x7fd754100fb0 2024-03-14T09:53:12.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.142+0000 7fd753e00700 1 --2- 172.21.15.78:0/2770136345 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd73c0a0b20 0x7fd73c0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:12.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.142+0000 7fd753e00700 1 --2- 172.21.15.78:0/2770136345 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd73c0a0b20 0x7fd73c0a2fe0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd7540fcc50 tx=0x7fd740005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:12.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.145+0000 7fd751600700 1 -- 172.21.15.78:0/2770136345 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd748095340 con 0x7fd754100fb0 2024-03-14T09:53:12.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.343+0000 7fd75b800700 1 -- 172.21.15.78:0/2770136345 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd7541110a0 con 0x7fd73c0a0b20 2024-03-14T09:53:12.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.355+0000 7fd751600700 1 -- 172.21.15.78:0/2770136345 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+425829 (secure 0 0 0) 0x7fd7541110a0 con 0x7fd73c0a0b20 2024-03-14T09:53:12.362 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:12.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.360+0000 7fd75b800700 1 -- 172.21.15.78:0/2770136345 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd73c0a0b20 msgr2=0x7fd73c0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:12.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.360+0000 7fd75b800700 1 --2- 172.21.15.78:0/2770136345 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd73c0a0b20 0x7fd73c0a2fe0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd7540fcc50 tx=0x7fd740005960 comp rx=0 tx=0).stop 2024-03-14T09:53:12.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.360+0000 7fd75b800700 1 -- 172.21.15.78:0/2770136345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd754100fb0 msgr2=0x7fd75419e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:12.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.360+0000 7fd75b800700 1 --2- 172.21.15.78:0/2770136345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd754100fb0 0x7fd75419e3b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd74800d750 tx=0x7fd74800da60 comp rx=0 tx=0).stop 2024-03-14T09:53:12.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.361+0000 7fd75b800700 1 -- 172.21.15.78:0/2770136345 shutdown_connections 2024-03-14T09:53:12.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.361+0000 7fd75b800700 1 --2- 172.21.15.78:0/2770136345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd754106090 0x7fd7541996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:12.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.361+0000 7fd75b800700 1 --2- 172.21.15.78:0/2770136345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd754100fb0 0x7fd75419e3b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:12.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.361+0000 7fd75b800700 1 --2- 172.21.15.78:0/2770136345 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd73c0a0b20 0x7fd73c0a2fe0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:12.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.361+0000 7fd75b800700 1 --2- 172.21.15.78:0/2770136345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd754101960 0x7fd75419e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:12.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.361+0000 7fd75b800700 1 -- 172.21.15.78:0/2770136345 >> 172.21.15.78:0/2770136345 conn(0x7fd7540fc800 msgr2=0x7fd7540febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:12.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.361+0000 7fd75b800700 1 -- 172.21.15.78:0/2770136345 shutdown_connections 2024-03-14T09:53:12.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:12.361+0000 7fd75b800700 1 -- 172.21.15.78:0/2770136345 wait complete. 2024-03-14T09:53:12.366 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-14T09:53:12.701 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:12 smithi078 ceph-mon[38143]: pgmap v270: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T09:53:12.729 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-03-14T09:53:11.083066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656237,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":393,"num_read_kb":318,"num_write":336,"num_write_kb":3052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1181430,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":335612,"kb_used_data":11368,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124606212,"statfs":{"total":1151940427776,"available":1151596761088,"internally_reserved":0,"allocated":11640832,"data_stored":7586014,"data_compressed":323961,"data_compressed_allocated":442368,"data_compressed_original":2383872,"omap_allocated":19131,"internal_metadata":331855173},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003203"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141879+0000","last_change":"2024-03-14T09:52:05.737617+0000","last_active":"2024-03-14T09:52:41.141879+0000","last_peered":"2024-03-14T09:52:41.141879+0000","last_clean":"2024-03-14T09:52:41.141879+0000","last_became_active":"2024-03-14T09:52:05.736796+0000","last_became_peered":"2024-03-14T09:52:05.736796+0000","last_unstale":"2024-03-14T09:52:41.141879+0000","last_undegraded":"2024-03-14T09:52:41.141879+0000","last_fullsized":"2024-03-14T09:52:41.141879+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:47:52.437748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142968+0000","last_change":"2024-03-14T09:52:05.749474+0000","last_active":"2024-03-14T09:52:41.142968+0000","last_peered":"2024-03-14T09:52:41.142968+0000","last_clean":"2024-03-14T09:52:41.142968+0000","last_became_active":"2024-03-14T09:52:05.748927+0000","last_became_peered":"2024-03-14T09:52:05.748927+0000","last_unstale":"2024-03-14T09:52:41.142968+0000","last_undegraded":"2024-03-14T09:52:41.142968+0000","last_fullsized":"2024-03-14T09:52:41.142968+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:28:15.852021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142862+0000","last_change":"2024-03-14T09:52:05.745098+0000","last_active":"2024-03-14T09:52:41.142862+0000","last_peered":"2024-03-14T09:52:41.142862+0000","last_clean":"2024-03-14T09:52:41.142862+0000","last_became_active":"2024-03-14T09:52:05.744755+0000","last_became_peered":"2024-03-14T09:52:05.744755+0000","last_unstale":"2024-03-14T09:52:41.142862+0000","last_undegraded":"2024-03-14T09:52:41.142862+0000","last_fullsized":"2024-03-14T09:52:41.142862+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:35:41.932386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142028+0000","last_change":"2024-03-14T09:52:05.737429+0000","last_active":"2024-03-14T09:52:41.142028+0000","last_peered":"2024-03-14T09:52:41.142028+0000","last_clean":"2024-03-14T09:52:41.142028+0000","last_became_active":"2024-03-14T09:52:05.736516+0000","last_became_peered":"2024-03-14T09:52:05.736516+0000","last_unstale":"2024-03-14T09:52:41.142028+0000","last_undegraded":"2024-03-14T09:52:41.142028+0000","last_fullsized":"2024-03-14T09:52:41.142028+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:48:38.158783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137401+0000","last_change":"2024-03-14T09:52:05.729634+0000","last_active":"2024-03-14T09:52:41.137401+0000","last_peered":"2024-03-14T09:52:41.137401+0000","last_clean":"2024-03-14T09:52:41.137401+0000","last_became_active":"2024-03-14T09:52:05.727889+0000","last_became_peered":"2024-03-14T09:52:05.727889+0000","last_unstale":"2024-03-14T09:52:41.137401+0000","last_undegraded":"2024-03-14T09:52:41.137401+0000","last_fullsized":"2024-03-14T09:52:41.137401+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:01:13.216763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.147610+0000","last_change":"2024-03-14T09:52:42.147610+0000","last_active":"2024-03-14T09:52:42.147610+0000","last_peered":"2024-03-14T09:52:42.147610+0000","last_clean":"2024-03-14T09:52:42.147610+0000","last_became_active":"2024-03-14T09:52:42.147262+0000","last_became_peered":"2024-03-14T09:52:42.147262+0000","last_unstale":"2024-03-14T09:52:42.147610+0000","last_undegraded":"2024-03-14T09:52:42.147610+0000","last_fullsized":"2024-03-14T09:52:42.147610+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:20:49.855564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136737+0000","last_change":"2024-03-14T09:52:05.727365+0000","last_active":"2024-03-14T09:52:41.136737+0000","last_peered":"2024-03-14T09:52:41.136737+0000","last_clean":"2024-03-14T09:52:41.136737+0000","last_became_active":"2024-03-14T09:52:05.726984+0000","last_became_peered":"2024-03-14T09:52:05.726984+0000","last_unstale":"2024-03-14T09:52:41.136737+0000","last_undegraded":"2024-03-14T09:52:41.136737+0000","last_fullsized":"2024-03-14T09:52:41.136737+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:42:57.129718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145478+0000","last_change":"2024-03-14T09:52:05.729622+0000","last_active":"2024-03-14T09:52:41.145478+0000","last_peered":"2024-03-14T09:52:41.145478+0000","last_clean":"2024-03-14T09:52:41.145478+0000","last_became_active":"2024-03-14T09:52:05.729293+0000","last_became_peered":"2024-03-14T09:52:05.729293+0000","last_unstale":"2024-03-14T09:52:41.145478+0000","last_undegraded":"2024-03-14T09:52:41.145478+0000","last_fullsized":"2024-03-14T09:52:41.145478+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:13:20.875857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.144495+0000","last_change":"2024-03-14T09:52:05.741620+0000","last_active":"2024-03-14T09:52:41.144495+0000","last_peered":"2024-03-14T09:52:41.144495+0000","last_clean":"2024-03-14T09:52:41.144495+0000","last_became_active":"2024-03-14T09:52:05.734079+0000","last_became_peered":"2024-03-14T09:52:05.734079+0000","last_unstale":"2024-03-14T09:52:41.144495+0000","last_undegraded":"2024-03-14T09:52:41.144495+0000","last_fullsized":"2024-03-14T09:52:41.144495+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:02:35.529671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143730+0000","last_change":"2024-03-14T09:52:05.743424+0000","last_active":"2024-03-14T09:52:41.143730+0000","last_peered":"2024-03-14T09:52:41.143730+0000","last_clean":"2024-03-14T09:52:41.143730+0000","last_became_active":"2024-03-14T09:52:05.743100+0000","last_became_peered":"2024-03-14T09:52:05.743100+0000","last_unstale":"2024-03-14T09:52:41.143730+0000","last_undegraded":"2024-03-14T09:52:41.143730+0000","last_fullsized":"2024-03-14T09:52:41.143730+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:32:33.492163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137363+0000","last_change":"2024-03-14T09:52:05.735654+0000","last_active":"2024-03-14T09:52:41.137363+0000","last_peered":"2024-03-14T09:52:41.137363+0000","last_clean":"2024-03-14T09:52:41.137363+0000","last_became_active":"2024-03-14T09:52:05.735371+0000","last_became_peered":"2024-03-14T09:52:05.735371+0000","last_unstale":"2024-03-14T09:52:41.137363+0000","last_undegraded":"2024-03-14T09:52:41.137363+0000","last_fullsized":"2024-03-14T09:52:41.137363+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:19:50.011554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.146700+0000","last_change":"2024-03-14T09:52:42.146700+0000","last_active":"2024-03-14T09:52:42.146700+0000","last_peered":"2024-03-14T09:52:42.146700+0000","last_clean":"2024-03-14T09:52:42.146700+0000","last_became_active":"2024-03-14T09:52:42.145377+0000","last_became_peered":"2024-03-14T09:52:42.145377+0000","last_unstale":"2024-03-14T09:52:42.146700+0000","last_undegraded":"2024-03-14T09:52:42.146700+0000","last_fullsized":"2024-03-14T09:52:42.146700+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:35:52.735091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136767+0000","last_change":"2024-03-14T09:52:05.740209+0000","last_active":"2024-03-14T09:52:41.136767+0000","last_peered":"2024-03-14T09:52:41.136767+0000","last_clean":"2024-03-14T09:52:41.136767+0000","last_became_active":"2024-03-14T09:52:05.732949+0000","last_became_peered":"2024-03-14T09:52:05.732949+0000","last_unstale":"2024-03-14T09:52:41.136767+0000","last_undegraded":"2024-03-14T09:52:41.136767+0000","last_fullsized":"2024-03-14T09:52:41.136767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:15:56.377569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141029+0000","last_change":"2024-03-14T09:52:05.748795+0000","last_active":"2024-03-14T09:52:41.141029+0000","last_peered":"2024-03-14T09:52:41.141029+0000","last_clean":"2024-03-14T09:52:41.141029+0000","last_became_active":"2024-03-14T09:52:05.746107+0000","last_became_peered":"2024-03-14T09:52:05.746107+0000","last_unstale":"2024-03-14T09:52:41.141029+0000","last_undegraded":"2024-03-14T09:52:41.141029+0000","last_fullsized":"2024-03-14T09:52:41.141029+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:49:07.932854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140932+0000","last_change":"2024-03-14T09:52:05.744174+0000","last_active":"2024-03-14T09:52:41.140932+0000","last_peered":"2024-03-14T09:52:41.140932+0000","last_clean":"2024-03-14T09:52:41.140932+0000","last_became_active":"2024-03-14T09:52:05.743848+0000","last_became_peered":"2024-03-14T09:52:05.743848+0000","last_unstale":"2024-03-14T09:52:41.140932+0000","last_undegraded":"2024-03-14T09:52:41.140932+0000","last_fullsized":"2024-03-14T09:52:41.140932+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:07:23.260734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143348+0000","last_change":"2024-03-14T09:52:05.733677+0000","last_active":"2024-03-14T09:52:41.143348+0000","last_peered":"2024-03-14T09:52:41.143348+0000","last_clean":"2024-03-14T09:52:41.143348+0000","last_became_active":"2024-03-14T09:52:05.730967+0000","last_became_peered":"2024-03-14T09:52:05.730967+0000","last_unstale":"2024-03-14T09:52:41.143348+0000","last_undegraded":"2024-03-14T09:52:41.143348+0000","last_fullsized":"2024-03-14T09:52:41.143348+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:33:53.484741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141583+0000","last_change":"2024-03-14T09:52:05.737836+0000","last_active":"2024-03-14T09:52:41.141583+0000","last_peered":"2024-03-14T09:52:41.141583+0000","last_clean":"2024-03-14T09:52:41.141583+0000","last_became_active":"2024-03-14T09:52:05.737257+0000","last_became_peered":"2024-03-14T09:52:05.737257+0000","last_unstale":"2024-03-14T09:52:41.141583+0000","last_undegraded":"2024-03-14T09:52:41.141583+0000","last_fullsized":"2024-03-14T09:52:41.141583+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:53.723415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142940+0000","last_change":"2024-03-14T09:52:05.741360+0000","last_active":"2024-03-14T09:52:41.142940+0000","last_peered":"2024-03-14T09:52:41.142940+0000","last_clean":"2024-03-14T09:52:41.142940+0000","last_became_active":"2024-03-14T09:52:05.740673+0000","last_became_peered":"2024-03-14T09:52:05.740673+0000","last_unstale":"2024-03-14T09:52:41.142940+0000","last_undegraded":"2024-03-14T09:52:41.142940+0000","last_fullsized":"2024-03-14T09:52:41.142940+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:13:41.561338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143917+0000","last_change":"2024-03-14T09:52:05.739134+0000","last_active":"2024-03-14T09:52:41.143917+0000","last_peered":"2024-03-14T09:52:41.143917+0000","last_clean":"2024-03-14T09:52:41.143917+0000","last_became_active":"2024-03-14T09:52:05.738439+0000","last_became_peered":"2024-03-14T09:52:05.738439+0000","last_unstale":"2024-03-14T09:52:41.143917+0000","last_undegraded":"2024-03-14T09:52:41.143917+0000","last_fullsized":"2024-03-14T09:52:41.143917+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:05:36.997462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142821+0000","last_change":"2024-03-14T09:52:09.769284+0000","last_active":"2024-03-14T09:52:41.142821+0000","last_peered":"2024-03-14T09:52:41.142821+0000","last_clean":"2024-03-14T09:52:41.142821+0000","last_became_active":"2024-03-14T09:52:09.768808+0000","last_became_peered":"2024-03-14T09:52:09.768808+0000","last_unstale":"2024-03-14T09:52:41.142821+0000","last_undegraded":"2024-03-14T09:52:41.142821+0000","last_fullsized":"2024-03-14T09:52:41.142821+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:48:56.015410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"83'9","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.154303+0000","last_change":"2024-03-14T09:52:42.154303+0000","last_active":"2024-03-14T09:52:42.154303+0000","last_peered":"2024-03-14T09:52:42.154303+0000","last_clean":"2024-03-14T09:52:42.154303+0000","last_became_active":"2024-03-14T09:52:42.147756+0000","last_became_peered":"2024-03-14T09:52:42.147756+0000","last_unstale":"2024-03-14T09:52:42.154303+0000","last_undegraded":"2024-03-14T09:52:42.154303+0000","last_fullsized":"2024-03-14T09:52:42.154303+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:01:27.630337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145550+0000","last_change":"2024-03-14T09:52:05.739004+0000","last_active":"2024-03-14T09:52:41.145550+0000","last_peered":"2024-03-14T09:52:41.145550+0000","last_clean":"2024-03-14T09:52:41.145550+0000","last_became_active":"2024-03-14T09:52:05.732217+0000","last_became_peered":"2024-03-14T09:52:05.732217+0000","last_unstale":"2024-03-14T09:52:41.145550+0000","last_undegraded":"2024-03-14T09:52:41.145550+0000","last_fullsized":"2024-03-14T09:52:41.145550+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:02:21.926623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140568+0000","last_change":"2024-03-14T09:52:09.776128+0000","last_active":"2024-03-14T09:52:41.140568+0000","last_peered":"2024-03-14T09:52:41.140568+0000","last_clean":"2024-03-14T09:52:41.140568+0000","last_became_active":"2024-03-14T09:52:09.775701+0000","last_became_peered":"2024-03-14T09:52:09.775701+0000","last_unstale":"2024-03-14T09:52:41.140568+0000","last_undegraded":"2024-03-14T09:52:41.140568+0000","last_fullsized":"2024-03-14T09:52:41.140568+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:41:51.790627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"83'2","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146080+0000","last_change":"2024-03-14T09:52:05.234269+0000","last_active":"2024-03-14T09:52:41.146080+0000","last_peered":"2024-03-14T09:52:41.146080+0000","last_clean":"2024-03-14T09:52:41.146080+0000","last_became_active":"2024-03-14T09:52:05.233418+0000","last_became_peered":"2024-03-14T09:52:05.233418+0000","last_unstale":"2024-03-14T09:52:41.146080+0000","last_undegraded":"2024-03-14T09:52:41.146080+0000","last_fullsized":"2024-03-14T09:52:41.146080+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:22:46.894461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143057+0000","last_change":"2024-03-14T09:52:05.741786+0000","last_active":"2024-03-14T09:52:41.143057+0000","last_peered":"2024-03-14T09:52:41.143057+0000","last_clean":"2024-03-14T09:52:41.143057+0000","last_became_active":"2024-03-14T09:52:05.741461+0000","last_became_peered":"2024-03-14T09:52:05.741461+0000","last_unstale":"2024-03-14T09:52:41.143057+0000","last_undegraded":"2024-03-14T09:52:41.143057+0000","last_fullsized":"2024-03-14T09:52:41.143057+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:21:17.409530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141735+0000","last_change":"2024-03-14T09:52:09.768851+0000","last_active":"2024-03-14T09:52:41.141735+0000","last_peered":"2024-03-14T09:52:41.141735+0000","last_clean":"2024-03-14T09:52:41.141735+0000","last_became_active":"2024-03-14T09:52:09.768485+0000","last_became_peered":"2024-03-14T09:52:09.768485+0000","last_unstale":"2024-03-14T09:52:41.141735+0000","last_undegraded":"2024-03-14T09:52:41.141735+0000","last_fullsized":"2024-03-14T09:52:41.141735+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:00:51.189790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"83'4","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.154227+0000","last_change":"2024-03-14T09:52:42.154227+0000","last_active":"2024-03-14T09:52:42.154227+0000","last_peered":"2024-03-14T09:52:42.154227+0000","last_clean":"2024-03-14T09:52:42.154227+0000","last_became_active":"2024-03-14T09:52:42.149841+0000","last_became_peered":"2024-03-14T09:52:42.149841+0000","last_unstale":"2024-03-14T09:52:42.154227+0000","last_undegraded":"2024-03-14T09:52:42.154227+0000","last_fullsized":"2024-03-14T09:52:42.154227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:04:46.911921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.144336+0000","last_change":"2024-03-14T09:52:05.742994+0000","last_active":"2024-03-14T09:52:41.144336+0000","last_peered":"2024-03-14T09:52:41.144336+0000","last_clean":"2024-03-14T09:52:41.144336+0000","last_became_active":"2024-03-14T09:52:05.742652+0000","last_became_peered":"2024-03-14T09:52:05.742652+0000","last_unstale":"2024-03-14T09:52:41.144336+0000","last_undegraded":"2024-03-14T09:52:41.144336+0000","last_fullsized":"2024-03-14T09:52:41.144336+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:46:47.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140467+0000","last_change":"2024-03-14T09:52:09.763009+0000","last_active":"2024-03-14T09:52:41.140467+0000","last_peered":"2024-03-14T09:52:41.140467+0000","last_clean":"2024-03-14T09:52:41.140467+0000","last_became_active":"2024-03-14T09:52:09.762621+0000","last_became_peered":"2024-03-14T09:52:09.762621+0000","last_unstale":"2024-03-14T09:52:41.140467+0000","last_undegraded":"2024-03-14T09:52:41.140467+0000","last_fullsized":"2024-03-14T09:52:41.140467+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:56:22.162409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137429+0000","last_change":"2024-03-14T09:52:05.234039+0000","last_active":"2024-03-14T09:52:41.137429+0000","last_peered":"2024-03-14T09:52:41.137429+0000","last_clean":"2024-03-14T09:52:41.137429+0000","last_became_active":"2024-03-14T09:52:05.233612+0000","last_became_peered":"2024-03-14T09:52:05.233612+0000","last_unstale":"2024-03-14T09:52:41.137429+0000","last_undegraded":"2024-03-14T09:52:41.137429+0000","last_fullsized":"2024-03-14T09:52:41.137429+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:29:52.968178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.426506+0000","last_change":"2024-03-14T09:52:42.426506+0000","last_active":"2024-03-14T09:52:42.426506+0000","last_peered":"2024-03-14T09:52:42.426506+0000","last_clean":"2024-03-14T09:52:42.426506+0000","last_became_active":"2024-03-14T09:52:42.426062+0000","last_became_peered":"2024-03-14T09:52:42.426062+0000","last_unstale":"2024-03-14T09:52:42.426506+0000","last_undegraded":"2024-03-14T09:52:42.426506+0000","last_fullsized":"2024-03-14T09:52:42.426506+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:23:42.943381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143755+0000","last_change":"2024-03-14T09:52:09.757856+0000","last_active":"2024-03-14T09:52:41.143755+0000","last_peered":"2024-03-14T09:52:41.143755+0000","last_clean":"2024-03-14T09:52:41.143755+0000","last_became_active":"2024-03-14T09:52:09.757395+0000","last_became_peered":"2024-03-14T09:52:09.757395+0000","last_unstale":"2024-03-14T09:52:41.143755+0000","last_undegraded":"2024-03-14T09:52:41.143755+0000","last_fullsized":"2024-03-14T09:52:41.143755+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:34:06.600727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"83'5","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:53:05.776205+0000","last_change":"2024-03-14T09:52:05.228527+0000","last_active":"2024-03-14T09:53:05.776205+0000","last_peered":"2024-03-14T09:53:05.776205+0000","last_clean":"2024-03-14T09:53:05.776205+0000","last_became_active":"2024-03-14T09:52:05.228177+0000","last_became_peered":"2024-03-14T09:52:05.228177+0000","last_unstale":"2024-03-14T09:53:05.776205+0000","last_undegraded":"2024-03-14T09:53:05.776205+0000","last_fullsized":"2024-03-14T09:53:05.776205+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:17:57.529308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.138384+0000","last_change":"2024-03-14T09:52:05.734041+0000","last_active":"2024-03-14T09:52:41.138384+0000","last_peered":"2024-03-14T09:52:41.138384+0000","last_clean":"2024-03-14T09:52:41.138384+0000","last_became_active":"2024-03-14T09:52:05.732491+0000","last_became_peered":"2024-03-14T09:52:05.732491+0000","last_unstale":"2024-03-14T09:52:41.138384+0000","last_undegraded":"2024-03-14T09:52:41.138384+0000","last_fullsized":"2024-03-14T09:52:41.138384+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:07:31.909382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143044+0000","last_change":"2024-03-14T09:52:09.769584+0000","last_active":"2024-03-14T09:52:41.143044+0000","last_peered":"2024-03-14T09:52:41.143044+0000","last_clean":"2024-03-14T09:52:41.143044+0000","last_became_active":"2024-03-14T09:52:09.768847+0000","last_became_peered":"2024-03-14T09:52:09.768847+0000","last_unstale":"2024-03-14T09:52:41.143044+0000","last_undegraded":"2024-03-14T09:52:41.143044+0000","last_fullsized":"2024-03-14T09:52:41.143044+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:12:18.005122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"83'3","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142733+0000","last_change":"2024-03-14T09:52:05.228893+0000","last_active":"2024-03-14T09:52:41.142733+0000","last_peered":"2024-03-14T09:52:41.142733+0000","last_clean":"2024-03-14T09:52:41.142733+0000","last_became_active":"2024-03-14T09:52:05.228569+0000","last_became_peered":"2024-03-14T09:52:05.228569+0000","last_unstale":"2024-03-14T09:52:41.142733+0000","last_undegraded":"2024-03-14T09:52:41.142733+0000","last_fullsized":"2024-03-14T09:52:41.142733+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:28:03.200437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146405+0000","last_change":"2024-03-14T09:52:05.736388+0000","last_active":"2024-03-14T09:52:41.146405+0000","last_peered":"2024-03-14T09:52:41.146405+0000","last_clean":"2024-03-14T09:52:41.146405+0000","last_became_active":"2024-03-14T09:52:05.735131+0000","last_became_peered":"2024-03-14T09:52:05.735131+0000","last_unstale":"2024-03-14T09:52:41.146405+0000","last_undegraded":"2024-03-14T09:52:41.146405+0000","last_fullsized":"2024-03-14T09:52:41.146405+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:00:54.120087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141484+0000","last_change":"2024-03-14T09:52:09.770321+0000","last_active":"2024-03-14T09:52:41.141484+0000","last_peered":"2024-03-14T09:52:41.141484+0000","last_clean":"2024-03-14T09:52:41.141484+0000","last_became_active":"2024-03-14T09:52:09.769710+0000","last_became_peered":"2024-03-14T09:52:09.769710+0000","last_unstale":"2024-03-14T09:52:41.141484+0000","last_undegraded":"2024-03-14T09:52:41.141484+0000","last_fullsized":"2024-03-14T09:52:41.141484+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:12:12.635248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"83'2","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136475+0000","last_change":"2024-03-14T09:52:05.223912+0000","last_active":"2024-03-14T09:52:41.136475+0000","last_peered":"2024-03-14T09:52:41.136475+0000","last_clean":"2024-03-14T09:52:41.136475+0000","last_became_active":"2024-03-14T09:52:05.223380+0000","last_became_peered":"2024-03-14T09:52:05.223380+0000","last_unstale":"2024-03-14T09:52:41.136475+0000","last_undegraded":"2024-03-14T09:52:41.136475+0000","last_fullsized":"2024-03-14T09:52:41.136475+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:08:33.098215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145370+0000","last_change":"2024-03-14T09:52:05.743277+0000","last_active":"2024-03-14T09:52:41.145370+0000","last_peered":"2024-03-14T09:52:41.145370+0000","last_clean":"2024-03-14T09:52:41.145370+0000","last_became_active":"2024-03-14T09:52:05.741386+0000","last_became_peered":"2024-03-14T09:52:05.741386+0000","last_unstale":"2024-03-14T09:52:41.145370+0000","last_undegraded":"2024-03-14T09:52:41.145370+0000","last_fullsized":"2024-03-14T09:52:41.145370+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:08:08.320279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141726+0000","last_change":"2024-03-14T09:52:09.766485+0000","last_active":"2024-03-14T09:52:41.141726+0000","last_peered":"2024-03-14T09:52:41.141726+0000","last_clean":"2024-03-14T09:52:41.141726+0000","last_became_active":"2024-03-14T09:52:09.766115+0000","last_became_peered":"2024-03-14T09:52:09.766115+0000","last_unstale":"2024-03-14T09:52:41.141726+0000","last_undegraded":"2024-03-14T09:52:41.141726+0000","last_fullsized":"2024-03-14T09:52:41.141726+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:34:45.067757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"83'1","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145976+0000","last_change":"2024-03-14T09:52:05.233852+0000","last_active":"2024-03-14T09:52:41.145976+0000","last_peered":"2024-03-14T09:52:41.145976+0000","last_clean":"2024-03-14T09:52:41.145976+0000","last_became_active":"2024-03-14T09:52:05.233548+0000","last_became_peered":"2024-03-14T09:52:05.233548+0000","last_unstale":"2024-03-14T09:52:41.145976+0000","last_undegraded":"2024-03-14T09:52:41.145976+0000","last_fullsized":"2024-03-14T09:52:41.145976+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:08:51.821465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141954+0000","last_change":"2024-03-14T09:52:05.731215+0000","last_active":"2024-03-14T09:52:41.141954+0000","last_peered":"2024-03-14T09:52:41.141954+0000","last_clean":"2024-03-14T09:52:41.141954+0000","last_became_active":"2024-03-14T09:52:05.730924+0000","last_became_peered":"2024-03-14T09:52:05.730924+0000","last_unstale":"2024-03-14T09:52:41.141954+0000","last_undegraded":"2024-03-14T09:52:41.141954+0000","last_fullsized":"2024-03-14T09:52:41.141954+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:12:17.560297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145920+0000","last_change":"2024-03-14T09:52:09.753775+0000","last_active":"2024-03-14T09:52:41.145920+0000","last_peered":"2024-03-14T09:52:41.145920+0000","last_clean":"2024-03-14T09:52:41.145920+0000","last_became_active":"2024-03-14T09:52:09.753422+0000","last_became_peered":"2024-03-14T09:52:09.753422+0000","last_unstale":"2024-03-14T09:52:41.145920+0000","last_undegraded":"2024-03-14T09:52:41.145920+0000","last_fullsized":"2024-03-14T09:52:41.145920+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:59:25.625909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"83'3","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:53:06.776870+0000","last_change":"2024-03-14T09:52:05.229064+0000","last_active":"2024-03-14T09:53:06.776870+0000","last_peered":"2024-03-14T09:53:06.776870+0000","last_clean":"2024-03-14T09:53:06.776870+0000","last_became_active":"2024-03-14T09:52:05.228751+0000","last_became_peered":"2024-03-14T09:52:05.228751+0000","last_unstale":"2024-03-14T09:53:06.776870+0000","last_undegraded":"2024-03-14T09:53:06.776870+0000","last_fullsized":"2024-03-14T09:53:06.776870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:13:29.832866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145697+0000","last_change":"2024-03-14T09:52:05.737798+0000","last_active":"2024-03-14T09:52:41.145697+0000","last_peered":"2024-03-14T09:52:41.145697+0000","last_clean":"2024-03-14T09:52:41.145697+0000","last_became_active":"2024-03-14T09:52:05.737449+0000","last_became_peered":"2024-03-14T09:52:05.737449+0000","last_unstale":"2024-03-14T09:52:41.145697+0000","last_undegraded":"2024-03-14T09:52:41.145697+0000","last_fullsized":"2024-03-14T09:52:41.145697+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:40:10.079770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141907+0000","last_change":"2024-03-14T09:52:09.768368+0000","last_active":"2024-03-14T09:52:41.141907+0000","last_peered":"2024-03-14T09:52:41.141907+0000","last_clean":"2024-03-14T09:52:41.141907+0000","last_became_active":"2024-03-14T09:52:09.767622+0000","last_became_peered":"2024-03-14T09:52:09.767622+0000","last_unstale":"2024-03-14T09:52:41.141907+0000","last_undegraded":"2024-03-14T09:52:41.141907+0000","last_fullsized":"2024-03-14T09:52:41.141907+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:19:28.836317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"83'2","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137100+0000","last_change":"2024-03-14T09:52:05.229778+0000","last_active":"2024-03-14T09:52:41.137100+0000","last_peered":"2024-03-14T09:52:41.137100+0000","last_clean":"2024-03-14T09:52:41.137100+0000","last_became_active":"2024-03-14T09:52:05.229468+0000","last_became_peered":"2024-03-14T09:52:05.229468+0000","last_unstale":"2024-03-14T09:52:41.137100+0000","last_undegraded":"2024-03-14T09:52:41.137100+0000","last_fullsized":"2024-03-14T09:52:41.137100+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:49:58.433646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143199+0000","last_change":"2024-03-14T09:52:05.751027+0000","last_active":"2024-03-14T09:52:41.143199+0000","last_peered":"2024-03-14T09:52:41.143199+0000","last_clean":"2024-03-14T09:52:41.143199+0000","last_became_active":"2024-03-14T09:52:05.750650+0000","last_became_peered":"2024-03-14T09:52:05.750650+0000","last_unstale":"2024-03-14T09:52:41.143199+0000","last_undegraded":"2024-03-14T09:52:41.143199+0000","last_fullsized":"2024-03-14T09:52:41.143199+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:05:37.264396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141928+0000","last_change":"2024-03-14T09:52:09.768207+0000","last_active":"2024-03-14T09:52:41.141928+0000","last_peered":"2024-03-14T09:52:41.141928+0000","last_clean":"2024-03-14T09:52:41.141928+0000","last_became_active":"2024-03-14T09:52:09.767756+0000","last_became_peered":"2024-03-14T09:52:09.767756+0000","last_unstale":"2024-03-14T09:52:41.141928+0000","last_undegraded":"2024-03-14T09:52:41.141928+0000","last_fullsized":"2024-03-14T09:52:41.141928+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:13:12.544995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"83'4","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146023+0000","last_change":"2024-03-14T09:52:05.235472+0000","last_active":"2024-03-14T09:52:41.146023+0000","last_peered":"2024-03-14T09:52:41.146023+0000","last_clean":"2024-03-14T09:52:41.146023+0000","last_became_active":"2024-03-14T09:52:05.235106+0000","last_became_peered":"2024-03-14T09:52:05.235106+0000","last_unstale":"2024-03-14T09:52:41.146023+0000","last_undegraded":"2024-03-14T09:52:41.146023+0000","last_fullsized":"2024-03-14T09:52:41.146023+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:30:36.446130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.144505+0000","last_change":"2024-03-14T09:52:05.747615+0000","last_active":"2024-03-14T09:52:41.144505+0000","last_peered":"2024-03-14T09:52:41.144505+0000","last_clean":"2024-03-14T09:52:41.144505+0000","last_became_active":"2024-03-14T09:52:05.747344+0000","last_became_peered":"2024-03-14T09:52:05.747344+0000","last_unstale":"2024-03-14T09:52:41.144505+0000","last_undegraded":"2024-03-14T09:52:41.144505+0000","last_fullsized":"2024-03-14T09:52:41.144505+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:52:23.096124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141847+0000","last_change":"2024-03-14T09:52:09.768295+0000","last_active":"2024-03-14T09:52:41.141847+0000","last_peered":"2024-03-14T09:52:41.141847+0000","last_clean":"2024-03-14T09:52:41.141847+0000","last_became_active":"2024-03-14T09:52:09.760997+0000","last_became_peered":"2024-03-14T09:52:09.760997+0000","last_unstale":"2024-03-14T09:52:41.141847+0000","last_undegraded":"2024-03-14T09:52:41.141847+0000","last_fullsized":"2024-03-14T09:52:41.141847+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:22:09.909302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'31","reported_seq":132,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:53:07.788981+0000","last_change":"2024-03-14T09:52:05.227876+0000","last_active":"2024-03-14T09:53:07.788981+0000","last_peered":"2024-03-14T09:53:07.788981+0000","last_clean":"2024-03-14T09:53:07.788981+0000","last_became_active":"2024-03-14T09:52:05.227519+0000","last_became_peered":"2024-03-14T09:52:05.227519+0000","last_unstale":"2024-03-14T09:53:07.788981+0000","last_undegraded":"2024-03-14T09:53:07.788981+0000","last_fullsized":"2024-03-14T09:53:07.788981+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:12.616609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61729,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":83,"num_read_kb":102,"num_write":32,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137730+0000","last_change":"2024-03-14T09:52:05.737432+0000","last_active":"2024-03-14T09:52:41.137730+0000","last_peered":"2024-03-14T09:52:41.137730+0000","last_clean":"2024-03-14T09:52:41.137730+0000","last_became_active":"2024-03-14T09:52:05.735050+0000","last_became_peered":"2024-03-14T09:52:05.735050+0000","last_unstale":"2024-03-14T09:52:41.137730+0000","last_undegraded":"2024-03-14T09:52:41.137730+0000","last_fullsized":"2024-03-14T09:52:41.137730+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T09:57:45.383155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146516+0000","last_change":"2024-03-14T09:52:09.754909+0000","last_active":"2024-03-14T09:52:41.146516+0000","last_peered":"2024-03-14T09:52:41.146516+0000","last_clean":"2024-03-14T09:52:41.146516+0000","last_became_active":"2024-03-14T09:52:09.754367+0000","last_became_peered":"2024-03-14T09:52:09.754367+0000","last_unstale":"2024-03-14T09:52:41.146516+0000","last_undegraded":"2024-03-14T09:52:41.146516+0000","last_fullsized":"2024-03-14T09:52:41.146516+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:52:34.251555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"83'6","reported_seq":25,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.144607+0000","last_change":"2024-03-14T09:52:05.224869+0000","last_active":"2024-03-14T09:52:41.144607+0000","last_peered":"2024-03-14T09:52:41.144607+0000","last_clean":"2024-03-14T09:52:41.144607+0000","last_became_active":"2024-03-14T09:52:05.224499+0000","last_became_peered":"2024-03-14T09:52:05.224499+0000","last_unstale":"2024-03-14T09:52:41.144607+0000","last_undegraded":"2024-03-14T09:52:41.144607+0000","last_fullsized":"2024-03-14T09:52:41.144607+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:49:01.729417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136754+0000","last_change":"2024-03-14T09:52:05.729168+0000","last_active":"2024-03-14T09:52:41.136754+0000","last_peered":"2024-03-14T09:52:41.136754+0000","last_clean":"2024-03-14T09:52:41.136754+0000","last_became_active":"2024-03-14T09:52:05.728806+0000","last_became_peered":"2024-03-14T09:52:05.728806+0000","last_unstale":"2024-03-14T09:52:41.136754+0000","last_undegraded":"2024-03-14T09:52:41.136754+0000","last_fullsized":"2024-03-14T09:52:41.136754+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:54:36.758853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140504+0000","last_change":"2024-03-14T09:52:09.770126+0000","last_active":"2024-03-14T09:52:41.140504+0000","last_peered":"2024-03-14T09:52:41.140504+0000","last_clean":"2024-03-14T09:52:41.140504+0000","last_became_active":"2024-03-14T09:52:09.769734+0000","last_became_peered":"2024-03-14T09:52:09.769734+0000","last_unstale":"2024-03-14T09:52:41.140504+0000","last_undegraded":"2024-03-14T09:52:41.140504+0000","last_fullsized":"2024-03-14T09:52:41.140504+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:50:37.839858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"83'3","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146334+0000","last_change":"2024-03-14T09:52:05.235201+0000","last_active":"2024-03-14T09:52:41.146334+0000","last_peered":"2024-03-14T09:52:41.146334+0000","last_clean":"2024-03-14T09:52:41.146334+0000","last_became_active":"2024-03-14T09:52:05.234816+0000","last_became_peered":"2024-03-14T09:52:05.234816+0000","last_unstale":"2024-03-14T09:52:41.146334+0000","last_undegraded":"2024-03-14T09:52:41.146334+0000","last_fullsized":"2024-03-14T09:52:41.146334+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:01:50.070913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137236+0000","last_change":"2024-03-14T09:52:05.735304+0000","last_active":"2024-03-14T09:52:41.137236+0000","last_peered":"2024-03-14T09:52:41.137236+0000","last_clean":"2024-03-14T09:52:41.137236+0000","last_became_active":"2024-03-14T09:52:05.734977+0000","last_became_peered":"2024-03-14T09:52:05.734977+0000","last_unstale":"2024-03-14T09:52:41.137236+0000","last_undegraded":"2024-03-14T09:52:41.137236+0000","last_fullsized":"2024-03-14T09:52:41.137236+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:08:28.071962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142729+0000","last_change":"2024-03-14T09:52:09.770721+0000","last_active":"2024-03-14T09:52:41.142729+0000","last_peered":"2024-03-14T09:52:41.142729+0000","last_clean":"2024-03-14T09:52:41.142729+0000","last_became_active":"2024-03-14T09:52:09.770270+0000","last_became_peered":"2024-03-14T09:52:09.770270+0000","last_unstale":"2024-03-14T09:52:41.142729+0000","last_undegraded":"2024-03-14T09:52:41.142729+0000","last_fullsized":"2024-03-14T09:52:41.142729+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:09:33.480235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"83'4","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:53:08.785465+0000","last_change":"2024-03-14T09:52:05.227260+0000","last_active":"2024-03-14T09:53:08.785465+0000","last_peered":"2024-03-14T09:53:08.785465+0000","last_clean":"2024-03-14T09:53:08.785465+0000","last_became_active":"2024-03-14T09:52:05.226922+0000","last_became_peered":"2024-03-14T09:52:05.226922+0000","last_unstale":"2024-03-14T09:53:08.785465+0000","last_undegraded":"2024-03-14T09:53:08.785465+0000","last_fullsized":"2024-03-14T09:53:08.785465+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:36:09.770478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":932,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137037+0000","last_change":"2024-03-14T09:52:05.734278+0000","last_active":"2024-03-14T09:52:41.137037+0000","last_peered":"2024-03-14T09:52:41.137037+0000","last_clean":"2024-03-14T09:52:41.137037+0000","last_became_active":"2024-03-14T09:52:05.732051+0000","last_became_peered":"2024-03-14T09:52:05.732051+0000","last_unstale":"2024-03-14T09:52:41.137037+0000","last_undegraded":"2024-03-14T09:52:41.137037+0000","last_fullsized":"2024-03-14T09:52:41.137037+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:33:07.137983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136993+0000","last_change":"2024-03-14T09:52:09.764068+0000","last_active":"2024-03-14T09:52:41.136993+0000","last_peered":"2024-03-14T09:52:41.136993+0000","last_clean":"2024-03-14T09:52:41.136993+0000","last_became_active":"2024-03-14T09:52:09.763613+0000","last_became_peered":"2024-03-14T09:52:09.763613+0000","last_unstale":"2024-03-14T09:52:41.136993+0000","last_undegraded":"2024-03-14T09:52:41.136993+0000","last_fullsized":"2024-03-14T09:52:41.136993+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:32:15.524602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"83'15","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.438205+0000","last_change":"2024-03-14T09:52:42.438205+0000","last_active":"2024-03-14T09:52:42.438205+0000","last_peered":"2024-03-14T09:52:42.438205+0000","last_clean":"2024-03-14T09:52:42.438205+0000","last_became_active":"2024-03-14T09:52:42.426268+0000","last_became_peered":"2024-03-14T09:52:42.426268+0000","last_unstale":"2024-03-14T09:52:42.438205+0000","last_undegraded":"2024-03-14T09:52:42.438205+0000","last_fullsized":"2024-03-14T09:52:42.438205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:10:09.651131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145909+0000","last_change":"2024-03-14T09:52:05.734366+0000","last_active":"2024-03-14T09:52:41.145909+0000","last_peered":"2024-03-14T09:52:41.145909+0000","last_clean":"2024-03-14T09:52:41.145909+0000","last_became_active":"2024-03-14T09:52:05.734058+0000","last_became_peered":"2024-03-14T09:52:05.734058+0000","last_unstale":"2024-03-14T09:52:41.145909+0000","last_undegraded":"2024-03-14T09:52:41.145909+0000","last_fullsized":"2024-03-14T09:52:41.145909+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:49:49.233683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145893+0000","last_change":"2024-03-14T09:52:09.760241+0000","last_active":"2024-03-14T09:52:41.145893+0000","last_peered":"2024-03-14T09:52:41.145893+0000","last_clean":"2024-03-14T09:52:41.145893+0000","last_became_active":"2024-03-14T09:52:09.759797+0000","last_became_peered":"2024-03-14T09:52:09.759797+0000","last_unstale":"2024-03-14T09:52:41.145893+0000","last_undegraded":"2024-03-14T09:52:41.145893+0000","last_fullsized":"2024-03-14T09:52:41.145893+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:50:33.035775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"83'2","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.138494+0000","last_change":"2024-03-14T09:52:05.232932+0000","last_active":"2024-03-14T09:52:41.138494+0000","last_peered":"2024-03-14T09:52:41.138494+0000","last_clean":"2024-03-14T09:52:41.138494+0000","last_became_active":"2024-03-14T09:52:05.232608+0000","last_became_peered":"2024-03-14T09:52:05.232608+0000","last_unstale":"2024-03-14T09:52:41.138494+0000","last_undegraded":"2024-03-14T09:52:41.138494+0000","last_fullsized":"2024-03-14T09:52:41.138494+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:43:59.613531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141511+0000","last_change":"2024-03-14T09:52:05.733762+0000","last_active":"2024-03-14T09:52:41.141511+0000","last_peered":"2024-03-14T09:52:41.141511+0000","last_clean":"2024-03-14T09:52:41.141511+0000","last_became_active":"2024-03-14T09:52:05.733417+0000","last_became_peered":"2024-03-14T09:52:05.733417+0000","last_unstale":"2024-03-14T09:52:41.141511+0000","last_undegraded":"2024-03-14T09:52:41.141511+0000","last_fullsized":"2024-03-14T09:52:41.141511+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:11:12.593952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.426896+0000","last_change":"2024-03-14T09:52:42.426896+0000","last_active":"2024-03-14T09:52:42.426896+0000","last_peered":"2024-03-14T09:52:42.426896+0000","last_clean":"2024-03-14T09:52:42.426896+0000","last_became_active":"2024-03-14T09:52:42.426287+0000","last_became_peered":"2024-03-14T09:52:42.426287+0000","last_unstale":"2024-03-14T09:52:42.426896+0000","last_undegraded":"2024-03-14T09:52:42.426896+0000","last_fullsized":"2024-03-14T09:52:42.426896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:53:16.523707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142321+0000","last_change":"2024-03-14T09:52:05.227636+0000","last_active":"2024-03-14T09:52:41.142321+0000","last_peered":"2024-03-14T09:52:41.142321+0000","last_clean":"2024-03-14T09:52:41.142321+0000","last_became_active":"2024-03-14T09:52:05.227274+0000","last_became_peered":"2024-03-14T09:52:05.227274+0000","last_unstale":"2024-03-14T09:52:41.142321+0000","last_undegraded":"2024-03-14T09:52:41.142321+0000","last_fullsized":"2024-03-14T09:52:41.142321+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:13:25.380090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142838+0000","last_change":"2024-03-14T09:52:05.733158+0000","last_active":"2024-03-14T09:52:41.142838+0000","last_peered":"2024-03-14T09:52:41.142838+0000","last_clean":"2024-03-14T09:52:41.142838+0000","last_became_active":"2024-03-14T09:52:05.732812+0000","last_became_peered":"2024-03-14T09:52:05.732812+0000","last_unstale":"2024-03-14T09:52:41.142838+0000","last_undegraded":"2024-03-14T09:52:41.142838+0000","last_fullsized":"2024-03-14T09:52:41.142838+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:01:49.154860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136641+0000","last_change":"2024-03-14T09:52:09.761596+0000","last_active":"2024-03-14T09:52:41.136641+0000","last_peered":"2024-03-14T09:52:41.136641+0000","last_clean":"2024-03-14T09:52:41.136641+0000","last_became_active":"2024-03-14T09:52:09.761184+0000","last_became_peered":"2024-03-14T09:52:09.761184+0000","last_unstale":"2024-03-14T09:52:41.136641+0000","last_undegraded":"2024-03-14T09:52:41.136641+0000","last_fullsized":"2024-03-14T09:52:41.136641+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:49:02.567472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"83'1","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136845+0000","last_change":"2024-03-14T09:52:05.231828+0000","last_active":"2024-03-14T09:52:41.136845+0000","last_peered":"2024-03-14T09:52:41.136845+0000","last_clean":"2024-03-14T09:52:41.136845+0000","last_became_active":"2024-03-14T09:52:05.231490+0000","last_became_peered":"2024-03-14T09:52:05.231490+0000","last_unstale":"2024-03-14T09:52:41.136845+0000","last_undegraded":"2024-03-14T09:52:41.136845+0000","last_fullsized":"2024-03-14T09:52:41.136845+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:24:53.899728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142678+0000","last_change":"2024-03-14T09:52:05.744646+0000","last_active":"2024-03-14T09:52:41.142678+0000","last_peered":"2024-03-14T09:52:41.142678+0000","last_clean":"2024-03-14T09:52:41.142678+0000","last_became_active":"2024-03-14T09:52:05.744327+0000","last_became_peered":"2024-03-14T09:52:05.744327+0000","last_unstale":"2024-03-14T09:52:41.142678+0000","last_undegraded":"2024-03-14T09:52:41.142678+0000","last_fullsized":"2024-03-14T09:52:41.142678+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:46:50.643350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137005+0000","last_change":"2024-03-14T09:52:09.765374+0000","last_active":"2024-03-14T09:52:41.137005+0000","last_peered":"2024-03-14T09:52:41.137005+0000","last_clean":"2024-03-14T09:52:41.137005+0000","last_became_active":"2024-03-14T09:52:09.764939+0000","last_became_peered":"2024-03-14T09:52:09.764939+0000","last_unstale":"2024-03-14T09:52:41.137005+0000","last_undegraded":"2024-03-14T09:52:41.137005+0000","last_fullsized":"2024-03-14T09:52:41.137005+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:08:10.688746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146095+0000","last_change":"2024-03-14T09:52:05.729076+0000","last_active":"2024-03-14T09:52:41.146095+0000","last_peered":"2024-03-14T09:52:41.146095+0000","last_clean":"2024-03-14T09:52:41.146095+0000","last_became_active":"2024-03-14T09:52:05.728711+0000","last_became_peered":"2024-03-14T09:52:05.728711+0000","last_unstale":"2024-03-14T09:52:41.146095+0000","last_undegraded":"2024-03-14T09:52:41.146095+0000","last_fullsized":"2024-03-14T09:52:41.146095+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:11:05.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"83'4","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:53:08.784542+0000","last_change":"2024-03-14T09:52:05.236209+0000","last_active":"2024-03-14T09:53:08.784542+0000","last_peered":"2024-03-14T09:53:08.784542+0000","last_clean":"2024-03-14T09:53:08.784542+0000","last_became_active":"2024-03-14T09:52:05.235517+0000","last_became_peered":"2024-03-14T09:52:05.235517+0000","last_unstale":"2024-03-14T09:53:08.784542+0000","last_undegraded":"2024-03-14T09:53:08.784542+0000","last_fullsized":"2024-03-14T09:53:08.784542+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:09:54.275728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137171+0000","last_change":"2024-03-14T09:52:09.761095+0000","last_active":"2024-03-14T09:52:41.137171+0000","last_peered":"2024-03-14T09:52:41.137171+0000","last_clean":"2024-03-14T09:52:41.137171+0000","last_became_active":"2024-03-14T09:52:09.760662+0000","last_became_peered":"2024-03-14T09:52:09.760662+0000","last_unstale":"2024-03-14T09:52:41.137171+0000","last_undegraded":"2024-03-14T09:52:41.137171+0000","last_fullsized":"2024-03-14T09:52:41.137171+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:07:17.670605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145420+0000","last_change":"2024-03-14T09:52:05.736078+0000","last_active":"2024-03-14T09:52:41.145420+0000","last_peered":"2024-03-14T09:52:41.145420+0000","last_clean":"2024-03-14T09:52:41.145420+0000","last_became_active":"2024-03-14T09:52:05.735715+0000","last_became_peered":"2024-03-14T09:52:05.735715+0000","last_unstale":"2024-03-14T09:52:41.145420+0000","last_undegraded":"2024-03-14T09:52:41.145420+0000","last_fullsized":"2024-03-14T09:52:41.145420+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:41:32.863680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"83'3","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141835+0000","last_change":"2024-03-14T09:52:05.225756+0000","last_active":"2024-03-14T09:52:41.141835+0000","last_peered":"2024-03-14T09:52:41.141835+0000","last_clean":"2024-03-14T09:52:41.141835+0000","last_became_active":"2024-03-14T09:52:05.225378+0000","last_became_peered":"2024-03-14T09:52:05.225378+0000","last_unstale":"2024-03-14T09:52:41.141835+0000","last_undegraded":"2024-03-14T09:52:41.141835+0000","last_fullsized":"2024-03-14T09:52:41.141835+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:14:49.594636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":337,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136821+0000","last_change":"2024-03-14T09:51:11.588790+0000","last_active":"2024-03-14T09:52:41.136821+0000","last_peered":"2024-03-14T09:52:41.136821+0000","last_clean":"2024-03-14T09:52:41.136821+0000","last_became_active":"2024-03-14T09:51:11.588287+0000","last_became_peered":"2024-03-14T09:51:11.588287+0000","last_unstale":"2024-03-14T09:52:41.136821+0000","last_undegraded":"2024-03-14T09:52:41.136821+0000","last_fullsized":"2024-03-14T09:52:41.136821+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:48:37.651174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:48:37.651174+0000","last_clean_scrub_stamp":"2024-03-14T09:48:37.651174+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:06:24.252565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146527+0000","last_change":"2024-03-14T09:52:09.761344+0000","last_active":"2024-03-14T09:52:41.146527+0000","last_peered":"2024-03-14T09:52:41.146527+0000","last_clean":"2024-03-14T09:52:41.146527+0000","last_became_active":"2024-03-14T09:52:09.760899+0000","last_became_peered":"2024-03-14T09:52:09.760899+0000","last_unstale":"2024-03-14T09:52:41.146527+0000","last_undegraded":"2024-03-14T09:52:41.146527+0000","last_fullsized":"2024-03-14T09:52:41.146527+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:24:26.212157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136377+0000","last_change":"2024-03-14T09:52:05.738735+0000","last_active":"2024-03-14T09:52:41.136377+0000","last_peered":"2024-03-14T09:52:41.136377+0000","last_clean":"2024-03-14T09:52:41.136377+0000","last_became_active":"2024-03-14T09:52:05.738447+0000","last_became_peered":"2024-03-14T09:52:05.738447+0000","last_unstale":"2024-03-14T09:52:41.136377+0000","last_undegraded":"2024-03-14T09:52:41.136377+0000","last_fullsized":"2024-03-14T09:52:41.136377+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:03:24.598291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'3","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137056+0000","last_change":"2024-03-14T09:52:05.230107+0000","last_active":"2024-03-14T09:52:41.137056+0000","last_peered":"2024-03-14T09:52:41.137056+0000","last_clean":"2024-03-14T09:52:41.137056+0000","last_became_active":"2024-03-14T09:52:05.229812+0000","last_became_peered":"2024-03-14T09:52:05.229812+0000","last_unstale":"2024-03-14T09:52:41.137056+0000","last_undegraded":"2024-03-14T09:52:41.137056+0000","last_fullsized":"2024-03-14T09:52:41.137056+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:09:44.884867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141204+0000","last_change":"2024-03-14T09:52:09.763424+0000","last_active":"2024-03-14T09:52:41.141204+0000","last_peered":"2024-03-14T09:52:41.141204+0000","last_clean":"2024-03-14T09:52:41.141204+0000","last_became_active":"2024-03-14T09:52:09.763028+0000","last_became_peered":"2024-03-14T09:52:09.763028+0000","last_unstale":"2024-03-14T09:52:41.141204+0000","last_undegraded":"2024-03-14T09:52:41.141204+0000","last_fullsized":"2024-03-14T09:52:41.141204+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:47:31.415788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"83'1","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137766+0000","last_change":"2024-03-14T09:52:05.235099+0000","last_active":"2024-03-14T09:52:41.137766+0000","last_peered":"2024-03-14T09:52:41.137766+0000","last_clean":"2024-03-14T09:52:41.137766+0000","last_became_active":"2024-03-14T09:52:05.234798+0000","last_became_peered":"2024-03-14T09:52:05.234798+0000","last_unstale":"2024-03-14T09:52:41.137766+0000","last_undegraded":"2024-03-14T09:52:41.137766+0000","last_fullsized":"2024-03-14T09:52:41.137766+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:00:07.461931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136781+0000","last_change":"2024-03-14T09:52:05.742736+0000","last_active":"2024-03-14T09:52:41.136781+0000","last_peered":"2024-03-14T09:52:41.136781+0000","last_clean":"2024-03-14T09:52:41.136781+0000","last_became_active":"2024-03-14T09:52:05.740927+0000","last_became_peered":"2024-03-14T09:52:05.740927+0000","last_unstale":"2024-03-14T09:52:41.136781+0000","last_undegraded":"2024-03-14T09:52:41.136781+0000","last_fullsized":"2024-03-14T09:52:41.136781+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:50:17.738763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145386+0000","last_change":"2024-03-14T09:52:09.762722+0000","last_active":"2024-03-14T09:52:41.145386+0000","last_peered":"2024-03-14T09:52:41.145386+0000","last_clean":"2024-03-14T09:52:41.145386+0000","last_became_active":"2024-03-14T09:52:09.759053+0000","last_became_peered":"2024-03-14T09:52:09.759053+0000","last_unstale":"2024-03-14T09:52:41.145386+0000","last_undegraded":"2024-03-14T09:52:41.145386+0000","last_fullsized":"2024-03-14T09:52:41.145386+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:40:29.680002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"83'5","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142908+0000","last_change":"2024-03-14T09:52:05.224778+0000","last_active":"2024-03-14T09:52:41.142908+0000","last_peered":"2024-03-14T09:52:41.142908+0000","last_clean":"2024-03-14T09:52:41.142908+0000","last_became_active":"2024-03-14T09:52:05.223158+0000","last_became_peered":"2024-03-14T09:52:05.223158+0000","last_unstale":"2024-03-14T09:52:41.142908+0000","last_undegraded":"2024-03-14T09:52:41.142908+0000","last_fullsized":"2024-03-14T09:52:41.142908+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:01:29.910099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146027+0000","last_change":"2024-03-14T09:52:05.741753+0000","last_active":"2024-03-14T09:52:41.146027+0000","last_peered":"2024-03-14T09:52:41.146027+0000","last_clean":"2024-03-14T09:52:41.146027+0000","last_became_active":"2024-03-14T09:52:05.741451+0000","last_became_peered":"2024-03-14T09:52:05.741451+0000","last_unstale":"2024-03-14T09:52:41.146027+0000","last_undegraded":"2024-03-14T09:52:41.146027+0000","last_fullsized":"2024-03-14T09:52:41.146027+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:42:59.920909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142537+0000","last_change":"2024-03-14T09:52:09.755884+0000","last_active":"2024-03-14T09:52:41.142537+0000","last_peered":"2024-03-14T09:52:41.142537+0000","last_clean":"2024-03-14T09:52:41.142537+0000","last_became_active":"2024-03-14T09:52:09.755474+0000","last_became_peered":"2024-03-14T09:52:09.755474+0000","last_unstale":"2024-03-14T09:52:41.142537+0000","last_undegraded":"2024-03-14T09:52:41.142537+0000","last_fullsized":"2024-03-14T09:52:41.142537+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:59:23.383388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"83'4","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141058+0000","last_change":"2024-03-14T09:52:05.232673+0000","last_active":"2024-03-14T09:52:41.141058+0000","last_peered":"2024-03-14T09:52:41.141058+0000","last_clean":"2024-03-14T09:52:41.141058+0000","last_became_active":"2024-03-14T09:52:05.232296+0000","last_became_peered":"2024-03-14T09:52:05.232296+0000","last_unstale":"2024-03-14T09:52:41.141058+0000","last_undegraded":"2024-03-14T09:52:41.141058+0000","last_fullsized":"2024-03-14T09:52:41.141058+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:24:56.638451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143026+0000","last_change":"2024-03-14T09:52:05.744906+0000","last_active":"2024-03-14T09:52:41.143026+0000","last_peered":"2024-03-14T09:52:41.143026+0000","last_clean":"2024-03-14T09:52:41.143026+0000","last_became_active":"2024-03-14T09:52:05.743753+0000","last_became_peered":"2024-03-14T09:52:05.743753+0000","last_unstale":"2024-03-14T09:52:41.143026+0000","last_undegraded":"2024-03-14T09:52:41.143026+0000","last_fullsized":"2024-03-14T09:52:41.143026+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T09:59:40.358352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140739+0000","last_change":"2024-03-14T09:52:09.768355+0000","last_active":"2024-03-14T09:52:41.140739+0000","last_peered":"2024-03-14T09:52:41.140739+0000","last_clean":"2024-03-14T09:52:41.140739+0000","last_became_active":"2024-03-14T09:52:09.767919+0000","last_became_peered":"2024-03-14T09:52:09.767919+0000","last_unstale":"2024-03-14T09:52:41.140739+0000","last_undegraded":"2024-03-14T09:52:41.140739+0000","last_fullsized":"2024-03-14T09:52:41.140739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:45:08.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"83'2","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143444+0000","last_change":"2024-03-14T09:52:05.230343+0000","last_active":"2024-03-14T09:52:41.143444+0000","last_peered":"2024-03-14T09:52:41.143444+0000","last_clean":"2024-03-14T09:52:41.143444+0000","last_became_active":"2024-03-14T09:52:05.229974+0000","last_became_peered":"2024-03-14T09:52:05.229974+0000","last_unstale":"2024-03-14T09:52:41.143444+0000","last_undegraded":"2024-03-14T09:52:41.143444+0000","last_fullsized":"2024-03-14T09:52:41.143444+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:21:02.458108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137283+0000","last_change":"2024-03-14T09:52:05.738347+0000","last_active":"2024-03-14T09:52:41.137283+0000","last_peered":"2024-03-14T09:52:41.137283+0000","last_clean":"2024-03-14T09:52:41.137283+0000","last_became_active":"2024-03-14T09:52:05.737961+0000","last_became_peered":"2024-03-14T09:52:05.737961+0000","last_unstale":"2024-03-14T09:52:41.137283+0000","last_undegraded":"2024-03-14T09:52:41.137283+0000","last_fullsized":"2024-03-14T09:52:41.137283+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:02:57.252069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142859+0000","last_change":"2024-03-14T09:52:09.761930+0000","last_active":"2024-03-14T09:52:41.142859+0000","last_peered":"2024-03-14T09:52:41.142859+0000","last_clean":"2024-03-14T09:52:41.142859+0000","last_became_active":"2024-03-14T09:52:09.761515+0000","last_became_peered":"2024-03-14T09:52:09.761515+0000","last_unstale":"2024-03-14T09:52:41.142859+0000","last_undegraded":"2024-03-14T09:52:41.142859+0000","last_fullsized":"2024-03-14T09:52:41.142859+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T15:49:39.262277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"83'3","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145432+0000","last_change":"2024-03-14T09:52:05.233579+0000","last_active":"2024-03-14T09:52:41.145432+0000","last_peered":"2024-03-14T09:52:41.145432+0000","last_clean":"2024-03-14T09:52:41.145432+0000","last_became_active":"2024-03-14T09:52:05.231605+0000","last_became_peered":"2024-03-14T09:52:05.231605+0000","last_unstale":"2024-03-14T09:52:41.145432+0000","last_undegraded":"2024-03-14T09:52:41.145432+0000","last_fullsized":"2024-03-14T09:52:41.145432+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:55:38.282203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140457+0000","last_change":"2024-03-14T09:52:05.736586+0000","last_active":"2024-03-14T09:52:41.140457+0000","last_peered":"2024-03-14T09:52:41.140457+0000","last_clean":"2024-03-14T09:52:41.140457+0000","last_became_active":"2024-03-14T09:52:05.736235+0000","last_became_peered":"2024-03-14T09:52:05.736235+0000","last_unstale":"2024-03-14T09:52:41.140457+0000","last_undegraded":"2024-03-14T09:52:41.140457+0000","last_fullsized":"2024-03-14T09:52:41.140457+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:46:53.293698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"83'1","reported_seq":15,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140743+0000","last_change":"2024-03-14T09:52:09.768203+0000","last_active":"2024-03-14T09:52:41.140743+0000","last_peered":"2024-03-14T09:52:41.140743+0000","last_clean":"2024-03-14T09:52:41.140743+0000","last_became_active":"2024-03-14T09:52:09.767777+0000","last_became_peered":"2024-03-14T09:52:09.767777+0000","last_unstale":"2024-03-14T09:52:41.140743+0000","last_undegraded":"2024-03-14T09:52:41.140743+0000","last_fullsized":"2024-03-14T09:52:41.140743+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:20:47.598905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"83'1","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142704+0000","last_change":"2024-03-14T09:52:05.232359+0000","last_active":"2024-03-14T09:52:41.142704+0000","last_peered":"2024-03-14T09:52:41.142704+0000","last_clean":"2024-03-14T09:52:41.142704+0000","last_became_active":"2024-03-14T09:52:05.231977+0000","last_became_peered":"2024-03-14T09:52:05.231977+0000","last_unstale":"2024-03-14T09:52:41.142704+0000","last_undegraded":"2024-03-14T09:52:41.142704+0000","last_fullsized":"2024-03-14T09:52:41.142704+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:32.678157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145654+0000","last_change":"2024-03-14T09:52:05.735822+0000","last_active":"2024-03-14T09:52:41.145654+0000","last_peered":"2024-03-14T09:52:41.145654+0000","last_clean":"2024-03-14T09:52:41.145654+0000","last_became_active":"2024-03-14T09:52:05.735059+0000","last_became_peered":"2024-03-14T09:52:05.735059+0000","last_unstale":"2024-03-14T09:52:41.145654+0000","last_undegraded":"2024-03-14T09:52:41.145654+0000","last_fullsized":"2024-03-14T09:52:41.145654+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T15:22:33.033689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140934+0000","last_change":"2024-03-14T09:52:09.759794+0000","last_active":"2024-03-14T09:52:41.140934+0000","last_peered":"2024-03-14T09:52:41.140934+0000","last_clean":"2024-03-14T09:52:41.140934+0000","last_became_active":"2024-03-14T09:52:09.759366+0000","last_became_peered":"2024-03-14T09:52:09.759366+0000","last_unstale":"2024-03-14T09:52:41.140934+0000","last_undegraded":"2024-03-14T09:52:41.140934+0000","last_fullsized":"2024-03-14T09:52:41.140934+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:52:40.641264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"83'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.325733+0000","last_change":"2024-03-14T09:52:42.325733+0000","last_active":"2024-03-14T09:52:42.325733+0000","last_peered":"2024-03-14T09:52:42.325733+0000","last_clean":"2024-03-14T09:52:42.325733+0000","last_became_active":"2024-03-14T09:52:42.322002+0000","last_became_peered":"2024-03-14T09:52:42.322002+0000","last_unstale":"2024-03-14T09:52:42.325733+0000","last_undegraded":"2024-03-14T09:52:42.325733+0000","last_fullsized":"2024-03-14T09:52:42.325733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:22:10.439271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137057+0000","last_change":"2024-03-14T09:52:05.734073+0000","last_active":"2024-03-14T09:52:41.137057+0000","last_peered":"2024-03-14T09:52:41.137057+0000","last_clean":"2024-03-14T09:52:41.137057+0000","last_became_active":"2024-03-14T09:52:05.731748+0000","last_became_peered":"2024-03-14T09:52:05.731748+0000","last_unstale":"2024-03-14T09:52:41.137057+0000","last_undegraded":"2024-03-14T09:52:41.137057+0000","last_fullsized":"2024-03-14T09:52:41.137057+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:01:11.789162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143980+0000","last_change":"2024-03-14T09:52:09.763309+0000","last_active":"2024-03-14T09:52:41.143980+0000","last_peered":"2024-03-14T09:52:41.143980+0000","last_clean":"2024-03-14T09:52:41.143980+0000","last_became_active":"2024-03-14T09:52:09.762868+0000","last_became_peered":"2024-03-14T09:52:09.762868+0000","last_unstale":"2024-03-14T09:52:41.143980+0000","last_undegraded":"2024-03-14T09:52:41.143980+0000","last_fullsized":"2024-03-14T09:52:41.143980+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:30:03.403911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"83'3","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143238+0000","last_change":"2024-03-14T09:52:05.232310+0000","last_active":"2024-03-14T09:52:41.143238+0000","last_peered":"2024-03-14T09:52:41.143238+0000","last_clean":"2024-03-14T09:52:41.143238+0000","last_became_active":"2024-03-14T09:52:05.231931+0000","last_became_peered":"2024-03-14T09:52:05.231931+0000","last_unstale":"2024-03-14T09:52:41.143238+0000","last_undegraded":"2024-03-14T09:52:41.143238+0000","last_fullsized":"2024-03-14T09:52:41.143238+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:35:44.274800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137430+0000","last_change":"2024-03-14T09:52:05.731797+0000","last_active":"2024-03-14T09:52:41.137430+0000","last_peered":"2024-03-14T09:52:41.137430+0000","last_clean":"2024-03-14T09:52:41.137430+0000","last_became_active":"2024-03-14T09:52:05.730478+0000","last_became_peered":"2024-03-14T09:52:05.730478+0000","last_unstale":"2024-03-14T09:52:41.137430+0000","last_undegraded":"2024-03-14T09:52:41.137430+0000","last_fullsized":"2024-03-14T09:52:41.137430+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:52:50.684430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140879+0000","last_change":"2024-03-14T09:52:09.772786+0000","last_active":"2024-03-14T09:52:41.140879+0000","last_peered":"2024-03-14T09:52:41.140879+0000","last_clean":"2024-03-14T09:52:41.140879+0000","last_became_active":"2024-03-14T09:52:09.772198+0000","last_became_peered":"2024-03-14T09:52:09.772198+0000","last_unstale":"2024-03-14T09:52:41.140879+0000","last_undegraded":"2024-03-14T09:52:41.140879+0000","last_fullsized":"2024-03-14T09:52:41.140879+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:28:56.620266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"78'1","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146209+0000","last_change":"2024-03-14T09:52:05.237657+0000","last_active":"2024-03-14T09:52:41.146209+0000","last_peered":"2024-03-14T09:52:41.146209+0000","last_clean":"2024-03-14T09:52:41.146209+0000","last_became_active":"2024-03-14T09:52:05.237381+0000","last_became_peered":"2024-03-14T09:52:05.237381+0000","last_unstale":"2024-03-14T09:52:41.146209+0000","last_undegraded":"2024-03-14T09:52:41.146209+0000","last_fullsized":"2024-03-14T09:52:41.146209+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:34:44.372128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137056+0000","last_change":"2024-03-14T09:52:05.730454+0000","last_active":"2024-03-14T09:52:41.137056+0000","last_peered":"2024-03-14T09:52:41.137056+0000","last_clean":"2024-03-14T09:52:41.137056+0000","last_became_active":"2024-03-14T09:52:05.730046+0000","last_became_peered":"2024-03-14T09:52:05.730046+0000","last_unstale":"2024-03-14T09:52:41.137056+0000","last_undegraded":"2024-03-14T09:52:41.137056+0000","last_fullsized":"2024-03-14T09:52:41.137056+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:33:19.296993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.145692+0000","last_change":"2024-03-14T09:52:09.755969+0000","last_active":"2024-03-14T09:52:41.145692+0000","last_peered":"2024-03-14T09:52:41.145692+0000","last_clean":"2024-03-14T09:52:41.145692+0000","last_became_active":"2024-03-14T09:52:09.755512+0000","last_became_peered":"2024-03-14T09:52:09.755512+0000","last_unstale":"2024-03-14T09:52:41.145692+0000","last_undegraded":"2024-03-14T09:52:41.145692+0000","last_fullsized":"2024-03-14T09:52:41.145692+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:31:03.388090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137026+0000","last_change":"2024-03-14T09:52:05.232676+0000","last_active":"2024-03-14T09:52:41.137026+0000","last_peered":"2024-03-14T09:52:41.137026+0000","last_clean":"2024-03-14T09:52:41.137026+0000","last_became_active":"2024-03-14T09:52:05.232332+0000","last_became_peered":"2024-03-14T09:52:05.232332+0000","last_unstale":"2024-03-14T09:52:41.137026+0000","last_undegraded":"2024-03-14T09:52:41.137026+0000","last_fullsized":"2024-03-14T09:52:41.137026+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:06:11.020555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.137349+0000","last_change":"2024-03-14T09:52:05.733338+0000","last_active":"2024-03-14T09:52:41.137349+0000","last_peered":"2024-03-14T09:52:41.137349+0000","last_clean":"2024-03-14T09:52:41.137349+0000","last_became_active":"2024-03-14T09:52:05.732994+0000","last_became_peered":"2024-03-14T09:52:05.732994+0000","last_unstale":"2024-03-14T09:52:41.137349+0000","last_undegraded":"2024-03-14T09:52:41.137349+0000","last_fullsized":"2024-03-14T09:52:41.137349+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:31:25.085916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140575+0000","last_change":"2024-03-14T09:52:05.733121+0000","last_active":"2024-03-14T09:52:41.140575+0000","last_peered":"2024-03-14T09:52:41.140575+0000","last_clean":"2024-03-14T09:52:41.140575+0000","last_became_active":"2024-03-14T09:52:05.732671+0000","last_became_peered":"2024-03-14T09:52:05.732671+0000","last_unstale":"2024-03-14T09:52:41.140575+0000","last_undegraded":"2024-03-14T09:52:41.140575+0000","last_fullsized":"2024-03-14T09:52:41.140575+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:18:54.180131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143025+0000","last_change":"2024-03-14T09:52:05.736534+0000","last_active":"2024-03-14T09:52:41.143025+0000","last_peered":"2024-03-14T09:52:41.143025+0000","last_clean":"2024-03-14T09:52:41.143025+0000","last_became_active":"2024-03-14T09:52:05.736202+0000","last_became_peered":"2024-03-14T09:52:05.736202+0000","last_unstale":"2024-03-14T09:52:41.143025+0000","last_undegraded":"2024-03-14T09:52:41.143025+0000","last_fullsized":"2024-03-14T09:52:41.143025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:22:46.161561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.142597+0000","last_change":"2024-03-14T09:52:05.739434+0000","last_active":"2024-03-14T09:52:41.142597+0000","last_peered":"2024-03-14T09:52:41.142597+0000","last_clean":"2024-03-14T09:52:41.142597+0000","last_became_active":"2024-03-14T09:52:05.738023+0000","last_became_peered":"2024-03-14T09:52:05.738023+0000","last_unstale":"2024-03-14T09:52:41.142597+0000","last_undegraded":"2024-03-14T09:52:41.142597+0000","last_fullsized":"2024-03-14T09:52:41.142597+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:51:44.173645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.141083+0000","last_change":"2024-03-14T09:52:05.751367+0000","last_active":"2024-03-14T09:52:41.141083+0000","last_peered":"2024-03-14T09:52:41.141083+0000","last_clean":"2024-03-14T09:52:41.141083+0000","last_became_active":"2024-03-14T09:52:05.750816+0000","last_became_peered":"2024-03-14T09:52:05.750816+0000","last_unstale":"2024-03-14T09:52:41.141083+0000","last_undegraded":"2024-03-14T09:52:41.141083+0000","last_fullsized":"2024-03-14T09:52:41.141083+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:57:58.403356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136880+0000","last_change":"2024-03-14T09:52:05.746652+0000","last_active":"2024-03-14T09:52:41.136880+0000","last_peered":"2024-03-14T09:52:41.136880+0000","last_clean":"2024-03-14T09:52:41.136880+0000","last_became_active":"2024-03-14T09:52:05.746214+0000","last_became_peered":"2024-03-14T09:52:05.746214+0000","last_unstale":"2024-03-14T09:52:41.136880+0000","last_undegraded":"2024-03-14T09:52:41.136880+0000","last_fullsized":"2024-03-14T09:52:41.136880+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:21:39.244617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.147977+0000","last_change":"2024-03-14T09:52:42.147977+0000","last_active":"2024-03-14T09:52:42.147977+0000","last_peered":"2024-03-14T09:52:42.147977+0000","last_clean":"2024-03-14T09:52:42.147977+0000","last_became_active":"2024-03-14T09:52:42.147623+0000","last_became_peered":"2024-03-14T09:52:42.147623+0000","last_unstale":"2024-03-14T09:52:42.147977+0000","last_undegraded":"2024-03-14T09:52:42.147977+0000","last_fullsized":"2024-03-14T09:52:42.147977+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:38:35.351709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.143440+0000","last_change":"2024-03-14T09:52:05.739758+0000","last_active":"2024-03-14T09:52:41.143440+0000","last_peered":"2024-03-14T09:52:41.143440+0000","last_clean":"2024-03-14T09:52:41.143440+0000","last_became_active":"2024-03-14T09:52:05.739423+0000","last_became_peered":"2024-03-14T09:52:05.739423+0000","last_unstale":"2024-03-14T09:52:41.143440+0000","last_undegraded":"2024-03-14T09:52:41.143440+0000","last_fullsized":"2024-03-14T09:52:41.143440+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:47:22.594910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146100+0000","last_change":"2024-03-14T09:52:05.733281+0000","last_active":"2024-03-14T09:52:41.146100+0000","last_peered":"2024-03-14T09:52:41.146100+0000","last_clean":"2024-03-14T09:52:41.146100+0000","last_became_active":"2024-03-14T09:52:05.730081+0000","last_became_peered":"2024-03-14T09:52:05.730081+0000","last_unstale":"2024-03-14T09:52:41.146100+0000","last_undegraded":"2024-03-14T09:52:41.146100+0000","last_fullsized":"2024-03-14T09:52:41.146100+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:40:26.354063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.146032+0000","last_change":"2024-03-14T09:52:05.726938+0000","last_active":"2024-03-14T09:52:41.146032+0000","last_peered":"2024-03-14T09:52:41.146032+0000","last_clean":"2024-03-14T09:52:41.146032+0000","last_became_active":"2024-03-14T09:52:05.726602+0000","last_became_peered":"2024-03-14T09:52:05.726602+0000","last_unstale":"2024-03-14T09:52:41.146032+0000","last_undegraded":"2024-03-14T09:52:41.146032+0000","last_fullsized":"2024-03-14T09:52:41.146032+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:26:20.354368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-14T09:52:42.149962+0000","last_change":"2024-03-14T09:52:42.149962+0000","last_active":"2024-03-14T09:52:42.149962+0000","last_peered":"2024-03-14T09:52:42.149962+0000","last_clean":"2024-03-14T09:52:42.149962+0000","last_became_active":"2024-03-14T09:52:42.149726+0000","last_became_peered":"2024-03-14T09:52:42.149726+0000","last_unstale":"2024-03-14T09:52:42.149962+0000","last_undegraded":"2024-03-14T09:52:42.149962+0000","last_fullsized":"2024-03-14T09:52:42.149962+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:20:18.294456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.144160+0000","last_change":"2024-03-14T09:52:05.742945+0000","last_active":"2024-03-14T09:52:41.144160+0000","last_peered":"2024-03-14T09:52:41.144160+0000","last_clean":"2024-03-14T09:52:41.144160+0000","last_became_active":"2024-03-14T09:52:05.742483+0000","last_became_peered":"2024-03-14T09:52:05.742483+0000","last_unstale":"2024-03-14T09:52:41.144160+0000","last_undegraded":"2024-03-14T09:52:41.144160+0000","last_fullsized":"2024-03-14T09:52:41.144160+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:05:31.053608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.136786+0000","last_change":"2024-03-14T09:52:05.737822+0000","last_active":"2024-03-14T09:52:41.136786+0000","last_peered":"2024-03-14T09:52:41.136786+0000","last_clean":"2024-03-14T09:52:41.136786+0000","last_became_active":"2024-03-14T09:52:05.735294+0000","last_became_peered":"2024-03-14T09:52:05.735294+0000","last_unstale":"2024-03-14T09:52:41.136786+0000","last_undegraded":"2024-03-14T09:52:41.136786+0000","last_fullsized":"2024-03-14T09:52:41.136786+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:06:28.437259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-14T09:52:41.140721+0000","last_change":"2024-03-14T09:52:05.746991+0000","last_active":"2024-03-14T09:52:41.140721+0000","last_peered":"2024-03-14T09:52:41.140721+0000","last_clean":"2024-03-14T09:52:41.140721+0000","last_became_active":"2024-03-14T09:52:05.746558+0000","last_became_peered":"2024-03-14T09:52:05.746558+0000","last_unstale":"2024-03-14T09:52:41.140721+0000","last_undegraded":"2024-03-14T09:52:41.140721+0000","last_fullsized":"2024-03-14T09:52:41.140721+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:46:11.540134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247929,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65635,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":257,"num_read_kb":199,"num_write":153,"num_write_kb":112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":694,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":196905,"data_compressed":4812,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579927,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":933888,"data_stored":470724,"data_compressed":17622,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.52900000000000003}]},{"osd":1,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.48299999999999998}]},{"osd":2,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.434}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.55500000000000005}]},{"osd":5,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.623}]},{"osd":6,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.64400000000000002}]},{"osd":7,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.63600000000000001}]},{"osd":8,"last update":"Thu Mar 14 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.65700000000000003}]},{"osd":9,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.71499999999999997}]},{"osd":10,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.60999999999999999}]}]},{"osd":10,"up_from":67,"seq":287762808858,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1119483,"data_compressed":55121,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.65100000000000002}]},{"osd":1,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.80800000000000005}]},{"osd":2,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.84699999999999998}]},{"osd":3,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.67500000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.77600000000000002}]},{"osd":5,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.89300000000000002}]},{"osd":6,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.83899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":8,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.98199999999999998}]},{"osd":9,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":1.014}]},{"osd":11,"last update":"Thu Mar 14 09:52:28 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.88300000000000001}]}]},{"osd":9,"up_from":62,"seq":266287972383,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":474474,"data_compressed":17622,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.56899999999999995,"15min":0.57699999999999996},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68400000000000005,"15min":0.68000000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.504,"5min":2.504,"15min":2.504},"last":0.505},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.73499999999999999,"15min":0.74299999999999999},"min":{"1min":0.50800000000000001,"5min":0.432,"15min":0.432},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":0.58899999999999997}]},{"osd":2,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.54200000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":2.3380000000000001,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":0.624},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.78000000000000003,"15min":0.79100000000000004},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.2360000000000002,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":0.81799999999999995}]},{"osd":3,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.71199999999999997,"15min":0.72499999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.0649999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.67600000000000005,"15min":0.68400000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.86699999999999999}]},{"osd":4,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.81000000000000005,"15min":0.82399999999999995},"min":{"1min":0.52500000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.79700000000000004,"15min":0.81000000000000005},"min":{"1min":0.53000000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.755}]},{"osd":5,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.85099999999999998,"15min":0.86699999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.83599999999999997,"15min":0.84199999999999997},"min":{"1min":0.57799999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":0.69999999999999996}]},{"osd":6,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.88600000000000001,"15min":0.90600000000000003},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.90500000000000003,"15min":0.92100000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.77300000000000002}]},{"osd":8,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.92100000000000004,"15min":0.94099999999999995},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.93600000000000005,"15min":0.95399999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":0.94899999999999995}]},{"osd":10,"last update":"Thu Mar 14 09:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.93600000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]}]},{"osd":8,"up_from":55,"seq":236223201313,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":469967,"data_compressed":17622,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.042,"15min":1.1200000000000001},"min":{"1min":0.36899999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80100000000000005,"5min":9.7769999999999992,"15min":9.7769999999999992},"last":0.58199999999999996}]},{"osd":1,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.39300000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.433,"5min":0.432,"15min":0.432},"max":{"1min":0.747,"5min":0.877,"15min":0.877},"last":0.68999999999999995}]},{"osd":2,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.435,"15min":0.42699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.45600000000000002,"15min":0.44900000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.435,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.72999999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.62,"15min":0.61599999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.79000000000000004,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.748}]},{"osd":4,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.57699999999999996,"15min":0.56899999999999995},"min":{"1min":0.48599999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.84999999999999998,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.59399999999999997,"15min":0.58399999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.63300000000000001}]},{"osd":5,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.503,"15min":0.499},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.52500000000000002,"15min":0.51700000000000002},"min":{"1min":0.38700000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.67900000000000005}]},{"osd":6,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.64800000000000002,"15min":0.64600000000000002},"min":{"1min":0.42799999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.78100000000000003}]},{"osd":7,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.629,"15min":0.62},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.56100000000000005,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.76700000000000002}]},{"osd":9,"last update":"Thu Mar 14 09:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.73299999999999998,"15min":0.72999999999999998},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.53800000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.83899999999999997}]},{"osd":10,"last update":"Thu Mar 14 09:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.80300000000000005}]},{"osd":11,"last update":"Thu Mar 14 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.81200000000000006}]}]},{"osd":7,"up_from":48,"seq":206158430246,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":1065530,"data_compressed":53517,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.40999999999999998}]},{"osd":1,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.42699999999999999,"15min":0.41999999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.41699999999999998,"15min":0.40899999999999997},"min":{"1min":0.29199999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.57699999999999996,"15min":0.56799999999999995},"min":{"1min":0.441,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.42199999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.59499999999999997,"15min":0.58899999999999997},"min":{"1min":0.47499999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.41099999999999998}]},{"osd":4,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.48799999999999999,"15min":0.47799999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.503,"15min":0.497},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.42699999999999999}]},{"osd":5,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.50900000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.58799999999999997,"15min":0.57599999999999996},"min":{"1min":0.47799999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.49099999999999999}]},{"osd":6,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.64300000000000002,"15min":0.63500000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66500000000000004,"15min":0.65900000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.54700000000000004}]},{"osd":8,"last update":"Thu Mar 14 09:52:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.56499999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.92300000000000004,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.55400000000000005,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.89500000000000002,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.505}]},{"osd":9,"last update":"Thu Mar 14 09:53:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.75700000000000001,"15min":0.76300000000000001},"min":{"1min":0.55500000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.98299999999999998,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.75,"5min":0.77000000000000002,"15min":0.77400000000000002},"min":{"1min":0.61399999999999999,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.92300000000000004,"5min":1.008,"15min":1.008},"last":0.53600000000000003}]},{"osd":10,"last update":"Thu Mar 14 09:52:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.59199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":6,"up_from":43,"seq":184683593769,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":465554,"data_compressed":17622,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.51200000000000001,"15min":0.48199999999999998},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.54,"5min":3.54,"15min":3.54},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.52300000000000002,"15min":0.497},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.9860000000000002,"5min":3.9860000000000002,"15min":3.9860000000000002},"last":0.38900000000000001}]},{"osd":1,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.70699999999999996,"15min":0.65900000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":3.7589999999999999,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":0.374},{"interface":"front","average":{"1min":0.75,"5min":0.64100000000000001,"15min":0.61099999999999999},"min":{"1min":0.52900000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.56899999999999995}]},{"osd":2,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.68400000000000005,"15min":0.64700000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":4.0190000000000001,"5min":4.0190000000000001,"15min":4.0190000000000001},"last":0.504},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.72399999999999998,"15min":0.68600000000000005},"min":{"1min":0.46100000000000002,"5min":0.38,"15min":0.38},"max":{"1min":3.5790000000000002,"5min":3.5790000000000002,"15min":3.5790000000000002},"last":0.48599999999999999}]},{"osd":3,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.63800000000000001,"15min":0.60699999999999998},"min":{"1min":0.41999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.7959999999999998,"5min":3.7959999999999998,"15min":3.7959999999999998},"last":0.75},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.55400000000000005,"15min":0.54600000000000004},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78300000000000003,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.626}]},{"osd":4,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.69599999999999995,"15min":0.65600000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.76700000000000002,"15min":0.72899999999999998},"min":{"1min":0.53000000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":3.8559999999999999,"5min":3.8559999999999999,"15min":3.8559999999999999},"last":0.61599999999999999}]},{"osd":5,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.76500000000000001,"15min":0.77500000000000002},"min":{"1min":0.58799999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1120000000000001,"5min":3.1059999999999999,"15min":3.1059999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.746,"15min":0.72099999999999997},"min":{"1min":0.53500000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":4.0469999999999997,"5min":4.0469999999999997,"15min":4.0469999999999997},"last":0.70099999999999996}]},{"osd":7,"last update":"Thu Mar 14 09:52:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.69299999999999995,"15min":0.68700000000000006},"min":{"1min":0.51100000000000001,"5min":0.495,"15min":0.495},"max":{"1min":0.90200000000000002,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.77300000000000002,"15min":0.77100000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.67000000000000004}]},{"osd":8,"last update":"Thu Mar 14 09:52:37 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.89300000000000002,"15min":0.91100000000000003},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":0.94699999999999995,"5min":3.117,"15min":3.117},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.75,"15min":0.753},"min":{"1min":0.55800000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.0029999999999999,"5min":1.143,"15min":1.143},"last":0.73799999999999999}]},{"osd":9,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.84999999999999998,"15min":0.86199999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.95299999999999996,"5min":1.119,"15min":1.119},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.79900000000000004,"15min":0.80500000000000005},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":1.0840000000000001}]},{"osd":10,"last update":"Thu Mar 14 09:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":1.05},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.79400000000000004}]},{"osd":11,"last update":"Thu Mar 14 09:53:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":3.9489999999999998,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.008}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":527305,"data_compressed":19226,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:30 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.56399999999999995,"15min":0.56699999999999995},"min":{"1min":0.34599999999999997,"5min":0.315,"15min":0.315},"max":{"1min":1.3879999999999999,"5min":1.917,"15min":1.917},"last":0.52600000000000002},{"interface":"front","average":{"1min":1.05,"5min":0.64300000000000002,"15min":0.56699999999999995},"min":{"1min":0.41899999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":7.6399999999999997,"5min":7.6399999999999997,"15min":7.6399999999999997},"last":0.498}]},{"osd":2,"last update":"Thu Mar 14 09:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.55100000000000005,"15min":0.54400000000000004},"min":{"1min":0.38800000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.4419999999999999,"5min":1.885,"15min":1.885},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.58699999999999997,"15min":0.54100000000000004},"min":{"1min":0.39900000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":3.9820000000000002},"last":0.45100000000000001}]},{"osd":3,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.59899999999999998,"15min":0.57699999999999996},"min":{"1min":0.39300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.1829999999999998,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.58699999999999997,"15min":0.57699999999999996},"min":{"1min":0.434,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.6299999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":0.47199999999999998}]},{"osd":4,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.378,"15min":0.36399999999999999},"min":{"1min":0.31,"5min":0.249,"15min":0.249},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.42999999999999999,"15min":0.40000000000000002},"min":{"1min":0.316,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":2.1339999999999999,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Mar 14 09:52:37 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.53600000000000003,"15min":0.48899999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":2.9849999999999999,"5min":2.9849999999999999,"15min":2.9849999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.55400000000000005,"15min":0.51700000000000002},"min":{"1min":0.44800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.57699999999999996}]},{"osd":6,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.69899999999999995,"15min":0.64900000000000002},"min":{"1min":0.60199999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":3.7879999999999998,"5min":3.7879999999999998,"15min":3.7879999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.80400000000000005,"15min":0.85999999999999999},"min":{"1min":0.46500000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.5910000000000002,"5min":5.5229999999999997,"15min":5.5229999999999997},"last":0.58599999999999997}]},{"osd":7,"last update":"Thu Mar 14 09:52:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.51100000000000001,"15min":0.48899999999999999},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.49199999999999999,"15min":0.48499999999999999},"min":{"1min":0.33300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.50900000000000001}]},{"osd":8,"last update":"Thu Mar 14 09:52:33 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.68000000000000005,"15min":0.623},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":6.4500000000000002,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.66000000000000003,"15min":0.64000000000000001},"min":{"1min":0.50600000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.59599999999999997}]},{"osd":9,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":0.76300000000000001,"15min":0.70599999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":6.5149999999999997,"5min":6.5149999999999997,"15min":6.5149999999999997},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.359,"5min":0.83799999999999997,"15min":0.751},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":9.8900000000000006,"5min":9.8900000000000006,"15min":9.8900000000000006},"last":0.64600000000000002}]},{"osd":10,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.621},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.68999999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738433,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":950272,"data_stored":466942,"data_compressed":17622,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 14 09:52:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.56699999999999995,"15min":0.52900000000000003},"min":{"1min":0.34999999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.58899999999999997,"15min":0.54900000000000004},"min":{"1min":0.28199999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":0.64000000000000001}]},{"osd":2,"last update":"Thu Mar 14 09:52:44 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.67000000000000004,"15min":0.57099999999999995},"min":{"1min":0.503,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":5.1349999999999998,"5min":5.1349999999999998,"15min":5.1349999999999998},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.64900000000000002,"15min":0.57599999999999996},"min":{"1min":0.53500000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":0.80900000000000005}]},{"osd":3,"last update":"Thu Mar 14 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":0.75600000000000001}]},{"osd":4,"last update":"Thu Mar 14 09:52:22 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.71399999999999997,"15min":0.66600000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.8220000000000001,"5min":3.8220000000000001,"15min":3.8220000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.61499999999999999,"15min":0.58299999999999996},"min":{"1min":0.38900000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.573,"5min":3.573,"15min":3.573},"last":0.65400000000000003}]},{"osd":5,"last update":"Thu Mar 14 09:52:39 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.73399999999999999,"15min":0.67700000000000005},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.415,"5min":3.863,"15min":3.863},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.69699999999999995,"15min":0.67700000000000005},"min":{"1min":0.48499999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.621}]},{"osd":6,"last update":"Thu Mar 14 09:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.65600000000000003,"15min":0.60999999999999999},"min":{"1min":0.41299999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2370000000000001,"5min":5.2370000000000001,"15min":5.2370000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.76500000000000001,"15min":0.72499999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.5890000000000004,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":0.84099999999999997}]},{"osd":7,"last update":"Thu Mar 14 09:51:13 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.7509999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.76700000000000002}]},{"osd":8,"last update":"Thu Mar 14 09:52:34 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.83699999999999997,"15min":0.83199999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.80100000000000005,"15min":0.79300000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":3.6709999999999998,"5min":3.6709999999999998,"15min":3.6709999999999998},"last":0.73799999999999999}]},{"osd":9,"last update":"Thu Mar 14 09:52:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.88600000000000001,"15min":0.83999999999999997},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":6.7699999999999996,"5min":6.7699999999999996,"15min":6.7699999999999996},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.2110000000000001,"5min":0.93100000000000005,"15min":0.88400000000000001},"min":{"1min":0.45400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":6.0199999999999996,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":0.872}]},{"osd":10,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.68700000000000006}]},{"osd":11,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":2.5590000000000002,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":3.71,"5min":3.71,"15min":3.71},"last":0.82999999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411384,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":528383,"data_compressed":19226,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52600000000000002,"15min":0.51600000000000001},"min":{"1min":0.42799999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.52600000000000002},"min":{"1min":0.439,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85199999999999998,"5min":0.879,"15min":0.879},"last":0.52000000000000002}]},{"osd":1,"last update":"Thu Mar 14 09:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.55900000000000005,"15min":0.52600000000000002},"min":{"1min":0.41799999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.55300000000000005,"15min":0.54100000000000004},"min":{"1min":0.438,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.81699999999999995,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.44400000000000001}]},{"osd":3,"last update":"Thu Mar 14 09:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.55000000000000004,"15min":0.52600000000000002},"min":{"1min":0.42599999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.624},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56999999999999995,"15min":0.54900000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.57599999999999996}]},{"osd":4,"last update":"Thu Mar 14 09:51:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.55700000000000005,"15min":0.56499999999999995},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.73199999999999998,"5min":0.87,"15min":0.87},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.39500000000000002}]},{"osd":5,"last update":"Thu Mar 14 09:52:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.47099999999999997,"15min":0.44500000000000001},"min":{"1min":0.315,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.48999999999999999,"15min":0.47699999999999998},"min":{"1min":0.34399999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.72599999999999998,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.32800000000000001}]},{"osd":6,"last update":"Thu Mar 14 09:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.66100000000000003,"15min":0.65000000000000002},"min":{"1min":0.55100000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.67600000000000005,"15min":0.66300000000000003},"min":{"1min":0.56699999999999995,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Mar 14 09:51:12 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.54400000000000004}]},{"osd":8,"last update":"Thu Mar 14 09:52:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.63,"15min":0.623},"min":{"1min":0.38800000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.505},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.64200000000000002,"15min":0.63700000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.41499999999999998}]},{"osd":9,"last update":"Thu Mar 14 09:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.63900000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.75700000000000001,"15min":0.746},"min":{"1min":0.59299999999999997,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.749}]},{"osd":10,"last update":"Thu Mar 14 09:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.59799999999999998}]},{"osd":11,"last update":"Thu Mar 14 09:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.55600000000000005}]}]},{"osd":3,"up_from":24,"seq":103079215158,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":465748,"data_compressed":17622,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.42199999999999999,"15min":0.40100000000000002},"min":{"1min":0.32300000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.44600000000000001,"15min":0.42999999999999999},"min":{"1min":0.373,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.502}]},{"osd":1,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61099999999999999,"15min":0.59099999999999997},"min":{"1min":0.39400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.82799999999999996,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60699999999999998,"15min":0.58499999999999996},"min":{"1min":0.437,"5min":0.374,"15min":0.374},"max":{"1min":0.76300000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.61799999999999999}]},{"osd":2,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63400000000000001,"15min":0.60699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84099999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.751},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64000000000000001,"15min":0.623},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.80400000000000005,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.67500000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.61799999999999999,"15min":0.59599999999999997},"min":{"1min":0.42099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.61399999999999999,"15min":0.59899999999999998},"min":{"1min":0.48199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Mar 14 09:52:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.46700000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.82499999999999996,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.70099999999999996,"15min":0.69399999999999995},"min":{"1min":0.51500000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.96199999999999997,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.79200000000000004}]},{"osd":6,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64800000000000002,"15min":0.65000000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86599999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65800000000000003,"15min":0.64300000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.98799999999999999,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.52700000000000002}]},{"osd":7,"last update":"Thu Mar 14 09:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.745,"15min":0.74299999999999999},"min":{"1min":0.53500000000000003,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.109,"5min":1.123,"15min":1.123},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77000000000000002,"15min":0.76900000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.69399999999999995}]},{"osd":8,"last update":"Thu Mar 14 09:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.871}]},{"osd":9,"last update":"Thu Mar 14 09:52:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.80000000000000004,"15min":0.81499999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.98099999999999998,"5min":1.097,"15min":1.097},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.83999999999999997,"5min":1.121,"15min":1.121},"last":0.63500000000000001}]},{"osd":10,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.78000000000000003}]},{"osd":11,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.81299999999999994}]}]},{"osd":4,"up_from":31,"seq":133143986226,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":470548,"data_compressed":17622,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.40100000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71599999999999997,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.52300000000000002,"15min":0.51600000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.35499999999999998,"15min":0.35099999999999998},"min":{"1min":0.26600000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.26900000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.36799999999999999}]},{"osd":2,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.55900000000000005,"15min":0.53600000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":4.3150000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53100000000000003,"15min":0.51500000000000001},"min":{"1min":0.433,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.43099999999999999}]},{"osd":3,"last update":"Thu Mar 14 09:52:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.40699999999999997,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76500000000000001,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.44},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55400000000000005,"15min":0.55700000000000005},"min":{"1min":0.39800000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.59599999999999997}]},{"osd":5,"last update":"Thu Mar 14 09:52:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.57099999999999995,"15min":0.52500000000000002},"min":{"1min":0.39100000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.3479999999999999,"5min":4.3479999999999999,"15min":4.3479999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.53800000000000003,"15min":0.52700000000000002},"min":{"1min":0.42099999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.75600000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.40699999999999997}]},{"osd":6,"last update":"Thu Mar 14 09:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.48099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73499999999999999,"5min":0.876,"15min":0.876},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.621,"5min":0.60899999999999999,"15min":0.59999999999999998},"min":{"1min":0.42799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.80500000000000005,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.54600000000000004}]},{"osd":7,"last update":"Thu Mar 14 09:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.379,"5min":0.318,"15min":0.318},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.45600000000000002,"15min":0.45000000000000001},"min":{"1min":0.33200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.51400000000000001}]},{"osd":8,"last update":"Thu Mar 14 09:52:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.63800000000000001,"15min":0.63100000000000001},"min":{"1min":0.41299999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.60699999999999998,"15min":0.59899999999999998},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.79300000000000004,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.47599999999999998}]},{"osd":9,"last update":"Thu Mar 14 09:52:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.47599999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.75,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71299999999999997,"15min":0.70899999999999996},"min":{"1min":0.61899999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.84099999999999997,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.67700000000000005}]},{"osd":10,"last update":"Thu Mar 14 09:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.502}]},{"osd":11,"last update":"Thu Mar 14 09:53:07 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.621}]}]},{"osd":5,"up_from":36,"seq":154618822702,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":1061356,"data_compressed":53517,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.40200000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56000000000000005,"15min":0.54900000000000004},"min":{"1min":0.46899999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81000000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.372}]},{"osd":1,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.53600000000000003,"15min":0.51600000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79100000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.628,"5min":0.54700000000000004,"15min":0.51900000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.40100000000000002,"15min":0.38100000000000001},"min":{"1min":0.30299999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.68899999999999995,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.40400000000000003,"15min":0.38300000000000001},"min":{"1min":0.28100000000000003,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.44,"5min":0.372,"15min":0.372},"max":{"1min":0.77800000000000002,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.59799999999999998,"15min":0.58599999999999997},"min":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.57599999999999996,"15min":0.55400000000000005},"min":{"1min":0.40999999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55000000000000004,"15min":0.53400000000000003},"min":{"1min":0.39300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78400000000000003,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.63}]},{"osd":6,"last update":"Thu Mar 14 09:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.628,"15min":0.626},"min":{"1min":0.51800000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.79100000000000004,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.64000000000000001,"15min":0.627},"min":{"1min":0.44600000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.82799999999999996,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.64500000000000002}]},{"osd":7,"last update":"Thu Mar 14 09:52:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.60799999999999998,"15min":0.59299999999999997},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.59899999999999998,"15min":0.58499999999999996},"min":{"1min":0.46400000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.70999999999999996}]},{"osd":8,"last update":"Thu Mar 14 09:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59699999999999998,"15min":0.59199999999999997},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.443},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.56399999999999995},"min":{"1min":0.374,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.433}]},{"osd":9,"last update":"Thu Mar 14 09:52:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.72099999999999997,"15min":0.71199999999999997},"min":{"1min":0.56799999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96299999999999997,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.74299999999999999,"15min":0.73699999999999999},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.54400000000000004}]},{"osd":10,"last update":"Thu Mar 14 09:52:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34227,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34227,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":61841,"data_compressed":1604,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62919,"data_compressed":1604,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":63651,"data_compressed":1604,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-14T09:53:12.736 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-14T09:53:12.737 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-14T09:53:12.737 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-14T09:53:12.737 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-14T09:53:12.770 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:12 smithi132 ceph-mon[47959]: pgmap v270: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T09:53:12.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:12 smithi152 ceph-mon[48279]: pgmap v270: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T09:53:13.100 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:53:13.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:13 smithi078 ceph-mon[38143]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:53:13.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:13 smithi152 ceph-mon[48279]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:53:14.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:13 smithi132 ceph-mon[47959]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:53:14.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.231+0000 7f40e2800700 1 -- 172.21.15.78:0/1464041449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40dc103b80 msgr2=0x7f40dc107bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:14.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.231+0000 7f40e2800700 1 --2- 172.21.15.78:0/1464041449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40dc103b80 0x7f40dc107bd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f40c8009980 tx=0x7f40c8009c90 comp rx=0 tx=0).stop 2024-03-14T09:53:14.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.231+0000 7f40e2800700 1 -- 172.21.15.78:0/1464041449 shutdown_connections 2024-03-14T09:53:14.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.231+0000 7f40e2800700 1 --2- 172.21.15.78:0/1464041449 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40dc1082b0 0x7f40dc10fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:14.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.231+0000 7f40e2800700 1 --2- 172.21.15.78:0/1464041449 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40dc1031d0 0x7f40dc1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:14.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.231+0000 7f40e2800700 1 --2- 172.21.15.78:0/1464041449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40dc103b80 0x7f40dc107bd0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:14.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.231+0000 7f40e2800700 1 -- 172.21.15.78:0/1464041449 >> 172.21.15.78:0/1464041449 conn(0x7f40dc0fea40 msgr2=0x7f40dc100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:14.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.231+0000 7f40e2800700 1 -- 172.21.15.78:0/1464041449 shutdown_connections 2024-03-14T09:53:14.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.231+0000 7f40e2800700 1 -- 172.21.15.78:0/1464041449 wait complete. 2024-03-14T09:53:14.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.232+0000 7f40e2800700 1 Processor -- start 2024-03-14T09:53:14.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.232+0000 7f40e2800700 1 -- start start 2024-03-14T09:53:14.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.233+0000 7f40e2800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40dc1031d0 0x7f40dc19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.233+0000 7f40e2800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40dc103b80 0x7f40dc19fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.233+0000 7f40e2800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40dc1082b0 0x7f40dc199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.233+0000 7f40e2800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc199e40 con 0x7f40dc103b80 2024-03-14T09:53:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.233+0000 7f40e2800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc199fb0 con 0x7f40dc1082b0 2024-03-14T09:53:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.233+0000 7f40e2800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc19a170 con 0x7f40dc1031d0 2024-03-14T09:53:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.233+0000 7f40daa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40dc103b80 0x7f40dc19fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.233+0000 7f40daa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40dc103b80 0x7f40dc19fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41460/0 (socket says 172.21.15.78:41460) 2024-03-14T09:53:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.233+0000 7f40db400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40dc1031d0 0x7f40dc19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.233+0000 7f40daa00700 1 -- 172.21.15.78:0/1050744542 learned_addr learned my addr 172.21.15.78:0/1050744542 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.233+0000 7f40dbe00700 1 --2- 172.21.15.78:0/1050744542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40dc1082b0 0x7f40dc199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.234+0000 7f40daa00700 1 -- 172.21.15.78:0/1050744542 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40dc1031d0 msgr2=0x7f40dc19f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.234+0000 7f40daa00700 1 --2- 172.21.15.78:0/1050744542 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40dc1031d0 0x7f40dc19f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.234+0000 7f40daa00700 1 -- 172.21.15.78:0/1050744542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40dc1082b0 msgr2=0x7f40dc199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:14.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.234+0000 7f40daa00700 1 --2- 172.21.15.78:0/1050744542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40dc1082b0 0x7f40dc199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.234+0000 7f40daa00700 1 -- 172.21.15.78:0/1050744542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c8009590 con 0x7f40dc103b80 2024-03-14T09:53:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.234+0000 7f40db400700 1 --2- 172.21.15.78:0/1050744542 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40dc1031d0 0x7f40dc19f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.234+0000 7f40daa00700 1 --2- 172.21.15.78:0/1050744542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40dc103b80 0x7f40dc19fb20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f40c8004730 tx=0x7f40c8004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.234+0000 7f40d3e00700 1 -- 172.21.15.78:0/1050744542 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c801d070 con 0x7f40dc103b80 2024-03-14T09:53:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.234+0000 7f40d3e00700 1 -- 172.21.15.78:0/1050744542 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40c800fb30 con 0x7f40dc103b80 2024-03-14T09:53:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.234+0000 7f40d3e00700 1 -- 172.21.15.78:0/1050744542 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c8022b50 con 0x7f40dc103b80 2024-03-14T09:53:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.234+0000 7f40e2800700 1 -- 172.21.15.78:0/1050744542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40dc19a3f0 con 0x7f40dc103b80 2024-03-14T09:53:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.234+0000 7f40e2800700 1 -- 172.21.15.78:0/1050744542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40dc1a6560 con 0x7f40dc103b80 2024-03-14T09:53:14.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.235+0000 7f40e2800700 1 -- 172.21.15.78:0/1050744542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40dc19a5b0 con 0x7f40dc103b80 2024-03-14T09:53:14.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.237+0000 7f40d3e00700 1 -- 172.21.15.78:0/1050744542 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f40c800fca0 con 0x7f40dc103b80 2024-03-14T09:53:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.239+0000 7f40d3e00700 1 --2- 172.21.15.78:0/1050744542 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f40c00a0ba0 0x7f40c00a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.239+0000 7f40d3e00700 1 -- 172.21.15.78:0/1050744542 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f40c80cdab0 con 0x7f40dc103b80 2024-03-14T09:53:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.239+0000 7f40db400700 1 --2- 172.21.15.78:0/1050744542 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f40c00a0ba0 0x7f40c00a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:14.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.240+0000 7f40db400700 1 --2- 172.21.15.78:0/1050744542 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f40c00a0ba0 0x7f40c00a3060 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f40cc011490 tx=0x7f40cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:14.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.241+0000 7f40d3e00700 1 -- 172.21.15.78:0/1050744542 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f40c8097040 con 0x7f40dc103b80 2024-03-14T09:53:14.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.487+0000 7f40e2800700 1 -- 172.21.15.78:0/1050744542 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f40dc10b5d0 con 0x7f40dc103b80 2024-03-14T09:53:14.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.487+0000 7f40d3e00700 1 -- 172.21.15.78:0/1050744542 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f40c809a660 con 0x7f40dc103b80 2024-03-14T09:53:14.488 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:14.488 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-14T09:53:14.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.490+0000 7f40e2800700 1 -- 172.21.15.78:0/1050744542 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f40c00a0ba0 msgr2=0x7f40c00a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:14.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.490+0000 7f40e2800700 1 --2- 172.21.15.78:0/1050744542 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f40c00a0ba0 0x7f40c00a3060 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f40cc011490 tx=0x7f40cc009260 comp rx=0 tx=0).stop 2024-03-14T09:53:14.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.490+0000 7f40e2800700 1 -- 172.21.15.78:0/1050744542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40dc103b80 msgr2=0x7f40dc19fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:14.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.490+0000 7f40e2800700 1 --2- 172.21.15.78:0/1050744542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40dc103b80 0x7f40dc19fb20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f40c8004730 tx=0x7f40c8004760 comp rx=0 tx=0).stop 2024-03-14T09:53:14.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.490+0000 7f40e2800700 1 -- 172.21.15.78:0/1050744542 shutdown_connections 2024-03-14T09:53:14.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.490+0000 7f40e2800700 1 --2- 172.21.15.78:0/1050744542 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f40c00a0ba0 0x7f40c00a3060 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:14.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.490+0000 7f40e2800700 1 --2- 172.21.15.78:0/1050744542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40dc1082b0 0x7f40dc199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:14.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.490+0000 7f40e2800700 1 --2- 172.21.15.78:0/1050744542 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40dc1031d0 0x7f40dc19f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:14.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.490+0000 7f40e2800700 1 --2- 172.21.15.78:0/1050744542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40dc103b80 0x7f40dc19fb20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:14.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.490+0000 7f40e2800700 1 -- 172.21.15.78:0/1050744542 >> 172.21.15.78:0/1050744542 conn(0x7f40dc0fea40 msgr2=0x7f40dc0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:14.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.490+0000 7f40e2800700 1 -- 172.21.15.78:0/1050744542 shutdown_connections 2024-03-14T09:53:14.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:14.490+0000 7f40e2800700 1 -- 172.21.15.78:0/1050744542 wait complete. 2024-03-14T09:53:14.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:14 smithi078 ceph-mon[38143]: pgmap v271: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:14.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:14 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1050744542' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-14T09:53:14.885 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-14T09:53:14.886 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:53:14.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:14 smithi152 ceph-mon[48279]: pgmap v271: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:14.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:14 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1050744542' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-14T09:53:15.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:14 smithi132 ceph-mon[47959]: pgmap v271: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:15.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:14 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1050744542' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-14T09:53:15.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.080+0000 7f2e2b800700 1 -- 172.21.15.78:0/1613556275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e24100c50 msgr2=0x7f2e24101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:15.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.080+0000 7f2e2b800700 1 --2- 172.21.15.78:0/1613556275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e24100c50 0x7f2e24101070 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2e180099c0 tx=0x7f2e18009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.080+0000 7f2e2b800700 1 -- 172.21.15.78:0/1613556275 shutdown_connections 2024-03-14T09:53:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.080+0000 7f2e2b800700 1 --2- 172.21.15.78:0/1613556275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e241027e0 0x7f2e24109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.080+0000 7f2e2b800700 1 --2- 172.21.15.78:0/1613556275 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e24100c50 0x7f2e24101070 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.080+0000 7f2e2b800700 1 --2- 172.21.15.78:0/1613556275 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e24101e40 0x7f2e241022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.081+0000 7f2e2b800700 1 -- 172.21.15.78:0/1613556275 >> 172.21.15.78:0/1613556275 conn(0x7f2e240fc1c0 msgr2=0x7f2e240fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.081+0000 7f2e2b800700 1 -- 172.21.15.78:0/1613556275 shutdown_connections 2024-03-14T09:53:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.081+0000 7f2e2b800700 1 -- 172.21.15.78:0/1613556275 wait complete. 2024-03-14T09:53:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.082+0000 7f2e2b800700 1 Processor -- start 2024-03-14T09:53:15.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.083+0000 7f2e2b800700 1 -- start start 2024-03-14T09:53:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.083+0000 7f2e2b800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e24100c50 0x7f2e241992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.083+0000 7f2e2b800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e24101e40 0x7f2e241997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.083+0000 7f2e2b800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e241027e0 0x7f2e241a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.083+0000 7f2e2b800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e241a0d80 con 0x7f2e24100c50 2024-03-14T09:53:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.083+0000 7f2e2b800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e241a0ec0 con 0x7f2e241027e0 2024-03-14T09:53:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.083+0000 7f2e2b800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e241a1050 con 0x7f2e24101e40 2024-03-14T09:53:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.083+0000 7f2e28e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e24100c50 0x7f2e241992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.083+0000 7f2e28e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e24100c50 0x7f2e241992a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41468/0 (socket says 172.21.15.78:41468) 2024-03-14T09:53:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.083+0000 7f2e23e00700 1 --2- 172.21.15.78:0/1539988031 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e24101e40 0x7f2e241997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.083+0000 7f2e28e00700 1 -- 172.21.15.78:0/1539988031 learned_addr learned my addr 172.21.15.78:0/1539988031 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.084+0000 7f2e29800700 1 --2- 172.21.15.78:0/1539988031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e241027e0 0x7f2e241a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:15.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.084+0000 7f2e28e00700 1 -- 172.21.15.78:0/1539988031 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e24101e40 msgr2=0x7f2e241997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:15.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.084+0000 7f2e28e00700 1 --2- 172.21.15.78:0/1539988031 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e24101e40 0x7f2e241997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.084+0000 7f2e28e00700 1 -- 172.21.15.78:0/1539988031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e241027e0 msgr2=0x7f2e241a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:15.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.084+0000 7f2e28e00700 1 --2- 172.21.15.78:0/1539988031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e241027e0 0x7f2e241a07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.084+0000 7f2e28e00700 1 -- 172.21.15.78:0/1539988031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e180095f0 con 0x7f2e24100c50 2024-03-14T09:53:15.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.084+0000 7f2e28e00700 1 --2- 172.21.15.78:0/1539988031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e24100c50 0x7f2e241992a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2e18003730 tx=0x7f2e1800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:15.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.084+0000 7f2e21600700 1 -- 172.21.15.78:0/1539988031 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1801d070 con 0x7f2e24100c50 2024-03-14T09:53:15.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.084+0000 7f2e21600700 1 -- 172.21.15.78:0/1539988031 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e18005050 con 0x7f2e24100c50 2024-03-14T09:53:15.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.084+0000 7f2e2b800700 1 -- 172.21.15.78:0/1539988031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e241a12d0 con 0x7f2e24100c50 2024-03-14T09:53:15.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.084+0000 7f2e21600700 1 -- 172.21.15.78:0/1539988031 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18004a50 con 0x7f2e24100c50 2024-03-14T09:53:15.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.085+0000 7f2e2b800700 1 -- 172.21.15.78:0/1539988031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e241a16f0 con 0x7f2e24100c50 2024-03-14T09:53:15.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.086+0000 7f2e2b800700 1 -- 172.21.15.78:0/1539988031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e240679e0 con 0x7f2e24100c50 2024-03-14T09:53:15.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.088+0000 7f2e21600700 1 -- 172.21.15.78:0/1539988031 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f2e180051c0 con 0x7f2e24100c50 2024-03-14T09:53:15.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.089+0000 7f2e21600700 1 --2- 172.21.15.78:0/1539988031 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2e0c0a0b20 0x7f2e0c0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:15.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.090+0000 7f2e21600700 1 -- 172.21.15.78:0/1539988031 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2e180cd260 con 0x7f2e24100c50 2024-03-14T09:53:15.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.090+0000 7f2e23e00700 1 --2- 172.21.15.78:0/1539988031 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2e0c0a0b20 0x7f2e0c0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:15.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.090+0000 7f2e23e00700 1 --2- 172.21.15.78:0/1539988031 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2e0c0a0b20 0x7f2e0c0a2fe0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2e10009c60 tx=0x7f2e10009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:15.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.094+0000 7f2e21600700 1 -- 172.21.15.78:0/1539988031 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2e180967f0 con 0x7f2e24100c50 2024-03-14T09:53:15.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.320+0000 7f2e2b800700 1 -- 172.21.15.78:0/1539988031 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e2410c8d0 con 0x7f2e24100c50 2024-03-14T09:53:15.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.321+0000 7f2e21600700 1 -- 172.21.15.78:0/1539988031 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+24453 (secure 0 0 0) 0x7f2e18099e10 con 0x7f2e24100c50 2024-03-14T09:53:15.322 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:15.322 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":85,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:52:42.134030+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:53:15.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.324+0000 7f2e2b800700 1 -- 172.21.15.78:0/1539988031 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2e0c0a0b20 msgr2=0x7f2e0c0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:15.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.324+0000 7f2e2b800700 1 --2- 172.21.15.78:0/1539988031 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2e0c0a0b20 0x7f2e0c0a2fe0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2e10009c60 tx=0x7f2e10009260 comp rx=0 tx=0).stop 2024-03-14T09:53:15.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.324+0000 7f2e2b800700 1 -- 172.21.15.78:0/1539988031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e24100c50 msgr2=0x7f2e241992a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:15.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.324+0000 7f2e2b800700 1 --2- 172.21.15.78:0/1539988031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e24100c50 0x7f2e241992a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2e18003730 tx=0x7f2e1800f690 comp rx=0 tx=0).stop 2024-03-14T09:53:15.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.324+0000 7f2e2b800700 1 -- 172.21.15.78:0/1539988031 shutdown_connections 2024-03-14T09:53:15.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.324+0000 7f2e2b800700 1 --2- 172.21.15.78:0/1539988031 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2e0c0a0b20 0x7f2e0c0a2fe0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.324+0000 7f2e2b800700 1 --2- 172.21.15.78:0/1539988031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e241027e0 0x7f2e241a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.324+0000 7f2e2b800700 1 --2- 172.21.15.78:0/1539988031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e24100c50 0x7f2e241992a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.324+0000 7f2e2b800700 1 --2- 172.21.15.78:0/1539988031 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2e24101e40 0x7f2e241997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.324+0000 7f2e2b800700 1 -- 172.21.15.78:0/1539988031 >> 172.21.15.78:0/1539988031 conn(0x7f2e240fc1c0 msgr2=0x7f2e24072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:15.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.324+0000 7f2e2b800700 1 -- 172.21.15.78:0/1539988031 shutdown_connections 2024-03-14T09:53:15.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.324+0000 7f2e2b800700 1 -- 172.21.15.78:0/1539988031 wait complete. 2024-03-14T09:53:15.344 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T09:48:37.618450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-14T09:52:03.938666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-14T09:52:05.149985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-14T09:52:08.227698+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-14T09:53:15.345 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T09:53:15.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.540+0000 7fe00b400700 1 -- 172.21.15.78:0/3736475956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004100d50 msgr2=0x7fe004101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:15.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.540+0000 7fe00b400700 1 --2- 172.21.15.78:0/3736475956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004100d50 0x7fe004101170 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdff80099c0 tx=0x7fdff8009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.540+0000 7fe00b400700 1 -- 172.21.15.78:0/3736475956 shutdown_connections 2024-03-14T09:53:15.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.540+0000 7fe00b400700 1 --2- 172.21.15.78:0/3736475956 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0041028e0 0x7fe004109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.540+0000 7fe00b400700 1 --2- 172.21.15.78:0/3736475956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004100d50 0x7fe004101170 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.540+0000 7fe00b400700 1 --2- 172.21.15.78:0/3736475956 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe004101f40 0x7fe0041023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.540+0000 7fe00b400700 1 -- 172.21.15.78:0/3736475956 >> 172.21.15.78:0/3736475956 conn(0x7fe0040fc2c0 msgr2=0x7fe0040fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:15.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.540+0000 7fe00b400700 1 -- 172.21.15.78:0/3736475956 shutdown_connections 2024-03-14T09:53:15.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.540+0000 7fe00b400700 1 -- 172.21.15.78:0/3736475956 wait complete. 2024-03-14T09:53:15.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.541+0000 7fe00b400700 1 Processor -- start 2024-03-14T09:53:15.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.542+0000 7fe00b400700 1 -- start start 2024-03-14T09:53:15.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.542+0000 7fe00b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004100d50 0x7fe004199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:15.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.542+0000 7fe00b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe004101f40 0x7fe004199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:15.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.542+0000 7fe00b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0041028e0 0x7fe0041a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:15.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.542+0000 7fe00b400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0041a0e50 con 0x7fe004100d50 2024-03-14T09:53:15.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.542+0000 7fe00b400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0041a0f90 con 0x7fe0041028e0 2024-03-14T09:53:15.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.542+0000 7fe00b400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0041a1120 con 0x7fe004101f40 2024-03-14T09:53:15.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.542+0000 7fe008a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004100d50 0x7fe004199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:15.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.542+0000 7fe008a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004100d50 0x7fe004199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41490/0 (socket says 172.21.15.78:41490) 2024-03-14T09:53:15.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.542+0000 7fe008a00700 1 -- 172.21.15.78:0/2782529477 learned_addr learned my addr 172.21.15.78:0/2782529477 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:15.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.542+0000 7fe003e00700 1 --2- 172.21.15.78:0/2782529477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe004101f40 0x7fe004199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:15.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.543+0000 7fe009400700 1 --2- 172.21.15.78:0/2782529477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0041028e0 0x7fe0041a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:15.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.543+0000 7fe008a00700 1 -- 172.21.15.78:0/2782529477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe004101f40 msgr2=0x7fe004199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:15.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.543+0000 7fe008a00700 1 --2- 172.21.15.78:0/2782529477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe004101f40 0x7fe004199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.543+0000 7fe008a00700 1 -- 172.21.15.78:0/2782529477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0041028e0 msgr2=0x7fe0041a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:15.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.543+0000 7fe008a00700 1 --2- 172.21.15.78:0/2782529477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0041028e0 0x7fe0041a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.543+0000 7fe008a00700 1 -- 172.21.15.78:0/2782529477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff80095f0 con 0x7fe004100d50 2024-03-14T09:53:15.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.543+0000 7fe008a00700 1 --2- 172.21.15.78:0/2782529477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004100d50 0x7fe004199320 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdff8003730 tx=0x7fdff800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:15.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.543+0000 7fe001600700 1 -- 172.21.15.78:0/2782529477 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff801d070 con 0x7fe004100d50 2024-03-14T09:53:15.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.543+0000 7fe001600700 1 -- 172.21.15.78:0/2782529477 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdff800bda0 con 0x7fe004100d50 2024-03-14T09:53:15.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.543+0000 7fe001600700 1 -- 172.21.15.78:0/2782529477 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff8004b30 con 0x7fe004100d50 2024-03-14T09:53:15.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.543+0000 7fe00b400700 1 -- 172.21.15.78:0/2782529477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0041a13a0 con 0x7fe004100d50 2024-03-14T09:53:15.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.543+0000 7fe00b400700 1 -- 172.21.15.78:0/2782529477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0041a17c0 con 0x7fe004100d50 2024-03-14T09:53:15.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.544+0000 7fe00b400700 1 -- 172.21.15.78:0/2782529477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe004067ac0 con 0x7fe004100d50 2024-03-14T09:53:15.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.546+0000 7fe001600700 1 -- 172.21.15.78:0/2782529477 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fdff8004c90 con 0x7fe004100d50 2024-03-14T09:53:15.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.548+0000 7fe001600700 1 --2- 172.21.15.78:0/2782529477 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdfec0a0ba0 0x7fdfec0a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:15.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.548+0000 7fe001600700 1 -- 172.21.15.78:0/2782529477 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdff80cd2f0 con 0x7fe004100d50 2024-03-14T09:53:15.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.548+0000 7fe003e00700 1 --2- 172.21.15.78:0/2782529477 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdfec0a0ba0 0x7fdfec0a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:15.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.549+0000 7fe003e00700 1 --2- 172.21.15.78:0/2782529477 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdfec0a0ba0 0x7fdfec0a3060 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdff0009c60 tx=0x7fdff0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:15.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.552+0000 7fe001600700 1 -- 172.21.15.78:0/2782529477 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fdff8096880 con 0x7fe004100d50 2024-03-14T09:53:15.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.757+0000 7fe00b400700 1 -- 172.21.15.78:0/2782529477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe00410cc10 con 0x7fe004100d50 2024-03-14T09:53:15.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.758+0000 7fe001600700 1 -- 172.21.15.78:0/2782529477 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdff8099ea0 con 0x7fe004100d50 2024-03-14T09:53:15.758 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-14T09:53:15.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.760+0000 7fe00b400700 1 -- 172.21.15.78:0/2782529477 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdfec0a0ba0 msgr2=0x7fdfec0a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:15.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.760+0000 7fe00b400700 1 --2- 172.21.15.78:0/2782529477 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdfec0a0ba0 0x7fdfec0a3060 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdff0009c60 tx=0x7fdff0009260 comp rx=0 tx=0).stop 2024-03-14T09:53:15.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.760+0000 7fe00b400700 1 -- 172.21.15.78:0/2782529477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004100d50 msgr2=0x7fe004199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:15.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.760+0000 7fe00b400700 1 --2- 172.21.15.78:0/2782529477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004100d50 0x7fe004199320 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdff8003730 tx=0x7fdff800fa00 comp rx=0 tx=0).stop 2024-03-14T09:53:15.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.760+0000 7fe00b400700 1 -- 172.21.15.78:0/2782529477 shutdown_connections 2024-03-14T09:53:15.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.760+0000 7fe00b400700 1 --2- 172.21.15.78:0/2782529477 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fdfec0a0ba0 0x7fdfec0a3060 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.760+0000 7fe00b400700 1 --2- 172.21.15.78:0/2782529477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0041028e0 0x7fe0041a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.760+0000 7fe00b400700 1 --2- 172.21.15.78:0/2782529477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe004100d50 0x7fe004199320 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.760+0000 7fe00b400700 1 --2- 172.21.15.78:0/2782529477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe004101f40 0x7fe004199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.760+0000 7fe00b400700 1 -- 172.21.15.78:0/2782529477 >> 172.21.15.78:0/2782529477 conn(0x7fe0040fc2c0 msgr2=0x7fe004107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:15.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.760+0000 7fe00b400700 1 -- 172.21.15.78:0/2782529477 shutdown_connections 2024-03-14T09:53:15.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.760+0000 7fe00b400700 1 -- 172.21.15.78:0/2782529477 wait complete. 2024-03-14T09:53:15.777 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:15 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1539988031' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:15.780 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-14T09:53:15.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.975+0000 7f3a73400700 1 -- 172.21.15.78:0/2986272206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c101f40 msgr2=0x7f3a6c1023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:15.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.975+0000 7f3a73400700 1 --2- 172.21.15.78:0/2986272206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c101f40 0x7f3a6c1023a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3a580099c0 tx=0x7f3a58009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.976+0000 7f3a73400700 1 -- 172.21.15.78:0/2986272206 shutdown_connections 2024-03-14T09:53:15.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.976+0000 7f3a73400700 1 --2- 172.21.15.78:0/2986272206 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a6c1028e0 0x7f3a6c109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.976+0000 7f3a73400700 1 --2- 172.21.15.78:0/2986272206 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a6c100d50 0x7f3a6c101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.976+0000 7f3a73400700 1 --2- 172.21.15.78:0/2986272206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c101f40 0x7f3a6c1023a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.976+0000 7f3a73400700 1 -- 172.21.15.78:0/2986272206 >> 172.21.15.78:0/2986272206 conn(0x7f3a6c0fc2c0 msgr2=0x7f3a6c0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:15.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.976+0000 7f3a73400700 1 -- 172.21.15.78:0/2986272206 shutdown_connections 2024-03-14T09:53:15.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.976+0000 7f3a73400700 1 -- 172.21.15.78:0/2986272206 wait complete. 2024-03-14T09:53:15.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.977+0000 7f3a73400700 1 Processor -- start 2024-03-14T09:53:15.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.977+0000 7f3a73400700 1 -- start start 2024-03-14T09:53:15.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a73400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c100d50 0x7f3a6c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:15.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a73400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a6c101f40 0x7f3a6c0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:15.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a73400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a6c1028e0 0x7f3a6c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:15.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a73400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c073e80 con 0x7f3a6c100d50 2024-03-14T09:53:15.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a73400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c073ff0 con 0x7f3a6c1028e0 2024-03-14T09:53:15.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a73400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c1a1160 con 0x7f3a6c101f40 2024-03-14T09:53:15.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a70a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c100d50 0x7f3a6c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:15.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a70a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c100d50 0x7f3a6c074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41510/0 (socket says 172.21.15.78:41510) 2024-03-14T09:53:15.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a70a00700 1 -- 172.21.15.78:0/91531789 learned_addr learned my addr 172.21.15.78:0/91531789 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:15.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a71400700 1 --2- 172.21.15.78:0/91531789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a6c1028e0 0x7f3a6c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:15.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a6be00700 1 --2- 172.21.15.78:0/91531789 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a6c101f40 0x7f3a6c0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:15.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a70a00700 1 -- 172.21.15.78:0/91531789 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a6c101f40 msgr2=0x7f3a6c0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:15.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a70a00700 1 --2- 172.21.15.78:0/91531789 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a6c101f40 0x7f3a6c0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a70a00700 1 -- 172.21.15.78:0/91531789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a6c1028e0 msgr2=0x7f3a6c0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:15.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a70a00700 1 --2- 172.21.15.78:0/91531789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a6c1028e0 0x7f3a6c0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:15.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a70a00700 1 -- 172.21.15.78:0/91531789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a580095f0 con 0x7f3a6c100d50 2024-03-14T09:53:15.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.978+0000 7f3a6be00700 1 --2- 172.21.15.78:0/91531789 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a6c101f40 0x7f3a6c0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:53:15.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.979+0000 7f3a70a00700 1 --2- 172.21.15.78:0/91531789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c100d50 0x7f3a6c074b10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3a6000e970 tx=0x7f3a6000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:15.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.979+0000 7f3a69600700 1 -- 172.21.15.78:0/91531789 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a60009790 con 0x7f3a6c100d50 2024-03-14T09:53:15.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.979+0000 7f3a69600700 1 -- 172.21.15.78:0/91531789 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a6000eec0 con 0x7f3a6c100d50 2024-03-14T09:53:15.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.979+0000 7f3a73400700 1 -- 172.21.15.78:0/91531789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6c1a13f0 con 0x7f3a6c100d50 2024-03-14T09:53:15.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.979+0000 7f3a69600700 1 -- 172.21.15.78:0/91531789 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a60003c90 con 0x7f3a6c100d50 2024-03-14T09:53:15.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.979+0000 7f3a73400700 1 -- 172.21.15.78:0/91531789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6c1a1900 con 0x7f3a6c100d50 2024-03-14T09:53:15.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.980+0000 7f3a73400700 1 -- 172.21.15.78:0/91531789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6c108220 con 0x7f3a6c100d50 2024-03-14T09:53:15.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.982+0000 7f3a69600700 1 -- 172.21.15.78:0/91531789 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f3a6000cc70 con 0x7f3a6c100d50 2024-03-14T09:53:15.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.983+0000 7f3a69600700 1 --2- 172.21.15.78:0/91531789 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3a540a0b20 0x7f3a540a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:15.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.983+0000 7f3a69600700 1 -- 172.21.15.78:0/91531789 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3a60014070 con 0x7f3a6c100d50 2024-03-14T09:53:15.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.983+0000 7f3a6be00700 1 --2- 172.21.15.78:0/91531789 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3a540a0b20 0x7f3a540a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:15.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.984+0000 7f3a6be00700 1 --2- 172.21.15.78:0/91531789 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3a540a0b20 0x7f3a540a2fe0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3a5800b580 tx=0x7f3a5801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:15.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:15.987+0000 7f3a69600700 1 -- 172.21.15.78:0/91531789 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3a60095c90 con 0x7f3a6c100d50 2024-03-14T09:53:15.998 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:15 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1539988031' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:16.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:15 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1539988031' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:16.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.194+0000 7f3a73400700 1 -- 172.21.15.78:0/91531789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3a6c10c9d0 con 0x7f3a6c100d50 2024-03-14T09:53:16.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.195+0000 7f3a69600700 1 -- 172.21.15.78:0/91531789 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3a600992b0 con 0x7f3a6c100d50 2024-03-14T09:53:16.195 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:53:16.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.197+0000 7f3a73400700 1 -- 172.21.15.78:0/91531789 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3a540a0b20 msgr2=0x7f3a540a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:16.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.197+0000 7f3a73400700 1 --2- 172.21.15.78:0/91531789 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3a540a0b20 0x7f3a540a2fe0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3a5800b580 tx=0x7f3a5801a040 comp rx=0 tx=0).stop 2024-03-14T09:53:16.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.197+0000 7f3a73400700 1 -- 172.21.15.78:0/91531789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c100d50 msgr2=0x7f3a6c074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:16.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.197+0000 7f3a73400700 1 --2- 172.21.15.78:0/91531789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c100d50 0x7f3a6c074b10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3a6000e970 tx=0x7f3a6000ec80 comp rx=0 tx=0).stop 2024-03-14T09:53:16.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.197+0000 7f3a73400700 1 -- 172.21.15.78:0/91531789 shutdown_connections 2024-03-14T09:53:16.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.197+0000 7f3a73400700 1 --2- 172.21.15.78:0/91531789 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3a540a0b20 0x7f3a540a2fe0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.197+0000 7f3a73400700 1 --2- 172.21.15.78:0/91531789 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3a6c1028e0 0x7f3a6c0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.197+0000 7f3a73400700 1 --2- 172.21.15.78:0/91531789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c100d50 0x7f3a6c074b10 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.197+0000 7f3a73400700 1 --2- 172.21.15.78:0/91531789 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3a6c101f40 0x7f3a6c0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.197+0000 7f3a73400700 1 -- 172.21.15.78:0/91531789 >> 172.21.15.78:0/91531789 conn(0x7f3a6c0fc2c0 msgr2=0x7f3a6c105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:16.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.198+0000 7f3a73400700 1 -- 172.21.15.78:0/91531789 shutdown_connections 2024-03-14T09:53:16.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.198+0000 7f3a73400700 1 -- 172.21.15.78:0/91531789 wait complete. 2024-03-14T09:53:16.216 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-14T09:53:16.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.410+0000 7f2528600700 1 -- 172.21.15.78:0/2487927780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25201028e0 msgr2=0x7f2520109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:16.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.410+0000 7f2528600700 1 --2- 172.21.15.78:0/2487927780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25201028e0 0x7f2520109190 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f251800b0d0 tx=0x7f251800b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.410+0000 7f2528600700 1 -- 172.21.15.78:0/2487927780 shutdown_connections 2024-03-14T09:53:16.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.410+0000 7f2528600700 1 --2- 172.21.15.78:0/2487927780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25201028e0 0x7f2520109190 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.410+0000 7f2528600700 1 --2- 172.21.15.78:0/2487927780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2520100d50 0x7f2520101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.410+0000 7f2528600700 1 --2- 172.21.15.78:0/2487927780 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2520101f40 0x7f25201023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.410+0000 7f2528600700 1 -- 172.21.15.78:0/2487927780 >> 172.21.15.78:0/2487927780 conn(0x7f25200fc2c0 msgr2=0x7f25200fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:16.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.411+0000 7f2528600700 1 -- 172.21.15.78:0/2487927780 shutdown_connections 2024-03-14T09:53:16.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.411+0000 7f2528600700 1 -- 172.21.15.78:0/2487927780 wait complete. 2024-03-14T09:53:16.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.411+0000 7f2528600700 1 Processor -- start 2024-03-14T09:53:16.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.412+0000 7f2528600700 1 -- start start 2024-03-14T09:53:16.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.412+0000 7f2528600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2520100d50 0x7f2520199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:16.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.412+0000 7f2528600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2520101f40 0x7f2520199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:16.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.412+0000 7f2528600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25201028e0 0x7f25201a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:16.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.412+0000 7f2528600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25201a0f50 con 0x7f2520101f40 2024-03-14T09:53:16.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.412+0000 7f2528600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25201a1090 con 0x7f2520100d50 2024-03-14T09:53:16.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.412+0000 7f2528600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25201a1220 con 0x7f25201028e0 2024-03-14T09:53:16.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.412+0000 7f2525200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2520101f40 0x7f2520199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:16.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.412+0000 7f2525200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2520101f40 0x7f2520199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41522/0 (socket says 172.21.15.78:41522) 2024-03-14T09:53:16.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.412+0000 7f2525200700 1 -- 172.21.15.78:0/2633672305 learned_addr learned my addr 172.21.15.78:0/2633672305 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:16.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.412+0000 7f2526600700 1 --2- 172.21.15.78:0/2633672305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25201028e0 0x7f25201a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:16.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.412+0000 7f2525c00700 1 --2- 172.21.15.78:0/2633672305 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2520100d50 0x7f2520199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:16.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.413+0000 7f2525200700 1 -- 172.21.15.78:0/2633672305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25201028e0 msgr2=0x7f25201a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:16.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.413+0000 7f2525200700 1 --2- 172.21.15.78:0/2633672305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25201028e0 0x7f25201a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.413+0000 7f2525200700 1 -- 172.21.15.78:0/2633672305 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2520100d50 msgr2=0x7f2520199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:16.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.413+0000 7f2525200700 1 --2- 172.21.15.78:0/2633672305 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2520100d50 0x7f2520199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.413+0000 7f2525200700 1 -- 172.21.15.78:0/2633672305 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2518009d00 con 0x7f2520101f40 2024-03-14T09:53:16.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.413+0000 7f2525200700 1 --2- 172.21.15.78:0/2633672305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2520101f40 0x7f2520199960 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f250c00e9a0 tx=0x7f250c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:16.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.413+0000 7f251e000700 1 -- 172.21.15.78:0/2633672305 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c009790 con 0x7f2520101f40 2024-03-14T09:53:16.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.413+0000 7f251e000700 1 -- 172.21.15.78:0/2633672305 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f250c010430 con 0x7f2520101f40 2024-03-14T09:53:16.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.413+0000 7f2528600700 1 -- 172.21.15.78:0/2633672305 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25201a1500 con 0x7f2520101f40 2024-03-14T09:53:16.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.413+0000 7f251e000700 1 -- 172.21.15.78:0/2633672305 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c018680 con 0x7f2520101f40 2024-03-14T09:53:16.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.413+0000 7f2528600700 1 -- 172.21.15.78:0/2633672305 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25201a19c0 con 0x7f2520101f40 2024-03-14T09:53:16.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.414+0000 7f2528600700 1 -- 172.21.15.78:0/2633672305 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2520067ac0 con 0x7f2520101f40 2024-03-14T09:53:16.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.416+0000 7f251e000700 1 -- 172.21.15.78:0/2633672305 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f250c0105a0 con 0x7f2520101f40 2024-03-14T09:53:16.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.417+0000 7f251e000700 1 --2- 172.21.15.78:0/2633672305 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f25080a0b20 0x7f25080a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:16.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.418+0000 7f251e000700 1 -- 172.21.15.78:0/2633672305 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f250c014070 con 0x7f2520101f40 2024-03-14T09:53:16.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.418+0000 7f2525c00700 1 --2- 172.21.15.78:0/2633672305 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f25080a0b20 0x7f25080a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:16.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.418+0000 7f2525c00700 1 --2- 172.21.15.78:0/2633672305 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f25080a0b20 0x7f25080a2fe0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2520101da0 tx=0x7f2514009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:16.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.421+0000 7f251e000700 1 -- 172.21.15.78:0/2633672305 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f250c052960 con 0x7f2520101f40 2024-03-14T09:53:16.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.629+0000 7f2528600700 1 -- 172.21.15.78:0/2633672305 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f252010cc30 con 0x7f2520101f40 2024-03-14T09:53:16.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.630+0000 7f251e000700 1 -- 172.21.15.78:0/2633672305 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f250c095bf0 con 0x7f2520101f40 2024-03-14T09:53:16.631 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-14T09:53:16.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.632+0000 7f2528600700 1 -- 172.21.15.78:0/2633672305 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f25080a0b20 msgr2=0x7f25080a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:16.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.632+0000 7f2528600700 1 --2- 172.21.15.78:0/2633672305 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f25080a0b20 0x7f25080a2fe0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2520101da0 tx=0x7f2514009260 comp rx=0 tx=0).stop 2024-03-14T09:53:16.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.632+0000 7f2528600700 1 -- 172.21.15.78:0/2633672305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2520101f40 msgr2=0x7f2520199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:16.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.632+0000 7f2528600700 1 --2- 172.21.15.78:0/2633672305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2520101f40 0x7f2520199960 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f250c00e9a0 tx=0x7f250c00ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.633+0000 7f2528600700 1 -- 172.21.15.78:0/2633672305 shutdown_connections 2024-03-14T09:53:16.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.633+0000 7f2528600700 1 --2- 172.21.15.78:0/2633672305 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f25080a0b20 0x7f25080a2fe0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.633+0000 7f2528600700 1 --2- 172.21.15.78:0/2633672305 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2520100d50 0x7f2520199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.633+0000 7f2528600700 1 --2- 172.21.15.78:0/2633672305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25201028e0 0x7f25201a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.633+0000 7f2528600700 1 --2- 172.21.15.78:0/2633672305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2520101f40 0x7f2520199960 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.633+0000 7f2528600700 1 -- 172.21.15.78:0/2633672305 >> 172.21.15.78:0/2633672305 conn(0x7f25200fc2c0 msgr2=0x7f2520072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:16.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.633+0000 7f2528600700 1 -- 172.21.15.78:0/2633672305 shutdown_connections 2024-03-14T09:53:16.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.633+0000 7f2528600700 1 -- 172.21.15.78:0/2633672305 wait complete. 2024-03-14T09:53:16.651 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-14T09:53:16.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:16 smithi078 ceph-mon[38143]: pgmap v272: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-14T09:53:16.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:16 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2782529477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:16.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:16 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/91531789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:16.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.846+0000 7fd93d600700 1 -- 172.21.15.78:0/1965379334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd938101f30 msgr2=0x7fd938102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:16.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.846+0000 7fd93d600700 1 --2- 172.21.15.78:0/1965379334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd938101f30 0x7fd938102390 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd9200099c0 tx=0x7fd920009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.846+0000 7fd93d600700 1 -- 172.21.15.78:0/1965379334 shutdown_connections 2024-03-14T09:53:16.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.846+0000 7fd93d600700 1 --2- 172.21.15.78:0/1965379334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9381028d0 0x7fd938109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.846+0000 7fd93d600700 1 --2- 172.21.15.78:0/1965379334 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd938100d40 0x7fd938101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.846+0000 7fd93d600700 1 --2- 172.21.15.78:0/1965379334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd938101f30 0x7fd938102390 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.846+0000 7fd93d600700 1 -- 172.21.15.78:0/1965379334 >> 172.21.15.78:0/1965379334 conn(0x7fd9380fc2f0 msgr2=0x7fd9380fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:16.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.846+0000 7fd93d600700 1 -- 172.21.15.78:0/1965379334 shutdown_connections 2024-03-14T09:53:16.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.846+0000 7fd93d600700 1 -- 172.21.15.78:0/1965379334 wait complete. 2024-03-14T09:53:16.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.847+0000 7fd93d600700 1 Processor -- start 2024-03-14T09:53:16.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd93d600700 1 -- start start 2024-03-14T09:53:16.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd93d600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd938100d40 0x7fd938199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:16.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd93d600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd938101f30 0x7fd9381998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:16.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd93d600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9381028d0 0x7fd9381a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:16.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd93d600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9381a0e20 con 0x7fd9381028d0 2024-03-14T09:53:16.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd93d600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9381a0f60 con 0x7fd938100d40 2024-03-14T09:53:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd93d600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9381a10f0 con 0x7fd938101f30 2024-03-14T09:53:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd937400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9381028d0 0x7fd9381a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd937400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9381028d0 0x7fd9381a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41532/0 (socket says 172.21.15.78:41532) 2024-03-14T09:53:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd937400700 1 -- 172.21.15.78:0/1220737066 learned_addr learned my addr 172.21.15.78:0/1220737066 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd936a00700 1 --2- 172.21.15.78:0/1220737066 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd938100d40 0x7fd938199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd936000700 1 --2- 172.21.15.78:0/1220737066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd938101f30 0x7fd9381998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd937400700 1 -- 172.21.15.78:0/1220737066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd938101f30 msgr2=0x7fd9381998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd937400700 1 --2- 172.21.15.78:0/1220737066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd938101f30 0x7fd9381998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd937400700 1 -- 172.21.15.78:0/1220737066 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd938100d40 msgr2=0x7fd938199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd937400700 1 --2- 172.21.15.78:0/1220737066 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd938100d40 0x7fd938199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.848+0000 7fd937400700 1 -- 172.21.15.78:0/1220737066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9200095f0 con 0x7fd9381028d0 2024-03-14T09:53:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.849+0000 7fd937400700 1 --2- 172.21.15.78:0/1220737066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9381028d0 0x7fd9381a0850 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd93000e940 tx=0x7fd93000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.849+0000 7fd92f400700 1 -- 172.21.15.78:0/1220737066 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9300097b0 con 0x7fd9381028d0 2024-03-14T09:53:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.849+0000 7fd93d600700 1 -- 172.21.15.78:0/1220737066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9381a13d0 con 0x7fd9381028d0 2024-03-14T09:53:16.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.849+0000 7fd92f400700 1 -- 172.21.15.78:0/1220737066 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd930005020 con 0x7fd9381028d0 2024-03-14T09:53:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.849+0000 7fd92f400700 1 -- 172.21.15.78:0/1220737066 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93000cbf0 con 0x7fd9381028d0 2024-03-14T09:53:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.849+0000 7fd93d600700 1 -- 172.21.15.78:0/1220737066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9381a1930 con 0x7fd9381028d0 2024-03-14T09:53:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.850+0000 7fd93d600700 1 -- 172.21.15.78:0/1220737066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd938067230 con 0x7fd9381028d0 2024-03-14T09:53:16.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.852+0000 7fd92f400700 1 -- 172.21.15.78:0/1220737066 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fd930005400 con 0x7fd9381028d0 2024-03-14T09:53:16.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.853+0000 7fd92f400700 1 --2- 172.21.15.78:0/1220737066 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd91c0a0b20 0x7fd91c0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:16.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.853+0000 7fd92f400700 1 -- 172.21.15.78:0/1220737066 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd930014070 con 0x7fd9381028d0 2024-03-14T09:53:16.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.854+0000 7fd936a00700 1 --2- 172.21.15.78:0/1220737066 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd91c0a0b20 0x7fd91c0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:16.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.854+0000 7fd936a00700 1 --2- 172.21.15.78:0/1220737066 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd91c0a0b20 0x7fd91c0a2fe0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd938101d90 tx=0x7fd92800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:16.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:16.857+0000 7fd92f400700 1 -- 172.21.15.78:0/1220737066 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd930095bc0 con 0x7fd9381028d0 2024-03-14T09:53:16.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:16 smithi152 ceph-mon[48279]: pgmap v272: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-14T09:53:16.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:16 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2782529477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:16.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:16 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/91531789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:17.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:16 smithi132 ceph-mon[47959]: pgmap v272: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-14T09:53:17.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:16 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2782529477' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:17.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:16 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/91531789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:17.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.062+0000 7fd93d600700 1 -- 172.21.15.78:0/1220737066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd9381a1c70 con 0x7fd9381028d0 2024-03-14T09:53:17.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.063+0000 7fd92f400700 1 -- 172.21.15.78:0/1220737066 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd930005190 con 0x7fd9381028d0 2024-03-14T09:53:17.064 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:53:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.065+0000 7fd93d600700 1 -- 172.21.15.78:0/1220737066 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd91c0a0b20 msgr2=0x7fd91c0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.065+0000 7fd93d600700 1 --2- 172.21.15.78:0/1220737066 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd91c0a0b20 0x7fd91c0a2fe0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd938101d90 tx=0x7fd92800b040 comp rx=0 tx=0).stop 2024-03-14T09:53:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.065+0000 7fd93d600700 1 -- 172.21.15.78:0/1220737066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9381028d0 msgr2=0x7fd9381a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.065+0000 7fd93d600700 1 --2- 172.21.15.78:0/1220737066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9381028d0 0x7fd9381a0850 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd93000e940 tx=0x7fd93000ec50 comp rx=0 tx=0).stop 2024-03-14T09:53:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.066+0000 7fd93d600700 1 -- 172.21.15.78:0/1220737066 shutdown_connections 2024-03-14T09:53:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.066+0000 7fd93d600700 1 --2- 172.21.15.78:0/1220737066 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd91c0a0b20 0x7fd91c0a2fe0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.066+0000 7fd93d600700 1 --2- 172.21.15.78:0/1220737066 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd938100d40 0x7fd938199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.066+0000 7fd93d600700 1 --2- 172.21.15.78:0/1220737066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd9381028d0 0x7fd9381a0850 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.066+0000 7fd93d600700 1 --2- 172.21.15.78:0/1220737066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd938101f30 0x7fd9381998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.066+0000 7fd93d600700 1 -- 172.21.15.78:0/1220737066 >> 172.21.15.78:0/1220737066 conn(0x7fd9380fc2f0 msgr2=0x7fd938072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.066+0000 7fd93d600700 1 -- 172.21.15.78:0/1220737066 shutdown_connections 2024-03-14T09:53:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.066+0000 7fd93d600700 1 -- 172.21.15.78:0/1220737066 wait complete. 2024-03-14T09:53:17.084 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:53:17.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.278+0000 7f2aa5a00700 1 -- 172.21.15.78:0/266358142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2aa0101e40 msgr2=0x7f2aa01022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:17.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.278+0000 7f2aa5a00700 1 --2- 172.21.15.78:0/266358142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2aa0101e40 0x7f2aa01022a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c009920 tx=0x7f2a8c009c30 comp rx=0 tx=0).stop 2024-03-14T09:53:17.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.278+0000 7f2aa5a00700 1 -- 172.21.15.78:0/266358142 shutdown_connections 2024-03-14T09:53:17.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.278+0000 7f2aa5a00700 1 --2- 172.21.15.78:0/266358142 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2aa01027e0 0x7f2aa0109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.278+0000 7f2aa5a00700 1 --2- 172.21.15.78:0/266358142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2aa0100c50 0x7f2aa0101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.278+0000 7f2aa5a00700 1 --2- 172.21.15.78:0/266358142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2aa0101e40 0x7f2aa01022a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.278+0000 7f2aa5a00700 1 -- 172.21.15.78:0/266358142 >> 172.21.15.78:0/266358142 conn(0x7f2aa00fc1c0 msgr2=0x7f2aa00fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:17.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.278+0000 7f2aa5a00700 1 -- 172.21.15.78:0/266358142 shutdown_connections 2024-03-14T09:53:17.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.278+0000 7f2aa5a00700 1 -- 172.21.15.78:0/266358142 wait complete. 2024-03-14T09:53:17.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.279+0000 7f2aa5a00700 1 Processor -- start 2024-03-14T09:53:17.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.279+0000 7f2aa5a00700 1 -- start start 2024-03-14T09:53:17.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.279+0000 7f2aa5a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2aa0100c50 0x7f2aa0074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:17.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.279+0000 7f2aa5a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2aa0101e40 0x7f2aa00731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:17.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.279+0000 7f2aa5a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2aa01027e0 0x7f2aa00736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:17.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.279+0000 7f2aa5a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa0073e80 con 0x7f2aa01027e0 2024-03-14T09:53:17.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.279+0000 7f2aa5a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa0073ff0 con 0x7f2aa0101e40 2024-03-14T09:53:17.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.279+0000 7f2aa5a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa01a10d0 con 0x7f2aa0100c50 2024-03-14T09:53:17.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.280+0000 7f2a9f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2aa01027e0 0x7f2aa00736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:17.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.280+0000 7f2a9f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2aa01027e0 0x7f2aa00736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41548/0 (socket says 172.21.15.78:41548) 2024-03-14T09:53:17.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.280+0000 7f2a9f400700 1 -- 172.21.15.78:0/2467731855 learned_addr learned my addr 172.21.15.78:0/2467731855 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:17.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.280+0000 7f2a9e000700 1 --2- 172.21.15.78:0/2467731855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2aa0101e40 0x7f2aa00731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:17.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.280+0000 7f2a9ea00700 1 --2- 172.21.15.78:0/2467731855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2aa0100c50 0x7f2aa0074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:17.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.280+0000 7f2a9f400700 1 -- 172.21.15.78:0/2467731855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2aa0100c50 msgr2=0x7f2aa0074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:17.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.280+0000 7f2a9f400700 1 --2- 172.21.15.78:0/2467731855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2aa0100c50 0x7f2aa0074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.280+0000 7f2a9f400700 1 -- 172.21.15.78:0/2467731855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2aa0101e40 msgr2=0x7f2aa00731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:17.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.280+0000 7f2a9f400700 1 --2- 172.21.15.78:0/2467731855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2aa0101e40 0x7f2aa00731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.280+0000 7f2a9f400700 1 -- 172.21.15.78:0/2467731855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c009520 con 0x7f2aa01027e0 2024-03-14T09:53:17.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.280+0000 7f2a9e000700 1 --2- 172.21.15.78:0/2467731855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2aa0101e40 0x7f2aa00731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:17.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.280+0000 7f2a9f400700 1 --2- 172.21.15.78:0/2467731855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2aa01027e0 0x7f2aa00736e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2a9800ef30 tx=0x7f2a9800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:17.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.281+0000 7f2a97400700 1 -- 172.21.15.78:0/2467731855 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98018070 con 0x7f2aa01027e0 2024-03-14T09:53:17.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.281+0000 7f2a97400700 1 -- 172.21.15.78:0/2467731855 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a9800f040 con 0x7f2aa01027e0 2024-03-14T09:53:17.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.281+0000 7f2aa5a00700 1 -- 172.21.15.78:0/2467731855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa01a1300 con 0x7f2aa01027e0 2024-03-14T09:53:17.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.281+0000 7f2a97400700 1 -- 172.21.15.78:0/2467731855 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9800cc90 con 0x7f2aa01027e0 2024-03-14T09:53:17.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.281+0000 7f2aa5a00700 1 -- 172.21.15.78:0/2467731855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa01a1860 con 0x7f2aa01027e0 2024-03-14T09:53:17.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.282+0000 7f2aa5a00700 1 -- 172.21.15.78:0/2467731855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa0108110 con 0x7f2aa01027e0 2024-03-14T09:53:17.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.283+0000 7f2a97400700 1 -- 172.21.15.78:0/2467731855 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f2a98003a00 con 0x7f2aa01027e0 2024-03-14T09:53:17.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.285+0000 7f2a97400700 1 --2- 172.21.15.78:0/2467731855 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2a840a0ba0 0x7f2a840a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:17.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.285+0000 7f2a97400700 1 -- 172.21.15.78:0/2467731855 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2a980cbf60 con 0x7f2aa01027e0 2024-03-14T09:53:17.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.285+0000 7f2a9ea00700 1 --2- 172.21.15.78:0/2467731855 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2a840a0ba0 0x7f2a840a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:17.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.286+0000 7f2a9ea00700 1 --2- 172.21.15.78:0/2467731855 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2a840a0ba0 0x7f2a840a3060 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0101ca0 tx=0x7f2a90009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:17.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.289+0000 7f2a97400700 1 -- 172.21.15.78:0/2467731855 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2a980954f0 con 0x7f2aa01027e0 2024-03-14T09:53:17.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.541+0000 7f2aa5a00700 1 -- 172.21.15.78:0/2467731855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2aa00679e0 con 0x7f2aa01027e0 2024-03-14T09:53:17.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.542+0000 7f2a97400700 1 -- 172.21.15.78:0/2467731855 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f2a98098b10 con 0x7f2aa01027e0 2024-03-14T09:53:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:17.543 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:52:30.715484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:53:17.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.545+0000 7f2aa5a00700 1 -- 172.21.15.78:0/2467731855 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2a840a0ba0 msgr2=0x7f2a840a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:17.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.545+0000 7f2aa5a00700 1 --2- 172.21.15.78:0/2467731855 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2a840a0ba0 0x7f2a840a3060 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0101ca0 tx=0x7f2a90009310 comp rx=0 tx=0).stop 2024-03-14T09:53:17.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.545+0000 7f2aa5a00700 1 -- 172.21.15.78:0/2467731855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2aa01027e0 msgr2=0x7f2aa00736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:17.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.545+0000 7f2aa5a00700 1 --2- 172.21.15.78:0/2467731855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2aa01027e0 0x7f2aa00736e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2a9800ef30 tx=0x7f2a9800c3a0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.545+0000 7f2aa5a00700 1 -- 172.21.15.78:0/2467731855 shutdown_connections 2024-03-14T09:53:17.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.545+0000 7f2aa5a00700 1 --2- 172.21.15.78:0/2467731855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2aa01027e0 0x7f2aa00736e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.545+0000 7f2aa5a00700 1 --2- 172.21.15.78:0/2467731855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2aa0100c50 0x7f2aa0074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.545+0000 7f2aa5a00700 1 --2- 172.21.15.78:0/2467731855 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f2a840a0ba0 0x7f2a840a3060 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.545+0000 7f2aa5a00700 1 --2- 172.21.15.78:0/2467731855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2aa0101e40 0x7f2aa00731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.545+0000 7f2aa5a00700 1 -- 172.21.15.78:0/2467731855 >> 172.21.15.78:0/2467731855 conn(0x7f2aa00fc1c0 msgr2=0x7f2aa0105250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:17.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.545+0000 7f2aa5a00700 1 -- 172.21.15.78:0/2467731855 shutdown_connections 2024-03-14T09:53:17.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.545+0000 7f2aa5a00700 1 -- 172.21.15.78:0/2467731855 wait complete. 2024-03-14T09:53:17.547 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2024-03-14T09:53:17.564 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:53:17.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.760+0000 7f6e3a200700 1 -- 172.21.15.78:0/2843227357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e341027e0 msgr2=0x7f6e34109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.760+0000 7f6e3a200700 1 --2- 172.21.15.78:0/2843227357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e341027e0 0x7f6e34109090 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00b0d0 tx=0x7f6e2c00b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.761+0000 7f6e3a200700 1 -- 172.21.15.78:0/2843227357 shutdown_connections 2024-03-14T09:53:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.761+0000 7f6e3a200700 1 --2- 172.21.15.78:0/2843227357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e34100c50 0x7f6e34101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.761+0000 7f6e3a200700 1 --2- 172.21.15.78:0/2843227357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e341027e0 0x7f6e34109090 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.761+0000 7f6e3a200700 1 --2- 172.21.15.78:0/2843227357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e34101e40 0x7f6e341022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.761+0000 7f6e3a200700 1 -- 172.21.15.78:0/2843227357 >> 172.21.15.78:0/2843227357 conn(0x7f6e340fc1c0 msgr2=0x7f6e340fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.761+0000 7f6e3a200700 1 -- 172.21.15.78:0/2843227357 shutdown_connections 2024-03-14T09:53:17.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.761+0000 7f6e3a200700 1 -- 172.21.15.78:0/2843227357 wait complete. 2024-03-14T09:53:17.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.762+0000 7f6e3a200700 1 Processor -- start 2024-03-14T09:53:17.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.762+0000 7f6e3a200700 1 -- start start 2024-03-14T09:53:17.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.762+0000 7f6e3a200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e34100c50 0x7f6e341992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:17.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.762+0000 7f6e3a200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e34101e40 0x7f6e34199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:17.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.762+0000 7f6e3a200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e341027e0 0x7f6e341a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:17.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.762+0000 7f6e3a200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e341a0e10 con 0x7f6e34100c50 2024-03-14T09:53:17.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.762+0000 7f6e3a200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e341a0f50 con 0x7f6e34101e40 2024-03-14T09:53:17.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.762+0000 7f6e3a200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e341a10e0 con 0x7f6e341027e0 2024-03-14T09:53:17.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.762+0000 7f6e33400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e34100c50 0x7f6e341992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:17.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.762+0000 7f6e33400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e34100c50 0x7f6e341992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41568/0 (socket says 172.21.15.78:41568) 2024-03-14T09:53:17.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.762+0000 7f6e33400700 1 -- 172.21.15.78:0/3423567290 learned_addr learned my addr 172.21.15.78:0/3423567290 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:17.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.763+0000 7f6e33e00700 1 --2- 172.21.15.78:0/3423567290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e341027e0 0x7f6e341a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:17.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.763+0000 7f6e32a00700 1 --2- 172.21.15.78:0/3423567290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e34101e40 0x7f6e34199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:17.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.763+0000 7f6e33400700 1 -- 172.21.15.78:0/3423567290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e341027e0 msgr2=0x7f6e341a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:17.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.763+0000 7f6e33400700 1 --2- 172.21.15.78:0/3423567290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e341027e0 0x7f6e341a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.763+0000 7f6e33400700 1 -- 172.21.15.78:0/3423567290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e34101e40 msgr2=0x7f6e34199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:17.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.763+0000 7f6e33400700 1 --2- 172.21.15.78:0/3423567290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e34101e40 0x7f6e34199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.763+0000 7f6e33400700 1 -- 172.21.15.78:0/3423567290 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2c009d00 con 0x7f6e34100c50 2024-03-14T09:53:17.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.763+0000 7f6e33400700 1 --2- 172.21.15.78:0/3423567290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e34100c50 0x7f6e341992e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6e2400e970 tx=0x7f6e2400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:17.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.763+0000 7f6e2be00700 1 -- 172.21.15.78:0/3423567290 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e24009790 con 0x7f6e34100c50 2024-03-14T09:53:17.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.763+0000 7f6e2be00700 1 -- 172.21.15.78:0/3423567290 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e2400c7e0 con 0x7f6e34100c50 2024-03-14T09:53:17.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.763+0000 7f6e2be00700 1 -- 172.21.15.78:0/3423567290 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e24003c70 con 0x7f6e34100c50 2024-03-14T09:53:17.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.763+0000 7f6e3a200700 1 -- 172.21.15.78:0/3423567290 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e341a13c0 con 0x7f6e34100c50 2024-03-14T09:53:17.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.763+0000 7f6e3a200700 1 -- 172.21.15.78:0/3423567290 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e341a18d0 con 0x7f6e34100c50 2024-03-14T09:53:17.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.764+0000 7f6e3a200700 1 -- 172.21.15.78:0/3423567290 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e340679e0 con 0x7f6e34100c50 2024-03-14T09:53:17.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.766+0000 7f6e2be00700 1 -- 172.21.15.78:0/3423567290 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f6e2400cc50 con 0x7f6e34100c50 2024-03-14T09:53:17.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.767+0000 7f6e2be00700 1 --2- 172.21.15.78:0/3423567290 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6e180a0ba0 0x7f6e180a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:17.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.768+0000 7f6e2be00700 1 -- 172.21.15.78:0/3423567290 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6e24014070 con 0x7f6e34100c50 2024-03-14T09:53:17.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.768+0000 7f6e32a00700 1 --2- 172.21.15.78:0/3423567290 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6e180a0ba0 0x7f6e180a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:17.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.768+0000 7f6e32a00700 1 --2- 172.21.15.78:0/3423567290 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6e180a0ba0 0x7f6e180a3060 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009c60 tx=0x7f6e20009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:17.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.772+0000 7f6e2be00700 1 -- 172.21.15.78:0/3423567290 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6e24095b90 con 0x7f6e34100c50 2024-03-14T09:53:17.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:17 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2633672305' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:17.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:17 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1220737066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.992+0000 7f6e3a200700 1 -- 172.21.15.78:0/3423567290 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e3410c8e0 con 0x7f6e34100c50 2024-03-14T09:53:17.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.994+0000 7f6e2be00700 1 -- 172.21.15.78:0/3423567290 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+24453 (secure 0 0 0) 0x7f6e24003dd0 con 0x7f6e34100c50 2024-03-14T09:53:17.994 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:17.995 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":85,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:52:42.134030+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:53:17.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.997+0000 7f6e3a200700 1 -- 172.21.15.78:0/3423567290 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6e180a0ba0 msgr2=0x7f6e180a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:17.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.997+0000 7f6e3a200700 1 --2- 172.21.15.78:0/3423567290 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6e180a0ba0 0x7f6e180a3060 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009c60 tx=0x7f6e20009260 comp rx=0 tx=0).stop 2024-03-14T09:53:17.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.997+0000 7f6e3a200700 1 -- 172.21.15.78:0/3423567290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e34100c50 msgr2=0x7f6e341992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:17.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.997+0000 7f6e3a200700 1 --2- 172.21.15.78:0/3423567290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e34100c50 0x7f6e341992e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6e2400e970 tx=0x7f6e2400ec80 comp rx=0 tx=0).stop 2024-03-14T09:53:17.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.997+0000 7f6e3a200700 1 -- 172.21.15.78:0/3423567290 shutdown_connections 2024-03-14T09:53:17.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.997+0000 7f6e3a200700 1 --2- 172.21.15.78:0/3423567290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6e341027e0 0x7f6e341a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.997+0000 7f6e3a200700 1 --2- 172.21.15.78:0/3423567290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e34100c50 0x7f6e341992e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.997+0000 7f6e3a200700 1 --2- 172.21.15.78:0/3423567290 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6e180a0ba0 0x7f6e180a3060 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.997+0000 7f6e3a200700 1 --2- 172.21.15.78:0/3423567290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6e34101e40 0x7f6e34199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:17.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.997+0000 7f6e3a200700 1 -- 172.21.15.78:0/3423567290 >> 172.21.15.78:0/3423567290 conn(0x7f6e340fc1c0 msgr2=0x7f6e34072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:17.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.997+0000 7f6e3a200700 1 -- 172.21.15.78:0/3423567290 shutdown_connections 2024-03-14T09:53:17.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:17.997+0000 7f6e3a200700 1 -- 172.21.15.78:0/3423567290 wait complete. 2024-03-14T09:53:17.998 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:17 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2633672305' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:17.998 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:17 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1220737066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:18.017 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T09:48:37.618450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-14T09:52:03.938666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-14T09:52:05.149985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-14T09:52:08.227698+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-14T09:53:18.017 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T09:53:18.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:17 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2633672305' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:18.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:17 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1220737066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:18.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.211+0000 7f3b5a800700 1 -- 172.21.15.78:0/833873066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b54100d50 msgr2=0x7f3b54101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:18.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.211+0000 7f3b5a800700 1 --2- 172.21.15.78:0/833873066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b54100d50 0x7f3b54101170 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3b44009990 tx=0x7f3b44009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.211+0000 7f3b5a800700 1 -- 172.21.15.78:0/833873066 shutdown_connections 2024-03-14T09:53:18.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.211+0000 7f3b5a800700 1 --2- 172.21.15.78:0/833873066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b541028e0 0x7f3b54109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.211+0000 7f3b5a800700 1 --2- 172.21.15.78:0/833873066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b54100d50 0x7f3b54101170 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.211+0000 7f3b5a800700 1 --2- 172.21.15.78:0/833873066 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b54101f40 0x7f3b541023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.211+0000 7f3b5a800700 1 -- 172.21.15.78:0/833873066 >> 172.21.15.78:0/833873066 conn(0x7f3b540fc2c0 msgr2=0x7f3b540fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:18.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.211+0000 7f3b5a800700 1 -- 172.21.15.78:0/833873066 shutdown_connections 2024-03-14T09:53:18.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.211+0000 7f3b5a800700 1 -- 172.21.15.78:0/833873066 wait complete. 2024-03-14T09:53:18.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.212+0000 7f3b5a800700 1 Processor -- start 2024-03-14T09:53:18.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.212+0000 7f3b5a800700 1 -- start start 2024-03-14T09:53:18.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.212+0000 7f3b5a800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b54100d50 0x7f3b54074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:18.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.212+0000 7f3b5a800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b54101f40 0x7f3b540731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:18.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.213+0000 7f3b5a800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b541028e0 0x7f3b540736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:18.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.213+0000 7f3b5a800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b54073e80 con 0x7f3b54101f40 2024-03-14T09:53:18.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.213+0000 7f3b5a800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b54073ff0 con 0x7f3b54100d50 2024-03-14T09:53:18.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.213+0000 7f3b5a800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b541a11b0 con 0x7f3b541028e0 2024-03-14T09:53:18.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.213+0000 7f3b52a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b54101f40 0x7f3b540731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:18.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.213+0000 7f3b52a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b54101f40 0x7f3b540731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41590/0 (socket says 172.21.15.78:41590) 2024-03-14T09:53:18.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.213+0000 7f3b52a00700 1 -- 172.21.15.78:0/2038144273 learned_addr learned my addr 172.21.15.78:0/2038144273 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:18.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.213+0000 7f3b53400700 1 --2- 172.21.15.78:0/2038144273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b54100d50 0x7f3b54074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:18.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.213+0000 7f3b53e00700 1 --2- 172.21.15.78:0/2038144273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b541028e0 0x7f3b540736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:18.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.213+0000 7f3b52a00700 1 -- 172.21.15.78:0/2038144273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b541028e0 msgr2=0x7f3b540736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:18.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.213+0000 7f3b52a00700 1 --2- 172.21.15.78:0/2038144273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b541028e0 0x7f3b540736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.213+0000 7f3b52a00700 1 -- 172.21.15.78:0/2038144273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b54100d50 msgr2=0x7f3b54074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:18.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.213+0000 7f3b52a00700 1 --2- 172.21.15.78:0/2038144273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b54100d50 0x7f3b54074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.213+0000 7f3b52a00700 1 -- 172.21.15.78:0/2038144273 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b440095f0 con 0x7f3b54101f40 2024-03-14T09:53:18.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.213+0000 7f3b53e00700 1 --2- 172.21.15.78:0/2038144273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b541028e0 0x7f3b540736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:18.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.214+0000 7f3b52a00700 1 --2- 172.21.15.78:0/2038144273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b54101f40 0x7f3b540731a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3b4000e9a0 tx=0x7f3b4000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:18.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.214+0000 7f3b4be00700 1 -- 172.21.15.78:0/2038144273 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b40009790 con 0x7f3b54101f40 2024-03-14T09:53:18.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.214+0000 7f3b4be00700 1 -- 172.21.15.78:0/2038144273 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b4000ce70 con 0x7f3b54101f40 2024-03-14T09:53:18.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.214+0000 7f3b5a800700 1 -- 172.21.15.78:0/2038144273 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b541a1440 con 0x7f3b54101f40 2024-03-14T09:53:18.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.214+0000 7f3b4be00700 1 -- 172.21.15.78:0/2038144273 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b40018420 con 0x7f3b54101f40 2024-03-14T09:53:18.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.214+0000 7f3b5a800700 1 -- 172.21.15.78:0/2038144273 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b541a19a0 con 0x7f3b54101f40 2024-03-14T09:53:18.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.215+0000 7f3b5a800700 1 -- 172.21.15.78:0/2038144273 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b541081a0 con 0x7f3b54101f40 2024-03-14T09:53:18.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.217+0000 7f3b4be00700 1 -- 172.21.15.78:0/2038144273 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f3b400106f0 con 0x7f3b54101f40 2024-03-14T09:53:18.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.219+0000 7f3b4be00700 1 --2- 172.21.15.78:0/2038144273 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3b380a0ba0 0x7f3b380a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:18.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.219+0000 7f3b4be00700 1 -- 172.21.15.78:0/2038144273 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3b40014070 con 0x7f3b54101f40 2024-03-14T09:53:18.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.219+0000 7f3b53400700 1 --2- 172.21.15.78:0/2038144273 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3b380a0ba0 0x7f3b380a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:18.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.220+0000 7f3b53400700 1 --2- 172.21.15.78:0/2038144273 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3b380a0ba0 0x7f3b380a3060 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400b680 tx=0x7f3b44005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:18.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.223+0000 7f3b4be00700 1 -- 172.21.15.78:0/2038144273 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3b40052960 con 0x7f3b54101f40 2024-03-14T09:53:18.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.428+0000 7f3b5a800700 1 -- 172.21.15.78:0/2038144273 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3b5410cae0 con 0x7f3b54101f40 2024-03-14T09:53:18.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.428+0000 7f3b4be00700 1 -- 172.21.15.78:0/2038144273 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3b400995b0 con 0x7f3b54101f40 2024-03-14T09:53:18.429 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-14T09:53:18.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.430+0000 7f3b5a800700 1 -- 172.21.15.78:0/2038144273 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3b380a0ba0 msgr2=0x7f3b380a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:18.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.430+0000 7f3b5a800700 1 --2- 172.21.15.78:0/2038144273 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3b380a0ba0 0x7f3b380a3060 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400b680 tx=0x7f3b44005fd0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.430+0000 7f3b5a800700 1 -- 172.21.15.78:0/2038144273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b54101f40 msgr2=0x7f3b540731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:18.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.430+0000 7f3b5a800700 1 --2- 172.21.15.78:0/2038144273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b54101f40 0x7f3b540731a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3b4000e9a0 tx=0x7f3b4000ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.431+0000 7f3b5a800700 1 -- 172.21.15.78:0/2038144273 shutdown_connections 2024-03-14T09:53:18.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.431+0000 7f3b5a800700 1 --2- 172.21.15.78:0/2038144273 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3b380a0ba0 0x7f3b380a3060 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.431+0000 7f3b5a800700 1 --2- 172.21.15.78:0/2038144273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3b54100d50 0x7f3b54074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.431+0000 7f3b5a800700 1 --2- 172.21.15.78:0/2038144273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3b541028e0 0x7f3b540736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.431+0000 7f3b5a800700 1 --2- 172.21.15.78:0/2038144273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b54101f40 0x7f3b540731a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.431+0000 7f3b5a800700 1 -- 172.21.15.78:0/2038144273 >> 172.21.15.78:0/2038144273 conn(0x7f3b540fc2c0 msgr2=0x7f3b541052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:18.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.431+0000 7f3b5a800700 1 -- 172.21.15.78:0/2038144273 shutdown_connections 2024-03-14T09:53:18.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.431+0000 7f3b5a800700 1 -- 172.21.15.78:0/2038144273 wait complete. 2024-03-14T09:53:18.450 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-14T09:53:18.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.646+0000 7f00dac00700 1 -- 172.21.15.78:0/2191906826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d4100d50 msgr2=0x7f00d4101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:18.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.646+0000 7f00dac00700 1 --2- 172.21.15.78:0/2191906826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d4100d50 0x7f00d4101170 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f00c40099c0 tx=0x7f00c4009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.646+0000 7f00dac00700 1 -- 172.21.15.78:0/2191906826 shutdown_connections 2024-03-14T09:53:18.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.646+0000 7f00dac00700 1 --2- 172.21.15.78:0/2191906826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00d41028e0 0x7f00d4109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.646+0000 7f00dac00700 1 --2- 172.21.15.78:0/2191906826 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d4100d50 0x7f00d4101170 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.646+0000 7f00dac00700 1 --2- 172.21.15.78:0/2191906826 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00d4101f40 0x7f00d41023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.646+0000 7f00dac00700 1 -- 172.21.15.78:0/2191906826 >> 172.21.15.78:0/2191906826 conn(0x7f00d40fc2c0 msgr2=0x7f00d40fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:18.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.646+0000 7f00dac00700 1 -- 172.21.15.78:0/2191906826 shutdown_connections 2024-03-14T09:53:18.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.646+0000 7f00dac00700 1 -- 172.21.15.78:0/2191906826 wait complete. 2024-03-14T09:53:18.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.648+0000 7f00dac00700 1 Processor -- start 2024-03-14T09:53:18.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.648+0000 7f00dac00700 1 -- start start 2024-03-14T09:53:18.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.648+0000 7f00dac00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00d4100d50 0x7f00d4197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:18.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.648+0000 7f00dac00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00d4101f40 0x7f00d4197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:18.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.648+0000 7f00dac00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d41028e0 0x7f00d419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:18.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.649+0000 7f00dac00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d419ec50 con 0x7f00d41028e0 2024-03-14T09:53:18.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.649+0000 7f00dac00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d419ed90 con 0x7f00d4101f40 2024-03-14T09:53:18.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.649+0000 7f00dac00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d419ef20 con 0x7f00d4100d50 2024-03-14T09:53:18.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.649+0000 7f00d8c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d41028e0 0x7f00d419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:18.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.649+0000 7f00d8c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d41028e0 0x7f00d419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41614/0 (socket says 172.21.15.78:41614) 2024-03-14T09:53:18.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.649+0000 7f00d8c00700 1 -- 172.21.15.78:0/2463470417 learned_addr learned my addr 172.21.15.78:0/2463470417 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:18.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.649+0000 7f00d3e00700 1 --2- 172.21.15.78:0/2463470417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00d4100d50 0x7f00d4197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:18.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.649+0000 7f00d3400700 1 --2- 172.21.15.78:0/2463470417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00d4101f40 0x7f00d4197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:18.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.649+0000 7f00d8c00700 1 -- 172.21.15.78:0/2463470417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00d4100d50 msgr2=0x7f00d4197120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:18.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.649+0000 7f00d8c00700 1 --2- 172.21.15.78:0/2463470417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00d4100d50 0x7f00d4197120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.649+0000 7f00d8c00700 1 -- 172.21.15.78:0/2463470417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00d4101f40 msgr2=0x7f00d4197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:18.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.649+0000 7f00d8c00700 1 --2- 172.21.15.78:0/2463470417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00d4101f40 0x7f00d4197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.649+0000 7f00d8c00700 1 -- 172.21.15.78:0/2463470417 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c40095f0 con 0x7f00d41028e0 2024-03-14T09:53:18.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.649+0000 7f00d3e00700 1 --2- 172.21.15.78:0/2463470417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00d4100d50 0x7f00d4197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:18.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.649+0000 7f00d8c00700 1 --2- 172.21.15.78:0/2463470417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d41028e0 0x7f00d419e680 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00ef30 tx=0x7f00cc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:18.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.650+0000 7f00d0c00700 1 -- 172.21.15.78:0/2463470417 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc018070 con 0x7f00d41028e0 2024-03-14T09:53:18.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.650+0000 7f00dac00700 1 -- 172.21.15.78:0/2463470417 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d419f1a0 con 0x7f00d41028e0 2024-03-14T09:53:18.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.650+0000 7f00d0c00700 1 -- 172.21.15.78:0/2463470417 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00cc00f040 con 0x7f00d41028e0 2024-03-14T09:53:18.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.650+0000 7f00d0c00700 1 -- 172.21.15.78:0/2463470417 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc00cd00 con 0x7f00d41028e0 2024-03-14T09:53:18.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.650+0000 7f00dac00700 1 -- 172.21.15.78:0/2463470417 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d419f6b0 con 0x7f00d41028e0 2024-03-14T09:53:18.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.651+0000 7f00dac00700 1 -- 172.21.15.78:0/2463470417 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d4108170 con 0x7f00d41028e0 2024-03-14T09:53:18.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.653+0000 7f00d0c00700 1 -- 172.21.15.78:0/2463470417 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f00cc003a00 con 0x7f00d41028e0 2024-03-14T09:53:18.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.654+0000 7f00d0c00700 1 --2- 172.21.15.78:0/2463470417 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f00b80a0b20 0x7f00b80a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:18.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.654+0000 7f00d0c00700 1 -- 172.21.15.78:0/2463470417 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f00cc0cc000 con 0x7f00d41028e0 2024-03-14T09:53:18.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.654+0000 7f00d3e00700 1 --2- 172.21.15.78:0/2463470417 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f00b80a0b20 0x7f00b80a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:18.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.655+0000 7f00d3e00700 1 --2- 172.21.15.78:0/2463470417 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f00b80a0b20 0x7f00b80a2fe0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f00c4003730 tx=0x7f00c400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:18.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.658+0000 7f00d0c00700 1 -- 172.21.15.78:0/2463470417 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f00cc0ff960 con 0x7f00d41028e0 2024-03-14T09:53:18.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:18 smithi078 ceph-mon[38143]: pgmap v273: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:18.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:18 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2467731855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:18.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:18 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3423567290' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:18.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:18 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2038144273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:18.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.861+0000 7f00dac00700 1 -- 172.21.15.78:0/2463470417 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f00d410cae0 con 0x7f00d41028e0 2024-03-14T09:53:18.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.862+0000 7f00d0c00700 1 -- 172.21.15.78:0/2463470417 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f00cc0955f0 con 0x7f00d41028e0 2024-03-14T09:53:18.863 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:53:18.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.864+0000 7f00dac00700 1 -- 172.21.15.78:0/2463470417 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f00b80a0b20 msgr2=0x7f00b80a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:18.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.864+0000 7f00dac00700 1 --2- 172.21.15.78:0/2463470417 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f00b80a0b20 0x7f00b80a2fe0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f00c4003730 tx=0x7f00c400b540 comp rx=0 tx=0).stop 2024-03-14T09:53:18.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.864+0000 7f00dac00700 1 -- 172.21.15.78:0/2463470417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d41028e0 msgr2=0x7f00d419e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:18.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.864+0000 7f00dac00700 1 --2- 172.21.15.78:0/2463470417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d41028e0 0x7f00d419e680 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00ef30 tx=0x7f00cc00c3a0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.864+0000 7f00dac00700 1 -- 172.21.15.78:0/2463470417 shutdown_connections 2024-03-14T09:53:18.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.864+0000 7f00dac00700 1 --2- 172.21.15.78:0/2463470417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f00d41028e0 0x7f00d419e680 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.864+0000 7f00dac00700 1 --2- 172.21.15.78:0/2463470417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f00d4100d50 0x7f00d4197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.864+0000 7f00dac00700 1 --2- 172.21.15.78:0/2463470417 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f00b80a0b20 0x7f00b80a2fe0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.864+0000 7f00dac00700 1 --2- 172.21.15.78:0/2463470417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00d4101f40 0x7f00d4197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:18.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.864+0000 7f00dac00700 1 -- 172.21.15.78:0/2463470417 >> 172.21.15.78:0/2463470417 conn(0x7f00d40fc2c0 msgr2=0x7f00d41052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:18.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.864+0000 7f00dac00700 1 -- 172.21.15.78:0/2463470417 shutdown_connections 2024-03-14T09:53:18.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:18.864+0000 7f00dac00700 1 -- 172.21.15.78:0/2463470417 wait complete. 2024-03-14T09:53:18.883 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-14T09:53:18.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:18 smithi152 ceph-mon[48279]: pgmap v273: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:18.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:18 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2467731855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:18.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:18 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3423567290' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:18.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:18 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2038144273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:19.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:18 smithi132 ceph-mon[47959]: pgmap v273: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:19.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:18 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2467731855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:19.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:18 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3423567290' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:19.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:18 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2038144273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.079+0000 7fc903200700 1 -- 172.21.15.78:0/1541106388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8fc1028d0 msgr2=0x7fc8fc109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.079+0000 7fc903200700 1 --2- 172.21.15.78:0/1541106388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8fc1028d0 0x7fc8fc109180 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc8f400b0d0 tx=0x7fc8f400b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.079+0000 7fc903200700 1 -- 172.21.15.78:0/1541106388 shutdown_connections 2024-03-14T09:53:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.079+0000 7fc903200700 1 --2- 172.21.15.78:0/1541106388 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8fc100d40 0x7fc8fc101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.079+0000 7fc903200700 1 --2- 172.21.15.78:0/1541106388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8fc1028d0 0x7fc8fc109180 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.079+0000 7fc903200700 1 --2- 172.21.15.78:0/1541106388 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8fc101f30 0x7fc8fc102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.079+0000 7fc903200700 1 -- 172.21.15.78:0/1541106388 >> 172.21.15.78:0/1541106388 conn(0x7fc8fc0fc2b0 msgr2=0x7fc8fc0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.079+0000 7fc903200700 1 -- 172.21.15.78:0/1541106388 shutdown_connections 2024-03-14T09:53:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.079+0000 7fc903200700 1 -- 172.21.15.78:0/1541106388 wait complete. 2024-03-14T09:53:19.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.080+0000 7fc903200700 1 Processor -- start 2024-03-14T09:53:19.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.080+0000 7fc903200700 1 -- start start 2024-03-14T09:53:19.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc903200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8fc100d40 0x7fc8fc074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc903200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8fc101f30 0x7fc8fc0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc903200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8fc1028d0 0x7fc8fc0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc903200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc073e80 con 0x7fc8fc100d40 2024-03-14T09:53:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc903200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc073ff0 con 0x7fc8fc1028d0 2024-03-14T09:53:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc903200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc1a1290 con 0x7fc8fc101f30 2024-03-14T09:53:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc8fbe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8fc100d40 0x7fc8fc074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc8fbe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8fc100d40 0x7fc8fc074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41622/0 (socket says 172.21.15.78:41622) 2024-03-14T09:53:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc8fbe00700 1 -- 172.21.15.78:0/613129493 learned_addr learned my addr 172.21.15.78:0/613129493 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc8fb400700 1 --2- 172.21.15.78:0/613129493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8fc101f30 0x7fc8fc0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc901200700 1 --2- 172.21.15.78:0/613129493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8fc1028d0 0x7fc8fc0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc8fbe00700 1 -- 172.21.15.78:0/613129493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8fc101f30 msgr2=0x7fc8fc0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc8fbe00700 1 --2- 172.21.15.78:0/613129493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8fc101f30 0x7fc8fc0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc8fbe00700 1 -- 172.21.15.78:0/613129493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8fc1028d0 msgr2=0x7fc8fc0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc8fbe00700 1 --2- 172.21.15.78:0/613129493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8fc1028d0 0x7fc8fc0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc8fbe00700 1 -- 172.21.15.78:0/613129493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f4009d00 con 0x7fc8fc100d40 2024-03-14T09:53:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc901200700 1 --2- 172.21.15.78:0/613129493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8fc1028d0 0x7fc8fc0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:53:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc8fb400700 1 --2- 172.21.15.78:0/613129493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8fc101f30 0x7fc8fc0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:53:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.081+0000 7fc8fbe00700 1 --2- 172.21.15.78:0/613129493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8fc100d40 0x7fc8fc074b10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec00e970 tx=0x7fc8ec00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.082+0000 7fc8f8c00700 1 -- 172.21.15.78:0/613129493 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec009790 con 0x7fc8fc100d40 2024-03-14T09:53:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.082+0000 7fc8f8c00700 1 -- 172.21.15.78:0/613129493 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8ec0044e0 con 0x7fc8fc100d40 2024-03-14T09:53:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.082+0000 7fc903200700 1 -- 172.21.15.78:0/613129493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8fc1a1520 con 0x7fc8fc100d40 2024-03-14T09:53:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.082+0000 7fc8f8c00700 1 -- 172.21.15.78:0/613129493 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec003d80 con 0x7fc8fc100d40 2024-03-14T09:53:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.082+0000 7fc903200700 1 -- 172.21.15.78:0/613129493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8fc1a1a80 con 0x7fc8fc100d40 2024-03-14T09:53:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.083+0000 7fc903200700 1 -- 172.21.15.78:0/613129493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8fc108140 con 0x7fc8fc100d40 2024-03-14T09:53:19.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.085+0000 7fc8f8c00700 1 -- 172.21.15.78:0/613129493 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fc8ec003ee0 con 0x7fc8fc100d40 2024-03-14T09:53:19.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.086+0000 7fc8f8c00700 1 --2- 172.21.15.78:0/613129493 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8e00a0ba0 0x7fc8e00a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:19.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.086+0000 7fc8f8c00700 1 -- 172.21.15.78:0/613129493 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc8ec014070 con 0x7fc8fc100d40 2024-03-14T09:53:19.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.086+0000 7fc8fb400700 1 --2- 172.21.15.78:0/613129493 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8e00a0ba0 0x7fc8e00a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:19.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.087+0000 7fc8fb400700 1 --2- 172.21.15.78:0/613129493 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8e00a0ba0 0x7fc8e00a3060 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009cb0 tx=0x7fc8e8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:19.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.090+0000 7fc8f8c00700 1 -- 172.21.15.78:0/613129493 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc8ec095c80 con 0x7fc8fc100d40 2024-03-14T09:53:19.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.299+0000 7fc903200700 1 -- 172.21.15.78:0/613129493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc8fc10cb60 con 0x7fc8fc100d40 2024-03-14T09:53:19.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.299+0000 7fc8f8c00700 1 -- 172.21.15.78:0/613129493 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc8ec0992a0 con 0x7fc8fc100d40 2024-03-14T09:53:19.300 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-14T09:53:19.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.302+0000 7fc903200700 1 -- 172.21.15.78:0/613129493 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8e00a0ba0 msgr2=0x7fc8e00a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:19.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.302+0000 7fc903200700 1 --2- 172.21.15.78:0/613129493 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8e00a0ba0 0x7fc8e00a3060 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009cb0 tx=0x7fc8e8009260 comp rx=0 tx=0).stop 2024-03-14T09:53:19.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.302+0000 7fc903200700 1 -- 172.21.15.78:0/613129493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8fc100d40 msgr2=0x7fc8fc074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:19.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.302+0000 7fc903200700 1 --2- 172.21.15.78:0/613129493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8fc100d40 0x7fc8fc074b10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec00e970 tx=0x7fc8ec00ed30 comp rx=0 tx=0).stop 2024-03-14T09:53:19.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.302+0000 7fc903200700 1 -- 172.21.15.78:0/613129493 shutdown_connections 2024-03-14T09:53:19.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.302+0000 7fc903200700 1 --2- 172.21.15.78:0/613129493 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc8e00a0ba0 0x7fc8e00a3060 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.303+0000 7fc903200700 1 --2- 172.21.15.78:0/613129493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc8fc1028d0 0x7fc8fc0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.303+0000 7fc903200700 1 --2- 172.21.15.78:0/613129493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8fc100d40 0x7fc8fc074b10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.303+0000 7fc903200700 1 --2- 172.21.15.78:0/613129493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc8fc101f30 0x7fc8fc0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.303+0000 7fc903200700 1 -- 172.21.15.78:0/613129493 >> 172.21.15.78:0/613129493 conn(0x7fc8fc0fc2b0 msgr2=0x7fc8fc105280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:19.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.303+0000 7fc903200700 1 -- 172.21.15.78:0/613129493 shutdown_connections 2024-03-14T09:53:19.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.303+0000 7fc903200700 1 -- 172.21.15.78:0/613129493 wait complete. 2024-03-14T09:53:19.322 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-14T09:53:19.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.519+0000 7f88f1400700 1 -- 172.21.15.78:0/176971319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88ec074df0 msgr2=0x7f88ec073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:19.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.519+0000 7f88f1400700 1 --2- 172.21.15.78:0/176971319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88ec074df0 0x7f88ec073240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f88dc009990 tx=0x7f88dc009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.519+0000 7f88f1400700 1 -- 172.21.15.78:0/176971319 shutdown_connections 2024-03-14T09:53:19.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.519+0000 7f88f1400700 1 --2- 172.21.15.78:0/176971319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec108c00 0x7f88ec10b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.519+0000 7f88f1400700 1 --2- 172.21.15.78:0/176971319 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88ec074df0 0x7f88ec073240 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.519+0000 7f88f1400700 1 --2- 172.21.15.78:0/176971319 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88ec073810 0x7f88ec073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.519+0000 7f88f1400700 1 -- 172.21.15.78:0/176971319 >> 172.21.15.78:0/176971319 conn(0x7f88ec0fc2b0 msgr2=0x7f88ec0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:19.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.519+0000 7f88f1400700 1 -- 172.21.15.78:0/176971319 shutdown_connections 2024-03-14T09:53:19.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.519+0000 7f88f1400700 1 -- 172.21.15.78:0/176971319 wait complete. 2024-03-14T09:53:19.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.520+0000 7f88f1400700 1 Processor -- start 2024-03-14T09:53:19.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.521+0000 7f88f1400700 1 -- start start 2024-03-14T09:53:19.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.521+0000 7f88f1400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88ec073810 0x7f88ec199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:19.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.521+0000 7f88f1400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88ec074df0 0x7f88ec199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:19.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.521+0000 7f88f1400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec108c00 0x7f88ec1a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:19.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.521+0000 7f88f1400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec1a0e50 con 0x7f88ec108c00 2024-03-14T09:53:19.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.521+0000 7f88f1400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec1a0f90 con 0x7f88ec074df0 2024-03-14T09:53:19.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.521+0000 7f88f1400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec1a1120 con 0x7f88ec073810 2024-03-14T09:53:19.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.521+0000 7f88eb200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec108c00 0x7f88ec1a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:19.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.521+0000 7f88eb200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec108c00 0x7f88ec1a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41634/0 (socket says 172.21.15.78:41634) 2024-03-14T09:53:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.521+0000 7f88eb200700 1 -- 172.21.15.78:0/2718878401 learned_addr learned my addr 172.21.15.78:0/2718878401 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.522+0000 7f88e9e00700 1 --2- 172.21.15.78:0/2718878401 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88ec074df0 0x7f88ec199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.522+0000 7f88ea800700 1 --2- 172.21.15.78:0/2718878401 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88ec073810 0x7f88ec199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.522+0000 7f88eb200700 1 -- 172.21.15.78:0/2718878401 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88ec073810 msgr2=0x7f88ec199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.522+0000 7f88eb200700 1 --2- 172.21.15.78:0/2718878401 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88ec073810 0x7f88ec199320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.522+0000 7f88eb200700 1 -- 172.21.15.78:0/2718878401 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88ec074df0 msgr2=0x7f88ec199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.522+0000 7f88eb200700 1 --2- 172.21.15.78:0/2718878401 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88ec074df0 0x7f88ec199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.522+0000 7f88eb200700 1 -- 172.21.15.78:0/2718878401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88dc0095f0 con 0x7f88ec108c00 2024-03-14T09:53:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.522+0000 7f88e9e00700 1 --2- 172.21.15.78:0/2718878401 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88ec074df0 0x7f88ec199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.522+0000 7f88eb200700 1 --2- 172.21.15.78:0/2718878401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec108c00 0x7f88ec1a0880 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f88e4007ee0 tx=0x7f88e400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.522+0000 7f88e3400700 1 -- 172.21.15.78:0/2718878401 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e400dd20 con 0x7f88ec108c00 2024-03-14T09:53:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.522+0000 7f88e3400700 1 -- 172.21.15.78:0/2718878401 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f88e400f040 con 0x7f88ec108c00 2024-03-14T09:53:19.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.522+0000 7f88e3400700 1 -- 172.21.15.78:0/2718878401 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e40048a0 con 0x7f88ec108c00 2024-03-14T09:53:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.522+0000 7f88f1400700 1 -- 172.21.15.78:0/2718878401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ec1a1400 con 0x7f88ec108c00 2024-03-14T09:53:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.523+0000 7f88f1400700 1 -- 172.21.15.78:0/2718878401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ec1a18c0 con 0x7f88ec108c00 2024-03-14T09:53:19.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.524+0000 7f88f1400700 1 -- 172.21.15.78:0/2718878401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88ec1079c0 con 0x7f88ec108c00 2024-03-14T09:53:19.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.525+0000 7f88e3400700 1 -- 172.21.15.78:0/2718878401 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f88e4003a00 con 0x7f88ec108c00 2024-03-14T09:53:19.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.527+0000 7f88e3400700 1 --2- 172.21.15.78:0/2718878401 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f88d00a0b20 0x7f88d00a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:19.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.527+0000 7f88e3400700 1 -- 172.21.15.78:0/2718878401 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f88e40cbd00 con 0x7f88ec108c00 2024-03-14T09:53:19.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.527+0000 7f88ea800700 1 --2- 172.21.15.78:0/2718878401 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f88d00a0b20 0x7f88d00a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:19.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.528+0000 7f88ea800700 1 --2- 172.21.15.78:0/2718878401 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f88d00a0b20 0x7f88d00a2fe0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00b2e0 tx=0x7f88dc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:19.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.531+0000 7f88e3400700 1 -- 172.21.15.78:0/2718878401 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f88e4095260 con 0x7f88ec108c00 2024-03-14T09:53:19.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.743+0000 7f88f1400700 1 -- 172.21.15.78:0/2718878401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f88ec10eba0 con 0x7f88ec108c00 2024-03-14T09:53:19.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.743+0000 7f88e3400700 1 -- 172.21.15.78:0/2718878401 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) v1 ==== 103+0+11 (secure 0 0 0) 0x7f88e4098880 con 0x7f88ec108c00 2024-03-14T09:53:19.744 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:53:19.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.746+0000 7f88f1400700 1 -- 172.21.15.78:0/2718878401 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f88d00a0b20 msgr2=0x7f88d00a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:19.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.746+0000 7f88f1400700 1 --2- 172.21.15.78:0/2718878401 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f88d00a0b20 0x7f88d00a2fe0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00b2e0 tx=0x7f88dc005fd0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.746+0000 7f88f1400700 1 -- 172.21.15.78:0/2718878401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec108c00 msgr2=0x7f88ec1a0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:19.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.746+0000 7f88f1400700 1 --2- 172.21.15.78:0/2718878401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec108c00 0x7f88ec1a0880 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f88e4007ee0 tx=0x7f88e400d3b0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.746+0000 7f88f1400700 1 -- 172.21.15.78:0/2718878401 shutdown_connections 2024-03-14T09:53:19.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.746+0000 7f88f1400700 1 --2- 172.21.15.78:0/2718878401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88ec108c00 0x7f88ec1a0880 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.746+0000 7f88f1400700 1 --2- 172.21.15.78:0/2718878401 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88ec073810 0x7f88ec199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.746+0000 7f88f1400700 1 --2- 172.21.15.78:0/2718878401 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f88d00a0b20 0x7f88d00a2fe0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.746+0000 7f88f1400700 1 --2- 172.21.15.78:0/2718878401 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88ec074df0 0x7f88ec199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.746+0000 7f88f1400700 1 -- 172.21.15.78:0/2718878401 >> 172.21.15.78:0/2718878401 conn(0x7f88ec0fc2b0 msgr2=0x7f88ec104b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:19.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.746+0000 7f88f1400700 1 -- 172.21.15.78:0/2718878401 shutdown_connections 2024-03-14T09:53:19.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.746+0000 7f88f1400700 1 -- 172.21.15.78:0/2718878401 wait complete. 2024-03-14T09:53:19.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:19 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2463470417' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:19.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:19 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/613129493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:19.765 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:53:19.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.956+0000 7fd3c1400700 1 -- 172.21.15.78:0/1469026518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3bc1027e0 msgr2=0x7fd3bc109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:19.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.956+0000 7fd3c1400700 1 --2- 172.21.15.78:0/1469026518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3bc1027e0 0x7fd3bc109090 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400b0d0 tx=0x7fd3b400b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.956+0000 7fd3c1400700 1 -- 172.21.15.78:0/1469026518 shutdown_connections 2024-03-14T09:53:19.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.956+0000 7fd3c1400700 1 --2- 172.21.15.78:0/1469026518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3bc1027e0 0x7fd3bc109090 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.956+0000 7fd3c1400700 1 --2- 172.21.15.78:0/1469026518 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3bc100c50 0x7fd3bc101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.956+0000 7fd3c1400700 1 --2- 172.21.15.78:0/1469026518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd3bc101e40 0x7fd3bc1022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.956+0000 7fd3c1400700 1 -- 172.21.15.78:0/1469026518 >> 172.21.15.78:0/1469026518 conn(0x7fd3bc0fc1c0 msgr2=0x7fd3bc0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:19.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.956+0000 7fd3c1400700 1 -- 172.21.15.78:0/1469026518 shutdown_connections 2024-03-14T09:53:19.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.956+0000 7fd3c1400700 1 -- 172.21.15.78:0/1469026518 wait complete. 2024-03-14T09:53:19.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.957+0000 7fd3c1400700 1 Processor -- start 2024-03-14T09:53:19.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.958+0000 7fd3c1400700 1 -- start start 2024-03-14T09:53:19.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.958+0000 7fd3c1400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3bc100c50 0x7fd3bc199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:19.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.958+0000 7fd3c1400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd3bc101e40 0x7fd3bc199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:19.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.958+0000 7fd3c1400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3bc1027e0 0x7fd3bc1a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:19.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.958+0000 7fd3c1400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc1a0e50 con 0x7fd3bc100c50 2024-03-14T09:53:19.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.958+0000 7fd3c1400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc1a0f90 con 0x7fd3bc101e40 2024-03-14T09:53:19.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.958+0000 7fd3c1400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc1a1120 con 0x7fd3bc1027e0 2024-03-14T09:53:19.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.958+0000 7fd3ba800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3bc100c50 0x7fd3bc199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:19.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.958+0000 7fd3ba800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3bc100c50 0x7fd3bc199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41652/0 (socket says 172.21.15.78:41652) 2024-03-14T09:53:19.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.958+0000 7fd3ba800700 1 -- 172.21.15.78:0/34879644 learned_addr learned my addr 172.21.15.78:0/34879644 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:19.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.958+0000 7fd3bb200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3bc1027e0 0x7fd3bc1a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:19.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.958+0000 7fd3b9e00700 1 --2- 172.21.15.78:0/34879644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd3bc101e40 0x7fd3bc199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:19.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.959+0000 7fd3ba800700 1 -- 172.21.15.78:0/34879644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3bc1027e0 msgr2=0x7fd3bc1a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:19.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.959+0000 7fd3ba800700 1 --2- 172.21.15.78:0/34879644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3bc1027e0 0x7fd3bc1a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.959+0000 7fd3ba800700 1 -- 172.21.15.78:0/34879644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd3bc101e40 msgr2=0x7fd3bc199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:19.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.959+0000 7fd3ba800700 1 --2- 172.21.15.78:0/34879644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd3bc101e40 0x7fd3bc199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:19.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.959+0000 7fd3ba800700 1 -- 172.21.15.78:0/34879644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a8009520 con 0x7fd3bc100c50 2024-03-14T09:53:19.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.959+0000 7fd3ba800700 1 --2- 172.21.15.78:0/34879644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3bc100c50 0x7fd3bc199320 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00e970 tx=0x7fd3ac00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:19.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.959+0000 7fd3b3400700 1 -- 172.21.15.78:0/34879644 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac009790 con 0x7fd3bc100c50 2024-03-14T09:53:19.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.959+0000 7fd3b3400700 1 -- 172.21.15.78:0/34879644 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd3ac00c7e0 con 0x7fd3bc100c50 2024-03-14T09:53:19.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.959+0000 7fd3c1400700 1 -- 172.21.15.78:0/34879644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b4009d00 con 0x7fd3bc100c50 2024-03-14T09:53:19.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.959+0000 7fd3b3400700 1 -- 172.21.15.78:0/34879644 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac003c70 con 0x7fd3bc100c50 2024-03-14T09:53:19.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.959+0000 7fd3c1400700 1 -- 172.21.15.78:0/34879644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3bc1a17d0 con 0x7fd3bc100c50 2024-03-14T09:53:19.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.960+0000 7fd3c1400700 1 -- 172.21.15.78:0/34879644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3bc0679e0 con 0x7fd3bc100c50 2024-03-14T09:53:19.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.962+0000 7fd3b3400700 1 -- 172.21.15.78:0/34879644 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fd3ac00cc50 con 0x7fd3bc100c50 2024-03-14T09:53:19.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.963+0000 7fd3b3400700 1 --2- 172.21.15.78:0/34879644 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd3a00a0ba0 0x7fd3a00a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:19.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.963+0000 7fd3b3400700 1 -- 172.21.15.78:0/34879644 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd3ac014070 con 0x7fd3bc100c50 2024-03-14T09:53:19.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.964+0000 7fd3b9e00700 1 --2- 172.21.15.78:0/34879644 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd3a00a0ba0 0x7fd3a00a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:19.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.964+0000 7fd3b9e00700 1 --2- 172.21.15.78:0/34879644 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd3a00a0ba0 0x7fd3a00a3060 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009df0 tx=0x7fd3a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:19.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:19.967+0000 7fd3b3400700 1 -- 172.21.15.78:0/34879644 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd3ac095b60 con 0x7fd3bc100c50 2024-03-14T09:53:19.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:19 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2463470417' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:19.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:19 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/613129493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:20.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:19 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2463470417' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:20.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:19 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/613129493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:20.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.216+0000 7fd3c1400700 1 -- 172.21.15.78:0/34879644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3bc10cb20 con 0x7fd3bc100c50 2024-03-14T09:53:20.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.217+0000 7fd3b3400700 1 -- 172.21.15.78:0/34879644 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10870 (secure 0 0 0) 0x7fd3ac003dd0 con 0x7fd3bc100c50 2024-03-14T09:53:20.217 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:20.217 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:52:30.715484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:53:20.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.219+0000 7fd3c1400700 1 -- 172.21.15.78:0/34879644 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd3a00a0ba0 msgr2=0x7fd3a00a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:20.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.219+0000 7fd3c1400700 1 --2- 172.21.15.78:0/34879644 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd3a00a0ba0 0x7fd3a00a3060 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009df0 tx=0x7fd3a8009260 comp rx=0 tx=0).stop 2024-03-14T09:53:20.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.219+0000 7fd3c1400700 1 -- 172.21.15.78:0/34879644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3bc100c50 msgr2=0x7fd3bc199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:20.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.219+0000 7fd3c1400700 1 --2- 172.21.15.78:0/34879644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3bc100c50 0x7fd3bc199320 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00e970 tx=0x7fd3ac00ec80 comp rx=0 tx=0).stop 2024-03-14T09:53:20.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.219+0000 7fd3c1400700 1 -- 172.21.15.78:0/34879644 shutdown_connections 2024-03-14T09:53:20.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.219+0000 7fd3c1400700 1 --2- 172.21.15.78:0/34879644 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd3bc1027e0 0x7fd3bc1a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.219+0000 7fd3c1400700 1 --2- 172.21.15.78:0/34879644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3bc100c50 0x7fd3bc199320 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.219+0000 7fd3c1400700 1 --2- 172.21.15.78:0/34879644 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd3a00a0ba0 0x7fd3a00a3060 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.219+0000 7fd3c1400700 1 --2- 172.21.15.78:0/34879644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd3bc101e40 0x7fd3bc199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.219+0000 7fd3c1400700 1 -- 172.21.15.78:0/34879644 >> 172.21.15.78:0/34879644 conn(0x7fd3bc0fc1c0 msgr2=0x7fd3bc072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:20.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.219+0000 7fd3c1400700 1 -- 172.21.15.78:0/34879644 shutdown_connections 2024-03-14T09:53:20.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.219+0000 7fd3c1400700 1 -- 172.21.15.78:0/34879644 wait complete. 2024-03-14T09:53:20.221 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2024-03-14T09:53:20.239 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:53:20.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.430+0000 7fb3a3e00700 1 -- 172.21.15.78:0/4275628750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c1027e0 msgr2=0x7fb39c109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:20.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.430+0000 7fb3a3e00700 1 --2- 172.21.15.78:0/4275628750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c1027e0 0x7fb39c109090 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb39400b0d0 tx=0x7fb39400b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.430+0000 7fb3a3e00700 1 -- 172.21.15.78:0/4275628750 shutdown_connections 2024-03-14T09:53:20.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.430+0000 7fb3a3e00700 1 --2- 172.21.15.78:0/4275628750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c1027e0 0x7fb39c109090 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.430+0000 7fb3a3e00700 1 --2- 172.21.15.78:0/4275628750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb39c100c50 0x7fb39c101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.430+0000 7fb3a3e00700 1 --2- 172.21.15.78:0/4275628750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb39c101e40 0x7fb39c1022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.430+0000 7fb3a3e00700 1 -- 172.21.15.78:0/4275628750 >> 172.21.15.78:0/4275628750 conn(0x7fb39c0fc1c0 msgr2=0x7fb39c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:20.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.430+0000 7fb3a3e00700 1 -- 172.21.15.78:0/4275628750 shutdown_connections 2024-03-14T09:53:20.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.430+0000 7fb3a3e00700 1 -- 172.21.15.78:0/4275628750 wait complete. 2024-03-14T09:53:20.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.431+0000 7fb3a3e00700 1 Processor -- start 2024-03-14T09:53:20.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.431+0000 7fb3a3e00700 1 -- start start 2024-03-14T09:53:20.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.431+0000 7fb3a3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c100c50 0x7fb39c199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:20.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.432+0000 7fb3a3e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb39c101e40 0x7fb39c199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:20.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.432+0000 7fb3a3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb39c1027e0 0x7fb39c1a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:20.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.432+0000 7fb3a3e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c1a0e50 con 0x7fb39c100c50 2024-03-14T09:53:20.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.432+0000 7fb3a3e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c1a0f90 con 0x7fb39c1027e0 2024-03-14T09:53:20.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.432+0000 7fb3a3e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c1a1120 con 0x7fb39c101e40 2024-03-14T09:53:20.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.432+0000 7fb3a1400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c100c50 0x7fb39c199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:20.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.432+0000 7fb3a1400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c100c50 0x7fb39c199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41666/0 (socket says 172.21.15.78:41666) 2024-03-14T09:53:20.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.432+0000 7fb3a1400700 1 -- 172.21.15.78:0/4167964939 learned_addr learned my addr 172.21.15.78:0/4167964939 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:20.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.432+0000 7fb3a0a00700 1 --2- 172.21.15.78:0/4167964939 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb39c101e40 0x7fb39c199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:20.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.432+0000 7fb3a1e00700 1 --2- 172.21.15.78:0/4167964939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb39c1027e0 0x7fb39c1a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:20.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.432+0000 7fb3a1400700 1 -- 172.21.15.78:0/4167964939 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb39c101e40 msgr2=0x7fb39c199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:20.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.432+0000 7fb3a1400700 1 --2- 172.21.15.78:0/4167964939 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb39c101e40 0x7fb39c199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.432+0000 7fb3a1400700 1 -- 172.21.15.78:0/4167964939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb39c1027e0 msgr2=0x7fb39c1a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:20.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.432+0000 7fb3a1400700 1 --2- 172.21.15.78:0/4167964939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb39c1027e0 0x7fb39c1a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.432+0000 7fb3a1400700 1 -- 172.21.15.78:0/4167964939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb388009520 con 0x7fb39c100c50 2024-03-14T09:53:20.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.433+0000 7fb3a1400700 1 --2- 172.21.15.78:0/4167964939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c100c50 0x7fb39c199320 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb39000e970 tx=0x7fb39000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:20.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.433+0000 7fb39a000700 1 -- 172.21.15.78:0/4167964939 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb390009790 con 0x7fb39c100c50 2024-03-14T09:53:20.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.433+0000 7fb39a000700 1 -- 172.21.15.78:0/4167964939 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb39000c7e0 con 0x7fb39c100c50 2024-03-14T09:53:20.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.433+0000 7fb3a3e00700 1 -- 172.21.15.78:0/4167964939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb394009d00 con 0x7fb39c100c50 2024-03-14T09:53:20.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.433+0000 7fb39a000700 1 -- 172.21.15.78:0/4167964939 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb390003c70 con 0x7fb39c100c50 2024-03-14T09:53:20.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.433+0000 7fb3a3e00700 1 -- 172.21.15.78:0/4167964939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39c1a1780 con 0x7fb39c100c50 2024-03-14T09:53:20.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.434+0000 7fb3a3e00700 1 -- 172.21.15.78:0/4167964939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb39c0679e0 con 0x7fb39c100c50 2024-03-14T09:53:20.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.436+0000 7fb39a000700 1 -- 172.21.15.78:0/4167964939 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fb39000cc50 con 0x7fb39c100c50 2024-03-14T09:53:20.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.437+0000 7fb39a000700 1 --2- 172.21.15.78:0/4167964939 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb3840a0b20 0x7fb3840a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:20.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.437+0000 7fb39a000700 1 -- 172.21.15.78:0/4167964939 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb390014070 con 0x7fb39c100c50 2024-03-14T09:53:20.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.437+0000 7fb3a0a00700 1 --2- 172.21.15.78:0/4167964939 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb3840a0b20 0x7fb3840a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:20.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.438+0000 7fb3a0a00700 1 --2- 172.21.15.78:0/4167964939 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb3840a0b20 0x7fb3840a2fe0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb388009df0 tx=0x7fb388009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:20.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.441+0000 7fb39a000700 1 -- 172.21.15.78:0/4167964939 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb390095c20 con 0x7fb39c100c50 2024-03-14T09:53:20.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.669+0000 7fb3a3e00700 1 -- 172.21.15.78:0/4167964939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb39c10c8e0 con 0x7fb39c100c50 2024-03-14T09:53:20.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.671+0000 7fb39a000700 1 -- 172.21.15.78:0/4167964939 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+24453 (secure 0 0 0) 0x7fb390003dd0 con 0x7fb39c100c50 2024-03-14T09:53:20.673 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:20.674 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":85,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:52:42.134030+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:53:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.673+0000 7fb3a3e00700 1 -- 172.21.15.78:0/4167964939 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb3840a0b20 msgr2=0x7fb3840a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.673+0000 7fb3a3e00700 1 --2- 172.21.15.78:0/4167964939 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb3840a0b20 0x7fb3840a2fe0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb388009df0 tx=0x7fb388009260 comp rx=0 tx=0).stop 2024-03-14T09:53:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.673+0000 7fb3a3e00700 1 -- 172.21.15.78:0/4167964939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c100c50 msgr2=0x7fb39c199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.673+0000 7fb3a3e00700 1 --2- 172.21.15.78:0/4167964939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c100c50 0x7fb39c199320 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb39000e970 tx=0x7fb39000ec80 comp rx=0 tx=0).stop 2024-03-14T09:53:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.673+0000 7fb3a3e00700 1 -- 172.21.15.78:0/4167964939 shutdown_connections 2024-03-14T09:53:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.673+0000 7fb3a3e00700 1 --2- 172.21.15.78:0/4167964939 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb3840a0b20 0x7fb3840a2fe0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.673+0000 7fb3a3e00700 1 --2- 172.21.15.78:0/4167964939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb39c1027e0 0x7fb39c1a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.673+0000 7fb3a3e00700 1 --2- 172.21.15.78:0/4167964939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39c100c50 0x7fb39c199320 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.673+0000 7fb3a3e00700 1 --2- 172.21.15.78:0/4167964939 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb39c101e40 0x7fb39c199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.673+0000 7fb3a3e00700 1 -- 172.21.15.78:0/4167964939 >> 172.21.15.78:0/4167964939 conn(0x7fb39c0fc1c0 msgr2=0x7fb39c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.673+0000 7fb3a3e00700 1 -- 172.21.15.78:0/4167964939 shutdown_connections 2024-03-14T09:53:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.673+0000 7fb3a3e00700 1 -- 172.21.15.78:0/4167964939 wait complete. 2024-03-14T09:53:20.694 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:53:20.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:20 smithi078 ceph-mon[38143]: pgmap v274: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-14T09:53:20.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:20 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2718878401' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:20.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:20 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/34879644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:20.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.887+0000 7f94eee00700 1 -- 172.21.15.78:0/463266842 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94e8100c50 msgr2=0x7f94e8101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:20.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.887+0000 7f94eee00700 1 --2- 172.21.15.78:0/463266842 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94e8100c50 0x7f94e8101070 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f94d80099c0 tx=0x7f94d8009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.887+0000 7f94eee00700 1 -- 172.21.15.78:0/463266842 shutdown_connections 2024-03-14T09:53:20.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.887+0000 7f94eee00700 1 --2- 172.21.15.78:0/463266842 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94e81027e0 0x7f94e8109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.887+0000 7f94eee00700 1 --2- 172.21.15.78:0/463266842 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94e8100c50 0x7f94e8101070 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.887+0000 7f94eee00700 1 --2- 172.21.15.78:0/463266842 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94e8101e40 0x7f94e81022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.887+0000 7f94eee00700 1 -- 172.21.15.78:0/463266842 >> 172.21.15.78:0/463266842 conn(0x7f94e80fc1c0 msgr2=0x7f94e80fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:20.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.887+0000 7f94eee00700 1 -- 172.21.15.78:0/463266842 shutdown_connections 2024-03-14T09:53:20.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.887+0000 7f94eee00700 1 -- 172.21.15.78:0/463266842 wait complete. 2024-03-14T09:53:20.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.888+0000 7f94eee00700 1 Processor -- start 2024-03-14T09:53:20.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.888+0000 7f94eee00700 1 -- start start 2024-03-14T09:53:20.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.889+0000 7f94eee00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94e8100c50 0x7f94e8074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:20.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.889+0000 7f94eee00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94e8101e40 0x7f94e80731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:20.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.889+0000 7f94eee00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94e81027e0 0x7f94e80736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:20.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.889+0000 7f94eee00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e8073e80 con 0x7f94e8101e40 2024-03-14T09:53:20.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.889+0000 7f94eee00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e8073ff0 con 0x7f94e8100c50 2024-03-14T09:53:20.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.889+0000 7f94eee00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e81a10b0 con 0x7f94e81027e0 2024-03-14T09:53:20.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.889+0000 7f94e7400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94e8101e40 0x7f94e80731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:20.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.889+0000 7f94e7400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94e8101e40 0x7f94e80731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41696/0 (socket says 172.21.15.78:41696) 2024-03-14T09:53:20.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.889+0000 7f94e7400700 1 -- 172.21.15.78:0/1312622874 learned_addr learned my addr 172.21.15.78:0/1312622874 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:20.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.889+0000 7f94e7e00700 1 --2- 172.21.15.78:0/1312622874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94e8100c50 0x7f94e8074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:20.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.889+0000 7f94ece00700 1 --2- 172.21.15.78:0/1312622874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94e81027e0 0x7f94e80736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:20.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.889+0000 7f94e7400700 1 -- 172.21.15.78:0/1312622874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94e81027e0 msgr2=0x7f94e80736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:20.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.889+0000 7f94e7400700 1 --2- 172.21.15.78:0/1312622874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94e81027e0 0x7f94e80736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.889+0000 7f94e7400700 1 -- 172.21.15.78:0/1312622874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94e8100c50 msgr2=0x7f94e8074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:20.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.889+0000 7f94e7400700 1 --2- 172.21.15.78:0/1312622874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94e8100c50 0x7f94e8074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:20.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.889+0000 7f94e7400700 1 -- 172.21.15.78:0/1312622874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d80095f0 con 0x7f94e8101e40 2024-03-14T09:53:20.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.890+0000 7f94e7400700 1 --2- 172.21.15.78:0/1312622874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94e8101e40 0x7f94e80731a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f94d400e9a0 tx=0x7f94d400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:20.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.890+0000 7f94e4c00700 1 -- 172.21.15.78:0/1312622874 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d4009790 con 0x7f94e8101e40 2024-03-14T09:53:20.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.890+0000 7f94e4c00700 1 -- 172.21.15.78:0/1312622874 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94d400cb70 con 0x7f94e8101e40 2024-03-14T09:53:20.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.890+0000 7f94e4c00700 1 -- 172.21.15.78:0/1312622874 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d4010ea0 con 0x7f94e8101e40 2024-03-14T09:53:20.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.890+0000 7f94eee00700 1 -- 172.21.15.78:0/1312622874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e81a1340 con 0x7f94e8101e40 2024-03-14T09:53:20.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.890+0000 7f94eee00700 1 -- 172.21.15.78:0/1312622874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e81a1800 con 0x7f94e8101e40 2024-03-14T09:53:20.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.891+0000 7f94eee00700 1 -- 172.21.15.78:0/1312622874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e8108120 con 0x7f94e8101e40 2024-03-14T09:53:20.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.893+0000 7f94e4c00700 1 -- 172.21.15.78:0/1312622874 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f94d400cce0 con 0x7f94e8101e40 2024-03-14T09:53:20.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.894+0000 7f94e4c00700 1 --2- 172.21.15.78:0/1312622874 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f94cc0a0b20 0x7f94cc0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:20.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.895+0000 7f94e4c00700 1 -- 172.21.15.78:0/1312622874 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f94d4014070 con 0x7f94e8101e40 2024-03-14T09:53:20.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.895+0000 7f94e7e00700 1 --2- 172.21.15.78:0/1312622874 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f94cc0a0b20 0x7f94cc0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:20.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.895+0000 7f94e7e00700 1 --2- 172.21.15.78:0/1312622874 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f94cc0a0b20 0x7f94cc0a2fe0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f94d8003730 tx=0x7f94d8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:20.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:20.897+0000 7f94e4c00700 1 -- 172.21.15.78:0/1312622874 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f94d4096120 con 0x7f94e8101e40 2024-03-14T09:53:20.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:20 smithi152 ceph-mon[48279]: pgmap v274: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-14T09:53:20.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:20 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2718878401' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:20.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:20 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/34879644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:21.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:20 smithi132 ceph-mon[47959]: pgmap v274: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-14T09:53:21.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:20 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2718878401' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:21.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:20 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/34879644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:21.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.148+0000 7f94eee00700 1 -- 172.21.15.78:0/1312622874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94e80679e0 con 0x7f94e8101e40 2024-03-14T09:53:21.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.149+0000 7f94e4c00700 1 -- 172.21.15.78:0/1312622874 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f94d4099740 con 0x7f94e8101e40 2024-03-14T09:53:21.150 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:21.150 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:52:30.715484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:53:21.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.152+0000 7f94eee00700 1 -- 172.21.15.78:0/1312622874 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f94cc0a0b20 msgr2=0x7f94cc0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.152+0000 7f94eee00700 1 --2- 172.21.15.78:0/1312622874 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f94cc0a0b20 0x7f94cc0a2fe0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f94d8003730 tx=0x7f94d8000bc0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.152+0000 7f94eee00700 1 -- 172.21.15.78:0/1312622874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94e8101e40 msgr2=0x7f94e80731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.152+0000 7f94eee00700 1 --2- 172.21.15.78:0/1312622874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94e8101e40 0x7f94e80731a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f94d400e9a0 tx=0x7f94d400ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.152+0000 7f94eee00700 1 -- 172.21.15.78:0/1312622874 shutdown_connections 2024-03-14T09:53:21.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.152+0000 7f94eee00700 1 --2- 172.21.15.78:0/1312622874 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f94cc0a0b20 0x7f94cc0a2fe0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.152+0000 7f94eee00700 1 --2- 172.21.15.78:0/1312622874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f94e8100c50 0x7f94e8074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.152+0000 7f94eee00700 1 --2- 172.21.15.78:0/1312622874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f94e81027e0 0x7f94e80736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.152+0000 7f94eee00700 1 --2- 172.21.15.78:0/1312622874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94e8101e40 0x7f94e80731a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.152+0000 7f94eee00700 1 -- 172.21.15.78:0/1312622874 >> 172.21.15.78:0/1312622874 conn(0x7f94e80fc1c0 msgr2=0x7f94e8105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.152+0000 7f94eee00700 1 -- 172.21.15.78:0/1312622874 shutdown_connections 2024-03-14T09:53:21.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.152+0000 7f94eee00700 1 -- 172.21.15.78:0/1312622874 wait complete. 2024-03-14T09:53:21.154 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2024-03-14T09:53:21.171 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-14T09:52:05.708711+0000', 'modified': '2024-03-14T09:52:30.715484+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24572, 'mds_1': 24538, 'mds_2': 24527}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24508': {'gid': 24508, 'name': 'c', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.152:6837/2123600730', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6836', 'nonce': 2123600730}, {'type': 'v1', 'addr': '172.21.15.152:6837', 'nonce': 2123600730}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24509': {'gid': 24509, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.132:6835/1709886688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6834', 'nonce': 1709886688}, {'type': 'v1', 'addr': '172.21.15.132:6835', 'nonce': 1709886688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24526': {'gid': 24526, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.152:6839/1965809497', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6838', 'nonce': 1965809497}, {'type': 'v1', 'addr': '172.21.15.152:6839', 'nonce': 1965809497}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24527': {'gid': 24527, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.132:6837/142126094', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6836', 'nonce': 142126094}, {'type': 'v1', 'addr': '172.21.15.132:6837', 'nonce': 142126094}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24538': {'gid': 24538, 'name': 'g', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.78:6839/4227580185', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6838', 'nonce': 4227580185}, {'type': 'v1', 'addr': '172.21.15.78:6839', 'nonce': 4227580185}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.132:6839/2192113559', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6838', 'nonce': 2192113559}, {'type': 'v1', 'addr': '172.21.15.132:6839', 'nonce': 2192113559}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-14T09:53:21.171 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-14T09:53:21.172 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-14T09:53:21.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.365+0000 7f71c1800700 1 -- 172.21.15.78:0/2479669567 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71bc1028e0 msgr2=0x7f71bc109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.365+0000 7f71c1800700 1 --2- 172.21.15.78:0/2479669567 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71bc1028e0 0x7f71bc109190 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f71b400b0d0 tx=0x7f71b400b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.365+0000 7f71c1800700 1 -- 172.21.15.78:0/2479669567 shutdown_connections 2024-03-14T09:53:21.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.365+0000 7f71c1800700 1 --2- 172.21.15.78:0/2479669567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71bc100d50 0x7f71bc101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.365+0000 7f71c1800700 1 --2- 172.21.15.78:0/2479669567 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71bc1028e0 0x7f71bc109190 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.365+0000 7f71c1800700 1 --2- 172.21.15.78:0/2479669567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71bc101f40 0x7f71bc1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.365+0000 7f71c1800700 1 -- 172.21.15.78:0/2479669567 >> 172.21.15.78:0/2479669567 conn(0x7f71bc0fc2c0 msgr2=0x7f71bc0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.365+0000 7f71c1800700 1 -- 172.21.15.78:0/2479669567 shutdown_connections 2024-03-14T09:53:21.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.365+0000 7f71c1800700 1 -- 172.21.15.78:0/2479669567 wait complete. 2024-03-14T09:53:21.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.366+0000 7f71c1800700 1 Processor -- start 2024-03-14T09:53:21.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.366+0000 7f71c1800700 1 -- start start 2024-03-14T09:53:21.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.366+0000 7f71c1800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71bc100d50 0x7f71bc074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.366+0000 7f71c1800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71bc101f40 0x7f71bc0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.366+0000 7f71c1800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71bc1028e0 0x7f71bc0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.366+0000 7f71c1800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc073e80 con 0x7f71bc1028e0 2024-03-14T09:53:21.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.366+0000 7f71c1800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc073ff0 con 0x7f71bc100d50 2024-03-14T09:53:21.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.366+0000 7f71c1800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc1a1270 con 0x7f71bc101f40 2024-03-14T09:53:21.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.367+0000 7f71bb400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71bc1028e0 0x7f71bc0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.367+0000 7f71bb400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71bc1028e0 0x7f71bc0736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41712/0 (socket says 172.21.15.78:41712) 2024-03-14T09:53:21.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.367+0000 7f71bb400700 1 -- 172.21.15.78:0/959664341 learned_addr learned my addr 172.21.15.78:0/959664341 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:21.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.367+0000 7f71ba000700 1 --2- 172.21.15.78:0/959664341 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71bc101f40 0x7f71bc0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.367+0000 7f71baa00700 1 --2- 172.21.15.78:0/959664341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71bc100d50 0x7f71bc074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.367+0000 7f71bb400700 1 -- 172.21.15.78:0/959664341 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71bc101f40 msgr2=0x7f71bc0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.367+0000 7f71bb400700 1 --2- 172.21.15.78:0/959664341 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71bc101f40 0x7f71bc0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.367+0000 7f71bb400700 1 -- 172.21.15.78:0/959664341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71bc100d50 msgr2=0x7f71bc074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.367+0000 7f71bb400700 1 --2- 172.21.15.78:0/959664341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71bc100d50 0x7f71bc074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.367+0000 7f71bb400700 1 -- 172.21.15.78:0/959664341 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b4009d00 con 0x7f71bc1028e0 2024-03-14T09:53:21.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.367+0000 7f71bb400700 1 --2- 172.21.15.78:0/959664341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71bc1028e0 0x7f71bc0736e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f71b4007b60 tx=0x7f71b4008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.368+0000 7f71b3400700 1 -- 172.21.15.78:0/959664341 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b400e040 con 0x7f71bc1028e0 2024-03-14T09:53:21.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.368+0000 7f71b3400700 1 -- 172.21.15.78:0/959664341 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71b401d930 con 0x7f71bc1028e0 2024-03-14T09:53:21.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.368+0000 7f71c1800700 1 -- 172.21.15.78:0/959664341 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71bc1a14a0 con 0x7f71bc1028e0 2024-03-14T09:53:21.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.368+0000 7f71b3400700 1 -- 172.21.15.78:0/959664341 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b401b9f0 con 0x7f71bc1028e0 2024-03-14T09:53:21.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.368+0000 7f71c1800700 1 -- 172.21.15.78:0/959664341 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71bc1a1870 con 0x7f71bc1028e0 2024-03-14T09:53:21.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.371+0000 7f71b3400700 1 -- 172.21.15.78:0/959664341 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f71b401daa0 con 0x7f71bc1028e0 2024-03-14T09:53:21.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.372+0000 7f71b3400700 1 --2- 172.21.15.78:0/959664341 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f71a00a0b20 0x7f71a00a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.372+0000 7f71b3400700 1 -- 172.21.15.78:0/959664341 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f71b40cd9b0 con 0x7f71bc1028e0 2024-03-14T09:53:21.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.372+0000 7f71d53e5b80 1 Processor -- start 2024-03-14T09:53:21.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.372+0000 7f71baa00700 1 --2- 172.21.15.78:0/959664341 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f71a00a0b20 0x7f71a00a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.372+0000 7f71d53e5b80 1 -- start start 2024-03-14T09:53:21.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71d53e5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bb97a0 0x559e06bb9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71d53e5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bba100 0x559e06bbaa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71d53e5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bbaf90 0x559e06bbb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71d53e5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06b12940 con 0x559e06bba100 2024-03-14T09:53:21.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71d53e5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06b10300 con 0x559e06bbaf90 2024-03-14T09:53:21.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71d53e5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06b0bbf0 con 0x559e06bb97a0 2024-03-14T09:53:21.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71baa00700 1 --2- 172.21.15.78:0/959664341 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f71a00a0b20 0x7f71a00a2fe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f71bc101da0 tx=0x7f71ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71ba000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bba100 0x559e06bbaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71ba000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bba100 0x559e06bbaa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41716/0 (socket says 172.21.15.78:41716) 2024-03-14T09:53:21.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71ba000700 1 -- 172.21.15.78:0/2421293350 learned_addr learned my addr 172.21.15.78:0/2421293350 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:21.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71baa00700 1 --2- 172.21.15.78:0/2421293350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bb97a0 0x559e06bb9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71bb400700 1 --2- 172.21.15.78:0/2421293350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bbaf90 0x559e06bbb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71ba000700 1 -- 172.21.15.78:0/2421293350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bb97a0 msgr2=0x559e06bb9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71ba000700 1 --2- 172.21.15.78:0/2421293350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bb97a0 0x559e06bb9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71ba000700 1 -- 172.21.15.78:0/2421293350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bbaf90 msgr2=0x559e06bbb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71ba000700 1 --2- 172.21.15.78:0/2421293350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bbaf90 0x559e06bbb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71ba000700 1 -- 172.21.15.78:0/2421293350 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e0698d360 con 0x559e06bba100 2024-03-14T09:53:21.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71baa00700 1 --2- 172.21.15.78:0/2421293350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bb97a0 0x559e06bb9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:21.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.373+0000 7f71bb400700 1 --2- 172.21.15.78:0/2421293350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bbaf90 0x559e06bbb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:21.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.374+0000 7f71ba000700 1 --2- 172.21.15.78:0/2421293350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bba100 0x559e06bbaa00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f71a8009d80 tx=0x7f71a8011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=679a40f9e1d868c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.374+0000 7f71a7e00700 1 -- 172.21.15.78:0/2421293350 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a8004230 con 0x559e06bba100 2024-03-14T09:53:21.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.374+0000 7f71a7e00700 1 -- 172.21.15.78:0/2421293350 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71a8004390 con 0x559e06bba100 2024-03-14T09:53:21.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.374+0000 7f71a7e00700 1 -- 172.21.15.78:0/2421293350 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a8019820 con 0x559e06bba100 2024-03-14T09:53:21.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.374+0000 7f71d53e5b80 1 -- 172.21.15.78:0/2421293350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bba100 msgr2=0x559e06bbaa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.374+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/2421293350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bba100 0x559e06bbaa00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f71a8009d80 tx=0x7f71a8011720 comp rx=0 tx=0).stop 2024-03-14T09:53:21.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.375+0000 7f71d53e5b80 1 -- 172.21.15.78:0/2421293350 shutdown_connections 2024-03-14T09:53:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.375+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/2421293350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bbaf90 0x559e06bbb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.375+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/2421293350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bb97a0 0x559e06bb9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.375+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/2421293350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bba100 0x559e06bbaa00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.375+0000 7f71d53e5b80 1 -- 172.21.15.78:0/2421293350 >> 172.21.15.78:0/2421293350 conn(0x559e06b2f1d0 msgr2=0x559e06bb4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.375+0000 7f71d53e5b80 1 -- 172.21.15.78:0/2421293350 shutdown_connections 2024-03-14T09:53:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.375+0000 7f71d53e5b80 1 -- 172.21.15.78:0/2421293350 wait complete. 2024-03-14T09:53:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.375+0000 7f71d53e5b80 1 -- start start 2024-03-14T09:53:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.375+0000 7f71d53e5b80 1 Processor -- start 2024-03-14T09:53:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.376+0000 7f71d53e5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bb97a0 0x559e06bc3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.376+0000 7f71d53e5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bba100 0x559e06bc3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.376+0000 7f71d53e5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bbaf90 0x559e06bcab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.376+0000 7f71d53e5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06aa7d30 con 0x559e06bb97a0 2024-03-14T09:53:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.376+0000 7f71d53e5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06aa79d0 con 0x559e06bba100 2024-03-14T09:53:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.376+0000 7f71d53e5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e0698f650 con 0x559e06bbaf90 2024-03-14T09:53:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.376+0000 7f71d53e5b80 10 client.0 _ll_register_callbacks cb 0x559e06913350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:53:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.376+0000 7f71baa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bb97a0 0x559e06bc3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.376+0000 7f71baa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bb97a0 0x559e06bc3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41724/0 (socket says 172.21.15.78:41724) 2024-03-14T09:53:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.376+0000 7f71ba000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bba100 0x559e06bc3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.376+0000 7f71baa00700 1 -- 172.21.15.78:0/642931127 learned_addr learned my addr 172.21.15.78:0/642931127 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.376+0000 7f71bb400700 1 --2- 172.21.15.78:0/642931127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bbaf90 0x559e06bcab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.377+0000 7f71ba000700 1 -- 172.21.15.78:0/642931127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bbaf90 msgr2=0x559e06bcab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.377+0000 7f71ba000700 1 --2- 172.21.15.78:0/642931127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bbaf90 0x559e06bcab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.377+0000 7f71ba000700 1 -- 172.21.15.78:0/642931127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bb97a0 msgr2=0x559e06bc3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.377+0000 7f71ba000700 1 --2- 172.21.15.78:0/642931127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bb97a0 0x559e06bc3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.377+0000 7f71ba000700 1 -- 172.21.15.78:0/642931127 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71bc1a1870 con 0x559e06bba100 2024-03-14T09:53:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.377+0000 7f71baa00700 1 --2- 172.21.15.78:0/642931127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bb97a0 0x559e06bc3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.377+0000 7f71ba000700 1 --2- 172.21.15.78:0/642931127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bba100 0x559e06bc3b90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f71a8009770 tx=0x7f71a8003930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=867414c6c431ea56 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.377+0000 7f71a4c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:53:21.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.377+0000 7f71a4c00700 1 -- 172.21.15.78:0/642931127 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a8005450 con 0x559e06bba100 2024-03-14T09:53:21.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.377+0000 7f71d53e5b80 1 -- 172.21.15.78:0/642931127 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559e06bac6d0 con 0x559e06bba100 2024-03-14T09:53:21.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.377+0000 7f71a4c00700 1 -- 172.21.15.78:0/642931127 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f71a8005ce0 con 0x559e06bba100 2024-03-14T09:53:21.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.378+0000 7f71a4c00700 1 -- 172.21.15.78:0/642931127 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71a8009ed0 con 0x559e06bba100 2024-03-14T09:53:21.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.378+0000 7f71d53e5b80 10 client.25133 fetch_fsmap learned FSMap version 24 2024-03-14T09:53:21.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.378+0000 7f71d53e5b80 1 -- 172.21.15.78:0/642931127 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7f71a80099e0 con 0x559e06bba100 2024-03-14T09:53:21.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.378+0000 7f71a4c00700 1 -- 172.21.15.78:0/642931127 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559e06bac6d0 con 0x559e06bba100 2024-03-14T09:53:21.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.378+0000 7f71a4c00700 1 -- 172.21.15.78:0/642931127 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f71a8021040 con 0x559e06bba100 2024-03-14T09:53:21.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.378+0000 7f71d53e5b80 10 client.25133 fetch_fsmap finished waiting for FSMap version 24 2024-03-14T09:53:21.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.378+0000 7f71d53e5b80 10 client.25133 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-14T09:53:21.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.378+0000 7f71d53e5b80 20 client.25133 populate_metadata read hostname 'smithi078' 2024-03-14T09:53:21.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.378+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/642931127 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559e06bce710 0x559e06bd3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.378+0000 7f71d53e5b80 4 client.25133 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:53:21.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.378+0000 7f71d53e5b80 1 -- 172.21.15.78:0/642931127 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559e06b046b0 con 0x559e06bce710 2024-03-14T09:53:21.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.379+0000 7f71baa00700 1 --2- 172.21.15.78:0/642931127 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559e06bce710 0x559e06bd3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.379+0000 7f71baa00700 1 --2- 172.21.15.78:0/642931127 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559e06bce710 0x559e06bd3a10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d01ac443da545e5 server_cookie=104fc7608a1655c7 in_seq=0 out_seq=0 2024-03-14T09:53:21.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.379+0000 7f71a4c00700 10 client.25133 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:53:21.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.380+0000 7f71a4c00700 1 -- 172.21.15.78:0/642931127 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x559e06b046b0 con 0x559e06bce710 2024-03-14T09:53:21.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.380+0000 7f71a4c00700 10 client.25133 handle_command_reply: tid=0 2024-03-14T09:53:21.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.380+0000 7f71d53e5b80 1 client.25133 shutdown 2024-03-14T09:53:21.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.380+0000 7f71d53e5b80 1 -- 172.21.15.78:0/642931127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bba100 msgr2=0x559e06bc3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.380+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/642931127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bba100 0x559e06bc3b90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f71a8009770 tx=0x7f71a8003930 comp rx=0 tx=0).stop 2024-03-14T09:53:21.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.380+0000 7f71d53e5b80 1 -- 172.21.15.78:0/642931127 shutdown_connections 2024-03-14T09:53:21.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.380+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/642931127 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559e06bce710 0x559e06bd3a10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.380+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/642931127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bbaf90 0x559e06bcab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.380+0000 7f71a4c00700 0 client.25133 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:53:21.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.380+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/642931127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bb97a0 0x559e06bc3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.380+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/642931127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bba100 0x559e06bc3b90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.380+0000 7f71d53e5b80 1 -- 172.21.15.78:0/642931127 >> 172.21.15.78:0/642931127 conn(0x559e06b2f1d0 msgr2=0x559e06bb3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.380+0000 7f71d53e5b80 1 -- 172.21.15.78:0/642931127 shutdown_connections 2024-03-14T09:53:21.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.380+0000 7f71d53e5b80 1 -- 172.21.15.78:0/642931127 wait complete. 2024-03-14T09:53:21.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.381+0000 7f71d53e5b80 20 client.25133 trim_cache size 0 max 16384 2024-03-14T09:53:21.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.398+0000 7f71d53e5b80 1 Processor -- start 2024-03-14T09:53:21.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.398+0000 7f71d53e5b80 1 -- start start 2024-03-14T09:53:21.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.398+0000 7f71d53e5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bb97a0 0x559e06bc8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.398+0000 7f71d53e5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bba100 0x559e06bc2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.398+0000 7f71d53e5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bbaf90 0x559e06bd7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.398+0000 7f71d53e5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06b046b0 con 0x559e06bbaf90 2024-03-14T09:53:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.398+0000 7f71d53e5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06bb5480 con 0x559e06bb97a0 2024-03-14T09:53:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.398+0000 7f71d53e5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06b08850 con 0x559e06bba100 2024-03-14T09:53:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.398+0000 7f71bb400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bbaf90 0x559e06bd7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.398+0000 7f71bb400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bbaf90 0x559e06bd7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41738/0 (socket says 172.21.15.78:41738) 2024-03-14T09:53:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.398+0000 7f71bb400700 1 -- 172.21.15.78:0/1262772390 learned_addr learned my addr 172.21.15.78:0/1262772390 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.398+0000 7f71ba000700 1 --2- 172.21.15.78:0/1262772390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bba100 0x559e06bc2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.398+0000 7f71baa00700 1 --2- 172.21.15.78:0/1262772390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bb97a0 0x559e06bc8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.399+0000 7f71bb400700 1 -- 172.21.15.78:0/1262772390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bba100 msgr2=0x559e06bc2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.399+0000 7f71bb400700 1 --2- 172.21.15.78:0/1262772390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bba100 0x559e06bc2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.399+0000 7f71bb400700 1 -- 172.21.15.78:0/1262772390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bb97a0 msgr2=0x559e06bc8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.399+0000 7f71bb400700 1 --2- 172.21.15.78:0/1262772390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bb97a0 0x559e06bc8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.399+0000 7f71bb400700 1 -- 172.21.15.78:0/1262772390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e06939b10 con 0x559e06bbaf90 2024-03-14T09:53:21.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.399+0000 7f71bb400700 1 --2- 172.21.15.78:0/1262772390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bbaf90 0x559e06bd7a30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f71b4097100 tx=0x7f71b4006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a252e34d5648cef3 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.399+0000 7f71b0c00700 1 -- 172.21.15.78:0/1262772390 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b40b2070 con 0x559e06bbaf90 2024-03-14T09:53:21.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.399+0000 7f71b0c00700 1 -- 172.21.15.78:0/1262772390 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71b409b300 con 0x559e06bbaf90 2024-03-14T09:53:21.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.399+0000 7f71b0c00700 1 -- 172.21.15.78:0/1262772390 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b409ae60 con 0x559e06bbaf90 2024-03-14T09:53:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.400+0000 7f71d53e5b80 1 -- 172.21.15.78:0/1262772390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bbaf90 msgr2=0x559e06bd7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.400+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/1262772390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bbaf90 0x559e06bd7a30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f71b4097100 tx=0x7f71b4006000 comp rx=0 tx=0).stop 2024-03-14T09:53:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.400+0000 7f71d53e5b80 1 -- 172.21.15.78:0/1262772390 shutdown_connections 2024-03-14T09:53:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.400+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/1262772390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bb97a0 0x559e06bc8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.400+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/1262772390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bbaf90 0x559e06bd7a30 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.400+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/1262772390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bba100 0x559e06bc2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.400+0000 7f71d53e5b80 1 -- 172.21.15.78:0/1262772390 >> 172.21.15.78:0/1262772390 conn(0x559e06b2f1d0 msgr2=0x559e06bc1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.400+0000 7f71d53e5b80 1 -- 172.21.15.78:0/1262772390 shutdown_connections 2024-03-14T09:53:21.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.400+0000 7f71d53e5b80 1 -- 172.21.15.78:0/1262772390 wait complete. 2024-03-14T09:53:21.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.400+0000 7f71d53e5b80 1 -- start start 2024-03-14T09:53:21.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.400+0000 7f71d53e5b80 1 Processor -- start 2024-03-14T09:53:21.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71d53e5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bb97a0 0x559e06bc7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71d53e5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bba100 0x559e06bc80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71d53e5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bbaf90 0x559e06bc8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71d53e5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06bc8e90 con 0x559e06bb97a0 2024-03-14T09:53:21.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71d53e5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06c20cf0 con 0x559e06bba100 2024-03-14T09:53:21.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71d53e5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e06c20e60 con 0x559e06bbaf90 2024-03-14T09:53:21.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71d53e5b80 10 client.0 _ll_register_callbacks cb 0x7f71b400bf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:53:21.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71baa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bb97a0 0x559e06bc7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71baa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bb97a0 0x559e06bc7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41742/0 (socket says 172.21.15.78:41742) 2024-03-14T09:53:21.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71baa00700 1 -- 172.21.15.78:0/2873604112 learned_addr learned my addr 172.21.15.78:0/2873604112 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:21.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71ba000700 1 --2- 172.21.15.78:0/2873604112 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bba100 0x559e06bc80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71bb400700 1 --2- 172.21.15.78:0/2873604112 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bbaf90 0x559e06bc8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71baa00700 1 -- 172.21.15.78:0/2873604112 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bbaf90 msgr2=0x559e06bc8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71baa00700 1 --2- 172.21.15.78:0/2873604112 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bbaf90 0x559e06bc8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71baa00700 1 -- 172.21.15.78:0/2873604112 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bba100 msgr2=0x559e06bc80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71baa00700 1 --2- 172.21.15.78:0/2873604112 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bba100 0x559e06bc80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.401+0000 7f71baa00700 1 -- 172.21.15.78:0/2873604112 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b40ba040 con 0x559e06bb97a0 2024-03-14T09:53:21.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.402+0000 7f71baa00700 1 --2- 172.21.15.78:0/2873604112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bb97a0 0x559e06bc7b70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x559e06bc41e0 tx=0x7f71ac0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4910b9bcade27788 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.402+0000 7f71a4c00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:53:21.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.402+0000 7f71a4c00700 1 -- 172.21.15.78:0/2873604112 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac022040 con 0x559e06bb97a0 2024-03-14T09:53:21.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.402+0000 7f71a4c00700 1 -- 172.21.15.78:0/2873604112 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f71ac004220 con 0x559e06bb97a0 2024-03-14T09:53:21.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.402+0000 7f71d53e5b80 1 -- 172.21.15.78:0/2873604112 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f71a8004390 con 0x559e06bb97a0 2024-03-14T09:53:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.402+0000 7f71a4c00700 1 -- 172.21.15.78:0/2873604112 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71ac0043e0 con 0x559e06bb97a0 2024-03-14T09:53:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.402+0000 7f71d53e5b80 10 client.15126 fetch_fsmap learned FSMap version 24 2024-03-14T09:53:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.402+0000 7f71d53e5b80 1 -- 172.21.15.78:0/2873604112 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x559e06ad1e50 con 0x559e06bb97a0 2024-03-14T09:53:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.402+0000 7f71a4c00700 1 -- 172.21.15.78:0/2873604112 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71a8004390 con 0x559e06bb97a0 2024-03-14T09:53:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.403+0000 7f71a4c00700 1 -- 172.21.15.78:0/2873604112 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f71ac003950 con 0x559e06bb97a0 2024-03-14T09:53:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.403+0000 7f71d53e5b80 10 client.15126 fetch_fsmap finished waiting for FSMap version 24 2024-03-14T09:53:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.403+0000 7f71d53e5b80 10 client.15126 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-14T09:53:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.403+0000 7f71d53e5b80 20 client.15126 populate_metadata read hostname 'smithi078' 2024-03-14T09:53:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.403+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/2873604112 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559e06c25d20 0x559e06c22650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.403+0000 7f71d53e5b80 4 client.15126 mds_command: new command op to 24527 tid=0[{"prefix": "status"}] 2024-03-14T09:53:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.403+0000 7f71d53e5b80 1 -- 172.21.15.78:0/2873604112 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559e0695aca0 con 0x559e06c25d20 2024-03-14T09:53:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.403+0000 7f71ba000700 1 --2- 172.21.15.78:0/2873604112 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559e06c25d20 0x559e06c22650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71a4c00700 10 client.15126 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:53:21.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71ba000700 1 --2- 172.21.15.78:0/2873604112 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559e06c25d20 0x559e06c22650 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3345f6d99f8b5f43 server_cookie=e03f6ad0c60149ed in_seq=0 out_seq=0 2024-03-14T09:53:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71a4c00700 1 -- 172.21.15.78:0/2873604112 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x559e0695aca0 con 0x559e06c25d20 2024-03-14T09:53:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71a4c00700 10 client.15126 handle_command_reply: tid=0 2024-03-14T09:53:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71d53e5b80 1 client.15126 shutdown 2024-03-14T09:53:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71d53e5b80 1 -- 172.21.15.78:0/2873604112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bb97a0 msgr2=0x559e06bc7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/2873604112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bb97a0 0x559e06bc7b70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x559e06bc41e0 tx=0x7f71ac0047c0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71d53e5b80 1 -- 172.21.15.78:0/2873604112 shutdown_connections 2024-03-14T09:53:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/2873604112 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559e06c25d20 0x559e06c22650 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/2873604112 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e06bbaf90 0x559e06bc8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/2873604112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e06bb97a0 0x559e06bc7b70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71d53e5b80 1 --2- 172.21.15.78:0/2873604112 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e06bba100 0x559e06bc80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71a4c00700 0 client.15126 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:53:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71d53e5b80 1 -- 172.21.15.78:0/2873604112 >> 172.21.15.78:0/2873604112 conn(0x559e06b2f1d0 msgr2=0x559e06bc2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71d53e5b80 1 -- 172.21.15.78:0/2873604112 shutdown_connections 2024-03-14T09:53:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71d53e5b80 1 -- 172.21.15.78:0/2873604112 wait complete. 2024-03-14T09:53:21.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.404+0000 7f71d53e5b80 20 client.15126 trim_cache size 0 max 16384 2024-03-14T09:53:21.410 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:53:21.410 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "83662c4e-e1e7-11ee-95c9-87774f69a715", 2024-03-14T09:53:21.410 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 2, 2024-03-14T09:53:21.410 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24527, 2024-03-14T09:53:21.410 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-14T09:53:21.410 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-14T09:53:21.410 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-14T09:53:21.410 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 51.695149827000002, 2024-03-14T09:53:21.410 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2024-03-14T09:53:21.410 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 85, 2024-03-14T09:53:21.410 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 83, 2024-03-14T09:53:21.411 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 91.234692925000004 2024-03-14T09:53:21.411 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:53:21.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.405+0000 7f71c1800700 1 -- 172.21.15.78:0/959664341 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f71a00a0b20 msgr2=0x7f71a00a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.405+0000 7f71c1800700 1 --2- 172.21.15.78:0/959664341 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f71a00a0b20 0x7f71a00a2fe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f71bc101da0 tx=0x7f71ac009260 comp rx=0 tx=0).stop 2024-03-14T09:53:21.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.405+0000 7f71c1800700 1 -- 172.21.15.78:0/959664341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71bc1028e0 msgr2=0x7f71bc0736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.405+0000 7f71c1800700 1 --2- 172.21.15.78:0/959664341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71bc1028e0 0x7f71bc0736e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f71b4007b60 tx=0x7f71b4008e40 comp rx=0 tx=0).stop 2024-03-14T09:53:21.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.405+0000 7f71c1800700 1 -- 172.21.15.78:0/959664341 shutdown_connections 2024-03-14T09:53:21.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.405+0000 7f71c1800700 1 --2- 172.21.15.78:0/959664341 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f71a00a0b20 0x7f71a00a2fe0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.405+0000 7f71c1800700 1 --2- 172.21.15.78:0/959664341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71bc100d50 0x7f71bc074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.405+0000 7f71c1800700 1 --2- 172.21.15.78:0/959664341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71bc1028e0 0x7f71bc0736e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.405+0000 7f71c1800700 1 --2- 172.21.15.78:0/959664341 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f71bc101f40 0x7f71bc0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.405+0000 7f71c1800700 1 -- 172.21.15.78:0/959664341 >> 172.21.15.78:0/959664341 conn(0x7f71bc0fc2c0 msgr2=0x7f71bc105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.405+0000 7f71c1800700 1 -- 172.21.15.78:0/959664341 shutdown_connections 2024-03-14T09:53:21.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.405+0000 7f71c1800700 1 -- 172.21.15.78:0/959664341 wait complete. 2024-03-14T09:53:21.424 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-03-14T09:53:21.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.618+0000 7fd217000700 1 -- 172.21.15.78:0/1468040637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2101028d0 msgr2=0x7fd210109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.618+0000 7fd217000700 1 --2- 172.21.15.78:0/1468040637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2101028d0 0x7fd210109180 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd20800b0d0 tx=0x7fd20800b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.618+0000 7fd217000700 1 -- 172.21.15.78:0/1468040637 shutdown_connections 2024-03-14T09:53:21.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.618+0000 7fd217000700 1 --2- 172.21.15.78:0/1468040637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2101028d0 0x7fd210109180 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.618+0000 7fd217000700 1 --2- 172.21.15.78:0/1468040637 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd210100d40 0x7fd210101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.618+0000 7fd217000700 1 --2- 172.21.15.78:0/1468040637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd210101f30 0x7fd210102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.618+0000 7fd217000700 1 -- 172.21.15.78:0/1468040637 >> 172.21.15.78:0/1468040637 conn(0x7fd2100fc2b0 msgr2=0x7fd2100fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.618+0000 7fd217000700 1 -- 172.21.15.78:0/1468040637 shutdown_connections 2024-03-14T09:53:21.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.618+0000 7fd217000700 1 -- 172.21.15.78:0/1468040637 wait complete. 2024-03-14T09:53:21.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.619+0000 7fd217000700 1 Processor -- start 2024-03-14T09:53:21.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.619+0000 7fd217000700 1 -- start start 2024-03-14T09:53:21.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.619+0000 7fd217000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd210100d40 0x7fd210074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.620+0000 7fd217000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd210101f30 0x7fd2100731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.620+0000 7fd217000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2101028d0 0x7fd2100736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.620+0000 7fd217000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd210073e80 con 0x7fd210101f30 2024-03-14T09:53:21.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.620+0000 7fd217000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd210073ff0 con 0x7fd210100d40 2024-03-14T09:53:21.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.620+0000 7fd217000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2101a1270 con 0x7fd2101028d0 2024-03-14T09:53:21.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.620+0000 7fd20f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd210101f30 0x7fd2100731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.620+0000 7fd20f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd210101f30 0x7fd2100731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41752/0 (socket says 172.21.15.78:41752) 2024-03-14T09:53:21.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.620+0000 7fd20f400700 1 -- 172.21.15.78:0/548937121 learned_addr learned my addr 172.21.15.78:0/548937121 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:21.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.620+0000 7fd215000700 1 --2- 172.21.15.78:0/548937121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2101028d0 0x7fd2100736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.620+0000 7fd20fe00700 1 --2- 172.21.15.78:0/548937121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd210100d40 0x7fd210074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.620+0000 7fd20f400700 1 -- 172.21.15.78:0/548937121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2101028d0 msgr2=0x7fd2100736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.620+0000 7fd20f400700 1 --2- 172.21.15.78:0/548937121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2101028d0 0x7fd2100736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.620+0000 7fd20f400700 1 -- 172.21.15.78:0/548937121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd210100d40 msgr2=0x7fd210074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.620+0000 7fd20f400700 1 --2- 172.21.15.78:0/548937121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd210100d40 0x7fd210074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.620+0000 7fd20f400700 1 -- 172.21.15.78:0/548937121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd208009d00 con 0x7fd210101f30 2024-03-14T09:53:21.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.620+0000 7fd20f400700 1 --2- 172.21.15.78:0/548937121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd210101f30 0x7fd2100731a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00e9a0 tx=0x7fd1fc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.621+0000 7fd20cc00700 1 -- 172.21.15.78:0/548937121 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc009790 con 0x7fd210101f30 2024-03-14T09:53:21.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.621+0000 7fd20cc00700 1 -- 172.21.15.78:0/548937121 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1fc00cb70 con 0x7fd210101f30 2024-03-14T09:53:21.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.621+0000 7fd217000700 1 -- 172.21.15.78:0/548937121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2101a1500 con 0x7fd210101f30 2024-03-14T09:53:21.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.621+0000 7fd20cc00700 1 -- 172.21.15.78:0/548937121 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc010ea0 con 0x7fd210101f30 2024-03-14T09:53:21.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.621+0000 7fd217000700 1 -- 172.21.15.78:0/548937121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2101a1a60 con 0x7fd210101f30 2024-03-14T09:53:21.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.625+0000 7fd20cc00700 1 -- 172.21.15.78:0/548937121 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fd1fc00cce0 con 0x7fd210101f30 2024-03-14T09:53:21.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.626+0000 7fd22ab5cb80 1 Processor -- start 2024-03-14T09:53:21.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.626+0000 7fd22ab5cb80 1 -- start start 2024-03-14T09:53:21.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.627+0000 7fd20cc00700 1 --2- 172.21.15.78:0/548937121 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd1f40a0ba0 0x7fd1f40a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.627+0000 7fd20cc00700 1 -- 172.21.15.78:0/548937121 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd1fc014070 con 0x7fd210101f30 2024-03-14T09:53:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.627+0000 7fd22ab5cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30d7a0 0x55559a30dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.627+0000 7fd215000700 1 --2- 172.21.15.78:0/548937121 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd1f40a0ba0 0x7fd1f40a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.627+0000 7fd22ab5cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30e100 0x55559a30ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.627+0000 7fd22ab5cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30ef90 0x55559a30f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.627+0000 7fd22ab5cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559a266940 con 0x55559a30e100 2024-03-14T09:53:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.627+0000 7fd22ab5cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559a264300 con 0x55559a30ef90 2024-03-14T09:53:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.627+0000 7fd22ab5cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559a25fbf0 con 0x55559a30d7a0 2024-03-14T09:53:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.627+0000 7fd20f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30e100 0x55559a30ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.627+0000 7fd20f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30e100 0x55559a30ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41754/0 (socket says 172.21.15.78:41754) 2024-03-14T09:53:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.627+0000 7fd20fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30d7a0 0x55559a30dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.627+0000 7fd20f400700 1 -- 172.21.15.78:0/4243044257 learned_addr learned my addr 172.21.15.78:0/4243044257 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.628+0000 7fd20f400700 1 -- 172.21.15.78:0/4243044257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30d7a0 msgr2=0x55559a30dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.628+0000 7fd215000700 1 --2- 172.21.15.78:0/4243044257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30ef90 0x55559a30f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.628+0000 7fd20f400700 1 --2- 172.21.15.78:0/4243044257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30d7a0 0x55559a30dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.628+0000 7fd20f400700 1 -- 172.21.15.78:0/4243044257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30ef90 msgr2=0x55559a30f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.628+0000 7fd20f400700 1 --2- 172.21.15.78:0/4243044257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30ef90 0x55559a30f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.628+0000 7fd20f400700 1 -- 172.21.15.78:0/4243044257 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55559a0e1360 con 0x55559a30e100 2024-03-14T09:53:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.628+0000 7fd215000700 1 --2- 172.21.15.78:0/4243044257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30ef90 0x55559a30f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:53:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.628+0000 7fd215000700 1 --2- 172.21.15.78:0/548937121 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd1f40a0ba0 0x7fd1f40a3060 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd208009fd0 tx=0x7fd208009340 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.628+0000 7fd20f400700 1 --2- 172.21.15.78:0/4243044257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30e100 0x55559a30ea00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc0a1800 tx=0x7fd1fc0992e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989f7d94bb5ff42b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.628+0000 7fd205600700 1 -- 172.21.15.78:0/4243044257 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0b7ea0 con 0x55559a30e100 2024-03-14T09:53:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.628+0000 7fd205600700 1 -- 172.21.15.78:0/4243044257 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1fc09d6e0 con 0x55559a30e100 2024-03-14T09:53:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.628+0000 7fd205600700 1 -- 172.21.15.78:0/4243044257 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc098fa0 con 0x55559a30e100 2024-03-14T09:53:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.629+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/4243044257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30e100 msgr2=0x55559a30ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.629+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/4243044257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30e100 0x55559a30ea00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc0a1800 tx=0x7fd1fc0992e0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.629+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/4243044257 shutdown_connections 2024-03-14T09:53:21.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.629+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/4243044257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30ef90 0x55559a30f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.629+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/4243044257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30d7a0 0x55559a30dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.629+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/4243044257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30e100 0x55559a30ea00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.629+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/4243044257 >> 172.21.15.78:0/4243044257 conn(0x55559a2831d0 msgr2=0x55559a308580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.629+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/4243044257 shutdown_connections 2024-03-14T09:53:21.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.629+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/4243044257 wait complete. 2024-03-14T09:53:21.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.629+0000 7fd22ab5cb80 1 -- start start 2024-03-14T09:53:21.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.630+0000 7fd22ab5cb80 1 Processor -- start 2024-03-14T09:53:21.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.630+0000 7fd22ab5cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30d7a0 0x55559a317670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.630+0000 7fd22ab5cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30e100 0x55559a317bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.630+0000 7fd22ab5cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30ef90 0x55559a31ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.630+0000 7fd22ab5cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559a1fbd30 con 0x55559a30ef90 2024-03-14T09:53:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.630+0000 7fd22ab5cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559a1fb9d0 con 0x55559a30e100 2024-03-14T09:53:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.630+0000 7fd22ab5cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559a0e3650 con 0x55559a30d7a0 2024-03-14T09:53:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.630+0000 7fd22ab5cb80 10 client.0 _ll_register_callbacks cb 0x55559a067350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:53:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.630+0000 7fd215000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30ef90 0x55559a31ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.630+0000 7fd20fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30d7a0 0x55559a317670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.630+0000 7fd20fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30d7a0 0x55559a317670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:39594/0 (socket says 172.21.15.78:39594) 2024-03-14T09:53:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.630+0000 7fd20f400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30e100 0x55559a317bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.630+0000 7fd215000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30ef90 0x55559a31ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41766/0 (socket says 172.21.15.78:41766) 2024-03-14T09:53:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.630+0000 7fd20fe00700 1 -- 172.21.15.78:0/1259739301 learned_addr learned my addr 172.21.15.78:0/1259739301 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.631+0000 7fd20fe00700 1 -- 172.21.15.78:0/1259739301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30e100 msgr2=0x55559a317bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.631+0000 7fd20fe00700 1 --2- 172.21.15.78:0/1259739301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30e100 0x55559a317bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.631+0000 7fd20fe00700 1 -- 172.21.15.78:0/1259739301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30ef90 msgr2=0x55559a31ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.631+0000 7fd20fe00700 1 --2- 172.21.15.78:0/1259739301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30ef90 0x55559a31ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.631+0000 7fd20fe00700 1 -- 172.21.15.78:0/1259739301 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2101a1a60 con 0x55559a30d7a0 2024-03-14T09:53:21.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.631+0000 7fd215000700 1 --2- 172.21.15.78:0/1259739301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30ef90 0x55559a31ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:21.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.631+0000 7fd20fe00700 1 --2- 172.21.15.78:0/1259739301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30d7a0 0x55559a317670 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd200012700 tx=0x7fd200012a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cb2e91038fbe3e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.631+0000 7fd1fa000700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:53:21.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.631+0000 7fd1fa000700 1 -- 172.21.15.78:0/1259739301 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd200017070 con 0x55559a30d7a0 2024-03-14T09:53:21.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.631+0000 7fd1fa000700 1 -- 172.21.15.78:0/1259739301 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd200004240 con 0x55559a30d7a0 2024-03-14T09:53:21.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.631+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/1259739301 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55559a3006d0 con 0x55559a30d7a0 2024-03-14T09:53:21.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.632+0000 7fd1fa000700 1 -- 172.21.15.78:0/1259739301 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd200004400 con 0x55559a30d7a0 2024-03-14T09:53:21.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.632+0000 7fd22ab5cb80 10 client.25150 fetch_fsmap learned FSMap version 24 2024-03-14T09:53:21.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.632+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/1259739301 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55559a08db10 con 0x55559a30d7a0 2024-03-14T09:53:21.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.632+0000 7fd1fa000700 1 -- 172.21.15.78:0/1259739301 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55559a3006d0 con 0x55559a30d7a0 2024-03-14T09:53:21.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.632+0000 7fd1fa000700 1 -- 172.21.15.78:0/1259739301 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd20000f730 con 0x55559a30d7a0 2024-03-14T09:53:21.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.632+0000 7fd22ab5cb80 10 client.25150 fetch_fsmap finished waiting for FSMap version 24 2024-03-14T09:53:21.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.632+0000 7fd22ab5cb80 10 client.25150 resolve_mds: resolved name 'g' to daemon mds.g 2024-03-14T09:53:21.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.632+0000 7fd22ab5cb80 20 client.25150 populate_metadata read hostname 'smithi078' 2024-03-14T09:53:21.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.632+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/1259739301 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55559a3227e0 0x55559a327ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.632+0000 7fd22ab5cb80 4 client.25150 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:53:21.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.632+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/1259739301 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55559a2586b0 con 0x55559a3227e0 2024-03-14T09:53:21.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.633+0000 7fd20f400700 1 --2- 172.21.15.78:0/1259739301 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55559a3227e0 0x55559a327ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.633+0000 7fd20f400700 1 --2- 172.21.15.78:0/1259739301 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55559a3227e0 0x55559a327ae0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8e8c0bf0de9a246d server_cookie=3fa17ca18775b166 in_seq=0 out_seq=0 2024-03-14T09:53:21.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.633+0000 7fd1fa000700 10 client.25150 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:53:21.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.634+0000 7fd1fa000700 1 -- 172.21.15.78:0/1259739301 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55559a2586b0 con 0x55559a3227e0 2024-03-14T09:53:21.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.634+0000 7fd1fa000700 10 client.25150 handle_command_reply: tid=0 2024-03-14T09:53:21.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.634+0000 7fd22ab5cb80 1 client.25150 shutdown 2024-03-14T09:53:21.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.634+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/1259739301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30d7a0 msgr2=0x55559a317670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.634+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/1259739301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30d7a0 0x55559a317670 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd200012700 tx=0x7fd200012a10 comp rx=0 tx=0).stop 2024-03-14T09:53:21.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.634+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/1259739301 shutdown_connections 2024-03-14T09:53:21.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.634+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/1259739301 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55559a3227e0 0x55559a327ae0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.634+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/1259739301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30ef90 0x55559a31ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.634+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/1259739301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30d7a0 0x55559a317670 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.634+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/1259739301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30e100 0x55559a317bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.634+0000 7fd1fa000700 0 client.25150 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:53:21.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.634+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/1259739301 >> 172.21.15.78:0/1259739301 conn(0x55559a2831d0 msgr2=0x55559a307bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.634+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/1259739301 shutdown_connections 2024-03-14T09:53:21.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.634+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/1259739301 wait complete. 2024-03-14T09:53:21.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.634+0000 7fd22ab5cb80 20 client.25150 trim_cache size 0 max 16384 2024-03-14T09:53:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.652+0000 7fd22ab5cb80 1 Processor -- start 2024-03-14T09:53:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.652+0000 7fd22ab5cb80 1 -- start start 2024-03-14T09:53:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.652+0000 7fd22ab5cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30d7a0 0x55559a30db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.652+0000 7fd22ab5cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30e100 0x55559a31cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.652+0000 7fd22ab5cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30ef90 0x55559a30f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.652+0000 7fd22ab5cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559a2589c0 con 0x55559a30ef90 2024-03-14T09:53:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.652+0000 7fd22ab5cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559a25c880 con 0x55559a30d7a0 2024-03-14T09:53:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.652+0000 7fd22ab5cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559a3208f0 con 0x55559a30e100 2024-03-14T09:53:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.652+0000 7fd215000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30ef90 0x55559a30f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.652+0000 7fd215000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30ef90 0x55559a30f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41770/0 (socket says 172.21.15.78:41770) 2024-03-14T09:53:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.652+0000 7fd215000700 1 -- 172.21.15.78:0/3561055357 learned_addr learned my addr 172.21.15.78:0/3561055357 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.652+0000 7fd20fe00700 1 --2- 172.21.15.78:0/3561055357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30d7a0 0x55559a30db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.652+0000 7fd20f400700 1 --2- 172.21.15.78:0/3561055357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30e100 0x55559a31cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.653+0000 7fd215000700 1 -- 172.21.15.78:0/3561055357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30e100 msgr2=0x55559a31cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.653+0000 7fd215000700 1 --2- 172.21.15.78:0/3561055357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30e100 0x55559a31cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.653+0000 7fd215000700 1 -- 172.21.15.78:0/3561055357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30d7a0 msgr2=0x55559a30db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.653+0000 7fd215000700 1 --2- 172.21.15.78:0/3561055357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30d7a0 0x55559a30db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.653+0000 7fd215000700 1 -- 172.21.15.78:0/3561055357 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55559a225e50 con 0x55559a30ef90 2024-03-14T09:53:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.653+0000 7fd20fe00700 1 --2- 172.21.15.78:0/3561055357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30d7a0 0x55559a30db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:53:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.653+0000 7fd215000700 1 --2- 172.21.15.78:0/3561055357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30ef90 0x55559a30f370 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd208015000 tx=0x7fd20800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a391b4cad7b5c58 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.653+0000 7fd206000700 1 -- 172.21.15.78:0/3561055357 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20800e040 con 0x55559a30ef90 2024-03-14T09:53:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.653+0000 7fd206000700 1 -- 172.21.15.78:0/3561055357 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd20802cb10 con 0x55559a30ef90 2024-03-14T09:53:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.653+0000 7fd206000700 1 -- 172.21.15.78:0/3561055357 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20801cac0 con 0x55559a30ef90 2024-03-14T09:53:21.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.654+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/3561055357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30ef90 msgr2=0x55559a30f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.654+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/3561055357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30ef90 0x55559a30f370 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd208015000 tx=0x7fd20800b040 comp rx=0 tx=0).stop 2024-03-14T09:53:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.654+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/3561055357 shutdown_connections 2024-03-14T09:53:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.654+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/3561055357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30d7a0 0x55559a30db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.654+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/3561055357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30ef90 0x55559a30f370 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.654+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/3561055357 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30e100 0x55559a31cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.654+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/3561055357 >> 172.21.15.78:0/3561055357 conn(0x55559a2831d0 msgr2=0x55559a308550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.654+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/3561055357 shutdown_connections 2024-03-14T09:53:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.654+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/3561055357 wait complete. 2024-03-14T09:53:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.654+0000 7fd22ab5cb80 1 -- start start 2024-03-14T09:53:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.654+0000 7fd22ab5cb80 1 Processor -- start 2024-03-14T09:53:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd22ab5cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30d7a0 0x55559a31bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd22ab5cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30e100 0x55559a31c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd22ab5cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30ef90 0x55559a31c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd22ab5cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559a31ccd0 con 0x55559a30d7a0 2024-03-14T09:53:21.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd22ab5cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559a31ce40 con 0x55559a30ef90 2024-03-14T09:53:21.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd22ab5cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55559a374dc0 con 0x55559a30e100 2024-03-14T09:53:21.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd22ab5cb80 10 client.0 _ll_register_callbacks cb 0x7fd200009ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:53:21.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd20fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30d7a0 0x55559a31bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd20fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30d7a0 0x55559a31bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41786/0 (socket says 172.21.15.78:41786) 2024-03-14T09:53:21.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd20f400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30e100 0x55559a31c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd20fe00700 1 -- 172.21.15.78:0/2625071788 learned_addr learned my addr 172.21.15.78:0/2625071788 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:21.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd215000700 1 --2- 172.21.15.78:0/2625071788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30ef90 0x55559a31c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd20fe00700 1 -- 172.21.15.78:0/2625071788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30e100 msgr2=0x55559a31c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd20fe00700 1 --2- 172.21.15.78:0/2625071788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30e100 0x55559a31c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd20fe00700 1 -- 172.21.15.78:0/2625071788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30ef90 msgr2=0x55559a31c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd20fe00700 1 --2- 172.21.15.78:0/2625071788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30ef90 0x55559a31c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.655+0000 7fd20fe00700 1 -- 172.21.15.78:0/2625071788 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd210109ad0 con 0x55559a30d7a0 2024-03-14T09:53:21.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.656+0000 7fd20fe00700 1 --2- 172.21.15.78:0/2625071788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30d7a0 0x55559a31bb40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x55559a30e860 tx=0x7fd200005990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2acfee0ec7238a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.656+0000 7fd1fa000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:53:21.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.656+0000 7fd1fa000700 1 -- 172.21.15.78:0/2625071788 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20002d030 con 0x55559a30d7a0 2024-03-14T09:53:21.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.656+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/2625071788 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd1fc09d6e0 con 0x55559a30d7a0 2024-03-14T09:53:21.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.656+0000 7fd1fa000700 1 -- 172.21.15.78:0/2625071788 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd20002e070 con 0x55559a30d7a0 2024-03-14T09:53:21.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.656+0000 7fd1fa000700 1 -- 172.21.15.78:0/2625071788 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd200017040 con 0x55559a30d7a0 2024-03-14T09:53:21.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.656+0000 7fd22ab5cb80 10 client.15144 fetch_fsmap learned FSMap version 24 2024-03-14T09:53:21.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.656+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/2625071788 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55559a0e5c80 con 0x55559a30d7a0 2024-03-14T09:53:21.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.656+0000 7fd1fa000700 1 -- 172.21.15.78:0/2625071788 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1fc09d6e0 con 0x55559a30d7a0 2024-03-14T09:53:21.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.657+0000 7fd1fa000700 1 -- 172.21.15.78:0/2625071788 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd2000104f0 con 0x55559a30d7a0 2024-03-14T09:53:21.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.657+0000 7fd22ab5cb80 10 client.15144 fetch_fsmap finished waiting for FSMap version 24 2024-03-14T09:53:21.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.657+0000 7fd22ab5cb80 10 client.15144 resolve_mds: resolved name 'g' to daemon mds.g 2024-03-14T09:53:21.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.657+0000 7fd22ab5cb80 20 client.15144 populate_metadata read hostname 'smithi078' 2024-03-14T09:53:21.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.657+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/2625071788 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55559a379be0 0x55559a376530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.657+0000 7fd22ab5cb80 4 client.15144 mds_command: new command op to 24538 tid=0[{"prefix": "status"}] 2024-03-14T09:53:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.657+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/2625071788 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55559a0aeca0 con 0x55559a379be0 2024-03-14T09:53:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.657+0000 7fd20f400700 1 --2- 172.21.15.78:0/2625071788 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55559a379be0 0x55559a376530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.657+0000 7fd20f400700 1 --2- 172.21.15.78:0/2625071788 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55559a379be0 0x55559a376530 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=488d0d7526aa973 server_cookie=9839966df8a22d7c in_seq=0 out_seq=0 2024-03-14T09:53:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.657+0000 7fd1fa000700 10 client.15144 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:53:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.658+0000 7fd1fa000700 1 -- 172.21.15.78:0/2625071788 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55559a0aeca0 con 0x55559a379be0 2024-03-14T09:53:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.658+0000 7fd1fa000700 10 client.15144 handle_command_reply: tid=0 2024-03-14T09:53:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.658+0000 7fd22ab5cb80 1 client.15144 shutdown 2024-03-14T09:53:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.658+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/2625071788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30d7a0 msgr2=0x55559a31bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.658+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/2625071788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30d7a0 0x55559a31bb40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x55559a30e860 tx=0x7fd200005990 comp rx=0 tx=0).stop 2024-03-14T09:53:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.658+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/2625071788 shutdown_connections 2024-03-14T09:53:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.658+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/2625071788 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55559a379be0 0x55559a376530 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.658+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/2625071788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55559a30ef90 0x55559a31c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.658+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/2625071788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55559a30d7a0 0x55559a31bb40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.658+0000 7fd1fa000700 0 client.15144 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:53:21.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.658+0000 7fd22ab5cb80 1 --2- 172.21.15.78:0/2625071788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55559a30e100 0x55559a31c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.658+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/2625071788 >> 172.21.15.78:0/2625071788 conn(0x55559a2831d0 msgr2=0x55559a315460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.658+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/2625071788 shutdown_connections 2024-03-14T09:53:21.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.658+0000 7fd22ab5cb80 1 -- 172.21.15.78:0/2625071788 wait complete. 2024-03-14T09:53:21.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.658+0000 7fd22ab5cb80 20 client.15144 trim_cache size 0 max 16384 2024-03-14T09:53:21.664 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:53:21.664 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "83662c4e-e1e7-11ee-95c9-87774f69a715", 2024-03-14T09:53:21.664 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 1, 2024-03-14T09:53:21.664 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24538, 2024-03-14T09:53:21.664 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-14T09:53:21.664 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-14T09:53:21.665 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-14T09:53:21.665 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 52.952901791000002, 2024-03-14T09:53:21.665 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2024-03-14T09:53:21.665 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 85, 2024-03-14T09:53:21.665 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 83, 2024-03-14T09:53:21.665 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 87.199693877000001 2024-03-14T09:53:21.665 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:53:21.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.659+0000 7fd217000700 1 -- 172.21.15.78:0/548937121 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd1f40a0ba0 msgr2=0x7fd1f40a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.659+0000 7fd217000700 1 --2- 172.21.15.78:0/548937121 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd1f40a0ba0 0x7fd1f40a3060 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd208009fd0 tx=0x7fd208009340 comp rx=0 tx=0).stop 2024-03-14T09:53:21.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.659+0000 7fd217000700 1 -- 172.21.15.78:0/548937121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd210101f30 msgr2=0x7fd2100731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.659+0000 7fd217000700 1 --2- 172.21.15.78:0/548937121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd210101f30 0x7fd2100731a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00e9a0 tx=0x7fd1fc00ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.659+0000 7fd217000700 1 -- 172.21.15.78:0/548937121 shutdown_connections 2024-03-14T09:53:21.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.659+0000 7fd217000700 1 --2- 172.21.15.78:0/548937121 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd1f40a0ba0 0x7fd1f40a3060 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.659+0000 7fd217000700 1 --2- 172.21.15.78:0/548937121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd210100d40 0x7fd210074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.659+0000 7fd217000700 1 --2- 172.21.15.78:0/548937121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2101028d0 0x7fd2100736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.659+0000 7fd217000700 1 --2- 172.21.15.78:0/548937121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd210101f30 0x7fd2100731a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.659+0000 7fd217000700 1 -- 172.21.15.78:0/548937121 >> 172.21.15.78:0/548937121 conn(0x7fd2100fc2b0 msgr2=0x7fd210105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.659+0000 7fd217000700 1 -- 172.21.15.78:0/548937121 shutdown_connections 2024-03-14T09:53:21.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.659+0000 7fd217000700 1 -- 172.21.15.78:0/548937121 wait complete. 2024-03-14T09:53:21.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:21 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/4167964939' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:21.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:21 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1312622874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:21.677 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-14T09:53:21.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.874+0000 7f15b1a00700 1 -- 172.21.15.78:0/3212690296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15ac101f30 msgr2=0x7f15ac102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.874+0000 7f15b1a00700 1 --2- 172.21.15.78:0/3212690296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15ac101f30 0x7f15ac102390 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f15980099c0 tx=0x7f1598009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.874+0000 7f15b1a00700 1 -- 172.21.15.78:0/3212690296 shutdown_connections 2024-03-14T09:53:21.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.874+0000 7f15b1a00700 1 --2- 172.21.15.78:0/3212690296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15ac1028d0 0x7f15ac109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.874+0000 7f15b1a00700 1 --2- 172.21.15.78:0/3212690296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15ac100d40 0x7f15ac101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.874+0000 7f15b1a00700 1 --2- 172.21.15.78:0/3212690296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15ac101f30 0x7f15ac102390 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.874+0000 7f15b1a00700 1 -- 172.21.15.78:0/3212690296 >> 172.21.15.78:0/3212690296 conn(0x7f15ac0fc2b0 msgr2=0x7f15ac0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.874+0000 7f15b1a00700 1 -- 172.21.15.78:0/3212690296 shutdown_connections 2024-03-14T09:53:21.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.874+0000 7f15b1a00700 1 -- 172.21.15.78:0/3212690296 wait complete. 2024-03-14T09:53:21.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.875+0000 7f15b1a00700 1 Processor -- start 2024-03-14T09:53:21.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.875+0000 7f15b1a00700 1 -- start start 2024-03-14T09:53:21.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.875+0000 7f15b1a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15ac100d40 0x7f15ac197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.875+0000 7f15b1a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15ac101f30 0x7f15ac197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.875+0000 7f15b1a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15ac1028d0 0x7f15ac19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.875+0000 7f15b1a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac19ec60 con 0x7f15ac100d40 2024-03-14T09:53:21.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.876+0000 7f15b1a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac19eda0 con 0x7f15ac101f30 2024-03-14T09:53:21.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.876+0000 7f15b1a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac19ef30 con 0x7f15ac1028d0 2024-03-14T09:53:21.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.876+0000 7f15aaa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15ac100d40 0x7f15ac197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.876+0000 7f15aaa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15ac100d40 0x7f15ac197130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41810/0 (socket says 172.21.15.78:41810) 2024-03-14T09:53:21.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.876+0000 7f15aaa00700 1 -- 172.21.15.78:0/4150440396 learned_addr learned my addr 172.21.15.78:0/4150440396 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:21.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.876+0000 7f15aa000700 1 --2- 172.21.15.78:0/4150440396 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15ac101f30 0x7f15ac197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.876+0000 7f15ab400700 1 --2- 172.21.15.78:0/4150440396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15ac1028d0 0x7f15ac19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.876+0000 7f15aaa00700 1 -- 172.21.15.78:0/4150440396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15ac1028d0 msgr2=0x7f15ac19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.876+0000 7f15aaa00700 1 --2- 172.21.15.78:0/4150440396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15ac1028d0 0x7f15ac19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.876+0000 7f15aaa00700 1 -- 172.21.15.78:0/4150440396 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15ac101f30 msgr2=0x7f15ac197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.876+0000 7f15aaa00700 1 --2- 172.21.15.78:0/4150440396 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15ac101f30 0x7f15ac197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.876+0000 7f15aaa00700 1 -- 172.21.15.78:0/4150440396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159c009520 con 0x7f15ac100d40 2024-03-14T09:53:21.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.876+0000 7f15ab400700 1 --2- 172.21.15.78:0/4150440396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15ac1028d0 0x7f15ac19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:53:21.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.876+0000 7f15aaa00700 1 --2- 172.21.15.78:0/4150440396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15ac100d40 0x7f15ac197130 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f159c00ea50 tx=0x7f159c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.877+0000 7f15a3400700 1 -- 172.21.15.78:0/4150440396 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c00cbd0 con 0x7f15ac100d40 2024-03-14T09:53:21.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.877+0000 7f15a3400700 1 -- 172.21.15.78:0/4150440396 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f159c00cd30 con 0x7f15ac100d40 2024-03-14T09:53:21.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.877+0000 7f15b1a00700 1 -- 172.21.15.78:0/4150440396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a4009c40 con 0x7f15ac100d40 2024-03-14T09:53:21.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.877+0000 7f15a3400700 1 -- 172.21.15.78:0/4150440396 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c0186c0 con 0x7f15ac100d40 2024-03-14T09:53:21.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.877+0000 7f15b1a00700 1 -- 172.21.15.78:0/4150440396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15980095f0 con 0x7f15ac100d40 2024-03-14T09:53:21.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.880+0000 7f15a3400700 1 -- 172.21.15.78:0/4150440396 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f159c018820 con 0x7f15ac100d40 2024-03-14T09:53:21.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.881+0000 7f15c55aab80 1 Processor -- start 2024-03-14T09:53:21.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.881+0000 7f15a3400700 1 --2- 172.21.15.78:0/4150440396 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f15900a0ba0 0x7f15900a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.881+0000 7f15c55aab80 1 -- start start 2024-03-14T09:53:21.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.881+0000 7f15a3400700 1 -- 172.21.15.78:0/4150440396 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f159c014070 con 0x7f15ac100d40 2024-03-14T09:53:21.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.881+0000 7f15ab400700 1 --2- 172.21.15.78:0/4150440396 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f15900a0ba0 0x7f15900a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.881+0000 7f15c55aab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804cf7a0 0x55c4804cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.881+0000 7f15c55aab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804d0100 0x55c4804d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.881+0000 7f15c55aab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 0x55c4804d1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.881+0000 7f15c55aab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c480428940 con 0x55c4804d0f90 2024-03-14T09:53:21.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.881+0000 7f15c55aab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c480426300 con 0x55c4804d0100 2024-03-14T09:53:21.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.882+0000 7f15c55aab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c480421bf0 con 0x55c4804cf7a0 2024-03-14T09:53:21.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.882+0000 7f15ab400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 0x55c4804d1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.882+0000 7f15ab400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 0x55c4804d1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41820/0 (socket says 172.21.15.78:41820) 2024-03-14T09:53:21.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.882+0000 7f15ab400700 1 -- 172.21.15.78:0/3290873238 learned_addr learned my addr 172.21.15.78:0/3290873238 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:21.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.882+0000 7f15aa000700 1 --2- 172.21.15.78:0/3290873238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804d0100 0x55c4804d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.882+0000 7f15aaa00700 1 --2- 172.21.15.78:0/3290873238 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804cf7a0 0x55c4804cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.882+0000 7f15aa000700 1 -- 172.21.15.78:0/3290873238 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804cf7a0 msgr2=0x55c4804cfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.882+0000 7f15aa000700 1 --2- 172.21.15.78:0/3290873238 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804cf7a0 0x55c4804cfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.882+0000 7f15ab400700 1 --2- 172.21.15.78:0/4150440396 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f15900a0ba0 0x7f15900a3060 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f15a40079b0 tx=0x7f15a4007d70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.882+0000 7f15aa000700 1 -- 172.21.15.78:0/3290873238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 msgr2=0x55c4804d1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.882+0000 7f15aa000700 1 --2- 172.21.15.78:0/3290873238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 0x55c4804d1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.882+0000 7f15aa000700 1 -- 172.21.15.78:0/3290873238 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4802a3360 con 0x55c4804d0100 2024-03-14T09:53:21.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.882+0000 7f15ab400700 1 --2- 172.21.15.78:0/3290873238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 0x55c4804d1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:21.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.883+0000 7f15aa000700 1 --2- 172.21.15.78:0/3290873238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804d0100 0x55c4804d0a00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1598009fd0 tx=0x7f1598005c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4e68ff48653250c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.883+0000 7f1597e00700 1 -- 172.21.15.78:0/3290873238 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159800ba10 con 0x55c4804d0100 2024-03-14T09:53:21.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.883+0000 7f1597e00700 1 -- 172.21.15.78:0/3290873238 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f159801e040 con 0x55c4804d0100 2024-03-14T09:53:21.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.883+0000 7f1597e00700 1 -- 172.21.15.78:0/3290873238 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1598003a30 con 0x55c4804d0100 2024-03-14T09:53:21.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.883+0000 7f15c55aab80 1 -- 172.21.15.78:0/3290873238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804d0100 msgr2=0x55c4804d0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.883+0000 7f15c55aab80 1 --2- 172.21.15.78:0/3290873238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804d0100 0x55c4804d0a00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1598009fd0 tx=0x7f1598005c30 comp rx=0 tx=0).stop 2024-03-14T09:53:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.884+0000 7f15c55aab80 1 -- 172.21.15.78:0/3290873238 shutdown_connections 2024-03-14T09:53:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.884+0000 7f15c55aab80 1 --2- 172.21.15.78:0/3290873238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 0x55c4804d1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.884+0000 7f15c55aab80 1 --2- 172.21.15.78:0/3290873238 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804cf7a0 0x55c4804cfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.884+0000 7f15c55aab80 1 --2- 172.21.15.78:0/3290873238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804d0100 0x55c4804d0a00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.884+0000 7f15c55aab80 1 -- 172.21.15.78:0/3290873238 >> 172.21.15.78:0/3290873238 conn(0x55c4804451d0 msgr2=0x55c4804ca580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.884+0000 7f15c55aab80 1 -- 172.21.15.78:0/3290873238 shutdown_connections 2024-03-14T09:53:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.884+0000 7f15c55aab80 1 -- 172.21.15.78:0/3290873238 wait complete. 2024-03-14T09:53:21.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.884+0000 7f15c55aab80 1 -- start start 2024-03-14T09:53:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.885+0000 7f15c55aab80 1 Processor -- start 2024-03-14T09:53:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.885+0000 7f15c55aab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804cf7a0 0x55c4804d9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.885+0000 7f15c55aab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 0x55c4804d9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.885+0000 7f15c55aab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 0x55c4804e0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.885+0000 7f15c55aab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4803bdd30 con 0x55c4804d0f90 2024-03-14T09:53:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.885+0000 7f15c55aab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4803bd9d0 con 0x55c4804cf7a0 2024-03-14T09:53:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.885+0000 7f15c55aab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4802a5650 con 0x55c4804d0100 2024-03-14T09:53:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.885+0000 7f15c55aab80 10 client.0 _ll_register_callbacks cb 0x55c480229350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:53:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.886+0000 7f15aa000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 0x55c4804d9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.886+0000 7f15aa000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 0x55c4804d9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:39636/0 (socket says 172.21.15.78:39636) 2024-03-14T09:53:21.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.886+0000 7f15aa000700 1 -- 172.21.15.78:0/2174896308 learned_addr learned my addr 172.21.15.78:0/2174896308 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:21.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.886+0000 7f15aaa00700 1 --2- 172.21.15.78:0/2174896308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804cf7a0 0x55c4804d9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.886+0000 7f15ab400700 1 --2- 172.21.15.78:0/2174896308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 0x55c4804e0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.886+0000 7f15aa000700 1 -- 172.21.15.78:0/2174896308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804cf7a0 msgr2=0x55c4804d9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.886+0000 7f15aa000700 1 --2- 172.21.15.78:0/2174896308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804cf7a0 0x55c4804d9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.886+0000 7f15aa000700 1 -- 172.21.15.78:0/2174896308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 msgr2=0x55c4804e0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.886+0000 7f15aa000700 1 --2- 172.21.15.78:0/2174896308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 0x55c4804e0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.886+0000 7f15aa000700 1 -- 172.21.15.78:0/2174896308 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a40177e0 con 0x55c4804d0100 2024-03-14T09:53:21.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.886+0000 7f15aa000700 1 --2- 172.21.15.78:0/2174896308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 0x55c4804d9b80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1598025040 tx=0x7f1598005ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60ee5c737e377f8f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.886+0000 7f1594c00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:53:21.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.886+0000 7f1594c00700 1 -- 172.21.15.78:0/2174896308 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1598005640 con 0x55c4804d0100 2024-03-14T09:53:21.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.886+0000 7f15c55aab80 1 -- 172.21.15.78:0/2174896308 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c4804c26d0 con 0x55c4804d0100 2024-03-14T09:53:21.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.887+0000 7f1594c00700 1 -- 172.21.15.78:0/2174896308 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f15980037d0 con 0x55c4804d0100 2024-03-14T09:53:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.887+0000 7f1594c00700 1 -- 172.21.15.78:0/2174896308 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1598003990 con 0x55c4804d0100 2024-03-14T09:53:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.887+0000 7f15c55aab80 10 client.25168 fetch_fsmap learned FSMap version 24 2024-03-14T09:53:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.887+0000 7f15c55aab80 1 -- 172.21.15.78:0/2174896308 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7f15ac109ad0 con 0x55c4804d0100 2024-03-14T09:53:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.887+0000 7f1594c00700 1 -- 172.21.15.78:0/2174896308 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c4804c26d0 con 0x55c4804d0100 2024-03-14T09:53:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.887+0000 7f1594c00700 1 -- 172.21.15.78:0/2174896308 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f159802b420 con 0x55c4804d0100 2024-03-14T09:53:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.887+0000 7f15c55aab80 10 client.25168 fetch_fsmap finished waiting for FSMap version 24 2024-03-14T09:53:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.887+0000 7f15c55aab80 10 client.25168 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-14T09:53:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.887+0000 7f15c55aab80 20 client.25168 populate_metadata read hostname 'smithi078' 2024-03-14T09:53:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.887+0000 7f15c55aab80 1 --2- 172.21.15.78:0/2174896308 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c4804e43a0 0x55c4804e9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.887+0000 7f15c55aab80 4 client.25168 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:53:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.887+0000 7f15c55aab80 1 -- 172.21.15.78:0/2174896308 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c48041a6b0 con 0x55c4804e43a0 2024-03-14T09:53:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.888+0000 7f15aaa00700 1 --2- 172.21.15.78:0/2174896308 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c4804e43a0 0x55c4804e9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.888+0000 7f1594c00700 10 client.25168 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:53:21.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.888+0000 7f15aaa00700 1 --2- 172.21.15.78:0/2174896308 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c4804e43a0 0x55c4804e9680 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7fce5373868c486 server_cookie=1d550b117f3f874c in_seq=0 out_seq=0 2024-03-14T09:53:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.889+0000 7f1594c00700 1 -- 172.21.15.78:0/2174896308 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55c48041a6b0 con 0x55c4804e43a0 2024-03-14T09:53:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.889+0000 7f1594c00700 10 client.25168 handle_command_reply: tid=0 2024-03-14T09:53:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.889+0000 7f15c55aab80 1 client.25168 shutdown 2024-03-14T09:53:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.889+0000 7f15c55aab80 1 -- 172.21.15.78:0/2174896308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 msgr2=0x55c4804d9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.889+0000 7f15c55aab80 1 --2- 172.21.15.78:0/2174896308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 0x55c4804d9b80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1598025040 tx=0x7f1598005ee0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.890+0000 7f15c55aab80 1 -- 172.21.15.78:0/2174896308 shutdown_connections 2024-03-14T09:53:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.890+0000 7f15c55aab80 1 --2- 172.21.15.78:0/2174896308 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c4804e43a0 0x55c4804e9680 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.890+0000 7f15c55aab80 1 --2- 172.21.15.78:0/2174896308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804cf7a0 0x55c4804d9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.890+0000 7f1594c00700 0 client.25168 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:53:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.890+0000 7f15c55aab80 1 --2- 172.21.15.78:0/2174896308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 0x55c4804e0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.890+0000 7f15c55aab80 1 --2- 172.21.15.78:0/2174896308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 0x55c4804d9b80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.890+0000 7f15c55aab80 1 -- 172.21.15.78:0/2174896308 >> 172.21.15.78:0/2174896308 conn(0x55c4804451d0 msgr2=0x55c4804c9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.890+0000 7f15c55aab80 1 -- 172.21.15.78:0/2174896308 shutdown_connections 2024-03-14T09:53:21.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.890+0000 7f15c55aab80 1 -- 172.21.15.78:0/2174896308 wait complete. 2024-03-14T09:53:21.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.890+0000 7f15c55aab80 20 client.25168 trim_cache size 0 max 16384 2024-03-14T09:53:21.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.907+0000 7f15c55aab80 1 Processor -- start 2024-03-14T09:53:21.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.907+0000 7f15c55aab80 1 -- start start 2024-03-14T09:53:21.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.908+0000 7f15c55aab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804cf7a0 0x55c4804dead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.908+0000 7f15c55aab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 0x55c4804d1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.908+0000 7f15c55aab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804d0f90 0x55c4804d74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.908+0000 7f15c55aab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c48041a6b0 con 0x55c4804cf7a0 2024-03-14T09:53:21.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.908+0000 7f15c55aab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4803dfad0 con 0x55c4804d0f90 2024-03-14T09:53:21.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.908+0000 7f15c55aab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c48041a990 con 0x55c4804d0100 2024-03-14T09:53:21.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.908+0000 7f15aaa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804cf7a0 0x55c4804dead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.908+0000 7f15aaa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804cf7a0 0x55c4804dead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41840/0 (socket says 172.21.15.78:41840) 2024-03-14T09:53:21.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.908+0000 7f15ab400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804d0f90 0x55c4804d74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.908+0000 7f15aaa00700 1 -- 172.21.15.78:0/2603491049 learned_addr learned my addr 172.21.15.78:0/2603491049 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:21.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.908+0000 7f15aa000700 1 --2- 172.21.15.78:0/2603491049 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 0x55c4804d1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.908+0000 7f15aaa00700 1 -- 172.21.15.78:0/2603491049 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 msgr2=0x55c4804d1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.908+0000 7f15aaa00700 1 --2- 172.21.15.78:0/2603491049 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 0x55c4804d1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.908+0000 7f15aaa00700 1 -- 172.21.15.78:0/2603491049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804d0f90 msgr2=0x55c4804d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.908+0000 7f15aaa00700 1 --2- 172.21.15.78:0/2603491049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804d0f90 0x55c4804d74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.908+0000 7f15aaa00700 1 -- 172.21.15.78:0/2603491049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c48024fb10 con 0x55c4804cf7a0 2024-03-14T09:53:21.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.909+0000 7f15aaa00700 1 --2- 172.21.15.78:0/2603491049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804cf7a0 0x55c4804dead0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f159c0998d0 tx=0x7f159c023040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73fa36b010519e15 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.909+0000 7f15a0c00700 1 -- 172.21.15.78:0/2603491049 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c0a2050 con 0x55c4804cf7a0 2024-03-14T09:53:21.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.909+0000 7f15a0c00700 1 -- 172.21.15.78:0/2603491049 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f159c097f70 con 0x55c4804cf7a0 2024-03-14T09:53:21.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.909+0000 7f15a0c00700 1 -- 172.21.15.78:0/2603491049 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c0990c0 con 0x55c4804cf7a0 2024-03-14T09:53:21.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.909+0000 7f15c55aab80 1 -- 172.21.15.78:0/2603491049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804cf7a0 msgr2=0x55c4804dead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.909+0000 7f15c55aab80 1 --2- 172.21.15.78:0/2603491049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804cf7a0 0x55c4804dead0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f159c0998d0 tx=0x7f159c023040 comp rx=0 tx=0).stop 2024-03-14T09:53:21.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.909+0000 7f15c55aab80 1 -- 172.21.15.78:0/2603491049 shutdown_connections 2024-03-14T09:53:21.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.909+0000 7f15c55aab80 1 --2- 172.21.15.78:0/2603491049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804d0f90 0x55c4804d74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.909+0000 7f15c55aab80 1 --2- 172.21.15.78:0/2603491049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804cf7a0 0x55c4804dead0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.909+0000 7f15c55aab80 1 --2- 172.21.15.78:0/2603491049 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 0x55c4804d1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.909+0000 7f15c55aab80 1 -- 172.21.15.78:0/2603491049 >> 172.21.15.78:0/2603491049 conn(0x55c4804451d0 msgr2=0x55c4804ca550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.909+0000 7f15c55aab80 1 -- 172.21.15.78:0/2603491049 shutdown_connections 2024-03-14T09:53:21.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.909+0000 7f15c55aab80 1 -- 172.21.15.78:0/2603491049 wait complete. 2024-03-14T09:53:21.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.910+0000 7f15c55aab80 1 -- start start 2024-03-14T09:53:21.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.910+0000 7f15c55aab80 1 Processor -- start 2024-03-14T09:53:21.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15c55aab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804cf7a0 0x55c4804dead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15c55aab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 0x55c4804d6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15c55aab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 0x55c4804d4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15c55aab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4804d54c0 con 0x55c4804d0f90 2024-03-14T09:53:21.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15c55aab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4804d5630 con 0x55c4804cf7a0 2024-03-14T09:53:21.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15c55aab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4804d57f0 con 0x55c4804d0100 2024-03-14T09:53:21.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15c55aab80 10 client.0 _ll_register_callbacks cb 0x7f15a4007ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:53:21.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15aaa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804cf7a0 0x55c4804dead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15aaa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804cf7a0 0x55c4804dead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48630/0 (socket says 172.21.15.78:48630) 2024-03-14T09:53:21.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15aaa00700 1 -- 172.21.15.78:0/1813047807 learned_addr learned my addr 172.21.15.78:0/1813047807 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:21.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15ab400700 1 --2- 172.21.15.78:0/1813047807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 0x55c4804d4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15aa000700 1 --2- 172.21.15.78:0/1813047807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 0x55c4804d6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15aaa00700 1 -- 172.21.15.78:0/1813047807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 msgr2=0x55c4804d6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15aaa00700 1 --2- 172.21.15.78:0/1813047807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 0x55c4804d6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15aaa00700 1 -- 172.21.15.78:0/1813047807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 msgr2=0x55c4804d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:21.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15aaa00700 1 --2- 172.21.15.78:0/1813047807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 0x55c4804d4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.911+0000 7f15aaa00700 1 -- 172.21.15.78:0/1813047807 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15980095f0 con 0x55c4804cf7a0 2024-03-14T09:53:21.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.912+0000 7f15aaa00700 1 --2- 172.21.15.78:0/1813047807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804cf7a0 0x55c4804dead0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f159c005950 tx=0x7f159c010cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95a2ae5c80422145 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:21.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.912+0000 7f1594c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:53:21.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.912+0000 7f1594c00700 1 -- 172.21.15.78:0/1813047807 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c018cd0 con 0x55c4804cf7a0 2024-03-14T09:53:21.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.912+0000 7f1594c00700 1 -- 172.21.15.78:0/1813047807 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f159c0b2070 con 0x55c4804cf7a0 2024-03-14T09:53:21.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.912+0000 7f15c55aab80 1 -- 172.21.15.78:0/1813047807 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f159801e040 con 0x55c4804cf7a0 2024-03-14T09:53:21.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.912+0000 7f1594c00700 1 -- 172.21.15.78:0/1813047807 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f159c0990c0 con 0x55c4804cf7a0 2024-03-14T09:53:21.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.912+0000 7f15c55aab80 10 client.25172 fetch_fsmap learned FSMap version 24 2024-03-14T09:53:21.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.912+0000 7f15c55aab80 1 -- 172.21.15.78:0/1813047807 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55c4803e7e50 con 0x55c4804cf7a0 2024-03-14T09:53:21.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.912+0000 7f1594c00700 1 -- 172.21.15.78:0/1813047807 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f159801e040 con 0x55c4804cf7a0 2024-03-14T09:53:21.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.912+0000 7f1594c00700 1 -- 172.21.15.78:0/1813047807 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f159c080070 con 0x55c4804cf7a0 2024-03-14T09:53:21.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.912+0000 7f15c55aab80 10 client.25172 fetch_fsmap finished waiting for FSMap version 24 2024-03-14T09:53:21.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.913+0000 7f15c55aab80 10 client.25172 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-14T09:53:21.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.913+0000 7f15c55aab80 20 client.25172 populate_metadata read hostname 'smithi078' 2024-03-14T09:53:21.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.913+0000 7f15c55aab80 1 --2- 172.21.15.78:0/1813047807 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c48053f140 0x55c48053bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:21.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.913+0000 7f15c55aab80 4 client.25172 mds_command: new command op to 24572 tid=0[{"prefix": "status"}] 2024-03-14T09:53:21.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.913+0000 7f15c55aab80 1 -- 172.21.15.78:0/1813047807 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c480270ca0 con 0x55c48053f140 2024-03-14T09:53:21.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.913+0000 7f15aa000700 1 --2- 172.21.15.78:0/1813047807 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c48053f140 0x55c48053bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:21.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.913+0000 7f1594c00700 10 client.25172 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:53:21.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.913+0000 7f15aa000700 1 --2- 172.21.15.78:0/1813047807 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c48053f140 0x55c48053bbf0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41bd51f961ad6701 server_cookie=8e59c10113f703 in_seq=0 out_seq=0 2024-03-14T09:53:21.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.913+0000 7f1594c00700 1 -- 172.21.15.78:0/1813047807 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55c480270ca0 con 0x55c48053f140 2024-03-14T09:53:21.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.913+0000 7f1594c00700 10 client.25172 handle_command_reply: tid=0 2024-03-14T09:53:21.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.914+0000 7f15c55aab80 1 client.25172 shutdown 2024-03-14T09:53:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.914+0000 7f15c55aab80 1 -- 172.21.15.78:0/1813047807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804cf7a0 msgr2=0x55c4804dead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.914+0000 7f15c55aab80 1 --2- 172.21.15.78:0/1813047807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804cf7a0 0x55c4804dead0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f159c005950 tx=0x7f159c010cf0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.914+0000 7f15c55aab80 1 -- 172.21.15.78:0/1813047807 shutdown_connections 2024-03-14T09:53:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.914+0000 7f15c55aab80 1 --2- 172.21.15.78:0/1813047807 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c48053f140 0x55c48053bbf0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.914+0000 7f15c55aab80 1 --2- 172.21.15.78:0/1813047807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c4804cf7a0 0x55c4804dead0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.914+0000 7f15c55aab80 1 --2- 172.21.15.78:0/1813047807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c4804d0f90 0x55c4804d4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.914+0000 7f1594c00700 0 client.25172 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:53:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.914+0000 7f15c55aab80 1 --2- 172.21.15.78:0/1813047807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c4804d0100 0x55c4804d6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.914+0000 7f15c55aab80 1 -- 172.21.15.78:0/1813047807 >> 172.21.15.78:0/1813047807 conn(0x55c4804451d0 msgr2=0x55c4804ed6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.914+0000 7f15c55aab80 1 -- 172.21.15.78:0/1813047807 shutdown_connections 2024-03-14T09:53:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.914+0000 7f15c55aab80 1 -- 172.21.15.78:0/1813047807 wait complete. 2024-03-14T09:53:21.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.914+0000 7f15c55aab80 20 client.25172 trim_cache size 0 max 16384 2024-03-14T09:53:21.919 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:53:21.920 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "83662c4e-e1e7-11ee-95c9-87774f69a715", 2024-03-14T09:53:21.920 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-03-14T09:53:21.920 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24572, 2024-03-14T09:53:21.920 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-14T09:53:21.920 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-14T09:53:21.920 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-14T09:53:21.920 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 76.202515980000001, 2024-03-14T09:53:21.920 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2024-03-14T09:53:21.920 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 85, 2024-03-14T09:53:21.920 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 83, 2024-03-14T09:53:21.920 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 83.179627081000007 2024-03-14T09:53:21.920 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:53:21.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.914+0000 7f15b1a00700 1 -- 172.21.15.78:0/4150440396 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f15900a0ba0 msgr2=0x7f15900a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.915+0000 7f15b1a00700 1 --2- 172.21.15.78:0/4150440396 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f15900a0ba0 0x7f15900a3060 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f15a40079b0 tx=0x7f15a4007d70 comp rx=0 tx=0).stop 2024-03-14T09:53:21.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.915+0000 7f15b1a00700 1 -- 172.21.15.78:0/4150440396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15ac100d40 msgr2=0x7f15ac197130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:21.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.915+0000 7f15b1a00700 1 --2- 172.21.15.78:0/4150440396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15ac100d40 0x7f15ac197130 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f159c00ea50 tx=0x7f159c00ed60 comp rx=0 tx=0).stop 2024-03-14T09:53:21.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.915+0000 7f15b1a00700 1 -- 172.21.15.78:0/4150440396 shutdown_connections 2024-03-14T09:53:21.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.915+0000 7f15b1a00700 1 --2- 172.21.15.78:0/4150440396 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15ac1028d0 0x7f15ac19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.915+0000 7f15b1a00700 1 --2- 172.21.15.78:0/4150440396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15ac100d40 0x7f15ac197130 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.915+0000 7f15b1a00700 1 --2- 172.21.15.78:0/4150440396 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f15900a0ba0 0x7f15900a3060 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.915+0000 7f15b1a00700 1 --2- 172.21.15.78:0/4150440396 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15ac101f30 0x7f15ac197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:21.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.915+0000 7f15b1a00700 1 -- 172.21.15.78:0/4150440396 >> 172.21.15.78:0/4150440396 conn(0x7f15ac0fc2b0 msgr2=0x7f15ac105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:21.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.915+0000 7f15b1a00700 1 -- 172.21.15.78:0/4150440396 shutdown_connections 2024-03-14T09:53:21.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:21.915+0000 7f15b1a00700 1 -- 172.21.15.78:0/4150440396 wait complete. 2024-03-14T09:53:21.933 INFO:tasks.cephadm:Setup complete, yielding 2024-03-14T09:53:21.934 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-14T09:53:21.946 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-03-14T09:53:21.947 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-14T09:53:21.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:21 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/4167964939' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:21.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:21 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1312622874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:22.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:21 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/4167964939' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:22.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:21 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1312622874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:22.304 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:53:22.793 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:22 smithi078 ceph-mon[38143]: pgmap v275: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T09:53:22.998 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:22 smithi152 ceph-mon[48279]: pgmap v275: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T09:53:23.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:22 smithi132 ceph-mon[47959]: pgmap v275: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T09:53:23.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.594+0000 7f25a9200700 1 -- 172.21.15.78:0/1646043428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a4108cb0 msgr2=0x7f25a410b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:23.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.594+0000 7f25a9200700 1 --2- 172.21.15.78:0/1646043428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a4108cb0 0x7f25a410b140 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f259c00b0a0 tx=0x7f259c00b3b0 comp rx=0 tx=0).stop 2024-03-14T09:53:23.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.594+0000 7f25a9200700 1 -- 172.21.15.78:0/1646043428 shutdown_connections 2024-03-14T09:53:23.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.594+0000 7f25a9200700 1 --2- 172.21.15.78:0/1646043428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a4108cb0 0x7f25a410b140 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:23.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.594+0000 7f25a9200700 1 --2- 172.21.15.78:0/1646043428 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25a4074de0 0x7f25a4073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:23.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.594+0000 7f25a9200700 1 --2- 172.21.15.78:0/1646043428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25a4073810 0x7f25a4073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:23.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.594+0000 7f25a9200700 1 -- 172.21.15.78:0/1646043428 >> 172.21.15.78:0/1646043428 conn(0x7f25a40fc380 msgr2=0x7f25a40fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:23.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.594+0000 7f25a9200700 1 -- 172.21.15.78:0/1646043428 shutdown_connections 2024-03-14T09:53:23.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.594+0000 7f25a9200700 1 -- 172.21.15.78:0/1646043428 wait complete. 2024-03-14T09:53:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.595+0000 7f25a9200700 1 Processor -- start 2024-03-14T09:53:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.595+0000 7f25a9200700 1 -- start start 2024-03-14T09:53:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.595+0000 7f25a9200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25a4073810 0x7f25a41994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.595+0000 7f25a9200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25a4074de0 0x7f25a4199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.596+0000 7f25a9200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a4108cb0 0x7f25a41a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.596+0000 7f25a9200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a41a0fd0 con 0x7f25a4108cb0 2024-03-14T09:53:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.596+0000 7f25a9200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a41a1110 con 0x7f25a4073810 2024-03-14T09:53:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.596+0000 7f25a9200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a41a12a0 con 0x7f25a4074de0 2024-03-14T09:53:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.596+0000 7f25a2800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a4108cb0 0x7f25a41a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.596+0000 7f25a1400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25a4074de0 0x7f25a4199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:23.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.596+0000 7f25a1e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25a4073810 0x7f25a41994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.596+0000 7f25a1400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25a4074de0 0x7f25a4199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:39680/0 (socket says 172.21.15.78:39680) 2024-03-14T09:53:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.596+0000 7f25a1400700 1 -- 172.21.15.78:0/11492595 learned_addr learned my addr 172.21.15.78:0/11492595 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.596+0000 7f25a2800700 1 -- 172.21.15.78:0/11492595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25a4074de0 msgr2=0x7f25a4199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.596+0000 7f25a2800700 1 --2- 172.21.15.78:0/11492595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25a4074de0 0x7f25a4199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.596+0000 7f25a2800700 1 -- 172.21.15.78:0/11492595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25a4073810 msgr2=0x7f25a41994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.596+0000 7f25a2800700 1 --2- 172.21.15.78:0/11492595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25a4073810 0x7f25a41994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.596+0000 7f25a2800700 1 -- 172.21.15.78:0/11492595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c009d00 con 0x7f25a4108cb0 2024-03-14T09:53:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.596+0000 7f25a1e00700 1 --2- 172.21.15.78:0/11492595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25a4073810 0x7f25a41994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:23.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.597+0000 7f25a2800700 1 --2- 172.21.15.78:0/11492595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a4108cb0 0x7f25a41a0a00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f259c00b070 tx=0x7f259c007930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:23.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.597+0000 7f259aa00700 1 -- 172.21.15.78:0/11492595 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c00e040 con 0x7f25a4108cb0 2024-03-14T09:53:23.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.597+0000 7f259aa00700 1 -- 172.21.15.78:0/11492595 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f259c00be40 con 0x7f25a4108cb0 2024-03-14T09:53:23.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.597+0000 7f25a9200700 1 -- 172.21.15.78:0/11492595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a41a1520 con 0x7f25a4108cb0 2024-03-14T09:53:23.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.597+0000 7f259aa00700 1 -- 172.21.15.78:0/11492595 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c01b460 con 0x7f25a4108cb0 2024-03-14T09:53:23.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.597+0000 7f25a9200700 1 -- 172.21.15.78:0/11492595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a41a18e0 con 0x7f25a4108cb0 2024-03-14T09:53:23.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.598+0000 7f25a9200700 1 -- 172.21.15.78:0/11492595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a4107a70 con 0x7f25a4108cb0 2024-03-14T09:53:23.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.600+0000 7f259aa00700 1 -- 172.21.15.78:0/11492595 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f259c01d470 con 0x7f25a4108cb0 2024-03-14T09:53:23.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.601+0000 7f259aa00700 1 --2- 172.21.15.78:0/11492595 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f25880a0b20 0x7f25880a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:23.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.601+0000 7f259aa00700 1 -- 172.21.15.78:0/11492595 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f259c0cd6c0 con 0x7f25a4108cb0 2024-03-14T09:53:23.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.601+0000 7f25a1e00700 1 --2- 172.21.15.78:0/11492595 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f25880a0b20 0x7f25880a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:23.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.602+0000 7f25a1e00700 1 --2- 172.21.15.78:0/11492595 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f25880a0b20 0x7f25880a2fe0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f25a40751c0 tx=0x7f2594009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:23.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.605+0000 7f259aa00700 1 -- 172.21.15.78:0/11492595 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f259c096c50 con 0x7f25a4108cb0 2024-03-14T09:53:23.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.830+0000 7f25a9200700 1 -- 172.21.15.78:0/11492595 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f25a40611f0 con 0x7f25880a0b20 2024-03-14T09:53:23.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.831+0000 7f259aa00700 1 -- 172.21.15.78:0/11492595 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f25a40611f0 con 0x7f25880a0b20 2024-03-14T09:53:23.830 INFO:teuthology.orchestra.run.smithi078.stdout:Backend: cephadm 2024-03-14T09:53:23.831 INFO:teuthology.orchestra.run.smithi078.stdout:Available: Yes 2024-03-14T09:53:23.831 INFO:teuthology.orchestra.run.smithi078.stdout:Paused: No 2024-03-14T09:53:23.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.833+0000 7f25a9200700 1 -- 172.21.15.78:0/11492595 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f25880a0b20 msgr2=0x7f25880a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:23.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.833+0000 7f25a9200700 1 --2- 172.21.15.78:0/11492595 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f25880a0b20 0x7f25880a2fe0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f25a40751c0 tx=0x7f2594009260 comp rx=0 tx=0).stop 2024-03-14T09:53:23.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.833+0000 7f25a9200700 1 -- 172.21.15.78:0/11492595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a4108cb0 msgr2=0x7f25a41a0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:23.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.833+0000 7f25a9200700 1 --2- 172.21.15.78:0/11492595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a4108cb0 0x7f25a41a0a00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f259c00b070 tx=0x7f259c007930 comp rx=0 tx=0).stop 2024-03-14T09:53:23.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.833+0000 7f25a9200700 1 -- 172.21.15.78:0/11492595 shutdown_connections 2024-03-14T09:53:23.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.833+0000 7f25a9200700 1 --2- 172.21.15.78:0/11492595 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f25880a0b20 0x7f25880a2fe0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:23.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.833+0000 7f25a9200700 1 --2- 172.21.15.78:0/11492595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25a4073810 0x7f25a41994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:23.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.833+0000 7f25a9200700 1 --2- 172.21.15.78:0/11492595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a4108cb0 0x7f25a41a0a00 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:23.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.833+0000 7f25a9200700 1 --2- 172.21.15.78:0/11492595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25a4074de0 0x7f25a4199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:23.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.833+0000 7f25a9200700 1 -- 172.21.15.78:0/11492595 >> 172.21.15.78:0/11492595 conn(0x7f25a40fc380 msgr2=0x7f25a4104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:23.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.833+0000 7f25a9200700 1 -- 172.21.15.78:0/11492595 shutdown_connections 2024-03-14T09:53:23.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:23.834+0000 7f25a9200700 1 -- 172.21.15.78:0/11492595 wait complete. 2024-03-14T09:53:23.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:23 smithi152 ceph-mon[48279]: pgmap v276: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:24.017 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:23 smithi078 ceph-mon[38143]: pgmap v276: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:24.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:23 smithi132 ceph-mon[47959]: pgmap v276: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:24.199 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-14T09:53:24.562 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:53:24.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:24 smithi152 ceph-mon[48279]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:25.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:24 smithi132 ceph-mon[47959]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:25.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:24 smithi078 ceph-mon[38143]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:25.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.747+0000 7fc9d7200700 1 -- 172.21.15.78:0/151307610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9d8103f80 msgr2=0x7fc9d8104400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:25.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.747+0000 7fc9d7200700 1 --2- 172.21.15.78:0/151307610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9d8103f80 0x7fc9d8104400 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc9c40098f0 tx=0x7fc9c4009c00 comp rx=0 tx=0).stop 2024-03-14T09:53:25.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.747+0000 7fc9d7200700 1 -- 172.21.15.78:0/151307610 shutdown_connections 2024-03-14T09:53:25.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.747+0000 7fc9d7200700 1 --2- 172.21.15.78:0/151307610 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc9d8104940 0x7fc9d810b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:25.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.747+0000 7fc9d7200700 1 --2- 172.21.15.78:0/151307610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d8102e20 0x7fc9d8103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:25.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.747+0000 7fc9d7200700 1 --2- 172.21.15.78:0/151307610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9d8103f80 0x7fc9d8104400 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:25.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.747+0000 7fc9d7200700 1 -- 172.21.15.78:0/151307610 >> 172.21.15.78:0/151307610 conn(0x7fc9d80fe3c0 msgr2=0x7fc9d8100800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:25.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.747+0000 7fc9d7200700 1 -- 172.21.15.78:0/151307610 shutdown_connections 2024-03-14T09:53:25.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.747+0000 7fc9d7200700 1 -- 172.21.15.78:0/151307610 wait complete. 2024-03-14T09:53:25.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.749+0000 7fc9d7200700 1 Processor -- start 2024-03-14T09:53:25.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.749+0000 7fc9d7200700 1 -- start start 2024-03-14T09:53:25.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.749+0000 7fc9d7200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc9d8102e20 0x7fc9d8199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:25.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.749+0000 7fc9d7200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9d8103f80 0x7fc9d81996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:25.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.749+0000 7fc9d7200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d8104940 0x7fc9d81a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:25.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.749+0000 7fc9d7200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d81a0cd0 con 0x7fc9d8104940 2024-03-14T09:53:25.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.749+0000 7fc9d7200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d81a0e10 con 0x7fc9d8103f80 2024-03-14T09:53:25.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.749+0000 7fc9d7200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d81a0fa0 con 0x7fc9d8102e20 2024-03-14T09:53:25.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.750+0000 7fc9d5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9d8103f80 0x7fc9d81996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:25.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.750+0000 7fc9d5e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc9d8102e20 0x7fc9d8199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:25.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.750+0000 7fc9d5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9d8103f80 0x7fc9d81996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:32920/0 (socket says 172.21.15.78:32920) 2024-03-14T09:53:25.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.750+0000 7fc9d6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d8104940 0x7fc9d81a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:25.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.750+0000 7fc9d5400700 1 -- 172.21.15.78:0/4017895745 learned_addr learned my addr 172.21.15.78:0/4017895745 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:25.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.750+0000 7fc9d5400700 1 -- 172.21.15.78:0/4017895745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc9d8102e20 msgr2=0x7fc9d8199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:25.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.750+0000 7fc9d5400700 1 --2- 172.21.15.78:0/4017895745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc9d8102e20 0x7fc9d8199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:25.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.750+0000 7fc9d5400700 1 -- 172.21.15.78:0/4017895745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d8104940 msgr2=0x7fc9d81a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:25.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.750+0000 7fc9d5400700 1 --2- 172.21.15.78:0/4017895745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d8104940 0x7fc9d81a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:25.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.750+0000 7fc9d5400700 1 -- 172.21.15.78:0/4017895745 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c4009520 con 0x7fc9d8103f80 2024-03-14T09:53:25.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.750+0000 7fc9d5400700 1 --2- 172.21.15.78:0/4017895745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9d8103f80 0x7fc9d81996d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4005e90 tx=0x7fc9c400b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:25.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.751+0000 7fc9cea00700 1 -- 172.21.15.78:0/4017895745 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c401c070 con 0x7fc9d8103f80 2024-03-14T09:53:25.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.751+0000 7fc9d7200700 1 -- 172.21.15.78:0/4017895745 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d81a1220 con 0x7fc9d8103f80 2024-03-14T09:53:25.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.751+0000 7fc9cea00700 1 -- 172.21.15.78:0/4017895745 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9c400f7f0 con 0x7fc9d8103f80 2024-03-14T09:53:25.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.751+0000 7fc9cea00700 1 -- 172.21.15.78:0/4017895745 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c40039f0 con 0x7fc9d8103f80 2024-03-14T09:53:25.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.751+0000 7fc9d7200700 1 -- 172.21.15.78:0/4017895745 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d81a1690 con 0x7fc9d8103f80 2024-03-14T09:53:25.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.752+0000 7fc9d7200700 1 -- 172.21.15.78:0/4017895745 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d810a2f0 con 0x7fc9d8103f80 2024-03-14T09:53:25.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.754+0000 7fc9cea00700 1 -- 172.21.15.78:0/4017895745 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fc9c4020030 con 0x7fc9d8103f80 2024-03-14T09:53:25.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.755+0000 7fc9cea00700 1 --2- 172.21.15.78:0/4017895745 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc9bc0a0b20 0x7fc9bc0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:25.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.755+0000 7fc9cea00700 1 -- 172.21.15.78:0/4017895745 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc9c40cdb20 con 0x7fc9d8103f80 2024-03-14T09:53:25.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.755+0000 7fc9d5e00700 1 --2- 172.21.15.78:0/4017895745 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc9bc0a0b20 0x7fc9bc0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:25.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.756+0000 7fc9d5e00700 1 --2- 172.21.15.78:0/4017895745 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc9bc0a0b20 0x7fc9bc0a2fe0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009c60 tx=0x7fc9c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:25.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.758+0000 7fc9cea00700 1 -- 172.21.15.78:0/4017895745 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc9c40970b0 con 0x7fc9d8103f80 2024-03-14T09:53:25.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:25 smithi078 ceph-mon[38143]: pgmap v277: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-14T09:53:25.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.973+0000 7fc9d7200700 1 -- 172.21.15.78:0/4017895745 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9d80611f0 con 0x7fc9bc0a0b20 2024-03-14T09:53:25.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.986+0000 7fc9cea00700 1 -- 172.21.15.78:0/4017895745 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4154 (secure 0 0 0) 0x7fc9d80611f0 con 0x7fc9bc0a0b20 2024-03-14T09:53:25.986 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T09:53:25.986 INFO:teuthology.orchestra.run.smithi078.stdout:mds.a smithi078 running (89s) 17s ago 89s 13.7M - 18.2.2-825-g2f465319 a9e65ad305ae 7fb9b0e19bcb 2024-03-14T09:53:25.986 INFO:teuthology.orchestra.run.smithi078.stdout:mds.b smithi132 running (87s) 21s ago 87s 30.1M - 18.2.2-825-g2f465319 a9e65ad305ae 3dcddc4b5522 2024-03-14T09:53:25.986 INFO:teuthology.orchestra.run.smithi078.stdout:mds.c smithi152 running (97s) 21s ago 97s 19.9M - 18.2.2-825-g2f465319 a9e65ad305ae 7c11fe5f6016 2024-03-14T09:53:25.986 INFO:teuthology.orchestra.run.smithi078.stdout:mds.d smithi078 running (103s) 17s ago 103s 13.8M - 18.2.2-825-g2f465319 a9e65ad305ae b53b9544ba2b 2024-03-14T09:53:25.986 INFO:teuthology.orchestra.run.smithi078.stdout:mds.e smithi132 running (101s) 21s ago 101s 18.9M - 18.2.2-825-g2f465319 a9e65ad305ae df6e50ca2a6d 2024-03-14T09:53:25.986 INFO:teuthology.orchestra.run.smithi078.stdout:mds.f smithi152 running (99s) 21s ago 99s 16.4M - 18.2.2-825-g2f465319 a9e65ad305ae bad61adda90b 2024-03-14T09:53:25.986 INFO:teuthology.orchestra.run.smithi078.stdout:mds.g smithi078 running (91s) 17s ago 91s 17.4M - 18.2.2-825-g2f465319 a9e65ad305ae 90b8142a99a1 2024-03-14T09:53:25.986 INFO:teuthology.orchestra.run.smithi078.stdout:mds.h smithi132 running (96s) 21s ago 95s 20.9M - 18.2.2-825-g2f465319 a9e65ad305ae ecd951122b1d 2024-03-14T09:53:25.987 INFO:teuthology.orchestra.run.smithi078.stdout:mds.i smithi152 running (94s) 21s ago 93s 21.2M - 18.2.2-825-g2f465319 a9e65ad305ae cfd7c7cd8790 2024-03-14T09:53:25.987 INFO:teuthology.orchestra.run.smithi078.stdout:mds.j smithi078 running (105s) 17s ago 105s 13.8M - 18.2.2-825-g2f465319 a9e65ad305ae 484e71a0bc44 2024-03-14T09:53:25.987 INFO:teuthology.orchestra.run.smithi078.stdout:mds.k smithi132 running (108s) 21s ago 107s 14.1M - 18.2.2-825-g2f465319 a9e65ad305ae db03ab2e61b8 2024-03-14T09:53:25.987 INFO:teuthology.orchestra.run.smithi078.stdout:mds.l smithi152 running (109s) 21s ago 109s 14.1M - 18.2.2-825-g2f465319 a9e65ad305ae e452c2e7aa9e 2024-03-14T09:53:25.987 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi078 *:9283,8765 running (7m) 17s ago 7m 534M - 18.2.2-825-g2f465319 a9e65ad305ae c98e9884dd1c 2024-03-14T09:53:25.987 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi132 *:8443,8765 running (5m) 21s ago 5m 459M - 18.2.2-825-g2f465319 a9e65ad305ae 3e8637b9cc83 2024-03-14T09:53:25.987 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.z smithi152 *:8443,8765 running (5m) 21s ago 5m 457M - 18.2.2-825-g2f465319 a9e65ad305ae a7ed9640f3c2 2024-03-14T09:53:25.987 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (7m) 17s ago 7m 90.7M 2048M 18.2.2-825-g2f465319 a9e65ad305ae f2a8137bfb2d 2024-03-14T09:53:25.987 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi132 running (6m) 21s ago 6m 63.6M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 54f3f9d88369 2024-03-14T09:53:25.987 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi152 running (6m) 21s ago 6m 60.7M 2048M 18.2.2-825-g2f465319 a9e65ad305ae a31b518db22e 2024-03-14T09:53:25.987 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (5m) 17s ago 5m 68.6M 4317M 18.2.2-825-g2f465319 a9e65ad305ae 58a94f77014a 2024-03-14T09:53:25.987 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi132 running (5m) 21s ago 5m 78.9M 4278M 18.2.2-825-g2f465319 a9e65ad305ae 11b62293429f 2024-03-14T09:53:25.987 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi152 running (4m) 21s ago 4m 73.5M 4278M 18.2.2-825-g2f465319 a9e65ad305ae fc847fcacdb8 2024-03-14T09:53:25.987 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4m) 17s ago 4m 63.2M 4317M 18.2.2-825-g2f465319 a9e65ad305ae 79178cc0925d 2024-03-14T09:53:25.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi132 running (4m) 21s ago 4m 61.7M 4278M 18.2.2-825-g2f465319 a9e65ad305ae c00c761bdca8 2024-03-14T09:53:25.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi152 running (4m) 21s ago 3m 67.6M 4278M 18.2.2-825-g2f465319 a9e65ad305ae 150f16e206c1 2024-03-14T09:53:25.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi078 running (3m) 17s ago 3m 60.8M 4317M 18.2.2-825-g2f465319 a9e65ad305ae 822dc3f51e37 2024-03-14T09:53:25.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi132 running (3m) 21s ago 3m 64.3M 4278M 18.2.2-825-g2f465319 a9e65ad305ae f18cc1d8d107 2024-03-14T09:53:25.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.8 smithi152 running (3m) 21s ago 3m 64.5M 4278M 18.2.2-825-g2f465319 a9e65ad305ae cd9983738a15 2024-03-14T09:53:25.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.9 smithi078 running (2m) 17s ago 2m 60.7M 4317M 18.2.2-825-g2f465319 a9e65ad305ae f4c5aa4eec47 2024-03-14T09:53:25.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.10 smithi132 running (2m) 21s ago 2m 61.2M 4278M 18.2.2-825-g2f465319 a9e65ad305ae a4647901612c 2024-03-14T09:53:25.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.11 smithi152 running (2m) 21s ago 2m 55.8M 4278M 18.2.2-825-g2f465319 a9e65ad305ae cca911ff8661 2024-03-14T09:53:25.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.988+0000 7fc9d7200700 1 -- 172.21.15.78:0/4017895745 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc9bc0a0b20 msgr2=0x7fc9bc0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:25.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.988+0000 7fc9d7200700 1 --2- 172.21.15.78:0/4017895745 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc9bc0a0b20 0x7fc9bc0a2fe0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009c60 tx=0x7fc9c8009260 comp rx=0 tx=0).stop 2024-03-14T09:53:25.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.989+0000 7fc9d7200700 1 -- 172.21.15.78:0/4017895745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9d8103f80 msgr2=0x7fc9d81996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:25.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.989+0000 7fc9d7200700 1 --2- 172.21.15.78:0/4017895745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9d8103f80 0x7fc9d81996d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4005e90 tx=0x7fc9c400b730 comp rx=0 tx=0).stop 2024-03-14T09:53:25.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.989+0000 7fc9d7200700 1 -- 172.21.15.78:0/4017895745 shutdown_connections 2024-03-14T09:53:25.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.989+0000 7fc9d7200700 1 --2- 172.21.15.78:0/4017895745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d8104940 0x7fc9d81a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:25.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.989+0000 7fc9d7200700 1 --2- 172.21.15.78:0/4017895745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc9d8102e20 0x7fc9d8199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:25.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.989+0000 7fc9d7200700 1 --2- 172.21.15.78:0/4017895745 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc9bc0a0b20 0x7fc9bc0a2fe0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:25.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.989+0000 7fc9d7200700 1 --2- 172.21.15.78:0/4017895745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9d8103f80 0x7fc9d81996d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:25.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.989+0000 7fc9d7200700 1 -- 172.21.15.78:0/4017895745 >> 172.21.15.78:0/4017895745 conn(0x7fc9d80fe3c0 msgr2=0x7fc9d8107430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:25.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.989+0000 7fc9d7200700 1 -- 172.21.15.78:0/4017895745 shutdown_connections 2024-03-14T09:53:25.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:25.989+0000 7fc9d7200700 1 -- 172.21.15.78:0/4017895745 wait complete. 2024-03-14T09:53:25.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:25 smithi152 ceph-mon[48279]: pgmap v277: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-14T09:53:26.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:25 smithi132 ceph-mon[47959]: pgmap v277: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-14T09:53:26.346 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-14T09:53:26.704 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:53:26.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:26 smithi078 ceph-mon[38143]: from='client.25184 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:26.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:26 smithi152 ceph-mon[48279]: from='client.25184 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:27.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:26 smithi132 ceph-mon[47959]: from='client.25184 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:27.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.835+0000 7f9d8ba00700 1 -- 172.21.15.78:0/1874341387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d84101fe0 msgr2=0x7f9d84102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:27.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.835+0000 7f9d8ba00700 1 --2- 172.21.15.78:0/1874341387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d84101fe0 0x7f9d84102440 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9d700099c0 tx=0x7f9d70009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:27.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.836+0000 7f9d8ba00700 1 -- 172.21.15.78:0/1874341387 shutdown_connections 2024-03-14T09:53:27.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.836+0000 7f9d8ba00700 1 --2- 172.21.15.78:0/1874341387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d84100de0 0x7f9d84101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:27.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.836+0000 7f9d8ba00700 1 --2- 172.21.15.78:0/1874341387 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d84102980 0x7f9d84109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:27.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.836+0000 7f9d8ba00700 1 --2- 172.21.15.78:0/1874341387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d84101fe0 0x7f9d84102440 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:27.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.836+0000 7f9d8ba00700 1 -- 172.21.15.78:0/1874341387 >> 172.21.15.78:0/1874341387 conn(0x7f9d840fc380 msgr2=0x7f9d840fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:27.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.836+0000 7f9d8ba00700 1 -- 172.21.15.78:0/1874341387 shutdown_connections 2024-03-14T09:53:27.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.836+0000 7f9d8ba00700 1 -- 172.21.15.78:0/1874341387 wait complete. 2024-03-14T09:53:27.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.837+0000 7f9d8ba00700 1 Processor -- start 2024-03-14T09:53:27.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.837+0000 7f9d8ba00700 1 -- start start 2024-03-14T09:53:27.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.837+0000 7f9d8ba00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d84100de0 0x7f9d84199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:27.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.837+0000 7f9d8ba00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d84101fe0 0x7f9d841998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:27.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.837+0000 7f9d8ba00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d84102980 0x7f9d841a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:27.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.837+0000 7f9d8ba00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d841a0ea0 con 0x7f9d84102980 2024-03-14T09:53:27.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.837+0000 7f9d8ba00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d841a0fe0 con 0x7f9d84101fe0 2024-03-14T09:53:27.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.837+0000 7f9d8ba00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d841a1170 con 0x7f9d84100de0 2024-03-14T09:53:27.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.837+0000 7f9d89a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d84102980 0x7f9d841a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:27.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.837+0000 7f9d89a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d84102980 0x7f9d841a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49114/0 (socket says 172.21.15.78:49114) 2024-03-14T09:53:27.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.837+0000 7f9d89a00700 1 -- 172.21.15.78:0/726672420 learned_addr learned my addr 172.21.15.78:0/726672420 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:27.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.838+0000 7f9d89000700 1 --2- 172.21.15.78:0/726672420 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d84100de0 0x7f9d84199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:27.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.838+0000 7f9d83e00700 1 --2- 172.21.15.78:0/726672420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d84101fe0 0x7f9d841998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:27.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.838+0000 7f9d89a00700 1 -- 172.21.15.78:0/726672420 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d84100de0 msgr2=0x7f9d84199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:27.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.838+0000 7f9d89a00700 1 --2- 172.21.15.78:0/726672420 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d84100de0 0x7f9d84199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:27.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.838+0000 7f9d89a00700 1 -- 172.21.15.78:0/726672420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d84101fe0 msgr2=0x7f9d841998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:27.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.838+0000 7f9d89a00700 1 --2- 172.21.15.78:0/726672420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d84101fe0 0x7f9d841998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:27.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.838+0000 7f9d89a00700 1 -- 172.21.15.78:0/726672420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d700095f0 con 0x7f9d84102980 2024-03-14T09:53:27.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.838+0000 7f9d89a00700 1 --2- 172.21.15.78:0/726672420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d84102980 0x7f9d841a08d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00ef60 tx=0x7f9d7c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:27.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.838+0000 7f9d81600700 1 -- 172.21.15.78:0/726672420 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c018070 con 0x7f9d84102980 2024-03-14T09:53:27.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.838+0000 7f9d81600700 1 -- 172.21.15.78:0/726672420 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d7c00f040 con 0x7f9d84102980 2024-03-14T09:53:27.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.838+0000 7f9d8ba00700 1 -- 172.21.15.78:0/726672420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d841a13f0 con 0x7f9d84102980 2024-03-14T09:53:27.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.838+0000 7f9d81600700 1 -- 172.21.15.78:0/726672420 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c00ccc0 con 0x7f9d84102980 2024-03-14T09:53:27.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.839+0000 7f9d8ba00700 1 -- 172.21.15.78:0/726672420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d841a18a0 con 0x7f9d84102980 2024-03-14T09:53:27.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.839+0000 7f9d8ba00700 1 -- 172.21.15.78:0/726672420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8404ea80 con 0x7f9d84102980 2024-03-14T09:53:27.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.841+0000 7f9d81600700 1 -- 172.21.15.78:0/726672420 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f9d7c003a00 con 0x7f9d84102980 2024-03-14T09:53:27.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.842+0000 7f9d81600700 1 --2- 172.21.15.78:0/726672420 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9d6c0a0b20 0x7f9d6c0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:27.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.843+0000 7f9d81600700 1 -- 172.21.15.78:0/726672420 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9d7c0cbfc0 con 0x7f9d84102980 2024-03-14T09:53:27.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.843+0000 7f9d89000700 1 --2- 172.21.15.78:0/726672420 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9d6c0a0b20 0x7f9d6c0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:27.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.843+0000 7f9d89000700 1 --2- 172.21.15.78:0/726672420 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9d6c0a0b20 0x7f9d6c0a2fe0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9d84101e40 tx=0x7f9d7800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:27.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:27.847+0000 7f9d81600700 1 -- 172.21.15.78:0/726672420 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9d7c095550 con 0x7f9d84102980 2024-03-14T09:53:27.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:27 smithi152 ceph-mon[48279]: pgmap v278: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:28.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:27 smithi132 ceph-mon[47959]: pgmap v278: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:28.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:27 smithi078 ceph-mon[38143]: pgmap v278: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:28.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:28.057+0000 7f9d8ba00700 1 -- 172.21.15.78:0/726672420 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d84074700 con 0x7f9d6c0a0b20 2024-03-14T09:53:28.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:28.063+0000 7f9d81600700 1 -- 172.21.15.78:0/726672420 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+985 (secure 0 0 0) 0x7f9d84074700 con 0x7f9d6c0a0b20 2024-03-14T09:53:28.063 INFO:teuthology.orchestra.run.smithi078.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-14T09:53:28.063 INFO:teuthology.orchestra.run.smithi078.stdout:mds.all 12/12 23s ago 113s smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi132=b;smithi132=e;smithi132=h;smithi132=k;smithi152=c;smithi152=f;smithi152=i;smithi152=l;count:12 2024-03-14T09:53:28.063 INFO:teuthology.orchestra.run.smithi078.stdout:mgr 3/3 23s ago 5m smithi078=x;smithi132=y;smithi152=z;count:3 2024-03-14T09:53:28.063 INFO:teuthology.orchestra.run.smithi078.stdout:mon 3/3 23s ago 6m smithi078:172.21.15.78=a;smithi132:172.21.15.132=b;smithi152:172.21.15.152=c;count:3 2024-03-14T09:53:28.063 INFO:teuthology.orchestra.run.smithi078.stdout:osd 12 23s ago - 2024-03-14T09:53:28.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:28.065+0000 7f9d8ba00700 1 -- 172.21.15.78:0/726672420 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9d6c0a0b20 msgr2=0x7f9d6c0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:28.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:28.065+0000 7f9d8ba00700 1 --2- 172.21.15.78:0/726672420 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9d6c0a0b20 0x7f9d6c0a2fe0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9d84101e40 tx=0x7f9d7800b410 comp rx=0 tx=0).stop 2024-03-14T09:53:28.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:28.065+0000 7f9d8ba00700 1 -- 172.21.15.78:0/726672420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d84102980 msgr2=0x7f9d841a08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:28.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:28.065+0000 7f9d8ba00700 1 --2- 172.21.15.78:0/726672420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d84102980 0x7f9d841a08d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00ef60 tx=0x7f9d7c00ef90 comp rx=0 tx=0).stop 2024-03-14T09:53:28.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:28.065+0000 7f9d8ba00700 1 -- 172.21.15.78:0/726672420 shutdown_connections 2024-03-14T09:53:28.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:28.065+0000 7f9d8ba00700 1 --2- 172.21.15.78:0/726672420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d84102980 0x7f9d841a08d0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:28.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:28.065+0000 7f9d8ba00700 1 --2- 172.21.15.78:0/726672420 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9d84100de0 0x7f9d84199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:28.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:28.065+0000 7f9d8ba00700 1 --2- 172.21.15.78:0/726672420 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9d6c0a0b20 0x7f9d6c0a2fe0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:28.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:28.065+0000 7f9d8ba00700 1 --2- 172.21.15.78:0/726672420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9d84101fe0 0x7f9d841998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:28.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:28.065+0000 7f9d8ba00700 1 -- 172.21.15.78:0/726672420 >> 172.21.15.78:0/726672420 conn(0x7f9d840fc380 msgr2=0x7f9d84072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:28.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:28.065+0000 7f9d8ba00700 1 -- 172.21.15.78:0/726672420 shutdown_connections 2024-03-14T09:53:28.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:28.066+0000 7f9d8ba00700 1 -- 172.21.15.78:0/726672420 wait complete. 2024-03-14T09:53:28.492 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-14T09:53:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:53:28.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:28 smithi152 ceph-mon[48279]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:29.000 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:28 smithi078 ceph-mon[38143]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:29.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:28 smithi132 ceph-mon[47959]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:29.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:29 smithi152 ceph-mon[48279]: pgmap v279: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-14T09:53:30.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.001+0000 7fe732e00700 1 -- 172.21.15.78:0/1322777793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe72c108cb0 msgr2=0x7fe72c10b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:30.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.001+0000 7fe732e00700 1 --2- 172.21.15.78:0/1322777793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe72c108cb0 0x7fe72c10b140 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe72400b0a0 tx=0x7fe72400b3b0 comp rx=0 tx=0).stop 2024-03-14T09:53:30.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.001+0000 7fe732e00700 1 -- 172.21.15.78:0/1322777793 shutdown_connections 2024-03-14T09:53:30.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.001+0000 7fe732e00700 1 --2- 172.21.15.78:0/1322777793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe72c108cb0 0x7fe72c10b140 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:30.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.001+0000 7fe732e00700 1 --2- 172.21.15.78:0/1322777793 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe72c074de0 0x7fe72c073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:30.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.001+0000 7fe732e00700 1 --2- 172.21.15.78:0/1322777793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe72c073810 0x7fe72c073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:30.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.001+0000 7fe732e00700 1 -- 172.21.15.78:0/1322777793 >> 172.21.15.78:0/1322777793 conn(0x7fe72c0fc380 msgr2=0x7fe72c0fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:30.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.001+0000 7fe732e00700 1 -- 172.21.15.78:0/1322777793 shutdown_connections 2024-03-14T09:53:30.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.001+0000 7fe732e00700 1 -- 172.21.15.78:0/1322777793 wait complete. 2024-03-14T09:53:30.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.002+0000 7fe732e00700 1 Processor -- start 2024-03-14T09:53:30.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.002+0000 7fe732e00700 1 -- start start 2024-03-14T09:53:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.002+0000 7fe732e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe72c073810 0x7fe72c199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.002+0000 7fe732e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe72c074de0 0x7fe72c199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.002+0000 7fe732e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe72c108cb0 0x7fe72c1a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.002+0000 7fe732e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c1a0ff0 con 0x7fe72c073810 2024-03-14T09:53:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.002+0000 7fe732e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c1a1130 con 0x7fe72c108cb0 2024-03-14T09:53:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.002+0000 7fe732e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c1a12c0 con 0x7fe72c074de0 2024-03-14T09:53:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.003+0000 7fe72be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe72c073810 0x7fe72c199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.003+0000 7fe72be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe72c073810 0x7fe72c199500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49124/0 (socket says 172.21.15.78:49124) 2024-03-14T09:53:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.003+0000 7fe72be00700 1 -- 172.21.15.78:0/1967410166 learned_addr learned my addr 172.21.15.78:0/1967410166 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.003+0000 7fe730e00700 1 --2- 172.21.15.78:0/1967410166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe72c108cb0 0x7fe72c1a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.003+0000 7fe72b400700 1 --2- 172.21.15.78:0/1967410166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe72c074de0 0x7fe72c199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.003+0000 7fe72be00700 1 -- 172.21.15.78:0/1967410166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe72c074de0 msgr2=0x7fe72c199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:30.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.003+0000 7fe72be00700 1 --2- 172.21.15.78:0/1967410166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe72c074de0 0x7fe72c199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.003+0000 7fe72be00700 1 -- 172.21.15.78:0/1967410166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe72c108cb0 msgr2=0x7fe72c1a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.003+0000 7fe72be00700 1 --2- 172.21.15.78:0/1967410166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe72c108cb0 0x7fe72c1a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.003+0000 7fe72be00700 1 -- 172.21.15.78:0/1967410166 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe724009d00 con 0x7fe72c073810 2024-03-14T09:53:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.003+0000 7fe72be00700 1 --2- 172.21.15.78:0/1967410166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe72c073810 0x7fe72c199500 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe71c00ea20 tx=0x7fe71c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:30.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.003+0000 7fe728c00700 1 -- 172.21.15.78:0/1967410166 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c009750 con 0x7fe72c073810 2024-03-14T09:53:30.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.004+0000 7fe732e00700 1 -- 172.21.15.78:0/1967410166 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72c1a15a0 con 0x7fe72c073810 2024-03-14T09:53:30.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.004+0000 7fe728c00700 1 -- 172.21.15.78:0/1967410166 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe71c00ee60 con 0x7fe72c073810 2024-03-14T09:53:30.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.004+0000 7fe728c00700 1 -- 172.21.15.78:0/1967410166 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c0054c0 con 0x7fe72c073810 2024-03-14T09:53:30.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.004+0000 7fe732e00700 1 -- 172.21.15.78:0/1967410166 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72c1a1aa0 con 0x7fe72c073810 2024-03-14T09:53:30.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.005+0000 7fe732e00700 1 -- 172.21.15.78:0/1967410166 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72c107a70 con 0x7fe72c073810 2024-03-14T09:53:30.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.006+0000 7fe728c00700 1 -- 172.21.15.78:0/1967410166 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fe71c00cbd0 con 0x7fe72c073810 2024-03-14T09:53:30.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.008+0000 7fe728c00700 1 --2- 172.21.15.78:0/1967410166 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe7100a0ba0 0x7fe7100a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:30.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.008+0000 7fe728c00700 1 -- 172.21.15.78:0/1967410166 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe71c014070 con 0x7fe72c073810 2024-03-14T09:53:30.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.008+0000 7fe72b400700 1 --2- 172.21.15.78:0/1967410166 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe7100a0ba0 0x7fe7100a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:30.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.009+0000 7fe72b400700 1 --2- 172.21.15.78:0/1967410166 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe7100a0ba0 0x7fe7100a3060 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe72c074b10 tx=0x7fe718009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:30.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.012+0000 7fe728c00700 1 -- 172.21.15.78:0/1967410166 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe71c099620 con 0x7fe72c073810 2024-03-14T09:53:30.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:29 smithi132 ceph-mon[47959]: pgmap v279: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-14T09:53:30.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:29 smithi078 ceph-mon[38143]: pgmap v279: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-14T09:53:30.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.223+0000 7fe732e00700 1 -- 172.21.15.78:0/1967410166 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe72c0611f0 con 0x7fe7100a0ba0 2024-03-14T09:53:30.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.225+0000 7fe728c00700 1 -- 172.21.15.78:0/1967410166 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fe72c0611f0 con 0x7fe7100a0ba0 2024-03-14T09:53:30.225 INFO:teuthology.orchestra.run.smithi078.stdout:HOST ADDR LABELS STATUS 2024-03-14T09:53:30.225 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 172.21.15.78 2024-03-14T09:53:30.225 INFO:teuthology.orchestra.run.smithi078.stdout:smithi132 172.21.15.132 2024-03-14T09:53:30.225 INFO:teuthology.orchestra.run.smithi078.stdout:smithi152 172.21.15.152 2024-03-14T09:53:30.225 INFO:teuthology.orchestra.run.smithi078.stdout:3 hosts in cluster 2024-03-14T09:53:30.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.227+0000 7fe732e00700 1 -- 172.21.15.78:0/1967410166 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe7100a0ba0 msgr2=0x7fe7100a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:30.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.227+0000 7fe732e00700 1 --2- 172.21.15.78:0/1967410166 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe7100a0ba0 0x7fe7100a3060 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe72c074b10 tx=0x7fe718009260 comp rx=0 tx=0).stop 2024-03-14T09:53:30.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.227+0000 7fe732e00700 1 -- 172.21.15.78:0/1967410166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe72c073810 msgr2=0x7fe72c199500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:30.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.227+0000 7fe732e00700 1 --2- 172.21.15.78:0/1967410166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe72c073810 0x7fe72c199500 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe71c00ea20 tx=0x7fe71c00ed30 comp rx=0 tx=0).stop 2024-03-14T09:53:30.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.227+0000 7fe732e00700 1 -- 172.21.15.78:0/1967410166 shutdown_connections 2024-03-14T09:53:30.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.227+0000 7fe732e00700 1 --2- 172.21.15.78:0/1967410166 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe7100a0ba0 0x7fe7100a3060 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:30.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.227+0000 7fe732e00700 1 --2- 172.21.15.78:0/1967410166 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe72c108cb0 0x7fe72c1a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:30.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.227+0000 7fe732e00700 1 --2- 172.21.15.78:0/1967410166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe72c073810 0x7fe72c199500 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:30.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.227+0000 7fe732e00700 1 --2- 172.21.15.78:0/1967410166 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe72c074de0 0x7fe72c199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:30.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.227+0000 7fe732e00700 1 -- 172.21.15.78:0/1967410166 >> 172.21.15.78:0/1967410166 conn(0x7fe72c0fc380 msgr2=0x7fe72c104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:30.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.227+0000 7fe732e00700 1 -- 172.21.15.78:0/1967410166 shutdown_connections 2024-03-14T09:53:30.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:30.227+0000 7fe732e00700 1 -- 172.21.15.78:0/1967410166 wait complete. 2024-03-14T09:53:30.585 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-14T09:53:30.948 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:53:30.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:30 smithi152 ceph-mon[48279]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:31.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:30 smithi132 ceph-mon[47959]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:31.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:30 smithi078 ceph-mon[38143]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:31.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:31 smithi152 ceph-mon[48279]: pgmap v280: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T09:53:32.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:31 smithi132 ceph-mon[47959]: pgmap v280: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T09:53:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.050+0000 7fec5a000700 1 -- 172.21.15.78:0/1866435715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec54073810 msgr2=0x7fec54073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.050+0000 7fec5a000700 1 --2- 172.21.15.78:0/1866435715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec54073810 0x7fec54073c90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fec40009990 tx=0x7fec40009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.050+0000 7fec5a000700 1 -- 172.21.15.78:0/1866435715 shutdown_connections 2024-03-14T09:53:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.050+0000 7fec5a000700 1 --2- 172.21.15.78:0/1866435715 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec54108cb0 0x7fec5410b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.050+0000 7fec5a000700 1 --2- 172.21.15.78:0/1866435715 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec54074de0 0x7fec54073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:32.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.050+0000 7fec5a000700 1 --2- 172.21.15.78:0/1866435715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec54073810 0x7fec54073c90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:32.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.050+0000 7fec5a000700 1 -- 172.21.15.78:0/1866435715 >> 172.21.15.78:0/1866435715 conn(0x7fec540fc380 msgr2=0x7fec540fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:32.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.050+0000 7fec5a000700 1 -- 172.21.15.78:0/1866435715 shutdown_connections 2024-03-14T09:53:32.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.050+0000 7fec5a000700 1 -- 172.21.15.78:0/1866435715 wait complete. 2024-03-14T09:53:32.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.051+0000 7fec5a000700 1 Processor -- start 2024-03-14T09:53:32.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.051+0000 7fec5a000700 1 -- start start 2024-03-14T09:53:32.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.052+0000 7fec5a000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec54073810 0x7fec541993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:32.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.052+0000 7fec5a000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec54074de0 0x7fec54199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:32.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.052+0000 7fec5a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec54108cb0 0x7fec541a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:32.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.052+0000 7fec5a000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec541a0f30 con 0x7fec54108cb0 2024-03-14T09:53:32.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.052+0000 7fec5a000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec541a1070 con 0x7fec54074de0 2024-03-14T09:53:32.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.052+0000 7fec5a000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec541a1200 con 0x7fec54073810 2024-03-14T09:53:32.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.052+0000 7fec53400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec54073810 0x7fec541993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:32.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.052+0000 7fec53e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec54108cb0 0x7fec541a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:32.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.052+0000 7fec53400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec54073810 0x7fec541993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:34506/0 (socket says 172.21.15.78:34506) 2024-03-14T09:53:32.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.052+0000 7fec53400700 1 -- 172.21.15.78:0/451868581 learned_addr learned my addr 172.21.15.78:0/451868581 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:32.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.052+0000 7fec52a00700 1 --2- 172.21.15.78:0/451868581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec54074de0 0x7fec54199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:32.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:31 smithi078 ceph-mon[38143]: pgmap v280: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T09:53:32.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.053+0000 7fec53e00700 1 -- 172.21.15.78:0/451868581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec54073810 msgr2=0x7fec541993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:32.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.053+0000 7fec53e00700 1 --2- 172.21.15.78:0/451868581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec54073810 0x7fec541993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:32.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.053+0000 7fec53e00700 1 -- 172.21.15.78:0/451868581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec54074de0 msgr2=0x7fec54199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:32.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.053+0000 7fec53e00700 1 --2- 172.21.15.78:0/451868581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec54074de0 0x7fec54199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:32.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.053+0000 7fec53e00700 1 -- 172.21.15.78:0/451868581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec400095f0 con 0x7fec54108cb0 2024-03-14T09:53:32.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.053+0000 7fec53e00700 1 --2- 172.21.15.78:0/451868581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec54108cb0 0x7fec541a0960 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fec4c007f40 tx=0x7fec4c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:32.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.053+0000 7fec4be00700 1 -- 172.21.15.78:0/451868581 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c009ea0 con 0x7fec54108cb0 2024-03-14T09:53:32.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.053+0000 7fec4be00700 1 -- 172.21.15.78:0/451868581 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec4c00f040 con 0x7fec54108cb0 2024-03-14T09:53:32.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.053+0000 7fec5a000700 1 -- 172.21.15.78:0/451868581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec541a14e0 con 0x7fec54108cb0 2024-03-14T09:53:32.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.053+0000 7fec4be00700 1 -- 172.21.15.78:0/451868581 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c004350 con 0x7fec54108cb0 2024-03-14T09:53:32.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.053+0000 7fec5a000700 1 -- 172.21.15.78:0/451868581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec541a1a30 con 0x7fec54108cb0 2024-03-14T09:53:32.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.054+0000 7fec5a000700 1 -- 172.21.15.78:0/451868581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec54107a70 con 0x7fec54108cb0 2024-03-14T09:53:32.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.056+0000 7fec4be00700 1 -- 172.21.15.78:0/451868581 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fec4c00db20 con 0x7fec54108cb0 2024-03-14T09:53:32.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.057+0000 7fec4be00700 1 --2- 172.21.15.78:0/451868581 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fec380a0ba0 0x7fec380a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:32.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.057+0000 7fec4be00700 1 -- 172.21.15.78:0/451868581 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fec4c0cbd40 con 0x7fec54108cb0 2024-03-14T09:53:32.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.058+0000 7fec53400700 1 --2- 172.21.15.78:0/451868581 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fec380a0ba0 0x7fec380a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:32.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.058+0000 7fec53400700 1 --2- 172.21.15.78:0/451868581 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fec380a0ba0 0x7fec380a3060 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fec4400bd10 tx=0x7fec4400b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:32.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.061+0000 7fec4be00700 1 -- 172.21.15.78:0/451868581 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fec4c095240 con 0x7fec54108cb0 2024-03-14T09:53:32.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.271+0000 7fec5a000700 1 -- 172.21.15.78:0/451868581 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fec540611f0 con 0x7fec380a0ba0 2024-03-14T09:53:32.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.273+0000 7fec4be00700 1 -- 172.21.15.78:0/451868581 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7fec540611f0 con 0x7fec380a0ba0 2024-03-14T09:53:32.273 INFO:teuthology.orchestra.run.smithi078.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-14T09:53:32.274 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400QR400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-14T09:53:32.274 INFO:teuthology.orchestra.run.smithi078.stdout:smithi132 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6232000B400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-14T09:53:32.274 INFO:teuthology.orchestra.run.smithi078.stdout:smithi152 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JT400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-14T09:53:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.276+0000 7fec5a000700 1 -- 172.21.15.78:0/451868581 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fec380a0ba0 msgr2=0x7fec380a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.276+0000 7fec5a000700 1 --2- 172.21.15.78:0/451868581 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fec380a0ba0 0x7fec380a3060 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fec4400bd10 tx=0x7fec4400b480 comp rx=0 tx=0).stop 2024-03-14T09:53:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.276+0000 7fec5a000700 1 -- 172.21.15.78:0/451868581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec54108cb0 msgr2=0x7fec541a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.276+0000 7fec5a000700 1 --2- 172.21.15.78:0/451868581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec54108cb0 0x7fec541a0960 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fec4c007f40 tx=0x7fec4c00d3b0 comp rx=0 tx=0).stop 2024-03-14T09:53:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.276+0000 7fec5a000700 1 -- 172.21.15.78:0/451868581 shutdown_connections 2024-03-14T09:53:32.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.276+0000 7fec5a000700 1 --2- 172.21.15.78:0/451868581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec54108cb0 0x7fec541a0960 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:32.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.276+0000 7fec5a000700 1 --2- 172.21.15.78:0/451868581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec54073810 0x7fec541993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:32.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.276+0000 7fec5a000700 1 --2- 172.21.15.78:0/451868581 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fec380a0ba0 0x7fec380a3060 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:32.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.276+0000 7fec5a000700 1 --2- 172.21.15.78:0/451868581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec54074de0 0x7fec54199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:32.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.276+0000 7fec5a000700 1 -- 172.21.15.78:0/451868581 >> 172.21.15.78:0/451868581 conn(0x7fec540fc380 msgr2=0x7fec54104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:32.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.276+0000 7fec5a000700 1 -- 172.21.15.78:0/451868581 shutdown_connections 2024-03-14T09:53:32.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:32.276+0000 7fec5a000700 1 -- 172.21.15.78:0/451868581 wait complete. 2024-03-14T09:53:32.664 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-14T09:53:32.674 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-03-14T09:53:32.674 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-14T09:53:32.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:32 smithi152 ceph-mon[48279]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:33.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:32 smithi132 ceph-mon[47959]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:33.035 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:53:33.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:32 smithi078 ceph-mon[38143]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:34.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:33 smithi132 ceph-mon[47959]: pgmap v281: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:34.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:33 smithi078 ceph-mon[38143]: pgmap v281: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:34.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:33 smithi152 ceph-mon[48279]: pgmap v281: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:34.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.282+0000 7f40fc000700 1 -- 172.21.15.78:0/2009331903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40f4102980 msgr2=0x7f40f4109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:34.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.282+0000 7f40fc000700 1 --2- 172.21.15.78:0/2009331903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40f4102980 0x7f40f4109230 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00b0d0 tx=0x7f40ec00b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:34.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.282+0000 7f40fc000700 1 -- 172.21.15.78:0/2009331903 shutdown_connections 2024-03-14T09:53:34.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.282+0000 7f40fc000700 1 --2- 172.21.15.78:0/2009331903 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40f4100de0 0x7f40f4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:34.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.282+0000 7f40fc000700 1 --2- 172.21.15.78:0/2009331903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40f4102980 0x7f40f4109230 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:34.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.282+0000 7f40fc000700 1 --2- 172.21.15.78:0/2009331903 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40f4101fe0 0x7f40f4102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:34.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.282+0000 7f40fc000700 1 -- 172.21.15.78:0/2009331903 >> 172.21.15.78:0/2009331903 conn(0x7f40f40fc380 msgr2=0x7f40f40fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:34.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.282+0000 7f40fc000700 1 -- 172.21.15.78:0/2009331903 shutdown_connections 2024-03-14T09:53:34.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.282+0000 7f40fc000700 1 -- 172.21.15.78:0/2009331903 wait complete. 2024-03-14T09:53:34.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.283+0000 7f40fc000700 1 Processor -- start 2024-03-14T09:53:34.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.283+0000 7f40fc000700 1 -- start start 2024-03-14T09:53:34.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.283+0000 7f40fc000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40f4100de0 0x7f40f4197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:34.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.283+0000 7f40fc000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40f4101fe0 0x7f40f41977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:34.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.283+0000 7f40fc000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40f4102980 0x7f40f419e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:34.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.283+0000 7f40fc000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f419ed60 con 0x7f40f4102980 2024-03-14T09:53:34.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.283+0000 7f40fc000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f419eea0 con 0x7f40f4101fe0 2024-03-14T09:53:34.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.283+0000 7f40fc000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f419f030 con 0x7f40f4100de0 2024-03-14T09:53:34.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.284+0000 7f40f9600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40f4100de0 0x7f40f4197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:34.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.284+0000 7f40fa000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40f4102980 0x7f40f419e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:34.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.284+0000 7f40f9600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40f4100de0 0x7f40f4197270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:51538/0 (socket says 172.21.15.78:51538) 2024-03-14T09:53:34.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.284+0000 7f40f9600700 1 -- 172.21.15.78:0/874337379 learned_addr learned my addr 172.21.15.78:0/874337379 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:34.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.284+0000 7f40f8c00700 1 --2- 172.21.15.78:0/874337379 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40f4101fe0 0x7f40f41977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:34.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.284+0000 7f40f9600700 1 -- 172.21.15.78:0/874337379 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40f4101fe0 msgr2=0x7f40f41977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:34.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.284+0000 7f40f9600700 1 --2- 172.21.15.78:0/874337379 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40f4101fe0 0x7f40f41977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:34.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.284+0000 7f40f9600700 1 -- 172.21.15.78:0/874337379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40f4102980 msgr2=0x7f40f419e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:34.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.284+0000 7f40f9600700 1 --2- 172.21.15.78:0/874337379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40f4102980 0x7f40f419e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:34.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.284+0000 7f40f9600700 1 -- 172.21.15.78:0/874337379 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ec009d00 con 0x7f40f4100de0 2024-03-14T09:53:34.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.284+0000 7f40fa000700 1 --2- 172.21.15.78:0/874337379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40f4102980 0x7f40f419e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:34.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.284+0000 7f40f9600700 1 --2- 172.21.15.78:0/874337379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40f4100de0 0x7f40f4197270 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f40e800e9a0 tx=0x7f40e800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:34.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.285+0000 7f40f2000700 1 -- 172.21.15.78:0/874337379 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e8009790 con 0x7f40f4100de0 2024-03-14T09:53:34.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.285+0000 7f40f2000700 1 -- 172.21.15.78:0/874337379 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40e800c810 con 0x7f40f4100de0 2024-03-14T09:53:34.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.285+0000 7f40fc000700 1 -- 172.21.15.78:0/874337379 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f419f310 con 0x7f40f4100de0 2024-03-14T09:53:34.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.285+0000 7f40f2000700 1 -- 172.21.15.78:0/874337379 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e8003c70 con 0x7f40f4100de0 2024-03-14T09:53:34.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.285+0000 7f40fc000700 1 -- 172.21.15.78:0/874337379 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f419f810 con 0x7f40f4100de0 2024-03-14T09:53:34.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.286+0000 7f40fc000700 1 -- 172.21.15.78:0/874337379 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f41081a0 con 0x7f40f4100de0 2024-03-14T09:53:34.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.288+0000 7f40f2000700 1 -- 172.21.15.78:0/874337379 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f40e800cc80 con 0x7f40f4100de0 2024-03-14T09:53:34.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.289+0000 7f40f2000700 1 --2- 172.21.15.78:0/874337379 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f40dc0a0b20 0x7f40dc0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:34.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.289+0000 7f40f2000700 1 -- 172.21.15.78:0/874337379 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f40e8014070 con 0x7f40f4100de0 2024-03-14T09:53:34.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.289+0000 7f40f8c00700 1 --2- 172.21.15.78:0/874337379 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f40dc0a0b20 0x7f40dc0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:34.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.290+0000 7f40f8c00700 1 --2- 172.21.15.78:0/874337379 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f40dc0a0b20 0x7f40dc0a2fe0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f40e0009c60 tx=0x7f40e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:34.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.291+0000 7f40f2000700 1 -- 172.21.15.78:0/874337379 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f40e8095c60 con 0x7f40f4100de0 2024-03-14T09:53:34.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.536+0000 7f40fc000700 1 -- 172.21.15.78:0/874337379 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f40f410ca60 con 0x7f40f4100de0 2024-03-14T09:53:34.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.537+0000 7f40f2000700 1 -- 172.21.15.78:0/874337379 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+2901 (secure 0 0 0) 0x7f40e8099280 con 0x7f40f4100de0 2024-03-14T09:53:34.538 INFO:teuthology.orchestra.run.smithi078.stdout:e24 2024-03-14T09:53:34.538 INFO:teuthology.orchestra.run.smithi078.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T09:53:34.538 INFO:teuthology.orchestra.run.smithi078.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T09:53:34.539 INFO:teuthology.orchestra.run.smithi078.stdout:legacy client fscid: 1 2024-03-14T09:53:34.539 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:34.539 INFO:teuthology.orchestra.run.smithi078.stdout:Filesystem 'cephfs' (1) 2024-03-14T09:53:34.539 INFO:teuthology.orchestra.run.smithi078.stdout:fs_name cephfs 2024-03-14T09:53:34.539 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 24 2024-03-14T09:53:34.539 INFO:teuthology.orchestra.run.smithi078.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-14T09:53:34.540 INFO:teuthology.orchestra.run.smithi078.stdout:created 2024-03-14T09:52:05.708711+0000 2024-03-14T09:53:34.540 INFO:teuthology.orchestra.run.smithi078.stdout:modified 2024-03-14T09:52:30.715484+0000 2024-03-14T09:53:34.540 INFO:teuthology.orchestra.run.smithi078.stdout:tableserver 0 2024-03-14T09:53:34.540 INFO:teuthology.orchestra.run.smithi078.stdout:root 0 2024-03-14T09:53:34.540 INFO:teuthology.orchestra.run.smithi078.stdout:session_timeout 300 2024-03-14T09:53:34.541 INFO:teuthology.orchestra.run.smithi078.stdout:session_autoclose 300 2024-03-14T09:53:34.541 INFO:teuthology.orchestra.run.smithi078.stdout:max_file_size 1099511627776 2024-03-14T09:53:34.541 INFO:teuthology.orchestra.run.smithi078.stdout:max_xattr_size 65536 2024-03-14T09:53:34.541 INFO:teuthology.orchestra.run.smithi078.stdout:required_client_features {} 2024-03-14T09:53:34.541 INFO:teuthology.orchestra.run.smithi078.stdout:last_failure 0 2024-03-14T09:53:34.541 INFO:teuthology.orchestra.run.smithi078.stdout:last_failure_osd_epoch 0 2024-03-14T09:53:34.541 INFO:teuthology.orchestra.run.smithi078.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T09:53:34.542 INFO:teuthology.orchestra.run.smithi078.stdout:max_mds 3 2024-03-14T09:53:34.542 INFO:teuthology.orchestra.run.smithi078.stdout:in 0,1,2 2024-03-14T09:53:34.542 INFO:teuthology.orchestra.run.smithi078.stdout:up {0=24572,1=24538,2=24527} 2024-03-14T09:53:34.542 INFO:teuthology.orchestra.run.smithi078.stdout:failed 2024-03-14T09:53:34.542 INFO:teuthology.orchestra.run.smithi078.stdout:damaged 2024-03-14T09:53:34.542 INFO:teuthology.orchestra.run.smithi078.stdout:stopped 2024-03-14T09:53:34.542 INFO:teuthology.orchestra.run.smithi078.stdout:data_pools [3,4] 2024-03-14T09:53:34.542 INFO:teuthology.orchestra.run.smithi078.stdout:metadata_pool 2 2024-03-14T09:53:34.543 INFO:teuthology.orchestra.run.smithi078.stdout:inline_data disabled 2024-03-14T09:53:34.543 INFO:teuthology.orchestra.run.smithi078.stdout:balancer 2024-03-14T09:53:34.543 INFO:teuthology.orchestra.run.smithi078.stdout:bal_rank_mask -1 2024-03-14T09:53:34.543 INFO:teuthology.orchestra.run.smithi078.stdout:standby_count_wanted 0 2024-03-14T09:53:34.543 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.b{0:24572} state up:active seq 3 addr [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T09:53:34.543 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.i{0:24526} state up:standby-replay seq 1 addr [v2:172.21.15.152:6838/1965809497,v1:172.21.15.152:6839/1965809497] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T09:53:34.543 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.g{1:24538} state up:active seq 10 addr [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T09:53:34.544 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.e{1:24509} state up:standby-replay seq 1 addr [v2:172.21.15.132:6834/1709886688,v1:172.21.15.132:6835/1709886688] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T09:53:34.544 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.h{2:24527} state up:active seq 11 addr [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T09:53:34.544 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.c{2:24508} state up:standby-replay seq 1 addr [v2:172.21.15.152:6836/2123600730,v1:172.21.15.152:6837/2123600730] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T09:53:34.544 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:34.544 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:34.544 INFO:teuthology.orchestra.run.smithi078.stdout:Standby daemons: 2024-03-14T09:53:34.544 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:34.544 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.j{-1:14454} state up:standby seq 1 addr [v2:172.21.15.78:6834/996011838,v1:172.21.15.78:6835/996011838] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T09:53:34.545 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.d{-1:14466} state up:standby seq 1 addr [v2:172.21.15.78:6836/176895986,v1:172.21.15.78:6837/176895986] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T09:53:34.545 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.a{-1:14532} state up:standby seq 1 addr [v2:172.21.15.78:6840/1109060843,v1:172.21.15.78:6841/1109060843] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T09:53:34.545 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.l{-1:24457} state up:standby seq 1 addr [v2:172.21.15.152:6832/4187874184,v1:172.21.15.152:6833/4187874184] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T09:53:34.545 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.k{-1:24479} state up:standby seq 1 addr [v2:172.21.15.132:6832/726198552,v1:172.21.15.132:6833/726198552] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T09:53:34.545 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.f{-1:24502} state up:standby seq 1 addr [v2:172.21.15.152:6834/2986677339,v1:172.21.15.152:6835/2986677339] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T09:53:34.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.539+0000 7f40fc000700 1 -- 172.21.15.78:0/874337379 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f40dc0a0b20 msgr2=0x7f40dc0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:34.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.539+0000 7f40fc000700 1 --2- 172.21.15.78:0/874337379 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f40dc0a0b20 0x7f40dc0a2fe0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f40e0009c60 tx=0x7f40e0009260 comp rx=0 tx=0).stop 2024-03-14T09:53:34.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.539+0000 7f40fc000700 1 -- 172.21.15.78:0/874337379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40f4100de0 msgr2=0x7f40f4197270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.539+0000 7f40fc000700 1 --2- 172.21.15.78:0/874337379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40f4100de0 0x7f40f4197270 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f40e800e9a0 tx=0x7f40e800ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.539+0000 7f40fc000700 1 -- 172.21.15.78:0/874337379 shutdown_connections 2024-03-14T09:53:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.539+0000 7f40fc000700 1 --2- 172.21.15.78:0/874337379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40f4102980 0x7f40f419e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.539+0000 7f40fc000700 1 --2- 172.21.15.78:0/874337379 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f40f4100de0 0x7f40f4197270 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.539+0000 7f40fc000700 1 --2- 172.21.15.78:0/874337379 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f40dc0a0b20 0x7f40dc0a2fe0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.539+0000 7f40fc000700 1 --2- 172.21.15.78:0/874337379 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40f4101fe0 0x7f40f41977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.539+0000 7f40fc000700 1 -- 172.21.15.78:0/874337379 >> 172.21.15.78:0/874337379 conn(0x7f40f40fc380 msgr2=0x7f40f41052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.539+0000 7f40fc000700 1 -- 172.21.15.78:0/874337379 shutdown_connections 2024-03-14T09:53:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:34.539+0000 7f40fc000700 1 -- 172.21.15.78:0/874337379 wait complete. 2024-03-14T09:53:34.547 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2024-03-14T09:53:34.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:34 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/874337379' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T09:53:34.948 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-14T09:53:35.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:34 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/874337379' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T09:53:35.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:34 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/874337379' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T09:53:35.309 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:53:36.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:35 smithi132 ceph-mon[47959]: pgmap v282: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-14T09:53:36.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:35 smithi078 ceph-mon[38143]: pgmap v282: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-14T09:53:36.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:35 smithi152 ceph-mon[48279]: pgmap v282: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-14T09:53:36.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.440+0000 7f4e52e00700 1 -- 172.21.15.78:0/2602830268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4c106ac0 msgr2=0x7f4e4c108f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.440+0000 7f4e52e00700 1 --2- 172.21.15.78:0/2602830268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4c106ac0 0x7f4e4c108f30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400b0a0 tx=0x7f4e4400b3b0 comp rx=0 tx=0).stop 2024-03-14T09:53:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.440+0000 7f4e52e00700 1 -- 172.21.15.78:0/2602830268 shutdown_connections 2024-03-14T09:53:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.440+0000 7f4e52e00700 1 --2- 172.21.15.78:0/2602830268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e4c101860 0x7f4e4c103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.440+0000 7f4e52e00700 1 --2- 172.21.15.78:0/2602830268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4c106ac0 0x7f4e4c108f30 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.440+0000 7f4e52e00700 1 --2- 172.21.15.78:0/2602830268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4e4c104190 0x7f4e4c106580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.440+0000 7f4e52e00700 1 -- 172.21.15.78:0/2602830268 >> 172.21.15.78:0/2602830268 conn(0x7f4e4c0fb270 msgr2=0x7f4e4c0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.440+0000 7f4e52e00700 1 -- 172.21.15.78:0/2602830268 shutdown_connections 2024-03-14T09:53:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.440+0000 7f4e52e00700 1 -- 172.21.15.78:0/2602830268 wait complete. 2024-03-14T09:53:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.441+0000 7f4e52e00700 1 Processor -- start 2024-03-14T09:53:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.441+0000 7f4e52e00700 1 -- start start 2024-03-14T09:53:36.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e52e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4c101860 0x7f4e4c197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:36.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e52e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e4c104190 0x7f4e4c197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:36.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e52e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4e4c106ac0 0x7f4e4c19e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:36.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e52e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c19ed40 con 0x7f4e4c101860 2024-03-14T09:53:36.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e52e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c19ee80 con 0x7f4e4c104190 2024-03-14T09:53:36.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e52e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c19f010 con 0x7f4e4c106ac0 2024-03-14T09:53:36.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e4be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4c101860 0x7f4e4c197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:36.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e4b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e4c104190 0x7f4e4c197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e50e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4e4c106ac0 0x7f4e4c19e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e4b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e4c104190 0x7f4e4c197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:35580/0 (socket says 172.21.15.78:35580) 2024-03-14T09:53:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e4b400700 1 -- 172.21.15.78:0/1582339672 learned_addr learned my addr 172.21.15.78:0/1582339672 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e50e00700 1 -- 172.21.15.78:0/1582339672 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e4c104190 msgr2=0x7f4e4c197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e50e00700 1 --2- 172.21.15.78:0/1582339672 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e4c104190 0x7f4e4c197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e50e00700 1 -- 172.21.15.78:0/1582339672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4c101860 msgr2=0x7f4e4c197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e50e00700 1 --2- 172.21.15.78:0/1582339672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4c101860 0x7f4e4c197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e50e00700 1 -- 172.21.15.78:0/1582339672 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e44009d00 con 0x7f4e4c106ac0 2024-03-14T09:53:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.442+0000 7f4e4b400700 1 --2- 172.21.15.78:0/1582339672 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e4c104190 0x7f4e4c197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.443+0000 7f4e50e00700 1 --2- 172.21.15.78:0/1582339672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4e4c106ac0 0x7f4e4c19e770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400b070 tx=0x7f4e44007930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.443+0000 7f4e48c00700 1 -- 172.21.15.78:0/1582339672 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4400e040 con 0x7f4e4c106ac0 2024-03-14T09:53:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.443+0000 7f4e48c00700 1 -- 172.21.15.78:0/1582339672 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e4400be40 con 0x7f4e4c106ac0 2024-03-14T09:53:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.443+0000 7f4e48c00700 1 -- 172.21.15.78:0/1582339672 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4401b460 con 0x7f4e4c106ac0 2024-03-14T09:53:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.443+0000 7f4e52e00700 1 -- 172.21.15.78:0/1582339672 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4c19f290 con 0x7f4e4c106ac0 2024-03-14T09:53:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.443+0000 7f4e52e00700 1 -- 172.21.15.78:0/1582339672 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4c19f6a0 con 0x7f4e4c106ac0 2024-03-14T09:53:36.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.444+0000 7f4e52e00700 1 -- 172.21.15.78:0/1582339672 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e4c0fcdf0 con 0x7f4e4c106ac0 2024-03-14T09:53:36.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.446+0000 7f4e48c00700 1 -- 172.21.15.78:0/1582339672 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f4e4401d470 con 0x7f4e4c106ac0 2024-03-14T09:53:36.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.447+0000 7f4e48c00700 1 --2- 172.21.15.78:0/1582339672 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4e300a0ba0 0x7f4e300a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:36.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.448+0000 7f4e48c00700 1 -- 172.21.15.78:0/1582339672 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4e440cda50 con 0x7f4e4c106ac0 2024-03-14T09:53:36.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.448+0000 7f4e4be00700 1 --2- 172.21.15.78:0/1582339672 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4e300a0ba0 0x7f4e300a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.448+0000 7f4e4be00700 1 --2- 172.21.15.78:0/1582339672 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4e300a0ba0 0x7f4e300a3060 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c009c60 tx=0x7f4e3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.450+0000 7f4e48c00700 1 -- 172.21.15.78:0/1582339672 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4e44096fe0 con 0x7f4e4c106ac0 2024-03-14T09:53:36.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.654+0000 7f4e52e00700 1 -- 172.21.15.78:0/1582339672 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f4e4c10ca30 con 0x7f4e4c106ac0 2024-03-14T09:53:36.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.656+0000 7f4e48c00700 1 -- 172.21.15.78:0/1582339672 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v85) v1 ==== 56+0+5591 (secure 0 0 0) 0x7f4e4409a600 con 0x7f4e4c106ac0 2024-03-14T09:53:36.656 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 85 2024-03-14T09:53:36.657 INFO:teuthology.orchestra.run.smithi078.stdout:fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T09:53:36.657 INFO:teuthology.orchestra.run.smithi078.stdout:created 2024-03-14T09:45:43.515085+0000 2024-03-14T09:53:36.657 INFO:teuthology.orchestra.run.smithi078.stdout:modified 2024-03-14T09:52:42.134030+0000 2024-03-14T09:53:36.657 INFO:teuthology.orchestra.run.smithi078.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-14T09:53:36.657 INFO:teuthology.orchestra.run.smithi078.stdout:crush_version 27 2024-03-14T09:53:36.657 INFO:teuthology.orchestra.run.smithi078.stdout:full_ratio 0.95 2024-03-14T09:53:36.657 INFO:teuthology.orchestra.run.smithi078.stdout:backfillfull_ratio 0.9 2024-03-14T09:53:36.657 INFO:teuthology.orchestra.run.smithi078.stdout:nearfull_ratio 0.85 2024-03-14T09:53:36.657 INFO:teuthology.orchestra.run.smithi078.stdout:require_min_compat_client luminous 2024-03-14T09:53:36.657 INFO:teuthology.orchestra.run.smithi078.stdout:min_compat_client luminous 2024-03-14T09:53:36.657 INFO:teuthology.orchestra.run.smithi078.stdout:require_osd_release reef 2024-03-14T09:53:36.657 INFO:teuthology.orchestra.run.smithi078.stdout:stretch_mode_enabled false 2024-03-14T09:53:36.657 INFO:teuthology.orchestra.run.smithi078.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-14T09:53:36.657 INFO:teuthology.orchestra.run.smithi078.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 78 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-14T09:53:36.658 INFO:teuthology.orchestra.run.smithi078.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-14T09:53:36.658 INFO:teuthology.orchestra.run.smithi078.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 83 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-14T09:53:36.658 INFO:teuthology.orchestra.run.smithi078.stdout:max_osd 12 2024-03-14T09:53:36.658 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 up in weight 1 up_from 8 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6802/1911630001,v1:172.21.15.78:6803/1911630001] [v2:172.21.15.78:6804/1911630001,v1:172.21.15.78:6805/1911630001] exists,up a9918c01-e344-4789-83f9-6f9425f98542 2024-03-14T09:53:36.658 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 up in weight 1 up_from 13 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6800/162295960,v1:172.21.15.132:6801/162295960] [v2:172.21.15.132:6802/162295960,v1:172.21.15.132:6803/162295960] exists,up 1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71 2024-03-14T09:53:36.658 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 up in weight 1 up_from 18 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6800/3822041535,v1:172.21.15.152:6801/3822041535] [v2:172.21.15.152:6802/3822041535,v1:172.21.15.152:6803/3822041535] exists,up 410edac5-366f-4f37-aaf3-75383d995b28 2024-03-14T09:53:36.658 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 up in weight 1 up_from 24 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6810/3422121058,v1:172.21.15.78:6811/3422121058] [v2:172.21.15.78:6812/3422121058,v1:172.21.15.78:6813/3422121058] exists,up 3f480434-66c7-4aba-bea5-e74d94e480d8 2024-03-14T09:53:36.658 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 up in weight 1 up_from 31 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6808/830577797,v1:172.21.15.132:6809/830577797] [v2:172.21.15.132:6810/830577797,v1:172.21.15.132:6811/830577797] exists,up fc8670cb-b4c1-42fc-9134-4fce4b31e03c 2024-03-14T09:53:36.658 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 up in weight 1 up_from 36 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6808/1175069776,v1:172.21.15.152:6809/1175069776] [v2:172.21.15.152:6810/1175069776,v1:172.21.15.152:6811/1175069776] exists,up a315a4be-a807-4aef-b476-21ca966b20fa 2024-03-14T09:53:36.658 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 up in weight 1 up_from 43 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6818/996423467,v1:172.21.15.78:6819/996423467] [v2:172.21.15.78:6820/996423467,v1:172.21.15.78:6821/996423467] exists,up 8b9f16ed-3677-433d-976b-99b3b21f3479 2024-03-14T09:53:36.658 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 up in weight 1 up_from 48 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6816/1879810712,v1:172.21.15.132:6817/1879810712] [v2:172.21.15.132:6818/1879810712,v1:172.21.15.132:6819/1879810712] exists,up f097c83a-d182-409b-bfe8-5bf6a6627fef 2024-03-14T09:53:36.658 INFO:teuthology.orchestra.run.smithi078.stdout:osd.8 up in weight 1 up_from 55 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6816/2604540397,v1:172.21.15.152:6817/2604540397] [v2:172.21.15.152:6818/2604540397,v1:172.21.15.152:6819/2604540397] exists,up 238d2afc-7d81-4089-98be-2852f0504b6c 2024-03-14T09:53:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:osd.9 up in weight 1 up_from 62 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6826/803422096,v1:172.21.15.78:6827/803422096] [v2:172.21.15.78:6828/803422096,v1:172.21.15.78:6829/803422096] exists,up 15f5ddad-e37a-4291-8d62-05fa69894552 2024-03-14T09:53:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:osd.10 up in weight 1 up_from 67 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6824/81920089,v1:172.21.15.132:6825/81920089] [v2:172.21.15.132:6826/81920089,v1:172.21.15.132:6827/81920089] exists,up 7a325f26-e31f-4f0e-b09f-8bda5b27fbcd 2024-03-14T09:53:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:osd.11 up in weight 1 up_from 74 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.152:6824/2318800042,v1:172.21.15.152:6825/2318800042] [v2:172.21.15.152:6826/2318800042,v1:172.21.15.152:6827/2318800042] exists,up 37c97c4c-a16b-4b02-9bd0-585aea0baafc 2024-03-14T09:53:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 2.b [2,4] 2024-03-14T09:53:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 2.10 [2,1] 2024-03-14T09:53:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 2.1d [8,11] 2024-03-14T09:53:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 2.1f [7,11] 2024-03-14T09:53:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.1d [4,6] 2024-03-14T09:53:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.25 [5,9] 2024-03-14T09:53:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.29 [3,9] 2024-03-14T09:53:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.34 [4,9] 2024-03-14T09:53:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.3a [4,8] 2024-03-14T09:53:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.2 [9,1] 2024-03-14T09:53:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6801/3890232539 expires 2024-03-15T09:46:40.988022+0000 2024-03-14T09:53:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6800/3890232539 expires 2024-03-15T09:46:40.988022+0000 2024-03-14T09:53:36.659 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/2260754576 expires 2024-03-15T09:46:40.988022+0000 2024-03-14T09:53:36.660 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6800/1325295736 expires 2024-03-15T09:46:17.100896+0000 2024-03-14T09:53:36.660 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/119757382 expires 2024-03-15T09:46:17.100896+0000 2024-03-14T09:53:36.660 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6801/1325295736 expires 2024-03-15T09:46:17.100896+0000 2024-03-14T09:53:36.660 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/1127810454 expires 2024-03-15T09:46:17.100896+0000 2024-03-14T09:53:36.660 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/2380275837 expires 2024-03-15T09:46:40.988022+0000 2024-03-14T09:53:36.660 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/2472038902 expires 2024-03-15T09:46:17.100896+0000 2024-03-14T09:53:36.660 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/4237457492 expires 2024-03-15T09:46:40.988022+0000 2024-03-14T09:53:36.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.658+0000 7f4e52e00700 1 -- 172.21.15.78:0/1582339672 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4e300a0ba0 msgr2=0x7f4e300a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:36.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.658+0000 7f4e52e00700 1 --2- 172.21.15.78:0/1582339672 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4e300a0ba0 0x7f4e300a3060 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c009c60 tx=0x7f4e3c009260 comp rx=0 tx=0).stop 2024-03-14T09:53:36.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.658+0000 7f4e52e00700 1 -- 172.21.15.78:0/1582339672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4e4c106ac0 msgr2=0x7f4e4c19e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:36.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.658+0000 7f4e52e00700 1 --2- 172.21.15.78:0/1582339672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4e4c106ac0 0x7f4e4c19e770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400b070 tx=0x7f4e44007930 comp rx=0 tx=0).stop 2024-03-14T09:53:36.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.658+0000 7f4e52e00700 1 -- 172.21.15.78:0/1582339672 shutdown_connections 2024-03-14T09:53:36.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.658+0000 7f4e52e00700 1 --2- 172.21.15.78:0/1582339672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4e4c106ac0 0x7f4e4c19e770 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:36.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.658+0000 7f4e52e00700 1 --2- 172.21.15.78:0/1582339672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e4c101860 0x7f4e4c197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:36.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.658+0000 7f4e52e00700 1 --2- 172.21.15.78:0/1582339672 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4e300a0ba0 0x7f4e300a3060 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:36.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.658+0000 7f4e52e00700 1 --2- 172.21.15.78:0/1582339672 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e4c104190 0x7f4e4c197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:36.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.658+0000 7f4e52e00700 1 -- 172.21.15.78:0/1582339672 >> 172.21.15.78:0/1582339672 conn(0x7f4e4c0fb270 msgr2=0x7f4e4c1021a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:36.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.658+0000 7f4e52e00700 1 -- 172.21.15.78:0/1582339672 shutdown_connections 2024-03-14T09:53:36.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:36.658+0000 7f4e52e00700 1 -- 172.21.15.78:0/1582339672 wait complete. 2024-03-14T09:53:37.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:36 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1582339672' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-14T09:53:37.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:36 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1582339672' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-14T09:53:37.072 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-14T09:53:37.083 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:53:37.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:36 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1582339672' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-14T09:53:37.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.280+0000 7f7008e00700 1 -- 172.21.15.78:0/1894007594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70041027e0 msgr2=0x7f7004109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:37.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.280+0000 7f7008e00700 1 --2- 172.21.15.78:0/1894007594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70041027e0 0x7f7004109090 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc00b0d0 tx=0x7f6ffc00b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.280+0000 7f7008e00700 1 -- 172.21.15.78:0/1894007594 shutdown_connections 2024-03-14T09:53:37.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.280+0000 7f7008e00700 1 --2- 172.21.15.78:0/1894007594 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7004100c50 0x7f7004101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.280+0000 7f7008e00700 1 --2- 172.21.15.78:0/1894007594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70041027e0 0x7f7004109090 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.280+0000 7f7008e00700 1 --2- 172.21.15.78:0/1894007594 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7004101e40 0x7f70041022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.280+0000 7f7008e00700 1 -- 172.21.15.78:0/1894007594 >> 172.21.15.78:0/1894007594 conn(0x7f70040fc1c0 msgr2=0x7f70040fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:37.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.280+0000 7f7008e00700 1 -- 172.21.15.78:0/1894007594 shutdown_connections 2024-03-14T09:53:37.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.280+0000 7f7008e00700 1 -- 172.21.15.78:0/1894007594 wait complete. 2024-03-14T09:53:37.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.281+0000 7f7008e00700 1 Processor -- start 2024-03-14T09:53:37.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.281+0000 7f7008e00700 1 -- start start 2024-03-14T09:53:37.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f7008e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7004100c50 0x7f7004074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:37.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f7008e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7004101e40 0x7f70040731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:37.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f7008e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70041027e0 0x7f70040736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:37.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f7008e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7004073e80 con 0x7f70041027e0 2024-03-14T09:53:37.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f7008e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7004073ff0 con 0x7f7004101e40 2024-03-14T09:53:37.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f7008e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70041a1110 con 0x7f7004100c50 2024-03-14T09:53:37.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f7001e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7004100c50 0x7f7004074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:37.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f7002800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70041027e0 0x7f70040736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:37.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f6ffbe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7004101e40 0x7f70040731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:37.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f7001e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7004100c50 0x7f7004074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:51584/0 (socket says 172.21.15.78:51584) 2024-03-14T09:53:37.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f7001e00700 1 -- 172.21.15.78:0/2179636690 learned_addr learned my addr 172.21.15.78:0/2179636690 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:37.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f7001e00700 1 -- 172.21.15.78:0/2179636690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7004101e40 msgr2=0x7f70040731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:37.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f7001e00700 1 --2- 172.21.15.78:0/2179636690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7004101e40 0x7f70040731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f7001e00700 1 -- 172.21.15.78:0/2179636690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70041027e0 msgr2=0x7f70040736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:37.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f7001e00700 1 --2- 172.21.15.78:0/2179636690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70041027e0 0x7f70040736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f7001e00700 1 -- 172.21.15.78:0/2179636690 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc009d00 con 0x7f7004100c50 2024-03-14T09:53:37.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.282+0000 7f7002800700 1 --2- 172.21.15.78:0/2179636690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70041027e0 0x7f70040736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:37.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.283+0000 7f7001e00700 1 --2- 172.21.15.78:0/2179636690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7004100c50 0x7f7004074b10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400d980 tx=0x7f6ff400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:37.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.283+0000 7f6ffaa00700 1 -- 172.21.15.78:0/2179636690 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff400f890 con 0x7f7004100c50 2024-03-14T09:53:37.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.283+0000 7f7008e00700 1 -- 172.21.15.78:0/2179636690 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70041a1340 con 0x7f7004100c50 2024-03-14T09:53:37.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.283+0000 7f6ffaa00700 1 -- 172.21.15.78:0/2179636690 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ff4010460 con 0x7f7004100c50 2024-03-14T09:53:37.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.283+0000 7f6ffaa00700 1 -- 172.21.15.78:0/2179636690 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40195c0 con 0x7f7004100c50 2024-03-14T09:53:37.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.283+0000 7f7008e00700 1 -- 172.21.15.78:0/2179636690 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70041a18a0 con 0x7f7004100c50 2024-03-14T09:53:37.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.284+0000 7f7008e00700 1 -- 172.21.15.78:0/2179636690 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70041080a0 con 0x7f7004100c50 2024-03-14T09:53:37.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.286+0000 7f6ffaa00700 1 -- 172.21.15.78:0/2179636690 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f6ff40105d0 con 0x7f7004100c50 2024-03-14T09:53:37.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.287+0000 7f6ffaa00700 1 --2- 172.21.15.78:0/2179636690 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6fe80a0b20 0x7f6fe80a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:37.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.287+0000 7f6ffaa00700 1 -- 172.21.15.78:0/2179636690 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6ff40cbfe0 con 0x7f7004100c50 2024-03-14T09:53:37.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.288+0000 7f6ffbe00700 1 --2- 172.21.15.78:0/2179636690 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6fe80a0b20 0x7f6fe80a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:37.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.288+0000 7f6ffbe00700 1 --2- 172.21.15.78:0/2179636690 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6fe80a0b20 0x7f6fe80a2fe0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009be0 tx=0x7f6fec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:37.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.291+0000 7f6ffaa00700 1 -- 172.21.15.78:0/2179636690 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6ff40ff960 con 0x7f7004100c50 2024-03-14T09:53:37.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.504+0000 7f7008e00700 1 -- 172.21.15.78:0/2179636690 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f700410c890 con 0x7f7004100c50 2024-03-14T09:53:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.506+0000 7f6ffaa00700 1 -- 172.21.15.78:0/2179636690 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+24453 (secure 0 0 0) 0x7f6ff4095500 con 0x7f7004100c50 2024-03-14T09:53:37.507 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:37.507 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":85,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:52:42.134030+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:53:37.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.508+0000 7f7008e00700 1 -- 172.21.15.78:0/2179636690 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6fe80a0b20 msgr2=0x7f6fe80a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:37.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.508+0000 7f7008e00700 1 --2- 172.21.15.78:0/2179636690 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6fe80a0b20 0x7f6fe80a2fe0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009be0 tx=0x7f6fec009260 comp rx=0 tx=0).stop 2024-03-14T09:53:37.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.508+0000 7f7008e00700 1 -- 172.21.15.78:0/2179636690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7004100c50 msgr2=0x7f7004074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:37.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.508+0000 7f7008e00700 1 --2- 172.21.15.78:0/2179636690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7004100c50 0x7f7004074b10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400d980 tx=0x7f6ff400dc90 comp rx=0 tx=0).stop 2024-03-14T09:53:37.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.508+0000 7f7008e00700 1 -- 172.21.15.78:0/2179636690 shutdown_connections 2024-03-14T09:53:37.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.508+0000 7f7008e00700 1 --2- 172.21.15.78:0/2179636690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70041027e0 0x7f70040736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.508+0000 7f7008e00700 1 --2- 172.21.15.78:0/2179636690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7004100c50 0x7f7004074b10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.508+0000 7f7008e00700 1 --2- 172.21.15.78:0/2179636690 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f6fe80a0b20 0x7f6fe80a2fe0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.508+0000 7f7008e00700 1 --2- 172.21.15.78:0/2179636690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7004101e40 0x7f70040731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.508+0000 7f7008e00700 1 -- 172.21.15.78:0/2179636690 >> 172.21.15.78:0/2179636690 conn(0x7f70040fc1c0 msgr2=0x7f70041051e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:37.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.508+0000 7f7008e00700 1 -- 172.21.15.78:0/2179636690 shutdown_connections 2024-03-14T09:53:37.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.508+0000 7f7008e00700 1 -- 172.21.15.78:0/2179636690 wait complete. 2024-03-14T09:53:37.528 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T09:48:37.618450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-14T09:52:03.938666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-14T09:52:05.149985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-14T09:52:08.227698+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-14T09:53:37.528 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T09:53:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.724+0000 7f4f64400700 1 -- 172.21.15.78:0/1017951533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f5c073790 msgr2=0x7f4f5c073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.724+0000 7f4f64400700 1 --2- 172.21.15.78:0/1017951533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f5c073790 0x7f4f5c073c10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009990 tx=0x7f4f48009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.724+0000 7f4f64400700 1 -- 172.21.15.78:0/1017951533 shutdown_connections 2024-03-14T09:53:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.724+0000 7f4f64400700 1 --2- 172.21.15.78:0/1017951533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f5c074d70 0x7f4f5c0731c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.724+0000 7f4f64400700 1 --2- 172.21.15.78:0/1017951533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f5c108c40 0x7f4f5c10b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.724+0000 7f4f64400700 1 --2- 172.21.15.78:0/1017951533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f5c073790 0x7f4f5c073c10 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.724+0000 7f4f64400700 1 -- 172.21.15.78:0/1017951533 >> 172.21.15.78:0/1017951533 conn(0x7f4f5c0fc2c0 msgr2=0x7f4f5c0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.724+0000 7f4f64400700 1 -- 172.21.15.78:0/1017951533 shutdown_connections 2024-03-14T09:53:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.724+0000 7f4f64400700 1 -- 172.21.15.78:0/1017951533 wait complete. 2024-03-14T09:53:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.725+0000 7f4f64400700 1 Processor -- start 2024-03-14T09:53:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.726+0000 7f4f64400700 1 -- start start 2024-03-14T09:53:37.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.726+0000 7f4f64400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f5c073790 0x7f4f5c1992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:37.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.726+0000 7f4f64400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f5c074d70 0x7f4f5c199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:37.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.726+0000 7f4f64400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f5c108c40 0x7f4f5c1a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:37.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.726+0000 7f4f64400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c1a0df0 con 0x7f4f5c074d70 2024-03-14T09:53:37.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.726+0000 7f4f64400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c1a0f30 con 0x7f4f5c073790 2024-03-14T09:53:37.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.726+0000 7f4f64400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c1a10c0 con 0x7f4f5c108c40 2024-03-14T09:53:37.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.726+0000 7f4f61000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f5c074d70 0x7f4f5c199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:37.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.726+0000 7f4f61000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f5c074d70 0x7f4f5c199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52422/0 (socket says 172.21.15.78:52422) 2024-03-14T09:53:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.726+0000 7f4f61000700 1 -- 172.21.15.78:0/2426481561 learned_addr learned my addr 172.21.15.78:0/2426481561 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.726+0000 7f4f61a00700 1 --2- 172.21.15.78:0/2426481561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f5c073790 0x7f4f5c1992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.727+0000 7f4f62400700 1 --2- 172.21.15.78:0/2426481561 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f5c108c40 0x7f4f5c1a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.727+0000 7f4f61000700 1 -- 172.21.15.78:0/2426481561 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f5c108c40 msgr2=0x7f4f5c1a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.727+0000 7f4f61000700 1 --2- 172.21.15.78:0/2426481561 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f5c108c40 0x7f4f5c1a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.727+0000 7f4f61000700 1 -- 172.21.15.78:0/2426481561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f5c073790 msgr2=0x7f4f5c1992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.727+0000 7f4f61000700 1 --2- 172.21.15.78:0/2426481561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f5c073790 0x7f4f5c1992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.727+0000 7f4f61000700 1 -- 172.21.15.78:0/2426481561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f480095f0 con 0x7f4f5c074d70 2024-03-14T09:53:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.727+0000 7f4f61a00700 1 --2- 172.21.15.78:0/2426481561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f5c073790 0x7f4f5c1992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.727+0000 7f4f61000700 1 --2- 172.21.15.78:0/2426481561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f5c074d70 0x7f4f5c199800 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4f4800b680 tx=0x7f4f48009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.727+0000 7f4f5a000700 1 -- 172.21.15.78:0/2426481561 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4801d070 con 0x7f4f5c074d70 2024-03-14T09:53:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.727+0000 7f4f5a000700 1 -- 172.21.15.78:0/2426481561 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f48022470 con 0x7f4f5c074d70 2024-03-14T09:53:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.727+0000 7f4f64400700 1 -- 172.21.15.78:0/2426481561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f5c1a1340 con 0x7f4f5c074d70 2024-03-14T09:53:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.727+0000 7f4f5a000700 1 -- 172.21.15.78:0/2426481561 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48017650 con 0x7f4f5c074d70 2024-03-14T09:53:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.728+0000 7f4f64400700 1 -- 172.21.15.78:0/2426481561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5c1a17c0 con 0x7f4f5c074d70 2024-03-14T09:53:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.729+0000 7f4f64400700 1 -- 172.21.15.78:0/2426481561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f5c107a50 con 0x7f4f5c074d70 2024-03-14T09:53:37.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.731+0000 7f4f5a000700 1 -- 172.21.15.78:0/2426481561 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f4f480225e0 con 0x7f4f5c074d70 2024-03-14T09:53:37.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.732+0000 7f4f5a000700 1 --2- 172.21.15.78:0/2426481561 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f440a0ba0 0x7f4f440a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:37.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.732+0000 7f4f5a000700 1 -- 172.21.15.78:0/2426481561 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4f480cd780 con 0x7f4f5c074d70 2024-03-14T09:53:37.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.732+0000 7f4f61a00700 1 --2- 172.21.15.78:0/2426481561 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f440a0ba0 0x7f4f440a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:37.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.733+0000 7f4f61a00700 1 --2- 172.21.15.78:0/2426481561 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f440a0ba0 0x7f4f440a3060 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4f50011490 tx=0x7f4f50009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:37.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.736+0000 7f4f5a000700 1 -- 172.21.15.78:0/2426481561 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4f48096d10 con 0x7f4f5c074d70 2024-03-14T09:53:37.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:37 smithi078 ceph-mon[38143]: pgmap v283: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:37.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:37 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2179636690' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:37.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.946+0000 7f4f64400700 1 -- 172.21.15.78:0/2426481561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4f5c10ecc0 con 0x7f4f5c074d70 2024-03-14T09:53:37.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.947+0000 7f4f5a000700 1 -- 172.21.15.78:0/2426481561 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4f4809a330 con 0x7f4f5c074d70 2024-03-14T09:53:37.947 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-14T09:53:37.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.949+0000 7f4f64400700 1 -- 172.21.15.78:0/2426481561 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f440a0ba0 msgr2=0x7f4f440a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:37.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.949+0000 7f4f64400700 1 --2- 172.21.15.78:0/2426481561 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f440a0ba0 0x7f4f440a3060 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4f50011490 tx=0x7f4f50009260 comp rx=0 tx=0).stop 2024-03-14T09:53:37.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.949+0000 7f4f64400700 1 -- 172.21.15.78:0/2426481561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f5c074d70 msgr2=0x7f4f5c199800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:37.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.949+0000 7f4f64400700 1 --2- 172.21.15.78:0/2426481561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f5c074d70 0x7f4f5c199800 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4f4800b680 tx=0x7f4f48009580 comp rx=0 tx=0).stop 2024-03-14T09:53:37.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.949+0000 7f4f64400700 1 -- 172.21.15.78:0/2426481561 shutdown_connections 2024-03-14T09:53:37.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.949+0000 7f4f64400700 1 --2- 172.21.15.78:0/2426481561 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f4f440a0ba0 0x7f4f440a3060 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.949+0000 7f4f64400700 1 --2- 172.21.15.78:0/2426481561 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f5c073790 0x7f4f5c1992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.949+0000 7f4f64400700 1 --2- 172.21.15.78:0/2426481561 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f5c108c40 0x7f4f5c1a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.949+0000 7f4f64400700 1 --2- 172.21.15.78:0/2426481561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f5c074d70 0x7f4f5c199800 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:37.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.949+0000 7f4f64400700 1 -- 172.21.15.78:0/2426481561 >> 172.21.15.78:0/2426481561 conn(0x7f4f5c0fc2c0 msgr2=0x7f4f5c104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:37.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.949+0000 7f4f64400700 1 -- 172.21.15.78:0/2426481561 shutdown_connections 2024-03-14T09:53:37.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:37.949+0000 7f4f64400700 1 -- 172.21.15.78:0/2426481561 wait complete. 2024-03-14T09:53:37.968 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-14T09:53:38.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.163+0000 7f4006200700 1 -- 172.21.15.78:0/1688310345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40000737b0 msgr2=0x7f4000073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:38.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.163+0000 7f4006200700 1 --2- 172.21.15.78:0/1688310345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40000737b0 0x7f4000073c30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3fec009990 tx=0x7f3fec009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.163+0000 7f4006200700 1 -- 172.21.15.78:0/1688310345 shutdown_connections 2024-03-14T09:53:38.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.163+0000 7f4006200700 1 --2- 172.21.15.78:0/1688310345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4000108c20 0x7f400010b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.163+0000 7f4006200700 1 --2- 172.21.15.78:0/1688310345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4000074d90 0x7f40000731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.163+0000 7f4006200700 1 --2- 172.21.15.78:0/1688310345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40000737b0 0x7f4000073c30 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.163+0000 7f4006200700 1 -- 172.21.15.78:0/1688310345 >> 172.21.15.78:0/1688310345 conn(0x7f40000fc2c0 msgr2=0x7f40000fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:38.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.164+0000 7f4006200700 1 -- 172.21.15.78:0/1688310345 shutdown_connections 2024-03-14T09:53:38.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.164+0000 7f4006200700 1 -- 172.21.15.78:0/1688310345 wait complete. 2024-03-14T09:53:38.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.164+0000 7f4006200700 1 Processor -- start 2024-03-14T09:53:38.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.165+0000 7f4006200700 1 -- start start 2024-03-14T09:53:38.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.165+0000 7f4006200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40000737b0 0x7f4000199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:38.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.165+0000 7f4006200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4000074d90 0x7f4000199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:38.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.165+0000 7f4006200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4000108c20 0x7f40001a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.165+0000 7f4006200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40001a0e30 con 0x7f40000737b0 2024-03-14T09:53:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.165+0000 7f4006200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40001a0f70 con 0x7f4000108c20 2024-03-14T09:53:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.165+0000 7f4006200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40001a1100 con 0x7f4000074d90 2024-03-14T09:53:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.165+0000 7f3fff400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40000737b0 0x7f4000199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.165+0000 7f3fff400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40000737b0 0x7f4000199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52430/0 (socket says 172.21.15.78:52430) 2024-03-14T09:53:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.165+0000 7f3fff400700 1 -- 172.21.15.78:0/4168899282 learned_addr learned my addr 172.21.15.78:0/4168899282 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.165+0000 7f3ffea00700 1 --2- 172.21.15.78:0/4168899282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4000074d90 0x7f4000199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.165+0000 7f3fffe00700 1 --2- 172.21.15.78:0/4168899282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4000108c20 0x7f40001a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.166+0000 7f3fff400700 1 -- 172.21.15.78:0/4168899282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4000074d90 msgr2=0x7f4000199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.166+0000 7f3fff400700 1 --2- 172.21.15.78:0/4168899282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4000074d90 0x7f4000199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.166+0000 7f3fff400700 1 -- 172.21.15.78:0/4168899282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4000108c20 msgr2=0x7f40001a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.166+0000 7f3fff400700 1 --2- 172.21.15.78:0/4168899282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4000108c20 0x7f40001a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.166+0000 7f3fff400700 1 -- 172.21.15.78:0/4168899282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fec0095f0 con 0x7f40000737b0 2024-03-14T09:53:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.166+0000 7f3ffea00700 1 --2- 172.21.15.78:0/4168899282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4000074d90 0x7f4000199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.166+0000 7f3fff400700 1 --2- 172.21.15.78:0/4168899282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40000737b0 0x7f4000199300 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000e9a0 tx=0x7f3ff000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.166+0000 7f3ff7e00700 1 -- 172.21.15.78:0/4168899282 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0009790 con 0x7f40000737b0 2024-03-14T09:53:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.166+0000 7f3ff7e00700 1 -- 172.21.15.78:0/4168899282 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ff0010430 con 0x7f40000737b0 2024-03-14T09:53:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.166+0000 7f4006200700 1 -- 172.21.15.78:0/4168899282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40001a13e0 con 0x7f40000737b0 2024-03-14T09:53:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.166+0000 7f3ff7e00700 1 -- 172.21.15.78:0/4168899282 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0018680 con 0x7f40000737b0 2024-03-14T09:53:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.166+0000 7f4006200700 1 -- 172.21.15.78:0/4168899282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40001a18f0 con 0x7f40000737b0 2024-03-14T09:53:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.167+0000 7f4006200700 1 -- 172.21.15.78:0/4168899282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4000107a30 con 0x7f40000737b0 2024-03-14T09:53:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.169+0000 7f3ff7e00700 1 -- 172.21.15.78:0/4168899282 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f3ff00105a0 con 0x7f40000737b0 2024-03-14T09:53:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.171+0000 7f3ff7e00700 1 --2- 172.21.15.78:0/4168899282 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3fe40a0b20 0x7f3fe40a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.171+0000 7f3ff7e00700 1 -- 172.21.15.78:0/4168899282 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3ff0014070 con 0x7f40000737b0 2024-03-14T09:53:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.171+0000 7f3ffea00700 1 --2- 172.21.15.78:0/4168899282 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3fe40a0b20 0x7f3fe40a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.172+0000 7f3ffea00700 1 --2- 172.21.15.78:0/4168899282 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3fe40a0b20 0x7f3fe40a2fe0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00b2e0 tx=0x7f3fec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:38.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.175+0000 7f3ff7e00700 1 -- 172.21.15.78:0/4168899282 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3ff0052960 con 0x7f40000737b0 2024-03-14T09:53:38.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:37 smithi152 ceph-mon[48279]: pgmap v283: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:38.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:37 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2179636690' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:38.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:37 smithi132 ceph-mon[47959]: pgmap v283: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-14T09:53:38.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:37 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2179636690' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:38.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.379+0000 7f4006200700 1 -- 172.21.15.78:0/4168899282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f400010ee60 con 0x7f40000737b0 2024-03-14T09:53:38.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.379+0000 7f3ff7e00700 1 -- 172.21.15.78:0/4168899282 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3ff0095bf0 con 0x7f40000737b0 2024-03-14T09:53:38.380 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:53:38.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.382+0000 7f4006200700 1 -- 172.21.15.78:0/4168899282 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3fe40a0b20 msgr2=0x7f3fe40a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:38.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.382+0000 7f4006200700 1 --2- 172.21.15.78:0/4168899282 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3fe40a0b20 0x7f3fe40a2fe0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00b2e0 tx=0x7f3fec005fd0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.382+0000 7f4006200700 1 -- 172.21.15.78:0/4168899282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40000737b0 msgr2=0x7f4000199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:38.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.382+0000 7f4006200700 1 --2- 172.21.15.78:0/4168899282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40000737b0 0x7f4000199300 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000e9a0 tx=0x7f3ff000ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.382+0000 7f4006200700 1 -- 172.21.15.78:0/4168899282 shutdown_connections 2024-03-14T09:53:38.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.382+0000 7f4006200700 1 --2- 172.21.15.78:0/4168899282 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f3fe40a0b20 0x7f3fe40a2fe0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.382+0000 7f4006200700 1 --2- 172.21.15.78:0/4168899282 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4000108c20 0x7f40001a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.382+0000 7f4006200700 1 --2- 172.21.15.78:0/4168899282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40000737b0 0x7f4000199300 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.382+0000 7f4006200700 1 --2- 172.21.15.78:0/4168899282 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4000074d90 0x7f4000199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.382+0000 7f4006200700 1 -- 172.21.15.78:0/4168899282 >> 172.21.15.78:0/4168899282 conn(0x7f40000fc2c0 msgr2=0x7f4000104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:38.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.382+0000 7f4006200700 1 -- 172.21.15.78:0/4168899282 shutdown_connections 2024-03-14T09:53:38.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.382+0000 7f4006200700 1 -- 172.21.15.78:0/4168899282 wait complete. 2024-03-14T09:53:38.402 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-14T09:53:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.596+0000 7f0584e00700 1 -- 172.21.15.78:0/569365233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0580100d40 msgr2=0x7f0580101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.596+0000 7f0584e00700 1 --2- 172.21.15.78:0/569365233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0580100d40 0x7f0580101160 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f05700099c0 tx=0x7f0570009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.596+0000 7f0584e00700 1 -- 172.21.15.78:0/569365233 shutdown_connections 2024-03-14T09:53:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.596+0000 7f0584e00700 1 --2- 172.21.15.78:0/569365233 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f05801028d0 0x7f0580109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.596+0000 7f0584e00700 1 --2- 172.21.15.78:0/569365233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0580100d40 0x7f0580101160 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.596+0000 7f0584e00700 1 --2- 172.21.15.78:0/569365233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0580101f30 0x7f0580102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.596+0000 7f0584e00700 1 -- 172.21.15.78:0/569365233 >> 172.21.15.78:0/569365233 conn(0x7f05800fc2b0 msgr2=0x7f05800fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:38.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.596+0000 7f0584e00700 1 -- 172.21.15.78:0/569365233 shutdown_connections 2024-03-14T09:53:38.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.596+0000 7f0584e00700 1 -- 172.21.15.78:0/569365233 wait complete. 2024-03-14T09:53:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.597+0000 7f0584e00700 1 Processor -- start 2024-03-14T09:53:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.597+0000 7f0584e00700 1 -- start start 2024-03-14T09:53:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.598+0000 7f0584e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0580100d40 0x7f0580199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.598+0000 7f0584e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0580101f30 0x7f0580199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.598+0000 7f0584e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05801028d0 0x7f05801a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.598+0000 7f0584e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05801a0e40 con 0x7f0580101f30 2024-03-14T09:53:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.598+0000 7f0584e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05801a0f80 con 0x7f05801028d0 2024-03-14T09:53:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.598+0000 7f0584e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05801a1110 con 0x7f0580100d40 2024-03-14T09:53:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.598+0000 7f057d400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0580101f30 0x7f0580199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.598+0000 7f057d400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0580101f30 0x7f0580199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52448/0 (socket says 172.21.15.78:52448) 2024-03-14T09:53:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.598+0000 7f057d400700 1 -- 172.21.15.78:0/82415942 learned_addr learned my addr 172.21.15.78:0/82415942 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.598+0000 7f057e800700 1 --2- 172.21.15.78:0/82415942 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05801028d0 0x7f05801a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.598+0000 7f057de00700 1 --2- 172.21.15.78:0/82415942 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0580100d40 0x7f0580199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.598+0000 7f057d400700 1 -- 172.21.15.78:0/82415942 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0580100d40 msgr2=0x7f0580199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.598+0000 7f057d400700 1 --2- 172.21.15.78:0/82415942 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0580100d40 0x7f0580199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.598+0000 7f057d400700 1 -- 172.21.15.78:0/82415942 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05801028d0 msgr2=0x7f05801a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.598+0000 7f057d400700 1 --2- 172.21.15.78:0/82415942 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05801028d0 0x7f05801a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.598+0000 7f057d400700 1 -- 172.21.15.78:0/82415942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05700095f0 con 0x7f0580101f30 2024-03-14T09:53:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.599+0000 7f057d400700 1 --2- 172.21.15.78:0/82415942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0580101f30 0x7f0580199850 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f056c00e9a0 tx=0x7f056c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.599+0000 7f0576a00700 1 -- 172.21.15.78:0/82415942 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c009790 con 0x7f0580101f30 2024-03-14T09:53:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.599+0000 7f0576a00700 1 -- 172.21.15.78:0/82415942 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f056c010430 con 0x7f0580101f30 2024-03-14T09:53:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.599+0000 7f0584e00700 1 -- 172.21.15.78:0/82415942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05801a13f0 con 0x7f0580101f30 2024-03-14T09:53:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.599+0000 7f0576a00700 1 -- 172.21.15.78:0/82415942 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c018680 con 0x7f0580101f30 2024-03-14T09:53:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.599+0000 7f0584e00700 1 -- 172.21.15.78:0/82415942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05801a1900 con 0x7f0580101f30 2024-03-14T09:53:38.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.600+0000 7f0584e00700 1 -- 172.21.15.78:0/82415942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0580067ac0 con 0x7f0580101f30 2024-03-14T09:53:38.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.602+0000 7f0576a00700 1 -- 172.21.15.78:0/82415942 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f056c0105a0 con 0x7f0580101f30 2024-03-14T09:53:38.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.603+0000 7f0576a00700 1 --2- 172.21.15.78:0/82415942 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f05640a0b20 0x7f05640a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:38.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.603+0000 7f0576a00700 1 -- 172.21.15.78:0/82415942 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f056c014070 con 0x7f0580101f30 2024-03-14T09:53:38.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.604+0000 7f057de00700 1 --2- 172.21.15.78:0/82415942 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f05640a0b20 0x7f05640a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:38.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.604+0000 7f057de00700 1 --2- 172.21.15.78:0/82415942 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f05640a0b20 0x7f05640a2fe0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0570003730 tx=0x7f0570000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:38.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.605+0000 7f0576a00700 1 -- 172.21.15.78:0/82415942 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f056c095dc0 con 0x7f0580101f30 2024-03-14T09:53:38.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.813+0000 7f0584e00700 1 -- 172.21.15.78:0/82415942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f05801a1bc0 con 0x7f0580101f30 2024-03-14T09:53:38.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.814+0000 7f0576a00700 1 -- 172.21.15.78:0/82415942 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f056c0260a0 con 0x7f0580101f30 2024-03-14T09:53:38.814 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-14T09:53:38.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.816+0000 7f0584e00700 1 -- 172.21.15.78:0/82415942 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f05640a0b20 msgr2=0x7f05640a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:38.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.816+0000 7f0584e00700 1 --2- 172.21.15.78:0/82415942 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f05640a0b20 0x7f05640a2fe0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0570003730 tx=0x7f0570000bc0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.816+0000 7f0584e00700 1 -- 172.21.15.78:0/82415942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0580101f30 msgr2=0x7f0580199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:38.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.816+0000 7f0584e00700 1 --2- 172.21.15.78:0/82415942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0580101f30 0x7f0580199850 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f056c00e9a0 tx=0x7f056c00ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.816+0000 7f0584e00700 1 -- 172.21.15.78:0/82415942 shutdown_connections 2024-03-14T09:53:38.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.816+0000 7f0584e00700 1 --2- 172.21.15.78:0/82415942 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f05640a0b20 0x7f05640a2fe0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.816+0000 7f0584e00700 1 --2- 172.21.15.78:0/82415942 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05801028d0 0x7f05801a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.816+0000 7f0584e00700 1 --2- 172.21.15.78:0/82415942 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0580100d40 0x7f0580199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.816+0000 7f0584e00700 1 --2- 172.21.15.78:0/82415942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0580101f30 0x7f0580199850 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:38.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.816+0000 7f0584e00700 1 -- 172.21.15.78:0/82415942 >> 172.21.15.78:0/82415942 conn(0x7f05800fc2b0 msgr2=0x7f0580107700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:38.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.816+0000 7f0584e00700 1 -- 172.21.15.78:0/82415942 shutdown_connections 2024-03-14T09:53:38.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:38.816+0000 7f0584e00700 1 -- 172.21.15.78:0/82415942 wait complete. 2024-03-14T09:53:38.836 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-14T09:53:39.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.032+0000 7f5f0c800700 1 -- 172.21.15.78:0/3457113401 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f041028e0 msgr2=0x7f5f04109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:39.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.032+0000 7f5f0c800700 1 --2- 172.21.15.78:0/3457113401 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f041028e0 0x7f5f04109190 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5f0000b0a0 tx=0x7f5f0000b3b0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.032+0000 7f5f0c800700 1 -- 172.21.15.78:0/3457113401 shutdown_connections 2024-03-14T09:53:39.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.032+0000 7f5f0c800700 1 --2- 172.21.15.78:0/3457113401 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f041028e0 0x7f5f04109190 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.032+0000 7f5f0c800700 1 --2- 172.21.15.78:0/3457113401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f04100d50 0x7f5f04101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.032+0000 7f5f0c800700 1 --2- 172.21.15.78:0/3457113401 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f04101f40 0x7f5f041023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.032+0000 7f5f0c800700 1 -- 172.21.15.78:0/3457113401 >> 172.21.15.78:0/3457113401 conn(0x7f5f040fc2c0 msgr2=0x7f5f040fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:39.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.032+0000 7f5f0c800700 1 -- 172.21.15.78:0/3457113401 shutdown_connections 2024-03-14T09:53:39.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.032+0000 7f5f0c800700 1 -- 172.21.15.78:0/3457113401 wait complete. 2024-03-14T09:53:39.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.033+0000 7f5f0c800700 1 Processor -- start 2024-03-14T09:53:39.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.033+0000 7f5f0c800700 1 -- start start 2024-03-14T09:53:39.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.034+0000 7f5f0c800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f04100d50 0x7f5f04074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:39.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.034+0000 7f5f0c800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f04101f40 0x7f5f040731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:39.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.034+0000 7f5f0c800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f041028e0 0x7f5f040736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:39.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.034+0000 7f5f0c800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f04073e80 con 0x7f5f04101f40 2024-03-14T09:53:39.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.034+0000 7f5f0c800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f04073ff0 con 0x7f5f041028e0 2024-03-14T09:53:39.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.034+0000 7f5f0c800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f041a1270 con 0x7f5f04100d50 2024-03-14T09:53:39.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.034+0000 7f5f09400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f04101f40 0x7f5f040731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:39.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.034+0000 7f5f09e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f04100d50 0x7f5f04074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:39.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.034+0000 7f5f09400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f04101f40 0x7f5f040731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52458/0 (socket says 172.21.15.78:52458) 2024-03-14T09:53:39.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.034+0000 7f5f09400700 1 -- 172.21.15.78:0/4065814202 learned_addr learned my addr 172.21.15.78:0/4065814202 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:39.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.034+0000 7f5f0a800700 1 --2- 172.21.15.78:0/4065814202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f041028e0 0x7f5f040736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:39.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.034+0000 7f5f09400700 1 -- 172.21.15.78:0/4065814202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f04100d50 msgr2=0x7f5f04074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:39.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.034+0000 7f5f09400700 1 --2- 172.21.15.78:0/4065814202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f04100d50 0x7f5f04074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.034+0000 7f5f09400700 1 -- 172.21.15.78:0/4065814202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f041028e0 msgr2=0x7f5f040736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:39.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.034+0000 7f5f09400700 1 --2- 172.21.15.78:0/4065814202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f041028e0 0x7f5f040736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.034+0000 7f5f09400700 1 -- 172.21.15.78:0/4065814202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f00009d00 con 0x7f5f04101f40 2024-03-14T09:53:39.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.035+0000 7f5f09400700 1 --2- 172.21.15.78:0/4065814202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f04101f40 0x7f5f040731a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5ef400d950 tx=0x7f5ef400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:39.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.035+0000 7f5efea00700 1 -- 172.21.15.78:0/4065814202 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef400f890 con 0x7f5f04101f40 2024-03-14T09:53:39.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.035+0000 7f5efea00700 1 -- 172.21.15.78:0/4065814202 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ef4010460 con 0x7f5f04101f40 2024-03-14T09:53:39.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.035+0000 7f5efea00700 1 -- 172.21.15.78:0/4065814202 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef40195c0 con 0x7f5f04101f40 2024-03-14T09:53:39.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.035+0000 7f5f0c800700 1 -- 172.21.15.78:0/4065814202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f041a1500 con 0x7f5f04101f40 2024-03-14T09:53:39.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.035+0000 7f5f0c800700 1 -- 172.21.15.78:0/4065814202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f041a19c0 con 0x7f5f04101f40 2024-03-14T09:53:39.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.036+0000 7f5f0c800700 1 -- 172.21.15.78:0/4065814202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f04108100 con 0x7f5f04101f40 2024-03-14T09:53:39.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.038+0000 7f5efea00700 1 -- 172.21.15.78:0/4065814202 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f5ef40108d0 con 0x7f5f04101f40 2024-03-14T09:53:39.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.040+0000 7f5efea00700 1 --2- 172.21.15.78:0/4065814202 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5eec0a0b20 0x7f5eec0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:39.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.041+0000 7f5efea00700 1 -- 172.21.15.78:0/4065814202 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5ef40cc2e0 con 0x7f5f04101f40 2024-03-14T09:53:39.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.041+0000 7f5f09e00700 1 --2- 172.21.15.78:0/4065814202 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5eec0a0b20 0x7f5eec0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:39.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.041+0000 7f5f09e00700 1 --2- 172.21.15.78:0/4065814202 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5eec0a0b20 0x7f5eec0a2fe0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5f04101da0 tx=0x7f5ef8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:39.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.045+0000 7f5efea00700 1 -- 172.21.15.78:0/4065814202 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5ef40957e0 con 0x7f5f04101f40 2024-03-14T09:53:39.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:38 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2426481561' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:39.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:38 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/4168899282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:39.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:38 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2426481561' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:39.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:38 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/4168899282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:39.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.249+0000 7f5f0c800700 1 -- 172.21.15.78:0/4065814202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5f0410cc60 con 0x7f5f04101f40 2024-03-14T09:53:39.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.250+0000 7f5efea00700 1 -- 172.21.15.78:0/4065814202 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5ef4098e00 con 0x7f5f04101f40 2024-03-14T09:53:39.251 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:53:39.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.252+0000 7f5f0c800700 1 -- 172.21.15.78:0/4065814202 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5eec0a0b20 msgr2=0x7f5eec0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:39.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.252+0000 7f5f0c800700 1 --2- 172.21.15.78:0/4065814202 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5eec0a0b20 0x7f5eec0a2fe0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5f04101da0 tx=0x7f5ef8009260 comp rx=0 tx=0).stop 2024-03-14T09:53:39.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.252+0000 7f5f0c800700 1 -- 172.21.15.78:0/4065814202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f04101f40 msgr2=0x7f5f040731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:39.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.252+0000 7f5f0c800700 1 --2- 172.21.15.78:0/4065814202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f04101f40 0x7f5f040731a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5ef400d950 tx=0x7f5ef400dc60 comp rx=0 tx=0).stop 2024-03-14T09:53:39.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.252+0000 7f5f0c800700 1 -- 172.21.15.78:0/4065814202 shutdown_connections 2024-03-14T09:53:39.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.252+0000 7f5f0c800700 1 --2- 172.21.15.78:0/4065814202 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f5eec0a0b20 0x7f5eec0a2fe0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.252+0000 7f5f0c800700 1 --2- 172.21.15.78:0/4065814202 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f041028e0 0x7f5f040736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.252+0000 7f5f0c800700 1 --2- 172.21.15.78:0/4065814202 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f04100d50 0x7f5f04074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.252+0000 7f5f0c800700 1 --2- 172.21.15.78:0/4065814202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f04101f40 0x7f5f040731a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.252+0000 7f5f0c800700 1 -- 172.21.15.78:0/4065814202 >> 172.21.15.78:0/4065814202 conn(0x7f5f040fc2c0 msgr2=0x7f5f04105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:39.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.252+0000 7f5f0c800700 1 -- 172.21.15.78:0/4065814202 shutdown_connections 2024-03-14T09:53:39.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.253+0000 7f5f0c800700 1 -- 172.21.15.78:0/4065814202 wait complete. 2024-03-14T09:53:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:38 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2426481561' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:38 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/4168899282' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:39.272 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:53:39.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.464+0000 7ff69cc00700 1 -- 172.21.15.78:0/4038600934 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff698073810 msgr2=0x7ff698073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:39.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.464+0000 7ff69cc00700 1 --2- 172.21.15.78:0/4038600934 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff698073810 0x7ff698073c90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff684009990 tx=0x7ff684009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.464+0000 7ff69cc00700 1 -- 172.21.15.78:0/4038600934 shutdown_connections 2024-03-14T09:53:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.464+0000 7ff69cc00700 1 --2- 172.21.15.78:0/4038600934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff698108b10 0x7ff69810afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.464+0000 7ff69cc00700 1 --2- 172.21.15.78:0/4038600934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff698074df0 0x7ff698073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.464+0000 7ff69cc00700 1 --2- 172.21.15.78:0/4038600934 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff698073810 0x7ff698073c90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.464+0000 7ff69cc00700 1 -- 172.21.15.78:0/4038600934 >> 172.21.15.78:0/4038600934 conn(0x7ff6980fc1c0 msgr2=0x7ff6980fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.464+0000 7ff69cc00700 1 -- 172.21.15.78:0/4038600934 shutdown_connections 2024-03-14T09:53:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.465+0000 7ff69cc00700 1 -- 172.21.15.78:0/4038600934 wait complete. 2024-03-14T09:53:39.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.465+0000 7ff69cc00700 1 Processor -- start 2024-03-14T09:53:39.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.466+0000 7ff69cc00700 1 -- start start 2024-03-14T09:53:39.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.466+0000 7ff69cc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff698073810 0x7ff698199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:39.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.466+0000 7ff69cc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff698074df0 0x7ff698199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:39.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.466+0000 7ff69cc00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff698108b10 0x7ff6981a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:39.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.466+0000 7ff69cc00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6981a0d20 con 0x7ff698073810 2024-03-14T09:53:39.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.466+0000 7ff69cc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6981a0e60 con 0x7ff698074df0 2024-03-14T09:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.466+0000 7ff69cc00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6981a0ff0 con 0x7ff698108b10 2024-03-14T09:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.466+0000 7ff695e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff698073810 0x7ff698199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.466+0000 7ff696800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff698108b10 0x7ff6981a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.466+0000 7ff695e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff698073810 0x7ff698199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52472/0 (socket says 172.21.15.78:52472) 2024-03-14T09:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.466+0000 7ff695e00700 1 -- 172.21.15.78:0/1900454867 learned_addr learned my addr 172.21.15.78:0/1900454867 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.466+0000 7ff695400700 1 --2- 172.21.15.78:0/1900454867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff698074df0 0x7ff698199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.466+0000 7ff696800700 1 -- 172.21.15.78:0/1900454867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff698074df0 msgr2=0x7ff698199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.467+0000 7ff696800700 1 --2- 172.21.15.78:0/1900454867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff698074df0 0x7ff698199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.467+0000 7ff696800700 1 -- 172.21.15.78:0/1900454867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff698073810 msgr2=0x7ff698199240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.467+0000 7ff696800700 1 --2- 172.21.15.78:0/1900454867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff698073810 0x7ff698199240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.467+0000 7ff696800700 1 -- 172.21.15.78:0/1900454867 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff690009c40 con 0x7ff698108b10 2024-03-14T09:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.467+0000 7ff695e00700 1 --2- 172.21.15.78:0/1900454867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff698073810 0x7ff698199240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.467+0000 7ff696800700 1 --2- 172.21.15.78:0/1900454867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff698108b10 0x7ff6981a0750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff69000efb0 tx=0x7ff69000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.467+0000 7ff68ea00700 1 -- 172.21.15.78:0/1900454867 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff690003a00 con 0x7ff698108b10 2024-03-14T09:53:39.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.467+0000 7ff68ea00700 1 -- 172.21.15.78:0/1900454867 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff69000f040 con 0x7ff698108b10 2024-03-14T09:53:39.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.467+0000 7ff69cc00700 1 -- 172.21.15.78:0/1900454867 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6840095f0 con 0x7ff698108b10 2024-03-14T09:53:39.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.467+0000 7ff68ea00700 1 -- 172.21.15.78:0/1900454867 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff690004540 con 0x7ff698108b10 2024-03-14T09:53:39.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.467+0000 7ff69cc00700 1 -- 172.21.15.78:0/1900454867 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6981a1650 con 0x7ff698108b10 2024-03-14T09:53:39.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.468+0000 7ff69cc00700 1 -- 172.21.15.78:0/1900454867 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6981078d0 con 0x7ff698108b10 2024-03-14T09:53:39.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.470+0000 7ff68ea00700 1 -- 172.21.15.78:0/1900454867 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7ff690003bf0 con 0x7ff698108b10 2024-03-14T09:53:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.471+0000 7ff68ea00700 1 --2- 172.21.15.78:0/1900454867 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff67c0a0b20 0x7ff67c0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:39.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.472+0000 7ff68ea00700 1 -- 172.21.15.78:0/1900454867 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff6900cc3a0 con 0x7ff698108b10 2024-03-14T09:53:39.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.472+0000 7ff695e00700 1 --2- 172.21.15.78:0/1900454867 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff67c0a0b20 0x7ff67c0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:39.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.472+0000 7ff695e00700 1 --2- 172.21.15.78:0/1900454867 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff67c0a0b20 0x7ff67c0a2fe0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff688011550 tx=0x7ff688009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:39.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.474+0000 7ff68ea00700 1 -- 172.21.15.78:0/1900454867 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff690095930 con 0x7ff698108b10 2024-03-14T09:53:39.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.724+0000 7ff69cc00700 1 -- 172.21.15.78:0/1900454867 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff69810ece0 con 0x7ff698108b10 2024-03-14T09:53:39.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.725+0000 7ff68ea00700 1 -- 172.21.15.78:0/1900454867 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10870 (secure 0 0 0) 0x7ff690098f50 con 0x7ff698108b10 2024-03-14T09:53:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:39.726 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:52:30.715484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:53:39.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.727+0000 7ff69cc00700 1 -- 172.21.15.78:0/1900454867 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff67c0a0b20 msgr2=0x7ff67c0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:39.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.727+0000 7ff69cc00700 1 --2- 172.21.15.78:0/1900454867 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff67c0a0b20 0x7ff67c0a2fe0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff688011550 tx=0x7ff688009260 comp rx=0 tx=0).stop 2024-03-14T09:53:39.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.727+0000 7ff69cc00700 1 -- 172.21.15.78:0/1900454867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff698108b10 msgr2=0x7ff6981a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:39.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.727+0000 7ff69cc00700 1 --2- 172.21.15.78:0/1900454867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff698108b10 0x7ff6981a0750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff69000efb0 tx=0x7ff69000c3a0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.728+0000 7ff69cc00700 1 -- 172.21.15.78:0/1900454867 shutdown_connections 2024-03-14T09:53:39.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.728+0000 7ff69cc00700 1 --2- 172.21.15.78:0/1900454867 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff698108b10 0x7ff6981a0750 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.728+0000 7ff69cc00700 1 --2- 172.21.15.78:0/1900454867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff698073810 0x7ff698199240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.728+0000 7ff69cc00700 1 --2- 172.21.15.78:0/1900454867 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff67c0a0b20 0x7ff67c0a2fe0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.728+0000 7ff69cc00700 1 --2- 172.21.15.78:0/1900454867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff698074df0 0x7ff698199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.728+0000 7ff69cc00700 1 -- 172.21.15.78:0/1900454867 >> 172.21.15.78:0/1900454867 conn(0x7ff6980fc1c0 msgr2=0x7ff698104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:39.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.728+0000 7ff69cc00700 1 -- 172.21.15.78:0/1900454867 shutdown_connections 2024-03-14T09:53:39.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.728+0000 7ff69cc00700 1 -- 172.21.15.78:0/1900454867 wait complete. 2024-03-14T09:53:39.729 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2024-03-14T09:53:39.747 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:53:39.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.942+0000 7ffa30a00700 1 -- 172.21.15.78:0/4223040784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa2c101e40 msgr2=0x7ffa2c1022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.942+0000 7ffa30a00700 1 --2- 172.21.15.78:0/4223040784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa2c101e40 0x7ffa2c1022a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffa180099c0 tx=0x7ffa18009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.942+0000 7ffa30a00700 1 -- 172.21.15.78:0/4223040784 shutdown_connections 2024-03-14T09:53:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.942+0000 7ffa30a00700 1 --2- 172.21.15.78:0/4223040784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa2c100c50 0x7ffa2c101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.942+0000 7ffa30a00700 1 --2- 172.21.15.78:0/4223040784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa2c1027e0 0x7ffa2c109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.942+0000 7ffa30a00700 1 --2- 172.21.15.78:0/4223040784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa2c101e40 0x7ffa2c1022a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.942+0000 7ffa30a00700 1 -- 172.21.15.78:0/4223040784 >> 172.21.15.78:0/4223040784 conn(0x7ffa2c0fc1c0 msgr2=0x7ffa2c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.943+0000 7ffa30a00700 1 -- 172.21.15.78:0/4223040784 shutdown_connections 2024-03-14T09:53:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.943+0000 7ffa30a00700 1 -- 172.21.15.78:0/4223040784 wait complete. 2024-03-14T09:53:39.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.943+0000 7ffa30a00700 1 Processor -- start 2024-03-14T09:53:39.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.944+0000 7ffa30a00700 1 -- start start 2024-03-14T09:53:39.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.944+0000 7ffa30a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa2c100c50 0x7ffa2c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:39.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.944+0000 7ffa30a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa2c101e40 0x7ffa2c0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:39.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.944+0000 7ffa30a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa2c1027e0 0x7ffa2c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:39.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.944+0000 7ffa30a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c073e80 con 0x7ffa2c100c50 2024-03-14T09:53:39.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.944+0000 7ffa30a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c073ff0 con 0x7ffa2c1027e0 2024-03-14T09:53:39.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.944+0000 7ffa30a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c1a1040 con 0x7ffa2c101e40 2024-03-14T09:53:39.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.944+0000 7ffa29e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa2c100c50 0x7ffa2c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:39.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.944+0000 7ffa29e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa2c100c50 0x7ffa2c074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52488/0 (socket says 172.21.15.78:52488) 2024-03-14T09:53:39.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.944+0000 7ffa29e00700 1 -- 172.21.15.78:0/4134823391 learned_addr learned my addr 172.21.15.78:0/4134823391 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:39.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.944+0000 7ffa29400700 1 --2- 172.21.15.78:0/4134823391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa2c101e40 0x7ffa2c0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:39.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.944+0000 7ffa2a800700 1 --2- 172.21.15.78:0/4134823391 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa2c1027e0 0x7ffa2c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:39.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.945+0000 7ffa29e00700 1 -- 172.21.15.78:0/4134823391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa2c101e40 msgr2=0x7ffa2c0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:39.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.945+0000 7ffa29e00700 1 --2- 172.21.15.78:0/4134823391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa2c101e40 0x7ffa2c0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.945+0000 7ffa29e00700 1 -- 172.21.15.78:0/4134823391 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa2c1027e0 msgr2=0x7ffa2c0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:39.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.945+0000 7ffa29e00700 1 --2- 172.21.15.78:0/4134823391 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa2c1027e0 0x7ffa2c0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:39.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.945+0000 7ffa29e00700 1 -- 172.21.15.78:0/4134823391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa180095f0 con 0x7ffa2c100c50 2024-03-14T09:53:39.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.945+0000 7ffa29e00700 1 --2- 172.21.15.78:0/4134823391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa2c100c50 0x7ffa2c074b10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00e970 tx=0x7ffa1c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:39.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.945+0000 7ffa22a00700 1 -- 172.21.15.78:0/4134823391 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c009790 con 0x7ffa2c100c50 2024-03-14T09:53:39.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.945+0000 7ffa22a00700 1 -- 172.21.15.78:0/4134823391 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa1c00c780 con 0x7ffa2c100c50 2024-03-14T09:53:39.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.945+0000 7ffa30a00700 1 -- 172.21.15.78:0/4134823391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2c1a12d0 con 0x7ffa2c100c50 2024-03-14T09:53:39.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.945+0000 7ffa22a00700 1 -- 172.21.15.78:0/4134823391 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c003be0 con 0x7ffa2c100c50 2024-03-14T09:53:39.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.945+0000 7ffa30a00700 1 -- 172.21.15.78:0/4134823391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2c1a17e0 con 0x7ffa2c100c50 2024-03-14T09:53:39.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.947+0000 7ffa30a00700 1 -- 172.21.15.78:0/4134823391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2c1080d0 con 0x7ffa2c100c50 2024-03-14T09:53:39.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.948+0000 7ffa22a00700 1 -- 172.21.15.78:0/4134823391 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7ffa1c00cbf0 con 0x7ffa2c100c50 2024-03-14T09:53:39.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.950+0000 7ffa22a00700 1 --2- 172.21.15.78:0/4134823391 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffa100a0b20 0x7ffa100a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:39.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.950+0000 7ffa22a00700 1 -- 172.21.15.78:0/4134823391 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ffa1c014070 con 0x7ffa2c100c50 2024-03-14T09:53:39.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.950+0000 7ffa29400700 1 --2- 172.21.15.78:0/4134823391 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffa100a0b20 0x7ffa100a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.951+0000 7ffa29400700 1 --2- 172.21.15.78:0/4134823391 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffa100a0b20 0x7ffa100a2fe0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffa18003730 tx=0x7ffa1800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:39.954+0000 7ffa22a00700 1 -- 172.21.15.78:0/4134823391 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ffa1c095c00 con 0x7ffa2c100c50 2024-03-14T09:53:40.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:39 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/82415942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:40.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:39 smithi078 ceph-mon[38143]: pgmap v284: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-14T09:53:40.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:39 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/4065814202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:40.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:39 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1900454867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:40.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.180+0000 7ffa30a00700 1 -- 172.21.15.78:0/4134823391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa2c0679e0 con 0x7ffa2c100c50 2024-03-14T09:53:40.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.182+0000 7ffa22a00700 1 -- 172.21.15.78:0/4134823391 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+24453 (secure 0 0 0) 0x7ffa1c099220 con 0x7ffa2c100c50 2024-03-14T09:53:40.183 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:40.183 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":85,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:52:42.134030+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:53:40.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.184+0000 7ffa30a00700 1 -- 172.21.15.78:0/4134823391 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffa100a0b20 msgr2=0x7ffa100a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:40.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.184+0000 7ffa30a00700 1 --2- 172.21.15.78:0/4134823391 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffa100a0b20 0x7ffa100a2fe0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffa18003730 tx=0x7ffa1800b540 comp rx=0 tx=0).stop 2024-03-14T09:53:40.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.184+0000 7ffa30a00700 1 -- 172.21.15.78:0/4134823391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa2c100c50 msgr2=0x7ffa2c074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:40.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.184+0000 7ffa30a00700 1 --2- 172.21.15.78:0/4134823391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa2c100c50 0x7ffa2c074b10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00e970 tx=0x7ffa1c00ec80 comp rx=0 tx=0).stop 2024-03-14T09:53:40.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.184+0000 7ffa30a00700 1 -- 172.21.15.78:0/4134823391 shutdown_connections 2024-03-14T09:53:40.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.184+0000 7ffa30a00700 1 --2- 172.21.15.78:0/4134823391 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffa100a0b20 0x7ffa100a2fe0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.184+0000 7ffa30a00700 1 --2- 172.21.15.78:0/4134823391 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa2c1027e0 0x7ffa2c0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.184+0000 7ffa30a00700 1 --2- 172.21.15.78:0/4134823391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa2c100c50 0x7ffa2c074b10 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.184+0000 7ffa30a00700 1 --2- 172.21.15.78:0/4134823391 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa2c101e40 0x7ffa2c0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.184+0000 7ffa30a00700 1 -- 172.21.15.78:0/4134823391 >> 172.21.15.78:0/4134823391 conn(0x7ffa2c0fc1c0 msgr2=0x7ffa2c105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:40.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.184+0000 7ffa30a00700 1 -- 172.21.15.78:0/4134823391 shutdown_connections 2024-03-14T09:53:40.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.185+0000 7ffa30a00700 1 -- 172.21.15.78:0/4134823391 wait complete. 2024-03-14T09:53:40.204 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T09:48:37.618450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-14T09:52:03.938666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-14T09:52:05.149985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-14T09:52:08.227698+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-14T09:53:40.205 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T09:53:40.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:39 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/82415942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:40.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:39 smithi152 ceph-mon[48279]: pgmap v284: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-14T09:53:40.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:39 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/4065814202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:40.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:39 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1900454867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:40.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:39 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/82415942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:40.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:39 smithi132 ceph-mon[47959]: pgmap v284: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-14T09:53:40.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:39 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/4065814202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:40.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:39 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1900454867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:40.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.399+0000 7f06d8a00700 1 -- 172.21.15.78:0/3677720966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06d4100c80 msgr2=0x7f06d41010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:40.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.399+0000 7f06d8a00700 1 --2- 172.21.15.78:0/3677720966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06d4100c80 0x7f06d41010a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f06c4009920 tx=0x7f06c4009c30 comp rx=0 tx=0).stop 2024-03-14T09:53:40.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.399+0000 7f06d8a00700 1 -- 172.21.15.78:0/3677720966 shutdown_connections 2024-03-14T09:53:40.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.399+0000 7f06d8a00700 1 --2- 172.21.15.78:0/3677720966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06d4100c80 0x7f06d41010a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.399+0000 7f06d8a00700 1 --2- 172.21.15.78:0/3677720966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d4102810 0x7f06d41090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.399+0000 7f06d8a00700 1 --2- 172.21.15.78:0/3677720966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d4101e70 0x7f06d41022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.399+0000 7f06d8a00700 1 -- 172.21.15.78:0/3677720966 >> 172.21.15.78:0/3677720966 conn(0x7f06d40fc1f0 msgr2=0x7f06d40fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:40.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.399+0000 7f06d8a00700 1 -- 172.21.15.78:0/3677720966 shutdown_connections 2024-03-14T09:53:40.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.399+0000 7f06d8a00700 1 -- 172.21.15.78:0/3677720966 wait complete. 2024-03-14T09:53:40.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.400+0000 7f06d8a00700 1 Processor -- start 2024-03-14T09:53:40.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.401+0000 7f06d8a00700 1 -- start start 2024-03-14T09:53:40.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.401+0000 7f06d8a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d4100c80 0x7f06d4199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:40.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.401+0000 7f06d8a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06d4101e70 0x7f06d4199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:40.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.401+0000 7f06d8a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d4102810 0x7f06d41a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:40.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.401+0000 7f06d8a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d41a0e60 con 0x7f06d4100c80 2024-03-14T09:53:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.401+0000 7f06d8a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d41a0fa0 con 0x7f06d4101e70 2024-03-14T09:53:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.401+0000 7f06d8a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d41a1130 con 0x7f06d4102810 2024-03-14T09:53:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.401+0000 7f06d1e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d4100c80 0x7f06d4199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.401+0000 7f06d1e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d4100c80 0x7f06d4199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52496/0 (socket says 172.21.15.78:52496) 2024-03-14T09:53:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.401+0000 7f06d1e00700 1 -- 172.21.15.78:0/838796323 learned_addr learned my addr 172.21.15.78:0/838796323 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.401+0000 7f06d1400700 1 --2- 172.21.15.78:0/838796323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06d4101e70 0x7f06d4199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.401+0000 7f06d2800700 1 --2- 172.21.15.78:0/838796323 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d4102810 0x7f06d41a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.402+0000 7f06d1e00700 1 -- 172.21.15.78:0/838796323 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d4102810 msgr2=0x7f06d41a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.402+0000 7f06d1e00700 1 --2- 172.21.15.78:0/838796323 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d4102810 0x7f06d41a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.402+0000 7f06d1e00700 1 -- 172.21.15.78:0/838796323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06d4101e70 msgr2=0x7f06d4199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.402+0000 7f06d1e00700 1 --2- 172.21.15.78:0/838796323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06d4101e70 0x7f06d4199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.402+0000 7f06d1e00700 1 -- 172.21.15.78:0/838796323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06cc009d00 con 0x7f06d4100c80 2024-03-14T09:53:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.402+0000 7f06d1400700 1 --2- 172.21.15.78:0/838796323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06d4101e70 0x7f06d4199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.402+0000 7f06d1e00700 1 --2- 172.21.15.78:0/838796323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d4100c80 0x7f06d4199330 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f06c4009230 tx=0x7f06c400bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:40.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.402+0000 7f06caa00700 1 -- 172.21.15.78:0/838796323 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c401c070 con 0x7f06d4100c80 2024-03-14T09:53:40.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.402+0000 7f06caa00700 1 -- 172.21.15.78:0/838796323 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f06c400bb20 con 0x7f06d4100c80 2024-03-14T09:53:40.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.402+0000 7f06d8a00700 1 -- 172.21.15.78:0/838796323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c4009520 con 0x7f06d4100c80 2024-03-14T09:53:40.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.402+0000 7f06caa00700 1 -- 172.21.15.78:0/838796323 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c40049c0 con 0x7f06d4100c80 2024-03-14T09:53:40.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.402+0000 7f06d8a00700 1 -- 172.21.15.78:0/838796323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d41a15f0 con 0x7f06d4100c80 2024-03-14T09:53:40.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.403+0000 7f06d8a00700 1 -- 172.21.15.78:0/838796323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d40679e0 con 0x7f06d4100c80 2024-03-14T09:53:40.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.405+0000 7f06caa00700 1 -- 172.21.15.78:0/838796323 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f06c4020070 con 0x7f06d4100c80 2024-03-14T09:53:40.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.406+0000 7f06caa00700 1 --2- 172.21.15.78:0/838796323 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f06b80a0b10 0x7f06b80a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:40.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.406+0000 7f06caa00700 1 -- 172.21.15.78:0/838796323 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f06c40cd5a0 con 0x7f06d4100c80 2024-03-14T09:53:40.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.407+0000 7f06d1400700 1 --2- 172.21.15.78:0/838796323 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f06b80a0b10 0x7f06b80a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:40.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.407+0000 7f06d1400700 1 --2- 172.21.15.78:0/838796323 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f06b80a0b10 0x7f06b80a2fd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f06c0011490 tx=0x7f06c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:40.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.409+0000 7f06caa00700 1 -- 172.21.15.78:0/838796323 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f06c4096b30 con 0x7f06d4100c80 2024-03-14T09:53:40.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.612+0000 7f06d8a00700 1 -- 172.21.15.78:0/838796323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f06d410ca90 con 0x7f06d4100c80 2024-03-14T09:53:40.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.613+0000 7f06caa00700 1 -- 172.21.15.78:0/838796323 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f06c409a150 con 0x7f06d4100c80 2024-03-14T09:53:40.614 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-14T09:53:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.615+0000 7f06d8a00700 1 -- 172.21.15.78:0/838796323 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f06b80a0b10 msgr2=0x7f06b80a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.615+0000 7f06d8a00700 1 --2- 172.21.15.78:0/838796323 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f06b80a0b10 0x7f06b80a2fd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f06c0011490 tx=0x7f06c0009260 comp rx=0 tx=0).stop 2024-03-14T09:53:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.615+0000 7f06d8a00700 1 -- 172.21.15.78:0/838796323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d4100c80 msgr2=0x7f06d4199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.615+0000 7f06d8a00700 1 --2- 172.21.15.78:0/838796323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d4100c80 0x7f06d4199330 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f06c4009230 tx=0x7f06c400bef0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.615+0000 7f06d8a00700 1 -- 172.21.15.78:0/838796323 shutdown_connections 2024-03-14T09:53:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.615+0000 7f06d8a00700 1 --2- 172.21.15.78:0/838796323 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f06d4102810 0x7f06d41a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.615+0000 7f06d8a00700 1 --2- 172.21.15.78:0/838796323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06d4100c80 0x7f06d4199330 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.615+0000 7f06d8a00700 1 --2- 172.21.15.78:0/838796323 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f06b80a0b10 0x7f06b80a2fd0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.615+0000 7f06d8a00700 1 --2- 172.21.15.78:0/838796323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06d4101e70 0x7f06d4199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.615+0000 7f06d8a00700 1 -- 172.21.15.78:0/838796323 >> 172.21.15.78:0/838796323 conn(0x7f06d40fc1f0 msgr2=0x7f06d41076e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:40.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.615+0000 7f06d8a00700 1 -- 172.21.15.78:0/838796323 shutdown_connections 2024-03-14T09:53:40.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.615+0000 7f06d8a00700 1 -- 172.21.15.78:0/838796323 wait complete. 2024-03-14T09:53:40.636 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-14T09:53:40.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.832+0000 7f118c800700 1 -- 172.21.15.78:0/4060014400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1184101810 msgr2=0x7f1184103c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:40.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.832+0000 7f118c800700 1 --2- 172.21.15.78:0/4060014400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1184101810 0x7f1184103c00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1178009990 tx=0x7f1178009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.832+0000 7f118c800700 1 -- 172.21.15.78:0/4060014400 shutdown_connections 2024-03-14T09:53:40.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.832+0000 7f118c800700 1 --2- 172.21.15.78:0/4060014400 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1184101810 0x7f1184103c00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.832+0000 7f118c800700 1 --2- 172.21.15.78:0/4060014400 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1184106a70 0x7f1184108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.832+0000 7f118c800700 1 --2- 172.21.15.78:0/4060014400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1184104140 0x7f1184106530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.832+0000 7f118c800700 1 -- 172.21.15.78:0/4060014400 >> 172.21.15.78:0/4060014400 conn(0x7f11840fb1a0 msgr2=0x7f11840fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:40.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.832+0000 7f118c800700 1 -- 172.21.15.78:0/4060014400 shutdown_connections 2024-03-14T09:53:40.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.832+0000 7f118c800700 1 -- 172.21.15.78:0/4060014400 wait complete. 2024-03-14T09:53:40.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.833+0000 7f118c800700 1 Processor -- start 2024-03-14T09:53:40.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.833+0000 7f118c800700 1 -- start start 2024-03-14T09:53:40.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.833+0000 7f118c800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1184101810 0x7f1184101130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:40.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.833+0000 7f118c800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1184104140 0x7f11840ff7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:40.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.833+0000 7f118c800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1184106a70 0x7f11840ffd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:40.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.833+0000 7f118c800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11841004a0 con 0x7f1184106a70 2024-03-14T09:53:40.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.833+0000 7f118c800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1184100610 con 0x7f1184104140 2024-03-14T09:53:40.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.833+0000 7f118c800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11841a1130 con 0x7f1184101810 2024-03-14T09:53:40.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.834+0000 7f1189400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1184104140 0x7f11840ff7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:40.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.834+0000 7f1189e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1184101810 0x7f1184101130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:40.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.834+0000 7f118a800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1184106a70 0x7f11840ffd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:40.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.834+0000 7f1189400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1184104140 0x7f11840ff7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:35726/0 (socket says 172.21.15.78:35726) 2024-03-14T09:53:40.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.834+0000 7f1189400700 1 -- 172.21.15.78:0/1730986116 learned_addr learned my addr 172.21.15.78:0/1730986116 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:40.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.834+0000 7f1189400700 1 -- 172.21.15.78:0/1730986116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1184101810 msgr2=0x7f1184101130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:40.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.834+0000 7f1189400700 1 --2- 172.21.15.78:0/1730986116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1184101810 0x7f1184101130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.834+0000 7f1189400700 1 -- 172.21.15.78:0/1730986116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1184106a70 msgr2=0x7f11840ffd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:40.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.834+0000 7f1189400700 1 --2- 172.21.15.78:0/1730986116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1184106a70 0x7f11840ffd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:40.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.834+0000 7f1189400700 1 -- 172.21.15.78:0/1730986116 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11780095f0 con 0x7f1184104140 2024-03-14T09:53:40.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.834+0000 7f1189400700 1 --2- 172.21.15.78:0/1730986116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1184104140 0x7f11840ff7c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f117400e9a0 tx=0x7f117400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:40.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.834+0000 7f117ea00700 1 -- 172.21.15.78:0/1730986116 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174009790 con 0x7f1184104140 2024-03-14T09:53:40.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.835+0000 7f117ea00700 1 -- 172.21.15.78:0/1730986116 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f117400ce70 con 0x7f1184104140 2024-03-14T09:53:40.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.835+0000 7f117ea00700 1 -- 172.21.15.78:0/1730986116 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174018420 con 0x7f1184104140 2024-03-14T09:53:40.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.835+0000 7f118c800700 1 -- 172.21.15.78:0/1730986116 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11841a13c0 con 0x7f1184104140 2024-03-14T09:53:40.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.835+0000 7f118c800700 1 -- 172.21.15.78:0/1730986116 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11841a1880 con 0x7f1184104140 2024-03-14T09:53:40.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.836+0000 7f118c800700 1 -- 172.21.15.78:0/1730986116 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11840fcd20 con 0x7f1184104140 2024-03-14T09:53:40.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.838+0000 7f117ea00700 1 -- 172.21.15.78:0/1730986116 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f11740106f0 con 0x7f1184104140 2024-03-14T09:53:40.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.839+0000 7f117ea00700 1 --2- 172.21.15.78:0/1730986116 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f116c0a0b20 0x7f116c0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:40.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.840+0000 7f117ea00700 1 -- 172.21.15.78:0/1730986116 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1174014070 con 0x7f1184104140 2024-03-14T09:53:40.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.840+0000 7f1189e00700 1 --2- 172.21.15.78:0/1730986116 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f116c0a0b20 0x7f116c0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:40.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.840+0000 7f1189e00700 1 --2- 172.21.15.78:0/1730986116 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f116c0a0b20 0x7f116c0a2fe0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f117800b2e0 tx=0x7f1178005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:40.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:40.841+0000 7f117ea00700 1 -- 172.21.15.78:0/1730986116 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1174096050 con 0x7f1184104140 2024-03-14T09:53:41.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.048+0000 7f118c800700 1 -- 172.21.15.78:0/1730986116 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f118410c980 con 0x7f1184104140 2024-03-14T09:53:41.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.049+0000 7f117ea00700 1 -- 172.21.15.78:0/1730986116 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1174099670 con 0x7f1184104140 2024-03-14T09:53:41.050 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:53:41.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.051+0000 7f118c800700 1 -- 172.21.15.78:0/1730986116 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f116c0a0b20 msgr2=0x7f116c0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:41.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.051+0000 7f118c800700 1 --2- 172.21.15.78:0/1730986116 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f116c0a0b20 0x7f116c0a2fe0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f117800b2e0 tx=0x7f1178005fb0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.051+0000 7f118c800700 1 -- 172.21.15.78:0/1730986116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1184104140 msgr2=0x7f11840ff7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.051+0000 7f118c800700 1 --2- 172.21.15.78:0/1730986116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1184104140 0x7f11840ff7c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f117400e9a0 tx=0x7f117400ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.051+0000 7f118c800700 1 -- 172.21.15.78:0/1730986116 shutdown_connections 2024-03-14T09:53:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.051+0000 7f118c800700 1 --2- 172.21.15.78:0/1730986116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1184106a70 0x7f11840ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.051+0000 7f118c800700 1 --2- 172.21.15.78:0/1730986116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1184101810 0x7f1184101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.051+0000 7f118c800700 1 --2- 172.21.15.78:0/1730986116 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f116c0a0b20 0x7f116c0a2fe0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.051+0000 7f118c800700 1 --2- 172.21.15.78:0/1730986116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1184104140 0x7f11840ff7c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.051+0000 7f118c800700 1 -- 172.21.15.78:0/1730986116 >> 172.21.15.78:0/1730986116 conn(0x7f11840fb1a0 msgr2=0x7f11841036a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:41.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.051+0000 7f118c800700 1 -- 172.21.15.78:0/1730986116 shutdown_connections 2024-03-14T09:53:41.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.051+0000 7f118c800700 1 -- 172.21.15.78:0/1730986116 wait complete. 2024-03-14T09:53:41.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:40 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/4134823391' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:41.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:40 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/838796323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:41.075 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-14T09:53:41.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:40 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/4134823391' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:41.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:40 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/838796323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:41.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:40 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/4134823391' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:41.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:40 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/838796323' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.296+0000 7fe9b6200700 1 -- 172.21.15.78:0/4171214214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b01028e0 msgr2=0x7fe9b0109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.296+0000 7fe9b6200700 1 --2- 172.21.15.78:0/4171214214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b01028e0 0x7fe9b0109190 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800b0d0 tx=0x7fe9a800b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.296+0000 7fe9b6200700 1 -- 172.21.15.78:0/4171214214 shutdown_connections 2024-03-14T09:53:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.296+0000 7fe9b6200700 1 --2- 172.21.15.78:0/4171214214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b01028e0 0x7fe9b0109190 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.296+0000 7fe9b6200700 1 --2- 172.21.15.78:0/4171214214 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9b0100d50 0x7fe9b0101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.296+0000 7fe9b6200700 1 --2- 172.21.15.78:0/4171214214 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9b0101f40 0x7fe9b01023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.296+0000 7fe9b6200700 1 -- 172.21.15.78:0/4171214214 >> 172.21.15.78:0/4171214214 conn(0x7fe9b00fc2c0 msgr2=0x7fe9b00fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.296+0000 7fe9b6200700 1 -- 172.21.15.78:0/4171214214 shutdown_connections 2024-03-14T09:53:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.296+0000 7fe9b6200700 1 -- 172.21.15.78:0/4171214214 wait complete. 2024-03-14T09:53:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.297+0000 7fe9b6200700 1 Processor -- start 2024-03-14T09:53:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.297+0000 7fe9b6200700 1 -- start start 2024-03-14T09:53:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.297+0000 7fe9b6200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9b0100d50 0x7fe9b0199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.297+0000 7fe9b6200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b0101f40 0x7fe9b0199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.297+0000 7fe9b6200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9b01028e0 0x7fe9b01a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.297+0000 7fe9b6200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b01a0f50 con 0x7fe9b0101f40 2024-03-14T09:53:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.297+0000 7fe9b6200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b01a1090 con 0x7fe9b0100d50 2024-03-14T09:53:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.297+0000 7fe9b6200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b01a1220 con 0x7fe9b01028e0 2024-03-14T09:53:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.297+0000 7fe9aea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b0101f40 0x7fe9b0199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.298+0000 7fe9aea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b0101f40 0x7fe9b0199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52542/0 (socket says 172.21.15.78:52542) 2024-03-14T09:53:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.298+0000 7fe9aea00700 1 -- 172.21.15.78:0/397316550 learned_addr learned my addr 172.21.15.78:0/397316550 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:41.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.298+0000 7fe9af400700 1 --2- 172.21.15.78:0/397316550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9b0100d50 0x7fe9b0199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:41.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.298+0000 7fe9afe00700 1 --2- 172.21.15.78:0/397316550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9b01028e0 0x7fe9b01a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:41.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.298+0000 7fe9aea00700 1 -- 172.21.15.78:0/397316550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9b01028e0 msgr2=0x7fe9b01a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:41.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.298+0000 7fe9aea00700 1 --2- 172.21.15.78:0/397316550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9b01028e0 0x7fe9b01a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.298+0000 7fe9aea00700 1 -- 172.21.15.78:0/397316550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9b0100d50 msgr2=0x7fe9b0199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:41.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.298+0000 7fe9aea00700 1 --2- 172.21.15.78:0/397316550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9b0100d50 0x7fe9b0199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.298+0000 7fe9aea00700 1 -- 172.21.15.78:0/397316550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a8009d00 con 0x7fe9b0101f40 2024-03-14T09:53:41.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.298+0000 7fe9aea00700 1 --2- 172.21.15.78:0/397316550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b0101f40 0x7fe9b0199960 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe99c00e9a0 tx=0x7fe99c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:41.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.298+0000 7fe9a7e00700 1 -- 172.21.15.78:0/397316550 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c009790 con 0x7fe9b0101f40 2024-03-14T09:53:41.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.298+0000 7fe9a7e00700 1 -- 172.21.15.78:0/397316550 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe99c010430 con 0x7fe9b0101f40 2024-03-14T09:53:41.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.299+0000 7fe9b6200700 1 -- 172.21.15.78:0/397316550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b01a1500 con 0x7fe9b0101f40 2024-03-14T09:53:41.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.299+0000 7fe9a7e00700 1 -- 172.21.15.78:0/397316550 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c018680 con 0x7fe9b0101f40 2024-03-14T09:53:41.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.299+0000 7fe9b6200700 1 -- 172.21.15.78:0/397316550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b01a1a10 con 0x7fe9b0101f40 2024-03-14T09:53:41.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.300+0000 7fe9b6200700 1 -- 172.21.15.78:0/397316550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b0067ac0 con 0x7fe9b0101f40 2024-03-14T09:53:41.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.301+0000 7fe9a7e00700 1 -- 172.21.15.78:0/397316550 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fe99c0105a0 con 0x7fe9b0101f40 2024-03-14T09:53:41.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.303+0000 7fe9a7e00700 1 --2- 172.21.15.78:0/397316550 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe9940a0b20 0x7fe9940a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:41.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.303+0000 7fe9a7e00700 1 -- 172.21.15.78:0/397316550 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe99c014070 con 0x7fe9b0101f40 2024-03-14T09:53:41.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.303+0000 7fe9af400700 1 --2- 172.21.15.78:0/397316550 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe9940a0b20 0x7fe9940a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:41.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.303+0000 7fe9af400700 1 --2- 172.21.15.78:0/397316550 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe9940a0b20 0x7fe9940a2fe0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0101da0 tx=0x7fe9a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:41.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.306+0000 7fe9a7e00700 1 -- 172.21.15.78:0/397316550 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe99c052960 con 0x7fe9b0101f40 2024-03-14T09:53:41.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.509+0000 7fe9b6200700 1 -- 172.21.15.78:0/397316550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe9b01a1cd0 con 0x7fe9b0101f40 2024-03-14T09:53:41.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.509+0000 7fe9a7e00700 1 -- 172.21.15.78:0/397316550 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe99c095bf0 con 0x7fe9b0101f40 2024-03-14T09:53:41.510 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-14T09:53:41.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.512+0000 7fe9b6200700 1 -- 172.21.15.78:0/397316550 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe9940a0b20 msgr2=0x7fe9940a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:41.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.512+0000 7fe9b6200700 1 --2- 172.21.15.78:0/397316550 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe9940a0b20 0x7fe9940a2fe0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0101da0 tx=0x7fe9a0009260 comp rx=0 tx=0).stop 2024-03-14T09:53:41.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.512+0000 7fe9b6200700 1 -- 172.21.15.78:0/397316550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b0101f40 msgr2=0x7fe9b0199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:41.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.512+0000 7fe9b6200700 1 --2- 172.21.15.78:0/397316550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b0101f40 0x7fe9b0199960 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe99c00e9a0 tx=0x7fe99c00ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.512+0000 7fe9b6200700 1 -- 172.21.15.78:0/397316550 shutdown_connections 2024-03-14T09:53:41.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.512+0000 7fe9b6200700 1 --2- 172.21.15.78:0/397316550 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe9940a0b20 0x7fe9940a2fe0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.512+0000 7fe9b6200700 1 --2- 172.21.15.78:0/397316550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9b0100d50 0x7fe9b0199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.512+0000 7fe9b6200700 1 --2- 172.21.15.78:0/397316550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9b01028e0 0x7fe9b01a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.512+0000 7fe9b6200700 1 --2- 172.21.15.78:0/397316550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b0101f40 0x7fe9b0199960 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.512+0000 7fe9b6200700 1 -- 172.21.15.78:0/397316550 >> 172.21.15.78:0/397316550 conn(0x7fe9b00fc2c0 msgr2=0x7fe9b0072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:41.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.512+0000 7fe9b6200700 1 -- 172.21.15.78:0/397316550 shutdown_connections 2024-03-14T09:53:41.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.512+0000 7fe9b6200700 1 -- 172.21.15.78:0/397316550 wait complete. 2024-03-14T09:53:41.531 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-14T09:53:41.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.727+0000 7fa927400700 1 -- 172.21.15.78:0/2561957987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa920101f30 msgr2=0x7fa920102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:41.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.727+0000 7fa927400700 1 --2- 172.21.15.78:0/2561957987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa920101f30 0x7fa920102390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009990 tx=0x7fa90c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.727+0000 7fa927400700 1 -- 172.21.15.78:0/2561957987 shutdown_connections 2024-03-14T09:53:41.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.727+0000 7fa927400700 1 --2- 172.21.15.78:0/2561957987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa9201028d0 0x7fa920109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.727+0000 7fa927400700 1 --2- 172.21.15.78:0/2561957987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa920100d40 0x7fa920101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.727+0000 7fa927400700 1 --2- 172.21.15.78:0/2561957987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa920101f30 0x7fa920102390 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.727+0000 7fa927400700 1 -- 172.21.15.78:0/2561957987 >> 172.21.15.78:0/2561957987 conn(0x7fa9200fc2b0 msgr2=0x7fa9200fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:41.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.727+0000 7fa927400700 1 -- 172.21.15.78:0/2561957987 shutdown_connections 2024-03-14T09:53:41.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.727+0000 7fa927400700 1 -- 172.21.15.78:0/2561957987 wait complete. 2024-03-14T09:53:41.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.728+0000 7fa927400700 1 Processor -- start 2024-03-14T09:53:41.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.728+0000 7fa927400700 1 -- start start 2024-03-14T09:53:41.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.729+0000 7fa927400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa920100d40 0x7fa920074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:41.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.729+0000 7fa927400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa920101f30 0x7fa9200731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:41.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.729+0000 7fa927400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9201028d0 0x7fa9200736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:41.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.729+0000 7fa927400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa920073e80 con 0x7fa920101f30 2024-03-14T09:53:41.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.729+0000 7fa927400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa920073ff0 con 0x7fa920100d40 2024-03-14T09:53:41.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.729+0000 7fa927400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9201a1100 con 0x7fa9201028d0 2024-03-14T09:53:41.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.729+0000 7fa924a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa920100d40 0x7fa920074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:41.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.729+0000 7fa91fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa920101f30 0x7fa9200731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:41.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.729+0000 7fa925400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9201028d0 0x7fa9200736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.729+0000 7fa924a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa920100d40 0x7fa920074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:35756/0 (socket says 172.21.15.78:35756) 2024-03-14T09:53:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.729+0000 7fa924a00700 1 -- 172.21.15.78:0/2080832632 learned_addr learned my addr 172.21.15.78:0/2080832632 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.730+0000 7fa924a00700 1 -- 172.21.15.78:0/2080832632 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9201028d0 msgr2=0x7fa9200736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.730+0000 7fa924a00700 1 --2- 172.21.15.78:0/2080832632 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9201028d0 0x7fa9200736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.730+0000 7fa924a00700 1 -- 172.21.15.78:0/2080832632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa920101f30 msgr2=0x7fa9200731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.730+0000 7fa924a00700 1 --2- 172.21.15.78:0/2080832632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa920101f30 0x7fa9200731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.730+0000 7fa924a00700 1 -- 172.21.15.78:0/2080832632 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa914009520 con 0x7fa920100d40 2024-03-14T09:53:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.730+0000 7fa924a00700 1 --2- 172.21.15.78:0/2080832632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa920100d40 0x7fa920074b10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa91400ea50 tx=0x7fa91400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.730+0000 7fa91d600700 1 -- 172.21.15.78:0/2080832632 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91400cb70 con 0x7fa920100d40 2024-03-14T09:53:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.730+0000 7fa91d600700 1 -- 172.21.15.78:0/2080832632 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa91400ccd0 con 0x7fa920100d40 2024-03-14T09:53:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.730+0000 7fa91d600700 1 -- 172.21.15.78:0/2080832632 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa914018680 con 0x7fa920100d40 2024-03-14T09:53:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.730+0000 7fa927400700 1 -- 172.21.15.78:0/2080832632 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90c0095f0 con 0x7fa920100d40 2024-03-14T09:53:41.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.730+0000 7fa927400700 1 -- 172.21.15.78:0/2080832632 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9201a16c0 con 0x7fa920100d40 2024-03-14T09:53:41.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.732+0000 7fa927400700 1 -- 172.21.15.78:0/2080832632 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9201081e0 con 0x7fa920100d40 2024-03-14T09:53:41.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.734+0000 7fa91d600700 1 -- 172.21.15.78:0/2080832632 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 17) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fa914010430 con 0x7fa920100d40 2024-03-14T09:53:41.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.735+0000 7fa91d600700 1 --2- 172.21.15.78:0/2080832632 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa9080a0b20 0x7fa9080a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.735+0000 7fa91d600700 1 -- 172.21.15.78:0/2080832632 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa914014070 con 0x7fa920100d40 2024-03-14T09:53:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.736+0000 7fa91fe00700 1 --2- 172.21.15.78:0/2080832632 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa9080a0b20 0x7fa9080a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.736+0000 7fa91fe00700 1 --2- 172.21.15.78:0/2080832632 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa9080a0b20 0x7fa9080a2fe0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa90c00b2e0 tx=0x7fa90c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:41.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.738+0000 7fa91d600700 1 -- 172.21.15.78:0/2080832632 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa914096170 con 0x7fa920100d40 2024-03-14T09:53:41.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.798+0000 7fa91d600700 1 -- 172.21.15.78:0/2080832632 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 18) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fa914096540 con 0x7fa920100d40 2024-03-14T09:53:41.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.958+0000 7fa927400700 1 -- 172.21.15.78:0/2080832632 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa92010c970 con 0x7fa920100d40 2024-03-14T09:53:41.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.958+0000 7fa91d600700 1 -- 172.21.15.78:0/2080832632 <== mon.2 v2:172.21.15.132:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v86) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa914099c00 con 0x7fa920100d40 2024-03-14T09:53:41.959 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:53:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.961+0000 7fa927400700 1 -- 172.21.15.78:0/2080832632 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa9080a0b20 msgr2=0x7fa9080a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.961+0000 7fa927400700 1 --2- 172.21.15.78:0/2080832632 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa9080a0b20 0x7fa9080a2fe0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa90c00b2e0 tx=0x7fa90c005fd0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.961+0000 7fa927400700 1 -- 172.21.15.78:0/2080832632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa920100d40 msgr2=0x7fa920074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.961+0000 7fa927400700 1 --2- 172.21.15.78:0/2080832632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa920100d40 0x7fa920074b10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa91400ea50 tx=0x7fa91400ed60 comp rx=0 tx=0).stop 2024-03-14T09:53:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.961+0000 7fa927400700 1 -- 172.21.15.78:0/2080832632 shutdown_connections 2024-03-14T09:53:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.961+0000 7fa927400700 1 --2- 172.21.15.78:0/2080832632 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa9080a0b20 0x7fa9080a2fe0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.961+0000 7fa927400700 1 --2- 172.21.15.78:0/2080832632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa920100d40 0x7fa920074b10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.961+0000 7fa927400700 1 --2- 172.21.15.78:0/2080832632 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa9201028d0 0x7fa9200736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.961+0000 7fa927400700 1 --2- 172.21.15.78:0/2080832632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa920101f30 0x7fa9200731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:41.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.961+0000 7fa927400700 1 -- 172.21.15.78:0/2080832632 >> 172.21.15.78:0/2080832632 conn(0x7fa9200fc2b0 msgr2=0x7fa920105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:41.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.961+0000 7fa927400700 1 -- 172.21.15.78:0/2080832632 shutdown_connections 2024-03-14T09:53:41.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:41.961+0000 7fa927400700 1 -- 172.21.15.78:0/2080832632 wait complete. 2024-03-14T09:53:41.980 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:53:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:41 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1730986116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:41 smithi078 ceph-mon[38143]: pgmap v285: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T09:53:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-14T09:53:42.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-14T09:53:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:41 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 10]}]: dispatch 2024-03-14T09:53:42.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:41 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/397316550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:42.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.173+0000 7fc27aa00700 1 -- 172.21.15.78:0/1314110955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc274101e40 msgr2=0x7fc2741022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:42.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.173+0000 7fc27aa00700 1 --2- 172.21.15.78:0/1314110955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc274101e40 0x7fc2741022a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc260009920 tx=0x7fc260009c30 comp rx=0 tx=0).stop 2024-03-14T09:53:42.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.173+0000 7fc27aa00700 1 -- 172.21.15.78:0/1314110955 shutdown_connections 2024-03-14T09:53:42.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.173+0000 7fc27aa00700 1 --2- 172.21.15.78:0/1314110955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2741027e0 0x7fc274109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.173+0000 7fc27aa00700 1 --2- 172.21.15.78:0/1314110955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc274100c50 0x7fc274101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.173+0000 7fc27aa00700 1 --2- 172.21.15.78:0/1314110955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc274101e40 0x7fc2741022a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.173+0000 7fc27aa00700 1 -- 172.21.15.78:0/1314110955 >> 172.21.15.78:0/1314110955 conn(0x7fc2740fc1c0 msgr2=0x7fc2740fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:42.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.173+0000 7fc27aa00700 1 -- 172.21.15.78:0/1314110955 shutdown_connections 2024-03-14T09:53:42.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.173+0000 7fc27aa00700 1 -- 172.21.15.78:0/1314110955 wait complete. 2024-03-14T09:53:42.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.174+0000 7fc27aa00700 1 Processor -- start 2024-03-14T09:53:42.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.174+0000 7fc27aa00700 1 -- start start 2024-03-14T09:53:42.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.175+0000 7fc27aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc274100c50 0x7fc2741992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:42.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.175+0000 7fc27aa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc274101e40 0x7fc2741997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:42.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.175+0000 7fc27aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2741027e0 0x7fc2741a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:42.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.175+0000 7fc27aa00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2741a0dd0 con 0x7fc274100c50 2024-03-14T09:53:42.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.175+0000 7fc27aa00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2741a0f10 con 0x7fc2741027e0 2024-03-14T09:53:42.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.175+0000 7fc27aa00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2741a10a0 con 0x7fc274101e40 2024-03-14T09:53:42.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.175+0000 7fc273e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc274100c50 0x7fc2741992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:42.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.175+0000 7fc273e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc274100c50 0x7fc2741992a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52578/0 (socket says 172.21.15.78:52578) 2024-03-14T09:53:42.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.175+0000 7fc273e00700 1 -- 172.21.15.78:0/1615659140 learned_addr learned my addr 172.21.15.78:0/1615659140 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:42.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.175+0000 7fc273400700 1 --2- 172.21.15.78:0/1615659140 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc274101e40 0x7fc2741997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:42.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.175+0000 7fc278a00700 1 --2- 172.21.15.78:0/1615659140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2741027e0 0x7fc2741a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:42.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.175+0000 7fc273e00700 1 -- 172.21.15.78:0/1615659140 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc274101e40 msgr2=0x7fc2741997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:42.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.175+0000 7fc273e00700 1 --2- 172.21.15.78:0/1615659140 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc274101e40 0x7fc2741997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.175+0000 7fc273e00700 1 -- 172.21.15.78:0/1615659140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2741027e0 msgr2=0x7fc2741a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:42.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.175+0000 7fc273e00700 1 --2- 172.21.15.78:0/1615659140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2741027e0 0x7fc2741a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.175+0000 7fc273e00700 1 -- 172.21.15.78:0/1615659140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26c009d00 con 0x7fc274100c50 2024-03-14T09:53:42.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.176+0000 7fc273e00700 1 --2- 172.21.15.78:0/1615659140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc274100c50 0x7fc2741992a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc26400e970 tx=0x7fc26400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:42.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.176+0000 7fc270c00700 1 -- 172.21.15.78:0/1615659140 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc264009790 con 0x7fc274100c50 2024-03-14T09:53:42.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.176+0000 7fc270c00700 1 -- 172.21.15.78:0/1615659140 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc26400c7e0 con 0x7fc274100c50 2024-03-14T09:53:42.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.176+0000 7fc27aa00700 1 -- 172.21.15.78:0/1615659140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc260009520 con 0x7fc274100c50 2024-03-14T09:53:42.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.176+0000 7fc270c00700 1 -- 172.21.15.78:0/1615659140 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc264003c70 con 0x7fc274100c50 2024-03-14T09:53:42.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.176+0000 7fc27aa00700 1 -- 172.21.15.78:0/1615659140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2741a1700 con 0x7fc274100c50 2024-03-14T09:53:42.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.177+0000 7fc27aa00700 1 -- 172.21.15.78:0/1615659140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2740679e0 con 0x7fc274100c50 2024-03-14T09:53:42.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.179+0000 7fc270c00700 1 -- 172.21.15.78:0/1615659140 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fc26400cc50 con 0x7fc274100c50 2024-03-14T09:53:42.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.180+0000 7fc270c00700 1 --2- 172.21.15.78:0/1615659140 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc2580a0a90 0x7fc2580a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:42.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.180+0000 7fc270c00700 1 -- 172.21.15.78:0/1615659140 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9107+0+0 (secure 0 0 0) 0x7fc264014070 con 0x7fc274100c50 2024-03-14T09:53:42.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.180+0000 7fc273400700 1 --2- 172.21.15.78:0/1615659140 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc2580a0a90 0x7fc2580a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:42.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.181+0000 7fc273400700 1 --2- 172.21.15.78:0/1615659140 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc2580a0a90 0x7fc2580a2f50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc26000fa40 tx=0x7fc260019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:42.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.184+0000 7fc270c00700 1 -- 172.21.15.78:0/1615659140 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc264095bd0 con 0x7fc274100c50 2024-03-14T09:53:42.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:41 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1730986116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:42.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:41 smithi152 ceph-mon[48279]: pgmap v285: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T09:53:42.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-14T09:53:42.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-14T09:53:42.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:41 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 10]}]: dispatch 2024-03-14T09:53:42.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:41 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/397316550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:42.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:41 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1730986116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:42.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:41 smithi132 ceph-mon[47959]: pgmap v285: 129 pgs: 129 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T09:53:42.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-14T09:53:42.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-14T09:53:42.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:41 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 10]}]: dispatch 2024-03-14T09:53:42.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:41 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/397316550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:42.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.432+0000 7fc27aa00700 1 -- 172.21.15.78:0/1615659140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc27410c8d0 con 0x7fc274100c50 2024-03-14T09:53:42.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.433+0000 7fc270c00700 1 -- 172.21.15.78:0/1615659140 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10870 (secure 0 0 0) 0x7fc264003dd0 con 0x7fc274100c50 2024-03-14T09:53:42.434 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:42.434 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:52:30.715484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:53:42.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.435+0000 7fc27aa00700 1 -- 172.21.15.78:0/1615659140 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc2580a0a90 msgr2=0x7fc2580a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:42.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.435+0000 7fc27aa00700 1 --2- 172.21.15.78:0/1615659140 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc2580a0a90 0x7fc2580a2f50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc26000fa40 tx=0x7fc260019040 comp rx=0 tx=0).stop 2024-03-14T09:53:42.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.435+0000 7fc27aa00700 1 -- 172.21.15.78:0/1615659140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc274100c50 msgr2=0x7fc2741992a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:42.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.435+0000 7fc27aa00700 1 --2- 172.21.15.78:0/1615659140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc274100c50 0x7fc2741992a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc26400e970 tx=0x7fc26400ec80 comp rx=0 tx=0).stop 2024-03-14T09:53:42.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.435+0000 7fc27aa00700 1 -- 172.21.15.78:0/1615659140 shutdown_connections 2024-03-14T09:53:42.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.435+0000 7fc27aa00700 1 --2- 172.21.15.78:0/1615659140 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc2580a0a90 0x7fc2580a2f50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.435+0000 7fc27aa00700 1 --2- 172.21.15.78:0/1615659140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2741027e0 0x7fc2741a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.435+0000 7fc27aa00700 1 --2- 172.21.15.78:0/1615659140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc274100c50 0x7fc2741992a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.435+0000 7fc27aa00700 1 --2- 172.21.15.78:0/1615659140 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc274101e40 0x7fc2741997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.435+0000 7fc27aa00700 1 -- 172.21.15.78:0/1615659140 >> 172.21.15.78:0/1615659140 conn(0x7fc2740fc1c0 msgr2=0x7fc2741076b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:42.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.436+0000 7fc27aa00700 1 -- 172.21.15.78:0/1615659140 shutdown_connections 2024-03-14T09:53:42.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.436+0000 7fc27aa00700 1 -- 172.21.15.78:0/1615659140 wait complete. 2024-03-14T09:53:42.437 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2024-03-14T09:53:42.455 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:53:42.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.647+0000 7fab95200700 1 -- 172.21.15.78:0/2875447353 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab901027e0 msgr2=0x7fab90109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:42.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.647+0000 7fab95200700 1 --2- 172.21.15.78:0/2875447353 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab901027e0 0x7fab90109090 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fab8800b0d0 tx=0x7fab8800b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.647+0000 7fab95200700 1 -- 172.21.15.78:0/2875447353 shutdown_connections 2024-03-14T09:53:42.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.647+0000 7fab95200700 1 --2- 172.21.15.78:0/2875447353 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab901027e0 0x7fab90109090 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.647+0000 7fab95200700 1 --2- 172.21.15.78:0/2875447353 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab90100c50 0x7fab90101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.647+0000 7fab95200700 1 --2- 172.21.15.78:0/2875447353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab90101e40 0x7fab901022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.647+0000 7fab95200700 1 -- 172.21.15.78:0/2875447353 >> 172.21.15.78:0/2875447353 conn(0x7fab900fc1c0 msgr2=0x7fab900fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:42.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.647+0000 7fab95200700 1 -- 172.21.15.78:0/2875447353 shutdown_connections 2024-03-14T09:53:42.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.648+0000 7fab95200700 1 -- 172.21.15.78:0/2875447353 wait complete. 2024-03-14T09:53:42.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.648+0000 7fab95200700 1 Processor -- start 2024-03-14T09:53:42.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.649+0000 7fab95200700 1 -- start start 2024-03-14T09:53:42.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.649+0000 7fab95200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab90100c50 0x7fab901993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:42.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.649+0000 7fab95200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab90101e40 0x7fab901998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:42.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.649+0000 7fab95200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab901027e0 0x7fab901a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:42.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.649+0000 7fab95200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab901a0e90 con 0x7fab90100c50 2024-03-14T09:53:42.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.649+0000 7fab95200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab901a0fd0 con 0x7fab90101e40 2024-03-14T09:53:42.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.649+0000 7fab95200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab901a1160 con 0x7fab901027e0 2024-03-14T09:53:42.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.649+0000 7fab8de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab90100c50 0x7fab901993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:42.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.649+0000 7fab8de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab90100c50 0x7fab901993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52596/0 (socket says 172.21.15.78:52596) 2024-03-14T09:53:42.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.649+0000 7fab8de00700 1 -- 172.21.15.78:0/2329565595 learned_addr learned my addr 172.21.15.78:0/2329565595 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:42.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.649+0000 7fab8d400700 1 --2- 172.21.15.78:0/2329565595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab90101e40 0x7fab901998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:42.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.649+0000 7fab8e800700 1 --2- 172.21.15.78:0/2329565595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab901027e0 0x7fab901a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:42.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.649+0000 7fab8de00700 1 -- 172.21.15.78:0/2329565595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab901027e0 msgr2=0x7fab901a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:42.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.649+0000 7fab8de00700 1 --2- 172.21.15.78:0/2329565595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab901027e0 0x7fab901a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.650+0000 7fab8de00700 1 -- 172.21.15.78:0/2329565595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab90101e40 msgr2=0x7fab901998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:42.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.650+0000 7fab8de00700 1 --2- 172.21.15.78:0/2329565595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab90101e40 0x7fab901998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.650+0000 7fab8de00700 1 -- 172.21.15.78:0/2329565595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab88009d00 con 0x7fab90100c50 2024-03-14T09:53:42.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.650+0000 7fab8de00700 1 --2- 172.21.15.78:0/2329565595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab90100c50 0x7fab901993b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fab8000e970 tx=0x7fab8000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:42.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.650+0000 7fab86a00700 1 -- 172.21.15.78:0/2329565595 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab80009790 con 0x7fab90100c50 2024-03-14T09:53:42.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.650+0000 7fab86a00700 1 -- 172.21.15.78:0/2329565595 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab8000c7e0 con 0x7fab90100c50 2024-03-14T09:53:42.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.650+0000 7fab95200700 1 -- 172.21.15.78:0/2329565595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab901a1440 con 0x7fab90100c50 2024-03-14T09:53:42.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.650+0000 7fab86a00700 1 -- 172.21.15.78:0/2329565595 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab80003c70 con 0x7fab90100c50 2024-03-14T09:53:42.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.650+0000 7fab95200700 1 -- 172.21.15.78:0/2329565595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab901a1950 con 0x7fab90100c50 2024-03-14T09:53:42.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.651+0000 7fab95200700 1 -- 172.21.15.78:0/2329565595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab900679e0 con 0x7fab90100c50 2024-03-14T09:53:42.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.653+0000 7fab86a00700 1 -- 172.21.15.78:0/2329565595 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fab8000cc50 con 0x7fab90100c50 2024-03-14T09:53:42.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.654+0000 7fab86a00700 1 --2- 172.21.15.78:0/2329565595 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fab740a0a90 0x7fab740a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:42.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.654+0000 7fab86a00700 1 -- 172.21.15.78:0/2329565595 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9107+0+0 (secure 0 0 0) 0x7fab80014070 con 0x7fab90100c50 2024-03-14T09:53:42.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.654+0000 7fab8d400700 1 --2- 172.21.15.78:0/2329565595 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fab740a0a90 0x7fab740a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:42.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.655+0000 7fab8d400700 1 --2- 172.21.15.78:0/2329565595 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fab740a0a90 0x7fab740a2f50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fab7c009c60 tx=0x7fab7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:42.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.658+0000 7fab86a00700 1 -- 172.21.15.78:0/2329565595 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fab80095bf0 con 0x7fab90100c50 2024-03-14T09:53:42.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.890+0000 7fab95200700 1 -- 172.21.15.78:0/2329565595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab9010c8e0 con 0x7fab90100c50 2024-03-14T09:53:42.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.891+0000 7fab86a00700 1 -- 172.21.15.78:0/2329565595 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24594 (secure 0 0 0) 0x7fab80003dd0 con 0x7fab90100c50 2024-03-14T09:53:42.892 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:42.892 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":87,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:53:42.786378+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:53:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.894+0000 7fab95200700 1 -- 172.21.15.78:0/2329565595 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fab740a0a90 msgr2=0x7fab740a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.894+0000 7fab95200700 1 --2- 172.21.15.78:0/2329565595 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fab740a0a90 0x7fab740a2f50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fab7c009c60 tx=0x7fab7c009260 comp rx=0 tx=0).stop 2024-03-14T09:53:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.894+0000 7fab95200700 1 -- 172.21.15.78:0/2329565595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab90100c50 msgr2=0x7fab901993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.894+0000 7fab95200700 1 --2- 172.21.15.78:0/2329565595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab90100c50 0x7fab901993b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fab8000e970 tx=0x7fab8000ec80 comp rx=0 tx=0).stop 2024-03-14T09:53:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.894+0000 7fab95200700 1 -- 172.21.15.78:0/2329565595 shutdown_connections 2024-03-14T09:53:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.894+0000 7fab95200700 1 --2- 172.21.15.78:0/2329565595 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fab901027e0 0x7fab901a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.894+0000 7fab95200700 1 --2- 172.21.15.78:0/2329565595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab90100c50 0x7fab901993b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.894+0000 7fab95200700 1 --2- 172.21.15.78:0/2329565595 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fab740a0a90 0x7fab740a2f50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.894+0000 7fab95200700 1 --2- 172.21.15.78:0/2329565595 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab90101e40 0x7fab901998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:42.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.894+0000 7fab95200700 1 -- 172.21.15.78:0/2329565595 >> 172.21.15.78:0/2329565595 conn(0x7fab900fc1c0 msgr2=0x7fab90072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:42.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.894+0000 7fab95200700 1 -- 172.21.15.78:0/2329565595 shutdown_connections 2024-03-14T09:53:42.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:42.894+0000 7fab95200700 1 -- 172.21.15.78:0/2329565595 wait complete. 2024-03-14T09:53:42.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-14T09:52:05.708711+0000', 'modified': '2024-03-14T09:52:30.715484+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24572, 'mds_1': 24538, 'mds_2': 24527}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24508': {'gid': 24508, 'name': 'c', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.152:6837/2123600730', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6836', 'nonce': 2123600730}, {'type': 'v1', 'addr': '172.21.15.152:6837', 'nonce': 2123600730}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24509': {'gid': 24509, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.132:6835/1709886688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6834', 'nonce': 1709886688}, {'type': 'v1', 'addr': '172.21.15.132:6835', 'nonce': 1709886688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24526': {'gid': 24526, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.152:6839/1965809497', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.152:6838', 'nonce': 1965809497}, {'type': 'v1', 'addr': '172.21.15.152:6839', 'nonce': 1965809497}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24527': {'gid': 24527, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.132:6837/142126094', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6836', 'nonce': 142126094}, {'type': 'v1', 'addr': '172.21.15.132:6837', 'nonce': 142126094}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24538': {'gid': 24538, 'name': 'g', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.78:6839/4227580185', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6838', 'nonce': 4227580185}, {'type': 'v1', 'addr': '172.21.15.78:6839', 'nonce': 4227580185}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.132:6839/2192113559', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6838', 'nonce': 2192113559}, {'type': 'v1', 'addr': '172.21.15.132:6839', 'nonce': 2192113559}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-14T09:53:42.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-14T09:53:42.914 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-14T09:53:43.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-14T09:53:43.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-14T09:53:43.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:42 smithi078 ceph-mon[38143]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 10]}]': finished 2024-03-14T09:53:43.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:42 smithi078 ceph-mon[38143]: osdmap e86: 12 total, 12 up, 12 in 2024-03-14T09:53:43.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:42 smithi078 ceph-mon[38143]: mgrmap e18: x(active, since 7m), standbys: z, y 2024-03-14T09:53:43.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:42 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2080832632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:43.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:42 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1615659140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.107+0000 7f7319000700 1 -- 172.21.15.78:0/2833231193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73141028e0 msgr2=0x7f7314109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.107+0000 7f7319000700 1 --2- 172.21.15.78:0/2833231193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73141028e0 0x7f7314109190 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f730c00b0d0 tx=0x7f730c00b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.107+0000 7f7319000700 1 -- 172.21.15.78:0/2833231193 shutdown_connections 2024-03-14T09:53:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.107+0000 7f7319000700 1 --2- 172.21.15.78:0/2833231193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7314100d50 0x7f7314101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.107+0000 7f7319000700 1 --2- 172.21.15.78:0/2833231193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73141028e0 0x7f7314109190 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.107+0000 7f7319000700 1 --2- 172.21.15.78:0/2833231193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7314101f40 0x7f73141023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.107+0000 7f7319000700 1 -- 172.21.15.78:0/2833231193 >> 172.21.15.78:0/2833231193 conn(0x7f73140fc2c0 msgr2=0x7f73140fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.107+0000 7f7319000700 1 -- 172.21.15.78:0/2833231193 shutdown_connections 2024-03-14T09:53:43.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.107+0000 7f7319000700 1 -- 172.21.15.78:0/2833231193 wait complete. 2024-03-14T09:53:43.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.108+0000 7f7319000700 1 Processor -- start 2024-03-14T09:53:43.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.108+0000 7f7319000700 1 -- start start 2024-03-14T09:53:43.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.108+0000 7f7319000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7314100d50 0x7f7314199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.108+0000 7f7319000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7314101f40 0x7f7314199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.109+0000 7f7319000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73141028e0 0x7f73141a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.109+0000 7f7319000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73141a0f50 con 0x7f7314100d50 2024-03-14T09:53:43.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.109+0000 7f7319000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73141a1090 con 0x7f73141028e0 2024-03-14T09:53:43.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.109+0000 7f7319000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73141a1220 con 0x7f7314101f40 2024-03-14T09:53:43.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.109+0000 7f7311e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7314100d50 0x7f7314199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.109+0000 7f7311400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7314101f40 0x7f7314199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.109+0000 7f7311e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7314100d50 0x7f7314199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52606/0 (socket says 172.21.15.78:52606) 2024-03-14T09:53:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.109+0000 7f7311e00700 1 -- 172.21.15.78:0/4093429048 learned_addr learned my addr 172.21.15.78:0/4093429048 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.109+0000 7f7312800700 1 --2- 172.21.15.78:0/4093429048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73141028e0 0x7f73141a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.109+0000 7f7311e00700 1 -- 172.21.15.78:0/4093429048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7314101f40 msgr2=0x7f7314199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.109+0000 7f7311e00700 1 --2- 172.21.15.78:0/4093429048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7314101f40 0x7f7314199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.109+0000 7f7311e00700 1 -- 172.21.15.78:0/4093429048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73141028e0 msgr2=0x7f73141a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.109+0000 7f7311e00700 1 --2- 172.21.15.78:0/4093429048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73141028e0 0x7f73141a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.109+0000 7f7311e00700 1 -- 172.21.15.78:0/4093429048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c009d00 con 0x7f7314100d50 2024-03-14T09:53:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.109+0000 7f7311e00700 1 --2- 172.21.15.78:0/4093429048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7314100d50 0x7f7314199420 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f730400e970 tx=0x7f730400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.110+0000 7f730aa00700 1 -- 172.21.15.78:0/4093429048 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304009790 con 0x7f7314100d50 2024-03-14T09:53:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.110+0000 7f730aa00700 1 -- 172.21.15.78:0/4093429048 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f730400c7e0 con 0x7f7314100d50 2024-03-14T09:53:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.110+0000 7f730aa00700 1 -- 172.21.15.78:0/4093429048 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304003c70 con 0x7f7314100d50 2024-03-14T09:53:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.110+0000 7f7319000700 1 -- 172.21.15.78:0/4093429048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73141a1500 con 0x7f7314100d50 2024-03-14T09:53:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.110+0000 7f7319000700 1 -- 172.21.15.78:0/4093429048 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73141a19c0 con 0x7f7314100d50 2024-03-14T09:53:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.113+0000 7f730aa00700 1 -- 172.21.15.78:0/4093429048 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f730400cc50 con 0x7f7314100d50 2024-03-14T09:53:43.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.114+0000 7f732cb45b80 1 Processor -- start 2024-03-14T09:53:43.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.114+0000 7f730aa00700 1 --2- 172.21.15.78:0/4093429048 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f72f80a0b10 0x7f72f80a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.114+0000 7f732cb45b80 1 -- start start 2024-03-14T09:53:43.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.114+0000 7f730aa00700 1 -- 172.21.15.78:0/4093429048 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f7304014070 con 0x7f7314100d50 2024-03-14T09:53:43.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f7312800700 1 --2- 172.21.15.78:0/4093429048 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f72f80a0b10 0x7f72f80a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f732cb45b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d47a0 0x5581393d4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f732cb45b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 0x5581393d5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f732cb45b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5f90 0x5581393d6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f732cb45b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55813932d940 con 0x5581393d47a0 2024-03-14T09:53:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f732cb45b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55813932b300 con 0x5581393d5f90 2024-03-14T09:53:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f732cb45b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558139326bf0 con 0x5581393d5100 2024-03-14T09:53:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f7311400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 0x5581393d5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f7311e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d47a0 0x5581393d4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f7312800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5f90 0x5581393d6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f7311400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 0x5581393d5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:51836/0 (socket says 172.21.15.78:51836) 2024-03-14T09:53:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f7311400700 1 -- 172.21.15.78:0/2238858691 learned_addr learned my addr 172.21.15.78:0/2238858691 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:43.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f7311400700 1 -- 172.21.15.78:0/2238858691 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5f90 msgr2=0x5581393d6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f7311400700 1 --2- 172.21.15.78:0/2238858691 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5f90 0x5581393d6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f7311400700 1 -- 172.21.15.78:0/2238858691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d47a0 msgr2=0x5581393d4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f7311400700 1 --2- 172.21.15.78:0/2238858691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d47a0 0x5581393d4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.115+0000 7f7311400700 1 -- 172.21.15.78:0/2238858691 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581391a8360 con 0x5581393d5100 2024-03-14T09:53:43.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.116+0000 7f7312800700 1 --2- 172.21.15.78:0/4093429048 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f72f80a0b10 0x7f72f80a2fd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f730c005bd0 tx=0x7f730c008c10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.116+0000 7f7311400700 1 --2- 172.21.15.78:0/2238858691 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 0x5581393d5a00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7300009d80 tx=0x7f7300011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=503b98dbb073dad8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.116+0000 7f72ff400700 1 -- 172.21.15.78:0/2238858691 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7300004230 con 0x5581393d5100 2024-03-14T09:53:43.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.116+0000 7f72ff400700 1 -- 172.21.15.78:0/2238858691 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7300004390 con 0x5581393d5100 2024-03-14T09:53:43.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.116+0000 7f72ff400700 1 -- 172.21.15.78:0/2238858691 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73000197e0 con 0x5581393d5100 2024-03-14T09:53:43.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.117+0000 7f732cb45b80 1 -- 172.21.15.78:0/2238858691 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 msgr2=0x5581393d5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.117+0000 7f732cb45b80 1 --2- 172.21.15.78:0/2238858691 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 0x5581393d5a00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7300009d80 tx=0x7f7300011720 comp rx=0 tx=0).stop 2024-03-14T09:53:43.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.117+0000 7f732cb45b80 1 -- 172.21.15.78:0/2238858691 shutdown_connections 2024-03-14T09:53:43.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.117+0000 7f732cb45b80 1 --2- 172.21.15.78:0/2238858691 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5f90 0x5581393d6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.117+0000 7f732cb45b80 1 --2- 172.21.15.78:0/2238858691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d47a0 0x5581393d4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.117+0000 7f732cb45b80 1 --2- 172.21.15.78:0/2238858691 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 0x5581393d5a00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.117+0000 7f732cb45b80 1 -- 172.21.15.78:0/2238858691 >> 172.21.15.78:0/2238858691 conn(0x55813934a1d0 msgr2=0x5581393cf580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.117+0000 7f732cb45b80 1 -- 172.21.15.78:0/2238858691 shutdown_connections 2024-03-14T09:53:43.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.117+0000 7f732cb45b80 1 -- 172.21.15.78:0/2238858691 wait complete. 2024-03-14T09:53:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.117+0000 7f732cb45b80 1 -- start start 2024-03-14T09:53:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.118+0000 7f732cb45b80 1 Processor -- start 2024-03-14T09:53:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.118+0000 7f732cb45b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d47a0 0x5581393de610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.118+0000 7f732cb45b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 0x5581393deb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.118+0000 7f732cb45b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5f90 0x5581393e5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.118+0000 7f732cb45b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581392c2d30 con 0x5581393d47a0 2024-03-14T09:53:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.118+0000 7f732cb45b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581392c29d0 con 0x5581393d5f90 2024-03-14T09:53:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.118+0000 7f732cb45b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581391aa650 con 0x5581393d5100 2024-03-14T09:53:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.118+0000 7f732cb45b80 10 client.0 _ll_register_callbacks cb 0x55813912e350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:53:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.118+0000 7f7311e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d47a0 0x5581393de610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.118+0000 7f7311400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 0x5581393deb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.118+0000 7f7311e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d47a0 0x5581393de610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52628/0 (socket says 172.21.15.78:52628) 2024-03-14T09:53:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.118+0000 7f7311e00700 1 -- 172.21.15.78:0/2160695217 learned_addr learned my addr 172.21.15.78:0/2160695217 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.118+0000 7f7312800700 1 --2- 172.21.15.78:0/2160695217 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5f90 0x5581393e5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.119+0000 7f7311e00700 1 -- 172.21.15.78:0/2160695217 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 msgr2=0x5581393deb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.119+0000 7f7311e00700 1 --2- 172.21.15.78:0/2160695217 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 0x5581393deb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.119+0000 7f7311e00700 1 -- 172.21.15.78:0/2160695217 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5f90 msgr2=0x5581393e5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.119+0000 7f7311e00700 1 --2- 172.21.15.78:0/2160695217 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5f90 0x5581393e5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.119+0000 7f7311e00700 1 -- 172.21.15.78:0/2160695217 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73000099e0 con 0x5581393d47a0 2024-03-14T09:53:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.119+0000 7f7311e00700 1 --2- 172.21.15.78:0/2160695217 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d47a0 0x5581393de610 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7304000c00 tx=0x7f7304003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d403edf7cbec1ba7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.119+0000 7f72f3e00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:53:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.119+0000 7f72f3e00700 1 -- 172.21.15.78:0/2160695217 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73040a1ca0 con 0x5581393d47a0 2024-03-14T09:53:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.119+0000 7f72f3e00700 1 -- 172.21.15.78:0/2160695217 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f7304095f30 con 0x5581393d47a0 2024-03-14T09:53:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.119+0000 7f732cb45b80 1 -- 172.21.15.78:0/2160695217 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5581393c76d0 con 0x5581393d47a0 2024-03-14T09:53:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.119+0000 7f72f3e00700 1 -- 172.21.15.78:0/2160695217 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73040960f0 con 0x5581393d47a0 2024-03-14T09:53:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.119+0000 7f732cb45b80 10 client.15294 fetch_fsmap learned FSMap version 24 2024-03-14T09:53:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.119+0000 7f732cb45b80 1 -- 172.21.15.78:0/2160695217 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x558139154b10 con 0x5581393d47a0 2024-03-14T09:53:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.120+0000 7f72f3e00700 1 -- 172.21.15.78:0/2160695217 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f730407e070 con 0x5581393d47a0 2024-03-14T09:53:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.120+0000 7f72f3e00700 1 -- 172.21.15.78:0/2160695217 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7304099b10 con 0x5581393d47a0 2024-03-14T09:53:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.120+0000 7f732cb45b80 10 client.15294 fetch_fsmap finished waiting for FSMap version 24 2024-03-14T09:53:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.120+0000 7f732cb45b80 10 client.15294 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-14T09:53:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.120+0000 7f732cb45b80 20 client.15294 populate_metadata read hostname 'smithi078' 2024-03-14T09:53:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.120+0000 7f732cb45b80 1 --2- 172.21.15.78:0/2160695217 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5581393e97b0 0x5581393eead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.120+0000 7f732cb45b80 4 client.15294 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:53:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.120+0000 7f732cb45b80 1 -- 172.21.15.78:0/2160695217 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55813931f6b0 con 0x5581393e97b0 2024-03-14T09:53:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.120+0000 7f7311400700 1 --2- 172.21.15.78:0/2160695217 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5581393e97b0 0x5581393eead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.121+0000 7f72f3e00700 10 client.15294 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:53:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.121+0000 7f7311400700 1 --2- 172.21.15.78:0/2160695217 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5581393e97b0 0x5581393eead0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d11c90d69063087d server_cookie=3a3f85e5d7e22588 in_seq=0 out_seq=0 2024-03-14T09:53:43.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.121+0000 7f72f3e00700 1 -- 172.21.15.78:0/2160695217 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55813931f6b0 con 0x5581393e97b0 2024-03-14T09:53:43.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.121+0000 7f72f3e00700 10 client.15294 handle_command_reply: tid=0 2024-03-14T09:53:43.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.121+0000 7f732cb45b80 1 client.15294 shutdown 2024-03-14T09:53:43.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.121+0000 7f732cb45b80 1 -- 172.21.15.78:0/2160695217 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d47a0 msgr2=0x5581393de610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.121+0000 7f732cb45b80 1 --2- 172.21.15.78:0/2160695217 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d47a0 0x5581393de610 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7304000c00 tx=0x7f7304003ea0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.122+0000 7f732cb45b80 1 -- 172.21.15.78:0/2160695217 shutdown_connections 2024-03-14T09:53:43.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.122+0000 7f732cb45b80 1 --2- 172.21.15.78:0/2160695217 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5581393e97b0 0x5581393eead0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.122+0000 7f732cb45b80 1 --2- 172.21.15.78:0/2160695217 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5f90 0x5581393e5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.122+0000 7f732cb45b80 1 --2- 172.21.15.78:0/2160695217 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d47a0 0x5581393de610 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.122+0000 7f72f3e00700 0 client.15294 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:53:43.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.122+0000 7f732cb45b80 1 --2- 172.21.15.78:0/2160695217 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 0x5581393deb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.122+0000 7f732cb45b80 1 -- 172.21.15.78:0/2160695217 >> 172.21.15.78:0/2160695217 conn(0x55813934a1d0 msgr2=0x5581393cebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.122+0000 7f732cb45b80 1 -- 172.21.15.78:0/2160695217 shutdown_connections 2024-03-14T09:53:43.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.122+0000 7f732cb45b80 1 -- 172.21.15.78:0/2160695217 wait complete. 2024-03-14T09:53:43.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.122+0000 7f732cb45b80 20 client.15294 trim_cache size 0 max 16384 2024-03-14T09:53:43.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.139+0000 7f732cb45b80 1 Processor -- start 2024-03-14T09:53:43.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.139+0000 7f732cb45b80 1 -- start start 2024-03-14T09:53:43.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.139+0000 7f732cb45b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d47a0 0x5581393e3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.139+0000 7f732cb45b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 0x5581393de3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.139+0000 7f732cb45b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d5f90 0x5581393de900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.139+0000 7f732cb45b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558139323850 con 0x5581393d5f90 2024-03-14T09:53:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.139+0000 7f732cb45b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581392c5de0 con 0x5581393d47a0 2024-03-14T09:53:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.139+0000 7f732cb45b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581393eb050 con 0x5581393d5100 2024-03-14T09:53:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.140+0000 7f7312800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d5f90 0x5581393de900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.140+0000 7f7312800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d5f90 0x5581393de900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52644/0 (socket says 172.21.15.78:52644) 2024-03-14T09:53:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.140+0000 7f7312800700 1 -- 172.21.15.78:0/204744516 learned_addr learned my addr 172.21.15.78:0/204744516 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.140+0000 7f7311e00700 1 --2- 172.21.15.78:0/204744516 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d47a0 0x5581393e3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.140+0000 7f7311400700 1 --2- 172.21.15.78:0/204744516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 0x5581393de3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.140+0000 7f7312800700 1 -- 172.21.15.78:0/204744516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 msgr2=0x5581393de3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.140+0000 7f7312800700 1 --2- 172.21.15.78:0/204744516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 0x5581393de3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.140+0000 7f7312800700 1 -- 172.21.15.78:0/204744516 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d47a0 msgr2=0x5581393e3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.140+0000 7f7312800700 1 --2- 172.21.15.78:0/204744516 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d47a0 0x5581393e3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.140+0000 7f7312800700 1 -- 172.21.15.78:0/204744516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581392ece50 con 0x5581393d5f90 2024-03-14T09:53:43.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.140+0000 7f7311400700 1 --2- 172.21.15.78:0/204744516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 0x5581393de3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:53:43.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.140+0000 7f7311e00700 1 --2- 172.21.15.78:0/204744516 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d47a0 0x5581393e3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:53:43.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.140+0000 7f7312800700 1 --2- 172.21.15.78:0/204744516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d5f90 0x5581393de900 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f730c015000 tx=0x7f730c024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14d9e60787dec74 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.141+0000 7f72ffe00700 1 -- 172.21.15.78:0/204744516 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c00e040 con 0x5581393d5f90 2024-03-14T09:53:43.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.141+0000 7f72ffe00700 1 -- 172.21.15.78:0/204744516 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f730c004a10 con 0x5581393d5f90 2024-03-14T09:53:43.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.141+0000 7f72ffe00700 1 -- 172.21.15.78:0/204744516 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c005500 con 0x5581393d5f90 2024-03-14T09:53:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.141+0000 7f732cb45b80 1 -- 172.21.15.78:0/204744516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d5f90 msgr2=0x5581393de900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.141+0000 7f732cb45b80 1 --2- 172.21.15.78:0/204744516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d5f90 0x5581393de900 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f730c015000 tx=0x7f730c024000 comp rx=0 tx=0).stop 2024-03-14T09:53:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.141+0000 7f732cb45b80 1 -- 172.21.15.78:0/204744516 shutdown_connections 2024-03-14T09:53:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.141+0000 7f732cb45b80 1 --2- 172.21.15.78:0/204744516 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d47a0 0x5581393e3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.141+0000 7f732cb45b80 1 --2- 172.21.15.78:0/204744516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d5f90 0x5581393de900 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.141+0000 7f732cb45b80 1 --2- 172.21.15.78:0/204744516 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d5100 0x5581393de3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.141+0000 7f732cb45b80 1 -- 172.21.15.78:0/204744516 >> 172.21.15.78:0/204744516 conn(0x55813934a1d0 msgr2=0x5581393d8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.141+0000 7f732cb45b80 1 -- 172.21.15.78:0/204744516 shutdown_connections 2024-03-14T09:53:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.141+0000 7f732cb45b80 1 -- 172.21.15.78:0/204744516 wait complete. 2024-03-14T09:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.142+0000 7f732cb45b80 1 -- start start 2024-03-14T09:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.142+0000 7f732cb45b80 1 Processor -- start 2024-03-14T09:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.142+0000 7f732cb45b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d47a0 0x5581393e0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.142+0000 7f732cb45b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5100 0x5581393e1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.142+0000 7f732cb45b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d5f90 0x5581393e2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.142+0000 7f732cb45b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581393e2570 con 0x5581393d5f90 2024-03-14T09:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.142+0000 7f732cb45b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581393e26e0 con 0x5581393d5100 2024-03-14T09:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.142+0000 7f732cb45b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581393e28a0 con 0x5581393d47a0 2024-03-14T09:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.142+0000 7f732cb45b80 10 client.0 _ll_register_callbacks cb 0x55813912e350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.143+0000 7f7311400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5100 0x5581393e1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.143+0000 7f7312800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d5f90 0x5581393e2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.143+0000 7f7311400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5100 0x5581393e1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:35866/0 (socket says 172.21.15.78:35866) 2024-03-14T09:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.143+0000 7f7311400700 1 -- 172.21.15.78:0/385254780 learned_addr learned my addr 172.21.15.78:0/385254780 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.143+0000 7f7311e00700 1 --2- 172.21.15.78:0/385254780 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d47a0 0x5581393e0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.143+0000 7f7311400700 1 -- 172.21.15.78:0/385254780 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d47a0 msgr2=0x5581393e0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.143+0000 7f7311400700 1 --2- 172.21.15.78:0/385254780 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d47a0 0x5581393e0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.143+0000 7f7311400700 1 -- 172.21.15.78:0/385254780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d5f90 msgr2=0x5581393e2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.143+0000 7f7311400700 1 --2- 172.21.15.78:0/385254780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d5f90 0x5581393e2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.143+0000 7f7311400700 1 -- 172.21.15.78:0/385254780 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c027050 con 0x5581393d5100 2024-03-14T09:53:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.143+0000 7f7311e00700 1 --2- 172.21.15.78:0/385254780 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d47a0 0x5581393e0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:53:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.143+0000 7f7312800700 1 --2- 172.21.15.78:0/385254780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d5f90 0x5581393e2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.143+0000 7f7311400700 1 --2- 172.21.15.78:0/385254780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5100 0x5581393e1020 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7300000c00 tx=0x7f7300009480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1abe500a9df92144 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.143+0000 7f72f3e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:53:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.144+0000 7f72f3e00700 1 -- 172.21.15.78:0/385254780 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7300029050 con 0x5581393d5100 2024-03-14T09:53:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.144+0000 7f732cb45b80 1 -- 172.21.15.78:0/385254780 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7300004390 con 0x5581393d5100 2024-03-14T09:53:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.144+0000 7f72f3e00700 1 -- 172.21.15.78:0/385254780 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f7300005c20 con 0x5581393d5100 2024-03-14T09:53:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.144+0000 7f72f3e00700 1 -- 172.21.15.78:0/385254780 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f730002a1b0 con 0x5581393d5100 2024-03-14T09:53:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.144+0000 7f732cb45b80 10 client.25310 fetch_fsmap learned FSMap version 24 2024-03-14T09:53:43.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.144+0000 7f732cb45b80 1 -- 172.21.15.78:0/385254780 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5581391acc80 con 0x5581393d5100 2024-03-14T09:53:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.144+0000 7f72f3e00700 1 -- 172.21.15.78:0/385254780 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7300004390 con 0x5581393d5100 2024-03-14T09:53:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.144+0000 7f72f3e00700 1 -- 172.21.15.78:0/385254780 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7300011440 con 0x5581393d5100 2024-03-14T09:53:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.144+0000 7f732cb45b80 10 client.25310 fetch_fsmap finished waiting for FSMap version 24 2024-03-14T09:53:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.144+0000 7f732cb45b80 10 client.25310 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-14T09:53:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.144+0000 7f732cb45b80 20 client.25310 populate_metadata read hostname 'smithi078' 2024-03-14T09:53:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.144+0000 7f732cb45b80 1 --2- 172.21.15.78:0/385254780 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x558139441460 0x55813943ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.144+0000 7f732cb45b80 4 client.25310 mds_command: new command op to 24527 tid=0[{"prefix": "status"}] 2024-03-14T09:53:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.144+0000 7f732cb45b80 1 -- 172.21.15.78:0/385254780 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558139175ca0 con 0x558139441460 2024-03-14T09:53:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.145+0000 7f7311e00700 1 --2- 172.21.15.78:0/385254780 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x558139441460 0x55813943ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.145+0000 7f72f3e00700 10 client.25310 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:53:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.145+0000 7f7311e00700 1 --2- 172.21.15.78:0/385254780 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x558139441460 0x55813943ddc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7b48e5db70b38aa7 server_cookie=91f3eeb183aa3510 in_seq=0 out_seq=0 2024-03-14T09:53:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.145+0000 7f72f3e00700 1 -- 172.21.15.78:0/385254780 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x558139175ca0 con 0x558139441460 2024-03-14T09:53:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.145+0000 7f72f3e00700 10 client.25310 handle_command_reply: tid=0 2024-03-14T09:53:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.145+0000 7f732cb45b80 1 client.25310 shutdown 2024-03-14T09:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.145+0000 7f732cb45b80 1 -- 172.21.15.78:0/385254780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5100 msgr2=0x5581393e1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.145+0000 7f732cb45b80 1 --2- 172.21.15.78:0/385254780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5100 0x5581393e1020 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7300000c00 tx=0x7f7300009480 comp rx=0 tx=0).stop 2024-03-14T09:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f732cb45b80 1 -- 172.21.15.78:0/385254780 shutdown_connections 2024-03-14T09:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f732cb45b80 1 --2- 172.21.15.78:0/385254780 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x558139441460 0x55813943ddc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f732cb45b80 1 --2- 172.21.15.78:0/385254780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5581393d5f90 0x5581393e2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f732cb45b80 1 --2- 172.21.15.78:0/385254780 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5581393d47a0 0x5581393e0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f732cb45b80 1 --2- 172.21.15.78:0/385254780 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5581393d5100 0x5581393e1020 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f72f3e00700 0 client.25310 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f732cb45b80 1 -- 172.21.15.78:0/385254780 >> 172.21.15.78:0/385254780 conn(0x55813934a1d0 msgr2=0x5581393de3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f732cb45b80 1 -- 172.21.15.78:0/385254780 shutdown_connections 2024-03-14T09:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f732cb45b80 1 -- 172.21.15.78:0/385254780 wait complete. 2024-03-14T09:53:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f732cb45b80 20 client.25310 trim_cache size 0 max 16384 2024-03-14T09:53:43.153 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:53:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "83662c4e-e1e7-11ee-95c9-87774f69a715", 2024-03-14T09:53:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 2, 2024-03-14T09:53:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24527, 2024-03-14T09:53:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-14T09:53:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-14T09:53:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-14T09:53:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 73.436707803000004, 2024-03-14T09:53:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2024-03-14T09:53:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 87, 2024-03-14T09:53:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 83, 2024-03-14T09:53:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 112.976250615 2024-03-14T09:53:43.153 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:53:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f7319000700 1 -- 172.21.15.78:0/4093429048 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f72f80a0b10 msgr2=0x7f72f80a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f7319000700 1 --2- 172.21.15.78:0/4093429048 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f72f80a0b10 0x7f72f80a2fd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f730c005bd0 tx=0x7f730c008c10 comp rx=0 tx=0).stop 2024-03-14T09:53:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f7319000700 1 -- 172.21.15.78:0/4093429048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7314100d50 msgr2=0x7f7314199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f7319000700 1 --2- 172.21.15.78:0/4093429048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7314100d50 0x7f7314199420 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f730400e970 tx=0x7f730400ec80 comp rx=0 tx=0).stop 2024-03-14T09:53:43.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f7319000700 1 -- 172.21.15.78:0/4093429048 shutdown_connections 2024-03-14T09:53:43.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f7319000700 1 --2- 172.21.15.78:0/4093429048 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f72f80a0b10 0x7f72f80a2fd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f7319000700 1 --2- 172.21.15.78:0/4093429048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f73141028e0 0x7f73141a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f7319000700 1 --2- 172.21.15.78:0/4093429048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7314100d50 0x7f7314199420 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f7319000700 1 --2- 172.21.15.78:0/4093429048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7314101f40 0x7f7314199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.146+0000 7f7319000700 1 -- 172.21.15.78:0/4093429048 >> 172.21.15.78:0/4093429048 conn(0x7f73140fc2c0 msgr2=0x7f7314072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.147+0000 7f7319000700 1 -- 172.21.15.78:0/4093429048 shutdown_connections 2024-03-14T09:53:43.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.147+0000 7f7319000700 1 -- 172.21.15.78:0/4093429048 wait complete. 2024-03-14T09:53:43.165 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-03-14T09:53:43.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-14T09:53:43.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-14T09:53:43.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:42 smithi152 ceph-mon[48279]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 10]}]': finished 2024-03-14T09:53:43.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:42 smithi152 ceph-mon[48279]: osdmap e86: 12 total, 12 up, 12 in 2024-03-14T09:53:43.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:42 smithi152 ceph-mon[48279]: mgrmap e18: x(active, since 7m), standbys: z, y 2024-03-14T09:53:43.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:42 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2080832632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:43.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:42 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1615659140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:43.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-14T09:53:43.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-14T09:53:43.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:42 smithi132 ceph-mon[47959]: from='mgr.14152 172.21.15.78:0/2823445952' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 10]}]': finished 2024-03-14T09:53:43.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:42 smithi132 ceph-mon[47959]: osdmap e86: 12 total, 12 up, 12 in 2024-03-14T09:53:43.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:42 smithi132 ceph-mon[47959]: mgrmap e18: x(active, since 7m), standbys: z, y 2024-03-14T09:53:43.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:42 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2080832632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:43.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:42 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1615659140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:43.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.355+0000 7f7f4f800700 1 -- 172.21.15.78:0/1214144725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48074df0 msgr2=0x7f7f48073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.355+0000 7f7f4f800700 1 --2- 172.21.15.78:0/1214144725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48074df0 0x7f7f48073240 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c009990 tx=0x7f7f3c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.355+0000 7f7f4f800700 1 -- 172.21.15.78:0/1214144725 shutdown_connections 2024-03-14T09:53:43.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.355+0000 7f7f4f800700 1 --2- 172.21.15.78:0/1214144725 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f48108c00 0x7f7f4810b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.356+0000 7f7f4f800700 1 --2- 172.21.15.78:0/1214144725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48074df0 0x7f7f48073240 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.356+0000 7f7f4f800700 1 --2- 172.21.15.78:0/1214144725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f48073810 0x7f7f48073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.356+0000 7f7f4f800700 1 -- 172.21.15.78:0/1214144725 >> 172.21.15.78:0/1214144725 conn(0x7f7f480fc2f0 msgr2=0x7f7f480fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.356+0000 7f7f4f800700 1 -- 172.21.15.78:0/1214144725 shutdown_connections 2024-03-14T09:53:43.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.356+0000 7f7f4f800700 1 -- 172.21.15.78:0/1214144725 wait complete. 2024-03-14T09:53:43.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.357+0000 7f7f4f800700 1 Processor -- start 2024-03-14T09:53:43.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.357+0000 7f7f4f800700 1 -- start start 2024-03-14T09:53:43.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.357+0000 7f7f4f800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48073810 0x7f7f481992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.357+0000 7f7f4f800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f48074df0 0x7f7f48199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.357+0000 7f7f4f800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f48108c00 0x7f7f481a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.357+0000 7f7f4f800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f481a0e10 con 0x7f7f48073810 2024-03-14T09:53:43.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.357+0000 7f7f4f800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f481a0f50 con 0x7f7f48108c00 2024-03-14T09:53:43.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.357+0000 7f7f4f800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f481a10e0 con 0x7f7f48074df0 2024-03-14T09:53:43.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.357+0000 7f7f4ce00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48073810 0x7f7f481992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.357+0000 7f7f4ce00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48073810 0x7f7f481992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52672/0 (socket says 172.21.15.78:52672) 2024-03-14T09:53:43.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.357+0000 7f7f4ce00700 1 -- 172.21.15.78:0/2844507703 learned_addr learned my addr 172.21.15.78:0/2844507703 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:43.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.357+0000 7f7f47e00700 1 --2- 172.21.15.78:0/2844507703 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f48074df0 0x7f7f48199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.357+0000 7f7f4d800700 1 --2- 172.21.15.78:0/2844507703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f48108c00 0x7f7f481a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.358+0000 7f7f4ce00700 1 -- 172.21.15.78:0/2844507703 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f48074df0 msgr2=0x7f7f48199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.358+0000 7f7f4ce00700 1 --2- 172.21.15.78:0/2844507703 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f48074df0 0x7f7f48199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.358+0000 7f7f4ce00700 1 -- 172.21.15.78:0/2844507703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f48108c00 msgr2=0x7f7f481a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.358+0000 7f7f4ce00700 1 --2- 172.21.15.78:0/2844507703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f48108c00 0x7f7f481a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.358+0000 7f7f4ce00700 1 -- 172.21.15.78:0/2844507703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f34009520 con 0x7f7f48073810 2024-03-14T09:53:43.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.358+0000 7f7f4ce00700 1 --2- 172.21.15.78:0/2844507703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48073810 0x7f7f481992e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c005910 tx=0x7f7f3c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.358+0000 7f7f45600700 1 -- 172.21.15.78:0/2844507703 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c01d070 con 0x7f7f48073810 2024-03-14T09:53:43.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.358+0000 7f7f45600700 1 -- 172.21.15.78:0/2844507703 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f3c021410 con 0x7f7f48073810 2024-03-14T09:53:43.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.358+0000 7f7f4f800700 1 -- 172.21.15.78:0/2844507703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3c0095f0 con 0x7f7f48073810 2024-03-14T09:53:43.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.358+0000 7f7f45600700 1 -- 172.21.15.78:0/2844507703 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c017650 con 0x7f7f48073810 2024-03-14T09:53:43.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.358+0000 7f7f4f800700 1 -- 172.21.15.78:0/2844507703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f481a15f0 con 0x7f7f48073810 2024-03-14T09:53:43.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.361+0000 7f7f45600700 1 -- 172.21.15.78:0/2844507703 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f7f3c021580 con 0x7f7f48073810 2024-03-14T09:53:43.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.363+0000 7f7f63355b80 1 Processor -- start 2024-03-14T09:53:43.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.363+0000 7f7f63355b80 1 -- start start 2024-03-14T09:53:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.363+0000 7f7f45600700 1 --2- 172.21.15.78:0/2844507703 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7f300a0a90 0x7f7f300a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.363+0000 7f7f45600700 1 -- 172.21.15.78:0/2844507703 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f7f3c0cd7c0 con 0x7f7f48073810 2024-03-14T09:53:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.363+0000 7f7f63355b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d777a0 0x562d38d77bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f63355b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78100 0x562d38d78a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f63355b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78f90 0x562d38d79410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f63355b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38cd0940 con 0x562d38d78100 2024-03-14T09:53:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f63355b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38cce300 con 0x562d38d78f90 2024-03-14T09:53:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f63355b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38cc9bf0 con 0x562d38d777a0 2024-03-14T09:53:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f4d800700 1 --2- 172.21.15.78:0/2844507703 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7f300a0a90 0x7f7f300a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f47e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78100 0x562d38d78a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f47e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78100 0x562d38d78a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52680/0 (socket says 172.21.15.78:52680) 2024-03-14T09:53:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f47e00700 1 -- 172.21.15.78:0/3876736618 learned_addr learned my addr 172.21.15.78:0/3876736618 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f4ce00700 1 --2- 172.21.15.78:0/3876736618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d777a0 0x562d38d77bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f4d800700 1 --2- 172.21.15.78:0/3876736618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78f90 0x562d38d79410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f47e00700 1 -- 172.21.15.78:0/3876736618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d777a0 msgr2=0x562d38d77bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f47e00700 1 --2- 172.21.15.78:0/3876736618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d777a0 0x562d38d77bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f47e00700 1 -- 172.21.15.78:0/3876736618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78f90 msgr2=0x562d38d79410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f47e00700 1 --2- 172.21.15.78:0/3876736618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78f90 0x562d38d79410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f47e00700 1 -- 172.21.15.78:0/3876736618 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d38b4b360 con 0x562d38d78100 2024-03-14T09:53:43.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.364+0000 7f7f4d800700 1 --2- 172.21.15.78:0/2844507703 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7f300a0a90 0x7f7f300a2f50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7f400076f0 tx=0x7f7f40007ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.365+0000 7f7f47e00700 1 --2- 172.21.15.78:0/3876736618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78100 0x562d38d78a00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7f34009eb0 tx=0x7f7f34011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c095c0e84d0b829 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.365+0000 7f7f3a000700 1 -- 172.21.15.78:0/3876736618 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f34019630 con 0x562d38d78100 2024-03-14T09:53:43.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.365+0000 7f7f3a000700 1 -- 172.21.15.78:0/3876736618 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f34019c50 con 0x562d38d78100 2024-03-14T09:53:43.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.365+0000 7f7f3a000700 1 -- 172.21.15.78:0/3876736618 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3400ed30 con 0x562d38d78100 2024-03-14T09:53:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.365+0000 7f7f63355b80 1 -- 172.21.15.78:0/3876736618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78100 msgr2=0x562d38d78a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.365+0000 7f7f63355b80 1 --2- 172.21.15.78:0/3876736618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78100 0x562d38d78a00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7f34009eb0 tx=0x7f7f34011720 comp rx=0 tx=0).stop 2024-03-14T09:53:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.366+0000 7f7f63355b80 1 -- 172.21.15.78:0/3876736618 shutdown_connections 2024-03-14T09:53:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.366+0000 7f7f63355b80 1 --2- 172.21.15.78:0/3876736618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78f90 0x562d38d79410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.366+0000 7f7f63355b80 1 --2- 172.21.15.78:0/3876736618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d777a0 0x562d38d77bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.366+0000 7f7f63355b80 1 --2- 172.21.15.78:0/3876736618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78100 0x562d38d78a00 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.366+0000 7f7f63355b80 1 -- 172.21.15.78:0/3876736618 >> 172.21.15.78:0/3876736618 conn(0x562d38ced1d0 msgr2=0x562d38d72580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.366+0000 7f7f63355b80 1 -- 172.21.15.78:0/3876736618 shutdown_connections 2024-03-14T09:53:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.366+0000 7f7f63355b80 1 -- 172.21.15.78:0/3876736618 wait complete. 2024-03-14T09:53:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.366+0000 7f7f63355b80 1 -- start start 2024-03-14T09:53:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.366+0000 7f7f63355b80 1 Processor -- start 2024-03-14T09:53:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.367+0000 7f7f63355b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d777a0 0x562d38d816b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.367+0000 7f7f63355b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78100 0x562d38d81bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.367+0000 7f7f63355b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78f90 0x562d38d88be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.367+0000 7f7f63355b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38c65d30 con 0x562d38d78100 2024-03-14T09:53:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.367+0000 7f7f63355b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38c659d0 con 0x562d38d78f90 2024-03-14T09:53:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.367+0000 7f7f63355b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38b4d650 con 0x562d38d777a0 2024-03-14T09:53:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.367+0000 7f7f63355b80 10 client.0 _ll_register_callbacks cb 0x562d38ad1350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:53:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.367+0000 7f7f47e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78100 0x562d38d81bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.367+0000 7f7f4d800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78f90 0x562d38d88be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.367+0000 7f7f47e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78100 0x562d38d81bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52692/0 (socket says 172.21.15.78:52692) 2024-03-14T09:53:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.367+0000 7f7f47e00700 1 -- 172.21.15.78:0/3953440363 learned_addr learned my addr 172.21.15.78:0/3953440363 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.367+0000 7f7f4ce00700 1 --2- 172.21.15.78:0/3953440363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d777a0 0x562d38d816b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.367+0000 7f7f47e00700 1 -- 172.21.15.78:0/3953440363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d777a0 msgr2=0x562d38d816b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.367+0000 7f7f47e00700 1 --2- 172.21.15.78:0/3953440363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d777a0 0x562d38d816b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.367+0000 7f7f47e00700 1 -- 172.21.15.78:0/3953440363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78f90 msgr2=0x562d38d88be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.367+0000 7f7f47e00700 1 --2- 172.21.15.78:0/3953440363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78f90 0x562d38d88be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.368+0000 7f7f47e00700 1 -- 172.21.15.78:0/3953440363 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f34009ae0 con 0x562d38d78100 2024-03-14T09:53:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.368+0000 7f7f4d800700 1 --2- 172.21.15.78:0/3953440363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78f90 0x562d38d88be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:43.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.368+0000 7f7f47e00700 1 --2- 172.21.15.78:0/3953440363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78100 0x562d38d81bf0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7f34011a70 tx=0x7f7f3400e780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b686eff9d22406 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.368+0000 7f7f2ea00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:53:43.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.368+0000 7f7f2ea00700 1 -- 172.21.15.78:0/3953440363 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f34019430 con 0x562d38d78100 2024-03-14T09:53:43.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.368+0000 7f7f63355b80 1 -- 172.21.15.78:0/3953440363 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562d38d6a6d0 con 0x562d38d78100 2024-03-14T09:53:43.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.368+0000 7f7f2ea00700 1 -- 172.21.15.78:0/3953440363 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f7f34019590 con 0x562d38d78100 2024-03-14T09:53:43.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.368+0000 7f7f2ea00700 1 -- 172.21.15.78:0/3953440363 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f34019750 con 0x562d38d78100 2024-03-14T09:53:43.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.368+0000 7f7f63355b80 10 client.15312 fetch_fsmap learned FSMap version 24 2024-03-14T09:53:43.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.368+0000 7f7f63355b80 1 -- 172.21.15.78:0/3953440363 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x562d38af7b10 con 0x562d38d78100 2024-03-14T09:53:43.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.369+0000 7f7f2ea00700 1 -- 172.21.15.78:0/3953440363 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562d38d6a6d0 con 0x562d38d78100 2024-03-14T09:53:43.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.369+0000 7f7f2ea00700 1 -- 172.21.15.78:0/3953440363 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7f340048a0 con 0x562d38d78100 2024-03-14T09:53:43.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.369+0000 7f7f63355b80 10 client.15312 fetch_fsmap finished waiting for FSMap version 24 2024-03-14T09:53:43.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.369+0000 7f7f63355b80 10 client.15312 resolve_mds: resolved name 'g' to daemon mds.g 2024-03-14T09:53:43.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.369+0000 7f7f63355b80 20 client.15312 populate_metadata read hostname 'smithi078' 2024-03-14T09:53:43.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.369+0000 7f7f63355b80 1 --2- 172.21.15.78:0/3953440363 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x562d38d8c790 0x562d38d91ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.369+0000 7f7f63355b80 4 client.15312 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:53:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.369+0000 7f7f63355b80 1 -- 172.21.15.78:0/3953440363 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562d38cc26b0 con 0x562d38d8c790 2024-03-14T09:53:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.369+0000 7f7f4ce00700 1 --2- 172.21.15.78:0/3953440363 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x562d38d8c790 0x562d38d91ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.370+0000 7f7f2ea00700 10 client.15312 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:53:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.370+0000 7f7f4ce00700 1 --2- 172.21.15.78:0/3953440363 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x562d38d8c790 0x562d38d91ab0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=32c9c41dedaec211 server_cookie=80ce8ef6eee2da4 in_seq=0 out_seq=0 2024-03-14T09:53:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.370+0000 7f7f2ea00700 1 -- 172.21.15.78:0/3953440363 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x562d38cc26b0 con 0x562d38d8c790 2024-03-14T09:53:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.370+0000 7f7f2ea00700 10 client.15312 handle_command_reply: tid=0 2024-03-14T09:53:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.370+0000 7f7f63355b80 1 client.15312 shutdown 2024-03-14T09:53:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.370+0000 7f7f63355b80 1 -- 172.21.15.78:0/3953440363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78100 msgr2=0x562d38d81bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.370+0000 7f7f63355b80 1 --2- 172.21.15.78:0/3953440363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78100 0x562d38d81bf0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7f34011a70 tx=0x7f7f3400e780 comp rx=0 tx=0).stop 2024-03-14T09:53:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.371+0000 7f7f63355b80 1 -- 172.21.15.78:0/3953440363 shutdown_connections 2024-03-14T09:53:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.371+0000 7f7f63355b80 1 --2- 172.21.15.78:0/3953440363 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x562d38d8c790 0x562d38d91ab0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.371+0000 7f7f63355b80 1 --2- 172.21.15.78:0/3953440363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78f90 0x562d38d88be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.371+0000 7f7f63355b80 1 --2- 172.21.15.78:0/3953440363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d777a0 0x562d38d816b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.371+0000 7f7f2ea00700 0 client.15312 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:53:43.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.371+0000 7f7f63355b80 1 --2- 172.21.15.78:0/3953440363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78100 0x562d38d81bf0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.371+0000 7f7f63355b80 1 -- 172.21.15.78:0/3953440363 >> 172.21.15.78:0/3953440363 conn(0x562d38ced1d0 msgr2=0x562d38d71bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.371+0000 7f7f63355b80 1 -- 172.21.15.78:0/3953440363 shutdown_connections 2024-03-14T09:53:43.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.371+0000 7f7f63355b80 1 -- 172.21.15.78:0/3953440363 wait complete. 2024-03-14T09:53:43.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.371+0000 7f7f63355b80 20 client.15312 trim_cache size 0 max 16384 2024-03-14T09:53:43.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.388+0000 7f7f63355b80 1 Processor -- start 2024-03-14T09:53:43.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.388+0000 7f7f63355b80 1 -- start start 2024-03-14T09:53:43.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.388+0000 7f7f63355b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d777a0 0x562d38d86b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.388+0000 7f7f63355b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78100 0x562d38d80bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.388+0000 7f7f63355b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78f90 0x562d38d79370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.388+0000 7f7f63355b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38cc6850 con 0x562d38d78f90 2024-03-14T09:53:43.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.388+0000 7f7f63355b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38c68de0 con 0x562d38d78100 2024-03-14T09:53:43.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.388+0000 7f7f63355b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38d89d60 con 0x562d38d777a0 2024-03-14T09:53:43.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.389+0000 7f7f47e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78100 0x562d38d80bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.389+0000 7f7f47e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78100 0x562d38d80bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:35922/0 (socket says 172.21.15.78:35922) 2024-03-14T09:53:43.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.389+0000 7f7f4d800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78f90 0x562d38d79370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.389+0000 7f7f47e00700 1 -- 172.21.15.78:0/4287769262 learned_addr learned my addr 172.21.15.78:0/4287769262 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:43.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.389+0000 7f7f4ce00700 1 --2- 172.21.15.78:0/4287769262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d777a0 0x562d38d86b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.389+0000 7f7f47e00700 1 -- 172.21.15.78:0/4287769262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d777a0 msgr2=0x562d38d86b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.389+0000 7f7f47e00700 1 --2- 172.21.15.78:0/4287769262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d777a0 0x562d38d86b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.389+0000 7f7f47e00700 1 -- 172.21.15.78:0/4287769262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78f90 msgr2=0x562d38d79370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.389+0000 7f7f47e00700 1 --2- 172.21.15.78:0/4287769262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78f90 0x562d38d79370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.389+0000 7f7f47e00700 1 -- 172.21.15.78:0/4287769262 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d38c8fe50 con 0x562d38d78100 2024-03-14T09:53:43.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.389+0000 7f7f47e00700 1 --2- 172.21.15.78:0/4287769262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78100 0x562d38d80bc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7f34009dd0 tx=0x7f7f34011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be10c46b055e5814 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.389+0000 7f7f3aa00700 1 -- 172.21.15.78:0/4287769262 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f34011aa0 con 0x562d38d78100 2024-03-14T09:53:43.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.389+0000 7f7f3aa00700 1 -- 172.21.15.78:0/4287769262 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f3400e7f0 con 0x562d38d78100 2024-03-14T09:53:43.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.390+0000 7f7f3aa00700 1 -- 172.21.15.78:0/4287769262 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f34005040 con 0x562d38d78100 2024-03-14T09:53:43.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.390+0000 7f7f63355b80 1 -- 172.21.15.78:0/4287769262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78100 msgr2=0x562d38d80bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.390+0000 7f7f63355b80 1 --2- 172.21.15.78:0/4287769262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78100 0x562d38d80bc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7f34009dd0 tx=0x7f7f34011720 comp rx=0 tx=0).stop 2024-03-14T09:53:43.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.390+0000 7f7f63355b80 1 -- 172.21.15.78:0/4287769262 shutdown_connections 2024-03-14T09:53:43.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.390+0000 7f7f63355b80 1 --2- 172.21.15.78:0/4287769262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78f90 0x562d38d79370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.390+0000 7f7f63355b80 1 --2- 172.21.15.78:0/4287769262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d777a0 0x562d38d86b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.390+0000 7f7f63355b80 1 --2- 172.21.15.78:0/4287769262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d78100 0x562d38d80bc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.390+0000 7f7f63355b80 1 -- 172.21.15.78:0/4287769262 >> 172.21.15.78:0/4287769262 conn(0x562d38ced1d0 msgr2=0x562d38d72550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.390+0000 7f7f63355b80 1 -- 172.21.15.78:0/4287769262 shutdown_connections 2024-03-14T09:53:43.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.390+0000 7f7f63355b80 1 -- 172.21.15.78:0/4287769262 wait complete. 2024-03-14T09:53:43.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.391+0000 7f7f63355b80 1 -- start start 2024-03-14T09:53:43.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.391+0000 7f7f63355b80 1 Processor -- start 2024-03-14T09:53:43.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.391+0000 7f7f63355b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d777a0 0x562d38d85bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.391+0000 7f7f63355b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d78100 0x562d38d86110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.391+0000 7f7f63355b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78f90 0x562d38d86650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.391+0000 7f7f63355b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38d86c20 con 0x562d38d78f90 2024-03-14T09:53:43.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.391+0000 7f7f63355b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38d86d90 con 0x562d38d777a0 2024-03-14T09:53:43.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.391+0000 7f7f63355b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d38d86f50 con 0x562d38d78100 2024-03-14T09:53:43.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.391+0000 7f7f63355b80 10 client.0 _ll_register_callbacks cb 0x562d38ad1350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:53:43.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.391+0000 7f7f4d800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78f90 0x562d38d86650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.392+0000 7f7f47e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d78100 0x562d38d86110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.392+0000 7f7f4d800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78f90 0x562d38d86650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52698/0 (socket says 172.21.15.78:52698) 2024-03-14T09:53:43.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.392+0000 7f7f4d800700 1 -- 172.21.15.78:0/3054221233 learned_addr learned my addr 172.21.15.78:0/3054221233 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:43.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.392+0000 7f7f4ce00700 1 --2- 172.21.15.78:0/3054221233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d777a0 0x562d38d85bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.392+0000 7f7f47e00700 1 -- 172.21.15.78:0/3054221233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d777a0 msgr2=0x562d38d85bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.392+0000 7f7f47e00700 1 --2- 172.21.15.78:0/3054221233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d777a0 0x562d38d85bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.392+0000 7f7f47e00700 1 -- 172.21.15.78:0/3054221233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78f90 msgr2=0x562d38d86650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.392+0000 7f7f47e00700 1 --2- 172.21.15.78:0/3054221233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78f90 0x562d38d86650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.392+0000 7f7f47e00700 1 -- 172.21.15.78:0/3054221233 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562d38af7b10 con 0x562d38d78100 2024-03-14T09:53:43.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.392+0000 7f7f4d800700 1 --2- 172.21.15.78:0/3054221233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78f90 0x562d38d86650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:43.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.392+0000 7f7f47e00700 1 --2- 172.21.15.78:0/3054221233 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d78100 0x562d38d86110 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7f34019860 tx=0x7f7f3400e710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=addd29edee65a854 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.392+0000 7f7f2ea00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:53:43.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.393+0000 7f7f2ea00700 1 -- 172.21.15.78:0/3054221233 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f34011a70 con 0x562d38d78100 2024-03-14T09:53:43.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.393+0000 7f7f63355b80 1 -- 172.21.15.78:0/3054221233 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7f34019c50 con 0x562d38d78100 2024-03-14T09:53:43.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.393+0000 7f7f2ea00700 1 -- 172.21.15.78:0/3054221233 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f7f34000e40 con 0x562d38d78100 2024-03-14T09:53:43.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.393+0000 7f7f2ea00700 1 -- 172.21.15.78:0/3054221233 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f34019990 con 0x562d38d78100 2024-03-14T09:53:43.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.393+0000 7f7f63355b80 10 client.25330 fetch_fsmap learned FSMap version 24 2024-03-14T09:53:43.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.393+0000 7f7f63355b80 1 -- 172.21.15.78:0/3054221233 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x562d38b4fc80 con 0x562d38d78100 2024-03-14T09:53:43.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.393+0000 7f7f2ea00700 1 -- 172.21.15.78:0/3054221233 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f34019c50 con 0x562d38d78100 2024-03-14T09:53:43.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.393+0000 7f7f2ea00700 1 -- 172.21.15.78:0/3054221233 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7f34003980 con 0x562d38d78100 2024-03-14T09:53:43.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.393+0000 7f7f63355b80 10 client.25330 fetch_fsmap finished waiting for FSMap version 24 2024-03-14T09:53:43.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.393+0000 7f7f63355b80 10 client.25330 resolve_mds: resolved name 'g' to daemon mds.g 2024-03-14T09:53:43.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.393+0000 7f7f63355b80 20 client.25330 populate_metadata read hostname 'smithi078' 2024-03-14T09:53:43.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.393+0000 7f7f63355b80 1 --2- 172.21.15.78:0/3054221233 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x562d38de35f0 0x562d38ddffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.393+0000 7f7f63355b80 4 client.25330 mds_command: new command op to 24538 tid=0[{"prefix": "status"}] 2024-03-14T09:53:43.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.393+0000 7f7f63355b80 1 -- 172.21.15.78:0/3054221233 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562d38b18ca0 con 0x562d38de35f0 2024-03-14T09:53:43.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.394+0000 7f7f4ce00700 1 --2- 172.21.15.78:0/3054221233 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x562d38de35f0 0x562d38ddffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.394+0000 7f7f2ea00700 10 client.25330 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:53:43.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.394+0000 7f7f4ce00700 1 --2- 172.21.15.78:0/3054221233 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x562d38de35f0 0x562d38ddffa0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d1630eaf372932da server_cookie=2b4bd61a527be139 in_seq=0 out_seq=0 2024-03-14T09:53:43.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.394+0000 7f7f2ea00700 1 -- 172.21.15.78:0/3054221233 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x562d38b18ca0 con 0x562d38de35f0 2024-03-14T09:53:43.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.394+0000 7f7f2ea00700 10 client.25330 handle_command_reply: tid=0 2024-03-14T09:53:43.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.394+0000 7f7f63355b80 1 client.25330 shutdown 2024-03-14T09:53:43.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.394+0000 7f7f63355b80 1 -- 172.21.15.78:0/3054221233 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d78100 msgr2=0x562d38d86110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.394+0000 7f7f63355b80 1 --2- 172.21.15.78:0/3054221233 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d78100 0x562d38d86110 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7f34019860 tx=0x7f7f3400e710 comp rx=0 tx=0).stop 2024-03-14T09:53:43.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f63355b80 1 -- 172.21.15.78:0/3054221233 shutdown_connections 2024-03-14T09:53:43.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f63355b80 1 --2- 172.21.15.78:0/3054221233 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x562d38de35f0 0x562d38ddffa0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f63355b80 1 --2- 172.21.15.78:0/3054221233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562d38d777a0 0x562d38d85bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f63355b80 1 --2- 172.21.15.78:0/3054221233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562d38d78f90 0x562d38d86650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f63355b80 1 --2- 172.21.15.78:0/3054221233 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562d38d78100 0x562d38d86110 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f2ea00700 0 client.25330 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:53:43.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f63355b80 1 -- 172.21.15.78:0/3054221233 >> 172.21.15.78:0/3054221233 conn(0x562d38ced1d0 msgr2=0x562d38d72480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f63355b80 1 -- 172.21.15.78:0/3054221233 shutdown_connections 2024-03-14T09:53:43.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f63355b80 1 -- 172.21.15.78:0/3054221233 wait complete. 2024-03-14T09:53:43.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f63355b80 20 client.25330 trim_cache size 0 max 16384 2024-03-14T09:53:43.403 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:53:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "83662c4e-e1e7-11ee-95c9-87774f69a715", 2024-03-14T09:53:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 1, 2024-03-14T09:53:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24538, 2024-03-14T09:53:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-14T09:53:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-14T09:53:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-14T09:53:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 74.689617953999999, 2024-03-14T09:53:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2024-03-14T09:53:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 87, 2024-03-14T09:53:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 83, 2024-03-14T09:53:43.403 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 108.936411357 2024-03-14T09:53:43.403 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:53:43.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f4f800700 1 -- 172.21.15.78:0/2844507703 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7f300a0a90 msgr2=0x7f7f300a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f4f800700 1 --2- 172.21.15.78:0/2844507703 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7f300a0a90 0x7f7f300a2f50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7f400076f0 tx=0x7f7f40007ab0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f4f800700 1 -- 172.21.15.78:0/2844507703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48073810 msgr2=0x7f7f481992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f4f800700 1 --2- 172.21.15.78:0/2844507703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48073810 0x7f7f481992e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c005910 tx=0x7f7f3c009580 comp rx=0 tx=0).stop 2024-03-14T09:53:43.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f4f800700 1 -- 172.21.15.78:0/2844507703 shutdown_connections 2024-03-14T09:53:43.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f4f800700 1 --2- 172.21.15.78:0/2844507703 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f7f300a0a90 0x7f7f300a2f50 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f4f800700 1 --2- 172.21.15.78:0/2844507703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f48108c00 0x7f7f481a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f4f800700 1 --2- 172.21.15.78:0/2844507703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48073810 0x7f7f481992e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f4f800700 1 --2- 172.21.15.78:0/2844507703 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f48074df0 0x7f7f48199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.395+0000 7f7f4f800700 1 -- 172.21.15.78:0/2844507703 >> 172.21.15.78:0/2844507703 conn(0x7f7f480fc2f0 msgr2=0x7f7f48104b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.396+0000 7f7f4f800700 1 -- 172.21.15.78:0/2844507703 shutdown_connections 2024-03-14T09:53:43.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.396+0000 7f7f4f800700 1 -- 172.21.15.78:0/2844507703 wait complete. 2024-03-14T09:53:43.414 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-14T09:53:43.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.607+0000 7f0c48a00700 1 -- 172.21.15.78:0/3775459766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c441040e0 msgr2=0x7f0c441064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.607+0000 7f0c48a00700 1 --2- 172.21.15.78:0/3775459766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c441040e0 0x7f0c441064d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0c30009990 tx=0x7f0c30009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.607+0000 7f0c48a00700 1 -- 172.21.15.78:0/3775459766 shutdown_connections 2024-03-14T09:53:43.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.607+0000 7f0c48a00700 1 --2- 172.21.15.78:0/3775459766 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c44106a10 0x7f0c44108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.607+0000 7f0c48a00700 1 --2- 172.21.15.78:0/3775459766 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c441017b0 0x7f0c44103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.607+0000 7f0c48a00700 1 --2- 172.21.15.78:0/3775459766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c441040e0 0x7f0c441064d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.607+0000 7f0c48a00700 1 -- 172.21.15.78:0/3775459766 >> 172.21.15.78:0/3775459766 conn(0x7f0c440fb1a0 msgr2=0x7f0c440fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.607+0000 7f0c48a00700 1 -- 172.21.15.78:0/3775459766 shutdown_connections 2024-03-14T09:53:43.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.607+0000 7f0c48a00700 1 -- 172.21.15.78:0/3775459766 wait complete. 2024-03-14T09:53:43.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.608+0000 7f0c48a00700 1 Processor -- start 2024-03-14T09:53:43.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.608+0000 7f0c48a00700 1 -- start start 2024-03-14T09:53:43.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.608+0000 7f0c48a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c441017b0 0x7f0c441992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.608+0000 7f0c48a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c441040e0 0x7f0c441997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.608+0000 7f0c48a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c44106a10 0x7f0c441a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.608+0000 7f0c48a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c441a0de0 con 0x7f0c441017b0 2024-03-14T09:53:43.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.608+0000 7f0c48a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c441a0f20 con 0x7f0c44106a10 2024-03-14T09:53:43.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.608+0000 7f0c48a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c441a10b0 con 0x7f0c441040e0 2024-03-14T09:53:43.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.609+0000 7f0c41e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c441017b0 0x7f0c441992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.609+0000 7f0c41e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c441017b0 0x7f0c441992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52702/0 (socket says 172.21.15.78:52702) 2024-03-14T09:53:43.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.609+0000 7f0c41e00700 1 -- 172.21.15.78:0/1837281551 learned_addr learned my addr 172.21.15.78:0/1837281551 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:43.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.609+0000 7f0c41400700 1 --2- 172.21.15.78:0/1837281551 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c441040e0 0x7f0c441997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.609+0000 7f0c42800700 1 --2- 172.21.15.78:0/1837281551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c44106a10 0x7f0c441a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.609+0000 7f0c41e00700 1 -- 172.21.15.78:0/1837281551 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c441040e0 msgr2=0x7f0c441997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.609+0000 7f0c41e00700 1 --2- 172.21.15.78:0/1837281551 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c441040e0 0x7f0c441997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.609+0000 7f0c41e00700 1 -- 172.21.15.78:0/1837281551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c44106a10 msgr2=0x7f0c441a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.609+0000 7f0c41e00700 1 --2- 172.21.15.78:0/1837281551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c44106a10 0x7f0c441a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.609+0000 7f0c41e00700 1 -- 172.21.15.78:0/1837281551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c34009520 con 0x7f0c441017b0 2024-03-14T09:53:43.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.609+0000 7f0c41e00700 1 --2- 172.21.15.78:0/1837281551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c441017b0 0x7f0c441992b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400eb30 tx=0x7f0c3400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.610+0000 7f0c3aa00700 1 -- 172.21.15.78:0/1837281551 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c34009880 con 0x7f0c441017b0 2024-03-14T09:53:43.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.610+0000 7f0c3aa00700 1 -- 172.21.15.78:0/1837281551 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c340044e0 con 0x7f0c441017b0 2024-03-14T09:53:43.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.610+0000 7f0c48a00700 1 -- 172.21.15.78:0/1837281551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c300095f0 con 0x7f0c441017b0 2024-03-14T09:53:43.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.610+0000 7f0c3aa00700 1 -- 172.21.15.78:0/1837281551 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c34003d80 con 0x7f0c441017b0 2024-03-14T09:53:43.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.610+0000 7f0c48a00700 1 -- 172.21.15.78:0/1837281551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c441a1760 con 0x7f0c441017b0 2024-03-14T09:53:43.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.612+0000 7f0c3aa00700 1 -- 172.21.15.78:0/1837281551 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f0c34003f20 con 0x7f0c441017b0 2024-03-14T09:53:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.614+0000 7f0c3aa00700 1 --2- 172.21.15.78:0/1837281551 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0c280a0b10 0x7f0c280a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.614+0000 7f0c5c619b80 1 Processor -- start 2024-03-14T09:53:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.614+0000 7f0c3aa00700 1 -- 172.21.15.78:0/1837281551 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f0c34014070 con 0x7f0c441017b0 2024-03-14T09:53:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.614+0000 7f0c5c619b80 1 -- start start 2024-03-14T09:53:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.614+0000 7f0c41400700 1 --2- 172.21.15.78:0/1837281551 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0c280a0b10 0x7f0c280a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.614+0000 7f0c5c619b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 0x559971a56bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.614+0000 7f0c5c619b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 0x559971a57a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.614+0000 7f0c5c619b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 0x559971a58410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.614+0000 7f0c5c619b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599719af940 con 0x559971a57100 2024-03-14T09:53:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.614+0000 7f0c5c619b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599719ad300 con 0x559971a57f90 2024-03-14T09:53:43.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.614+0000 7f0c5c619b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599719a8bf0 con 0x559971a567a0 2024-03-14T09:53:43.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.615+0000 7f0c41e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 0x559971a56bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.615+0000 7f0c42800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 0x559971a58410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.615+0000 7f0c41e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 0x559971a56bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:51952/0 (socket says 172.21.15.78:51952) 2024-03-14T09:53:43.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.615+0000 7f0c41e00700 1 -- 172.21.15.78:0/464981690 learned_addr learned my addr 172.21.15.78:0/464981690 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:43.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.615+0000 7f0c41400700 1 --2- 172.21.15.78:0/464981690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 0x559971a57a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.615+0000 7f0c41e00700 1 -- 172.21.15.78:0/464981690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 msgr2=0x559971a58410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.615+0000 7f0c41e00700 1 --2- 172.21.15.78:0/464981690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 0x559971a58410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.615+0000 7f0c41e00700 1 -- 172.21.15.78:0/464981690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 msgr2=0x559971a57a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.615+0000 7f0c41e00700 1 --2- 172.21.15.78:0/464981690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 0x559971a57a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.615+0000 7f0c41e00700 1 -- 172.21.15.78:0/464981690 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55997182a360 con 0x559971a567a0 2024-03-14T09:53:43.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.615+0000 7f0c41400700 1 --2- 172.21.15.78:0/1837281551 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0c280a0b10 0x7f0c280a2fd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0c3000b2e0 tx=0x7f0c3000b910 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.615+0000 7f0c41e00700 1 --2- 172.21.15.78:0/464981690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 0x559971a56bc0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0c34099b10 tx=0x7f0c34099560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ed9abf5cdeaf868 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.615+0000 7f0c2f000700 1 -- 172.21.15.78:0/464981690 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3409a280 con 0x559971a567a0 2024-03-14T09:53:43.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.616+0000 7f0c2f000700 1 -- 172.21.15.78:0/464981690 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c34099b80 con 0x559971a567a0 2024-03-14T09:53:43.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.616+0000 7f0c2f000700 1 -- 172.21.15.78:0/464981690 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3401fde0 con 0x559971a567a0 2024-03-14T09:53:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.616+0000 7f0c5c619b80 1 -- 172.21.15.78:0/464981690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 msgr2=0x559971a56bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.616+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/464981690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 0x559971a56bc0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0c34099b10 tx=0x7f0c34099560 comp rx=0 tx=0).stop 2024-03-14T09:53:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.616+0000 7f0c5c619b80 1 -- 172.21.15.78:0/464981690 shutdown_connections 2024-03-14T09:53:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.616+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/464981690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 0x559971a58410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.616+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/464981690 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 0x559971a56bc0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.616+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/464981690 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 0x559971a57a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.616+0000 7f0c5c619b80 1 -- 172.21.15.78:0/464981690 >> 172.21.15.78:0/464981690 conn(0x5599719cc1d0 msgr2=0x559971a51580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.616+0000 7f0c5c619b80 1 -- 172.21.15.78:0/464981690 shutdown_connections 2024-03-14T09:53:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.616+0000 7f0c5c619b80 1 -- 172.21.15.78:0/464981690 wait complete. 2024-03-14T09:53:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.617+0000 7f0c5c619b80 1 -- start start 2024-03-14T09:53:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.617+0000 7f0c5c619b80 1 Processor -- start 2024-03-14T09:53:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c5c619b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 0x559971a60640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c5c619b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 0x559971a60b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c5c619b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 0x559971a67bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c5c619b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559971944d30 con 0x559971a57100 2024-03-14T09:53:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c5c619b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599719449d0 con 0x559971a57f90 2024-03-14T09:53:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c5c619b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55997182c650 con 0x559971a567a0 2024-03-14T09:53:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c5c619b80 10 client.0 _ll_register_callbacks cb 0x5599717b0350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:53:43.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c41400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 0x559971a60b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c41400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 0x559971a60b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52724/0 (socket says 172.21.15.78:52724) 2024-03-14T09:53:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c41400700 1 -- 172.21.15.78:0/1952523163 learned_addr learned my addr 172.21.15.78:0/1952523163 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c42800700 1 --2- 172.21.15.78:0/1952523163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 0x559971a67bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c41e00700 1 --2- 172.21.15.78:0/1952523163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 0x559971a60640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c41400700 1 -- 172.21.15.78:0/1952523163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 msgr2=0x559971a60640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c41400700 1 --2- 172.21.15.78:0/1952523163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 0x559971a60640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c41400700 1 -- 172.21.15.78:0/1952523163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 msgr2=0x559971a67bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c41400700 1 --2- 172.21.15.78:0/1952523163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 0x559971a67bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c41400700 1 -- 172.21.15.78:0/1952523163 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c441a1760 con 0x559971a57100 2024-03-14T09:53:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.618+0000 7f0c41400700 1 --2- 172.21.15.78:0/1952523163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 0x559971a60b80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0c300098e0 tx=0x7f0c30009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4167e3df91a486cb server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.619+0000 7f0c23400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:53:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.619+0000 7f0c23400700 1 -- 172.21.15.78:0/1952523163 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c300056e0 con 0x559971a57100 2024-03-14T09:53:43.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.619+0000 7f0c23400700 1 -- 172.21.15.78:0/1952523163 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f0c3002c040 con 0x559971a57100 2024-03-14T09:53:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.619+0000 7f0c5c619b80 1 -- 172.21.15.78:0/1952523163 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559971a496d0 con 0x559971a57100 2024-03-14T09:53:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.619+0000 7f0c23400700 1 -- 172.21.15.78:0/1952523163 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c30004b70 con 0x559971a57100 2024-03-14T09:53:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.619+0000 7f0c5c619b80 10 client.15330 fetch_fsmap learned FSMap version 24 2024-03-14T09:53:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.619+0000 7f0c5c619b80 1 -- 172.21.15.78:0/1952523163 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5599717d6b10 con 0x559971a57100 2024-03-14T09:53:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.619+0000 7f0c23400700 1 -- 172.21.15.78:0/1952523163 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559971a496d0 con 0x559971a57100 2024-03-14T09:53:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.619+0000 7f0c23400700 1 -- 172.21.15.78:0/1952523163 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0c30030450 con 0x559971a57100 2024-03-14T09:53:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.619+0000 7f0c5c619b80 10 client.15330 fetch_fsmap finished waiting for FSMap version 24 2024-03-14T09:53:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.620+0000 7f0c5c619b80 10 client.15330 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-14T09:53:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.620+0000 7f0c5c619b80 20 client.15330 populate_metadata read hostname 'smithi078' 2024-03-14T09:53:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.620+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/1952523163 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559971a6b6e0 0x559971a709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.620+0000 7f0c5c619b80 4 client.15330 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:53:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.620+0000 7f0c5c619b80 1 -- 172.21.15.78:0/1952523163 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5599719a16b0 con 0x559971a6b6e0 2024-03-14T09:53:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.620+0000 7f0c41e00700 1 --2- 172.21.15.78:0/1952523163 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559971a6b6e0 0x559971a709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.620+0000 7f0c23400700 10 client.15330 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:53:43.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.620+0000 7f0c41e00700 1 --2- 172.21.15.78:0/1952523163 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559971a6b6e0 0x559971a709c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa763c6b671c4c9f server_cookie=f44deea7ffaac14b in_seq=0 out_seq=0 2024-03-14T09:53:43.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.621+0000 7f0c23400700 1 -- 172.21.15.78:0/1952523163 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5599719a16b0 con 0x559971a6b6e0 2024-03-14T09:53:43.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.621+0000 7f0c23400700 10 client.15330 handle_command_reply: tid=0 2024-03-14T09:53:43.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.621+0000 7f0c5c619b80 1 client.15330 shutdown 2024-03-14T09:53:43.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.621+0000 7f0c5c619b80 1 -- 172.21.15.78:0/1952523163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 msgr2=0x559971a60b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.621+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/1952523163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 0x559971a60b80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0c300098e0 tx=0x7f0c30009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.621+0000 7f0c5c619b80 1 -- 172.21.15.78:0/1952523163 shutdown_connections 2024-03-14T09:53:43.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.621+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/1952523163 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559971a6b6e0 0x559971a709c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.621+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/1952523163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 0x559971a67bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.621+0000 7f0c23400700 0 client.15330 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:53:43.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.621+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/1952523163 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 0x559971a60640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.621+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/1952523163 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 0x559971a60b80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.621+0000 7f0c5c619b80 1 -- 172.21.15.78:0/1952523163 >> 172.21.15.78:0/1952523163 conn(0x5599719cc1d0 msgr2=0x559971a50bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.621+0000 7f0c5c619b80 1 -- 172.21.15.78:0/1952523163 shutdown_connections 2024-03-14T09:53:43.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.622+0000 7f0c5c619b80 1 -- 172.21.15.78:0/1952523163 wait complete. 2024-03-14T09:53:43.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.622+0000 7f0c5c619b80 20 client.15330 trim_cache size 0 max 16384 2024-03-14T09:53:43.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.641+0000 7f0c5c619b80 1 Processor -- start 2024-03-14T09:53:43.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.641+0000 7f0c5c619b80 1 -- start start 2024-03-14T09:53:43.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.641+0000 7f0c5c619b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a567a0 0x559971a65b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.641+0000 7f0c5c619b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a57100 0x559971a5fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.641+0000 7f0c5c619b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 0x559971a5e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.641+0000 7f0c5c619b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599719a19d0 con 0x559971a567a0 2024-03-14T09:53:43.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.641+0000 7f0c5c619b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599719a5880 con 0x559971a57f90 2024-03-14T09:53:43.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.641+0000 7f0c5c619b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559971a6cf40 con 0x559971a57100 2024-03-14T09:53:43.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.641+0000 7f0c41e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a567a0 0x559971a65b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.641+0000 7f0c41e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a567a0 0x559971a65b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52740/0 (socket says 172.21.15.78:52740) 2024-03-14T09:53:43.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.641+0000 7f0c41e00700 1 -- 172.21.15.78:0/869869209 learned_addr learned my addr 172.21.15.78:0/869869209 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:43.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.641+0000 7f0c41400700 1 --2- 172.21.15.78:0/869869209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a57100 0x559971a5fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.641+0000 7f0c42800700 1 --2- 172.21.15.78:0/869869209 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 0x559971a5e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.642+0000 7f0c41e00700 1 -- 172.21.15.78:0/869869209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a57100 msgr2=0x559971a5fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.642+0000 7f0c41e00700 1 --2- 172.21.15.78:0/869869209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a57100 0x559971a5fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.642+0000 7f0c41e00700 1 -- 172.21.15.78:0/869869209 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 msgr2=0x559971a5e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.642+0000 7f0c41e00700 1 --2- 172.21.15.78:0/869869209 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 0x559971a5e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.642+0000 7f0c41e00700 1 -- 172.21.15.78:0/869869209 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55997196ee50 con 0x559971a567a0 2024-03-14T09:53:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.642+0000 7f0c41e00700 1 --2- 172.21.15.78:0/869869209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a567a0 0x559971a65b70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400eb00 tx=0x7f0c3409a280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5954b921ad3b4c0f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.642+0000 7f0c2fa00700 1 -- 172.21.15.78:0/869869209 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3409da30 con 0x559971a567a0 2024-03-14T09:53:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.642+0000 7f0c2fa00700 1 -- 172.21.15.78:0/869869209 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c340a2000 con 0x559971a567a0 2024-03-14T09:53:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.642+0000 7f0c2fa00700 1 -- 172.21.15.78:0/869869209 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c340992d0 con 0x559971a567a0 2024-03-14T09:53:43.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.643+0000 7f0c5c619b80 1 -- 172.21.15.78:0/869869209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a567a0 msgr2=0x559971a65b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.643+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/869869209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a567a0 0x559971a65b70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400eb00 tx=0x7f0c3409a280 comp rx=0 tx=0).stop 2024-03-14T09:53:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.643+0000 7f0c5c619b80 1 -- 172.21.15.78:0/869869209 shutdown_connections 2024-03-14T09:53:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.643+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/869869209 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 0x559971a5e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.643+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/869869209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a567a0 0x559971a65b70 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.643+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/869869209 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a57100 0x559971a5fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.643+0000 7f0c5c619b80 1 -- 172.21.15.78:0/869869209 >> 172.21.15.78:0/869869209 conn(0x5599719cc1d0 msgr2=0x559971a51550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.643+0000 7f0c5c619b80 1 -- 172.21.15.78:0/869869209 shutdown_connections 2024-03-14T09:53:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.643+0000 7f0c5c619b80 1 -- 172.21.15.78:0/869869209 wait complete. 2024-03-14T09:53:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.643+0000 7f0c5c619b80 1 -- start start 2024-03-14T09:53:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.644+0000 7f0c5c619b80 1 Processor -- start 2024-03-14T09:53:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.644+0000 7f0c5c619b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 0x559971a62df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.644+0000 7f0c5c619b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 0x559971a63330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.644+0000 7f0c5c619b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 0x559971abdca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.644+0000 7f0c5c619b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559971a65d60 con 0x559971a57100 2024-03-14T09:53:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.644+0000 7f0c5c619b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559971a65ed0 con 0x559971a57f90 2024-03-14T09:53:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.644+0000 7f0c5c619b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559971abe1e0 con 0x559971a567a0 2024-03-14T09:53:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.644+0000 7f0c5c619b80 10 client.0 _ll_register_callbacks cb 0x7f0c3000bf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:53:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.644+0000 7f0c41400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 0x559971a63330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.644+0000 7f0c41e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 0x559971a62df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.644+0000 7f0c41400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 0x559971a63330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52756/0 (socket says 172.21.15.78:52756) 2024-03-14T09:53:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.644+0000 7f0c41400700 1 -- 172.21.15.78:0/412822428 learned_addr learned my addr 172.21.15.78:0/412822428 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.644+0000 7f0c42800700 1 --2- 172.21.15.78:0/412822428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 0x559971abdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.645+0000 7f0c41400700 1 -- 172.21.15.78:0/412822428 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 msgr2=0x559971a62df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.645+0000 7f0c41400700 1 --2- 172.21.15.78:0/412822428 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 0x559971a62df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.645+0000 7f0c41400700 1 -- 172.21.15.78:0/412822428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 msgr2=0x559971abdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:53:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.645+0000 7f0c41400700 1 --2- 172.21.15.78:0/412822428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 0x559971abdca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.645+0000 7f0c41400700 1 -- 172.21.15.78:0/412822428 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55997182a360 con 0x559971a57100 2024-03-14T09:53:43.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.645+0000 7f0c41400700 1 --2- 172.21.15.78:0/412822428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 0x559971a63330 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0c30000c00 tx=0x7f0c3000bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cf382f1fa6715e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.645+0000 7f0c23400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:53:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.645+0000 7f0c23400700 1 -- 172.21.15.78:0/412822428 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c300056e0 con 0x559971a57100 2024-03-14T09:53:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.645+0000 7f0c23400700 1 -- 172.21.15.78:0/412822428 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f0c300058e0 con 0x559971a57100 2024-03-14T09:53:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.645+0000 7f0c5c619b80 1 -- 172.21.15.78:0/412822428 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c34099b80 con 0x559971a57100 2024-03-14T09:53:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.645+0000 7f0c23400700 1 -- 172.21.15.78:0/412822428 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c30030a80 con 0x559971a57100 2024-03-14T09:53:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.645+0000 7f0c5c619b80 10 client.15336 fetch_fsmap learned FSMap version 24 2024-03-14T09:53:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.645+0000 7f0c5c619b80 1 -- 172.21.15.78:0/412822428 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55997182ec80 con 0x559971a57100 2024-03-14T09:53:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.645+0000 7f0c23400700 1 -- 172.21.15.78:0/412822428 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c34099b80 con 0x559971a57100 2024-03-14T09:53:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.646+0000 7f0c23400700 1 -- 172.21.15.78:0/412822428 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0c30005340 con 0x559971a57100 2024-03-14T09:53:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.646+0000 7f0c5c619b80 10 client.15336 fetch_fsmap finished waiting for FSMap version 24 2024-03-14T09:53:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.646+0000 7f0c5c619b80 10 client.15336 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-14T09:53:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.646+0000 7f0c5c619b80 20 client.15336 populate_metadata read hostname 'smithi078' 2024-03-14T09:53:43.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.646+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/412822428 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559971ac29d0 0x559971abf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.646+0000 7f0c5c619b80 4 client.15336 mds_command: new command op to 24572 tid=0[{"prefix": "status"}] 2024-03-14T09:53:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.646+0000 7f0c5c619b80 1 -- 172.21.15.78:0/412822428 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5599717f7ca0 con 0x559971ac29d0 2024-03-14T09:53:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.646+0000 7f0c41e00700 1 --2- 172.21.15.78:0/412822428 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559971ac29d0 0x559971abf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c41e00700 1 --2- 172.21.15.78:0/412822428 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559971ac29d0 0x559971abf430 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a409d4407299b03 server_cookie=990b67629335b280 in_seq=0 out_seq=0 2024-03-14T09:53:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c23400700 10 client.15336 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:53:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c23400700 1 -- 172.21.15.78:0/412822428 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5599717f7ca0 con 0x559971ac29d0 2024-03-14T09:53:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c23400700 10 client.15336 handle_command_reply: tid=0 2024-03-14T09:53:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c5c619b80 1 client.15336 shutdown 2024-03-14T09:53:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c5c619b80 1 -- 172.21.15.78:0/412822428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 msgr2=0x559971a63330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/412822428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 0x559971a63330 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0c30000c00 tx=0x7f0c3000bce0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c5c619b80 1 -- 172.21.15.78:0/412822428 shutdown_connections 2024-03-14T09:53:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/412822428 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559971ac29d0 0x559971abf430 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/412822428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559971a57f90 0x559971abdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/412822428 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559971a567a0 0x559971a62df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c23400700 0 client.15336 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:53:43.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c5c619b80 1 --2- 172.21.15.78:0/412822428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559971a57100 0x559971a63330 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c5c619b80 1 -- 172.21.15.78:0/412822428 >> 172.21.15.78:0/412822428 conn(0x5599719cc1d0 msgr2=0x559971a50130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c5c619b80 1 -- 172.21.15.78:0/412822428 shutdown_connections 2024-03-14T09:53:43.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c5c619b80 1 -- 172.21.15.78:0/412822428 wait complete. 2024-03-14T09:53:43.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.647+0000 7f0c5c619b80 20 client.15336 trim_cache size 0 max 16384 2024-03-14T09:53:43.653 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:53:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "83662c4e-e1e7-11ee-95c9-87774f69a715", 2024-03-14T09:53:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2024-03-14T09:53:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24572, 2024-03-14T09:53:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2024-03-14T09:53:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2024-03-14T09:53:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2024-03-14T09:53:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 97.935838046000001, 2024-03-14T09:53:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2024-03-14T09:53:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 87, 2024-03-14T09:53:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 83, 2024-03-14T09:53:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 104.912951892 2024-03-14T09:53:43.654 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:53:43.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.648+0000 7f0c48a00700 1 -- 172.21.15.78:0/1837281551 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0c280a0b10 msgr2=0x7f0c280a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.648+0000 7f0c48a00700 1 --2- 172.21.15.78:0/1837281551 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0c280a0b10 0x7f0c280a2fd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0c3000b2e0 tx=0x7f0c3000b910 comp rx=0 tx=0).stop 2024-03-14T09:53:43.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.648+0000 7f0c48a00700 1 -- 172.21.15.78:0/1837281551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c441017b0 msgr2=0x7f0c441992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:43.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.648+0000 7f0c48a00700 1 --2- 172.21.15.78:0/1837281551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c441017b0 0x7f0c441992b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400eb30 tx=0x7f0c3400ee40 comp rx=0 tx=0).stop 2024-03-14T09:53:43.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.648+0000 7f0c48a00700 1 -- 172.21.15.78:0/1837281551 shutdown_connections 2024-03-14T09:53:43.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.648+0000 7f0c48a00700 1 --2- 172.21.15.78:0/1837281551 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0c280a0b10 0x7f0c280a2fd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.648+0000 7f0c48a00700 1 --2- 172.21.15.78:0/1837281551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0c44106a10 0x7f0c441a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.648+0000 7f0c48a00700 1 --2- 172.21.15.78:0/1837281551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c441017b0 0x7f0c441992b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.648+0000 7f0c48a00700 1 --2- 172.21.15.78:0/1837281551 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0c441040e0 0x7f0c441997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:43.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.648+0000 7f0c48a00700 1 -- 172.21.15.78:0/1837281551 >> 172.21.15.78:0/1837281551 conn(0x7f0c440fb1a0 msgr2=0x7f0c44100ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:43.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.648+0000 7f0c48a00700 1 -- 172.21.15.78:0/1837281551 shutdown_connections 2024-03-14T09:53:43.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:43.648+0000 7f0c48a00700 1 -- 172.21.15.78:0/1837281551 wait complete. 2024-03-14T09:53:43.667 INFO:teuthology.run_tasks:Running task kclient... 2024-03-14T09:53:43.681 INFO:tasks.kclient:Mounting kernel clients... 2024-03-14T09:53:43.681 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-14T09:53:43.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:43.681 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-03-14T09:53:43.723 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-14T09:53:43.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:43.725 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-14T09:53:43.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:43.834 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2024-03-14T09:53:43.940 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-14T09:53:43.940 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-14T09:53:43.941 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-14T09:53:43.941 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-14T09:53:44.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:43 smithi078 ceph-mon[38143]: osdmap e87: 12 total, 12 up, 12 in 2024-03-14T09:53:44.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:43 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2329565595' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:44.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:43 smithi078 ceph-mon[38143]: pgmap v288: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-03-14T09:53:44.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:43 smithi152 ceph-mon[48279]: osdmap e87: 12 total, 12 up, 12 in 2024-03-14T09:53:44.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:43 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2329565595' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:44.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:43 smithi152 ceph-mon[48279]: pgmap v288: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-03-14T09:53:44.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:43 smithi132 ceph-mon[47959]: osdmap e87: 12 total, 12 up, 12 in 2024-03-14T09:53:44.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:43 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2329565595' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:44.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:43 smithi132 ceph-mon[47959]: pgmap v288: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-03-14T09:53:44.300 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T09:53:45.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:44 smithi078 ceph-mon[38143]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-14T09:53:45.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:44 smithi152 ceph-mon[48279]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-14T09:53:45.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:44 smithi132 ceph-mon[47959]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-14T09:53:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.420+0000 7f02b3a00700 1 -- 172.21.15.78:0/3730199788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100fb0 msgr2=0x7f02ac101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.420+0000 7f02b3a00700 1 --2- 172.21.15.78:0/3730199788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100fb0 0x7f02ac101390 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f029c009990 tx=0x7f029c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:45.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.420+0000 7f02b3a00700 1 -- 172.21.15.78:0/3730199788 shutdown_connections 2024-03-14T09:53:45.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.420+0000 7f02b3a00700 1 --2- 172.21.15.78:0/3730199788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02ac106090 0x7f02ac10dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:45.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.420+0000 7f02b3a00700 1 --2- 172.21.15.78:0/3730199788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100fb0 0x7f02ac101390 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:45.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.420+0000 7f02b3a00700 1 --2- 172.21.15.78:0/3730199788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02ac101960 0x7f02ac1059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:45.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.420+0000 7f02b3a00700 1 -- 172.21.15.78:0/3730199788 >> 172.21.15.78:0/3730199788 conn(0x7f02ac0fc800 msgr2=0x7f02ac0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:45.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.420+0000 7f02b3a00700 1 -- 172.21.15.78:0/3730199788 shutdown_connections 2024-03-14T09:53:45.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.420+0000 7f02b3a00700 1 -- 172.21.15.78:0/3730199788 wait complete. 2024-03-14T09:53:45.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.421+0000 7f02b3a00700 1 Processor -- start 2024-03-14T09:53:45.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.422+0000 7f02b3a00700 1 -- start start 2024-03-14T09:53:45.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.422+0000 7f02b3a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100fb0 0x7f02ac1975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:45.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.422+0000 7f02b3a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02ac101960 0x7f02ac197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:45.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.422+0000 7f02b3a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02ac106090 0x7f02ac19beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:45.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.422+0000 7f02b3a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac19c4c0 con 0x7f02ac100fb0 2024-03-14T09:53:45.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.422+0000 7f02b3a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac19c630 con 0x7f02ac101960 2024-03-14T09:53:45.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.422+0000 7f02b3a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac19c7f0 con 0x7f02ac106090 2024-03-14T09:53:45.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.422+0000 7f02b1000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100fb0 0x7f02ac1975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:45.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.422+0000 7f02b1000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100fb0 0x7f02ac1975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52634/0 (socket says 172.21.15.78:52634) 2024-03-14T09:53:45.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.422+0000 7f02b1000700 1 -- 172.21.15.78:0/755755544 learned_addr learned my addr 172.21.15.78:0/755755544 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:45.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.422+0000 7f02a3e00700 1 --2- 172.21.15.78:0/755755544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02ac101960 0x7f02ac197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:45.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.423+0000 7f02b1a00700 1 --2- 172.21.15.78:0/755755544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02ac106090 0x7f02ac19beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:45.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.423+0000 7f02b1000700 1 -- 172.21.15.78:0/755755544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02ac106090 msgr2=0x7f02ac19beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:45.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.423+0000 7f02b1000700 1 --2- 172.21.15.78:0/755755544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02ac106090 0x7f02ac19beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:45.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.423+0000 7f02b1000700 1 -- 172.21.15.78:0/755755544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02ac101960 msgr2=0x7f02ac197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:45.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.423+0000 7f02b1000700 1 --2- 172.21.15.78:0/755755544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02ac101960 0x7f02ac197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:45.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.423+0000 7f02b1000700 1 -- 172.21.15.78:0/755755544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c0095f0 con 0x7f02ac100fb0 2024-03-14T09:53:45.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.423+0000 7f02b1000700 1 --2- 172.21.15.78:0/755755544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100fb0 0x7f02ac1975c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f029c005770 tx=0x7f029c0057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:45.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.423+0000 7f02aa000700 1 -- 172.21.15.78:0/755755544 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c01d070 con 0x7f02ac100fb0 2024-03-14T09:53:45.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.423+0000 7f02b3a00700 1 -- 172.21.15.78:0/755755544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ac19ca70 con 0x7f02ac100fb0 2024-03-14T09:53:45.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.423+0000 7f02aa000700 1 -- 172.21.15.78:0/755755544 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f029c021410 con 0x7f02ac100fb0 2024-03-14T09:53:45.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.423+0000 7f02aa000700 1 -- 172.21.15.78:0/755755544 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c0175f0 con 0x7f02ac100fb0 2024-03-14T09:53:45.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.423+0000 7f02b3a00700 1 -- 172.21.15.78:0/755755544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ac1a4330 con 0x7f02ac100fb0 2024-03-14T09:53:45.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.424+0000 7f02b3a00700 1 -- 172.21.15.78:0/755755544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02ac109300 con 0x7f02ac100fb0 2024-03-14T09:53:45.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.426+0000 7f02aa000700 1 -- 172.21.15.78:0/755755544 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f029c0217f0 con 0x7f02ac100fb0 2024-03-14T09:53:45.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.427+0000 7f02aa000700 1 --2- 172.21.15.78:0/755755544 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f02900a0a90 0x7f02900a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:45.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.428+0000 7f02aa000700 1 -- 172.21.15.78:0/755755544 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f029c0cd3f0 con 0x7f02ac100fb0 2024-03-14T09:53:45.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.428+0000 7f02a3e00700 1 --2- 172.21.15.78:0/755755544 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f02900a0a90 0x7f02900a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:45.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.428+0000 7f02a3e00700 1 --2- 172.21.15.78:0/755755544 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f02900a0a90 0x7f02900a2f50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f02ac198b20 tx=0x7f0294009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:45.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.431+0000 7f02aa000700 1 -- 172.21.15.78:0/755755544 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f029c096a80 con 0x7f02ac100fb0 2024-03-14T09:53:45.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.676+0000 7f02b3a00700 1 -- 172.21.15.78:0/755755544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f02ac1988d0 con 0x7f02ac100fb0 2024-03-14T09:53:45.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.677+0000 7f02aa000700 1 -- 172.21.15.78:0/755755544 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+88 (secure 0 0 0) 0x7f029c09a0a0 con 0x7f02ac100fb0 2024-03-14T09:53:45.678 INFO:teuthology.orchestra.run.smithi078.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-14T09:53:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.679+0000 7f02b3a00700 1 -- 172.21.15.78:0/755755544 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f02900a0a90 msgr2=0x7f02900a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.679+0000 7f02b3a00700 1 --2- 172.21.15.78:0/755755544 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f02900a0a90 0x7f02900a2f50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f02ac198b20 tx=0x7f0294009260 comp rx=0 tx=0).stop 2024-03-14T09:53:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.679+0000 7f02b3a00700 1 -- 172.21.15.78:0/755755544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100fb0 msgr2=0x7f02ac1975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.679+0000 7f02b3a00700 1 --2- 172.21.15.78:0/755755544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100fb0 0x7f02ac1975c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f029c005770 tx=0x7f029c0057a0 comp rx=0 tx=0).stop 2024-03-14T09:53:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.679+0000 7f02b3a00700 1 -- 172.21.15.78:0/755755544 shutdown_connections 2024-03-14T09:53:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.679+0000 7f02b3a00700 1 --2- 172.21.15.78:0/755755544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f02ac106090 0x7f02ac19beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.679+0000 7f02b3a00700 1 --2- 172.21.15.78:0/755755544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02ac100fb0 0x7f02ac1975c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.679+0000 7f02b3a00700 1 --2- 172.21.15.78:0/755755544 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f02900a0a90 0x7f02900a2f50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:45.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.679+0000 7f02b3a00700 1 --2- 172.21.15.78:0/755755544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02ac101960 0x7f02ac197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:45.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.679+0000 7f02b3a00700 1 -- 172.21.15.78:0/755755544 >> 172.21.15.78:0/755755544 conn(0x7f02ac0fc800 msgr2=0x7f02ac0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:45.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.680+0000 7f02b3a00700 1 -- 172.21.15.78:0/755755544 shutdown_connections 2024-03-14T09:53:45.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:45.680+0000 7f02b3a00700 1 -- 172.21.15.78:0/755755544 wait complete. 2024-03-14T09:53:46.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:45 smithi078 ceph-mon[38143]: pgmap v289: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-14T09:53:46.038 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:45 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/755755544' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-14T09:53:46.067 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-14T09:53:46.067 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-14T09:53:46.067 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2024-03-14T09:53:46.067 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-14T09:53:46.067 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-14T09:53:46.067 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-14T09:53:46.067 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-14T09:53:46.067 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-14T09:53:46.067 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-14T09:53:46.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:46.068 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2024-03-14T09:53:46.111 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-14T09:53:46.111 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-14T09:53:46.111 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2024-03-14T09:53:46.111 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-03-14T09:53:46.111 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2024-03-14T09:53:46.111 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-03-14T09:53:46.111 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-14T09:53:46.111 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2024-03-14T09:53:46.111 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f0 2024-03-14T09:53:46.111 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-14T09:53:46.111 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2024-03-14T09:53:46.111 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f1 2024-03-14T09:53:46.111 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-14T09:53:46.112 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2024-03-14T09:53:46.112 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-14T09:53:46.112 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2024-03-14T09:53:46.112 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-14T09:53:46.112 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 41849sec preferred_lft 41849sec 2024-03-14T09:53:46.112 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2024-03-14T09:53:46.112 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-03-14T09:53:46.113 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-14T09:53:46.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:46.113 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-14T09:53:46.113 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-14T09:53:46.113 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add name ceph-brx type bridge 2024-03-14T09:53:46.113 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr flush dev ceph-brx 2024-03-14T09:53:46.113 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set ceph-brx up 2024-03-14T09:53:46.113 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-14T09:53:46.113 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-14T09:53:46.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:45 smithi152 ceph-mon[48279]: pgmap v289: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-14T09:53:46.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:45 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/755755544' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-14T09:53:46.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:45 smithi132 ceph-mon[47959]: pgmap v289: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-14T09:53:46.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:45 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/755755544' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-14T09:53:46.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:46.350 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-14T09:53:46.415 INFO:teuthology.orchestra.run.smithi078.stdout:1 2024-03-14T09:53:46.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:46.419 DEBUG:teuthology.orchestra.run.smithi078:> route 2024-03-14T09:53:46.464 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2024-03-14T09:53:46.464 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-14T09:53:46.464 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-14T09:53:46.464 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-14T09:53:46.464 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-14T09:53:46.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:46.464 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-14T09:53:46.464 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-14T09:53:46.464 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-14T09:53:46.465 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-14T09:53:46.465 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-14T09:53:46.465 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-14T09:53:46.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:46.668 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-03-14T09:53:46.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:46.711 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list-id 2024-03-14T09:53:46.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:46.794 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-14T09:53:46.794 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-14T09:53:46.795 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-14T09:53:46.795 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-14T09:53:46.795 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-14T09:53:46.971 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-14T09:53:46.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:46.972 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-14T09:53:46.972 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-14T09:53:46.972 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-14T09:53:46.972 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-14T09:53:46.972 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-14T09:53:46.972 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-14T09:53:46.972 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-14T09:53:46.972 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-14T09:53:47.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:47.211 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2024-03-14T09:53:47.212 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-03-14T09:53:47.212 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2024-03-14T09:53:47.212 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set dev brx.0 master ceph-brx 2024-03-14T09:53:47.212 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-03-14T09:53:47.378 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-14T09:53:47.378 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-14T09:53:47.421 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-14T09:53:47.422 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-14T09:53:47.422 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-14T09:53:47.503 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-14T09:53:47.504 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-14T09:53:47.504 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/ee9e45a3-d5e7-49ff-9604-f8158c25a08c 2024-03-14T09:53:47.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:47.504 DEBUG:teuthology.orchestra.run.smithi078:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/ee9e45a3-d5e7-49ff-9604-f8158c25a08c /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-14T09:53:47.644 INFO:teuthology.orchestra.run.smithi078.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-14T09:53:47.644 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-03-14T09:53:47.644 INFO:teuthology.orchestra.run.smithi078.stdout:invalid new device string format 2024-03-14T09:53:48.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:48 smithi078 ceph-mon[38143]: pgmap v290: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 639 B/s wr, 5 op/s 2024-03-14T09:53:48.309 INFO:teuthology.orchestra.run.smithi078.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-14T09:53:48.309 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-03-14T09:53:48.309 INFO:teuthology.orchestra.run.smithi078.stdout:invalid new device string format 2024-03-14T09:53:48.309 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: resolved to: "172.21.15.78:6789,172.21.15.132:6789,172.21.15.152:6789" 2024-03-14T09:53:48.309 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.78:6789,172.21.15.132:6789,172.21.15.152:6789:/volumes/qa/sv_1/ee9e45a3-d5e7-49ff-9604-f8158c25a08c 2024-03-14T09:53:48.310 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,key=0,fsid=83662c4e-e1e7-11ee-95c9-87774f69a715" will pass to kernel 2024-03-14T09:53:48.315 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-14T09:53:48.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:48.316 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-14T09:53:48.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:48.388 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-14T09:53:48.470 INFO:teuthology.orchestra.run.smithi078.stdout:client25355 2024-03-14T09:53:48.471 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:53:48.471 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-14T09:53:48.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:48 smithi152 ceph-mon[48279]: pgmap v290: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 639 B/s wr, 5 op/s 2024-03-14T09:53:48.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:48.514 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-14T09:53:48.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:48 smithi132 ceph-mon[47959]: pgmap v290: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 639 B/s wr, 5 op/s 2024-03-14T09:53:48.636 INFO:teuthology.orchestra.run.smithi078.stdout:client25355 2024-03-14T09:53:48.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:48.637 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-14T09:53:48.718 INFO:teuthology.orchestra.run.smithi078.stdout:client25355 2024-03-14T09:53:48.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T09:53:48.719 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715.client25355/status') 2024-03-14T09:53:48.821 INFO:teuthology.orchestra.run.smithi078.stdout:instance: client.25355 (3)172.21.15.78:0/3912012263 2024-03-14T09:53:48.821 INFO:teuthology.orchestra.run.smithi078.stdout:blocklisted: false 2024-03-14T09:53:48.822 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2024-03-14T09:53:48.822 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2024-03-14T09:53:48.822 INFO:teuthology.orchestra.run.smithi078.stderr:71 bytes copied, 5.4959e-05 s, 1.3 MB/s 2024-03-14T09:53:48.825 INFO:teuthology.run_tasks:Running task exec... 2024-03-14T09:53:48.835 INFO:teuthology.task.exec:Executing custom commands... 2024-03-14T09:53:48.835 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-03-14T09:53:48.835 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-14T09:53:49.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.025+0000 7f15fc600700 1 -- 172.21.15.78:0/761754927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15f4100ce0 msgr2=0x7f15f4101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:49.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.025+0000 7f15fc600700 1 --2- 172.21.15.78:0/761754927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15f4100ce0 0x7f15f4101100 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f15e80099c0 tx=0x7f15e8009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:49.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.025+0000 7f15fc600700 1 -- 172.21.15.78:0/761754927 shutdown_connections 2024-03-14T09:53:49.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.025+0000 7f15fc600700 1 --2- 172.21.15.78:0/761754927 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15f4102870 0x7f15f4109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:49.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.025+0000 7f15fc600700 1 --2- 172.21.15.78:0/761754927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15f4100ce0 0x7f15f4101100 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:49.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.025+0000 7f15fc600700 1 --2- 172.21.15.78:0/761754927 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15f4101ed0 0x7f15f4102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:49.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.025+0000 7f15fc600700 1 -- 172.21.15.78:0/761754927 >> 172.21.15.78:0/761754927 conn(0x7f15f40fc250 msgr2=0x7f15f40fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:49.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.025+0000 7f15fc600700 1 -- 172.21.15.78:0/761754927 shutdown_connections 2024-03-14T09:53:49.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.025+0000 7f15fc600700 1 -- 172.21.15.78:0/761754927 wait complete. 2024-03-14T09:53:49.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.026+0000 7f15fc600700 1 Processor -- start 2024-03-14T09:53:49.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.027+0000 7f15fc600700 1 -- start start 2024-03-14T09:53:49.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.027+0000 7f15fc600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15f4100ce0 0x7f15f41970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:49.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.027+0000 7f15fc600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15f4101ed0 0x7f15f4197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:49.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.027+0000 7f15fc600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15f4102870 0x7f15f419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:49.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.027+0000 7f15fc600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f419ebf0 con 0x7f15f4102870 2024-03-14T09:53:49.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.027+0000 7f15fc600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f419ed30 con 0x7f15f4101ed0 2024-03-14T09:53:49.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.027+0000 7f15fc600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f419eec0 con 0x7f15f4100ce0 2024-03-14T09:53:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.027+0000 7f15f9c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15f4100ce0 0x7f15f41970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.027+0000 7f15fa600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15f4102870 0x7f15f419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.027+0000 7f15f9c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15f4100ce0 0x7f15f41970c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:45034/0 (socket says 172.21.15.78:45034) 2024-03-14T09:53:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.027+0000 7f15f9c00700 1 -- 172.21.15.78:0/1949788881 learned_addr learned my addr 172.21.15.78:0/1949788881 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.027+0000 7f15f9200700 1 --2- 172.21.15.78:0/1949788881 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15f4101ed0 0x7f15f4197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.028+0000 7f15f9c00700 1 -- 172.21.15.78:0/1949788881 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15f4101ed0 msgr2=0x7f15f4197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.028+0000 7f15f9c00700 1 --2- 172.21.15.78:0/1949788881 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15f4101ed0 0x7f15f4197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.028+0000 7f15f9c00700 1 -- 172.21.15.78:0/1949788881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15f4102870 msgr2=0x7f15f419e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:49.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.028+0000 7f15f9c00700 1 --2- 172.21.15.78:0/1949788881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15f4102870 0x7f15f419e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:49.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.028+0000 7f15f9c00700 1 -- 172.21.15.78:0/1949788881 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e80095f0 con 0x7f15f4100ce0 2024-03-14T09:53:49.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.028+0000 7f15fa600700 1 --2- 172.21.15.78:0/1949788881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15f4102870 0x7f15f419e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:49.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.028+0000 7f15f9c00700 1 --2- 172.21.15.78:0/1949788881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15f4100ce0 0x7f15f41970c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f15e8006010 tx=0x7f15e800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:49.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.028+0000 7f15f2000700 1 -- 172.21.15.78:0/1949788881 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e801d070 con 0x7f15f4100ce0 2024-03-14T09:53:49.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.028+0000 7f15f2000700 1 -- 172.21.15.78:0/1949788881 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f15e8005020 con 0x7f15f4100ce0 2024-03-14T09:53:49.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.028+0000 7f15fc600700 1 -- 172.21.15.78:0/1949788881 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f419f140 con 0x7f15f4100ce0 2024-03-14T09:53:49.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.028+0000 7f15f2000700 1 -- 172.21.15.78:0/1949788881 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e8004b10 con 0x7f15f4100ce0 2024-03-14T09:53:49.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.028+0000 7f15fc600700 1 -- 172.21.15.78:0/1949788881 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f419f500 con 0x7f15f4100ce0 2024-03-14T09:53:49.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.029+0000 7f15fc600700 1 -- 172.21.15.78:0/1949788881 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f41081b0 con 0x7f15f4100ce0 2024-03-14T09:53:49.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.031+0000 7f15f2000700 1 -- 172.21.15.78:0/1949788881 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 18) v1 ==== 135388+0+0 (secure 0 0 0) 0x7f15e8005190 con 0x7f15f4100ce0 2024-03-14T09:53:49.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.032+0000 7f15f2000700 1 --2- 172.21.15.78:0/1949788881 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f15dc0a0a90 0x7f15dc0a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:49.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.033+0000 7f15f2000700 1 -- 172.21.15.78:0/1949788881 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f15e80ccfe0 con 0x7f15f4100ce0 2024-03-14T09:53:49.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.033+0000 7f15f9200700 1 --2- 172.21.15.78:0/1949788881 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f15dc0a0a90 0x7f15dc0a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:49.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.033+0000 7f15f9200700 1 --2- 172.21.15.78:0/1949788881 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f15dc0a0a90 0x7f15dc0a2f50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f15e0009c60 tx=0x7f15e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:49.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.035+0000 7f15f2000700 1 -- 172.21.15.78:0/1949788881 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f15e8096670 con 0x7f15f4100ce0 2024-03-14T09:53:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:49.297+0000 7f15fc600700 1 -- 172.21.15.78:0/1949788881 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7f15f4067c00 con 0x7f15f4100ce0 2024-03-14T09:53:49.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:49 smithi152 ceph-mon[48279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-14T09:53:49.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:49 smithi152 ceph-mon[48279]: Cluster is now healthy 2024-03-14T09:53:49.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:49 smithi132 ceph-mon[47959]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-14T09:53:49.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:49 smithi132 ceph-mon[47959]: Cluster is now healthy 2024-03-14T09:53:49.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:49 smithi078 ceph-mon[38143]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-14T09:53:49.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:49 smithi078 ceph-mon[38143]: Cluster is now healthy 2024-03-14T09:53:50.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.206+0000 7f15f2000700 1 -- 172.21.15.78:0/1949788881 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v25) v1 ==== 118+0+0 (secure 0 0 0) 0x7f15e8099c90 con 0x7f15f4100ce0 2024-03-14T09:53:50.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.209+0000 7f15fc600700 1 -- 172.21.15.78:0/1949788881 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f15dc0a0a90 msgr2=0x7f15dc0a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:50.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.209+0000 7f15fc600700 1 --2- 172.21.15.78:0/1949788881 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f15dc0a0a90 0x7f15dc0a2f50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f15e0009c60 tx=0x7f15e0009260 comp rx=0 tx=0).stop 2024-03-14T09:53:50.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.209+0000 7f15fc600700 1 -- 172.21.15.78:0/1949788881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15f4100ce0 msgr2=0x7f15f41970c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:50.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.209+0000 7f15fc600700 1 --2- 172.21.15.78:0/1949788881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15f4100ce0 0x7f15f41970c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f15e8006010 tx=0x7f15e800f690 comp rx=0 tx=0).stop 2024-03-14T09:53:50.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.210+0000 7f15fc600700 1 -- 172.21.15.78:0/1949788881 shutdown_connections 2024-03-14T09:53:50.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.210+0000 7f15fc600700 1 --2- 172.21.15.78:0/1949788881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15f4102870 0x7f15f419e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.210+0000 7f15fc600700 1 --2- 172.21.15.78:0/1949788881 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f15f4100ce0 0x7f15f41970c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.210+0000 7f15fc600700 1 --2- 172.21.15.78:0/1949788881 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f15dc0a0a90 0x7f15dc0a2f50 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.210+0000 7f15fc600700 1 --2- 172.21.15.78:0/1949788881 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15f4101ed0 0x7f15f4197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.210+0000 7f15fc600700 1 -- 172.21.15.78:0/1949788881 >> 172.21.15.78:0/1949788881 conn(0x7f15f40fc250 msgr2=0x7f15f41052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:50.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.210+0000 7f15fc600700 1 -- 172.21.15.78:0/1949788881 shutdown_connections 2024-03-14T09:53:50.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.210+0000 7f15fc600700 1 -- 172.21.15.78:0/1949788881 wait complete. 2024-03-14T09:53:50.230 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-14T09:53:50.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.422+0000 7ff203c00700 1 -- 172.21.15.78:0/3772042467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1fc100ce0 msgr2=0x7ff1fc101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:50.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.422+0000 7ff203c00700 1 --2- 172.21.15.78:0/3772042467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1fc100ce0 0x7ff1fc101100 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00099c0 tx=0x7ff1f0009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.422+0000 7ff203c00700 1 -- 172.21.15.78:0/3772042467 shutdown_connections 2024-03-14T09:53:50.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.422+0000 7ff203c00700 1 --2- 172.21.15.78:0/3772042467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1fc100ce0 0x7ff1fc101100 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.422+0000 7ff203c00700 1 --2- 172.21.15.78:0/3772042467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1fc102870 0x7ff1fc109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.422+0000 7ff203c00700 1 --2- 172.21.15.78:0/3772042467 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1fc101ed0 0x7ff1fc102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.422+0000 7ff203c00700 1 -- 172.21.15.78:0/3772042467 >> 172.21.15.78:0/3772042467 conn(0x7ff1fc0fc250 msgr2=0x7ff1fc0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:50.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.422+0000 7ff203c00700 1 -- 172.21.15.78:0/3772042467 shutdown_connections 2024-03-14T09:53:50.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.422+0000 7ff203c00700 1 -- 172.21.15.78:0/3772042467 wait complete. 2024-03-14T09:53:50.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.423+0000 7ff203c00700 1 Processor -- start 2024-03-14T09:53:50.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.423+0000 7ff203c00700 1 -- start start 2024-03-14T09:53:50.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.423+0000 7ff203c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1fc100ce0 0x7ff1fc074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:50.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.423+0000 7ff203c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1fc101ed0 0x7ff1fc0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:50.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.423+0000 7ff203c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1fc102870 0x7ff1fc0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:50.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.423+0000 7ff203c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc073e80 con 0x7ff1fc102870 2024-03-14T09:53:50.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.424+0000 7ff203c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc073ff0 con 0x7ff1fc101ed0 2024-03-14T09:53:50.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.424+0000 7ff203c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc1a1160 con 0x7ff1fc100ce0 2024-03-14T09:53:50.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.424+0000 7ff201c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1fc102870 0x7ff1fc0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:50.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.424+0000 7ff1fbe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1fc101ed0 0x7ff1fc0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:50.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.424+0000 7ff1fbe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1fc101ed0 0x7ff1fc0731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:36842/0 (socket says 172.21.15.78:36842) 2024-03-14T09:53:50.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.424+0000 7ff1fbe00700 1 -- 172.21.15.78:0/44447951 learned_addr learned my addr 172.21.15.78:0/44447951 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:50.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.424+0000 7ff201200700 1 --2- 172.21.15.78:0/44447951 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1fc100ce0 0x7ff1fc074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:50.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.424+0000 7ff1fbe00700 1 -- 172.21.15.78:0/44447951 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1fc100ce0 msgr2=0x7ff1fc074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:50.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.424+0000 7ff1fbe00700 1 --2- 172.21.15.78:0/44447951 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1fc100ce0 0x7ff1fc074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.424+0000 7ff1fbe00700 1 -- 172.21.15.78:0/44447951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1fc102870 msgr2=0x7ff1fc0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:50.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.424+0000 7ff1fbe00700 1 --2- 172.21.15.78:0/44447951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1fc102870 0x7ff1fc0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.424+0000 7ff1fbe00700 1 -- 172.21.15.78:0/44447951 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f4009c40 con 0x7ff1fc101ed0 2024-03-14T09:53:50.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.424+0000 7ff201c00700 1 --2- 172.21.15.78:0/44447951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1fc102870 0x7ff1fc0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:50.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.424+0000 7ff1fbe00700 1 --2- 172.21.15.78:0/44447951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1fc101ed0 0x7ff1fc0731a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800e9a0 tx=0x7ff1e800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:50.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.425+0000 7ff1f9600700 1 -- 172.21.15.78:0/44447951 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8009790 con 0x7ff1fc101ed0 2024-03-14T09:53:50.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.425+0000 7ff1f9600700 1 -- 172.21.15.78:0/44447951 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1e800ce20 con 0x7ff1fc101ed0 2024-03-14T09:53:50.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.425+0000 7ff203c00700 1 -- 172.21.15.78:0/44447951 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f00095f0 con 0x7ff1fc101ed0 2024-03-14T09:53:50.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.425+0000 7ff1f9600700 1 -- 172.21.15.78:0/44447951 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8018420 con 0x7ff1fc101ed0 2024-03-14T09:53:50.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.425+0000 7ff203c00700 1 -- 172.21.15.78:0/44447951 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1fc1a1770 con 0x7ff1fc101ed0 2024-03-14T09:53:50.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.426+0000 7ff203c00700 1 -- 172.21.15.78:0/44447951 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1fc108220 con 0x7ff1fc101ed0 2024-03-14T09:53:50.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.428+0000 7ff1f9600700 1 -- 172.21.15.78:0/44447951 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 18) v1 ==== 135388+0+0 (secure 0 0 0) 0x7ff1e80106a0 con 0x7ff1fc101ed0 2024-03-14T09:53:50.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.429+0000 7ff1f9600700 1 --2- 172.21.15.78:0/44447951 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff1e40a0a90 0x7ff1e40a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:50.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.429+0000 7ff1f9600700 1 -- 172.21.15.78:0/44447951 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7ff1e8014070 con 0x7ff1fc101ed0 2024-03-14T09:53:50.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.429+0000 7ff201200700 1 --2- 172.21.15.78:0/44447951 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff1e40a0a90 0x7ff1e40a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:50.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.430+0000 7ff201200700 1 --2- 172.21.15.78:0/44447951 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff1e40a0a90 0x7ff1e40a2f50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0003730 tx=0x7ff1f0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:50.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.433+0000 7ff1f9600700 1 -- 172.21.15.78:0/44447951 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff1e8052960 con 0x7ff1fc101ed0 2024-03-14T09:53:50.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:50 smithi152 ceph-mon[48279]: pgmap v291: 129 pgs: 129 active+clean; 642 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-14T09:53:50.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:50 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1949788881' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-14T09:53:50.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:50 smithi152 ceph-mon[48279]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-14T09:53:50.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:50 smithi132 ceph-mon[47959]: pgmap v291: 129 pgs: 129 active+clean; 642 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-14T09:53:50.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:50 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1949788881' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-14T09:53:50.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:50 smithi132 ceph-mon[47959]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-14T09:53:50.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:50 smithi078 ceph-mon[38143]: pgmap v291: 129 pgs: 129 active+clean; 642 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-14T09:53:50.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:50 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1949788881' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-14T09:53:50.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:50 smithi078 ceph-mon[38143]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-14T09:53:50.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.678+0000 7ff203c00700 1 -- 172.21.15.78:0/44447951 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1fc02cdc0 con 0x7ff1e40a0a90 2024-03-14T09:53:50.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.698+0000 7ff1f9600700 1 -- 172.21.15.78:0/44447951 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7ff1fc02cdc0 con 0x7ff1e40a0a90 2024-03-14T09:53:50.698 INFO:teuthology.orchestra.run.smithi078.stdout:{} 2024-03-14T09:53:50.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.700+0000 7ff203c00700 1 -- 172.21.15.78:0/44447951 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff1e40a0a90 msgr2=0x7ff1e40a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:50.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.700+0000 7ff203c00700 1 --2- 172.21.15.78:0/44447951 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff1e40a0a90 0x7ff1e40a2f50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0003730 tx=0x7ff1f0000bc0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.700+0000 7ff203c00700 1 -- 172.21.15.78:0/44447951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1fc101ed0 msgr2=0x7ff1fc0731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:50.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.700+0000 7ff203c00700 1 --2- 172.21.15.78:0/44447951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1fc101ed0 0x7ff1fc0731a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800e9a0 tx=0x7ff1e800ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.700+0000 7ff203c00700 1 -- 172.21.15.78:0/44447951 shutdown_connections 2024-03-14T09:53:50.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.700+0000 7ff203c00700 1 --2- 172.21.15.78:0/44447951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1fc102870 0x7ff1fc0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.700+0000 7ff203c00700 1 --2- 172.21.15.78:0/44447951 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1fc100ce0 0x7ff1fc074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.700+0000 7ff203c00700 1 --2- 172.21.15.78:0/44447951 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff1e40a0a90 0x7ff1e40a2f50 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.700+0000 7ff203c00700 1 --2- 172.21.15.78:0/44447951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1fc101ed0 0x7ff1fc0731a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.700+0000 7ff203c00700 1 -- 172.21.15.78:0/44447951 >> 172.21.15.78:0/44447951 conn(0x7ff1fc0fc250 msgr2=0x7ff1fc105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:50.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.700+0000 7ff203c00700 1 -- 172.21.15.78:0/44447951 shutdown_connections 2024-03-14T09:53:50.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.700+0000 7ff203c00700 1 -- 172.21.15.78:0/44447951 wait complete. 2024-03-14T09:53:50.719 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-14T09:53:50.731 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-14T09:53:50.774 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:53:50.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.967+0000 7fe0b3000700 1 -- 172.21.15.78:0/4066560682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0ac1027e0 msgr2=0x7fe0ac109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.967+0000 7fe0b3000700 1 --2- 172.21.15.78:0/4066560682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0ac1027e0 0x7fe0ac109090 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400b0d0 tx=0x7fe0a400b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.967+0000 7fe0b3000700 1 -- 172.21.15.78:0/4066560682 shutdown_connections 2024-03-14T09:53:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.967+0000 7fe0b3000700 1 --2- 172.21.15.78:0/4066560682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0ac100c50 0x7fe0ac101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.967+0000 7fe0b3000700 1 --2- 172.21.15.78:0/4066560682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0ac1027e0 0x7fe0ac109090 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.967+0000 7fe0b3000700 1 --2- 172.21.15.78:0/4066560682 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0ac101e40 0x7fe0ac1022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.967+0000 7fe0b3000700 1 -- 172.21.15.78:0/4066560682 >> 172.21.15.78:0/4066560682 conn(0x7fe0ac0fc1c0 msgr2=0x7fe0ac0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.967+0000 7fe0b3000700 1 -- 172.21.15.78:0/4066560682 shutdown_connections 2024-03-14T09:53:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.967+0000 7fe0b3000700 1 -- 172.21.15.78:0/4066560682 wait complete. 2024-03-14T09:53:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.968+0000 7fe0b3000700 1 Processor -- start 2024-03-14T09:53:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.968+0000 7fe0b3000700 1 -- start start 2024-03-14T09:53:50.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.968+0000 7fe0b3000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0ac100c50 0x7fe0ac074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.968+0000 7fe0b3000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0ac101e40 0x7fe0ac0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.969+0000 7fe0b3000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0ac1027e0 0x7fe0ac0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.969+0000 7fe0b3000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac073e80 con 0x7fe0ac101e40 2024-03-14T09:53:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.969+0000 7fe0b3000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac073ff0 con 0x7fe0ac100c50 2024-03-14T09:53:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.969+0000 7fe0b3000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac1a1140 con 0x7fe0ac1027e0 2024-03-14T09:53:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.969+0000 7fe0ab400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0ac101e40 0x7fe0ac0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.969+0000 7fe0ab400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0ac101e40 0x7fe0ac0731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52704/0 (socket says 172.21.15.78:52704) 2024-03-14T09:53:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.969+0000 7fe0ab400700 1 -- 172.21.15.78:0/603443324 learned_addr learned my addr 172.21.15.78:0/603443324 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.969+0000 7fe0abe00700 1 --2- 172.21.15.78:0/603443324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0ac100c50 0x7fe0ac074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.969+0000 7fe0b1000700 1 --2- 172.21.15.78:0/603443324 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0ac1027e0 0x7fe0ac0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.969+0000 7fe0ab400700 1 -- 172.21.15.78:0/603443324 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0ac1027e0 msgr2=0x7fe0ac0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.969+0000 7fe0ab400700 1 --2- 172.21.15.78:0/603443324 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0ac1027e0 0x7fe0ac0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.969+0000 7fe0ab400700 1 -- 172.21.15.78:0/603443324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0ac100c50 msgr2=0x7fe0ac074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.969+0000 7fe0ab400700 1 --2- 172.21.15.78:0/603443324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0ac100c50 0x7fe0ac074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.969+0000 7fe0ab400700 1 -- 172.21.15.78:0/603443324 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a4009d00 con 0x7fe0ac101e40 2024-03-14T09:53:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.969+0000 7fe0ab400700 1 --2- 172.21.15.78:0/603443324 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0ac101e40 0x7fe0ac0731a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe09800e9a0 tx=0x7fe09800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.970+0000 7fe0a8c00700 1 -- 172.21.15.78:0/603443324 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe098009790 con 0x7fe0ac101e40 2024-03-14T09:53:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.970+0000 7fe0a8c00700 1 -- 172.21.15.78:0/603443324 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe09800ce70 con 0x7fe0ac101e40 2024-03-14T09:53:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.970+0000 7fe0a8c00700 1 -- 172.21.15.78:0/603443324 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe098018420 con 0x7fe0ac101e40 2024-03-14T09:53:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.970+0000 7fe0b3000700 1 -- 172.21.15.78:0/603443324 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ac1a13d0 con 0x7fe0ac101e40 2024-03-14T09:53:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.970+0000 7fe0b3000700 1 -- 172.21.15.78:0/603443324 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ac1a18e0 con 0x7fe0ac101e40 2024-03-14T09:53:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.971+0000 7fe0b3000700 1 -- 172.21.15.78:0/603443324 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ac108080 con 0x7fe0ac101e40 2024-03-14T09:53:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.973+0000 7fe0a8c00700 1 -- 172.21.15.78:0/603443324 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135388+0+0 (secure 0 0 0) 0x7fe0980106f0 con 0x7fe0ac101e40 2024-03-14T09:53:50.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.974+0000 7fe0a8c00700 1 --2- 172.21.15.78:0/603443324 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe0900a0b10 0x7fe0900a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:50.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.974+0000 7fe0a8c00700 1 -- 172.21.15.78:0/603443324 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7fe098014070 con 0x7fe0ac101e40 2024-03-14T09:53:50.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.974+0000 7fe0abe00700 1 --2- 172.21.15.78:0/603443324 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe0900a0b10 0x7fe0900a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:50.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.975+0000 7fe0abe00700 1 --2- 172.21.15.78:0/603443324 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe0900a0b10 0x7fe0900a2fd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac101ca0 tx=0x7fe09c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:50.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:50.978+0000 7fe0a8c00700 1 -- 172.21.15.78:0/603443324 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe098052960 con 0x7fe0ac101e40 2024-03-14T09:53:51.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:53:50 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:53:50.684+0000 7fca6a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:53:51.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:53:50 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:53:50.684+0000 7fca6a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:53:51.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:53:50 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:53:50.684+0000 7fca6a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:53:51.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:53:50 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:53:50.684+0000 7fca6a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:53:51.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:53:50 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:53:50.684+0000 7fca6a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:53:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.206+0000 7fe0b3000700 1 -- 172.21.15.78:0/603443324 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0ac0679e0 con 0x7fe0ac101e40 2024-03-14T09:53:51.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.208+0000 7fe0a8c00700 1 -- 172.21.15.78:0/603443324 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24594 (secure 0 0 0) 0x7fe098095af0 con 0x7fe0ac101e40 2024-03-14T09:53:51.209 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:51.209 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":87,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:53:42.786378+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:53:51.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.210+0000 7fe0b3000700 1 -- 172.21.15.78:0/603443324 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe0900a0b10 msgr2=0x7fe0900a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:51.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.210+0000 7fe0b3000700 1 --2- 172.21.15.78:0/603443324 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe0900a0b10 0x7fe0900a2fd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac101ca0 tx=0x7fe09c009260 comp rx=0 tx=0).stop 2024-03-14T09:53:51.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.210+0000 7fe0b3000700 1 -- 172.21.15.78:0/603443324 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0ac101e40 msgr2=0x7fe0ac0731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:51.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.210+0000 7fe0b3000700 1 --2- 172.21.15.78:0/603443324 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0ac101e40 0x7fe0ac0731a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe09800e9a0 tx=0x7fe09800ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.210+0000 7fe0b3000700 1 -- 172.21.15.78:0/603443324 shutdown_connections 2024-03-14T09:53:51.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.210+0000 7fe0b3000700 1 --2- 172.21.15.78:0/603443324 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe0900a0b10 0x7fe0900a2fd0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.210+0000 7fe0b3000700 1 --2- 172.21.15.78:0/603443324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0ac100c50 0x7fe0ac074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.210+0000 7fe0b3000700 1 --2- 172.21.15.78:0/603443324 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe0ac1027e0 0x7fe0ac0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.210+0000 7fe0b3000700 1 --2- 172.21.15.78:0/603443324 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0ac101e40 0x7fe0ac0731a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.210+0000 7fe0b3000700 1 -- 172.21.15.78:0/603443324 >> 172.21.15.78:0/603443324 conn(0x7fe0ac0fc1c0 msgr2=0x7fe0ac1051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:51.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.211+0000 7fe0b3000700 1 -- 172.21.15.78:0/603443324 shutdown_connections 2024-03-14T09:53:51.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.211+0000 7fe0b3000700 1 -- 172.21.15.78:0/603443324 wait complete. 2024-03-14T09:53:51.234 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T09:48:37.618450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-14T09:52:03.938666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-14T09:52:05.149985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-14T09:52:08.227698+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-14T09:53:51.235 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T09:53:51.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.431+0000 7f32bac00700 1 -- 172.21.15.78:0/2297216442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32b41028d0 msgr2=0x7f32b4109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:51.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.431+0000 7f32bac00700 1 --2- 172.21.15.78:0/2297216442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32b41028d0 0x7f32b4109180 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00b0d0 tx=0x7f32ac00b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.431+0000 7f32bac00700 1 -- 172.21.15.78:0/2297216442 shutdown_connections 2024-03-14T09:53:51.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.431+0000 7f32bac00700 1 --2- 172.21.15.78:0/2297216442 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32b4100d40 0x7f32b4101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.431+0000 7f32bac00700 1 --2- 172.21.15.78:0/2297216442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32b41028d0 0x7f32b4109180 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.431+0000 7f32bac00700 1 --2- 172.21.15.78:0/2297216442 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4101f30 0x7f32b4102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.431+0000 7f32bac00700 1 -- 172.21.15.78:0/2297216442 >> 172.21.15.78:0/2297216442 conn(0x7f32b40fc2b0 msgr2=0x7f32b40fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:51.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.431+0000 7f32bac00700 1 -- 172.21.15.78:0/2297216442 shutdown_connections 2024-03-14T09:53:51.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.431+0000 7f32bac00700 1 -- 172.21.15.78:0/2297216442 wait complete. 2024-03-14T09:53:51.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.432+0000 7f32bac00700 1 Processor -- start 2024-03-14T09:53:51.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.433+0000 7f32bac00700 1 -- start start 2024-03-14T09:53:51.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.433+0000 7f32bac00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4100d40 0x7f32b4197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:51.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.433+0000 7f32bac00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32b4101f30 0x7f32b4197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:51.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.433+0000 7f32bac00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32b41028d0 0x7f32b419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:51.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.433+0000 7f32bac00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b419ec70 con 0x7f32b41028d0 2024-03-14T09:53:51.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.433+0000 7f32bac00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b419edb0 con 0x7f32b4101f30 2024-03-14T09:53:51.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.433+0000 7f32bac00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b419ef40 con 0x7f32b4100d40 2024-03-14T09:53:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.433+0000 7f32b8c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32b41028d0 0x7f32b419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.433+0000 7f32b3e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4100d40 0x7f32b4197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.433+0000 7f32b3400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32b4101f30 0x7f32b4197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.433+0000 7f32b3e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4100d40 0x7f32b4197140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:45094/0 (socket says 172.21.15.78:45094) 2024-03-14T09:53:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.433+0000 7f32b3e00700 1 -- 172.21.15.78:0/1116478050 learned_addr learned my addr 172.21.15.78:0/1116478050 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.434+0000 7f32b8c00700 1 -- 172.21.15.78:0/1116478050 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4100d40 msgr2=0x7f32b4197140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.434+0000 7f32b8c00700 1 --2- 172.21.15.78:0/1116478050 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4100d40 0x7f32b4197140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.434+0000 7f32b8c00700 1 -- 172.21.15.78:0/1116478050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32b4101f30 msgr2=0x7f32b4197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.434+0000 7f32b8c00700 1 --2- 172.21.15.78:0/1116478050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32b4101f30 0x7f32b4197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.434+0000 7f32b8c00700 1 -- 172.21.15.78:0/1116478050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ac009d00 con 0x7f32b41028d0 2024-03-14T09:53:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.434+0000 7f32b3e00700 1 --2- 172.21.15.78:0/1116478050 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4100d40 0x7f32b4197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.434+0000 7f32b8c00700 1 --2- 172.21.15.78:0/1116478050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32b41028d0 0x7f32b419e6a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f32ac007b60 tx=0x7f32ac008f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.434+0000 7f32b0c00700 1 -- 172.21.15.78:0/1116478050 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac00e040 con 0x7f32b41028d0 2024-03-14T09:53:51.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.434+0000 7f32b0c00700 1 -- 172.21.15.78:0/1116478050 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f32ac01c6c0 con 0x7f32b41028d0 2024-03-14T09:53:51.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.434+0000 7f32b0c00700 1 -- 172.21.15.78:0/1116478050 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac0052f0 con 0x7f32b41028d0 2024-03-14T09:53:51.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.434+0000 7f32bac00700 1 -- 172.21.15.78:0/1116478050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b4072fe0 con 0x7f32b41028d0 2024-03-14T09:53:51.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.435+0000 7f32bac00700 1 -- 172.21.15.78:0/1116478050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b4073440 con 0x7f32b41028d0 2024-03-14T09:53:51.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.436+0000 7f32bac00700 1 -- 172.21.15.78:0/1116478050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b41080f0 con 0x7f32b41028d0 2024-03-14T09:53:51.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.437+0000 7f32b0c00700 1 -- 172.21.15.78:0/1116478050 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f32ac024030 con 0x7f32b41028d0 2024-03-14T09:53:51.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.439+0000 7f32b0c00700 1 --2- 172.21.15.78:0/1116478050 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f32980a0b20 0x7f32980a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:51.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.439+0000 7f32b0c00700 1 -- 172.21.15.78:0/1116478050 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f32ac022070 con 0x7f32b41028d0 2024-03-14T09:53:51.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.439+0000 7f32b3e00700 1 --2- 172.21.15.78:0/1116478050 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f32980a0b20 0x7f32980a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:51.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.439+0000 7f32b3e00700 1 --2- 172.21.15.78:0/1116478050 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f32980a0b20 0x7f32980a2fe0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f32a4005950 tx=0x7f32a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:51.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.443+0000 7f32b0c00700 1 -- 172.21.15.78:0/1116478050 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f32ac09a950 con 0x7f32b41028d0 2024-03-14T09:53:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:51 smithi152 ceph-mon[48279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-14T09:53:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:51 smithi152 ceph-mon[48279]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-14T09:53:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:51 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/603443324' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:51.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:51 smithi132 ceph-mon[47959]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-14T09:53:51.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:51 smithi132 ceph-mon[47959]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-14T09:53:51.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:51 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/603443324' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:51.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:51 smithi078 ceph-mon[38143]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-14T09:53:51.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:51 smithi078 ceph-mon[38143]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-14T09:53:51.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:51 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/603443324' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:51.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.648+0000 7f32bac00700 1 -- 172.21.15.78:0/1116478050 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f32b410c980 con 0x7f32b41028d0 2024-03-14T09:53:51.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.649+0000 7f32b0c00700 1 -- 172.21.15.78:0/1116478050 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f32ac09df70 con 0x7f32b41028d0 2024-03-14T09:53:51.649 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-14T09:53:51.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.651+0000 7f32bac00700 1 -- 172.21.15.78:0/1116478050 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f32980a0b20 msgr2=0x7f32980a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:51.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.651+0000 7f32bac00700 1 --2- 172.21.15.78:0/1116478050 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f32980a0b20 0x7f32980a2fe0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f32a4005950 tx=0x7f32a4009260 comp rx=0 tx=0).stop 2024-03-14T09:53:51.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.651+0000 7f32bac00700 1 -- 172.21.15.78:0/1116478050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32b41028d0 msgr2=0x7f32b419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:51.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.651+0000 7f32bac00700 1 --2- 172.21.15.78:0/1116478050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32b41028d0 0x7f32b419e6a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f32ac007b60 tx=0x7f32ac008f30 comp rx=0 tx=0).stop 2024-03-14T09:53:51.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.651+0000 7f32bac00700 1 -- 172.21.15.78:0/1116478050 shutdown_connections 2024-03-14T09:53:51.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.651+0000 7f32bac00700 1 --2- 172.21.15.78:0/1116478050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32b41028d0 0x7f32b419e6a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.651+0000 7f32bac00700 1 --2- 172.21.15.78:0/1116478050 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f32b4100d40 0x7f32b4197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.651+0000 7f32bac00700 1 --2- 172.21.15.78:0/1116478050 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f32980a0b20 0x7f32980a2fe0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.651+0000 7f32bac00700 1 --2- 172.21.15.78:0/1116478050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32b4101f30 0x7f32b4197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.652+0000 7f32bac00700 1 -- 172.21.15.78:0/1116478050 >> 172.21.15.78:0/1116478050 conn(0x7f32b40fc2b0 msgr2=0x7f32b4105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:51.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.652+0000 7f32bac00700 1 -- 172.21.15.78:0/1116478050 shutdown_connections 2024-03-14T09:53:51.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.652+0000 7f32bac00700 1 -- 172.21.15.78:0/1116478050 wait complete. 2024-03-14T09:53:51.671 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-14T09:53:51.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.868+0000 7fa6b3c00700 1 -- 172.21.15.78:0/1311781825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6ac100d50 msgr2=0x7fa6ac101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:51.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.868+0000 7fa6b3c00700 1 --2- 172.21.15.78:0/1311781825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6ac100d50 0x7fa6ac101170 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa6a00099c0 tx=0x7fa6a0009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.868+0000 7fa6b3c00700 1 -- 172.21.15.78:0/1311781825 shutdown_connections 2024-03-14T09:53:51.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.868+0000 7fa6b3c00700 1 --2- 172.21.15.78:0/1311781825 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6ac1028e0 0x7fa6ac109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.868+0000 7fa6b3c00700 1 --2- 172.21.15.78:0/1311781825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6ac100d50 0x7fa6ac101170 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.868+0000 7fa6b3c00700 1 --2- 172.21.15.78:0/1311781825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6ac101f40 0x7fa6ac1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.868+0000 7fa6b3c00700 1 -- 172.21.15.78:0/1311781825 >> 172.21.15.78:0/1311781825 conn(0x7fa6ac0fc2c0 msgr2=0x7fa6ac0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:51.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.868+0000 7fa6b3c00700 1 -- 172.21.15.78:0/1311781825 shutdown_connections 2024-03-14T09:53:51.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.868+0000 7fa6b3c00700 1 -- 172.21.15.78:0/1311781825 wait complete. 2024-03-14T09:53:51.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.869+0000 7fa6b3c00700 1 Processor -- start 2024-03-14T09:53:51.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.870+0000 7fa6b3c00700 1 -- start start 2024-03-14T09:53:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.870+0000 7fa6b3c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6ac100d50 0x7fa6ac199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.870+0000 7fa6b3c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6ac101f40 0x7fa6ac1998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.870+0000 7fa6b3c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6ac1028e0 0x7fa6ac1a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.870+0000 7fa6b3c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac1a0ea0 con 0x7fa6ac101f40 2024-03-14T09:53:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.870+0000 7fa6b3c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac1a0fe0 con 0x7fa6ac1028e0 2024-03-14T09:53:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.870+0000 7fa6b3c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac1a1170 con 0x7fa6ac100d50 2024-03-14T09:53:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.870+0000 7fa6abe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6ac101f40 0x7fa6ac1998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.871+0000 7fa6b1200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6ac100d50 0x7fa6ac199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.871+0000 7fa6abe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6ac101f40 0x7fa6ac1998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52742/0 (socket says 172.21.15.78:52742) 2024-03-14T09:53:51.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.871+0000 7fa6abe00700 1 -- 172.21.15.78:0/2040623118 learned_addr learned my addr 172.21.15.78:0/2040623118 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.871+0000 7fa6b1c00700 1 --2- 172.21.15.78:0/2040623118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6ac1028e0 0x7fa6ac1a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.871+0000 7fa6b1200700 1 -- 172.21.15.78:0/2040623118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6ac1028e0 msgr2=0x7fa6ac1a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.871+0000 7fa6b1200700 1 --2- 172.21.15.78:0/2040623118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6ac1028e0 0x7fa6ac1a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.871+0000 7fa6b1200700 1 -- 172.21.15.78:0/2040623118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6ac101f40 msgr2=0x7fa6ac1998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.871+0000 7fa6b1200700 1 --2- 172.21.15.78:0/2040623118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6ac101f40 0x7fa6ac1998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.871+0000 7fa6b1200700 1 -- 172.21.15.78:0/2040623118 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a00095f0 con 0x7fa6ac100d50 2024-03-14T09:53:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.871+0000 7fa6abe00700 1 --2- 172.21.15.78:0/2040623118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6ac101f40 0x7fa6ac1998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.871+0000 7fa6b1200700 1 --2- 172.21.15.78:0/2040623118 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6ac100d50 0x7fa6ac199370 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa6a00098e0 tx=0x7fa6a000fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.871+0000 7fa6a9600700 1 -- 172.21.15.78:0/2040623118 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a001d070 con 0x7fa6ac100d50 2024-03-14T09:53:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.872+0000 7fa6a9600700 1 -- 172.21.15.78:0/2040623118 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6a000bda0 con 0x7fa6ac100d50 2024-03-14T09:53:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.872+0000 7fa6b3c00700 1 -- 172.21.15.78:0/2040623118 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ac1a13f0 con 0x7fa6ac100d50 2024-03-14T09:53:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.872+0000 7fa6a9600700 1 -- 172.21.15.78:0/2040623118 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0004a90 con 0x7fa6ac100d50 2024-03-14T09:53:51.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.872+0000 7fa6b3c00700 1 -- 172.21.15.78:0/2040623118 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ac1a1810 con 0x7fa6ac100d50 2024-03-14T09:53:51.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.873+0000 7fa6b3c00700 1 -- 172.21.15.78:0/2040623118 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ac067ac0 con 0x7fa6ac100d50 2024-03-14T09:53:51.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.874+0000 7fa6a9600700 1 -- 172.21.15.78:0/2040623118 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fa6a0004c30 con 0x7fa6ac100d50 2024-03-14T09:53:51.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.876+0000 7fa6a9600700 1 --2- 172.21.15.78:0/2040623118 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa6940a0b20 0x7fa6940a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:51.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.876+0000 7fa6a9600700 1 -- 172.21.15.78:0/2040623118 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7fa6a00cd3d0 con 0x7fa6ac100d50 2024-03-14T09:53:51.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.876+0000 7fa6abe00700 1 --2- 172.21.15.78:0/2040623118 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa6940a0b20 0x7fa6940a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:51.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.876+0000 7fa6abe00700 1 --2- 172.21.15.78:0/2040623118 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa6940a0b20 0x7fa6940a2fe0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa698011550 tx=0x7fa698009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:51.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:51.880+0000 7fa6a9600700 1 -- 172.21.15.78:0/2040623118 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa6a0096900 con 0x7fa6ac100d50 2024-03-14T09:53:52.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.087+0000 7fa6b3c00700 1 -- 172.21.15.78:0/2040623118 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa6ac1a1aa0 con 0x7fa6ac100d50 2024-03-14T09:53:52.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.088+0000 7fa6a9600700 1 -- 172.21.15.78:0/2040623118 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa6a0099f20 con 0x7fa6ac100d50 2024-03-14T09:53:52.089 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:53:52.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.091+0000 7fa6b3c00700 1 -- 172.21.15.78:0/2040623118 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa6940a0b20 msgr2=0x7fa6940a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:52.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.091+0000 7fa6b3c00700 1 --2- 172.21.15.78:0/2040623118 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa6940a0b20 0x7fa6940a2fe0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa698011550 tx=0x7fa698009260 comp rx=0 tx=0).stop 2024-03-14T09:53:52.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.091+0000 7fa6b3c00700 1 -- 172.21.15.78:0/2040623118 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6ac100d50 msgr2=0x7fa6ac199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:52.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.091+0000 7fa6b3c00700 1 --2- 172.21.15.78:0/2040623118 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6ac100d50 0x7fa6ac199370 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa6a00098e0 tx=0x7fa6a000fa00 comp rx=0 tx=0).stop 2024-03-14T09:53:52.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.091+0000 7fa6b3c00700 1 -- 172.21.15.78:0/2040623118 shutdown_connections 2024-03-14T09:53:52.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.091+0000 7fa6b3c00700 1 --2- 172.21.15.78:0/2040623118 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa6940a0b20 0x7fa6940a2fe0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.091+0000 7fa6b3c00700 1 --2- 172.21.15.78:0/2040623118 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6ac1028e0 0x7fa6ac1a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.091+0000 7fa6b3c00700 1 --2- 172.21.15.78:0/2040623118 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6ac100d50 0x7fa6ac199370 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.091+0000 7fa6b3c00700 1 --2- 172.21.15.78:0/2040623118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6ac101f40 0x7fa6ac1998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.091+0000 7fa6b3c00700 1 -- 172.21.15.78:0/2040623118 >> 172.21.15.78:0/2040623118 conn(0x7fa6ac0fc2c0 msgr2=0x7fa6ac072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:52.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.091+0000 7fa6b3c00700 1 -- 172.21.15.78:0/2040623118 shutdown_connections 2024-03-14T09:53:52.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.091+0000 7fa6b3c00700 1 -- 172.21.15.78:0/2040623118 wait complete. 2024-03-14T09:53:52.110 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-14T09:53:52.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.309+0000 7f0bbcc00700 1 -- 172.21.15.78:0/889033199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb8101f30 msgr2=0x7f0bb8102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:52.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.309+0000 7f0bbcc00700 1 --2- 172.21.15.78:0/889033199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb8101f30 0x7f0bb8102390 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0ba40099c0 tx=0x7f0ba4009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.309+0000 7f0bbcc00700 1 -- 172.21.15.78:0/889033199 shutdown_connections 2024-03-14T09:53:52.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.309+0000 7f0bbcc00700 1 --2- 172.21.15.78:0/889033199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0bb81028d0 0x7f0bb8109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.309+0000 7f0bbcc00700 1 --2- 172.21.15.78:0/889033199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bb8100d40 0x7f0bb8101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.309+0000 7f0bbcc00700 1 --2- 172.21.15.78:0/889033199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb8101f30 0x7f0bb8102390 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.309+0000 7f0bbcc00700 1 -- 172.21.15.78:0/889033199 >> 172.21.15.78:0/889033199 conn(0x7f0bb80fc2b0 msgr2=0x7f0bb80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:52.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.309+0000 7f0bbcc00700 1 -- 172.21.15.78:0/889033199 shutdown_connections 2024-03-14T09:53:52.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.309+0000 7f0bbcc00700 1 -- 172.21.15.78:0/889033199 wait complete. 2024-03-14T09:53:52.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.310+0000 7f0bbcc00700 1 Processor -- start 2024-03-14T09:53:52.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.310+0000 7f0bbcc00700 1 -- start start 2024-03-14T09:53:52.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.310+0000 7f0bbcc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb8100d40 0x7f0bb8199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:52.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.310+0000 7f0bbcc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0bb8101f30 0x7f0bb81998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:52.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.310+0000 7f0bbcc00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bb81028d0 0x7f0bb81a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:52.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.310+0000 7f0bbcc00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb81a0e40 con 0x7f0bb8100d40 2024-03-14T09:53:52.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.310+0000 7f0bbcc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb81a0f80 con 0x7f0bb8101f30 2024-03-14T09:53:52.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.310+0000 7f0bbcc00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb81a1110 con 0x7f0bb81028d0 2024-03-14T09:53:52.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.310+0000 7f0bb5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb8100d40 0x7f0bb8199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:52.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.311+0000 7f0bb5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb8100d40 0x7f0bb8199360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52752/0 (socket says 172.21.15.78:52752) 2024-03-14T09:53:52.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.311+0000 7f0bb5e00700 1 -- 172.21.15.78:0/4186748946 learned_addr learned my addr 172.21.15.78:0/4186748946 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:52.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.311+0000 7f0bb5400700 1 --2- 172.21.15.78:0/4186748946 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0bb8101f30 0x7f0bb81998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:52.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.311+0000 7f0bb6800700 1 --2- 172.21.15.78:0/4186748946 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bb81028d0 0x7f0bb81a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:52.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.311+0000 7f0bb5e00700 1 -- 172.21.15.78:0/4186748946 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bb81028d0 msgr2=0x7f0bb81a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:52.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.311+0000 7f0bb5e00700 1 --2- 172.21.15.78:0/4186748946 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bb81028d0 0x7f0bb81a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.311+0000 7f0bb5e00700 1 -- 172.21.15.78:0/4186748946 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0bb8101f30 msgr2=0x7f0bb81998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:52.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.311+0000 7f0bb5e00700 1 --2- 172.21.15.78:0/4186748946 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0bb8101f30 0x7f0bb81998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.311+0000 7f0bb5e00700 1 -- 172.21.15.78:0/4186748946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba40095f0 con 0x7f0bb8100d40 2024-03-14T09:53:52.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.311+0000 7f0bb6800700 1 --2- 172.21.15.78:0/4186748946 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bb81028d0 0x7f0bb81a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:53:52.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.311+0000 7f0bb5e00700 1 --2- 172.21.15.78:0/4186748946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb8100d40 0x7f0bb8199360 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0ba800e970 tx=0x7f0ba800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:52.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.311+0000 7f0baea00700 1 -- 172.21.15.78:0/4186748946 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8009790 con 0x7f0bb8100d40 2024-03-14T09:53:52.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.311+0000 7f0baea00700 1 -- 172.21.15.78:0/4186748946 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ba800eec0 con 0x7f0bb8100d40 2024-03-14T09:53:52.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.311+0000 7f0bbcc00700 1 -- 172.21.15.78:0/4186748946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb81a13f0 con 0x7f0bb8100d40 2024-03-14T09:53:52.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.312+0000 7f0baea00700 1 -- 172.21.15.78:0/4186748946 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8003cf0 con 0x7f0bb8100d40 2024-03-14T09:53:52.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.312+0000 7f0bbcc00700 1 -- 172.21.15.78:0/4186748946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb81a1900 con 0x7f0bb8100d40 2024-03-14T09:53:52.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.313+0000 7f0bbcc00700 1 -- 172.21.15.78:0/4186748946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb8067ac0 con 0x7f0bb8100d40 2024-03-14T09:53:52.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.314+0000 7f0baea00700 1 -- 172.21.15.78:0/4186748946 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f0ba800ccd0 con 0x7f0bb8100d40 2024-03-14T09:53:52.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.316+0000 7f0baea00700 1 --2- 172.21.15.78:0/4186748946 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0b9c0a0b20 0x7f0b9c0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:52.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.316+0000 7f0baea00700 1 -- 172.21.15.78:0/4186748946 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f0ba8014070 con 0x7f0bb8100d40 2024-03-14T09:53:52.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.316+0000 7f0bb5400700 1 --2- 172.21.15.78:0/4186748946 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0b9c0a0b20 0x7f0b9c0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:52.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.316+0000 7f0bb5400700 1 --2- 172.21.15.78:0/4186748946 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0b9c0a0b20 0x7f0b9c0a2fe0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4006010 tx=0x7f0ba400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:52.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.319+0000 7f0baea00700 1 -- 172.21.15.78:0/4186748946 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0ba8095cb0 con 0x7f0bb8100d40 2024-03-14T09:53:52.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:52 smithi152 ceph-mon[48279]: from='client.25367 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:52.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:52 smithi152 ceph-mon[48279]: pgmap v292: 129 pgs: 129 active+clean; 642 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 769 B/s wr, 7 op/s 2024-03-14T09:53:52.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:52 smithi152 ceph-mon[48279]: mgrmap e19: x(active, since 7m), standbys: z, y 2024-03-14T09:53:52.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:52 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1116478050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:52.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:52 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2040623118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:52.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:52 smithi132 ceph-mon[47959]: from='client.25367 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:52.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:52 smithi132 ceph-mon[47959]: pgmap v292: 129 pgs: 129 active+clean; 642 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 769 B/s wr, 7 op/s 2024-03-14T09:53:52.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:52 smithi132 ceph-mon[47959]: mgrmap e19: x(active, since 7m), standbys: z, y 2024-03-14T09:53:52.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:52 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1116478050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:52.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:52 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2040623118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:52.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.526+0000 7f0bbcc00700 1 -- 172.21.15.78:0/4186748946 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0bb81a1be0 con 0x7f0bb8100d40 2024-03-14T09:53:52.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.527+0000 7f0baea00700 1 -- 172.21.15.78:0/4186748946 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0ba80992d0 con 0x7f0bb8100d40 2024-03-14T09:53:52.527 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-14T09:53:52.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.529+0000 7f0bbcc00700 1 -- 172.21.15.78:0/4186748946 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0b9c0a0b20 msgr2=0x7f0b9c0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:52.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.529+0000 7f0bbcc00700 1 --2- 172.21.15.78:0/4186748946 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0b9c0a0b20 0x7f0b9c0a2fe0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4006010 tx=0x7f0ba400b540 comp rx=0 tx=0).stop 2024-03-14T09:53:52.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.529+0000 7f0bbcc00700 1 -- 172.21.15.78:0/4186748946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb8100d40 msgr2=0x7f0bb8199360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:52.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.529+0000 7f0bbcc00700 1 --2- 172.21.15.78:0/4186748946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb8100d40 0x7f0bb8199360 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0ba800e970 tx=0x7f0ba800ec80 comp rx=0 tx=0).stop 2024-03-14T09:53:52.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.529+0000 7f0bbcc00700 1 -- 172.21.15.78:0/4186748946 shutdown_connections 2024-03-14T09:53:52.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.529+0000 7f0bbcc00700 1 --2- 172.21.15.78:0/4186748946 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0bb81028d0 0x7f0bb81a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.529+0000 7f0bbcc00700 1 --2- 172.21.15.78:0/4186748946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bb8100d40 0x7f0bb8199360 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.529+0000 7f0bbcc00700 1 --2- 172.21.15.78:0/4186748946 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f0b9c0a0b20 0x7f0b9c0a2fe0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.529+0000 7f0bbcc00700 1 --2- 172.21.15.78:0/4186748946 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0bb8101f30 0x7f0bb81998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.529+0000 7f0bbcc00700 1 -- 172.21.15.78:0/4186748946 >> 172.21.15.78:0/4186748946 conn(0x7f0bb80fc2b0 msgr2=0x7f0bb8072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:52.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.530+0000 7f0bbcc00700 1 -- 172.21.15.78:0/4186748946 shutdown_connections 2024-03-14T09:53:52.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.530+0000 7f0bbcc00700 1 -- 172.21.15.78:0/4186748946 wait complete. 2024-03-14T09:53:52.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:52 smithi078 ceph-mon[38143]: from='client.25367 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:53:52.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:52 smithi078 ceph-mon[38143]: pgmap v292: 129 pgs: 129 active+clean; 642 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 769 B/s wr, 7 op/s 2024-03-14T09:53:52.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:52 smithi078 ceph-mon[38143]: mgrmap e19: x(active, since 7m), standbys: z, y 2024-03-14T09:53:52.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:52 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1116478050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:52.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:52 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2040623118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:52.549 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-14T09:53:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.746+0000 7f21a0600700 1 -- 172.21.15.78:0/254566087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2198100d50 msgr2=0x7f2198101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:52.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.746+0000 7f21a0600700 1 --2- 172.21.15.78:0/254566087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2198100d50 0x7f2198101170 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f218c0099c0 tx=0x7f218c009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.746+0000 7f21a0600700 1 -- 172.21.15.78:0/254566087 shutdown_connections 2024-03-14T09:53:52.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.746+0000 7f21a0600700 1 --2- 172.21.15.78:0/254566087 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21981028e0 0x7f2198109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.746+0000 7f21a0600700 1 --2- 172.21.15.78:0/254566087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2198100d50 0x7f2198101170 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.746+0000 7f21a0600700 1 --2- 172.21.15.78:0/254566087 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2198101f40 0x7f21981023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.746+0000 7f21a0600700 1 -- 172.21.15.78:0/254566087 >> 172.21.15.78:0/254566087 conn(0x7f21980fc2c0 msgr2=0x7f21980fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:52.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.746+0000 7f21a0600700 1 -- 172.21.15.78:0/254566087 shutdown_connections 2024-03-14T09:53:52.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.746+0000 7f21a0600700 1 -- 172.21.15.78:0/254566087 wait complete. 2024-03-14T09:53:52.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.747+0000 7f21a0600700 1 Processor -- start 2024-03-14T09:53:52.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.747+0000 7f21a0600700 1 -- start start 2024-03-14T09:53:52.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.747+0000 7f21a0600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2198100d50 0x7f2198074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:52.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.747+0000 7f21a0600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2198101f40 0x7f21980731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:52.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.747+0000 7f21a0600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21981028e0 0x7f21980736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:52.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.747+0000 7f21a0600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2198073e80 con 0x7f2198101f40 2024-03-14T09:53:52.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.747+0000 7f21a0600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2198073ff0 con 0x7f2198100d50 2024-03-14T09:53:52.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.747+0000 7f21a0600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21981a1180 con 0x7f21981028e0 2024-03-14T09:53:52.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.747+0000 7f219d200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2198101f40 0x7f21980731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:52.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.748+0000 7f219d200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2198101f40 0x7f21980731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52768/0 (socket says 172.21.15.78:52768) 2024-03-14T09:53:52.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.748+0000 7f219d200700 1 -- 172.21.15.78:0/4028475995 learned_addr learned my addr 172.21.15.78:0/4028475995 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:52.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.748+0000 7f219e600700 1 --2- 172.21.15.78:0/4028475995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21981028e0 0x7f21980736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:52.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.748+0000 7f219dc00700 1 --2- 172.21.15.78:0/4028475995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2198100d50 0x7f2198074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:52.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.748+0000 7f219d200700 1 -- 172.21.15.78:0/4028475995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21981028e0 msgr2=0x7f21980736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:52.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.748+0000 7f219d200700 1 --2- 172.21.15.78:0/4028475995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21981028e0 0x7f21980736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.748+0000 7f219d200700 1 -- 172.21.15.78:0/4028475995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2198100d50 msgr2=0x7f2198074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:52.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.748+0000 7f219d200700 1 --2- 172.21.15.78:0/4028475995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2198100d50 0x7f2198074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.748+0000 7f219d200700 1 -- 172.21.15.78:0/4028475995 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218c0095f0 con 0x7f2198101f40 2024-03-14T09:53:52.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.748+0000 7f219d200700 1 --2- 172.21.15.78:0/4028475995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2198101f40 0x7f21980731a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f218400e9a0 tx=0x7f218400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:52.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.748+0000 7f2196000700 1 -- 172.21.15.78:0/4028475995 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2184009790 con 0x7f2198101f40 2024-03-14T09:53:52.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.748+0000 7f2196000700 1 -- 172.21.15.78:0/4028475995 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f218400ce70 con 0x7f2198101f40 2024-03-14T09:53:52.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.748+0000 7f2196000700 1 -- 172.21.15.78:0/4028475995 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2184018420 con 0x7f2198101f40 2024-03-14T09:53:52.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.748+0000 7f21a0600700 1 -- 172.21.15.78:0/4028475995 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21981a1410 con 0x7f2198101f40 2024-03-14T09:53:52.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.749+0000 7f21a0600700 1 -- 172.21.15.78:0/4028475995 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21981a1920 con 0x7f2198101f40 2024-03-14T09:53:52.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.750+0000 7f21a0600700 1 -- 172.21.15.78:0/4028475995 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2198108170 con 0x7f2198101f40 2024-03-14T09:53:52.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.751+0000 7f2196000700 1 -- 172.21.15.78:0/4028475995 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f21840106f0 con 0x7f2198101f40 2024-03-14T09:53:52.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.753+0000 7f2196000700 1 --2- 172.21.15.78:0/4028475995 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f21800a0ba0 0x7f21800a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:52.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.753+0000 7f2196000700 1 -- 172.21.15.78:0/4028475995 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f2184014070 con 0x7f2198101f40 2024-03-14T09:53:52.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.753+0000 7f219dc00700 1 --2- 172.21.15.78:0/4028475995 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f21800a0ba0 0x7f21800a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:52.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.753+0000 7f219dc00700 1 --2- 172.21.15.78:0/4028475995 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f21800a0ba0 0x7f21800a3060 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f218c009990 tx=0x7f218c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:52.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.757+0000 7f2196000700 1 -- 172.21.15.78:0/4028475995 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2184052960 con 0x7f2198101f40 2024-03-14T09:53:52.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.962+0000 7f21a0600700 1 -- 172.21.15.78:0/4028475995 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f219810c9d0 con 0x7f2198101f40 2024-03-14T09:53:52.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.962+0000 7f2196000700 1 -- 172.21.15.78:0/4028475995 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7f21840995b0 con 0x7f2198101f40 2024-03-14T09:53:52.963 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:53:52.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.965+0000 7f21a0600700 1 -- 172.21.15.78:0/4028475995 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f21800a0ba0 msgr2=0x7f21800a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:52.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.965+0000 7f21a0600700 1 --2- 172.21.15.78:0/4028475995 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f21800a0ba0 0x7f21800a3060 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f218c009990 tx=0x7f218c000bc0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.965+0000 7f21a0600700 1 -- 172.21.15.78:0/4028475995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2198101f40 msgr2=0x7f21980731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:52.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.965+0000 7f21a0600700 1 --2- 172.21.15.78:0/4028475995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2198101f40 0x7f21980731a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f218400e9a0 tx=0x7f218400ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.965+0000 7f21a0600700 1 -- 172.21.15.78:0/4028475995 shutdown_connections 2024-03-14T09:53:52.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.965+0000 7f21a0600700 1 --2- 172.21.15.78:0/4028475995 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f21800a0ba0 0x7f21800a3060 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.965+0000 7f21a0600700 1 --2- 172.21.15.78:0/4028475995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2198100d50 0x7f2198074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.965+0000 7f21a0600700 1 --2- 172.21.15.78:0/4028475995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21981028e0 0x7f21980736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.965+0000 7f21a0600700 1 --2- 172.21.15.78:0/4028475995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2198101f40 0x7f21980731a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:52.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.965+0000 7f21a0600700 1 -- 172.21.15.78:0/4028475995 >> 172.21.15.78:0/4028475995 conn(0x7f21980fc2c0 msgr2=0x7f21981052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:52.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.965+0000 7f21a0600700 1 -- 172.21.15.78:0/4028475995 shutdown_connections 2024-03-14T09:53:52.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:52.965+0000 7f21a0600700 1 -- 172.21.15.78:0/4028475995 wait complete. 2024-03-14T09:53:52.984 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:53:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.178+0000 7f2340400700 1 -- 172.21.15.78:0/2534781509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2338100c50 msgr2=0x7f2338101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.178+0000 7f2340400700 1 --2- 172.21.15.78:0/2534781509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2338100c50 0x7f2338101070 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f232c0099c0 tx=0x7f232c009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.178+0000 7f2340400700 1 -- 172.21.15.78:0/2534781509 shutdown_connections 2024-03-14T09:53:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.178+0000 7f2340400700 1 --2- 172.21.15.78:0/2534781509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23381027e0 0x7f2338109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.178+0000 7f2340400700 1 --2- 172.21.15.78:0/2534781509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2338100c50 0x7f2338101070 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.178+0000 7f2340400700 1 --2- 172.21.15.78:0/2534781509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2338101e40 0x7f23381022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.178+0000 7f2340400700 1 -- 172.21.15.78:0/2534781509 >> 172.21.15.78:0/2534781509 conn(0x7f23380fc1c0 msgr2=0x7f23380fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.178+0000 7f2340400700 1 -- 172.21.15.78:0/2534781509 shutdown_connections 2024-03-14T09:53:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.178+0000 7f2340400700 1 -- 172.21.15.78:0/2534781509 wait complete. 2024-03-14T09:53:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.179+0000 7f2340400700 1 Processor -- start 2024-03-14T09:53:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.179+0000 7f2340400700 1 -- start start 2024-03-14T09:53:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.180+0000 7f2340400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2338100c50 0x7f2338199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.180+0000 7f2340400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2338101e40 0x7f2338199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.180+0000 7f2340400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23381027e0 0x7f23381a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.180+0000 7f2340400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23381a0d50 con 0x7f23381027e0 2024-03-14T09:53:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.180+0000 7f2340400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23381a0e90 con 0x7f2338100c50 2024-03-14T09:53:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.180+0000 7f2340400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23381a1020 con 0x7f2338101e40 2024-03-14T09:53:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.180+0000 7f233e400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23381027e0 0x7f23381a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.180+0000 7f233e400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23381027e0 0x7f23381a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:52798/0 (socket says 172.21.15.78:52798) 2024-03-14T09:53:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.180+0000 7f233e400700 1 -- 172.21.15.78:0/624374165 learned_addr learned my addr 172.21.15.78:0/624374165 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.180+0000 7f233da00700 1 --2- 172.21.15.78:0/624374165 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2338100c50 0x7f2338199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.180+0000 7f233d000700 1 --2- 172.21.15.78:0/624374165 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2338101e40 0x7f2338199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.180+0000 7f233e400700 1 -- 172.21.15.78:0/624374165 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2338101e40 msgr2=0x7f2338199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.180+0000 7f233e400700 1 --2- 172.21.15.78:0/624374165 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2338101e40 0x7f2338199760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.180+0000 7f233e400700 1 -- 172.21.15.78:0/624374165 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2338100c50 msgr2=0x7f2338199220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.180+0000 7f233e400700 1 --2- 172.21.15.78:0/624374165 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2338100c50 0x7f2338199220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.180+0000 7f233e400700 1 -- 172.21.15.78:0/624374165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c0095f0 con 0x7f23381027e0 2024-03-14T09:53:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.181+0000 7f233e400700 1 --2- 172.21.15.78:0/624374165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23381027e0 0x7f23381a0780 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f233000ef30 tx=0x7f233000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.181+0000 7f2336000700 1 -- 172.21.15.78:0/624374165 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2330018070 con 0x7f23381027e0 2024-03-14T09:53:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.181+0000 7f2336000700 1 -- 172.21.15.78:0/624374165 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f233000f040 con 0x7f23381027e0 2024-03-14T09:53:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.181+0000 7f2340400700 1 -- 172.21.15.78:0/624374165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23381a12a0 con 0x7f23381027e0 2024-03-14T09:53:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.181+0000 7f2336000700 1 -- 172.21.15.78:0/624374165 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233000cd00 con 0x7f23381027e0 2024-03-14T09:53:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.181+0000 7f2340400700 1 -- 172.21.15.78:0/624374165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23381a17b0 con 0x7f23381027e0 2024-03-14T09:53:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.182+0000 7f2340400700 1 -- 172.21.15.78:0/624374165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23380679e0 con 0x7f23381027e0 2024-03-14T09:53:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.184+0000 7f2336000700 1 -- 172.21.15.78:0/624374165 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f2330003a00 con 0x7f23381027e0 2024-03-14T09:53:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.185+0000 7f2336000700 1 --2- 172.21.15.78:0/624374165 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f23200a0b20 0x7f23200a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.185+0000 7f2336000700 1 -- 172.21.15.78:0/624374165 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f23300cbfe0 con 0x7f23381027e0 2024-03-14T09:53:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.186+0000 7f233da00700 1 --2- 172.21.15.78:0/624374165 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f23200a0b20 0x7f23200a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.186+0000 7f233da00700 1 --2- 172.21.15.78:0/624374165 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f23200a0b20 0x7f23200a2fe0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f232c003730 tx=0x7f232c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:53.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.189+0000 7f2336000700 1 -- 172.21.15.78:0/624374165 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f23300ff960 con 0x7f23381027e0 2024-03-14T09:53:53.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:53 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/4186748946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:53.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:53 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/4028475995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:53.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.413+0000 7f2340400700 1 -- 172.21.15.78:0/624374165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f233810c8d0 con 0x7f23381027e0 2024-03-14T09:53:53.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.415+0000 7f2336000700 1 -- 172.21.15.78:0/624374165 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24594 (secure 0 0 0) 0x7f2330095570 con 0x7f23381027e0 2024-03-14T09:53:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:53.416 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":87,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:53:42.786378+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:53:53.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.417+0000 7f2340400700 1 -- 172.21.15.78:0/624374165 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f23200a0b20 msgr2=0x7f23200a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:53.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.417+0000 7f2340400700 1 --2- 172.21.15.78:0/624374165 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f23200a0b20 0x7f23200a2fe0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f232c003730 tx=0x7f232c000bc0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.417+0000 7f2340400700 1 -- 172.21.15.78:0/624374165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23381027e0 msgr2=0x7f23381a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:53.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.417+0000 7f2340400700 1 --2- 172.21.15.78:0/624374165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23381027e0 0x7f23381a0780 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f233000ef30 tx=0x7f233000c3a0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.417+0000 7f2340400700 1 -- 172.21.15.78:0/624374165 shutdown_connections 2024-03-14T09:53:53.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.417+0000 7f2340400700 1 --2- 172.21.15.78:0/624374165 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f23200a0b20 0x7f23200a2fe0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.417+0000 7f2340400700 1 --2- 172.21.15.78:0/624374165 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2338100c50 0x7f2338199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.417+0000 7f2340400700 1 --2- 172.21.15.78:0/624374165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23381027e0 0x7f23381a0780 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.417+0000 7f2340400700 1 --2- 172.21.15.78:0/624374165 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2338101e40 0x7f2338199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.417+0000 7f2340400700 1 -- 172.21.15.78:0/624374165 >> 172.21.15.78:0/624374165 conn(0x7f23380fc1c0 msgr2=0x7f2338072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:53.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.417+0000 7f2340400700 1 -- 172.21.15.78:0/624374165 shutdown_connections 2024-03-14T09:53:53.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.417+0000 7f2340400700 1 -- 172.21.15.78:0/624374165 wait complete. 2024-03-14T09:53:53.437 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T09:48:37.618450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-14T09:52:03.938666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-14T09:52:05.149985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-14T09:52:08.227698+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-14T09:53:53.438 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T09:53:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:53 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/4186748946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:53 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/4028475995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:53.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:53 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/4186748946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:53.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:53 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/4028475995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:53.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.633+0000 7f9e38000700 1 -- 172.21.15.78:0/2274317141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e30101850 msgr2=0x7f9e30103c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:53.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.633+0000 7f9e38000700 1 --2- 172.21.15.78:0/2274317141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e30101850 0x7f9e30103c40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009990 tx=0x7f9e24009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.633+0000 7f9e38000700 1 -- 172.21.15.78:0/2274317141 shutdown_connections 2024-03-14T09:53:53.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.633+0000 7f9e38000700 1 --2- 172.21.15.78:0/2274317141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e30106ab0 0x7f9e30108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.633+0000 7f9e38000700 1 --2- 172.21.15.78:0/2274317141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e30101850 0x7f9e30103c40 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.633+0000 7f9e38000700 1 --2- 172.21.15.78:0/2274317141 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9e30104180 0x7f9e30106570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.633+0000 7f9e38000700 1 -- 172.21.15.78:0/2274317141 >> 172.21.15.78:0/2274317141 conn(0x7f9e300fb1a0 msgr2=0x7f9e300fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:53.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.633+0000 7f9e38000700 1 -- 172.21.15.78:0/2274317141 shutdown_connections 2024-03-14T09:53:53.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.633+0000 7f9e38000700 1 -- 172.21.15.78:0/2274317141 wait complete. 2024-03-14T09:53:53.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.634+0000 7f9e38000700 1 Processor -- start 2024-03-14T09:53:53.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.634+0000 7f9e38000700 1 -- start start 2024-03-14T09:53:53.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e38000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e30101850 0x7f9e301970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e38000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9e30104180 0x7f9e30197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e38000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e30106ab0 0x7f9e3019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e38000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3019ec20 con 0x7f9e30106ab0 2024-03-14T09:53:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e38000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3019ed60 con 0x7f9e30101850 2024-03-14T09:53:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e38000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3019eef0 con 0x7f9e30104180 2024-03-14T09:53:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e34c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9e30104180 0x7f9e30197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e36000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e30106ab0 0x7f9e3019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e34c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9e30104180 0x7f9e30197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:45188/0 (socket says 172.21.15.78:45188) 2024-03-14T09:53:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e34c00700 1 -- 172.21.15.78:0/750058768 learned_addr learned my addr 172.21.15.78:0/750058768 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e35600700 1 --2- 172.21.15.78:0/750058768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e30101850 0x7f9e301970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e34c00700 1 -- 172.21.15.78:0/750058768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e30101850 msgr2=0x7f9e301970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e34c00700 1 --2- 172.21.15.78:0/750058768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e30101850 0x7f9e301970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e34c00700 1 -- 172.21.15.78:0/750058768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e30106ab0 msgr2=0x7f9e3019e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e34c00700 1 --2- 172.21.15.78:0/750058768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e30106ab0 0x7f9e3019e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e34c00700 1 -- 172.21.15.78:0/750058768 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e240095f0 con 0x7f9e30104180 2024-03-14T09:53:53.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.635+0000 7f9e36000700 1 --2- 172.21.15.78:0/750058768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e30106ab0 0x7f9e3019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:53.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.636+0000 7f9e34c00700 1 --2- 172.21.15.78:0/750058768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9e30104180 0x7f9e30197630 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00e9a0 tx=0x7f9e1c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:53.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.636+0000 7f9e2e000700 1 -- 172.21.15.78:0/750058768 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c009790 con 0x7f9e30104180 2024-03-14T09:53:53.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.636+0000 7f9e2e000700 1 -- 172.21.15.78:0/750058768 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e1c010430 con 0x7f9e30104180 2024-03-14T09:53:53.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.636+0000 7f9e38000700 1 -- 172.21.15.78:0/750058768 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e3019f1d0 con 0x7f9e30104180 2024-03-14T09:53:53.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.636+0000 7f9e2e000700 1 -- 172.21.15.78:0/750058768 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c018680 con 0x7f9e30104180 2024-03-14T09:53:53.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.636+0000 7f9e38000700 1 -- 172.21.15.78:0/750058768 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e3019f6e0 con 0x7f9e30104180 2024-03-14T09:53:53.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.637+0000 7f9e38000700 1 -- 172.21.15.78:0/750058768 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e300fcd20 con 0x7f9e30104180 2024-03-14T09:53:53.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.639+0000 7f9e2e000700 1 -- 172.21.15.78:0/750058768 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f9e1c0105a0 con 0x7f9e30104180 2024-03-14T09:53:53.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.640+0000 7f9e2e000700 1 --2- 172.21.15.78:0/750058768 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9e180a0b20 0x7f9e180a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:53.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.640+0000 7f9e2e000700 1 -- 172.21.15.78:0/750058768 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f9e1c014070 con 0x7f9e30104180 2024-03-14T09:53:53.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.641+0000 7f9e35600700 1 --2- 172.21.15.78:0/750058768 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9e180a0b20 0x7f9e180a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:53.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.641+0000 7f9e35600700 1 --2- 172.21.15.78:0/750058768 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9e180a0b20 0x7f9e180a2fe0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400b2e0 tx=0x7f9e24005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:53.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.643+0000 7f9e2e000700 1 -- 172.21.15.78:0/750058768 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9e1c095ee0 con 0x7f9e30104180 2024-03-14T09:53:53.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.853+0000 7f9e38000700 1 -- 172.21.15.78:0/750058768 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9e3010ca90 con 0x7f9e30104180 2024-03-14T09:53:53.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.853+0000 7f9e2e000700 1 -- 172.21.15.78:0/750058768 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9e1c099500 con 0x7f9e30104180 2024-03-14T09:53:53.854 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-14T09:53:53.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.856+0000 7f9e38000700 1 -- 172.21.15.78:0/750058768 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9e180a0b20 msgr2=0x7f9e180a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:53.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.856+0000 7f9e38000700 1 --2- 172.21.15.78:0/750058768 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9e180a0b20 0x7f9e180a2fe0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400b2e0 tx=0x7f9e24005fb0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.856+0000 7f9e38000700 1 -- 172.21.15.78:0/750058768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9e30104180 msgr2=0x7f9e30197630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:53.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.856+0000 7f9e38000700 1 --2- 172.21.15.78:0/750058768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9e30104180 0x7f9e30197630 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00e9a0 tx=0x7f9e1c00ecb0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.856+0000 7f9e38000700 1 -- 172.21.15.78:0/750058768 shutdown_connections 2024-03-14T09:53:53.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.856+0000 7f9e38000700 1 --2- 172.21.15.78:0/750058768 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f9e180a0b20 0x7f9e180a2fe0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.856+0000 7f9e38000700 1 --2- 172.21.15.78:0/750058768 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e30101850 0x7f9e301970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.856+0000 7f9e38000700 1 --2- 172.21.15.78:0/750058768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e30106ab0 0x7f9e3019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.856+0000 7f9e38000700 1 --2- 172.21.15.78:0/750058768 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9e30104180 0x7f9e30197630 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:53.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.856+0000 7f9e38000700 1 -- 172.21.15.78:0/750058768 >> 172.21.15.78:0/750058768 conn(0x7f9e300fb1a0 msgr2=0x7f9e30102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:53.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.856+0000 7f9e38000700 1 -- 172.21.15.78:0/750058768 shutdown_connections 2024-03-14T09:53:53.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:53.856+0000 7f9e38000700 1 -- 172.21.15.78:0/750058768 wait complete. 2024-03-14T09:53:53.875 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-14T09:53:54.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.073+0000 7f249fa00700 1 -- 172.21.15.78:0/3278860717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24981028e0 msgr2=0x7f2498109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:54.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.073+0000 7f249fa00700 1 --2- 172.21.15.78:0/3278860717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24981028e0 0x7f2498109190 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f249000b0d0 tx=0x7f249000b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.073+0000 7f249fa00700 1 -- 172.21.15.78:0/3278860717 shutdown_connections 2024-03-14T09:53:54.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.074+0000 7f249fa00700 1 --2- 172.21.15.78:0/3278860717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2498100d50 0x7f2498101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.074+0000 7f249fa00700 1 --2- 172.21.15.78:0/3278860717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24981028e0 0x7f2498109190 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.074+0000 7f249fa00700 1 --2- 172.21.15.78:0/3278860717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2498101f40 0x7f24981023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.074+0000 7f249fa00700 1 -- 172.21.15.78:0/3278860717 >> 172.21.15.78:0/3278860717 conn(0x7f24980fc2c0 msgr2=0x7f24980fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:54.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.074+0000 7f249fa00700 1 -- 172.21.15.78:0/3278860717 shutdown_connections 2024-03-14T09:53:54.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.074+0000 7f249fa00700 1 -- 172.21.15.78:0/3278860717 wait complete. 2024-03-14T09:53:54.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.075+0000 7f249fa00700 1 Processor -- start 2024-03-14T09:53:54.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.075+0000 7f249fa00700 1 -- start start 2024-03-14T09:53:54.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.075+0000 7f249fa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2498100d50 0x7f2498074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:54.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.075+0000 7f249fa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2498101f40 0x7f24980731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:54.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.075+0000 7f249fa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f24981028e0 0x7f24980736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.075+0000 7f249fa00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2498073e80 con 0x7f2498100d50 2024-03-14T09:53:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.075+0000 7f249fa00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2498073ff0 con 0x7f2498101f40 2024-03-14T09:53:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.075+0000 7f249fa00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24981a1250 con 0x7f24981028e0 2024-03-14T09:53:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.075+0000 7f249d000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2498100d50 0x7f2498074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.075+0000 7f249d000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2498100d50 0x7f2498074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43818/0 (socket says 172.21.15.78:43818) 2024-03-14T09:53:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.075+0000 7f249d000700 1 -- 172.21.15.78:0/2603148342 learned_addr learned my addr 172.21.15.78:0/2603148342 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.075+0000 7f249da00700 1 --2- 172.21.15.78:0/2603148342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f24981028e0 0x7f24980736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.075+0000 7f2497e00700 1 --2- 172.21.15.78:0/2603148342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2498101f40 0x7f24980731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.076+0000 7f249d000700 1 -- 172.21.15.78:0/2603148342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f24981028e0 msgr2=0x7f24980736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.076+0000 7f249d000700 1 --2- 172.21.15.78:0/2603148342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f24981028e0 0x7f24980736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.076+0000 7f249d000700 1 -- 172.21.15.78:0/2603148342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2498101f40 msgr2=0x7f24980731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.076+0000 7f249d000700 1 --2- 172.21.15.78:0/2603148342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2498101f40 0x7f24980731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.076+0000 7f249d000700 1 -- 172.21.15.78:0/2603148342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2490009d00 con 0x7f2498100d50 2024-03-14T09:53:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.076+0000 7f249d000700 1 --2- 172.21.15.78:0/2603148342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2498100d50 0x7f2498074b10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f248c00e970 tx=0x7f248c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.076+0000 7f2495600700 1 -- 172.21.15.78:0/2603148342 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c009790 con 0x7f2498100d50 2024-03-14T09:53:54.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.076+0000 7f249fa00700 1 -- 172.21.15.78:0/2603148342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24981a14e0 con 0x7f2498100d50 2024-03-14T09:53:54.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.076+0000 7f2495600700 1 -- 172.21.15.78:0/2603148342 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f248c00c780 con 0x7f2498100d50 2024-03-14T09:53:54.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.076+0000 7f2495600700 1 -- 172.21.15.78:0/2603148342 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c003bc0 con 0x7f2498100d50 2024-03-14T09:53:54.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.076+0000 7f249fa00700 1 -- 172.21.15.78:0/2603148342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24981a19f0 con 0x7f2498100d50 2024-03-14T09:53:54.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.077+0000 7f249fa00700 1 -- 172.21.15.78:0/2603148342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2498108100 con 0x7f2498100d50 2024-03-14T09:53:54.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.079+0000 7f2495600700 1 -- 172.21.15.78:0/2603148342 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f248c003d60 con 0x7f2498100d50 2024-03-14T09:53:54.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.080+0000 7f2495600700 1 --2- 172.21.15.78:0/2603148342 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f24800a0b20 0x7f24800a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:54.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.080+0000 7f2495600700 1 -- 172.21.15.78:0/2603148342 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f248c014070 con 0x7f2498100d50 2024-03-14T09:53:54.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.081+0000 7f2497e00700 1 --2- 172.21.15.78:0/2603148342 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f24800a0b20 0x7f24800a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:54.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.081+0000 7f2497e00700 1 --2- 172.21.15.78:0/2603148342 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f24800a0b20 0x7f24800a2fe0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2484009c60 tx=0x7f2484009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:54.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.084+0000 7f2495600700 1 -- 172.21.15.78:0/2603148342 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f248c095b20 con 0x7f2498100d50 2024-03-14T09:53:54.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.289+0000 7f249fa00700 1 -- 172.21.15.78:0/2603148342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2498107ae0 con 0x7f2498100d50 2024-03-14T09:53:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.289+0000 7f2495600700 1 -- 172.21.15.78:0/2603148342 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f248c099140 con 0x7f2498100d50 2024-03-14T09:53:54.290 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:53:54.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.292+0000 7f249fa00700 1 -- 172.21.15.78:0/2603148342 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f24800a0b20 msgr2=0x7f24800a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.292+0000 7f249fa00700 1 --2- 172.21.15.78:0/2603148342 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f24800a0b20 0x7f24800a2fe0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2484009c60 tx=0x7f2484009260 comp rx=0 tx=0).stop 2024-03-14T09:53:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.292+0000 7f249fa00700 1 -- 172.21.15.78:0/2603148342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2498100d50 msgr2=0x7f2498074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.292+0000 7f249fa00700 1 --2- 172.21.15.78:0/2603148342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2498100d50 0x7f2498074b10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f248c00e970 tx=0x7f248c00ec80 comp rx=0 tx=0).stop 2024-03-14T09:53:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.292+0000 7f249fa00700 1 -- 172.21.15.78:0/2603148342 shutdown_connections 2024-03-14T09:53:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.292+0000 7f249fa00700 1 --2- 172.21.15.78:0/2603148342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f24981028e0 0x7f24980736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.292+0000 7f249fa00700 1 --2- 172.21.15.78:0/2603148342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2498100d50 0x7f2498074b10 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.292+0000 7f249fa00700 1 --2- 172.21.15.78:0/2603148342 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f24800a0b20 0x7f24800a2fe0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.292+0000 7f249fa00700 1 --2- 172.21.15.78:0/2603148342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2498101f40 0x7f24980731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.292+0000 7f249fa00700 1 -- 172.21.15.78:0/2603148342 >> 172.21.15.78:0/2603148342 conn(0x7f24980fc2c0 msgr2=0x7f2498105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.292+0000 7f249fa00700 1 -- 172.21.15.78:0/2603148342 shutdown_connections 2024-03-14T09:53:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.292+0000 7f249fa00700 1 -- 172.21.15.78:0/2603148342 wait complete. 2024-03-14T09:53:54.310 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-14T09:53:54.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:54 smithi152 ceph-mon[48279]: pgmap v293: 129 pgs: 129 active+clean; 642 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 893 B/s wr, 7 op/s 2024-03-14T09:53:54.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:54 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/624374165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:54.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:54 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/750058768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:54.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.507+0000 7fa639c00700 1 -- 172.21.15.78:0/3451466639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa634101f40 msgr2=0x7fa6341023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:54.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.507+0000 7fa639c00700 1 --2- 172.21.15.78:0/3451466639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa634101f40 0x7fa6341023a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa6200099c0 tx=0x7fa620009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.507+0000 7fa639c00700 1 -- 172.21.15.78:0/3451466639 shutdown_connections 2024-03-14T09:53:54.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.507+0000 7fa639c00700 1 --2- 172.21.15.78:0/3451466639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa634100d50 0x7fa634101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.507+0000 7fa639c00700 1 --2- 172.21.15.78:0/3451466639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa6341028e0 0x7fa634109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.507+0000 7fa639c00700 1 --2- 172.21.15.78:0/3451466639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa634101f40 0x7fa6341023a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.507+0000 7fa639c00700 1 -- 172.21.15.78:0/3451466639 >> 172.21.15.78:0/3451466639 conn(0x7fa6340fc2c0 msgr2=0x7fa6340fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:54.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.507+0000 7fa639c00700 1 -- 172.21.15.78:0/3451466639 shutdown_connections 2024-03-14T09:53:54.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.507+0000 7fa639c00700 1 -- 172.21.15.78:0/3451466639 wait complete. 2024-03-14T09:53:54.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.508+0000 7fa639c00700 1 Processor -- start 2024-03-14T09:53:54.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.508+0000 7fa639c00700 1 -- start start 2024-03-14T09:53:54.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.508+0000 7fa639c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa634100d50 0x7fa634074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.509+0000 7fa639c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa634101f40 0x7fa6340731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.509+0000 7fa639c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6341028e0 0x7fa6340736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.509+0000 7fa639c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa634073e80 con 0x7fa634101f40 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.509+0000 7fa639c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa634073ff0 con 0x7fa6341028e0 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.509+0000 7fa639c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6341a1180 con 0x7fa634100d50 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.509+0000 7fa632000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa634101f40 0x7fa6340731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.509+0000 7fa632000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa634101f40 0x7fa6340731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43832/0 (socket says 172.21.15.78:43832) 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.509+0000 7fa632000700 1 -- 172.21.15.78:0/1111894409 learned_addr learned my addr 172.21.15.78:0/1111894409 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.509+0000 7fa632a00700 1 --2- 172.21.15.78:0/1111894409 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa634100d50 0x7fa634074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.509+0000 7fa633400700 1 --2- 172.21.15.78:0/1111894409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6341028e0 0x7fa6340736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.509+0000 7fa632000700 1 -- 172.21.15.78:0/1111894409 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa634100d50 msgr2=0x7fa634074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.509+0000 7fa632000700 1 --2- 172.21.15.78:0/1111894409 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa634100d50 0x7fa634074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.509+0000 7fa632000700 1 -- 172.21.15.78:0/1111894409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6341028e0 msgr2=0x7fa6340736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.509+0000 7fa632000700 1 --2- 172.21.15.78:0/1111894409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6341028e0 0x7fa6340736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.509+0000 7fa632000700 1 -- 172.21.15.78:0/1111894409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6200095f0 con 0x7fa634101f40 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.509+0000 7fa632000700 1 --2- 172.21.15.78:0/1111894409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa634101f40 0x7fa6340731a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa620006010 tx=0x7fa62000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:54.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.510+0000 7fa62b400700 1 -- 172.21.15.78:0/1111894409 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62001d070 con 0x7fa634101f40 2024-03-14T09:53:54.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.510+0000 7fa62b400700 1 -- 172.21.15.78:0/1111894409 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa620005030 con 0x7fa634101f40 2024-03-14T09:53:54.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.510+0000 7fa639c00700 1 -- 172.21.15.78:0/1111894409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6341a13b0 con 0x7fa634101f40 2024-03-14T09:53:54.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.510+0000 7fa62b400700 1 -- 172.21.15.78:0/1111894409 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa620004b10 con 0x7fa634101f40 2024-03-14T09:53:54.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.510+0000 7fa639c00700 1 -- 172.21.15.78:0/1111894409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6341a17d0 con 0x7fa634101f40 2024-03-14T09:53:54.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.511+0000 7fa639c00700 1 -- 172.21.15.78:0/1111894409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6341081d0 con 0x7fa634101f40 2024-03-14T09:53:54.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.513+0000 7fa62b400700 1 -- 172.21.15.78:0/1111894409 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fa6200051a0 con 0x7fa634101f40 2024-03-14T09:53:54.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.514+0000 7fa62b400700 1 --2- 172.21.15.78:0/1111894409 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa6180a0ba0 0x7fa6180a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:54.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.514+0000 7fa62b400700 1 -- 172.21.15.78:0/1111894409 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7fa6200cd200 con 0x7fa634101f40 2024-03-14T09:53:54.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.514+0000 7fa632a00700 1 --2- 172.21.15.78:0/1111894409 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa6180a0ba0 0x7fa6180a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:54.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.515+0000 7fa632a00700 1 --2- 172.21.15.78:0/1111894409 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa6180a0ba0 0x7fa6180a3060 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa634101da0 tx=0x7fa624009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:54.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.518+0000 7fa62b400700 1 -- 172.21.15.78:0/1111894409 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa620096730 con 0x7fa634101f40 2024-03-14T09:53:54.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:54 smithi132 ceph-mon[47959]: pgmap v293: 129 pgs: 129 active+clean; 642 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 893 B/s wr, 7 op/s 2024-03-14T09:53:54.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:54 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/624374165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:54.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:54 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/750058768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:54.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:54 smithi078 ceph-mon[38143]: pgmap v293: 129 pgs: 129 active+clean; 642 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 893 B/s wr, 7 op/s 2024-03-14T09:53:54.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:54 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/624374165' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:54.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:54 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/750058768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:54.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.721+0000 7fa639c00700 1 -- 172.21.15.78:0/1111894409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa63410cae0 con 0x7fa634101f40 2024-03-14T09:53:54.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.721+0000 7fa62b400700 1 -- 172.21.15.78:0/1111894409 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa620099d50 con 0x7fa634101f40 2024-03-14T09:53:54.722 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-14T09:53:54.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.724+0000 7fa639c00700 1 -- 172.21.15.78:0/1111894409 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa6180a0ba0 msgr2=0x7fa6180a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:54.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.724+0000 7fa639c00700 1 --2- 172.21.15.78:0/1111894409 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa6180a0ba0 0x7fa6180a3060 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa634101da0 tx=0x7fa624009260 comp rx=0 tx=0).stop 2024-03-14T09:53:54.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.724+0000 7fa639c00700 1 -- 172.21.15.78:0/1111894409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa634101f40 msgr2=0x7fa6340731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:54.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.724+0000 7fa639c00700 1 --2- 172.21.15.78:0/1111894409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa634101f40 0x7fa6340731a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa620006010 tx=0x7fa62000f690 comp rx=0 tx=0).stop 2024-03-14T09:53:54.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.724+0000 7fa639c00700 1 -- 172.21.15.78:0/1111894409 shutdown_connections 2024-03-14T09:53:54.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.724+0000 7fa639c00700 1 --2- 172.21.15.78:0/1111894409 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa6180a0ba0 0x7fa6180a3060 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.724+0000 7fa639c00700 1 --2- 172.21.15.78:0/1111894409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6341028e0 0x7fa6340736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.724+0000 7fa639c00700 1 --2- 172.21.15.78:0/1111894409 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa634100d50 0x7fa634074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.724+0000 7fa639c00700 1 --2- 172.21.15.78:0/1111894409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa634101f40 0x7fa6340731a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.724+0000 7fa639c00700 1 -- 172.21.15.78:0/1111894409 >> 172.21.15.78:0/1111894409 conn(0x7fa6340fc2c0 msgr2=0x7fa634105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:54.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.724+0000 7fa639c00700 1 -- 172.21.15.78:0/1111894409 shutdown_connections 2024-03-14T09:53:54.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.724+0000 7fa639c00700 1 -- 172.21.15.78:0/1111894409 wait complete. 2024-03-14T09:53:54.743 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-14T09:53:54.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.940+0000 7ffb2d200700 1 -- 172.21.15.78:0/2437601476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb28100d50 msgr2=0x7ffb28101170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:54.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.940+0000 7ffb2d200700 1 --2- 172.21.15.78:0/2437601476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb28100d50 0x7ffb28101170 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ffb180099c0 tx=0x7ffb18009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.940+0000 7ffb2d200700 1 -- 172.21.15.78:0/2437601476 shutdown_connections 2024-03-14T09:53:54.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.940+0000 7ffb2d200700 1 --2- 172.21.15.78:0/2437601476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb281028e0 0x7ffb28109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.940+0000 7ffb2d200700 1 --2- 172.21.15.78:0/2437601476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb28100d50 0x7ffb28101170 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.940+0000 7ffb2d200700 1 --2- 172.21.15.78:0/2437601476 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb28101f40 0x7ffb281023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.940+0000 7ffb2d200700 1 -- 172.21.15.78:0/2437601476 >> 172.21.15.78:0/2437601476 conn(0x7ffb280fc2c0 msgr2=0x7ffb280fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:54.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.940+0000 7ffb2d200700 1 -- 172.21.15.78:0/2437601476 shutdown_connections 2024-03-14T09:53:54.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.940+0000 7ffb2d200700 1 -- 172.21.15.78:0/2437601476 wait complete. 2024-03-14T09:53:54.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.941+0000 7ffb2d200700 1 Processor -- start 2024-03-14T09:53:54.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.941+0000 7ffb2d200700 1 -- start start 2024-03-14T09:53:54.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.941+0000 7ffb2d200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb28100d50 0x7ffb28197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:54.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.941+0000 7ffb2d200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb28101f40 0x7ffb28197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:54.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.941+0000 7ffb2d200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb281028e0 0x7ffb2819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:54.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.941+0000 7ffb2d200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2819ec50 con 0x7ffb28101f40 2024-03-14T09:53:54.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.941+0000 7ffb2d200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2819ed90 con 0x7ffb28100d50 2024-03-14T09:53:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.941+0000 7ffb2d200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2819ef20 con 0x7ffb281028e0 2024-03-14T09:53:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.941+0000 7ffb25400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb28101f40 0x7ffb28197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.941+0000 7ffb25e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb28100d50 0x7ffb28197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.941+0000 7ffb25e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb28100d50 0x7ffb28197120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:44128/0 (socket says 172.21.15.78:44128) 2024-03-14T09:53:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.941+0000 7ffb25400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb28101f40 0x7ffb28197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43860/0 (socket says 172.21.15.78:43860) 2024-03-14T09:53:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.941+0000 7ffb25e00700 1 -- 172.21.15.78:0/311079999 learned_addr learned my addr 172.21.15.78:0/311079999 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.942+0000 7ffb26800700 1 --2- 172.21.15.78:0/311079999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb281028e0 0x7ffb2819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.942+0000 7ffb25400700 1 -- 172.21.15.78:0/311079999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb281028e0 msgr2=0x7ffb2819e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.942+0000 7ffb25400700 1 --2- 172.21.15.78:0/311079999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb281028e0 0x7ffb2819e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.942+0000 7ffb25400700 1 -- 172.21.15.78:0/311079999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb28100d50 msgr2=0x7ffb28197120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.942+0000 7ffb25400700 1 --2- 172.21.15.78:0/311079999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb28100d50 0x7ffb28197120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.942+0000 7ffb25400700 1 -- 172.21.15.78:0/311079999 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb180095f0 con 0x7ffb28101f40 2024-03-14T09:53:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.942+0000 7ffb25400700 1 --2- 172.21.15.78:0/311079999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb28101f40 0x7ffb28197660 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400d950 tx=0x7ffb1400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.942+0000 7ffb1ea00700 1 -- 172.21.15.78:0/311079999 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1400f890 con 0x7ffb28101f40 2024-03-14T09:53:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.942+0000 7ffb2d200700 1 -- 172.21.15.78:0/311079999 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb28073100 con 0x7ffb28101f40 2024-03-14T09:53:54.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.942+0000 7ffb1ea00700 1 -- 172.21.15.78:0/311079999 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb14010460 con 0x7ffb28101f40 2024-03-14T09:53:54.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.942+0000 7ffb1ea00700 1 -- 172.21.15.78:0/311079999 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1400e610 con 0x7ffb28101f40 2024-03-14T09:53:54.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.942+0000 7ffb2d200700 1 -- 172.21.15.78:0/311079999 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb28073610 con 0x7ffb28101f40 2024-03-14T09:53:54.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.943+0000 7ffb2d200700 1 -- 172.21.15.78:0/311079999 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb28108170 con 0x7ffb28101f40 2024-03-14T09:53:54.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.945+0000 7ffb1ea00700 1 -- 172.21.15.78:0/311079999 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7ffb140105d0 con 0x7ffb28101f40 2024-03-14T09:53:54.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.946+0000 7ffb1ea00700 1 --2- 172.21.15.78:0/311079999 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffb0c0a0b20 0x7ffb0c0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:54.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.947+0000 7ffb1ea00700 1 -- 172.21.15.78:0/311079999 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7ffb140cbe10 con 0x7ffb28101f40 2024-03-14T09:53:54.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.947+0000 7ffb25e00700 1 --2- 172.21.15.78:0/311079999 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffb0c0a0b20 0x7ffb0c0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:54.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.947+0000 7ffb25e00700 1 --2- 172.21.15.78:0/311079999 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffb0c0a0b20 0x7ffb0c0a2fe0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ffb18003820 tx=0x7ffb18005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:54.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:54.951+0000 7ffb1ea00700 1 -- 172.21.15.78:0/311079999 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ffb14095310 con 0x7ffb28101f40 2024-03-14T09:53:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.158+0000 7ffb2d200700 1 -- 172.21.15.78:0/311079999 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ffb28107b50 con 0x7ffb28101f40 2024-03-14T09:53:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.159+0000 7ffb1ea00700 1 -- 172.21.15.78:0/311079999 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7ffb140160a0 con 0x7ffb28101f40 2024-03-14T09:53:55.159 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:53:55.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.161+0000 7ffb2d200700 1 -- 172.21.15.78:0/311079999 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffb0c0a0b20 msgr2=0x7ffb0c0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:55.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.161+0000 7ffb2d200700 1 --2- 172.21.15.78:0/311079999 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffb0c0a0b20 0x7ffb0c0a2fe0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ffb18003820 tx=0x7ffb18005ee0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.161+0000 7ffb2d200700 1 -- 172.21.15.78:0/311079999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb28101f40 msgr2=0x7ffb28197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:55.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.161+0000 7ffb2d200700 1 --2- 172.21.15.78:0/311079999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb28101f40 0x7ffb28197660 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400d950 tx=0x7ffb1400dc60 comp rx=0 tx=0).stop 2024-03-14T09:53:55.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.161+0000 7ffb2d200700 1 -- 172.21.15.78:0/311079999 shutdown_connections 2024-03-14T09:53:55.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.161+0000 7ffb2d200700 1 --2- 172.21.15.78:0/311079999 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ffb0c0a0b20 0x7ffb0c0a2fe0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.161+0000 7ffb2d200700 1 --2- 172.21.15.78:0/311079999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffb28100d50 0x7ffb28197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.161+0000 7ffb2d200700 1 --2- 172.21.15.78:0/311079999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffb281028e0 0x7ffb2819e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.161+0000 7ffb2d200700 1 --2- 172.21.15.78:0/311079999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb28101f40 0x7ffb28197660 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.161+0000 7ffb2d200700 1 -- 172.21.15.78:0/311079999 >> 172.21.15.78:0/311079999 conn(0x7ffb280fc2c0 msgr2=0x7ffb281052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:55.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.161+0000 7ffb2d200700 1 -- 172.21.15.78:0/311079999 shutdown_connections 2024-03-14T09:53:55.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.161+0000 7ffb2d200700 1 -- 172.21.15.78:0/311079999 wait complete. 2024-03-14T09:53:55.181 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-14T09:53:55.181 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:53:55.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.373+0000 7fe396400700 1 -- 172.21.15.78:0/712995170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe390101e40 msgr2=0x7fe3901022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:55.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.373+0000 7fe396400700 1 --2- 172.21.15.78:0/712995170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe390101e40 0x7fe3901022a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0099c0 tx=0x7fe37c009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.374+0000 7fe396400700 1 -- 172.21.15.78:0/712995170 shutdown_connections 2024-03-14T09:53:55.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.374+0000 7fe396400700 1 --2- 172.21.15.78:0/712995170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe390100c50 0x7fe390101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.374+0000 7fe396400700 1 --2- 172.21.15.78:0/712995170 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe3901027e0 0x7fe390109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.374+0000 7fe396400700 1 --2- 172.21.15.78:0/712995170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe390101e40 0x7fe3901022a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.374+0000 7fe396400700 1 -- 172.21.15.78:0/712995170 >> 172.21.15.78:0/712995170 conn(0x7fe3900fc1c0 msgr2=0x7fe3900fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:55.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.374+0000 7fe396400700 1 -- 172.21.15.78:0/712995170 shutdown_connections 2024-03-14T09:53:55.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.374+0000 7fe396400700 1 -- 172.21.15.78:0/712995170 wait complete. 2024-03-14T09:53:55.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.375+0000 7fe396400700 1 Processor -- start 2024-03-14T09:53:55.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.375+0000 7fe396400700 1 -- start start 2024-03-14T09:53:55.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.375+0000 7fe396400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe390100c50 0x7fe390199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:55.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.375+0000 7fe396400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe390101e40 0x7fe3901997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:55.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.375+0000 7fe396400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3901027e0 0x7fe3901a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:55.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.375+0000 7fe396400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3901a0d50 con 0x7fe390101e40 2024-03-14T09:53:55.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.375+0000 7fe396400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3901a0e90 con 0x7fe3901027e0 2024-03-14T09:53:55.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.375+0000 7fe396400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3901a1020 con 0x7fe390100c50 2024-03-14T09:53:55.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.375+0000 7fe38ea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe390101e40 0x7fe3901997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:55.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.375+0000 7fe38f400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe390100c50 0x7fe390199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:55.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.375+0000 7fe38ea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe390101e40 0x7fe3901997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43878/0 (socket says 172.21.15.78:43878) 2024-03-14T09:53:55.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.375+0000 7fe38ea00700 1 -- 172.21.15.78:0/3522177793 learned_addr learned my addr 172.21.15.78:0/3522177793 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:55.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.376+0000 7fe38fe00700 1 --2- 172.21.15.78:0/3522177793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3901027e0 0x7fe3901a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:55.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.376+0000 7fe38ea00700 1 -- 172.21.15.78:0/3522177793 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe390100c50 msgr2=0x7fe390199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:55.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.376+0000 7fe38ea00700 1 --2- 172.21.15.78:0/3522177793 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe390100c50 0x7fe390199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.376+0000 7fe38ea00700 1 -- 172.21.15.78:0/3522177793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3901027e0 msgr2=0x7fe3901a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:55.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.376+0000 7fe38ea00700 1 --2- 172.21.15.78:0/3522177793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3901027e0 0x7fe3901a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.376+0000 7fe38ea00700 1 -- 172.21.15.78:0/3522177793 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe37c0095f0 con 0x7fe390101e40 2024-03-14T09:53:55.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.376+0000 7fe38fe00700 1 --2- 172.21.15.78:0/3522177793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3901027e0 0x7fe3901a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:55.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.376+0000 7fe38ea00700 1 --2- 172.21.15.78:0/3522177793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe390101e40 0x7fe3901997b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe37c006010 tx=0x7fe37c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:55.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.376+0000 7fe387e00700 1 -- 172.21.15.78:0/3522177793 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c01d070 con 0x7fe390101e40 2024-03-14T09:53:55.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.376+0000 7fe387e00700 1 -- 172.21.15.78:0/3522177793 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe37c005030 con 0x7fe390101e40 2024-03-14T09:53:55.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.376+0000 7fe396400700 1 -- 172.21.15.78:0/3522177793 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3901a12a0 con 0x7fe390101e40 2024-03-14T09:53:55.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.376+0000 7fe387e00700 1 -- 172.21.15.78:0/3522177793 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c004b10 con 0x7fe390101e40 2024-03-14T09:53:55.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.377+0000 7fe396400700 1 -- 172.21.15.78:0/3522177793 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3901a16c0 con 0x7fe390101e40 2024-03-14T09:53:55.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.378+0000 7fe396400700 1 -- 172.21.15.78:0/3522177793 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3900679e0 con 0x7fe390101e40 2024-03-14T09:53:55.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.379+0000 7fe387e00700 1 -- 172.21.15.78:0/3522177793 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fe37c0051a0 con 0x7fe390101e40 2024-03-14T09:53:55.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.381+0000 7fe387e00700 1 --2- 172.21.15.78:0/3522177793 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe3740a0b20 0x7fe3740a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:55.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.381+0000 7fe387e00700 1 -- 172.21.15.78:0/3522177793 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7fe37c0cd1f0 con 0x7fe390101e40 2024-03-14T09:53:55.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.381+0000 7fe38f400700 1 --2- 172.21.15.78:0/3522177793 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe3740a0b20 0x7fe3740a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:55.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.382+0000 7fe38f400700 1 --2- 172.21.15.78:0/3522177793 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe3740a0b20 0x7fe3740a2fe0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe390101ca0 tx=0x7fe380009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:55.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.385+0000 7fe387e00700 1 -- 172.21.15.78:0/3522177793 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe37c096780 con 0x7fe390101e40 2024-03-14T09:53:55.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:55 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2603148342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:55.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:55 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1111894409' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:55.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:55 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/311079999' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:55 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2603148342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:55 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1111894409' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:55 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/311079999' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:55.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:55 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2603148342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:55.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:55 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1111894409' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:55.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:55 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/311079999' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:53:55.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.635+0000 7fe396400700 1 -- 172.21.15.78:0/3522177793 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe39010cad0 con 0x7fe390101e40 2024-03-14T09:53:55.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.636+0000 7fe387e00700 1 -- 172.21.15.78:0/3522177793 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7fe37c099da0 con 0x7fe390101e40 2024-03-14T09:53:55.636 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:55.636 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:53:55.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.638+0000 7fe396400700 1 -- 172.21.15.78:0/3522177793 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe3740a0b20 msgr2=0x7fe3740a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:55.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.638+0000 7fe396400700 1 --2- 172.21.15.78:0/3522177793 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe3740a0b20 0x7fe3740a2fe0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe390101ca0 tx=0x7fe380009260 comp rx=0 tx=0).stop 2024-03-14T09:53:55.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.638+0000 7fe396400700 1 -- 172.21.15.78:0/3522177793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe390101e40 msgr2=0x7fe3901997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:55.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.638+0000 7fe396400700 1 --2- 172.21.15.78:0/3522177793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe390101e40 0x7fe3901997b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe37c006010 tx=0x7fe37c00f690 comp rx=0 tx=0).stop 2024-03-14T09:53:55.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.638+0000 7fe396400700 1 -- 172.21.15.78:0/3522177793 shutdown_connections 2024-03-14T09:53:55.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.638+0000 7fe396400700 1 --2- 172.21.15.78:0/3522177793 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe3740a0b20 0x7fe3740a2fe0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.638+0000 7fe396400700 1 --2- 172.21.15.78:0/3522177793 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3901027e0 0x7fe3901a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.638+0000 7fe396400700 1 --2- 172.21.15.78:0/3522177793 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe390100c50 0x7fe390199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.638+0000 7fe396400700 1 --2- 172.21.15.78:0/3522177793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe390101e40 0x7fe3901997b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.638+0000 7fe396400700 1 -- 172.21.15.78:0/3522177793 >> 172.21.15.78:0/3522177793 conn(0x7fe3900fc1c0 msgr2=0x7fe390072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:55.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.638+0000 7fe396400700 1 -- 172.21.15.78:0/3522177793 shutdown_connections 2024-03-14T09:53:55.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.638+0000 7fe396400700 1 -- 172.21.15.78:0/3522177793 wait complete. 2024-03-14T09:53:55.640 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T09:53:55.657 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-14T09:53:55.657 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-14T09:53:55.657 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-14T09:53:55.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.850+0000 7f8941e00700 1 -- 172.21.15.78:0/2496730757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f893c1027e0 msgr2=0x7f893c109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:55.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.850+0000 7f8941e00700 1 --2- 172.21.15.78:0/2496730757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f893c1027e0 0x7f893c109090 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f893400b0d0 tx=0x7f893400b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.850+0000 7f8941e00700 1 -- 172.21.15.78:0/2496730757 shutdown_connections 2024-03-14T09:53:55.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.850+0000 7f8941e00700 1 --2- 172.21.15.78:0/2496730757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f893c1027e0 0x7f893c109090 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.850+0000 7f8941e00700 1 --2- 172.21.15.78:0/2496730757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f893c100c50 0x7f893c101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.850+0000 7f8941e00700 1 --2- 172.21.15.78:0/2496730757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c101e40 0x7f893c1022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.850+0000 7f8941e00700 1 -- 172.21.15.78:0/2496730757 >> 172.21.15.78:0/2496730757 conn(0x7f893c0fc1c0 msgr2=0x7f893c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:55.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.850+0000 7f8941e00700 1 -- 172.21.15.78:0/2496730757 shutdown_connections 2024-03-14T09:53:55.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.850+0000 7f8941e00700 1 -- 172.21.15.78:0/2496730757 wait complete. 2024-03-14T09:53:55.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.851+0000 7f8941e00700 1 Processor -- start 2024-03-14T09:53:55.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.851+0000 7f8941e00700 1 -- start start 2024-03-14T09:53:55.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.851+0000 7f8941e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c100c50 0x7f893c199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:55.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.851+0000 7f8941e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f893c101e40 0x7f893c199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:55.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.851+0000 7f8941e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f893c1027e0 0x7f893c1a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:55.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.851+0000 7f8941e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c1a0e70 con 0x7f893c101e40 2024-03-14T09:53:55.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.851+0000 7f8941e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c1a0fb0 con 0x7f893c100c50 2024-03-14T09:53:55.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.851+0000 7f8941e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c1a1140 con 0x7f893c1027e0 2024-03-14T09:53:55.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.851+0000 7f893aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c100c50 0x7f893c199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:55.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.852+0000 7f893aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c100c50 0x7f893c199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:44142/0 (socket says 172.21.15.78:44142) 2024-03-14T09:53:55.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.852+0000 7f893aa00700 1 -- 172.21.15.78:0/3839694234 learned_addr learned my addr 172.21.15.78:0/3839694234 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:55.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.852+0000 7f893a000700 1 --2- 172.21.15.78:0/3839694234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f893c101e40 0x7f893c199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:55.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.852+0000 7f893b400700 1 --2- 172.21.15.78:0/3839694234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f893c1027e0 0x7f893c1a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:55.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.852+0000 7f893aa00700 1 -- 172.21.15.78:0/3839694234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f893c1027e0 msgr2=0x7f893c1a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:55.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.852+0000 7f893aa00700 1 --2- 172.21.15.78:0/3839694234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f893c1027e0 0x7f893c1a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.852+0000 7f893aa00700 1 -- 172.21.15.78:0/3839694234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f893c101e40 msgr2=0x7f893c199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:55.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.852+0000 7f893aa00700 1 --2- 172.21.15.78:0/3839694234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f893c101e40 0x7f893c199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:55.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.852+0000 7f893aa00700 1 -- 172.21.15.78:0/3839694234 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8934009d00 con 0x7f893c100c50 2024-03-14T09:53:55.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.852+0000 7f893a000700 1 --2- 172.21.15.78:0/3839694234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f893c101e40 0x7f893c199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:55.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.852+0000 7f893aa00700 1 --2- 172.21.15.78:0/3839694234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c100c50 0x7f893c199340 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f892c00e970 tx=0x7f892c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:55.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.853+0000 7f8940a00700 1 -- 172.21.15.78:0/3839694234 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c009790 con 0x7f893c100c50 2024-03-14T09:53:55.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.853+0000 7f8940a00700 1 -- 172.21.15.78:0/3839694234 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f892c00eec0 con 0x7f893c100c50 2024-03-14T09:53:55.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.853+0000 7f8941e00700 1 -- 172.21.15.78:0/3839694234 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893c1a1420 con 0x7f893c100c50 2024-03-14T09:53:55.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.853+0000 7f8940a00700 1 -- 172.21.15.78:0/3839694234 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c003cf0 con 0x7f893c100c50 2024-03-14T09:53:55.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.853+0000 7f8941e00700 1 -- 172.21.15.78:0/3839694234 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893c1a1980 con 0x7f893c100c50 2024-03-14T09:53:55.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.854+0000 7f8941e00700 1 -- 172.21.15.78:0/3839694234 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893c0679e0 con 0x7f893c100c50 2024-03-14T09:53:55.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.856+0000 7f8940a00700 1 -- 172.21.15.78:0/3839694234 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f892c00ccd0 con 0x7f893c100c50 2024-03-14T09:53:55.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.857+0000 7f8940a00700 1 --2- 172.21.15.78:0/3839694234 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f89200a0ba0 0x7f89200a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:55.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.857+0000 7f8940a00700 1 -- 172.21.15.78:0/3839694234 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f892c014070 con 0x7f893c100c50 2024-03-14T09:53:55.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.857+0000 7f893a000700 1 --2- 172.21.15.78:0/3839694234 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f89200a0ba0 0x7f89200a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:55.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.858+0000 7f893a000700 1 --2- 172.21.15.78:0/3839694234 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f89200a0ba0 0x7f89200a3060 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8928011490 tx=0x7f8928009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:55.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:55.860+0000 7f8940a00700 1 -- 172.21.15.78:0/3839694234 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f892c095d70 con 0x7f893c100c50 2024-03-14T09:53:56.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.061+0000 7f8941e00700 1 -- 172.21.15.78:0/3839694234 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f893c074700 con 0x7f89200a0ba0 2024-03-14T09:53:56.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.075+0000 7f8940a00700 1 -- 172.21.15.78:0/3839694234 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427290 (secure 0 0 0) 0x7f893c074700 con 0x7f89200a0ba0 2024-03-14T09:53:56.083 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:56.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.081+0000 7f8941e00700 1 -- 172.21.15.78:0/3839694234 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f89200a0ba0 msgr2=0x7f89200a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:56.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.081+0000 7f8941e00700 1 --2- 172.21.15.78:0/3839694234 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f89200a0ba0 0x7f89200a3060 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8928011490 tx=0x7f8928009260 comp rx=0 tx=0).stop 2024-03-14T09:53:56.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.081+0000 7f8941e00700 1 -- 172.21.15.78:0/3839694234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c100c50 msgr2=0x7f893c199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:56.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.081+0000 7f8941e00700 1 --2- 172.21.15.78:0/3839694234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c100c50 0x7f893c199340 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f892c00e970 tx=0x7f892c00ec80 comp rx=0 tx=0).stop 2024-03-14T09:53:56.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.081+0000 7f8941e00700 1 -- 172.21.15.78:0/3839694234 shutdown_connections 2024-03-14T09:53:56.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.081+0000 7f8941e00700 1 --2- 172.21.15.78:0/3839694234 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f89200a0ba0 0x7f89200a3060 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.081+0000 7f8941e00700 1 --2- 172.21.15.78:0/3839694234 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c100c50 0x7f893c199340 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.081+0000 7f8941e00700 1 --2- 172.21.15.78:0/3839694234 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f893c1027e0 0x7f893c1a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.081+0000 7f8941e00700 1 --2- 172.21.15.78:0/3839694234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f893c101e40 0x7f893c199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.081+0000 7f8941e00700 1 -- 172.21.15.78:0/3839694234 >> 172.21.15.78:0/3839694234 conn(0x7f893c0fc1c0 msgr2=0x7f893c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:56.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.081+0000 7f8941e00700 1 -- 172.21.15.78:0/3839694234 shutdown_connections 2024-03-14T09:53:56.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.081+0000 7f8941e00700 1 -- 172.21.15.78:0/3839694234 wait complete. 2024-03-14T09:53:56.088 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-14T09:53:56.101 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-03-14T09:53:55.097758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661953,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":663,"num_read_kb":462,"num_write":351,"num_write_kb":3067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1181430,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":786,"ondisk_log_size":786,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":335892,"kb_used_data":11680,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124605932,"statfs":{"total":1151940427776,"available":1151596474368,"internally_reserved":0,"allocated":11960320,"data_stored":7828646,"data_compressed":395397,"data_compressed_allocated":540672,"data_compressed_original":2580480,"omap_allocated":19131,"internal_metadata":331855173},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5716,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":85,"num_read_kb":54,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003232"},"pg_stats":[{"pgid":"3.3f","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899578+0000","last_change":"2024-03-14T09:52:05.737617+0000","last_active":"2024-03-14T09:53:41.899578+0000","last_peered":"2024-03-14T09:53:41.899578+0000","last_clean":"2024-03-14T09:53:41.899578+0000","last_became_active":"2024-03-14T09:52:05.736796+0000","last_became_peered":"2024-03-14T09:52:05.736796+0000","last_unstale":"2024-03-14T09:53:41.899578+0000","last_undegraded":"2024-03-14T09:53:41.899578+0000","last_fullsized":"2024-03-14T09:53:41.899578+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:47:52.437748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"85'4","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899167+0000","last_change":"2024-03-14T09:52:05.749474+0000","last_active":"2024-03-14T09:53:41.899167+0000","last_peered":"2024-03-14T09:53:41.899167+0000","last_clean":"2024-03-14T09:53:41.899167+0000","last_became_active":"2024-03-14T09:52:05.748927+0000","last_became_peered":"2024-03-14T09:52:05.748927+0000","last_unstale":"2024-03-14T09:53:41.899167+0000","last_undegraded":"2024-03-14T09:53:41.899167+0000","last_fullsized":"2024-03-14T09:53:41.899167+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:28:15.852021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899117+0000","last_change":"2024-03-14T09:52:05.745098+0000","last_active":"2024-03-14T09:53:41.899117+0000","last_peered":"2024-03-14T09:53:41.899117+0000","last_clean":"2024-03-14T09:53:41.899117+0000","last_became_active":"2024-03-14T09:52:05.744755+0000","last_became_peered":"2024-03-14T09:52:05.744755+0000","last_unstale":"2024-03-14T09:53:41.899117+0000","last_undegraded":"2024-03-14T09:53:41.899117+0000","last_fullsized":"2024-03-14T09:53:41.899117+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:35:41.932386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786320+0000","last_change":"2024-03-14T09:52:05.737429+0000","last_active":"2024-03-14T09:53:41.786320+0000","last_peered":"2024-03-14T09:53:41.786320+0000","last_clean":"2024-03-14T09:53:41.786320+0000","last_became_active":"2024-03-14T09:52:05.736516+0000","last_became_peered":"2024-03-14T09:52:05.736516+0000","last_unstale":"2024-03-14T09:53:41.786320+0000","last_undegraded":"2024-03-14T09:53:41.786320+0000","last_fullsized":"2024-03-14T09:53:41.786320+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:48:38.158783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"85'4","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788000+0000","last_change":"2024-03-14T09:52:05.729634+0000","last_active":"2024-03-14T09:53:41.788000+0000","last_peered":"2024-03-14T09:53:41.788000+0000","last_clean":"2024-03-14T09:53:41.788000+0000","last_became_active":"2024-03-14T09:52:05.727889+0000","last_became_peered":"2024-03-14T09:52:05.727889+0000","last_unstale":"2024-03-14T09:53:41.788000+0000","last_undegraded":"2024-03-14T09:53:41.788000+0000","last_fullsized":"2024-03-14T09:53:41.788000+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:01:13.216763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"85'6","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787456+0000","last_change":"2024-03-14T09:52:42.147610+0000","last_active":"2024-03-14T09:53:41.787456+0000","last_peered":"2024-03-14T09:53:41.787456+0000","last_clean":"2024-03-14T09:53:41.787456+0000","last_became_active":"2024-03-14T09:52:42.147262+0000","last_became_peered":"2024-03-14T09:52:42.147262+0000","last_unstale":"2024-03-14T09:53:41.787456+0000","last_undegraded":"2024-03-14T09:53:41.787456+0000","last_fullsized":"2024-03-14T09:53:41.787456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:20:49.855564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783351+0000","last_change":"2024-03-14T09:52:05.727365+0000","last_active":"2024-03-14T09:53:41.783351+0000","last_peered":"2024-03-14T09:53:41.783351+0000","last_clean":"2024-03-14T09:53:41.783351+0000","last_became_active":"2024-03-14T09:52:05.726984+0000","last_became_peered":"2024-03-14T09:52:05.726984+0000","last_unstale":"2024-03-14T09:53:41.783351+0000","last_undegraded":"2024-03-14T09:53:41.783351+0000","last_fullsized":"2024-03-14T09:53:41.783351+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:42:57.129718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785018+0000","last_change":"2024-03-14T09:52:05.729622+0000","last_active":"2024-03-14T09:53:41.785018+0000","last_peered":"2024-03-14T09:53:41.785018+0000","last_clean":"2024-03-14T09:53:41.785018+0000","last_became_active":"2024-03-14T09:52:05.729293+0000","last_became_peered":"2024-03-14T09:52:05.729293+0000","last_unstale":"2024-03-14T09:53:41.785018+0000","last_undegraded":"2024-03-14T09:53:41.785018+0000","last_fullsized":"2024-03-14T09:53:41.785018+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:13:20.875857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"85'11","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790229+0000","last_change":"2024-03-14T09:52:05.741620+0000","last_active":"2024-03-14T09:53:41.790229+0000","last_peered":"2024-03-14T09:53:41.790229+0000","last_clean":"2024-03-14T09:53:41.790229+0000","last_became_active":"2024-03-14T09:52:05.734079+0000","last_became_peered":"2024-03-14T09:52:05.734079+0000","last_unstale":"2024-03-14T09:53:41.790229+0000","last_undegraded":"2024-03-14T09:53:41.790229+0000","last_fullsized":"2024-03-14T09:53:41.790229+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:02:35.529671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"85'16","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788064+0000","last_change":"2024-03-14T09:52:05.743424+0000","last_active":"2024-03-14T09:53:41.788064+0000","last_peered":"2024-03-14T09:53:41.788064+0000","last_clean":"2024-03-14T09:53:41.788064+0000","last_became_active":"2024-03-14T09:52:05.743100+0000","last_became_peered":"2024-03-14T09:52:05.743100+0000","last_unstale":"2024-03-14T09:53:41.788064+0000","last_undegraded":"2024-03-14T09:53:41.788064+0000","last_fullsized":"2024-03-14T09:53:41.788064+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:32:33.492163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783875+0000","last_change":"2024-03-14T09:52:05.735654+0000","last_active":"2024-03-14T09:53:41.783875+0000","last_peered":"2024-03-14T09:53:41.783875+0000","last_clean":"2024-03-14T09:53:41.783875+0000","last_became_active":"2024-03-14T09:52:05.735371+0000","last_became_peered":"2024-03-14T09:52:05.735371+0000","last_unstale":"2024-03-14T09:53:41.783875+0000","last_undegraded":"2024-03-14T09:53:41.783875+0000","last_fullsized":"2024-03-14T09:53:41.783875+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:19:50.011554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"85'7","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789097+0000","last_change":"2024-03-14T09:52:42.146700+0000","last_active":"2024-03-14T09:53:41.789097+0000","last_peered":"2024-03-14T09:53:41.789097+0000","last_clean":"2024-03-14T09:53:41.789097+0000","last_became_active":"2024-03-14T09:52:42.145377+0000","last_became_peered":"2024-03-14T09:52:42.145377+0000","last_unstale":"2024-03-14T09:53:41.789097+0000","last_undegraded":"2024-03-14T09:53:41.789097+0000","last_fullsized":"2024-03-14T09:53:41.789097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:35:52.735091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"85'10","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790265+0000","last_change":"2024-03-14T09:52:05.740209+0000","last_active":"2024-03-14T09:53:41.790265+0000","last_peered":"2024-03-14T09:53:41.790265+0000","last_clean":"2024-03-14T09:53:41.790265+0000","last_became_active":"2024-03-14T09:52:05.732949+0000","last_became_peered":"2024-03-14T09:52:05.732949+0000","last_unstale":"2024-03-14T09:53:41.790265+0000","last_undegraded":"2024-03-14T09:53:41.790265+0000","last_fullsized":"2024-03-14T09:53:41.790265+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:15:56.377569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.898939+0000","last_change":"2024-03-14T09:52:05.748795+0000","last_active":"2024-03-14T09:53:41.898939+0000","last_peered":"2024-03-14T09:53:41.898939+0000","last_clean":"2024-03-14T09:53:41.898939+0000","last_became_active":"2024-03-14T09:52:05.746107+0000","last_became_peered":"2024-03-14T09:52:05.746107+0000","last_unstale":"2024-03-14T09:53:41.898939+0000","last_undegraded":"2024-03-14T09:53:41.898939+0000","last_fullsized":"2024-03-14T09:53:41.898939+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:49:07.932854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.898932+0000","last_change":"2024-03-14T09:52:05.744174+0000","last_active":"2024-03-14T09:53:41.898932+0000","last_peered":"2024-03-14T09:53:41.898932+0000","last_clean":"2024-03-14T09:53:41.898932+0000","last_became_active":"2024-03-14T09:52:05.743848+0000","last_became_peered":"2024-03-14T09:52:05.743848+0000","last_unstale":"2024-03-14T09:53:41.898932+0000","last_undegraded":"2024-03-14T09:53:41.898932+0000","last_fullsized":"2024-03-14T09:53:41.898932+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:07:23.260734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789366+0000","last_change":"2024-03-14T09:52:05.733677+0000","last_active":"2024-03-14T09:53:41.789366+0000","last_peered":"2024-03-14T09:53:41.789366+0000","last_clean":"2024-03-14T09:53:41.789366+0000","last_became_active":"2024-03-14T09:52:05.730967+0000","last_became_peered":"2024-03-14T09:52:05.730967+0000","last_unstale":"2024-03-14T09:53:41.789366+0000","last_undegraded":"2024-03-14T09:53:41.789366+0000","last_fullsized":"2024-03-14T09:53:41.789366+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:33:53.484741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899444+0000","last_change":"2024-03-14T09:52:05.737836+0000","last_active":"2024-03-14T09:53:41.899444+0000","last_peered":"2024-03-14T09:53:41.899444+0000","last_clean":"2024-03-14T09:53:41.899444+0000","last_became_active":"2024-03-14T09:52:05.737257+0000","last_became_peered":"2024-03-14T09:52:05.737257+0000","last_unstale":"2024-03-14T09:53:41.899444+0000","last_undegraded":"2024-03-14T09:53:41.899444+0000","last_fullsized":"2024-03-14T09:53:41.899444+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:53.723415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"85'10","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789627+0000","last_change":"2024-03-14T09:52:05.741360+0000","last_active":"2024-03-14T09:53:41.789627+0000","last_peered":"2024-03-14T09:53:41.789627+0000","last_clean":"2024-03-14T09:53:41.789627+0000","last_became_active":"2024-03-14T09:52:05.740673+0000","last_became_peered":"2024-03-14T09:52:05.740673+0000","last_unstale":"2024-03-14T09:53:41.789627+0000","last_undegraded":"2024-03-14T09:53:41.789627+0000","last_fullsized":"2024-03-14T09:53:41.789627+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:13:41.561338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.784072+0000","last_change":"2024-03-14T09:52:05.739134+0000","last_active":"2024-03-14T09:53:41.784072+0000","last_peered":"2024-03-14T09:53:41.784072+0000","last_clean":"2024-03-14T09:53:41.784072+0000","last_became_active":"2024-03-14T09:52:05.738439+0000","last_became_peered":"2024-03-14T09:52:05.738439+0000","last_unstale":"2024-03-14T09:53:41.784072+0000","last_undegraded":"2024-03-14T09:53:41.784072+0000","last_fullsized":"2024-03-14T09:53:41.784072+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:05:36.997462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783550+0000","last_change":"2024-03-14T09:52:09.769284+0000","last_active":"2024-03-14T09:53:41.783550+0000","last_peered":"2024-03-14T09:53:41.783550+0000","last_clean":"2024-03-14T09:53:41.783550+0000","last_became_active":"2024-03-14T09:52:09.768808+0000","last_became_peered":"2024-03-14T09:52:09.768808+0000","last_unstale":"2024-03-14T09:53:41.783550+0000","last_undegraded":"2024-03-14T09:53:41.783550+0000","last_fullsized":"2024-03-14T09:53:41.783550+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:48:56.015410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"83'9","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789540+0000","last_change":"2024-03-14T09:52:42.154303+0000","last_active":"2024-03-14T09:53:41.789540+0000","last_peered":"2024-03-14T09:53:41.789540+0000","last_clean":"2024-03-14T09:53:41.789540+0000","last_became_active":"2024-03-14T09:52:42.147756+0000","last_became_peered":"2024-03-14T09:52:42.147756+0000","last_unstale":"2024-03-14T09:53:41.789540+0000","last_undegraded":"2024-03-14T09:53:41.789540+0000","last_fullsized":"2024-03-14T09:53:41.789540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:01:27.630337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785328+0000","last_change":"2024-03-14T09:52:05.739004+0000","last_active":"2024-03-14T09:53:41.785328+0000","last_peered":"2024-03-14T09:53:41.785328+0000","last_clean":"2024-03-14T09:53:41.785328+0000","last_became_active":"2024-03-14T09:52:05.732217+0000","last_became_peered":"2024-03-14T09:52:05.732217+0000","last_unstale":"2024-03-14T09:53:41.785328+0000","last_undegraded":"2024-03-14T09:53:41.785328+0000","last_fullsized":"2024-03-14T09:53:41.785328+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:02:21.926623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785929+0000","last_change":"2024-03-14T09:52:09.776128+0000","last_active":"2024-03-14T09:53:41.785929+0000","last_peered":"2024-03-14T09:53:41.785929+0000","last_clean":"2024-03-14T09:53:41.785929+0000","last_became_active":"2024-03-14T09:52:09.775701+0000","last_became_peered":"2024-03-14T09:52:09.775701+0000","last_unstale":"2024-03-14T09:53:41.785929+0000","last_undegraded":"2024-03-14T09:53:41.785929+0000","last_fullsized":"2024-03-14T09:53:41.785929+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:41:51.790627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"83'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785672+0000","last_change":"2024-03-14T09:52:05.234269+0000","last_active":"2024-03-14T09:53:41.785672+0000","last_peered":"2024-03-14T09:53:41.785672+0000","last_clean":"2024-03-14T09:53:41.785672+0000","last_became_active":"2024-03-14T09:52:05.233418+0000","last_became_peered":"2024-03-14T09:52:05.233418+0000","last_unstale":"2024-03-14T09:53:41.785672+0000","last_undegraded":"2024-03-14T09:53:41.785672+0000","last_fullsized":"2024-03-14T09:53:41.785672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:22:46.894461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899167+0000","last_change":"2024-03-14T09:52:05.741786+0000","last_active":"2024-03-14T09:53:41.899167+0000","last_peered":"2024-03-14T09:53:41.899167+0000","last_clean":"2024-03-14T09:53:41.899167+0000","last_became_active":"2024-03-14T09:52:05.741461+0000","last_became_peered":"2024-03-14T09:52:05.741461+0000","last_unstale":"2024-03-14T09:53:41.899167+0000","last_undegraded":"2024-03-14T09:53:41.899167+0000","last_fullsized":"2024-03-14T09:53:41.899167+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:21:17.409530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786363+0000","last_change":"2024-03-14T09:52:09.768851+0000","last_active":"2024-03-14T09:53:41.786363+0000","last_peered":"2024-03-14T09:53:41.786363+0000","last_clean":"2024-03-14T09:53:41.786363+0000","last_became_active":"2024-03-14T09:52:09.768485+0000","last_became_peered":"2024-03-14T09:52:09.768485+0000","last_unstale":"2024-03-14T09:53:41.786363+0000","last_undegraded":"2024-03-14T09:53:41.786363+0000","last_fullsized":"2024-03-14T09:53:41.786363+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:00:51.189790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"83'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899465+0000","last_change":"2024-03-14T09:52:42.154227+0000","last_active":"2024-03-14T09:53:41.899465+0000","last_peered":"2024-03-14T09:53:41.899465+0000","last_clean":"2024-03-14T09:53:41.899465+0000","last_became_active":"2024-03-14T09:52:42.149841+0000","last_became_peered":"2024-03-14T09:52:42.149841+0000","last_unstale":"2024-03-14T09:53:41.899465+0000","last_undegraded":"2024-03-14T09:53:41.899465+0000","last_fullsized":"2024-03-14T09:53:41.899465+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:04:46.911921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789719+0000","last_change":"2024-03-14T09:52:05.742994+0000","last_active":"2024-03-14T09:53:41.789719+0000","last_peered":"2024-03-14T09:53:41.789719+0000","last_clean":"2024-03-14T09:53:41.789719+0000","last_became_active":"2024-03-14T09:52:05.742652+0000","last_became_peered":"2024-03-14T09:52:05.742652+0000","last_unstale":"2024-03-14T09:53:41.789719+0000","last_undegraded":"2024-03-14T09:53:41.789719+0000","last_fullsized":"2024-03-14T09:53:41.789719+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:46:47.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785978+0000","last_change":"2024-03-14T09:52:09.763009+0000","last_active":"2024-03-14T09:53:41.785978+0000","last_peered":"2024-03-14T09:53:41.785978+0000","last_clean":"2024-03-14T09:53:41.785978+0000","last_became_active":"2024-03-14T09:52:09.762621+0000","last_became_peered":"2024-03-14T09:52:09.762621+0000","last_unstale":"2024-03-14T09:53:41.785978+0000","last_undegraded":"2024-03-14T09:53:41.785978+0000","last_fullsized":"2024-03-14T09:53:41.785978+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:56:22.162409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788297+0000","last_change":"2024-03-14T09:52:05.234039+0000","last_active":"2024-03-14T09:53:41.788297+0000","last_peered":"2024-03-14T09:53:41.788297+0000","last_clean":"2024-03-14T09:53:41.788297+0000","last_became_active":"2024-03-14T09:52:05.233612+0000","last_became_peered":"2024-03-14T09:52:05.233612+0000","last_unstale":"2024-03-14T09:53:41.788297+0000","last_undegraded":"2024-03-14T09:53:41.788297+0000","last_fullsized":"2024-03-14T09:53:41.788297+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:29:52.968178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"85'15","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787964+0000","last_change":"2024-03-14T09:52:42.426506+0000","last_active":"2024-03-14T09:53:41.787964+0000","last_peered":"2024-03-14T09:53:41.787964+0000","last_clean":"2024-03-14T09:53:41.787964+0000","last_became_active":"2024-03-14T09:52:42.426062+0000","last_became_peered":"2024-03-14T09:52:42.426062+0000","last_unstale":"2024-03-14T09:53:41.787964+0000","last_undegraded":"2024-03-14T09:53:41.787964+0000","last_fullsized":"2024-03-14T09:53:41.787964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:23:42.943381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783824+0000","last_change":"2024-03-14T09:52:09.757856+0000","last_active":"2024-03-14T09:53:41.783824+0000","last_peered":"2024-03-14T09:53:41.783824+0000","last_clean":"2024-03-14T09:53:41.783824+0000","last_became_active":"2024-03-14T09:52:09.757395+0000","last_became_peered":"2024-03-14T09:52:09.757395+0000","last_unstale":"2024-03-14T09:53:41.783824+0000","last_undegraded":"2024-03-14T09:53:41.783824+0000","last_fullsized":"2024-03-14T09:53:41.783824+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:34:06.600727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"83'5","reported_seq":109,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:50.849824+0000","last_change":"2024-03-14T09:52:05.228527+0000","last_active":"2024-03-14T09:53:50.849824+0000","last_peered":"2024-03-14T09:53:50.849824+0000","last_clean":"2024-03-14T09:53:50.849824+0000","last_became_active":"2024-03-14T09:52:05.228177+0000","last_became_peered":"2024-03-14T09:52:05.228177+0000","last_unstale":"2024-03-14T09:53:50.849824+0000","last_undegraded":"2024-03-14T09:53:50.849824+0000","last_fullsized":"2024-03-14T09:53:50.849824+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:17:57.529308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789022+0000","last_change":"2024-03-14T09:52:05.734041+0000","last_active":"2024-03-14T09:53:41.789022+0000","last_peered":"2024-03-14T09:53:41.789022+0000","last_clean":"2024-03-14T09:53:41.789022+0000","last_became_active":"2024-03-14T09:52:05.732491+0000","last_became_peered":"2024-03-14T09:52:05.732491+0000","last_unstale":"2024-03-14T09:53:41.789022+0000","last_undegraded":"2024-03-14T09:53:41.789022+0000","last_fullsized":"2024-03-14T09:53:41.789022+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:07:31.909382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:43.640334+0000","last_change":"2024-03-14T09:53:43.640334+0000","last_active":"2024-03-14T09:53:43.640334+0000","last_peered":"2024-03-14T09:53:43.640334+0000","last_clean":"2024-03-14T09:53:43.640334+0000","last_became_active":"2024-03-14T09:53:43.639821+0000","last_became_peered":"2024-03-14T09:53:43.639821+0000","last_unstale":"2024-03-14T09:53:43.640334+0000","last_undegraded":"2024-03-14T09:53:43.640334+0000","last_fullsized":"2024-03-14T09:53:43.640334+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:23:30.760149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,10],"acting":[2,6,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"83'3","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787260+0000","last_change":"2024-03-14T09:52:05.228893+0000","last_active":"2024-03-14T09:53:41.787260+0000","last_peered":"2024-03-14T09:53:41.787260+0000","last_clean":"2024-03-14T09:53:41.787260+0000","last_became_active":"2024-03-14T09:52:05.228569+0000","last_became_peered":"2024-03-14T09:52:05.228569+0000","last_unstale":"2024-03-14T09:53:41.787260+0000","last_undegraded":"2024-03-14T09:53:41.787260+0000","last_fullsized":"2024-03-14T09:53:41.787260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:28:03.200437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788237+0000","last_change":"2024-03-14T09:52:05.736388+0000","last_active":"2024-03-14T09:53:41.788237+0000","last_peered":"2024-03-14T09:53:41.788237+0000","last_clean":"2024-03-14T09:53:41.788237+0000","last_became_active":"2024-03-14T09:52:05.735131+0000","last_became_peered":"2024-03-14T09:52:05.735131+0000","last_unstale":"2024-03-14T09:53:41.788237+0000","last_undegraded":"2024-03-14T09:53:41.788237+0000","last_fullsized":"2024-03-14T09:53:41.788237+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:00:54.120087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899317+0000","last_change":"2024-03-14T09:52:09.770321+0000","last_active":"2024-03-14T09:53:41.899317+0000","last_peered":"2024-03-14T09:53:41.899317+0000","last_clean":"2024-03-14T09:53:41.899317+0000","last_became_active":"2024-03-14T09:52:09.769710+0000","last_became_peered":"2024-03-14T09:52:09.769710+0000","last_unstale":"2024-03-14T09:53:41.899317+0000","last_undegraded":"2024-03-14T09:53:41.899317+0000","last_fullsized":"2024-03-14T09:53:41.899317+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:12:12.635248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"83'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789908+0000","last_change":"2024-03-14T09:52:05.223912+0000","last_active":"2024-03-14T09:53:41.789908+0000","last_peered":"2024-03-14T09:53:41.789908+0000","last_clean":"2024-03-14T09:53:41.789908+0000","last_became_active":"2024-03-14T09:52:05.223380+0000","last_became_peered":"2024-03-14T09:52:05.223380+0000","last_unstale":"2024-03-14T09:53:41.789908+0000","last_undegraded":"2024-03-14T09:53:41.789908+0000","last_fullsized":"2024-03-14T09:53:41.789908+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:08:33.098215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785120+0000","last_change":"2024-03-14T09:52:05.743277+0000","last_active":"2024-03-14T09:53:41.785120+0000","last_peered":"2024-03-14T09:53:41.785120+0000","last_clean":"2024-03-14T09:53:41.785120+0000","last_became_active":"2024-03-14T09:52:05.741386+0000","last_became_peered":"2024-03-14T09:52:05.741386+0000","last_unstale":"2024-03-14T09:53:41.785120+0000","last_undegraded":"2024-03-14T09:53:41.785120+0000","last_fullsized":"2024-03-14T09:53:41.785120+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:08:08.320279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899160+0000","last_change":"2024-03-14T09:52:09.766485+0000","last_active":"2024-03-14T09:53:41.899160+0000","last_peered":"2024-03-14T09:53:41.899160+0000","last_clean":"2024-03-14T09:53:41.899160+0000","last_became_active":"2024-03-14T09:52:09.766115+0000","last_became_peered":"2024-03-14T09:52:09.766115+0000","last_unstale":"2024-03-14T09:53:41.899160+0000","last_undegraded":"2024-03-14T09:53:41.899160+0000","last_fullsized":"2024-03-14T09:53:41.899160+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:34:45.067757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"83'1","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786965+0000","last_change":"2024-03-14T09:52:05.233852+0000","last_active":"2024-03-14T09:53:41.786965+0000","last_peered":"2024-03-14T09:53:41.786965+0000","last_clean":"2024-03-14T09:53:41.786965+0000","last_became_active":"2024-03-14T09:52:05.233548+0000","last_became_peered":"2024-03-14T09:52:05.233548+0000","last_unstale":"2024-03-14T09:53:41.786965+0000","last_undegraded":"2024-03-14T09:53:41.786965+0000","last_fullsized":"2024-03-14T09:53:41.786965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:08:51.821465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"85'11","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788942+0000","last_change":"2024-03-14T09:52:05.731215+0000","last_active":"2024-03-14T09:53:41.788942+0000","last_peered":"2024-03-14T09:53:41.788942+0000","last_clean":"2024-03-14T09:53:41.788942+0000","last_became_active":"2024-03-14T09:52:05.730924+0000","last_became_peered":"2024-03-14T09:52:05.730924+0000","last_unstale":"2024-03-14T09:53:41.788942+0000","last_undegraded":"2024-03-14T09:53:41.788942+0000","last_fullsized":"2024-03-14T09:53:41.788942+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:12:17.560297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785051+0000","last_change":"2024-03-14T09:52:09.753775+0000","last_active":"2024-03-14T09:53:41.785051+0000","last_peered":"2024-03-14T09:53:41.785051+0000","last_clean":"2024-03-14T09:53:41.785051+0000","last_became_active":"2024-03-14T09:52:09.753422+0000","last_became_peered":"2024-03-14T09:52:09.753422+0000","last_unstale":"2024-03-14T09:53:41.785051+0000","last_undegraded":"2024-03-14T09:53:41.785051+0000","last_fullsized":"2024-03-14T09:53:41.785051+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:59:25.625909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"83'3","reported_seq":106,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:51.850659+0000","last_change":"2024-03-14T09:52:05.229064+0000","last_active":"2024-03-14T09:53:51.850659+0000","last_peered":"2024-03-14T09:53:51.850659+0000","last_clean":"2024-03-14T09:53:51.850659+0000","last_became_active":"2024-03-14T09:52:05.228751+0000","last_became_peered":"2024-03-14T09:52:05.228751+0000","last_unstale":"2024-03-14T09:53:51.850659+0000","last_undegraded":"2024-03-14T09:53:51.850659+0000","last_fullsized":"2024-03-14T09:53:51.850659+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:13:29.832866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"85'10","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790141+0000","last_change":"2024-03-14T09:52:05.737798+0000","last_active":"2024-03-14T09:53:41.790141+0000","last_peered":"2024-03-14T09:53:41.790141+0000","last_clean":"2024-03-14T09:53:41.790141+0000","last_became_active":"2024-03-14T09:52:05.737449+0000","last_became_peered":"2024-03-14T09:52:05.737449+0000","last_unstale":"2024-03-14T09:53:41.790141+0000","last_undegraded":"2024-03-14T09:53:41.790141+0000","last_fullsized":"2024-03-14T09:53:41.790141+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:40:10.079770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788972+0000","last_change":"2024-03-14T09:52:09.768368+0000","last_active":"2024-03-14T09:53:41.788972+0000","last_peered":"2024-03-14T09:53:41.788972+0000","last_clean":"2024-03-14T09:53:41.788972+0000","last_became_active":"2024-03-14T09:52:09.767622+0000","last_became_peered":"2024-03-14T09:52:09.767622+0000","last_unstale":"2024-03-14T09:53:41.788972+0000","last_undegraded":"2024-03-14T09:53:41.788972+0000","last_fullsized":"2024-03-14T09:53:41.788972+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:19:28.836317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"83'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783894+0000","last_change":"2024-03-14T09:52:05.229778+0000","last_active":"2024-03-14T09:53:41.783894+0000","last_peered":"2024-03-14T09:53:41.783894+0000","last_clean":"2024-03-14T09:53:41.783894+0000","last_became_active":"2024-03-14T09:52:05.229468+0000","last_became_peered":"2024-03-14T09:52:05.229468+0000","last_unstale":"2024-03-14T09:53:41.783894+0000","last_undegraded":"2024-03-14T09:53:41.783894+0000","last_fullsized":"2024-03-14T09:53:41.783894+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:49:58.433646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899333+0000","last_change":"2024-03-14T09:52:05.751027+0000","last_active":"2024-03-14T09:53:41.899333+0000","last_peered":"2024-03-14T09:53:41.899333+0000","last_clean":"2024-03-14T09:53:41.899333+0000","last_became_active":"2024-03-14T09:52:05.750650+0000","last_became_peered":"2024-03-14T09:52:05.750650+0000","last_unstale":"2024-03-14T09:53:41.899333+0000","last_undegraded":"2024-03-14T09:53:41.899333+0000","last_fullsized":"2024-03-14T09:53:41.899333+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:05:37.264396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:43.631132+0000","last_change":"2024-03-14T09:53:43.631132+0000","last_active":"2024-03-14T09:53:43.631132+0000","last_peered":"2024-03-14T09:53:43.631132+0000","last_clean":"2024-03-14T09:53:43.631132+0000","last_became_active":"2024-03-14T09:53:43.630718+0000","last_became_peered":"2024-03-14T09:53:43.630718+0000","last_unstale":"2024-03-14T09:53:43.631132+0000","last_undegraded":"2024-03-14T09:53:43.631132+0000","last_fullsized":"2024-03-14T09:53:43.631132+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T15:45:17.156095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"83'4","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787859+0000","last_change":"2024-03-14T09:52:05.235472+0000","last_active":"2024-03-14T09:53:41.787859+0000","last_peered":"2024-03-14T09:53:41.787859+0000","last_clean":"2024-03-14T09:53:41.787859+0000","last_became_active":"2024-03-14T09:52:05.235106+0000","last_became_peered":"2024-03-14T09:52:05.235106+0000","last_unstale":"2024-03-14T09:53:41.787859+0000","last_undegraded":"2024-03-14T09:53:41.787859+0000","last_fullsized":"2024-03-14T09:53:41.787859+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:30:36.446130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"85'11","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789899+0000","last_change":"2024-03-14T09:52:05.747615+0000","last_active":"2024-03-14T09:53:41.789899+0000","last_peered":"2024-03-14T09:53:41.789899+0000","last_clean":"2024-03-14T09:53:41.789899+0000","last_became_active":"2024-03-14T09:52:05.747344+0000","last_became_peered":"2024-03-14T09:52:05.747344+0000","last_unstale":"2024-03-14T09:53:41.789899+0000","last_undegraded":"2024-03-14T09:53:41.789899+0000","last_fullsized":"2024-03-14T09:53:41.789899+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:52:23.096124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788627+0000","last_change":"2024-03-14T09:52:09.768295+0000","last_active":"2024-03-14T09:53:41.788627+0000","last_peered":"2024-03-14T09:53:41.788627+0000","last_clean":"2024-03-14T09:53:41.788627+0000","last_became_active":"2024-03-14T09:52:09.760997+0000","last_became_peered":"2024-03-14T09:52:09.760997+0000","last_unstale":"2024-03-14T09:53:41.788627+0000","last_undegraded":"2024-03-14T09:53:41.788627+0000","last_fullsized":"2024-03-14T09:53:41.788627+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:22:09.909302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'37","reported_seq":233,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:52.852579+0000","last_change":"2024-03-14T09:52:05.227876+0000","last_active":"2024-03-14T09:53:52.852579+0000","last_peered":"2024-03-14T09:53:52.852579+0000","last_clean":"2024-03-14T09:53:52.852579+0000","last_became_active":"2024-03-14T09:52:05.227519+0000","last_became_peered":"2024-03-14T09:52:05.227519+0000","last_unstale":"2024-03-14T09:53:52.852579+0000","last_undegraded":"2024-03-14T09:53:52.852579+0000","last_fullsized":"2024-03-14T09:53:52.852579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:12.616609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66994,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":176,"num_read_kb":154,"num_write":38,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788944+0000","last_change":"2024-03-14T09:52:05.737432+0000","last_active":"2024-03-14T09:53:41.788944+0000","last_peered":"2024-03-14T09:53:41.788944+0000","last_clean":"2024-03-14T09:53:41.788944+0000","last_became_active":"2024-03-14T09:52:05.735050+0000","last_became_peered":"2024-03-14T09:52:05.735050+0000","last_unstale":"2024-03-14T09:53:41.788944+0000","last_undegraded":"2024-03-14T09:53:41.788944+0000","last_fullsized":"2024-03-14T09:53:41.788944+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T09:57:45.383155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788252+0000","last_change":"2024-03-14T09:52:09.754909+0000","last_active":"2024-03-14T09:53:41.788252+0000","last_peered":"2024-03-14T09:53:41.788252+0000","last_clean":"2024-03-14T09:53:41.788252+0000","last_became_active":"2024-03-14T09:52:09.754367+0000","last_became_peered":"2024-03-14T09:52:09.754367+0000","last_unstale":"2024-03-14T09:53:41.788252+0000","last_undegraded":"2024-03-14T09:53:41.788252+0000","last_fullsized":"2024-03-14T09:53:41.788252+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:52:34.251555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"83'6","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783813+0000","last_change":"2024-03-14T09:52:05.224869+0000","last_active":"2024-03-14T09:53:41.783813+0000","last_peered":"2024-03-14T09:53:41.783813+0000","last_clean":"2024-03-14T09:53:41.783813+0000","last_became_active":"2024-03-14T09:52:05.224499+0000","last_became_peered":"2024-03-14T09:52:05.224499+0000","last_unstale":"2024-03-14T09:53:41.783813+0000","last_undegraded":"2024-03-14T09:53:41.783813+0000","last_fullsized":"2024-03-14T09:53:41.783813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:49:01.729417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789760+0000","last_change":"2024-03-14T09:52:05.729168+0000","last_active":"2024-03-14T09:53:41.789760+0000","last_peered":"2024-03-14T09:53:41.789760+0000","last_clean":"2024-03-14T09:53:41.789760+0000","last_became_active":"2024-03-14T09:52:05.728806+0000","last_became_peered":"2024-03-14T09:52:05.728806+0000","last_unstale":"2024-03-14T09:53:41.789760+0000","last_undegraded":"2024-03-14T09:53:41.789760+0000","last_fullsized":"2024-03-14T09:53:41.789760+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:54:36.758853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899244+0000","last_change":"2024-03-14T09:52:09.770126+0000","last_active":"2024-03-14T09:53:41.899244+0000","last_peered":"2024-03-14T09:53:41.899244+0000","last_clean":"2024-03-14T09:53:41.899244+0000","last_became_active":"2024-03-14T09:52:09.769734+0000","last_became_peered":"2024-03-14T09:52:09.769734+0000","last_unstale":"2024-03-14T09:53:41.899244+0000","last_undegraded":"2024-03-14T09:53:41.899244+0000","last_fullsized":"2024-03-14T09:53:41.899244+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:50:37.839858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"83'3","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787719+0000","last_change":"2024-03-14T09:52:05.235201+0000","last_active":"2024-03-14T09:53:41.787719+0000","last_peered":"2024-03-14T09:53:41.787719+0000","last_clean":"2024-03-14T09:53:41.787719+0000","last_became_active":"2024-03-14T09:52:05.234816+0000","last_became_peered":"2024-03-14T09:52:05.234816+0000","last_unstale":"2024-03-14T09:53:41.787719+0000","last_undegraded":"2024-03-14T09:53:41.787719+0000","last_fullsized":"2024-03-14T09:53:41.787719+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:01:50.070913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'15","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783755+0000","last_change":"2024-03-14T09:52:05.735304+0000","last_active":"2024-03-14T09:53:41.783755+0000","last_peered":"2024-03-14T09:53:41.783755+0000","last_clean":"2024-03-14T09:53:41.783755+0000","last_became_active":"2024-03-14T09:52:05.734977+0000","last_became_peered":"2024-03-14T09:52:05.734977+0000","last_unstale":"2024-03-14T09:53:41.783755+0000","last_undegraded":"2024-03-14T09:53:41.783755+0000","last_fullsized":"2024-03-14T09:53:41.783755+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:08:28.071962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789402+0000","last_change":"2024-03-14T09:52:09.770721+0000","last_active":"2024-03-14T09:53:41.789402+0000","last_peered":"2024-03-14T09:53:41.789402+0000","last_clean":"2024-03-14T09:53:41.789402+0000","last_became_active":"2024-03-14T09:52:09.770270+0000","last_became_peered":"2024-03-14T09:52:09.770270+0000","last_unstale":"2024-03-14T09:53:41.789402+0000","last_undegraded":"2024-03-14T09:53:41.789402+0000","last_fullsized":"2024-03-14T09:53:41.789402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:09:33.480235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'6","reported_seq":111,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:48.852068+0000","last_change":"2024-03-14T09:52:05.227260+0000","last_active":"2024-03-14T09:53:48.852068+0000","last_peered":"2024-03-14T09:53:48.852068+0000","last_clean":"2024-03-14T09:53:48.852068+0000","last_became_active":"2024-03-14T09:52:05.226922+0000","last_became_peered":"2024-03-14T09:52:05.226922+0000","last_unstale":"2024-03-14T09:53:48.852068+0000","last_undegraded":"2024-03-14T09:53:48.852068+0000","last_fullsized":"2024-03-14T09:53:48.852068+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:36:09.770478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783445+0000","last_change":"2024-03-14T09:52:05.734278+0000","last_active":"2024-03-14T09:53:41.783445+0000","last_peered":"2024-03-14T09:53:41.783445+0000","last_clean":"2024-03-14T09:53:41.783445+0000","last_became_active":"2024-03-14T09:52:05.732051+0000","last_became_peered":"2024-03-14T09:52:05.732051+0000","last_unstale":"2024-03-14T09:53:41.783445+0000","last_undegraded":"2024-03-14T09:53:41.783445+0000","last_fullsized":"2024-03-14T09:53:41.783445+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:33:07.137983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790644+0000","last_change":"2024-03-14T09:52:09.764068+0000","last_active":"2024-03-14T09:53:41.790644+0000","last_peered":"2024-03-14T09:53:41.790644+0000","last_clean":"2024-03-14T09:53:41.790644+0000","last_became_active":"2024-03-14T09:52:09.763613+0000","last_became_peered":"2024-03-14T09:52:09.763613+0000","last_unstale":"2024-03-14T09:53:41.790644+0000","last_undegraded":"2024-03-14T09:53:41.790644+0000","last_fullsized":"2024-03-14T09:53:41.790644+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:32:15.524602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"83'15","reported_seq":63,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787502+0000","last_change":"2024-03-14T09:52:42.438205+0000","last_active":"2024-03-14T09:53:41.787502+0000","last_peered":"2024-03-14T09:53:41.787502+0000","last_clean":"2024-03-14T09:53:41.787502+0000","last_became_active":"2024-03-14T09:52:42.426268+0000","last_became_peered":"2024-03-14T09:52:42.426268+0000","last_unstale":"2024-03-14T09:53:41.787502+0000","last_undegraded":"2024-03-14T09:53:41.787502+0000","last_fullsized":"2024-03-14T09:53:41.787502+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:10:09.651131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788220+0000","last_change":"2024-03-14T09:52:05.734366+0000","last_active":"2024-03-14T09:53:41.788220+0000","last_peered":"2024-03-14T09:53:41.788220+0000","last_clean":"2024-03-14T09:53:41.788220+0000","last_became_active":"2024-03-14T09:52:05.734058+0000","last_became_peered":"2024-03-14T09:52:05.734058+0000","last_unstale":"2024-03-14T09:53:41.788220+0000","last_undegraded":"2024-03-14T09:53:41.788220+0000","last_fullsized":"2024-03-14T09:53:41.788220+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:49:49.233683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788119+0000","last_change":"2024-03-14T09:52:09.760241+0000","last_active":"2024-03-14T09:53:41.788119+0000","last_peered":"2024-03-14T09:53:41.788119+0000","last_clean":"2024-03-14T09:53:41.788119+0000","last_became_active":"2024-03-14T09:52:09.759797+0000","last_became_peered":"2024-03-14T09:52:09.759797+0000","last_unstale":"2024-03-14T09:53:41.788119+0000","last_undegraded":"2024-03-14T09:53:41.788119+0000","last_fullsized":"2024-03-14T09:53:41.788119+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:50:33.035775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:42.792396+0000","last_change":"2024-03-14T09:52:05.232932+0000","last_active":"2024-03-14T09:53:42.792396+0000","last_peered":"2024-03-14T09:53:42.792396+0000","last_clean":"2024-03-14T09:53:42.792396+0000","last_became_active":"2024-03-14T09:52:05.232608+0000","last_became_peered":"2024-03-14T09:52:05.232608+0000","last_unstale":"2024-03-14T09:53:42.792396+0000","last_undegraded":"2024-03-14T09:53:42.792396+0000","last_fullsized":"2024-03-14T09:53:42.792396+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:43:59.613531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899344+0000","last_change":"2024-03-14T09:52:05.733762+0000","last_active":"2024-03-14T09:53:41.899344+0000","last_peered":"2024-03-14T09:53:41.899344+0000","last_clean":"2024-03-14T09:53:41.899344+0000","last_became_active":"2024-03-14T09:52:05.733417+0000","last_became_peered":"2024-03-14T09:52:05.733417+0000","last_unstale":"2024-03-14T09:53:41.899344+0000","last_undegraded":"2024-03-14T09:53:41.899344+0000","last_fullsized":"2024-03-14T09:53:41.899344+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:11:12.593952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789125+0000","last_change":"2024-03-14T09:52:42.426896+0000","last_active":"2024-03-14T09:53:41.789125+0000","last_peered":"2024-03-14T09:53:41.789125+0000","last_clean":"2024-03-14T09:53:41.789125+0000","last_became_active":"2024-03-14T09:52:42.426287+0000","last_became_peered":"2024-03-14T09:52:42.426287+0000","last_unstale":"2024-03-14T09:53:41.789125+0000","last_undegraded":"2024-03-14T09:53:41.789125+0000","last_fullsized":"2024-03-14T09:53:41.789125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:53:16.523707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789199+0000","last_change":"2024-03-14T09:52:05.227636+0000","last_active":"2024-03-14T09:53:41.789199+0000","last_peered":"2024-03-14T09:53:41.789199+0000","last_clean":"2024-03-14T09:53:41.789199+0000","last_became_active":"2024-03-14T09:52:05.227274+0000","last_became_peered":"2024-03-14T09:52:05.227274+0000","last_unstale":"2024-03-14T09:53:41.789199+0000","last_undegraded":"2024-03-14T09:53:41.789199+0000","last_fullsized":"2024-03-14T09:53:41.789199+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:13:25.380090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'4","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787833+0000","last_change":"2024-03-14T09:52:05.733158+0000","last_active":"2024-03-14T09:53:41.787833+0000","last_peered":"2024-03-14T09:53:41.787833+0000","last_clean":"2024-03-14T09:53:41.787833+0000","last_became_active":"2024-03-14T09:52:05.732812+0000","last_became_peered":"2024-03-14T09:52:05.732812+0000","last_unstale":"2024-03-14T09:53:41.787833+0000","last_undegraded":"2024-03-14T09:53:41.787833+0000","last_fullsized":"2024-03-14T09:53:41.787833+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:01:49.154860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790946+0000","last_change":"2024-03-14T09:52:09.761596+0000","last_active":"2024-03-14T09:53:41.790946+0000","last_peered":"2024-03-14T09:53:41.790946+0000","last_clean":"2024-03-14T09:53:41.790946+0000","last_became_active":"2024-03-14T09:52:09.761184+0000","last_became_peered":"2024-03-14T09:52:09.761184+0000","last_unstale":"2024-03-14T09:53:41.790946+0000","last_undegraded":"2024-03-14T09:53:41.790946+0000","last_fullsized":"2024-03-14T09:53:41.790946+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:49:02.567472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"83'1","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783455+0000","last_change":"2024-03-14T09:52:05.231828+0000","last_active":"2024-03-14T09:53:41.783455+0000","last_peered":"2024-03-14T09:53:41.783455+0000","last_clean":"2024-03-14T09:53:41.783455+0000","last_became_active":"2024-03-14T09:52:05.231490+0000","last_became_peered":"2024-03-14T09:52:05.231490+0000","last_unstale":"2024-03-14T09:53:41.783455+0000","last_undegraded":"2024-03-14T09:53:41.783455+0000","last_fullsized":"2024-03-14T09:53:41.783455+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:24:53.899728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787604+0000","last_change":"2024-03-14T09:52:05.744646+0000","last_active":"2024-03-14T09:53:41.787604+0000","last_peered":"2024-03-14T09:53:41.787604+0000","last_clean":"2024-03-14T09:53:41.787604+0000","last_became_active":"2024-03-14T09:52:05.744327+0000","last_became_peered":"2024-03-14T09:52:05.744327+0000","last_unstale":"2024-03-14T09:53:41.787604+0000","last_undegraded":"2024-03-14T09:53:41.787604+0000","last_fullsized":"2024-03-14T09:53:41.787604+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:46:50.643350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.791013+0000","last_change":"2024-03-14T09:52:09.765374+0000","last_active":"2024-03-14T09:53:41.791013+0000","last_peered":"2024-03-14T09:53:41.791013+0000","last_clean":"2024-03-14T09:53:41.791013+0000","last_became_active":"2024-03-14T09:52:09.764939+0000","last_became_peered":"2024-03-14T09:52:09.764939+0000","last_unstale":"2024-03-14T09:53:41.791013+0000","last_undegraded":"2024-03-14T09:53:41.791013+0000","last_fullsized":"2024-03-14T09:53:41.791013+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:08:10.688746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787063+0000","last_change":"2024-03-14T09:52:05.729076+0000","last_active":"2024-03-14T09:53:41.787063+0000","last_peered":"2024-03-14T09:53:41.787063+0000","last_clean":"2024-03-14T09:53:41.787063+0000","last_became_active":"2024-03-14T09:52:05.728711+0000","last_became_peered":"2024-03-14T09:52:05.728711+0000","last_unstale":"2024-03-14T09:53:41.787063+0000","last_undegraded":"2024-03-14T09:53:41.787063+0000","last_fullsized":"2024-03-14T09:53:41.787063+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:11:05.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":110,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:53.860698+0000","last_change":"2024-03-14T09:52:05.236209+0000","last_active":"2024-03-14T09:53:53.860698+0000","last_peered":"2024-03-14T09:53:53.860698+0000","last_clean":"2024-03-14T09:53:53.860698+0000","last_became_active":"2024-03-14T09:52:05.235517+0000","last_became_peered":"2024-03-14T09:52:05.235517+0000","last_unstale":"2024-03-14T09:53:53.860698+0000","last_undegraded":"2024-03-14T09:53:53.860698+0000","last_fullsized":"2024-03-14T09:53:53.860698+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:09:54.275728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":85,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"83'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783734+0000","last_change":"2024-03-14T09:52:09.761095+0000","last_active":"2024-03-14T09:53:41.783734+0000","last_peered":"2024-03-14T09:53:41.783734+0000","last_clean":"2024-03-14T09:53:41.783734+0000","last_became_active":"2024-03-14T09:52:09.760662+0000","last_became_peered":"2024-03-14T09:52:09.760662+0000","last_unstale":"2024-03-14T09:53:41.783734+0000","last_undegraded":"2024-03-14T09:53:41.783734+0000","last_fullsized":"2024-03-14T09:53:41.783734+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:07:17.670605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785163+0000","last_change":"2024-03-14T09:52:05.736078+0000","last_active":"2024-03-14T09:53:41.785163+0000","last_peered":"2024-03-14T09:53:41.785163+0000","last_clean":"2024-03-14T09:53:41.785163+0000","last_became_active":"2024-03-14T09:52:05.735715+0000","last_became_peered":"2024-03-14T09:52:05.735715+0000","last_unstale":"2024-03-14T09:53:41.785163+0000","last_undegraded":"2024-03-14T09:53:41.785163+0000","last_fullsized":"2024-03-14T09:53:41.785163+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:41:32.863680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"83'3","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786470+0000","last_change":"2024-03-14T09:52:05.225756+0000","last_active":"2024-03-14T09:53:41.786470+0000","last_peered":"2024-03-14T09:53:41.786470+0000","last_clean":"2024-03-14T09:53:41.786470+0000","last_became_active":"2024-03-14T09:52:05.225378+0000","last_became_peered":"2024-03-14T09:52:05.225378+0000","last_unstale":"2024-03-14T09:53:41.786470+0000","last_undegraded":"2024-03-14T09:53:41.786470+0000","last_fullsized":"2024-03-14T09:53:41.786470+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:14:49.594636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":339,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783295+0000","last_change":"2024-03-14T09:51:11.588790+0000","last_active":"2024-03-14T09:53:41.783295+0000","last_peered":"2024-03-14T09:53:41.783295+0000","last_clean":"2024-03-14T09:53:41.783295+0000","last_became_active":"2024-03-14T09:51:11.588287+0000","last_became_peered":"2024-03-14T09:51:11.588287+0000","last_unstale":"2024-03-14T09:53:41.783295+0000","last_undegraded":"2024-03-14T09:53:41.783295+0000","last_fullsized":"2024-03-14T09:53:41.783295+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:48:37.651174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:48:37.651174+0000","last_clean_scrub_stamp":"2024-03-14T09:48:37.651174+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:06:24.252565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787689+0000","last_change":"2024-03-14T09:52:09.761344+0000","last_active":"2024-03-14T09:53:41.787689+0000","last_peered":"2024-03-14T09:53:41.787689+0000","last_clean":"2024-03-14T09:53:41.787689+0000","last_became_active":"2024-03-14T09:52:09.760899+0000","last_became_peered":"2024-03-14T09:52:09.760899+0000","last_unstale":"2024-03-14T09:53:41.787689+0000","last_undegraded":"2024-03-14T09:53:41.787689+0000","last_fullsized":"2024-03-14T09:53:41.787689+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:24:26.212157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789785+0000","last_change":"2024-03-14T09:52:05.738735+0000","last_active":"2024-03-14T09:53:41.789785+0000","last_peered":"2024-03-14T09:53:41.789785+0000","last_clean":"2024-03-14T09:53:41.789785+0000","last_became_active":"2024-03-14T09:52:05.738447+0000","last_became_peered":"2024-03-14T09:52:05.738447+0000","last_unstale":"2024-03-14T09:53:41.789785+0000","last_undegraded":"2024-03-14T09:53:41.789785+0000","last_fullsized":"2024-03-14T09:53:41.789785+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:03:24.598291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'3","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783341+0000","last_change":"2024-03-14T09:52:05.230107+0000","last_active":"2024-03-14T09:53:41.783341+0000","last_peered":"2024-03-14T09:53:41.783341+0000","last_clean":"2024-03-14T09:53:41.783341+0000","last_became_active":"2024-03-14T09:52:05.229812+0000","last_became_peered":"2024-03-14T09:52:05.229812+0000","last_unstale":"2024-03-14T09:53:41.783341+0000","last_undegraded":"2024-03-14T09:53:41.783341+0000","last_fullsized":"2024-03-14T09:53:41.783341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:09:44.884867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899259+0000","last_change":"2024-03-14T09:52:09.763424+0000","last_active":"2024-03-14T09:53:41.899259+0000","last_peered":"2024-03-14T09:53:41.899259+0000","last_clean":"2024-03-14T09:53:41.899259+0000","last_became_active":"2024-03-14T09:52:09.763028+0000","last_became_peered":"2024-03-14T09:52:09.763028+0000","last_unstale":"2024-03-14T09:53:41.899259+0000","last_undegraded":"2024-03-14T09:53:41.899259+0000","last_fullsized":"2024-03-14T09:53:41.899259+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:47:31.415788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"83'1","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789049+0000","last_change":"2024-03-14T09:52:05.235099+0000","last_active":"2024-03-14T09:53:41.789049+0000","last_peered":"2024-03-14T09:53:41.789049+0000","last_clean":"2024-03-14T09:53:41.789049+0000","last_became_active":"2024-03-14T09:52:05.234798+0000","last_became_peered":"2024-03-14T09:52:05.234798+0000","last_unstale":"2024-03-14T09:53:41.789049+0000","last_undegraded":"2024-03-14T09:53:41.789049+0000","last_fullsized":"2024-03-14T09:53:41.789049+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:00:07.461931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.791106+0000","last_change":"2024-03-14T09:52:05.742736+0000","last_active":"2024-03-14T09:53:41.791106+0000","last_peered":"2024-03-14T09:53:41.791106+0000","last_clean":"2024-03-14T09:53:41.791106+0000","last_became_active":"2024-03-14T09:52:05.740927+0000","last_became_peered":"2024-03-14T09:52:05.740927+0000","last_unstale":"2024-03-14T09:53:41.791106+0000","last_undegraded":"2024-03-14T09:53:41.791106+0000","last_fullsized":"2024-03-14T09:53:41.791106+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:50:17.738763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.784877+0000","last_change":"2024-03-14T09:52:09.762722+0000","last_active":"2024-03-14T09:53:41.784877+0000","last_peered":"2024-03-14T09:53:41.784877+0000","last_clean":"2024-03-14T09:53:41.784877+0000","last_became_active":"2024-03-14T09:52:09.759053+0000","last_became_peered":"2024-03-14T09:52:09.759053+0000","last_unstale":"2024-03-14T09:53:41.784877+0000","last_undegraded":"2024-03-14T09:53:41.784877+0000","last_fullsized":"2024-03-14T09:53:41.784877+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:40:29.680002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:43.740679+0000","last_change":"2024-03-14T09:52:05.224778+0000","last_active":"2024-03-14T09:53:43.740679+0000","last_peered":"2024-03-14T09:53:43.740679+0000","last_clean":"2024-03-14T09:53:43.740679+0000","last_became_active":"2024-03-14T09:52:05.223158+0000","last_became_peered":"2024-03-14T09:52:05.223158+0000","last_unstale":"2024-03-14T09:53:43.740679+0000","last_undegraded":"2024-03-14T09:53:43.740679+0000","last_fullsized":"2024-03-14T09:53:43.740679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:01:29.910099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785742+0000","last_change":"2024-03-14T09:52:05.741753+0000","last_active":"2024-03-14T09:53:41.785742+0000","last_peered":"2024-03-14T09:53:41.785742+0000","last_clean":"2024-03-14T09:53:41.785742+0000","last_became_active":"2024-03-14T09:52:05.741451+0000","last_became_peered":"2024-03-14T09:52:05.741451+0000","last_unstale":"2024-03-14T09:53:41.785742+0000","last_undegraded":"2024-03-14T09:53:41.785742+0000","last_fullsized":"2024-03-14T09:53:41.785742+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:42:59.920909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783437+0000","last_change":"2024-03-14T09:52:09.755884+0000","last_active":"2024-03-14T09:53:41.783437+0000","last_peered":"2024-03-14T09:53:41.783437+0000","last_clean":"2024-03-14T09:53:41.783437+0000","last_became_active":"2024-03-14T09:52:09.755474+0000","last_became_peered":"2024-03-14T09:52:09.755474+0000","last_unstale":"2024-03-14T09:53:41.783437+0000","last_undegraded":"2024-03-14T09:53:41.783437+0000","last_fullsized":"2024-03-14T09:53:41.783437+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:59:23.383388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"83'4","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786361+0000","last_change":"2024-03-14T09:52:05.232673+0000","last_active":"2024-03-14T09:53:41.786361+0000","last_peered":"2024-03-14T09:53:41.786361+0000","last_clean":"2024-03-14T09:53:41.786361+0000","last_became_active":"2024-03-14T09:52:05.232296+0000","last_became_peered":"2024-03-14T09:52:05.232296+0000","last_unstale":"2024-03-14T09:53:41.786361+0000","last_undegraded":"2024-03-14T09:53:41.786361+0000","last_fullsized":"2024-03-14T09:53:41.786361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:24:56.638451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899250+0000","last_change":"2024-03-14T09:52:05.744906+0000","last_active":"2024-03-14T09:53:41.899250+0000","last_peered":"2024-03-14T09:53:41.899250+0000","last_clean":"2024-03-14T09:53:41.899250+0000","last_became_active":"2024-03-14T09:52:05.743753+0000","last_became_peered":"2024-03-14T09:52:05.743753+0000","last_unstale":"2024-03-14T09:53:41.899250+0000","last_undegraded":"2024-03-14T09:53:41.899250+0000","last_fullsized":"2024-03-14T09:53:41.899250+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T09:59:40.358352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786261+0000","last_change":"2024-03-14T09:52:09.768355+0000","last_active":"2024-03-14T09:53:41.786261+0000","last_peered":"2024-03-14T09:53:41.786261+0000","last_clean":"2024-03-14T09:53:41.786261+0000","last_became_active":"2024-03-14T09:52:09.767919+0000","last_became_peered":"2024-03-14T09:52:09.767919+0000","last_unstale":"2024-03-14T09:53:41.786261+0000","last_undegraded":"2024-03-14T09:53:41.786261+0000","last_fullsized":"2024-03-14T09:53:41.786261+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:45:08.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"83'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789425+0000","last_change":"2024-03-14T09:52:05.230343+0000","last_active":"2024-03-14T09:53:41.789425+0000","last_peered":"2024-03-14T09:53:41.789425+0000","last_clean":"2024-03-14T09:53:41.789425+0000","last_became_active":"2024-03-14T09:52:05.229974+0000","last_became_peered":"2024-03-14T09:52:05.229974+0000","last_unstale":"2024-03-14T09:53:41.789425+0000","last_undegraded":"2024-03-14T09:53:41.789425+0000","last_fullsized":"2024-03-14T09:53:41.789425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:21:02.458108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"85'2","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787651+0000","last_change":"2024-03-14T09:52:05.738347+0000","last_active":"2024-03-14T09:53:41.787651+0000","last_peered":"2024-03-14T09:53:41.787651+0000","last_clean":"2024-03-14T09:53:41.787651+0000","last_became_active":"2024-03-14T09:52:05.737961+0000","last_became_peered":"2024-03-14T09:52:05.737961+0000","last_unstale":"2024-03-14T09:53:41.787651+0000","last_undegraded":"2024-03-14T09:53:41.787651+0000","last_fullsized":"2024-03-14T09:53:41.787651+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:02:57.252069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:42.843871+0000","last_change":"2024-03-14T09:53:42.843441+0000","last_active":"2024-03-14T09:53:42.843871+0000","last_peered":"2024-03-14T09:53:42.843871+0000","last_clean":"2024-03-14T09:53:42.843871+0000","last_became_active":"2024-03-14T09:53:42.842860+0000","last_became_peered":"2024-03-14T09:53:42.842860+0000","last_unstale":"2024-03-14T09:53:42.843871+0000","last_undegraded":"2024-03-14T09:53:42.843871+0000","last_fullsized":"2024-03-14T09:53:42.843871+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:20:23.073703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"83'3","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.784877+0000","last_change":"2024-03-14T09:52:05.233579+0000","last_active":"2024-03-14T09:53:41.784877+0000","last_peered":"2024-03-14T09:53:41.784877+0000","last_clean":"2024-03-14T09:53:41.784877+0000","last_became_active":"2024-03-14T09:52:05.231605+0000","last_became_peered":"2024-03-14T09:52:05.231605+0000","last_unstale":"2024-03-14T09:53:41.784877+0000","last_undegraded":"2024-03-14T09:53:41.784877+0000","last_fullsized":"2024-03-14T09:53:41.784877+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:55:38.282203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899095+0000","last_change":"2024-03-14T09:52:05.736586+0000","last_active":"2024-03-14T09:53:41.899095+0000","last_peered":"2024-03-14T09:53:41.899095+0000","last_clean":"2024-03-14T09:53:41.899095+0000","last_became_active":"2024-03-14T09:52:05.736235+0000","last_became_peered":"2024-03-14T09:52:05.736235+0000","last_unstale":"2024-03-14T09:53:41.899095+0000","last_undegraded":"2024-03-14T09:53:41.899095+0000","last_fullsized":"2024-03-14T09:53:41.899095+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:46:53.293698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"83'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785979+0000","last_change":"2024-03-14T09:52:09.768203+0000","last_active":"2024-03-14T09:53:41.785979+0000","last_peered":"2024-03-14T09:53:41.785979+0000","last_clean":"2024-03-14T09:53:41.785979+0000","last_became_active":"2024-03-14T09:52:09.767777+0000","last_became_peered":"2024-03-14T09:52:09.767777+0000","last_unstale":"2024-03-14T09:53:41.785979+0000","last_undegraded":"2024-03-14T09:53:41.785979+0000","last_fullsized":"2024-03-14T09:53:41.785979+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:20:47.598905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"83'1","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783567+0000","last_change":"2024-03-14T09:52:05.232359+0000","last_active":"2024-03-14T09:53:41.783567+0000","last_peered":"2024-03-14T09:53:41.783567+0000","last_clean":"2024-03-14T09:53:41.783567+0000","last_became_active":"2024-03-14T09:52:05.231977+0000","last_became_peered":"2024-03-14T09:52:05.231977+0000","last_unstale":"2024-03-14T09:53:41.783567+0000","last_undegraded":"2024-03-14T09:53:41.783567+0000","last_fullsized":"2024-03-14T09:53:41.783567+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:32.678157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785461+0000","last_change":"2024-03-14T09:52:05.735822+0000","last_active":"2024-03-14T09:53:41.785461+0000","last_peered":"2024-03-14T09:53:41.785461+0000","last_clean":"2024-03-14T09:53:41.785461+0000","last_became_active":"2024-03-14T09:52:05.735059+0000","last_became_peered":"2024-03-14T09:52:05.735059+0000","last_unstale":"2024-03-14T09:53:41.785461+0000","last_undegraded":"2024-03-14T09:53:41.785461+0000","last_fullsized":"2024-03-14T09:53:41.785461+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T15:22:33.033689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786204+0000","last_change":"2024-03-14T09:52:09.759794+0000","last_active":"2024-03-14T09:53:41.786204+0000","last_peered":"2024-03-14T09:53:41.786204+0000","last_clean":"2024-03-14T09:53:41.786204+0000","last_became_active":"2024-03-14T09:52:09.759366+0000","last_became_peered":"2024-03-14T09:52:09.759366+0000","last_unstale":"2024-03-14T09:53:41.786204+0000","last_undegraded":"2024-03-14T09:53:41.786204+0000","last_fullsized":"2024-03-14T09:53:41.786204+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:52:40.641264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"83'4","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788103+0000","last_change":"2024-03-14T09:52:42.325733+0000","last_active":"2024-03-14T09:53:41.788103+0000","last_peered":"2024-03-14T09:53:41.788103+0000","last_clean":"2024-03-14T09:53:41.788103+0000","last_became_active":"2024-03-14T09:52:42.322002+0000","last_became_peered":"2024-03-14T09:52:42.322002+0000","last_unstale":"2024-03-14T09:53:41.788103+0000","last_undegraded":"2024-03-14T09:53:41.788103+0000","last_fullsized":"2024-03-14T09:53:41.788103+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:22:10.439271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783569+0000","last_change":"2024-03-14T09:52:05.734073+0000","last_active":"2024-03-14T09:53:41.783569+0000","last_peered":"2024-03-14T09:53:41.783569+0000","last_clean":"2024-03-14T09:53:41.783569+0000","last_became_active":"2024-03-14T09:52:05.731748+0000","last_became_peered":"2024-03-14T09:52:05.731748+0000","last_unstale":"2024-03-14T09:53:41.783569+0000","last_undegraded":"2024-03-14T09:53:41.783569+0000","last_fullsized":"2024-03-14T09:53:41.783569+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:01:11.789162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789540+0000","last_change":"2024-03-14T09:52:09.763309+0000","last_active":"2024-03-14T09:53:41.789540+0000","last_peered":"2024-03-14T09:53:41.789540+0000","last_clean":"2024-03-14T09:53:41.789540+0000","last_became_active":"2024-03-14T09:52:09.762868+0000","last_became_peered":"2024-03-14T09:52:09.762868+0000","last_unstale":"2024-03-14T09:53:41.789540+0000","last_undegraded":"2024-03-14T09:53:41.789540+0000","last_fullsized":"2024-03-14T09:53:41.789540+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:30:03.403911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"83'3","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.784042+0000","last_change":"2024-03-14T09:52:05.232310+0000","last_active":"2024-03-14T09:53:41.784042+0000","last_peered":"2024-03-14T09:53:41.784042+0000","last_clean":"2024-03-14T09:53:41.784042+0000","last_became_active":"2024-03-14T09:52:05.231931+0000","last_became_peered":"2024-03-14T09:52:05.231931+0000","last_unstale":"2024-03-14T09:53:41.784042+0000","last_undegraded":"2024-03-14T09:53:41.784042+0000","last_fullsized":"2024-03-14T09:53:41.784042+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:35:44.274800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783928+0000","last_change":"2024-03-14T09:52:05.731797+0000","last_active":"2024-03-14T09:53:41.783928+0000","last_peered":"2024-03-14T09:53:41.783928+0000","last_clean":"2024-03-14T09:53:41.783928+0000","last_became_active":"2024-03-14T09:52:05.730478+0000","last_became_peered":"2024-03-14T09:52:05.730478+0000","last_unstale":"2024-03-14T09:53:41.783928+0000","last_undegraded":"2024-03-14T09:53:41.783928+0000","last_fullsized":"2024-03-14T09:53:41.783928+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:52:50.684430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.898921+0000","last_change":"2024-03-14T09:52:09.772786+0000","last_active":"2024-03-14T09:53:41.898921+0000","last_peered":"2024-03-14T09:53:41.898921+0000","last_clean":"2024-03-14T09:53:41.898921+0000","last_became_active":"2024-03-14T09:52:09.772198+0000","last_became_peered":"2024-03-14T09:52:09.772198+0000","last_unstale":"2024-03-14T09:53:41.898921+0000","last_undegraded":"2024-03-14T09:53:41.898921+0000","last_fullsized":"2024-03-14T09:53:41.898921+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:28:56.620266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"78'1","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788052+0000","last_change":"2024-03-14T09:52:05.237657+0000","last_active":"2024-03-14T09:53:41.788052+0000","last_peered":"2024-03-14T09:53:41.788052+0000","last_clean":"2024-03-14T09:53:41.788052+0000","last_became_active":"2024-03-14T09:52:05.237381+0000","last_became_peered":"2024-03-14T09:52:05.237381+0000","last_unstale":"2024-03-14T09:53:41.788052+0000","last_undegraded":"2024-03-14T09:53:41.788052+0000","last_fullsized":"2024-03-14T09:53:41.788052+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:34:44.372128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783767+0000","last_change":"2024-03-14T09:52:05.730454+0000","last_active":"2024-03-14T09:53:41.783767+0000","last_peered":"2024-03-14T09:53:41.783767+0000","last_clean":"2024-03-14T09:53:41.783767+0000","last_became_active":"2024-03-14T09:52:05.730046+0000","last_became_peered":"2024-03-14T09:52:05.730046+0000","last_unstale":"2024-03-14T09:53:41.783767+0000","last_undegraded":"2024-03-14T09:53:41.783767+0000","last_fullsized":"2024-03-14T09:53:41.783767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:33:19.296993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786764+0000","last_change":"2024-03-14T09:52:09.755969+0000","last_active":"2024-03-14T09:53:41.786764+0000","last_peered":"2024-03-14T09:53:41.786764+0000","last_clean":"2024-03-14T09:53:41.786764+0000","last_became_active":"2024-03-14T09:52:09.755512+0000","last_became_peered":"2024-03-14T09:52:09.755512+0000","last_unstale":"2024-03-14T09:53:41.786764+0000","last_undegraded":"2024-03-14T09:53:41.786764+0000","last_fullsized":"2024-03-14T09:53:41.786764+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:31:03.388090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790735+0000","last_change":"2024-03-14T09:52:05.232676+0000","last_active":"2024-03-14T09:53:41.790735+0000","last_peered":"2024-03-14T09:53:41.790735+0000","last_clean":"2024-03-14T09:53:41.790735+0000","last_became_active":"2024-03-14T09:52:05.232332+0000","last_became_peered":"2024-03-14T09:52:05.232332+0000","last_unstale":"2024-03-14T09:53:41.790735+0000","last_undegraded":"2024-03-14T09:53:41.790735+0000","last_fullsized":"2024-03-14T09:53:41.790735+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:06:11.020555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783875+0000","last_change":"2024-03-14T09:52:05.733338+0000","last_active":"2024-03-14T09:53:41.783875+0000","last_peered":"2024-03-14T09:53:41.783875+0000","last_clean":"2024-03-14T09:53:41.783875+0000","last_became_active":"2024-03-14T09:52:05.732994+0000","last_became_peered":"2024-03-14T09:52:05.732994+0000","last_unstale":"2024-03-14T09:53:41.783875+0000","last_undegraded":"2024-03-14T09:53:41.783875+0000","last_fullsized":"2024-03-14T09:53:41.783875+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:31:25.085916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899170+0000","last_change":"2024-03-14T09:52:05.733121+0000","last_active":"2024-03-14T09:53:41.899170+0000","last_peered":"2024-03-14T09:53:41.899170+0000","last_clean":"2024-03-14T09:53:41.899170+0000","last_became_active":"2024-03-14T09:52:05.732671+0000","last_became_peered":"2024-03-14T09:52:05.732671+0000","last_unstale":"2024-03-14T09:53:41.899170+0000","last_undegraded":"2024-03-14T09:53:41.899170+0000","last_fullsized":"2024-03-14T09:53:41.899170+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:18:54.180131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"85'12","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783676+0000","last_change":"2024-03-14T09:52:05.736534+0000","last_active":"2024-03-14T09:53:41.783676+0000","last_peered":"2024-03-14T09:53:41.783676+0000","last_clean":"2024-03-14T09:53:41.783676+0000","last_became_active":"2024-03-14T09:52:05.736202+0000","last_became_peered":"2024-03-14T09:52:05.736202+0000","last_unstale":"2024-03-14T09:53:41.783676+0000","last_undegraded":"2024-03-14T09:53:41.783676+0000","last_fullsized":"2024-03-14T09:53:41.783676+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:22:46.161561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783472+0000","last_change":"2024-03-14T09:52:05.739434+0000","last_active":"2024-03-14T09:53:41.783472+0000","last_peered":"2024-03-14T09:53:41.783472+0000","last_clean":"2024-03-14T09:53:41.783472+0000","last_became_active":"2024-03-14T09:52:05.738023+0000","last_became_peered":"2024-03-14T09:52:05.738023+0000","last_unstale":"2024-03-14T09:53:41.783472+0000","last_undegraded":"2024-03-14T09:53:41.783472+0000","last_fullsized":"2024-03-14T09:53:41.783472+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:51:44.173645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899212+0000","last_change":"2024-03-14T09:52:05.751367+0000","last_active":"2024-03-14T09:53:41.899212+0000","last_peered":"2024-03-14T09:53:41.899212+0000","last_clean":"2024-03-14T09:53:41.899212+0000","last_became_active":"2024-03-14T09:52:05.750816+0000","last_became_peered":"2024-03-14T09:52:05.750816+0000","last_unstale":"2024-03-14T09:53:41.899212+0000","last_undegraded":"2024-03-14T09:53:41.899212+0000","last_fullsized":"2024-03-14T09:53:41.899212+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:57:58.403356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"85'4","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.791204+0000","last_change":"2024-03-14T09:52:05.746652+0000","last_active":"2024-03-14T09:53:41.791204+0000","last_peered":"2024-03-14T09:53:41.791204+0000","last_clean":"2024-03-14T09:53:41.791204+0000","last_became_active":"2024-03-14T09:52:05.746214+0000","last_became_peered":"2024-03-14T09:52:05.746214+0000","last_unstale":"2024-03-14T09:53:41.791204+0000","last_undegraded":"2024-03-14T09:53:41.791204+0000","last_fullsized":"2024-03-14T09:53:41.791204+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:21:39.244617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"85'6","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899490+0000","last_change":"2024-03-14T09:52:42.147977+0000","last_active":"2024-03-14T09:53:41.899490+0000","last_peered":"2024-03-14T09:53:41.899490+0000","last_clean":"2024-03-14T09:53:41.899490+0000","last_became_active":"2024-03-14T09:52:42.147623+0000","last_became_peered":"2024-03-14T09:52:42.147623+0000","last_unstale":"2024-03-14T09:53:41.899490+0000","last_undegraded":"2024-03-14T09:53:41.899490+0000","last_fullsized":"2024-03-14T09:53:41.899490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:38:35.351709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"85'15","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.784157+0000","last_change":"2024-03-14T09:52:05.739758+0000","last_active":"2024-03-14T09:53:41.784157+0000","last_peered":"2024-03-14T09:53:41.784157+0000","last_clean":"2024-03-14T09:53:41.784157+0000","last_became_active":"2024-03-14T09:52:05.739423+0000","last_became_peered":"2024-03-14T09:52:05.739423+0000","last_unstale":"2024-03-14T09:53:41.784157+0000","last_undegraded":"2024-03-14T09:53:41.784157+0000","last_fullsized":"2024-03-14T09:53:41.784157+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:47:22.594910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"85'2","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785881+0000","last_change":"2024-03-14T09:52:05.733281+0000","last_active":"2024-03-14T09:53:41.785881+0000","last_peered":"2024-03-14T09:53:41.785881+0000","last_clean":"2024-03-14T09:53:41.785881+0000","last_became_active":"2024-03-14T09:52:05.730081+0000","last_became_peered":"2024-03-14T09:52:05.730081+0000","last_unstale":"2024-03-14T09:53:41.785881+0000","last_undegraded":"2024-03-14T09:53:41.785881+0000","last_fullsized":"2024-03-14T09:53:41.785881+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:40:26.354063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786998+0000","last_change":"2024-03-14T09:52:05.726938+0000","last_active":"2024-03-14T09:53:41.786998+0000","last_peered":"2024-03-14T09:53:41.786998+0000","last_clean":"2024-03-14T09:53:41.786998+0000","last_became_active":"2024-03-14T09:52:05.726602+0000","last_became_peered":"2024-03-14T09:52:05.726602+0000","last_unstale":"2024-03-14T09:53:41.786998+0000","last_undegraded":"2024-03-14T09:53:41.786998+0000","last_fullsized":"2024-03-14T09:53:41.786998+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:26:20.354368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"85'8","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788428+0000","last_change":"2024-03-14T09:52:42.149962+0000","last_active":"2024-03-14T09:53:41.788428+0000","last_peered":"2024-03-14T09:53:41.788428+0000","last_clean":"2024-03-14T09:53:41.788428+0000","last_became_active":"2024-03-14T09:52:42.149726+0000","last_became_peered":"2024-03-14T09:52:42.149726+0000","last_unstale":"2024-03-14T09:53:41.788428+0000","last_undegraded":"2024-03-14T09:53:41.788428+0000","last_fullsized":"2024-03-14T09:53:41.788428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:20:18.294456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"85'10","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789837+0000","last_change":"2024-03-14T09:52:05.742945+0000","last_active":"2024-03-14T09:53:41.789837+0000","last_peered":"2024-03-14T09:53:41.789837+0000","last_clean":"2024-03-14T09:53:41.789837+0000","last_became_active":"2024-03-14T09:52:05.742483+0000","last_became_peered":"2024-03-14T09:52:05.742483+0000","last_unstale":"2024-03-14T09:53:41.789837+0000","last_undegraded":"2024-03-14T09:53:41.789837+0000","last_fullsized":"2024-03-14T09:53:41.789837+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:05:31.053608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"85'11","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790355+0000","last_change":"2024-03-14T09:52:05.737822+0000","last_active":"2024-03-14T09:53:41.790355+0000","last_peered":"2024-03-14T09:53:41.790355+0000","last_clean":"2024-03-14T09:53:41.790355+0000","last_became_active":"2024-03-14T09:52:05.735294+0000","last_became_peered":"2024-03-14T09:52:05.735294+0000","last_unstale":"2024-03-14T09:53:41.790355+0000","last_undegraded":"2024-03-14T09:53:41.790355+0000","last_fullsized":"2024-03-14T09:53:41.790355+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:06:28.437259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899359+0000","last_change":"2024-03-14T09:52:05.746991+0000","last_active":"2024-03-14T09:53:41.899359+0000","last_peered":"2024-03-14T09:53:41.899359+0000","last_clean":"2024-03-14T09:53:41.899359+0000","last_became_active":"2024-03-14T09:52:05.746558+0000","last_became_peered":"2024-03-14T09:52:05.746558+0000","last_unstale":"2024-03-14T09:53:41.899359+0000","last_undegraded":"2024-03-14T09:53:41.899359+0000","last_fullsized":"2024-03-14T09:53:41.899359+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:46:11.540134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247929,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71351,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":527,"num_read_kb":343,"num_write":168,"num_write_kb":127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":694,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":217357,"data_compressed":4812,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":147,"ondisk_log_size":147,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579936,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":489449,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.627,"15min":0.61599999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.681,"5min":2.681,"15min":2.681},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.59999999999999998,"15min":0.60899999999999999},"min":{"1min":0.379,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.64700000000000002,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.496}]},{"osd":1,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.64900000000000002,"15min":0.64400000000000002},"min":{"1min":0.57099999999999995,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.624},{"interface":"front","average":{"1min":0.623,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77300000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.47899999999999998}]},{"osd":2,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51300000000000001,"15min":0.51100000000000001},"min":{"1min":0.39900000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.77900000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.83099999999999996,"5min":0.88,"15min":0.88},"last":0.371}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.623,"15min":0.61699999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.68000000000000005},"min":{"1min":0.45800000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.81499999999999995,"5min":1.325,"15min":1.325},"last":0.64100000000000001}]},{"osd":5,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.748,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.44},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63800000000000001,"15min":0.64800000000000002},"min":{"1min":0.44600000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.77800000000000002,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.51100000000000001}]},{"osd":6,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70899999999999996,"15min":0.71499999999999997},"min":{"1min":0.53500000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97999999999999998,"5min":1.135,"15min":1.135},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.79800000000000004,"15min":0.79100000000000004},"min":{"1min":0.61799999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.7000000000000002,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.752,"15min":0.75800000000000001},"min":{"1min":0.56599999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.88800000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.75800000000000001,"15min":0.75700000000000001},"min":{"1min":0.57099999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.874,"5min":1.363,"15min":1.363},"last":0.69399999999999995}]},{"osd":8,"last update":"Thu Mar 14 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.73499999999999999}]},{"osd":9,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.81699999999999995,"15min":0.82499999999999996},"min":{"1min":0.626,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.96499999999999997,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.874,"15min":0.874},"min":{"1min":0.66400000000000003,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.66000000000000003}]},{"osd":10,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.81699999999999995,"15min":0.82099999999999995},"min":{"1min":0.65500000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.92500000000000004,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.78100000000000003,"15min":0.78500000000000003},"min":{"1min":0.60999999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.93500000000000005,"5min":1.448,"15min":1.448},"last":0.63300000000000001}]}]},{"osd":10,"up_from":67,"seq":287762808867,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":1144316,"data_compressed":61074,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46200000000000002,"15min":0.46400000000000002},"min":{"1min":0.32500000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46000000000000002,"15min":0.45900000000000002},"min":{"1min":0.29899999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.35599999999999998}]},{"osd":2,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57799999999999996,"15min":0.57999999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79400000000000004,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60199999999999998,"15min":0.60299999999999998},"min":{"1min":0.441,"5min":0.309,"15min":0.309},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.52300000000000002}]},{"osd":3,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.73399999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.63900000000000001,"15min":0.64100000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.80400000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.375,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.75800000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55500000000000005,"15min":0.56100000000000005},"min":{"1min":0.39200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77600000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.624,"15min":0.621},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.42299999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.622}]},{"osd":6,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.752,"15min":0.76100000000000001},"min":{"1min":0.51000000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0089999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.499,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.95899999999999996,"5min":1.05,"15min":1.05},"last":0.67600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":8,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.74099999999999999,"15min":0.74199999999999999},"min":{"1min":0.47499999999999998,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.75,"15min":0.75600000000000001},"min":{"1min":0.52300000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0009999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.56499999999999995}]},{"osd":9,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.79500000000000004,"15min":0.79900000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.76700000000000002,"15min":0.77400000000000002},"min":{"1min":0.55000000000000004,"5min":0.502,"15min":0.502},"max":{"1min":1.014,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.59499999999999997}]},{"osd":11,"last update":"Thu Mar 14 09:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.748,"15min":0.75},"min":{"1min":0.56399999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.754,"5min":0.75800000000000001,"15min":0.75900000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.73199999999999998}]}]},{"osd":9,"up_from":62,"seq":266287972392,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":493199,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.56899999999999995,"15min":0.57699999999999996},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.376},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.34999999999999998}]},{"osd":1,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68400000000000005,"15min":0.68000000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.504,"5min":2.504,"15min":2.504},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.73499999999999999,"15min":0.74299999999999999},"min":{"1min":0.50800000000000001,"5min":0.432,"15min":0.432},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":0.76800000000000002}]},{"osd":2,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.54200000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":2.3380000000000001,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.78000000000000003,"15min":0.79100000000000004},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.2360000000000002,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.71199999999999997,"15min":0.72499999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.0649999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.67600000000000005,"15min":0.68400000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.499}]},{"osd":4,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.81000000000000005,"15min":0.82399999999999995},"min":{"1min":0.52500000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.79700000000000004,"15min":0.81000000000000005},"min":{"1min":0.53000000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.54300000000000004}]},{"osd":5,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.85099999999999998,"15min":0.86699999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.83599999999999997,"15min":0.84199999999999997},"min":{"1min":0.57799999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":0.87}]},{"osd":6,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.88600000000000001,"15min":0.90600000000000003},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.90500000000000003,"15min":0.92100000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.69399999999999995}]},{"osd":8,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.92100000000000004,"15min":0.94099999999999995},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.93600000000000005,"15min":0.95399999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":0.84699999999999998}]},{"osd":10,"last update":"Thu Mar 14 09:53:09 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.88,"15min":0.89000000000000001},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.88500000000000001,"15min":0.89300000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":0.752}]},{"osd":11,"last update":"Thu Mar 14 09:53:09 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.569,"5min":2.569,"15min":2.569},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":2.266,"5min":2.266,"15min":2.266},"last":0.85599999999999998}]}]},{"osd":8,"up_from":55,"seq":236223201322,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":488692,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57099999999999995,"15min":0.58199999999999996},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66700000000000004,"5min":0.872,"15min":0.872},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.92500000000000004,"15min":1.081},"min":{"1min":0.38800000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80900000000000005,"5min":9.7769999999999992,"15min":9.7769999999999992},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.38600000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.748,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.59399999999999997,"15min":0.57399999999999995},"min":{"1min":0.46400000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.78900000000000003,"5min":0.877,"15min":0.877},"last":0.65900000000000003}]},{"osd":2,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.44900000000000001,"15min":0.432},"min":{"1min":0.33000000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46500000000000002,"15min":0.45200000000000001},"min":{"1min":0.34699999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.71399999999999997,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.42099999999999999}]},{"osd":3,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61199999999999999,"15min":0.60599999999999998},"min":{"1min":0.46200000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80400000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.64100000000000001,"15min":0.623},"min":{"1min":0.47699999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59799999999999998,"15min":0.57599999999999996},"min":{"1min":0.52100000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82199999999999995,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.61499999999999999,"15min":0.59099999999999997},"min":{"1min":0.501,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84099999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.60399999999999998}]},{"osd":5,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.36199999999999999,"5min":0.312,"15min":0.312},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54100000000000004,"15min":0.52200000000000002},"min":{"1min":0.41299999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.67100000000000004}]},{"osd":6,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.68600000000000005,"15min":0.66800000000000004},"min":{"1min":0.63300000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.755},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.66700000000000004,"15min":0.65300000000000002},"min":{"1min":0.50900000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.64700000000000002}]},{"osd":7,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.65200000000000002,"15min":0.628},"min":{"1min":0.55600000000000005,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.69299999999999995,"15min":0.67400000000000004},"min":{"1min":0.60899999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.69899999999999995}]},{"osd":9,"last update":"Thu Mar 14 09:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.73299999999999998,"15min":0.72999999999999998},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.53800000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.745}]},{"osd":10,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.72499999999999998,"15min":0.71099999999999997},"min":{"1min":0.60999999999999999,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.58599999999999997,"5min":0.499,"15min":0.499},"max":{"1min":0.98099999999999998,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.72299999999999998}]},{"osd":11,"last update":"Thu Mar 14 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.76600000000000001}]}]},{"osd":7,"up_from":48,"seq":206158430255,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":1085098,"data_compressed":59470,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.40999999999999998}]},{"osd":1,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42999999999999999,"15min":0.42099999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.57999999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.42899999999999999,"15min":0.41299999999999998},"min":{"1min":0.28799999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.57399999999999995,"15min":0.56200000000000006},"min":{"1min":0.47599999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82299999999999995,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58199999999999996,"15min":0.56999999999999995},"min":{"1min":0.38900000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.748,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.52800000000000002}]},{"osd":3,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81200000000000006,"5min":0.872,"15min":0.872},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60399999999999998,"15min":0.59199999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82599999999999996,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.627}]},{"osd":4,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.502,"15min":0.48199999999999998},"min":{"1min":0.33700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.5,"15min":0.496},"min":{"1min":0.32700000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76600000000000001,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.60399999999999998,"15min":0.58099999999999996},"min":{"1min":0.47099999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86899999999999999,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60899999999999999}]},{"osd":6,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.65400000000000003,"15min":0.63800000000000001},"min":{"1min":0.51400000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.878,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.66100000000000003,"15min":0.65800000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85499999999999998,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.70899999999999996}]},{"osd":8,"last update":"Thu Mar 14 09:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.69999999999999996,"15min":0.69499999999999995},"min":{"1min":0.52800000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.95699999999999996,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.70099999999999996,"15min":0.70199999999999996},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.90100000000000002,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.65700000000000003}]},{"osd":9,"last update":"Thu Mar 14 09:53:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.75700000000000001,"15min":0.76300000000000001},"min":{"1min":0.55500000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.98299999999999998,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.77000000000000002,"15min":0.77400000000000002},"min":{"1min":0.61399999999999999,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.92300000000000004,"5min":1.008,"15min":1.008},"last":0.76500000000000001}]},{"osd":10,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.57099999999999995,"5min":0.434,"15min":0.434},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.72099999999999997,"15min":0.72499999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90400000000000003,"5min":1.044,"15min":1.044},"last":0.68999999999999995}]},{"osd":11,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.747},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.78700000000000003}]}]},{"osd":6,"up_from":43,"seq":184683593778,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":950272,"data_stored":484279,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.51200000000000001,"15min":0.48199999999999998},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.54,"5min":3.54,"15min":3.54},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.52300000000000002,"15min":0.497},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.9860000000000002,"5min":3.9860000000000002,"15min":3.9860000000000002},"last":0.47499999999999998}]},{"osd":1,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.70699999999999996,"15min":0.65900000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":3.7589999999999999,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":0.626},{"interface":"front","average":{"1min":0.75,"5min":0.64100000000000001,"15min":0.61099999999999999},"min":{"1min":0.52900000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.68799999999999994}]},{"osd":2,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.68400000000000005,"15min":0.64700000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":4.0190000000000001,"5min":4.0190000000000001,"15min":4.0190000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.72399999999999998,"15min":0.68600000000000005},"min":{"1min":0.46100000000000002,"5min":0.38,"15min":0.38},"max":{"1min":3.5790000000000002,"5min":3.5790000000000002,"15min":3.5790000000000002},"last":0.76000000000000001}]},{"osd":3,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.63800000000000001,"15min":0.60699999999999998},"min":{"1min":0.41999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.7959999999999998,"5min":3.7959999999999998,"15min":3.7959999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.55400000000000005,"15min":0.54600000000000004},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78300000000000003,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.499}]},{"osd":4,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.69599999999999995,"15min":0.65600000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.76700000000000002,"15min":0.72899999999999998},"min":{"1min":0.53000000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":3.8559999999999999,"5min":3.8559999999999999,"15min":3.8559999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.76500000000000001,"15min":0.77500000000000002},"min":{"1min":0.58799999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1120000000000001,"5min":3.1059999999999999,"15min":3.1059999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.746,"15min":0.72099999999999997},"min":{"1min":0.53500000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":4.0469999999999997,"5min":4.0469999999999997,"15min":4.0469999999999997},"last":0.69999999999999996}]},{"osd":7,"last update":"Thu Mar 14 09:53:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.71799999999999997,"15min":0.69599999999999995},"min":{"1min":0.59299999999999997,"5min":0.495,"15min":0.495},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.81599999999999995,"15min":0.78500000000000003},"min":{"1min":0.57099999999999995,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.8260000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.79300000000000004}]},{"osd":8,"last update":"Thu Mar 14 09:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.89300000000000002,"15min":0.91100000000000003},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":3.8860000000000001,"5min":3.8860000000000001,"15min":3.8860000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.79500000000000004,"15min":0.76800000000000002},"min":{"1min":0.54200000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":4.1079999999999997,"5min":4.1079999999999997,"15min":4.1079999999999997},"last":0.78200000000000003}]},{"osd":9,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.84999999999999998,"15min":0.86199999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.95299999999999996,"5min":1.119,"15min":1.119},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.79900000000000004,"15min":0.80500000000000005},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.77600000000000002}]},{"osd":10,"last update":"Thu Mar 14 09:53:14 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.84599999999999997,"15min":0.83499999999999996},"min":{"1min":0.68200000000000005,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.85399999999999998,"15min":0.82199999999999995},"min":{"1min":0.66400000000000003,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":4.1360000000000001,"5min":4.1360000000000001,"15min":4.1360000000000001},"last":0.749}]},{"osd":11,"last update":"Thu Mar 14 09:53:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":3.9489999999999998,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.82599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":551295,"data_compressed":25179,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.57099999999999995,"15min":0.56899999999999995},"min":{"1min":0.45000000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.877,"5min":1.917,"15min":1.917},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.66000000000000003,"15min":0.57299999999999995},"min":{"1min":0.437,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.877,"5min":7.6399999999999997,"15min":7.6399999999999997},"last":0.61099999999999999}]},{"osd":2,"last update":"Thu Mar 14 09:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56499999999999995,"15min":0.54900000000000004},"min":{"1min":0.42199999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76700000000000002,"5min":1.885,"15min":1.885},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61199999999999999,"15min":0.55000000000000004},"min":{"1min":0.438,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.88400000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"last":0.75}]},{"osd":3,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.59899999999999998,"15min":0.57699999999999996},"min":{"1min":0.39300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.1829999999999998,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.58699999999999997,"15min":0.57699999999999996},"min":{"1min":0.434,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.6299999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":0.51600000000000001}]},{"osd":4,"last update":"Thu Mar 14 09:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.39800000000000002,"15min":0.371},"min":{"1min":0.35799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.624,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.45900000000000002,"15min":0.40999999999999998},"min":{"1min":0.36399999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.64100000000000001,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Mar 14 09:53:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55800000000000005,"15min":0.496},"min":{"1min":0.40799999999999997,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70799999999999996,"5min":2.9849999999999999,"15min":2.9849999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.59499999999999997,"15min":0.53100000000000003},"min":{"1min":0.57199999999999995,"5min":0.309,"15min":0.309},"max":{"1min":0.86199999999999999,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.70999999999999996}]},{"osd":6,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.69899999999999995,"15min":0.64900000000000002},"min":{"1min":0.60199999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":3.7879999999999998,"5min":3.7879999999999998,"15min":3.7879999999999998},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.80400000000000005,"15min":0.85999999999999999},"min":{"1min":0.46500000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.5910000000000002,"5min":5.5229999999999997,"15min":5.5229999999999997},"last":0.72799999999999998}]},{"osd":7,"last update":"Thu Mar 14 09:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53200000000000003,"15min":0.496},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81499999999999995,"5min":2.331,"15min":2.331},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.50900000000000001,"15min":0.49099999999999999},"min":{"1min":0.39900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.753,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63500000000000001}]},{"osd":8,"last update":"Thu Mar 14 09:53:34 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.69899999999999995,"15min":0.629},"min":{"1min":0.54100000000000004,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.84199999999999997,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.68000000000000005,"15min":0.64700000000000002},"min":{"1min":0.57999999999999996,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.95399999999999996,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.76000000000000001}]},{"osd":9,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":0.76300000000000001,"15min":0.70599999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":6.5149999999999997,"5min":6.5149999999999997,"15min":6.5149999999999997},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.359,"5min":0.83799999999999997,"15min":0.751},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":9.8900000000000006,"5min":9.8900000000000006,"15min":9.8900000000000006},"last":0.73799999999999999}]},{"osd":10,"last update":"Thu Mar 14 09:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.68600000000000005,"15min":0.68200000000000005},"min":{"1min":0.61699999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.85999999999999999,"5min":0.877,"15min":0.877},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.73599999999999999,"15min":0.72999999999999998},"min":{"1min":0.64600000000000002,"5min":0.497,"15min":0.497},"max":{"1min":0.88500000000000001,"5min":1.171,"15min":1.171},"last":0.97699999999999998}]},{"osd":11,"last update":"Thu Mar 14 09:53:09 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.387,"5min":2.387,"15min":2.387},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.018}]}]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":485667,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58699999999999997,"15min":0.53500000000000003},"min":{"1min":0.40500000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85199999999999998,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.626,"15min":0.56100000000000005},"min":{"1min":0.42599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.998,"5min":2.056,"15min":2.056},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Mar 14 09:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.69199999999999995,"15min":0.57799999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.94799999999999995,"5min":5.1349999999999998,"15min":5.1349999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.68999999999999995,"15min":0.58999999999999997},"min":{"1min":0.52100000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0880000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":0.95699999999999996}]},{"osd":3,"last update":"Thu Mar 14 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":0.998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.1040000000000001}]},{"osd":4,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.71799999999999997,"15min":0.66800000000000004},"min":{"1min":0.47999999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.871,"5min":3.8220000000000001,"15min":3.8220000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.66600000000000004,"15min":0.59999999999999998},"min":{"1min":0.496,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":5.2949999999999999,"5min":5.2949999999999999,"15min":5.2949999999999999},"last":0.75800000000000001}]},{"osd":5,"last update":"Thu Mar 14 09:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.76500000000000001,"15min":0.68700000000000006},"min":{"1min":0.58099999999999996,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1279999999999999,"5min":3.863,"15min":3.863},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69599999999999995,"15min":0.67700000000000005},"min":{"1min":0.42999999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95799999999999996,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.56599999999999995}]},{"osd":6,"last update":"Thu Mar 14 09:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.65600000000000003,"15min":0.60999999999999999},"min":{"1min":0.41299999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2370000000000001,"5min":5.2370000000000001,"15min":5.2370000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.76500000000000001,"15min":0.72499999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.5890000000000004,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":1.2310000000000001}]},{"osd":7,"last update":"Thu Mar 14 09:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.4449999999999998,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.0669999999999999},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":3.544,"5min":3.544,"15min":3.544},"last":0.59499999999999997}]},{"osd":8,"last update":"Thu Mar 14 09:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.84799999999999998,"15min":0.83599999999999997},"min":{"1min":0.71499999999999997,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.385,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":1.1559999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.79200000000000004,"15min":0.79000000000000004},"min":{"1min":0.51500000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.92700000000000005,"5min":3.6709999999999998,"15min":3.6709999999999998},"last":0.625}]},{"osd":9,"last update":"Thu Mar 14 09:52:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.88600000000000001,"15min":0.83999999999999997},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":6.7699999999999996,"5min":6.7699999999999996,"15min":6.7699999999999996},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.2110000000000001,"5min":0.93100000000000005,"15min":0.88400000000000001},"min":{"1min":0.45400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":6.0199999999999996,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":1.1950000000000001}]},{"osd":10,"last update":"Thu Mar 14 09:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.82399999999999995,"15min":0.81200000000000006},"min":{"1min":0.627,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.6339999999999999,"5min":3.6339999999999999,"15min":3.6339999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.88,"15min":0.84399999999999997},"min":{"1min":0.68700000000000006,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":5.3399999999999999,"5min":5.3399999999999999,"15min":5.3399999999999999},"last":0.80900000000000005}]},{"osd":11,"last update":"Thu Mar 14 09:53:27 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.91200000000000003,"15min":0.90800000000000003},"min":{"1min":0.77100000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.329,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":1.2629999999999999},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.91700000000000004,"15min":0.92900000000000005},"min":{"1min":0.64300000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.1220000000000001,"5min":3.71,"15min":3.71},"last":0.73099999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":552373,"data_compressed":25179,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53300000000000003,"15min":0.51800000000000002},"min":{"1min":0.37,"5min":0.31,"15min":0.31},"max":{"1min":0.76800000000000002,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.5},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54900000000000004,"15min":0.53000000000000003},"min":{"1min":0.435,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.83999999999999997,"5min":0.879,"15min":0.879},"last":0.68999999999999995}]},{"osd":1,"last update":"Thu Mar 14 09:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.59799999999999998,"15min":0.53900000000000003},"min":{"1min":0.44900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.91100000000000003,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56299999999999994,"15min":0.54400000000000004},"min":{"1min":0.44400000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.70199999999999996,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.57899999999999996}]},{"osd":3,"last update":"Thu Mar 14 09:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.55000000000000004,"15min":0.52600000000000002},"min":{"1min":0.42599999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56999999999999995,"15min":0.54900000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.66200000000000003}]},{"osd":4,"last update":"Thu Mar 14 09:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.624}]},{"osd":5,"last update":"Thu Mar 14 09:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.48699999999999999,"15min":0.45100000000000001},"min":{"1min":0.41699999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.82899999999999996,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.50600000000000001,"15min":0.48199999999999998},"min":{"1min":0.32800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.76100000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.52300000000000002}]},{"osd":6,"last update":"Thu Mar 14 09:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.66100000000000003,"15min":0.65000000000000002},"min":{"1min":0.55100000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.67600000000000005,"15min":0.66300000000000003},"min":{"1min":0.56699999999999995,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.85799999999999998}]},{"osd":7,"last update":"Thu Mar 14 09:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.81000000000000005}]},{"osd":8,"last update":"Thu Mar 14 09:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.64500000000000002,"15min":0.628},"min":{"1min":0.505,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.66000000000000003,"15min":0.64300000000000002},"min":{"1min":0.41499999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.745}]},{"osd":9,"last update":"Thu Mar 14 09:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.63900000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.75700000000000001,"15min":0.746},"min":{"1min":0.59299999999999997,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.83399999999999996}]},{"osd":10,"last update":"Thu Mar 14 09:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73399999999999999,"15min":0.73299999999999998},"min":{"1min":0.56899999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.91800000000000004,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.76400000000000001,"15min":0.754},"min":{"1min":0.59799999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1,"5min":1.008,"15min":1.008},"last":0.872}]},{"osd":11,"last update":"Thu Mar 14 09:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.84799999999999998}]}]},{"osd":3,"up_from":24,"seq":103079215167,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":484473,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.42199999999999999,"15min":0.40100000000000002},"min":{"1min":0.32300000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.44600000000000001,"15min":0.42999999999999999},"min":{"1min":0.373,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.66900000000000004}]},{"osd":1,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61099999999999999,"15min":0.59099999999999997},"min":{"1min":0.39400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.82799999999999996,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60699999999999998,"15min":0.58499999999999996},"min":{"1min":0.437,"5min":0.374,"15min":0.374},"max":{"1min":0.76300000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.60299999999999998}]},{"osd":2,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63400000000000001,"15min":0.60699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84099999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64000000000000001,"15min":0.623},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.80400000000000005,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.80300000000000005}]},{"osd":4,"last update":"Thu Mar 14 09:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.624,"15min":0.59799999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77200000000000002,"5min":1.091,"15min":1.091},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.629,"15min":0.60399999999999998},"min":{"1min":0.54000000000000004,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81999999999999995,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.50600000000000001}]},{"osd":5,"last update":"Thu Mar 14 09:53:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.67600000000000005,"15min":0.67900000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83799999999999997,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.46400000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.80200000000000005,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.78000000000000003}]},{"osd":6,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64800000000000002,"15min":0.65000000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86599999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65800000000000003,"15min":0.64300000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.98799999999999999,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.63500000000000001}]},{"osd":7,"last update":"Thu Mar 14 09:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.72999999999999998,"15min":0.73799999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.88900000000000001,"5min":1.123,"15min":1.123},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.748,"5min":0.76600000000000001,"15min":0.76800000000000002},"min":{"1min":0.64100000000000001,"5min":0.498,"15min":0.498},"max":{"1min":0.89200000000000002,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.753}]},{"osd":8,"last update":"Thu Mar 14 09:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.871}]},{"osd":9,"last update":"Thu Mar 14 09:52:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.80000000000000004,"15min":0.81499999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.98099999999999998,"5min":1.097,"15min":1.097},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.83999999999999997,"5min":1.121,"15min":1.121},"last":0.83399999999999996}]},{"osd":10,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.86899999999999999,"15min":0.88500000000000001},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":0.90200000000000002,"5min":1.127,"15min":1.127},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.83099999999999996,"15min":0.83899999999999997},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.97699999999999998,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.79100000000000004}]},{"osd":11,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.86399999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986234,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":489724,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.40100000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71599999999999997,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.52300000000000002,"15min":0.51600000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:53:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.36599999999999999,"15min":0.35399999999999998},"min":{"1min":0.27000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.37,"15min":0.36399999999999999},"min":{"1min":0.33000000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.51300000000000001,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.61099999999999999}]},{"osd":2,"last update":"Thu Mar 14 09:53:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56499999999999995,"15min":0.53800000000000003},"min":{"1min":0.38300000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67200000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.55600000000000005,"15min":0.52400000000000002},"min":{"1min":0.43099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.69199999999999995}]},{"osd":3,"last update":"Thu Mar 14 09:53:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.44,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38500000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.748,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.67800000000000005}]},{"osd":5,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59499999999999997,"15min":0.53300000000000003},"min":{"1min":0.44600000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78300000000000003,"5min":4.3479999999999999,"15min":4.3479999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55000000000000004,"15min":0.53100000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.73699999999999999,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.72399999999999998}]},{"osd":6,"last update":"Thu Mar 14 09:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.48099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73499999999999999,"5min":0.876,"15min":0.876},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.60899999999999999,"15min":0.59999999999999998},"min":{"1min":0.42799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.80500000000000005,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.85799999999999998}]},{"osd":7,"last update":"Thu Mar 14 09:53:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.50800000000000001,"15min":0.48799999999999999},"min":{"1min":0.34499999999999997,"5min":0.318,"15min":0.318},"max":{"1min":0.72499999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.47199999999999998,"15min":0.45600000000000002},"min":{"1min":0.36599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.71299999999999997}]},{"osd":8,"last update":"Thu Mar 14 09:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.65600000000000003,"15min":0.63700000000000001},"min":{"1min":0.57499999999999996,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.623,"15min":0.60499999999999998},"min":{"1min":0.47599999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82699999999999996,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.873}]},{"osd":9,"last update":"Thu Mar 14 09:52:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.47599999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.75,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71299999999999997,"15min":0.70899999999999996},"min":{"1min":0.61899999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.84099999999999997,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.96899999999999997}]},{"osd":10,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.63300000000000001,"5min":0.502,"15min":0.502},"max":{"1min":0.82799999999999996,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66500000000000004,"15min":0.66300000000000003},"min":{"1min":0.502,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.77900000000000003,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.90500000000000003}]},{"osd":11,"last update":"Thu Mar 14 09:53:07 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.84599999999999997}]}]},{"osd":5,"up_from":36,"seq":154618822711,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":1080081,"data_compressed":59470,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54500000000000004,"15min":0.53000000000000003},"min":{"1min":0.40400000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70799999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.372,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.625,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.48399999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.51900000000000002},"min":{"1min":0.372,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81899999999999995,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56599999999999995,"15min":0.52500000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75600000000000001,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.57299999999999995}]},{"osd":2,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.41599999999999998,"15min":0.38600000000000001},"min":{"1min":0.27300000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.70699999999999996,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.42599999999999999,"15min":0.39000000000000001},"min":{"1min":0.248,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.67400000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71199999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.58799999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.77700000000000002,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56399999999999995}]},{"osd":4,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59399999999999997,"15min":0.56000000000000005},"min":{"1min":0.48699999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80200000000000005,"5min":0.87,"15min":0.87},"last":0.626},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56299999999999994,"15min":0.53900000000000003},"min":{"1min":0.42699999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.76300000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.53900000000000003}]},{"osd":6,"last update":"Thu Mar 14 09:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.628,"15min":0.626},"min":{"1min":0.51800000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.79100000000000004,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.64000000000000001,"15min":0.627},"min":{"1min":0.44600000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.82799999999999996,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.52400000000000002}]},{"osd":7,"last update":"Thu Mar 14 09:53:16 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.63300000000000001,"15min":0.60099999999999998},"min":{"1min":0.48399999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.63,"15min":0.59599999999999997},"min":{"1min":0.56299999999999994,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.89600000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.64200000000000002}]},{"osd":8,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60399999999999998,"15min":0.59399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56499999999999995,"15min":0.56399999999999995},"min":{"1min":0.33400000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70799999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.58299999999999996}]},{"osd":9,"last update":"Thu Mar 14 09:52:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.72099999999999997,"15min":0.71199999999999997},"min":{"1min":0.56799999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96299999999999997,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.74299999999999999,"15min":0.73699999999999999},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.68600000000000005}]},{"osd":10,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.69999999999999996,"15min":0.69599999999999995},"min":{"1min":0.58699999999999997,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.75800000000000001,"15min":0.753},"min":{"1min":0.60899999999999999,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.70099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34227,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34227,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":69626,"data_compressed":1604,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68184,"data_compressed":1604,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":69759,"data_compressed":1604,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-14T09:53:56.109 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-14T09:53:56.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.335+0000 7f8809200700 1 -- 172.21.15.78:0/2798971204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8804101e40 msgr2=0x7f88041022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:56.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.335+0000 7f8809200700 1 --2- 172.21.15.78:0/2798971204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8804101e40 0x7f88041022a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f87f00099c0 tx=0x7f87f0009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.335+0000 7f8809200700 1 -- 172.21.15.78:0/2798971204 shutdown_connections 2024-03-14T09:53:56.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.335+0000 7f8809200700 1 --2- 172.21.15.78:0/2798971204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88041027e0 0x7f8804109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.335+0000 7f8809200700 1 --2- 172.21.15.78:0/2798971204 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8804100c50 0x7f8804101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.335+0000 7f8809200700 1 --2- 172.21.15.78:0/2798971204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8804101e40 0x7f88041022a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.335+0000 7f8809200700 1 -- 172.21.15.78:0/2798971204 >> 172.21.15.78:0/2798971204 conn(0x7f88040fc1c0 msgr2=0x7f88040fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:56.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.335+0000 7f8809200700 1 -- 172.21.15.78:0/2798971204 shutdown_connections 2024-03-14T09:53:56.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.335+0000 7f8809200700 1 -- 172.21.15.78:0/2798971204 wait complete. 2024-03-14T09:53:56.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.336+0000 7f8809200700 1 Processor -- start 2024-03-14T09:53:56.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.336+0000 7f8809200700 1 -- start start 2024-03-14T09:53:56.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.336+0000 7f8809200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8804100c50 0x7f8804199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:56.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.336+0000 7f8809200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8804101e40 0x7f8804199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:56.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.336+0000 7f8809200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88041027e0 0x7f88041a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:56.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.336+0000 7f8809200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88041a0d50 con 0x7f8804100c50 2024-03-14T09:53:56.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.336+0000 7f8809200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88041a0e90 con 0x7f8804101e40 2024-03-14T09:53:56.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.336+0000 7f8809200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88041a1020 con 0x7f88041027e0 2024-03-14T09:53:56.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.337+0000 7f8801e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8804100c50 0x7f8804199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:56.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.337+0000 7f8801e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8804100c50 0x7f8804199220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43926/0 (socket says 172.21.15.78:43926) 2024-03-14T09:53:56.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.337+0000 7f8801e00700 1 -- 172.21.15.78:0/838847457 learned_addr learned my addr 172.21.15.78:0/838847457 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:56.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.337+0000 7f8801400700 1 --2- 172.21.15.78:0/838847457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8804101e40 0x7f8804199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:56.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.337+0000 7f8802800700 1 --2- 172.21.15.78:0/838847457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88041027e0 0x7f88041a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:56.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.337+0000 7f8801e00700 1 -- 172.21.15.78:0/838847457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88041027e0 msgr2=0x7f88041a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:56.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.337+0000 7f8801e00700 1 --2- 172.21.15.78:0/838847457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88041027e0 0x7f88041a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.337+0000 7f8801e00700 1 -- 172.21.15.78:0/838847457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8804101e40 msgr2=0x7f8804199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:56.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.337+0000 7f8801e00700 1 --2- 172.21.15.78:0/838847457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8804101e40 0x7f8804199760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.337+0000 7f8801e00700 1 -- 172.21.15.78:0/838847457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f00095f0 con 0x7f8804100c50 2024-03-14T09:53:56.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.337+0000 7f8801e00700 1 --2- 172.21.15.78:0/838847457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8804100c50 0x7f8804199220 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f87f400e970 tx=0x7f87f400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:56.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.338+0000 7f87faa00700 1 -- 172.21.15.78:0/838847457 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f4009790 con 0x7f8804100c50 2024-03-14T09:53:56.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.338+0000 7f8809200700 1 -- 172.21.15.78:0/838847457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88041a1300 con 0x7f8804100c50 2024-03-14T09:53:56.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.338+0000 7f87faa00700 1 -- 172.21.15.78:0/838847457 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f87f400c7e0 con 0x7f8804100c50 2024-03-14T09:53:56.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.338+0000 7f87faa00700 1 -- 172.21.15.78:0/838847457 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f4003c10 con 0x7f8804100c50 2024-03-14T09:53:56.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.338+0000 7f8809200700 1 -- 172.21.15.78:0/838847457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88041a1810 con 0x7f8804100c50 2024-03-14T09:53:56.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.339+0000 7f8809200700 1 -- 172.21.15.78:0/838847457 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88040679e0 con 0x7f8804100c50 2024-03-14T09:53:56.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.341+0000 7f87faa00700 1 -- 172.21.15.78:0/838847457 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f87f400cc50 con 0x7f8804100c50 2024-03-14T09:53:56.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.342+0000 7f87faa00700 1 --2- 172.21.15.78:0/838847457 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87e80a0b20 0x7f87e80a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:56.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.342+0000 7f87faa00700 1 -- 172.21.15.78:0/838847457 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f87f4014070 con 0x7f8804100c50 2024-03-14T09:53:56.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.342+0000 7f8801400700 1 --2- 172.21.15.78:0/838847457 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87e80a0b20 0x7f87e80a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:56.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.343+0000 7f8801400700 1 --2- 172.21.15.78:0/838847457 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87e80a0b20 0x7f87e80a2fe0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f87f0003730 tx=0x7f87f000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:56.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.345+0000 7f87faa00700 1 -- 172.21.15.78:0/838847457 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f87f4095b80 con 0x7f8804100c50 2024-03-14T09:53:56.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:56 smithi078 ceph-mon[38143]: pgmap v294: 129 pgs: 129 active+clean; 646 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-14T09:53:56.390 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:56 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3522177793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:56.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:56 smithi152 ceph-mon[48279]: pgmap v294: 129 pgs: 129 active+clean; 646 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-14T09:53:56.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:56 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3522177793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:56.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:56 smithi132 ceph-mon[47959]: pgmap v294: 129 pgs: 129 active+clean; 646 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-14T09:53:56.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:56 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3522177793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:53:56.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.564+0000 7f8809200700 1 -- 172.21.15.78:0/838847457 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8804074700 con 0x7f87e80a0b20 2024-03-14T09:53:56.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.577+0000 7f87faa00700 1 -- 172.21.15.78:0/838847457 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427290 (secure 0 0 0) 0x7f8804074700 con 0x7f87e80a0b20 2024-03-14T09:53:56.584 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.582+0000 7f8809200700 1 -- 172.21.15.78:0/838847457 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87e80a0b20 msgr2=0x7f87e80a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.582+0000 7f8809200700 1 --2- 172.21.15.78:0/838847457 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87e80a0b20 0x7f87e80a2fe0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f87f0003730 tx=0x7f87f000b540 comp rx=0 tx=0).stop 2024-03-14T09:53:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.582+0000 7f8809200700 1 -- 172.21.15.78:0/838847457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8804100c50 msgr2=0x7f8804199220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.582+0000 7f8809200700 1 --2- 172.21.15.78:0/838847457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8804100c50 0x7f8804199220 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f87f400e970 tx=0x7f87f400ec80 comp rx=0 tx=0).stop 2024-03-14T09:53:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.582+0000 7f8809200700 1 -- 172.21.15.78:0/838847457 shutdown_connections 2024-03-14T09:53:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.582+0000 7f8809200700 1 --2- 172.21.15.78:0/838847457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f88041027e0 0x7f88041a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.582+0000 7f8809200700 1 --2- 172.21.15.78:0/838847457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8804100c50 0x7f8804199220 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.582+0000 7f8809200700 1 --2- 172.21.15.78:0/838847457 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f87e80a0b20 0x7f87e80a2fe0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.582+0000 7f8809200700 1 --2- 172.21.15.78:0/838847457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8804101e40 0x7f8804199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.582+0000 7f8809200700 1 -- 172.21.15.78:0/838847457 >> 172.21.15.78:0/838847457 conn(0x7f88040fc1c0 msgr2=0x7f8804072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:56.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.583+0000 7f8809200700 1 -- 172.21.15.78:0/838847457 shutdown_connections 2024-03-14T09:53:56.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.583+0000 7f8809200700 1 -- 172.21.15.78:0/838847457 wait complete. 2024-03-14T09:53:56.588 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-14T09:53:56.602 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-03-14T09:53:55.097758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661953,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":663,"num_read_kb":462,"num_write":351,"num_write_kb":3067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1181430,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":786,"ondisk_log_size":786,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":335892,"kb_used_data":11680,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124605932,"statfs":{"total":1151940427776,"available":1151596474368,"internally_reserved":0,"allocated":11960320,"data_stored":7828646,"data_compressed":395397,"data_compressed_allocated":540672,"data_compressed_original":2580480,"omap_allocated":19131,"internal_metadata":331855173},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5716,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":85,"num_read_kb":54,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003232"},"pg_stats":[{"pgid":"3.3f","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899578+0000","last_change":"2024-03-14T09:52:05.737617+0000","last_active":"2024-03-14T09:53:41.899578+0000","last_peered":"2024-03-14T09:53:41.899578+0000","last_clean":"2024-03-14T09:53:41.899578+0000","last_became_active":"2024-03-14T09:52:05.736796+0000","last_became_peered":"2024-03-14T09:52:05.736796+0000","last_unstale":"2024-03-14T09:53:41.899578+0000","last_undegraded":"2024-03-14T09:53:41.899578+0000","last_fullsized":"2024-03-14T09:53:41.899578+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:47:52.437748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"85'4","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899167+0000","last_change":"2024-03-14T09:52:05.749474+0000","last_active":"2024-03-14T09:53:41.899167+0000","last_peered":"2024-03-14T09:53:41.899167+0000","last_clean":"2024-03-14T09:53:41.899167+0000","last_became_active":"2024-03-14T09:52:05.748927+0000","last_became_peered":"2024-03-14T09:52:05.748927+0000","last_unstale":"2024-03-14T09:53:41.899167+0000","last_undegraded":"2024-03-14T09:53:41.899167+0000","last_fullsized":"2024-03-14T09:53:41.899167+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:28:15.852021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899117+0000","last_change":"2024-03-14T09:52:05.745098+0000","last_active":"2024-03-14T09:53:41.899117+0000","last_peered":"2024-03-14T09:53:41.899117+0000","last_clean":"2024-03-14T09:53:41.899117+0000","last_became_active":"2024-03-14T09:52:05.744755+0000","last_became_peered":"2024-03-14T09:52:05.744755+0000","last_unstale":"2024-03-14T09:53:41.899117+0000","last_undegraded":"2024-03-14T09:53:41.899117+0000","last_fullsized":"2024-03-14T09:53:41.899117+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:35:41.932386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786320+0000","last_change":"2024-03-14T09:52:05.737429+0000","last_active":"2024-03-14T09:53:41.786320+0000","last_peered":"2024-03-14T09:53:41.786320+0000","last_clean":"2024-03-14T09:53:41.786320+0000","last_became_active":"2024-03-14T09:52:05.736516+0000","last_became_peered":"2024-03-14T09:52:05.736516+0000","last_unstale":"2024-03-14T09:53:41.786320+0000","last_undegraded":"2024-03-14T09:53:41.786320+0000","last_fullsized":"2024-03-14T09:53:41.786320+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:48:38.158783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"85'4","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788000+0000","last_change":"2024-03-14T09:52:05.729634+0000","last_active":"2024-03-14T09:53:41.788000+0000","last_peered":"2024-03-14T09:53:41.788000+0000","last_clean":"2024-03-14T09:53:41.788000+0000","last_became_active":"2024-03-14T09:52:05.727889+0000","last_became_peered":"2024-03-14T09:52:05.727889+0000","last_unstale":"2024-03-14T09:53:41.788000+0000","last_undegraded":"2024-03-14T09:53:41.788000+0000","last_fullsized":"2024-03-14T09:53:41.788000+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:01:13.216763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"85'6","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787456+0000","last_change":"2024-03-14T09:52:42.147610+0000","last_active":"2024-03-14T09:53:41.787456+0000","last_peered":"2024-03-14T09:53:41.787456+0000","last_clean":"2024-03-14T09:53:41.787456+0000","last_became_active":"2024-03-14T09:52:42.147262+0000","last_became_peered":"2024-03-14T09:52:42.147262+0000","last_unstale":"2024-03-14T09:53:41.787456+0000","last_undegraded":"2024-03-14T09:53:41.787456+0000","last_fullsized":"2024-03-14T09:53:41.787456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:20:49.855564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783351+0000","last_change":"2024-03-14T09:52:05.727365+0000","last_active":"2024-03-14T09:53:41.783351+0000","last_peered":"2024-03-14T09:53:41.783351+0000","last_clean":"2024-03-14T09:53:41.783351+0000","last_became_active":"2024-03-14T09:52:05.726984+0000","last_became_peered":"2024-03-14T09:52:05.726984+0000","last_unstale":"2024-03-14T09:53:41.783351+0000","last_undegraded":"2024-03-14T09:53:41.783351+0000","last_fullsized":"2024-03-14T09:53:41.783351+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:42:57.129718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785018+0000","last_change":"2024-03-14T09:52:05.729622+0000","last_active":"2024-03-14T09:53:41.785018+0000","last_peered":"2024-03-14T09:53:41.785018+0000","last_clean":"2024-03-14T09:53:41.785018+0000","last_became_active":"2024-03-14T09:52:05.729293+0000","last_became_peered":"2024-03-14T09:52:05.729293+0000","last_unstale":"2024-03-14T09:53:41.785018+0000","last_undegraded":"2024-03-14T09:53:41.785018+0000","last_fullsized":"2024-03-14T09:53:41.785018+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:13:20.875857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"85'11","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790229+0000","last_change":"2024-03-14T09:52:05.741620+0000","last_active":"2024-03-14T09:53:41.790229+0000","last_peered":"2024-03-14T09:53:41.790229+0000","last_clean":"2024-03-14T09:53:41.790229+0000","last_became_active":"2024-03-14T09:52:05.734079+0000","last_became_peered":"2024-03-14T09:52:05.734079+0000","last_unstale":"2024-03-14T09:53:41.790229+0000","last_undegraded":"2024-03-14T09:53:41.790229+0000","last_fullsized":"2024-03-14T09:53:41.790229+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:02:35.529671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"85'16","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788064+0000","last_change":"2024-03-14T09:52:05.743424+0000","last_active":"2024-03-14T09:53:41.788064+0000","last_peered":"2024-03-14T09:53:41.788064+0000","last_clean":"2024-03-14T09:53:41.788064+0000","last_became_active":"2024-03-14T09:52:05.743100+0000","last_became_peered":"2024-03-14T09:52:05.743100+0000","last_unstale":"2024-03-14T09:53:41.788064+0000","last_undegraded":"2024-03-14T09:53:41.788064+0000","last_fullsized":"2024-03-14T09:53:41.788064+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:32:33.492163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783875+0000","last_change":"2024-03-14T09:52:05.735654+0000","last_active":"2024-03-14T09:53:41.783875+0000","last_peered":"2024-03-14T09:53:41.783875+0000","last_clean":"2024-03-14T09:53:41.783875+0000","last_became_active":"2024-03-14T09:52:05.735371+0000","last_became_peered":"2024-03-14T09:52:05.735371+0000","last_unstale":"2024-03-14T09:53:41.783875+0000","last_undegraded":"2024-03-14T09:53:41.783875+0000","last_fullsized":"2024-03-14T09:53:41.783875+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:19:50.011554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"85'7","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789097+0000","last_change":"2024-03-14T09:52:42.146700+0000","last_active":"2024-03-14T09:53:41.789097+0000","last_peered":"2024-03-14T09:53:41.789097+0000","last_clean":"2024-03-14T09:53:41.789097+0000","last_became_active":"2024-03-14T09:52:42.145377+0000","last_became_peered":"2024-03-14T09:52:42.145377+0000","last_unstale":"2024-03-14T09:53:41.789097+0000","last_undegraded":"2024-03-14T09:53:41.789097+0000","last_fullsized":"2024-03-14T09:53:41.789097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:35:52.735091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"85'10","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790265+0000","last_change":"2024-03-14T09:52:05.740209+0000","last_active":"2024-03-14T09:53:41.790265+0000","last_peered":"2024-03-14T09:53:41.790265+0000","last_clean":"2024-03-14T09:53:41.790265+0000","last_became_active":"2024-03-14T09:52:05.732949+0000","last_became_peered":"2024-03-14T09:52:05.732949+0000","last_unstale":"2024-03-14T09:53:41.790265+0000","last_undegraded":"2024-03-14T09:53:41.790265+0000","last_fullsized":"2024-03-14T09:53:41.790265+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:15:56.377569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.898939+0000","last_change":"2024-03-14T09:52:05.748795+0000","last_active":"2024-03-14T09:53:41.898939+0000","last_peered":"2024-03-14T09:53:41.898939+0000","last_clean":"2024-03-14T09:53:41.898939+0000","last_became_active":"2024-03-14T09:52:05.746107+0000","last_became_peered":"2024-03-14T09:52:05.746107+0000","last_unstale":"2024-03-14T09:53:41.898939+0000","last_undegraded":"2024-03-14T09:53:41.898939+0000","last_fullsized":"2024-03-14T09:53:41.898939+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:49:07.932854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.898932+0000","last_change":"2024-03-14T09:52:05.744174+0000","last_active":"2024-03-14T09:53:41.898932+0000","last_peered":"2024-03-14T09:53:41.898932+0000","last_clean":"2024-03-14T09:53:41.898932+0000","last_became_active":"2024-03-14T09:52:05.743848+0000","last_became_peered":"2024-03-14T09:52:05.743848+0000","last_unstale":"2024-03-14T09:53:41.898932+0000","last_undegraded":"2024-03-14T09:53:41.898932+0000","last_fullsized":"2024-03-14T09:53:41.898932+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:07:23.260734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789366+0000","last_change":"2024-03-14T09:52:05.733677+0000","last_active":"2024-03-14T09:53:41.789366+0000","last_peered":"2024-03-14T09:53:41.789366+0000","last_clean":"2024-03-14T09:53:41.789366+0000","last_became_active":"2024-03-14T09:52:05.730967+0000","last_became_peered":"2024-03-14T09:52:05.730967+0000","last_unstale":"2024-03-14T09:53:41.789366+0000","last_undegraded":"2024-03-14T09:53:41.789366+0000","last_fullsized":"2024-03-14T09:53:41.789366+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:33:53.484741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899444+0000","last_change":"2024-03-14T09:52:05.737836+0000","last_active":"2024-03-14T09:53:41.899444+0000","last_peered":"2024-03-14T09:53:41.899444+0000","last_clean":"2024-03-14T09:53:41.899444+0000","last_became_active":"2024-03-14T09:52:05.737257+0000","last_became_peered":"2024-03-14T09:52:05.737257+0000","last_unstale":"2024-03-14T09:53:41.899444+0000","last_undegraded":"2024-03-14T09:53:41.899444+0000","last_fullsized":"2024-03-14T09:53:41.899444+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:53.723415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"85'10","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789627+0000","last_change":"2024-03-14T09:52:05.741360+0000","last_active":"2024-03-14T09:53:41.789627+0000","last_peered":"2024-03-14T09:53:41.789627+0000","last_clean":"2024-03-14T09:53:41.789627+0000","last_became_active":"2024-03-14T09:52:05.740673+0000","last_became_peered":"2024-03-14T09:52:05.740673+0000","last_unstale":"2024-03-14T09:53:41.789627+0000","last_undegraded":"2024-03-14T09:53:41.789627+0000","last_fullsized":"2024-03-14T09:53:41.789627+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:13:41.561338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.784072+0000","last_change":"2024-03-14T09:52:05.739134+0000","last_active":"2024-03-14T09:53:41.784072+0000","last_peered":"2024-03-14T09:53:41.784072+0000","last_clean":"2024-03-14T09:53:41.784072+0000","last_became_active":"2024-03-14T09:52:05.738439+0000","last_became_peered":"2024-03-14T09:52:05.738439+0000","last_unstale":"2024-03-14T09:53:41.784072+0000","last_undegraded":"2024-03-14T09:53:41.784072+0000","last_fullsized":"2024-03-14T09:53:41.784072+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:05:36.997462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783550+0000","last_change":"2024-03-14T09:52:09.769284+0000","last_active":"2024-03-14T09:53:41.783550+0000","last_peered":"2024-03-14T09:53:41.783550+0000","last_clean":"2024-03-14T09:53:41.783550+0000","last_became_active":"2024-03-14T09:52:09.768808+0000","last_became_peered":"2024-03-14T09:52:09.768808+0000","last_unstale":"2024-03-14T09:53:41.783550+0000","last_undegraded":"2024-03-14T09:53:41.783550+0000","last_fullsized":"2024-03-14T09:53:41.783550+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:48:56.015410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"83'9","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789540+0000","last_change":"2024-03-14T09:52:42.154303+0000","last_active":"2024-03-14T09:53:41.789540+0000","last_peered":"2024-03-14T09:53:41.789540+0000","last_clean":"2024-03-14T09:53:41.789540+0000","last_became_active":"2024-03-14T09:52:42.147756+0000","last_became_peered":"2024-03-14T09:52:42.147756+0000","last_unstale":"2024-03-14T09:53:41.789540+0000","last_undegraded":"2024-03-14T09:53:41.789540+0000","last_fullsized":"2024-03-14T09:53:41.789540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:01:27.630337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785328+0000","last_change":"2024-03-14T09:52:05.739004+0000","last_active":"2024-03-14T09:53:41.785328+0000","last_peered":"2024-03-14T09:53:41.785328+0000","last_clean":"2024-03-14T09:53:41.785328+0000","last_became_active":"2024-03-14T09:52:05.732217+0000","last_became_peered":"2024-03-14T09:52:05.732217+0000","last_unstale":"2024-03-14T09:53:41.785328+0000","last_undegraded":"2024-03-14T09:53:41.785328+0000","last_fullsized":"2024-03-14T09:53:41.785328+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:02:21.926623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785929+0000","last_change":"2024-03-14T09:52:09.776128+0000","last_active":"2024-03-14T09:53:41.785929+0000","last_peered":"2024-03-14T09:53:41.785929+0000","last_clean":"2024-03-14T09:53:41.785929+0000","last_became_active":"2024-03-14T09:52:09.775701+0000","last_became_peered":"2024-03-14T09:52:09.775701+0000","last_unstale":"2024-03-14T09:53:41.785929+0000","last_undegraded":"2024-03-14T09:53:41.785929+0000","last_fullsized":"2024-03-14T09:53:41.785929+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:41:51.790627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"83'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785672+0000","last_change":"2024-03-14T09:52:05.234269+0000","last_active":"2024-03-14T09:53:41.785672+0000","last_peered":"2024-03-14T09:53:41.785672+0000","last_clean":"2024-03-14T09:53:41.785672+0000","last_became_active":"2024-03-14T09:52:05.233418+0000","last_became_peered":"2024-03-14T09:52:05.233418+0000","last_unstale":"2024-03-14T09:53:41.785672+0000","last_undegraded":"2024-03-14T09:53:41.785672+0000","last_fullsized":"2024-03-14T09:53:41.785672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:22:46.894461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899167+0000","last_change":"2024-03-14T09:52:05.741786+0000","last_active":"2024-03-14T09:53:41.899167+0000","last_peered":"2024-03-14T09:53:41.899167+0000","last_clean":"2024-03-14T09:53:41.899167+0000","last_became_active":"2024-03-14T09:52:05.741461+0000","last_became_peered":"2024-03-14T09:52:05.741461+0000","last_unstale":"2024-03-14T09:53:41.899167+0000","last_undegraded":"2024-03-14T09:53:41.899167+0000","last_fullsized":"2024-03-14T09:53:41.899167+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:21:17.409530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786363+0000","last_change":"2024-03-14T09:52:09.768851+0000","last_active":"2024-03-14T09:53:41.786363+0000","last_peered":"2024-03-14T09:53:41.786363+0000","last_clean":"2024-03-14T09:53:41.786363+0000","last_became_active":"2024-03-14T09:52:09.768485+0000","last_became_peered":"2024-03-14T09:52:09.768485+0000","last_unstale":"2024-03-14T09:53:41.786363+0000","last_undegraded":"2024-03-14T09:53:41.786363+0000","last_fullsized":"2024-03-14T09:53:41.786363+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:00:51.189790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"83'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899465+0000","last_change":"2024-03-14T09:52:42.154227+0000","last_active":"2024-03-14T09:53:41.899465+0000","last_peered":"2024-03-14T09:53:41.899465+0000","last_clean":"2024-03-14T09:53:41.899465+0000","last_became_active":"2024-03-14T09:52:42.149841+0000","last_became_peered":"2024-03-14T09:52:42.149841+0000","last_unstale":"2024-03-14T09:53:41.899465+0000","last_undegraded":"2024-03-14T09:53:41.899465+0000","last_fullsized":"2024-03-14T09:53:41.899465+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:04:46.911921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789719+0000","last_change":"2024-03-14T09:52:05.742994+0000","last_active":"2024-03-14T09:53:41.789719+0000","last_peered":"2024-03-14T09:53:41.789719+0000","last_clean":"2024-03-14T09:53:41.789719+0000","last_became_active":"2024-03-14T09:52:05.742652+0000","last_became_peered":"2024-03-14T09:52:05.742652+0000","last_unstale":"2024-03-14T09:53:41.789719+0000","last_undegraded":"2024-03-14T09:53:41.789719+0000","last_fullsized":"2024-03-14T09:53:41.789719+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:46:47.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785978+0000","last_change":"2024-03-14T09:52:09.763009+0000","last_active":"2024-03-14T09:53:41.785978+0000","last_peered":"2024-03-14T09:53:41.785978+0000","last_clean":"2024-03-14T09:53:41.785978+0000","last_became_active":"2024-03-14T09:52:09.762621+0000","last_became_peered":"2024-03-14T09:52:09.762621+0000","last_unstale":"2024-03-14T09:53:41.785978+0000","last_undegraded":"2024-03-14T09:53:41.785978+0000","last_fullsized":"2024-03-14T09:53:41.785978+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:56:22.162409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788297+0000","last_change":"2024-03-14T09:52:05.234039+0000","last_active":"2024-03-14T09:53:41.788297+0000","last_peered":"2024-03-14T09:53:41.788297+0000","last_clean":"2024-03-14T09:53:41.788297+0000","last_became_active":"2024-03-14T09:52:05.233612+0000","last_became_peered":"2024-03-14T09:52:05.233612+0000","last_unstale":"2024-03-14T09:53:41.788297+0000","last_undegraded":"2024-03-14T09:53:41.788297+0000","last_fullsized":"2024-03-14T09:53:41.788297+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:29:52.968178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"85'15","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787964+0000","last_change":"2024-03-14T09:52:42.426506+0000","last_active":"2024-03-14T09:53:41.787964+0000","last_peered":"2024-03-14T09:53:41.787964+0000","last_clean":"2024-03-14T09:53:41.787964+0000","last_became_active":"2024-03-14T09:52:42.426062+0000","last_became_peered":"2024-03-14T09:52:42.426062+0000","last_unstale":"2024-03-14T09:53:41.787964+0000","last_undegraded":"2024-03-14T09:53:41.787964+0000","last_fullsized":"2024-03-14T09:53:41.787964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:23:42.943381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783824+0000","last_change":"2024-03-14T09:52:09.757856+0000","last_active":"2024-03-14T09:53:41.783824+0000","last_peered":"2024-03-14T09:53:41.783824+0000","last_clean":"2024-03-14T09:53:41.783824+0000","last_became_active":"2024-03-14T09:52:09.757395+0000","last_became_peered":"2024-03-14T09:52:09.757395+0000","last_unstale":"2024-03-14T09:53:41.783824+0000","last_undegraded":"2024-03-14T09:53:41.783824+0000","last_fullsized":"2024-03-14T09:53:41.783824+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:34:06.600727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"83'5","reported_seq":109,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:50.849824+0000","last_change":"2024-03-14T09:52:05.228527+0000","last_active":"2024-03-14T09:53:50.849824+0000","last_peered":"2024-03-14T09:53:50.849824+0000","last_clean":"2024-03-14T09:53:50.849824+0000","last_became_active":"2024-03-14T09:52:05.228177+0000","last_became_peered":"2024-03-14T09:52:05.228177+0000","last_unstale":"2024-03-14T09:53:50.849824+0000","last_undegraded":"2024-03-14T09:53:50.849824+0000","last_fullsized":"2024-03-14T09:53:50.849824+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:17:57.529308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789022+0000","last_change":"2024-03-14T09:52:05.734041+0000","last_active":"2024-03-14T09:53:41.789022+0000","last_peered":"2024-03-14T09:53:41.789022+0000","last_clean":"2024-03-14T09:53:41.789022+0000","last_became_active":"2024-03-14T09:52:05.732491+0000","last_became_peered":"2024-03-14T09:52:05.732491+0000","last_unstale":"2024-03-14T09:53:41.789022+0000","last_undegraded":"2024-03-14T09:53:41.789022+0000","last_fullsized":"2024-03-14T09:53:41.789022+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:07:31.909382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:43.640334+0000","last_change":"2024-03-14T09:53:43.640334+0000","last_active":"2024-03-14T09:53:43.640334+0000","last_peered":"2024-03-14T09:53:43.640334+0000","last_clean":"2024-03-14T09:53:43.640334+0000","last_became_active":"2024-03-14T09:53:43.639821+0000","last_became_peered":"2024-03-14T09:53:43.639821+0000","last_unstale":"2024-03-14T09:53:43.640334+0000","last_undegraded":"2024-03-14T09:53:43.640334+0000","last_fullsized":"2024-03-14T09:53:43.640334+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:23:30.760149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,10],"acting":[2,6,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"83'3","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787260+0000","last_change":"2024-03-14T09:52:05.228893+0000","last_active":"2024-03-14T09:53:41.787260+0000","last_peered":"2024-03-14T09:53:41.787260+0000","last_clean":"2024-03-14T09:53:41.787260+0000","last_became_active":"2024-03-14T09:52:05.228569+0000","last_became_peered":"2024-03-14T09:52:05.228569+0000","last_unstale":"2024-03-14T09:53:41.787260+0000","last_undegraded":"2024-03-14T09:53:41.787260+0000","last_fullsized":"2024-03-14T09:53:41.787260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:28:03.200437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788237+0000","last_change":"2024-03-14T09:52:05.736388+0000","last_active":"2024-03-14T09:53:41.788237+0000","last_peered":"2024-03-14T09:53:41.788237+0000","last_clean":"2024-03-14T09:53:41.788237+0000","last_became_active":"2024-03-14T09:52:05.735131+0000","last_became_peered":"2024-03-14T09:52:05.735131+0000","last_unstale":"2024-03-14T09:53:41.788237+0000","last_undegraded":"2024-03-14T09:53:41.788237+0000","last_fullsized":"2024-03-14T09:53:41.788237+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:00:54.120087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899317+0000","last_change":"2024-03-14T09:52:09.770321+0000","last_active":"2024-03-14T09:53:41.899317+0000","last_peered":"2024-03-14T09:53:41.899317+0000","last_clean":"2024-03-14T09:53:41.899317+0000","last_became_active":"2024-03-14T09:52:09.769710+0000","last_became_peered":"2024-03-14T09:52:09.769710+0000","last_unstale":"2024-03-14T09:53:41.899317+0000","last_undegraded":"2024-03-14T09:53:41.899317+0000","last_fullsized":"2024-03-14T09:53:41.899317+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:12:12.635248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"83'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789908+0000","last_change":"2024-03-14T09:52:05.223912+0000","last_active":"2024-03-14T09:53:41.789908+0000","last_peered":"2024-03-14T09:53:41.789908+0000","last_clean":"2024-03-14T09:53:41.789908+0000","last_became_active":"2024-03-14T09:52:05.223380+0000","last_became_peered":"2024-03-14T09:52:05.223380+0000","last_unstale":"2024-03-14T09:53:41.789908+0000","last_undegraded":"2024-03-14T09:53:41.789908+0000","last_fullsized":"2024-03-14T09:53:41.789908+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:08:33.098215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785120+0000","last_change":"2024-03-14T09:52:05.743277+0000","last_active":"2024-03-14T09:53:41.785120+0000","last_peered":"2024-03-14T09:53:41.785120+0000","last_clean":"2024-03-14T09:53:41.785120+0000","last_became_active":"2024-03-14T09:52:05.741386+0000","last_became_peered":"2024-03-14T09:52:05.741386+0000","last_unstale":"2024-03-14T09:53:41.785120+0000","last_undegraded":"2024-03-14T09:53:41.785120+0000","last_fullsized":"2024-03-14T09:53:41.785120+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:08:08.320279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899160+0000","last_change":"2024-03-14T09:52:09.766485+0000","last_active":"2024-03-14T09:53:41.899160+0000","last_peered":"2024-03-14T09:53:41.899160+0000","last_clean":"2024-03-14T09:53:41.899160+0000","last_became_active":"2024-03-14T09:52:09.766115+0000","last_became_peered":"2024-03-14T09:52:09.766115+0000","last_unstale":"2024-03-14T09:53:41.899160+0000","last_undegraded":"2024-03-14T09:53:41.899160+0000","last_fullsized":"2024-03-14T09:53:41.899160+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:34:45.067757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"83'1","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786965+0000","last_change":"2024-03-14T09:52:05.233852+0000","last_active":"2024-03-14T09:53:41.786965+0000","last_peered":"2024-03-14T09:53:41.786965+0000","last_clean":"2024-03-14T09:53:41.786965+0000","last_became_active":"2024-03-14T09:52:05.233548+0000","last_became_peered":"2024-03-14T09:52:05.233548+0000","last_unstale":"2024-03-14T09:53:41.786965+0000","last_undegraded":"2024-03-14T09:53:41.786965+0000","last_fullsized":"2024-03-14T09:53:41.786965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:08:51.821465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"85'11","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788942+0000","last_change":"2024-03-14T09:52:05.731215+0000","last_active":"2024-03-14T09:53:41.788942+0000","last_peered":"2024-03-14T09:53:41.788942+0000","last_clean":"2024-03-14T09:53:41.788942+0000","last_became_active":"2024-03-14T09:52:05.730924+0000","last_became_peered":"2024-03-14T09:52:05.730924+0000","last_unstale":"2024-03-14T09:53:41.788942+0000","last_undegraded":"2024-03-14T09:53:41.788942+0000","last_fullsized":"2024-03-14T09:53:41.788942+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:12:17.560297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785051+0000","last_change":"2024-03-14T09:52:09.753775+0000","last_active":"2024-03-14T09:53:41.785051+0000","last_peered":"2024-03-14T09:53:41.785051+0000","last_clean":"2024-03-14T09:53:41.785051+0000","last_became_active":"2024-03-14T09:52:09.753422+0000","last_became_peered":"2024-03-14T09:52:09.753422+0000","last_unstale":"2024-03-14T09:53:41.785051+0000","last_undegraded":"2024-03-14T09:53:41.785051+0000","last_fullsized":"2024-03-14T09:53:41.785051+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:59:25.625909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"83'3","reported_seq":106,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:51.850659+0000","last_change":"2024-03-14T09:52:05.229064+0000","last_active":"2024-03-14T09:53:51.850659+0000","last_peered":"2024-03-14T09:53:51.850659+0000","last_clean":"2024-03-14T09:53:51.850659+0000","last_became_active":"2024-03-14T09:52:05.228751+0000","last_became_peered":"2024-03-14T09:52:05.228751+0000","last_unstale":"2024-03-14T09:53:51.850659+0000","last_undegraded":"2024-03-14T09:53:51.850659+0000","last_fullsized":"2024-03-14T09:53:51.850659+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:13:29.832866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"85'10","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790141+0000","last_change":"2024-03-14T09:52:05.737798+0000","last_active":"2024-03-14T09:53:41.790141+0000","last_peered":"2024-03-14T09:53:41.790141+0000","last_clean":"2024-03-14T09:53:41.790141+0000","last_became_active":"2024-03-14T09:52:05.737449+0000","last_became_peered":"2024-03-14T09:52:05.737449+0000","last_unstale":"2024-03-14T09:53:41.790141+0000","last_undegraded":"2024-03-14T09:53:41.790141+0000","last_fullsized":"2024-03-14T09:53:41.790141+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:40:10.079770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788972+0000","last_change":"2024-03-14T09:52:09.768368+0000","last_active":"2024-03-14T09:53:41.788972+0000","last_peered":"2024-03-14T09:53:41.788972+0000","last_clean":"2024-03-14T09:53:41.788972+0000","last_became_active":"2024-03-14T09:52:09.767622+0000","last_became_peered":"2024-03-14T09:52:09.767622+0000","last_unstale":"2024-03-14T09:53:41.788972+0000","last_undegraded":"2024-03-14T09:53:41.788972+0000","last_fullsized":"2024-03-14T09:53:41.788972+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:19:28.836317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"83'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783894+0000","last_change":"2024-03-14T09:52:05.229778+0000","last_active":"2024-03-14T09:53:41.783894+0000","last_peered":"2024-03-14T09:53:41.783894+0000","last_clean":"2024-03-14T09:53:41.783894+0000","last_became_active":"2024-03-14T09:52:05.229468+0000","last_became_peered":"2024-03-14T09:52:05.229468+0000","last_unstale":"2024-03-14T09:53:41.783894+0000","last_undegraded":"2024-03-14T09:53:41.783894+0000","last_fullsized":"2024-03-14T09:53:41.783894+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:49:58.433646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899333+0000","last_change":"2024-03-14T09:52:05.751027+0000","last_active":"2024-03-14T09:53:41.899333+0000","last_peered":"2024-03-14T09:53:41.899333+0000","last_clean":"2024-03-14T09:53:41.899333+0000","last_became_active":"2024-03-14T09:52:05.750650+0000","last_became_peered":"2024-03-14T09:52:05.750650+0000","last_unstale":"2024-03-14T09:53:41.899333+0000","last_undegraded":"2024-03-14T09:53:41.899333+0000","last_fullsized":"2024-03-14T09:53:41.899333+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:05:37.264396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:43.631132+0000","last_change":"2024-03-14T09:53:43.631132+0000","last_active":"2024-03-14T09:53:43.631132+0000","last_peered":"2024-03-14T09:53:43.631132+0000","last_clean":"2024-03-14T09:53:43.631132+0000","last_became_active":"2024-03-14T09:53:43.630718+0000","last_became_peered":"2024-03-14T09:53:43.630718+0000","last_unstale":"2024-03-14T09:53:43.631132+0000","last_undegraded":"2024-03-14T09:53:43.631132+0000","last_fullsized":"2024-03-14T09:53:43.631132+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T15:45:17.156095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"83'4","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787859+0000","last_change":"2024-03-14T09:52:05.235472+0000","last_active":"2024-03-14T09:53:41.787859+0000","last_peered":"2024-03-14T09:53:41.787859+0000","last_clean":"2024-03-14T09:53:41.787859+0000","last_became_active":"2024-03-14T09:52:05.235106+0000","last_became_peered":"2024-03-14T09:52:05.235106+0000","last_unstale":"2024-03-14T09:53:41.787859+0000","last_undegraded":"2024-03-14T09:53:41.787859+0000","last_fullsized":"2024-03-14T09:53:41.787859+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:30:36.446130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"85'11","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789899+0000","last_change":"2024-03-14T09:52:05.747615+0000","last_active":"2024-03-14T09:53:41.789899+0000","last_peered":"2024-03-14T09:53:41.789899+0000","last_clean":"2024-03-14T09:53:41.789899+0000","last_became_active":"2024-03-14T09:52:05.747344+0000","last_became_peered":"2024-03-14T09:52:05.747344+0000","last_unstale":"2024-03-14T09:53:41.789899+0000","last_undegraded":"2024-03-14T09:53:41.789899+0000","last_fullsized":"2024-03-14T09:53:41.789899+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:52:23.096124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788627+0000","last_change":"2024-03-14T09:52:09.768295+0000","last_active":"2024-03-14T09:53:41.788627+0000","last_peered":"2024-03-14T09:53:41.788627+0000","last_clean":"2024-03-14T09:53:41.788627+0000","last_became_active":"2024-03-14T09:52:09.760997+0000","last_became_peered":"2024-03-14T09:52:09.760997+0000","last_unstale":"2024-03-14T09:53:41.788627+0000","last_undegraded":"2024-03-14T09:53:41.788627+0000","last_fullsized":"2024-03-14T09:53:41.788627+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:22:09.909302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'37","reported_seq":233,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:52.852579+0000","last_change":"2024-03-14T09:52:05.227876+0000","last_active":"2024-03-14T09:53:52.852579+0000","last_peered":"2024-03-14T09:53:52.852579+0000","last_clean":"2024-03-14T09:53:52.852579+0000","last_became_active":"2024-03-14T09:52:05.227519+0000","last_became_peered":"2024-03-14T09:52:05.227519+0000","last_unstale":"2024-03-14T09:53:52.852579+0000","last_undegraded":"2024-03-14T09:53:52.852579+0000","last_fullsized":"2024-03-14T09:53:52.852579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:12.616609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66994,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":176,"num_read_kb":154,"num_write":38,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788944+0000","last_change":"2024-03-14T09:52:05.737432+0000","last_active":"2024-03-14T09:53:41.788944+0000","last_peered":"2024-03-14T09:53:41.788944+0000","last_clean":"2024-03-14T09:53:41.788944+0000","last_became_active":"2024-03-14T09:52:05.735050+0000","last_became_peered":"2024-03-14T09:52:05.735050+0000","last_unstale":"2024-03-14T09:53:41.788944+0000","last_undegraded":"2024-03-14T09:53:41.788944+0000","last_fullsized":"2024-03-14T09:53:41.788944+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T09:57:45.383155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788252+0000","last_change":"2024-03-14T09:52:09.754909+0000","last_active":"2024-03-14T09:53:41.788252+0000","last_peered":"2024-03-14T09:53:41.788252+0000","last_clean":"2024-03-14T09:53:41.788252+0000","last_became_active":"2024-03-14T09:52:09.754367+0000","last_became_peered":"2024-03-14T09:52:09.754367+0000","last_unstale":"2024-03-14T09:53:41.788252+0000","last_undegraded":"2024-03-14T09:53:41.788252+0000","last_fullsized":"2024-03-14T09:53:41.788252+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:52:34.251555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"83'6","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783813+0000","last_change":"2024-03-14T09:52:05.224869+0000","last_active":"2024-03-14T09:53:41.783813+0000","last_peered":"2024-03-14T09:53:41.783813+0000","last_clean":"2024-03-14T09:53:41.783813+0000","last_became_active":"2024-03-14T09:52:05.224499+0000","last_became_peered":"2024-03-14T09:52:05.224499+0000","last_unstale":"2024-03-14T09:53:41.783813+0000","last_undegraded":"2024-03-14T09:53:41.783813+0000","last_fullsized":"2024-03-14T09:53:41.783813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:49:01.729417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789760+0000","last_change":"2024-03-14T09:52:05.729168+0000","last_active":"2024-03-14T09:53:41.789760+0000","last_peered":"2024-03-14T09:53:41.789760+0000","last_clean":"2024-03-14T09:53:41.789760+0000","last_became_active":"2024-03-14T09:52:05.728806+0000","last_became_peered":"2024-03-14T09:52:05.728806+0000","last_unstale":"2024-03-14T09:53:41.789760+0000","last_undegraded":"2024-03-14T09:53:41.789760+0000","last_fullsized":"2024-03-14T09:53:41.789760+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:54:36.758853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899244+0000","last_change":"2024-03-14T09:52:09.770126+0000","last_active":"2024-03-14T09:53:41.899244+0000","last_peered":"2024-03-14T09:53:41.899244+0000","last_clean":"2024-03-14T09:53:41.899244+0000","last_became_active":"2024-03-14T09:52:09.769734+0000","last_became_peered":"2024-03-14T09:52:09.769734+0000","last_unstale":"2024-03-14T09:53:41.899244+0000","last_undegraded":"2024-03-14T09:53:41.899244+0000","last_fullsized":"2024-03-14T09:53:41.899244+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:50:37.839858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"83'3","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787719+0000","last_change":"2024-03-14T09:52:05.235201+0000","last_active":"2024-03-14T09:53:41.787719+0000","last_peered":"2024-03-14T09:53:41.787719+0000","last_clean":"2024-03-14T09:53:41.787719+0000","last_became_active":"2024-03-14T09:52:05.234816+0000","last_became_peered":"2024-03-14T09:52:05.234816+0000","last_unstale":"2024-03-14T09:53:41.787719+0000","last_undegraded":"2024-03-14T09:53:41.787719+0000","last_fullsized":"2024-03-14T09:53:41.787719+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:01:50.070913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'15","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783755+0000","last_change":"2024-03-14T09:52:05.735304+0000","last_active":"2024-03-14T09:53:41.783755+0000","last_peered":"2024-03-14T09:53:41.783755+0000","last_clean":"2024-03-14T09:53:41.783755+0000","last_became_active":"2024-03-14T09:52:05.734977+0000","last_became_peered":"2024-03-14T09:52:05.734977+0000","last_unstale":"2024-03-14T09:53:41.783755+0000","last_undegraded":"2024-03-14T09:53:41.783755+0000","last_fullsized":"2024-03-14T09:53:41.783755+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:08:28.071962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789402+0000","last_change":"2024-03-14T09:52:09.770721+0000","last_active":"2024-03-14T09:53:41.789402+0000","last_peered":"2024-03-14T09:53:41.789402+0000","last_clean":"2024-03-14T09:53:41.789402+0000","last_became_active":"2024-03-14T09:52:09.770270+0000","last_became_peered":"2024-03-14T09:52:09.770270+0000","last_unstale":"2024-03-14T09:53:41.789402+0000","last_undegraded":"2024-03-14T09:53:41.789402+0000","last_fullsized":"2024-03-14T09:53:41.789402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:09:33.480235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'6","reported_seq":111,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:48.852068+0000","last_change":"2024-03-14T09:52:05.227260+0000","last_active":"2024-03-14T09:53:48.852068+0000","last_peered":"2024-03-14T09:53:48.852068+0000","last_clean":"2024-03-14T09:53:48.852068+0000","last_became_active":"2024-03-14T09:52:05.226922+0000","last_became_peered":"2024-03-14T09:52:05.226922+0000","last_unstale":"2024-03-14T09:53:48.852068+0000","last_undegraded":"2024-03-14T09:53:48.852068+0000","last_fullsized":"2024-03-14T09:53:48.852068+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:36:09.770478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783445+0000","last_change":"2024-03-14T09:52:05.734278+0000","last_active":"2024-03-14T09:53:41.783445+0000","last_peered":"2024-03-14T09:53:41.783445+0000","last_clean":"2024-03-14T09:53:41.783445+0000","last_became_active":"2024-03-14T09:52:05.732051+0000","last_became_peered":"2024-03-14T09:52:05.732051+0000","last_unstale":"2024-03-14T09:53:41.783445+0000","last_undegraded":"2024-03-14T09:53:41.783445+0000","last_fullsized":"2024-03-14T09:53:41.783445+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:33:07.137983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790644+0000","last_change":"2024-03-14T09:52:09.764068+0000","last_active":"2024-03-14T09:53:41.790644+0000","last_peered":"2024-03-14T09:53:41.790644+0000","last_clean":"2024-03-14T09:53:41.790644+0000","last_became_active":"2024-03-14T09:52:09.763613+0000","last_became_peered":"2024-03-14T09:52:09.763613+0000","last_unstale":"2024-03-14T09:53:41.790644+0000","last_undegraded":"2024-03-14T09:53:41.790644+0000","last_fullsized":"2024-03-14T09:53:41.790644+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:32:15.524602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"83'15","reported_seq":63,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787502+0000","last_change":"2024-03-14T09:52:42.438205+0000","last_active":"2024-03-14T09:53:41.787502+0000","last_peered":"2024-03-14T09:53:41.787502+0000","last_clean":"2024-03-14T09:53:41.787502+0000","last_became_active":"2024-03-14T09:52:42.426268+0000","last_became_peered":"2024-03-14T09:52:42.426268+0000","last_unstale":"2024-03-14T09:53:41.787502+0000","last_undegraded":"2024-03-14T09:53:41.787502+0000","last_fullsized":"2024-03-14T09:53:41.787502+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:10:09.651131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788220+0000","last_change":"2024-03-14T09:52:05.734366+0000","last_active":"2024-03-14T09:53:41.788220+0000","last_peered":"2024-03-14T09:53:41.788220+0000","last_clean":"2024-03-14T09:53:41.788220+0000","last_became_active":"2024-03-14T09:52:05.734058+0000","last_became_peered":"2024-03-14T09:52:05.734058+0000","last_unstale":"2024-03-14T09:53:41.788220+0000","last_undegraded":"2024-03-14T09:53:41.788220+0000","last_fullsized":"2024-03-14T09:53:41.788220+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:49:49.233683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788119+0000","last_change":"2024-03-14T09:52:09.760241+0000","last_active":"2024-03-14T09:53:41.788119+0000","last_peered":"2024-03-14T09:53:41.788119+0000","last_clean":"2024-03-14T09:53:41.788119+0000","last_became_active":"2024-03-14T09:52:09.759797+0000","last_became_peered":"2024-03-14T09:52:09.759797+0000","last_unstale":"2024-03-14T09:53:41.788119+0000","last_undegraded":"2024-03-14T09:53:41.788119+0000","last_fullsized":"2024-03-14T09:53:41.788119+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:50:33.035775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:42.792396+0000","last_change":"2024-03-14T09:52:05.232932+0000","last_active":"2024-03-14T09:53:42.792396+0000","last_peered":"2024-03-14T09:53:42.792396+0000","last_clean":"2024-03-14T09:53:42.792396+0000","last_became_active":"2024-03-14T09:52:05.232608+0000","last_became_peered":"2024-03-14T09:52:05.232608+0000","last_unstale":"2024-03-14T09:53:42.792396+0000","last_undegraded":"2024-03-14T09:53:42.792396+0000","last_fullsized":"2024-03-14T09:53:42.792396+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:43:59.613531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899344+0000","last_change":"2024-03-14T09:52:05.733762+0000","last_active":"2024-03-14T09:53:41.899344+0000","last_peered":"2024-03-14T09:53:41.899344+0000","last_clean":"2024-03-14T09:53:41.899344+0000","last_became_active":"2024-03-14T09:52:05.733417+0000","last_became_peered":"2024-03-14T09:52:05.733417+0000","last_unstale":"2024-03-14T09:53:41.899344+0000","last_undegraded":"2024-03-14T09:53:41.899344+0000","last_fullsized":"2024-03-14T09:53:41.899344+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:11:12.593952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789125+0000","last_change":"2024-03-14T09:52:42.426896+0000","last_active":"2024-03-14T09:53:41.789125+0000","last_peered":"2024-03-14T09:53:41.789125+0000","last_clean":"2024-03-14T09:53:41.789125+0000","last_became_active":"2024-03-14T09:52:42.426287+0000","last_became_peered":"2024-03-14T09:52:42.426287+0000","last_unstale":"2024-03-14T09:53:41.789125+0000","last_undegraded":"2024-03-14T09:53:41.789125+0000","last_fullsized":"2024-03-14T09:53:41.789125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:53:16.523707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789199+0000","last_change":"2024-03-14T09:52:05.227636+0000","last_active":"2024-03-14T09:53:41.789199+0000","last_peered":"2024-03-14T09:53:41.789199+0000","last_clean":"2024-03-14T09:53:41.789199+0000","last_became_active":"2024-03-14T09:52:05.227274+0000","last_became_peered":"2024-03-14T09:52:05.227274+0000","last_unstale":"2024-03-14T09:53:41.789199+0000","last_undegraded":"2024-03-14T09:53:41.789199+0000","last_fullsized":"2024-03-14T09:53:41.789199+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:13:25.380090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'4","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787833+0000","last_change":"2024-03-14T09:52:05.733158+0000","last_active":"2024-03-14T09:53:41.787833+0000","last_peered":"2024-03-14T09:53:41.787833+0000","last_clean":"2024-03-14T09:53:41.787833+0000","last_became_active":"2024-03-14T09:52:05.732812+0000","last_became_peered":"2024-03-14T09:52:05.732812+0000","last_unstale":"2024-03-14T09:53:41.787833+0000","last_undegraded":"2024-03-14T09:53:41.787833+0000","last_fullsized":"2024-03-14T09:53:41.787833+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:01:49.154860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790946+0000","last_change":"2024-03-14T09:52:09.761596+0000","last_active":"2024-03-14T09:53:41.790946+0000","last_peered":"2024-03-14T09:53:41.790946+0000","last_clean":"2024-03-14T09:53:41.790946+0000","last_became_active":"2024-03-14T09:52:09.761184+0000","last_became_peered":"2024-03-14T09:52:09.761184+0000","last_unstale":"2024-03-14T09:53:41.790946+0000","last_undegraded":"2024-03-14T09:53:41.790946+0000","last_fullsized":"2024-03-14T09:53:41.790946+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:49:02.567472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"83'1","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783455+0000","last_change":"2024-03-14T09:52:05.231828+0000","last_active":"2024-03-14T09:53:41.783455+0000","last_peered":"2024-03-14T09:53:41.783455+0000","last_clean":"2024-03-14T09:53:41.783455+0000","last_became_active":"2024-03-14T09:52:05.231490+0000","last_became_peered":"2024-03-14T09:52:05.231490+0000","last_unstale":"2024-03-14T09:53:41.783455+0000","last_undegraded":"2024-03-14T09:53:41.783455+0000","last_fullsized":"2024-03-14T09:53:41.783455+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:24:53.899728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787604+0000","last_change":"2024-03-14T09:52:05.744646+0000","last_active":"2024-03-14T09:53:41.787604+0000","last_peered":"2024-03-14T09:53:41.787604+0000","last_clean":"2024-03-14T09:53:41.787604+0000","last_became_active":"2024-03-14T09:52:05.744327+0000","last_became_peered":"2024-03-14T09:52:05.744327+0000","last_unstale":"2024-03-14T09:53:41.787604+0000","last_undegraded":"2024-03-14T09:53:41.787604+0000","last_fullsized":"2024-03-14T09:53:41.787604+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:46:50.643350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.791013+0000","last_change":"2024-03-14T09:52:09.765374+0000","last_active":"2024-03-14T09:53:41.791013+0000","last_peered":"2024-03-14T09:53:41.791013+0000","last_clean":"2024-03-14T09:53:41.791013+0000","last_became_active":"2024-03-14T09:52:09.764939+0000","last_became_peered":"2024-03-14T09:52:09.764939+0000","last_unstale":"2024-03-14T09:53:41.791013+0000","last_undegraded":"2024-03-14T09:53:41.791013+0000","last_fullsized":"2024-03-14T09:53:41.791013+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:08:10.688746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787063+0000","last_change":"2024-03-14T09:52:05.729076+0000","last_active":"2024-03-14T09:53:41.787063+0000","last_peered":"2024-03-14T09:53:41.787063+0000","last_clean":"2024-03-14T09:53:41.787063+0000","last_became_active":"2024-03-14T09:52:05.728711+0000","last_became_peered":"2024-03-14T09:52:05.728711+0000","last_unstale":"2024-03-14T09:53:41.787063+0000","last_undegraded":"2024-03-14T09:53:41.787063+0000","last_fullsized":"2024-03-14T09:53:41.787063+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:11:05.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":110,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:53.860698+0000","last_change":"2024-03-14T09:52:05.236209+0000","last_active":"2024-03-14T09:53:53.860698+0000","last_peered":"2024-03-14T09:53:53.860698+0000","last_clean":"2024-03-14T09:53:53.860698+0000","last_became_active":"2024-03-14T09:52:05.235517+0000","last_became_peered":"2024-03-14T09:52:05.235517+0000","last_unstale":"2024-03-14T09:53:53.860698+0000","last_undegraded":"2024-03-14T09:53:53.860698+0000","last_fullsized":"2024-03-14T09:53:53.860698+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:09:54.275728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":85,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"83'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783734+0000","last_change":"2024-03-14T09:52:09.761095+0000","last_active":"2024-03-14T09:53:41.783734+0000","last_peered":"2024-03-14T09:53:41.783734+0000","last_clean":"2024-03-14T09:53:41.783734+0000","last_became_active":"2024-03-14T09:52:09.760662+0000","last_became_peered":"2024-03-14T09:52:09.760662+0000","last_unstale":"2024-03-14T09:53:41.783734+0000","last_undegraded":"2024-03-14T09:53:41.783734+0000","last_fullsized":"2024-03-14T09:53:41.783734+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:07:17.670605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785163+0000","last_change":"2024-03-14T09:52:05.736078+0000","last_active":"2024-03-14T09:53:41.785163+0000","last_peered":"2024-03-14T09:53:41.785163+0000","last_clean":"2024-03-14T09:53:41.785163+0000","last_became_active":"2024-03-14T09:52:05.735715+0000","last_became_peered":"2024-03-14T09:52:05.735715+0000","last_unstale":"2024-03-14T09:53:41.785163+0000","last_undegraded":"2024-03-14T09:53:41.785163+0000","last_fullsized":"2024-03-14T09:53:41.785163+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:41:32.863680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"83'3","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786470+0000","last_change":"2024-03-14T09:52:05.225756+0000","last_active":"2024-03-14T09:53:41.786470+0000","last_peered":"2024-03-14T09:53:41.786470+0000","last_clean":"2024-03-14T09:53:41.786470+0000","last_became_active":"2024-03-14T09:52:05.225378+0000","last_became_peered":"2024-03-14T09:52:05.225378+0000","last_unstale":"2024-03-14T09:53:41.786470+0000","last_undegraded":"2024-03-14T09:53:41.786470+0000","last_fullsized":"2024-03-14T09:53:41.786470+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:14:49.594636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":339,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783295+0000","last_change":"2024-03-14T09:51:11.588790+0000","last_active":"2024-03-14T09:53:41.783295+0000","last_peered":"2024-03-14T09:53:41.783295+0000","last_clean":"2024-03-14T09:53:41.783295+0000","last_became_active":"2024-03-14T09:51:11.588287+0000","last_became_peered":"2024-03-14T09:51:11.588287+0000","last_unstale":"2024-03-14T09:53:41.783295+0000","last_undegraded":"2024-03-14T09:53:41.783295+0000","last_fullsized":"2024-03-14T09:53:41.783295+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:48:37.651174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:48:37.651174+0000","last_clean_scrub_stamp":"2024-03-14T09:48:37.651174+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:06:24.252565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787689+0000","last_change":"2024-03-14T09:52:09.761344+0000","last_active":"2024-03-14T09:53:41.787689+0000","last_peered":"2024-03-14T09:53:41.787689+0000","last_clean":"2024-03-14T09:53:41.787689+0000","last_became_active":"2024-03-14T09:52:09.760899+0000","last_became_peered":"2024-03-14T09:52:09.760899+0000","last_unstale":"2024-03-14T09:53:41.787689+0000","last_undegraded":"2024-03-14T09:53:41.787689+0000","last_fullsized":"2024-03-14T09:53:41.787689+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:24:26.212157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789785+0000","last_change":"2024-03-14T09:52:05.738735+0000","last_active":"2024-03-14T09:53:41.789785+0000","last_peered":"2024-03-14T09:53:41.789785+0000","last_clean":"2024-03-14T09:53:41.789785+0000","last_became_active":"2024-03-14T09:52:05.738447+0000","last_became_peered":"2024-03-14T09:52:05.738447+0000","last_unstale":"2024-03-14T09:53:41.789785+0000","last_undegraded":"2024-03-14T09:53:41.789785+0000","last_fullsized":"2024-03-14T09:53:41.789785+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:03:24.598291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'3","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783341+0000","last_change":"2024-03-14T09:52:05.230107+0000","last_active":"2024-03-14T09:53:41.783341+0000","last_peered":"2024-03-14T09:53:41.783341+0000","last_clean":"2024-03-14T09:53:41.783341+0000","last_became_active":"2024-03-14T09:52:05.229812+0000","last_became_peered":"2024-03-14T09:52:05.229812+0000","last_unstale":"2024-03-14T09:53:41.783341+0000","last_undegraded":"2024-03-14T09:53:41.783341+0000","last_fullsized":"2024-03-14T09:53:41.783341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:09:44.884867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899259+0000","last_change":"2024-03-14T09:52:09.763424+0000","last_active":"2024-03-14T09:53:41.899259+0000","last_peered":"2024-03-14T09:53:41.899259+0000","last_clean":"2024-03-14T09:53:41.899259+0000","last_became_active":"2024-03-14T09:52:09.763028+0000","last_became_peered":"2024-03-14T09:52:09.763028+0000","last_unstale":"2024-03-14T09:53:41.899259+0000","last_undegraded":"2024-03-14T09:53:41.899259+0000","last_fullsized":"2024-03-14T09:53:41.899259+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:47:31.415788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"83'1","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789049+0000","last_change":"2024-03-14T09:52:05.235099+0000","last_active":"2024-03-14T09:53:41.789049+0000","last_peered":"2024-03-14T09:53:41.789049+0000","last_clean":"2024-03-14T09:53:41.789049+0000","last_became_active":"2024-03-14T09:52:05.234798+0000","last_became_peered":"2024-03-14T09:52:05.234798+0000","last_unstale":"2024-03-14T09:53:41.789049+0000","last_undegraded":"2024-03-14T09:53:41.789049+0000","last_fullsized":"2024-03-14T09:53:41.789049+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:00:07.461931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.791106+0000","last_change":"2024-03-14T09:52:05.742736+0000","last_active":"2024-03-14T09:53:41.791106+0000","last_peered":"2024-03-14T09:53:41.791106+0000","last_clean":"2024-03-14T09:53:41.791106+0000","last_became_active":"2024-03-14T09:52:05.740927+0000","last_became_peered":"2024-03-14T09:52:05.740927+0000","last_unstale":"2024-03-14T09:53:41.791106+0000","last_undegraded":"2024-03-14T09:53:41.791106+0000","last_fullsized":"2024-03-14T09:53:41.791106+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:50:17.738763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.784877+0000","last_change":"2024-03-14T09:52:09.762722+0000","last_active":"2024-03-14T09:53:41.784877+0000","last_peered":"2024-03-14T09:53:41.784877+0000","last_clean":"2024-03-14T09:53:41.784877+0000","last_became_active":"2024-03-14T09:52:09.759053+0000","last_became_peered":"2024-03-14T09:52:09.759053+0000","last_unstale":"2024-03-14T09:53:41.784877+0000","last_undegraded":"2024-03-14T09:53:41.784877+0000","last_fullsized":"2024-03-14T09:53:41.784877+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:40:29.680002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:43.740679+0000","last_change":"2024-03-14T09:52:05.224778+0000","last_active":"2024-03-14T09:53:43.740679+0000","last_peered":"2024-03-14T09:53:43.740679+0000","last_clean":"2024-03-14T09:53:43.740679+0000","last_became_active":"2024-03-14T09:52:05.223158+0000","last_became_peered":"2024-03-14T09:52:05.223158+0000","last_unstale":"2024-03-14T09:53:43.740679+0000","last_undegraded":"2024-03-14T09:53:43.740679+0000","last_fullsized":"2024-03-14T09:53:43.740679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:01:29.910099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785742+0000","last_change":"2024-03-14T09:52:05.741753+0000","last_active":"2024-03-14T09:53:41.785742+0000","last_peered":"2024-03-14T09:53:41.785742+0000","last_clean":"2024-03-14T09:53:41.785742+0000","last_became_active":"2024-03-14T09:52:05.741451+0000","last_became_peered":"2024-03-14T09:52:05.741451+0000","last_unstale":"2024-03-14T09:53:41.785742+0000","last_undegraded":"2024-03-14T09:53:41.785742+0000","last_fullsized":"2024-03-14T09:53:41.785742+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:42:59.920909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783437+0000","last_change":"2024-03-14T09:52:09.755884+0000","last_active":"2024-03-14T09:53:41.783437+0000","last_peered":"2024-03-14T09:53:41.783437+0000","last_clean":"2024-03-14T09:53:41.783437+0000","last_became_active":"2024-03-14T09:52:09.755474+0000","last_became_peered":"2024-03-14T09:52:09.755474+0000","last_unstale":"2024-03-14T09:53:41.783437+0000","last_undegraded":"2024-03-14T09:53:41.783437+0000","last_fullsized":"2024-03-14T09:53:41.783437+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:59:23.383388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"83'4","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786361+0000","last_change":"2024-03-14T09:52:05.232673+0000","last_active":"2024-03-14T09:53:41.786361+0000","last_peered":"2024-03-14T09:53:41.786361+0000","last_clean":"2024-03-14T09:53:41.786361+0000","last_became_active":"2024-03-14T09:52:05.232296+0000","last_became_peered":"2024-03-14T09:52:05.232296+0000","last_unstale":"2024-03-14T09:53:41.786361+0000","last_undegraded":"2024-03-14T09:53:41.786361+0000","last_fullsized":"2024-03-14T09:53:41.786361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:24:56.638451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899250+0000","last_change":"2024-03-14T09:52:05.744906+0000","last_active":"2024-03-14T09:53:41.899250+0000","last_peered":"2024-03-14T09:53:41.899250+0000","last_clean":"2024-03-14T09:53:41.899250+0000","last_became_active":"2024-03-14T09:52:05.743753+0000","last_became_peered":"2024-03-14T09:52:05.743753+0000","last_unstale":"2024-03-14T09:53:41.899250+0000","last_undegraded":"2024-03-14T09:53:41.899250+0000","last_fullsized":"2024-03-14T09:53:41.899250+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T09:59:40.358352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786261+0000","last_change":"2024-03-14T09:52:09.768355+0000","last_active":"2024-03-14T09:53:41.786261+0000","last_peered":"2024-03-14T09:53:41.786261+0000","last_clean":"2024-03-14T09:53:41.786261+0000","last_became_active":"2024-03-14T09:52:09.767919+0000","last_became_peered":"2024-03-14T09:52:09.767919+0000","last_unstale":"2024-03-14T09:53:41.786261+0000","last_undegraded":"2024-03-14T09:53:41.786261+0000","last_fullsized":"2024-03-14T09:53:41.786261+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:45:08.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"83'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789425+0000","last_change":"2024-03-14T09:52:05.230343+0000","last_active":"2024-03-14T09:53:41.789425+0000","last_peered":"2024-03-14T09:53:41.789425+0000","last_clean":"2024-03-14T09:53:41.789425+0000","last_became_active":"2024-03-14T09:52:05.229974+0000","last_became_peered":"2024-03-14T09:52:05.229974+0000","last_unstale":"2024-03-14T09:53:41.789425+0000","last_undegraded":"2024-03-14T09:53:41.789425+0000","last_fullsized":"2024-03-14T09:53:41.789425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:21:02.458108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"85'2","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787651+0000","last_change":"2024-03-14T09:52:05.738347+0000","last_active":"2024-03-14T09:53:41.787651+0000","last_peered":"2024-03-14T09:53:41.787651+0000","last_clean":"2024-03-14T09:53:41.787651+0000","last_became_active":"2024-03-14T09:52:05.737961+0000","last_became_peered":"2024-03-14T09:52:05.737961+0000","last_unstale":"2024-03-14T09:53:41.787651+0000","last_undegraded":"2024-03-14T09:53:41.787651+0000","last_fullsized":"2024-03-14T09:53:41.787651+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:02:57.252069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:42.843871+0000","last_change":"2024-03-14T09:53:42.843441+0000","last_active":"2024-03-14T09:53:42.843871+0000","last_peered":"2024-03-14T09:53:42.843871+0000","last_clean":"2024-03-14T09:53:42.843871+0000","last_became_active":"2024-03-14T09:53:42.842860+0000","last_became_peered":"2024-03-14T09:53:42.842860+0000","last_unstale":"2024-03-14T09:53:42.843871+0000","last_undegraded":"2024-03-14T09:53:42.843871+0000","last_fullsized":"2024-03-14T09:53:42.843871+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:20:23.073703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"83'3","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.784877+0000","last_change":"2024-03-14T09:52:05.233579+0000","last_active":"2024-03-14T09:53:41.784877+0000","last_peered":"2024-03-14T09:53:41.784877+0000","last_clean":"2024-03-14T09:53:41.784877+0000","last_became_active":"2024-03-14T09:52:05.231605+0000","last_became_peered":"2024-03-14T09:52:05.231605+0000","last_unstale":"2024-03-14T09:53:41.784877+0000","last_undegraded":"2024-03-14T09:53:41.784877+0000","last_fullsized":"2024-03-14T09:53:41.784877+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:55:38.282203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899095+0000","last_change":"2024-03-14T09:52:05.736586+0000","last_active":"2024-03-14T09:53:41.899095+0000","last_peered":"2024-03-14T09:53:41.899095+0000","last_clean":"2024-03-14T09:53:41.899095+0000","last_became_active":"2024-03-14T09:52:05.736235+0000","last_became_peered":"2024-03-14T09:52:05.736235+0000","last_unstale":"2024-03-14T09:53:41.899095+0000","last_undegraded":"2024-03-14T09:53:41.899095+0000","last_fullsized":"2024-03-14T09:53:41.899095+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:46:53.293698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"83'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785979+0000","last_change":"2024-03-14T09:52:09.768203+0000","last_active":"2024-03-14T09:53:41.785979+0000","last_peered":"2024-03-14T09:53:41.785979+0000","last_clean":"2024-03-14T09:53:41.785979+0000","last_became_active":"2024-03-14T09:52:09.767777+0000","last_became_peered":"2024-03-14T09:52:09.767777+0000","last_unstale":"2024-03-14T09:53:41.785979+0000","last_undegraded":"2024-03-14T09:53:41.785979+0000","last_fullsized":"2024-03-14T09:53:41.785979+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:20:47.598905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"83'1","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783567+0000","last_change":"2024-03-14T09:52:05.232359+0000","last_active":"2024-03-14T09:53:41.783567+0000","last_peered":"2024-03-14T09:53:41.783567+0000","last_clean":"2024-03-14T09:53:41.783567+0000","last_became_active":"2024-03-14T09:52:05.231977+0000","last_became_peered":"2024-03-14T09:52:05.231977+0000","last_unstale":"2024-03-14T09:53:41.783567+0000","last_undegraded":"2024-03-14T09:53:41.783567+0000","last_fullsized":"2024-03-14T09:53:41.783567+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:32.678157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785461+0000","last_change":"2024-03-14T09:52:05.735822+0000","last_active":"2024-03-14T09:53:41.785461+0000","last_peered":"2024-03-14T09:53:41.785461+0000","last_clean":"2024-03-14T09:53:41.785461+0000","last_became_active":"2024-03-14T09:52:05.735059+0000","last_became_peered":"2024-03-14T09:52:05.735059+0000","last_unstale":"2024-03-14T09:53:41.785461+0000","last_undegraded":"2024-03-14T09:53:41.785461+0000","last_fullsized":"2024-03-14T09:53:41.785461+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T15:22:33.033689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786204+0000","last_change":"2024-03-14T09:52:09.759794+0000","last_active":"2024-03-14T09:53:41.786204+0000","last_peered":"2024-03-14T09:53:41.786204+0000","last_clean":"2024-03-14T09:53:41.786204+0000","last_became_active":"2024-03-14T09:52:09.759366+0000","last_became_peered":"2024-03-14T09:52:09.759366+0000","last_unstale":"2024-03-14T09:53:41.786204+0000","last_undegraded":"2024-03-14T09:53:41.786204+0000","last_fullsized":"2024-03-14T09:53:41.786204+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:52:40.641264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"83'4","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788103+0000","last_change":"2024-03-14T09:52:42.325733+0000","last_active":"2024-03-14T09:53:41.788103+0000","last_peered":"2024-03-14T09:53:41.788103+0000","last_clean":"2024-03-14T09:53:41.788103+0000","last_became_active":"2024-03-14T09:52:42.322002+0000","last_became_peered":"2024-03-14T09:52:42.322002+0000","last_unstale":"2024-03-14T09:53:41.788103+0000","last_undegraded":"2024-03-14T09:53:41.788103+0000","last_fullsized":"2024-03-14T09:53:41.788103+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:22:10.439271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783569+0000","last_change":"2024-03-14T09:52:05.734073+0000","last_active":"2024-03-14T09:53:41.783569+0000","last_peered":"2024-03-14T09:53:41.783569+0000","last_clean":"2024-03-14T09:53:41.783569+0000","last_became_active":"2024-03-14T09:52:05.731748+0000","last_became_peered":"2024-03-14T09:52:05.731748+0000","last_unstale":"2024-03-14T09:53:41.783569+0000","last_undegraded":"2024-03-14T09:53:41.783569+0000","last_fullsized":"2024-03-14T09:53:41.783569+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:01:11.789162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789540+0000","last_change":"2024-03-14T09:52:09.763309+0000","last_active":"2024-03-14T09:53:41.789540+0000","last_peered":"2024-03-14T09:53:41.789540+0000","last_clean":"2024-03-14T09:53:41.789540+0000","last_became_active":"2024-03-14T09:52:09.762868+0000","last_became_peered":"2024-03-14T09:52:09.762868+0000","last_unstale":"2024-03-14T09:53:41.789540+0000","last_undegraded":"2024-03-14T09:53:41.789540+0000","last_fullsized":"2024-03-14T09:53:41.789540+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:30:03.403911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"83'3","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.784042+0000","last_change":"2024-03-14T09:52:05.232310+0000","last_active":"2024-03-14T09:53:41.784042+0000","last_peered":"2024-03-14T09:53:41.784042+0000","last_clean":"2024-03-14T09:53:41.784042+0000","last_became_active":"2024-03-14T09:52:05.231931+0000","last_became_peered":"2024-03-14T09:52:05.231931+0000","last_unstale":"2024-03-14T09:53:41.784042+0000","last_undegraded":"2024-03-14T09:53:41.784042+0000","last_fullsized":"2024-03-14T09:53:41.784042+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:35:44.274800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783928+0000","last_change":"2024-03-14T09:52:05.731797+0000","last_active":"2024-03-14T09:53:41.783928+0000","last_peered":"2024-03-14T09:53:41.783928+0000","last_clean":"2024-03-14T09:53:41.783928+0000","last_became_active":"2024-03-14T09:52:05.730478+0000","last_became_peered":"2024-03-14T09:52:05.730478+0000","last_unstale":"2024-03-14T09:53:41.783928+0000","last_undegraded":"2024-03-14T09:53:41.783928+0000","last_fullsized":"2024-03-14T09:53:41.783928+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:52:50.684430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.898921+0000","last_change":"2024-03-14T09:52:09.772786+0000","last_active":"2024-03-14T09:53:41.898921+0000","last_peered":"2024-03-14T09:53:41.898921+0000","last_clean":"2024-03-14T09:53:41.898921+0000","last_became_active":"2024-03-14T09:52:09.772198+0000","last_became_peered":"2024-03-14T09:52:09.772198+0000","last_unstale":"2024-03-14T09:53:41.898921+0000","last_undegraded":"2024-03-14T09:53:41.898921+0000","last_fullsized":"2024-03-14T09:53:41.898921+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:28:56.620266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"78'1","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788052+0000","last_change":"2024-03-14T09:52:05.237657+0000","last_active":"2024-03-14T09:53:41.788052+0000","last_peered":"2024-03-14T09:53:41.788052+0000","last_clean":"2024-03-14T09:53:41.788052+0000","last_became_active":"2024-03-14T09:52:05.237381+0000","last_became_peered":"2024-03-14T09:52:05.237381+0000","last_unstale":"2024-03-14T09:53:41.788052+0000","last_undegraded":"2024-03-14T09:53:41.788052+0000","last_fullsized":"2024-03-14T09:53:41.788052+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:34:44.372128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783767+0000","last_change":"2024-03-14T09:52:05.730454+0000","last_active":"2024-03-14T09:53:41.783767+0000","last_peered":"2024-03-14T09:53:41.783767+0000","last_clean":"2024-03-14T09:53:41.783767+0000","last_became_active":"2024-03-14T09:52:05.730046+0000","last_became_peered":"2024-03-14T09:52:05.730046+0000","last_unstale":"2024-03-14T09:53:41.783767+0000","last_undegraded":"2024-03-14T09:53:41.783767+0000","last_fullsized":"2024-03-14T09:53:41.783767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:33:19.296993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786764+0000","last_change":"2024-03-14T09:52:09.755969+0000","last_active":"2024-03-14T09:53:41.786764+0000","last_peered":"2024-03-14T09:53:41.786764+0000","last_clean":"2024-03-14T09:53:41.786764+0000","last_became_active":"2024-03-14T09:52:09.755512+0000","last_became_peered":"2024-03-14T09:52:09.755512+0000","last_unstale":"2024-03-14T09:53:41.786764+0000","last_undegraded":"2024-03-14T09:53:41.786764+0000","last_fullsized":"2024-03-14T09:53:41.786764+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:31:03.388090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790735+0000","last_change":"2024-03-14T09:52:05.232676+0000","last_active":"2024-03-14T09:53:41.790735+0000","last_peered":"2024-03-14T09:53:41.790735+0000","last_clean":"2024-03-14T09:53:41.790735+0000","last_became_active":"2024-03-14T09:52:05.232332+0000","last_became_peered":"2024-03-14T09:52:05.232332+0000","last_unstale":"2024-03-14T09:53:41.790735+0000","last_undegraded":"2024-03-14T09:53:41.790735+0000","last_fullsized":"2024-03-14T09:53:41.790735+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:06:11.020555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783875+0000","last_change":"2024-03-14T09:52:05.733338+0000","last_active":"2024-03-14T09:53:41.783875+0000","last_peered":"2024-03-14T09:53:41.783875+0000","last_clean":"2024-03-14T09:53:41.783875+0000","last_became_active":"2024-03-14T09:52:05.732994+0000","last_became_peered":"2024-03-14T09:52:05.732994+0000","last_unstale":"2024-03-14T09:53:41.783875+0000","last_undegraded":"2024-03-14T09:53:41.783875+0000","last_fullsized":"2024-03-14T09:53:41.783875+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:31:25.085916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899170+0000","last_change":"2024-03-14T09:52:05.733121+0000","last_active":"2024-03-14T09:53:41.899170+0000","last_peered":"2024-03-14T09:53:41.899170+0000","last_clean":"2024-03-14T09:53:41.899170+0000","last_became_active":"2024-03-14T09:52:05.732671+0000","last_became_peered":"2024-03-14T09:52:05.732671+0000","last_unstale":"2024-03-14T09:53:41.899170+0000","last_undegraded":"2024-03-14T09:53:41.899170+0000","last_fullsized":"2024-03-14T09:53:41.899170+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:18:54.180131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"85'12","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783676+0000","last_change":"2024-03-14T09:52:05.736534+0000","last_active":"2024-03-14T09:53:41.783676+0000","last_peered":"2024-03-14T09:53:41.783676+0000","last_clean":"2024-03-14T09:53:41.783676+0000","last_became_active":"2024-03-14T09:52:05.736202+0000","last_became_peered":"2024-03-14T09:52:05.736202+0000","last_unstale":"2024-03-14T09:53:41.783676+0000","last_undegraded":"2024-03-14T09:53:41.783676+0000","last_fullsized":"2024-03-14T09:53:41.783676+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:22:46.161561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783472+0000","last_change":"2024-03-14T09:52:05.739434+0000","last_active":"2024-03-14T09:53:41.783472+0000","last_peered":"2024-03-14T09:53:41.783472+0000","last_clean":"2024-03-14T09:53:41.783472+0000","last_became_active":"2024-03-14T09:52:05.738023+0000","last_became_peered":"2024-03-14T09:52:05.738023+0000","last_unstale":"2024-03-14T09:53:41.783472+0000","last_undegraded":"2024-03-14T09:53:41.783472+0000","last_fullsized":"2024-03-14T09:53:41.783472+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:51:44.173645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899212+0000","last_change":"2024-03-14T09:52:05.751367+0000","last_active":"2024-03-14T09:53:41.899212+0000","last_peered":"2024-03-14T09:53:41.899212+0000","last_clean":"2024-03-14T09:53:41.899212+0000","last_became_active":"2024-03-14T09:52:05.750816+0000","last_became_peered":"2024-03-14T09:52:05.750816+0000","last_unstale":"2024-03-14T09:53:41.899212+0000","last_undegraded":"2024-03-14T09:53:41.899212+0000","last_fullsized":"2024-03-14T09:53:41.899212+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:57:58.403356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"85'4","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.791204+0000","last_change":"2024-03-14T09:52:05.746652+0000","last_active":"2024-03-14T09:53:41.791204+0000","last_peered":"2024-03-14T09:53:41.791204+0000","last_clean":"2024-03-14T09:53:41.791204+0000","last_became_active":"2024-03-14T09:52:05.746214+0000","last_became_peered":"2024-03-14T09:52:05.746214+0000","last_unstale":"2024-03-14T09:53:41.791204+0000","last_undegraded":"2024-03-14T09:53:41.791204+0000","last_fullsized":"2024-03-14T09:53:41.791204+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:21:39.244617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"85'6","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899490+0000","last_change":"2024-03-14T09:52:42.147977+0000","last_active":"2024-03-14T09:53:41.899490+0000","last_peered":"2024-03-14T09:53:41.899490+0000","last_clean":"2024-03-14T09:53:41.899490+0000","last_became_active":"2024-03-14T09:52:42.147623+0000","last_became_peered":"2024-03-14T09:52:42.147623+0000","last_unstale":"2024-03-14T09:53:41.899490+0000","last_undegraded":"2024-03-14T09:53:41.899490+0000","last_fullsized":"2024-03-14T09:53:41.899490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:38:35.351709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"85'15","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.784157+0000","last_change":"2024-03-14T09:52:05.739758+0000","last_active":"2024-03-14T09:53:41.784157+0000","last_peered":"2024-03-14T09:53:41.784157+0000","last_clean":"2024-03-14T09:53:41.784157+0000","last_became_active":"2024-03-14T09:52:05.739423+0000","last_became_peered":"2024-03-14T09:52:05.739423+0000","last_unstale":"2024-03-14T09:53:41.784157+0000","last_undegraded":"2024-03-14T09:53:41.784157+0000","last_fullsized":"2024-03-14T09:53:41.784157+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:47:22.594910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"85'2","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785881+0000","last_change":"2024-03-14T09:52:05.733281+0000","last_active":"2024-03-14T09:53:41.785881+0000","last_peered":"2024-03-14T09:53:41.785881+0000","last_clean":"2024-03-14T09:53:41.785881+0000","last_became_active":"2024-03-14T09:52:05.730081+0000","last_became_peered":"2024-03-14T09:52:05.730081+0000","last_unstale":"2024-03-14T09:53:41.785881+0000","last_undegraded":"2024-03-14T09:53:41.785881+0000","last_fullsized":"2024-03-14T09:53:41.785881+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:40:26.354063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786998+0000","last_change":"2024-03-14T09:52:05.726938+0000","last_active":"2024-03-14T09:53:41.786998+0000","last_peered":"2024-03-14T09:53:41.786998+0000","last_clean":"2024-03-14T09:53:41.786998+0000","last_became_active":"2024-03-14T09:52:05.726602+0000","last_became_peered":"2024-03-14T09:52:05.726602+0000","last_unstale":"2024-03-14T09:53:41.786998+0000","last_undegraded":"2024-03-14T09:53:41.786998+0000","last_fullsized":"2024-03-14T09:53:41.786998+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:26:20.354368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"85'8","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788428+0000","last_change":"2024-03-14T09:52:42.149962+0000","last_active":"2024-03-14T09:53:41.788428+0000","last_peered":"2024-03-14T09:53:41.788428+0000","last_clean":"2024-03-14T09:53:41.788428+0000","last_became_active":"2024-03-14T09:52:42.149726+0000","last_became_peered":"2024-03-14T09:52:42.149726+0000","last_unstale":"2024-03-14T09:53:41.788428+0000","last_undegraded":"2024-03-14T09:53:41.788428+0000","last_fullsized":"2024-03-14T09:53:41.788428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:20:18.294456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"85'10","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789837+0000","last_change":"2024-03-14T09:52:05.742945+0000","last_active":"2024-03-14T09:53:41.789837+0000","last_peered":"2024-03-14T09:53:41.789837+0000","last_clean":"2024-03-14T09:53:41.789837+0000","last_became_active":"2024-03-14T09:52:05.742483+0000","last_became_peered":"2024-03-14T09:52:05.742483+0000","last_unstale":"2024-03-14T09:53:41.789837+0000","last_undegraded":"2024-03-14T09:53:41.789837+0000","last_fullsized":"2024-03-14T09:53:41.789837+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:05:31.053608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"85'11","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790355+0000","last_change":"2024-03-14T09:52:05.737822+0000","last_active":"2024-03-14T09:53:41.790355+0000","last_peered":"2024-03-14T09:53:41.790355+0000","last_clean":"2024-03-14T09:53:41.790355+0000","last_became_active":"2024-03-14T09:52:05.735294+0000","last_became_peered":"2024-03-14T09:52:05.735294+0000","last_unstale":"2024-03-14T09:53:41.790355+0000","last_undegraded":"2024-03-14T09:53:41.790355+0000","last_fullsized":"2024-03-14T09:53:41.790355+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:06:28.437259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899359+0000","last_change":"2024-03-14T09:52:05.746991+0000","last_active":"2024-03-14T09:53:41.899359+0000","last_peered":"2024-03-14T09:53:41.899359+0000","last_clean":"2024-03-14T09:53:41.899359+0000","last_became_active":"2024-03-14T09:52:05.746558+0000","last_became_peered":"2024-03-14T09:52:05.746558+0000","last_unstale":"2024-03-14T09:53:41.899359+0000","last_undegraded":"2024-03-14T09:53:41.899359+0000","last_fullsized":"2024-03-14T09:53:41.899359+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:46:11.540134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247929,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71351,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":527,"num_read_kb":343,"num_write":168,"num_write_kb":127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":694,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":217357,"data_compressed":4812,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":147,"ondisk_log_size":147,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579936,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":489449,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.627,"15min":0.61599999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.681,"5min":2.681,"15min":2.681},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.59999999999999998,"15min":0.60899999999999999},"min":{"1min":0.379,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.64700000000000002,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.496}]},{"osd":1,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.64900000000000002,"15min":0.64400000000000002},"min":{"1min":0.57099999999999995,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.624},{"interface":"front","average":{"1min":0.623,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77300000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.47899999999999998}]},{"osd":2,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51300000000000001,"15min":0.51100000000000001},"min":{"1min":0.39900000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.77900000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.83099999999999996,"5min":0.88,"15min":0.88},"last":0.371}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.623,"15min":0.61699999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.68000000000000005},"min":{"1min":0.45800000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.81499999999999995,"5min":1.325,"15min":1.325},"last":0.64100000000000001}]},{"osd":5,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.748,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.44},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63800000000000001,"15min":0.64800000000000002},"min":{"1min":0.44600000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.77800000000000002,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.51100000000000001}]},{"osd":6,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70899999999999996,"15min":0.71499999999999997},"min":{"1min":0.53500000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97999999999999998,"5min":1.135,"15min":1.135},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.79800000000000004,"15min":0.79100000000000004},"min":{"1min":0.61799999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.7000000000000002,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.752,"15min":0.75800000000000001},"min":{"1min":0.56599999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.88800000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.75800000000000001,"15min":0.75700000000000001},"min":{"1min":0.57099999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.874,"5min":1.363,"15min":1.363},"last":0.69399999999999995}]},{"osd":8,"last update":"Thu Mar 14 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.73499999999999999}]},{"osd":9,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.81699999999999995,"15min":0.82499999999999996},"min":{"1min":0.626,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.96499999999999997,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.874,"15min":0.874},"min":{"1min":0.66400000000000003,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.66000000000000003}]},{"osd":10,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.81699999999999995,"15min":0.82099999999999995},"min":{"1min":0.65500000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.92500000000000004,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.78100000000000003,"15min":0.78500000000000003},"min":{"1min":0.60999999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.93500000000000005,"5min":1.448,"15min":1.448},"last":0.63300000000000001}]}]},{"osd":10,"up_from":67,"seq":287762808867,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":1144316,"data_compressed":61074,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46200000000000002,"15min":0.46400000000000002},"min":{"1min":0.32500000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46000000000000002,"15min":0.45900000000000002},"min":{"1min":0.29899999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.35599999999999998}]},{"osd":2,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57799999999999996,"15min":0.57999999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79400000000000004,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60199999999999998,"15min":0.60299999999999998},"min":{"1min":0.441,"5min":0.309,"15min":0.309},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.52300000000000002}]},{"osd":3,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.73399999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.63900000000000001,"15min":0.64100000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.80400000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.375,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.75800000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55500000000000005,"15min":0.56100000000000005},"min":{"1min":0.39200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77600000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.624,"15min":0.621},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.42299999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.622}]},{"osd":6,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.752,"15min":0.76100000000000001},"min":{"1min":0.51000000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0089999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.499,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.95899999999999996,"5min":1.05,"15min":1.05},"last":0.67600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":8,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.74099999999999999,"15min":0.74199999999999999},"min":{"1min":0.47499999999999998,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.75,"15min":0.75600000000000001},"min":{"1min":0.52300000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0009999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.56499999999999995}]},{"osd":9,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.79500000000000004,"15min":0.79900000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.76700000000000002,"15min":0.77400000000000002},"min":{"1min":0.55000000000000004,"5min":0.502,"15min":0.502},"max":{"1min":1.014,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.59499999999999997}]},{"osd":11,"last update":"Thu Mar 14 09:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.748,"15min":0.75},"min":{"1min":0.56399999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.754,"5min":0.75800000000000001,"15min":0.75900000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.73199999999999998}]}]},{"osd":9,"up_from":62,"seq":266287972392,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":493199,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.56899999999999995,"15min":0.57699999999999996},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.376},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.34999999999999998}]},{"osd":1,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68400000000000005,"15min":0.68000000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.504,"5min":2.504,"15min":2.504},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.73499999999999999,"15min":0.74299999999999999},"min":{"1min":0.50800000000000001,"5min":0.432,"15min":0.432},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":0.76800000000000002}]},{"osd":2,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.54200000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":2.3380000000000001,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.78000000000000003,"15min":0.79100000000000004},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.2360000000000002,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.71199999999999997,"15min":0.72499999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.0649999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.67600000000000005,"15min":0.68400000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.499}]},{"osd":4,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.81000000000000005,"15min":0.82399999999999995},"min":{"1min":0.52500000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.79700000000000004,"15min":0.81000000000000005},"min":{"1min":0.53000000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.54300000000000004}]},{"osd":5,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.85099999999999998,"15min":0.86699999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.83599999999999997,"15min":0.84199999999999997},"min":{"1min":0.57799999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":0.87}]},{"osd":6,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.88600000000000001,"15min":0.90600000000000003},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.90500000000000003,"15min":0.92100000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.69399999999999995}]},{"osd":8,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.92100000000000004,"15min":0.94099999999999995},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.93600000000000005,"15min":0.95399999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":0.84699999999999998}]},{"osd":10,"last update":"Thu Mar 14 09:53:09 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.88,"15min":0.89000000000000001},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.88500000000000001,"15min":0.89300000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":0.752}]},{"osd":11,"last update":"Thu Mar 14 09:53:09 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.569,"5min":2.569,"15min":2.569},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":2.266,"5min":2.266,"15min":2.266},"last":0.85599999999999998}]}]},{"osd":8,"up_from":55,"seq":236223201322,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":488692,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57099999999999995,"15min":0.58199999999999996},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66700000000000004,"5min":0.872,"15min":0.872},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.92500000000000004,"15min":1.081},"min":{"1min":0.38800000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80900000000000005,"5min":9.7769999999999992,"15min":9.7769999999999992},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.38600000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.748,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.59399999999999997,"15min":0.57399999999999995},"min":{"1min":0.46400000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.78900000000000003,"5min":0.877,"15min":0.877},"last":0.65900000000000003}]},{"osd":2,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.44900000000000001,"15min":0.432},"min":{"1min":0.33000000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46500000000000002,"15min":0.45200000000000001},"min":{"1min":0.34699999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.71399999999999997,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.42099999999999999}]},{"osd":3,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61199999999999999,"15min":0.60599999999999998},"min":{"1min":0.46200000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80400000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.64100000000000001,"15min":0.623},"min":{"1min":0.47699999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59799999999999998,"15min":0.57599999999999996},"min":{"1min":0.52100000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82199999999999995,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.61499999999999999,"15min":0.59099999999999997},"min":{"1min":0.501,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84099999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.60399999999999998}]},{"osd":5,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.36199999999999999,"5min":0.312,"15min":0.312},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54100000000000004,"15min":0.52200000000000002},"min":{"1min":0.41299999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.67100000000000004}]},{"osd":6,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.68600000000000005,"15min":0.66800000000000004},"min":{"1min":0.63300000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.755},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.66700000000000004,"15min":0.65300000000000002},"min":{"1min":0.50900000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.64700000000000002}]},{"osd":7,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.65200000000000002,"15min":0.628},"min":{"1min":0.55600000000000005,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.69299999999999995,"15min":0.67400000000000004},"min":{"1min":0.60899999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.69899999999999995}]},{"osd":9,"last update":"Thu Mar 14 09:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.73299999999999998,"15min":0.72999999999999998},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.53800000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.745}]},{"osd":10,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.72499999999999998,"15min":0.71099999999999997},"min":{"1min":0.60999999999999999,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.58599999999999997,"5min":0.499,"15min":0.499},"max":{"1min":0.98099999999999998,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.72299999999999998}]},{"osd":11,"last update":"Thu Mar 14 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.76600000000000001}]}]},{"osd":7,"up_from":48,"seq":206158430255,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":1085098,"data_compressed":59470,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.40999999999999998}]},{"osd":1,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42999999999999999,"15min":0.42099999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.57999999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.42899999999999999,"15min":0.41299999999999998},"min":{"1min":0.28799999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.57399999999999995,"15min":0.56200000000000006},"min":{"1min":0.47599999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82299999999999995,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58199999999999996,"15min":0.56999999999999995},"min":{"1min":0.38900000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.748,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.52800000000000002}]},{"osd":3,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81200000000000006,"5min":0.872,"15min":0.872},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60399999999999998,"15min":0.59199999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82599999999999996,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.627}]},{"osd":4,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.502,"15min":0.48199999999999998},"min":{"1min":0.33700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.5,"15min":0.496},"min":{"1min":0.32700000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76600000000000001,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.60399999999999998,"15min":0.58099999999999996},"min":{"1min":0.47099999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86899999999999999,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60899999999999999}]},{"osd":6,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.65400000000000003,"15min":0.63800000000000001},"min":{"1min":0.51400000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.878,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.66100000000000003,"15min":0.65800000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85499999999999998,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.70899999999999996}]},{"osd":8,"last update":"Thu Mar 14 09:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.69999999999999996,"15min":0.69499999999999995},"min":{"1min":0.52800000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.95699999999999996,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.70099999999999996,"15min":0.70199999999999996},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.90100000000000002,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.65700000000000003}]},{"osd":9,"last update":"Thu Mar 14 09:53:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.75700000000000001,"15min":0.76300000000000001},"min":{"1min":0.55500000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.98299999999999998,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.77000000000000002,"15min":0.77400000000000002},"min":{"1min":0.61399999999999999,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.92300000000000004,"5min":1.008,"15min":1.008},"last":0.76500000000000001}]},{"osd":10,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.57099999999999995,"5min":0.434,"15min":0.434},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.72099999999999997,"15min":0.72499999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90400000000000003,"5min":1.044,"15min":1.044},"last":0.68999999999999995}]},{"osd":11,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.747},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.78700000000000003}]}]},{"osd":6,"up_from":43,"seq":184683593778,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":950272,"data_stored":484279,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.51200000000000001,"15min":0.48199999999999998},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.54,"5min":3.54,"15min":3.54},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.52300000000000002,"15min":0.497},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.9860000000000002,"5min":3.9860000000000002,"15min":3.9860000000000002},"last":0.47499999999999998}]},{"osd":1,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.70699999999999996,"15min":0.65900000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":3.7589999999999999,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":0.626},{"interface":"front","average":{"1min":0.75,"5min":0.64100000000000001,"15min":0.61099999999999999},"min":{"1min":0.52900000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.68799999999999994}]},{"osd":2,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.68400000000000005,"15min":0.64700000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":4.0190000000000001,"5min":4.0190000000000001,"15min":4.0190000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.72399999999999998,"15min":0.68600000000000005},"min":{"1min":0.46100000000000002,"5min":0.38,"15min":0.38},"max":{"1min":3.5790000000000002,"5min":3.5790000000000002,"15min":3.5790000000000002},"last":0.76000000000000001}]},{"osd":3,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.63800000000000001,"15min":0.60699999999999998},"min":{"1min":0.41999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.7959999999999998,"5min":3.7959999999999998,"15min":3.7959999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.55400000000000005,"15min":0.54600000000000004},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78300000000000003,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.499}]},{"osd":4,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.69599999999999995,"15min":0.65600000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.76700000000000002,"15min":0.72899999999999998},"min":{"1min":0.53000000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":3.8559999999999999,"5min":3.8559999999999999,"15min":3.8559999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.76500000000000001,"15min":0.77500000000000002},"min":{"1min":0.58799999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1120000000000001,"5min":3.1059999999999999,"15min":3.1059999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.746,"15min":0.72099999999999997},"min":{"1min":0.53500000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":4.0469999999999997,"5min":4.0469999999999997,"15min":4.0469999999999997},"last":0.69999999999999996}]},{"osd":7,"last update":"Thu Mar 14 09:53:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.71799999999999997,"15min":0.69599999999999995},"min":{"1min":0.59299999999999997,"5min":0.495,"15min":0.495},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.81599999999999995,"15min":0.78500000000000003},"min":{"1min":0.57099999999999995,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.8260000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.79300000000000004}]},{"osd":8,"last update":"Thu Mar 14 09:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.89300000000000002,"15min":0.91100000000000003},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":3.8860000000000001,"5min":3.8860000000000001,"15min":3.8860000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.79500000000000004,"15min":0.76800000000000002},"min":{"1min":0.54200000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":4.1079999999999997,"5min":4.1079999999999997,"15min":4.1079999999999997},"last":0.78200000000000003}]},{"osd":9,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.84999999999999998,"15min":0.86199999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.95299999999999996,"5min":1.119,"15min":1.119},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.79900000000000004,"15min":0.80500000000000005},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.77600000000000002}]},{"osd":10,"last update":"Thu Mar 14 09:53:14 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.84599999999999997,"15min":0.83499999999999996},"min":{"1min":0.68200000000000005,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.85399999999999998,"15min":0.82199999999999995},"min":{"1min":0.66400000000000003,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":4.1360000000000001,"5min":4.1360000000000001,"15min":4.1360000000000001},"last":0.749}]},{"osd":11,"last update":"Thu Mar 14 09:53:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":3.9489999999999998,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.82599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":551295,"data_compressed":25179,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.57099999999999995,"15min":0.56899999999999995},"min":{"1min":0.45000000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.877,"5min":1.917,"15min":1.917},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.66000000000000003,"15min":0.57299999999999995},"min":{"1min":0.437,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.877,"5min":7.6399999999999997,"15min":7.6399999999999997},"last":0.61099999999999999}]},{"osd":2,"last update":"Thu Mar 14 09:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56499999999999995,"15min":0.54900000000000004},"min":{"1min":0.42199999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76700000000000002,"5min":1.885,"15min":1.885},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61199999999999999,"15min":0.55000000000000004},"min":{"1min":0.438,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.88400000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"last":0.75}]},{"osd":3,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.59899999999999998,"15min":0.57699999999999996},"min":{"1min":0.39300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.1829999999999998,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.58699999999999997,"15min":0.57699999999999996},"min":{"1min":0.434,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.6299999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":0.51600000000000001}]},{"osd":4,"last update":"Thu Mar 14 09:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.39800000000000002,"15min":0.371},"min":{"1min":0.35799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.624,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.45900000000000002,"15min":0.40999999999999998},"min":{"1min":0.36399999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.64100000000000001,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Mar 14 09:53:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55800000000000005,"15min":0.496},"min":{"1min":0.40799999999999997,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70799999999999996,"5min":2.9849999999999999,"15min":2.9849999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.59499999999999997,"15min":0.53100000000000003},"min":{"1min":0.57199999999999995,"5min":0.309,"15min":0.309},"max":{"1min":0.86199999999999999,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.70999999999999996}]},{"osd":6,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.69899999999999995,"15min":0.64900000000000002},"min":{"1min":0.60199999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":3.7879999999999998,"5min":3.7879999999999998,"15min":3.7879999999999998},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.80400000000000005,"15min":0.85999999999999999},"min":{"1min":0.46500000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.5910000000000002,"5min":5.5229999999999997,"15min":5.5229999999999997},"last":0.72799999999999998}]},{"osd":7,"last update":"Thu Mar 14 09:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53200000000000003,"15min":0.496},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81499999999999995,"5min":2.331,"15min":2.331},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.50900000000000001,"15min":0.49099999999999999},"min":{"1min":0.39900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.753,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63500000000000001}]},{"osd":8,"last update":"Thu Mar 14 09:53:34 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.69899999999999995,"15min":0.629},"min":{"1min":0.54100000000000004,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.84199999999999997,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.68000000000000005,"15min":0.64700000000000002},"min":{"1min":0.57999999999999996,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.95399999999999996,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.76000000000000001}]},{"osd":9,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":0.76300000000000001,"15min":0.70599999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":6.5149999999999997,"5min":6.5149999999999997,"15min":6.5149999999999997},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.359,"5min":0.83799999999999997,"15min":0.751},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":9.8900000000000006,"5min":9.8900000000000006,"15min":9.8900000000000006},"last":0.73799999999999999}]},{"osd":10,"last update":"Thu Mar 14 09:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.68600000000000005,"15min":0.68200000000000005},"min":{"1min":0.61699999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.85999999999999999,"5min":0.877,"15min":0.877},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.73599999999999999,"15min":0.72999999999999998},"min":{"1min":0.64600000000000002,"5min":0.497,"15min":0.497},"max":{"1min":0.88500000000000001,"5min":1.171,"15min":1.171},"last":0.97699999999999998}]},{"osd":11,"last update":"Thu Mar 14 09:53:09 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.387,"5min":2.387,"15min":2.387},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.018}]}]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":485667,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58699999999999997,"15min":0.53500000000000003},"min":{"1min":0.40500000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85199999999999998,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.626,"15min":0.56100000000000005},"min":{"1min":0.42599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.998,"5min":2.056,"15min":2.056},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Mar 14 09:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.69199999999999995,"15min":0.57799999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.94799999999999995,"5min":5.1349999999999998,"15min":5.1349999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.68999999999999995,"15min":0.58999999999999997},"min":{"1min":0.52100000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0880000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":0.95699999999999996}]},{"osd":3,"last update":"Thu Mar 14 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":0.998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.1040000000000001}]},{"osd":4,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.71799999999999997,"15min":0.66800000000000004},"min":{"1min":0.47999999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.871,"5min":3.8220000000000001,"15min":3.8220000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.66600000000000004,"15min":0.59999999999999998},"min":{"1min":0.496,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":5.2949999999999999,"5min":5.2949999999999999,"15min":5.2949999999999999},"last":0.75800000000000001}]},{"osd":5,"last update":"Thu Mar 14 09:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.76500000000000001,"15min":0.68700000000000006},"min":{"1min":0.58099999999999996,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1279999999999999,"5min":3.863,"15min":3.863},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69599999999999995,"15min":0.67700000000000005},"min":{"1min":0.42999999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95799999999999996,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.56599999999999995}]},{"osd":6,"last update":"Thu Mar 14 09:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.65600000000000003,"15min":0.60999999999999999},"min":{"1min":0.41299999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2370000000000001,"5min":5.2370000000000001,"15min":5.2370000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.76500000000000001,"15min":0.72499999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.5890000000000004,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":1.2310000000000001}]},{"osd":7,"last update":"Thu Mar 14 09:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.4449999999999998,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.0669999999999999},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":3.544,"5min":3.544,"15min":3.544},"last":0.59499999999999997}]},{"osd":8,"last update":"Thu Mar 14 09:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.84799999999999998,"15min":0.83599999999999997},"min":{"1min":0.71499999999999997,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.385,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":1.1559999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.79200000000000004,"15min":0.79000000000000004},"min":{"1min":0.51500000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.92700000000000005,"5min":3.6709999999999998,"15min":3.6709999999999998},"last":0.625}]},{"osd":9,"last update":"Thu Mar 14 09:52:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.88600000000000001,"15min":0.83999999999999997},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":6.7699999999999996,"5min":6.7699999999999996,"15min":6.7699999999999996},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.2110000000000001,"5min":0.93100000000000005,"15min":0.88400000000000001},"min":{"1min":0.45400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":6.0199999999999996,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":1.1950000000000001}]},{"osd":10,"last update":"Thu Mar 14 09:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.82399999999999995,"15min":0.81200000000000006},"min":{"1min":0.627,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.6339999999999999,"5min":3.6339999999999999,"15min":3.6339999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.88,"15min":0.84399999999999997},"min":{"1min":0.68700000000000006,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":5.3399999999999999,"5min":5.3399999999999999,"15min":5.3399999999999999},"last":0.80900000000000005}]},{"osd":11,"last update":"Thu Mar 14 09:53:27 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.91200000000000003,"15min":0.90800000000000003},"min":{"1min":0.77100000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.329,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":1.2629999999999999},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.91700000000000004,"15min":0.92900000000000005},"min":{"1min":0.64300000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.1220000000000001,"5min":3.71,"15min":3.71},"last":0.73099999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":552373,"data_compressed":25179,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53300000000000003,"15min":0.51800000000000002},"min":{"1min":0.37,"5min":0.31,"15min":0.31},"max":{"1min":0.76800000000000002,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.5},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54900000000000004,"15min":0.53000000000000003},"min":{"1min":0.435,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.83999999999999997,"5min":0.879,"15min":0.879},"last":0.68999999999999995}]},{"osd":1,"last update":"Thu Mar 14 09:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.59799999999999998,"15min":0.53900000000000003},"min":{"1min":0.44900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.91100000000000003,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56299999999999994,"15min":0.54400000000000004},"min":{"1min":0.44400000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.70199999999999996,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.57899999999999996}]},{"osd":3,"last update":"Thu Mar 14 09:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.55000000000000004,"15min":0.52600000000000002},"min":{"1min":0.42599999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56999999999999995,"15min":0.54900000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.66200000000000003}]},{"osd":4,"last update":"Thu Mar 14 09:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.624}]},{"osd":5,"last update":"Thu Mar 14 09:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.48699999999999999,"15min":0.45100000000000001},"min":{"1min":0.41699999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.82899999999999996,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.50600000000000001,"15min":0.48199999999999998},"min":{"1min":0.32800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.76100000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.52300000000000002}]},{"osd":6,"last update":"Thu Mar 14 09:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.66100000000000003,"15min":0.65000000000000002},"min":{"1min":0.55100000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.67600000000000005,"15min":0.66300000000000003},"min":{"1min":0.56699999999999995,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.85799999999999998}]},{"osd":7,"last update":"Thu Mar 14 09:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.81000000000000005}]},{"osd":8,"last update":"Thu Mar 14 09:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.64500000000000002,"15min":0.628},"min":{"1min":0.505,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.66000000000000003,"15min":0.64300000000000002},"min":{"1min":0.41499999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.745}]},{"osd":9,"last update":"Thu Mar 14 09:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.63900000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.75700000000000001,"15min":0.746},"min":{"1min":0.59299999999999997,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.83399999999999996}]},{"osd":10,"last update":"Thu Mar 14 09:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73399999999999999,"15min":0.73299999999999998},"min":{"1min":0.56899999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.91800000000000004,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.76400000000000001,"15min":0.754},"min":{"1min":0.59799999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1,"5min":1.008,"15min":1.008},"last":0.872}]},{"osd":11,"last update":"Thu Mar 14 09:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.84799999999999998}]}]},{"osd":3,"up_from":24,"seq":103079215167,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":484473,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.42199999999999999,"15min":0.40100000000000002},"min":{"1min":0.32300000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.44600000000000001,"15min":0.42999999999999999},"min":{"1min":0.373,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.66900000000000004}]},{"osd":1,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61099999999999999,"15min":0.59099999999999997},"min":{"1min":0.39400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.82799999999999996,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60699999999999998,"15min":0.58499999999999996},"min":{"1min":0.437,"5min":0.374,"15min":0.374},"max":{"1min":0.76300000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.60299999999999998}]},{"osd":2,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63400000000000001,"15min":0.60699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84099999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64000000000000001,"15min":0.623},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.80400000000000005,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.80300000000000005}]},{"osd":4,"last update":"Thu Mar 14 09:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.624,"15min":0.59799999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77200000000000002,"5min":1.091,"15min":1.091},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.629,"15min":0.60399999999999998},"min":{"1min":0.54000000000000004,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81999999999999995,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.50600000000000001}]},{"osd":5,"last update":"Thu Mar 14 09:53:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.67600000000000005,"15min":0.67900000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83799999999999997,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.46400000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.80200000000000005,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.78000000000000003}]},{"osd":6,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64800000000000002,"15min":0.65000000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86599999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65800000000000003,"15min":0.64300000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.98799999999999999,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.63500000000000001}]},{"osd":7,"last update":"Thu Mar 14 09:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.72999999999999998,"15min":0.73799999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.88900000000000001,"5min":1.123,"15min":1.123},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.748,"5min":0.76600000000000001,"15min":0.76800000000000002},"min":{"1min":0.64100000000000001,"5min":0.498,"15min":0.498},"max":{"1min":0.89200000000000002,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.753}]},{"osd":8,"last update":"Thu Mar 14 09:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.871}]},{"osd":9,"last update":"Thu Mar 14 09:52:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.80000000000000004,"15min":0.81499999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.98099999999999998,"5min":1.097,"15min":1.097},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.83999999999999997,"5min":1.121,"15min":1.121},"last":0.83399999999999996}]},{"osd":10,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.86899999999999999,"15min":0.88500000000000001},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":0.90200000000000002,"5min":1.127,"15min":1.127},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.83099999999999996,"15min":0.83899999999999997},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.97699999999999998,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.79100000000000004}]},{"osd":11,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.86399999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986234,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":489724,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.40100000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71599999999999997,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.52300000000000002,"15min":0.51600000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:53:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.36599999999999999,"15min":0.35399999999999998},"min":{"1min":0.27000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.37,"15min":0.36399999999999999},"min":{"1min":0.33000000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.51300000000000001,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.61099999999999999}]},{"osd":2,"last update":"Thu Mar 14 09:53:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56499999999999995,"15min":0.53800000000000003},"min":{"1min":0.38300000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67200000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.55600000000000005,"15min":0.52400000000000002},"min":{"1min":0.43099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.69199999999999995}]},{"osd":3,"last update":"Thu Mar 14 09:53:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.44,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38500000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.748,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.67800000000000005}]},{"osd":5,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59499999999999997,"15min":0.53300000000000003},"min":{"1min":0.44600000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78300000000000003,"5min":4.3479999999999999,"15min":4.3479999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55000000000000004,"15min":0.53100000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.73699999999999999,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.72399999999999998}]},{"osd":6,"last update":"Thu Mar 14 09:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.48099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73499999999999999,"5min":0.876,"15min":0.876},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.60899999999999999,"15min":0.59999999999999998},"min":{"1min":0.42799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.80500000000000005,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.85799999999999998}]},{"osd":7,"last update":"Thu Mar 14 09:53:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.50800000000000001,"15min":0.48799999999999999},"min":{"1min":0.34499999999999997,"5min":0.318,"15min":0.318},"max":{"1min":0.72499999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.47199999999999998,"15min":0.45600000000000002},"min":{"1min":0.36599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.71299999999999997}]},{"osd":8,"last update":"Thu Mar 14 09:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.65600000000000003,"15min":0.63700000000000001},"min":{"1min":0.57499999999999996,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.623,"15min":0.60499999999999998},"min":{"1min":0.47599999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82699999999999996,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.873}]},{"osd":9,"last update":"Thu Mar 14 09:52:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.47599999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.75,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71299999999999997,"15min":0.70899999999999996},"min":{"1min":0.61899999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.84099999999999997,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.96899999999999997}]},{"osd":10,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.63300000000000001,"5min":0.502,"15min":0.502},"max":{"1min":0.82799999999999996,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66500000000000004,"15min":0.66300000000000003},"min":{"1min":0.502,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.77900000000000003,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.90500000000000003}]},{"osd":11,"last update":"Thu Mar 14 09:53:07 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.84599999999999997}]}]},{"osd":5,"up_from":36,"seq":154618822711,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":1080081,"data_compressed":59470,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54500000000000004,"15min":0.53000000000000003},"min":{"1min":0.40400000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70799999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.372,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.625,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.48399999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.51900000000000002},"min":{"1min":0.372,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81899999999999995,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56599999999999995,"15min":0.52500000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75600000000000001,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.57299999999999995}]},{"osd":2,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.41599999999999998,"15min":0.38600000000000001},"min":{"1min":0.27300000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.70699999999999996,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.42599999999999999,"15min":0.39000000000000001},"min":{"1min":0.248,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.67400000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71199999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.58799999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.77700000000000002,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56399999999999995}]},{"osd":4,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59399999999999997,"15min":0.56000000000000005},"min":{"1min":0.48699999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80200000000000005,"5min":0.87,"15min":0.87},"last":0.626},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56299999999999994,"15min":0.53900000000000003},"min":{"1min":0.42699999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.76300000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.53900000000000003}]},{"osd":6,"last update":"Thu Mar 14 09:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.628,"15min":0.626},"min":{"1min":0.51800000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.79100000000000004,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.64000000000000001,"15min":0.627},"min":{"1min":0.44600000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.82799999999999996,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.52400000000000002}]},{"osd":7,"last update":"Thu Mar 14 09:53:16 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.63300000000000001,"15min":0.60099999999999998},"min":{"1min":0.48399999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.63,"15min":0.59599999999999997},"min":{"1min":0.56299999999999994,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.89600000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.64200000000000002}]},{"osd":8,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60399999999999998,"15min":0.59399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56499999999999995,"15min":0.56399999999999995},"min":{"1min":0.33400000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70799999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.58299999999999996}]},{"osd":9,"last update":"Thu Mar 14 09:52:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.72099999999999997,"15min":0.71199999999999997},"min":{"1min":0.56799999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96299999999999997,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.74299999999999999,"15min":0.73699999999999999},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.68600000000000005}]},{"osd":10,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.69999999999999996,"15min":0.69599999999999995},"min":{"1min":0.58699999999999997,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.75800000000000001,"15min":0.753},"min":{"1min":0.60899999999999999,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.70099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34227,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34227,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":69626,"data_compressed":1604,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68184,"data_compressed":1604,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":69759,"data_compressed":1604,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-14T09:53:56.609 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-14T09:53:56.609 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-14T09:53:56.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.837+0000 7fc1bb800700 1 -- 172.21.15.78:0/3386750050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1b4103fc0 msgr2=0x7fc1b4104440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:56.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.837+0000 7fc1bb800700 1 --2- 172.21.15.78:0/3386750050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1b4103fc0 0x7fc1b4104440 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0009990 tx=0x7fc1a0009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.837+0000 7fc1bb800700 1 -- 172.21.15.78:0/3386750050 shutdown_connections 2024-03-14T09:53:56.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.837+0000 7fc1bb800700 1 --2- 172.21.15.78:0/3386750050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1b4104980 0x7fc1b410b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.837+0000 7fc1bb800700 1 --2- 172.21.15.78:0/3386750050 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1b4102e70 0x7fc1b4103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.837+0000 7fc1bb800700 1 --2- 172.21.15.78:0/3386750050 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1b4103fc0 0x7fc1b4104440 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.837+0000 7fc1bb800700 1 -- 172.21.15.78:0/3386750050 >> 172.21.15.78:0/3386750050 conn(0x7fc1b40fe3e0 msgr2=0x7fc1b4100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:56.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.837+0000 7fc1bb800700 1 -- 172.21.15.78:0/3386750050 shutdown_connections 2024-03-14T09:53:56.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.837+0000 7fc1bb800700 1 -- 172.21.15.78:0/3386750050 wait complete. 2024-03-14T09:53:56.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.838+0000 7fc1bb800700 1 Processor -- start 2024-03-14T09:53:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.838+0000 7fc1bb800700 1 -- start start 2024-03-14T09:53:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1bb800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1b4102e70 0x7fc1b4199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1bb800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1b4103fc0 0x7fc1b4199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1bb800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1b4104980 0x7fc1b41a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1bb800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b41a0d40 con 0x7fc1b4104980 2024-03-14T09:53:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1bb800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b41a0e80 con 0x7fc1b4102e70 2024-03-14T09:53:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1bb800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b41a1010 con 0x7fc1b4103fc0 2024-03-14T09:53:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1b9800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1b4104980 0x7fc1b41a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1b9800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1b4104980 0x7fc1b41a0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43954/0 (socket says 172.21.15.78:43954) 2024-03-14T09:53:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1b3e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1b4103fc0 0x7fc1b4199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1b9800700 1 -- 172.21.15.78:0/1507262091 learned_addr learned my addr 172.21.15.78:0/1507262091 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1b8e00700 1 --2- 172.21.15.78:0/1507262091 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1b4102e70 0x7fc1b4199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1b9800700 1 -- 172.21.15.78:0/1507262091 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1b4103fc0 msgr2=0x7fc1b4199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:56.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1b9800700 1 --2- 172.21.15.78:0/1507262091 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1b4103fc0 0x7fc1b4199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1b9800700 1 -- 172.21.15.78:0/1507262091 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1b4102e70 msgr2=0x7fc1b4199210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:56.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1b9800700 1 --2- 172.21.15.78:0/1507262091 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1b4102e70 0x7fc1b4199210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:56.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1b9800700 1 -- 172.21.15.78:0/1507262091 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a00095f0 con 0x7fc1b4104980 2024-03-14T09:53:56.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.839+0000 7fc1b9800700 1 --2- 172.21.15.78:0/1507262091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1b4104980 0x7fc1b41a0770 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac00ef30 tx=0x7fc1ac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:56.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.840+0000 7fc1b1600700 1 -- 172.21.15.78:0/1507262091 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac018070 con 0x7fc1b4104980 2024-03-14T09:53:56.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.840+0000 7fc1b1600700 1 -- 172.21.15.78:0/1507262091 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc1ac00f040 con 0x7fc1b4104980 2024-03-14T09:53:56.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.840+0000 7fc1b1600700 1 -- 172.21.15.78:0/1507262091 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac00cd00 con 0x7fc1b4104980 2024-03-14T09:53:56.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.840+0000 7fc1bb800700 1 -- 172.21.15.78:0/1507262091 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b41a1290 con 0x7fc1b4104980 2024-03-14T09:53:56.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.840+0000 7fc1bb800700 1 -- 172.21.15.78:0/1507262091 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b41a17a0 con 0x7fc1b4104980 2024-03-14T09:53:56.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.841+0000 7fc1bb800700 1 -- 172.21.15.78:0/1507262091 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b410a230 con 0x7fc1b4104980 2024-03-14T09:53:56.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.843+0000 7fc1b1600700 1 -- 172.21.15.78:0/1507262091 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fc1ac003a00 con 0x7fc1b4104980 2024-03-14T09:53:56.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.844+0000 7fc1b1600700 1 --2- 172.21.15.78:0/1507262091 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc19c0a0ba0 0x7fc19c0a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:56.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.844+0000 7fc1b1600700 1 -- 172.21.15.78:0/1507262091 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7fc1ac0cbfe0 con 0x7fc1b4104980 2024-03-14T09:53:56.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.844+0000 7fc1b8e00700 1 --2- 172.21.15.78:0/1507262091 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc19c0a0ba0 0x7fc19c0a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:56.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.845+0000 7fc1b8e00700 1 --2- 172.21.15.78:0/1507262091 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc19c0a0ba0 0x7fc19c0a3060 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8009c60 tx=0x7fc1a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:56.848+0000 7fc1b1600700 1 -- 172.21.15.78:0/1507262091 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc1ac0ff960 con 0x7fc1b4104980 2024-03-14T09:53:57.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.052+0000 7fc1bb800700 1 -- 172.21.15.78:0/1507262091 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1b402ce10 con 0x7fc19c0a0ba0 2024-03-14T09:53:57.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.065+0000 7fc1b1600700 1 -- 172.21.15.78:0/1507262091 <== mgr.14152 v2:172.21.15.78:6800/2580616009 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427290 (secure 0 0 0) 0x7fc1b402ce10 con 0x7fc19c0a0ba0 2024-03-14T09:53:57.071 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:57.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.070+0000 7fc1bb800700 1 -- 172.21.15.78:0/1507262091 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc19c0a0ba0 msgr2=0x7fc19c0a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:57.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.070+0000 7fc1bb800700 1 --2- 172.21.15.78:0/1507262091 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc19c0a0ba0 0x7fc19c0a3060 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8009c60 tx=0x7fc1a8009260 comp rx=0 tx=0).stop 2024-03-14T09:53:57.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.070+0000 7fc1bb800700 1 -- 172.21.15.78:0/1507262091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1b4104980 msgr2=0x7fc1b41a0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:57.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.070+0000 7fc1bb800700 1 --2- 172.21.15.78:0/1507262091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1b4104980 0x7fc1b41a0770 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac00ef30 tx=0x7fc1ac00c3a0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.070+0000 7fc1bb800700 1 -- 172.21.15.78:0/1507262091 shutdown_connections 2024-03-14T09:53:57.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.070+0000 7fc1bb800700 1 --2- 172.21.15.78:0/1507262091 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fc19c0a0ba0 0x7fc19c0a3060 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.070+0000 7fc1bb800700 1 --2- 172.21.15.78:0/1507262091 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1b4102e70 0x7fc1b4199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.070+0000 7fc1bb800700 1 --2- 172.21.15.78:0/1507262091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1b4104980 0x7fc1b41a0770 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.070+0000 7fc1bb800700 1 --2- 172.21.15.78:0/1507262091 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc1b4103fc0 0x7fc1b4199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.070+0000 7fc1bb800700 1 -- 172.21.15.78:0/1507262091 >> 172.21.15.78:0/1507262091 conn(0x7fc1b40fe3e0 msgr2=0x7fc1b4107370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:57.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.070+0000 7fc1bb800700 1 -- 172.21.15.78:0/1507262091 shutdown_connections 2024-03-14T09:53:57.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.070+0000 7fc1bb800700 1 -- 172.21.15.78:0/1507262091 wait complete. 2024-03-14T09:53:57.076 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-03-14T09:53:57.089 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-03-14T09:53:55.097758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661953,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":663,"num_read_kb":462,"num_write":351,"num_write_kb":3067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1181430,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":786,"ondisk_log_size":786,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":335892,"kb_used_data":11680,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124605932,"statfs":{"total":1151940427776,"available":1151596474368,"internally_reserved":0,"allocated":11960320,"data_stored":7828646,"data_compressed":395397,"data_compressed_allocated":540672,"data_compressed_original":2580480,"omap_allocated":19131,"internal_metadata":331855173},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5716,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":85,"num_read_kb":54,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003232"},"pg_stats":[{"pgid":"3.3f","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899578+0000","last_change":"2024-03-14T09:52:05.737617+0000","last_active":"2024-03-14T09:53:41.899578+0000","last_peered":"2024-03-14T09:53:41.899578+0000","last_clean":"2024-03-14T09:53:41.899578+0000","last_became_active":"2024-03-14T09:52:05.736796+0000","last_became_peered":"2024-03-14T09:52:05.736796+0000","last_unstale":"2024-03-14T09:53:41.899578+0000","last_undegraded":"2024-03-14T09:53:41.899578+0000","last_fullsized":"2024-03-14T09:53:41.899578+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:47:52.437748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"85'4","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899167+0000","last_change":"2024-03-14T09:52:05.749474+0000","last_active":"2024-03-14T09:53:41.899167+0000","last_peered":"2024-03-14T09:53:41.899167+0000","last_clean":"2024-03-14T09:53:41.899167+0000","last_became_active":"2024-03-14T09:52:05.748927+0000","last_became_peered":"2024-03-14T09:52:05.748927+0000","last_unstale":"2024-03-14T09:53:41.899167+0000","last_undegraded":"2024-03-14T09:53:41.899167+0000","last_fullsized":"2024-03-14T09:53:41.899167+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:28:15.852021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899117+0000","last_change":"2024-03-14T09:52:05.745098+0000","last_active":"2024-03-14T09:53:41.899117+0000","last_peered":"2024-03-14T09:53:41.899117+0000","last_clean":"2024-03-14T09:53:41.899117+0000","last_became_active":"2024-03-14T09:52:05.744755+0000","last_became_peered":"2024-03-14T09:52:05.744755+0000","last_unstale":"2024-03-14T09:53:41.899117+0000","last_undegraded":"2024-03-14T09:53:41.899117+0000","last_fullsized":"2024-03-14T09:53:41.899117+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:35:41.932386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786320+0000","last_change":"2024-03-14T09:52:05.737429+0000","last_active":"2024-03-14T09:53:41.786320+0000","last_peered":"2024-03-14T09:53:41.786320+0000","last_clean":"2024-03-14T09:53:41.786320+0000","last_became_active":"2024-03-14T09:52:05.736516+0000","last_became_peered":"2024-03-14T09:52:05.736516+0000","last_unstale":"2024-03-14T09:53:41.786320+0000","last_undegraded":"2024-03-14T09:53:41.786320+0000","last_fullsized":"2024-03-14T09:53:41.786320+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:48:38.158783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"85'4","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788000+0000","last_change":"2024-03-14T09:52:05.729634+0000","last_active":"2024-03-14T09:53:41.788000+0000","last_peered":"2024-03-14T09:53:41.788000+0000","last_clean":"2024-03-14T09:53:41.788000+0000","last_became_active":"2024-03-14T09:52:05.727889+0000","last_became_peered":"2024-03-14T09:52:05.727889+0000","last_unstale":"2024-03-14T09:53:41.788000+0000","last_undegraded":"2024-03-14T09:53:41.788000+0000","last_fullsized":"2024-03-14T09:53:41.788000+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:01:13.216763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"85'6","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787456+0000","last_change":"2024-03-14T09:52:42.147610+0000","last_active":"2024-03-14T09:53:41.787456+0000","last_peered":"2024-03-14T09:53:41.787456+0000","last_clean":"2024-03-14T09:53:41.787456+0000","last_became_active":"2024-03-14T09:52:42.147262+0000","last_became_peered":"2024-03-14T09:52:42.147262+0000","last_unstale":"2024-03-14T09:53:41.787456+0000","last_undegraded":"2024-03-14T09:53:41.787456+0000","last_fullsized":"2024-03-14T09:53:41.787456+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:20:49.855564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783351+0000","last_change":"2024-03-14T09:52:05.727365+0000","last_active":"2024-03-14T09:53:41.783351+0000","last_peered":"2024-03-14T09:53:41.783351+0000","last_clean":"2024-03-14T09:53:41.783351+0000","last_became_active":"2024-03-14T09:52:05.726984+0000","last_became_peered":"2024-03-14T09:52:05.726984+0000","last_unstale":"2024-03-14T09:53:41.783351+0000","last_undegraded":"2024-03-14T09:53:41.783351+0000","last_fullsized":"2024-03-14T09:53:41.783351+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:42:57.129718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785018+0000","last_change":"2024-03-14T09:52:05.729622+0000","last_active":"2024-03-14T09:53:41.785018+0000","last_peered":"2024-03-14T09:53:41.785018+0000","last_clean":"2024-03-14T09:53:41.785018+0000","last_became_active":"2024-03-14T09:52:05.729293+0000","last_became_peered":"2024-03-14T09:52:05.729293+0000","last_unstale":"2024-03-14T09:53:41.785018+0000","last_undegraded":"2024-03-14T09:53:41.785018+0000","last_fullsized":"2024-03-14T09:53:41.785018+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:13:20.875857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"85'11","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790229+0000","last_change":"2024-03-14T09:52:05.741620+0000","last_active":"2024-03-14T09:53:41.790229+0000","last_peered":"2024-03-14T09:53:41.790229+0000","last_clean":"2024-03-14T09:53:41.790229+0000","last_became_active":"2024-03-14T09:52:05.734079+0000","last_became_peered":"2024-03-14T09:52:05.734079+0000","last_unstale":"2024-03-14T09:53:41.790229+0000","last_undegraded":"2024-03-14T09:53:41.790229+0000","last_fullsized":"2024-03-14T09:53:41.790229+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:02:35.529671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"85'16","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788064+0000","last_change":"2024-03-14T09:52:05.743424+0000","last_active":"2024-03-14T09:53:41.788064+0000","last_peered":"2024-03-14T09:53:41.788064+0000","last_clean":"2024-03-14T09:53:41.788064+0000","last_became_active":"2024-03-14T09:52:05.743100+0000","last_became_peered":"2024-03-14T09:52:05.743100+0000","last_unstale":"2024-03-14T09:53:41.788064+0000","last_undegraded":"2024-03-14T09:53:41.788064+0000","last_fullsized":"2024-03-14T09:53:41.788064+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:32:33.492163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783875+0000","last_change":"2024-03-14T09:52:05.735654+0000","last_active":"2024-03-14T09:53:41.783875+0000","last_peered":"2024-03-14T09:53:41.783875+0000","last_clean":"2024-03-14T09:53:41.783875+0000","last_became_active":"2024-03-14T09:52:05.735371+0000","last_became_peered":"2024-03-14T09:52:05.735371+0000","last_unstale":"2024-03-14T09:53:41.783875+0000","last_undegraded":"2024-03-14T09:53:41.783875+0000","last_fullsized":"2024-03-14T09:53:41.783875+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:19:50.011554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"85'7","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789097+0000","last_change":"2024-03-14T09:52:42.146700+0000","last_active":"2024-03-14T09:53:41.789097+0000","last_peered":"2024-03-14T09:53:41.789097+0000","last_clean":"2024-03-14T09:53:41.789097+0000","last_became_active":"2024-03-14T09:52:42.145377+0000","last_became_peered":"2024-03-14T09:52:42.145377+0000","last_unstale":"2024-03-14T09:53:41.789097+0000","last_undegraded":"2024-03-14T09:53:41.789097+0000","last_fullsized":"2024-03-14T09:53:41.789097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:35:52.735091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"85'10","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790265+0000","last_change":"2024-03-14T09:52:05.740209+0000","last_active":"2024-03-14T09:53:41.790265+0000","last_peered":"2024-03-14T09:53:41.790265+0000","last_clean":"2024-03-14T09:53:41.790265+0000","last_became_active":"2024-03-14T09:52:05.732949+0000","last_became_peered":"2024-03-14T09:52:05.732949+0000","last_unstale":"2024-03-14T09:53:41.790265+0000","last_undegraded":"2024-03-14T09:53:41.790265+0000","last_fullsized":"2024-03-14T09:53:41.790265+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:15:56.377569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.898939+0000","last_change":"2024-03-14T09:52:05.748795+0000","last_active":"2024-03-14T09:53:41.898939+0000","last_peered":"2024-03-14T09:53:41.898939+0000","last_clean":"2024-03-14T09:53:41.898939+0000","last_became_active":"2024-03-14T09:52:05.746107+0000","last_became_peered":"2024-03-14T09:52:05.746107+0000","last_unstale":"2024-03-14T09:53:41.898939+0000","last_undegraded":"2024-03-14T09:53:41.898939+0000","last_fullsized":"2024-03-14T09:53:41.898939+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:49:07.932854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.898932+0000","last_change":"2024-03-14T09:52:05.744174+0000","last_active":"2024-03-14T09:53:41.898932+0000","last_peered":"2024-03-14T09:53:41.898932+0000","last_clean":"2024-03-14T09:53:41.898932+0000","last_became_active":"2024-03-14T09:52:05.743848+0000","last_became_peered":"2024-03-14T09:52:05.743848+0000","last_unstale":"2024-03-14T09:53:41.898932+0000","last_undegraded":"2024-03-14T09:53:41.898932+0000","last_fullsized":"2024-03-14T09:53:41.898932+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:07:23.260734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789366+0000","last_change":"2024-03-14T09:52:05.733677+0000","last_active":"2024-03-14T09:53:41.789366+0000","last_peered":"2024-03-14T09:53:41.789366+0000","last_clean":"2024-03-14T09:53:41.789366+0000","last_became_active":"2024-03-14T09:52:05.730967+0000","last_became_peered":"2024-03-14T09:52:05.730967+0000","last_unstale":"2024-03-14T09:53:41.789366+0000","last_undegraded":"2024-03-14T09:53:41.789366+0000","last_fullsized":"2024-03-14T09:53:41.789366+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:33:53.484741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899444+0000","last_change":"2024-03-14T09:52:05.737836+0000","last_active":"2024-03-14T09:53:41.899444+0000","last_peered":"2024-03-14T09:53:41.899444+0000","last_clean":"2024-03-14T09:53:41.899444+0000","last_became_active":"2024-03-14T09:52:05.737257+0000","last_became_peered":"2024-03-14T09:52:05.737257+0000","last_unstale":"2024-03-14T09:53:41.899444+0000","last_undegraded":"2024-03-14T09:53:41.899444+0000","last_fullsized":"2024-03-14T09:53:41.899444+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:53.723415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"85'10","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789627+0000","last_change":"2024-03-14T09:52:05.741360+0000","last_active":"2024-03-14T09:53:41.789627+0000","last_peered":"2024-03-14T09:53:41.789627+0000","last_clean":"2024-03-14T09:53:41.789627+0000","last_became_active":"2024-03-14T09:52:05.740673+0000","last_became_peered":"2024-03-14T09:52:05.740673+0000","last_unstale":"2024-03-14T09:53:41.789627+0000","last_undegraded":"2024-03-14T09:53:41.789627+0000","last_fullsized":"2024-03-14T09:53:41.789627+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:13:41.561338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.784072+0000","last_change":"2024-03-14T09:52:05.739134+0000","last_active":"2024-03-14T09:53:41.784072+0000","last_peered":"2024-03-14T09:53:41.784072+0000","last_clean":"2024-03-14T09:53:41.784072+0000","last_became_active":"2024-03-14T09:52:05.738439+0000","last_became_peered":"2024-03-14T09:52:05.738439+0000","last_unstale":"2024-03-14T09:53:41.784072+0000","last_undegraded":"2024-03-14T09:53:41.784072+0000","last_fullsized":"2024-03-14T09:53:41.784072+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:05:36.997462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783550+0000","last_change":"2024-03-14T09:52:09.769284+0000","last_active":"2024-03-14T09:53:41.783550+0000","last_peered":"2024-03-14T09:53:41.783550+0000","last_clean":"2024-03-14T09:53:41.783550+0000","last_became_active":"2024-03-14T09:52:09.768808+0000","last_became_peered":"2024-03-14T09:52:09.768808+0000","last_unstale":"2024-03-14T09:53:41.783550+0000","last_undegraded":"2024-03-14T09:53:41.783550+0000","last_fullsized":"2024-03-14T09:53:41.783550+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:48:56.015410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"83'9","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789540+0000","last_change":"2024-03-14T09:52:42.154303+0000","last_active":"2024-03-14T09:53:41.789540+0000","last_peered":"2024-03-14T09:53:41.789540+0000","last_clean":"2024-03-14T09:53:41.789540+0000","last_became_active":"2024-03-14T09:52:42.147756+0000","last_became_peered":"2024-03-14T09:52:42.147756+0000","last_unstale":"2024-03-14T09:53:41.789540+0000","last_undegraded":"2024-03-14T09:53:41.789540+0000","last_fullsized":"2024-03-14T09:53:41.789540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:01:27.630337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785328+0000","last_change":"2024-03-14T09:52:05.739004+0000","last_active":"2024-03-14T09:53:41.785328+0000","last_peered":"2024-03-14T09:53:41.785328+0000","last_clean":"2024-03-14T09:53:41.785328+0000","last_became_active":"2024-03-14T09:52:05.732217+0000","last_became_peered":"2024-03-14T09:52:05.732217+0000","last_unstale":"2024-03-14T09:53:41.785328+0000","last_undegraded":"2024-03-14T09:53:41.785328+0000","last_fullsized":"2024-03-14T09:53:41.785328+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:02:21.926623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785929+0000","last_change":"2024-03-14T09:52:09.776128+0000","last_active":"2024-03-14T09:53:41.785929+0000","last_peered":"2024-03-14T09:53:41.785929+0000","last_clean":"2024-03-14T09:53:41.785929+0000","last_became_active":"2024-03-14T09:52:09.775701+0000","last_became_peered":"2024-03-14T09:52:09.775701+0000","last_unstale":"2024-03-14T09:53:41.785929+0000","last_undegraded":"2024-03-14T09:53:41.785929+0000","last_fullsized":"2024-03-14T09:53:41.785929+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:41:51.790627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"83'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785672+0000","last_change":"2024-03-14T09:52:05.234269+0000","last_active":"2024-03-14T09:53:41.785672+0000","last_peered":"2024-03-14T09:53:41.785672+0000","last_clean":"2024-03-14T09:53:41.785672+0000","last_became_active":"2024-03-14T09:52:05.233418+0000","last_became_peered":"2024-03-14T09:52:05.233418+0000","last_unstale":"2024-03-14T09:53:41.785672+0000","last_undegraded":"2024-03-14T09:53:41.785672+0000","last_fullsized":"2024-03-14T09:53:41.785672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:22:46.894461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899167+0000","last_change":"2024-03-14T09:52:05.741786+0000","last_active":"2024-03-14T09:53:41.899167+0000","last_peered":"2024-03-14T09:53:41.899167+0000","last_clean":"2024-03-14T09:53:41.899167+0000","last_became_active":"2024-03-14T09:52:05.741461+0000","last_became_peered":"2024-03-14T09:52:05.741461+0000","last_unstale":"2024-03-14T09:53:41.899167+0000","last_undegraded":"2024-03-14T09:53:41.899167+0000","last_fullsized":"2024-03-14T09:53:41.899167+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:21:17.409530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786363+0000","last_change":"2024-03-14T09:52:09.768851+0000","last_active":"2024-03-14T09:53:41.786363+0000","last_peered":"2024-03-14T09:53:41.786363+0000","last_clean":"2024-03-14T09:53:41.786363+0000","last_became_active":"2024-03-14T09:52:09.768485+0000","last_became_peered":"2024-03-14T09:52:09.768485+0000","last_unstale":"2024-03-14T09:53:41.786363+0000","last_undegraded":"2024-03-14T09:53:41.786363+0000","last_fullsized":"2024-03-14T09:53:41.786363+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:00:51.189790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"83'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899465+0000","last_change":"2024-03-14T09:52:42.154227+0000","last_active":"2024-03-14T09:53:41.899465+0000","last_peered":"2024-03-14T09:53:41.899465+0000","last_clean":"2024-03-14T09:53:41.899465+0000","last_became_active":"2024-03-14T09:52:42.149841+0000","last_became_peered":"2024-03-14T09:52:42.149841+0000","last_unstale":"2024-03-14T09:53:41.899465+0000","last_undegraded":"2024-03-14T09:53:41.899465+0000","last_fullsized":"2024-03-14T09:53:41.899465+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:04:46.911921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789719+0000","last_change":"2024-03-14T09:52:05.742994+0000","last_active":"2024-03-14T09:53:41.789719+0000","last_peered":"2024-03-14T09:53:41.789719+0000","last_clean":"2024-03-14T09:53:41.789719+0000","last_became_active":"2024-03-14T09:52:05.742652+0000","last_became_peered":"2024-03-14T09:52:05.742652+0000","last_unstale":"2024-03-14T09:53:41.789719+0000","last_undegraded":"2024-03-14T09:53:41.789719+0000","last_fullsized":"2024-03-14T09:53:41.789719+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:46:47.212245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785978+0000","last_change":"2024-03-14T09:52:09.763009+0000","last_active":"2024-03-14T09:53:41.785978+0000","last_peered":"2024-03-14T09:53:41.785978+0000","last_clean":"2024-03-14T09:53:41.785978+0000","last_became_active":"2024-03-14T09:52:09.762621+0000","last_became_peered":"2024-03-14T09:52:09.762621+0000","last_unstale":"2024-03-14T09:53:41.785978+0000","last_undegraded":"2024-03-14T09:53:41.785978+0000","last_fullsized":"2024-03-14T09:53:41.785978+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:56:22.162409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"78'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788297+0000","last_change":"2024-03-14T09:52:05.234039+0000","last_active":"2024-03-14T09:53:41.788297+0000","last_peered":"2024-03-14T09:53:41.788297+0000","last_clean":"2024-03-14T09:53:41.788297+0000","last_became_active":"2024-03-14T09:52:05.233612+0000","last_became_peered":"2024-03-14T09:52:05.233612+0000","last_unstale":"2024-03-14T09:53:41.788297+0000","last_undegraded":"2024-03-14T09:53:41.788297+0000","last_fullsized":"2024-03-14T09:53:41.788297+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:29:52.968178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"85'15","reported_seq":43,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787964+0000","last_change":"2024-03-14T09:52:42.426506+0000","last_active":"2024-03-14T09:53:41.787964+0000","last_peered":"2024-03-14T09:53:41.787964+0000","last_clean":"2024-03-14T09:53:41.787964+0000","last_became_active":"2024-03-14T09:52:42.426062+0000","last_became_peered":"2024-03-14T09:52:42.426062+0000","last_unstale":"2024-03-14T09:53:41.787964+0000","last_undegraded":"2024-03-14T09:53:41.787964+0000","last_fullsized":"2024-03-14T09:53:41.787964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:23:42.943381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783824+0000","last_change":"2024-03-14T09:52:09.757856+0000","last_active":"2024-03-14T09:53:41.783824+0000","last_peered":"2024-03-14T09:53:41.783824+0000","last_clean":"2024-03-14T09:53:41.783824+0000","last_became_active":"2024-03-14T09:52:09.757395+0000","last_became_peered":"2024-03-14T09:52:09.757395+0000","last_unstale":"2024-03-14T09:53:41.783824+0000","last_undegraded":"2024-03-14T09:53:41.783824+0000","last_fullsized":"2024-03-14T09:53:41.783824+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:34:06.600727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"83'5","reported_seq":109,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:50.849824+0000","last_change":"2024-03-14T09:52:05.228527+0000","last_active":"2024-03-14T09:53:50.849824+0000","last_peered":"2024-03-14T09:53:50.849824+0000","last_clean":"2024-03-14T09:53:50.849824+0000","last_became_active":"2024-03-14T09:52:05.228177+0000","last_became_peered":"2024-03-14T09:52:05.228177+0000","last_unstale":"2024-03-14T09:53:50.849824+0000","last_undegraded":"2024-03-14T09:53:50.849824+0000","last_fullsized":"2024-03-14T09:53:50.849824+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:17:57.529308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789022+0000","last_change":"2024-03-14T09:52:05.734041+0000","last_active":"2024-03-14T09:53:41.789022+0000","last_peered":"2024-03-14T09:53:41.789022+0000","last_clean":"2024-03-14T09:53:41.789022+0000","last_became_active":"2024-03-14T09:52:05.732491+0000","last_became_peered":"2024-03-14T09:52:05.732491+0000","last_unstale":"2024-03-14T09:53:41.789022+0000","last_undegraded":"2024-03-14T09:53:41.789022+0000","last_fullsized":"2024-03-14T09:53:41.789022+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:07:31.909382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:43.640334+0000","last_change":"2024-03-14T09:53:43.640334+0000","last_active":"2024-03-14T09:53:43.640334+0000","last_peered":"2024-03-14T09:53:43.640334+0000","last_clean":"2024-03-14T09:53:43.640334+0000","last_became_active":"2024-03-14T09:53:43.639821+0000","last_became_peered":"2024-03-14T09:53:43.639821+0000","last_unstale":"2024-03-14T09:53:43.640334+0000","last_undegraded":"2024-03-14T09:53:43.640334+0000","last_fullsized":"2024-03-14T09:53:43.640334+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:23:30.760149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,10],"acting":[2,6,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"83'3","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787260+0000","last_change":"2024-03-14T09:52:05.228893+0000","last_active":"2024-03-14T09:53:41.787260+0000","last_peered":"2024-03-14T09:53:41.787260+0000","last_clean":"2024-03-14T09:53:41.787260+0000","last_became_active":"2024-03-14T09:52:05.228569+0000","last_became_peered":"2024-03-14T09:52:05.228569+0000","last_unstale":"2024-03-14T09:53:41.787260+0000","last_undegraded":"2024-03-14T09:53:41.787260+0000","last_fullsized":"2024-03-14T09:53:41.787260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:28:03.200437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788237+0000","last_change":"2024-03-14T09:52:05.736388+0000","last_active":"2024-03-14T09:53:41.788237+0000","last_peered":"2024-03-14T09:53:41.788237+0000","last_clean":"2024-03-14T09:53:41.788237+0000","last_became_active":"2024-03-14T09:52:05.735131+0000","last_became_peered":"2024-03-14T09:52:05.735131+0000","last_unstale":"2024-03-14T09:53:41.788237+0000","last_undegraded":"2024-03-14T09:53:41.788237+0000","last_fullsized":"2024-03-14T09:53:41.788237+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:00:54.120087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899317+0000","last_change":"2024-03-14T09:52:09.770321+0000","last_active":"2024-03-14T09:53:41.899317+0000","last_peered":"2024-03-14T09:53:41.899317+0000","last_clean":"2024-03-14T09:53:41.899317+0000","last_became_active":"2024-03-14T09:52:09.769710+0000","last_became_peered":"2024-03-14T09:52:09.769710+0000","last_unstale":"2024-03-14T09:53:41.899317+0000","last_undegraded":"2024-03-14T09:53:41.899317+0000","last_fullsized":"2024-03-14T09:53:41.899317+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:12:12.635248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"83'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789908+0000","last_change":"2024-03-14T09:52:05.223912+0000","last_active":"2024-03-14T09:53:41.789908+0000","last_peered":"2024-03-14T09:53:41.789908+0000","last_clean":"2024-03-14T09:53:41.789908+0000","last_became_active":"2024-03-14T09:52:05.223380+0000","last_became_peered":"2024-03-14T09:52:05.223380+0000","last_unstale":"2024-03-14T09:53:41.789908+0000","last_undegraded":"2024-03-14T09:53:41.789908+0000","last_fullsized":"2024-03-14T09:53:41.789908+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:08:33.098215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785120+0000","last_change":"2024-03-14T09:52:05.743277+0000","last_active":"2024-03-14T09:53:41.785120+0000","last_peered":"2024-03-14T09:53:41.785120+0000","last_clean":"2024-03-14T09:53:41.785120+0000","last_became_active":"2024-03-14T09:52:05.741386+0000","last_became_peered":"2024-03-14T09:52:05.741386+0000","last_unstale":"2024-03-14T09:53:41.785120+0000","last_undegraded":"2024-03-14T09:53:41.785120+0000","last_fullsized":"2024-03-14T09:53:41.785120+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:08:08.320279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899160+0000","last_change":"2024-03-14T09:52:09.766485+0000","last_active":"2024-03-14T09:53:41.899160+0000","last_peered":"2024-03-14T09:53:41.899160+0000","last_clean":"2024-03-14T09:53:41.899160+0000","last_became_active":"2024-03-14T09:52:09.766115+0000","last_became_peered":"2024-03-14T09:52:09.766115+0000","last_unstale":"2024-03-14T09:53:41.899160+0000","last_undegraded":"2024-03-14T09:53:41.899160+0000","last_fullsized":"2024-03-14T09:53:41.899160+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:34:45.067757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"83'1","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786965+0000","last_change":"2024-03-14T09:52:05.233852+0000","last_active":"2024-03-14T09:53:41.786965+0000","last_peered":"2024-03-14T09:53:41.786965+0000","last_clean":"2024-03-14T09:53:41.786965+0000","last_became_active":"2024-03-14T09:52:05.233548+0000","last_became_peered":"2024-03-14T09:52:05.233548+0000","last_unstale":"2024-03-14T09:53:41.786965+0000","last_undegraded":"2024-03-14T09:53:41.786965+0000","last_fullsized":"2024-03-14T09:53:41.786965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:08:51.821465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"85'11","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788942+0000","last_change":"2024-03-14T09:52:05.731215+0000","last_active":"2024-03-14T09:53:41.788942+0000","last_peered":"2024-03-14T09:53:41.788942+0000","last_clean":"2024-03-14T09:53:41.788942+0000","last_became_active":"2024-03-14T09:52:05.730924+0000","last_became_peered":"2024-03-14T09:52:05.730924+0000","last_unstale":"2024-03-14T09:53:41.788942+0000","last_undegraded":"2024-03-14T09:53:41.788942+0000","last_fullsized":"2024-03-14T09:53:41.788942+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:12:17.560297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785051+0000","last_change":"2024-03-14T09:52:09.753775+0000","last_active":"2024-03-14T09:53:41.785051+0000","last_peered":"2024-03-14T09:53:41.785051+0000","last_clean":"2024-03-14T09:53:41.785051+0000","last_became_active":"2024-03-14T09:52:09.753422+0000","last_became_peered":"2024-03-14T09:52:09.753422+0000","last_unstale":"2024-03-14T09:53:41.785051+0000","last_undegraded":"2024-03-14T09:53:41.785051+0000","last_fullsized":"2024-03-14T09:53:41.785051+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:59:25.625909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"83'3","reported_seq":106,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:51.850659+0000","last_change":"2024-03-14T09:52:05.229064+0000","last_active":"2024-03-14T09:53:51.850659+0000","last_peered":"2024-03-14T09:53:51.850659+0000","last_clean":"2024-03-14T09:53:51.850659+0000","last_became_active":"2024-03-14T09:52:05.228751+0000","last_became_peered":"2024-03-14T09:52:05.228751+0000","last_unstale":"2024-03-14T09:53:51.850659+0000","last_undegraded":"2024-03-14T09:53:51.850659+0000","last_fullsized":"2024-03-14T09:53:51.850659+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:13:29.832866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"85'10","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790141+0000","last_change":"2024-03-14T09:52:05.737798+0000","last_active":"2024-03-14T09:53:41.790141+0000","last_peered":"2024-03-14T09:53:41.790141+0000","last_clean":"2024-03-14T09:53:41.790141+0000","last_became_active":"2024-03-14T09:52:05.737449+0000","last_became_peered":"2024-03-14T09:52:05.737449+0000","last_unstale":"2024-03-14T09:53:41.790141+0000","last_undegraded":"2024-03-14T09:53:41.790141+0000","last_fullsized":"2024-03-14T09:53:41.790141+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:40:10.079770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788972+0000","last_change":"2024-03-14T09:52:09.768368+0000","last_active":"2024-03-14T09:53:41.788972+0000","last_peered":"2024-03-14T09:53:41.788972+0000","last_clean":"2024-03-14T09:53:41.788972+0000","last_became_active":"2024-03-14T09:52:09.767622+0000","last_became_peered":"2024-03-14T09:52:09.767622+0000","last_unstale":"2024-03-14T09:53:41.788972+0000","last_undegraded":"2024-03-14T09:53:41.788972+0000","last_fullsized":"2024-03-14T09:53:41.788972+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:19:28.836317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"83'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783894+0000","last_change":"2024-03-14T09:52:05.229778+0000","last_active":"2024-03-14T09:53:41.783894+0000","last_peered":"2024-03-14T09:53:41.783894+0000","last_clean":"2024-03-14T09:53:41.783894+0000","last_became_active":"2024-03-14T09:52:05.229468+0000","last_became_peered":"2024-03-14T09:52:05.229468+0000","last_unstale":"2024-03-14T09:53:41.783894+0000","last_undegraded":"2024-03-14T09:53:41.783894+0000","last_fullsized":"2024-03-14T09:53:41.783894+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:49:58.433646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899333+0000","last_change":"2024-03-14T09:52:05.751027+0000","last_active":"2024-03-14T09:53:41.899333+0000","last_peered":"2024-03-14T09:53:41.899333+0000","last_clean":"2024-03-14T09:53:41.899333+0000","last_became_active":"2024-03-14T09:52:05.750650+0000","last_became_peered":"2024-03-14T09:52:05.750650+0000","last_unstale":"2024-03-14T09:53:41.899333+0000","last_undegraded":"2024-03-14T09:53:41.899333+0000","last_fullsized":"2024-03-14T09:53:41.899333+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:05:37.264396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:43.631132+0000","last_change":"2024-03-14T09:53:43.631132+0000","last_active":"2024-03-14T09:53:43.631132+0000","last_peered":"2024-03-14T09:53:43.631132+0000","last_clean":"2024-03-14T09:53:43.631132+0000","last_became_active":"2024-03-14T09:53:43.630718+0000","last_became_peered":"2024-03-14T09:53:43.630718+0000","last_unstale":"2024-03-14T09:53:43.631132+0000","last_undegraded":"2024-03-14T09:53:43.631132+0000","last_fullsized":"2024-03-14T09:53:43.631132+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T15:45:17.156095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"83'4","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787859+0000","last_change":"2024-03-14T09:52:05.235472+0000","last_active":"2024-03-14T09:53:41.787859+0000","last_peered":"2024-03-14T09:53:41.787859+0000","last_clean":"2024-03-14T09:53:41.787859+0000","last_became_active":"2024-03-14T09:52:05.235106+0000","last_became_peered":"2024-03-14T09:52:05.235106+0000","last_unstale":"2024-03-14T09:53:41.787859+0000","last_undegraded":"2024-03-14T09:53:41.787859+0000","last_fullsized":"2024-03-14T09:53:41.787859+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:30:36.446130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"85'11","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789899+0000","last_change":"2024-03-14T09:52:05.747615+0000","last_active":"2024-03-14T09:53:41.789899+0000","last_peered":"2024-03-14T09:53:41.789899+0000","last_clean":"2024-03-14T09:53:41.789899+0000","last_became_active":"2024-03-14T09:52:05.747344+0000","last_became_peered":"2024-03-14T09:52:05.747344+0000","last_unstale":"2024-03-14T09:53:41.789899+0000","last_undegraded":"2024-03-14T09:53:41.789899+0000","last_fullsized":"2024-03-14T09:53:41.789899+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:52:23.096124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788627+0000","last_change":"2024-03-14T09:52:09.768295+0000","last_active":"2024-03-14T09:53:41.788627+0000","last_peered":"2024-03-14T09:53:41.788627+0000","last_clean":"2024-03-14T09:53:41.788627+0000","last_became_active":"2024-03-14T09:52:09.760997+0000","last_became_peered":"2024-03-14T09:52:09.760997+0000","last_unstale":"2024-03-14T09:53:41.788627+0000","last_undegraded":"2024-03-14T09:53:41.788627+0000","last_fullsized":"2024-03-14T09:53:41.788627+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:22:09.909302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'37","reported_seq":233,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:52.852579+0000","last_change":"2024-03-14T09:52:05.227876+0000","last_active":"2024-03-14T09:53:52.852579+0000","last_peered":"2024-03-14T09:53:52.852579+0000","last_clean":"2024-03-14T09:53:52.852579+0000","last_became_active":"2024-03-14T09:52:05.227519+0000","last_became_peered":"2024-03-14T09:52:05.227519+0000","last_unstale":"2024-03-14T09:53:52.852579+0000","last_undegraded":"2024-03-14T09:53:52.852579+0000","last_fullsized":"2024-03-14T09:53:52.852579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:12.616609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66994,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":176,"num_read_kb":154,"num_write":38,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788944+0000","last_change":"2024-03-14T09:52:05.737432+0000","last_active":"2024-03-14T09:53:41.788944+0000","last_peered":"2024-03-14T09:53:41.788944+0000","last_clean":"2024-03-14T09:53:41.788944+0000","last_became_active":"2024-03-14T09:52:05.735050+0000","last_became_peered":"2024-03-14T09:52:05.735050+0000","last_unstale":"2024-03-14T09:53:41.788944+0000","last_undegraded":"2024-03-14T09:53:41.788944+0000","last_fullsized":"2024-03-14T09:53:41.788944+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T09:57:45.383155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788252+0000","last_change":"2024-03-14T09:52:09.754909+0000","last_active":"2024-03-14T09:53:41.788252+0000","last_peered":"2024-03-14T09:53:41.788252+0000","last_clean":"2024-03-14T09:53:41.788252+0000","last_became_active":"2024-03-14T09:52:09.754367+0000","last_became_peered":"2024-03-14T09:52:09.754367+0000","last_unstale":"2024-03-14T09:53:41.788252+0000","last_undegraded":"2024-03-14T09:53:41.788252+0000","last_fullsized":"2024-03-14T09:53:41.788252+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:52:34.251555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"83'6","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783813+0000","last_change":"2024-03-14T09:52:05.224869+0000","last_active":"2024-03-14T09:53:41.783813+0000","last_peered":"2024-03-14T09:53:41.783813+0000","last_clean":"2024-03-14T09:53:41.783813+0000","last_became_active":"2024-03-14T09:52:05.224499+0000","last_became_peered":"2024-03-14T09:52:05.224499+0000","last_unstale":"2024-03-14T09:53:41.783813+0000","last_undegraded":"2024-03-14T09:53:41.783813+0000","last_fullsized":"2024-03-14T09:53:41.783813+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:49:01.729417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789760+0000","last_change":"2024-03-14T09:52:05.729168+0000","last_active":"2024-03-14T09:53:41.789760+0000","last_peered":"2024-03-14T09:53:41.789760+0000","last_clean":"2024-03-14T09:53:41.789760+0000","last_became_active":"2024-03-14T09:52:05.728806+0000","last_became_peered":"2024-03-14T09:52:05.728806+0000","last_unstale":"2024-03-14T09:53:41.789760+0000","last_undegraded":"2024-03-14T09:53:41.789760+0000","last_fullsized":"2024-03-14T09:53:41.789760+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:54:36.758853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899244+0000","last_change":"2024-03-14T09:52:09.770126+0000","last_active":"2024-03-14T09:53:41.899244+0000","last_peered":"2024-03-14T09:53:41.899244+0000","last_clean":"2024-03-14T09:53:41.899244+0000","last_became_active":"2024-03-14T09:52:09.769734+0000","last_became_peered":"2024-03-14T09:52:09.769734+0000","last_unstale":"2024-03-14T09:53:41.899244+0000","last_undegraded":"2024-03-14T09:53:41.899244+0000","last_fullsized":"2024-03-14T09:53:41.899244+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:50:37.839858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"83'3","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787719+0000","last_change":"2024-03-14T09:52:05.235201+0000","last_active":"2024-03-14T09:53:41.787719+0000","last_peered":"2024-03-14T09:53:41.787719+0000","last_clean":"2024-03-14T09:53:41.787719+0000","last_became_active":"2024-03-14T09:52:05.234816+0000","last_became_peered":"2024-03-14T09:52:05.234816+0000","last_unstale":"2024-03-14T09:53:41.787719+0000","last_undegraded":"2024-03-14T09:53:41.787719+0000","last_fullsized":"2024-03-14T09:53:41.787719+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:01:50.070913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'15","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783755+0000","last_change":"2024-03-14T09:52:05.735304+0000","last_active":"2024-03-14T09:53:41.783755+0000","last_peered":"2024-03-14T09:53:41.783755+0000","last_clean":"2024-03-14T09:53:41.783755+0000","last_became_active":"2024-03-14T09:52:05.734977+0000","last_became_peered":"2024-03-14T09:52:05.734977+0000","last_unstale":"2024-03-14T09:53:41.783755+0000","last_undegraded":"2024-03-14T09:53:41.783755+0000","last_fullsized":"2024-03-14T09:53:41.783755+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:08:28.071962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789402+0000","last_change":"2024-03-14T09:52:09.770721+0000","last_active":"2024-03-14T09:53:41.789402+0000","last_peered":"2024-03-14T09:53:41.789402+0000","last_clean":"2024-03-14T09:53:41.789402+0000","last_became_active":"2024-03-14T09:52:09.770270+0000","last_became_peered":"2024-03-14T09:52:09.770270+0000","last_unstale":"2024-03-14T09:53:41.789402+0000","last_undegraded":"2024-03-14T09:53:41.789402+0000","last_fullsized":"2024-03-14T09:53:41.789402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:09:33.480235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'6","reported_seq":111,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:48.852068+0000","last_change":"2024-03-14T09:52:05.227260+0000","last_active":"2024-03-14T09:53:48.852068+0000","last_peered":"2024-03-14T09:53:48.852068+0000","last_clean":"2024-03-14T09:53:48.852068+0000","last_became_active":"2024-03-14T09:52:05.226922+0000","last_became_peered":"2024-03-14T09:52:05.226922+0000","last_unstale":"2024-03-14T09:53:48.852068+0000","last_undegraded":"2024-03-14T09:53:48.852068+0000","last_fullsized":"2024-03-14T09:53:48.852068+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:36:09.770478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783445+0000","last_change":"2024-03-14T09:52:05.734278+0000","last_active":"2024-03-14T09:53:41.783445+0000","last_peered":"2024-03-14T09:53:41.783445+0000","last_clean":"2024-03-14T09:53:41.783445+0000","last_became_active":"2024-03-14T09:52:05.732051+0000","last_became_peered":"2024-03-14T09:52:05.732051+0000","last_unstale":"2024-03-14T09:53:41.783445+0000","last_undegraded":"2024-03-14T09:53:41.783445+0000","last_fullsized":"2024-03-14T09:53:41.783445+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:33:07.137983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790644+0000","last_change":"2024-03-14T09:52:09.764068+0000","last_active":"2024-03-14T09:53:41.790644+0000","last_peered":"2024-03-14T09:53:41.790644+0000","last_clean":"2024-03-14T09:53:41.790644+0000","last_became_active":"2024-03-14T09:52:09.763613+0000","last_became_peered":"2024-03-14T09:52:09.763613+0000","last_unstale":"2024-03-14T09:53:41.790644+0000","last_undegraded":"2024-03-14T09:53:41.790644+0000","last_fullsized":"2024-03-14T09:53:41.790644+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:32:15.524602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"83'15","reported_seq":63,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787502+0000","last_change":"2024-03-14T09:52:42.438205+0000","last_active":"2024-03-14T09:53:41.787502+0000","last_peered":"2024-03-14T09:53:41.787502+0000","last_clean":"2024-03-14T09:53:41.787502+0000","last_became_active":"2024-03-14T09:52:42.426268+0000","last_became_peered":"2024-03-14T09:52:42.426268+0000","last_unstale":"2024-03-14T09:53:41.787502+0000","last_undegraded":"2024-03-14T09:53:41.787502+0000","last_fullsized":"2024-03-14T09:53:41.787502+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:10:09.651131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788220+0000","last_change":"2024-03-14T09:52:05.734366+0000","last_active":"2024-03-14T09:53:41.788220+0000","last_peered":"2024-03-14T09:53:41.788220+0000","last_clean":"2024-03-14T09:53:41.788220+0000","last_became_active":"2024-03-14T09:52:05.734058+0000","last_became_peered":"2024-03-14T09:52:05.734058+0000","last_unstale":"2024-03-14T09:53:41.788220+0000","last_undegraded":"2024-03-14T09:53:41.788220+0000","last_fullsized":"2024-03-14T09:53:41.788220+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:49:49.233683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788119+0000","last_change":"2024-03-14T09:52:09.760241+0000","last_active":"2024-03-14T09:53:41.788119+0000","last_peered":"2024-03-14T09:53:41.788119+0000","last_clean":"2024-03-14T09:53:41.788119+0000","last_became_active":"2024-03-14T09:52:09.759797+0000","last_became_peered":"2024-03-14T09:52:09.759797+0000","last_unstale":"2024-03-14T09:53:41.788119+0000","last_undegraded":"2024-03-14T09:53:41.788119+0000","last_fullsized":"2024-03-14T09:53:41.788119+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:50:33.035775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:42.792396+0000","last_change":"2024-03-14T09:52:05.232932+0000","last_active":"2024-03-14T09:53:42.792396+0000","last_peered":"2024-03-14T09:53:42.792396+0000","last_clean":"2024-03-14T09:53:42.792396+0000","last_became_active":"2024-03-14T09:52:05.232608+0000","last_became_peered":"2024-03-14T09:52:05.232608+0000","last_unstale":"2024-03-14T09:53:42.792396+0000","last_undegraded":"2024-03-14T09:53:42.792396+0000","last_fullsized":"2024-03-14T09:53:42.792396+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:43:59.613531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899344+0000","last_change":"2024-03-14T09:52:05.733762+0000","last_active":"2024-03-14T09:53:41.899344+0000","last_peered":"2024-03-14T09:53:41.899344+0000","last_clean":"2024-03-14T09:53:41.899344+0000","last_became_active":"2024-03-14T09:52:05.733417+0000","last_became_peered":"2024-03-14T09:52:05.733417+0000","last_unstale":"2024-03-14T09:53:41.899344+0000","last_undegraded":"2024-03-14T09:53:41.899344+0000","last_fullsized":"2024-03-14T09:53:41.899344+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:11:12.593952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789125+0000","last_change":"2024-03-14T09:52:42.426896+0000","last_active":"2024-03-14T09:53:41.789125+0000","last_peered":"2024-03-14T09:53:41.789125+0000","last_clean":"2024-03-14T09:53:41.789125+0000","last_became_active":"2024-03-14T09:52:42.426287+0000","last_became_peered":"2024-03-14T09:52:42.426287+0000","last_unstale":"2024-03-14T09:53:41.789125+0000","last_undegraded":"2024-03-14T09:53:41.789125+0000","last_fullsized":"2024-03-14T09:53:41.789125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:53:16.523707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789199+0000","last_change":"2024-03-14T09:52:05.227636+0000","last_active":"2024-03-14T09:53:41.789199+0000","last_peered":"2024-03-14T09:53:41.789199+0000","last_clean":"2024-03-14T09:53:41.789199+0000","last_became_active":"2024-03-14T09:52:05.227274+0000","last_became_peered":"2024-03-14T09:52:05.227274+0000","last_unstale":"2024-03-14T09:53:41.789199+0000","last_undegraded":"2024-03-14T09:53:41.789199+0000","last_fullsized":"2024-03-14T09:53:41.789199+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:13:25.380090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'4","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787833+0000","last_change":"2024-03-14T09:52:05.733158+0000","last_active":"2024-03-14T09:53:41.787833+0000","last_peered":"2024-03-14T09:53:41.787833+0000","last_clean":"2024-03-14T09:53:41.787833+0000","last_became_active":"2024-03-14T09:52:05.732812+0000","last_became_peered":"2024-03-14T09:52:05.732812+0000","last_unstale":"2024-03-14T09:53:41.787833+0000","last_undegraded":"2024-03-14T09:53:41.787833+0000","last_fullsized":"2024-03-14T09:53:41.787833+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:01:49.154860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790946+0000","last_change":"2024-03-14T09:52:09.761596+0000","last_active":"2024-03-14T09:53:41.790946+0000","last_peered":"2024-03-14T09:53:41.790946+0000","last_clean":"2024-03-14T09:53:41.790946+0000","last_became_active":"2024-03-14T09:52:09.761184+0000","last_became_peered":"2024-03-14T09:52:09.761184+0000","last_unstale":"2024-03-14T09:53:41.790946+0000","last_undegraded":"2024-03-14T09:53:41.790946+0000","last_fullsized":"2024-03-14T09:53:41.790946+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:49:02.567472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"83'1","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783455+0000","last_change":"2024-03-14T09:52:05.231828+0000","last_active":"2024-03-14T09:53:41.783455+0000","last_peered":"2024-03-14T09:53:41.783455+0000","last_clean":"2024-03-14T09:53:41.783455+0000","last_became_active":"2024-03-14T09:52:05.231490+0000","last_became_peered":"2024-03-14T09:52:05.231490+0000","last_unstale":"2024-03-14T09:53:41.783455+0000","last_undegraded":"2024-03-14T09:53:41.783455+0000","last_fullsized":"2024-03-14T09:53:41.783455+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:24:53.899728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787604+0000","last_change":"2024-03-14T09:52:05.744646+0000","last_active":"2024-03-14T09:53:41.787604+0000","last_peered":"2024-03-14T09:53:41.787604+0000","last_clean":"2024-03-14T09:53:41.787604+0000","last_became_active":"2024-03-14T09:52:05.744327+0000","last_became_peered":"2024-03-14T09:52:05.744327+0000","last_unstale":"2024-03-14T09:53:41.787604+0000","last_undegraded":"2024-03-14T09:53:41.787604+0000","last_fullsized":"2024-03-14T09:53:41.787604+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:46:50.643350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.791013+0000","last_change":"2024-03-14T09:52:09.765374+0000","last_active":"2024-03-14T09:53:41.791013+0000","last_peered":"2024-03-14T09:53:41.791013+0000","last_clean":"2024-03-14T09:53:41.791013+0000","last_became_active":"2024-03-14T09:52:09.764939+0000","last_became_peered":"2024-03-14T09:52:09.764939+0000","last_unstale":"2024-03-14T09:53:41.791013+0000","last_undegraded":"2024-03-14T09:53:41.791013+0000","last_fullsized":"2024-03-14T09:53:41.791013+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:08:10.688746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787063+0000","last_change":"2024-03-14T09:52:05.729076+0000","last_active":"2024-03-14T09:53:41.787063+0000","last_peered":"2024-03-14T09:53:41.787063+0000","last_clean":"2024-03-14T09:53:41.787063+0000","last_became_active":"2024-03-14T09:52:05.728711+0000","last_became_peered":"2024-03-14T09:52:05.728711+0000","last_unstale":"2024-03-14T09:53:41.787063+0000","last_undegraded":"2024-03-14T09:53:41.787063+0000","last_fullsized":"2024-03-14T09:53:41.787063+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:11:05.608223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":110,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:53.860698+0000","last_change":"2024-03-14T09:52:05.236209+0000","last_active":"2024-03-14T09:53:53.860698+0000","last_peered":"2024-03-14T09:53:53.860698+0000","last_clean":"2024-03-14T09:53:53.860698+0000","last_became_active":"2024-03-14T09:52:05.235517+0000","last_became_peered":"2024-03-14T09:52:05.235517+0000","last_unstale":"2024-03-14T09:53:53.860698+0000","last_undegraded":"2024-03-14T09:53:53.860698+0000","last_fullsized":"2024-03-14T09:53:53.860698+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:09:54.275728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":85,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"83'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783734+0000","last_change":"2024-03-14T09:52:09.761095+0000","last_active":"2024-03-14T09:53:41.783734+0000","last_peered":"2024-03-14T09:53:41.783734+0000","last_clean":"2024-03-14T09:53:41.783734+0000","last_became_active":"2024-03-14T09:52:09.760662+0000","last_became_peered":"2024-03-14T09:52:09.760662+0000","last_unstale":"2024-03-14T09:53:41.783734+0000","last_undegraded":"2024-03-14T09:53:41.783734+0000","last_fullsized":"2024-03-14T09:53:41.783734+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:07:17.670605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785163+0000","last_change":"2024-03-14T09:52:05.736078+0000","last_active":"2024-03-14T09:53:41.785163+0000","last_peered":"2024-03-14T09:53:41.785163+0000","last_clean":"2024-03-14T09:53:41.785163+0000","last_became_active":"2024-03-14T09:52:05.735715+0000","last_became_peered":"2024-03-14T09:52:05.735715+0000","last_unstale":"2024-03-14T09:53:41.785163+0000","last_undegraded":"2024-03-14T09:53:41.785163+0000","last_fullsized":"2024-03-14T09:53:41.785163+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:41:32.863680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"83'3","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786470+0000","last_change":"2024-03-14T09:52:05.225756+0000","last_active":"2024-03-14T09:53:41.786470+0000","last_peered":"2024-03-14T09:53:41.786470+0000","last_clean":"2024-03-14T09:53:41.786470+0000","last_became_active":"2024-03-14T09:52:05.225378+0000","last_became_peered":"2024-03-14T09:52:05.225378+0000","last_unstale":"2024-03-14T09:53:41.786470+0000","last_undegraded":"2024-03-14T09:53:41.786470+0000","last_fullsized":"2024-03-14T09:53:41.786470+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:14:49.594636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":339,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783295+0000","last_change":"2024-03-14T09:51:11.588790+0000","last_active":"2024-03-14T09:53:41.783295+0000","last_peered":"2024-03-14T09:53:41.783295+0000","last_clean":"2024-03-14T09:53:41.783295+0000","last_became_active":"2024-03-14T09:51:11.588287+0000","last_became_peered":"2024-03-14T09:51:11.588287+0000","last_unstale":"2024-03-14T09:53:41.783295+0000","last_undegraded":"2024-03-14T09:53:41.783295+0000","last_fullsized":"2024-03-14T09:53:41.783295+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:48:37.651174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:48:37.651174+0000","last_clean_scrub_stamp":"2024-03-14T09:48:37.651174+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:06:24.252565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787689+0000","last_change":"2024-03-14T09:52:09.761344+0000","last_active":"2024-03-14T09:53:41.787689+0000","last_peered":"2024-03-14T09:53:41.787689+0000","last_clean":"2024-03-14T09:53:41.787689+0000","last_became_active":"2024-03-14T09:52:09.760899+0000","last_became_peered":"2024-03-14T09:52:09.760899+0000","last_unstale":"2024-03-14T09:53:41.787689+0000","last_undegraded":"2024-03-14T09:53:41.787689+0000","last_fullsized":"2024-03-14T09:53:41.787689+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:24:26.212157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789785+0000","last_change":"2024-03-14T09:52:05.738735+0000","last_active":"2024-03-14T09:53:41.789785+0000","last_peered":"2024-03-14T09:53:41.789785+0000","last_clean":"2024-03-14T09:53:41.789785+0000","last_became_active":"2024-03-14T09:52:05.738447+0000","last_became_peered":"2024-03-14T09:52:05.738447+0000","last_unstale":"2024-03-14T09:53:41.789785+0000","last_undegraded":"2024-03-14T09:53:41.789785+0000","last_fullsized":"2024-03-14T09:53:41.789785+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:03:24.598291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'3","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783341+0000","last_change":"2024-03-14T09:52:05.230107+0000","last_active":"2024-03-14T09:53:41.783341+0000","last_peered":"2024-03-14T09:53:41.783341+0000","last_clean":"2024-03-14T09:53:41.783341+0000","last_became_active":"2024-03-14T09:52:05.229812+0000","last_became_peered":"2024-03-14T09:52:05.229812+0000","last_unstale":"2024-03-14T09:53:41.783341+0000","last_undegraded":"2024-03-14T09:53:41.783341+0000","last_fullsized":"2024-03-14T09:53:41.783341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:09:44.884867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899259+0000","last_change":"2024-03-14T09:52:09.763424+0000","last_active":"2024-03-14T09:53:41.899259+0000","last_peered":"2024-03-14T09:53:41.899259+0000","last_clean":"2024-03-14T09:53:41.899259+0000","last_became_active":"2024-03-14T09:52:09.763028+0000","last_became_peered":"2024-03-14T09:52:09.763028+0000","last_unstale":"2024-03-14T09:53:41.899259+0000","last_undegraded":"2024-03-14T09:53:41.899259+0000","last_fullsized":"2024-03-14T09:53:41.899259+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:47:31.415788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"83'1","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789049+0000","last_change":"2024-03-14T09:52:05.235099+0000","last_active":"2024-03-14T09:53:41.789049+0000","last_peered":"2024-03-14T09:53:41.789049+0000","last_clean":"2024-03-14T09:53:41.789049+0000","last_became_active":"2024-03-14T09:52:05.234798+0000","last_became_peered":"2024-03-14T09:52:05.234798+0000","last_unstale":"2024-03-14T09:53:41.789049+0000","last_undegraded":"2024-03-14T09:53:41.789049+0000","last_fullsized":"2024-03-14T09:53:41.789049+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:00:07.461931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.791106+0000","last_change":"2024-03-14T09:52:05.742736+0000","last_active":"2024-03-14T09:53:41.791106+0000","last_peered":"2024-03-14T09:53:41.791106+0000","last_clean":"2024-03-14T09:53:41.791106+0000","last_became_active":"2024-03-14T09:52:05.740927+0000","last_became_peered":"2024-03-14T09:52:05.740927+0000","last_unstale":"2024-03-14T09:53:41.791106+0000","last_undegraded":"2024-03-14T09:53:41.791106+0000","last_fullsized":"2024-03-14T09:53:41.791106+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:50:17.738763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.784877+0000","last_change":"2024-03-14T09:52:09.762722+0000","last_active":"2024-03-14T09:53:41.784877+0000","last_peered":"2024-03-14T09:53:41.784877+0000","last_clean":"2024-03-14T09:53:41.784877+0000","last_became_active":"2024-03-14T09:52:09.759053+0000","last_became_peered":"2024-03-14T09:52:09.759053+0000","last_unstale":"2024-03-14T09:53:41.784877+0000","last_undegraded":"2024-03-14T09:53:41.784877+0000","last_fullsized":"2024-03-14T09:53:41.784877+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:40:29.680002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:43.740679+0000","last_change":"2024-03-14T09:52:05.224778+0000","last_active":"2024-03-14T09:53:43.740679+0000","last_peered":"2024-03-14T09:53:43.740679+0000","last_clean":"2024-03-14T09:53:43.740679+0000","last_became_active":"2024-03-14T09:52:05.223158+0000","last_became_peered":"2024-03-14T09:52:05.223158+0000","last_unstale":"2024-03-14T09:53:43.740679+0000","last_undegraded":"2024-03-14T09:53:43.740679+0000","last_fullsized":"2024-03-14T09:53:43.740679+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:01:29.910099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785742+0000","last_change":"2024-03-14T09:52:05.741753+0000","last_active":"2024-03-14T09:53:41.785742+0000","last_peered":"2024-03-14T09:53:41.785742+0000","last_clean":"2024-03-14T09:53:41.785742+0000","last_became_active":"2024-03-14T09:52:05.741451+0000","last_became_peered":"2024-03-14T09:52:05.741451+0000","last_unstale":"2024-03-14T09:53:41.785742+0000","last_undegraded":"2024-03-14T09:53:41.785742+0000","last_fullsized":"2024-03-14T09:53:41.785742+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:42:59.920909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783437+0000","last_change":"2024-03-14T09:52:09.755884+0000","last_active":"2024-03-14T09:53:41.783437+0000","last_peered":"2024-03-14T09:53:41.783437+0000","last_clean":"2024-03-14T09:53:41.783437+0000","last_became_active":"2024-03-14T09:52:09.755474+0000","last_became_peered":"2024-03-14T09:52:09.755474+0000","last_unstale":"2024-03-14T09:53:41.783437+0000","last_undegraded":"2024-03-14T09:53:41.783437+0000","last_fullsized":"2024-03-14T09:53:41.783437+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:59:23.383388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"83'4","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786361+0000","last_change":"2024-03-14T09:52:05.232673+0000","last_active":"2024-03-14T09:53:41.786361+0000","last_peered":"2024-03-14T09:53:41.786361+0000","last_clean":"2024-03-14T09:53:41.786361+0000","last_became_active":"2024-03-14T09:52:05.232296+0000","last_became_peered":"2024-03-14T09:52:05.232296+0000","last_unstale":"2024-03-14T09:53:41.786361+0000","last_undegraded":"2024-03-14T09:53:41.786361+0000","last_fullsized":"2024-03-14T09:53:41.786361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:24:56.638451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899250+0000","last_change":"2024-03-14T09:52:05.744906+0000","last_active":"2024-03-14T09:53:41.899250+0000","last_peered":"2024-03-14T09:53:41.899250+0000","last_clean":"2024-03-14T09:53:41.899250+0000","last_became_active":"2024-03-14T09:52:05.743753+0000","last_became_peered":"2024-03-14T09:52:05.743753+0000","last_unstale":"2024-03-14T09:53:41.899250+0000","last_undegraded":"2024-03-14T09:53:41.899250+0000","last_fullsized":"2024-03-14T09:53:41.899250+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T09:59:40.358352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786261+0000","last_change":"2024-03-14T09:52:09.768355+0000","last_active":"2024-03-14T09:53:41.786261+0000","last_peered":"2024-03-14T09:53:41.786261+0000","last_clean":"2024-03-14T09:53:41.786261+0000","last_became_active":"2024-03-14T09:52:09.767919+0000","last_became_peered":"2024-03-14T09:52:09.767919+0000","last_unstale":"2024-03-14T09:53:41.786261+0000","last_undegraded":"2024-03-14T09:53:41.786261+0000","last_fullsized":"2024-03-14T09:53:41.786261+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:45:08.436456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"83'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789425+0000","last_change":"2024-03-14T09:52:05.230343+0000","last_active":"2024-03-14T09:53:41.789425+0000","last_peered":"2024-03-14T09:53:41.789425+0000","last_clean":"2024-03-14T09:53:41.789425+0000","last_became_active":"2024-03-14T09:52:05.229974+0000","last_became_peered":"2024-03-14T09:52:05.229974+0000","last_unstale":"2024-03-14T09:53:41.789425+0000","last_undegraded":"2024-03-14T09:53:41.789425+0000","last_fullsized":"2024-03-14T09:53:41.789425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:21:02.458108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"85'2","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.787651+0000","last_change":"2024-03-14T09:52:05.738347+0000","last_active":"2024-03-14T09:53:41.787651+0000","last_peered":"2024-03-14T09:53:41.787651+0000","last_clean":"2024-03-14T09:53:41.787651+0000","last_became_active":"2024-03-14T09:52:05.737961+0000","last_became_peered":"2024-03-14T09:52:05.737961+0000","last_unstale":"2024-03-14T09:53:41.787651+0000","last_undegraded":"2024-03-14T09:53:41.787651+0000","last_fullsized":"2024-03-14T09:53:41.787651+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:02:57.252069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-14T09:53:42.843871+0000","last_change":"2024-03-14T09:53:42.843441+0000","last_active":"2024-03-14T09:53:42.843871+0000","last_peered":"2024-03-14T09:53:42.843871+0000","last_clean":"2024-03-14T09:53:42.843871+0000","last_became_active":"2024-03-14T09:53:42.842860+0000","last_became_peered":"2024-03-14T09:53:42.842860+0000","last_unstale":"2024-03-14T09:53:42.843871+0000","last_undegraded":"2024-03-14T09:53:42.843871+0000","last_fullsized":"2024-03-14T09:53:42.843871+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T21:20:23.073703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"83'3","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.784877+0000","last_change":"2024-03-14T09:52:05.233579+0000","last_active":"2024-03-14T09:53:41.784877+0000","last_peered":"2024-03-14T09:53:41.784877+0000","last_clean":"2024-03-14T09:53:41.784877+0000","last_became_active":"2024-03-14T09:52:05.231605+0000","last_became_peered":"2024-03-14T09:52:05.231605+0000","last_unstale":"2024-03-14T09:53:41.784877+0000","last_undegraded":"2024-03-14T09:53:41.784877+0000","last_fullsized":"2024-03-14T09:53:41.784877+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:55:38.282203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899095+0000","last_change":"2024-03-14T09:52:05.736586+0000","last_active":"2024-03-14T09:53:41.899095+0000","last_peered":"2024-03-14T09:53:41.899095+0000","last_clean":"2024-03-14T09:53:41.899095+0000","last_became_active":"2024-03-14T09:52:05.736235+0000","last_became_peered":"2024-03-14T09:52:05.736235+0000","last_unstale":"2024-03-14T09:53:41.899095+0000","last_undegraded":"2024-03-14T09:53:41.899095+0000","last_fullsized":"2024-03-14T09:53:41.899095+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:46:53.293698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"83'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785979+0000","last_change":"2024-03-14T09:52:09.768203+0000","last_active":"2024-03-14T09:53:41.785979+0000","last_peered":"2024-03-14T09:53:41.785979+0000","last_clean":"2024-03-14T09:53:41.785979+0000","last_became_active":"2024-03-14T09:52:09.767777+0000","last_became_peered":"2024-03-14T09:52:09.767777+0000","last_unstale":"2024-03-14T09:53:41.785979+0000","last_undegraded":"2024-03-14T09:53:41.785979+0000","last_fullsized":"2024-03-14T09:53:41.785979+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:20:47.598905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"83'1","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783567+0000","last_change":"2024-03-14T09:52:05.232359+0000","last_active":"2024-03-14T09:53:41.783567+0000","last_peered":"2024-03-14T09:53:41.783567+0000","last_clean":"2024-03-14T09:53:41.783567+0000","last_became_active":"2024-03-14T09:52:05.231977+0000","last_became_peered":"2024-03-14T09:52:05.231977+0000","last_unstale":"2024-03-14T09:53:41.783567+0000","last_undegraded":"2024-03-14T09:53:41.783567+0000","last_fullsized":"2024-03-14T09:53:41.783567+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:42:32.678157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785461+0000","last_change":"2024-03-14T09:52:05.735822+0000","last_active":"2024-03-14T09:53:41.785461+0000","last_peered":"2024-03-14T09:53:41.785461+0000","last_clean":"2024-03-14T09:53:41.785461+0000","last_became_active":"2024-03-14T09:52:05.735059+0000","last_became_peered":"2024-03-14T09:52:05.735059+0000","last_unstale":"2024-03-14T09:53:41.785461+0000","last_undegraded":"2024-03-14T09:53:41.785461+0000","last_fullsized":"2024-03-14T09:53:41.785461+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T15:22:33.033689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786204+0000","last_change":"2024-03-14T09:52:09.759794+0000","last_active":"2024-03-14T09:53:41.786204+0000","last_peered":"2024-03-14T09:53:41.786204+0000","last_clean":"2024-03-14T09:53:41.786204+0000","last_became_active":"2024-03-14T09:52:09.759366+0000","last_became_peered":"2024-03-14T09:52:09.759366+0000","last_unstale":"2024-03-14T09:53:41.786204+0000","last_undegraded":"2024-03-14T09:53:41.786204+0000","last_fullsized":"2024-03-14T09:53:41.786204+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:52:40.641264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"83'4","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788103+0000","last_change":"2024-03-14T09:52:42.325733+0000","last_active":"2024-03-14T09:53:41.788103+0000","last_peered":"2024-03-14T09:53:41.788103+0000","last_clean":"2024-03-14T09:53:41.788103+0000","last_became_active":"2024-03-14T09:52:42.322002+0000","last_became_peered":"2024-03-14T09:52:42.322002+0000","last_unstale":"2024-03-14T09:53:41.788103+0000","last_undegraded":"2024-03-14T09:53:41.788103+0000","last_fullsized":"2024-03-14T09:53:41.788103+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:22:10.439271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"85'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783569+0000","last_change":"2024-03-14T09:52:05.734073+0000","last_active":"2024-03-14T09:53:41.783569+0000","last_peered":"2024-03-14T09:53:41.783569+0000","last_clean":"2024-03-14T09:53:41.783569+0000","last_became_active":"2024-03-14T09:52:05.731748+0000","last_became_peered":"2024-03-14T09:52:05.731748+0000","last_unstale":"2024-03-14T09:53:41.783569+0000","last_undegraded":"2024-03-14T09:53:41.783569+0000","last_fullsized":"2024-03-14T09:53:41.783569+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:01:11.789162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789540+0000","last_change":"2024-03-14T09:52:09.763309+0000","last_active":"2024-03-14T09:53:41.789540+0000","last_peered":"2024-03-14T09:53:41.789540+0000","last_clean":"2024-03-14T09:53:41.789540+0000","last_became_active":"2024-03-14T09:52:09.762868+0000","last_became_peered":"2024-03-14T09:52:09.762868+0000","last_unstale":"2024-03-14T09:53:41.789540+0000","last_undegraded":"2024-03-14T09:53:41.789540+0000","last_fullsized":"2024-03-14T09:53:41.789540+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:30:03.403911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"83'3","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.784042+0000","last_change":"2024-03-14T09:52:05.232310+0000","last_active":"2024-03-14T09:53:41.784042+0000","last_peered":"2024-03-14T09:53:41.784042+0000","last_clean":"2024-03-14T09:53:41.784042+0000","last_became_active":"2024-03-14T09:52:05.231931+0000","last_became_peered":"2024-03-14T09:52:05.231931+0000","last_unstale":"2024-03-14T09:53:41.784042+0000","last_undegraded":"2024-03-14T09:53:41.784042+0000","last_fullsized":"2024-03-14T09:53:41.784042+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:35:44.274800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783928+0000","last_change":"2024-03-14T09:52:05.731797+0000","last_active":"2024-03-14T09:53:41.783928+0000","last_peered":"2024-03-14T09:53:41.783928+0000","last_clean":"2024-03-14T09:53:41.783928+0000","last_became_active":"2024-03-14T09:52:05.730478+0000","last_became_peered":"2024-03-14T09:52:05.730478+0000","last_unstale":"2024-03-14T09:53:41.783928+0000","last_undegraded":"2024-03-14T09:53:41.783928+0000","last_fullsized":"2024-03-14T09:53:41.783928+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:52:50.684430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.898921+0000","last_change":"2024-03-14T09:52:09.772786+0000","last_active":"2024-03-14T09:53:41.898921+0000","last_peered":"2024-03-14T09:53:41.898921+0000","last_clean":"2024-03-14T09:53:41.898921+0000","last_became_active":"2024-03-14T09:52:09.772198+0000","last_became_peered":"2024-03-14T09:52:09.772198+0000","last_unstale":"2024-03-14T09:53:41.898921+0000","last_undegraded":"2024-03-14T09:53:41.898921+0000","last_fullsized":"2024-03-14T09:53:41.898921+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T14:28:56.620266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"78'1","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788052+0000","last_change":"2024-03-14T09:52:05.237657+0000","last_active":"2024-03-14T09:53:41.788052+0000","last_peered":"2024-03-14T09:53:41.788052+0000","last_clean":"2024-03-14T09:53:41.788052+0000","last_became_active":"2024-03-14T09:52:05.237381+0000","last_became_peered":"2024-03-14T09:52:05.237381+0000","last_unstale":"2024-03-14T09:53:41.788052+0000","last_undegraded":"2024-03-14T09:53:41.788052+0000","last_fullsized":"2024-03-14T09:53:41.788052+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:34:44.372128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783767+0000","last_change":"2024-03-14T09:52:05.730454+0000","last_active":"2024-03-14T09:53:41.783767+0000","last_peered":"2024-03-14T09:53:41.783767+0000","last_clean":"2024-03-14T09:53:41.783767+0000","last_became_active":"2024-03-14T09:52:05.730046+0000","last_became_peered":"2024-03-14T09:52:05.730046+0000","last_unstale":"2024-03-14T09:53:41.783767+0000","last_undegraded":"2024-03-14T09:53:41.783767+0000","last_fullsized":"2024-03-14T09:53:41.783767+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:33:19.296993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786764+0000","last_change":"2024-03-14T09:52:09.755969+0000","last_active":"2024-03-14T09:53:41.786764+0000","last_peered":"2024-03-14T09:53:41.786764+0000","last_clean":"2024-03-14T09:53:41.786764+0000","last_became_active":"2024-03-14T09:52:09.755512+0000","last_became_peered":"2024-03-14T09:52:09.755512+0000","last_unstale":"2024-03-14T09:53:41.786764+0000","last_undegraded":"2024-03-14T09:53:41.786764+0000","last_fullsized":"2024-03-14T09:53:41.786764+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:08.727942+0000","last_clean_scrub_stamp":"2024-03-14T09:52:08.727942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:31:03.388090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"78'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790735+0000","last_change":"2024-03-14T09:52:05.232676+0000","last_active":"2024-03-14T09:53:41.790735+0000","last_peered":"2024-03-14T09:53:41.790735+0000","last_clean":"2024-03-14T09:53:41.790735+0000","last_became_active":"2024-03-14T09:52:05.232332+0000","last_became_peered":"2024-03-14T09:52:05.232332+0000","last_unstale":"2024-03-14T09:53:41.790735+0000","last_undegraded":"2024-03-14T09:53:41.790735+0000","last_fullsized":"2024-03-14T09:53:41.790735+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:04.196645+0000","last_clean_scrub_stamp":"2024-03-14T09:52:04.196645+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:06:11.020555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783875+0000","last_change":"2024-03-14T09:52:05.733338+0000","last_active":"2024-03-14T09:53:41.783875+0000","last_peered":"2024-03-14T09:53:41.783875+0000","last_clean":"2024-03-14T09:53:41.783875+0000","last_became_active":"2024-03-14T09:52:05.732994+0000","last_became_peered":"2024-03-14T09:52:05.732994+0000","last_unstale":"2024-03-14T09:53:41.783875+0000","last_undegraded":"2024-03-14T09:53:41.783875+0000","last_fullsized":"2024-03-14T09:53:41.783875+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:31:25.085916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"85'9","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899170+0000","last_change":"2024-03-14T09:52:05.733121+0000","last_active":"2024-03-14T09:53:41.899170+0000","last_peered":"2024-03-14T09:53:41.899170+0000","last_clean":"2024-03-14T09:53:41.899170+0000","last_became_active":"2024-03-14T09:52:05.732671+0000","last_became_peered":"2024-03-14T09:52:05.732671+0000","last_unstale":"2024-03-14T09:53:41.899170+0000","last_undegraded":"2024-03-14T09:53:41.899170+0000","last_fullsized":"2024-03-14T09:53:41.899170+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:18:54.180131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"85'12","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783676+0000","last_change":"2024-03-14T09:52:05.736534+0000","last_active":"2024-03-14T09:53:41.783676+0000","last_peered":"2024-03-14T09:53:41.783676+0000","last_clean":"2024-03-14T09:53:41.783676+0000","last_became_active":"2024-03-14T09:52:05.736202+0000","last_became_peered":"2024-03-14T09:52:05.736202+0000","last_unstale":"2024-03-14T09:53:41.783676+0000","last_undegraded":"2024-03-14T09:53:41.783676+0000","last_fullsized":"2024-03-14T09:53:41.783676+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T20:22:46.161561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.783472+0000","last_change":"2024-03-14T09:52:05.739434+0000","last_active":"2024-03-14T09:53:41.783472+0000","last_peered":"2024-03-14T09:53:41.783472+0000","last_clean":"2024-03-14T09:53:41.783472+0000","last_became_active":"2024-03-14T09:52:05.738023+0000","last_became_peered":"2024-03-14T09:52:05.738023+0000","last_unstale":"2024-03-14T09:53:41.783472+0000","last_undegraded":"2024-03-14T09:53:41.783472+0000","last_fullsized":"2024-03-14T09:53:41.783472+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T11:51:44.173645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"85'7","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899212+0000","last_change":"2024-03-14T09:52:05.751367+0000","last_active":"2024-03-14T09:53:41.899212+0000","last_peered":"2024-03-14T09:53:41.899212+0000","last_clean":"2024-03-14T09:53:41.899212+0000","last_became_active":"2024-03-14T09:52:05.750816+0000","last_became_peered":"2024-03-14T09:52:05.750816+0000","last_unstale":"2024-03-14T09:53:41.899212+0000","last_undegraded":"2024-03-14T09:53:41.899212+0000","last_fullsized":"2024-03-14T09:53:41.899212+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:57:58.403356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"85'4","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.791204+0000","last_change":"2024-03-14T09:52:05.746652+0000","last_active":"2024-03-14T09:53:41.791204+0000","last_peered":"2024-03-14T09:53:41.791204+0000","last_clean":"2024-03-14T09:53:41.791204+0000","last_became_active":"2024-03-14T09:52:05.746214+0000","last_became_peered":"2024-03-14T09:52:05.746214+0000","last_unstale":"2024-03-14T09:53:41.791204+0000","last_undegraded":"2024-03-14T09:53:41.791204+0000","last_fullsized":"2024-03-14T09:53:41.791204+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T12:21:39.244617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"85'6","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899490+0000","last_change":"2024-03-14T09:52:42.147977+0000","last_active":"2024-03-14T09:53:41.899490+0000","last_peered":"2024-03-14T09:53:41.899490+0000","last_clean":"2024-03-14T09:53:41.899490+0000","last_became_active":"2024-03-14T09:52:42.147623+0000","last_became_peered":"2024-03-14T09:52:42.147623+0000","last_unstale":"2024-03-14T09:53:41.899490+0000","last_undegraded":"2024-03-14T09:53:41.899490+0000","last_fullsized":"2024-03-14T09:53:41.899490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:38:35.351709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"85'15","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.784157+0000","last_change":"2024-03-14T09:52:05.739758+0000","last_active":"2024-03-14T09:53:41.784157+0000","last_peered":"2024-03-14T09:53:41.784157+0000","last_clean":"2024-03-14T09:53:41.784157+0000","last_became_active":"2024-03-14T09:52:05.739423+0000","last_became_peered":"2024-03-14T09:52:05.739423+0000","last_unstale":"2024-03-14T09:53:41.784157+0000","last_undegraded":"2024-03-14T09:53:41.784157+0000","last_fullsized":"2024-03-14T09:53:41.784157+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T10:47:22.594910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"85'2","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.785881+0000","last_change":"2024-03-14T09:52:05.733281+0000","last_active":"2024-03-14T09:53:41.785881+0000","last_peered":"2024-03-14T09:53:41.785881+0000","last_clean":"2024-03-14T09:53:41.785881+0000","last_became_active":"2024-03-14T09:52:05.730081+0000","last_became_peered":"2024-03-14T09:52:05.730081+0000","last_unstale":"2024-03-14T09:53:41.785881+0000","last_undegraded":"2024-03-14T09:53:41.785881+0000","last_fullsized":"2024-03-14T09:53:41.785881+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:40:26.354063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"85'6","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.786998+0000","last_change":"2024-03-14T09:52:05.726938+0000","last_active":"2024-03-14T09:53:41.786998+0000","last_peered":"2024-03-14T09:53:41.786998+0000","last_clean":"2024-03-14T09:53:41.786998+0000","last_became_active":"2024-03-14T09:52:05.726602+0000","last_became_peered":"2024-03-14T09:52:05.726602+0000","last_unstale":"2024-03-14T09:53:41.786998+0000","last_undegraded":"2024-03-14T09:53:41.786998+0000","last_fullsized":"2024-03-14T09:53:41.786998+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T18:26:20.354368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"85'8","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.788428+0000","last_change":"2024-03-14T09:52:42.149962+0000","last_active":"2024-03-14T09:53:41.788428+0000","last_peered":"2024-03-14T09:53:41.788428+0000","last_clean":"2024-03-14T09:53:41.788428+0000","last_became_active":"2024-03-14T09:52:42.149726+0000","last_became_peered":"2024-03-14T09:52:42.149726+0000","last_unstale":"2024-03-14T09:53:41.788428+0000","last_undegraded":"2024-03-14T09:53:41.788428+0000","last_fullsized":"2024-03-14T09:53:41.788428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:20:18.294456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"85'10","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.789837+0000","last_change":"2024-03-14T09:52:05.742945+0000","last_active":"2024-03-14T09:53:41.789837+0000","last_peered":"2024-03-14T09:53:41.789837+0000","last_clean":"2024-03-14T09:53:41.789837+0000","last_became_active":"2024-03-14T09:52:05.742483+0000","last_became_peered":"2024-03-14T09:52:05.742483+0000","last_unstale":"2024-03-14T09:53:41.789837+0000","last_undegraded":"2024-03-14T09:53:41.789837+0000","last_fullsized":"2024-03-14T09:53:41.789837+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T19:05:31.053608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"85'11","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.790355+0000","last_change":"2024-03-14T09:52:05.737822+0000","last_active":"2024-03-14T09:53:41.790355+0000","last_peered":"2024-03-14T09:53:41.790355+0000","last_clean":"2024-03-14T09:53:41.790355+0000","last_became_active":"2024-03-14T09:52:05.735294+0000","last_became_peered":"2024-03-14T09:52:05.735294+0000","last_unstale":"2024-03-14T09:53:41.790355+0000","last_undegraded":"2024-03-14T09:53:41.790355+0000","last_fullsized":"2024-03-14T09:53:41.790355+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T16:06:28.437259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"85'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-14T09:53:41.899359+0000","last_change":"2024-03-14T09:52:05.746991+0000","last_active":"2024-03-14T09:53:41.899359+0000","last_peered":"2024-03-14T09:53:41.899359+0000","last_clean":"2024-03-14T09:53:41.899359+0000","last_became_active":"2024-03-14T09:52:05.746558+0000","last_became_peered":"2024-03-14T09:52:05.746558+0000","last_unstale":"2024-03-14T09:53:41.899359+0000","last_undegraded":"2024-03-14T09:53:41.899359+0000","last_fullsized":"2024-03-14T09:53:41.899359+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T09:52:05.202663+0000","last_clean_scrub_stamp":"2024-03-14T09:52:05.202663+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T17:46:11.540134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247929,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71351,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":527,"num_read_kb":343,"num_write":168,"num_write_kb":127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":694,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":217357,"data_compressed":4812,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":147,"ondisk_log_size":147,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579936,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":958464,"data_stored":489449,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.627,"15min":0.61599999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.681,"5min":2.681,"15min":2.681},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.59999999999999998,"15min":0.60899999999999999},"min":{"1min":0.379,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.64700000000000002,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.496}]},{"osd":1,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.64900000000000002,"15min":0.64400000000000002},"min":{"1min":0.57099999999999995,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.624},{"interface":"front","average":{"1min":0.623,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77300000000000002,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.47899999999999998}]},{"osd":2,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51300000000000001,"15min":0.51100000000000001},"min":{"1min":0.39900000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.77900000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.83099999999999996,"5min":0.88,"15min":0.88},"last":0.371}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.623,"15min":0.61699999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.68000000000000005},"min":{"1min":0.45800000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.81499999999999995,"5min":1.325,"15min":1.325},"last":0.64100000000000001}]},{"osd":5,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.748,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.44},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63800000000000001,"15min":0.64800000000000002},"min":{"1min":0.44600000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.77800000000000002,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.51100000000000001}]},{"osd":6,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70899999999999996,"15min":0.71499999999999997},"min":{"1min":0.53500000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.97999999999999998,"5min":1.135,"15min":1.135},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.79800000000000004,"15min":0.79100000000000004},"min":{"1min":0.61799999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.7000000000000002,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.752,"15min":0.75800000000000001},"min":{"1min":0.56599999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.88800000000000001,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.75800000000000001,"15min":0.75700000000000001},"min":{"1min":0.57099999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.874,"5min":1.363,"15min":1.363},"last":0.69399999999999995}]},{"osd":8,"last update":"Thu Mar 14 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.73499999999999999}]},{"osd":9,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.81699999999999995,"15min":0.82499999999999996},"min":{"1min":0.626,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.96499999999999997,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.874,"15min":0.874},"min":{"1min":0.66400000000000003,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.804,"5min":1.804,"15min":1.804},"last":0.66000000000000003}]},{"osd":10,"last update":"Thu Mar 14 09:53:28 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.81699999999999995,"15min":0.82099999999999995},"min":{"1min":0.65500000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.92500000000000004,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.78100000000000003,"15min":0.78500000000000003},"min":{"1min":0.60999999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.93500000000000005,"5min":1.448,"15min":1.448},"last":0.63300000000000001}]}]},{"osd":10,"up_from":67,"seq":287762808867,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":1144316,"data_compressed":61074,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46200000000000002,"15min":0.46400000000000002},"min":{"1min":0.32500000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46000000000000002,"15min":0.45900000000000002},"min":{"1min":0.29899999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.35599999999999998}]},{"osd":2,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57799999999999996,"15min":0.57999999999999996},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79400000000000004,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60199999999999998,"15min":0.60299999999999998},"min":{"1min":0.441,"5min":0.309,"15min":0.309},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.52300000000000002}]},{"osd":3,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.64900000000000002,"15min":0.66100000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.73399999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.63900000000000001,"15min":0.64100000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.80400000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.375,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.75800000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55500000000000005,"15min":0.56100000000000005},"min":{"1min":0.39200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77600000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.39900000000000002}]},{"osd":5,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.624,"15min":0.621},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.42299999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.622}]},{"osd":6,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.752,"15min":0.76100000000000001},"min":{"1min":0.51000000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0089999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.499,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.95899999999999996,"5min":1.05,"15min":1.05},"last":0.67600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":8,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.74099999999999999,"15min":0.74199999999999999},"min":{"1min":0.47499999999999998,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.75,"15min":0.75600000000000001},"min":{"1min":0.52300000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0009999999999999,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.56499999999999995}]},{"osd":9,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.79500000000000004,"15min":0.79900000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.76700000000000002,"15min":0.77400000000000002},"min":{"1min":0.55000000000000004,"5min":0.502,"15min":0.502},"max":{"1min":1.014,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.59499999999999997}]},{"osd":11,"last update":"Thu Mar 14 09:53:30 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.748,"15min":0.75},"min":{"1min":0.56399999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.754,"5min":0.75800000000000001,"15min":0.75900000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.73199999999999998}]}]},{"osd":9,"up_from":62,"seq":266287972392,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":493199,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.56899999999999995,"15min":0.57699999999999996},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":0.376},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.34999999999999998}]},{"osd":1,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68400000000000005,"15min":0.68000000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.504,"5min":2.504,"15min":2.504},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.73499999999999999,"15min":0.74299999999999999},"min":{"1min":0.50800000000000001,"5min":0.432,"15min":0.432},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":0.76800000000000002}]},{"osd":2,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.54200000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":2.3380000000000001,"5min":2.3380000000000001,"15min":2.3380000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.78000000000000003,"15min":0.79100000000000004},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.2360000000000002,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.71199999999999997,"15min":0.72499999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.0649999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.67600000000000005,"15min":0.68400000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.5990000000000002,"5min":2.5990000000000002,"15min":2.5990000000000002},"last":0.499}]},{"osd":4,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.81000000000000005,"15min":0.82399999999999995},"min":{"1min":0.52500000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.79700000000000004,"15min":0.81000000000000005},"min":{"1min":0.53000000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.54300000000000004}]},{"osd":5,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.85099999999999998,"15min":0.86699999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.83599999999999997,"15min":0.84199999999999997},"min":{"1min":0.57799999999999996,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":2.4390000000000001,"5min":2.4390000000000001,"15min":2.4390000000000001},"last":0.87}]},{"osd":6,"last update":"Thu Mar 14 09:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.88600000000000001,"15min":0.90600000000000003},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.90500000000000003,"15min":0.92100000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.69399999999999995}]},{"osd":8,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.92100000000000004,"15min":0.94099999999999995},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.93600000000000005,"15min":0.95399999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.137,"5min":2.137,"15min":2.137},"last":0.84699999999999998}]},{"osd":10,"last update":"Thu Mar 14 09:53:09 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.88,"15min":0.89000000000000001},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.88500000000000001,"15min":0.89300000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":0.752}]},{"osd":11,"last update":"Thu Mar 14 09:53:09 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.569,"5min":2.569,"15min":2.569},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":2.266,"5min":2.266,"15min":2.266},"last":0.85599999999999998}]}]},{"osd":8,"up_from":55,"seq":236223201322,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":488692,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57099999999999995,"15min":0.58199999999999996},"min":{"1min":0.38100000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66700000000000004,"5min":0.872,"15min":0.872},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.92500000000000004,"15min":1.081},"min":{"1min":0.38800000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80900000000000005,"5min":9.7769999999999992,"15min":9.7769999999999992},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.38600000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.748,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.59399999999999997,"15min":0.57399999999999995},"min":{"1min":0.46400000000000002,"5min":0.432,"15min":0.432},"max":{"1min":0.78900000000000003,"5min":0.877,"15min":0.877},"last":0.65900000000000003}]},{"osd":2,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.44900000000000001,"15min":0.432},"min":{"1min":0.33000000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46500000000000002,"15min":0.45200000000000001},"min":{"1min":0.34699999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.71399999999999997,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.42099999999999999}]},{"osd":3,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.61199999999999999,"15min":0.60599999999999998},"min":{"1min":0.46200000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80400000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.64100000000000001,"15min":0.623},"min":{"1min":0.47699999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59799999999999998,"15min":0.57599999999999996},"min":{"1min":0.52100000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82199999999999995,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.61499999999999999,"15min":0.59099999999999997},"min":{"1min":0.501,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84099999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.60399999999999998}]},{"osd":5,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.51600000000000001,"15min":0.504},"min":{"1min":0.36199999999999999,"5min":0.312,"15min":0.312},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54100000000000004,"15min":0.52200000000000002},"min":{"1min":0.41299999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.67100000000000004}]},{"osd":6,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.68600000000000005,"15min":0.66800000000000004},"min":{"1min":0.63300000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.755},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.66700000000000004,"15min":0.65300000000000002},"min":{"1min":0.50900000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.64700000000000002}]},{"osd":7,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.65200000000000002,"15min":0.628},"min":{"1min":0.55600000000000005,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.69299999999999995,"15min":0.67400000000000004},"min":{"1min":0.60899999999999999,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.69899999999999995}]},{"osd":9,"last update":"Thu Mar 14 09:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.73299999999999998,"15min":0.72999999999999998},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.53800000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.745}]},{"osd":10,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.72499999999999998,"15min":0.71099999999999997},"min":{"1min":0.60999999999999999,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.73299999999999998,"15min":0.72499999999999998},"min":{"1min":0.58599999999999997,"5min":0.499,"15min":0.499},"max":{"1min":0.98099999999999998,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.72299999999999998}]},{"osd":11,"last update":"Thu Mar 14 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.76600000000000001}]}]},{"osd":7,"up_from":48,"seq":206158430255,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":1085098,"data_compressed":59470,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.40999999999999998}]},{"osd":1,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42999999999999999,"15min":0.42099999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.57999999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.42899999999999999,"15min":0.41299999999999998},"min":{"1min":0.28799999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.57399999999999995,"15min":0.56200000000000006},"min":{"1min":0.47599999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82299999999999995,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58199999999999996,"15min":0.56999999999999995},"min":{"1min":0.38900000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.748,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.52800000000000002}]},{"osd":3,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58699999999999997,"15min":0.58799999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81200000000000006,"5min":0.872,"15min":0.872},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60399999999999998,"15min":0.59199999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82599999999999996,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.627}]},{"osd":4,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.502,"15min":0.48199999999999998},"min":{"1min":0.33700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.5,"15min":0.496},"min":{"1min":0.32700000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76600000000000001,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.60399999999999998,"15min":0.58099999999999996},"min":{"1min":0.47099999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.86899999999999999,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60899999999999999}]},{"osd":6,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.65400000000000003,"15min":0.63800000000000001},"min":{"1min":0.51400000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.878,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.66100000000000003,"15min":0.65800000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85499999999999998,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.70899999999999996}]},{"osd":8,"last update":"Thu Mar 14 09:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.69999999999999996,"15min":0.69499999999999995},"min":{"1min":0.52800000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.95699999999999996,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.70099999999999996,"15min":0.70199999999999996},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.90100000000000002,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.65700000000000003}]},{"osd":9,"last update":"Thu Mar 14 09:53:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.75700000000000001,"15min":0.76300000000000001},"min":{"1min":0.55500000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.98299999999999998,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.77000000000000002,"15min":0.77400000000000002},"min":{"1min":0.61399999999999999,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.92300000000000004,"5min":1.008,"15min":1.008},"last":0.76500000000000001}]},{"osd":10,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.57099999999999995,"5min":0.434,"15min":0.434},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.72099999999999997,"15min":0.72499999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90400000000000003,"5min":1.044,"15min":1.044},"last":0.68999999999999995}]},{"osd":11,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.747},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.78700000000000003}]}]},{"osd":6,"up_from":43,"seq":184683593778,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":950272,"data_stored":484279,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.51200000000000001,"15min":0.48199999999999998},"min":{"1min":0.31,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.54,"5min":3.54,"15min":3.54},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.52300000000000002,"15min":0.497},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.9860000000000002,"5min":3.9860000000000002,"15min":3.9860000000000002},"last":0.47499999999999998}]},{"osd":1,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.70699999999999996,"15min":0.65900000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":3.7589999999999999,"5min":3.7589999999999999,"15min":3.7589999999999999},"last":0.626},{"interface":"front","average":{"1min":0.75,"5min":0.64100000000000001,"15min":0.61099999999999999},"min":{"1min":0.52900000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.68799999999999994}]},{"osd":2,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.68400000000000005,"15min":0.64700000000000002},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":4.0190000000000001,"5min":4.0190000000000001,"15min":4.0190000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.72399999999999998,"15min":0.68600000000000005},"min":{"1min":0.46100000000000002,"5min":0.38,"15min":0.38},"max":{"1min":3.5790000000000002,"5min":3.5790000000000002,"15min":3.5790000000000002},"last":0.76000000000000001}]},{"osd":3,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.63800000000000001,"15min":0.60699999999999998},"min":{"1min":0.41999999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.7959999999999998,"5min":3.7959999999999998,"15min":3.7959999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.55400000000000005,"15min":0.54600000000000004},"min":{"1min":0.41099999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78300000000000003,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.499}]},{"osd":4,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.69599999999999995,"15min":0.65600000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.76700000000000002,"15min":0.72899999999999998},"min":{"1min":0.53000000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":3.8559999999999999,"5min":3.8559999999999999,"15min":3.8559999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.76500000000000001,"15min":0.77500000000000002},"min":{"1min":0.58799999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1120000000000001,"5min":3.1059999999999999,"15min":3.1059999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.746,"15min":0.72099999999999997},"min":{"1min":0.53500000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":4.0469999999999997,"5min":4.0469999999999997,"15min":4.0469999999999997},"last":0.69999999999999996}]},{"osd":7,"last update":"Thu Mar 14 09:53:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.71799999999999997,"15min":0.69599999999999995},"min":{"1min":0.59299999999999997,"5min":0.495,"15min":0.495},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.81599999999999995,"15min":0.78500000000000003},"min":{"1min":0.57099999999999995,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.8260000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.79300000000000004}]},{"osd":8,"last update":"Thu Mar 14 09:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.89300000000000002,"15min":0.91100000000000003},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":3.8860000000000001,"5min":3.8860000000000001,"15min":3.8860000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.79500000000000004,"15min":0.76800000000000002},"min":{"1min":0.54200000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":4.1079999999999997,"5min":4.1079999999999997,"15min":4.1079999999999997},"last":0.78200000000000003}]},{"osd":9,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.84999999999999998,"15min":0.86199999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.95299999999999996,"5min":1.119,"15min":1.119},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.79900000000000004,"15min":0.80500000000000005},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.77600000000000002}]},{"osd":10,"last update":"Thu Mar 14 09:53:14 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.84599999999999997,"15min":0.83499999999999996},"min":{"1min":0.68200000000000005,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.0449999999999999,"5min":0.85399999999999998,"15min":0.82199999999999995},"min":{"1min":0.66400000000000003,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":4.1360000000000001,"5min":4.1360000000000001,"15min":4.1360000000000001},"last":0.749}]},{"osd":11,"last update":"Thu Mar 14 09:53:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":3.9489999999999998,"5min":3.9489999999999998,"15min":3.9489999999999998},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.82599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":551295,"data_compressed":25179,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:32 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.57099999999999995,"15min":0.56899999999999995},"min":{"1min":0.45000000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.877,"5min":1.917,"15min":1.917},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.66000000000000003,"15min":0.57299999999999995},"min":{"1min":0.437,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.877,"5min":7.6399999999999997,"15min":7.6399999999999997},"last":0.61099999999999999}]},{"osd":2,"last update":"Thu Mar 14 09:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56499999999999995,"15min":0.54900000000000004},"min":{"1min":0.42199999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76700000000000002,"5min":1.885,"15min":1.885},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61199999999999999,"15min":0.55000000000000004},"min":{"1min":0.438,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.88400000000000001,"5min":3.9820000000000002,"15min":3.9820000000000002},"last":0.75}]},{"osd":3,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.59899999999999998,"15min":0.57699999999999996},"min":{"1min":0.39300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.1829999999999998,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.58699999999999997,"15min":0.57699999999999996},"min":{"1min":0.434,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.6299999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":0.51600000000000001}]},{"osd":4,"last update":"Thu Mar 14 09:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.39800000000000002,"15min":0.371},"min":{"1min":0.35799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.624,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.45900000000000002,"15min":0.40999999999999998},"min":{"1min":0.36399999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.64100000000000001,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Mar 14 09:53:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.55800000000000005,"15min":0.496},"min":{"1min":0.40799999999999997,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70799999999999996,"5min":2.9849999999999999,"15min":2.9849999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.59499999999999997,"15min":0.53100000000000003},"min":{"1min":0.57199999999999995,"5min":0.309,"15min":0.309},"max":{"1min":0.86199999999999999,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.70999999999999996}]},{"osd":6,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.69899999999999995,"15min":0.64900000000000002},"min":{"1min":0.60199999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":3.7879999999999998,"5min":3.7879999999999998,"15min":3.7879999999999998},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.80400000000000005,"15min":0.85999999999999999},"min":{"1min":0.46500000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.5910000000000002,"5min":5.5229999999999997,"15min":5.5229999999999997},"last":0.72799999999999998}]},{"osd":7,"last update":"Thu Mar 14 09:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53200000000000003,"15min":0.496},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81499999999999995,"5min":2.331,"15min":2.331},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.50900000000000001,"15min":0.49099999999999999},"min":{"1min":0.39900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.753,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63500000000000001}]},{"osd":8,"last update":"Thu Mar 14 09:53:34 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.69899999999999995,"15min":0.629},"min":{"1min":0.54100000000000004,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.84199999999999997,"5min":6.4500000000000002,"15min":6.4500000000000002},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.68000000000000005,"15min":0.64700000000000002},"min":{"1min":0.57999999999999996,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.95399999999999996,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":0.76000000000000001}]},{"osd":9,"last update":"Thu Mar 14 09:52:54 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":0.76300000000000001,"15min":0.70599999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":6.5149999999999997,"5min":6.5149999999999997,"15min":6.5149999999999997},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.359,"5min":0.83799999999999997,"15min":0.751},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":9.8900000000000006,"5min":9.8900000000000006,"15min":9.8900000000000006},"last":0.73799999999999999}]},{"osd":10,"last update":"Thu Mar 14 09:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.68600000000000005,"15min":0.68200000000000005},"min":{"1min":0.61699999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.85999999999999999,"5min":0.877,"15min":0.877},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.73599999999999999,"15min":0.72999999999999998},"min":{"1min":0.64600000000000002,"5min":0.497,"15min":0.497},"max":{"1min":0.88500000000000001,"5min":1.171,"15min":1.171},"last":0.97699999999999998}]},{"osd":11,"last update":"Thu Mar 14 09:53:09 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.387,"5min":2.387,"15min":2.387},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":2.2170000000000001,"5min":2.2170000000000001,"15min":2.2170000000000001},"last":1.018}]}]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":485667,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58699999999999997,"15min":0.53500000000000003},"min":{"1min":0.40500000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85199999999999998,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.626,"15min":0.56100000000000005},"min":{"1min":0.42599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.998,"5min":2.056,"15min":2.056},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Mar 14 09:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.69199999999999995,"15min":0.57799999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.94799999999999995,"5min":5.1349999999999998,"15min":5.1349999999999998},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.68999999999999995,"15min":0.58999999999999997},"min":{"1min":0.52100000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0880000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":0.95699999999999996}]},{"osd":3,"last update":"Thu Mar 14 09:53:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":0.998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.1040000000000001}]},{"osd":4,"last update":"Thu Mar 14 09:53:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.71799999999999997,"15min":0.66800000000000004},"min":{"1min":0.47999999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.871,"5min":3.8220000000000001,"15min":3.8220000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.66600000000000004,"15min":0.59999999999999998},"min":{"1min":0.496,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":5.2949999999999999,"5min":5.2949999999999999,"15min":5.2949999999999999},"last":0.75800000000000001}]},{"osd":5,"last update":"Thu Mar 14 09:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.76500000000000001,"15min":0.68700000000000006},"min":{"1min":0.58099999999999996,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1279999999999999,"5min":3.863,"15min":3.863},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.69599999999999995,"15min":0.67700000000000005},"min":{"1min":0.42999999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95799999999999996,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.56599999999999995}]},{"osd":6,"last update":"Thu Mar 14 09:52:58 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.65600000000000003,"15min":0.60999999999999999},"min":{"1min":0.41299999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2370000000000001,"5min":5.2370000000000001,"15min":5.2370000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.76500000000000001,"15min":0.72499999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":4.5890000000000004,"5min":4.5890000000000004,"15min":4.5890000000000004},"last":1.2310000000000001}]},{"osd":7,"last update":"Thu Mar 14 09:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.4449999999999998,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.0669999999999999},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":3.544,"5min":3.544,"15min":3.544},"last":0.59499999999999997}]},{"osd":8,"last update":"Thu Mar 14 09:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.84799999999999998,"15min":0.83599999999999997},"min":{"1min":0.71499999999999997,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.385,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":1.1559999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.79200000000000004,"15min":0.79000000000000004},"min":{"1min":0.51500000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.92700000000000005,"5min":3.6709999999999998,"15min":3.6709999999999998},"last":0.625}]},{"osd":9,"last update":"Thu Mar 14 09:52:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":0.88600000000000001,"15min":0.83999999999999997},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":6.7699999999999996,"5min":6.7699999999999996,"15min":6.7699999999999996},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.2110000000000001,"5min":0.93100000000000005,"15min":0.88400000000000001},"min":{"1min":0.45400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":6.0199999999999996,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":1.1950000000000001}]},{"osd":10,"last update":"Thu Mar 14 09:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.82399999999999995,"15min":0.81200000000000006},"min":{"1min":0.627,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.6339999999999999,"5min":3.6339999999999999,"15min":3.6339999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.88,"15min":0.84399999999999997},"min":{"1min":0.68700000000000006,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":5.3399999999999999,"5min":5.3399999999999999,"15min":5.3399999999999999},"last":0.80900000000000005}]},{"osd":11,"last update":"Thu Mar 14 09:53:27 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.91200000000000003,"15min":0.90800000000000003},"min":{"1min":0.77100000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.329,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":1.2629999999999999},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.91700000000000004,"15min":0.92900000000000005},"min":{"1min":0.64300000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.1220000000000001,"5min":3.71,"15min":3.71},"last":0.73099999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":552373,"data_compressed":25179,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53300000000000003,"15min":0.51800000000000002},"min":{"1min":0.37,"5min":0.31,"15min":0.31},"max":{"1min":0.76800000000000002,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.5},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54900000000000004,"15min":0.53000000000000003},"min":{"1min":0.435,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.83999999999999997,"5min":0.879,"15min":0.879},"last":0.68999999999999995}]},{"osd":1,"last update":"Thu Mar 14 09:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.59799999999999998,"15min":0.53900000000000003},"min":{"1min":0.44900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.91100000000000003,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56299999999999994,"15min":0.54400000000000004},"min":{"1min":0.44400000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.70199999999999996,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.57899999999999996}]},{"osd":3,"last update":"Thu Mar 14 09:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.55000000000000004,"15min":0.52600000000000002},"min":{"1min":0.42599999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56999999999999995,"15min":0.54900000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.66200000000000003}]},{"osd":4,"last update":"Thu Mar 14 09:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.624}]},{"osd":5,"last update":"Thu Mar 14 09:53:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.48699999999999999,"15min":0.45100000000000001},"min":{"1min":0.41699999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.82899999999999996,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.50600000000000001,"15min":0.48199999999999998},"min":{"1min":0.32800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.76100000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.52300000000000002}]},{"osd":6,"last update":"Thu Mar 14 09:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.66100000000000003,"15min":0.65000000000000002},"min":{"1min":0.55100000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.67600000000000005,"15min":0.66300000000000003},"min":{"1min":0.56699999999999995,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.85799999999999998}]},{"osd":7,"last update":"Thu Mar 14 09:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.81000000000000005}]},{"osd":8,"last update":"Thu Mar 14 09:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.64500000000000002,"15min":0.628},"min":{"1min":0.505,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.66000000000000003,"15min":0.64300000000000002},"min":{"1min":0.41499999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.745}]},{"osd":9,"last update":"Thu Mar 14 09:52:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.63900000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.75700000000000001,"15min":0.746},"min":{"1min":0.59299999999999997,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.83399999999999996}]},{"osd":10,"last update":"Thu Mar 14 09:53:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73399999999999999,"15min":0.73299999999999998},"min":{"1min":0.56899999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.91800000000000004,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.76400000000000001,"15min":0.754},"min":{"1min":0.59799999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1,"5min":1.008,"15min":1.008},"last":0.872}]},{"osd":11,"last update":"Thu Mar 14 09:53:04 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.84799999999999998}]}]},{"osd":3,"up_from":24,"seq":103079215167,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":484473,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.42199999999999999,"15min":0.40100000000000002},"min":{"1min":0.32300000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.44600000000000001,"15min":0.42999999999999999},"min":{"1min":0.373,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.66900000000000004}]},{"osd":1,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61099999999999999,"15min":0.59099999999999997},"min":{"1min":0.39400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.82799999999999996,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60699999999999998,"15min":0.58499999999999996},"min":{"1min":0.437,"5min":0.374,"15min":0.374},"max":{"1min":0.76300000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.60299999999999998}]},{"osd":2,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63400000000000001,"15min":0.60699999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84099999999999997,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64000000000000001,"15min":0.623},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.80400000000000005,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.80300000000000005}]},{"osd":4,"last update":"Thu Mar 14 09:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.624,"15min":0.59799999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77200000000000002,"5min":1.091,"15min":1.091},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.629,"15min":0.60399999999999998},"min":{"1min":0.54000000000000004,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81999999999999995,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.50600000000000001}]},{"osd":5,"last update":"Thu Mar 14 09:53:42 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.67600000000000005,"15min":0.67900000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.83799999999999997,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.46400000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.80200000000000005,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.78000000000000003}]},{"osd":6,"last update":"Thu Mar 14 09:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64800000000000002,"15min":0.65000000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.86599999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65800000000000003,"15min":0.64300000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.98799999999999999,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.63500000000000001}]},{"osd":7,"last update":"Thu Mar 14 09:53:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.72999999999999998,"15min":0.73799999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.88900000000000001,"5min":1.123,"15min":1.123},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.748,"5min":0.76600000000000001,"15min":0.76800000000000002},"min":{"1min":0.64100000000000001,"5min":0.498,"15min":0.498},"max":{"1min":0.89200000000000002,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.753}]},{"osd":8,"last update":"Thu Mar 14 09:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.871}]},{"osd":9,"last update":"Thu Mar 14 09:52:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.80000000000000004,"15min":0.81499999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.98099999999999998,"5min":1.097,"15min":1.097},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.83999999999999997,"5min":1.121,"15min":1.121},"last":0.83399999999999996}]},{"osd":10,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.86899999999999999,"15min":0.88500000000000001},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":0.90200000000000002,"5min":1.127,"15min":1.127},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.83099999999999996,"15min":0.83899999999999997},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":0.97699999999999998,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.79100000000000004}]},{"osd":11,"last update":"Thu Mar 14 09:53:06 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.86399999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986234,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":489724,"data_compressed":23575,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:51:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51500000000000001,"15min":0.51200000000000001},"min":{"1min":0.40100000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71599999999999997,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.52300000000000002,"15min":0.51600000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:53:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.36599999999999999,"15min":0.35399999999999998},"min":{"1min":0.27000000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.37,"15min":0.36399999999999999},"min":{"1min":0.33000000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.51300000000000001,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.61099999999999999}]},{"osd":2,"last update":"Thu Mar 14 09:53:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56499999999999995,"15min":0.53800000000000003},"min":{"1min":0.38300000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67200000000000004,"5min":4.3150000000000004,"15min":4.3150000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.55600000000000005,"15min":0.52400000000000002},"min":{"1min":0.43099999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.69199999999999995}]},{"osd":3,"last update":"Thu Mar 14 09:53:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.44,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55500000000000005,"15min":0.55700000000000005},"min":{"1min":0.38500000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.748,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.67800000000000005}]},{"osd":5,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59499999999999997,"15min":0.53300000000000003},"min":{"1min":0.44600000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78300000000000003,"5min":4.3479999999999999,"15min":4.3479999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55000000000000004,"15min":0.53100000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.73699999999999999,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.72399999999999998}]},{"osd":6,"last update":"Thu Mar 14 09:52:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.48099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73499999999999999,"5min":0.876,"15min":0.876},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.60899999999999999,"15min":0.59999999999999998},"min":{"1min":0.42799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.80500000000000005,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.85799999999999998}]},{"osd":7,"last update":"Thu Mar 14 09:53:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.50800000000000001,"15min":0.48799999999999999},"min":{"1min":0.34499999999999997,"5min":0.318,"15min":0.318},"max":{"1min":0.72499999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.47199999999999998,"15min":0.45600000000000002},"min":{"1min":0.36599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.71299999999999997}]},{"osd":8,"last update":"Thu Mar 14 09:53:36 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.65600000000000003,"15min":0.63700000000000001},"min":{"1min":0.57499999999999996,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.623,"15min":0.60499999999999998},"min":{"1min":0.47599999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82699999999999996,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.873}]},{"osd":9,"last update":"Thu Mar 14 09:52:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.47599999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.75,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71299999999999997,"15min":0.70899999999999996},"min":{"1min":0.61899999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.84099999999999997,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.96899999999999997}]},{"osd":10,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.63300000000000001,"5min":0.502,"15min":0.502},"max":{"1min":0.82799999999999996,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.66500000000000004,"15min":0.66300000000000003},"min":{"1min":0.502,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.77900000000000003,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.90500000000000003}]},{"osd":11,"last update":"Thu Mar 14 09:53:07 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.84599999999999997}]}]},{"osd":5,"up_from":36,"seq":154618822711,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":1080081,"data_compressed":59470,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54500000000000004,"15min":0.53000000000000003},"min":{"1min":0.40400000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.70799999999999996,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.372,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.625,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.48399999999999999}]},{"osd":1,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.51900000000000002},"min":{"1min":0.372,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.81899999999999995,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56599999999999995,"15min":0.52500000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75600000000000001,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.57299999999999995}]},{"osd":2,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.41599999999999998,"15min":0.38600000000000001},"min":{"1min":0.27300000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.70699999999999996,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.42599999999999999,"15min":0.39000000000000001},"min":{"1min":0.248,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.67400000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.38900000000000001}]},{"osd":3,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.71199999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.58799999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.77700000000000002,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56399999999999995}]},{"osd":4,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59399999999999997,"15min":0.56000000000000005},"min":{"1min":0.48699999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80200000000000005,"5min":0.87,"15min":0.87},"last":0.626},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56299999999999994,"15min":0.53900000000000003},"min":{"1min":0.42699999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.76300000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.53900000000000003}]},{"osd":6,"last update":"Thu Mar 14 09:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.628,"15min":0.626},"min":{"1min":0.51800000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.79100000000000004,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.64000000000000001,"15min":0.627},"min":{"1min":0.44600000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.82799999999999996,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.52400000000000002}]},{"osd":7,"last update":"Thu Mar 14 09:53:16 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.63300000000000001,"15min":0.60099999999999998},"min":{"1min":0.48399999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.63,"15min":0.59599999999999997},"min":{"1min":0.56299999999999994,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.89600000000000002,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.64200000000000002}]},{"osd":8,"last update":"Thu Mar 14 09:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60399999999999998,"15min":0.59399999999999997},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56499999999999995,"15min":0.56399999999999995},"min":{"1min":0.33400000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.70799999999999996,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.58299999999999996}]},{"osd":9,"last update":"Thu Mar 14 09:52:53 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.72099999999999997,"15min":0.71199999999999997},"min":{"1min":0.56799999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96299999999999997,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.74299999999999999,"15min":0.73699999999999999},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.68600000000000005}]},{"osd":10,"last update":"Thu Mar 14 09:53:11 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.69999999999999996,"15min":0.69599999999999995},"min":{"1min":0.58699999999999997,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.75800000000000001,"15min":0.753},"min":{"1min":0.60899999999999999,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.70099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34227,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34227,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35895,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":69626,"data_compressed":1604,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68184,"data_compressed":1604,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":69759,"data_compressed":1604,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-14T09:53:57.097 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:53:57.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.323+0000 7f01e8400700 1 -- 172.21.15.78:0/503291899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01e01027e0 msgr2=0x7f01e0109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:57.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.323+0000 7f01e8400700 1 --2- 172.21.15.78:0/503291899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01e01027e0 0x7f01e0109090 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f01d800b0d0 tx=0x7f01d800b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.323+0000 7f01e8400700 1 -- 172.21.15.78:0/503291899 shutdown_connections 2024-03-14T09:53:57.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.323+0000 7f01e8400700 1 --2- 172.21.15.78:0/503291899 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01e0100c50 0x7f01e0101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.323+0000 7f01e8400700 1 --2- 172.21.15.78:0/503291899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01e01027e0 0x7f01e0109090 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.323+0000 7f01e8400700 1 --2- 172.21.15.78:0/503291899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f01e0101e40 0x7f01e01022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.323+0000 7f01e8400700 1 -- 172.21.15.78:0/503291899 >> 172.21.15.78:0/503291899 conn(0x7f01e00fc1c0 msgr2=0x7f01e00fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:57.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.323+0000 7f01e8400700 1 -- 172.21.15.78:0/503291899 shutdown_connections 2024-03-14T09:53:57.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.323+0000 7f01e8400700 1 -- 172.21.15.78:0/503291899 wait complete. 2024-03-14T09:53:57.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.324+0000 7f01e8400700 1 Processor -- start 2024-03-14T09:53:57.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.324+0000 7f01e8400700 1 -- start start 2024-03-14T09:53:57.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.325+0000 7f01e8400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01e0100c50 0x7f01e0074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:57.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.325+0000 7f01e8400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f01e0101e40 0x7f01e00731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:57.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.325+0000 7f01e8400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01e01027e0 0x7f01e00736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:57.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.325+0000 7f01e8400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e0073e80 con 0x7f01e0100c50 2024-03-14T09:53:57.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.325+0000 7f01e8400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e0073ff0 con 0x7f01e01027e0 2024-03-14T09:53:57.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.325+0000 7f01e8400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e01a1160 con 0x7f01e0101e40 2024-03-14T09:53:57.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.325+0000 7f01e5a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01e0100c50 0x7f01e0074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:57.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.325+0000 7f01e5a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01e0100c50 0x7f01e0074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43978/0 (socket says 172.21.15.78:43978) 2024-03-14T09:53:57.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.325+0000 7f01e5a00700 1 -- 172.21.15.78:0/3104297287 learned_addr learned my addr 172.21.15.78:0/3104297287 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:57.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.325+0000 7f01e5000700 1 --2- 172.21.15.78:0/3104297287 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f01e0101e40 0x7f01e00731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:57.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.325+0000 7f01e6400700 1 --2- 172.21.15.78:0/3104297287 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01e01027e0 0x7f01e00736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:57.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.325+0000 7f01e5a00700 1 -- 172.21.15.78:0/3104297287 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f01e0101e40 msgr2=0x7f01e00731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:57.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.325+0000 7f01e5a00700 1 --2- 172.21.15.78:0/3104297287 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f01e0101e40 0x7f01e00731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.325+0000 7f01e5a00700 1 -- 172.21.15.78:0/3104297287 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01e01027e0 msgr2=0x7f01e00736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.325+0000 7f01e5a00700 1 --2- 172.21.15.78:0/3104297287 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01e01027e0 0x7f01e00736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.325+0000 7f01e5a00700 1 -- 172.21.15.78:0/3104297287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d8009d00 con 0x7f01e0100c50 2024-03-14T09:53:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.326+0000 7f01e5a00700 1 --2- 172.21.15.78:0/3104297287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01e0100c50 0x7f01e0074b10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f01d400d980 tx=0x7f01d400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.326+0000 7f01de000700 1 -- 172.21.15.78:0/3104297287 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d400f890 con 0x7f01e0100c50 2024-03-14T09:53:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.326+0000 7f01de000700 1 -- 172.21.15.78:0/3104297287 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01d4010460 con 0x7f01e0100c50 2024-03-14T09:53:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.326+0000 7f01e8400700 1 -- 172.21.15.78:0/3104297287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e01a13f0 con 0x7f01e0100c50 2024-03-14T09:53:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.326+0000 7f01de000700 1 -- 172.21.15.78:0/3104297287 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d40195c0 con 0x7f01e0100c50 2024-03-14T09:53:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.326+0000 7f01e8400700 1 -- 172.21.15.78:0/3104297287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e01a1950 con 0x7f01e0100c50 2024-03-14T09:53:57.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.327+0000 7f01e8400700 1 -- 172.21.15.78:0/3104297287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e01080a0 con 0x7f01e0100c50 2024-03-14T09:53:57.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.329+0000 7f01de000700 1 -- 172.21.15.78:0/3104297287 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f01d40105d0 con 0x7f01e0100c50 2024-03-14T09:53:57.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.330+0000 7f01de000700 1 --2- 172.21.15.78:0/3104297287 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f01c80a0ba0 0x7f01c80a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:57.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.330+0000 7f01de000700 1 -- 172.21.15.78:0/3104297287 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f01d40cc1e0 con 0x7f01e0100c50 2024-03-14T09:53:57.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.331+0000 7f01e5000700 1 --2- 172.21.15.78:0/3104297287 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f01c80a0ba0 0x7f01c80a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:57.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.331+0000 7f01e5000700 1 --2- 172.21.15.78:0/3104297287 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f01c80a0ba0 0x7f01c80a3060 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f01cc005950 tx=0x7f01cc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:57.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.334+0000 7f01de000700 1 -- 172.21.15.78:0/3104297287 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f01d40956e0 con 0x7f01e0100c50 2024-03-14T09:53:57.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:57 smithi152 ceph-mon[48279]: from='client.25445 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:53:57.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:57 smithi152 ceph-mon[48279]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:53:57.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:57 smithi132 ceph-mon[47959]: from='client.25445 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:53:57.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:57 smithi132 ceph-mon[47959]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:53:57.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.544+0000 7f01e8400700 1 -- 172.21.15.78:0/3104297287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01e00679e0 con 0x7f01e0100c50 2024-03-14T09:53:57.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.546+0000 7f01de000700 1 -- 172.21.15.78:0/3104297287 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24594 (secure 0 0 0) 0x7f01d4098d00 con 0x7f01e0100c50 2024-03-14T09:53:57.546 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:57.547 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":87,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:53:42.786378+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:53:57.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.548+0000 7f01e8400700 1 -- 172.21.15.78:0/3104297287 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f01c80a0ba0 msgr2=0x7f01c80a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:57.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.548+0000 7f01e8400700 1 --2- 172.21.15.78:0/3104297287 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f01c80a0ba0 0x7f01c80a3060 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f01cc005950 tx=0x7f01cc00b410 comp rx=0 tx=0).stop 2024-03-14T09:53:57.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.548+0000 7f01e8400700 1 -- 172.21.15.78:0/3104297287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01e0100c50 msgr2=0x7f01e0074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:57.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.548+0000 7f01e8400700 1 --2- 172.21.15.78:0/3104297287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01e0100c50 0x7f01e0074b10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f01d400d980 tx=0x7f01d400dc90 comp rx=0 tx=0).stop 2024-03-14T09:53:57.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.548+0000 7f01e8400700 1 -- 172.21.15.78:0/3104297287 shutdown_connections 2024-03-14T09:53:57.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.548+0000 7f01e8400700 1 --2- 172.21.15.78:0/3104297287 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f01c80a0ba0 0x7f01c80a3060 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.548+0000 7f01e8400700 1 --2- 172.21.15.78:0/3104297287 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01e01027e0 0x7f01e00736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.548+0000 7f01e8400700 1 --2- 172.21.15.78:0/3104297287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01e0100c50 0x7f01e0074b10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.548+0000 7f01e8400700 1 --2- 172.21.15.78:0/3104297287 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f01e0101e40 0x7f01e00731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.548+0000 7f01e8400700 1 -- 172.21.15.78:0/3104297287 >> 172.21.15.78:0/3104297287 conn(0x7f01e00fc1c0 msgr2=0x7f01e01051e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:57.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.548+0000 7f01e8400700 1 -- 172.21.15.78:0/3104297287 shutdown_connections 2024-03-14T09:53:57.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.548+0000 7f01e8400700 1 -- 172.21.15.78:0/3104297287 wait complete. 2024-03-14T09:53:57.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:57 smithi078 ceph-mon[38143]: from='client.25445 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:53:57.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:57 smithi078 ceph-mon[38143]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:53:57.568 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T09:48:37.618450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-14T09:52:03.938666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-14T09:52:05.149985+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-14T09:52:08.227698+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-14T09:53:57.569 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T09:53:57.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.768+0000 7f19b9400700 1 -- 172.21.15.78:0/373809764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19b41028d0 msgr2=0x7f19b4109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:57.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.768+0000 7f19b9400700 1 --2- 172.21.15.78:0/373809764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19b41028d0 0x7f19b4109180 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f19ac00b0d0 tx=0x7f19ac00b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.768+0000 7f19b9400700 1 -- 172.21.15.78:0/373809764 shutdown_connections 2024-03-14T09:53:57.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.768+0000 7f19b9400700 1 --2- 172.21.15.78:0/373809764 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19b4100d40 0x7f19b4101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.768+0000 7f19b9400700 1 --2- 172.21.15.78:0/373809764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19b41028d0 0x7f19b4109180 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.768+0000 7f19b9400700 1 --2- 172.21.15.78:0/373809764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19b4101f30 0x7f19b4102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.768+0000 7f19b9400700 1 -- 172.21.15.78:0/373809764 >> 172.21.15.78:0/373809764 conn(0x7f19b40fc2b0 msgr2=0x7f19b40fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.768+0000 7f19b9400700 1 -- 172.21.15.78:0/373809764 shutdown_connections 2024-03-14T09:53:57.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.768+0000 7f19b9400700 1 -- 172.21.15.78:0/373809764 wait complete. 2024-03-14T09:53:57.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.769+0000 7f19b9400700 1 Processor -- start 2024-03-14T09:53:57.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.769+0000 7f19b9400700 1 -- start start 2024-03-14T09:53:57.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.770+0000 7f19b9400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19b4100d40 0x7f19b4197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:57.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.770+0000 7f19b9400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19b4101f30 0x7f19b4197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:57.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.770+0000 7f19b9400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19b41028d0 0x7f19b419e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:57.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.770+0000 7f19b9400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b419ed60 con 0x7f19b4101f30 2024-03-14T09:53:57.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.770+0000 7f19b9400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b419eea0 con 0x7f19b4100d40 2024-03-14T09:53:57.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.770+0000 7f19b9400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b419f030 con 0x7f19b41028d0 2024-03-14T09:53:57.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.770+0000 7f19b2800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19b4100d40 0x7f19b4197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:57.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.770+0000 7f19b3200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19b41028d0 0x7f19b419e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:57.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.770+0000 7f19b2800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19b4100d40 0x7f19b4197230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:44200/0 (socket says 172.21.15.78:44200) 2024-03-14T09:53:57.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.770+0000 7f19b1e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19b4101f30 0x7f19b4197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:57.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.770+0000 7f19b2800700 1 -- 172.21.15.78:0/2982517048 learned_addr learned my addr 172.21.15.78:0/2982517048 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:57.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.770+0000 7f19b2800700 1 -- 172.21.15.78:0/2982517048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19b41028d0 msgr2=0x7f19b419e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:57.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.770+0000 7f19b2800700 1 --2- 172.21.15.78:0/2982517048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19b41028d0 0x7f19b419e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.770+0000 7f19b2800700 1 -- 172.21.15.78:0/2982517048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19b4101f30 msgr2=0x7f19b4197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:57.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.770+0000 7f19b2800700 1 --2- 172.21.15.78:0/2982517048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19b4101f30 0x7f19b4197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.770+0000 7f19b2800700 1 -- 172.21.15.78:0/2982517048 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac009d00 con 0x7f19b4100d40 2024-03-14T09:53:57.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.771+0000 7f19b2800700 1 --2- 172.21.15.78:0/2982517048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19b4100d40 0x7f19b4197230 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f19a400e970 tx=0x7f19a400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:57.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.771+0000 7f19ab400700 1 -- 172.21.15.78:0/2982517048 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a4009790 con 0x7f19b4100d40 2024-03-14T09:53:57.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.771+0000 7f19b9400700 1 -- 172.21.15.78:0/2982517048 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b419f310 con 0x7f19b4100d40 2024-03-14T09:53:57.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.771+0000 7f19ab400700 1 -- 172.21.15.78:0/2982517048 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f19a400c7e0 con 0x7f19b4100d40 2024-03-14T09:53:57.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.771+0000 7f19ab400700 1 -- 172.21.15.78:0/2982517048 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a4003c70 con 0x7f19b4100d40 2024-03-14T09:53:57.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.771+0000 7f19b9400700 1 -- 172.21.15.78:0/2982517048 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b419f820 con 0x7f19b4100d40 2024-03-14T09:53:57.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.772+0000 7f19b9400700 1 -- 172.21.15.78:0/2982517048 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b4108140 con 0x7f19b4100d40 2024-03-14T09:53:57.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.774+0000 7f19ab400700 1 -- 172.21.15.78:0/2982517048 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f19a400cc50 con 0x7f19b4100d40 2024-03-14T09:53:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.775+0000 7f19ab400700 1 --2- 172.21.15.78:0/2982517048 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19980a0b20 0x7f19980a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.775+0000 7f19ab400700 1 -- 172.21.15.78:0/2982517048 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f19a4014070 con 0x7f19b4100d40 2024-03-14T09:53:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.776+0000 7f19b1e00700 1 --2- 172.21.15.78:0/2982517048 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19980a0b20 0x7f19980a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:57.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.776+0000 7f19b1e00700 1 --2- 172.21.15.78:0/2982517048 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19980a0b20 0x7f19980a2fe0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f19a0009c60 tx=0x7f19a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:57.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.780+0000 7f19ab400700 1 -- 172.21.15.78:0/2982517048 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f19a4095c00 con 0x7f19b4100d40 2024-03-14T09:53:57.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.987+0000 7f19b9400700 1 -- 172.21.15.78:0/2982517048 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f19b410cc00 con 0x7f19b4100d40 2024-03-14T09:53:57.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.988+0000 7f19ab400700 1 -- 172.21.15.78:0/2982517048 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f19a4003dd0 con 0x7f19b4100d40 2024-03-14T09:53:57.989 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-14T09:53:57.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.990+0000 7f19b9400700 1 -- 172.21.15.78:0/2982517048 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19980a0b20 msgr2=0x7f19980a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:57.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.990+0000 7f19b9400700 1 --2- 172.21.15.78:0/2982517048 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19980a0b20 0x7f19980a2fe0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f19a0009c60 tx=0x7f19a0009260 comp rx=0 tx=0).stop 2024-03-14T09:53:57.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.990+0000 7f19b9400700 1 -- 172.21.15.78:0/2982517048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19b4100d40 msgr2=0x7f19b4197230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:57.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.990+0000 7f19b9400700 1 --2- 172.21.15.78:0/2982517048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19b4100d40 0x7f19b4197230 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f19a400e970 tx=0x7f19a400ec80 comp rx=0 tx=0).stop 2024-03-14T09:53:57.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.991+0000 7f19b9400700 1 -- 172.21.15.78:0/2982517048 shutdown_connections 2024-03-14T09:53:57.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.991+0000 7f19b9400700 1 --2- 172.21.15.78:0/2982517048 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19980a0b20 0x7f19980a2fe0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.991+0000 7f19b9400700 1 --2- 172.21.15.78:0/2982517048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19b4100d40 0x7f19b4197230 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.991+0000 7f19b9400700 1 --2- 172.21.15.78:0/2982517048 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19b41028d0 0x7f19b419e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.991+0000 7f19b9400700 1 --2- 172.21.15.78:0/2982517048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19b4101f30 0x7f19b4197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:57.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.991+0000 7f19b9400700 1 -- 172.21.15.78:0/2982517048 >> 172.21.15.78:0/2982517048 conn(0x7f19b40fc2b0 msgr2=0x7f19b4105280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:57.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.991+0000 7f19b9400700 1 -- 172.21.15.78:0/2982517048 shutdown_connections 2024-03-14T09:53:57.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:57.991+0000 7f19b9400700 1 -- 172.21.15.78:0/2982517048 wait complete. 2024-03-14T09:53:58.010 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-14T09:53:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.206+0000 7f2813200700 1 -- 172.21.15.78:0/2325465496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f280c100d40 msgr2=0x7f280c101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.206+0000 7f2813200700 1 --2- 172.21.15.78:0/2325465496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f280c100d40 0x7f280c101160 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0099c0 tx=0x7f27fc009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.206+0000 7f2813200700 1 -- 172.21.15.78:0/2325465496 shutdown_connections 2024-03-14T09:53:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.206+0000 7f2813200700 1 --2- 172.21.15.78:0/2325465496 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f280c1028d0 0x7f280c109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.206+0000 7f2813200700 1 --2- 172.21.15.78:0/2325465496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f280c100d40 0x7f280c101160 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.206+0000 7f2813200700 1 --2- 172.21.15.78:0/2325465496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f280c101f30 0x7f280c102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.206+0000 7f2813200700 1 -- 172.21.15.78:0/2325465496 >> 172.21.15.78:0/2325465496 conn(0x7f280c0fc2b0 msgr2=0x7f280c0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.206+0000 7f2813200700 1 -- 172.21.15.78:0/2325465496 shutdown_connections 2024-03-14T09:53:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.206+0000 7f2813200700 1 -- 172.21.15.78:0/2325465496 wait complete. 2024-03-14T09:53:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.207+0000 7f2813200700 1 Processor -- start 2024-03-14T09:53:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.207+0000 7f2813200700 1 -- start start 2024-03-14T09:53:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.207+0000 7f2813200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f280c100d40 0x7f280c199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.207+0000 7f2813200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f280c101f30 0x7f280c1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.207+0000 7f2813200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f280c1028d0 0x7f280c1a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.207+0000 7f2813200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280c1a0e90 con 0x7f280c1028d0 2024-03-14T09:53:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.207+0000 7f2813200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280c1a0fd0 con 0x7f280c101f30 2024-03-14T09:53:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.207+0000 7f2813200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280c1a1160 con 0x7f280c100d40 2024-03-14T09:53:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.208+0000 7f2811200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f280c1028d0 0x7f280c1a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.208+0000 7f2811200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f280c1028d0 0x7f280c1a08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44008/0 (socket says 172.21.15.78:44008) 2024-03-14T09:53:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.208+0000 7f2811200700 1 -- 172.21.15.78:0/1771418080 learned_addr learned my addr 172.21.15.78:0/1771418080 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.208+0000 7f2811200700 1 -- 172.21.15.78:0/1771418080 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f280c100d40 msgr2=0x7f280c199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.208+0000 7f280be00700 1 --2- 172.21.15.78:0/1771418080 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f280c100d40 0x7f280c199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.208+0000 7f2811200700 1 --2- 172.21.15.78:0/1771418080 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f280c100d40 0x7f280c199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.208+0000 7f2811200700 1 -- 172.21.15.78:0/1771418080 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f280c101f30 msgr2=0x7f280c1998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.208+0000 7f2811200700 1 --2- 172.21.15.78:0/1771418080 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f280c101f30 0x7f280c1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.208+0000 7f2811200700 1 -- 172.21.15.78:0/1771418080 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc0095f0 con 0x7f280c1028d0 2024-03-14T09:53:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.208+0000 7f280be00700 1 --2- 172.21.15.78:0/1771418080 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f280c100d40 0x7f280c199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:53:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.208+0000 7f2811200700 1 --2- 172.21.15.78:0/1771418080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f280c1028d0 0x7f280c1a08c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2804007ee0 tx=0x7f280400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.209+0000 7f2808c00700 1 -- 172.21.15.78:0/1771418080 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2804017070 con 0x7f280c1028d0 2024-03-14T09:53:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.209+0000 7f2808c00700 1 -- 172.21.15.78:0/1771418080 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f280400f040 con 0x7f280c1028d0 2024-03-14T09:53:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.209+0000 7f2813200700 1 -- 172.21.15.78:0/1771418080 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f280c1a1440 con 0x7f280c1028d0 2024-03-14T09:53:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.209+0000 7f2808c00700 1 -- 172.21.15.78:0/1771418080 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2804005740 con 0x7f280c1028d0 2024-03-14T09:53:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.209+0000 7f2813200700 1 -- 172.21.15.78:0/1771418080 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f280c1a1950 con 0x7f280c1028d0 2024-03-14T09:53:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.210+0000 7f2813200700 1 -- 172.21.15.78:0/1771418080 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f280c067ac0 con 0x7f280c1028d0 2024-03-14T09:53:58.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.212+0000 7f2808c00700 1 -- 172.21.15.78:0/1771418080 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f28040058a0 con 0x7f280c1028d0 2024-03-14T09:53:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.213+0000 7f2808c00700 1 --2- 172.21.15.78:0/1771418080 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27f00a0b20 0x7f27f00a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.213+0000 7f2808c00700 1 -- 172.21.15.78:0/1771418080 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f28040cbe30 con 0x7f280c1028d0 2024-03-14T09:53:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.213+0000 7f280be00700 1 --2- 172.21.15.78:0/1771418080 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27f00a0b20 0x7f27f00a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.214+0000 7f280be00700 1 --2- 172.21.15.78:0/1771418080 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27f00a0b20 0x7f27f00a2fe0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f27fc006010 tx=0x7f27fc009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:58.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.217+0000 7f2808c00700 1 -- 172.21.15.78:0/1771418080 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f28040952d0 con 0x7f280c1028d0 2024-03-14T09:53:58.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:58 smithi078 ceph-mon[38143]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:53:58.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:58 smithi078 ceph-mon[38143]: pgmap v295: 129 pgs: 129 active+clean; 647 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-14T09:53:58.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:58 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3104297287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:58.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:58 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2982517048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:58.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.429+0000 7f2813200700 1 -- 172.21.15.78:0/1771418080 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f280c1a1c80 con 0x7f280c1028d0 2024-03-14T09:53:58.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.430+0000 7f2808c00700 1 -- 172.21.15.78:0/1771418080 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f28040988f0 con 0x7f280c1028d0 2024-03-14T09:53:58.430 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:53:58.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.432+0000 7f2813200700 1 -- 172.21.15.78:0/1771418080 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27f00a0b20 msgr2=0x7f27f00a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:58.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.432+0000 7f2813200700 1 --2- 172.21.15.78:0/1771418080 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27f00a0b20 0x7f27f00a2fe0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f27fc006010 tx=0x7f27fc009f10 comp rx=0 tx=0).stop 2024-03-14T09:53:58.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.432+0000 7f2813200700 1 -- 172.21.15.78:0/1771418080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f280c1028d0 msgr2=0x7f280c1a08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:58.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.432+0000 7f2813200700 1 --2- 172.21.15.78:0/1771418080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f280c1028d0 0x7f280c1a08c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f2804007ee0 tx=0x7f280400d3b0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.432+0000 7f2813200700 1 -- 172.21.15.78:0/1771418080 shutdown_connections 2024-03-14T09:53:58.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.432+0000 7f2813200700 1 --2- 172.21.15.78:0/1771418080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f280c1028d0 0x7f280c1a08c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.432+0000 7f2813200700 1 --2- 172.21.15.78:0/1771418080 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f280c100d40 0x7f280c199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.432+0000 7f2813200700 1 --2- 172.21.15.78:0/1771418080 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27f00a0b20 0x7f27f00a2fe0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.432+0000 7f2813200700 1 --2- 172.21.15.78:0/1771418080 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f280c101f30 0x7f280c1998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.432+0000 7f2813200700 1 -- 172.21.15.78:0/1771418080 >> 172.21.15.78:0/1771418080 conn(0x7f280c0fc2b0 msgr2=0x7f280c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:58.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.432+0000 7f2813200700 1 -- 172.21.15.78:0/1771418080 shutdown_connections 2024-03-14T09:53:58.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.432+0000 7f2813200700 1 -- 172.21.15.78:0/1771418080 wait complete. 2024-03-14T09:53:58.451 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-14T09:53:58.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:58 smithi152 ceph-mon[48279]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:53:58.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:58 smithi152 ceph-mon[48279]: pgmap v295: 129 pgs: 129 active+clean; 647 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-14T09:53:58.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:58 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3104297287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:58.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:58 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2982517048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:58.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:58 smithi132 ceph-mon[47959]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T09:53:58.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:58 smithi132 ceph-mon[47959]: pgmap v295: 129 pgs: 129 active+clean; 647 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-14T09:53:58.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:58 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3104297287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:53:58.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:58 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2982517048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T09:53:58.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.646+0000 7ff329c00700 1 -- 172.21.15.78:0/887325758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324073810 msgr2=0x7ff324073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:58.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.646+0000 7ff329c00700 1 --2- 172.21.15.78:0/887325758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324073810 0x7ff324073c90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff310009990 tx=0x7ff310009ca0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.646+0000 7ff329c00700 1 -- 172.21.15.78:0/887325758 shutdown_connections 2024-03-14T09:53:58.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.646+0000 7ff329c00700 1 --2- 172.21.15.78:0/887325758 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff324074df0 0x7ff324073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.646+0000 7ff329c00700 1 --2- 172.21.15.78:0/887325758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff324108c00 0x7ff32410b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.646+0000 7ff329c00700 1 --2- 172.21.15.78:0/887325758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324073810 0x7ff324073c90 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.646+0000 7ff329c00700 1 -- 172.21.15.78:0/887325758 >> 172.21.15.78:0/887325758 conn(0x7ff3240fc2b0 msgr2=0x7ff3240fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:58.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.646+0000 7ff329c00700 1 -- 172.21.15.78:0/887325758 shutdown_connections 2024-03-14T09:53:58.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.647+0000 7ff329c00700 1 -- 172.21.15.78:0/887325758 wait complete. 2024-03-14T09:53:58.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.647+0000 7ff329c00700 1 Processor -- start 2024-03-14T09:53:58.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.648+0000 7ff329c00700 1 -- start start 2024-03-14T09:53:58.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.648+0000 7ff329c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324073810 0x7ff324199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:58.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.648+0000 7ff329c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff324074df0 0x7ff3241998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:58.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.648+0000 7ff329c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff324108c00 0x7ff3241a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:58.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.648+0000 7ff329c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3241a0e70 con 0x7ff324073810 2024-03-14T09:53:58.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.648+0000 7ff329c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3241a0fb0 con 0x7ff324074df0 2024-03-14T09:53:58.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.648+0000 7ff329c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3241a1140 con 0x7ff324108c00 2024-03-14T09:53:58.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.648+0000 7ff322a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324073810 0x7ff324199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:58.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.648+0000 7ff322000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff324074df0 0x7ff3241998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:58.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.648+0000 7ff322a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324073810 0x7ff324199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44022/0 (socket says 172.21.15.78:44022) 2024-03-14T09:53:58.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.648+0000 7ff322a00700 1 -- 172.21.15.78:0/3767398090 learned_addr learned my addr 172.21.15.78:0/3767398090 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:58.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.648+0000 7ff323400700 1 --2- 172.21.15.78:0/3767398090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff324108c00 0x7ff3241a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:58.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.649+0000 7ff322000700 1 -- 172.21.15.78:0/3767398090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff324108c00 msgr2=0x7ff3241a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:58.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.649+0000 7ff322000700 1 --2- 172.21.15.78:0/3767398090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff324108c00 0x7ff3241a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.649+0000 7ff322000700 1 -- 172.21.15.78:0/3767398090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324073810 msgr2=0x7ff324199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:58.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.649+0000 7ff322000700 1 --2- 172.21.15.78:0/3767398090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324073810 0x7ff324199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.649+0000 7ff322000700 1 -- 172.21.15.78:0/3767398090 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3100095f0 con 0x7ff324074df0 2024-03-14T09:53:58.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.649+0000 7ff323400700 1 --2- 172.21.15.78:0/3767398090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff324108c00 0x7ff3241a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:53:58.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.649+0000 7ff322000700 1 --2- 172.21.15.78:0/3767398090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff324074df0 0x7ff3241998d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff31000b680 tx=0x7ff310004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:58.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.649+0000 7ff31b400700 1 -- 172.21.15.78:0/3767398090 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff31001d070 con 0x7ff324074df0 2024-03-14T09:53:58.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.649+0000 7ff31b400700 1 -- 172.21.15.78:0/3767398090 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff310021410 con 0x7ff324074df0 2024-03-14T09:53:58.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.649+0000 7ff329c00700 1 -- 172.21.15.78:0/3767398090 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3241a13c0 con 0x7ff324074df0 2024-03-14T09:53:58.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.649+0000 7ff31b400700 1 -- 172.21.15.78:0/3767398090 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff310017650 con 0x7ff324074df0 2024-03-14T09:53:58.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.649+0000 7ff329c00700 1 -- 172.21.15.78:0/3767398090 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3241a17e0 con 0x7ff324074df0 2024-03-14T09:53:58.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.650+0000 7ff329c00700 1 -- 172.21.15.78:0/3767398090 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3241079c0 con 0x7ff324074df0 2024-03-14T09:53:58.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.652+0000 7ff31b400700 1 -- 172.21.15.78:0/3767398090 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7ff310021580 con 0x7ff324074df0 2024-03-14T09:53:58.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.653+0000 7ff31b400700 1 --2- 172.21.15.78:0/3767398090 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff3080a0b20 0x7ff3080a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:58.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.654+0000 7ff31b400700 1 -- 172.21.15.78:0/3767398090 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7ff3100cd9b0 con 0x7ff324074df0 2024-03-14T09:53:58.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.654+0000 7ff322a00700 1 --2- 172.21.15.78:0/3767398090 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff3080a0b20 0x7ff3080a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:58.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.654+0000 7ff322a00700 1 --2- 172.21.15.78:0/3767398090 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff3080a0b20 0x7ff3080a2fe0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff314009d20 tx=0x7ff314009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:58.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.658+0000 7ff31b400700 1 -- 172.21.15.78:0/3767398090 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff310096ee0 con 0x7ff324074df0 2024-03-14T09:53:58.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.865+0000 7ff329c00700 1 -- 172.21.15.78:0/3767398090 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff32410edd0 con 0x7ff324074df0 2024-03-14T09:53:58.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.866+0000 7ff31b400700 1 -- 172.21.15.78:0/3767398090 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff31009a500 con 0x7ff324074df0 2024-03-14T09:53:58.867 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-14T09:53:58.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.869+0000 7ff329c00700 1 -- 172.21.15.78:0/3767398090 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff3080a0b20 msgr2=0x7ff3080a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:58.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.869+0000 7ff329c00700 1 --2- 172.21.15.78:0/3767398090 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff3080a0b20 0x7ff3080a2fe0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff314009d20 tx=0x7ff314009260 comp rx=0 tx=0).stop 2024-03-14T09:53:58.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.869+0000 7ff329c00700 1 -- 172.21.15.78:0/3767398090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff324074df0 msgr2=0x7ff3241998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:58.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.869+0000 7ff329c00700 1 --2- 172.21.15.78:0/3767398090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff324074df0 0x7ff3241998d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff31000b680 tx=0x7ff310004730 comp rx=0 tx=0).stop 2024-03-14T09:53:58.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.869+0000 7ff329c00700 1 -- 172.21.15.78:0/3767398090 shutdown_connections 2024-03-14T09:53:58.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.869+0000 7ff329c00700 1 --2- 172.21.15.78:0/3767398090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff324108c00 0x7ff3241a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.869+0000 7ff329c00700 1 --2- 172.21.15.78:0/3767398090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff324073810 0x7ff324199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.869+0000 7ff329c00700 1 --2- 172.21.15.78:0/3767398090 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7ff3080a0b20 0x7ff3080a2fe0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.869+0000 7ff329c00700 1 --2- 172.21.15.78:0/3767398090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff324074df0 0x7ff3241998d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:58.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.869+0000 7ff329c00700 1 -- 172.21.15.78:0/3767398090 >> 172.21.15.78:0/3767398090 conn(0x7ff3240fc2b0 msgr2=0x7ff324104b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:58.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.869+0000 7ff329c00700 1 -- 172.21.15.78:0/3767398090 shutdown_connections 2024-03-14T09:53:58.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:58.869+0000 7ff329c00700 1 -- 172.21.15.78:0/3767398090 wait complete. 2024-03-14T09:53:58.888 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-14T09:53:59.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.083+0000 7fe062a00700 1 -- 172.21.15.78:0/2836652172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c101f40 msgr2=0x7fe05c1023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:59.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.083+0000 7fe062a00700 1 --2- 172.21.15.78:0/2836652172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c101f40 0x7fe05c1023a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe0480099c0 tx=0x7fe048009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.083+0000 7fe062a00700 1 -- 172.21.15.78:0/2836652172 shutdown_connections 2024-03-14T09:53:59.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.083+0000 7fe062a00700 1 --2- 172.21.15.78:0/2836652172 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe05c1028e0 0x7fe05c109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.083+0000 7fe062a00700 1 --2- 172.21.15.78:0/2836652172 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe05c100d50 0x7fe05c101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.083+0000 7fe062a00700 1 --2- 172.21.15.78:0/2836652172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c101f40 0x7fe05c1023a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.083+0000 7fe062a00700 1 -- 172.21.15.78:0/2836652172 >> 172.21.15.78:0/2836652172 conn(0x7fe05c0fc2c0 msgr2=0x7fe05c0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:59.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.083+0000 7fe062a00700 1 -- 172.21.15.78:0/2836652172 shutdown_connections 2024-03-14T09:53:59.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.083+0000 7fe062a00700 1 -- 172.21.15.78:0/2836652172 wait complete. 2024-03-14T09:53:59.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.084+0000 7fe062a00700 1 Processor -- start 2024-03-14T09:53:59.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.084+0000 7fe062a00700 1 -- start start 2024-03-14T09:53:59.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe062a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe05c100d50 0x7fe05c197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:59.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe062a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe05c101f40 0x7fe05c197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:59.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe062a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c1028e0 0x7fe05c19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:59.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe062a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c19ec30 con 0x7fe05c1028e0 2024-03-14T09:53:59.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe062a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c19ed70 con 0x7fe05c101f40 2024-03-14T09:53:59.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe062a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c19ef00 con 0x7fe05c100d50 2024-03-14T09:53:59.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe060a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c1028e0 0x7fe05c19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:59.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe05be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe05c100d50 0x7fe05c197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:59.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe05b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe05c101f40 0x7fe05c197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:59.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe05be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe05c100d50 0x7fe05c197100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35542/0 (socket says 172.21.15.78:35542) 2024-03-14T09:53:59.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe05be00700 1 -- 172.21.15.78:0/2292744312 learned_addr learned my addr 172.21.15.78:0/2292744312 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:59.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe05be00700 1 -- 172.21.15.78:0/2292744312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe05c101f40 msgr2=0x7fe05c197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:59.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe05be00700 1 --2- 172.21.15.78:0/2292744312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe05c101f40 0x7fe05c197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe05be00700 1 -- 172.21.15.78:0/2292744312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c1028e0 msgr2=0x7fe05c19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:59.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe05be00700 1 --2- 172.21.15.78:0/2292744312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c1028e0 0x7fe05c19e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe05be00700 1 -- 172.21.15.78:0/2292744312 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0480095f0 con 0x7fe05c100d50 2024-03-14T09:53:59.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.085+0000 7fe060a00700 1 --2- 172.21.15.78:0/2292744312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c1028e0 0x7fe05c19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:53:59.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.086+0000 7fe05be00700 1 --2- 172.21.15.78:0/2292744312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe05c100d50 0x7fe05c197100 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe04c00e970 tx=0x7fe04c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:59.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.086+0000 7fe058c00700 1 -- 172.21.15.78:0/2292744312 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04c009790 con 0x7fe05c100d50 2024-03-14T09:53:59.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.086+0000 7fe058c00700 1 -- 172.21.15.78:0/2292744312 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe04c00c7e0 con 0x7fe05c100d50 2024-03-14T09:53:59.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.086+0000 7fe062a00700 1 -- 172.21.15.78:0/2292744312 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe05c19f1e0 con 0x7fe05c100d50 2024-03-14T09:53:59.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.086+0000 7fe058c00700 1 -- 172.21.15.78:0/2292744312 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04c003c70 con 0x7fe05c100d50 2024-03-14T09:53:59.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.086+0000 7fe062a00700 1 -- 172.21.15.78:0/2292744312 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05c19f6f0 con 0x7fe05c100d50 2024-03-14T09:53:59.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.087+0000 7fe062a00700 1 -- 172.21.15.78:0/2292744312 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe05c108220 con 0x7fe05c100d50 2024-03-14T09:53:59.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.089+0000 7fe058c00700 1 -- 172.21.15.78:0/2292744312 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fe04c00cc50 con 0x7fe05c100d50 2024-03-14T09:53:59.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.090+0000 7fe058c00700 1 --2- 172.21.15.78:0/2292744312 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe0400a0b20 0x7fe0400a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:59.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.090+0000 7fe058c00700 1 -- 172.21.15.78:0/2292744312 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7fe04c014070 con 0x7fe05c100d50 2024-03-14T09:53:59.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.091+0000 7fe05b400700 1 --2- 172.21.15.78:0/2292744312 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe0400a0b20 0x7fe0400a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:59.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.091+0000 7fe05b400700 1 --2- 172.21.15.78:0/2292744312 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe0400a0b20 0x7fe0400a2fe0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe048003730 tx=0x7fe04800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:59.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.093+0000 7fe058c00700 1 -- 172.21.15.78:0/2292744312 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe04c095d70 con 0x7fe05c100d50 2024-03-14T09:53:59.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.300+0000 7fe062a00700 1 -- 172.21.15.78:0/2292744312 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe05c10cc80 con 0x7fe05c100d50 2024-03-14T09:53:59.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.301+0000 7fe058c00700 1 -- 172.21.15.78:0/2292744312 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe04c003dd0 con 0x7fe05c100d50 2024-03-14T09:53:59.301 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T09:53:59.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.303+0000 7fe062a00700 1 -- 172.21.15.78:0/2292744312 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe0400a0b20 msgr2=0x7fe0400a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:59.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.303+0000 7fe062a00700 1 --2- 172.21.15.78:0/2292744312 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe0400a0b20 0x7fe0400a2fe0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe048003730 tx=0x7fe04800b540 comp rx=0 tx=0).stop 2024-03-14T09:53:59.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.303+0000 7fe062a00700 1 -- 172.21.15.78:0/2292744312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe05c100d50 msgr2=0x7fe05c197100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:59.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.303+0000 7fe062a00700 1 --2- 172.21.15.78:0/2292744312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe05c100d50 0x7fe05c197100 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe04c00e970 tx=0x7fe04c00ec80 comp rx=0 tx=0).stop 2024-03-14T09:53:59.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.303+0000 7fe062a00700 1 -- 172.21.15.78:0/2292744312 shutdown_connections 2024-03-14T09:53:59.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.303+0000 7fe062a00700 1 --2- 172.21.15.78:0/2292744312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe05c1028e0 0x7fe05c19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.303+0000 7fe062a00700 1 --2- 172.21.15.78:0/2292744312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe05c100d50 0x7fe05c197100 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.303+0000 7fe062a00700 1 --2- 172.21.15.78:0/2292744312 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fe0400a0b20 0x7fe0400a2fe0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.303+0000 7fe062a00700 1 --2- 172.21.15.78:0/2292744312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe05c101f40 0x7fe05c197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.303+0000 7fe062a00700 1 -- 172.21.15.78:0/2292744312 >> 172.21.15.78:0/2292744312 conn(0x7fe05c0fc2c0 msgr2=0x7fe05c105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:59.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.303+0000 7fe062a00700 1 -- 172.21.15.78:0/2292744312 shutdown_connections 2024-03-14T09:53:59.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.303+0000 7fe062a00700 1 -- 172.21.15.78:0/2292744312 wait complete. 2024-03-14T09:53:59.323 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:53:59.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:59 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1771418080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:59.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:53:59 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3767398090' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:59.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.515+0000 7f19c9600700 1 -- 172.21.15.78:0/1781086286 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19c4100c50 msgr2=0x7f19c4101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:59.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.515+0000 7f19c9600700 1 --2- 172.21.15.78:0/1781086286 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19c4100c50 0x7f19c4101070 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f19b40099c0 tx=0x7f19b4009cd0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.515+0000 7f19c9600700 1 -- 172.21.15.78:0/1781086286 shutdown_connections 2024-03-14T09:53:59.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.515+0000 7f19c9600700 1 --2- 172.21.15.78:0/1781086286 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19c41027e0 0x7f19c4109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.515+0000 7f19c9600700 1 --2- 172.21.15.78:0/1781086286 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19c4100c50 0x7f19c4101070 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.515+0000 7f19c9600700 1 --2- 172.21.15.78:0/1781086286 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19c4101e40 0x7f19c41022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.515+0000 7f19c9600700 1 -- 172.21.15.78:0/1781086286 >> 172.21.15.78:0/1781086286 conn(0x7f19c40fc1c0 msgr2=0x7f19c40fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:59.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.515+0000 7f19c9600700 1 -- 172.21.15.78:0/1781086286 shutdown_connections 2024-03-14T09:53:59.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.515+0000 7f19c9600700 1 -- 172.21.15.78:0/1781086286 wait complete. 2024-03-14T09:53:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.516+0000 7f19c9600700 1 Processor -- start 2024-03-14T09:53:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.516+0000 7f19c9600700 1 -- start start 2024-03-14T09:53:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.516+0000 7f19c9600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19c4100c50 0x7f19c4197040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.516+0000 7f19c9600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19c4101e40 0x7f19c4197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.516+0000 7f19c9600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19c41027e0 0x7f19c419e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.516+0000 7f19c9600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c419eb70 con 0x7f19c4100c50 2024-03-14T09:53:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.516+0000 7f19c9600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c419ecb0 con 0x7f19c4101e40 2024-03-14T09:53:59.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.516+0000 7f19c9600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c419ee40 con 0x7f19c41027e0 2024-03-14T09:53:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.517+0000 7f19c2a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19c4100c50 0x7f19c4197040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.517+0000 7f19c2a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19c4100c50 0x7f19c4197040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44066/0 (socket says 172.21.15.78:44066) 2024-03-14T09:53:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.517+0000 7f19c2a00700 1 -- 172.21.15.78:0/3176073465 learned_addr learned my addr 172.21.15.78:0/3176073465 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.517+0000 7f19c3400700 1 --2- 172.21.15.78:0/3176073465 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19c41027e0 0x7f19c419e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.517+0000 7f19c2000700 1 --2- 172.21.15.78:0/3176073465 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19c4101e40 0x7f19c4197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.517+0000 7f19c2a00700 1 -- 172.21.15.78:0/3176073465 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19c41027e0 msgr2=0x7f19c419e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.517+0000 7f19c2a00700 1 --2- 172.21.15.78:0/3176073465 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19c41027e0 0x7f19c419e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.517+0000 7f19c2a00700 1 -- 172.21.15.78:0/3176073465 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19c4101e40 msgr2=0x7f19c4197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.517+0000 7f19c2a00700 1 --2- 172.21.15.78:0/3176073465 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19c4101e40 0x7f19c4197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.517+0000 7f19c2a00700 1 -- 172.21.15.78:0/3176073465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b40095f0 con 0x7f19c4100c50 2024-03-14T09:53:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.517+0000 7f19c2a00700 1 --2- 172.21.15.78:0/3176073465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19c4100c50 0x7f19c4197040 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f19b4006010 tx=0x7f19b4005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.517+0000 7f19bb400700 1 -- 172.21.15.78:0/3176073465 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b401d070 con 0x7f19c4100c50 2024-03-14T09:53:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.517+0000 7f19c9600700 1 -- 172.21.15.78:0/3176073465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c419f0c0 con 0x7f19c4100c50 2024-03-14T09:53:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.517+0000 7f19bb400700 1 -- 172.21.15.78:0/3176073465 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f19b4004050 con 0x7f19c4100c50 2024-03-14T09:53:59.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.518+0000 7f19c9600700 1 -- 172.21.15.78:0/3176073465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c419f4e0 con 0x7f19c4100c50 2024-03-14T09:53:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.518+0000 7f19bb400700 1 -- 172.21.15.78:0/3176073465 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b4022da0 con 0x7f19c4100c50 2024-03-14T09:53:59.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:59 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1771418080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:59.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:53:59 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3767398090' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:59.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.519+0000 7f19c9600700 1 -- 172.21.15.78:0/3176073465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c4108120 con 0x7f19c4100c50 2024-03-14T09:53:59.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.520+0000 7f19bb400700 1 -- 172.21.15.78:0/3176073465 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f19b4022500 con 0x7f19c4100c50 2024-03-14T09:53:59.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.521+0000 7f19bb400700 1 --2- 172.21.15.78:0/3176073465 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19a80a0b20 0x7f19a80a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:59.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.522+0000 7f19bb400700 1 -- 172.21.15.78:0/3176073465 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f19b40cd850 con 0x7f19c4100c50 2024-03-14T09:53:59.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.522+0000 7f19c2000700 1 --2- 172.21.15.78:0/3176073465 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19a80a0b20 0x7f19a80a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:59.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.522+0000 7f19c2000700 1 --2- 172.21.15.78:0/3176073465 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19a80a0b20 0x7f19a80a2fe0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f19b0005950 tx=0x7f19b00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:59.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.525+0000 7f19bb400700 1 -- 172.21.15.78:0/3176073465 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f19b4096de0 con 0x7f19c4100c50 2024-03-14T09:53:59.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:59 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1771418080' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T09:53:59.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:53:59 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3767398090' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T09:53:59.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.777+0000 7f19c9600700 1 -- 172.21.15.78:0/3176073465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19c40679e0 con 0x7f19c4100c50 2024-03-14T09:53:59.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.778+0000 7f19bb400700 1 -- 172.21.15.78:0/3176073465 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f19b409a400 con 0x7f19c4100c50 2024-03-14T09:53:59.778 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:53:59.779 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:53:59.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.780+0000 7f19c9600700 1 -- 172.21.15.78:0/3176073465 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19a80a0b20 msgr2=0x7f19a80a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:59.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.780+0000 7f19c9600700 1 --2- 172.21.15.78:0/3176073465 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19a80a0b20 0x7f19a80a2fe0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f19b0005950 tx=0x7f19b00058e0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.780+0000 7f19c9600700 1 -- 172.21.15.78:0/3176073465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19c4100c50 msgr2=0x7f19c4197040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:59.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.780+0000 7f19c9600700 1 --2- 172.21.15.78:0/3176073465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19c4100c50 0x7f19c4197040 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f19b4006010 tx=0x7f19b4005740 comp rx=0 tx=0).stop 2024-03-14T09:53:59.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.780+0000 7f19c9600700 1 -- 172.21.15.78:0/3176073465 shutdown_connections 2024-03-14T09:53:59.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.780+0000 7f19c9600700 1 --2- 172.21.15.78:0/3176073465 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f19c41027e0 0x7f19c419e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.780+0000 7f19c9600700 1 --2- 172.21.15.78:0/3176073465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19c4100c50 0x7f19c4197040 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.780+0000 7f19c9600700 1 --2- 172.21.15.78:0/3176073465 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f19a80a0b20 0x7f19a80a2fe0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.780+0000 7f19c9600700 1 --2- 172.21.15.78:0/3176073465 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19c4101e40 0x7f19c4197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.780+0000 7f19c9600700 1 -- 172.21.15.78:0/3176073465 >> 172.21.15.78:0/3176073465 conn(0x7f19c40fc1c0 msgr2=0x7f19c4105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:59.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.780+0000 7f19c9600700 1 -- 172.21.15.78:0/3176073465 shutdown_connections 2024-03-14T09:53:59.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.780+0000 7f19c9600700 1 -- 172.21.15.78:0/3176073465 wait complete. 2024-03-14T09:53:59.782 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T09:53:59.800 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T09:53:59.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.992+0000 7fd26f000700 1 -- 172.21.15.78:0/1256586197 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2681027e0 msgr2=0x7fd268109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:59.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.992+0000 7fd26f000700 1 --2- 172.21.15.78:0/1256586197 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2681027e0 0x7fd268109090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd26000b0d0 tx=0x7fd26000b3e0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.992+0000 7fd26f000700 1 -- 172.21.15.78:0/1256586197 shutdown_connections 2024-03-14T09:53:59.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.992+0000 7fd26f000700 1 --2- 172.21.15.78:0/1256586197 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2681027e0 0x7fd268109090 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.992+0000 7fd26f000700 1 --2- 172.21.15.78:0/1256586197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd268100c50 0x7fd268101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.992+0000 7fd26f000700 1 --2- 172.21.15.78:0/1256586197 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd268101e40 0x7fd2681022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.992+0000 7fd26f000700 1 -- 172.21.15.78:0/1256586197 >> 172.21.15.78:0/1256586197 conn(0x7fd2680fc1c0 msgr2=0x7fd2680fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:53:59.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.992+0000 7fd26f000700 1 -- 172.21.15.78:0/1256586197 shutdown_connections 2024-03-14T09:53:59.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.992+0000 7fd26f000700 1 -- 172.21.15.78:0/1256586197 wait complete. 2024-03-14T09:53:59.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.993+0000 7fd26f000700 1 Processor -- start 2024-03-14T09:53:59.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.993+0000 7fd26f000700 1 -- start start 2024-03-14T09:53:59.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.993+0000 7fd26f000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd268100c50 0x7fd268199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.993+0000 7fd26f000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd268101e40 0x7fd268199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.993+0000 7fd26f000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2681027e0 0x7fd2681a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.993+0000 7fd26f000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2681a0e50 con 0x7fd268100c50 2024-03-14T09:53:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.993+0000 7fd26f000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2681a0f90 con 0x7fd268101e40 2024-03-14T09:53:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.993+0000 7fd26f000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2681a1120 con 0x7fd2681027e0 2024-03-14T09:53:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.994+0000 7fd267e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd268100c50 0x7fd268199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.994+0000 7fd267e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd268100c50 0x7fd268199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44088/0 (socket says 172.21.15.78:44088) 2024-03-14T09:53:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.994+0000 7fd267e00700 1 -- 172.21.15.78:0/3327204701 learned_addr learned my addr 172.21.15.78:0/3327204701 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:53:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.994+0000 7fd267400700 1 --2- 172.21.15.78:0/3327204701 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd268101e40 0x7fd268199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.994+0000 7fd26d000700 1 --2- 172.21.15.78:0/3327204701 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2681027e0 0x7fd2681a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:53:59.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.994+0000 7fd267e00700 1 -- 172.21.15.78:0/3327204701 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2681027e0 msgr2=0x7fd2681a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.994+0000 7fd267e00700 1 --2- 172.21.15.78:0/3327204701 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2681027e0 0x7fd2681a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.994+0000 7fd267e00700 1 -- 172.21.15.78:0/3327204701 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd268101e40 msgr2=0x7fd268199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:53:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.994+0000 7fd267e00700 1 --2- 172.21.15.78:0/3327204701 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd268101e40 0x7fd268199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:53:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.994+0000 7fd267e00700 1 -- 172.21.15.78:0/3327204701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd258009520 con 0x7fd268100c50 2024-03-14T09:53:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.994+0000 7fd267e00700 1 --2- 172.21.15.78:0/3327204701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd268100c50 0x7fd268199320 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd25800ea50 tx=0x7fd25800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:53:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.995+0000 7fd264c00700 1 -- 172.21.15.78:0/3327204701 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25800cbd0 con 0x7fd268100c50 2024-03-14T09:53:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.995+0000 7fd264c00700 1 -- 172.21.15.78:0/3327204701 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd25800cd30 con 0x7fd268100c50 2024-03-14T09:53:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.995+0000 7fd26f000700 1 -- 172.21.15.78:0/3327204701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd260009d00 con 0x7fd268100c50 2024-03-14T09:53:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.995+0000 7fd264c00700 1 -- 172.21.15.78:0/3327204701 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258018680 con 0x7fd268100c50 2024-03-14T09:53:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.995+0000 7fd26f000700 1 -- 172.21.15.78:0/3327204701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2681a1730 con 0x7fd268100c50 2024-03-14T09:53:59.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.996+0000 7fd26f000700 1 -- 172.21.15.78:0/3327204701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2680679e0 con 0x7fd268100c50 2024-03-14T09:53:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.998+0000 7fd264c00700 1 -- 172.21.15.78:0/3327204701 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7fd258010430 con 0x7fd268100c50 2024-03-14T09:53:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.999+0000 7fd264c00700 1 --2- 172.21.15.78:0/3327204701 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd24c0a0b20 0x7fd24c0a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:53:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.999+0000 7fd264c00700 1 -- 172.21.15.78:0/3327204701 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7fd258014070 con 0x7fd268100c50 2024-03-14T09:53:59.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:53:59.999+0000 7fd267400700 1 --2- 172.21.15.78:0/3327204701 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd24c0a0b20 0x7fd24c0a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.000+0000 7fd267400700 1 --2- 172.21.15.78:0/3327204701 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd24c0a0b20 0x7fd24c0a2fe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd254009c60 tx=0x7fd254009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:00.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.003+0000 7fd264c00700 1 -- 172.21.15.78:0/3327204701 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd258095ef0 con 0x7fd268100c50 2024-03-14T09:54:00.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.222+0000 7fd26f000700 1 -- 172.21.15.78:0/3327204701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd26810c8e0 con 0x7fd268100c50 2024-03-14T09:54:00.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.224+0000 7fd264c00700 1 -- 172.21.15.78:0/3327204701 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24594 (secure 0 0 0) 0x7fd258099510 con 0x7fd268100c50 2024-03-14T09:54:00.224 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:54:00.225 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":87,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T09:53:42.786378+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T09:54:00.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.226+0000 7fd26f000700 1 -- 172.21.15.78:0/3327204701 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd24c0a0b20 msgr2=0x7fd24c0a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.226+0000 7fd26f000700 1 --2- 172.21.15.78:0/3327204701 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd24c0a0b20 0x7fd24c0a2fe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd254009c60 tx=0x7fd254009260 comp rx=0 tx=0).stop 2024-03-14T09:54:00.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.226+0000 7fd26f000700 1 -- 172.21.15.78:0/3327204701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd268100c50 msgr2=0x7fd268199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.226+0000 7fd26f000700 1 --2- 172.21.15.78:0/3327204701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd268100c50 0x7fd268199320 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd25800ea50 tx=0x7fd25800ed60 comp rx=0 tx=0).stop 2024-03-14T09:54:00.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.226+0000 7fd26f000700 1 -- 172.21.15.78:0/3327204701 shutdown_connections 2024-03-14T09:54:00.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.226+0000 7fd26f000700 1 --2- 172.21.15.78:0/3327204701 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd2681027e0 0x7fd2681a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.226+0000 7fd26f000700 1 --2- 172.21.15.78:0/3327204701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd268100c50 0x7fd268199320 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.226+0000 7fd26f000700 1 --2- 172.21.15.78:0/3327204701 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fd24c0a0b20 0x7fd24c0a2fe0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.226+0000 7fd26f000700 1 --2- 172.21.15.78:0/3327204701 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd268101e40 0x7fd268199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.226+0000 7fd26f000700 1 -- 172.21.15.78:0/3327204701 >> 172.21.15.78:0/3327204701 conn(0x7fd2680fc1c0 msgr2=0x7fd268072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:00.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.227+0000 7fd26f000700 1 -- 172.21.15.78:0/3327204701 shutdown_connections 2024-03-14T09:54:00.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.227+0000 7fd26f000700 1 -- 172.21.15.78:0/3327204701 wait complete. 2024-03-14T09:54:00.247 DEBUG:tasks.fwd_scrub:Yielding 2024-03-14T09:54:00.247 INFO:teuthology.run_tasks:Running task exec... 2024-03-14T09:54:00.257 INFO:teuthology.task.exec:Executing custom commands... 2024-03-14T09:54:00.257 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-03-14T09:54:00.257 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-14T09:54:00.257 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-14T09:54:00.257 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T09:54:00.257 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T09:54:00.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.451+0000 7f604a000700 1 -- 172.21.15.78:0/895311124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6044100cf0 msgr2=0x7f6044101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.451+0000 7f604a000700 1 --2- 172.21.15.78:0/895311124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6044100cf0 0x7f6044101110 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f60340099c0 tx=0x7f6034009cd0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.451+0000 7f604a000700 1 -- 172.21.15.78:0/895311124 shutdown_connections 2024-03-14T09:54:00.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.451+0000 7f604a000700 1 --2- 172.21.15.78:0/895311124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6044102880 0x7f6044109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.451+0000 7f604a000700 1 --2- 172.21.15.78:0/895311124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6044100cf0 0x7f6044101110 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.451+0000 7f604a000700 1 --2- 172.21.15.78:0/895311124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6044101ee0 0x7f6044102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.451+0000 7f604a000700 1 -- 172.21.15.78:0/895311124 >> 172.21.15.78:0/895311124 conn(0x7f60440fc280 msgr2=0x7f60440fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:00.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.451+0000 7f604a000700 1 -- 172.21.15.78:0/895311124 shutdown_connections 2024-03-14T09:54:00.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.451+0000 7f604a000700 1 -- 172.21.15.78:0/895311124 wait complete. 2024-03-14T09:54:00.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.452+0000 7f604a000700 1 Processor -- start 2024-03-14T09:54:00.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.452+0000 7f604a000700 1 -- start start 2024-03-14T09:54:00.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.452+0000 7f604a000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6044100cf0 0x7f6044199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.452+0000 7f604a000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6044101ee0 0x7f6044199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.452+0000 7f604a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6044102880 0x7f60441a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.452+0000 7f604a000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60441a0dd0 con 0x7f6044102880 2024-03-14T09:54:00.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.452+0000 7f604a000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60441a0f10 con 0x7f6044101ee0 2024-03-14T09:54:00.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.452+0000 7f604a000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60441a10a0 con 0x7f6044100cf0 2024-03-14T09:54:00.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.453+0000 7f6043e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6044102880 0x7f60441a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.453+0000 7f6043e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6044102880 0x7f60441a0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44114/0 (socket says 172.21.15.78:44114) 2024-03-14T09:54:00.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.453+0000 7f6043e00700 1 -- 172.21.15.78:0/4253601430 learned_addr learned my addr 172.21.15.78:0/4253601430 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:00.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.453+0000 7f6042a00700 1 --2- 172.21.15.78:0/4253601430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6044101ee0 0x7f6044199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.453+0000 7f6043400700 1 --2- 172.21.15.78:0/4253601430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6044100cf0 0x7f6044199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.453+0000 7f6043e00700 1 -- 172.21.15.78:0/4253601430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6044100cf0 msgr2=0x7f6044199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.453+0000 7f6043e00700 1 --2- 172.21.15.78:0/4253601430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6044100cf0 0x7f6044199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.453+0000 7f6043e00700 1 -- 172.21.15.78:0/4253601430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6044101ee0 msgr2=0x7f6044199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.453+0000 7f6043e00700 1 --2- 172.21.15.78:0/4253601430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6044101ee0 0x7f6044199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.453+0000 7f6043e00700 1 -- 172.21.15.78:0/4253601430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60340095f0 con 0x7f6044102880 2024-03-14T09:54:00.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.453+0000 7f6043e00700 1 --2- 172.21.15.78:0/4253601430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6044102880 0x7f60441a0800 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f603c00ef30 tx=0x7f603c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:00.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.454+0000 7f603be00700 1 -- 172.21.15.78:0/4253601430 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c018070 con 0x7f6044102880 2024-03-14T09:54:00.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.454+0000 7f603be00700 1 -- 172.21.15.78:0/4253601430 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f603c00f040 con 0x7f6044102880 2024-03-14T09:54:00.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.454+0000 7f604a000700 1 -- 172.21.15.78:0/4253601430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60441a1320 con 0x7f6044102880 2024-03-14T09:54:00.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.454+0000 7f603be00700 1 -- 172.21.15.78:0/4253601430 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c00cd00 con 0x7f6044102880 2024-03-14T09:54:00.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.454+0000 7f604a000700 1 -- 172.21.15.78:0/4253601430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60441a1880 con 0x7f6044102880 2024-03-14T09:54:00.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.455+0000 7f604a000700 1 -- 172.21.15.78:0/4253601430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6044067370 con 0x7f6044102880 2024-03-14T09:54:00.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.457+0000 7f603be00700 1 -- 172.21.15.78:0/4253601430 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f603c003a00 con 0x7f6044102880 2024-03-14T09:54:00.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.458+0000 7f603be00700 1 --2- 172.21.15.78:0/4253601430 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f60280a0ba0 0x7f60280a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.458+0000 7f603be00700 1 -- 172.21.15.78:0/4253601430 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f603c0cbfe0 con 0x7f6044102880 2024-03-14T09:54:00.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.458+0000 7f6043400700 1 --2- 172.21.15.78:0/4253601430 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f60280a0ba0 0x7f60280a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.459+0000 7f6043400700 1 --2- 172.21.15.78:0/4253601430 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f60280a0ba0 0x7f60280a3060 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6034009990 tx=0x7f60340036f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:00.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.462+0000 7f603be00700 1 -- 172.21.15.78:0/4253601430 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f603c0ff960 con 0x7f6044102880 2024-03-14T09:54:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.463+0000 7f2331a00700 1 -- 172.21.15.78:0/4154769279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f232c101f40 msgr2=0x7f232c1023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.463+0000 7f2331a00700 1 --2- 172.21.15.78:0/4154769279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f232c101f40 0x7f232c1023a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f23180099c0 tx=0x7f2318009cd0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.463+0000 7f2331a00700 1 -- 172.21.15.78:0/4154769279 shutdown_connections 2024-03-14T09:54:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.463+0000 7f2331a00700 1 --2- 172.21.15.78:0/4154769279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f232c100d50 0x7f232c101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.463+0000 7f2331a00700 1 --2- 172.21.15.78:0/4154769279 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f232c1028e0 0x7f232c109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.463+0000 7f2331a00700 1 --2- 172.21.15.78:0/4154769279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f232c101f40 0x7f232c1023a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.463+0000 7f2331a00700 1 -- 172.21.15.78:0/4154769279 >> 172.21.15.78:0/4154769279 conn(0x7f232c0fc2c0 msgr2=0x7f232c0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.463+0000 7f2331a00700 1 -- 172.21.15.78:0/4154769279 shutdown_connections 2024-03-14T09:54:00.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.463+0000 7f2331a00700 1 -- 172.21.15.78:0/4154769279 wait complete. 2024-03-14T09:54:00.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.464+0000 7f2331a00700 1 Processor -- start 2024-03-14T09:54:00.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.465+0000 7f2331a00700 1 -- start start 2024-03-14T09:54:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.465+0000 7f2331a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f232c100d50 0x7f232c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.465+0000 7f2331a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f232c101f40 0x7f232c0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.465+0000 7f2331a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f232c1028e0 0x7f232c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.465+0000 7f2331a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232c073e80 con 0x7f232c100d50 2024-03-14T09:54:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.465+0000 7f2331a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232c073ff0 con 0x7f232c1028e0 2024-03-14T09:54:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.465+0000 7f2331a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232c1a1160 con 0x7f232c101f40 2024-03-14T09:54:00.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.465+0000 7f232aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f232c100d50 0x7f232c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.466+0000 7f232aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f232c100d50 0x7f232c074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44124/0 (socket says 172.21.15.78:44124) 2024-03-14T09:54:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.466+0000 7f232aa00700 1 -- 172.21.15.78:0/3612925652 learned_addr learned my addr 172.21.15.78:0/3612925652 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.466+0000 7f232a000700 1 --2- 172.21.15.78:0/3612925652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f232c101f40 0x7f232c0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.466+0000 7f232b400700 1 --2- 172.21.15.78:0/3612925652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f232c1028e0 0x7f232c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.466+0000 7f232aa00700 1 -- 172.21.15.78:0/3612925652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f232c101f40 msgr2=0x7f232c0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.466+0000 7f232aa00700 1 --2- 172.21.15.78:0/3612925652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f232c101f40 0x7f232c0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.466+0000 7f232aa00700 1 -- 172.21.15.78:0/3612925652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f232c1028e0 msgr2=0x7f232c0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.466+0000 7f232aa00700 1 --2- 172.21.15.78:0/3612925652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f232c1028e0 0x7f232c0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.466+0000 7f232aa00700 1 -- 172.21.15.78:0/3612925652 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23180095f0 con 0x7f232c100d50 2024-03-14T09:54:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.466+0000 7f232aa00700 1 --2- 172.21.15.78:0/3612925652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f232c100d50 0x7f232c074b10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f231c00d980 tx=0x7f231c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.466+0000 7f2323400700 1 -- 172.21.15.78:0/3612925652 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c004020 con 0x7f232c100d50 2024-03-14T09:54:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.467+0000 7f2331a00700 1 -- 172.21.15.78:0/3612925652 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232c1a13f0 con 0x7f232c100d50 2024-03-14T09:54:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.467+0000 7f2323400700 1 -- 172.21.15.78:0/3612925652 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f231c00f690 con 0x7f232c100d50 2024-03-14T09:54:00.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.467+0000 7f2323400700 1 -- 172.21.15.78:0/3612925652 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c003a60 con 0x7f232c100d50 2024-03-14T09:54:00.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.467+0000 7f2331a00700 1 -- 172.21.15.78:0/3612925652 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232c1a1950 con 0x7f232c100d50 2024-03-14T09:54:00.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.470+0000 7f2323400700 1 -- 172.21.15.78:0/3612925652 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7f231c00fb00 con 0x7f232c100d50 2024-03-14T09:54:00.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.471+0000 7f2345438b80 1 Processor -- start 2024-03-14T09:54:00.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.471+0000 7f2345438b80 1 -- start start 2024-03-14T09:54:00.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.472+0000 7f2345438b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150b870 0x55e15150bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.472+0000 7f2345438b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150c1d0 0x55e15150cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.472+0000 7f2345438b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150d060 0x55e15150d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.472+0000 7f2345438b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e151464a10 con 0x55e15150b870 2024-03-14T09:54:00.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.472+0000 7f2345438b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1514623d0 con 0x55e15150d060 2024-03-14T09:54:00.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.472+0000 7f2345438b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e15145dcc0 con 0x55e15150c1d0 2024-03-14T09:54:00.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.472+0000 7f232b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150b870 0x55e15150bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.472+0000 7f232b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150b870 0x55e15150bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44130/0 (socket says 172.21.15.78:44130) 2024-03-14T09:54:00.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.472+0000 7f232b400700 1 -- 172.21.15.78:0/465906495 learned_addr learned my addr 172.21.15.78:0/465906495 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:00.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.472+0000 7f232aa00700 1 --2- 172.21.15.78:0/465906495 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150c1d0 0x55e15150cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.472+0000 7f232a000700 1 --2- 172.21.15.78:0/465906495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150d060 0x55e15150d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.472+0000 7f2323400700 1 --2- 172.21.15.78:0/3612925652 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f23100a0b20 0x7f23100a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.472+0000 7f232b400700 1 -- 172.21.15.78:0/465906495 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150c1d0 msgr2=0x55e15150cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:00.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.472+0000 7f232b400700 1 --2- 172.21.15.78:0/465906495 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150c1d0 0x55e15150cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.473+0000 7f232b400700 1 -- 172.21.15.78:0/465906495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150d060 msgr2=0x55e15150d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:00.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.473+0000 7f232b400700 1 --2- 172.21.15.78:0/465906495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150d060 0x55e15150d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.473+0000 7f232b400700 1 -- 172.21.15.78:0/465906495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e1512df430 con 0x55e15150b870 2024-03-14T09:54:00.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.473+0000 7f232aa00700 1 --2- 172.21.15.78:0/465906495 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150c1d0 0x55e15150cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:00.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.473+0000 7f2323400700 1 -- 172.21.15.78:0/3612925652 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f231c0cc140 con 0x7f232c100d50 2024-03-14T09:54:00.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.473+0000 7f232b400700 1 --2- 172.21.15.78:0/3612925652 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f23100a0b20 0x7f23100a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.473+0000 7f232b400700 1 --2- 172.21.15.78:0/465906495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150b870 0x55e15150bc90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2324009a90 tx=0x7f2324003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d5acb74529a4ec server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.473+0000 7f2317e00700 1 -- 172.21.15.78:0/465906495 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2324004bb0 con 0x55e15150b870 2024-03-14T09:54:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.473+0000 7f2317e00700 1 -- 172.21.15.78:0/465906495 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f232400f040 con 0x55e15150b870 2024-03-14T09:54:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.473+0000 7f2317e00700 1 -- 172.21.15.78:0/465906495 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2324022650 con 0x55e15150b870 2024-03-14T09:54:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.474+0000 7f2345438b80 1 -- 172.21.15.78:0/465906495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150b870 msgr2=0x55e15150bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.474+0000 7f232b400700 1 --2- 172.21.15.78:0/3612925652 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f23100a0b20 0x7f23100a2fe0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2324021000 tx=0x7f232401e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.474+0000 7f2345438b80 1 --2- 172.21.15.78:0/465906495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150b870 0x55e15150bc90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2324009a90 tx=0x7f2324003f90 comp rx=0 tx=0).stop 2024-03-14T09:54:00.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.474+0000 7f2345438b80 1 -- 172.21.15.78:0/465906495 shutdown_connections 2024-03-14T09:54:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.474+0000 7f2345438b80 1 --2- 172.21.15.78:0/465906495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150d060 0x55e15150d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.474+0000 7f2345438b80 1 --2- 172.21.15.78:0/465906495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150b870 0x55e15150bc90 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.474+0000 7f2345438b80 1 --2- 172.21.15.78:0/465906495 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150c1d0 0x55e15150cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.474+0000 7f2345438b80 1 -- 172.21.15.78:0/465906495 >> 172.21.15.78:0/465906495 conn(0x55e1514812a0 msgr2=0x55e151506650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.474+0000 7f2345438b80 1 -- 172.21.15.78:0/465906495 shutdown_connections 2024-03-14T09:54:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.474+0000 7f2345438b80 1 -- 172.21.15.78:0/465906495 wait complete. 2024-03-14T09:54:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.475+0000 7f2345438b80 1 -- start start 2024-03-14T09:54:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.475+0000 7f2345438b80 1 Processor -- start 2024-03-14T09:54:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.475+0000 7f2345438b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150b870 0x55e151515770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.475+0000 7f2345438b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 0x55e151515cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.475+0000 7f2345438b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150d060 0x55e15151ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.475+0000 7f2345438b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1513f9e00 con 0x55e15150c1d0 2024-03-14T09:54:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.475+0000 7f2345438b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1513f9aa0 con 0x55e15150b870 2024-03-14T09:54:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.475+0000 7f2345438b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1512e1720 con 0x55e15150d060 2024-03-14T09:54:00.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.475+0000 7f2345438b80 10 client.0 _ll_register_callbacks cb 0x55e151265420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f232a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 0x55e151515cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f232aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150b870 0x55e151515770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f232aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150b870 0x55e151515770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:44344/0 (socket says 172.21.15.78:44344) 2024-03-14T09:54:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f232aa00700 1 -- 172.21.15.78:0/1479098972 learned_addr learned my addr 172.21.15.78:0/1479098972 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f232a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 0x55e151515cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44144/0 (socket says 172.21.15.78:44144) 2024-03-14T09:54:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f232b400700 1 --2- 172.21.15.78:0/1479098972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150d060 0x55e15151ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f232aa00700 1 -- 172.21.15.78:0/1479098972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150d060 msgr2=0x55e15151ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f232aa00700 1 --2- 172.21.15.78:0/1479098972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150d060 0x55e15151ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f232aa00700 1 -- 172.21.15.78:0/1479098972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 msgr2=0x55e151515cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f232aa00700 1 --2- 172.21.15.78:0/1479098972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 0x55e151515cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f232aa00700 1 -- 172.21.15.78:0/1479098972 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232c1a1950 con 0x55e15150b870 2024-03-14T09:54:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f232b400700 1 --2- 172.21.15.78:0/1479098972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150d060 0x55e15151ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:54:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f232aa00700 1 --2- 172.21.15.78:0/1479098972 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150b870 0x55e151515770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f231c0c9750 tx=0x7f231c098e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6a2d8635fd0a25 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f2314c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:54:00.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f2314c00700 1 -- 172.21.15.78:0/1479098972 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c09d0d0 con 0x55e15150b870 2024-03-14T09:54:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f2314c00700 1 -- 172.21.15.78:0/1479098972 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f231c0a1720 con 0x55e15150b870 2024-03-14T09:54:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.476+0000 7f2345438b80 1 -- 172.21.15.78:0/1479098972 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e1514fe7a0 con 0x55e15150b870 2024-03-14T09:54:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.477+0000 7f2314c00700 1 -- 172.21.15.78:0/1479098972 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f231c0992a0 con 0x55e15150b870 2024-03-14T09:54:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.477+0000 7f2345438b80 10 client.25514 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.477+0000 7f2345438b80 1 -- 172.21.15.78:0/1479098972 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f2324003ca0 con 0x55e15150b870 2024-03-14T09:54:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.477+0000 7f2314c00700 1 -- 172.21.15.78:0/1479098972 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e1514fe7a0 con 0x55e15150b870 2024-03-14T09:54:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.477+0000 7f2314c00700 1 -- 172.21.15.78:0/1479098972 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f231c009350 con 0x55e15150b870 2024-03-14T09:54:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.477+0000 7f2345438b80 10 client.25514 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.477+0000 7f2345438b80 10 client.25514 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.477+0000 7f2345438b80 20 client.25514 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.477+0000 7f2345438b80 1 --2- 172.21.15.78:0/1479098972 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e151521500 0x55e1515238f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.477+0000 7f2345438b80 4 client.25514 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:54:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.477+0000 7f2345438b80 1 -- 172.21.15.78:0/1479098972 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e151456780 con 0x55e151521500 2024-03-14T09:54:00.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.478+0000 7f232a000700 1 --2- 172.21.15.78:0/1479098972 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e151521500 0x55e1515238f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.478+0000 7f2314c00700 10 client.25514 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:00.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.478+0000 7f232a000700 1 --2- 172.21.15.78:0/1479098972 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e151521500 0x55e1515238f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e72b0bc4d97ab38e server_cookie=761c38e79548aad0 in_seq=0 out_seq=0 2024-03-14T09:54:00.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.479+0000 7f2314c00700 1 -- 172.21.15.78:0/1479098972 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55e151456780 con 0x55e151521500 2024-03-14T09:54:00.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.479+0000 7f2314c00700 10 client.25514 handle_command_reply: tid=0 2024-03-14T09:54:00.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.479+0000 7f2345438b80 1 client.25514 shutdown 2024-03-14T09:54:00.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.479+0000 7f2345438b80 1 -- 172.21.15.78:0/1479098972 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150b870 msgr2=0x55e151515770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.479+0000 7f2345438b80 1 --2- 172.21.15.78:0/1479098972 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150b870 0x55e151515770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f231c0c9750 tx=0x7f231c098e60 comp rx=0 tx=0).stop 2024-03-14T09:54:00.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.479+0000 7f2345438b80 1 -- 172.21.15.78:0/1479098972 shutdown_connections 2024-03-14T09:54:00.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.479+0000 7f2345438b80 1 --2- 172.21.15.78:0/1479098972 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e151521500 0x55e1515238f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.479+0000 7f2345438b80 1 --2- 172.21.15.78:0/1479098972 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150b870 0x55e151515770 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.479+0000 7f2345438b80 1 --2- 172.21.15.78:0/1479098972 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150d060 0x55e15151ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.479+0000 7f2345438b80 1 --2- 172.21.15.78:0/1479098972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 0x55e151515cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.479+0000 7f2314c00700 0 client.25514 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:00.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.479+0000 7f2345438b80 1 -- 172.21.15.78:0/1479098972 >> 172.21.15.78:0/1479098972 conn(0x55e1514812a0 msgr2=0x55e151505ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:00.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.479+0000 7f2345438b80 1 -- 172.21.15.78:0/1479098972 shutdown_connections 2024-03-14T09:54:00.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.479+0000 7f2345438b80 1 -- 172.21.15.78:0/1479098972 wait complete. 2024-03-14T09:54:00.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.480+0000 7f2345438b80 20 client.25514 trim_cache size 0 max 16384 2024-03-14T09:54:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.497+0000 7f2345438b80 1 Processor -- start 2024-03-14T09:54:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.497+0000 7f2345438b80 1 -- start start 2024-03-14T09:54:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.497+0000 7f2345438b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150b870 0x55e151514c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.497+0000 7f2345438b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 0x55e15151ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.497+0000 7f2345438b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150d060 0x55e151514450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.497+0000 7f2345438b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e151500d80 con 0x55e15150c1d0 2024-03-14T09:54:00.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.497+0000 7f2345438b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e151516960 con 0x55e15150d060 2024-03-14T09:54:00.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.497+0000 7f2345438b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e151513210 con 0x55e15150b870 2024-03-14T09:54:00.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.497+0000 7f232a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 0x55e15151ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.497+0000 7f232a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 0x55e15151ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44160/0 (socket says 172.21.15.78:44160) 2024-03-14T09:54:00.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.497+0000 7f232a000700 1 -- 172.21.15.78:0/2114721402 learned_addr learned my addr 172.21.15.78:0/2114721402 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:00.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.497+0000 7f232b400700 1 --2- 172.21.15.78:0/2114721402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150d060 0x55e151514450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.497+0000 7f232aa00700 1 --2- 172.21.15.78:0/2114721402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150b870 0x55e151514c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.498+0000 7f232b400700 1 -- 172.21.15.78:0/2114721402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150b870 msgr2=0x55e151514c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:00.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.498+0000 7f232b400700 1 --2- 172.21.15.78:0/2114721402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150b870 0x55e151514c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.498+0000 7f232b400700 1 -- 172.21.15.78:0/2114721402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 msgr2=0x55e15151ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:00.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.498+0000 7f232b400700 1 --2- 172.21.15.78:0/2114721402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 0x55e15151ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.498+0000 7f232b400700 1 -- 172.21.15.78:0/2114721402 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e1515029d0 con 0x55e15150d060 2024-03-14T09:54:00.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.498+0000 7f232a000700 1 --2- 172.21.15.78:0/2114721402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 0x55e15151ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:00.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.498+0000 7f232b400700 1 --2- 172.21.15.78:0/2114721402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150d060 0x55e151514450 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2324000f80 tx=0x7f23240094e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8af1f6c1fb38f425 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:00.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.498+0000 7f2320c00700 1 -- 172.21.15.78:0/2114721402 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2324007eb0 con 0x55e15150d060 2024-03-14T09:54:00.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.498+0000 7f2320c00700 1 -- 172.21.15.78:0/2114721402 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f23240223f0 con 0x55e15150d060 2024-03-14T09:54:00.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.498+0000 7f2320c00700 1 -- 172.21.15.78:0/2114721402 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2324022e30 con 0x55e15150d060 2024-03-14T09:54:00.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:00 smithi152 ceph-mon[48279]: pgmap v296: 129 pgs: 129 active+clean; 649 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-03-14T09:54:00.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:00 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2292744312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:54:00.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:00 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3176073465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:54:00.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:00 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3327204701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:54:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.499+0000 7f2345438b80 1 -- 172.21.15.78:0/2114721402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150d060 msgr2=0x55e151514450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.499+0000 7f2345438b80 1 --2- 172.21.15.78:0/2114721402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150d060 0x55e151514450 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2324000f80 tx=0x7f23240094e0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.499+0000 7f2345438b80 1 -- 172.21.15.78:0/2114721402 shutdown_connections 2024-03-14T09:54:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.499+0000 7f2345438b80 1 --2- 172.21.15.78:0/2114721402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150d060 0x55e151514450 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.499+0000 7f2345438b80 1 --2- 172.21.15.78:0/2114721402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150b870 0x55e151514c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.499+0000 7f2345438b80 1 --2- 172.21.15.78:0/2114721402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 0x55e15151ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.499+0000 7f2345438b80 1 -- 172.21.15.78:0/2114721402 >> 172.21.15.78:0/2114721402 conn(0x55e1514812a0 msgr2=0x55e151513590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.499+0000 7f2345438b80 1 -- 172.21.15.78:0/2114721402 shutdown_connections 2024-03-14T09:54:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.499+0000 7f2345438b80 1 -- 172.21.15.78:0/2114721402 wait complete. 2024-03-14T09:54:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.500+0000 7f2345438b80 1 -- start start 2024-03-14T09:54:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.500+0000 7f2345438b80 1 Processor -- start 2024-03-14T09:54:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.500+0000 7f2345438b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150b870 0x55e151519980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.500+0000 7f2345438b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 0x55e151519ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.500+0000 7f2345438b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150d060 0x55e151572fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.500+0000 7f2345438b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e15151cda0 con 0x55e15150c1d0 2024-03-14T09:54:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.500+0000 7f2345438b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e15151cf10 con 0x55e15150b870 2024-03-14T09:54:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.500+0000 7f2345438b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e151573520 con 0x55e15150d060 2024-03-14T09:54:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.500+0000 7f2345438b80 10 client.0 _ll_register_callbacks cb 0x7f231c00ffa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.501+0000 7f232a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 0x55e151519ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.501+0000 7f232aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150b870 0x55e151519980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.501+0000 7f232a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 0x55e151519ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44168/0 (socket says 172.21.15.78:44168) 2024-03-14T09:54:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.501+0000 7f232a000700 1 -- 172.21.15.78:0/4293429354 learned_addr learned my addr 172.21.15.78:0/4293429354 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.501+0000 7f232b400700 1 --2- 172.21.15.78:0/4293429354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150d060 0x55e151572fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.501+0000 7f232aa00700 1 -- 172.21.15.78:0/4293429354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150d060 msgr2=0x55e151572fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.501+0000 7f232aa00700 1 --2- 172.21.15.78:0/4293429354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150d060 0x55e151572fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.501+0000 7f232aa00700 1 -- 172.21.15.78:0/4293429354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 msgr2=0x55e151519ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.501+0000 7f232aa00700 1 --2- 172.21.15.78:0/4293429354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 0x55e151519ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.501+0000 7f232aa00700 1 -- 172.21.15.78:0/4293429354 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232c109ae0 con 0x55e15150b870 2024-03-14T09:54:00.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.501+0000 7f232b400700 1 --2- 172.21.15.78:0/4293429354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150d060 0x55e151572fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:54:00.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.501+0000 7f232aa00700 1 --2- 172.21.15.78:0/4293429354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150b870 0x55e151519980 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x55e151516910 tx=0x7f231c0c9c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef46e77ff2f93293 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:00.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.501+0000 7f2314c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:54:00.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.502+0000 7f2314c00700 1 -- 172.21.15.78:0/4293429354 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c0b5070 con 0x55e15150b870 2024-03-14T09:54:00.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.502+0000 7f2314c00700 1 -- 172.21.15.78:0/4293429354 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f231c0094b0 con 0x55e15150b870 2024-03-14T09:54:00.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.502+0000 7f2345438b80 1 -- 172.21.15.78:0/4293429354 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f232400f040 con 0x55e15150b870 2024-03-14T09:54:00.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.502+0000 7f2314c00700 1 -- 172.21.15.78:0/4293429354 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f231c0989d0 con 0x55e15150b870 2024-03-14T09:54:00.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.502+0000 7f2345438b80 10 client.25520 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:00.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.502+0000 7f2345438b80 1 -- 172.21.15.78:0/4293429354 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f2324019e70 con 0x55e15150b870 2024-03-14T09:54:00.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.502+0000 7f2314c00700 1 -- 172.21.15.78:0/4293429354 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f232400f040 con 0x55e15150b870 2024-03-14T09:54:00.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.502+0000 7f2314c00700 1 -- 172.21.15.78:0/4293429354 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f231c0ce0d0 con 0x55e15150b870 2024-03-14T09:54:00.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.502+0000 7f2345438b80 10 client.25520 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:00.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.502+0000 7f2345438b80 10 client.25520 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:00.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.502+0000 7f2345438b80 20 client.25520 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.502+0000 7f2345438b80 1 --2- 172.21.15.78:0/4293429354 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e151574b40 0x55e151576f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.502+0000 7f2345438b80 4 client.25520 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T09:54:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.502+0000 7f2345438b80 1 -- 172.21.15.78:0/4293429354 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e1512acd70 con 0x55e151574b40 2024-03-14T09:54:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.503+0000 7f232a000700 1 --2- 172.21.15.78:0/4293429354 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e151574b40 0x55e151576f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.503+0000 7f2314c00700 10 client.25520 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:00.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.503+0000 7f232a000700 1 --2- 172.21.15.78:0/4293429354 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e151574b40 0x55e151576f30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ed862fd2536c442 server_cookie=5cae1cdee31bdb20 in_seq=0 out_seq=0 2024-03-14T09:54:00.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.505+0000 7f2314c00700 1 -- 172.21.15.78:0/4293429354 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e1512acd70 con 0x55e151574b40 2024-03-14T09:54:00.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.505+0000 7f2314c00700 10 client.25520 handle_command_reply: tid=0 2024-03-14T09:54:00.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.505+0000 7f2345438b80 1 client.25520 shutdown 2024-03-14T09:54:00.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.505+0000 7f2345438b80 1 -- 172.21.15.78:0/4293429354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150b870 msgr2=0x55e151519980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.505+0000 7f2345438b80 1 --2- 172.21.15.78:0/4293429354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150b870 0x55e151519980 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x55e151516910 tx=0x7f231c0c9c20 comp rx=0 tx=0).stop 2024-03-14T09:54:00.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.505+0000 7f2345438b80 1 -- 172.21.15.78:0/4293429354 shutdown_connections 2024-03-14T09:54:00.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.505+0000 7f2345438b80 1 --2- 172.21.15.78:0/4293429354 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e151574b40 0x55e151576f30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.505+0000 7f2345438b80 1 --2- 172.21.15.78:0/4293429354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e15150b870 0x55e151519980 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.505+0000 7f2345438b80 1 --2- 172.21.15.78:0/4293429354 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e15150d060 0x55e151572fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.505+0000 7f2345438b80 1 --2- 172.21.15.78:0/4293429354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e15150c1d0 0x55e151519ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.505+0000 7f2314c00700 0 client.25520 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:00.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.505+0000 7f2345438b80 1 -- 172.21.15.78:0/4293429354 >> 172.21.15.78:0/4293429354 conn(0x55e1514812a0 msgr2=0x55e15150ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:00.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.505+0000 7f2345438b80 1 -- 172.21.15.78:0/4293429354 shutdown_connections 2024-03-14T09:54:00.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.505+0000 7f2345438b80 1 -- 172.21.15.78:0/4293429354 wait complete. 2024-03-14T09:54:00.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.506+0000 7f2345438b80 20 client.25520 trim_cache size 0 max 16384 2024-03-14T09:54:00.509 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:54:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T09:54:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "c784aa4a-0c43-472b-809c-8c5888d8dfcb", 2024-03-14T09:54:00.509 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T09:54:00.509 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:54:00.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.506+0000 7f2331a00700 1 -- 172.21.15.78:0/3612925652 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f23100a0b20 msgr2=0x7f23100a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.506+0000 7f2331a00700 1 --2- 172.21.15.78:0/3612925652 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f23100a0b20 0x7f23100a2fe0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2324021000 tx=0x7f232401e000 comp rx=0 tx=0).stop 2024-03-14T09:54:00.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.506+0000 7f2331a00700 1 -- 172.21.15.78:0/3612925652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f232c100d50 msgr2=0x7f232c074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.506+0000 7f2331a00700 1 --2- 172.21.15.78:0/3612925652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f232c100d50 0x7f232c074b10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f231c00d980 tx=0x7f231c00dc90 comp rx=0 tx=0).stop 2024-03-14T09:54:00.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.506+0000 7f2331a00700 1 -- 172.21.15.78:0/3612925652 shutdown_connections 2024-03-14T09:54:00.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.506+0000 7f2331a00700 1 --2- 172.21.15.78:0/3612925652 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f23100a0b20 0x7f23100a2fe0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.506+0000 7f2331a00700 1 --2- 172.21.15.78:0/3612925652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f232c1028e0 0x7f232c0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.506+0000 7f2331a00700 1 --2- 172.21.15.78:0/3612925652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f232c100d50 0x7f232c074b10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.506+0000 7f2331a00700 1 --2- 172.21.15.78:0/3612925652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f232c101f40 0x7f232c0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.506+0000 7f2331a00700 1 -- 172.21.15.78:0/3612925652 >> 172.21.15.78:0/3612925652 conn(0x7f232c0fc2c0 msgr2=0x7f232c105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:00.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.506+0000 7f2331a00700 1 -- 172.21.15.78:0/3612925652 shutdown_connections 2024-03-14T09:54:00.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.506+0000 7f2331a00700 1 -- 172.21.15.78:0/3612925652 wait complete. 2024-03-14T09:54:00.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:00 smithi132 ceph-mon[47959]: pgmap v296: 129 pgs: 129 active+clean; 649 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-03-14T09:54:00.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:00 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2292744312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:54:00.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:00 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3176073465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:54:00.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:00 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3327204701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:54:00.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:00 smithi078 ceph-mon[38143]: pgmap v296: 129 pgs: 129 active+clean; 649 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-03-14T09:54:00.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:00 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2292744312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T09:54:00.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:00 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3176073465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:54:00.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:00 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3327204701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T09:54:00.525 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:54:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.722+0000 7efcd5000700 1 -- 172.21.15.78:0/51971547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0073810 msgr2=0x7efcd0073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.722+0000 7efcd5000700 1 --2- 172.21.15.78:0/51971547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0073810 0x7efcd0073c90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efcbc009990 tx=0x7efcbc009ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.722+0000 7efcd5000700 1 -- 172.21.15.78:0/51971547 shutdown_connections 2024-03-14T09:54:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.722+0000 7efcd5000700 1 --2- 172.21.15.78:0/51971547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcd0108c00 0x7efcd010b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.722+0000 7efcd5000700 1 --2- 172.21.15.78:0/51971547 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcd0074df0 0x7efcd0073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.722+0000 7efcd5000700 1 --2- 172.21.15.78:0/51971547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0073810 0x7efcd0073c90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.722+0000 7efcd5000700 1 -- 172.21.15.78:0/51971547 >> 172.21.15.78:0/51971547 conn(0x7efcd00fc2f0 msgr2=0x7efcd00fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.722+0000 7efcd5000700 1 -- 172.21.15.78:0/51971547 shutdown_connections 2024-03-14T09:54:00.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.722+0000 7efcd5000700 1 -- 172.21.15.78:0/51971547 wait complete. 2024-03-14T09:54:00.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.723+0000 7efcd5000700 1 Processor -- start 2024-03-14T09:54:00.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.723+0000 7efcd5000700 1 -- start start 2024-03-14T09:54:00.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.723+0000 7efcd5000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcd0073810 0x7efcd0199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.723+0000 7efcd5000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0074df0 0x7efcd0199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.723+0000 7efcd5000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcd0108c00 0x7efcd01a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.723+0000 7efcd5000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd01a0e70 con 0x7efcd0073810 2024-03-14T09:54:00.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.723+0000 7efcd5000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd01a0fb0 con 0x7efcd0074df0 2024-03-14T09:54:00.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.723+0000 7efcd5000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd01a1140 con 0x7efcd0108c00 2024-03-14T09:54:00.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.724+0000 7efccde00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcd0073810 0x7efcd0199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.724+0000 7efccd400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0074df0 0x7efcd0199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.724+0000 7efccde00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcd0073810 0x7efcd0199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44188/0 (socket says 172.21.15.78:44188) 2024-03-14T09:54:00.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.724+0000 7efccde00700 1 -- 172.21.15.78:0/4160609570 learned_addr learned my addr 172.21.15.78:0/4160609570 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:00.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.724+0000 7efcce800700 1 --2- 172.21.15.78:0/4160609570 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcd0108c00 0x7efcd01a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.724+0000 7efccd400700 1 -- 172.21.15.78:0/4160609570 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcd0108c00 msgr2=0x7efcd01a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.724+0000 7efccd400700 1 --2- 172.21.15.78:0/4160609570 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcd0108c00 0x7efcd01a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.724+0000 7efccd400700 1 -- 172.21.15.78:0/4160609570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcd0073810 msgr2=0x7efcd0199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.724+0000 7efccd400700 1 --2- 172.21.15.78:0/4160609570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcd0073810 0x7efcd0199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.724+0000 7efccd400700 1 -- 172.21.15.78:0/4160609570 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc0095f0 con 0x7efcd0074df0 2024-03-14T09:54:00.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.724+0000 7efccde00700 1 --2- 172.21.15.78:0/4160609570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcd0073810 0x7efcd0199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:00.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.724+0000 7efccd400700 1 --2- 172.21.15.78:0/4160609570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0074df0 0x7efcd0199880 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efcbc00b680 tx=0x7efcbc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:00.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.725+0000 7efcc6a00700 1 -- 172.21.15.78:0/4160609570 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc01d070 con 0x7efcd0074df0 2024-03-14T09:54:00.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.725+0000 7efcc6a00700 1 -- 172.21.15.78:0/4160609570 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcbc021410 con 0x7efcd0074df0 2024-03-14T09:54:00.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.725+0000 7efcd5000700 1 -- 172.21.15.78:0/4160609570 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd01a13c0 con 0x7efcd0074df0 2024-03-14T09:54:00.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.725+0000 7efcc6a00700 1 -- 172.21.15.78:0/4160609570 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc017650 con 0x7efcd0074df0 2024-03-14T09:54:00.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.725+0000 7efcd5000700 1 -- 172.21.15.78:0/4160609570 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd01a17e0 con 0x7efcd0074df0 2024-03-14T09:54:00.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.728+0000 7efcc6a00700 1 -- 172.21.15.78:0/4160609570 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) v1 ==== 135439+0+0 (secure 0 0 0) 0x7efcbc021580 con 0x7efcd0074df0 2024-03-14T09:54:00.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.729+0000 7efce8b52b80 1 Processor -- start 2024-03-14T09:54:00.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.729+0000 7efcc6a00700 1 --2- 172.21.15.78:0/4160609570 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7efcb40a0b20 0x7efcb40a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.729+0000 7efce8b52b80 1 -- start start 2024-03-14T09:54:00.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.729+0000 7efcc6a00700 1 -- 172.21.15.78:0/4160609570 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7efcbc0cd9b0 con 0x7efcd0074df0 2024-03-14T09:54:00.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.730+0000 7efcce800700 1 --2- 172.21.15.78:0/4160609570 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7efcb40a0b20 0x7efcb40a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.730+0000 7efce8b52b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad87f0 0x55ed11ad8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.730+0000 7efce8b52b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 0x55ed11ad9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.730+0000 7efce8b52b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad9fe0 0x55ed11ada460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.730+0000 7efce8b52b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed11a31990 con 0x55ed11ad87f0 2024-03-14T09:54:00.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.730+0000 7efce8b52b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed11a2f350 con 0x55ed11ad9150 2024-03-14T09:54:00.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.730+0000 7efce8b52b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed11a2ac40 con 0x55ed11ad9fe0 2024-03-14T09:54:00.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.730+0000 7efccd400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 0x55ed11ad9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.730+0000 7efcce800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad9fe0 0x55ed11ada460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.730+0000 7efccd400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 0x55ed11ad9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:44390/0 (socket says 172.21.15.78:44390) 2024-03-14T09:54:00.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.730+0000 7efccd400700 1 -- 172.21.15.78:0/3838894696 learned_addr learned my addr 172.21.15.78:0/3838894696 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:00.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.730+0000 7efcce800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad9fe0 0x55ed11ada460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35702/0 (socket says 172.21.15.78:35702) 2024-03-14T09:54:00.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.730+0000 7efccde00700 1 --2- 172.21.15.78:0/3838894696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad87f0 0x55ed11ad8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.730+0000 7efcce800700 1 --2- 172.21.15.78:0/4160609570 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7efcb40a0b20 0x7efcb40a2fe0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efcc800f540 tx=0x7efcc8009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:00.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.730+0000 7efccd400700 1 -- 172.21.15.78:0/3838894696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad9fe0 msgr2=0x55ed11ada460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:00.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.731+0000 7efccd400700 1 --2- 172.21.15.78:0/3838894696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad9fe0 0x55ed11ada460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.731+0000 7efccd400700 1 -- 172.21.15.78:0/3838894696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad87f0 msgr2=0x55ed11ad8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:00.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.731+0000 7efccd400700 1 --2- 172.21.15.78:0/3838894696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad87f0 0x55ed11ad8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.731+0000 7efccd400700 1 -- 172.21.15.78:0/3838894696 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed118ac3b0 con 0x55ed11ad9150 2024-03-14T09:54:00.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.731+0000 7efccd400700 1 --2- 172.21.15.78:0/3838894696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 0x55ed11ad9a50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efcbc0a2a60 tx=0x7efcbc09aa80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75fb2c3d223fc1e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:00.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.731+0000 7efcbb400700 1 -- 172.21.15.78:0/3838894696 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc09aba0 con 0x55ed11ad9150 2024-03-14T09:54:00.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.731+0000 7efcbb400700 1 -- 172.21.15.78:0/3838894696 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcbc09e9d0 con 0x55ed11ad9150 2024-03-14T09:54:00.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.731+0000 7efcbb400700 1 -- 172.21.15.78:0/3838894696 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc09a090 con 0x55ed11ad9150 2024-03-14T09:54:00.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.732+0000 7efce8b52b80 1 -- 172.21.15.78:0/3838894696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 msgr2=0x55ed11ad9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.732+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3838894696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 0x55ed11ad9a50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efcbc0a2a60 tx=0x7efcbc09aa80 comp rx=0 tx=0).stop 2024-03-14T09:54:00.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.732+0000 7efce8b52b80 1 -- 172.21.15.78:0/3838894696 shutdown_connections 2024-03-14T09:54:00.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.732+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3838894696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad9fe0 0x55ed11ada460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.732+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3838894696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad87f0 0x55ed11ad8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.732+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3838894696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 0x55ed11ad9a50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.732+0000 7efce8b52b80 1 -- 172.21.15.78:0/3838894696 >> 172.21.15.78:0/3838894696 conn(0x55ed11a4e220 msgr2=0x55ed11ad35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:00.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.732+0000 7efce8b52b80 1 -- 172.21.15.78:0/3838894696 shutdown_connections 2024-03-14T09:54:00.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.732+0000 7efce8b52b80 1 -- 172.21.15.78:0/3838894696 wait complete. 2024-03-14T09:54:00.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.733+0000 7efce8b52b80 1 -- start start 2024-03-14T09:54:00.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.733+0000 7efce8b52b80 1 Processor -- start 2024-03-14T09:54:00.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.734+0000 7efce8b52b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 0x55ed11ae2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.734+0000 7efce8b52b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 0x55ed11ae28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.734+0000 7efce8b52b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.734+0000 7efce8b52b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed119c6d80 con 0x55ed11ad9fe0 2024-03-14T09:54:00.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.734+0000 7efce8b52b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed119c6a20 con 0x55ed11ad9150 2024-03-14T09:54:00.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.734+0000 7efce8b52b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed118ae6a0 con 0x55ed11ad87f0 2024-03-14T09:54:00.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.734+0000 7efce8b52b80 10 client.0 _ll_register_callbacks cb 0x55ed118323a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:00.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.734+0000 7efcce800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.734+0000 7efccde00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 0x55ed11ae2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.734+0000 7efccde00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 0x55ed11ae2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35708/0 (socket says 172.21.15.78:35708) 2024-03-14T09:54:00.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.734+0000 7efccde00700 1 -- 172.21.15.78:0/3244723978 learned_addr learned my addr 172.21.15.78:0/3244723978 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:00.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.734+0000 7efcce800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44208/0 (socket says 172.21.15.78:44208) 2024-03-14T09:54:00.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.734+0000 7efccd400700 1 --2- 172.21.15.78:0/3244723978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 0x55ed11ae28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.735+0000 7efccde00700 1 -- 172.21.15.78:0/3244723978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 msgr2=0x55ed11ae28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:00.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.735+0000 7efccde00700 1 --2- 172.21.15.78:0/3244723978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 0x55ed11ae28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.735+0000 7efccde00700 1 -- 172.21.15.78:0/3244723978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9fe0 msgr2=0x55ed11ae9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:00.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.735+0000 7efccde00700 1 --2- 172.21.15.78:0/3244723978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.735+0000 7efccde00700 1 -- 172.21.15.78:0/3244723978 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd01a17e0 con 0x55ed11ad87f0 2024-03-14T09:54:00.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.735+0000 7efcce800700 1 --2- 172.21.15.78:0/3244723978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.735+0000 7efccde00700 1 --2- 172.21.15.78:0/3244723978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 0x55ed11ae2390 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efcc0011970 tx=0x7efcc0011c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3b6c28858638f06 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.735+0000 7efcafe00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:54:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.735+0000 7efcafe00700 1 -- 172.21.15.78:0/3244723978 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc0016070 con 0x55ed11ad87f0 2024-03-14T09:54:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.735+0000 7efce8b52b80 1 -- 172.21.15.78:0/3244723978 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ed11acb720 con 0x55ed11ad87f0 2024-03-14T09:54:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.735+0000 7efcafe00700 1 -- 172.21.15.78:0/3244723978 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7efcc0004260 con 0x55ed11ad87f0 2024-03-14T09:54:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.735+0000 7efcafe00700 1 -- 172.21.15.78:0/3244723978 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7efcc0004420 con 0x55ed11ad87f0 2024-03-14T09:54:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.735+0000 7efce8b52b80 10 client.25531 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.735+0000 7efce8b52b80 1 -- 172.21.15.78:0/3244723978 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ed11858b60 con 0x55ed11ad87f0 2024-03-14T09:54:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.736+0000 7efcafe00700 1 -- 172.21.15.78:0/3244723978 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ed11acb720 con 0x55ed11ad87f0 2024-03-14T09:54:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.736+0000 7efcafe00700 1 -- 172.21.15.78:0/3244723978 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7efcc0003be0 con 0x55ed11ad87f0 2024-03-14T09:54:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.736+0000 7efce8b52b80 10 client.25531 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.736+0000 7efce8b52b80 10 client.25531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.736+0000 7efce8b52b80 20 client.25531 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.736+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3244723978 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55ed11aee150 0x55ed11af0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.736+0000 7efce8b52b80 4 client.25531 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:54:00.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.736+0000 7efce8b52b80 1 -- 172.21.15.78:0/3244723978 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed11a23700 con 0x55ed11aee150 2024-03-14T09:54:00.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.736+0000 7efccd400700 1 --2- 172.21.15.78:0/3244723978 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55ed11aee150 0x55ed11af0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.737+0000 7efcafe00700 10 client.25531 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:00.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.737+0000 7efccd400700 1 --2- 172.21.15.78:0/3244723978 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55ed11aee150 0x55ed11af0570 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b309f8c9db3513e server_cookie=d5dc3a700393a03b in_seq=0 out_seq=0 2024-03-14T09:54:00.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.737+0000 7efcafe00700 1 -- 172.21.15.78:0/3244723978 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55ed11a23700 con 0x55ed11aee150 2024-03-14T09:54:00.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.738+0000 7efcafe00700 10 client.25531 handle_command_reply: tid=0 2024-03-14T09:54:00.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.738+0000 7efce8b52b80 1 client.25531 shutdown 2024-03-14T09:54:00.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.738+0000 7efce8b52b80 1 -- 172.21.15.78:0/3244723978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 msgr2=0x55ed11ae2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.738+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3244723978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 0x55ed11ae2390 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efcc0011970 tx=0x7efcc0011c80 comp rx=0 tx=0).stop 2024-03-14T09:54:00.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.738+0000 7efce8b52b80 1 -- 172.21.15.78:0/3244723978 shutdown_connections 2024-03-14T09:54:00.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.738+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3244723978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.738+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3244723978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 0x55ed11ae2390 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.738+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3244723978 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55ed11aee150 0x55ed11af0570 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.738+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3244723978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 0x55ed11ae28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.738+0000 7efcafe00700 0 client.25531 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:00.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.738+0000 7efce8b52b80 1 -- 172.21.15.78:0/3244723978 >> 172.21.15.78:0/3244723978 conn(0x55ed11a4e220 msgr2=0x55ed11ad2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:00.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.738+0000 7efce8b52b80 1 -- 172.21.15.78:0/3244723978 shutdown_connections 2024-03-14T09:54:00.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.738+0000 7efce8b52b80 1 -- 172.21.15.78:0/3244723978 wait complete. 2024-03-14T09:54:00.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.738+0000 7efce8b52b80 20 client.25531 trim_cache size 0 max 16384 2024-03-14T09:54:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.752+0000 7f604a000700 1 -- 172.21.15.78:0/4253601430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f604410cb40 con 0x7f6044102880 2024-03-14T09:54:00.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.755+0000 7efce8b52b80 1 Processor -- start 2024-03-14T09:54:00.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.755+0000 7efce8b52b80 1 -- start start 2024-03-14T09:54:00.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.755+0000 7efce8b52b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 0x55ed11ae7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.755+0000 7efce8b52b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9150 0x55ed11ae1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.755+0000 7efce8b52b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.755+0000 7efce8b52b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed11aeaa90 con 0x55ed11ad9150 2024-03-14T09:54:00.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.755+0000 7efce8b52b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed11ad9550 con 0x55ed11ad9fe0 2024-03-14T09:54:00.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.755+0000 7efce8b52b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed11acdd00 con 0x55ed11ad87f0 2024-03-14T09:54:00.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.756+0000 7efccd400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9150 0x55ed11ae1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.756+0000 7efccd400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9150 0x55ed11ae1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44218/0 (socket says 172.21.15.78:44218) 2024-03-14T09:54:00.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.756+0000 7efccd400700 1 -- 172.21.15.78:0/3738671865 learned_addr learned my addr 172.21.15.78:0/3738671865 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:00.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.756+0000 7efccde00700 1 --2- 172.21.15.78:0/3738671865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 0x55ed11ae7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.756+0000 7efcce800700 1 --2- 172.21.15.78:0/3738671865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.756+0000 7efccd400700 1 -- 172.21.15.78:0/3738671865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 msgr2=0x55ed11ae7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:00.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.756+0000 7efccd400700 1 --2- 172.21.15.78:0/3738671865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 0x55ed11ae7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.756+0000 7efccd400700 1 -- 172.21.15.78:0/3738671865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9fe0 msgr2=0x55ed11ae2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:00.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.756+0000 7efccd400700 1 --2- 172.21.15.78:0/3738671865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.756+0000 7efccd400700 1 -- 172.21.15.78:0/3738671865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed119f0ea0 con 0x55ed11ad9150 2024-03-14T09:54:00.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.756+0000 7efcce800700 1 --2- 172.21.15.78:0/3738671865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:54:00.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.756+0000 7efccd400700 1 --2- 172.21.15.78:0/3738671865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9150 0x55ed11ae1070 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7efcbc09a850 tx=0x7efcbc009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e93f3f2ac9b5544 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:00.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.756+0000 7efcbbe00700 1 -- 172.21.15.78:0/3738671865 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc09aba0 con 0x55ed11ad9150 2024-03-14T09:54:00.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.756+0000 7efcbbe00700 1 -- 172.21.15.78:0/3738671865 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcbc0cb1e0 con 0x55ed11ad9150 2024-03-14T09:54:00.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.756+0000 7efcbbe00700 1 -- 172.21.15.78:0/3738671865 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc09a2f0 con 0x55ed11ad9150 2024-03-14T09:54:00.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.757+0000 7efce8b52b80 1 -- 172.21.15.78:0/3738671865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9150 msgr2=0x55ed11ae1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.757+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3738671865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9150 0x55ed11ae1070 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7efcbc09a850 tx=0x7efcbc009ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.757+0000 7efce8b52b80 1 -- 172.21.15.78:0/3738671865 shutdown_connections 2024-03-14T09:54:00.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.757+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3738671865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.757+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3738671865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 0x55ed11ae7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.757+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3738671865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9150 0x55ed11ae1070 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.757+0000 7efce8b52b80 1 -- 172.21.15.78:0/3738671865 >> 172.21.15.78:0/3738671865 conn(0x55ed11a4e220 msgr2=0x55ed11ae18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:00.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.757+0000 7efce8b52b80 1 -- 172.21.15.78:0/3738671865 shutdown_connections 2024-03-14T09:54:00.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.757+0000 7efce8b52b80 1 -- 172.21.15.78:0/3738671865 wait complete. 2024-03-14T09:54:00.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.758+0000 7efce8b52b80 1 -- start start 2024-03-14T09:54:00.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.758+0000 7efce8b52b80 1 Processor -- start 2024-03-14T09:54:00.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.758+0000 7efce8b52b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 0x55ed11ae71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.758+0000 7efce8b52b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 0x55ed11ae7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.758+0000 7efce8b52b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.758+0000 7efce8b52b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed11ae5f30 con 0x55ed11ad9fe0 2024-03-14T09:54:00.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.758+0000 7efce8b52b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed11ae60a0 con 0x55ed11ad9150 2024-03-14T09:54:00.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.758+0000 7efce8b52b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed11ae6260 con 0x55ed11ad87f0 2024-03-14T09:54:00.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.758+0000 7efce8b52b80 10 client.0 _ll_register_callbacks cb 0x7efcc0011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:00.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.758+0000 7efcce800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.758+0000 7efcce800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44232/0 (socket says 172.21.15.78:44232) 2024-03-14T09:54:00.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.758+0000 7efcce800700 1 -- 172.21.15.78:0/3390398981 learned_addr learned my addr 172.21.15.78:0/3390398981 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:00.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.758+0000 7efccde00700 1 --2- 172.21.15.78:0/3390398981 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 0x55ed11ae71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.759+0000 7efccd400700 1 --2- 172.21.15.78:0/3390398981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 0x55ed11ae7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.759+0000 7efcce800700 1 -- 172.21.15.78:0/3390398981 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 msgr2=0x55ed11ae71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:00.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.759+0000 7efcce800700 1 --2- 172.21.15.78:0/3390398981 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 0x55ed11ae71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.759+0000 7efcce800700 1 -- 172.21.15.78:0/3390398981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 msgr2=0x55ed11ae7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:00.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.759+0000 7efcce800700 1 --2- 172.21.15.78:0/3390398981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 0x55ed11ae7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.759+0000 7efcce800700 1 -- 172.21.15.78:0/3390398981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed118ac3b0 con 0x55ed11ad9fe0 2024-03-14T09:54:00.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.759+0000 7efcce800700 1 --2- 172.21.15.78:0/3390398981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae5820 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efcc8019450 tx=0x7efcc801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5995d93793799308 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:00.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.759+0000 7efcafe00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:54:00.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.759+0000 7efcafe00700 1 -- 172.21.15.78:0/3390398981 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc800e040 con 0x55ed11ad9fe0 2024-03-14T09:54:00.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.759+0000 7efcafe00700 1 -- 172.21.15.78:0/3390398981 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7efcc8018af0 con 0x55ed11ad9fe0 2024-03-14T09:54:00.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.759+0000 7efce8b52b80 1 -- 172.21.15.78:0/3390398981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efcbc09e9d0 con 0x55ed11ad9fe0 2024-03-14T09:54:00.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.760+0000 7efcafe00700 1 -- 172.21.15.78:0/3390398981 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7efcc8018cb0 con 0x55ed11ad9fe0 2024-03-14T09:54:00.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.760+0000 7efce8b52b80 10 client.15528 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:00.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.760+0000 7efce8b52b80 1 -- 172.21.15.78:0/3390398981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ed118b0cd0 con 0x55ed11ad9fe0 2024-03-14T09:54:00.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.760+0000 7efcafe00700 1 -- 172.21.15.78:0/3390398981 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efcbc09e9d0 con 0x55ed11ad9fe0 2024-03-14T09:54:00.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.760+0000 7efcafe00700 1 -- 172.21.15.78:0/3390398981 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7efcc8022050 con 0x55ed11ad9fe0 2024-03-14T09:54:00.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.760+0000 7efce8b52b80 10 client.15528 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:00.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.760+0000 7efce8b52b80 10 client.15528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:00.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.760+0000 7efce8b52b80 20 client.15528 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:00.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.760+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3390398981 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55ed11b411b0 0x55ed11b435d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:00.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.760+0000 7efce8b52b80 4 client.15528 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:54:00.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.760+0000 7efce8b52b80 1 -- 172.21.15.78:0/3390398981 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ed11879cf0 con 0x55ed11b411b0 2024-03-14T09:54:00.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.760+0000 7efccde00700 1 --2- 172.21.15.78:0/3390398981 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55ed11b411b0 0x55ed11b435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:00.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.761+0000 7efcafe00700 10 client.15528 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:00.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.761+0000 7efccde00700 1 --2- 172.21.15.78:0/3390398981 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55ed11b411b0 0x55ed11b435d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=368cd148d3cd955b server_cookie=a2f36ab9a9fdc2c in_seq=0 out_seq=0 2024-03-14T09:54:00.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.761+0000 7efcafe00700 1 -- 172.21.15.78:0/3390398981 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ed11879cf0 con 0x55ed11b411b0 2024-03-14T09:54:00.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.761+0000 7efcafe00700 10 client.15528 handle_command_reply: tid=0 2024-03-14T09:54:00.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.761+0000 7efce8b52b80 1 client.15528 shutdown 2024-03-14T09:54:00.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.761+0000 7efce8b52b80 1 -- 172.21.15.78:0/3390398981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9fe0 msgr2=0x55ed11ae5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.761+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3390398981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae5820 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efcc8019450 tx=0x7efcc801f000 comp rx=0 tx=0).stop 2024-03-14T09:54:00.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efce8b52b80 1 -- 172.21.15.78:0/3390398981 shutdown_connections 2024-03-14T09:54:00.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3390398981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed11ad9fe0 0x55ed11ae5820 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3390398981 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed11ad87f0 0x55ed11ae71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3390398981 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55ed11b411b0 0x55ed11b435d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efce8b52b80 1 --2- 172.21.15.78:0/3390398981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed11ad9150 0x55ed11ae7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efcafe00700 0 client.15528 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:00.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efce8b52b80 1 -- 172.21.15.78:0/3390398981 >> 172.21.15.78:0/3390398981 conn(0x55ed11a4e220 msgr2=0x55ed11ae1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:00.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efce8b52b80 1 -- 172.21.15.78:0/3390398981 shutdown_connections 2024-03-14T09:54:00.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efce8b52b80 1 -- 172.21.15.78:0/3390398981 wait complete. 2024-03-14T09:54:00.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efce8b52b80 20 client.15528 trim_cache size 0 max 16384 2024-03-14T09:54:00.767 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:54:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-14T09:54:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T09:54:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "c784aa4a-0c43-472b-809c-8c5888d8dfcb": { 2024-03-14T09:54:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T09:54:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "c784aa4a-0c43-472b-809c-8c5888d8dfcb", 2024-03-14T09:54:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T09:54:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:54:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:54:00.767 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:54:00.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efcd5000700 1 -- 172.21.15.78:0/4160609570 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7efcb40a0b20 msgr2=0x7efcb40a2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efcd5000700 1 --2- 172.21.15.78:0/4160609570 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7efcb40a0b20 0x7efcb40a2fe0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efcc800f540 tx=0x7efcc8009920 comp rx=0 tx=0).stop 2024-03-14T09:54:00.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efcd5000700 1 -- 172.21.15.78:0/4160609570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0074df0 msgr2=0x7efcd0199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:00.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efcd5000700 1 --2- 172.21.15.78:0/4160609570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0074df0 0x7efcd0199880 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efcbc00b680 tx=0x7efcbc004730 comp rx=0 tx=0).stop 2024-03-14T09:54:00.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efcd5000700 1 -- 172.21.15.78:0/4160609570 shutdown_connections 2024-03-14T09:54:00.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efcd5000700 1 --2- 172.21.15.78:0/4160609570 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efcd0108c00 0x7efcd01a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efcd5000700 1 --2- 172.21.15.78:0/4160609570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efcd0073810 0x7efcd0199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efcd5000700 1 --2- 172.21.15.78:0/4160609570 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7efcb40a0b20 0x7efcb40a2fe0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efcd5000700 1 --2- 172.21.15.78:0/4160609570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efcd0074df0 0x7efcd0199880 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:00.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.762+0000 7efcd5000700 1 -- 172.21.15.78:0/4160609570 >> 172.21.15.78:0/4160609570 conn(0x7efcd00fc2f0 msgr2=0x7efcd0104b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:00.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.763+0000 7efcd5000700 1 -- 172.21.15.78:0/4160609570 shutdown_connections 2024-03-14T09:54:00.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:00.763+0000 7efcd5000700 1 -- 172.21.15.78:0/4160609570 wait complete. 2024-03-14T09:54:00.780 INFO:tasks.cephfs.filesystem:scrub status for tag:c784aa4a-0c43-472b-809c-8c5888d8dfcb - {'path': '/', 'tag': 'c784aa4a-0c43-472b-809c-8c5888d8dfcb', 'options': 'recursive,force'} 2024-03-14T09:54:01.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.258+0000 7f603be00700 1 -- 172.21.15.78:0/4253601430 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 20) v1 ==== 135456+0+0 (secure 0 0 0) 0x7f603c003cb0 con 0x7f6044102880 2024-03-14T09:54:01.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.259+0000 7f603be00700 1 -- 172.21.15.78:0/4253601430 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v20) v1 ==== 92+0+0 (secure 0 0 0) 0x7f603c093a70 con 0x7f6044102880 2024-03-14T09:54:01.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.262+0000 7f604a000700 1 -- 172.21.15.78:0/4253601430 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f60280a0ba0 msgr2=0x7f60280a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:01.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.262+0000 7f604a000700 1 --2- 172.21.15.78:0/4253601430 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f60280a0ba0 0x7f60280a3060 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6034009990 tx=0x7f60340036f0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.262+0000 7f604a000700 1 -- 172.21.15.78:0/4253601430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6044102880 msgr2=0x7f60441a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:01.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.262+0000 7f604a000700 1 --2- 172.21.15.78:0/4253601430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6044102880 0x7f60441a0800 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f603c00ef30 tx=0x7f603c00c3a0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.262+0000 7f604a000700 1 -- 172.21.15.78:0/4253601430 shutdown_connections 2024-03-14T09:54:01.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.262+0000 7f604a000700 1 --2- 172.21.15.78:0/4253601430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6044102880 0x7f60441a0800 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.262+0000 7f604a000700 1 --2- 172.21.15.78:0/4253601430 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6044100cf0 0x7f6044199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.262+0000 7f604a000700 1 --2- 172.21.15.78:0/4253601430 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f60280a0ba0 0x7f60280a3060 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.262+0000 7f604a000700 1 --2- 172.21.15.78:0/4253601430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6044101ee0 0x7f6044199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.262+0000 7f604a000700 1 -- 172.21.15.78:0/4253601430 >> 172.21.15.78:0/4253601430 conn(0x7f60440fc280 msgr2=0x7f6044072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:01.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.262+0000 7f604a000700 1 -- 172.21.15.78:0/4253601430 shutdown_connections 2024-03-14T09:54:01.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.262+0000 7f604a000700 1 -- 172.21.15.78:0/4253601430 wait complete. 2024-03-14T09:54:01.284 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-14T09:54:01.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.489+0000 7fb6b1c00700 1 -- 172.21.15.78:0/3707100059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6ac101ee0 msgr2=0x7fb6ac102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:01.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.489+0000 7fb6b1c00700 1 --2- 172.21.15.78:0/3707100059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6ac101ee0 0x7fb6ac102340 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb698009920 tx=0x7fb698009c30 comp rx=0 tx=0).stop 2024-03-14T09:54:01.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.489+0000 7fb6b1c00700 1 -- 172.21.15.78:0/3707100059 shutdown_connections 2024-03-14T09:54:01.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.489+0000 7fb6b1c00700 1 --2- 172.21.15.78:0/3707100059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb6ac102880 0x7fb6ac109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.489+0000 7fb6b1c00700 1 --2- 172.21.15.78:0/3707100059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6ac100cf0 0x7fb6ac101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.489+0000 7fb6b1c00700 1 --2- 172.21.15.78:0/3707100059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6ac101ee0 0x7fb6ac102340 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.489+0000 7fb6b1c00700 1 -- 172.21.15.78:0/3707100059 >> 172.21.15.78:0/3707100059 conn(0x7fb6ac0fc280 msgr2=0x7fb6ac0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:01.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.489+0000 7fb6b1c00700 1 -- 172.21.15.78:0/3707100059 shutdown_connections 2024-03-14T09:54:01.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.489+0000 7fb6b1c00700 1 -- 172.21.15.78:0/3707100059 wait complete. 2024-03-14T09:54:01.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.490+0000 7fb6b1c00700 1 Processor -- start 2024-03-14T09:54:01.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.490+0000 7fb6b1c00700 1 -- start start 2024-03-14T09:54:01.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6b1c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb6ac100cf0 0x7fb6ac199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:01.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6b1c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6ac101ee0 0x7fb6ac1998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:01.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6b1c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6ac102880 0x7fb6ac1a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:01.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6b1c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac1a0ea0 con 0x7fb6ac101ee0 2024-03-14T09:54:01.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6b1c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac1a0fe0 con 0x7fb6ac100cf0 2024-03-14T09:54:01.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6b1c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac1a1170 con 0x7fb6ac102880 2024-03-14T09:54:01.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6aa000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6ac101ee0 0x7fb6ac1998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:01.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6ab400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6ac102880 0x7fb6ac1a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:01.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6aaa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb6ac100cf0 0x7fb6ac199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:01.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6ab400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6ac102880 0x7fb6ac1a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35756/0 (socket says 172.21.15.78:35756) 2024-03-14T09:54:01.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6aaa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb6ac100cf0 0x7fb6ac199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:44438/0 (socket says 172.21.15.78:44438) 2024-03-14T09:54:01.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6ab400700 1 -- 172.21.15.78:0/2578450187 learned_addr learned my addr 172.21.15.78:0/2578450187 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:01.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6aa000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6ac101ee0 0x7fb6ac1998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44274/0 (socket says 172.21.15.78:44274) 2024-03-14T09:54:01.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6ab400700 1 -- 172.21.15.78:0/2578450187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb6ac100cf0 msgr2=0x7fb6ac199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:01.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6ab400700 1 --2- 172.21.15.78:0/2578450187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb6ac100cf0 0x7fb6ac199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6ab400700 1 -- 172.21.15.78:0/2578450187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6ac101ee0 msgr2=0x7fb6ac1998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:01.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6ab400700 1 --2- 172.21.15.78:0/2578450187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6ac101ee0 0x7fb6ac1998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6ab400700 1 -- 172.21.15.78:0/2578450187 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c0095f0 con 0x7fb6ac102880 2024-03-14T09:54:01.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.491+0000 7fb6aa000700 1 --2- 172.21.15.78:0/2578450187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6ac101ee0 0x7fb6ac1998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:01.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.492+0000 7fb6ab400700 1 --2- 172.21.15.78:0/2578450187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6ac102880 0x7fb6ac1a08d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb6a400ef30 tx=0x7fb6a400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:01.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.492+0000 7fb6a3400700 1 -- 172.21.15.78:0/2578450187 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4018070 con 0x7fb6ac102880 2024-03-14T09:54:01.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.492+0000 7fb6a3400700 1 -- 172.21.15.78:0/2578450187 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb6a400f040 con 0x7fb6ac102880 2024-03-14T09:54:01.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.492+0000 7fb6b1c00700 1 -- 172.21.15.78:0/2578450187 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb698009520 con 0x7fb6ac102880 2024-03-14T09:54:01.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.492+0000 7fb6a3400700 1 -- 172.21.15.78:0/2578450187 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a400cd00 con 0x7fb6ac102880 2024-03-14T09:54:01.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.492+0000 7fb6b1c00700 1 -- 172.21.15.78:0/2578450187 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ac1a1770 con 0x7fb6ac102880 2024-03-14T09:54:01.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.493+0000 7fb6b1c00700 1 -- 172.21.15.78:0/2578450187 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ac067c00 con 0x7fb6ac102880 2024-03-14T09:54:01.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.495+0000 7fb6a3400700 1 -- 172.21.15.78:0/2578450187 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 20) v1 ==== 135456+0+0 (secure 0 0 0) 0x7fb6a4003a00 con 0x7fb6ac102880 2024-03-14T09:54:01.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.496+0000 7fb6a3400700 1 --2- 172.21.15.78:0/2578450187 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb6900a0b70 0x7fb6900a3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:01.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.496+0000 7fb6a3400700 1 -- 172.21.15.78:0/2578450187 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7fb6a40cc0f0 con 0x7fb6ac102880 2024-03-14T09:54:01.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.496+0000 7fb6aaa00700 1 -- 172.21.15.78:0/2578450187 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb6900a0b70 msgr2=0x7fb6900a3030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2580616009 2024-03-14T09:54:01.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.496+0000 7fb6aaa00700 1 --2- 172.21.15.78:0/2578450187 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb6900a0b70 0x7fb6900a3030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T09:54:01.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.500+0000 7fb6a3400700 1 -- 172.21.15.78:0/2578450187 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb6a4095620 con 0x7fb6ac102880 2024-03-14T09:54:01.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:01 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T09:54:01.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:01 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:01.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:01 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T09:54:01.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:01 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:01.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:01 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/4253601430' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-14T09:54:01.521 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:01 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: ignoring --setuser ceph since I am not root 2024-03-14T09:54:01.521 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:01 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: ignoring --setgroup ceph since I am not root 2024-03-14T09:54:01.521 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:01 smithi132 ceph-mgr[49465]: -- 172.21.15.132:0/4290231271 <== mon.2 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e14c0e41a0 con 0x55e14c0ba800 2024-03-14T09:54:01.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:01 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: ignoring --setuser ceph since I am not root 2024-03-14T09:54:01.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:01 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: ignoring --setgroup ceph since I am not root 2024-03-14T09:54:01.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:01 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T09:54:01.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:01 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:01.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:01 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T09:54:01.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:01 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:01.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:01 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/4253601430' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-14T09:54:01.694 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:01 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T09:54:01.694 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:01 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:01.695 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:01 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T09:54:01.695 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:01 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:01.695 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:01 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/4253601430' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-14T09:54:01.695 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:01 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: ignoring --setuser ceph since I am not root 2024-03-14T09:54:01.695 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:01 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: ignoring --setgroup ceph since I am not root 2024-03-14T09:54:01.695 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:01 smithi152 ceph-mgr[50168]: -- 172.21.15.152:0/1529775803 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b36db241a0 con 0x55b36dafb000 2024-03-14T09:54:01.695 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:01 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:01.693+0000 7f9d3e174200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-14T09:54:01.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.697+0000 7fb6aaa00700 1 -- 172.21.15.78:0/2578450187 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb6900a0b70 msgr2=0x7fb6900a3030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2580616009 2024-03-14T09:54:01.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.697+0000 7fb6aaa00700 1 --2- 172.21.15.78:0/2578450187 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb6900a0b70 0x7fb6900a3030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-14T09:54:01.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.716+0000 7fb6b1c00700 1 -- 172.21.15.78:0/2578450187 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fb6ac1a1a30 con 0x7fb6ac102880 2024-03-14T09:54:01.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.722+0000 7fb6a3400700 1 -- 172.21.15.78:0/2578450187 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb6a4098c40 con 0x7fb6ac102880 2024-03-14T09:54:01.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.724+0000 7fb6b1c00700 1 -- 172.21.15.78:0/2578450187 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb6900a0b70 msgr2=0x7fb6900a3030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T09:54:01.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.724+0000 7fb6b1c00700 1 --2- 172.21.15.78:0/2578450187 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb6900a0b70 0x7fb6900a3030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.724+0000 7fb6b1c00700 1 -- 172.21.15.78:0/2578450187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6ac102880 msgr2=0x7fb6ac1a08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.725+0000 7fb6b1c00700 1 --2- 172.21.15.78:0/2578450187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6ac102880 0x7fb6ac1a08d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb6a400ef30 tx=0x7fb6a400c3a0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.725+0000 7fb6b1c00700 1 -- 172.21.15.78:0/2578450187 shutdown_connections 2024-03-14T09:54:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.725+0000 7fb6b1c00700 1 --2- 172.21.15.78:0/2578450187 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fb6900a0b70 0x7fb6900a3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.725+0000 7fb6b1c00700 1 --2- 172.21.15.78:0/2578450187 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb6ac100cf0 0x7fb6ac199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.725+0000 7fb6b1c00700 1 --2- 172.21.15.78:0/2578450187 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb6ac102880 0x7fb6ac1a08d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.725+0000 7fb6b1c00700 1 --2- 172.21.15.78:0/2578450187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6ac101ee0 0x7fb6ac1998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.725+0000 7fb6b1c00700 1 -- 172.21.15.78:0/2578450187 >> 172.21.15.78:0/2578450187 conn(0x7fb6ac0fc280 msgr2=0x7fb6ac1076b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.725+0000 7fb6b1c00700 1 -- 172.21.15.78:0/2578450187 shutdown_connections 2024-03-14T09:54:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.725+0000 7fb6b1c00700 1 -- 172.21.15.78:0/2578450187 wait complete. 2024-03-14T09:54:01.744 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-14T09:54:01.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.938+0000 7fa3ed000700 1 -- 172.21.15.78:0/1960912665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3e8073790 msgr2=0x7fa3e8073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:01.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.938+0000 7fa3ed000700 1 --2- 172.21.15.78:0/1960912665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3e8073790 0x7fa3e8073c10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4009990 tx=0x7fa3d4009ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.939+0000 7fa3ed000700 1 -- 172.21.15.78:0/1960912665 shutdown_connections 2024-03-14T09:54:01.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.939+0000 7fa3ed000700 1 --2- 172.21.15.78:0/1960912665 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3e8108be0 0x7fa3e810b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.939+0000 7fa3ed000700 1 --2- 172.21.15.78:0/1960912665 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3e8074d70 0x7fa3e80731c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.939+0000 7fa3ed000700 1 --2- 172.21.15.78:0/1960912665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3e8073790 0x7fa3e8073c10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.939+0000 7fa3ed000700 1 -- 172.21.15.78:0/1960912665 >> 172.21.15.78:0/1960912665 conn(0x7fa3e80fc280 msgr2=0x7fa3e80fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:01.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.939+0000 7fa3ed000700 1 -- 172.21.15.78:0/1960912665 shutdown_connections 2024-03-14T09:54:01.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.939+0000 7fa3ed000700 1 -- 172.21.15.78:0/1960912665 wait complete. 2024-03-14T09:54:01.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.940+0000 7fa3ed000700 1 Processor -- start 2024-03-14T09:54:01.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.940+0000 7fa3ed000700 1 -- start start 2024-03-14T09:54:01.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.940+0000 7fa3ed000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3e8073790 0x7fa3e8199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:01.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.940+0000 7fa3ed000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3e8074d70 0x7fa3e8199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:01.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.940+0000 7fa3ed000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3e8108be0 0x7fa3e81a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:01.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.940+0000 7fa3ed000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e81a0d50 con 0x7fa3e8074d70 2024-03-14T09:54:01.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.940+0000 7fa3ed000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e81a0e90 con 0x7fa3e8073790 2024-03-14T09:54:01.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.940+0000 7fa3ed000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e81a1020 con 0x7fa3e8108be0 2024-03-14T09:54:01.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.941+0000 7fa3e5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3e8073790 0x7fa3e8199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:01.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.941+0000 7fa3e5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3e8073790 0x7fa3e8199220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:44462/0 (socket says 172.21.15.78:44462) 2024-03-14T09:54:01.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.941+0000 7fa3e5e00700 1 -- 172.21.15.78:0/1025623244 learned_addr learned my addr 172.21.15.78:0/1025623244 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:01.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.941+0000 7fa3e6800700 1 --2- 172.21.15.78:0/1025623244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3e8108be0 0x7fa3e81a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:01.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.941+0000 7fa3e5400700 1 --2- 172.21.15.78:0/1025623244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3e8074d70 0x7fa3e8199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:01.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.941+0000 7fa3e6800700 1 -- 172.21.15.78:0/1025623244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3e8073790 msgr2=0x7fa3e8199220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:01.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.941+0000 7fa3e6800700 1 --2- 172.21.15.78:0/1025623244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3e8073790 0x7fa3e8199220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.941+0000 7fa3e6800700 1 -- 172.21.15.78:0/1025623244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3e8074d70 msgr2=0x7fa3e8199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:01.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.941+0000 7fa3e6800700 1 --2- 172.21.15.78:0/1025623244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3e8074d70 0x7fa3e8199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:01.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.941+0000 7fa3e6800700 1 -- 172.21.15.78:0/1025623244 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d8009520 con 0x7fa3e8108be0 2024-03-14T09:54:01.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.941+0000 7fa3e5e00700 1 --2- 172.21.15.78:0/1025623244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3e8073790 0x7fa3e8199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:01.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.942+0000 7fa3e6800700 1 --2- 172.21.15.78:0/1025623244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3e8108be0 0x7fa3e81a0780 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa3e000efc0 tx=0x7fa3e000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:01.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.942+0000 7fa3dea00700 1 -- 172.21.15.78:0/1025623244 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e0018070 con 0x7fa3e8108be0 2024-03-14T09:54:01.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.942+0000 7fa3ed000700 1 -- 172.21.15.78:0/1025623244 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d40095f0 con 0x7fa3e8108be0 2024-03-14T09:54:01.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.942+0000 7fa3dea00700 1 -- 172.21.15.78:0/1025623244 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3e000f040 con 0x7fa3e8108be0 2024-03-14T09:54:01.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.942+0000 7fa3dea00700 1 -- 172.21.15.78:0/1025623244 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e000cdf0 con 0x7fa3e8108be0 2024-03-14T09:54:01.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.942+0000 7fa3ed000700 1 -- 172.21.15.78:0/1025623244 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e8100af0 con 0x7fa3e8108be0 2024-03-14T09:54:01.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.943+0000 7fa3ed000700 1 -- 172.21.15.78:0/1025623244 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e81079f0 con 0x7fa3e8108be0 2024-03-14T09:54:01.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.945+0000 7fa3dea00700 1 -- 172.21.15.78:0/1025623244 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 20) v1 ==== 135456+0+0 (secure 0 0 0) 0x7fa3e0003a00 con 0x7fa3e8108be0 2024-03-14T09:54:01.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.946+0000 7fa3dea00700 1 --2- 172.21.15.78:0/1025623244 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa3cc0a0bf0 0x7fa3cc0a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:01.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.947+0000 7fa3e5e00700 1 -- 172.21.15.78:0/1025623244 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa3cc0a0bf0 msgr2=0x7fa3cc0a30b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2580616009 2024-03-14T09:54:01.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.947+0000 7fa3dea00700 1 -- 172.21.15.78:0/1025623244 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7fa3e00cc7a0 con 0x7fa3e8108be0 2024-03-14T09:54:01.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.947+0000 7fa3e5e00700 1 --2- 172.21.15.78:0/1025623244 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa3cc0a0bf0 0x7fa3cc0a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T09:54:01.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:01.952+0000 7fa3dea00700 1 -- 172.21.15.78:0/1025623244 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa3e0095cd0 con 0x7fa3e8108be0 2024-03-14T09:54:01.965 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:01 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:01.788+0000 7f9d3e174200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-14T09:54:01.965 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:01 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:01.876+0000 7f9d3e174200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-14T09:54:01.972 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:01 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:01.703+0000 7f5df8cc5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-14T09:54:01.972 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:01 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:01.799+0000 7f5df8cc5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-14T09:54:01.972 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:01 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:01.886+0000 7f5df8cc5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-14T09:54:02.004 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:01 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:01.876+0000 7fd97f342200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-14T09:54:02.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.148+0000 7fa3e5e00700 1 -- 172.21.15.78:0/1025623244 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa3cc0a0bf0 msgr2=0x7fa3cc0a30b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2580616009 2024-03-14T09:54:02.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.148+0000 7fa3e5e00700 1 --2- 172.21.15.78:0/1025623244 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa3cc0a0bf0 0x7fa3cc0a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-14T09:54:02.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.168+0000 7fa3ed000700 1 -- 172.21.15.78:0/1025623244 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fa3e810eb00 con 0x7fa3e8108be0 2024-03-14T09:54:02.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.177+0000 7fa3dea00700 1 -- 172.21.15.78:0/1025623244 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7fa3e0024330 con 0x7fa3e8108be0 2024-03-14T09:54:02.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.180+0000 7fa3ed000700 1 -- 172.21.15.78:0/1025623244 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa3cc0a0bf0 msgr2=0x7fa3cc0a30b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T09:54:02.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.180+0000 7fa3ed000700 1 --2- 172.21.15.78:0/1025623244 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa3cc0a0bf0 0x7fa3cc0a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.180+0000 7fa3ed000700 1 -- 172.21.15.78:0/1025623244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3e8108be0 msgr2=0x7fa3e81a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.180+0000 7fa3ed000700 1 --2- 172.21.15.78:0/1025623244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3e8108be0 0x7fa3e81a0780 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa3e000efc0 tx=0x7fa3e000c3a0 comp rx=0 tx=0).stop 2024-03-14T09:54:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.180+0000 7fa3ed000700 1 -- 172.21.15.78:0/1025623244 shutdown_connections 2024-03-14T09:54:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.180+0000 7fa3ed000700 1 --2- 172.21.15.78:0/1025623244 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7fa3cc0a0bf0 0x7fa3cc0a30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.180+0000 7fa3ed000700 1 --2- 172.21.15.78:0/1025623244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3e8073790 0x7fa3e8199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.180+0000 7fa3ed000700 1 --2- 172.21.15.78:0/1025623244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3e8108be0 0x7fa3e81a0780 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.180+0000 7fa3ed000700 1 --2- 172.21.15.78:0/1025623244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3e8074d70 0x7fa3e8199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.180+0000 7fa3ed000700 1 -- 172.21.15.78:0/1025623244 >> 172.21.15.78:0/1025623244 conn(0x7fa3e80fc280 msgr2=0x7fa3e8104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.180+0000 7fa3ed000700 1 -- 172.21.15.78:0/1025623244 shutdown_connections 2024-03-14T09:54:02.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.180+0000 7fa3ed000700 1 -- 172.21.15.78:0/1025623244 wait complete. 2024-03-14T09:54:02.199 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-14T09:54:02.248 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:01 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:01.965+0000 7f9d3e174200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-14T09:54:02.249 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:01 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:01.971+0000 7f5df8cc5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-14T09:54:02.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:02 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/4253601430' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-14T09:54:02.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:02 smithi078 ceph-mon[38143]: mgrmap e20: x(active, since 7m), standbys: z, y 2024-03-14T09:54:02.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:02 smithi078 ceph-mon[38143]: from='client.? ' entity='client.admin' 2024-03-14T09:54:02.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:02 smithi078 ceph-mon[38143]: from='client.? ' entity='client.admin' 2024-03-14T09:54:02.257 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:02 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:02.003+0000 7fd97f342200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-14T09:54:02.258 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:02 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:02.121+0000 7fd97f342200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-14T09:54:02.258 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:02 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:02.239+0000 7fd97f342200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-14T09:54:02.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.394+0000 7f278aa00700 1 -- 172.21.15.78:0/2124185578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2784074df0 msgr2=0x7f2784073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:02.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.394+0000 7f278aa00700 1 --2- 172.21.15.78:0/2124185578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2784074df0 0x7f2784073240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2774009990 tx=0x7f2774009ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:02.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.394+0000 7f278aa00700 1 -- 172.21.15.78:0/2124185578 shutdown_connections 2024-03-14T09:54:02.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.394+0000 7f278aa00700 1 --2- 172.21.15.78:0/2124185578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2784074df0 0x7f2784073240 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:02.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.394+0000 7f278aa00700 1 --2- 172.21.15.78:0/2124185578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f278410d2c0 0x7f278410f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:02.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.394+0000 7f278aa00700 1 --2- 172.21.15.78:0/2124185578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2784073810 0x7f2784073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:02.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.394+0000 7f278aa00700 1 -- 172.21.15.78:0/2124185578 >> 172.21.15.78:0/2124185578 conn(0x7f27840fc280 msgr2=0x7f27840fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:02.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.394+0000 7f278aa00700 1 -- 172.21.15.78:0/2124185578 shutdown_connections 2024-03-14T09:54:02.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.395+0000 7f278aa00700 1 -- 172.21.15.78:0/2124185578 wait complete. 2024-03-14T09:54:02.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.395+0000 7f278aa00700 1 Processor -- start 2024-03-14T09:54:02.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.396+0000 7f278aa00700 1 -- start start 2024-03-14T09:54:02.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.396+0000 7f278aa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2784073810 0x7f278419d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:02.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.396+0000 7f278aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2784074df0 0x7f278419dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:02.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.396+0000 7f278aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f278410d2c0 0x7f27841a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:02.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.396+0000 7f278aa00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27841a51c0 con 0x7f2784074df0 2024-03-14T09:54:02.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.396+0000 7f278aa00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27841a5300 con 0x7f278410d2c0 2024-03-14T09:54:02.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.396+0000 7f278aa00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27841a5490 con 0x7f2784073810 2024-03-14T09:54:02.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.396+0000 7f2783e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2784073810 0x7f278419d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:02.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.396+0000 7f2788a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f278410d2c0 0x7f27841a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:02.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.396+0000 7f2783e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2784073810 0x7f278419d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35776/0 (socket says 172.21.15.78:35776) 2024-03-14T09:54:02.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.396+0000 7f2783e00700 1 -- 172.21.15.78:0/3196010531 learned_addr learned my addr 172.21.15.78:0/3196010531 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:02.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.396+0000 7f2783400700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2784074df0 0x7f278419dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:02.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.397+0000 7f2788a00700 1 -- 172.21.15.78:0/3196010531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2784073810 msgr2=0x7f278419d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:02.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.397+0000 7f2788a00700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2784073810 0x7f278419d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:02.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.397+0000 7f2788a00700 1 -- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2784074df0 msgr2=0x7f278419dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:02.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.397+0000 7f2788a00700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2784074df0 0x7f278419dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:02.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.397+0000 7f2788a00700 1 -- 172.21.15.78:0/3196010531 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27740095f0 con 0x7f278410d2c0 2024-03-14T09:54:02.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.397+0000 7f2788a00700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f278410d2c0 0x7f27841a4bf0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f277c00efc0 tx=0x7f277c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:02.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.397+0000 7f2780c00700 1 -- 172.21.15.78:0/3196010531 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c018070 con 0x7f278410d2c0 2024-03-14T09:54:02.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.397+0000 7f2780c00700 1 -- 172.21.15.78:0/3196010531 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f277c00f040 con 0x7f278410d2c0 2024-03-14T09:54:02.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.397+0000 7f278aa00700 1 -- 172.21.15.78:0/3196010531 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27841a5710 con 0x7f278410d2c0 2024-03-14T09:54:02.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.397+0000 7f2780c00700 1 -- 172.21.15.78:0/3196010531 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c00cd40 con 0x7f278410d2c0 2024-03-14T09:54:02.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.397+0000 7f278aa00700 1 -- 172.21.15.78:0/3196010531 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27841a5c70 con 0x7f278410d2c0 2024-03-14T09:54:02.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.398+0000 7f278aa00700 1 -- 172.21.15.78:0/3196010531 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2784103520 con 0x7f278410d2c0 2024-03-14T09:54:02.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.400+0000 7f2780c00700 1 -- 172.21.15.78:0/3196010531 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 20) v1 ==== 135456+0+0 (secure 0 0 0) 0x7f277c003a00 con 0x7f278410d2c0 2024-03-14T09:54:02.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.401+0000 7f2780c00700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27680a0bf0 0x7f27680a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:02.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.402+0000 7f2780c00700 1 -- 172.21.15.78:0/3196010531 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9107+0+0 (secure 0 0 0) 0x7f277c0cca90 con 0x7f278410d2c0 2024-03-14T09:54:02.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.402+0000 7f2783e00700 1 -- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27680a0bf0 msgr2=0x7f27680a30b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2580616009 2024-03-14T09:54:02.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.402+0000 7f2783e00700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27680a0bf0 0x7f27680a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T09:54:02.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.407+0000 7f2780c00700 1 -- 172.21.15.78:0/3196010531 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f277c095fc0 con 0x7f278410d2c0 2024-03-14T09:54:02.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:02 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/4253601430' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-14T09:54:02.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:02 smithi132 ceph-mon[47959]: mgrmap e20: x(active, since 7m), standbys: z, y 2024-03-14T09:54:02.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:02 smithi132 ceph-mon[47959]: from='client.? ' entity='client.admin' 2024-03-14T09:54:02.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:02 smithi132 ceph-mon[47959]: from='client.? ' entity='client.admin' 2024-03-14T09:54:02.596 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:02 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/4253601430' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-14T09:54:02.596 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:02 smithi152 ceph-mon[48279]: mgrmap e20: x(active, since 7m), standbys: z, y 2024-03-14T09:54:02.596 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:02 smithi152 ceph-mon[48279]: from='client.? ' entity='client.admin' 2024-03-14T09:54:02.596 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:02 smithi152 ceph-mon[48279]: from='client.? ' entity='client.admin' 2024-03-14T09:54:02.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.603+0000 7f2783e00700 1 -- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27680a0bf0 msgr2=0x7f27680a30b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2580616009 2024-03-14T09:54:02.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.603+0000 7f2783e00700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27680a0bf0 0x7f27680a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-14T09:54:02.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:02.645+0000 7f278aa00700 1 -- 172.21.15.78:0/3196010531 --> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f27840611f0 con 0x7f27680a0bf0 2024-03-14T09:54:02.848 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:02 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:02.595+0000 7f9d3e174200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-14T09:54:02.856 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:02 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:02.602+0000 7f5df8cc5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-14T09:54:03.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:03.004+0000 7f2783e00700 1 -- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27680a0bf0 msgr2=0x7f27680a30b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2580616009 2024-03-14T09:54:03.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:03.004+0000 7f2783e00700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27680a0bf0 0x7f27680a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-14T09:54:03.201 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:02 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:02.847+0000 7f9d3e174200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-14T09:54:03.202 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:02 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:02.934+0000 7f9d3e174200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-14T09:54:03.202 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:03 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:03.043+0000 7f9d3e174200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-14T09:54:03.212 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:02 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:02.855+0000 7f5df8cc5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-14T09:54:03.212 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:02 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:02.942+0000 7f5df8cc5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-14T09:54:03.212 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:03 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:03.053+0000 7f5df8cc5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-14T09:54:03.398 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:03 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:03.087+0000 7fd97f342200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-14T09:54:03.499 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:03 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:03.201+0000 7f9d3e174200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-14T09:54:03.499 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:03 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:03.328+0000 7f9d3e174200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-14T09:54:03.499 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:03 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:03.413+0000 7f9d3e174200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-14T09:54:03.520 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:03 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:03.211+0000 7f5df8cc5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-14T09:54:03.520 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:03 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:03.339+0000 7f5df8cc5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-14T09:54:03.520 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:03 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:03.425+0000 7f5df8cc5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-14T09:54:03.803 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:03 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:03.397+0000 7fd97f342200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-14T09:54:03.803 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:03 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:03.508+0000 7fd97f342200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-14T09:54:03.803 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:03 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:03.646+0000 7fd97f342200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-14T09:54:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:03.805+0000 7f2783e00700 1 -- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27680a0bf0 msgr2=0x7f27680a30b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2580616009 2024-03-14T09:54:03.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:03.805+0000 7f2783e00700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27680a0bf0 0x7f27680a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-14T09:54:04.119 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:03 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:03.842+0000 7fd97f342200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-14T09:54:04.120 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:04 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:04.007+0000 7fd97f342200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-14T09:54:04.171 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:03 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:03.783+0000 7f9d3e174200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-14T09:54:04.184 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:03 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:03.796+0000 7f5df8cc5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-14T09:54:04.458 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:04 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:04.170+0000 7f9d3e174200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-14T09:54:04.458 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:04 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:04.363+0000 7f9d3e174200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-14T09:54:04.472 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:04 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:04.183+0000 7f5df8cc5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-14T09:54:04.472 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:04 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:04.378+0000 7f5df8cc5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-14T09:54:04.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:04 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:04.119+0000 7fd97f342200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-14T09:54:04.749 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:04 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:04.457+0000 7f9d3e174200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-14T09:54:04.770 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:04 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:04.471+0000 7f5df8cc5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-14T09:54:05.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:04 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:04.585+0000 7fd97f342200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-14T09:54:05.322 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:05 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:05.071+0000 7fd97f342200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-14T09:54:05.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:05.407+0000 7f2783e00700 1 -- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27680a0bf0 msgr2=0x7f27680a30b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2580616009 2024-03-14T09:54:05.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:05.407+0000 7f2783e00700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27680a0bf0 0x7f27680a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-14T09:54:05.499 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:05 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:05.216+0000 7f9d3e174200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-14T09:54:05.499 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:05 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:05.315+0000 7f9d3e174200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-14T09:54:05.499 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:05 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:05.463+0000 7f9d3e174200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-14T09:54:05.520 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:05 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:05.234+0000 7f5df8cc5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-14T09:54:05.520 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:05 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:05.332+0000 7f5df8cc5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-14T09:54:05.520 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:05 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:05.481+0000 7f5df8cc5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-14T09:54:05.803 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:05 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:05.321+0000 7fd97f342200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-14T09:54:05.803 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:05 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:05.444+0000 7fd97f342200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-14T09:54:06.464 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:06 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:06.089+0000 7f9d3e174200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-14T09:54:06.464 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:06 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:06.270+0000 7f9d3e174200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-14T09:54:06.489 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:06.112+0000 7f5df8cc5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-14T09:54:06.489 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:06.295+0000 7f5df8cc5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-14T09:54:06.717 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:06 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:06.396+0000 7fd97f342200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-14T09:54:06.717 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:06 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:06.523+0000 7fd97f342200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-14T09:54:06.749 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:06 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:06.463+0000 7f9d3e174200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-14T09:54:06.770 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:06.488+0000 7f5df8cc5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-14T09:54:07.053 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:06 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:06.716+0000 7fd97f342200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-14T09:54:07.676 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:07 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:07.392+0000 7f9d3e174200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-14T09:54:07.710 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:07 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:07.423+0000 7f5df8cc5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-14T09:54:07.803 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:07 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:07.521+0000 7fd97f342200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-14T09:54:07.803 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:07 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:07.763+0000 7fd97f342200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-14T09:54:07.958 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:07 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:07.675+0000 7f9d3e174200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-14T09:54:07.958 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:07 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:07.764+0000 7f9d3e174200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-14T09:54:07.992 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:07 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:07.710+0000 7f5df8cc5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-14T09:54:07.992 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:07 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:07.799+0000 7f5df8cc5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-14T09:54:08.249 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 09:54:07 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z[50145]: 2024-03-14T09:54:07.955+0000 7f9d3e174200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-14T09:54:08.270 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 09:54:07 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y[49442]: 2024-03-14T09:54:07.991+0000 7f5df8cc5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-14T09:54:08.303 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:08 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:08.019+0000 7fd97f342200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-14T09:54:08.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:08.610+0000 7f2783e00700 1 -- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27680a0bf0 msgr2=0x7f27680a30b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/2580616009 2024-03-14T09:54:08.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:08.610+0000 7f2783e00700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27680a0bf0 0x7f27680a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-14T09:54:08.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:08.744+0000 7f2780c00700 1 -- 172.21.15.78:0/3196010531 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 21) v1 ==== 135456+0+0 (secure 0 0 0) 0x7f277c003cb0 con 0x7f278410d2c0 2024-03-14T09:54:08.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:08 smithi152 ceph-mon[48279]: Standby manager daemon z restarted 2024-03-14T09:54:08.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:08 smithi152 ceph-mon[48279]: Standby manager daemon z started 2024-03-14T09:54:08.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:08 smithi152 ceph-mon[48279]: from='mgr.? 172.21.15.152:0/1653643' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-14T09:54:09.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:08 smithi152 ceph-mon[48279]: from='mgr.? 172.21.15.152:0/1653643' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T09:54:09.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:08 smithi152 ceph-mon[48279]: from='mgr.? 172.21.15.152:0/1653643' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-14T09:54:09.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:08 smithi152 ceph-mon[48279]: from='mgr.? 172.21.15.152:0/1653643' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T09:54:09.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:08 smithi152 ceph-mon[48279]: Standby manager daemon y restarted 2024-03-14T09:54:09.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:08 smithi152 ceph-mon[48279]: Standby manager daemon y started 2024-03-14T09:54:09.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:08 smithi152 ceph-mon[48279]: from='mgr.? 172.21.15.132:0/1081265264' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-14T09:54:09.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:08 smithi152 ceph-mon[48279]: from='mgr.? 172.21.15.132:0/1081265264' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T09:54:09.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:08 smithi152 ceph-mon[48279]: from='mgr.? 172.21.15.132:0/1081265264' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-14T09:54:09.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:08 smithi152 ceph-mon[48279]: from='mgr.? 172.21.15.132:0/1081265264' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T09:54:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:08 smithi132 ceph-mon[47959]: Standby manager daemon z restarted 2024-03-14T09:54:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:08 smithi132 ceph-mon[47959]: Standby manager daemon z started 2024-03-14T09:54:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:08 smithi132 ceph-mon[47959]: from='mgr.? 172.21.15.152:0/1653643' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-14T09:54:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:08 smithi132 ceph-mon[47959]: from='mgr.? 172.21.15.152:0/1653643' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T09:54:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:08 smithi132 ceph-mon[47959]: from='mgr.? 172.21.15.152:0/1653643' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-14T09:54:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:08 smithi132 ceph-mon[47959]: from='mgr.? 172.21.15.152:0/1653643' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T09:54:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:08 smithi132 ceph-mon[47959]: Standby manager daemon y restarted 2024-03-14T09:54:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:08 smithi132 ceph-mon[47959]: Standby manager daemon y started 2024-03-14T09:54:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:08 smithi132 ceph-mon[47959]: from='mgr.? 172.21.15.132:0/1081265264' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-14T09:54:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:08 smithi132 ceph-mon[47959]: from='mgr.? 172.21.15.132:0/1081265264' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T09:54:09.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:08 smithi132 ceph-mon[47959]: from='mgr.? 172.21.15.132:0/1081265264' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-14T09:54:09.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:08 smithi132 ceph-mon[47959]: from='mgr.? 172.21.15.132:0/1081265264' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T09:54:09.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:08 smithi078 ceph-mon[38143]: Standby manager daemon z restarted 2024-03-14T09:54:09.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:08 smithi078 ceph-mon[38143]: Standby manager daemon z started 2024-03-14T09:54:09.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:08 smithi078 ceph-mon[38143]: from='mgr.? 172.21.15.152:0/1653643' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-14T09:54:09.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:08 smithi078 ceph-mon[38143]: from='mgr.? 172.21.15.152:0/1653643' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T09:54:09.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:08 smithi078 ceph-mon[38143]: from='mgr.? 172.21.15.152:0/1653643' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-14T09:54:09.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:08 smithi078 ceph-mon[38143]: from='mgr.? 172.21.15.152:0/1653643' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T09:54:09.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:08 smithi078 ceph-mon[38143]: Standby manager daemon y restarted 2024-03-14T09:54:09.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:08 smithi078 ceph-mon[38143]: Standby manager daemon y started 2024-03-14T09:54:09.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:08 smithi078 ceph-mon[38143]: from='mgr.? 172.21.15.132:0/1081265264' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-14T09:54:09.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:08 smithi078 ceph-mon[38143]: from='mgr.? 172.21.15.132:0/1081265264' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T09:54:09.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:08 smithi078 ceph-mon[38143]: from='mgr.? 172.21.15.132:0/1081265264' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-14T09:54:09.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:08 smithi078 ceph-mon[38143]: from='mgr.? 172.21.15.132:0/1081265264' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T09:54:09.545 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:09 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:09.168+0000 7fd97f342200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-14T09:54:09.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:09 smithi078 ceph-mon[38143]: mgrmap e21: x(active, since 7m), standbys: z, y 2024-03-14T09:54:09.803 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:09 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:09.544+0000 7fd97f342200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-14T09:54:09.803 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:09 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:09.664+0000 7fd97f342200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-14T09:54:09.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:09.937+0000 7f2780c00700 1 -- 172.21.15.78:0/3196010531 <== mon.2 v2:172.21.15.132:3300/0 8 ==== mgrmap(e 22) v1 ==== 135073+0+0 (secure 0 0 0) 0x7f277c096390 con 0x7f278410d2c0 2024-03-14T09:54:09.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:09.938+0000 7f2780c00700 1 -- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27680a0bf0 msgr2=0x7f27680a30b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T09:54:09.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:09.938+0000 7f2780c00700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2580616009,v1:172.21.15.78:6801/2580616009] conn(0x7f27680a0bf0 0x7f27680a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:09.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:09 smithi152 ceph-mon[48279]: mgrmap e21: x(active, since 7m), standbys: z, y 2024-03-14T09:54:10.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:09 smithi132 ceph-mon[47959]: mgrmap e21: x(active, since 7m), standbys: z, y 2024-03-14T09:54:10.188 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:09 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:09.922+0000 7fd97f342200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-14T09:54:10.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:10.206+0000 7fd8d5c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:54:10.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:10.206+0000 7fd8d5c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:54:10.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:10.206+0000 7fd8d5c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:54:10.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:10.206+0000 7fd8d5c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:54:10.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:10.206+0000 7fd8d5c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:54:10.553 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:10.207+0000 7fd8d2000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:54:10.554 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:10.207+0000 7fd8d2000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:54:10.554 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:10.207+0000 7fd8d2000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:54:10.554 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:10.207+0000 7fd8d2000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:54:10.554 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:54:10.207+0000 7fd8d2000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:54:10.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:10.945+0000 7f2780c00700 1 -- 172.21.15.78:0/3196010531 <== mon.2 v2:172.21.15.132:3300/0 9 ==== mgrmap(e 23) v1 ==== 135200+0+0 (secure 0 0 0) 0x7f277c050230 con 0x7f278410d2c0 2024-03-14T09:54:10.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:10.947+0000 7f2780c00700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f276801c1e0 0x7f276801c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:10.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:10.947+0000 7f2780c00700 1 -- 172.21.15.78:0/3196010531 --> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f27840611f0 con 0x7f276801c1e0 2024-03-14T09:54:10.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:10.949+0000 7f2783e00700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f276801c1e0 0x7f276801c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:10.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:10.951+0000 7f2783e00700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f276801c1e0 0x7f276801c5c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f277400b2e0 tx=0x7f277400b750 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:10.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: Active manager daemon x restarted 2024-03-14T09:54:11.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: Activating manager daemon x 2024-03-14T09:54:11.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: osdmap e88: 12 total, 12 up, 12 in 2024-03-14T09:54:11.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: mgrmap e22: x(active, starting, since 0.0124503s), standbys: z, y 2024-03-14T09:54:11.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:54:11.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:54:11.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:54:11.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-14T09:54:11.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-14T09:54:11.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-14T09:54:11.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-14T09:54:11.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-14T09:54:11.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-14T09:54:11.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-14T09:54:11.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-14T09:54:11.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-14T09:54:11.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-14T09:54:11.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-14T09:54:11.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-14T09:54:11.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-14T09:54:11.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-14T09:54:11.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-14T09:54:11.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:54:11.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:54:11.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:54:11.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:54:11.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:54:11.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:54:11.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:54:11.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:54:11.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:54:11.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:54:11.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:54:11.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:54:11.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-14T09:54:11.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-14T09:54:11.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-14T09:54:11.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: Manager daemon x is now available 2024-03-14T09:54:11.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:54:11.002 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:54:11.003 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-14T09:54:11.003 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:10 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-14T09:54:11.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: Active manager daemon x restarted 2024-03-14T09:54:11.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: Activating manager daemon x 2024-03-14T09:54:11.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: osdmap e88: 12 total, 12 up, 12 in 2024-03-14T09:54:11.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: mgrmap e22: x(active, starting, since 0.0124503s), standbys: z, y 2024-03-14T09:54:11.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:54:11.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:54:11.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:54:11.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-14T09:54:11.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-14T09:54:11.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-14T09:54:11.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-14T09:54:11.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-14T09:54:11.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-14T09:54:11.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-14T09:54:11.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-14T09:54:11.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-14T09:54:11.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-14T09:54:11.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-14T09:54:11.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-14T09:54:11.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-14T09:54:11.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-14T09:54:11.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-14T09:54:11.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:54:11.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:54:11.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:54:11.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:54:11.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:54:11.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:54:11.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:54:11.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:54:11.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:54:11.023 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:54:11.023 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:54:11.023 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:54:11.023 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-14T09:54:11.023 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-14T09:54:11.023 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-14T09:54:11.023 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: Manager daemon x is now available 2024-03-14T09:54:11.023 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:54:11.023 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:54:11.023 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-14T09:54:11.023 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:10 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-14T09:54:11.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.024+0000 7f2780c00700 1 -- 172.21.15.78:0/3196010531 <== mgr.15546 v2:172.21.15.78:6800/2192201293 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f27840611f0 con 0x7f276801c1e0 2024-03-14T09:54:11.024 INFO:teuthology.orchestra.run.smithi078.stdout:Schedule set for path / 2024-03-14T09:54:11.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.026+0000 7f278aa00700 1 -- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f276801c1e0 msgr2=0x7f276801c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:11.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.026+0000 7f278aa00700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f276801c1e0 0x7f276801c5c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f277400b2e0 tx=0x7f277400b750 comp rx=0 tx=0).stop 2024-03-14T09:54:11.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.026+0000 7f278aa00700 1 -- 172.21.15.78:0/3196010531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f278410d2c0 msgr2=0x7f27841a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:11.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.026+0000 7f278aa00700 1 --2- 172.21.15.78:0/3196010531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f278410d2c0 0x7f27841a4bf0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f277c00efc0 tx=0x7f277c00c3a0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.026+0000 7f2788a00700 1 -- 172.21.15.78:0/3196010531 reap_dead start 2024-03-14T09:54:11.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.026+0000 7f278aa00700 1 -- 172.21.15.78:0/3196010531 shutdown_connections 2024-03-14T09:54:11.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.026+0000 7f278aa00700 1 -- 172.21.15.78:0/3196010531 >> 172.21.15.78:0/3196010531 conn(0x7f27840fc280 msgr2=0x7f2784100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:11.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.027+0000 7f278aa00700 1 -- 172.21.15.78:0/3196010531 shutdown_connections 2024-03-14T09:54:11.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.027+0000 7f278aa00700 1 -- 172.21.15.78:0/3196010531 wait complete. 2024-03-14T09:54:11.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: Active manager daemon x restarted 2024-03-14T09:54:11.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: Activating manager daemon x 2024-03-14T09:54:11.045 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: osdmap e88: 12 total, 12 up, 12 in 2024-03-14T09:54:11.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: mgrmap e22: x(active, starting, since 0.0124503s), standbys: z, y 2024-03-14T09:54:11.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-14T09:54:11.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-14T09:54:11.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-14T09:54:11.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-14T09:54:11.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-14T09:54:11.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-14T09:54:11.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-14T09:54:11.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-14T09:54:11.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-14T09:54:11.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-14T09:54:11.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-14T09:54:11.046 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-14T09:54:11.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-14T09:54:11.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-14T09:54:11.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-14T09:54:11.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-14T09:54:11.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-14T09:54:11.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-14T09:54:11.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T09:54:11.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T09:54:11.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T09:54:11.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T09:54:11.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T09:54:11.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T09:54:11.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-14T09:54:11.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-14T09:54:11.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-14T09:54:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-14T09:54:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-14T09:54:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-14T09:54:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-14T09:54:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-14T09:54:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-14T09:54:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: Manager daemon x is now available 2024-03-14T09:54:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:54:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:54:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-14T09:54:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:10 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-14T09:54:11.049 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-14T09:54:11.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.250+0000 7efd65200700 1 -- 172.21.15.78:0/2680323968 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd6010d2c0 msgr2=0x7efd6010f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:11.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.250+0000 7efd65200700 1 --2- 172.21.15.78:0/2680323968 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd6010d2c0 0x7efd6010f6b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efd5800b0a0 tx=0x7efd5800b3b0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.250+0000 7efd65200700 1 -- 172.21.15.78:0/2680323968 shutdown_connections 2024-03-14T09:54:11.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.250+0000 7efd65200700 1 --2- 172.21.15.78:0/2680323968 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd6010d2c0 0x7efd6010f6b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.250+0000 7efd65200700 1 --2- 172.21.15.78:0/2680323968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd60074df0 0x7efd60073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.250+0000 7efd65200700 1 --2- 172.21.15.78:0/2680323968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd60073810 0x7efd60073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.250+0000 7efd65200700 1 -- 172.21.15.78:0/2680323968 >> 172.21.15.78:0/2680323968 conn(0x7efd600fc280 msgr2=0x7efd600fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:11.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.250+0000 7efd65200700 1 -- 172.21.15.78:0/2680323968 shutdown_connections 2024-03-14T09:54:11.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.250+0000 7efd65200700 1 -- 172.21.15.78:0/2680323968 wait complete. 2024-03-14T09:54:11.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.251+0000 7efd65200700 1 Processor -- start 2024-03-14T09:54:11.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.251+0000 7efd65200700 1 -- start start 2024-03-14T09:54:11.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.251+0000 7efd65200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd60073810 0x7efd6019d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:11.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.251+0000 7efd65200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd60074df0 0x7efd6019dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:11.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.251+0000 7efd65200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd6010d2c0 0x7efd601a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:11.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.252+0000 7efd65200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd601a52e0 con 0x7efd60073810 2024-03-14T09:54:11.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.252+0000 7efd65200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd601a5420 con 0x7efd6010d2c0 2024-03-14T09:54:11.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.252+0000 7efd65200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd601a55b0 con 0x7efd60074df0 2024-03-14T09:54:11.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.252+0000 7efd5e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd6010d2c0 0x7efd601a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:11.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.252+0000 7efd5d400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd60074df0 0x7efd6019dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:11.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.252+0000 7efd5e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd6010d2c0 0x7efd601a4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:44206/0 (socket says 172.21.15.78:44206) 2024-03-14T09:54:11.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.252+0000 7efd5d400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd60074df0 0x7efd6019dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:41044/0 (socket says 172.21.15.78:41044) 2024-03-14T09:54:11.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.252+0000 7efd5de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd60073810 0x7efd6019d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:11.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.252+0000 7efd5e800700 1 -- 172.21.15.78:0/2815263798 learned_addr learned my addr 172.21.15.78:0/2815263798 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:11.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.252+0000 7efd5e800700 1 -- 172.21.15.78:0/2815263798 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd60074df0 msgr2=0x7efd6019dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:11.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.252+0000 7efd5e800700 1 --2- 172.21.15.78:0/2815263798 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd60074df0 0x7efd6019dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.252+0000 7efd5e800700 1 -- 172.21.15.78:0/2815263798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd60073810 msgr2=0x7efd6019d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:11.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.252+0000 7efd5e800700 1 --2- 172.21.15.78:0/2815263798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd60073810 0x7efd6019d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.252+0000 7efd5e800700 1 -- 172.21.15.78:0/2815263798 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd58009d00 con 0x7efd6010d2c0 2024-03-14T09:54:11.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.252+0000 7efd5e800700 1 --2- 172.21.15.78:0/2815263798 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd6010d2c0 0x7efd601a4d10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efd58007a60 tx=0x7efd58007930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:11.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.253+0000 7efd56a00700 1 -- 172.21.15.78:0/2815263798 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5800e040 con 0x7efd6010d2c0 2024-03-14T09:54:11.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.253+0000 7efd56a00700 1 -- 172.21.15.78:0/2815263798 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd58004c50 con 0x7efd6010d2c0 2024-03-14T09:54:11.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.253+0000 7efd65200700 1 -- 172.21.15.78:0/2815263798 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd601a5830 con 0x7efd6010d2c0 2024-03-14T09:54:11.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.253+0000 7efd56a00700 1 -- 172.21.15.78:0/2815263798 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5801b6c0 con 0x7efd6010d2c0 2024-03-14T09:54:11.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.253+0000 7efd65200700 1 -- 172.21.15.78:0/2815263798 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd601a5c00 con 0x7efd6010d2c0 2024-03-14T09:54:11.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.254+0000 7efd65200700 1 -- 172.21.15.78:0/2815263798 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd60103520 con 0x7efd6010d2c0 2024-03-14T09:54:11.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.256+0000 7efd56a00700 1 -- 172.21.15.78:0/2815263798 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 23) v1 ==== 135200+0+0 (secure 0 0 0) 0x7efd5801d470 con 0x7efd6010d2c0 2024-03-14T09:54:11.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.257+0000 7efd56a00700 1 --2- 172.21.15.78:0/2815263798 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efd440a0890 0x7efd440a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:11.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.257+0000 7efd56a00700 1 -- 172.21.15.78:0/2815263798 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7efd580cd760 con 0x7efd6010d2c0 2024-03-14T09:54:11.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.257+0000 7efd5de00700 1 --2- 172.21.15.78:0/2815263798 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efd440a0890 0x7efd440a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:11.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.258+0000 7efd5de00700 1 --2- 172.21.15.78:0/2815263798 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efd440a0890 0x7efd440a2d50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efd50009c60 tx=0x7efd50009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:11.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.262+0000 7efd56a00700 1 -- 172.21.15.78:0/2815263798 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7efd58097000 con 0x7efd6010d2c0 2024-03-14T09:54:11.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.521+0000 7efd65200700 1 -- 172.21.15.78:0/2815263798 --> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7efd6002cdc0 con 0x7efd440a0890 2024-03-14T09:54:11.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.600+0000 7efd56a00700 1 -- 172.21.15.78:0/2815263798 <== mgr.15546 v2:172.21.15.78:6800/2192201293 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7efd6002cdc0 con 0x7efd440a0890 2024-03-14T09:54:11.601 INFO:teuthology.orchestra.run.smithi078.stdout:Retention added to path / 2024-03-14T09:54:11.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.603+0000 7efd65200700 1 -- 172.21.15.78:0/2815263798 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efd440a0890 msgr2=0x7efd440a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.603+0000 7efd65200700 1 --2- 172.21.15.78:0/2815263798 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efd440a0890 0x7efd440a2d50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efd50009c60 tx=0x7efd50009260 comp rx=0 tx=0).stop 2024-03-14T09:54:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.603+0000 7efd65200700 1 -- 172.21.15.78:0/2815263798 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd6010d2c0 msgr2=0x7efd601a4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.603+0000 7efd65200700 1 --2- 172.21.15.78:0/2815263798 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd6010d2c0 0x7efd601a4d10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efd58007a60 tx=0x7efd58007930 comp rx=0 tx=0).stop 2024-03-14T09:54:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.603+0000 7efd65200700 1 -- 172.21.15.78:0/2815263798 shutdown_connections 2024-03-14T09:54:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.603+0000 7efd65200700 1 --2- 172.21.15.78:0/2815263798 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efd440a0890 0x7efd440a2d50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.603+0000 7efd65200700 1 --2- 172.21.15.78:0/2815263798 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd6010d2c0 0x7efd601a4d10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.603+0000 7efd65200700 1 --2- 172.21.15.78:0/2815263798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd60073810 0x7efd6019d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.603+0000 7efd65200700 1 --2- 172.21.15.78:0/2815263798 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efd60074df0 0x7efd6019dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.603+0000 7efd65200700 1 -- 172.21.15.78:0/2815263798 >> 172.21.15.78:0/2815263798 conn(0x7efd600fc280 msgr2=0x7efd60100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.603+0000 7efd65200700 1 -- 172.21.15.78:0/2815263798 shutdown_connections 2024-03-14T09:54:11.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.603+0000 7efd65200700 1 -- 172.21.15.78:0/2815263798 wait complete. 2024-03-14T09:54:11.626 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-14T09:54:11.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.855+0000 7f6ae4200700 1 -- 172.21.15.78:0/4223242123 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6adc104080 msgr2=0x7f6adc106470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:11.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.855+0000 7f6ae4200700 1 --2- 172.21.15.78:0/4223242123 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6adc104080 0x7f6adc106470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8009990 tx=0x7f6ac8009ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.855+0000 7f6ae4200700 1 -- 172.21.15.78:0/4223242123 shutdown_connections 2024-03-14T09:54:11.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.855+0000 7f6ae4200700 1 --2- 172.21.15.78:0/4223242123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6adc101750 0x7f6adc103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.855+0000 7f6ae4200700 1 --2- 172.21.15.78:0/4223242123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6adc1069b0 0x7f6adc108e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.855+0000 7f6ae4200700 1 --2- 172.21.15.78:0/4223242123 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6adc104080 0x7f6adc106470 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.855+0000 7f6ae4200700 1 -- 172.21.15.78:0/4223242123 >> 172.21.15.78:0/4223242123 conn(0x7f6adc0fb160 msgr2=0x7f6adc0fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:11.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.855+0000 7f6ae4200700 1 -- 172.21.15.78:0/4223242123 shutdown_connections 2024-03-14T09:54:11.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.855+0000 7f6ae4200700 1 -- 172.21.15.78:0/4223242123 wait complete. 2024-03-14T09:54:11.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.856+0000 7f6ae4200700 1 Processor -- start 2024-03-14T09:54:11.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.856+0000 7f6ae4200700 1 -- start start 2024-03-14T09:54:11.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae4200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6adc101750 0x7f6adc1992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:11.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae4200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6adc104080 0x7f6adc1997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:11.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae4200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6adc1069b0 0x7f6adc1a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:11.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae4200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc1a0dd0 con 0x7f6adc104080 2024-03-14T09:54:11.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae4200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc1a0f10 con 0x7f6adc1069b0 2024-03-14T09:54:11.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae4200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc1a10a0 con 0x7f6adc101750 2024-03-14T09:54:11.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae1800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6adc101750 0x7f6adc1992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:11.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae0e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6adc104080 0x7f6adc1997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:11.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae1800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6adc101750 0x7f6adc1992a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:41070/0 (socket says 172.21.15.78:41070) 2024-03-14T09:54:11.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae1800700 1 -- 172.21.15.78:0/729920745 learned_addr learned my addr 172.21.15.78:0/729920745 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:11.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae2200700 1 --2- 172.21.15.78:0/729920745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6adc1069b0 0x7f6adc1a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:11.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae0e00700 1 -- 172.21.15.78:0/729920745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6adc101750 msgr2=0x7f6adc1992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:11.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae0e00700 1 --2- 172.21.15.78:0/729920745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6adc101750 0x7f6adc1992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae0e00700 1 -- 172.21.15.78:0/729920745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6adc1069b0 msgr2=0x7f6adc1a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:11.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae0e00700 1 --2- 172.21.15.78:0/729920745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6adc1069b0 0x7f6adc1a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:11.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae0e00700 1 -- 172.21.15.78:0/729920745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad4009c40 con 0x7f6adc104080 2024-03-14T09:54:11.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae1800700 1 --2- 172.21.15.78:0/729920745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6adc101750 0x7f6adc1992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:11.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.857+0000 7f6ae2200700 1 --2- 172.21.15.78:0/729920745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6adc1069b0 0x7f6adc1a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:11.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.858+0000 7f6ae0e00700 1 --2- 172.21.15.78:0/729920745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6adc104080 0x7f6adc1997e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800b680 tx=0x7f6ac8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:11.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.858+0000 7f6ada000700 1 -- 172.21.15.78:0/729920745 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac801d070 con 0x7f6adc104080 2024-03-14T09:54:11.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.858+0000 7f6ae4200700 1 -- 172.21.15.78:0/729920745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac80095f0 con 0x7f6adc104080 2024-03-14T09:54:11.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.858+0000 7f6ada000700 1 -- 172.21.15.78:0/729920745 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ac8021410 con 0x7f6adc104080 2024-03-14T09:54:11.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.858+0000 7f6ada000700 1 -- 172.21.15.78:0/729920745 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8017650 con 0x7f6adc104080 2024-03-14T09:54:11.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.858+0000 7f6ae4200700 1 -- 172.21.15.78:0/729920745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6adc1a1610 con 0x7f6adc104080 2024-03-14T09:54:11.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.860+0000 7f6ae4200700 1 -- 172.21.15.78:0/729920745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6adc108580 con 0x7f6adc104080 2024-03-14T09:54:11.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.861+0000 7f6ada000700 1 -- 172.21.15.78:0/729920745 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 23) v1 ==== 135200+0+0 (secure 0 0 0) 0x7f6ac8021580 con 0x7f6adc104080 2024-03-14T09:54:11.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.862+0000 7f6ada000700 1 --2- 172.21.15.78:0/729920745 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6ac40a0890 0x7f6ac40a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:11.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.863+0000 7f6ada000700 1 -- 172.21.15.78:0/729920745 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f6ac80cd420 con 0x7f6adc104080 2024-03-14T09:54:11.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.863+0000 7f6ae1800700 1 --2- 172.21.15.78:0/729920745 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6ac40a0890 0x7f6ac40a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:11.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.863+0000 7f6ae1800700 1 --2- 172.21.15.78:0/729920745 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6ac40a0890 0x7f6ac40a2d50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0011550 tx=0x7f6ad0009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:11.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:11.871+0000 7f6ada000700 1 -- 172.21.15.78:0/729920745 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6ac8096cc0 con 0x7f6adc104080 2024-03-14T09:54:12.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.164+0000 7f6ae4200700 1 -- 172.21.15.78:0/729920745 --> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f6adc0611f0 con 0x7f6ac40a0890 2024-03-14T09:54:12.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.172+0000 7f6ada000700 1 -- 172.21.15.78:0/729920745 <== mgr.15546 v2:172.21.15.78:6800/2192201293 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7f6adc0611f0 con 0x7f6ac40a0890 2024-03-14T09:54:12.173 INFO:teuthology.orchestra.run.smithi078.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-14T00:00:00", "created": "2024-03-14T09:54:10", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-14T09:54:12.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.174+0000 7f6ae4200700 1 -- 172.21.15.78:0/729920745 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6ac40a0890 msgr2=0x7f6ac40a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:12.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.175+0000 7f6ae4200700 1 --2- 172.21.15.78:0/729920745 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6ac40a0890 0x7f6ac40a2d50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0011550 tx=0x7f6ad0009260 comp rx=0 tx=0).stop 2024-03-14T09:54:12.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.175+0000 7f6ae4200700 1 -- 172.21.15.78:0/729920745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6adc104080 msgr2=0x7f6adc1997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:12.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.175+0000 7f6ae4200700 1 --2- 172.21.15.78:0/729920745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6adc104080 0x7f6adc1997e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800b680 tx=0x7f6ac8004730 comp rx=0 tx=0).stop 2024-03-14T09:54:12.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.175+0000 7f6ae4200700 1 -- 172.21.15.78:0/729920745 shutdown_connections 2024-03-14T09:54:12.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.175+0000 7f6ae4200700 1 --2- 172.21.15.78:0/729920745 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6ac40a0890 0x7f6ac40a2d50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:12.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.175+0000 7f6ae4200700 1 --2- 172.21.15.78:0/729920745 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6adc1069b0 0x7f6adc1a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:12.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.175+0000 7f6ae4200700 1 --2- 172.21.15.78:0/729920745 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6adc101750 0x7f6adc1992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:12.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.175+0000 7f6ae4200700 1 --2- 172.21.15.78:0/729920745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6adc104080 0x7f6adc1997e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:12.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.175+0000 7f6ae4200700 1 -- 172.21.15.78:0/729920745 >> 172.21.15.78:0/729920745 conn(0x7f6adc0fb160 msgr2=0x7f6adc100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:12.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.175+0000 7f6ae4200700 1 -- 172.21.15.78:0/729920745 shutdown_connections 2024-03-14T09:54:12.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.175+0000 7f6ae4200700 1 -- 172.21.15.78:0/729920745 wait complete. 2024-03-14T09:54:12.194 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-14T09:54:12.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:11 smithi152 ceph-mon[48279]: [14/Mar/2024:09:54:10] ENGINE Bus STARTING 2024-03-14T09:54:12.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:11 smithi152 ceph-mon[48279]: [14/Mar/2024:09:54:10] ENGINE Serving on http://172.21.15.78:8765 2024-03-14T09:54:12.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:11 smithi152 ceph-mon[48279]: [14/Mar/2024:09:54:10] ENGINE Serving on https://172.21.15.78:7150 2024-03-14T09:54:12.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:11 smithi152 ceph-mon[48279]: [14/Mar/2024:09:54:10] ENGINE Bus STARTED 2024-03-14T09:54:12.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:11 smithi152 ceph-mon[48279]: mgrmap e23: x(active, since 1.02261s), standbys: z, y 2024-03-14T09:54:12.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:11 smithi152 ceph-mon[48279]: from='client.25571 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:54:12.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:11 smithi152 ceph-mon[48279]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:54:12.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:11 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:12.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:11 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:12.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:11 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:12.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:11 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:12.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:11 smithi078 ceph-mon[38143]: [14/Mar/2024:09:54:10] ENGINE Bus STARTING 2024-03-14T09:54:12.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:11 smithi078 ceph-mon[38143]: [14/Mar/2024:09:54:10] ENGINE Serving on http://172.21.15.78:8765 2024-03-14T09:54:12.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:11 smithi078 ceph-mon[38143]: [14/Mar/2024:09:54:10] ENGINE Serving on https://172.21.15.78:7150 2024-03-14T09:54:12.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:11 smithi078 ceph-mon[38143]: [14/Mar/2024:09:54:10] ENGINE Bus STARTED 2024-03-14T09:54:12.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:11 smithi078 ceph-mon[38143]: mgrmap e23: x(active, since 1.02261s), standbys: z, y 2024-03-14T09:54:12.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:11 smithi078 ceph-mon[38143]: from='client.25571 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:54:12.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:11 smithi078 ceph-mon[38143]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:54:12.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:11 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:12.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:11 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:12.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:11 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:12.253 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:11 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:12.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:11 smithi132 ceph-mon[47959]: [14/Mar/2024:09:54:10] ENGINE Bus STARTING 2024-03-14T09:54:12.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:11 smithi132 ceph-mon[47959]: [14/Mar/2024:09:54:10] ENGINE Serving on http://172.21.15.78:8765 2024-03-14T09:54:12.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:11 smithi132 ceph-mon[47959]: [14/Mar/2024:09:54:10] ENGINE Serving on https://172.21.15.78:7150 2024-03-14T09:54:12.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:11 smithi132 ceph-mon[47959]: [14/Mar/2024:09:54:10] ENGINE Bus STARTED 2024-03-14T09:54:12.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:11 smithi132 ceph-mon[47959]: mgrmap e23: x(active, since 1.02261s), standbys: z, y 2024-03-14T09:54:12.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:11 smithi132 ceph-mon[47959]: from='client.25571 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:54:12.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:11 smithi132 ceph-mon[47959]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:54:12.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:11 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:12.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:11 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:12.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:11 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:12.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:11 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.396+0000 7fe1dae00700 1 -- 172.21.15.78:0/3495107289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1d4100cf0 msgr2=0x7fe1d4101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.396+0000 7fe1dae00700 1 --2- 172.21.15.78:0/3495107289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1d4100cf0 0x7fe1d4101110 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe1c40099c0 tx=0x7fe1c4009cd0 comp rx=0 tx=0).stop 2024-03-14T09:54:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.396+0000 7fe1dae00700 1 -- 172.21.15.78:0/3495107289 shutdown_connections 2024-03-14T09:54:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.396+0000 7fe1dae00700 1 --2- 172.21.15.78:0/3495107289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1d4100cf0 0x7fe1d4101110 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.396+0000 7fe1dae00700 1 --2- 172.21.15.78:0/3495107289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1d4102880 0x7fe1d4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.396+0000 7fe1dae00700 1 --2- 172.21.15.78:0/3495107289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1d4101ee0 0x7fe1d4102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.396+0000 7fe1dae00700 1 -- 172.21.15.78:0/3495107289 >> 172.21.15.78:0/3495107289 conn(0x7fe1d40fc280 msgr2=0x7fe1d40fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.396+0000 7fe1dae00700 1 -- 172.21.15.78:0/3495107289 shutdown_connections 2024-03-14T09:54:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.396+0000 7fe1dae00700 1 -- 172.21.15.78:0/3495107289 wait complete. 2024-03-14T09:54:12.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.397+0000 7fe1dae00700 1 Processor -- start 2024-03-14T09:54:12.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.397+0000 7fe1dae00700 1 -- start start 2024-03-14T09:54:12.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.397+0000 7fe1dae00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1d4100cf0 0x7fe1d41992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:12.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.397+0000 7fe1dae00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1d4101ee0 0x7fe1d41997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:12.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.397+0000 7fe1dae00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1d4102880 0x7fe1d41a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:12.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.397+0000 7fe1dae00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d41a0de0 con 0x7fe1d4100cf0 2024-03-14T09:54:12.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.397+0000 7fe1dae00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d41a0f20 con 0x7fe1d4102880 2024-03-14T09:54:12.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.397+0000 7fe1dae00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d41a10b0 con 0x7fe1d4101ee0 2024-03-14T09:54:12.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.397+0000 7fe1d3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1d4100cf0 0x7fe1d41992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:12.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.398+0000 7fe1d3400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1d4101ee0 0x7fe1d41997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:12.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.398+0000 7fe1d8e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1d4102880 0x7fe1d41a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:12.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.398+0000 7fe1d3400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1d4101ee0 0x7fe1d41997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:41112/0 (socket says 172.21.15.78:41112) 2024-03-14T09:54:12.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.398+0000 7fe1d3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1d4100cf0 0x7fe1d41992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58032/0 (socket says 172.21.15.78:58032) 2024-03-14T09:54:12.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.398+0000 7fe1d3400700 1 -- 172.21.15.78:0/2552569564 learned_addr learned my addr 172.21.15.78:0/2552569564 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:12.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.398+0000 7fe1d8e00700 1 -- 172.21.15.78:0/2552569564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1d4101ee0 msgr2=0x7fe1d41997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:12.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.398+0000 7fe1d8e00700 1 --2- 172.21.15.78:0/2552569564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1d4101ee0 0x7fe1d41997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:12.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.398+0000 7fe1d8e00700 1 -- 172.21.15.78:0/2552569564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1d4100cf0 msgr2=0x7fe1d41992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:12.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.398+0000 7fe1d8e00700 1 --2- 172.21.15.78:0/2552569564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1d4100cf0 0x7fe1d41992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:12.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.398+0000 7fe1d8e00700 1 -- 172.21.15.78:0/2552569564 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c40095f0 con 0x7fe1d4102880 2024-03-14T09:54:12.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.398+0000 7fe1d3400700 1 --2- 172.21.15.78:0/2552569564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1d4101ee0 0x7fe1d41997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:12.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.398+0000 7fe1d3e00700 1 --2- 172.21.15.78:0/2552569564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1d4100cf0 0x7fe1d41992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:12.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.398+0000 7fe1d8e00700 1 --2- 172.21.15.78:0/2552569564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1d4102880 0x7fe1d41a0810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc00efc0 tx=0x7fe1cc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:12.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.398+0000 7fe1d0c00700 1 -- 172.21.15.78:0/2552569564 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc018070 con 0x7fe1d4102880 2024-03-14T09:54:12.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.399+0000 7fe1d0c00700 1 -- 172.21.15.78:0/2552569564 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe1cc00f040 con 0x7fe1d4102880 2024-03-14T09:54:12.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.399+0000 7fe1dae00700 1 -- 172.21.15.78:0/2552569564 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d41a1330 con 0x7fe1d4102880 2024-03-14T09:54:12.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.399+0000 7fe1d0c00700 1 -- 172.21.15.78:0/2552569564 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc00cd40 con 0x7fe1d4102880 2024-03-14T09:54:12.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.399+0000 7fe1dae00700 1 -- 172.21.15.78:0/2552569564 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d41a1890 con 0x7fe1d4102880 2024-03-14T09:54:12.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.400+0000 7fe1dae00700 1 -- 172.21.15.78:0/2552569564 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d4067c00 con 0x7fe1d4102880 2024-03-14T09:54:12.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.401+0000 7fe1d0c00700 1 -- 172.21.15.78:0/2552569564 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 23) v1 ==== 135200+0+0 (secure 0 0 0) 0x7fe1cc003a00 con 0x7fe1d4102880 2024-03-14T09:54:12.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.403+0000 7fe1d0c00700 1 --2- 172.21.15.78:0/2552569564 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe1b80a0910 0x7fe1b80a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:12.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.403+0000 7fe1d0c00700 1 -- 172.21.15.78:0/2552569564 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe1cc0cbbc0 con 0x7fe1d4102880 2024-03-14T09:54:12.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.403+0000 7fe1d3e00700 1 --2- 172.21.15.78:0/2552569564 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe1b80a0910 0x7fe1b80a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.404+0000 7fe1d3e00700 1 --2- 172.21.15.78:0/2552569564 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe1b80a0910 0x7fe1b80a2dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4009990 tx=0x7fe1c401a040 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:12.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.407+0000 7fe1d0c00700 1 -- 172.21.15.78:0/2552569564 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe1cc095480 con 0x7fe1d4102880 2024-03-14T09:54:12.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.622+0000 7fe1d0c00700 1 -- 172.21.15.78:0/2552569564 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 24) v1 ==== 135508+0+0 (secure 0 0 0) 0x7fe1cc003cb0 con 0x7fe1d4102880 2024-03-14T09:54:12.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.664+0000 7fe1dae00700 1 -- 172.21.15.78:0/2552569564 --> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fe1d4074700 con 0x7fe1b80a0910 2024-03-14T09:54:12.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.671+0000 7fe1d0c00700 1 -- 172.21.15.78:0/2552569564 <== mgr.15546 v2:172.21.15.78:6800/2192201293 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fe1d4074700 con 0x7fe1b80a0910 2024-03-14T09:54:12.672 INFO:teuthology.orchestra.run.smithi078.stdout:/ 1m 6m3h 2024-03-14T09:54:12.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.674+0000 7fe1dae00700 1 -- 172.21.15.78:0/2552569564 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe1b80a0910 msgr2=0x7fe1b80a2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.674+0000 7fe1dae00700 1 --2- 172.21.15.78:0/2552569564 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe1b80a0910 0x7fe1b80a2dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe1c4009990 tx=0x7fe1c401a040 comp rx=0 tx=0).stop 2024-03-14T09:54:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.674+0000 7fe1dae00700 1 -- 172.21.15.78:0/2552569564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1d4102880 msgr2=0x7fe1d41a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.674+0000 7fe1dae00700 1 --2- 172.21.15.78:0/2552569564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1d4102880 0x7fe1d41a0810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc00efc0 tx=0x7fe1cc00c3a0 comp rx=0 tx=0).stop 2024-03-14T09:54:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.674+0000 7fe1dae00700 1 -- 172.21.15.78:0/2552569564 shutdown_connections 2024-03-14T09:54:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.674+0000 7fe1dae00700 1 --2- 172.21.15.78:0/2552569564 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe1b80a0910 0x7fe1b80a2dd0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.674+0000 7fe1dae00700 1 --2- 172.21.15.78:0/2552569564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe1d4102880 0x7fe1d41a0810 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.674+0000 7fe1dae00700 1 --2- 172.21.15.78:0/2552569564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1d4100cf0 0x7fe1d41992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.674+0000 7fe1dae00700 1 --2- 172.21.15.78:0/2552569564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe1d4101ee0 0x7fe1d41997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.674+0000 7fe1dae00700 1 -- 172.21.15.78:0/2552569564 >> 172.21.15.78:0/2552569564 conn(0x7fe1d40fc280 msgr2=0x7fe1d4072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.674+0000 7fe1dae00700 1 -- 172.21.15.78:0/2552569564 shutdown_connections 2024-03-14T09:54:12.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:12.674+0000 7fe1dae00700 1 -- 172.21.15.78:0/2552569564 wait complete. 2024-03-14T09:54:12.693 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-14T09:54:12.769 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-14T09:54:12.784 INFO:teuthology.run_tasks:Running task workunit... 2024-03-14T09:54:12.798 INFO:tasks.workunit:Pulling workunits from ref 2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T09:54:12.798 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-14T09:54:12.798 INFO:tasks.workunit:timeout=3h 2024-03-14T09:54:12.798 INFO:tasks.workunit:cleanup=True 2024-03-14T09:54:12.799 DEBUG:teuthology.orchestra.run.smithi078:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-14T09:54:12.842 INFO:teuthology.orchestra.run.smithi078.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-14T09:54:12.842 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-14T09:54:12.842 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 7ah/122d Inode: 1099511628283 Links: 2 2024-03-14T09:54:12.842 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T09:54:12.842 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:cephfs_t:s0 2024-03-14T09:54:12.842 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-03-14 09:52:32.646122979 +0000 2024-03-14T09:54:12.843 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-03-14 09:52:32.646122979 +0000 2024-03-14T09:54:12.843 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-03-14 09:53:48.378715322 +0000 2024-03-14T09:54:12.843 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-03-14 09:52:32.646122979 +0000 2024-03-14T09:54:12.843 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-14T09:54:12.843 DEBUG:teuthology.orchestra.run.smithi078:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-14T09:54:12.961 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T09:54:13.007 INFO:tasks.workunit.client.0.smithi078.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-14T09:54:13.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: from='client.25604 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:54:13.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:54:13.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi132 to 182.5M 2024-03-14T09:54:13.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: Unable to set osd_memory_target on smithi132 to 191379148: error parsing value: Value '191379148' is below minimum 939524096 2024-03-14T09:54:13.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:54:13.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi152 to 182.5M 2024-03-14T09:54:13.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: Unable to set osd_memory_target on smithi152 to 191374848: error parsing value: Value '191374848' is below minimum 939524096 2024-03-14T09:54:13.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:13 smithi078 ceph-mon[38143]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-03-14T09:54:13.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: from='client.25604 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:54:13.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:54:13.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi132 to 182.5M 2024-03-14T09:54:13.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: Unable to set osd_memory_target on smithi132 to 191379148: error parsing value: Value '191379148' is below minimum 939524096 2024-03-14T09:54:13.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:54:13.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi152 to 182.5M 2024-03-14T09:54:13.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: Unable to set osd_memory_target on smithi152 to 191374848: error parsing value: Value '191374848' is below minimum 939524096 2024-03-14T09:54:13.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:13 smithi152 ceph-mon[48279]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-03-14T09:54:13.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: from='client.25604 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:54:13.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:54:13.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi132 to 182.5M 2024-03-14T09:54:13.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: Unable to set osd_memory_target on smithi132 to 191379148: error parsing value: Value '191379148' is below minimum 939524096 2024-03-14T09:54:13.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: from='client.15603 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:54:13.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:13.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:13.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi152 to 182.5M 2024-03-14T09:54:13.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: Unable to set osd_memory_target on smithi152 to 191374848: error parsing value: Value '191374848' is below minimum 939524096 2024-03-14T09:54:13.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:13 smithi132 ceph-mon[47959]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-03-14T09:54:14.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:14 smithi152 ceph-mon[48279]: from='client.25622 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:54:14.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:14 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:14.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:14 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:14.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:14 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:14.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:14 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:14.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:14 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:14.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:14 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:14.502 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:14 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:54:14.502 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:14 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:54:14.502 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:14 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T09:54:14.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:14 smithi132 ceph-mon[47959]: from='client.25622 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:54:14.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:14 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:14.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:14 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:14.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:14 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:14.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:14 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:14.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:14 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:14.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:14 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:14.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:14 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:54:14.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:14 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:54:14.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:14 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T09:54:14.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:14 smithi078 ceph-mon[38143]: from='client.25622 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T09:54:14.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:14 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:14.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:14 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:14.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:14 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:14.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:14 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:14.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:14 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:14.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:14 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-14T09:54:14.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:14 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:54:14.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:14 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:54:14.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:14 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T09:54:15.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi078 to 221.5M 2024-03-14T09:54:15.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: Unable to set osd_memory_target on smithi078 to 232331366: error parsing value: Value '232331366' is below minimum 939524096 2024-03-14T09:54:15.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: Updating smithi078:/etc/ceph/ceph.conf 2024-03-14T09:54:15.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: Updating smithi132:/etc/ceph/ceph.conf 2024-03-14T09:54:15.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-14T09:54:15.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: Updating smithi132:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:54:15.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: Updating smithi152:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:54:15.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:54:15.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: Updating smithi078:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:54:15.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:54:15.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:54:15.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: Updating smithi132:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T09:54:15.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: Updating smithi152:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T09:54:15.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-03-14T09:54:15.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:15.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:15.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:15.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:15 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:15.523 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi078 to 221.5M 2024-03-14T09:54:15.523 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: Unable to set osd_memory_target on smithi078 to 232331366: error parsing value: Value '232331366' is below minimum 939524096 2024-03-14T09:54:15.523 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: Updating smithi078:/etc/ceph/ceph.conf 2024-03-14T09:54:15.523 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: Updating smithi132:/etc/ceph/ceph.conf 2024-03-14T09:54:15.523 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-14T09:54:15.523 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: Updating smithi132:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:54:15.523 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: Updating smithi152:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:54:15.523 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:54:15.523 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: Updating smithi078:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:54:15.523 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:54:15.523 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:54:15.523 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: Updating smithi132:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T09:54:15.524 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: Updating smithi152:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T09:54:15.524 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-03-14T09:54:15.524 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:15.524 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:15.524 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:15.524 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:15 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:15.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi078 to 221.5M 2024-03-14T09:54:15.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: Unable to set osd_memory_target on smithi078 to 232331366: error parsing value: Value '232331366' is below minimum 939524096 2024-03-14T09:54:15.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: Updating smithi078:/etc/ceph/ceph.conf 2024-03-14T09:54:15.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: Updating smithi132:/etc/ceph/ceph.conf 2024-03-14T09:54:15.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: Updating smithi152:/etc/ceph/ceph.conf 2024-03-14T09:54:15.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: Updating smithi132:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:54:15.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: Updating smithi152:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:54:15.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:54:15.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: Updating smithi078:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T09:54:15.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:54:15.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: Updating smithi152:/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:54:15.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: Updating smithi132:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T09:54:15.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: Updating smithi152:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T09:54:15.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-03-14T09:54:15.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:15.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:15.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:15.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:15 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:16.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:16 smithi152 ceph-mon[48279]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:54:16.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:16 smithi152 ceph-mon[48279]: Updating smithi078:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T09:54:16.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:16 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:16.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:16 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:16.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:16 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:16.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:16 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:54:16.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:16 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:54:16.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:16 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:54:16.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:16 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:16.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:16 smithi132 ceph-mon[47959]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:54:16.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:16 smithi132 ceph-mon[47959]: Updating smithi078:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T09:54:16.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:16 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:16.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:16 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:16.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:16 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:16.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:16 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:54:16.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:16 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:54:16.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:16 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:54:16.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:16 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:16.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:16 smithi078 ceph-mon[38143]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2024-03-14T09:54:16.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:16 smithi078 ceph-mon[38143]: Updating smithi078:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T09:54:16.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:16 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:16.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:16 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:16.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:16 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:16.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:16 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:54:16.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:16 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:54:16.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:16 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:54:16.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:16 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:54:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:17 smithi152 ceph-mon[48279]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:54:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:17 smithi132 ceph-mon[47959]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:54:17.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:17 smithi078 ceph-mon[38143]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-14T09:54:18.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:18 smithi152 ceph-mon[48279]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 132 KiB/s wr, 75 op/s 2024-03-14T09:54:19.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:18 smithi132 ceph-mon[47959]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 132 KiB/s wr, 75 op/s 2024-03-14T09:54:19.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:18 smithi078 ceph-mon[38143]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 132 KiB/s wr, 75 op/s 2024-03-14T09:54:21.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:21 smithi132 ceph-mon[47959]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 58 op/s 2024-03-14T09:54:21.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:21 smithi078 ceph-mon[38143]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 58 op/s 2024-03-14T09:54:21.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:21 smithi152 ceph-mon[48279]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 58 op/s 2024-03-14T09:54:23.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:23 smithi132 ceph-mon[47959]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 50 op/s 2024-03-14T09:54:23.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:23 smithi078 ceph-mon[38143]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 50 op/s 2024-03-14T09:54:23.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:23 smithi152 ceph-mon[48279]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 50 op/s 2024-03-14T09:54:25.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:25 smithi132 ceph-mon[47959]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-03-14T09:54:25.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:25 smithi078 ceph-mon[38143]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-03-14T09:54:25.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:25 smithi152 ceph-mon[48279]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-03-14T09:54:27.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:27 smithi132 ceph-mon[47959]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-03-14T09:54:27.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:27 smithi078 ceph-mon[38143]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-03-14T09:54:27.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:27 smithi152 ceph-mon[48279]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-03-14T09:54:28.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:28 smithi152 ceph-mon[48279]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-03-14T09:54:29.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:28 smithi132 ceph-mon[47959]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-03-14T09:54:29.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:28 smithi078 ceph-mon[38143]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-03-14T09:54:30.782 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:54:31.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.003+0000 7f8b07200700 1 -- 172.21.15.78:0/1914248455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b08101d40 msgr2=0x7f8b08102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.003+0000 7f8b07200700 1 --2- 172.21.15.78:0/1914248455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b08101d40 0x7f8b08102120 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8af8009990 tx=0x7f8af8009ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.003+0000 7f8b07200700 1 -- 172.21.15.78:0/1914248455 shutdown_connections 2024-03-14T09:54:31.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.003+0000 7f8b07200700 1 --2- 172.21.15.78:0/1914248455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b0810b110 0x7f8b0810d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.003+0000 7f8b07200700 1 --2- 172.21.15.78:0/1914248455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b08101d40 0x7f8b08102120 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.003+0000 7f8b07200700 1 --2- 172.21.15.78:0/1914248455 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b08102660 0x7f8b0810ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.003+0000 7f8b07200700 1 -- 172.21.15.78:0/1914248455 >> 172.21.15.78:0/1914248455 conn(0x7f8b080fb4e0 msgr2=0x7f8b080fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:31.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.004+0000 7f8b07200700 1 -- 172.21.15.78:0/1914248455 shutdown_connections 2024-03-14T09:54:31.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.004+0000 7f8b07200700 1 -- 172.21.15.78:0/1914248455 wait complete. 2024-03-14T09:54:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.005+0000 7f8b07200700 1 Processor -- start 2024-03-14T09:54:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.005+0000 7f8b07200700 1 -- start start 2024-03-14T09:54:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.005+0000 7f8b07200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b08101d40 0x7f8b080ff990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.005+0000 7f8b07200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b08102660 0x7f8b080ffed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.005+0000 7f8b07200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b0810b110 0x7f8b08100410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.005+0000 7f8b07200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b08101960 con 0x7f8b08102660 2024-03-14T09:54:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.005+0000 7f8b07200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b081a9dc0 con 0x7f8b0810b110 2024-03-14T09:54:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.005+0000 7f8b07200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b081a9f80 con 0x7f8b08101d40 2024-03-14T09:54:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.006+0000 7f8b05e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b08101d40 0x7f8b080ff990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.006+0000 7f8b05e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b08101d40 0x7f8b080ff990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37138/0 (socket says 172.21.15.78:37138) 2024-03-14T09:54:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.006+0000 7f8b05e00700 1 -- 172.21.15.78:0/2642314389 learned_addr learned my addr 172.21.15.78:0/2642314389 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.006+0000 7f8b05400700 1 --2- 172.21.15.78:0/2642314389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b08102660 0x7f8b080ffed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.006+0000 7f8b06800700 1 --2- 172.21.15.78:0/2642314389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b0810b110 0x7f8b08100410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.006+0000 7f8b05e00700 1 -- 172.21.15.78:0/2642314389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b0810b110 msgr2=0x7f8b08100410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.006+0000 7f8b05e00700 1 --2- 172.21.15.78:0/2642314389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b0810b110 0x7f8b08100410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.006+0000 7f8b05e00700 1 -- 172.21.15.78:0/2642314389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b08102660 msgr2=0x7f8b080ffed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.006+0000 7f8b05e00700 1 --2- 172.21.15.78:0/2642314389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b08102660 0x7f8b080ffed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.006+0000 7f8b05e00700 1 -- 172.21.15.78:0/2642314389 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af80095f0 con 0x7f8b08101d40 2024-03-14T09:54:31.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.006+0000 7f8b05e00700 1 --2- 172.21.15.78:0/2642314389 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b08101d40 0x7f8b080ff990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8af8005910 tx=0x7f8af8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:31.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.007+0000 7f8afea00700 1 -- 172.21.15.78:0/2642314389 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af801d070 con 0x7f8b08101d40 2024-03-14T09:54:31.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.007+0000 7f8afea00700 1 -- 172.21.15.78:0/2642314389 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8af8022470 con 0x7f8b08101d40 2024-03-14T09:54:31.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.007+0000 7f8b07200700 1 -- 172.21.15.78:0/2642314389 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b081aa200 con 0x7f8b08101d40 2024-03-14T09:54:31.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.007+0000 7f8afea00700 1 -- 172.21.15.78:0/2642314389 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af8017650 con 0x7f8b08101d40 2024-03-14T09:54:31.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.007+0000 7f8b07200700 1 -- 172.21.15.78:0/2642314389 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b081aa700 con 0x7f8b08101d40 2024-03-14T09:54:31.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.011+0000 7f8afea00700 1 -- 172.21.15.78:0/2642314389 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 25) v1 ==== 135564+0+0 (secure 0 0 0) 0x7f8af80225e0 con 0x7f8b08101d40 2024-03-14T09:54:31.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.013+0000 7f8afea00700 1 --2- 172.21.15.78:0/2642314389 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8aec0a0d10 0x7f8aec0a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.013+0000 7f8afea00700 1 -- 172.21.15.78:0/2642314389 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f8af80cdaf0 con 0x7f8b08101d40 2024-03-14T09:54:31.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.013+0000 7f8b05400700 1 --2- 172.21.15.78:0/2642314389 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8aec0a0d10 0x7f8aec0a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.013+0000 7f8b20694b80 1 Processor -- start 2024-03-14T09:54:31.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.014+0000 7f8b20694b80 1 -- start start 2024-03-14T09:54:31.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.014+0000 7f8b05400700 1 --2- 172.21.15.78:0/2642314389 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8aec0a0d10 0x7f8aec0a31d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8b081010c0 tx=0x7f8af4009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:31.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.014+0000 7f8b20694b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a3860 0x558cda2a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.014+0000 7f8b20694b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2a4180 0x558cda2aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.014+0000 7f8b20694b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2af030 0x558cda2b1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.014+0000 7f8b20694b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cda1fd990 con 0x558cda2af030 2024-03-14T09:54:31.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.014+0000 7f8b20694b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cda1fb350 con 0x558cda2a4180 2024-03-14T09:54:31.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.014+0000 7f8b20694b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cda1f6c40 con 0x558cda2a3860 2024-03-14T09:54:31.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.014+0000 7f8b05e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a3860 0x558cda2a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.014+0000 7f8b06800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2af030 0x558cda2b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.014+0000 7f8b05e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a3860 0x558cda2a3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37150/0 (socket says 172.21.15.78:37150) 2024-03-14T09:54:31.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.014+0000 7f8b05e00700 1 -- 172.21.15.78:0/3377812461 learned_addr learned my addr 172.21.15.78:0/3377812461 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:31.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.015+0000 7f8b05400700 1 --2- 172.21.15.78:0/3377812461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2a4180 0x558cda2aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.015+0000 7f8b05e00700 1 -- 172.21.15.78:0/3377812461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2a4180 msgr2=0x558cda2aeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:31.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.015+0000 7f8b05e00700 1 --2- 172.21.15.78:0/3377812461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2a4180 0x558cda2aeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.015+0000 7f8b05e00700 1 -- 172.21.15.78:0/3377812461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2af030 msgr2=0x558cda2b1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:31.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.015+0000 7f8b05e00700 1 --2- 172.21.15.78:0/3377812461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2af030 0x558cda2b1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.015+0000 7f8b05e00700 1 -- 172.21.15.78:0/3377812461 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558cda0783b0 con 0x558cda2a3860 2024-03-14T09:54:31.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.015+0000 7f8b05e00700 1 --2- 172.21.15.78:0/3377812461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a3860 0x558cda2a3c40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8af800f740 tx=0x7f8af8022f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5c3bdf714ce0c37 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:31.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.015+0000 7f8af3000700 1 -- 172.21.15.78:0/3377812461 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af809a650 con 0x558cda2a3860 2024-03-14T09:54:31.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.015+0000 7f8af3000700 1 -- 172.21.15.78:0/3377812461 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8af809a470 con 0x558cda2a3860 2024-03-14T09:54:31.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.015+0000 7f8af3000700 1 -- 172.21.15.78:0/3377812461 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af80b3210 con 0x558cda2a3860 2024-03-14T09:54:31.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.016+0000 7f8b20694b80 1 -- 172.21.15.78:0/3377812461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a3860 msgr2=0x558cda2a3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.016+0000 7f8b20694b80 1 --2- 172.21.15.78:0/3377812461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a3860 0x558cda2a3c40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8af800f740 tx=0x7f8af8022f90 comp rx=0 tx=0).stop 2024-03-14T09:54:31.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.016+0000 7f8b20694b80 1 -- 172.21.15.78:0/3377812461 shutdown_connections 2024-03-14T09:54:31.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.016+0000 7f8b20694b80 1 --2- 172.21.15.78:0/3377812461 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2af030 0x558cda2b1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.016+0000 7f8b20694b80 1 --2- 172.21.15.78:0/3377812461 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a3860 0x558cda2a3c40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.016+0000 7f8b20694b80 1 --2- 172.21.15.78:0/3377812461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2a4180 0x558cda2aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.016+0000 7f8b20694b80 1 -- 172.21.15.78:0/3377812461 >> 172.21.15.78:0/3377812461 conn(0x558cda21a220 msgr2=0x558cda29fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:31.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.016+0000 7f8b20694b80 1 -- 172.21.15.78:0/3377812461 shutdown_connections 2024-03-14T09:54:31.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.016+0000 7f8b20694b80 1 -- 172.21.15.78:0/3377812461 wait complete. 2024-03-14T09:54:31.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.017+0000 7f8b20694b80 1 -- start start 2024-03-14T09:54:31.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.017+0000 7f8b20694b80 1 Processor -- start 2024-03-14T09:54:31.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.018+0000 7f8b20694b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a3860 0x558cda2ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.018+0000 7f8b20694b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2a4180 0x558cda2abd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.018+0000 7f8b20694b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2af030 0x558cda2ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.018+0000 7f8b20694b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cda192d80 con 0x558cda2af030 2024-03-14T09:54:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.018+0000 7f8b20694b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cda192a20 con 0x558cda2a4180 2024-03-14T09:54:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.018+0000 7f8b20694b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cda07a6a0 con 0x558cda2a3860 2024-03-14T09:54:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.018+0000 7f8b20694b80 10 client.0 _ll_register_callbacks cb 0x558cd9ffe3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.018+0000 7f8b05e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a3860 0x558cda2ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.018+0000 7f8b05e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a3860 0x558cda2ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37162/0 (socket says 172.21.15.78:37162) 2024-03-14T09:54:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.018+0000 7f8b05e00700 1 -- 172.21.15.78:0/1854267380 learned_addr learned my addr 172.21.15.78:0/1854267380 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.018+0000 7f8b06800700 1 --2- 172.21.15.78:0/1854267380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2af030 0x558cda2ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.019+0000 7f8b05e00700 1 -- 172.21.15.78:0/1854267380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2a4180 msgr2=0x558cda2abd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.019+0000 7f8b05e00700 1 --2- 172.21.15.78:0/1854267380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2a4180 0x558cda2abd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.019+0000 7f8b05e00700 1 -- 172.21.15.78:0/1854267380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2af030 msgr2=0x558cda2ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.019+0000 7f8b05e00700 1 --2- 172.21.15.78:0/1854267380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2af030 0x558cda2ac2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.019+0000 7f8b05e00700 1 -- 172.21.15.78:0/1854267380 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b081aa700 con 0x558cda2a3860 2024-03-14T09:54:31.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.019+0000 7f8b05e00700 1 --2- 172.21.15.78:0/1854267380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a3860 0x558cda2ae9e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8af8000c00 tx=0x7f8af80b3560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34ef91b764d1bf24 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:31.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.019+0000 7f8ae7400700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:54:31.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.019+0000 7f8ae7400700 1 -- 172.21.15.78:0/1854267380 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af809e940 con 0x558cda2a3860 2024-03-14T09:54:31.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.019+0000 7f8ae7400700 1 -- 172.21.15.78:0/1854267380 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f8af809ab10 con 0x558cda2a3860 2024-03-14T09:54:31.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.019+0000 7f8b20694b80 1 -- 172.21.15.78:0/1854267380 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558cda297720 con 0x558cda2a3860 2024-03-14T09:54:31.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.019+0000 7f8ae7400700 1 -- 172.21.15.78:0/1854267380 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8af809b210 con 0x558cda2a3860 2024-03-14T09:54:31.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.019+0000 7f8b20694b80 10 client.25639 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:31.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.019+0000 7f8b20694b80 1 -- 172.21.15.78:0/1854267380 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558cda024b60 con 0x558cda2a3860 2024-03-14T09:54:31.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.019+0000 7f8ae7400700 1 -- 172.21.15.78:0/1854267380 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558cda297720 con 0x558cda2a3860 2024-03-14T09:54:31.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.020+0000 7f8ae7400700 1 -- 172.21.15.78:0/1854267380 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8af80a2a70 con 0x558cda2a3860 2024-03-14T09:54:31.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.020+0000 7f8b20694b80 10 client.25639 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:31.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.020+0000 7f8b20694b80 10 client.25639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:31.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.020+0000 7f8b20694b80 20 client.25639 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:31.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.020+0000 7f8b20694b80 1 --2- 172.21.15.78:0/1854267380 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558cda2be3c0 0x558cda2c07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.020+0000 7f8b20694b80 4 client.25639 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:54:31.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.020+0000 7f8b20694b80 1 -- 172.21.15.78:0/1854267380 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558cda1ef700 con 0x558cda2be3c0 2024-03-14T09:54:31.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.020+0000 7f8b05400700 1 --2- 172.21.15.78:0/1854267380 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558cda2be3c0 0x558cda2c07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.021+0000 7f8b05400700 1 --2- 172.21.15.78:0/1854267380 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558cda2be3c0 0x558cda2c07e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e42a65dda6691dd server_cookie=ac4d1168be323fa2 in_seq=0 out_seq=0 2024-03-14T09:54:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.021+0000 7f8ae7400700 10 client.25639 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.021+0000 7f8ae7400700 1 -- 172.21.15.78:0/1854267380 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x558cda1ef700 con 0x558cda2be3c0 2024-03-14T09:54:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.021+0000 7f8ae7400700 10 client.25639 handle_command_reply: tid=0 2024-03-14T09:54:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.021+0000 7f8b20694b80 1 client.25639 shutdown 2024-03-14T09:54:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.021+0000 7f8b20694b80 1 -- 172.21.15.78:0/1854267380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a3860 msgr2=0x558cda2ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.021+0000 7f8b20694b80 1 --2- 172.21.15.78:0/1854267380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a3860 0x558cda2ae9e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8af8000c00 tx=0x7f8af80b3560 comp rx=0 tx=0).stop 2024-03-14T09:54:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.022+0000 7f8b20694b80 1 -- 172.21.15.78:0/1854267380 shutdown_connections 2024-03-14T09:54:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.022+0000 7f8b20694b80 1 --2- 172.21.15.78:0/1854267380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2af030 0x558cda2ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.022+0000 7f8b20694b80 1 --2- 172.21.15.78:0/1854267380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a3860 0x558cda2ae9e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.022+0000 7f8b20694b80 1 --2- 172.21.15.78:0/1854267380 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558cda2be3c0 0x558cda2c07e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.022+0000 7f8b20694b80 1 --2- 172.21.15.78:0/1854267380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2a4180 0x558cda2abd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.022+0000 7f8ae7400700 0 client.25639 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:31.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.022+0000 7f8b20694b80 1 -- 172.21.15.78:0/1854267380 >> 172.21.15.78:0/1854267380 conn(0x558cda21a220 msgr2=0x558cda29f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:31.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.022+0000 7f8b20694b80 1 -- 172.21.15.78:0/1854267380 shutdown_connections 2024-03-14T09:54:31.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.022+0000 7f8b20694b80 1 -- 172.21.15.78:0/1854267380 wait complete. 2024-03-14T09:54:31.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.022+0000 7f8b20694b80 20 client.25639 trim_cache size 0 max 16384 2024-03-14T09:54:31.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.040+0000 7f8b20694b80 1 Processor -- start 2024-03-14T09:54:31.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.040+0000 7f8b20694b80 1 -- start start 2024-03-14T09:54:31.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.041+0000 7f8b20694b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a4180 0x558cda29fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.041+0000 7f8b20694b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2af030 0x558cda2b53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.041+0000 7f8b20694b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2ab5e0 0x558cda2ae510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.041+0000 7f8b20694b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cda2ac660 con 0x558cda2ab5e0 2024-03-14T09:54:31.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.041+0000 7f8b20694b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cda299d00 con 0x558cda2af030 2024-03-14T09:54:31.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.041+0000 7f8b20694b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cda2a4820 con 0x558cda2a4180 2024-03-14T09:54:31.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.041+0000 7f8b05e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a4180 0x558cda29fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.041+0000 7f8b05400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2af030 0x558cda2b53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.041+0000 7f8b06800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2ab5e0 0x558cda2ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.041+0000 7f8b05e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a4180 0x558cda29fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37164/0 (socket says 172.21.15.78:37164) 2024-03-14T09:54:31.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.041+0000 7f8b05400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2af030 0x558cda2b53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:39618/0 (socket says 172.21.15.78:39618) 2024-03-14T09:54:31.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.041+0000 7f8b05e00700 1 -- 172.21.15.78:0/596508515 learned_addr learned my addr 172.21.15.78:0/596508515 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:31.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.042+0000 7f8b05400700 1 -- 172.21.15.78:0/596508515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a4180 msgr2=0x558cda29fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:31.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.042+0000 7f8b05400700 1 --2- 172.21.15.78:0/596508515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a4180 0x558cda29fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.042+0000 7f8b05400700 1 -- 172.21.15.78:0/596508515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2ab5e0 msgr2=0x558cda2ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.042+0000 7f8b05400700 1 --2- 172.21.15.78:0/596508515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2ab5e0 0x558cda2ae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.042+0000 7f8b05400700 1 -- 172.21.15.78:0/596508515 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558cda1bcea0 con 0x558cda2af030 2024-03-14T09:54:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.042+0000 7f8b05e00700 1 --2- 172.21.15.78:0/596508515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a4180 0x558cda29fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.042+0000 7f8b06800700 1 --2- 172.21.15.78:0/596508515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2ab5e0 0x558cda2ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.042+0000 7f8b05400700 1 --2- 172.21.15.78:0/596508515 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2af030 0x558cda2b53f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x558cda2c10c0 tx=0x7f8af4013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2895a6104f3cdd8d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.042+0000 7f8af3a00700 1 -- 172.21.15.78:0/596508515 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af4022040 con 0x558cda2af030 2024-03-14T09:54:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.042+0000 7f8af3a00700 1 -- 172.21.15.78:0/596508515 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8af4026bd0 con 0x558cda2af030 2024-03-14T09:54:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.042+0000 7f8af3a00700 1 -- 172.21.15.78:0/596508515 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af40265b0 con 0x558cda2af030 2024-03-14T09:54:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.043+0000 7f8b20694b80 1 -- 172.21.15.78:0/596508515 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2af030 msgr2=0x558cda2b53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.043+0000 7f8b20694b80 1 --2- 172.21.15.78:0/596508515 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2af030 0x558cda2b53f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x558cda2c10c0 tx=0x7f8af4013040 comp rx=0 tx=0).stop 2024-03-14T09:54:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.043+0000 7f8b20694b80 1 -- 172.21.15.78:0/596508515 shutdown_connections 2024-03-14T09:54:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.043+0000 7f8b20694b80 1 --2- 172.21.15.78:0/596508515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2ab5e0 0x558cda2ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.043+0000 7f8b20694b80 1 --2- 172.21.15.78:0/596508515 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a4180 0x558cda29fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.043+0000 7f8b20694b80 1 --2- 172.21.15.78:0/596508515 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2af030 0x558cda2b53f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.043+0000 7f8b20694b80 1 -- 172.21.15.78:0/596508515 >> 172.21.15.78:0/596508515 conn(0x558cda21a220 msgr2=0x558cda29e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:31.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.043+0000 7f8b20694b80 1 -- 172.21.15.78:0/596508515 shutdown_connections 2024-03-14T09:54:31.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.043+0000 7f8b20694b80 1 -- 172.21.15.78:0/596508515 wait complete. 2024-03-14T09:54:31.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.044+0000 7f8b20694b80 1 -- start start 2024-03-14T09:54:31.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.044+0000 7f8b20694b80 1 Processor -- start 2024-03-14T09:54:31.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.044+0000 7f8b20694b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a4180 0x558cda2b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.044+0000 7f8b20694b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2ab5e0 0x558cda2b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.044+0000 7f8b20694b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2af030 0x558cda30fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.044+0000 7f8b20694b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cda2b2be0 con 0x558cda2af030 2024-03-14T09:54:31.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.045+0000 7f8b20694b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cda310420 con 0x558cda2ab5e0 2024-03-14T09:54:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.045+0000 7f8b20694b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cda310560 con 0x558cda2a4180 2024-03-14T09:54:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.045+0000 7f8b20694b80 10 client.0 _ll_register_callbacks cb 0x7f8b00009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.045+0000 7f8b05400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2ab5e0 0x558cda2b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.045+0000 7f8b06800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2af030 0x558cda30fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.045+0000 7f8b05400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2ab5e0 0x558cda2b2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:39630/0 (socket says 172.21.15.78:39630) 2024-03-14T09:54:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.045+0000 7f8b05e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a4180 0x558cda2b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.045+0000 7f8b05400700 1 -- 172.21.15.78:0/4108244651 learned_addr learned my addr 172.21.15.78:0/4108244651 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.045+0000 7f8b05400700 1 -- 172.21.15.78:0/4108244651 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a4180 msgr2=0x558cda2b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.045+0000 7f8b05400700 1 --2- 172.21.15.78:0/4108244651 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a4180 0x558cda2b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.045+0000 7f8b05400700 1 -- 172.21.15.78:0/4108244651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2af030 msgr2=0x558cda30fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.045+0000 7f8b05400700 1 --2- 172.21.15.78:0/4108244651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2af030 0x558cda30fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.045+0000 7f8b05400700 1 -- 172.21.15.78:0/4108244651 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558cda024b60 con 0x558cda2ab5e0 2024-03-14T09:54:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.045+0000 7f8b06800700 1 --2- 172.21.15.78:0/4108244651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2af030 0x558cda30fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:31.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.046+0000 7f8b05400700 1 --2- 172.21.15.78:0/4108244651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2ab5e0 0x558cda2b2320 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8af4003b00 tx=0x7f8af4003e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6138850806a56e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:31.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.046+0000 7f8ae7400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:54:31.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.046+0000 7f8ae7400700 1 -- 172.21.15.78:0/4108244651 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af4022040 con 0x558cda2ab5e0 2024-03-14T09:54:31.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.046+0000 7f8b20694b80 1 -- 172.21.15.78:0/4108244651 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8af809a470 con 0x558cda2ab5e0 2024-03-14T09:54:31.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.046+0000 7f8ae7400700 1 -- 172.21.15.78:0/4108244651 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f8af40263b0 con 0x558cda2ab5e0 2024-03-14T09:54:31.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.046+0000 7f8ae7400700 1 -- 172.21.15.78:0/4108244651 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8af4003900 con 0x558cda2ab5e0 2024-03-14T09:54:31.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.046+0000 7f8b20694b80 10 client.25637 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:31.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.046+0000 7f8b20694b80 1 -- 172.21.15.78:0/4108244651 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f8b000114d0 con 0x558cda2ab5e0 2024-03-14T09:54:31.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.046+0000 7f8ae7400700 1 -- 172.21.15.78:0/4108244651 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8af809a470 con 0x558cda2ab5e0 2024-03-14T09:54:31.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.047+0000 7f8ae7400700 1 -- 172.21.15.78:0/4108244651 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8af4025070 con 0x558cda2ab5e0 2024-03-14T09:54:31.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.047+0000 7f8b20694b80 10 client.25637 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:31.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.047+0000 7f8b20694b80 10 client.25637 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:31.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.047+0000 7f8b20694b80 20 client.25637 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:31.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.047+0000 7f8b20694b80 1 --2- 172.21.15.78:0/4108244651 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558cda311ce0 0x558cda314100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.047+0000 7f8b20694b80 4 client.25637 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:54:31.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.047+0000 7f8b20694b80 1 -- 172.21.15.78:0/4108244651 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558cda045cf0 con 0x558cda311ce0 2024-03-14T09:54:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.047+0000 7f8b05e00700 1 --2- 172.21.15.78:0/4108244651 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558cda311ce0 0x558cda314100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.048+0000 7f8b05e00700 1 --2- 172.21.15.78:0/4108244651 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558cda311ce0 0x558cda314100 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ff7ab32fa32f82f server_cookie=bbc936dfbf414a90 in_seq=0 out_seq=0 2024-03-14T09:54:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.048+0000 7f8ae7400700 10 client.25637 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.048+0000 7f8ae7400700 1 -- 172.21.15.78:0/4108244651 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x558cda045cf0 con 0x558cda311ce0 2024-03-14T09:54:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.048+0000 7f8ae7400700 10 client.25637 handle_command_reply: tid=0 2024-03-14T09:54:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.048+0000 7f8b20694b80 1 client.25637 shutdown 2024-03-14T09:54:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.048+0000 7f8b20694b80 1 -- 172.21.15.78:0/4108244651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2ab5e0 msgr2=0x558cda2b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.048+0000 7f8b20694b80 1 --2- 172.21.15.78:0/4108244651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2ab5e0 0x558cda2b2320 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8af4003b00 tx=0x7f8af4003e20 comp rx=0 tx=0).stop 2024-03-14T09:54:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.048+0000 7f8b20694b80 1 -- 172.21.15.78:0/4108244651 shutdown_connections 2024-03-14T09:54:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.048+0000 7f8b20694b80 1 --2- 172.21.15.78:0/4108244651 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558cda2af030 0x558cda30fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.048+0000 7f8b20694b80 1 --2- 172.21.15.78:0/4108244651 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558cda2a4180 0x558cda2b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.048+0000 7f8b20694b80 1 --2- 172.21.15.78:0/4108244651 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558cda311ce0 0x558cda314100 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.048+0000 7f8b20694b80 1 --2- 172.21.15.78:0/4108244651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558cda2ab5e0 0x558cda2b2320 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.048+0000 7f8b20694b80 1 -- 172.21.15.78:0/4108244651 >> 172.21.15.78:0/4108244651 conn(0x558cda21a220 msgr2=0x558cda2a5900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.048+0000 7f8ae7400700 0 client.25637 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:31.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.048+0000 7f8b20694b80 1 -- 172.21.15.78:0/4108244651 shutdown_connections 2024-03-14T09:54:31.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.048+0000 7f8b20694b80 1 -- 172.21.15.78:0/4108244651 wait complete. 2024-03-14T09:54:31.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.049+0000 7f8b20694b80 20 client.25637 trim_cache size 0 max 16384 2024-03-14T09:54:31.053 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:54:31.053 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T09:54:31.053 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T09:54:31.053 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:54:31.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.049+0000 7f8b07200700 1 -- 172.21.15.78:0/2642314389 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8aec0a0d10 msgr2=0x7f8aec0a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.049+0000 7f8b07200700 1 --2- 172.21.15.78:0/2642314389 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8aec0a0d10 0x7f8aec0a31d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8b081010c0 tx=0x7f8af4009260 comp rx=0 tx=0).stop 2024-03-14T09:54:31.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.049+0000 7f8b07200700 1 -- 172.21.15.78:0/2642314389 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b08101d40 msgr2=0x7f8b080ff990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.049+0000 7f8b07200700 1 --2- 172.21.15.78:0/2642314389 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b08101d40 0x7f8b080ff990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8af8005910 tx=0x7f8af8004730 comp rx=0 tx=0).stop 2024-03-14T09:54:31.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.049+0000 7f8b07200700 1 -- 172.21.15.78:0/2642314389 shutdown_connections 2024-03-14T09:54:31.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.049+0000 7f8b07200700 1 --2- 172.21.15.78:0/2642314389 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8aec0a0d10 0x7f8aec0a31d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.049+0000 7f8b07200700 1 --2- 172.21.15.78:0/2642314389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b0810b110 0x7f8b08100410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.049+0000 7f8b07200700 1 --2- 172.21.15.78:0/2642314389 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8b08101d40 0x7f8b080ff990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.049+0000 7f8b07200700 1 --2- 172.21.15.78:0/2642314389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b08102660 0x7f8b080ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.049+0000 7f8b07200700 1 -- 172.21.15.78:0/2642314389 >> 172.21.15.78:0/2642314389 conn(0x7f8b080fb4e0 msgr2=0x7f8b080fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.049+0000 7f8b07200700 1 -- 172.21.15.78:0/2642314389 shutdown_connections 2024-03-14T09:54:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.050+0000 7f8b07200700 1 -- 172.21.15.78:0/2642314389 wait complete. 2024-03-14T09:54:31.069 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T09:54:31.069 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:54:31.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:31 smithi078 ceph-mon[38143]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T09:54:31.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:31 smithi132 ceph-mon[47959]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T09:54:31.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.271+0000 7f406ca00700 1 -- 172.21.15.78:0/2480358784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40680689e0 msgr2=0x7f4068105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.271+0000 7f406ca00700 1 --2- 172.21.15.78:0/2480358784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40680689e0 0x7f4068105800 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4054009990 tx=0x7f4054009ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.271+0000 7f406ca00700 1 -- 172.21.15.78:0/2480358784 shutdown_connections 2024-03-14T09:54:31.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.271+0000 7f406ca00700 1 --2- 172.21.15.78:0/2480358784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4068105ee0 0x7f406810dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.271+0000 7f406ca00700 1 --2- 172.21.15.78:0/2480358784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4068068030 0x7f4068068410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.271+0000 7f406ca00700 1 --2- 172.21.15.78:0/2480358784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f40680689e0 0x7f4068105800 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.271+0000 7f406ca00700 1 -- 172.21.15.78:0/2480358784 >> 172.21.15.78:0/2480358784 conn(0x7f40680755f0 msgr2=0x7f4068075a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:31.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.271+0000 7f406ca00700 1 -- 172.21.15.78:0/2480358784 shutdown_connections 2024-03-14T09:54:31.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.271+0000 7f406ca00700 1 -- 172.21.15.78:0/2480358784 wait complete. 2024-03-14T09:54:31.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.272+0000 7f406ca00700 1 Processor -- start 2024-03-14T09:54:31.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.273+0000 7f406ca00700 1 -- start start 2024-03-14T09:54:31.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.273+0000 7f406ca00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4068068030 0x7f40681996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.273+0000 7f406ca00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4068105ee0 0x7f4068199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.273+0000 7f406ca00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f406819dfd0 0x7f406819e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.273+0000 7f406ca00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406819ea50 con 0x7f406819dfd0 2024-03-14T09:54:31.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.273+0000 7f406ca00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406819ebc0 con 0x7f4068105ee0 2024-03-14T09:54:31.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.273+0000 7f406ca00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406819ed80 con 0x7f4068068030 2024-03-14T09:54:31.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.273+0000 7f4067e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f406819dfd0 0x7f406819e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.273+0000 7f4067e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f406819dfd0 0x7f406819e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45610/0 (socket says 172.21.15.78:45610) 2024-03-14T09:54:31.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.273+0000 7f4067e00700 1 -- 172.21.15.78:0/3616246567 learned_addr learned my addr 172.21.15.78:0/3616246567 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:31.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.273+0000 7f4067400700 1 --2- 172.21.15.78:0/3616246567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4068068030 0x7f40681996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.274+0000 7f4066a00700 1 --2- 172.21.15.78:0/3616246567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4068105ee0 0x7f4068199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.274+0000 7f4067400700 1 -- 172.21.15.78:0/3616246567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4068105ee0 msgr2=0x7f4068199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.274+0000 7f4067400700 1 --2- 172.21.15.78:0/3616246567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4068105ee0 0x7f4068199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.274+0000 7f4067400700 1 -- 172.21.15.78:0/3616246567 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f406819dfd0 msgr2=0x7f406819e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.274+0000 7f4067400700 1 --2- 172.21.15.78:0/3616246567 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f406819dfd0 0x7f406819e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.274+0000 7f4067400700 1 -- 172.21.15.78:0/3616246567 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40540095f0 con 0x7f4068068030 2024-03-14T09:54:31.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.274+0000 7f4067e00700 1 --2- 172.21.15.78:0/3616246567 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f406819dfd0 0x7f406819e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:31.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.274+0000 7f4067400700 1 --2- 172.21.15.78:0/3616246567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4068068030 0x7f40681996f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f405800e9f0 tx=0x7f405800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:31.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.275+0000 7f405fe00700 1 -- 172.21.15.78:0/3616246567 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405800eee0 con 0x7f4068068030 2024-03-14T09:54:31.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.275+0000 7f405fe00700 1 -- 172.21.15.78:0/3616246567 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f405800cbd0 con 0x7f4068068030 2024-03-14T09:54:31.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.275+0000 7f406ca00700 1 -- 172.21.15.78:0/3616246567 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40681a62e0 con 0x7f4068068030 2024-03-14T09:54:31.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.275+0000 7f405fe00700 1 -- 172.21.15.78:0/3616246567 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4058018680 con 0x7f4068068030 2024-03-14T09:54:31.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.275+0000 7f406ca00700 1 -- 172.21.15.78:0/3616246567 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40681a6810 con 0x7f4068068030 2024-03-14T09:54:31.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.276+0000 7f406ca00700 1 -- 172.21.15.78:0/3616246567 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40681022e0 con 0x7f4068068030 2024-03-14T09:54:31.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.278+0000 7f405fe00700 1 -- 172.21.15.78:0/3616246567 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 25) v1 ==== 135564+0+0 (secure 0 0 0) 0x7f405800cd40 con 0x7f4068068030 2024-03-14T09:54:31.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.279+0000 7f405fe00700 1 --2- 172.21.15.78:0/3616246567 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f404c0a0d10 0x7f404c0a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.279+0000 7f405fe00700 1 -- 172.21.15.78:0/3616246567 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f4058014070 con 0x7f4068068030 2024-03-14T09:54:31.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.279+0000 7f4066a00700 1 --2- 172.21.15.78:0/3616246567 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f404c0a0d10 0x7f404c0a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.280+0000 7f4066a00700 1 --2- 172.21.15.78:0/3616246567 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f404c0a0d10 0x7f404c0a31d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f406819ac40 tx=0x7f4054005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:31.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.281+0000 7f405fe00700 1 -- 172.21.15.78:0/3616246567 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4058095f10 con 0x7f4068068030 2024-03-14T09:54:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:31 smithi152 ceph-mon[48279]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T09:54:31.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.584+0000 7f406ca00700 1 -- 172.21.15.78:0/3616246567 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f406819a9f0 con 0x7f4068068030 2024-03-14T09:54:31.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.585+0000 7f405fe00700 1 -- 172.21.15.78:0/3616246567 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f4058026090 con 0x7f4068068030 2024-03-14T09:54:31.586 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:54:31.586 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:54:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.587+0000 7f406ca00700 1 -- 172.21.15.78:0/3616246567 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f404c0a0d10 msgr2=0x7f404c0a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.587+0000 7f406ca00700 1 --2- 172.21.15.78:0/3616246567 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f404c0a0d10 0x7f404c0a31d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f406819ac40 tx=0x7f4054005fd0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.587+0000 7f406ca00700 1 -- 172.21.15.78:0/3616246567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4068068030 msgr2=0x7f40681996f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.587+0000 7f406ca00700 1 --2- 172.21.15.78:0/3616246567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4068068030 0x7f40681996f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f405800e9f0 tx=0x7f405800ed00 comp rx=0 tx=0).stop 2024-03-14T09:54:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.588+0000 7f406ca00700 1 -- 172.21.15.78:0/3616246567 shutdown_connections 2024-03-14T09:54:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.588+0000 7f406ca00700 1 --2- 172.21.15.78:0/3616246567 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f406819dfd0 0x7f406819e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.588+0000 7f406ca00700 1 --2- 172.21.15.78:0/3616246567 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4068068030 0x7f40681996f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.588+0000 7f406ca00700 1 --2- 172.21.15.78:0/3616246567 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f404c0a0d10 0x7f404c0a31d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.588+0000 7f406ca00700 1 --2- 172.21.15.78:0/3616246567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4068105ee0 0x7f4068199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.588+0000 7f406ca00700 1 -- 172.21.15.78:0/3616246567 >> 172.21.15.78:0/3616246567 conn(0x7f40680755f0 msgr2=0x7f406810be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.588+0000 7f406ca00700 1 -- 172.21.15.78:0/3616246567 shutdown_connections 2024-03-14T09:54:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.588+0000 7f406ca00700 1 -- 172.21.15.78:0/3616246567 wait complete. 2024-03-14T09:54:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T09:54:31.608 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T09:54:31.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.824+0000 7fa3b7200700 1 -- 172.21.15.78:0/4092347815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b81014b0 msgr2=0x7fa3b810d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.824+0000 7fa3b7200700 1 --2- 172.21.15.78:0/4092347815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b81014b0 0x7fa3b810d9c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0009f90 tx=0x7fa3b000b330 comp rx=0 tx=0).stop 2024-03-14T09:54:31.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.824+0000 7fa3b7200700 1 -- 172.21.15.78:0/4092347815 shutdown_connections 2024-03-14T09:54:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.824+0000 7fa3b7200700 1 --2- 172.21.15.78:0/4092347815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b81014b0 0x7fa3b810d9c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.824+0000 7fa3b7200700 1 --2- 172.21.15.78:0/4092347815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3b8106b90 0x7fa3b8106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.824+0000 7fa3b7200700 1 --2- 172.21.15.78:0/4092347815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b8100af0 0x7fa3b8100f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.824+0000 7fa3b7200700 1 -- 172.21.15.78:0/4092347815 >> 172.21.15.78:0/4092347815 conn(0x7fa3b80fc5e0 msgr2=0x7fa3b80fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.825+0000 7fa3b7200700 1 -- 172.21.15.78:0/4092347815 shutdown_connections 2024-03-14T09:54:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.825+0000 7fa3b7200700 1 -- 172.21.15.78:0/4092347815 wait complete. 2024-03-14T09:54:31.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.826+0000 7fa3b7200700 1 Processor -- start 2024-03-14T09:54:31.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.826+0000 7fa3b7200700 1 -- start start 2024-03-14T09:54:31.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.826+0000 7fa3b7200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b8100af0 0x7fa3b80730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.826+0000 7fa3b7200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b81014b0 0x7fa3b80735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.826+0000 7fa3b7200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3b8106b90 0x7fa3b8073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.826+0000 7fa3b7200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b81a5760 con 0x7fa3b81014b0 2024-03-14T09:54:31.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.826+0000 7fa3b7200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b81a58d0 con 0x7fa3b8100af0 2024-03-14T09:54:31.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.826+0000 7fa3b7200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b81a5a90 con 0x7fa3b8106b90 2024-03-14T09:54:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.826+0000 7fa3b5400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b81014b0 0x7fa3b80735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.826+0000 7fa3b5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b8100af0 0x7fa3b80730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.827+0000 7fa3b5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b8100af0 0x7fa3b80730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:39670/0 (socket says 172.21.15.78:39670) 2024-03-14T09:54:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.827+0000 7fa3b5400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b81014b0 0x7fa3b80735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45624/0 (socket says 172.21.15.78:45624) 2024-03-14T09:54:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.827+0000 7fa3b5e00700 1 -- 172.21.15.78:0/43807097 learned_addr learned my addr 172.21.15.78:0/43807097 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.827+0000 7fa3b6800700 1 --2- 172.21.15.78:0/43807097 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3b8106b90 0x7fa3b8073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.827+0000 7fa3b5400700 1 -- 172.21.15.78:0/43807097 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3b8106b90 msgr2=0x7fa3b8073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.827+0000 7fa3b5400700 1 --2- 172.21.15.78:0/43807097 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3b8106b90 0x7fa3b8073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.827+0000 7fa3b5400700 1 -- 172.21.15.78:0/43807097 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b8100af0 msgr2=0x7fa3b80730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.827+0000 7fa3b5400700 1 --2- 172.21.15.78:0/43807097 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b8100af0 0x7fa3b80730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.827+0000 7fa3b5400700 1 -- 172.21.15.78:0/43807097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b0009c40 con 0x7fa3b81014b0 2024-03-14T09:54:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.827+0000 7fa3b5e00700 1 --2- 172.21.15.78:0/43807097 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b8100af0 0x7fa3b80730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.827+0000 7fa3b5400700 1 --2- 172.21.15.78:0/43807097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b81014b0 0x7fa3b80735f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa3a400e9c0 tx=0x7fa3a400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.827+0000 7fa3aea00700 1 -- 172.21.15.78:0/43807097 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a400c9c0 con 0x7fa3b81014b0 2024-03-14T09:54:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.828+0000 7fa3aea00700 1 -- 172.21.15.78:0/43807097 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3a4010430 con 0x7fa3b81014b0 2024-03-14T09:54:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.828+0000 7fa3b7200700 1 -- 172.21.15.78:0/43807097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b81a5d70 con 0x7fa3b81014b0 2024-03-14T09:54:31.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.828+0000 7fa3aea00700 1 -- 172.21.15.78:0/43807097 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4018680 con 0x7fa3b81014b0 2024-03-14T09:54:31.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.828+0000 7fa3b7200700 1 -- 172.21.15.78:0/43807097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b81a6270 con 0x7fa3b81014b0 2024-03-14T09:54:31.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.831+0000 7fa3aea00700 1 -- 172.21.15.78:0/43807097 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 135564+0+0 (secure 0 0 0) 0x7fa3a4010860 con 0x7fa3b81014b0 2024-03-14T09:54:31.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.832+0000 7fa3d0ab8b80 1 Processor -- start 2024-03-14T09:54:31.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.832+0000 7fa3d0ab8b80 1 -- start start 2024-03-14T09:54:31.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3d0ab8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312667860 0x564312667c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3d0ab8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312668180 0x564312672af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3d0ab8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312673030 0x564312675420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3d0ab8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643125c1990 con 0x564312667860 2024-03-14T09:54:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3d0ab8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643125bf350 con 0x564312668180 2024-03-14T09:54:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3d0ab8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643125bac40 con 0x564312673030 2024-03-14T09:54:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3b6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312667860 0x564312667c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3b5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312668180 0x564312672af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3b6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312667860 0x564312667c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45628/0 (socket says 172.21.15.78:45628) 2024-03-14T09:54:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3b6800700 1 -- 172.21.15.78:0/2054603047 learned_addr learned my addr 172.21.15.78:0/2054603047 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3b5400700 1 --2- 172.21.15.78:0/2054603047 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312673030 0x564312675420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3b6800700 1 -- 172.21.15.78:0/2054603047 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312673030 msgr2=0x564312675420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3b6800700 1 --2- 172.21.15.78:0/2054603047 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312673030 0x564312675420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3b6800700 1 -- 172.21.15.78:0/2054603047 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312668180 msgr2=0x564312672af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3b6800700 1 --2- 172.21.15.78:0/2054603047 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312668180 0x564312672af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3b6800700 1 -- 172.21.15.78:0/2054603047 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56431243c3b0 con 0x564312667860 2024-03-14T09:54:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.833+0000 7fa3b5e00700 1 --2- 172.21.15.78:0/2054603047 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312668180 0x564312672af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:31.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.834+0000 7fa3aea00700 1 --2- 172.21.15.78:0/43807097 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa39c0a0d10 0x7fa39c0a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.834+0000 7fa3b6800700 1 --2- 172.21.15.78:0/2054603047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312667860 0x564312667c40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0006070 tx=0x7fa3b0007ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d58daf1f3bb8ce5 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:31.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.834+0000 7fa3aea00700 1 -- 172.21.15.78:0/43807097 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fa3a4014070 con 0x7fa3b81014b0 2024-03-14T09:54:31.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.834+0000 7fa3b5e00700 1 --2- 172.21.15.78:0/43807097 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa39c0a0d10 0x7fa39c0a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.834+0000 7fa3a3400700 1 -- 172.21.15.78:0/2054603047 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b000e040 con 0x564312667860 2024-03-14T09:54:31.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.834+0000 7fa3a3400700 1 -- 172.21.15.78:0/2054603047 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3b000b9f0 con 0x564312667860 2024-03-14T09:54:31.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.834+0000 7fa3a3400700 1 -- 172.21.15.78:0/2054603047 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b00077e0 con 0x564312667860 2024-03-14T09:54:31.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.835+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/2054603047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312667860 msgr2=0x564312667c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.835+0000 7fa3b5e00700 1 --2- 172.21.15.78:0/43807097 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa39c0a0d10 0x7fa39c0a31d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80096b0 tx=0x7fa3a8009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:31.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.835+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/2054603047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312667860 0x564312667c40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0006070 tx=0x7fa3b0007ab0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.835+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/2054603047 shutdown_connections 2024-03-14T09:54:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.835+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/2054603047 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312673030 0x564312675420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.835+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/2054603047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312667860 0x564312667c40 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.835+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/2054603047 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312668180 0x564312672af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.835+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/2054603047 >> 172.21.15.78:0/2054603047 conn(0x5643125de220 msgr2=0x564312663c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.835+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/2054603047 shutdown_connections 2024-03-14T09:54:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.835+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/2054603047 wait complete. 2024-03-14T09:54:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.836+0000 7fa3d0ab8b80 1 -- start start 2024-03-14T09:54:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.836+0000 7fa3d0ab8b80 1 Processor -- start 2024-03-14T09:54:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.836+0000 7fa3d0ab8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312667860 0x5643126729f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.836+0000 7fa3d0ab8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312668180 0x56431266fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.836+0000 7fa3d0ab8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312673030 0x564312670340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.836+0000 7fa3d0ab8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564312556d80 con 0x564312668180 2024-03-14T09:54:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.836+0000 7fa3d0ab8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x564312556a20 con 0x564312673030 2024-03-14T09:54:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.836+0000 7fa3d0ab8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431243e6a0 con 0x564312667860 2024-03-14T09:54:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.836+0000 7fa3d0ab8b80 10 client.0 _ll_register_callbacks cb 0x5643123c23a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa3b5400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312668180 0x56431266fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa3b5400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312668180 0x56431266fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45638/0 (socket says 172.21.15.78:45638) 2024-03-14T09:54:31.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa3b5400700 1 -- 172.21.15.78:0/1718247527 learned_addr learned my addr 172.21.15.78:0/1718247527 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:31.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa3b5e00700 1 --2- 172.21.15.78:0/1718247527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312667860 0x5643126729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa3b6800700 1 --2- 172.21.15.78:0/1718247527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312673030 0x564312670340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa3b5400700 1 -- 172.21.15.78:0/1718247527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312667860 msgr2=0x5643126729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:31.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa3b5400700 1 --2- 172.21.15.78:0/1718247527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312667860 0x5643126729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa3b5400700 1 -- 172.21.15.78:0/1718247527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312673030 msgr2=0x564312670340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:31.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa3b5400700 1 --2- 172.21.15.78:0/1718247527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312673030 0x564312670340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa3b5400700 1 -- 172.21.15.78:0/1718247527 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a8009ce0 con 0x564312668180 2024-03-14T09:54:31.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa3b6800700 1 --2- 172.21.15.78:0/1718247527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312673030 0x564312670340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:54:31.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa3b5e00700 1 --2- 172.21.15.78:0/1718247527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312667860 0x5643126729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:54:31.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa3b5400700 1 --2- 172.21.15.78:0/1718247527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312668180 0x56431266fd30 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4005fd0 tx=0x7fa3a4099ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2737bb2adb0dd55 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:31.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa397e00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:54:31.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa397e00700 1 -- 172.21.15.78:0/1718247527 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4099530 con 0x564312668180 2024-03-14T09:54:31.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa397e00700 1 -- 172.21.15.78:0/1718247527 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fa3a4099e60 con 0x564312668180 2024-03-14T09:54:31.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.837+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/1718247527 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56431265b720 con 0x564312668180 2024-03-14T09:54:31.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.838+0000 7fa397e00700 1 -- 172.21.15.78:0/1718247527 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3a40992a0 con 0x564312668180 2024-03-14T09:54:31.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.838+0000 7fa3d0ab8b80 10 client.15651 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:31.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.838+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/1718247527 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7fa3b810e2c0 con 0x564312668180 2024-03-14T09:54:31.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.838+0000 7fa397e00700 1 -- 172.21.15.78:0/1718247527 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3a40b1030 con 0x564312668180 2024-03-14T09:54:31.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.838+0000 7fa397e00700 1 -- 172.21.15.78:0/1718247527 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa3a40a1750 con 0x564312668180 2024-03-14T09:54:31.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.838+0000 7fa3d0ab8b80 10 client.15651 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:31.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.838+0000 7fa3d0ab8b80 10 client.15651 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:54:31.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.838+0000 7fa3d0ab8b80 20 client.15651 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:31.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.838+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/1718247527 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x564312682a80 0x564312684ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.838+0000 7fa3d0ab8b80 4 client.15651 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:54:31.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.838+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/1718247527 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5643125b3700 con 0x564312682a80 2024-03-14T09:54:31.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.839+0000 7fa3b5e00700 1 --2- 172.21.15.78:0/1718247527 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x564312682a80 0x564312684ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.839+0000 7fa3b5e00700 1 --2- 172.21.15.78:0/1718247527 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x564312682a80 0x564312684ea0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=83a8943bd2979e4a server_cookie=4dd25649a0f5e42b in_seq=0 out_seq=0 2024-03-14T09:54:31.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.839+0000 7fa397e00700 10 client.15651 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:54:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.840+0000 7fa397e00700 1 -- 172.21.15.78:0/1718247527 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5643125b3700 con 0x564312682a80 2024-03-14T09:54:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.840+0000 7fa397e00700 10 client.15651 handle_command_reply: tid=0 2024-03-14T09:54:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.840+0000 7fa3d0ab8b80 1 client.15651 shutdown 2024-03-14T09:54:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.840+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/1718247527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312668180 msgr2=0x56431266fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.840+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/1718247527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312668180 0x56431266fd30 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4005fd0 tx=0x7fa3a4099ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.840+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/1718247527 shutdown_connections 2024-03-14T09:54:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.840+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/1718247527 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x564312682a80 0x564312684ea0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.840+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/1718247527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312673030 0x564312670340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.840+0000 7fa397e00700 0 client.15651 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:54:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.840+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/1718247527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312667860 0x5643126729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.840+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/1718247527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312668180 0x56431266fd30 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.840+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/1718247527 >> 172.21.15.78:0/1718247527 conn(0x5643125de220 msgr2=0x564312663880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.840+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/1718247527 shutdown_connections 2024-03-14T09:54:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.840+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/1718247527 wait complete. 2024-03-14T09:54:31.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.841+0000 7fa3d0ab8b80 20 client.15651 trim_cache size 0 max 16384 2024-03-14T09:54:31.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.854+0000 7fa3d0ab8b80 1 Processor -- start 2024-03-14T09:54:31.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.854+0000 7fa3d0ab8b80 1 -- start start 2024-03-14T09:54:31.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.854+0000 7fa3d0ab8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312667860 0x564312689800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.854+0000 7fa3d0ab8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312668180 0x564312679c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.854+0000 7fa3d0ab8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312673030 0x56431266fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.854+0000 7fa3d0ab8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564312668560 con 0x564312673030 2024-03-14T09:54:31.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.854+0000 7fa3d0ab8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x564312680270 con 0x564312668180 2024-03-14T09:54:31.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.854+0000 7fa3d0ab8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643126712e0 con 0x564312667860 2024-03-14T09:54:31.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.854+0000 7fa3b5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312668180 0x564312679c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.854+0000 7fa3b6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312673030 0x56431266fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.854+0000 7fa3b5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312668180 0x564312679c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:39694/0 (socket says 172.21.15.78:39694) 2024-03-14T09:54:31.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.854+0000 7fa3b5400700 1 -- 172.21.15.78:0/4209963025 learned_addr learned my addr 172.21.15.78:0/4209963025 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:31.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.855+0000 7fa3b5e00700 1 --2- 172.21.15.78:0/4209963025 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312667860 0x564312689800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.855+0000 7fa3b5400700 1 -- 172.21.15.78:0/4209963025 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312667860 msgr2=0x564312689800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:31.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.855+0000 7fa3b5400700 1 --2- 172.21.15.78:0/4209963025 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312667860 0x564312689800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.855+0000 7fa3b5400700 1 -- 172.21.15.78:0/4209963025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312673030 msgr2=0x56431266fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:31.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.855+0000 7fa3b5400700 1 --2- 172.21.15.78:0/4209963025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312673030 0x56431266fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.855+0000 7fa3b5400700 1 -- 172.21.15.78:0/4209963025 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643123e8b60 con 0x564312668180 2024-03-14T09:54:31.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.855+0000 7fa3b5400700 1 --2- 172.21.15.78:0/4209963025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312668180 0x564312679c40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa3a40961d0 tx=0x7fa3a40b2240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4ebe10b205b08a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:31.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.855+0000 7fa3a3e00700 1 -- 172.21.15.78:0/4209963025 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a40187e0 con 0x564312668180 2024-03-14T09:54:31.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.855+0000 7fa3a3e00700 1 -- 172.21.15.78:0/4209963025 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3a40b1070 con 0x564312668180 2024-03-14T09:54:31.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.855+0000 7fa3a3e00700 1 -- 172.21.15.78:0/4209963025 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a40d22d0 con 0x564312668180 2024-03-14T09:54:31.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.856+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/4209963025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312668180 msgr2=0x564312679c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.856+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/4209963025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312668180 0x564312679c40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa3a40961d0 tx=0x7fa3a40b2240 comp rx=0 tx=0).stop 2024-03-14T09:54:31.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.856+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/4209963025 shutdown_connections 2024-03-14T09:54:31.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.856+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/4209963025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312673030 0x56431266fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.856+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/4209963025 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312667860 0x564312689800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.856+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/4209963025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312668180 0x564312679c40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.856+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/4209963025 >> 172.21.15.78:0/4209963025 conn(0x5643125de220 msgr2=0x56431266f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:31.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.856+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/4209963025 shutdown_connections 2024-03-14T09:54:31.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.856+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/4209963025 wait complete. 2024-03-14T09:54:31.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.857+0000 7fa3d0ab8b80 1 -- start start 2024-03-14T09:54:31.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.857+0000 7fa3d0ab8b80 1 Processor -- start 2024-03-14T09:54:31.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.858+0000 7fa3d0ab8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312667860 0x5643126780f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.858+0000 7fa3d0ab8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312668180 0x564312678630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.858+0000 7fa3d0ab8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312673030 0x564312678b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.858+0000 7fa3d0ab8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643126d45b0 con 0x564312673030 2024-03-14T09:54:31.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.858+0000 7fa3d0ab8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643126d46f0 con 0x564312667860 2024-03-14T09:54:31.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.858+0000 7fa3d0ab8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643126d4830 con 0x564312668180 2024-03-14T09:54:31.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.858+0000 7fa3d0ab8b80 10 client.0 _ll_register_callbacks cb 0x7fa3a4010b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:31.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.858+0000 7fa3b5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312667860 0x5643126780f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.858+0000 7fa3b5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312667860 0x5643126780f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:39710/0 (socket says 172.21.15.78:39710) 2024-03-14T09:54:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.858+0000 7fa3b5e00700 1 -- 172.21.15.78:0/4057582625 learned_addr learned my addr 172.21.15.78:0/4057582625 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.858+0000 7fa3b5400700 1 --2- 172.21.15.78:0/4057582625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312668180 0x564312678630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.858+0000 7fa3b6800700 1 --2- 172.21.15.78:0/4057582625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312673030 0x564312678b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.859+0000 7fa3b5e00700 1 -- 172.21.15.78:0/4057582625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312668180 msgr2=0x564312678630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.859+0000 7fa3b5e00700 1 --2- 172.21.15.78:0/4057582625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312668180 0x564312678630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.859+0000 7fa3b5e00700 1 -- 172.21.15.78:0/4057582625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312673030 msgr2=0x564312678b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.859+0000 7fa3b5e00700 1 --2- 172.21.15.78:0/4057582625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312673030 0x564312678b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.859+0000 7fa3b5e00700 1 -- 172.21.15.78:0/4057582625 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b810e2c0 con 0x564312667860 2024-03-14T09:54:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.859+0000 7fa3b5400700 1 --2- 172.21.15.78:0/4057582625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312668180 0x564312678630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.859+0000 7fa3b5e00700 1 --2- 172.21.15.78:0/4057582625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312667860 0x5643126780f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009fd0 tx=0x7fa3a80047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b871012b24466375 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.859+0000 7fa397e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:54:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.859+0000 7fa397e00700 1 -- 172.21.15.78:0/4057582625 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8024040 con 0x564312667860 2024-03-14T09:54:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.859+0000 7fa397e00700 1 -- 172.21.15.78:0/4057582625 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fa3a8029310 con 0x564312667860 2024-03-14T09:54:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.859+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/4057582625 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa3b000b9f0 con 0x564312667860 2024-03-14T09:54:31.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.860+0000 7fa397e00700 1 -- 172.21.15.78:0/4057582625 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3a8016070 con 0x564312667860 2024-03-14T09:54:31.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.860+0000 7fa3d0ab8b80 10 client.25661 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:31.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.860+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/4057582625 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564312580ea0 con 0x564312667860 2024-03-14T09:54:31.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.860+0000 7fa397e00700 1 -- 172.21.15.78:0/4057582625 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3b000b9f0 con 0x564312667860 2024-03-14T09:54:31.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.860+0000 7fa397e00700 1 -- 172.21.15.78:0/4057582625 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa3a8003680 con 0x564312667860 2024-03-14T09:54:31.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.860+0000 7fa3d0ab8b80 10 client.25661 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:31.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.860+0000 7fa3d0ab8b80 10 client.25661 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:54:31.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.860+0000 7fa3d0ab8b80 20 client.25661 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:31.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.860+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/4057582625 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5643126d6080 0x5643126d84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:31.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.860+0000 7fa3d0ab8b80 4 client.25661 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:54:31.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.860+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/4057582625 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564312409cf0 con 0x5643126d6080 2024-03-14T09:54:31.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.860+0000 7fa3b5400700 1 --2- 172.21.15.78:0/4057582625 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5643126d6080 0x5643126d84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:31.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.861+0000 7fa397e00700 10 client.25661 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:54:31.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.861+0000 7fa3b5400700 1 --2- 172.21.15.78:0/4057582625 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5643126d6080 0x5643126d84a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a76d995a7b2a5c97 server_cookie=440bcc843511f68d in_seq=0 out_seq=0 2024-03-14T09:54:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.861+0000 7fa397e00700 1 -- 172.21.15.78:0/4057582625 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564312409cf0 con 0x5643126d6080 2024-03-14T09:54:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.861+0000 7fa397e00700 10 client.25661 handle_command_reply: tid=0 2024-03-14T09:54:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.861+0000 7fa3d0ab8b80 1 client.25661 shutdown 2024-03-14T09:54:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.861+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/4057582625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312667860 msgr2=0x5643126780f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.861+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/4057582625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312667860 0x5643126780f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009fd0 tx=0x7fa3a80047c0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.861+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/4057582625 shutdown_connections 2024-03-14T09:54:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.861+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/4057582625 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5643126d6080 0x5643126d84a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.861+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/4057582625 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564312667860 0x5643126780f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.861+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/4057582625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564312673030 0x564312678b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.861+0000 7fa397e00700 0 client.25661 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:54:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.861+0000 7fa3d0ab8b80 1 --2- 172.21.15.78:0/4057582625 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564312668180 0x564312678630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.861+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/4057582625 >> 172.21.15.78:0/4057582625 conn(0x5643125de220 msgr2=0x564312669900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.862+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/4057582625 shutdown_connections 2024-03-14T09:54:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.862+0000 7fa3d0ab8b80 1 -- 172.21.15.78:0/4057582625 wait complete. 2024-03-14T09:54:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.862+0000 7fa3d0ab8b80 20 client.25661 trim_cache size 0 max 16384 2024-03-14T09:54:31.866 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.862+0000 7fa3b7200700 1 -- 172.21.15.78:0/43807097 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa39c0a0d10 msgr2=0x7fa39c0a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.862+0000 7fa3b7200700 1 --2- 172.21.15.78:0/43807097 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa39c0a0d10 0x7fa39c0a31d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80096b0 tx=0x7fa3a8009260 comp rx=0 tx=0).stop 2024-03-14T09:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.862+0000 7fa3b7200700 1 -- 172.21.15.78:0/43807097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b81014b0 msgr2=0x7fa3b80735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.862+0000 7fa3b7200700 1 --2- 172.21.15.78:0/43807097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b81014b0 0x7fa3b80735f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa3a400e9c0 tx=0x7fa3a400ecd0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.862+0000 7fa3b7200700 1 -- 172.21.15.78:0/43807097 shutdown_connections 2024-03-14T09:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.863+0000 7fa3b7200700 1 --2- 172.21.15.78:0/43807097 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa39c0a0d10 0x7fa39c0a31d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.863+0000 7fa3b7200700 1 --2- 172.21.15.78:0/43807097 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3b8100af0 0x7fa3b80730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.863+0000 7fa3b7200700 1 --2- 172.21.15.78:0/43807097 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa3b8106b90 0x7fa3b8073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.863+0000 7fa3b7200700 1 --2- 172.21.15.78:0/43807097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3b81014b0 0x7fa3b80735f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.863+0000 7fa3b7200700 1 -- 172.21.15.78:0/43807097 >> 172.21.15.78:0/43807097 conn(0x7fa3b80fc5e0 msgr2=0x7fa3b80fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.863+0000 7fa3b7200700 1 -- 172.21.15.78:0/43807097 shutdown_connections 2024-03-14T09:54:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:31.863+0000 7fa3b7200700 1 -- 172.21.15.78:0/43807097 wait complete. 2024-03-14T09:54:31.888 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T09:54:32.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.093+0000 7f1c74c00700 1 -- 172.21.15.78:0/3338827003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c701018d0 msgr2=0x7f1c70105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.093+0000 7f1c74c00700 1 --2- 172.21.15.78:0/3338827003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c701018d0 0x7f1c70105910 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009990 tx=0x7f1c5c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.094+0000 7f1c74c00700 1 -- 172.21.15.78:0/3338827003 shutdown_connections 2024-03-14T09:54:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.094+0000 7f1c74c00700 1 --2- 172.21.15.78:0/3338827003 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c70100f20 0x7f1c70101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.094+0000 7f1c74c00700 1 --2- 172.21.15.78:0/3338827003 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c70106040 0x7f1c7010dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.094+0000 7f1c74c00700 1 --2- 172.21.15.78:0/3338827003 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c701018d0 0x7f1c70105910 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.094+0000 7f1c74c00700 1 -- 172.21.15.78:0/3338827003 >> 172.21.15.78:0/3338827003 conn(0x7f1c700fc7a0 msgr2=0x7f1c700febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.094+0000 7f1c74c00700 1 -- 172.21.15.78:0/3338827003 shutdown_connections 2024-03-14T09:54:32.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.094+0000 7f1c74c00700 1 -- 172.21.15.78:0/3338827003 wait complete. 2024-03-14T09:54:32.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.095+0000 7f1c74c00700 1 Processor -- start 2024-03-14T09:54:32.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.095+0000 7f1c74c00700 1 -- start start 2024-03-14T09:54:32.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.095+0000 7f1c74c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c70100f20 0x7f1c70197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.095+0000 7f1c74c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c70106040 0x7f1c70197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.095+0000 7f1c74c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c7019bed0 0x7f1c7019c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.095+0000 7f1c74c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7019c950 con 0x7f1c70106040 2024-03-14T09:54:32.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.095+0000 7f1c74c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7019cac0 con 0x7f1c7019bed0 2024-03-14T09:54:32.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.095+0000 7f1c74c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7019cc80 con 0x7f1c70100f20 2024-03-14T09:54:32.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.095+0000 7f1c6d400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c70106040 0x7f1c70197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.096+0000 7f1c6d400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c70106040 0x7f1c70197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45678/0 (socket says 172.21.15.78:45678) 2024-03-14T09:54:32.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.096+0000 7f1c6d400700 1 -- 172.21.15.78:0/190001205 learned_addr learned my addr 172.21.15.78:0/190001205 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:32.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.096+0000 7f1c6de00700 1 --2- 172.21.15.78:0/190001205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c70100f20 0x7f1c70197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.096+0000 7f1c6e800700 1 --2- 172.21.15.78:0/190001205 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c7019bed0 0x7f1c7019c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.096+0000 7f1c6d400700 1 -- 172.21.15.78:0/190001205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c70100f20 msgr2=0x7f1c70197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.096+0000 7f1c6d400700 1 --2- 172.21.15.78:0/190001205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c70100f20 0x7f1c70197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.096+0000 7f1c6d400700 1 -- 172.21.15.78:0/190001205 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c7019bed0 msgr2=0x7f1c7019c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.096+0000 7f1c6d400700 1 --2- 172.21.15.78:0/190001205 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c7019bed0 0x7f1c7019c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.096+0000 7f1c6d400700 1 -- 172.21.15.78:0/190001205 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c0095f0 con 0x7f1c70106040 2024-03-14T09:54:32.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.096+0000 7f1c6de00700 1 --2- 172.21.15.78:0/190001205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c70100f20 0x7f1c70197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:32.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.096+0000 7f1c6d400700 1 --2- 172.21.15.78:0/190001205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c70106040 0x7f1c70197b80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c000c00 tx=0x7f1c5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:32.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.097+0000 7f1c66a00700 1 -- 172.21.15.78:0/190001205 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c01d070 con 0x7f1c70106040 2024-03-14T09:54:32.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.097+0000 7f1c66a00700 1 -- 172.21.15.78:0/190001205 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c5c021410 con 0x7f1c70106040 2024-03-14T09:54:32.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.097+0000 7f1c74c00700 1 -- 172.21.15.78:0/190001205 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c701a4180 con 0x7f1c70106040 2024-03-14T09:54:32.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.097+0000 7f1c66a00700 1 -- 172.21.15.78:0/190001205 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c017650 con 0x7f1c70106040 2024-03-14T09:54:32.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.097+0000 7f1c74c00700 1 -- 172.21.15.78:0/190001205 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c701a4600 con 0x7f1c70106040 2024-03-14T09:54:32.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.100+0000 7f1c66a00700 1 -- 172.21.15.78:0/190001205 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 135564+0+0 (secure 0 0 0) 0x7f1c5c021580 con 0x7f1c70106040 2024-03-14T09:54:32.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.101+0000 7f1c66a00700 1 --2- 172.21.15.78:0/190001205 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1c540a0d10 0x7f1c540a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.101+0000 7f1c66a00700 1 -- 172.21.15.78:0/190001205 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f1c5c0cda20 con 0x7f1c70106040 2024-03-14T09:54:32.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.102+0000 7f1c6de00700 1 --2- 172.21.15.78:0/190001205 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1c540a0d10 0x7f1c540a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.102+0000 7f1c886bdb80 1 Processor -- start 2024-03-14T09:54:32.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.102+0000 7f1c886bdb80 1 -- start start 2024-03-14T09:54:32.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.102+0000 7f1c6de00700 1 --2- 172.21.15.78:0/190001205 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1c540a0d10 0x7f1c540a31d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1c60011490 tx=0x7f1c60009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:32.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c886bdb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fe680 0x56472f7fea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c886bdb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f7fefa0 0x56472f808980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c886bdb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f808ec0 0x56472f80b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c886bdb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56472f757990 con 0x56472f7fefa0 2024-03-14T09:54:32.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c886bdb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56472f755350 con 0x56472f808ec0 2024-03-14T09:54:32.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c886bdb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56472f750c40 con 0x56472f7fe680 2024-03-14T09:54:32.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c6de00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fe680 0x56472f7fea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c6e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f808ec0 0x56472f80b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c6de00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fe680 0x56472f7fea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37280/0 (socket says 172.21.15.78:37280) 2024-03-14T09:54:32.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c6e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f808ec0 0x56472f80b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:39732/0 (socket says 172.21.15.78:39732) 2024-03-14T09:54:32.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c6de00700 1 -- 172.21.15.78:0/2980198500 learned_addr learned my addr 172.21.15.78:0/2980198500 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:32.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c6d400700 1 --2- 172.21.15.78:0/2980198500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f7fefa0 0x56472f808980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c6de00700 1 -- 172.21.15.78:0/2980198500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f808ec0 msgr2=0x56472f80b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:32.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c6de00700 1 --2- 172.21.15.78:0/2980198500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f808ec0 0x56472f80b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c6de00700 1 -- 172.21.15.78:0/2980198500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f7fefa0 msgr2=0x56472f808980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:32.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c6de00700 1 --2- 172.21.15.78:0/2980198500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f7fefa0 0x56472f808980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.103+0000 7f1c6de00700 1 -- 172.21.15.78:0/2980198500 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56472f5d23b0 con 0x56472f7fe680 2024-03-14T09:54:32.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.104+0000 7f1c6de00700 1 --2- 172.21.15.78:0/2980198500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fe680 0x56472f7fea60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1c600043c0 tx=0x7f1c60014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72ab785c6429de1b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:32.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.104+0000 7f1c5b000700 1 -- 172.21.15.78:0/2980198500 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c60003d10 con 0x56472f7fe680 2024-03-14T09:54:32.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.104+0000 7f1c5b000700 1 -- 172.21.15.78:0/2980198500 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c60003e70 con 0x56472f7fe680 2024-03-14T09:54:32.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.104+0000 7f1c5b000700 1 -- 172.21.15.78:0/2980198500 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6001e7e0 con 0x56472f7fe680 2024-03-14T09:54:32.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.105+0000 7f1c886bdb80 1 -- 172.21.15.78:0/2980198500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fe680 msgr2=0x56472f7fea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.105+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/2980198500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fe680 0x56472f7fea60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1c600043c0 tx=0x7f1c60014000 comp rx=0 tx=0).stop 2024-03-14T09:54:32.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.105+0000 7f1c886bdb80 1 -- 172.21.15.78:0/2980198500 shutdown_connections 2024-03-14T09:54:32.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.105+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/2980198500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f808ec0 0x56472f80b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.105+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/2980198500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fe680 0x56472f7fea60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.105+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/2980198500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f7fefa0 0x56472f808980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.105+0000 7f1c886bdb80 1 -- 172.21.15.78:0/2980198500 >> 172.21.15.78:0/2980198500 conn(0x56472f774220 msgr2=0x56472f7f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:32.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.105+0000 7f1c886bdb80 1 -- 172.21.15.78:0/2980198500 shutdown_connections 2024-03-14T09:54:32.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.105+0000 7f1c886bdb80 1 -- 172.21.15.78:0/2980198500 wait complete. 2024-03-14T09:54:32.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.105+0000 7f1c886bdb80 1 -- start start 2024-03-14T09:54:32.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.106+0000 7f1c886bdb80 1 Processor -- start 2024-03-14T09:54:32.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.106+0000 7f1c886bdb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f7fe680 0x56472f8088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.106+0000 7f1c886bdb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fefa0 0x56472f805ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.106+0000 7f1c886bdb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f808ec0 0x56472f806220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.106+0000 7f1c886bdb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56472f6ecd80 con 0x56472f808ec0 2024-03-14T09:54:32.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.106+0000 7f1c886bdb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56472f6eca20 con 0x56472f7fe680 2024-03-14T09:54:32.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.106+0000 7f1c886bdb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56472f5d46a0 con 0x56472f7fefa0 2024-03-14T09:54:32.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.106+0000 7f1c886bdb80 10 client.0 _ll_register_callbacks cb 0x56472f5583a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:32.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.106+0000 7f1c6d400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fefa0 0x56472f805ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.106+0000 7f1c6e800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f808ec0 0x56472f806220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.106+0000 7f1c6d400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fefa0 0x56472f805ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37288/0 (socket says 172.21.15.78:37288) 2024-03-14T09:54:32.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.107+0000 7f1c6d400700 1 -- 172.21.15.78:0/3143003172 learned_addr learned my addr 172.21.15.78:0/3143003172 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:32.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.107+0000 7f1c6de00700 1 --2- 172.21.15.78:0/3143003172 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f7fe680 0x56472f8088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.107+0000 7f1c6d400700 1 -- 172.21.15.78:0/3143003172 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f7fe680 msgr2=0x56472f8088b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:32.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.107+0000 7f1c6d400700 1 --2- 172.21.15.78:0/3143003172 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f7fe680 0x56472f8088b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.107+0000 7f1c6d400700 1 -- 172.21.15.78:0/3143003172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f808ec0 msgr2=0x56472f806220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:32.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.107+0000 7f1c6d400700 1 --2- 172.21.15.78:0/3143003172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f808ec0 0x56472f806220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.107+0000 7f1c6d400700 1 -- 172.21.15.78:0/3143003172 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c60017070 con 0x56472f7fefa0 2024-03-14T09:54:32.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.107+0000 7f1c6d400700 1 --2- 172.21.15.78:0/3143003172 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fefa0 0x56472f805ba0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c096e20 tx=0x7f1c5c0cb2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94866567b4bfebf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:32.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.107+0000 7f1c4f400700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:54:32.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.107+0000 7f1c4f400700 1 -- 172.21.15.78:0/3143003172 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0ba2c0 con 0x56472f7fefa0 2024-03-14T09:54:32.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.107+0000 7f1c4f400700 1 -- 172.21.15.78:0/3143003172 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f1c5c09b020 con 0x56472f7fefa0 2024-03-14T09:54:32.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.107+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3143003172 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56472f7f1720 con 0x56472f7fefa0 2024-03-14T09:54:32.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.108+0000 7f1c4f400700 1 -- 172.21.15.78:0/3143003172 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c5c0b3240 con 0x56472f7fefa0 2024-03-14T09:54:32.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.108+0000 7f1c886bdb80 10 client.25681 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:32.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.108+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3143003172 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56472f57eb60 con 0x56472f7fefa0 2024-03-14T09:54:32.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.108+0000 7f1c4f400700 1 -- 172.21.15.78:0/3143003172 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56472f7f1720 con 0x56472f7fefa0 2024-03-14T09:54:32.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.108+0000 7f1c4f400700 1 -- 172.21.15.78:0/3143003172 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1c5c099e10 con 0x56472f7fefa0 2024-03-14T09:54:32.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.108+0000 7f1c886bdb80 10 client.25681 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:32.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.108+0000 7f1c886bdb80 10 client.25681 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:54:32.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.108+0000 7f1c886bdb80 20 client.25681 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:32.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.108+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/3143003172 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x56472f81cbe0 0x56472f81efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.108+0000 7f1c886bdb80 4 client.25681 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:54:32.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.108+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3143003172 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56472f749700 con 0x56472f81cbe0 2024-03-14T09:54:32.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.108+0000 7f1c6de00700 1 --2- 172.21.15.78:0/3143003172 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x56472f81cbe0 0x56472f81efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.109+0000 7f1c4f400700 10 client.25681 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:54:32.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.109+0000 7f1c6de00700 1 --2- 172.21.15.78:0/3143003172 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x56472f81cbe0 0x56472f81efd0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=69cc9692c632329d server_cookie=47ae38e75f7bf7e5 in_seq=0 out_seq=0 2024-03-14T09:54:32.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.110+0000 7f1c4f400700 1 -- 172.21.15.78:0/3143003172 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56472f749700 con 0x56472f81cbe0 2024-03-14T09:54:32.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.110+0000 7f1c4f400700 10 client.25681 handle_command_reply: tid=0 2024-03-14T09:54:32.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.110+0000 7f1c886bdb80 1 client.25681 shutdown 2024-03-14T09:54:32.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.110+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3143003172 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fefa0 msgr2=0x56472f805ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.110+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/3143003172 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fefa0 0x56472f805ba0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c096e20 tx=0x7f1c5c0cb2d0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.110+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3143003172 shutdown_connections 2024-03-14T09:54:32.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.110+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/3143003172 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x56472f81cbe0 0x56472f81efd0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.110+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/3143003172 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f7fe680 0x56472f8088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.110+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/3143003172 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f808ec0 0x56472f806220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.110+0000 7f1c4f400700 0 client.25681 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:54:32.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.110+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/3143003172 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fefa0 0x56472f805ba0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.110+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3143003172 >> 172.21.15.78:0/3143003172 conn(0x56472f774220 msgr2=0x56472f7f9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:32.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.110+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3143003172 shutdown_connections 2024-03-14T09:54:32.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.110+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3143003172 wait complete. 2024-03-14T09:54:32.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.110+0000 7f1c886bdb80 20 client.25681 trim_cache size 0 max 16384 2024-03-14T09:54:32.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.123+0000 7f1c886bdb80 1 Processor -- start 2024-03-14T09:54:32.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.124+0000 7f1c886bdb80 1 -- start start 2024-03-14T09:54:32.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.124+0000 7f1c886bdb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f7fe680 0x56472f814020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.124+0000 7f1c886bdb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fefa0 0x56472f7f95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.124+0000 7f1c886bdb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f808ec0 0x56472f7f9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.124+0000 7f1c886bdb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56472f800390 con 0x56472f808ec0 2024-03-14T09:54:32.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.124+0000 7f1c886bdb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56472f7fa030 con 0x56472f7fe680 2024-03-14T09:54:32.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.124+0000 7f1c886bdb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56472f805970 con 0x56472f7fefa0 2024-03-14T09:54:32.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.124+0000 7f1c6e800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f808ec0 0x56472f7f9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.124+0000 7f1c6d400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fefa0 0x56472f7f95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.124+0000 7f1c6d400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fefa0 0x56472f7f95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37290/0 (socket says 172.21.15.78:37290) 2024-03-14T09:54:32.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.124+0000 7f1c6e800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f808ec0 0x56472f7f9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45700/0 (socket says 172.21.15.78:45700) 2024-03-14T09:54:32.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.124+0000 7f1c6d400700 1 -- 172.21.15.78:0/3717118285 learned_addr learned my addr 172.21.15.78:0/3717118285 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:32.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.124+0000 7f1c6de00700 1 --2- 172.21.15.78:0/3717118285 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f7fe680 0x56472f814020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.125+0000 7f1c6d400700 1 -- 172.21.15.78:0/3717118285 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f7fe680 msgr2=0x56472f814020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:32.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.125+0000 7f1c6d400700 1 --2- 172.21.15.78:0/3717118285 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f7fe680 0x56472f814020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.125+0000 7f1c6d400700 1 -- 172.21.15.78:0/3717118285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f808ec0 msgr2=0x56472f7f9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:32.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.125+0000 7f1c6d400700 1 --2- 172.21.15.78:0/3717118285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f808ec0 0x56472f7f9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.125+0000 7f1c6d400700 1 -- 172.21.15.78:0/3717118285 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56472f716ea0 con 0x56472f7fefa0 2024-03-14T09:54:32.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.125+0000 7f1c6d400700 1 --2- 172.21.15.78:0/3717118285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fefa0 0x56472f7f95b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c098c70 tx=0x7f1c5c005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e672153cfde6197 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:32.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.125+0000 7f1c5ba00700 1 -- 172.21.15.78:0/3717118285 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0cac80 con 0x56472f7fefa0 2024-03-14T09:54:32.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.125+0000 7f1c5ba00700 1 -- 172.21.15.78:0/3717118285 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c5c0b3240 con 0x56472f7fefa0 2024-03-14T09:54:32.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.125+0000 7f1c5ba00700 1 -- 172.21.15.78:0/3717118285 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0ba350 con 0x56472f7fefa0 2024-03-14T09:54:32.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.126+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3717118285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fefa0 msgr2=0x56472f7f95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.126+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/3717118285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fefa0 0x56472f7f95b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c098c70 tx=0x7f1c5c005c90 comp rx=0 tx=0).stop 2024-03-14T09:54:32.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.126+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3717118285 shutdown_connections 2024-03-14T09:54:32.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.126+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/3717118285 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f7fe680 0x56472f814020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.126+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/3717118285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f808ec0 0x56472f7f9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.126+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/3717118285 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f7fefa0 0x56472f7f95b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.126+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3717118285 >> 172.21.15.78:0/3717118285 conn(0x56472f774220 msgr2=0x56472f800e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:32.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.126+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3717118285 shutdown_connections 2024-03-14T09:54:32.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.126+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3717118285 wait complete. 2024-03-14T09:54:32.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.127+0000 7f1c886bdb80 1 -- start start 2024-03-14T09:54:32.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.127+0000 7f1c886bdb80 1 Processor -- start 2024-03-14T09:54:32.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.127+0000 7f1c886bdb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f7fe680 0x56472f813c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.127+0000 7f1c886bdb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f7fefa0 0x56472f812080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.127+0000 7f1c886bdb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f808ec0 0x56472f8125c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.127+0000 7f1c886bdb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56472f812b00 con 0x56472f7fe680 2024-03-14T09:54:32.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.127+0000 7f1c886bdb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56472f812c40 con 0x56472f7fefa0 2024-03-14T09:54:32.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.127+0000 7f1c886bdb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56472f812e00 con 0x56472f808ec0 2024-03-14T09:54:32.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.127+0000 7f1c886bdb80 10 client.0 _ll_register_callbacks cb 0x56472f5583a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:32.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.127+0000 7f1c6de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f7fe680 0x56472f813c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.127+0000 7f1c6de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f7fe680 0x56472f813c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45704/0 (socket says 172.21.15.78:45704) 2024-03-14T09:54:32.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.127+0000 7f1c6de00700 1 -- 172.21.15.78:0/3374926027 learned_addr learned my addr 172.21.15.78:0/3374926027 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:32.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.128+0000 7f1c6d400700 1 --2- 172.21.15.78:0/3374926027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f7fefa0 0x56472f812080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.128+0000 7f1c6e800700 1 --2- 172.21.15.78:0/3374926027 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f808ec0 0x56472f8125c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.128+0000 7f1c6de00700 1 -- 172.21.15.78:0/3374926027 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f808ec0 msgr2=0x56472f8125c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.128+0000 7f1c6de00700 1 --2- 172.21.15.78:0/3374926027 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f808ec0 0x56472f8125c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.128+0000 7f1c6de00700 1 -- 172.21.15.78:0/3374926027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f7fefa0 msgr2=0x56472f812080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.128+0000 7f1c6de00700 1 --2- 172.21.15.78:0/3374926027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f7fefa0 0x56472f812080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.128+0000 7f1c6de00700 1 -- 172.21.15.78:0/3374926027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56472f57eb60 con 0x56472f7fe680 2024-03-14T09:54:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.128+0000 7f1c6de00700 1 --2- 172.21.15.78:0/3374926027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f7fe680 0x56472f813c90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1c60026f50 tx=0x7f1c600091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a942df62a5a6b4c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.128+0000 7f1c4f400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:54:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.128+0000 7f1c4f400700 1 -- 172.21.15.78:0/3374926027 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6002b030 con 0x56472f7fe680 2024-03-14T09:54:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.128+0000 7f1c4f400700 1 -- 172.21.15.78:0/3374926027 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f1c60032c10 con 0x56472f7fe680 2024-03-14T09:54:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.128+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3374926027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1c60003e70 con 0x56472f7fe680 2024-03-14T09:54:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.129+0000 7f1c4f400700 1 -- 172.21.15.78:0/3374926027 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c60032430 con 0x56472f7fe680 2024-03-14T09:54:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.129+0000 7f1c886bdb80 10 client.15675 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.129+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3374926027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56472f5d6cd0 con 0x56472f7fe680 2024-03-14T09:54:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.129+0000 7f1c4f400700 1 -- 172.21.15.78:0/3374926027 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c60003e70 con 0x56472f7fe680 2024-03-14T09:54:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.129+0000 7f1c4f400700 1 -- 172.21.15.78:0/3374926027 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1c60024020 con 0x56472f7fe680 2024-03-14T09:54:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.129+0000 7f1c886bdb80 10 client.15675 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:32.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.129+0000 7f1c886bdb80 10 client.15675 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:54:32.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.129+0000 7f1c886bdb80 20 client.15675 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:32.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.129+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/3374926027 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x56472f86fb30 0x56472f871f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.129+0000 7f1c886bdb80 4 client.15675 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:54:32.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.129+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3374926027 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56472f59fcf0 con 0x56472f86fb30 2024-03-14T09:54:32.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.129+0000 7f1c6d400700 1 --2- 172.21.15.78:0/3374926027 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x56472f86fb30 0x56472f871f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.130+0000 7f1c4f400700 10 client.15675 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:54:32.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.130+0000 7f1c6d400700 1 --2- 172.21.15.78:0/3374926027 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x56472f86fb30 0x56472f871f50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=52ba41c0397dde9a server_cookie=95a3500c532dee26 in_seq=0 out_seq=0 2024-03-14T09:54:32.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.130+0000 7f1c4f400700 1 -- 172.21.15.78:0/3374926027 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56472f59fcf0 con 0x56472f86fb30 2024-03-14T09:54:32.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.130+0000 7f1c4f400700 10 client.15675 handle_command_reply: tid=0 2024-03-14T09:54:32.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.130+0000 7f1c886bdb80 1 client.15675 shutdown 2024-03-14T09:54:32.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.130+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3374926027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f7fe680 msgr2=0x56472f813c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.130+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/3374926027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f7fe680 0x56472f813c90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1c60026f50 tx=0x7f1c600091e0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.131+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3374926027 shutdown_connections 2024-03-14T09:54:32.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.131+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/3374926027 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x56472f86fb30 0x56472f871f50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.131+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/3374926027 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56472f808ec0 0x56472f8125c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.131+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/3374926027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56472f7fe680 0x56472f813c90 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.131+0000 7f1c4f400700 0 client.15675 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:54:32.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.131+0000 7f1c886bdb80 1 --2- 172.21.15.78:0/3374926027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56472f7fefa0 0x56472f812080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.131+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3374926027 >> 172.21.15.78:0/3374926027 conn(0x56472f774220 msgr2=0x56472f800e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:32.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.131+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3374926027 shutdown_connections 2024-03-14T09:54:32.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.131+0000 7f1c886bdb80 1 -- 172.21.15.78:0/3374926027 wait complete. 2024-03-14T09:54:32.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.131+0000 7f1c886bdb80 20 client.15675 trim_cache size 0 max 16384 2024-03-14T09:54:32.135 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:54:32.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.132+0000 7f1c74c00700 1 -- 172.21.15.78:0/190001205 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1c540a0d10 msgr2=0x7f1c540a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.132+0000 7f1c74c00700 1 --2- 172.21.15.78:0/190001205 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1c540a0d10 0x7f1c540a31d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1c60011490 tx=0x7f1c60009260 comp rx=0 tx=0).stop 2024-03-14T09:54:32.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.132+0000 7f1c74c00700 1 -- 172.21.15.78:0/190001205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c70106040 msgr2=0x7f1c70197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.132+0000 7f1c74c00700 1 --2- 172.21.15.78:0/190001205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c70106040 0x7f1c70197b80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c000c00 tx=0x7f1c5c004730 comp rx=0 tx=0).stop 2024-03-14T09:54:32.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.132+0000 7f1c74c00700 1 -- 172.21.15.78:0/190001205 shutdown_connections 2024-03-14T09:54:32.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.132+0000 7f1c74c00700 1 --2- 172.21.15.78:0/190001205 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1c540a0d10 0x7f1c540a31d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.132+0000 7f1c74c00700 1 --2- 172.21.15.78:0/190001205 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c7019bed0 0x7f1c7019c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.132+0000 7f1c74c00700 1 --2- 172.21.15.78:0/190001205 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c70100f20 0x7f1c70197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.132+0000 7f1c74c00700 1 --2- 172.21.15.78:0/190001205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c70106040 0x7f1c70197b80 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.132+0000 7f1c74c00700 1 -- 172.21.15.78:0/190001205 >> 172.21.15.78:0/190001205 conn(0x7f1c700fc7a0 msgr2=0x7f1c700fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:32.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.132+0000 7f1c74c00700 1 -- 172.21.15.78:0/190001205 shutdown_connections 2024-03-14T09:54:32.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.132+0000 7f1c74c00700 1 -- 172.21.15.78:0/190001205 wait complete. 2024-03-14T09:54:32.147 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:32 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3616246567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:54:32.150 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T09:54:32.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:32 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3616246567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:54:32.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.354+0000 7fd749a00700 1 -- 172.21.15.78:0/1241199007 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7441014b0 msgr2=0x7fd74410d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.354+0000 7fd749a00700 1 --2- 172.21.15.78:0/1241199007 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7441014b0 0x7fd74410d9c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00b0a0 tx=0x7fd73c00b3b0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.354+0000 7fd749a00700 1 -- 172.21.15.78:0/1241199007 shutdown_connections 2024-03-14T09:54:32.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.354+0000 7fd749a00700 1 --2- 172.21.15.78:0/1241199007 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd744106b90 0x7fd744106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.354+0000 7fd749a00700 1 --2- 172.21.15.78:0/1241199007 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd7441014b0 0x7fd74410d9c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.354+0000 7fd749a00700 1 --2- 172.21.15.78:0/1241199007 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd744100af0 0x7fd744100f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.354+0000 7fd749a00700 1 -- 172.21.15.78:0/1241199007 >> 172.21.15.78:0/1241199007 conn(0x7fd7440fc640 msgr2=0x7fd7440fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:32.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.354+0000 7fd749a00700 1 -- 172.21.15.78:0/1241199007 shutdown_connections 2024-03-14T09:54:32.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.354+0000 7fd749a00700 1 -- 172.21.15.78:0/1241199007 wait complete. 2024-03-14T09:54:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.355+0000 7fd749a00700 1 Processor -- start 2024-03-14T09:54:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.355+0000 7fd749a00700 1 -- start start 2024-03-14T09:54:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.356+0000 7fd749a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd744100af0 0x7fd744199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.356+0000 7fd749a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7441014b0 0x7fd744199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.356+0000 7fd749a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd744106b90 0x7fd74419de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.356+0000 7fd749a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74419e440 con 0x7fd7441014b0 2024-03-14T09:54:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.356+0000 7fd749a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74419e5b0 con 0x7fd744100af0 2024-03-14T09:54:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.356+0000 7fd749a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74419e770 con 0x7fd744106b90 2024-03-14T09:54:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.356+0000 7fd743e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd744100af0 0x7fd744199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.356+0000 7fd743400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7441014b0 0x7fd744199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.356+0000 7fd743e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd744100af0 0x7fd744199520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:39768/0 (socket says 172.21.15.78:39768) 2024-03-14T09:54:32.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.356+0000 7fd743e00700 1 -- 172.21.15.78:0/2702201170 learned_addr learned my addr 172.21.15.78:0/2702201170 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:32.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.356+0000 7fd749000700 1 --2- 172.21.15.78:0/2702201170 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd744106b90 0x7fd74419de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.357+0000 7fd743e00700 1 -- 172.21.15.78:0/2702201170 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd744106b90 msgr2=0x7fd74419de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.357+0000 7fd743e00700 1 --2- 172.21.15.78:0/2702201170 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd744106b90 0x7fd74419de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.357+0000 7fd743e00700 1 -- 172.21.15.78:0/2702201170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7441014b0 msgr2=0x7fd744199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.357+0000 7fd743e00700 1 --2- 172.21.15.78:0/2702201170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7441014b0 0x7fd744199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.357+0000 7fd743e00700 1 -- 172.21.15.78:0/2702201170 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c009d00 con 0x7fd744100af0 2024-03-14T09:54:32.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.357+0000 7fd743400700 1 --2- 172.21.15.78:0/2702201170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7441014b0 0x7fd744199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:32.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.357+0000 7fd743e00700 1 --2- 172.21.15.78:0/2702201170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd744100af0 0x7fd744199520 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd73400e9a0 tx=0x7fd73400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:32.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.357+0000 7fd740c00700 1 -- 172.21.15.78:0/2702201170 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd734009790 con 0x7fd744100af0 2024-03-14T09:54:32.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.357+0000 7fd740c00700 1 -- 172.21.15.78:0/2702201170 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd734010430 con 0x7fd744100af0 2024-03-14T09:54:32.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.357+0000 7fd749a00700 1 -- 172.21.15.78:0/2702201170 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74419ea50 con 0x7fd744100af0 2024-03-14T09:54:32.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.357+0000 7fd740c00700 1 -- 172.21.15.78:0/2702201170 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd734018680 con 0x7fd744100af0 2024-03-14T09:54:32.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.357+0000 7fd749a00700 1 -- 172.21.15.78:0/2702201170 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7441a6290 con 0x7fd744100af0 2024-03-14T09:54:32.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.360+0000 7fd740c00700 1 -- 172.21.15.78:0/2702201170 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 25) v1 ==== 135564+0+0 (secure 0 0 0) 0x7fd7340105a0 con 0x7fd744100af0 2024-03-14T09:54:32.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.361+0000 7fd740c00700 1 --2- 172.21.15.78:0/2702201170 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd7280a0c90 0x7fd7280a3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.362+0000 7fd740c00700 1 -- 172.21.15.78:0/2702201170 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fd734014070 con 0x7fd744100af0 2024-03-14T09:54:32.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.362+0000 7fd743400700 1 --2- 172.21.15.78:0/2702201170 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd7280a0c90 0x7fd7280a3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.362+0000 7fd75eb21b80 1 Processor -- start 2024-03-14T09:54:32.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.362+0000 7fd75eb21b80 1 -- start start 2024-03-14T09:54:32.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd743400700 1 --2- 172.21.15.78:0/2702201170 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd7280a0c90 0x7fd7280a3150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd74419aa70 tx=0x7fd730009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:32.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd75eb21b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e7860 0x55a2ce6e7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd75eb21b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6e8180 0x55a2ce6f2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd75eb21b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd75eb21b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2ce641990 con 0x55a2ce6e7860 2024-03-14T09:54:32.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd75eb21b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2ce63f350 con 0x55a2ce6e8180 2024-03-14T09:54:32.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd75eb21b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2ce63ac40 con 0x55a2ce6f3030 2024-03-14T09:54:32.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd743e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e7860 0x55a2ce6e7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd743e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e7860 0x55a2ce6e7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45732/0 (socket says 172.21.15.78:45732) 2024-03-14T09:54:32.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd743e00700 1 -- 172.21.15.78:0/1239430340 learned_addr learned my addr 172.21.15.78:0/1239430340 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:32.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd743400700 1 --2- 172.21.15.78:0/1239430340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6e8180 0x55a2ce6f2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd749000700 1 --2- 172.21.15.78:0/1239430340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd743e00700 1 -- 172.21.15.78:0/1239430340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6f3030 msgr2=0x55a2ce6f5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:32.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd743e00700 1 --2- 172.21.15.78:0/1239430340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd743e00700 1 -- 172.21.15.78:0/1239430340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6e8180 msgr2=0x55a2ce6f2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:32.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd743e00700 1 --2- 172.21.15.78:0/1239430340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6e8180 0x55a2ce6f2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.363+0000 7fd743e00700 1 -- 172.21.15.78:0/1239430340 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2ce4bc3b0 con 0x55a2ce6e7860 2024-03-14T09:54:32.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.364+0000 7fd743e00700 1 --2- 172.21.15.78:0/1239430340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e7860 0x55a2ce6e7c40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd734099620 tx=0x7fd734099280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48942dd5ce4ad142 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:32.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.364+0000 7fd739600700 1 -- 172.21.15.78:0/1239430340 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd734099920 con 0x55a2ce6e7860 2024-03-14T09:54:32.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.364+0000 7fd739600700 1 -- 172.21.15.78:0/1239430340 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd73409d750 con 0x55a2ce6e7860 2024-03-14T09:54:32.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.364+0000 7fd739600700 1 -- 172.21.15.78:0/1239430340 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd734099070 con 0x55a2ce6e7860 2024-03-14T09:54:32.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.365+0000 7fd75eb21b80 1 -- 172.21.15.78:0/1239430340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e7860 msgr2=0x55a2ce6e7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.365+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/1239430340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e7860 0x55a2ce6e7c40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd734099620 tx=0x7fd734099280 comp rx=0 tx=0).stop 2024-03-14T09:54:32.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.365+0000 7fd75eb21b80 1 -- 172.21.15.78:0/1239430340 shutdown_connections 2024-03-14T09:54:32.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.365+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/1239430340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.365+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/1239430340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e7860 0x55a2ce6e7c40 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.365+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/1239430340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6e8180 0x55a2ce6f2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.365+0000 7fd75eb21b80 1 -- 172.21.15.78:0/1239430340 >> 172.21.15.78:0/1239430340 conn(0x55a2ce65e220 msgr2=0x55a2ce6e3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:32.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.365+0000 7fd75eb21b80 1 -- 172.21.15.78:0/1239430340 shutdown_connections 2024-03-14T09:54:32.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.365+0000 7fd75eb21b80 1 -- 172.21.15.78:0/1239430340 wait complete. 2024-03-14T09:54:32.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.365+0000 7fd75eb21b80 1 -- start start 2024-03-14T09:54:32.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.366+0000 7fd75eb21b80 1 Processor -- start 2024-03-14T09:54:32.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.366+0000 7fd75eb21b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e7860 0x55a2ce6f2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.366+0000 7fd75eb21b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e8180 0x55a2ce6efd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.366+0000 7fd75eb21b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.366+0000 7fd75eb21b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2ce5d6d80 con 0x55a2ce6e8180 2024-03-14T09:54:32.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.366+0000 7fd75eb21b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2ce5d6a20 con 0x55a2ce6f3030 2024-03-14T09:54:32.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.366+0000 7fd75eb21b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2ce4be6a0 con 0x55a2ce6e7860 2024-03-14T09:54:32.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.366+0000 7fd75eb21b80 10 client.0 _ll_register_callbacks cb 0x55a2ce4423a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:32.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.366+0000 7fd743400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e8180 0x55a2ce6efd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.367+0000 7fd743e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e7860 0x55a2ce6f2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.367+0000 7fd743400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e8180 0x55a2ce6efd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45746/0 (socket says 172.21.15.78:45746) 2024-03-14T09:54:32.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.367+0000 7fd743e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e7860 0x55a2ce6f2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37340/0 (socket says 172.21.15.78:37340) 2024-03-14T09:54:32.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.367+0000 7fd743400700 1 -- 172.21.15.78:0/224975390 learned_addr learned my addr 172.21.15.78:0/224975390 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:32.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.367+0000 7fd749000700 1 --2- 172.21.15.78:0/224975390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.367+0000 7fd743e00700 1 -- 172.21.15.78:0/224975390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6f3030 msgr2=0x55a2ce6f03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:32.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.367+0000 7fd743e00700 1 --2- 172.21.15.78:0/224975390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.367+0000 7fd743e00700 1 -- 172.21.15.78:0/224975390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e8180 msgr2=0x55a2ce6efd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:32.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.367+0000 7fd743e00700 1 --2- 172.21.15.78:0/224975390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e8180 0x55a2ce6efd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.367+0000 7fd743e00700 1 -- 172.21.15.78:0/224975390 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7441a6290 con 0x55a2ce6e7860 2024-03-14T09:54:32.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.367+0000 7fd743400700 1 --2- 172.21.15.78:0/224975390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e8180 0x55a2ce6efd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:32.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.367+0000 7fd743e00700 1 --2- 172.21.15.78:0/224975390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e7860 0x55a2ce6f2a50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd7340a1f60 tx=0x7fd734010b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73adeab7ef63c017 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:32.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.367+0000 7fd72e000700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:54:32.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.367+0000 7fd72e000700 1 -- 172.21.15.78:0/224975390 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd734099920 con 0x55a2ce6e7860 2024-03-14T09:54:32.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.367+0000 7fd75eb21b80 1 -- 172.21.15.78:0/224975390 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a2ce6db720 con 0x55a2ce6e7860 2024-03-14T09:54:32.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.368+0000 7fd72e000700 1 -- 172.21.15.78:0/224975390 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fd734097cc0 con 0x55a2ce6e7860 2024-03-14T09:54:32.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.368+0000 7fd72e000700 1 -- 172.21.15.78:0/224975390 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7340b2270 con 0x55a2ce6e7860 2024-03-14T09:54:32.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.368+0000 7fd75eb21b80 10 client.25699 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:32.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.368+0000 7fd75eb21b80 1 -- 172.21.15.78:0/224975390 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a2ce468b60 con 0x55a2ce6e7860 2024-03-14T09:54:32.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.368+0000 7fd72e000700 1 -- 172.21.15.78:0/224975390 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd73407c080 con 0x55a2ce6e7860 2024-03-14T09:54:32.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.368+0000 7fd72e000700 1 -- 172.21.15.78:0/224975390 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd734095710 con 0x55a2ce6e7860 2024-03-14T09:54:32.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.368+0000 7fd75eb21b80 10 client.25699 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:32.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.368+0000 7fd75eb21b80 10 client.25699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:32.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.368+0000 7fd75eb21b80 20 client.25699 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:32.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.368+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/224975390 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a2ce702800 0x55a2ce704c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.368+0000 7fd75eb21b80 4 client.25699 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:54:32.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.368+0000 7fd75eb21b80 1 -- 172.21.15.78:0/224975390 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a2ce633700 con 0x55a2ce702800 2024-03-14T09:54:32.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.369+0000 7fd743400700 1 --2- 172.21.15.78:0/224975390 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a2ce702800 0x55a2ce704c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.369+0000 7fd743400700 1 --2- 172.21.15.78:0/224975390 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a2ce702800 0x55a2ce704c20 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91b0c17110171ea6 server_cookie=a95117abf0909696 in_seq=0 out_seq=0 2024-03-14T09:54:32.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.369+0000 7fd72e000700 10 client.25699 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:32.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.370+0000 7fd72e000700 1 -- 172.21.15.78:0/224975390 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55a2ce633700 con 0x55a2ce702800 2024-03-14T09:54:32.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.370+0000 7fd72e000700 10 client.25699 handle_command_reply: tid=0 2024-03-14T09:54:32.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.370+0000 7fd75eb21b80 1 client.25699 shutdown 2024-03-14T09:54:32.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.370+0000 7fd75eb21b80 1 -- 172.21.15.78:0/224975390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e7860 msgr2=0x55a2ce6f2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.370+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/224975390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e7860 0x55a2ce6f2a50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd7340a1f60 tx=0x7fd734010b80 comp rx=0 tx=0).stop 2024-03-14T09:54:32.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.370+0000 7fd75eb21b80 1 -- 172.21.15.78:0/224975390 shutdown_connections 2024-03-14T09:54:32.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.370+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/224975390 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a2ce702800 0x55a2ce704c20 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.370+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/224975390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.370+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/224975390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e7860 0x55a2ce6f2a50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.370+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/224975390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e8180 0x55a2ce6efd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.370+0000 7fd72e000700 0 client.25699 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:32.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.370+0000 7fd75eb21b80 1 -- 172.21.15.78:0/224975390 >> 172.21.15.78:0/224975390 conn(0x55a2ce65e220 msgr2=0x55a2ce6e3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:32.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.370+0000 7fd75eb21b80 1 -- 172.21.15.78:0/224975390 shutdown_connections 2024-03-14T09:54:32.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.370+0000 7fd75eb21b80 1 -- 172.21.15.78:0/224975390 wait complete. 2024-03-14T09:54:32.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.371+0000 7fd75eb21b80 20 client.25699 trim_cache size 0 max 16384 2024-03-14T09:54:32.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.385+0000 7fd75eb21b80 1 Processor -- start 2024-03-14T09:54:32.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.385+0000 7fd75eb21b80 1 -- start start 2024-03-14T09:54:32.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.386+0000 7fd75eb21b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e7860 0x55a2ce6e3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.386+0000 7fd75eb21b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e8180 0x55a2ce6efd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.386+0000 7fd75eb21b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.386+0000 7fd75eb21b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2ce6e4860 con 0x55a2ce6e7860 2024-03-14T09:54:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.386+0000 7fd75eb21b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2ce6e5020 con 0x55a2ce6f3030 2024-03-14T09:54:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.386+0000 7fd75eb21b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2ce6ddd00 con 0x55a2ce6e8180 2024-03-14T09:54:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.386+0000 7fd743e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e7860 0x55a2ce6e3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.386+0000 7fd743e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e7860 0x55a2ce6e3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45752/0 (socket says 172.21.15.78:45752) 2024-03-14T09:54:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.386+0000 7fd743e00700 1 -- 172.21.15.78:0/481109506 learned_addr learned my addr 172.21.15.78:0/481109506 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.386+0000 7fd743400700 1 --2- 172.21.15.78:0/481109506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e8180 0x55a2ce6efd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.386+0000 7fd749000700 1 --2- 172.21.15.78:0/481109506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.387+0000 7fd743e00700 1 -- 172.21.15.78:0/481109506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e8180 msgr2=0x55a2ce6efd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.387+0000 7fd743e00700 1 --2- 172.21.15.78:0/481109506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e8180 0x55a2ce6efd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.387+0000 7fd743e00700 1 -- 172.21.15.78:0/481109506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6f3030 msgr2=0x55a2ce6f2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.387+0000 7fd743e00700 1 --2- 172.21.15.78:0/481109506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.387+0000 7fd743e00700 1 -- 172.21.15.78:0/481109506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2ce600ea0 con 0x55a2ce6e7860 2024-03-14T09:54:32.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.387+0000 7fd743e00700 1 --2- 172.21.15.78:0/481109506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e7860 0x55a2ce6e3e50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd7340998b0 tx=0x7fd734010bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=522bcb6c29c26c1a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:32.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.387+0000 7fd73a000700 1 -- 172.21.15.78:0/481109506 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd734098880 con 0x55a2ce6e7860 2024-03-14T09:54:32.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.387+0000 7fd73a000700 1 -- 172.21.15.78:0/481109506 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a2ce6db720 con 0x55a2ce6e7860 2024-03-14T09:54:32.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.387+0000 7fd73a000700 1 -- 172.21.15.78:0/481109506 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd734095710 con 0x55a2ce6e7860 2024-03-14T09:54:32.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.388+0000 7fd75eb21b80 1 -- 172.21.15.78:0/481109506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e7860 msgr2=0x55a2ce6e3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.388+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/481109506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e7860 0x55a2ce6e3e50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd7340998b0 tx=0x7fd734010bf0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.388+0000 7fd75eb21b80 1 -- 172.21.15.78:0/481109506 shutdown_connections 2024-03-14T09:54:32.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.388+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/481109506 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.388+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/481109506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e7860 0x55a2ce6e3e50 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.388+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/481109506 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e8180 0x55a2ce6efd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.388+0000 7fd75eb21b80 1 -- 172.21.15.78:0/481109506 >> 172.21.15.78:0/481109506 conn(0x55a2ce65e220 msgr2=0x55a2ce6df680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:32.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.388+0000 7fd75eb21b80 1 -- 172.21.15.78:0/481109506 shutdown_connections 2024-03-14T09:54:32.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.388+0000 7fd75eb21b80 1 -- 172.21.15.78:0/481109506 wait complete. 2024-03-14T09:54:32.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.388+0000 7fd75eb21b80 1 -- start start 2024-03-14T09:54:32.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.389+0000 7fd75eb21b80 1 Processor -- start 2024-03-14T09:54:32.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.389+0000 7fd75eb21b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e7860 0x55a2ce6f7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.389+0000 7fd75eb21b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e8180 0x55a2ce6f8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.389+0000 7fd75eb21b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.389+0000 7fd75eb21b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2ce753450 con 0x55a2ce6e8180 2024-03-14T09:54:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.389+0000 7fd75eb21b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2ce7535c0 con 0x55a2ce6f3030 2024-03-14T09:54:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.389+0000 7fd75eb21b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2ce753780 con 0x55a2ce6e7860 2024-03-14T09:54:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.389+0000 7fd75eb21b80 10 client.0 _ll_register_callbacks cb 0x7fd730004030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.389+0000 7fd743400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e8180 0x55a2ce6f8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.389+0000 7fd743400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e8180 0x55a2ce6f8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45760/0 (socket says 172.21.15.78:45760) 2024-03-14T09:54:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.389+0000 7fd743e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e7860 0x55a2ce6f7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.389+0000 7fd743400700 1 -- 172.21.15.78:0/3456263953 learned_addr learned my addr 172.21.15.78:0/3456263953 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.390+0000 7fd749000700 1 --2- 172.21.15.78:0/3456263953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.390+0000 7fd743400700 1 -- 172.21.15.78:0/3456263953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e7860 msgr2=0x55a2ce6f7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.390+0000 7fd743400700 1 --2- 172.21.15.78:0/3456263953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e7860 0x55a2ce6f7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.390+0000 7fd743400700 1 -- 172.21.15.78:0/3456263953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6f3030 msgr2=0x55a2ce6f8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.390+0000 7fd743400700 1 --2- 172.21.15.78:0/3456263953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.390+0000 7fd743400700 1 -- 172.21.15.78:0/3456263953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a2ce468b60 con 0x55a2ce6e8180 2024-03-14T09:54:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.390+0000 7fd749000700 1 --2- 172.21.15.78:0/3456263953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.390+0000 7fd743e00700 1 --2- 172.21.15.78:0/3456263953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e7860 0x55a2ce6f7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.390+0000 7fd743400700 1 --2- 172.21.15.78:0/3456263953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e8180 0x55a2ce6f8120 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd730005d90 tx=0x7fd7300047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c54124f75adc141 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.390+0000 7fd72e000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:54:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.390+0000 7fd72e000700 1 -- 172.21.15.78:0/3456263953 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730024040 con 0x55a2ce6e8180 2024-03-14T09:54:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.391+0000 7fd72e000700 1 -- 172.21.15.78:0/3456263953 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fd730029990 con 0x55a2ce6e8180 2024-03-14T09:54:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.391+0000 7fd75eb21b80 1 -- 172.21.15.78:0/3456263953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd73409d750 con 0x55a2ce6e8180 2024-03-14T09:54:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.391+0000 7fd72e000700 1 -- 172.21.15.78:0/3456263953 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd730003ea0 con 0x55a2ce6e8180 2024-03-14T09:54:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.391+0000 7fd75eb21b80 10 client.15693 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.391+0000 7fd75eb21b80 1 -- 172.21.15.78:0/3456263953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a2ce4c0cd0 con 0x55a2ce6e8180 2024-03-14T09:54:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.391+0000 7fd72e000700 1 -- 172.21.15.78:0/3456263953 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd73409d750 con 0x55a2ce6e8180 2024-03-14T09:54:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.391+0000 7fd72e000700 1 -- 172.21.15.78:0/3456263953 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd73000e3b0 con 0x55a2ce6e8180 2024-03-14T09:54:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.391+0000 7fd75eb21b80 10 client.15693 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:32.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.391+0000 7fd75eb21b80 10 client.15693 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:32.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.391+0000 7fd75eb21b80 20 client.15693 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:32.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.391+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/3456263953 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a2ce754f30 0x55a2ce757350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:32.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.391+0000 7fd75eb21b80 4 client.15693 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:54:32.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.391+0000 7fd75eb21b80 1 -- 172.21.15.78:0/3456263953 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a2ce489cf0 con 0x55a2ce754f30 2024-03-14T09:54:32.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.392+0000 7fd743e00700 1 --2- 172.21.15.78:0/3456263953 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a2ce754f30 0x55a2ce757350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:32.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.392+0000 7fd743e00700 1 --2- 172.21.15.78:0/3456263953 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a2ce754f30 0x55a2ce757350 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c0494ebf18a3bcc server_cookie=f0671c8141684aaa in_seq=0 out_seq=0 2024-03-14T09:54:32.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.392+0000 7fd72e000700 10 client.15693 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:32.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.392+0000 7fd72e000700 1 -- 172.21.15.78:0/3456263953 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a2ce489cf0 con 0x55a2ce754f30 2024-03-14T09:54:32.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.392+0000 7fd72e000700 10 client.15693 handle_command_reply: tid=0 2024-03-14T09:54:32.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.392+0000 7fd75eb21b80 1 client.15693 shutdown 2024-03-14T09:54:32.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.393+0000 7fd75eb21b80 1 -- 172.21.15.78:0/3456263953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e8180 msgr2=0x55a2ce6f8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.393+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/3456263953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e8180 0x55a2ce6f8120 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd730005d90 tx=0x7fd7300047c0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.393+0000 7fd75eb21b80 1 -- 172.21.15.78:0/3456263953 shutdown_connections 2024-03-14T09:54:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.393+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/3456263953 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a2ce754f30 0x55a2ce757350 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.393+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/3456263953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2ce6f3030 0x55a2ce6f8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.393+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/3456263953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2ce6e7860 0x55a2ce6f7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.393+0000 7fd72e000700 0 client.15693 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.393+0000 7fd75eb21b80 1 --2- 172.21.15.78:0/3456263953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2ce6e8180 0x55a2ce6f8120 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.393+0000 7fd75eb21b80 1 -- 172.21.15.78:0/3456263953 >> 172.21.15.78:0/3456263953 conn(0x55a2ce65e220 msgr2=0x55a2ce6ee270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.393+0000 7fd75eb21b80 1 -- 172.21.15.78:0/3456263953 shutdown_connections 2024-03-14T09:54:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.393+0000 7fd75eb21b80 1 -- 172.21.15.78:0/3456263953 wait complete. 2024-03-14T09:54:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.393+0000 7fd75eb21b80 20 client.15693 trim_cache size 0 max 16384 2024-03-14T09:54:32.397 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:54:32.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.394+0000 7fd749a00700 1 -- 172.21.15.78:0/2702201170 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd7280a0c90 msgr2=0x7fd7280a3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.394+0000 7fd749a00700 1 --2- 172.21.15.78:0/2702201170 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd7280a0c90 0x7fd7280a3150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd74419aa70 tx=0x7fd730009260 comp rx=0 tx=0).stop 2024-03-14T09:54:32.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.394+0000 7fd749a00700 1 -- 172.21.15.78:0/2702201170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd744100af0 msgr2=0x7fd744199520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:32.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.394+0000 7fd749a00700 1 --2- 172.21.15.78:0/2702201170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd744100af0 0x7fd744199520 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd73400e9a0 tx=0x7fd73400ecb0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.394+0000 7fd749a00700 1 -- 172.21.15.78:0/2702201170 shutdown_connections 2024-03-14T09:54:32.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.394+0000 7fd749a00700 1 --2- 172.21.15.78:0/2702201170 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd7280a0c90 0x7fd7280a3150 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.394+0000 7fd749a00700 1 --2- 172.21.15.78:0/2702201170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd744100af0 0x7fd744199520 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.394+0000 7fd749a00700 1 --2- 172.21.15.78:0/2702201170 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd744106b90 0x7fd74419de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.394+0000 7fd749a00700 1 --2- 172.21.15.78:0/2702201170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7441014b0 0x7fd744199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:32.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.394+0000 7fd749a00700 1 -- 172.21.15.78:0/2702201170 >> 172.21.15.78:0/2702201170 conn(0x7fd7440fc640 msgr2=0x7fd7440fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:32.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.394+0000 7fd749a00700 1 -- 172.21.15.78:0/2702201170 shutdown_connections 2024-03-14T09:54:32.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:32.394+0000 7fd749a00700 1 -- 172.21.15.78:0/2702201170 wait complete. 2024-03-14T09:54:32.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:32 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3616246567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:54:33.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:33 smithi132 ceph-mon[47959]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T09:54:33.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:33 smithi078 ceph-mon[38143]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T09:54:33.413 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T09:54:33.414 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T09:54:33.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:33 smithi152 ceph-mon[48279]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T09:54:33.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.641+0000 7fea53200700 1 -- 172.21.15.78:0/2636333025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea540ff8a0 msgr2=0x7fea540ffc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.641+0000 7fea53200700 1 --2- 172.21.15.78:0/2636333025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea540ff8a0 0x7fea540ffc80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fea44009990 tx=0x7fea44009ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.641+0000 7fea53200700 1 -- 172.21.15.78:0/2636333025 shutdown_connections 2024-03-14T09:54:33.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.641+0000 7fea53200700 1 --2- 172.21.15.78:0/2636333025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea5410d450 0x7fea5410f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.641+0000 7fea53200700 1 --2- 172.21.15.78:0/2636333025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea540ff8a0 0x7fea540ffc80 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.641+0000 7fea53200700 1 --2- 172.21.15.78:0/2636333025 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea54100250 0x7fea5410cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.641+0000 7fea53200700 1 -- 172.21.15.78:0/2636333025 >> 172.21.15.78:0/2636333025 conn(0x7fea540fb4e0 msgr2=0x7fea540fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:33.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.642+0000 7fea53200700 1 -- 172.21.15.78:0/2636333025 shutdown_connections 2024-03-14T09:54:33.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.642+0000 7fea53200700 1 -- 172.21.15.78:0/2636333025 wait complete. 2024-03-14T09:54:33.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.643+0000 7fea53200700 1 Processor -- start 2024-03-14T09:54:33.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.643+0000 7fea53200700 1 -- start start 2024-03-14T09:54:33.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.643+0000 7fea53200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea540ff8a0 0x7fea54199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.644+0000 7fea53200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea54100250 0x7fea54199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.644+0000 7fea53200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea5410d450 0x7fea5419de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.644+0000 7fea53200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5419e4d0 con 0x7fea54100250 2024-03-14T09:54:33.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.644+0000 7fea53200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5419e640 con 0x7fea5410d450 2024-03-14T09:54:33.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.644+0000 7fea53200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5419e800 con 0x7fea540ff8a0 2024-03-14T09:54:33.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.644+0000 7fea52800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea5410d450 0x7fea5419de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.644+0000 7fea51400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea54100250 0x7fea54199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.644+0000 7fea51e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea540ff8a0 0x7fea54199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.644+0000 7fea51400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea54100250 0x7fea54199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45772/0 (socket says 172.21.15.78:45772) 2024-03-14T09:54:33.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.644+0000 7fea51400700 1 -- 172.21.15.78:0/408144474 learned_addr learned my addr 172.21.15.78:0/408144474 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:33.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.644+0000 7fea51400700 1 -- 172.21.15.78:0/408144474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea540ff8a0 msgr2=0x7fea54199600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.644+0000 7fea51400700 1 --2- 172.21.15.78:0/408144474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea540ff8a0 0x7fea54199600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.644+0000 7fea51400700 1 -- 172.21.15.78:0/408144474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea5410d450 msgr2=0x7fea5419de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.644+0000 7fea51400700 1 --2- 172.21.15.78:0/408144474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea5410d450 0x7fea5419de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.645+0000 7fea51400700 1 -- 172.21.15.78:0/408144474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea440095f0 con 0x7fea54100250 2024-03-14T09:54:33.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.645+0000 7fea51400700 1 --2- 172.21.15.78:0/408144474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea54100250 0x7fea54199b40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fea4000e9a0 tx=0x7fea4000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:33.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.645+0000 7fea4aa00700 1 -- 172.21.15.78:0/408144474 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea40009790 con 0x7fea54100250 2024-03-14T09:54:33.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.645+0000 7fea53200700 1 -- 172.21.15.78:0/408144474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea5419eae0 con 0x7fea54100250 2024-03-14T09:54:33.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.645+0000 7fea4aa00700 1 -- 172.21.15.78:0/408144474 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea4000cba0 con 0x7fea54100250 2024-03-14T09:54:33.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.645+0000 7fea4aa00700 1 -- 172.21.15.78:0/408144474 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4001fca0 con 0x7fea54100250 2024-03-14T09:54:33.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.646+0000 7fea53200700 1 -- 172.21.15.78:0/408144474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea54101c90 con 0x7fea54100250 2024-03-14T09:54:33.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.649+0000 7fea4aa00700 1 -- 172.21.15.78:0/408144474 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 135564+0+0 (secure 0 0 0) 0x7fea4000c6e0 con 0x7fea54100250 2024-03-14T09:54:33.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.650+0000 7fea4aa00700 1 --2- 172.21.15.78:0/408144474 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fea380a0d10 0x7fea380a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.650+0000 7fea4aa00700 1 -- 172.21.15.78:0/408144474 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fea400cc740 con 0x7fea54100250 2024-03-14T09:54:33.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.651+0000 7fea51e00700 1 --2- 172.21.15.78:0/408144474 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fea380a0d10 0x7fea380a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.651+0000 7fea51e00700 1 --2- 172.21.15.78:0/408144474 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fea380a0d10 0x7fea380a31d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fea4400b2e0 tx=0x7fea44005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:33.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.652+0000 7fea6cbb6b80 1 Processor -- start 2024-03-14T09:54:33.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.652+0000 7fea6cbb6b80 1 -- start start 2024-03-14T09:54:33.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.652+0000 7fea6cbb6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f358e0 0x55d218f35cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.652+0000 7fea6cbb6b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f36200 0x55d218f40b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.652+0000 7fea6cbb6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f410b0 0x55d218f434a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.653+0000 7fea6cbb6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d218e8fa10 con 0x55d218f358e0 2024-03-14T09:54:33.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.653+0000 7fea6cbb6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d218e8d3d0 con 0x55d218f410b0 2024-03-14T09:54:33.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.653+0000 7fea6cbb6b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d218e88cc0 con 0x55d218f36200 2024-03-14T09:54:33.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.653+0000 7fea51e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f358e0 0x55d218f35cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.653+0000 7fea51e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f358e0 0x55d218f35cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45778/0 (socket says 172.21.15.78:45778) 2024-03-14T09:54:33.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.653+0000 7fea51e00700 1 -- 172.21.15.78:0/962117720 learned_addr learned my addr 172.21.15.78:0/962117720 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:33.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.653+0000 7fea51400700 1 --2- 172.21.15.78:0/962117720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f36200 0x55d218f40b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.653+0000 7fea52800700 1 --2- 172.21.15.78:0/962117720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f410b0 0x55d218f434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.653+0000 7fea51400700 1 -- 172.21.15.78:0/962117720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f410b0 msgr2=0x55d218f434a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:33.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.653+0000 7fea51400700 1 --2- 172.21.15.78:0/962117720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f410b0 0x55d218f434a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.653+0000 7fea51400700 1 -- 172.21.15.78:0/962117720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f358e0 msgr2=0x55d218f35cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:33.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.653+0000 7fea51400700 1 --2- 172.21.15.78:0/962117720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f358e0 0x55d218f35cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.653+0000 7fea51400700 1 -- 172.21.15.78:0/962117720 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d218d0a430 con 0x55d218f36200 2024-03-14T09:54:33.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.653+0000 7fea51e00700 1 --2- 172.21.15.78:0/962117720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f358e0 0x55d218f35cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:54:33.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.654+0000 7fea51400700 1 --2- 172.21.15.78:0/962117720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f36200 0x55d218f40b70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fea400c9b30 tx=0x7fea40099240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90b9cba3e8ec2967 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:33.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.654+0000 7fea3f400700 1 -- 172.21.15.78:0/962117720 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea40099d80 con 0x55d218f36200 2024-03-14T09:54:33.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.654+0000 7fea3f400700 1 -- 172.21.15.78:0/962117720 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea40099680 con 0x55d218f36200 2024-03-14T09:54:33.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.654+0000 7fea3f400700 1 -- 172.21.15.78:0/962117720 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4009d120 con 0x55d218f36200 2024-03-14T09:54:33.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.655+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/962117720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f36200 msgr2=0x55d218f40b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.655+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/962117720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f36200 0x55d218f40b70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fea400c9b30 tx=0x7fea40099240 comp rx=0 tx=0).stop 2024-03-14T09:54:33.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.655+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/962117720 shutdown_connections 2024-03-14T09:54:33.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.655+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/962117720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f410b0 0x55d218f434a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.655+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/962117720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f358e0 0x55d218f35cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.655+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/962117720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f36200 0x55d218f40b70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.655+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/962117720 >> 172.21.15.78:0/962117720 conn(0x55d218eac2a0 msgr2=0x55d218f31d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.655+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/962117720 shutdown_connections 2024-03-14T09:54:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.655+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/962117720 wait complete. 2024-03-14T09:54:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.656+0000 7fea6cbb6b80 1 -- start start 2024-03-14T09:54:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.656+0000 7fea6cbb6b80 1 Processor -- start 2024-03-14T09:54:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.657+0000 7fea6cbb6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f358e0 0x55d218f40b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.657+0000 7fea6cbb6b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f36200 0x55d218f3df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.657+0000 7fea6cbb6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f410b0 0x55d218f3e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.657+0000 7fea6cbb6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d218e24e00 con 0x55d218f358e0 2024-03-14T09:54:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.657+0000 7fea6cbb6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d218e24aa0 con 0x55d218f410b0 2024-03-14T09:54:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.657+0000 7fea6cbb6b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d218d0c720 con 0x55d218f36200 2024-03-14T09:54:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.657+0000 7fea51400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f36200 0x55d218f3df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.657+0000 7fea51400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f36200 0x55d218f3df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37398/0 (socket says 172.21.15.78:37398) 2024-03-14T09:54:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.657+0000 7fea51400700 1 -- 172.21.15.78:0/1166802599 learned_addr learned my addr 172.21.15.78:0/1166802599 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.657+0000 7fea6cbb6b80 10 client.0 _ll_register_callbacks cb 0x55d218c90420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.657+0000 7fea52800700 1 --2- 172.21.15.78:0/1166802599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f410b0 0x55d218f3e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.658+0000 7fea51400700 1 -- 172.21.15.78:0/1166802599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f410b0 msgr2=0x55d218f3e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.658+0000 7fea51400700 1 --2- 172.21.15.78:0/1166802599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f410b0 0x55d218f3e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.658+0000 7fea51e00700 1 --2- 172.21.15.78:0/1166802599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f358e0 0x55d218f40b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.658+0000 7fea51400700 1 -- 172.21.15.78:0/1166802599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f358e0 msgr2=0x55d218f40b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.658+0000 7fea51400700 1 --2- 172.21.15.78:0/1166802599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f358e0 0x55d218f40b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.658+0000 7fea51400700 1 -- 172.21.15.78:0/1166802599 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea54101c90 con 0x55d218f36200 2024-03-14T09:54:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.658+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/1166802599 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d218f297a0 con 0x55d218f36200 2024-03-14T09:54:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.658+0000 7fea51e00700 1 --2- 172.21.15.78:0/1166802599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f358e0 0x55d218f40b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:54:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.658+0000 7fea51400700 1 --2- 172.21.15.78:0/1166802599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f36200 0x55d218f3df50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fea4000e970 tx=0x7fea4001f660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4031227ab2d9ef5a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.658+0000 7fea33e00700 10 client.25717 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:54:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.658+0000 7fea33e00700 1 -- 172.21.15.78:0/1166802599 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4001f440 con 0x55d218f36200 2024-03-14T09:54:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.658+0000 7fea33e00700 1 -- 172.21.15.78:0/1166802599 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fea4001fe40 con 0x55d218f36200 2024-03-14T09:54:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.659+0000 7fea33e00700 1 -- 172.21.15.78:0/1166802599 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea400a1b00 con 0x55d218f36200 2024-03-14T09:54:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.659+0000 7fea6cbb6b80 10 client.25717 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.659+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/1166802599 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x55d218cb6be0 con 0x55d218f36200 2024-03-14T09:54:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.659+0000 7fea33e00700 1 -- 172.21.15.78:0/1166802599 <== mon.1 v2:172.21.15.152:3300/0 4 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fea400cc330 con 0x55d218f36200 2024-03-14T09:54:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.659+0000 7fea6cbb6b80 10 client.25717 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.659+0000 7fea6cbb6b80 10 client.25717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.659+0000 7fea6cbb6b80 20 client.25717 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.659+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/1166802599 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d218f507a0 0x55d218f52b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.659+0000 7fea6cbb6b80 4 client.25717 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:54:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.659+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/1166802599 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d218e81780 con 0x55d218f507a0 2024-03-14T09:54:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.660+0000 7fea51e00700 1 --2- 172.21.15.78:0/1166802599 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d218f507a0 0x55d218f52b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.662+0000 7fea51e00700 1 --2- 172.21.15.78:0/1166802599 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d218f507a0 0x55d218f52b90 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1af408deab75b9 server_cookie=48b92fe04908c7f1 in_seq=0 out_seq=0 2024-03-14T09:54:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.662+0000 7fea33e00700 10 client.25717 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:33.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.663+0000 7fea33e00700 1 -- 172.21.15.78:0/1166802599 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55d218e81780 con 0x55d218f507a0 2024-03-14T09:54:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.663+0000 7fea33e00700 10 client.25717 handle_command_reply: tid=0 2024-03-14T09:54:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.663+0000 7fea6cbb6b80 1 client.25717 shutdown 2024-03-14T09:54:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.663+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/1166802599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f36200 msgr2=0x55d218f3df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.663+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/1166802599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f36200 0x55d218f3df50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fea4000e970 tx=0x7fea4001f660 comp rx=0 tx=0).stop 2024-03-14T09:54:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.663+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/1166802599 shutdown_connections 2024-03-14T09:54:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.663+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/1166802599 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d218f507a0 0x55d218f52b90 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.663+0000 7fea33e00700 0 client.25717 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.663+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/1166802599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f410b0 0x55d218f3e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.663+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/1166802599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f358e0 0x55d218f40b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.663+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/1166802599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f36200 0x55d218f3df50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.663+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/1166802599 >> 172.21.15.78:0/1166802599 conn(0x55d218eac2a0 msgr2=0x55d218f31900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.664+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/1166802599 shutdown_connections 2024-03-14T09:54:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.664+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/1166802599 wait complete. 2024-03-14T09:54:33.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.664+0000 7fea6cbb6b80 20 client.25717 trim_cache size 0 max 16384 2024-03-14T09:54:33.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.681+0000 7fea6cbb6b80 1 Processor -- start 2024-03-14T09:54:33.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.681+0000 7fea6cbb6b80 1 -- start start 2024-03-14T09:54:33.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea6cbb6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f358e0 0x55d218f3b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea6cbb6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f36200 0x55d218f3b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea6cbb6b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f410b0 0x55d218f37980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea6cbb6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d218f57900 con 0x55d218f36200 2024-03-14T09:54:33.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea6cbb6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d218f368a0 con 0x55d218f358e0 2024-03-14T09:54:33.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea6cbb6b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d218f3c4e0 con 0x55d218f410b0 2024-03-14T09:54:33.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea51400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f36200 0x55d218f3b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea51e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f358e0 0x55d218f3b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea51400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f36200 0x55d218f3b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45794/0 (socket says 172.21.15.78:45794) 2024-03-14T09:54:33.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea51e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f358e0 0x55d218f3b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:39878/0 (socket says 172.21.15.78:39878) 2024-03-14T09:54:33.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea51400700 1 -- 172.21.15.78:0/1001070766 learned_addr learned my addr 172.21.15.78:0/1001070766 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:33.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea52800700 1 --2- 172.21.15.78:0/1001070766 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f410b0 0x55d218f37980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea51e00700 1 -- 172.21.15.78:0/1001070766 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f410b0 msgr2=0x55d218f37980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:33.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea51e00700 1 --2- 172.21.15.78:0/1001070766 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f410b0 0x55d218f37980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea51e00700 1 -- 172.21.15.78:0/1001070766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f36200 msgr2=0x55d218f3b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:33.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea51e00700 1 --2- 172.21.15.78:0/1001070766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f36200 0x55d218f3b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea51e00700 1 -- 172.21.15.78:0/1001070766 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d218f3e080 con 0x55d218f358e0 2024-03-14T09:54:33.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.682+0000 7fea52800700 1 --2- 172.21.15.78:0/1001070766 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f410b0 0x55d218f37980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:54:33.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.683+0000 7fea51e00700 1 --2- 172.21.15.78:0/1001070766 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f358e0 0x55d218f3b210 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fea4400b6f0 tx=0x7fea44020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=786550322554c9d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:33.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.683+0000 7fea3fe00700 1 -- 172.21.15.78:0/1001070766 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea44033a30 con 0x55d218f358e0 2024-03-14T09:54:33.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.683+0000 7fea3fe00700 1 -- 172.21.15.78:0/1001070766 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea4401e040 con 0x55d218f358e0 2024-03-14T09:54:33.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.683+0000 7fea3fe00700 1 -- 172.21.15.78:0/1001070766 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4400fd40 con 0x55d218f358e0 2024-03-14T09:54:33.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.684+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/1001070766 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f358e0 msgr2=0x55d218f3b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.684+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/1001070766 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f358e0 0x55d218f3b210 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fea4400b6f0 tx=0x7fea44020000 comp rx=0 tx=0).stop 2024-03-14T09:54:33.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.684+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/1001070766 shutdown_connections 2024-03-14T09:54:33.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.684+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/1001070766 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f358e0 0x55d218f3b210 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.684+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/1001070766 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f410b0 0x55d218f37980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.684+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/1001070766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f36200 0x55d218f3b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.684+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/1001070766 >> 172.21.15.78:0/1001070766 conn(0x55d218eac2a0 msgr2=0x55d218f31840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:33.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.684+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/1001070766 shutdown_connections 2024-03-14T09:54:33.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.684+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/1001070766 wait complete. 2024-03-14T09:54:33.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.684+0000 7fea6cbb6b80 1 -- start start 2024-03-14T09:54:33.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.685+0000 7fea6cbb6b80 1 Processor -- start 2024-03-14T09:54:33.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.686+0000 7fea6cbb6b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f358e0 0x55d218f9b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.686+0000 7fea6cbb6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f36200 0x55d218f9bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.686+0000 7fea6cbb6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f410b0 0x55d218f46da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.686+0000 7fea6cbb6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d218f47320 con 0x55d218f410b0 2024-03-14T09:54:33.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.686+0000 7fea6cbb6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d218f47490 con 0x55d218f36200 2024-03-14T09:54:33.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.686+0000 7fea6cbb6b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d218f47650 con 0x55d218f358e0 2024-03-14T09:54:33.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.686+0000 7fea6cbb6b80 10 client.0 _ll_register_callbacks cb 0x7fea4c007c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:33.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.686+0000 7fea51e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f358e0 0x55d218f9b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.686+0000 7fea51e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f358e0 0x55d218f9b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37408/0 (socket says 172.21.15.78:37408) 2024-03-14T09:54:33.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.686+0000 7fea51e00700 1 -- 172.21.15.78:0/2469234212 learned_addr learned my addr 172.21.15.78:0/2469234212 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:33.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.687+0000 7fea51400700 1 --2- 172.21.15.78:0/2469234212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f36200 0x55d218f9bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.687+0000 7fea51e00700 1 -- 172.21.15.78:0/2469234212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f36200 msgr2=0x55d218f9bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:33.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.687+0000 7fea51e00700 1 --2- 172.21.15.78:0/2469234212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f36200 0x55d218f9bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.687+0000 7fea51e00700 1 -- 172.21.15.78:0/2469234212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f410b0 msgr2=0x55d218f46da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:33.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.687+0000 7fea51e00700 1 --2- 172.21.15.78:0/2469234212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f410b0 0x55d218f46da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.687+0000 7fea51e00700 1 -- 172.21.15.78:0/2469234212 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea54110470 con 0x55d218f358e0 2024-03-14T09:54:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.687+0000 7fea51e00700 1 --2- 172.21.15.78:0/2469234212 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f358e0 0x55d218f9b6c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fea4400b420 tx=0x7fea440338e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a37bf21caebb469f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.687+0000 7fea33e00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:54:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.688+0000 7fea33e00700 1 -- 172.21.15.78:0/2469234212 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4400f460 con 0x55d218f358e0 2024-03-14T09:54:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.688+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/2469234212 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fea40099680 con 0x55d218f358e0 2024-03-14T09:54:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.688+0000 7fea33e00700 1 -- 172.21.15.78:0/2469234212 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fea4400f5c0 con 0x55d218f358e0 2024-03-14T09:54:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.688+0000 7fea33e00700 1 -- 172.21.15.78:0/2469234212 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea4400f780 con 0x55d218f358e0 2024-03-14T09:54:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.688+0000 7fea6cbb6b80 10 client.25723 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.688+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/2469234212 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d218e4ef20 con 0x55d218f358e0 2024-03-14T09:54:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.689+0000 7fea33e00700 1 -- 172.21.15.78:0/2469234212 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea40099680 con 0x55d218f358e0 2024-03-14T09:54:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.689+0000 7fea33e00700 1 -- 172.21.15.78:0/2469234212 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fea4404e830 con 0x55d218f358e0 2024-03-14T09:54:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.689+0000 7fea6cbb6b80 10 client.25723 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.689+0000 7fea6cbb6b80 10 client.25723 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.689+0000 7fea6cbb6b80 20 client.25723 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.689+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/2469234212 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d218fa4200 0x55d218fa65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.689+0000 7fea6cbb6b80 4 client.25723 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T09:54:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.689+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/2469234212 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d218cd7d70 con 0x55d218fa4200 2024-03-14T09:54:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.689+0000 7fea51400700 1 --2- 172.21.15.78:0/2469234212 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d218fa4200 0x55d218fa65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.690+0000 7fea33e00700 10 client.25723 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.690+0000 7fea51400700 1 --2- 172.21.15.78:0/2469234212 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d218fa4200 0x55d218fa65f0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a20be1d6ec1c23 server_cookie=82a6ad6e0eefbd5b in_seq=0 out_seq=0 2024-03-14T09:54:33.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.691+0000 7fea33e00700 1 -- 172.21.15.78:0/2469234212 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d218cd7d70 con 0x55d218fa4200 2024-03-14T09:54:33.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.691+0000 7fea33e00700 10 client.25723 handle_command_reply: tid=0 2024-03-14T09:54:33.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.691+0000 7fea6cbb6b80 1 client.25723 shutdown 2024-03-14T09:54:33.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.691+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/2469234212 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f358e0 msgr2=0x55d218f9b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.691+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/2469234212 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f358e0 0x55d218f9b6c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fea4400b420 tx=0x7fea440338e0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.692+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/2469234212 shutdown_connections 2024-03-14T09:54:33.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.692+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/2469234212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d218f410b0 0x55d218f46da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.692+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/2469234212 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d218f358e0 0x55d218f9b6c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.692+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/2469234212 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d218fa4200 0x55d218fa65f0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.692+0000 7fea6cbb6b80 1 --2- 172.21.15.78:0/2469234212 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d218f36200 0x55d218f9bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.692+0000 7fea33e00700 0 client.25723 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:33.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.692+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/2469234212 >> 172.21.15.78:0/2469234212 conn(0x55d218eac2a0 msgr2=0x55d218f37980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:33.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.692+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/2469234212 shutdown_connections 2024-03-14T09:54:33.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.692+0000 7fea6cbb6b80 1 -- 172.21.15.78:0/2469234212 wait complete. 2024-03-14T09:54:33.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.692+0000 7fea6cbb6b80 20 client.25723 trim_cache size 0 max 16384 2024-03-14T09:54:33.694 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:54:33.694 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T09:54:33.694 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "01ebf85c-3a23-48d9-8bd3-f37826ea363b", 2024-03-14T09:54:33.694 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T09:54:33.694 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:54:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.694+0000 7fea53200700 1 -- 172.21.15.78:0/408144474 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fea380a0d10 msgr2=0x7fea380a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.694+0000 7fea53200700 1 --2- 172.21.15.78:0/408144474 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fea380a0d10 0x7fea380a31d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fea4400b2e0 tx=0x7fea44005fd0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.694+0000 7fea53200700 1 -- 172.21.15.78:0/408144474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea54100250 msgr2=0x7fea54199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.694+0000 7fea53200700 1 --2- 172.21.15.78:0/408144474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea54100250 0x7fea54199b40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fea4000e9a0 tx=0x7fea4000ecb0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.694+0000 7fea53200700 1 -- 172.21.15.78:0/408144474 shutdown_connections 2024-03-14T09:54:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.694+0000 7fea53200700 1 --2- 172.21.15.78:0/408144474 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fea380a0d10 0x7fea380a31d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.694+0000 7fea53200700 1 --2- 172.21.15.78:0/408144474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea5410d450 0x7fea5419de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.694+0000 7fea53200700 1 --2- 172.21.15.78:0/408144474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fea540ff8a0 0x7fea54199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.694+0000 7fea53200700 1 --2- 172.21.15.78:0/408144474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea54100250 0x7fea54199b40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.694+0000 7fea53200700 1 -- 172.21.15.78:0/408144474 >> 172.21.15.78:0/408144474 conn(0x7fea540fb4e0 msgr2=0x7fea540fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.694+0000 7fea53200700 1 -- 172.21.15.78:0/408144474 shutdown_connections 2024-03-14T09:54:33.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.694+0000 7fea53200700 1 -- 172.21.15.78:0/408144474 wait complete. 2024-03-14T09:54:33.721 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:54:33.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.947+0000 7fb928200700 1 -- 172.21.15.78:0/4023946581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb920068360 msgr2=0x7fb9200687e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.947+0000 7fb928200700 1 --2- 172.21.15.78:0/4023946581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb920068360 0x7fb9200687e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb90c009990 tx=0x7fb90c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.947+0000 7fb928200700 1 -- 172.21.15.78:0/4023946581 shutdown_connections 2024-03-14T09:54:33.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.947+0000 7fb928200700 1 --2- 172.21.15.78:0/4023946581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb920105c00 0x7fb920067e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.947+0000 7fb928200700 1 --2- 172.21.15.78:0/4023946581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb920103700 0x7fb920103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.947+0000 7fb928200700 1 --2- 172.21.15.78:0/4023946581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb920068360 0x7fb9200687e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.947+0000 7fb928200700 1 -- 172.21.15.78:0/4023946581 >> 172.21.15.78:0/4023946581 conn(0x7fb920076d80 msgr2=0x7fb920077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:33.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.948+0000 7fb928200700 1 -- 172.21.15.78:0/4023946581 shutdown_connections 2024-03-14T09:54:33.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.948+0000 7fb928200700 1 -- 172.21.15.78:0/4023946581 wait complete. 2024-03-14T09:54:33.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.949+0000 7fb928200700 1 Processor -- start 2024-03-14T09:54:33.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.949+0000 7fb928200700 1 -- start start 2024-03-14T09:54:33.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.949+0000 7fb928200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb920068360 0x7fb9201a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.949+0000 7fb928200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb920103700 0x7fb9201a3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.949+0000 7fb928200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb920105c00 0x7fb92019d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.949+0000 7fb928200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92019e040 con 0x7fb920103700 2024-03-14T09:54:33.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.949+0000 7fb928200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92019e1b0 con 0x7fb920105c00 2024-03-14T09:54:33.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.949+0000 7fb928200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92019e370 con 0x7fb920068360 2024-03-14T09:54:33.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.950+0000 7fb925800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb920068360 0x7fb9201a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.950+0000 7fb925800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb920068360 0x7fb9201a38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:58976/0 (socket says 172.21.15.78:58976) 2024-03-14T09:54:33.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.950+0000 7fb925800700 1 -- 172.21.15.78:0/692444668 learned_addr learned my addr 172.21.15.78:0/692444668 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:33.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.950+0000 7fb924e00700 1 --2- 172.21.15.78:0/692444668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb920103700 0x7fb9201a3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.950+0000 7fb925800700 1 -- 172.21.15.78:0/692444668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb920105c00 msgr2=0x7fb92019d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.950+0000 7fb925800700 1 --2- 172.21.15.78:0/692444668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb920105c00 0x7fb92019d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.950+0000 7fb925800700 1 -- 172.21.15.78:0/692444668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb920103700 msgr2=0x7fb9201a3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.950+0000 7fb925800700 1 --2- 172.21.15.78:0/692444668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb920103700 0x7fb9201a3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.950+0000 7fb925800700 1 -- 172.21.15.78:0/692444668 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90c0095f0 con 0x7fb920068360 2024-03-14T09:54:33.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.950+0000 7fb925800700 1 --2- 172.21.15.78:0/692444668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb920068360 0x7fb9201a38b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb91400e9a0 tx=0x7fb91400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.951+0000 7fb91e000700 1 -- 172.21.15.78:0/692444668 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb914009790 con 0x7fb920068360 2024-03-14T09:54:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.951+0000 7fb91e000700 1 -- 172.21.15.78:0/692444668 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb91400cbd0 con 0x7fb920068360 2024-03-14T09:54:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.951+0000 7fb91e000700 1 -- 172.21.15.78:0/692444668 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb914018680 con 0x7fb920068360 2024-03-14T09:54:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.951+0000 7fb928200700 1 -- 172.21.15.78:0/692444668 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92019e650 con 0x7fb920068360 2024-03-14T09:54:33.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.951+0000 7fb928200700 1 -- 172.21.15.78:0/692444668 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9201aa8a0 con 0x7fb920068360 2024-03-14T09:54:33.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.956+0000 7fb91e000700 1 -- 172.21.15.78:0/692444668 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 25) v1 ==== 135564+0+0 (secure 0 0 0) 0x7fb914010430 con 0x7fb920068360 2024-03-14T09:54:33.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.957+0000 7fb91e000700 1 --2- 172.21.15.78:0/692444668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb9080a0d10 0x7fb9080a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.957+0000 7fb93bc93b80 1 Processor -- start 2024-03-14T09:54:33.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.957+0000 7fb91e000700 1 -- 172.21.15.78:0/692444668 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fb914014070 con 0x7fb920068360 2024-03-14T09:54:33.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.957+0000 7fb93bc93b80 1 -- start start 2024-03-14T09:54:33.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.957+0000 7fb924e00700 1 --2- 172.21.15.78:0/692444668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb9080a0d10 0x7fb9080a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.958+0000 7fb93bc93b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11680 0x559c63e11a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.958+0000 7fb93bc93b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11fa0 0x559c63e1b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.958+0000 7fb93bc93b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e1bec0 0x559c63e1e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.958+0000 7fb93bc93b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c63d6a990 con 0x559c63e1bec0 2024-03-14T09:54:33.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.958+0000 7fb93bc93b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c63d68350 con 0x559c63e11680 2024-03-14T09:54:33.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.958+0000 7fb93bc93b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c63d63c40 con 0x559c63e11fa0 2024-03-14T09:54:33.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.958+0000 7fb925800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11680 0x559c63e11a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.958+0000 7fb925800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11680 0x559c63e11a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57256/0 (socket says 172.21.15.78:57256) 2024-03-14T09:54:33.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.958+0000 7fb926200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e1bec0 0x559c63e1e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.958+0000 7fb925800700 1 -- 172.21.15.78:0/2989448348 learned_addr learned my addr 172.21.15.78:0/2989448348 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:33.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.958+0000 7fb924e00700 1 --2- 172.21.15.78:0/692444668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb9080a0d10 0x7fb9080a31d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0098e0 tx=0x7fb90c005960 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:33.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.958+0000 7fb924e00700 1 --2- 172.21.15.78:0/2989448348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11fa0 0x559c63e1b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.959+0000 7fb925800700 1 -- 172.21.15.78:0/2989448348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11fa0 msgr2=0x559c63e1b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:33.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.959+0000 7fb925800700 1 --2- 172.21.15.78:0/2989448348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11fa0 0x559c63e1b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.959+0000 7fb925800700 1 -- 172.21.15.78:0/2989448348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e1bec0 msgr2=0x559c63e1e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:33.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.959+0000 7fb925800700 1 --2- 172.21.15.78:0/2989448348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e1bec0 0x559c63e1e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.959+0000 7fb925800700 1 -- 172.21.15.78:0/2989448348 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c63be53b0 con 0x559c63e11680 2024-03-14T09:54:33.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.959+0000 7fb924e00700 1 --2- 172.21.15.78:0/2989448348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11fa0 0x559c63e1b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:54:33.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.959+0000 7fb925800700 1 --2- 172.21.15.78:0/2989448348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11680 0x559c63e11a60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb914099870 tx=0x7fb914099420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1acedad999c92e2b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:33.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.959+0000 7fb912a00700 1 -- 172.21.15.78:0/2989448348 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91400cd40 con 0x559c63e11680 2024-03-14T09:54:33.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.959+0000 7fb912a00700 1 -- 172.21.15.78:0/2989448348 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb9140998e0 con 0x559c63e11680 2024-03-14T09:54:33.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.959+0000 7fb912a00700 1 -- 172.21.15.78:0/2989448348 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb914099da0 con 0x559c63e11680 2024-03-14T09:54:33.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.960+0000 7fb93bc93b80 1 -- 172.21.15.78:0/2989448348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11680 msgr2=0x559c63e11a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.960+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/2989448348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11680 0x559c63e11a60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb914099870 tx=0x7fb914099420 comp rx=0 tx=0).stop 2024-03-14T09:54:33.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.960+0000 7fb93bc93b80 1 -- 172.21.15.78:0/2989448348 shutdown_connections 2024-03-14T09:54:33.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.960+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/2989448348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11680 0x559c63e11a60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.960+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/2989448348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e1bec0 0x559c63e1e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.960+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/2989448348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11fa0 0x559c63e1b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.960+0000 7fb93bc93b80 1 -- 172.21.15.78:0/2989448348 >> 172.21.15.78:0/2989448348 conn(0x559c63d87220 msgr2=0x559c63e0caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:33.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.960+0000 7fb93bc93b80 1 -- 172.21.15.78:0/2989448348 shutdown_connections 2024-03-14T09:54:33.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.960+0000 7fb93bc93b80 1 -- 172.21.15.78:0/2989448348 wait complete. 2024-03-14T09:54:33.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.961+0000 7fb93bc93b80 1 -- start start 2024-03-14T09:54:33.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.961+0000 7fb93bc93b80 1 Processor -- start 2024-03-14T09:54:33.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb93bc93b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11680 0x559c63e1b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb93bc93b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11fa0 0x559c63e18b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb93bc93b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e1bec0 0x559c63e191a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb93bc93b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c63cffd80 con 0x559c63e1bec0 2024-03-14T09:54:33.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb93bc93b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c63cffa20 con 0x559c63e11680 2024-03-14T09:54:33.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb93bc93b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c63be76a0 con 0x559c63e11fa0 2024-03-14T09:54:33.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb93bc93b80 10 client.0 _ll_register_callbacks cb 0x559c63b6b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:33.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb925800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11680 0x559c63e1b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb925800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11680 0x559c63e1b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57258/0 (socket says 172.21.15.78:57258) 2024-03-14T09:54:33.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb925800700 1 -- 172.21.15.78:0/332472117 learned_addr learned my addr 172.21.15.78:0/332472117 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:33.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb924e00700 1 --2- 172.21.15.78:0/332472117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11fa0 0x559c63e18b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb926200700 1 --2- 172.21.15.78:0/332472117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e1bec0 0x559c63e191a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb925800700 1 -- 172.21.15.78:0/332472117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11fa0 msgr2=0x559c63e18b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:33.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb925800700 1 --2- 172.21.15.78:0/332472117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11fa0 0x559c63e18b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb925800700 1 -- 172.21.15.78:0/332472117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e1bec0 msgr2=0x559c63e191a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:33.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb925800700 1 --2- 172.21.15.78:0/332472117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e1bec0 0x559c63e191a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.962+0000 7fb925800700 1 -- 172.21.15.78:0/332472117 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9201aa8a0 con 0x559c63e11680 2024-03-14T09:54:33.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.963+0000 7fb925800700 1 --2- 172.21.15.78:0/332472117 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11680 0x559c63e1b850 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb9140995e0 tx=0x7fb91400cf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d964680738482a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:33.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.963+0000 7fb907400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:54:33.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.963+0000 7fb907400700 1 -- 172.21.15.78:0/332472117 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9140b2270 con 0x559c63e11680 2024-03-14T09:54:33.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.963+0000 7fb93bc93b80 1 -- 172.21.15.78:0/332472117 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c63e04720 con 0x559c63e11680 2024-03-14T09:54:33.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.963+0000 7fb907400700 1 -- 172.21.15.78:0/332472117 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fb91400cd40 con 0x559c63e11680 2024-03-14T09:54:33.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.963+0000 7fb907400700 1 -- 172.21.15.78:0/332472117 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb9140a1d60 con 0x559c63e11680 2024-03-14T09:54:33.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.963+0000 7fb93bc93b80 10 client.25721 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:33.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.963+0000 7fb93bc93b80 1 -- 172.21.15.78:0/332472117 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559c63b91b60 con 0x559c63e11680 2024-03-14T09:54:33.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.964+0000 7fb907400700 1 -- 172.21.15.78:0/332472117 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559c63e04720 con 0x559c63e11680 2024-03-14T09:54:33.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.964+0000 7fb907400700 1 -- 172.21.15.78:0/332472117 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb9140a1960 con 0x559c63e11680 2024-03-14T09:54:33.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.964+0000 7fb93bc93b80 10 client.25721 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:33.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.964+0000 7fb93bc93b80 10 client.25721 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:33.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.964+0000 7fb93bc93b80 20 client.25721 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:33.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.964+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/332472117 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559c63e3c680 0x559c63e3eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.964+0000 7fb93bc93b80 4 client.25721 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:54:33.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.964+0000 7fb93bc93b80 1 -- 172.21.15.78:0/332472117 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559c63d5c700 con 0x559c63e3c680 2024-03-14T09:54:33.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.964+0000 7fb924e00700 1 --2- 172.21.15.78:0/332472117 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559c63e3c680 0x559c63e3eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.965+0000 7fb924e00700 1 --2- 172.21.15.78:0/332472117 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559c63e3c680 0x559c63e3eaa0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf25ae6d2929977a server_cookie=eaac43fd7c9a22a1 in_seq=0 out_seq=0 2024-03-14T09:54:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.965+0000 7fb907400700 10 client.25721 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.965+0000 7fb907400700 1 -- 172.21.15.78:0/332472117 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x559c63d5c700 con 0x559c63e3c680 2024-03-14T09:54:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.965+0000 7fb907400700 10 client.25721 handle_command_reply: tid=0 2024-03-14T09:54:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.966+0000 7fb93bc93b80 1 client.25721 shutdown 2024-03-14T09:54:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.966+0000 7fb93bc93b80 1 -- 172.21.15.78:0/332472117 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11680 msgr2=0x559c63e1b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.966+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/332472117 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11680 0x559c63e1b850 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb9140995e0 tx=0x7fb91400cf40 comp rx=0 tx=0).stop 2024-03-14T09:54:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.966+0000 7fb93bc93b80 1 -- 172.21.15.78:0/332472117 shutdown_connections 2024-03-14T09:54:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.966+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/332472117 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559c63e3c680 0x559c63e3eaa0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.966+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/332472117 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11680 0x559c63e1b850 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.966+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/332472117 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e1bec0 0x559c63e191a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.966+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/332472117 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11fa0 0x559c63e18b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.966+0000 7fb93bc93b80 1 -- 172.21.15.78:0/332472117 >> 172.21.15.78:0/332472117 conn(0x559c63d87220 msgr2=0x559c63e0c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.966+0000 7fb907400700 0 client.25721 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:33.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.966+0000 7fb93bc93b80 1 -- 172.21.15.78:0/332472117 shutdown_connections 2024-03-14T09:54:33.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.966+0000 7fb93bc93b80 1 -- 172.21.15.78:0/332472117 wait complete. 2024-03-14T09:54:33.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.966+0000 7fb93bc93b80 20 client.25721 trim_cache size 0 max 16384 2024-03-14T09:54:33.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.983+0000 7fb93bc93b80 1 Processor -- start 2024-03-14T09:54:33.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.983+0000 7fb93bc93b80 1 -- start start 2024-03-14T09:54:33.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.984+0000 7fb93bc93b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11680 0x559c63e190e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.984+0000 7fb93bc93b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11fa0 0x559c63e15df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.984+0000 7fb93bc93b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e1bec0 0x559c63e16330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.984+0000 7fb93bc93b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c63e13200 con 0x559c63e1bec0 2024-03-14T09:54:33.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.984+0000 7fb93bc93b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c63e1a1b0 con 0x559c63e11fa0 2024-03-14T09:54:33.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.984+0000 7fb93bc93b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c63e395b0 con 0x559c63e11680 2024-03-14T09:54:33.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.984+0000 7fb925800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11680 0x559c63e190e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.984+0000 7fb924e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11fa0 0x559c63e15df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.984+0000 7fb925800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11680 0x559c63e190e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59006/0 (socket says 172.21.15.78:59006) 2024-03-14T09:54:33.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.984+0000 7fb925800700 1 -- 172.21.15.78:0/614613869 learned_addr learned my addr 172.21.15.78:0/614613869 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:33.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.984+0000 7fb924e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11fa0 0x559c63e15df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57264/0 (socket says 172.21.15.78:57264) 2024-03-14T09:54:33.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.985+0000 7fb926200700 1 --2- 172.21.15.78:0/614613869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e1bec0 0x559c63e16330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.985+0000 7fb925800700 1 -- 172.21.15.78:0/614613869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11fa0 msgr2=0x559c63e15df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:33.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.985+0000 7fb925800700 1 --2- 172.21.15.78:0/614613869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11fa0 0x559c63e15df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.985+0000 7fb925800700 1 -- 172.21.15.78:0/614613869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e1bec0 msgr2=0x559c63e16330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:33.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.985+0000 7fb925800700 1 --2- 172.21.15.78:0/614613869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e1bec0 0x559c63e16330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.985+0000 7fb925800700 1 -- 172.21.15.78:0/614613869 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c63d29ea0 con 0x559c63e11680 2024-03-14T09:54:33.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.985+0000 7fb924e00700 1 --2- 172.21.15.78:0/614613869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11fa0 0x559c63e15df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:33.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.985+0000 7fb925800700 1 --2- 172.21.15.78:0/614613869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11680 0x559c63e190e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb914099870 tx=0x7fb91409a150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ef163d0f7db2ece server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:33.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.985+0000 7fb913400700 1 -- 172.21.15.78:0/614613869 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb914099420 con 0x559c63e11680 2024-03-14T09:54:33.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.986+0000 7fb913400700 1 -- 172.21.15.78:0/614613869 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb91400cd40 con 0x559c63e11680 2024-03-14T09:54:33.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.986+0000 7fb913400700 1 -- 172.21.15.78:0/614613869 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb914099b40 con 0x559c63e11680 2024-03-14T09:54:33.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.986+0000 7fb93bc93b80 1 -- 172.21.15.78:0/614613869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11680 msgr2=0x559c63e190e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.986+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/614613869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11680 0x559c63e190e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb914099870 tx=0x7fb91409a150 comp rx=0 tx=0).stop 2024-03-14T09:54:33.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.986+0000 7fb93bc93b80 1 -- 172.21.15.78:0/614613869 shutdown_connections 2024-03-14T09:54:33.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.986+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/614613869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e1bec0 0x559c63e16330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.986+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/614613869 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e11680 0x559c63e190e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.986+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/614613869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11fa0 0x559c63e15df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.986+0000 7fb93bc93b80 1 -- 172.21.15.78:0/614613869 >> 172.21.15.78:0/614613869 conn(0x559c63d87220 msgr2=0x559c63e17460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:33.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.986+0000 7fb93bc93b80 1 -- 172.21.15.78:0/614613869 shutdown_connections 2024-03-14T09:54:33.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.986+0000 7fb93bc93b80 1 -- 172.21.15.78:0/614613869 wait complete. 2024-03-14T09:54:33.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.987+0000 7fb93bc93b80 1 -- start start 2024-03-14T09:54:33.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.987+0000 7fb93bc93b80 1 Processor -- start 2024-03-14T09:54:33.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.988+0000 7fb93bc93b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e11680 0x559c63e317a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.988+0000 7fb93bc93b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11fa0 0x559c63e31ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.988+0000 7fb93bc93b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e1bec0 0x559c63e8d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.988+0000 7fb93bc93b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c63e32470 con 0x559c63e11680 2024-03-14T09:54:33.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.988+0000 7fb93bc93b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c63e8d6d0 con 0x559c63e11fa0 2024-03-14T09:54:33.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.988+0000 7fb93bc93b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c63e8d890 con 0x559c63e1bec0 2024-03-14T09:54:33.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.988+0000 7fb93bc93b80 10 client.0 _ll_register_callbacks cb 0x7fb918009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:33.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.988+0000 7fb924e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11fa0 0x559c63e31ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.988+0000 7fb924e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11fa0 0x559c63e31ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57270/0 (socket says 172.21.15.78:57270) 2024-03-14T09:54:33.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.988+0000 7fb924e00700 1 -- 172.21.15.78:0/524984557 learned_addr learned my addr 172.21.15.78:0/524984557 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:33.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.988+0000 7fb925800700 1 --2- 172.21.15.78:0/524984557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e11680 0x559c63e317a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.989+0000 7fb924e00700 1 -- 172.21.15.78:0/524984557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e1bec0 msgr2=0x559c63e8d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:33.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.989+0000 7fb924e00700 1 --2- 172.21.15.78:0/524984557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e1bec0 0x559c63e8d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.989+0000 7fb924e00700 1 -- 172.21.15.78:0/524984557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e11680 msgr2=0x559c63e317a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:33.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.989+0000 7fb924e00700 1 --2- 172.21.15.78:0/524984557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e11680 0x559c63e317a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.989+0000 7fb924e00700 1 -- 172.21.15.78:0/524984557 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb920114af0 con 0x559c63e11fa0 2024-03-14T09:54:33.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.989+0000 7fb925800700 1 --2- 172.21.15.78:0/524984557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e11680 0x559c63e317a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:54:33.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.989+0000 7fb924e00700 1 --2- 172.21.15.78:0/524984557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11fa0 0x559c63e31ce0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb90c000c00 tx=0x7fb90c009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd3041df61ab385b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:33.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.989+0000 7fb907400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:54:33.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.989+0000 7fb907400700 1 -- 172.21.15.78:0/524984557 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c00b7d0 con 0x559c63e11fa0 2024-03-14T09:54:33.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.989+0000 7fb93bc93b80 1 -- 172.21.15.78:0/524984557 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb9140998e0 con 0x559c63e11fa0 2024-03-14T09:54:33.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.989+0000 7fb907400700 1 -- 172.21.15.78:0/524984557 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fb90c00b930 con 0x559c63e11fa0 2024-03-14T09:54:33.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.990+0000 7fb907400700 1 -- 172.21.15.78:0/524984557 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb90c00baf0 con 0x559c63e11fa0 2024-03-14T09:54:33.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.990+0000 7fb93bc93b80 10 client.25727 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:33.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.990+0000 7fb93bc93b80 1 -- 172.21.15.78:0/524984557 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559c63b91b60 con 0x559c63e11fa0 2024-03-14T09:54:33.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.990+0000 7fb907400700 1 -- 172.21.15.78:0/524984557 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb9140998e0 con 0x559c63e11fa0 2024-03-14T09:54:33.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.991+0000 7fb907400700 1 -- 172.21.15.78:0/524984557 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb90c004500 con 0x559c63e11fa0 2024-03-14T09:54:33.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.991+0000 7fb93bc93b80 10 client.25727 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:33.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.991+0000 7fb93bc93b80 10 client.25727 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:33.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.991+0000 7fb93bc93b80 20 client.25727 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:33.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.991+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/524984557 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559c63e8f120 0x559c63e91540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:33.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.991+0000 7fb93bc93b80 4 client.25727 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:54:33.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.991+0000 7fb93bc93b80 1 -- 172.21.15.78:0/524984557 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559c63bb2cf0 con 0x559c63e8f120 2024-03-14T09:54:33.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.991+0000 7fb925800700 1 --2- 172.21.15.78:0/524984557 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559c63e8f120 0x559c63e91540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:33.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.991+0000 7fb925800700 1 --2- 172.21.15.78:0/524984557 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559c63e8f120 0x559c63e91540 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abd2505349e86e6d server_cookie=992e9cb821ae1f81 in_seq=0 out_seq=0 2024-03-14T09:54:33.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.991+0000 7fb907400700 10 client.25727 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.991+0000 7fb907400700 1 -- 172.21.15.78:0/524984557 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x559c63bb2cf0 con 0x559c63e8f120 2024-03-14T09:54:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.992+0000 7fb907400700 10 client.25727 handle_command_reply: tid=0 2024-03-14T09:54:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.992+0000 7fb93bc93b80 1 client.25727 shutdown 2024-03-14T09:54:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.992+0000 7fb93bc93b80 1 -- 172.21.15.78:0/524984557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11fa0 msgr2=0x559c63e31ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.992+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/524984557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11fa0 0x559c63e31ce0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb90c000c00 tx=0x7fb90c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.992+0000 7fb93bc93b80 1 -- 172.21.15.78:0/524984557 shutdown_connections 2024-03-14T09:54:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.992+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/524984557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559c63e1bec0 0x559c63e8d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.992+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/524984557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559c63e11680 0x559c63e317a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.992+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/524984557 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559c63e8f120 0x559c63e91540 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.992+0000 7fb93bc93b80 1 --2- 172.21.15.78:0/524984557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559c63e11fa0 0x559c63e31ce0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.992+0000 7fb93bc93b80 1 -- 172.21.15.78:0/524984557 >> 172.21.15.78:0/524984557 conn(0x559c63d87220 msgr2=0x559c63e1a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.992+0000 7fb907400700 0 client.25727 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.992+0000 7fb93bc93b80 1 -- 172.21.15.78:0/524984557 shutdown_connections 2024-03-14T09:54:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.992+0000 7fb93bc93b80 1 -- 172.21.15.78:0/524984557 wait complete. 2024-03-14T09:54:33.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.992+0000 7fb93bc93b80 20 client.25727 trim_cache size 0 max 16384 2024-03-14T09:54:33.996 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:54:33.996 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T09:54:33.996 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T09:54:33.996 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:54:33.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.993+0000 7fb928200700 1 -- 172.21.15.78:0/692444668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb9080a0d10 msgr2=0x7fb9080a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.993+0000 7fb928200700 1 --2- 172.21.15.78:0/692444668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb9080a0d10 0x7fb9080a31d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0098e0 tx=0x7fb90c005960 comp rx=0 tx=0).stop 2024-03-14T09:54:33.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.993+0000 7fb928200700 1 -- 172.21.15.78:0/692444668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb920068360 msgr2=0x7fb9201a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:33.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.993+0000 7fb928200700 1 --2- 172.21.15.78:0/692444668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb920068360 0x7fb9201a38b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb91400e9a0 tx=0x7fb91400ecb0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.993+0000 7fb928200700 1 -- 172.21.15.78:0/692444668 shutdown_connections 2024-03-14T09:54:33.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.993+0000 7fb928200700 1 --2- 172.21.15.78:0/692444668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb9080a0d10 0x7fb9080a31d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.993+0000 7fb928200700 1 --2- 172.21.15.78:0/692444668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb920105c00 0x7fb92019d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.993+0000 7fb928200700 1 --2- 172.21.15.78:0/692444668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb920068360 0x7fb9201a38b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.993+0000 7fb928200700 1 --2- 172.21.15.78:0/692444668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb920103700 0x7fb9201a3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:33.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.993+0000 7fb928200700 1 -- 172.21.15.78:0/692444668 >> 172.21.15.78:0/692444668 conn(0x7fb920076d80 msgr2=0x7fb9200fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:33.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.993+0000 7fb928200700 1 -- 172.21.15.78:0/692444668 shutdown_connections 2024-03-14T09:54:33.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:33.993+0000 7fb928200700 1 -- 172.21.15.78:0/692444668 wait complete. 2024-03-14T09:54:34.013 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T09:54:34.014 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:54:34.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:34 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T09:54:34.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:34 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:34.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:34 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T09:54:34.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:34 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:34.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:34 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T09:54:34.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.228+0000 7f0a87200700 1 -- 172.21.15.78:0/2376115908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a8810d3f0 msgr2=0x7f0a8810f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:34.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.228+0000 7f0a87200700 1 --2- 172.21.15.78:0/2376115908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a8810d3f0 0x7f0a8810f820 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000b0a0 tx=0x7f0a8000b3b0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.229+0000 7f0a87200700 1 -- 172.21.15.78:0/2376115908 shutdown_connections 2024-03-14T09:54:34.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.229+0000 7f0a87200700 1 --2- 172.21.15.78:0/2376115908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0a88101a90 0x7f0a88067dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.229+0000 7f0a87200700 1 --2- 172.21.15.78:0/2376115908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a8810d3f0 0x7f0a8810f820 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.229+0000 7f0a87200700 1 --2- 172.21.15.78:0/2376115908 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a88068300 0x7f0a88068780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.229+0000 7f0a87200700 1 -- 172.21.15.78:0/2376115908 >> 172.21.15.78:0/2376115908 conn(0x7f0a880755f0 msgr2=0x7f0a88075a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:34.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.229+0000 7f0a87200700 1 -- 172.21.15.78:0/2376115908 shutdown_connections 2024-03-14T09:54:34.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.229+0000 7f0a87200700 1 -- 172.21.15.78:0/2376115908 wait complete. 2024-03-14T09:54:34.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.230+0000 7f0a87200700 1 Processor -- start 2024-03-14T09:54:34.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.231+0000 7f0a87200700 1 -- start start 2024-03-14T09:54:34.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.231+0000 7f0a87200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a88068300 0x7f0a88102060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.231+0000 7f0a87200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0a88101a90 0x7f0a881025a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.231+0000 7f0a87200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a8810d3f0 0x7f0a88102ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.232+0000 7f0a87200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a881a9de0 con 0x7f0a88068300 2024-03-14T09:54:34.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.232+0000 7f0a85e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a88068300 0x7f0a88102060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.232+0000 7f0a85400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0a88101a90 0x7f0a881025a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.232+0000 7f0a85400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0a88101a90 0x7f0a881025a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57288/0 (socket says 172.21.15.78:57288) 2024-03-14T09:54:34.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.232+0000 7f0a85400700 1 -- 172.21.15.78:0/2035485244 learned_addr learned my addr 172.21.15.78:0/2035485244 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:34.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.232+0000 7f0a86800700 1 --2- 172.21.15.78:0/2035485244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a8810d3f0 0x7f0a88102ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.232+0000 7f0a87200700 1 -- 172.21.15.78:0/2035485244 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a881a9f20 con 0x7f0a88101a90 2024-03-14T09:54:34.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.232+0000 7f0a87200700 1 -- 172.21.15.78:0/2035485244 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a881aa090 con 0x7f0a8810d3f0 2024-03-14T09:54:34.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.233+0000 7f0a85400700 1 -- 172.21.15.78:0/2035485244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a8810d3f0 msgr2=0x7f0a88102ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:34.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.233+0000 7f0a85400700 1 --2- 172.21.15.78:0/2035485244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a8810d3f0 0x7f0a88102ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.233+0000 7f0a85400700 1 -- 172.21.15.78:0/2035485244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a88068300 msgr2=0x7f0a88102060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:34.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.233+0000 7f0a85400700 1 --2- 172.21.15.78:0/2035485244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a88068300 0x7f0a88102060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.233+0000 7f0a85400700 1 -- 172.21.15.78:0/2035485244 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a80009d00 con 0x7f0a88101a90 2024-03-14T09:54:34.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.233+0000 7f0a85e00700 1 --2- 172.21.15.78:0/2035485244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a88068300 0x7f0a88102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:34.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.233+0000 7f0a85400700 1 --2- 172.21.15.78:0/2035485244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0a88101a90 0x7f0a881025a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400e9a0 tx=0x7f0a7400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:34.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.234+0000 7f0a7ea00700 1 -- 172.21.15.78:0/2035485244 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a74009790 con 0x7f0a88101a90 2024-03-14T09:54:34.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.234+0000 7f0a7ea00700 1 -- 172.21.15.78:0/2035485244 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a74010430 con 0x7f0a88101a90 2024-03-14T09:54:34.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.234+0000 7f0a7ea00700 1 -- 172.21.15.78:0/2035485244 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a74018680 con 0x7f0a88101a90 2024-03-14T09:54:34.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.234+0000 7f0a87200700 1 -- 172.21.15.78:0/2035485244 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a881aa370 con 0x7f0a88101a90 2024-03-14T09:54:34.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.234+0000 7f0a87200700 1 -- 172.21.15.78:0/2035485244 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a881aa830 con 0x7f0a88101a90 2024-03-14T09:54:34.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.235+0000 7f0a87200700 1 -- 172.21.15.78:0/2035485244 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a88100100 con 0x7f0a88101a90 2024-03-14T09:54:34.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.239+0000 7f0a7ea00700 1 -- 172.21.15.78:0/2035485244 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 25) v1 ==== 135564+0+0 (secure 0 0 0) 0x7f0a740105a0 con 0x7f0a88101a90 2024-03-14T09:54:34.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.240+0000 7f0a7ea00700 1 --2- 172.21.15.78:0/2035485244 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0a6c0a0d10 0x7f0a6c0a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.240+0000 7f0a7ea00700 1 -- 172.21.15.78:0/2035485244 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f0a74014070 con 0x7f0a88101a90 2024-03-14T09:54:34.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.240+0000 7f0a85e00700 1 --2- 172.21.15.78:0/2035485244 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0a6c0a0d10 0x7f0a6c0a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.241+0000 7f0a85e00700 1 --2- 172.21.15.78:0/2035485244 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0a6c0a0d10 0x7f0a6c0a31d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0a78005950 tx=0x7f0a7800b410 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:34.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.241+0000 7f0a7ea00700 1 -- 172.21.15.78:0/2035485244 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0a74096070 con 0x7f0a88101a90 2024-03-14T09:54:34.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:34 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T09:54:34.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:34 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:34.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:34 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T09:54:34.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:34 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:34.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:34 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T09:54:34.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.506+0000 7f0a87200700 1 -- 172.21.15.78:0/2035485244 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a88103570 con 0x7f0a88101a90 2024-03-14T09:54:34.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.507+0000 7f0a7ea00700 1 -- 172.21.15.78:0/2035485244 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f0a74099690 con 0x7f0a88101a90 2024-03-14T09:54:34.508 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:54:34.508 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:54:34.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.509+0000 7f0a87200700 1 -- 172.21.15.78:0/2035485244 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0a6c0a0d10 msgr2=0x7f0a6c0a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:34.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.509+0000 7f0a87200700 1 --2- 172.21.15.78:0/2035485244 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0a6c0a0d10 0x7f0a6c0a31d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0a78005950 tx=0x7f0a7800b410 comp rx=0 tx=0).stop 2024-03-14T09:54:34.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.509+0000 7f0a87200700 1 -- 172.21.15.78:0/2035485244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0a88101a90 msgr2=0x7f0a881025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:34.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.509+0000 7f0a87200700 1 --2- 172.21.15.78:0/2035485244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0a88101a90 0x7f0a881025a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400e9a0 tx=0x7f0a7400ecb0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.510+0000 7f0a87200700 1 -- 172.21.15.78:0/2035485244 shutdown_connections 2024-03-14T09:54:34.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.510+0000 7f0a87200700 1 --2- 172.21.15.78:0/2035485244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0a8810d3f0 0x7f0a88102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.510+0000 7f0a87200700 1 --2- 172.21.15.78:0/2035485244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a88068300 0x7f0a88102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.510+0000 7f0a87200700 1 --2- 172.21.15.78:0/2035485244 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0a6c0a0d10 0x7f0a6c0a31d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.510+0000 7f0a87200700 1 --2- 172.21.15.78:0/2035485244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0a88101a90 0x7f0a881025a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.510+0000 7f0a87200700 1 -- 172.21.15.78:0/2035485244 >> 172.21.15.78:0/2035485244 conn(0x7f0a880755f0 msgr2=0x7f0a880fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:34.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.510+0000 7f0a87200700 1 -- 172.21.15.78:0/2035485244 shutdown_connections 2024-03-14T09:54:34.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.510+0000 7f0a87200700 1 -- 172.21.15.78:0/2035485244 wait complete. 2024-03-14T09:54:34.512 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T09:54:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:34 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T09:54:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:34 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:34 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T09:54:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:34 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:34 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T09:54:34.538 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T09:54:34.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.833+0000 7f31af200700 1 -- 172.21.15.78:0/2406886564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31b0106b90 msgr2=0x7f31b0106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:34.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.833+0000 7f31af200700 1 --2- 172.21.15.78:0/2406886564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31b0106b90 0x7f31b0106f70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009990 tx=0x7f31a0009ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.833+0000 7f31af200700 1 -- 172.21.15.78:0/2406886564 shutdown_connections 2024-03-14T09:54:34.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.833+0000 7f31af200700 1 --2- 172.21.15.78:0/2406886564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31b01014b0 0x7f31b010d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.833+0000 7f31af200700 1 --2- 172.21.15.78:0/2406886564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31b0106b90 0x7f31b0106f70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.833+0000 7f31af200700 1 --2- 172.21.15.78:0/2406886564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b0100af0 0x7f31b0100f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.833+0000 7f31af200700 1 -- 172.21.15.78:0/2406886564 >> 172.21.15.78:0/2406886564 conn(0x7f31b00fc5e0 msgr2=0x7f31b00fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.833+0000 7f31af200700 1 -- 172.21.15.78:0/2406886564 shutdown_connections 2024-03-14T09:54:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.833+0000 7f31af200700 1 -- 172.21.15.78:0/2406886564 wait complete. 2024-03-14T09:54:34.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.835+0000 7f31af200700 1 Processor -- start 2024-03-14T09:54:34.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.835+0000 7f31af200700 1 -- start start 2024-03-14T09:54:34.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.835+0000 7f31af200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31b0100af0 0x7f31b01995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.835+0000 7f31af200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b01014b0 0x7f31b0199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.835+0000 7f31af200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31b0106b90 0x7f31b019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.835+0000 7f31af200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b019e4c0 con 0x7f31b01014b0 2024-03-14T09:54:34.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.835+0000 7f31af200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b019e630 con 0x7f31b0100af0 2024-03-14T09:54:34.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.835+0000 7f31af200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b019e7f0 con 0x7f31b0106b90 2024-03-14T09:54:34.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.836+0000 7f31ade00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31b0100af0 0x7f31b01995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.836+0000 7f31ade00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31b0100af0 0x7f31b01995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57312/0 (socket says 172.21.15.78:57312) 2024-03-14T09:54:34.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.836+0000 7f31ade00700 1 -- 172.21.15.78:0/262128822 learned_addr learned my addr 172.21.15.78:0/262128822 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:34.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.836+0000 7f31ad400700 1 --2- 172.21.15.78:0/262128822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b01014b0 0x7f31b0199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.836+0000 7f31ae800700 1 --2- 172.21.15.78:0/262128822 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31b0106b90 0x7f31b019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.836+0000 7f31ade00700 1 -- 172.21.15.78:0/262128822 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31b0106b90 msgr2=0x7f31b019de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:34.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.836+0000 7f31ade00700 1 --2- 172.21.15.78:0/262128822 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31b0106b90 0x7f31b019de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.836+0000 7f31ade00700 1 -- 172.21.15.78:0/262128822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b01014b0 msgr2=0x7f31b0199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:34.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.836+0000 7f31ade00700 1 --2- 172.21.15.78:0/262128822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b01014b0 0x7f31b0199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.836+0000 7f31ade00700 1 -- 172.21.15.78:0/262128822 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a00095f0 con 0x7f31b0100af0 2024-03-14T09:54:34.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.837+0000 7f31ade00700 1 --2- 172.21.15.78:0/262128822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31b0100af0 0x7f31b01995a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f31a0005910 tx=0x7f31a0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:34.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.837+0000 7f31a6a00700 1 -- 172.21.15.78:0/262128822 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a001d070 con 0x7f31b0100af0 2024-03-14T09:54:34.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.837+0000 7f31a6a00700 1 -- 172.21.15.78:0/262128822 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f31a0022470 con 0x7f31b0100af0 2024-03-14T09:54:34.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.837+0000 7f31a6a00700 1 -- 172.21.15.78:0/262128822 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a0017650 con 0x7f31b0100af0 2024-03-14T09:54:34.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.837+0000 7f31af200700 1 -- 172.21.15.78:0/262128822 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b019ea70 con 0x7f31b0100af0 2024-03-14T09:54:34.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.837+0000 7f31af200700 1 -- 172.21.15.78:0/262128822 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b01a6310 con 0x7f31b0100af0 2024-03-14T09:54:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.842+0000 7f31a6a00700 1 -- 172.21.15.78:0/262128822 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 25) v1 ==== 135564+0+0 (secure 0 0 0) 0x7f31a00225e0 con 0x7f31b0100af0 2024-03-14T09:54:34.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.843+0000 7f31c8ba8b80 1 Processor -- start 2024-03-14T09:54:34.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.843+0000 7f31c8ba8b80 1 -- start start 2024-03-14T09:54:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.843+0000 7f31c8ba8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54c860 0x55ca2a54cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.843+0000 7f31c8ba8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54d180 0x55ca2a557af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.843+0000 7f31c8ba8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 0x55ca2a55a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.843+0000 7f31c8ba8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2a4a6990 con 0x55ca2a54d180 2024-03-14T09:54:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.844+0000 7f31c8ba8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2a4a4350 con 0x55ca2a54c860 2024-03-14T09:54:34.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.844+0000 7f31c8ba8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2a49fc40 con 0x55ca2a558030 2024-03-14T09:54:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.844+0000 7f31ae800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54c860 0x55ca2a54cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.844+0000 7f31ae800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54c860 0x55ca2a54cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57318/0 (socket says 172.21.15.78:57318) 2024-03-14T09:54:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.844+0000 7f31ae800700 1 -- 172.21.15.78:0/423878389 learned_addr learned my addr 172.21.15.78:0/423878389 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.844+0000 7f31ad400700 1 --2- 172.21.15.78:0/423878389 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 0x55ca2a55a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.844+0000 7f31a6a00700 1 --2- 172.21.15.78:0/262128822 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f31940a0c90 0x7f31940a3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.844+0000 7f31ade00700 1 --2- 172.21.15.78:0/423878389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54d180 0x55ca2a557af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.844+0000 7f31ad400700 1 -- 172.21.15.78:0/423878389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54c860 msgr2=0x55ca2a54cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.844+0000 7f31ad400700 1 --2- 172.21.15.78:0/423878389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54c860 0x55ca2a54cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.844+0000 7f31ad400700 1 -- 172.21.15.78:0/423878389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54d180 msgr2=0x55ca2a557af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.844+0000 7f31ad400700 1 --2- 172.21.15.78:0/423878389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54d180 0x55ca2a557af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.845+0000 7f31ad400700 1 -- 172.21.15.78:0/423878389 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca2a3213b0 con 0x55ca2a558030 2024-03-14T09:54:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.845+0000 7f31a6a00700 1 -- 172.21.15.78:0/262128822 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f31a00cdbe0 con 0x7f31b0100af0 2024-03-14T09:54:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.845+0000 7f31ad400700 1 --2- 172.21.15.78:0/423878389 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 0x55ca2a55a420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f319c005940 tx=0x7f319c009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=838a8796696f45d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:34.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.845+0000 7f31ae800700 1 --2- 172.21.15.78:0/262128822 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f31940a0c90 0x7f31940a3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.845+0000 7f319b400700 1 -- 172.21.15.78:0/423878389 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c011ce0 con 0x55ca2a558030 2024-03-14T09:54:34.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.845+0000 7f319b400700 1 -- 172.21.15.78:0/423878389 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f319c011e40 con 0x55ca2a558030 2024-03-14T09:54:34.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.845+0000 7f319b400700 1 -- 172.21.15.78:0/423878389 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319c004a90 con 0x55ca2a558030 2024-03-14T09:54:34.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.846+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/423878389 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 msgr2=0x55ca2a55a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.846+0000 7f31ae800700 1 --2- 172.21.15.78:0/262128822 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f31940a0c90 0x7f31940a3150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f31a80075c0 tx=0x7f31a8009040 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.846+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/423878389 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 0x55ca2a55a420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f319c005940 tx=0x7f319c009f60 comp rx=0 tx=0).stop 2024-03-14T09:54:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.846+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/423878389 shutdown_connections 2024-03-14T09:54:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.846+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/423878389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54c860 0x55ca2a54cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.846+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/423878389 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 0x55ca2a55a420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.846+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/423878389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54d180 0x55ca2a557af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.846+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/423878389 >> 172.21.15.78:0/423878389 conn(0x55ca2a4c3220 msgr2=0x55ca2a548c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.847+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/423878389 shutdown_connections 2024-03-14T09:54:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.847+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/423878389 wait complete. 2024-03-14T09:54:34.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.847+0000 7f31c8ba8b80 1 -- start start 2024-03-14T09:54:34.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.848+0000 7f31c8ba8b80 1 Processor -- start 2024-03-14T09:54:34.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.848+0000 7f31c8ba8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54c860 0x55ca2a557a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.848+0000 7f31c8ba8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54d180 0x55ca2a554d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.848+0000 7f31c8ba8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 0x55ca2a555440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.848+0000 7f31c8ba8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2a43bd80 con 0x55ca2a54d180 2024-03-14T09:54:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.848+0000 7f31c8ba8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2a43ba20 con 0x55ca2a54c860 2024-03-14T09:54:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.848+0000 7f31c8ba8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2a3236a0 con 0x55ca2a558030 2024-03-14T09:54:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.848+0000 7f31c8ba8b80 10 client.0 _ll_register_callbacks cb 0x55ca2a2a73a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.849+0000 7f31ae800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 0x55ca2a555440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.849+0000 7f31ade00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54c860 0x55ca2a557a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.849+0000 7f31ae800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 0x55ca2a555440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59050/0 (socket says 172.21.15.78:59050) 2024-03-14T09:54:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.849+0000 7f31ae800700 1 -- 172.21.15.78:0/4061769318 learned_addr learned my addr 172.21.15.78:0/4061769318 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:34.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.849+0000 7f31ad400700 1 --2- 172.21.15.78:0/4061769318 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54d180 0x55ca2a554d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.849+0000 7f31ae800700 1 -- 172.21.15.78:0/4061769318 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54c860 msgr2=0x55ca2a557a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.850+0000 7f31ae800700 1 --2- 172.21.15.78:0/4061769318 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54c860 0x55ca2a557a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.850+0000 7f31ae800700 1 -- 172.21.15.78:0/4061769318 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54d180 msgr2=0x55ca2a554d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.850+0000 7f31ae800700 1 --2- 172.21.15.78:0/4061769318 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54d180 0x55ca2a554d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.850+0000 7f31ae800700 1 -- 172.21.15.78:0/4061769318 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319c0098e0 con 0x55ca2a558030 2024-03-14T09:54:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.850+0000 7f31ad400700 1 --2- 172.21.15.78:0/4061769318 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54d180 0x55ca2a554d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.850+0000 7f31ade00700 1 --2- 172.21.15.78:0/4061769318 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54c860 0x55ca2a557a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.850+0000 7f31ae800700 1 --2- 172.21.15.78:0/4061769318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 0x55ca2a555440 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f31a8017f90 tx=0x7f31a8004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a36cad94da3fdf3f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.850+0000 7f318fe00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:54:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.850+0000 7f318fe00700 1 -- 172.21.15.78:0/4061769318 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a800e040 con 0x55ca2a558030 2024-03-14T09:54:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.851+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/4061769318 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ca2a540720 con 0x55ca2a558030 2024-03-14T09:54:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.851+0000 7f318fe00700 1 -- 172.21.15.78:0/4061769318 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f31a801c070 con 0x55ca2a558030 2024-03-14T09:54:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.851+0000 7f318fe00700 1 -- 172.21.15.78:0/4061769318 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31a8003a00 con 0x55ca2a558030 2024-03-14T09:54:34.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.851+0000 7f31c8ba8b80 10 client.25753 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:34.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.851+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/4061769318 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ca2a2cdb60 con 0x55ca2a558030 2024-03-14T09:54:34.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.851+0000 7f318fe00700 1 -- 172.21.15.78:0/4061769318 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f31a800f040 con 0x55ca2a558030 2024-03-14T09:54:34.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.852+0000 7f318fe00700 1 -- 172.21.15.78:0/4061769318 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f31a8004050 con 0x55ca2a558030 2024-03-14T09:54:34.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.852+0000 7f31c8ba8b80 10 client.25753 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.852+0000 7f31c8ba8b80 10 client.25753 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:54:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.852+0000 7f31c8ba8b80 20 client.25753 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.852+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/4061769318 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55ca2a567d50 0x55ca2a56a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.852+0000 7f31c8ba8b80 4 client.25753 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:54:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.852+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/4061769318 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ca2a540720 con 0x55ca2a567d50 2024-03-14T09:54:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.853+0000 7f31ade00700 1 --2- 172.21.15.78:0/4061769318 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55ca2a567d50 0x55ca2a56a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.853+0000 7f31ade00700 1 --2- 172.21.15.78:0/4061769318 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55ca2a567d50 0x55ca2a56a190 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1a882d47da894bae server_cookie=829ee3b810774cb7 in_seq=0 out_seq=0 2024-03-14T09:54:34.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.853+0000 7f318fe00700 10 client.25753 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:54:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.854+0000 7f318fe00700 1 -- 172.21.15.78:0/4061769318 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55ca2a540720 con 0x55ca2a567d50 2024-03-14T09:54:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.854+0000 7f318fe00700 10 client.25753 handle_command_reply: tid=0 2024-03-14T09:54:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.854+0000 7f31c8ba8b80 1 client.25753 shutdown 2024-03-14T09:54:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.854+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/4061769318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 msgr2=0x55ca2a555440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.854+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/4061769318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 0x55ca2a555440 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f31a8017f90 tx=0x7f31a8004ad0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.854+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/4061769318 shutdown_connections 2024-03-14T09:54:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.855+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/4061769318 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55ca2a567d50 0x55ca2a56a190 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.855+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/4061769318 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54c860 0x55ca2a557a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.855+0000 7f318fe00700 0 client.25753 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:54:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.855+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/4061769318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 0x55ca2a555440 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.855+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/4061769318 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54d180 0x55ca2a554d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.855+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/4061769318 >> 172.21.15.78:0/4061769318 conn(0x55ca2a4c3220 msgr2=0x55ca2a548880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:34.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.856+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/4061769318 shutdown_connections 2024-03-14T09:54:34.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.856+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/4061769318 wait complete. 2024-03-14T09:54:34.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.857+0000 7f31c8ba8b80 20 client.25753 trim_cache size 0 max 16384 2024-03-14T09:54:34.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.881+0000 7f31c8ba8b80 1 Processor -- start 2024-03-14T09:54:34.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.882+0000 7f31c8ba8b80 1 -- start start 2024-03-14T09:54:34.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.882+0000 7f31c8ba8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54c860 0x55ca2a56a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.882+0000 7f31c8ba8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a54d180 0x55ca2a55ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.882+0000 7f31c8ba8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a558030 0x55ca2a5575b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.882+0000 7f31c8ba8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2a54d820 con 0x55ca2a54c860 2024-03-14T09:54:34.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.882+0000 7f31c8ba8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2a553250 con 0x55ca2a558030 2024-03-14T09:54:34.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.882+0000 7f31c8ba8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2a555a40 con 0x55ca2a54d180 2024-03-14T09:54:34.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.882+0000 7f31ade00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54c860 0x55ca2a56a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.882+0000 7f31ade00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54c860 0x55ca2a56a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37350/0 (socket says 172.21.15.78:37350) 2024-03-14T09:54:34.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.883+0000 7f31ade00700 1 -- 172.21.15.78:0/2548781955 learned_addr learned my addr 172.21.15.78:0/2548781955 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:34.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.883+0000 7f31ae800700 1 --2- 172.21.15.78:0/2548781955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a558030 0x55ca2a5575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.883+0000 7f31ade00700 1 -- 172.21.15.78:0/2548781955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a54d180 msgr2=0x55ca2a55ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:34.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.883+0000 7f31ad400700 1 --2- 172.21.15.78:0/2548781955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a54d180 0x55ca2a55ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.883+0000 7f31ade00700 1 --2- 172.21.15.78:0/2548781955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a54d180 0x55ca2a55ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.883+0000 7f31ade00700 1 -- 172.21.15.78:0/2548781955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a558030 msgr2=0x55ca2a5575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:34.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.883+0000 7f31ade00700 1 --2- 172.21.15.78:0/2548781955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a558030 0x55ca2a5575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.883+0000 7f31ade00700 1 -- 172.21.15.78:0/2548781955 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca2a465ea0 con 0x55ca2a54c860 2024-03-14T09:54:34.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.883+0000 7f31ade00700 1 --2- 172.21.15.78:0/2548781955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54c860 0x55ca2a56a140 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f31a000f740 tx=0x7f31a006d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b9fed2292cde88 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:34.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.884+0000 7f319be00700 1 -- 172.21.15.78:0/2548781955 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a009a990 con 0x55ca2a54c860 2024-03-14T09:54:34.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.884+0000 7f319be00700 1 -- 172.21.15.78:0/2548781955 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f31a009a700 con 0x55ca2a54c860 2024-03-14T09:54:34.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.884+0000 7f319be00700 1 -- 172.21.15.78:0/2548781955 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a009e9f0 con 0x55ca2a54c860 2024-03-14T09:54:34.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.885+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/2548781955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54c860 msgr2=0x55ca2a56a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:34.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.885+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/2548781955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54c860 0x55ca2a56a140 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f31a000f740 tx=0x7f31a006d000 comp rx=0 tx=0).stop 2024-03-14T09:54:34.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.885+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/2548781955 shutdown_connections 2024-03-14T09:54:34.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.885+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/2548781955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a558030 0x55ca2a5575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.885+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/2548781955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54c860 0x55ca2a56a140 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.885+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/2548781955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a54d180 0x55ca2a55ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.885+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/2548781955 >> 172.21.15.78:0/2548781955 conn(0x55ca2a4c3220 msgr2=0x55ca2a544680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:34.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.885+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/2548781955 shutdown_connections 2024-03-14T09:54:34.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.885+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/2548781955 wait complete. 2024-03-14T09:54:34.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.886+0000 7f31c8ba8b80 1 -- start start 2024-03-14T09:54:34.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.886+0000 7f31c8ba8b80 1 Processor -- start 2024-03-14T09:54:34.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.886+0000 7f31c8ba8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54c860 0x55ca2a55cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.886+0000 7f31c8ba8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54d180 0x55ca2a55d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.887+0000 7f31c8ba8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 0x55ca2a5b8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.887+0000 7f31c8ba8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2a5b8dd0 con 0x55ca2a54c860 2024-03-14T09:54:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.887+0000 7f31c8ba8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2a5b8f10 con 0x55ca2a54d180 2024-03-14T09:54:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.887+0000 7f31c8ba8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca2a5b90d0 con 0x55ca2a558030 2024-03-14T09:54:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.887+0000 7f31c8ba8b80 10 client.0 _ll_register_callbacks cb 0x7f31a8017930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.887+0000 7f31ade00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54c860 0x55ca2a55cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.887+0000 7f31ade00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54c860 0x55ca2a55cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37358/0 (socket says 172.21.15.78:37358) 2024-03-14T09:54:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.887+0000 7f31ade00700 1 -- 172.21.15.78:0/2245936582 learned_addr learned my addr 172.21.15.78:0/2245936582 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.887+0000 7f31ad400700 1 --2- 172.21.15.78:0/2245936582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54d180 0x55ca2a55d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.887+0000 7f31ae800700 1 --2- 172.21.15.78:0/2245936582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 0x55ca2a5b8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.887+0000 7f31ae800700 1 -- 172.21.15.78:0/2245936582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54d180 msgr2=0x55ca2a55d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:34.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.887+0000 7f31ae800700 1 --2- 172.21.15.78:0/2245936582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54d180 0x55ca2a55d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.887+0000 7f31ae800700 1 -- 172.21.15.78:0/2245936582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54c860 msgr2=0x55ca2a55cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:34.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.888+0000 7f31ae800700 1 --2- 172.21.15.78:0/2245936582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54c860 0x55ca2a55cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.888+0000 7f31ae800700 1 -- 172.21.15.78:0/2245936582 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b01a6310 con 0x55ca2a558030 2024-03-14T09:54:34.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.888+0000 7f31ade00700 1 --2- 172.21.15.78:0/2245936582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54c860 0x55ca2a55cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:34.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.888+0000 7f31ae800700 1 --2- 172.21.15.78:0/2245936582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 0x55ca2a5b8890 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f31a8004430 tx=0x7f31a8016f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab8fd3dbb0ff4d1e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:34.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.888+0000 7f318fe00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:54:34.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.888+0000 7f318fe00700 1 -- 172.21.15.78:0/2245936582 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a800e070 con 0x55ca2a558030 2024-03-14T09:54:34.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.888+0000 7f318fe00700 1 -- 172.21.15.78:0/2245936582 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f31a801c050 con 0x55ca2a558030 2024-03-14T09:54:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.889+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/2245936582 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f319c011e40 con 0x55ca2a558030 2024-03-14T09:54:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.889+0000 7f318fe00700 1 -- 172.21.15.78:0/2245936582 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31a80041e0 con 0x55ca2a558030 2024-03-14T09:54:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.889+0000 7f31c8ba8b80 10 client.25756 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.889+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/2245936582 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ca2a325cd0 con 0x55ca2a558030 2024-03-14T09:54:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.889+0000 7f318fe00700 1 -- 172.21.15.78:0/2245936582 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f319c011e40 con 0x55ca2a558030 2024-03-14T09:54:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.889+0000 7f318fe00700 1 -- 172.21.15.78:0/2245936582 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f31a80163a0 con 0x55ca2a558030 2024-03-14T09:54:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.889+0000 7f31c8ba8b80 10 client.25756 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.889+0000 7f31c8ba8b80 10 client.25756 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:54:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.889+0000 7f31c8ba8b80 20 client.25756 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.889+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/2245936582 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55ca2a5ba710 0x55ca2a5bcb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.889+0000 7f31c8ba8b80 4 client.25756 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:54:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.889+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/2245936582 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ca2a498700 con 0x55ca2a5ba710 2024-03-14T09:54:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.890+0000 7f31ade00700 1 --2- 172.21.15.78:0/2245936582 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55ca2a5ba710 0x55ca2a5bcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.890+0000 7f31ade00700 1 --2- 172.21.15.78:0/2245936582 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55ca2a5ba710 0x55ca2a5bcb30 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=aa1cb3e0dc5addb9 server_cookie=b76c1e43d8c054fe in_seq=0 out_seq=0 2024-03-14T09:54:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.890+0000 7f318fe00700 10 client.25756 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:54:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.891+0000 7f318fe00700 1 -- 172.21.15.78:0/2245936582 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ca2a498700 con 0x55ca2a5ba710 2024-03-14T09:54:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.891+0000 7f318fe00700 10 client.25756 handle_command_reply: tid=0 2024-03-14T09:54:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.891+0000 7f31c8ba8b80 1 client.25756 shutdown 2024-03-14T09:54:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.891+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/2245936582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 msgr2=0x55ca2a5b8890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.891+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/2245936582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 0x55ca2a5b8890 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f31a8004430 tx=0x7f31a8016f50 comp rx=0 tx=0).stop 2024-03-14T09:54:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.891+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/2245936582 shutdown_connections 2024-03-14T09:54:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.891+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/2245936582 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55ca2a5ba710 0x55ca2a5bcb30 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.891+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/2245936582 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ca2a558030 0x55ca2a5b8890 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.891+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/2245936582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ca2a54c860 0x55ca2a55cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.891+0000 7f318fe00700 0 client.25756 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:54:34.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.891+0000 7f31c8ba8b80 1 --2- 172.21.15.78:0/2245936582 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ca2a54d180 0x55ca2a55d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.891+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/2245936582 >> 172.21.15.78:0/2245936582 conn(0x55ca2a4c3220 msgr2=0x55ca2a54fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:34.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.891+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/2245936582 shutdown_connections 2024-03-14T09:54:34.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.891+0000 7f31c8ba8b80 1 -- 172.21.15.78:0/2245936582 wait complete. 2024-03-14T09:54:34.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.891+0000 7f31c8ba8b80 20 client.25756 trim_cache size 0 max 16384 2024-03-14T09:54:34.894 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:54:34.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.892+0000 7f31af200700 1 -- 172.21.15.78:0/262128822 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f31940a0c90 msgr2=0x7f31940a3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:34.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.892+0000 7f31af200700 1 --2- 172.21.15.78:0/262128822 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f31940a0c90 0x7f31940a3150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f31a80075c0 tx=0x7f31a8009040 comp rx=0 tx=0).stop 2024-03-14T09:54:34.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.892+0000 7f31af200700 1 -- 172.21.15.78:0/262128822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31b0100af0 msgr2=0x7f31b01995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:34.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.892+0000 7f31af200700 1 --2- 172.21.15.78:0/262128822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31b0100af0 0x7f31b01995a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f31a0005910 tx=0x7f31a0004730 comp rx=0 tx=0).stop 2024-03-14T09:54:34.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.892+0000 7f31af200700 1 -- 172.21.15.78:0/262128822 shutdown_connections 2024-03-14T09:54:34.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.892+0000 7f31af200700 1 --2- 172.21.15.78:0/262128822 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f31940a0c90 0x7f31940a3150 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.892+0000 7f31af200700 1 --2- 172.21.15.78:0/262128822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31b0100af0 0x7f31b01995a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.892+0000 7f31af200700 1 --2- 172.21.15.78:0/262128822 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f31b0106b90 0x7f31b019de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.892+0000 7f31af200700 1 --2- 172.21.15.78:0/262128822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b01014b0 0x7f31b0199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:34.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.892+0000 7f31af200700 1 -- 172.21.15.78:0/262128822 >> 172.21.15.78:0/262128822 conn(0x7f31b00fc5e0 msgr2=0x7f31b010c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:34.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.893+0000 7f31af200700 1 -- 172.21.15.78:0/262128822 shutdown_connections 2024-03-14T09:54:34.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:34.893+0000 7f31af200700 1 -- 172.21.15.78:0/262128822 wait complete. 2024-03-14T09:54:34.922 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T09:54:35.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.153+0000 7fe232a00700 1 -- 172.21.15.78:0/2899019305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe22c1018d0 msgr2=0x7fe22c105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.153+0000 7fe232a00700 1 --2- 172.21.15.78:0/2899019305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe22c1018d0 0x7fe22c105910 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe218009990 tx=0x7fe218009ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.154+0000 7fe232a00700 1 -- 172.21.15.78:0/2899019305 shutdown_connections 2024-03-14T09:54:35.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.154+0000 7fe232a00700 1 --2- 172.21.15.78:0/2899019305 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe22c106040 0x7fe22c10dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.154+0000 7fe232a00700 1 --2- 172.21.15.78:0/2899019305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c100f20 0x7fe22c101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.154+0000 7fe232a00700 1 --2- 172.21.15.78:0/2899019305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe22c1018d0 0x7fe22c105910 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.154+0000 7fe232a00700 1 -- 172.21.15.78:0/2899019305 >> 172.21.15.78:0/2899019305 conn(0x7fe22c0fc7a0 msgr2=0x7fe22c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:35.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.154+0000 7fe232a00700 1 -- 172.21.15.78:0/2899019305 shutdown_connections 2024-03-14T09:54:35.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.154+0000 7fe232a00700 1 -- 172.21.15.78:0/2899019305 wait complete. 2024-03-14T09:54:35.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.155+0000 7fe232a00700 1 Processor -- start 2024-03-14T09:54:35.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.156+0000 7fe232a00700 1 -- start start 2024-03-14T09:54:35.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.156+0000 7fe232a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe22c100f20 0x7fe22c197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.156+0000 7fe232a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c106040 0x7fe22c197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.156+0000 7fe232a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe22c19bf90 0x7fe22c19c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.156+0000 7fe232a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c19ca10 con 0x7fe22c106040 2024-03-14T09:54:35.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.156+0000 7fe232a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c19cb50 con 0x7fe22c100f20 2024-03-14T09:54:35.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.156+0000 7fe232a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c19ccc0 con 0x7fe22c19bf90 2024-03-14T09:54:35.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.157+0000 7fe22be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe22c100f20 0x7fe22c197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.157+0000 7fe22be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe22c100f20 0x7fe22c197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57362/0 (socket says 172.21.15.78:57362) 2024-03-14T09:54:35.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.157+0000 7fe230a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe22c19bf90 0x7fe22c19c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.157+0000 7fe22be00700 1 -- 172.21.15.78:0/1487106146 learned_addr learned my addr 172.21.15.78:0/1487106146 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.157+0000 7fe22be00700 1 -- 172.21.15.78:0/1487106146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe22c19bf90 msgr2=0x7fe22c19c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.157+0000 7fe22be00700 1 --2- 172.21.15.78:0/1487106146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe22c19bf90 0x7fe22c19c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.157+0000 7fe22b400700 1 --2- 172.21.15.78:0/1487106146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c106040 0x7fe22c197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.157+0000 7fe22be00700 1 -- 172.21.15.78:0/1487106146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c106040 msgr2=0x7fe22c197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.157+0000 7fe22be00700 1 --2- 172.21.15.78:0/1487106146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c106040 0x7fe22c197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.157+0000 7fe22be00700 1 -- 172.21.15.78:0/1487106146 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2180095f0 con 0x7fe22c100f20 2024-03-14T09:54:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.157+0000 7fe230a00700 1 --2- 172.21.15.78:0/1487106146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe22c19bf90 0x7fe22c19c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.157+0000 7fe22b400700 1 --2- 172.21.15.78:0/1487106146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c106040 0x7fe22c197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:54:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.158+0000 7fe22be00700 1 --2- 172.21.15.78:0/1487106146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe22c100f20 0x7fe22c197700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00e9a0 tx=0x7fe21c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.158+0000 7fe228c00700 1 -- 172.21.15.78:0/1487106146 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c009790 con 0x7fe22c100f20 2024-03-14T09:54:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.158+0000 7fe228c00700 1 -- 172.21.15.78:0/1487106146 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe21c00cb50 con 0x7fe22c100f20 2024-03-14T09:54:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.158+0000 7fe232a00700 1 -- 172.21.15.78:0/1487106146 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22c06a7d0 con 0x7fe22c100f20 2024-03-14T09:54:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.158+0000 7fe228c00700 1 -- 172.21.15.78:0/1487106146 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c018680 con 0x7fe22c100f20 2024-03-14T09:54:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.158+0000 7fe232a00700 1 -- 172.21.15.78:0/1487106146 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22c06ad30 con 0x7fe22c100f20 2024-03-14T09:54:35.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.163+0000 7fe228c00700 1 -- 172.21.15.78:0/1487106146 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 25) v1 ==== 135564+0+0 (secure 0 0 0) 0x7fe21c00ccc0 con 0x7fe22c100f20 2024-03-14T09:54:35.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.165+0000 7fe2464f4b80 1 Processor -- start 2024-03-14T09:54:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.165+0000 7fe2464f4b80 1 -- start start 2024-03-14T09:54:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.165+0000 7fe228c00700 1 --2- 172.21.15.78:0/1487106146 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe2100a0d10 0x7fe2100a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.166+0000 7fe2464f4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46680 0x5601b1f46a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.166+0000 7fe2464f4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46fa0 0x5601b1f50980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.166+0000 7fe2464f4b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f50ec0 0x5601b1f532b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.166+0000 7fe2464f4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b1e9f990 con 0x5601b1f46fa0 2024-03-14T09:54:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.166+0000 7fe2464f4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b1e9d350 con 0x5601b1f46680 2024-03-14T09:54:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.166+0000 7fe2464f4b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b1e98c40 con 0x5601b1f50ec0 2024-03-14T09:54:35.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.166+0000 7fe228c00700 1 -- 172.21.15.78:0/1487106146 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe21c014070 con 0x7fe22c100f20 2024-03-14T09:54:35.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.166+0000 7fe230a00700 1 --2- 172.21.15.78:0/1487106146 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe2100a0d10 0x7fe2100a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.166+0000 7fe22be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46680 0x5601b1f46a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.167+0000 7fe230a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f50ec0 0x5601b1f532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.167+0000 7fe22b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46fa0 0x5601b1f50980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.167+0000 7fe22b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46fa0 0x5601b1f50980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37380/0 (socket says 172.21.15.78:37380) 2024-03-14T09:54:35.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.167+0000 7fe22b400700 1 -- 172.21.15.78:0/3974782346 learned_addr learned my addr 172.21.15.78:0/3974782346 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:35.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.167+0000 7fe22be00700 1 -- 172.21.15.78:0/3974782346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f50ec0 msgr2=0x5601b1f532b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:35.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.167+0000 7fe22be00700 1 --2- 172.21.15.78:0/3974782346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f50ec0 0x5601b1f532b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.167+0000 7fe22be00700 1 -- 172.21.15.78:0/3974782346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46fa0 msgr2=0x5601b1f50980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:35.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.167+0000 7fe22be00700 1 --2- 172.21.15.78:0/3974782346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46fa0 0x5601b1f50980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.167+0000 7fe22be00700 1 -- 172.21.15.78:0/3974782346 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601b1d1a3b0 con 0x5601b1f46680 2024-03-14T09:54:35.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.167+0000 7fe230a00700 1 --2- 172.21.15.78:0/1487106146 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe2100a0d10 0x7fe2100a31d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe224009fd0 tx=0x7fe224009920 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:35.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.168+0000 7fe22be00700 1 --2- 172.21.15.78:0/3974782346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46680 0x5601b1f46a60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe21c0c9df0 tx=0x7fe21c095db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41a816837e1d1675 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:35.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.168+0000 7fe221200700 1 -- 172.21.15.78:0/3974782346 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c09a040 con 0x5601b1f46680 2024-03-14T09:54:35.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.168+0000 7fe221200700 1 -- 172.21.15.78:0/3974782346 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe21c099940 con 0x5601b1f46680 2024-03-14T09:54:35.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.168+0000 7fe221200700 1 -- 172.21.15.78:0/3974782346 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c099030 con 0x5601b1f46680 2024-03-14T09:54:35.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.169+0000 7fe2464f4b80 1 -- 172.21.15.78:0/3974782346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46680 msgr2=0x5601b1f46a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.169+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/3974782346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46680 0x5601b1f46a60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe21c0c9df0 tx=0x7fe21c095db0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.169+0000 7fe2464f4b80 1 -- 172.21.15.78:0/3974782346 shutdown_connections 2024-03-14T09:54:35.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.169+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/3974782346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46680 0x5601b1f46a60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.169+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/3974782346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f50ec0 0x5601b1f532b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.169+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/3974782346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46fa0 0x5601b1f50980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.169+0000 7fe2464f4b80 1 -- 172.21.15.78:0/3974782346 >> 172.21.15.78:0/3974782346 conn(0x5601b1ebc220 msgr2=0x5601b1f41aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:35.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.170+0000 7fe2464f4b80 1 -- 172.21.15.78:0/3974782346 shutdown_connections 2024-03-14T09:54:35.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.170+0000 7fe2464f4b80 1 -- 172.21.15.78:0/3974782346 wait complete. 2024-03-14T09:54:35.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.170+0000 7fe2464f4b80 1 -- start start 2024-03-14T09:54:35.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.171+0000 7fe2464f4b80 1 Processor -- start 2024-03-14T09:54:35.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.171+0000 7fe2464f4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46680 0x5601b1f50950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.171+0000 7fe2464f4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46fa0 0x5601b1f4dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.171+0000 7fe2464f4b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f50ec0 0x5601b1f4e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.171+0000 7fe2464f4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b1e34d80 con 0x5601b1f46680 2024-03-14T09:54:35.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.171+0000 7fe2464f4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b1e34a20 con 0x5601b1f46fa0 2024-03-14T09:54:35.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.171+0000 7fe2464f4b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b1d1c6a0 con 0x5601b1f50ec0 2024-03-14T09:54:35.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.171+0000 7fe2464f4b80 10 client.0 _ll_register_callbacks cb 0x5601b1ca03a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:35.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.172+0000 7fe22be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46680 0x5601b1f50950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.172+0000 7fe22be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46680 0x5601b1f50950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37396/0 (socket says 172.21.15.78:37396) 2024-03-14T09:54:35.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.172+0000 7fe22be00700 1 -- 172.21.15.78:0/342775905 learned_addr learned my addr 172.21.15.78:0/342775905 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:35.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.172+0000 7fe22b400700 1 --2- 172.21.15.78:0/342775905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46fa0 0x5601b1f4dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.172+0000 7fe230a00700 1 --2- 172.21.15.78:0/342775905 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f50ec0 0x5601b1f4e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.172+0000 7fe22be00700 1 -- 172.21.15.78:0/342775905 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f50ec0 msgr2=0x5601b1f4e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:35.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.172+0000 7fe22be00700 1 --2- 172.21.15.78:0/342775905 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f50ec0 0x5601b1f4e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.172+0000 7fe22be00700 1 -- 172.21.15.78:0/342775905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46fa0 msgr2=0x5601b1f4dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:35.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.172+0000 7fe22be00700 1 --2- 172.21.15.78:0/342775905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46fa0 0x5601b1f4dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.172+0000 7fe22be00700 1 -- 172.21.15.78:0/342775905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22c06ad30 con 0x5601b1f46680 2024-03-14T09:54:35.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.173+0000 7fe22be00700 1 --2- 172.21.15.78:0/342775905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46680 0x5601b1f50950 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00e970 tx=0x7fe21c010aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de00904c76044fd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:35.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.173+0000 7fe215600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:54:35.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.173+0000 7fe215600700 1 -- 172.21.15.78:0/342775905 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c09d770 con 0x5601b1f46680 2024-03-14T09:54:35.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.173+0000 7fe215600700 1 -- 172.21.15.78:0/342775905 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe21c0a1dc0 con 0x5601b1f46680 2024-03-14T09:54:35.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.173+0000 7fe2464f4b80 1 -- 172.21.15.78:0/342775905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5601b1f39720 con 0x5601b1f46680 2024-03-14T09:54:35.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.173+0000 7fe215600700 1 -- 172.21.15.78:0/342775905 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe21c0b2250 con 0x5601b1f46680 2024-03-14T09:54:35.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.173+0000 7fe2464f4b80 10 client.15756 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:35.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.173+0000 7fe2464f4b80 1 -- 172.21.15.78:0/342775905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5601b1cc6b60 con 0x5601b1f46680 2024-03-14T09:54:35.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.174+0000 7fe215600700 1 -- 172.21.15.78:0/342775905 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5601b1f39720 con 0x5601b1f46680 2024-03-14T09:54:35.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.174+0000 7fe215600700 1 -- 172.21.15.78:0/342775905 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe21c0b90f0 con 0x5601b1f46680 2024-03-14T09:54:35.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.174+0000 7fe2464f4b80 10 client.15756 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:35.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.174+0000 7fe2464f4b80 10 client.15756 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:54:35.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.175+0000 7fe2464f4b80 20 client.15756 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:35.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.175+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/342775905 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5601b1f60890 0x5601b1f62cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.175+0000 7fe2464f4b80 4 client.15756 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:54:35.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.175+0000 7fe2464f4b80 1 -- 172.21.15.78:0/342775905 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5601b1e91700 con 0x5601b1f60890 2024-03-14T09:54:35.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.175+0000 7fe22b400700 1 --2- 172.21.15.78:0/342775905 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5601b1f60890 0x5601b1f62cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.175+0000 7fe22b400700 1 --2- 172.21.15.78:0/342775905 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5601b1f60890 0x5601b1f62cb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=db0265190870755d server_cookie=bdbad324f20fc780 in_seq=0 out_seq=0 2024-03-14T09:54:35.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.175+0000 7fe215600700 10 client.15756 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:54:35.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.176+0000 7fe215600700 1 -- 172.21.15.78:0/342775905 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5601b1e91700 con 0x5601b1f60890 2024-03-14T09:54:35.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.176+0000 7fe215600700 10 client.15756 handle_command_reply: tid=0 2024-03-14T09:54:35.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.176+0000 7fe2464f4b80 1 client.15756 shutdown 2024-03-14T09:54:35.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.176+0000 7fe2464f4b80 1 -- 172.21.15.78:0/342775905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46680 msgr2=0x5601b1f50950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.176+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/342775905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46680 0x5601b1f50950 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00e970 tx=0x7fe21c010aa0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.177+0000 7fe2464f4b80 1 -- 172.21.15.78:0/342775905 shutdown_connections 2024-03-14T09:54:35.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.177+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/342775905 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5601b1f60890 0x5601b1f62cb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.177+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/342775905 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f50ec0 0x5601b1f4e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.177+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/342775905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46680 0x5601b1f50950 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.177+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/342775905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46fa0 0x5601b1f4dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.177+0000 7fe2464f4b80 1 -- 172.21.15.78:0/342775905 >> 172.21.15.78:0/342775905 conn(0x5601b1ebc220 msgr2=0x5601b1f41690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:35.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.177+0000 7fe215600700 0 client.15756 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:54:35.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.177+0000 7fe2464f4b80 1 -- 172.21.15.78:0/342775905 shutdown_connections 2024-03-14T09:54:35.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.177+0000 7fe2464f4b80 1 -- 172.21.15.78:0/342775905 wait complete. 2024-03-14T09:54:35.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.177+0000 7fe2464f4b80 20 client.15756 trim_cache size 0 max 16384 2024-03-14T09:54:35.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.191+0000 7fe2464f4b80 1 Processor -- start 2024-03-14T09:54:35.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.191+0000 7fe2464f4b80 1 -- start start 2024-03-14T09:54:35.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.191+0000 7fe2464f4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46680 0x5601b1f41c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.191+0000 7fe2464f4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46fa0 0x5601b1f67610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.191+0000 7fe2464f4b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f50ec0 0x5601b1f50480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.191+0000 7fe2464f4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b1f42e30 con 0x5601b1f46fa0 2024-03-14T09:54:35.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.191+0000 7fe2464f4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b1f43060 con 0x5601b1f46680 2024-03-14T09:54:35.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.191+0000 7fe2464f4b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b1f4e060 con 0x5601b1f50ec0 2024-03-14T09:54:35.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.192+0000 7fe22be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46680 0x5601b1f41c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.192+0000 7fe22be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46680 0x5601b1f41c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57386/0 (socket says 172.21.15.78:57386) 2024-03-14T09:54:35.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.192+0000 7fe22be00700 1 -- 172.21.15.78:0/3349526004 learned_addr learned my addr 172.21.15.78:0/3349526004 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:35.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.192+0000 7fe230a00700 1 --2- 172.21.15.78:0/3349526004 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f50ec0 0x5601b1f50480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.192+0000 7fe22be00700 1 -- 172.21.15.78:0/3349526004 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f50ec0 msgr2=0x5601b1f50480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:35.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.192+0000 7fe22be00700 1 --2- 172.21.15.78:0/3349526004 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f50ec0 0x5601b1f50480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.192+0000 7fe22be00700 1 -- 172.21.15.78:0/3349526004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46fa0 msgr2=0x5601b1f67610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:35.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.192+0000 7fe22b400700 1 --2- 172.21.15.78:0/3349526004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46fa0 0x5601b1f67610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.192+0000 7fe22be00700 1 --2- 172.21.15.78:0/3349526004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46fa0 0x5601b1f67610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.192+0000 7fe22be00700 1 -- 172.21.15.78:0/3349526004 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601b1e5eea0 con 0x5601b1f46680 2024-03-14T09:54:35.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.192+0000 7fe22b400700 1 --2- 172.21.15.78:0/3349526004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46fa0 0x5601b1f67610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:54:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.192+0000 7fe22be00700 1 --2- 172.21.15.78:0/3349526004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46680 0x5601b1f41c30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe21c000c00 tx=0x7fe21c05df90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a3f2da6eaa21109 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.193+0000 7fe221c00700 1 -- 172.21.15.78:0/3349526004 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c0c9d30 con 0x5601b1f46680 2024-03-14T09:54:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.193+0000 7fe221c00700 1 -- 172.21.15.78:0/3349526004 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe21c0a1dc0 con 0x5601b1f46680 2024-03-14T09:54:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.193+0000 7fe221c00700 1 -- 172.21.15.78:0/3349526004 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c09a950 con 0x5601b1f46680 2024-03-14T09:54:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.194+0000 7fe2464f4b80 1 -- 172.21.15.78:0/3349526004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46680 msgr2=0x5601b1f41c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.194+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/3349526004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46680 0x5601b1f41c30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe21c000c00 tx=0x7fe21c05df90 comp rx=0 tx=0).stop 2024-03-14T09:54:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.194+0000 7fe2464f4b80 1 -- 172.21.15.78:0/3349526004 shutdown_connections 2024-03-14T09:54:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.194+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/3349526004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f46680 0x5601b1f41c30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.194+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/3349526004 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f50ec0 0x5601b1f50480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.194+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/3349526004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46fa0 0x5601b1f67610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.194+0000 7fe2464f4b80 1 -- 172.21.15.78:0/3349526004 >> 172.21.15.78:0/3349526004 conn(0x5601b1ebc220 msgr2=0x5601b1f4d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.194+0000 7fe2464f4b80 1 -- 172.21.15.78:0/3349526004 shutdown_connections 2024-03-14T09:54:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.194+0000 7fe2464f4b80 1 -- 172.21.15.78:0/3349526004 wait complete. 2024-03-14T09:54:35.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.194+0000 7fe2464f4b80 1 -- start start 2024-03-14T09:54:35.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.195+0000 7fe2464f4b80 1 Processor -- start 2024-03-14T09:54:35.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.195+0000 7fe2464f4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46680 0x5601b1f55b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.195+0000 7fe2464f4b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f46fa0 0x5601b1f56070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.195+0000 7fe2464f4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f50ec0 0x5601b1f565b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.195+0000 7fe2464f4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b1fb1440 con 0x5601b1f46680 2024-03-14T09:54:35.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.195+0000 7fe2464f4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b1fb15b0 con 0x5601b1f50ec0 2024-03-14T09:54:35.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.195+0000 7fe2464f4b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b1fb1770 con 0x5601b1f46fa0 2024-03-14T09:54:35.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.195+0000 7fe2464f4b80 10 client.0 _ll_register_callbacks cb 0x7fe2240097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:35.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.195+0000 7fe22be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46680 0x5601b1f55b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.196+0000 7fe22b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f46fa0 0x5601b1f56070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.196+0000 7fe22be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46680 0x5601b1f55b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37418/0 (socket says 172.21.15.78:37418) 2024-03-14T09:54:35.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.196+0000 7fe22be00700 1 -- 172.21.15.78:0/399960647 learned_addr learned my addr 172.21.15.78:0/399960647 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:35.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.196+0000 7fe230a00700 1 --2- 172.21.15.78:0/399960647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f50ec0 0x5601b1f565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.196+0000 7fe22b400700 1 -- 172.21.15.78:0/399960647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f50ec0 msgr2=0x5601b1f565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:35.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.196+0000 7fe22b400700 1 --2- 172.21.15.78:0/399960647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f50ec0 0x5601b1f565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.196+0000 7fe22b400700 1 -- 172.21.15.78:0/399960647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46680 msgr2=0x5601b1f55b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:35.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.196+0000 7fe22b400700 1 --2- 172.21.15.78:0/399960647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46680 0x5601b1f55b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.196+0000 7fe22b400700 1 -- 172.21.15.78:0/399960647 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601b1cc6b60 con 0x5601b1f46fa0 2024-03-14T09:54:35.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.196+0000 7fe22be00700 1 --2- 172.21.15.78:0/399960647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46680 0x5601b1f55b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:35.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.197+0000 7fe22b400700 1 --2- 172.21.15.78:0/399960647 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f46fa0 0x5601b1f56070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe218009990 tx=0x7fe218005a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cd843e06140d5d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:35.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.197+0000 7fe215600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:54:35.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.197+0000 7fe215600700 1 -- 172.21.15.78:0/399960647 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe218020040 con 0x5601b1f46fa0 2024-03-14T09:54:35.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.197+0000 7fe2464f4b80 1 -- 172.21.15.78:0/399960647 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe21c099940 con 0x5601b1f46fa0 2024-03-14T09:54:35.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.197+0000 7fe215600700 1 -- 172.21.15.78:0/399960647 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe21800bad0 con 0x5601b1f46fa0 2024-03-14T09:54:35.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.197+0000 7fe215600700 1 -- 172.21.15.78:0/399960647 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe21800bc90 con 0x5601b1f46fa0 2024-03-14T09:54:35.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.197+0000 7fe2464f4b80 10 client.25774 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:35.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.197+0000 7fe2464f4b80 1 -- 172.21.15.78:0/399960647 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5601b1d1ecd0 con 0x5601b1f46fa0 2024-03-14T09:54:35.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.197+0000 7fe215600700 1 -- 172.21.15.78:0/399960647 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe21c099940 con 0x5601b1f46fa0 2024-03-14T09:54:35.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.198+0000 7fe215600700 1 -- 172.21.15.78:0/399960647 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe21800f620 con 0x5601b1f46fa0 2024-03-14T09:54:35.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.198+0000 7fe2464f4b80 10 client.25774 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:35.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.198+0000 7fe2464f4b80 10 client.25774 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:54:35.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.198+0000 7fe2464f4b80 20 client.25774 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:35.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.198+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/399960647 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5601b1fb2cf0 0x5601b1fb5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.198+0000 7fe2464f4b80 4 client.25774 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:54:35.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.198+0000 7fe2464f4b80 1 -- 172.21.15.78:0/399960647 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5601b1ce7cf0 con 0x5601b1fb2cf0 2024-03-14T09:54:35.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.198+0000 7fe22be00700 1 --2- 172.21.15.78:0/399960647 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5601b1fb2cf0 0x5601b1fb5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.199+0000 7fe22be00700 1 --2- 172.21.15.78:0/399960647 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5601b1fb2cf0 0x5601b1fb5110 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6803f2c91662bc0a server_cookie=bae7e969b37d838d in_seq=0 out_seq=0 2024-03-14T09:54:35.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.199+0000 7fe215600700 10 client.25774 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:54:35.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.199+0000 7fe215600700 1 -- 172.21.15.78:0/399960647 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5601b1ce7cf0 con 0x5601b1fb2cf0 2024-03-14T09:54:35.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.199+0000 7fe215600700 10 client.25774 handle_command_reply: tid=0 2024-03-14T09:54:35.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.199+0000 7fe2464f4b80 1 client.25774 shutdown 2024-03-14T09:54:35.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.199+0000 7fe2464f4b80 1 -- 172.21.15.78:0/399960647 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f46fa0 msgr2=0x5601b1f56070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.199+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/399960647 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f46fa0 0x5601b1f56070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe218009990 tx=0x7fe218005a90 comp rx=0 tx=0).stop 2024-03-14T09:54:35.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.200+0000 7fe2464f4b80 1 -- 172.21.15.78:0/399960647 shutdown_connections 2024-03-14T09:54:35.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.200+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/399960647 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5601b1fb2cf0 0x5601b1fb5110 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.200+0000 7fe215600700 0 client.25774 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:54:35.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.200+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/399960647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5601b1f50ec0 0x5601b1f565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.200+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/399960647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5601b1f46680 0x5601b1f55b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.200+0000 7fe2464f4b80 1 --2- 172.21.15.78:0/399960647 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5601b1f46fa0 0x5601b1f56070 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.200+0000 7fe2464f4b80 1 -- 172.21.15.78:0/399960647 >> 172.21.15.78:0/399960647 conn(0x5601b1ebc220 msgr2=0x5601b1f5c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:35.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.200+0000 7fe2464f4b80 1 -- 172.21.15.78:0/399960647 shutdown_connections 2024-03-14T09:54:35.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.200+0000 7fe2464f4b80 1 -- 172.21.15.78:0/399960647 wait complete. 2024-03-14T09:54:35.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.200+0000 7fe2464f4b80 20 client.25774 trim_cache size 0 max 16384 2024-03-14T09:54:35.203 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:54:35.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.201+0000 7fe232a00700 1 -- 172.21.15.78:0/1487106146 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe2100a0d10 msgr2=0x7fe2100a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.201+0000 7fe232a00700 1 --2- 172.21.15.78:0/1487106146 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe2100a0d10 0x7fe2100a31d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe224009fd0 tx=0x7fe224009920 comp rx=0 tx=0).stop 2024-03-14T09:54:35.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.201+0000 7fe232a00700 1 -- 172.21.15.78:0/1487106146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe22c100f20 msgr2=0x7fe22c197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.201+0000 7fe232a00700 1 --2- 172.21.15.78:0/1487106146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe22c100f20 0x7fe22c197700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00e9a0 tx=0x7fe21c00ecb0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.203+0000 7fe232a00700 1 -- 172.21.15.78:0/1487106146 shutdown_connections 2024-03-14T09:54:35.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.203+0000 7fe232a00700 1 --2- 172.21.15.78:0/1487106146 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe2100a0d10 0x7fe2100a31d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.203+0000 7fe232a00700 1 --2- 172.21.15.78:0/1487106146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe22c100f20 0x7fe22c197700 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.203+0000 7fe232a00700 1 --2- 172.21.15.78:0/1487106146 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe22c19bf90 0x7fe22c19c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.203+0000 7fe232a00700 1 --2- 172.21.15.78:0/1487106146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c106040 0x7fe22c197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.203+0000 7fe232a00700 1 -- 172.21.15.78:0/1487106146 >> 172.21.15.78:0/1487106146 conn(0x7fe22c0fc7a0 msgr2=0x7fe22c0fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:35.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.203+0000 7fe232a00700 1 -- 172.21.15.78:0/1487106146 shutdown_connections 2024-03-14T09:54:35.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.203+0000 7fe232a00700 1 -- 172.21.15.78:0/1487106146 wait complete. 2024-03-14T09:54:35.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:35 smithi078 ceph-mon[38143]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T09:54:35.226 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:35 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2035485244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:54:35.229 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T09:54:35.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.438+0000 7fd847a00700 1 -- 172.21.15.78:0/1482639666 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd840073b00 msgr2=0x7fd84010cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.438+0000 7fd847a00700 1 --2- 172.21.15.78:0/1482639666 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd840073b00 0x7fd84010cfb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009990 tx=0x7fd82c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.439+0000 7fd847a00700 1 -- 172.21.15.78:0/1482639666 shutdown_connections 2024-03-14T09:54:35.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.439+0000 7fd847a00700 1 --2- 172.21.15.78:0/1482639666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd84010d4f0 0x7fd84010f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.439+0000 7fd847a00700 1 --2- 172.21.15.78:0/1482639666 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd8400731e0 0x7fd8400735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.439+0000 7fd847a00700 1 --2- 172.21.15.78:0/1482639666 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd840073b00 0x7fd84010cfb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.439+0000 7fd847a00700 1 -- 172.21.15.78:0/1482639666 >> 172.21.15.78:0/1482639666 conn(0x7fd8400fc7a0 msgr2=0x7fd8400febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:35.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.439+0000 7fd847a00700 1 -- 172.21.15.78:0/1482639666 shutdown_connections 2024-03-14T09:54:35.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.439+0000 7fd847a00700 1 -- 172.21.15.78:0/1482639666 wait complete. 2024-03-14T09:54:35.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.440+0000 7fd847a00700 1 Processor -- start 2024-03-14T09:54:35.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.440+0000 7fd847a00700 1 -- start start 2024-03-14T09:54:35.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.440+0000 7fd847a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8400731e0 0x7fd84019f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.440+0000 7fd847a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd840073b00 0x7fd84019fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.440+0000 7fd847a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd84010d4f0 0x7fd840199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.440+0000 7fd847a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd840199cc0 con 0x7fd84010d4f0 2024-03-14T09:54:35.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.440+0000 7fd847a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd840199e30 con 0x7fd8400731e0 2024-03-14T09:54:35.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.440+0000 7fd847a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd840199ff0 con 0x7fd840073b00 2024-03-14T09:54:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.441+0000 7fd845a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd84010d4f0 0x7fd840199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.441+0000 7fd845a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd84010d4f0 0x7fd840199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37432/0 (socket says 172.21.15.78:37432) 2024-03-14T09:54:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.441+0000 7fd845a00700 1 -- 172.21.15.78:0/3756034458 learned_addr learned my addr 172.21.15.78:0/3756034458 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.441+0000 7fd845000700 1 --2- 172.21.15.78:0/3756034458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8400731e0 0x7fd84019f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.441+0000 7fd83fe00700 1 --2- 172.21.15.78:0/3756034458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd840073b00 0x7fd84019fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.441+0000 7fd845a00700 1 -- 172.21.15.78:0/3756034458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd840073b00 msgr2=0x7fd84019fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.441+0000 7fd845a00700 1 --2- 172.21.15.78:0/3756034458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd840073b00 0x7fd84019fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.441+0000 7fd845a00700 1 -- 172.21.15.78:0/3756034458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8400731e0 msgr2=0x7fd84019f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.441+0000 7fd845a00700 1 --2- 172.21.15.78:0/3756034458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8400731e0 0x7fd84019f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.441+0000 7fd845a00700 1 -- 172.21.15.78:0/3756034458 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82c0095f0 con 0x7fd84010d4f0 2024-03-14T09:54:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.441+0000 7fd845000700 1 --2- 172.21.15.78:0/3756034458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8400731e0 0x7fd84019f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:54:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.442+0000 7fd845a00700 1 --2- 172.21.15.78:0/3756034458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd84010d4f0 0x7fd840199660 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd83800ef30 tx=0x7fd83800c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:35.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.442+0000 7fd83d600700 1 -- 172.21.15.78:0/3756034458 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838018070 con 0x7fd84010d4f0 2024-03-14T09:54:35.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.442+0000 7fd847a00700 1 -- 172.21.15.78:0/3756034458 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd84019a270 con 0x7fd84010d4f0 2024-03-14T09:54:35.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.442+0000 7fd83d600700 1 -- 172.21.15.78:0/3756034458 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd83800f040 con 0x7fd84010d4f0 2024-03-14T09:54:35.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.442+0000 7fd847a00700 1 -- 172.21.15.78:0/3756034458 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8401a6520 con 0x7fd84010d4f0 2024-03-14T09:54:35.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.442+0000 7fd83d600700 1 -- 172.21.15.78:0/3756034458 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838003e00 con 0x7fd84010d4f0 2024-03-14T09:54:35.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.447+0000 7fd83d600700 1 -- 172.21.15.78:0/3756034458 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 25) v1 ==== 135564+0+0 (secure 0 0 0) 0x7fd83800caa0 con 0x7fd84010d4f0 2024-03-14T09:54:35.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.448+0000 7fd85b54bb80 1 Processor -- start 2024-03-14T09:54:35.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.449+0000 7fd85b54bb80 1 -- start start 2024-03-14T09:54:35.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.449+0000 7fd85b54bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478680 0x5607f8478a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.449+0000 7fd85b54bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478fa0 0x5607f8482980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.449+0000 7fd85b54bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8482ec0 0x5607f84852b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.449+0000 7fd85b54bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f83d1990 con 0x5607f8478680 2024-03-14T09:54:35.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.449+0000 7fd85b54bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f83cf350 con 0x5607f8482ec0 2024-03-14T09:54:35.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.449+0000 7fd85b54bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f83cac40 con 0x5607f8478fa0 2024-03-14T09:54:35.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.449+0000 7fd83d600700 1 --2- 172.21.15.78:0/3756034458 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd8280a0d90 0x7fd8280a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.450+0000 7fd83d600700 1 -- 172.21.15.78:0/3756034458 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fd8380cc2f0 con 0x7fd84010d4f0 2024-03-14T09:54:35.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.450+0000 7fd845000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478fa0 0x5607f8482980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.450+0000 7fd845000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478fa0 0x5607f8482980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59142/0 (socket says 172.21.15.78:59142) 2024-03-14T09:54:35.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.450+0000 7fd845000700 1 -- 172.21.15.78:0/2231789182 learned_addr learned my addr 172.21.15.78:0/2231789182 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:35.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.450+0000 7fd83fe00700 1 --2- 172.21.15.78:0/2231789182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8482ec0 0x5607f84852b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.450+0000 7fd83fe00700 1 --2- 172.21.15.78:0/2231789182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478680 0x5607f8478a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.450+0000 7fd845a00700 1 --2- 172.21.15.78:0/3756034458 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd8280a0d90 0x7fd8280a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.450+0000 7fd83fe00700 1 -- 172.21.15.78:0/2231789182 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478fa0 msgr2=0x5607f8482980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:35.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.450+0000 7fd83fe00700 1 --2- 172.21.15.78:0/2231789182 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478fa0 0x5607f8482980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.450+0000 7fd83fe00700 1 -- 172.21.15.78:0/2231789182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478680 msgr2=0x5607f8478a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:35.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.450+0000 7fd83fe00700 1 --2- 172.21.15.78:0/2231789182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478680 0x5607f8478a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.450+0000 7fd83fe00700 1 -- 172.21.15.78:0/2231789182 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607f824c3b0 con 0x5607f8482ec0 2024-03-14T09:54:35.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.451+0000 7fd83fe00700 1 --2- 172.21.15.78:0/2231789182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8482ec0 0x5607f84852b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd82c0098e0 tx=0x7fd82c01a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40a04267b04f714e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:35.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.451+0000 7fd832000700 1 -- 172.21.15.78:0/2231789182 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c005700 con 0x5607f8482ec0 2024-03-14T09:54:35.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.451+0000 7fd832000700 1 -- 172.21.15.78:0/2231789182 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd82c01e070 con 0x5607f8482ec0 2024-03-14T09:54:35.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.451+0000 7fd832000700 1 -- 172.21.15.78:0/2231789182 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c00fe00 con 0x5607f8482ec0 2024-03-14T09:54:35.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.452+0000 7fd85b54bb80 1 -- 172.21.15.78:0/2231789182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8482ec0 msgr2=0x5607f84852b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.452+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/2231789182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8482ec0 0x5607f84852b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd82c0098e0 tx=0x7fd82c01a040 comp rx=0 tx=0).stop 2024-03-14T09:54:35.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.452+0000 7fd845a00700 1 --2- 172.21.15.78:0/3756034458 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd8280a0d90 0x7fd8280a3250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd8380c9630 tx=0x7fd838003a00 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:35.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.452+0000 7fd85b54bb80 1 -- 172.21.15.78:0/2231789182 shutdown_connections 2024-03-14T09:54:35.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.452+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/2231789182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8482ec0 0x5607f84852b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.452+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/2231789182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478680 0x5607f8478a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.452+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/2231789182 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478fa0 0x5607f8482980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.452+0000 7fd85b54bb80 1 -- 172.21.15.78:0/2231789182 >> 172.21.15.78:0/2231789182 conn(0x5607f83ee220 msgr2=0x5607f8473aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:35.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.452+0000 7fd85b54bb80 1 -- 172.21.15.78:0/2231789182 shutdown_connections 2024-03-14T09:54:35.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.452+0000 7fd85b54bb80 1 -- 172.21.15.78:0/2231789182 wait complete. 2024-03-14T09:54:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.453+0000 7fd85b54bb80 1 -- start start 2024-03-14T09:54:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.453+0000 7fd85b54bb80 1 Processor -- start 2024-03-14T09:54:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.454+0000 7fd85b54bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478680 0x5607f84828e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.454+0000 7fd85b54bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478fa0 0x5607f847fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.454+0000 7fd85b54bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8482ec0 0x5607f8480330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.454+0000 7fd85b54bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f8366d80 con 0x5607f8478fa0 2024-03-14T09:54:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.454+0000 7fd85b54bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f8366a20 con 0x5607f8482ec0 2024-03-14T09:54:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.454+0000 7fd85b54bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f824e6a0 con 0x5607f8478680 2024-03-14T09:54:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.454+0000 7fd85b54bb80 10 client.0 _ll_register_callbacks cb 0x5607f81d23a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.454+0000 7fd845000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478fa0 0x5607f847fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.454+0000 7fd845000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478fa0 0x5607f847fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37458/0 (socket says 172.21.15.78:37458) 2024-03-14T09:54:35.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.454+0000 7fd845000700 1 -- 172.21.15.78:0/817002494 learned_addr learned my addr 172.21.15.78:0/817002494 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.455+0000 7fd83fe00700 1 --2- 172.21.15.78:0/817002494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478680 0x5607f84828e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.455+0000 7fd845000700 1 -- 172.21.15.78:0/817002494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478680 msgr2=0x5607f84828e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.455+0000 7fd845000700 1 --2- 172.21.15.78:0/817002494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478680 0x5607f84828e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.455+0000 7fd845000700 1 -- 172.21.15.78:0/817002494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8482ec0 msgr2=0x5607f8480330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.455+0000 7fd83fe00700 1 --2- 172.21.15.78:0/817002494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8482ec0 0x5607f8480330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.455+0000 7fd845000700 1 --2- 172.21.15.78:0/817002494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8482ec0 0x5607f8480330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.455+0000 7fd845000700 1 -- 172.21.15.78:0/817002494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd840110620 con 0x5607f8478fa0 2024-03-14T09:54:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.455+0000 7fd845000700 1 --2- 172.21.15.78:0/817002494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478fa0 0x5607f847fd20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd834005950 tx=0x7fd834013a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a662c94ea6692e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.455+0000 7fd826a00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:54:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.455+0000 7fd826a00700 1 -- 172.21.15.78:0/817002494 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd834011730 con 0x5607f8478fa0 2024-03-14T09:54:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.455+0000 7fd826a00700 1 -- 172.21.15.78:0/817002494 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fd834011890 con 0x5607f8478fa0 2024-03-14T09:54:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.455+0000 7fd85b54bb80 1 -- 172.21.15.78:0/817002494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5607f846b720 con 0x5607f8478fa0 2024-03-14T09:54:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.456+0000 7fd826a00700 1 -- 172.21.15.78:0/817002494 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd834011a50 con 0x5607f8478fa0 2024-03-14T09:54:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.456+0000 7fd85b54bb80 10 client.15771 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.456+0000 7fd85b54bb80 1 -- 172.21.15.78:0/817002494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5607f81f8b60 con 0x5607f8478fa0 2024-03-14T09:54:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.456+0000 7fd826a00700 1 -- 172.21.15.78:0/817002494 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5607f846b720 con 0x5607f8478fa0 2024-03-14T09:54:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.456+0000 7fd826a00700 1 -- 172.21.15.78:0/817002494 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd83400e3d0 con 0x5607f8478fa0 2024-03-14T09:54:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.456+0000 7fd85b54bb80 10 client.15771 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.456+0000 7fd85b54bb80 10 client.15771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.457+0000 7fd85b54bb80 20 client.15771 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:35.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.457+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/817002494 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5607f8492b60 0x5607f8494f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.457+0000 7fd85b54bb80 4 client.15771 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:54:35.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.457+0000 7fd85b54bb80 1 -- 172.21.15.78:0/817002494 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5607f83c3700 con 0x5607f8492b60 2024-03-14T09:54:35.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.457+0000 7fd83fe00700 1 --2- 172.21.15.78:0/817002494 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5607f8492b60 0x5607f8494f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.458+0000 7fd826a00700 10 client.15771 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:35.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.458+0000 7fd83fe00700 1 --2- 172.21.15.78:0/817002494 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5607f8492b60 0x5607f8494f80 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2eb69b9f726c4dd2 server_cookie=a7f3f062e088df16 in_seq=0 out_seq=0 2024-03-14T09:54:35.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.459+0000 7fd826a00700 1 -- 172.21.15.78:0/817002494 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5607f83c3700 con 0x5607f8492b60 2024-03-14T09:54:35.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.459+0000 7fd826a00700 10 client.15771 handle_command_reply: tid=0 2024-03-14T09:54:35.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.459+0000 7fd85b54bb80 1 client.15771 shutdown 2024-03-14T09:54:35.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.459+0000 7fd85b54bb80 1 -- 172.21.15.78:0/817002494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478fa0 msgr2=0x5607f847fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.459+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/817002494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478fa0 0x5607f847fd20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd834005950 tx=0x7fd834013a70 comp rx=0 tx=0).stop 2024-03-14T09:54:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.459+0000 7fd85b54bb80 1 -- 172.21.15.78:0/817002494 shutdown_connections 2024-03-14T09:54:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.459+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/817002494 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5607f8492b60 0x5607f8494f80 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.459+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/817002494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8482ec0 0x5607f8480330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.459+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/817002494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478680 0x5607f84828e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.459+0000 7fd826a00700 0 client.15771 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.459+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/817002494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478fa0 0x5607f847fd20 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.459+0000 7fd85b54bb80 1 -- 172.21.15.78:0/817002494 >> 172.21.15.78:0/817002494 conn(0x5607f83ee220 msgr2=0x5607f8473690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.460+0000 7fd85b54bb80 1 -- 172.21.15.78:0/817002494 shutdown_connections 2024-03-14T09:54:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.460+0000 7fd85b54bb80 1 -- 172.21.15.78:0/817002494 wait complete. 2024-03-14T09:54:35.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.460+0000 7fd85b54bb80 20 client.15771 trim_cache size 0 max 16384 2024-03-14T09:54:35.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.473+0000 7fd85b54bb80 1 Processor -- start 2024-03-14T09:54:35.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.473+0000 7fd85b54bb80 1 -- start start 2024-03-14T09:54:35.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.473+0000 7fd85b54bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8478680 0x5607f8473c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.473+0000 7fd85b54bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478fa0 0x5607f84998e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.474+0000 7fd85b54bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8482ec0 0x5607f8489b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.474+0000 7fd85b54bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f8479af0 con 0x5607f8478fa0 2024-03-14T09:54:35.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.474+0000 7fd85b54bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f84901a0 con 0x5607f8478680 2024-03-14T09:54:35.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.474+0000 7fd85b54bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f8470230 con 0x5607f8482ec0 2024-03-14T09:54:35.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.474+0000 7fd845000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478fa0 0x5607f84998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.474+0000 7fd845000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478fa0 0x5607f84998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37466/0 (socket says 172.21.15.78:37466) 2024-03-14T09:54:35.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.474+0000 7fd845000700 1 -- 172.21.15.78:0/337074370 learned_addr learned my addr 172.21.15.78:0/337074370 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:35.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.474+0000 7fd845000700 1 -- 172.21.15.78:0/337074370 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8482ec0 msgr2=0x5607f8489b30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-14T09:54:35.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.474+0000 7fd845000700 1 --2- 172.21.15.78:0/337074370 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8482ec0 0x5607f8489b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.474+0000 7fd845000700 1 -- 172.21.15.78:0/337074370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8478680 msgr2=0x5607f8473c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:35.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.475+0000 7fd83fe00700 1 --2- 172.21.15.78:0/337074370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8478680 0x5607f8473c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.475+0000 7fd845000700 1 --2- 172.21.15.78:0/337074370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8478680 0x5607f8473c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.475+0000 7fd845000700 1 -- 172.21.15.78:0/337074370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607f8390ea0 con 0x5607f8478fa0 2024-03-14T09:54:35.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.475+0000 7fd83fe00700 1 --2- 172.21.15.78:0/337074370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8478680 0x5607f8473c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:54:35.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.475+0000 7fd845000700 1 --2- 172.21.15.78:0/337074370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478fa0 0x5607f84998e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd834005dc0 tx=0x7fd8340058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d695f346c55c3b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:35.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.475+0000 7fd832a00700 1 -- 172.21.15.78:0/337074370 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83400ec90 con 0x5607f8478fa0 2024-03-14T09:54:35.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.475+0000 7fd832a00700 1 -- 172.21.15.78:0/337074370 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd83400edf0 con 0x5607f8478fa0 2024-03-14T09:54:35.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.475+0000 7fd832a00700 1 -- 172.21.15.78:0/337074370 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd834011b90 con 0x5607f8478fa0 2024-03-14T09:54:35.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.476+0000 7fd85b54bb80 1 -- 172.21.15.78:0/337074370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478fa0 msgr2=0x5607f84998e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.476+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/337074370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478fa0 0x5607f84998e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd834005dc0 tx=0x7fd8340058e0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.476+0000 7fd85b54bb80 1 -- 172.21.15.78:0/337074370 shutdown_connections 2024-03-14T09:54:35.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.476+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/337074370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8478680 0x5607f8473c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.476+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/337074370 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8482ec0 0x5607f8489b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.476+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/337074370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8478fa0 0x5607f84998e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.476+0000 7fd85b54bb80 1 -- 172.21.15.78:0/337074370 >> 172.21.15.78:0/337074370 conn(0x5607f83ee220 msgr2=0x5607f84826d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:35.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.476+0000 7fd85b54bb80 1 -- 172.21.15.78:0/337074370 shutdown_connections 2024-03-14T09:54:35.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.476+0000 7fd85b54bb80 1 -- 172.21.15.78:0/337074370 wait complete. 2024-03-14T09:54:35.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.477+0000 7fd85b54bb80 1 -- start start 2024-03-14T09:54:35.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.477+0000 7fd85b54bb80 1 Processor -- start 2024-03-14T09:54:35.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.477+0000 7fd85b54bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478680 0x5607f8487b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.477+0000 7fd85b54bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8478fa0 0x5607f8488050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.477+0000 7fd85b54bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8482ec0 0x5607f8488590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.477+0000 7fd85b54bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f84ebfc0 con 0x5607f8482ec0 2024-03-14T09:54:35.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.477+0000 7fd85b54bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f84ec130 con 0x5607f8478fa0 2024-03-14T09:54:35.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.477+0000 7fd85b54bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f84ec2f0 con 0x5607f8478680 2024-03-14T09:54:35.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.478+0000 7fd85b54bb80 10 client.0 _ll_register_callbacks cb 0x7fd834009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:35.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.478+0000 7fd845000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8478fa0 0x5607f8488050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.478+0000 7fd83fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478680 0x5607f8487b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.478+0000 7fd83fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478680 0x5607f8487b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59174/0 (socket says 172.21.15.78:59174) 2024-03-14T09:54:35.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.478+0000 7fd83fe00700 1 -- 172.21.15.78:0/267723897 learned_addr learned my addr 172.21.15.78:0/267723897 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:35.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.478+0000 7fd83fe00700 1 --2- 172.21.15.78:0/267723897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8482ec0 0x5607f8488590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.478+0000 7fd83fe00700 1 -- 172.21.15.78:0/267723897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8478fa0 msgr2=0x5607f8488050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:35.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.478+0000 7fd83fe00700 1 --2- 172.21.15.78:0/267723897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8478fa0 0x5607f8488050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.478+0000 7fd83fe00700 1 -- 172.21.15.78:0/267723897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8482ec0 msgr2=0x5607f8488590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:35.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.478+0000 7fd83fe00700 1 --2- 172.21.15.78:0/267723897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8482ec0 0x5607f8488590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.478+0000 7fd83fe00700 1 -- 172.21.15.78:0/267723897 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5607f81f8b60 con 0x5607f8478680 2024-03-14T09:54:35.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.479+0000 7fd83fe00700 1 --2- 172.21.15.78:0/267723897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478680 0x5607f8487b10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x5607f84811d0 tx=0x7fd82c01a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bda8ed9644902c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:35.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.479+0000 7fd826a00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:54:35.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.479+0000 7fd826a00700 1 -- 172.21.15.78:0/267723897 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c02b040 con 0x5607f8478680 2024-03-14T09:54:35.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.479+0000 7fd826a00700 1 -- 172.21.15.78:0/267723897 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fd82c000e40 con 0x5607f8478680 2024-03-14T09:54:35.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.479+0000 7fd85b54bb80 1 -- 172.21.15.78:0/267723897 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd82c01e070 con 0x5607f8478680 2024-03-14T09:54:35.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.479+0000 7fd826a00700 1 -- 172.21.15.78:0/267723897 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd82c00bec0 con 0x5607f8478680 2024-03-14T09:54:35.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.479+0000 7fd85b54bb80 10 client.25786 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:35.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.479+0000 7fd85b54bb80 1 -- 172.21.15.78:0/267723897 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5607f8250cd0 con 0x5607f8478680 2024-03-14T09:54:35.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.480+0000 7fd826a00700 1 -- 172.21.15.78:0/267723897 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd82c01e070 con 0x5607f8478680 2024-03-14T09:54:35.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.480+0000 7fd826a00700 1 -- 172.21.15.78:0/267723897 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd82c00f950 con 0x5607f8478680 2024-03-14T09:54:35.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.480+0000 7fd85b54bb80 10 client.25786 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:35.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.480+0000 7fd85b54bb80 10 client.25786 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:35.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.480+0000 7fd85b54bb80 20 client.25786 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:35.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.480+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/267723897 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5607f84ed600 0x5607f84efa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:35.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.480+0000 7fd85b54bb80 4 client.25786 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:54:35.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.480+0000 7fd85b54bb80 1 -- 172.21.15.78:0/267723897 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5607f8219cf0 con 0x5607f84ed600 2024-03-14T09:54:35.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.481+0000 7fd845000700 1 --2- 172.21.15.78:0/267723897 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5607f84ed600 0x5607f84efa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:35.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.481+0000 7fd826a00700 10 client.25786 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:35.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.481+0000 7fd845000700 1 --2- 172.21.15.78:0/267723897 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5607f84ed600 0x5607f84efa20 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4d2acc0534fdbf0 server_cookie=bbd4f125c2f6f4d8 in_seq=0 out_seq=0 2024-03-14T09:54:35.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.482+0000 7fd826a00700 1 -- 172.21.15.78:0/267723897 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5607f8219cf0 con 0x5607f84ed600 2024-03-14T09:54:35.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.482+0000 7fd826a00700 10 client.25786 handle_command_reply: tid=0 2024-03-14T09:54:35.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.482+0000 7fd85b54bb80 1 client.25786 shutdown 2024-03-14T09:54:35.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.482+0000 7fd85b54bb80 1 -- 172.21.15.78:0/267723897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478680 msgr2=0x5607f8487b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.482+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/267723897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478680 0x5607f8487b10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x5607f84811d0 tx=0x7fd82c01a040 comp rx=0 tx=0).stop 2024-03-14T09:54:35.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.482+0000 7fd85b54bb80 1 -- 172.21.15.78:0/267723897 shutdown_connections 2024-03-14T09:54:35.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.482+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/267723897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5607f8482ec0 0x5607f8488590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.482+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/267723897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5607f8478680 0x5607f8487b10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.482+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/267723897 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5607f84ed600 0x5607f84efa20 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.482+0000 7fd85b54bb80 1 --2- 172.21.15.78:0/267723897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5607f8478fa0 0x5607f8488050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.482+0000 7fd826a00700 0 client.25786 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:35.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.482+0000 7fd85b54bb80 1 -- 172.21.15.78:0/267723897 >> 172.21.15.78:0/267723897 conn(0x5607f83ee220 msgr2=0x5607f847e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:35.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.482+0000 7fd85b54bb80 1 -- 172.21.15.78:0/267723897 shutdown_connections 2024-03-14T09:54:35.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.482+0000 7fd85b54bb80 1 -- 172.21.15.78:0/267723897 wait complete. 2024-03-14T09:54:35.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.482+0000 7fd85b54bb80 20 client.25786 trim_cache size 0 max 16384 2024-03-14T09:54:35.485 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:54:35.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.483+0000 7fd847a00700 1 -- 172.21.15.78:0/3756034458 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd8280a0d90 msgr2=0x7fd8280a3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.483+0000 7fd847a00700 1 --2- 172.21.15.78:0/3756034458 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd8280a0d90 0x7fd8280a3250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd8380c9630 tx=0x7fd838003a00 comp rx=0 tx=0).stop 2024-03-14T09:54:35.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.483+0000 7fd847a00700 1 -- 172.21.15.78:0/3756034458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd84010d4f0 msgr2=0x7fd840199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:35.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.483+0000 7fd847a00700 1 --2- 172.21.15.78:0/3756034458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd84010d4f0 0x7fd840199660 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd83800ef30 tx=0x7fd83800c450 comp rx=0 tx=0).stop 2024-03-14T09:54:35.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.483+0000 7fd847a00700 1 -- 172.21.15.78:0/3756034458 shutdown_connections 2024-03-14T09:54:35.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.483+0000 7fd847a00700 1 --2- 172.21.15.78:0/3756034458 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd8280a0d90 0x7fd8280a3250 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.483+0000 7fd847a00700 1 --2- 172.21.15.78:0/3756034458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8400731e0 0x7fd84019f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.483+0000 7fd847a00700 1 --2- 172.21.15.78:0/3756034458 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd84010d4f0 0x7fd840199660 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.483+0000 7fd847a00700 1 --2- 172.21.15.78:0/3756034458 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd840073b00 0x7fd84019fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:35.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.483+0000 7fd847a00700 1 -- 172.21.15.78:0/3756034458 >> 172.21.15.78:0/3756034458 conn(0x7fd8400fc7a0 msgr2=0x7fd8400fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:35.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.484+0000 7fd847a00700 1 -- 172.21.15.78:0/3756034458 shutdown_connections 2024-03-14T09:54:35.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:35.484+0000 7fd847a00700 1 -- 172.21.15.78:0/3756034458 wait complete. 2024-03-14T09:54:35.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:35 smithi152 ceph-mon[48279]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T09:54:35.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:35 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2035485244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:54:35.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:35 smithi132 ceph-mon[47959]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T09:54:35.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:35 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2035485244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:54:36.504 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T09:54:36.505 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T09:54:36.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.746+0000 7fbaf7200700 1 -- 172.21.15.78:0/2168949241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbaf8100250 msgr2=0x7fbaf810cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:36.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.746+0000 7fbaf7200700 1 --2- 172.21.15.78:0/2168949241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbaf8100250 0x7fbaf810cf10 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbae4009990 tx=0x7fbae4009ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.746+0000 7fbaf7200700 1 -- 172.21.15.78:0/2168949241 shutdown_connections 2024-03-14T09:54:36.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.746+0000 7fbaf7200700 1 --2- 172.21.15.78:0/2168949241 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbaf80ff8a0 0x7fbaf80ffc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.746+0000 7fbaf7200700 1 --2- 172.21.15.78:0/2168949241 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbaf810d450 0x7fbaf810f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.746+0000 7fbaf7200700 1 --2- 172.21.15.78:0/2168949241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbaf8100250 0x7fbaf810cf10 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.746+0000 7fbaf7200700 1 -- 172.21.15.78:0/2168949241 >> 172.21.15.78:0/2168949241 conn(0x7fbaf80fb4e0 msgr2=0x7fbaf80fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:36.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.746+0000 7fbaf7200700 1 -- 172.21.15.78:0/2168949241 shutdown_connections 2024-03-14T09:54:36.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.746+0000 7fbaf7200700 1 -- 172.21.15.78:0/2168949241 wait complete. 2024-03-14T09:54:36.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.748+0000 7fbaf7200700 1 Processor -- start 2024-03-14T09:54:36.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.748+0000 7fbaf7200700 1 -- start start 2024-03-14T09:54:36.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.748+0000 7fbaf7200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbaf80ff8a0 0x7fbaf8199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.748+0000 7fbaf7200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbaf8100250 0x7fbaf8199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.749+0000 7fbaf5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbaf80ff8a0 0x7fbaf8199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.749+0000 7fbaf5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbaf80ff8a0 0x7fbaf8199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37492/0 (socket says 172.21.15.78:37492) 2024-03-14T09:54:36.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.749+0000 7fbaf7200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbaf810d450 0x7fbaf819dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.749+0000 7fbaf7200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf819e520 con 0x7fbaf80ff8a0 2024-03-14T09:54:36.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.749+0000 7fbaf7200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf819e690 con 0x7fbaf810d450 2024-03-14T09:54:36.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.749+0000 7fbaf5400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbaf8100250 0x7fbaf8199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.749+0000 7fbaf7200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf819e850 con 0x7fbaf8100250 2024-03-14T09:54:36.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.749+0000 7fbaf5e00700 1 -- 172.21.15.78:0/3873334235 learned_addr learned my addr 172.21.15.78:0/3873334235 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:36.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.749+0000 7fbaf5400700 1 -- 172.21.15.78:0/3873334235 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbaf810d450 msgr2=0x7fbaf819dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:36.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.749+0000 7fbaf6800700 1 --2- 172.21.15.78:0/3873334235 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbaf810d450 0x7fbaf819dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.749+0000 7fbaf5400700 1 --2- 172.21.15.78:0/3873334235 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbaf810d450 0x7fbaf819dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.749+0000 7fbaf5400700 1 -- 172.21.15.78:0/3873334235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbaf80ff8a0 msgr2=0x7fbaf8199600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:36.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.749+0000 7fbaf5400700 1 --2- 172.21.15.78:0/3873334235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbaf80ff8a0 0x7fbaf8199600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.750+0000 7fbaf5400700 1 -- 172.21.15.78:0/3873334235 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae8009520 con 0x7fbaf8100250 2024-03-14T09:54:36.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.750+0000 7fbaf6800700 1 --2- 172.21.15.78:0/3873334235 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbaf810d450 0x7fbaf819dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:54:36.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.750+0000 7fbaf5400700 1 --2- 172.21.15.78:0/3873334235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbaf8100250 0x7fbaf8199b40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbae40098e0 tx=0x7fbae4005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:36.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.750+0000 7fbaeea00700 1 -- 172.21.15.78:0/3873334235 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae401d070 con 0x7fbaf8100250 2024-03-14T09:54:36.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.750+0000 7fbaeea00700 1 -- 172.21.15.78:0/3873334235 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbae4022470 con 0x7fbaf8100250 2024-03-14T09:54:36.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.750+0000 7fbaf7200700 1 -- 172.21.15.78:0/3873334235 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae40095f0 con 0x7fbaf8100250 2024-03-14T09:54:36.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.750+0000 7fbaeea00700 1 -- 172.21.15.78:0/3873334235 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae4017650 con 0x7fbaf8100250 2024-03-14T09:54:36.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.750+0000 7fbaf7200700 1 -- 172.21.15.78:0/3873334235 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf81a6190 con 0x7fbaf8100250 2024-03-14T09:54:36.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.755+0000 7fbaeea00700 1 -- 172.21.15.78:0/3873334235 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 25) v1 ==== 135564+0+0 (secure 0 0 0) 0x7fbae40225e0 con 0x7fbaf8100250 2024-03-14T09:54:36.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.757+0000 7fbb10db1b80 1 Processor -- start 2024-03-14T09:54:36.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.757+0000 7fbaeea00700 1 --2- 172.21.15.78:0/3873334235 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbadc0a0d10 0x7fbadc0a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.757+0000 7fbb10db1b80 1 -- start start 2024-03-14T09:54:36.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.757+0000 7fbaeea00700 1 -- 172.21.15.78:0/3873334235 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fbae40cdba0 con 0x7fbaf8100250 2024-03-14T09:54:36.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.757+0000 7fbaf6800700 1 --2- 172.21.15.78:0/3873334235 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbadc0a0d10 0x7fbadc0a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.757+0000 7fbb10db1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee8f0f0 0x55c66ee8f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.757+0000 7fbb10db1b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 0x55c66ee9a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.757+0000 7fbb10db1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee9a8c0 0x55c66ee9ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.757+0000 7fbb10db1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66ede0a10 con 0x55c66ee9a8c0 2024-03-14T09:54:36.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.757+0000 7fbb10db1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66edde3d0 con 0x55c66ee8f0f0 2024-03-14T09:54:36.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.757+0000 7fbb10db1b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66edd9cc0 con 0x55c66ee8fa10 2024-03-14T09:54:36.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.758+0000 7fbaf5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee8f0f0 0x55c66ee8f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.758+0000 7fbaf5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee8f0f0 0x55c66ee8f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57462/0 (socket says 172.21.15.78:57462) 2024-03-14T09:54:36.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.758+0000 7fbaf5e00700 1 -- 172.21.15.78:0/2571644149 learned_addr learned my addr 172.21.15.78:0/2571644149 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:36.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.758+0000 7fbaf5400700 1 --2- 172.21.15.78:0/2571644149 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 0x55c66ee9a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.758+0000 7fbaf6800700 1 --2- 172.21.15.78:0/2571644149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee9a8c0 0x55c66ee9ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.758+0000 7fbaf5e00700 1 -- 172.21.15.78:0/2571644149 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 msgr2=0x55c66ee9a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:36.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.758+0000 7fbaf6800700 1 --2- 172.21.15.78:0/3873334235 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbadc0a0d10 0x7fbadc0a31d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0009fd0 tx=0x7fbaf0009920 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:36.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.758+0000 7fbaf5e00700 1 --2- 172.21.15.78:0/2571644149 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 0x55c66ee9a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.758+0000 7fbaf5e00700 1 -- 172.21.15.78:0/2571644149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee9a8c0 msgr2=0x55c66ee9ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:36.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.758+0000 7fbaf5e00700 1 --2- 172.21.15.78:0/2571644149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee9a8c0 0x55c66ee9ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.758+0000 7fbaf5e00700 1 -- 172.21.15.78:0/2571644149 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c66ec5b430 con 0x55c66ee8f0f0 2024-03-14T09:54:36.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.759+0000 7fbaf5e00700 1 --2- 172.21.15.78:0/2571644149 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee8f0f0 0x55c66ee8f4d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbae80114c0 tx=0x7fbae80117d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e46cf1fdc7566838 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:36.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.759+0000 7fbae3400700 1 -- 172.21.15.78:0/2571644149 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae8019630 con 0x55c66ee8f0f0 2024-03-14T09:54:36.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.759+0000 7fbae3400700 1 -- 172.21.15.78:0/2571644149 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbae8019c50 con 0x55c66ee8f0f0 2024-03-14T09:54:36.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.759+0000 7fbae3400700 1 -- 172.21.15.78:0/2571644149 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae800fd20 con 0x55c66ee8f0f0 2024-03-14T09:54:36.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.760+0000 7fbb10db1b80 1 -- 172.21.15.78:0/2571644149 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee8f0f0 msgr2=0x55c66ee8f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:36.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.760+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/2571644149 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee8f0f0 0x55c66ee8f4d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbae80114c0 tx=0x7fbae80117d0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.760+0000 7fbb10db1b80 1 -- 172.21.15.78:0/2571644149 shutdown_connections 2024-03-14T09:54:36.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.760+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/2571644149 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee8f0f0 0x55c66ee8f4d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.760+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/2571644149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee9a8c0 0x55c66ee9ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.760+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/2571644149 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 0x55c66ee9a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.760+0000 7fbb10db1b80 1 -- 172.21.15.78:0/2571644149 >> 172.21.15.78:0/2571644149 conn(0x55c66edfd2a0 msgr2=0x55c66ee82d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:36.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.760+0000 7fbb10db1b80 1 -- 172.21.15.78:0/2571644149 shutdown_connections 2024-03-14T09:54:36.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.760+0000 7fbb10db1b80 1 -- 172.21.15.78:0/2571644149 wait complete. 2024-03-14T09:54:36.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.761+0000 7fbb10db1b80 1 -- start start 2024-03-14T09:54:36.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.761+0000 7fbb10db1b80 1 Processor -- start 2024-03-14T09:54:36.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.762+0000 7fbb10db1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee9a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.762+0000 7fbb10db1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee8fa10 0x55c66ee97620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.762+0000 7fbb10db1b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee9a8c0 0x55c66ee97d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.762+0000 7fbb10db1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66ed75e00 con 0x55c66ee8f0f0 2024-03-14T09:54:36.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.762+0000 7fbb10db1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66ed75aa0 con 0x55c66ee8fa10 2024-03-14T09:54:36.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.762+0000 7fbb10db1b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66ec5d720 con 0x55c66ee9a8c0 2024-03-14T09:54:36.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.762+0000 7fbb10db1b80 10 client.0 _ll_register_callbacks cb 0x55c66ebe1420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:36.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.762+0000 7fbaf5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee9a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.763+0000 7fbaf5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee9a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37510/0 (socket says 172.21.15.78:37510) 2024-03-14T09:54:36.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.763+0000 7fbaf5e00700 1 -- 172.21.15.78:0/3193050028 learned_addr learned my addr 172.21.15.78:0/3193050028 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:36.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.763+0000 7fbaf6800700 1 --2- 172.21.15.78:0/3193050028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee9a8c0 0x55c66ee97d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.763+0000 7fbaf5400700 1 --2- 172.21.15.78:0/3193050028 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee8fa10 0x55c66ee97620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.763+0000 7fbaf5e00700 1 -- 172.21.15.78:0/3193050028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee9a8c0 msgr2=0x55c66ee97d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:36.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.763+0000 7fbaf5e00700 1 --2- 172.21.15.78:0/3193050028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee9a8c0 0x55c66ee97d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.763+0000 7fbaf5e00700 1 -- 172.21.15.78:0/3193050028 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee8fa10 msgr2=0x55c66ee97620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:36.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.763+0000 7fbaf5e00700 1 --2- 172.21.15.78:0/3193050028 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee8fa10 0x55c66ee97620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.763+0000 7fbaf5e00700 1 -- 172.21.15.78:0/3193050028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae8009c90 con 0x55c66ee8f0f0 2024-03-14T09:54:36.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.764+0000 7fbaf5e00700 1 --2- 172.21.15.78:0/3193050028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee9a330 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbae800f8a0 tx=0x7fbae8005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=125b2952747f7dd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:36.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.764+0000 7fbad7e00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:54:36.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.764+0000 7fbad7e00700 1 -- 172.21.15.78:0/3193050028 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae8004440 con 0x55c66ee8f0f0 2024-03-14T09:54:36.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.764+0000 7fbad7e00700 1 -- 172.21.15.78:0/3193050028 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fbae8019430 con 0x55c66ee8f0f0 2024-03-14T09:54:36.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.764+0000 7fbb10db1b80 1 -- 172.21.15.78:0/3193050028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c66ee7a7a0 con 0x55c66ee8f0f0 2024-03-14T09:54:36.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.764+0000 7fbad7e00700 1 -- 172.21.15.78:0/3193050028 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbae80195f0 con 0x55c66ee8f0f0 2024-03-14T09:54:36.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.765+0000 7fbb10db1b80 10 client.15789 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:36.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.765+0000 7fbb10db1b80 1 -- 172.21.15.78:0/3193050028 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c66ec07be0 con 0x55c66ee8f0f0 2024-03-14T09:54:36.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.765+0000 7fbad7e00700 1 -- 172.21.15.78:0/3193050028 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c66ee7a7a0 con 0x55c66ee8f0f0 2024-03-14T09:54:36.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.765+0000 7fbad7e00700 1 -- 172.21.15.78:0/3193050028 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbae8005520 con 0x55c66ee8f0f0 2024-03-14T09:54:36.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.765+0000 7fbb10db1b80 10 client.15789 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:36.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.765+0000 7fbb10db1b80 10 client.15789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:36.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.765+0000 7fbb10db1b80 20 client.15789 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:36.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.765+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/3193050028 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c66eeaa3e0 0x55c66eeac800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.765+0000 7fbb10db1b80 4 client.15789 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:54:36.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.765+0000 7fbb10db1b80 1 -- 172.21.15.78:0/3193050028 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c66edd2780 con 0x55c66eeaa3e0 2024-03-14T09:54:36.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.766+0000 7fbaf5400700 1 --2- 172.21.15.78:0/3193050028 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c66eeaa3e0 0x55c66eeac800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.766+0000 7fbad7e00700 10 client.15789 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:36.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.766+0000 7fbaf5400700 1 --2- 172.21.15.78:0/3193050028 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c66eeaa3e0 0x55c66eeac800 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae53ad0635bcd5bd server_cookie=bc7be18a603be010 in_seq=0 out_seq=0 2024-03-14T09:54:36.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.767+0000 7fbad7e00700 1 -- 172.21.15.78:0/3193050028 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55c66edd2780 con 0x55c66eeaa3e0 2024-03-14T09:54:36.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.767+0000 7fbad7e00700 10 client.15789 handle_command_reply: tid=0 2024-03-14T09:54:36.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.767+0000 7fbb10db1b80 1 client.15789 shutdown 2024-03-14T09:54:36.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.767+0000 7fbb10db1b80 1 -- 172.21.15.78:0/3193050028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 msgr2=0x55c66ee9a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:36.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.767+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/3193050028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee9a330 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbae800f8a0 tx=0x7fbae8005ce0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.768+0000 7fbb10db1b80 1 -- 172.21.15.78:0/3193050028 shutdown_connections 2024-03-14T09:54:36.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.768+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/3193050028 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee9a8c0 0x55c66ee97d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.768+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/3193050028 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee9a330 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.768+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/3193050028 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c66eeaa3e0 0x55c66eeac800 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.768+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/3193050028 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee8fa10 0x55c66ee97620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.768+0000 7fbb10db1b80 1 -- 172.21.15.78:0/3193050028 >> 172.21.15.78:0/3193050028 conn(0x55c66edfd2a0 msgr2=0x55c66ee82900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:36.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.768+0000 7fbad7e00700 0 client.15789 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:36.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.768+0000 7fbb10db1b80 1 -- 172.21.15.78:0/3193050028 shutdown_connections 2024-03-14T09:54:36.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.768+0000 7fbb10db1b80 1 -- 172.21.15.78:0/3193050028 wait complete. 2024-03-14T09:54:36.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.768+0000 7fbb10db1b80 20 client.15789 trim_cache size 0 max 16384 2024-03-14T09:54:36.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.800+0000 7fbb10db1b80 1 Processor -- start 2024-03-14T09:54:36.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.801+0000 7fbb10db1b80 1 -- start start 2024-03-14T09:54:36.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.801+0000 7fbb10db1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee91f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.801+0000 7fbb10db1b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 0x55c66ee972b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.801+0000 7fbb10db1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee9a8c0 0x55c66ee948e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.801+0000 7fbb10db1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66ee8fdf0 con 0x55c66ee8f0f0 2024-03-14T09:54:36.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.801+0000 7fbb10db1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66ee8ce50 con 0x55c66ee9a8c0 2024-03-14T09:54:36.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.801+0000 7fbb10db1b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66ee7f2b0 con 0x55c66ee8fa10 2024-03-14T09:54:36.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.802+0000 7fbaf5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee91f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.802+0000 7fbaf5400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 0x55c66ee972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.802+0000 7fbaf5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee91f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37518/0 (socket says 172.21.15.78:37518) 2024-03-14T09:54:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.802+0000 7fbaf5e00700 1 -- 172.21.15.78:0/3989423411 learned_addr learned my addr 172.21.15.78:0/3989423411 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.802+0000 7fbaf6800700 1 --2- 172.21.15.78:0/3989423411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee9a8c0 0x55c66ee948e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.802+0000 7fbaf5400700 1 -- 172.21.15.78:0/3989423411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee9a8c0 msgr2=0x55c66ee948e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.802+0000 7fbaf5400700 1 --2- 172.21.15.78:0/3989423411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee9a8c0 0x55c66ee948e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.802+0000 7fbaf5400700 1 -- 172.21.15.78:0/3989423411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 msgr2=0x55c66ee91f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.802+0000 7fbaf5400700 1 --2- 172.21.15.78:0/3989423411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee91f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.802+0000 7fbaf5400700 1 -- 172.21.15.78:0/3989423411 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c66ed9ff20 con 0x55c66ee8fa10 2024-03-14T09:54:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.802+0000 7fbaf6800700 1 --2- 172.21.15.78:0/3989423411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee9a8c0 0x55c66ee948e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:54:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.803+0000 7fbaf5400700 1 --2- 172.21.15.78:0/3989423411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 0x55c66ee972b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbae40cae90 tx=0x7fbae402e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8465317a8d945042 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.803+0000 7fbae3e00700 1 -- 172.21.15.78:0/3989423411 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae409b2c0 con 0x55c66ee8fa10 2024-03-14T09:54:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.803+0000 7fbae3e00700 1 -- 172.21.15.78:0/3989423411 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbae409abc0 con 0x55c66ee8fa10 2024-03-14T09:54:36.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.803+0000 7fbae3e00700 1 -- 172.21.15.78:0/3989423411 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae409b080 con 0x55c66ee8fa10 2024-03-14T09:54:36.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.804+0000 7fbb10db1b80 1 -- 172.21.15.78:0/3989423411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 msgr2=0x55c66ee972b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:36.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.804+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/3989423411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 0x55c66ee972b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbae40cae90 tx=0x7fbae402e040 comp rx=0 tx=0).stop 2024-03-14T09:54:36.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.804+0000 7fbb10db1b80 1 -- 172.21.15.78:0/3989423411 shutdown_connections 2024-03-14T09:54:36.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.804+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/3989423411 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee9a8c0 0x55c66ee948e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.804+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/3989423411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee91f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.804+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/3989423411 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 0x55c66ee972b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.804+0000 7fbb10db1b80 1 -- 172.21.15.78:0/3989423411 >> 172.21.15.78:0/3989423411 conn(0x55c66edfd2a0 msgr2=0x55c66ee96ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:36.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.804+0000 7fbb10db1b80 1 -- 172.21.15.78:0/3989423411 shutdown_connections 2024-03-14T09:54:36.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.804+0000 7fbb10db1b80 1 -- 172.21.15.78:0/3989423411 wait complete. 2024-03-14T09:54:36.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.805+0000 7fbb10db1b80 1 -- start start 2024-03-14T09:54:36.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.805+0000 7fbb10db1b80 1 Processor -- start 2024-03-14T09:54:36.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.806+0000 7fbb10db1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee9ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.806+0000 7fbb10db1b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 0x55c66eea0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.806+0000 7fbb10db1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee9a8c0 0x55c66eefbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.806+0000 7fbb10db1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66eea0d70 con 0x55c66ee8f0f0 2024-03-14T09:54:36.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.806+0000 7fbb10db1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66eefc450 con 0x55c66ee9a8c0 2024-03-14T09:54:36.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.806+0000 7fbb10db1b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66eefc590 con 0x55c66ee8fa10 2024-03-14T09:54:36.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.806+0000 7fbb10db1b80 10 client.0 _ll_register_callbacks cb 0x7fbae801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:36.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.806+0000 7fbaf5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee9ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.806+0000 7fbaf5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee9ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37530/0 (socket says 172.21.15.78:37530) 2024-03-14T09:54:36.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.806+0000 7fbaf5e00700 1 -- 172.21.15.78:0/2193927605 learned_addr learned my addr 172.21.15.78:0/2193927605 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:36.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.806+0000 7fbaf6800700 1 --2- 172.21.15.78:0/2193927605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee9a8c0 0x55c66eefbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.806+0000 7fbaf5400700 1 --2- 172.21.15.78:0/2193927605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 0x55c66eea0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.807+0000 7fbaf5e00700 1 -- 172.21.15.78:0/2193927605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 msgr2=0x55c66eea0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:36.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.807+0000 7fbaf5e00700 1 --2- 172.21.15.78:0/2193927605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 0x55c66eea0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.807+0000 7fbaf5e00700 1 -- 172.21.15.78:0/2193927605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee9a8c0 msgr2=0x55c66eefbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:36.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.807+0000 7fbaf5e00700 1 --2- 172.21.15.78:0/2193927605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee9a8c0 0x55c66eefbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.807+0000 7fbaf5e00700 1 -- 172.21.15.78:0/2193927605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf81a6190 con 0x55c66ee8f0f0 2024-03-14T09:54:36.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.807+0000 7fbaf5400700 1 --2- 172.21.15.78:0/2193927605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 0x55c66eea0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:54:36.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.807+0000 7fbaf5e00700 1 --2- 172.21.15.78:0/2193927605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee9ff50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fbae800ee40 tx=0x7fbae800fd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dd5f4a35c0a2b15 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:36.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.807+0000 7fbad7e00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:54:36.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.807+0000 7fbad7e00700 1 -- 172.21.15.78:0/2193927605 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae8004440 con 0x55c66ee8f0f0 2024-03-14T09:54:36.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.807+0000 7fbad7e00700 1 -- 172.21.15.78:0/2193927605 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fbae80045a0 con 0x55c66ee8f0f0 2024-03-14T09:54:36.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.807+0000 7fbb10db1b80 1 -- 172.21.15.78:0/2193927605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbae8019c50 con 0x55c66ee8f0f0 2024-03-14T09:54:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.808+0000 7fbad7e00700 1 -- 172.21.15.78:0/2193927605 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbae800ee70 con 0x55c66ee8f0f0 2024-03-14T09:54:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.808+0000 7fbb10db1b80 10 client.15795 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.808+0000 7fbb10db1b80 1 -- 172.21.15.78:0/2193927605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c66ec5fd50 con 0x55c66ee8f0f0 2024-03-14T09:54:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.808+0000 7fbad7e00700 1 -- 172.21.15.78:0/2193927605 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbae8004a40 con 0x55c66ee8f0f0 2024-03-14T09:54:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.809+0000 7fbad7e00700 1 -- 172.21.15.78:0/2193927605 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbae8019430 con 0x55c66ee8f0f0 2024-03-14T09:54:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.809+0000 7fbb10db1b80 10 client.15795 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.809+0000 7fbb10db1b80 10 client.15795 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.809+0000 7fbb10db1b80 20 client.15795 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:36.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.809+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/2193927605 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c66eefd940 0x55c66eeffd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:36.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.809+0000 7fbb10db1b80 4 client.15795 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T09:54:36.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.809+0000 7fbb10db1b80 1 -- 172.21.15.78:0/2193927605 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x7fbae8019c50 con 0x55c66eefd940 2024-03-14T09:54:36.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.810+0000 7fbaf5400700 1 --2- 172.21.15.78:0/2193927605 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c66eefd940 0x55c66eeffd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:36.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.810+0000 7fbad7e00700 10 client.15795 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:36.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.810+0000 7fbaf5400700 1 --2- 172.21.15.78:0/2193927605 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c66eefd940 0x55c66eeffd60 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc237c5d0ae8580e server_cookie=7bd6d09538d692fd in_seq=0 out_seq=0 2024-03-14T09:54:36.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.812+0000 7fbad7e00700 1 -- 172.21.15.78:0/2193927605 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x7fbae8019c50 con 0x55c66eefd940 2024-03-14T09:54:36.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.812+0000 7fbad7e00700 10 client.15795 handle_command_reply: tid=0 2024-03-14T09:54:36.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.812+0000 7fbb10db1b80 1 client.15795 shutdown 2024-03-14T09:54:36.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.812+0000 7fbb10db1b80 1 -- 172.21.15.78:0/2193927605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 msgr2=0x55c66ee9ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:36.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.812+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/2193927605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee9ff50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fbae800ee40 tx=0x7fbae800fd60 comp rx=0 tx=0).stop 2024-03-14T09:54:36.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.812+0000 7fbb10db1b80 1 -- 172.21.15.78:0/2193927605 shutdown_connections 2024-03-14T09:54:36.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.812+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/2193927605 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c66eefd940 0x55c66eeffd60 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.812+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/2193927605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c66ee9a8c0 0x55c66eefbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.812+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/2193927605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66ee8f0f0 0x55c66ee9ff50 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.812+0000 7fbb10db1b80 1 --2- 172.21.15.78:0/2193927605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c66ee8fa10 0x55c66eea0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.812+0000 7fbb10db1b80 1 -- 172.21.15.78:0/2193927605 >> 172.21.15.78:0/2193927605 conn(0x55c66edfd2a0 msgr2=0x55c66ee95b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:36.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.812+0000 7fbad7e00700 0 client.15795 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:36.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.812+0000 7fbb10db1b80 1 -- 172.21.15.78:0/2193927605 shutdown_connections 2024-03-14T09:54:36.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.812+0000 7fbb10db1b80 1 -- 172.21.15.78:0/2193927605 wait complete. 2024-03-14T09:54:36.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.813+0000 7fbb10db1b80 20 client.15795 trim_cache size 0 max 16384 2024-03-14T09:54:36.814 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:54:36.814 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T09:54:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "1f99ba69-06c5-492b-8192-8662ab61eb88", 2024-03-14T09:54:36.815 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T09:54:36.815 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:54:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.813+0000 7fbaf7200700 1 -- 172.21.15.78:0/3873334235 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbadc0a0d10 msgr2=0x7fbadc0a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.813+0000 7fbaf7200700 1 --2- 172.21.15.78:0/3873334235 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbadc0a0d10 0x7fbadc0a31d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0009fd0 tx=0x7fbaf0009920 comp rx=0 tx=0).stop 2024-03-14T09:54:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.813+0000 7fbaf7200700 1 -- 172.21.15.78:0/3873334235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbaf8100250 msgr2=0x7fbaf8199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.813+0000 7fbaf7200700 1 --2- 172.21.15.78:0/3873334235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbaf8100250 0x7fbaf8199b40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbae40098e0 tx=0x7fbae4005740 comp rx=0 tx=0).stop 2024-03-14T09:54:36.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.814+0000 7fbaf7200700 1 -- 172.21.15.78:0/3873334235 shutdown_connections 2024-03-14T09:54:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.814+0000 7fbaf7200700 1 --2- 172.21.15.78:0/3873334235 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbadc0a0d10 0x7fbadc0a31d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.814+0000 7fbaf7200700 1 --2- 172.21.15.78:0/3873334235 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbaf810d450 0x7fbaf819dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.814+0000 7fbaf7200700 1 --2- 172.21.15.78:0/3873334235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbaf80ff8a0 0x7fbaf8199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.814+0000 7fbaf7200700 1 --2- 172.21.15.78:0/3873334235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbaf8100250 0x7fbaf8199b40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:36.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.814+0000 7fbaf7200700 1 -- 172.21.15.78:0/3873334235 >> 172.21.15.78:0/3873334235 conn(0x7fbaf80fb4e0 msgr2=0x7fbaf80fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:36.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.817+0000 7fbaf7200700 1 -- 172.21.15.78:0/3873334235 shutdown_connections 2024-03-14T09:54:36.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:36.817+0000 7fbaf7200700 1 -- 172.21.15.78:0/3873334235 wait complete. 2024-03-14T09:54:36.842 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:54:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.056+0000 7f21e8e00700 1 -- 172.21.15.78:0/3724505133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e41018d0 msgr2=0x7f21e4105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.056+0000 7f21e8e00700 1 --2- 172.21.15.78:0/3724505133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e41018d0 0x7f21e4105910 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009990 tx=0x7f21d0009ca0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.057+0000 7f21e8e00700 1 -- 172.21.15.78:0/3724505133 shutdown_connections 2024-03-14T09:54:37.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.057+0000 7f21e8e00700 1 --2- 172.21.15.78:0/3724505133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21e4106040 0x7f21e410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.057+0000 7f21e8e00700 1 --2- 172.21.15.78:0/3724505133 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21e4100f20 0x7f21e4101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.057+0000 7f21e8e00700 1 --2- 172.21.15.78:0/3724505133 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e41018d0 0x7f21e4105910 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.057+0000 7f21e8e00700 1 -- 172.21.15.78:0/3724505133 >> 172.21.15.78:0/3724505133 conn(0x7f21e40fc780 msgr2=0x7f21e40feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:37.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.057+0000 7f21e8e00700 1 -- 172.21.15.78:0/3724505133 shutdown_connections 2024-03-14T09:54:37.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.057+0000 7f21e8e00700 1 -- 172.21.15.78:0/3724505133 wait complete. 2024-03-14T09:54:37.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.058+0000 7f21e8e00700 1 Processor -- start 2024-03-14T09:54:37.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.058+0000 7f21e8e00700 1 -- start start 2024-03-14T09:54:37.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.059+0000 7f21e8e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e4100f20 0x7f21e4107c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.059+0000 7f21e8e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21e4106040 0x7f21e4108180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.059+0000 7f21e8e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21e41086c0 0x7f21e41a9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.059+0000 7f21e8e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e410bde0 con 0x7f21e4106040 2024-03-14T09:54:37.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.059+0000 7f21e8e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e41aa2a0 con 0x7f21e4100f20 2024-03-14T09:54:37.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.059+0000 7f21e8e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e41aa430 con 0x7f21e41086c0 2024-03-14T09:54:37.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.059+0000 7f21e1400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21e4106040 0x7f21e4108180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.059+0000 7f21e1e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e4100f20 0x7f21e4107c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.059+0000 7f21e1400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21e4106040 0x7f21e4108180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37544/0 (socket says 172.21.15.78:37544) 2024-03-14T09:54:37.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.059+0000 7f21e1400700 1 -- 172.21.15.78:0/3802558135 learned_addr learned my addr 172.21.15.78:0/3802558135 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:37.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.060+0000 7f21e2800700 1 --2- 172.21.15.78:0/3802558135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21e41086c0 0x7f21e41a9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.060+0000 7f21e1e00700 1 -- 172.21.15.78:0/3802558135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21e41086c0 msgr2=0x7f21e41a9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:37.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.060+0000 7f21e1e00700 1 --2- 172.21.15.78:0/3802558135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21e41086c0 0x7f21e41a9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.060+0000 7f21e1e00700 1 -- 172.21.15.78:0/3802558135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21e4106040 msgr2=0x7f21e4108180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:37.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.060+0000 7f21e1e00700 1 --2- 172.21.15.78:0/3802558135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21e4106040 0x7f21e4108180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.060+0000 7f21e1e00700 1 -- 172.21.15.78:0/3802558135 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d00095f0 con 0x7f21e4100f20 2024-03-14T09:54:37.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.060+0000 7f21e2800700 1 --2- 172.21.15.78:0/3802558135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21e41086c0 0x7f21e41a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:54:37.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.060+0000 7f21e1e00700 1 --2- 172.21.15.78:0/3802558135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e4100f20 0x7f21e4107c40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f21d400e9f0 tx=0x7f21d400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:37.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.060+0000 7f21daa00700 1 -- 172.21.15.78:0/3802558135 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d400eee0 con 0x7f21e4100f20 2024-03-14T09:54:37.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.060+0000 7f21daa00700 1 -- 172.21.15.78:0/3802558135 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f21d400cbd0 con 0x7f21e4100f20 2024-03-14T09:54:37.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.060+0000 7f21e8e00700 1 -- 172.21.15.78:0/3802558135 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e41aa710 con 0x7f21e4100f20 2024-03-14T09:54:37.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.061+0000 7f21e8e00700 1 -- 172.21.15.78:0/3802558135 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e41aac70 con 0x7f21e4100f20 2024-03-14T09:54:37.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.061+0000 7f21daa00700 1 -- 172.21.15.78:0/3802558135 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d4018680 con 0x7f21e4100f20 2024-03-14T09:54:37.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.065+0000 7f21daa00700 1 -- 172.21.15.78:0/3802558135 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 25) v1 ==== 135564+0+0 (secure 0 0 0) 0x7f21d40187e0 con 0x7f21e4100f20 2024-03-14T09:54:37.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.066+0000 7f21fc8c2b80 1 Processor -- start 2024-03-14T09:54:37.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.066+0000 7f21fc8c2b80 1 -- start start 2024-03-14T09:54:37.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.067+0000 7f21fc8c2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada7a680 0x5616ada7aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.067+0000 7f21fc8c2b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 0x5616ada95990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.067+0000 7f21fc8c2b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada95ed0 0x5616ada982c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.067+0000 7f21fc8c2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ad9d3990 con 0x5616ada7a680 2024-03-14T09:54:37.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.067+0000 7f21fc8c2b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ad9d1350 con 0x5616ada95ed0 2024-03-14T09:54:37.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.067+0000 7f21fc8c2b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ad9ccc40 con 0x5616ada7afa0 2024-03-14T09:54:37.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.067+0000 7f21daa00700 1 --2- 172.21.15.78:0/3802558135 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f21c80a0d10 0x7f21c80a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.067+0000 7f21e1e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 0x5616ada95990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.067+0000 7f21e1e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 0x5616ada95990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59230/0 (socket says 172.21.15.78:59230) 2024-03-14T09:54:37.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.067+0000 7f21e1e00700 1 -- 172.21.15.78:0/49689556 learned_addr learned my addr 172.21.15.78:0/49689556 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:37.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.067+0000 7f21e1400700 1 --2- 172.21.15.78:0/49689556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada95ed0 0x5616ada982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.068+0000 7f21e2800700 1 --2- 172.21.15.78:0/3802558135 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f21c80a0d10 0x7f21c80a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.068+0000 7f21daa00700 1 -- 172.21.15.78:0/3802558135 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f21d4014070 con 0x7f21e4100f20 2024-03-14T09:54:37.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.068+0000 7f21e1400700 1 -- 172.21.15.78:0/49689556 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 msgr2=0x5616ada95990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:37.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.068+0000 7f21e1400700 1 --2- 172.21.15.78:0/49689556 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 0x5616ada95990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.068+0000 7f21e1400700 1 -- 172.21.15.78:0/49689556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada7a680 msgr2=0x5616ada7aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:37.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.068+0000 7f21e1400700 1 --2- 172.21.15.78:0/49689556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada7a680 0x5616ada7aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.068+0000 7f21e1400700 1 -- 172.21.15.78:0/49689556 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616ad84e3b0 con 0x5616ada95ed0 2024-03-14T09:54:37.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.068+0000 7f21e1400700 1 --2- 172.21.15.78:0/49689556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada95ed0 0x5616ada982c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f21d0006010 tx=0x7f21d000b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4482f95d5377f8c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:37.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.068+0000 7f21cf400700 1 -- 172.21.15.78:0/49689556 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d001d070 con 0x5616ada95ed0 2024-03-14T09:54:37.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.069+0000 7f21e2800700 1 --2- 172.21.15.78:0/3802558135 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f21c80a0d10 0x7f21c80a31d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0105b0 tx=0x7f21dc009920 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:37.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.069+0000 7f21cf400700 1 -- 172.21.15.78:0/49689556 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f21d001e040 con 0x5616ada95ed0 2024-03-14T09:54:37.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.069+0000 7f21cf400700 1 -- 172.21.15.78:0/49689556 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0004880 con 0x5616ada95ed0 2024-03-14T09:54:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.069+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/49689556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada95ed0 msgr2=0x5616ada982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.069+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/49689556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada95ed0 0x5616ada982c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f21d0006010 tx=0x7f21d000b610 comp rx=0 tx=0).stop 2024-03-14T09:54:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.070+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/49689556 shutdown_connections 2024-03-14T09:54:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.070+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/49689556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada95ed0 0x5616ada982c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.070+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/49689556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada7a680 0x5616ada7aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.070+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/49689556 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 0x5616ada95990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.070+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/49689556 >> 172.21.15.78:0/49689556 conn(0x5616ad9f0220 msgr2=0x5616ada75aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.070+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/49689556 shutdown_connections 2024-03-14T09:54:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.070+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/49689556 wait complete. 2024-03-14T09:54:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.070+0000 7f21fc8c2b80 1 -- start start 2024-03-14T09:54:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.070+0000 7f21fc8c2b80 1 Processor -- start 2024-03-14T09:54:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.071+0000 7f21fc8c2b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada7a680 0x5616ada86af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.071+0000 7f21fc8c2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada7afa0 0x5616ada87030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.071+0000 7f21fc8c2b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada95ed0 0x5616ada81ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.071+0000 7f21fc8c2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ad968d80 con 0x5616ada7afa0 2024-03-14T09:54:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.071+0000 7f21fc8c2b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ad968a20 con 0x5616ada7a680 2024-03-14T09:54:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.071+0000 7f21fc8c2b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ad8506a0 con 0x5616ada95ed0 2024-03-14T09:54:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.071+0000 7f21fc8c2b80 10 client.0 _ll_register_callbacks cb 0x5616ad7d43a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.071+0000 7f21e1400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada7afa0 0x5616ada87030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.072+0000 7f21e1e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada7a680 0x5616ada86af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.072+0000 7f21e1e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada7a680 0x5616ada86af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57540/0 (socket says 172.21.15.78:57540) 2024-03-14T09:54:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.072+0000 7f21e1e00700 1 -- 172.21.15.78:0/96300900 learned_addr learned my addr 172.21.15.78:0/96300900 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.072+0000 7f21e2800700 1 --2- 172.21.15.78:0/96300900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada95ed0 0x5616ada81ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.072+0000 7f21e1e00700 1 -- 172.21.15.78:0/96300900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada95ed0 msgr2=0x5616ada81ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.072+0000 7f21e1e00700 1 --2- 172.21.15.78:0/96300900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada95ed0 0x5616ada81ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.072+0000 7f21e1e00700 1 -- 172.21.15.78:0/96300900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada7afa0 msgr2=0x5616ada87030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.072+0000 7f21e1e00700 1 --2- 172.21.15.78:0/96300900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada7afa0 0x5616ada87030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.072+0000 7f21e1e00700 1 -- 172.21.15.78:0/96300900 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e410e430 con 0x5616ada7a680 2024-03-14T09:54:37.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.072+0000 7f21e1400700 1 --2- 172.21.15.78:0/96300900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada7afa0 0x5616ada87030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:54:37.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.073+0000 7f21e1e00700 1 --2- 172.21.15.78:0/96300900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada7a680 0x5616ada86af0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f21d400c5b0 tx=0x7f21d4099050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=440820aad311fd2a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:37.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.073+0000 7f21c3e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:54:37.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.073+0000 7f21c3e00700 1 -- 172.21.15.78:0/96300900 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d40f3040 con 0x5616ada7a680 2024-03-14T09:54:37.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.073+0000 7f21c3e00700 1 -- 172.21.15.78:0/96300900 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f21d4099f80 con 0x5616ada7a680 2024-03-14T09:54:37.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.073+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/96300900 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5616ada6d720 con 0x5616ada7a680 2024-03-14T09:54:37.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.073+0000 7f21c3e00700 1 -- 172.21.15.78:0/96300900 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21d409d6b0 con 0x5616ada7a680 2024-03-14T09:54:37.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.073+0000 7f21fc8c2b80 10 client.25805 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:37.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.073+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/96300900 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5616ad7fab60 con 0x5616ada7a680 2024-03-14T09:54:37.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.073+0000 7f21c3e00700 1 -- 172.21.15.78:0/96300900 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5616ada6d720 con 0x5616ada7a680 2024-03-14T09:54:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.074+0000 7f21c3e00700 1 -- 172.21.15.78:0/96300900 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f21d40c9fe0 con 0x5616ada7a680 2024-03-14T09:54:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.074+0000 7f21fc8c2b80 10 client.25805 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.074+0000 7f21fc8c2b80 10 client.25805 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.074+0000 7f21fc8c2b80 20 client.25805 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.075+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/96300900 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5616ada92080 0x5616ada944a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.075+0000 7f21fc8c2b80 4 client.25805 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:54:37.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.075+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/96300900 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616ad9c5700 con 0x5616ada92080 2024-03-14T09:54:37.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.075+0000 7f21e1400700 1 --2- 172.21.15.78:0/96300900 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5616ada92080 0x5616ada944a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.076+0000 7f21e1400700 1 --2- 172.21.15.78:0/96300900 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5616ada92080 0x5616ada944a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19e80a01ba3246ce server_cookie=c777c19cf0b4ab7a in_seq=0 out_seq=0 2024-03-14T09:54:37.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.076+0000 7f21c3e00700 10 client.25805 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:37.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.076+0000 7f21c3e00700 1 -- 172.21.15.78:0/96300900 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5616ad9c5700 con 0x5616ada92080 2024-03-14T09:54:37.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.076+0000 7f21c3e00700 10 client.25805 handle_command_reply: tid=0 2024-03-14T09:54:37.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.076+0000 7f21fc8c2b80 1 client.25805 shutdown 2024-03-14T09:54:37.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.078+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/96300900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada7a680 msgr2=0x5616ada86af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:37.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.078+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/96300900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada7a680 0x5616ada86af0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f21d400c5b0 tx=0x7f21d4099050 comp rx=0 tx=0).stop 2024-03-14T09:54:37.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.078+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/96300900 shutdown_connections 2024-03-14T09:54:37.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.078+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/96300900 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5616ada92080 0x5616ada944a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.078+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/96300900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada7a680 0x5616ada86af0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.078+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/96300900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada95ed0 0x5616ada81ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.078+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/96300900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada7afa0 0x5616ada87030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.078+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/96300900 >> 172.21.15.78:0/96300900 conn(0x5616ad9f0220 msgr2=0x5616ada75690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:37.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.078+0000 7f21c3e00700 0 client.25805 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:37.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.078+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/96300900 shutdown_connections 2024-03-14T09:54:37.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.079+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/96300900 wait complete. 2024-03-14T09:54:37.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.079+0000 7f21fc8c2b80 20 client.25805 trim_cache size 0 max 16384 2024-03-14T09:54:37.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.096+0000 7f21fc8c2b80 1 Processor -- start 2024-03-14T09:54:37.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.096+0000 7f21fc8c2b80 1 -- start start 2024-03-14T09:54:37.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.096+0000 7f21fc8c2b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada7a680 0x5616ada807c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.096+0000 7f21fc8c2b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 0x5616ada80d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.096+0000 7f21fc8c2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada95ed0 0x5616ada866f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.096+0000 7f21fc8c2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ada7baf0 con 0x5616ada95ed0 2024-03-14T09:54:37.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.096+0000 7f21fc8c2b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ada81240 con 0x5616ada7a680 2024-03-14T09:54:37.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.096+0000 7f21fc8c2b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ada77060 con 0x5616ada7afa0 2024-03-14T09:54:37.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.097+0000 7f21e1e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada7a680 0x5616ada807c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.097+0000 7f21e1e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada7a680 0x5616ada807c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57542/0 (socket says 172.21.15.78:57542) 2024-03-14T09:54:37.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.097+0000 7f21e1e00700 1 -- 172.21.15.78:0/130912818 learned_addr learned my addr 172.21.15.78:0/130912818 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:37.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.097+0000 7f21e2800700 1 --2- 172.21.15.78:0/130912818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada95ed0 0x5616ada866f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.097+0000 7f21e1400700 1 --2- 172.21.15.78:0/130912818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 0x5616ada80d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.097+0000 7f21e1e00700 1 -- 172.21.15.78:0/130912818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 msgr2=0x5616ada80d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:37.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.097+0000 7f21e1e00700 1 --2- 172.21.15.78:0/130912818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 0x5616ada80d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.097+0000 7f21e1e00700 1 -- 172.21.15.78:0/130912818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada95ed0 msgr2=0x5616ada866f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:37.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.097+0000 7f21e1e00700 1 --2- 172.21.15.78:0/130912818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada95ed0 0x5616ada866f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.097+0000 7f21e1e00700 1 -- 172.21.15.78:0/130912818 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616ad992ea0 con 0x5616ada7a680 2024-03-14T09:54:37.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.097+0000 7f21e1400700 1 --2- 172.21.15.78:0/130912818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 0x5616ada80d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:54:37.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.098+0000 7f21e1e00700 1 --2- 172.21.15.78:0/130912818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada7a680 0x5616ada807c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f21d40a17d0 tx=0x7f21d400e980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=431cd79128a6ce5a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:37.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.098+0000 7f21cfe00700 1 -- 172.21.15.78:0/130912818 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d40f3070 con 0x5616ada7a680 2024-03-14T09:54:37.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.098+0000 7f21cfe00700 1 -- 172.21.15.78:0/130912818 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f21d409d6b0 con 0x5616ada7a680 2024-03-14T09:54:37.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.098+0000 7f21cfe00700 1 -- 172.21.15.78:0/130912818 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d40b90b0 con 0x5616ada7a680 2024-03-14T09:54:37.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.098+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/130912818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada7a680 msgr2=0x5616ada807c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:37.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.099+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/130912818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada7a680 0x5616ada807c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f21d40a17d0 tx=0x7f21d400e980 comp rx=0 tx=0).stop 2024-03-14T09:54:37.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.099+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/130912818 shutdown_connections 2024-03-14T09:54:37.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.099+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/130912818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada7a680 0x5616ada807c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.099+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/130912818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada95ed0 0x5616ada866f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.099+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/130912818 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 0x5616ada80d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.099+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/130912818 >> 172.21.15.78:0/130912818 conn(0x5616ad9f0220 msgr2=0x5616ada7d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:37.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.099+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/130912818 shutdown_connections 2024-03-14T09:54:37.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.099+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/130912818 wait complete. 2024-03-14T09:54:37.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.099+0000 7f21fc8c2b80 1 -- start start 2024-03-14T09:54:37.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.100+0000 7f21fc8c2b80 1 Processor -- start 2024-03-14T09:54:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.100+0000 7f21fc8c2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada7a680 0x5616ada7ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.100+0000 7f21fc8c2b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 0x5616ada7f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.100+0000 7f21fc8c2b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada95ed0 0x5616adae5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.100+0000 7f21fc8c2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ada7faa0 con 0x5616ada7a680 2024-03-14T09:54:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.100+0000 7f21fc8c2b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616adae5b50 con 0x5616ada95ed0 2024-03-14T09:54:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.100+0000 7f21fc8c2b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616adae5cc0 con 0x5616ada7afa0 2024-03-14T09:54:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.100+0000 7f21fc8c2b80 10 client.0 _ll_register_callbacks cb 0x5616ad7d43a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:54:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.101+0000 7f21e1e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada7a680 0x5616ada7ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.101+0000 7f21e1400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 0x5616ada7f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.101+0000 7f21e1400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 0x5616ada7f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59270/0 (socket says 172.21.15.78:59270) 2024-03-14T09:54:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.101+0000 7f21e1400700 1 -- 172.21.15.78:0/2648539376 learned_addr learned my addr 172.21.15.78:0/2648539376 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:54:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.101+0000 7f21e2800700 1 --2- 172.21.15.78:0/2648539376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada95ed0 0x5616adae5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.101+0000 7f21e1400700 1 -- 172.21.15.78:0/2648539376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada95ed0 msgr2=0x5616adae5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:37.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.101+0000 7f21e1400700 1 --2- 172.21.15.78:0/2648539376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada95ed0 0x5616adae5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.101+0000 7f21e1400700 1 -- 172.21.15.78:0/2648539376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada7a680 msgr2=0x5616ada7ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:54:37.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.101+0000 7f21e1400700 1 --2- 172.21.15.78:0/2648539376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada7a680 0x5616ada7ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.101+0000 7f21e1400700 1 -- 172.21.15.78:0/2648539376 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5616ad7fab60 con 0x5616ada7afa0 2024-03-14T09:54:37.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.102+0000 7f21e1400700 1 --2- 172.21.15.78:0/2648539376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 0x5616ada7f390 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f21d00059d0 tx=0x7f21d0009290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7966c6d8af784a80 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:54:37.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.102+0000 7f21c3e00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:54:37.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.102+0000 7f21c3e00700 1 -- 172.21.15.78:0/2648539376 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0032040 con 0x5616ada7afa0 2024-03-14T09:54:37.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.102+0000 7f21c3e00700 1 -- 172.21.15.78:0/2648539376 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f21d000b950 con 0x5616ada7afa0 2024-03-14T09:54:37.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.102+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/2648539376 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f21d001e040 con 0x5616ada7afa0 2024-03-14T09:54:37.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.102+0000 7f21c3e00700 1 -- 172.21.15.78:0/2648539376 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21d001d040 con 0x5616ada7afa0 2024-03-14T09:54:37.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.102+0000 7f21fc8c2b80 10 client.25822 fetch_fsmap learned FSMap version 25 2024-03-14T09:54:37.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.102+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/2648539376 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5616ad852cd0 con 0x5616ada7afa0 2024-03-14T09:54:37.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.103+0000 7f21c3e00700 1 -- 172.21.15.78:0/2648539376 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f21d001e040 con 0x5616ada7afa0 2024-03-14T09:54:37.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.103+0000 7f21c3e00700 1 -- 172.21.15.78:0/2648539376 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f21d0035050 con 0x5616ada7afa0 2024-03-14T09:54:37.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.103+0000 7f21fc8c2b80 10 client.25822 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:54:37.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.103+0000 7f21fc8c2b80 10 client.25822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:54:37.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.103+0000 7f21fc8c2b80 20 client.25822 populate_metadata read hostname 'smithi078' 2024-03-14T09:54:37.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.103+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/2648539376 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5616adae7060 0x5616adae9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:54:37.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.103+0000 7f21fc8c2b80 4 client.25822 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:54:37.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.103+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/2648539376 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5616ad81bcf0 con 0x5616adae7060 2024-03-14T09:54:37.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.103+0000 7f21e1e00700 1 --2- 172.21.15.78:0/2648539376 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5616adae7060 0x5616adae9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:54:37.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.104+0000 7f21c3e00700 10 client.25822 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:37.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.104+0000 7f21e1e00700 1 --2- 172.21.15.78:0/2648539376 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5616adae7060 0x5616adae9480 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae3f800adce3641b server_cookie=c22061deaf76cc5 in_seq=0 out_seq=0 2024-03-14T09:54:37.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.104+0000 7f21c3e00700 1 -- 172.21.15.78:0/2648539376 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5616ad81bcf0 con 0x5616adae7060 2024-03-14T09:54:37.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.104+0000 7f21c3e00700 10 client.25822 handle_command_reply: tid=0 2024-03-14T09:54:37.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.104+0000 7f21fc8c2b80 1 client.25822 shutdown 2024-03-14T09:54:37.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.104+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/2648539376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 msgr2=0x5616ada7f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:37.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.104+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/2648539376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 0x5616ada7f390 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f21d00059d0 tx=0x7f21d0009290 comp rx=0 tx=0).stop 2024-03-14T09:54:37.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.104+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/2648539376 shutdown_connections 2024-03-14T09:54:37.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.104+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/2648539376 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5616adae7060 0x5616adae9480 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.104+0000 7f21c3e00700 0 client.25822 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:54:37.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.104+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/2648539376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5616ada95ed0 0x5616adae5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.104+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/2648539376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5616ada7a680 0x5616ada7ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.104+0000 7f21fc8c2b80 1 --2- 172.21.15.78:0/2648539376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5616ada7afa0 0x5616ada7f390 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.104+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/2648539376 >> 172.21.15.78:0/2648539376 conn(0x5616ad9f0220 msgr2=0x5616ada7d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:37.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.104+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/2648539376 shutdown_connections 2024-03-14T09:54:37.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.104+0000 7f21fc8c2b80 1 -- 172.21.15.78:0/2648539376 wait complete. 2024-03-14T09:54:37.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.105+0000 7f21fc8c2b80 20 client.25822 trim_cache size 0 max 16384 2024-03-14T09:54:37.107 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:54:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-14T09:54:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T09:54:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: "1f99ba69-06c5-492b-8192-8662ab61eb88": { 2024-03-14T09:54:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T09:54:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "1f99ba69-06c5-492b-8192-8662ab61eb88", 2024-03-14T09:54:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T09:54:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:54:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:54:37.108 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:54:37.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.105+0000 7f21e8e00700 1 -- 172.21.15.78:0/3802558135 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f21c80a0d10 msgr2=0x7f21c80a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:37.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.105+0000 7f21e8e00700 1 --2- 172.21.15.78:0/3802558135 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f21c80a0d10 0x7f21c80a31d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21dc0105b0 tx=0x7f21dc009920 comp rx=0 tx=0).stop 2024-03-14T09:54:37.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.105+0000 7f21e8e00700 1 -- 172.21.15.78:0/3802558135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e4100f20 msgr2=0x7f21e4107c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:54:37.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.105+0000 7f21e8e00700 1 --2- 172.21.15.78:0/3802558135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e4100f20 0x7f21e4107c40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f21d400e9f0 tx=0x7f21d400ed00 comp rx=0 tx=0).stop 2024-03-14T09:54:37.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.105+0000 7f21e8e00700 1 -- 172.21.15.78:0/3802558135 shutdown_connections 2024-03-14T09:54:37.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.105+0000 7f21e8e00700 1 --2- 172.21.15.78:0/3802558135 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f21c80a0d10 0x7f21c80a31d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.105+0000 7f21e8e00700 1 --2- 172.21.15.78:0/3802558135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e4100f20 0x7f21e4107c40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.105+0000 7f21e8e00700 1 --2- 172.21.15.78:0/3802558135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f21e41086c0 0x7f21e41a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.105+0000 7f21e8e00700 1 --2- 172.21.15.78:0/3802558135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21e4106040 0x7f21e4108180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:54:37.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.105+0000 7f21e8e00700 1 -- 172.21.15.78:0/3802558135 >> 172.21.15.78:0/3802558135 conn(0x7f21e40fc780 msgr2=0x7f21e40feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:54:37.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.106+0000 7f21e8e00700 1 -- 172.21.15.78:0/3802558135 shutdown_connections 2024-03-14T09:54:37.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:54:37.106+0000 7f21e8e00700 1 -- 172.21.15.78:0/3802558135 wait complete. 2024-03-14T09:54:37.123 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:37 smithi078 ceph-mon[38143]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-14T09:54:37.125 INFO:tasks.cephfs.filesystem:scrub status for tag:1f99ba69-06c5-492b-8192-8662ab61eb88 - {'path': '/', 'tag': '1f99ba69-06c5-492b-8192-8662ab61eb88', 'options': 'recursive,force'} 2024-03-14T09:54:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:37 smithi152 ceph-mon[48279]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-14T09:54:37.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:37 smithi132 ceph-mon[47959]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-14T09:54:38.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:38 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T09:54:38.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:38 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:38.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:38 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T09:54:38.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:38 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:38.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:38 smithi152 ceph-mon[48279]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T09:54:39.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:38 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T09:54:39.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:38 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:39.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:38 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T09:54:39.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:38 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:39.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:38 smithi132 ceph-mon[47959]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T09:54:39.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:38 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T09:54:39.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:38 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:39.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:38 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T09:54:39.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:38 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:54:39.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:38 smithi078 ceph-mon[38143]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T09:54:39.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:39 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T09:54:40.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:39 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T09:54:40.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:39 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T09:54:41.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:40 smithi132 ceph-mon[47959]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T09:54:41.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:40 smithi078 ceph-mon[38143]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T09:54:41.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:40 smithi152 ceph-mon[48279]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T09:54:43.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:43 smithi132 ceph-mon[47959]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-03-14T09:54:43.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:43 smithi078 ceph-mon[38143]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-03-14T09:54:43.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:43 smithi152 ceph-mon[48279]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-03-14T09:54:45.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:45 smithi132 ceph-mon[47959]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-14T09:54:45.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:45 smithi078 ceph-mon[38143]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-14T09:54:45.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:45 smithi152 ceph-mon[48279]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-14T09:54:47.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:47 smithi132 ceph-mon[47959]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-14T09:54:47.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:47 smithi078 ceph-mon[38143]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-14T09:54:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:47 smithi152 ceph-mon[48279]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-14T09:54:48.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:48 smithi152 ceph-mon[48279]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-14T09:54:49.019 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:48 smithi132 ceph-mon[47959]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-14T09:54:49.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:48 smithi078 ceph-mon[38143]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-14T09:54:51.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:51 smithi132 ceph-mon[47959]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T09:54:51.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:51 smithi078 ceph-mon[38143]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T09:54:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:51 smithi152 ceph-mon[48279]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T09:54:53.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:53 smithi132 ceph-mon[47959]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-14T09:54:53.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:53 smithi078 ceph-mon[38143]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-14T09:54:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:53 smithi152 ceph-mon[48279]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-14T09:54:55.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:55 smithi132 ceph-mon[47959]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-14T09:54:55.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:55 smithi078 ceph-mon[38143]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-14T09:54:55.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:55 smithi152 ceph-mon[48279]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-14T09:54:57.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:57 smithi078 ceph-mon[38143]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-14T09:54:57.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:57 smithi152 ceph-mon[48279]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-14T09:54:57.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:57 smithi132 ceph-mon[47959]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-14T09:54:58.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:54:58 smithi152 ceph-mon[48279]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-14T09:54:59.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:54:58 smithi132 ceph-mon[47959]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-14T09:54:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:54:58 smithi078 ceph-mon[38143]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-14T09:55:01.013 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:55:00 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:55:00.614+0000 7fd8ab400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:55:01.013 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:55:00 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:55:00.614+0000 7fd8ab400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:55:01.013 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:55:00 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:55:00.614+0000 7fd8ab400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:55:01.013 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:55:00 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:55:00.614+0000 7fd8ab400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:55:01.013 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 09:55:00 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T09:55:00.614+0000 7fd8ab400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-14T09:55:01.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:01 smithi132 ceph-mon[47959]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-14T09:55:01.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:01 smithi078 ceph-mon[38143]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-14T09:55:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:01 smithi152 ceph-mon[48279]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-14T09:55:03.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:03 smithi078 ceph-mon[38143]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-14T09:55:03.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:03 smithi078 ceph-mon[38143]: mgrmap e26: x(active, since 52s), standbys: z, y 2024-03-14T09:55:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:03 smithi152 ceph-mon[48279]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-14T09:55:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:03 smithi152 ceph-mon[48279]: mgrmap e26: x(active, since 52s), standbys: z, y 2024-03-14T09:55:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:03 smithi132 ceph-mon[47959]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-14T09:55:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:03 smithi132 ceph-mon[47959]: mgrmap e26: x(active, since 52s), standbys: z, y 2024-03-14T09:55:05.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:05 smithi078 ceph-mon[38143]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-14T09:55:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:05 smithi152 ceph-mon[48279]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-14T09:55:05.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:05 smithi132 ceph-mon[47959]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-14T09:55:05.640 INFO:tasks.workunit.client.0.smithi078.stderr:Updating files: 67% (8110/12088) Updating files: 68% (8220/12088) Updating files: 69% (8341/12088) Updating files: 70% (8462/12088) Updating files: 71% (8583/12088) Updating files: 72% (8704/12088) Updating files: 73% (8825/12088) Updating files: 74% (8946/12088) Updating files: 75% (9066/12088) Updating files: 76% (9187/12088) Updating files: 77% (9308/12088) Updating files: 78% (9429/12088) Updating files: 79% (9550/12088) Updating files: 80% (9671/12088) Updating files: 81% (9792/12088) Updating files: 82% (9913/12088) Updating files: 83% (10034/12088) Updating files: 84% (10154/12088) Updating files: 85% (10275/12088) Updating files: 86% (10396/12088) Updating files: 87% (10517/12088) Updating files: 88% (10638/12088) Updating files: 89% (10759/12088) Updating files: 90% (10880/12088) Updating files: 91% (11001/12088) Updating files: 92% (11121/12088) Updating files: 93% (11242/12088) Updating files: 94% (11363/12088) Updating files: 95% (11484/12088) Updating files: 96% (11605/12088) Updating files: 97% (11726/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-03-14T09:55:06.530 INFO:tasks.workunit.client.0.smithi078.stderr:Note: switching to '2f465319e3c83d48f999325459bcdfd0346449c0'. 2024-03-14T09:55:06.530 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-14T09:55:06.530 INFO:tasks.workunit.client.0.smithi078.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-14T09:55:06.530 INFO:tasks.workunit.client.0.smithi078.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-14T09:55:06.530 INFO:tasks.workunit.client.0.smithi078.stderr:state without impacting any branches by switching back to a branch. 2024-03-14T09:55:06.530 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-14T09:55:06.530 INFO:tasks.workunit.client.0.smithi078.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-14T09:55:06.530 INFO:tasks.workunit.client.0.smithi078.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-14T09:55:06.530 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-14T09:55:06.531 INFO:tasks.workunit.client.0.smithi078.stderr: git switch -c 2024-03-14T09:55:06.531 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-14T09:55:06.531 INFO:tasks.workunit.client.0.smithi078.stderr:Or undo this operation with: 2024-03-14T09:55:06.531 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-14T09:55:06.531 INFO:tasks.workunit.client.0.smithi078.stderr: git switch - 2024-03-14T09:55:06.531 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-14T09:55:06.531 INFO:tasks.workunit.client.0.smithi078.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-14T09:55:06.531 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-14T09:55:06.531 INFO:tasks.workunit.client.0.smithi078.stderr:HEAD is now at 2f465319e3c Merge branch 'wip-64737-reef' of https://github.com/xhernandez/ceph into wip-yuri-testing-2024-03-12-1240-reef 2024-03-14T09:55:06.540 DEBUG:teuthology.orchestra.run.smithi078:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-14T09:55:06.589 INFO:tasks.workunit.client.0.smithi078.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-14T09:55:06.593 INFO:tasks.workunit.client.0.smithi078.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-14T09:55:06.594 INFO:tasks.workunit.client.0.smithi078.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-14T09:55:07.125 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:55:07.169 INFO:tasks.workunit.client.0.smithi078.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-14T09:55:07.251 INFO:tasks.workunit.client.0.smithi078.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-14T09:55:07.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:07 smithi078 ceph-mon[38143]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-14T09:55:07.315 INFO:tasks.workunit.client.0.smithi078.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-14T09:55:07.319 INFO:tasks.workunit.client.0.smithi078.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-14T09:55:07.319 INFO:tasks.workunit.client.0.smithi078.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-14T09:55:07.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.328+0000 7fec62000700 1 -- 172.21.15.78:0/1748130190 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec5c073b00 msgr2=0x7fec5c10cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:07.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.328+0000 7fec62000700 1 --2- 172.21.15.78:0/1748130190 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec5c073b00 0x7fec5c10cfc0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fec480098f0 tx=0x7fec48009c00 comp rx=0 tx=0).stop 2024-03-14T09:55:07.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.328+0000 7fec62000700 1 -- 172.21.15.78:0/1748130190 shutdown_connections 2024-03-14T09:55:07.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.328+0000 7fec62000700 1 --2- 172.21.15.78:0/1748130190 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec5c0731e0 0x7fec5c0735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.328+0000 7fec62000700 1 --2- 172.21.15.78:0/1748130190 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec5c10d500 0x7fec5c10f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.328+0000 7fec62000700 1 --2- 172.21.15.78:0/1748130190 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec5c073b00 0x7fec5c10cfc0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.328+0000 7fec62000700 1 -- 172.21.15.78:0/1748130190 >> 172.21.15.78:0/1748130190 conn(0x7fec5c0fc7b0 msgr2=0x7fec5c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:07.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.328+0000 7fec62000700 1 -- 172.21.15.78:0/1748130190 shutdown_connections 2024-03-14T09:55:07.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.328+0000 7fec62000700 1 -- 172.21.15.78:0/1748130190 wait complete. 2024-03-14T09:55:07.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.329+0000 7fec62000700 1 Processor -- start 2024-03-14T09:55:07.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.329+0000 7fec62000700 1 -- start start 2024-03-14T09:55:07.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec62000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec5c0731e0 0x7fec5c19f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec62000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec5c073b00 0x7fec5c19fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec62000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec5c10d500 0x7fec5c199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec62000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c199db0 con 0x7fec5c073b00 2024-03-14T09:55:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec62000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c199f20 con 0x7fec5c10d500 2024-03-14T09:55:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec62000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c19a0e0 con 0x7fec5c0731e0 2024-03-14T09:55:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec5aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec5c073b00 0x7fec5c19fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec5b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec5c0731e0 0x7fec5c19f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec5b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec5c0731e0 0x7fec5c19f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:52976/0 (socket says 172.21.15.78:52976) 2024-03-14T09:55:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec5aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec5c073b00 0x7fec5c19fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44568/0 (socket says 172.21.15.78:44568) 2024-03-14T09:55:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec5b400700 1 -- 172.21.15.78:0/1646061390 learned_addr learned my addr 172.21.15.78:0/1646061390 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec5be00700 1 --2- 172.21.15.78:0/1646061390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec5c10d500 0x7fec5c199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec5aa00700 1 -- 172.21.15.78:0/1646061390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec5c0731e0 msgr2=0x7fec5c19f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec5aa00700 1 --2- 172.21.15.78:0/1646061390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec5c0731e0 0x7fec5c19f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec5aa00700 1 -- 172.21.15.78:0/1646061390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec5c10d500 msgr2=0x7fec5c199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:07.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec5aa00700 1 --2- 172.21.15.78:0/1646061390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec5c10d500 0x7fec5c199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec5aa00700 1 -- 172.21.15.78:0/1646061390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec48009520 con 0x7fec5c073b00 2024-03-14T09:55:07.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.330+0000 7fec5b400700 1 --2- 172.21.15.78:0/1646061390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec5c0731e0 0x7fec5c19f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:07.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.331+0000 7fec5aa00700 1 --2- 172.21.15.78:0/1646061390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec5c073b00 0x7fec5c19fb60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fec48009fd0 tx=0x7fec4800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:07.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.331+0000 7fec53e00700 1 -- 172.21.15.78:0/1646061390 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4801c070 con 0x7fec5c073b00 2024-03-14T09:55:07.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.331+0000 7fec53e00700 1 -- 172.21.15.78:0/1646061390 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec4800baa0 con 0x7fec5c073b00 2024-03-14T09:55:07.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.331+0000 7fec62000700 1 -- 172.21.15.78:0/1646061390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5c19a360 con 0x7fec5c073b00 2024-03-14T09:55:07.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.331+0000 7fec53e00700 1 -- 172.21.15.78:0/1646061390 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec480038e0 con 0x7fec5c073b00 2024-03-14T09:55:07.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.331+0000 7fec62000700 1 -- 172.21.15.78:0/1646061390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5c1a6610 con 0x7fec5c073b00 2024-03-14T09:55:07.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.335+0000 7fec53e00700 1 -- 172.21.15.78:0/1646061390 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) v1 ==== 135621+0+0 (secure 0 0 0) 0x7fec48020030 con 0x7fec5c073b00 2024-03-14T09:55:07.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.336+0000 7fec75b31b80 1 Processor -- start 2024-03-14T09:55:07.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.337+0000 7fec75b31b80 1 -- start start 2024-03-14T09:55:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.337+0000 7fec75b31b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 0x558c260daa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.337+0000 7fec75b31b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.337+0000 7fec75b31b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260e4ec0 0x558c260e72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.337+0000 7fec75b31b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c26033990 con 0x558c260dafa0 2024-03-14T09:55:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.337+0000 7fec75b31b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c26031350 con 0x558c260e4ec0 2024-03-14T09:55:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.337+0000 7fec75b31b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c2602cc40 con 0x558c260da680 2024-03-14T09:55:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.337+0000 7fec5b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.337+0000 7fec5be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 0x558c260daa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.337+0000 7fec5be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 0x558c260daa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:52980/0 (socket says 172.21.15.78:52980) 2024-03-14T09:55:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.337+0000 7fec5be00700 1 -- 172.21.15.78:0/3036442716 learned_addr learned my addr 172.21.15.78:0/3036442716 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.337+0000 7fec5b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44578/0 (socket says 172.21.15.78:44578) 2024-03-14T09:55:07.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.337+0000 7fec5aa00700 1 --2- 172.21.15.78:0/3036442716 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260e4ec0 0x558c260e72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.338+0000 7fec5be00700 1 -- 172.21.15.78:0/3036442716 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260e4ec0 msgr2=0x558c260e72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:07.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.338+0000 7fec5be00700 1 --2- 172.21.15.78:0/3036442716 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260e4ec0 0x558c260e72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.338+0000 7fec5be00700 1 -- 172.21.15.78:0/3036442716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 msgr2=0x558c260e4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.338+0000 7fec5be00700 1 --2- 172.21.15.78:0/3036442716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.338+0000 7fec5be00700 1 -- 172.21.15.78:0/3036442716 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c25eae3b0 con 0x558c260da680 2024-03-14T09:55:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.338+0000 7fec5b400700 1 --2- 172.21.15.78:0/3036442716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.338+0000 7fec5be00700 1 --2- 172.21.15.78:0/3036442716 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 0x558c260daa60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fec54011680 tx=0x7fec54009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cb724ae44b6a785 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.338+0000 7fec53e00700 1 --2- 172.21.15.78:0/1646061390 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fec400a0da0 0x7fec400a3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.338+0000 7fec53e00700 1 -- 172.21.15.78:0/1646061390 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fec480cd6c0 con 0x7fec5c073b00 2024-03-14T09:55:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.338+0000 7fec5b400700 1 --2- 172.21.15.78:0/1646061390 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fec400a0da0 0x7fec400a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.338+0000 7fec47e00700 1 -- 172.21.15.78:0/3036442716 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5400e040 con 0x558c260da680 2024-03-14T09:55:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.338+0000 7fec47e00700 1 -- 172.21.15.78:0/3036442716 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec5400f040 con 0x558c260da680 2024-03-14T09:55:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.338+0000 7fec47e00700 1 -- 172.21.15.78:0/3036442716 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec54004740 con 0x558c260da680 2024-03-14T09:55:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.339+0000 7fec75b31b80 1 -- 172.21.15.78:0/3036442716 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 msgr2=0x558c260daa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.339+0000 7fec5b400700 1 --2- 172.21.15.78:0/1646061390 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fec400a0da0 0x7fec400a3260 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fec4c005950 tx=0x7fec4c009310 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.339+0000 7fec75b31b80 1 --2- 172.21.15.78:0/3036442716 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 0x558c260daa60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fec54011680 tx=0x7fec54009920 comp rx=0 tx=0).stop 2024-03-14T09:55:07.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.339+0000 7fec75b31b80 1 -- 172.21.15.78:0/3036442716 shutdown_connections 2024-03-14T09:55:07.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.339+0000 7fec75b31b80 1 --2- 172.21.15.78:0/3036442716 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260e4ec0 0x558c260e72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.339+0000 7fec75b31b80 1 --2- 172.21.15.78:0/3036442716 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 0x558c260daa60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.339+0000 7fec75b31b80 1 --2- 172.21.15.78:0/3036442716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.339+0000 7fec75b31b80 1 -- 172.21.15.78:0/3036442716 >> 172.21.15.78:0/3036442716 conn(0x558c26050220 msgr2=0x558c260d5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:07.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.340+0000 7fec75b31b80 1 -- 172.21.15.78:0/3036442716 shutdown_connections 2024-03-14T09:55:07.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.340+0000 7fec75b31b80 1 -- 172.21.15.78:0/3036442716 wait complete. 2024-03-14T09:55:07.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.340+0000 7fec75b31b80 1 -- start start 2024-03-14T09:55:07.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.341+0000 7fec75b31b80 1 Processor -- start 2024-03-14T09:55:07.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.341+0000 7fec75b31b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 0x558c260e4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.341+0000 7fec75b31b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260dafa0 0x558c260e1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.341+0000 7fec75b31b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260e4ec0 0x558c260e2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.341+0000 7fec75b31b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c25fc8d80 con 0x558c260e4ec0 2024-03-14T09:55:07.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.341+0000 7fec75b31b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c25fc8a20 con 0x558c260dafa0 2024-03-14T09:55:07.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.341+0000 7fec75b31b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c25eb06a0 con 0x558c260da680 2024-03-14T09:55:07.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.341+0000 7fec75b31b80 10 client.0 _ll_register_callbacks cb 0x558c25e343a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:07.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.342+0000 7fec5aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260dafa0 0x558c260e1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.342+0000 7fec5be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260e4ec0 0x558c260e2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.342+0000 7fec5aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260dafa0 0x558c260e1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60714/0 (socket says 172.21.15.78:60714) 2024-03-14T09:55:07.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.342+0000 7fec5aa00700 1 -- 172.21.15.78:0/1129195544 learned_addr learned my addr 172.21.15.78:0/1129195544 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:07.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.342+0000 7fec5b400700 1 --2- 172.21.15.78:0/1129195544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 0x558c260e4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.342+0000 7fec5be00700 1 -- 172.21.15.78:0/1129195544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 msgr2=0x558c260e4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:07.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.342+0000 7fec5be00700 1 --2- 172.21.15.78:0/1129195544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 0x558c260e4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.342+0000 7fec5be00700 1 -- 172.21.15.78:0/1129195544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260dafa0 msgr2=0x558c260e1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:07.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.342+0000 7fec5be00700 1 --2- 172.21.15.78:0/1129195544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260dafa0 0x558c260e1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.342+0000 7fec5be00700 1 -- 172.21.15.78:0/1129195544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec4c009d90 con 0x558c260e4ec0 2024-03-14T09:55:07.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.342+0000 7fec5be00700 1 --2- 172.21.15.78:0/1129195544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260e4ec0 0x558c260e2160 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fec54011650 tx=0x7fec54011f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fba6c7d51685972 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:07.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.342+0000 7fec44c00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:55:07.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.343+0000 7fec44c00700 1 -- 172.21.15.78:0/1129195544 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5400e070 con 0x558c260e4ec0 2024-03-14T09:55:07.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.343+0000 7fec44c00700 1 -- 172.21.15.78:0/1129195544 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fec54004590 con 0x558c260e4ec0 2024-03-14T09:55:07.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.343+0000 7fec75b31b80 1 -- 172.21.15.78:0/1129195544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558c260cd720 con 0x558c260e4ec0 2024-03-14T09:55:07.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.343+0000 7fec44c00700 1 -- 172.21.15.78:0/1129195544 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec54004790 con 0x558c260e4ec0 2024-03-14T09:55:07.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.343+0000 7fec75b31b80 10 client.15825 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:07.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.343+0000 7fec75b31b80 1 -- 172.21.15.78:0/1129195544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7fec54011330 con 0x558c260e4ec0 2024-03-14T09:55:07.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.343+0000 7fec44c00700 1 -- 172.21.15.78:0/1129195544 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558c260cd720 con 0x558c260e4ec0 2024-03-14T09:55:07.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.343+0000 7fec44c00700 1 -- 172.21.15.78:0/1129195544 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fec54027460 con 0x558c260e4ec0 2024-03-14T09:55:07.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.343+0000 7fec75b31b80 10 client.15825 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:07.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.343+0000 7fec75b31b80 10 client.15825 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:55:07.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.343+0000 7fec75b31b80 20 client.15825 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:07.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.343+0000 7fec75b31b80 1 --2- 172.21.15.78:0/1129195544 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558c260f49f0 0x558c260f6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.344+0000 7fec75b31b80 4 client.15825 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:55:07.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.344+0000 7fec75b31b80 1 -- 172.21.15.78:0/1129195544 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558c26025700 con 0x558c260f49f0 2024-03-14T09:55:07.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.344+0000 7fec5b400700 1 --2- 172.21.15.78:0/1129195544 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558c260f49f0 0x558c260f6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.344+0000 7fec5b400700 1 --2- 172.21.15.78:0/1129195544 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558c260f49f0 0x558c260f6de0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f6d89ff43495ce6 server_cookie=f7c0374c79dff1e1 in_seq=0 out_seq=0 2024-03-14T09:55:07.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.344+0000 7fec44c00700 10 client.15825 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:07.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.345+0000 7fec44c00700 1 -- 172.21.15.78:0/1129195544 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x558c26025700 con 0x558c260f49f0 2024-03-14T09:55:07.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.345+0000 7fec44c00700 10 client.15825 handle_command_reply: tid=0 2024-03-14T09:55:07.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.345+0000 7fec75b31b80 1 client.15825 shutdown 2024-03-14T09:55:07.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.345+0000 7fec75b31b80 1 -- 172.21.15.78:0/1129195544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260e4ec0 msgr2=0x558c260e2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:07.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.345+0000 7fec75b31b80 1 --2- 172.21.15.78:0/1129195544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260e4ec0 0x558c260e2160 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fec54011650 tx=0x7fec54011f20 comp rx=0 tx=0).stop 2024-03-14T09:55:07.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.345+0000 7fec75b31b80 1 -- 172.21.15.78:0/1129195544 shutdown_connections 2024-03-14T09:55:07.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.345+0000 7fec75b31b80 1 --2- 172.21.15.78:0/1129195544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260e4ec0 0x558c260e2160 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.345+0000 7fec75b31b80 1 --2- 172.21.15.78:0/1129195544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 0x558c260e4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.345+0000 7fec75b31b80 1 --2- 172.21.15.78:0/1129195544 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558c260f49f0 0x558c260f6de0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.345+0000 7fec75b31b80 1 --2- 172.21.15.78:0/1129195544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260dafa0 0x558c260e1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.345+0000 7fec75b31b80 1 -- 172.21.15.78:0/1129195544 >> 172.21.15.78:0/1129195544 conn(0x558c26050220 msgr2=0x558c260d5690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:07.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.345+0000 7fec44c00700 0 client.15825 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:07.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.345+0000 7fec75b31b80 1 -- 172.21.15.78:0/1129195544 shutdown_connections 2024-03-14T09:55:07.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.346+0000 7fec75b31b80 1 -- 172.21.15.78:0/1129195544 wait complete. 2024-03-14T09:55:07.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.346+0000 7fec75b31b80 20 client.15825 trim_cache size 0 max 16384 2024-03-14T09:55:07.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.363+0000 7fec75b31b80 1 Processor -- start 2024-03-14T09:55:07.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.363+0000 7fec75b31b80 1 -- start start 2024-03-14T09:55:07.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.363+0000 7fec75b31b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260da680 0x558c260f0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.363+0000 7fec75b31b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.364+0000 7fec75b31b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260e4ec0 0x558c260dee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.364+0000 7fec75b31b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c260f47d0 con 0x558c260dafa0 2024-03-14T09:55:07.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.364+0000 7fec75b31b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c260cfd00 con 0x558c260da680 2024-03-14T09:55:07.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.364+0000 7fec75b31b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c260e2740 con 0x558c260e4ec0 2024-03-14T09:55:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.364+0000 7fec5aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.364+0000 7fec5b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260da680 0x558c260f0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.364+0000 7fec5aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44586/0 (socket says 172.21.15.78:44586) 2024-03-14T09:55:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.364+0000 7fec5aa00700 1 -- 172.21.15.78:0/1194169624 learned_addr learned my addr 172.21.15.78:0/1194169624 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.364+0000 7fec5be00700 1 --2- 172.21.15.78:0/1194169624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260e4ec0 0x558c260dee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.364+0000 7fec5b400700 1 -- 172.21.15.78:0/1194169624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260e4ec0 msgr2=0x558c260dee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.364+0000 7fec5b400700 1 --2- 172.21.15.78:0/1194169624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260e4ec0 0x558c260dee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.364+0000 7fec5b400700 1 -- 172.21.15.78:0/1194169624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 msgr2=0x558c260e4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.364+0000 7fec5b400700 1 --2- 172.21.15.78:0/1194169624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.364+0000 7fec5b400700 1 -- 172.21.15.78:0/1194169624 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c25e5ab60 con 0x558c260da680 2024-03-14T09:55:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.365+0000 7fec5b400700 1 --2- 172.21.15.78:0/1194169624 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260da680 0x558c260f0870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x558c260e3120 tx=0x7fec4c01b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=618b73548e55403a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.365+0000 7fec51200700 1 -- 172.21.15.78:0/1194169624 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c025040 con 0x558c260da680 2024-03-14T09:55:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.365+0000 7fec51200700 1 -- 172.21.15.78:0/1194169624 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec4c012e90 con 0x558c260da680 2024-03-14T09:55:07.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.365+0000 7fec51200700 1 -- 172.21.15.78:0/1194169624 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c017070 con 0x558c260da680 2024-03-14T09:55:07.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.366+0000 7fec75b31b80 1 -- 172.21.15.78:0/1194169624 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260da680 msgr2=0x558c260f0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:07.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.366+0000 7fec75b31b80 1 --2- 172.21.15.78:0/1194169624 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260da680 0x558c260f0870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x558c260e3120 tx=0x7fec4c01b000 comp rx=0 tx=0).stop 2024-03-14T09:55:07.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.366+0000 7fec75b31b80 1 -- 172.21.15.78:0/1194169624 shutdown_connections 2024-03-14T09:55:07.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.366+0000 7fec75b31b80 1 --2- 172.21.15.78:0/1194169624 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260da680 0x558c260f0870 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.366+0000 7fec75b31b80 1 --2- 172.21.15.78:0/1194169624 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260e4ec0 0x558c260dee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.366+0000 7fec75b31b80 1 --2- 172.21.15.78:0/1194169624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.366+0000 7fec75b31b80 1 -- 172.21.15.78:0/1194169624 >> 172.21.15.78:0/1194169624 conn(0x558c26050220 msgr2=0x558c260f0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:07.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.366+0000 7fec75b31b80 1 -- 172.21.15.78:0/1194169624 shutdown_connections 2024-03-14T09:55:07.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.366+0000 7fec75b31b80 1 -- 172.21.15.78:0/1194169624 wait complete. 2024-03-14T09:55:07.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.366+0000 7fec75b31b80 1 -- start start 2024-03-14T09:55:07.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.367+0000 7fec75b31b80 1 Processor -- start 2024-03-14T09:55:07.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.367+0000 7fec75b31b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 0x558c260eba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.367+0000 7fec75b31b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.367+0000 7fec75b31b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260e4ec0 0x558c260ea4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.367+0000 7fec75b31b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c260eac00 con 0x558c260dafa0 2024-03-14T09:55:07.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.367+0000 7fec75b31b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c260ead70 con 0x558c260e4ec0 2024-03-14T09:55:07.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.367+0000 7fec75b31b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c261464f0 con 0x558c260da680 2024-03-14T09:55:07.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.367+0000 7fec75b31b80 10 client.0 _ll_register_callbacks cb 0x7fec480d3400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:07.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.367+0000 7fec5aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.368+0000 7fec5aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44598/0 (socket says 172.21.15.78:44598) 2024-03-14T09:55:07.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.368+0000 7fec5aa00700 1 -- 172.21.15.78:0/2119203533 learned_addr learned my addr 172.21.15.78:0/2119203533 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:07.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.368+0000 7fec5b400700 1 --2- 172.21.15.78:0/2119203533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 0x558c260eba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.368+0000 7fec5be00700 1 --2- 172.21.15.78:0/2119203533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260e4ec0 0x558c260ea4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.368+0000 7fec5aa00700 1 -- 172.21.15.78:0/2119203533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 msgr2=0x558c260eba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:07.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.368+0000 7fec5aa00700 1 --2- 172.21.15.78:0/2119203533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 0x558c260eba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.368+0000 7fec5aa00700 1 -- 172.21.15.78:0/2119203533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260e4ec0 msgr2=0x558c260ea4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:07.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.368+0000 7fec5aa00700 1 --2- 172.21.15.78:0/2119203533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260e4ec0 0x558c260ea4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.368+0000 7fec5aa00700 1 -- 172.21.15.78:0/2119203533 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec4c004240 con 0x558c260dafa0 2024-03-14T09:55:07.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.368+0000 7fec5be00700 1 --2- 172.21.15.78:0/2119203533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260e4ec0 0x558c260ea4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:55:07.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.368+0000 7fec5aa00700 1 --2- 172.21.15.78:0/2119203533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e9f80 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fec4809a1e0 tx=0x7fec480097f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caecba434bc18101 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:07.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.368+0000 7fec44c00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:55:07.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.368+0000 7fec44c00700 1 -- 172.21.15.78:0/2119203533 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4809e120 con 0x558c260dafa0 2024-03-14T09:55:07.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.368+0000 7fec44c00700 1 -- 172.21.15.78:0/2119203533 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fec480052b0 con 0x558c260dafa0 2024-03-14T09:55:07.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.368+0000 7fec75b31b80 1 -- 172.21.15.78:0/2119203533 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558c25e7bcf0 con 0x558c260dafa0 2024-03-14T09:55:07.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.369+0000 7fec44c00700 1 -- 172.21.15.78:0/2119203533 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec48096820 con 0x558c260dafa0 2024-03-14T09:55:07.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.369+0000 7fec75b31b80 10 client.15831 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:07.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.369+0000 7fec75b31b80 1 -- 172.21.15.78:0/2119203533 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558c25ff2ea0 con 0x558c260dafa0 2024-03-14T09:55:07.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.369+0000 7fec44c00700 1 -- 172.21.15.78:0/2119203533 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec480b6080 con 0x558c260dafa0 2024-03-14T09:55:07.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.369+0000 7fec44c00700 1 -- 172.21.15.78:0/2119203533 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fec480cd310 con 0x558c260dafa0 2024-03-14T09:55:07.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.369+0000 7fec75b31b80 10 client.15831 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:07.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.369+0000 7fec75b31b80 10 client.15831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:55:07.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.369+0000 7fec75b31b80 20 client.15831 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:07.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.369+0000 7fec75b31b80 1 --2- 172.21.15.78:0/2119203533 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558c26147930 0x558c26149d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.369+0000 7fec75b31b80 4 client.15831 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:55:07.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.369+0000 7fec75b31b80 1 -- 172.21.15.78:0/2119203533 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558c25eb0ce0 con 0x558c26147930 2024-03-14T09:55:07.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.370+0000 7fec5b400700 1 --2- 172.21.15.78:0/2119203533 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558c26147930 0x558c26149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.370+0000 7fec5b400700 1 --2- 172.21.15.78:0/2119203533 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558c26147930 0x558c26149d50 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=828fec72639b006b server_cookie=2deba95cddca3e49 in_seq=0 out_seq=0 2024-03-14T09:55:07.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.370+0000 7fec44c00700 10 client.15831 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:07.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.370+0000 7fec44c00700 1 -- 172.21.15.78:0/2119203533 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x558c25eb0ce0 con 0x558c26147930 2024-03-14T09:55:07.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.370+0000 7fec44c00700 10 client.15831 handle_command_reply: tid=0 2024-03-14T09:55:07.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.370+0000 7fec75b31b80 1 client.15831 shutdown 2024-03-14T09:55:07.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.371+0000 7fec75b31b80 1 -- 172.21.15.78:0/2119203533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 msgr2=0x558c260e9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:07.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.371+0000 7fec75b31b80 1 --2- 172.21.15.78:0/2119203533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e9f80 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fec4809a1e0 tx=0x7fec480097f0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.371+0000 7fec75b31b80 1 -- 172.21.15.78:0/2119203533 shutdown_connections 2024-03-14T09:55:07.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.371+0000 7fec75b31b80 1 --2- 172.21.15.78:0/2119203533 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558c26147930 0x558c26149d50 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.371+0000 7fec75b31b80 1 --2- 172.21.15.78:0/2119203533 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558c260e4ec0 0x558c260ea4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.371+0000 7fec75b31b80 1 --2- 172.21.15.78:0/2119203533 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558c260da680 0x558c260eba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.371+0000 7fec75b31b80 1 --2- 172.21.15.78:0/2119203533 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558c260dafa0 0x558c260e9f80 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.371+0000 7fec44c00700 0 client.15831 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:07.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.371+0000 7fec75b31b80 1 -- 172.21.15.78:0/2119203533 >> 172.21.15.78:0/2119203533 conn(0x558c26050220 msgr2=0x558c260d9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:07.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.371+0000 7fec75b31b80 1 -- 172.21.15.78:0/2119203533 shutdown_connections 2024-03-14T09:55:07.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.371+0000 7fec75b31b80 1 -- 172.21.15.78:0/2119203533 wait complete. 2024-03-14T09:55:07.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.371+0000 7fec75b31b80 20 client.15831 trim_cache size 0 max 16384 2024-03-14T09:55:07.376 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:55:07.376 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T09:55:07.376 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T09:55:07.376 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:55:07.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.372+0000 7fec62000700 1 -- 172.21.15.78:0/1646061390 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fec400a0da0 msgr2=0x7fec400a3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:07.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.372+0000 7fec62000700 1 --2- 172.21.15.78:0/1646061390 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fec400a0da0 0x7fec400a3260 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fec4c005950 tx=0x7fec4c009310 comp rx=0 tx=0).stop 2024-03-14T09:55:07.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.372+0000 7fec62000700 1 -- 172.21.15.78:0/1646061390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec5c073b00 msgr2=0x7fec5c19fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:07.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.372+0000 7fec62000700 1 --2- 172.21.15.78:0/1646061390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec5c073b00 0x7fec5c19fb60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fec48009fd0 tx=0x7fec4800fa00 comp rx=0 tx=0).stop 2024-03-14T09:55:07.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.372+0000 7fec62000700 1 -- 172.21.15.78:0/1646061390 shutdown_connections 2024-03-14T09:55:07.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.372+0000 7fec62000700 1 --2- 172.21.15.78:0/1646061390 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fec400a0da0 0x7fec400a3260 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.372+0000 7fec62000700 1 --2- 172.21.15.78:0/1646061390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec5c10d500 0x7fec5c199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.372+0000 7fec62000700 1 --2- 172.21.15.78:0/1646061390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec5c0731e0 0x7fec5c19f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.372+0000 7fec62000700 1 --2- 172.21.15.78:0/1646061390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec5c073b00 0x7fec5c19fb60 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.372+0000 7fec62000700 1 -- 172.21.15.78:0/1646061390 >> 172.21.15.78:0/1646061390 conn(0x7fec5c0fc7b0 msgr2=0x7fec5c0fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:07.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.372+0000 7fec62000700 1 -- 172.21.15.78:0/1646061390 shutdown_connections 2024-03-14T09:55:07.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.372+0000 7fec62000700 1 -- 172.21.15.78:0/1646061390 wait complete. 2024-03-14T09:55:07.388 INFO:tasks.workunit.client.0.smithi078.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-14T09:55:07.391 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T09:55:07.391 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:55:07.395 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T09:55:07.395 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-14T09:55:07.481 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-03-14T09:55:07.482 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-03-14T09:55:07.483 DEBUG:teuthology.orchestra.run.smithi078:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2f465319e3c83d48f999325459bcdfd0346449c0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-03-14T09:55:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:07 smithi152 ceph-mon[48279]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-14T09:55:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:07 smithi132 ceph-mon[47959]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-14T09:55:07.576 INFO:tasks.workunit.client.0.smithi078.stderr:+ set -e 2024-03-14T09:55:07.576 INFO:tasks.workunit.client.0.smithi078.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-03-14T09:55:07.581 INFO:tasks.workunit.client.0.smithi078.stderr:Cloning into 'xfstests-dev'... 2024-03-14T09:55:07.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.636+0000 7fe305c00700 1 -- 172.21.15.78:0/1066862791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe300101e20 msgr2=0x7fe300102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:07.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.636+0000 7fe305c00700 1 --2- 172.21.15.78:0/1066862791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe300101e20 0x7fe300102200 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0009990 tx=0x7fe2f0009ca0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.636+0000 7fe305c00700 1 -- 172.21.15.78:0/1066862791 shutdown_connections 2024-03-14T09:55:07.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.636+0000 7fe305c00700 1 --2- 172.21.15.78:0/1066862791 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe30010b1f0 0x7fe30010d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.636+0000 7fe305c00700 1 --2- 172.21.15.78:0/1066862791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe300101e20 0x7fe300102200 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.636+0000 7fe305c00700 1 --2- 172.21.15.78:0/1066862791 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe300102740 0x7fe30010ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.636+0000 7fe305c00700 1 -- 172.21.15.78:0/1066862791 >> 172.21.15.78:0/1066862791 conn(0x7fe3000fb5c0 msgr2=0x7fe3000fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:07.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.636+0000 7fe305c00700 1 -- 172.21.15.78:0/1066862791 shutdown_connections 2024-03-14T09:55:07.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.636+0000 7fe305c00700 1 -- 172.21.15.78:0/1066862791 wait complete. 2024-03-14T09:55:07.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.637+0000 7fe305c00700 1 Processor -- start 2024-03-14T09:55:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.637+0000 7fe305c00700 1 -- start start 2024-03-14T09:55:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.637+0000 7fe305c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe300101e20 0x7fe3000ffa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe305c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe300102740 0x7fe3000fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe305c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe30010b1f0 0x7fe3001004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe305c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3001a58a0 con 0x7fe300102740 2024-03-14T09:55:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe305c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3001a5a10 con 0x7fe300101e20 2024-03-14T09:55:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe305c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3001a5bd0 con 0x7fe30010b1f0 2024-03-14T09:55:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe2fea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe300101e20 0x7fe3000ffa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe2fe000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe300102740 0x7fe3000fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe2ff400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe30010b1f0 0x7fe3001004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe2fe000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe300102740 0x7fe3000fffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44614/0 (socket says 172.21.15.78:44614) 2024-03-14T09:55:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe2fea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe300101e20 0x7fe3000ffa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60760/0 (socket says 172.21.15.78:60760) 2024-03-14T09:55:07.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe2ff400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe30010b1f0 0x7fe3001004f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:53022/0 (socket says 172.21.15.78:53022) 2024-03-14T09:55:07.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe2fea00700 1 -- 172.21.15.78:0/992183168 learned_addr learned my addr 172.21.15.78:0/992183168 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:07.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe2fe000700 1 -- 172.21.15.78:0/992183168 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe30010b1f0 msgr2=0x7fe3001004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:07.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe2fe000700 1 --2- 172.21.15.78:0/992183168 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe30010b1f0 0x7fe3001004f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe2fe000700 1 -- 172.21.15.78:0/992183168 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe300101e20 msgr2=0x7fe3000ffa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:07.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe2fe000700 1 --2- 172.21.15.78:0/992183168 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe300101e20 0x7fe3000ffa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe2fe000700 1 -- 172.21.15.78:0/992183168 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f00095f0 con 0x7fe300102740 2024-03-14T09:55:07.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.638+0000 7fe2fea00700 1 --2- 172.21.15.78:0/992183168 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe300101e20 0x7fe3000ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:07.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.639+0000 7fe2fe000700 1 --2- 172.21.15.78:0/992183168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe300102740 0x7fe3000fffb0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00e9f0 tx=0x7fe2ec00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:07.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.639+0000 7fe2f7400700 1 -- 172.21.15.78:0/992183168 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec00c9c0 con 0x7fe300102740 2024-03-14T09:55:07.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.639+0000 7fe2f7400700 1 -- 172.21.15.78:0/992183168 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2ec010430 con 0x7fe300102740 2024-03-14T09:55:07.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.639+0000 7fe305c00700 1 -- 172.21.15.78:0/992183168 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3001a5eb0 con 0x7fe300102740 2024-03-14T09:55:07.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.639+0000 7fe2f7400700 1 -- 172.21.15.78:0/992183168 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec018680 con 0x7fe300102740 2024-03-14T09:55:07.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.639+0000 7fe305c00700 1 -- 172.21.15.78:0/992183168 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3001a6330 con 0x7fe300102740 2024-03-14T09:55:07.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.640+0000 7fe305c00700 1 -- 172.21.15.78:0/992183168 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3001062d0 con 0x7fe300102740 2024-03-14T09:55:07.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.642+0000 7fe2f7400700 1 -- 172.21.15.78:0/992183168 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) v1 ==== 135621+0+0 (secure 0 0 0) 0x7fe2ec010860 con 0x7fe300102740 2024-03-14T09:55:07.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.643+0000 7fe2f7400700 1 --2- 172.21.15.78:0/992183168 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe2e40a0d20 0x7fe2e40a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:07.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.643+0000 7fe2f7400700 1 -- 172.21.15.78:0/992183168 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe2ec014070 con 0x7fe300102740 2024-03-14T09:55:07.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.643+0000 7fe2fea00700 1 --2- 172.21.15.78:0/992183168 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe2e40a0d20 0x7fe2e40a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:07.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.644+0000 7fe2fea00700 1 --2- 172.21.15.78:0/992183168 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe2e40a0d20 0x7fe2e40a31e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0006010 tx=0x7fe2f000b280 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:07.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.648+0000 7fe2f7400700 1 -- 172.21.15.78:0/992183168 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe2ec052960 con 0x7fe300102740 2024-03-14T09:55:07.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.910+0000 7fe305c00700 1 -- 172.21.15.78:0/992183168 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe300068600 con 0x7fe300102740 2024-03-14T09:55:07.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.911+0000 7fe2f7400700 1 -- 172.21.15.78:0/992183168 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7fe2ec095c50 con 0x7fe300102740 2024-03-14T09:55:07.912 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:55:07.912 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:55:07.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.914+0000 7fe305c00700 1 -- 172.21.15.78:0/992183168 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe2e40a0d20 msgr2=0x7fe2e40a31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:07.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.914+0000 7fe305c00700 1 --2- 172.21.15.78:0/992183168 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe2e40a0d20 0x7fe2e40a31e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0006010 tx=0x7fe2f000b280 comp rx=0 tx=0).stop 2024-03-14T09:55:07.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.914+0000 7fe305c00700 1 -- 172.21.15.78:0/992183168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe300102740 msgr2=0x7fe3000fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:07.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.914+0000 7fe305c00700 1 --2- 172.21.15.78:0/992183168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe300102740 0x7fe3000fffb0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00e9f0 tx=0x7fe2ec00ed00 comp rx=0 tx=0).stop 2024-03-14T09:55:07.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.914+0000 7fe305c00700 1 -- 172.21.15.78:0/992183168 shutdown_connections 2024-03-14T09:55:07.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.914+0000 7fe305c00700 1 --2- 172.21.15.78:0/992183168 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe2e40a0d20 0x7fe2e40a31e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.914+0000 7fe305c00700 1 --2- 172.21.15.78:0/992183168 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe300101e20 0x7fe3000ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.914+0000 7fe305c00700 1 --2- 172.21.15.78:0/992183168 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe30010b1f0 0x7fe3001004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.914+0000 7fe305c00700 1 --2- 172.21.15.78:0/992183168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe300102740 0x7fe3000fffb0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:07.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.914+0000 7fe305c00700 1 -- 172.21.15.78:0/992183168 >> 172.21.15.78:0/992183168 conn(0x7fe3000fb5c0 msgr2=0x7fe3000fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:07.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.914+0000 7fe305c00700 1 -- 172.21.15.78:0/992183168 shutdown_connections 2024-03-14T09:55:07.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:07.914+0000 7fe305c00700 1 -- 172.21.15.78:0/992183168 wait complete. 2024-03-14T09:55:07.916 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T09:55:07.934 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T09:55:08.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.130+0000 7fe9dbe00700 1 -- 172.21.15.78:0/2690201512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9d4100f30 msgr2=0x7fe9d4101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.130+0000 7fe9dbe00700 1 --2- 172.21.15.78:0/2690201512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9d4100f30 0x7fe9d4101310 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009990 tx=0x7fe9c8009ca0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.131+0000 7fe9dbe00700 1 -- 172.21.15.78:0/2690201512 shutdown_connections 2024-03-14T09:55:08.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.131+0000 7fe9dbe00700 1 --2- 172.21.15.78:0/2690201512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9d4106050 0x7fe9d410dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.131+0000 7fe9dbe00700 1 --2- 172.21.15.78:0/2690201512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9d4100f30 0x7fe9d4101310 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.131+0000 7fe9dbe00700 1 --2- 172.21.15.78:0/2690201512 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9d41018e0 0x7fe9d4105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.131+0000 7fe9dbe00700 1 -- 172.21.15.78:0/2690201512 >> 172.21.15.78:0/2690201512 conn(0x7fe9d40fc7b0 msgr2=0x7fe9d40febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.131+0000 7fe9dbe00700 1 -- 172.21.15.78:0/2690201512 shutdown_connections 2024-03-14T09:55:08.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.131+0000 7fe9dbe00700 1 -- 172.21.15.78:0/2690201512 wait complete. 2024-03-14T09:55:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.132+0000 7fe9dbe00700 1 Processor -- start 2024-03-14T09:55:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.132+0000 7fe9dbe00700 1 -- start start 2024-03-14T09:55:08.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.132+0000 7fe9dbe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9d4100f30 0x7fe9d419f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.132+0000 7fe9dbe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9d4106050 0x7fe9d419fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.132+0000 7fe9dbe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9d4199660 0x7fe9d4199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.132+0000 7fe9dbe00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d419a0e0 con 0x7fe9d4199660 2024-03-14T09:55:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.132+0000 7fe9dbe00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d419a250 con 0x7fe9d4100f30 2024-03-14T09:55:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.132+0000 7fe9dbe00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d419a410 con 0x7fe9d4106050 2024-03-14T09:55:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.133+0000 7fe9d9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9d4199660 0x7fe9d4199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.133+0000 7fe9d9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9d4199660 0x7fe9d4199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44628/0 (socket says 172.21.15.78:44628) 2024-03-14T09:55:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.133+0000 7fe9d9e00700 1 -- 172.21.15.78:0/4150370795 learned_addr learned my addr 172.21.15.78:0/4150370795 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.133+0000 7fe9d8a00700 1 --2- 172.21.15.78:0/4150370795 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9d4106050 0x7fe9d419fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.133+0000 7fe9d9400700 1 --2- 172.21.15.78:0/4150370795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9d4100f30 0x7fe9d419f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.133+0000 7fe9d9e00700 1 -- 172.21.15.78:0/4150370795 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9d4106050 msgr2=0x7fe9d419fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.133+0000 7fe9d9e00700 1 --2- 172.21.15.78:0/4150370795 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9d4106050 0x7fe9d419fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.133+0000 7fe9d9e00700 1 -- 172.21.15.78:0/4150370795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9d4100f30 msgr2=0x7fe9d419f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.133+0000 7fe9d9e00700 1 --2- 172.21.15.78:0/4150370795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9d4100f30 0x7fe9d419f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.133+0000 7fe9d9e00700 1 -- 172.21.15.78:0/4150370795 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c80095f0 con 0x7fe9d4199660 2024-03-14T09:55:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.133+0000 7fe9d9400700 1 --2- 172.21.15.78:0/4150370795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9d4100f30 0x7fe9d419f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.134+0000 7fe9d9e00700 1 --2- 172.21.15.78:0/4150370795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9d4199660 0x7fe9d4199b10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc007ee0 tx=0x7fe9cc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.134+0000 7fe9d2000700 1 -- 172.21.15.78:0/4150370795 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc017070 con 0x7fe9d4199660 2024-03-14T09:55:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.134+0000 7fe9d2000700 1 -- 172.21.15.78:0/4150370795 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9cc00f040 con 0x7fe9d4199660 2024-03-14T09:55:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.134+0000 7fe9d2000700 1 -- 172.21.15.78:0/4150370795 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc005580 con 0x7fe9d4199660 2024-03-14T09:55:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.134+0000 7fe9dbe00700 1 -- 172.21.15.78:0/4150370795 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d41a6400 con 0x7fe9d4199660 2024-03-14T09:55:08.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.134+0000 7fe9dbe00700 1 -- 172.21.15.78:0/4150370795 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d41a6890 con 0x7fe9d4199660 2024-03-14T09:55:08.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.137+0000 7fe9d2000700 1 -- 172.21.15.78:0/4150370795 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) v1 ==== 135621+0+0 (secure 0 0 0) 0x7fe9cc00db20 con 0x7fe9d4199660 2024-03-14T09:55:08.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.138+0000 7fe9d2000700 1 --2- 172.21.15.78:0/4150370795 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe9bc0a0da0 0x7fe9bc0a3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.138+0000 7fe9ef9dfb80 1 Processor -- start 2024-03-14T09:55:08.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.138+0000 7fe9d2000700 1 -- 172.21.15.78:0/4150370795 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe9cc0cc060 con 0x7fe9d4199660 2024-03-14T09:55:08.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.138+0000 7fe9ef9dfb80 1 -- start start 2024-03-14T09:55:08.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.138+0000 7fe9d9400700 1 --2- 172.21.15.78:0/4150370795 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe9bc0a0da0 0x7fe9bc0a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.139+0000 7fe9ef9dfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4e680 0x5654f9c4ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.139+0000 7fe9ef9dfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4efa0 0x5654f9c58980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.139+0000 7fe9ef9dfb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c58ec0 0x5654f9c5b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.139+0000 7fe9ef9dfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654f9ba7990 con 0x5654f9c4e680 2024-03-14T09:55:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.139+0000 7fe9ef9dfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654f9ba5350 con 0x5654f9c4efa0 2024-03-14T09:55:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.139+0000 7fe9ef9dfb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654f9ba0c40 con 0x5654f9c58ec0 2024-03-14T09:55:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.139+0000 7fe9d8a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4efa0 0x5654f9c58980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.139+0000 7fe9d9e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c58ec0 0x5654f9c5b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.139+0000 7fe9d8a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4efa0 0x5654f9c58980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60786/0 (socket says 172.21.15.78:60786) 2024-03-14T09:55:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.139+0000 7fe9d8a00700 1 -- 172.21.15.78:0/810697726 learned_addr learned my addr 172.21.15.78:0/810697726 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.139+0000 7fe9d9400700 1 --2- 172.21.15.78:0/810697726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4e680 0x5654f9c4ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.140+0000 7fe9d9400700 1 --2- 172.21.15.78:0/4150370795 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe9bc0a0da0 0x7fe9bc0a3260 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800b5c0 tx=0x7fe9c8005c20 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.140+0000 7fe9d8a00700 1 -- 172.21.15.78:0/810697726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c58ec0 msgr2=0x5654f9c5b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.140+0000 7fe9d8a00700 1 --2- 172.21.15.78:0/810697726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c58ec0 0x5654f9c5b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.140+0000 7fe9d8a00700 1 -- 172.21.15.78:0/810697726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4e680 msgr2=0x5654f9c4ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.140+0000 7fe9d8a00700 1 --2- 172.21.15.78:0/810697726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4e680 0x5654f9c4ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.140+0000 7fe9d8a00700 1 -- 172.21.15.78:0/810697726 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654f9a223b0 con 0x5654f9c4efa0 2024-03-14T09:55:08.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.140+0000 7fe9d8a00700 1 --2- 172.21.15.78:0/810697726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4efa0 0x5654f9c58980 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000bcf0 tx=0x7fe9c00103d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d0eb066ca85cd5c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.140+0000 7fe9c6600700 1 -- 172.21.15.78:0/810697726 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0010c50 con 0x5654f9c4efa0 2024-03-14T09:55:08.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.140+0000 7fe9c6600700 1 -- 172.21.15.78:0/810697726 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9c0015070 con 0x5654f9c4efa0 2024-03-14T09:55:08.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.140+0000 7fe9c6600700 1 -- 172.21.15.78:0/810697726 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0005020 con 0x5654f9c4efa0 2024-03-14T09:55:08.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.141+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/810697726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4efa0 msgr2=0x5654f9c58980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.141+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/810697726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4efa0 0x5654f9c58980 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000bcf0 tx=0x7fe9c00103d0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.141+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/810697726 shutdown_connections 2024-03-14T09:55:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.141+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/810697726 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c58ec0 0x5654f9c5b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.141+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/810697726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4e680 0x5654f9c4ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.141+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/810697726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4efa0 0x5654f9c58980 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.141+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/810697726 >> 172.21.15.78:0/810697726 conn(0x5654f9bc4220 msgr2=0x5654f9c49aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.141+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/810697726 shutdown_connections 2024-03-14T09:55:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.141+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/810697726 wait complete. 2024-03-14T09:55:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.142+0000 7fe9ef9dfb80 1 -- start start 2024-03-14T09:55:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.142+0000 7fe9ef9dfb80 1 Processor -- start 2024-03-14T09:55:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.142+0000 7fe9ef9dfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4e680 0x5654f9c58830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.142+0000 7fe9ef9dfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4efa0 0x5654f9c55ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.142+0000 7fe9ef9dfb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c58ec0 0x5654f9c56030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.142+0000 7fe9ef9dfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654f9b3cd80 con 0x5654f9c4efa0 2024-03-14T09:55:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.142+0000 7fe9ef9dfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654f9b3ca20 con 0x5654f9c4e680 2024-03-14T09:55:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.142+0000 7fe9ef9dfb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654f9a246a0 con 0x5654f9c58ec0 2024-03-14T09:55:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.142+0000 7fe9ef9dfb80 10 client.0 _ll_register_callbacks cb 0x5654f99a83a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.143+0000 7fe9d8a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4efa0 0x5654f9c55ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.143+0000 7fe9d9e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c58ec0 0x5654f9c56030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.143+0000 7fe9d8a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4efa0 0x5654f9c55ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44644/0 (socket says 172.21.15.78:44644) 2024-03-14T09:55:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.143+0000 7fe9d8a00700 1 -- 172.21.15.78:0/3284952784 learned_addr learned my addr 172.21.15.78:0/3284952784 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.143+0000 7fe9d9400700 1 --2- 172.21.15.78:0/3284952784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4e680 0x5654f9c58830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.143+0000 7fe9d8a00700 1 -- 172.21.15.78:0/3284952784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c58ec0 msgr2=0x5654f9c56030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.143+0000 7fe9d8a00700 1 --2- 172.21.15.78:0/3284952784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c58ec0 0x5654f9c56030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.143+0000 7fe9d8a00700 1 -- 172.21.15.78:0/3284952784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4e680 msgr2=0x5654f9c58830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.143+0000 7fe9d8a00700 1 --2- 172.21.15.78:0/3284952784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4e680 0x5654f9c58830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.143+0000 7fe9d8a00700 1 -- 172.21.15.78:0/3284952784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c000b920 con 0x5654f9c4efa0 2024-03-14T09:55:08.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.143+0000 7fe9d8a00700 1 --2- 172.21.15.78:0/3284952784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4efa0 0x5654f9c55ab0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0004a10 tx=0x7fe9c0010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f12db2e819d7ea6 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.143+0000 7fe9baa00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:55:08.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.143+0000 7fe9baa00700 1 -- 172.21.15.78:0/3284952784 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0020070 con 0x5654f9c4efa0 2024-03-14T09:55:08.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.143+0000 7fe9baa00700 1 -- 172.21.15.78:0/3284952784 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe9c0017070 con 0x5654f9c4efa0 2024-03-14T09:55:08.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.143+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/3284952784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5654f9c41720 con 0x5654f9c4efa0 2024-03-14T09:55:08.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.144+0000 7fe9baa00700 1 -- 172.21.15.78:0/3284952784 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9c00047c0 con 0x5654f9c4efa0 2024-03-14T09:55:08.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.144+0000 7fe9ef9dfb80 10 client.15849 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:08.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.144+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/3284952784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5654f99ceb60 con 0x5654f9c4efa0 2024-03-14T09:55:08.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.144+0000 7fe9baa00700 1 -- 172.21.15.78:0/3284952784 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5654f9c41720 con 0x5654f9c4efa0 2024-03-14T09:55:08.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.144+0000 7fe9baa00700 1 -- 172.21.15.78:0/3284952784 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe9c0011440 con 0x5654f9c4efa0 2024-03-14T09:55:08.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.144+0000 7fe9ef9dfb80 10 client.15849 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:08.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.144+0000 7fe9ef9dfb80 10 client.15849 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:55:08.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.144+0000 7fe9ef9dfb80 20 client.15849 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:08.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.144+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/3284952784 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5654f9c68a30 0x5654f9c6ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.144+0000 7fe9ef9dfb80 4 client.15849 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:55:08.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.144+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/3284952784 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5654f9b99700 con 0x5654f9c68a30 2024-03-14T09:55:08.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.145+0000 7fe9d9400700 1 --2- 172.21.15.78:0/3284952784 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5654f9c68a30 0x5654f9c6ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.145+0000 7fe9d9400700 1 --2- 172.21.15.78:0/3284952784 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5654f9c68a30 0x5654f9c6ae50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5352370a27ef62f4 server_cookie=c77a5c2c468c85d2 in_seq=0 out_seq=0 2024-03-14T09:55:08.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.145+0000 7fe9baa00700 10 client.15849 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:55:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.146+0000 7fe9baa00700 1 -- 172.21.15.78:0/3284952784 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5654f9b99700 con 0x5654f9c68a30 2024-03-14T09:55:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.146+0000 7fe9baa00700 10 client.15849 handle_command_reply: tid=0 2024-03-14T09:55:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.146+0000 7fe9ef9dfb80 1 client.15849 shutdown 2024-03-14T09:55:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.146+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/3284952784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4efa0 msgr2=0x5654f9c55ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.146+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/3284952784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4efa0 0x5654f9c55ab0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0004a10 tx=0x7fe9c0010c80 comp rx=0 tx=0).stop 2024-03-14T09:55:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.146+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/3284952784 shutdown_connections 2024-03-14T09:55:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.146+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/3284952784 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5654f9c68a30 0x5654f9c6ae50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.146+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/3284952784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4e680 0x5654f9c58830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.146+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/3284952784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c58ec0 0x5654f9c56030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.146+0000 7fe9baa00700 0 client.15849 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:55:08.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.146+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/3284952784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4efa0 0x5654f9c55ab0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.146+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/3284952784 >> 172.21.15.78:0/3284952784 conn(0x5654f9bc4220 msgr2=0x5654f9c49690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.146+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/3284952784 shutdown_connections 2024-03-14T09:55:08.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.146+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/3284952784 wait complete. 2024-03-14T09:55:08.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.146+0000 7fe9ef9dfb80 20 client.15849 trim_cache size 0 max 16384 2024-03-14T09:55:08.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.159+0000 7fe9ef9dfb80 1 Processor -- start 2024-03-14T09:55:08.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.159+0000 7fe9ef9dfb80 1 -- start start 2024-03-14T09:55:08.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9ef9dfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4e680 0x5654f9c648a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9ef9dfb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c4efa0 0x5654f9c58360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9ef9dfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c58ec0 0x5654f9c52d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9ef9dfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654f9c56630 con 0x5654f9c58ec0 2024-03-14T09:55:08.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9ef9dfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654f9c50920 con 0x5654f9c4e680 2024-03-14T09:55:08.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9ef9dfb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654f9c4ae30 con 0x5654f9c4efa0 2024-03-14T09:55:08.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9d9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c58ec0 0x5654f9c52d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9d9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c58ec0 0x5654f9c52d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44660/0 (socket says 172.21.15.78:44660) 2024-03-14T09:55:08.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9d9e00700 1 -- 172.21.15.78:0/940883124 learned_addr learned my addr 172.21.15.78:0/940883124 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:08.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9d8a00700 1 --2- 172.21.15.78:0/940883124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c4efa0 0x5654f9c58360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9d9400700 1 --2- 172.21.15.78:0/940883124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4e680 0x5654f9c648a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9d9e00700 1 -- 172.21.15.78:0/940883124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c4efa0 msgr2=0x5654f9c58360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9d9e00700 1 --2- 172.21.15.78:0/940883124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c4efa0 0x5654f9c58360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9d9e00700 1 -- 172.21.15.78:0/940883124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4e680 msgr2=0x5654f9c648a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9d9e00700 1 --2- 172.21.15.78:0/940883124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4e680 0x5654f9c648a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9d9e00700 1 -- 172.21.15.78:0/940883124 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654f9b66ea0 con 0x5654f9c58ec0 2024-03-14T09:55:08.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9d9400700 1 --2- 172.21.15.78:0/940883124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4e680 0x5654f9c648a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:55:08.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9d8a00700 1 --2- 172.21.15.78:0/940883124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c4efa0 0x5654f9c58360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:55:08.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.160+0000 7fe9d9e00700 1 --2- 172.21.15.78:0/940883124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c58ec0 0x5654f9c52d10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc098d80 tx=0x7fe9cc01e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba734e7ce69ca25 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.161+0000 7fe9c7000700 1 -- 172.21.15.78:0/940883124 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc098df0 con 0x5654f9c58ec0 2024-03-14T09:55:08.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.161+0000 7fe9c7000700 1 -- 172.21.15.78:0/940883124 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9cc099050 con 0x5654f9c58ec0 2024-03-14T09:55:08.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.161+0000 7fe9c7000700 1 -- 172.21.15.78:0/940883124 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc0a1270 con 0x5654f9c58ec0 2024-03-14T09:55:08.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.161+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/940883124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c58ec0 msgr2=0x5654f9c52d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.161+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/940883124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c58ec0 0x5654f9c52d10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc098d80 tx=0x7fe9cc01e040 comp rx=0 tx=0).stop 2024-03-14T09:55:08.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.161+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/940883124 shutdown_connections 2024-03-14T09:55:08.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.161+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/940883124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4e680 0x5654f9c648a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.161+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/940883124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c58ec0 0x5654f9c52d10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.161+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/940883124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c4efa0 0x5654f9c58360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.161+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/940883124 >> 172.21.15.78:0/940883124 conn(0x5654f9bc4220 msgr2=0x5654f9c64190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.161+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/940883124 shutdown_connections 2024-03-14T09:55:08.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.161+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/940883124 wait complete. 2024-03-14T09:55:08.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.162+0000 7fe9ef9dfb80 1 -- start start 2024-03-14T09:55:08.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.162+0000 7fe9ef9dfb80 1 Processor -- start 2024-03-14T09:55:08.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.162+0000 7fe9ef9dfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4e680 0x5654f9c5f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.162+0000 7fe9ef9dfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4efa0 0x5654f9c5dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.162+0000 7fe9ef9dfb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c58ec0 0x5654f9c5e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.162+0000 7fe9ef9dfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654f9c5e6f0 con 0x5654f9c4e680 2024-03-14T09:55:08.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.162+0000 7fe9ef9dfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654f9c5e830 con 0x5654f9c4efa0 2024-03-14T09:55:08.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.162+0000 7fe9ef9dfb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654f9c5e9f0 con 0x5654f9c58ec0 2024-03-14T09:55:08.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.162+0000 7fe9ef9dfb80 10 client.0 _ll_register_callbacks cb 0x7fe9cc095130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:08.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.163+0000 7fe9d9400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4e680 0x5654f9c5f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.163+0000 7fe9d9400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4e680 0x5654f9c5f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44676/0 (socket says 172.21.15.78:44676) 2024-03-14T09:55:08.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.163+0000 7fe9d9400700 1 -- 172.21.15.78:0/3058493774 learned_addr learned my addr 172.21.15.78:0/3058493774 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:08.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.163+0000 7fe9d8a00700 1 --2- 172.21.15.78:0/3058493774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4efa0 0x5654f9c5dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.163+0000 7fe9d9e00700 1 --2- 172.21.15.78:0/3058493774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c58ec0 0x5654f9c5e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.163+0000 7fe9d9400700 1 -- 172.21.15.78:0/3058493774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c58ec0 msgr2=0x5654f9c5e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.163+0000 7fe9d9400700 1 --2- 172.21.15.78:0/3058493774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c58ec0 0x5654f9c5e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.163+0000 7fe9d9400700 1 -- 172.21.15.78:0/3058493774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4efa0 msgr2=0x5654f9c5dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.163+0000 7fe9d9400700 1 --2- 172.21.15.78:0/3058493774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4efa0 0x5654f9c5dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.163+0000 7fe9d9400700 1 -- 172.21.15.78:0/3058493774 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d41a6890 con 0x5654f9c4e680 2024-03-14T09:55:08.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.163+0000 7fe9d8a00700 1 --2- 172.21.15.78:0/3058493774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4efa0 0x5654f9c5dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:08.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.163+0000 7fe9d9400700 1 --2- 172.21.15.78:0/3058493774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4e680 0x5654f9c5f830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009990 tx=0x7fe9c8009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce5e7b9800d203 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.163+0000 7fe9baa00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:55:08.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.164+0000 7fe9baa00700 1 -- 172.21.15.78:0/3058493774 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8005250 con 0x5654f9c4e680 2024-03-14T09:55:08.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.164+0000 7fe9baa00700 1 -- 172.21.15.78:0/3058493774 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe9c802c040 con 0x5654f9c4e680 2024-03-14T09:55:08.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.164+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/3058493774 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe9c0015070 con 0x5654f9c4e680 2024-03-14T09:55:08.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.164+0000 7fe9baa00700 1 -- 172.21.15.78:0/3058493774 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9c80053b0 con 0x5654f9c4e680 2024-03-14T09:55:08.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.164+0000 7fe9ef9dfb80 10 client.15855 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:08.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.164+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/3058493774 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5654f9a26cd0 con 0x5654f9c4e680 2024-03-14T09:55:08.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.164+0000 7fe9baa00700 1 -- 172.21.15.78:0/3058493774 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9c0015070 con 0x5654f9c4e680 2024-03-14T09:55:08.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.164+0000 7fe9baa00700 1 -- 172.21.15.78:0/3058493774 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe9c80335a0 con 0x5654f9c4e680 2024-03-14T09:55:08.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.164+0000 7fe9ef9dfb80 10 client.15855 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:08.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.164+0000 7fe9ef9dfb80 10 client.15855 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:55:08.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.164+0000 7fe9ef9dfb80 20 client.15855 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:08.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.164+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/3058493774 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5654f9cbb8f0 0x5654f9cbdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.164+0000 7fe9ef9dfb80 4 client.15855 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:55:08.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.164+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/3058493774 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5654f99efcf0 con 0x5654f9cbb8f0 2024-03-14T09:55:08.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.165+0000 7fe9d8a00700 1 --2- 172.21.15.78:0/3058493774 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5654f9cbb8f0 0x5654f9cbdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.165+0000 7fe9baa00700 10 client.15855 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:55:08.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.165+0000 7fe9d8a00700 1 --2- 172.21.15.78:0/3058493774 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5654f9cbb8f0 0x5654f9cbdd10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=97d5e6dbd8f9b6e2 server_cookie=97b153ab17151792 in_seq=0 out_seq=0 2024-03-14T09:55:08.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.165+0000 7fe9baa00700 1 -- 172.21.15.78:0/3058493774 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5654f99efcf0 con 0x5654f9cbb8f0 2024-03-14T09:55:08.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.166+0000 7fe9baa00700 10 client.15855 handle_command_reply: tid=0 2024-03-14T09:55:08.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.166+0000 7fe9ef9dfb80 1 client.15855 shutdown 2024-03-14T09:55:08.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.166+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/3058493774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4e680 msgr2=0x5654f9c5f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.166+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/3058493774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4e680 0x5654f9c5f830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009990 tx=0x7fe9c8009ca0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.166+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/3058493774 shutdown_connections 2024-03-14T09:55:08.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.166+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/3058493774 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5654f9cbb8f0 0x5654f9cbdd10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.166+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/3058493774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5654f9c58ec0 0x5654f9c5e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.166+0000 7fe9baa00700 0 client.15855 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:55:08.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.166+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/3058493774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5654f9c4e680 0x5654f9c5f830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.166+0000 7fe9ef9dfb80 1 --2- 172.21.15.78:0/3058493774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654f9c4efa0 0x5654f9c5dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.166+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/3058493774 >> 172.21.15.78:0/3058493774 conn(0x5654f9bc4220 msgr2=0x5654f9c4ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.166+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/3058493774 shutdown_connections 2024-03-14T09:55:08.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.166+0000 7fe9ef9dfb80 1 -- 172.21.15.78:0/3058493774 wait complete. 2024-03-14T09:55:08.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.166+0000 7fe9ef9dfb80 20 client.15855 trim_cache size 0 max 16384 2024-03-14T09:55:08.174 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:55:08.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.167+0000 7fe9dbe00700 1 -- 172.21.15.78:0/4150370795 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe9bc0a0da0 msgr2=0x7fe9bc0a3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.167+0000 7fe9dbe00700 1 --2- 172.21.15.78:0/4150370795 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe9bc0a0da0 0x7fe9bc0a3260 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800b5c0 tx=0x7fe9c8005c20 comp rx=0 tx=0).stop 2024-03-14T09:55:08.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.167+0000 7fe9dbe00700 1 -- 172.21.15.78:0/4150370795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9d4199660 msgr2=0x7fe9d4199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.167+0000 7fe9dbe00700 1 --2- 172.21.15.78:0/4150370795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9d4199660 0x7fe9d4199b10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc007ee0 tx=0x7fe9cc00d3b0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.167+0000 7fe9dbe00700 1 -- 172.21.15.78:0/4150370795 shutdown_connections 2024-03-14T09:55:08.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.167+0000 7fe9dbe00700 1 --2- 172.21.15.78:0/4150370795 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe9bc0a0da0 0x7fe9bc0a3260 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.167+0000 7fe9dbe00700 1 --2- 172.21.15.78:0/4150370795 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9d4100f30 0x7fe9d419f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.167+0000 7fe9dbe00700 1 --2- 172.21.15.78:0/4150370795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9d4199660 0x7fe9d4199b10 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.167+0000 7fe9dbe00700 1 --2- 172.21.15.78:0/4150370795 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe9d4106050 0x7fe9d419fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.167+0000 7fe9dbe00700 1 -- 172.21.15.78:0/4150370795 >> 172.21.15.78:0/4150370795 conn(0x7fe9d40fc7b0 msgr2=0x7fe9d40fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.167+0000 7fe9dbe00700 1 -- 172.21.15.78:0/4150370795 shutdown_connections 2024-03-14T09:55:08.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.167+0000 7fe9dbe00700 1 -- 172.21.15.78:0/4150370795 wait complete. 2024-03-14T09:55:08.185 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T09:55:08.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.379+0000 7f91e8400700 1 -- 172.21.15.78:0/2210090242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e0106040 msgr2=0x7f91e010dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.379+0000 7f91e8400700 1 --2- 172.21.15.78:0/2210090242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e0106040 0x7f91e010dbc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f91d800b0a0 tx=0x7f91d800b3b0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.380+0000 7f91e8400700 1 -- 172.21.15.78:0/2210090242 shutdown_connections 2024-03-14T09:55:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.380+0000 7f91e8400700 1 --2- 172.21.15.78:0/2210090242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91e0100f20 0x7f91e0101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.380+0000 7f91e8400700 1 --2- 172.21.15.78:0/2210090242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e0106040 0x7f91e010dbc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.380+0000 7f91e8400700 1 --2- 172.21.15.78:0/2210090242 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f91e01018d0 0x7f91e0105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.380+0000 7f91e8400700 1 -- 172.21.15.78:0/2210090242 >> 172.21.15.78:0/2210090242 conn(0x7f91e00fc7a0 msgr2=0x7f91e00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.380+0000 7f91e8400700 1 -- 172.21.15.78:0/2210090242 shutdown_connections 2024-03-14T09:55:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.380+0000 7f91e8400700 1 -- 172.21.15.78:0/2210090242 wait complete. 2024-03-14T09:55:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.381+0000 7f91e8400700 1 Processor -- start 2024-03-14T09:55:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.381+0000 7f91e8400700 1 -- start start 2024-03-14T09:55:08.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.382+0000 7f91e8400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91e0100f20 0x7f91e01977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.382+0000 7f91e8400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f91e0106040 0x7f91e0197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.382+0000 7f91e8400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e019afd0 0x7f91e019b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.382+0000 7f91e8400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e019b9c0 con 0x7f91e019afd0 2024-03-14T09:55:08.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.382+0000 7f91e8400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e019bb30 con 0x7f91e0100f20 2024-03-14T09:55:08.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.382+0000 7f91e8400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e019bcf0 con 0x7f91e0106040 2024-03-14T09:55:08.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.382+0000 7f91e6400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e019afd0 0x7f91e019b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.382+0000 7f91e6400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e019afd0 0x7f91e019b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44694/0 (socket says 172.21.15.78:44694) 2024-03-14T09:55:08.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.382+0000 7f91e6400700 1 -- 172.21.15.78:0/1735240760 learned_addr learned my addr 172.21.15.78:0/1735240760 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:08.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.382+0000 7f91e5a00700 1 --2- 172.21.15.78:0/1735240760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91e0100f20 0x7f91e01977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.382+0000 7f91e5000700 1 --2- 172.21.15.78:0/1735240760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f91e0106040 0x7f91e0197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.382+0000 7f91e6400700 1 -- 172.21.15.78:0/1735240760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f91e0106040 msgr2=0x7f91e0197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.382+0000 7f91e6400700 1 --2- 172.21.15.78:0/1735240760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f91e0106040 0x7f91e0197ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.382+0000 7f91e6400700 1 -- 172.21.15.78:0/1735240760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91e0100f20 msgr2=0x7f91e01977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.382+0000 7f91e6400700 1 --2- 172.21.15.78:0/1735240760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91e0100f20 0x7f91e01977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.382+0000 7f91e6400700 1 -- 172.21.15.78:0/1735240760 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d8009d00 con 0x7f91e019afd0 2024-03-14T09:55:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.383+0000 7f91e6400700 1 --2- 172.21.15.78:0/1735240760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e019afd0 0x7f91e019b480 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f91d800b070 tx=0x7f91d800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.383+0000 7f91de000700 1 -- 172.21.15.78:0/1735240760 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d800e040 con 0x7f91e019afd0 2024-03-14T09:55:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.383+0000 7f91de000700 1 -- 172.21.15.78:0/1735240760 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91d801d480 con 0x7f91e019afd0 2024-03-14T09:55:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.383+0000 7f91e8400700 1 -- 172.21.15.78:0/1735240760 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e01a4360 con 0x7f91e019afd0 2024-03-14T09:55:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.383+0000 7f91de000700 1 -- 172.21.15.78:0/1735240760 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d80052a0 con 0x7f91e019afd0 2024-03-14T09:55:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.383+0000 7f91e8400700 1 -- 172.21.15.78:0/1735240760 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e01a4730 con 0x7f91e019afd0 2024-03-14T09:55:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.386+0000 7f91de000700 1 -- 172.21.15.78:0/1735240760 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) v1 ==== 135621+0+0 (secure 0 0 0) 0x7f91d801d5f0 con 0x7f91e019afd0 2024-03-14T09:55:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.387+0000 7f91de000700 1 --2- 172.21.15.78:0/1735240760 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f91c80a0d50 0x7f91c80a3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.387+0000 7f91fbe7db80 1 Processor -- start 2024-03-14T09:55:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.387+0000 7f91de000700 1 -- 172.21.15.78:0/1735240760 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f91d80cdac0 con 0x7f91e019afd0 2024-03-14T09:55:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.388+0000 7f91e5a00700 1 --2- 172.21.15.78:0/1735240760 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f91c80a0d50 0x7f91c80a3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.388+0000 7f91fbe7db80 1 -- start start 2024-03-14T09:55:08.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.388+0000 7f91fbe7db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 0x55b0abfe2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.388+0000 7f91fbe7db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2fa0 0x55b0abfec980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.388+0000 7f91fbe7db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abfef2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.388+0000 7f91fbe7db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0abf3b990 con 0x55b0abfecec0 2024-03-14T09:55:08.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.388+0000 7f91fbe7db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0abf39350 con 0x55b0abfe2680 2024-03-14T09:55:08.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.388+0000 7f91fbe7db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0abf34c40 con 0x55b0abfe2fa0 2024-03-14T09:55:08.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.388+0000 7f91e5a00700 1 --2- 172.21.15.78:0/1735240760 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f91c80a0d50 0x7f91c80a3210 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009c60 tx=0x7f91d4009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.388+0000 7f91e6400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abfef2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.388+0000 7f91e6400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abfef2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44702/0 (socket says 172.21.15.78:44702) 2024-03-14T09:55:08.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.388+0000 7f91e6400700 1 -- 172.21.15.78:0/3697278264 learned_addr learned my addr 172.21.15.78:0/3697278264 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:08.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.388+0000 7f91e5a00700 1 --2- 172.21.15.78:0/3697278264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 0x55b0abfe2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.388+0000 7f91e5000700 1 --2- 172.21.15.78:0/3697278264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2fa0 0x55b0abfec980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.389+0000 7f91e6400700 1 -- 172.21.15.78:0/3697278264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2fa0 msgr2=0x55b0abfec980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.389+0000 7f91e6400700 1 --2- 172.21.15.78:0/3697278264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2fa0 0x55b0abfec980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.389+0000 7f91e6400700 1 -- 172.21.15.78:0/3697278264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 msgr2=0x55b0abfe2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.389+0000 7f91e6400700 1 --2- 172.21.15.78:0/3697278264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 0x55b0abfe2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.389+0000 7f91e6400700 1 -- 172.21.15.78:0/3697278264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0abdb63b0 con 0x55b0abfecec0 2024-03-14T09:55:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.389+0000 7f91e6400700 1 --2- 172.21.15.78:0/3697278264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abfef2b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f91d809a5a0 tx=0x7f91d80099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7c6ff3854232e11 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.389+0000 7f91d2a00700 1 -- 172.21.15.78:0/3697278264 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8022220 con 0x55b0abfecec0 2024-03-14T09:55:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.389+0000 7f91d2a00700 1 -- 172.21.15.78:0/3697278264 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91d809a8a0 con 0x55b0abfecec0 2024-03-14T09:55:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.389+0000 7f91d2a00700 1 -- 172.21.15.78:0/3697278264 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d801dad0 con 0x55b0abfecec0 2024-03-14T09:55:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.390+0000 7f91fbe7db80 1 -- 172.21.15.78:0/3697278264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 msgr2=0x55b0abfef2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.390+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/3697278264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abfef2b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f91d809a5a0 tx=0x7f91d80099e0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.390+0000 7f91fbe7db80 1 -- 172.21.15.78:0/3697278264 shutdown_connections 2024-03-14T09:55:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.390+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/3697278264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 0x55b0abfe2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.390+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/3697278264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abfef2b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.390+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/3697278264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2fa0 0x55b0abfec980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.390+0000 7f91fbe7db80 1 -- 172.21.15.78:0/3697278264 >> 172.21.15.78:0/3697278264 conn(0x55b0abf58220 msgr2=0x55b0abfddaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.390+0000 7f91fbe7db80 1 -- 172.21.15.78:0/3697278264 shutdown_connections 2024-03-14T09:55:08.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.390+0000 7f91fbe7db80 1 -- 172.21.15.78:0/3697278264 wait complete. 2024-03-14T09:55:08.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.391+0000 7f91fbe7db80 1 -- start start 2024-03-14T09:55:08.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.391+0000 7f91fbe7db80 1 Processor -- start 2024-03-14T09:55:08.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91fbe7db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 0x55b0abfec810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91fbe7db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2fa0 0x55b0abfe9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91fbe7db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abfea130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91fbe7db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0abed0d80 con 0x55b0abfecec0 2024-03-14T09:55:08.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91fbe7db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0abed0a20 con 0x55b0abfe2680 2024-03-14T09:55:08.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91fbe7db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0abdb86a0 con 0x55b0abfe2fa0 2024-03-14T09:55:08.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91fbe7db80 10 client.0 _ll_register_callbacks cb 0x55b0abd3c3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:08.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91e5a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 0x55b0abfec810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91e5000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2fa0 0x55b0abfe9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91e5a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 0x55b0abfec810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60836/0 (socket says 172.21.15.78:60836) 2024-03-14T09:55:08.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91e5a00700 1 -- 172.21.15.78:0/3393739905 learned_addr learned my addr 172.21.15.78:0/3393739905 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:08.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91e6400700 1 --2- 172.21.15.78:0/3393739905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abfea130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91e5a00700 1 -- 172.21.15.78:0/3393739905 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2fa0 msgr2=0x55b0abfe9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91e5a00700 1 --2- 172.21.15.78:0/3393739905 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2fa0 0x55b0abfe9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91e5a00700 1 -- 172.21.15.78:0/3393739905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 msgr2=0x55b0abfea130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91e5a00700 1 --2- 172.21.15.78:0/3393739905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abfea130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.392+0000 7f91e5a00700 1 -- 172.21.15.78:0/3393739905 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e01a4730 con 0x55b0abfe2680 2024-03-14T09:55:08.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.393+0000 7f91e5a00700 1 --2- 172.21.15.78:0/3393739905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 0x55b0abfec810 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f91d4003900 tx=0x7f91d40047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e915fbd8a03e574e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.393+0000 7f91c7400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:55:08.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.393+0000 7f91c7400700 1 -- 172.21.15.78:0/3393739905 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4022070 con 0x55b0abfe2680 2024-03-14T09:55:08.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.393+0000 7f91fbe7db80 1 -- 172.21.15.78:0/3393739905 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b0abfd5720 con 0x55b0abfe2680 2024-03-14T09:55:08.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.393+0000 7f91c7400700 1 -- 172.21.15.78:0/3393739905 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f91d401d620 con 0x55b0abfe2680 2024-03-14T09:55:08.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.393+0000 7f91c7400700 1 -- 172.21.15.78:0/3393739905 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91d401d7e0 con 0x55b0abfe2680 2024-03-14T09:55:08.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.393+0000 7f91fbe7db80 10 client.25871 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:08.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.393+0000 7f91fbe7db80 1 -- 172.21.15.78:0/3393739905 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b0abd62b60 con 0x55b0abfe2680 2024-03-14T09:55:08.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.393+0000 7f91c7400700 1 -- 172.21.15.78:0/3393739905 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b0abfd5720 con 0x55b0abfe2680 2024-03-14T09:55:08.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.393+0000 7f91c7400700 1 -- 172.21.15.78:0/3393739905 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f91d4027b90 con 0x55b0abfe2680 2024-03-14T09:55:08.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.394+0000 7f91fbe7db80 10 client.25871 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:08.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.394+0000 7f91fbe7db80 10 client.25871 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:55:08.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.394+0000 7f91fbe7db80 20 client.25871 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:08.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.394+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/3393739905 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55b0abffc8e0 0x55b0abffed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.394+0000 7f91fbe7db80 4 client.25871 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:55:08.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.394+0000 7f91fbe7db80 1 -- 172.21.15.78:0/3393739905 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b0abf2d700 con 0x55b0abffc8e0 2024-03-14T09:55:08.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.394+0000 7f91e5000700 1 --2- 172.21.15.78:0/3393739905 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55b0abffc8e0 0x55b0abffed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.394+0000 7f91c7400700 10 client.25871 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:55:08.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.394+0000 7f91e5000700 1 --2- 172.21.15.78:0/3393739905 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55b0abffc8e0 0x55b0abffed00 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=380b0b5e5a03619a server_cookie=aa46e94081de29ff in_seq=0 out_seq=0 2024-03-14T09:55:08.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.395+0000 7f91c7400700 1 -- 172.21.15.78:0/3393739905 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55b0abf2d700 con 0x55b0abffc8e0 2024-03-14T09:55:08.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.395+0000 7f91c7400700 10 client.25871 handle_command_reply: tid=0 2024-03-14T09:55:08.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.395+0000 7f91fbe7db80 1 client.25871 shutdown 2024-03-14T09:55:08.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.395+0000 7f91fbe7db80 1 -- 172.21.15.78:0/3393739905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 msgr2=0x55b0abfec810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.395+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/3393739905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 0x55b0abfec810 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f91d4003900 tx=0x7f91d40047c0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.395+0000 7f91fbe7db80 1 -- 172.21.15.78:0/3393739905 shutdown_connections 2024-03-14T09:55:08.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.395+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/3393739905 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55b0abffc8e0 0x55b0abffed00 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.395+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/3393739905 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 0x55b0abfec810 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.395+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/3393739905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abfea130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.395+0000 7f91c7400700 0 client.25871 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:55:08.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.395+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/3393739905 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2fa0 0x55b0abfe9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.395+0000 7f91fbe7db80 1 -- 172.21.15.78:0/3393739905 >> 172.21.15.78:0/3393739905 conn(0x55b0abf58220 msgr2=0x55b0abfdd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.395+0000 7f91fbe7db80 1 -- 172.21.15.78:0/3393739905 shutdown_connections 2024-03-14T09:55:08.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.395+0000 7f91fbe7db80 1 -- 172.21.15.78:0/3393739905 wait complete. 2024-03-14T09:55:08.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.396+0000 7f91fbe7db80 20 client.25871 trim_cache size 0 max 16384 2024-03-14T09:55:08.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.408+0000 7f91fbe7db80 1 Processor -- start 2024-03-14T09:55:08.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.409+0000 7f91fbe7db80 1 -- start start 2024-03-14T09:55:08.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.409+0000 7f91fbe7db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2680 0x55b0abfddc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.409+0000 7f91fbe7db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2fa0 0x55b0abff3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.409+0000 7f91fbe7db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abfe6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.409+0000 7f91fbe7db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0abfd7d00 con 0x55b0abfecec0 2024-03-14T09:55:08.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.409+0000 7f91fbe7db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0abfe49b0 con 0x55b0abfe2fa0 2024-03-14T09:55:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.409+0000 7f91fbe7db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0abfde770 con 0x55b0abfe2680 2024-03-14T09:55:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.409+0000 7f91e6400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abfe6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.409+0000 7f91e6400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abfe6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44722/0 (socket says 172.21.15.78:44722) 2024-03-14T09:55:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.409+0000 7f91e6400700 1 -- 172.21.15.78:0/63536794 learned_addr learned my addr 172.21.15.78:0/63536794 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.409+0000 7f91e5a00700 1 --2- 172.21.15.78:0/63536794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2680 0x55b0abfddc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.409+0000 7f91e5000700 1 --2- 172.21.15.78:0/63536794 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2fa0 0x55b0abff3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.409+0000 7f91e6400700 1 -- 172.21.15.78:0/63536794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2680 msgr2=0x55b0abfddc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.409+0000 7f91e6400700 1 --2- 172.21.15.78:0/63536794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2680 0x55b0abfddc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.409+0000 7f91e6400700 1 -- 172.21.15.78:0/63536794 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2fa0 msgr2=0x55b0abff3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.409+0000 7f91e6400700 1 --2- 172.21.15.78:0/63536794 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2fa0 0x55b0abff3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.410+0000 7f91e6400700 1 -- 172.21.15.78:0/63536794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0abefaea0 con 0x55b0abfecec0 2024-03-14T09:55:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.410+0000 7f91e5000700 1 --2- 172.21.15.78:0/63536794 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2fa0 0x55b0abff3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:55:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.410+0000 7f91e6400700 1 --2- 172.21.15.78:0/63536794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abfe6db0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f91d800b780 tx=0x7f91d801dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc3a4700bb4e552 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.410+0000 7f91d3400700 1 -- 172.21.15.78:0/63536794 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8022220 con 0x55b0abfecec0 2024-03-14T09:55:08.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.410+0000 7f91d3400700 1 -- 172.21.15.78:0/63536794 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91d80a2c90 con 0x55b0abfecec0 2024-03-14T09:55:08.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.410+0000 7f91d3400700 1 -- 172.21.15.78:0/63536794 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d80ba0b0 con 0x55b0abfecec0 2024-03-14T09:55:08.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.410+0000 7f91fbe7db80 1 -- 172.21.15.78:0/63536794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 msgr2=0x55b0abfe6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.410+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/63536794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abfe6db0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f91d800b780 tx=0x7f91d801dc70 comp rx=0 tx=0).stop 2024-03-14T09:55:08.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.411+0000 7f91fbe7db80 1 -- 172.21.15.78:0/63536794 shutdown_connections 2024-03-14T09:55:08.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.411+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/63536794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abfe6db0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.411+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/63536794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2680 0x55b0abfddc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.411+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/63536794 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2fa0 0x55b0abff3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.411+0000 7f91fbe7db80 1 -- 172.21.15.78:0/63536794 >> 172.21.15.78:0/63536794 conn(0x55b0abf58220 msgr2=0x55b0abfe8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.411+0000 7f91fbe7db80 1 -- 172.21.15.78:0/63536794 shutdown_connections 2024-03-14T09:55:08.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.411+0000 7f91fbe7db80 1 -- 172.21.15.78:0/63536794 wait complete. 2024-03-14T09:55:08.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.411+0000 7f91fbe7db80 1 -- start start 2024-03-14T09:55:08.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.411+0000 7f91fbe7db80 1 Processor -- start 2024-03-14T09:55:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91fbe7db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 0x55b0abff3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91fbe7db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2fa0 0x55b0abff3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91fbe7db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abff1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91fbe7db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0abff1dd0 con 0x55b0abfecec0 2024-03-14T09:55:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91fbe7db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0abff1f40 con 0x55b0abfe2680 2024-03-14T09:55:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91fbe7db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0abff2100 con 0x55b0abfe2fa0 2024-03-14T09:55:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91fbe7db80 10 client.0 _ll_register_callbacks cb 0x7f91d4004410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91e6400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abff1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91e5a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 0x55b0abff3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91e5a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 0x55b0abff3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60846/0 (socket says 172.21.15.78:60846) 2024-03-14T09:55:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91e5a00700 1 -- 172.21.15.78:0/1983379058 learned_addr learned my addr 172.21.15.78:0/1983379058 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91e5000700 1 --2- 172.21.15.78:0/1983379058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2fa0 0x55b0abff3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91e6400700 1 -- 172.21.15.78:0/1983379058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2fa0 msgr2=0x55b0abff3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91e6400700 1 --2- 172.21.15.78:0/1983379058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2fa0 0x55b0abff3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91e6400700 1 -- 172.21.15.78:0/1983379058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 msgr2=0x55b0abff3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91e6400700 1 --2- 172.21.15.78:0/1983379058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 0x55b0abff3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.412+0000 7f91e6400700 1 -- 172.21.15.78:0/1983379058 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b0abdb63b0 con 0x55b0abfecec0 2024-03-14T09:55:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.413+0000 7f91e6400700 1 --2- 172.21.15.78:0/1983379058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abff1800 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f91d809a830 tx=0x7f91d801db00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358e0b5c2ae05879 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.413+0000 7f91c7400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:55:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.413+0000 7f91c7400700 1 -- 172.21.15.78:0/1983379058 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d80ba2c0 con 0x55b0abfecec0 2024-03-14T09:55:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.413+0000 7f91c7400700 1 -- 172.21.15.78:0/1983379058 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f91d809e6d0 con 0x55b0abfecec0 2024-03-14T09:55:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.413+0000 7f91fbe7db80 1 -- 172.21.15.78:0/1983379058 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f91d809a8a0 con 0x55b0abfecec0 2024-03-14T09:55:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.413+0000 7f91c7400700 1 -- 172.21.15.78:0/1983379058 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91d8022220 con 0x55b0abfecec0 2024-03-14T09:55:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.413+0000 7f91fbe7db80 10 client.15873 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.413+0000 7f91fbe7db80 1 -- 172.21.15.78:0/1983379058 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b0abdbacd0 con 0x55b0abfecec0 2024-03-14T09:55:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.413+0000 7f91c7400700 1 -- 172.21.15.78:0/1983379058 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91d809a8a0 con 0x55b0abfecec0 2024-03-14T09:55:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.413+0000 7f91c7400700 1 -- 172.21.15.78:0/1983379058 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f91d8099d90 con 0x55b0abfecec0 2024-03-14T09:55:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.413+0000 7f91fbe7db80 10 client.15873 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.413+0000 7f91fbe7db80 10 client.15873 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:55:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.413+0000 7f91fbe7db80 20 client.15873 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.413+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/1983379058 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55b0ac04f160 0x55b0ac051550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.413+0000 7f91fbe7db80 4 client.15873 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:55:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.413+0000 7f91fbe7db80 1 -- 172.21.15.78:0/1983379058 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b0abd83cf0 con 0x55b0ac04f160 2024-03-14T09:55:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.414+0000 7f91e5a00700 1 --2- 172.21.15.78:0/1983379058 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55b0ac04f160 0x55b0ac051550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.414+0000 7f91c7400700 10 client.15873 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:55:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.414+0000 7f91e5a00700 1 --2- 172.21.15.78:0/1983379058 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55b0ac04f160 0x55b0ac051550 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=97c8d047a2a5c2f server_cookie=1b96bb789b8179ee in_seq=0 out_seq=0 2024-03-14T09:55:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.414+0000 7f91c7400700 1 -- 172.21.15.78:0/1983379058 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b0abd83cf0 con 0x55b0ac04f160 2024-03-14T09:55:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.414+0000 7f91c7400700 10 client.15873 handle_command_reply: tid=0 2024-03-14T09:55:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.414+0000 7f91fbe7db80 1 client.15873 shutdown 2024-03-14T09:55:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.415+0000 7f91fbe7db80 1 -- 172.21.15.78:0/1983379058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 msgr2=0x55b0abff1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.415+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/1983379058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abff1800 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f91d809a830 tx=0x7f91d801db00 comp rx=0 tx=0).stop 2024-03-14T09:55:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.415+0000 7f91fbe7db80 1 -- 172.21.15.78:0/1983379058 shutdown_connections 2024-03-14T09:55:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.415+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/1983379058 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55b0ac04f160 0x55b0ac051550 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.415+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/1983379058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b0abfe2680 0x55b0abff3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.415+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/1983379058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0abfecec0 0x55b0abff1800 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.415+0000 7f91fbe7db80 1 --2- 172.21.15.78:0/1983379058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b0abfe2fa0 0x55b0abff3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.415+0000 7f91c7400700 0 client.15873 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:55:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.415+0000 7f91fbe7db80 1 -- 172.21.15.78:0/1983379058 >> 172.21.15.78:0/1983379058 conn(0x55b0abf58220 msgr2=0x55b0abfe1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.415+0000 7f91fbe7db80 1 -- 172.21.15.78:0/1983379058 shutdown_connections 2024-03-14T09:55:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.415+0000 7f91fbe7db80 1 -- 172.21.15.78:0/1983379058 wait complete. 2024-03-14T09:55:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.415+0000 7f91fbe7db80 20 client.15873 trim_cache size 0 max 16384 2024-03-14T09:55:08.420 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:55:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.416+0000 7f91e8400700 1 -- 172.21.15.78:0/1735240760 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f91c80a0d50 msgr2=0x7f91c80a3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.416+0000 7f91e8400700 1 --2- 172.21.15.78:0/1735240760 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f91c80a0d50 0x7f91c80a3210 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009c60 tx=0x7f91d4009260 comp rx=0 tx=0).stop 2024-03-14T09:55:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.416+0000 7f91e8400700 1 -- 172.21.15.78:0/1735240760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e019afd0 msgr2=0x7f91e019b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.416+0000 7f91e8400700 1 --2- 172.21.15.78:0/1735240760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e019afd0 0x7f91e019b480 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f91d800b070 tx=0x7f91d800bda0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.416+0000 7f91e8400700 1 -- 172.21.15.78:0/1735240760 shutdown_connections 2024-03-14T09:55:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.416+0000 7f91e8400700 1 --2- 172.21.15.78:0/1735240760 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f91c80a0d50 0x7f91c80a3210 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.416+0000 7f91e8400700 1 --2- 172.21.15.78:0/1735240760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91e0100f20 0x7f91e01977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.416+0000 7f91e8400700 1 --2- 172.21.15.78:0/1735240760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91e019afd0 0x7f91e019b480 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.416+0000 7f91e8400700 1 --2- 172.21.15.78:0/1735240760 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f91e0106040 0x7f91e0197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.416+0000 7f91e8400700 1 -- 172.21.15.78:0/1735240760 >> 172.21.15.78:0/1735240760 conn(0x7f91e00fc7a0 msgr2=0x7f91e00feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.416+0000 7f91e8400700 1 -- 172.21.15.78:0/1735240760 shutdown_connections 2024-03-14T09:55:08.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.416+0000 7f91e8400700 1 -- 172.21.15.78:0/1735240760 wait complete. 2024-03-14T09:55:08.434 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T09:55:08.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.632+0000 7f0795200700 1 -- 172.21.15.78:0/2431666978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0790103af0 msgr2=0x7f0790107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.632+0000 7f0795200700 1 --2- 172.21.15.78:0/2431666978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0790103af0 0x7f0790107b30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0778009990 tx=0x7f0778009ca0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.632+0000 7f0795200700 1 -- 172.21.15.78:0/2431666978 shutdown_connections 2024-03-14T09:55:08.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.632+0000 7f0795200700 1 --2- 172.21.15.78:0/2431666978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0790108260 0x7f079010fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.632+0000 7f0795200700 1 --2- 172.21.15.78:0/2431666978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0790103140 0x7f0790103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.632+0000 7f0795200700 1 --2- 172.21.15.78:0/2431666978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0790103af0 0x7f0790107b30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.632+0000 7f0795200700 1 -- 172.21.15.78:0/2431666978 >> 172.21.15.78:0/2431666978 conn(0x7f07900fe980 msgr2=0x7f0790100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.632+0000 7f0795200700 1 -- 172.21.15.78:0/2431666978 shutdown_connections 2024-03-14T09:55:08.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.632+0000 7f0795200700 1 -- 172.21.15.78:0/2431666978 wait complete. 2024-03-14T09:55:08.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.633+0000 7f0795200700 1 Processor -- start 2024-03-14T09:55:08.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.634+0000 7f0795200700 1 -- start start 2024-03-14T09:55:08.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.634+0000 7f0795200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0790103140 0x7f079019f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.634+0000 7f0795200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0790108260 0x7f079019faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.634+0000 7f0795200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07901996e0 0x7f0790199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.634+0000 7f0795200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079019a160 con 0x7f07901996e0 2024-03-14T09:55:08.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.634+0000 7f0795200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079019a2a0 con 0x7f0790103140 2024-03-14T09:55:08.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.634+0000 7f0795200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079019a460 con 0x7f0790108260 2024-03-14T09:55:08.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.634+0000 7f078de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0790103140 0x7f079019f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.634+0000 7f078e800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07901996e0 0x7f0790199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.634+0000 7f078de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0790103140 0x7f079019f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60862/0 (socket says 172.21.15.78:60862) 2024-03-14T09:55:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.634+0000 7f078de00700 1 -- 172.21.15.78:0/200737161 learned_addr learned my addr 172.21.15.78:0/200737161 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.634+0000 7f0785400700 1 --2- 172.21.15.78:0/200737161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0790108260 0x7f079019faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.635+0000 7f078e800700 1 -- 172.21.15.78:0/200737161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0790108260 msgr2=0x7f079019faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.635+0000 7f078e800700 1 --2- 172.21.15.78:0/200737161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0790108260 0x7f079019faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.635+0000 7f078e800700 1 -- 172.21.15.78:0/200737161 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0790103140 msgr2=0x7f079019f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.635+0000 7f078e800700 1 --2- 172.21.15.78:0/200737161 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0790103140 0x7f079019f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.635+0000 7f078e800700 1 -- 172.21.15.78:0/200737161 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07780095f0 con 0x7f07901996e0 2024-03-14T09:55:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.635+0000 7f078e800700 1 --2- 172.21.15.78:0/200737161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07901996e0 0x7f0790199b90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f078800efc0 tx=0x7f078800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.635+0000 7f0787400700 1 -- 172.21.15.78:0/200737161 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0788018070 con 0x7f07901996e0 2024-03-14T09:55:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.635+0000 7f0795200700 1 -- 172.21.15.78:0/200737161 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07901a6420 con 0x7f07901996e0 2024-03-14T09:55:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.635+0000 7f0787400700 1 -- 172.21.15.78:0/200737161 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f078800f040 con 0x7f07901996e0 2024-03-14T09:55:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.635+0000 7f0787400700 1 -- 172.21.15.78:0/200737161 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078800cd40 con 0x7f07901996e0 2024-03-14T09:55:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.635+0000 7f0795200700 1 -- 172.21.15.78:0/200737161 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07901a6930 con 0x7f07901996e0 2024-03-14T09:55:08.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.638+0000 7f0787400700 1 -- 172.21.15.78:0/200737161 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) v1 ==== 135621+0+0 (secure 0 0 0) 0x7f0788003a00 con 0x7f07901996e0 2024-03-14T09:55:08.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.639+0000 7f0787400700 1 --2- 172.21.15.78:0/200737161 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f07740a0da0 0x7f07740a3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.640+0000 7f0787400700 1 -- 172.21.15.78:0/200737161 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f07880cc570 con 0x7f07901996e0 2024-03-14T09:55:08.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.640+0000 7f078de00700 1 --2- 172.21.15.78:0/200737161 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f07740a0da0 0x7f07740a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.640+0000 7f07a8cd1b80 1 Processor -- start 2024-03-14T09:55:08.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.640+0000 7f07a8cd1b80 1 -- start start 2024-03-14T09:55:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.640+0000 7f07a8cd1b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53e96a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f078de00700 1 --2- 172.21.15.78:0/200737161 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f07740a0da0 0x7f07740a3260 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0780009d20 tx=0x7f0780009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f07a8cd1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f07a8cd1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 0x55dd53ea32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f07a8cd1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd53def990 con 0x55dd53ea0ec0 2024-03-14T09:55:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f07a8cd1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd53ded350 con 0x55dd53e96fa0 2024-03-14T09:55:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f07a8cd1b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd53de8c40 con 0x55dd53e96680 2024-03-14T09:55:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f078de00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53e96a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f0785400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f078de00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53e96a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:53126/0 (socket says 172.21.15.78:53126) 2024-03-14T09:55:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f078e800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 0x55dd53ea32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f0785400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60866/0 (socket says 172.21.15.78:60866) 2024-03-14T09:55:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f078de00700 1 -- 172.21.15.78:0/2773613193 learned_addr learned my addr 172.21.15.78:0/2773613193 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f078de00700 1 -- 172.21.15.78:0/2773613193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 msgr2=0x55dd53ea0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f078de00700 1 --2- 172.21.15.78:0/2773613193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f078de00700 1 -- 172.21.15.78:0/2773613193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 msgr2=0x55dd53ea32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f078de00700 1 --2- 172.21.15.78:0/2773613193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 0x55dd53ea32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f078de00700 1 -- 172.21.15.78:0/2773613193 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd53c6a3b0 con 0x55dd53e96680 2024-03-14T09:55:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.641+0000 7f078e800700 1 --2- 172.21.15.78:0/2773613193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 0x55dd53ea32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.642+0000 7f078de00700 1 --2- 172.21.15.78:0/2773613193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53e96a60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f078000e540 tx=0x7f0780014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea39d851ce6dfe73 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.642+0000 7f077f400700 1 -- 172.21.15.78:0/2773613193 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0780026430 con 0x55dd53e96680 2024-03-14T09:55:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.642+0000 7f077f400700 1 -- 172.21.15.78:0/2773613193 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0780021070 con 0x55dd53e96680 2024-03-14T09:55:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.642+0000 7f077f400700 1 -- 172.21.15.78:0/2773613193 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078001d720 con 0x55dd53e96680 2024-03-14T09:55:08.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.643+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/2773613193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 msgr2=0x55dd53e96a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.643+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/2773613193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53e96a60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f078000e540 tx=0x7f0780014000 comp rx=0 tx=0).stop 2024-03-14T09:55:08.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.643+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/2773613193 shutdown_connections 2024-03-14T09:55:08.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.643+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/2773613193 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 0x55dd53ea32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.643+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/2773613193 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53e96a60 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.643+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/2773613193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.643+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/2773613193 >> 172.21.15.78:0/2773613193 conn(0x55dd53e0c220 msgr2=0x55dd53e91aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.643+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/2773613193 shutdown_connections 2024-03-14T09:55:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.643+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/2773613193 wait complete. 2024-03-14T09:55:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.644+0000 7f07a8cd1b80 1 -- start start 2024-03-14T09:55:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.644+0000 7f07a8cd1b80 1 Processor -- start 2024-03-14T09:55:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.644+0000 7f07a8cd1b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53ea08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.644+0000 7f07a8cd1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53e96fa0 0x55dd53e9dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.644+0000 7f07a8cd1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53ea0ec0 0x55dd53e9e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.644+0000 7f07a8cd1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd53d84d80 con 0x55dd53e96fa0 2024-03-14T09:55:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.644+0000 7f07a8cd1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd53d84a20 con 0x55dd53ea0ec0 2024-03-14T09:55:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.644+0000 7f07a8cd1b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd53c6c6a0 con 0x55dd53e96680 2024-03-14T09:55:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.644+0000 7f07a8cd1b80 10 client.0 _ll_register_callbacks cb 0x55dd53bf03a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.645+0000 7f0785400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53e96fa0 0x55dd53e9dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.645+0000 7f078e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53ea0ec0 0x55dd53e9e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.645+0000 7f078e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53ea0ec0 0x55dd53e9e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60878/0 (socket says 172.21.15.78:60878) 2024-03-14T09:55:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.645+0000 7f078e800700 1 -- 172.21.15.78:0/3487762815 learned_addr learned my addr 172.21.15.78:0/3487762815 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.645+0000 7f078de00700 1 --2- 172.21.15.78:0/3487762815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53ea08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.645+0000 7f078e800700 1 -- 172.21.15.78:0/3487762815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 msgr2=0x55dd53ea08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.645+0000 7f078e800700 1 --2- 172.21.15.78:0/3487762815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53ea08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.645+0000 7f078e800700 1 -- 172.21.15.78:0/3487762815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53e96fa0 msgr2=0x55dd53e9dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.645+0000 7f078e800700 1 --2- 172.21.15.78:0/3487762815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53e96fa0 0x55dd53e9dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.645+0000 7f078e800700 1 -- 172.21.15.78:0/3487762815 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07901a6930 con 0x55dd53ea0ec0 2024-03-14T09:55:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.645+0000 7f0785400700 1 --2- 172.21.15.78:0/3487762815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53e96fa0 0x55dd53e9dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.645+0000 7f078e800700 1 --2- 172.21.15.78:0/3487762815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53ea0ec0 0x55dd53e9e370 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0788095890 tx=0x7f0788098d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa2f04b74103bda7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.645+0000 7f0773e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:55:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.646+0000 7f0773e00700 1 -- 172.21.15.78:0/3487762815 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078801b320 con 0x55dd53ea0ec0 2024-03-14T09:55:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.646+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3487762815 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dd53e89720 con 0x55dd53ea0ec0 2024-03-14T09:55:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.646+0000 7f0773e00700 1 -- 172.21.15.78:0/3487762815 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f0788099a90 con 0x55dd53ea0ec0 2024-03-14T09:55:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.646+0000 7f0773e00700 1 -- 172.21.15.78:0/3487762815 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0788099390 con 0x55dd53ea0ec0 2024-03-14T09:55:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.646+0000 7f07a8cd1b80 10 client.25889 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.646+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3487762815 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f0780017070 con 0x55dd53ea0ec0 2024-03-14T09:55:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.646+0000 7f0773e00700 1 -- 172.21.15.78:0/3487762815 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dd53e89720 con 0x55dd53ea0ec0 2024-03-14T09:55:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.646+0000 7f0773e00700 1 -- 172.21.15.78:0/3487762815 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0788098880 con 0x55dd53ea0ec0 2024-03-14T09:55:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.646+0000 7f07a8cd1b80 10 client.25889 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.646+0000 7f07a8cd1b80 10 client.25889 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:55:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.646+0000 7f07a8cd1b80 20 client.25889 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.646+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/3487762815 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55dd53eb0a70 0x55dd53eb2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.646+0000 7f07a8cd1b80 4 client.25889 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:55:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.646+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3487762815 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dd53de1700 con 0x55dd53eb0a70 2024-03-14T09:55:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.647+0000 7f078de00700 1 --2- 172.21.15.78:0/3487762815 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55dd53eb0a70 0x55dd53eb2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.647+0000 7f078de00700 1 --2- 172.21.15.78:0/3487762815 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55dd53eb0a70 0x55dd53eb2e60 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8be4c59681428fe server_cookie=b7725c9e0c766908 in_seq=0 out_seq=0 2024-03-14T09:55:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.647+0000 7f0773e00700 10 client.25889 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.648+0000 7f0773e00700 1 -- 172.21.15.78:0/3487762815 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55dd53de1700 con 0x55dd53eb0a70 2024-03-14T09:55:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.648+0000 7f0773e00700 10 client.25889 handle_command_reply: tid=0 2024-03-14T09:55:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.648+0000 7f07a8cd1b80 1 client.25889 shutdown 2024-03-14T09:55:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.648+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3487762815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53ea0ec0 msgr2=0x55dd53e9e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.648+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/3487762815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53ea0ec0 0x55dd53e9e370 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0788095890 tx=0x7f0788098d70 comp rx=0 tx=0).stop 2024-03-14T09:55:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.649+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3487762815 shutdown_connections 2024-03-14T09:55:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.649+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/3487762815 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55dd53eb0a70 0x55dd53eb2e60 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.649+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/3487762815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53ea0ec0 0x55dd53e9e370 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.649+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/3487762815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53ea08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.649+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/3487762815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53e96fa0 0x55dd53e9dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.649+0000 7f0773e00700 0 client.25889 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.649+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3487762815 >> 172.21.15.78:0/3487762815 conn(0x55dd53e0c220 msgr2=0x55dd53e91690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.649+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3487762815 shutdown_connections 2024-03-14T09:55:08.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.649+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3487762815 wait complete. 2024-03-14T09:55:08.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.649+0000 7f07a8cd1b80 20 client.25889 trim_cache size 0 max 16384 2024-03-14T09:55:08.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.662+0000 7f07a8cd1b80 1 Processor -- start 2024-03-14T09:55:08.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.662+0000 7f07a8cd1b80 1 -- start start 2024-03-14T09:55:08.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.663+0000 7f07a8cd1b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53eb7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.663+0000 7f07a8cd1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.663+0000 7f07a8cd1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 0x55dd53ea0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.663+0000 7f07a8cd1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd53e9e970 con 0x55dd53ea0ec0 2024-03-14T09:55:08.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.663+0000 7f07a8cd1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd53e92e30 con 0x55dd53e96fa0 2024-03-14T09:55:08.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.663+0000 7f07a8cd1b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd53e93060 con 0x55dd53e96680 2024-03-14T09:55:08.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.663+0000 7f0785400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.663+0000 7f078e800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 0x55dd53ea0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.663+0000 7f0785400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60892/0 (socket says 172.21.15.78:60892) 2024-03-14T09:55:08.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.663+0000 7f0785400700 1 -- 172.21.15.78:0/3222658280 learned_addr learned my addr 172.21.15.78:0/3222658280 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:08.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.663+0000 7f078de00700 1 --2- 172.21.15.78:0/3222658280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53eb7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.663+0000 7f0785400700 1 -- 172.21.15.78:0/3222658280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 msgr2=0x55dd53eb7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.663+0000 7f0785400700 1 --2- 172.21.15.78:0/3222658280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53eb7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.663+0000 7f0785400700 1 -- 172.21.15.78:0/3222658280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 msgr2=0x55dd53ea0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.663+0000 7f0785400700 1 --2- 172.21.15.78:0/3222658280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 0x55dd53ea0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.663+0000 7f0785400700 1 -- 172.21.15.78:0/3222658280 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd53c16b60 con 0x55dd53e96fa0 2024-03-14T09:55:08.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.664+0000 7f0785400700 1 --2- 172.21.15.78:0/3222658280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea7b50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f079006a3a0 tx=0x7f0778005350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8368849af813f824 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.664+0000 7f077fe00700 1 -- 172.21.15.78:0/3222658280 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077801d070 con 0x55dd53e96fa0 2024-03-14T09:55:08.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.664+0000 7f077fe00700 1 -- 172.21.15.78:0/3222658280 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f077801e040 con 0x55dd53e96fa0 2024-03-14T09:55:08.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.664+0000 7f077fe00700 1 -- 172.21.15.78:0/3222658280 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0778004d60 con 0x55dd53e96fa0 2024-03-14T09:55:08.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.665+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3222658280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 msgr2=0x55dd53ea7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.665+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/3222658280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea7b50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f079006a3a0 tx=0x7f0778005350 comp rx=0 tx=0).stop 2024-03-14T09:55:08.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.665+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3222658280 shutdown_connections 2024-03-14T09:55:08.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.665+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/3222658280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 0x55dd53ea0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.665+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/3222658280 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53eb7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.665+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/3222658280 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea7b50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.665+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3222658280 >> 172.21.15.78:0/3222658280 conn(0x55dd53e0c220 msgr2=0x55dd53e90180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.665+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3222658280 shutdown_connections 2024-03-14T09:55:08.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.665+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3222658280 wait complete. 2024-03-14T09:55:08.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.665+0000 7f07a8cd1b80 1 -- start start 2024-03-14T09:55:08.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.665+0000 7f07a8cd1b80 1 Processor -- start 2024-03-14T09:55:08.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.666+0000 7f07a8cd1b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53ea5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.666+0000 7f07a8cd1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.666+0000 7f07a8cd1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 0x55dd53f01530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.666+0000 7f07a8cd1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd53ea6930 con 0x55dd53ea0ec0 2024-03-14T09:55:08.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.666+0000 7f07a8cd1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd53f01a70 con 0x55dd53e96fa0 2024-03-14T09:55:08.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.666+0000 7f07a8cd1b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd53f01bb0 con 0x55dd53e96680 2024-03-14T09:55:08.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.666+0000 7f07a8cd1b80 10 client.0 _ll_register_callbacks cb 0x7f0788099160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:08.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.666+0000 7f0785400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.666+0000 7f078e800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 0x55dd53f01530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.666+0000 7f0785400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60908/0 (socket says 172.21.15.78:60908) 2024-03-14T09:55:08.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.666+0000 7f078de00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53ea5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.666+0000 7f0785400700 1 -- 172.21.15.78:0/3114408938 learned_addr learned my addr 172.21.15.78:0/3114408938 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:08.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.667+0000 7f0785400700 1 -- 172.21.15.78:0/3114408938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 msgr2=0x55dd53ea5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.667+0000 7f0785400700 1 --2- 172.21.15.78:0/3114408938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53ea5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.667+0000 7f0785400700 1 -- 172.21.15.78:0/3114408938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 msgr2=0x55dd53f01530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:08.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.667+0000 7f0785400700 1 --2- 172.21.15.78:0/3114408938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 0x55dd53f01530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.667+0000 7f0785400700 1 -- 172.21.15.78:0/3114408938 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0790110650 con 0x55dd53e96fa0 2024-03-14T09:55:08.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.667+0000 7f078de00700 1 --2- 172.21.15.78:0/3114408938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53ea5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:08.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.667+0000 7f078e800700 1 --2- 172.21.15.78:0/3114408938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 0x55dd53f01530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:08.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.667+0000 7f0785400700 1 --2- 172.21.15.78:0/3114408938 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0778006010 tx=0x7f077800b630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c85b0416fd6a51a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:08.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.667+0000 7f0773e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:55:08.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.667+0000 7f0773e00700 1 -- 172.21.15.78:0/3114408938 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077801d070 con 0x55dd53e96fa0 2024-03-14T09:55:08.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.667+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3114408938 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0780021070 con 0x55dd53e96fa0 2024-03-14T09:55:08.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.667+0000 7f0773e00700 1 -- 172.21.15.78:0/3114408938 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f077800bb60 con 0x55dd53e96fa0 2024-03-14T09:55:08.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.668+0000 7f0773e00700 1 -- 172.21.15.78:0/3114408938 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0778004680 con 0x55dd53e96fa0 2024-03-14T09:55:08.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.668+0000 7f07a8cd1b80 10 client.25895 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:08.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.668+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3114408938 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55dd53daeea0 con 0x55dd53e96fa0 2024-03-14T09:55:08.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.668+0000 7f0773e00700 1 -- 172.21.15.78:0/3114408938 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0780021070 con 0x55dd53e96fa0 2024-03-14T09:55:08.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.668+0000 7f0773e00700 1 -- 172.21.15.78:0/3114408938 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0778037440 con 0x55dd53e96fa0 2024-03-14T09:55:08.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.668+0000 7f07a8cd1b80 10 client.25895 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:08.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.668+0000 7f07a8cd1b80 10 client.25895 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:55:08.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.668+0000 7f07a8cd1b80 20 client.25895 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:08.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.668+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/3114408938 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55dd53f02f00 0x55dd53f05320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:08.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.668+0000 7f07a8cd1b80 4 client.25895 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:55:08.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.668+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3114408938 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dd53c37cf0 con 0x55dd53f02f00 2024-03-14T09:55:08.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.668+0000 7f078de00700 1 --2- 172.21.15.78:0/3114408938 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55dd53f02f00 0x55dd53f05320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:08.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.669+0000 7f078de00700 1 --2- 172.21.15.78:0/3114408938 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55dd53f02f00 0x55dd53f05320 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f530f9526fc116fd server_cookie=18986005152eab53 in_seq=0 out_seq=0 2024-03-14T09:55:08.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.669+0000 7f0773e00700 10 client.25895 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:08.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.669+0000 7f0773e00700 1 -- 172.21.15.78:0/3114408938 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dd53c37cf0 con 0x55dd53f02f00 2024-03-14T09:55:08.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.669+0000 7f0773e00700 10 client.25895 handle_command_reply: tid=0 2024-03-14T09:55:08.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.669+0000 7f07a8cd1b80 1 client.25895 shutdown 2024-03-14T09:55:08.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.669+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3114408938 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 msgr2=0x55dd53ea6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.669+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/3114408938 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0778006010 tx=0x7f077800b630 comp rx=0 tx=0).stop 2024-03-14T09:55:08.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.669+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3114408938 shutdown_connections 2024-03-14T09:55:08.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.669+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/3114408938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55dd53ea0ec0 0x55dd53f01530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.669+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/3114408938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55dd53e96680 0x55dd53ea5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.669+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/3114408938 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55dd53f02f00 0x55dd53f05320 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.669+0000 7f07a8cd1b80 1 --2- 172.21.15.78:0/3114408938 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55dd53e96fa0 0x55dd53ea6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.669+0000 7f0773e00700 0 client.25895 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:08.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.669+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3114408938 >> 172.21.15.78:0/3114408938 conn(0x55dd53e0c220 msgr2=0x55dd53e95560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.670+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3114408938 shutdown_connections 2024-03-14T09:55:08.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.670+0000 7f07a8cd1b80 1 -- 172.21.15.78:0/3114408938 wait complete. 2024-03-14T09:55:08.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.670+0000 7f07a8cd1b80 20 client.25895 trim_cache size 0 max 16384 2024-03-14T09:55:08.675 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:55:08.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.670+0000 7f0795200700 1 -- 172.21.15.78:0/200737161 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f07740a0da0 msgr2=0x7f07740a3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.670+0000 7f0795200700 1 --2- 172.21.15.78:0/200737161 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f07740a0da0 0x7f07740a3260 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0780009d20 tx=0x7f0780009260 comp rx=0 tx=0).stop 2024-03-14T09:55:08.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.670+0000 7f0795200700 1 -- 172.21.15.78:0/200737161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07901996e0 msgr2=0x7f0790199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:08.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.670+0000 7f0795200700 1 --2- 172.21.15.78:0/200737161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07901996e0 0x7f0790199b90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f078800efc0 tx=0x7f078800c3a0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.670+0000 7f0795200700 1 -- 172.21.15.78:0/200737161 shutdown_connections 2024-03-14T09:55:08.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.670+0000 7f0795200700 1 --2- 172.21.15.78:0/200737161 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f07740a0da0 0x7f07740a3260 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.670+0000 7f0795200700 1 --2- 172.21.15.78:0/200737161 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0790103140 0x7f079019f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.670+0000 7f0795200700 1 --2- 172.21.15.78:0/200737161 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f07901996e0 0x7f0790199b90 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.670+0000 7f0795200700 1 --2- 172.21.15.78:0/200737161 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0790108260 0x7f079019faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:08.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.670+0000 7f0795200700 1 -- 172.21.15.78:0/200737161 >> 172.21.15.78:0/200737161 conn(0x7f07900fe980 msgr2=0x7f07900ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:08.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.671+0000 7f0795200700 1 -- 172.21.15.78:0/200737161 shutdown_connections 2024-03-14T09:55:08.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:08.671+0000 7f0795200700 1 -- 172.21.15.78:0/200737161 wait complete. 2024-03-14T09:55:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:08 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/992183168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:55:09.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:08 smithi132 ceph-mon[47959]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-14T09:55:09.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:08 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/992183168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:55:09.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:08 smithi078 ceph-mon[38143]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-14T09:55:09.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:08 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/992183168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:55:09.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:08 smithi152 ceph-mon[48279]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-14T09:55:09.693 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T09:55:09.694 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T09:55:09.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.919+0000 7fdf34c00700 1 -- 172.21.15.78:0/3363449129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf30100f30 msgr2=0x7fdf30101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:09.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.919+0000 7fdf34c00700 1 --2- 172.21.15.78:0/3363449129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf30100f30 0x7fdf30101310 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdf20009990 tx=0x7fdf20009ca0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.920+0000 7fdf34c00700 1 -- 172.21.15.78:0/3363449129 shutdown_connections 2024-03-14T09:55:09.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.920+0000 7fdf34c00700 1 --2- 172.21.15.78:0/3363449129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf30100f30 0x7fdf30101310 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.920+0000 7fdf34c00700 1 --2- 172.21.15.78:0/3363449129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf30106050 0x7fdf3010dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.920+0000 7fdf34c00700 1 --2- 172.21.15.78:0/3363449129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf301018e0 0x7fdf30105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.920+0000 7fdf34c00700 1 -- 172.21.15.78:0/3363449129 >> 172.21.15.78:0/3363449129 conn(0x7fdf300fc7b0 msgr2=0x7fdf300febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:09.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.920+0000 7fdf34c00700 1 -- 172.21.15.78:0/3363449129 shutdown_connections 2024-03-14T09:55:09.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.920+0000 7fdf34c00700 1 -- 172.21.15.78:0/3363449129 wait complete. 2024-03-14T09:55:09.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.921+0000 7fdf34c00700 1 Processor -- start 2024-03-14T09:55:09.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.921+0000 7fdf34c00700 1 -- start start 2024-03-14T09:55:09.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.921+0000 7fdf34c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf30100f30 0x7fdf301975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.921+0000 7fdf34c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf30106050 0x7fdf30197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.921+0000 7fdf34c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf3019bed0 0x7fdf3019c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.921+0000 7fdf34c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3019c950 con 0x7fdf30100f30 2024-03-14T09:55:09.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.921+0000 7fdf34c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3019cac0 con 0x7fdf3019bed0 2024-03-14T09:55:09.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.921+0000 7fdf34c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3019cc80 con 0x7fdf30106050 2024-03-14T09:55:09.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.922+0000 7fdf2de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf30100f30 0x7fdf301975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.922+0000 7fdf2e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf3019bed0 0x7fdf3019c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.922+0000 7fdf2de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf30100f30 0x7fdf301975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44804/0 (socket says 172.21.15.78:44804) 2024-03-14T09:55:09.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.922+0000 7fdf2de00700 1 -- 172.21.15.78:0/3302471039 learned_addr learned my addr 172.21.15.78:0/3302471039 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:09.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.922+0000 7fdf2d400700 1 --2- 172.21.15.78:0/3302471039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf30106050 0x7fdf30197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.922+0000 7fdf2e800700 1 -- 172.21.15.78:0/3302471039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf30106050 msgr2=0x7fdf30197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:09.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.922+0000 7fdf2e800700 1 --2- 172.21.15.78:0/3302471039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf30106050 0x7fdf30197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.922+0000 7fdf2e800700 1 -- 172.21.15.78:0/3302471039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf30100f30 msgr2=0x7fdf301975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:09.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.922+0000 7fdf2e800700 1 --2- 172.21.15.78:0/3302471039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf30100f30 0x7fdf301975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.922+0000 7fdf2e800700 1 -- 172.21.15.78:0/3302471039 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c009520 con 0x7fdf3019bed0 2024-03-14T09:55:09.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.922+0000 7fdf2de00700 1 --2- 172.21.15.78:0/3302471039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf30100f30 0x7fdf301975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:09.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.922+0000 7fdf2e800700 1 --2- 172.21.15.78:0/3302471039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf3019bed0 0x7fdf3019c350 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800efb0 tx=0x7fdf2800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:09.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.923+0000 7fdf26a00700 1 -- 172.21.15.78:0/3302471039 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28003a00 con 0x7fdf3019bed0 2024-03-14T09:55:09.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.923+0000 7fdf26a00700 1 -- 172.21.15.78:0/3302471039 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf2800f040 con 0x7fdf3019bed0 2024-03-14T09:55:09.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.923+0000 7fdf34c00700 1 -- 172.21.15.78:0/3302471039 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf28009c40 con 0x7fdf3019bed0 2024-03-14T09:55:09.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.923+0000 7fdf26a00700 1 -- 172.21.15.78:0/3302471039 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28004540 con 0x7fdf3019bed0 2024-03-14T09:55:09.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.923+0000 7fdf34c00700 1 -- 172.21.15.78:0/3302471039 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf200095f0 con 0x7fdf3019bed0 2024-03-14T09:55:09.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.926+0000 7fdf26a00700 1 -- 172.21.15.78:0/3302471039 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) v1 ==== 135621+0+0 (secure 0 0 0) 0x7fdf28003bf0 con 0x7fdf3019bed0 2024-03-14T09:55:09.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.927+0000 7fdf4864eb80 1 Processor -- start 2024-03-14T09:55:09.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.927+0000 7fdf4864eb80 1 -- start start 2024-03-14T09:55:09.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.927+0000 7fdf4864eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab28700 0x55635ab28ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.927+0000 7fdf4864eb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab29020 0x55635ab36e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.927+0000 7fdf4864eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab37350 0x55635ab39740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.927+0000 7fdf4864eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55635aa81a10 con 0x55635ab37350 2024-03-14T09:55:09.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.927+0000 7fdf4864eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55635aa7f3d0 con 0x55635ab28700 2024-03-14T09:55:09.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.927+0000 7fdf4864eb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55635aa7acc0 con 0x55635ab29020 2024-03-14T09:55:09.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.928+0000 7fdf2d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab28700 0x55635ab28ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.928+0000 7fdf2d400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab37350 0x55635ab39740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.928+0000 7fdf2d400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab37350 0x55635ab39740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44812/0 (socket says 172.21.15.78:44812) 2024-03-14T09:55:09.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.928+0000 7fdf2d400700 1 -- 172.21.15.78:0/2555984410 learned_addr learned my addr 172.21.15.78:0/2555984410 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:09.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.928+0000 7fdf2de00700 1 --2- 172.21.15.78:0/2555984410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab29020 0x55635ab36e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.928+0000 7fdf2de00700 1 -- 172.21.15.78:0/2555984410 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab28700 msgr2=0x55635ab28ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:09.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.928+0000 7fdf2de00700 1 --2- 172.21.15.78:0/2555984410 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab28700 0x55635ab28ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.928+0000 7fdf2de00700 1 -- 172.21.15.78:0/2555984410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab37350 msgr2=0x55635ab39740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:09.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.928+0000 7fdf2de00700 1 --2- 172.21.15.78:0/2555984410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab37350 0x55635ab39740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.928+0000 7fdf2de00700 1 -- 172.21.15.78:0/2555984410 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55635a8fc430 con 0x55635ab29020 2024-03-14T09:55:09.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.928+0000 7fdf26a00700 1 --2- 172.21.15.78:0/3302471039 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fdf140a0da0 0x7fdf140a3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.928+0000 7fdf2de00700 1 --2- 172.21.15.78:0/2555984410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab29020 0x55635ab36e10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdf200098e0 tx=0x7fdf2000b610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4663242c942c8929 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:09.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.928+0000 7fdf26a00700 1 -- 172.21.15.78:0/3302471039 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fdf280cc5f0 con 0x7fdf3019bed0 2024-03-14T09:55:09.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.929+0000 7fdf2d400700 1 --2- 172.21.15.78:0/3302471039 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fdf140a0da0 0x7fdf140a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.929+0000 7fdf1b000700 1 -- 172.21.15.78:0/2555984410 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2001d070 con 0x55635ab29020 2024-03-14T09:55:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.929+0000 7fdf1b000700 1 -- 172.21.15.78:0/2555984410 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf2001e040 con 0x55635ab29020 2024-03-14T09:55:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.929+0000 7fdf1b000700 1 -- 172.21.15.78:0/2555984410 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf20004ba0 con 0x55635ab29020 2024-03-14T09:55:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.930+0000 7fdf2d400700 1 --2- 172.21.15.78:0/3302471039 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fdf140a0da0 0x7fdf140a3260 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c009fd0 tx=0x7fdf1c0047c0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.930+0000 7fdf4864eb80 1 -- 172.21.15.78:0/2555984410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab29020 msgr2=0x55635ab36e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.930+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/2555984410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab29020 0x55635ab36e10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdf200098e0 tx=0x7fdf2000b610 comp rx=0 tx=0).stop 2024-03-14T09:55:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.930+0000 7fdf4864eb80 1 -- 172.21.15.78:0/2555984410 shutdown_connections 2024-03-14T09:55:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.930+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/2555984410 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab28700 0x55635ab28ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.930+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/2555984410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab37350 0x55635ab39740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.930+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/2555984410 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab29020 0x55635ab36e10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.930+0000 7fdf4864eb80 1 -- 172.21.15.78:0/2555984410 >> 172.21.15.78:0/2555984410 conn(0x55635aa9e2a0 msgr2=0x55635ab23b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:09.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.930+0000 7fdf4864eb80 1 -- 172.21.15.78:0/2555984410 shutdown_connections 2024-03-14T09:55:09.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.930+0000 7fdf4864eb80 1 -- 172.21.15.78:0/2555984410 wait complete. 2024-03-14T09:55:09.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.930+0000 7fdf4864eb80 1 -- start start 2024-03-14T09:55:09.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.931+0000 7fdf4864eb80 1 Processor -- start 2024-03-14T09:55:09.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf4864eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab28700 0x55635ab36cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf4864eb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab29020 0x55635ab34100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf4864eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab37350 0x55635ab346a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf4864eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55635aa16e00 con 0x55635ab28700 2024-03-14T09:55:09.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf4864eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55635aa16aa0 con 0x55635ab37350 2024-03-14T09:55:09.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf4864eb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55635a8fe720 con 0x55635ab29020 2024-03-14T09:55:09.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf2de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab28700 0x55635ab36cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf2e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab37350 0x55635ab346a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf2d400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab29020 0x55635ab34100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf4864eb80 10 client.0 _ll_register_callbacks cb 0x55635a882420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:09.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf2e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab37350 0x55635ab346a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60922/0 (socket says 172.21.15.78:60922) 2024-03-14T09:55:09.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf2e800700 1 -- 172.21.15.78:0/828867627 learned_addr learned my addr 172.21.15.78:0/828867627 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:09.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf2e800700 1 -- 172.21.15.78:0/828867627 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab29020 msgr2=0x55635ab34100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:09.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf2e800700 1 --2- 172.21.15.78:0/828867627 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab29020 0x55635ab34100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf2e800700 1 -- 172.21.15.78:0/828867627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab28700 msgr2=0x55635ab36cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:09.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf2e800700 1 --2- 172.21.15.78:0/828867627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab28700 0x55635ab36cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.932+0000 7fdf2e800700 1 -- 172.21.15.78:0/828867627 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf3010e440 con 0x55635ab37350 2024-03-14T09:55:09.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.933+0000 7fdf2e800700 1 --2- 172.21.15.78:0/828867627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab37350 0x55635ab346a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800c6f0 tx=0x7fdf28099000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4b9ca7330511afe server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:09.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.933+0000 7fdf0f400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:55:09.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.933+0000 7fdf0f400700 1 -- 172.21.15.78:0/828867627 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf280842b0 con 0x55635ab37350 2024-03-14T09:55:09.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.933+0000 7fdf0f400700 1 -- 172.21.15.78:0/828867627 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fdf280993d0 con 0x55635ab37350 2024-03-14T09:55:09.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.933+0000 7fdf4864eb80 1 -- 172.21.15.78:0/828867627 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55635ab1b7a0 con 0x55635ab37350 2024-03-14T09:55:09.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.933+0000 7fdf0f400700 1 -- 172.21.15.78:0/828867627 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf280cc430 con 0x55635ab37350 2024-03-14T09:55:09.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.933+0000 7fdf4864eb80 10 client.25907 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:09.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.933+0000 7fdf4864eb80 1 -- 172.21.15.78:0/828867627 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55635a8a8be0 con 0x55635ab37350 2024-03-14T09:55:09.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.933+0000 7fdf0f400700 1 -- 172.21.15.78:0/828867627 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55635ab1b7a0 con 0x55635ab37350 2024-03-14T09:55:09.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.933+0000 7fdf0f400700 1 -- 172.21.15.78:0/828867627 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fdf280988c0 con 0x55635ab37350 2024-03-14T09:55:09.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.934+0000 7fdf4864eb80 10 client.25907 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.934+0000 7fdf4864eb80 10 client.25907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.934+0000 7fdf4864eb80 20 client.25907 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.934+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/828867627 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55635ab46e60 0x55635ab49280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.934+0000 7fdf4864eb80 4 client.25907 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.934+0000 7fdf4864eb80 1 -- 172.21.15.78:0/828867627 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55635aa73780 con 0x55635ab46e60 2024-03-14T09:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.934+0000 7fdf2de00700 1 --2- 172.21.15.78:0/828867627 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55635ab46e60 0x55635ab49280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.934+0000 7fdf0f400700 10 client.25907 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.934+0000 7fdf2de00700 1 --2- 172.21.15.78:0/828867627 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55635ab46e60 0x55635ab49280 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28c45231becfc3ec server_cookie=403129bb3692097f in_seq=0 out_seq=0 2024-03-14T09:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.936+0000 7fdf0f400700 1 -- 172.21.15.78:0/828867627 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55635aa73780 con 0x55635ab46e60 2024-03-14T09:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.936+0000 7fdf0f400700 10 client.25907 handle_command_reply: tid=0 2024-03-14T09:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.936+0000 7fdf4864eb80 1 client.25907 shutdown 2024-03-14T09:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.936+0000 7fdf4864eb80 1 -- 172.21.15.78:0/828867627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab37350 msgr2=0x55635ab346a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.936+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/828867627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab37350 0x55635ab346a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800c6f0 tx=0x7fdf28099000 comp rx=0 tx=0).stop 2024-03-14T09:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.936+0000 7fdf4864eb80 1 -- 172.21.15.78:0/828867627 shutdown_connections 2024-03-14T09:55:09.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.936+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/828867627 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55635ab46e60 0x55635ab49280 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.936+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/828867627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab37350 0x55635ab346a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.936+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/828867627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab28700 0x55635ab36cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.936+0000 7fdf0f400700 0 client.25907 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.936+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/828867627 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab29020 0x55635ab34100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.936+0000 7fdf4864eb80 1 -- 172.21.15.78:0/828867627 >> 172.21.15.78:0/828867627 conn(0x55635aa9e2a0 msgr2=0x55635ab23710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.936+0000 7fdf4864eb80 1 -- 172.21.15.78:0/828867627 shutdown_connections 2024-03-14T09:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.936+0000 7fdf4864eb80 1 -- 172.21.15.78:0/828867627 wait complete. 2024-03-14T09:55:09.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.936+0000 7fdf4864eb80 20 client.25907 trim_cache size 0 max 16384 2024-03-14T09:55:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.954+0000 7fdf4864eb80 1 Processor -- start 2024-03-14T09:55:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.954+0000 7fdf4864eb80 1 -- start start 2024-03-14T09:55:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.954+0000 7fdf4864eb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab28700 0x55635ab3df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.954+0000 7fdf4864eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab29020 0x55635ab42de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.954+0000 7fdf4864eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab37350 0x55635ab36800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.954+0000 7fdf4864eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55635ab247f0 con 0x55635ab37350 2024-03-14T09:55:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.954+0000 7fdf4864eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55635ab34ca0 con 0x55635ab29020 2024-03-14T09:55:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.954+0000 7fdf4864eb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55635ab355c0 con 0x55635ab28700 2024-03-14T09:55:09.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.955+0000 7fdf2d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab29020 0x55635ab42de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.955+0000 7fdf2d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab29020 0x55635ab42de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60924/0 (socket says 172.21.15.78:60924) 2024-03-14T09:55:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.955+0000 7fdf2d400700 1 -- 172.21.15.78:0/1670512746 learned_addr learned my addr 172.21.15.78:0/1670512746 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.955+0000 7fdf2e800700 1 --2- 172.21.15.78:0/1670512746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab37350 0x55635ab36800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.955+0000 7fdf2de00700 1 --2- 172.21.15.78:0/1670512746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab28700 0x55635ab3df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.955+0000 7fdf2d400700 1 -- 172.21.15.78:0/1670512746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab28700 msgr2=0x55635ab3df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.955+0000 7fdf2d400700 1 --2- 172.21.15.78:0/1670512746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab28700 0x55635ab3df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.955+0000 7fdf2d400700 1 -- 172.21.15.78:0/1670512746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab37350 msgr2=0x55635ab36800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.955+0000 7fdf2d400700 1 --2- 172.21.15.78:0/1670512746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab37350 0x55635ab36800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.955+0000 7fdf2d400700 1 -- 172.21.15.78:0/1670512746 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55635aa40f20 con 0x55635ab29020 2024-03-14T09:55:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.955+0000 7fdf2e800700 1 --2- 172.21.15.78:0/1670512746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab37350 0x55635ab36800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:55:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.955+0000 7fdf2d400700 1 --2- 172.21.15.78:0/1670512746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab29020 0x55635ab42de0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c009320 tx=0x7fdf1c012070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfaffe80781e2475 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.955+0000 7fdf1ba00700 1 -- 172.21.15.78:0/1670512746 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c01e4c0 con 0x55635ab29020 2024-03-14T09:55:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.956+0000 7fdf1ba00700 1 -- 172.21.15.78:0/1670512746 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf1c01eae0 con 0x55635ab29020 2024-03-14T09:55:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.956+0000 7fdf1ba00700 1 -- 172.21.15.78:0/1670512746 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c02aa10 con 0x55635ab29020 2024-03-14T09:55:09.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.956+0000 7fdf4864eb80 1 -- 172.21.15.78:0/1670512746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab29020 msgr2=0x55635ab42de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:09.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.956+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/1670512746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab29020 0x55635ab42de0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c009320 tx=0x7fdf1c012070 comp rx=0 tx=0).stop 2024-03-14T09:55:09.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.956+0000 7fdf4864eb80 1 -- 172.21.15.78:0/1670512746 shutdown_connections 2024-03-14T09:55:09.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.956+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/1670512746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab37350 0x55635ab36800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.956+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/1670512746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab28700 0x55635ab3df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.956+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/1670512746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab29020 0x55635ab42de0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.956+0000 7fdf4864eb80 1 -- 172.21.15.78:0/1670512746 >> 172.21.15.78:0/1670512746 conn(0x55635aa9e2a0 msgr2=0x55635ab275e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:09.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.956+0000 7fdf4864eb80 1 -- 172.21.15.78:0/1670512746 shutdown_connections 2024-03-14T09:55:09.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.956+0000 7fdf4864eb80 1 -- 172.21.15.78:0/1670512746 wait complete. 2024-03-14T09:55:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.957+0000 7fdf4864eb80 1 -- start start 2024-03-14T09:55:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.957+0000 7fdf4864eb80 1 Processor -- start 2024-03-14T09:55:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf4864eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab28700 0x55635ab3c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf4864eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab29020 0x55635ab3cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf4864eb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab37350 0x55635ab98950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf4864eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55635ab3d6f0 con 0x55635ab29020 2024-03-14T09:55:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf4864eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55635ab98e90 con 0x55635ab28700 2024-03-14T09:55:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf4864eb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55635ab98fd0 con 0x55635ab37350 2024-03-14T09:55:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf4864eb80 10 client.0 _ll_register_callbacks cb 0x7fdf280957a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf2de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab28700 0x55635ab3c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf2de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab28700 0x55635ab3c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60926/0 (socket says 172.21.15.78:60926) 2024-03-14T09:55:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf2de00700 1 -- 172.21.15.78:0/249508579 learned_addr learned my addr 172.21.15.78:0/249508579 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf2d400700 1 --2- 172.21.15.78:0/249508579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab29020 0x55635ab3cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf2e800700 1 --2- 172.21.15.78:0/249508579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab37350 0x55635ab98950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf2de00700 1 -- 172.21.15.78:0/249508579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab37350 msgr2=0x55635ab98950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf2de00700 1 --2- 172.21.15.78:0/249508579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab37350 0x55635ab98950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf2de00700 1 -- 172.21.15.78:0/249508579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab29020 msgr2=0x55635ab3cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf2de00700 1 --2- 172.21.15.78:0/249508579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab29020 0x55635ab3cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.958+0000 7fdf2de00700 1 -- 172.21.15.78:0/249508579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1c026040 con 0x55635ab28700 2024-03-14T09:55:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.959+0000 7fdf2de00700 1 --2- 172.21.15.78:0/249508579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab28700 0x55635ab3c8a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdf2000b6f0 tx=0x7fdf2000b2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fd823c6aecb49f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.959+0000 7fdf0f400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:55:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.959+0000 7fdf0f400700 1 -- 172.21.15.78:0/249508579 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf20031040 con 0x55635ab28700 2024-03-14T09:55:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.959+0000 7fdf4864eb80 1 -- 172.21.15.78:0/249508579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdf2001e040 con 0x55635ab28700 2024-03-14T09:55:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.959+0000 7fdf0f400700 1 -- 172.21.15.78:0/249508579 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fdf200353f0 con 0x55635ab28700 2024-03-14T09:55:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.959+0000 7fdf0f400700 1 -- 172.21.15.78:0/249508579 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf2001d040 con 0x55635ab28700 2024-03-14T09:55:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.959+0000 7fdf4864eb80 10 client.25913 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.959+0000 7fdf4864eb80 1 -- 172.21.15.78:0/249508579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55635a900d50 con 0x55635ab28700 2024-03-14T09:55:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.959+0000 7fdf0f400700 1 -- 172.21.15.78:0/249508579 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf2001e040 con 0x55635ab28700 2024-03-14T09:55:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.959+0000 7fdf0f400700 1 -- 172.21.15.78:0/249508579 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fdf20034070 con 0x55635ab28700 2024-03-14T09:55:09.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.960+0000 7fdf4864eb80 10 client.25913 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:09.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.960+0000 7fdf4864eb80 10 client.25913 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:55:09.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.960+0000 7fdf4864eb80 20 client.25913 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:09.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.960+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/249508579 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55635ab9a2b0 0x55635ab9c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:09.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.960+0000 7fdf4864eb80 4 client.25913 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T09:55:09.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.960+0000 7fdf4864eb80 1 -- 172.21.15.78:0/249508579 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55635a8c9d70 con 0x55635ab9a2b0 2024-03-14T09:55:09.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.960+0000 7fdf2d400700 1 --2- 172.21.15.78:0/249508579 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55635ab9a2b0 0x55635ab9c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:09.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.960+0000 7fdf2d400700 1 --2- 172.21.15.78:0/249508579 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55635ab9a2b0 0x55635ab9c6a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f76de662c2c881f server_cookie=58535978aee2292d in_seq=0 out_seq=0 2024-03-14T09:55:09.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.960+0000 7fdf0f400700 10 client.25913 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.962+0000 7fdf0f400700 1 -- 172.21.15.78:0/249508579 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55635a8c9d70 con 0x55635ab9a2b0 2024-03-14T09:55:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.962+0000 7fdf0f400700 10 client.25913 handle_command_reply: tid=0 2024-03-14T09:55:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.962+0000 7fdf4864eb80 1 client.25913 shutdown 2024-03-14T09:55:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.962+0000 7fdf4864eb80 1 -- 172.21.15.78:0/249508579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab28700 msgr2=0x55635ab3c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.962+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/249508579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab28700 0x55635ab3c8a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdf2000b6f0 tx=0x7fdf2000b2f0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.962+0000 7fdf4864eb80 1 -- 172.21.15.78:0/249508579 shutdown_connections 2024-03-14T09:55:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.962+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/249508579 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55635ab9a2b0 0x55635ab9c6a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.962+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/249508579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55635ab28700 0x55635ab3c8a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.962+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/249508579 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55635ab37350 0x55635ab98950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.962+0000 7fdf0f400700 0 client.25913 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.962+0000 7fdf4864eb80 1 --2- 172.21.15.78:0/249508579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55635ab29020 0x55635ab3cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.962+0000 7fdf4864eb80 1 -- 172.21.15.78:0/249508579 >> 172.21.15.78:0/249508579 conn(0x55635aa9e2a0 msgr2=0x55635ab27e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.963+0000 7fdf4864eb80 1 -- 172.21.15.78:0/249508579 shutdown_connections 2024-03-14T09:55:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.963+0000 7fdf4864eb80 1 -- 172.21.15.78:0/249508579 wait complete. 2024-03-14T09:55:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.963+0000 7fdf4864eb80 20 client.25913 trim_cache size 0 max 16384 2024-03-14T09:55:09.966 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:55:09.966 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T09:55:09.967 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "8468f3ed-6cb1-420a-80b7-29f482805ccb", 2024-03-14T09:55:09.967 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T09:55:09.967 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:55:09.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.963+0000 7fdf34c00700 1 -- 172.21.15.78:0/3302471039 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fdf140a0da0 msgr2=0x7fdf140a3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:09.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.963+0000 7fdf34c00700 1 --2- 172.21.15.78:0/3302471039 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fdf140a0da0 0x7fdf140a3260 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c009fd0 tx=0x7fdf1c0047c0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.963+0000 7fdf34c00700 1 -- 172.21.15.78:0/3302471039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf3019bed0 msgr2=0x7fdf3019c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:09.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.963+0000 7fdf34c00700 1 --2- 172.21.15.78:0/3302471039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf3019bed0 0x7fdf3019c350 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800efb0 tx=0x7fdf2800c3a0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.963+0000 7fdf34c00700 1 -- 172.21.15.78:0/3302471039 shutdown_connections 2024-03-14T09:55:09.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.963+0000 7fdf34c00700 1 --2- 172.21.15.78:0/3302471039 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fdf140a0da0 0x7fdf140a3260 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.963+0000 7fdf34c00700 1 --2- 172.21.15.78:0/3302471039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf3019bed0 0x7fdf3019c350 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.963+0000 7fdf34c00700 1 --2- 172.21.15.78:0/3302471039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf30100f30 0x7fdf301975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.964+0000 7fdf34c00700 1 --2- 172.21.15.78:0/3302471039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf30106050 0x7fdf30197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:09.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.964+0000 7fdf34c00700 1 -- 172.21.15.78:0/3302471039 >> 172.21.15.78:0/3302471039 conn(0x7fdf300fc7b0 msgr2=0x7fdf300fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:09.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.964+0000 7fdf34c00700 1 -- 172.21.15.78:0/3302471039 shutdown_connections 2024-03-14T09:55:09.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:09.964+0000 7fdf34c00700 1 -- 172.21.15.78:0/3302471039 wait complete. 2024-03-14T09:55:09.982 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:55:10.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.192+0000 7f287ac00700 1 -- 172.21.15.78:0/1867800603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2874100f20 msgr2=0x7f2874101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:10.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.192+0000 7f287ac00700 1 --2- 172.21.15.78:0/1867800603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2874100f20 0x7f2874101300 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2864009990 tx=0x7f2864009ca0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.192+0000 7f287ac00700 1 -- 172.21.15.78:0/1867800603 shutdown_connections 2024-03-14T09:55:10.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.192+0000 7f287ac00700 1 --2- 172.21.15.78:0/1867800603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2874100f20 0x7f2874101300 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.192+0000 7f287ac00700 1 --2- 172.21.15.78:0/1867800603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2874106040 0x7f287410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.192+0000 7f287ac00700 1 --2- 172.21.15.78:0/1867800603 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28741018d0 0x7f2874105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.192+0000 7f287ac00700 1 -- 172.21.15.78:0/1867800603 >> 172.21.15.78:0/1867800603 conn(0x7f28740fc780 msgr2=0x7f28740feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:10.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.192+0000 7f287ac00700 1 -- 172.21.15.78:0/1867800603 shutdown_connections 2024-03-14T09:55:10.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.192+0000 7f287ac00700 1 -- 172.21.15.78:0/1867800603 wait complete. 2024-03-14T09:55:10.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.194+0000 7f287ac00700 1 Processor -- start 2024-03-14T09:55:10.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.194+0000 7f287ac00700 1 -- start start 2024-03-14T09:55:10.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.194+0000 7f287ac00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2874100f20 0x7f28740730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.194+0000 7f287ac00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2874106040 0x7f28740735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.194+0000 7f287ac00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2874073b30 0x7f28741a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.194+0000 7f287ac00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28741a5cf0 con 0x7f2874073b30 2024-03-14T09:55:10.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.194+0000 7f287ac00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28741a5e60 con 0x7f2874100f20 2024-03-14T09:55:10.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.194+0000 7f287ac00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28741a6020 con 0x7f2874106040 2024-03-14T09:55:10.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.194+0000 7f2873e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2874100f20 0x7f28740730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.194+0000 7f2873e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2874100f20 0x7f28740730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60952/0 (socket says 172.21.15.78:60952) 2024-03-14T09:55:10.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.194+0000 7f2878c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2874073b30 0x7f28741a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.194+0000 7f2873e00700 1 -- 172.21.15.78:0/4111747649 learned_addr learned my addr 172.21.15.78:0/4111747649 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:10.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.195+0000 7f2873400700 1 --2- 172.21.15.78:0/4111747649 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2874106040 0x7f28740735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.195+0000 7f2873e00700 1 -- 172.21.15.78:0/4111747649 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2874106040 msgr2=0x7f28740735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:10.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.195+0000 7f2873e00700 1 --2- 172.21.15.78:0/4111747649 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2874106040 0x7f28740735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.195+0000 7f2873e00700 1 -- 172.21.15.78:0/4111747649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2874073b30 msgr2=0x7f28741a5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:10.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.195+0000 7f2873e00700 1 --2- 172.21.15.78:0/4111747649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2874073b30 0x7f28741a5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.195+0000 7f2873e00700 1 -- 172.21.15.78:0/4111747649 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28640095f0 con 0x7f2874100f20 2024-03-14T09:55:10.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.195+0000 7f2873400700 1 --2- 172.21.15.78:0/4111747649 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2874106040 0x7f28740735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:55:10.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.195+0000 7f2873e00700 1 --2- 172.21.15.78:0/4111747649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2874100f20 0x7f28740730b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2864004760 tx=0x7f2864004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:10.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.195+0000 7f2870c00700 1 -- 172.21.15.78:0/4111747649 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286401d070 con 0x7f2874100f20 2024-03-14T09:55:10.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.195+0000 7f2870c00700 1 -- 172.21.15.78:0/4111747649 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2864022470 con 0x7f2874100f20 2024-03-14T09:55:10.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.195+0000 7f287ac00700 1 -- 172.21.15.78:0/4111747649 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28741a62a0 con 0x7f2874100f20 2024-03-14T09:55:10.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.195+0000 7f2870c00700 1 -- 172.21.15.78:0/4111747649 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2864017650 con 0x7f2874100f20 2024-03-14T09:55:10.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.195+0000 7f287ac00700 1 -- 172.21.15.78:0/4111747649 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28741a6720 con 0x7f2874100f20 2024-03-14T09:55:10.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.199+0000 7f2870c00700 1 -- 172.21.15.78:0/4111747649 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) v1 ==== 135621+0+0 (secure 0 0 0) 0x7f28640225e0 con 0x7f2874100f20 2024-03-14T09:55:10.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.200+0000 7f2870c00700 1 --2- 172.21.15.78:0/4111747649 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f28580a0d20 0x7f28580a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.200+0000 7f2870c00700 1 -- 172.21.15.78:0/4111747649 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f28640cdc90 con 0x7f2874100f20 2024-03-14T09:55:10.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.200+0000 7f2873400700 1 --2- 172.21.15.78:0/4111747649 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f28580a0d20 0x7f28580a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.200+0000 7f288e767b80 1 Processor -- start 2024-03-14T09:55:10.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.200+0000 7f288e767b80 1 -- start start 2024-03-14T09:55:10.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f288e767b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae3a680 0x564a2ae3aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f288e767b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3afa0 0x564a2ae44980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f288e767b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae44ec0 0x564a2ae472b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f288e767b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a2ad93990 con 0x564a2ae3afa0 2024-03-14T09:55:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f288e767b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a2ad91350 con 0x564a2ae44ec0 2024-03-14T09:55:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f288e767b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a2ad8cc40 con 0x564a2ae3a680 2024-03-14T09:55:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f2873e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae3a680 0x564a2ae3aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f2873400700 1 --2- 172.21.15.78:0/4111747649 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f28580a0d20 0x7f28580a31e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f28740747e0 tx=0x7f2860009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f2873e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae3a680 0x564a2ae3aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:53208/0 (socket says 172.21.15.78:53208) 2024-03-14T09:55:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f2873400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3afa0 0x564a2ae44980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f2873e00700 1 -- 172.21.15.78:0/754981577 learned_addr learned my addr 172.21.15.78:0/754981577 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f2878c00700 1 --2- 172.21.15.78:0/754981577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae44ec0 0x564a2ae472b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f2873e00700 1 -- 172.21.15.78:0/754981577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae44ec0 msgr2=0x564a2ae472b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f2873e00700 1 --2- 172.21.15.78:0/754981577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae44ec0 0x564a2ae472b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f2873e00700 1 -- 172.21.15.78:0/754981577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3afa0 msgr2=0x564a2ae44980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f2873e00700 1 --2- 172.21.15.78:0/754981577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3afa0 0x564a2ae44980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f2873e00700 1 -- 172.21.15.78:0/754981577 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a2ac0e3b0 con 0x564a2ae3a680 2024-03-14T09:55:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f2873400700 1 --2- 172.21.15.78:0/754981577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3afa0 0x564a2ae44980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:55:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.201+0000 7f2873e00700 1 --2- 172.21.15.78:0/754981577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae3a680 0x564a2ae3aa60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2864096f70 tx=0x7f286409a5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d975b44417531c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.202+0000 7f2869200700 1 -- 172.21.15.78:0/754981577 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286409b170 con 0x564a2ae3a680 2024-03-14T09:55:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.202+0000 7f2869200700 1 -- 172.21.15.78:0/754981577 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f286409aa70 con 0x564a2ae3a680 2024-03-14T09:55:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.202+0000 7f2869200700 1 -- 172.21.15.78:0/754981577 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28640a29d0 con 0x564a2ae3a680 2024-03-14T09:55:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.202+0000 7f288e767b80 1 -- 172.21.15.78:0/754981577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae3a680 msgr2=0x564a2ae3aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.202+0000 7f288e767b80 1 --2- 172.21.15.78:0/754981577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae3a680 0x564a2ae3aa60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2864096f70 tx=0x7f286409a5b0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.202+0000 7f288e767b80 1 -- 172.21.15.78:0/754981577 shutdown_connections 2024-03-14T09:55:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.202+0000 7f288e767b80 1 --2- 172.21.15.78:0/754981577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae44ec0 0x564a2ae472b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.202+0000 7f288e767b80 1 --2- 172.21.15.78:0/754981577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae3a680 0x564a2ae3aa60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.202+0000 7f288e767b80 1 --2- 172.21.15.78:0/754981577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3afa0 0x564a2ae44980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.202+0000 7f288e767b80 1 -- 172.21.15.78:0/754981577 >> 172.21.15.78:0/754981577 conn(0x564a2adb0220 msgr2=0x564a2ae35aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.203+0000 7f288e767b80 1 -- 172.21.15.78:0/754981577 shutdown_connections 2024-03-14T09:55:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.203+0000 7f288e767b80 1 -- 172.21.15.78:0/754981577 wait complete. 2024-03-14T09:55:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.203+0000 7f288e767b80 1 -- start start 2024-03-14T09:55:10.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.203+0000 7f288e767b80 1 Processor -- start 2024-03-14T09:55:10.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f288e767b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 0x564a2ae44820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f288e767b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae3afa0 0x564a2ae41b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f288e767b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae44ec0 0x564a2ae420f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f288e767b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a2ad28d80 con 0x564a2ae3a680 2024-03-14T09:55:10.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f288e767b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a2ad28a20 con 0x564a2ae3afa0 2024-03-14T09:55:10.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f288e767b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a2ac106a0 con 0x564a2ae44ec0 2024-03-14T09:55:10.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f288e767b80 10 client.0 _ll_register_callbacks cb 0x564a2ab943a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:10.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f2873400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae3afa0 0x564a2ae41b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f2873e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 0x564a2ae44820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f2878c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae44ec0 0x564a2ae420f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f2873400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae3afa0 0x564a2ae41b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60964/0 (socket says 172.21.15.78:60964) 2024-03-14T09:55:10.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f2873400700 1 -- 172.21.15.78:0/1324206273 learned_addr learned my addr 172.21.15.78:0/1324206273 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:10.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f2878c00700 1 -- 172.21.15.78:0/1324206273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae3afa0 msgr2=0x564a2ae41b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:10.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f2878c00700 1 --2- 172.21.15.78:0/1324206273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae3afa0 0x564a2ae41b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f2878c00700 1 -- 172.21.15.78:0/1324206273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 msgr2=0x564a2ae44820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:10.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f2878c00700 1 --2- 172.21.15.78:0/1324206273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 0x564a2ae44820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.204+0000 7f2878c00700 1 -- 172.21.15.78:0/1324206273 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28741a6720 con 0x564a2ae44ec0 2024-03-14T09:55:10.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.205+0000 7f2878c00700 1 --2- 172.21.15.78:0/1324206273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae44ec0 0x564a2ae420f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f286c012dc0 tx=0x7f286c014600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9034a9a222094e4a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:10.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.205+0000 7f285d600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:55:10.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.205+0000 7f285d600700 1 -- 172.21.15.78:0/1324206273 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286c00e040 con 0x564a2ae44ec0 2024-03-14T09:55:10.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.205+0000 7f288e767b80 1 -- 172.21.15.78:0/1324206273 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564a2ae2d720 con 0x564a2ae44ec0 2024-03-14T09:55:10.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.205+0000 7f285d600700 1 -- 172.21.15.78:0/1324206273 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f286c014a80 con 0x564a2ae44ec0 2024-03-14T09:55:10.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.205+0000 7f285d600700 1 -- 172.21.15.78:0/1324206273 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f286c014c40 con 0x564a2ae44ec0 2024-03-14T09:55:10.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.205+0000 7f288e767b80 10 client.25933 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:10.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.205+0000 7f288e767b80 1 -- 172.21.15.78:0/1324206273 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564a2abbab60 con 0x564a2ae44ec0 2024-03-14T09:55:10.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.205+0000 7f285d600700 1 -- 172.21.15.78:0/1324206273 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564a2ae2d720 con 0x564a2ae44ec0 2024-03-14T09:55:10.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.206+0000 7f285d600700 1 -- 172.21.15.78:0/1324206273 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f286c01c020 con 0x564a2ae44ec0 2024-03-14T09:55:10.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.206+0000 7f288e767b80 10 client.25933 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:10.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.206+0000 7f288e767b80 10 client.25933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:55:10.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.206+0000 7f288e767b80 20 client.25933 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:10.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.206+0000 7f288e767b80 1 --2- 172.21.15.78:0/1324206273 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x564a2ae54770 0x564a2ae56b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.206+0000 7f288e767b80 4 client.25933 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:55:10.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.206+0000 7f288e767b80 1 -- 172.21.15.78:0/1324206273 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564a2ad85700 con 0x564a2ae54770 2024-03-14T09:55:10.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.206+0000 7f2873e00700 1 --2- 172.21.15.78:0/1324206273 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x564a2ae54770 0x564a2ae56b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.206+0000 7f285d600700 10 client.25933 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.206+0000 7f2873e00700 1 --2- 172.21.15.78:0/1324206273 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x564a2ae54770 0x564a2ae56b90 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d15e5a0fcd6129d server_cookie=786fbecbfef14ec0 in_seq=0 out_seq=0 2024-03-14T09:55:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.208+0000 7f285d600700 1 -- 172.21.15.78:0/1324206273 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x564a2ad85700 con 0x564a2ae54770 2024-03-14T09:55:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.208+0000 7f285d600700 10 client.25933 handle_command_reply: tid=0 2024-03-14T09:55:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.208+0000 7f288e767b80 1 client.25933 shutdown 2024-03-14T09:55:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.208+0000 7f288e767b80 1 -- 172.21.15.78:0/1324206273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae44ec0 msgr2=0x564a2ae420f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.208+0000 7f288e767b80 1 --2- 172.21.15.78:0/1324206273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae44ec0 0x564a2ae420f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f286c012dc0 tx=0x7f286c014600 comp rx=0 tx=0).stop 2024-03-14T09:55:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.208+0000 7f288e767b80 1 -- 172.21.15.78:0/1324206273 shutdown_connections 2024-03-14T09:55:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.208+0000 7f288e767b80 1 --2- 172.21.15.78:0/1324206273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae44ec0 0x564a2ae420f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.208+0000 7f288e767b80 1 --2- 172.21.15.78:0/1324206273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 0x564a2ae44820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.208+0000 7f288e767b80 1 --2- 172.21.15.78:0/1324206273 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x564a2ae54770 0x564a2ae56b90 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.208+0000 7f288e767b80 1 --2- 172.21.15.78:0/1324206273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae3afa0 0x564a2ae41b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.208+0000 7f285d600700 0 client.25933 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.208+0000 7f288e767b80 1 -- 172.21.15.78:0/1324206273 >> 172.21.15.78:0/1324206273 conn(0x564a2adb0220 msgr2=0x564a2ae35690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.208+0000 7f288e767b80 1 -- 172.21.15.78:0/1324206273 shutdown_connections 2024-03-14T09:55:10.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.208+0000 7f288e767b80 1 -- 172.21.15.78:0/1324206273 wait complete. 2024-03-14T09:55:10.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.208+0000 7f288e767b80 20 client.25933 trim_cache size 0 max 16384 2024-03-14T09:55:10.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.225+0000 7f288e767b80 1 Processor -- start 2024-03-14T09:55:10.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.225+0000 7f288e767b80 1 -- start start 2024-03-14T09:55:10.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.225+0000 7f288e767b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 0x564a2ae420c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.225+0000 7f288e767b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae3afa0 0x564a2ae3edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.225+0000 7f288e767b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae44ec0 0x564a2ae3f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.225+0000 7f288e767b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a2ae51240 con 0x564a2ae3a680 2024-03-14T09:55:10.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.225+0000 7f288e767b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a2ae36770 con 0x564a2ae3afa0 2024-03-14T09:55:10.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.225+0000 7f288e767b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a2ae3baf0 con 0x564a2ae44ec0 2024-03-14T09:55:10.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.226+0000 7f2873400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae3afa0 0x564a2ae3edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.226+0000 7f2873400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae3afa0 0x564a2ae3edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60966/0 (socket says 172.21.15.78:60966) 2024-03-14T09:55:10.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.226+0000 7f2873400700 1 -- 172.21.15.78:0/2162077748 learned_addr learned my addr 172.21.15.78:0/2162077748 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:10.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.226+0000 7f2873e00700 1 --2- 172.21.15.78:0/2162077748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 0x564a2ae420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.226+0000 7f2878c00700 1 --2- 172.21.15.78:0/2162077748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae44ec0 0x564a2ae3f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.226+0000 7f2873400700 1 -- 172.21.15.78:0/2162077748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae44ec0 msgr2=0x564a2ae3f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:10.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.226+0000 7f2873400700 1 --2- 172.21.15.78:0/2162077748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae44ec0 0x564a2ae3f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.226+0000 7f2873400700 1 -- 172.21.15.78:0/2162077748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 msgr2=0x564a2ae420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:10.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.226+0000 7f2873400700 1 --2- 172.21.15.78:0/2162077748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 0x564a2ae420c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.226+0000 7f2873400700 1 -- 172.21.15.78:0/2162077748 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a2ad52ea0 con 0x564a2ae3afa0 2024-03-14T09:55:10.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.226+0000 7f2878c00700 1 --2- 172.21.15.78:0/2162077748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae44ec0 0x564a2ae3f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:55:10.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.226+0000 7f2873e00700 1 --2- 172.21.15.78:0/2162077748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 0x564a2ae420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:55:10.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.226+0000 7f2873400700 1 --2- 172.21.15.78:0/2162077748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae3afa0 0x564a2ae3edf0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2860012010 tx=0x7f2860014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef3be70dce90bb94 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:10.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.227+0000 7f2869c00700 1 -- 172.21.15.78:0/2162077748 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2860003b10 con 0x564a2ae3afa0 2024-03-14T09:55:10.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.227+0000 7f2869c00700 1 -- 172.21.15.78:0/2162077748 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2860003c70 con 0x564a2ae3afa0 2024-03-14T09:55:10.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.227+0000 7f2869c00700 1 -- 172.21.15.78:0/2162077748 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286002a6a0 con 0x564a2ae3afa0 2024-03-14T09:55:10.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.227+0000 7f288e767b80 1 -- 172.21.15.78:0/2162077748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae3afa0 msgr2=0x564a2ae3edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:10.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.227+0000 7f288e767b80 1 --2- 172.21.15.78:0/2162077748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae3afa0 0x564a2ae3edf0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2860012010 tx=0x7f2860014000 comp rx=0 tx=0).stop 2024-03-14T09:55:10.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.227+0000 7f288e767b80 1 -- 172.21.15.78:0/2162077748 shutdown_connections 2024-03-14T09:55:10.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.227+0000 7f288e767b80 1 --2- 172.21.15.78:0/2162077748 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae44ec0 0x564a2ae3f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.227+0000 7f288e767b80 1 --2- 172.21.15.78:0/2162077748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 0x564a2ae420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.227+0000 7f288e767b80 1 --2- 172.21.15.78:0/2162077748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae3afa0 0x564a2ae3edf0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.227+0000 7f288e767b80 1 -- 172.21.15.78:0/2162077748 >> 172.21.15.78:0/2162077748 conn(0x564a2adb0220 msgr2=0x564a2ae40440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:10.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.227+0000 7f288e767b80 1 -- 172.21.15.78:0/2162077748 shutdown_connections 2024-03-14T09:55:10.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.227+0000 7f288e767b80 1 -- 172.21.15.78:0/2162077748 wait complete. 2024-03-14T09:55:10.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.228+0000 7f288e767b80 1 -- start start 2024-03-14T09:55:10.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.228+0000 7f288e767b80 1 Processor -- start 2024-03-14T09:55:10.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.228+0000 7f288e767b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 0x564a2ae47a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.228+0000 7f288e767b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae3afa0 0x564a2ae47fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.228+0000 7f288e767b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae44ec0 0x564a2ae48510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.228+0000 7f288e767b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a2aea62b0 con 0x564a2ae3a680 2024-03-14T09:55:10.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.228+0000 7f288e767b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a2aea6420 con 0x564a2ae44ec0 2024-03-14T09:55:10.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.228+0000 7f288e767b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a2aea65e0 con 0x564a2ae3afa0 2024-03-14T09:55:10.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.228+0000 7f288e767b80 10 client.0 _ll_register_callbacks cb 0x7f286c0126f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:10.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.229+0000 7f2873e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 0x564a2ae47a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.229+0000 7f2873400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae3afa0 0x564a2ae47fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.229+0000 7f2873e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 0x564a2ae47a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44904/0 (socket says 172.21.15.78:44904) 2024-03-14T09:55:10.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.229+0000 7f2878c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae44ec0 0x564a2ae48510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.229+0000 7f2873e00700 1 -- 172.21.15.78:0/4030915322 learned_addr learned my addr 172.21.15.78:0/4030915322 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:10.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.229+0000 7f2873e00700 1 -- 172.21.15.78:0/4030915322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae3afa0 msgr2=0x564a2ae47fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:10.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.229+0000 7f2873e00700 1 --2- 172.21.15.78:0/4030915322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae3afa0 0x564a2ae47fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.229+0000 7f2873e00700 1 -- 172.21.15.78:0/4030915322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae44ec0 msgr2=0x564a2ae48510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.229+0000 7f2873e00700 1 --2- 172.21.15.78:0/4030915322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae44ec0 0x564a2ae48510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.229+0000 7f2873e00700 1 -- 172.21.15.78:0/4030915322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2860011cf0 con 0x564a2ae3a680 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.229+0000 7f2873400700 1 --2- 172.21.15.78:0/4030915322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae3afa0 0x564a2ae47fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.229+0000 7f2878c00700 1 --2- 172.21.15.78:0/4030915322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae44ec0 0x564a2ae48510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.229+0000 7f2873e00700 1 --2- 172.21.15.78:0/4030915322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 0x564a2ae47a90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2864022890 tx=0x7f2864009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed4d6ac4b6072fd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.229+0000 7f285d600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.230+0000 7f285d600700 1 -- 172.21.15.78:0/4030915322 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286409b170 con 0x564a2ae3a680 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.230+0000 7f285d600700 1 -- 172.21.15.78:0/4030915322 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f28640a2e60 con 0x564a2ae3a680 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.230+0000 7f288e767b80 1 -- 172.21.15.78:0/4030915322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f286409aa70 con 0x564a2ae3a680 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.230+0000 7f285d600700 1 -- 172.21.15.78:0/4030915322 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28640b3170 con 0x564a2ae3a680 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.230+0000 7f288e767b80 10 client.15927 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.230+0000 7f288e767b80 1 -- 172.21.15.78:0/4030915322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564a2ac12cd0 con 0x564a2ae3a680 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.230+0000 7f285d600700 1 -- 172.21.15.78:0/4030915322 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f286409aa70 con 0x564a2ae3a680 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.230+0000 7f285d600700 1 -- 172.21.15.78:0/4030915322 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f28640a29d0 con 0x564a2ae3a680 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.230+0000 7f288e767b80 10 client.15927 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:10.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.230+0000 7f288e767b80 10 client.15927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:55:10.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.230+0000 7f288e767b80 20 client.15927 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:10.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.230+0000 7f288e767b80 1 --2- 172.21.15.78:0/4030915322 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x564a2aea7a30 0x564a2aea9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:10.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.230+0000 7f288e767b80 4 client.15927 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:55:10.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.230+0000 7f288e767b80 1 -- 172.21.15.78:0/4030915322 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564a2abdbcf0 con 0x564a2aea7a30 2024-03-14T09:55:10.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.231+0000 7f2873400700 1 --2- 172.21.15.78:0/4030915322 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x564a2aea7a30 0x564a2aea9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:10.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.231+0000 7f2873400700 1 --2- 172.21.15.78:0/4030915322 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x564a2aea7a30 0x564a2aea9e50 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c68c7f0d49b840d server_cookie=db899c8460e27d7b in_seq=0 out_seq=0 2024-03-14T09:55:10.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.231+0000 7f285d600700 10 client.15927 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:10.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.231+0000 7f285d600700 1 -- 172.21.15.78:0/4030915322 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564a2abdbcf0 con 0x564a2aea7a30 2024-03-14T09:55:10.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.231+0000 7f285d600700 10 client.15927 handle_command_reply: tid=0 2024-03-14T09:55:10.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.231+0000 7f288e767b80 1 client.15927 shutdown 2024-03-14T09:55:10.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.231+0000 7f288e767b80 1 -- 172.21.15.78:0/4030915322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 msgr2=0x564a2ae47a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:10.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.231+0000 7f288e767b80 1 --2- 172.21.15.78:0/4030915322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 0x564a2ae47a90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2864022890 tx=0x7f2864009310 comp rx=0 tx=0).stop 2024-03-14T09:55:10.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.231+0000 7f288e767b80 1 -- 172.21.15.78:0/4030915322 shutdown_connections 2024-03-14T09:55:10.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.231+0000 7f288e767b80 1 --2- 172.21.15.78:0/4030915322 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x564a2aea7a30 0x564a2aea9e50 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.231+0000 7f288e767b80 1 --2- 172.21.15.78:0/4030915322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a2ae44ec0 0x564a2ae48510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.231+0000 7f288e767b80 1 --2- 172.21.15.78:0/4030915322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a2ae3a680 0x564a2ae47a90 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.231+0000 7f285d600700 0 client.15927 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:10.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.231+0000 7f288e767b80 1 --2- 172.21.15.78:0/4030915322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a2ae3afa0 0x564a2ae47fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.231+0000 7f288e767b80 1 -- 172.21.15.78:0/4030915322 >> 172.21.15.78:0/4030915322 conn(0x564a2adb0220 msgr2=0x564a2ae433b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:10.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.232+0000 7f288e767b80 1 -- 172.21.15.78:0/4030915322 shutdown_connections 2024-03-14T09:55:10.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.232+0000 7f288e767b80 1 -- 172.21.15.78:0/4030915322 wait complete. 2024-03-14T09:55:10.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.232+0000 7f288e767b80 20 client.15927 trim_cache size 0 max 16384 2024-03-14T09:55:10.237 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:55:10.237 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-14T09:55:10.237 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T09:55:10.238 INFO:teuthology.orchestra.run.smithi078.stdout: "8468f3ed-6cb1-420a-80b7-29f482805ccb": { 2024-03-14T09:55:10.238 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T09:55:10.238 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "8468f3ed-6cb1-420a-80b7-29f482805ccb", 2024-03-14T09:55:10.238 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T09:55:10.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:55:10.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:55:10.238 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:55:10.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.232+0000 7f287ac00700 1 -- 172.21.15.78:0/4111747649 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f28580a0d20 msgr2=0x7f28580a31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:10.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.232+0000 7f287ac00700 1 --2- 172.21.15.78:0/4111747649 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f28580a0d20 0x7f28580a31e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f28740747e0 tx=0x7f2860009260 comp rx=0 tx=0).stop 2024-03-14T09:55:10.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.232+0000 7f287ac00700 1 -- 172.21.15.78:0/4111747649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2874100f20 msgr2=0x7f28740730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:10.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.232+0000 7f287ac00700 1 --2- 172.21.15.78:0/4111747649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2874100f20 0x7f28740730b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2864004760 tx=0x7f2864004790 comp rx=0 tx=0).stop 2024-03-14T09:55:10.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.232+0000 7f287ac00700 1 -- 172.21.15.78:0/4111747649 shutdown_connections 2024-03-14T09:55:10.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.232+0000 7f287ac00700 1 --2- 172.21.15.78:0/4111747649 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f28580a0d20 0x7f28580a31e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.232+0000 7f287ac00700 1 --2- 172.21.15.78:0/4111747649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2874100f20 0x7f28740730b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.232+0000 7f287ac00700 1 --2- 172.21.15.78:0/4111747649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2874073b30 0x7f28741a5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.232+0000 7f287ac00700 1 --2- 172.21.15.78:0/4111747649 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2874106040 0x7f28740735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:10.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.232+0000 7f287ac00700 1 -- 172.21.15.78:0/4111747649 >> 172.21.15.78:0/4111747649 conn(0x7f28740fc780 msgr2=0x7f28740fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:10.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.233+0000 7f287ac00700 1 -- 172.21.15.78:0/4111747649 shutdown_connections 2024-03-14T09:55:10.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:10.233+0000 7f287ac00700 1 -- 172.21.15.78:0/4111747649 wait complete. 2024-03-14T09:55:10.251 INFO:tasks.cephfs.filesystem:scrub status for tag:8468f3ed-6cb1-420a-80b7-29f482805ccb - {'path': '/', 'tag': '8468f3ed-6cb1-420a-80b7-29f482805ccb', 'options': 'recursive,force'} 2024-03-14T09:55:11.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:11 smithi132 ceph-mon[47959]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-14T09:55:11.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:11 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T09:55:11.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:11 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:55:11.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:11 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T09:55:11.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:11 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:55:11.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:11 smithi078 ceph-mon[38143]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-14T09:55:11.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:11 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T09:55:11.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:11 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:55:11.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:11 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T09:55:11.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:11 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:55:11.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:11 smithi152 ceph-mon[48279]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-14T09:55:11.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:11 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T09:55:11.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:11 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:55:11.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:11 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T09:55:11.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:11 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:55:13.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:13 smithi078 ceph-mon[38143]: pgmap v34: 129 pgs: 129 active+clean; 15 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2024-03-14T09:55:13.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:13 smithi152 ceph-mon[48279]: pgmap v34: 129 pgs: 129 active+clean; 15 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2024-03-14T09:55:13.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:13 smithi132 ceph-mon[47959]: pgmap v34: 129 pgs: 129 active+clean; 15 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2024-03-14T09:55:14.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:14 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T09:55:14.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:14 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T09:55:14.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:14 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T09:55:15.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:15 smithi078 ceph-mon[38143]: pgmap v35: 129 pgs: 129 active+clean; 15 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2024-03-14T09:55:15.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:15 smithi152 ceph-mon[48279]: pgmap v35: 129 pgs: 129 active+clean; 15 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2024-03-14T09:55:15.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:15 smithi132 ceph-mon[47959]: pgmap v35: 129 pgs: 129 active+clean; 15 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 7 op/s 2024-03-14T09:55:17.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:17 smithi078 ceph-mon[38143]: pgmap v36: 129 pgs: 129 active+clean; 16 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-03-14T09:55:17.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:17 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:55:17.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:17 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:55:17.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:17 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:55:17.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:17 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:55:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:17 smithi152 ceph-mon[48279]: pgmap v36: 129 pgs: 129 active+clean; 16 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-03-14T09:55:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:17 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:55:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:17 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:55:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:17 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:55:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:17 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:55:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:17 smithi132 ceph-mon[47959]: pgmap v36: 129 pgs: 129 active+clean; 16 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-03-14T09:55:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:17 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:55:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:17 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:55:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:17 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:55:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:17 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:55:19.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:18 smithi132 ceph-mon[47959]: pgmap v37: 129 pgs: 129 active+clean; 16 MiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-03-14T09:55:19.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:18 smithi078 ceph-mon[38143]: pgmap v37: 129 pgs: 129 active+clean; 16 MiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-03-14T09:55:19.248 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:18 smithi152 ceph-mon[48279]: pgmap v37: 129 pgs: 129 active+clean; 16 MiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-03-14T09:55:21.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:21 smithi132 ceph-mon[47959]: pgmap v38: 129 pgs: 129 active+clean; 16 MiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-03-14T09:55:21.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:21 smithi078 ceph-mon[38143]: pgmap v38: 129 pgs: 129 active+clean; 16 MiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-03-14T09:55:21.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:21 smithi152 ceph-mon[48279]: pgmap v38: 129 pgs: 129 active+clean; 16 MiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-03-14T09:55:23.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:23 smithi078 ceph-mon[38143]: pgmap v39: 129 pgs: 129 active+clean; 18 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2024-03-14T09:55:23.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:23 smithi152 ceph-mon[48279]: pgmap v39: 129 pgs: 129 active+clean; 18 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2024-03-14T09:55:23.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:23 smithi132 ceph-mon[47959]: pgmap v39: 129 pgs: 129 active+clean; 18 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2024-03-14T09:55:25.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:25 smithi078 ceph-mon[38143]: pgmap v40: 129 pgs: 129 active+clean; 18 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 316 KiB/s wr, 11 op/s 2024-03-14T09:55:25.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:25 smithi152 ceph-mon[48279]: pgmap v40: 129 pgs: 129 active+clean; 18 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 316 KiB/s wr, 11 op/s 2024-03-14T09:55:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:25 smithi132 ceph-mon[47959]: pgmap v40: 129 pgs: 129 active+clean; 18 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 316 KiB/s wr, 11 op/s 2024-03-14T09:55:27.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:27 smithi078 ceph-mon[38143]: pgmap v41: 129 pgs: 129 active+clean; 22 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 687 KiB/s wr, 18 op/s 2024-03-14T09:55:27.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:27 smithi152 ceph-mon[48279]: pgmap v41: 129 pgs: 129 active+clean; 22 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 687 KiB/s wr, 18 op/s 2024-03-14T09:55:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:27 smithi132 ceph-mon[47959]: pgmap v41: 129 pgs: 129 active+clean; 22 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 687 KiB/s wr, 18 op/s 2024-03-14T09:55:29.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:28 smithi132 ceph-mon[47959]: pgmap v42: 129 pgs: 129 active+clean; 23 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 698 KiB/s wr, 15 op/s 2024-03-14T09:55:29.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:28 smithi078 ceph-mon[38143]: pgmap v42: 129 pgs: 129 active+clean; 23 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 698 KiB/s wr, 15 op/s 2024-03-14T09:55:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:28 smithi152 ceph-mon[48279]: pgmap v42: 129 pgs: 129 active+clean; 23 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 698 KiB/s wr, 15 op/s 2024-03-14T09:55:31.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:31 smithi078 ceph-mon[38143]: pgmap v43: 129 pgs: 129 active+clean; 23 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 689 KiB/s wr, 13 op/s 2024-03-14T09:55:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:31 smithi152 ceph-mon[48279]: pgmap v43: 129 pgs: 129 active+clean; 23 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 689 KiB/s wr, 13 op/s 2024-03-14T09:55:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:31 smithi132 ceph-mon[47959]: pgmap v43: 129 pgs: 129 active+clean; 23 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 689 KiB/s wr, 13 op/s 2024-03-14T09:55:31.986 INFO:tasks.workunit.client.0.smithi078.stderr:Updating files: 7% (321/4261) Updating files: 8% (341/4261) Updating files: 9% (384/4261) Updating files: 10% (427/4261) Updating files: 11% (469/4261) Updating files: 12% (512/4261) Updating files: 13% (554/4261) Updating files: 14% (597/4261) Updating files: 15% (640/4261) Updating files: 16% (682/4261) Updating files: 17% (725/4261) Updating files: 17% (742/4261) Updating files: 18% (767/4261) Updating files: 19% (810/4261) Updating files: 20% (853/4261) Updating files: 21% (895/4261) Updating files: 22% (938/4261) Updating files: 23% (981/4261) Updating files: 24% (1023/4261) Updating files: 25% (1066/4261) Updating files: 26% (1108/4261) Updating files: 27% (1151/4261) Updating files: 28% (1194/4261) Updating files: 29% (1236/4261) Updating files: 30% (1279/4261) Updating files: 31% (1321/4261) Updating files: 32% (1364/4261) Updating files: 33% (1407/4261) Updating files: 34% (1449/4261) Updating files: 35% (1492/4261) Updating files: 35% (1502/4261) Updating files: 36% (1534/4261) Updating files: 37% (1577/4261) Updating files: 38% (1620/4261) Updating files: 39% (1662/4261) Updating files: 40% (1705/4261) Updating files: 41% (1748/4261) Updating files: 42% (1790/4261) Updating files: 43% (1833/4261) Updating files: 44% (1875/4261) Updating files: 45% (1918/4261) Updating files: 46% (1961/4261) Updating files: 47% (2003/4261) Updating files: 48% (2046/4261) Updating files: 49% (2088/4261) Updating files: 50% (2131/4261) Updating files: 51% (2174/4261) Updating files: 52% (2216/4261) Updating files: 52% (2257/4261) Updating files: 53% (2259/4261) Updating files: 53% (2261/4261) Updating files: 54% (2301/4261) Updating files: 54% (2322/4261) Updating files: 55% (2344/4261) Updating files: 56% (2387/4261) Updating files: 57% (2429/4261) Updating files: 58% (2472/4261) Updating files: 59% (2514/4261) Updating files: 60% (2557/4261) Updating files: 61% (2600/4261) Updating files: 62% (2642/4261) Updating files: 63% (2685/4261) Updating files: 64% (2728/4261) Updating files: 65% (2770/4261) Updating files: 66% (2813/4261) Updating files: 66% (2815/4261) Updating files: 67% (2855/4261) Updating files: 68% (2898/4261) Updating files: 69% (2941/4261) Updating files: 70% (2983/4261) Updating files: 71% (3026/4261) Updating files: 72% (3068/4261) Updating files: 73% (3111/4261) Updating files: 74% (3154/4261) Updating files: 75% (3196/4261) Updating files: 76% (3239/4261) Updating files: 77% (3281/4261) Updating files: 78% (3324/4261) Updating files: 79% (3367/4261) Updating files: 80% (3409/4261) Updating files: 80% (3441/4261) Updating files: 81% (3452/4261) Updating files: 82% (3495/4261) Updating files: 83% (3537/4261) Updating files: 84% (3580/4261) Updating files: 85% (3622/4261) Updating files: 86% (3665/4261) Updating files: 87% (3708/4261) Updating files: 88% (3750/4261) Updating files: 89% (3793/4261) Updating files: 90% (3835/4261) Updating files: 91% (3878/4261) Updating files: 92% (3921/4261) Updating files: 93% (3963/4261) Updating files: 94% (4006/4261) Updating files: 95% (4048/4261) Updating files: 96% (4091/4261) Updating files: 97% (4134/4261) Updating files: 98% (4176/4261) Updating files: 99% (4219/4261) Updating files: 99% (4249/4261) Updating files: 100% (4261/4261) Updating files: 100% (4261/4261), done. 2024-03-14T09:55:32.009 INFO:tasks.workunit.client.0.smithi078.stderr:+ cd xfstests-dev 2024-03-14T09:55:32.009 INFO:tasks.workunit.client.0.smithi078.stderr:+ git checkout 12973fc04fd10d4af086901e10ffa8e48866b735 2024-03-14T09:55:33.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:33 smithi078 ceph-mon[38143]: pgmap v44: 129 pgs: 129 active+clean; 28 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 1.2 MiB/s wr, 33 op/s 2024-03-14T09:55:33.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:33 smithi152 ceph-mon[48279]: pgmap v44: 129 pgs: 129 active+clean; 28 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 1.2 MiB/s wr, 33 op/s 2024-03-14T09:55:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:33 smithi132 ceph-mon[47959]: pgmap v44: 129 pgs: 129 active+clean; 28 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 1.2 MiB/s wr, 33 op/s 2024-03-14T09:55:35.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:35 smithi078 ceph-mon[38143]: pgmap v45: 129 pgs: 129 active+clean; 28 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 979 KiB/s wr, 27 op/s 2024-03-14T09:55:35.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:35 smithi152 ceph-mon[48279]: pgmap v45: 129 pgs: 129 active+clean; 28 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 979 KiB/s wr, 27 op/s 2024-03-14T09:55:35.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:35 smithi132 ceph-mon[47959]: pgmap v45: 129 pgs: 129 active+clean; 28 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 979 KiB/s wr, 27 op/s 2024-03-14T09:55:37.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:37 smithi078 ceph-mon[38143]: pgmap v46: 129 pgs: 129 active+clean; 38 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 48 op/s 2024-03-14T09:55:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:37 smithi152 ceph-mon[48279]: pgmap v46: 129 pgs: 129 active+clean; 38 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 48 op/s 2024-03-14T09:55:37.519 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:37 smithi132 ceph-mon[47959]: pgmap v46: 129 pgs: 129 active+clean; 38 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 48 op/s 2024-03-14T09:55:39.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:38 smithi132 ceph-mon[47959]: pgmap v47: 129 pgs: 129 active+clean; 38 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 41 op/s 2024-03-14T09:55:39.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:38 smithi078 ceph-mon[38143]: pgmap v47: 129 pgs: 129 active+clean; 38 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 41 op/s 2024-03-14T09:55:39.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:38 smithi152 ceph-mon[48279]: pgmap v47: 129 pgs: 129 active+clean; 38 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.5 MiB/s wr, 41 op/s 2024-03-14T09:55:39.533 INFO:tasks.workunit.client.0.smithi078.stderr:Note: switching to '12973fc04fd10d4af086901e10ffa8e48866b735'. 2024-03-14T09:55:39.533 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-14T09:55:39.533 INFO:tasks.workunit.client.0.smithi078.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-14T09:55:39.533 INFO:tasks.workunit.client.0.smithi078.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-14T09:55:39.533 INFO:tasks.workunit.client.0.smithi078.stderr:state without impacting any branches by switching back to a branch. 2024-03-14T09:55:39.534 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-14T09:55:39.534 INFO:tasks.workunit.client.0.smithi078.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-14T09:55:39.534 INFO:tasks.workunit.client.0.smithi078.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-14T09:55:39.534 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-14T09:55:39.534 INFO:tasks.workunit.client.0.smithi078.stderr: git switch -c 2024-03-14T09:55:39.534 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-14T09:55:39.534 INFO:tasks.workunit.client.0.smithi078.stderr:Or undo this operation with: 2024-03-14T09:55:39.534 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-14T09:55:39.534 INFO:tasks.workunit.client.0.smithi078.stderr: git switch - 2024-03-14T09:55:39.534 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-14T09:55:39.534 INFO:tasks.workunit.client.0.smithi078.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-14T09:55:39.534 INFO:tasks.workunit.client.0.smithi078.stderr: 2024-03-14T09:55:39.535 INFO:tasks.workunit.client.0.smithi078.stderr:HEAD is now at 12973fc0 btrfs/022: Correct seq number in golden output 2024-03-14T09:55:39.538 INFO:tasks.workunit.client.0.smithi078.stderr:+ make -j4 2024-03-14T09:55:39.555 INFO:tasks.workunit.client.0.smithi078.stdout:libtoolize -cfi 2024-03-14T09:55:40.251 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:55:40.430 INFO:tasks.workunit.client.0.smithi078.stdout:libtoolize: putting auxiliary files in '.'. 2024-03-14T09:55:40.430 INFO:tasks.workunit.client.0.smithi078.stdout:libtoolize: copying file './config.guess' 2024-03-14T09:55:40.440 INFO:tasks.workunit.client.0.smithi078.stdout:libtoolize: copying file './config.sub' 2024-03-14T09:55:40.450 INFO:tasks.workunit.client.0.smithi078.stdout:libtoolize: copying file './install-sh' 2024-03-14T09:55:40.461 INFO:tasks.workunit.client.0.smithi078.stdout:libtoolize: copying file './ltmain.sh' 2024-03-14T09:55:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.478+0000 7f2dd0800700 1 -- 172.21.15.78:0/1719038183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dc8073b00 msgr2=0x7f2dc810cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.478+0000 7f2dd0800700 1 --2- 172.21.15.78:0/1719038183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dc8073b00 0x7f2dc810cfc0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2db80098f0 tx=0x7f2db8009c00 comp rx=0 tx=0).stop 2024-03-14T09:55:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.478+0000 7f2dd0800700 1 -- 172.21.15.78:0/1719038183 shutdown_connections 2024-03-14T09:55:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.478+0000 7f2dd0800700 1 --2- 172.21.15.78:0/1719038183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dc80731e0 0x7f2dc80735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.478+0000 7f2dd0800700 1 --2- 172.21.15.78:0/1719038183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dc810d500 0x7f2dc810f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.478+0000 7f2dd0800700 1 --2- 172.21.15.78:0/1719038183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dc8073b00 0x7f2dc810cfc0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.478+0000 7f2dd0800700 1 -- 172.21.15.78:0/1719038183 >> 172.21.15.78:0/1719038183 conn(0x7f2dc80fc7b0 msgr2=0x7f2dc80febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.479+0000 7f2dd0800700 1 -- 172.21.15.78:0/1719038183 shutdown_connections 2024-03-14T09:55:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.479+0000 7f2dd0800700 1 -- 172.21.15.78:0/1719038183 wait complete. 2024-03-14T09:55:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.480+0000 7f2dd0800700 1 Processor -- start 2024-03-14T09:55:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.480+0000 7f2dd0800700 1 -- start start 2024-03-14T09:55:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.480+0000 7f2dd0800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dc80731e0 0x7f2dc819f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.480+0000 7f2dd0800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dc8073b00 0x7f2dc819fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.480+0000 7f2dd0800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dc810d500 0x7f2dc8199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.481+0000 7f2dd0800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc8199d70 con 0x7f2dc80731e0 2024-03-14T09:55:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.481+0000 7f2dd0800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc8199ee0 con 0x7f2dc810d500 2024-03-14T09:55:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.481+0000 7f2dd0800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc819a0a0 con 0x7f2dc8073b00 2024-03-14T09:55:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.481+0000 7f2dcde00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dc80731e0 0x7f2dc819f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.481+0000 7f2dcd400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dc8073b00 0x7f2dc819fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.481+0000 7f2dcde00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dc80731e0 0x7f2dc819f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47140/0 (socket says 172.21.15.78:47140) 2024-03-14T09:55:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.481+0000 7f2dcde00700 1 -- 172.21.15.78:0/3645161350 learned_addr learned my addr 172.21.15.78:0/3645161350 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.481+0000 7f2dce800700 1 --2- 172.21.15.78:0/3645161350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dc810d500 0x7f2dc8199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.481+0000 7f2dcde00700 1 -- 172.21.15.78:0/3645161350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dc8073b00 msgr2=0x7f2dc819fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.481+0000 7f2dcde00700 1 --2- 172.21.15.78:0/3645161350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dc8073b00 0x7f2dc819fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.481+0000 7f2dcde00700 1 -- 172.21.15.78:0/3645161350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dc810d500 msgr2=0x7f2dc8199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.481+0000 7f2dcde00700 1 --2- 172.21.15.78:0/3645161350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dc810d500 0x7f2dc8199710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.481+0000 7f2dcde00700 1 -- 172.21.15.78:0/3645161350 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc4009d00 con 0x7f2dc80731e0 2024-03-14T09:55:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.481+0000 7f2dcde00700 1 --2- 172.21.15.78:0/3645161350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dc80731e0 0x7f2dc819f5e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc00e9a0 tx=0x7f2dbc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.482+0000 7f2dc2a00700 1 -- 172.21.15.78:0/3645161350 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc009790 con 0x7f2dc80731e0 2024-03-14T09:55:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.482+0000 7f2dc2a00700 1 -- 172.21.15.78:0/3645161350 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2dbc010430 con 0x7f2dc80731e0 2024-03-14T09:55:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.482+0000 7f2dd0800700 1 -- 172.21.15.78:0/3645161350 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db8009520 con 0x7f2dc80731e0 2024-03-14T09:55:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.482+0000 7f2dc2a00700 1 -- 172.21.15.78:0/3645161350 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc018680 con 0x7f2dc80731e0 2024-03-14T09:55:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.482+0000 7f2dd0800700 1 -- 172.21.15.78:0/3645161350 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc81a63f0 con 0x7f2dc80731e0 2024-03-14T09:55:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.485+0000 7f2dc2a00700 1 -- 172.21.15.78:0/3645161350 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 26) v1 ==== 135621+0+0 (secure 0 0 0) 0x7f2dbc0105a0 con 0x7f2dc80731e0 2024-03-14T09:55:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.486+0000 7f2de42c9b80 1 Processor -- start 2024-03-14T09:55:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.487+0000 7f2de42c9b80 1 -- start start 2024-03-14T09:55:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.487+0000 7f2de42c9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7de680 0x563bab7dea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.487+0000 7f2de42c9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7defa0 0x563bab7e8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.487+0000 7f2de42c9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7e8ec0 0x563bab7eb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.487+0000 7f2de42c9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bab737990 con 0x563bab7de680 2024-03-14T09:55:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.487+0000 7f2de42c9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bab735350 con 0x563bab7e8ec0 2024-03-14T09:55:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.487+0000 7f2de42c9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bab730c40 con 0x563bab7defa0 2024-03-14T09:55:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.487+0000 7f2dcde00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7defa0 0x563bab7e8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.487+0000 7f2dce800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7de680 0x563bab7dea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.487+0000 7f2dcde00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7defa0 0x563bab7e8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:56250/0 (socket says 172.21.15.78:56250) 2024-03-14T09:55:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.487+0000 7f2dce800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7de680 0x563bab7dea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47144/0 (socket says 172.21.15.78:47144) 2024-03-14T09:55:40.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.487+0000 7f2dcd400700 1 --2- 172.21.15.78:0/492021289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7e8ec0 0x563bab7eb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.487+0000 7f2dcde00700 1 -- 172.21.15.78:0/492021289 learned_addr learned my addr 172.21.15.78:0/492021289 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:40.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.488+0000 7f2dc2a00700 1 --2- 172.21.15.78:0/3645161350 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2db00a0d20 0x7f2db00a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.488+0000 7f2dc2a00700 1 -- 172.21.15.78:0/3645161350 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f2dbc014070 con 0x7f2dc80731e0 2024-03-14T09:55:40.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.488+0000 7f2dcd400700 1 -- 172.21.15.78:0/492021289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7defa0 msgr2=0x563bab7e8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:40.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.488+0000 7f2dcd400700 1 --2- 172.21.15.78:0/492021289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7defa0 0x563bab7e8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.488+0000 7f2dce800700 1 --2- 172.21.15.78:0/3645161350 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2db00a0d20 0x7f2db00a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.488+0000 7f2dcd400700 1 -- 172.21.15.78:0/492021289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7de680 msgr2=0x563bab7dea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:40.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.488+0000 7f2dcd400700 1 --2- 172.21.15.78:0/492021289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7de680 0x563bab7dea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.488+0000 7f2dcd400700 1 -- 172.21.15.78:0/492021289 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563bab5b23b0 con 0x563bab7e8ec0 2024-03-14T09:55:40.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.488+0000 7f2dcd400700 1 --2- 172.21.15.78:0/492021289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7e8ec0 0x563bab7eb2b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009fd0 tx=0x7f2db8005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2102bf72f2c393c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:40.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.489+0000 7f2dce800700 1 --2- 172.21.15.78:0/3645161350 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2db00a0d20 0x7f2db00a31e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009fd0 tx=0x7f2dc4004b40 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:40.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.489+0000 7f2db7000700 1 -- 172.21.15.78:0/492021289 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db800fdd0 con 0x563bab7e8ec0 2024-03-14T09:55:40.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.489+0000 7f2db7000700 1 -- 172.21.15.78:0/492021289 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2db801d040 con 0x563bab7e8ec0 2024-03-14T09:55:40.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.489+0000 7f2db7000700 1 -- 172.21.15.78:0/492021289 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db80052a0 con 0x563bab7e8ec0 2024-03-14T09:55:40.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.489+0000 7f2de42c9b80 1 -- 172.21.15.78:0/492021289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7e8ec0 msgr2=0x563bab7eb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:40.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.489+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/492021289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7e8ec0 0x563bab7eb2b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009fd0 tx=0x7f2db8005dc0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.490+0000 7f2de42c9b80 1 -- 172.21.15.78:0/492021289 shutdown_connections 2024-03-14T09:55:40.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.490+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/492021289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7e8ec0 0x563bab7eb2b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.490+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/492021289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7de680 0x563bab7dea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.490+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/492021289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7defa0 0x563bab7e8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.490+0000 7f2de42c9b80 1 -- 172.21.15.78:0/492021289 >> 172.21.15.78:0/492021289 conn(0x563bab754220 msgr2=0x563bab7d9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:40.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.490+0000 7f2de42c9b80 1 -- 172.21.15.78:0/492021289 shutdown_connections 2024-03-14T09:55:40.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.490+0000 7f2de42c9b80 1 -- 172.21.15.78:0/492021289 wait complete. 2024-03-14T09:55:40.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.490+0000 7f2de42c9b80 1 -- start start 2024-03-14T09:55:40.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.491+0000 7f2de42c9b80 1 Processor -- start 2024-03-14T09:55:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.491+0000 7f2de42c9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7de680 0x563bab7e8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.491+0000 7f2de42c9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7defa0 0x563bab7e5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.491+0000 7f2de42c9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7e8ec0 0x563bab7e62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.491+0000 7f2de42c9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bab6ccd80 con 0x563bab7defa0 2024-03-14T09:55:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.491+0000 7f2de42c9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bab6cca20 con 0x563bab7e8ec0 2024-03-14T09:55:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.491+0000 7f2de42c9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bab5b46a0 con 0x563bab7de680 2024-03-14T09:55:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.492+0000 7f2de42c9b80 10 client.0 _ll_register_callbacks cb 0x563bab5383a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.492+0000 7f2dcde00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7de680 0x563bab7e8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.492+0000 7f2dcd400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7defa0 0x563bab7e5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.492+0000 7f2dcde00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7de680 0x563bab7e8890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:56262/0 (socket says 172.21.15.78:56262) 2024-03-14T09:55:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.492+0000 7f2dcde00700 1 -- 172.21.15.78:0/3470918835 learned_addr learned my addr 172.21.15.78:0/3470918835 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.492+0000 7f2dce800700 1 --2- 172.21.15.78:0/3470918835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7e8ec0 0x563bab7e62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.492+0000 7f2dcd400700 1 -- 172.21.15.78:0/3470918835 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7de680 msgr2=0x563bab7e8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.492+0000 7f2dcd400700 1 --2- 172.21.15.78:0/3470918835 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7de680 0x563bab7e8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.492+0000 7f2dcd400700 1 -- 172.21.15.78:0/3470918835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7e8ec0 msgr2=0x563bab7e62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.492+0000 7f2dcd400700 1 --2- 172.21.15.78:0/3470918835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7e8ec0 0x563bab7e62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.492+0000 7f2dcd400700 1 -- 172.21.15.78:0/3470918835 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc8110630 con 0x563bab7defa0 2024-03-14T09:55:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.492+0000 7f2dcd400700 1 --2- 172.21.15.78:0/3470918835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7defa0 0x563bab7e5c20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009c40 tx=0x7f2db8009470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2f6e35091d06ac server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:40.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.492+0000 7f2dab400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:55:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.493+0000 7f2dab400700 1 -- 172.21.15.78:0/3470918835 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db800f460 con 0x563bab7defa0 2024-03-14T09:55:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.493+0000 7f2dab400700 1 -- 172.21.15.78:0/3470918835 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f2db800f5c0 con 0x563bab7defa0 2024-03-14T09:55:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.493+0000 7f2de42c9b80 1 -- 172.21.15.78:0/3470918835 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563bab7d1720 con 0x563bab7defa0 2024-03-14T09:55:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.493+0000 7f2dab400700 1 -- 172.21.15.78:0/3470918835 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2db800f780 con 0x563bab7defa0 2024-03-14T09:55:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.493+0000 7f2de42c9b80 10 client.15939 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.493+0000 7f2de42c9b80 1 -- 172.21.15.78:0/3470918835 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563bab55eb60 con 0x563bab7defa0 2024-03-14T09:55:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.493+0000 7f2dab400700 1 -- 172.21.15.78:0/3470918835 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563bab7d1720 con 0x563bab7defa0 2024-03-14T09:55:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.494+0000 7f2dab400700 1 -- 172.21.15.78:0/3470918835 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2db8015020 con 0x563bab7defa0 2024-03-14T09:55:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.494+0000 7f2de42c9b80 10 client.15939 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.494+0000 7f2de42c9b80 10 client.15939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:55:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.494+0000 7f2de42c9b80 20 client.15939 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.494+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/3470918835 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563bab7f8650 0x563bab7faa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.494+0000 7f2de42c9b80 4 client.15939 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:55:40.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.494+0000 7f2de42c9b80 1 -- 172.21.15.78:0/3470918835 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563bab729700 con 0x563bab7f8650 2024-03-14T09:55:40.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.494+0000 7f2dcde00700 1 --2- 172.21.15.78:0/3470918835 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563bab7f8650 0x563bab7faa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.494+0000 7f2dab400700 10 client.15939 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:40.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.494+0000 7f2dcde00700 1 --2- 172.21.15.78:0/3470918835 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563bab7f8650 0x563bab7faa70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2acff64ecb5eac9 server_cookie=e8fecf21bda9c550 in_seq=0 out_seq=0 2024-03-14T09:55:40.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.495+0000 7f2dab400700 1 -- 172.21.15.78:0/3470918835 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x563bab729700 con 0x563bab7f8650 2024-03-14T09:55:40.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.495+0000 7f2dab400700 10 client.15939 handle_command_reply: tid=0 2024-03-14T09:55:40.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.495+0000 7f2de42c9b80 1 client.15939 shutdown 2024-03-14T09:55:40.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.495+0000 7f2de42c9b80 1 -- 172.21.15.78:0/3470918835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7defa0 msgr2=0x563bab7e5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:40.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.495+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/3470918835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7defa0 0x563bab7e5c20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2db8009c40 tx=0x7f2db8009470 comp rx=0 tx=0).stop 2024-03-14T09:55:40.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.496+0000 7f2de42c9b80 1 -- 172.21.15.78:0/3470918835 shutdown_connections 2024-03-14T09:55:40.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.496+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/3470918835 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563bab7f8650 0x563bab7faa70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.496+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/3470918835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7e8ec0 0x563bab7e62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.496+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/3470918835 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7de680 0x563bab7e8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.496+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/3470918835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7defa0 0x563bab7e5c20 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.496+0000 7f2de42c9b80 1 -- 172.21.15.78:0/3470918835 >> 172.21.15.78:0/3470918835 conn(0x563bab754220 msgr2=0x563bab7d9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.496+0000 7f2dab400700 0 client.15939 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.496+0000 7f2de42c9b80 1 -- 172.21.15.78:0/3470918835 shutdown_connections 2024-03-14T09:55:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.496+0000 7f2de42c9b80 1 -- 172.21.15.78:0/3470918835 wait complete. 2024-03-14T09:55:40.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.496+0000 7f2de42c9b80 20 client.15939 trim_cache size 0 max 16384 2024-03-14T09:55:40.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.517+0000 7f2de42c9b80 1 Processor -- start 2024-03-14T09:55:40.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.518+0000 7f2de42c9b80 1 -- start start 2024-03-14T09:55:40.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.518+0000 7f2de42c9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7de680 0x563bab7d9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.518+0000 7f2de42c9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7defa0 0x563bab7efae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.518+0000 7f2de42c9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7e8ec0 0x563bab7e2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.518+0000 7f2de42c9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bab7e6b80 con 0x563bab7defa0 2024-03-14T09:55:40.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.518+0000 7f2de42c9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bab7f55e0 con 0x563bab7de680 2024-03-14T09:55:40.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.518+0000 7f2de42c9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bab7d6230 con 0x563bab7e8ec0 2024-03-14T09:55:40.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.518+0000 7f2dcde00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7de680 0x563bab7d9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.518+0000 7f2dcd400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7defa0 0x563bab7efae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.518+0000 7f2dcde00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7de680 0x563bab7d9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:59876/0 (socket says 172.21.15.78:59876) 2024-03-14T09:55:40.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.518+0000 7f2dcde00700 1 -- 172.21.15.78:0/4023838600 learned_addr learned my addr 172.21.15.78:0/4023838600 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:40.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.518+0000 7f2dce800700 1 --2- 172.21.15.78:0/4023838600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7e8ec0 0x563bab7e2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.519+0000 7f2dcde00700 1 -- 172.21.15.78:0/4023838600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7e8ec0 msgr2=0x563bab7e2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:40.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.519+0000 7f2dcde00700 1 --2- 172.21.15.78:0/4023838600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7e8ec0 0x563bab7e2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.519+0000 7f2dcde00700 1 -- 172.21.15.78:0/4023838600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7defa0 msgr2=0x563bab7efae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:40.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.519+0000 7f2dcde00700 1 --2- 172.21.15.78:0/4023838600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7defa0 0x563bab7efae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.519+0000 7f2dcde00700 1 -- 172.21.15.78:0/4023838600 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563bab6f6ea0 con 0x563bab7de680 2024-03-14T09:55:40.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.519+0000 7f2dcde00700 1 --2- 172.21.15.78:0/4023838600 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7de680 0x563bab7d9c30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc095ec0 tx=0x7f2dbc010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab44eede482b630a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:40.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.519+0000 7f2db7a00700 1 -- 172.21.15.78:0/4023838600 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc09a0c0 con 0x563bab7de680 2024-03-14T09:55:40.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.519+0000 7f2db7a00700 1 -- 172.21.15.78:0/4023838600 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2dbc0d3070 con 0x563bab7de680 2024-03-14T09:55:40.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.519+0000 7f2db7a00700 1 -- 172.21.15.78:0/4023838600 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc099c20 con 0x563bab7de680 2024-03-14T09:55:40.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.520+0000 7f2de42c9b80 1 -- 172.21.15.78:0/4023838600 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7de680 msgr2=0x563bab7d9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:40.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.520+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/4023838600 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7de680 0x563bab7d9c30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc095ec0 tx=0x7f2dbc010c80 comp rx=0 tx=0).stop 2024-03-14T09:55:40.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.520+0000 7f2de42c9b80 1 -- 172.21.15.78:0/4023838600 shutdown_connections 2024-03-14T09:55:40.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.520+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/4023838600 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7de680 0x563bab7d9c30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.520+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/4023838600 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7e8ec0 0x563bab7e2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.520+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/4023838600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7defa0 0x563bab7efae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.520+0000 7f2de42c9b80 1 -- 172.21.15.78:0/4023838600 >> 172.21.15.78:0/4023838600 conn(0x563bab754220 msgr2=0x563bab7e44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:40.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.520+0000 7f2de42c9b80 1 -- 172.21.15.78:0/4023838600 shutdown_connections 2024-03-14T09:55:40.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.520+0000 7f2de42c9b80 1 -- 172.21.15.78:0/4023838600 wait complete. 2024-03-14T09:55:40.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.521+0000 7f2de42c9b80 1 -- start start 2024-03-14T09:55:40.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.521+0000 7f2de42c9b80 1 Processor -- start 2024-03-14T09:55:40.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.521+0000 7f2de42c9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7de680 0x563bab7ef7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.521+0000 7f2de42c9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7defa0 0x563bab7edaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.521+0000 7f2de42c9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7e8ec0 0x563bab7ee030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.521+0000 7f2de42c9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bab7ee5b0 con 0x563bab7e8ec0 2024-03-14T09:55:40.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.521+0000 7f2de42c9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bab7ee6f0 con 0x563bab7de680 2024-03-14T09:55:40.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.521+0000 7f2de42c9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bab7ee880 con 0x563bab7defa0 2024-03-14T09:55:40.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.521+0000 7f2de42c9b80 10 client.0 _ll_register_callbacks cb 0x7f2dc4009050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:40.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.522+0000 7f2dcd400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7defa0 0x563bab7edaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.522+0000 7f2dcd400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7defa0 0x563bab7edaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:56282/0 (socket says 172.21.15.78:56282) 2024-03-14T09:55:40.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.522+0000 7f2dcd400700 1 -- 172.21.15.78:0/3177699417 learned_addr learned my addr 172.21.15.78:0/3177699417 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:40.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.522+0000 7f2dcde00700 1 --2- 172.21.15.78:0/3177699417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7de680 0x563bab7ef7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.522+0000 7f2dce800700 1 --2- 172.21.15.78:0/3177699417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7e8ec0 0x563bab7ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.522+0000 7f2dcd400700 1 -- 172.21.15.78:0/3177699417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7de680 msgr2=0x563bab7ef7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:40.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.522+0000 7f2dcd400700 1 --2- 172.21.15.78:0/3177699417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7de680 0x563bab7ef7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.522+0000 7f2dcd400700 1 -- 172.21.15.78:0/3177699417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7e8ec0 msgr2=0x563bab7ee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:40.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.522+0000 7f2dcd400700 1 --2- 172.21.15.78:0/3177699417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7e8ec0 0x563bab7ee030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.522+0000 7f2dcd400700 1 -- 172.21.15.78:0/3177699417 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc81a63f0 con 0x563bab7defa0 2024-03-14T09:55:40.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.522+0000 7f2dcde00700 1 --2- 172.21.15.78:0/3177699417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7de680 0x563bab7ef7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:40.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.523+0000 7f2dcd400700 1 --2- 172.21.15.78:0/3177699417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7defa0 0x563bab7edaf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2db80059d0 tx=0x7f2db800b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f314d339ec409cf server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:40.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.523+0000 7f2dab400700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:55:40.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.523+0000 7f2dab400700 1 -- 172.21.15.78:0/3177699417 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db800f660 con 0x563bab7defa0 2024-03-14T09:55:40.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.523+0000 7f2de42c9b80 1 -- 172.21.15.78:0/3177699417 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2db801d040 con 0x563bab7defa0 2024-03-14T09:55:40.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.523+0000 7f2dab400700 1 -- 172.21.15.78:0/3177699417 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f2db8000e40 con 0x563bab7defa0 2024-03-14T09:55:40.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.523+0000 7f2dab400700 1 -- 172.21.15.78:0/3177699417 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2db8003d10 con 0x563bab7defa0 2024-03-14T09:55:40.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.523+0000 7f2de42c9b80 10 client.25957 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:40.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.523+0000 7f2de42c9b80 1 -- 172.21.15.78:0/3177699417 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563bab5b6cd0 con 0x563bab7defa0 2024-03-14T09:55:40.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.523+0000 7f2dab400700 1 -- 172.21.15.78:0/3177699417 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2db801d040 con 0x563bab7defa0 2024-03-14T09:55:40.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.524+0000 7f2dab400700 1 -- 172.21.15.78:0/3177699417 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2db8009920 con 0x563bab7defa0 2024-03-14T09:55:40.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.524+0000 7f2de42c9b80 10 client.25957 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:40.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.524+0000 7f2de42c9b80 10 client.25957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:55:40.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.524+0000 7f2de42c9b80 20 client.25957 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:40.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.524+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/3177699417 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563bab84b210 0x563bab84d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.524+0000 7f2de42c9b80 4 client.25957 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:55:40.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.524+0000 7f2de42c9b80 1 -- 172.21.15.78:0/3177699417 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563bab57fcf0 con 0x563bab84b210 2024-03-14T09:55:40.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.524+0000 7f2dcde00700 1 --2- 172.21.15.78:0/3177699417 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563bab84b210 0x563bab84d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.524+0000 7f2dcde00700 1 --2- 172.21.15.78:0/3177699417 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563bab84b210 0x563bab84d600 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b81354c499f9d84e server_cookie=181a3e5a657eb009 in_seq=0 out_seq=0 2024-03-14T09:55:40.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.524+0000 7f2dab400700 10 client.25957 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:40.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.525+0000 7f2dab400700 1 -- 172.21.15.78:0/3177699417 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x563bab57fcf0 con 0x563bab84b210 2024-03-14T09:55:40.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.525+0000 7f2dab400700 10 client.25957 handle_command_reply: tid=0 2024-03-14T09:55:40.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.525+0000 7f2de42c9b80 1 client.25957 shutdown 2024-03-14T09:55:40.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.525+0000 7f2de42c9b80 1 -- 172.21.15.78:0/3177699417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7defa0 msgr2=0x563bab7edaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:40.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.525+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/3177699417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7defa0 0x563bab7edaf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2db80059d0 tx=0x7f2db800b980 comp rx=0 tx=0).stop 2024-03-14T09:55:40.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.525+0000 7f2de42c9b80 1 -- 172.21.15.78:0/3177699417 shutdown_connections 2024-03-14T09:55:40.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.525+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/3177699417 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563bab84b210 0x563bab84d600 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.525+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/3177699417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563bab7de680 0x563bab7ef7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.525+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/3177699417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563bab7e8ec0 0x563bab7ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.525+0000 7f2de42c9b80 1 --2- 172.21.15.78:0/3177699417 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563bab7defa0 0x563bab7edaf0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.525+0000 7f2de42c9b80 1 -- 172.21.15.78:0/3177699417 >> 172.21.15.78:0/3177699417 conn(0x563bab754220 msgr2=0x563bab7e5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:40.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.525+0000 7f2dab400700 0 client.25957 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:40.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.525+0000 7f2de42c9b80 1 -- 172.21.15.78:0/3177699417 shutdown_connections 2024-03-14T09:55:40.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.525+0000 7f2de42c9b80 1 -- 172.21.15.78:0/3177699417 wait complete. 2024-03-14T09:55:40.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.525+0000 7f2de42c9b80 20 client.25957 trim_cache size 0 max 16384 2024-03-14T09:55:40.530 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:55:40.530 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T09:55:40.530 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T09:55:40.530 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:55:40.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.526+0000 7f2dd0800700 1 -- 172.21.15.78:0/3645161350 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2db00a0d20 msgr2=0x7f2db00a31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:40.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.526+0000 7f2dd0800700 1 --2- 172.21.15.78:0/3645161350 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2db00a0d20 0x7f2db00a31e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009fd0 tx=0x7f2dc4004b40 comp rx=0 tx=0).stop 2024-03-14T09:55:40.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.526+0000 7f2dd0800700 1 -- 172.21.15.78:0/3645161350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dc80731e0 msgr2=0x7f2dc819f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:40.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.526+0000 7f2dd0800700 1 --2- 172.21.15.78:0/3645161350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dc80731e0 0x7f2dc819f5e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc00e9a0 tx=0x7f2dbc00ecb0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.526+0000 7f2dd0800700 1 -- 172.21.15.78:0/3645161350 shutdown_connections 2024-03-14T09:55:40.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.526+0000 7f2dd0800700 1 --2- 172.21.15.78:0/3645161350 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2db00a0d20 0x7f2db00a31e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.526+0000 7f2dd0800700 1 --2- 172.21.15.78:0/3645161350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2dc810d500 0x7f2dc8199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.526+0000 7f2dd0800700 1 --2- 172.21.15.78:0/3645161350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2dc80731e0 0x7f2dc819f5e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.526+0000 7f2dd0800700 1 --2- 172.21.15.78:0/3645161350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2dc8073b00 0x7f2dc819fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.526+0000 7f2dd0800700 1 -- 172.21.15.78:0/3645161350 >> 172.21.15.78:0/3645161350 conn(0x7f2dc80fc7b0 msgr2=0x7f2dc80fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:40.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.526+0000 7f2dd0800700 1 -- 172.21.15.78:0/3645161350 shutdown_connections 2024-03-14T09:55:40.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.526+0000 7f2dd0800700 1 -- 172.21.15.78:0/3645161350 wait complete. 2024-03-14T09:55:40.532 INFO:tasks.workunit.client.0.smithi078.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2024-03-14T09:55:40.532 INFO:tasks.workunit.client.0.smithi078.stdout:libtoolize: copying file 'm4/libtool.m4' 2024-03-14T09:55:40.544 INFO:tasks.workunit.client.0.smithi078.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2024-03-14T09:55:40.546 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T09:55:40.546 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:55:40.558 INFO:tasks.workunit.client.0.smithi078.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2024-03-14T09:55:40.570 INFO:tasks.workunit.client.0.smithi078.stdout:libtoolize: copying file 'm4/ltversion.m4' 2024-03-14T09:55:40.583 INFO:tasks.workunit.client.0.smithi078.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2024-03-14T09:55:40.600 INFO:tasks.workunit.client.0.smithi078.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2024-03-14T09:55:40.603 INFO:tasks.workunit.client.0.smithi078.stdout:cp include/install-sh . 2024-03-14T09:55:40.615 INFO:tasks.workunit.client.0.smithi078.stdout:aclocal -I m4 2024-03-14T09:55:40.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.745+0000 7ff1df800700 1 -- 172.21.15.78:0/3616643641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d8103040 msgr2=0x7ff1d8103420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:40.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.745+0000 7ff1df800700 1 --2- 172.21.15.78:0/3616643641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d8103040 0x7ff1d8103420 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc009990 tx=0x7ff1cc009ca0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.745+0000 7ff1df800700 1 -- 172.21.15.78:0/3616643641 shutdown_connections 2024-03-14T09:55:40.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.745+0000 7ff1df800700 1 --2- 172.21.15.78:0/3616643641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1d8108160 0x7ff1d810fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.745+0000 7ff1df800700 1 --2- 172.21.15.78:0/3616643641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d8103040 0x7ff1d8103420 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.745+0000 7ff1df800700 1 --2- 172.21.15.78:0/3616643641 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1d81039f0 0x7ff1d8107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.745+0000 7ff1df800700 1 -- 172.21.15.78:0/3616643641 >> 172.21.15.78:0/3616643641 conn(0x7ff1d80fe8a0 msgr2=0x7ff1d8100cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:40.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.746+0000 7ff1df800700 1 -- 172.21.15.78:0/3616643641 shutdown_connections 2024-03-14T09:55:40.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.746+0000 7ff1df800700 1 -- 172.21.15.78:0/3616643641 wait complete. 2024-03-14T09:55:40.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.747+0000 7ff1df800700 1 Processor -- start 2024-03-14T09:55:40.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.747+0000 7ff1df800700 1 -- start start 2024-03-14T09:55:40.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.747+0000 7ff1df800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1d8103040 0x7ff1d81997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.747+0000 7ff1df800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1d8108160 0x7ff1d8199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.747+0000 7ff1df800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d819cfb0 0x7ff1d819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.747+0000 7ff1df800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d819d9a0 con 0x7ff1d819cfb0 2024-03-14T09:55:40.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.747+0000 7ff1df800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d819db10 con 0x7ff1d8103040 2024-03-14T09:55:40.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.747+0000 7ff1df800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d819dcd0 con 0x7ff1d8108160 2024-03-14T09:55:40.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.747+0000 7ff1dd800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d819cfb0 0x7ff1d819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.747+0000 7ff1dce00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1d8103040 0x7ff1d81997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.747+0000 7ff1dce00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1d8103040 0x7ff1d81997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:59902/0 (socket says 172.21.15.78:59902) 2024-03-14T09:55:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.747+0000 7ff1dd800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d819cfb0 0x7ff1d819d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47172/0 (socket says 172.21.15.78:47172) 2024-03-14T09:55:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.747+0000 7ff1dce00700 1 -- 172.21.15.78:0/715511122 learned_addr learned my addr 172.21.15.78:0/715511122 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.747+0000 7ff1d7e00700 1 --2- 172.21.15.78:0/715511122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1d8108160 0x7ff1d8199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.748+0000 7ff1dce00700 1 -- 172.21.15.78:0/715511122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1d8108160 msgr2=0x7ff1d8199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.748+0000 7ff1dce00700 1 --2- 172.21.15.78:0/715511122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1d8108160 0x7ff1d8199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.748+0000 7ff1dce00700 1 -- 172.21.15.78:0/715511122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d819cfb0 msgr2=0x7ff1d819d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.748+0000 7ff1dce00700 1 --2- 172.21.15.78:0/715511122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d819cfb0 0x7ff1d819d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.748+0000 7ff1dce00700 1 -- 172.21.15.78:0/715511122 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1cc0095f0 con 0x7ff1d8103040 2024-03-14T09:55:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.748+0000 7ff1d7e00700 1 --2- 172.21.15.78:0/715511122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1d8108160 0x7ff1d8199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:55:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.748+0000 7ff1dd800700 1 --2- 172.21.15.78:0/715511122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d819cfb0 0x7ff1d819d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.748+0000 7ff1dce00700 1 --2- 172.21.15.78:0/715511122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1d8103040 0x7ff1d81997d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc000c00 tx=0x7ff1cc005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.748+0000 7ff1d5600700 1 -- 172.21.15.78:0/715511122 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc01d070 con 0x7ff1d8103040 2024-03-14T09:55:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.748+0000 7ff1d5600700 1 -- 172.21.15.78:0/715511122 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1cc022470 con 0x7ff1d8103040 2024-03-14T09:55:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.748+0000 7ff1d5600700 1 -- 172.21.15.78:0/715511122 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc017650 con 0x7ff1d8103040 2024-03-14T09:55:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.748+0000 7ff1df800700 1 -- 172.21.15.78:0/715511122 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d81a6340 con 0x7ff1d8103040 2024-03-14T09:55:40.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.748+0000 7ff1df800700 1 -- 172.21.15.78:0/715511122 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d81a6710 con 0x7ff1d8103040 2024-03-14T09:55:40.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.749+0000 7ff1df800700 1 -- 172.21.15.78:0/715511122 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d810b3d0 con 0x7ff1d8103040 2024-03-14T09:55:40.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.751+0000 7ff1d5600700 1 -- 172.21.15.78:0/715511122 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) v1 ==== 135621+0+0 (secure 0 0 0) 0x7ff1cc0225e0 con 0x7ff1d8103040 2024-03-14T09:55:40.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.752+0000 7ff1d5600700 1 --2- 172.21.15.78:0/715511122 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff1c00a0d50 0x7ff1c00a3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:40.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.752+0000 7ff1d5600700 1 -- 172.21.15.78:0/715511122 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7ff1cc0ce190 con 0x7ff1d8103040 2024-03-14T09:55:40.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.753+0000 7ff1d7e00700 1 --2- 172.21.15.78:0/715511122 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff1c00a0d50 0x7ff1c00a3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:40.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.753+0000 7ff1d7e00700 1 --2- 172.21.15.78:0/715511122 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff1c00a0d50 0x7ff1c00a3210 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff1d819acd0 tx=0x7ff1c4009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:40.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:40.755+0000 7ff1d5600700 1 -- 172.21.15.78:0/715511122 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff1cc0972f0 con 0x7ff1d8103040 2024-03-14T09:55:41.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.030+0000 7ff1df800700 1 -- 172.21.15.78:0/715511122 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1d819ab20 con 0x7ff1d8103040 2024-03-14T09:55:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.031+0000 7ff1d5600700 1 -- 172.21.15.78:0/715511122 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7ff1cc09a910 con 0x7ff1d8103040 2024-03-14T09:55:41.032 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:55:41.032 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:55:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.034+0000 7ff1df800700 1 -- 172.21.15.78:0/715511122 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff1c00a0d50 msgr2=0x7ff1c00a3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.034+0000 7ff1df800700 1 --2- 172.21.15.78:0/715511122 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff1c00a0d50 0x7ff1c00a3210 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff1d819acd0 tx=0x7ff1c4009260 comp rx=0 tx=0).stop 2024-03-14T09:55:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.034+0000 7ff1df800700 1 -- 172.21.15.78:0/715511122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1d8103040 msgr2=0x7ff1d81997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.034+0000 7ff1df800700 1 --2- 172.21.15.78:0/715511122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1d8103040 0x7ff1d81997d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc000c00 tx=0x7ff1cc005740 comp rx=0 tx=0).stop 2024-03-14T09:55:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.034+0000 7ff1df800700 1 -- 172.21.15.78:0/715511122 shutdown_connections 2024-03-14T09:55:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.034+0000 7ff1df800700 1 --2- 172.21.15.78:0/715511122 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff1c00a0d50 0x7ff1c00a3210 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.034+0000 7ff1df800700 1 --2- 172.21.15.78:0/715511122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1d8103040 0x7ff1d81997d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.034+0000 7ff1df800700 1 --2- 172.21.15.78:0/715511122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1d819cfb0 0x7ff1d819d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.034+0000 7ff1df800700 1 --2- 172.21.15.78:0/715511122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff1d8108160 0x7ff1d8199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.034+0000 7ff1df800700 1 -- 172.21.15.78:0/715511122 >> 172.21.15.78:0/715511122 conn(0x7ff1d80fe8a0 msgr2=0x7ff1d80ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.034+0000 7ff1df800700 1 -- 172.21.15.78:0/715511122 shutdown_connections 2024-03-14T09:55:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.034+0000 7ff1df800700 1 -- 172.21.15.78:0/715511122 wait complete. 2024-03-14T09:55:41.036 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T09:55:41.053 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T09:55:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.294+0000 7fa567a00700 1 -- 172.21.15.78:0/2092526844 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5600731e0 msgr2=0x7fa5600735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.294+0000 7fa567a00700 1 --2- 172.21.15.78:0/2092526844 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5600731e0 0x7fa5600735c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa5540098f0 tx=0x7fa554009c00 comp rx=0 tx=0).stop 2024-03-14T09:55:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.294+0000 7fa567a00700 1 -- 172.21.15.78:0/2092526844 shutdown_connections 2024-03-14T09:55:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.294+0000 7fa567a00700 1 --2- 172.21.15.78:0/2092526844 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa5600731e0 0x7fa5600735c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.294+0000 7fa567a00700 1 --2- 172.21.15.78:0/2092526844 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa56010d500 0x7fa56010f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.294+0000 7fa567a00700 1 --2- 172.21.15.78:0/2092526844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa560073b00 0x7fa56010cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.294+0000 7fa567a00700 1 -- 172.21.15.78:0/2092526844 >> 172.21.15.78:0/2092526844 conn(0x7fa5600fc7b0 msgr2=0x7fa5600febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.294+0000 7fa567a00700 1 -- 172.21.15.78:0/2092526844 shutdown_connections 2024-03-14T09:55:41.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.294+0000 7fa567a00700 1 -- 172.21.15.78:0/2092526844 wait complete. 2024-03-14T09:55:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.295+0000 7fa567a00700 1 Processor -- start 2024-03-14T09:55:41.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.296+0000 7fa567a00700 1 -- start start 2024-03-14T09:55:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.296+0000 7fa567a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5600731e0 0x7fa56019f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.296+0000 7fa567a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa560073b00 0x7fa56019fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.296+0000 7fa567a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa56010d500 0x7fa560199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.296+0000 7fa567a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa560199da0 con 0x7fa56010d500 2024-03-14T09:55:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.296+0000 7fa567a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa560199f10 con 0x7fa560073b00 2024-03-14T09:55:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.296+0000 7fa567a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56019a0d0 con 0x7fa5600731e0 2024-03-14T09:55:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.297+0000 7fa55fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa560073b00 0x7fa56019fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.297+0000 7fa55fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa560073b00 0x7fa56019fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:59920/0 (socket says 172.21.15.78:59920) 2024-03-14T09:55:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.297+0000 7fa565000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5600731e0 0x7fa56019f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.297+0000 7fa55fe00700 1 -- 172.21.15.78:0/1253918978 learned_addr learned my addr 172.21.15.78:0/1253918978 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.297+0000 7fa565a00700 1 --2- 172.21.15.78:0/1253918978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa56010d500 0x7fa560199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.297+0000 7fa55fe00700 1 -- 172.21.15.78:0/1253918978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5600731e0 msgr2=0x7fa56019f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.297+0000 7fa55fe00700 1 --2- 172.21.15.78:0/1253918978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5600731e0 0x7fa56019f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.297+0000 7fa55fe00700 1 -- 172.21.15.78:0/1253918978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa56010d500 msgr2=0x7fa560199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.297+0000 7fa55fe00700 1 --2- 172.21.15.78:0/1253918978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa56010d500 0x7fa560199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.297+0000 7fa55fe00700 1 -- 172.21.15.78:0/1253918978 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c0095f0 con 0x7fa560073b00 2024-03-14T09:55:41.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.297+0000 7fa55fe00700 1 --2- 172.21.15.78:0/1253918978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa560073b00 0x7fa56019fb50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa54c00ec10 tx=0x7fa54c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.297+0000 7fa55d600700 1 -- 172.21.15.78:0/1253918978 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54c00cc90 con 0x7fa560073b00 2024-03-14T09:55:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.297+0000 7fa55d600700 1 -- 172.21.15.78:0/1253918978 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa54c0044e0 con 0x7fa560073b00 2024-03-14T09:55:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.297+0000 7fa567a00700 1 -- 172.21.15.78:0/1253918978 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa554009520 con 0x7fa560073b00 2024-03-14T09:55:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.298+0000 7fa55d600700 1 -- 172.21.15.78:0/1253918978 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54c010430 con 0x7fa560073b00 2024-03-14T09:55:41.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.298+0000 7fa567a00700 1 -- 172.21.15.78:0/1253918978 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56019a5a0 con 0x7fa560073b00 2024-03-14T09:55:41.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.301+0000 7fa55d600700 1 -- 172.21.15.78:0/1253918978 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) v1 ==== 135621+0+0 (secure 0 0 0) 0x7fa54c003fb0 con 0x7fa560073b00 2024-03-14T09:55:41.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.302+0000 7fa57b501b80 1 Processor -- start 2024-03-14T09:55:41.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.302+0000 7fa57b501b80 1 -- start start 2024-03-14T09:55:41.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.302+0000 7fa55d600700 1 --2- 172.21.15.78:0/1253918978 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa5480a0da0 0x7fa5480a3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.302+0000 7fa57b501b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8680 0x5574e26e8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.302+0000 7fa57b501b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26e8fa0 0x5574e26f2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.302+0000 7fa55d600700 1 -- 172.21.15.78:0/1253918978 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fa54c014070 con 0x7fa560073b00 2024-03-14T09:55:41.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.302+0000 7fa57b501b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26f2ec0 0x5574e26f52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.302+0000 7fa57b501b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e2641990 con 0x5574e26e8fa0 2024-03-14T09:55:41.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.302+0000 7fa57b501b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e263f350 con 0x5574e26f2ec0 2024-03-14T09:55:41.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.302+0000 7fa57b501b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e263ac40 con 0x5574e26e8680 2024-03-14T09:55:41.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.303+0000 7fa565a00700 1 --2- 172.21.15.78:0/1253918978 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa5480a0da0 0x7fa5480a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.303+0000 7fa55fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26e8fa0 0x5574e26f2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.303+0000 7fa55fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26e8fa0 0x5574e26f2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47200/0 (socket says 172.21.15.78:47200) 2024-03-14T09:55:41.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.303+0000 7fa55fe00700 1 -- 172.21.15.78:0/1133900195 learned_addr learned my addr 172.21.15.78:0/1133900195 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:41.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.303+0000 7fa565a00700 1 --2- 172.21.15.78:0/1133900195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26f2ec0 0x5574e26f52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.303+0000 7fa565000700 1 --2- 172.21.15.78:0/1133900195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8680 0x5574e26e8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.303+0000 7fa55fe00700 1 -- 172.21.15.78:0/1133900195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8680 msgr2=0x5574e26e8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.303+0000 7fa55fe00700 1 --2- 172.21.15.78:0/1133900195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8680 0x5574e26e8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.303+0000 7fa55fe00700 1 -- 172.21.15.78:0/1133900195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26f2ec0 msgr2=0x5574e26f52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.303+0000 7fa55fe00700 1 --2- 172.21.15.78:0/1133900195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26f2ec0 0x5574e26f52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.305 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:41 smithi078 ceph-mon[38143]: pgmap v48: 129 pgs: 129 active+clean; 38 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 40 op/s 2024-03-14T09:55:41.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.303+0000 7fa55fe00700 1 -- 172.21.15.78:0/1133900195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574e24bc3b0 con 0x5574e26e8fa0 2024-03-14T09:55:41.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.303+0000 7fa565000700 1 --2- 172.21.15.78:0/1133900195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8680 0x5574e26e8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:55:41.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.304+0000 7fa565a00700 1 --2- 172.21.15.78:0/1253918978 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa5480a0da0 0x7fa5480a3260 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa558009ec0 tx=0x7fa558013250 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.304+0000 7fa55fe00700 1 --2- 172.21.15.78:0/1133900195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26e8fa0 0x5574e26f2980 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0a1c10 tx=0x7fa54c099800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a1915ea2178142e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.304+0000 7fa552000700 1 -- 172.21.15.78:0/1133900195 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54c0b1070 con 0x5574e26e8fa0 2024-03-14T09:55:41.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.304+0000 7fa552000700 1 -- 172.21.15.78:0/1133900195 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa54c099cc0 con 0x5574e26e8fa0 2024-03-14T09:55:41.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.304+0000 7fa552000700 1 -- 172.21.15.78:0/1133900195 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54c003780 con 0x5574e26e8fa0 2024-03-14T09:55:41.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.305+0000 7fa57b501b80 1 -- 172.21.15.78:0/1133900195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26e8fa0 msgr2=0x5574e26f2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.305+0000 7fa57b501b80 1 --2- 172.21.15.78:0/1133900195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26e8fa0 0x5574e26f2980 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0a1c10 tx=0x7fa54c099800 comp rx=0 tx=0).stop 2024-03-14T09:55:41.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.305+0000 7fa57b501b80 1 -- 172.21.15.78:0/1133900195 shutdown_connections 2024-03-14T09:55:41.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.305+0000 7fa57b501b80 1 --2- 172.21.15.78:0/1133900195 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26f2ec0 0x5574e26f52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.305+0000 7fa57b501b80 1 --2- 172.21.15.78:0/1133900195 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8680 0x5574e26e8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.305+0000 7fa57b501b80 1 --2- 172.21.15.78:0/1133900195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26e8fa0 0x5574e26f2980 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.305+0000 7fa57b501b80 1 -- 172.21.15.78:0/1133900195 >> 172.21.15.78:0/1133900195 conn(0x5574e265e220 msgr2=0x5574e26e3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.305+0000 7fa57b501b80 1 -- 172.21.15.78:0/1133900195 shutdown_connections 2024-03-14T09:55:41.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.305+0000 7fa57b501b80 1 -- 172.21.15.78:0/1133900195 wait complete. 2024-03-14T09:55:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.305+0000 7fa57b501b80 1 -- start start 2024-03-14T09:55:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.306+0000 7fa57b501b80 1 Processor -- start 2024-03-14T09:55:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.306+0000 7fa57b501b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8680 0x5574e26f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.307+0000 7fa57b501b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8fa0 0x5574e26efc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.307+0000 7fa57b501b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26f2ec0 0x5574e26f02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.307+0000 7fa57b501b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e25d6d80 con 0x5574e26f2ec0 2024-03-14T09:55:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.307+0000 7fa57b501b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e25d6a20 con 0x5574e26e8680 2024-03-14T09:55:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.307+0000 7fa57b501b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e24be6a0 con 0x5574e26e8fa0 2024-03-14T09:55:41.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.307+0000 7fa57b501b80 10 client.0 _ll_register_callbacks cb 0x5574e24423a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:41.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.307+0000 7fa565000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8680 0x5574e26f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.307+0000 7fa565000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8680 0x5574e26f28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:59940/0 (socket says 172.21.15.78:59940) 2024-03-14T09:55:41.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.307+0000 7fa565000700 1 -- 172.21.15.78:0/3535518457 learned_addr learned my addr 172.21.15.78:0/3535518457 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:41.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.307+0000 7fa55fe00700 1 --2- 172.21.15.78:0/3535518457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8fa0 0x5574e26efc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.307+0000 7fa565a00700 1 --2- 172.21.15.78:0/3535518457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26f2ec0 0x5574e26f02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.307+0000 7fa565000700 1 -- 172.21.15.78:0/3535518457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8fa0 msgr2=0x5574e26efc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.307+0000 7fa565000700 1 --2- 172.21.15.78:0/3535518457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8fa0 0x5574e26efc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.307+0000 7fa565000700 1 -- 172.21.15.78:0/3535518457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26f2ec0 msgr2=0x5574e26f02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.307+0000 7fa565000700 1 --2- 172.21.15.78:0/3535518457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26f2ec0 0x5574e26f02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.307+0000 7fa565000700 1 -- 172.21.15.78:0/3535518457 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56019a5a0 con 0x5574e26e8680 2024-03-14T09:55:41.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.308+0000 7fa565000700 1 --2- 172.21.15.78:0/3535518457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8680 0x5574e26f28d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa554009230 tx=0x7fa55400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=818f65d5d49200a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.308+0000 7fa546a00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:55:41.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.308+0000 7fa546a00700 1 -- 172.21.15.78:0/3535518457 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55401c070 con 0x5574e26e8680 2024-03-14T09:55:41.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.308+0000 7fa57b501b80 1 -- 172.21.15.78:0/3535518457 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5574e26db720 con 0x5574e26e8680 2024-03-14T09:55:41.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.308+0000 7fa546a00700 1 -- 172.21.15.78:0/3535518457 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fa55400b760 con 0x5574e26e8680 2024-03-14T09:55:41.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.308+0000 7fa546a00700 1 -- 172.21.15.78:0/3535518457 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa55400b920 con 0x5574e26e8680 2024-03-14T09:55:41.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.308+0000 7fa57b501b80 10 client.25967 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:41.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.308+0000 7fa57b501b80 1 -- 172.21.15.78:0/3535518457 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5574e2468b60 con 0x5574e26e8680 2024-03-14T09:55:41.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.309+0000 7fa546a00700 1 -- 172.21.15.78:0/3535518457 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5574e26db720 con 0x5574e26e8680 2024-03-14T09:55:41.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.309+0000 7fa546a00700 1 -- 172.21.15.78:0/3535518457 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa554015020 con 0x5574e26e8680 2024-03-14T09:55:41.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.309+0000 7fa57b501b80 10 client.25967 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:41.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.309+0000 7fa57b501b80 10 client.25967 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:55:41.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.309+0000 7fa57b501b80 20 client.25967 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:41.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.309+0000 7fa57b501b80 1 --2- 172.21.15.78:0/3535518457 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5574e2702ac0 0x5574e2704ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.309+0000 7fa57b501b80 4 client.25967 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:55:41.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.309+0000 7fa57b501b80 1 -- 172.21.15.78:0/3535518457 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5574e2633700 con 0x5574e2702ac0 2024-03-14T09:55:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.309+0000 7fa55fe00700 1 --2- 172.21.15.78:0/3535518457 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5574e2702ac0 0x5574e2704ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.310+0000 7fa55fe00700 1 --2- 172.21.15.78:0/3535518457 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5574e2702ac0 0x5574e2704ee0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5281443f7f4b9415 server_cookie=ede5580420594350 in_seq=0 out_seq=0 2024-03-14T09:55:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.310+0000 7fa546a00700 10 client.25967 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:55:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.310+0000 7fa546a00700 1 -- 172.21.15.78:0/3535518457 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5574e2633700 con 0x5574e2702ac0 2024-03-14T09:55:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.310+0000 7fa546a00700 10 client.25967 handle_command_reply: tid=0 2024-03-14T09:55:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.310+0000 7fa57b501b80 1 client.25967 shutdown 2024-03-14T09:55:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.311+0000 7fa57b501b80 1 -- 172.21.15.78:0/3535518457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8680 msgr2=0x5574e26f28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.311+0000 7fa57b501b80 1 --2- 172.21.15.78:0/3535518457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8680 0x5574e26f28d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa554009230 tx=0x7fa55400f690 comp rx=0 tx=0).stop 2024-03-14T09:55:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.311+0000 7fa57b501b80 1 -- 172.21.15.78:0/3535518457 shutdown_connections 2024-03-14T09:55:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.311+0000 7fa57b501b80 1 --2- 172.21.15.78:0/3535518457 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5574e2702ac0 0x5574e2704ee0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.311+0000 7fa57b501b80 1 --2- 172.21.15.78:0/3535518457 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8680 0x5574e26f28d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.311+0000 7fa57b501b80 1 --2- 172.21.15.78:0/3535518457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26f2ec0 0x5574e26f02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.311+0000 7fa57b501b80 1 --2- 172.21.15.78:0/3535518457 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8fa0 0x5574e26efc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.311+0000 7fa57b501b80 1 -- 172.21.15.78:0/3535518457 >> 172.21.15.78:0/3535518457 conn(0x5574e265e220 msgr2=0x5574e26e3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.311+0000 7fa546a00700 0 client.25967 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:55:41.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.311+0000 7fa57b501b80 1 -- 172.21.15.78:0/3535518457 shutdown_connections 2024-03-14T09:55:41.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.312+0000 7fa57b501b80 1 -- 172.21.15.78:0/3535518457 wait complete. 2024-03-14T09:55:41.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.312+0000 7fa57b501b80 20 client.25967 trim_cache size 0 max 16384 2024-03-14T09:55:41.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.334+0000 7fa57b501b80 1 Processor -- start 2024-03-14T09:55:41.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.334+0000 7fa57b501b80 1 -- start start 2024-03-14T09:55:41.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.334+0000 7fa57b501b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8680 0x5574e26e3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.334+0000 7fa57b501b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8fa0 0x5574e26ef150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.334+0000 7fa57b501b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26f2ec0 0x5574e26eaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.334+0000 7fa57b501b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e26e9890 con 0x5574e26f2ec0 2024-03-14T09:55:41.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.334+0000 7fa57b501b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e26f05f0 con 0x5574e26e8fa0 2024-03-14T09:55:41.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.334+0000 7fa57b501b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e26ee160 con 0x5574e26e8680 2024-03-14T09:55:41.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.334+0000 7fa55fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8fa0 0x5574e26ef150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.334+0000 7fa565a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26f2ec0 0x5574e26eaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.334+0000 7fa565000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8680 0x5574e26e3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.334+0000 7fa55fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8fa0 0x5574e26ef150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:59952/0 (socket says 172.21.15.78:59952) 2024-03-14T09:55:41.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.334+0000 7fa55fe00700 1 -- 172.21.15.78:0/1633607708 learned_addr learned my addr 172.21.15.78:0/1633607708 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:41.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.335+0000 7fa55fe00700 1 -- 172.21.15.78:0/1633607708 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8680 msgr2=0x5574e26e3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.335+0000 7fa55fe00700 1 --2- 172.21.15.78:0/1633607708 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8680 0x5574e26e3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.335+0000 7fa55fe00700 1 -- 172.21.15.78:0/1633607708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26f2ec0 msgr2=0x5574e26eaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.335+0000 7fa55fe00700 1 --2- 172.21.15.78:0/1633607708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26f2ec0 0x5574e26eaf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.335+0000 7fa55fe00700 1 -- 172.21.15.78:0/1633607708 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574e2600ea0 con 0x5574e26e8fa0 2024-03-14T09:55:41.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.335+0000 7fa565000700 1 --2- 172.21.15.78:0/1633607708 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8680 0x5574e26e3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:41.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.335+0000 7fa55fe00700 1 --2- 172.21.15.78:0/1633607708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8fa0 0x5574e26ef150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa54c099c00 tx=0x7fa54c003a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=deb6cc27eb52f080 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.335+0000 7fa552a00700 1 -- 172.21.15.78:0/1633607708 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54c0b1040 con 0x5574e26e8fa0 2024-03-14T09:55:41.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.335+0000 7fa552a00700 1 -- 172.21.15.78:0/1633607708 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa54c0a2140 con 0x5574e26e8fa0 2024-03-14T09:55:41.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.335+0000 7fa552a00700 1 -- 172.21.15.78:0/1633607708 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54c099f20 con 0x5574e26e8fa0 2024-03-14T09:55:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.336+0000 7fa57b501b80 1 -- 172.21.15.78:0/1633607708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8fa0 msgr2=0x5574e26ef150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.336+0000 7fa57b501b80 1 --2- 172.21.15.78:0/1633607708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8fa0 0x5574e26ef150 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa54c099c00 tx=0x7fa54c003a10 comp rx=0 tx=0).stop 2024-03-14T09:55:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.336+0000 7fa57b501b80 1 -- 172.21.15.78:0/1633607708 shutdown_connections 2024-03-14T09:55:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.336+0000 7fa57b501b80 1 --2- 172.21.15.78:0/1633607708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26f2ec0 0x5574e26eaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.336+0000 7fa57b501b80 1 --2- 172.21.15.78:0/1633607708 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8680 0x5574e26e3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.336+0000 7fa57b501b80 1 --2- 172.21.15.78:0/1633607708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8fa0 0x5574e26ef150 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.336+0000 7fa57b501b80 1 -- 172.21.15.78:0/1633607708 >> 172.21.15.78:0/1633607708 conn(0x5574e265e220 msgr2=0x5574e2704ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.336+0000 7fa57b501b80 1 -- 172.21.15.78:0/1633607708 shutdown_connections 2024-03-14T09:55:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.336+0000 7fa57b501b80 1 -- 172.21.15.78:0/1633607708 wait complete. 2024-03-14T09:55:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.337+0000 7fa57b501b80 1 -- start start 2024-03-14T09:55:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.337+0000 7fa57b501b80 1 Processor -- start 2024-03-14T09:55:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa57b501b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8680 0x5574e26f7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa57b501b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8fa0 0x5574e26f8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa57b501b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26f2ec0 0x5574e27535e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa57b501b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e26f88e0 con 0x5574e26f2ec0 2024-03-14T09:55:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa57b501b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e2753b20 con 0x5574e26e8680 2024-03-14T09:55:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa57b501b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574e2753c60 con 0x5574e26e8fa0 2024-03-14T09:55:41.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa57b501b80 10 client.0 _ll_register_callbacks cb 0x7fa554009740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:41.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa565000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8680 0x5574e26f7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa565000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8680 0x5574e26f7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:59966/0 (socket says 172.21.15.78:59966) 2024-03-14T09:55:41.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa565000700 1 -- 172.21.15.78:0/2072655127 learned_addr learned my addr 172.21.15.78:0/2072655127 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:41.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa565a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26f2ec0 0x5574e27535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa55fe00700 1 --2- 172.21.15.78:0/2072655127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8fa0 0x5574e26f8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa565000700 1 -- 172.21.15.78:0/2072655127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8fa0 msgr2=0x5574e26f8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa565000700 1 --2- 172.21.15.78:0/2072655127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8fa0 0x5574e26f8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa565000700 1 -- 172.21.15.78:0/2072655127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26f2ec0 msgr2=0x5574e27535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa565000700 1 --2- 172.21.15.78:0/2072655127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26f2ec0 0x5574e27535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.338+0000 7fa565000700 1 -- 172.21.15.78:0/2072655127 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574e2468b60 con 0x5574e26e8680 2024-03-14T09:55:41.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.339+0000 7fa565000700 1 --2- 172.21.15.78:0/2072655127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8680 0x5574e26f7cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa554009fd0 tx=0x7fa554003680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f512e9f30532c6c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.339+0000 7fa546a00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:55:41.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.339+0000 7fa546a00700 1 -- 172.21.15.78:0/2072655127 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55401c070 con 0x5574e26e8680 2024-03-14T09:55:41.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.339+0000 7fa546a00700 1 -- 172.21.15.78:0/2072655127 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fa554000e40 con 0x5574e26e8680 2024-03-14T09:55:41.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.339+0000 7fa57b501b80 1 -- 172.21.15.78:0/2072655127 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa54c099cc0 con 0x5574e26e8680 2024-03-14T09:55:41.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.339+0000 7fa546a00700 1 -- 172.21.15.78:0/2072655127 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa55400b2e0 con 0x5574e26e8680 2024-03-14T09:55:41.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.339+0000 7fa57b501b80 10 client.25973 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:41.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.339+0000 7fa57b501b80 1 -- 172.21.15.78:0/2072655127 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5574e24bc3b0 con 0x5574e26e8680 2024-03-14T09:55:41.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.340+0000 7fa546a00700 1 -- 172.21.15.78:0/2072655127 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa54c099cc0 con 0x5574e26e8680 2024-03-14T09:55:41.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.340+0000 7fa546a00700 1 -- 172.21.15.78:0/2072655127 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa554009c70 con 0x5574e26e8680 2024-03-14T09:55:41.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.340+0000 7fa57b501b80 10 client.25973 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:41.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.340+0000 7fa57b501b80 10 client.25973 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:55:41.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.340+0000 7fa57b501b80 20 client.25973 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:41.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.340+0000 7fa57b501b80 1 --2- 172.21.15.78:0/2072655127 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5574e2754f50 0x5574e2757370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.340+0000 7fa57b501b80 4 client.25973 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:55:41.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.340+0000 7fa57b501b80 1 -- 172.21.15.78:0/2072655127 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5574e2489cf0 con 0x5574e2754f50 2024-03-14T09:55:41.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.340+0000 7fa55fe00700 1 --2- 172.21.15.78:0/2072655127 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5574e2754f50 0x5574e2757370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa55fe00700 1 --2- 172.21.15.78:0/2072655127 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5574e2754f50 0x5574e2757370 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e7fbf0e8fb1f637 server_cookie=9ef53cef71149a53 in_seq=0 out_seq=0 2024-03-14T09:55:41.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa546a00700 10 client.25973 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:55:41.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa546a00700 1 -- 172.21.15.78:0/2072655127 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5574e2489cf0 con 0x5574e2754f50 2024-03-14T09:55:41.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa546a00700 10 client.25973 handle_command_reply: tid=0 2024-03-14T09:55:41.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa57b501b80 1 client.25973 shutdown 2024-03-14T09:55:41.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa57b501b80 1 -- 172.21.15.78:0/2072655127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8680 msgr2=0x5574e26f7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa57b501b80 1 --2- 172.21.15.78:0/2072655127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8680 0x5574e26f7cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa554009fd0 tx=0x7fa554003680 comp rx=0 tx=0).stop 2024-03-14T09:55:41.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa57b501b80 1 -- 172.21.15.78:0/2072655127 shutdown_connections 2024-03-14T09:55:41.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa57b501b80 1 --2- 172.21.15.78:0/2072655127 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5574e2754f50 0x5574e2757370 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa57b501b80 1 --2- 172.21.15.78:0/2072655127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574e26e8680 0x5574e26f7cc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa57b501b80 1 --2- 172.21.15.78:0/2072655127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5574e26f2ec0 0x5574e27535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa57b501b80 1 --2- 172.21.15.78:0/2072655127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5574e26e8fa0 0x5574e26f8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa57b501b80 1 -- 172.21.15.78:0/2072655127 >> 172.21.15.78:0/2072655127 conn(0x5574e265e220 msgr2=0x5574e26ef150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa546a00700 0 client.25973 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:55:41.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa57b501b80 1 -- 172.21.15.78:0/2072655127 shutdown_connections 2024-03-14T09:55:41.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa57b501b80 1 -- 172.21.15.78:0/2072655127 wait complete. 2024-03-14T09:55:41.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.341+0000 7fa57b501b80 20 client.25973 trim_cache size 0 max 16384 2024-03-14T09:55:41.351 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:55:41.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.342+0000 7fa567a00700 1 -- 172.21.15.78:0/1253918978 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa5480a0da0 msgr2=0x7fa5480a3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.342+0000 7fa567a00700 1 --2- 172.21.15.78:0/1253918978 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa5480a0da0 0x7fa5480a3260 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa558009ec0 tx=0x7fa558013250 comp rx=0 tx=0).stop 2024-03-14T09:55:41.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.342+0000 7fa567a00700 1 -- 172.21.15.78:0/1253918978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa560073b00 msgr2=0x7fa56019fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.342+0000 7fa567a00700 1 --2- 172.21.15.78:0/1253918978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa560073b00 0x7fa56019fb50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa54c00ec10 tx=0x7fa54c00ef20 comp rx=0 tx=0).stop 2024-03-14T09:55:41.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.342+0000 7fa567a00700 1 -- 172.21.15.78:0/1253918978 shutdown_connections 2024-03-14T09:55:41.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.342+0000 7fa567a00700 1 --2- 172.21.15.78:0/1253918978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa56010d500 0x7fa560199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.342+0000 7fa567a00700 1 --2- 172.21.15.78:0/1253918978 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa5600731e0 0x7fa56019f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.342+0000 7fa567a00700 1 --2- 172.21.15.78:0/1253918978 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa5480a0da0 0x7fa5480a3260 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.342+0000 7fa567a00700 1 --2- 172.21.15.78:0/1253918978 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa560073b00 0x7fa56019fb50 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.342+0000 7fa567a00700 1 -- 172.21.15.78:0/1253918978 >> 172.21.15.78:0/1253918978 conn(0x7fa5600fc7b0 msgr2=0x7fa5600fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.342+0000 7fa567a00700 1 -- 172.21.15.78:0/1253918978 shutdown_connections 2024-03-14T09:55:41.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.342+0000 7fa567a00700 1 -- 172.21.15.78:0/1253918978 wait complete. 2024-03-14T09:55:41.361 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T09:55:41.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:41 smithi152 ceph-mon[48279]: pgmap v48: 129 pgs: 129 active+clean; 38 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 40 op/s 2024-03-14T09:55:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:41 smithi132 ceph-mon[47959]: pgmap v48: 129 pgs: 129 active+clean; 38 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 40 op/s 2024-03-14T09:55:41.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.568+0000 7f3663e00700 1 -- 172.21.15.78:0/2743992312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f365c073b00 msgr2=0x7f365c10cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.568+0000 7f3663e00700 1 --2- 172.21.15.78:0/2743992312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f365c073b00 0x7f365c10cfc0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3648009990 tx=0x7f3648009ca0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.568+0000 7f3663e00700 1 -- 172.21.15.78:0/2743992312 shutdown_connections 2024-03-14T09:55:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.568+0000 7f3663e00700 1 --2- 172.21.15.78:0/2743992312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f365c10d500 0x7f365c10f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.568+0000 7f3663e00700 1 --2- 172.21.15.78:0/2743992312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f365c0731e0 0x7f365c0735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.568+0000 7f3663e00700 1 --2- 172.21.15.78:0/2743992312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f365c073b00 0x7f365c10cfc0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.568+0000 7f3663e00700 1 -- 172.21.15.78:0/2743992312 >> 172.21.15.78:0/2743992312 conn(0x7f365c0fc7b0 msgr2=0x7f365c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.568+0000 7f3663e00700 1 -- 172.21.15.78:0/2743992312 shutdown_connections 2024-03-14T09:55:41.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.568+0000 7f3663e00700 1 -- 172.21.15.78:0/2743992312 wait complete. 2024-03-14T09:55:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.569+0000 7f3663e00700 1 Processor -- start 2024-03-14T09:55:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.569+0000 7f3663e00700 1 -- start start 2024-03-14T09:55:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.569+0000 7f3663e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f365c0731e0 0x7f365c1997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.569+0000 7f3663e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f365c073b00 0x7f365c199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.569+0000 7f3663e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f365c10d500 0x7f365c19d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.570+0000 7f3663e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c19d600 con 0x7f365c0731e0 2024-03-14T09:55:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.570+0000 7f3663e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c19d770 con 0x7f365c10d500 2024-03-14T09:55:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.570+0000 7f3663e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c19d930 con 0x7f365c073b00 2024-03-14T09:55:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.570+0000 7f3661400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f365c0731e0 0x7f365c1997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.570+0000 7f3660a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f365c073b00 0x7f365c199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.570+0000 7f3661400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f365c0731e0 0x7f365c1997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47236/0 (socket says 172.21.15.78:47236) 2024-03-14T09:55:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.570+0000 7f3661400700 1 -- 172.21.15.78:0/3597838938 learned_addr learned my addr 172.21.15.78:0/3597838938 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:41.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.570+0000 7f3661e00700 1 --2- 172.21.15.78:0/3597838938 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f365c10d500 0x7f365c19d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.570+0000 7f3660a00700 1 -- 172.21.15.78:0/3597838938 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f365c10d500 msgr2=0x7f365c19d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.570+0000 7f3660a00700 1 --2- 172.21.15.78:0/3597838938 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f365c10d500 0x7f365c19d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.570+0000 7f3660a00700 1 -- 172.21.15.78:0/3597838938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f365c0731e0 msgr2=0x7f365c1997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.570+0000 7f3660a00700 1 --2- 172.21.15.78:0/3597838938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f365c0731e0 0x7f365c1997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.570+0000 7f3660a00700 1 -- 172.21.15.78:0/3597838938 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3650009520 con 0x7f365c073b00 2024-03-14T09:55:41.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.570+0000 7f3661400700 1 --2- 172.21.15.78:0/3597838938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f365c0731e0 0x7f365c1997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:41.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.570+0000 7f3660a00700 1 --2- 172.21.15.78:0/3597838938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f365c073b00 0x7f365c199d10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f36480098e0 tx=0x7f3648005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.571+0000 7f365a000700 1 -- 172.21.15.78:0/3597838938 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364801d070 con 0x7f365c073b00 2024-03-14T09:55:41.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.571+0000 7f3663e00700 1 -- 172.21.15.78:0/3597838938 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36480095f0 con 0x7f365c073b00 2024-03-14T09:55:41.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.571+0000 7f3663e00700 1 -- 172.21.15.78:0/3597838938 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f365c1a6390 con 0x7f365c073b00 2024-03-14T09:55:41.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.571+0000 7f365a000700 1 -- 172.21.15.78:0/3597838938 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3648022470 con 0x7f365c073b00 2024-03-14T09:55:41.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.571+0000 7f365a000700 1 -- 172.21.15.78:0/3597838938 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3648017650 con 0x7f365c073b00 2024-03-14T09:55:41.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.574+0000 7f365a000700 1 -- 172.21.15.78:0/3597838938 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 26) v1 ==== 135621+0+0 (secure 0 0 0) 0x7f36480228a0 con 0x7f365c073b00 2024-03-14T09:55:41.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.575+0000 7f36779e5b80 1 Processor -- start 2024-03-14T09:55:41.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.575+0000 7f36779e5b80 1 -- start start 2024-03-14T09:55:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.575+0000 7f36779e5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace733680 0x563ace733a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.575+0000 7f36779e5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace73d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.575+0000 7f36779e5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace73dec0 0x563ace7402b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.575+0000 7f36779e5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ace68c990 con 0x563ace733680 2024-03-14T09:55:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.576+0000 7f36779e5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ace68a350 con 0x563ace733fa0 2024-03-14T09:55:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.576+0000 7f36779e5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ace685c40 con 0x563ace73dec0 2024-03-14T09:55:41.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.576+0000 7f3661e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace733680 0x563ace733a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.576+0000 7f3661e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace733680 0x563ace733a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47252/0 (socket says 172.21.15.78:47252) 2024-03-14T09:55:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.576+0000 7f3661e00700 1 -- 172.21.15.78:0/1879421892 learned_addr learned my addr 172.21.15.78:0/1879421892 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.576+0000 7f3661400700 1 --2- 172.21.15.78:0/1879421892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace73d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.576+0000 7f3660a00700 1 --2- 172.21.15.78:0/1879421892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace73dec0 0x563ace7402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.576+0000 7f3661e00700 1 -- 172.21.15.78:0/1879421892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace73dec0 msgr2=0x563ace7402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.576+0000 7f3661e00700 1 --2- 172.21.15.78:0/1879421892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace73dec0 0x563ace7402b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.576+0000 7f3661e00700 1 -- 172.21.15.78:0/1879421892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 msgr2=0x563ace73d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.576+0000 7f3661e00700 1 --2- 172.21.15.78:0/1879421892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace73d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.576+0000 7f3661e00700 1 -- 172.21.15.78:0/1879421892 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ace5073b0 con 0x563ace733680 2024-03-14T09:55:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.576+0000 7f3661e00700 1 --2- 172.21.15.78:0/1879421892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace733680 0x563ace733a60 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f3654011680 tx=0x7f3654009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf702717ba6f667f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.577+0000 7f364e600700 1 -- 172.21.15.78:0/1879421892 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365400e040 con 0x563ace733680 2024-03-14T09:55:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.577+0000 7f365a000700 1 --2- 172.21.15.78:0/3597838938 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f36440a0da0 0x7f36440a3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.577+0000 7f364e600700 1 -- 172.21.15.78:0/1879421892 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f365400f040 con 0x563ace733680 2024-03-14T09:55:41.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.577+0000 7f364e600700 1 -- 172.21.15.78:0/1879421892 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36540048c0 con 0x563ace733680 2024-03-14T09:55:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.577+0000 7f365a000700 1 -- 172.21.15.78:0/3597838938 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f36480cdac0 con 0x7f365c073b00 2024-03-14T09:55:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.577+0000 7f3661400700 1 --2- 172.21.15.78:0/3597838938 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f36440a0da0 0x7f36440a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.577+0000 7f36779e5b80 1 -- 172.21.15.78:0/1879421892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace733680 msgr2=0x563ace733a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.577+0000 7f36779e5b80 1 --2- 172.21.15.78:0/1879421892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace733680 0x563ace733a60 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f3654011680 tx=0x7f3654009920 comp rx=0 tx=0).stop 2024-03-14T09:55:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.578+0000 7f36779e5b80 1 -- 172.21.15.78:0/1879421892 shutdown_connections 2024-03-14T09:55:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.578+0000 7f36779e5b80 1 --2- 172.21.15.78:0/1879421892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace73dec0 0x563ace7402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.578+0000 7f36779e5b80 1 --2- 172.21.15.78:0/1879421892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace733680 0x563ace733a60 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.578+0000 7f36779e5b80 1 --2- 172.21.15.78:0/1879421892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace73d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.578+0000 7f36779e5b80 1 -- 172.21.15.78:0/1879421892 >> 172.21.15.78:0/1879421892 conn(0x563ace6a9220 msgr2=0x563ace72eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.578+0000 7f3661400700 1 --2- 172.21.15.78:0/3597838938 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f36440a0da0 0x7f36440a3260 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3650012900 tx=0x7f3650009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.578+0000 7f36779e5b80 1 -- 172.21.15.78:0/1879421892 shutdown_connections 2024-03-14T09:55:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.578+0000 7f36779e5b80 1 -- 172.21.15.78:0/1879421892 wait complete. 2024-03-14T09:55:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.578+0000 7f36779e5b80 1 -- start start 2024-03-14T09:55:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.579+0000 7f36779e5b80 1 Processor -- start 2024-03-14T09:55:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.579+0000 7f36779e5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace733680 0x563ace73d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.579+0000 7f36779e5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace73abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.579+0000 7f36779e5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace73dec0 0x563ace73b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.579+0000 7f36779e5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ace621d80 con 0x563ace733680 2024-03-14T09:55:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.580+0000 7f36779e5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ace621a20 con 0x563ace733fa0 2024-03-14T09:55:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.580+0000 7f36779e5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ace5096a0 con 0x563ace73dec0 2024-03-14T09:55:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.580+0000 7f36779e5b80 10 client.0 _ll_register_callbacks cb 0x563ace48d3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.580+0000 7f3661400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace733680 0x563ace73d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.580+0000 7f3661400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace733680 0x563ace73d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47258/0 (socket says 172.21.15.78:47258) 2024-03-14T09:55:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.580+0000 7f3660a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace73abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.580+0000 7f3661e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace73dec0 0x563ace73b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.580+0000 7f3661400700 1 -- 172.21.15.78:0/1956312740 learned_addr learned my addr 172.21.15.78:0/1956312740 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.580+0000 7f3660a00700 1 -- 172.21.15.78:0/1956312740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace73dec0 msgr2=0x563ace73b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.580+0000 7f3660a00700 1 --2- 172.21.15.78:0/1956312740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace73dec0 0x563ace73b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.580+0000 7f3660a00700 1 -- 172.21.15.78:0/1956312740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace733680 msgr2=0x563ace73d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.580+0000 7f3660a00700 1 --2- 172.21.15.78:0/1956312740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace733680 0x563ace73d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.580+0000 7f3660a00700 1 -- 172.21.15.78:0/1956312740 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3654011330 con 0x563ace733fa0 2024-03-14T09:55:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.580+0000 7f3661e00700 1 --2- 172.21.15.78:0/1956312740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace73dec0 0x563ace73b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.580+0000 7f3661400700 1 --2- 172.21.15.78:0/1956312740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace733680 0x563ace73d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.581+0000 7f3660a00700 1 --2- 172.21.15.78:0/1956312740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace73abd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f36480cace0 tx=0x7f364809ade0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6627259d0f91e769 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.581+0000 7f3642a00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:55:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.581+0000 7f3642a00700 1 -- 172.21.15.78:0/1956312740 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3648096be0 con 0x563ace733fa0 2024-03-14T09:55:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.581+0000 7f3642a00700 1 -- 172.21.15.78:0/1956312740 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f364809afa0 con 0x563ace733fa0 2024-03-14T09:55:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.581+0000 7f36779e5b80 1 -- 172.21.15.78:0/1956312740 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563ace726720 con 0x563ace733fa0 2024-03-14T09:55:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.581+0000 7f3642a00700 1 -- 172.21.15.78:0/1956312740 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f364809a670 con 0x563ace733fa0 2024-03-14T09:55:41.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.581+0000 7f36779e5b80 10 client.25985 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.581+0000 7f36779e5b80 1 -- 172.21.15.78:0/1956312740 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563ace4b3b60 con 0x563ace733fa0 2024-03-14T09:55:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.581+0000 7f3642a00700 1 -- 172.21.15.78:0/1956312740 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563ace726720 con 0x563ace733fa0 2024-03-14T09:55:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.582+0000 7f3642a00700 1 -- 172.21.15.78:0/1956312740 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f36480ba0e0 con 0x563ace733fa0 2024-03-14T09:55:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.582+0000 7f36779e5b80 10 client.25985 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.582+0000 7f36779e5b80 10 client.25985 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:55:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.582+0000 7f36779e5b80 20 client.25985 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.582+0000 7f36779e5b80 1 --2- 172.21.15.78:0/1956312740 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x563ace74d660 0x563ace74fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.582+0000 7f36779e5b80 4 client.25985 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:55:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.582+0000 7f36779e5b80 1 -- 172.21.15.78:0/1956312740 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563ace67e700 con 0x563ace74d660 2024-03-14T09:55:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.582+0000 7f3661400700 1 --2- 172.21.15.78:0/1956312740 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x563ace74d660 0x563ace74fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.583+0000 7f3642a00700 10 client.25985 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:55:41.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.583+0000 7f3661400700 1 --2- 172.21.15.78:0/1956312740 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x563ace74d660 0x563ace74fa80 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7cec92e82b913918 server_cookie=d16daf53aa6fcff2 in_seq=0 out_seq=0 2024-03-14T09:55:41.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.583+0000 7f3642a00700 1 -- 172.21.15.78:0/1956312740 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x563ace67e700 con 0x563ace74d660 2024-03-14T09:55:41.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.583+0000 7f3642a00700 10 client.25985 handle_command_reply: tid=0 2024-03-14T09:55:41.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.583+0000 7f36779e5b80 1 client.25985 shutdown 2024-03-14T09:55:41.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.584+0000 7f36779e5b80 1 -- 172.21.15.78:0/1956312740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 msgr2=0x563ace73abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.584+0000 7f36779e5b80 1 --2- 172.21.15.78:0/1956312740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace73abd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f36480cace0 tx=0x7f364809ade0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.584+0000 7f36779e5b80 1 -- 172.21.15.78:0/1956312740 shutdown_connections 2024-03-14T09:55:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.584+0000 7f36779e5b80 1 --2- 172.21.15.78:0/1956312740 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x563ace74d660 0x563ace74fa80 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.584+0000 7f36779e5b80 1 --2- 172.21.15.78:0/1956312740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace73dec0 0x563ace73b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.584+0000 7f36779e5b80 1 --2- 172.21.15.78:0/1956312740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace733680 0x563ace73d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.584+0000 7f3642a00700 0 client.25985 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:55:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.584+0000 7f36779e5b80 1 --2- 172.21.15.78:0/1956312740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace73abd0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.584+0000 7f36779e5b80 1 -- 172.21.15.78:0/1956312740 >> 172.21.15.78:0/1956312740 conn(0x563ace6a9220 msgr2=0x563ace72e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.584+0000 7f36779e5b80 1 -- 172.21.15.78:0/1956312740 shutdown_connections 2024-03-14T09:55:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.584+0000 7f36779e5b80 1 -- 172.21.15.78:0/1956312740 wait complete. 2024-03-14T09:55:41.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.584+0000 7f36779e5b80 20 client.25985 trim_cache size 0 max 16384 2024-03-14T09:55:41.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.608+0000 7f36779e5b80 1 Processor -- start 2024-03-14T09:55:41.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.608+0000 7f36779e5b80 1 -- start start 2024-03-14T09:55:41.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.609+0000 7f36779e5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace733680 0x563ace72ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.609+0000 7f36779e5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace7543e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.609+0000 7f36779e5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace73dec0 0x563ace744b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.609+0000 7f36779e5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ace7300e0 con 0x563ace73dec0 2024-03-14T09:55:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.609+0000 7f36779e5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ace73af80 con 0x563ace733fa0 2024-03-14T09:55:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.609+0000 7f36779e5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ace728d30 con 0x563ace733680 2024-03-14T09:55:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.609+0000 7f3661e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace73dec0 0x563ace744b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.609+0000 7f3661e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace73dec0 0x563ace744b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47272/0 (socket says 172.21.15.78:47272) 2024-03-14T09:55:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.609+0000 7f3661e00700 1 -- 172.21.15.78:0/2051130999 learned_addr learned my addr 172.21.15.78:0/2051130999 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.609+0000 7f3661400700 1 --2- 172.21.15.78:0/2051130999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace733680 0x563ace72ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.609+0000 7f3660a00700 1 --2- 172.21.15.78:0/2051130999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace7543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.610+0000 7f3661e00700 1 -- 172.21.15.78:0/2051130999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace733680 msgr2=0x563ace72ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.610+0000 7f3661e00700 1 --2- 172.21.15.78:0/2051130999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace733680 0x563ace72ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.610+0000 7f3661e00700 1 -- 172.21.15.78:0/2051130999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 msgr2=0x563ace7543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.610+0000 7f3661e00700 1 --2- 172.21.15.78:0/2051130999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace7543e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.610+0000 7f3661e00700 1 -- 172.21.15.78:0/2051130999 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ace64bea0 con 0x563ace73dec0 2024-03-14T09:55:41.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.610+0000 7f3661e00700 1 --2- 172.21.15.78:0/2051130999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace73dec0 0x563ace744b90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f3654011300 tx=0x7f3654009b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a293143e441259 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.610+0000 7f364f000700 1 -- 172.21.15.78:0/2051130999 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f365400e050 con 0x563ace73dec0 2024-03-14T09:55:41.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.610+0000 7f364f000700 1 -- 172.21.15.78:0/2051130999 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3654009360 con 0x563ace73dec0 2024-03-14T09:55:41.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.610+0000 7f364f000700 1 -- 172.21.15.78:0/2051130999 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3654004d80 con 0x563ace73dec0 2024-03-14T09:55:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.611+0000 7f36779e5b80 1 -- 172.21.15.78:0/2051130999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace73dec0 msgr2=0x563ace744b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.611+0000 7f36779e5b80 1 --2- 172.21.15.78:0/2051130999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace73dec0 0x563ace744b90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f3654011300 tx=0x7f3654009b70 comp rx=0 tx=0).stop 2024-03-14T09:55:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.611+0000 7f36779e5b80 1 -- 172.21.15.78:0/2051130999 shutdown_connections 2024-03-14T09:55:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.611+0000 7f36779e5b80 1 --2- 172.21.15.78:0/2051130999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace73dec0 0x563ace744b90 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.611+0000 7f36779e5b80 1 --2- 172.21.15.78:0/2051130999 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace733680 0x563ace72ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.611+0000 7f36779e5b80 1 --2- 172.21.15.78:0/2051130999 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace7543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.611+0000 7f36779e5b80 1 -- 172.21.15.78:0/2051130999 >> 172.21.15.78:0/2051130999 conn(0x563ace6a9220 msgr2=0x563ace72d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.611+0000 7f36779e5b80 1 -- 172.21.15.78:0/2051130999 shutdown_connections 2024-03-14T09:55:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.611+0000 7f36779e5b80 1 -- 172.21.15.78:0/2051130999 wait complete. 2024-03-14T09:55:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.611+0000 7f36779e5b80 1 -- start start 2024-03-14T09:55:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.612+0000 7f36779e5b80 1 Processor -- start 2024-03-14T09:55:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.612+0000 7f36779e5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace733680 0x563ace742d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.612+0000 7f36779e5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace7432a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.612+0000 7f36779e5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace73dec0 0x563ace7981c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.612+0000 7f36779e5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ace798700 con 0x563ace73dec0 2024-03-14T09:55:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.612+0000 7f36779e5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ace798870 con 0x563ace733fa0 2024-03-14T09:55:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.612+0000 7f36779e5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ace798a30 con 0x563ace733680 2024-03-14T09:55:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.612+0000 7f36779e5b80 10 client.0 _ll_register_callbacks cb 0x7f3648005c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.612+0000 7f3661e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace73dec0 0x563ace7981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.612+0000 7f3661400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace733680 0x563ace742d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.612+0000 7f3661e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace73dec0 0x563ace7981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47274/0 (socket says 172.21.15.78:47274) 2024-03-14T09:55:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.612+0000 7f3661400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace733680 0x563ace742d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:56412/0 (socket says 172.21.15.78:56412) 2024-03-14T09:55:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.612+0000 7f3661e00700 1 -- 172.21.15.78:0/207875155 learned_addr learned my addr 172.21.15.78:0/207875155 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.612+0000 7f3660a00700 1 --2- 172.21.15.78:0/207875155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace7432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.613+0000 7f3661e00700 1 -- 172.21.15.78:0/207875155 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace733680 msgr2=0x563ace742d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.613+0000 7f3661e00700 1 --2- 172.21.15.78:0/207875155 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace733680 0x563ace742d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.613+0000 7f3661e00700 1 -- 172.21.15.78:0/207875155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 msgr2=0x563ace7432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.613+0000 7f3661e00700 1 --2- 172.21.15.78:0/207875155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace7432a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.613+0000 7f3661e00700 1 -- 172.21.15.78:0/207875155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563ace5073b0 con 0x563ace73dec0 2024-03-14T09:55:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.613+0000 7f3660a00700 1 --2- 172.21.15.78:0/207875155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace7432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:55:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.613+0000 7f3661e00700 1 --2- 172.21.15.78:0/207875155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace73dec0 0x563ace7981c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3654011c70 tx=0x7f3654003a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4dc0ac0411f1863 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.613+0000 7f3642a00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:55:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.613+0000 7f3642a00700 1 -- 172.21.15.78:0/207875155 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3654003f00 con 0x563ace73dec0 2024-03-14T09:55:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.613+0000 7f3642a00700 1 -- 172.21.15.78:0/207875155 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f3654004060 con 0x563ace73dec0 2024-03-14T09:55:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.613+0000 7f36779e5b80 1 -- 172.21.15.78:0/207875155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f365400f040 con 0x563ace73dec0 2024-03-14T09:55:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.614+0000 7f3642a00700 1 -- 172.21.15.78:0/207875155 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3654004220 con 0x563ace73dec0 2024-03-14T09:55:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.614+0000 7f36779e5b80 10 client.15987 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.614+0000 7f36779e5b80 1 -- 172.21.15.78:0/207875155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563ace50bcd0 con 0x563ace73dec0 2024-03-14T09:55:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.614+0000 7f3642a00700 1 -- 172.21.15.78:0/207875155 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f365400f040 con 0x563ace73dec0 2024-03-14T09:55:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.614+0000 7f3642a00700 1 -- 172.21.15.78:0/207875155 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f36540156b0 con 0x563ace73dec0 2024-03-14T09:55:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.614+0000 7f36779e5b80 10 client.15987 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.614+0000 7f36779e5b80 10 client.15987 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:55:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.614+0000 7f36779e5b80 20 client.15987 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.614+0000 7f36779e5b80 1 --2- 172.21.15.78:0/207875155 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x563ace79fe40 0x563ace7a2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.614+0000 7f36779e5b80 4 client.15987 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:55:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.614+0000 7f36779e5b80 1 -- 172.21.15.78:0/207875155 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563ace4d4cf0 con 0x563ace79fe40 2024-03-14T09:55:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.614+0000 7f3661400700 1 --2- 172.21.15.78:0/207875155 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x563ace79fe40 0x563ace7a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.615+0000 7f3661400700 1 --2- 172.21.15.78:0/207875155 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x563ace79fe40 0x563ace7a2230 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6ecd6159a030aa45 server_cookie=c38f6bcf6ba36feb in_seq=0 out_seq=0 2024-03-14T09:55:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.615+0000 7f3642a00700 10 client.15987 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:55:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.615+0000 7f3642a00700 1 -- 172.21.15.78:0/207875155 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563ace4d4cf0 con 0x563ace79fe40 2024-03-14T09:55:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.615+0000 7f3642a00700 10 client.15987 handle_command_reply: tid=0 2024-03-14T09:55:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.615+0000 7f36779e5b80 1 client.15987 shutdown 2024-03-14T09:55:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.615+0000 7f36779e5b80 1 -- 172.21.15.78:0/207875155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace73dec0 msgr2=0x563ace7981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.615+0000 7f36779e5b80 1 --2- 172.21.15.78:0/207875155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace73dec0 0x563ace7981c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3654011c70 tx=0x7f3654003a90 comp rx=0 tx=0).stop 2024-03-14T09:55:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.615+0000 7f36779e5b80 1 -- 172.21.15.78:0/207875155 shutdown_connections 2024-03-14T09:55:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.615+0000 7f36779e5b80 1 --2- 172.21.15.78:0/207875155 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x563ace79fe40 0x563ace7a2230 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.615+0000 7f36779e5b80 1 --2- 172.21.15.78:0/207875155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563ace73dec0 0x563ace7981c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.615+0000 7f36779e5b80 1 --2- 172.21.15.78:0/207875155 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563ace733680 0x563ace742d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.615+0000 7f36779e5b80 1 --2- 172.21.15.78:0/207875155 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563ace733fa0 0x563ace7432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.615+0000 7f36779e5b80 1 -- 172.21.15.78:0/207875155 >> 172.21.15.78:0/207875155 conn(0x563ace6a9220 msgr2=0x563ace735f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.615+0000 7f3642a00700 0 client.15987 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:55:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.616+0000 7f36779e5b80 1 -- 172.21.15.78:0/207875155 shutdown_connections 2024-03-14T09:55:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.616+0000 7f36779e5b80 1 -- 172.21.15.78:0/207875155 wait complete. 2024-03-14T09:55:41.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.616+0000 7f36779e5b80 20 client.15987 trim_cache size 0 max 16384 2024-03-14T09:55:41.621 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:55:41.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.616+0000 7f3663e00700 1 -- 172.21.15.78:0/3597838938 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f36440a0da0 msgr2=0x7f36440a3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.616+0000 7f3663e00700 1 --2- 172.21.15.78:0/3597838938 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f36440a0da0 0x7f36440a3260 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3650012900 tx=0x7f3650009260 comp rx=0 tx=0).stop 2024-03-14T09:55:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.616+0000 7f3663e00700 1 -- 172.21.15.78:0/3597838938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f365c073b00 msgr2=0x7f365c199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.616+0000 7f3663e00700 1 --2- 172.21.15.78:0/3597838938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f365c073b00 0x7f365c199d10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f36480098e0 tx=0x7f3648005740 comp rx=0 tx=0).stop 2024-03-14T09:55:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.616+0000 7f3663e00700 1 -- 172.21.15.78:0/3597838938 shutdown_connections 2024-03-14T09:55:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.616+0000 7f3663e00700 1 --2- 172.21.15.78:0/3597838938 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f36440a0da0 0x7f36440a3260 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.616+0000 7f3663e00700 1 --2- 172.21.15.78:0/3597838938 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f365c10d500 0x7f365c19d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.617+0000 7f3663e00700 1 --2- 172.21.15.78:0/3597838938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f365c0731e0 0x7f365c1997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.617+0000 7f3663e00700 1 --2- 172.21.15.78:0/3597838938 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f365c073b00 0x7f365c199d10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.617+0000 7f3663e00700 1 -- 172.21.15.78:0/3597838938 >> 172.21.15.78:0/3597838938 conn(0x7f365c0fc7b0 msgr2=0x7f365c0fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.617+0000 7f3663e00700 1 -- 172.21.15.78:0/3597838938 shutdown_connections 2024-03-14T09:55:41.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.617+0000 7f3663e00700 1 -- 172.21.15.78:0/3597838938 wait complete. 2024-03-14T09:55:41.635 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T09:55:41.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.837+0000 7f9fbd400700 1 -- 172.21.15.78:0/2755990130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb8106040 msgr2=0x7f9fb810dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.837+0000 7f9fbd400700 1 --2- 172.21.15.78:0/2755990130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb8106040 0x7f9fb810dbc0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9fb000b0a0 tx=0x7f9fb000b3b0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.837+0000 7f9fbd400700 1 -- 172.21.15.78:0/2755990130 shutdown_connections 2024-03-14T09:55:41.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.837+0000 7f9fbd400700 1 --2- 172.21.15.78:0/2755990130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb8106040 0x7f9fb810dbc0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.837+0000 7f9fbd400700 1 --2- 172.21.15.78:0/2755990130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9fb8100f20 0x7f9fb8101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.837+0000 7f9fbd400700 1 --2- 172.21.15.78:0/2755990130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9fb81018d0 0x7f9fb8105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.837+0000 7f9fbd400700 1 -- 172.21.15.78:0/2755990130 >> 172.21.15.78:0/2755990130 conn(0x7f9fb80fc7a0 msgr2=0x7f9fb80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.838+0000 7f9fbd400700 1 -- 172.21.15.78:0/2755990130 shutdown_connections 2024-03-14T09:55:41.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.838+0000 7f9fbd400700 1 -- 172.21.15.78:0/2755990130 wait complete. 2024-03-14T09:55:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.839+0000 7f9fbd400700 1 Processor -- start 2024-03-14T09:55:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.839+0000 7f9fbd400700 1 -- start start 2024-03-14T09:55:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.839+0000 7f9fbd400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9fb8100f20 0x7f9fb819f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.839+0000 7f9fbd400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb8106040 0x7f9fb819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.839+0000 7f9fbd400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9fb81997d0 0x7f9fb8199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.839+0000 7f9fbd400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb819a250 con 0x7f9fb8106040 2024-03-14T09:55:41.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.839+0000 7f9fbd400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb819a390 con 0x7f9fb81997d0 2024-03-14T09:55:41.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.839+0000 7f9fbd400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb819a550 con 0x7f9fb8100f20 2024-03-14T09:55:41.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.839+0000 7f9fb5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb8106040 0x7f9fb819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.839+0000 7f9fb7200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9fb81997d0 0x7f9fb8199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.839+0000 7f9fb5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb8106040 0x7f9fb819fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47284/0 (socket says 172.21.15.78:47284) 2024-03-14T09:55:41.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.839+0000 7f9fb6800700 1 --2- 172.21.15.78:0/3937031093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9fb8100f20 0x7f9fb819f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.839+0000 7f9fb5e00700 1 -- 172.21.15.78:0/3937031093 learned_addr learned my addr 172.21.15.78:0/3937031093 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:41.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.840+0000 7f9fb7200700 1 -- 172.21.15.78:0/3937031093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9fb8100f20 msgr2=0x7f9fb819f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.840+0000 7f9fb7200700 1 --2- 172.21.15.78:0/3937031093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9fb8100f20 0x7f9fb819f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.840+0000 7f9fb7200700 1 -- 172.21.15.78:0/3937031093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb8106040 msgr2=0x7f9fb819fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.840+0000 7f9fb7200700 1 --2- 172.21.15.78:0/3937031093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb8106040 0x7f9fb819fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.840+0000 7f9fb7200700 1 -- 172.21.15.78:0/3937031093 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa4009520 con 0x7f9fb81997d0 2024-03-14T09:55:41.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.840+0000 7f9fb5e00700 1 --2- 172.21.15.78:0/3937031093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb8106040 0x7f9fb819fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:41.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.840+0000 7f9fb7200700 1 --2- 172.21.15.78:0/3937031093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9fb81997d0 0x7f9fb8199c80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9fb000b070 tx=0x7f9fb000bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.840+0000 7f9faf400700 1 -- 172.21.15.78:0/3937031093 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb000e040 con 0x7f9fb81997d0 2024-03-14T09:55:41.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.840+0000 7f9faf400700 1 -- 172.21.15.78:0/3937031093 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fb001d480 con 0x7f9fb81997d0 2024-03-14T09:55:41.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.840+0000 7f9faf400700 1 -- 172.21.15.78:0/3937031093 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb00052a0 con 0x7f9fb81997d0 2024-03-14T09:55:41.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.840+0000 7f9fbd400700 1 -- 172.21.15.78:0/3937031093 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb0009d00 con 0x7f9fb81997d0 2024-03-14T09:55:41.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.841+0000 7f9fbd400700 1 -- 172.21.15.78:0/3937031093 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb81a6750 con 0x7f9fb81997d0 2024-03-14T09:55:41.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.844+0000 7f9faf400700 1 -- 172.21.15.78:0/3937031093 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 26) v1 ==== 135621+0+0 (secure 0 0 0) 0x7f9fb001d5f0 con 0x7f9fb81997d0 2024-03-14T09:55:41.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.845+0000 7f9fd0f61b80 1 Processor -- start 2024-03-14T09:55:41.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.845+0000 7f9fd0f61b80 1 -- start start 2024-03-14T09:55:41.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.845+0000 7f9faf400700 1 --2- 172.21.15.78:0/3937031093 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f9f9c0a0d20 0x7f9f9c0a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.845+0000 7f9fd0f61b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633680 0x55e273633a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.845+0000 7f9fd0f61b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 0x55e27363d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.845+0000 7f9fd0f61b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e27363dec0 0x55e2736402b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.845+0000 7f9fd0f61b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27358c990 con 0x55e273633680 2024-03-14T09:55:41.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.845+0000 7f9fd0f61b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27358a350 con 0x55e27363dec0 2024-03-14T09:55:41.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.845+0000 7f9fd0f61b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e273585c40 con 0x55e273633fa0 2024-03-14T09:55:41.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.845+0000 7f9faf400700 1 -- 172.21.15.78:0/3937031093 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f9fb00cda40 con 0x7f9fb81997d0 2024-03-14T09:55:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.846+0000 7f9fb5e00700 1 --2- 172.21.15.78:0/3937031093 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f9f9c0a0d20 0x7f9f9c0a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.846+0000 7f9fb6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633680 0x55e273633a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.846+0000 7f9fb6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633680 0x55e273633a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47290/0 (socket says 172.21.15.78:47290) 2024-03-14T09:55:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.846+0000 7f9fb6800700 1 -- 172.21.15.78:0/3438051339 learned_addr learned my addr 172.21.15.78:0/3438051339 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.846+0000 7f9fb7200700 1 --2- 172.21.15.78:0/3438051339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e27363dec0 0x55e2736402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.846+0000 7f9fb5e00700 1 --2- 172.21.15.78:0/3438051339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 0x55e27363d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.846+0000 7f9fb6800700 1 -- 172.21.15.78:0/3438051339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 msgr2=0x55e27363d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.846+0000 7f9fb6800700 1 --2- 172.21.15.78:0/3438051339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 0x55e27363d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.846+0000 7f9fb6800700 1 -- 172.21.15.78:0/3438051339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e27363dec0 msgr2=0x55e2736402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.846+0000 7f9fb6800700 1 --2- 172.21.15.78:0/3438051339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e27363dec0 0x55e2736402b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.846+0000 7f9fb6800700 1 -- 172.21.15.78:0/3438051339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2734073b0 con 0x55e273633680 2024-03-14T09:55:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.846+0000 7f9fb7200700 1 --2- 172.21.15.78:0/3438051339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e27363dec0 0x55e2736402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.846+0000 7f9fb5e00700 1 --2- 172.21.15.78:0/3937031093 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f9f9c0a0d20 0x7f9f9c0a31e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4009910 tx=0x7f9fa4009e00 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.846+0000 7f9fb6800700 1 --2- 172.21.15.78:0/3438051339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633680 0x55e273633a60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8009d80 tx=0x7f9fa8011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf28d93228f6715f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.847+0000 7f9fa3400700 1 -- 172.21.15.78:0/3438051339 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8004230 con 0x55e273633680 2024-03-14T09:55:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.847+0000 7f9fa3400700 1 -- 172.21.15.78:0/3438051339 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fa8004390 con 0x55e273633680 2024-03-14T09:55:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.847+0000 7f9fa3400700 1 -- 172.21.15.78:0/3438051339 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa80197c0 con 0x55e273633680 2024-03-14T09:55:41.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.847+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/3438051339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633680 msgr2=0x55e273633a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.847+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/3438051339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633680 0x55e273633a60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8009d80 tx=0x7f9fa8011720 comp rx=0 tx=0).stop 2024-03-14T09:55:41.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.847+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/3438051339 shutdown_connections 2024-03-14T09:55:41.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.847+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/3438051339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e27363dec0 0x55e2736402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.847+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/3438051339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633680 0x55e273633a60 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.847+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/3438051339 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 0x55e27363d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.848+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/3438051339 >> 172.21.15.78:0/3438051339 conn(0x55e2735a9220 msgr2=0x55e27362eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.848+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/3438051339 shutdown_connections 2024-03-14T09:55:41.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.848+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/3438051339 wait complete. 2024-03-14T09:55:41.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.848+0000 7f9fd0f61b80 1 -- start start 2024-03-14T09:55:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.848+0000 7f9fd0f61b80 1 Processor -- start 2024-03-14T09:55:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fd0f61b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633680 0x55e27363d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fd0f61b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 0x55e27363ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fd0f61b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e27363dec0 0x55e27363b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fd0f61b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e273521d80 con 0x55e273633680 2024-03-14T09:55:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fd0f61b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e273521a20 con 0x55e27363dec0 2024-03-14T09:55:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fd0f61b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2734096a0 con 0x55e273633fa0 2024-03-14T09:55:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fd0f61b80 10 client.0 _ll_register_callbacks cb 0x55e27338d3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fb6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633680 0x55e27363d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fb6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633680 0x55e27363d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47298/0 (socket says 172.21.15.78:47298) 2024-03-14T09:55:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fb6800700 1 -- 172.21.15.78:0/2643270247 learned_addr learned my addr 172.21.15.78:0/2643270247 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fb5e00700 1 --2- 172.21.15.78:0/2643270247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 0x55e27363ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fb7200700 1 --2- 172.21.15.78:0/2643270247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e27363dec0 0x55e27363b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fb6800700 1 -- 172.21.15.78:0/2643270247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 msgr2=0x55e27363ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fb6800700 1 --2- 172.21.15.78:0/2643270247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 0x55e27363ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fb6800700 1 -- 172.21.15.78:0/2643270247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e27363dec0 msgr2=0x55e27363b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fb6800700 1 --2- 172.21.15.78:0/2643270247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e27363dec0 0x55e27363b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fb6800700 1 -- 172.21.15.78:0/2643270247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb81a6750 con 0x55e273633680 2024-03-14T09:55:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.849+0000 7f9fb7200700 1 --2- 172.21.15.78:0/2643270247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e27363dec0 0x55e27363b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:55:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.850+0000 7f9fb6800700 1 --2- 172.21.15.78:0/2643270247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633680 0x55e27363d820 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9fa801b040 tx=0x7f9fa8003930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b06e83ab8e3b9b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.850+0000 7f9f97e00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:55:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.850+0000 7f9f97e00700 1 -- 172.21.15.78:0/2643270247 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8005040 con 0x55e273633680 2024-03-14T09:55:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.850+0000 7f9f97e00700 1 -- 172.21.15.78:0/2643270247 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f9fa80051a0 con 0x55e273633680 2024-03-14T09:55:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.850+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/2643270247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e273626720 con 0x55e273633680 2024-03-14T09:55:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.850+0000 7f9f97e00700 1 -- 172.21.15.78:0/2643270247 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9fa8005360 con 0x55e273633680 2024-03-14T09:55:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.850+0000 7f9fd0f61b80 10 client.15999 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.850+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/2643270247 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f9fa80099e0 con 0x55e273633680 2024-03-14T09:55:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.851+0000 7f9f97e00700 1 -- 172.21.15.78:0/2643270247 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e273626720 con 0x55e273633680 2024-03-14T09:55:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.851+0000 7f9f97e00700 1 -- 172.21.15.78:0/2643270247 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9fa8021040 con 0x55e273633680 2024-03-14T09:55:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.851+0000 7f9fd0f61b80 10 client.15999 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.851+0000 7f9fd0f61b80 10 client.15999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:55:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.851+0000 7f9fd0f61b80 20 client.15999 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.851+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/2643270247 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e27364d9b0 0x55e27364fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.851+0000 7f9fd0f61b80 4 client.15999 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:55:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.851+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/2643270247 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e27357e700 con 0x55e27364d9b0 2024-03-14T09:55:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.851+0000 7f9fb5e00700 1 --2- 172.21.15.78:0/2643270247 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e27364d9b0 0x55e27364fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.852+0000 7f9f97e00700 10 client.15999 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:41.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.852+0000 7f9fb5e00700 1 --2- 172.21.15.78:0/2643270247 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e27364d9b0 0x55e27364fdd0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6f789974ef9b949 server_cookie=e82904e9196266a0 in_seq=0 out_seq=0 2024-03-14T09:55:41.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.852+0000 7f9f97e00700 1 -- 172.21.15.78:0/2643270247 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55e27357e700 con 0x55e27364d9b0 2024-03-14T09:55:41.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.852+0000 7f9f97e00700 10 client.15999 handle_command_reply: tid=0 2024-03-14T09:55:41.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.852+0000 7f9fd0f61b80 1 client.15999 shutdown 2024-03-14T09:55:41.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.853+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/2643270247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633680 msgr2=0x55e27363d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.853+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/2643270247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633680 0x55e27363d820 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9fa801b040 tx=0x7f9fa8003930 comp rx=0 tx=0).stop 2024-03-14T09:55:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.853+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/2643270247 shutdown_connections 2024-03-14T09:55:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.853+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/2643270247 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e27364d9b0 0x55e27364fdd0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.853+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/2643270247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e27363dec0 0x55e27363b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.853+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/2643270247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633680 0x55e27363d820 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.853+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/2643270247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 0x55e27363ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.853+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/2643270247 >> 172.21.15.78:0/2643270247 conn(0x55e2735a9220 msgr2=0x55e27362e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.853+0000 7f9f97e00700 0 client.15999 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.853+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/2643270247 shutdown_connections 2024-03-14T09:55:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.853+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/2643270247 wait complete. 2024-03-14T09:55:41.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.853+0000 7f9fd0f61b80 20 client.15999 trim_cache size 0 max 16384 2024-03-14T09:55:41.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.867+0000 7f9fd0f61b80 1 Processor -- start 2024-03-14T09:55:41.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.867+0000 7f9fd0f61b80 1 -- start start 2024-03-14T09:55:41.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.868+0000 7f9fd0f61b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e273633680 0x55e273644b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.868+0000 7f9fd0f61b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 0x55e2736393e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.868+0000 7f9fd0f61b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27363dec0 0x55e273639920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.868+0000 7f9fd0f61b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27362b230 con 0x55e27363dec0 2024-03-14T09:55:41.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.868+0000 7f9fd0f61b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e273639e60 con 0x55e273633680 2024-03-14T09:55:41.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.868+0000 7f9fd0f61b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e273630060 con 0x55e273633fa0 2024-03-14T09:55:41.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.868+0000 7f9fb6800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e273633680 0x55e273644b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.868+0000 7f9fb5e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 0x55e2736393e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.868+0000 7f9fb7200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27363dec0 0x55e273639920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.868+0000 7f9fb6800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e273633680 0x55e273644b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60066/0 (socket says 172.21.15.78:60066) 2024-03-14T09:55:41.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.868+0000 7f9fb6800700 1 -- 172.21.15.78:0/165199129 learned_addr learned my addr 172.21.15.78:0/165199129 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:41.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.868+0000 7f9fb5e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 0x55e2736393e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:56436/0 (socket says 172.21.15.78:56436) 2024-03-14T09:55:41.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.869+0000 7f9fb6800700 1 -- 172.21.15.78:0/165199129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 msgr2=0x55e2736393e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.869+0000 7f9fb6800700 1 --2- 172.21.15.78:0/165199129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 0x55e2736393e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.869+0000 7f9fb6800700 1 -- 172.21.15.78:0/165199129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27363dec0 msgr2=0x55e273639920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.869+0000 7f9fb6800700 1 --2- 172.21.15.78:0/165199129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27363dec0 0x55e273639920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.869+0000 7f9fb6800700 1 -- 172.21.15.78:0/165199129 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2733b3b60 con 0x55e273633680 2024-03-14T09:55:41.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.869+0000 7f9fb5e00700 1 --2- 172.21.15.78:0/165199129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 0x55e2736393e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:41.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.869+0000 7f9fb7200700 1 --2- 172.21.15.78:0/165199129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27363dec0 0x55e273639920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:41.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.869+0000 7f9fb6800700 1 --2- 172.21.15.78:0/165199129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e273633680 0x55e273644b50 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8023040 tx=0x7f9fa80235a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=377b3b3e18f09621 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.869+0000 7f9fa3e00700 1 -- 172.21.15.78:0/165199129 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa80183d0 con 0x55e273633680 2024-03-14T09:55:41.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.869+0000 7f9fa3e00700 1 -- 172.21.15.78:0/165199129 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fa8011a70 con 0x55e273633680 2024-03-14T09:55:41.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.869+0000 7f9fa3e00700 1 -- 172.21.15.78:0/165199129 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8005490 con 0x55e273633680 2024-03-14T09:55:41.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.870+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/165199129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e273633680 msgr2=0x55e273644b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.870+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/165199129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e273633680 0x55e273644b50 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8023040 tx=0x7f9fa80235a0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.870+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/165199129 shutdown_connections 2024-03-14T09:55:41.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.870+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/165199129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e273633680 0x55e273644b50 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.870+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/165199129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27363dec0 0x55e273639920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.870+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/165199129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e273633fa0 0x55e2736393e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.870+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/165199129 >> 172.21.15.78:0/165199129 conn(0x55e2735a9220 msgr2=0x55e273635de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.870+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/165199129 shutdown_connections 2024-03-14T09:55:41.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.870+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/165199129 wait complete. 2024-03-14T09:55:41.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.870+0000 7f9fd0f61b80 1 -- start start 2024-03-14T09:55:41.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.871+0000 7f9fd0f61b80 1 Processor -- start 2024-03-14T09:55:41.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.871+0000 7f9fd0f61b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e273633680 0x55e273641ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.871+0000 7f9fd0f61b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633fa0 0x55e273642530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.871+0000 7f9fd0f61b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e27363dec0 0x55e273642a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.872+0000 7f9fd0f61b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27369e4d0 con 0x55e273633fa0 2024-03-14T09:55:41.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.872+0000 7f9fd0f61b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27369e610 con 0x55e273633680 2024-03-14T09:55:41.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.872+0000 7f9fd0f61b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27369e750 con 0x55e27363dec0 2024-03-14T09:55:41.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.872+0000 7f9fd0f61b80 10 client.0 _ll_register_callbacks cb 0x7f9fb00074e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:41.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.872+0000 7f9fb6800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e273633680 0x55e273641ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.872+0000 7f9fb5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633fa0 0x55e273642530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.872+0000 7f9fb6800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e273633680 0x55e273641ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60068/0 (socket says 172.21.15.78:60068) 2024-03-14T09:55:41.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.872+0000 7f9fb6800700 1 -- 172.21.15.78:0/101918149 learned_addr learned my addr 172.21.15.78:0/101918149 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:41.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.872+0000 7f9fb7200700 1 --2- 172.21.15.78:0/101918149 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e27363dec0 0x55e273642a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.872+0000 7f9fb5e00700 1 -- 172.21.15.78:0/101918149 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e27363dec0 msgr2=0x55e273642a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.872+0000 7f9fb5e00700 1 --2- 172.21.15.78:0/101918149 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e27363dec0 0x55e273642a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.872+0000 7f9fb5e00700 1 -- 172.21.15.78:0/101918149 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e273633680 msgr2=0x55e273641ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:41.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.872+0000 7f9fb5e00700 1 --2- 172.21.15.78:0/101918149 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e273633680 0x55e273641ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.872+0000 7f9fb5e00700 1 -- 172.21.15.78:0/101918149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa4018040 con 0x55e273633fa0 2024-03-14T09:55:41.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.872+0000 7f9fb7200700 1 --2- 172.21.15.78:0/101918149 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e27363dec0 0x55e273642a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:55:41.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.872+0000 7f9fb6800700 1 --2- 172.21.15.78:0/101918149 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e273633680 0x55e273641ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:41.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.873+0000 7f9fb5e00700 1 --2- 172.21.15.78:0/101918149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633fa0 0x55e273642530 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x55e2736506e0 tx=0x7f9fa40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74880f894f499a06 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:41.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.873+0000 7f9f97e00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:55:41.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.873+0000 7f9f97e00700 1 -- 172.21.15.78:0/101918149 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4025040 con 0x55e273633fa0 2024-03-14T09:55:41.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.873+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/101918149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9fa8004390 con 0x55e273633fa0 2024-03-14T09:55:41.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.873+0000 7f9f97e00700 1 -- 172.21.15.78:0/101918149 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f9fa40293c0 con 0x55e273633fa0 2024-03-14T09:55:41.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.873+0000 7f9f97e00700 1 -- 172.21.15.78:0/101918149 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9fa40295b0 con 0x55e273633fa0 2024-03-14T09:55:41.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.873+0000 7f9fd0f61b80 10 client.16005 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:41.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.873+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/101918149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f9fb002c070 con 0x55e273633fa0 2024-03-14T09:55:41.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.873+0000 7f9f97e00700 1 -- 172.21.15.78:0/101918149 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fa8004390 con 0x55e273633fa0 2024-03-14T09:55:41.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.873+0000 7f9f97e00700 1 -- 172.21.15.78:0/101918149 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9fa402f030 con 0x55e273633fa0 2024-03-14T09:55:41.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.873+0000 7f9fd0f61b80 10 client.16005 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:41.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.873+0000 7f9fd0f61b80 10 client.16005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:55:41.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.873+0000 7f9fd0f61b80 20 client.16005 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:41.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.873+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/101918149 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e27369fea0 0x55e2736a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:41.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.874+0000 7f9fd0f61b80 4 client.16005 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:55:41.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.874+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/101918149 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e2733d4cf0 con 0x55e27369fea0 2024-03-14T09:55:41.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.874+0000 7f9fb6800700 1 --2- 172.21.15.78:0/101918149 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e27369fea0 0x55e2736a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:41.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.874+0000 7f9f97e00700 10 client.16005 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:41.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.874+0000 7f9fb6800700 1 --2- 172.21.15.78:0/101918149 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e27369fea0 0x55e2736a22c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=705f121a21c04506 server_cookie=feb175bf6e90df3e in_seq=0 out_seq=0 2024-03-14T09:55:41.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.875+0000 7f9f97e00700 1 -- 172.21.15.78:0/101918149 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e2733d4cf0 con 0x55e27369fea0 2024-03-14T09:55:41.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.875+0000 7f9f97e00700 10 client.16005 handle_command_reply: tid=0 2024-03-14T09:55:41.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.875+0000 7f9fd0f61b80 1 client.16005 shutdown 2024-03-14T09:55:41.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.875+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/101918149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633fa0 msgr2=0x55e273642530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.875+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/101918149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633fa0 0x55e273642530 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x55e2736506e0 tx=0x7f9fa40047c0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.875+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/101918149 shutdown_connections 2024-03-14T09:55:41.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.875+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/101918149 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e27369fea0 0x55e2736a22c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.875+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/101918149 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e273633680 0x55e273641ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.875+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/101918149 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e27363dec0 0x55e273642a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.875+0000 7f9fd0f61b80 1 --2- 172.21.15.78:0/101918149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e273633fa0 0x55e273642530 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.875+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/101918149 >> 172.21.15.78:0/101918149 conn(0x55e2735a9220 msgr2=0x55e273639320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.875+0000 7f9f97e00700 0 client.16005 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:41.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.875+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/101918149 shutdown_connections 2024-03-14T09:55:41.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.875+0000 7f9fd0f61b80 1 -- 172.21.15.78:0/101918149 wait complete. 2024-03-14T09:55:41.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.875+0000 7f9fd0f61b80 20 client.16005 trim_cache size 0 max 16384 2024-03-14T09:55:41.880 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:55:41.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.876+0000 7f9fbd400700 1 -- 172.21.15.78:0/3937031093 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f9f9c0a0d20 msgr2=0x7f9f9c0a31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.876+0000 7f9fbd400700 1 --2- 172.21.15.78:0/3937031093 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f9f9c0a0d20 0x7f9f9c0a31e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4009910 tx=0x7f9fa4009e00 comp rx=0 tx=0).stop 2024-03-14T09:55:41.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.876+0000 7f9fbd400700 1 -- 172.21.15.78:0/3937031093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9fb81997d0 msgr2=0x7f9fb8199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:41.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.876+0000 7f9fbd400700 1 --2- 172.21.15.78:0/3937031093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9fb81997d0 0x7f9fb8199c80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9fb000b070 tx=0x7f9fb000bcf0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.876+0000 7f9fbd400700 1 -- 172.21.15.78:0/3937031093 shutdown_connections 2024-03-14T09:55:41.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.876+0000 7f9fbd400700 1 --2- 172.21.15.78:0/3937031093 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f9f9c0a0d20 0x7f9f9c0a31e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.876+0000 7f9fbd400700 1 --2- 172.21.15.78:0/3937031093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9fb81997d0 0x7f9fb8199c80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.876+0000 7f9fbd400700 1 --2- 172.21.15.78:0/3937031093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9fb8100f20 0x7f9fb819f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.876+0000 7f9fbd400700 1 --2- 172.21.15.78:0/3937031093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb8106040 0x7f9fb819fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:41.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.876+0000 7f9fbd400700 1 -- 172.21.15.78:0/3937031093 >> 172.21.15.78:0/3937031093 conn(0x7f9fb80fc7a0 msgr2=0x7f9fb80fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:41.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.876+0000 7f9fbd400700 1 -- 172.21.15.78:0/3937031093 shutdown_connections 2024-03-14T09:55:41.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:41.876+0000 7f9fbd400700 1 -- 172.21.15.78:0/3937031093 wait complete. 2024-03-14T09:55:42.111 INFO:tasks.workunit.client.0.smithi078.stdout:autoheader 2024-03-14T09:55:42.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:42 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/715511122' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:55:42.495 INFO:tasks.workunit.client.0.smithi078.stdout:autoconf 2024-03-14T09:55:42.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:42 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/715511122' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:55:42.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:42 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/715511122' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:55:42.709 INFO:tasks.workunit.client.0.smithi078.stdout:./configure \ 2024-03-14T09:55:42.710 INFO:tasks.workunit.client.0.smithi078.stdout: --libexecdir=/usr/lib \ 2024-03-14T09:55:42.710 INFO:tasks.workunit.client.0.smithi078.stdout: --exec_prefix=/var/lib 2024-03-14T09:55:42.895 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T09:55:42.895 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T09:55:43.001 INFO:tasks.workunit.client.0.smithi078.stdout:checking build system type... x86_64-pc-linux-gnu 2024-03-14T09:55:43.001 INFO:tasks.workunit.client.0.smithi078.stdout:checking host system type... x86_64-pc-linux-gnu 2024-03-14T09:55:43.004 INFO:tasks.workunit.client.0.smithi078.stdout:checking how to print strings... printf 2024-03-14T09:55:43.005 INFO:tasks.workunit.client.0.smithi078.stdout:checking for gcc... gcc 2024-03-14T09:55:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.103+0000 7fb0d7800700 1 -- 172.21.15.78:0/2860272803 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0d00731e0 msgr2=0x7fb0d00735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.103+0000 7fb0d7800700 1 --2- 172.21.15.78:0/2860272803 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0d00731e0 0x7fb0d00735c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4009990 tx=0x7fb0c4009ca0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.103+0000 7fb0d7800700 1 -- 172.21.15.78:0/2860272803 shutdown_connections 2024-03-14T09:55:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.103+0000 7fb0d7800700 1 --2- 172.21.15.78:0/2860272803 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0d010d4f0 0x7fb0d010f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.103+0000 7fb0d7800700 1 --2- 172.21.15.78:0/2860272803 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0d00731e0 0x7fb0d00735c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.103+0000 7fb0d7800700 1 --2- 172.21.15.78:0/2860272803 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0d0073b00 0x7fb0d010cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.103+0000 7fb0d7800700 1 -- 172.21.15.78:0/2860272803 >> 172.21.15.78:0/2860272803 conn(0x7fb0d00fc7a0 msgr2=0x7fb0d00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:43.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.103+0000 7fb0d7800700 1 -- 172.21.15.78:0/2860272803 shutdown_connections 2024-03-14T09:55:43.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.103+0000 7fb0d7800700 1 -- 172.21.15.78:0/2860272803 wait complete. 2024-03-14T09:55:43.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.104+0000 7fb0d7800700 1 Processor -- start 2024-03-14T09:55:43.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.105+0000 7fb0d7800700 1 -- start start 2024-03-14T09:55:43.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.105+0000 7fb0d7800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0d00731e0 0x7fb0d019f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.105+0000 7fb0d7800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0d0073b00 0x7fb0d019f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.105+0000 7fb0d7800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0d010d4f0 0x7fb0d0199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.105+0000 7fb0d7800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d0199cb0 con 0x7fb0d010d4f0 2024-03-14T09:55:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.105+0000 7fb0d7800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d0199e20 con 0x7fb0d0073b00 2024-03-14T09:55:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.105+0000 7fb0d7800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d0199fe0 con 0x7fb0d00731e0 2024-03-14T09:55:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.105+0000 7fb0cfe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0d0073b00 0x7fb0d019f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.105+0000 7fb0d4e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0d00731e0 0x7fb0d019f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.105+0000 7fb0d5800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0d010d4f0 0x7fb0d0199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.105+0000 7fb0cfe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0d0073b00 0x7fb0d019f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60080/0 (socket says 172.21.15.78:60080) 2024-03-14T09:55:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.105+0000 7fb0cfe00700 1 -- 172.21.15.78:0/1309298960 learned_addr learned my addr 172.21.15.78:0/1309298960 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.106+0000 7fb0cfe00700 1 -- 172.21.15.78:0/1309298960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0d00731e0 msgr2=0x7fb0d019f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.106+0000 7fb0cfe00700 1 --2- 172.21.15.78:0/1309298960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0d00731e0 0x7fb0d019f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.106+0000 7fb0cfe00700 1 -- 172.21.15.78:0/1309298960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0d010d4f0 msgr2=0x7fb0d0199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.106+0000 7fb0cfe00700 1 --2- 172.21.15.78:0/1309298960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0d010d4f0 0x7fb0d0199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.106+0000 7fb0cfe00700 1 -- 172.21.15.78:0/1309298960 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c40095f0 con 0x7fb0d0073b00 2024-03-14T09:55:43.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.106+0000 7fb0d5800700 1 --2- 172.21.15.78:0/1309298960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0d010d4f0 0x7fb0d0199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:43.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.106+0000 7fb0cfe00700 1 --2- 172.21.15.78:0/1309298960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0d0073b00 0x7fb0d019f9b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00d950 tx=0x7fb0bc00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:43.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.106+0000 7fb0cd600700 1 -- 172.21.15.78:0/1309298960 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc00f890 con 0x7fb0d0073b00 2024-03-14T09:55:43.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.106+0000 7fb0d7800700 1 -- 172.21.15.78:0/1309298960 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d019a2c0 con 0x7fb0d0073b00 2024-03-14T09:55:43.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.106+0000 7fb0cd600700 1 -- 172.21.15.78:0/1309298960 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0bc010460 con 0x7fb0d0073b00 2024-03-14T09:55:43.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.106+0000 7fb0cd600700 1 -- 172.21.15.78:0/1309298960 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc00e610 con 0x7fb0d0073b00 2024-03-14T09:55:43.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.106+0000 7fb0d7800700 1 -- 172.21.15.78:0/1309298960 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d01a6510 con 0x7fb0d0073b00 2024-03-14T09:55:43.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.110+0000 7fb0cd600700 1 -- 172.21.15.78:0/1309298960 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fb0bc0108d0 con 0x7fb0d0073b00 2024-03-14T09:55:43.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.111+0000 7fb0cd600700 1 --2- 172.21.15.78:0/1309298960 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0b80a0c00 0x7fb0b80a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.111+0000 7fb0cd600700 1 -- 172.21.15.78:0/1309298960 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fb0bc0cc5b0 con 0x7fb0d0073b00 2024-03-14T09:55:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.111+0000 7fb0d4e00700 1 --2- 172.21.15.78:0/1309298960 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0b80a0c00 0x7fb0b80a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.112+0000 7fb0eb41fb80 1 Processor -- start 2024-03-14T09:55:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.112+0000 7fb0eb41fb80 1 -- start start 2024-03-14T09:55:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.112+0000 7fb0d4e00700 1 --2- 172.21.15.78:0/1309298960 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0b80a0c00 0x7fb0b80a30c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4006010 tx=0x7fb0c400b260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:43.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.112+0000 7fb0eb41fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262f9700 0x55fe262f9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.112+0000 7fb0eb41fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe262fa020 0x55fe26303a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.112+0000 7fb0eb41fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe26303f40 0x55fe26306330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.112+0000 7fb0eb41fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe26252a10 con 0x55fe26303f40 2024-03-14T09:55:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.112+0000 7fb0eb41fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe262503d0 con 0x55fe262f9700 2024-03-14T09:55:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.112+0000 7fb0eb41fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe2624bcc0 con 0x55fe262fa020 2024-03-14T09:55:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.113+0000 7fb0d4e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262f9700 0x55fe262f9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.113+0000 7fb0d4e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262f9700 0x55fe262f9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60082/0 (socket says 172.21.15.78:60082) 2024-03-14T09:55:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.113+0000 7fb0d4e00700 1 -- 172.21.15.78:0/3091750792 learned_addr learned my addr 172.21.15.78:0/3091750792 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.113+0000 7fb0d5800700 1 --2- 172.21.15.78:0/3091750792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe26303f40 0x55fe26306330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.113+0000 7fb0cfe00700 1 --2- 172.21.15.78:0/3091750792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe262fa020 0x55fe26303a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.113+0000 7fb0d4e00700 1 -- 172.21.15.78:0/3091750792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe262fa020 msgr2=0x55fe26303a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.113+0000 7fb0d4e00700 1 --2- 172.21.15.78:0/3091750792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe262fa020 0x55fe26303a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.113+0000 7fb0d4e00700 1 -- 172.21.15.78:0/3091750792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe26303f40 msgr2=0x55fe26306330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:43.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.113+0000 7fb0d4e00700 1 --2- 172.21.15.78:0/3091750792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe26303f40 0x55fe26306330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.113+0000 7fb0d4e00700 1 -- 172.21.15.78:0/3091750792 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe260cd430 con 0x55fe262f9700 2024-03-14T09:55:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.113+0000 7fb0cfe00700 1 --2- 172.21.15.78:0/3091750792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe262fa020 0x55fe26303a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.113+0000 7fb0d4e00700 1 --2- 172.21.15.78:0/3091750792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262f9700 0x55fe262f9ae0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4015000 tx=0x7fb0c4020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15e916929bedd5f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.114+0000 7fb0c2000700 1 -- 172.21.15.78:0/3091750792 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4004700 con 0x55fe262f9700 2024-03-14T09:55:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.114+0000 7fb0c2000700 1 -- 172.21.15.78:0/3091750792 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0c401e040 con 0x55fe262f9700 2024-03-14T09:55:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.114+0000 7fb0c2000700 1 -- 172.21.15.78:0/3091750792 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c400fa00 con 0x55fe262f9700 2024-03-14T09:55:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.115+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/3091750792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262f9700 msgr2=0x55fe262f9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.115+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/3091750792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262f9700 0x55fe262f9ae0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4015000 tx=0x7fb0c4020000 comp rx=0 tx=0).stop 2024-03-14T09:55:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.115+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/3091750792 shutdown_connections 2024-03-14T09:55:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.115+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/3091750792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262f9700 0x55fe262f9ae0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.115+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/3091750792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe26303f40 0x55fe26306330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.115+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/3091750792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe262fa020 0x55fe26303a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.115+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/3091750792 >> 172.21.15.78:0/3091750792 conn(0x55fe2626f2a0 msgr2=0x55fe262f4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.115+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/3091750792 shutdown_connections 2024-03-14T09:55:43.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.115+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/3091750792 wait complete. 2024-03-14T09:55:43.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.115+0000 7fb0eb41fb80 1 -- start start 2024-03-14T09:55:43.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.116+0000 7fb0eb41fb80 1 Processor -- start 2024-03-14T09:55:43.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.116+0000 7fb0eb41fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262f9700 0x55fe263039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.116+0000 7fb0eb41fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262fa020 0x55fe26300ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.116+0000 7fb0eb41fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe26303f40 0x55fe26301420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.116+0000 7fb0eb41fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe261e7e00 con 0x55fe262f9700 2024-03-14T09:55:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.116+0000 7fb0eb41fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe261e7aa0 con 0x55fe262fa020 2024-03-14T09:55:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.116+0000 7fb0eb41fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe260cf720 con 0x55fe26303f40 2024-03-14T09:55:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.116+0000 7fb0eb41fb80 10 client.0 _ll_register_callbacks cb 0x55fe26053420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.116+0000 7fb0d4e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262f9700 0x55fe263039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.116+0000 7fb0d4e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262f9700 0x55fe263039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47354/0 (socket says 172.21.15.78:47354) 2024-03-14T09:55:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.116+0000 7fb0d4e00700 1 -- 172.21.15.78:0/2247710810 learned_addr learned my addr 172.21.15.78:0/2247710810 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.117+0000 7fb0cfe00700 1 --2- 172.21.15.78:0/2247710810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262fa020 0x55fe26300ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.117+0000 7fb0d5800700 1 --2- 172.21.15.78:0/2247710810 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe26303f40 0x55fe26301420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.117+0000 7fb0d4e00700 1 -- 172.21.15.78:0/2247710810 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe26303f40 msgr2=0x55fe26301420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.117+0000 7fb0d4e00700 1 --2- 172.21.15.78:0/2247710810 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe26303f40 0x55fe26301420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.117+0000 7fb0d4e00700 1 -- 172.21.15.78:0/2247710810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262fa020 msgr2=0x55fe26300ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.117+0000 7fb0d4e00700 1 --2- 172.21.15.78:0/2247710810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262fa020 0x55fe26300ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.117+0000 7fb0d4e00700 1 -- 172.21.15.78:0/2247710810 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0bc095040 con 0x55fe262f9700 2024-03-14T09:55:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.117+0000 7fb0d4e00700 1 --2- 172.21.15.78:0/2247710810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262f9700 0x55fe263039a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4009350 tx=0x7fb0c400fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb90fa20e7eb04b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.117+0000 7fb0b6a00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:55:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.117+0000 7fb0b6a00700 1 -- 172.21.15.78:0/2247710810 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4004910 con 0x55fe262f9700 2024-03-14T09:55:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.117+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2247710810 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe262ec7a0 con 0x55fe262f9700 2024-03-14T09:55:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.118+0000 7fb0b6a00700 1 -- 172.21.15.78:0/2247710810 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fb0c4004a70 con 0x55fe262f9700 2024-03-14T09:55:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.118+0000 7fb0b6a00700 1 -- 172.21.15.78:0/2247710810 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0c4005690 con 0x55fe262f9700 2024-03-14T09:55:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.118+0000 7fb0eb41fb80 10 client.16017 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.118+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2247710810 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7fb0d0110620 con 0x55fe262f9700 2024-03-14T09:55:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.118+0000 7fb0b6a00700 1 -- 172.21.15.78:0/2247710810 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fe262ec7a0 con 0x55fe262f9700 2024-03-14T09:55:43.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.118+0000 7fb0b6a00700 1 -- 172.21.15.78:0/2247710810 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb0c402fc00 con 0x55fe262f9700 2024-03-14T09:55:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.118+0000 7fb0eb41fb80 10 client.16017 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.118+0000 7fb0eb41fb80 10 client.16017 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:55:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.118+0000 7fb0eb41fb80 20 client.16017 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.118+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/2247710810 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55fe26313650 0x55fe26315a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.118+0000 7fb0eb41fb80 4 client.16017 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:55:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.118+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2247710810 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe26244780 con 0x55fe26313650 2024-03-14T09:55:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.119+0000 7fb0cfe00700 1 --2- 172.21.15.78:0/2247710810 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55fe26313650 0x55fe26315a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.119+0000 7fb0cfe00700 1 --2- 172.21.15.78:0/2247710810 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55fe26313650 0x55fe26315a40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f631a253106d33a9 server_cookie=afd2c1f1ac2d65d9 in_seq=0 out_seq=0 2024-03-14T09:55:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.119+0000 7fb0b6a00700 10 client.16017 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.120+0000 7fb0b6a00700 1 -- 172.21.15.78:0/2247710810 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55fe26244780 con 0x55fe26313650 2024-03-14T09:55:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.120+0000 7fb0b6a00700 10 client.16017 handle_command_reply: tid=0 2024-03-14T09:55:43.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.120+0000 7fb0eb41fb80 1 client.16017 shutdown 2024-03-14T09:55:43.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.120+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2247710810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262f9700 msgr2=0x55fe263039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.120+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/2247710810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262f9700 0x55fe263039a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4009350 tx=0x7fb0c400fce0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.120+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2247710810 shutdown_connections 2024-03-14T09:55:43.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.120+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/2247710810 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe26303f40 0x55fe26301420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.120+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/2247710810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262f9700 0x55fe263039a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.120+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/2247710810 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55fe26313650 0x55fe26315a40 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.120+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/2247710810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262fa020 0x55fe26300ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.120+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2247710810 >> 172.21.15.78:0/2247710810 conn(0x55fe2626f2a0 msgr2=0x55fe262f4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:43.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.120+0000 7fb0b6a00700 0 client.16017 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:43.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.120+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2247710810 shutdown_connections 2024-03-14T09:55:43.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.120+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2247710810 wait complete. 2024-03-14T09:55:43.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.121+0000 7fb0eb41fb80 20 client.16017 trim_cache size 0 max 16384 2024-03-14T09:55:43.136 INFO:tasks.workunit.client.0.smithi078.stdout:checking whether the C compiler works... yes 2024-03-14T09:55:43.136 INFO:tasks.workunit.client.0.smithi078.stdout:checking for C compiler default output file name... a.out 2024-03-14T09:55:43.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.138+0000 7fb0eb41fb80 1 Processor -- start 2024-03-14T09:55:43.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.138+0000 7fb0eb41fb80 1 -- start start 2024-03-14T09:55:43.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.139+0000 7fb0eb41fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe262f9700 0x55fe2630abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.139+0000 7fb0eb41fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262fa020 0x55fe263005c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.139+0000 7fb0eb41fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe26303f40 0x55fe26300b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.139+0000 7fb0eb41fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe26301c30 con 0x55fe262fa020 2024-03-14T09:55:43.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.139+0000 7fb0eb41fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe262fab70 con 0x55fe26303f40 2024-03-14T09:55:43.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.139+0000 7fb0eb41fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe262eed80 con 0x55fe262f9700 2024-03-14T09:55:43.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.139+0000 7fb0cfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262fa020 0x55fe263005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.139+0000 7fb0d4e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe262f9700 0x55fe2630abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.139+0000 7fb0d5800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe26303f40 0x55fe26300b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.139+0000 7fb0cfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262fa020 0x55fe263005c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47370/0 (socket says 172.21.15.78:47370) 2024-03-14T09:55:43.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.139+0000 7fb0cfe00700 1 -- 172.21.15.78:0/2604542490 learned_addr learned my addr 172.21.15.78:0/2604542490 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:43.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.139+0000 7fb0d4e00700 1 -- 172.21.15.78:0/2604542490 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe26303f40 msgr2=0x55fe26300b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:43.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.140+0000 7fb0d4e00700 1 --2- 172.21.15.78:0/2604542490 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe26303f40 0x55fe26300b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.140+0000 7fb0d4e00700 1 -- 172.21.15.78:0/2604542490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262fa020 msgr2=0x55fe263005c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:43.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.140+0000 7fb0d4e00700 1 --2- 172.21.15.78:0/2604542490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262fa020 0x55fe263005c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.140+0000 7fb0d4e00700 1 -- 172.21.15.78:0/2604542490 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe26079be0 con 0x55fe262f9700 2024-03-14T09:55:43.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.140+0000 7fb0d5800700 1 --2- 172.21.15.78:0/2604542490 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe26303f40 0x55fe26300b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:43.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.140+0000 7fb0cfe00700 1 --2- 172.21.15.78:0/2604542490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262fa020 0x55fe263005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:43.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.140+0000 7fb0d4e00700 1 --2- 172.21.15.78:0/2604542490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe262f9700 0x55fe2630abb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4000c00 tx=0x7fb0c4009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8446693e443acea server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:43.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.140+0000 7fb0c2a00700 1 -- 172.21.15.78:0/2604542490 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c400fd50 con 0x55fe262f9700 2024-03-14T09:55:43.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.140+0000 7fb0c2a00700 1 -- 172.21.15.78:0/2604542490 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0c4005640 con 0x55fe262f9700 2024-03-14T09:55:43.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.140+0000 7fb0c2a00700 1 -- 172.21.15.78:0/2604542490 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4004af0 con 0x55fe262f9700 2024-03-14T09:55:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.141+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2604542490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe262f9700 msgr2=0x55fe2630abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.141+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/2604542490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe262f9700 0x55fe2630abb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4000c00 tx=0x7fb0c4009ca0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.141+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2604542490 shutdown_connections 2024-03-14T09:55:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.141+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/2604542490 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe26303f40 0x55fe26300b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.141+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/2604542490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe262f9700 0x55fe2630abb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.141+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/2604542490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262fa020 0x55fe263005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.141+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2604542490 >> 172.21.15.78:0/2604542490 conn(0x55fe2626f2a0 msgr2=0x55fe262f4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.141+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2604542490 shutdown_connections 2024-03-14T09:55:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.141+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2604542490 wait complete. 2024-03-14T09:55:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.142+0000 7fb0eb41fb80 1 -- start start 2024-03-14T09:55:43.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.142+0000 7fb0eb41fb80 1 Processor -- start 2024-03-14T09:55:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0eb41fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262f9700 0x55fe26309310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0eb41fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262fa020 0x55fe26309850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0eb41fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe26303f40 0x55fe26309d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0eb41fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe26365180 con 0x55fe262fa020 2024-03-14T09:55:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0eb41fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe263652f0 con 0x55fe262f9700 2024-03-14T09:55:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0eb41fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe263654b0 con 0x55fe26303f40 2024-03-14T09:55:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0eb41fb80 10 client.0 _ll_register_callbacks cb 0x7fb0c8009f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0d4e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262f9700 0x55fe26309310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0d4e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262f9700 0x55fe26309310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60100/0 (socket says 172.21.15.78:60100) 2024-03-14T09:55:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0cfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262fa020 0x55fe26309850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0d4e00700 1 -- 172.21.15.78:0/2423228750 learned_addr learned my addr 172.21.15.78:0/2423228750 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:43.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0d5800700 1 --2- 172.21.15.78:0/2423228750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe26303f40 0x55fe26309d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0d4e00700 1 -- 172.21.15.78:0/2423228750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe26303f40 msgr2=0x55fe26309d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0d4e00700 1 --2- 172.21.15.78:0/2423228750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe26303f40 0x55fe26309d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0d4e00700 1 -- 172.21.15.78:0/2423228750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262fa020 msgr2=0x55fe26309850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0d4e00700 1 --2- 172.21.15.78:0/2423228750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262fa020 0x55fe26309850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0d4e00700 1 -- 172.21.15.78:0/2423228750 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d01a6510 con 0x55fe262f9700 2024-03-14T09:55:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.143+0000 7fb0cfe00700 1 --2- 172.21.15.78:0/2423228750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262fa020 0x55fe26309850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.144+0000 7fb0d4e00700 1 --2- 172.21.15.78:0/2423228750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262f9700 0x55fe26309310 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40057d0 tx=0x7fb0c4039d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc1500c3dd79f6e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.144+0000 7fb0b6a00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:55:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.144+0000 7fb0b6a00700 1 -- 172.21.15.78:0/2423228750 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4004710 con 0x55fe262f9700 2024-03-14T09:55:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.144+0000 7fb0b6a00700 1 -- 172.21.15.78:0/2423228750 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fb0c4004870 con 0x55fe262f9700 2024-03-14T09:55:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.144+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2423228750 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb0c401e040 con 0x55fe262f9700 2024-03-14T09:55:43.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.144+0000 7fb0b6a00700 1 -- 172.21.15.78:0/2423228750 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0c4004a30 con 0x55fe262f9700 2024-03-14T09:55:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.144+0000 7fb0eb41fb80 10 client.26027 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:43.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.144+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2423228750 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7fb0c800e600 con 0x55fe262f9700 2024-03-14T09:55:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.145+0000 7fb0b6a00700 1 -- 172.21.15.78:0/2423228750 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0c401e040 con 0x55fe262f9700 2024-03-14T09:55:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.145+0000 7fb0b6a00700 1 -- 172.21.15.78:0/2423228750 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb0c400eb60 con 0x55fe262f9700 2024-03-14T09:55:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.145+0000 7fb0eb41fb80 10 client.26027 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.145+0000 7fb0eb41fb80 10 client.26027 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:55:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.145+0000 7fb0eb41fb80 20 client.26027 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.145+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/2423228750 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55fe26366810 0x55fe26368c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.145+0000 7fb0eb41fb80 4 client.26027 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T09:55:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.145+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2423228750 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55fe2609ad70 con 0x55fe26366810 2024-03-14T09:55:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.145+0000 7fb0cfe00700 1 --2- 172.21.15.78:0/2423228750 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55fe26366810 0x55fe26368c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.146+0000 7fb0cfe00700 1 --2- 172.21.15.78:0/2423228750 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55fe26366810 0x55fe26368c00 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=880a52db1b645b64 server_cookie=5f50394e1a85248f in_seq=0 out_seq=0 2024-03-14T09:55:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.146+0000 7fb0b6a00700 10 client.26027 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:43.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.147+0000 7fb0b6a00700 1 -- 172.21.15.78:0/2423228750 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55fe2609ad70 con 0x55fe26366810 2024-03-14T09:55:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.147+0000 7fb0b6a00700 10 client.26027 handle_command_reply: tid=0 2024-03-14T09:55:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.147+0000 7fb0eb41fb80 1 client.26027 shutdown 2024-03-14T09:55:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.147+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2423228750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262f9700 msgr2=0x55fe26309310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.147+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/2423228750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262f9700 0x55fe26309310 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40057d0 tx=0x7fb0c4039d50 comp rx=0 tx=0).stop 2024-03-14T09:55:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.147+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2423228750 shutdown_connections 2024-03-14T09:55:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.147+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/2423228750 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55fe26366810 0x55fe26368c00 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.147+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/2423228750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fe262f9700 0x55fe26309310 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.147+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/2423228750 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fe26303f40 0x55fe26309d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.147+0000 7fb0eb41fb80 1 --2- 172.21.15.78:0/2423228750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe262fa020 0x55fe26309850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.147+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2423228750 >> 172.21.15.78:0/2423228750 conn(0x55fe2626f2a0 msgr2=0x55fe262f85e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.147+0000 7fb0b6a00700 0 client.26027 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.147+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2423228750 shutdown_connections 2024-03-14T09:55:43.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.147+0000 7fb0eb41fb80 1 -- 172.21.15.78:0/2423228750 wait complete. 2024-03-14T09:55:43.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.148+0000 7fb0eb41fb80 20 client.26027 trim_cache size 0 max 16384 2024-03-14T09:55:43.153 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:55:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T09:55:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "a8535772-78a2-4891-9785-461fe2d1675c", 2024-03-14T09:55:43.153 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T09:55:43.153 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:55:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.148+0000 7fb0d7800700 1 -- 172.21.15.78:0/1309298960 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0b80a0c00 msgr2=0x7fb0b80a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.148+0000 7fb0d7800700 1 --2- 172.21.15.78:0/1309298960 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0b80a0c00 0x7fb0b80a30c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4006010 tx=0x7fb0c400b260 comp rx=0 tx=0).stop 2024-03-14T09:55:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.148+0000 7fb0d7800700 1 -- 172.21.15.78:0/1309298960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0d0073b00 msgr2=0x7fb0d019f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.148+0000 7fb0d7800700 1 --2- 172.21.15.78:0/1309298960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0d0073b00 0x7fb0d019f9b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc00d950 tx=0x7fb0bc00dc60 comp rx=0 tx=0).stop 2024-03-14T09:55:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.148+0000 7fb0d7800700 1 -- 172.21.15.78:0/1309298960 shutdown_connections 2024-03-14T09:55:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.148+0000 7fb0d7800700 1 --2- 172.21.15.78:0/1309298960 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0d010d4f0 0x7fb0d0199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.148+0000 7fb0d7800700 1 --2- 172.21.15.78:0/1309298960 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0d00731e0 0x7fb0d019f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.148+0000 7fb0d7800700 1 --2- 172.21.15.78:0/1309298960 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0b80a0c00 0x7fb0b80a30c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.148+0000 7fb0d7800700 1 --2- 172.21.15.78:0/1309298960 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0d0073b00 0x7fb0d019f9b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.148+0000 7fb0d7800700 1 -- 172.21.15.78:0/1309298960 >> 172.21.15.78:0/1309298960 conn(0x7fb0d00fc7a0 msgr2=0x7fb0d00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:43.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.148+0000 7fb0d7800700 1 -- 172.21.15.78:0/1309298960 shutdown_connections 2024-03-14T09:55:43.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.148+0000 7fb0d7800700 1 -- 172.21.15.78:0/1309298960 wait complete. 2024-03-14T09:55:43.169 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:55:43.233 INFO:tasks.workunit.client.0.smithi078.stdout:checking for suffix of executables... 2024-03-14T09:55:43.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:43 smithi078 ceph-mon[38143]: pgmap v49: 129 pgs: 129 active+clean; 49 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 54 op/s 2024-03-14T09:55:43.272 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:43 smithi078 ceph-mon[38143]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-03-14T09:55:43.302 INFO:tasks.workunit.client.0.smithi078.stdout:checking whether we are cross compiling... no 2024-03-14T09:55:43.365 INFO:tasks.workunit.client.0.smithi078.stdout:checking for suffix of object files... o 2024-03-14T09:55:43.411 INFO:tasks.workunit.client.0.smithi078.stdout:checking whether we are using the GNU C compiler... yes 2024-03-14T09:55:43.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.441+0000 7fd895000700 1 -- 172.21.15.78:0/805391913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd890108260 msgr2=0x7fd89010fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.441+0000 7fd895000700 1 --2- 172.21.15.78:0/805391913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd890108260 0x7fd89010fde0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd888009f90 tx=0x7fd88800b330 comp rx=0 tx=0).stop 2024-03-14T09:55:43.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.441+0000 7fd895000700 1 -- 172.21.15.78:0/805391913 shutdown_connections 2024-03-14T09:55:43.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.441+0000 7fd895000700 1 --2- 172.21.15.78:0/805391913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd890103140 0x7fd890103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.441+0000 7fd895000700 1 --2- 172.21.15.78:0/805391913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd890108260 0x7fd89010fde0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.441+0000 7fd895000700 1 --2- 172.21.15.78:0/805391913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd890103af0 0x7fd890107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.441+0000 7fd895000700 1 -- 172.21.15.78:0/805391913 >> 172.21.15.78:0/805391913 conn(0x7fd8900fe980 msgr2=0x7fd890100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:43.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.441+0000 7fd895000700 1 -- 172.21.15.78:0/805391913 shutdown_connections 2024-03-14T09:55:43.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.441+0000 7fd895000700 1 -- 172.21.15.78:0/805391913 wait complete. 2024-03-14T09:55:43.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.443+0000 7fd895000700 1 Processor -- start 2024-03-14T09:55:43.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.443+0000 7fd895000700 1 -- start start 2024-03-14T09:55:43.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.443+0000 7fd895000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd890103140 0x7fd89019f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.443+0000 7fd895000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd890108260 0x7fd89019fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.444+0000 7fd895000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd890199790 0x7fd890199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.444+0000 7fd895000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89019a210 con 0x7fd890108260 2024-03-14T09:55:43.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.444+0000 7fd895000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89019a350 con 0x7fd890103140 2024-03-14T09:55:43.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.444+0000 7fd895000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89019a510 con 0x7fd890199790 2024-03-14T09:55:43.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.444+0000 7fd88de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd890103140 0x7fd89019f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.444+0000 7fd88de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd890103140 0x7fd89019f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60116/0 (socket says 172.21.15.78:60116) 2024-03-14T09:55:43.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.444+0000 7fd88de00700 1 -- 172.21.15.78:0/2394855022 learned_addr learned my addr 172.21.15.78:0/2394855022 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:43.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.444+0000 7fd88e800700 1 --2- 172.21.15.78:0/2394855022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd890199790 0x7fd890199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.444+0000 7fd88d400700 1 --2- 172.21.15.78:0/2394855022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd890108260 0x7fd89019fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.444+0000 7fd88de00700 1 -- 172.21.15.78:0/2394855022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd890199790 msgr2=0x7fd890199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.444+0000 7fd88de00700 1 --2- 172.21.15.78:0/2394855022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd890199790 0x7fd890199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.444+0000 7fd88de00700 1 -- 172.21.15.78:0/2394855022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd890108260 msgr2=0x7fd89019fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.444+0000 7fd88de00700 1 --2- 172.21.15.78:0/2394855022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd890108260 0x7fd89019fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.444+0000 7fd88de00700 1 -- 172.21.15.78:0/2394855022 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd888009c40 con 0x7fd890103140 2024-03-14T09:55:43.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.444+0000 7fd88e800700 1 --2- 172.21.15.78:0/2394855022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd890199790 0x7fd890199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:43.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.444+0000 7fd88d400700 1 --2- 172.21.15.78:0/2394855022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd890108260 0x7fd89019fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:55:43.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.445+0000 7fd88de00700 1 --2- 172.21.15.78:0/2394855022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd890103140 0x7fd89019f660 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd88000e9c0 tx=0x7fd88000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:43.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.445+0000 7fd886a00700 1 -- 172.21.15.78:0/2394855022 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88000cbd0 con 0x7fd890103140 2024-03-14T09:55:43.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.445+0000 7fd886a00700 1 -- 172.21.15.78:0/2394855022 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd88000eeb0 con 0x7fd890103140 2024-03-14T09:55:43.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.445+0000 7fd886a00700 1 -- 172.21.15.78:0/2394855022 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880005580 con 0x7fd890103140 2024-03-14T09:55:43.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.445+0000 7fd895000700 1 -- 172.21.15.78:0/2394855022 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8901a64d0 con 0x7fd890103140 2024-03-14T09:55:43.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.445+0000 7fd895000700 1 -- 172.21.15.78:0/2394855022 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8901a6990 con 0x7fd890103140 2024-03-14T09:55:43.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.449+0000 7fd886a00700 1 -- 172.21.15.78:0/2394855022 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fd880004750 con 0x7fd890103140 2024-03-14T09:55:43.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.450+0000 7fd8a8acab80 1 Processor -- start 2024-03-14T09:55:43.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.451+0000 7fd8a8acab80 1 -- start start 2024-03-14T09:55:43.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.451+0000 7fd886a00700 1 --2- 172.21.15.78:0/2394855022 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd8740a0c80 0x7fd8740a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.451+0000 7fd8a8acab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228b0680 0x5571228b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.451+0000 7fd8a8acab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0fa0 0x5571228ba980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.451+0000 7fd8a8acab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228baec0 0x5571228bd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.451+0000 7fd8a8acab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557122809990 con 0x5571228baec0 2024-03-14T09:55:43.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.451+0000 7fd8a8acab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557122807350 con 0x5571228b0680 2024-03-14T09:55:43.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.451+0000 7fd8a8acab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557122802c40 con 0x5571228b0fa0 2024-03-14T09:55:43.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.451+0000 7fd886a00700 1 -- 172.21.15.78:0/2394855022 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fd880014070 con 0x7fd890103140 2024-03-14T09:55:43.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.451+0000 7fd88de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228b0680 0x5571228b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.452+0000 7fd88de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228b0680 0x5571228b0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60126/0 (socket says 172.21.15.78:60126) 2024-03-14T09:55:43.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.452+0000 7fd88d400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0fa0 0x5571228ba980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.452+0000 7fd88de00700 1 -- 172.21.15.78:0/3744503144 learned_addr learned my addr 172.21.15.78:0/3744503144 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:43.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.452+0000 7fd88e800700 1 --2- 172.21.15.78:0/3744503144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228baec0 0x5571228bd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.452+0000 7fd88e800700 1 --2- 172.21.15.78:0/2394855022 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd8740a0c80 0x7fd8740a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.452+0000 7fd88d400700 1 -- 172.21.15.78:0/3744503144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228b0680 msgr2=0x5571228b0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:43.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.452+0000 7fd88d400700 1 --2- 172.21.15.78:0/3744503144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228b0680 0x5571228b0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.452+0000 7fd88d400700 1 -- 172.21.15.78:0/3744503144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228baec0 msgr2=0x5571228bd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:43.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.452+0000 7fd88d400700 1 --2- 172.21.15.78:0/3744503144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228baec0 0x5571228bd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.452+0000 7fd88d400700 1 -- 172.21.15.78:0/3744503144 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571226843b0 con 0x5571228b0fa0 2024-03-14T09:55:43.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.452+0000 7fd88d400700 1 --2- 172.21.15.78:0/3744503144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0fa0 0x5571228ba980 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd89006a3a0 tx=0x7fd87c009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dbb21467a2401d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:43.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.453+0000 7fd87b400700 1 -- 172.21.15.78:0/3744503144 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c017e90 con 0x5571228b0fa0 2024-03-14T09:55:43.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.453+0000 7fd87b400700 1 -- 172.21.15.78:0/3744503144 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd87c00ea10 con 0x5571228b0fa0 2024-03-14T09:55:43.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.453+0000 7fd87b400700 1 -- 172.21.15.78:0/3744503144 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c022b10 con 0x5571228b0fa0 2024-03-14T09:55:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.453+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3744503144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0fa0 msgr2=0x5571228ba980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.453+0000 7fd88e800700 1 --2- 172.21.15.78:0/2394855022 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd8740a0c80 0x7fd8740a3140 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd8880060b0 tx=0x7fd888009700 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.453+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/3744503144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0fa0 0x5571228ba980 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd89006a3a0 tx=0x7fd87c009b60 comp rx=0 tx=0).stop 2024-03-14T09:55:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.453+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3744503144 shutdown_connections 2024-03-14T09:55:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.453+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/3744503144 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228b0680 0x5571228b0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.453+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/3744503144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228baec0 0x5571228bd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.453+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/3744503144 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0fa0 0x5571228ba980 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.453+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3744503144 >> 172.21.15.78:0/3744503144 conn(0x557122826220 msgr2=0x5571228abaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.454+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3744503144 shutdown_connections 2024-03-14T09:55:43.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.454+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3744503144 wait complete. 2024-03-14T09:55:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.454+0000 7fd8a8acab80 1 -- start start 2024-03-14T09:55:43.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.454+0000 7fd8a8acab80 1 Processor -- start 2024-03-14T09:55:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.455+0000 7fd8a8acab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0680 0x5571228ba830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.455+0000 7fd8a8acab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0fa0 0x5571228b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.455+0000 7fd8a8acab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228baec0 0x5571228b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.455+0000 7fd8a8acab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55712279ed80 con 0x5571228b0fa0 2024-03-14T09:55:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.455+0000 7fd8a8acab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55712279ea20 con 0x5571228baec0 2024-03-14T09:55:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.455+0000 7fd8a8acab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571226866a0 con 0x5571228b0680 2024-03-14T09:55:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.455+0000 7fd8a8acab80 10 client.0 _ll_register_callbacks cb 0x55712260a3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.455+0000 7fd88de00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0680 0x5571228ba830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.455+0000 7fd88de00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0680 0x5571228ba830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:56532/0 (socket says 172.21.15.78:56532) 2024-03-14T09:55:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.455+0000 7fd88de00700 1 -- 172.21.15.78:0/2300436586 learned_addr learned my addr 172.21.15.78:0/2300436586 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.455+0000 7fd88e800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228baec0 0x5571228b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.456+0000 7fd88d400700 1 --2- 172.21.15.78:0/2300436586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0fa0 0x5571228b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.456+0000 7fd88de00700 1 -- 172.21.15.78:0/2300436586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228baec0 msgr2=0x5571228b81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.456+0000 7fd88de00700 1 --2- 172.21.15.78:0/2300436586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228baec0 0x5571228b81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.456+0000 7fd88de00700 1 -- 172.21.15.78:0/2300436586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0fa0 msgr2=0x5571228b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.456+0000 7fd88de00700 1 --2- 172.21.15.78:0/2300436586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0fa0 0x5571228b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.456+0000 7fd88de00700 1 -- 172.21.15.78:0/2300436586 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd87c018d00 con 0x5571228b0680 2024-03-14T09:55:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.456+0000 7fd88d400700 1 --2- 172.21.15.78:0/2300436586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0fa0 0x5571228b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:55:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.456+0000 7fd88e800700 1 --2- 172.21.15.78:0/2300436586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228baec0 0x5571228b81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.456+0000 7fd88de00700 1 --2- 172.21.15.78:0/2300436586 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0680 0x5571228ba830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd880018000 tx=0x7fd88000cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2bf40e469802208 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.456+0000 7fd86fe00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:55:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.456+0000 7fd86fe00700 1 -- 172.21.15.78:0/2300436586 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880099b00 con 0x5571228b0680 2024-03-14T09:55:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.456+0000 7fd86fe00700 1 -- 172.21.15.78:0/2300436586 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fd88009d930 con 0x5571228b0680 2024-03-14T09:55:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.456+0000 7fd8a8acab80 1 -- 172.21.15.78:0/2300436586 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5571228a3720 con 0x5571228b0680 2024-03-14T09:55:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.456+0000 7fd86fe00700 1 -- 172.21.15.78:0/2300436586 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8800afea0 con 0x5571228b0680 2024-03-14T09:55:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.456+0000 7fd8a8acab80 10 client.26047 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.456+0000 7fd8a8acab80 1 -- 172.21.15.78:0/2300436586 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557122630b60 con 0x5571228b0680 2024-03-14T09:55:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.457+0000 7fd86fe00700 1 -- 172.21.15.78:0/2300436586 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5571228a3720 con 0x5571228b0680 2024-03-14T09:55:43.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.457+0000 7fd86fe00700 1 -- 172.21.15.78:0/2300436586 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd880099110 con 0x5571228b0680 2024-03-14T09:55:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.457+0000 7fd8a8acab80 10 client.26047 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.457+0000 7fd8a8acab80 10 client.26047 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:55:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.457+0000 7fd8a8acab80 20 client.26047 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.457+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/2300436586 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5571228caa60 0x5571228cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.457+0000 7fd8a8acab80 4 client.26047 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:55:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.457+0000 7fd8a8acab80 1 -- 172.21.15.78:0/2300436586 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5571227fb700 con 0x5571228caa60 2024-03-14T09:55:43.461 INFO:tasks.workunit.client.0.smithi078.stdout:checking whether gcc accepts -g... yes 2024-03-14T09:55:43.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.457+0000 7fd88d400700 1 --2- 172.21.15.78:0/2300436586 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5571228caa60 0x5571228cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.458+0000 7fd88d400700 1 --2- 172.21.15.78:0/2300436586 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5571228caa60 0x5571228cce80 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de11acb13c7e3416 server_cookie=a2131b9ba0a940e3 in_seq=0 out_seq=0 2024-03-14T09:55:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.458+0000 7fd86fe00700 10 client.26047 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.459+0000 7fd86fe00700 1 -- 172.21.15.78:0/2300436586 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5571227fb700 con 0x5571228caa60 2024-03-14T09:55:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.459+0000 7fd86fe00700 10 client.26047 handle_command_reply: tid=0 2024-03-14T09:55:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.459+0000 7fd8a8acab80 1 client.26047 shutdown 2024-03-14T09:55:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.459+0000 7fd8a8acab80 1 -- 172.21.15.78:0/2300436586 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0680 msgr2=0x5571228ba830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.459+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/2300436586 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0680 0x5571228ba830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd880018000 tx=0x7fd88000cde0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.459+0000 7fd8a8acab80 1 -- 172.21.15.78:0/2300436586 shutdown_connections 2024-03-14T09:55:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.459+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/2300436586 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5571228caa60 0x5571228cce80 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.459+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/2300436586 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228baec0 0x5571228b81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.459+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/2300436586 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0680 0x5571228ba830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.459+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/2300436586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0fa0 0x5571228b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.459+0000 7fd86fe00700 0 client.26047 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.459+0000 7fd8a8acab80 1 -- 172.21.15.78:0/2300436586 >> 172.21.15.78:0/2300436586 conn(0x557122826220 msgr2=0x5571228ab690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.459+0000 7fd8a8acab80 1 -- 172.21.15.78:0/2300436586 shutdown_connections 2024-03-14T09:55:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.459+0000 7fd8a8acab80 1 -- 172.21.15.78:0/2300436586 wait complete. 2024-03-14T09:55:43.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.460+0000 7fd8a8acab80 20 client.26047 trim_cache size 0 max 16384 2024-03-14T09:55:43.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.481+0000 7fd8a8acab80 1 Processor -- start 2024-03-14T09:55:43.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.481+0000 7fd8a8acab80 1 -- start start 2024-03-14T09:55:43.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.481+0000 7fd8a8acab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228b0680 0x5571228abc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.481+0000 7fd8a8acab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0fa0 0x5571228c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.481+0000 7fd8a8acab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228baec0 0x5571228b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.481+0000 7fd8a8acab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571228b2740 con 0x5571228b0fa0 2024-03-14T09:55:43.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.481+0000 7fd8a8acab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571228b1af0 con 0x5571228b0680 2024-03-14T09:55:43.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.481+0000 7fd8a8acab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571228ad090 con 0x5571228baec0 2024-03-14T09:55:43.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.481+0000 7fd88e800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228baec0 0x5571228b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.481+0000 7fd88e800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228baec0 0x5571228b4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:56544/0 (socket says 172.21.15.78:56544) 2024-03-14T09:55:43.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.481+0000 7fd88de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228b0680 0x5571228abc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.481+0000 7fd88e800700 1 -- 172.21.15.78:0/3555176538 learned_addr learned my addr 172.21.15.78:0/3555176538 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:43.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.481+0000 7fd88d400700 1 --2- 172.21.15.78:0/3555176538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0fa0 0x5571228c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.482+0000 7fd88e800700 1 -- 172.21.15.78:0/3555176538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228b0680 msgr2=0x5571228abc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:43.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.482+0000 7fd88e800700 1 --2- 172.21.15.78:0/3555176538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228b0680 0x5571228abc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.482+0000 7fd88e800700 1 -- 172.21.15.78:0/3555176538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0fa0 msgr2=0x5571228c1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:43.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.482+0000 7fd88e800700 1 --2- 172.21.15.78:0/3555176538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0fa0 0x5571228c1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.482+0000 7fd88e800700 1 -- 172.21.15.78:0/3555176538 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571227c8ea0 con 0x5571228baec0 2024-03-14T09:55:43.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.482+0000 7fd88e800700 1 --2- 172.21.15.78:0/3555176538 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228baec0 0x5571228b4dd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd888015000 tx=0x7fd888009360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57d134b47072fa5a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:43.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.482+0000 7fd87be00700 1 -- 172.21.15.78:0/3555176538 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd888026040 con 0x5571228baec0 2024-03-14T09:55:43.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.482+0000 7fd87be00700 1 -- 172.21.15.78:0/3555176538 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd88801e920 con 0x5571228baec0 2024-03-14T09:55:43.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.482+0000 7fd87be00700 1 -- 172.21.15.78:0/3555176538 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88800e040 con 0x5571228baec0 2024-03-14T09:55:43.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.483+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3555176538 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228baec0 msgr2=0x5571228b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.483+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/3555176538 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228baec0 0x5571228b4dd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd888015000 tx=0x7fd888009360 comp rx=0 tx=0).stop 2024-03-14T09:55:43.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.483+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3555176538 shutdown_connections 2024-03-14T09:55:43.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.483+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/3555176538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228b0680 0x5571228abc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.483+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/3555176538 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228baec0 0x5571228b4dd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.483+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/3555176538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0fa0 0x5571228c1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.483+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3555176538 >> 172.21.15.78:0/3555176538 conn(0x557122826220 msgr2=0x5571228b6440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:43.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.483+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3555176538 shutdown_connections 2024-03-14T09:55:43.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.483+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3555176538 wait complete. 2024-03-14T09:55:43.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.484+0000 7fd8a8acab80 1 -- start start 2024-03-14T09:55:43.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.484+0000 7fd8a8acab80 1 Processor -- start 2024-03-14T09:55:43.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.485+0000 7fd8a8acab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0680 0x5571228c19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.485+0000 7fd8a8acab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0fa0 0x5571228bfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.485+0000 7fd8a8acab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228baec0 0x5571228c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.485+0000 7fd8a8acab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571228c09d0 con 0x5571228b0680 2024-03-14T09:55:43.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.485+0000 7fd8a8acab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571228c0b40 con 0x5571228baec0 2024-03-14T09:55:43.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.485+0000 7fd8a8acab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55712291c590 con 0x5571228b0fa0 2024-03-14T09:55:43.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.485+0000 7fd8a8acab80 10 client.0 _ll_register_callbacks cb 0x7fd8800b26b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:55:43.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.485+0000 7fd88de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0680 0x5571228c19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.485+0000 7fd88d400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0fa0 0x5571228bfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.485+0000 7fd88de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0680 0x5571228c19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47430/0 (socket says 172.21.15.78:47430) 2024-03-14T09:55:43.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.485+0000 7fd88de00700 1 -- 172.21.15.78:0/3029820718 learned_addr learned my addr 172.21.15.78:0/3029820718 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:55:43.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.485+0000 7fd88e800700 1 --2- 172.21.15.78:0/3029820718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228baec0 0x5571228c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.485+0000 7fd88d400700 1 -- 172.21.15.78:0/3029820718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228baec0 msgr2=0x5571228c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:43.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.485+0000 7fd88d400700 1 --2- 172.21.15.78:0/3029820718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228baec0 0x5571228c02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.485+0000 7fd88d400700 1 -- 172.21.15.78:0/3029820718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0680 msgr2=0x5571228c19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:55:43.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.485+0000 7fd88d400700 1 --2- 172.21.15.78:0/3029820718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0680 0x5571228c19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.486+0000 7fd88d400700 1 -- 172.21.15.78:0/3029820718 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd890110650 con 0x5571228b0fa0 2024-03-14T09:55:43.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.486+0000 7fd88de00700 1 --2- 172.21.15.78:0/3029820718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0680 0x5571228c19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:55:43.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.486+0000 7fd88d400700 1 --2- 172.21.15.78:0/3029820718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0fa0 0x5571228bfda0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd87c022650 tx=0x7fd87c009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b463e03b40a009ed server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:55:43.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.486+0000 7fd86fe00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:55:43.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.486+0000 7fd86fe00700 1 -- 172.21.15.78:0/3029820718 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c00e550 con 0x5571228b0fa0 2024-03-14T09:55:43.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.486+0000 7fd86fe00700 1 -- 172.21.15.78:0/3029820718 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fd87c00e6b0 con 0x5571228b0fa0 2024-03-14T09:55:43.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.486+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3029820718 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd87c00ea10 con 0x5571228b0fa0 2024-03-14T09:55:43.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.487+0000 7fd86fe00700 1 -- 172.21.15.78:0/3029820718 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd87c00e870 con 0x5571228b0fa0 2024-03-14T09:55:43.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.487+0000 7fd8a8acab80 10 client.26053 fetch_fsmap learned FSMap version 25 2024-03-14T09:55:43.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.487+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3029820718 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557122688cd0 con 0x5571228b0fa0 2024-03-14T09:55:43.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.487+0000 7fd86fe00700 1 -- 172.21.15.78:0/3029820718 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd87c00ea10 con 0x5571228b0fa0 2024-03-14T09:55:43.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.487+0000 7fd86fe00700 1 -- 172.21.15.78:0/3029820718 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd87c034020 con 0x5571228b0fa0 2024-03-14T09:55:43.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.487+0000 7fd8a8acab80 10 client.26053 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:55:43.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.487+0000 7fd8a8acab80 10 client.26053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:55:43.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.487+0000 7fd8a8acab80 20 client.26053 populate_metadata read hostname 'smithi078' 2024-03-14T09:55:43.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.487+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/3029820718 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55712291dc80 0x5571229200a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:55:43.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.487+0000 7fd8a8acab80 4 client.26053 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:55:43.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.487+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3029820718 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557122651cf0 con 0x55712291dc80 2024-03-14T09:55:43.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.487+0000 7fd88de00700 1 --2- 172.21.15.78:0/3029820718 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55712291dc80 0x5571229200a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:55:43.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.488+0000 7fd88de00700 1 --2- 172.21.15.78:0/3029820718 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55712291dc80 0x5571229200a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=deac8724270a32f3 server_cookie=423cfb17da1ee5d5 in_seq=0 out_seq=0 2024-03-14T09:55:43.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.488+0000 7fd86fe00700 10 client.26053 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:43.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.488+0000 7fd86fe00700 1 -- 172.21.15.78:0/3029820718 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+267 (crc 0 0 0) 0x557122651cf0 con 0x55712291dc80 2024-03-14T09:55:43.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.488+0000 7fd86fe00700 10 client.26053 handle_command_reply: tid=0 2024-03-14T09:55:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.488+0000 7fd8a8acab80 1 client.26053 shutdown 2024-03-14T09:55:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.489+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3029820718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0fa0 msgr2=0x5571228bfda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.489+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/3029820718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0fa0 0x5571228bfda0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd87c022650 tx=0x7fd87c009240 comp rx=0 tx=0).stop 2024-03-14T09:55:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.489+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3029820718 shutdown_connections 2024-03-14T09:55:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.489+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/3029820718 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55712291dc80 0x5571229200a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.489+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/3029820718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571228baec0 0x5571228c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.489+0000 7fd86fe00700 0 client.26053 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:55:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.489+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/3029820718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571228b0680 0x5571228c19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.489+0000 7fd8a8acab80 1 --2- 172.21.15.78:0/3029820718 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571228b0fa0 0x5571228bfda0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.489+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3029820718 >> 172.21.15.78:0/3029820718 conn(0x557122826220 msgr2=0x5571228b70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.489+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3029820718 shutdown_connections 2024-03-14T09:55:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.489+0000 7fd8a8acab80 1 -- 172.21.15.78:0/3029820718 wait complete. 2024-03-14T09:55:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.489+0000 7fd8a8acab80 20 client.26053 trim_cache size 0 max 16384 2024-03-14T09:55:43.493 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:55:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "scrub active (813 inodes in the stack)", 2024-03-14T09:55:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T09:55:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "a8535772-78a2-4891-9785-461fe2d1675c": { 2024-03-14T09:55:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T09:55:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "a8535772-78a2-4891-9785-461fe2d1675c", 2024-03-14T09:55:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T09:55:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:55:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:55:43.493 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:55:43.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.490+0000 7fd895000700 1 -- 172.21.15.78:0/2394855022 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd8740a0c80 msgr2=0x7fd8740a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.490+0000 7fd895000700 1 --2- 172.21.15.78:0/2394855022 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd8740a0c80 0x7fd8740a3140 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd8880060b0 tx=0x7fd888009700 comp rx=0 tx=0).stop 2024-03-14T09:55:43.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.490+0000 7fd895000700 1 -- 172.21.15.78:0/2394855022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd890103140 msgr2=0x7fd89019f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:55:43.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.490+0000 7fd895000700 1 --2- 172.21.15.78:0/2394855022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd890103140 0x7fd89019f660 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd88000e9c0 tx=0x7fd88000ecd0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.490+0000 7fd895000700 1 -- 172.21.15.78:0/2394855022 shutdown_connections 2024-03-14T09:55:43.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.490+0000 7fd895000700 1 --2- 172.21.15.78:0/2394855022 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd8740a0c80 0x7fd8740a3140 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.490+0000 7fd895000700 1 --2- 172.21.15.78:0/2394855022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd890103140 0x7fd89019f660 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.490+0000 7fd895000700 1 --2- 172.21.15.78:0/2394855022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd890199790 0x7fd890199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.490+0000 7fd895000700 1 --2- 172.21.15.78:0/2394855022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd890108260 0x7fd89019fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:55:43.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.490+0000 7fd895000700 1 -- 172.21.15.78:0/2394855022 >> 172.21.15.78:0/2394855022 conn(0x7fd8900fe980 msgr2=0x7fd890100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:55:43.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.490+0000 7fd895000700 1 -- 172.21.15.78:0/2394855022 shutdown_connections 2024-03-14T09:55:43.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:55:43.490+0000 7fd895000700 1 -- 172.21.15.78:0/2394855022 wait complete. 2024-03-14T09:55:43.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:43 smithi152 ceph-mon[48279]: pgmap v49: 129 pgs: 129 active+clean; 49 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 54 op/s 2024-03-14T09:55:43.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:43 smithi152 ceph-mon[48279]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-03-14T09:55:43.513 INFO:tasks.cephfs.filesystem:scrub status for tag:a8535772-78a2-4891-9785-461fe2d1675c - {'path': '/', 'tag': 'a8535772-78a2-4891-9785-461fe2d1675c', 'options': 'recursive,force'} 2024-03-14T09:55:43.519 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:43 smithi132 ceph-mon[47959]: pgmap v49: 129 pgs: 129 active+clean; 49 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.5 MiB/s wr, 54 op/s 2024-03-14T09:55:43.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:43 smithi132 ceph-mon[47959]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-03-14T09:55:43.530 INFO:tasks.workunit.client.0.smithi078.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-14T09:55:43.538 INFO:tasks.workunit.client.0.smithi078.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2024-03-14T09:55:43.545 INFO:tasks.workunit.client.0.smithi078.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-14T09:55:43.548 INFO:tasks.workunit.client.0.smithi078.stdout:checking for egrep... /usr/bin/grep -E 2024-03-14T09:55:43.550 INFO:tasks.workunit.client.0.smithi078.stdout:checking for fgrep... /usr/bin/grep -F 2024-03-14T09:55:43.561 INFO:tasks.workunit.client.0.smithi078.stdout:checking for ld used by gcc... /usr/bin/ld 2024-03-14T09:55:43.568 INFO:tasks.workunit.client.0.smithi078.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2024-03-14T09:55:43.583 INFO:tasks.workunit.client.0.smithi078.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2024-03-14T09:55:44.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:44 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T09:55:44.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:44 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:55:44.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:44 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T09:55:44.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:44 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T09:55:44.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:44 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:55:44.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:44 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T09:55:44.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:44 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T09:55:44.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:44 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:55:44.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:44 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T09:55:44.818 INFO:tasks.workunit.client.0.smithi078.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2024-03-14T09:55:44.818 INFO:tasks.workunit.client.0.smithi078.stdout:checking whether ln -s works... yes 2024-03-14T09:55:44.851 INFO:tasks.workunit.client.0.smithi078.stdout:checking the maximum length of command line arguments... 1572864 2024-03-14T09:55:44.857 INFO:tasks.workunit.client.0.smithi078.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2024-03-14T09:55:44.857 INFO:tasks.workunit.client.0.smithi078.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2024-03-14T09:55:44.858 INFO:tasks.workunit.client.0.smithi078.stdout:checking for /usr/bin/ld option to reload object files... -r 2024-03-14T09:55:44.858 INFO:tasks.workunit.client.0.smithi078.stdout:checking for objdump... objdump 2024-03-14T09:55:44.858 INFO:tasks.workunit.client.0.smithi078.stdout:checking how to recognize dependent libraries... pass_all 2024-03-14T09:55:44.859 INFO:tasks.workunit.client.0.smithi078.stdout:checking for dlltool... no 2024-03-14T09:55:44.859 INFO:tasks.workunit.client.0.smithi078.stdout:checking how to associate runtime and link libraries... printf %s\n 2024-03-14T09:55:44.859 INFO:tasks.workunit.client.0.smithi078.stdout:checking for ar... ar 2024-03-14T09:55:45.085 INFO:tasks.workunit.client.0.smithi078.stdout:checking for archiver @FILE support... @ 2024-03-14T09:55:45.086 INFO:tasks.workunit.client.0.smithi078.stdout:checking for strip... strip 2024-03-14T09:55:45.086 INFO:tasks.workunit.client.0.smithi078.stdout:checking for ranlib... ranlib 2024-03-14T09:55:45.087 INFO:tasks.workunit.client.0.smithi078.stdout:checking for gawk... gawk 2024-03-14T09:55:45.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:45 smithi078 ceph-mon[38143]: pgmap v50: 129 pgs: 129 active+clean; 49 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 35 op/s 2024-03-14T09:55:45.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:45 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:55:45.323 INFO:tasks.workunit.client.0.smithi078.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2024-03-14T09:55:45.327 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sysroot... no 2024-03-14T09:55:45.349 INFO:tasks.workunit.client.0.smithi078.stdout:checking for a working dd... /usr/bin/dd 2024-03-14T09:55:45.366 INFO:tasks.workunit.client.0.smithi078.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2024-03-14T09:55:45.449 INFO:tasks.workunit.client.0.smithi078.stdout:checking for mt... no 2024-03-14T09:55:45.478 INFO:tasks.workunit.client.0.smithi078.stdout:checking if : is a manifest tool... no 2024-03-14T09:55:45.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:45 smithi152 ceph-mon[48279]: pgmap v50: 129 pgs: 129 active+clean; 49 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 35 op/s 2024-03-14T09:55:45.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:45 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:55:45.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:45 smithi132 ceph-mon[47959]: pgmap v50: 129 pgs: 129 active+clean; 49 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 35 op/s 2024-03-14T09:55:45.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:45 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:55:45.581 INFO:tasks.workunit.client.0.smithi078.stdout:checking how to run the C preprocessor... gcc -E 2024-03-14T09:55:45.844 INFO:tasks.workunit.client.0.smithi078.stdout:checking for ANSI C header files... yes 2024-03-14T09:55:45.901 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/types.h... yes 2024-03-14T09:55:46.025 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/stat.h... yes 2024-03-14T09:55:46.130 INFO:tasks.workunit.client.0.smithi078.stdout:checking for stdlib.h... yes 2024-03-14T09:55:46.195 INFO:tasks.workunit.client.0.smithi078.stdout:checking for string.h... yes 2024-03-14T09:55:46.261 INFO:tasks.workunit.client.0.smithi078.stdout:checking for memory.h... yes 2024-03-14T09:55:46.326 INFO:tasks.workunit.client.0.smithi078.stdout:checking for strings.h... yes 2024-03-14T09:55:46.391 INFO:tasks.workunit.client.0.smithi078.stdout:checking for inttypes.h... yes 2024-03-14T09:55:46.456 INFO:tasks.workunit.client.0.smithi078.stdout:checking for stdint.h... yes 2024-03-14T09:55:46.525 INFO:tasks.workunit.client.0.smithi078.stdout:checking for unistd.h... yes 2024-03-14T09:55:46.629 INFO:tasks.workunit.client.0.smithi078.stdout:checking for dlfcn.h... yes 2024-03-14T09:55:46.640 INFO:tasks.workunit.client.0.smithi078.stdout:checking for objdir... .libs 2024-03-14T09:55:46.816 INFO:tasks.workunit.client.0.smithi078.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2024-03-14T09:55:46.816 INFO:tasks.workunit.client.0.smithi078.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2024-03-14T09:55:46.870 INFO:tasks.workunit.client.0.smithi078.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2024-03-14T09:55:46.949 INFO:tasks.workunit.client.0.smithi078.stdout:checking if gcc static flag -static works... no 2024-03-14T09:55:47.033 INFO:tasks.workunit.client.0.smithi078.stdout:checking if gcc supports -c -o file.o... yes 2024-03-14T09:55:47.034 INFO:tasks.workunit.client.0.smithi078.stdout:checking if gcc supports -c -o file.o... (cached) yes 2024-03-14T09:55:47.048 INFO:tasks.workunit.client.0.smithi078.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2024-03-14T09:55:47.111 INFO:tasks.workunit.client.0.smithi078.stdout:checking whether -lc should be explicitly linked in... no 2024-03-14T09:55:47.213 INFO:tasks.workunit.client.0.smithi078.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2024-03-14T09:55:47.213 INFO:tasks.workunit.client.0.smithi078.stdout:checking how to hardcode library paths into programs... immediate 2024-03-14T09:55:47.217 INFO:tasks.workunit.client.0.smithi078.stdout:checking whether stripping libraries is possible... yes 2024-03-14T09:55:47.217 INFO:tasks.workunit.client.0.smithi078.stdout:checking if libtool supports shared libraries... yes 2024-03-14T09:55:47.218 INFO:tasks.workunit.client.0.smithi078.stdout:checking whether to build shared libraries... yes 2024-03-14T09:55:47.218 INFO:tasks.workunit.client.0.smithi078.stdout:checking whether to build static libraries... yes 2024-03-14T09:55:47.229 INFO:tasks.workunit.client.0.smithi078.stdout:checking for gcc... (cached) gcc 2024-03-14T09:55:47.277 INFO:tasks.workunit.client.0.smithi078.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-14T09:55:47.277 INFO:tasks.workunit.client.0.smithi078.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-14T09:55:47.277 INFO:tasks.workunit.client.0.smithi078.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-14T09:55:47.277 INFO:tasks.workunit.client.0.smithi078.stdout:checking for gmake... /usr/bin/gmake 2024-03-14T09:55:47.277 INFO:tasks.workunit.client.0.smithi078.stderr:./configure: line 11956: test: $(SHELL): binary operator expected 2024-03-14T09:55:47.278 INFO:tasks.workunit.client.0.smithi078.stdout:checking for tar... /bin/tar 2024-03-14T09:55:47.278 INFO:tasks.workunit.client.0.smithi078.stdout:checking for gzip... /bin/gzip 2024-03-14T09:55:47.295 INFO:tasks.workunit.client.0.smithi078.stdout:checking whether gcc -MM is supported... yes 2024-03-14T09:55:47.296 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sort... /bin/sort 2024-03-14T09:55:47.296 INFO:tasks.workunit.client.0.smithi078.stdout:checking whether ln -s works... yes 2024-03-14T09:55:47.297 INFO:tasks.workunit.client.0.smithi078.stdout:checking for rpm... /bin/rpm 2024-03-14T09:55:47.305 INFO:tasks.workunit.client.0.smithi078.stdout:checking for rpmbuild... no 2024-03-14T09:55:47.305 INFO:tasks.workunit.client.0.smithi078.stdout:checking for ANSI C header files... (cached) yes 2024-03-14T09:55:47.378 INFO:tasks.workunit.client.0.smithi078.stdout:checking assert.h usability... yes 2024-03-14T09:55:47.398 INFO:tasks.workunit.client.0.smithi078.stdout:checking assert.h presence... yes 2024-03-14T09:55:47.398 INFO:tasks.workunit.client.0.smithi078.stdout:checking for assert.h... yes 2024-03-14T09:55:47.475 INFO:tasks.workunit.client.0.smithi078.stdout:checking bstring.h usability... no 2024-03-14T09:55:47.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:47 smithi152 ceph-mon[48279]: pgmap v51: 129 pgs: 129 active+clean; 57 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 274 op/s 2024-03-14T09:55:47.510 INFO:tasks.workunit.client.0.smithi078.stdout:checking bstring.h presence... no 2024-03-14T09:55:47.510 INFO:tasks.workunit.client.0.smithi078.stdout:checking for bstring.h... no 2024-03-14T09:55:47.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:47 smithi132 ceph-mon[47959]: pgmap v51: 129 pgs: 129 active+clean; 57 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 274 op/s 2024-03-14T09:55:47.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:47 smithi078 ceph-mon[38143]: pgmap v51: 129 pgs: 129 active+clean; 57 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 274 op/s 2024-03-14T09:55:47.571 INFO:tasks.workunit.client.0.smithi078.stdout:checking libgen.h usability... yes 2024-03-14T09:55:47.592 INFO:tasks.workunit.client.0.smithi078.stdout:checking libgen.h presence... yes 2024-03-14T09:55:47.592 INFO:tasks.workunit.client.0.smithi078.stdout:checking for libgen.h... yes 2024-03-14T09:55:47.679 INFO:tasks.workunit.client.0.smithi078.stdout:checking dirent.h usability... yes 2024-03-14T09:55:47.701 INFO:tasks.workunit.client.0.smithi078.stdout:checking dirent.h presence... yes 2024-03-14T09:55:47.701 INFO:tasks.workunit.client.0.smithi078.stdout:checking for dirent.h... yes 2024-03-14T09:55:47.775 INFO:tasks.workunit.client.0.smithi078.stdout:checking errno.h usability... yes 2024-03-14T09:55:47.796 INFO:tasks.workunit.client.0.smithi078.stdout:checking errno.h presence... yes 2024-03-14T09:55:47.796 INFO:tasks.workunit.client.0.smithi078.stdout:checking for errno.h... yes 2024-03-14T09:55:47.872 INFO:tasks.workunit.client.0.smithi078.stdout:checking malloc.h usability... yes 2024-03-14T09:55:47.894 INFO:tasks.workunit.client.0.smithi078.stdout:checking malloc.h presence... yes 2024-03-14T09:55:47.894 INFO:tasks.workunit.client.0.smithi078.stdout:checking for malloc.h... yes 2024-03-14T09:55:47.968 INFO:tasks.workunit.client.0.smithi078.stdout:checking uuid.h usability... no 2024-03-14T09:55:48.003 INFO:tasks.workunit.client.0.smithi078.stdout:checking uuid.h presence... no 2024-03-14T09:55:48.003 INFO:tasks.workunit.client.0.smithi078.stdout:checking for uuid.h... no 2024-03-14T09:55:48.073 INFO:tasks.workunit.client.0.smithi078.stdout:checking uuid/uuid.h usability... yes 2024-03-14T09:55:48.096 INFO:tasks.workunit.client.0.smithi078.stdout:checking uuid/uuid.h presence... yes 2024-03-14T09:55:48.096 INFO:tasks.workunit.client.0.smithi078.stdout:checking for uuid/uuid.h... yes 2024-03-14T09:55:48.170 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/uuid.h usability... no 2024-03-14T09:55:48.205 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/uuid.h presence... no 2024-03-14T09:55:48.205 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/uuid.h... no 2024-03-14T09:55:48.274 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/file.h usability... yes 2024-03-14T09:55:48.296 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/file.h presence... yes 2024-03-14T09:55:48.296 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/file.h... yes 2024-03-14T09:55:48.368 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/fcntl.h usability... yes 2024-03-14T09:55:48.389 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/fcntl.h presence... yes 2024-03-14T09:55:48.389 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/fcntl.h... yes 2024-03-14T09:55:48.463 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/syssgi.h usability... no 2024-03-14T09:55:48.499 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/syssgi.h presence... no 2024-03-14T09:55:48.499 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/syssgi.h... no 2024-03-14T09:55:48.601 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/param.h usability... yes 2024-03-14T09:55:48.626 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/param.h presence... yes 2024-03-14T09:55:48.627 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/param.h... yes 2024-03-14T09:55:48.652 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/stat.h... (cached) yes 2024-03-14T09:55:48.716 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/statvfs.h usability... yes 2024-03-14T09:55:48.736 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/statvfs.h presence... yes 2024-03-14T09:55:48.736 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/statvfs.h... yes 2024-03-14T09:55:48.813 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/time.h usability... yes 2024-03-14T09:55:48.835 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/time.h presence... yes 2024-03-14T09:55:48.835 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/time.h... yes 2024-03-14T09:55:48.917 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/ioctl.h usability... yes 2024-03-14T09:55:48.940 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/ioctl.h presence... yes 2024-03-14T09:55:48.940 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/ioctl.h... yes 2024-03-14T09:55:49.017 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/wait.h usability... yes 2024-03-14T09:55:49.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:48 smithi132 ceph-mon[47959]: pgmap v52: 129 pgs: 129 active+clean; 57 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 253 op/s 2024-03-14T09:55:49.041 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/wait.h presence... yes 2024-03-14T09:55:49.041 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/wait.h... yes 2024-03-14T09:55:49.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:48 smithi078 ceph-mon[38143]: pgmap v52: 129 pgs: 129 active+clean; 57 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 253 op/s 2024-03-14T09:55:49.064 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/types.h... (cached) yes 2024-03-14T09:55:49.077 INFO:tasks.workunit.client.0.smithi078.stdout:checking for strings.h... (cached) yes 2024-03-14T09:55:49.149 INFO:tasks.workunit.client.0.smithi078.stdout:checking err.h usability... yes 2024-03-14T09:55:49.169 INFO:tasks.workunit.client.0.smithi078.stdout:checking err.h presence... yes 2024-03-14T09:55:49.169 INFO:tasks.workunit.client.0.smithi078.stdout:checking for err.h... yes 2024-03-14T09:55:49.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:48 smithi152 ceph-mon[48279]: pgmap v52: 129 pgs: 129 active+clean; 57 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 253 op/s 2024-03-14T09:55:50.051 INFO:tasks.workunit.client.0.smithi078.stdout:checking linux/falloc.h usability... yes 2024-03-14T09:55:50.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:49 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T09:55:50.248 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:49 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T09:55:50.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:49 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T09:55:51.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:50 smithi078 ceph-mon[38143]: pgmap v53: 129 pgs: 129 active+clean; 57 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 253 op/s 2024-03-14T09:55:51.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:50 smithi152 ceph-mon[48279]: pgmap v53: 129 pgs: 129 active+clean; 57 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 253 op/s 2024-03-14T09:55:51.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:50 smithi132 ceph-mon[47959]: pgmap v53: 129 pgs: 129 active+clean; 57 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 253 op/s 2024-03-14T09:55:52.230 INFO:tasks.workunit.client.0.smithi078.stdout:checking linux/falloc.h presence... yes 2024-03-14T09:55:52.231 INFO:tasks.workunit.client.0.smithi078.stdout:checking for linux/falloc.h... yes 2024-03-14T09:55:52.800 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/fs/xfs_fsops.h usability... no 2024-03-14T09:55:52.835 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/fs/xfs_fsops.h presence... no 2024-03-14T09:55:52.836 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/fs/xfs_fsops.h... no 2024-03-14T09:55:52.905 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/fs/xfs_itable.h usability... no 2024-03-14T09:55:52.940 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/fs/xfs_itable.h presence... no 2024-03-14T09:55:52.940 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/fs/xfs_itable.h... no 2024-03-14T09:55:53.098 INFO:tasks.workunit.client.0.smithi078.stdout:checking xfs/platform_defs.h usability... no 2024-03-14T09:55:53.151 INFO:tasks.workunit.client.0.smithi078.stdout:checking xfs/platform_defs.h presence... no 2024-03-14T09:55:53.151 INFO:tasks.workunit.client.0.smithi078.stdout:checking for xfs/platform_defs.h... no 2024-03-14T09:55:53.254 INFO:tasks.workunit.client.0.smithi078.stdout:checking btrfs/ioctl.h usability... no 2024-03-14T09:55:53.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:53 smithi078 ceph-mon[38143]: pgmap v54: 129 pgs: 129 active+clean; 76 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 648 op/s 2024-03-14T09:55:53.314 INFO:tasks.workunit.client.0.smithi078.stdout:checking btrfs/ioctl.h presence... no 2024-03-14T09:55:53.314 INFO:tasks.workunit.client.0.smithi078.stdout:checking for btrfs/ioctl.h... no 2024-03-14T09:55:53.400 INFO:tasks.workunit.client.0.smithi078.stdout:checking cifs/ioctl.h usability... no 2024-03-14T09:55:53.437 INFO:tasks.workunit.client.0.smithi078.stdout:checking cifs/ioctl.h presence... no 2024-03-14T09:55:53.437 INFO:tasks.workunit.client.0.smithi078.stdout:checking for cifs/ioctl.h... no 2024-03-14T09:55:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:53 smithi152 ceph-mon[48279]: pgmap v54: 129 pgs: 129 active+clean; 76 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 648 op/s 2024-03-14T09:55:53.513 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/mman.h usability... yes 2024-03-14T09:55:53.519 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:53 smithi132 ceph-mon[47959]: pgmap v54: 129 pgs: 129 active+clean; 76 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 648 op/s 2024-03-14T09:55:53.535 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/mman.h presence... yes 2024-03-14T09:55:53.535 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/mman.h... yes 2024-03-14T09:55:53.593 INFO:tasks.workunit.client.0.smithi078.stdout:checking for xfs/xfs_log_format.h... no 2024-03-14T09:55:53.674 INFO:tasks.workunit.client.0.smithi078.stdout:checking for uuid_compare... no 2024-03-14T09:55:53.736 INFO:tasks.workunit.client.0.smithi078.stdout:checking for uuid_compare in -luuid... yes 2024-03-14T09:55:53.825 INFO:tasks.workunit.client.0.smithi078.stdout:checking for xfs/xfs.h... yes 2024-03-14T09:55:53.867 INFO:tasks.workunit.client.0.smithi078.stdout:checking for xfs/libxfs.h... no 2024-03-14T09:55:53.915 INFO:tasks.workunit.client.0.smithi078.stdout:checking whether xlog_assign_lsn is declared... no 2024-03-14T09:55:53.972 INFO:tasks.workunit.client.0.smithi078.stdout:checking for xfs/xqm.h... yes 2024-03-14T09:55:54.053 INFO:tasks.workunit.client.0.smithi078.stdout:checking xfsctl from xfs/xfs.h... ok 2024-03-14T09:55:54.107 INFO:tasks.workunit.client.0.smithi078.stdout:checking xfs/handle.h usability... yes 2024-03-14T09:55:54.127 INFO:tasks.workunit.client.0.smithi078.stdout:checking xfs/handle.h presence... yes 2024-03-14T09:55:54.127 INFO:tasks.workunit.client.0.smithi078.stdout:checking for xfs/handle.h... yes 2024-03-14T09:55:54.192 INFO:tasks.workunit.client.0.smithi078.stdout:checking for attr_list_by_handle in -lhandle... yes 2024-03-14T09:55:54.276 INFO:tasks.workunit.client.0.smithi078.stdout:checking attr/attributes.h usability... yes 2024-03-14T09:55:54.297 INFO:tasks.workunit.client.0.smithi078.stdout:checking attr/attributes.h presence... yes 2024-03-14T09:55:54.297 INFO:tasks.workunit.client.0.smithi078.stdout:checking for attr/attributes.h... yes 2024-03-14T09:55:54.373 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/attributes.h usability... no 2024-03-14T09:55:54.408 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/attributes.h presence... no 2024-03-14T09:55:54.408 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/attributes.h... no 2024-03-14T09:55:54.470 INFO:tasks.workunit.client.0.smithi078.stdout:checking for attr_list in -lattr... yes 2024-03-14T09:55:54.534 INFO:tasks.workunit.client.0.smithi078.stdout:checking for attr_set in -lattr... yes 2024-03-14T09:55:54.588 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/acl.h usability... yes 2024-03-14T09:55:54.611 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/acl.h presence... yes 2024-03-14T09:55:54.611 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/acl.h... yes 2024-03-14T09:55:54.673 INFO:tasks.workunit.client.0.smithi078.stdout:checking acl/libacl.h usability... yes 2024-03-14T09:55:54.695 INFO:tasks.workunit.client.0.smithi078.stdout:checking acl/libacl.h presence... yes 2024-03-14T09:55:54.695 INFO:tasks.workunit.client.0.smithi078.stdout:checking for acl/libacl.h... yes 2024-03-14T09:55:54.761 INFO:tasks.workunit.client.0.smithi078.stdout:checking for acl_init in -lacl... yes 2024-03-14T09:55:54.834 INFO:tasks.workunit.client.0.smithi078.stdout:checking gdbm-ndbm.h usability... no 2024-03-14T09:55:54.869 INFO:tasks.workunit.client.0.smithi078.stdout:checking gdbm-ndbm.h presence... no 2024-03-14T09:55:54.869 INFO:tasks.workunit.client.0.smithi078.stdout:checking for gdbm-ndbm.h... no 2024-03-14T09:55:54.922 INFO:tasks.workunit.client.0.smithi078.stdout:checking gdbm/ndbm.h usability... no 2024-03-14T09:55:54.957 INFO:tasks.workunit.client.0.smithi078.stdout:checking gdbm/ndbm.h presence... no 2024-03-14T09:55:54.957 INFO:tasks.workunit.client.0.smithi078.stdout:checking for gdbm/ndbm.h... no 2024-03-14T09:55:55.014 INFO:tasks.workunit.client.0.smithi078.stdout:checking ndbm.h usability... no 2024-03-14T09:55:55.051 INFO:tasks.workunit.client.0.smithi078.stdout:checking ndbm.h presence... no 2024-03-14T09:55:55.051 INFO:tasks.workunit.client.0.smithi078.stdout:checking for ndbm.h... no 2024-03-14T09:55:55.102 INFO:tasks.workunit.client.0.smithi078.stdout:checking libaio.h usability... yes 2024-03-14T09:55:55.127 INFO:tasks.workunit.client.0.smithi078.stdout:checking libaio.h presence... yes 2024-03-14T09:55:55.127 INFO:tasks.workunit.client.0.smithi078.stdout:checking for libaio.h... yes 2024-03-14T09:55:55.203 INFO:tasks.workunit.client.0.smithi078.stdout:checking for dm_make_handle in -ldm... no 2024-03-14T09:55:55.203 INFO:tasks.workunit.client.0.smithi078.stdout: 2024-03-14T09:55:55.203 INFO:tasks.workunit.client.0.smithi078.stdout:WARNING: could not find a valid DMAPI base library. 2024-03-14T09:55:55.203 INFO:tasks.workunit.client.0.smithi078.stdout:If you want DMAPI support please install the data migration 2024-03-14T09:55:55.203 INFO:tasks.workunit.client.0.smithi078.stdout:API (dmapi) library package. Alternatively, run "make install" 2024-03-14T09:55:55.203 INFO:tasks.workunit.client.0.smithi078.stdout:from the dmapi source. 2024-03-14T09:55:55.204 INFO:tasks.workunit.client.0.smithi078.stdout: 2024-03-14T09:55:55.251 INFO:tasks.workunit.client.0.smithi078.stdout:checking linux/fiemap.h usability... yes 2024-03-14T09:55:55.271 INFO:tasks.workunit.client.0.smithi078.stdout:checking linux/fiemap.h presence... yes 2024-03-14T09:55:55.271 INFO:tasks.workunit.client.0.smithi078.stdout:checking for linux/fiemap.h... yes 2024-03-14T09:55:55.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:55 smithi078 ceph-mon[38143]: pgmap v55: 129 pgs: 129 active+clean; 76 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 2.9 MiB/s wr, 633 op/s 2024-03-14T09:55:55.333 INFO:tasks.workunit.client.0.smithi078.stdout:checking for fallocate... yes 2024-03-14T09:55:55.399 INFO:tasks.workunit.client.0.smithi078.stdout:checking for open_by_handle_at... yes 2024-03-14T09:55:55.464 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/prctl.h usability... yes 2024-03-14T09:55:55.485 INFO:tasks.workunit.client.0.smithi078.stdout:checking sys/prctl.h presence... yes 2024-03-14T09:55:55.485 INFO:tasks.workunit.client.0.smithi078.stdout:checking for sys/prctl.h... yes 2024-03-14T09:55:55.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:55 smithi152 ceph-mon[48279]: pgmap v55: 129 pgs: 129 active+clean; 76 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 2.9 MiB/s wr, 633 op/s 2024-03-14T09:55:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:55 smithi132 ceph-mon[47959]: pgmap v55: 129 pgs: 129 active+clean; 76 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 2.9 MiB/s wr, 633 op/s 2024-03-14T09:55:55.536 INFO:tasks.workunit.client.0.smithi078.stdout:checking linux/fs.h usability... yes 2024-03-14T09:55:55.557 INFO:tasks.workunit.client.0.smithi078.stdout:checking linux/fs.h presence... yes 2024-03-14T09:55:55.557 INFO:tasks.workunit.client.0.smithi078.stdout:checking for linux/fs.h... yes 2024-03-14T09:55:55.631 INFO:tasks.workunit.client.0.smithi078.stdout:checking btrfsutil.h usability... no 2024-03-14T09:55:55.665 INFO:tasks.workunit.client.0.smithi078.stdout:checking btrfsutil.h presence... no 2024-03-14T09:55:55.665 INFO:tasks.workunit.client.0.smithi078.stdout:checking for btrfsutil.h... no 2024-03-14T09:55:55.739 INFO:tasks.workunit.client.0.smithi078.stdout:checking for copy_file_range... yes 2024-03-14T09:55:55.808 INFO:tasks.workunit.client.0.smithi078.stdout:checking for renameat2... yes 2024-03-14T09:55:55.840 INFO:tasks.workunit.client.0.smithi078.stdout:configure: creating ./config.status 2024-03-14T09:55:56.735 INFO:tasks.workunit.client.0.smithi078.stdout:config.status: creating include/builddefs 2024-03-14T09:55:56.767 INFO:tasks.workunit.client.0.smithi078.stdout:config.status: creating include/config.h 2024-03-14T09:55:56.792 INFO:tasks.workunit.client.0.smithi078.stdout:config.status: executing libtool commands 2024-03-14T09:55:56.907 INFO:tasks.workunit.client.0.smithi078.stdout:Building include 2024-03-14T09:55:56.908 INFO:tasks.workunit.client.0.smithi078.stdout:Building lib 2024-03-14T09:55:56.908 INFO:tasks.workunit.client.0.smithi078.stdout:Building tests 2024-03-14T09:55:56.917 INFO:tasks.workunit.client.0.smithi078.stdout: [LTDEP] 2024-03-14T09:55:57.157 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] dataascii.lo 2024-03-14T09:55:57.157 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] databin.lo 2024-03-14T09:55:57.157 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] datapid.lo 2024-03-14T09:55:57.157 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] file_lock.lo 2024-03-14T09:55:57.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:57 smithi078 ceph-mon[38143]: pgmap v56: 129 pgs: 129 active+clean; 79 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 775 op/s 2024-03-14T09:55:57.388 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] forker.lo 2024-03-14T09:55:57.394 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] pattern.lo 2024-03-14T09:55:57.437 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] open_flags.lo 2024-03-14T09:55:57.445 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] random_range.lo 2024-03-14T09:55:57.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:57 smithi152 ceph-mon[48279]: pgmap v56: 129 pgs: 129 active+clean; 79 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 775 op/s 2024-03-14T09:55:57.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:57 smithi132 ceph-mon[47959]: pgmap v56: 129 pgs: 129 active+clean; 79 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.4 MiB/s wr, 775 op/s 2024-03-14T09:55:57.634 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] string_to_tokens.lo 2024-03-14T09:55:57.656 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] str_to_bytes.lo 2024-03-14T09:55:57.720 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] tlibio.lo 2024-03-14T09:55:57.834 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] write_log.lo 2024-03-14T09:55:57.860 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] random.lo 2024-03-14T09:55:58.192 INFO:tasks.workunit.client.0.smithi078.stdout: [LD] 2024-03-14T09:55:58.353 INFO:tasks.workunit.client.0.smithi078.stdout:Building ltp 2024-03-14T09:55:58.353 INFO:tasks.workunit.client.0.smithi078.stdout:Building src 2024-03-14T09:55:58.353 INFO:tasks.workunit.client.0.smithi078.stdout:Building m4 2024-03-14T09:55:58.353 INFO:tasks.workunit.client.0.smithi078.stdout:Building common 2024-03-14T09:55:58.367 INFO:tasks.workunit.client.0.smithi078.stdout: [DEP] 2024-03-14T09:55:58.379 INFO:tasks.workunit.client.0.smithi078.stdout: [DEP] 2024-03-14T09:55:58.642 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] doio 2024-03-14T09:55:58.642 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] fsstress 2024-03-14T09:55:58.642 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] fsx 2024-03-14T09:55:58.932 INFO:tasks.workunit.client.0.smithi078.stderr:fsstress.c: In function 'attr_list_path': 2024-03-14T09:55:58.932 INFO:tasks.workunit.client.0.smithi078.stderr:fsstress.c:859:2: warning: 'attr_list' is deprecated: Use listxattr or llistxattr instead [-Wdeprecated-declarations] 2024-03-14T09:55:58.932 INFO:tasks.workunit.client.0.smithi078.stderr: rval = attr_list(name->path, buffer, buffersize, flags, cursor); 2024-03-14T09:55:58.932 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~ 2024-03-14T09:55:58.932 INFO:tasks.workunit.client.0.smithi078.stderr:In file included from ../src/global.h:25, 2024-03-14T09:55:58.932 INFO:tasks.workunit.client.0.smithi078.stderr: from fsstress.c:12: 2024-03-14T09:55:58.932 INFO:tasks.workunit.client.0.smithi078.stderr:/usr/include/attr/attributes.h:165:12: note: declared here 2024-03-14T09:55:58.932 INFO:tasks.workunit.client.0.smithi078.stderr: extern int attr_list(const char *__path, char *__buffer, const int __buffersize, 2024-03-14T09:55:58.932 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~~~~~~ 2024-03-14T09:55:58.933 INFO:tasks.workunit.client.0.smithi078.stderr:fsstress.c: In function 'attr_remove_path': 2024-03-14T09:55:58.933 INFO:tasks.workunit.client.0.smithi078.stderr:fsstress.c:878:2: warning: 'attr_remove' is deprecated: Use removexattr or lremovexattr instead [-Wdeprecated-declarations] 2024-03-14T09:55:58.933 INFO:tasks.workunit.client.0.smithi078.stderr: rval = attr_remove(name->path, attrname, flags); 2024-03-14T09:55:58.933 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~ 2024-03-14T09:55:58.933 INFO:tasks.workunit.client.0.smithi078.stderr:In file included from ../src/global.h:25, 2024-03-14T09:55:58.933 INFO:tasks.workunit.client.0.smithi078.stderr: from fsstress.c:12: 2024-03-14T09:55:58.933 INFO:tasks.workunit.client.0.smithi078.stderr:/usr/include/attr/attributes.h:152:12: note: declared here 2024-03-14T09:55:58.933 INFO:tasks.workunit.client.0.smithi078.stderr: extern int attr_remove (const char *__path, const char *__attrname, 2024-03-14T09:55:58.933 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~~~~~~~~ 2024-03-14T09:55:58.933 INFO:tasks.workunit.client.0.smithi078.stderr:fsstress.c: In function 'attr_set_path': 2024-03-14T09:55:58.933 INFO:tasks.workunit.client.0.smithi078.stderr:fsstress.c:898:2: warning: 'attr_set' is deprecated: Use setxattr or lsetxattr instead [-Wdeprecated-declarations] 2024-03-14T09:55:58.933 INFO:tasks.workunit.client.0.smithi078.stderr: rval = attr_set(name->path, attrname, attrvalue, valuelength, flags); 2024-03-14T09:55:58.933 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~ 2024-03-14T09:55:58.933 INFO:tasks.workunit.client.0.smithi078.stderr:In file included from ../src/global.h:25, 2024-03-14T09:55:58.933 INFO:tasks.workunit.client.0.smithi078.stderr: from fsstress.c:12: 2024-03-14T09:55:58.933 INFO:tasks.workunit.client.0.smithi078.stderr:/usr/include/attr/attributes.h:139:12: note: declared here 2024-03-14T09:55:58.934 INFO:tasks.workunit.client.0.smithi078.stderr: extern int attr_set (const char *__path, const char *__attrname, 2024-03-14T09:55:58.934 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~~~~~ 2024-03-14T09:55:59.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:55:58 smithi132 ceph-mon[47959]: pgmap v57: 129 pgs: 129 active+clean; 79 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 536 op/s 2024-03-14T09:55:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:55:58 smithi078 ceph-mon[38143]: pgmap v57: 129 pgs: 129 active+clean; 79 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 536 op/s 2024-03-14T09:55:59.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:55:58 smithi152 ceph-mon[48279]: pgmap v57: 129 pgs: 129 active+clean; 79 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 536 op/s 2024-03-14T09:55:59.484 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] growfiles 2024-03-14T09:55:59.502 INFO:tasks.workunit.client.0.smithi078.stderr:fsx.c: In function 'main': 2024-03-14T09:55:59.502 INFO:tasks.workunit.client.0.smithi078.stderr:fsx.c:2662:5: warning: 'strncpy' specified bound 4096 equals destination size [-Wstringop-truncation] 2024-03-14T09:55:59.502 INFO:tasks.workunit.client.0.smithi078.stderr: strncpy(opsfile, optarg, sizeof(opsfile)); 2024-03-14T09:55:59.502 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-14T09:55:59.502 INFO:tasks.workunit.client.0.smithi078.stderr:fsx.c:2632:4: warning: 'strncpy' specified bound 1024 equals destination size [-Wstringop-truncation] 2024-03-14T09:55:59.502 INFO:tasks.workunit.client.0.smithi078.stderr: strncpy(dname, optarg, sizeof(dname)); 2024-03-14T09:55:59.502 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-14T09:55:59.641 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] iogen 2024-03-14T09:56:00.166 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aio-stress 2024-03-14T09:56:00.761 INFO:tasks.workunit.client.0.smithi078.stdout: [DEP] 2024-03-14T09:56:00.762 INFO:tasks.workunit.client.0.smithi078.stdout: [DEP] 2024-03-14T09:56:00.762 INFO:tasks.workunit.client.0.smithi078.stdout: [DEP] 2024-03-14T09:56:01.140 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] dirstress 2024-03-14T09:56:01.144 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] fill 2024-03-14T09:56:01.144 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] fill2 2024-03-14T09:56:01.146 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] getpagesize 2024-03-14T09:56:01.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:01 smithi078 ceph-mon[38143]: pgmap v58: 129 pgs: 129 active+clean; 79 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 536 op/s 2024-03-14T09:56:01.443 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] holes 2024-03-14T09:56:01.451 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] lstat64 2024-03-14T09:56:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:01 smithi152 ceph-mon[48279]: pgmap v58: 129 pgs: 129 active+clean; 79 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 536 op/s 2024-03-14T09:56:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:01 smithi132 ceph-mon[47959]: pgmap v58: 129 pgs: 129 active+clean; 79 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 536 op/s 2024-03-14T09:56:01.546 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] nametest 2024-03-14T09:56:01.569 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] permname 2024-03-14T09:56:01.805 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] randholes 2024-03-14T09:56:01.808 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] runas 2024-03-14T09:56:01.946 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] truncfile 2024-03-14T09:56:01.984 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] usemem 2024-03-14T09:56:02.139 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] mmapcat 2024-03-14T09:56:02.243 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] append_reader 2024-03-14T09:56:02.279 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] append_writer 2024-03-14T09:56:02.280 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] dirperf 2024-03-14T09:56:02.436 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] metaperf 2024-03-14T09:56:02.564 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] devzero 2024-03-14T09:56:02.598 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] feature 2024-03-14T09:56:02.679 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] alloc 2024-03-14T09:56:02.883 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] fault 2024-03-14T09:56:02.905 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] fstest 2024-03-14T09:56:02.967 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_access_root 2024-03-14T09:56:03.077 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] godown 2024-03-14T09:56:03.233 INFO:tasks.workunit.client.0.smithi078.stderr:fstest.c: In function 'run_child': 2024-03-14T09:56:03.234 INFO:tasks.workunit.client.0.smithi078.stderr:fstest.c:95:20: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-03-14T09:56:03.234 INFO:tasks.workunit.client.0.smithi078.stderr: sprintf(fname, "%s/file%d", dir, fnum); 2024-03-14T09:56:03.234 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~~ 2024-03-14T09:56:03.234 INFO:tasks.workunit.client.0.smithi078.stderr:fstest.c:95:17: note: directive argument in the range [0, 2147483647] 2024-03-14T09:56:03.234 INFO:tasks.workunit.client.0.smithi078.stderr: sprintf(fname, "%s/file%d", dir, fnum); 2024-03-14T09:56:03.234 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~~~~~~~~ 2024-03-14T09:56:03.234 INFO:tasks.workunit.client.0.smithi078.stderr:fstest.c:95:2: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-03-14T09:56:03.234 INFO:tasks.workunit.client.0.smithi078.stderr: sprintf(fname, "%s/file%d", dir, fnum); 2024-03-14T09:56:03.234 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-14T09:56:03.234 INFO:tasks.workunit.client.0.smithi078.stderr:fstest.c:166:20: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-03-14T09:56:03.234 INFO:tasks.workunit.client.0.smithi078.stderr: sprintf(fname, "%s/file%d", dir, fnum); 2024-03-14T09:56:03.234 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~~ 2024-03-14T09:56:03.234 INFO:tasks.workunit.client.0.smithi078.stderr:fstest.c:166:17: note: directive argument in the range [0, 2147483647] 2024-03-14T09:56:03.235 INFO:tasks.workunit.client.0.smithi078.stderr: sprintf(fname, "%s/file%d", dir, fnum); 2024-03-14T09:56:03.235 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~~~~~~~~ 2024-03-14T09:56:03.235 INFO:tasks.workunit.client.0.smithi078.stderr:fstest.c:166:2: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-03-14T09:56:03.235 INFO:tasks.workunit.client.0.smithi078.stderr: sprintf(fname, "%s/file%d", dir, fnum); 2024-03-14T09:56:03.235 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-14T09:56:03.241 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] resvtest 2024-03-14T09:56:03.270 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] writemod 2024-03-14T09:56:03.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:03 smithi078 ceph-mon[38143]: pgmap v59: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 558 op/s 2024-03-14T09:56:03.341 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] writev_on_pagefault 2024-03-14T09:56:03.404 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] makeextents 2024-03-14T09:56:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:03 smithi152 ceph-mon[48279]: pgmap v59: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 558 op/s 2024-03-14T09:56:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:03 smithi132 ceph-mon[47959]: pgmap v59: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 558 op/s 2024-03-14T09:56:03.590 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] itrash 2024-03-14T09:56:03.603 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] rename 2024-03-14T09:56:03.658 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] multi_open_unlink 2024-03-14T09:56:03.766 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] dmiperf 2024-03-14T09:56:03.894 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] unwritten_sync 2024-03-14T09:56:03.913 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] genhashnames 2024-03-14T09:56:03.926 INFO:tasks.workunit.client.0.smithi078.stderr:multi_open_unlink.c: In function 'main': 2024-03-14T09:56:03.926 INFO:tasks.workunit.client.0.smithi078.stderr:multi_open_unlink.c:101:4: warning: 'attr_set' is deprecated: Use setxattr or lsetxattr instead [-Wdeprecated-declarations] 2024-03-14T09:56:03.926 INFO:tasks.workunit.client.0.smithi078.stderr: sts = attr_set(path, attrname, attrvalue, value_size, flags); 2024-03-14T09:56:03.926 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~ 2024-03-14T09:56:03.926 INFO:tasks.workunit.client.0.smithi078.stderr:In file included from multi_open_unlink.c:15: 2024-03-14T09:56:03.926 INFO:tasks.workunit.client.0.smithi078.stderr:/usr/include/attr/attributes.h:139:12: note: declared here 2024-03-14T09:56:03.926 INFO:tasks.workunit.client.0.smithi078.stderr: extern int attr_set (const char *__path, const char *__attrname, 2024-03-14T09:56:03.926 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~~~~~ 2024-03-14T09:56:04.006 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_holes 2024-03-14T09:56:04.065 INFO:tasks.workunit.client.0.smithi078.stderr:dmiperf.c: In function 'mkfile': 2024-03-14T09:56:04.066 INFO:tasks.workunit.client.0.smithi078.stderr:dmiperf.c:192:2: warning: 'attr_setf' is deprecated: Use fsetxattr instead [-Wdeprecated-declarations] 2024-03-14T09:56:04.066 INFO:tasks.workunit.client.0.smithi078.stderr: if (attr_setf(fd, DMFATTRNAME, attr, DMFATTRLEN, ATTR_ROOT) < 0) { 2024-03-14T09:56:04.066 INFO:tasks.workunit.client.0.smithi078.stderr: ^~ 2024-03-14T09:56:04.066 INFO:tasks.workunit.client.0.smithi078.stderr:In file included from dmiperf.c:20: 2024-03-14T09:56:04.066 INFO:tasks.workunit.client.0.smithi078.stderr:/usr/include/attr/attributes.h:143:12: note: declared here 2024-03-14T09:56:04.066 INFO:tasks.workunit.client.0.smithi078.stderr: extern int attr_setf (int __fd, const char *__attrname, 2024-03-14T09:56:04.066 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~~~~~~ 2024-03-14T09:56:04.182 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_mmap_writev 2024-03-14T09:56:04.243 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_truncate_cmtime 2024-03-14T09:56:04.294 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] dirhash_collide 2024-03-14T09:56:04.322 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_rename_overwrite 2024-03-14T09:56:04.526 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] holetest 2024-03-14T09:56:04.559 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_truncate_self 2024-03-14T09:56:04.638 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_mmap_dio 2024-03-14T09:56:04.642 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] af_unix 2024-03-14T09:56:04.855 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_mmap_stale_pmd 2024-03-14T09:56:04.920 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_mmap_cow_race 2024-03-14T09:56:04.945 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_mmap_fallocate 2024-03-14T09:56:04.971 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] fsync-err 2024-03-14T09:56:05.172 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_mmap_write_ro 2024-03-14T09:56:05.252 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_ext4_dax_journal_corruption 2024-03-14T09:56:05.261 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_ext4_dax_inline_corruption 2024-03-14T09:56:05.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:05 smithi078 ceph-mon[38143]: pgmap v60: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 607 KiB/s wr, 163 op/s 2024-03-14T09:56:05.426 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_ofd_locks 2024-03-14T09:56:05.486 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_locks_execve 2024-03-14T09:56:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:05 smithi152 ceph-mon[48279]: pgmap v60: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 607 KiB/s wr, 163 op/s 2024-03-14T09:56:05.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:05 smithi132 ceph-mon[47959]: pgmap v60: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 607 KiB/s wr, 163 op/s 2024-03-14T09:56:05.566 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_mmap_collision 2024-03-14T09:56:05.580 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] mmap-write-concurrent 2024-03-14T09:56:05.732 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] xfsctl 2024-03-14T09:56:05.813 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] bstat 2024-03-14T09:56:05.910 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_mtab 2024-03-14T09:56:05.945 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] getdevicesize 2024-03-14T09:56:06.104 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] preallo_rw_pattern_reader 2024-03-14T09:56:06.208 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] preallo_rw_pattern_writer 2024-03-14T09:56:06.257 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] ftrunc 2024-03-14T09:56:06.286 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] trunc 2024-03-14T09:56:06.443 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] fs_perms 2024-03-14T09:56:06.527 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] testx 2024-03-14T09:56:06.558 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] looptest 2024-03-14T09:56:06.632 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] locktest 2024-03-14T09:56:06.788 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] unwritten_mmap 2024-03-14T09:56:06.814 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] bulkstat_unlink_test 2024-03-14T09:56:06.922 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] bulkstat_unlink_test_modified 2024-03-14T09:56:07.158 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_dir_offset 2024-03-14T09:56:07.159 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_futimens 2024-03-14T09:56:07.182 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_immutable 2024-03-14T09:56:07.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:07 smithi078 ceph-mon[38143]: pgmap v61: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 619 KiB/s wr, 167 op/s 2024-03-14T09:56:07.305 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] stale_handle 2024-03-14T09:56:07.452 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] pwrite_mmap_blocked 2024-03-14T09:56:07.465 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_dir_offset2 2024-03-14T09:56:07.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:07 smithi152 ceph-mon[48279]: pgmap v61: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 619 KiB/s wr, 167 op/s 2024-03-14T09:56:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:07 smithi132 ceph-mon[47959]: pgmap v61: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 619 KiB/s wr, 167 op/s 2024-03-14T09:56:07.639 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] seek_sanity_test 2024-03-14T09:56:07.768 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] seek_copy_test 2024-03-14T09:56:07.796 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_readdir_1 2024-03-14T09:56:08.044 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_readdir_2 2024-03-14T09:56:08.079 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] fsync-tester 2024-03-14T09:56:08.151 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] nsexec 2024-03-14T09:56:08.295 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] cloner 2024-03-14T09:56:08.357 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] renameat2 2024-03-14T09:56:08.483 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_getcwd 2024-03-14T09:56:08.500 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] e4compact 2024-03-14T09:56:08.638 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] test-nextquota 2024-03-14T09:56:08.703 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] punch-alternating 2024-03-14T09:56:08.759 INFO:tasks.workunit.client.0.smithi078.stderr:t_getcwd.c: In function 'do_rename': 2024-03-14T09:56:08.759 INFO:tasks.workunit.client.0.smithi078.stderr:t_getcwd.c:40:2: warning: 'strncpy' specified bound 256 equals destination size [-Wstringop-truncation] 2024-03-14T09:56:08.759 INFO:tasks.workunit.client.0.smithi078.stderr: strncpy(c_name, prefix, BUF_SIZE); 2024-03-14T09:56:08.759 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-14T09:56:08.817 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] attr-list-by-handle-cursor-test 2024-03-14T09:56:08.873 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] listxattr 2024-03-14T09:56:08.988 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] dio-interleaved 2024-03-14T09:56:09.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:08 smithi078 ceph-mon[38143]: pgmap v62: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 75 KiB/s wr, 24 op/s 2024-03-14T09:56:09.062 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_dir_type 2024-03-14T09:56:09.154 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] dio-invalidate-cache 2024-03-14T09:56:09.196 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] stat_test 2024-03-14T09:56:09.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:08 smithi152 ceph-mon[48279]: pgmap v62: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 75 KiB/s wr, 24 op/s 2024-03-14T09:56:09.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:08 smithi132 ceph-mon[47959]: pgmap v62: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 75 KiB/s wr, 24 op/s 2024-03-14T09:56:09.313 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_encrypted_d_revalidate 2024-03-14T09:56:09.402 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] attr_replace_test 2024-03-14T09:56:09.545 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] swapon 2024-03-14T09:56:09.653 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] mkswap 2024-03-14T09:56:09.681 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_attr_corruption 2024-03-14T09:56:09.719 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_open_tmpfiles 2024-03-14T09:56:09.836 INFO:tasks.workunit.client.0.smithi078.stderr:swapon.c: In function 'main': 2024-03-14T09:56:09.836 INFO:tasks.workunit.client.0.smithi078.stderr:swapon.c:135:3: warning: 'p' may be used uninitialized in this function [-Wmaybe-uninitialized] 2024-03-14T09:56:09.836 INFO:tasks.workunit.client.0.smithi078.stderr: memcpy(p, buf, BUF_SIZE); 2024-03-14T09:56:09.836 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-14T09:56:09.894 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] fscrypt-crypt-util 2024-03-14T09:56:09.978 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] bulkstat_null_ocount 2024-03-14T09:56:09.988 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] splice-test 2024-03-14T09:56:10.085 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] fiemap-tester 2024-03-14T09:56:10.298 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] t_stripealign 2024-03-14T09:56:10.336 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] open_by_handle 2024-03-14T09:56:10.564 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] fssum 2024-03-14T09:56:10.637 INFO:tasks.workunit.client.0.smithi078.stdout:Building log-writes 2024-03-14T09:56:10.647 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] replay-log 2024-03-14T09:56:10.707 INFO:tasks.workunit.client.0.smithi078.stdout:Building perf 2024-03-14T09:56:10.715 INFO:tasks.workunit.client.0.smithi078.stdout:Building aio-dio-regress 2024-03-14T09:56:10.729 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aio-dio-append-write-fallocate-race 2024-03-14T09:56:10.762 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aio-dio-fcntl-race 2024-03-14T09:56:11.081 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aio-dio-subblock-eof-read 2024-03-14T09:56:11.086 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aio-dio-append-write-read-race 2024-03-14T09:56:11.141 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aio-dio-invalidate-readahead 2024-03-14T09:56:11.205 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aio-dio-hole-filling-race 2024-03-14T09:56:11.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:11 smithi078 ceph-mon[38143]: pgmap v63: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 75 KiB/s wr, 24 op/s 2024-03-14T09:56:11.394 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aio-free-ring-with-bogus-nr-pages 2024-03-14T09:56:11.414 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aio-last-ref-held-by-io 2024-03-14T09:56:11.473 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aio-dio-cycle-write 2024-03-14T09:56:11.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:11 smithi152 ceph-mon[48279]: pgmap v63: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 75 KiB/s wr, 24 op/s 2024-03-14T09:56:11.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:11 smithi132 ceph-mon[47959]: pgmap v63: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 75 KiB/s wr, 24 op/s 2024-03-14T09:56:11.536 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aio-dio-extend-stat 2024-03-14T09:56:11.696 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aio-io-setup-with-nonwritable-context-pointer 2024-03-14T09:56:11.766 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aiocp 2024-03-14T09:56:11.834 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aio-dio-invalidate-failure 2024-03-14T09:56:11.875 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aiodio_sparse2 2024-03-14T09:56:12.013 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aio-dio-cow-race 2024-03-14T09:56:12.157 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aio-dio-write-verify 2024-03-14T09:56:12.170 INFO:tasks.workunit.client.0.smithi078.stdout: [CC] aio-dio-eof-race 2024-03-14T09:56:12.220 INFO:tasks.workunit.client.0.smithi078.stderr:aiodio_sparse2.c: In function 'main': 2024-03-14T09:56:12.220 INFO:tasks.workunit.client.0.smithi078.stderr:aiodio_sparse2.c:404:2: warning: 'strncpy' specified bound 4096 equals destination size [-Wstringop-truncation] 2024-03-14T09:56:12.220 INFO:tasks.workunit.client.0.smithi078.stderr: strncpy(filename, argv[argc-1], PATH_MAX); 2024-03-14T09:56:12.220 INFO:tasks.workunit.client.0.smithi078.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-14T09:56:12.500 INFO:tasks.workunit.client.0.smithi078.stderr:+ cd .. 2024-03-14T09:56:12.500 INFO:tasks.workunit.client.0.smithi078.stderr:+ cp xfstests-dev/ltp/fsx . 2024-03-14T09:56:12.505 INFO:tasks.workunit.client.0.smithi078.stderr:+ OPTIONS=-z 2024-03-14T09:56:12.505 INFO:tasks.workunit.client.0.smithi078.stderr:+ ./fsx -z 1MB -N 50000 -p 10000 -l 1048576 2024-03-14T09:56:12.506 INFO:tasks.workunit.client.0.smithi078.stdout:Seed set to 1 2024-03-14T09:56:12.539 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support fallocate mode 0, disabling! 2024-03-14T09:56:12.539 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support fallocate mode FALLOC_FL_KEEP_SIZE, disabling! 2024-03-14T09:56:12.541 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support fallocate mode FALLOC_FL_COLLAPSE_RANGE, disabling! 2024-03-14T09:56:12.541 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support fallocate mode FALLOC_FL_INSERT_RANGE, disabling! 2024-03-14T09:56:12.542 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support clone range, disabling! 2024-03-14T09:56:12.543 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support dedupe range, disabling! 2024-03-14T09:56:12.552 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0x38fe9 2024-03-14T09:56:12.591 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0xead53 2024-03-14T09:56:12.695 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0xf2cdd 2024-03-14T09:56:12.704 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0xfca47 2024-03-14T09:56:13.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:13 smithi078 ceph-mon[38143]: pgmap v64: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 92 KiB/s wr, 27 op/s 2024-03-14T09:56:13.406 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0xfcdd6 2024-03-14T09:56:13.430 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0xfe854 2024-03-14T09:56:13.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:13 smithi152 ceph-mon[48279]: pgmap v64: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 92 KiB/s wr, 27 op/s 2024-03-14T09:56:13.514 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:56:13.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:13 smithi132 ceph-mon[47959]: pgmap v64: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 92 KiB/s wr, 27 op/s 2024-03-14T09:56:13.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.758+0000 7f6da2c00700 1 -- 172.21.15.78:0/2383813682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d9c073150 msgr2=0x7f6d9c073530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:13.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.758+0000 7f6da2c00700 1 --2- 172.21.15.78:0/2383813682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d9c073150 0x7f6d9c073530 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009990 tx=0x7f6d8c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.758+0000 7f6da2c00700 1 -- 172.21.15.78:0/2383813682 shutdown_connections 2024-03-14T09:56:13.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.758+0000 7f6da2c00700 1 --2- 172.21.15.78:0/2383813682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d9c073150 0x7f6d9c073530 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.758+0000 7f6da2c00700 1 --2- 172.21.15.78:0/2383813682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d9c111cc0 0x7f6d9c1140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.758+0000 7f6da2c00700 1 --2- 172.21.15.78:0/2383813682 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d9c073a70 0x7f6d9c111780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.758+0000 7f6da2c00700 1 -- 172.21.15.78:0/2383813682 >> 172.21.15.78:0/2383813682 conn(0x7f6d9c0fc7a0 msgr2=0x7f6d9c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:13.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.758+0000 7f6da2c00700 1 -- 172.21.15.78:0/2383813682 shutdown_connections 2024-03-14T09:56:13.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.758+0000 7f6da2c00700 1 -- 172.21.15.78:0/2383813682 wait complete. 2024-03-14T09:56:13.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.759+0000 7f6da2c00700 1 Processor -- start 2024-03-14T09:56:13.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.759+0000 7f6da2c00700 1 -- start start 2024-03-14T09:56:13.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.759+0000 7f6da2c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d9c073150 0x7f6d9c1a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.759+0000 7f6da2c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d9c073a70 0x7f6d9c1a3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.759+0000 7f6da2c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d9c111cc0 0x7f6d9c19da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.759+0000 7f6da2c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c19e0e0 con 0x7f6d9c111cc0 2024-03-14T09:56:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.759+0000 7f6da2c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c19e250 con 0x7f6d9c073150 2024-03-14T09:56:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.759+0000 7f6da2c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c19e410 con 0x7f6d9c073a70 2024-03-14T09:56:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.760+0000 7f6d9be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d9c073150 0x7f6d9c1a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.760+0000 7f6d9be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d9c073150 0x7f6d9c1a3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:59556/0 (socket says 172.21.15.78:59556) 2024-03-14T09:56:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.760+0000 7f6d9b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d9c073a70 0x7f6d9c1a3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.760+0000 7f6d9be00700 1 -- 172.21.15.78:0/4190387472 learned_addr learned my addr 172.21.15.78:0/4190387472 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.760+0000 7f6da0c00700 1 --2- 172.21.15.78:0/4190387472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d9c111cc0 0x7f6d9c19da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.760+0000 7f6d9be00700 1 -- 172.21.15.78:0/4190387472 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d9c073a70 msgr2=0x7f6d9c1a3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.760+0000 7f6d9be00700 1 --2- 172.21.15.78:0/4190387472 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d9c073a70 0x7f6d9c1a3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.760+0000 7f6d9be00700 1 -- 172.21.15.78:0/4190387472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d9c111cc0 msgr2=0x7f6d9c19da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.760+0000 7f6d9be00700 1 --2- 172.21.15.78:0/4190387472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d9c111cc0 0x7f6d9c19da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.760+0000 7f6d9be00700 1 -- 172.21.15.78:0/4190387472 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d88009520 con 0x7f6d9c073150 2024-03-14T09:56:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.760+0000 7f6d9b400700 1 --2- 172.21.15.78:0/4190387472 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d9c073a70 0x7f6d9c1a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:56:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.760+0000 7f6d9be00700 1 --2- 172.21.15.78:0/4190387472 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d9c073150 0x7f6d9c1a3950 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c000c00 tx=0x7f6d8c005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.761+0000 7f6d98c00700 1 -- 172.21.15.78:0/4190387472 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c01d070 con 0x7f6d9c073150 2024-03-14T09:56:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.761+0000 7f6d98c00700 1 -- 172.21.15.78:0/4190387472 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d8c022470 con 0x7f6d9c073150 2024-03-14T09:56:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.761+0000 7f6da2c00700 1 -- 172.21.15.78:0/4190387472 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8c0095f0 con 0x7f6d9c073150 2024-03-14T09:56:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.761+0000 7f6d98c00700 1 -- 172.21.15.78:0/4190387472 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c017650 con 0x7f6d9c073150 2024-03-14T09:56:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.761+0000 7f6da2c00700 1 -- 172.21.15.78:0/4190387472 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9c1aa760 con 0x7f6d9c073150 2024-03-14T09:56:13.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.765+0000 7f6d98c00700 1 -- 172.21.15.78:0/4190387472 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f6d8c0225e0 con 0x7f6d9c073150 2024-03-14T09:56:13.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.767+0000 7f6db676cb80 1 Processor -- start 2024-03-14T09:56:13.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.768+0000 7f6d98c00700 1 --2- 172.21.15.78:0/4190387472 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6d800a0c80 0x7f6d800a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.768+0000 7f6d98c00700 1 -- 172.21.15.78:0/4190387472 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f6d8c0cdac0 con 0x7f6d9c073150 2024-03-14T09:56:13.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.768+0000 7f6db676cb80 1 -- start start 2024-03-14T09:56:13.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.768+0000 7f6da0c00700 1 --2- 172.21.15.78:0/4190387472 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6d800a0c80 0x7f6d800a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.769+0000 7f6db676cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x556931614a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.769+0000 7f6db676cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x55693161e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.769+0000 7f6db676cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 0x5569316212b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.769+0000 7f6db676cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55693156d990 con 0x556931614fa0 2024-03-14T09:56:13.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.769+0000 7f6db676cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55693156b350 con 0x556931614680 2024-03-14T09:56:13.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.769+0000 7f6db676cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556931566c40 con 0x55693161eec0 2024-03-14T09:56:13.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.769+0000 7f6d9be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x556931614a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.769+0000 7f6d9be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x556931614a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:59560/0 (socket says 172.21.15.78:59560) 2024-03-14T09:56:13.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.769+0000 7f6d9be00700 1 -- 172.21.15.78:0/4249747982 learned_addr learned my addr 172.21.15.78:0/4249747982 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:13.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.769+0000 7f6da0c00700 1 --2- 172.21.15.78:0/4249747982 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 0x5569316212b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.769+0000 7f6da0c00700 1 --2- 172.21.15.78:0/4190387472 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6d800a0c80 0x7f6d800a3140 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c06a350 tx=0x7f6d9400b040 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:13.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.770+0000 7f6d9b400700 1 --2- 172.21.15.78:0/4249747982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x55693161e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.770+0000 7f6d9be00700 1 -- 172.21.15.78:0/4249747982 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 msgr2=0x5569316212b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:13.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.770+0000 7f6d9be00700 1 --2- 172.21.15.78:0/4249747982 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 0x5569316212b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.770+0000 7f6d9be00700 1 -- 172.21.15.78:0/4249747982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 msgr2=0x55693161e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:13.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.770+0000 7f6d9be00700 1 --2- 172.21.15.78:0/4249747982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x55693161e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.770+0000 7f6d9be00700 1 -- 172.21.15.78:0/4249747982 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569313e83b0 con 0x556931614680 2024-03-14T09:56:13.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.770+0000 7f6d9b400700 1 --2- 172.21.15.78:0/4249747982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x55693161e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:56:13.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.770+0000 7f6d9be00700 1 --2- 172.21.15.78:0/4249747982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x556931614a60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c097110 tx=0x7f6d8c022b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af3321f64f2bb5a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:13.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.771+0000 7f6d91200700 1 -- 172.21.15.78:0/4249747982 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c09b310 con 0x556931614680 2024-03-14T09:56:13.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.771+0000 7f6d91200700 1 -- 172.21.15.78:0/4249747982 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d8c09ac10 con 0x556931614680 2024-03-14T09:56:13.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.771+0000 7f6d91200700 1 -- 172.21.15.78:0/4249747982 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c09b0d0 con 0x556931614680 2024-03-14T09:56:13.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.771+0000 7f6db676cb80 1 -- 172.21.15.78:0/4249747982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 msgr2=0x556931614a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:13.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.771+0000 7f6db676cb80 1 --2- 172.21.15.78:0/4249747982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x556931614a60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c097110 tx=0x7f6d8c022b90 comp rx=0 tx=0).stop 2024-03-14T09:56:13.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.772+0000 7f6db676cb80 1 -- 172.21.15.78:0/4249747982 shutdown_connections 2024-03-14T09:56:13.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.772+0000 7f6db676cb80 1 --2- 172.21.15.78:0/4249747982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x556931614a60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.772+0000 7f6db676cb80 1 --2- 172.21.15.78:0/4249747982 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 0x5569316212b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.772+0000 7f6db676cb80 1 --2- 172.21.15.78:0/4249747982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x55693161e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.772+0000 7f6db676cb80 1 -- 172.21.15.78:0/4249747982 >> 172.21.15.78:0/4249747982 conn(0x55693158a220 msgr2=0x55693160faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:13.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.772+0000 7f6db676cb80 1 -- 172.21.15.78:0/4249747982 shutdown_connections 2024-03-14T09:56:13.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.772+0000 7f6db676cb80 1 -- 172.21.15.78:0/4249747982 wait complete. 2024-03-14T09:56:13.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.772+0000 7f6db676cb80 1 -- start start 2024-03-14T09:56:13.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.773+0000 7f6db676cb80 1 Processor -- start 2024-03-14T09:56:13.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.773+0000 7f6db676cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x55693161e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6db676cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x55693161bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6db676cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 0x55693161c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6db676cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556931502d80 con 0x556931614fa0 2024-03-14T09:56:13.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6db676cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x556931502a20 con 0x556931614680 2024-03-14T09:56:13.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6db676cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569313ea6a0 con 0x55693161eec0 2024-03-14T09:56:13.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6db676cb80 10 client.0 _ll_register_callbacks cb 0x55693136e3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:13.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6da0c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 0x55693161c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6da0c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 0x55693161c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60886/0 (socket says 172.21.15.78:60886) 2024-03-14T09:56:13.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6da0c00700 1 -- 172.21.15.78:0/3978691135 learned_addr learned my addr 172.21.15.78:0/3978691135 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:13.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6d9b400700 1 --2- 172.21.15.78:0/3978691135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x55693161bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6d9be00700 1 --2- 172.21.15.78:0/3978691135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x55693161e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6da0c00700 1 -- 172.21.15.78:0/3978691135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 msgr2=0x55693161e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:13.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6da0c00700 1 --2- 172.21.15.78:0/3978691135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x55693161e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6da0c00700 1 -- 172.21.15.78:0/3978691135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 msgr2=0x55693161bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:13.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6da0c00700 1 --2- 172.21.15.78:0/3978691135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x55693161bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6da0c00700 1 -- 172.21.15.78:0/3978691135 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9c1aa760 con 0x55693161eec0 2024-03-14T09:56:13.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.774+0000 7f6d9be00700 1 --2- 172.21.15.78:0/3978691135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x55693161e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:56:13.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.775+0000 7f6da0c00700 1 --2- 172.21.15.78:0/3978691135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 0x55693161c3e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6d940087c0 tx=0x7f6d94004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64d362ec50bb0be2 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:13.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.775+0000 7f6d85600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:56:13.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.775+0000 7f6d85600700 1 -- 172.21.15.78:0/3978691135 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9401c040 con 0x55693161eec0 2024-03-14T09:56:13.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.775+0000 7f6db676cb80 1 -- 172.21.15.78:0/3978691135 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556931607720 con 0x55693161eec0 2024-03-14T09:56:13.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.775+0000 7f6d85600700 1 -- 172.21.15.78:0/3978691135 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f6d940178d0 con 0x55693161eec0 2024-03-14T09:56:13.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.775+0000 7f6d85600700 1 -- 172.21.15.78:0/3978691135 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d9400e040 con 0x55693161eec0 2024-03-14T09:56:13.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.775+0000 7f6db676cb80 10 client.26065 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:13.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.775+0000 7f6db676cb80 1 -- 172.21.15.78:0/3978691135 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556931394b60 con 0x55693161eec0 2024-03-14T09:56:13.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.775+0000 7f6d85600700 1 -- 172.21.15.78:0/3978691135 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556931607720 con 0x55693161eec0 2024-03-14T09:56:13.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.776+0000 7f6d85600700 1 -- 172.21.15.78:0/3978691135 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6d94022c90 con 0x55693161eec0 2024-03-14T09:56:13.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.776+0000 7f6db676cb80 10 client.26065 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:13.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.776+0000 7f6db676cb80 10 client.26065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:56:13.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.776+0000 7f6db676cb80 20 client.26065 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:13.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.776+0000 7f6db676cb80 1 --2- 172.21.15.78:0/3978691135 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55693162e7d0 0x556931630bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.776+0000 7f6db676cb80 4 client.26065 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:56:13.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.776+0000 7f6db676cb80 1 -- 172.21.15.78:0/3978691135 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55693155f700 con 0x55693162e7d0 2024-03-14T09:56:13.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.776+0000 7f6d9be00700 1 --2- 172.21.15.78:0/3978691135 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55693162e7d0 0x556931630bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.776+0000 7f6d85600700 10 client.26065 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:13.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.776+0000 7f6d9be00700 1 --2- 172.21.15.78:0/3978691135 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55693162e7d0 0x556931630bf0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db9f5c7d086824e6 server_cookie=f3a25d20029a174b in_seq=0 out_seq=0 2024-03-14T09:56:13.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.777+0000 7f6d85600700 1 -- 172.21.15.78:0/3978691135 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55693155f700 con 0x55693162e7d0 2024-03-14T09:56:13.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.777+0000 7f6d85600700 10 client.26065 handle_command_reply: tid=0 2024-03-14T09:56:13.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.777+0000 7f6db676cb80 1 client.26065 shutdown 2024-03-14T09:56:13.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.777+0000 7f6db676cb80 1 -- 172.21.15.78:0/3978691135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 msgr2=0x55693161c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:13.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.777+0000 7f6db676cb80 1 --2- 172.21.15.78:0/3978691135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 0x55693161c3e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6d940087c0 tx=0x7f6d94004ad0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.778+0000 7f6db676cb80 1 -- 172.21.15.78:0/3978691135 shutdown_connections 2024-03-14T09:56:13.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.778+0000 7f6db676cb80 1 --2- 172.21.15.78:0/3978691135 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55693162e7d0 0x556931630bf0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.778+0000 7f6db676cb80 1 --2- 172.21.15.78:0/3978691135 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x55693161e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.778+0000 7f6d85600700 0 client.26065 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:13.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.778+0000 7f6db676cb80 1 --2- 172.21.15.78:0/3978691135 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 0x55693161c3e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.778+0000 7f6db676cb80 1 --2- 172.21.15.78:0/3978691135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x55693161bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.778+0000 7f6db676cb80 1 -- 172.21.15.78:0/3978691135 >> 172.21.15.78:0/3978691135 conn(0x55693158a220 msgr2=0x55693160f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:13.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.778+0000 7f6db676cb80 1 -- 172.21.15.78:0/3978691135 shutdown_connections 2024-03-14T09:56:13.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.778+0000 7f6db676cb80 1 -- 172.21.15.78:0/3978691135 wait complete. 2024-03-14T09:56:13.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.778+0000 7f6db676cb80 20 client.26065 trim_cache size 0 max 16384 2024-03-14T09:56:13.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.801+0000 7f6db676cb80 1 Processor -- start 2024-03-14T09:56:13.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.801+0000 7f6db676cb80 1 -- start start 2024-03-14T09:56:13.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.801+0000 7f6db676cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556931614680 0x556931625880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.801+0000 7f6db676cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x55693160ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.801+0000 7f6db676cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693161eec0 0x55693161b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.801+0000 7f6db676cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569316358d0 con 0x556931614fa0 2024-03-14T09:56:13.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.801+0000 7f6db676cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x556931615890 con 0x55693161eec0 2024-03-14T09:56:13.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.801+0000 7f6db676cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556931617b00 con 0x556931614680 2024-03-14T09:56:13.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.802+0000 7f6d9be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556931614680 0x556931625880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.802+0000 7f6d9b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x55693160ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.802+0000 7f6d9be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556931614680 0x556931625880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60892/0 (socket says 172.21.15.78:60892) 2024-03-14T09:56:13.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.802+0000 7f6d9be00700 1 -- 172.21.15.78:0/2961743038 learned_addr learned my addr 172.21.15.78:0/2961743038 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:13.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.802+0000 7f6da0c00700 1 --2- 172.21.15.78:0/2961743038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693161eec0 0x55693161b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.802+0000 7f6d9be00700 1 -- 172.21.15.78:0/2961743038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693161eec0 msgr2=0x55693161b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:13.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.802+0000 7f6d9be00700 1 --2- 172.21.15.78:0/2961743038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693161eec0 0x55693161b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.802+0000 7f6d9be00700 1 -- 172.21.15.78:0/2961743038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 msgr2=0x55693160ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:13.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.802+0000 7f6d9be00700 1 --2- 172.21.15.78:0/2961743038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x55693160ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.802+0000 7f6d9be00700 1 -- 172.21.15.78:0/2961743038 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55693152cea0 con 0x556931614680 2024-03-14T09:56:13.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.802+0000 7f6d9b400700 1 --2- 172.21.15.78:0/2961743038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x55693160ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:56:13.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.803+0000 7f6d9be00700 1 --2- 172.21.15.78:0/2961743038 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556931614680 0x556931625880 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c097680 tx=0x7f6d8c022c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c6998a2bfa3f41a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:13.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.803+0000 7f6d91c00700 1 -- 172.21.15.78:0/2961743038 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c09eac0 con 0x556931614680 2024-03-14T09:56:13.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.803+0000 7f6d91c00700 1 -- 172.21.15.78:0/2961743038 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d8c09b310 con 0x556931614680 2024-03-14T09:56:13.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.803+0000 7f6d91c00700 1 -- 172.21.15.78:0/2961743038 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c09ae70 con 0x556931614680 2024-03-14T09:56:13.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.804+0000 7f6db676cb80 1 -- 172.21.15.78:0/2961743038 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556931614680 msgr2=0x556931625880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:13.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.804+0000 7f6db676cb80 1 --2- 172.21.15.78:0/2961743038 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556931614680 0x556931625880 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c097680 tx=0x7f6d8c022c90 comp rx=0 tx=0).stop 2024-03-14T09:56:13.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.804+0000 7f6db676cb80 1 -- 172.21.15.78:0/2961743038 shutdown_connections 2024-03-14T09:56:13.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.804+0000 7f6db676cb80 1 --2- 172.21.15.78:0/2961743038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55693161eec0 0x55693161b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.804+0000 7f6db676cb80 1 --2- 172.21.15.78:0/2961743038 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556931614680 0x556931625880 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.804+0000 7f6db676cb80 1 --2- 172.21.15.78:0/2961743038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x55693160ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.804+0000 7f6db676cb80 1 -- 172.21.15.78:0/2961743038 >> 172.21.15.78:0/2961743038 conn(0x55693158a220 msgr2=0x55693160b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:13.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.804+0000 7f6db676cb80 1 -- 172.21.15.78:0/2961743038 shutdown_connections 2024-03-14T09:56:13.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.804+0000 7f6db676cb80 1 -- 172.21.15.78:0/2961743038 wait complete. 2024-03-14T09:56:13.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.805+0000 7f6db676cb80 1 -- start start 2024-03-14T09:56:13.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.805+0000 7f6db676cb80 1 Processor -- start 2024-03-14T09:56:13.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.806+0000 7f6db676cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x5569316253d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.806+0000 7f6db676cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x556931623860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.806+0000 7f6db676cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 0x556931623da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.806+0000 7f6db676cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569316242e0 con 0x556931614fa0 2024-03-14T09:56:13.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.806+0000 7f6db676cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x556931624450 con 0x556931614680 2024-03-14T09:56:13.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.806+0000 7f6db676cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556931624590 con 0x55693161eec0 2024-03-14T09:56:13.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.807+0000 7f6db676cb80 10 client.0 _ll_register_callbacks cb 0x55693136e3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:13.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.807+0000 7f6d9be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x5569316253d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.807+0000 7f6d9be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x5569316253d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:59586/0 (socket says 172.21.15.78:59586) 2024-03-14T09:56:13.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.807+0000 7f6d9be00700 1 -- 172.21.15.78:0/2766982796 learned_addr learned my addr 172.21.15.78:0/2766982796 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:13.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.807+0000 7f6d9b400700 1 --2- 172.21.15.78:0/2766982796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x556931623860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.807+0000 7f6da0c00700 1 --2- 172.21.15.78:0/2766982796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 0x556931623da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.807+0000 7f6da0c00700 1 -- 172.21.15.78:0/2766982796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 msgr2=0x5569316253d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:13.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.808+0000 7f6da0c00700 1 --2- 172.21.15.78:0/2766982796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x5569316253d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.808+0000 7f6da0c00700 1 -- 172.21.15.78:0/2766982796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 msgr2=0x556931623860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:13.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.808+0000 7f6da0c00700 1 --2- 172.21.15.78:0/2766982796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x556931623860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.808+0000 7f6da0c00700 1 -- 172.21.15.78:0/2766982796 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d880135f0 con 0x55693161eec0 2024-03-14T09:56:13.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.808+0000 7f6d9b400700 1 --2- 172.21.15.78:0/2766982796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x556931623860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:56:13.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.808+0000 7f6d9be00700 1 --2- 172.21.15.78:0/2766982796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x5569316253d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:56:13.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.808+0000 7f6da0c00700 1 --2- 172.21.15.78:0/2766982796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 0x556931623da0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6d94008e60 tx=0x7f6d94017d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85fbbd898cc47b39 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:13.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.808+0000 7f6d85600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:56:13.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.808+0000 7f6d85600700 1 -- 172.21.15.78:0/2766982796 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9400e040 con 0x55693161eec0 2024-03-14T09:56:13.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.808+0000 7f6d85600700 1 -- 172.21.15.78:0/2766982796 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f6d940178d0 con 0x55693161eec0 2024-03-14T09:56:13.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.808+0000 7f6db676cb80 1 -- 172.21.15.78:0/2766982796 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d8c09ac10 con 0x55693161eec0 2024-03-14T09:56:13.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.809+0000 7f6d85600700 1 -- 172.21.15.78:0/2766982796 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d94009920 con 0x55693161eec0 2024-03-14T09:56:13.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.809+0000 7f6db676cb80 10 client.26071 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:13.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.809+0000 7f6db676cb80 1 -- 172.21.15.78:0/2766982796 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5569313e83b0 con 0x55693161eec0 2024-03-14T09:56:13.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.809+0000 7f6d85600700 1 -- 172.21.15.78:0/2766982796 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d8c09ac10 con 0x55693161eec0 2024-03-14T09:56:13.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.809+0000 7f6d85600700 1 -- 172.21.15.78:0/2766982796 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6d94015890 con 0x55693161eec0 2024-03-14T09:56:13.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.809+0000 7f6db676cb80 10 client.26071 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:13.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.809+0000 7f6db676cb80 10 client.26071 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:56:13.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.809+0000 7f6db676cb80 20 client.26071 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:13.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.809+0000 7f6db676cb80 1 --2- 172.21.15.78:0/2766982796 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x556931692610 0x556931694a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:13.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.809+0000 7f6db676cb80 4 client.26071 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:56:13.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.809+0000 7f6db676cb80 1 -- 172.21.15.78:0/2766982796 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5569313b5cf0 con 0x556931692610 2024-03-14T09:56:13.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.810+0000 7f6d9be00700 1 --2- 172.21.15.78:0/2766982796 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x556931692610 0x556931694a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:13.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.810+0000 7f6d85600700 10 client.26071 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:13.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.810+0000 7f6d9be00700 1 --2- 172.21.15.78:0/2766982796 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x556931692610 0x556931694a30 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4767cee0c33e1c server_cookie=95037e745cba8f92 in_seq=0 out_seq=0 2024-03-14T09:56:13.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.811+0000 7f6d85600700 1 -- 172.21.15.78:0/2766982796 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5569313b5cf0 con 0x556931692610 2024-03-14T09:56:13.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.811+0000 7f6d85600700 10 client.26071 handle_command_reply: tid=0 2024-03-14T09:56:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.811+0000 7f6db676cb80 1 client.26071 shutdown 2024-03-14T09:56:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.811+0000 7f6db676cb80 1 -- 172.21.15.78:0/2766982796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 msgr2=0x556931623da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.811+0000 7f6db676cb80 1 --2- 172.21.15.78:0/2766982796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 0x556931623da0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6d94008e60 tx=0x7f6d94017d70 comp rx=0 tx=0).stop 2024-03-14T09:56:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.811+0000 7f6db676cb80 1 -- 172.21.15.78:0/2766982796 shutdown_connections 2024-03-14T09:56:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.811+0000 7f6db676cb80 1 --2- 172.21.15.78:0/2766982796 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x556931692610 0x556931694a30 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.811+0000 7f6db676cb80 1 --2- 172.21.15.78:0/2766982796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556931614680 0x5569316253d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.811+0000 7f6db676cb80 1 --2- 172.21.15.78:0/2766982796 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55693161eec0 0x556931623da0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.811+0000 7f6db676cb80 1 --2- 172.21.15.78:0/2766982796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556931614fa0 0x556931623860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.811+0000 7f6db676cb80 1 -- 172.21.15.78:0/2766982796 >> 172.21.15.78:0/2766982796 conn(0x55693158a220 msgr2=0x55693160ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.811+0000 7f6d85600700 0 client.26071 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.811+0000 7f6db676cb80 1 -- 172.21.15.78:0/2766982796 shutdown_connections 2024-03-14T09:56:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.811+0000 7f6db676cb80 1 -- 172.21.15.78:0/2766982796 wait complete. 2024-03-14T09:56:13.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.812+0000 7f6db676cb80 20 client.26071 trim_cache size 0 max 16384 2024-03-14T09:56:13.814 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:56:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T09:56:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T09:56:13.814 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:56:13.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.812+0000 7f6da2c00700 1 -- 172.21.15.78:0/4190387472 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6d800a0c80 msgr2=0x7f6d800a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:13.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.812+0000 7f6da2c00700 1 --2- 172.21.15.78:0/4190387472 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6d800a0c80 0x7f6d800a3140 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c06a350 tx=0x7f6d9400b040 comp rx=0 tx=0).stop 2024-03-14T09:56:13.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.812+0000 7f6da2c00700 1 -- 172.21.15.78:0/4190387472 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d9c073150 msgr2=0x7f6d9c1a3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.812+0000 7f6da2c00700 1 --2- 172.21.15.78:0/4190387472 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d9c073150 0x7f6d9c1a3950 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c000c00 tx=0x7f6d8c005740 comp rx=0 tx=0).stop 2024-03-14T09:56:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.813+0000 7f6da2c00700 1 -- 172.21.15.78:0/4190387472 shutdown_connections 2024-03-14T09:56:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.813+0000 7f6da2c00700 1 --2- 172.21.15.78:0/4190387472 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6d800a0c80 0x7f6d800a3140 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.813+0000 7f6da2c00700 1 --2- 172.21.15.78:0/4190387472 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d9c073150 0x7f6d9c1a3950 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.813+0000 7f6da2c00700 1 --2- 172.21.15.78:0/4190387472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d9c111cc0 0x7f6d9c19da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.813+0000 7f6da2c00700 1 --2- 172.21.15.78:0/4190387472 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d9c073a70 0x7f6d9c1a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.813+0000 7f6da2c00700 1 -- 172.21.15.78:0/4190387472 >> 172.21.15.78:0/4190387472 conn(0x7f6d9c0fc7a0 msgr2=0x7f6d9c0fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.813+0000 7f6da2c00700 1 -- 172.21.15.78:0/4190387472 shutdown_connections 2024-03-14T09:56:13.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:13.813+0000 7f6da2c00700 1 -- 172.21.15.78:0/4190387472 wait complete. 2024-03-14T09:56:13.836 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T09:56:13.836 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:56:14.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.081+0000 7f77d7800700 1 -- 172.21.15.78:0/3852378880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77d01026a0 msgr2=0x7f77d010ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:14.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.081+0000 7f77d7800700 1 --2- 172.21.15.78:0/3852378880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77d01026a0 0x7f77d010ab80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009990 tx=0x7f77bc009ca0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.081+0000 7f77d7800700 1 -- 172.21.15.78:0/3852378880 shutdown_connections 2024-03-14T09:56:14.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.081+0000 7f77d7800700 1 --2- 172.21.15.78:0/3852378880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77d010b1a0 0x7f77d010d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.081+0000 7f77d7800700 1 --2- 172.21.15.78:0/3852378880 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f77d0101d80 0x7f77d0102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.081+0000 7f77d7800700 1 --2- 172.21.15.78:0/3852378880 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77d01026a0 0x7f77d010ab80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.081+0000 7f77d7800700 1 -- 172.21.15.78:0/3852378880 >> 172.21.15.78:0/3852378880 conn(0x7f77d00fb5c0 msgr2=0x7f77d00fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:14.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.081+0000 7f77d7800700 1 -- 172.21.15.78:0/3852378880 shutdown_connections 2024-03-14T09:56:14.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.082+0000 7f77d7800700 1 -- 172.21.15.78:0/3852378880 wait complete. 2024-03-14T09:56:14.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.083+0000 7f77d7800700 1 Processor -- start 2024-03-14T09:56:14.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.083+0000 7f77d7800700 1 -- start start 2024-03-14T09:56:14.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.084+0000 7f77d7800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77d0101d80 0x7f77d0102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.084+0000 7f77d7800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f77d01026a0 0x7f77d01030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.084+0000 7f77d7800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77d010b1a0 0x7f77d0103610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.084+0000 7f77d7800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d01a9d40 con 0x7f77d010b1a0 2024-03-14T09:56:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.084+0000 7f77d7800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d01994e0 con 0x7f77d0101d80 2024-03-14T09:56:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.084+0000 7f77d7800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d01996a0 con 0x7f77d01026a0 2024-03-14T09:56:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.084+0000 7f77d4e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77d0101d80 0x7f77d0102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.084+0000 7f77d4e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77d0101d80 0x7f77d0102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:56334/0 (socket says 172.21.15.78:56334) 2024-03-14T09:56:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.084+0000 7f77d4e00700 1 -- 172.21.15.78:0/3854515548 learned_addr learned my addr 172.21.15.78:0/3854515548 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.084+0000 7f77d5800700 1 --2- 172.21.15.78:0/3854515548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77d010b1a0 0x7f77d0103610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.084+0000 7f77cfe00700 1 --2- 172.21.15.78:0/3854515548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f77d01026a0 0x7f77d01030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.085+0000 7f77d4e00700 1 -- 172.21.15.78:0/3854515548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f77d01026a0 msgr2=0x7f77d01030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.085+0000 7f77d4e00700 1 --2- 172.21.15.78:0/3854515548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f77d01026a0 0x7f77d01030d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.085+0000 7f77d4e00700 1 -- 172.21.15.78:0/3854515548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77d010b1a0 msgr2=0x7f77d0103610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:14.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.085+0000 7f77d4e00700 1 --2- 172.21.15.78:0/3854515548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77d010b1a0 0x7f77d0103610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.085+0000 7f77d4e00700 1 -- 172.21.15.78:0/3854515548 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77bc0095f0 con 0x7f77d0101d80 2024-03-14T09:56:14.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.085+0000 7f77d4e00700 1 --2- 172.21.15.78:0/3854515548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77d0101d80 0x7f77d0102b90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f77c400e9a0 tx=0x7f77c400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:14.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.085+0000 7f77cd600700 1 -- 172.21.15.78:0/3854515548 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c4009790 con 0x7f77d0101d80 2024-03-14T09:56:14.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.085+0000 7f77cd600700 1 -- 172.21.15.78:0/3854515548 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77c4010430 con 0x7f77d0101d80 2024-03-14T09:56:14.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.085+0000 7f77cd600700 1 -- 172.21.15.78:0/3854515548 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c4018680 con 0x7f77d0101d80 2024-03-14T09:56:14.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.086+0000 7f77d7800700 1 -- 172.21.15.78:0/3854515548 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d0199980 con 0x7f77d0101d80 2024-03-14T09:56:14.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.086+0000 7f77d7800700 1 -- 172.21.15.78:0/3854515548 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d0199e40 con 0x7f77d0101d80 2024-03-14T09:56:14.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.087+0000 7f77d7800700 1 -- 172.21.15.78:0/3854515548 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d0108670 con 0x7f77d0101d80 2024-03-14T09:56:14.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.090+0000 7f77cd600700 1 -- 172.21.15.78:0/3854515548 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f77c40105a0 con 0x7f77d0101d80 2024-03-14T09:56:14.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.092+0000 7f77cd600700 1 --2- 172.21.15.78:0/3854515548 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f77b80a0c80 0x7f77b80a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.092+0000 7f77cd600700 1 -- 172.21.15.78:0/3854515548 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f77c4014070 con 0x7f77d0101d80 2024-03-14T09:56:14.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.092+0000 7f77cfe00700 1 --2- 172.21.15.78:0/3854515548 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f77b80a0c80 0x7f77b80a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.092+0000 7f77cfe00700 1 --2- 172.21.15.78:0/3854515548 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f77b80a0c80 0x7f77b80a3140 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f77d0104270 tx=0x7f77bc00b280 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:14.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.096+0000 7f77cd600700 1 -- 172.21.15.78:0/3854515548 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f77c4052960 con 0x7f77d0101d80 2024-03-14T09:56:14.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.438+0000 7f77d7800700 1 -- 172.21.15.78:0/3854515548 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77d00686a0 con 0x7f77d0101d80 2024-03-14T09:56:14.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.439+0000 7f77cd600700 1 -- 172.21.15.78:0/3854515548 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f77d00686a0 con 0x7f77d0101d80 2024-03-14T09:56:14.439 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:56:14.439 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:56:14.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.442+0000 7f77d7800700 1 -- 172.21.15.78:0/3854515548 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f77b80a0c80 msgr2=0x7f77b80a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:14.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.442+0000 7f77d7800700 1 --2- 172.21.15.78:0/3854515548 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f77b80a0c80 0x7f77b80a3140 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f77d0104270 tx=0x7f77bc00b280 comp rx=0 tx=0).stop 2024-03-14T09:56:14.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.442+0000 7f77d7800700 1 -- 172.21.15.78:0/3854515548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77d0101d80 msgr2=0x7f77d0102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:14.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.442+0000 7f77d7800700 1 --2- 172.21.15.78:0/3854515548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77d0101d80 0x7f77d0102b90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f77c400e9a0 tx=0x7f77c400ecb0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.442+0000 7f77d7800700 1 -- 172.21.15.78:0/3854515548 shutdown_connections 2024-03-14T09:56:14.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.442+0000 7f77d7800700 1 --2- 172.21.15.78:0/3854515548 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f77b80a0c80 0x7f77b80a3140 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.442+0000 7f77d7800700 1 --2- 172.21.15.78:0/3854515548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77d0101d80 0x7f77d0102b90 secure :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f77c400e9a0 tx=0x7f77c400ecb0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.442+0000 7f77d7800700 1 --2- 172.21.15.78:0/3854515548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77d010b1a0 0x7f77d0103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.442+0000 7f77d7800700 1 --2- 172.21.15.78:0/3854515548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f77d01026a0 0x7f77d01030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.442+0000 7f77d7800700 1 -- 172.21.15.78:0/3854515548 >> 172.21.15.78:0/3854515548 conn(0x7f77d00fb5c0 msgr2=0x7f77d0100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:14.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.442+0000 7f77d7800700 1 -- 172.21.15.78:0/3854515548 shutdown_connections 2024-03-14T09:56:14.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.442+0000 7f77d7800700 1 -- 172.21.15.78:0/3854515548 wait complete. 2024-03-14T09:56:14.444 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T09:56:14.465 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T09:56:14.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.713+0000 7f57f2200700 1 -- 172.21.15.78:0/69271711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57ec1018e0 msgr2=0x7f57ec105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:14.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.713+0000 7f57f2200700 1 --2- 172.21.15.78:0/69271711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57ec1018e0 0x7f57ec105920 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f57d8009990 tx=0x7f57d8009ca0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.714+0000 7f57f2200700 1 -- 172.21.15.78:0/69271711 shutdown_connections 2024-03-14T09:56:14.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.714+0000 7f57f2200700 1 --2- 172.21.15.78:0/69271711 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57ec100f30 0x7f57ec101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.714+0000 7f57f2200700 1 --2- 172.21.15.78:0/69271711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57ec106050 0x7f57ec10dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.714+0000 7f57f2200700 1 --2- 172.21.15.78:0/69271711 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57ec1018e0 0x7f57ec105920 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.714+0000 7f57f2200700 1 -- 172.21.15.78:0/69271711 >> 172.21.15.78:0/69271711 conn(0x7f57ec0fc7b0 msgr2=0x7f57ec0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:14.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.714+0000 7f57f2200700 1 -- 172.21.15.78:0/69271711 shutdown_connections 2024-03-14T09:56:14.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.714+0000 7f57f2200700 1 -- 172.21.15.78:0/69271711 wait complete. 2024-03-14T09:56:14.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.715+0000 7f57f2200700 1 Processor -- start 2024-03-14T09:56:14.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.715+0000 7f57f2200700 1 -- start start 2024-03-14T09:56:14.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.716+0000 7f57f2200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57ec100f30 0x7f57ec1976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.716+0000 7f57f2200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57ec106050 0x7f57ec197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.716+0000 7f57f2200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57ec19bf40 0x7f57ec19c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.716+0000 7f57f2200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec19c9c0 con 0x7f57ec106050 2024-03-14T09:56:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.716+0000 7f57f2200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec19cb30 con 0x7f57ec19bf40 2024-03-14T09:56:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.716+0000 7f57f2200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec19ccf0 con 0x7f57ec100f30 2024-03-14T09:56:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.716+0000 7f57eb400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57ec100f30 0x7f57ec1976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.716+0000 7f57eb400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57ec100f30 0x7f57ec1976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:54976/0 (socket says 172.21.15.78:54976) 2024-03-14T09:56:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.716+0000 7f57eb400700 1 -- 172.21.15.78:0/4242406131 learned_addr learned my addr 172.21.15.78:0/4242406131 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.716+0000 7f57eaa00700 1 --2- 172.21.15.78:0/4242406131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57ec106050 0x7f57ec197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.716+0000 7f57ebe00700 1 --2- 172.21.15.78:0/4242406131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57ec19bf40 0x7f57ec19c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.717+0000 7f57eb400700 1 -- 172.21.15.78:0/4242406131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57ec19bf40 msgr2=0x7f57ec19c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:14.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.717+0000 7f57eb400700 1 --2- 172.21.15.78:0/4242406131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57ec19bf40 0x7f57ec19c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.717+0000 7f57eb400700 1 -- 172.21.15.78:0/4242406131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57ec106050 msgr2=0x7f57ec197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:14.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.717+0000 7f57eb400700 1 --2- 172.21.15.78:0/4242406131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57ec106050 0x7f57ec197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.717+0000 7f57eb400700 1 -- 172.21.15.78:0/4242406131 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d80095f0 con 0x7f57ec100f30 2024-03-14T09:56:14.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.717+0000 7f57eb400700 1 --2- 172.21.15.78:0/4242406131 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57ec100f30 0x7f57ec1976b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00e9a0 tx=0x7f57dc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:14.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.717+0000 7f57e3e00700 1 -- 172.21.15.78:0/4242406131 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc009790 con 0x7f57ec100f30 2024-03-14T09:56:14.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.717+0000 7f57e3e00700 1 -- 172.21.15.78:0/4242406131 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57dc010430 con 0x7f57ec100f30 2024-03-14T09:56:14.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.717+0000 7f57e3e00700 1 -- 172.21.15.78:0/4242406131 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc018680 con 0x7f57ec100f30 2024-03-14T09:56:14.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.717+0000 7f57f2200700 1 -- 172.21.15.78:0/4242406131 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57ec1a4250 con 0x7f57ec100f30 2024-03-14T09:56:14.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.717+0000 7f57f2200700 1 -- 172.21.15.78:0/4242406131 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57ec1a4730 con 0x7f57ec100f30 2024-03-14T09:56:14.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.720+0000 7f57e3e00700 1 -- 172.21.15.78:0/4242406131 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f57dc0105a0 con 0x7f57ec100f30 2024-03-14T09:56:14.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.722+0000 7f5805cceb80 1 Processor -- start 2024-03-14T09:56:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.722+0000 7f5805cceb80 1 -- start start 2024-03-14T09:56:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.723+0000 7f57e3e00700 1 --2- 172.21.15.78:0/4242406131 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f57d00a0c80 0x7f57d00a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.723+0000 7f57e3e00700 1 -- 172.21.15.78:0/4242406131 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f57dc014070 con 0x7f57ec100f30 2024-03-14T09:56:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.723+0000 7f5805cceb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d674764460 0x55d674762680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.723+0000 7f5805cceb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674762bc0 0x55d674763040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.723+0000 7f5805cceb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674775560 0x55d674777950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.723+0000 7f5805cceb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6746bb990 con 0x55d674764460 2024-03-14T09:56:14.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.723+0000 7f5805cceb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6746b9350 con 0x55d674762bc0 2024-03-14T09:56:14.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.723+0000 7f5805cceb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6746b4c40 con 0x55d674775560 2024-03-14T09:56:14.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.723+0000 7f57eb400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d674764460 0x55d674762680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.723+0000 7f57eaa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674762bc0 0x55d674763040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.723+0000 7f57eaa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674762bc0 0x55d674763040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:56364/0 (socket says 172.21.15.78:56364) 2024-03-14T09:56:14.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.723+0000 7f57eaa00700 1 -- 172.21.15.78:0/1240389246 learned_addr learned my addr 172.21.15.78:0/1240389246 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:14.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.723+0000 7f57ebe00700 1 --2- 172.21.15.78:0/4242406131 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f57d00a0c80 0x7f57d00a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.724+0000 7f57eaa00700 1 -- 172.21.15.78:0/1240389246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674775560 msgr2=0x55d674777950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:14.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.724+0000 7f57eaa00700 1 --2- 172.21.15.78:0/1240389246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674775560 0x55d674777950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.724+0000 7f57eaa00700 1 -- 172.21.15.78:0/1240389246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d674764460 msgr2=0x55d674762680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:14.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.724+0000 7f57eaa00700 1 --2- 172.21.15.78:0/1240389246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d674764460 0x55d674762680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.724+0000 7f57eaa00700 1 -- 172.21.15.78:0/1240389246 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6745363b0 con 0x55d674762bc0 2024-03-14T09:56:14.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.724+0000 7f57eaa00700 1 --2- 172.21.15.78:0/1240389246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674762bc0 0x55d674763040 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f57d8009990 tx=0x7f57d8005bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c63325bb7736a6e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:14.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.724+0000 7f57d7e00700 1 -- 172.21.15.78:0/1240389246 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d801d070 con 0x55d674762bc0 2024-03-14T09:56:14.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.724+0000 7f57d7e00700 1 -- 172.21.15.78:0/1240389246 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57d801e040 con 0x55d674762bc0 2024-03-14T09:56:14.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.724+0000 7f57d7e00700 1 -- 172.21.15.78:0/1240389246 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d8004920 con 0x55d674762bc0 2024-03-14T09:56:14.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.724+0000 7f57ebe00700 1 --2- 172.21.15.78:0/4242406131 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f57d00a0c80 0x7f57d00a3140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f57e4003f70 tx=0x7f57e400b040 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:14.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.725+0000 7f5805cceb80 1 -- 172.21.15.78:0/1240389246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674762bc0 msgr2=0x55d674763040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:14.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.725+0000 7f5805cceb80 1 --2- 172.21.15.78:0/1240389246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674762bc0 0x55d674763040 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f57d8009990 tx=0x7f57d8005bc0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.726+0000 7f5805cceb80 1 -- 172.21.15.78:0/1240389246 shutdown_connections 2024-03-14T09:56:14.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.726+0000 7f5805cceb80 1 --2- 172.21.15.78:0/1240389246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674775560 0x55d674777950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.726+0000 7f5805cceb80 1 --2- 172.21.15.78:0/1240389246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d674764460 0x55d674762680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.726+0000 7f5805cceb80 1 --2- 172.21.15.78:0/1240389246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674762bc0 0x55d674763040 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.726+0000 7f5805cceb80 1 -- 172.21.15.78:0/1240389246 >> 172.21.15.78:0/1240389246 conn(0x55d6746d8220 msgr2=0x55d67475daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:14.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.726+0000 7f5805cceb80 1 -- 172.21.15.78:0/1240389246 shutdown_connections 2024-03-14T09:56:14.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.726+0000 7f5805cceb80 1 -- 172.21.15.78:0/1240389246 wait complete. 2024-03-14T09:56:14.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.727+0000 7f5805cceb80 1 -- start start 2024-03-14T09:56:14.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.727+0000 7f5805cceb80 1 Processor -- start 2024-03-14T09:56:14.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.727+0000 7f5805cceb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674762bc0 0x55d674768490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.727+0000 7f5805cceb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674764460 0x55d674765780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.727+0000 7f5805cceb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d674775560 0x55d674765ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.728+0000 7f5805cceb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d674650d80 con 0x55d674775560 2024-03-14T09:56:14.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.728+0000 7f5805cceb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d674650a20 con 0x55d674762bc0 2024-03-14T09:56:14.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.728+0000 7f5805cceb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6745386a0 con 0x55d674764460 2024-03-14T09:56:14.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.728+0000 7f5805cceb80 10 client.0 _ll_register_callbacks cb 0x55d6744bc3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:14.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.728+0000 7f57ebe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d674775560 0x55d674765ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.728+0000 7f57eb400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674762bc0 0x55d674768490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.728+0000 7f57eb400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674762bc0 0x55d674768490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:56374/0 (socket says 172.21.15.78:56374) 2024-03-14T09:56:14.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.728+0000 7f57eb400700 1 -- 172.21.15.78:0/772822864 learned_addr learned my addr 172.21.15.78:0/772822864 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:14.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.728+0000 7f57eaa00700 1 --2- 172.21.15.78:0/772822864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674764460 0x55d674765780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.728+0000 7f57eb400700 1 -- 172.21.15.78:0/772822864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674764460 msgr2=0x55d674765780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:14.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.728+0000 7f57eb400700 1 --2- 172.21.15.78:0/772822864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674764460 0x55d674765780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.728+0000 7f57eb400700 1 -- 172.21.15.78:0/772822864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d674775560 msgr2=0x55d674765ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:14.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.728+0000 7f57eb400700 1 --2- 172.21.15.78:0/772822864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d674775560 0x55d674765ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.728+0000 7f57eb400700 1 -- 172.21.15.78:0/772822864 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57ec10e440 con 0x55d674762bc0 2024-03-14T09:56:14.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.728+0000 7f57eb400700 1 --2- 172.21.15.78:0/772822864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674762bc0 0x55d674768490 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f57dc095dd0 tx=0x7f57dc0998d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4f920454b8ab202 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:14.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.728+0000 7f57d4c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:56:14.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.729+0000 7f57d4c00700 1 -- 172.21.15.78:0/772822864 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc098fc0 con 0x55d674762bc0 2024-03-14T09:56:14.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.729+0000 7f57d4c00700 1 -- 172.21.15.78:0/772822864 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f57dc0a1d50 con 0x55d674762bc0 2024-03-14T09:56:14.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.729+0000 7f5805cceb80 1 -- 172.21.15.78:0/772822864 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d674755720 con 0x55d674762bc0 2024-03-14T09:56:14.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.729+0000 7f57d4c00700 1 -- 172.21.15.78:0/772822864 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57dc0b2200 con 0x55d674762bc0 2024-03-14T09:56:14.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.729+0000 7f5805cceb80 10 client.26081 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:14.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.729+0000 7f5805cceb80 1 -- 172.21.15.78:0/772822864 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d6744e2b60 con 0x55d674762bc0 2024-03-14T09:56:14.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.730+0000 7f57d4c00700 1 -- 172.21.15.78:0/772822864 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d674755720 con 0x55d674762bc0 2024-03-14T09:56:14.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.730+0000 7f57d4c00700 1 -- 172.21.15.78:0/772822864 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f57dc0b9200 con 0x55d674762bc0 2024-03-14T09:56:14.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.730+0000 7f5805cceb80 10 client.26081 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:14.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.730+0000 7f5805cceb80 10 client.26081 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:56:14.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.730+0000 7f5805cceb80 20 client.26081 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:14.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.730+0000 7f5805cceb80 1 --2- 172.21.15.78:0/772822864 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d674785290 0x55d674787680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.730+0000 7f5805cceb80 4 client.26081 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:56:14.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.730+0000 7f5805cceb80 1 -- 172.21.15.78:0/772822864 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d6746ad700 con 0x55d674785290 2024-03-14T09:56:14.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.730+0000 7f57eaa00700 1 --2- 172.21.15.78:0/772822864 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d674785290 0x55d674787680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.731+0000 7f57d4c00700 10 client.26081 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:56:14.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.731+0000 7f57eaa00700 1 --2- 172.21.15.78:0/772822864 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d674785290 0x55d674787680 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=45d72f184d4e332c server_cookie=eca4f4eed108def6 in_seq=0 out_seq=0 2024-03-14T09:56:14.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.732+0000 7f57d4c00700 1 -- 172.21.15.78:0/772822864 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55d6746ad700 con 0x55d674785290 2024-03-14T09:56:14.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.732+0000 7f57d4c00700 10 client.26081 handle_command_reply: tid=0 2024-03-14T09:56:14.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.732+0000 7f5805cceb80 1 client.26081 shutdown 2024-03-14T09:56:14.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.733+0000 7f5805cceb80 1 -- 172.21.15.78:0/772822864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674762bc0 msgr2=0x55d674768490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:14.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.733+0000 7f5805cceb80 1 --2- 172.21.15.78:0/772822864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674762bc0 0x55d674768490 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f57dc095dd0 tx=0x7f57dc0998d0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.733+0000 7f5805cceb80 1 -- 172.21.15.78:0/772822864 shutdown_connections 2024-03-14T09:56:14.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.733+0000 7f5805cceb80 1 --2- 172.21.15.78:0/772822864 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d674785290 0x55d674787680 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.733+0000 7f5805cceb80 1 --2- 172.21.15.78:0/772822864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674762bc0 0x55d674768490 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.733+0000 7f5805cceb80 1 --2- 172.21.15.78:0/772822864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d674775560 0x55d674765ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.733+0000 7f5805cceb80 1 --2- 172.21.15.78:0/772822864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674764460 0x55d674765780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.733+0000 7f5805cceb80 1 -- 172.21.15.78:0/772822864 >> 172.21.15.78:0/772822864 conn(0x55d6746d8220 msgr2=0x55d67475ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:14.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.733+0000 7f57d4c00700 0 client.26081 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:56:14.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.733+0000 7f5805cceb80 1 -- 172.21.15.78:0/772822864 shutdown_connections 2024-03-14T09:56:14.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.733+0000 7f5805cceb80 1 -- 172.21.15.78:0/772822864 wait complete. 2024-03-14T09:56:14.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.733+0000 7f5805cceb80 20 client.26081 trim_cache size 0 max 16384 2024-03-14T09:56:14.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.752+0000 7f5805cceb80 1 Processor -- start 2024-03-14T09:56:14.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.752+0000 7f5805cceb80 1 -- start start 2024-03-14T09:56:14.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.753+0000 7f5805cceb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d674762bc0 0x55d67475dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.753+0000 7f5805cceb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674764460 0x55d67477c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.753+0000 7f5805cceb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674775560 0x55d674771840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.753+0000 7f5805cceb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6747664c0 con 0x55d674762bc0 2024-03-14T09:56:14.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.753+0000 7f5805cceb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d67475e770 con 0x55d674775560 2024-03-14T09:56:14.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.753+0000 7f5805cceb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d674755260 con 0x55d674764460 2024-03-14T09:56:14.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.753+0000 7f57eaa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674764460 0x55d67477c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.753+0000 7f57eaa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674764460 0x55d67477c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:55008/0 (socket says 172.21.15.78:55008) 2024-03-14T09:56:14.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.753+0000 7f57eaa00700 1 -- 172.21.15.78:0/3428146529 learned_addr learned my addr 172.21.15.78:0/3428146529 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:14.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.753+0000 7f57eb400700 1 --2- 172.21.15.78:0/3428146529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d674762bc0 0x55d67475dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.753+0000 7f57ebe00700 1 --2- 172.21.15.78:0/3428146529 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674775560 0x55d674771840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.754+0000 7f57eb400700 1 -- 172.21.15.78:0/3428146529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674764460 msgr2=0x55d67477c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:14.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.754+0000 7f57eb400700 1 --2- 172.21.15.78:0/3428146529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674764460 0x55d67477c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.754+0000 7f57eb400700 1 -- 172.21.15.78:0/3428146529 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674775560 msgr2=0x55d674771840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:14.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.754+0000 7f57eb400700 1 --2- 172.21.15.78:0/3428146529 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674775560 0x55d674771840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.754+0000 7f57eb400700 1 -- 172.21.15.78:0/3428146529 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d67467aea0 con 0x55d674762bc0 2024-03-14T09:56:14.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.754+0000 7f57ebe00700 1 --2- 172.21.15.78:0/3428146529 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674775560 0x55d674771840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:56:14.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.754+0000 7f57eb400700 1 --2- 172.21.15.78:0/3428146529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d674762bc0 0x55d67475dc30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f57dc0a1820 tx=0x7f57dc010a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7386b609d4197ba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:14.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.754+0000 7f57e1200700 1 -- 172.21.15.78:0/3428146529 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc0a1ac0 con 0x55d674762bc0 2024-03-14T09:56:14.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.754+0000 7f57e1200700 1 -- 172.21.15.78:0/3428146529 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57dc0996a0 con 0x55d674762bc0 2024-03-14T09:56:14.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.754+0000 7f57e1200700 1 -- 172.21.15.78:0/3428146529 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc099020 con 0x55d674762bc0 2024-03-14T09:56:14.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.755+0000 7f5805cceb80 1 -- 172.21.15.78:0/3428146529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d674762bc0 msgr2=0x55d67475dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:14.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.755+0000 7f5805cceb80 1 --2- 172.21.15.78:0/3428146529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d674762bc0 0x55d67475dc30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f57dc0a1820 tx=0x7f57dc010a90 comp rx=0 tx=0).stop 2024-03-14T09:56:14.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.756+0000 7f5805cceb80 1 -- 172.21.15.78:0/3428146529 shutdown_connections 2024-03-14T09:56:14.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.756+0000 7f5805cceb80 1 --2- 172.21.15.78:0/3428146529 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674775560 0x55d674771840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.756+0000 7f5805cceb80 1 --2- 172.21.15.78:0/3428146529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d674762bc0 0x55d67475dc30 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.756+0000 7f5805cceb80 1 --2- 172.21.15.78:0/3428146529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674764460 0x55d67477c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.756+0000 7f5805cceb80 1 -- 172.21.15.78:0/3428146529 >> 172.21.15.78:0/3428146529 conn(0x55d6746d8220 msgr2=0x55d6747872a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:14.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.756+0000 7f5805cceb80 1 -- 172.21.15.78:0/3428146529 shutdown_connections 2024-03-14T09:56:14.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.756+0000 7f5805cceb80 1 -- 172.21.15.78:0/3428146529 wait complete. 2024-03-14T09:56:14.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.756+0000 7f5805cceb80 1 -- start start 2024-03-14T09:56:14.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.756+0000 7f5805cceb80 1 Processor -- start 2024-03-14T09:56:14.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.757+0000 7f5805cceb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674764460 0x55d67477a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.757+0000 7f5805cceb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674775560 0x55d67477a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.757+0000 7f5805cceb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d67477ad60 0x55d6747d5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.757+0000 7f5805cceb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d67477c4f0 con 0x55d67477ad60 2024-03-14T09:56:14.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.757+0000 7f5805cceb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d67477c630 con 0x55d674775560 2024-03-14T09:56:14.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.757+0000 7f5805cceb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6747d62c0 con 0x55d674764460 2024-03-14T09:56:14.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.757+0000 7f5805cceb80 10 client.0 _ll_register_callbacks cb 0x7f57dc099250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:14.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.758+0000 7f57ebe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d67477ad60 0x55d6747d5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.758+0000 7f57ebe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d67477ad60 0x55d6747d5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41072/0 (socket says 172.21.15.78:41072) 2024-03-14T09:56:14.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.758+0000 7f57ebe00700 1 -- 172.21.15.78:0/617600328 learned_addr learned my addr 172.21.15.78:0/617600328 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:14.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.758+0000 7f57eb400700 1 --2- 172.21.15.78:0/617600328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674764460 0x55d67477a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.758+0000 7f57ebe00700 1 -- 172.21.15.78:0/617600328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674764460 msgr2=0x55d67477a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:14.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.758+0000 7f57ebe00700 1 --2- 172.21.15.78:0/617600328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674764460 0x55d67477a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.758+0000 7f57ebe00700 1 -- 172.21.15.78:0/617600328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674775560 msgr2=0x55d67477a820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-14T09:56:14.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.758+0000 7f57ebe00700 1 --2- 172.21.15.78:0/617600328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674775560 0x55d67477a820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.758+0000 7f57ebe00700 1 -- 172.21.15.78:0/617600328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e4011040 con 0x55d67477ad60 2024-03-14T09:56:14.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.758+0000 7f57ebe00700 1 --2- 172.21.15.78:0/617600328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d67477ad60 0x55d6747d5d80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f57e40077b0 tx=0x7f57e4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64561dc112200bf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:14.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.759+0000 7f57d4c00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:56:14.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.759+0000 7f57d4c00700 1 -- 172.21.15.78:0/617600328 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e4029070 con 0x55d67477ad60 2024-03-14T09:56:14.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.759+0000 7f5805cceb80 1 -- 172.21.15.78:0/617600328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57d801e040 con 0x55d67477ad60 2024-03-14T09:56:14.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.759+0000 7f57d4c00700 1 -- 172.21.15.78:0/617600328 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f57e4004bb0 con 0x55d67477ad60 2024-03-14T09:56:14.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.759+0000 7f57d4c00700 1 -- 172.21.15.78:0/617600328 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57e4004d70 con 0x55d67477ad60 2024-03-14T09:56:14.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.759+0000 7f5805cceb80 10 client.16077 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:14.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.759+0000 7f5805cceb80 1 -- 172.21.15.78:0/617600328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d6744e2b60 con 0x55d67477ad60 2024-03-14T09:56:14.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.759+0000 7f57d4c00700 1 -- 172.21.15.78:0/617600328 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57d801e040 con 0x55d67477ad60 2024-03-14T09:56:14.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.760+0000 7f57d4c00700 1 -- 172.21.15.78:0/617600328 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f57e402c020 con 0x55d67477ad60 2024-03-14T09:56:14.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.760+0000 7f5805cceb80 10 client.16077 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:14.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.760+0000 7f5805cceb80 10 client.16077 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:56:14.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.760+0000 7f5805cceb80 20 client.16077 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:14.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.760+0000 7f5805cceb80 1 --2- 172.21.15.78:0/617600328 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d6747d7ab0 0x55d6747d9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:14.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.760+0000 7f5805cceb80 4 client.16077 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:56:14.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.760+0000 7f5805cceb80 1 -- 172.21.15.78:0/617600328 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d674503cf0 con 0x55d6747d7ab0 2024-03-14T09:56:14.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.761+0000 7f57eb400700 1 --2- 172.21.15.78:0/617600328 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d6747d7ab0 0x55d6747d9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:14.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.761+0000 7f57eb400700 1 --2- 172.21.15.78:0/617600328 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d6747d7ab0 0x55d6747d9ed0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b3db8657d336a31e server_cookie=69feb45af9ec6846 in_seq=0 out_seq=0 2024-03-14T09:56:14.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.761+0000 7f57d4c00700 10 client.16077 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:56:14.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.762+0000 7f57d4c00700 1 -- 172.21.15.78:0/617600328 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d674503cf0 con 0x55d6747d7ab0 2024-03-14T09:56:14.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.762+0000 7f57d4c00700 10 client.16077 handle_command_reply: tid=0 2024-03-14T09:56:14.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.762+0000 7f5805cceb80 1 client.16077 shutdown 2024-03-14T09:56:14.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.762+0000 7f5805cceb80 1 -- 172.21.15.78:0/617600328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d67477ad60 msgr2=0x55d6747d5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:14.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.762+0000 7f5805cceb80 1 --2- 172.21.15.78:0/617600328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d67477ad60 0x55d6747d5d80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f57e40077b0 tx=0x7f57e4004ad0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.762+0000 7f5805cceb80 1 -- 172.21.15.78:0/617600328 shutdown_connections 2024-03-14T09:56:14.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.762+0000 7f5805cceb80 1 --2- 172.21.15.78:0/617600328 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d6747d7ab0 0x55d6747d9ed0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.762+0000 7f5805cceb80 1 --2- 172.21.15.78:0/617600328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d67477ad60 0x55d6747d5d80 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.762+0000 7f5805cceb80 1 --2- 172.21.15.78:0/617600328 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d674764460 0x55d67477a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.762+0000 7f57d4c00700 0 client.16077 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:56:14.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.762+0000 7f5805cceb80 1 --2- 172.21.15.78:0/617600328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d674775560 0x55d67477a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.762+0000 7f5805cceb80 1 -- 172.21.15.78:0/617600328 >> 172.21.15.78:0/617600328 conn(0x55d6746d8220 msgr2=0x55d674771840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:14.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.762+0000 7f5805cceb80 1 -- 172.21.15.78:0/617600328 shutdown_connections 2024-03-14T09:56:14.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.762+0000 7f5805cceb80 1 -- 172.21.15.78:0/617600328 wait complete. 2024-03-14T09:56:14.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.763+0000 7f5805cceb80 20 client.16077 trim_cache size 0 max 16384 2024-03-14T09:56:14.764 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:56:14.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.763+0000 7f57f2200700 1 -- 172.21.15.78:0/4242406131 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f57d00a0c80 msgr2=0x7f57d00a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:14.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.763+0000 7f57f2200700 1 --2- 172.21.15.78:0/4242406131 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f57d00a0c80 0x7f57d00a3140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f57e4003f70 tx=0x7f57e400b040 comp rx=0 tx=0).stop 2024-03-14T09:56:14.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.763+0000 7f57f2200700 1 -- 172.21.15.78:0/4242406131 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57ec100f30 msgr2=0x7f57ec1976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:14.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.763+0000 7f57f2200700 1 --2- 172.21.15.78:0/4242406131 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57ec100f30 0x7f57ec1976b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00e9a0 tx=0x7f57dc00ecb0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.763+0000 7f57f2200700 1 -- 172.21.15.78:0/4242406131 shutdown_connections 2024-03-14T09:56:14.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.763+0000 7f57f2200700 1 --2- 172.21.15.78:0/4242406131 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f57d00a0c80 0x7f57d00a3140 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.763+0000 7f57f2200700 1 --2- 172.21.15.78:0/4242406131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57ec19bf40 0x7f57ec19c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.763+0000 7f57f2200700 1 --2- 172.21.15.78:0/4242406131 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f57ec100f30 0x7f57ec1976b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.763+0000 7f57f2200700 1 --2- 172.21.15.78:0/4242406131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57ec106050 0x7f57ec197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:14.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.763+0000 7f57f2200700 1 -- 172.21.15.78:0/4242406131 >> 172.21.15.78:0/4242406131 conn(0x7f57ec0fc7b0 msgr2=0x7f57ec0fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:14.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.763+0000 7f57f2200700 1 -- 172.21.15.78:0/4242406131 shutdown_connections 2024-03-14T09:56:14.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:14.764+0000 7f57f2200700 1 -- 172.21.15.78:0/4242406131 wait complete. 2024-03-14T09:56:14.786 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T09:56:15.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.036+0000 7f4f80e00700 1 -- 172.21.15.78:0/159683089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f7c068a60 msgr2=0x7f4f7c105930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.036+0000 7f4f80e00700 1 --2- 172.21.15.78:0/159683089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f7c068a60 0x7f4f7c105930 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4f68009990 tx=0x7f4f68009ca0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.036+0000 7f4f80e00700 1 -- 172.21.15.78:0/159683089 shutdown_connections 2024-03-14T09:56:15.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.036+0000 7f4f80e00700 1 --2- 172.21.15.78:0/159683089 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f7c106060 0x7f4f7c10dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.036+0000 7f4f80e00700 1 --2- 172.21.15.78:0/159683089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f7c0680b0 0x7f4f7c068490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.036+0000 7f4f80e00700 1 --2- 172.21.15.78:0/159683089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f7c068a60 0x7f4f7c105930 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.036+0000 7f4f80e00700 1 -- 172.21.15.78:0/159683089 >> 172.21.15.78:0/159683089 conn(0x7f4f7c076d80 msgr2=0x7f4f7c077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:15.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.036+0000 7f4f80e00700 1 -- 172.21.15.78:0/159683089 shutdown_connections 2024-03-14T09:56:15.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.036+0000 7f4f80e00700 1 -- 172.21.15.78:0/159683089 wait complete. 2024-03-14T09:56:15.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.037+0000 7f4f80e00700 1 Processor -- start 2024-03-14T09:56:15.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.038+0000 7f4f80e00700 1 -- start start 2024-03-14T09:56:15.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.038+0000 7f4f80e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f7c0680b0 0x7f4f7c10bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.038+0000 7f4f80e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f7c106060 0x7f4f7c10c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.038+0000 7f4f80e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f7c1a24a0 0x7f4f7c1a2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.038+0000 7f4f80e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c1a2f20 con 0x7f4f7c106060 2024-03-14T09:56:15.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.038+0000 7f4f80e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c1a3090 con 0x7f4f7c0680b0 2024-03-14T09:56:15.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.038+0000 7f4f80e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c1a3250 con 0x7f4f7c1a24a0 2024-03-14T09:56:15.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.039+0000 7f4f79e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f7c0680b0 0x7f4f7c10bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.039+0000 7f4f79e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f7c0680b0 0x7f4f7c10bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:56410/0 (socket says 172.21.15.78:56410) 2024-03-14T09:56:15.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.039+0000 7f4f79e00700 1 -- 172.21.15.78:0/814564672 learned_addr learned my addr 172.21.15.78:0/814564672 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:15.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.039+0000 7f4f79400700 1 --2- 172.21.15.78:0/814564672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f7c106060 0x7f4f7c10c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.039+0000 7f4f7a800700 1 --2- 172.21.15.78:0/814564672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f7c1a24a0 0x7f4f7c1a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.039+0000 7f4f79e00700 1 -- 172.21.15.78:0/814564672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f7c1a24a0 msgr2=0x7f4f7c1a2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.039+0000 7f4f79e00700 1 --2- 172.21.15.78:0/814564672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f7c1a24a0 0x7f4f7c1a2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.039+0000 7f4f79e00700 1 -- 172.21.15.78:0/814564672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f7c106060 msgr2=0x7f4f7c10c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.039+0000 7f4f79e00700 1 --2- 172.21.15.78:0/814564672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f7c106060 0x7f4f7c10c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.039+0000 7f4f79e00700 1 -- 172.21.15.78:0/814564672 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6c009520 con 0x7f4f7c0680b0 2024-03-14T09:56:15.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.039+0000 7f4f79e00700 1 --2- 172.21.15.78:0/814564672 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f7c0680b0 0x7f4f7c10bc70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c00eb30 tx=0x7f4f6c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:15.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.041+0000 7f4f72a00700 1 -- 172.21.15.78:0/814564672 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c009880 con 0x7f4f7c0680b0 2024-03-14T09:56:15.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.041+0000 7f4f72a00700 1 -- 172.21.15.78:0/814564672 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f6c0044e0 con 0x7f4f7c0680b0 2024-03-14T09:56:15.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.041+0000 7f4f80e00700 1 -- 172.21.15.78:0/814564672 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f680095f0 con 0x7f4f7c0680b0 2024-03-14T09:56:15.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.041+0000 7f4f72a00700 1 -- 172.21.15.78:0/814564672 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c003e90 con 0x7f4f7c0680b0 2024-03-14T09:56:15.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.041+0000 7f4f80e00700 1 -- 172.21.15.78:0/814564672 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7c1aab50 con 0x7f4f7c0680b0 2024-03-14T09:56:15.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.045+0000 7f4f72a00700 1 -- 172.21.15.78:0/814564672 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f4f6c004020 con 0x7f4f7c0680b0 2024-03-14T09:56:15.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.046+0000 7f4f72a00700 1 --2- 172.21.15.78:0/814564672 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4f600a0c80 0x7f4f600a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.046+0000 7f4f72a00700 1 -- 172.21.15.78:0/814564672 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f4f6c014070 con 0x7f4f7c0680b0 2024-03-14T09:56:15.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.046+0000 7f4f79400700 1 --2- 172.21.15.78:0/814564672 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4f600a0c80 0x7f4f600a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.048+0000 7f4f949aeb80 1 Processor -- start 2024-03-14T09:56:15.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.048+0000 7f4f949aeb80 1 -- start start 2024-03-14T09:56:15.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.048+0000 7f4f949aeb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad8205a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.048+0000 7f4f949aeb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad8205fa0 0x55fad820f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.048+0000 7f4f949aeb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad82122b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.048+0000 7f4f949aeb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fad815e990 con 0x55fad8205fa0 2024-03-14T09:56:15.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.048+0000 7f4f949aeb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fad815c350 con 0x55fad8205680 2024-03-14T09:56:15.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.048+0000 7f4f949aeb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fad8157c40 con 0x55fad820fec0 2024-03-14T09:56:15.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.049+0000 7f4f7a800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad82122b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.049+0000 7f4f7a800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad82122b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:55054/0 (socket says 172.21.15.78:55054) 2024-03-14T09:56:15.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.049+0000 7f4f79400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad8205fa0 0x55fad820f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.049+0000 7f4f7a800700 1 -- 172.21.15.78:0/843198823 learned_addr learned my addr 172.21.15.78:0/843198823 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:15.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.049+0000 7f4f79400700 1 --2- 172.21.15.78:0/814564672 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4f600a0c80 0x7f4f600a3140 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4f68006010 tx=0x7f4f68005960 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.049+0000 7f4f79e00700 1 --2- 172.21.15.78:0/843198823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad8205a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.049+0000 7f4f7a800700 1 -- 172.21.15.78:0/843198823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 msgr2=0x55fad8205a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.049+0000 7f4f7a800700 1 --2- 172.21.15.78:0/843198823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad8205a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.049+0000 7f4f7a800700 1 -- 172.21.15.78:0/843198823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad8205fa0 msgr2=0x55fad820f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.049+0000 7f4f7a800700 1 --2- 172.21.15.78:0/843198823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad8205fa0 0x55fad820f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.049+0000 7f4f7a800700 1 -- 172.21.15.78:0/843198823 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fad7fd93b0 con 0x55fad820fec0 2024-03-14T09:56:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.049+0000 7f4f7a800700 1 --2- 172.21.15.78:0/843198823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad82122b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4f74009d20 tx=0x7f4f740077e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=192ccbfbb3554652 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.050+0000 7f4f67400700 1 -- 172.21.15.78:0/843198823 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74018070 con 0x55fad820fec0 2024-03-14T09:56:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.050+0000 7f4f67400700 1 -- 172.21.15.78:0/843198823 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f7400f050 con 0x55fad820fec0 2024-03-14T09:56:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.050+0000 7f4f67400700 1 -- 172.21.15.78:0/843198823 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74014410 con 0x55fad820fec0 2024-03-14T09:56:15.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.051+0000 7f4f949aeb80 1 -- 172.21.15.78:0/843198823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 msgr2=0x55fad82122b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.051+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/843198823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad82122b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4f74009d20 tx=0x7f4f740077e0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.051+0000 7f4f949aeb80 1 -- 172.21.15.78:0/843198823 shutdown_connections 2024-03-14T09:56:15.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.051+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/843198823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad8205a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.051+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/843198823 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad82122b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.051+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/843198823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad8205fa0 0x55fad820f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.051+0000 7f4f949aeb80 1 -- 172.21.15.78:0/843198823 >> 172.21.15.78:0/843198823 conn(0x55fad817b220 msgr2=0x55fad8200aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:15.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.051+0000 7f4f949aeb80 1 -- 172.21.15.78:0/843198823 shutdown_connections 2024-03-14T09:56:15.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.051+0000 7f4f949aeb80 1 -- 172.21.15.78:0/843198823 wait complete. 2024-03-14T09:56:15.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.051+0000 7f4f949aeb80 1 -- start start 2024-03-14T09:56:15.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.052+0000 7f4f949aeb80 1 Processor -- start 2024-03-14T09:56:15.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.052+0000 7f4f949aeb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad820f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.052+0000 7f4f949aeb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad8205fa0 0x55fad820cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.053+0000 7f4f949aeb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad820d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.053+0000 7f4f949aeb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fad80f3d80 con 0x55fad8205fa0 2024-03-14T09:56:15.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.053+0000 7f4f949aeb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fad80f3a20 con 0x55fad8205680 2024-03-14T09:56:15.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.053+0000 7f4f949aeb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fad7fdb6a0 con 0x55fad820fec0 2024-03-14T09:56:15.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.053+0000 7f4f79e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad820f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.053+0000 7f4f949aeb80 10 client.0 _ll_register_callbacks cb 0x55fad7f5f3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:15.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.053+0000 7f4f7a800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad820d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.053+0000 7f4f79400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad8205fa0 0x55fad820cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.053+0000 7f4f79e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad820f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:56438/0 (socket says 172.21.15.78:56438) 2024-03-14T09:56:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.053+0000 7f4f79e00700 1 -- 172.21.15.78:0/130077018 learned_addr learned my addr 172.21.15.78:0/130077018 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.054+0000 7f4f79e00700 1 -- 172.21.15.78:0/130077018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 msgr2=0x55fad820d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.054+0000 7f4f79e00700 1 --2- 172.21.15.78:0/130077018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad820d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.054+0000 7f4f79e00700 1 -- 172.21.15.78:0/130077018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad8205fa0 msgr2=0x55fad820cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.054+0000 7f4f79e00700 1 --2- 172.21.15.78:0/130077018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad8205fa0 0x55fad820cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.054+0000 7f4f79e00700 1 -- 172.21.15.78:0/130077018 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f74009980 con 0x55fad8205680 2024-03-14T09:56:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.054+0000 7f4f79e00700 1 --2- 172.21.15.78:0/130077018 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad820f8a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c096050 tx=0x7f4f6c016330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff8fe3c00126edc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.054+0000 7f4f5be00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:56:15.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.055+0000 7f4f5be00700 1 -- 172.21.15.78:0/130077018 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c099fb0 con 0x55fad8205680 2024-03-14T09:56:15.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.055+0000 7f4f5be00700 1 -- 172.21.15.78:0/130077018 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f4f6c00cc90 con 0x55fad8205680 2024-03-14T09:56:15.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.055+0000 7f4f949aeb80 1 -- 172.21.15.78:0/130077018 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fad81f8720 con 0x55fad8205680 2024-03-14T09:56:15.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.055+0000 7f4f5be00700 1 -- 172.21.15.78:0/130077018 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f6c099b50 con 0x55fad8205680 2024-03-14T09:56:15.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.055+0000 7f4f949aeb80 10 client.26093 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:15.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.055+0000 7f4f949aeb80 1 -- 172.21.15.78:0/130077018 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55fad7f85b60 con 0x55fad8205680 2024-03-14T09:56:15.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.055+0000 7f4f5be00700 1 -- 172.21.15.78:0/130077018 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f6c07c080 con 0x55fad8205680 2024-03-14T09:56:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.056+0000 7f4f5be00700 1 -- 172.21.15.78:0/130077018 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4f6c0a1b40 con 0x55fad8205680 2024-03-14T09:56:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.056+0000 7f4f949aeb80 10 client.26093 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.056+0000 7f4f949aeb80 10 client.26093 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:56:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.056+0000 7f4f949aeb80 20 client.26093 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.056+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/130077018 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55fad821f990 0x55fad8221db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.056+0000 7f4f949aeb80 4 client.26093 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:56:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.056+0000 7f4f949aeb80 1 -- 172.21.15.78:0/130077018 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fad8150700 con 0x55fad821f990 2024-03-14T09:56:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.056+0000 7f4f79400700 1 --2- 172.21.15.78:0/130077018 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55fad821f990 0x55fad8221db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.057+0000 7f4f79400700 1 --2- 172.21.15.78:0/130077018 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55fad821f990 0x55fad8221db0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e4e291a49a0903f7 server_cookie=495506b235025745 in_seq=0 out_seq=0 2024-03-14T09:56:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.057+0000 7f4f5be00700 10 client.26093 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:56:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.058+0000 7f4f5be00700 1 -- 172.21.15.78:0/130077018 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55fad8150700 con 0x55fad821f990 2024-03-14T09:56:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.058+0000 7f4f5be00700 10 client.26093 handle_command_reply: tid=0 2024-03-14T09:56:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.058+0000 7f4f949aeb80 1 client.26093 shutdown 2024-03-14T09:56:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.058+0000 7f4f949aeb80 1 -- 172.21.15.78:0/130077018 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 msgr2=0x55fad820f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.058+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/130077018 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad820f8a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c096050 tx=0x7f4f6c016330 comp rx=0 tx=0).stop 2024-03-14T09:56:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.058+0000 7f4f949aeb80 1 -- 172.21.15.78:0/130077018 shutdown_connections 2024-03-14T09:56:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.058+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/130077018 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55fad821f990 0x55fad8221db0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.058+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/130077018 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad820f8a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.058+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/130077018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad820d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.058+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/130077018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad8205fa0 0x55fad820cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.058+0000 7f4f949aeb80 1 -- 172.21.15.78:0/130077018 >> 172.21.15.78:0/130077018 conn(0x55fad817b220 msgr2=0x55fad8200690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.058+0000 7f4f5be00700 0 client.26093 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:56:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.059+0000 7f4f949aeb80 1 -- 172.21.15.78:0/130077018 shutdown_connections 2024-03-14T09:56:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.059+0000 7f4f949aeb80 1 -- 172.21.15.78:0/130077018 wait complete. 2024-03-14T09:56:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.059+0000 7f4f949aeb80 20 client.26093 trim_cache size 0 max 16384 2024-03-14T09:56:15.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.077+0000 7f4f949aeb80 1 Processor -- start 2024-03-14T09:56:15.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.077+0000 7f4f949aeb80 1 -- start start 2024-03-14T09:56:15.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.077+0000 7f4f949aeb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad8200c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.077+0000 7f4f949aeb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad8205fa0 0x55fad8216b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.077+0000 7f4f949aeb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad820b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.077+0000 7f4f949aeb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fad8201770 con 0x55fad8205fa0 2024-03-14T09:56:15.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.077+0000 7f4f949aeb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fad8202060 con 0x55fad8205680 2024-03-14T09:56:15.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.077+0000 7f4f949aeb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fad81fad00 con 0x55fad820fec0 2024-03-14T09:56:15.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.078+0000 7f4f79e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad8200c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.078+0000 7f4f79400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad8205fa0 0x55fad8216b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.078+0000 7f4f79e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad8200c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:56442/0 (socket says 172.21.15.78:56442) 2024-03-14T09:56:15.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.078+0000 7f4f79e00700 1 -- 172.21.15.78:0/1159653877 learned_addr learned my addr 172.21.15.78:0/1159653877 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:15.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.078+0000 7f4f7a800700 1 --2- 172.21.15.78:0/1159653877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad820b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.078+0000 7f4f79400700 1 -- 172.21.15.78:0/1159653877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 msgr2=0x55fad820b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:15.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.078+0000 7f4f79400700 1 --2- 172.21.15.78:0/1159653877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad820b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.078+0000 7f4f79400700 1 -- 172.21.15.78:0/1159653877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 msgr2=0x55fad8200c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:15.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.078+0000 7f4f79400700 1 --2- 172.21.15.78:0/1159653877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad8200c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.078+0000 7f4f79400700 1 -- 172.21.15.78:0/1159653877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fad811dea0 con 0x55fad8205fa0 2024-03-14T09:56:15.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.078+0000 7f4f79e00700 1 --2- 172.21.15.78:0/1159653877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad8200c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:56:15.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.079+0000 7f4f79400700 1 --2- 172.21.15.78:0/1159653877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad8205fa0 0x55fad8216b40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4f680098e0 tx=0x7f4f68020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45f49bba649ad07 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:15.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.079+0000 7f4f67e00700 1 -- 172.21.15.78:0/1159653877 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f680340d0 con 0x55fad8205fa0 2024-03-14T09:56:15.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.079+0000 7f4f67e00700 1 -- 172.21.15.78:0/1159653877 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f6801e040 con 0x55fad8205fa0 2024-03-14T09:56:15.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.079+0000 7f4f67e00700 1 -- 172.21.15.78:0/1159653877 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f68004760 con 0x55fad8205fa0 2024-03-14T09:56:15.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.080+0000 7f4f949aeb80 1 -- 172.21.15.78:0/1159653877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad8205fa0 msgr2=0x55fad8216b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.080+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/1159653877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad8205fa0 0x55fad8216b40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4f680098e0 tx=0x7f4f68020000 comp rx=0 tx=0).stop 2024-03-14T09:56:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.080+0000 7f4f949aeb80 1 -- 172.21.15.78:0/1159653877 shutdown_connections 2024-03-14T09:56:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.080+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/1159653877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad8200c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.080+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/1159653877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad820b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.080+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/1159653877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad8205fa0 0x55fad8216b40 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.080+0000 7f4f949aeb80 1 -- 172.21.15.78:0/1159653877 >> 172.21.15.78:0/1159653877 conn(0x55fad817b220 msgr2=0x55fad8207ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.081+0000 7f4f949aeb80 1 -- 172.21.15.78:0/1159653877 shutdown_connections 2024-03-14T09:56:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.081+0000 7f4f949aeb80 1 -- 172.21.15.78:0/1159653877 wait complete. 2024-03-14T09:56:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.081+0000 7f4f949aeb80 1 -- start start 2024-03-14T09:56:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.081+0000 7f4f949aeb80 1 Processor -- start 2024-03-14T09:56:15.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.082+0000 7f4f949aeb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad82148d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.082+0000 7f4f949aeb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad8214e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.082+0000 7f4f949aeb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad826e4a0 0x55fad8270890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.082+0000 7f4f949aeb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fad8270dd0 con 0x55fad826e4a0 2024-03-14T09:56:15.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.082+0000 7f4f949aeb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fad8270f10 con 0x55fad8205680 2024-03-14T09:56:15.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.082+0000 7f4f949aeb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fad82820e0 con 0x55fad820fec0 2024-03-14T09:56:15.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.082+0000 7f4f949aeb80 10 client.0 _ll_register_callbacks cb 0x7f4f6c099980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.083+0000 7f4f79e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad82148d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.083+0000 7f4f79e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad82148d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:56456/0 (socket says 172.21.15.78:56456) 2024-03-14T09:56:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.083+0000 7f4f79e00700 1 -- 172.21.15.78:0/2592008605 learned_addr learned my addr 172.21.15.78:0/2592008605 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.083+0000 7f4f79400700 1 --2- 172.21.15.78:0/2592008605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad8214e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.083+0000 7f4f7a800700 1 --2- 172.21.15.78:0/2592008605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad826e4a0 0x55fad8270890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.083+0000 7f4f79e00700 1 -- 172.21.15.78:0/2592008605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 msgr2=0x55fad8214e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.083+0000 7f4f79e00700 1 --2- 172.21.15.78:0/2592008605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad8214e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.083+0000 7f4f79e00700 1 -- 172.21.15.78:0/2592008605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad826e4a0 msgr2=0x55fad8270890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.083+0000 7f4f79e00700 1 --2- 172.21.15.78:0/2592008605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad826e4a0 0x55fad8270890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.083+0000 7f4f79e00700 1 -- 172.21.15.78:0/2592008605 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f6c0b7070 con 0x55fad8205680 2024-03-14T09:56:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.083+0000 7f4f7a800700 1 --2- 172.21.15.78:0/2592008605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad826e4a0 0x55fad8270890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:56:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.083+0000 7f4f79e00700 1 --2- 172.21.15.78:0/2592008605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad82148d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c099850 tx=0x7f4f6c099780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f27b3da72838054 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.083+0000 7f4f5be00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:56:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.084+0000 7f4f5be00700 1 -- 172.21.15.78:0/2592008605 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c099240 con 0x55fad8205680 2024-03-14T09:56:15.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.084+0000 7f4f5be00700 1 -- 172.21.15.78:0/2592008605 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f4f6c00cc90 con 0x55fad8205680 2024-03-14T09:56:15.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.084+0000 7f4f949aeb80 1 -- 172.21.15.78:0/2592008605 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f7400f050 con 0x55fad8205680 2024-03-14T09:56:15.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.084+0000 7f4f5be00700 1 -- 172.21.15.78:0/2592008605 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f6c099b50 con 0x55fad8205680 2024-03-14T09:56:15.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.084+0000 7f4f949aeb80 10 client.26099 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:15.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.084+0000 7f4f949aeb80 1 -- 172.21.15.78:0/2592008605 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f4f7c10e420 con 0x55fad8205680 2024-03-14T09:56:15.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.084+0000 7f4f5be00700 1 -- 172.21.15.78:0/2592008605 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f7400f050 con 0x55fad8205680 2024-03-14T09:56:15.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.085+0000 7f4f5be00700 1 -- 172.21.15.78:0/2592008605 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4f6c081020 con 0x55fad8205680 2024-03-14T09:56:15.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.085+0000 7f4f949aeb80 10 client.26099 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:15.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.085+0000 7f4f949aeb80 10 client.26099 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:56:15.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.085+0000 7f4f949aeb80 20 client.26099 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:15.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.085+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/2592008605 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55fad8283450 0x55fad8285870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.085+0000 7f4f949aeb80 4 client.26099 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:56:15.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.085+0000 7f4f949aeb80 1 -- 172.21.15.78:0/2592008605 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fad7fa6cf0 con 0x55fad8283450 2024-03-14T09:56:15.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.085+0000 7f4f79400700 1 --2- 172.21.15.78:0/2592008605 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55fad8283450 0x55fad8285870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.086+0000 7f4f79400700 1 --2- 172.21.15.78:0/2592008605 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55fad8283450 0x55fad8285870 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c642c2f579135752 server_cookie=cba89e2b27b11ad2 in_seq=0 out_seq=0 2024-03-14T09:56:15.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.086+0000 7f4f5be00700 10 client.26099 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:56:15.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.086+0000 7f4f5be00700 1 -- 172.21.15.78:0/2592008605 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fad7fa6cf0 con 0x55fad8283450 2024-03-14T09:56:15.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.086+0000 7f4f5be00700 10 client.26099 handle_command_reply: tid=0 2024-03-14T09:56:15.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.086+0000 7f4f949aeb80 1 client.26099 shutdown 2024-03-14T09:56:15.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.087+0000 7f4f949aeb80 1 -- 172.21.15.78:0/2592008605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 msgr2=0x55fad82148d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.087+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/2592008605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad82148d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c099850 tx=0x7f4f6c099780 comp rx=0 tx=0).stop 2024-03-14T09:56:15.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.087+0000 7f4f949aeb80 1 -- 172.21.15.78:0/2592008605 shutdown_connections 2024-03-14T09:56:15.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.087+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/2592008605 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55fad8283450 0x55fad8285870 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.087+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/2592008605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fad8205680 0x55fad82148d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.087+0000 7f4f5be00700 0 client.26099 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:56:15.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.087+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/2592008605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fad826e4a0 0x55fad8270890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.087+0000 7f4f949aeb80 1 --2- 172.21.15.78:0/2592008605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fad820fec0 0x55fad8214e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.087+0000 7f4f949aeb80 1 -- 172.21.15.78:0/2592008605 >> 172.21.15.78:0/2592008605 conn(0x55fad817b220 msgr2=0x55fad820b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:15.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.087+0000 7f4f949aeb80 1 -- 172.21.15.78:0/2592008605 shutdown_connections 2024-03-14T09:56:15.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.087+0000 7f4f949aeb80 1 -- 172.21.15.78:0/2592008605 wait complete. 2024-03-14T09:56:15.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.088+0000 7f4f949aeb80 20 client.26099 trim_cache size 0 max 16384 2024-03-14T09:56:15.090 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:56:15.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.088+0000 7f4f80e00700 1 -- 172.21.15.78:0/814564672 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4f600a0c80 msgr2=0x7f4f600a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.088+0000 7f4f80e00700 1 --2- 172.21.15.78:0/814564672 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4f600a0c80 0x7f4f600a3140 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4f68006010 tx=0x7f4f68005960 comp rx=0 tx=0).stop 2024-03-14T09:56:15.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.088+0000 7f4f80e00700 1 -- 172.21.15.78:0/814564672 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f7c0680b0 msgr2=0x7f4f7c10bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.088+0000 7f4f80e00700 1 --2- 172.21.15.78:0/814564672 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f7c0680b0 0x7f4f7c10bc70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c00eb30 tx=0x7f4f6c00ee40 comp rx=0 tx=0).stop 2024-03-14T09:56:15.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.089+0000 7f4f80e00700 1 -- 172.21.15.78:0/814564672 shutdown_connections 2024-03-14T09:56:15.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.089+0000 7f4f80e00700 1 --2- 172.21.15.78:0/814564672 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4f600a0c80 0x7f4f600a3140 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.089+0000 7f4f80e00700 1 --2- 172.21.15.78:0/814564672 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4f7c0680b0 0x7f4f7c10bc70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.089+0000 7f4f80e00700 1 --2- 172.21.15.78:0/814564672 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4f7c1a24a0 0x7f4f7c1a2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.089+0000 7f4f80e00700 1 --2- 172.21.15.78:0/814564672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f7c106060 0x7f4f7c10c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.089+0000 7f4f80e00700 1 -- 172.21.15.78:0/814564672 >> 172.21.15.78:0/814564672 conn(0x7f4f7c076d80 msgr2=0x7f4f7c104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:15.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.089+0000 7f4f80e00700 1 -- 172.21.15.78:0/814564672 shutdown_connections 2024-03-14T09:56:15.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.089+0000 7f4f80e00700 1 -- 172.21.15.78:0/814564672 wait complete. 2024-03-14T09:56:15.112 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T09:56:15.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:15 smithi078 ceph-mon[38143]: pgmap v65: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 29 KiB/s wr, 6 op/s 2024-03-14T09:56:15.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:15 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3854515548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:56:15.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.362+0000 7fa787c00700 1 -- 172.21.15.78:0/315873437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7801018d0 msgr2=0x7fa780105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.362+0000 7fa787c00700 1 --2- 172.21.15.78:0/315873437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7801018d0 0x7fa780105910 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa76c0098f0 tx=0x7fa76c009c00 comp rx=0 tx=0).stop 2024-03-14T09:56:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.363+0000 7fa787c00700 1 -- 172.21.15.78:0/315873437 shutdown_connections 2024-03-14T09:56:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.363+0000 7fa787c00700 1 --2- 172.21.15.78:0/315873437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa780106040 0x7fa78010dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.363+0000 7fa787c00700 1 --2- 172.21.15.78:0/315873437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa780100f20 0x7fa780101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.363+0000 7fa787c00700 1 --2- 172.21.15.78:0/315873437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa7801018d0 0x7fa780105910 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.363+0000 7fa787c00700 1 -- 172.21.15.78:0/315873437 >> 172.21.15.78:0/315873437 conn(0x7fa7800fc7a0 msgr2=0x7fa7800febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.363+0000 7fa787c00700 1 -- 172.21.15.78:0/315873437 shutdown_connections 2024-03-14T09:56:15.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.363+0000 7fa787c00700 1 -- 172.21.15.78:0/315873437 wait complete. 2024-03-14T09:56:15.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.364+0000 7fa787c00700 1 Processor -- start 2024-03-14T09:56:15.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.364+0000 7fa787c00700 1 -- start start 2024-03-14T09:56:15.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.364+0000 7fa787c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa780100f20 0x7fa780197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.364+0000 7fa787c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa780106040 0x7fa780197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.364+0000 7fa787c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa78019bf00 0x7fa78019c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.364+0000 7fa787c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78019c980 con 0x7fa780100f20 2024-03-14T09:56:15.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.364+0000 7fa787c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78019caf0 con 0x7fa78019bf00 2024-03-14T09:56:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.364+0000 7fa787c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78019ccb0 con 0x7fa780106040 2024-03-14T09:56:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.364+0000 7fa785c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa78019bf00 0x7fa78019c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.364+0000 7fa77fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa780106040 0x7fa780197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.364+0000 7fa785200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa780100f20 0x7fa780197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.365+0000 7fa785c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa78019bf00 0x7fa78019c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:56470/0 (socket says 172.21.15.78:56470) 2024-03-14T09:56:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.365+0000 7fa785c00700 1 -- 172.21.15.78:0/675100886 learned_addr learned my addr 172.21.15.78:0/675100886 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.365+0000 7fa785c00700 1 -- 172.21.15.78:0/675100886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa780106040 msgr2=0x7fa780197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.365+0000 7fa785c00700 1 --2- 172.21.15.78:0/675100886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa780106040 0x7fa780197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.365+0000 7fa785c00700 1 -- 172.21.15.78:0/675100886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa780100f20 msgr2=0x7fa780197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.365+0000 7fa785c00700 1 --2- 172.21.15.78:0/675100886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa780100f20 0x7fa780197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.365+0000 7fa785c00700 1 -- 172.21.15.78:0/675100886 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c009520 con 0x7fa78019bf00 2024-03-14T09:56:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.365+0000 7fa77fe00700 1 --2- 172.21.15.78:0/675100886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa780106040 0x7fa780197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:56:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.365+0000 7fa785c00700 1 --2- 172.21.15.78:0/675100886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa78019bf00 0x7fa78019c380 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa77800ef30 tx=0x7fa77800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.365+0000 7fa77d600700 1 -- 172.21.15.78:0/675100886 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778018070 con 0x7fa78019bf00 2024-03-14T09:56:15.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.365+0000 7fa787c00700 1 -- 172.21.15.78:0/675100886 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7801a41b0 con 0x7fa78019bf00 2024-03-14T09:56:15.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.365+0000 7fa77d600700 1 -- 172.21.15.78:0/675100886 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa77800f040 con 0x7fa78019bf00 2024-03-14T09:56:15.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.365+0000 7fa77d600700 1 -- 172.21.15.78:0/675100886 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77800ccf0 con 0x7fa78019bf00 2024-03-14T09:56:15.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.366+0000 7fa787c00700 1 -- 172.21.15.78:0/675100886 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7801a4690 con 0x7fa78019bf00 2024-03-14T09:56:15.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.369+0000 7fa77d600700 1 -- 172.21.15.78:0/675100886 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fa778003a00 con 0x7fa78019bf00 2024-03-14T09:56:15.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.370+0000 7fa77d600700 1 --2- 172.21.15.78:0/675100886 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa7680a0c00 0x7fa7680a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.371+0000 7fa79b678b80 1 Processor -- start 2024-03-14T09:56:15.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.371+0000 7fa77d600700 1 -- 172.21.15.78:0/675100886 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fa7780cc270 con 0x7fa78019bf00 2024-03-14T09:56:15.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.371+0000 7fa785200700 1 --2- 172.21.15.78:0/675100886 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa7680a0c00 0x7fa7680a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.371+0000 7fa79b678b80 1 -- start start 2024-03-14T09:56:15.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.372+0000 7fa785200700 1 --2- 172.21.15.78:0/675100886 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa7680a0c00 0x7fa7680a30c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa774005fd0 tx=0x7fa774009310 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:15.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.372+0000 7fa79b678b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec404a460 0x562ec4048680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.372+0000 7fa79b678b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec4048bc0 0x562ec4049040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.372+0000 7fa79b678b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec405b560 0x562ec405d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.372+0000 7fa79b678b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec3fa1990 con 0x562ec405b560 2024-03-14T09:56:15.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.372+0000 7fa79b678b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec3f9f350 con 0x562ec4048bc0 2024-03-14T09:56:15.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.372+0000 7fa79b678b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec3f9ac40 con 0x562ec404a460 2024-03-14T09:56:15.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.373+0000 7fa77fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec4048bc0 0x562ec4049040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.373+0000 7fa77fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec4048bc0 0x562ec4049040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:56472/0 (socket says 172.21.15.78:56472) 2024-03-14T09:56:15.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.373+0000 7fa77fe00700 1 -- 172.21.15.78:0/1756172235 learned_addr learned my addr 172.21.15.78:0/1756172235 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:15.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.373+0000 7fa785200700 1 --2- 172.21.15.78:0/1756172235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec404a460 0x562ec4048680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.373+0000 7fa785c00700 1 --2- 172.21.15.78:0/1756172235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec405b560 0x562ec405d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.373+0000 7fa77fe00700 1 -- 172.21.15.78:0/1756172235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec404a460 msgr2=0x562ec4048680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:15.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.373+0000 7fa77fe00700 1 --2- 172.21.15.78:0/1756172235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec404a460 0x562ec4048680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.373+0000 7fa77fe00700 1 -- 172.21.15.78:0/1756172235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec405b560 msgr2=0x562ec405d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:15.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.373+0000 7fa77fe00700 1 --2- 172.21.15.78:0/1756172235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec405b560 0x562ec405d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.373+0000 7fa77fe00700 1 -- 172.21.15.78:0/1756172235 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ec3e1c3b0 con 0x562ec4048bc0 2024-03-14T09:56:15.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.373+0000 7fa785200700 1 --2- 172.21.15.78:0/1756172235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec404a460 0x562ec4048680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:56:15.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.373+0000 7fa785c00700 1 --2- 172.21.15.78:0/1756172235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec405b560 0x562ec405d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:56:15.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.373+0000 7fa77fe00700 1 --2- 172.21.15.78:0/1756172235 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec4048bc0 0x562ec4049040 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa78006a350 tx=0x7fa76c005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a633ff22a6b5962 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:15.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.374+0000 7fa772000700 1 -- 172.21.15.78:0/1756172235 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c01c050 con 0x562ec4048bc0 2024-03-14T09:56:15.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.374+0000 7fa772000700 1 -- 172.21.15.78:0/1756172235 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa76c01d040 con 0x562ec4048bc0 2024-03-14T09:56:15.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.374+0000 7fa772000700 1 -- 172.21.15.78:0/1756172235 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c0052a0 con 0x562ec4048bc0 2024-03-14T09:56:15.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.375+0000 7fa79b678b80 1 -- 172.21.15.78:0/1756172235 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec4048bc0 msgr2=0x562ec4049040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.375+0000 7fa79b678b80 1 --2- 172.21.15.78:0/1756172235 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec4048bc0 0x562ec4049040 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa78006a350 tx=0x7fa76c005dc0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.375+0000 7fa79b678b80 1 -- 172.21.15.78:0/1756172235 shutdown_connections 2024-03-14T09:56:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.375+0000 7fa79b678b80 1 --2- 172.21.15.78:0/1756172235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec405b560 0x562ec405d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.375+0000 7fa79b678b80 1 --2- 172.21.15.78:0/1756172235 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec404a460 0x562ec4048680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.375+0000 7fa79b678b80 1 --2- 172.21.15.78:0/1756172235 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec4048bc0 0x562ec4049040 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.375+0000 7fa79b678b80 1 -- 172.21.15.78:0/1756172235 >> 172.21.15.78:0/1756172235 conn(0x562ec3fbe220 msgr2=0x562ec4043aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.375+0000 7fa79b678b80 1 -- 172.21.15.78:0/1756172235 shutdown_connections 2024-03-14T09:56:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.375+0000 7fa79b678b80 1 -- 172.21.15.78:0/1756172235 wait complete. 2024-03-14T09:56:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.376+0000 7fa79b678b80 1 -- start start 2024-03-14T09:56:15.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.376+0000 7fa79b678b80 1 Processor -- start 2024-03-14T09:56:15.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.377+0000 7fa79b678b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec4048bc0 0x562ec404e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.377+0000 7fa79b678b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 0x562ec404b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.377+0000 7fa79b678b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec405b560 0x562ec404bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.377+0000 7fa79b678b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec3f36d80 con 0x562ec405b560 2024-03-14T09:56:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.377+0000 7fa79b678b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec3f36a20 con 0x562ec404a460 2024-03-14T09:56:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.377+0000 7fa79b678b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec3e1e6a0 con 0x562ec4048bc0 2024-03-14T09:56:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.377+0000 7fa79b678b80 10 client.0 _ll_register_callbacks cb 0x562ec3da23a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.377+0000 7fa785200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec4048bc0 0x562ec404e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.377+0000 7fa785c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec405b560 0x562ec404bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.377+0000 7fa785200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec4048bc0 0x562ec404e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:55108/0 (socket says 172.21.15.78:55108) 2024-03-14T09:56:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.378+0000 7fa785200700 1 -- 172.21.15.78:0/4003556802 learned_addr learned my addr 172.21.15.78:0/4003556802 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.378+0000 7fa77fe00700 1 --2- 172.21.15.78:0/4003556802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 0x562ec404b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.378+0000 7fa785c00700 1 -- 172.21.15.78:0/4003556802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec4048bc0 msgr2=0x562ec404e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:15.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.378+0000 7fa785c00700 1 --2- 172.21.15.78:0/4003556802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec4048bc0 0x562ec404e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.378+0000 7fa785c00700 1 -- 172.21.15.78:0/4003556802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 msgr2=0x562ec404b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:15.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.378+0000 7fa785c00700 1 --2- 172.21.15.78:0/4003556802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 0x562ec404b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.378+0000 7fa785c00700 1 -- 172.21.15.78:0/4003556802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78010e430 con 0x562ec405b560 2024-03-14T09:56:15.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.378+0000 7fa77fe00700 1 --2- 172.21.15.78:0/4003556802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 0x562ec404b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:56:15.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.379+0000 7fa785c00700 1 --2- 172.21.15.78:0/4003556802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec405b560 0x562ec404bdf0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa7780c9610 tx=0x7fa778004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a12c806e5e5130 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:15.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.379+0000 7fa766a00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:56:15.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.379+0000 7fa766a00700 1 -- 172.21.15.78:0/4003556802 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778099700 con 0x562ec405b560 2024-03-14T09:56:15.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.379+0000 7fa766a00700 1 -- 172.21.15.78:0/4003556802 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fa7780999a0 con 0x562ec405b560 2024-03-14T09:56:15.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.379+0000 7fa79b678b80 1 -- 172.21.15.78:0/4003556802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562ec403b720 con 0x562ec405b560 2024-03-14T09:56:15.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.380+0000 7fa766a00700 1 -- 172.21.15.78:0/4003556802 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7780992a0 con 0x562ec405b560 2024-03-14T09:56:15.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.380+0000 7fa79b678b80 10 client.16104 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:15.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.380+0000 7fa79b678b80 1 -- 172.21.15.78:0/4003556802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562ec3dc8b60 con 0x562ec405b560 2024-03-14T09:56:15.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.380+0000 7fa766a00700 1 -- 172.21.15.78:0/4003556802 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562ec403b720 con 0x562ec405b560 2024-03-14T09:56:15.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.380+0000 7fa766a00700 1 -- 172.21.15.78:0/4003556802 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa77809b070 con 0x562ec405b560 2024-03-14T09:56:15.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.380+0000 7fa79b678b80 10 client.16104 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:15.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.380+0000 7fa79b678b80 10 client.16104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:56:15.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.380+0000 7fa79b678b80 20 client.16104 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:15.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.380+0000 7fa79b678b80 1 --2- 172.21.15.78:0/4003556802 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x562ec406b350 0x562ec406d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.380+0000 7fa79b678b80 4 client.16104 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:56:15.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.380+0000 7fa79b678b80 1 -- 172.21.15.78:0/4003556802 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562ec3f93700 con 0x562ec406b350 2024-03-14T09:56:15.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.381+0000 7fa785200700 1 --2- 172.21.15.78:0/4003556802 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x562ec406b350 0x562ec406d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.381+0000 7fa766a00700 10 client.16104 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:15.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.382+0000 7fa785200700 1 --2- 172.21.15.78:0/4003556802 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x562ec406b350 0x562ec406d740 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=864ae72fa5f67786 server_cookie=814f46c6f224590f in_seq=0 out_seq=0 2024-03-14T09:56:15.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.382+0000 7fa766a00700 1 -- 172.21.15.78:0/4003556802 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x562ec3f93700 con 0x562ec406b350 2024-03-14T09:56:15.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.382+0000 7fa766a00700 10 client.16104 handle_command_reply: tid=0 2024-03-14T09:56:15.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.382+0000 7fa79b678b80 1 client.16104 shutdown 2024-03-14T09:56:15.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.383+0000 7fa79b678b80 1 -- 172.21.15.78:0/4003556802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec405b560 msgr2=0x562ec404bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.383+0000 7fa79b678b80 1 --2- 172.21.15.78:0/4003556802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec405b560 0x562ec404bdf0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa7780c9610 tx=0x7fa778004b50 comp rx=0 tx=0).stop 2024-03-14T09:56:15.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.383+0000 7fa79b678b80 1 -- 172.21.15.78:0/4003556802 shutdown_connections 2024-03-14T09:56:15.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.383+0000 7fa79b678b80 1 --2- 172.21.15.78:0/4003556802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec405b560 0x562ec404bdf0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.383+0000 7fa79b678b80 1 --2- 172.21.15.78:0/4003556802 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec4048bc0 0x562ec404e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.383+0000 7fa79b678b80 1 --2- 172.21.15.78:0/4003556802 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x562ec406b350 0x562ec406d740 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.384+0000 7fa79b678b80 1 --2- 172.21.15.78:0/4003556802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 0x562ec404b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.384+0000 7fa766a00700 0 client.16104 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:15.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.384+0000 7fa79b678b80 1 -- 172.21.15.78:0/4003556802 >> 172.21.15.78:0/4003556802 conn(0x562ec3fbe220 msgr2=0x562ec4043690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:15.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.384+0000 7fa79b678b80 1 -- 172.21.15.78:0/4003556802 shutdown_connections 2024-03-14T09:56:15.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.384+0000 7fa79b678b80 1 -- 172.21.15.78:0/4003556802 wait complete. 2024-03-14T09:56:15.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.385+0000 7fa79b678b80 20 client.16104 trim_cache size 0 max 16384 2024-03-14T09:56:15.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.404+0000 7fa79b678b80 1 Processor -- start 2024-03-14T09:56:15.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.404+0000 7fa79b678b80 1 -- start start 2024-03-14T09:56:15.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.404+0000 7fa79b678b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec4048bc0 0x562ec4062330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.404+0000 7fa79b678b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 0x562ec403f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.404+0000 7fa79b678b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec405b560 0x562ec4048fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.404+0000 7fa79b678b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec403fbc0 con 0x562ec4048bc0 2024-03-14T09:56:15.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.404+0000 7fa79b678b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec403b260 con 0x562ec404a460 2024-03-14T09:56:15.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.404+0000 7fa79b678b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec404c3f0 con 0x562ec405b560 2024-03-14T09:56:15.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.405+0000 7fa77fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 0x562ec403f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.405+0000 7fa77fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 0x562ec403f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:56500/0 (socket says 172.21.15.78:56500) 2024-03-14T09:56:15.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.405+0000 7fa77fe00700 1 -- 172.21.15.78:0/2173445862 learned_addr learned my addr 172.21.15.78:0/2173445862 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:15.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.405+0000 7fa785200700 1 --2- 172.21.15.78:0/2173445862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec4048bc0 0x562ec4062330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.405+0000 7fa785c00700 1 --2- 172.21.15.78:0/2173445862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec405b560 0x562ec4048fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.405+0000 7fa77fe00700 1 -- 172.21.15.78:0/2173445862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec405b560 msgr2=0x562ec4048fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:15.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.405+0000 7fa77fe00700 1 --2- 172.21.15.78:0/2173445862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec405b560 0x562ec4048fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.405+0000 7fa77fe00700 1 -- 172.21.15.78:0/2173445862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec4048bc0 msgr2=0x562ec4062330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:15.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.405+0000 7fa77fe00700 1 --2- 172.21.15.78:0/2173445862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec4048bc0 0x562ec4062330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.405+0000 7fa77fe00700 1 -- 172.21.15.78:0/2173445862 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ec3f60ea0 con 0x562ec404a460 2024-03-14T09:56:15.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.405+0000 7fa785c00700 1 --2- 172.21.15.78:0/2173445862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec405b560 0x562ec4048fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:56:15.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.406+0000 7fa77fe00700 1 --2- 172.21.15.78:0/2173445862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 0x562ec403f680 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa78006a350 tx=0x7fa76c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1769ecb378bea00e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:15.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.406+0000 7fa772a00700 1 -- 172.21.15.78:0/2173445862 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c0052b0 con 0x562ec404a460 2024-03-14T09:56:15.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.406+0000 7fa772a00700 1 -- 172.21.15.78:0/2173445862 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa76c00ba20 con 0x562ec404a460 2024-03-14T09:56:15.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.406+0000 7fa772a00700 1 -- 172.21.15.78:0/2173445862 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c003be0 con 0x562ec404a460 2024-03-14T09:56:15.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.407+0000 7fa79b678b80 1 -- 172.21.15.78:0/2173445862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 msgr2=0x562ec403f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.407+0000 7fa79b678b80 1 --2- 172.21.15.78:0/2173445862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 0x562ec403f680 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa78006a350 tx=0x7fa76c009c00 comp rx=0 tx=0).stop 2024-03-14T09:56:15.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.407+0000 7fa79b678b80 1 -- 172.21.15.78:0/2173445862 shutdown_connections 2024-03-14T09:56:15.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.407+0000 7fa79b678b80 1 --2- 172.21.15.78:0/2173445862 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec405b560 0x562ec4048fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.407+0000 7fa79b678b80 1 --2- 172.21.15.78:0/2173445862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec4048bc0 0x562ec4062330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.407+0000 7fa79b678b80 1 --2- 172.21.15.78:0/2173445862 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 0x562ec403f680 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.407+0000 7fa79b678b80 1 -- 172.21.15.78:0/2173445862 >> 172.21.15.78:0/2173445862 conn(0x562ec3fbe220 msgr2=0x562ec404af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:15.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.408+0000 7fa79b678b80 1 -- 172.21.15.78:0/2173445862 shutdown_connections 2024-03-14T09:56:15.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.408+0000 7fa79b678b80 1 -- 172.21.15.78:0/2173445862 wait complete. 2024-03-14T09:56:15.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.408+0000 7fa79b678b80 1 -- start start 2024-03-14T09:56:15.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.409+0000 7fa79b678b80 1 Processor -- start 2024-03-14T09:56:15.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.409+0000 7fa79b678b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec4048bc0 0x562ec4060310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.409+0000 7fa79b678b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 0x562ec4060850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.409+0000 7fa79b678b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec405b560 0x562ec4060d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.409+0000 7fa79b678b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec40bbe40 con 0x562ec4048bc0 2024-03-14T09:56:15.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.409+0000 7fa79b678b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec40bbf80 con 0x562ec404a460 2024-03-14T09:56:15.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.409+0000 7fa79b678b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ec40bc0f0 con 0x562ec405b560 2024-03-14T09:56:15.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.409+0000 7fa79b678b80 10 client.0 _ll_register_callbacks cb 0x7fa778098de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:15.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.410+0000 7fa785c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec405b560 0x562ec4060d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.410+0000 7fa785c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec405b560 0x562ec4060d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:55116/0 (socket says 172.21.15.78:55116) 2024-03-14T09:56:15.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.410+0000 7fa785c00700 1 -- 172.21.15.78:0/540086164 learned_addr learned my addr 172.21.15.78:0/540086164 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:15.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.410+0000 7fa785c00700 1 -- 172.21.15.78:0/540086164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 msgr2=0x562ec4060850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-14T09:56:15.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.410+0000 7fa785c00700 1 --2- 172.21.15.78:0/540086164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 0x562ec4060850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.410+0000 7fa785c00700 1 -- 172.21.15.78:0/540086164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec4048bc0 msgr2=0x562ec4060310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:15.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.410+0000 7fa785c00700 1 --2- 172.21.15.78:0/540086164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec4048bc0 0x562ec4060310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.410+0000 7fa785c00700 1 -- 172.21.15.78:0/540086164 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ec3e1c3b0 con 0x562ec405b560 2024-03-14T09:56:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.411+0000 7fa785c00700 1 --2- 172.21.15.78:0/540086164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec405b560 0x562ec4060d90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa7780090a0 tx=0x7fa778099420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab9522bde9e6021d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.411+0000 7fa766a00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:56:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.411+0000 7fa766a00700 1 -- 172.21.15.78:0/540086164 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7780992a0 con 0x562ec405b560 2024-03-14T09:56:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.411+0000 7fa766a00700 1 -- 172.21.15.78:0/540086164 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fa7780b7800 con 0x562ec405b560 2024-03-14T09:56:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.411+0000 7fa79b678b80 1 -- 172.21.15.78:0/540086164 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa76c01d040 con 0x562ec405b560 2024-03-14T09:56:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.411+0000 7fa766a00700 1 -- 172.21.15.78:0/540086164 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7780955e0 con 0x562ec405b560 2024-03-14T09:56:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.411+0000 7fa79b678b80 10 client.26125 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.411+0000 7fa79b678b80 1 -- 172.21.15.78:0/540086164 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562ec3e20cd0 con 0x562ec405b560 2024-03-14T09:56:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.412+0000 7fa766a00700 1 -- 172.21.15.78:0/540086164 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa76c01d040 con 0x562ec405b560 2024-03-14T09:56:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.412+0000 7fa766a00700 1 -- 172.21.15.78:0/540086164 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa77809b070 con 0x562ec405b560 2024-03-14T09:56:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.412+0000 7fa79b678b80 10 client.26125 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.412+0000 7fa79b678b80 10 client.26125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:56:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.412+0000 7fa79b678b80 20 client.26125 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.412+0000 7fa79b678b80 1 --2- 172.21.15.78:0/540086164 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x562ec40bd3f0 0x562ec40bf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.412+0000 7fa79b678b80 4 client.26125 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:56:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.412+0000 7fa79b678b80 1 -- 172.21.15.78:0/540086164 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562ec3de9cf0 con 0x562ec40bd3f0 2024-03-14T09:56:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.412+0000 7fa785200700 1 --2- 172.21.15.78:0/540086164 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x562ec40bd3f0 0x562ec40bf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:15.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.413+0000 7fa785200700 1 --2- 172.21.15.78:0/540086164 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x562ec40bd3f0 0x562ec40bf810 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3c220094c85eb6b server_cookie=5bddbc448570de21 in_seq=0 out_seq=0 2024-03-14T09:56:15.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.413+0000 7fa766a00700 10 client.26125 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:15.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.413+0000 7fa766a00700 1 -- 172.21.15.78:0/540086164 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562ec3de9cf0 con 0x562ec40bd3f0 2024-03-14T09:56:15.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.413+0000 7fa766a00700 10 client.26125 handle_command_reply: tid=0 2024-03-14T09:56:15.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.413+0000 7fa79b678b80 1 client.26125 shutdown 2024-03-14T09:56:15.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.413+0000 7fa79b678b80 1 -- 172.21.15.78:0/540086164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec405b560 msgr2=0x562ec4060d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.413+0000 7fa79b678b80 1 --2- 172.21.15.78:0/540086164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec405b560 0x562ec4060d90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa7780090a0 tx=0x7fa778099420 comp rx=0 tx=0).stop 2024-03-14T09:56:15.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.413+0000 7fa79b678b80 1 -- 172.21.15.78:0/540086164 shutdown_connections 2024-03-14T09:56:15.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.413+0000 7fa79b678b80 1 --2- 172.21.15.78:0/540086164 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562ec405b560 0x562ec4060d90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.413+0000 7fa79b678b80 1 --2- 172.21.15.78:0/540086164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562ec4048bc0 0x562ec4060310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.413+0000 7fa79b678b80 1 --2- 172.21.15.78:0/540086164 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x562ec40bd3f0 0x562ec40bf810 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.413+0000 7fa79b678b80 1 --2- 172.21.15.78:0/540086164 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562ec404a460 0x562ec4060850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.413+0000 7fa766a00700 0 client.26125 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:15.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.413+0000 7fa79b678b80 1 -- 172.21.15.78:0/540086164 >> 172.21.15.78:0/540086164 conn(0x562ec3fbe220 msgr2=0x562ec404af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:15.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.414+0000 7fa79b678b80 1 -- 172.21.15.78:0/540086164 shutdown_connections 2024-03-14T09:56:15.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.414+0000 7fa79b678b80 1 -- 172.21.15.78:0/540086164 wait complete. 2024-03-14T09:56:15.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.414+0000 7fa79b678b80 20 client.26125 trim_cache size 0 max 16384 2024-03-14T09:56:15.417 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:56:15.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.415+0000 7fa787c00700 1 -- 172.21.15.78:0/675100886 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa7680a0c00 msgr2=0x7fa7680a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.415+0000 7fa787c00700 1 --2- 172.21.15.78:0/675100886 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa7680a0c00 0x7fa7680a30c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa774005fd0 tx=0x7fa774009310 comp rx=0 tx=0).stop 2024-03-14T09:56:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.415+0000 7fa787c00700 1 -- 172.21.15.78:0/675100886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa78019bf00 msgr2=0x7fa78019c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.415+0000 7fa787c00700 1 --2- 172.21.15.78:0/675100886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa78019bf00 0x7fa78019c380 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa77800ef30 tx=0x7fa77800c3a0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.415+0000 7fa787c00700 1 -- 172.21.15.78:0/675100886 shutdown_connections 2024-03-14T09:56:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.415+0000 7fa787c00700 1 --2- 172.21.15.78:0/675100886 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa7680a0c00 0x7fa7680a30c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.415+0000 7fa787c00700 1 --2- 172.21.15.78:0/675100886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa78019bf00 0x7fa78019c380 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.415+0000 7fa787c00700 1 --2- 172.21.15.78:0/675100886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa780100f20 0x7fa780197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.415+0000 7fa787c00700 1 --2- 172.21.15.78:0/675100886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa780106040 0x7fa780197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.415+0000 7fa787c00700 1 -- 172.21.15.78:0/675100886 >> 172.21.15.78:0/675100886 conn(0x7fa7800fc7a0 msgr2=0x7fa7800fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.415+0000 7fa787c00700 1 -- 172.21.15.78:0/675100886 shutdown_connections 2024-03-14T09:56:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:15.415+0000 7fa787c00700 1 -- 172.21.15.78:0/675100886 wait complete. 2024-03-14T09:56:15.499 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0xffce4 2024-03-14T09:56:15.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:15 smithi152 ceph-mon[48279]: pgmap v65: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 29 KiB/s wr, 6 op/s 2024-03-14T09:56:15.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:15 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3854515548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:56:15.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:15 smithi132 ceph-mon[47959]: pgmap v65: 129 pgs: 129 active+clean; 80 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 29 KiB/s wr, 6 op/s 2024-03-14T09:56:15.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:15 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3854515548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:56:16.438 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T09:56:16.438 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T09:56:16.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.688+0000 7ff964a00700 1 -- 172.21.15.78:0/1413265924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff96010f4d0 msgr2=0x7ff960107db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:16.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.688+0000 7ff964a00700 1 --2- 172.21.15.78:0/1413265924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff96010f4d0 0x7ff960107db0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff950009990 tx=0x7ff950009ca0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.688+0000 7ff964a00700 1 -- 172.21.15.78:0/1413265924 shutdown_connections 2024-03-14T09:56:16.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.688+0000 7ff964a00700 1 --2- 172.21.15.78:0/1413265924 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff96011a590 0x7ff96011c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.688+0000 7ff964a00700 1 --2- 172.21.15.78:0/1413265924 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff96010f4d0 0x7ff960107db0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.688+0000 7ff964a00700 1 --2- 172.21.15.78:0/1413265924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff96010f940 0x7ff9601082f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.688+0000 7ff964a00700 1 -- 172.21.15.78:0/1413265924 >> 172.21.15.78:0/1413265924 conn(0x7ff96006bdc0 msgr2=0x7ff96006c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:16.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.688+0000 7ff964a00700 1 -- 172.21.15.78:0/1413265924 shutdown_connections 2024-03-14T09:56:16.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.688+0000 7ff964a00700 1 -- 172.21.15.78:0/1413265924 wait complete. 2024-03-14T09:56:16.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.689+0000 7ff964a00700 1 Processor -- start 2024-03-14T09:56:16.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.689+0000 7ff964a00700 1 -- start start 2024-03-14T09:56:16.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.690+0000 7ff964a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff96010f4d0 0x7ff9601a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.690+0000 7ff964a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff96010f940 0x7ff9601a6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.690+0000 7ff964a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff96011a590 0x7ff9601aae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.690+0000 7ff964a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9601ab450 con 0x7ff96010f940 2024-03-14T09:56:16.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.690+0000 7ff964a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9601ab5c0 con 0x7ff96010f4d0 2024-03-14T09:56:16.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.690+0000 7ff964a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9601ab780 con 0x7ff96011a590 2024-03-14T09:56:16.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.690+0000 7ff95de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff96010f4d0 0x7ff9601a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.690+0000 7ff95d400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff96010f940 0x7ff9601a6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.690+0000 7ff95e800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff96011a590 0x7ff9601aae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.690+0000 7ff95d400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff96010f940 0x7ff9601a6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41230/0 (socket says 172.21.15.78:41230) 2024-03-14T09:56:16.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.690+0000 7ff95d400700 1 -- 172.21.15.78:0/2290781245 learned_addr learned my addr 172.21.15.78:0/2290781245 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:16.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.691+0000 7ff95e800700 1 -- 172.21.15.78:0/2290781245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff96010f4d0 msgr2=0x7ff9601a6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:16.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.691+0000 7ff95e800700 1 --2- 172.21.15.78:0/2290781245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff96010f4d0 0x7ff9601a6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.691+0000 7ff95e800700 1 -- 172.21.15.78:0/2290781245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff96010f940 msgr2=0x7ff9601a6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:16.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.691+0000 7ff95e800700 1 --2- 172.21.15.78:0/2290781245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff96010f940 0x7ff9601a6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.691+0000 7ff95e800700 1 -- 172.21.15.78:0/2290781245 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9500095f0 con 0x7ff96011a590 2024-03-14T09:56:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.691+0000 7ff95e800700 1 --2- 172.21.15.78:0/2290781245 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff96011a590 0x7ff9601aae10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff95800ef90 tx=0x7ff95800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.691+0000 7ff956a00700 1 -- 172.21.15.78:0/2290781245 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff958018070 con 0x7ff96011a590 2024-03-14T09:56:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.691+0000 7ff964a00700 1 -- 172.21.15.78:0/2290781245 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9601aba00 con 0x7ff96011a590 2024-03-14T09:56:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.691+0000 7ff964a00700 1 -- 172.21.15.78:0/2290781245 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9601b32a0 con 0x7ff96011a590 2024-03-14T09:56:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.691+0000 7ff956a00700 1 -- 172.21.15.78:0/2290781245 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff95800f040 con 0x7ff96011a590 2024-03-14T09:56:16.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.692+0000 7ff956a00700 1 -- 172.21.15.78:0/2290781245 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95800cc30 con 0x7ff96011a590 2024-03-14T09:56:16.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.695+0000 7ff956a00700 1 -- 172.21.15.78:0/2290781245 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7ff958004000 con 0x7ff96011a590 2024-03-14T09:56:16.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.697+0000 7ff956a00700 1 --2- 172.21.15.78:0/2290781245 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff9440a0c80 0x7ff9440a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.697+0000 7ff97860db80 1 Processor -- start 2024-03-14T09:56:16.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.697+0000 7ff956a00700 1 -- 172.21.15.78:0/2290781245 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7ff9580cc190 con 0x7ff96011a590 2024-03-14T09:56:16.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.698+0000 7ff97860db80 1 -- start start 2024-03-14T09:56:16.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.698+0000 7ff95de00700 1 --2- 172.21.15.78:0/2290781245 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff9440a0c80 0x7ff9440a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.698+0000 7ff97860db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377aa4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.698+0000 7ff97860db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aa5020 0x565377aaea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.698+0000 7ff97860db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aaef40 0x565377ab1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.698+0000 7ff97860db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653779fda10 con 0x565377aaef40 2024-03-14T09:56:16.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.698+0000 7ff97860db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653779fb3d0 con 0x565377aa4700 2024-03-14T09:56:16.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.698+0000 7ff97860db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653779f6cc0 con 0x565377aa5020 2024-03-14T09:56:16.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.698+0000 7ff95de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377aa4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.698+0000 7ff95e800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aaef40 0x565377ab1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.698+0000 7ff95de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377aa4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:56524/0 (socket says 172.21.15.78:56524) 2024-03-14T09:56:16.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.698+0000 7ff95de00700 1 -- 172.21.15.78:0/4058488260 learned_addr learned my addr 172.21.15.78:0/4058488260 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.698+0000 7ff95d400700 1 --2- 172.21.15.78:0/4058488260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aa5020 0x565377aaea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.699+0000 7ff95de00700 1 --2- 172.21.15.78:0/2290781245 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff9440a0c80 0x7ff9440a3140 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff95000b2e0 tx=0x7ff950005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.699+0000 7ff95e800700 1 -- 172.21.15.78:0/4058488260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aa5020 msgr2=0x565377aaea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.699+0000 7ff95e800700 1 --2- 172.21.15.78:0/4058488260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aa5020 0x565377aaea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.699+0000 7ff95e800700 1 -- 172.21.15.78:0/4058488260 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 msgr2=0x565377aa4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.699+0000 7ff95e800700 1 --2- 172.21.15.78:0/4058488260 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377aa4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.699+0000 7ff95e800700 1 -- 172.21.15.78:0/4058488260 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565377878430 con 0x565377aaef40 2024-03-14T09:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.699+0000 7ff95de00700 1 --2- 172.21.15.78:0/4058488260 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377aa4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:56:16.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.699+0000 7ff95e800700 1 --2- 172.21.15.78:0/4058488260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aaef40 0x565377ab1330 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff9580c9530 tx=0x7ff958099c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3810efbadfc70a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.699+0000 7ff94b000700 1 -- 172.21.15.78:0/4058488260 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95809a0a0 con 0x565377aaef40 2024-03-14T09:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.699+0000 7ff94b000700 1 -- 172.21.15.78:0/4058488260 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff95809ded0 con 0x565377aaef40 2024-03-14T09:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.699+0000 7ff94b000700 1 -- 172.21.15.78:0/4058488260 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9580c97b0 con 0x565377aaef40 2024-03-14T09:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.700+0000 7ff97860db80 1 -- 172.21.15.78:0/4058488260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aaef40 msgr2=0x565377ab1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.700+0000 7ff97860db80 1 --2- 172.21.15.78:0/4058488260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aaef40 0x565377ab1330 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff9580c9530 tx=0x7ff958099c60 comp rx=0 tx=0).stop 2024-03-14T09:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.700+0000 7ff97860db80 1 -- 172.21.15.78:0/4058488260 shutdown_connections 2024-03-14T09:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.700+0000 7ff97860db80 1 --2- 172.21.15.78:0/4058488260 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377aa4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.700+0000 7ff97860db80 1 --2- 172.21.15.78:0/4058488260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aaef40 0x565377ab1330 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.700+0000 7ff97860db80 1 --2- 172.21.15.78:0/4058488260 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aa5020 0x565377aaea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.700+0000 7ff97860db80 1 -- 172.21.15.78:0/4058488260 >> 172.21.15.78:0/4058488260 conn(0x565377a1a2a0 msgr2=0x565377a9fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.700+0000 7ff97860db80 1 -- 172.21.15.78:0/4058488260 shutdown_connections 2024-03-14T09:56:16.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.700+0000 7ff97860db80 1 -- 172.21.15.78:0/4058488260 wait complete. 2024-03-14T09:56:16.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.701+0000 7ff97860db80 1 -- start start 2024-03-14T09:56:16.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.701+0000 7ff97860db80 1 Processor -- start 2024-03-14T09:56:16.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.702+0000 7ff97860db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377aae960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.702+0000 7ff97860db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aa5020 0x565377aabce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.702+0000 7ff97860db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aaef40 0x565377aac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.702+0000 7ff97860db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x565377992e00 con 0x565377aaef40 2024-03-14T09:56:16.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.702+0000 7ff97860db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x565377992aa0 con 0x565377aa4700 2024-03-14T09:56:16.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.702+0000 7ff97860db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56537787a720 con 0x565377aa5020 2024-03-14T09:56:16.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.702+0000 7ff97860db80 10 client.0 _ll_register_callbacks cb 0x5653777fe420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:16.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.702+0000 7ff95de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377aae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.702+0000 7ff95de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377aae960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:56536/0 (socket says 172.21.15.78:56536) 2024-03-14T09:56:16.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.702+0000 7ff95de00700 1 -- 172.21.15.78:0/22336492 learned_addr learned my addr 172.21.15.78:0/22336492 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:16.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.703+0000 7ff95d400700 1 --2- 172.21.15.78:0/22336492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aa5020 0x565377aabce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.703+0000 7ff95e800700 1 --2- 172.21.15.78:0/22336492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aaef40 0x565377aac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.703+0000 7ff95de00700 1 -- 172.21.15.78:0/22336492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aa5020 msgr2=0x565377aabce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:16.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.703+0000 7ff95de00700 1 --2- 172.21.15.78:0/22336492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aa5020 0x565377aabce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.703+0000 7ff95de00700 1 -- 172.21.15.78:0/22336492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aaef40 msgr2=0x565377aac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:16.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.703+0000 7ff95de00700 1 --2- 172.21.15.78:0/22336492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aaef40 0x565377aac2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.703+0000 7ff95de00700 1 -- 172.21.15.78:0/22336492 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff96011d300 con 0x565377aa4700 2024-03-14T09:56:16.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.703+0000 7ff95d400700 1 --2- 172.21.15.78:0/22336492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aa5020 0x565377aabce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.703+0000 7ff95de00700 1 --2- 172.21.15.78:0/22336492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377aae960 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff95000be60 tx=0x7ff950009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fe44101053ef14d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.704+0000 7ff93f400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:56:16.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.704+0000 7ff93f400700 1 -- 172.21.15.78:0/22336492 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950004d20 con 0x565377aa4700 2024-03-14T09:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.704+0000 7ff93f400700 1 -- 172.21.15.78:0/22336492 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7ff95000f460 con 0x565377aa4700 2024-03-14T09:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.704+0000 7ff97860db80 1 -- 172.21.15.78:0/22336492 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565377a977a0 con 0x565377aa4700 2024-03-14T09:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.704+0000 7ff93f400700 1 -- 172.21.15.78:0/22336492 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff950004e80 con 0x565377aa4700 2024-03-14T09:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.704+0000 7ff97860db80 10 client.26126 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.704+0000 7ff97860db80 1 -- 172.21.15.78:0/22336492 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7ff9601b32a0 con 0x565377aa4700 2024-03-14T09:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.705+0000 7ff93f400700 1 -- 172.21.15.78:0/22336492 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x565377a977a0 con 0x565377aa4700 2024-03-14T09:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.705+0000 7ff93f400700 1 -- 172.21.15.78:0/22336492 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff95002f770 con 0x565377aa4700 2024-03-14T09:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.705+0000 7ff97860db80 10 client.26126 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.705+0000 7ff97860db80 10 client.26126 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.705+0000 7ff97860db80 20 client.26126 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.705+0000 7ff97860db80 1 --2- 172.21.15.78:0/22336492 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x565377abeb00 0x565377ac0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.705+0000 7ff97860db80 4 client.26126 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.705+0000 7ff97860db80 1 -- 172.21.15.78:0/22336492 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5653779ef780 con 0x565377abeb00 2024-03-14T09:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.705+0000 7ff95d400700 1 --2- 172.21.15.78:0/22336492 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x565377abeb00 0x565377ac0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.706+0000 7ff93f400700 10 client.26126 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:16.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.706+0000 7ff95d400700 1 --2- 172.21.15.78:0/22336492 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x565377abeb00 0x565377ac0f20 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dc25c5faf754aef server_cookie=65439605db4fde34 in_seq=0 out_seq=0 2024-03-14T09:56:16.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.707+0000 7ff93f400700 1 -- 172.21.15.78:0/22336492 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5653779ef780 con 0x565377abeb00 2024-03-14T09:56:16.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.707+0000 7ff93f400700 10 client.26126 handle_command_reply: tid=0 2024-03-14T09:56:16.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.707+0000 7ff97860db80 1 client.26126 shutdown 2024-03-14T09:56:16.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.707+0000 7ff97860db80 1 -- 172.21.15.78:0/22336492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 msgr2=0x565377aae960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:16.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.707+0000 7ff97860db80 1 --2- 172.21.15.78:0/22336492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377aae960 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff95000be60 tx=0x7ff950009ca0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.707+0000 7ff97860db80 1 -- 172.21.15.78:0/22336492 shutdown_connections 2024-03-14T09:56:16.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.707+0000 7ff97860db80 1 --2- 172.21.15.78:0/22336492 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x565377abeb00 0x565377ac0f20 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.707+0000 7ff97860db80 1 --2- 172.21.15.78:0/22336492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377aae960 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.707+0000 7ff97860db80 1 --2- 172.21.15.78:0/22336492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aaef40 0x565377aac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.707+0000 7ff97860db80 1 --2- 172.21.15.78:0/22336492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aa5020 0x565377aabce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.707+0000 7ff93f400700 0 client.26126 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:16.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.707+0000 7ff97860db80 1 -- 172.21.15.78:0/22336492 >> 172.21.15.78:0/22336492 conn(0x565377a1a2a0 msgr2=0x565377a9f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:16.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.707+0000 7ff97860db80 1 -- 172.21.15.78:0/22336492 shutdown_connections 2024-03-14T09:56:16.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.707+0000 7ff97860db80 1 -- 172.21.15.78:0/22336492 wait complete. 2024-03-14T09:56:16.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.708+0000 7ff97860db80 20 client.26126 trim_cache size 0 max 16384 2024-03-14T09:56:16.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.730+0000 7ff97860db80 1 Processor -- start 2024-03-14T09:56:16.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.730+0000 7ff97860db80 1 -- start start 2024-03-14T09:56:16.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.731+0000 7ff97860db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377a9fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.731+0000 7ff97860db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aa5020 0x565377aba800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.731+0000 7ff97860db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aaef40 0x565377aa8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.731+0000 7ff97860db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x565377aacf20 con 0x565377aa5020 2024-03-14T09:56:16.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.731+0000 7ff97860db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x565377aa5b70 con 0x565377aa4700 2024-03-14T09:56:16.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.731+0000 7ff97860db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x565377aa4e60 con 0x565377aaef40 2024-03-14T09:56:16.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.731+0000 7ff95e800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aaef40 0x565377aa8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.731+0000 7ff95e800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aaef40 0x565377aa8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:55146/0 (socket says 172.21.15.78:55146) 2024-03-14T09:56:16.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.731+0000 7ff95d400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aa5020 0x565377aba800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.731+0000 7ff95e800700 1 -- 172.21.15.78:0/2299492299 learned_addr learned my addr 172.21.15.78:0/2299492299 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:16.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.731+0000 7ff95de00700 1 --2- 172.21.15.78:0/2299492299 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377a9fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.731+0000 7ff95e800700 1 -- 172.21.15.78:0/2299492299 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 msgr2=0x565377a9fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:16.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.731+0000 7ff95e800700 1 --2- 172.21.15.78:0/2299492299 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377a9fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.731+0000 7ff95e800700 1 -- 172.21.15.78:0/2299492299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aa5020 msgr2=0x565377aba800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:16.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.731+0000 7ff95e800700 1 --2- 172.21.15.78:0/2299492299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aa5020 0x565377aba800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.731+0000 7ff95e800700 1 -- 172.21.15.78:0/2299492299 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565377824be0 con 0x565377aaef40 2024-03-14T09:56:16.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.732+0000 7ff95e800700 1 --2- 172.21.15.78:0/2299492299 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aaef40 0x565377aa8f20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff9580965a0 tx=0x7ff958099f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9ecb435dea68583 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:16.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.732+0000 7ff94ba00700 1 -- 172.21.15.78:0/2299492299 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9580a2490 con 0x565377aaef40 2024-03-14T09:56:16.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.732+0000 7ff94ba00700 1 -- 172.21.15.78:0/2299492299 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9580b1070 con 0x565377aaef40 2024-03-14T09:56:16.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.732+0000 7ff94ba00700 1 -- 172.21.15.78:0/2299492299 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff958005890 con 0x565377aaef40 2024-03-14T09:56:16.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.733+0000 7ff97860db80 1 -- 172.21.15.78:0/2299492299 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aaef40 msgr2=0x565377aa8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:16.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.733+0000 7ff97860db80 1 --2- 172.21.15.78:0/2299492299 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aaef40 0x565377aa8f20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff9580965a0 tx=0x7ff958099f60 comp rx=0 tx=0).stop 2024-03-14T09:56:16.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.733+0000 7ff97860db80 1 -- 172.21.15.78:0/2299492299 shutdown_connections 2024-03-14T09:56:16.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.733+0000 7ff97860db80 1 --2- 172.21.15.78:0/2299492299 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377a9fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.733+0000 7ff97860db80 1 --2- 172.21.15.78:0/2299492299 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aaef40 0x565377aa8f20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.733+0000 7ff97860db80 1 --2- 172.21.15.78:0/2299492299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aa5020 0x565377aba800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.733+0000 7ff97860db80 1 -- 172.21.15.78:0/2299492299 >> 172.21.15.78:0/2299492299 conn(0x565377a1a2a0 msgr2=0x565377aba0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:16.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.733+0000 7ff97860db80 1 -- 172.21.15.78:0/2299492299 shutdown_connections 2024-03-14T09:56:16.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.733+0000 7ff97860db80 1 -- 172.21.15.78:0/2299492299 wait complete. 2024-03-14T09:56:16.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.733+0000 7ff97860db80 1 -- start start 2024-03-14T09:56:16.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.733+0000 7ff97860db80 1 Processor -- start 2024-03-14T09:56:16.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff97860db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377ab4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff97860db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aa5020 0x565377ab4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff97860db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aaef40 0x565377b10630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff97860db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x565377ab5130 con 0x565377aa5020 2024-03-14T09:56:16.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff97860db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x565377ab5270 con 0x565377aa4700 2024-03-14T09:56:16.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff97860db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x565377b10b70 con 0x565377aaef40 2024-03-14T09:56:16.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff97860db80 10 client.0 _ll_register_callbacks cb 0x7ff95000ba60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:16.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff95de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377ab4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff95de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377ab4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:56548/0 (socket says 172.21.15.78:56548) 2024-03-14T09:56:16.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff95de00700 1 -- 172.21.15.78:0/2060901492 learned_addr learned my addr 172.21.15.78:0/2060901492 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:16.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff95e800700 1 --2- 172.21.15.78:0/2060901492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aaef40 0x565377b10630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff95d400700 1 --2- 172.21.15.78:0/2060901492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aa5020 0x565377ab4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff95e800700 1 -- 172.21.15.78:0/2060901492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 msgr2=0x565377ab4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:16.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff95e800700 1 --2- 172.21.15.78:0/2060901492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377ab4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff95e800700 1 -- 172.21.15.78:0/2060901492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aa5020 msgr2=0x565377ab4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:16.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff95e800700 1 --2- 172.21.15.78:0/2060901492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aa5020 0x565377ab4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.734+0000 7ff95e800700 1 -- 172.21.15.78:0/2060901492 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565377878430 con 0x565377aaef40 2024-03-14T09:56:16.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.735+0000 7ff95de00700 1 --2- 172.21.15.78:0/2060901492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377ab4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:56:16.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.735+0000 7ff95e800700 1 --2- 172.21.15.78:0/2060901492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aaef40 0x565377b10630 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff9580a1f60 tx=0x7ff9580a2490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc77ed98794319ce server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:16.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.735+0000 7ff93f400700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:56:16.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.735+0000 7ff93f400700 1 -- 172.21.15.78:0/2060901492 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95809a0a0 con 0x565377aaef40 2024-03-14T09:56:16.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.735+0000 7ff97860db80 1 -- 172.21.15.78:0/2060901492 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff95809ded0 con 0x565377aaef40 2024-03-14T09:56:16.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.735+0000 7ff93f400700 1 -- 172.21.15.78:0/2060901492 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7ff958005890 con 0x565377aaef40 2024-03-14T09:56:16.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.735+0000 7ff93f400700 1 -- 172.21.15.78:0/2060901492 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff958005a50 con 0x565377aaef40 2024-03-14T09:56:16.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.735+0000 7ff97860db80 10 client.26143 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:16.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.735+0000 7ff97860db80 1 -- 172.21.15.78:0/2060901492 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5653779bcf20 con 0x565377aaef40 2024-03-14T09:56:16.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.735+0000 7ff93f400700 1 -- 172.21.15.78:0/2060901492 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff95809ded0 con 0x565377aaef40 2024-03-14T09:56:16.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.736+0000 7ff93f400700 1 -- 172.21.15.78:0/2060901492 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff958082070 con 0x565377aaef40 2024-03-14T09:56:16.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.736+0000 7ff97860db80 10 client.26143 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:16.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.736+0000 7ff97860db80 10 client.26143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:56:16.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.736+0000 7ff97860db80 20 client.26143 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:16.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.736+0000 7ff97860db80 1 --2- 172.21.15.78:0/2060901492 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x565377b11ea0 0x565377b14290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:16.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.736+0000 7ff97860db80 4 client.26143 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T09:56:16.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.736+0000 7ff97860db80 1 -- 172.21.15.78:0/2060901492 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x565377845d70 con 0x565377b11ea0 2024-03-14T09:56:16.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.736+0000 7ff95de00700 1 --2- 172.21.15.78:0/2060901492 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x565377b11ea0 0x565377b14290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:16.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.737+0000 7ff95de00700 1 --2- 172.21.15.78:0/2060901492 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x565377b11ea0 0x565377b14290 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba86cb4d6502396c server_cookie=8b538ca907237e4f in_seq=0 out_seq=0 2024-03-14T09:56:16.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.737+0000 7ff93f400700 10 client.26143 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:16.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.738+0000 7ff93f400700 1 -- 172.21.15.78:0/2060901492 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x565377845d70 con 0x565377b11ea0 2024-03-14T09:56:16.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.738+0000 7ff93f400700 10 client.26143 handle_command_reply: tid=0 2024-03-14T09:56:16.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.738+0000 7ff97860db80 1 client.26143 shutdown 2024-03-14T09:56:16.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.738+0000 7ff97860db80 1 -- 172.21.15.78:0/2060901492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aaef40 msgr2=0x565377b10630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:16.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.738+0000 7ff97860db80 1 --2- 172.21.15.78:0/2060901492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aaef40 0x565377b10630 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff9580a1f60 tx=0x7ff9580a2490 comp rx=0 tx=0).stop 2024-03-14T09:56:16.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.738+0000 7ff97860db80 1 -- 172.21.15.78:0/2060901492 shutdown_connections 2024-03-14T09:56:16.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.738+0000 7ff97860db80 1 --2- 172.21.15.78:0/2060901492 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x565377b11ea0 0x565377b14290 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.738+0000 7ff93f400700 0 client.26143 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:16.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.738+0000 7ff97860db80 1 --2- 172.21.15.78:0/2060901492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377aa4700 0x565377ab4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.738+0000 7ff97860db80 1 --2- 172.21.15.78:0/2060901492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377aaef40 0x565377b10630 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.739+0000 7ff97860db80 1 --2- 172.21.15.78:0/2060901492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377aa5020 0x565377ab4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.739+0000 7ff97860db80 1 -- 172.21.15.78:0/2060901492 >> 172.21.15.78:0/2060901492 conn(0x565377a1a2a0 msgr2=0x565377aaba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:16.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.739+0000 7ff97860db80 1 -- 172.21.15.78:0/2060901492 shutdown_connections 2024-03-14T09:56:16.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.739+0000 7ff97860db80 1 -- 172.21.15.78:0/2060901492 wait complete. 2024-03-14T09:56:16.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.739+0000 7ff97860db80 20 client.26143 trim_cache size 0 max 16384 2024-03-14T09:56:16.741 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:56:16.742 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T09:56:16.742 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "aa9055ad-10bd-49b6-b26f-f9b5affb7831", 2024-03-14T09:56:16.742 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T09:56:16.742 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:56:16.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.740+0000 7ff964a00700 1 -- 172.21.15.78:0/2290781245 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff9440a0c80 msgr2=0x7ff9440a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:16.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.740+0000 7ff964a00700 1 --2- 172.21.15.78:0/2290781245 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff9440a0c80 0x7ff9440a3140 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff95000b2e0 tx=0x7ff950005fd0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.740+0000 7ff964a00700 1 -- 172.21.15.78:0/2290781245 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff96011a590 msgr2=0x7ff9601aae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:16.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.740+0000 7ff964a00700 1 --2- 172.21.15.78:0/2290781245 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff96011a590 0x7ff9601aae10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff95800ef90 tx=0x7ff95800c3a0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.740+0000 7ff964a00700 1 -- 172.21.15.78:0/2290781245 shutdown_connections 2024-03-14T09:56:16.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.740+0000 7ff964a00700 1 --2- 172.21.15.78:0/2290781245 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff9440a0c80 0x7ff9440a3140 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.740+0000 7ff964a00700 1 --2- 172.21.15.78:0/2290781245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff96010f4d0 0x7ff9601a6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.740+0000 7ff964a00700 1 --2- 172.21.15.78:0/2290781245 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff96011a590 0x7ff9601aae10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.740+0000 7ff964a00700 1 --2- 172.21.15.78:0/2290781245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff96010f940 0x7ff9601a6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:16.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.740+0000 7ff964a00700 1 -- 172.21.15.78:0/2290781245 >> 172.21.15.78:0/2290781245 conn(0x7ff96006bdc0 msgr2=0x7ff960118580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:16.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.740+0000 7ff964a00700 1 -- 172.21.15.78:0/2290781245 shutdown_connections 2024-03-14T09:56:16.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:16.740+0000 7ff964a00700 1 -- 172.21.15.78:0/2290781245 wait complete. 2024-03-14T09:56:16.762 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:56:17.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.020+0000 7f0681c00700 1 -- 172.21.15.78:0/3306932694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f067c068a60 msgr2=0x7f067c105930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:17.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.020+0000 7f0681c00700 1 --2- 172.21.15.78:0/3306932694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f067c068a60 0x7f067c105930 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f06680099c0 tx=0x7f0668009cd0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.020+0000 7f0681c00700 1 -- 172.21.15.78:0/3306932694 shutdown_connections 2024-03-14T09:56:17.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.020+0000 7f0681c00700 1 --2- 172.21.15.78:0/3306932694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f067c106060 0x7f067c10dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.021+0000 7f0681c00700 1 --2- 172.21.15.78:0/3306932694 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f067c0680b0 0x7f067c068490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.021+0000 7f0681c00700 1 --2- 172.21.15.78:0/3306932694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f067c068a60 0x7f067c105930 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.021+0000 7f0681c00700 1 -- 172.21.15.78:0/3306932694 >> 172.21.15.78:0/3306932694 conn(0x7f067c076d80 msgr2=0x7f067c077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:17.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.021+0000 7f0681c00700 1 -- 172.21.15.78:0/3306932694 shutdown_connections 2024-03-14T09:56:17.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.021+0000 7f0681c00700 1 -- 172.21.15.78:0/3306932694 wait complete. 2024-03-14T09:56:17.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.022+0000 7f0681c00700 1 Processor -- start 2024-03-14T09:56:17.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.022+0000 7f0681c00700 1 -- start start 2024-03-14T09:56:17.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.023+0000 7f0681c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f067c0680b0 0x7f067c19f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.023+0000 7f0681c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f067c106060 0x7f067c19fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.023+0000 7f0681c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f067c199bf0 0x7f067c19a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.023+0000 7f0681c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c19a640 con 0x7f067c199bf0 2024-03-14T09:56:17.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.023+0000 7f0681c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c19a7b0 con 0x7f067c106060 2024-03-14T09:56:17.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.023+0000 7f0681c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c19a970 con 0x7f067c0680b0 2024-03-14T09:56:17.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.023+0000 7f067aa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f067c0680b0 0x7f067c19f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:17.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.023+0000 7f067aa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f067c0680b0 0x7f067c19f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:55174/0 (socket says 172.21.15.78:55174) 2024-03-14T09:56:17.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.023+0000 7f067aa00700 1 -- 172.21.15.78:0/367447345 learned_addr learned my addr 172.21.15.78:0/367447345 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:17.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.023+0000 7f067a000700 1 --2- 172.21.15.78:0/367447345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f067c106060 0x7f067c19fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:17.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.023+0000 7f067b400700 1 --2- 172.21.15.78:0/367447345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f067c199bf0 0x7f067c19a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:17.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.023+0000 7f067aa00700 1 -- 172.21.15.78:0/367447345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f067c106060 msgr2=0x7f067c19fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:17.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.023+0000 7f067aa00700 1 --2- 172.21.15.78:0/367447345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f067c106060 0x7f067c19fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.023+0000 7f067aa00700 1 -- 172.21.15.78:0/367447345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f067c199bf0 msgr2=0x7f067c19a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:17.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.023+0000 7f067aa00700 1 --2- 172.21.15.78:0/367447345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f067c199bf0 0x7f067c19a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.023+0000 7f067aa00700 1 -- 172.21.15.78:0/367447345 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06680095f0 con 0x7f067c0680b0 2024-03-14T09:56:17.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.024+0000 7f067aa00700 1 --2- 172.21.15.78:0/367447345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f067c0680b0 0x7f067c19f530 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f066c00e9a0 tx=0x7f066c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:17.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.024+0000 7f0673400700 1 -- 172.21.15.78:0/367447345 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c009790 con 0x7f067c0680b0 2024-03-14T09:56:17.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.024+0000 7f0673400700 1 -- 172.21.15.78:0/367447345 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f066c010430 con 0x7f067c0680b0 2024-03-14T09:56:17.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.024+0000 7f0673400700 1 -- 172.21.15.78:0/367447345 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c018680 con 0x7f067c0680b0 2024-03-14T09:56:17.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.024+0000 7f0681c00700 1 -- 172.21.15.78:0/367447345 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067c1a63a0 con 0x7f067c0680b0 2024-03-14T09:56:17.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.025+0000 7f0681c00700 1 -- 172.21.15.78:0/367447345 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067c1a6830 con 0x7f067c0680b0 2024-03-14T09:56:17.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.030+0000 7f0673400700 1 -- 172.21.15.78:0/367447345 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f066c0105a0 con 0x7f067c0680b0 2024-03-14T09:56:17.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.031+0000 7f069573bb80 1 Processor -- start 2024-03-14T09:56:17.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.032+0000 7f0673400700 1 --2- 172.21.15.78:0/367447345 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f06600a0c00 0x7f06600a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.032+0000 7f0673400700 1 -- 172.21.15.78:0/367447345 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f066c014070 con 0x7f067c0680b0 2024-03-14T09:56:17.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.032+0000 7f069573bb80 1 -- start start 2024-03-14T09:56:17.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.033+0000 7f067b400700 1 --2- 172.21.15.78:0/367447345 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f06600a0c00 0x7f06600a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:17.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.034+0000 7f069573bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 0x5571194b6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.034+0000 7f069573bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 0x5571194c0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.034+0000 7f069573bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194c2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.034+0000 7f069573bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557119406990 con 0x5571194b67b0 2024-03-14T09:56:17.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.034+0000 7f069573bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557119404350 con 0x5571194c06d0 2024-03-14T09:56:17.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.034+0000 7f069573bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571193ffc40 con 0x5571194b5e90 2024-03-14T09:56:17.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.034+0000 7f067aa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 0x5571194b6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:17.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.034+0000 7f067aa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 0x5571194b6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:55186/0 (socket says 172.21.15.78:55186) 2024-03-14T09:56:17.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.034+0000 7f067aa00700 1 -- 172.21.15.78:0/1280346114 learned_addr learned my addr 172.21.15.78:0/1280346114 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:17.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.035+0000 7f067a000700 1 --2- 172.21.15.78:0/1280346114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 0x5571194c0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:17.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.035+0000 7f067aa00700 1 -- 172.21.15.78:0/1280346114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 msgr2=0x5571194c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:17.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.035+0000 7f067aa00700 1 --2- 172.21.15.78:0/1280346114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194c2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.035+0000 7f067aa00700 1 -- 172.21.15.78:0/1280346114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 msgr2=0x5571194c0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:17.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.035+0000 7f067b400700 1 --2- 172.21.15.78:0/367447345 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f06600a0c00 0x7f06600a30c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f067400f590 tx=0x7f067400b040 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:17.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.035+0000 7f067aa00700 1 --2- 172.21.15.78:0/1280346114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 0x5571194c0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.035+0000 7f067aa00700 1 -- 172.21.15.78:0/1280346114 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571192813b0 con 0x5571194b5e90 2024-03-14T09:56:17.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.035+0000 7f067a000700 1 --2- 172.21.15.78:0/1280346114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 0x5571194c0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:56:17.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.035+0000 7f067aa00700 1 --2- 172.21.15.78:0/1280346114 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 0x5571194b6270 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f066c0a18e0 tx=0x7f066c099890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1ead70c39f0a4de server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:17.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.036+0000 7f0667e00700 1 -- 172.21.15.78:0/1280346114 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c09d7c0 con 0x5571194b5e90 2024-03-14T09:56:17.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.036+0000 7f0667e00700 1 -- 172.21.15.78:0/1280346114 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f066c0a1e10 con 0x5571194b5e90 2024-03-14T09:56:17.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.036+0000 7f0667e00700 1 -- 172.21.15.78:0/1280346114 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c0990e0 con 0x5571194b5e90 2024-03-14T09:56:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.037+0000 7f069573bb80 1 -- 172.21.15.78:0/1280346114 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 msgr2=0x5571194b6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.037+0000 7f069573bb80 1 --2- 172.21.15.78:0/1280346114 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 0x5571194b6270 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f066c0a18e0 tx=0x7f066c099890 comp rx=0 tx=0).stop 2024-03-14T09:56:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.037+0000 7f069573bb80 1 -- 172.21.15.78:0/1280346114 shutdown_connections 2024-03-14T09:56:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.037+0000 7f069573bb80 1 --2- 172.21.15.78:0/1280346114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194c2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.037+0000 7f069573bb80 1 --2- 172.21.15.78:0/1280346114 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 0x5571194b6270 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.037+0000 7f069573bb80 1 --2- 172.21.15.78:0/1280346114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 0x5571194c0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.037+0000 7f069573bb80 1 -- 172.21.15.78:0/1280346114 >> 172.21.15.78:0/1280346114 conn(0x557119423220 msgr2=0x5571194a8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.037+0000 7f069573bb80 1 -- 172.21.15.78:0/1280346114 shutdown_connections 2024-03-14T09:56:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.037+0000 7f069573bb80 1 -- 172.21.15.78:0/1280346114 wait complete. 2024-03-14T09:56:17.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.038+0000 7f069573bb80 1 -- start start 2024-03-14T09:56:17.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.038+0000 7f069573bb80 1 Processor -- start 2024-03-14T09:56:17.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.039+0000 7f069573bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b5e90 0x5571194c0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.039+0000 7f069573bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b67b0 0x5571194bd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.039+0000 7f069573bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194bdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.039+0000 7f069573bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55711939bd80 con 0x5571194b5e90 2024-03-14T09:56:17.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.039+0000 7f069573bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55711939ba20 con 0x5571194c06d0 2024-03-14T09:56:17.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.039+0000 7f069573bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571192836a0 con 0x5571194b67b0 2024-03-14T09:56:17.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.039+0000 7f069573bb80 10 client.0 _ll_register_callbacks cb 0x5571192073a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:17.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.039+0000 7f067a000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b67b0 0x5571194bd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:17.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.039+0000 7f067a000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b67b0 0x5571194bd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:55192/0 (socket says 172.21.15.78:55192) 2024-03-14T09:56:17.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.039+0000 7f067a000700 1 -- 172.21.15.78:0/455485865 learned_addr learned my addr 172.21.15.78:0/455485865 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:17.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.039+0000 7f067b400700 1 --2- 172.21.15.78:0/455485865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194bdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.040+0000 7f067a000700 1 -- 172.21.15.78:0/455485865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 msgr2=0x5571194bdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.040+0000 7f067a000700 1 --2- 172.21.15.78:0/455485865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194bdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.040+0000 7f067a000700 1 -- 172.21.15.78:0/455485865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b5e90 msgr2=0x5571194c0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.040+0000 7f067a000700 1 --2- 172.21.15.78:0/455485865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b5e90 0x5571194c0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.040+0000 7f067a000700 1 -- 172.21.15.78:0/455485865 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067c1a6830 con 0x5571194b67b0 2024-03-14T09:56:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.040+0000 7f067a000700 1 --2- 172.21.15.78:0/455485865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b67b0 0x5571194bd450 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0668009990 tx=0x7f0668005750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3d8edcb7e67ab96 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.040+0000 7f0664c00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:56:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.040+0000 7f0664c00700 1 -- 172.21.15.78:0/455485865 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066801d070 con 0x5571194b67b0 2024-03-14T09:56:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.040+0000 7f0664c00700 1 -- 172.21.15.78:0/455485865 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f066800bba0 con 0x5571194b67b0 2024-03-14T09:56:17.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.040+0000 7f069573bb80 1 -- 172.21.15.78:0/455485865 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5571194a0720 con 0x5571194b67b0 2024-03-14T09:56:17.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.047+0000 7f0664c00700 1 -- 172.21.15.78:0/455485865 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f066800bd60 con 0x5571194b67b0 2024-03-14T09:56:17.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.047+0000 7f069573bb80 10 client.26155 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:17.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.047+0000 7f069573bb80 1 -- 172.21.15.78:0/455485865 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55711922db60 con 0x5571194b67b0 2024-03-14T09:56:17.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.047+0000 7f0664c00700 1 -- 172.21.15.78:0/455485865 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5571194a0720 con 0x5571194b67b0 2024-03-14T09:56:17.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.047+0000 7f0664c00700 1 -- 172.21.15.78:0/455485865 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0668017430 con 0x5571194b67b0 2024-03-14T09:56:17.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.047+0000 7f069573bb80 10 client.26155 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:17.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.047+0000 7f069573bb80 10 client.26155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:56:17.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.047+0000 7f069573bb80 20 client.26155 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:17.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.047+0000 7f069573bb80 1 --2- 172.21.15.78:0/455485865 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5571194d0260 0x5571194d2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.047+0000 7f069573bb80 4 client.26155 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:56:17.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.047+0000 7f069573bb80 1 -- 172.21.15.78:0/455485865 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5571193f8700 con 0x5571194d0260 2024-03-14T09:56:17.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.048+0000 7f067aa00700 1 --2- 172.21.15.78:0/455485865 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5571194d0260 0x5571194d2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:17.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.049+0000 7f067aa00700 1 --2- 172.21.15.78:0/455485865 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5571194d0260 0x5571194d2680 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e8f71c95cb2ba66 server_cookie=24ec3b510ea44196 in_seq=0 out_seq=0 2024-03-14T09:56:17.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.049+0000 7f0664c00700 10 client.26155 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:17.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.050+0000 7f0664c00700 1 -- 172.21.15.78:0/455485865 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5571193f8700 con 0x5571194d0260 2024-03-14T09:56:17.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.050+0000 7f0664c00700 10 client.26155 handle_command_reply: tid=0 2024-03-14T09:56:17.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.050+0000 7f069573bb80 1 client.26155 shutdown 2024-03-14T09:56:17.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.051+0000 7f069573bb80 1 -- 172.21.15.78:0/455485865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b67b0 msgr2=0x5571194bd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:17.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.051+0000 7f069573bb80 1 --2- 172.21.15.78:0/455485865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b67b0 0x5571194bd450 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0668009990 tx=0x7f0668005750 comp rx=0 tx=0).stop 2024-03-14T09:56:17.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.051+0000 7f069573bb80 1 -- 172.21.15.78:0/455485865 shutdown_connections 2024-03-14T09:56:17.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.051+0000 7f069573bb80 1 --2- 172.21.15.78:0/455485865 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5571194d0260 0x5571194d2680 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.051+0000 7f0664c00700 0 client.26155 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:17.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.051+0000 7f069573bb80 1 --2- 172.21.15.78:0/455485865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194bdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.051+0000 7f069573bb80 1 --2- 172.21.15.78:0/455485865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b5e90 0x5571194c0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.051+0000 7f069573bb80 1 --2- 172.21.15.78:0/455485865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b67b0 0x5571194bd450 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.051+0000 7f069573bb80 1 -- 172.21.15.78:0/455485865 >> 172.21.15.78:0/455485865 conn(0x557119423220 msgr2=0x5571194a8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:17.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.052+0000 7f069573bb80 1 -- 172.21.15.78:0/455485865 shutdown_connections 2024-03-14T09:56:17.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.052+0000 7f069573bb80 1 -- 172.21.15.78:0/455485865 wait complete. 2024-03-14T09:56:17.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.052+0000 7f069573bb80 20 client.26155 trim_cache size 0 max 16384 2024-03-14T09:56:17.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.080+0000 7f069573bb80 1 Processor -- start 2024-03-14T09:56:17.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.080+0000 7f069573bb80 1 -- start start 2024-03-14T09:56:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.080+0000 7f069573bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 0x5571194c7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.080+0000 7f069573bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 0x5571194cb930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.080+0000 7f069573bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.080+0000 7f069573bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571194d2a10 con 0x5571194b67b0 2024-03-14T09:56:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.080+0000 7f069573bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571194b8310 con 0x5571194c06d0 2024-03-14T09:56:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.080+0000 7f069573bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571194a9770 con 0x5571194b5e90 2024-03-14T09:56:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.081+0000 7f067b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.082+0000 7f067b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194cbe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:56590/0 (socket says 172.21.15.78:56590) 2024-03-14T09:56:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.082+0000 7f067b400700 1 -- 172.21.15.78:0/526082988 learned_addr learned my addr 172.21.15.78:0/526082988 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.082+0000 7f067a000700 1 --2- 172.21.15.78:0/526082988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 0x5571194cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.082+0000 7f067b400700 1 -- 172.21.15.78:0/526082988 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 msgr2=0x5571194c7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.082+0000 7f067b400700 1 --2- 172.21.15.78:0/526082988 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 0x5571194c7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.082+0000 7f067b400700 1 -- 172.21.15.78:0/526082988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 msgr2=0x5571194cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.082+0000 7f067b400700 1 --2- 172.21.15.78:0/526082988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 0x5571194cb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.082+0000 7f067b400700 1 -- 172.21.15.78:0/526082988 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571193c5ea0 con 0x5571194c06d0 2024-03-14T09:56:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.082+0000 7f067a000700 1 --2- 172.21.15.78:0/526082988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 0x5571194cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:56:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.082+0000 7f067b400700 1 --2- 172.21.15.78:0/526082988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194cbe70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0674009400 tx=0x7f0674004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9da60329859d93bf server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.083+0000 7f0670c00700 1 -- 172.21.15.78:0/526082988 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067400e040 con 0x5571194c06d0 2024-03-14T09:56:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.083+0000 7f0670c00700 1 -- 172.21.15.78:0/526082988 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f067400f040 con 0x5571194c06d0 2024-03-14T09:56:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.083+0000 7f0670c00700 1 -- 172.21.15.78:0/526082988 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674017440 con 0x5571194c06d0 2024-03-14T09:56:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.084+0000 7f069573bb80 1 -- 172.21.15.78:0/526082988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 msgr2=0x5571194cbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:17.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.084+0000 7f069573bb80 1 --2- 172.21.15.78:0/526082988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194cbe70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0674009400 tx=0x7f0674004b40 comp rx=0 tx=0).stop 2024-03-14T09:56:17.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.084+0000 7f069573bb80 1 -- 172.21.15.78:0/526082988 shutdown_connections 2024-03-14T09:56:17.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.084+0000 7f069573bb80 1 --2- 172.21.15.78:0/526082988 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194cbe70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.084+0000 7f069573bb80 1 --2- 172.21.15.78:0/526082988 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 0x5571194c7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.084+0000 7f069573bb80 1 --2- 172.21.15.78:0/526082988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 0x5571194cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.084+0000 7f069573bb80 1 -- 172.21.15.78:0/526082988 >> 172.21.15.78:0/526082988 conn(0x557119423220 msgr2=0x5571194b87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:17.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.085+0000 7f069573bb80 1 -- 172.21.15.78:0/526082988 shutdown_connections 2024-03-14T09:56:17.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.088+0000 7f069573bb80 1 -- 172.21.15.78:0/526082988 wait complete. 2024-03-14T09:56:17.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.089+0000 7f069573bb80 1 -- start start 2024-03-14T09:56:17.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.090+0000 7f069573bb80 1 Processor -- start 2024-03-14T09:56:17.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.091+0000 7f069573bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 0x5571194c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.091+0000 7f069573bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 0x5571194c5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.091+0000 7f069573bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.091+0000 7f069573bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571194c6760 con 0x5571194b67b0 2024-03-14T09:56:17.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.091+0000 7f069573bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557119521d90 con 0x5571194c06d0 2024-03-14T09:56:17.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.091+0000 7f069573bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557119521ed0 con 0x5571194b5e90 2024-03-14T09:56:17.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.092+0000 7f069573bb80 10 client.0 _ll_register_callbacks cb 0x7f0668009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:17.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.091+0000 7f067aa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 0x5571194c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:17.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.092+0000 7f067a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 0x5571194c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:17.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.092+0000 7f067b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:17.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.092+0000 7f067a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 0x5571194c5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41332/0 (socket says 172.21.15.78:41332) 2024-03-14T09:56:17.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.092+0000 7f067a000700 1 -- 172.21.15.78:0/151593639 learned_addr learned my addr 172.21.15.78:0/151593639 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:17.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.092+0000 7f067a000700 1 -- 172.21.15.78:0/151593639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 msgr2=0x5571194c73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:17.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.092+0000 7f067a000700 1 --2- 172.21.15.78:0/151593639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 0x5571194c73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.092+0000 7f067a000700 1 -- 172.21.15.78:0/151593639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 msgr2=0x5571194c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:17.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.092+0000 7f067a000700 1 --2- 172.21.15.78:0/151593639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.092+0000 7f067a000700 1 -- 172.21.15.78:0/151593639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0674009080 con 0x5571194b67b0 2024-03-14T09:56:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.092+0000 7f067aa00700 1 --2- 172.21.15.78:0/151593639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 0x5571194c73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:56:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.093+0000 7f067a000700 1 --2- 172.21.15.78:0/151593639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 0x5571194c5940 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0668000c00 tx=0x7f0668017720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e048483cd1df8f2e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.093+0000 7f0664c00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:56:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.093+0000 7f0664c00700 1 -- 172.21.15.78:0/151593639 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066801d070 con 0x5571194b67b0 2024-03-14T09:56:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.093+0000 7f069573bb80 1 -- 172.21.15.78:0/151593639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f066c0a1e10 con 0x5571194b67b0 2024-03-14T09:56:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.093+0000 7f0664c00700 1 -- 172.21.15.78:0/151593639 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f0668000e40 con 0x5571194b67b0 2024-03-14T09:56:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.093+0000 7f0664c00700 1 -- 172.21.15.78:0/151593639 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0668005c40 con 0x5571194b67b0 2024-03-14T09:56:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.093+0000 7f069573bb80 10 client.16137 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.093+0000 7f069573bb80 1 -- 172.21.15.78:0/151593639 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557119285cd0 con 0x5571194b67b0 2024-03-14T09:56:17.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.094+0000 7f0664c00700 1 -- 172.21.15.78:0/151593639 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f066c0a1e10 con 0x5571194b67b0 2024-03-14T09:56:17.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.094+0000 7f0664c00700 1 -- 172.21.15.78:0/151593639 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0668003680 con 0x5571194b67b0 2024-03-14T09:56:17.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.094+0000 7f069573bb80 10 client.16137 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:17.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.094+0000 7f069573bb80 10 client.16137 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:56:17.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.094+0000 7f069573bb80 20 client.16137 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:17.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.094+0000 7f069573bb80 1 --2- 172.21.15.78:0/151593639 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557119523200 0x557119525620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:17.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.094+0000 7f069573bb80 4 client.16137 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:56:17.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.094+0000 7f069573bb80 1 -- 172.21.15.78:0/151593639 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55711924ecf0 con 0x557119523200 2024-03-14T09:56:17.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.094+0000 7f067aa00700 1 --2- 172.21.15.78:0/151593639 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557119523200 0x557119525620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:17.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.095+0000 7f0664c00700 10 client.16137 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:17.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.095+0000 7f067aa00700 1 --2- 172.21.15.78:0/151593639 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557119523200 0x557119525620 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46ac1a667f4322a server_cookie=79d8898506f8d66 in_seq=0 out_seq=0 2024-03-14T09:56:17.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.095+0000 7f0664c00700 1 -- 172.21.15.78:0/151593639 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55711924ecf0 con 0x557119523200 2024-03-14T09:56:17.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.095+0000 7f0664c00700 10 client.16137 handle_command_reply: tid=0 2024-03-14T09:56:17.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.095+0000 7f069573bb80 1 client.16137 shutdown 2024-03-14T09:56:17.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.097+0000 7f069573bb80 1 -- 172.21.15.78:0/151593639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 msgr2=0x5571194c5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:17.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.097+0000 7f069573bb80 1 --2- 172.21.15.78:0/151593639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 0x5571194c5940 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0668000c00 tx=0x7f0668017720 comp rx=0 tx=0).stop 2024-03-14T09:56:17.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.097+0000 7f069573bb80 1 -- 172.21.15.78:0/151593639 shutdown_connections 2024-03-14T09:56:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.097+0000 7f069573bb80 1 --2- 172.21.15.78:0/151593639 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557119523200 0x557119525620 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.097+0000 7f069573bb80 1 --2- 172.21.15.78:0/151593639 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5571194c06d0 0x5571194c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.097+0000 7f069573bb80 1 --2- 172.21.15.78:0/151593639 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5571194b5e90 0x5571194c73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.097+0000 7f069573bb80 1 --2- 172.21.15.78:0/151593639 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5571194b67b0 0x5571194c5940 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.097+0000 7f069573bb80 1 -- 172.21.15.78:0/151593639 >> 172.21.15.78:0/151593639 conn(0x557119423220 msgr2=0x5571194b87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.097+0000 7f0664c00700 0 client.16137 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.098+0000 7f069573bb80 1 -- 172.21.15.78:0/151593639 shutdown_connections 2024-03-14T09:56:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.098+0000 7f069573bb80 1 -- 172.21.15.78:0/151593639 wait complete. 2024-03-14T09:56:17.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.099+0000 7f069573bb80 20 client.16137 trim_cache size 0 max 16384 2024-03-14T09:56:17.100 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:56:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "scrub active (1150 inodes in the stack)", 2024-03-14T09:56:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T09:56:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "aa9055ad-10bd-49b6-b26f-f9b5affb7831": { 2024-03-14T09:56:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T09:56:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "aa9055ad-10bd-49b6-b26f-f9b5affb7831", 2024-03-14T09:56:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T09:56:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:56:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:56:17.100 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:56:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.099+0000 7f0681c00700 1 -- 172.21.15.78:0/367447345 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f06600a0c00 msgr2=0x7f06600a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.099+0000 7f0681c00700 1 --2- 172.21.15.78:0/367447345 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f06600a0c00 0x7f06600a30c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f067400f590 tx=0x7f067400b040 comp rx=0 tx=0).stop 2024-03-14T09:56:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.099+0000 7f0681c00700 1 -- 172.21.15.78:0/367447345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f067c0680b0 msgr2=0x7f067c19f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:17.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.099+0000 7f0681c00700 1 --2- 172.21.15.78:0/367447345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f067c0680b0 0x7f067c19f530 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f066c00e9a0 tx=0x7f066c00ecb0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.099+0000 7f0681c00700 1 -- 172.21.15.78:0/367447345 shutdown_connections 2024-03-14T09:56:17.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.099+0000 7f0681c00700 1 --2- 172.21.15.78:0/367447345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f067c199bf0 0x7f067c19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.100+0000 7f0681c00700 1 --2- 172.21.15.78:0/367447345 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f067c0680b0 0x7f067c19f530 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.100+0000 7f0681c00700 1 --2- 172.21.15.78:0/367447345 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f06600a0c00 0x7f06600a30c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.100+0000 7f0681c00700 1 --2- 172.21.15.78:0/367447345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f067c106060 0x7f067c19fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:17.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.100+0000 7f0681c00700 1 -- 172.21.15.78:0/367447345 >> 172.21.15.78:0/367447345 conn(0x7f067c076d80 msgr2=0x7f067c0fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:17.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.100+0000 7f0681c00700 1 -- 172.21.15.78:0/367447345 shutdown_connections 2024-03-14T09:56:17.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:17.100+0000 7f0681c00700 1 -- 172.21.15.78:0/367447345 wait complete. 2024-03-14T09:56:17.126 INFO:tasks.cephfs.filesystem:scrub status for tag:aa9055ad-10bd-49b6-b26f-f9b5affb7831 - {'path': '/', 'tag': 'aa9055ad-10bd-49b6-b26f-f9b5affb7831', 'options': 'recursive,force'} 2024-03-14T09:56:17.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:17 smithi078 ceph-mon[38143]: pgmap v66: 129 pgs: 129 active+clean; 82 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.1 MiB/s wr, 163 op/s 2024-03-14T09:56:17.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:17 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T09:56:17.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:17 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:56:17.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:17 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T09:56:17.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:17 smithi152 ceph-mon[48279]: pgmap v66: 129 pgs: 129 active+clean; 82 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.1 MiB/s wr, 163 op/s 2024-03-14T09:56:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:17 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T09:56:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:17 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:56:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:17 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T09:56:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:17 smithi132 ceph-mon[47959]: pgmap v66: 129 pgs: 129 active+clean; 82 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.1 MiB/s wr, 163 op/s 2024-03-14T09:56:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:17 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T09:56:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:17 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:56:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:17 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T09:56:18.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:18 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:56:18.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:18 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:56:18.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:18 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:56:18.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:18 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:56:18.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:18 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:56:18.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:18 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:56:18.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:18 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:56:18.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:18 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:56:18.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:18 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:56:18.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:18 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:56:18.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:18 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:56:18.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:18 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:56:18.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:18 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:56:18.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:18 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:56:18.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:18 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:56:19.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:19 smithi078 ceph-mon[38143]: pgmap v67: 129 pgs: 129 active+clean; 82 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 1.1 MiB/s wr, 159 op/s 2024-03-14T09:56:19.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:19 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T09:56:19.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:19 smithi152 ceph-mon[48279]: pgmap v67: 129 pgs: 129 active+clean; 82 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 1.1 MiB/s wr, 159 op/s 2024-03-14T09:56:19.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:19 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T09:56:19.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:19 smithi132 ceph-mon[47959]: pgmap v67: 129 pgs: 129 active+clean; 82 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 1.1 MiB/s wr, 159 op/s 2024-03-14T09:56:19.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:19 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T09:56:21.180 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0xfffc5 2024-03-14T09:56:21.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:21 smithi078 ceph-mon[38143]: pgmap v68: 129 pgs: 129 active+clean; 82 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 1.1 MiB/s wr, 159 op/s 2024-03-14T09:56:21.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:21 smithi152 ceph-mon[48279]: pgmap v68: 129 pgs: 129 active+clean; 82 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 1.1 MiB/s wr, 159 op/s 2024-03-14T09:56:21.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:21 smithi132 ceph-mon[47959]: pgmap v68: 129 pgs: 129 active+clean; 82 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 1.1 MiB/s wr, 159 op/s 2024-03-14T09:56:22.541 INFO:tasks.workunit.client.0.smithi078.stdout:10000 mapread 0x32cdc thru 0x3e88f (0xbbb4 bytes) 2024-03-14T09:56:23.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:23 smithi078 ceph-mon[38143]: pgmap v69: 129 pgs: 129 active+clean; 85 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 382 op/s 2024-03-14T09:56:23.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:23 smithi152 ceph-mon[48279]: pgmap v69: 129 pgs: 129 active+clean; 85 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 382 op/s 2024-03-14T09:56:23.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:23 smithi132 ceph-mon[47959]: pgmap v69: 129 pgs: 129 active+clean; 85 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 382 op/s 2024-03-14T09:56:25.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:25 smithi152 ceph-mon[48279]: pgmap v70: 129 pgs: 129 active+clean; 85 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 379 op/s 2024-03-14T09:56:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:25 smithi132 ceph-mon[47959]: pgmap v70: 129 pgs: 129 active+clean; 85 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 379 op/s 2024-03-14T09:56:25.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:25 smithi078 ceph-mon[38143]: pgmap v70: 129 pgs: 129 active+clean; 85 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 379 op/s 2024-03-14T09:56:27.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:27 smithi152 ceph-mon[48279]: pgmap v71: 129 pgs: 129 active+clean; 88 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 600 op/s 2024-03-14T09:56:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:27 smithi132 ceph-mon[47959]: pgmap v71: 129 pgs: 129 active+clean; 88 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 600 op/s 2024-03-14T09:56:27.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:27 smithi078 ceph-mon[38143]: pgmap v71: 129 pgs: 129 active+clean; 88 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.6 MiB/s wr, 600 op/s 2024-03-14T09:56:29.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:28 smithi078 ceph-mon[38143]: pgmap v72: 129 pgs: 129 active+clean; 88 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 443 op/s 2024-03-14T09:56:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:28 smithi152 ceph-mon[48279]: pgmap v72: 129 pgs: 129 active+clean; 88 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 443 op/s 2024-03-14T09:56:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:28 smithi132 ceph-mon[47959]: pgmap v72: 129 pgs: 129 active+clean; 88 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 443 op/s 2024-03-14T09:56:31.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:31 smithi078 ceph-mon[38143]: pgmap v73: 129 pgs: 129 active+clean; 88 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 443 op/s 2024-03-14T09:56:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:31 smithi152 ceph-mon[48279]: pgmap v73: 129 pgs: 129 active+clean; 88 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 443 op/s 2024-03-14T09:56:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:31 smithi132 ceph-mon[47959]: pgmap v73: 129 pgs: 129 active+clean; 88 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.6 MiB/s wr, 443 op/s 2024-03-14T09:56:33.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:33 smithi078 ceph-mon[38143]: pgmap v74: 129 pgs: 129 active+clean; 92 MiB data, 667 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.9 MiB/s wr, 646 op/s 2024-03-14T09:56:33.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:33 smithi152 ceph-mon[48279]: pgmap v74: 129 pgs: 129 active+clean; 92 MiB data, 667 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.9 MiB/s wr, 646 op/s 2024-03-14T09:56:33.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:33 smithi132 ceph-mon[47959]: pgmap v74: 129 pgs: 129 active+clean; 92 MiB data, 667 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.9 MiB/s wr, 646 op/s 2024-03-14T09:56:35.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:35 smithi078 ceph-mon[38143]: pgmap v75: 129 pgs: 129 active+clean; 92 MiB data, 667 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 424 op/s 2024-03-14T09:56:35.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:35 smithi152 ceph-mon[48279]: pgmap v75: 129 pgs: 129 active+clean; 92 MiB data, 667 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 424 op/s 2024-03-14T09:56:35.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:35 smithi132 ceph-mon[47959]: pgmap v75: 129 pgs: 129 active+clean; 92 MiB data, 667 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 424 op/s 2024-03-14T09:56:37.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:37 smithi078 ceph-mon[38143]: pgmap v76: 129 pgs: 129 active+clean; 95 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 604 op/s 2024-03-14T09:56:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:37 smithi152 ceph-mon[48279]: pgmap v76: 129 pgs: 129 active+clean; 95 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 604 op/s 2024-03-14T09:56:37.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:37 smithi132 ceph-mon[47959]: pgmap v76: 129 pgs: 129 active+clean; 95 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 604 op/s 2024-03-14T09:56:39.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:38 smithi078 ceph-mon[38143]: pgmap v77: 129 pgs: 129 active+clean; 95 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 383 op/s 2024-03-14T09:56:39.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:38 smithi152 ceph-mon[48279]: pgmap v77: 129 pgs: 129 active+clean; 95 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 383 op/s 2024-03-14T09:56:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:38 smithi132 ceph-mon[47959]: pgmap v77: 129 pgs: 129 active+clean; 95 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 383 op/s 2024-03-14T09:56:39.852 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0xfffe8 2024-03-14T09:56:41.306 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:41 smithi078 ceph-mon[38143]: pgmap v78: 129 pgs: 129 active+clean; 95 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 383 op/s 2024-03-14T09:56:41.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:41 smithi152 ceph-mon[48279]: pgmap v78: 129 pgs: 129 active+clean; 95 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 383 op/s 2024-03-14T09:56:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:41 smithi132 ceph-mon[47959]: pgmap v78: 129 pgs: 129 active+clean; 95 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 383 op/s 2024-03-14T09:56:43.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:43 smithi078 ceph-mon[38143]: pgmap v79: 129 pgs: 129 active+clean; 99 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 551 op/s 2024-03-14T09:56:43.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:43 smithi152 ceph-mon[48279]: pgmap v79: 129 pgs: 129 active+clean; 99 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 551 op/s 2024-03-14T09:56:43.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:43 smithi132 ceph-mon[47959]: pgmap v79: 129 pgs: 129 active+clean; 99 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 551 op/s 2024-03-14T09:56:45.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:45 smithi078 ceph-mon[38143]: pgmap v80: 129 pgs: 129 active+clean; 99 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 347 op/s 2024-03-14T09:56:45.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:45 smithi152 ceph-mon[48279]: pgmap v80: 129 pgs: 129 active+clean; 99 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 347 op/s 2024-03-14T09:56:45.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:45 smithi132 ceph-mon[47959]: pgmap v80: 129 pgs: 129 active+clean; 99 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 347 op/s 2024-03-14T09:56:47.127 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:56:47.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:47 smithi078 ceph-mon[38143]: pgmap v81: 129 pgs: 129 active+clean; 101 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 500 op/s 2024-03-14T09:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.363+0000 7fb32de00700 1 -- 172.21.15.78:0/3050883756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb328106050 msgr2=0x7fb32810dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.363+0000 7fb32de00700 1 --2- 172.21.15.78:0/3050883756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb328106050 0x7fb32810dbd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb32000b0a0 tx=0x7fb32000b3b0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.363+0000 7fb32de00700 1 -- 172.21.15.78:0/3050883756 shutdown_connections 2024-03-14T09:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.363+0000 7fb32de00700 1 --2- 172.21.15.78:0/3050883756 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb328106050 0x7fb32810dbd0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.363+0000 7fb32de00700 1 --2- 172.21.15.78:0/3050883756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb328100f30 0x7fb328101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.363+0000 7fb32de00700 1 --2- 172.21.15.78:0/3050883756 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb3281018e0 0x7fb328105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.363+0000 7fb32de00700 1 -- 172.21.15.78:0/3050883756 >> 172.21.15.78:0/3050883756 conn(0x7fb3280fc7b0 msgr2=0x7fb3280febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.363+0000 7fb32de00700 1 -- 172.21.15.78:0/3050883756 shutdown_connections 2024-03-14T09:56:47.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.363+0000 7fb32de00700 1 -- 172.21.15.78:0/3050883756 wait complete. 2024-03-14T09:56:47.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.365+0000 7fb32de00700 1 Processor -- start 2024-03-14T09:56:47.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.365+0000 7fb32de00700 1 -- start start 2024-03-14T09:56:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.365+0000 7fb32de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb328100f30 0x7fb3281952f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.365+0000 7fb32de00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb328106050 0x7fb328195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.365+0000 7fb32de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb328199b80 0x7fb32819a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.365+0000 7fb32de00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32819a600 con 0x7fb328199b80 2024-03-14T09:56:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.365+0000 7fb32de00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32819a770 con 0x7fb328100f30 2024-03-14T09:56:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.365+0000 7fb32de00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32819a930 con 0x7fb328106050 2024-03-14T09:56:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.366+0000 7fb326a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb328100f30 0x7fb3281952f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.366+0000 7fb326000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb328106050 0x7fb328195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.366+0000 7fb326a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb328100f30 0x7fb3281952f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48148/0 (socket says 172.21.15.78:48148) 2024-03-14T09:56:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.366+0000 7fb326000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb328106050 0x7fb328195830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:33786/0 (socket says 172.21.15.78:33786) 2024-03-14T09:56:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.366+0000 7fb326a00700 1 -- 172.21.15.78:0/258373621 learned_addr learned my addr 172.21.15.78:0/258373621 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.366+0000 7fb327400700 1 --2- 172.21.15.78:0/258373621 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb328199b80 0x7fb32819a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.366+0000 7fb326a00700 1 -- 172.21.15.78:0/258373621 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb328106050 msgr2=0x7fb328195830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:47.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.366+0000 7fb326a00700 1 --2- 172.21.15.78:0/258373621 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb328106050 0x7fb328195830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.366+0000 7fb326a00700 1 -- 172.21.15.78:0/258373621 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb328199b80 msgr2=0x7fb32819a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:47.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.366+0000 7fb326a00700 1 --2- 172.21.15.78:0/258373621 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb328199b80 0x7fb32819a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.366+0000 7fb326a00700 1 -- 172.21.15.78:0/258373621 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb320009d00 con 0x7fb328100f30 2024-03-14T09:56:47.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.366+0000 7fb326000700 1 --2- 172.21.15.78:0/258373621 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb328106050 0x7fb328195830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:56:47.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.366+0000 7fb326a00700 1 --2- 172.21.15.78:0/258373621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb328100f30 0x7fb3281952f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb31800e9a0 tx=0x7fb31800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.367+0000 7fb32ca00700 1 -- 172.21.15.78:0/258373621 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb318009790 con 0x7fb328100f30 2024-03-14T09:56:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.367+0000 7fb32ca00700 1 -- 172.21.15.78:0/258373621 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb318010430 con 0x7fb328100f30 2024-03-14T09:56:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.367+0000 7fb32de00700 1 -- 172.21.15.78:0/258373621 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3281a1e90 con 0x7fb328100f30 2024-03-14T09:56:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.367+0000 7fb32ca00700 1 -- 172.21.15.78:0/258373621 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb318018680 con 0x7fb328100f30 2024-03-14T09:56:47.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.367+0000 7fb32de00700 1 -- 172.21.15.78:0/258373621 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3281a2360 con 0x7fb328100f30 2024-03-14T09:56:47.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.371+0000 7fb32ca00700 1 -- 172.21.15.78:0/258373621 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fb3180105a0 con 0x7fb328100f30 2024-03-14T09:56:47.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.372+0000 7fb341838b80 1 Processor -- start 2024-03-14T09:56:47.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.372+0000 7fb341838b80 1 -- start start 2024-03-14T09:56:47.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.372+0000 7fb341838b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14b9e680 0x55eb14b9ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.372+0000 7fb341838b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14b9efa0 0x55eb14ba8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.372+0000 7fb341838b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14ba8ec0 0x55eb14bab2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.372+0000 7fb341838b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb14af7990 con 0x55eb14b9e680 2024-03-14T09:56:47.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.372+0000 7fb341838b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb14af5350 con 0x55eb14ba8ec0 2024-03-14T09:56:47.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.372+0000 7fb341838b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb14af0c40 con 0x55eb14b9efa0 2024-03-14T09:56:47.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.373+0000 7fb326000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14ba8ec0 0x55eb14bab2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.373+0000 7fb327400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14b9e680 0x55eb14b9ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.373+0000 7fb326000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14ba8ec0 0x55eb14bab2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48164/0 (socket says 172.21.15.78:48164) 2024-03-14T09:56:47.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.373+0000 7fb326000700 1 -- 172.21.15.78:0/3754080993 learned_addr learned my addr 172.21.15.78:0/3754080993 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:47.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.373+0000 7fb326a00700 1 --2- 172.21.15.78:0/3754080993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14b9efa0 0x55eb14ba8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.373+0000 7fb32ca00700 1 --2- 172.21.15.78:0/258373621 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb30c0a0d00 0x7fb30c0a31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.373+0000 7fb326000700 1 -- 172.21.15.78:0/3754080993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14b9efa0 msgr2=0x55eb14ba8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:47.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.373+0000 7fb326000700 1 --2- 172.21.15.78:0/3754080993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14b9efa0 0x55eb14ba8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.373+0000 7fb326000700 1 -- 172.21.15.78:0/3754080993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14b9e680 msgr2=0x55eb14b9ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:47.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.373+0000 7fb326000700 1 --2- 172.21.15.78:0/3754080993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14b9e680 0x55eb14b9ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.373+0000 7fb32ca00700 1 -- 172.21.15.78:0/258373621 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fb318014070 con 0x7fb328100f30 2024-03-14T09:56:47.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.373+0000 7fb326000700 1 -- 172.21.15.78:0/3754080993 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb149723b0 con 0x55eb14ba8ec0 2024-03-14T09:56:47.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.373+0000 7fb327400700 1 --2- 172.21.15.78:0/3754080993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14b9e680 0x55eb14b9ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:56:47.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.374+0000 7fb327400700 1 --2- 172.21.15.78:0/258373621 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb30c0a0d00 0x7fb30c0a31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.374+0000 7fb326000700 1 --2- 172.21.15.78:0/3754080993 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14ba8ec0 0x55eb14bab2b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb314013630 tx=0x7fb31400d670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7d5e7eab98566c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:47.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.374+0000 7fb313e00700 1 -- 172.21.15.78:0/3754080993 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb314012040 con 0x55eb14ba8ec0 2024-03-14T09:56:47.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.374+0000 7fb313e00700 1 -- 172.21.15.78:0/3754080993 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb314013040 con 0x55eb14ba8ec0 2024-03-14T09:56:47.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.374+0000 7fb313e00700 1 -- 172.21.15.78:0/3754080993 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3140186c0 con 0x55eb14ba8ec0 2024-03-14T09:56:47.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.375+0000 7fb341838b80 1 -- 172.21.15.78:0/3754080993 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14ba8ec0 msgr2=0x55eb14bab2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:47.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.375+0000 7fb327400700 1 --2- 172.21.15.78:0/258373621 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb30c0a0d00 0x7fb30c0a31c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb320009610 tx=0x7fb3200099e0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:47.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.375+0000 7fb341838b80 1 --2- 172.21.15.78:0/3754080993 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14ba8ec0 0x55eb14bab2b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb314013630 tx=0x7fb31400d670 comp rx=0 tx=0).stop 2024-03-14T09:56:47.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.375+0000 7fb341838b80 1 -- 172.21.15.78:0/3754080993 shutdown_connections 2024-03-14T09:56:47.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.375+0000 7fb341838b80 1 --2- 172.21.15.78:0/3754080993 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14ba8ec0 0x55eb14bab2b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.375+0000 7fb341838b80 1 --2- 172.21.15.78:0/3754080993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14b9e680 0x55eb14b9ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.375+0000 7fb341838b80 1 --2- 172.21.15.78:0/3754080993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14b9efa0 0x55eb14ba8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.375+0000 7fb341838b80 1 -- 172.21.15.78:0/3754080993 >> 172.21.15.78:0/3754080993 conn(0x55eb14b14220 msgr2=0x55eb14b99aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:47.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.375+0000 7fb341838b80 1 -- 172.21.15.78:0/3754080993 shutdown_connections 2024-03-14T09:56:47.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.375+0000 7fb341838b80 1 -- 172.21.15.78:0/3754080993 wait complete. 2024-03-14T09:56:47.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.376+0000 7fb341838b80 1 -- start start 2024-03-14T09:56:47.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.376+0000 7fb341838b80 1 Processor -- start 2024-03-14T09:56:47.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.377+0000 7fb341838b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14b9e680 0x55eb14ba8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.377+0000 7fb341838b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14b9efa0 0x55eb14ba5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.377+0000 7fb341838b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14ba8ec0 0x55eb14ba61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.377+0000 7fb341838b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb14a8cd80 con 0x55eb14b9e680 2024-03-14T09:56:47.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.377+0000 7fb341838b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb14a8ca20 con 0x55eb14ba8ec0 2024-03-14T09:56:47.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.377+0000 7fb341838b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb149746a0 con 0x55eb14b9efa0 2024-03-14T09:56:47.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.377+0000 7fb341838b80 10 client.0 _ll_register_callbacks cb 0x55eb148f83a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:47.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.377+0000 7fb327400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14ba8ec0 0x55eb14ba61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.377+0000 7fb327400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14ba8ec0 0x55eb14ba61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48178/0 (socket says 172.21.15.78:48178) 2024-03-14T09:56:47.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.377+0000 7fb327400700 1 -- 172.21.15.78:0/3663525631 learned_addr learned my addr 172.21.15.78:0/3663525631 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:47.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.378+0000 7fb326000700 1 --2- 172.21.15.78:0/3663525631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14b9efa0 0x55eb14ba5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.378+0000 7fb327400700 1 -- 172.21.15.78:0/3663525631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14b9efa0 msgr2=0x55eb14ba5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:47.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.378+0000 7fb327400700 1 --2- 172.21.15.78:0/3663525631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14b9efa0 0x55eb14ba5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.378+0000 7fb327400700 1 -- 172.21.15.78:0/3663525631 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14b9e680 msgr2=0x55eb14ba8880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-14T09:56:47.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.378+0000 7fb327400700 1 --2- 172.21.15.78:0/3663525631 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14b9e680 0x55eb14ba8880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.378+0000 7fb327400700 1 -- 172.21.15.78:0/3663525631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb32810e440 con 0x55eb14ba8ec0 2024-03-14T09:56:47.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.378+0000 7fb327400700 1 --2- 172.21.15.78:0/3663525631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14ba8ec0 0x55eb14ba61d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb320015000 tx=0x7fb320006230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=575f1dbbedb72c90 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:47.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.378+0000 7fb310c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:56:47.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.378+0000 7fb310c00700 1 -- 172.21.15.78:0/3663525631 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32000e040 con 0x55eb14ba8ec0 2024-03-14T09:56:47.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.378+0000 7fb341838b80 1 -- 172.21.15.78:0/3663525631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eb14b91720 con 0x55eb14ba8ec0 2024-03-14T09:56:47.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.378+0000 7fb310c00700 1 -- 172.21.15.78:0/3663525631 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fb320029440 con 0x55eb14ba8ec0 2024-03-14T09:56:47.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.379+0000 7fb310c00700 1 -- 172.21.15.78:0/3663525631 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb320029600 con 0x55eb14ba8ec0 2024-03-14T09:56:47.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.379+0000 7fb341838b80 10 client.26159 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:47.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.379+0000 7fb341838b80 1 -- 172.21.15.78:0/3663525631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7fb3140132e0 con 0x55eb14ba8ec0 2024-03-14T09:56:47.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.379+0000 7fb310c00700 1 -- 172.21.15.78:0/3663525631 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55eb14b91720 con 0x55eb14ba8ec0 2024-03-14T09:56:47.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.379+0000 7fb310c00700 1 -- 172.21.15.78:0/3663525631 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb320026050 con 0x55eb14ba8ec0 2024-03-14T09:56:47.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.379+0000 7fb341838b80 10 client.26159 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:47.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.379+0000 7fb341838b80 10 client.26159 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:56:47.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.379+0000 7fb341838b80 20 client.26159 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:47.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.379+0000 7fb341838b80 1 --2- 172.21.15.78:0/3663525631 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55eb14bb88a0 0x55eb14bbacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.379+0000 7fb341838b80 4 client.26159 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:56:47.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.379+0000 7fb341838b80 1 -- 172.21.15.78:0/3663525631 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eb14ae9700 con 0x55eb14bb88a0 2024-03-14T09:56:47.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.380+0000 7fb326a00700 1 --2- 172.21.15.78:0/3663525631 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55eb14bb88a0 0x55eb14bbacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.380+0000 7fb326a00700 1 --2- 172.21.15.78:0/3663525631 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55eb14bb88a0 0x55eb14bbacc0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2459bb736cdcb0d3 server_cookie=6713d33570d11ff1 in_seq=0 out_seq=0 2024-03-14T09:56:47.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.380+0000 7fb310c00700 10 client.26159 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:47.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.381+0000 7fb310c00700 1 -- 172.21.15.78:0/3663525631 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55eb14ae9700 con 0x55eb14bb88a0 2024-03-14T09:56:47.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.381+0000 7fb310c00700 10 client.26159 handle_command_reply: tid=0 2024-03-14T09:56:47.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.381+0000 7fb341838b80 1 client.26159 shutdown 2024-03-14T09:56:47.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.381+0000 7fb341838b80 1 -- 172.21.15.78:0/3663525631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14ba8ec0 msgr2=0x55eb14ba61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:47.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.381+0000 7fb341838b80 1 --2- 172.21.15.78:0/3663525631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14ba8ec0 0x55eb14ba61d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb320015000 tx=0x7fb320006230 comp rx=0 tx=0).stop 2024-03-14T09:56:47.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.381+0000 7fb341838b80 1 -- 172.21.15.78:0/3663525631 shutdown_connections 2024-03-14T09:56:47.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.381+0000 7fb341838b80 1 --2- 172.21.15.78:0/3663525631 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55eb14bb88a0 0x55eb14bbacc0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.381+0000 7fb341838b80 1 --2- 172.21.15.78:0/3663525631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14ba8ec0 0x55eb14ba61d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.381+0000 7fb310c00700 0 client.26159 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:47.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.381+0000 7fb341838b80 1 --2- 172.21.15.78:0/3663525631 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14b9e680 0x55eb14ba8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.381+0000 7fb341838b80 1 --2- 172.21.15.78:0/3663525631 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14b9efa0 0x55eb14ba5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.381+0000 7fb341838b80 1 -- 172.21.15.78:0/3663525631 >> 172.21.15.78:0/3663525631 conn(0x55eb14b14220 msgr2=0x55eb14b99690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:47.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.382+0000 7fb341838b80 1 -- 172.21.15.78:0/3663525631 shutdown_connections 2024-03-14T09:56:47.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.382+0000 7fb341838b80 1 -- 172.21.15.78:0/3663525631 wait complete. 2024-03-14T09:56:47.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.382+0000 7fb341838b80 20 client.26159 trim_cache size 0 max 16384 2024-03-14T09:56:47.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.401+0000 7fb341838b80 1 Processor -- start 2024-03-14T09:56:47.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.401+0000 7fb341838b80 1 -- start start 2024-03-14T09:56:47.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.402+0000 7fb341838b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14b9e680 0x55eb14bafb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.402+0000 7fb341838b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14b9efa0 0x55eb14ba83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.402+0000 7fb341838b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14ba8ec0 0x55eb14ba4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.402+0000 7fb341838b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb14b9b060 con 0x55eb14ba8ec0 2024-03-14T09:56:47.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.402+0000 7fb341838b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb14bbf9c0 con 0x55eb14b9efa0 2024-03-14T09:56:47.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.402+0000 7fb341838b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb14b9a770 con 0x55eb14b9e680 2024-03-14T09:56:47.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.402+0000 7fb326000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14b9efa0 0x55eb14ba83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.402+0000 7fb327400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14ba8ec0 0x55eb14ba4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.402+0000 7fb326a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14b9e680 0x55eb14bafb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.402+0000 7fb326000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14b9efa0 0x55eb14ba83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48180/0 (socket says 172.21.15.78:48180) 2024-03-14T09:56:47.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.402+0000 7fb326000700 1 -- 172.21.15.78:0/3108348539 learned_addr learned my addr 172.21.15.78:0/3108348539 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:47.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.403+0000 7fb326000700 1 -- 172.21.15.78:0/3108348539 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14b9e680 msgr2=0x55eb14bafb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:47.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.403+0000 7fb326000700 1 --2- 172.21.15.78:0/3108348539 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14b9e680 0x55eb14bafb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.403+0000 7fb326000700 1 -- 172.21.15.78:0/3108348539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14ba8ec0 msgr2=0x55eb14ba4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:47.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.403+0000 7fb326000700 1 --2- 172.21.15.78:0/3108348539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14ba8ec0 0x55eb14ba4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.403+0000 7fb326000700 1 -- 172.21.15.78:0/3108348539 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb1491eb60 con 0x55eb14b9efa0 2024-03-14T09:56:47.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.403+0000 7fb326a00700 1 --2- 172.21.15.78:0/3108348539 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14b9e680 0x55eb14bafb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:56:47.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.403+0000 7fb326000700 1 --2- 172.21.15.78:0/3108348539 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14b9efa0 0x55eb14ba83b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb3140135d0 tx=0x7fb314005d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37a68de250c0a75b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:47.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.403+0000 7fb31d200700 1 -- 172.21.15.78:0/3108348539 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb314012040 con 0x55eb14b9efa0 2024-03-14T09:56:47.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.403+0000 7fb31d200700 1 -- 172.21.15.78:0/3108348539 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb314003680 con 0x55eb14b9efa0 2024-03-14T09:56:47.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.403+0000 7fb31d200700 1 -- 172.21.15.78:0/3108348539 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3140186c0 con 0x55eb14b9efa0 2024-03-14T09:56:47.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.404+0000 7fb341838b80 1 -- 172.21.15.78:0/3108348539 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14b9efa0 msgr2=0x55eb14ba83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:47.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.404+0000 7fb341838b80 1 --2- 172.21.15.78:0/3108348539 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14b9efa0 0x55eb14ba83b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb3140135d0 tx=0x7fb314005d40 comp rx=0 tx=0).stop 2024-03-14T09:56:47.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.404+0000 7fb341838b80 1 -- 172.21.15.78:0/3108348539 shutdown_connections 2024-03-14T09:56:47.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.404+0000 7fb341838b80 1 --2- 172.21.15.78:0/3108348539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14ba8ec0 0x55eb14ba4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.404+0000 7fb341838b80 1 --2- 172.21.15.78:0/3108348539 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14b9e680 0x55eb14bafb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.404+0000 7fb341838b80 1 --2- 172.21.15.78:0/3108348539 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14b9efa0 0x55eb14ba83b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.404+0000 7fb341838b80 1 -- 172.21.15.78:0/3108348539 >> 172.21.15.78:0/3108348539 conn(0x55eb14b14220 msgr2=0x55eb14ba0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:47.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.404+0000 7fb341838b80 1 -- 172.21.15.78:0/3108348539 shutdown_connections 2024-03-14T09:56:47.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.404+0000 7fb341838b80 1 -- 172.21.15.78:0/3108348539 wait complete. 2024-03-14T09:56:47.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.405+0000 7fb341838b80 1 -- start start 2024-03-14T09:56:47.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.405+0000 7fb341838b80 1 Processor -- start 2024-03-14T09:56:47.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.405+0000 7fb341838b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14b9e680 0x55eb14baf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.405+0000 7fb341838b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14b9efa0 0x55eb14badca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.405+0000 7fb341838b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14ba8ec0 0x55eb14bae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.405+0000 7fb341838b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb14bae720 con 0x55eb14b9e680 2024-03-14T09:56:47.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.405+0000 7fb341838b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb14bae890 con 0x55eb14b9efa0 2024-03-14T09:56:47.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.405+0000 7fb341838b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb14baea50 con 0x55eb14ba8ec0 2024-03-14T09:56:47.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.406+0000 7fb341838b80 10 client.0 _ll_register_callbacks cb 0x7fb32000cdd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:47.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.406+0000 7fb326000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14b9efa0 0x55eb14badca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.406+0000 7fb326a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14b9e680 0x55eb14baf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.406+0000 7fb326000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14b9efa0 0x55eb14badca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48190/0 (socket says 172.21.15.78:48190) 2024-03-14T09:56:47.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.406+0000 7fb326000700 1 -- 172.21.15.78:0/2245567350 learned_addr learned my addr 172.21.15.78:0/2245567350 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:47.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.406+0000 7fb327400700 1 --2- 172.21.15.78:0/2245567350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14ba8ec0 0x55eb14bae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.406+0000 7fb326000700 1 -- 172.21.15.78:0/2245567350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14ba8ec0 msgr2=0x55eb14bae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:47.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.406+0000 7fb326000700 1 --2- 172.21.15.78:0/2245567350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14ba8ec0 0x55eb14bae1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.406+0000 7fb326000700 1 -- 172.21.15.78:0/2245567350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14b9e680 msgr2=0x55eb14baf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:47.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.406+0000 7fb326000700 1 --2- 172.21.15.78:0/2245567350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14b9e680 0x55eb14baf7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.406+0000 7fb326000700 1 -- 172.21.15.78:0/2245567350 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3281a2360 con 0x55eb14b9efa0 2024-03-14T09:56:47.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.406+0000 7fb326000700 1 --2- 172.21.15.78:0/2245567350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14b9efa0 0x55eb14badca0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb314014010 tx=0x7fb314003b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b19262963cc64657 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:47.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.406+0000 7fb310c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:56:47.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.407+0000 7fb310c00700 1 -- 172.21.15.78:0/2245567350 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb314018870 con 0x55eb14b9efa0 2024-03-14T09:56:47.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.407+0000 7fb341838b80 1 -- 172.21.15.78:0/2245567350 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb314013040 con 0x55eb14b9efa0 2024-03-14T09:56:47.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.407+0000 7fb310c00700 1 -- 172.21.15.78:0/2245567350 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fb3140189d0 con 0x55eb14b9efa0 2024-03-14T09:56:47.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.407+0000 7fb310c00700 1 -- 172.21.15.78:0/2245567350 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb31400d5f0 con 0x55eb14b9efa0 2024-03-14T09:56:47.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.407+0000 7fb341838b80 10 client.26165 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:47.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.407+0000 7fb341838b80 1 -- 172.21.15.78:0/2245567350 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55eb149723b0 con 0x55eb14b9efa0 2024-03-14T09:56:47.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.407+0000 7fb310c00700 1 -- 172.21.15.78:0/2245567350 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb314013040 con 0x55eb14b9efa0 2024-03-14T09:56:47.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.408+0000 7fb310c00700 1 -- 172.21.15.78:0/2245567350 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb314024b70 con 0x55eb14b9efa0 2024-03-14T09:56:47.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.408+0000 7fb341838b80 10 client.26165 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:47.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.408+0000 7fb341838b80 10 client.26165 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:56:47.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.408+0000 7fb341838b80 20 client.26165 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:47.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.408+0000 7fb341838b80 1 --2- 172.21.15.78:0/2245567350 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55eb14c0b540 0x55eb14c0d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.408+0000 7fb341838b80 4 client.26165 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:56:47.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.408+0000 7fb341838b80 1 -- 172.21.15.78:0/2245567350 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55eb1493fcf0 con 0x55eb14c0b540 2024-03-14T09:56:47.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.408+0000 7fb326a00700 1 --2- 172.21.15.78:0/2245567350 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55eb14c0b540 0x55eb14c0d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.409+0000 7fb326a00700 1 --2- 172.21.15.78:0/2245567350 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55eb14c0b540 0x55eb14c0d960 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af99691d43253c57 server_cookie=bca27266278253d0 in_seq=0 out_seq=0 2024-03-14T09:56:47.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.409+0000 7fb310c00700 10 client.26165 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:47.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.409+0000 7fb310c00700 1 -- 172.21.15.78:0/2245567350 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55eb1493fcf0 con 0x55eb14c0b540 2024-03-14T09:56:47.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.409+0000 7fb310c00700 10 client.26165 handle_command_reply: tid=0 2024-03-14T09:56:47.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.409+0000 7fb341838b80 1 client.26165 shutdown 2024-03-14T09:56:47.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.410+0000 7fb341838b80 1 -- 172.21.15.78:0/2245567350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14b9efa0 msgr2=0x55eb14badca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:47.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.410+0000 7fb341838b80 1 --2- 172.21.15.78:0/2245567350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14b9efa0 0x55eb14badca0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb314014010 tx=0x7fb314003b50 comp rx=0 tx=0).stop 2024-03-14T09:56:47.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.410+0000 7fb341838b80 1 -- 172.21.15.78:0/2245567350 shutdown_connections 2024-03-14T09:56:47.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.410+0000 7fb341838b80 1 --2- 172.21.15.78:0/2245567350 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55eb14ba8ec0 0x55eb14bae1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.410+0000 7fb341838b80 1 --2- 172.21.15.78:0/2245567350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eb14b9e680 0x55eb14baf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.410+0000 7fb341838b80 1 --2- 172.21.15.78:0/2245567350 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55eb14c0b540 0x55eb14c0d960 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.410+0000 7fb341838b80 1 --2- 172.21.15.78:0/2245567350 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55eb14b9efa0 0x55eb14badca0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.410+0000 7fb310c00700 0 client.26165 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:47.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.410+0000 7fb341838b80 1 -- 172.21.15.78:0/2245567350 >> 172.21.15.78:0/2245567350 conn(0x55eb14b14220 msgr2=0x55eb14ba4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:47.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.410+0000 7fb341838b80 1 -- 172.21.15.78:0/2245567350 shutdown_connections 2024-03-14T09:56:47.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.410+0000 7fb341838b80 1 -- 172.21.15.78:0/2245567350 wait complete. 2024-03-14T09:56:47.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.410+0000 7fb341838b80 20 client.26165 trim_cache size 0 max 16384 2024-03-14T09:56:47.413 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:56:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T09:56:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T09:56:47.414 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:56:47.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.411+0000 7fb32de00700 1 -- 172.21.15.78:0/258373621 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb30c0a0d00 msgr2=0x7fb30c0a31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:47.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.411+0000 7fb32de00700 1 --2- 172.21.15.78:0/258373621 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb30c0a0d00 0x7fb30c0a31c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb320009610 tx=0x7fb3200099e0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.411+0000 7fb32de00700 1 -- 172.21.15.78:0/258373621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb328100f30 msgr2=0x7fb3281952f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:47.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.411+0000 7fb32de00700 1 --2- 172.21.15.78:0/258373621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb328100f30 0x7fb3281952f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb31800e9a0 tx=0x7fb31800ecb0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.411+0000 7fb32de00700 1 -- 172.21.15.78:0/258373621 shutdown_connections 2024-03-14T09:56:47.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.411+0000 7fb32de00700 1 --2- 172.21.15.78:0/258373621 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb30c0a0d00 0x7fb30c0a31c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.411+0000 7fb32de00700 1 --2- 172.21.15.78:0/258373621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb328100f30 0x7fb3281952f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.411+0000 7fb32de00700 1 --2- 172.21.15.78:0/258373621 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb328199b80 0x7fb32819a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.411+0000 7fb32de00700 1 --2- 172.21.15.78:0/258373621 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb328106050 0x7fb328195830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.411+0000 7fb32de00700 1 -- 172.21.15.78:0/258373621 >> 172.21.15.78:0/258373621 conn(0x7fb3280fc7b0 msgr2=0x7fb3280feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:47.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.411+0000 7fb32de00700 1 -- 172.21.15.78:0/258373621 shutdown_connections 2024-03-14T09:56:47.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.411+0000 7fb32de00700 1 -- 172.21.15.78:0/258373621 wait complete. 2024-03-14T09:56:47.431 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T09:56:47.432 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:56:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:47 smithi152 ceph-mon[48279]: pgmap v81: 129 pgs: 129 active+clean; 101 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 500 op/s 2024-03-14T09:56:47.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:47 smithi132 ceph-mon[47959]: pgmap v81: 129 pgs: 129 active+clean; 101 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 500 op/s 2024-03-14T09:56:47.619 INFO:tasks.workunit.client.0.smithi078.stdout:30000 trunc from 0xf3838 to 0x7d16c 2024-03-14T09:56:47.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.668+0000 7fd017600700 1 -- 172.21.15.78:0/1302445475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0101017d0 msgr2=0x7fd010105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:47.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.668+0000 7fd017600700 1 --2- 172.21.15.78:0/1302445475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0101017d0 0x7fd010105810 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009990 tx=0x7fcffc009ca0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.668+0000 7fd017600700 1 -- 172.21.15.78:0/1302445475 shutdown_connections 2024-03-14T09:56:47.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.668+0000 7fd017600700 1 --2- 172.21.15.78:0/1302445475 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010100e20 0x7fd010101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.668+0000 7fd017600700 1 --2- 172.21.15.78:0/1302445475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd010105f40 0x7fd01010dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.668+0000 7fd017600700 1 --2- 172.21.15.78:0/1302445475 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd0101017d0 0x7fd010105810 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.668+0000 7fd017600700 1 -- 172.21.15.78:0/1302445475 >> 172.21.15.78:0/1302445475 conn(0x7fd0100fc700 msgr2=0x7fd0100feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:47.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.668+0000 7fd017600700 1 -- 172.21.15.78:0/1302445475 shutdown_connections 2024-03-14T09:56:47.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.668+0000 7fd017600700 1 -- 172.21.15.78:0/1302445475 wait complete. 2024-03-14T09:56:47.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.669+0000 7fd017600700 1 Processor -- start 2024-03-14T09:56:47.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.669+0000 7fd017600700 1 -- start start 2024-03-14T09:56:47.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.669+0000 7fd017600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd010100e20 0x7fd0100730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.669+0000 7fd017600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd010105f40 0x7fd0100735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.669+0000 7fd017600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010073b30 0x7fd0101a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.669+0000 7fd017600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0101a5d60 con 0x7fd010105f40 2024-03-14T09:56:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.669+0000 7fd017600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0101a5ea0 con 0x7fd010073b30 2024-03-14T09:56:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.669+0000 7fd017600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0101a6060 con 0x7fd010100e20 2024-03-14T09:56:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.670+0000 7fd015600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010073b30 0x7fd0101a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.670+0000 7fd00fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd010105f40 0x7fd0100735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.670+0000 7fd014c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd010100e20 0x7fd0100730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.670+0000 7fd015600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010073b30 0x7fd0101a5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48194/0 (socket says 172.21.15.78:48194) 2024-03-14T09:56:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.670+0000 7fd015600700 1 -- 172.21.15.78:0/3198893454 learned_addr learned my addr 172.21.15.78:0/3198893454 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.670+0000 7fd014c00700 1 -- 172.21.15.78:0/3198893454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010073b30 msgr2=0x7fd0101a5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.670+0000 7fd014c00700 1 --2- 172.21.15.78:0/3198893454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010073b30 0x7fd0101a5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.670+0000 7fd014c00700 1 -- 172.21.15.78:0/3198893454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd010105f40 msgr2=0x7fd0100735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.670+0000 7fd014c00700 1 --2- 172.21.15.78:0/3198893454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd010105f40 0x7fd0100735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.670+0000 7fd014c00700 1 -- 172.21.15.78:0/3198893454 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd008009c40 con 0x7fd010100e20 2024-03-14T09:56:47.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.670+0000 7fd00fe00700 1 --2- 172.21.15.78:0/3198893454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd010105f40 0x7fd0100735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:56:47.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.670+0000 7fd014c00700 1 --2- 172.21.15.78:0/3198893454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd010100e20 0x7fd0100730b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd00400e9f0 tx=0x7fd00400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:47.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.671+0000 7fd00d600700 1 -- 172.21.15.78:0/3198893454 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00400c9c0 con 0x7fd010100e20 2024-03-14T09:56:47.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.671+0000 7fd00d600700 1 -- 172.21.15.78:0/3198893454 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd004010430 con 0x7fd010100e20 2024-03-14T09:56:47.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.671+0000 7fd017600700 1 -- 172.21.15.78:0/3198893454 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcffc0095f0 con 0x7fd010100e20 2024-03-14T09:56:47.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.671+0000 7fd00d600700 1 -- 172.21.15.78:0/3198893454 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd004018680 con 0x7fd010100e20 2024-03-14T09:56:47.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.671+0000 7fd017600700 1 -- 172.21.15.78:0/3198893454 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0101a6710 con 0x7fd010100e20 2024-03-14T09:56:47.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.672+0000 7fd017600700 1 -- 172.21.15.78:0/3198893454 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd010109210 con 0x7fd010100e20 2024-03-14T09:56:47.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.674+0000 7fd00d600700 1 -- 172.21.15.78:0/3198893454 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fd004010860 con 0x7fd010100e20 2024-03-14T09:56:47.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.675+0000 7fd00d600700 1 --2- 172.21.15.78:0/3198893454 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcff80a0c80 0x7fcff80a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:47.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.675+0000 7fd00d600700 1 -- 172.21.15.78:0/3198893454 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fd004014070 con 0x7fd010100e20 2024-03-14T09:56:47.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.675+0000 7fd00fe00700 1 --2- 172.21.15.78:0/3198893454 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcff80a0c80 0x7fcff80a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:47.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.676+0000 7fd00fe00700 1 --2- 172.21.15.78:0/3198893454 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcff80a0c80 0x7fcff80a3140 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd010074790 tx=0x7fcffc005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:47.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.679+0000 7fd00d600700 1 -- 172.21.15.78:0/3198893454 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd004052960 con 0x7fd010100e20 2024-03-14T09:56:47.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.992+0000 7fd017600700 1 -- 172.21.15.78:0/3198893454 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0101022f0 con 0x7fd010100e20 2024-03-14T09:56:47.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.993+0000 7fd00d600700 1 -- 172.21.15.78:0/3198893454 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7fd004095ba0 con 0x7fd010100e20 2024-03-14T09:56:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:56:47.994 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:56:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.996+0000 7fd017600700 1 -- 172.21.15.78:0/3198893454 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcff80a0c80 msgr2=0x7fcff80a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.996+0000 7fd017600700 1 --2- 172.21.15.78:0/3198893454 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcff80a0c80 0x7fcff80a3140 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd010074790 tx=0x7fcffc005fd0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.996+0000 7fd017600700 1 -- 172.21.15.78:0/3198893454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd010100e20 msgr2=0x7fd0100730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.996+0000 7fd017600700 1 --2- 172.21.15.78:0/3198893454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd010100e20 0x7fd0100730b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd00400e9f0 tx=0x7fd00400ed00 comp rx=0 tx=0).stop 2024-03-14T09:56:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.996+0000 7fd017600700 1 -- 172.21.15.78:0/3198893454 shutdown_connections 2024-03-14T09:56:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.996+0000 7fd017600700 1 --2- 172.21.15.78:0/3198893454 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcff80a0c80 0x7fcff80a3140 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.996+0000 7fd017600700 1 --2- 172.21.15.78:0/3198893454 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010073b30 0x7fd0101a5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.996+0000 7fd017600700 1 --2- 172.21.15.78:0/3198893454 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd010100e20 0x7fd0100730b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.996+0000 7fd017600700 1 --2- 172.21.15.78:0/3198893454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd010105f40 0x7fd0100735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.996+0000 7fd017600700 1 -- 172.21.15.78:0/3198893454 >> 172.21.15.78:0/3198893454 conn(0x7fd0100fc700 msgr2=0x7fd0100fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.996+0000 7fd017600700 1 -- 172.21.15.78:0/3198893454 shutdown_connections 2024-03-14T09:56:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:47.996+0000 7fd017600700 1 -- 172.21.15.78:0/3198893454 wait complete. 2024-03-14T09:56:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T09:56:48.018 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T09:56:48.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.247+0000 7f8e72a00700 1 -- 172.21.15.78:0/770258546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8e6c0ff610 msgr2=0x7f8e6c0ffa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.247+0000 7f8e72a00700 1 --2- 172.21.15.78:0/770258546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8e6c0ff610 0x7f8e6c0ffa90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8e580099c0 tx=0x7f8e58009cd0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.247+0000 7f8e72a00700 1 -- 172.21.15.78:0/770258546 shutdown_connections 2024-03-14T09:56:48.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.247+0000 7f8e72a00700 1 --2- 172.21.15.78:0/770258546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0fffd0 0x7f8e6c10fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.247+0000 7f8e72a00700 1 --2- 172.21.15.78:0/770258546 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e6c101b60 0x7f8e6c101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.247+0000 7f8e72a00700 1 --2- 172.21.15.78:0/770258546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8e6c0ff610 0x7f8e6c0ffa90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.247+0000 7f8e72a00700 1 -- 172.21.15.78:0/770258546 >> 172.21.15.78:0/770258546 conn(0x7f8e6c074f70 msgr2=0x7f8e6c075380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.247+0000 7f8e72a00700 1 -- 172.21.15.78:0/770258546 shutdown_connections 2024-03-14T09:56:48.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.247+0000 7f8e72a00700 1 -- 172.21.15.78:0/770258546 wait complete. 2024-03-14T09:56:48.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.248+0000 7f8e72a00700 1 Processor -- start 2024-03-14T09:56:48.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.248+0000 7f8e72a00700 1 -- start start 2024-03-14T09:56:48.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.249+0000 7f8e72a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0ff610 0x7f8e6c1a3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.249+0000 7f8e72a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e6c0fffd0 0x7f8e6c1a3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.249+0000 7f8e72a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8e6c101b60 0x7f8e6c19d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.249+0000 7f8e72a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c19e000 con 0x7f8e6c0ff610 2024-03-14T09:56:48.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.249+0000 7f8e72a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c19e170 con 0x7f8e6c101b60 2024-03-14T09:56:48.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.249+0000 7f8e72a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6c19e330 con 0x7f8e6c0fffd0 2024-03-14T09:56:48.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.249+0000 7f8e6b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e6c0fffd0 0x7f8e6c1a3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.249+0000 7f8e6be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0ff610 0x7f8e6c1a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.249+0000 7f8e6b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e6c0fffd0 0x7f8e6c1a3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:33862/0 (socket says 172.21.15.78:33862) 2024-03-14T09:56:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.249+0000 7f8e6b400700 1 -- 172.21.15.78:0/3111555296 learned_addr learned my addr 172.21.15.78:0/3111555296 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.249+0000 7f8e70a00700 1 --2- 172.21.15.78:0/3111555296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8e6c101b60 0x7f8e6c19d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.250+0000 7f8e6b400700 1 -- 172.21.15.78:0/3111555296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8e6c101b60 msgr2=0x7f8e6c19d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.250+0000 7f8e6b400700 1 --2- 172.21.15.78:0/3111555296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8e6c101b60 0x7f8e6c19d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.250+0000 7f8e6b400700 1 -- 172.21.15.78:0/3111555296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0ff610 msgr2=0x7f8e6c1a3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.250+0000 7f8e6b400700 1 --2- 172.21.15.78:0/3111555296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0ff610 0x7f8e6c1a3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.250+0000 7f8e6b400700 1 -- 172.21.15.78:0/3111555296 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e580095f0 con 0x7f8e6c0fffd0 2024-03-14T09:56:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.250+0000 7f8e6b400700 1 --2- 172.21.15.78:0/3111555296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e6c0fffd0 0x7f8e6c1a3d60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800b580 tx=0x7f8e5800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.250+0000 7f8e68c00700 1 -- 172.21.15.78:0/3111555296 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5801c070 con 0x7f8e6c0fffd0 2024-03-14T09:56:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.250+0000 7f8e68c00700 1 -- 172.21.15.78:0/3111555296 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e5800fe30 con 0x7f8e6c0fffd0 2024-03-14T09:56:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.250+0000 7f8e72a00700 1 -- 172.21.15.78:0/3111555296 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e6c19e5b0 con 0x7f8e6c0fffd0 2024-03-14T09:56:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.250+0000 7f8e68c00700 1 -- 172.21.15.78:0/3111555296 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58022d40 con 0x7f8e6c0fffd0 2024-03-14T09:56:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.250+0000 7f8e72a00700 1 -- 172.21.15.78:0/3111555296 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6c1aa810 con 0x7f8e6c0fffd0 2024-03-14T09:56:48.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.255+0000 7f8e68c00700 1 -- 172.21.15.78:0/3111555296 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f8e58020070 con 0x7f8e6c0fffd0 2024-03-14T09:56:48.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.256+0000 7f8e864deb80 1 Processor -- start 2024-03-14T09:56:48.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.256+0000 7f8e68c00700 1 --2- 172.21.15.78:0/3111555296 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8e500a0c80 0x7f8e500a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.256+0000 7f8e864deb80 1 -- start start 2024-03-14T09:56:48.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.256+0000 7f8e68c00700 1 -- 172.21.15.78:0/3111555296 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f8e580cd440 con 0x7f8e6c0fffd0 2024-03-14T09:56:48.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e6be00700 1 --2- 172.21.15.78:0/3111555296 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8e500a0c80 0x7f8e500a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e864deb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 0x55ebc181ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e864deb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181cfa0 0x55ebc1826980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e864deb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc1826ec0 0x55ebc18292b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e864deb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc1775990 con 0x55ebc181cfa0 2024-03-14T09:56:48.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e864deb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc1773350 con 0x55ebc181c680 2024-03-14T09:56:48.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e864deb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc176ec40 con 0x55ebc1826ec0 2024-03-14T09:56:48.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e6be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 0x55ebc181ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e6be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 0x55ebc181ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48224/0 (socket says 172.21.15.78:48224) 2024-03-14T09:56:48.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e6be00700 1 -- 172.21.15.78:0/4255527136 learned_addr learned my addr 172.21.15.78:0/4255527136 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:48.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e6b400700 1 --2- 172.21.15.78:0/4255527136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181cfa0 0x55ebc1826980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e70a00700 1 --2- 172.21.15.78:0/4255527136 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc1826ec0 0x55ebc18292b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e6b400700 1 -- 172.21.15.78:0/4255527136 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc1826ec0 msgr2=0x55ebc18292b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e6b400700 1 --2- 172.21.15.78:0/4255527136 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc1826ec0 0x55ebc18292b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e6b400700 1 -- 172.21.15.78:0/4255527136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 msgr2=0x55ebc181ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e6b400700 1 --2- 172.21.15.78:0/4255527136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 0x55ebc181ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e6b400700 1 -- 172.21.15.78:0/4255527136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebc15f03b0 con 0x55ebc181cfa0 2024-03-14T09:56:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.257+0000 7f8e6be00700 1 --2- 172.21.15.78:0/4255527136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 0x55ebc181ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:56:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.258+0000 7f8e6be00700 1 --2- 172.21.15.78:0/3111555296 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8e500a0c80 0x7f8e500a3140 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c009fd0 tx=0x7f8e5c009c60 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.258+0000 7f8e6b400700 1 --2- 172.21.15.78:0/4255527136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181cfa0 0x55ebc1826980 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8e580ca820 tx=0x7f8e5809a3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b6985482bd98a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.258+0000 7f8e61200700 1 -- 172.21.15.78:0/4255527136 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58022ea0 con 0x55ebc181cfa0 2024-03-14T09:56:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.258+0000 7f8e61200700 1 -- 172.21.15.78:0/4255527136 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e5809e2e0 con 0x55ebc181cfa0 2024-03-14T09:56:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.258+0000 7f8e61200700 1 -- 172.21.15.78:0/4255527136 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5809a710 con 0x55ebc181cfa0 2024-03-14T09:56:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.259+0000 7f8e864deb80 1 -- 172.21.15.78:0/4255527136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181cfa0 msgr2=0x55ebc1826980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.259+0000 7f8e864deb80 1 --2- 172.21.15.78:0/4255527136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181cfa0 0x55ebc1826980 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8e580ca820 tx=0x7f8e5809a3b0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.259+0000 7f8e864deb80 1 -- 172.21.15.78:0/4255527136 shutdown_connections 2024-03-14T09:56:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.259+0000 7f8e864deb80 1 --2- 172.21.15.78:0/4255527136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 0x55ebc181ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.259+0000 7f8e864deb80 1 --2- 172.21.15.78:0/4255527136 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc1826ec0 0x55ebc18292b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.259+0000 7f8e864deb80 1 --2- 172.21.15.78:0/4255527136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181cfa0 0x55ebc1826980 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.259+0000 7f8e864deb80 1 -- 172.21.15.78:0/4255527136 >> 172.21.15.78:0/4255527136 conn(0x55ebc1792220 msgr2=0x55ebc1817aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.259+0000 7f8e864deb80 1 -- 172.21.15.78:0/4255527136 shutdown_connections 2024-03-14T09:56:48.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.259+0000 7f8e864deb80 1 -- 172.21.15.78:0/4255527136 wait complete. 2024-03-14T09:56:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.260+0000 7f8e864deb80 1 -- start start 2024-03-14T09:56:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.260+0000 7f8e864deb80 1 Processor -- start 2024-03-14T09:56:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.260+0000 7f8e864deb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181c680 0x55ebc1826920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.260+0000 7f8e864deb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc181cfa0 0x55ebc1823c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.260+0000 7f8e864deb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc1826ec0 0x55ebc18243a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.260+0000 7f8e864deb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc170ad80 con 0x55ebc181c680 2024-03-14T09:56:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.260+0000 7f8e864deb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc170aa20 con 0x55ebc1826ec0 2024-03-14T09:56:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.260+0000 7f8e864deb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc15f26a0 con 0x55ebc181cfa0 2024-03-14T09:56:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.260+0000 7f8e864deb80 10 client.0 _ll_register_callbacks cb 0x55ebc15763a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.261+0000 7f8e6be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181c680 0x55ebc1826920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.261+0000 7f8e6be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181c680 0x55ebc1826920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60152/0 (socket says 172.21.15.78:60152) 2024-03-14T09:56:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.261+0000 7f8e6be00700 1 -- 172.21.15.78:0/3044594369 learned_addr learned my addr 172.21.15.78:0/3044594369 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.261+0000 7f8e70a00700 1 --2- 172.21.15.78:0/3044594369 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc1826ec0 0x55ebc18243a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.261+0000 7f8e6b400700 1 --2- 172.21.15.78:0/3044594369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc181cfa0 0x55ebc1823c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.261+0000 7f8e70a00700 1 -- 172.21.15.78:0/3044594369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc181cfa0 msgr2=0x55ebc1823c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.261+0000 7f8e70a00700 1 --2- 172.21.15.78:0/3044594369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc181cfa0 0x55ebc1823c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.261+0000 7f8e70a00700 1 -- 172.21.15.78:0/3044594369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181c680 msgr2=0x55ebc1826920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.261+0000 7f8e70a00700 1 --2- 172.21.15.78:0/3044594369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181c680 0x55ebc1826920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.261+0000 7f8e70a00700 1 -- 172.21.15.78:0/3044594369 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e6c1aa810 con 0x55ebc1826ec0 2024-03-14T09:56:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.262+0000 7f8e70a00700 1 --2- 172.21.15.78:0/3044594369 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc1826ec0 0x55ebc18243a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8e640107a0 tx=0x7f8e64010ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1209fd1a539ce0f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.262+0000 7f8e55600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:56:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.262+0000 7f8e55600700 1 -- 172.21.15.78:0/3044594369 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e64015870 con 0x55ebc1826ec0 2024-03-14T09:56:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.262+0000 7f8e55600700 1 -- 172.21.15.78:0/3044594369 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f8e640159d0 con 0x55ebc1826ec0 2024-03-14T09:56:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.262+0000 7f8e864deb80 1 -- 172.21.15.78:0/3044594369 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ebc180f720 con 0x55ebc1826ec0 2024-03-14T09:56:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.262+0000 7f8e55600700 1 -- 172.21.15.78:0/3044594369 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e64015b90 con 0x55ebc1826ec0 2024-03-14T09:56:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.262+0000 7f8e864deb80 10 client.26183 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.262+0000 7f8e864deb80 1 -- 172.21.15.78:0/3044594369 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ebc159cb60 con 0x55ebc1826ec0 2024-03-14T09:56:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.262+0000 7f8e55600700 1 -- 172.21.15.78:0/3044594369 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ebc180f720 con 0x55ebc1826ec0 2024-03-14T09:56:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.263+0000 7f8e55600700 1 -- 172.21.15.78:0/3044594369 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8e640143c0 con 0x55ebc1826ec0 2024-03-14T09:56:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.263+0000 7f8e864deb80 10 client.26183 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.263+0000 7f8e864deb80 10 client.26183 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:56:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.263+0000 7f8e864deb80 20 client.26183 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.263+0000 7f8e864deb80 1 --2- 172.21.15.78:0/3044594369 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55ebc18368a0 0x55ebc1838cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.263+0000 7f8e864deb80 4 client.26183 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:56:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.263+0000 7f8e864deb80 1 -- 172.21.15.78:0/3044594369 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ebc1767700 con 0x55ebc18368a0 2024-03-14T09:56:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.263+0000 7f8e6be00700 1 --2- 172.21.15.78:0/3044594369 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55ebc18368a0 0x55ebc1838cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.264+0000 7f8e55600700 10 client.26183 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:56:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.264+0000 7f8e6be00700 1 --2- 172.21.15.78:0/3044594369 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55ebc18368a0 0x55ebc1838cc0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=154f5b4344a44386 server_cookie=c90355c9cfe0476c in_seq=0 out_seq=0 2024-03-14T09:56:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.264+0000 7f8e55600700 1 -- 172.21.15.78:0/3044594369 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55ebc1767700 con 0x55ebc18368a0 2024-03-14T09:56:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.264+0000 7f8e55600700 10 client.26183 handle_command_reply: tid=0 2024-03-14T09:56:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.264+0000 7f8e864deb80 1 client.26183 shutdown 2024-03-14T09:56:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.265+0000 7f8e864deb80 1 -- 172.21.15.78:0/3044594369 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc1826ec0 msgr2=0x55ebc18243a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.265+0000 7f8e864deb80 1 --2- 172.21.15.78:0/3044594369 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc1826ec0 0x55ebc18243a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8e640107a0 tx=0x7f8e64010ab0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.265+0000 7f8e864deb80 1 -- 172.21.15.78:0/3044594369 shutdown_connections 2024-03-14T09:56:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.265+0000 7f8e864deb80 1 --2- 172.21.15.78:0/3044594369 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55ebc18368a0 0x55ebc1838cc0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.265+0000 7f8e864deb80 1 --2- 172.21.15.78:0/3044594369 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc1826ec0 0x55ebc18243a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.265+0000 7f8e864deb80 1 --2- 172.21.15.78:0/3044594369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181c680 0x55ebc1826920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.265+0000 7f8e864deb80 1 --2- 172.21.15.78:0/3044594369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc181cfa0 0x55ebc1823c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.265+0000 7f8e864deb80 1 -- 172.21.15.78:0/3044594369 >> 172.21.15.78:0/3044594369 conn(0x55ebc1792220 msgr2=0x55ebc1817690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.265+0000 7f8e55600700 0 client.26183 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:56:48.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.265+0000 7f8e864deb80 1 -- 172.21.15.78:0/3044594369 shutdown_connections 2024-03-14T09:56:48.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.265+0000 7f8e864deb80 1 -- 172.21.15.78:0/3044594369 wait complete. 2024-03-14T09:56:48.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.265+0000 7f8e864deb80 20 client.26183 trim_cache size 0 max 16384 2024-03-14T09:56:48.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.281+0000 7f8e864deb80 1 Processor -- start 2024-03-14T09:56:48.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.281+0000 7f8e864deb80 1 -- start start 2024-03-14T09:56:48.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.281+0000 7f8e864deb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 0x55ebc182d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.281+0000 7f8e864deb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc181cfa0 0x55ebc1824260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.281+0000 7f8e864deb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc1826ec0 0x55ebc1826450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.281+0000 7f8e864deb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc1824f60 con 0x55ebc1826ec0 2024-03-14T09:56:48.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.281+0000 7f8e864deb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc183d9e0 con 0x55ebc181c680 2024-03-14T09:56:48.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.281+0000 7f8e864deb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc181e190 con 0x55ebc181cfa0 2024-03-14T09:56:48.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.282+0000 7f8e6b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc181cfa0 0x55ebc1824260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.282+0000 7f8e6b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc181cfa0 0x55ebc1824260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:33894/0 (socket says 172.21.15.78:33894) 2024-03-14T09:56:48.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.282+0000 7f8e6b400700 1 -- 172.21.15.78:0/3356418967 learned_addr learned my addr 172.21.15.78:0/3356418967 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:48.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.282+0000 7f8e70a00700 1 --2- 172.21.15.78:0/3356418967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc1826ec0 0x55ebc1826450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.282+0000 7f8e6be00700 1 --2- 172.21.15.78:0/3356418967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 0x55ebc182d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.282+0000 7f8e6b400700 1 -- 172.21.15.78:0/3356418967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 msgr2=0x55ebc182d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.282+0000 7f8e6b400700 1 --2- 172.21.15.78:0/3356418967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 0x55ebc182d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.282+0000 7f8e6b400700 1 -- 172.21.15.78:0/3356418967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc1826ec0 msgr2=0x55ebc1826450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.282+0000 7f8e6b400700 1 --2- 172.21.15.78:0/3356418967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc1826ec0 0x55ebc1826450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.282+0000 7f8e6b400700 1 -- 172.21.15.78:0/3356418967 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebc1734ea0 con 0x55ebc181cfa0 2024-03-14T09:56:48.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.282+0000 7f8e6b400700 1 --2- 172.21.15.78:0/3356418967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc181cfa0 0x55ebc1824260 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8e5809a440 tx=0x7f8e5809a630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=183c2f29cd887aa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.282+0000 7f8e61c00700 1 -- 172.21.15.78:0/3356418967 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58022ea0 con 0x55ebc181cfa0 2024-03-14T09:56:48.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.282+0000 7f8e61c00700 1 -- 172.21.15.78:0/3356418967 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e580a28a0 con 0x55ebc181cfa0 2024-03-14T09:56:48.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.283+0000 7f8e61c00700 1 -- 172.21.15.78:0/3356418967 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e580999a0 con 0x55ebc181cfa0 2024-03-14T09:56:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.283+0000 7f8e864deb80 1 -- 172.21.15.78:0/3356418967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc181cfa0 msgr2=0x55ebc1824260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.283+0000 7f8e864deb80 1 --2- 172.21.15.78:0/3356418967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc181cfa0 0x55ebc1824260 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8e5809a440 tx=0x7f8e5809a630 comp rx=0 tx=0).stop 2024-03-14T09:56:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.283+0000 7f8e864deb80 1 -- 172.21.15.78:0/3356418967 shutdown_connections 2024-03-14T09:56:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.283+0000 7f8e864deb80 1 --2- 172.21.15.78:0/3356418967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 0x55ebc182d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.283+0000 7f8e864deb80 1 --2- 172.21.15.78:0/3356418967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc1826ec0 0x55ebc1826450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.283+0000 7f8e864deb80 1 --2- 172.21.15.78:0/3356418967 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc181cfa0 0x55ebc1824260 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.283+0000 7f8e864deb80 1 -- 172.21.15.78:0/3356418967 >> 172.21.15.78:0/3356418967 conn(0x55ebc1792220 msgr2=0x55ebc1813680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.283+0000 7f8e864deb80 1 -- 172.21.15.78:0/3356418967 shutdown_connections 2024-03-14T09:56:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.284+0000 7f8e864deb80 1 -- 172.21.15.78:0/3356418967 wait complete. 2024-03-14T09:56:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.284+0000 7f8e864deb80 1 -- start start 2024-03-14T09:56:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.284+0000 7f8e864deb80 1 Processor -- start 2024-03-14T09:56:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.284+0000 7f8e864deb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 0x55ebc182b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.285+0000 7f8e864deb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181cfa0 0x55ebc182bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.285+0000 7f8e864deb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc1826ec0 0x55ebc18873c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.285+0000 7f8e864deb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc1887900 con 0x55ebc181cfa0 2024-03-14T09:56:48.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.285+0000 7f8e864deb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc1887a40 con 0x55ebc181c680 2024-03-14T09:56:48.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.285+0000 7f8e864deb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebc1887c00 con 0x55ebc1826ec0 2024-03-14T09:56:48.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.285+0000 7f8e864deb80 10 client.0 _ll_register_callbacks cb 0x7f8e64010270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:48.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.285+0000 7f8e6b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181cfa0 0x55ebc182bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.285+0000 7f8e6b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181cfa0 0x55ebc182bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60166/0 (socket says 172.21.15.78:60166) 2024-03-14T09:56:48.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.285+0000 7f8e6b400700 1 -- 172.21.15.78:0/3586076629 learned_addr learned my addr 172.21.15.78:0/3586076629 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:48.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.285+0000 7f8e70a00700 1 --2- 172.21.15.78:0/3586076629 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc1826ec0 0x55ebc18873c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.285+0000 7f8e6be00700 1 --2- 172.21.15.78:0/3586076629 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 0x55ebc182b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.285+0000 7f8e6b400700 1 -- 172.21.15.78:0/3586076629 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc1826ec0 msgr2=0x55ebc18873c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.285+0000 7f8e6b400700 1 --2- 172.21.15.78:0/3586076629 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc1826ec0 0x55ebc18873c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.285+0000 7f8e6b400700 1 -- 172.21.15.78:0/3586076629 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 msgr2=0x55ebc182b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.285+0000 7f8e6b400700 1 --2- 172.21.15.78:0/3586076629 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 0x55ebc182b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.285+0000 7f8e6b400700 1 -- 172.21.15.78:0/3586076629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ebc15f03b0 con 0x55ebc181cfa0 2024-03-14T09:56:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.286+0000 7f8e6be00700 1 --2- 172.21.15.78:0/3586076629 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 0x55ebc182b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:56:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.286+0000 7f8e6b400700 1 --2- 172.21.15.78:0/3586076629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181cfa0 0x55ebc182bae0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8e580cd6a0 tx=0x7f8e58005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57c80bc2be179ef server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.286+0000 7f8e55600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:56:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.286+0000 7f8e55600700 1 -- 172.21.15.78:0/3586076629 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58022ea0 con 0x55ebc181cfa0 2024-03-14T09:56:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.286+0000 7f8e55600700 1 -- 172.21.15.78:0/3586076629 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f8e580cd1e0 con 0x55ebc181cfa0 2024-03-14T09:56:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.286+0000 7f8e864deb80 1 -- 172.21.15.78:0/3586076629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e5809e2e0 con 0x55ebc181cfa0 2024-03-14T09:56:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.286+0000 7f8e55600700 1 -- 172.21.15.78:0/3586076629 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e580999a0 con 0x55ebc181cfa0 2024-03-14T09:56:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.286+0000 7f8e864deb80 10 client.16173 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.286+0000 7f8e864deb80 1 -- 172.21.15.78:0/3586076629 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ebc15f4cd0 con 0x55ebc181cfa0 2024-03-14T09:56:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.287+0000 7f8e55600700 1 -- 172.21.15.78:0/3586076629 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e5809e2e0 con 0x55ebc181cfa0 2024-03-14T09:56:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.287+0000 7f8e55600700 1 -- 172.21.15.78:0/3586076629 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8e58081070 con 0x55ebc181cfa0 2024-03-14T09:56:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.287+0000 7f8e864deb80 10 client.16173 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.287+0000 7f8e864deb80 10 client.16173 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:56:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.287+0000 7f8e864deb80 20 client.16173 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.287+0000 7f8e864deb80 1 --2- 172.21.15.78:0/3586076629 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55ebc1888ef0 0x55ebc188b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.287+0000 7f8e864deb80 4 client.16173 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:56:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.287+0000 7f8e864deb80 1 -- 172.21.15.78:0/3586076629 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ebc15bdcf0 con 0x55ebc1888ef0 2024-03-14T09:56:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.287+0000 7f8e6be00700 1 --2- 172.21.15.78:0/3586076629 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55ebc1888ef0 0x55ebc188b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.287+0000 7f8e6be00700 1 --2- 172.21.15.78:0/3586076629 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55ebc1888ef0 0x55ebc188b2e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6f4c254f231bf03e server_cookie=e7a689622e749c14 in_seq=0 out_seq=0 2024-03-14T09:56:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.287+0000 7f8e55600700 10 client.16173 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:56:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.288+0000 7f8e55600700 1 -- 172.21.15.78:0/3586076629 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ebc15bdcf0 con 0x55ebc1888ef0 2024-03-14T09:56:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.288+0000 7f8e55600700 10 client.16173 handle_command_reply: tid=0 2024-03-14T09:56:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.288+0000 7f8e864deb80 1 client.16173 shutdown 2024-03-14T09:56:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.288+0000 7f8e864deb80 1 -- 172.21.15.78:0/3586076629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181cfa0 msgr2=0x55ebc182bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.288+0000 7f8e864deb80 1 --2- 172.21.15.78:0/3586076629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181cfa0 0x55ebc182bae0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8e580cd6a0 tx=0x7f8e58005ef0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.288+0000 7f8e864deb80 1 -- 172.21.15.78:0/3586076629 shutdown_connections 2024-03-14T09:56:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.288+0000 7f8e864deb80 1 --2- 172.21.15.78:0/3586076629 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55ebc1888ef0 0x55ebc188b2e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.288+0000 7f8e864deb80 1 --2- 172.21.15.78:0/3586076629 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ebc181c680 0x55ebc182b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.288+0000 7f8e55600700 0 client.16173 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:56:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.288+0000 7f8e864deb80 1 --2- 172.21.15.78:0/3586076629 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ebc1826ec0 0x55ebc18873c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.288+0000 7f8e864deb80 1 --2- 172.21.15.78:0/3586076629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ebc181cfa0 0x55ebc182bae0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.288+0000 7f8e864deb80 1 -- 172.21.15.78:0/3586076629 >> 172.21.15.78:0/3586076629 conn(0x55ebc1792220 msgr2=0x55ebc181ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.288+0000 7f8e864deb80 1 -- 172.21.15.78:0/3586076629 shutdown_connections 2024-03-14T09:56:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.288+0000 7f8e864deb80 1 -- 172.21.15.78:0/3586076629 wait complete. 2024-03-14T09:56:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.288+0000 7f8e864deb80 20 client.16173 trim_cache size 0 max 16384 2024-03-14T09:56:48.293 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:56:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.289+0000 7f8e72a00700 1 -- 172.21.15.78:0/3111555296 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8e500a0c80 msgr2=0x7f8e500a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.289+0000 7f8e72a00700 1 --2- 172.21.15.78:0/3111555296 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8e500a0c80 0x7f8e500a3140 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c009fd0 tx=0x7f8e5c009c60 comp rx=0 tx=0).stop 2024-03-14T09:56:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.289+0000 7f8e72a00700 1 -- 172.21.15.78:0/3111555296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e6c0fffd0 msgr2=0x7f8e6c1a3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.289+0000 7f8e72a00700 1 --2- 172.21.15.78:0/3111555296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e6c0fffd0 0x7f8e6c1a3d60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800b580 tx=0x7f8e5800f690 comp rx=0 tx=0).stop 2024-03-14T09:56:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.289+0000 7f8e72a00700 1 -- 172.21.15.78:0/3111555296 shutdown_connections 2024-03-14T09:56:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.289+0000 7f8e72a00700 1 --2- 172.21.15.78:0/3111555296 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8e500a0c80 0x7f8e500a3140 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.289+0000 7f8e72a00700 1 --2- 172.21.15.78:0/3111555296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8e6c101b60 0x7f8e6c19d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.289+0000 7f8e72a00700 1 --2- 172.21.15.78:0/3111555296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8e6c0ff610 0x7f8e6c1a3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.289+0000 7f8e72a00700 1 --2- 172.21.15.78:0/3111555296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8e6c0fffd0 0x7f8e6c1a3d60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.289+0000 7f8e72a00700 1 -- 172.21.15.78:0/3111555296 >> 172.21.15.78:0/3111555296 conn(0x7f8e6c074f70 msgr2=0x7f8e6c10f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.289+0000 7f8e72a00700 1 -- 172.21.15.78:0/3111555296 shutdown_connections 2024-03-14T09:56:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.289+0000 7f8e72a00700 1 -- 172.21.15.78:0/3111555296 wait complete. 2024-03-14T09:56:48.311 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T09:56:48.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.542+0000 7f41e6a00700 1 -- 172.21.15.78:0/157976811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41e00680b0 msgr2=0x7f41e0068490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.542+0000 7f41e6a00700 1 --2- 172.21.15.78:0/157976811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41e00680b0 0x7f41e0068490 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009990 tx=0x7f41d0009ca0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.542+0000 7f41e6a00700 1 -- 172.21.15.78:0/157976811 shutdown_connections 2024-03-14T09:56:48.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.542+0000 7f41e6a00700 1 --2- 172.21.15.78:0/157976811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41e00680b0 0x7f41e0068490 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.542+0000 7f41e6a00700 1 --2- 172.21.15.78:0/157976811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e0106060 0x7f41e010dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.542+0000 7f41e6a00700 1 --2- 172.21.15.78:0/157976811 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41e0068a60 0x7f41e0105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.542+0000 7f41e6a00700 1 -- 172.21.15.78:0/157976811 >> 172.21.15.78:0/157976811 conn(0x7f41e0076d80 msgr2=0x7f41e0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.542+0000 7f41e6a00700 1 -- 172.21.15.78:0/157976811 shutdown_connections 2024-03-14T09:56:48.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.542+0000 7f41e6a00700 1 -- 172.21.15.78:0/157976811 wait complete. 2024-03-14T09:56:48.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.544+0000 7f41e6a00700 1 Processor -- start 2024-03-14T09:56:48.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.544+0000 7f41e6a00700 1 -- start start 2024-03-14T09:56:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.544+0000 7f41e6a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41e00680b0 0x7f41e0197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.544+0000 7f41e6a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e0106060 0x7f41e0197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.544+0000 7f41e6a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41e019bef0 0x7f41e019c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.544+0000 7f41e6a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e019c970 con 0x7f41e0106060 2024-03-14T09:56:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.544+0000 7f41e6a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e019cae0 con 0x7f41e019bef0 2024-03-14T09:56:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.544+0000 7f41e6a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e019cca0 con 0x7f41e00680b0 2024-03-14T09:56:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.544+0000 7f41df400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e0106060 0x7f41e0197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.544+0000 7f41dfe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41e00680b0 0x7f41e0197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.544+0000 7f41dfe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41e00680b0 0x7f41e0197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:33918/0 (socket says 172.21.15.78:33918) 2024-03-14T09:56:48.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.544+0000 7f41dfe00700 1 -- 172.21.15.78:0/3572958015 learned_addr learned my addr 172.21.15.78:0/3572958015 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.544+0000 7f41e4a00700 1 --2- 172.21.15.78:0/3572958015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41e019bef0 0x7f41e019c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.545+0000 7f41dfe00700 1 -- 172.21.15.78:0/3572958015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41e019bef0 msgr2=0x7f41e019c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.545+0000 7f41dfe00700 1 --2- 172.21.15.78:0/3572958015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41e019bef0 0x7f41e019c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.545+0000 7f41dfe00700 1 -- 172.21.15.78:0/3572958015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e0106060 msgr2=0x7f41e0197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.545+0000 7f41dfe00700 1 --2- 172.21.15.78:0/3572958015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e0106060 0x7f41e0197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.545+0000 7f41dfe00700 1 -- 172.21.15.78:0/3572958015 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d00095f0 con 0x7f41e00680b0 2024-03-14T09:56:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.545+0000 7f41dfe00700 1 --2- 172.21.15.78:0/3572958015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41e00680b0 0x7f41e0197610 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f41d0005910 tx=0x7f41d0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.545+0000 7f41dcc00700 1 -- 172.21.15.78:0/3572958015 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d001d070 con 0x7f41e00680b0 2024-03-14T09:56:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.545+0000 7f41dcc00700 1 -- 172.21.15.78:0/3572958015 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41d0022470 con 0x7f41e00680b0 2024-03-14T09:56:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.545+0000 7f41e6a00700 1 -- 172.21.15.78:0/3572958015 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e01a41a0 con 0x7f41e00680b0 2024-03-14T09:56:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.546+0000 7f41dcc00700 1 -- 172.21.15.78:0/3572958015 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d0017650 con 0x7f41e00680b0 2024-03-14T09:56:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.546+0000 7f41e6a00700 1 -- 172.21.15.78:0/3572958015 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e01a4620 con 0x7f41e00680b0 2024-03-14T09:56:48.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.550+0000 7f41dcc00700 1 -- 172.21.15.78:0/3572958015 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f41d00177b0 con 0x7f41e00680b0 2024-03-14T09:56:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.552+0000 7f41fa49db80 1 Processor -- start 2024-03-14T09:56:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.552+0000 7f41fa49db80 1 -- start start 2024-03-14T09:56:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.552+0000 7f41fa49db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff845680 0x5640ff845a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.552+0000 7f41fa49db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845fa0 0x5640ff84f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.552+0000 7f41fa49db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff84fec0 0x5640ff8522b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.552+0000 7f41dcc00700 1 --2- 172.21.15.78:0/3572958015 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f41c40a0c80 0x7f41c40a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.552+0000 7f41fa49db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640ff79e990 con 0x5640ff845680 2024-03-14T09:56:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.552+0000 7f41fa49db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640ff79c350 con 0x5640ff84fec0 2024-03-14T09:56:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.552+0000 7f41fa49db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640ff797c40 con 0x5640ff845fa0 2024-03-14T09:56:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.552+0000 7f41dcc00700 1 -- 172.21.15.78:0/3572958015 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f41d00cd850 con 0x7f41e00680b0 2024-03-14T09:56:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.552+0000 7f41dfe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845fa0 0x5640ff84f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.552+0000 7f41dfe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845fa0 0x5640ff84f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:33934/0 (socket says 172.21.15.78:33934) 2024-03-14T09:56:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.552+0000 7f41dfe00700 1 -- 172.21.15.78:0/1261399355 learned_addr learned my addr 172.21.15.78:0/1261399355 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.553+0000 7f41e4a00700 1 --2- 172.21.15.78:0/1261399355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff84fec0 0x5640ff8522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.553+0000 7f41dfe00700 1 -- 172.21.15.78:0/1261399355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff84fec0 msgr2=0x5640ff8522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.553+0000 7f41dfe00700 1 --2- 172.21.15.78:0/1261399355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff84fec0 0x5640ff8522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.553+0000 7f41dfe00700 1 -- 172.21.15.78:0/1261399355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff845680 msgr2=0x5640ff845a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.553+0000 7f41dfe00700 1 --2- 172.21.15.78:0/1261399355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff845680 0x5640ff845a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.553+0000 7f41dfe00700 1 -- 172.21.15.78:0/1261399355 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640ff6193b0 con 0x5640ff845fa0 2024-03-14T09:56:48.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.553+0000 7f41dfe00700 1 --2- 172.21.15.78:0/1261399355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845fa0 0x5640ff84f980 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f41d0006010 tx=0x7f41d0096600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8aca9544c8f38d35 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.553+0000 7f41d5200700 1 -- 172.21.15.78:0/1261399355 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d009a220 con 0x5640ff845fa0 2024-03-14T09:56:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.553+0000 7f41df400700 1 --2- 172.21.15.78:0/3572958015 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f41c40a0c80 0x7f41c40a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.553+0000 7f41d5200700 1 -- 172.21.15.78:0/1261399355 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41d009afc0 con 0x5640ff845fa0 2024-03-14T09:56:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.553+0000 7f41d5200700 1 -- 172.21.15.78:0/1261399355 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d009ab20 con 0x5640ff845fa0 2024-03-14T09:56:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.554+0000 7f41fa49db80 1 -- 172.21.15.78:0/1261399355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845fa0 msgr2=0x5640ff84f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.554+0000 7f41df400700 1 --2- 172.21.15.78:0/3572958015 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f41c40a0c80 0x7f41c40a3140 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f41e0198b60 tx=0x7f41cc009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.554+0000 7f41fa49db80 1 --2- 172.21.15.78:0/1261399355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845fa0 0x5640ff84f980 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f41d0006010 tx=0x7f41d0096600 comp rx=0 tx=0).stop 2024-03-14T09:56:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.554+0000 7f41fa49db80 1 -- 172.21.15.78:0/1261399355 shutdown_connections 2024-03-14T09:56:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.554+0000 7f41fa49db80 1 --2- 172.21.15.78:0/1261399355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff84fec0 0x5640ff8522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.554+0000 7f41fa49db80 1 --2- 172.21.15.78:0/1261399355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff845680 0x5640ff845a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.554+0000 7f41fa49db80 1 --2- 172.21.15.78:0/1261399355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845fa0 0x5640ff84f980 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.554+0000 7f41fa49db80 1 -- 172.21.15.78:0/1261399355 >> 172.21.15.78:0/1261399355 conn(0x5640ff7bb220 msgr2=0x5640ff840aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.555+0000 7f41fa49db80 1 -- 172.21.15.78:0/1261399355 shutdown_connections 2024-03-14T09:56:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.555+0000 7f41fa49db80 1 -- 172.21.15.78:0/1261399355 wait complete. 2024-03-14T09:56:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.555+0000 7f41fa49db80 1 -- start start 2024-03-14T09:56:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.556+0000 7f41fa49db80 1 Processor -- start 2024-03-14T09:56:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.556+0000 7f41fa49db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff845680 0x5640ff84f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.556+0000 7f41fa49db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff845fa0 0x5640ff84cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.556+0000 7f41fa49db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff84fec0 0x5640ff84d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.556+0000 7f41fa49db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640ff733d80 con 0x5640ff845680 2024-03-14T09:56:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.556+0000 7f41fa49db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640ff733a20 con 0x5640ff845fa0 2024-03-14T09:56:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.556+0000 7f41fa49db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640ff61b6a0 con 0x5640ff84fec0 2024-03-14T09:56:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.556+0000 7f41fa49db80 10 client.0 _ll_register_callbacks cb 0x5640ff59f3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.557+0000 7f41e4a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff84fec0 0x5640ff84d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.557+0000 7f41e4a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff84fec0 0x5640ff84d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:33944/0 (socket says 172.21.15.78:33944) 2024-03-14T09:56:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.557+0000 7f41e4a00700 1 -- 172.21.15.78:0/3533326926 learned_addr learned my addr 172.21.15.78:0/3533326926 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.557+0000 7f41e4a00700 1 -- 172.21.15.78:0/3533326926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff845fa0 msgr2=0x5640ff84cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.557+0000 7f41df400700 1 --2- 172.21.15.78:0/3533326926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff845fa0 0x5640ff84cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.557+0000 7f41e4a00700 1 --2- 172.21.15.78:0/3533326926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff845fa0 0x5640ff84cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.557+0000 7f41e4a00700 1 -- 172.21.15.78:0/3533326926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff845680 msgr2=0x5640ff84f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.557+0000 7f41e4a00700 1 --2- 172.21.15.78:0/3533326926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff845680 0x5640ff84f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.557+0000 7f41e4a00700 1 -- 172.21.15.78:0/3533326926 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e01a4620 con 0x5640ff84fec0 2024-03-14T09:56:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.558+0000 7f41e4a00700 1 --2- 172.21.15.78:0/3533326926 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff84fec0 0x5640ff84d120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f41d8007ba0 tx=0x7f41d8007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ccbc32bf7c74417 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.558+0000 7f41c9600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:56:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.558+0000 7f41c9600700 1 -- 172.21.15.78:0/3533326926 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d800e070 con 0x5640ff84fec0 2024-03-14T09:56:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.558+0000 7f41fa49db80 1 -- 172.21.15.78:0/3533326926 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5640ff838720 con 0x5640ff84fec0 2024-03-14T09:56:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.558+0000 7f41c9600700 1 -- 172.21.15.78:0/3533326926 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f41d8013b30 con 0x5640ff84fec0 2024-03-14T09:56:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.558+0000 7f41c9600700 1 -- 172.21.15.78:0/3533326926 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41d8013cf0 con 0x5640ff84fec0 2024-03-14T09:56:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.558+0000 7f41fa49db80 10 client.26212 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.558+0000 7f41fa49db80 1 -- 172.21.15.78:0/3533326926 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5640ff5c5b60 con 0x5640ff84fec0 2024-03-14T09:56:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.558+0000 7f41c9600700 1 -- 172.21.15.78:0/3533326926 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5640ff838720 con 0x5640ff84fec0 2024-03-14T09:56:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.559+0000 7f41c9600700 1 -- 172.21.15.78:0/3533326926 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f41d80163d0 con 0x5640ff84fec0 2024-03-14T09:56:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.559+0000 7f41fa49db80 10 client.26212 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.559+0000 7f41fa49db80 10 client.26212 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:56:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.559+0000 7f41fa49db80 20 client.26212 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.559+0000 7f41fa49db80 1 --2- 172.21.15.78:0/3533326926 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5640ff863bc0 0x5640ff865fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.559+0000 7f41fa49db80 4 client.26212 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:56:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.559+0000 7f41fa49db80 1 -- 172.21.15.78:0/3533326926 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5640ff790700 con 0x5640ff863bc0 2024-03-14T09:56:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.560+0000 7f41dfe00700 1 --2- 172.21.15.78:0/3533326926 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5640ff863bc0 0x5640ff865fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.560+0000 7f41dfe00700 1 --2- 172.21.15.78:0/3533326926 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5640ff863bc0 0x5640ff865fe0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e15c0304ba27e619 server_cookie=1a690b6a63352907 in_seq=0 out_seq=0 2024-03-14T09:56:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.560+0000 7f41c9600700 10 client.26212 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:56:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.561+0000 7f41c9600700 1 -- 172.21.15.78:0/3533326926 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5640ff790700 con 0x5640ff863bc0 2024-03-14T09:56:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.561+0000 7f41c9600700 10 client.26212 handle_command_reply: tid=0 2024-03-14T09:56:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.561+0000 7f41fa49db80 1 client.26212 shutdown 2024-03-14T09:56:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.561+0000 7f41fa49db80 1 -- 172.21.15.78:0/3533326926 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff84fec0 msgr2=0x5640ff84d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.561+0000 7f41fa49db80 1 --2- 172.21.15.78:0/3533326926 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff84fec0 0x5640ff84d120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f41d8007ba0 tx=0x7f41d8007f60 comp rx=0 tx=0).stop 2024-03-14T09:56:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.561+0000 7f41fa49db80 1 -- 172.21.15.78:0/3533326926 shutdown_connections 2024-03-14T09:56:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.561+0000 7f41fa49db80 1 --2- 172.21.15.78:0/3533326926 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5640ff863bc0 0x5640ff865fe0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.561+0000 7f41c9600700 0 client.26212 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:56:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.561+0000 7f41fa49db80 1 --2- 172.21.15.78:0/3533326926 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff84fec0 0x5640ff84d120 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.561+0000 7f41fa49db80 1 --2- 172.21.15.78:0/3533326926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff845680 0x5640ff84f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.561+0000 7f41fa49db80 1 --2- 172.21.15.78:0/3533326926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff845fa0 0x5640ff84cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.561+0000 7f41fa49db80 1 -- 172.21.15.78:0/3533326926 >> 172.21.15.78:0/3533326926 conn(0x5640ff7bb220 msgr2=0x5640ff840690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.562+0000 7f41fa49db80 1 -- 172.21.15.78:0/3533326926 shutdown_connections 2024-03-14T09:56:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.562+0000 7f41fa49db80 1 -- 172.21.15.78:0/3533326926 wait complete. 2024-03-14T09:56:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.562+0000 7f41fa49db80 20 client.26212 trim_cache size 0 max 16384 2024-03-14T09:56:48.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.577+0000 7f41fa49db80 1 Processor -- start 2024-03-14T09:56:48.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.577+0000 7f41fa49db80 1 -- start start 2024-03-14T09:56:48.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.577+0000 7f41fa49db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff845680 0x5640ff84d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.578+0000 7f41fa49db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845fa0 0x5640ff85f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.578+0000 7f41fa49db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff84fec0 0x5640ff84f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.578+0000 7f41fa49db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640ff841e60 con 0x5640ff845680 2024-03-14T09:56:48.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.578+0000 7f41fa49db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640ff83ad00 con 0x5640ff84fec0 2024-03-14T09:56:48.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.578+0000 7f41fa49db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640ff86ad00 con 0x5640ff845fa0 2024-03-14T09:56:48.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.578+0000 7f41df400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845fa0 0x5640ff85f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.578+0000 7f41dfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff845680 0x5640ff84d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.578+0000 7f41df400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845fa0 0x5640ff85f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:33956/0 (socket says 172.21.15.78:33956) 2024-03-14T09:56:48.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.578+0000 7f41df400700 1 -- 172.21.15.78:0/91157776 learned_addr learned my addr 172.21.15.78:0/91157776 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:48.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.578+0000 7f41dfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff845680 0x5640ff84d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60210/0 (socket says 172.21.15.78:60210) 2024-03-14T09:56:48.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.578+0000 7f41e4a00700 1 --2- 172.21.15.78:0/91157776 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff84fec0 0x5640ff84f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.578+0000 7f41df400700 1 -- 172.21.15.78:0/91157776 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff84fec0 msgr2=0x5640ff84f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.578+0000 7f41df400700 1 --2- 172.21.15.78:0/91157776 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff84fec0 0x5640ff84f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.578+0000 7f41df400700 1 -- 172.21.15.78:0/91157776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff845680 msgr2=0x5640ff84d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.578+0000 7f41df400700 1 --2- 172.21.15.78:0/91157776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff845680 0x5640ff84d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.578+0000 7f41df400700 1 -- 172.21.15.78:0/91157776 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640ff75dea0 con 0x5640ff845fa0 2024-03-14T09:56:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.579+0000 7f41df400700 1 --2- 172.21.15.78:0/91157776 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845fa0 0x5640ff85f9b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f41cc011cb0 tx=0x7f41cc014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8c97122ff2267da server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.579+0000 7f41d5c00700 1 -- 172.21.15.78:0/91157776 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc004050 con 0x5640ff845fa0 2024-03-14T09:56:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.579+0000 7f41d5c00700 1 -- 172.21.15.78:0/91157776 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41cc01d7c0 con 0x5640ff845fa0 2024-03-14T09:56:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.579+0000 7f41d5c00700 1 -- 172.21.15.78:0/91157776 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc003a80 con 0x5640ff845fa0 2024-03-14T09:56:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.580+0000 7f41fa49db80 1 -- 172.21.15.78:0/91157776 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845fa0 msgr2=0x5640ff85f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.580+0000 7f41fa49db80 1 --2- 172.21.15.78:0/91157776 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845fa0 0x5640ff85f9b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f41cc011cb0 tx=0x7f41cc014000 comp rx=0 tx=0).stop 2024-03-14T09:56:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.580+0000 7f41fa49db80 1 -- 172.21.15.78:0/91157776 shutdown_connections 2024-03-14T09:56:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.580+0000 7f41fa49db80 1 --2- 172.21.15.78:0/91157776 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff84fec0 0x5640ff84f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.580+0000 7f41fa49db80 1 --2- 172.21.15.78:0/91157776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff845680 0x5640ff84d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.580+0000 7f41fa49db80 1 --2- 172.21.15.78:0/91157776 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845fa0 0x5640ff85f9b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.580+0000 7f41fa49db80 1 -- 172.21.15.78:0/91157776 >> 172.21.15.78:0/91157776 conn(0x5640ff7bb220 msgr2=0x5640ff85f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.580+0000 7f41fa49db80 1 -- 172.21.15.78:0/91157776 shutdown_connections 2024-03-14T09:56:48.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.580+0000 7f41fa49db80 1 -- 172.21.15.78:0/91157776 wait complete. 2024-03-14T09:56:48.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.580+0000 7f41fa49db80 1 -- start start 2024-03-14T09:56:48.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.581+0000 7f41fa49db80 1 Processor -- start 2024-03-14T09:56:48.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.581+0000 7f41fa49db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845680 0x5640ff85a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.581+0000 7f41fa49db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff845fa0 0x5640ff858d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.581+0000 7f41fa49db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff84fec0 0x5640ff859270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.581+0000 7f41fa49db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640ff8597b0 con 0x5640ff84fec0 2024-03-14T09:56:48.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.581+0000 7f41fa49db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640ff859920 con 0x5640ff845fa0 2024-03-14T09:56:48.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.581+0000 7f41fa49db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640ff859ae0 con 0x5640ff845680 2024-03-14T09:56:48.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.581+0000 7f41fa49db80 10 client.0 _ll_register_callbacks cb 0x5640ff59f3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:48.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.581+0000 7f41dfe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845680 0x5640ff85a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.582+0000 7f41dfe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845680 0x5640ff85a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:33966/0 (socket says 172.21.15.78:33966) 2024-03-14T09:56:48.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.582+0000 7f41dfe00700 1 -- 172.21.15.78:0/2467479001 learned_addr learned my addr 172.21.15.78:0/2467479001 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:48.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.582+0000 7f41df400700 1 --2- 172.21.15.78:0/2467479001 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff845fa0 0x5640ff858d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.582+0000 7f41e4a00700 1 --2- 172.21.15.78:0/2467479001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff84fec0 0x5640ff859270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.582+0000 7f41dfe00700 1 -- 172.21.15.78:0/2467479001 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff845fa0 msgr2=0x5640ff858d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.582+0000 7f41dfe00700 1 --2- 172.21.15.78:0/2467479001 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff845fa0 0x5640ff858d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.582+0000 7f41dfe00700 1 -- 172.21.15.78:0/2467479001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff84fec0 msgr2=0x5640ff859270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.582+0000 7f41dfe00700 1 --2- 172.21.15.78:0/2467479001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff84fec0 0x5640ff859270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.582+0000 7f41dfe00700 1 -- 172.21.15.78:0/2467479001 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41cc017070 con 0x5640ff845680 2024-03-14T09:56:48.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.582+0000 7f41dfe00700 1 --2- 172.21.15.78:0/2467479001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845680 0x5640ff85a8a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f41d009e8b0 tx=0x7f41d0009910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37379b8beaac5668 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.582+0000 7f41c9600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:56:48.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.582+0000 7f41c9600700 1 -- 172.21.15.78:0/2467479001 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d00ba2e0 con 0x5640ff845680 2024-03-14T09:56:48.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.582+0000 7f41c9600700 1 -- 172.21.15.78:0/2467479001 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f41d009a8c0 con 0x5640ff845680 2024-03-14T09:56:48.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.582+0000 7f41fa49db80 1 -- 172.21.15.78:0/2467479001 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f41d009afc0 con 0x5640ff845680 2024-03-14T09:56:48.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.583+0000 7f41c9600700 1 -- 172.21.15.78:0/2467479001 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41d00a2cb0 con 0x5640ff845680 2024-03-14T09:56:48.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.583+0000 7f41fa49db80 10 client.26218 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:48.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.583+0000 7f41fa49db80 1 -- 172.21.15.78:0/2467479001 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5640ff61dcd0 con 0x5640ff845680 2024-03-14T09:56:48.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.583+0000 7f41c9600700 1 -- 172.21.15.78:0/2467479001 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41d00b2070 con 0x5640ff845680 2024-03-14T09:56:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.583+0000 7f41c9600700 1 -- 172.21.15.78:0/2467479001 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f41d0099ed0 con 0x5640ff845680 2024-03-14T09:56:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.583+0000 7f41fa49db80 10 client.26218 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.583+0000 7f41fa49db80 10 client.26218 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:56:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.583+0000 7f41fa49db80 20 client.26218 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.583+0000 7f41fa49db80 1 --2- 172.21.15.78:0/2467479001 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5640ff8b6700 0x5640ff8b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.583+0000 7f41fa49db80 4 client.26218 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:56:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.583+0000 7f41fa49db80 1 -- 172.21.15.78:0/2467479001 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5640ff5e6cf0 con 0x5640ff8b6700 2024-03-14T09:56:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.583+0000 7f41df400700 1 --2- 172.21.15.78:0/2467479001 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5640ff8b6700 0x5640ff8b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.584+0000 7f41df400700 1 --2- 172.21.15.78:0/2467479001 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5640ff8b6700 0x5640ff8b8b20 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7c2da286bf5dd7a5 server_cookie=d4284014ec70af2e in_seq=0 out_seq=0 2024-03-14T09:56:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.584+0000 7f41c9600700 10 client.26218 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:56:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.584+0000 7f41c9600700 1 -- 172.21.15.78:0/2467479001 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5640ff5e6cf0 con 0x5640ff8b6700 2024-03-14T09:56:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.584+0000 7f41c9600700 10 client.26218 handle_command_reply: tid=0 2024-03-14T09:56:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.584+0000 7f41fa49db80 1 client.26218 shutdown 2024-03-14T09:56:48.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.584+0000 7f41fa49db80 1 -- 172.21.15.78:0/2467479001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845680 msgr2=0x5640ff85a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.584+0000 7f41fa49db80 1 --2- 172.21.15.78:0/2467479001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845680 0x5640ff85a8a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f41d009e8b0 tx=0x7f41d0009910 comp rx=0 tx=0).stop 2024-03-14T09:56:48.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.585+0000 7f41fa49db80 1 -- 172.21.15.78:0/2467479001 shutdown_connections 2024-03-14T09:56:48.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.585+0000 7f41fa49db80 1 --2- 172.21.15.78:0/2467479001 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5640ff8b6700 0x5640ff8b8b20 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.585+0000 7f41fa49db80 1 --2- 172.21.15.78:0/2467479001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5640ff84fec0 0x5640ff859270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.585+0000 7f41fa49db80 1 --2- 172.21.15.78:0/2467479001 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5640ff845680 0x5640ff85a8a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.585+0000 7f41fa49db80 1 --2- 172.21.15.78:0/2467479001 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5640ff845fa0 0x5640ff858d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.585+0000 7f41c9600700 0 client.26218 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:56:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.585+0000 7f41fa49db80 1 -- 172.21.15.78:0/2467479001 >> 172.21.15.78:0/2467479001 conn(0x5640ff7bb220 msgr2=0x5640ff844df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.585+0000 7f41fa49db80 1 -- 172.21.15.78:0/2467479001 shutdown_connections 2024-03-14T09:56:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.585+0000 7f41fa49db80 1 -- 172.21.15.78:0/2467479001 wait complete. 2024-03-14T09:56:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.585+0000 7f41fa49db80 20 client.26218 trim_cache size 0 max 16384 2024-03-14T09:56:48.589 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:56:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.586+0000 7f41e6a00700 1 -- 172.21.15.78:0/3572958015 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f41c40a0c80 msgr2=0x7f41c40a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.586+0000 7f41e6a00700 1 --2- 172.21.15.78:0/3572958015 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f41c40a0c80 0x7f41c40a3140 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f41e0198b60 tx=0x7f41cc009260 comp rx=0 tx=0).stop 2024-03-14T09:56:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.586+0000 7f41e6a00700 1 -- 172.21.15.78:0/3572958015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41e00680b0 msgr2=0x7f41e0197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.586+0000 7f41e6a00700 1 --2- 172.21.15.78:0/3572958015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41e00680b0 0x7f41e0197610 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f41d0005910 tx=0x7f41d0004730 comp rx=0 tx=0).stop 2024-03-14T09:56:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.586+0000 7f41e6a00700 1 -- 172.21.15.78:0/3572958015 shutdown_connections 2024-03-14T09:56:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.586+0000 7f41e6a00700 1 --2- 172.21.15.78:0/3572958015 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f41c40a0c80 0x7f41c40a3140 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.586+0000 7f41e6a00700 1 --2- 172.21.15.78:0/3572958015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41e019bef0 0x7f41e019c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.586+0000 7f41e6a00700 1 --2- 172.21.15.78:0/3572958015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f41e00680b0 0x7f41e0197610 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.586+0000 7f41e6a00700 1 --2- 172.21.15.78:0/3572958015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41e0106060 0x7f41e0197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.586+0000 7f41e6a00700 1 -- 172.21.15.78:0/3572958015 >> 172.21.15.78:0/3572958015 conn(0x7f41e0076d80 msgr2=0x7f41e00fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.586+0000 7f41e6a00700 1 -- 172.21.15.78:0/3572958015 shutdown_connections 2024-03-14T09:56:48.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.586+0000 7f41e6a00700 1 -- 172.21.15.78:0/3572958015 wait complete. 2024-03-14T09:56:48.607 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T09:56:48.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.840+0000 7f3d53600700 1 -- 172.21.15.78:0/1250977492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d4c106050 msgr2=0x7f3d4c10dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.840+0000 7f3d53600700 1 --2- 172.21.15.78:0/1250977492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d4c106050 0x7f3d4c10dbd0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3d44009f90 tx=0x7f3d4400b330 comp rx=0 tx=0).stop 2024-03-14T09:56:48.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.840+0000 7f3d53600700 1 -- 172.21.15.78:0/1250977492 shutdown_connections 2024-03-14T09:56:48.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.840+0000 7f3d53600700 1 --2- 172.21.15.78:0/1250977492 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d4c106050 0x7f3d4c10dbd0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.840+0000 7f3d53600700 1 --2- 172.21.15.78:0/1250977492 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d4c100f30 0x7f3d4c101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.840+0000 7f3d53600700 1 --2- 172.21.15.78:0/1250977492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d4c1018e0 0x7f3d4c105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.840+0000 7f3d53600700 1 -- 172.21.15.78:0/1250977492 >> 172.21.15.78:0/1250977492 conn(0x7f3d4c0fc7b0 msgr2=0x7f3d4c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.840+0000 7f3d53600700 1 -- 172.21.15.78:0/1250977492 shutdown_connections 2024-03-14T09:56:48.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.840+0000 7f3d53600700 1 -- 172.21.15.78:0/1250977492 wait complete. 2024-03-14T09:56:48.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.841+0000 7f3d53600700 1 Processor -- start 2024-03-14T09:56:48.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.841+0000 7f3d53600700 1 -- start start 2024-03-14T09:56:48.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d53600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d4c100f30 0x7f3d4c1954a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d53600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d4c106050 0x7f3d4c1959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d53600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d4c199d80 0x7f3d4c19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d53600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c19a800 con 0x7f3d4c199d80 2024-03-14T09:56:48.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d53600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c19a970 con 0x7f3d4c100f30 2024-03-14T09:56:48.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d53600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c19ab30 con 0x7f3d4c106050 2024-03-14T09:56:48.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d4be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d4c106050 0x7f3d4c1959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d50c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d4c100f30 0x7f3d4c1954a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d4be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d4c106050 0x7f3d4c1959e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:33974/0 (socket says 172.21.15.78:33974) 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d50c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d4c100f30 0x7f3d4c1954a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48326/0 (socket says 172.21.15.78:48326) 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d4be00700 1 -- 172.21.15.78:0/1184702168 learned_addr learned my addr 172.21.15.78:0/1184702168 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d51600700 1 --2- 172.21.15.78:0/1184702168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d4c199d80 0x7f3d4c19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d4be00700 1 -- 172.21.15.78:0/1184702168 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d4c100f30 msgr2=0x7f3d4c1954a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d4be00700 1 --2- 172.21.15.78:0/1184702168 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d4c100f30 0x7f3d4c1954a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d4be00700 1 -- 172.21.15.78:0/1184702168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d4c199d80 msgr2=0x7f3d4c19a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d4be00700 1 --2- 172.21.15.78:0/1184702168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d4c199d80 0x7f3d4c19a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d4be00700 1 -- 172.21.15.78:0/1184702168 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d44009c40 con 0x7f3d4c106050 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.842+0000 7f3d51600700 1 --2- 172.21.15.78:0/1184702168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d4c199d80 0x7f3d4c19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.843+0000 7f3d4be00700 1 --2- 172.21.15.78:0/1184702168 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d4c106050 0x7f3d4c1959e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3d3800e9f0 tx=0x7f3d3800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.843+0000 7f3d49600700 1 -- 172.21.15.78:0/1184702168 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d38009690 con 0x7f3d4c106050 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.843+0000 7f3d49600700 1 -- 172.21.15.78:0/1184702168 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d3800ce80 con 0x7f3d4c106050 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.843+0000 7f3d53600700 1 -- 172.21.15.78:0/1184702168 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d4c1a2090 con 0x7f3d4c106050 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.843+0000 7f3d49600700 1 -- 172.21.15.78:0/1184702168 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d38018680 con 0x7f3d4c106050 2024-03-14T09:56:48.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.843+0000 7f3d53600700 1 -- 172.21.15.78:0/1184702168 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4c1a2570 con 0x7f3d4c106050 2024-03-14T09:56:48.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.847+0000 7f3d49600700 1 -- 172.21.15.78:0/1184702168 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f3d38010430 con 0x7f3d4c106050 2024-03-14T09:56:48.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.848+0000 7f3d670f0b80 1 Processor -- start 2024-03-14T09:56:48.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.848+0000 7f3d670f0b80 1 -- start start 2024-03-14T09:56:48.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.848+0000 7f3d670f0b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309680 0x55627a309a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.848+0000 7f3d670f0b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 0x55627a313980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.848+0000 7f3d670f0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a313ec0 0x55627a3162b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.848+0000 7f3d670f0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55627a262990 con 0x55627a313ec0 2024-03-14T09:56:48.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.848+0000 7f3d670f0b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55627a260350 con 0x55627a309680 2024-03-14T09:56:48.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.848+0000 7f3d670f0b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55627a25bc40 con 0x55627a309fa0 2024-03-14T09:56:48.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.849+0000 7f3d50c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 0x55627a313980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.849+0000 7f3d51600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309680 0x55627a309a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.849+0000 7f3d51600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309680 0x55627a309a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48328/0 (socket says 172.21.15.78:48328) 2024-03-14T09:56:48.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.849+0000 7f3d50c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 0x55627a313980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:33978/0 (socket says 172.21.15.78:33978) 2024-03-14T09:56:48.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.849+0000 7f3d51600700 1 -- 172.21.15.78:0/2428617620 learned_addr learned my addr 172.21.15.78:0/2428617620 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:48.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.849+0000 7f3d49600700 1 --2- 172.21.15.78:0/1184702168 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f3d340a0c00 0x7f3d340a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.849+0000 7f3d4be00700 1 --2- 172.21.15.78:0/2428617620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a313ec0 0x55627a3162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.849+0000 7f3d49600700 1 -- 172.21.15.78:0/1184702168 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f3d38014070 con 0x7f3d4c106050 2024-03-14T09:56:48.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.849+0000 7f3d51600700 1 --2- 172.21.15.78:0/1184702168 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f3d340a0c00 0x7f3d340a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.849+0000 7f3d50c00700 1 -- 172.21.15.78:0/2428617620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309680 msgr2=0x55627a309a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.849+0000 7f3d50c00700 1 --2- 172.21.15.78:0/2428617620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309680 0x55627a309a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.849+0000 7f3d50c00700 1 -- 172.21.15.78:0/2428617620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a313ec0 msgr2=0x55627a3162b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.849+0000 7f3d50c00700 1 --2- 172.21.15.78:0/2428617620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a313ec0 0x55627a3162b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.849+0000 7f3d50c00700 1 -- 172.21.15.78:0/2428617620 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55627a0dd3b0 con 0x55627a309fa0 2024-03-14T09:56:48.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.849+0000 7f3d4be00700 1 --2- 172.21.15.78:0/2428617620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a313ec0 0x55627a3162b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:56:48.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.850+0000 7f3d50c00700 1 --2- 172.21.15.78:0/2428617620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 0x55627a313980 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000bc80 tx=0x7f3d4000bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f79112f94ca8f8b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.850+0000 7f3d3dc00700 1 -- 172.21.15.78:0/2428617620 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40010ca0 con 0x55627a309fa0 2024-03-14T09:56:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.850+0000 7f3d3dc00700 1 -- 172.21.15.78:0/2428617620 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d40015070 con 0x55627a309fa0 2024-03-14T09:56:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.850+0000 7f3d3dc00700 1 -- 172.21.15.78:0/2428617620 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40005020 con 0x55627a309fa0 2024-03-14T09:56:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.851+0000 7f3d670f0b80 1 -- 172.21.15.78:0/2428617620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 msgr2=0x55627a313980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.851+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/2428617620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 0x55627a313980 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000bc80 tx=0x7f3d4000bf90 comp rx=0 tx=0).stop 2024-03-14T09:56:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.851+0000 7f3d51600700 1 --2- 172.21.15.78:0/1184702168 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f3d340a0c00 0x7f3d340a30c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3d44009c10 tx=0x7f3d4400b330 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.851+0000 7f3d670f0b80 1 -- 172.21.15.78:0/2428617620 shutdown_connections 2024-03-14T09:56:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.851+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/2428617620 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309680 0x55627a309a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.851+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/2428617620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a313ec0 0x55627a3162b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.851+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/2428617620 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 0x55627a313980 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.851+0000 7f3d670f0b80 1 -- 172.21.15.78:0/2428617620 >> 172.21.15.78:0/2428617620 conn(0x55627a27f220 msgr2=0x55627a304aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.851+0000 7f3d670f0b80 1 -- 172.21.15.78:0/2428617620 shutdown_connections 2024-03-14T09:56:48.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.851+0000 7f3d670f0b80 1 -- 172.21.15.78:0/2428617620 wait complete. 2024-03-14T09:56:48.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.852+0000 7f3d670f0b80 1 -- start start 2024-03-14T09:56:48.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.852+0000 7f3d670f0b80 1 Processor -- start 2024-03-14T09:56:48.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d670f0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a309680 0x55627a313890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d670f0b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309fa0 0x55627a310c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d670f0b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a313ec0 0x55627a311290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d670f0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55627a1f7d80 con 0x55627a309680 2024-03-14T09:56:48.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d670f0b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55627a1f7a20 con 0x55627a309fa0 2024-03-14T09:56:48.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d670f0b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55627a0df6a0 con 0x55627a313ec0 2024-03-14T09:56:48.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d670f0b80 10 client.0 _ll_register_callbacks cb 0x55627a0633a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:48.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d50c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a309680 0x55627a313890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d50c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a309680 0x55627a313890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60256/0 (socket says 172.21.15.78:60256) 2024-03-14T09:56:48.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d50c00700 1 -- 172.21.15.78:0/4207774136 learned_addr learned my addr 172.21.15.78:0/4207774136 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:48.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d4be00700 1 --2- 172.21.15.78:0/4207774136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309fa0 0x55627a310c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d51600700 1 --2- 172.21.15.78:0/4207774136 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a313ec0 0x55627a311290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d50c00700 1 -- 172.21.15.78:0/4207774136 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a313ec0 msgr2=0x55627a311290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d50c00700 1 --2- 172.21.15.78:0/4207774136 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a313ec0 0x55627a311290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d50c00700 1 -- 172.21.15.78:0/4207774136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309fa0 msgr2=0x55627a310c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d50c00700 1 --2- 172.21.15.78:0/4207774136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309fa0 0x55627a310c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d50c00700 1 -- 172.21.15.78:0/4207774136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4000b8e0 con 0x55627a309680 2024-03-14T09:56:48.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.853+0000 7f3d51600700 1 --2- 172.21.15.78:0/4207774136 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a313ec0 0x55627a311290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:56:48.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.854+0000 7f3d50c00700 1 --2- 172.21.15.78:0/4207774136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a309680 0x55627a313890 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3d40010cd0 tx=0x7f3d40010d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb2fb4f80581eb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.854+0000 7f3d32000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:56:48.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.854+0000 7f3d32000700 1 -- 172.21.15.78:0/4207774136 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40020070 con 0x55627a309680 2024-03-14T09:56:48.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.854+0000 7f3d32000700 1 -- 172.21.15.78:0/4207774136 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f3d40017070 con 0x55627a309680 2024-03-14T09:56:48.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.854+0000 7f3d670f0b80 1 -- 172.21.15.78:0/4207774136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55627a2fc720 con 0x55627a309680 2024-03-14T09:56:48.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.854+0000 7f3d32000700 1 -- 172.21.15.78:0/4207774136 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d40004750 con 0x55627a309680 2024-03-14T09:56:48.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.854+0000 7f3d670f0b80 10 client.16194 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:48.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.854+0000 7f3d670f0b80 1 -- 172.21.15.78:0/4207774136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55627a089b60 con 0x55627a309680 2024-03-14T09:56:48.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.855+0000 7f3d32000700 1 -- 172.21.15.78:0/4207774136 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55627a2fc720 con 0x55627a309680 2024-03-14T09:56:48.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.855+0000 7f3d32000700 1 -- 172.21.15.78:0/4207774136 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3d40011440 con 0x55627a309680 2024-03-14T09:56:48.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.855+0000 7f3d670f0b80 10 client.16194 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:48.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.855+0000 7f3d670f0b80 10 client.16194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:56:48.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.855+0000 7f3d670f0b80 20 client.16194 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:48.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.855+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/4207774136 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55627a3238c0 0x55627a325ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.855+0000 7f3d670f0b80 4 client.16194 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:56:48.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.855+0000 7f3d670f0b80 1 -- 172.21.15.78:0/4207774136 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55627a254700 con 0x55627a3238c0 2024-03-14T09:56:48.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.855+0000 7f3d4be00700 1 --2- 172.21.15.78:0/4207774136 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55627a3238c0 0x55627a325ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.856+0000 7f3d4be00700 1 --2- 172.21.15.78:0/4207774136 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55627a3238c0 0x55627a325ce0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcc914f337dc1bd9 server_cookie=133c8f108cb721a7 in_seq=0 out_seq=0 2024-03-14T09:56:48.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.856+0000 7f3d32000700 10 client.16194 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:48.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.857+0000 7f3d32000700 1 -- 172.21.15.78:0/4207774136 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55627a254700 con 0x55627a3238c0 2024-03-14T09:56:48.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.857+0000 7f3d32000700 10 client.16194 handle_command_reply: tid=0 2024-03-14T09:56:48.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.857+0000 7f3d670f0b80 1 client.16194 shutdown 2024-03-14T09:56:48.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.857+0000 7f3d670f0b80 1 -- 172.21.15.78:0/4207774136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a309680 msgr2=0x55627a313890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.857+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/4207774136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a309680 0x55627a313890 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3d40010cd0 tx=0x7f3d40010d00 comp rx=0 tx=0).stop 2024-03-14T09:56:48.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.857+0000 7f3d670f0b80 1 -- 172.21.15.78:0/4207774136 shutdown_connections 2024-03-14T09:56:48.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.857+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/4207774136 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a313ec0 0x55627a311290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.857+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/4207774136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a309680 0x55627a313890 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.857+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/4207774136 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55627a3238c0 0x55627a325ce0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.857+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/4207774136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309fa0 0x55627a310c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.857+0000 7f3d670f0b80 1 -- 172.21.15.78:0/4207774136 >> 172.21.15.78:0/4207774136 conn(0x55627a27f220 msgr2=0x55627a304690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.857+0000 7f3d32000700 0 client.16194 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:48.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.857+0000 7f3d670f0b80 1 -- 172.21.15.78:0/4207774136 shutdown_connections 2024-03-14T09:56:48.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.857+0000 7f3d670f0b80 1 -- 172.21.15.78:0/4207774136 wait complete. 2024-03-14T09:56:48.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.858+0000 7f3d670f0b80 20 client.16194 trim_cache size 0 max 16384 2024-03-14T09:56:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.871+0000 7f3d670f0b80 1 Processor -- start 2024-03-14T09:56:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.871+0000 7f3d670f0b80 1 -- start start 2024-03-14T09:56:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.872+0000 7f3d670f0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a309680 0x55627a3162b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.872+0000 7f3d670f0b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 0x55627a31ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.872+0000 7f3d670f0b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a313ec0 0x55627a30f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.872+0000 7f3d670f0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55627a30b730 con 0x55627a309680 2024-03-14T09:56:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.872+0000 7f3d670f0b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55627a301260 con 0x55627a313ec0 2024-03-14T09:56:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.872+0000 7f3d670f0b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55627a311fd0 con 0x55627a309fa0 2024-03-14T09:56:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.872+0000 7f3d50c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a309680 0x55627a3162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.872+0000 7f3d50c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a309680 0x55627a3162b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60262/0 (socket says 172.21.15.78:60262) 2024-03-14T09:56:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.872+0000 7f3d50c00700 1 -- 172.21.15.78:0/3102145402 learned_addr learned my addr 172.21.15.78:0/3102145402 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:48.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.872+0000 7f3d51600700 1 --2- 172.21.15.78:0/3102145402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a313ec0 0x55627a30f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.872+0000 7f3d4be00700 1 --2- 172.21.15.78:0/3102145402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 0x55627a31ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.873+0000 7f3d50c00700 1 -- 172.21.15.78:0/3102145402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 msgr2=0x55627a31ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.873+0000 7f3d50c00700 1 --2- 172.21.15.78:0/3102145402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 0x55627a31ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.873+0000 7f3d50c00700 1 -- 172.21.15.78:0/3102145402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a313ec0 msgr2=0x55627a30f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.873+0000 7f3d50c00700 1 --2- 172.21.15.78:0/3102145402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a313ec0 0x55627a30f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.873+0000 7f3d50c00700 1 -- 172.21.15.78:0/3102145402 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55627a221ea0 con 0x55627a309680 2024-03-14T09:56:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.873+0000 7f3d51600700 1 --2- 172.21.15.78:0/3102145402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a313ec0 0x55627a30f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:56:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.873+0000 7f3d50c00700 1 --2- 172.21.15.78:0/3102145402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a309680 0x55627a3162b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000bbd0 tx=0x7f3d40004a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ba29ab2b264a35 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.873+0000 7f3d3e600700 1 -- 172.21.15.78:0/3102145402 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40020070 con 0x55627a309680 2024-03-14T09:56:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.873+0000 7f3d3e600700 1 -- 172.21.15.78:0/3102145402 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d40021040 con 0x55627a309680 2024-03-14T09:56:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.873+0000 7f3d3e600700 1 -- 172.21.15.78:0/3102145402 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40024e90 con 0x55627a309680 2024-03-14T09:56:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.874+0000 7f3d670f0b80 1 -- 172.21.15.78:0/3102145402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a309680 msgr2=0x55627a3162b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.874+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/3102145402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a309680 0x55627a3162b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000bbd0 tx=0x7f3d40004a70 comp rx=0 tx=0).stop 2024-03-14T09:56:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.874+0000 7f3d670f0b80 1 -- 172.21.15.78:0/3102145402 shutdown_connections 2024-03-14T09:56:48.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.874+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/3102145402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a313ec0 0x55627a30f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.874+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/3102145402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a309680 0x55627a3162b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.874+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/3102145402 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 0x55627a31ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.874+0000 7f3d670f0b80 1 -- 172.21.15.78:0/3102145402 >> 172.21.15.78:0/3102145402 conn(0x55627a27f220 msgr2=0x55627a30bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.874+0000 7f3d670f0b80 1 -- 172.21.15.78:0/3102145402 shutdown_connections 2024-03-14T09:56:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.874+0000 7f3d670f0b80 1 -- 172.21.15.78:0/3102145402 wait complete. 2024-03-14T09:56:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.875+0000 7f3d670f0b80 1 -- start start 2024-03-14T09:56:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.875+0000 7f3d670f0b80 1 Processor -- start 2024-03-14T09:56:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.875+0000 7f3d670f0b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309680 0x55627a318340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.875+0000 7f3d670f0b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 0x55627a318880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.875+0000 7f3d670f0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a313ec0 0x55627a3743a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.875+0000 7f3d670f0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55627a318ff0 con 0x55627a313ec0 2024-03-14T09:56:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.875+0000 7f3d670f0b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55627a319160 con 0x55627a309680 2024-03-14T09:56:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.875+0000 7f3d670f0b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55627a3748e0 con 0x55627a309fa0 2024-03-14T09:56:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.875+0000 7f3d670f0b80 10 client.0 _ll_register_callbacks cb 0x7f3d44009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:48.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d4be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 0x55627a318880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d50c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309680 0x55627a318340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d50c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309680 0x55627a318340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48358/0 (socket says 172.21.15.78:48358) 2024-03-14T09:56:48.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d4be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 0x55627a318880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:34004/0 (socket says 172.21.15.78:34004) 2024-03-14T09:56:48.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d50c00700 1 -- 172.21.15.78:0/1422476696 learned_addr learned my addr 172.21.15.78:0/1422476696 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:48.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d51600700 1 --2- 172.21.15.78:0/1422476696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a313ec0 0x55627a3743a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d4be00700 1 -- 172.21.15.78:0/1422476696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309680 msgr2=0x55627a318340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d4be00700 1 --2- 172.21.15.78:0/1422476696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309680 0x55627a318340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d4be00700 1 -- 172.21.15.78:0/1422476696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a313ec0 msgr2=0x55627a3743a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:48.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d4be00700 1 --2- 172.21.15.78:0/1422476696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a313ec0 0x55627a3743a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d4be00700 1 -- 172.21.15.78:0/1422476696 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55627a089b60 con 0x55627a309fa0 2024-03-14T09:56:48.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d50c00700 1 --2- 172.21.15.78:0/1422476696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309680 0x55627a318340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:56:48.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d4be00700 1 --2- 172.21.15.78:0/1422476696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 0x55627a318880 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3d380c9f10 tx=0x7f3d3806be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e5ef354fdf88f07 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:48.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d32000700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:56:48.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d32000700 1 -- 172.21.15.78:0/1422476696 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d38099b10 con 0x55627a309fa0 2024-03-14T09:56:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d32000700 1 -- 172.21.15.78:0/1422476696 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f3d3809d940 con 0x55627a309fa0 2024-03-14T09:56:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.876+0000 7f3d670f0b80 1 -- 172.21.15.78:0/1422476696 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d40015070 con 0x55627a309fa0 2024-03-14T09:56:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.877+0000 7f3d32000700 1 -- 172.21.15.78:0/1422476696 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d380a1f90 con 0x55627a309fa0 2024-03-14T09:56:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.877+0000 7f3d670f0b80 10 client.26233 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.877+0000 7f3d670f0b80 1 -- 172.21.15.78:0/1422476696 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55627a0e1cd0 con 0x55627a309fa0 2024-03-14T09:56:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.877+0000 7f3d32000700 1 -- 172.21.15.78:0/1422476696 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d40015070 con 0x55627a309fa0 2024-03-14T09:56:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.877+0000 7f3d32000700 1 -- 172.21.15.78:0/1422476696 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3d38099120 con 0x55627a309fa0 2024-03-14T09:56:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.877+0000 7f3d670f0b80 10 client.26233 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.877+0000 7f3d670f0b80 10 client.26233 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:56:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.877+0000 7f3d670f0b80 20 client.26233 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.877+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/1422476696 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55627a376150 0x55627a378570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.877+0000 7f3d670f0b80 4 client.26233 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:56:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.877+0000 7f3d670f0b80 1 -- 172.21.15.78:0/1422476696 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55627a0aacf0 con 0x55627a376150 2024-03-14T09:56:48.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.877+0000 7f3d50c00700 1 --2- 172.21.15.78:0/1422476696 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55627a376150 0x55627a378570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:48.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.878+0000 7f3d50c00700 1 --2- 172.21.15.78:0/1422476696 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55627a376150 0x55627a378570 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b60e5613f64f97c server_cookie=3f86726429674303 in_seq=0 out_seq=0 2024-03-14T09:56:48.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.878+0000 7f3d32000700 10 client.26233 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:48.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.878+0000 7f3d32000700 1 -- 172.21.15.78:0/1422476696 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55627a0aacf0 con 0x55627a376150 2024-03-14T09:56:48.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.878+0000 7f3d32000700 10 client.26233 handle_command_reply: tid=0 2024-03-14T09:56:48.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.878+0000 7f3d670f0b80 1 client.26233 shutdown 2024-03-14T09:56:48.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.878+0000 7f3d670f0b80 1 -- 172.21.15.78:0/1422476696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 msgr2=0x55627a318880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.878+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/1422476696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 0x55627a318880 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3d380c9f10 tx=0x7f3d3806be30 comp rx=0 tx=0).stop 2024-03-14T09:56:48.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.878+0000 7f3d670f0b80 1 -- 172.21.15.78:0/1422476696 shutdown_connections 2024-03-14T09:56:48.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.878+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/1422476696 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55627a376150 0x55627a378570 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.878+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/1422476696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55627a309680 0x55627a318340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.878+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/1422476696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55627a313ec0 0x55627a3743a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.878+0000 7f3d32000700 0 client.26233 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:48.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.878+0000 7f3d670f0b80 1 --2- 172.21.15.78:0/1422476696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55627a309fa0 0x55627a318880 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.878+0000 7f3d670f0b80 1 -- 172.21.15.78:0/1422476696 >> 172.21.15.78:0/1422476696 conn(0x55627a27f220 msgr2=0x55627a30f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.878+0000 7f3d670f0b80 1 -- 172.21.15.78:0/1422476696 shutdown_connections 2024-03-14T09:56:48.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.878+0000 7f3d670f0b80 1 -- 172.21.15.78:0/1422476696 wait complete. 2024-03-14T09:56:48.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.879+0000 7f3d670f0b80 20 client.26233 trim_cache size 0 max 16384 2024-03-14T09:56:48.886 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:56:48.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.879+0000 7f3d53600700 1 -- 172.21.15.78:0/1184702168 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f3d340a0c00 msgr2=0x7f3d340a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.879+0000 7f3d53600700 1 --2- 172.21.15.78:0/1184702168 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f3d340a0c00 0x7f3d340a30c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3d44009c10 tx=0x7f3d4400b330 comp rx=0 tx=0).stop 2024-03-14T09:56:48.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.879+0000 7f3d53600700 1 -- 172.21.15.78:0/1184702168 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d4c106050 msgr2=0x7f3d4c1959e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:48.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.879+0000 7f3d53600700 1 --2- 172.21.15.78:0/1184702168 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d4c106050 0x7f3d4c1959e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3d3800e9f0 tx=0x7f3d3800ed00 comp rx=0 tx=0).stop 2024-03-14T09:56:48.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.879+0000 7f3d53600700 1 -- 172.21.15.78:0/1184702168 shutdown_connections 2024-03-14T09:56:48.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.879+0000 7f3d53600700 1 --2- 172.21.15.78:0/1184702168 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f3d340a0c00 0x7f3d340a30c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.879+0000 7f3d53600700 1 --2- 172.21.15.78:0/1184702168 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d4c100f30 0x7f3d4c1954a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.879+0000 7f3d53600700 1 --2- 172.21.15.78:0/1184702168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d4c199d80 0x7f3d4c19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.879+0000 7f3d53600700 1 --2- 172.21.15.78:0/1184702168 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3d4c106050 0x7f3d4c1959e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:48.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.879+0000 7f3d53600700 1 -- 172.21.15.78:0/1184702168 >> 172.21.15.78:0/1184702168 conn(0x7f3d4c0fc7b0 msgr2=0x7f3d4c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:48.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.880+0000 7f3d53600700 1 -- 172.21.15.78:0/1184702168 shutdown_connections 2024-03-14T09:56:48.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:48.880+0000 7f3d53600700 1 -- 172.21.15.78:0/1184702168 wait complete. 2024-03-14T09:56:48.898 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:48 smithi078 ceph-mon[38143]: pgmap v82: 129 pgs: 129 active+clean; 101 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 321 op/s 2024-03-14T09:56:48.898 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:48 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3198893454' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:56:49.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:48 smithi152 ceph-mon[48279]: pgmap v82: 129 pgs: 129 active+clean; 101 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 321 op/s 2024-03-14T09:56:49.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:48 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3198893454' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:56:49.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:48 smithi132 ceph-mon[47959]: pgmap v82: 129 pgs: 129 active+clean; 101 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 321 op/s 2024-03-14T09:56:49.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:48 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3198893454' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:56:49.901 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T09:56:49.902 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T09:56:50.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.130+0000 7f1c92c00700 1 -- 172.21.15.78:0/97462778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c8c103140 msgr2=0x7f1c8c103520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.130+0000 7f1c92c00700 1 --2- 172.21.15.78:0/97462778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c8c103140 0x7f1c8c103520 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c009990 tx=0x7f1c7c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.130+0000 7f1c92c00700 1 -- 172.21.15.78:0/97462778 shutdown_connections 2024-03-14T09:56:50.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.130+0000 7f1c92c00700 1 --2- 172.21.15.78:0/97462778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c8c103140 0x7f1c8c103520 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.130+0000 7f1c92c00700 1 --2- 172.21.15.78:0/97462778 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c8c108260 0x7f1c8c10fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.130+0000 7f1c92c00700 1 --2- 172.21.15.78:0/97462778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c8c103af0 0x7f1c8c107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.130+0000 7f1c92c00700 1 -- 172.21.15.78:0/97462778 >> 172.21.15.78:0/97462778 conn(0x7f1c8c0fe980 msgr2=0x7f1c8c100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:50.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.130+0000 7f1c92c00700 1 -- 172.21.15.78:0/97462778 shutdown_connections 2024-03-14T09:56:50.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.130+0000 7f1c92c00700 1 -- 172.21.15.78:0/97462778 wait complete. 2024-03-14T09:56:50.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.132+0000 7f1c92c00700 1 Processor -- start 2024-03-14T09:56:50.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.132+0000 7f1c92c00700 1 -- start start 2024-03-14T09:56:50.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c92c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c8c103140 0x7f1c8c19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c92c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c8c108260 0x7f1c8c19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c92c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c8c199650 0x7f1c8c199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c92c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c19a0d0 con 0x7f1c8c199650 2024-03-14T09:56:50.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c92c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c19a240 con 0x7f1c8c103140 2024-03-14T09:56:50.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c92c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c19a400 con 0x7f1c8c108260 2024-03-14T09:56:50.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c8b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c8c108260 0x7f1c8c19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c8be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c8c103140 0x7f1c8c19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c8be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c8c103140 0x7f1c8c19f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48374/0 (socket says 172.21.15.78:48374) 2024-03-14T09:56:50.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c8be00700 1 -- 172.21.15.78:0/2184222104 learned_addr learned my addr 172.21.15.78:0/2184222104 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:50.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c8b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c8c108260 0x7f1c8c19fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:34024/0 (socket says 172.21.15.78:34024) 2024-03-14T09:56:50.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c90c00700 1 --2- 172.21.15.78:0/2184222104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c8c199650 0x7f1c8c199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c8be00700 1 -- 172.21.15.78:0/2184222104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c8c108260 msgr2=0x7f1c8c19fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c8be00700 1 --2- 172.21.15.78:0/2184222104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c8c108260 0x7f1c8c19fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c8be00700 1 -- 172.21.15.78:0/2184222104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c8c199650 msgr2=0x7f1c8c199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c8be00700 1 --2- 172.21.15.78:0/2184222104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c8c199650 0x7f1c8c199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c8be00700 1 -- 172.21.15.78:0/2184222104 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c78009520 con 0x7f1c8c103140 2024-03-14T09:56:50.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.133+0000 7f1c8b400700 1 --2- 172.21.15.78:0/2184222104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c8c108260 0x7f1c8c19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:56:50.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.134+0000 7f1c8be00700 1 --2- 172.21.15.78:0/2184222104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c8c103140 0x7f1c8c19f520 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c000c00 tx=0x7f1c7c005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:50.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.134+0000 7f1c88c00700 1 -- 172.21.15.78:0/2184222104 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c01d070 con 0x7f1c8c103140 2024-03-14T09:56:50.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.134+0000 7f1c88c00700 1 -- 172.21.15.78:0/2184222104 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c7c022470 con 0x7f1c8c103140 2024-03-14T09:56:50.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.134+0000 7f1c92c00700 1 -- 172.21.15.78:0/2184222104 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7c0095f0 con 0x7f1c8c103140 2024-03-14T09:56:50.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.134+0000 7f1c88c00700 1 -- 172.21.15.78:0/2184222104 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c017650 con 0x7f1c8c103140 2024-03-14T09:56:50.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.134+0000 7f1c92c00700 1 -- 172.21.15.78:0/2184222104 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c8c1a6680 con 0x7f1c8c103140 2024-03-14T09:56:50.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.137+0000 7f1c88c00700 1 -- 172.21.15.78:0/2184222104 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f1c7c0225e0 con 0x7f1c8c103140 2024-03-14T09:56:50.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.138+0000 7f1ca6737b80 1 Processor -- start 2024-03-14T09:56:50.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.139+0000 7f1ca6737b80 1 -- start start 2024-03-14T09:56:50.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.139+0000 7f1c88c00700 1 --2- 172.21.15.78:0/2184222104 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1c700a0c00 0x7f1c700a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.139+0000 7f1c88c00700 1 -- 172.21.15.78:0/2184222104 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f1c7c0cd9a0 con 0x7f1c8c103140 2024-03-14T09:56:50.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.139+0000 7f1ca6737b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1177700 0x55beb1177ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.139+0000 7f1ca6737b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1178020 0x55beb1181a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.139+0000 7f1ca6737b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1181f40 0x55beb1184330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.139+0000 7f1ca6737b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55beb10d0a10 con 0x55beb1178020 2024-03-14T09:56:50.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.139+0000 7f1ca6737b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55beb10ce3d0 con 0x55beb1181f40 2024-03-14T09:56:50.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.139+0000 7f1ca6737b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55beb10c9cc0 con 0x55beb1177700 2024-03-14T09:56:50.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.140+0000 7f1c8be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1177700 0x55beb1177ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.140+0000 7f1c8be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1177700 0x55beb1177ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:34028/0 (socket says 172.21.15.78:34028) 2024-03-14T09:56:50.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.140+0000 7f1c8be00700 1 -- 172.21.15.78:0/3750127078 learned_addr learned my addr 172.21.15.78:0/3750127078 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:50.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.140+0000 7f1c90c00700 1 --2- 172.21.15.78:0/3750127078 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1181f40 0x55beb1184330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.140+0000 7f1c90c00700 1 --2- 172.21.15.78:0/2184222104 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1c700a0c00 0x7f1c700a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.140+0000 7f1c8be00700 1 -- 172.21.15.78:0/3750127078 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1181f40 msgr2=0x55beb1184330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:50.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.140+0000 7f1c8be00700 1 --2- 172.21.15.78:0/3750127078 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1181f40 0x55beb1184330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.140+0000 7f1c8be00700 1 -- 172.21.15.78:0/3750127078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1178020 msgr2=0x55beb1181a00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-14T09:56:50.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.140+0000 7f1c8be00700 1 --2- 172.21.15.78:0/3750127078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1178020 0x55beb1181a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.140+0000 7f1c8be00700 1 -- 172.21.15.78:0/3750127078 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55beb0f4b430 con 0x55beb1177700 2024-03-14T09:56:50.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.140+0000 7f1c8be00700 1 --2- 172.21.15.78:0/3750127078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1177700 0x55beb1177ae0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c096f10 tx=0x7f1c7c096e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd5197d57a4af399 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:50.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.141+0000 7f1c81200700 1 -- 172.21.15.78:0/3750127078 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c09b110 con 0x55beb1177700 2024-03-14T09:56:50.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.141+0000 7f1c81200700 1 -- 172.21.15.78:0/3750127078 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c7c09aa10 con 0x55beb1177700 2024-03-14T09:56:50.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.141+0000 7f1c90c00700 1 --2- 172.21.15.78:0/2184222104 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1c700a0c00 0x7f1c700a30c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009ec0 tx=0x7f1c8400b040 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:50.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.141+0000 7f1c81200700 1 -- 172.21.15.78:0/3750127078 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c09aeb0 con 0x55beb1177700 2024-03-14T09:56:50.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.142+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3750127078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1177700 msgr2=0x55beb1177ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.142+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/3750127078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1177700 0x55beb1177ae0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c096f10 tx=0x7f1c7c096e20 comp rx=0 tx=0).stop 2024-03-14T09:56:50.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.142+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3750127078 shutdown_connections 2024-03-14T09:56:50.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.142+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/3750127078 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1181f40 0x55beb1184330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.142+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/3750127078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1177700 0x55beb1177ae0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.142+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/3750127078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1178020 0x55beb1181a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.142+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3750127078 >> 172.21.15.78:0/3750127078 conn(0x55beb10ed2a0 msgr2=0x55beb1172b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:50.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.142+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3750127078 shutdown_connections 2024-03-14T09:56:50.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.142+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3750127078 wait complete. 2024-03-14T09:56:50.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.142+0000 7f1ca6737b80 1 -- start start 2024-03-14T09:56:50.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.143+0000 7f1ca6737b80 1 Processor -- start 2024-03-14T09:56:50.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.143+0000 7f1ca6737b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1177700 0x55beb1181950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.143+0000 7f1ca6737b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1178020 0x55beb117ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.143+0000 7f1ca6737b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1181f40 0x55beb117f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.143+0000 7f1ca6737b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55beb1065e00 con 0x55beb1181f40 2024-03-14T09:56:50.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.143+0000 7f1ca6737b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55beb1065aa0 con 0x55beb1177700 2024-03-14T09:56:50.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.143+0000 7f1ca6737b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55beb0f4d720 con 0x55beb1178020 2024-03-14T09:56:50.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.143+0000 7f1ca6737b80 10 client.0 _ll_register_callbacks cb 0x55beb0ed1420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:50.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.143+0000 7f1c8be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1177700 0x55beb1181950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.144+0000 7f1c8be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1177700 0x55beb1181950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48400/0 (socket says 172.21.15.78:48400) 2024-03-14T09:56:50.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.144+0000 7f1c8be00700 1 -- 172.21.15.78:0/3673117608 learned_addr learned my addr 172.21.15.78:0/3673117608 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:50.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.144+0000 7f1c8b400700 1 --2- 172.21.15.78:0/3673117608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1178020 0x55beb117ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.144+0000 7f1c90c00700 1 --2- 172.21.15.78:0/3673117608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1181f40 0x55beb117f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.145+0000 7f1c8be00700 1 -- 172.21.15.78:0/3673117608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1178020 msgr2=0x55beb117ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:50.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.145+0000 7f1c8be00700 1 --2- 172.21.15.78:0/3673117608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1178020 0x55beb117ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.145+0000 7f1c8be00700 1 -- 172.21.15.78:0/3673117608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1181f40 msgr2=0x55beb117f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:50.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.145+0000 7f1c8be00700 1 --2- 172.21.15.78:0/3673117608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1181f40 0x55beb117f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.145+0000 7f1c8be00700 1 -- 172.21.15.78:0/3673117608 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c8c1a6680 con 0x55beb1177700 2024-03-14T09:56:50.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.145+0000 7f1c8be00700 1 --2- 172.21.15.78:0/3673117608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1177700 0x55beb1181950 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c09a710 tx=0x7f1c7c009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e59ad5869a094a08 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:50.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.145+0000 7f1c75600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:56:50.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.145+0000 7f1c75600700 1 -- 172.21.15.78:0/3673117608 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c0b32d0 con 0x55beb1177700 2024-03-14T09:56:50.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.145+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3673117608 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55beb116a7a0 con 0x55beb1177700 2024-03-14T09:56:50.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.145+0000 7f1c75600700 1 -- 172.21.15.78:0/3673117608 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f1c7c09b110 con 0x55beb1177700 2024-03-14T09:56:50.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.145+0000 7f1c75600700 1 -- 172.21.15.78:0/3673117608 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c7c0a2e00 con 0x55beb1177700 2024-03-14T09:56:50.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.145+0000 7f1ca6737b80 10 client.26231 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:50.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.145+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3673117608 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55beb0ef7be0 con 0x55beb1177700 2024-03-14T09:56:50.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.145+0000 7f1c75600700 1 -- 172.21.15.78:0/3673117608 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55beb116a7a0 con 0x55beb1177700 2024-03-14T09:56:50.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.146+0000 7f1c75600700 1 -- 172.21.15.78:0/3673117608 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1c7c096800 con 0x55beb1177700 2024-03-14T09:56:50.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.146+0000 7f1ca6737b80 10 client.26231 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:50.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.146+0000 7f1ca6737b80 10 client.26231 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:56:50.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.146+0000 7f1ca6737b80 20 client.26231 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:50.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.146+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/3673117608 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55beb11913c0 0x55beb11937e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.146+0000 7f1ca6737b80 4 client.26231 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:56:50.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.146+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3673117608 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55beb10c2780 con 0x55beb11913c0 2024-03-14T09:56:50.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.146+0000 7f1c8b400700 1 --2- 172.21.15.78:0/3673117608 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55beb11913c0 0x55beb11937e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.147+0000 7f1c8b400700 1 --2- 172.21.15.78:0/3673117608 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55beb11913c0 0x55beb11937e0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca070a27003e43fe server_cookie=3461a7d49d1cf9f4 in_seq=0 out_seq=0 2024-03-14T09:56:50.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.147+0000 7f1c75600700 10 client.26231 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:50.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.147+0000 7f1c75600700 1 -- 172.21.15.78:0/3673117608 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55beb10c2780 con 0x55beb11913c0 2024-03-14T09:56:50.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.147+0000 7f1c75600700 10 client.26231 handle_command_reply: tid=0 2024-03-14T09:56:50.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.147+0000 7f1ca6737b80 1 client.26231 shutdown 2024-03-14T09:56:50.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.147+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3673117608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1177700 msgr2=0x55beb1181950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.148+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/3673117608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1177700 0x55beb1181950 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c09a710 tx=0x7f1c7c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.148+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3673117608 shutdown_connections 2024-03-14T09:56:50.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.148+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/3673117608 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55beb11913c0 0x55beb11937e0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.148+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/3673117608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1177700 0x55beb1181950 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.148+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/3673117608 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1181f40 0x55beb117f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.148+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/3673117608 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1178020 0x55beb117ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.148+0000 7f1c75600700 0 client.26231 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:50.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.148+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3673117608 >> 172.21.15.78:0/3673117608 conn(0x55beb10ed2a0 msgr2=0x55beb1172710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:50.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.148+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3673117608 shutdown_connections 2024-03-14T09:56:50.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.148+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3673117608 wait complete. 2024-03-14T09:56:50.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.148+0000 7f1ca6737b80 20 client.26231 trim_cache size 0 max 16384 2024-03-14T09:56:50.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.174+0000 7f1ca6737b80 1 Processor -- start 2024-03-14T09:56:50.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.174+0000 7f1ca6737b80 1 -- start start 2024-03-14T09:56:50.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1ca6737b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1177700 0x55beb1186350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1ca6737b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1178020 0x55beb117ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1ca6737b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1181f40 0x55beb1176cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1ca6737b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55beb1179780 con 0x55beb1181f40 2024-03-14T09:56:50.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1ca6737b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55beb116f2b0 con 0x55beb1178020 2024-03-14T09:56:50.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1ca6737b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55beb1179140 con 0x55beb1177700 2024-03-14T09:56:50.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1c8be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1177700 0x55beb1186350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1c8be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1177700 0x55beb1186350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:34040/0 (socket says 172.21.15.78:34040) 2024-03-14T09:56:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1c8be00700 1 -- 172.21.15.78:0/1811865257 learned_addr learned my addr 172.21.15.78:0/1811865257 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1c8b400700 1 --2- 172.21.15.78:0/1811865257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1178020 0x55beb117ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1c90c00700 1 --2- 172.21.15.78:0/1811865257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1181f40 0x55beb1176cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1c8be00700 1 -- 172.21.15.78:0/1811865257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1178020 msgr2=0x55beb117ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1c8be00700 1 --2- 172.21.15.78:0/1811865257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1178020 0x55beb117ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1c8be00700 1 -- 172.21.15.78:0/1811865257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1181f40 msgr2=0x55beb1176cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1c8be00700 1 --2- 172.21.15.78:0/1811865257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1181f40 0x55beb1176cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1c8be00700 1 -- 172.21.15.78:0/1811865257 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55beb117e7d0 con 0x55beb1177700 2024-03-14T09:56:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.175+0000 7f1c90c00700 1 --2- 172.21.15.78:0/1811865257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1181f40 0x55beb1176cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:56:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.176+0000 7f1c8be00700 1 --2- 172.21.15.78:0/1811865257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1177700 0x55beb1186350 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00f740 tx=0x7f1c7c0cb300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e5e37c7bdc941ae server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.176+0000 7f1c81c00700 1 -- 172.21.15.78:0/1811865257 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c0a2e00 con 0x55beb1177700 2024-03-14T09:56:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.176+0000 7f1c81c00700 1 -- 172.21.15.78:0/1811865257 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c7c0a2bb0 con 0x55beb1177700 2024-03-14T09:56:50.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.176+0000 7f1c81c00700 1 -- 172.21.15.78:0/1811865257 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c0b3330 con 0x55beb1177700 2024-03-14T09:56:50.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.177+0000 7f1ca6737b80 1 -- 172.21.15.78:0/1811865257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1177700 msgr2=0x55beb1186350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.177+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/1811865257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1177700 0x55beb1186350 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00f740 tx=0x7f1c7c0cb300 comp rx=0 tx=0).stop 2024-03-14T09:56:50.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.177+0000 7f1ca6737b80 1 -- 172.21.15.78:0/1811865257 shutdown_connections 2024-03-14T09:56:50.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.177+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/1811865257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1181f40 0x55beb1176cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.177+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/1811865257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1177700 0x55beb1186350 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.177+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/1811865257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1178020 0x55beb117ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.177+0000 7f1ca6737b80 1 -- 172.21.15.78:0/1811865257 >> 172.21.15.78:0/1811865257 conn(0x55beb10ed2a0 msgr2=0x55beb11765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:50.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.177+0000 7f1ca6737b80 1 -- 172.21.15.78:0/1811865257 shutdown_connections 2024-03-14T09:56:50.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.177+0000 7f1ca6737b80 1 -- 172.21.15.78:0/1811865257 wait complete. 2024-03-14T09:56:50.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.177+0000 7f1ca6737b80 1 -- start start 2024-03-14T09:56:50.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.178+0000 7f1ca6737b80 1 Processor -- start 2024-03-14T09:56:50.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.178+0000 7f1ca6737b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1177700 0x55beb11860f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.178+0000 7f1ca6737b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1178020 0x55beb1184680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.178+0000 7f1ca6737b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1181f40 0x55beb1184bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.178+0000 7f1ca6737b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55beb11851e0 con 0x55beb1178020 2024-03-14T09:56:50.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.178+0000 7f1ca6737b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55beb1185320 con 0x55beb1177700 2024-03-14T09:56:50.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.179+0000 7f1ca6737b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55beb11854e0 con 0x55beb1181f40 2024-03-14T09:56:50.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.179+0000 7f1c8b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1178020 0x55beb1184680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.179+0000 7f1c8b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1178020 0x55beb1184680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60320/0 (socket says 172.21.15.78:60320) 2024-03-14T09:56:50.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.179+0000 7f1c8b400700 1 -- 172.21.15.78:0/3245803580 learned_addr learned my addr 172.21.15.78:0/3245803580 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:50.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.179+0000 7f1ca6737b80 10 client.0 _ll_register_callbacks cb 0x7f1c7c0057d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:50.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.179+0000 7f1c8b400700 1 -- 172.21.15.78:0/3245803580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1181f40 msgr2=0x55beb1184bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:50.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.179+0000 7f1c8b400700 1 --2- 172.21.15.78:0/3245803580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1181f40 0x55beb1184bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.179+0000 7f1c8b400700 1 -- 172.21.15.78:0/3245803580 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1177700 msgr2=0x55beb11860f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:50.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.179+0000 7f1c8b400700 1 --2- 172.21.15.78:0/3245803580 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1177700 0x55beb11860f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.179+0000 7f1c8b400700 1 -- 172.21.15.78:0/3245803580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55beb0ef7be0 con 0x55beb1178020 2024-03-14T09:56:50.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.180+0000 7f1c8b400700 1 --2- 172.21.15.78:0/3245803580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1178020 0x55beb1184680 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1c780092a0 tx=0x7f1c78004460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b64c4b1bbf23cc server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:50.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.180+0000 7f1c75600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:56:50.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.180+0000 7f1c75600700 1 -- 172.21.15.78:0/3245803580 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c78017040 con 0x55beb1178020 2024-03-14T09:56:50.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.180+0000 7f1c75600700 1 -- 172.21.15.78:0/3245803580 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f1c78012d90 con 0x55beb1178020 2024-03-14T09:56:50.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.180+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3245803580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1c7c09aa10 con 0x55beb1178020 2024-03-14T09:56:50.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.181+0000 7f1c75600700 1 -- 172.21.15.78:0/3245803580 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c7801c780 con 0x55beb1178020 2024-03-14T09:56:50.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.181+0000 7f1ca6737b80 10 client.16212 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:50.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.181+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3245803580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55beb108ff20 con 0x55beb1178020 2024-03-14T09:56:50.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.181+0000 7f1c75600700 1 -- 172.21.15.78:0/3245803580 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c7c09aa10 con 0x55beb1178020 2024-03-14T09:56:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.181+0000 7f1c75600700 1 -- 172.21.15.78:0/3245803580 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1c78003a90 con 0x55beb1178020 2024-03-14T09:56:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.182+0000 7f1ca6737b80 10 client.16212 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.182+0000 7f1ca6737b80 10 client.16212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:56:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.182+0000 7f1ca6737b80 20 client.16212 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.182+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/3245803580 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55beb11da4b0 0x55beb11dc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.182+0000 7f1ca6737b80 4 client.16212 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T09:56:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.182+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3245803580 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55beb0f18d70 con 0x55beb11da4b0 2024-03-14T09:56:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.182+0000 7f1c8be00700 1 --2- 172.21.15.78:0/3245803580 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55beb11da4b0 0x55beb11dc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.183+0000 7f1c8be00700 1 --2- 172.21.15.78:0/3245803580 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55beb11da4b0 0x55beb11dc8d0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc96d7d0f40dc82c server_cookie=b4bd9e2e46c44b87 in_seq=0 out_seq=0 2024-03-14T09:56:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.183+0000 7f1c75600700 10 client.16212 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.184+0000 7f1c75600700 1 -- 172.21.15.78:0/3245803580 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55beb0f18d70 con 0x55beb11da4b0 2024-03-14T09:56:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.184+0000 7f1c75600700 10 client.16212 handle_command_reply: tid=0 2024-03-14T09:56:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.184+0000 7f1ca6737b80 1 client.16212 shutdown 2024-03-14T09:56:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.184+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3245803580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1178020 msgr2=0x55beb1184680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.184+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/3245803580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1178020 0x55beb1184680 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1c780092a0 tx=0x7f1c78004460 comp rx=0 tx=0).stop 2024-03-14T09:56:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.184+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3245803580 shutdown_connections 2024-03-14T09:56:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.184+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/3245803580 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55beb11da4b0 0x55beb11dc8d0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.184+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/3245803580 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55beb1177700 0x55beb11860f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.184+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/3245803580 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55beb1181f40 0x55beb1184bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.184+0000 7f1ca6737b80 1 --2- 172.21.15.78:0/3245803580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55beb1178020 0x55beb1184680 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.184+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3245803580 >> 172.21.15.78:0/3245803580 conn(0x55beb10ed2a0 msgr2=0x55beb117dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.184+0000 7f1c75600700 0 client.16212 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.185+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3245803580 shutdown_connections 2024-03-14T09:56:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.185+0000 7f1ca6737b80 1 -- 172.21.15.78:0/3245803580 wait complete. 2024-03-14T09:56:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.185+0000 7f1ca6737b80 20 client.16212 trim_cache size 0 max 16384 2024-03-14T09:56:50.187 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:56:50.187 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T09:56:50.187 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "0163b6e2-bc79-495d-9276-18e6498a59e2", 2024-03-14T09:56:50.187 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T09:56:50.187 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:56:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.186+0000 7f1c92c00700 1 -- 172.21.15.78:0/2184222104 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1c700a0c00 msgr2=0x7f1c700a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.186+0000 7f1c92c00700 1 --2- 172.21.15.78:0/2184222104 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1c700a0c00 0x7f1c700a30c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009ec0 tx=0x7f1c8400b040 comp rx=0 tx=0).stop 2024-03-14T09:56:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.186+0000 7f1c92c00700 1 -- 172.21.15.78:0/2184222104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c8c103140 msgr2=0x7f1c8c19f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.186+0000 7f1c92c00700 1 --2- 172.21.15.78:0/2184222104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c8c103140 0x7f1c8c19f520 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c000c00 tx=0x7f1c7c005740 comp rx=0 tx=0).stop 2024-03-14T09:56:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.186+0000 7f1c92c00700 1 -- 172.21.15.78:0/2184222104 shutdown_connections 2024-03-14T09:56:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.186+0000 7f1c92c00700 1 --2- 172.21.15.78:0/2184222104 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1c700a0c00 0x7f1c700a30c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.186+0000 7f1c92c00700 1 --2- 172.21.15.78:0/2184222104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c8c103140 0x7f1c8c19f520 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.186+0000 7f1c92c00700 1 --2- 172.21.15.78:0/2184222104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c8c199650 0x7f1c8c199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.186+0000 7f1c92c00700 1 --2- 172.21.15.78:0/2184222104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1c8c108260 0x7f1c8c19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.186+0000 7f1c92c00700 1 -- 172.21.15.78:0/2184222104 >> 172.21.15.78:0/2184222104 conn(0x7f1c8c0fe980 msgr2=0x7f1c8c0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.186+0000 7f1c92c00700 1 -- 172.21.15.78:0/2184222104 shutdown_connections 2024-03-14T09:56:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.186+0000 7f1c92c00700 1 -- 172.21.15.78:0/2184222104 wait complete. 2024-03-14T09:56:50.208 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:56:50.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.458+0000 7f4691a00700 1 -- 172.21.15.78:0/1558451273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f468c106050 msgr2=0x7f468c10dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.458+0000 7f4691a00700 1 --2- 172.21.15.78:0/1558451273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f468c106050 0x7f468c10dbd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f467c0098f0 tx=0x7f467c009c00 comp rx=0 tx=0).stop 2024-03-14T09:56:50.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.459+0000 7f4691a00700 1 -- 172.21.15.78:0/1558451273 shutdown_connections 2024-03-14T09:56:50.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.459+0000 7f4691a00700 1 --2- 172.21.15.78:0/1558451273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f468c106050 0x7f468c10dbd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.459+0000 7f4691a00700 1 --2- 172.21.15.78:0/1558451273 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f468c100f30 0x7f468c101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.459+0000 7f4691a00700 1 --2- 172.21.15.78:0/1558451273 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f468c1018e0 0x7f468c105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.459+0000 7f4691a00700 1 -- 172.21.15.78:0/1558451273 >> 172.21.15.78:0/1558451273 conn(0x7f468c0fc7b0 msgr2=0x7f468c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:50.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.459+0000 7f4691a00700 1 -- 172.21.15.78:0/1558451273 shutdown_connections 2024-03-14T09:56:50.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.459+0000 7f4691a00700 1 -- 172.21.15.78:0/1558451273 wait complete. 2024-03-14T09:56:50.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.461+0000 7f4691a00700 1 Processor -- start 2024-03-14T09:56:50.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.461+0000 7f4691a00700 1 -- start start 2024-03-14T09:56:50.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.462+0000 7f4691a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f468c100f30 0x7f468c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.462+0000 7f4691a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f468c106050 0x7f468c0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.462+0000 7f4691a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f468c073b30 0x7f468c1a5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.462+0000 7f4691a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c1a5da0 con 0x7f468c073b30 2024-03-14T09:56:50.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.462+0000 7f4691a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c1a5f10 con 0x7f468c100f30 2024-03-14T09:56:50.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.462+0000 7f4691a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c1a60d0 con 0x7f468c106050 2024-03-14T09:56:50.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.463+0000 7f468aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f468c100f30 0x7f468c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.463+0000 7f468aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f468c100f30 0x7f468c0730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48434/0 (socket says 172.21.15.78:48434) 2024-03-14T09:56:50.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.463+0000 7f468aa00700 1 -- 172.21.15.78:0/1464476857 learned_addr learned my addr 172.21.15.78:0/1464476857 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:50.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.463+0000 7f468aa00700 1 -- 172.21.15.78:0/1464476857 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f468c106050 msgr2=0x7f468c0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.463+0000 7f468a000700 1 --2- 172.21.15.78:0/1464476857 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f468c106050 0x7f468c0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.463+0000 7f468aa00700 1 --2- 172.21.15.78:0/1464476857 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f468c106050 0x7f468c0735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.463+0000 7f468aa00700 1 -- 172.21.15.78:0/1464476857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f468c073b30 msgr2=0x7f468c1a5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.463+0000 7f468aa00700 1 --2- 172.21.15.78:0/1464476857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f468c073b30 0x7f468c1a5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.463+0000 7f468aa00700 1 -- 172.21.15.78:0/1464476857 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4678009520 con 0x7f468c100f30 2024-03-14T09:56:50.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.463+0000 7f468a000700 1 --2- 172.21.15.78:0/1464476857 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f468c106050 0x7f468c0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:56:50.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.464+0000 7f468aa00700 1 --2- 172.21.15.78:0/1464476857 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f468c100f30 0x7f468c0730b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f468400c0c0 tx=0x7f468400c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:50.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.464+0000 7f4683400700 1 -- 172.21.15.78:0/1464476857 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684018070 con 0x7f468c100f30 2024-03-14T09:56:50.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.464+0000 7f4683400700 1 -- 172.21.15.78:0/1464476857 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f468400f040 con 0x7f468c100f30 2024-03-14T09:56:50.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.464+0000 7f4683400700 1 -- 172.21.15.78:0/1464476857 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468400e020 con 0x7f468c100f30 2024-03-14T09:56:50.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.464+0000 7f4691a00700 1 -- 172.21.15.78:0/1464476857 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467c009520 con 0x7f468c100f30 2024-03-14T09:56:50.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.464+0000 7f4691a00700 1 -- 172.21.15.78:0/1464476857 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468c197610 con 0x7f468c100f30 2024-03-14T09:56:50.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.468+0000 7f4683400700 1 -- 172.21.15.78:0/1464476857 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f46840075c0 con 0x7f468c100f30 2024-03-14T09:56:50.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.470+0000 7f46a547fb80 1 Processor -- start 2024-03-14T09:56:50.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.470+0000 7f4683400700 1 --2- 172.21.15.78:0/1464476857 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f46700a0c80 0x7f46700a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.470+0000 7f4683400700 1 -- 172.21.15.78:0/1464476857 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f46840cc600 con 0x7f468c100f30 2024-03-14T09:56:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.470+0000 7f46a547fb80 1 -- start start 2024-03-14T09:56:50.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.471+0000 7f46a547fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395680 0x5564f1395a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.471+0000 7f46a547fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395fa0 0x5564f139f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.471+0000 7f46a547fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f139fec0 0x5564f13a22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.471+0000 7f46a547fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564f12ee990 con 0x5564f139fec0 2024-03-14T09:56:50.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.471+0000 7f46a547fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564f12ec350 con 0x5564f1395680 2024-03-14T09:56:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.471+0000 7f46a547fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564f12e7c40 con 0x5564f1395fa0 2024-03-14T09:56:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.471+0000 7f468aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395680 0x5564f1395a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.472+0000 7f468aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395680 0x5564f1395a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48448/0 (socket says 172.21.15.78:48448) 2024-03-14T09:56:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.472+0000 7f468aa00700 1 -- 172.21.15.78:0/3147992769 learned_addr learned my addr 172.21.15.78:0/3147992769 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.472+0000 7f468aa00700 1 -- 172.21.15.78:0/3147992769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395fa0 msgr2=0x5564f139f980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-14T09:56:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.472+0000 7f468aa00700 1 --2- 172.21.15.78:0/3147992769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395fa0 0x5564f139f980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.472+0000 7f468aa00700 1 -- 172.21.15.78:0/3147992769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f139fec0 msgr2=0x5564f13a22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.472+0000 7f468aa00700 1 --2- 172.21.15.78:0/3147992769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f139fec0 0x5564f13a22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.472+0000 7f468aa00700 1 -- 172.21.15.78:0/3147992769 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564f11693b0 con 0x5564f1395680 2024-03-14T09:56:50.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.472+0000 7f468aa00700 1 --2- 172.21.15.78:0/3147992769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395680 0x5564f1395a60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4684096010 tx=0x7f4684099210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaa7df9fbbf90570 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:50.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.472+0000 7f4677e00700 1 -- 172.21.15.78:0/3147992769 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46840995a0 con 0x5564f1395680 2024-03-14T09:56:50.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.473+0000 7f4677e00700 1 -- 172.21.15.78:0/3147992769 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f468409d3d0 con 0x5564f1395680 2024-03-14T09:56:50.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.473+0000 7f4677e00700 1 -- 172.21.15.78:0/3147992769 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46840cc230 con 0x5564f1395680 2024-03-14T09:56:50.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.473+0000 7f468b400700 1 --2- 172.21.15.78:0/1464476857 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f46700a0c80 0x7f46700a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.474+0000 7f46a547fb80 1 -- 172.21.15.78:0/3147992769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395680 msgr2=0x5564f1395a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.474+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3147992769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395680 0x5564f1395a60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4684096010 tx=0x7f4684099210 comp rx=0 tx=0).stop 2024-03-14T09:56:50.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.474+0000 7f468b400700 1 --2- 172.21.15.78:0/1464476857 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f46700a0c80 0x7f46700a3140 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f467c009230 tx=0x7f467c005b60 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:50.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.474+0000 7f46a547fb80 1 -- 172.21.15.78:0/3147992769 shutdown_connections 2024-03-14T09:56:50.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.474+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3147992769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395680 0x5564f1395a60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.474+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3147992769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f139fec0 0x5564f13a22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.474+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3147992769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395fa0 0x5564f139f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.474+0000 7f46a547fb80 1 -- 172.21.15.78:0/3147992769 >> 172.21.15.78:0/3147992769 conn(0x5564f130b220 msgr2=0x5564f1390aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:50.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.474+0000 7f46a547fb80 1 -- 172.21.15.78:0/3147992769 shutdown_connections 2024-03-14T09:56:50.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.474+0000 7f46a547fb80 1 -- 172.21.15.78:0/3147992769 wait complete. 2024-03-14T09:56:50.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.475+0000 7f46a547fb80 1 -- start start 2024-03-14T09:56:50.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.476+0000 7f46a547fb80 1 Processor -- start 2024-03-14T09:56:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.476+0000 7f46a547fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f1395680 0x5564f139f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.476+0000 7f46a547fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395fa0 0x5564f139cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.476+0000 7f46a547fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f139fec0 0x5564f139d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.476+0000 7f46a547fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564f1283d80 con 0x5564f1395680 2024-03-14T09:56:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.476+0000 7f46a547fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564f1283a20 con 0x5564f139fec0 2024-03-14T09:56:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.476+0000 7f46a547fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564f116b6a0 con 0x5564f1395fa0 2024-03-14T09:56:50.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.476+0000 7f46a547fb80 10 client.0 _ll_register_callbacks cb 0x5564f10ef3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:50.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.477+0000 7f468aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f1395680 0x5564f139f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.477+0000 7f468aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f1395680 0x5564f139f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:60340/0 (socket says 172.21.15.78:60340) 2024-03-14T09:56:50.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.477+0000 7f468aa00700 1 -- 172.21.15.78:0/3002600154 learned_addr learned my addr 172.21.15.78:0/3002600154 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:50.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.477+0000 7f468b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f139fec0 0x5564f139d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.477+0000 7f468a000700 1 --2- 172.21.15.78:0/3002600154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395fa0 0x5564f139cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.478+0000 7f468aa00700 1 -- 172.21.15.78:0/3002600154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395fa0 msgr2=0x5564f139cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:50.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.478+0000 7f468aa00700 1 --2- 172.21.15.78:0/3002600154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395fa0 0x5564f139cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.478+0000 7f468aa00700 1 -- 172.21.15.78:0/3002600154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f139fec0 msgr2=0x5564f139d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:50.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.478+0000 7f468aa00700 1 --2- 172.21.15.78:0/3002600154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f139fec0 0x5564f139d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.478+0000 7f468aa00700 1 -- 172.21.15.78:0/3002600154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468c197610 con 0x5564f1395680 2024-03-14T09:56:50.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.478+0000 7f468b400700 1 --2- 172.21.15.78:0/3002600154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f139fec0 0x5564f139d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:56:50.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.478+0000 7f468aa00700 1 --2- 172.21.15.78:0/3002600154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f1395680 0x5564f139f960 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f46840992a0 tx=0x7f468409d120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d884c732887d25 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:50.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.478+0000 7f4674c00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:56:50.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.479+0000 7f4674c00700 1 -- 172.21.15.78:0/3002600154 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684099800 con 0x5564f1395680 2024-03-14T09:56:50.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.479+0000 7f4674c00700 1 -- 172.21.15.78:0/3002600154 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f46840cc070 con 0x5564f1395680 2024-03-14T09:56:50.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.479+0000 7f46a547fb80 1 -- 172.21.15.78:0/3002600154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5564f1388720 con 0x5564f1395680 2024-03-14T09:56:50.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.479+0000 7f4674c00700 1 -- 172.21.15.78:0/3002600154 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4684099960 con 0x5564f1395680 2024-03-14T09:56:50.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.479+0000 7f46a547fb80 10 client.16218 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:50.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.479+0000 7f46a547fb80 1 -- 172.21.15.78:0/3002600154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5564f1115b60 con 0x5564f1395680 2024-03-14T09:56:50.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.480+0000 7f4674c00700 1 -- 172.21.15.78:0/3002600154 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5564f1388720 con 0x5564f1395680 2024-03-14T09:56:50.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.480+0000 7f4674c00700 1 -- 172.21.15.78:0/3002600154 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f46840a1590 con 0x5564f1395680 2024-03-14T09:56:50.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.480+0000 7f46a547fb80 10 client.16218 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:50.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.480+0000 7f46a547fb80 10 client.16218 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:56:50.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.480+0000 7f46a547fb80 20 client.16218 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:50.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.480+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3002600154 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5564f13af830 0x5564f13b1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.480+0000 7f46a547fb80 4 client.16218 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:56:50.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.480+0000 7f46a547fb80 1 -- 172.21.15.78:0/3002600154 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5564f12e0700 con 0x5564f13af830 2024-03-14T09:56:50.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.480+0000 7f468a000700 1 --2- 172.21.15.78:0/3002600154 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5564f13af830 0x5564f13b1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.481+0000 7f4674c00700 10 client.16218 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:50.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.481+0000 7f468a000700 1 --2- 172.21.15.78:0/3002600154 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5564f13af830 0x5564f13b1c50 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dc17af5fbdac377 server_cookie=a9b01899a980641e in_seq=0 out_seq=0 2024-03-14T09:56:50.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.482+0000 7f4674c00700 1 -- 172.21.15.78:0/3002600154 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5564f12e0700 con 0x5564f13af830 2024-03-14T09:56:50.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.482+0000 7f4674c00700 10 client.16218 handle_command_reply: tid=0 2024-03-14T09:56:50.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.482+0000 7f46a547fb80 1 client.16218 shutdown 2024-03-14T09:56:50.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.483+0000 7f46a547fb80 1 -- 172.21.15.78:0/3002600154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f1395680 msgr2=0x5564f139f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.483+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3002600154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f1395680 0x5564f139f960 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f46840992a0 tx=0x7f468409d120 comp rx=0 tx=0).stop 2024-03-14T09:56:50.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.483+0000 7f46a547fb80 1 -- 172.21.15.78:0/3002600154 shutdown_connections 2024-03-14T09:56:50.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.483+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3002600154 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5564f13af830 0x5564f13b1c50 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.483+0000 7f4674c00700 0 client.16218 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:50.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.483+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3002600154 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f139fec0 0x5564f139d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.483+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3002600154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f1395680 0x5564f139f960 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.483+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3002600154 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395fa0 0x5564f139cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.483+0000 7f46a547fb80 1 -- 172.21.15.78:0/3002600154 >> 172.21.15.78:0/3002600154 conn(0x5564f130b220 msgr2=0x5564f1390690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:50.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.483+0000 7f46a547fb80 1 -- 172.21.15.78:0/3002600154 shutdown_connections 2024-03-14T09:56:50.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.483+0000 7f46a547fb80 1 -- 172.21.15.78:0/3002600154 wait complete. 2024-03-14T09:56:50.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.485+0000 7f46a547fb80 20 client.16218 trim_cache size 0 max 16384 2024-03-14T09:56:50.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.511+0000 7f46a547fb80 1 Processor -- start 2024-03-14T09:56:50.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.511+0000 7f46a547fb80 1 -- start start 2024-03-14T09:56:50.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.511+0000 7f46a547fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395680 0x5564f13905d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.511+0000 7f46a547fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395fa0 0x5564f1390b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.511+0000 7f46a547fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f139fec0 0x5564f139c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.511+0000 7f46a547fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564f12e0730 con 0x5564f139fec0 2024-03-14T09:56:50.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.511+0000 7f46a547fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564f1396b20 con 0x5564f1395fa0 2024-03-14T09:56:50.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.511+0000 7f46a547fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564f139e0a0 con 0x5564f1395680 2024-03-14T09:56:50.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.511+0000 7f468aa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395680 0x5564f13905d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.511+0000 7f468aa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395680 0x5564f13905d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:34086/0 (socket says 172.21.15.78:34086) 2024-03-14T09:56:50.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.511+0000 7f468a000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395fa0 0x5564f1390b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.511+0000 7f468aa00700 1 -- 172.21.15.78:0/3787313110 learned_addr learned my addr 172.21.15.78:0/3787313110 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:50.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.512+0000 7f468b400700 1 --2- 172.21.15.78:0/3787313110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f139fec0 0x5564f139c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.512+0000 7f468aa00700 1 -- 172.21.15.78:0/3787313110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395fa0 msgr2=0x5564f1390b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:50.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.512+0000 7f468aa00700 1 --2- 172.21.15.78:0/3787313110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395fa0 0x5564f1390b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.512+0000 7f468aa00700 1 -- 172.21.15.78:0/3787313110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f139fec0 msgr2=0x5564f139c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:50.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.512+0000 7f468aa00700 1 --2- 172.21.15.78:0/3787313110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f139fec0 0x5564f139c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.512+0000 7f468aa00700 1 -- 172.21.15.78:0/3787313110 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564f12adea0 con 0x5564f1395680 2024-03-14T09:56:50.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.512+0000 7f468b400700 1 --2- 172.21.15.78:0/3787313110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f139fec0 0x5564f139c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:56:50.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.512+0000 7f468aa00700 1 --2- 172.21.15.78:0/3787313110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395680 0x5564f13905d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4684095aa0 tx=0x7f4684097b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9f63e4f8314218b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:50.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.513+0000 7f4680c00700 1 -- 172.21.15.78:0/3787313110 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46840b7380 con 0x5564f1395680 2024-03-14T09:56:50.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.513+0000 7f4680c00700 1 -- 172.21.15.78:0/3787313110 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46840b7ce0 con 0x5564f1395680 2024-03-14T09:56:50.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.513+0000 7f4680c00700 1 -- 172.21.15.78:0/3787313110 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684099a60 con 0x5564f1395680 2024-03-14T09:56:50.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.514+0000 7f46a547fb80 1 -- 172.21.15.78:0/3787313110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395680 msgr2=0x5564f13905d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.514+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3787313110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395680 0x5564f13905d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4684095aa0 tx=0x7f4684097b50 comp rx=0 tx=0).stop 2024-03-14T09:56:50.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.514+0000 7f46a547fb80 1 -- 172.21.15.78:0/3787313110 shutdown_connections 2024-03-14T09:56:50.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.514+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3787313110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f139fec0 0x5564f139c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.514+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3787313110 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395680 0x5564f13905d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.514+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3787313110 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395fa0 0x5564f1390b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.514+0000 7f46a547fb80 1 -- 172.21.15.78:0/3787313110 >> 172.21.15.78:0/3787313110 conn(0x5564f130b220 msgr2=0x5564f1398020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:50.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.514+0000 7f46a547fb80 1 -- 172.21.15.78:0/3787313110 shutdown_connections 2024-03-14T09:56:50.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.514+0000 7f46a547fb80 1 -- 172.21.15.78:0/3787313110 wait complete. 2024-03-14T09:56:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.514+0000 7f46a547fb80 1 -- start start 2024-03-14T09:56:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.515+0000 7f46a547fb80 1 Processor -- start 2024-03-14T09:56:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.515+0000 7f46a547fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395680 0x5564f13a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.515+0000 7f46a547fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395fa0 0x5564f13a5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.515+0000 7f46a547fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f139fec0 0x5564f1400350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.515+0000 7f46a547fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564f13a57e0 con 0x5564f139fec0 2024-03-14T09:56:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.515+0000 7f46a547fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564f13a5950 con 0x5564f1395fa0 2024-03-14T09:56:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.515+0000 7f46a547fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564f1400890 con 0x5564f1395680 2024-03-14T09:56:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.515+0000 7f46a547fb80 10 client.0 _ll_register_callbacks cb 0x7f46840990e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:56:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.515+0000 7f468a000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395fa0 0x5564f13a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.516+0000 7f468a000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395fa0 0x5564f13a5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:48474/0 (socket says 172.21.15.78:48474) 2024-03-14T09:56:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.516+0000 7f468a000700 1 -- 172.21.15.78:0/3677784855 learned_addr learned my addr 172.21.15.78:0/3677784855 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:56:50.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.516+0000 7f468b400700 1 --2- 172.21.15.78:0/3677784855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f139fec0 0x5564f1400350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.516+0000 7f468aa00700 1 --2- 172.21.15.78:0/3677784855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395680 0x5564f13a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.516+0000 7f468a000700 1 -- 172.21.15.78:0/3677784855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395680 msgr2=0x5564f13a4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:50.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.516+0000 7f468a000700 1 --2- 172.21.15.78:0/3677784855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395680 0x5564f13a4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.516+0000 7f468a000700 1 -- 172.21.15.78:0/3677784855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f139fec0 msgr2=0x5564f1400350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:56:50.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.516+0000 7f468a000700 1 --2- 172.21.15.78:0/3677784855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f139fec0 0x5564f1400350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.516+0000 7f468a000700 1 -- 172.21.15.78:0/3677784855 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5564f1115b60 con 0x5564f1395fa0 2024-03-14T09:56:50.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.516+0000 7f468a000700 1 --2- 172.21.15.78:0/3677784855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395fa0 0x5564f13a5070 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f468c074b60 tx=0x7f4678009fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33060e44d90e2695 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:56:50.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.516+0000 7f4674c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:56:50.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.517+0000 7f4674c00700 1 -- 172.21.15.78:0/3677784855 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4678017040 con 0x5564f1395fa0 2024-03-14T09:56:50.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.517+0000 7f4674c00700 1 -- 172.21.15.78:0/3677784855 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f4678005450 con 0x5564f1395fa0 2024-03-14T09:56:50.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.517+0000 7f46a547fb80 1 -- 172.21.15.78:0/3677784855 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f468409d3d0 con 0x5564f1395fa0 2024-03-14T09:56:50.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.517+0000 7f4674c00700 1 -- 172.21.15.78:0/3677784855 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4678004e50 con 0x5564f1395fa0 2024-03-14T09:56:50.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.517+0000 7f46a547fb80 10 client.26252 fetch_fsmap learned FSMap version 25 2024-03-14T09:56:50.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.517+0000 7f46a547fb80 1 -- 172.21.15.78:0/3677784855 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5564f116dcd0 con 0x5564f1395fa0 2024-03-14T09:56:50.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.517+0000 7f4674c00700 1 -- 172.21.15.78:0/3677784855 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f468409d3d0 con 0x5564f1395fa0 2024-03-14T09:56:50.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.518+0000 7f4674c00700 1 -- 172.21.15.78:0/3677784855 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f467801ac50 con 0x5564f1395fa0 2024-03-14T09:56:50.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.518+0000 7f46a547fb80 10 client.26252 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:56:50.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.518+0000 7f46a547fb80 10 client.26252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:56:50.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.518+0000 7f46a547fb80 20 client.26252 populate_metadata read hostname 'smithi078' 2024-03-14T09:56:50.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.519+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3677784855 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5564f13af830 0x5564f14040a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:56:50.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.519+0000 7f46a547fb80 4 client.26252 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:56:50.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.519+0000 7f46a547fb80 1 -- 172.21.15.78:0/3677784855 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5564f1136cf0 con 0x5564f13af830 2024-03-14T09:56:50.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.519+0000 7f468aa00700 1 --2- 172.21.15.78:0/3677784855 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5564f13af830 0x5564f14040a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:56:50.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.519+0000 7f468aa00700 1 --2- 172.21.15.78:0/3677784855 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5564f13af830 0x5564f14040a0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9664a2c4f88afab server_cookie=8f9cc0dbba0270cd in_seq=0 out_seq=0 2024-03-14T09:56:50.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.519+0000 7f4674c00700 10 client.26252 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:50.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.520+0000 7f4674c00700 1 -- 172.21.15.78:0/3677784855 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x5564f1136cf0 con 0x5564f13af830 2024-03-14T09:56:50.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.520+0000 7f4674c00700 10 client.26252 handle_command_reply: tid=0 2024-03-14T09:56:50.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.520+0000 7f46a547fb80 1 client.26252 shutdown 2024-03-14T09:56:50.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.520+0000 7f46a547fb80 1 -- 172.21.15.78:0/3677784855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395fa0 msgr2=0x5564f13a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.520+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3677784855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395fa0 0x5564f13a5070 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f468c074b60 tx=0x7f4678009fb0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.520+0000 7f46a547fb80 1 -- 172.21.15.78:0/3677784855 shutdown_connections 2024-03-14T09:56:50.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.520+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3677784855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5564f139fec0 0x5564f1400350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.520+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3677784855 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5564f1395680 0x5564f13a4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.520+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3677784855 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5564f13af830 0x5564f14040a0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.520+0000 7f4674c00700 0 client.26252 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:56:50.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.521+0000 7f46a547fb80 1 --2- 172.21.15.78:0/3677784855 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5564f1395fa0 0x5564f13a5070 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.521+0000 7f46a547fb80 1 -- 172.21.15.78:0/3677784855 >> 172.21.15.78:0/3677784855 conn(0x5564f130b220 msgr2=0x5564f1394560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:50.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.521+0000 7f46a547fb80 1 -- 172.21.15.78:0/3677784855 shutdown_connections 2024-03-14T09:56:50.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.521+0000 7f46a547fb80 1 -- 172.21.15.78:0/3677784855 wait complete. 2024-03-14T09:56:50.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.522+0000 7f46a547fb80 20 client.26252 trim_cache size 0 max 16384 2024-03-14T09:56:50.523 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:56:50.523 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "scrub active (1298 inodes in the stack)", 2024-03-14T09:56:50.523 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T09:56:50.524 INFO:teuthology.orchestra.run.smithi078.stdout: "0163b6e2-bc79-495d-9276-18e6498a59e2": { 2024-03-14T09:56:50.524 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T09:56:50.524 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "0163b6e2-bc79-495d-9276-18e6498a59e2", 2024-03-14T09:56:50.524 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T09:56:50.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:56:50.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:56:50.524 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:56:50.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.522+0000 7f4691a00700 1 -- 172.21.15.78:0/1464476857 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f46700a0c80 msgr2=0x7f46700a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.522+0000 7f4691a00700 1 --2- 172.21.15.78:0/1464476857 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f46700a0c80 0x7f46700a3140 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f467c009230 tx=0x7f467c005b60 comp rx=0 tx=0).stop 2024-03-14T09:56:50.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.522+0000 7f4691a00700 1 -- 172.21.15.78:0/1464476857 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f468c100f30 msgr2=0x7f468c0730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:56:50.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.522+0000 7f4691a00700 1 --2- 172.21.15.78:0/1464476857 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f468c100f30 0x7f468c0730b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f468400c0c0 tx=0x7f468400c480 comp rx=0 tx=0).stop 2024-03-14T09:56:50.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.523+0000 7f4691a00700 1 -- 172.21.15.78:0/1464476857 shutdown_connections 2024-03-14T09:56:50.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.523+0000 7f4691a00700 1 --2- 172.21.15.78:0/1464476857 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f46700a0c80 0x7f46700a3140 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.523+0000 7f4691a00700 1 --2- 172.21.15.78:0/1464476857 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f468c100f30 0x7f468c0730b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.523+0000 7f4691a00700 1 --2- 172.21.15.78:0/1464476857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f468c073b30 0x7f468c1a5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.523+0000 7f4691a00700 1 --2- 172.21.15.78:0/1464476857 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f468c106050 0x7f468c0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:56:50.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.523+0000 7f4691a00700 1 -- 172.21.15.78:0/1464476857 >> 172.21.15.78:0/1464476857 conn(0x7f468c0fc7b0 msgr2=0x7f468c0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:56:50.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.523+0000 7f4691a00700 1 -- 172.21.15.78:0/1464476857 shutdown_connections 2024-03-14T09:56:50.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:56:50.523+0000 7f4691a00700 1 -- 172.21.15.78:0/1464476857 wait complete. 2024-03-14T09:56:50.545 INFO:tasks.cephfs.filesystem:scrub status for tag:0163b6e2-bc79-495d-9276-18e6498a59e2 - {'path': '/', 'tag': '0163b6e2-bc79-495d-9276-18e6498a59e2', 'options': 'recursive,force'} 2024-03-14T09:56:51.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:51 smithi078 ceph-mon[38143]: pgmap v83: 129 pgs: 129 active+clean; 101 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 321 op/s 2024-03-14T09:56:51.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:51 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T09:56:51.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:51 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:56:51.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:51 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T09:56:51.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:51 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:56:51.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:51 smithi152 ceph-mon[48279]: pgmap v83: 129 pgs: 129 active+clean; 101 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 321 op/s 2024-03-14T09:56:51.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:51 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T09:56:51.502 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:51 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:56:51.502 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:51 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T09:56:51.502 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:51 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:56:51.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:51 smithi132 ceph-mon[47959]: pgmap v83: 129 pgs: 129 active+clean; 101 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 321 op/s 2024-03-14T09:56:51.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:51 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T09:56:51.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:51 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:56:51.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:51 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T09:56:51.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:51 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:56:53.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:53 smithi078 ceph-mon[38143]: pgmap v84: 129 pgs: 129 active+clean; 104 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 452 op/s 2024-03-14T09:56:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:53 smithi152 ceph-mon[48279]: pgmap v84: 129 pgs: 129 active+clean; 104 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 452 op/s 2024-03-14T09:56:53.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:53 smithi132 ceph-mon[47959]: pgmap v84: 129 pgs: 129 active+clean; 104 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 452 op/s 2024-03-14T09:56:54.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:54 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T09:56:54.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:54 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T09:56:54.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:54 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T09:56:55.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:55 smithi078 ceph-mon[38143]: pgmap v85: 129 pgs: 129 active+clean; 104 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 284 op/s 2024-03-14T09:56:55.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:55 smithi152 ceph-mon[48279]: pgmap v85: 129 pgs: 129 active+clean; 104 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 284 op/s 2024-03-14T09:56:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:55 smithi132 ceph-mon[47959]: pgmap v85: 129 pgs: 129 active+clean; 104 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 284 op/s 2024-03-14T09:56:57.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:57 smithi152 ceph-mon[48279]: pgmap v86: 129 pgs: 129 active+clean; 105 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 411 op/s 2024-03-14T09:56:57.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:57 smithi132 ceph-mon[47959]: pgmap v86: 129 pgs: 129 active+clean; 105 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 411 op/s 2024-03-14T09:56:57.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:57 smithi078 ceph-mon[38143]: pgmap v86: 129 pgs: 129 active+clean; 105 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 411 op/s 2024-03-14T09:56:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:56:58 smithi078 ceph-mon[38143]: pgmap v87: 129 pgs: 129 active+clean; 105 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 258 op/s 2024-03-14T09:56:59.248 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:56:58 smithi152 ceph-mon[48279]: pgmap v87: 129 pgs: 129 active+clean; 105 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 258 op/s 2024-03-14T09:56:59.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:56:58 smithi132 ceph-mon[47959]: pgmap v87: 129 pgs: 129 active+clean; 105 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 258 op/s 2024-03-14T09:57:01.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:01 smithi078 ceph-mon[38143]: pgmap v88: 129 pgs: 129 active+clean; 105 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 258 op/s 2024-03-14T09:57:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:01 smithi152 ceph-mon[48279]: pgmap v88: 129 pgs: 129 active+clean; 105 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 258 op/s 2024-03-14T09:57:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:01 smithi132 ceph-mon[47959]: pgmap v88: 129 pgs: 129 active+clean; 105 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 258 op/s 2024-03-14T09:57:03.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:03 smithi078 ceph-mon[38143]: pgmap v89: 129 pgs: 129 active+clean; 107 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 395 op/s 2024-03-14T09:57:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:03 smithi152 ceph-mon[48279]: pgmap v89: 129 pgs: 129 active+clean; 107 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 395 op/s 2024-03-14T09:57:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:03 smithi132 ceph-mon[47959]: pgmap v89: 129 pgs: 129 active+clean; 107 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 395 op/s 2024-03-14T09:57:04.689 INFO:tasks.workunit.client.0.smithi078.stdout:40000 copy from 0x5dfcb to 0x607bc, (0x27f1 bytes) at 0xf5933 2024-03-14T09:57:05.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:05 smithi078 ceph-mon[38143]: pgmap v90: 129 pgs: 129 active+clean; 107 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 263 op/s 2024-03-14T09:57:05.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:05 smithi152 ceph-mon[48279]: pgmap v90: 129 pgs: 129 active+clean; 107 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 263 op/s 2024-03-14T09:57:05.519 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:05 smithi132 ceph-mon[47959]: pgmap v90: 129 pgs: 129 active+clean; 107 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 263 op/s 2024-03-14T09:57:07.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:07 smithi078 ceph-mon[38143]: pgmap v91: 129 pgs: 129 active+clean; 109 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 390 op/s 2024-03-14T09:57:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:07 smithi152 ceph-mon[48279]: pgmap v91: 129 pgs: 129 active+clean; 109 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 390 op/s 2024-03-14T09:57:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:07 smithi132 ceph-mon[47959]: pgmap v91: 129 pgs: 129 active+clean; 109 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.3 MiB/s wr, 390 op/s 2024-03-14T09:57:09.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:08 smithi078 ceph-mon[38143]: pgmap v92: 129 pgs: 129 active+clean; 109 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 264 op/s 2024-03-14T09:57:09.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:08 smithi152 ceph-mon[48279]: pgmap v92: 129 pgs: 129 active+clean; 109 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 264 op/s 2024-03-14T09:57:09.269 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:08 smithi132 ceph-mon[47959]: pgmap v92: 129 pgs: 129 active+clean; 109 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 264 op/s 2024-03-14T09:57:11.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:11 smithi078 ceph-mon[38143]: pgmap v93: 129 pgs: 129 active+clean; 109 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 264 op/s 2024-03-14T09:57:11.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:11 smithi152 ceph-mon[48279]: pgmap v93: 129 pgs: 129 active+clean; 109 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 264 op/s 2024-03-14T09:57:11.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:11 smithi132 ceph-mon[47959]: pgmap v93: 129 pgs: 129 active+clean; 109 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 264 op/s 2024-03-14T09:57:13.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:13 smithi078 ceph-mon[38143]: pgmap v94: 129 pgs: 129 active+clean; 110 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 396 op/s 2024-03-14T09:57:13.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:13 smithi152 ceph-mon[48279]: pgmap v94: 129 pgs: 129 active+clean; 110 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 396 op/s 2024-03-14T09:57:13.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:13 smithi132 ceph-mon[47959]: pgmap v94: 129 pgs: 129 active+clean; 110 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 396 op/s 2024-03-14T09:57:15.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:15 smithi078 ceph-mon[38143]: pgmap v95: 129 pgs: 129 active+clean; 110 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 259 op/s 2024-03-14T09:57:15.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:15 smithi152 ceph-mon[48279]: pgmap v95: 129 pgs: 129 active+clean; 110 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 259 op/s 2024-03-14T09:57:15.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:15 smithi132 ceph-mon[47959]: pgmap v95: 129 pgs: 129 active+clean; 110 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 259 op/s 2024-03-14T09:57:17.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:17 smithi078 ceph-mon[38143]: pgmap v96: 129 pgs: 129 active+clean; 112 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 392 op/s 2024-03-14T09:57:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:17 smithi152 ceph-mon[48279]: pgmap v96: 129 pgs: 129 active+clean; 112 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 392 op/s 2024-03-14T09:57:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:17 smithi132 ceph-mon[47959]: pgmap v96: 129 pgs: 129 active+clean; 112 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 392 op/s 2024-03-14T09:57:19.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:18 smithi078 ceph-mon[38143]: pgmap v97: 129 pgs: 129 active+clean; 112 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 265 op/s 2024-03-14T09:57:19.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:18 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:57:19.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:18 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:19.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:18 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:19.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:18 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:19.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:18 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:19.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:18 smithi152 ceph-mon[48279]: pgmap v97: 129 pgs: 129 active+clean; 112 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 265 op/s 2024-03-14T09:57:19.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:18 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:57:19.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:18 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:19.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:18 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:19.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:18 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:19.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:18 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:19.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:18 smithi132 ceph-mon[47959]: pgmap v97: 129 pgs: 129 active+clean; 112 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 265 op/s 2024-03-14T09:57:19.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:18 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:57:19.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:18 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:19.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:18 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:19.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:18 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:19.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:18 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:20.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:19 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:20.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:19 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:20.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:19 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:57:20.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:19 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:57:20.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:19 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:20.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:19 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:20.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:19 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:20.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:19 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:57:20.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:19 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:57:20.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:19 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:20.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:19 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:20.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:19 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:20.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:19 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:57:20.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:19 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:57:20.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:19 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:57:20.546 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:57:20.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.771+0000 7f8046800700 1 -- 172.21.15.78:0/642316743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f80401018d0 msgr2=0x7f8040105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:20.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.771+0000 7f8046800700 1 --2- 172.21.15.78:0/642316743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f80401018d0 0x7f8040105910 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f802c009990 tx=0x7f802c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.771+0000 7f8046800700 1 -- 172.21.15.78:0/642316743 shutdown_connections 2024-03-14T09:57:20.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.771+0000 7f8046800700 1 --2- 172.21.15.78:0/642316743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8040106040 0x7f804010dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.771+0000 7f8046800700 1 --2- 172.21.15.78:0/642316743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8040100f20 0x7f8040101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.771+0000 7f8046800700 1 --2- 172.21.15.78:0/642316743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f80401018d0 0x7f8040105910 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.771+0000 7f8046800700 1 -- 172.21.15.78:0/642316743 >> 172.21.15.78:0/642316743 conn(0x7f80400fc7a0 msgr2=0x7f80400febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:20.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.771+0000 7f8046800700 1 -- 172.21.15.78:0/642316743 shutdown_connections 2024-03-14T09:57:20.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.771+0000 7f8046800700 1 -- 172.21.15.78:0/642316743 wait complete. 2024-03-14T09:57:20.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.772+0000 7f8046800700 1 Processor -- start 2024-03-14T09:57:20.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.772+0000 7f8046800700 1 -- start start 2024-03-14T09:57:20.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.773+0000 7f8046800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8040100f20 0x7f804019f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.773+0000 7f8046800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8040106040 0x7f804019f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.773+0000 7f8046800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8040199600 0x7f8040199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.773+0000 7f8046800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804019a050 con 0x7f8040100f20 2024-03-14T09:57:20.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.773+0000 7f8046800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804019a190 con 0x7f8040199600 2024-03-14T09:57:20.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.773+0000 7f8046800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804019a350 con 0x7f8040106040 2024-03-14T09:57:20.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.773+0000 7f803ea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8040106040 0x7f804019f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.773+0000 7f803ea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8040106040 0x7f804019f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47096/0 (socket says 172.21.15.78:47096) 2024-03-14T09:57:20.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.773+0000 7f803ea00700 1 -- 172.21.15.78:0/2155964270 learned_addr learned my addr 172.21.15.78:0/2155964270 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:20.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.773+0000 7f803fe00700 1 --2- 172.21.15.78:0/2155964270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8040199600 0x7f8040199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.773+0000 7f803f400700 1 --2- 172.21.15.78:0/2155964270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8040100f20 0x7f804019f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.773+0000 7f803ea00700 1 -- 172.21.15.78:0/2155964270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8040199600 msgr2=0x7f8040199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:20.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.774+0000 7f803ea00700 1 --2- 172.21.15.78:0/2155964270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8040199600 0x7f8040199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.774+0000 7f803ea00700 1 -- 172.21.15.78:0/2155964270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8040100f20 msgr2=0x7f804019f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:20.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.774+0000 7f803ea00700 1 --2- 172.21.15.78:0/2155964270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8040100f20 0x7f804019f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.774+0000 7f803ea00700 1 -- 172.21.15.78:0/2155964270 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8038009c40 con 0x7f8040106040 2024-03-14T09:57:20.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.774+0000 7f803ea00700 1 --2- 172.21.15.78:0/2155964270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8040106040 0x7f804019f9b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f802c000c00 tx=0x7f802c00bfb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:20.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.774+0000 7f8037e00700 1 -- 172.21.15.78:0/2155964270 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c01d070 con 0x7f8040106040 2024-03-14T09:57:20.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.774+0000 7f8037e00700 1 -- 172.21.15.78:0/2155964270 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f802c0047e0 con 0x7f8040106040 2024-03-14T09:57:20.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.774+0000 7f8046800700 1 -- 172.21.15.78:0/2155964270 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802c0095f0 con 0x7f8040106040 2024-03-14T09:57:20.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.774+0000 7f8037e00700 1 -- 172.21.15.78:0/2155964270 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c005250 con 0x7f8040106040 2024-03-14T09:57:20.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.774+0000 7f8046800700 1 -- 172.21.15.78:0/2155964270 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80401a65c0 con 0x7f8040106040 2024-03-14T09:57:20.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.779+0000 7f8037e00700 1 -- 172.21.15.78:0/2155964270 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f802c004950 con 0x7f8040106040 2024-03-14T09:57:20.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.780+0000 7f805a3e4b80 1 Processor -- start 2024-03-14T09:57:20.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.780+0000 7f805a3e4b80 1 -- start start 2024-03-14T09:57:20.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.781+0000 7f8037e00700 1 --2- 172.21.15.78:0/2155964270 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f80240a0c00 0x7f80240a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.781+0000 7f805a3e4b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23680 0x56432cc23a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.781+0000 7f805a3e4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc2d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.781+0000 7f805a3e4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc2dec0 0x56432cc302b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.781+0000 7f805a3e4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56432cb7c990 con 0x56432cc2dec0 2024-03-14T09:57:20.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.781+0000 7f805a3e4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56432cb7a350 con 0x56432cc23fa0 2024-03-14T09:57:20.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.781+0000 7f805a3e4b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56432cb75c40 con 0x56432cc23680 2024-03-14T09:57:20.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.781+0000 7f8037e00700 1 -- 172.21.15.78:0/2155964270 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f802c0cd800 con 0x7f8040106040 2024-03-14T09:57:20.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.781+0000 7f803ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc2d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.781+0000 7f803ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc2d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:52350/0 (socket says 172.21.15.78:52350) 2024-03-14T09:57:20.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.781+0000 7f803ea00700 1 -- 172.21.15.78:0/228864720 learned_addr learned my addr 172.21.15.78:0/228864720 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:20.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.781+0000 7f803f400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23680 0x56432cc23a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.781+0000 7f803fe00700 1 --2- 172.21.15.78:0/2155964270 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f80240a0c00 0x7f80240a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.782+0000 7f803fe00700 1 --2- 172.21.15.78:0/228864720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc2dec0 0x56432cc302b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.782+0000 7f803ea00700 1 -- 172.21.15.78:0/228864720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23680 msgr2=0x56432cc23a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:20.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.782+0000 7f803ea00700 1 --2- 172.21.15.78:0/228864720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23680 0x56432cc23a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.782+0000 7f803ea00700 1 -- 172.21.15.78:0/228864720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc2dec0 msgr2=0x56432cc302b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:20.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.782+0000 7f803ea00700 1 --2- 172.21.15.78:0/228864720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc2dec0 0x56432cc302b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.782+0000 7f803ea00700 1 -- 172.21.15.78:0/228864720 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56432c9f73b0 con 0x56432cc23fa0 2024-03-14T09:57:20.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.782+0000 7f803f400700 1 --2- 172.21.15.78:0/228864720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23680 0x56432cc23a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:20.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.782+0000 7f803fe00700 1 --2- 172.21.15.78:0/228864720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc2dec0 0x56432cc302b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:57:20.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.782+0000 7f803ea00700 1 --2- 172.21.15.78:0/228864720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc2d980 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f802c005000 tx=0x7f802c096c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cffe011199fba965 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:20.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.782+0000 7f8034c00700 1 -- 172.21.15.78:0/228864720 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c09a870 con 0x56432cc23fa0 2024-03-14T09:57:20.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.782+0000 7f8034c00700 1 -- 172.21.15.78:0/228864720 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f802c09e6a0 con 0x56432cc23fa0 2024-03-14T09:57:20.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.782+0000 7f8034c00700 1 -- 172.21.15.78:0/228864720 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c098cf0 con 0x56432cc23fa0 2024-03-14T09:57:20.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.783+0000 7f805a3e4b80 1 -- 172.21.15.78:0/228864720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 msgr2=0x56432cc2d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:20.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.783+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/228864720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc2d980 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f802c005000 tx=0x7f802c096c30 comp rx=0 tx=0).stop 2024-03-14T09:57:20.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.783+0000 7f803fe00700 1 --2- 172.21.15.78:0/2155964270 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f80240a0c00 0x7f80240a30c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8038009fd0 tx=0x7f8038009920 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:20.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.783+0000 7f805a3e4b80 1 -- 172.21.15.78:0/228864720 shutdown_connections 2024-03-14T09:57:20.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.783+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/228864720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc2dec0 0x56432cc302b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.783+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/228864720 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23680 0x56432cc23a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.783+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/228864720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc2d980 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.783+0000 7f805a3e4b80 1 -- 172.21.15.78:0/228864720 >> 172.21.15.78:0/228864720 conn(0x56432cb99220 msgr2=0x56432cc1eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:20.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.784+0000 7f805a3e4b80 1 -- 172.21.15.78:0/228864720 shutdown_connections 2024-03-14T09:57:20.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.784+0000 7f805a3e4b80 1 -- 172.21.15.78:0/228864720 wait complete. 2024-03-14T09:57:20.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.784+0000 7f805a3e4b80 1 -- start start 2024-03-14T09:57:20.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.785+0000 7f805a3e4b80 1 Processor -- start 2024-03-14T09:57:20.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.785+0000 7f805a3e4b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23680 0x56432cc2d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.785+0000 7f805a3e4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc2abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.785+0000 7f805a3e4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc2dec0 0x56432cc2b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.785+0000 7f805a3e4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56432cb11d80 con 0x56432cc2dec0 2024-03-14T09:57:20.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.785+0000 7f805a3e4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56432cb11a20 con 0x56432cc23fa0 2024-03-14T09:57:20.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.785+0000 7f805a3e4b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56432c9f96a0 con 0x56432cc23680 2024-03-14T09:57:20.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.785+0000 7f805a3e4b80 10 client.0 _ll_register_callbacks cb 0x56432c97d3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:20.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.786+0000 7f803fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc2dec0 0x56432cc2b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.786+0000 7f803f400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23680 0x56432cc2d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.786+0000 7f803f400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23680 0x56432cc2d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47108/0 (socket says 172.21.15.78:47108) 2024-03-14T09:57:20.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.786+0000 7f803f400700 1 -- 172.21.15.78:0/4115011505 learned_addr learned my addr 172.21.15.78:0/4115011505 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:20.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.786+0000 7f803ea00700 1 --2- 172.21.15.78:0/4115011505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc2abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.786+0000 7f803ea00700 1 -- 172.21.15.78:0/4115011505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23680 msgr2=0x56432cc2d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:20.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.786+0000 7f803ea00700 1 --2- 172.21.15.78:0/4115011505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23680 0x56432cc2d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.786+0000 7f803ea00700 1 -- 172.21.15.78:0/4115011505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc2dec0 msgr2=0x56432cc2b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:20.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.786+0000 7f803ea00700 1 --2- 172.21.15.78:0/4115011505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc2dec0 0x56432cc2b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.786+0000 7f803ea00700 1 -- 172.21.15.78:0/4115011505 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8030009830 con 0x56432cc23fa0 2024-03-14T09:57:20.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.786+0000 7f803f400700 1 --2- 172.21.15.78:0/4115011505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23680 0x56432cc2d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:20.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.786+0000 7f803fe00700 1 --2- 172.21.15.78:0/4115011505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc2dec0 0x56432cc2b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:20.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.787+0000 7f803ea00700 1 --2- 172.21.15.78:0/4115011505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc2abc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f802c09a570 tx=0x7f802c0cb090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=756fb6379c8ad33f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:20.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.787+0000 7f8029600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:57:20.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.787+0000 7f8029600700 1 -- 172.21.15.78:0/4115011505 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c09acd0 con 0x56432cc23fa0 2024-03-14T09:57:20.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.787+0000 7f8029600700 1 -- 172.21.15.78:0/4115011505 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f802c004c30 con 0x56432cc23fa0 2024-03-14T09:57:20.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.787+0000 7f805a3e4b80 1 -- 172.21.15.78:0/4115011505 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56432cc16720 con 0x56432cc23fa0 2024-03-14T09:57:20.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.788+0000 7f8029600700 1 -- 172.21.15.78:0/4115011505 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f802c0cd650 con 0x56432cc23fa0 2024-03-14T09:57:20.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.788+0000 7f805a3e4b80 10 client.26264 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:20.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.788+0000 7f805a3e4b80 1 -- 172.21.15.78:0/4115011505 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f80401a65c0 con 0x56432cc23fa0 2024-03-14T09:57:20.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.788+0000 7f8029600700 1 -- 172.21.15.78:0/4115011505 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56432cc16720 con 0x56432cc23fa0 2024-03-14T09:57:20.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.788+0000 7f8029600700 1 -- 172.21.15.78:0/4115011505 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f802c05bd20 con 0x56432cc23fa0 2024-03-14T09:57:20.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.788+0000 7f805a3e4b80 10 client.26264 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:20.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.788+0000 7f805a3e4b80 10 client.26264 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:57:20.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.788+0000 7f805a3e4b80 20 client.26264 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:20.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.788+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/4115011505 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56432cc457d0 0x56432cc47bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.788+0000 7f805a3e4b80 4 client.26264 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:57:20.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.788+0000 7f805a3e4b80 1 -- 172.21.15.78:0/4115011505 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56432cb6e700 con 0x56432cc457d0 2024-03-14T09:57:20.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.789+0000 7f803f400700 1 --2- 172.21.15.78:0/4115011505 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56432cc457d0 0x56432cc47bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.789+0000 7f803f400700 1 --2- 172.21.15.78:0/4115011505 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56432cc457d0 0x56432cc47bc0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1db265dd5fb22aa4 server_cookie=6eab6897b4137c0e in_seq=0 out_seq=0 2024-03-14T09:57:20.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.789+0000 7f8029600700 10 client.26264 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:20.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.790+0000 7f8029600700 1 -- 172.21.15.78:0/4115011505 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56432cb6e700 con 0x56432cc457d0 2024-03-14T09:57:20.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.790+0000 7f8029600700 10 client.26264 handle_command_reply: tid=0 2024-03-14T09:57:20.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.790+0000 7f805a3e4b80 1 client.26264 shutdown 2024-03-14T09:57:20.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.790+0000 7f805a3e4b80 1 -- 172.21.15.78:0/4115011505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 msgr2=0x56432cc2abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:20.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.790+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/4115011505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc2abc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f802c09a570 tx=0x7f802c0cb090 comp rx=0 tx=0).stop 2024-03-14T09:57:20.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.790+0000 7f805a3e4b80 1 -- 172.21.15.78:0/4115011505 shutdown_connections 2024-03-14T09:57:20.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.790+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/4115011505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc2dec0 0x56432cc2b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.790+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/4115011505 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23680 0x56432cc2d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.790+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/4115011505 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56432cc457d0 0x56432cc47bc0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.790+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/4115011505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc2abc0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.790+0000 7f8029600700 0 client.26264 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:20.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.790+0000 7f805a3e4b80 1 -- 172.21.15.78:0/4115011505 >> 172.21.15.78:0/4115011505 conn(0x56432cb99220 msgr2=0x56432cc1e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:20.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.791+0000 7f805a3e4b80 1 -- 172.21.15.78:0/4115011505 shutdown_connections 2024-03-14T09:57:20.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.791+0000 7f805a3e4b80 1 -- 172.21.15.78:0/4115011505 wait complete. 2024-03-14T09:57:20.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.791+0000 7f805a3e4b80 20 client.26264 trim_cache size 0 max 16384 2024-03-14T09:57:20.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.812+0000 7f805a3e4b80 1 Processor -- start 2024-03-14T09:57:20.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.812+0000 7f805a3e4b80 1 -- start start 2024-03-14T09:57:20.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.813+0000 7f805a3e4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc23680 0x56432cc34880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.813+0000 7f805a3e4b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23fa0 0x56432cc25f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.813+0000 7f805a3e4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc2dec0 0x56432cc26460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.813+0000 7f805a3e4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56432cc24af0 con 0x56432cc23680 2024-03-14T09:57:20.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.813+0000 7f805a3e4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56432cc250c0 con 0x56432cc2dec0 2024-03-14T09:57:20.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.813+0000 7f805a3e4b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56432cc23de0 con 0x56432cc23fa0 2024-03-14T09:57:20.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.813+0000 7f803fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc2dec0 0x56432cc26460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.813+0000 7f803fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc2dec0 0x56432cc26460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:52364/0 (socket says 172.21.15.78:52364) 2024-03-14T09:57:20.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.813+0000 7f803fe00700 1 -- 172.21.15.78:0/1728676283 learned_addr learned my addr 172.21.15.78:0/1728676283 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:20.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.813+0000 7f803ea00700 1 --2- 172.21.15.78:0/1728676283 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23fa0 0x56432cc25f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.814+0000 7f803f400700 1 --2- 172.21.15.78:0/1728676283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc23680 0x56432cc34880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.814+0000 7f803fe00700 1 -- 172.21.15.78:0/1728676283 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23fa0 msgr2=0x56432cc25f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.814+0000 7f803fe00700 1 --2- 172.21.15.78:0/1728676283 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23fa0 0x56432cc25f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.814+0000 7f803fe00700 1 -- 172.21.15.78:0/1728676283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc23680 msgr2=0x56432cc34880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.814+0000 7f803fe00700 1 --2- 172.21.15.78:0/1728676283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc23680 0x56432cc34880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.814+0000 7f803fe00700 1 -- 172.21.15.78:0/1728676283 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56432c9a3b60 con 0x56432cc2dec0 2024-03-14T09:57:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.814+0000 7f803f400700 1 --2- 172.21.15.78:0/1728676283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc23680 0x56432cc34880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:57:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.814+0000 7f803fe00700 1 --2- 172.21.15.78:0/1728676283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc2dec0 0x56432cc26460 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8038009250 tx=0x7f8038015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=361adf7d64228d93 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.814+0000 7f8035600700 1 -- 172.21.15.78:0/1728676283 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803800e040 con 0x56432cc2dec0 2024-03-14T09:57:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.814+0000 7f8035600700 1 -- 172.21.15.78:0/1728676283 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f803800f040 con 0x56432cc2dec0 2024-03-14T09:57:20.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.814+0000 7f8035600700 1 -- 172.21.15.78:0/1728676283 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8038024960 con 0x56432cc2dec0 2024-03-14T09:57:20.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.815+0000 7f805a3e4b80 1 -- 172.21.15.78:0/1728676283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc2dec0 msgr2=0x56432cc26460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:20.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.815+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/1728676283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc2dec0 0x56432cc26460 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8038009250 tx=0x7f8038015000 comp rx=0 tx=0).stop 2024-03-14T09:57:20.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.815+0000 7f805a3e4b80 1 -- 172.21.15.78:0/1728676283 shutdown_connections 2024-03-14T09:57:20.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.815+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/1728676283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc2dec0 0x56432cc26460 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.815+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/1728676283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc23680 0x56432cc34880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.815+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/1728676283 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc23fa0 0x56432cc25f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.815+0000 7f805a3e4b80 1 -- 172.21.15.78:0/1728676283 >> 172.21.15.78:0/1728676283 conn(0x56432cb99220 msgr2=0x56432cc38e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:20.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.815+0000 7f805a3e4b80 1 -- 172.21.15.78:0/1728676283 shutdown_connections 2024-03-14T09:57:20.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.815+0000 7f805a3e4b80 1 -- 172.21.15.78:0/1728676283 wait complete. 2024-03-14T09:57:20.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.816+0000 7f805a3e4b80 1 -- start start 2024-03-14T09:57:20.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.816+0000 7f805a3e4b80 1 Processor -- start 2024-03-14T09:57:20.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.817+0000 7f805a3e4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc23680 0x56432cc32e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.817+0000 7f805a3e4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc333c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.817+0000 7f805a3e4b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc2dec0 0x56432cc33900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.817+0000 7f805a3e4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56432cc97290 con 0x56432cc23680 2024-03-14T09:57:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.817+0000 7f805a3e4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56432cc973d0 con 0x56432cc23fa0 2024-03-14T09:57:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.817+0000 7f805a3e4b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56432cc97510 con 0x56432cc2dec0 2024-03-14T09:57:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.817+0000 7f805a3e4b80 10 client.0 _ll_register_callbacks cb 0x56432c97d3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.817+0000 7f803ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc333c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.817+0000 7f803ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc333c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:52376/0 (socket says 172.21.15.78:52376) 2024-03-14T09:57:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.817+0000 7f803ea00700 1 -- 172.21.15.78:0/2609420068 learned_addr learned my addr 172.21.15.78:0/2609420068 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.817+0000 7f803f400700 1 --2- 172.21.15.78:0/2609420068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc23680 0x56432cc32e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.817+0000 7f803fe00700 1 --2- 172.21.15.78:0/2609420068 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc2dec0 0x56432cc33900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.817+0000 7f803ea00700 1 -- 172.21.15.78:0/2609420068 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc2dec0 msgr2=0x56432cc33900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.818+0000 7f803ea00700 1 --2- 172.21.15.78:0/2609420068 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc2dec0 0x56432cc33900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.818+0000 7f803ea00700 1 -- 172.21.15.78:0/2609420068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc23680 msgr2=0x56432cc32e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.818+0000 7f803ea00700 1 --2- 172.21.15.78:0/2609420068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc23680 0x56432cc32e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.818+0000 7f803ea00700 1 -- 172.21.15.78:0/2609420068 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8038007500 con 0x56432cc23fa0 2024-03-14T09:57:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.818+0000 7f803ea00700 1 --2- 172.21.15.78:0/2609420068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc333c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x56432cc2c060 tx=0x7f802c009380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe8c215e4998d80e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.818+0000 7f8029600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:57:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.818+0000 7f8029600700 1 -- 172.21.15.78:0/2609420068 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c085320 con 0x56432cc23fa0 2024-03-14T09:57:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.818+0000 7f8029600700 1 -- 172.21.15.78:0/2609420068 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f802c09a870 con 0x56432cc23fa0 2024-03-14T09:57:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.818+0000 7f805a3e4b80 1 -- 172.21.15.78:0/2609420068 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f802c09e6a0 con 0x56432cc23fa0 2024-03-14T09:57:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.818+0000 7f8029600700 1 -- 172.21.15.78:0/2609420068 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f802c0a2c60 con 0x56432cc23fa0 2024-03-14T09:57:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.818+0000 7f805a3e4b80 10 client.26270 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.818+0000 7f805a3e4b80 1 -- 172.21.15.78:0/2609420068 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56432cb3bea0 con 0x56432cc23fa0 2024-03-14T09:57:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.819+0000 7f8029600700 1 -- 172.21.15.78:0/2609420068 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f802c09e6a0 con 0x56432cc23fa0 2024-03-14T09:57:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.819+0000 7f8029600700 1 -- 172.21.15.78:0/2609420068 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f802c09aad0 con 0x56432cc23fa0 2024-03-14T09:57:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.819+0000 7f805a3e4b80 10 client.26270 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.819+0000 7f805a3e4b80 10 client.26270 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:57:20.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.819+0000 7f805a3e4b80 20 client.26270 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.819+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/2609420068 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56432cc98700 0x56432cc9aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.819+0000 7f805a3e4b80 4 client.26270 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:57:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.819+0000 7f805a3e4b80 1 -- 172.21.15.78:0/2609420068 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56432c9c4cf0 con 0x56432cc98700 2024-03-14T09:57:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.819+0000 7f803f400700 1 --2- 172.21.15.78:0/2609420068 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56432cc98700 0x56432cc9aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.820+0000 7f803f400700 1 --2- 172.21.15.78:0/2609420068 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56432cc98700 0x56432cc9aaf0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fd33ed32c388110 server_cookie=93487f7b2c332d2c in_seq=0 out_seq=0 2024-03-14T09:57:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.820+0000 7f8029600700 10 client.26270 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.820+0000 7f8029600700 1 -- 172.21.15.78:0/2609420068 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56432c9c4cf0 con 0x56432cc98700 2024-03-14T09:57:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.820+0000 7f8029600700 10 client.26270 handle_command_reply: tid=0 2024-03-14T09:57:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.820+0000 7f805a3e4b80 1 client.26270 shutdown 2024-03-14T09:57:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.820+0000 7f805a3e4b80 1 -- 172.21.15.78:0/2609420068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 msgr2=0x56432cc333c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.820+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/2609420068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc333c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x56432cc2c060 tx=0x7f802c009380 comp rx=0 tx=0).stop 2024-03-14T09:57:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.820+0000 7f805a3e4b80 1 -- 172.21.15.78:0/2609420068 shutdown_connections 2024-03-14T09:57:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.820+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/2609420068 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56432cc2dec0 0x56432cc33900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.820+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/2609420068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56432cc23680 0x56432cc32e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.820+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/2609420068 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56432cc98700 0x56432cc9aaf0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.820+0000 7f805a3e4b80 1 --2- 172.21.15.78:0/2609420068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56432cc23fa0 0x56432cc333c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.820+0000 7f8029600700 0 client.26270 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.820+0000 7f805a3e4b80 1 -- 172.21.15.78:0/2609420068 >> 172.21.15.78:0/2609420068 conn(0x56432cb99220 msgr2=0x56432cc29490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.820+0000 7f805a3e4b80 1 -- 172.21.15.78:0/2609420068 shutdown_connections 2024-03-14T09:57:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.820+0000 7f805a3e4b80 1 -- 172.21.15.78:0/2609420068 wait complete. 2024-03-14T09:57:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.821+0000 7f805a3e4b80 20 client.26270 trim_cache size 0 max 16384 2024-03-14T09:57:20.824 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:57:20.824 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T09:57:20.824 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T09:57:20.824 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:57:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.821+0000 7f8046800700 1 -- 172.21.15.78:0/2155964270 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f80240a0c00 msgr2=0x7f80240a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.821+0000 7f8046800700 1 --2- 172.21.15.78:0/2155964270 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f80240a0c00 0x7f80240a30c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8038009fd0 tx=0x7f8038009920 comp rx=0 tx=0).stop 2024-03-14T09:57:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.821+0000 7f8046800700 1 -- 172.21.15.78:0/2155964270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8040106040 msgr2=0x7f804019f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.821+0000 7f8046800700 1 --2- 172.21.15.78:0/2155964270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8040106040 0x7f804019f9b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f802c000c00 tx=0x7f802c00bfb0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.821+0000 7f8046800700 1 -- 172.21.15.78:0/2155964270 shutdown_connections 2024-03-14T09:57:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.821+0000 7f8046800700 1 --2- 172.21.15.78:0/2155964270 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f80240a0c00 0x7f80240a30c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.821+0000 7f8046800700 1 --2- 172.21.15.78:0/2155964270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8040199600 0x7f8040199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.821+0000 7f8046800700 1 --2- 172.21.15.78:0/2155964270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8040100f20 0x7f804019f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.821+0000 7f8046800700 1 --2- 172.21.15.78:0/2155964270 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8040106040 0x7f804019f9b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.821+0000 7f8046800700 1 -- 172.21.15.78:0/2155964270 >> 172.21.15.78:0/2155964270 conn(0x7f80400fc7a0 msgr2=0x7f80400feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.821+0000 7f8046800700 1 -- 172.21.15.78:0/2155964270 shutdown_connections 2024-03-14T09:57:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:20.822+0000 7f8046800700 1 -- 172.21.15.78:0/2155964270 wait complete. 2024-03-14T09:57:20.838 INFO:tasks.workunit.client.0.smithi078.stdout:50000 mapwrite 0xd9535 thru 0xdf124 (0x5bf0 bytes) 2024-03-14T09:57:20.841 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T09:57:20.841 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:57:20.846 INFO:tasks.workunit.client.0.smithi078.stdout:All 50000 operations completed A-OK! 2024-03-14T09:57:20.850 INFO:tasks.workunit.client.0.smithi078.stderr:+ ./fsx -z 10MB -N 50000 -p 10000 -l 10485760 2024-03-14T09:57:20.852 INFO:tasks.workunit.client.0.smithi078.stdout:Seed set to 1 2024-03-14T09:57:20.899 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:20 smithi078 ceph-mon[38143]: pgmap v98: 129 pgs: 129 active+clean; 112 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 265 op/s 2024-03-14T09:57:20.994 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support fallocate mode 0, disabling! 2024-03-14T09:57:20.995 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support fallocate mode FALLOC_FL_KEEP_SIZE, disabling! 2024-03-14T09:57:20.996 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support fallocate mode FALLOC_FL_COLLAPSE_RANGE, disabling! 2024-03-14T09:57:20.996 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support fallocate mode FALLOC_FL_INSERT_RANGE, disabling! 2024-03-14T09:57:20.996 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support clone range, disabling! 2024-03-14T09:57:20.998 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support dedupe range, disabling! 2024-03-14T09:57:21.000 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0x5a0dfb 2024-03-14T09:57:21.011 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0x5c415c 2024-03-14T09:57:21.012 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0x9c27b4 2024-03-14T09:57:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.047+0000 7efe06200700 1 -- 172.21.15.78:0/1181196289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe00100e20 msgr2=0x7efe00101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.047+0000 7efe06200700 1 --2- 172.21.15.78:0/1181196289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe00100e20 0x7efe00101200 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009990 tx=0x7efdf0009ca0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.047+0000 7efe06200700 1 -- 172.21.15.78:0/1181196289 shutdown_connections 2024-03-14T09:57:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.047+0000 7efe06200700 1 --2- 172.21.15.78:0/1181196289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe00105f40 0x7efe0010dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.047+0000 7efe06200700 1 --2- 172.21.15.78:0/1181196289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe00100e20 0x7efe00101200 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.047+0000 7efe06200700 1 --2- 172.21.15.78:0/1181196289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe001017d0 0x7efe00105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.047+0000 7efe06200700 1 -- 172.21.15.78:0/1181196289 >> 172.21.15.78:0/1181196289 conn(0x7efe000fc700 msgr2=0x7efe000feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.047+0000 7efe06200700 1 -- 172.21.15.78:0/1181196289 shutdown_connections 2024-03-14T09:57:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.047+0000 7efe06200700 1 -- 172.21.15.78:0/1181196289 wait complete. 2024-03-14T09:57:21.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.048+0000 7efe06200700 1 Processor -- start 2024-03-14T09:57:21.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.049+0000 7efe06200700 1 -- start start 2024-03-14T09:57:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.049+0000 7efe06200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe00100e20 0x7efe0019f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.049+0000 7efe06200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe00105f40 0x7efe0019faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.049+0000 7efe06200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe00199690 0x7efe00199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.049+0000 7efe06200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0019a110 con 0x7efe00199690 2024-03-14T09:57:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.049+0000 7efe06200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0019a280 con 0x7efe00100e20 2024-03-14T09:57:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.049+0000 7efe06200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0019a440 con 0x7efe00105f40 2024-03-14T09:57:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.050+0000 7efdfea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe00105f40 0x7efe0019faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.050+0000 7efdff400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe00100e20 0x7efe0019f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.050+0000 7efdfea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe00105f40 0x7efe0019faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47144/0 (socket says 172.21.15.78:47144) 2024-03-14T09:57:21.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.050+0000 7efdffe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe00199690 0x7efe00199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.050+0000 7efdff400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe00100e20 0x7efe0019f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:52390/0 (socket says 172.21.15.78:52390) 2024-03-14T09:57:21.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.050+0000 7efdfea00700 1 -- 172.21.15.78:0/30519917 learned_addr learned my addr 172.21.15.78:0/30519917 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.050+0000 7efdfea00700 1 -- 172.21.15.78:0/30519917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe00100e20 msgr2=0x7efe0019f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.050+0000 7efdfea00700 1 --2- 172.21.15.78:0/30519917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe00100e20 0x7efe0019f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.050+0000 7efdfea00700 1 -- 172.21.15.78:0/30519917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe00199690 msgr2=0x7efe00199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.050+0000 7efdfea00700 1 --2- 172.21.15.78:0/30519917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe00199690 0x7efe00199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.050+0000 7efdfea00700 1 -- 172.21.15.78:0/30519917 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf00095f0 con 0x7efe00105f40 2024-03-14T09:57:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.050+0000 7efdff400700 1 --2- 172.21.15.78:0/30519917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe00100e20 0x7efe0019f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.050+0000 7efdfea00700 1 --2- 172.21.15.78:0/30519917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe00105f40 0x7efe0019faa0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efdec00e970 tx=0x7efdec00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.050+0000 7efdf7e00700 1 -- 172.21.15.78:0/30519917 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec009790 con 0x7efe00105f40 2024-03-14T09:57:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.051+0000 7efe06200700 1 -- 172.21.15.78:0/30519917 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe001a6430 con 0x7efe00105f40 2024-03-14T09:57:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.051+0000 7efdf7e00700 1 -- 172.21.15.78:0/30519917 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdec00c7e0 con 0x7efe00105f40 2024-03-14T09:57:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.051+0000 7efdf7e00700 1 -- 172.21.15.78:0/30519917 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec0054e0 con 0x7efe00105f40 2024-03-14T09:57:21.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.051+0000 7efe06200700 1 -- 172.21.15.78:0/30519917 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe001a6960 con 0x7efe00105f40 2024-03-14T09:57:21.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.052+0000 7efe06200700 1 -- 172.21.15.78:0/30519917 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe001022f0 con 0x7efe00105f40 2024-03-14T09:57:21.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.054+0000 7efdf7e00700 1 -- 172.21.15.78:0/30519917 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7efdec00c950 con 0x7efe00105f40 2024-03-14T09:57:21.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.055+0000 7efdf7e00700 1 --2- 172.21.15.78:0/30519917 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efde40a0c80 0x7efde40a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.055+0000 7efdf7e00700 1 -- 172.21.15.78:0/30519917 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7efdec014070 con 0x7efe00105f40 2024-03-14T09:57:21.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.055+0000 7efdff400700 1 --2- 172.21.15.78:0/30519917 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efde40a0c80 0x7efde40a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.056+0000 7efdff400700 1 --2- 172.21.15.78:0/30519917 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efde40a0c80 0x7efde40a3140 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efdf000b2e0 tx=0x7efdf0005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:21.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.060+0000 7efdf7e00700 1 -- 172.21.15.78:0/30519917 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7efdec095c50 con 0x7efe00105f40 2024-03-14T09:57:21.115 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0x9d03f4 2024-03-14T09:57:21.230 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0x9f6462 2024-03-14T09:57:21.248 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:20 smithi152 ceph-mon[48279]: pgmap v98: 129 pgs: 129 active+clean; 112 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 265 op/s 2024-03-14T09:57:21.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:20 smithi132 ceph-mon[47959]: pgmap v98: 129 pgs: 129 active+clean; 112 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 265 op/s 2024-03-14T09:57:21.374 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0x9fa7e5 2024-03-14T09:57:21.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.379+0000 7efe06200700 1 -- 172.21.15.78:0/30519917 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe001017d0 con 0x7efe00105f40 2024-03-14T09:57:21.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.380+0000 7efdf7e00700 1 -- 172.21.15.78:0/30519917 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7efdec099270 con 0x7efe00105f40 2024-03-14T09:57:21.380 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:57:21.380 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:57:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.382+0000 7efe06200700 1 -- 172.21.15.78:0/30519917 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efde40a0c80 msgr2=0x7efde40a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.382+0000 7efe06200700 1 --2- 172.21.15.78:0/30519917 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efde40a0c80 0x7efde40a3140 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efdf000b2e0 tx=0x7efdf0005fd0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.382+0000 7efe06200700 1 -- 172.21.15.78:0/30519917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe00105f40 msgr2=0x7efe0019faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.382+0000 7efe06200700 1 --2- 172.21.15.78:0/30519917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe00105f40 0x7efe0019faa0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efdec00e970 tx=0x7efdec00ec80 comp rx=0 tx=0).stop 2024-03-14T09:57:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.382+0000 7efe06200700 1 -- 172.21.15.78:0/30519917 shutdown_connections 2024-03-14T09:57:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.382+0000 7efe06200700 1 --2- 172.21.15.78:0/30519917 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efde40a0c80 0x7efde40a3140 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.382+0000 7efe06200700 1 --2- 172.21.15.78:0/30519917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe00100e20 0x7efe0019f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.382+0000 7efe06200700 1 --2- 172.21.15.78:0/30519917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe00199690 0x7efe00199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.382+0000 7efe06200700 1 --2- 172.21.15.78:0/30519917 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe00105f40 0x7efe0019faa0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.383+0000 7efe06200700 1 -- 172.21.15.78:0/30519917 >> 172.21.15.78:0/30519917 conn(0x7efe000fc700 msgr2=0x7efe000fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.383+0000 7efe06200700 1 -- 172.21.15.78:0/30519917 shutdown_connections 2024-03-14T09:57:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.383+0000 7efe06200700 1 -- 172.21.15.78:0/30519917 wait complete. 2024-03-14T09:57:21.383 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T09:57:21.403 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T09:57:21.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.683+0000 7f567d200700 1 -- 172.21.15.78:0/453627603 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5678106bf0 msgr2=0x7f5678106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.683+0000 7f567d200700 1 --2- 172.21.15.78:0/453627603 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5678106bf0 0x7f5678106fd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5668009990 tx=0x7f5668009ca0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.684+0000 7f567d200700 1 -- 172.21.15.78:0/453627603 shutdown_connections 2024-03-14T09:57:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.684+0000 7f567d200700 1 --2- 172.21.15.78:0/453627603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56781014f0 0x7f567810d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.684+0000 7f567d200700 1 --2- 172.21.15.78:0/453627603 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5678106bf0 0x7f5678106fd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.684+0000 7f567d200700 1 --2- 172.21.15.78:0/453627603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5678100b30 0x7f5678100fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.684+0000 7f567d200700 1 -- 172.21.15.78:0/453627603 >> 172.21.15.78:0/453627603 conn(0x7f56780fc620 msgr2=0x7f56780fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.684+0000 7f567d200700 1 -- 172.21.15.78:0/453627603 shutdown_connections 2024-03-14T09:57:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.684+0000 7f567d200700 1 -- 172.21.15.78:0/453627603 wait complete. 2024-03-14T09:57:21.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.685+0000 7f567d200700 1 Processor -- start 2024-03-14T09:57:21.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.685+0000 7f567d200700 1 -- start start 2024-03-14T09:57:21.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.686+0000 7f567d200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5678100b30 0x7f5678197450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.686+0000 7f567d200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56781014f0 0x7f5678197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.686+0000 7f567d200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5678106bf0 0x7f567819bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.686+0000 7f567d200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567819c370 con 0x7f5678100b30 2024-03-14T09:57:21.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.686+0000 7f567d200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567819c4e0 con 0x7f56781014f0 2024-03-14T09:57:21.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.686+0000 7f567d200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567819c6a0 con 0x7f5678106bf0 2024-03-14T09:57:21.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.686+0000 7f5677400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5678100b30 0x7f5678197450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.686+0000 7f5677400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5678100b30 0x7f5678197450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57144/0 (socket says 172.21.15.78:57144) 2024-03-14T09:57:21.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.686+0000 7f5677400700 1 -- 172.21.15.78:0/3299698550 learned_addr learned my addr 172.21.15.78:0/3299698550 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:21.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.686+0000 7f5676a00700 1 --2- 172.21.15.78:0/3299698550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56781014f0 0x7f5678197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.687+0000 7f5677e00700 1 --2- 172.21.15.78:0/3299698550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5678106bf0 0x7f567819bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.687+0000 7f5676a00700 1 -- 172.21.15.78:0/3299698550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5678106bf0 msgr2=0x7f567819bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:21.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.687+0000 7f5676a00700 1 --2- 172.21.15.78:0/3299698550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5678106bf0 0x7f567819bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.687+0000 7f5676a00700 1 -- 172.21.15.78:0/3299698550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5678100b30 msgr2=0x7f5678197450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:21.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.687+0000 7f5676a00700 1 --2- 172.21.15.78:0/3299698550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5678100b30 0x7f5678197450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.687+0000 7f5676a00700 1 -- 172.21.15.78:0/3299698550 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56680095f0 con 0x7f56781014f0 2024-03-14T09:57:21.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.687+0000 7f5676a00700 1 --2- 172.21.15.78:0/3299698550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56781014f0 0x7f5678197990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f566400e9f0 tx=0x7f566400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:21.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.687+0000 7f566fe00700 1 -- 172.21.15.78:0/3299698550 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566400c9c0 con 0x7f56781014f0 2024-03-14T09:57:21.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.687+0000 7f566fe00700 1 -- 172.21.15.78:0/3299698550 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5664010430 con 0x7f56781014f0 2024-03-14T09:57:21.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.687+0000 7f567d200700 1 -- 172.21.15.78:0/3299698550 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f567819c980 con 0x7f56781014f0 2024-03-14T09:57:21.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.688+0000 7f566fe00700 1 -- 172.21.15.78:0/3299698550 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5664018680 con 0x7f56781014f0 2024-03-14T09:57:21.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.688+0000 7f567d200700 1 -- 172.21.15.78:0/3299698550 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56780731c0 con 0x7f56781014f0 2024-03-14T09:57:21.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.691+0000 7f566fe00700 1 -- 172.21.15.78:0/3299698550 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f56640187e0 con 0x7f56781014f0 2024-03-14T09:57:21.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.692+0000 7f566fe00700 1 --2- 172.21.15.78:0/3299698550 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f565c0a0c00 0x7f565c0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.692+0000 7f5692360b80 1 Processor -- start 2024-03-14T09:57:21.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.692+0000 7f566fe00700 1 -- 172.21.15.78:0/3299698550 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f5664014070 con 0x7f56781014f0 2024-03-14T09:57:21.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.692+0000 7f5677400700 1 --2- 172.21.15.78:0/3299698550 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f565c0a0c00 0x7f565c0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.692+0000 7f5692360b80 1 -- start start 2024-03-14T09:57:21.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.693+0000 7f5677400700 1 --2- 172.21.15.78:0/3299698550 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f565c0a0c00 0x7f565c0a30c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5668006010 tx=0x7f566800b280 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:21.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.693+0000 7f5692360b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50b860 0x56411f50bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.693+0000 7f5692360b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f516af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.693+0000 7f5692360b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f517030 0x56411f519420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.693+0000 7f5692360b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411f465990 con 0x56411f50b860 2024-03-14T09:57:21.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.693+0000 7f5692360b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411f463350 con 0x56411f517030 2024-03-14T09:57:21.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.693+0000 7f5692360b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411f45ec40 con 0x56411f50c180 2024-03-14T09:57:21.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.693+0000 7f5677400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50b860 0x56411f50bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.693+0000 7f5677400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50b860 0x56411f50bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57152/0 (socket says 172.21.15.78:57152) 2024-03-14T09:57:21.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.693+0000 7f5677400700 1 -- 172.21.15.78:0/919040599 learned_addr learned my addr 172.21.15.78:0/919040599 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:21.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.693+0000 7f5676a00700 1 --2- 172.21.15.78:0/919040599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f516af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.693+0000 7f5677e00700 1 --2- 172.21.15.78:0/919040599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f517030 0x56411f519420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.694+0000 7f5676a00700 1 -- 172.21.15.78:0/919040599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f517030 msgr2=0x56411f519420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:21.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.694+0000 7f5676a00700 1 --2- 172.21.15.78:0/919040599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f517030 0x56411f519420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.694+0000 7f5676a00700 1 -- 172.21.15.78:0/919040599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50b860 msgr2=0x56411f50bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:21.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.694+0000 7f5676a00700 1 --2- 172.21.15.78:0/919040599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50b860 0x56411f50bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.694+0000 7f5676a00700 1 -- 172.21.15.78:0/919040599 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56411f2e03b0 con 0x56411f50c180 2024-03-14T09:57:21.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.694+0000 7f5676a00700 1 --2- 172.21.15.78:0/919040599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f516af0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f56640c9c80 tx=0x7f5664099260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=540f6c8530ff0bea server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:21.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.694+0000 7f566cc00700 1 -- 172.21.15.78:0/919040599 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566409d6b0 con 0x56411f50c180 2024-03-14T09:57:21.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.694+0000 7f566cc00700 1 -- 172.21.15.78:0/919040599 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56640a1d00 con 0x56411f50c180 2024-03-14T09:57:21.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.694+0000 7f566cc00700 1 -- 172.21.15.78:0/919040599 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5664098d70 con 0x56411f50c180 2024-03-14T09:57:21.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.695+0000 7f5692360b80 1 -- 172.21.15.78:0/919040599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 msgr2=0x56411f516af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:21.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.695+0000 7f5692360b80 1 --2- 172.21.15.78:0/919040599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f516af0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f56640c9c80 tx=0x7f5664099260 comp rx=0 tx=0).stop 2024-03-14T09:57:21.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.695+0000 7f5692360b80 1 -- 172.21.15.78:0/919040599 shutdown_connections 2024-03-14T09:57:21.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.696+0000 7f5692360b80 1 --2- 172.21.15.78:0/919040599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f517030 0x56411f519420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.696+0000 7f5692360b80 1 --2- 172.21.15.78:0/919040599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50b860 0x56411f50bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.696+0000 7f5692360b80 1 --2- 172.21.15.78:0/919040599 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f516af0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.696+0000 7f5692360b80 1 -- 172.21.15.78:0/919040599 >> 172.21.15.78:0/919040599 conn(0x56411f482220 msgr2=0x56411f507c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:21.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.696+0000 7f5692360b80 1 -- 172.21.15.78:0/919040599 shutdown_connections 2024-03-14T09:57:21.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.696+0000 7f5692360b80 1 -- 172.21.15.78:0/919040599 wait complete. 2024-03-14T09:57:21.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.696+0000 7f5692360b80 1 -- start start 2024-03-14T09:57:21.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.697+0000 7f5692360b80 1 Processor -- start 2024-03-14T09:57:21.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.697+0000 7f5692360b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50b860 0x56411f516af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.697+0000 7f5692360b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f513ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.697+0000 7f5692360b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f517030 0x56411f514620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.697+0000 7f5692360b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411f3fad80 con 0x56411f50b860 2024-03-14T09:57:21.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.697+0000 7f5692360b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411f3faa20 con 0x56411f517030 2024-03-14T09:57:21.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.697+0000 7f5692360b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411f2e26a0 con 0x56411f50c180 2024-03-14T09:57:21.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.698+0000 7f5692360b80 10 client.0 _ll_register_callbacks cb 0x56411f2663a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:21.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.698+0000 7f5676a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f513ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.698+0000 7f5677400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50b860 0x56411f516af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.698+0000 7f5676a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f513ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47172/0 (socket says 172.21.15.78:47172) 2024-03-14T09:57:21.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.698+0000 7f5676a00700 1 -- 172.21.15.78:0/872953581 learned_addr learned my addr 172.21.15.78:0/872953581 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:21.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.698+0000 7f5677e00700 1 --2- 172.21.15.78:0/872953581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f517030 0x56411f514620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.698+0000 7f5676a00700 1 -- 172.21.15.78:0/872953581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f517030 msgr2=0x56411f514620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:21.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.698+0000 7f5676a00700 1 --2- 172.21.15.78:0/872953581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f517030 0x56411f514620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.698+0000 7f5676a00700 1 -- 172.21.15.78:0/872953581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50b860 msgr2=0x56411f516af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:21.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.698+0000 7f5676a00700 1 --2- 172.21.15.78:0/872953581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50b860 0x56411f516af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.698+0000 7f5676a00700 1 -- 172.21.15.78:0/872953581 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56780731c0 con 0x56411f50c180 2024-03-14T09:57:21.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.698+0000 7f5676a00700 1 --2- 172.21.15.78:0/872953581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f513ee0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f56640a17d0 tx=0x7f566400e940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe0d0cf89d6d01b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:21.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.698+0000 7f5661600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:57:21.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.699+0000 7f5661600700 1 -- 172.21.15.78:0/872953581 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56640b2220 con 0x56411f50c180 2024-03-14T09:57:21.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.699+0000 7f5661600700 1 -- 172.21.15.78:0/872953581 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f566409d6b0 con 0x56411f50c180 2024-03-14T09:57:21.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.699+0000 7f5692360b80 1 -- 172.21.15.78:0/872953581 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56411f4ff720 con 0x56411f50c180 2024-03-14T09:57:21.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.699+0000 7f5661600700 1 -- 172.21.15.78:0/872953581 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5664099ae0 con 0x56411f50c180 2024-03-14T09:57:21.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.699+0000 7f5692360b80 10 client.26296 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:21.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.699+0000 7f5692360b80 1 -- 172.21.15.78:0/872953581 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56411f28cb60 con 0x56411f50c180 2024-03-14T09:57:21.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.699+0000 7f5661600700 1 -- 172.21.15.78:0/872953581 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56411f4ff720 con 0x56411f50c180 2024-03-14T09:57:21.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.699+0000 7f5661600700 1 -- 172.21.15.78:0/872953581 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f56640a1900 con 0x56411f50c180 2024-03-14T09:57:21.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.699+0000 7f5692360b80 10 client.26296 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:21.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.699+0000 7f5692360b80 10 client.26296 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:57:21.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.699+0000 7f5692360b80 20 client.26296 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:21.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.699+0000 7f5692360b80 1 --2- 172.21.15.78:0/872953581 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56411f526850 0x56411f528c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.699+0000 7f5692360b80 4 client.26296 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:57:21.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.699+0000 7f5692360b80 1 -- 172.21.15.78:0/872953581 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56411f457700 con 0x56411f526850 2024-03-14T09:57:21.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.700+0000 7f5677400700 1 --2- 172.21.15.78:0/872953581 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56411f526850 0x56411f528c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.700+0000 7f5661600700 10 client.26296 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:57:21.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.700+0000 7f5677400700 1 --2- 172.21.15.78:0/872953581 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56411f526850 0x56411f528c40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d02fcd0923c045a3 server_cookie=3881d2b1c458bab7 in_seq=0 out_seq=0 2024-03-14T09:57:21.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.701+0000 7f5661600700 1 -- 172.21.15.78:0/872953581 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56411f457700 con 0x56411f526850 2024-03-14T09:57:21.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.701+0000 7f5661600700 10 client.26296 handle_command_reply: tid=0 2024-03-14T09:57:21.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.701+0000 7f5692360b80 1 client.26296 shutdown 2024-03-14T09:57:21.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.701+0000 7f5692360b80 1 -- 172.21.15.78:0/872953581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 msgr2=0x56411f513ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:21.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.701+0000 7f5692360b80 1 --2- 172.21.15.78:0/872953581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f513ee0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f56640a17d0 tx=0x7f566400e940 comp rx=0 tx=0).stop 2024-03-14T09:57:21.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.701+0000 7f5692360b80 1 -- 172.21.15.78:0/872953581 shutdown_connections 2024-03-14T09:57:21.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.701+0000 7f5692360b80 1 --2- 172.21.15.78:0/872953581 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56411f526850 0x56411f528c40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.701+0000 7f5692360b80 1 --2- 172.21.15.78:0/872953581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f517030 0x56411f514620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.701+0000 7f5661600700 0 client.26296 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:57:21.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.701+0000 7f5692360b80 1 --2- 172.21.15.78:0/872953581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50b860 0x56411f516af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.701+0000 7f5692360b80 1 --2- 172.21.15.78:0/872953581 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f513ee0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.701+0000 7f5692360b80 1 -- 172.21.15.78:0/872953581 >> 172.21.15.78:0/872953581 conn(0x56411f482220 msgr2=0x56411f507880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:21.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.701+0000 7f5692360b80 1 -- 172.21.15.78:0/872953581 shutdown_connections 2024-03-14T09:57:21.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.701+0000 7f5692360b80 1 -- 172.21.15.78:0/872953581 wait complete. 2024-03-14T09:57:21.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.702+0000 7f5692360b80 20 client.26296 trim_cache size 0 max 16384 2024-03-14T09:57:21.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.723+0000 7f5692360b80 1 Processor -- start 2024-03-14T09:57:21.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.723+0000 7f5692360b80 1 -- start start 2024-03-14T09:57:21.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.723+0000 7f5692360b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f50b860 0x56411f507e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.723+0000 7f5692360b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f5145e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.723+0000 7f5692360b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f517030 0x56411f52d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.723+0000 7f5692360b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411f50cad0 con 0x56411f517030 2024-03-14T09:57:21.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.723+0000 7f5692360b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411f501d30 con 0x56411f50b860 2024-03-14T09:57:21.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.723+0000 7f5692360b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411f512460 con 0x56411f50c180 2024-03-14T09:57:21.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.724+0000 7f5677e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f517030 0x56411f52d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.724+0000 7f5676a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f5145e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.724+0000 7f5677e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f517030 0x56411f52d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57164/0 (socket says 172.21.15.78:57164) 2024-03-14T09:57:21.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.724+0000 7f5676a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f5145e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47184/0 (socket says 172.21.15.78:47184) 2024-03-14T09:57:21.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.724+0000 7f5677e00700 1 -- 172.21.15.78:0/690635890 learned_addr learned my addr 172.21.15.78:0/690635890 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:21.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.724+0000 7f5677400700 1 --2- 172.21.15.78:0/690635890 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f50b860 0x56411f507e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.724+0000 7f5676a00700 1 -- 172.21.15.78:0/690635890 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f50b860 msgr2=0x56411f507e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:21.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.724+0000 7f5676a00700 1 --2- 172.21.15.78:0/690635890 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f50b860 0x56411f507e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.724+0000 7f5676a00700 1 -- 172.21.15.78:0/690635890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f517030 msgr2=0x56411f52d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:21.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.724+0000 7f5676a00700 1 --2- 172.21.15.78:0/690635890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f517030 0x56411f52d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.724+0000 7f5676a00700 1 -- 172.21.15.78:0/690635890 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56411f424ea0 con 0x56411f50c180 2024-03-14T09:57:21.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.725+0000 7f5676a00700 1 --2- 172.21.15.78:0/690635890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f5145e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5664095d80 tx=0x7f5664010c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9187b2e901c9532c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:21.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.725+0000 7f566d600700 1 -- 172.21.15.78:0/690635890 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566409a210 con 0x56411f50c180 2024-03-14T09:57:21.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.725+0000 7f566d600700 1 -- 172.21.15.78:0/690635890 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5664095860 con 0x56411f50c180 2024-03-14T09:57:21.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.725+0000 7f566d600700 1 -- 172.21.15.78:0/690635890 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56640662a0 con 0x56411f50c180 2024-03-14T09:57:21.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.726+0000 7f5692360b80 1 -- 172.21.15.78:0/690635890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 msgr2=0x56411f5145e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:21.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.726+0000 7f5692360b80 1 --2- 172.21.15.78:0/690635890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f5145e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5664095d80 tx=0x7f5664010c60 comp rx=0 tx=0).stop 2024-03-14T09:57:21.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.726+0000 7f5692360b80 1 -- 172.21.15.78:0/690635890 shutdown_connections 2024-03-14T09:57:21.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.726+0000 7f5692360b80 1 --2- 172.21.15.78:0/690635890 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f50b860 0x56411f507e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.726+0000 7f5692360b80 1 --2- 172.21.15.78:0/690635890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f517030 0x56411f52d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.726+0000 7f5692360b80 1 --2- 172.21.15.78:0/690635890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50c180 0x56411f5145e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.726+0000 7f5692360b80 1 -- 172.21.15.78:0/690635890 >> 172.21.15.78:0/690635890 conn(0x56411f482220 msgr2=0x56411f503680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:21.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.726+0000 7f5692360b80 1 -- 172.21.15.78:0/690635890 shutdown_connections 2024-03-14T09:57:21.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.726+0000 7f5692360b80 1 -- 172.21.15.78:0/690635890 wait complete. 2024-03-14T09:57:21.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.726+0000 7f5692360b80 1 -- start start 2024-03-14T09:57:21.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.726+0000 7f5692360b80 1 Processor -- start 2024-03-14T09:57:21.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5692360b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50b860 0x56411f51bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5692360b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50c180 0x56411f51c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5692360b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f517030 0x56411f577340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5692360b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411f51c9f0 con 0x56411f50c180 2024-03-14T09:57:21.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5692360b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411f577880 con 0x56411f517030 2024-03-14T09:57:21.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5692360b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411f5779c0 con 0x56411f50b860 2024-03-14T09:57:21.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5692360b80 10 client.0 _ll_register_callbacks cb 0x7f5668005e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:21.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5676a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50c180 0x56411f51c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5677400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50b860 0x56411f51bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5677e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f517030 0x56411f577340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5677400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50b860 0x56411f51bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47200/0 (socket says 172.21.15.78:47200) 2024-03-14T09:57:21.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5676a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50c180 0x56411f51c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57170/0 (socket says 172.21.15.78:57170) 2024-03-14T09:57:21.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5677400700 1 -- 172.21.15.78:0/530472923 learned_addr learned my addr 172.21.15.78:0/530472923 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:21.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5676a00700 1 -- 172.21.15.78:0/530472923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50b860 msgr2=0x56411f51bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:21.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5676a00700 1 --2- 172.21.15.78:0/530472923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50b860 0x56411f51bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5676a00700 1 -- 172.21.15.78:0/530472923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f517030 msgr2=0x56411f577340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:21.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5676a00700 1 --2- 172.21.15.78:0/530472923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f517030 0x56411f577340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.727+0000 7f5676a00700 1 -- 172.21.15.78:0/530472923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56411f28cb60 con 0x56411f50c180 2024-03-14T09:57:21.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.728+0000 7f5676a00700 1 --2- 172.21.15.78:0/530472923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50c180 0x56411f51c310 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x56411f50d390 tx=0x7f5664099260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc9a80181440940 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:21.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.728+0000 7f5661600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:57:21.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.728+0000 7f5661600700 1 -- 172.21.15.78:0/530472923 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566409a210 con 0x56411f50c180 2024-03-14T09:57:21.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.728+0000 7f5661600700 1 -- 172.21.15.78:0/530472923 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f5664010660 con 0x56411f50c180 2024-03-14T09:57:21.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.728+0000 7f5692360b80 1 -- 172.21.15.78:0/530472923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f56640a1d00 con 0x56411f50c180 2024-03-14T09:57:21.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.728+0000 7f5661600700 1 -- 172.21.15.78:0/530472923 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56640a1a80 con 0x56411f50c180 2024-03-14T09:57:21.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.728+0000 7f5692360b80 10 client.16260 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:21.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.728+0000 7f5692360b80 1 -- 172.21.15.78:0/530472923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56411f2e4cd0 con 0x56411f50c180 2024-03-14T09:57:21.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.729+0000 7f5661600700 1 -- 172.21.15.78:0/530472923 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56640a1d00 con 0x56411f50c180 2024-03-14T09:57:21.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.729+0000 7f5661600700 1 -- 172.21.15.78:0/530472923 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5664099ae0 con 0x56411f50c180 2024-03-14T09:57:21.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.729+0000 7f5692360b80 10 client.16260 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:21.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.729+0000 7f5692360b80 10 client.16260 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:57:21.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.729+0000 7f5692360b80 20 client.16260 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:21.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.729+0000 7f5692360b80 1 --2- 172.21.15.78:0/530472923 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56411f578cb0 0x56411f57b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:21.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.729+0000 7f5692360b80 4 client.16260 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:57:21.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.729+0000 7f5692360b80 1 -- 172.21.15.78:0/530472923 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56411f2adcf0 con 0x56411f578cb0 2024-03-14T09:57:21.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.729+0000 7f5677400700 1 --2- 172.21.15.78:0/530472923 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56411f578cb0 0x56411f57b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:21.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.730+0000 7f5677400700 1 --2- 172.21.15.78:0/530472923 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56411f578cb0 0x56411f57b0a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eda4165728e0c00b server_cookie=2909b09bd22f187 in_seq=0 out_seq=0 2024-03-14T09:57:21.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.730+0000 7f5661600700 10 client.16260 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:57:21.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.730+0000 7f5661600700 1 -- 172.21.15.78:0/530472923 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56411f2adcf0 con 0x56411f578cb0 2024-03-14T09:57:21.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.730+0000 7f5661600700 10 client.16260 handle_command_reply: tid=0 2024-03-14T09:57:21.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.730+0000 7f5692360b80 1 client.16260 shutdown 2024-03-14T09:57:21.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.730+0000 7f5692360b80 1 -- 172.21.15.78:0/530472923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50c180 msgr2=0x56411f51c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:21.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.730+0000 7f5692360b80 1 --2- 172.21.15.78:0/530472923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50c180 0x56411f51c310 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x56411f50d390 tx=0x7f5664099260 comp rx=0 tx=0).stop 2024-03-14T09:57:21.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.730+0000 7f5692360b80 1 -- 172.21.15.78:0/530472923 shutdown_connections 2024-03-14T09:57:21.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.730+0000 7f5692360b80 1 --2- 172.21.15.78:0/530472923 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56411f578cb0 0x56411f57b0a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.730+0000 7f5692360b80 1 --2- 172.21.15.78:0/530472923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56411f517030 0x56411f577340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.730+0000 7f5692360b80 1 --2- 172.21.15.78:0/530472923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56411f50b860 0x56411f51bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.730+0000 7f5692360b80 1 --2- 172.21.15.78:0/530472923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56411f50c180 0x56411f51c310 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.730+0000 7f5692360b80 1 -- 172.21.15.78:0/530472923 >> 172.21.15.78:0/530472923 conn(0x56411f482220 msgr2=0x56411f522a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:21.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.730+0000 7f5661600700 0 client.16260 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:57:21.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.730+0000 7f5692360b80 1 -- 172.21.15.78:0/530472923 shutdown_connections 2024-03-14T09:57:21.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.731+0000 7f5692360b80 1 -- 172.21.15.78:0/530472923 wait complete. 2024-03-14T09:57:21.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.731+0000 7f5692360b80 20 client.16260 trim_cache size 0 max 16384 2024-03-14T09:57:21.735 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:57:21.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.731+0000 7f567d200700 1 -- 172.21.15.78:0/3299698550 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f565c0a0c00 msgr2=0x7f565c0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:21.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.731+0000 7f567d200700 1 --2- 172.21.15.78:0/3299698550 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f565c0a0c00 0x7f565c0a30c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5668006010 tx=0x7f566800b280 comp rx=0 tx=0).stop 2024-03-14T09:57:21.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.731+0000 7f567d200700 1 -- 172.21.15.78:0/3299698550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56781014f0 msgr2=0x7f5678197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:21.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.731+0000 7f567d200700 1 --2- 172.21.15.78:0/3299698550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56781014f0 0x7f5678197990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f566400e9f0 tx=0x7f566400ed00 comp rx=0 tx=0).stop 2024-03-14T09:57:21.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.732+0000 7f567d200700 1 -- 172.21.15.78:0/3299698550 shutdown_connections 2024-03-14T09:57:21.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.732+0000 7f567d200700 1 --2- 172.21.15.78:0/3299698550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5678106bf0 0x7f567819bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.732+0000 7f567d200700 1 --2- 172.21.15.78:0/3299698550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5678100b30 0x7f5678197450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.732+0000 7f567d200700 1 --2- 172.21.15.78:0/3299698550 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f565c0a0c00 0x7f565c0a30c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.732+0000 7f567d200700 1 --2- 172.21.15.78:0/3299698550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f56781014f0 0x7f5678197990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:21.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.732+0000 7f567d200700 1 -- 172.21.15.78:0/3299698550 >> 172.21.15.78:0/3299698550 conn(0x7f56780fc620 msgr2=0x7f56780fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:21.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.732+0000 7f567d200700 1 -- 172.21.15.78:0/3299698550 shutdown_connections 2024-03-14T09:57:21.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:21.732+0000 7f567d200700 1 -- 172.21.15.78:0/3299698550 wait complete. 2024-03-14T09:57:21.754 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T09:57:22.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.000+0000 7fac62400700 1 -- 172.21.15.78:0/1285803448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac5c073180 msgr2=0x7fac5c073560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.000+0000 7fac62400700 1 --2- 172.21.15.78:0/1285803448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac5c073180 0x7fac5c073560 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fac4c009990 tx=0x7fac4c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.000+0000 7fac62400700 1 -- 172.21.15.78:0/1285803448 shutdown_connections 2024-03-14T09:57:22.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.000+0000 7fac62400700 1 --2- 172.21.15.78:0/1285803448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac5c10d550 0x7fac5c10f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.000+0000 7fac62400700 1 --2- 172.21.15.78:0/1285803448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac5c073180 0x7fac5c073560 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.000+0000 7fac62400700 1 --2- 172.21.15.78:0/1285803448 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac5c073aa0 0x7fac5c10d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.000+0000 7fac62400700 1 -- 172.21.15.78:0/1285803448 >> 172.21.15.78:0/1285803448 conn(0x7fac5c0fc780 msgr2=0x7fac5c0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:22.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.001+0000 7fac62400700 1 -- 172.21.15.78:0/1285803448 shutdown_connections 2024-03-14T09:57:22.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.001+0000 7fac62400700 1 -- 172.21.15.78:0/1285803448 wait complete. 2024-03-14T09:57:22.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.002+0000 7fac62400700 1 Processor -- start 2024-03-14T09:57:22.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.002+0000 7fac62400700 1 -- start start 2024-03-14T09:57:22.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.002+0000 7fac62400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac5c073180 0x7fac5c19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.002+0000 7fac62400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac5c073aa0 0x7fac5c19fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.002+0000 7fac62400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac5c10d550 0x7fac5c199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.002+0000 7fac62400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c199e50 con 0x7fac5c10d550 2024-03-14T09:57:22.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.002+0000 7fac62400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c199fc0 con 0x7fac5c073aa0 2024-03-14T09:57:22.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.002+0000 7fac62400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c19a180 con 0x7fac5c073180 2024-03-14T09:57:22.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.002+0000 7fac5b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac5c073180 0x7fac5c19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.003+0000 7fac5b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac5c073180 0x7fac5c19f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47212/0 (socket says 172.21.15.78:47212) 2024-03-14T09:57:22.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.003+0000 7fac5b400700 1 -- 172.21.15.78:0/679222015 learned_addr learned my addr 172.21.15.78:0/679222015 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:22.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.003+0000 7fac5be00700 1 --2- 172.21.15.78:0/679222015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac5c10d550 0x7fac5c199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.003+0000 7fac5aa00700 1 --2- 172.21.15.78:0/679222015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac5c073aa0 0x7fac5c19fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.003+0000 7fac5aa00700 1 -- 172.21.15.78:0/679222015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac5c073180 msgr2=0x7fac5c19f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.003+0000 7fac5aa00700 1 --2- 172.21.15.78:0/679222015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac5c073180 0x7fac5c19f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.003+0000 7fac5aa00700 1 -- 172.21.15.78:0/679222015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac5c10d550 msgr2=0x7fac5c199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.003+0000 7fac5aa00700 1 --2- 172.21.15.78:0/679222015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac5c10d550 0x7fac5c199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.003+0000 7fac5aa00700 1 -- 172.21.15.78:0/679222015 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4c0095f0 con 0x7fac5c073aa0 2024-03-14T09:57:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.003+0000 7fac5be00700 1 --2- 172.21.15.78:0/679222015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac5c10d550 0x7fac5c199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.003+0000 7fac5aa00700 1 --2- 172.21.15.78:0/679222015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac5c073aa0 0x7fac5c19fb20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fac4800e9f0 tx=0x7fac4800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.004+0000 7fac53e00700 1 -- 172.21.15.78:0/679222015 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4800c9c0 con 0x7fac5c073aa0 2024-03-14T09:57:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.004+0000 7fac53e00700 1 -- 172.21.15.78:0/679222015 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac48010430 con 0x7fac5c073aa0 2024-03-14T09:57:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.004+0000 7fac62400700 1 -- 172.21.15.78:0/679222015 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5c19a460 con 0x7fac5c073aa0 2024-03-14T09:57:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.004+0000 7fac53e00700 1 -- 172.21.15.78:0/679222015 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48018680 con 0x7fac5c073aa0 2024-03-14T09:57:22.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.004+0000 7fac62400700 1 -- 172.21.15.78:0/679222015 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5c1a6560 con 0x7fac5c073aa0 2024-03-14T09:57:22.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.007+0000 7fac53e00700 1 -- 172.21.15.78:0/679222015 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fac48010860 con 0x7fac5c073aa0 2024-03-14T09:57:22.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.008+0000 7fac53e00700 1 --2- 172.21.15.78:0/679222015 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fac400a0c40 0x7fac400a3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.008+0000 7fac53e00700 1 -- 172.21.15.78:0/679222015 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fac48014070 con 0x7fac5c073aa0 2024-03-14T09:57:22.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.008+0000 7fac75fb8b80 1 Processor -- start 2024-03-14T09:57:22.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.008+0000 7fac5b400700 1 --2- 172.21.15.78:0/679222015 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fac400a0c40 0x7fac400a3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.009+0000 7fac75fb8b80 1 -- start start 2024-03-14T09:57:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.009+0000 7fac75fb8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15680 0x560513b15a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.009+0000 7fac75fb8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 0x560513b1f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.009+0000 7fac75fb8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b1fec0 0x560513b222b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.009+0000 7fac75fb8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560513a6e990 con 0x560513b15680 2024-03-14T09:57:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.009+0000 7fac75fb8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x560513a6c350 con 0x560513b15fa0 2024-03-14T09:57:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.009+0000 7fac75fb8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560513a67c40 con 0x560513b1fec0 2024-03-14T09:57:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.009+0000 7fac5b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15680 0x560513b15a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.009+0000 7fac5b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15680 0x560513b15a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57206/0 (socket says 172.21.15.78:57206) 2024-03-14T09:57:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.009+0000 7fac5b400700 1 -- 172.21.15.78:0/2811654597 learned_addr learned my addr 172.21.15.78:0/2811654597 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:22.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.009+0000 7fac5aa00700 1 --2- 172.21.15.78:0/2811654597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 0x560513b1f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.009+0000 7fac5b400700 1 --2- 172.21.15.78:0/679222015 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fac400a0c40 0x7fac400a3100 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00b6f0 tx=0x7fac4c005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.010+0000 7fac5be00700 1 --2- 172.21.15.78:0/2811654597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b1fec0 0x560513b222b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.010+0000 7fac5aa00700 1 -- 172.21.15.78:0/2811654597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b1fec0 msgr2=0x560513b222b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.010+0000 7fac5aa00700 1 --2- 172.21.15.78:0/2811654597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b1fec0 0x560513b222b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.010+0000 7fac5aa00700 1 -- 172.21.15.78:0/2811654597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15680 msgr2=0x560513b15a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.010+0000 7fac5aa00700 1 --2- 172.21.15.78:0/2811654597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15680 0x560513b15a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.010+0000 7fac5aa00700 1 -- 172.21.15.78:0/2811654597 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605138e93b0 con 0x560513b15fa0 2024-03-14T09:57:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.010+0000 7fac5be00700 1 --2- 172.21.15.78:0/2811654597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b1fec0 0x560513b222b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:57:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.010+0000 7fac5aa00700 1 --2- 172.21.15.78:0/2811654597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 0x560513b1f980 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fac480c9cc0 tx=0x7fac480992f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff73d2b49c515f7d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.010+0000 7fac50c00700 1 -- 172.21.15.78:0/2811654597 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4809d6f0 con 0x560513b15fa0 2024-03-14T09:57:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.010+0000 7fac50c00700 1 -- 172.21.15.78:0/2811654597 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac480a1d40 con 0x560513b15fa0 2024-03-14T09:57:22.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.010+0000 7fac50c00700 1 -- 172.21.15.78:0/2811654597 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48099010 con 0x560513b15fa0 2024-03-14T09:57:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.011+0000 7fac75fb8b80 1 -- 172.21.15.78:0/2811654597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 msgr2=0x560513b1f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.011+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/2811654597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 0x560513b1f980 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fac480c9cc0 tx=0x7fac480992f0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.011+0000 7fac75fb8b80 1 -- 172.21.15.78:0/2811654597 shutdown_connections 2024-03-14T09:57:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.011+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/2811654597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b1fec0 0x560513b222b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.011+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/2811654597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15680 0x560513b15a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.011+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/2811654597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 0x560513b1f980 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.011+0000 7fac75fb8b80 1 -- 172.21.15.78:0/2811654597 >> 172.21.15.78:0/2811654597 conn(0x560513a8b220 msgr2=0x560513b10aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.011+0000 7fac75fb8b80 1 -- 172.21.15.78:0/2811654597 shutdown_connections 2024-03-14T09:57:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.011+0000 7fac75fb8b80 1 -- 172.21.15.78:0/2811654597 wait complete. 2024-03-14T09:57:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.012+0000 7fac75fb8b80 1 -- start start 2024-03-14T09:57:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.012+0000 7fac75fb8b80 1 Processor -- start 2024-03-14T09:57:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac75fb8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b15680 0x560513b1ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac75fb8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 0x560513b1d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac75fb8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b1fec0 0x560513b1d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac75fb8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560513a03d80 con 0x560513b1fec0 2024-03-14T09:57:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac75fb8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x560513a03a20 con 0x560513b15fa0 2024-03-14T09:57:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac75fb8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605138eb6a0 con 0x560513b15680 2024-03-14T09:57:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac75fb8b80 10 client.0 _ll_register_callbacks cb 0x56051386f3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac5b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b15680 0x560513b1ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac5b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b15680 0x560513b1ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47218/0 (socket says 172.21.15.78:47218) 2024-03-14T09:57:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac5b400700 1 -- 172.21.15.78:0/763801436 learned_addr learned my addr 172.21.15.78:0/763801436 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac5aa00700 1 --2- 172.21.15.78:0/763801436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 0x560513b1d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac5be00700 1 --2- 172.21.15.78:0/763801436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b1fec0 0x560513b1d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac5b400700 1 -- 172.21.15.78:0/763801436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 msgr2=0x560513b1d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac5b400700 1 --2- 172.21.15.78:0/763801436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 0x560513b1d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac5b400700 1 -- 172.21.15.78:0/763801436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b1fec0 msgr2=0x560513b1d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac5b400700 1 --2- 172.21.15.78:0/763801436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b1fec0 0x560513b1d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac5b400700 1 -- 172.21.15.78:0/763801436 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5c1a6560 con 0x560513b15680 2024-03-14T09:57:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.013+0000 7fac5aa00700 1 --2- 172.21.15.78:0/763801436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 0x560513b1d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.014+0000 7fac5b400700 1 --2- 172.21.15.78:0/763801436 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b15680 0x560513b1ccf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00be80 tx=0x7fac4c01a050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=125f10f8f52aa779 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.014+0000 7fac45600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:57:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.014+0000 7fac45600700 1 -- 172.21.15.78:0/763801436 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c00f660 con 0x560513b15680 2024-03-14T09:57:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.014+0000 7fac45600700 1 -- 172.21.15.78:0/763801436 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fac4c00f7c0 con 0x560513b15680 2024-03-14T09:57:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.014+0000 7fac75fb8b80 1 -- 172.21.15.78:0/763801436 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560513b08720 con 0x560513b15680 2024-03-14T09:57:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.014+0000 7fac45600700 1 -- 172.21.15.78:0/763801436 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac4c00beb0 con 0x560513b15680 2024-03-14T09:57:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.014+0000 7fac75fb8b80 10 client.26311 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.014+0000 7fac75fb8b80 1 -- 172.21.15.78:0/763801436 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560513895b60 con 0x560513b15680 2024-03-14T09:57:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.014+0000 7fac45600700 1 -- 172.21.15.78:0/763801436 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560513b08720 con 0x560513b15680 2024-03-14T09:57:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.015+0000 7fac45600700 1 -- 172.21.15.78:0/763801436 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fac4c02f9b0 con 0x560513b15680 2024-03-14T09:57:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.015+0000 7fac75fb8b80 10 client.26311 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:22.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.015+0000 7fac75fb8b80 10 client.26311 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.015+0000 7fac75fb8b80 20 client.26311 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.015+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/763801436 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x560513b2fa40 0x560513b31e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.015+0000 7fac75fb8b80 4 client.26311 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.015+0000 7fac75fb8b80 1 -- 172.21.15.78:0/763801436 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560513a60700 con 0x560513b2fa40 2024-03-14T09:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.015+0000 7fac5aa00700 1 --2- 172.21.15.78:0/763801436 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x560513b2fa40 0x560513b31e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.016+0000 7fac45600700 10 client.26311 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:57:22.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.016+0000 7fac5aa00700 1 --2- 172.21.15.78:0/763801436 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x560513b2fa40 0x560513b31e60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c23d76d51996b04d server_cookie=d4146e2142466437 in_seq=0 out_seq=0 2024-03-14T09:57:22.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.016+0000 7fac45600700 1 -- 172.21.15.78:0/763801436 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x560513a60700 con 0x560513b2fa40 2024-03-14T09:57:22.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.016+0000 7fac45600700 10 client.26311 handle_command_reply: tid=0 2024-03-14T09:57:22.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.016+0000 7fac75fb8b80 1 client.26311 shutdown 2024-03-14T09:57:22.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.016+0000 7fac75fb8b80 1 -- 172.21.15.78:0/763801436 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b15680 msgr2=0x560513b1ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.016+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/763801436 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b15680 0x560513b1ccf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00be80 tx=0x7fac4c01a050 comp rx=0 tx=0).stop 2024-03-14T09:57:22.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.016+0000 7fac75fb8b80 1 -- 172.21.15.78:0/763801436 shutdown_connections 2024-03-14T09:57:22.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.016+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/763801436 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x560513b2fa40 0x560513b31e60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.016+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/763801436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b1fec0 0x560513b1d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.016+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/763801436 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b15680 0x560513b1ccf0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.016+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/763801436 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 0x560513b1d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.016+0000 7fac75fb8b80 1 -- 172.21.15.78:0/763801436 >> 172.21.15.78:0/763801436 conn(0x560513a8b220 msgr2=0x560513b10690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:22.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.017+0000 7fac45600700 0 client.26311 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:57:22.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.017+0000 7fac75fb8b80 1 -- 172.21.15.78:0/763801436 shutdown_connections 2024-03-14T09:57:22.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.017+0000 7fac75fb8b80 1 -- 172.21.15.78:0/763801436 wait complete. 2024-03-14T09:57:22.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.017+0000 7fac75fb8b80 20 client.26311 trim_cache size 0 max 16384 2024-03-14T09:57:22.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.032+0000 7fac75fb8b80 1 Processor -- start 2024-03-14T09:57:22.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.032+0000 7fac75fb8b80 1 -- start start 2024-03-14T09:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.032+0000 7fac75fb8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b15680 0x560513b10c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.032+0000 7fac75fb8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15fa0 0x560513b222b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.032+0000 7fac75fb8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b1fec0 0x560513b1d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.032+0000 7fac75fb8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560513b36b80 con 0x560513b15fa0 2024-03-14T09:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.032+0000 7fac75fb8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x560513b1b240 con 0x560513b1fec0 2024-03-14T09:57:22.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.032+0000 7fac75fb8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560513b18b30 con 0x560513b15680 2024-03-14T09:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.033+0000 7fac5b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b15680 0x560513b10c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.033+0000 7fac5b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b15680 0x560513b10c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47226/0 (socket says 172.21.15.78:47226) 2024-03-14T09:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.033+0000 7fac5b400700 1 -- 172.21.15.78:0/2976849854 learned_addr learned my addr 172.21.15.78:0/2976849854 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:22.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.033+0000 7fac5be00700 1 --2- 172.21.15.78:0/2976849854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b1fec0 0x560513b1d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.033+0000 7fac5aa00700 1 --2- 172.21.15.78:0/2976849854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15fa0 0x560513b222b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.033+0000 7fac5b400700 1 -- 172.21.15.78:0/2976849854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b1fec0 msgr2=0x560513b1d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.033+0000 7fac5b400700 1 --2- 172.21.15.78:0/2976849854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b1fec0 0x560513b1d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.033+0000 7fac5b400700 1 -- 172.21.15.78:0/2976849854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15fa0 msgr2=0x560513b222b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.033+0000 7fac5b400700 1 --2- 172.21.15.78:0/2976849854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15fa0 0x560513b222b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.033+0000 7fac5b400700 1 -- 172.21.15.78:0/2976849854 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560513a2dea0 con 0x560513b15680 2024-03-14T09:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.033+0000 7fac5be00700 1 --2- 172.21.15.78:0/2976849854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b1fec0 0x560513b1d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.034+0000 7fac5b400700 1 --2- 172.21.15.78:0/2976849854 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b15680 0x560513b10c30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0098e0 tx=0x7fac4c00b320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4298817bca58eb1b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.034+0000 7fac51600700 1 -- 172.21.15.78:0/2976849854 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c02f610 con 0x560513b15680 2024-03-14T09:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.034+0000 7fac51600700 1 -- 172.21.15.78:0/2976849854 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac4c01e040 con 0x560513b15680 2024-03-14T09:57:22.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.034+0000 7fac51600700 1 -- 172.21.15.78:0/2976849854 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c0249b0 con 0x560513b15680 2024-03-14T09:57:22.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.035+0000 7fac75fb8b80 1 -- 172.21.15.78:0/2976849854 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b15680 msgr2=0x560513b10c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.035+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/2976849854 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b15680 0x560513b10c30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0098e0 tx=0x7fac4c00b320 comp rx=0 tx=0).stop 2024-03-14T09:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.035+0000 7fac75fb8b80 1 -- 172.21.15.78:0/2976849854 shutdown_connections 2024-03-14T09:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.035+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/2976849854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b1fec0 0x560513b1d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.035+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/2976849854 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b15680 0x560513b10c30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.035+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/2976849854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15fa0 0x560513b222b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.035+0000 7fac75fb8b80 1 -- 172.21.15.78:0/2976849854 >> 172.21.15.78:0/2976849854 conn(0x560513a8b220 msgr2=0x560513b17fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.035+0000 7fac75fb8b80 1 -- 172.21.15.78:0/2976849854 shutdown_connections 2024-03-14T09:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.035+0000 7fac75fb8b80 1 -- 172.21.15.78:0/2976849854 wait complete. 2024-03-14T09:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.036+0000 7fac75fb8b80 1 -- start start 2024-03-14T09:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.036+0000 7fac75fb8b80 1 Processor -- start 2024-03-14T09:57:22.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.036+0000 7fac75fb8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15680 0x560513b24a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.036+0000 7fac75fb8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 0x560513b24fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.036+0000 7fac75fb8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b1fec0 0x560513b254f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.036+0000 7fac75fb8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560513b805a0 con 0x560513b15680 2024-03-14T09:57:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.036+0000 7fac75fb8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x560513b806e0 con 0x560513b15fa0 2024-03-14T09:57:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.036+0000 7fac75fb8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560513b808a0 con 0x560513b1fec0 2024-03-14T09:57:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.036+0000 7fac75fb8b80 10 client.0 _ll_register_callbacks cb 0x7fac4c00ba60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.037+0000 7fac5b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15680 0x560513b24a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.037+0000 7fac5be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b1fec0 0x560513b254f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.037+0000 7fac5b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15680 0x560513b24a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57232/0 (socket says 172.21.15.78:57232) 2024-03-14T09:57:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.037+0000 7fac5b400700 1 -- 172.21.15.78:0/1964712865 learned_addr learned my addr 172.21.15.78:0/1964712865 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.037+0000 7fac5aa00700 1 --2- 172.21.15.78:0/1964712865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 0x560513b24fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.037+0000 7fac5b400700 1 -- 172.21.15.78:0/1964712865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b1fec0 msgr2=0x560513b254f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.037+0000 7fac5b400700 1 --2- 172.21.15.78:0/1964712865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b1fec0 0x560513b254f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.037+0000 7fac5b400700 1 -- 172.21.15.78:0/1964712865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 msgr2=0x560513b24fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.037+0000 7fac5b400700 1 --2- 172.21.15.78:0/1964712865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 0x560513b24fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.037+0000 7fac5b400700 1 -- 172.21.15.78:0/1964712865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560513895b60 con 0x560513b15680 2024-03-14T09:57:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.037+0000 7fac5be00700 1 --2- 172.21.15.78:0/1964712865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b1fec0 0x560513b254f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.037+0000 7fac5aa00700 1 --2- 172.21.15.78:0/1964712865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 0x560513b24fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:57:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.037+0000 7fac5b400700 1 --2- 172.21.15.78:0/1964712865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15680 0x560513b24a70 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fac4c005ac0 tx=0x7fac4c02f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76eff655ac6cf91b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.037+0000 7fac45600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:57:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.038+0000 7fac45600700 1 -- 172.21.15.78:0/1964712865 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c00f660 con 0x560513b15680 2024-03-14T09:57:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.038+0000 7fac75fb8b80 1 -- 172.21.15.78:0/1964712865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fac480a1d40 con 0x560513b15680 2024-03-14T09:57:22.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.038+0000 7fac45600700 1 -- 172.21.15.78:0/1964712865 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fac4c00be80 con 0x560513b15680 2024-03-14T09:57:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.038+0000 7fac45600700 1 -- 172.21.15.78:0/1964712865 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac4c0056f0 con 0x560513b15680 2024-03-14T09:57:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.038+0000 7fac75fb8b80 10 client.16278 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.038+0000 7fac75fb8b80 1 -- 172.21.15.78:0/1964712865 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5605138edcd0 con 0x560513b15680 2024-03-14T09:57:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.038+0000 7fac45600700 1 -- 172.21.15.78:0/1964712865 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac480a1d40 con 0x560513b15680 2024-03-14T09:57:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.039+0000 7fac45600700 1 -- 172.21.15.78:0/1964712865 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fac4c04a870 con 0x560513b15680 2024-03-14T09:57:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.039+0000 7fac75fb8b80 10 client.16278 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.039+0000 7fac75fb8b80 10 client.16278 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:57:22.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.039+0000 7fac75fb8b80 20 client.16278 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:22.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.039+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/1964712865 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x560513b8a470 0x560513b8c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.039+0000 7fac75fb8b80 4 client.16278 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:57:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.039+0000 7fac75fb8b80 1 -- 172.21.15.78:0/1964712865 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5605138b6cf0 con 0x560513b8a470 2024-03-14T09:57:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.039+0000 7fac5aa00700 1 --2- 172.21.15.78:0/1964712865 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x560513b8a470 0x560513b8c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.040+0000 7fac5aa00700 1 --2- 172.21.15.78:0/1964712865 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x560513b8a470 0x560513b8c860 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ba8f0a149f8475a0 server_cookie=8d1862d4c3a1ab32 in_seq=0 out_seq=0 2024-03-14T09:57:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.040+0000 7fac45600700 10 client.16278 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:57:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.040+0000 7fac45600700 1 -- 172.21.15.78:0/1964712865 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5605138b6cf0 con 0x560513b8a470 2024-03-14T09:57:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.040+0000 7fac45600700 10 client.16278 handle_command_reply: tid=0 2024-03-14T09:57:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.040+0000 7fac75fb8b80 1 client.16278 shutdown 2024-03-14T09:57:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.040+0000 7fac75fb8b80 1 -- 172.21.15.78:0/1964712865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15680 msgr2=0x560513b24a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.040+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/1964712865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15680 0x560513b24a70 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fac4c005ac0 tx=0x7fac4c02f6c0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.040+0000 7fac75fb8b80 1 -- 172.21.15.78:0/1964712865 shutdown_connections 2024-03-14T09:57:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.040+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/1964712865 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x560513b8a470 0x560513b8c860 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.040+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/1964712865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560513b1fec0 0x560513b254f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.040+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/1964712865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560513b15680 0x560513b24a70 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.040+0000 7fac75fb8b80 1 --2- 172.21.15.78:0/1964712865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560513b15fa0 0x560513b24fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.040+0000 7fac75fb8b80 1 -- 172.21.15.78:0/1964712865 >> 172.21.15.78:0/1964712865 conn(0x560513a8b220 msgr2=0x560513b1b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.041+0000 7fac45600700 0 client.16278 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:57:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.041+0000 7fac75fb8b80 1 -- 172.21.15.78:0/1964712865 shutdown_connections 2024-03-14T09:57:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.041+0000 7fac75fb8b80 1 -- 172.21.15.78:0/1964712865 wait complete. 2024-03-14T09:57:22.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.042+0000 7fac75fb8b80 20 client.16278 trim_cache size 0 max 16384 2024-03-14T09:57:22.043 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:57:22.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.042+0000 7fac62400700 1 -- 172.21.15.78:0/679222015 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fac400a0c40 msgr2=0x7fac400a3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.042+0000 7fac62400700 1 --2- 172.21.15.78:0/679222015 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fac400a0c40 0x7fac400a3100 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00b6f0 tx=0x7fac4c005fd0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.042+0000 7fac62400700 1 -- 172.21.15.78:0/679222015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac5c073aa0 msgr2=0x7fac5c19fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.042+0000 7fac62400700 1 --2- 172.21.15.78:0/679222015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac5c073aa0 0x7fac5c19fb20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fac4800e9f0 tx=0x7fac4800ed00 comp rx=0 tx=0).stop 2024-03-14T09:57:22.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.043+0000 7fac62400700 1 -- 172.21.15.78:0/679222015 shutdown_connections 2024-03-14T09:57:22.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.043+0000 7fac62400700 1 --2- 172.21.15.78:0/679222015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac5c10d550 0x7fac5c199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.043+0000 7fac62400700 1 --2- 172.21.15.78:0/679222015 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fac5c073180 0x7fac5c19f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.043+0000 7fac62400700 1 --2- 172.21.15.78:0/679222015 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fac400a0c40 0x7fac400a3100 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00b6f0 tx=0x7fac4c005fd0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.043+0000 7fac62400700 1 --2- 172.21.15.78:0/679222015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fac5c073aa0 0x7fac5c19fb20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.043+0000 7fac62400700 1 -- 172.21.15.78:0/679222015 >> 172.21.15.78:0/679222015 conn(0x7fac5c0fc780 msgr2=0x7fac5c0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:22.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.043+0000 7fac62400700 1 -- 172.21.15.78:0/679222015 shutdown_connections 2024-03-14T09:57:22.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.043+0000 7fac62400700 1 -- 172.21.15.78:0/679222015 wait complete. 2024-03-14T09:57:22.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:21 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/30519917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:57:22.064 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T09:57:22.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:21 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/30519917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:57:22.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:21 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/30519917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:57:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.296+0000 7f4eab200700 1 -- 172.21.15.78:0/1653532523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ea4100f30 msgr2=0x7f4ea4101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.296+0000 7f4eab200700 1 --2- 172.21.15.78:0/1653532523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ea4100f30 0x7f4ea4101310 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4e94009990 tx=0x7f4e94009ca0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.296+0000 7f4eab200700 1 -- 172.21.15.78:0/1653532523 shutdown_connections 2024-03-14T09:57:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.296+0000 7f4eab200700 1 --2- 172.21.15.78:0/1653532523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ea4100f30 0x7f4ea4101310 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.296+0000 7f4eab200700 1 --2- 172.21.15.78:0/1653532523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ea4106050 0x7f4ea410dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.296+0000 7f4eab200700 1 --2- 172.21.15.78:0/1653532523 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ea41018e0 0x7f4ea4105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.296+0000 7f4eab200700 1 -- 172.21.15.78:0/1653532523 >> 172.21.15.78:0/1653532523 conn(0x7f4ea40fc7b0 msgr2=0x7f4ea40febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:22.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.296+0000 7f4eab200700 1 -- 172.21.15.78:0/1653532523 shutdown_connections 2024-03-14T09:57:22.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.296+0000 7f4eab200700 1 -- 172.21.15.78:0/1653532523 wait complete. 2024-03-14T09:57:22.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.297+0000 7f4eab200700 1 Processor -- start 2024-03-14T09:57:22.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.298+0000 7f4eab200700 1 -- start start 2024-03-14T09:57:22.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.298+0000 7f4eab200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ea4100f30 0x7f4ea41a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.298+0000 7f4eab200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ea4106050 0x7f4ea41a3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.298+0000 7f4eab200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ea410cc30 0x7f4ea410d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.298+0000 7f4eab200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea410d730 con 0x7f4ea4106050 2024-03-14T09:57:22.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.298+0000 7f4eab200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea410d8a0 con 0x7f4ea4100f30 2024-03-14T09:57:22.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.298+0000 7f4eab200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea410da60 con 0x7f4ea410cc30 2024-03-14T09:57:22.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.299+0000 7f4ea9200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ea410cc30 0x7f4ea410d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.299+0000 7f4ea3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ea4100f30 0x7f4ea41a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.299+0000 7f4ea9200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ea410cc30 0x7f4ea410d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47254/0 (socket says 172.21.15.78:47254) 2024-03-14T09:57:22.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.299+0000 7f4ea9200700 1 -- 172.21.15.78:0/478688455 learned_addr learned my addr 172.21.15.78:0/478688455 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:22.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.299+0000 7f4ea3400700 1 --2- 172.21.15.78:0/478688455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ea4106050 0x7f4ea41a3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.299+0000 7f4ea9200700 1 -- 172.21.15.78:0/478688455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ea4100f30 msgr2=0x7f4ea41a3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.299+0000 7f4ea9200700 1 --2- 172.21.15.78:0/478688455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ea4100f30 0x7f4ea41a3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.299+0000 7f4ea9200700 1 -- 172.21.15.78:0/478688455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ea4106050 msgr2=0x7f4ea41a3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.299+0000 7f4ea9200700 1 --2- 172.21.15.78:0/478688455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ea4106050 0x7f4ea41a3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.299+0000 7f4ea9200700 1 -- 172.21.15.78:0/478688455 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e940095f0 con 0x7f4ea410cc30 2024-03-14T09:57:22.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.299+0000 7f4ea3400700 1 --2- 172.21.15.78:0/478688455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ea4106050 0x7f4ea41a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:57:22.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.299+0000 7f4ea3e00700 1 --2- 172.21.15.78:0/478688455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ea4100f30 0x7f4ea41a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:22.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.299+0000 7f4ea9200700 1 --2- 172.21.15.78:0/478688455 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ea410cc30 0x7f4ea410d0e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00ef30 tx=0x7f4e9c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:22.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.300+0000 7f4ea0c00700 1 -- 172.21.15.78:0/478688455 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c018070 con 0x7f4ea410cc30 2024-03-14T09:57:22.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.300+0000 7f4eab200700 1 -- 172.21.15.78:0/478688455 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea41aa7b0 con 0x7f4ea410cc30 2024-03-14T09:57:22.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.300+0000 7f4ea0c00700 1 -- 172.21.15.78:0/478688455 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e9c00f040 con 0x7f4ea410cc30 2024-03-14T09:57:22.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.300+0000 7f4ea0c00700 1 -- 172.21.15.78:0/478688455 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c00cdb0 con 0x7f4ea410cc30 2024-03-14T09:57:22.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.300+0000 7f4eab200700 1 -- 172.21.15.78:0/478688455 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea41aad10 con 0x7f4ea410cc30 2024-03-14T09:57:22.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.304+0000 7f4ea0c00700 1 -- 172.21.15.78:0/478688455 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f4e9c003a00 con 0x7f4ea410cc30 2024-03-14T09:57:22.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.305+0000 7f4ebec8fb80 1 Processor -- start 2024-03-14T09:57:22.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.306+0000 7f4ebec8fb80 1 -- start start 2024-03-14T09:57:22.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.306+0000 7f4ebec8fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9680 0x5591477a9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.306+0000 7f4ebec8fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477a9fa0 0x5591477b3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.306+0000 7f4ebec8fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477b3ec0 0x5591477b62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.306+0000 7f4ebec8fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559147702990 con 0x5591477b3ec0 2024-03-14T09:57:22.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.306+0000 7f4ebec8fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559147700350 con 0x5591477a9680 2024-03-14T09:57:22.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.306+0000 7f4ebec8fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591476fbc40 con 0x5591477a9fa0 2024-03-14T09:57:22.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.306+0000 7f4ea3e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477a9fa0 0x5591477b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.306+0000 7f4ea3e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477a9fa0 0x5591477b3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47266/0 (socket says 172.21.15.78:47266) 2024-03-14T09:57:22.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.306+0000 7f4ea3e00700 1 -- 172.21.15.78:0/3434176527 learned_addr learned my addr 172.21.15.78:0/3434176527 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:22.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.307+0000 7f4ea0c00700 1 --2- 172.21.15.78:0/478688455 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4e880a0c00 0x7f4e880a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.307+0000 7f4ea3400700 1 --2- 172.21.15.78:0/3434176527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9680 0x5591477a9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.307+0000 7f4ea3400700 1 --2- 172.21.15.78:0/3434176527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477b3ec0 0x5591477b62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.307+0000 7f4ea9200700 1 --2- 172.21.15.78:0/478688455 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4e880a0c00 0x7f4e880a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.307+0000 7f4ea3e00700 1 -- 172.21.15.78:0/3434176527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9680 msgr2=0x5591477a9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:22.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.307+0000 7f4ea3e00700 1 --2- 172.21.15.78:0/3434176527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9680 0x5591477a9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.307+0000 7f4ea3e00700 1 -- 172.21.15.78:0/3434176527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477b3ec0 msgr2=0x5591477b62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:22.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.307+0000 7f4ea3e00700 1 --2- 172.21.15.78:0/3434176527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477b3ec0 0x5591477b62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.307+0000 7f4ea3e00700 1 -- 172.21.15.78:0/3434176527 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55914757d3b0 con 0x5591477a9fa0 2024-03-14T09:57:22.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.307+0000 7f4ea3400700 1 --2- 172.21.15.78:0/3434176527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477b3ec0 0x5591477b62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:57:22.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.307+0000 7f4ea0c00700 1 -- 172.21.15.78:0/478688455 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f4e9c0cc300 con 0x7f4ea410cc30 2024-03-14T09:57:22.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.308+0000 7f4ea3e00700 1 --2- 172.21.15.78:0/3434176527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477a9fa0 0x5591477b3980 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4e94005ec0 tx=0x7f4e94005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d834776458465ed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:22.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.308+0000 7f4e99600700 1 -- 172.21.15.78:0/3434176527 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94027070 con 0x5591477a9fa0 2024-03-14T09:57:22.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.308+0000 7f4ea9200700 1 --2- 172.21.15.78:0/478688455 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4e880a0c00 0x7f4e880a30c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00c7a0 tx=0x7f4e9c0092e0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:22.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.308+0000 7f4e99600700 1 -- 172.21.15.78:0/3434176527 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e9401e040 con 0x5591477a9fa0 2024-03-14T09:57:22.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.308+0000 7f4e99600700 1 -- 172.21.15.78:0/3434176527 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9401d050 con 0x5591477a9fa0 2024-03-14T09:57:22.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.309+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/3434176527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477a9fa0 msgr2=0x5591477b3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.309+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/3434176527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477a9fa0 0x5591477b3980 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4e94005ec0 tx=0x7f4e94005ef0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.309+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/3434176527 shutdown_connections 2024-03-14T09:57:22.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.309+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/3434176527 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9680 0x5591477a9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.309+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/3434176527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477b3ec0 0x5591477b62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.309+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/3434176527 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477a9fa0 0x5591477b3980 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.309+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/3434176527 >> 172.21.15.78:0/3434176527 conn(0x55914771f220 msgr2=0x5591477a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.309+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/3434176527 shutdown_connections 2024-03-14T09:57:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.309+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/3434176527 wait complete. 2024-03-14T09:57:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.310+0000 7f4ebec8fb80 1 -- start start 2024-03-14T09:57:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.310+0000 7f4ebec8fb80 1 Processor -- start 2024-03-14T09:57:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.310+0000 7f4ebec8fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9680 0x5591477b3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.310+0000 7f4ebec8fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9fa0 0x5591477b0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.310+0000 7f4ebec8fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477b3ec0 0x5591477b13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.310+0000 7f4ebec8fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559147697d80 con 0x5591477a9680 2024-03-14T09:57:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.310+0000 7f4ebec8fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559147697a20 con 0x5591477a9fa0 2024-03-14T09:57:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.310+0000 7f4ebec8fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55914757f6a0 con 0x5591477b3ec0 2024-03-14T09:57:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.311+0000 7f4ebec8fb80 10 client.0 _ll_register_callbacks cb 0x5591475033a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.311+0000 7f4ea3400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9680 0x5591477b3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.311+0000 7f4ea3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9fa0 0x5591477b0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.311+0000 7f4ea3400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9680 0x5591477b3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57268/0 (socket says 172.21.15.78:57268) 2024-03-14T09:57:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.311+0000 7f4ea3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9fa0 0x5591477b0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:52536/0 (socket says 172.21.15.78:52536) 2024-03-14T09:57:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.311+0000 7f4ea3400700 1 -- 172.21.15.78:0/1866631385 learned_addr learned my addr 172.21.15.78:0/1866631385 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.311+0000 7f4ea3400700 1 --2- 172.21.15.78:0/1866631385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477b3ec0 0x5591477b13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.311+0000 7f4ea3e00700 1 -- 172.21.15.78:0/1866631385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477b3ec0 msgr2=0x5591477b13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.311+0000 7f4ea3e00700 1 --2- 172.21.15.78:0/1866631385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477b3ec0 0x5591477b13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.311+0000 7f4ea3e00700 1 -- 172.21.15.78:0/1866631385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9680 msgr2=0x5591477b3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.311+0000 7f4ea3e00700 1 --2- 172.21.15.78:0/1866631385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9680 0x5591477b3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.311+0000 7f4ea3e00700 1 -- 172.21.15.78:0/1866631385 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea410e440 con 0x5591477a9fa0 2024-03-14T09:57:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.312+0000 7f4ea3e00700 1 --2- 172.21.15.78:0/1866631385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9fa0 0x5591477b0c90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400b2e0 tx=0x7f4e94009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c7d2f874b847fbb server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.312+0000 7f4e8e000700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:57:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.312+0000 7f4e8e000700 1 -- 172.21.15.78:0/1866631385 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94027030 con 0x5591477a9fa0 2024-03-14T09:57:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.312+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/1866631385 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55914779c720 con 0x5591477a9fa0 2024-03-14T09:57:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.312+0000 7f4e8e000700 1 -- 172.21.15.78:0/1866631385 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f4e940052b0 con 0x5591477a9fa0 2024-03-14T09:57:22.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.312+0000 7f4e8e000700 1 -- 172.21.15.78:0/1866631385 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e94005470 con 0x5591477a9fa0 2024-03-14T09:57:22.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.312+0000 7f4ebec8fb80 10 client.26321 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:22.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.312+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/1866631385 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559147529b60 con 0x5591477a9fa0 2024-03-14T09:57:22.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.312+0000 7f4e8e000700 1 -- 172.21.15.78:0/1866631385 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55914779c720 con 0x5591477a9fa0 2024-03-14T09:57:22.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.313+0000 7f4e8e000700 1 -- 172.21.15.78:0/1866631385 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4e94023460 con 0x5591477a9fa0 2024-03-14T09:57:22.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.313+0000 7f4ebec8fb80 10 client.26321 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:22.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.313+0000 7f4ebec8fb80 10 client.26321 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:57:22.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.313+0000 7f4ebec8fb80 20 client.26321 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:22.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.313+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/1866631385 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5591477c35b0 0x5591477c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.313+0000 7f4ebec8fb80 4 client.26321 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:57:22.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.313+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/1866631385 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5591476f4700 con 0x5591477c35b0 2024-03-14T09:57:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.313+0000 7f4ea3400700 1 --2- 172.21.15.78:0/1866631385 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5591477c35b0 0x5591477c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.314+0000 7f4ea3400700 1 --2- 172.21.15.78:0/1866631385 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5591477c35b0 0x5591477c59a0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23fb90fe21ea8ba1 server_cookie=90d4485efe67e890 in_seq=0 out_seq=0 2024-03-14T09:57:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.314+0000 7f4e8e000700 10 client.26321 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.314+0000 7f4e8e000700 1 -- 172.21.15.78:0/1866631385 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5591476f4700 con 0x5591477c35b0 2024-03-14T09:57:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.314+0000 7f4e8e000700 10 client.26321 handle_command_reply: tid=0 2024-03-14T09:57:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.314+0000 7f4ebec8fb80 1 client.26321 shutdown 2024-03-14T09:57:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.315+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/1866631385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9fa0 msgr2=0x5591477b0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.315+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/1866631385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9fa0 0x5591477b0c90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400b2e0 tx=0x7f4e94009580 comp rx=0 tx=0).stop 2024-03-14T09:57:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.315+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/1866631385 shutdown_connections 2024-03-14T09:57:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.315+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/1866631385 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477b3ec0 0x5591477b13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.315+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/1866631385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9680 0x5591477b3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.315+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/1866631385 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5591477c35b0 0x5591477c59a0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.315+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/1866631385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9fa0 0x5591477b0c90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.315+0000 7f4e8e000700 0 client.26321 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:22.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.315+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/1866631385 >> 172.21.15.78:0/1866631385 conn(0x55914771f220 msgr2=0x5591477a4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:22.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.315+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/1866631385 shutdown_connections 2024-03-14T09:57:22.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.315+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/1866631385 wait complete. 2024-03-14T09:57:22.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.315+0000 7f4ebec8fb80 20 client.26321 trim_cache size 0 max 16384 2024-03-14T09:57:22.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.330+0000 7f4ebec8fb80 1 Processor -- start 2024-03-14T09:57:22.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.330+0000 7f4ebec8fb80 1 -- start start 2024-03-14T09:57:22.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.331+0000 7f4ebec8fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9680 0x5591477b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.331+0000 7f4ebec8fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9fa0 0x5591477bab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.331+0000 7f4ebec8fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477b3ec0 0x5591477b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.331+0000 7f4ebec8fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591477a9cb0 con 0x5591477a9680 2024-03-14T09:57:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.331+0000 7f4ebec8fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591477b19b0 con 0x5591477a9fa0 2024-03-14T09:57:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.331+0000 7f4ebec8fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591477aa890 con 0x5591477b3ec0 2024-03-14T09:57:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.331+0000 7f4ea3400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9680 0x5591477b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.331+0000 7f4ea3400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9680 0x5591477b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57284/0 (socket says 172.21.15.78:57284) 2024-03-14T09:57:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.331+0000 7f4ea3400700 1 -- 172.21.15.78:0/3277560181 learned_addr learned my addr 172.21.15.78:0/3277560181 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.331+0000 7f4ea3e00700 1 --2- 172.21.15.78:0/3277560181 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9fa0 0x5591477bab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.332+0000 7f4ea3400700 1 --2- 172.21.15.78:0/3277560181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477b3ec0 0x5591477b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.332+0000 7f4ea3400700 1 -- 172.21.15.78:0/3277560181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477b3ec0 msgr2=0x5591477b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.332+0000 7f4ea3400700 1 --2- 172.21.15.78:0/3277560181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477b3ec0 0x5591477b3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.332+0000 7f4ea3400700 1 -- 172.21.15.78:0/3277560181 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9fa0 msgr2=0x5591477bab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:22.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.332+0000 7f4ea3400700 1 --2- 172.21.15.78:0/3277560181 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9fa0 0x5591477bab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.332+0000 7f4ea3400700 1 -- 172.21.15.78:0/3277560181 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5591476c1ea0 con 0x5591477a9680 2024-03-14T09:57:22.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.332+0000 7f4ea3400700 1 --2- 172.21.15.78:0/3277560181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9680 0x5591477b0c30 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4e900098d0 tx=0x7f4e900047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7dd62344f42bf00 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:22.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.332+0000 7f4e9a000700 1 -- 172.21.15.78:0/3277560181 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e900044b0 con 0x5591477a9680 2024-03-14T09:57:22.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.333+0000 7f4e9a000700 1 -- 172.21.15.78:0/3277560181 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e900238b0 con 0x5591477a9680 2024-03-14T09:57:22.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.333+0000 7f4e9a000700 1 -- 172.21.15.78:0/3277560181 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e900039e0 con 0x5591477a9680 2024-03-14T09:57:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.333+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/3277560181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9680 msgr2=0x5591477b0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.333+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/3277560181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9680 0x5591477b0c30 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4e900098d0 tx=0x7f4e900047c0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.333+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/3277560181 shutdown_connections 2024-03-14T09:57:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.334+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/3277560181 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477b3ec0 0x5591477b3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.334+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/3277560181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9680 0x5591477b0c30 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.334+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/3277560181 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477a9fa0 0x5591477bab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.334+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/3277560181 >> 172.21.15.78:0/3277560181 conn(0x55914771f220 msgr2=0x5591477b0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.334+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/3277560181 shutdown_connections 2024-03-14T09:57:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.334+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/3277560181 wait complete. 2024-03-14T09:57:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.334+0000 7f4ebec8fb80 1 -- start start 2024-03-14T09:57:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.334+0000 7f4ebec8fb80 1 Processor -- start 2024-03-14T09:57:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.335+0000 7f4ebec8fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477a9680 0x5591477ba670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.335+0000 7f4ebec8fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9fa0 0x5591477b8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.335+0000 7f4ebec8fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477b3ec0 0x5591477b9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.335+0000 7f4ebec8fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591477baf50 con 0x5591477a9fa0 2024-03-14T09:57:22.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.335+0000 7f4ebec8fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591477b9590 con 0x5591477b3ec0 2024-03-14T09:57:22.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.335+0000 7f4ebec8fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591477b9750 con 0x5591477a9680 2024-03-14T09:57:22.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.335+0000 7f4ebec8fb80 10 client.0 _ll_register_callbacks cb 0x7f4e90009ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:22.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.335+0000 7f4ea3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9fa0 0x5591477b8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.335+0000 7f4ea3400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477a9680 0x5591477ba670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.335+0000 7f4ea3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9fa0 0x5591477b8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57300/0 (socket says 172.21.15.78:57300) 2024-03-14T09:57:22.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.335+0000 7f4ea3e00700 1 -- 172.21.15.78:0/4052078095 learned_addr learned my addr 172.21.15.78:0/4052078095 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:22.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.335+0000 7f4ea3400700 1 --2- 172.21.15.78:0/4052078095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477b3ec0 0x5591477b9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.336+0000 7f4ea3400700 1 -- 172.21.15.78:0/4052078095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477b3ec0 msgr2=0x5591477b9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.336+0000 7f4ea3400700 1 --2- 172.21.15.78:0/4052078095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477b3ec0 0x5591477b9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.336+0000 7f4ea3400700 1 -- 172.21.15.78:0/4052078095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9fa0 msgr2=0x5591477b8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.336+0000 7f4ea3400700 1 --2- 172.21.15.78:0/4052078095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9fa0 0x5591477b8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.336+0000 7f4ea3400700 1 -- 172.21.15.78:0/4052078095 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9001a040 con 0x5591477a9680 2024-03-14T09:57:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.336+0000 7f4ea3400700 1 --2- 172.21.15.78:0/4052078095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477a9680 0x5591477ba670 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4e90000c00 tx=0x7f4e90009750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf0f056bea20a663 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.336+0000 7f4e8e000700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:57:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.336+0000 7f4e8e000700 1 -- 172.21.15.78:0/4052078095 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e900044b0 con 0x5591477a9680 2024-03-14T09:57:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.336+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/4052078095 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e9401e040 con 0x5591477a9680 2024-03-14T09:57:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.336+0000 7f4e8e000700 1 -- 172.21.15.78:0/4052078095 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f4e90023c20 con 0x5591477a9680 2024-03-14T09:57:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.336+0000 7f4e8e000700 1 -- 172.21.15.78:0/4052078095 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e90023de0 con 0x5591477a9680 2024-03-14T09:57:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.336+0000 7f4ebec8fb80 10 client.26335 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.336+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/4052078095 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559147581cd0 con 0x5591477a9680 2024-03-14T09:57:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.337+0000 7f4e8e000700 1 -- 172.21.15.78:0/4052078095 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e9401e040 con 0x5591477a9680 2024-03-14T09:57:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.337+0000 7f4e8e000700 1 -- 172.21.15.78:0/4052078095 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4e90032420 con 0x5591477a9680 2024-03-14T09:57:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.337+0000 7f4ebec8fb80 10 client.26335 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.337+0000 7f4ebec8fb80 10 client.26335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:57:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.337+0000 7f4ebec8fb80 20 client.26335 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.337+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/4052078095 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559147816330 0x559147818750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.337+0000 7f4ebec8fb80 4 client.26335 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:57:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.337+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/4052078095 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55914754acf0 con 0x559147816330 2024-03-14T09:57:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.337+0000 7f4ea3e00700 1 --2- 172.21.15.78:0/4052078095 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559147816330 0x559147818750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.338+0000 7f4e8e000700 10 client.26335 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.338+0000 7f4ea3e00700 1 --2- 172.21.15.78:0/4052078095 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559147816330 0x559147818750 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fcf465422e8a8be server_cookie=c06c12d56ce3c66f in_seq=0 out_seq=0 2024-03-14T09:57:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.338+0000 7f4e8e000700 1 -- 172.21.15.78:0/4052078095 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55914754acf0 con 0x559147816330 2024-03-14T09:57:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.338+0000 7f4e8e000700 10 client.26335 handle_command_reply: tid=0 2024-03-14T09:57:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.338+0000 7f4ebec8fb80 1 client.26335 shutdown 2024-03-14T09:57:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.338+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/4052078095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477a9680 msgr2=0x5591477ba670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.338+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/4052078095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477a9680 0x5591477ba670 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4e90000c00 tx=0x7f4e90009750 comp rx=0 tx=0).stop 2024-03-14T09:57:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.338+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/4052078095 shutdown_connections 2024-03-14T09:57:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.338+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/4052078095 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x559147816330 0x559147818750 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.338+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/4052078095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5591477b3ec0 0x5591477b9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.338+0000 7f4e8e000700 0 client.26335 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.338+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/4052078095 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5591477a9680 0x5591477ba670 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.338+0000 7f4ebec8fb80 1 --2- 172.21.15.78:0/4052078095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591477a9fa0 0x5591477b8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.338+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/4052078095 >> 172.21.15.78:0/4052078095 conn(0x55914771f220 msgr2=0x5591477af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.338+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/4052078095 shutdown_connections 2024-03-14T09:57:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.339+0000 7f4ebec8fb80 1 -- 172.21.15.78:0/4052078095 wait complete. 2024-03-14T09:57:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.339+0000 7f4ebec8fb80 20 client.26335 trim_cache size 0 max 16384 2024-03-14T09:57:22.341 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:57:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.340+0000 7f4eab200700 1 -- 172.21.15.78:0/478688455 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4e880a0c00 msgr2=0x7f4e880a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.340+0000 7f4eab200700 1 --2- 172.21.15.78:0/478688455 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4e880a0c00 0x7f4e880a30c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00c7a0 tx=0x7f4e9c0092e0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.340+0000 7f4eab200700 1 -- 172.21.15.78:0/478688455 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ea410cc30 msgr2=0x7f4ea410d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.340+0000 7f4eab200700 1 --2- 172.21.15.78:0/478688455 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ea410cc30 0x7f4ea410d0e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00ef30 tx=0x7f4e9c00c3a0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.340+0000 7f4eab200700 1 -- 172.21.15.78:0/478688455 shutdown_connections 2024-03-14T09:57:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.340+0000 7f4eab200700 1 --2- 172.21.15.78:0/478688455 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4e880a0c00 0x7f4e880a30c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.340+0000 7f4eab200700 1 --2- 172.21.15.78:0/478688455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ea4100f30 0x7f4ea41a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.340+0000 7f4eab200700 1 --2- 172.21.15.78:0/478688455 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ea410cc30 0x7f4ea410d0e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.340+0000 7f4eab200700 1 --2- 172.21.15.78:0/478688455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ea4106050 0x7f4ea41a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:22.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.340+0000 7f4eab200700 1 -- 172.21.15.78:0/478688455 >> 172.21.15.78:0/478688455 conn(0x7f4ea40fc7b0 msgr2=0x7f4ea40febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:22.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.340+0000 7f4eab200700 1 -- 172.21.15.78:0/478688455 shutdown_connections 2024-03-14T09:57:22.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:22.340+0000 7f4eab200700 1 -- 172.21.15.78:0/478688455 wait complete. 2024-03-14T09:57:23.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:22 smithi078 ceph-mon[38143]: pgmap v99: 129 pgs: 129 active+clean; 114 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 385 op/s 2024-03-14T09:57:23.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:22 smithi152 ceph-mon[48279]: pgmap v99: 129 pgs: 129 active+clean; 114 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 385 op/s 2024-03-14T09:57:23.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:22 smithi132 ceph-mon[47959]: pgmap v99: 129 pgs: 129 active+clean; 114 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 385 op/s 2024-03-14T09:57:23.362 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T09:57:23.362 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T09:57:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.605+0000 7f2b46800700 1 -- 172.21.15.78:0/1804893765 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2b40106040 msgr2=0x7f2b4010dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.605+0000 7f2b46800700 1 --2- 172.21.15.78:0/1804893765 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2b40106040 0x7f2b4010dbc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2b38009f90 tx=0x7f2b3800b330 comp rx=0 tx=0).stop 2024-03-14T09:57:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.605+0000 7f2b46800700 1 -- 172.21.15.78:0/1804893765 shutdown_connections 2024-03-14T09:57:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.605+0000 7f2b46800700 1 --2- 172.21.15.78:0/1804893765 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2b40106040 0x7f2b4010dbc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.605+0000 7f2b46800700 1 --2- 172.21.15.78:0/1804893765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b40100f20 0x7f2b40101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.605+0000 7f2b46800700 1 --2- 172.21.15.78:0/1804893765 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b401018d0 0x7f2b40105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.605+0000 7f2b46800700 1 -- 172.21.15.78:0/1804893765 >> 172.21.15.78:0/1804893765 conn(0x7f2b400fc7a0 msgr2=0x7f2b400febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:23.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.605+0000 7f2b46800700 1 -- 172.21.15.78:0/1804893765 shutdown_connections 2024-03-14T09:57:23.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.605+0000 7f2b46800700 1 -- 172.21.15.78:0/1804893765 wait complete. 2024-03-14T09:57:23.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.607+0000 7f2b46800700 1 Processor -- start 2024-03-14T09:57:23.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.607+0000 7f2b46800700 1 -- start start 2024-03-14T09:57:23.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.607+0000 7f2b46800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b40100f20 0x7f2b40197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.607+0000 7f2b46800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b40106040 0x7f2b40197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.607+0000 7f2b46800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2b4019bde0 0x7f2b4019c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.607+0000 7f2b46800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4019c860 con 0x7f2b40100f20 2024-03-14T09:57:23.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.607+0000 7f2b46800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4019c9d0 con 0x7f2b4019bde0 2024-03-14T09:57:23.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.607+0000 7f2b46800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4019cb90 con 0x7f2b40106040 2024-03-14T09:57:23.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.607+0000 7f2b3ea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b40106040 0x7f2b40197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.607+0000 7f2b3f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b40100f20 0x7f2b40197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.607+0000 7f2b3ea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b40106040 0x7f2b40197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47306/0 (socket says 172.21.15.78:47306) 2024-03-14T09:57:23.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.607+0000 7f2b3fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2b4019bde0 0x7f2b4019c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.607+0000 7f2b3ea00700 1 -- 172.21.15.78:0/1114486122 learned_addr learned my addr 172.21.15.78:0/1114486122 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:23.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.608+0000 7f2b3fe00700 1 -- 172.21.15.78:0/1114486122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b40106040 msgr2=0x7f2b40197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:23.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.608+0000 7f2b3fe00700 1 --2- 172.21.15.78:0/1114486122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b40106040 0x7f2b40197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.608+0000 7f2b3fe00700 1 -- 172.21.15.78:0/1114486122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b40100f20 msgr2=0x7f2b40197500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:23.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.608+0000 7f2b3fe00700 1 --2- 172.21.15.78:0/1114486122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b40100f20 0x7f2b40197500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.608+0000 7f2b3fe00700 1 -- 172.21.15.78:0/1114486122 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b300095f0 con 0x7f2b4019bde0 2024-03-14T09:57:23.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.608+0000 7f2b3f400700 1 --2- 172.21.15.78:0/1114486122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b40100f20 0x7f2b40197500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:23.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.608+0000 7f2b3ea00700 1 --2- 172.21.15.78:0/1114486122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b40106040 0x7f2b40197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:23.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.608+0000 7f2b3fe00700 1 --2- 172.21.15.78:0/1114486122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2b4019bde0 0x7f2b4019c260 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2b3800b6c0 tx=0x7f2b38008fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:23.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.608+0000 7f2b37e00700 1 -- 172.21.15.78:0/1114486122 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3800e040 con 0x7f2b4019bde0 2024-03-14T09:57:23.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.608+0000 7f2b37e00700 1 -- 172.21.15.78:0/1114486122 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b3801d470 con 0x7f2b4019bde0 2024-03-14T09:57:23.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.608+0000 7f2b46800700 1 -- 172.21.15.78:0/1114486122 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b38009c40 con 0x7f2b4019bde0 2024-03-14T09:57:23.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.608+0000 7f2b37e00700 1 -- 172.21.15.78:0/1114486122 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b380052a0 con 0x7f2b4019bde0 2024-03-14T09:57:23.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.608+0000 7f2b46800700 1 -- 172.21.15.78:0/1114486122 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b401a4330 con 0x7f2b4019bde0 2024-03-14T09:57:23.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.613+0000 7f2b37e00700 1 -- 172.21.15.78:0/1114486122 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f2b3801d5e0 con 0x7f2b4019bde0 2024-03-14T09:57:23.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.615+0000 7f2b37e00700 1 --2- 172.21.15.78:0/1114486122 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2b240a0c00 0x7f2b240a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.615+0000 7f2b5a374b80 1 Processor -- start 2024-03-14T09:57:23.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.615+0000 7f2b37e00700 1 -- 172.21.15.78:0/1114486122 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f2b380cdcc0 con 0x7f2b4019bde0 2024-03-14T09:57:23.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.615+0000 7f2b5a374b80 1 -- start start 2024-03-14T09:57:23.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.615+0000 7f2b3f400700 1 --2- 172.21.15.78:0/1114486122 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2b240a0c00 0x7f2b240a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.615+0000 7f2b5a374b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a4fa0 0x55aaed9a5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.615+0000 7f2b5a374b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a58c0 0x55aaed991600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.616+0000 7f2b5a374b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 0x55aaed991f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.616+0000 7f2b5a374b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaed8efa10 con 0x55aaed991b40 2024-03-14T09:57:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.616+0000 7f2b5a374b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaed8ed3d0 con 0x55aaed9a58c0 2024-03-14T09:57:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.616+0000 7f2b5a374b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaed8e8cc0 con 0x55aaed9a4fa0 2024-03-14T09:57:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.616+0000 7f2b3f400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a4fa0 0x55aaed9a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.616+0000 7f2b3f400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a4fa0 0x55aaed9a5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47314/0 (socket says 172.21.15.78:47314) 2024-03-14T09:57:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.616+0000 7f2b3f400700 1 -- 172.21.15.78:0/2169169769 learned_addr learned my addr 172.21.15.78:0/2169169769 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.616+0000 7f2b3ea00700 1 --2- 172.21.15.78:0/2169169769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a58c0 0x55aaed991600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.616+0000 7f2b3fe00700 1 --2- 172.21.15.78:0/2169169769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 0x55aaed991f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.616+0000 7f2b3f400700 1 --2- 172.21.15.78:0/1114486122 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2b240a0c00 0x7f2b240a30c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b30005fd0 tx=0x7f2b30009310 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.616+0000 7f2b3ea00700 1 -- 172.21.15.78:0/2169169769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a4fa0 msgr2=0x55aaed9a5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:23.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.616+0000 7f2b3ea00700 1 --2- 172.21.15.78:0/2169169769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a4fa0 0x55aaed9a5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.616+0000 7f2b3ea00700 1 -- 172.21.15.78:0/2169169769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 msgr2=0x55aaed991f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:23.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.617+0000 7f2b3ea00700 1 --2- 172.21.15.78:0/2169169769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 0x55aaed991f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.617+0000 7f2b3ea00700 1 -- 172.21.15.78:0/2169169769 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aaed76a430 con 0x55aaed9a58c0 2024-03-14T09:57:23.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.617+0000 7f2b3f400700 1 --2- 172.21.15.78:0/2169169769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a4fa0 0x55aaed9a5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:23.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.617+0000 7f2b3ea00700 1 --2- 172.21.15.78:0/2169169769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a58c0 0x55aaed991600 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2b40198a50 tx=0x7f2b2c0117a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bed626113f64c680 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:23.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.617+0000 7f2b34c00700 1 -- 172.21.15.78:0/2169169769 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c00e670 con 0x55aaed9a58c0 2024-03-14T09:57:23.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.617+0000 7f2b34c00700 1 -- 172.21.15.78:0/2169169769 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b2c00ec90 con 0x55aaed9a58c0 2024-03-14T09:57:23.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.617+0000 7f2b34c00700 1 -- 172.21.15.78:0/2169169769 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c019d10 con 0x55aaed9a58c0 2024-03-14T09:57:23.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.618+0000 7f2b5a374b80 1 -- 172.21.15.78:0/2169169769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a58c0 msgr2=0x55aaed991600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:23.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.618+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/2169169769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a58c0 0x55aaed991600 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2b40198a50 tx=0x7f2b2c0117a0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.618+0000 7f2b5a374b80 1 -- 172.21.15.78:0/2169169769 shutdown_connections 2024-03-14T09:57:23.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.618+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/2169169769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 0x55aaed991f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.618+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/2169169769 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a4fa0 0x55aaed9a5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.618+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/2169169769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a58c0 0x55aaed991600 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.618+0000 7f2b5a374b80 1 -- 172.21.15.78:0/2169169769 >> 172.21.15.78:0/2169169769 conn(0x55aaed90c2a0 msgr2=0x55aaed996c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:23.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.618+0000 7f2b5a374b80 1 -- 172.21.15.78:0/2169169769 shutdown_connections 2024-03-14T09:57:23.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.618+0000 7f2b5a374b80 1 -- 172.21.15.78:0/2169169769 wait complete. 2024-03-14T09:57:23.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.619+0000 7f2b5a374b80 1 -- start start 2024-03-14T09:57:23.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.619+0000 7f2b5a374b80 1 Processor -- start 2024-03-14T09:57:23.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.619+0000 7f2b5a374b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 0x55aaed9ab7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.619+0000 7f2b5a374b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a4fa0 0x55aaed9a8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.619+0000 7f2b5a374b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a58c0 0x55aaed9a90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.619+0000 7f2b5a374b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaed884e00 con 0x55aaed991b40 2024-03-14T09:57:23.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.619+0000 7f2b5a374b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaed884aa0 con 0x55aaed9a4fa0 2024-03-14T09:57:23.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.619+0000 7f2b5a374b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaed76c720 con 0x55aaed9a58c0 2024-03-14T09:57:23.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.620+0000 7f2b5a374b80 10 client.0 _ll_register_callbacks cb 0x55aaed6f0420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:23.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.620+0000 7f2b3ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a4fa0 0x55aaed9a8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.620+0000 7f2b3f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 0x55aaed9ab7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.620+0000 7f2b3ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a4fa0 0x55aaed9a8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:52606/0 (socket says 172.21.15.78:52606) 2024-03-14T09:57:23.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.620+0000 7f2b3ea00700 1 -- 172.21.15.78:0/3580582949 learned_addr learned my addr 172.21.15.78:0/3580582949 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:23.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.620+0000 7f2b3fe00700 1 --2- 172.21.15.78:0/3580582949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a58c0 0x55aaed9a90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.620+0000 7f2b3ea00700 1 -- 172.21.15.78:0/3580582949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a58c0 msgr2=0x55aaed9a90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:23.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.620+0000 7f2b3ea00700 1 --2- 172.21.15.78:0/3580582949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a58c0 0x55aaed9a90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.620+0000 7f2b3ea00700 1 -- 172.21.15.78:0/3580582949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 msgr2=0x55aaed9ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:23.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.620+0000 7f2b3ea00700 1 --2- 172.21.15.78:0/3580582949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 0x55aaed9ab7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.620+0000 7f2b3ea00700 1 -- 172.21.15.78:0/3580582949 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b30017070 con 0x55aaed9a4fa0 2024-03-14T09:57:23.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.620+0000 7f2b3fe00700 1 --2- 172.21.15.78:0/3580582949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a58c0 0x55aaed9a90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.620+0000 7f2b3ea00700 1 --2- 172.21.15.78:0/3580582949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a4fa0 0x55aaed9a8b10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c01c040 tx=0x7f2b2c005d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95a76aa66dfa6cae server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.620+0000 7f2b29600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.621+0000 7f2b29600700 1 -- 172.21.15.78:0/3580582949 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c005240 con 0x55aaed9a4fa0 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.621+0000 7f2b5a374b80 1 -- 172.21.15.78:0/3580582949 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aaed9897a0 con 0x55aaed9a4fa0 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.621+0000 7f2b29600700 1 -- 172.21.15.78:0/3580582949 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f2b2c00ee80 con 0x55aaed9a4fa0 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.621+0000 7f2b29600700 1 -- 172.21.15.78:0/3580582949 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b2c0053a0 con 0x55aaed9a4fa0 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.621+0000 7f2b5a374b80 10 client.26339 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.621+0000 7f2b5a374b80 1 -- 172.21.15.78:0/3580582949 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f2b2c009ce0 con 0x55aaed9a4fa0 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.621+0000 7f2b29600700 1 -- 172.21.15.78:0/3580582949 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55aaed9897a0 con 0x55aaed9a4fa0 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.621+0000 7f2b29600700 1 -- 172.21.15.78:0/3580582949 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2b2c028020 con 0x55aaed9a4fa0 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.621+0000 7f2b5a374b80 10 client.26339 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.621+0000 7f2b5a374b80 10 client.26339 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.621+0000 7f2b5a374b80 20 client.26339 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.621+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/3580582949 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55aaed9b91d0 0x55aaed9bb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.621+0000 7f2b5a374b80 4 client.26339 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.621+0000 7f2b5a374b80 1 -- 172.21.15.78:0/3580582949 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aaed8e1780 con 0x55aaed9b91d0 2024-03-14T09:57:23.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.622+0000 7f2b3f400700 1 --2- 172.21.15.78:0/3580582949 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55aaed9b91d0 0x55aaed9bb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.622+0000 7f2b3f400700 1 --2- 172.21.15.78:0/3580582949 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55aaed9b91d0 0x55aaed9bb5f0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8b551472d04e66b server_cookie=83819f44d7fa69bc in_seq=0 out_seq=0 2024-03-14T09:57:23.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.622+0000 7f2b29600700 10 client.26339 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:23.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.623+0000 7f2b29600700 1 -- 172.21.15.78:0/3580582949 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55aaed8e1780 con 0x55aaed9b91d0 2024-03-14T09:57:23.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.623+0000 7f2b29600700 10 client.26339 handle_command_reply: tid=0 2024-03-14T09:57:23.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.623+0000 7f2b5a374b80 1 client.26339 shutdown 2024-03-14T09:57:23.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.623+0000 7f2b5a374b80 1 -- 172.21.15.78:0/3580582949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a4fa0 msgr2=0x55aaed9a8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:23.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.623+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/3580582949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a4fa0 0x55aaed9a8b10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c01c040 tx=0x7f2b2c005d40 comp rx=0 tx=0).stop 2024-03-14T09:57:23.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.623+0000 7f2b5a374b80 1 -- 172.21.15.78:0/3580582949 shutdown_connections 2024-03-14T09:57:23.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.623+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/3580582949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a58c0 0x55aaed9a90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.623+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/3580582949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 0x55aaed9ab7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.623+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/3580582949 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55aaed9b91d0 0x55aaed9bb5f0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.623+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/3580582949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a4fa0 0x55aaed9a8b10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.623+0000 7f2b5a374b80 1 -- 172.21.15.78:0/3580582949 >> 172.21.15.78:0/3580582949 conn(0x55aaed90c2a0 msgr2=0x55aaed996870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:23.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.623+0000 7f2b29600700 0 client.26339 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:23.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.624+0000 7f2b5a374b80 1 -- 172.21.15.78:0/3580582949 shutdown_connections 2024-03-14T09:57:23.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.624+0000 7f2b5a374b80 1 -- 172.21.15.78:0/3580582949 wait complete. 2024-03-14T09:57:23.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.624+0000 7f2b5a374b80 20 client.26339 trim_cache size 0 max 16384 2024-03-14T09:57:23.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.644+0000 7f2b5a374b80 1 Processor -- start 2024-03-14T09:57:23.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.644+0000 7f2b5a374b80 1 -- start start 2024-03-14T09:57:23.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.644+0000 7f2b5a374b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed991b40 0x55aaed9a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.644+0000 7f2b5a374b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed9a4fa0 0x55aaed9a2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.644+0000 7f2b5a374b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a58c0 0x55aaed990ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.644+0000 7f2b5a374b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaed9ac1f0 con 0x55aaed9a4fa0 2024-03-14T09:57:23.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.644+0000 7f2b5a374b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaed9a37a0 con 0x55aaed9a58c0 2024-03-14T09:57:23.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.644+0000 7f2b5a374b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaed9a99e0 con 0x55aaed991b40 2024-03-14T09:57:23.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.644+0000 7f2b3f400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed991b40 0x55aaed9a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.644+0000 7f2b3ea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed9a4fa0 0x55aaed9a2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.644+0000 7f2b3f400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed991b40 0x55aaed9a1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47334/0 (socket says 172.21.15.78:47334) 2024-03-14T09:57:23.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.644+0000 7f2b3f400700 1 -- 172.21.15.78:0/2821743229 learned_addr learned my addr 172.21.15.78:0/2821743229 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:23.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.645+0000 7f2b3fe00700 1 --2- 172.21.15.78:0/2821743229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a58c0 0x55aaed990ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.645+0000 7f2b3f400700 1 -- 172.21.15.78:0/2821743229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a58c0 msgr2=0x55aaed990ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:23.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.645+0000 7f2b3f400700 1 --2- 172.21.15.78:0/2821743229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a58c0 0x55aaed990ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.645+0000 7f2b3f400700 1 -- 172.21.15.78:0/2821743229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed9a4fa0 msgr2=0x55aaed9a2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:23.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.645+0000 7f2b3f400700 1 --2- 172.21.15.78:0/2821743229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed9a4fa0 0x55aaed9a2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.645+0000 7f2b3f400700 1 -- 172.21.15.78:0/2821743229 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aaed716be0 con 0x55aaed991b40 2024-03-14T09:57:23.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.645+0000 7f2b3fe00700 1 --2- 172.21.15.78:0/2821743229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a58c0 0x55aaed990ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:57:23.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.645+0000 7f2b3f400700 1 --2- 172.21.15.78:0/2821743229 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed991b40 0x55aaed9a1ae0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2b30009fe0 tx=0x7f2b30014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff261182bbfaf0b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:23.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.645+0000 7f2b35600700 1 -- 172.21.15.78:0/2821743229 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b30024040 con 0x55aaed991b40 2024-03-14T09:57:23.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.645+0000 7f2b35600700 1 -- 172.21.15.78:0/2821743229 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b30003d40 con 0x55aaed991b40 2024-03-14T09:57:23.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.645+0000 7f2b35600700 1 -- 172.21.15.78:0/2821743229 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b30011b80 con 0x55aaed991b40 2024-03-14T09:57:23.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.646+0000 7f2b5a374b80 1 -- 172.21.15.78:0/2821743229 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed991b40 msgr2=0x55aaed9a1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:23.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.646+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/2821743229 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed991b40 0x55aaed9a1ae0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2b30009fe0 tx=0x7f2b30014000 comp rx=0 tx=0).stop 2024-03-14T09:57:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.646+0000 7f2b5a374b80 1 -- 172.21.15.78:0/2821743229 shutdown_connections 2024-03-14T09:57:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.646+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/2821743229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a58c0 0x55aaed990ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.646+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/2821743229 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed991b40 0x55aaed9a1ae0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.646+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/2821743229 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed9a4fa0 0x55aaed9a2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.646+0000 7f2b5a374b80 1 -- 172.21.15.78:0/2821743229 >> 172.21.15.78:0/2821743229 conn(0x55aaed90c2a0 msgr2=0x55aaed9a73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.646+0000 7f2b5a374b80 1 -- 172.21.15.78:0/2821743229 shutdown_connections 2024-03-14T09:57:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.646+0000 7f2b5a374b80 1 -- 172.21.15.78:0/2821743229 wait complete. 2024-03-14T09:57:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.647+0000 7f2b5a374b80 1 -- start start 2024-03-14T09:57:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.647+0000 7f2b5a374b80 1 Processor -- start 2024-03-14T09:57:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.647+0000 7f2b5a374b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 0x55aaed9a2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.647+0000 7f2b5a374b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a4fa0 0x55aaed9ae470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.647+0000 7f2b5a374b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a58c0 0x55aaed9ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.647+0000 7f2b5a374b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaed9aef90 con 0x55aaed991b40 2024-03-14T09:57:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.647+0000 7f2b5a374b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaed9af100 con 0x55aaed9a4fa0 2024-03-14T09:57:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.647+0000 7f2b5a374b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaed9af2c0 con 0x55aaed9a58c0 2024-03-14T09:57:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.648+0000 7f2b5a374b80 10 client.0 _ll_register_callbacks cb 0x7f2b38009020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.648+0000 7f2b3f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 0x55aaed9a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.648+0000 7f2b3fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a58c0 0x55aaed9ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.648+0000 7f2b3ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a4fa0 0x55aaed9ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.648+0000 7f2b3fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a58c0 0x55aaed9ae9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47348/0 (socket says 172.21.15.78:47348) 2024-03-14T09:57:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.648+0000 7f2b3f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 0x55aaed9a2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57344/0 (socket says 172.21.15.78:57344) 2024-03-14T09:57:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.648+0000 7f2b3fe00700 1 -- 172.21.15.78:0/3554849618 learned_addr learned my addr 172.21.15.78:0/3554849618 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.648+0000 7f2b3ea00700 1 -- 172.21.15.78:0/3554849618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a58c0 msgr2=0x55aaed9ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.648+0000 7f2b3ea00700 1 --2- 172.21.15.78:0/3554849618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a58c0 0x55aaed9ae9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.648+0000 7f2b3ea00700 1 -- 172.21.15.78:0/3554849618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 msgr2=0x55aaed9a2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.648+0000 7f2b3ea00700 1 --2- 172.21.15.78:0/3554849618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 0x55aaed9a2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.648+0000 7f2b3ea00700 1 -- 172.21.15.78:0/3554849618 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b300042e0 con 0x55aaed9a4fa0 2024-03-14T09:57:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.648+0000 7f2b3f400700 1 --2- 172.21.15.78:0/3554849618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 0x55aaed9a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.648+0000 7f2b3ea00700 1 --2- 172.21.15.78:0/3554849618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a4fa0 0x55aaed9ae470 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c019fd0 tx=0x7f2b2c00eb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=286cef3dedea8eb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.648+0000 7f2b29600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:57:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.649+0000 7f2b29600700 1 -- 172.21.15.78:0/3554849618 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c009290 con 0x55aaed9a4fa0 2024-03-14T09:57:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.649+0000 7f2b5a374b80 1 -- 172.21.15.78:0/3554849618 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2b2c00ec90 con 0x55aaed9a4fa0 2024-03-14T09:57:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.649+0000 7f2b29600700 1 -- 172.21.15.78:0/3554849618 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f2b2c000e40 con 0x55aaed9a4fa0 2024-03-14T09:57:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.649+0000 7f2b29600700 1 -- 172.21.15.78:0/3554849618 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b2c0197c0 con 0x55aaed9a4fa0 2024-03-14T09:57:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.649+0000 7f2b5a374b80 10 client.26345 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.649+0000 7f2b5a374b80 1 -- 172.21.15.78:0/3554849618 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55aaed8aef20 con 0x55aaed9a4fa0 2024-03-14T09:57:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.649+0000 7f2b29600700 1 -- 172.21.15.78:0/3554849618 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b2c00ec90 con 0x55aaed9a4fa0 2024-03-14T09:57:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.650+0000 7f2b29600700 1 -- 172.21.15.78:0/3554849618 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2b2c028070 con 0x55aaed9a4fa0 2024-03-14T09:57:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.650+0000 7f2b5a374b80 10 client.26345 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.650+0000 7f2b5a374b80 10 client.26345 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:57:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.650+0000 7f2b5a374b80 20 client.26345 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.650+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/3554849618 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55aaeda0d010 0x55aaeda0f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.650+0000 7f2b5a374b80 4 client.26345 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T09:57:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.650+0000 7f2b5a374b80 1 -- 172.21.15.78:0/3554849618 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55aaed737d70 con 0x55aaeda0d010 2024-03-14T09:57:23.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.650+0000 7f2b3f400700 1 --2- 172.21.15.78:0/3554849618 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55aaeda0d010 0x55aaeda0f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.650+0000 7f2b3f400700 1 --2- 172.21.15.78:0/3554849618 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55aaeda0d010 0x55aaeda0f430 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfbced583a140903 server_cookie=8fba5060e2ff3e03 in_seq=0 out_seq=0 2024-03-14T09:57:23.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.650+0000 7f2b29600700 10 client.26345 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:23.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.652+0000 7f2b29600700 1 -- 172.21.15.78:0/3554849618 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55aaed737d70 con 0x55aaeda0d010 2024-03-14T09:57:23.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.652+0000 7f2b29600700 10 client.26345 handle_command_reply: tid=0 2024-03-14T09:57:23.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.652+0000 7f2b5a374b80 1 client.26345 shutdown 2024-03-14T09:57:23.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.652+0000 7f2b5a374b80 1 -- 172.21.15.78:0/3554849618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a4fa0 msgr2=0x55aaed9ae470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:23.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.652+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/3554849618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a4fa0 0x55aaed9ae470 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c019fd0 tx=0x7f2b2c00eb10 comp rx=0 tx=0).stop 2024-03-14T09:57:23.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.652+0000 7f2b5a374b80 1 -- 172.21.15.78:0/3554849618 shutdown_connections 2024-03-14T09:57:23.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.652+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/3554849618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55aaed9a58c0 0x55aaed9ae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.652+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/3554849618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55aaed991b40 0x55aaed9a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.652+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/3554849618 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55aaeda0d010 0x55aaeda0f430 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.652+0000 7f2b5a374b80 1 --2- 172.21.15.78:0/3554849618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aaed9a4fa0 0x55aaed9ae470 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.652+0000 7f2b5a374b80 1 -- 172.21.15.78:0/3554849618 >> 172.21.15.78:0/3554849618 conn(0x55aaed90c2a0 msgr2=0x55aaed990ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:23.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.652+0000 7f2b29600700 0 client.26345 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:23.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.652+0000 7f2b5a374b80 1 -- 172.21.15.78:0/3554849618 shutdown_connections 2024-03-14T09:57:23.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.652+0000 7f2b5a374b80 1 -- 172.21.15.78:0/3554849618 wait complete. 2024-03-14T09:57:23.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.652+0000 7f2b5a374b80 20 client.26345 trim_cache size 0 max 16384 2024-03-14T09:57:23.655 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:57:23.655 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T09:57:23.655 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "36005c13-5ff8-43ff-9d3b-f359504f334f", 2024-03-14T09:57:23.656 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T09:57:23.656 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:57:23.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.653+0000 7f2b46800700 1 -- 172.21.15.78:0/1114486122 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2b240a0c00 msgr2=0x7f2b240a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:23.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.653+0000 7f2b46800700 1 --2- 172.21.15.78:0/1114486122 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2b240a0c00 0x7f2b240a30c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b30005fd0 tx=0x7f2b30009310 comp rx=0 tx=0).stop 2024-03-14T09:57:23.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.653+0000 7f2b46800700 1 -- 172.21.15.78:0/1114486122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2b4019bde0 msgr2=0x7f2b4019c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:23.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.653+0000 7f2b46800700 1 --2- 172.21.15.78:0/1114486122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2b4019bde0 0x7f2b4019c260 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2b3800b6c0 tx=0x7f2b38008fb0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.653+0000 7f2b46800700 1 -- 172.21.15.78:0/1114486122 shutdown_connections 2024-03-14T09:57:23.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.653+0000 7f2b46800700 1 --2- 172.21.15.78:0/1114486122 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2b240a0c00 0x7f2b240a30c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.653+0000 7f2b46800700 1 --2- 172.21.15.78:0/1114486122 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2b4019bde0 0x7f2b4019c260 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.653+0000 7f2b46800700 1 --2- 172.21.15.78:0/1114486122 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2b40100f20 0x7f2b40197500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.653+0000 7f2b46800700 1 --2- 172.21.15.78:0/1114486122 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2b40106040 0x7f2b40197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.653+0000 7f2b46800700 1 -- 172.21.15.78:0/1114486122 >> 172.21.15.78:0/1114486122 conn(0x7f2b400fc7a0 msgr2=0x7f2b400fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:23.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.654+0000 7f2b46800700 1 -- 172.21.15.78:0/1114486122 shutdown_connections 2024-03-14T09:57:23.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.654+0000 7f2b46800700 1 -- 172.21.15.78:0/1114486122 wait complete. 2024-03-14T09:57:23.676 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:57:23.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.933+0000 7f1989200700 1 -- 172.21.15.78:0/3412174040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19841018d0 msgr2=0x7f1984105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:23.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.933+0000 7f1989200700 1 --2- 172.21.15.78:0/3412174040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19841018d0 0x7f1984105910 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1970009990 tx=0x7f1970009ca0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.934+0000 7f1989200700 1 -- 172.21.15.78:0/3412174040 shutdown_connections 2024-03-14T09:57:23.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.934+0000 7f1989200700 1 --2- 172.21.15.78:0/3412174040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1984106040 0x7f198410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.934+0000 7f1989200700 1 --2- 172.21.15.78:0/3412174040 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1984100f20 0x7f1984101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.934+0000 7f1989200700 1 --2- 172.21.15.78:0/3412174040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f19841018d0 0x7f1984105910 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.934+0000 7f1989200700 1 -- 172.21.15.78:0/3412174040 >> 172.21.15.78:0/3412174040 conn(0x7f19840fc7a0 msgr2=0x7f19840febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:23.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.934+0000 7f1989200700 1 -- 172.21.15.78:0/3412174040 shutdown_connections 2024-03-14T09:57:23.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.934+0000 7f1989200700 1 -- 172.21.15.78:0/3412174040 wait complete. 2024-03-14T09:57:23.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.935+0000 7f1989200700 1 Processor -- start 2024-03-14T09:57:23.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.936+0000 7f1989200700 1 -- start start 2024-03-14T09:57:23.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.936+0000 7f1989200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1984100f20 0x7f198419f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.936+0000 7f1989200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1984106040 0x7f198419fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.936+0000 7f1989200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1984199680 0x7f1984199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.936+0000 7f1989200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198419a100 con 0x7f1984199680 2024-03-14T09:57:23.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.936+0000 7f1989200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198419a270 con 0x7f1984106040 2024-03-14T09:57:23.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.936+0000 7f1989200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198419a430 con 0x7f1984100f20 2024-03-14T09:57:23.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.936+0000 7f1981e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1984100f20 0x7f198419f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.936+0000 7f1981e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1984100f20 0x7f198419f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:44264/0 (socket says 172.21.15.78:44264) 2024-03-14T09:57:23.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.936+0000 7f1981e00700 1 -- 172.21.15.78:0/4013243284 learned_addr learned my addr 172.21.15.78:0/4013243284 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:23.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.936+0000 7f1981400700 1 --2- 172.21.15.78:0/4013243284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1984106040 0x7f198419fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.937+0000 7f1982800700 1 --2- 172.21.15.78:0/4013243284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1984199680 0x7f1984199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.937+0000 7f1981e00700 1 -- 172.21.15.78:0/4013243284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1984106040 msgr2=0x7f198419fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:23.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.937+0000 7f1981e00700 1 --2- 172.21.15.78:0/4013243284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1984106040 0x7f198419fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.937+0000 7f1981e00700 1 -- 172.21.15.78:0/4013243284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1984199680 msgr2=0x7f1984199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:23.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.937+0000 7f1981e00700 1 --2- 172.21.15.78:0/4013243284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1984199680 0x7f1984199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.937+0000 7f1981e00700 1 -- 172.21.15.78:0/4013243284 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19700095f0 con 0x7f1984100f20 2024-03-14T09:57:23.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.937+0000 7f1981e00700 1 --2- 172.21.15.78:0/4013243284 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1984100f20 0x7f198419f500 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f197400d950 tx=0x7f197400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:23.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.938+0000 7f197aa00700 1 -- 172.21.15.78:0/4013243284 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197400f890 con 0x7f1984100f20 2024-03-14T09:57:23.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.938+0000 7f197aa00700 1 -- 172.21.15.78:0/4013243284 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1974010460 con 0x7f1984100f20 2024-03-14T09:57:23.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.938+0000 7f197aa00700 1 -- 172.21.15.78:0/4013243284 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197400e610 con 0x7f1984100f20 2024-03-14T09:57:23.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.938+0000 7f1989200700 1 -- 172.21.15.78:0/4013243284 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19841a63d0 con 0x7f1984100f20 2024-03-14T09:57:23.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.938+0000 7f1989200700 1 -- 172.21.15.78:0/4013243284 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19841a6860 con 0x7f1984100f20 2024-03-14T09:57:23.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.942+0000 7f197aa00700 1 -- 172.21.15.78:0/4013243284 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f19740108d0 con 0x7f1984100f20 2024-03-14T09:57:23.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.943+0000 7f197aa00700 1 --2- 172.21.15.78:0/4013243284 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f19680a0d50 0x7f19680a3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.943+0000 7f197aa00700 1 -- 172.21.15.78:0/4013243284 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f19740cc490 con 0x7f1984100f20 2024-03-14T09:57:23.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.944+0000 7f1981400700 1 --2- 172.21.15.78:0/4013243284 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f19680a0d50 0x7f19680a3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.945+0000 7f1981400700 1 --2- 172.21.15.78:0/4013243284 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f19680a0d50 0x7f19680a3210 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f198419ad40 tx=0x7f1970005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:23.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.946+0000 7f199cc54b80 1 Processor -- start 2024-03-14T09:57:23.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.946+0000 7f199cc54b80 1 -- start start 2024-03-14T09:57:23.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.947+0000 7f199cc54b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040680 0x55af65040a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.947+0000 7f199cc54b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af65040fa0 0x55af6504a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.947+0000 7f199cc54b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af6504aec0 0x55af6504d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.947+0000 7f199cc54b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af64f99990 con 0x55af65040680 2024-03-14T09:57:23.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.947+0000 7f199cc54b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af64f97350 con 0x55af6504aec0 2024-03-14T09:57:23.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.947+0000 7f199cc54b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af64f92c40 con 0x55af65040fa0 2024-03-14T09:57:23.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.947+0000 7f1981e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040680 0x55af65040a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.947+0000 7f1981400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af65040fa0 0x55af6504a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.947+0000 7f1981e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040680 0x55af65040a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47272/0 (socket says 172.21.15.78:47272) 2024-03-14T09:57:23.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.947+0000 7f1981e00700 1 -- 172.21.15.78:0/2411211594 learned_addr learned my addr 172.21.15.78:0/2411211594 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:23.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.947+0000 7f1982800700 1 --2- 172.21.15.78:0/2411211594 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af6504aec0 0x55af6504d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.948+0000 7f1981e00700 1 -- 172.21.15.78:0/2411211594 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af65040fa0 msgr2=0x55af6504a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:23.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.948+0000 7f1981e00700 1 --2- 172.21.15.78:0/2411211594 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af65040fa0 0x55af6504a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.948+0000 7f1981e00700 1 -- 172.21.15.78:0/2411211594 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af6504aec0 msgr2=0x55af6504d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:23.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.948+0000 7f1981e00700 1 --2- 172.21.15.78:0/2411211594 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af6504aec0 0x55af6504d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.948+0000 7f1981e00700 1 -- 172.21.15.78:0/2411211594 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af64e143b0 con 0x55af65040680 2024-03-14T09:57:23.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.948+0000 7f1981e00700 1 --2- 172.21.15.78:0/2411211594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040680 0x55af65040a60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f19740a1380 tx=0x7f1974010cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769b7fc55b56c17f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:23.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.948+0000 7f196f400700 1 -- 172.21.15.78:0/2411211594 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1974099430 con 0x55af65040680 2024-03-14T09:57:23.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.948+0000 7f196f400700 1 -- 172.21.15.78:0/2411211594 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f197409d260 con 0x55af65040680 2024-03-14T09:57:23.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.948+0000 7f196f400700 1 -- 172.21.15.78:0/2411211594 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19740998f0 con 0x55af65040680 2024-03-14T09:57:23.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.949+0000 7f199cc54b80 1 -- 172.21.15.78:0/2411211594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040680 msgr2=0x55af65040a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:23.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.949+0000 7f199cc54b80 1 --2- 172.21.15.78:0/2411211594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040680 0x55af65040a60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f19740a1380 tx=0x7f1974010cd0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.951+0000 7f199cc54b80 1 -- 172.21.15.78:0/2411211594 shutdown_connections 2024-03-14T09:57:23.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.951+0000 7f199cc54b80 1 --2- 172.21.15.78:0/2411211594 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af6504aec0 0x55af6504d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.951+0000 7f199cc54b80 1 --2- 172.21.15.78:0/2411211594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040680 0x55af65040a60 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.951+0000 7f199cc54b80 1 --2- 172.21.15.78:0/2411211594 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af65040fa0 0x55af6504a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.951+0000 7f199cc54b80 1 -- 172.21.15.78:0/2411211594 >> 172.21.15.78:0/2411211594 conn(0x55af64fb6220 msgr2=0x55af6503baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:23.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.951+0000 7f199cc54b80 1 -- 172.21.15.78:0/2411211594 shutdown_connections 2024-03-14T09:57:23.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.952+0000 7f199cc54b80 1 -- 172.21.15.78:0/2411211594 wait complete. 2024-03-14T09:57:23.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.953+0000 7f199cc54b80 1 -- start start 2024-03-14T09:57:23.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.953+0000 7f199cc54b80 1 Processor -- start 2024-03-14T09:57:23.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.954+0000 7f199cc54b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040680 0x55af6504a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.954+0000 7f199cc54b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040fa0 0x55af65047b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.954+0000 7f199cc54b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af6504aec0 0x55af65048190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.954+0000 7f199cc54b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af64f2ed80 con 0x55af65040fa0 2024-03-14T09:57:23.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.954+0000 7f199cc54b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af64f2ea20 con 0x55af65040680 2024-03-14T09:57:23.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.954+0000 7f199cc54b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af64e166a0 con 0x55af6504aec0 2024-03-14T09:57:23.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.954+0000 7f199cc54b80 10 client.0 _ll_register_callbacks cb 0x55af64d9a3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:23.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.955+0000 7f1982800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af6504aec0 0x55af65048190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.955+0000 7f1982800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af6504aec0 0x55af65048190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:44274/0 (socket says 172.21.15.78:44274) 2024-03-14T09:57:23.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.955+0000 7f1982800700 1 -- 172.21.15.78:0/3057803272 learned_addr learned my addr 172.21.15.78:0/3057803272 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:23.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.955+0000 7f1981e00700 1 --2- 172.21.15.78:0/3057803272 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040680 0x55af6504a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.955+0000 7f1981400700 1 --2- 172.21.15.78:0/3057803272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040fa0 0x55af65047b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.955+0000 7f1982800700 1 -- 172.21.15.78:0/3057803272 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040680 msgr2=0x55af6504a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:23.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.955+0000 7f1982800700 1 --2- 172.21.15.78:0/3057803272 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040680 0x55af6504a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.955+0000 7f1982800700 1 -- 172.21.15.78:0/3057803272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040fa0 msgr2=0x55af65047b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:23.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.955+0000 7f1982800700 1 --2- 172.21.15.78:0/3057803272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040fa0 0x55af65047b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.955+0000 7f1982800700 1 -- 172.21.15.78:0/3057803272 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19841a6860 con 0x55af6504aec0 2024-03-14T09:57:23.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.955+0000 7f1981400700 1 --2- 172.21.15.78:0/3057803272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040fa0 0x55af65047b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:57:23.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.956+0000 7f1982800700 1 --2- 172.21.15.78:0/3057803272 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af6504aec0 0x55af65048190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f197c003fd0 tx=0x7f197c004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2875d44ffdb4fbed server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:23.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.957+0000 7f1963e00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:57:23.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.957+0000 7f1963e00700 1 -- 172.21.15.78:0/3057803272 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197c004db0 con 0x55af6504aec0 2024-03-14T09:57:23.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.957+0000 7f1963e00700 1 -- 172.21.15.78:0/3057803272 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f197c007500 con 0x55af6504aec0 2024-03-14T09:57:23.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.957+0000 7f199cc54b80 1 -- 172.21.15.78:0/3057803272 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55af65033720 con 0x55af6504aec0 2024-03-14T09:57:23.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.958+0000 7f1963e00700 1 -- 172.21.15.78:0/3057803272 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f197c0076c0 con 0x55af6504aec0 2024-03-14T09:57:23.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.958+0000 7f199cc54b80 10 client.26365 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:23.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.958+0000 7f199cc54b80 1 -- 172.21.15.78:0/3057803272 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55af64dc0b60 con 0x55af6504aec0 2024-03-14T09:57:23.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.958+0000 7f1963e00700 1 -- 172.21.15.78:0/3057803272 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55af65033720 con 0x55af6504aec0 2024-03-14T09:57:23.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.959+0000 7f1963e00700 1 -- 172.21.15.78:0/3057803272 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f197c016840 con 0x55af6504aec0 2024-03-14T09:57:23.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.959+0000 7f199cc54b80 10 client.26365 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:23.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.959+0000 7f199cc54b80 10 client.26365 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:57:23.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.959+0000 7f199cc54b80 20 client.26365 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:23.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.959+0000 7f199cc54b80 1 --2- 172.21.15.78:0/3057803272 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55af65051930 0x55af65053d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.959+0000 7f199cc54b80 4 client.26365 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:57:23.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.959+0000 7f199cc54b80 1 -- 172.21.15.78:0/3057803272 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55af64f8b700 con 0x55af65051930 2024-03-14T09:57:23.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.959+0000 7f1981e00700 1 --2- 172.21.15.78:0/3057803272 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55af65051930 0x55af65053d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.960+0000 7f1963e00700 10 client.26365 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:23.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.960+0000 7f1981e00700 1 --2- 172.21.15.78:0/3057803272 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55af65051930 0x55af65053d50 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=343385be7455dd69 server_cookie=576325eeff13a452 in_seq=0 out_seq=0 2024-03-14T09:57:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.961+0000 7f1963e00700 1 -- 172.21.15.78:0/3057803272 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55af64f8b700 con 0x55af65051930 2024-03-14T09:57:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.961+0000 7f1963e00700 10 client.26365 handle_command_reply: tid=0 2024-03-14T09:57:23.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.961+0000 7f199cc54b80 1 client.26365 shutdown 2024-03-14T09:57:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.961+0000 7f199cc54b80 1 -- 172.21.15.78:0/3057803272 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af6504aec0 msgr2=0x55af65048190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.961+0000 7f199cc54b80 1 --2- 172.21.15.78:0/3057803272 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af6504aec0 0x55af65048190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f197c003fd0 tx=0x7f197c004040 comp rx=0 tx=0).stop 2024-03-14T09:57:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.961+0000 7f199cc54b80 1 -- 172.21.15.78:0/3057803272 shutdown_connections 2024-03-14T09:57:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.961+0000 7f199cc54b80 1 --2- 172.21.15.78:0/3057803272 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55af65051930 0x55af65053d50 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.961+0000 7f1963e00700 0 client.26365 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.961+0000 7f199cc54b80 1 --2- 172.21.15.78:0/3057803272 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040680 0x55af6504a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.961+0000 7f199cc54b80 1 --2- 172.21.15.78:0/3057803272 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af6504aec0 0x55af65048190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.961+0000 7f199cc54b80 1 --2- 172.21.15.78:0/3057803272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040fa0 0x55af65047b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.961+0000 7f199cc54b80 1 -- 172.21.15.78:0/3057803272 >> 172.21.15.78:0/3057803272 conn(0x55af64fb6220 msgr2=0x55af6503b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:23.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.962+0000 7f199cc54b80 1 -- 172.21.15.78:0/3057803272 shutdown_connections 2024-03-14T09:57:23.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.962+0000 7f199cc54b80 1 -- 172.21.15.78:0/3057803272 wait complete. 2024-03-14T09:57:23.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.962+0000 7f199cc54b80 20 client.26365 trim_cache size 0 max 16384 2024-03-14T09:57:23.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.988+0000 7f199cc54b80 1 Processor -- start 2024-03-14T09:57:23.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.988+0000 7f199cc54b80 1 -- start start 2024-03-14T09:57:23.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.988+0000 7f199cc54b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af65040680 0x55af6503bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.988+0000 7f199cc54b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040fa0 0x55af650480b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.988+0000 7f199cc54b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af6504aec0 0x55af65046430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.988+0000 7f199cc54b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af65042310 con 0x55af6504aec0 2024-03-14T09:57:23.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.988+0000 7f199cc54b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af6504f060 con 0x55af65040fa0 2024-03-14T09:57:23.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.988+0000 7f199cc54b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af6503d060 con 0x55af65040680 2024-03-14T09:57:23.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.989+0000 7f1982800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af6504aec0 0x55af65046430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.989+0000 7f1982800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af6504aec0 0x55af65046430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47290/0 (socket says 172.21.15.78:47290) 2024-03-14T09:57:23.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.989+0000 7f1982800700 1 -- 172.21.15.78:0/605098322 learned_addr learned my addr 172.21.15.78:0/605098322 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:23.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.989+0000 7f1981400700 1 --2- 172.21.15.78:0/605098322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040fa0 0x55af650480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.989+0000 7f1981e00700 1 --2- 172.21.15.78:0/605098322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af65040680 0x55af6503bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.990+0000 7f1981400700 1 -- 172.21.15.78:0/605098322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af65040680 msgr2=0x55af6503bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.990+0000 7f1981400700 1 --2- 172.21.15.78:0/605098322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af65040680 0x55af6503bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.990+0000 7f1981400700 1 -- 172.21.15.78:0/605098322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af6504aec0 msgr2=0x55af65046430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.990+0000 7f1981400700 1 --2- 172.21.15.78:0/605098322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af6504aec0 0x55af65046430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.990+0000 7f1981400700 1 -- 172.21.15.78:0/605098322 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af64f58ea0 con 0x55af65040fa0 2024-03-14T09:57:23.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.990+0000 7f1981400700 1 --2- 172.21.15.78:0/605098322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040fa0 0x55af650480b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1970000c00 tx=0x7f1970021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0084a18205274b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:23.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.990+0000 7f196fe00700 1 -- 172.21.15.78:0/605098322 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970029050 con 0x55af65040fa0 2024-03-14T09:57:23.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.991+0000 7f196fe00700 1 -- 172.21.15.78:0/605098322 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f197002a040 con 0x55af65040fa0 2024-03-14T09:57:23.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.991+0000 7f196fe00700 1 -- 172.21.15.78:0/605098322 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197000fa00 con 0x55af65040fa0 2024-03-14T09:57:23.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.991+0000 7f199cc54b80 1 -- 172.21.15.78:0/605098322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040fa0 msgr2=0x55af650480b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:23.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.991+0000 7f199cc54b80 1 --2- 172.21.15.78:0/605098322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040fa0 0x55af650480b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1970000c00 tx=0x7f1970021000 comp rx=0 tx=0).stop 2024-03-14T09:57:23.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.992+0000 7f199cc54b80 1 -- 172.21.15.78:0/605098322 shutdown_connections 2024-03-14T09:57:23.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.992+0000 7f199cc54b80 1 --2- 172.21.15.78:0/605098322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af6504aec0 0x55af65046430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.992+0000 7f199cc54b80 1 --2- 172.21.15.78:0/605098322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af65040680 0x55af6503bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.992+0000 7f199cc54b80 1 --2- 172.21.15.78:0/605098322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040fa0 0x55af650480b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.992+0000 7f199cc54b80 1 -- 172.21.15.78:0/605098322 >> 172.21.15.78:0/605098322 conn(0x55af64fb6220 msgr2=0x55af65042e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:23.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.992+0000 7f199cc54b80 1 -- 172.21.15.78:0/605098322 shutdown_connections 2024-03-14T09:57:23.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.992+0000 7f199cc54b80 1 -- 172.21.15.78:0/605098322 wait complete. 2024-03-14T09:57:23.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.993+0000 7f199cc54b80 1 -- start start 2024-03-14T09:57:23.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.993+0000 7f199cc54b80 1 Processor -- start 2024-03-14T09:57:23.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.993+0000 7f199cc54b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040680 0x55af65057110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.993+0000 7f199cc54b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040fa0 0x55af65057650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.993+0000 7f199cc54b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af6504aec0 0x55af650bd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.994+0000 7f199cc54b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af65057f60 con 0x55af65040680 2024-03-14T09:57:23.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.994+0000 7f199cc54b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af650bd610 con 0x55af65040fa0 2024-03-14T09:57:23.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.994+0000 7f199cc54b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af650bd750 con 0x55af6504aec0 2024-03-14T09:57:23.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.994+0000 7f199cc54b80 10 client.0 _ll_register_callbacks cb 0x55af64d9a3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:23.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.994+0000 7f1981e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040680 0x55af65057110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.994+0000 7f1981400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040fa0 0x55af65057650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.994+0000 7f1981400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040fa0 0x55af65057650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:36264/0 (socket says 172.21.15.78:36264) 2024-03-14T09:57:23.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.994+0000 7f1981400700 1 -- 172.21.15.78:0/3718614425 learned_addr learned my addr 172.21.15.78:0/3718614425 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:23.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.994+0000 7f1982800700 1 --2- 172.21.15.78:0/3718614425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af6504aec0 0x55af650bd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.994+0000 7f1981400700 1 -- 172.21.15.78:0/3718614425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af6504aec0 msgr2=0x55af650bd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:23.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.994+0000 7f1981400700 1 --2- 172.21.15.78:0/3718614425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af6504aec0 0x55af650bd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.994+0000 7f1981400700 1 -- 172.21.15.78:0/3718614425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040680 msgr2=0x55af65057110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:23.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.994+0000 7f1981400700 1 --2- 172.21.15.78:0/3718614425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040680 0x55af65057110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:23.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.994+0000 7f1981400700 1 -- 172.21.15.78:0/3718614425 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f198410e430 con 0x55af65040fa0 2024-03-14T09:57:23.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.995+0000 7f1981400700 1 --2- 172.21.15.78:0/3718614425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040fa0 0x55af65057650 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1970005f50 tx=0x7f1970003ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48c8fb5862b84257 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:23.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.995+0000 7f1963e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:57:23.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.995+0000 7f1963e00700 1 -- 172.21.15.78:0/3718614425 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197000f670 con 0x55af65040fa0 2024-03-14T09:57:23.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.996+0000 7f1963e00700 1 -- 172.21.15.78:0/3718614425 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f197000f7d0 con 0x55af65040fa0 2024-03-14T09:57:23.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.996+0000 7f199cc54b80 1 -- 172.21.15.78:0/3718614425 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f197409d260 con 0x55af65040fa0 2024-03-14T09:57:23.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.996+0000 7f1963e00700 1 -- 172.21.15.78:0/3718614425 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1970029020 con 0x55af65040fa0 2024-03-14T09:57:23.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.996+0000 7f199cc54b80 10 client.26363 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:23.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.996+0000 7f199cc54b80 1 -- 172.21.15.78:0/3718614425 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55af64e18cd0 con 0x55af65040fa0 2024-03-14T09:57:23.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.996+0000 7f1963e00700 1 -- 172.21.15.78:0/3718614425 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f197409d260 con 0x55af65040fa0 2024-03-14T09:57:23.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.997+0000 7f1963e00700 1 -- 172.21.15.78:0/3718614425 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1970037a20 con 0x55af65040fa0 2024-03-14T09:57:23.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.998+0000 7f199cc54b80 10 client.26363 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:23.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.998+0000 7f199cc54b80 10 client.26363 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:57:23.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.998+0000 7f199cc54b80 20 client.26363 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:23.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.998+0000 7f199cc54b80 1 --2- 172.21.15.78:0/3718614425 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55af65051930 0x55af650c0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:23.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.998+0000 7f199cc54b80 4 client.26363 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:57:23.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.998+0000 7f199cc54b80 1 -- 172.21.15.78:0/3718614425 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55af64de1cf0 con 0x55af65051930 2024-03-14T09:57:23.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.998+0000 7f1981e00700 1 --2- 172.21.15.78:0/3718614425 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55af65051930 0x55af650c0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:23.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.998+0000 7f1981e00700 1 --2- 172.21.15.78:0/3718614425 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55af65051930 0x55af650c0b60 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c810a17b5773fc42 server_cookie=71d7d77c5baabc82 in_seq=0 out_seq=0 2024-03-14T09:57:23.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.998+0000 7f1963e00700 10 client.26363 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:23.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.999+0000 7f1963e00700 1 -- 172.21.15.78:0/3718614425 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55af64de1cf0 con 0x55af65051930 2024-03-14T09:57:23.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.999+0000 7f1963e00700 10 client.26363 handle_command_reply: tid=0 2024-03-14T09:57:23.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:23.999+0000 7f199cc54b80 1 client.26363 shutdown 2024-03-14T09:57:24.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.000+0000 7f199cc54b80 1 -- 172.21.15.78:0/3718614425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040fa0 msgr2=0x55af65057650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:24.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.000+0000 7f199cc54b80 1 --2- 172.21.15.78:0/3718614425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040fa0 0x55af65057650 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1970005f50 tx=0x7f1970003ea0 comp rx=0 tx=0).stop 2024-03-14T09:57:24.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.000+0000 7f199cc54b80 1 -- 172.21.15.78:0/3718614425 shutdown_connections 2024-03-14T09:57:24.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.000+0000 7f199cc54b80 1 --2- 172.21.15.78:0/3718614425 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55af6504aec0 0x55af650bd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:24.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.000+0000 7f199cc54b80 1 --2- 172.21.15.78:0/3718614425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55af65040680 0x55af65057110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:24.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.000+0000 7f199cc54b80 1 --2- 172.21.15.78:0/3718614425 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55af65051930 0x55af650c0b60 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:24.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.000+0000 7f199cc54b80 1 --2- 172.21.15.78:0/3718614425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55af65040fa0 0x55af65057650 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:24.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.000+0000 7f199cc54b80 1 -- 172.21.15.78:0/3718614425 >> 172.21.15.78:0/3718614425 conn(0x55af64fb6220 msgr2=0x55af650463b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:24.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.000+0000 7f1963e00700 0 client.26363 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:24.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.000+0000 7f199cc54b80 1 -- 172.21.15.78:0/3718614425 shutdown_connections 2024-03-14T09:57:24.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.000+0000 7f199cc54b80 1 -- 172.21.15.78:0/3718614425 wait complete. 2024-03-14T09:57:24.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.001+0000 7f199cc54b80 20 client.26363 trim_cache size 0 max 16384 2024-03-14T09:57:24.001 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:57:24.001 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "scrub active (1050 inodes in the stack)", 2024-03-14T09:57:24.001 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T09:57:24.001 INFO:teuthology.orchestra.run.smithi078.stdout: "36005c13-5ff8-43ff-9d3b-f359504f334f": { 2024-03-14T09:57:24.001 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T09:57:24.001 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "36005c13-5ff8-43ff-9d3b-f359504f334f", 2024-03-14T09:57:24.001 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T09:57:24.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:57:24.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:57:24.002 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:57:24.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.002+0000 7f1989200700 1 -- 172.21.15.78:0/4013243284 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f19680a0d50 msgr2=0x7f19680a3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:24.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.002+0000 7f1989200700 1 --2- 172.21.15.78:0/4013243284 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f19680a0d50 0x7f19680a3210 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f198419ad40 tx=0x7f1970005fd0 comp rx=0 tx=0).stop 2024-03-14T09:57:24.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.002+0000 7f1989200700 1 -- 172.21.15.78:0/4013243284 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1984100f20 msgr2=0x7f198419f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:24.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.002+0000 7f1989200700 1 --2- 172.21.15.78:0/4013243284 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1984100f20 0x7f198419f500 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f197400d950 tx=0x7f197400dc60 comp rx=0 tx=0).stop 2024-03-14T09:57:24.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.002+0000 7f1989200700 1 -- 172.21.15.78:0/4013243284 shutdown_connections 2024-03-14T09:57:24.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.002+0000 7f1989200700 1 --2- 172.21.15.78:0/4013243284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1984199680 0x7f1984199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:24.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.002+0000 7f1989200700 1 --2- 172.21.15.78:0/4013243284 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1984100f20 0x7f198419f500 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:24.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.002+0000 7f1989200700 1 --2- 172.21.15.78:0/4013243284 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f19680a0d50 0x7f19680a3210 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:24.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.002+0000 7f1989200700 1 --2- 172.21.15.78:0/4013243284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1984106040 0x7f198419fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:24.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.002+0000 7f1989200700 1 -- 172.21.15.78:0/4013243284 >> 172.21.15.78:0/4013243284 conn(0x7f19840fc7a0 msgr2=0x7f19840fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:24.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.002+0000 7f1989200700 1 -- 172.21.15.78:0/4013243284 shutdown_connections 2024-03-14T09:57:24.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:24.002+0000 7f1989200700 1 -- 172.21.15.78:0/4013243284 wait complete. 2024-03-14T09:57:24.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:23 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T09:57:24.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:23 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:57:24.022 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:23 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T09:57:24.025 INFO:tasks.cephfs.filesystem:scrub status for tag:36005c13-5ff8-43ff-9d3b-f359504f334f - {'path': '/', 'tag': '36005c13-5ff8-43ff-9d3b-f359504f334f', 'options': 'recursive,force'} 2024-03-14T09:57:24.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:23 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T09:57:24.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:23 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:57:24.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:23 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T09:57:24.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:23 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T09:57:24.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:23 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:57:24.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:23 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T09:57:24.967 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0x9fd4ba 2024-03-14T09:57:25.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:24 smithi152 ceph-mon[48279]: pgmap v100: 129 pgs: 129 active+clean; 114 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 253 op/s 2024-03-14T09:57:25.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:24 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:57:25.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:24 smithi132 ceph-mon[47959]: pgmap v100: 129 pgs: 129 active+clean; 114 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 253 op/s 2024-03-14T09:57:25.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:24 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:57:25.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:24 smithi078 ceph-mon[38143]: pgmap v100: 129 pgs: 129 active+clean; 114 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 253 op/s 2024-03-14T09:57:25.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:24 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:57:25.651 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0x9fe323 2024-03-14T09:57:27.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:27 smithi078 ceph-mon[38143]: pgmap v101: 129 pgs: 129 active+clean; 121 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 451 op/s 2024-03-14T09:57:27.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:27 smithi152 ceph-mon[48279]: pgmap v101: 129 pgs: 129 active+clean; 121 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 451 op/s 2024-03-14T09:57:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:27 smithi132 ceph-mon[47959]: pgmap v101: 129 pgs: 129 active+clean; 121 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 451 op/s 2024-03-14T09:57:28.364 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0x9fec09 2024-03-14T09:57:29.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:28 smithi078 ceph-mon[38143]: pgmap v102: 129 pgs: 129 active+clean; 121 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 318 op/s 2024-03-14T09:57:29.248 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:28 smithi152 ceph-mon[48279]: pgmap v102: 129 pgs: 129 active+clean; 121 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 318 op/s 2024-03-14T09:57:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:28 smithi132 ceph-mon[47959]: pgmap v102: 129 pgs: 129 active+clean; 121 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 318 op/s 2024-03-14T09:57:30.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:29 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T09:57:30.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:29 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T09:57:30.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:29 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T09:57:30.744 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0x9ff6e7 2024-03-14T09:57:31.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:30 smithi078 ceph-mon[38143]: pgmap v103: 129 pgs: 129 active+clean; 121 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 318 op/s 2024-03-14T09:57:31.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:30 smithi152 ceph-mon[48279]: pgmap v103: 129 pgs: 129 active+clean; 121 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 318 op/s 2024-03-14T09:57:31.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:30 smithi132 ceph-mon[47959]: pgmap v103: 129 pgs: 129 active+clean; 121 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.0 MiB/s wr, 318 op/s 2024-03-14T09:57:33.088 INFO:tasks.workunit.client.0.smithi078.stdout:10000 copy from 0x2ab252 to 0x2b6f08, (0xbcb6 bytes) at 0x1cd955 2024-03-14T09:57:33.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:33 smithi078 ceph-mon[38143]: pgmap v104: 129 pgs: 129 active+clean; 127 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.2 MiB/s wr, 515 op/s 2024-03-14T09:57:33.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:33 smithi152 ceph-mon[48279]: pgmap v104: 129 pgs: 129 active+clean; 127 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.2 MiB/s wr, 515 op/s 2024-03-14T09:57:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:33 smithi132 ceph-mon[47959]: pgmap v104: 129 pgs: 129 active+clean; 127 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.2 MiB/s wr, 515 op/s 2024-03-14T09:57:35.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:35 smithi152 ceph-mon[48279]: pgmap v105: 129 pgs: 129 active+clean; 127 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.4 MiB/s wr, 394 op/s 2024-03-14T09:57:35.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:35 smithi132 ceph-mon[47959]: pgmap v105: 129 pgs: 129 active+clean; 127 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.4 MiB/s wr, 394 op/s 2024-03-14T09:57:35.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:35 smithi078 ceph-mon[38143]: pgmap v105: 129 pgs: 129 active+clean; 127 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.4 MiB/s wr, 394 op/s 2024-03-14T09:57:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:37 smithi152 ceph-mon[48279]: pgmap v106: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.5 MiB/s wr, 587 op/s 2024-03-14T09:57:37.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:37 smithi132 ceph-mon[47959]: pgmap v106: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.5 MiB/s wr, 587 op/s 2024-03-14T09:57:37.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:37 smithi078 ceph-mon[38143]: pgmap v106: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.5 MiB/s wr, 587 op/s 2024-03-14T09:57:39.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:38 smithi078 ceph-mon[38143]: pgmap v107: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 2.3 MiB/s wr, 389 op/s 2024-03-14T09:57:39.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:38 smithi152 ceph-mon[48279]: pgmap v107: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 2.3 MiB/s wr, 389 op/s 2024-03-14T09:57:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:38 smithi132 ceph-mon[47959]: pgmap v107: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 2.3 MiB/s wr, 389 op/s 2024-03-14T09:57:41.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:41 smithi078 ceph-mon[38143]: pgmap v108: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 2.3 MiB/s wr, 389 op/s 2024-03-14T09:57:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:41 smithi152 ceph-mon[48279]: pgmap v108: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 2.3 MiB/s wr, 389 op/s 2024-03-14T09:57:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:41 smithi132 ceph-mon[47959]: pgmap v108: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 2.3 MiB/s wr, 389 op/s 2024-03-14T09:57:43.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:43 smithi152 ceph-mon[48279]: pgmap v109: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.4 MiB/s wr, 578 op/s 2024-03-14T09:57:43.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:43 smithi132 ceph-mon[47959]: pgmap v109: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.4 MiB/s wr, 578 op/s 2024-03-14T09:57:43.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:43 smithi078 ceph-mon[38143]: pgmap v109: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.4 MiB/s wr, 578 op/s 2024-03-14T09:57:45.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:45 smithi152 ceph-mon[48279]: pgmap v110: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 2.2 MiB/s wr, 382 op/s 2024-03-14T09:57:45.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:45 smithi132 ceph-mon[47959]: pgmap v110: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 2.2 MiB/s wr, 382 op/s 2024-03-14T09:57:45.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:45 smithi078 ceph-mon[38143]: pgmap v110: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 2.2 MiB/s wr, 382 op/s 2024-03-14T09:57:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:47 smithi152 ceph-mon[48279]: pgmap v111: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.3 MiB/s wr, 564 op/s 2024-03-14T09:57:47.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:47 smithi132 ceph-mon[47959]: pgmap v111: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.3 MiB/s wr, 564 op/s 2024-03-14T09:57:47.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:47 smithi078 ceph-mon[38143]: pgmap v111: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.3 MiB/s wr, 564 op/s 2024-03-14T09:57:49.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:48 smithi078 ceph-mon[38143]: pgmap v112: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 2.2 MiB/s wr, 371 op/s 2024-03-14T09:57:49.248 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:48 smithi152 ceph-mon[48279]: pgmap v112: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 2.2 MiB/s wr, 371 op/s 2024-03-14T09:57:49.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:48 smithi132 ceph-mon[47959]: pgmap v112: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 2.2 MiB/s wr, 371 op/s 2024-03-14T09:57:51.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:51 smithi152 ceph-mon[48279]: pgmap v113: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 2.2 MiB/s wr, 372 op/s 2024-03-14T09:57:51.519 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:51 smithi132 ceph-mon[47959]: pgmap v113: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 2.2 MiB/s wr, 372 op/s 2024-03-14T09:57:51.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:51 smithi078 ceph-mon[38143]: pgmap v113: 129 pgs: 129 active+clean; 129 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 2.2 MiB/s wr, 372 op/s 2024-03-14T09:57:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:53 smithi152 ceph-mon[48279]: pgmap v114: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 3.2 MiB/s wr, 551 op/s 2024-03-14T09:57:53.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:53 smithi132 ceph-mon[47959]: pgmap v114: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 3.2 MiB/s wr, 551 op/s 2024-03-14T09:57:53.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:53 smithi078 ceph-mon[38143]: pgmap v114: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 3.2 MiB/s wr, 551 op/s 2024-03-14T09:57:54.026 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:57:54.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.270+0000 7f0b77400700 1 -- 172.21.15.78:0/3772470554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b700680b0 msgr2=0x7f0b70068490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:54.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.270+0000 7f0b77400700 1 --2- 172.21.15.78:0/3772470554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b700680b0 0x7f0b70068490 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009990 tx=0x7f0b64009ca0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.271+0000 7f0b77400700 1 -- 172.21.15.78:0/3772470554 shutdown_connections 2024-03-14T09:57:54.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.271+0000 7f0b77400700 1 --2- 172.21.15.78:0/3772470554 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b700680b0 0x7f0b70068490 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.271+0000 7f0b77400700 1 --2- 172.21.15.78:0/3772470554 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b70106060 0x7f0b7010dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.271+0000 7f0b77400700 1 --2- 172.21.15.78:0/3772470554 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b70068a60 0x7f0b70105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.271+0000 7f0b77400700 1 -- 172.21.15.78:0/3772470554 >> 172.21.15.78:0/3772470554 conn(0x7f0b70076d80 msgr2=0x7f0b70077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:54.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.271+0000 7f0b77400700 1 -- 172.21.15.78:0/3772470554 shutdown_connections 2024-03-14T09:57:54.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.271+0000 7f0b77400700 1 -- 172.21.15.78:0/3772470554 wait complete. 2024-03-14T09:57:54.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.272+0000 7f0b77400700 1 Processor -- start 2024-03-14T09:57:54.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.272+0000 7f0b77400700 1 -- start start 2024-03-14T09:57:54.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.273+0000 7f0b77400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b700680b0 0x7f0b700ffc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.273+0000 7f0b77400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b70106060 0x7f0b70100160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.273+0000 7f0b77400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b701006a0 0x7f0b701a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.273+0000 7f0b77400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b701a5db0 con 0x7f0b701006a0 2024-03-14T09:57:54.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.273+0000 7f0b77400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b701a5f20 con 0x7f0b700680b0 2024-03-14T09:57:54.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.273+0000 7f0b77400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b701a60e0 con 0x7f0b70106060 2024-03-14T09:57:54.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.273+0000 7f0b74a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b700680b0 0x7f0b700ffc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.273+0000 7f0b74a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b700680b0 0x7f0b700ffc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41310/0 (socket says 172.21.15.78:41310) 2024-03-14T09:57:54.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.273+0000 7f0b74a00700 1 -- 172.21.15.78:0/2966969687 learned_addr learned my addr 172.21.15.78:0/2966969687 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:54.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.273+0000 7f0b6fe00700 1 --2- 172.21.15.78:0/2966969687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b70106060 0x7f0b70100160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.273+0000 7f0b75400700 1 --2- 172.21.15.78:0/2966969687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b701006a0 0x7f0b701a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.274+0000 7f0b74a00700 1 -- 172.21.15.78:0/2966969687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b70106060 msgr2=0x7f0b70100160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:54.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.274+0000 7f0b74a00700 1 --2- 172.21.15.78:0/2966969687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b70106060 0x7f0b70100160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.274+0000 7f0b74a00700 1 -- 172.21.15.78:0/2966969687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b701006a0 msgr2=0x7f0b701a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:54.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.274+0000 7f0b74a00700 1 --2- 172.21.15.78:0/2966969687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b701006a0 0x7f0b701a57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.274+0000 7f0b74a00700 1 -- 172.21.15.78:0/2966969687 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b640095f0 con 0x7f0b700680b0 2024-03-14T09:57:54.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.274+0000 7f0b75400700 1 --2- 172.21.15.78:0/2966969687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b701006a0 0x7f0b701a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:57:54.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.274+0000 7f0b74a00700 1 --2- 172.21.15.78:0/2966969687 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b700680b0 0x7f0b700ffc20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0b64000c00 tx=0x7f0b64005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:54.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.274+0000 7f0b6d600700 1 -- 172.21.15.78:0/2966969687 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6401d070 con 0x7f0b700680b0 2024-03-14T09:57:54.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.274+0000 7f0b77400700 1 -- 172.21.15.78:0/2966969687 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b701a6360 con 0x7f0b700680b0 2024-03-14T09:57:54.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.274+0000 7f0b6d600700 1 -- 172.21.15.78:0/2966969687 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b64022470 con 0x7f0b700680b0 2024-03-14T09:57:54.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.274+0000 7f0b6d600700 1 -- 172.21.15.78:0/2966969687 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64017650 con 0x7f0b700680b0 2024-03-14T09:57:54.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.274+0000 7f0b77400700 1 -- 172.21.15.78:0/2966969687 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b701a67e0 con 0x7f0b700680b0 2024-03-14T09:57:54.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.278+0000 7f0b6d600700 1 -- 172.21.15.78:0/2966969687 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f0b640225e0 con 0x7f0b700680b0 2024-03-14T09:57:54.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.280+0000 7f0b6d600700 1 --2- 172.21.15.78:0/2966969687 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0b580a0c80 0x7f0b580a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.280+0000 7f0b6d600700 1 -- 172.21.15.78:0/2966969687 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f0b640cd9f0 con 0x7f0b700680b0 2024-03-14T09:57:54.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.280+0000 7f0b6fe00700 1 --2- 172.21.15.78:0/2966969687 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0b580a0c80 0x7f0b580a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.280+0000 7f0b8ae53b80 1 Processor -- start 2024-03-14T09:57:54.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.280+0000 7f0b8ae53b80 1 -- start start 2024-03-14T09:57:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.281+0000 7f0b6fe00700 1 --2- 172.21.15.78:0/2966969687 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0b580a0c80 0x7f0b580a3140 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0b70101350 tx=0x7f0b5c009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.281+0000 7f0b8ae53b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30b4680 0x5603b30b4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.281+0000 7f0b8ae53b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4fa0 0x5603b30be980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.281+0000 7f0b8ae53b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30beec0 0x5603b30c12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.281+0000 7f0b8ae53b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603b300d990 con 0x5603b30b4680 2024-03-14T09:57:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.281+0000 7f0b8ae53b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603b300b350 con 0x5603b30b4fa0 2024-03-14T09:57:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.281+0000 7f0b8ae53b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603b3006c40 con 0x5603b30beec0 2024-03-14T09:57:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.281+0000 7f0b6fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4fa0 0x5603b30be980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.281+0000 7f0b74a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30b4680 0x5603b30b4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.281+0000 7f0b6fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4fa0 0x5603b30be980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41326/0 (socket says 172.21.15.78:41326) 2024-03-14T09:57:54.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.281+0000 7f0b6fe00700 1 -- 172.21.15.78:0/713412022 learned_addr learned my addr 172.21.15.78:0/713412022 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.282+0000 7f0b74a00700 1 -- 172.21.15.78:0/713412022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30beec0 msgr2=0x5603b30c12b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-14T09:57:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.282+0000 7f0b75400700 1 --2- 172.21.15.78:0/713412022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30beec0 0x5603b30c12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.282+0000 7f0b74a00700 1 --2- 172.21.15.78:0/713412022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30beec0 0x5603b30c12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.282+0000 7f0b74a00700 1 -- 172.21.15.78:0/713412022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4fa0 msgr2=0x5603b30be980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.282+0000 7f0b74a00700 1 --2- 172.21.15.78:0/713412022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4fa0 0x5603b30be980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.282+0000 7f0b74a00700 1 -- 172.21.15.78:0/713412022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603b2e883b0 con 0x5603b30b4680 2024-03-14T09:57:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.282+0000 7f0b6fe00700 1 --2- 172.21.15.78:0/713412022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4fa0 0x5603b30be980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.282+0000 7f0b75400700 1 --2- 172.21.15.78:0/713412022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30beec0 0x5603b30c12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:57:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.282+0000 7f0b74a00700 1 --2- 172.21.15.78:0/713412022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30b4680 0x5603b30b4a60 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0b640cad90 tx=0x7f0b64005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964a4e84f7ab2726 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:54.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.282+0000 7f0b61c00700 1 -- 172.21.15.78:0/713412022 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6409b120 con 0x5603b30b4680 2024-03-14T09:57:54.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.282+0000 7f0b61c00700 1 -- 172.21.15.78:0/713412022 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b6409aa20 con 0x5603b30b4680 2024-03-14T09:57:54.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.283+0000 7f0b61c00700 1 -- 172.21.15.78:0/713412022 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6409aee0 con 0x5603b30b4680 2024-03-14T09:57:54.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.283+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/713412022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30b4680 msgr2=0x5603b30b4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:54.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.283+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/713412022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30b4680 0x5603b30b4a60 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0b640cad90 tx=0x7f0b64005fd0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.284+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/713412022 shutdown_connections 2024-03-14T09:57:54.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.284+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/713412022 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30beec0 0x5603b30c12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.284+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/713412022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30b4680 0x5603b30b4a60 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.284+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/713412022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4fa0 0x5603b30be980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.284+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/713412022 >> 172.21.15.78:0/713412022 conn(0x5603b302a220 msgr2=0x5603b30afaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:54.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.284+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/713412022 shutdown_connections 2024-03-14T09:57:54.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.284+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/713412022 wait complete. 2024-03-14T09:57:54.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.285+0000 7f0b8ae53b80 1 -- start start 2024-03-14T09:57:54.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.286+0000 7f0b8ae53b80 1 Processor -- start 2024-03-14T09:57:54.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.287+0000 7f0b8ae53b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4680 0x5603b30be8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.288+0000 7f0b8ae53b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4fa0 0x5603b30bbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.288+0000 7f0b8ae53b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30beec0 0x5603b30bc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.288+0000 7f0b8ae53b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603b2fa2d80 con 0x5603b30beec0 2024-03-14T09:57:54.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.288+0000 7f0b8ae53b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603b2fa2a20 con 0x5603b30b4fa0 2024-03-14T09:57:54.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.288+0000 7f0b8ae53b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603b2e8a6a0 con 0x5603b30b4680 2024-03-14T09:57:54.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.288+0000 7f0b8ae53b80 10 client.0 _ll_register_callbacks cb 0x5603b2e0e3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:54.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.288+0000 7f0b74a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4680 0x5603b30be8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.288+0000 7f0b6fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4fa0 0x5603b30bbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.288+0000 7f0b74a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4680 0x5603b30be8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59214/0 (socket says 172.21.15.78:59214) 2024-03-14T09:57:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.288+0000 7f0b6fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4fa0 0x5603b30bbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41330/0 (socket says 172.21.15.78:41330) 2024-03-14T09:57:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.288+0000 7f0b74a00700 1 -- 172.21.15.78:0/2496762467 learned_addr learned my addr 172.21.15.78:0/2496762467 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.288+0000 7f0b75400700 1 --2- 172.21.15.78:0/2496762467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30beec0 0x5603b30bc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.289+0000 7f0b74a00700 1 -- 172.21.15.78:0/2496762467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4fa0 msgr2=0x5603b30bbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.289+0000 7f0b74a00700 1 --2- 172.21.15.78:0/2496762467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4fa0 0x5603b30bbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.289+0000 7f0b74a00700 1 -- 172.21.15.78:0/2496762467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30beec0 msgr2=0x5603b30bc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.289+0000 7f0b74a00700 1 --2- 172.21.15.78:0/2496762467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30beec0 0x5603b30bc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.289+0000 7f0b74a00700 1 -- 172.21.15.78:0/2496762467 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5c017070 con 0x5603b30b4680 2024-03-14T09:57:54.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.289+0000 7f0b74a00700 1 --2- 172.21.15.78:0/2496762467 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4680 0x5603b30be8a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0b640cb310 tx=0x7f0b64022bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1708e52e35d96439 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:54.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.289+0000 7f0b56000700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:57:54.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.289+0000 7f0b56000700 1 -- 172.21.15.78:0/2496762467 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6409ae80 con 0x5603b30b4680 2024-03-14T09:57:54.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.289+0000 7f0b56000700 1 -- 172.21.15.78:0/2496762467 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f0b6409b120 con 0x5603b30b4680 2024-03-14T09:57:54.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.289+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/2496762467 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5603b30a7720 con 0x5603b30b4680 2024-03-14T09:57:54.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.290+0000 7f0b56000700 1 -- 172.21.15.78:0/2496762467 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b640a2980 con 0x5603b30b4680 2024-03-14T09:57:54.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.290+0000 7f0b8ae53b80 10 client.26380 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:54.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.290+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/2496762467 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f0b701a67e0 con 0x5603b30b4680 2024-03-14T09:57:54.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.290+0000 7f0b56000700 1 -- 172.21.15.78:0/2496762467 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b640b7070 con 0x5603b30b4680 2024-03-14T09:57:54.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.290+0000 7f0b56000700 1 -- 172.21.15.78:0/2496762467 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0b64099980 con 0x5603b30b4680 2024-03-14T09:57:54.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.290+0000 7f0b8ae53b80 10 client.26380 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:54.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.290+0000 7f0b8ae53b80 10 client.26380 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:57:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.290+0000 7f0b8ae53b80 20 client.26380 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.290+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/2496762467 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5603b30ce290 0x5603b30d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.290+0000 7f0b8ae53b80 4 client.26380 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:57:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.290+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/2496762467 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5603b2fff700 con 0x5603b30ce290 2024-03-14T09:57:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.291+0000 7f0b6fe00700 1 --2- 172.21.15.78:0/2496762467 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5603b30ce290 0x5603b30d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.291+0000 7f0b56000700 10 client.26380 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:54.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.291+0000 7f0b6fe00700 1 --2- 172.21.15.78:0/2496762467 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5603b30ce290 0x5603b30d06b0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2410074a205da1ae server_cookie=929d19cbc59f3ae7 in_seq=0 out_seq=0 2024-03-14T09:57:54.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.292+0000 7f0b56000700 1 -- 172.21.15.78:0/2496762467 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5603b2fff700 con 0x5603b30ce290 2024-03-14T09:57:54.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.292+0000 7f0b56000700 10 client.26380 handle_command_reply: tid=0 2024-03-14T09:57:54.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.292+0000 7f0b8ae53b80 1 client.26380 shutdown 2024-03-14T09:57:54.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.292+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/2496762467 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4680 msgr2=0x5603b30be8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:54.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.292+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/2496762467 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4680 0x5603b30be8a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0b640cb310 tx=0x7f0b64022bb0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.292+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/2496762467 shutdown_connections 2024-03-14T09:57:54.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.292+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/2496762467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30beec0 0x5603b30bc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.292+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/2496762467 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4680 0x5603b30be8a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.292+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/2496762467 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5603b30ce290 0x5603b30d06b0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.292+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/2496762467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4fa0 0x5603b30bbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.292+0000 7f0b56000700 0 client.26380 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:54.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.292+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/2496762467 >> 172.21.15.78:0/2496762467 conn(0x5603b302a220 msgr2=0x5603b30af690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:54.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.293+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/2496762467 shutdown_connections 2024-03-14T09:57:54.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.293+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/2496762467 wait complete. 2024-03-14T09:57:54.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.293+0000 7f0b8ae53b80 20 client.26380 trim_cache size 0 max 16384 2024-03-14T09:57:54.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.312+0000 7f0b8ae53b80 1 Processor -- start 2024-03-14T09:57:54.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.312+0000 7f0b8ae53b80 1 -- start start 2024-03-14T09:57:54.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.312+0000 7f0b8ae53b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4680 0x5603b30afc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.312+0000 7f0b8ae53b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4fa0 0x5603b30ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.312+0000 7f0b8ae53b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30beec0 0x5603b30bbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.312+0000 7f0b8ae53b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603b30bcba0 con 0x5603b30beec0 2024-03-14T09:57:54.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.312+0000 7f0b8ae53b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603b30a9d00 con 0x5603b30b4680 2024-03-14T09:57:54.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.312+0000 7f0b8ae53b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603b30bba00 con 0x5603b30b4fa0 2024-03-14T09:57:54.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.312+0000 7f0b74a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4680 0x5603b30afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.312+0000 7f0b74a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4680 0x5603b30afc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41342/0 (socket says 172.21.15.78:41342) 2024-03-14T09:57:54.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.312+0000 7f0b74a00700 1 -- 172.21.15.78:0/3840493373 learned_addr learned my addr 172.21.15.78:0/3840493373 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:54.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.313+0000 7f0b75400700 1 --2- 172.21.15.78:0/3840493373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30beec0 0x5603b30bbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.313+0000 7f0b6fe00700 1 --2- 172.21.15.78:0/3840493373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4fa0 0x5603b30ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.313+0000 7f0b74a00700 1 -- 172.21.15.78:0/3840493373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4fa0 msgr2=0x5603b30ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:54.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.313+0000 7f0b74a00700 1 --2- 172.21.15.78:0/3840493373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4fa0 0x5603b30ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.313+0000 7f0b74a00700 1 -- 172.21.15.78:0/3840493373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30beec0 msgr2=0x5603b30bbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:54.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.313+0000 7f0b74a00700 1 --2- 172.21.15.78:0/3840493373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30beec0 0x5603b30bbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.313+0000 7f0b74a00700 1 -- 172.21.15.78:0/3840493373 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603b2e34b60 con 0x5603b30b4680 2024-03-14T09:57:54.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.313+0000 7f0b74a00700 1 --2- 172.21.15.78:0/3840493373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4680 0x5603b30afc30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0b6409a9b0 tx=0x7f0b64022c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51e5745e16e117c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:54.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.313+0000 7f0b62600700 1 -- 172.21.15.78:0/3840493373 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6409a7f0 con 0x5603b30b4680 2024-03-14T09:57:54.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.313+0000 7f0b62600700 1 -- 172.21.15.78:0/3840493373 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5603b30a7720 con 0x5603b30b4680 2024-03-14T09:57:54.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.313+0000 7f0b62600700 1 -- 172.21.15.78:0/3840493373 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b640a2980 con 0x5603b30b4680 2024-03-14T09:57:54.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.314+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/3840493373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4680 msgr2=0x5603b30afc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:54.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.314+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/3840493373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4680 0x5603b30afc30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0b6409a9b0 tx=0x7f0b64022c20 comp rx=0 tx=0).stop 2024-03-14T09:57:54.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.314+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/3840493373 shutdown_connections 2024-03-14T09:57:54.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.314+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/3840493373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30b4680 0x5603b30afc30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.314+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/3840493373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30beec0 0x5603b30bbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.314+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/3840493373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4fa0 0x5603b30ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.314+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/3840493373 >> 172.21.15.78:0/3840493373 conn(0x5603b302a220 msgr2=0x5603b30b6f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:54.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.314+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/3840493373 shutdown_connections 2024-03-14T09:57:54.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.314+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/3840493373 wait complete. 2024-03-14T09:57:54.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.314+0000 7f0b8ae53b80 1 -- start start 2024-03-14T09:57:54.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.315+0000 7f0b8ae53b80 1 Processor -- start 2024-03-14T09:57:54.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.315+0000 7f0b8ae53b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4680 0x5603b30c0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.315+0000 7f0b8ae53b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30b4fa0 0x5603b30c1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.315+0000 7f0b8ae53b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30beec0 0x5603b311ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.315+0000 7f0b8ae53b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603b30c1c10 con 0x5603b30b4fa0 2024-03-14T09:57:54.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.315+0000 7f0b8ae53b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603b311f2b0 con 0x5603b30beec0 2024-03-14T09:57:54.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.315+0000 7f0b8ae53b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603b311f420 con 0x5603b30b4680 2024-03-14T09:57:54.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.315+0000 7f0b8ae53b80 10 client.0 _ll_register_callbacks cb 0x7f0b5c011a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:54.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.315+0000 7f0b6fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30b4fa0 0x5603b30c1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.315+0000 7f0b6fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30b4fa0 0x5603b30c1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58776/0 (socket says 172.21.15.78:58776) 2024-03-14T09:57:54.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.315+0000 7f0b6fe00700 1 -- 172.21.15.78:0/374117941 learned_addr learned my addr 172.21.15.78:0/374117941 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:54.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.316+0000 7f0b74a00700 1 --2- 172.21.15.78:0/374117941 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4680 0x5603b30c0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.316+0000 7f0b75400700 1 --2- 172.21.15.78:0/374117941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30beec0 0x5603b311ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.316+0000 7f0b6fe00700 1 -- 172.21.15.78:0/374117941 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4680 msgr2=0x5603b30c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:54.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.316+0000 7f0b6fe00700 1 --2- 172.21.15.78:0/374117941 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4680 0x5603b30c0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.316+0000 7f0b6fe00700 1 -- 172.21.15.78:0/374117941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30beec0 msgr2=0x5603b311ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:54.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.316+0000 7f0b6fe00700 1 --2- 172.21.15.78:0/374117941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30beec0 0x5603b311ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.316+0000 7f0b6fe00700 1 -- 172.21.15.78:0/374117941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b701a67e0 con 0x5603b30b4fa0 2024-03-14T09:57:54.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.316+0000 7f0b75400700 1 --2- 172.21.15.78:0/374117941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30beec0 0x5603b311ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:57:54.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.316+0000 7f0b6fe00700 1 --2- 172.21.15.78:0/374117941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30b4fa0 0x5603b30c1500 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x5603b30d0f50 tx=0x7f0b5c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa63008d09ea22ba server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:54.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.316+0000 7f0b56000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:57:54.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.317+0000 7f0b56000700 1 -- 172.21.15.78:0/374117941 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5c022070 con 0x5603b30b4fa0 2024-03-14T09:57:54.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.317+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/374117941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0b6409aa20 con 0x5603b30b4fa0 2024-03-14T09:57:54.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.317+0000 7f0b56000700 1 -- 172.21.15.78:0/374117941 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f0b5c003ce0 con 0x5603b30b4fa0 2024-03-14T09:57:54.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.317+0000 7f0b56000700 1 -- 172.21.15.78:0/374117941 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b5c003ea0 con 0x5603b30b4fa0 2024-03-14T09:57:54.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.317+0000 7f0b8ae53b80 10 client.16344 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:54.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.317+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/374117941 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5603b2fccea0 con 0x5603b30b4fa0 2024-03-14T09:57:54.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.317+0000 7f0b56000700 1 -- 172.21.15.78:0/374117941 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b6409aa20 con 0x5603b30b4fa0 2024-03-14T09:57:54.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.317+0000 7f0b56000700 1 -- 172.21.15.78:0/374117941 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0b5c033460 con 0x5603b30b4fa0 2024-03-14T09:57:54.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.318+0000 7f0b8ae53b80 10 client.16344 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:54.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.318+0000 7f0b8ae53b80 10 client.16344 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:57:54.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.318+0000 7f0b8ae53b80 20 client.16344 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:54.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.318+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/374117941 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5603b3120d00 0x5603b3123120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.318+0000 7f0b8ae53b80 4 client.16344 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:57:54.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.318+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/374117941 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5603b2e55cf0 con 0x5603b3120d00 2024-03-14T09:57:54.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.318+0000 7f0b74a00700 1 --2- 172.21.15.78:0/374117941 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5603b3120d00 0x5603b3123120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.318+0000 7f0b74a00700 1 --2- 172.21.15.78:0/374117941 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5603b3120d00 0x5603b3123120 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c8b3f2e219684e1 server_cookie=de0e57e48722c9d0 in_seq=0 out_seq=0 2024-03-14T09:57:54.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.318+0000 7f0b56000700 10 client.16344 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:54.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.319+0000 7f0b56000700 1 -- 172.21.15.78:0/374117941 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5603b2e55cf0 con 0x5603b3120d00 2024-03-14T09:57:54.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.319+0000 7f0b56000700 10 client.16344 handle_command_reply: tid=0 2024-03-14T09:57:54.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.319+0000 7f0b8ae53b80 1 client.16344 shutdown 2024-03-14T09:57:54.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.319+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/374117941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30b4fa0 msgr2=0x5603b30c1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:54.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.319+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/374117941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30b4fa0 0x5603b30c1500 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x5603b30d0f50 tx=0x7f0b5c0047c0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.319+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/374117941 shutdown_connections 2024-03-14T09:57:54.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.319+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/374117941 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5603b3120d00 0x5603b3123120 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.319+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/374117941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5603b30beec0 0x5603b311ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.319+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/374117941 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5603b30b4680 0x5603b30c0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.319+0000 7f0b56000700 0 client.16344 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:54.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.319+0000 7f0b8ae53b80 1 --2- 172.21.15.78:0/374117941 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5603b30b4fa0 0x5603b30c1500 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.319+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/374117941 >> 172.21.15.78:0/374117941 conn(0x5603b302a220 msgr2=0x5603b30bbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:54.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.319+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/374117941 shutdown_connections 2024-03-14T09:57:54.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.319+0000 7f0b8ae53b80 1 -- 172.21.15.78:0/374117941 wait complete. 2024-03-14T09:57:54.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.319+0000 7f0b8ae53b80 20 client.16344 trim_cache size 0 max 16384 2024-03-14T09:57:54.324 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:57:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T09:57:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T09:57:54.324 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:57:54.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.320+0000 7f0b77400700 1 -- 172.21.15.78:0/2966969687 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0b580a0c80 msgr2=0x7f0b580a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:54.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.320+0000 7f0b77400700 1 --2- 172.21.15.78:0/2966969687 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0b580a0c80 0x7f0b580a3140 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0b70101350 tx=0x7f0b5c009260 comp rx=0 tx=0).stop 2024-03-14T09:57:54.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.320+0000 7f0b77400700 1 -- 172.21.15.78:0/2966969687 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b700680b0 msgr2=0x7f0b700ffc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:54.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.320+0000 7f0b77400700 1 --2- 172.21.15.78:0/2966969687 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b700680b0 0x7f0b700ffc20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0b64000c00 tx=0x7f0b64005740 comp rx=0 tx=0).stop 2024-03-14T09:57:54.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.320+0000 7f0b77400700 1 -- 172.21.15.78:0/2966969687 shutdown_connections 2024-03-14T09:57:54.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.320+0000 7f0b77400700 1 --2- 172.21.15.78:0/2966969687 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0b580a0c80 0x7f0b580a3140 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.320+0000 7f0b77400700 1 --2- 172.21.15.78:0/2966969687 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b700680b0 0x7f0b700ffc20 secure :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0b64000c00 tx=0x7f0b64005740 comp rx=0 tx=0).stop 2024-03-14T09:57:54.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.320+0000 7f0b77400700 1 --2- 172.21.15.78:0/2966969687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b701006a0 0x7f0b701a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.320+0000 7f0b77400700 1 --2- 172.21.15.78:0/2966969687 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b70106060 0x7f0b70100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.320+0000 7f0b77400700 1 -- 172.21.15.78:0/2966969687 >> 172.21.15.78:0/2966969687 conn(0x7f0b70076d80 msgr2=0x7f0b700fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:54.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.320+0000 7f0b77400700 1 -- 172.21.15.78:0/2966969687 shutdown_connections 2024-03-14T09:57:54.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.320+0000 7f0b77400700 1 -- 172.21.15.78:0/2966969687 wait complete. 2024-03-14T09:57:54.343 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T09:57:54.343 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:57:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.585+0000 7f2837200700 1 -- 172.21.15.78:0/1299333433 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28301026a0 msgr2=0x7f283010ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.585+0000 7f2837200700 1 --2- 172.21.15.78:0/1299333433 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28301026a0 0x7f283010ab80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f281c009990 tx=0x7f281c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.585+0000 7f2837200700 1 -- 172.21.15.78:0/1299333433 shutdown_connections 2024-03-14T09:57:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.585+0000 7f2837200700 1 --2- 172.21.15.78:0/1299333433 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f283010b1a0 0x7f283010d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.585+0000 7f2837200700 1 --2- 172.21.15.78:0/1299333433 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2830101d80 0x7f2830102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.585+0000 7f2837200700 1 --2- 172.21.15.78:0/1299333433 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f28301026a0 0x7f283010ab80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.585+0000 7f2837200700 1 -- 172.21.15.78:0/1299333433 >> 172.21.15.78:0/1299333433 conn(0x7f28300fb5c0 msgr2=0x7f28300fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:54.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.585+0000 7f2837200700 1 -- 172.21.15.78:0/1299333433 shutdown_connections 2024-03-14T09:57:54.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.585+0000 7f2837200700 1 -- 172.21.15.78:0/1299333433 wait complete. 2024-03-14T09:57:54.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.586+0000 7f2837200700 1 Processor -- start 2024-03-14T09:57:54.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.587+0000 7f2837200700 1 -- start start 2024-03-14T09:57:54.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.587+0000 7f2837200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2830101d80 0x7f283010a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.587+0000 7f2837200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28301026a0 0x7f2830107ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.587+0000 7f2837200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f283010b1a0 0x7f2830107ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.587+0000 7f2837200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2830108780 con 0x7f28301026a0 2024-03-14T09:57:54.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.587+0000 7f2837200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28301088f0 con 0x7f2830101d80 2024-03-14T09:57:54.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.587+0000 7f2837200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28301aa030 con 0x7f283010b1a0 2024-03-14T09:57:54.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.587+0000 7f282fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2830101d80 0x7f283010a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.588+0000 7f282f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28301026a0 0x7f2830107ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.588+0000 7f2835200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f283010b1a0 0x7f2830107ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.588+0000 7f282fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2830101d80 0x7f283010a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41358/0 (socket says 172.21.15.78:41358) 2024-03-14T09:57:54.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.588+0000 7f282fe00700 1 -- 172.21.15.78:0/983277860 learned_addr learned my addr 172.21.15.78:0/983277860 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:54.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.588+0000 7f2835200700 1 -- 172.21.15.78:0/983277860 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2830101d80 msgr2=0x7f283010a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:54.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.588+0000 7f2835200700 1 --2- 172.21.15.78:0/983277860 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2830101d80 0x7f283010a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.588+0000 7f2835200700 1 -- 172.21.15.78:0/983277860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28301026a0 msgr2=0x7f2830107ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:54.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.588+0000 7f2835200700 1 --2- 172.21.15.78:0/983277860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28301026a0 0x7f2830107ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.588+0000 7f2835200700 1 -- 172.21.15.78:0/983277860 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c0095f0 con 0x7f283010b1a0 2024-03-14T09:57:54.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.588+0000 7f282fe00700 1 --2- 172.21.15.78:0/983277860 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2830101d80 0x7f283010a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:54.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.588+0000 7f2835200700 1 --2- 172.21.15.78:0/983277860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f283010b1a0 0x7f2830107ff0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2828009f60 tx=0x7f2828009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:54.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.588+0000 7f282cc00700 1 -- 172.21.15.78:0/983277860 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2828018070 con 0x7f283010b1a0 2024-03-14T09:57:54.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.588+0000 7f282cc00700 1 -- 172.21.15.78:0/983277860 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f282800f040 con 0x7f283010b1a0 2024-03-14T09:57:54.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.588+0000 7f2837200700 1 -- 172.21.15.78:0/983277860 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28301aa260 con 0x7f283010b1a0 2024-03-14T09:57:54.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.588+0000 7f282cc00700 1 -- 172.21.15.78:0/983277860 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282800ccc0 con 0x7f283010b1a0 2024-03-14T09:57:54.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.589+0000 7f2837200700 1 -- 172.21.15.78:0/983277860 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28301aa7c0 con 0x7f283010b1a0 2024-03-14T09:57:54.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.590+0000 7f2837200700 1 -- 172.21.15.78:0/983277860 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28301115e0 con 0x7f283010b1a0 2024-03-14T09:57:54.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.592+0000 7f282cc00700 1 -- 172.21.15.78:0/983277860 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f2828003a00 con 0x7f283010b1a0 2024-03-14T09:57:54.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.594+0000 7f282cc00700 1 --2- 172.21.15.78:0/983277860 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f28140a0c80 0x7f28140a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:54.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.594+0000 7f282cc00700 1 -- 172.21.15.78:0/983277860 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f28280cc340 con 0x7f283010b1a0 2024-03-14T09:57:54.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.594+0000 7f282fe00700 1 --2- 172.21.15.78:0/983277860 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f28140a0c80 0x7f28140a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:54.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.595+0000 7f282fe00700 1 --2- 172.21.15.78:0/983277860 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f28140a0c80 0x7f28140a3140 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2830109220 tx=0x7f2820009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:54.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.597+0000 7f282cc00700 1 -- 172.21.15.78:0/983277860 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f28280956b0 con 0x7f283010b1a0 2024-03-14T09:57:54.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.920+0000 7f2837200700 1 -- 172.21.15.78:0/983277860 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2830108f70 con 0x7f283010b1a0 2024-03-14T09:57:54.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.921+0000 7f282cc00700 1 -- 172.21.15.78:0/983277860 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f2828098cd0 con 0x7f283010b1a0 2024-03-14T09:57:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:57:54.922 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:57:54.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.924+0000 7f2837200700 1 -- 172.21.15.78:0/983277860 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f28140a0c80 msgr2=0x7f28140a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:54.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.924+0000 7f2837200700 1 --2- 172.21.15.78:0/983277860 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f28140a0c80 0x7f28140a3140 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2830109220 tx=0x7f2820009260 comp rx=0 tx=0).stop 2024-03-14T09:57:54.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.924+0000 7f2837200700 1 -- 172.21.15.78:0/983277860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f283010b1a0 msgr2=0x7f2830107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:54.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.924+0000 7f2837200700 1 --2- 172.21.15.78:0/983277860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f283010b1a0 0x7f2830107ff0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2828009f60 tx=0x7f2828009f90 comp rx=0 tx=0).stop 2024-03-14T09:57:54.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.924+0000 7f2837200700 1 -- 172.21.15.78:0/983277860 shutdown_connections 2024-03-14T09:57:54.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.924+0000 7f2837200700 1 --2- 172.21.15.78:0/983277860 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f28140a0c80 0x7f28140a3140 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.924+0000 7f2837200700 1 --2- 172.21.15.78:0/983277860 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2830101d80 0x7f283010a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.924+0000 7f2837200700 1 --2- 172.21.15.78:0/983277860 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f283010b1a0 0x7f2830107ff0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.924+0000 7f2837200700 1 --2- 172.21.15.78:0/983277860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28301026a0 0x7f2830107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:54.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.924+0000 7f2837200700 1 -- 172.21.15.78:0/983277860 >> 172.21.15.78:0/983277860 conn(0x7f28300fb5c0 msgr2=0x7f28300fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:54.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.924+0000 7f2837200700 1 -- 172.21.15.78:0/983277860 shutdown_connections 2024-03-14T09:57:54.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:54.924+0000 7f2837200700 1 -- 172.21.15.78:0/983277860 wait complete. 2024-03-14T09:57:54.926 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T09:57:54.947 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T09:57:55.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.184+0000 7efe03800700 1 -- 172.21.15.78:0/4038569309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efdfc0680b0 msgr2=0x7efdfc068490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.184+0000 7efe03800700 1 --2- 172.21.15.78:0/4038569309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efdfc0680b0 0x7efdfc068490 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efdf00099c0 tx=0x7efdf0009cd0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.184+0000 7efe03800700 1 -- 172.21.15.78:0/4038569309 shutdown_connections 2024-03-14T09:57:55.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.184+0000 7efe03800700 1 --2- 172.21.15.78:0/4038569309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdfc106060 0x7efdfc10dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.184+0000 7efe03800700 1 --2- 172.21.15.78:0/4038569309 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efdfc0680b0 0x7efdfc068490 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.184+0000 7efe03800700 1 --2- 172.21.15.78:0/4038569309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc068a60 0x7efdfc105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.184+0000 7efe03800700 1 -- 172.21.15.78:0/4038569309 >> 172.21.15.78:0/4038569309 conn(0x7efdfc076d80 msgr2=0x7efdfc077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.184+0000 7efe03800700 1 -- 172.21.15.78:0/4038569309 shutdown_connections 2024-03-14T09:57:55.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.184+0000 7efe03800700 1 -- 172.21.15.78:0/4038569309 wait complete. 2024-03-14T09:57:55.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.186+0000 7efe03800700 1 Processor -- start 2024-03-14T09:57:55.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.186+0000 7efe03800700 1 -- start start 2024-03-14T09:57:55.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.186+0000 7efe03800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdfc0680b0 0x7efdfc0ffc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.186+0000 7efe03800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efdfc106060 0x7efdfc100160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.186+0000 7efe03800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc1006a0 0x7efdfc1a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.186+0000 7efe03800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc1a5d50 con 0x7efdfc0680b0 2024-03-14T09:57:55.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.186+0000 7efe03800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc1a5ec0 con 0x7efdfc1006a0 2024-03-14T09:57:55.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.186+0000 7efe03800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc1a6080 con 0x7efdfc106060 2024-03-14T09:57:55.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.187+0000 7efe00e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdfc0680b0 0x7efdfc0ffc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.187+0000 7efdfbe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efdfc106060 0x7efdfc100160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.187+0000 7efe00e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdfc0680b0 0x7efdfc0ffc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58804/0 (socket says 172.21.15.78:58804) 2024-03-14T09:57:55.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.187+0000 7efe00e00700 1 -- 172.21.15.78:0/4056757949 learned_addr learned my addr 172.21.15.78:0/4056757949 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:55.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.187+0000 7efe01800700 1 --2- 172.21.15.78:0/4056757949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc1006a0 0x7efdfc1a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.187+0000 7efe00e00700 1 -- 172.21.15.78:0/4056757949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efdfc106060 msgr2=0x7efdfc100160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.187+0000 7efe00e00700 1 --2- 172.21.15.78:0/4056757949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efdfc106060 0x7efdfc100160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.187+0000 7efe00e00700 1 -- 172.21.15.78:0/4056757949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc1006a0 msgr2=0x7efdfc1a5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.187+0000 7efe00e00700 1 --2- 172.21.15.78:0/4056757949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc1006a0 0x7efdfc1a5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.187+0000 7efe00e00700 1 -- 172.21.15.78:0/4056757949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf00095f0 con 0x7efdfc0680b0 2024-03-14T09:57:55.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.187+0000 7efe00e00700 1 --2- 172.21.15.78:0/4056757949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdfc0680b0 0x7efdfc0ffc20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7efdf000b580 tx=0x7efdf000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.188+0000 7efdf9600700 1 -- 172.21.15.78:0/4056757949 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf001c070 con 0x7efdfc0680b0 2024-03-14T09:57:55.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.188+0000 7efdf9600700 1 -- 172.21.15.78:0/4056757949 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdf000fd80 con 0x7efdfc0680b0 2024-03-14T09:57:55.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.188+0000 7efe03800700 1 -- 172.21.15.78:0/4056757949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdfc1a6300 con 0x7efdfc0680b0 2024-03-14T09:57:55.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.188+0000 7efdf9600700 1 -- 172.21.15.78:0/4056757949 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0022c40 con 0x7efdfc0680b0 2024-03-14T09:57:55.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.188+0000 7efe03800700 1 -- 172.21.15.78:0/4056757949 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdfc1a6720 con 0x7efdfc0680b0 2024-03-14T09:57:55.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.192+0000 7efdf9600700 1 -- 172.21.15.78:0/4056757949 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7efdf0020070 con 0x7efdfc0680b0 2024-03-14T09:57:55.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.193+0000 7efe173dfb80 1 Processor -- start 2024-03-14T09:57:55.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.193+0000 7efdf9600700 1 --2- 172.21.15.78:0/4056757949 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efde40a0c00 0x7efde40a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.193+0000 7efe173dfb80 1 -- start start 2024-03-14T09:57:55.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.193+0000 7efdf9600700 1 -- 172.21.15.78:0/4056757949 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7efdf00cd470 con 0x7efdfc0680b0 2024-03-14T09:57:55.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.193+0000 7efe01800700 1 --2- 172.21.15.78:0/4056757949 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efde40a0c00 0x7efde40a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.193+0000 7efe173dfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83680 0x55d7e4e83a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efe173dfb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e96190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efe173dfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e966d0 0x55d7e4e98ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efe173dfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7e4ddc990 con 0x55d7e4e966d0 2024-03-14T09:57:55.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efe173dfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7e4dda350 con 0x55d7e4e83680 2024-03-14T09:57:55.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efe173dfb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7e4dd5c40 con 0x55d7e4e83fa0 2024-03-14T09:57:55.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efe00e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83680 0x55d7e4e83a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efe00e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83680 0x55d7e4e83a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41390/0 (socket says 172.21.15.78:41390) 2024-03-14T09:57:55.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efe00e00700 1 -- 172.21.15.78:0/3712323618 learned_addr learned my addr 172.21.15.78:0/3712323618 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:55.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efdfbe00700 1 --2- 172.21.15.78:0/3712323618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e96190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efe01800700 1 --2- 172.21.15.78:0/3712323618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e966d0 0x55d7e4e98ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efe00e00700 1 -- 172.21.15.78:0/3712323618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83fa0 msgr2=0x55d7e4e96190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efe00e00700 1 --2- 172.21.15.78:0/3712323618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e96190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efe00e00700 1 -- 172.21.15.78:0/3712323618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e966d0 msgr2=0x55d7e4e98ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efe00e00700 1 --2- 172.21.15.78:0/3712323618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e966d0 0x55d7e4e98ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efe00e00700 1 -- 172.21.15.78:0/3712323618 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d7e4c573b0 con 0x55d7e4e83680 2024-03-14T09:57:55.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efdfbe00700 1 --2- 172.21.15.78:0/3712323618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e96190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:57:55.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.194+0000 7efe00e00700 1 --2- 172.21.15.78:0/3712323618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83680 0x55d7e4e83a60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efdf009a510 tx=0x7efdf009a140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf912abfa8d96d31 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.195+0000 7efe01800700 1 --2- 172.21.15.78:0/4056757949 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efde40a0c00 0x7efde40a30c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efdf40117f0 tx=0x7efdf4011bb0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.195+0000 7efdee000700 1 -- 172.21.15.78:0/3712323618 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf009a580 con 0x55d7e4e83680 2024-03-14T09:57:55.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.195+0000 7efdee000700 1 -- 172.21.15.78:0/3712323618 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdf009e3b0 con 0x55d7e4e83680 2024-03-14T09:57:55.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.195+0000 7efdee000700 1 -- 172.21.15.78:0/3712323618 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf009a7e0 con 0x55d7e4e83680 2024-03-14T09:57:55.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.195+0000 7efe173dfb80 1 -- 172.21.15.78:0/3712323618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83680 msgr2=0x55d7e4e83a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.196+0000 7efe173dfb80 1 --2- 172.21.15.78:0/3712323618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83680 0x55d7e4e83a60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efdf009a510 tx=0x7efdf009a140 comp rx=0 tx=0).stop 2024-03-14T09:57:55.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.196+0000 7efe173dfb80 1 -- 172.21.15.78:0/3712323618 shutdown_connections 2024-03-14T09:57:55.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.196+0000 7efe173dfb80 1 --2- 172.21.15.78:0/3712323618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83680 0x55d7e4e83a60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.196+0000 7efe173dfb80 1 --2- 172.21.15.78:0/3712323618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e966d0 0x55d7e4e98ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.196+0000 7efe173dfb80 1 --2- 172.21.15.78:0/3712323618 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e96190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.196+0000 7efe173dfb80 1 -- 172.21.15.78:0/3712323618 >> 172.21.15.78:0/3712323618 conn(0x55d7e4df9220 msgr2=0x55d7e4e7eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.196+0000 7efe173dfb80 1 -- 172.21.15.78:0/3712323618 shutdown_connections 2024-03-14T09:57:55.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.196+0000 7efe173dfb80 1 -- 172.21.15.78:0/3712323618 wait complete. 2024-03-14T09:57:55.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.196+0000 7efe173dfb80 1 -- start start 2024-03-14T09:57:55.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.197+0000 7efe173dfb80 1 Processor -- start 2024-03-14T09:57:55.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.197+0000 7efe173dfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e83680 0x55d7e4e96060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.197+0000 7efe173dfb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e934f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.197+0000 7efe173dfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e966d0 0x55d7e4e93a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.197+0000 7efe173dfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7e4d71d80 con 0x55d7e4e83680 2024-03-14T09:57:55.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.197+0000 7efe173dfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7e4d71a20 con 0x55d7e4e966d0 2024-03-14T09:57:55.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.197+0000 7efe173dfb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7e4c596a0 con 0x55d7e4e83fa0 2024-03-14T09:57:55.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.197+0000 7efe173dfb80 10 client.0 _ll_register_callbacks cb 0x55d7e4bdd3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:55.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.198+0000 7efe00e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e83680 0x55d7e4e96060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.198+0000 7efe01800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e966d0 0x55d7e4e93a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.198+0000 7efe01800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e966d0 0x55d7e4e93a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41396/0 (socket says 172.21.15.78:41396) 2024-03-14T09:57:55.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.198+0000 7efe01800700 1 -- 172.21.15.78:0/294294640 learned_addr learned my addr 172.21.15.78:0/294294640 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:55.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.198+0000 7efdfbe00700 1 --2- 172.21.15.78:0/294294640 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e934f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.199+0000 7efe01800700 1 -- 172.21.15.78:0/294294640 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83fa0 msgr2=0x55d7e4e934f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.199+0000 7efe01800700 1 --2- 172.21.15.78:0/294294640 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e934f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.199+0000 7efe01800700 1 -- 172.21.15.78:0/294294640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e83680 msgr2=0x55d7e4e96060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.199+0000 7efe01800700 1 --2- 172.21.15.78:0/294294640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e83680 0x55d7e4e96060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.199+0000 7efe01800700 1 -- 172.21.15.78:0/294294640 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdfc1a6720 con 0x55d7e4e966d0 2024-03-14T09:57:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.199+0000 7efe01800700 1 --2- 172.21.15.78:0/294294640 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e966d0 0x55d7e4e93a70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efdf4016ac0 tx=0x7efdf4004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acf346527b520ef server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.199+0000 7efde2a00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:57:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.199+0000 7efde2a00700 1 -- 172.21.15.78:0/294294640 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf4006000 con 0x55d7e4e966d0 2024-03-14T09:57:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.199+0000 7efde2a00700 1 -- 172.21.15.78:0/294294640 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7efdf4016e80 con 0x55d7e4e966d0 2024-03-14T09:57:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.199+0000 7efe173dfb80 1 -- 172.21.15.78:0/294294640 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d7e4e76720 con 0x55d7e4e966d0 2024-03-14T09:57:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.200+0000 7efde2a00700 1 -- 172.21.15.78:0/294294640 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdf4019b10 con 0x55d7e4e966d0 2024-03-14T09:57:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.200+0000 7efe173dfb80 10 client.26399 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.200+0000 7efe173dfb80 1 -- 172.21.15.78:0/294294640 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d7e4c03b60 con 0x55d7e4e966d0 2024-03-14T09:57:55.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.200+0000 7efde2a00700 1 -- 172.21.15.78:0/294294640 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d7e4e76720 con 0x55d7e4e966d0 2024-03-14T09:57:55.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.201+0000 7efde2a00700 1 -- 172.21.15.78:0/294294640 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7efdf40243c0 con 0x55d7e4e966d0 2024-03-14T09:57:55.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.201+0000 7efe173dfb80 10 client.26399 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:55.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.201+0000 7efe173dfb80 10 client.26399 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:57:55.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.201+0000 7efe173dfb80 20 client.26399 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:55.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.201+0000 7efe173dfb80 1 --2- 172.21.15.78:0/294294640 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d7e4ea5f30 0x55d7e4ea8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.201+0000 7efe173dfb80 4 client.26399 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:57:55.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.201+0000 7efe173dfb80 1 -- 172.21.15.78:0/294294640 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d7e4dce700 con 0x55d7e4ea5f30 2024-03-14T09:57:55.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.201+0000 7efe00e00700 1 --2- 172.21.15.78:0/294294640 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d7e4ea5f30 0x55d7e4ea8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.202+0000 7efe00e00700 1 --2- 172.21.15.78:0/294294640 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d7e4ea5f30 0x55d7e4ea8350 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e25e54b0b49e86da server_cookie=ead445ee81290574 in_seq=0 out_seq=0 2024-03-14T09:57:55.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.202+0000 7efde2a00700 10 client.26399 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:57:55.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.202+0000 7efde2a00700 1 -- 172.21.15.78:0/294294640 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55d7e4dce700 con 0x55d7e4ea5f30 2024-03-14T09:57:55.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.202+0000 7efde2a00700 10 client.26399 handle_command_reply: tid=0 2024-03-14T09:57:55.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.202+0000 7efe173dfb80 1 client.26399 shutdown 2024-03-14T09:57:55.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.202+0000 7efe173dfb80 1 -- 172.21.15.78:0/294294640 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e966d0 msgr2=0x55d7e4e93a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.203+0000 7efe173dfb80 1 --2- 172.21.15.78:0/294294640 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e966d0 0x55d7e4e93a70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efdf4016ac0 tx=0x7efdf4004b40 comp rx=0 tx=0).stop 2024-03-14T09:57:55.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.203+0000 7efe173dfb80 1 -- 172.21.15.78:0/294294640 shutdown_connections 2024-03-14T09:57:55.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.203+0000 7efe173dfb80 1 --2- 172.21.15.78:0/294294640 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d7e4ea5f30 0x55d7e4ea8350 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.203+0000 7efe173dfb80 1 --2- 172.21.15.78:0/294294640 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e966d0 0x55d7e4e93a70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.203+0000 7efe173dfb80 1 --2- 172.21.15.78:0/294294640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e83680 0x55d7e4e96060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.203+0000 7efe173dfb80 1 --2- 172.21.15.78:0/294294640 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e934f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.203+0000 7efde2a00700 0 client.26399 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:57:55.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.203+0000 7efe173dfb80 1 -- 172.21.15.78:0/294294640 >> 172.21.15.78:0/294294640 conn(0x55d7e4df9220 msgr2=0x55d7e4e7e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.203+0000 7efe173dfb80 1 -- 172.21.15.78:0/294294640 shutdown_connections 2024-03-14T09:57:55.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.203+0000 7efe173dfb80 1 -- 172.21.15.78:0/294294640 wait complete. 2024-03-14T09:57:55.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.203+0000 7efe173dfb80 20 client.26399 trim_cache size 0 max 16384 2024-03-14T09:57:55.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.219+0000 7efe173dfb80 1 Processor -- start 2024-03-14T09:57:55.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.220+0000 7efe173dfb80 1 -- start start 2024-03-14T09:57:55.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.220+0000 7efe173dfb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83680 0x55d7e4e7ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.220+0000 7efe173dfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e8e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.220+0000 7efe173dfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e966d0 0x55d7e4e8ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.220+0000 7efe173dfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7e4ea2a30 con 0x55d7e4e966d0 2024-03-14T09:57:55.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.220+0000 7efe173dfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7e4e84af0 con 0x55d7e4e83fa0 2024-03-14T09:57:55.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.220+0000 7efe173dfb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7e4e94870 con 0x55d7e4e83680 2024-03-14T09:57:55.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.221+0000 7efdfbe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e8e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.221+0000 7efe01800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e966d0 0x55d7e4e8ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.221+0000 7efdfbe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e8e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41404/0 (socket says 172.21.15.78:41404) 2024-03-14T09:57:55.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.221+0000 7efdfbe00700 1 -- 172.21.15.78:0/3220032993 learned_addr learned my addr 172.21.15.78:0/3220032993 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.221+0000 7efe00e00700 1 --2- 172.21.15.78:0/3220032993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83680 0x55d7e4e7ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.221+0000 7efdfbe00700 1 -- 172.21.15.78:0/3220032993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83680 msgr2=0x55d7e4e7ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.221+0000 7efdfbe00700 1 --2- 172.21.15.78:0/3220032993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83680 0x55d7e4e7ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.221+0000 7efdfbe00700 1 -- 172.21.15.78:0/3220032993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e966d0 msgr2=0x55d7e4e8ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.221+0000 7efdfbe00700 1 --2- 172.21.15.78:0/3220032993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e966d0 0x55d7e4e8ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.221+0000 7efdfbe00700 1 -- 172.21.15.78:0/3220032993 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d7e4d9bea0 con 0x55d7e4e83fa0 2024-03-14T09:57:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.221+0000 7efe00e00700 1 --2- 172.21.15.78:0/3220032993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83680 0x55d7e4e7ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:57:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.222+0000 7efdfbe00700 1 --2- 172.21.15.78:0/3220032993 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e8e7d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efde8009d00 tx=0x7efde8010710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73dfb9bafab82028 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.222+0000 7efdeea00700 1 -- 172.21.15.78:0/3220032993 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde8010df0 con 0x55d7e4e83fa0 2024-03-14T09:57:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.222+0000 7efdeea00700 1 -- 172.21.15.78:0/3220032993 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efde8018040 con 0x55d7e4e83fa0 2024-03-14T09:57:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.222+0000 7efdeea00700 1 -- 172.21.15.78:0/3220032993 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde801c3f0 con 0x55d7e4e83fa0 2024-03-14T09:57:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.223+0000 7efe173dfb80 1 -- 172.21.15.78:0/3220032993 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83fa0 msgr2=0x55d7e4e8e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.223+0000 7efe173dfb80 1 --2- 172.21.15.78:0/3220032993 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e8e7d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efde8009d00 tx=0x7efde8010710 comp rx=0 tx=0).stop 2024-03-14T09:57:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.223+0000 7efe173dfb80 1 -- 172.21.15.78:0/3220032993 shutdown_connections 2024-03-14T09:57:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.223+0000 7efe173dfb80 1 --2- 172.21.15.78:0/3220032993 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e966d0 0x55d7e4e8ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.223+0000 7efe173dfb80 1 --2- 172.21.15.78:0/3220032993 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e83680 0x55d7e4e7ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.223+0000 7efe173dfb80 1 --2- 172.21.15.78:0/3220032993 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e8e7d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.223+0000 7efe173dfb80 1 -- 172.21.15.78:0/3220032993 >> 172.21.15.78:0/3220032993 conn(0x55d7e4df9220 msgr2=0x55d7e4e7d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.223+0000 7efe173dfb80 1 -- 172.21.15.78:0/3220032993 shutdown_connections 2024-03-14T09:57:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.223+0000 7efe173dfb80 1 -- 172.21.15.78:0/3220032993 wait complete. 2024-03-14T09:57:55.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.224+0000 7efe173dfb80 1 -- start start 2024-03-14T09:57:55.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.224+0000 7efe173dfb80 1 Processor -- start 2024-03-14T09:57:55.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.224+0000 7efe173dfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e83680 0x55d7e4e98930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.224+0000 7efe173dfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e98e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.224+0000 7efe173dfb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e966d0 0x55d7e4ef6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.225+0000 7efe173dfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7e4ef6f90 con 0x55d7e4e83680 2024-03-14T09:57:55.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.225+0000 7efe173dfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7e4ef7100 con 0x55d7e4e83fa0 2024-03-14T09:57:55.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.225+0000 7efe173dfb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7e4ef72c0 con 0x55d7e4e966d0 2024-03-14T09:57:55.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.225+0000 7efe173dfb80 10 client.0 _ll_register_callbacks cb 0x7efdf4007550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:55.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.225+0000 7efdfbe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e98e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.225+0000 7efdfbe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e98e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41416/0 (socket says 172.21.15.78:41416) 2024-03-14T09:57:55.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.225+0000 7efdfbe00700 1 -- 172.21.15.78:0/3909112471 learned_addr learned my addr 172.21.15.78:0/3909112471 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:55.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.225+0000 7efe00e00700 1 --2- 172.21.15.78:0/3909112471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e83680 0x55d7e4e98930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.225+0000 7efe01800700 1 --2- 172.21.15.78:0/3909112471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e966d0 0x55d7e4ef6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.225+0000 7efdfbe00700 1 -- 172.21.15.78:0/3909112471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e966d0 msgr2=0x55d7e4ef6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.225+0000 7efdfbe00700 1 --2- 172.21.15.78:0/3909112471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e966d0 0x55d7e4ef6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.225+0000 7efdfbe00700 1 -- 172.21.15.78:0/3909112471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e83680 msgr2=0x55d7e4e98930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.225+0000 7efdfbe00700 1 --2- 172.21.15.78:0/3909112471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e83680 0x55d7e4e98930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.225+0000 7efdfbe00700 1 -- 172.21.15.78:0/3909112471 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde80099e0 con 0x55d7e4e83fa0 2024-03-14T09:57:55.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.226+0000 7efdfbe00700 1 --2- 172.21.15.78:0/3909112471 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e98e70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efde8010a60 tx=0x7efde8004330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8c4a03306d33275 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.226+0000 7efde2a00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:57:55.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.226+0000 7efde2a00700 1 -- 172.21.15.78:0/3909112471 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde8005460 con 0x55d7e4e83fa0 2024-03-14T09:57:55.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.226+0000 7efde2a00700 1 -- 172.21.15.78:0/3909112471 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7efde8009e50 con 0x55d7e4e83fa0 2024-03-14T09:57:55.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.226+0000 7efe173dfb80 1 -- 172.21.15.78:0/3909112471 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efdf009e3b0 con 0x55d7e4e83fa0 2024-03-14T09:57:55.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.226+0000 7efde2a00700 1 -- 172.21.15.78:0/3909112471 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7efde80055c0 con 0x55d7e4e83fa0 2024-03-14T09:57:55.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.226+0000 7efe173dfb80 10 client.26405 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:55.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.226+0000 7efe173dfb80 1 -- 172.21.15.78:0/3909112471 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d7e4c5bcd0 con 0x55d7e4e83fa0 2024-03-14T09:57:55.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.227+0000 7efde2a00700 1 -- 172.21.15.78:0/3909112471 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdf009e3b0 con 0x55d7e4e83fa0 2024-03-14T09:57:55.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.227+0000 7efde2a00700 1 -- 172.21.15.78:0/3909112471 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7efde8026440 con 0x55d7e4e83fa0 2024-03-14T09:57:55.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.227+0000 7efe173dfb80 10 client.26405 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:55.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.227+0000 7efe173dfb80 10 client.26405 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:57:55.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.227+0000 7efe173dfb80 20 client.26405 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:55.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.227+0000 7efe173dfb80 1 --2- 172.21.15.78:0/3909112471 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d7e4ef88a0 0x55d7e4efacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.227+0000 7efe173dfb80 4 client.26405 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:57:55.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.227+0000 7efe173dfb80 1 -- 172.21.15.78:0/3909112471 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d7e4c24cf0 con 0x55d7e4ef88a0 2024-03-14T09:57:55.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.227+0000 7efe00e00700 1 --2- 172.21.15.78:0/3909112471 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d7e4ef88a0 0x55d7e4efacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.228+0000 7efe00e00700 1 --2- 172.21.15.78:0/3909112471 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d7e4ef88a0 0x55d7e4efacc0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3c05aa59bfd00ce2 server_cookie=1fcb68e2c778dce2 in_seq=0 out_seq=0 2024-03-14T09:57:55.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.228+0000 7efde2a00700 10 client.26405 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:57:55.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.228+0000 7efde2a00700 1 -- 172.21.15.78:0/3909112471 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d7e4c24cf0 con 0x55d7e4ef88a0 2024-03-14T09:57:55.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.228+0000 7efde2a00700 10 client.26405 handle_command_reply: tid=0 2024-03-14T09:57:55.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.228+0000 7efe173dfb80 1 client.26405 shutdown 2024-03-14T09:57:55.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.228+0000 7efe173dfb80 1 -- 172.21.15.78:0/3909112471 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83fa0 msgr2=0x55d7e4e98e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.228+0000 7efe173dfb80 1 --2- 172.21.15.78:0/3909112471 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e98e70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efde8010a60 tx=0x7efde8004330 comp rx=0 tx=0).stop 2024-03-14T09:57:55.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.229+0000 7efe173dfb80 1 -- 172.21.15.78:0/3909112471 shutdown_connections 2024-03-14T09:57:55.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.229+0000 7efe173dfb80 1 --2- 172.21.15.78:0/3909112471 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d7e4ef88a0 0x55d7e4efacc0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.229+0000 7efe173dfb80 1 --2- 172.21.15.78:0/3909112471 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d7e4e966d0 0x55d7e4ef6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.229+0000 7efe173dfb80 1 --2- 172.21.15.78:0/3909112471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d7e4e83680 0x55d7e4e98930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.229+0000 7efde2a00700 0 client.26405 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:57:55.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.229+0000 7efe173dfb80 1 --2- 172.21.15.78:0/3909112471 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d7e4e83fa0 0x55d7e4e98e70 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.229+0000 7efe173dfb80 1 -- 172.21.15.78:0/3909112471 >> 172.21.15.78:0/3909112471 conn(0x55d7e4df9220 msgr2=0x55d7e4e933e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.229+0000 7efe173dfb80 1 -- 172.21.15.78:0/3909112471 shutdown_connections 2024-03-14T09:57:55.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.229+0000 7efe173dfb80 1 -- 172.21.15.78:0/3909112471 wait complete. 2024-03-14T09:57:55.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.229+0000 7efe173dfb80 20 client.26405 trim_cache size 0 max 16384 2024-03-14T09:57:55.234 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:57:55.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.230+0000 7efe03800700 1 -- 172.21.15.78:0/4056757949 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efde40a0c00 msgr2=0x7efde40a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.230+0000 7efe03800700 1 --2- 172.21.15.78:0/4056757949 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efde40a0c00 0x7efde40a30c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efdf40117f0 tx=0x7efdf4011bb0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.230+0000 7efe03800700 1 -- 172.21.15.78:0/4056757949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdfc0680b0 msgr2=0x7efdfc0ffc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.230+0000 7efe03800700 1 --2- 172.21.15.78:0/4056757949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdfc0680b0 0x7efdfc0ffc20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7efdf000b580 tx=0x7efdf000f690 comp rx=0 tx=0).stop 2024-03-14T09:57:55.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.230+0000 7efe03800700 1 -- 172.21.15.78:0/4056757949 shutdown_connections 2024-03-14T09:57:55.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.230+0000 7efe03800700 1 --2- 172.21.15.78:0/4056757949 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efde40a0c00 0x7efde40a30c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.230+0000 7efe03800700 1 --2- 172.21.15.78:0/4056757949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efdfc1006a0 0x7efdfc1a5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.230+0000 7efe03800700 1 --2- 172.21.15.78:0/4056757949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdfc0680b0 0x7efdfc0ffc20 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.230+0000 7efe03800700 1 --2- 172.21.15.78:0/4056757949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efdfc106060 0x7efdfc100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.230+0000 7efe03800700 1 -- 172.21.15.78:0/4056757949 >> 172.21.15.78:0/4056757949 conn(0x7efdfc076d80 msgr2=0x7efdfc0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.230+0000 7efe03800700 1 -- 172.21.15.78:0/4056757949 shutdown_connections 2024-03-14T09:57:55.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.230+0000 7efe03800700 1 -- 172.21.15.78:0/4056757949 wait complete. 2024-03-14T09:57:55.249 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:55 smithi078 ceph-mon[38143]: pgmap v115: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 2.1 MiB/s wr, 361 op/s 2024-03-14T09:57:55.249 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:55 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/983277860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:57:55.252 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T09:57:55.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.489+0000 7f304f200700 1 -- 172.21.15.78:0/190798162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3050101cc0 msgr2=0x7f30501020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.489+0000 7f304f200700 1 --2- 172.21.15.78:0/190798162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3050101cc0 0x7f30501020a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f30400098f0 tx=0x7f3040009c00 comp rx=0 tx=0).stop 2024-03-14T09:57:55.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.489+0000 7f304f200700 1 -- 172.21.15.78:0/190798162 shutdown_connections 2024-03-14T09:57:55.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.489+0000 7f304f200700 1 --2- 172.21.15.78:0/190798162 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3050101cc0 0x7f30501020a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.489+0000 7f304f200700 1 --2- 172.21.15.78:0/190798162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f305010b0e0 0x7f305010d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.489+0000 7f304f200700 1 --2- 172.21.15.78:0/190798162 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f30501025e0 0x7f305010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.489+0000 7f304f200700 1 -- 172.21.15.78:0/190798162 >> 172.21.15.78:0/190798162 conn(0x7f30500fb4e0 msgr2=0x7f30500fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.489+0000 7f304f200700 1 -- 172.21.15.78:0/190798162 shutdown_connections 2024-03-14T09:57:55.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.489+0000 7f304f200700 1 -- 172.21.15.78:0/190798162 wait complete. 2024-03-14T09:57:55.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.490+0000 7f304f200700 1 Processor -- start 2024-03-14T09:57:55.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.490+0000 7f304f200700 1 -- start start 2024-03-14T09:57:55.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.491+0000 7f304f200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3050101cc0 0x7f30500ff910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.491+0000 7f304f200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30501025e0 0x7f30500ffe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.491+0000 7f304f200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f305010b0e0 0x7f3050100390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.491+0000 7f304f200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30501a5810 con 0x7f30501025e0 2024-03-14T09:57:55.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.491+0000 7f304f200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30501a5980 con 0x7f3050101cc0 2024-03-14T09:57:55.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.491+0000 7f304f200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30501a5b40 con 0x7f305010b0e0 2024-03-14T09:57:55.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.491+0000 7f304de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3050101cc0 0x7f30500ff910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.491+0000 7f304e800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f305010b0e0 0x7f3050100390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.491+0000 7f304de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3050101cc0 0x7f30500ff910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41428/0 (socket says 172.21.15.78:41428) 2024-03-14T09:57:55.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.491+0000 7f304de00700 1 -- 172.21.15.78:0/345786758 learned_addr learned my addr 172.21.15.78:0/345786758 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:55.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.491+0000 7f304d400700 1 --2- 172.21.15.78:0/345786758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30501025e0 0x7f30500ffe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.492+0000 7f304de00700 1 -- 172.21.15.78:0/345786758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f305010b0e0 msgr2=0x7f3050100390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.492+0000 7f304de00700 1 --2- 172.21.15.78:0/345786758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f305010b0e0 0x7f3050100390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.492+0000 7f304de00700 1 -- 172.21.15.78:0/345786758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30501025e0 msgr2=0x7f30500ffe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.492+0000 7f304de00700 1 --2- 172.21.15.78:0/345786758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30501025e0 0x7f30500ffe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.492+0000 7f304de00700 1 -- 172.21.15.78:0/345786758 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3040009520 con 0x7f3050101cc0 2024-03-14T09:57:55.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.492+0000 7f304e800700 1 --2- 172.21.15.78:0/345786758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f305010b0e0 0x7f3050100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:55.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.492+0000 7f304de00700 1 --2- 172.21.15.78:0/345786758 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3050101cc0 0x7f30500ff910 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3040005950 tx=0x7f304000fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.492+0000 7f3046a00700 1 -- 172.21.15.78:0/345786758 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304001c070 con 0x7f3050101cc0 2024-03-14T09:57:55.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.492+0000 7f3046a00700 1 -- 172.21.15.78:0/345786758 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f304000baa0 con 0x7f3050101cc0 2024-03-14T09:57:55.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.492+0000 7f304f200700 1 -- 172.21.15.78:0/345786758 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30501a5dc0 con 0x7f3050101cc0 2024-03-14T09:57:55.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.492+0000 7f3046a00700 1 -- 172.21.15.78:0/345786758 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30400038e0 con 0x7f3050101cc0 2024-03-14T09:57:55.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.492+0000 7f304f200700 1 -- 172.21.15.78:0/345786758 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30501a6210 con 0x7f3050101cc0 2024-03-14T09:57:55.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.497+0000 7f3046a00700 1 -- 172.21.15.78:0/345786758 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f3040020030 con 0x7f3050101cc0 2024-03-14T09:57:55.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:55 smithi152 ceph-mon[48279]: pgmap v115: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 2.1 MiB/s wr, 361 op/s 2024-03-14T09:57:55.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:55 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/983277860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:57:55.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.499+0000 7f3068cafb80 1 Processor -- start 2024-03-14T09:57:55.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.499+0000 7f3068cafb80 1 -- start start 2024-03-14T09:57:55.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.500+0000 7f3068cafb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650070 0x557eb4650450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.500+0000 7f3068cafb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4650990 0x557eb465b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.500+0000 7f3068cafb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 0x557eb465dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.500+0000 7f3068cafb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eb45a1990 con 0x557eb4650070 2024-03-14T09:57:55.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.500+0000 7f3068cafb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eb459f350 con 0x557eb4650990 2024-03-14T09:57:55.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.500+0000 7f3068cafb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eb459ac40 con 0x557eb465b840 2024-03-14T09:57:55.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.500+0000 7f304de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4650990 0x557eb465b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.500+0000 7f304de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4650990 0x557eb465b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41444/0 (socket says 172.21.15.78:41444) 2024-03-14T09:57:55.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.500+0000 7f304de00700 1 -- 172.21.15.78:0/1437200104 learned_addr learned my addr 172.21.15.78:0/1437200104 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:55.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.500+0000 7f3046a00700 1 --2- 172.21.15.78:0/345786758 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f30340a0c80 0x7f30340a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.501+0000 7f304de00700 1 -- 172.21.15.78:0/1437200104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 msgr2=0x557eb465dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.501+0000 7f304e800700 1 --2- 172.21.15.78:0/1437200104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650070 0x557eb4650450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.501+0000 7f3046a00700 1 -- 172.21.15.78:0/345786758 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f30400cd550 con 0x7f3050101cc0 2024-03-14T09:57:55.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.501+0000 7f304d400700 1 --2- 172.21.15.78:0/1437200104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 0x557eb465dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.501+0000 7f304de00700 1 --2- 172.21.15.78:0/1437200104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 0x557eb465dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.501+0000 7f304de00700 1 -- 172.21.15.78:0/1437200104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650070 msgr2=0x557eb4650450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.501+0000 7f304de00700 1 --2- 172.21.15.78:0/1437200104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650070 0x557eb4650450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.501+0000 7f304de00700 1 -- 172.21.15.78:0/1437200104 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557eb441c3b0 con 0x557eb4650990 2024-03-14T09:57:55.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.501+0000 7f304e800700 1 --2- 172.21.15.78:0/1437200104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650070 0x557eb4650450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:57:55.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.502+0000 7f304e800700 1 --2- 172.21.15.78:0/345786758 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f30340a0c80 0x7f30340a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.502+0000 7f304de00700 1 --2- 172.21.15.78:0/1437200104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4650990 0x557eb465b300 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f30400ca860 tx=0x7f3040009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b58b767fd7f68869 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.502+0000 7f304e800700 1 --2- 172.21.15.78:0/345786758 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f30340a0c80 0x7f30340a3140 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3048009fd0 tx=0x7f3048004ad0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.502+0000 7f303b400700 1 -- 172.21.15.78:0/1437200104 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304000bc10 con 0x557eb4650990 2024-03-14T09:57:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.503+0000 7f303b400700 1 -- 172.21.15.78:0/1437200104 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f304009a0c0 con 0x557eb4650990 2024-03-14T09:57:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.503+0000 7f303b400700 1 -- 172.21.15.78:0/1437200104 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304009e3b0 con 0x557eb4650990 2024-03-14T09:57:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.503+0000 7f3068cafb80 1 -- 172.21.15.78:0/1437200104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4650990 msgr2=0x557eb465b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.503+0000 7f3068cafb80 1 --2- 172.21.15.78:0/1437200104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4650990 0x557eb465b300 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f30400ca860 tx=0x7f3040009260 comp rx=0 tx=0).stop 2024-03-14T09:57:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.503+0000 7f3068cafb80 1 -- 172.21.15.78:0/1437200104 shutdown_connections 2024-03-14T09:57:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.503+0000 7f3068cafb80 1 --2- 172.21.15.78:0/1437200104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 0x557eb465dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.503+0000 7f3068cafb80 1 --2- 172.21.15.78:0/1437200104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650070 0x557eb4650450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.503+0000 7f3068cafb80 1 --2- 172.21.15.78:0/1437200104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4650990 0x557eb465b300 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.503+0000 7f3068cafb80 1 -- 172.21.15.78:0/1437200104 >> 172.21.15.78:0/1437200104 conn(0x557eb45be220 msgr2=0x557eb4643c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.504+0000 7f3068cafb80 1 -- 172.21.15.78:0/1437200104 shutdown_connections 2024-03-14T09:57:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.504+0000 7f3068cafb80 1 -- 172.21.15.78:0/1437200104 wait complete. 2024-03-14T09:57:55.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.504+0000 7f3068cafb80 1 -- start start 2024-03-14T09:57:55.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.504+0000 7f3068cafb80 1 Processor -- start 2024-03-14T09:57:55.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.505+0000 7f3068cafb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650070 0x557eb465bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.505+0000 7f3068cafb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4650990 0x557eb465c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.505+0000 7f3068cafb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 0x557eb4666ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.505+0000 7f3068cafb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eb4536d80 con 0x557eb4650070 2024-03-14T09:57:55.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.505+0000 7f3068cafb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eb4536a20 con 0x557eb4650990 2024-03-14T09:57:55.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.505+0000 7f3068cafb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eb441e6a0 con 0x557eb465b840 2024-03-14T09:57:55.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.505+0000 7f3068cafb80 10 client.0 _ll_register_callbacks cb 0x557eb43a23a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:55.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.505+0000 7f304d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4650990 0x557eb465c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.505+0000 7f304e800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 0x557eb4666ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.505+0000 7f304d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4650990 0x557eb465c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41458/0 (socket says 172.21.15.78:41458) 2024-03-14T09:57:55.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.505+0000 7f304d400700 1 -- 172.21.15.78:0/2427611346 learned_addr learned my addr 172.21.15.78:0/2427611346 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:55.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.505+0000 7f304de00700 1 --2- 172.21.15.78:0/2427611346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650070 0x557eb465bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.506+0000 7f304de00700 1 -- 172.21.15.78:0/2427611346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 msgr2=0x557eb4666ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.506+0000 7f304de00700 1 --2- 172.21.15.78:0/2427611346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 0x557eb4666ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.506+0000 7f304de00700 1 -- 172.21.15.78:0/2427611346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4650990 msgr2=0x557eb465c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.506+0000 7f304de00700 1 --2- 172.21.15.78:0/2427611346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4650990 0x557eb465c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.506+0000 7f304de00700 1 -- 172.21.15.78:0/2427611346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30501a6210 con 0x557eb4650070 2024-03-14T09:57:55.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.506+0000 7f304d400700 1 --2- 172.21.15.78:0/2427611346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4650990 0x557eb465c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:55.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.506+0000 7f304e800700 1 --2- 172.21.15.78:0/2427611346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 0x557eb4666ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:55.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.506+0000 7f304de00700 1 --2- 172.21.15.78:0/2427611346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650070 0x557eb465bc20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f30400d0fd0 tx=0x7f3040005470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b939e29a2a5e67 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.506+0000 7f302fe00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:57:55.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.506+0000 7f302fe00700 1 -- 172.21.15.78:0/2427611346 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040099ee0 con 0x557eb4650070 2024-03-14T09:57:55.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.506+0000 7f302fe00700 1 -- 172.21.15.78:0/2427611346 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f304009ac80 con 0x557eb4650070 2024-03-14T09:57:55.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.506+0000 7f3068cafb80 1 -- 172.21.15.78:0/2427611346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557eb463b720 con 0x557eb4650070 2024-03-14T09:57:55.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.507+0000 7f302fe00700 1 -- 172.21.15.78:0/2427611346 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f304002b9f0 con 0x557eb4650070 2024-03-14T09:57:55.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.507+0000 7f3068cafb80 10 client.16374 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:55.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.507+0000 7f3068cafb80 1 -- 172.21.15.78:0/2427611346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557eb43c8b60 con 0x557eb4650070 2024-03-14T09:57:55.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.507+0000 7f302fe00700 1 -- 172.21.15.78:0/2427611346 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557eb463b720 con 0x557eb4650070 2024-03-14T09:57:55.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.507+0000 7f302fe00700 1 -- 172.21.15.78:0/2427611346 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f30400cab70 con 0x557eb4650070 2024-03-14T09:57:55.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.507+0000 7f3068cafb80 10 client.16374 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:55.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.507+0000 7f3068cafb80 10 client.16374 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:57:55.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.507+0000 7f3068cafb80 20 client.16374 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:55.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.507+0000 7f3068cafb80 1 --2- 172.21.15.78:0/2427611346 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x557eb466abc0 0x557eb466cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.507+0000 7f3068cafb80 4 client.16374 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:57:55.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.507+0000 7f3068cafb80 1 -- 172.21.15.78:0/2427611346 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557eb4593700 con 0x557eb466abc0 2024-03-14T09:57:55.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.507+0000 7f304d400700 1 --2- 172.21.15.78:0/2427611346 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x557eb466abc0 0x557eb466cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.508+0000 7f304d400700 1 --2- 172.21.15.78:0/2427611346 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x557eb466abc0 0x557eb466cfe0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f43651f363a71786 server_cookie=4f19aa9d7fbb41c3 in_seq=0 out_seq=0 2024-03-14T09:57:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.508+0000 7f302fe00700 10 client.16374 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:57:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.509+0000 7f302fe00700 1 -- 172.21.15.78:0/2427611346 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x557eb4593700 con 0x557eb466abc0 2024-03-14T09:57:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.509+0000 7f302fe00700 10 client.16374 handle_command_reply: tid=0 2024-03-14T09:57:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.509+0000 7f3068cafb80 1 client.16374 shutdown 2024-03-14T09:57:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.509+0000 7f3068cafb80 1 -- 172.21.15.78:0/2427611346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650070 msgr2=0x557eb465bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.509+0000 7f3068cafb80 1 --2- 172.21.15.78:0/2427611346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650070 0x557eb465bc20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f30400d0fd0 tx=0x7f3040005470 comp rx=0 tx=0).stop 2024-03-14T09:57:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.509+0000 7f3068cafb80 1 -- 172.21.15.78:0/2427611346 shutdown_connections 2024-03-14T09:57:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.509+0000 7f3068cafb80 1 --2- 172.21.15.78:0/2427611346 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x557eb466abc0 0x557eb466cfe0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.509+0000 7f3068cafb80 1 --2- 172.21.15.78:0/2427611346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 0x557eb4666ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.509+0000 7f302fe00700 0 client.16374 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:57:55.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.509+0000 7f3068cafb80 1 --2- 172.21.15.78:0/2427611346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650070 0x557eb465bc20 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.509+0000 7f3068cafb80 1 --2- 172.21.15.78:0/2427611346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4650990 0x557eb465c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.509+0000 7f3068cafb80 1 -- 172.21.15.78:0/2427611346 >> 172.21.15.78:0/2427611346 conn(0x557eb45be220 msgr2=0x557eb4643880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.509+0000 7f3068cafb80 1 -- 172.21.15.78:0/2427611346 shutdown_connections 2024-03-14T09:57:55.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.509+0000 7f3068cafb80 1 -- 172.21.15.78:0/2427611346 wait complete. 2024-03-14T09:57:55.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.509+0000 7f3068cafb80 20 client.16374 trim_cache size 0 max 16384 2024-03-14T09:57:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:55 smithi132 ceph-mon[47959]: pgmap v115: 129 pgs: 129 active+clean; 130 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 2.1 MiB/s wr, 361 op/s 2024-03-14T09:57:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:55 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/983277860' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:57:55.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.526+0000 7f3068cafb80 1 Processor -- start 2024-03-14T09:57:55.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.527+0000 7f3068cafb80 1 -- start start 2024-03-14T09:57:55.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.527+0000 7f3068cafb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650990 0x557eb4660b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.527+0000 7f3068cafb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 0x557eb46551c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.527+0000 7f3068cafb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4655d30 0x557eb4656170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.527+0000 7f3068cafb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eb46566b0 con 0x557eb4650990 2024-03-14T09:57:55.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.527+0000 7f3068cafb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eb463dd00 con 0x557eb4655d30 2024-03-14T09:57:55.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.527+0000 7f3068cafb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eb465c4d0 con 0x557eb465b840 2024-03-14T09:57:55.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.527+0000 7f304d400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 0x557eb46551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.527+0000 7f304d400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 0x557eb46551c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59308/0 (socket says 172.21.15.78:59308) 2024-03-14T09:57:55.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.527+0000 7f304de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650990 0x557eb4660b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.527+0000 7f304d400700 1 -- 172.21.15.78:0/2850460526 learned_addr learned my addr 172.21.15.78:0/2850460526 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:55.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.527+0000 7f304e800700 1 --2- 172.21.15.78:0/2850460526 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4655d30 0x557eb4656170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.528+0000 7f304d400700 1 -- 172.21.15.78:0/2850460526 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4655d30 msgr2=0x557eb4656170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.528+0000 7f304d400700 1 --2- 172.21.15.78:0/2850460526 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4655d30 0x557eb4656170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.528+0000 7f304d400700 1 -- 172.21.15.78:0/2850460526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650990 msgr2=0x557eb4660b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.528+0000 7f304d400700 1 --2- 172.21.15.78:0/2850460526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650990 0x557eb4660b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.528+0000 7f304d400700 1 -- 172.21.15.78:0/2850460526 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557eb4560ea0 con 0x557eb465b840 2024-03-14T09:57:55.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.528+0000 7f304e800700 1 --2- 172.21.15.78:0/2850460526 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4655d30 0x557eb4656170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:55.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.528+0000 7f304de00700 1 --2- 172.21.15.78:0/2850460526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650990 0x557eb4660b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:55.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.528+0000 7f304d400700 1 --2- 172.21.15.78:0/2850460526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 0x557eb46551c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x557eb4640350 tx=0x7f303c005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e98d0a65af1ab38a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.528+0000 7f303be00700 1 -- 172.21.15.78:0/2850460526 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303c018040 con 0x557eb465b840 2024-03-14T09:57:55.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.528+0000 7f303be00700 1 -- 172.21.15.78:0/2850460526 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f303c023070 con 0x557eb465b840 2024-03-14T09:57:55.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.528+0000 7f303be00700 1 -- 172.21.15.78:0/2850460526 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303c005440 con 0x557eb465b840 2024-03-14T09:57:55.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.529+0000 7f3068cafb80 1 -- 172.21.15.78:0/2850460526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 msgr2=0x557eb46551c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.529+0000 7f3068cafb80 1 --2- 172.21.15.78:0/2850460526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 0x557eb46551c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x557eb4640350 tx=0x7f303c005f10 comp rx=0 tx=0).stop 2024-03-14T09:57:55.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.529+0000 7f3068cafb80 1 -- 172.21.15.78:0/2850460526 shutdown_connections 2024-03-14T09:57:55.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.529+0000 7f3068cafb80 1 --2- 172.21.15.78:0/2850460526 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4655d30 0x557eb4656170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.529+0000 7f3068cafb80 1 --2- 172.21.15.78:0/2850460526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb4650990 0x557eb4660b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.529+0000 7f3068cafb80 1 --2- 172.21.15.78:0/2850460526 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb465b840 0x557eb46551c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.529+0000 7f3068cafb80 1 -- 172.21.15.78:0/2850460526 >> 172.21.15.78:0/2850460526 conn(0x557eb45be220 msgr2=0x557eb465a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.529+0000 7f3068cafb80 1 -- 172.21.15.78:0/2850460526 shutdown_connections 2024-03-14T09:57:55.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.530+0000 7f3068cafb80 1 -- 172.21.15.78:0/2850460526 wait complete. 2024-03-14T09:57:55.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.530+0000 7f3068cafb80 1 -- start start 2024-03-14T09:57:55.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.530+0000 7f3068cafb80 1 Processor -- start 2024-03-14T09:57:55.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f3068cafb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb4650990 0x557eb46606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f3068cafb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4655d30 0x557eb465eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f3068cafb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb465b840 0x557eb465f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f3068cafb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eb465f5c0 con 0x557eb465b840 2024-03-14T09:57:55.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f3068cafb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eb465f730 con 0x557eb4655d30 2024-03-14T09:57:55.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f3068cafb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eb465f8f0 con 0x557eb4650990 2024-03-14T09:57:55.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f3068cafb80 10 client.0 _ll_register_callbacks cb 0x7f30400cd7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:55.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f304de00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb4650990 0x557eb46606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f304d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4655d30 0x557eb465eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f304de00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb4650990 0x557eb46606a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59318/0 (socket says 172.21.15.78:59318) 2024-03-14T09:57:55.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f304de00700 1 -- 172.21.15.78:0/1901892955 learned_addr learned my addr 172.21.15.78:0/1901892955 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:55.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f304e800700 1 --2- 172.21.15.78:0/1901892955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb465b840 0x557eb465f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f304d400700 1 -- 172.21.15.78:0/1901892955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb4650990 msgr2=0x557eb46606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f304d400700 1 --2- 172.21.15.78:0/1901892955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb4650990 0x557eb46606a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f304d400700 1 -- 172.21.15.78:0/1901892955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb465b840 msgr2=0x557eb465f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f304d400700 1 --2- 172.21.15.78:0/1901892955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb465b840 0x557eb465f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.531+0000 7f304d400700 1 -- 172.21.15.78:0/1901892955 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557eb43c8b60 con 0x557eb4655d30 2024-03-14T09:57:55.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.532+0000 7f304d400700 1 --2- 172.21.15.78:0/1901892955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4655d30 0x557eb465eb40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x557eb45937d0 tx=0x7f303c0130c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76b49da9bce55326 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.532+0000 7f302fe00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:57:55.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.532+0000 7f302fe00700 1 -- 172.21.15.78:0/1901892955 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303c018070 con 0x557eb4655d30 2024-03-14T09:57:55.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.532+0000 7f302fe00700 1 -- 172.21.15.78:0/1901892955 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f303c004250 con 0x557eb4655d30 2024-03-14T09:57:55.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.532+0000 7f3068cafb80 1 -- 172.21.15.78:0/1901892955 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f304009a0c0 con 0x557eb4655d30 2024-03-14T09:57:55.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.532+0000 7f302fe00700 1 -- 172.21.15.78:0/1901892955 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f303c004410 con 0x557eb4655d30 2024-03-14T09:57:55.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.532+0000 7f3068cafb80 10 client.26423 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:55.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.532+0000 7f3068cafb80 1 -- 172.21.15.78:0/1901892955 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f304800e910 con 0x557eb4655d30 2024-03-14T09:57:55.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.533+0000 7f302fe00700 1 -- 172.21.15.78:0/1901892955 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f304009a0c0 con 0x557eb4655d30 2024-03-14T09:57:55.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.533+0000 7f302fe00700 1 -- 172.21.15.78:0/1901892955 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f303c004b00 con 0x557eb4655d30 2024-03-14T09:57:55.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.533+0000 7f3068cafb80 10 client.26423 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:55.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.533+0000 7f3068cafb80 10 client.26423 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:57:55.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.533+0000 7f3068cafb80 20 client.26423 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:55.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.533+0000 7f3068cafb80 1 --2- 172.21.15.78:0/1901892955 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x557eb46bdad0 0x557eb46bfef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.533+0000 7f3068cafb80 4 client.26423 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:57:55.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.533+0000 7f3068cafb80 1 -- 172.21.15.78:0/1901892955 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557eb43e9cf0 con 0x557eb46bdad0 2024-03-14T09:57:55.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.533+0000 7f304de00700 1 --2- 172.21.15.78:0/1901892955 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x557eb46bdad0 0x557eb46bfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.534+0000 7f304de00700 1 --2- 172.21.15.78:0/1901892955 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x557eb46bdad0 0x557eb46bfef0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d939d5c1631d3dbe server_cookie=cf2e0ca773ccc3ec in_seq=0 out_seq=0 2024-03-14T09:57:55.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.534+0000 7f302fe00700 10 client.26423 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:57:55.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.534+0000 7f302fe00700 1 -- 172.21.15.78:0/1901892955 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557eb43e9cf0 con 0x557eb46bdad0 2024-03-14T09:57:55.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.534+0000 7f302fe00700 10 client.26423 handle_command_reply: tid=0 2024-03-14T09:57:55.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.534+0000 7f3068cafb80 1 client.26423 shutdown 2024-03-14T09:57:55.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.534+0000 7f3068cafb80 1 -- 172.21.15.78:0/1901892955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4655d30 msgr2=0x557eb465eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.534+0000 7f3068cafb80 1 --2- 172.21.15.78:0/1901892955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4655d30 0x557eb465eb40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x557eb45937d0 tx=0x7f303c0130c0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.534+0000 7f3068cafb80 1 -- 172.21.15.78:0/1901892955 shutdown_connections 2024-03-14T09:57:55.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.534+0000 7f3068cafb80 1 --2- 172.21.15.78:0/1901892955 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x557eb46bdad0 0x557eb46bfef0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.534+0000 7f3068cafb80 1 --2- 172.21.15.78:0/1901892955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557eb465b840 0x557eb465f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.534+0000 7f3068cafb80 1 --2- 172.21.15.78:0/1901892955 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557eb4650990 0x557eb46606a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.534+0000 7f3068cafb80 1 --2- 172.21.15.78:0/1901892955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557eb4655d30 0x557eb465eb40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.534+0000 7f302fe00700 0 client.26423 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:57:55.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.534+0000 7f3068cafb80 1 -- 172.21.15.78:0/1901892955 >> 172.21.15.78:0/1901892955 conn(0x557eb45be220 msgr2=0x557eb4655210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.534+0000 7f3068cafb80 1 -- 172.21.15.78:0/1901892955 shutdown_connections 2024-03-14T09:57:55.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.534+0000 7f3068cafb80 1 -- 172.21.15.78:0/1901892955 wait complete. 2024-03-14T09:57:55.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.535+0000 7f3068cafb80 20 client.26423 trim_cache size 0 max 16384 2024-03-14T09:57:55.538 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:57:55.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.535+0000 7f304f200700 1 -- 172.21.15.78:0/345786758 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f30340a0c80 msgr2=0x7f30340a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.535+0000 7f304f200700 1 --2- 172.21.15.78:0/345786758 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f30340a0c80 0x7f30340a3140 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3048009fd0 tx=0x7f3048004ad0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.535+0000 7f304f200700 1 -- 172.21.15.78:0/345786758 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3050101cc0 msgr2=0x7f30500ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.535+0000 7f304f200700 1 --2- 172.21.15.78:0/345786758 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3050101cc0 0x7f30500ff910 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3040005950 tx=0x7f304000fa00 comp rx=0 tx=0).stop 2024-03-14T09:57:55.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.535+0000 7f304f200700 1 -- 172.21.15.78:0/345786758 shutdown_connections 2024-03-14T09:57:55.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.535+0000 7f304f200700 1 --2- 172.21.15.78:0/345786758 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f30340a0c80 0x7f30340a3140 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.535+0000 7f304f200700 1 --2- 172.21.15.78:0/345786758 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3050101cc0 0x7f30500ff910 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.535+0000 7f304f200700 1 --2- 172.21.15.78:0/345786758 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f305010b0e0 0x7f3050100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.535+0000 7f304f200700 1 --2- 172.21.15.78:0/345786758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30501025e0 0x7f30500ffe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.535+0000 7f304f200700 1 -- 172.21.15.78:0/345786758 >> 172.21.15.78:0/345786758 conn(0x7f30500fb4e0 msgr2=0x7f30500fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.536+0000 7f304f200700 1 -- 172.21.15.78:0/345786758 shutdown_connections 2024-03-14T09:57:55.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.536+0000 7f304f200700 1 -- 172.21.15.78:0/345786758 wait complete. 2024-03-14T09:57:55.558 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T09:57:55.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.801+0000 7f1bfe200700 1 -- 172.21.15.78:0/969134634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1bf8102f90 msgr2=0x7f1bf8103370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.801+0000 7f1bfe200700 1 --2- 172.21.15.78:0/969134634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1bf8102f90 0x7f1bf8103370 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1bec009990 tx=0x7f1bec009ca0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.801+0000 7f1bfe200700 1 -- 172.21.15.78:0/969134634 shutdown_connections 2024-03-14T09:57:55.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.801+0000 7f1bfe200700 1 --2- 172.21.15.78:0/969134634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1bf8102f90 0x7f1bf8103370 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.801+0000 7f1bfe200700 1 --2- 172.21.15.78:0/969134634 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1bf81080b0 0x7f1bf810fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.801+0000 7f1bfe200700 1 --2- 172.21.15.78:0/969134634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bf8103940 0x7f1bf8107980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.801+0000 7f1bfe200700 1 -- 172.21.15.78:0/969134634 >> 172.21.15.78:0/969134634 conn(0x7f1bf80fe7b0 msgr2=0x7f1bf8100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.801+0000 7f1bfe200700 1 -- 172.21.15.78:0/969134634 shutdown_connections 2024-03-14T09:57:55.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.801+0000 7f1bfe200700 1 -- 172.21.15.78:0/969134634 wait complete. 2024-03-14T09:57:55.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.802+0000 7f1bfe200700 1 Processor -- start 2024-03-14T09:57:55.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.803+0000 7f1bfe200700 1 -- start start 2024-03-14T09:57:55.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.803+0000 7f1bfe200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1bf8102f90 0x7f1bf8199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.803+0000 7f1bfe200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bf81080b0 0x7f1bf8199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.803+0000 7f1bfe200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1bf819df10 0x7f1bf819e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.803+0000 7f1bfe200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf819e990 con 0x7f1bf81080b0 2024-03-14T09:57:55.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.803+0000 7f1bfe200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf819eb00 con 0x7f1bf8102f90 2024-03-14T09:57:55.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.803+0000 7f1bfe200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf819ecc0 con 0x7f1bf819df10 2024-03-14T09:57:55.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.803+0000 7f1bfd800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1bf819df10 0x7f1bf819e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.803+0000 7f1bf7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bf81080b0 0x7f1bf8199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.803+0000 7f1bfd800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1bf819df10 0x7f1bf819e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59324/0 (socket says 172.21.15.78:59324) 2024-03-14T09:57:55.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.803+0000 7f1bfd800700 1 -- 172.21.15.78:0/3373909954 learned_addr learned my addr 172.21.15.78:0/3373909954 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:55.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.803+0000 7f1bfce00700 1 --2- 172.21.15.78:0/3373909954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1bf8102f90 0x7f1bf8199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.804+0000 7f1bfd800700 1 -- 172.21.15.78:0/3373909954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1bf8102f90 msgr2=0x7f1bf8199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.804+0000 7f1bfd800700 1 --2- 172.21.15.78:0/3373909954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1bf8102f90 0x7f1bf8199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.804+0000 7f1bfd800700 1 -- 172.21.15.78:0/3373909954 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bf81080b0 msgr2=0x7f1bf8199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.804+0000 7f1bfd800700 1 --2- 172.21.15.78:0/3373909954 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bf81080b0 0x7f1bf8199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.804+0000 7f1bfd800700 1 -- 172.21.15.78:0/3373909954 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bec0095f0 con 0x7f1bf819df10 2024-03-14T09:57:55.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.804+0000 7f1bfd800700 1 --2- 172.21.15.78:0/3373909954 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1bf819df10 0x7f1bf819e390 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1bf000ef30 tx=0x7f1bf000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.804+0000 7f1bf5600700 1 -- 172.21.15.78:0/3373909954 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0018070 con 0x7f1bf819df10 2024-03-14T09:57:55.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.804+0000 7f1bfe200700 1 -- 172.21.15.78:0/3373909954 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf81a61c0 con 0x7f1bf819df10 2024-03-14T09:57:55.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.804+0000 7f1bf5600700 1 -- 172.21.15.78:0/3373909954 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1bf000f040 con 0x7f1bf819df10 2024-03-14T09:57:55.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.804+0000 7f1bf5600700 1 -- 172.21.15.78:0/3373909954 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf000cd00 con 0x7f1bf819df10 2024-03-14T09:57:55.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.804+0000 7f1bfe200700 1 -- 172.21.15.78:0/3373909954 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf81a66f0 con 0x7f1bf819df10 2024-03-14T09:57:55.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.808+0000 7f1bf5600700 1 -- 172.21.15.78:0/3373909954 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f1bf0003a00 con 0x7f1bf819df10 2024-03-14T09:57:55.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.809+0000 7f1c132eab80 1 Processor -- start 2024-03-14T09:57:55.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.810+0000 7f1c132eab80 1 -- start start 2024-03-14T09:57:55.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.810+0000 7f1c132eab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556556143860 0x556556143c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.810+0000 7f1c132eab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556144180 0x55655614eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.810+0000 7f1c132eab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55655614f030 0x556556151420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.810+0000 7f1c132eab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655609d990 con 0x556556144180 2024-03-14T09:57:55.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.810+0000 7f1c132eab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655609b350 con 0x55655614f030 2024-03-14T09:57:55.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.810+0000 7f1c132eab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556556096c40 con 0x556556143860 2024-03-14T09:57:55.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.810+0000 7f1bf7e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556556143860 0x556556143c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.810+0000 7f1bfce00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556144180 0x55655614eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.810+0000 7f1bf7e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556556143860 0x556556143c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59338/0 (socket says 172.21.15.78:59338) 2024-03-14T09:57:55.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.810+0000 7f1bf7e00700 1 -- 172.21.15.78:0/2828480043 learned_addr learned my addr 172.21.15.78:0/2828480043 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:55.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.810+0000 7f1bf5600700 1 --2- 172.21.15.78:0/3373909954 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1be00a0c00 0x7f1be00a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.810+0000 7f1bf7e00700 1 --2- 172.21.15.78:0/2828480043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55655614f030 0x556556151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.811+0000 7f1bf5600700 1 -- 172.21.15.78:0/3373909954 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f1bf00cc2a0 con 0x7f1bf819df10 2024-03-14T09:57:55.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.811+0000 7f1bfd800700 1 --2- 172.21.15.78:0/3373909954 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1be00a0c00 0x7f1be00a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.811+0000 7f1bf7e00700 1 -- 172.21.15.78:0/2828480043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55655614f030 msgr2=0x556556151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.811+0000 7f1bf7e00700 1 --2- 172.21.15.78:0/2828480043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55655614f030 0x556556151420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.811+0000 7f1bf7e00700 1 -- 172.21.15.78:0/2828480043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556144180 msgr2=0x55655614eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.811+0000 7f1bf7e00700 1 --2- 172.21.15.78:0/2828480043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556144180 0x55655614eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.811+0000 7f1bf7e00700 1 -- 172.21.15.78:0/2828480043 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556555f183b0 con 0x556556143860 2024-03-14T09:57:55.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.811+0000 7f1bf7e00700 1 --2- 172.21.15.78:0/2828480043 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556556143860 0x556556143c40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1be4005d90 tx=0x7f1be4005d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cf1868355a78209 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.811+0000 7f1bea000700 1 -- 172.21.15.78:0/2828480043 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be4012e00 con 0x556556143860 2024-03-14T09:57:55.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.811+0000 7f1bea000700 1 -- 172.21.15.78:0/2828480043 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1be4004cf0 con 0x556556143860 2024-03-14T09:57:55.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.811+0000 7f1bea000700 1 -- 172.21.15.78:0/2828480043 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be400f610 con 0x556556143860 2024-03-14T09:57:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.812+0000 7f1c132eab80 1 -- 172.21.15.78:0/2828480043 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556556143860 msgr2=0x556556143c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.812+0000 7f1bfd800700 1 --2- 172.21.15.78:0/3373909954 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1be00a0c00 0x7f1be00a30c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1bf00c96a0 tx=0x7f1bf0009ed0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.812+0000 7f1c132eab80 1 --2- 172.21.15.78:0/2828480043 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556556143860 0x556556143c40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1be4005d90 tx=0x7f1be4005d20 comp rx=0 tx=0).stop 2024-03-14T09:57:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.812+0000 7f1c132eab80 1 -- 172.21.15.78:0/2828480043 shutdown_connections 2024-03-14T09:57:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.812+0000 7f1c132eab80 1 --2- 172.21.15.78:0/2828480043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55655614f030 0x556556151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.812+0000 7f1c132eab80 1 --2- 172.21.15.78:0/2828480043 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556556143860 0x556556143c40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.812+0000 7f1c132eab80 1 --2- 172.21.15.78:0/2828480043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556144180 0x55655614eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.812+0000 7f1c132eab80 1 -- 172.21.15.78:0/2828480043 >> 172.21.15.78:0/2828480043 conn(0x5565560ba220 msgr2=0x55655613fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.812+0000 7f1c132eab80 1 -- 172.21.15.78:0/2828480043 shutdown_connections 2024-03-14T09:57:55.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.812+0000 7f1c132eab80 1 -- 172.21.15.78:0/2828480043 wait complete. 2024-03-14T09:57:55.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.813+0000 7f1c132eab80 1 -- start start 2024-03-14T09:57:55.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.813+0000 7f1c132eab80 1 Processor -- start 2024-03-14T09:57:55.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.814+0000 7f1c132eab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556143860 0x55655614ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.814+0000 7f1c132eab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556556144180 0x55655614bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.814+0000 7f1c132eab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55655614f030 0x55655614c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.814+0000 7f1c132eab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556556032d80 con 0x556556143860 2024-03-14T09:57:55.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.814+0000 7f1c132eab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x556556032a20 con 0x556556144180 2024-03-14T09:57:55.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.814+0000 7f1c132eab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556555f1a6a0 con 0x55655614f030 2024-03-14T09:57:55.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.814+0000 7f1c132eab80 10 client.0 _ll_register_callbacks cb 0x556555e9e3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:55.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.814+0000 7f1bfce00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556556144180 0x55655614bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.814+0000 7f1bfce00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556556144180 0x55655614bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41522/0 (socket says 172.21.15.78:41522) 2024-03-14T09:57:55.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.814+0000 7f1bfce00700 1 -- 172.21.15.78:0/2980621807 learned_addr learned my addr 172.21.15.78:0/2980621807 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:55.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.814+0000 7f1bf7e00700 1 --2- 172.21.15.78:0/2980621807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556143860 0x55655614ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.814+0000 7f1bf7e00700 1 --2- 172.21.15.78:0/2980621807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55655614f030 0x55655614c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.815+0000 7f1bfce00700 1 -- 172.21.15.78:0/2980621807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55655614f030 msgr2=0x55655614c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.815+0000 7f1bfce00700 1 --2- 172.21.15.78:0/2980621807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55655614f030 0x55655614c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.815+0000 7f1bfce00700 1 -- 172.21.15.78:0/2980621807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556143860 msgr2=0x55655614ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.815+0000 7f1bfce00700 1 --2- 172.21.15.78:0/2980621807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556143860 0x55655614ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.815+0000 7f1bfce00700 1 -- 172.21.15.78:0/2980621807 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be401a040 con 0x556556144180 2024-03-14T09:57:55.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.815+0000 7f1bf7e00700 1 --2- 172.21.15.78:0/2980621807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55655614f030 0x55655614c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:57:55.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.815+0000 7f1bfce00700 1 --2- 172.21.15.78:0/2980621807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556556144180 0x55655614bdd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1bec005750 tx=0x7f1bec005bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a64bdf80e5501469 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.815+0000 7f1bdea00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:57:55.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.816+0000 7f1bdea00700 1 -- 172.21.15.78:0/2980621807 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec01d070 con 0x556556144180 2024-03-14T09:57:55.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.816+0000 7f1bdea00700 1 -- 172.21.15.78:0/2980621807 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f1bec004500 con 0x556556144180 2024-03-14T09:57:55.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.816+0000 7f1c132eab80 1 -- 172.21.15.78:0/2980621807 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556556137720 con 0x556556144180 2024-03-14T09:57:55.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.816+0000 7f1bdea00700 1 -- 172.21.15.78:0/2980621807 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bec0046c0 con 0x556556144180 2024-03-14T09:57:55.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.816+0000 7f1c132eab80 10 client.26435 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:55.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.816+0000 7f1c132eab80 1 -- 172.21.15.78:0/2980621807 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556555ec4b60 con 0x556556144180 2024-03-14T09:57:55.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.817+0000 7f1bdea00700 1 -- 172.21.15.78:0/2980621807 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556556137720 con 0x556556144180 2024-03-14T09:57:55.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.817+0000 7f1bdea00700 1 -- 172.21.15.78:0/2980621807 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1bec00f9c0 con 0x556556144180 2024-03-14T09:57:55.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.817+0000 7f1c132eab80 10 client.26435 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:55.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.817+0000 7f1c132eab80 10 client.26435 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:57:55.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.817+0000 7f1c132eab80 20 client.26435 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:55.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.817+0000 7f1c132eab80 1 --2- 172.21.15.78:0/2980621807 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55655615ec00 0x556556161020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.817+0000 7f1c132eab80 4 client.26435 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:57:55.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.817+0000 7f1c132eab80 1 -- 172.21.15.78:0/2980621807 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55655608f700 con 0x55655615ec00 2024-03-14T09:57:55.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.817+0000 7f1bf7e00700 1 --2- 172.21.15.78:0/2980621807 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55655615ec00 0x556556161020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.818+0000 7f1bdea00700 10 client.26435 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:55.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.818+0000 7f1bf7e00700 1 --2- 172.21.15.78:0/2980621807 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55655615ec00 0x556556161020 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d85b70805503b510 server_cookie=541fed9d27d36345 in_seq=0 out_seq=0 2024-03-14T09:57:55.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.820+0000 7f1bdea00700 1 -- 172.21.15.78:0/2980621807 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55655608f700 con 0x55655615ec00 2024-03-14T09:57:55.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.820+0000 7f1bdea00700 10 client.26435 handle_command_reply: tid=0 2024-03-14T09:57:55.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.820+0000 7f1c132eab80 1 client.26435 shutdown 2024-03-14T09:57:55.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.820+0000 7f1c132eab80 1 -- 172.21.15.78:0/2980621807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556556144180 msgr2=0x55655614bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.820+0000 7f1c132eab80 1 --2- 172.21.15.78:0/2980621807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556556144180 0x55655614bdd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1bec005750 tx=0x7f1bec005bc0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.821+0000 7f1c132eab80 1 -- 172.21.15.78:0/2980621807 shutdown_connections 2024-03-14T09:57:55.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.821+0000 7f1c132eab80 1 --2- 172.21.15.78:0/2980621807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55655614f030 0x55655614c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.821+0000 7f1c132eab80 1 --2- 172.21.15.78:0/2980621807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556143860 0x55655614ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.821+0000 7f1c132eab80 1 --2- 172.21.15.78:0/2980621807 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55655615ec00 0x556556161020 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.821+0000 7f1c132eab80 1 --2- 172.21.15.78:0/2980621807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556556144180 0x55655614bdd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.821+0000 7f1c132eab80 1 -- 172.21.15.78:0/2980621807 >> 172.21.15.78:0/2980621807 conn(0x5565560ba220 msgr2=0x55655613f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.821+0000 7f1bdea00700 0 client.26435 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:55.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.821+0000 7f1c132eab80 1 -- 172.21.15.78:0/2980621807 shutdown_connections 2024-03-14T09:57:55.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.822+0000 7f1c132eab80 1 -- 172.21.15.78:0/2980621807 wait complete. 2024-03-14T09:57:55.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.822+0000 7f1c132eab80 20 client.26435 trim_cache size 0 max 16384 2024-03-14T09:57:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.836+0000 7f1c132eab80 1 Processor -- start 2024-03-14T09:57:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.836+0000 7f1c132eab80 1 -- start start 2024-03-14T09:57:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.836+0000 7f1c132eab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556556143860 0x556556155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.836+0000 7f1c132eab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556144180 0x55655614e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.836+0000 7f1c132eab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55655614f030 0x55655614a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.836+0000 7f1c132eab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655614cce0 con 0x556556144180 2024-03-14T09:57:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.836+0000 7f1c132eab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x556556144820 con 0x55655614f030 2024-03-14T09:57:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.836+0000 7f1c132eab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556556141020 con 0x556556143860 2024-03-14T09:57:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.837+0000 7f1bf7e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556556143860 0x556556155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.837+0000 7f1bfce00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556144180 0x55655614e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.837+0000 7f1bf7e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55655614f030 0x55655614a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.837+0000 7f1bf7e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55655614f030 0x55655614a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41536/0 (socket says 172.21.15.78:41536) 2024-03-14T09:57:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.837+0000 7f1bf7e00700 1 -- 172.21.15.78:0/3190510695 learned_addr learned my addr 172.21.15.78:0/3190510695 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.837+0000 7f1bf7e00700 1 -- 172.21.15.78:0/3190510695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556556143860 msgr2=0x556556155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.837+0000 7f1bf7e00700 1 --2- 172.21.15.78:0/3190510695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556556143860 0x556556155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.837+0000 7f1bf7e00700 1 -- 172.21.15.78:0/3190510695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556144180 msgr2=0x55655614e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.837+0000 7f1bf7e00700 1 --2- 172.21.15.78:0/3190510695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556144180 0x55655614e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.837+0000 7f1bf7e00700 1 -- 172.21.15.78:0/3190510695 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55655605cea0 con 0x55655614f030 2024-03-14T09:57:55.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.838+0000 7f1bf7e00700 1 --2- 172.21.15.78:0/3190510695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55655614f030 0x55655614a6a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x556556145810 tx=0x7f1be4024040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd0c45f60d5c7ded server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.838+0000 7f1beaa00700 1 -- 172.21.15.78:0/3190510695 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be4027040 con 0x55655614f030 2024-03-14T09:57:55.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.838+0000 7f1beaa00700 1 -- 172.21.15.78:0/3190510695 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1be400bc80 con 0x55655614f030 2024-03-14T09:57:55.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.838+0000 7f1beaa00700 1 -- 172.21.15.78:0/3190510695 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be4004e60 con 0x55655614f030 2024-03-14T09:57:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.839+0000 7f1c132eab80 1 -- 172.21.15.78:0/3190510695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55655614f030 msgr2=0x55655614a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.839+0000 7f1c132eab80 1 --2- 172.21.15.78:0/3190510695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55655614f030 0x55655614a6a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x556556145810 tx=0x7f1be4024040 comp rx=0 tx=0).stop 2024-03-14T09:57:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.839+0000 7f1c132eab80 1 -- 172.21.15.78:0/3190510695 shutdown_connections 2024-03-14T09:57:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.839+0000 7f1c132eab80 1 --2- 172.21.15.78:0/3190510695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55655614f030 0x55655614a6a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.839+0000 7f1c132eab80 1 --2- 172.21.15.78:0/3190510695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556556143860 0x556556155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.839+0000 7f1c132eab80 1 --2- 172.21.15.78:0/3190510695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556144180 0x55655614e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.839+0000 7f1c132eab80 1 -- 172.21.15.78:0/3190510695 >> 172.21.15.78:0/3190510695 conn(0x5565560ba220 msgr2=0x55655613e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.839+0000 7f1c132eab80 1 -- 172.21.15.78:0/3190510695 shutdown_connections 2024-03-14T09:57:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.839+0000 7f1c132eab80 1 -- 172.21.15.78:0/3190510695 wait complete. 2024-03-14T09:57:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.839+0000 7f1c132eab80 1 -- start start 2024-03-14T09:57:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.840+0000 7f1c132eab80 1 Processor -- start 2024-03-14T09:57:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.840+0000 7f1c132eab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556143860 0x5565561558d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.840+0000 7f1c132eab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556556144180 0x556556153c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.840+0000 7f1c132eab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5565561541b0 0x556556154630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.840+0000 7f1c132eab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556556156050 con 0x556556143860 2024-03-14T09:57:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.840+0000 7f1c132eab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565561b0710 con 0x556556144180 2024-03-14T09:57:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.840+0000 7f1c132eab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565561b08d0 con 0x5565561541b0 2024-03-14T09:57:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.840+0000 7f1c132eab80 10 client.0 _ll_register_callbacks cb 0x556555e9e3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.841+0000 7f1bf7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556143860 0x5565561558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.841+0000 7f1bf7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556143860 0x5565561558d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58940/0 (socket says 172.21.15.78:58940) 2024-03-14T09:57:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.841+0000 7f1bf7e00700 1 -- 172.21.15.78:0/704577696 learned_addr learned my addr 172.21.15.78:0/704577696 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.841+0000 7f1bfce00700 1 --2- 172.21.15.78:0/704577696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556556144180 0x556556153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.841+0000 7f1bf7e00700 1 -- 172.21.15.78:0/704577696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5565561541b0 msgr2=0x556556154630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.841+0000 7f1bf7e00700 1 --2- 172.21.15.78:0/704577696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5565561541b0 0x556556154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.841+0000 7f1bf7e00700 1 -- 172.21.15.78:0/704577696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556556144180 msgr2=0x556556153c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.841+0000 7f1bf7e00700 1 --2- 172.21.15.78:0/704577696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556556144180 0x556556153c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.841+0000 7f1bf7e00700 1 -- 172.21.15.78:0/704577696 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556555f183b0 con 0x556556143860 2024-03-14T09:57:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.842+0000 7f1bf7e00700 1 --2- 172.21.15.78:0/704577696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556143860 0x5565561558d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1be40127e0 tx=0x7f1be402a4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f6146e8d992ef8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.842+0000 7f1bdea00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:57:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.842+0000 7f1bdea00700 1 -- 172.21.15.78:0/704577696 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be4005680 con 0x556556143860 2024-03-14T09:57:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.842+0000 7f1bdea00700 1 -- 172.21.15.78:0/704577696 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f1be4004e60 con 0x556556143860 2024-03-14T09:57:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.842+0000 7f1c132eab80 1 -- 172.21.15.78:0/704577696 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1be4004cf0 con 0x556556143860 2024-03-14T09:57:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.842+0000 7f1bdea00700 1 -- 172.21.15.78:0/704577696 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1be4005020 con 0x556556143860 2024-03-14T09:57:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.843+0000 7f1c132eab80 10 client.16398 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.843+0000 7f1c132eab80 1 -- 172.21.15.78:0/704577696 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556555f1ccd0 con 0x556556143860 2024-03-14T09:57:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.843+0000 7f1bdea00700 1 -- 172.21.15.78:0/704577696 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1be4004cf0 con 0x556556143860 2024-03-14T09:57:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.843+0000 7f1bdea00700 1 -- 172.21.15.78:0/704577696 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1be4003810 con 0x556556143860 2024-03-14T09:57:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.843+0000 7f1c132eab80 10 client.16398 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.843+0000 7f1c132eab80 10 client.16398 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:57:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.843+0000 7f1c132eab80 20 client.16398 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.843+0000 7f1c132eab80 1 --2- 172.21.15.78:0/704577696 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5565561b1b80 0x5565561b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.843+0000 7f1c132eab80 4 client.16398 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:57:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.843+0000 7f1c132eab80 1 -- 172.21.15.78:0/704577696 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556555ee5cf0 con 0x5565561b1b80 2024-03-14T09:57:55.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.844+0000 7f1bfce00700 1 --2- 172.21.15.78:0/704577696 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5565561b1b80 0x5565561b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:55.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.844+0000 7f1bfce00700 1 --2- 172.21.15.78:0/704577696 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5565561b1b80 0x5565561b3f70 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b6cc9841892722e server_cookie=3ef0e93c78fe24d9 in_seq=0 out_seq=0 2024-03-14T09:57:55.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.844+0000 7f1bdea00700 10 client.16398 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:55.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.844+0000 7f1bdea00700 1 -- 172.21.15.78:0/704577696 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556555ee5cf0 con 0x5565561b1b80 2024-03-14T09:57:55.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.844+0000 7f1bdea00700 10 client.16398 handle_command_reply: tid=0 2024-03-14T09:57:55.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.844+0000 7f1c132eab80 1 client.16398 shutdown 2024-03-14T09:57:55.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.845+0000 7f1c132eab80 1 -- 172.21.15.78:0/704577696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556143860 msgr2=0x5565561558d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.845+0000 7f1c132eab80 1 --2- 172.21.15.78:0/704577696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556143860 0x5565561558d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1be40127e0 tx=0x7f1be402a4c0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.845+0000 7f1c132eab80 1 -- 172.21.15.78:0/704577696 shutdown_connections 2024-03-14T09:57:55.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.845+0000 7f1c132eab80 1 --2- 172.21.15.78:0/704577696 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5565561541b0 0x556556154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.845+0000 7f1c132eab80 1 --2- 172.21.15.78:0/704577696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556556143860 0x5565561558d0 secure :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1be40127e0 tx=0x7f1be402a4c0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.845+0000 7f1c132eab80 1 --2- 172.21.15.78:0/704577696 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5565561b1b80 0x5565561b3f70 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.845+0000 7f1c132eab80 1 --2- 172.21.15.78:0/704577696 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556556144180 0x556556153c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.845+0000 7f1bdea00700 0 client.16398 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:55.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.845+0000 7f1c132eab80 1 -- 172.21.15.78:0/704577696 >> 172.21.15.78:0/704577696 conn(0x5565560ba220 msgr2=0x55655614a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.845+0000 7f1c132eab80 1 -- 172.21.15.78:0/704577696 shutdown_connections 2024-03-14T09:57:55.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.845+0000 7f1c132eab80 1 -- 172.21.15.78:0/704577696 wait complete. 2024-03-14T09:57:55.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.845+0000 7f1c132eab80 20 client.16398 trim_cache size 0 max 16384 2024-03-14T09:57:55.849 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:57:55.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.846+0000 7f1bfe200700 1 -- 172.21.15.78:0/3373909954 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1be00a0c00 msgr2=0x7f1be00a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.846+0000 7f1bfe200700 1 --2- 172.21.15.78:0/3373909954 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1be00a0c00 0x7f1be00a30c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1bf00c96a0 tx=0x7f1bf0009ed0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.846+0000 7f1bfe200700 1 -- 172.21.15.78:0/3373909954 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1bf819df10 msgr2=0x7f1bf819e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:55.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.846+0000 7f1bfe200700 1 --2- 172.21.15.78:0/3373909954 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1bf819df10 0x7f1bf819e390 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1bf000ef30 tx=0x7f1bf000c3a0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.846+0000 7f1bfe200700 1 -- 172.21.15.78:0/3373909954 shutdown_connections 2024-03-14T09:57:55.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.846+0000 7f1bfe200700 1 --2- 172.21.15.78:0/3373909954 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1be00a0c00 0x7f1be00a30c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.846+0000 7f1bfe200700 1 --2- 172.21.15.78:0/3373909954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1bf8102f90 0x7f1bf8199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.846+0000 7f1bfe200700 1 --2- 172.21.15.78:0/3373909954 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1bf819df10 0x7f1bf819e390 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.846+0000 7f1bfe200700 1 --2- 172.21.15.78:0/3373909954 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bf81080b0 0x7f1bf8199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:55.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.846+0000 7f1bfe200700 1 -- 172.21.15.78:0/3373909954 >> 172.21.15.78:0/3373909954 conn(0x7f1bf80fe7b0 msgr2=0x7f1bf80ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:55.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.847+0000 7f1bfe200700 1 -- 172.21.15.78:0/3373909954 shutdown_connections 2024-03-14T09:57:55.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:55.847+0000 7f1bfe200700 1 -- 172.21.15.78:0/3373909954 wait complete. 2024-03-14T09:57:56.869 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T09:57:56.869 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T09:57:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.105+0000 7fb100200700 1 -- 172.21.15.78:0/1515780096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0f8100f20 msgr2=0x7fb0f8101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.105+0000 7fb100200700 1 --2- 172.21.15.78:0/1515780096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0f8100f20 0x7fb0f8101300 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec009990 tx=0x7fb0ec009ca0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.105+0000 7fb100200700 1 -- 172.21.15.78:0/1515780096 shutdown_connections 2024-03-14T09:57:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.105+0000 7fb100200700 1 --2- 172.21.15.78:0/1515780096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0f8106040 0x7fb0f810dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.105+0000 7fb100200700 1 --2- 172.21.15.78:0/1515780096 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0f8100f20 0x7fb0f8101300 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.105+0000 7fb100200700 1 --2- 172.21.15.78:0/1515780096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0f81018d0 0x7fb0f8105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.105+0000 7fb100200700 1 -- 172.21.15.78:0/1515780096 >> 172.21.15.78:0/1515780096 conn(0x7fb0f80fc7a0 msgr2=0x7fb0f80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.105+0000 7fb100200700 1 -- 172.21.15.78:0/1515780096 shutdown_connections 2024-03-14T09:57:57.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.105+0000 7fb100200700 1 -- 172.21.15.78:0/1515780096 wait complete. 2024-03-14T09:57:57.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.106+0000 7fb100200700 1 Processor -- start 2024-03-14T09:57:57.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.106+0000 7fb100200700 1 -- start start 2024-03-14T09:57:57.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.107+0000 7fb100200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0f8100f20 0x7fb0f819f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.107+0000 7fb100200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0f8106040 0x7fb0f819fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.107+0000 7fb100200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0f8199770 0x7fb0f8199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.107+0000 7fb100200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f819a1f0 con 0x7fb0f8100f20 2024-03-14T09:57:57.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.107+0000 7fb100200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f819a360 con 0x7fb0f8199770 2024-03-14T09:57:57.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.107+0000 7fb100200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f819a520 con 0x7fb0f8106040 2024-03-14T09:57:57.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.107+0000 7fb0fce00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0f8106040 0x7fb0f819fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.107+0000 7fb0fd800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0f8100f20 0x7fb0f819f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.107+0000 7fb0fce00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0f8106040 0x7fb0f819fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59390/0 (socket says 172.21.15.78:59390) 2024-03-14T09:57:57.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.107+0000 7fb0fce00700 1 -- 172.21.15.78:0/3228183013 learned_addr learned my addr 172.21.15.78:0/3228183013 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:57.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.107+0000 7fb0fe200700 1 --2- 172.21.15.78:0/3228183013 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0f8199770 0x7fb0f8199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.107+0000 7fb0fce00700 1 -- 172.21.15.78:0/3228183013 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0f8199770 msgr2=0x7fb0f8199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.107+0000 7fb0fce00700 1 --2- 172.21.15.78:0/3228183013 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0f8199770 0x7fb0f8199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.107+0000 7fb0fce00700 1 -- 172.21.15.78:0/3228183013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0f8100f20 msgr2=0x7fb0f819f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.107+0000 7fb0fce00700 1 --2- 172.21.15.78:0/3228183013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0f8100f20 0x7fb0f819f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.107+0000 7fb0fce00700 1 -- 172.21.15.78:0/3228183013 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec0095f0 con 0x7fb0f8106040 2024-03-14T09:57:57.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.108+0000 7fb0fce00700 1 --2- 172.21.15.78:0/3228183013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0f8106040 0x7fb0f819fa90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb0e400e970 tx=0x7fb0e400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:57.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.108+0000 7fb0f6000700 1 -- 172.21.15.78:0/3228183013 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4009790 con 0x7fb0f8106040 2024-03-14T09:57:57.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.108+0000 7fb0f6000700 1 -- 172.21.15.78:0/3228183013 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0e400c7e0 con 0x7fb0f8106040 2024-03-14T09:57:57.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.108+0000 7fb100200700 1 -- 172.21.15.78:0/3228183013 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f81a6420 con 0x7fb0f8106040 2024-03-14T09:57:57.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.108+0000 7fb0f6000700 1 -- 172.21.15.78:0/3228183013 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e40054e0 con 0x7fb0f8106040 2024-03-14T09:57:57.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.108+0000 7fb100200700 1 -- 172.21.15.78:0/3228183013 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f81a6900 con 0x7fb0f8106040 2024-03-14T09:57:57.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.112+0000 7fb0f6000700 1 -- 172.21.15.78:0/3228183013 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fb0e400c950 con 0x7fb0f8106040 2024-03-14T09:57:57.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.114+0000 7fb0f6000700 1 --2- 172.21.15.78:0/3228183013 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0e00a0c00 0x7fb0e00a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.114+0000 7fb113c5bb80 1 Processor -- start 2024-03-14T09:57:57.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.114+0000 7fb0f6000700 1 -- 172.21.15.78:0/3228183013 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fb0e4014070 con 0x7fb0f8106040 2024-03-14T09:57:57.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.114+0000 7fb113c5bb80 1 -- start start 2024-03-14T09:57:57.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.115+0000 7fb113c5bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a07197d700 0x55a07197dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.115+0000 7fb113c5bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 0x55a071987a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.115+0000 7fb113c5bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a071987f40 0x55a07198a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.115+0000 7fb113c5bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0718d6a10 con 0x55a07197d700 2024-03-14T09:57:57.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.115+0000 7fb113c5bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0718d43d0 con 0x55a071987f40 2024-03-14T09:57:57.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.115+0000 7fb113c5bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0718cfcc0 con 0x55a07197e020 2024-03-14T09:57:57.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.115+0000 7fb0fce00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 0x55a071987a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.115+0000 7fb0fce00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 0x55a071987a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59394/0 (socket says 172.21.15.78:59394) 2024-03-14T09:57:57.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.115+0000 7fb0fd800700 1 --2- 172.21.15.78:0/3228183013 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0e00a0c00 0x7fb0e00a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.115+0000 7fb0fce00700 1 -- 172.21.15.78:0/4278920127 learned_addr learned my addr 172.21.15.78:0/4278920127 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:57.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.115+0000 7fb0fe200700 1 --2- 172.21.15.78:0/4278920127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a071987f40 0x55a07198a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.115+0000 7fb0fd800700 1 --2- 172.21.15.78:0/4278920127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a07197d700 0x55a07197dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.115+0000 7fb0fce00700 1 -- 172.21.15.78:0/4278920127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a071987f40 msgr2=0x55a07198a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:57.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.115+0000 7fb0fce00700 1 --2- 172.21.15.78:0/4278920127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a071987f40 0x55a07198a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.115+0000 7fb0fce00700 1 -- 172.21.15.78:0/4278920127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a07197d700 msgr2=0x55a07197dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:57.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.116+0000 7fb0fce00700 1 --2- 172.21.15.78:0/4278920127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a07197d700 0x55a07197dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.116+0000 7fb0fce00700 1 -- 172.21.15.78:0/4278920127 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a071751430 con 0x55a07197e020 2024-03-14T09:57:57.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.116+0000 7fb0fce00700 1 --2- 172.21.15.78:0/4278920127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 0x55a071987a00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb0e40056d0 tx=0x7fb0e4005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5807d5c574a93c7c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:57.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.116+0000 7fb0fe200700 1 --2- 172.21.15.78:0/4278920127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a071987f40 0x55a07198a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:57:57.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.116+0000 7fb0fd800700 1 --2- 172.21.15.78:0/4278920127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a07197d700 0x55a07197dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:57:57.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.116+0000 7fb0eaa00700 1 -- 172.21.15.78:0/4278920127 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e40999d0 con 0x55a07197e020 2024-03-14T09:57:57.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.116+0000 7fb0eaa00700 1 -- 172.21.15.78:0/4278920127 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0e409d800 con 0x55a07197e020 2024-03-14T09:57:57.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.116+0000 7fb0eaa00700 1 -- 172.21.15.78:0/4278920127 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4099c30 con 0x55a07197e020 2024-03-14T09:57:57.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.116+0000 7fb0fd800700 1 --2- 172.21.15.78:0/3228183013 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0e00a0c00 0x7fb0e00a30c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec006010 tx=0x7fb0ec01a040 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.117+0000 7fb113c5bb80 1 -- 172.21.15.78:0/4278920127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 msgr2=0x55a071987a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.117+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/4278920127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 0x55a071987a00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb0e40056d0 tx=0x7fb0e4005740 comp rx=0 tx=0).stop 2024-03-14T09:57:57.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.117+0000 7fb113c5bb80 1 -- 172.21.15.78:0/4278920127 shutdown_connections 2024-03-14T09:57:57.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.117+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/4278920127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a071987f40 0x55a07198a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.117+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/4278920127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a07197d700 0x55a07197dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.117+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/4278920127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 0x55a071987a00 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.117+0000 7fb113c5bb80 1 -- 172.21.15.78:0/4278920127 >> 172.21.15.78:0/4278920127 conn(0x55a0718f32a0 msgr2=0x55a071978b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:57.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.117+0000 7fb113c5bb80 1 -- 172.21.15.78:0/4278920127 shutdown_connections 2024-03-14T09:57:57.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.117+0000 7fb113c5bb80 1 -- 172.21.15.78:0/4278920127 wait complete. 2024-03-14T09:57:57.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.118+0000 7fb113c5bb80 1 -- start start 2024-03-14T09:57:57.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.118+0000 7fb113c5bb80 1 Processor -- start 2024-03-14T09:57:57.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.118+0000 7fb113c5bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a07197d700 0x55a0719879c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.118+0000 7fb113c5bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 0x55a071984d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.119+0000 7fb113c5bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a071987f40 0x55a071985440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.119+0000 7fb113c5bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07186be00 con 0x55a07197d700 2024-03-14T09:57:57.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.119+0000 7fb113c5bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07186baa0 con 0x55a071987f40 2024-03-14T09:57:57.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.119+0000 7fb113c5bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a071753720 con 0x55a07197e020 2024-03-14T09:57:57.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.119+0000 7fb113c5bb80 10 client.0 _ll_register_callbacks cb 0x55a0716d7420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:57.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.119+0000 7fb0fd800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a07197d700 0x55a0719879c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.119+0000 7fb0fe200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a071987f40 0x55a071985440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.119+0000 7fb0fe200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a071987f40 0x55a071985440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41580/0 (socket says 172.21.15.78:41580) 2024-03-14T09:57:57.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.119+0000 7fb0fe200700 1 -- 172.21.15.78:0/2499252039 learned_addr learned my addr 172.21.15.78:0/2499252039 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:57.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.119+0000 7fb0fce00700 1 --2- 172.21.15.78:0/2499252039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 0x55a071984d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.119+0000 7fb0fe200700 1 -- 172.21.15.78:0/2499252039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 msgr2=0x55a071984d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:57.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.119+0000 7fb0fe200700 1 --2- 172.21.15.78:0/2499252039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 0x55a071984d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.119+0000 7fb0fe200700 1 -- 172.21.15.78:0/2499252039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a07197d700 msgr2=0x55a0719879c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:57.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.119+0000 7fb0fe200700 1 --2- 172.21.15.78:0/2499252039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a07197d700 0x55a0719879c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.119+0000 7fb0fe200700 1 -- 172.21.15.78:0/2499252039 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f81a6900 con 0x55a071987f40 2024-03-14T09:57:57.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.120+0000 7fb0fe200700 1 --2- 172.21.15.78:0/2499252039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a071987f40 0x55a071985440 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0007be0 tx=0x7fb0f0007c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=228241f12166095c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:57.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.120+0000 7fb0df400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:57:57.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.120+0000 7fb0df400700 1 -- 172.21.15.78:0/2499252039 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f000e040 con 0x55a071987f40 2024-03-14T09:57:57.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.120+0000 7fb113c5bb80 1 -- 172.21.15.78:0/2499252039 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a0719707a0 con 0x55a071987f40 2024-03-14T09:57:57.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.120+0000 7fb0df400700 1 -- 172.21.15.78:0/2499252039 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fb0f0013bf0 con 0x55a071987f40 2024-03-14T09:57:57.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.120+0000 7fb0df400700 1 -- 172.21.15.78:0/2499252039 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0f0013df0 con 0x55a071987f40 2024-03-14T09:57:57.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.120+0000 7fb113c5bb80 10 client.26453 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:57.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.120+0000 7fb113c5bb80 1 -- 172.21.15.78:0/2499252039 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a0716fdbe0 con 0x55a071987f40 2024-03-14T09:57:57.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.120+0000 7fb0df400700 1 -- 172.21.15.78:0/2499252039 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a0719707a0 con 0x55a071987f40 2024-03-14T09:57:57.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.121+0000 7fb0df400700 1 -- 172.21.15.78:0/2499252039 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb0f001e3d0 con 0x55a071987f40 2024-03-14T09:57:57.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.121+0000 7fb113c5bb80 10 client.26453 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:57.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.121+0000 7fb113c5bb80 10 client.26453 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:57:57.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.121+0000 7fb113c5bb80 20 client.26453 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:57.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.121+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/2499252039 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a071997b40 0x55a071999f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.121+0000 7fb113c5bb80 4 client.26453 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:57:57.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.121+0000 7fb113c5bb80 1 -- 172.21.15.78:0/2499252039 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a0718c8780 con 0x55a071997b40 2024-03-14T09:57:57.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.121+0000 7fb0fd800700 1 --2- 172.21.15.78:0/2499252039 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a071997b40 0x55a071999f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.122+0000 7fb0df400700 10 client.26453 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.122+0000 7fb0fd800700 1 --2- 172.21.15.78:0/2499252039 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a071997b40 0x55a071999f60 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8db6cb9301056e66 server_cookie=f17087745fdb8ef8 in_seq=0 out_seq=0 2024-03-14T09:57:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.122+0000 7fb0df400700 1 -- 172.21.15.78:0/2499252039 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55a0718c8780 con 0x55a071997b40 2024-03-14T09:57:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.122+0000 7fb0df400700 10 client.26453 handle_command_reply: tid=0 2024-03-14T09:57:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.122+0000 7fb113c5bb80 1 client.26453 shutdown 2024-03-14T09:57:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.122+0000 7fb113c5bb80 1 -- 172.21.15.78:0/2499252039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a071987f40 msgr2=0x55a071985440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.122+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/2499252039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a071987f40 0x55a071985440 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0007be0 tx=0x7fb0f0007c10 comp rx=0 tx=0).stop 2024-03-14T09:57:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.123+0000 7fb113c5bb80 1 -- 172.21.15.78:0/2499252039 shutdown_connections 2024-03-14T09:57:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.123+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/2499252039 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a071997b40 0x55a071999f60 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.123+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/2499252039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a071987f40 0x55a071985440 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.123+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/2499252039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a07197d700 0x55a0719879c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.123+0000 7fb0df400700 0 client.26453 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.123+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/2499252039 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 0x55a071984d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.123+0000 7fb113c5bb80 1 -- 172.21.15.78:0/2499252039 >> 172.21.15.78:0/2499252039 conn(0x55a0718f32a0 msgr2=0x55a071978710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.123+0000 7fb113c5bb80 1 -- 172.21.15.78:0/2499252039 shutdown_connections 2024-03-14T09:57:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.123+0000 7fb113c5bb80 1 -- 172.21.15.78:0/2499252039 wait complete. 2024-03-14T09:57:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.123+0000 7fb113c5bb80 20 client.26453 trim_cache size 0 max 16384 2024-03-14T09:57:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.145+0000 7fb113c5bb80 1 Processor -- start 2024-03-14T09:57:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.145+0000 7fb113c5bb80 1 -- start start 2024-03-14T09:57:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb113c5bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197d700 0x55a07198ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb113c5bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a07197e020 0x55a0719874f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb113c5bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a071987f40 0x55a071978630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb113c5bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a071972dc0 con 0x55a071987f40 2024-03-14T09:57:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb113c5bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07197eb70 con 0x55a07197e020 2024-03-14T09:57:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb113c5bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a07197a110 con 0x55a07197d700 2024-03-14T09:57:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb0fce00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a07197e020 0x55a0719874f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb0fe200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a071987f40 0x55a071978630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb0fd800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197d700 0x55a07198ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb0fce00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a07197e020 0x55a0719874f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41594/0 (socket says 172.21.15.78:41594) 2024-03-14T09:57:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb0fd800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197d700 0x55a07198ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59406/0 (socket says 172.21.15.78:59406) 2024-03-14T09:57:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb0fce00700 1 -- 172.21.15.78:0/3477067073 learned_addr learned my addr 172.21.15.78:0/3477067073 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb0fe200700 1 -- 172.21.15.78:0/3477067073 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197d700 msgr2=0x55a07198ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb0fe200700 1 --2- 172.21.15.78:0/3477067073 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197d700 0x55a07198ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb0fe200700 1 -- 172.21.15.78:0/3477067073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a07197e020 msgr2=0x55a0719874f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb0fe200700 1 --2- 172.21.15.78:0/3477067073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a07197e020 0x55a0719874f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb0fe200700 1 -- 172.21.15.78:0/3477067073 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0719848b0 con 0x55a071987f40 2024-03-14T09:57:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.146+0000 7fb0fce00700 1 --2- 172.21.15.78:0/3477067073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a07197e020 0x55a0719874f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.147+0000 7fb0fe200700 1 --2- 172.21.15.78:0/3477067073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a071987f40 0x55a071978630 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb0f001e7e0 tx=0x7fb0f0009e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad25f08e7713337d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:57.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.147+0000 7fb0eb400700 1 -- 172.21.15.78:0/3477067073 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f000e040 con 0x55a071987f40 2024-03-14T09:57:57.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.147+0000 7fb0eb400700 1 -- 172.21.15.78:0/3477067073 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0f000f040 con 0x55a071987f40 2024-03-14T09:57:57.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.147+0000 7fb0eb400700 1 -- 172.21.15.78:0/3477067073 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0013c50 con 0x55a071987f40 2024-03-14T09:57:57.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.148+0000 7fb113c5bb80 1 -- 172.21.15.78:0/3477067073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a071987f40 msgr2=0x55a071978630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.148+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/3477067073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a071987f40 0x55a071978630 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb0f001e7e0 tx=0x7fb0f0009e50 comp rx=0 tx=0).stop 2024-03-14T09:57:57.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.148+0000 7fb113c5bb80 1 -- 172.21.15.78:0/3477067073 shutdown_connections 2024-03-14T09:57:57.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.148+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/3477067073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a071987f40 0x55a071978630 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.148+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/3477067073 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197d700 0x55a07198ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.148+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/3477067073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a07197e020 0x55a0719874f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.148+0000 7fb113c5bb80 1 -- 172.21.15.78:0/3477067073 >> 172.21.15.78:0/3477067073 conn(0x55a0718f32a0 msgr2=0x55a071977200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.148+0000 7fb113c5bb80 1 -- 172.21.15.78:0/3477067073 shutdown_connections 2024-03-14T09:57:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.148+0000 7fb113c5bb80 1 -- 172.21.15.78:0/3477067073 wait complete. 2024-03-14T09:57:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.148+0000 7fb113c5bb80 1 -- start start 2024-03-14T09:57:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.149+0000 7fb113c5bb80 1 Processor -- start 2024-03-14T09:57:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.149+0000 7fb113c5bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a07197d700 0x55a07198d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.149+0000 7fb113c5bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 0x55a07198d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.149+0000 7fb113c5bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a071987f40 0x55a07198ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.149+0000 7fb113c5bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0719e9670 con 0x55a071987f40 2024-03-14T09:57:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.149+0000 7fb113c5bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0719e97e0 con 0x55a07197d700 2024-03-14T09:57:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.149+0000 7fb113c5bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0719e99a0 con 0x55a07197e020 2024-03-14T09:57:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.150+0000 7fb113c5bb80 10 client.0 _ll_register_callbacks cb 0x7fb0f00094b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.150+0000 7fb0fd800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a07197d700 0x55a07198d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.150+0000 7fb0fd800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a07197d700 0x55a07198d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41608/0 (socket says 172.21.15.78:41608) 2024-03-14T09:57:57.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.150+0000 7fb0fd800700 1 -- 172.21.15.78:0/2254120537 learned_addr learned my addr 172.21.15.78:0/2254120537 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:57.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.150+0000 7fb0fce00700 1 --2- 172.21.15.78:0/2254120537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 0x55a07198d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.150+0000 7fb0fd800700 1 -- 172.21.15.78:0/2254120537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 msgr2=0x55a07198d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:57.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.150+0000 7fb0fd800700 1 --2- 172.21.15.78:0/2254120537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 0x55a07198d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.150+0000 7fb0fd800700 1 -- 172.21.15.78:0/2254120537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a071987f40 msgr2=0x55a07198ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:57.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.150+0000 7fb0fd800700 1 --2- 172.21.15.78:0/2254120537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a071987f40 0x55a07198ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.150+0000 7fb0fd800700 1 -- 172.21.15.78:0/2254120537 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a071751430 con 0x55a07197d700 2024-03-14T09:57:57.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.150+0000 7fb0fce00700 1 --2- 172.21.15.78:0/2254120537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 0x55a07198d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:57:57.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.151+0000 7fb0fd800700 1 --2- 172.21.15.78:0/2254120537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a07197d700 0x55a07198d350 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00b3e0 tx=0x7fb0ec009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56b6d73da1000fd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:57.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.151+0000 7fb0df400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:57:57.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.151+0000 7fb0df400700 1 -- 172.21.15.78:0/2254120537 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec0353a0 con 0x55a07197d700 2024-03-14T09:57:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.151+0000 7fb113c5bb80 1 -- 172.21.15.78:0/2254120537 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb0e409d800 con 0x55a07197d700 2024-03-14T09:57:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.151+0000 7fb0df400700 1 -- 172.21.15.78:0/2254120537 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fb0ec00bad0 con 0x55a07197d700 2024-03-14T09:57:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.151+0000 7fb0df400700 1 -- 172.21.15.78:0/2254120537 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0ec00bc90 con 0x55a07197d700 2024-03-14T09:57:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.151+0000 7fb113c5bb80 10 client.26459 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.151+0000 7fb113c5bb80 1 -- 172.21.15.78:0/2254120537 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a0716fdbe0 con 0x55a07197d700 2024-03-14T09:57:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.151+0000 7fb0df400700 1 -- 172.21.15.78:0/2254120537 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0e409d800 con 0x55a07197d700 2024-03-14T09:57:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.152+0000 7fb0df400700 1 -- 172.21.15.78:0/2254120537 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb0ec03e8c0 con 0x55a07197d700 2024-03-14T09:57:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.152+0000 7fb113c5bb80 10 client.26459 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.152+0000 7fb113c5bb80 10 client.26459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:57:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.152+0000 7fb113c5bb80 20 client.26459 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.152+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/2254120537 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a0719eb140 0x55a0719ed560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.152+0000 7fb113c5bb80 4 client.26459 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T09:57:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.152+0000 7fb113c5bb80 1 -- 172.21.15.78:0/2254120537 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a07171ed70 con 0x55a0719eb140 2024-03-14T09:57:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.152+0000 7fb0fce00700 1 --2- 172.21.15.78:0/2254120537 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a0719eb140 0x55a0719ed560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.152+0000 7fb0df400700 10 client.26459 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.152+0000 7fb0fce00700 1 --2- 172.21.15.78:0/2254120537 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a0719eb140 0x55a0719ed560 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a77796b49ab2a6a5 server_cookie=120da8f0b5663b30 in_seq=0 out_seq=0 2024-03-14T09:57:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.153+0000 7fb0df400700 1 -- 172.21.15.78:0/2254120537 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a07171ed70 con 0x55a0719eb140 2024-03-14T09:57:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.153+0000 7fb0df400700 10 client.26459 handle_command_reply: tid=0 2024-03-14T09:57:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.153+0000 7fb113c5bb80 1 client.26459 shutdown 2024-03-14T09:57:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.154+0000 7fb113c5bb80 1 -- 172.21.15.78:0/2254120537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a07197d700 msgr2=0x55a07198d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.154+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/2254120537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a07197d700 0x55a07198d350 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00b3e0 tx=0x7fb0ec009310 comp rx=0 tx=0).stop 2024-03-14T09:57:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.154+0000 7fb113c5bb80 1 -- 172.21.15.78:0/2254120537 shutdown_connections 2024-03-14T09:57:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.154+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/2254120537 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a0719eb140 0x55a0719ed560 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.154+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/2254120537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a07197d700 0x55a07198d350 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.154+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/2254120537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a071987f40 0x55a07198ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.154+0000 7fb113c5bb80 1 --2- 172.21.15.78:0/2254120537 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a07197e020 0x55a07198d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.154+0000 7fb0df400700 0 client.26459 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.154+0000 7fb113c5bb80 1 -- 172.21.15.78:0/2254120537 >> 172.21.15.78:0/2254120537 conn(0x55a0718f32a0 msgr2=0x55a071978630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:57.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.154+0000 7fb113c5bb80 1 -- 172.21.15.78:0/2254120537 shutdown_connections 2024-03-14T09:57:57.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.154+0000 7fb113c5bb80 1 -- 172.21.15.78:0/2254120537 wait complete. 2024-03-14T09:57:57.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.154+0000 7fb113c5bb80 20 client.26459 trim_cache size 0 max 16384 2024-03-14T09:57:57.158 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:57:57.158 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T09:57:57.158 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "03f9e43a-d786-46b5-bff7-77c6b7d7bc8a", 2024-03-14T09:57:57.158 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T09:57:57.158 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:57:57.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.155+0000 7fb100200700 1 -- 172.21.15.78:0/3228183013 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0e00a0c00 msgr2=0x7fb0e00a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.155+0000 7fb100200700 1 --2- 172.21.15.78:0/3228183013 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0e00a0c00 0x7fb0e00a30c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec006010 tx=0x7fb0ec01a040 comp rx=0 tx=0).stop 2024-03-14T09:57:57.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.155+0000 7fb100200700 1 -- 172.21.15.78:0/3228183013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0f8106040 msgr2=0x7fb0f819fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.155+0000 7fb100200700 1 --2- 172.21.15.78:0/3228183013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0f8106040 0x7fb0f819fa90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb0e400e970 tx=0x7fb0e400ec80 comp rx=0 tx=0).stop 2024-03-14T09:57:57.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.155+0000 7fb100200700 1 -- 172.21.15.78:0/3228183013 shutdown_connections 2024-03-14T09:57:57.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.155+0000 7fb100200700 1 --2- 172.21.15.78:0/3228183013 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0e00a0c00 0x7fb0e00a30c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.155+0000 7fb100200700 1 --2- 172.21.15.78:0/3228183013 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0f8199770 0x7fb0f8199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.155+0000 7fb100200700 1 --2- 172.21.15.78:0/3228183013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0f8100f20 0x7fb0f819f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.155+0000 7fb100200700 1 --2- 172.21.15.78:0/3228183013 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb0f8106040 0x7fb0f819fa90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.155+0000 7fb100200700 1 -- 172.21.15.78:0/3228183013 >> 172.21.15.78:0/3228183013 conn(0x7fb0f80fc7a0 msgr2=0x7fb0f80fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:57.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.156+0000 7fb100200700 1 -- 172.21.15.78:0/3228183013 shutdown_connections 2024-03-14T09:57:57.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.156+0000 7fb100200700 1 -- 172.21.15.78:0/3228183013 wait complete. 2024-03-14T09:57:57.179 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:57:57.244 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:57 smithi078 ceph-mon[38143]: pgmap v116: 129 pgs: 129 active+clean; 133 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.1 MiB/s wr, 533 op/s 2024-03-14T09:57:57.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.436+0000 7fc722000700 1 -- 172.21.15.78:0/622783078 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc71c073150 msgr2=0x7fc71c073530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.436+0000 7fc722000700 1 --2- 172.21.15.78:0/622783078 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc71c073150 0x7fc71c073530 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc70c009990 tx=0x7fc70c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.436+0000 7fc722000700 1 -- 172.21.15.78:0/622783078 shutdown_connections 2024-03-14T09:57:57.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.436+0000 7fc722000700 1 --2- 172.21.15.78:0/622783078 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc71c073150 0x7fc71c073530 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.436+0000 7fc722000700 1 --2- 172.21.15.78:0/622783078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc71c111cd0 0x7fc71c1140c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.436+0000 7fc722000700 1 --2- 172.21.15.78:0/622783078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc71c073a70 0x7fc71c111790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.436+0000 7fc722000700 1 -- 172.21.15.78:0/622783078 >> 172.21.15.78:0/622783078 conn(0x7fc71c0fc7b0 msgr2=0x7fc71c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:57.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.436+0000 7fc722000700 1 -- 172.21.15.78:0/622783078 shutdown_connections 2024-03-14T09:57:57.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.436+0000 7fc722000700 1 -- 172.21.15.78:0/622783078 wait complete. 2024-03-14T09:57:57.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.438+0000 7fc722000700 1 Processor -- start 2024-03-14T09:57:57.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.438+0000 7fc722000700 1 -- start start 2024-03-14T09:57:57.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.439+0000 7fc722000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc71c073150 0x7fc71c1a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.439+0000 7fc722000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc71c073a70 0x7fc71c1a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.439+0000 7fc722000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc71c111cd0 0x7fc71c19da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.439+0000 7fc722000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c19e0d0 con 0x7fc71c073150 2024-03-14T09:57:57.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.439+0000 7fc722000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c19e240 con 0x7fc71c111cd0 2024-03-14T09:57:57.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.439+0000 7fc722000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c19e400 con 0x7fc71c073a70 2024-03-14T09:57:57.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.439+0000 7fc71b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc71c073150 0x7fc71c1a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.439+0000 7fc71b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc71c073150 0x7fc71c1a3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:59018/0 (socket says 172.21.15.78:59018) 2024-03-14T09:57:57.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.439+0000 7fc71b400700 1 -- 172.21.15.78:0/3814301574 learned_addr learned my addr 172.21.15.78:0/3814301574 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:57.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.439+0000 7fc71aa00700 1 --2- 172.21.15.78:0/3814301574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc71c073a70 0x7fc71c1a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.439+0000 7fc71be00700 1 --2- 172.21.15.78:0/3814301574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc71c111cd0 0x7fc71c19da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.440+0000 7fc71b400700 1 -- 172.21.15.78:0/3814301574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc71c073a70 msgr2=0x7fc71c1a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.440+0000 7fc71b400700 1 --2- 172.21.15.78:0/3814301574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc71c073a70 0x7fc71c1a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.440+0000 7fc71b400700 1 -- 172.21.15.78:0/3814301574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc71c111cd0 msgr2=0x7fc71c19da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.440+0000 7fc71b400700 1 --2- 172.21.15.78:0/3814301574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc71c111cd0 0x7fc71c19da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.440+0000 7fc71b400700 1 -- 172.21.15.78:0/3814301574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c0095f0 con 0x7fc71c073150 2024-03-14T09:57:57.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.440+0000 7fc71aa00700 1 --2- 172.21.15.78:0/3814301574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc71c073a70 0x7fc71c1a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:57.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.440+0000 7fc71be00700 1 --2- 172.21.15.78:0/3814301574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc71c111cd0 0x7fc71c19da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:57.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.440+0000 7fc71b400700 1 --2- 172.21.15.78:0/3814301574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc71c073150 0x7fc71c1a3940 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc70c005910 tx=0x7fc70c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:57.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.440+0000 7fc713e00700 1 -- 172.21.15.78:0/3814301574 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c01d070 con 0x7fc71c073150 2024-03-14T09:57:57.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.441+0000 7fc713e00700 1 -- 172.21.15.78:0/3814301574 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc70c022470 con 0x7fc71c073150 2024-03-14T09:57:57.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.441+0000 7fc722000700 1 -- 172.21.15.78:0/3814301574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71c19e680 con 0x7fc71c073150 2024-03-14T09:57:57.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.441+0000 7fc713e00700 1 -- 172.21.15.78:0/3814301574 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c017650 con 0x7fc71c073150 2024-03-14T09:57:57.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.441+0000 7fc722000700 1 -- 172.21.15.78:0/3814301574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71c1aa930 con 0x7fc71c073150 2024-03-14T09:57:57.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.446+0000 7fc713e00700 1 -- 172.21.15.78:0/3814301574 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fc70c00fac0 con 0x7fc71c073150 2024-03-14T09:57:57.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.446+0000 7fc735ae5b80 1 Processor -- start 2024-03-14T09:57:57.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.447+0000 7fc735ae5b80 1 -- start start 2024-03-14T09:57:57.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.447+0000 7fc735ae5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4a680 0x563f83b4aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.447+0000 7fc735ae5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b4afa0 0x563f83b54980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.447+0000 7fc735ae5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b54ec0 0x563f83b572b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.447+0000 7fc735ae5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f83aa3990 con 0x563f83b54ec0 2024-03-14T09:57:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.447+0000 7fc735ae5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f83aa1350 con 0x563f83b4afa0 2024-03-14T09:57:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.447+0000 7fc735ae5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f83a9cc40 con 0x563f83b4a680 2024-03-14T09:57:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.448+0000 7fc71be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4a680 0x563f83b4aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.448+0000 7fc71be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4a680 0x563f83b4aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59436/0 (socket says 172.21.15.78:59436) 2024-03-14T09:57:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.448+0000 7fc71be00700 1 -- 172.21.15.78:0/2275466979 learned_addr learned my addr 172.21.15.78:0/2275466979 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:57.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.448+0000 7fc71b400700 1 --2- 172.21.15.78:0/2275466979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b4afa0 0x563f83b54980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.448+0000 7fc713e00700 1 --2- 172.21.15.78:0/3814301574 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fc7000a0c80 0x7fc7000a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.448+0000 7fc71be00700 1 -- 172.21.15.78:0/2275466979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b4afa0 msgr2=0x563f83b54980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.448+0000 7fc71be00700 1 --2- 172.21.15.78:0/2275466979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b4afa0 0x563f83b54980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.448+0000 7fc71be00700 1 -- 172.21.15.78:0/2275466979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b54ec0 msgr2=0x563f83b572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.448+0000 7fc71be00700 1 --2- 172.21.15.78:0/2275466979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b54ec0 0x563f83b572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.448+0000 7fc71be00700 1 -- 172.21.15.78:0/2275466979 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f8391e3b0 con 0x563f83b4a680 2024-03-14T09:57:57.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.448+0000 7fc713e00700 1 -- 172.21.15.78:0/3814301574 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fc70c0cda40 con 0x7fc71c073150 2024-03-14T09:57:57.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.449+0000 7fc71be00700 1 --2- 172.21.15.78:0/3814301574 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fc7000a0c80 0x7fc7000a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.449+0000 7fc71be00700 1 --2- 172.21.15.78:0/2275466979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4a680 0x563f83b4aa60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc714007680 tx=0x7fc714007990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc5a2382523d7dbd server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:57.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.449+0000 7fc707e00700 1 -- 172.21.15.78:0/2275466979 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc714012970 con 0x563f83b4a680 2024-03-14T09:57:57.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.449+0000 7fc707e00700 1 -- 172.21.15.78:0/2275466979 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc71400f040 con 0x563f83b4a680 2024-03-14T09:57:57.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.449+0000 7fc707e00700 1 -- 172.21.15.78:0/2275466979 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc714020940 con 0x563f83b4a680 2024-03-14T09:57:57.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.450+0000 7fc735ae5b80 1 -- 172.21.15.78:0/2275466979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4a680 msgr2=0x563f83b4aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.450+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/2275466979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4a680 0x563f83b4aa60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc714007680 tx=0x7fc714007990 comp rx=0 tx=0).stop 2024-03-14T09:57:57.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.450+0000 7fc71be00700 1 --2- 172.21.15.78:0/3814301574 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fc7000a0c80 0x7fc7000a3140 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc71401f000 tx=0x7fc714020000 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:57.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.451+0000 7fc735ae5b80 1 -- 172.21.15.78:0/2275466979 shutdown_connections 2024-03-14T09:57:57.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.451+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/2275466979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b54ec0 0x563f83b572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.451+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/2275466979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4a680 0x563f83b4aa60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.451+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/2275466979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b4afa0 0x563f83b54980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.451+0000 7fc735ae5b80 1 -- 172.21.15.78:0/2275466979 >> 172.21.15.78:0/2275466979 conn(0x563f83ac0220 msgr2=0x563f83b45aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:57.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.451+0000 7fc735ae5b80 1 -- 172.21.15.78:0/2275466979 shutdown_connections 2024-03-14T09:57:57.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.451+0000 7fc735ae5b80 1 -- 172.21.15.78:0/2275466979 wait complete. 2024-03-14T09:57:57.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.452+0000 7fc735ae5b80 1 -- start start 2024-03-14T09:57:57.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.453+0000 7fc735ae5b80 1 Processor -- start 2024-03-14T09:57:57.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.453+0000 7fc735ae5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4a680 0x563f83b548c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.453+0000 7fc735ae5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b4afa0 0x563f83b51c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.453+0000 7fc735ae5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b54ec0 0x563f83b52280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.453+0000 7fc735ae5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f83a38d80 con 0x563f83b4a680 2024-03-14T09:57:57.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.453+0000 7fc735ae5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f83a38a20 con 0x563f83b4afa0 2024-03-14T09:57:57.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.453+0000 7fc735ae5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f839206a0 con 0x563f83b54ec0 2024-03-14T09:57:57.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.453+0000 7fc735ae5b80 10 client.0 _ll_register_callbacks cb 0x563f838a43a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:57.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.454+0000 7fc71be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b54ec0 0x563f83b52280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.454+0000 7fc71aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b4afa0 0x563f83b51c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.454+0000 7fc71aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b4afa0 0x563f83b51c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41654/0 (socket says 172.21.15.78:41654) 2024-03-14T09:57:57.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.454+0000 7fc71be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b54ec0 0x563f83b52280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59444/0 (socket says 172.21.15.78:59444) 2024-03-14T09:57:57.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.454+0000 7fc71aa00700 1 -- 172.21.15.78:0/548468794 learned_addr learned my addr 172.21.15.78:0/548468794 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:57.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.454+0000 7fc71b400700 1 --2- 172.21.15.78:0/548468794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4a680 0x563f83b548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.454+0000 7fc71aa00700 1 -- 172.21.15.78:0/548468794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b54ec0 msgr2=0x563f83b52280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:57.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.454+0000 7fc71aa00700 1 --2- 172.21.15.78:0/548468794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b54ec0 0x563f83b52280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.454+0000 7fc71aa00700 1 -- 172.21.15.78:0/548468794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4a680 msgr2=0x563f83b548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:57.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.454+0000 7fc71aa00700 1 --2- 172.21.15.78:0/548468794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4a680 0x563f83b548c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.454+0000 7fc71aa00700 1 -- 172.21.15.78:0/548468794 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc714009dd0 con 0x563f83b4afa0 2024-03-14T09:57:57.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.454+0000 7fc71b400700 1 --2- 172.21.15.78:0/548468794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4a680 0x563f83b548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:57.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.455+0000 7fc71aa00700 1 --2- 172.21.15.78:0/548468794 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b4afa0 0x563f83b51c00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc708011a70 tx=0x7fc708011d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b373979a0616f6e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:57.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.455+0000 7fc704c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:57:57.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.455+0000 7fc704c00700 1 -- 172.21.15.78:0/548468794 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc708004520 con 0x563f83b4afa0 2024-03-14T09:57:57.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.455+0000 7fc704c00700 1 -- 172.21.15.78:0/548468794 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fc70800f7b0 con 0x563f83b4afa0 2024-03-14T09:57:57.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.455+0000 7fc735ae5b80 1 -- 172.21.15.78:0/548468794 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563f83b3d720 con 0x563f83b4afa0 2024-03-14T09:57:57.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.455+0000 7fc704c00700 1 -- 172.21.15.78:0/548468794 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc70800f970 con 0x563f83b4afa0 2024-03-14T09:57:57.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.455+0000 7fc735ae5b80 10 client.26471 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:57.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.455+0000 7fc735ae5b80 1 -- 172.21.15.78:0/548468794 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563f838cab60 con 0x563f83b4afa0 2024-03-14T09:57:57.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.456+0000 7fc704c00700 1 -- 172.21.15.78:0/548468794 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563f83b3d720 con 0x563f83b4afa0 2024-03-14T09:57:57.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.456+0000 7fc704c00700 1 -- 172.21.15.78:0/548468794 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc70801a440 con 0x563f83b4afa0 2024-03-14T09:57:57.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.456+0000 7fc735ae5b80 10 client.26471 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:57.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.456+0000 7fc735ae5b80 10 client.26471 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:57:57.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.456+0000 7fc735ae5b80 20 client.26471 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:57.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.456+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/548468794 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563f83b75b90 0x563f83b77fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.456+0000 7fc735ae5b80 4 client.26471 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:57:57.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.456+0000 7fc735ae5b80 1 -- 172.21.15.78:0/548468794 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f83a95700 con 0x563f83b75b90 2024-03-14T09:57:57.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.457+0000 7fc71b400700 1 --2- 172.21.15.78:0/548468794 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563f83b75b90 0x563f83b77fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.457+0000 7fc71b400700 1 --2- 172.21.15.78:0/548468794 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563f83b75b90 0x563f83b77fb0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd787646895ba721 server_cookie=ad0146e129a65eb0 in_seq=0 out_seq=0 2024-03-14T09:57:57.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.457+0000 7fc704c00700 10 client.26471 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:57.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.458+0000 7fc704c00700 1 -- 172.21.15.78:0/548468794 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x563f83a95700 con 0x563f83b75b90 2024-03-14T09:57:57.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.458+0000 7fc704c00700 10 client.26471 handle_command_reply: tid=0 2024-03-14T09:57:57.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.459+0000 7fc735ae5b80 1 client.26471 shutdown 2024-03-14T09:57:57.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.459+0000 7fc735ae5b80 1 -- 172.21.15.78:0/548468794 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b4afa0 msgr2=0x563f83b51c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.459+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/548468794 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b4afa0 0x563f83b51c00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc708011a70 tx=0x7fc708011d80 comp rx=0 tx=0).stop 2024-03-14T09:57:57.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.460+0000 7fc735ae5b80 1 -- 172.21.15.78:0/548468794 shutdown_connections 2024-03-14T09:57:57.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.460+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/548468794 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b54ec0 0x563f83b52280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.460+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/548468794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4a680 0x563f83b548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.460+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/548468794 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563f83b75b90 0x563f83b77fb0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.460+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/548468794 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b4afa0 0x563f83b51c00 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.460+0000 7fc735ae5b80 1 -- 172.21.15.78:0/548468794 >> 172.21.15.78:0/548468794 conn(0x563f83ac0220 msgr2=0x563f83b45690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:57.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.460+0000 7fc704c00700 0 client.26471 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:57.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.460+0000 7fc735ae5b80 1 -- 172.21.15.78:0/548468794 shutdown_connections 2024-03-14T09:57:57.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.460+0000 7fc735ae5b80 1 -- 172.21.15.78:0/548468794 wait complete. 2024-03-14T09:57:57.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.460+0000 7fc735ae5b80 20 client.26471 trim_cache size 0 max 16384 2024-03-14T09:57:57.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.481+0000 7fc735ae5b80 1 Processor -- start 2024-03-14T09:57:57.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.482+0000 7fc735ae5b80 1 -- start start 2024-03-14T09:57:57.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.482+0000 7fc735ae5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4a680 0x563f83b6cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.482+0000 7fc735ae5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4afa0 0x563f83b51140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.482+0000 7fc735ae5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b54ec0 0x563f83b4cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.482+0000 7fc735ae5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f83b42280 con 0x563f83b4a680 2024-03-14T09:57:57.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.482+0000 7fc735ae5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f83b467a0 con 0x563f83b54ec0 2024-03-14T09:57:57.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.482+0000 7fc735ae5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f83b4c110 con 0x563f83b4afa0 2024-03-14T09:57:57.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.482+0000 7fc71aa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4afa0 0x563f83b51140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.482+0000 7fc71b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4a680 0x563f83b6cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.482+0000 7fc71aa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4afa0 0x563f83b51140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59458/0 (socket says 172.21.15.78:59458) 2024-03-14T09:57:57.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.482+0000 7fc71aa00700 1 -- 172.21.15.78:0/2252036014 learned_addr learned my addr 172.21.15.78:0/2252036014 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:57.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.482+0000 7fc71be00700 1 --2- 172.21.15.78:0/2252036014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b54ec0 0x563f83b4cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.483+0000 7fc71aa00700 1 -- 172.21.15.78:0/2252036014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b54ec0 msgr2=0x563f83b4cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:57.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.483+0000 7fc71aa00700 1 --2- 172.21.15.78:0/2252036014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b54ec0 0x563f83b4cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.483+0000 7fc71aa00700 1 -- 172.21.15.78:0/2252036014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4a680 msgr2=0x563f83b6cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:57.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.483+0000 7fc71aa00700 1 --2- 172.21.15.78:0/2252036014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4a680 0x563f83b6cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.483+0000 7fc71aa00700 1 -- 172.21.15.78:0/2252036014 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f83a62ea0 con 0x563f83b4afa0 2024-03-14T09:57:57.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.483+0000 7fc71b400700 1 --2- 172.21.15.78:0/2252036014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4a680 0x563f83b6cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:57.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.483+0000 7fc71aa00700 1 --2- 172.21.15.78:0/2252036014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4afa0 0x563f83b51140 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc70801a8c0 tx=0x7fc708005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a40284cab2bcc4f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:57.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.484+0000 7fc711200700 1 -- 172.21.15.78:0/2252036014 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc708009290 con 0x563f83b4afa0 2024-03-14T09:57:57.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.484+0000 7fc711200700 1 -- 172.21.15.78:0/2252036014 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc708004320 con 0x563f83b4afa0 2024-03-14T09:57:57.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.484+0000 7fc711200700 1 -- 172.21.15.78:0/2252036014 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70800fde0 con 0x563f83b4afa0 2024-03-14T09:57:57.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.485+0000 7fc735ae5b80 1 -- 172.21.15.78:0/2252036014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4afa0 msgr2=0x563f83b51140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.485+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/2252036014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4afa0 0x563f83b51140 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc70801a8c0 tx=0x7fc708005780 comp rx=0 tx=0).stop 2024-03-14T09:57:57.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.486+0000 7fc735ae5b80 1 -- 172.21.15.78:0/2252036014 shutdown_connections 2024-03-14T09:57:57.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.486+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/2252036014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b54ec0 0x563f83b4cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.486+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/2252036014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4a680 0x563f83b6cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.486+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/2252036014 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4afa0 0x563f83b51140 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.486+0000 7fc735ae5b80 1 -- 172.21.15.78:0/2252036014 >> 172.21.15.78:0/2252036014 conn(0x563f83ac0220 msgr2=0x563f83b77ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:57.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.486+0000 7fc735ae5b80 1 -- 172.21.15.78:0/2252036014 shutdown_connections 2024-03-14T09:57:57.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.486+0000 7fc735ae5b80 1 -- 172.21.15.78:0/2252036014 wait complete. 2024-03-14T09:57:57.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.487+0000 7fc735ae5b80 1 -- start start 2024-03-14T09:57:57.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.487+0000 7fc735ae5b80 1 Processor -- start 2024-03-14T09:57:57.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.488+0000 7fc735ae5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4a680 0x563f83b6ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.488+0000 7fc735ae5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4afa0 0x563f83b6b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.488+0000 7fc735ae5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b54ec0 0x563f83b6b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.488+0000 7fc735ae5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f83bc67e0 con 0x563f83b4afa0 2024-03-14T09:57:57.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.488+0000 7fc735ae5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f83bc6950 con 0x563f83b54ec0 2024-03-14T09:57:57.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.488+0000 7fc735ae5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f83bc6b10 con 0x563f83b4a680 2024-03-14T09:57:57.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.488+0000 7fc735ae5b80 10 client.0 _ll_register_callbacks cb 0x7fc708011540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:57:57.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.488+0000 7fc71be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b54ec0 0x563f83b6b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.488+0000 7fc71be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b54ec0 0x563f83b6b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:41672/0 (socket says 172.21.15.78:41672) 2024-03-14T09:57:57.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.488+0000 7fc71be00700 1 -- 172.21.15.78:0/3525373185 learned_addr learned my addr 172.21.15.78:0/3525373185 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:57:57.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.488+0000 7fc71b400700 1 --2- 172.21.15.78:0/3525373185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4a680 0x563f83b6ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.488+0000 7fc71aa00700 1 --2- 172.21.15.78:0/3525373185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4afa0 0x563f83b6b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.489+0000 7fc71be00700 1 -- 172.21.15.78:0/3525373185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4a680 msgr2=0x563f83b6ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:57.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.489+0000 7fc71be00700 1 --2- 172.21.15.78:0/3525373185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4a680 0x563f83b6ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.489+0000 7fc71be00700 1 -- 172.21.15.78:0/3525373185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4afa0 msgr2=0x563f83b6b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:57:57.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.489+0000 7fc71be00700 1 --2- 172.21.15.78:0/3525373185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4afa0 0x563f83b6b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.489+0000 7fc71be00700 1 -- 172.21.15.78:0/3525373185 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71c1aa930 con 0x563f83b54ec0 2024-03-14T09:57:57.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.489+0000 7fc71aa00700 1 --2- 172.21.15.78:0/3525373185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4afa0 0x563f83b6b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:57:57.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.489+0000 7fc71be00700 1 --2- 172.21.15.78:0/3525373185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b54ec0 0x563f83b6b5c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc714007d90 tx=0x7fc714009d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e157e7781edf09f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:57:57.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.489+0000 7fc704c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:57:57.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.489+0000 7fc704c00700 1 -- 172.21.15.78:0/3525373185 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc714020c00 con 0x563f83b54ec0 2024-03-14T09:57:57.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.490+0000 7fc704c00700 1 -- 172.21.15.78:0/3525373185 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fc714020d60 con 0x563f83b54ec0 2024-03-14T09:57:57.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.490+0000 7fc735ae5b80 1 -- 172.21.15.78:0/3525373185 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc71400f040 con 0x563f83b54ec0 2024-03-14T09:57:57.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.490+0000 7fc704c00700 1 -- 172.21.15.78:0/3525373185 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc714020630 con 0x563f83b54ec0 2024-03-14T09:57:57.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.490+0000 7fc735ae5b80 10 client.26477 fetch_fsmap learned FSMap version 25 2024-03-14T09:57:57.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.490+0000 7fc735ae5b80 1 -- 172.21.15.78:0/3525373185 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563f838cab60 con 0x563f83b54ec0 2024-03-14T09:57:57.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.491+0000 7fc704c00700 1 -- 172.21.15.78:0/3525373185 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc71400f040 con 0x563f83b54ec0 2024-03-14T09:57:57.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.491+0000 7fc704c00700 1 -- 172.21.15.78:0/3525373185 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc71402d550 con 0x563f83b54ec0 2024-03-14T09:57:57.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.491+0000 7fc735ae5b80 10 client.26477 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:57:57.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.491+0000 7fc735ae5b80 10 client.26477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:57:57.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.491+0000 7fc735ae5b80 20 client.26477 populate_metadata read hostname 'smithi078' 2024-03-14T09:57:57.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.491+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/3525373185 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563f83bc8220 0x563f83bca640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:57:57.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.491+0000 7fc735ae5b80 4 client.26477 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:57:57.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.491+0000 7fc735ae5b80 1 -- 172.21.15.78:0/3525373185 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563f838ebcf0 con 0x563f83bc8220 2024-03-14T09:57:57.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.492+0000 7fc71b400700 1 --2- 172.21.15.78:0/3525373185 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563f83bc8220 0x563f83bca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:57:57.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.492+0000 7fc704c00700 10 client.26477 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:57.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.492+0000 7fc71b400700 1 --2- 172.21.15.78:0/3525373185 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563f83bc8220 0x563f83bca640 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=567ca049c60d92b4 server_cookie=52232ab93d08a0f9 in_seq=0 out_seq=0 2024-03-14T09:57:57.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.493+0000 7fc704c00700 1 -- 172.21.15.78:0/3525373185 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x563f838ebcf0 con 0x563f83bc8220 2024-03-14T09:57:57.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.493+0000 7fc704c00700 10 client.26477 handle_command_reply: tid=0 2024-03-14T09:57:57.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.493+0000 7fc735ae5b80 1 client.26477 shutdown 2024-03-14T09:57:57.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.494+0000 7fc735ae5b80 1 -- 172.21.15.78:0/3525373185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b54ec0 msgr2=0x563f83b6b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.494+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/3525373185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b54ec0 0x563f83b6b5c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc714007d90 tx=0x7fc714009d40 comp rx=0 tx=0).stop 2024-03-14T09:57:57.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.494+0000 7fc735ae5b80 1 -- 172.21.15.78:0/3525373185 shutdown_connections 2024-03-14T09:57:57.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.494+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/3525373185 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563f83bc8220 0x563f83bca640 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.494+0000 7fc704c00700 0 client.26477 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:57:57.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.494+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/3525373185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f83b54ec0 0x563f83b6b5c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.494+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/3525373185 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f83b4a680 0x563f83b6ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.494+0000 7fc735ae5b80 1 --2- 172.21.15.78:0/3525373185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f83b4afa0 0x563f83b6b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.494+0000 7fc735ae5b80 1 -- 172.21.15.78:0/3525373185 >> 172.21.15.78:0/3525373185 conn(0x563f83ac0220 msgr2=0x563f83b4cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:57.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.494+0000 7fc735ae5b80 1 -- 172.21.15.78:0/3525373185 shutdown_connections 2024-03-14T09:57:57.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.494+0000 7fc735ae5b80 1 -- 172.21.15.78:0/3525373185 wait complete. 2024-03-14T09:57:57.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.495+0000 7fc735ae5b80 20 client.26477 trim_cache size 0 max 16384 2024-03-14T09:57:57.497 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:57:57.497 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "scrub active (1216 inodes in the stack)", 2024-03-14T09:57:57.497 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T09:57:57.497 INFO:teuthology.orchestra.run.smithi078.stdout: "03f9e43a-d786-46b5-bff7-77c6b7d7bc8a": { 2024-03-14T09:57:57.497 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T09:57:57.497 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "03f9e43a-d786-46b5-bff7-77c6b7d7bc8a", 2024-03-14T09:57:57.497 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T09:57:57.497 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:57:57.497 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:57:57.497 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:57:57.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.495+0000 7fc722000700 1 -- 172.21.15.78:0/3814301574 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fc7000a0c80 msgr2=0x7fc7000a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.495+0000 7fc722000700 1 --2- 172.21.15.78:0/3814301574 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fc7000a0c80 0x7fc7000a3140 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc71401f000 tx=0x7fc714020000 comp rx=0 tx=0).stop 2024-03-14T09:57:57.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.495+0000 7fc722000700 1 -- 172.21.15.78:0/3814301574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc71c073150 msgr2=0x7fc71c1a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:57:57.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.495+0000 7fc722000700 1 --2- 172.21.15.78:0/3814301574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc71c073150 0x7fc71c1a3940 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc70c005910 tx=0x7fc70c004730 comp rx=0 tx=0).stop 2024-03-14T09:57:57.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.495+0000 7fc722000700 1 -- 172.21.15.78:0/3814301574 shutdown_connections 2024-03-14T09:57:57.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.495+0000 7fc722000700 1 --2- 172.21.15.78:0/3814301574 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fc7000a0c80 0x7fc7000a3140 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.495+0000 7fc722000700 1 --2- 172.21.15.78:0/3814301574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc71c111cd0 0x7fc71c19da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.495+0000 7fc722000700 1 --2- 172.21.15.78:0/3814301574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc71c073150 0x7fc71c1a3940 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.495+0000 7fc722000700 1 --2- 172.21.15.78:0/3814301574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc71c073a70 0x7fc71c1a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:57:57.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.495+0000 7fc722000700 1 -- 172.21.15.78:0/3814301574 >> 172.21.15.78:0/3814301574 conn(0x7fc71c0fc7b0 msgr2=0x7fc71c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:57:57.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.496+0000 7fc722000700 1 -- 172.21.15.78:0/3814301574 shutdown_connections 2024-03-14T09:57:57.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:57:57.496+0000 7fc722000700 1 -- 172.21.15.78:0/3814301574 wait complete. 2024-03-14T09:57:57.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:57 smithi152 ceph-mon[48279]: pgmap v116: 129 pgs: 129 active+clean; 133 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.1 MiB/s wr, 533 op/s 2024-03-14T09:57:57.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:57 smithi132 ceph-mon[47959]: pgmap v116: 129 pgs: 129 active+clean; 133 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.1 MiB/s wr, 533 op/s 2024-03-14T09:57:57.521 INFO:tasks.cephfs.filesystem:scrub status for tag:03f9e43a-d786-46b5-bff7-77c6b7d7bc8a - {'path': '/', 'tag': '03f9e43a-d786-46b5-bff7-77c6b7d7bc8a', 'options': 'recursive,force'} 2024-03-14T09:57:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:58 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T09:57:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:58 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:57:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:58 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T09:57:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:58 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:57:59.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:58 smithi078 ceph-mon[38143]: pgmap v117: 129 pgs: 129 active+clean; 133 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.0 MiB/s wr, 351 op/s 2024-03-14T09:57:59.248 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:58 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T09:57:59.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:58 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:57:59.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:58 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T09:57:59.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:58 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:57:59.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:58 smithi152 ceph-mon[48279]: pgmap v117: 129 pgs: 129 active+clean; 133 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.0 MiB/s wr, 351 op/s 2024-03-14T09:57:59.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:58 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T09:57:59.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:58 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:57:59.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:58 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T09:57:59.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:58 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:57:59.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:58 smithi132 ceph-mon[47959]: pgmap v117: 129 pgs: 129 active+clean; 133 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.0 MiB/s wr, 351 op/s 2024-03-14T09:58:00.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:57:59 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T09:58:00.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:57:59 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T09:58:00.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:57:59 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T09:58:01.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:00 smithi152 ceph-mon[48279]: pgmap v118: 129 pgs: 129 active+clean; 133 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.0 MiB/s wr, 351 op/s 2024-03-14T09:58:01.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:00 smithi132 ceph-mon[47959]: pgmap v118: 129 pgs: 129 active+clean; 133 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.0 MiB/s wr, 351 op/s 2024-03-14T09:58:01.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:00 smithi078 ceph-mon[38143]: pgmap v118: 129 pgs: 129 active+clean; 133 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 2.0 MiB/s wr, 351 op/s 2024-03-14T09:58:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:03 smithi152 ceph-mon[48279]: pgmap v119: 129 pgs: 129 active+clean; 143 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 3.1 MiB/s wr, 532 op/s 2024-03-14T09:58:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:03 smithi132 ceph-mon[47959]: pgmap v119: 129 pgs: 129 active+clean; 143 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 3.1 MiB/s wr, 532 op/s 2024-03-14T09:58:03.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:03 smithi078 ceph-mon[38143]: pgmap v119: 129 pgs: 129 active+clean; 143 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 3.1 MiB/s wr, 532 op/s 2024-03-14T09:58:05.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:05 smithi152 ceph-mon[48279]: pgmap v120: 129 pgs: 129 active+clean; 143 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.0 MiB/s wr, 353 op/s 2024-03-14T09:58:05.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:05 smithi132 ceph-mon[47959]: pgmap v120: 129 pgs: 129 active+clean; 143 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.0 MiB/s wr, 353 op/s 2024-03-14T09:58:05.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:05 smithi078 ceph-mon[38143]: pgmap v120: 129 pgs: 129 active+clean; 143 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.0 MiB/s wr, 353 op/s 2024-03-14T09:58:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:07 smithi152 ceph-mon[48279]: pgmap v121: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.1 MiB/s wr, 521 op/s 2024-03-14T09:58:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:07 smithi132 ceph-mon[47959]: pgmap v121: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.1 MiB/s wr, 521 op/s 2024-03-14T09:58:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:07 smithi078 ceph-mon[38143]: pgmap v121: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.1 MiB/s wr, 521 op/s 2024-03-14T09:58:09.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:08 smithi078 ceph-mon[38143]: pgmap v122: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.1 MiB/s wr, 349 op/s 2024-03-14T09:58:09.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:08 smithi152 ceph-mon[48279]: pgmap v122: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.1 MiB/s wr, 349 op/s 2024-03-14T09:58:09.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:08 smithi132 ceph-mon[47959]: pgmap v122: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.1 MiB/s wr, 349 op/s 2024-03-14T09:58:11.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:11 smithi152 ceph-mon[48279]: pgmap v123: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.1 MiB/s wr, 349 op/s 2024-03-14T09:58:11.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:11 smithi132 ceph-mon[47959]: pgmap v123: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.1 MiB/s wr, 349 op/s 2024-03-14T09:58:11.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:11 smithi078 ceph-mon[38143]: pgmap v123: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.1 MiB/s wr, 349 op/s 2024-03-14T09:58:13.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:13 smithi152 ceph-mon[48279]: pgmap v124: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.1 MiB/s wr, 518 op/s 2024-03-14T09:58:13.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:13 smithi132 ceph-mon[47959]: pgmap v124: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.1 MiB/s wr, 518 op/s 2024-03-14T09:58:13.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:13 smithi078 ceph-mon[38143]: pgmap v124: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.1 MiB/s wr, 518 op/s 2024-03-14T09:58:14.342 INFO:tasks.workunit.client.0.smithi078.stdout:40000 copy from 0x19d87f to 0x1a73ec, (0x9b6d bytes) at 0x9241fd 2024-03-14T09:58:15.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:15 smithi152 ceph-mon[48279]: pgmap v125: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.1 MiB/s wr, 336 op/s 2024-03-14T09:58:15.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:15 smithi132 ceph-mon[47959]: pgmap v125: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.1 MiB/s wr, 336 op/s 2024-03-14T09:58:15.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:15 smithi078 ceph-mon[38143]: pgmap v125: 129 pgs: 129 active+clean; 152 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.1 MiB/s wr, 336 op/s 2024-03-14T09:58:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:17 smithi152 ceph-mon[48279]: pgmap v126: 129 pgs: 129 active+clean; 150 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.0 MiB/s wr, 499 op/s 2024-03-14T09:58:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:17 smithi132 ceph-mon[47959]: pgmap v126: 129 pgs: 129 active+clean; 150 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.0 MiB/s wr, 499 op/s 2024-03-14T09:58:17.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:17 smithi078 ceph-mon[38143]: pgmap v126: 129 pgs: 129 active+clean; 150 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.0 MiB/s wr, 499 op/s 2024-03-14T09:58:19.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:18 smithi078 ceph-mon[38143]: pgmap v127: 129 pgs: 129 active+clean; 150 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 330 op/s 2024-03-14T09:58:19.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:18 smithi152 ceph-mon[48279]: pgmap v127: 129 pgs: 129 active+clean; 150 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 330 op/s 2024-03-14T09:58:19.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:18 smithi132 ceph-mon[47959]: pgmap v127: 129 pgs: 129 active+clean; 150 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 330 op/s 2024-03-14T09:58:20.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:20 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:58:20.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:20 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:58:20.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:20 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:58:21.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:21 smithi152 ceph-mon[48279]: pgmap v128: 129 pgs: 129 active+clean; 150 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 330 op/s 2024-03-14T09:58:21.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:21 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:58:21.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:21 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:58:21.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:21 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:58:21.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:21 smithi132 ceph-mon[47959]: pgmap v128: 129 pgs: 129 active+clean; 150 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 330 op/s 2024-03-14T09:58:21.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:21 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:58:21.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:21 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:58:21.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:21 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:58:21.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:21 smithi078 ceph-mon[38143]: pgmap v128: 129 pgs: 129 active+clean; 150 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 330 op/s 2024-03-14T09:58:21.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:21 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:58:21.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:21 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:58:21.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:21 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:58:23.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:23 smithi152 ceph-mon[48279]: pgmap v129: 129 pgs: 129 active+clean; 154 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.9 MiB/s wr, 482 op/s 2024-03-14T09:58:23.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:23 smithi132 ceph-mon[47959]: pgmap v129: 129 pgs: 129 active+clean; 154 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.9 MiB/s wr, 482 op/s 2024-03-14T09:58:23.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:23 smithi078 ceph-mon[38143]: pgmap v129: 129 pgs: 129 active+clean; 154 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.9 MiB/s wr, 482 op/s 2024-03-14T09:58:25.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:25 smithi152 ceph-mon[48279]: pgmap v130: 129 pgs: 129 active+clean; 154 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.9 MiB/s wr, 314 op/s 2024-03-14T09:58:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:25 smithi132 ceph-mon[47959]: pgmap v130: 129 pgs: 129 active+clean; 154 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.9 MiB/s wr, 314 op/s 2024-03-14T09:58:25.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:25 smithi078 ceph-mon[38143]: pgmap v130: 129 pgs: 129 active+clean; 154 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.9 MiB/s wr, 314 op/s 2024-03-14T09:58:27.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:27 smithi152 ceph-mon[48279]: pgmap v131: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.9 MiB/s wr, 469 op/s 2024-03-14T09:58:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:27 smithi132 ceph-mon[47959]: pgmap v131: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.9 MiB/s wr, 469 op/s 2024-03-14T09:58:27.522 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:58:27.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:27 smithi078 ceph-mon[38143]: pgmap v131: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.9 MiB/s wr, 469 op/s 2024-03-14T09:58:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.764+0000 7f6217a00700 1 -- 172.21.15.78:0/3930103373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6210106050 msgr2=0x7f621010dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.764+0000 7f6217a00700 1 --2- 172.21.15.78:0/3930103373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6210106050 0x7f621010dbd0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f620800b0a0 tx=0x7f620800b3b0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.764+0000 7f6217a00700 1 -- 172.21.15.78:0/3930103373 shutdown_connections 2024-03-14T09:58:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.764+0000 7f6217a00700 1 --2- 172.21.15.78:0/3930103373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6210106050 0x7f621010dbd0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.764+0000 7f6217a00700 1 --2- 172.21.15.78:0/3930103373 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6210100f30 0x7f6210101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.764+0000 7f6217a00700 1 --2- 172.21.15.78:0/3930103373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f62101018e0 0x7f6210105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.764+0000 7f6217a00700 1 -- 172.21.15.78:0/3930103373 >> 172.21.15.78:0/3930103373 conn(0x7f62100fc7b0 msgr2=0x7f62100febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:27.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.764+0000 7f6217a00700 1 -- 172.21.15.78:0/3930103373 shutdown_connections 2024-03-14T09:58:27.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.764+0000 7f6217a00700 1 -- 172.21.15.78:0/3930103373 wait complete. 2024-03-14T09:58:27.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.765+0000 7f6217a00700 1 Processor -- start 2024-03-14T09:58:27.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.766+0000 7f6217a00700 1 -- start start 2024-03-14T09:58:27.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.766+0000 7f6217a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6210100f30 0x7f621019f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.766+0000 7f6217a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6210106050 0x7f621019fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.766+0000 7f6217a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6210199790 0x7f6210199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.766+0000 7f6217a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621019a210 con 0x7f6210100f30 2024-03-14T09:58:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.766+0000 7f6217a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621019a350 con 0x7f6210199790 2024-03-14T09:58:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.766+0000 7f6217a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621019a510 con 0x7f6210106050 2024-03-14T09:58:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.766+0000 7f6215000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6210100f30 0x7f621019f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.766+0000 7f620fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6210106050 0x7f621019fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.766+0000 7f6215000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6210100f30 0x7f621019f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50956/0 (socket says 172.21.15.78:50956) 2024-03-14T09:58:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.766+0000 7f620fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6210106050 0x7f621019fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:58864/0 (socket says 172.21.15.78:58864) 2024-03-14T09:58:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.766+0000 7f6215000700 1 -- 172.21.15.78:0/2847062313 learned_addr learned my addr 172.21.15.78:0/2847062313 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.766+0000 7f6215a00700 1 --2- 172.21.15.78:0/2847062313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6210199790 0x7f6210199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.767+0000 7f620fe00700 1 -- 172.21.15.78:0/2847062313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6210199790 msgr2=0x7f6210199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.767+0000 7f620fe00700 1 --2- 172.21.15.78:0/2847062313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6210199790 0x7f6210199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.767+0000 7f620fe00700 1 -- 172.21.15.78:0/2847062313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6210100f30 msgr2=0x7f621019f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.767+0000 7f620fe00700 1 --2- 172.21.15.78:0/2847062313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6210100f30 0x7f621019f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.767+0000 7f620fe00700 1 -- 172.21.15.78:0/2847062313 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6208009d00 con 0x7f6210106050 2024-03-14T09:58:27.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.767+0000 7f6215a00700 1 --2- 172.21.15.78:0/2847062313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6210199790 0x7f6210199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:58:27.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.767+0000 7f620fe00700 1 --2- 172.21.15.78:0/2847062313 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6210106050 0x7f621019fba0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f61fc00e9c0 tx=0x7f61fc00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:27.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.767+0000 7f620d600700 1 -- 172.21.15.78:0/2847062313 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc00c9c0 con 0x7f6210106050 2024-03-14T09:58:27.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.767+0000 7f620d600700 1 -- 172.21.15.78:0/2847062313 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61fc010430 con 0x7f6210106050 2024-03-14T09:58:27.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.767+0000 7f6217a00700 1 -- 172.21.15.78:0/2847062313 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62101a64d0 con 0x7f6210106050 2024-03-14T09:58:27.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.767+0000 7f620d600700 1 -- 172.21.15.78:0/2847062313 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc018680 con 0x7f6210106050 2024-03-14T09:58:27.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.768+0000 7f6217a00700 1 -- 172.21.15.78:0/2847062313 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62101a6a30 con 0x7f6210106050 2024-03-14T09:58:27.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.771+0000 7f620d600700 1 -- 172.21.15.78:0/2847062313 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f61fc00cb20 con 0x7f6210106050 2024-03-14T09:58:27.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.772+0000 7f620d600700 1 --2- 172.21.15.78:0/2847062313 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f61f80a0c80 0x7f61f80a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.773+0000 7f620d600700 1 -- 172.21.15.78:0/2847062313 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f61fc014070 con 0x7f6210106050 2024-03-14T09:58:27.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.773+0000 7f622b434b80 1 Processor -- start 2024-03-14T09:58:27.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.773+0000 7f6215000700 1 --2- 172.21.15.78:0/2847062313 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f61f80a0c80 0x7f61f80a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.773+0000 7f622b434b80 1 -- start start 2024-03-14T09:58:27.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.773+0000 7f622b434b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfc6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.773+0000 7f622b434b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfc6fa0 0x5578dcfd0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.773+0000 7f622b434b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfd0ec0 0x5578dcfd32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.773+0000 7f622b434b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578dcf1f990 con 0x5578dcfd0ec0 2024-03-14T09:58:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.773+0000 7f622b434b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578dcf1d350 con 0x5578dcfc6fa0 2024-03-14T09:58:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.773+0000 7f622b434b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578dcf18c40 con 0x5578dcfc6680 2024-03-14T09:58:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.774+0000 7f620fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfc6fa0 0x5578dcfd0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.774+0000 7f620fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfc6fa0 0x5578dcfd0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:38622/0 (socket says 172.21.15.78:38622) 2024-03-14T09:58:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.774+0000 7f620fe00700 1 -- 172.21.15.78:0/1717752245 learned_addr learned my addr 172.21.15.78:0/1717752245 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:27.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.774+0000 7f6215a00700 1 --2- 172.21.15.78:0/1717752245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfd0ec0 0x5578dcfd32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.774+0000 7f6215000700 1 --2- 172.21.15.78:0/2847062313 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f61f80a0c80 0x7f61f80a3140 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6204009d20 tx=0x7f6204009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.774+0000 7f620fe00700 1 -- 172.21.15.78:0/1717752245 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 msgr2=0x5578dcfc6a60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-14T09:58:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.774+0000 7f620fe00700 1 --2- 172.21.15.78:0/1717752245 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfc6a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.774+0000 7f620fe00700 1 -- 172.21.15.78:0/1717752245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfd0ec0 msgr2=0x5578dcfd32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.774+0000 7f620fe00700 1 --2- 172.21.15.78:0/1717752245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfd0ec0 0x5578dcfd32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.774+0000 7f620fe00700 1 -- 172.21.15.78:0/1717752245 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578dcd9a3b0 con 0x5578dcfc6fa0 2024-03-14T09:58:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.774+0000 7f6215a00700 1 --2- 172.21.15.78:0/1717752245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfd0ec0 0x5578dcfd32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:58:27.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.775+0000 7f620fe00700 1 --2- 172.21.15.78:0/1717752245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfc6fa0 0x5578dcfd0980 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0a1700 tx=0x7f61fc0993c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10d26fbd0c02568a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.775+0000 7f6202000700 1 -- 172.21.15.78:0/1717752245 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc0997b0 con 0x5578dcfc6fa0 2024-03-14T09:58:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.775+0000 7f6202000700 1 -- 172.21.15.78:0/1717752245 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61fc09d5e0 con 0x5578dcfc6fa0 2024-03-14T09:58:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.775+0000 7f6202000700 1 -- 172.21.15.78:0/1717752245 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc0b90b0 con 0x5578dcfc6fa0 2024-03-14T09:58:27.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.776+0000 7f622b434b80 1 -- 172.21.15.78:0/1717752245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfc6fa0 msgr2=0x5578dcfd0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:27.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.776+0000 7f622b434b80 1 --2- 172.21.15.78:0/1717752245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfc6fa0 0x5578dcfd0980 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0a1700 tx=0x7f61fc0993c0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.776+0000 7f622b434b80 1 -- 172.21.15.78:0/1717752245 shutdown_connections 2024-03-14T09:58:27.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.776+0000 7f622b434b80 1 --2- 172.21.15.78:0/1717752245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfd0ec0 0x5578dcfd32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.776+0000 7f622b434b80 1 --2- 172.21.15.78:0/1717752245 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfc6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.776+0000 7f622b434b80 1 --2- 172.21.15.78:0/1717752245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfc6fa0 0x5578dcfd0980 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.776+0000 7f622b434b80 1 -- 172.21.15.78:0/1717752245 >> 172.21.15.78:0/1717752245 conn(0x5578dcf3c220 msgr2=0x5578dcfc1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:27.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.777+0000 7f622b434b80 1 -- 172.21.15.78:0/1717752245 shutdown_connections 2024-03-14T09:58:27.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.777+0000 7f622b434b80 1 -- 172.21.15.78:0/1717752245 wait complete. 2024-03-14T09:58:27.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.778+0000 7f622b434b80 1 -- start start 2024-03-14T09:58:27.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.778+0000 7f622b434b80 1 Processor -- start 2024-03-14T09:58:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.779+0000 7f622b434b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfcdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.779+0000 7f622b434b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfc6fa0 0x5578dcfce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.779+0000 7f622b434b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfd0ec0 0x5578dcfce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.779+0000 7f622b434b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578dceb4d80 con 0x5578dcfd0ec0 2024-03-14T09:58:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.779+0000 7f622b434b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578dceb4a20 con 0x5578dcfc6fa0 2024-03-14T09:58:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.779+0000 7f622b434b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578dcd9c6a0 con 0x5578dcfc6680 2024-03-14T09:58:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.779+0000 7f622b434b80 10 client.0 _ll_register_callbacks cb 0x5578dcd203a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:58:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.780+0000 7f6215000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfcdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.780+0000 7f6215000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfcdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:58880/0 (socket says 172.21.15.78:58880) 2024-03-14T09:58:27.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.780+0000 7f6215000700 1 -- 172.21.15.78:0/3675627577 learned_addr learned my addr 172.21.15.78:0/3675627577 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.780+0000 7f620fe00700 1 --2- 172.21.15.78:0/3675627577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfc6fa0 0x5578dcfce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.780+0000 7f6215a00700 1 --2- 172.21.15.78:0/3675627577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfd0ec0 0x5578dcfce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.781+0000 7f620fe00700 1 -- 172.21.15.78:0/3675627577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 msgr2=0x5578dcfcdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.781+0000 7f620fe00700 1 --2- 172.21.15.78:0/3675627577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfcdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.781+0000 7f620fe00700 1 -- 172.21.15.78:0/3675627577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfd0ec0 msgr2=0x5578dcfce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.781+0000 7f620fe00700 1 --2- 172.21.15.78:0/3675627577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfd0ec0 0x5578dcfce700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.781+0000 7f620fe00700 1 -- 172.21.15.78:0/3675627577 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f621010e440 con 0x5578dcfc6fa0 2024-03-14T09:58:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.781+0000 7f6215000700 1 --2- 172.21.15.78:0/3675627577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfcdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:58:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.781+0000 7f6215a00700 1 --2- 172.21.15.78:0/3675627577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfd0ec0 0x5578dcfce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:58:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.781+0000 7f620fe00700 1 --2- 172.21.15.78:0/3675627577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfc6fa0 0x5578dcfce1c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0a1700 tx=0x7f61fc099fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fa784629f2bb236 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.781+0000 7f61f6a00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:58:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.781+0000 7f61f6a00700 1 -- 172.21.15.78:0/3675627577 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc0105a0 con 0x5578dcfc6fa0 2024-03-14T09:58:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.781+0000 7f622b434b80 1 -- 172.21.15.78:0/3675627577 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5578dcfb9720 con 0x5578dcfc6fa0 2024-03-14T09:58:27.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.781+0000 7f61f6a00700 1 -- 172.21.15.78:0/3675627577 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f61fc0cc400 con 0x5578dcfc6fa0 2024-03-14T09:58:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.781+0000 7f61f6a00700 1 -- 172.21.15.78:0/3675627577 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61fc0cc5c0 con 0x5578dcfc6fa0 2024-03-14T09:58:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.781+0000 7f622b434b80 10 client.26489 fetch_fsmap learned FSMap version 25 2024-03-14T09:58:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.781+0000 7f622b434b80 1 -- 172.21.15.78:0/3675627577 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f62101a6a30 con 0x5578dcfc6fa0 2024-03-14T09:58:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.782+0000 7f61f6a00700 1 -- 172.21.15.78:0/3675627577 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5578dcfb9720 con 0x5578dcfc6fa0 2024-03-14T09:58:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.782+0000 7f61f6a00700 1 -- 172.21.15.78:0/3675627577 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f61fc07f020 con 0x5578dcfc6fa0 2024-03-14T09:58:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.782+0000 7f622b434b80 10 client.26489 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:58:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.782+0000 7f622b434b80 10 client.26489 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:58:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.782+0000 7f622b434b80 20 client.26489 populate_metadata read hostname 'smithi078' 2024-03-14T09:58:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.782+0000 7f622b434b80 1 --2- 172.21.15.78:0/3675627577 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5578dcff1830 0x5578dcff3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.782+0000 7f622b434b80 4 client.26489 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:58:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.782+0000 7f622b434b80 1 -- 172.21.15.78:0/3675627577 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5578dcf11700 con 0x5578dcff1830 2024-03-14T09:58:27.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.782+0000 7f6215000700 1 --2- 172.21.15.78:0/3675627577 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5578dcff1830 0x5578dcff3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.783+0000 7f6215000700 1 --2- 172.21.15.78:0/3675627577 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5578dcff1830 0x5578dcff3c20 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf4cba668f936831 server_cookie=1a9df02d838f129b in_seq=0 out_seq=0 2024-03-14T09:58:27.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.783+0000 7f61f6a00700 10 client.26489 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:58:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.784+0000 7f61f6a00700 1 -- 172.21.15.78:0/3675627577 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5578dcf11700 con 0x5578dcff1830 2024-03-14T09:58:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.784+0000 7f61f6a00700 10 client.26489 handle_command_reply: tid=0 2024-03-14T09:58:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.784+0000 7f622b434b80 1 client.26489 shutdown 2024-03-14T09:58:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.785+0000 7f622b434b80 1 -- 172.21.15.78:0/3675627577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfc6fa0 msgr2=0x5578dcfce1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.785+0000 7f622b434b80 1 --2- 172.21.15.78:0/3675627577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfc6fa0 0x5578dcfce1c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0a1700 tx=0x7f61fc099fe0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.785+0000 7f622b434b80 1 -- 172.21.15.78:0/3675627577 shutdown_connections 2024-03-14T09:58:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.785+0000 7f622b434b80 1 --2- 172.21.15.78:0/3675627577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfd0ec0 0x5578dcfce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.785+0000 7f622b434b80 1 --2- 172.21.15.78:0/3675627577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfcdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.785+0000 7f622b434b80 1 --2- 172.21.15.78:0/3675627577 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5578dcff1830 0x5578dcff3c20 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.785+0000 7f622b434b80 1 --2- 172.21.15.78:0/3675627577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfc6fa0 0x5578dcfce1c0 secure :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0a1700 tx=0x7f61fc099fe0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.785+0000 7f61f6a00700 0 client.26489 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:58:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.785+0000 7f622b434b80 1 -- 172.21.15.78:0/3675627577 >> 172.21.15.78:0/3675627577 conn(0x5578dcf3c220 msgr2=0x5578dcfc1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.785+0000 7f622b434b80 1 -- 172.21.15.78:0/3675627577 shutdown_connections 2024-03-14T09:58:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.786+0000 7f622b434b80 1 -- 172.21.15.78:0/3675627577 wait complete. 2024-03-14T09:58:27.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.786+0000 7f622b434b80 20 client.26489 trim_cache size 0 max 16384 2024-03-14T09:58:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.812+0000 7f622b434b80 1 Processor -- start 2024-03-14T09:58:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.812+0000 7f622b434b80 1 -- start start 2024-03-14T09:58:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.812+0000 7f622b434b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfc1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.812+0000 7f622b434b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfd0ec0 0x5578dcfce520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.812+0000 7f622b434b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfbd680 0x5578dcfcaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.812+0000 7f622b434b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578dcfc8b70 con 0x5578dcfbd680 2024-03-14T09:58:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.812+0000 7f622b434b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578dcfc2e30 con 0x5578dcfd0ec0 2024-03-14T09:58:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.812+0000 7f622b434b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578dcfc3060 con 0x5578dcfc6680 2024-03-14T09:58:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.813+0000 7f620fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfd0ec0 0x5578dcfce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.813+0000 7f6215000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfc1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.813+0000 7f620fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfd0ec0 0x5578dcfce520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:38650/0 (socket says 172.21.15.78:38650) 2024-03-14T09:58:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.813+0000 7f620fe00700 1 -- 172.21.15.78:0/68678861 learned_addr learned my addr 172.21.15.78:0/68678861 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.813+0000 7f6215000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfc1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:58884/0 (socket says 172.21.15.78:58884) 2024-03-14T09:58:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.813+0000 7f6215a00700 1 --2- 172.21.15.78:0/68678861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfbd680 0x5578dcfcaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.813+0000 7f620fe00700 1 -- 172.21.15.78:0/68678861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 msgr2=0x5578dcfc1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.813+0000 7f620fe00700 1 --2- 172.21.15.78:0/68678861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfc1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.813+0000 7f620fe00700 1 -- 172.21.15.78:0/68678861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfbd680 msgr2=0x5578dcfcaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.813+0000 7f620fe00700 1 --2- 172.21.15.78:0/68678861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfbd680 0x5578dcfcaf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.813+0000 7f620fe00700 1 -- 172.21.15.78:0/68678861 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578dcd46b60 con 0x5578dcfd0ec0 2024-03-14T09:58:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.813+0000 7f6215a00700 1 --2- 172.21.15.78:0/68678861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfbd680 0x5578dcfcaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:58:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.813+0000 7f620fe00700 1 --2- 172.21.15.78:0/68678861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfd0ec0 0x5578dcfce520 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f61fc098ca0 tx=0x7f61fc097d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9e4a3b73be1056a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.814+0000 7f6202a00700 1 -- 172.21.15.78:0/68678861 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc098f60 con 0x5578dcfd0ec0 2024-03-14T09:58:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.814+0000 7f6202a00700 1 -- 172.21.15.78:0/68678861 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61fc0a1e80 con 0x5578dcfd0ec0 2024-03-14T09:58:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.814+0000 7f6202a00700 1 -- 172.21.15.78:0/68678861 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc0a17a0 con 0x5578dcfd0ec0 2024-03-14T09:58:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.814+0000 7f622b434b80 1 -- 172.21.15.78:0/68678861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfd0ec0 msgr2=0x5578dcfce520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.814+0000 7f622b434b80 1 --2- 172.21.15.78:0/68678861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfd0ec0 0x5578dcfce520 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f61fc098ca0 tx=0x7f61fc097d60 comp rx=0 tx=0).stop 2024-03-14T09:58:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.815+0000 7f622b434b80 1 -- 172.21.15.78:0/68678861 shutdown_connections 2024-03-14T09:58:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.815+0000 7f622b434b80 1 --2- 172.21.15.78:0/68678861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfbd680 0x5578dcfcaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.815+0000 7f622b434b80 1 --2- 172.21.15.78:0/68678861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfc1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.815+0000 7f622b434b80 1 --2- 172.21.15.78:0/68678861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfd0ec0 0x5578dcfce520 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.815+0000 7f622b434b80 1 -- 172.21.15.78:0/68678861 >> 172.21.15.78:0/68678861 conn(0x5578dcf3c220 msgr2=0x5578dcfcc570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.815+0000 7f622b434b80 1 -- 172.21.15.78:0/68678861 shutdown_connections 2024-03-14T09:58:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.815+0000 7f622b434b80 1 -- 172.21.15.78:0/68678861 wait complete. 2024-03-14T09:58:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.815+0000 7f622b434b80 1 -- start start 2024-03-14T09:58:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.815+0000 7f622b434b80 1 Processor -- start 2024-03-14T09:58:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.816+0000 7f622b434b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfbd680 0x5578dcfd5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.816+0000 7f622b434b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfd62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.816+0000 7f622b434b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfd0ec0 0x5578dd042320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.816+0000 7f622b434b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578dd042860 con 0x5578dcfbd680 2024-03-14T09:58:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.816+0000 7f622b434b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578dd0429a0 con 0x5578dcfd0ec0 2024-03-14T09:58:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.816+0000 7f622b434b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578dd042b60 con 0x5578dcfc6680 2024-03-14T09:58:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.816+0000 7f622b434b80 10 client.0 _ll_register_callbacks cb 0x7f61fc005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:58:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.816+0000 7f6215000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfbd680 0x5578dcfd5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.816+0000 7f6215000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfbd680 0x5578dcfd5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:50986/0 (socket says 172.21.15.78:50986) 2024-03-14T09:58:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.816+0000 7f6215000700 1 -- 172.21.15.78:0/2824839488 learned_addr learned my addr 172.21.15.78:0/2824839488 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.816+0000 7f6215a00700 1 --2- 172.21.15.78:0/2824839488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfd0ec0 0x5578dd042320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.816+0000 7f620fe00700 1 --2- 172.21.15.78:0/2824839488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfd62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.816+0000 7f6215000700 1 -- 172.21.15.78:0/2824839488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 msgr2=0x5578dcfd62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.816+0000 7f6215000700 1 --2- 172.21.15.78:0/2824839488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfd62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.817+0000 7f6215000700 1 -- 172.21.15.78:0/2824839488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfd0ec0 msgr2=0x5578dd042320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.817+0000 7f6215000700 1 --2- 172.21.15.78:0/2824839488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfd0ec0 0x5578dd042320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.817+0000 7f6215000700 1 -- 172.21.15.78:0/2824839488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62101a6a30 con 0x5578dcfbd680 2024-03-14T09:58:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.817+0000 7f6215000700 1 --2- 172.21.15.78:0/2824839488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfbd680 0x5578dcfd5d60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x5578dcff44d0 tx=0x7f6204013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dcd5c205382fbc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.817+0000 7f61f6a00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:58:27.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.817+0000 7f61f6a00700 1 -- 172.21.15.78:0/2824839488 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6204022040 con 0x5578dcfbd680 2024-03-14T09:58:27.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.817+0000 7f61f6a00700 1 -- 172.21.15.78:0/2824839488 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f6204028a00 con 0x5578dcfbd680 2024-03-14T09:58:27.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.817+0000 7f622b434b80 1 -- 172.21.15.78:0/2824839488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5578dcd67cf0 con 0x5578dcfbd680 2024-03-14T09:58:27.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.817+0000 7f61f6a00700 1 -- 172.21.15.78:0/2824839488 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f620400e760 con 0x5578dcfbd680 2024-03-14T09:58:27.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.817+0000 7f622b434b80 10 client.16443 fetch_fsmap learned FSMap version 25 2024-03-14T09:58:27.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.817+0000 7f622b434b80 1 -- 172.21.15.78:0/2824839488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5578dcedeea0 con 0x5578dcfbd680 2024-03-14T09:58:27.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.818+0000 7f61f6a00700 1 -- 172.21.15.78:0/2824839488 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5578dcd67cf0 con 0x5578dcfbd680 2024-03-14T09:58:27.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.818+0000 7f61f6a00700 1 -- 172.21.15.78:0/2824839488 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f620401dce0 con 0x5578dcfbd680 2024-03-14T09:58:27.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.818+0000 7f622b434b80 10 client.16443 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:58:27.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.818+0000 7f622b434b80 10 client.16443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:58:27.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.818+0000 7f622b434b80 20 client.16443 populate_metadata read hostname 'smithi078' 2024-03-14T09:58:27.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.818+0000 7f622b434b80 1 --2- 172.21.15.78:0/2824839488 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5578dd044390 0x5578dd0467b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:27.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.818+0000 7f622b434b80 4 client.16443 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:58:27.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.818+0000 7f622b434b80 1 -- 172.21.15.78:0/2824839488 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5578dcd9cce0 con 0x5578dd044390 2024-03-14T09:58:27.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.818+0000 7f620fe00700 1 --2- 172.21.15.78:0/2824839488 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5578dd044390 0x5578dd0467b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:27.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.819+0000 7f620fe00700 1 --2- 172.21.15.78:0/2824839488 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5578dd044390 0x5578dd0467b0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c87cb1a5c2ff623 server_cookie=323e24ddd2d4f902 in_seq=0 out_seq=0 2024-03-14T09:58:27.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.819+0000 7f61f6a00700 10 client.16443 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:58:27.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.820+0000 7f61f6a00700 1 -- 172.21.15.78:0/2824839488 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5578dcd9cce0 con 0x5578dd044390 2024-03-14T09:58:27.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.820+0000 7f61f6a00700 10 client.16443 handle_command_reply: tid=0 2024-03-14T09:58:27.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.820+0000 7f622b434b80 1 client.16443 shutdown 2024-03-14T09:58:27.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.820+0000 7f622b434b80 1 -- 172.21.15.78:0/2824839488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfbd680 msgr2=0x5578dcfd5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:27.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.820+0000 7f622b434b80 1 --2- 172.21.15.78:0/2824839488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfbd680 0x5578dcfd5d60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x5578dcff44d0 tx=0x7f6204013040 comp rx=0 tx=0).stop 2024-03-14T09:58:27.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.820+0000 7f622b434b80 1 -- 172.21.15.78:0/2824839488 shutdown_connections 2024-03-14T09:58:27.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.820+0000 7f622b434b80 1 --2- 172.21.15.78:0/2824839488 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5578dd044390 0x5578dd0467b0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.820+0000 7f622b434b80 1 --2- 172.21.15.78:0/2824839488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5578dcfd0ec0 0x5578dd042320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.820+0000 7f622b434b80 1 --2- 172.21.15.78:0/2824839488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5578dcfbd680 0x5578dcfd5d60 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.820+0000 7f61f6a00700 0 client.16443 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:58:27.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.820+0000 7f622b434b80 1 --2- 172.21.15.78:0/2824839488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5578dcfc6680 0x5578dcfd62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.820+0000 7f622b434b80 1 -- 172.21.15.78:0/2824839488 >> 172.21.15.78:0/2824839488 conn(0x5578dcf3c220 msgr2=0x5578dcfc5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:27.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.820+0000 7f622b434b80 1 -- 172.21.15.78:0/2824839488 shutdown_connections 2024-03-14T09:58:27.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.820+0000 7f622b434b80 1 -- 172.21.15.78:0/2824839488 wait complete. 2024-03-14T09:58:27.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.821+0000 7f622b434b80 20 client.16443 trim_cache size 0 max 16384 2024-03-14T09:58:27.824 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:58:27.824 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T09:58:27.824 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T09:58:27.824 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:58:27.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.821+0000 7f6217a00700 1 -- 172.21.15.78:0/2847062313 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f61f80a0c80 msgr2=0x7f61f80a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:27.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.821+0000 7f6217a00700 1 --2- 172.21.15.78:0/2847062313 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f61f80a0c80 0x7f61f80a3140 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6204009d20 tx=0x7f6204009260 comp rx=0 tx=0).stop 2024-03-14T09:58:27.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.821+0000 7f6217a00700 1 -- 172.21.15.78:0/2847062313 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6210106050 msgr2=0x7f621019fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:27.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.821+0000 7f6217a00700 1 --2- 172.21.15.78:0/2847062313 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6210106050 0x7f621019fba0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f61fc00e9c0 tx=0x7f61fc00ecd0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.821+0000 7f6217a00700 1 -- 172.21.15.78:0/2847062313 shutdown_connections 2024-03-14T09:58:27.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.821+0000 7f6217a00700 1 --2- 172.21.15.78:0/2847062313 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f61f80a0c80 0x7f61f80a3140 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.821+0000 7f6217a00700 1 --2- 172.21.15.78:0/2847062313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6210199790 0x7f6210199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.821+0000 7f6217a00700 1 --2- 172.21.15.78:0/2847062313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6210100f30 0x7f621019f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.821+0000 7f6217a00700 1 --2- 172.21.15.78:0/2847062313 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6210106050 0x7f621019fba0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:27.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.821+0000 7f6217a00700 1 -- 172.21.15.78:0/2847062313 >> 172.21.15.78:0/2847062313 conn(0x7f62100fc7b0 msgr2=0x7f62100feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:27.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.821+0000 7f6217a00700 1 -- 172.21.15.78:0/2847062313 shutdown_connections 2024-03-14T09:58:27.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:27.821+0000 7f6217a00700 1 -- 172.21.15.78:0/2847062313 wait complete. 2024-03-14T09:58:27.842 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T09:58:27.843 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:58:28.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.118+0000 7f37b5c00700 1 -- 172.21.15.78:0/1407533844 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37b01026a0 msgr2=0x7f37b010ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:28.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.118+0000 7f37b5c00700 1 --2- 172.21.15.78:0/1407533844 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37b01026a0 0x7f37b010ab80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f379c009990 tx=0x7f379c009ca0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.118+0000 7f37b5c00700 1 -- 172.21.15.78:0/1407533844 shutdown_connections 2024-03-14T09:58:28.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.118+0000 7f37b5c00700 1 --2- 172.21.15.78:0/1407533844 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37b0101d80 0x7f37b0102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.118+0000 7f37b5c00700 1 --2- 172.21.15.78:0/1407533844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37b010b1a0 0x7f37b010d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.118+0000 7f37b5c00700 1 --2- 172.21.15.78:0/1407533844 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37b01026a0 0x7f37b010ab80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.118+0000 7f37b5c00700 1 -- 172.21.15.78:0/1407533844 >> 172.21.15.78:0/1407533844 conn(0x7f37b00fb5c0 msgr2=0x7f37b00fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:28.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.119+0000 7f37b5c00700 1 -- 172.21.15.78:0/1407533844 shutdown_connections 2024-03-14T09:58:28.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.120+0000 7f37b5c00700 1 -- 172.21.15.78:0/1407533844 wait complete. 2024-03-14T09:58:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.121+0000 7f37b5c00700 1 Processor -- start 2024-03-14T09:58:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.121+0000 7f37b5c00700 1 -- start start 2024-03-14T09:58:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.122+0000 7f37b5c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37b0101d80 0x7f37b0072850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.122+0000 7f37b5c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37b01026a0 0x7f37b00ff9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.122+0000 7f37b5c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37b010b1a0 0x7f37b00fff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.122+0000 7f37b5c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b0100450 con 0x7f37b01026a0 2024-03-14T09:58:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.122+0000 7f37b5c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b0100590 con 0x7f37b010b1a0 2024-03-14T09:58:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.122+0000 7f37b5c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b0100750 con 0x7f37b0101d80 2024-03-14T09:58:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.122+0000 7f37aea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37b0101d80 0x7f37b0072850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.122+0000 7f37aea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37b0101d80 0x7f37b0072850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:58900/0 (socket says 172.21.15.78:58900) 2024-03-14T09:58:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.122+0000 7f37aea00700 1 -- 172.21.15.78:0/2219874127 learned_addr learned my addr 172.21.15.78:0/2219874127 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.122+0000 7f37ae000700 1 --2- 172.21.15.78:0/2219874127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37b01026a0 0x7f37b00ff9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.123+0000 7f37af400700 1 --2- 172.21.15.78:0/2219874127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37b010b1a0 0x7f37b00fff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.123+0000 7f37aea00700 1 -- 172.21.15.78:0/2219874127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37b010b1a0 msgr2=0x7f37b00fff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.123+0000 7f37aea00700 1 --2- 172.21.15.78:0/2219874127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37b010b1a0 0x7f37b00fff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.123+0000 7f37aea00700 1 -- 172.21.15.78:0/2219874127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37b01026a0 msgr2=0x7f37b00ff9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.123+0000 7f37aea00700 1 --2- 172.21.15.78:0/2219874127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37b01026a0 0x7f37b00ff9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.123+0000 7f37aea00700 1 -- 172.21.15.78:0/2219874127 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379c0095f0 con 0x7f37b0101d80 2024-03-14T09:58:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.123+0000 7f37aea00700 1 --2- 172.21.15.78:0/2219874127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37b0101d80 0x7f37b0072850 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f37a000e9a0 tx=0x7f37a000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.123+0000 7f37a7400700 1 -- 172.21.15.78:0/2219874127 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a0009790 con 0x7f37b0101d80 2024-03-14T09:58:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.123+0000 7f37a7400700 1 -- 172.21.15.78:0/2219874127 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37a000cbd0 con 0x7f37b0101d80 2024-03-14T09:58:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.123+0000 7f37a7400700 1 -- 172.21.15.78:0/2219874127 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a0018680 con 0x7f37b0101d80 2024-03-14T09:58:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.124+0000 7f37b5c00700 1 -- 172.21.15.78:0/2219874127 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b01ae630 con 0x7f37b0101d80 2024-03-14T09:58:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.124+0000 7f37b5c00700 1 -- 172.21.15.78:0/2219874127 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b01aeaf0 con 0x7f37b0101d80 2024-03-14T09:58:28.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.126+0000 7f37b5c00700 1 -- 172.21.15.78:0/2219874127 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b0106270 con 0x7f37b0101d80 2024-03-14T09:58:28.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.127+0000 7f37a7400700 1 -- 172.21.15.78:0/2219874127 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f37a000cd40 con 0x7f37b0101d80 2024-03-14T09:58:28.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.129+0000 7f37a7400700 1 --2- 172.21.15.78:0/2219874127 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f37940a0c00 0x7f37940a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.129+0000 7f37a7400700 1 -- 172.21.15.78:0/2219874127 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f37a0014070 con 0x7f37b0101d80 2024-03-14T09:58:28.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.129+0000 7f37ae000700 1 --2- 172.21.15.78:0/2219874127 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f37940a0c00 0x7f37940a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.130+0000 7f37ae000700 1 --2- 172.21.15.78:0/2219874127 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f37940a0c00 0x7f37940a30c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f379c006010 tx=0x7f379c00b260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:28.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.133+0000 7f37a7400700 1 -- 172.21.15.78:0/2219874127 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f37a0052960 con 0x7f37b0101d80 2024-03-14T09:58:28.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.447+0000 7f37b5c00700 1 -- 172.21.15.78:0/2219874127 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37b0100eb0 con 0x7f37b0101d80 2024-03-14T09:58:28.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.448+0000 7f37a7400700 1 -- 172.21.15.78:0/2219874127 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f37a0095e20 con 0x7f37b0101d80 2024-03-14T09:58:28.449 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:58:28.449 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:58:28.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.451+0000 7f37b5c00700 1 -- 172.21.15.78:0/2219874127 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f37940a0c00 msgr2=0x7f37940a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:28.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.451+0000 7f37b5c00700 1 --2- 172.21.15.78:0/2219874127 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f37940a0c00 0x7f37940a30c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f379c006010 tx=0x7f379c00b260 comp rx=0 tx=0).stop 2024-03-14T09:58:28.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.451+0000 7f37b5c00700 1 -- 172.21.15.78:0/2219874127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37b0101d80 msgr2=0x7f37b0072850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:28.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.451+0000 7f37b5c00700 1 --2- 172.21.15.78:0/2219874127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37b0101d80 0x7f37b0072850 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f37a000e9a0 tx=0x7f37a000ed60 comp rx=0 tx=0).stop 2024-03-14T09:58:28.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.451+0000 7f37b5c00700 1 -- 172.21.15.78:0/2219874127 shutdown_connections 2024-03-14T09:58:28.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.451+0000 7f37b5c00700 1 --2- 172.21.15.78:0/2219874127 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f37940a0c00 0x7f37940a30c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.451+0000 7f37b5c00700 1 --2- 172.21.15.78:0/2219874127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37b010b1a0 0x7f37b00fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.451+0000 7f37b5c00700 1 --2- 172.21.15.78:0/2219874127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f37b0101d80 0x7f37b0072850 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.451+0000 7f37b5c00700 1 --2- 172.21.15.78:0/2219874127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37b01026a0 0x7f37b00ff9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.451+0000 7f37b5c00700 1 -- 172.21.15.78:0/2219874127 >> 172.21.15.78:0/2219874127 conn(0x7f37b00fb5c0 msgr2=0x7f37b00fcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:28.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.451+0000 7f37b5c00700 1 -- 172.21.15.78:0/2219874127 shutdown_connections 2024-03-14T09:58:28.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.451+0000 7f37b5c00700 1 -- 172.21.15.78:0/2219874127 wait complete. 2024-03-14T09:58:28.453 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T09:58:28.473 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T09:58:28.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.716+0000 7f377dc00700 1 -- 172.21.15.78:0/3472855141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3778106050 msgr2=0x7f377810dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:28.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.716+0000 7f377dc00700 1 --2- 172.21.15.78:0/3472855141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3778106050 0x7f377810dbd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f377000b0a0 tx=0x7f377000b3b0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.716+0000 7f377dc00700 1 -- 172.21.15.78:0/3472855141 shutdown_connections 2024-03-14T09:58:28.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.716+0000 7f377dc00700 1 --2- 172.21.15.78:0/3472855141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3778106050 0x7f377810dbd0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.716+0000 7f377dc00700 1 --2- 172.21.15.78:0/3472855141 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3778100f30 0x7f3778101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.716+0000 7f377dc00700 1 --2- 172.21.15.78:0/3472855141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37781018e0 0x7f3778105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.716+0000 7f377dc00700 1 -- 172.21.15.78:0/3472855141 >> 172.21.15.78:0/3472855141 conn(0x7f37780fc7b0 msgr2=0x7f37780febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:28.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.716+0000 7f377dc00700 1 -- 172.21.15.78:0/3472855141 shutdown_connections 2024-03-14T09:58:28.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.716+0000 7f377dc00700 1 -- 172.21.15.78:0/3472855141 wait complete. 2024-03-14T09:58:28.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.717+0000 7f377dc00700 1 Processor -- start 2024-03-14T09:58:28.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.717+0000 7f377dc00700 1 -- start start 2024-03-14T09:58:28.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.718+0000 7f377dc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3778100f30 0x7f37780730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.718+0000 7f377dc00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3778106050 0x7f37780735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.718+0000 7f377dc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3778073b30 0x7f37781a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.718+0000 7f377dc00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37781a5cd0 con 0x7f3778073b30 2024-03-14T09:58:28.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.718+0000 7f377dc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37781a5e40 con 0x7f3778100f30 2024-03-14T09:58:28.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.718+0000 7f377dc00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37781a6000 con 0x7f3778106050 2024-03-14T09:58:28.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.718+0000 7f3776000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3778106050 0x7f37780735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.718+0000 7f3776000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3778106050 0x7f37780735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:58920/0 (socket says 172.21.15.78:58920) 2024-03-14T09:58:28.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.718+0000 7f3776000700 1 -- 172.21.15.78:0/993737949 learned_addr learned my addr 172.21.15.78:0/993737949 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:28.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.718+0000 7f3776a00700 1 --2- 172.21.15.78:0/993737949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3778100f30 0x7f37780730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.718+0000 7f3777400700 1 --2- 172.21.15.78:0/993737949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3778073b30 0x7f37781a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.718+0000 7f3776000700 1 -- 172.21.15.78:0/993737949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3778100f30 msgr2=0x7f37780730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:28.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.718+0000 7f3776000700 1 --2- 172.21.15.78:0/993737949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3778100f30 0x7f37780730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.718+0000 7f3776000700 1 -- 172.21.15.78:0/993737949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3778073b30 msgr2=0x7f37781a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.718+0000 7f3776000700 1 --2- 172.21.15.78:0/993737949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3778073b30 0x7f37781a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.718+0000 7f3776000700 1 -- 172.21.15.78:0/993737949 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3770009d00 con 0x7f3778106050 2024-03-14T09:58:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.719+0000 7f3776000700 1 --2- 172.21.15.78:0/993737949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3778106050 0x7f37780735f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f376400e970 tx=0x7f376400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.719+0000 7f376f400700 1 -- 172.21.15.78:0/993737949 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3764009790 con 0x7f3778106050 2024-03-14T09:58:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.719+0000 7f377dc00700 1 -- 172.21.15.78:0/993737949 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37781a62e0 con 0x7f3778106050 2024-03-14T09:58:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.719+0000 7f376f400700 1 -- 172.21.15.78:0/993737949 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f376400c7e0 con 0x7f3778106050 2024-03-14T09:58:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.719+0000 7f376f400700 1 -- 172.21.15.78:0/993737949 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37640054e0 con 0x7f3778106050 2024-03-14T09:58:28.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.719+0000 7f377dc00700 1 -- 172.21.15.78:0/993737949 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37781a6840 con 0x7f3778106050 2024-03-14T09:58:28.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.723+0000 7f376f400700 1 -- 172.21.15.78:0/993737949 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f376400c950 con 0x7f3778106050 2024-03-14T09:58:28.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.724+0000 7f379178bb80 1 Processor -- start 2024-03-14T09:58:28.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.725+0000 7f379178bb80 1 -- start start 2024-03-14T09:58:28.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.725+0000 7f379178bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2680 0x55d9374d2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.725+0000 7f379178bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2fa0 0x55d9374e5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.725+0000 7f379178bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 0x55d9374e7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.725+0000 7f379178bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93742b990 con 0x55d9374d2680 2024-03-14T09:58:28.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.725+0000 7f379178bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d937429350 con 0x55d9374d2fa0 2024-03-14T09:58:28.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.725+0000 7f379178bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d937424c40 con 0x55d9374e56d0 2024-03-14T09:58:28.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.726+0000 7f3776000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 0x55d9374e7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.726+0000 7f376f400700 1 --2- 172.21.15.78:0/993737949 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f375c0a0c00 0x7f375c0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.726+0000 7f3776a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2fa0 0x55d9374e5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.726+0000 7f3776000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 0x55d9374e7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:58934/0 (socket says 172.21.15.78:58934) 2024-03-14T09:58:28.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.726+0000 7f3776000700 1 -- 172.21.15.78:0/3302378398 learned_addr learned my addr 172.21.15.78:0/3302378398 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:28.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.726+0000 7f3777400700 1 --2- 172.21.15.78:0/3302378398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2680 0x55d9374d2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.726+0000 7f3776a00700 1 -- 172.21.15.78:0/3302378398 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 msgr2=0x55d9374e7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:28.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.726+0000 7f3776a00700 1 --2- 172.21.15.78:0/3302378398 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 0x55d9374e7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.726+0000 7f3776a00700 1 -- 172.21.15.78:0/3302378398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2680 msgr2=0x55d9374d2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:28.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.726+0000 7f376f400700 1 -- 172.21.15.78:0/993737949 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f3764014070 con 0x7f3778106050 2024-03-14T09:58:28.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.726+0000 7f3776a00700 1 --2- 172.21.15.78:0/3302378398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2680 0x55d9374d2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.726+0000 7f3776a00700 1 -- 172.21.15.78:0/3302378398 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9372a63b0 con 0x55d9374d2fa0 2024-03-14T09:58:28.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.726+0000 7f3777400700 1 --2- 172.21.15.78:0/3302378398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2680 0x55d9374d2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:58:28.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.726+0000 7f3776000700 1 --2- 172.21.15.78:0/3302378398 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 0x55d9374e7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:58:28.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.726+0000 7f3777400700 1 --2- 172.21.15.78:0/993737949 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f375c0a0c00 0x7f375c0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.727+0000 7f3776a00700 1 --2- 172.21.15.78:0/3302378398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2fa0 0x55d9374e5190 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3768009cb0 tx=0x7f3768011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94cfa915478f44e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:28.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.727+0000 7f3763e00700 1 -- 172.21.15.78:0/3302378398 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3768011d20 con 0x55d9374d2fa0 2024-03-14T09:58:28.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.727+0000 7f3763e00700 1 -- 172.21.15.78:0/3302378398 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3768011e80 con 0x55d9374d2fa0 2024-03-14T09:58:28.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.727+0000 7f3763e00700 1 -- 172.21.15.78:0/3302378398 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3768005280 con 0x55d9374d2fa0 2024-03-14T09:58:28.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.728+0000 7f379178bb80 1 -- 172.21.15.78:0/3302378398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2fa0 msgr2=0x55d9374e5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:28.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.728+0000 7f379178bb80 1 --2- 172.21.15.78:0/3302378398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2fa0 0x55d9374e5190 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3768009cb0 tx=0x7f3768011440 comp rx=0 tx=0).stop 2024-03-14T09:58:28.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.728+0000 7f3777400700 1 --2- 172.21.15.78:0/993737949 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f375c0a0c00 0x7f375c0a30c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f377000b780 tx=0x7f37700099e0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:28.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.728+0000 7f379178bb80 1 -- 172.21.15.78:0/3302378398 shutdown_connections 2024-03-14T09:58:28.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.728+0000 7f379178bb80 1 --2- 172.21.15.78:0/3302378398 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 0x55d9374e7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.728+0000 7f379178bb80 1 --2- 172.21.15.78:0/3302378398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2680 0x55d9374d2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.728+0000 7f379178bb80 1 --2- 172.21.15.78:0/3302378398 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2fa0 0x55d9374e5190 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.728+0000 7f379178bb80 1 -- 172.21.15.78:0/3302378398 >> 172.21.15.78:0/3302378398 conn(0x55d937448220 msgr2=0x55d9374cdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:28.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.728+0000 7f379178bb80 1 -- 172.21.15.78:0/3302378398 shutdown_connections 2024-03-14T09:58:28.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.728+0000 7f379178bb80 1 -- 172.21.15.78:0/3302378398 wait complete. 2024-03-14T09:58:28.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.729+0000 7f379178bb80 1 -- start start 2024-03-14T09:58:28.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.729+0000 7f379178bb80 1 Processor -- start 2024-03-14T09:58:28.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f379178bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374d2680 0x55d9374debe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f379178bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2fa0 0x55d9374df120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f379178bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374e56d0 0x55d9374e4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f379178bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9373c0d80 con 0x55d9374d2fa0 2024-03-14T09:58:28.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f379178bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9373c0a20 con 0x55d9374e56d0 2024-03-14T09:58:28.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f379178bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9372a86a0 con 0x55d9374d2680 2024-03-14T09:58:28.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f379178bb80 10 client.0 _ll_register_callbacks cb 0x55d93722c3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:58:28.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f3776a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374d2680 0x55d9374debe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f3777400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374e56d0 0x55d9374e4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f3776000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2fa0 0x55d9374df120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f3776a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374d2680 0x55d9374debe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:58946/0 (socket says 172.21.15.78:58946) 2024-03-14T09:58:28.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f3776a00700 1 -- 172.21.15.78:0/4025789779 learned_addr learned my addr 172.21.15.78:0/4025789779 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:28.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f3777400700 1 -- 172.21.15.78:0/4025789779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374d2680 msgr2=0x55d9374debe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:28.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f3777400700 1 --2- 172.21.15.78:0/4025789779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374d2680 0x55d9374debe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f3777400700 1 -- 172.21.15.78:0/4025789779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2fa0 msgr2=0x55d9374df120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:28.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f3777400700 1 --2- 172.21.15.78:0/4025789779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2fa0 0x55d9374df120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f3777400700 1 -- 172.21.15.78:0/4025789779 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37781a6840 con 0x55d9374e56d0 2024-03-14T09:58:28.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.730+0000 7f3776000700 1 --2- 172.21.15.78:0/4025789779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2fa0 0x55d9374df120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:58:28.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.731+0000 7f3777400700 1 --2- 172.21.15.78:0/4025789779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374e56d0 0x55d9374e4900 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3770009fd0 tx=0x7f3770006230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65ab617b166dcf16 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:28.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.731+0000 7f3760c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:58:28.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.731+0000 7f3760c00700 1 -- 172.21.15.78:0/4025789779 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377000e040 con 0x55d9374e56d0 2024-03-14T09:58:28.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.731+0000 7f379178bb80 1 -- 172.21.15.78:0/4025789779 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d93741d700 con 0x55d9374e56d0 2024-03-14T09:58:28.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.731+0000 7f3760c00700 1 -- 172.21.15.78:0/4025789779 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f3770028440 con 0x55d9374e56d0 2024-03-14T09:58:28.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.731+0000 7f3760c00700 1 -- 172.21.15.78:0/4025789779 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3770028600 con 0x55d9374e56d0 2024-03-14T09:58:28.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.731+0000 7f379178bb80 10 client.26513 fetch_fsmap learned FSMap version 25 2024-03-14T09:58:28.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.731+0000 7f379178bb80 1 -- 172.21.15.78:0/4025789779 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f37680098e0 con 0x55d9374e56d0 2024-03-14T09:58:28.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.732+0000 7f3760c00700 1 -- 172.21.15.78:0/4025789779 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d93741d700 con 0x55d9374e56d0 2024-03-14T09:58:28.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.732+0000 7f3760c00700 1 -- 172.21.15.78:0/4025789779 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3770025050 con 0x55d9374e56d0 2024-03-14T09:58:28.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.732+0000 7f379178bb80 10 client.26513 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:58:28.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.732+0000 7f379178bb80 10 client.26513 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:58:28.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.732+0000 7f379178bb80 20 client.26513 populate_metadata read hostname 'smithi078' 2024-03-14T09:58:28.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.732+0000 7f379178bb80 1 --2- 172.21.15.78:0/4025789779 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d9374fd790 0x55d9374ffb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.732+0000 7f379178bb80 4 client.26513 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:58:28.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.732+0000 7f379178bb80 1 -- 172.21.15.78:0/4025789779 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d937273cf0 con 0x55d9374fd790 2024-03-14T09:58:28.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.732+0000 7f3776a00700 1 --2- 172.21.15.78:0/4025789779 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d9374fd790 0x55d9374ffb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.733+0000 7f3776a00700 1 --2- 172.21.15.78:0/4025789779 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d9374fd790 0x55d9374ffb80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b66ca98a5644ffe server_cookie=c08b713b0343420c in_seq=0 out_seq=0 2024-03-14T09:58:28.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.733+0000 7f3760c00700 10 client.26513 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:58:28.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.734+0000 7f3760c00700 1 -- 172.21.15.78:0/4025789779 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55d937273cf0 con 0x55d9374fd790 2024-03-14T09:58:28.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.734+0000 7f3760c00700 10 client.26513 handle_command_reply: tid=0 2024-03-14T09:58:28.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.734+0000 7f379178bb80 1 client.26513 shutdown 2024-03-14T09:58:28.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.734+0000 7f379178bb80 1 -- 172.21.15.78:0/4025789779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374e56d0 msgr2=0x55d9374e4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:28.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.734+0000 7f379178bb80 1 --2- 172.21.15.78:0/4025789779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374e56d0 0x55d9374e4900 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3770009fd0 tx=0x7f3770006230 comp rx=0 tx=0).stop 2024-03-14T09:58:28.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.734+0000 7f379178bb80 1 -- 172.21.15.78:0/4025789779 shutdown_connections 2024-03-14T09:58:28.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.734+0000 7f379178bb80 1 --2- 172.21.15.78:0/4025789779 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d9374fd790 0x55d9374ffb80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.734+0000 7f379178bb80 1 --2- 172.21.15.78:0/4025789779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374e56d0 0x55d9374e4900 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.734+0000 7f379178bb80 1 --2- 172.21.15.78:0/4025789779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374d2680 0x55d9374debe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.734+0000 7f379178bb80 1 --2- 172.21.15.78:0/4025789779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2fa0 0x55d9374df120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.734+0000 7f3760c00700 0 client.26513 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:58:28.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.734+0000 7f379178bb80 1 -- 172.21.15.78:0/4025789779 >> 172.21.15.78:0/4025789779 conn(0x55d937448220 msgr2=0x55d9374cd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:28.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.734+0000 7f379178bb80 1 -- 172.21.15.78:0/4025789779 shutdown_connections 2024-03-14T09:58:28.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.734+0000 7f379178bb80 1 -- 172.21.15.78:0/4025789779 wait complete. 2024-03-14T09:58:28.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.734+0000 7f379178bb80 20 client.26513 trim_cache size 0 max 16384 2024-03-14T09:58:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.753+0000 7f379178bb80 1 Processor -- start 2024-03-14T09:58:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.753+0000 7f379178bb80 1 -- start start 2024-03-14T09:58:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.753+0000 7f379178bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2680 0x55d9375044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.753+0000 7f379178bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2fa0 0x55d9374d8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.753+0000 7f379178bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 0x55d9374d8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.753+0000 7f379178bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9374db540 con 0x55d9374d2fa0 2024-03-14T09:58:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.753+0000 7f379178bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9374ce830 con 0x55d9374d2680 2024-03-14T09:58:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.753+0000 7f379178bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9374d3af0 con 0x55d9374e56d0 2024-03-14T09:58:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.754+0000 7f3776a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2680 0x55d9375044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.754+0000 7f3776000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2fa0 0x55d9374d8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.754+0000 7f3776a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2680 0x55d9375044e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:38736/0 (socket says 172.21.15.78:38736) 2024-03-14T09:58:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.754+0000 7f3776a00700 1 -- 172.21.15.78:0/1683521199 learned_addr learned my addr 172.21.15.78:0/1683521199 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.754+0000 7f3777400700 1 --2- 172.21.15.78:0/1683521199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 0x55d9374d8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.754+0000 7f3776a00700 1 -- 172.21.15.78:0/1683521199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 msgr2=0x55d9374d8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.754+0000 7f3776a00700 1 --2- 172.21.15.78:0/1683521199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 0x55d9374d8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.754+0000 7f3776a00700 1 -- 172.21.15.78:0/1683521199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2fa0 msgr2=0x55d9374d8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.754+0000 7f3776a00700 1 --2- 172.21.15.78:0/1683521199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2fa0 0x55d9374d8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.754+0000 7f3776a00700 1 -- 172.21.15.78:0/1683521199 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d937252b60 con 0x55d9374d2680 2024-03-14T09:58:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.754+0000 7f3776000700 1 --2- 172.21.15.78:0/1683521199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2fa0 0x55d9374d8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:58:28.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.754+0000 7f3776a00700 1 --2- 172.21.15.78:0/1683521199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2680 0x55d9375044e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3768009c80 tx=0x7f3768009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbfc83c70e70c122 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:28.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.755+0000 7f376cc00700 1 -- 172.21.15.78:0/1683521199 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3768004af0 con 0x55d9374d2680 2024-03-14T09:58:28.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.755+0000 7f376cc00700 1 -- 172.21.15.78:0/1683521199 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3768029070 con 0x55d9374d2680 2024-03-14T09:58:28.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.755+0000 7f376cc00700 1 -- 172.21.15.78:0/1683521199 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37680038c0 con 0x55d9374d2680 2024-03-14T09:58:28.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.756+0000 7f379178bb80 1 -- 172.21.15.78:0/1683521199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2680 msgr2=0x55d9375044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:28.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.756+0000 7f379178bb80 1 --2- 172.21.15.78:0/1683521199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2680 0x55d9375044e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3768009c80 tx=0x7f3768009260 comp rx=0 tx=0).stop 2024-03-14T09:58:28.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.756+0000 7f379178bb80 1 -- 172.21.15.78:0/1683521199 shutdown_connections 2024-03-14T09:58:28.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.756+0000 7f379178bb80 1 --2- 172.21.15.78:0/1683521199 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2680 0x55d9375044e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.756+0000 7f379178bb80 1 --2- 172.21.15.78:0/1683521199 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 0x55d9374d8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.756+0000 7f379178bb80 1 --2- 172.21.15.78:0/1683521199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2fa0 0x55d9374d8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.756+0000 7f379178bb80 1 -- 172.21.15.78:0/1683521199 >> 172.21.15.78:0/1683521199 conn(0x55d937448220 msgr2=0x55d9374c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:28.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.756+0000 7f379178bb80 1 -- 172.21.15.78:0/1683521199 shutdown_connections 2024-03-14T09:58:28.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.756+0000 7f379178bb80 1 -- 172.21.15.78:0/1683521199 wait complete. 2024-03-14T09:58:28.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.756+0000 7f379178bb80 1 -- start start 2024-03-14T09:58:28.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.757+0000 7f379178bb80 1 Processor -- start 2024-03-14T09:58:28.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.757+0000 7f379178bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2680 0x55d9374e20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.757+0000 7f379178bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2fa0 0x55d9374e25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.757+0000 7f379178bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 0x55d9374e2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.757+0000 7f379178bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93754e360 con 0x55d9374d2680 2024-03-14T09:58:28.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.757+0000 7f379178bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93754e4d0 con 0x55d9374d2fa0 2024-03-14T09:58:28.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.757+0000 7f379178bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93754e690 con 0x55d9374e56d0 2024-03-14T09:58:28.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.757+0000 7f379178bb80 10 client.0 _ll_register_callbacks cb 0x7f377000cdd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:58:28.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.757+0000 7f3776a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2680 0x55d9374e20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.757+0000 7f3777400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 0x55d9374e2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.757+0000 7f3776000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2fa0 0x55d9374e25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.757+0000 7f3776a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2680 0x55d9374e20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51080/0 (socket says 172.21.15.78:51080) 2024-03-14T09:58:28.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.757+0000 7f3776a00700 1 -- 172.21.15.78:0/1462232588 learned_addr learned my addr 172.21.15.78:0/1462232588 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:28.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.758+0000 7f3777400700 1 -- 172.21.15.78:0/1462232588 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2fa0 msgr2=0x55d9374e25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:28.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.758+0000 7f3777400700 1 --2- 172.21.15.78:0/1462232588 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2fa0 0x55d9374e25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.758+0000 7f3777400700 1 -- 172.21.15.78:0/1462232588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2680 msgr2=0x55d9374e20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:28.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.758+0000 7f3777400700 1 --2- 172.21.15.78:0/1462232588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2680 0x55d9374e20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.758+0000 7f3777400700 1 -- 172.21.15.78:0/1462232588 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37781a62e0 con 0x55d9374e56d0 2024-03-14T09:58:28.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.758+0000 7f3776a00700 1 --2- 172.21.15.78:0/1462232588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2680 0x55d9374e20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:58:28.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.758+0000 7f3776000700 1 --2- 172.21.15.78:0/1462232588 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2fa0 0x55d9374e25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:58:28.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.758+0000 7f3777400700 1 --2- 172.21.15.78:0/1462232588 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 0x55d9374e2b30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f37700054b0 tx=0x7f3770005b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af15cbe2c3a55cc server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:28.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.758+0000 7f3760c00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:58:28.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.758+0000 7f3760c00700 1 -- 172.21.15.78:0/1462232588 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37700077c0 con 0x55d9374e56d0 2024-03-14T09:58:28.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.758+0000 7f379178bb80 1 -- 172.21.15.78:0/1462232588 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d9374c5720 con 0x55d9374e56d0 2024-03-14T09:58:28.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.758+0000 7f3760c00700 1 -- 172.21.15.78:0/1462232588 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f37700056a0 con 0x55d9374e56d0 2024-03-14T09:58:28.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.759+0000 7f3760c00700 1 -- 172.21.15.78:0/1462232588 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3770028650 con 0x55d9374e56d0 2024-03-14T09:58:28.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.759+0000 7f379178bb80 10 client.26506 fetch_fsmap learned FSMap version 25 2024-03-14T09:58:28.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.759+0000 7f379178bb80 1 -- 172.21.15.78:0/1462232588 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f376801d040 con 0x55d9374e56d0 2024-03-14T09:58:28.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.759+0000 7f3760c00700 1 -- 172.21.15.78:0/1462232588 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d9374c5720 con 0x55d9374e56d0 2024-03-14T09:58:28.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.759+0000 7f3760c00700 1 -- 172.21.15.78:0/1462232588 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3770019030 con 0x55d9374e56d0 2024-03-14T09:58:28.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.759+0000 7f379178bb80 10 client.26506 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:58:28.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.759+0000 7f379178bb80 10 client.26506 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:58:28.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.759+0000 7f379178bb80 20 client.26506 populate_metadata read hostname 'smithi078' 2024-03-14T09:58:28.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.759+0000 7f379178bb80 1 --2- 172.21.15.78:0/1462232588 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d93754f9e0 0x55d937551e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:28.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.759+0000 7f379178bb80 4 client.26506 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:58:28.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.759+0000 7f379178bb80 1 -- 172.21.15.78:0/1462232588 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d9372a8ce0 con 0x55d93754f9e0 2024-03-14T09:58:28.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.760+0000 7f3776a00700 1 --2- 172.21.15.78:0/1462232588 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d93754f9e0 0x55d937551e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:28.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.760+0000 7f3776a00700 1 --2- 172.21.15.78:0/1462232588 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d93754f9e0 0x55d937551e00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=117ad8226a1c750e server_cookie=ac86df896362b868 in_seq=0 out_seq=0 2024-03-14T09:58:28.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.760+0000 7f3760c00700 10 client.26506 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:58:28.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.761+0000 7f3760c00700 1 -- 172.21.15.78:0/1462232588 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d9372a8ce0 con 0x55d93754f9e0 2024-03-14T09:58:28.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.761+0000 7f3760c00700 10 client.26506 handle_command_reply: tid=0 2024-03-14T09:58:28.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.761+0000 7f379178bb80 1 client.26506 shutdown 2024-03-14T09:58:28.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.761+0000 7f379178bb80 1 -- 172.21.15.78:0/1462232588 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 msgr2=0x55d9374e2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:28.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.761+0000 7f379178bb80 1 --2- 172.21.15.78:0/1462232588 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 0x55d9374e2b30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f37700054b0 tx=0x7f3770005b90 comp rx=0 tx=0).stop 2024-03-14T09:58:28.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.761+0000 7f379178bb80 1 -- 172.21.15.78:0/1462232588 shutdown_connections 2024-03-14T09:58:28.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.761+0000 7f379178bb80 1 --2- 172.21.15.78:0/1462232588 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d93754f9e0 0x55d937551e00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.761+0000 7f379178bb80 1 --2- 172.21.15.78:0/1462232588 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d9374e56d0 0x55d9374e2b30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.761+0000 7f379178bb80 1 --2- 172.21.15.78:0/1462232588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9374d2680 0x55d9374e20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.761+0000 7f3760c00700 0 client.26506 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:58:28.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.761+0000 7f379178bb80 1 --2- 172.21.15.78:0/1462232588 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d9374d2fa0 0x55d9374e25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.761+0000 7f379178bb80 1 -- 172.21.15.78:0/1462232588 >> 172.21.15.78:0/1462232588 conn(0x55d937448220 msgr2=0x55d9374d8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:28.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.761+0000 7f379178bb80 1 -- 172.21.15.78:0/1462232588 shutdown_connections 2024-03-14T09:58:28.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.761+0000 7f379178bb80 1 -- 172.21.15.78:0/1462232588 wait complete. 2024-03-14T09:58:28.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.761+0000 7f379178bb80 20 client.26506 trim_cache size 0 max 16384 2024-03-14T09:58:28.766 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:58:28.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.762+0000 7f377dc00700 1 -- 172.21.15.78:0/993737949 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f375c0a0c00 msgr2=0x7f375c0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:28.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.762+0000 7f377dc00700 1 --2- 172.21.15.78:0/993737949 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f375c0a0c00 0x7f375c0a30c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f377000b780 tx=0x7f37700099e0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.762+0000 7f377dc00700 1 -- 172.21.15.78:0/993737949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3778106050 msgr2=0x7f37780735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:28.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.762+0000 7f377dc00700 1 --2- 172.21.15.78:0/993737949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3778106050 0x7f37780735f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f376400e970 tx=0x7f376400ec80 comp rx=0 tx=0).stop 2024-03-14T09:58:28.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.762+0000 7f377dc00700 1 -- 172.21.15.78:0/993737949 shutdown_connections 2024-03-14T09:58:28.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.762+0000 7f377dc00700 1 --2- 172.21.15.78:0/993737949 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f375c0a0c00 0x7f375c0a30c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.762+0000 7f377dc00700 1 --2- 172.21.15.78:0/993737949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3778100f30 0x7f37780730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.762+0000 7f377dc00700 1 --2- 172.21.15.78:0/993737949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3778073b30 0x7f37781a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.762+0000 7f377dc00700 1 --2- 172.21.15.78:0/993737949 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3778106050 0x7f37780735f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:28.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.762+0000 7f377dc00700 1 -- 172.21.15.78:0/993737949 >> 172.21.15.78:0/993737949 conn(0x7f37780fc7b0 msgr2=0x7f37780feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:28.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.762+0000 7f377dc00700 1 -- 172.21.15.78:0/993737949 shutdown_connections 2024-03-14T09:58:28.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:28.763+0000 7f377dc00700 1 -- 172.21.15.78:0/993737949 wait complete. 2024-03-14T09:58:28.785 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T09:58:29.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.008+0000 7febac800700 1 -- 172.21.15.78:0/4123937265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba4106040 msgr2=0x7feba410dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.008+0000 7febac800700 1 --2- 172.21.15.78:0/4123937265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba4106040 0x7feba410dbc0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7feba000b0a0 tx=0x7feba000b3b0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.008+0000 7febac800700 1 -- 172.21.15.78:0/4123937265 shutdown_connections 2024-03-14T09:58:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.008+0000 7febac800700 1 --2- 172.21.15.78:0/4123937265 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba4106040 0x7feba410dbc0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.008+0000 7febac800700 1 --2- 172.21.15.78:0/4123937265 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feba4100f20 0x7feba4101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.008+0000 7febac800700 1 --2- 172.21.15.78:0/4123937265 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feba41018d0 0x7feba4105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.008+0000 7febac800700 1 -- 172.21.15.78:0/4123937265 >> 172.21.15.78:0/4123937265 conn(0x7feba40fc7a0 msgr2=0x7feba40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.009+0000 7febac800700 1 -- 172.21.15.78:0/4123937265 shutdown_connections 2024-03-14T09:58:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.009+0000 7febac800700 1 -- 172.21.15.78:0/4123937265 wait complete. 2024-03-14T09:58:29.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.009+0000 7febac800700 1 Processor -- start 2024-03-14T09:58:29.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.010+0000 7febac800700 1 -- start start 2024-03-14T09:58:29.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.010+0000 7febac800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba4100f20 0x7feba40730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.010+0000 7febac800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feba4106040 0x7feba40735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.010+0000 7febac800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feba4073b30 0x7feba41a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.010+0000 7febac800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba41a5ee0 con 0x7feba4100f20 2024-03-14T09:58:29.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.010+0000 7febac800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba41a6020 con 0x7feba4073b30 2024-03-14T09:58:29.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.010+0000 7febac800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba41a61e0 con 0x7feba4106040 2024-03-14T09:58:29.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.010+0000 7feba9400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feba4106040 0x7feba40735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.010+0000 7feba9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba4100f20 0x7feba40730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.010+0000 7feba9400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feba4106040 0x7feba40735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:58988/0 (socket says 172.21.15.78:58988) 2024-03-14T09:58:29.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.010+0000 7feba9400700 1 -- 172.21.15.78:0/1166495668 learned_addr learned my addr 172.21.15.78:0/1166495668 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:29.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.010+0000 7febaa800700 1 --2- 172.21.15.78:0/1166495668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feba4073b30 0x7feba41a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.011+0000 7feba9400700 1 -- 172.21.15.78:0/1166495668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feba4073b30 msgr2=0x7feba41a5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.011+0000 7feba9400700 1 --2- 172.21.15.78:0/1166495668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feba4073b30 0x7feba41a5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.011+0000 7feba9400700 1 -- 172.21.15.78:0/1166495668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba4100f20 msgr2=0x7feba40730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.011+0000 7feba9400700 1 --2- 172.21.15.78:0/1166495668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba4100f20 0x7feba40730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.011+0000 7feba9400700 1 -- 172.21.15.78:0/1166495668 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba0009d00 con 0x7feba4106040 2024-03-14T09:58:29.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.011+0000 7feba9e00700 1 --2- 172.21.15.78:0/1166495668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba4100f20 0x7feba40730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:58:29.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.011+0000 7feba9400700 1 --2- 172.21.15.78:0/1166495668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feba4106040 0x7feba40735f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7feb9400e970 tx=0x7feb9400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:29.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.011+0000 7feb9ea00700 1 -- 172.21.15.78:0/1166495668 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94009790 con 0x7feba4106040 2024-03-14T09:58:29.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.011+0000 7febac800700 1 -- 172.21.15.78:0/1166495668 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba41a64c0 con 0x7feba4106040 2024-03-14T09:58:29.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.011+0000 7feb9ea00700 1 -- 172.21.15.78:0/1166495668 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb9400c7e0 con 0x7feba4106040 2024-03-14T09:58:29.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.011+0000 7feb9ea00700 1 -- 172.21.15.78:0/1166495668 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb940054e0 con 0x7feba4106040 2024-03-14T09:58:29.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.011+0000 7febac800700 1 -- 172.21.15.78:0/1166495668 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba41a6a20 con 0x7feba4106040 2024-03-14T09:58:29.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.014+0000 7feb9ea00700 1 -- 172.21.15.78:0/1166495668 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7feb9400c950 con 0x7feba4106040 2024-03-14T09:58:29.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.016+0000 7febc028cb80 1 Processor -- start 2024-03-14T09:58:29.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.016+0000 7febc028cb80 1 -- start start 2024-03-14T09:58:29.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.016+0000 7feb9ea00700 1 --2- 172.21.15.78:0/1166495668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7feb8c0a0c00 0x7feb8c0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7feb9ea00700 1 -- 172.21.15.78:0/1166495668 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7feb94014070 con 0x7feba4106040 2024-03-14T09:58:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7febc028cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 0x55a34d3f0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7febc028cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 0x55a34d3fa980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7febaa800700 1 --2- 172.21.15.78:0/1166495668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7feb8c0a0c00 0x7feb8c0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7febc028cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3faec0 0x55a34d3fd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7febc028cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34d349990 con 0x55a34d3faec0 2024-03-14T09:58:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7febc028cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34d347350 con 0x55a34d3f0fa0 2024-03-14T09:58:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7febc028cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34d342c40 con 0x55a34d3f0680 2024-03-14T09:58:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7feba9e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 0x55a34d3f0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7feba9e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 0x55a34d3f0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59000/0 (socket says 172.21.15.78:59000) 2024-03-14T09:58:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7feba9e00700 1 -- 172.21.15.78:0/18654894 learned_addr learned my addr 172.21.15.78:0/18654894 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7febaa800700 1 --2- 172.21.15.78:0/18654894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3faec0 0x55a34d3fd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7feba9400700 1 --2- 172.21.15.78:0/18654894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 0x55a34d3fa980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7feba9e00700 1 -- 172.21.15.78:0/18654894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 msgr2=0x55a34d3fa980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7feba9e00700 1 --2- 172.21.15.78:0/18654894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 0x55a34d3fa980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7feba9e00700 1 -- 172.21.15.78:0/18654894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3faec0 msgr2=0x55a34d3fd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7feba9e00700 1 --2- 172.21.15.78:0/18654894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3faec0 0x55a34d3fd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.017+0000 7feba9e00700 1 -- 172.21.15.78:0/18654894 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a34d1c43b0 con 0x55a34d3f0680 2024-03-14T09:58:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.018+0000 7feba9e00700 1 --2- 172.21.15.78:0/18654894 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 0x55a34d3f0a60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7feb980115f0 tx=0x7feb98011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb5816b29b17aaa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.018+0000 7febaa800700 1 --2- 172.21.15.78:0/1166495668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7feb8c0a0c00 0x7feb8c0a30c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7feba000b780 tx=0x7feba0007d70 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.018+0000 7feb93000700 1 -- 172.21.15.78:0/18654894 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb98019630 con 0x55a34d3f0680 2024-03-14T09:58:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.018+0000 7feb93000700 1 -- 172.21.15.78:0/18654894 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb98019c50 con 0x55a34d3f0680 2024-03-14T09:58:29.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.018+0000 7feb93000700 1 -- 172.21.15.78:0/18654894 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9800ec20 con 0x55a34d3f0680 2024-03-14T09:58:29.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.018+0000 7febc028cb80 1 -- 172.21.15.78:0/18654894 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 msgr2=0x55a34d3f0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.018+0000 7febc028cb80 1 --2- 172.21.15.78:0/18654894 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 0x55a34d3f0a60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7feb980115f0 tx=0x7feb98011900 comp rx=0 tx=0).stop 2024-03-14T09:58:29.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.019+0000 7febc028cb80 1 -- 172.21.15.78:0/18654894 shutdown_connections 2024-03-14T09:58:29.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.019+0000 7febc028cb80 1 --2- 172.21.15.78:0/18654894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3faec0 0x55a34d3fd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.019+0000 7febc028cb80 1 --2- 172.21.15.78:0/18654894 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 0x55a34d3f0a60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.019+0000 7febc028cb80 1 --2- 172.21.15.78:0/18654894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 0x55a34d3fa980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.019+0000 7febc028cb80 1 -- 172.21.15.78:0/18654894 >> 172.21.15.78:0/18654894 conn(0x55a34d366220 msgr2=0x55a34d3ebaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.019+0000 7febc028cb80 1 -- 172.21.15.78:0/18654894 shutdown_connections 2024-03-14T09:58:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.019+0000 7febc028cb80 1 -- 172.21.15.78:0/18654894 wait complete. 2024-03-14T09:58:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.019+0000 7febc028cb80 1 -- start start 2024-03-14T09:58:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.019+0000 7febc028cb80 1 Processor -- start 2024-03-14T09:58:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7febc028cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 0x55a34d3fa900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7febc028cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 0x55a34d3f7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7febc028cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3faec0 0x55a34d3f8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7febc028cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34d2ded80 con 0x55a34d3faec0 2024-03-14T09:58:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7febc028cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34d2dea20 con 0x55a34d3f0fa0 2024-03-14T09:58:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7febc028cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34d1c66a0 con 0x55a34d3f0680 2024-03-14T09:58:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7febc028cb80 10 client.0 _ll_register_callbacks cb 0x55a34d14a3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:58:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7febaa800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3faec0 0x55a34d3f8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7feba9e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 0x55a34d3fa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7feba9400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 0x55a34d3f7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7febaa800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3faec0 0x55a34d3f8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51136/0 (socket says 172.21.15.78:51136) 2024-03-14T09:58:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7feba9e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 0x55a34d3fa900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59010/0 (socket says 172.21.15.78:59010) 2024-03-14T09:58:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7febaa800700 1 -- 172.21.15.78:0/3035162305 learned_addr learned my addr 172.21.15.78:0/3035162305 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7feba9400700 1 -- 172.21.15.78:0/3035162305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 msgr2=0x55a34d3fa900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7feba9400700 1 --2- 172.21.15.78:0/3035162305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 0x55a34d3fa900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7feba9400700 1 -- 172.21.15.78:0/3035162305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3faec0 msgr2=0x55a34d3f8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7feba9400700 1 --2- 172.21.15.78:0/3035162305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3faec0 0x55a34d3f8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.020+0000 7feba9400700 1 -- 172.21.15.78:0/3035162305 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb98009dd0 con 0x55a34d3f0fa0 2024-03-14T09:58:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.021+0000 7feba9400700 1 --2- 172.21.15.78:0/3035162305 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 0x55a34d3f7c40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7feb940c9d50 tx=0x7feb9409a140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7d4fb5a003375f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.021+0000 7feb87400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:58:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.021+0000 7feb87400700 1 -- 172.21.15.78:0/3035162305 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9409d780 con 0x55a34d3f0fa0 2024-03-14T09:58:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.021+0000 7febc028cb80 1 -- 172.21.15.78:0/3035162305 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a34d3e3720 con 0x55a34d3f0fa0 2024-03-14T09:58:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.021+0000 7feb87400700 1 -- 172.21.15.78:0/3035162305 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7feb940a1dd0 con 0x55a34d3f0fa0 2024-03-14T09:58:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.021+0000 7feb87400700 1 -- 172.21.15.78:0/3035162305 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb94016330 con 0x55a34d3f0fa0 2024-03-14T09:58:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.021+0000 7febc028cb80 10 client.26531 fetch_fsmap learned FSMap version 25 2024-03-14T09:58:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.021+0000 7febc028cb80 1 -- 172.21.15.78:0/3035162305 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a34d170b60 con 0x55a34d3f0fa0 2024-03-14T09:58:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.022+0000 7feb87400700 1 -- 172.21.15.78:0/3035162305 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a34d3e3720 con 0x55a34d3f0fa0 2024-03-14T09:58:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.022+0000 7feb87400700 1 -- 172.21.15.78:0/3035162305 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7feb94098e40 con 0x55a34d3f0fa0 2024-03-14T09:58:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.022+0000 7febc028cb80 10 client.26531 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:58:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.022+0000 7febc028cb80 10 client.26531 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:58:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.022+0000 7febc028cb80 20 client.26531 populate_metadata read hostname 'smithi078' 2024-03-14T09:58:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.022+0000 7febc028cb80 1 --2- 172.21.15.78:0/3035162305 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55a34d40ab10 0x55a34d40cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.022+0000 7febc028cb80 4 client.26531 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:58:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.022+0000 7febc028cb80 1 -- 172.21.15.78:0/3035162305 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a34d33b700 con 0x55a34d40ab10 2024-03-14T09:58:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.022+0000 7feba9e00700 1 --2- 172.21.15.78:0/3035162305 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55a34d40ab10 0x55a34d40cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.023+0000 7feba9e00700 1 --2- 172.21.15.78:0/3035162305 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55a34d40ab10 0x55a34d40cf30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fc024f140725ec6a server_cookie=4dd72e7a1cfd33b5 in_seq=0 out_seq=0 2024-03-14T09:58:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.023+0000 7feb87400700 10 client.26531 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:58:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.023+0000 7feb87400700 1 -- 172.21.15.78:0/3035162305 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55a34d33b700 con 0x55a34d40ab10 2024-03-14T09:58:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.023+0000 7feb87400700 10 client.26531 handle_command_reply: tid=0 2024-03-14T09:58:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.023+0000 7febc028cb80 1 client.26531 shutdown 2024-03-14T09:58:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.023+0000 7febc028cb80 1 -- 172.21.15.78:0/3035162305 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 msgr2=0x55a34d3f7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.023+0000 7febc028cb80 1 --2- 172.21.15.78:0/3035162305 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 0x55a34d3f7c40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7feb940c9d50 tx=0x7feb9409a140 comp rx=0 tx=0).stop 2024-03-14T09:58:29.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.023+0000 7febc028cb80 1 -- 172.21.15.78:0/3035162305 shutdown_connections 2024-03-14T09:58:29.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.023+0000 7febc028cb80 1 --2- 172.21.15.78:0/3035162305 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55a34d40ab10 0x55a34d40cf30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.024+0000 7febc028cb80 1 --2- 172.21.15.78:0/3035162305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3faec0 0x55a34d3f8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.024+0000 7febc028cb80 1 --2- 172.21.15.78:0/3035162305 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 0x55a34d3fa900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.024+0000 7febc028cb80 1 --2- 172.21.15.78:0/3035162305 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 0x55a34d3f7c40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.024+0000 7feb87400700 0 client.26531 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:58:29.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.024+0000 7febc028cb80 1 -- 172.21.15.78:0/3035162305 >> 172.21.15.78:0/3035162305 conn(0x55a34d366220 msgr2=0x55a34d3eb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:29.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.024+0000 7febc028cb80 1 -- 172.21.15.78:0/3035162305 shutdown_connections 2024-03-14T09:58:29.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.024+0000 7febc028cb80 1 -- 172.21.15.78:0/3035162305 wait complete. 2024-03-14T09:58:29.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.024+0000 7febc028cb80 20 client.26531 trim_cache size 0 max 16384 2024-03-14T09:58:29.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.037+0000 7febc028cb80 1 Processor -- start 2024-03-14T09:58:29.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.037+0000 7febc028cb80 1 -- start start 2024-03-14T09:58:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.037+0000 7febc028cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3f0680 0x55a34d4118b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.037+0000 7febc028cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0fa0 0x55a34d401b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.037+0000 7febc028cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3faec0 0x55a34d406120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.037+0000 7febc028cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34d3ec770 con 0x55a34d3f0680 2024-03-14T09:58:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.037+0000 7febc028cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34d3f2700 con 0x55a34d3faec0 2024-03-14T09:58:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.037+0000 7febc028cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34d3f2310 con 0x55a34d3f0fa0 2024-03-14T09:58:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.037+0000 7feba9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3f0680 0x55a34d4118b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.037+0000 7feba9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3f0680 0x55a34d4118b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51148/0 (socket says 172.21.15.78:51148) 2024-03-14T09:58:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.037+0000 7feba9400700 1 --2- 172.21.15.78:0/478598290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0fa0 0x55a34d401b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.037+0000 7feba9e00700 1 -- 172.21.15.78:0/478598290 learned_addr learned my addr 172.21.15.78:0/478598290 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.038+0000 7febaa800700 1 --2- 172.21.15.78:0/478598290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3faec0 0x55a34d406120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.038+0000 7feba9e00700 1 -- 172.21.15.78:0/478598290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0fa0 msgr2=0x55a34d401b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.038+0000 7feba9e00700 1 --2- 172.21.15.78:0/478598290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0fa0 0x55a34d401b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.038+0000 7feba9e00700 1 -- 172.21.15.78:0/478598290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3faec0 msgr2=0x55a34d406120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.038+0000 7feba9e00700 1 --2- 172.21.15.78:0/478598290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3faec0 0x55a34d406120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.038+0000 7feba9e00700 1 -- 172.21.15.78:0/478598290 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a34d308ea0 con 0x55a34d3f0680 2024-03-14T09:58:29.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.038+0000 7feba9e00700 1 --2- 172.21.15.78:0/478598290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3f0680 0x55a34d4118b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7feb98000c00 tx=0x7feb98011540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f0830fe098d2d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:29.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.038+0000 7feb93a00700 1 -- 172.21.15.78:0/478598290 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb98028070 con 0x55a34d3f0680 2024-03-14T09:58:29.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.038+0000 7feb93a00700 1 -- 172.21.15.78:0/478598290 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb98005570 con 0x55a34d3f0680 2024-03-14T09:58:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.038+0000 7feb93a00700 1 -- 172.21.15.78:0/478598290 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb98019790 con 0x55a34d3f0680 2024-03-14T09:58:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.039+0000 7febc028cb80 1 -- 172.21.15.78:0/478598290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3f0680 msgr2=0x55a34d4118b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.039+0000 7febc028cb80 1 --2- 172.21.15.78:0/478598290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3f0680 0x55a34d4118b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7feb98000c00 tx=0x7feb98011540 comp rx=0 tx=0).stop 2024-03-14T09:58:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.039+0000 7febc028cb80 1 -- 172.21.15.78:0/478598290 shutdown_connections 2024-03-14T09:58:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.039+0000 7febc028cb80 1 --2- 172.21.15.78:0/478598290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3faec0 0x55a34d406120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.039+0000 7febc028cb80 1 --2- 172.21.15.78:0/478598290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3f0680 0x55a34d4118b0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.039+0000 7febc028cb80 1 --2- 172.21.15.78:0/478598290 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0fa0 0x55a34d401b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.039+0000 7febc028cb80 1 -- 172.21.15.78:0/478598290 >> 172.21.15.78:0/478598290 conn(0x55a34d366220 msgr2=0x55a34d3fa510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.039+0000 7febc028cb80 1 -- 172.21.15.78:0/478598290 shutdown_connections 2024-03-14T09:58:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.039+0000 7febc028cb80 1 -- 172.21.15.78:0/478598290 wait complete. 2024-03-14T09:58:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.039+0000 7febc028cb80 1 -- start start 2024-03-14T09:58:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.040+0000 7febc028cb80 1 Processor -- start 2024-03-14T09:58:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.040+0000 7febc028cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 0x55a34d401370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.040+0000 7febc028cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 0x55a34d4018b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.040+0000 7febc028cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3faec0 0x55a34d3ff8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.040+0000 7febc028cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34d3ffec0 con 0x55a34d3faec0 2024-03-14T09:58:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.040+0000 7febc028cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34d400030 con 0x55a34d3f0fa0 2024-03-14T09:58:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.040+0000 7febc028cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34d4001f0 con 0x55a34d3f0680 2024-03-14T09:58:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.040+0000 7febc028cb80 10 client.0 _ll_register_callbacks cb 0x7feb9400cc50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:58:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.040+0000 7feba9400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 0x55a34d4018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.040+0000 7febaa800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3faec0 0x55a34d3ff8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.040+0000 7feba9400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 0x55a34d4018b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:38796/0 (socket says 172.21.15.78:38796) 2024-03-14T09:58:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.040+0000 7feba9400700 1 -- 172.21.15.78:0/2780073839 learned_addr learned my addr 172.21.15.78:0/2780073839 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.040+0000 7feba9e00700 1 --2- 172.21.15.78:0/2780073839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 0x55a34d401370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.041+0000 7feba9400700 1 -- 172.21.15.78:0/2780073839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 msgr2=0x55a34d401370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.041+0000 7feba9400700 1 --2- 172.21.15.78:0/2780073839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 0x55a34d401370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.041+0000 7feba9400700 1 -- 172.21.15.78:0/2780073839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3faec0 msgr2=0x55a34d3ff8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.041+0000 7feba9400700 1 --2- 172.21.15.78:0/2780073839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3faec0 0x55a34d3ff8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.041+0000 7feba9400700 1 -- 172.21.15.78:0/2780073839 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a34d1c43b0 con 0x55a34d3f0fa0 2024-03-14T09:58:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.041+0000 7feba9400700 1 --2- 172.21.15.78:0/2780073839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 0x55a34d4018b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7feb94061040 tx=0x7feb9400e8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=565369e8234fb435 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.041+0000 7feb87400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:58:29.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.041+0000 7feb87400700 1 -- 172.21.15.78:0/2780073839 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94084600 con 0x55a34d3f0fa0 2024-03-14T09:58:29.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.041+0000 7feb87400700 1 -- 172.21.15.78:0/2780073839 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7feb940a1dd0 con 0x55a34d3f0fa0 2024-03-14T09:58:29.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.041+0000 7febc028cb80 1 -- 172.21.15.78:0/2780073839 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feb98019c50 con 0x55a34d3f0fa0 2024-03-14T09:58:29.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.041+0000 7feb87400700 1 -- 172.21.15.78:0/2780073839 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb940a1a60 con 0x55a34d3f0fa0 2024-03-14T09:58:29.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.041+0000 7febc028cb80 10 client.26537 fetch_fsmap learned FSMap version 25 2024-03-14T09:58:29.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.041+0000 7febc028cb80 1 -- 172.21.15.78:0/2780073839 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a34d1c8cd0 con 0x55a34d3f0fa0 2024-03-14T09:58:29.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.042+0000 7feb87400700 1 -- 172.21.15.78:0/2780073839 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb98019c50 con 0x55a34d3f0fa0 2024-03-14T09:58:29.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.042+0000 7feb87400700 1 -- 172.21.15.78:0/2780073839 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7feb94099bb0 con 0x55a34d3f0fa0 2024-03-14T09:58:29.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.042+0000 7febc028cb80 10 client.26537 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:58:29.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.042+0000 7febc028cb80 10 client.26537 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:58:29.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.042+0000 7febc028cb80 20 client.26537 populate_metadata read hostname 'smithi078' 2024-03-14T09:58:29.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.042+0000 7febc028cb80 1 --2- 172.21.15.78:0/2780073839 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55a34d45d370 0x55a34d45f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.042+0000 7febc028cb80 4 client.26537 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:58:29.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.042+0000 7febc028cb80 1 -- 172.21.15.78:0/2780073839 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a34d191cf0 con 0x55a34d45d370 2024-03-14T09:58:29.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.042+0000 7feba9e00700 1 --2- 172.21.15.78:0/2780073839 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55a34d45d370 0x55a34d45f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.042+0000 7feb87400700 10 client.26537 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:58:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.042+0000 7feba9e00700 1 --2- 172.21.15.78:0/2780073839 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55a34d45d370 0x55a34d45f760 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5d8e0743b8c74c38 server_cookie=c416ccb4d95ae34d in_seq=0 out_seq=0 2024-03-14T09:58:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.043+0000 7feb87400700 1 -- 172.21.15.78:0/2780073839 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a34d191cf0 con 0x55a34d45d370 2024-03-14T09:58:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.043+0000 7feb87400700 10 client.26537 handle_command_reply: tid=0 2024-03-14T09:58:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.043+0000 7febc028cb80 1 client.26537 shutdown 2024-03-14T09:58:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.043+0000 7febc028cb80 1 -- 172.21.15.78:0/2780073839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 msgr2=0x55a34d4018b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.043+0000 7febc028cb80 1 --2- 172.21.15.78:0/2780073839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 0x55a34d4018b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7feb94061040 tx=0x7feb9400e8f0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.043+0000 7febc028cb80 1 -- 172.21.15.78:0/2780073839 shutdown_connections 2024-03-14T09:58:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.043+0000 7febc028cb80 1 --2- 172.21.15.78:0/2780073839 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55a34d45d370 0x55a34d45f760 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.043+0000 7febc028cb80 1 --2- 172.21.15.78:0/2780073839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a34d3faec0 0x55a34d3ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.043+0000 7febc028cb80 1 --2- 172.21.15.78:0/2780073839 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a34d3f0680 0x55a34d401370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.043+0000 7febc028cb80 1 --2- 172.21.15.78:0/2780073839 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a34d3f0fa0 0x55a34d4018b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.043+0000 7febc028cb80 1 -- 172.21.15.78:0/2780073839 >> 172.21.15.78:0/2780073839 conn(0x55a34d366220 msgr2=0x55a34d3f2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.043+0000 7feb87400700 0 client.26537 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:58:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.043+0000 7febc028cb80 1 -- 172.21.15.78:0/2780073839 shutdown_connections 2024-03-14T09:58:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.043+0000 7febc028cb80 1 -- 172.21.15.78:0/2780073839 wait complete. 2024-03-14T09:58:29.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.043+0000 7febc028cb80 20 client.26537 trim_cache size 0 max 16384 2024-03-14T09:58:29.048 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:58:29.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.044+0000 7febac800700 1 -- 172.21.15.78:0/1166495668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7feb8c0a0c00 msgr2=0x7feb8c0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.044+0000 7febac800700 1 --2- 172.21.15.78:0/1166495668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7feb8c0a0c00 0x7feb8c0a30c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7feba000b780 tx=0x7feba0007d70 comp rx=0 tx=0).stop 2024-03-14T09:58:29.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.044+0000 7febac800700 1 -- 172.21.15.78:0/1166495668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feba4106040 msgr2=0x7feba40735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.044+0000 7febac800700 1 --2- 172.21.15.78:0/1166495668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feba4106040 0x7feba40735f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7feb9400e970 tx=0x7feb9400ec80 comp rx=0 tx=0).stop 2024-03-14T09:58:29.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.044+0000 7febac800700 1 -- 172.21.15.78:0/1166495668 shutdown_connections 2024-03-14T09:58:29.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.044+0000 7febac800700 1 --2- 172.21.15.78:0/1166495668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7feb8c0a0c00 0x7feb8c0a30c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.044+0000 7febac800700 1 --2- 172.21.15.78:0/1166495668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feba4073b30 0x7feba41a5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.044+0000 7febac800700 1 --2- 172.21.15.78:0/1166495668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feba4100f20 0x7feba40730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.044+0000 7febac800700 1 --2- 172.21.15.78:0/1166495668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feba4106040 0x7feba40735f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.044+0000 7febac800700 1 -- 172.21.15.78:0/1166495668 >> 172.21.15.78:0/1166495668 conn(0x7feba40fc7a0 msgr2=0x7feba40fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:29.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.044+0000 7febac800700 1 -- 172.21.15.78:0/1166495668 shutdown_connections 2024-03-14T09:58:29.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.044+0000 7febac800700 1 -- 172.21.15.78:0/1166495668 wait complete. 2024-03-14T09:58:29.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:28 smithi078 ceph-mon[38143]: pgmap v132: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.9 MiB/s wr, 306 op/s 2024-03-14T09:58:29.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:28 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2219874127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:58:29.065 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T09:58:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:28 smithi152 ceph-mon[48279]: pgmap v132: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.9 MiB/s wr, 306 op/s 2024-03-14T09:58:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:28 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2219874127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:58:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:28 smithi132 ceph-mon[47959]: pgmap v132: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.9 MiB/s wr, 306 op/s 2024-03-14T09:58:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:28 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2219874127' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:58:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.311+0000 7f7058200700 1 -- 172.21.15.78:0/2704844544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70500680b0 msgr2=0x7f7050068490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.311+0000 7f7058200700 1 --2- 172.21.15.78:0/2704844544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70500680b0 0x7f7050068490 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f70440099c0 tx=0x7f7044009cd0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.311+0000 7f7058200700 1 -- 172.21.15.78:0/2704844544 shutdown_connections 2024-03-14T09:58:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.311+0000 7f7058200700 1 --2- 172.21.15.78:0/2704844544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7050106060 0x7f705010dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.311+0000 7f7058200700 1 --2- 172.21.15.78:0/2704844544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f70500680b0 0x7f7050068490 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.311+0000 7f7058200700 1 --2- 172.21.15.78:0/2704844544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7050068a60 0x7f7050105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.311+0000 7f7058200700 1 -- 172.21.15.78:0/2704844544 >> 172.21.15.78:0/2704844544 conn(0x7f7050076d80 msgr2=0x7f7050077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.311+0000 7f7058200700 1 -- 172.21.15.78:0/2704844544 shutdown_connections 2024-03-14T09:58:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.312+0000 7f7058200700 1 -- 172.21.15.78:0/2704844544 wait complete. 2024-03-14T09:58:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.313+0000 7f7058200700 1 Processor -- start 2024-03-14T09:58:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.313+0000 7f7058200700 1 -- start start 2024-03-14T09:58:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.313+0000 7f7058200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70500680b0 0x7f705019f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.313+0000 7f7058200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7050106060 0x7f705019fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.313+0000 7f7058200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70501996c0 0x7f7050199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.313+0000 7f7058200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705019a140 con 0x7f70501996c0 2024-03-14T09:58:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.313+0000 7f7058200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705019a280 con 0x7f70500680b0 2024-03-14T09:58:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.313+0000 7f7058200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705019a440 con 0x7f7050106060 2024-03-14T09:58:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.314+0000 7f7054e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7050106060 0x7f705019fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.314+0000 7f7055800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70500680b0 0x7f705019f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.314+0000 7f7054e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7050106060 0x7f705019fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59048/0 (socket says 172.21.15.78:59048) 2024-03-14T09:58:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.314+0000 7f7054e00700 1 -- 172.21.15.78:0/1337808868 learned_addr learned my addr 172.21.15.78:0/1337808868 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.314+0000 7f7055800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70500680b0 0x7f705019f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:38806/0 (socket says 172.21.15.78:38806) 2024-03-14T09:58:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.314+0000 7f7056200700 1 --2- 172.21.15.78:0/1337808868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70501996c0 0x7f7050199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.314+0000 7f7054e00700 1 -- 172.21.15.78:0/1337808868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70500680b0 msgr2=0x7f705019f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.314+0000 7f7054e00700 1 --2- 172.21.15.78:0/1337808868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70500680b0 0x7f705019f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.314+0000 7f7054e00700 1 -- 172.21.15.78:0/1337808868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70501996c0 msgr2=0x7f7050199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.314+0000 7f7054e00700 1 --2- 172.21.15.78:0/1337808868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70501996c0 0x7f7050199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.314+0000 7f7054e00700 1 -- 172.21.15.78:0/1337808868 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703c009520 con 0x7f7050106060 2024-03-14T09:58:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.314+0000 7f7056200700 1 --2- 172.21.15.78:0/1337808868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70501996c0 0x7f7050199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:58:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.314+0000 7f7054e00700 1 --2- 172.21.15.78:0/1337808868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7050106060 0x7f705019fa80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f703c00ea30 tx=0x7f703c00ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.314+0000 7f704e000700 1 -- 172.21.15.78:0/1337808868 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c009820 con 0x7f7050106060 2024-03-14T09:58:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.315+0000 7f7058200700 1 -- 172.21.15.78:0/1337808868 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70440095f0 con 0x7f7050106060 2024-03-14T09:58:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.315+0000 7f704e000700 1 -- 172.21.15.78:0/1337808868 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f703c0044e0 con 0x7f7050106060 2024-03-14T09:58:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.315+0000 7f704e000700 1 -- 172.21.15.78:0/1337808868 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c003dc0 con 0x7f7050106060 2024-03-14T09:58:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.315+0000 7f7058200700 1 -- 172.21.15.78:0/1337808868 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70501a6730 con 0x7f7050106060 2024-03-14T09:58:29.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.318+0000 7f704e000700 1 -- 172.21.15.78:0/1337808868 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f703c004000 con 0x7f7050106060 2024-03-14T09:58:29.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.321+0000 7f704e000700 1 --2- 172.21.15.78:0/1337808868 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f70380a0c00 0x7f70380a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.321+0000 7f706bdb0b80 1 Processor -- start 2024-03-14T09:58:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.321+0000 7f706bdb0b80 1 -- start start 2024-03-14T09:58:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.321+0000 7f704e000700 1 -- 172.21.15.78:0/1337808868 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f703c014070 con 0x7f7050106060 2024-03-14T09:58:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.321+0000 7f7055800700 1 --2- 172.21.15.78:0/1337808868 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f70380a0c00 0x7f70380a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.321+0000 7f706bdb0b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff65680 0x56155ff65a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.321+0000 7f706bdb0b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff65fa0 0x56155ff6f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.321+0000 7f706bdb0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff6fec0 0x56155ff722b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.321+0000 7f706bdb0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56155febe990 con 0x56155ff6fec0 2024-03-14T09:58:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.321+0000 7f706bdb0b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56155febc350 con 0x56155ff65680 2024-03-14T09:58:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.321+0000 7f706bdb0b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56155feb7c40 con 0x56155ff65fa0 2024-03-14T09:58:29.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.321+0000 7f7055800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff65680 0x56155ff65a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.321+0000 7f7056200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff6fec0 0x56155ff722b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.321+0000 7f7054e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff65fa0 0x56155ff6f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.321+0000 7f7055800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff65680 0x56155ff65a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:38808/0 (socket says 172.21.15.78:38808) 2024-03-14T09:58:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.321+0000 7f7055800700 1 -- 172.21.15.78:0/2529987370 learned_addr learned my addr 172.21.15.78:0/2529987370 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.322+0000 7f7054e00700 1 -- 172.21.15.78:0/2529987370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff65680 msgr2=0x56155ff65a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.322+0000 7f7054e00700 1 --2- 172.21.15.78:0/2529987370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff65680 0x56155ff65a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.322+0000 7f7054e00700 1 -- 172.21.15.78:0/2529987370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff6fec0 msgr2=0x56155ff722b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.322+0000 7f7054e00700 1 --2- 172.21.15.78:0/2529987370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff6fec0 0x56155ff722b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.322+0000 7f7054e00700 1 -- 172.21.15.78:0/2529987370 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56155fd393b0 con 0x56155ff65fa0 2024-03-14T09:58:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.322+0000 7f7055800700 1 --2- 172.21.15.78:0/1337808868 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f70380a0c00 0x7f70380a30c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7044000c00 tx=0x7f70440054d0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.322+0000 7f7054e00700 1 --2- 172.21.15.78:0/2529987370 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff65fa0 0x56155ff6f980 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f703c0c9fa0 tx=0x7f703c099810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=698b06d8a6b27896 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.322+0000 7f7042a00700 1 -- 172.21.15.78:0/2529987370 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c09a2a0 con 0x56155ff65fa0 2024-03-14T09:58:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.322+0000 7f7042a00700 1 -- 172.21.15.78:0/2529987370 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f703c099ba0 con 0x56155ff65fa0 2024-03-14T09:58:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.322+0000 7f7042a00700 1 -- 172.21.15.78:0/2529987370 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c0962b0 con 0x56155ff65fa0 2024-03-14T09:58:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.323+0000 7f706bdb0b80 1 -- 172.21.15.78:0/2529987370 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff65fa0 msgr2=0x56155ff6f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.323+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/2529987370 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff65fa0 0x56155ff6f980 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f703c0c9fa0 tx=0x7f703c099810 comp rx=0 tx=0).stop 2024-03-14T09:58:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.323+0000 7f706bdb0b80 1 -- 172.21.15.78:0/2529987370 shutdown_connections 2024-03-14T09:58:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.323+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/2529987370 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff65680 0x56155ff65a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.323+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/2529987370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff6fec0 0x56155ff722b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.323+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/2529987370 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff65fa0 0x56155ff6f980 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.323+0000 7f706bdb0b80 1 -- 172.21.15.78:0/2529987370 >> 172.21.15.78:0/2529987370 conn(0x56155fedb220 msgr2=0x56155ff60aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.324+0000 7f706bdb0b80 1 -- 172.21.15.78:0/2529987370 shutdown_connections 2024-03-14T09:58:29.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.324+0000 7f706bdb0b80 1 -- 172.21.15.78:0/2529987370 wait complete. 2024-03-14T09:58:29.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.324+0000 7f706bdb0b80 1 -- start start 2024-03-14T09:58:29.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.325+0000 7f706bdb0b80 1 Processor -- start 2024-03-14T09:58:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.325+0000 7f706bdb0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff65680 0x56155ff6cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.325+0000 7f706bdb0b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff65fa0 0x56155ff6d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.325+0000 7f706bdb0b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff6fec0 0x56155ff6d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.325+0000 7f706bdb0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56155fe53d80 con 0x56155ff65680 2024-03-14T09:58:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.325+0000 7f706bdb0b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56155fe53a20 con 0x56155ff6fec0 2024-03-14T09:58:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.325+0000 7f706bdb0b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56155fd3b6a0 con 0x56155ff65fa0 2024-03-14T09:58:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.325+0000 7f706bdb0b80 10 client.0 _ll_register_callbacks cb 0x56155fcbf3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:58:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.326+0000 7f7054e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff65fa0 0x56155ff6d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.326+0000 7f7056200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff6fec0 0x56155ff6d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.326+0000 7f7054e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff65fa0 0x56155ff6d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59066/0 (socket says 172.21.15.78:59066) 2024-03-14T09:58:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.326+0000 7f7056200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff6fec0 0x56155ff6d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:38818/0 (socket says 172.21.15.78:38818) 2024-03-14T09:58:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.326+0000 7f7056200700 1 -- 172.21.15.78:0/1125571151 learned_addr learned my addr 172.21.15.78:0/1125571151 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.326+0000 7f7055800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff65680 0x56155ff6cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.326+0000 7f7054e00700 1 -- 172.21.15.78:0/1125571151 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff6fec0 msgr2=0x56155ff6d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.326+0000 7f7054e00700 1 --2- 172.21.15.78:0/1125571151 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff6fec0 0x56155ff6d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.326+0000 7f7054e00700 1 -- 172.21.15.78:0/1125571151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff65680 msgr2=0x56155ff6cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.326+0000 7f7054e00700 1 --2- 172.21.15.78:0/1125571151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff65680 0x56155ff6cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.326+0000 7f7054e00700 1 -- 172.21.15.78:0/1125571151 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70501a6730 con 0x56155ff65fa0 2024-03-14T09:58:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.326+0000 7f7054e00700 1 --2- 172.21.15.78:0/1125571151 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff65fa0 0x56155ff6d250 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x56155ff665b0 tx=0x7f703c095ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccb1e97bb5eb12a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.328+0000 7f7037400700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:58:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.328+0000 7f7037400700 1 -- 172.21.15.78:0/1125571151 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c09a2a0 con 0x56155ff65fa0 2024-03-14T09:58:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.328+0000 7f7037400700 1 -- 172.21.15.78:0/1125571151 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f703c0b9050 con 0x56155ff65fa0 2024-03-14T09:58:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.328+0000 7f706bdb0b80 1 -- 172.21.15.78:0/1125571151 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56155ff58720 con 0x56155ff65fa0 2024-03-14T09:58:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.328+0000 7f7037400700 1 -- 172.21.15.78:0/1125571151 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f703c097f40 con 0x56155ff65fa0 2024-03-14T09:58:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.328+0000 7f706bdb0b80 10 client.26536 fetch_fsmap learned FSMap version 25 2024-03-14T09:58:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.328+0000 7f706bdb0b80 1 -- 172.21.15.78:0/1125571151 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56155fce5b60 con 0x56155ff65fa0 2024-03-14T09:58:29.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.329+0000 7f7037400700 1 -- 172.21.15.78:0/1125571151 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56155ff58720 con 0x56155ff65fa0 2024-03-14T09:58:29.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.329+0000 7f7037400700 1 -- 172.21.15.78:0/1125571151 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f703c099090 con 0x56155ff65fa0 2024-03-14T09:58:29.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.329+0000 7f706bdb0b80 10 client.26536 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:58:29.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.329+0000 7f706bdb0b80 10 client.26536 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:58:29.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.329+0000 7f706bdb0b80 20 client.26536 populate_metadata read hostname 'smithi078' 2024-03-14T09:58:29.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.329+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/1125571151 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56155ff87f40 0x56155ff8a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.329+0000 7f706bdb0b80 4 client.26536 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:58:29.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.329+0000 7f706bdb0b80 1 -- 172.21.15.78:0/1125571151 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56155feb0700 con 0x56155ff87f40 2024-03-14T09:58:29.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.330+0000 7f7055800700 1 --2- 172.21.15.78:0/1125571151 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56155ff87f40 0x56155ff8a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.330+0000 7f7037400700 10 client.26536 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:58:29.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.330+0000 7f7055800700 1 --2- 172.21.15.78:0/1125571151 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56155ff87f40 0x56155ff8a330 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=540c810f31a999cb server_cookie=f8d2c95849fa4e73 in_seq=0 out_seq=0 2024-03-14T09:58:29.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.331+0000 7f7037400700 1 -- 172.21.15.78:0/1125571151 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56155feb0700 con 0x56155ff87f40 2024-03-14T09:58:29.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.331+0000 7f7037400700 10 client.26536 handle_command_reply: tid=0 2024-03-14T09:58:29.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.331+0000 7f706bdb0b80 1 client.26536 shutdown 2024-03-14T09:58:29.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.331+0000 7f706bdb0b80 1 -- 172.21.15.78:0/1125571151 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff65fa0 msgr2=0x56155ff6d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.331+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/1125571151 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff65fa0 0x56155ff6d250 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x56155ff665b0 tx=0x7f703c095ee0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.331+0000 7f706bdb0b80 1 -- 172.21.15.78:0/1125571151 shutdown_connections 2024-03-14T09:58:29.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.331+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/1125571151 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56155ff87f40 0x56155ff8a330 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.331+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/1125571151 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff6fec0 0x56155ff6d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.331+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/1125571151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff65680 0x56155ff6cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.331+0000 7f7037400700 0 client.26536 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:58:29.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.331+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/1125571151 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff65fa0 0x56155ff6d250 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.331+0000 7f706bdb0b80 1 -- 172.21.15.78:0/1125571151 >> 172.21.15.78:0/1125571151 conn(0x56155fedb220 msgr2=0x56155ff60690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:29.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.331+0000 7f706bdb0b80 1 -- 172.21.15.78:0/1125571151 shutdown_connections 2024-03-14T09:58:29.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.332+0000 7f706bdb0b80 1 -- 172.21.15.78:0/1125571151 wait complete. 2024-03-14T09:58:29.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.332+0000 7f706bdb0b80 20 client.26536 trim_cache size 0 max 16384 2024-03-14T09:58:29.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.348+0000 7f706bdb0b80 1 Processor -- start 2024-03-14T09:58:29.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.348+0000 7f706bdb0b80 1 -- start start 2024-03-14T09:58:29.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.348+0000 7f706bdb0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff65680 0x56155ff60c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.348+0000 7f706bdb0b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff6fec0 0x56155ff7f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.348+0000 7f706bdb0b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff5c680 0x56155ff69f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.348+0000 7f706bdb0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56155feb0700 con 0x56155ff65680 2024-03-14T09:58:29.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.348+0000 7f706bdb0b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56155ff6e2c0 con 0x56155ff6fec0 2024-03-14T09:58:29.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.348+0000 7f706bdb0b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56155ff842d0 con 0x56155ff5c680 2024-03-14T09:58:29.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.348+0000 7f7054e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff6fec0 0x56155ff7f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.349+0000 7f7055800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff65680 0x56155ff60c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.349+0000 7f7054e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff6fec0 0x56155ff7f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:38824/0 (socket says 172.21.15.78:38824) 2024-03-14T09:58:29.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.349+0000 7f7054e00700 1 -- 172.21.15.78:0/928791289 learned_addr learned my addr 172.21.15.78:0/928791289 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:29.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.349+0000 7f7056200700 1 --2- 172.21.15.78:0/928791289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff5c680 0x56155ff69f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.349+0000 7f7054e00700 1 -- 172.21.15.78:0/928791289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff5c680 msgr2=0x56155ff69f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:29.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.349+0000 7f7054e00700 1 --2- 172.21.15.78:0/928791289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff5c680 0x56155ff69f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.349+0000 7f7054e00700 1 -- 172.21.15.78:0/928791289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff65680 msgr2=0x56155ff60c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:29.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.349+0000 7f7054e00700 1 --2- 172.21.15.78:0/928791289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff65680 0x56155ff60c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.349+0000 7f7054e00700 1 -- 172.21.15.78:0/928791289 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56155fe7dea0 con 0x56155ff6fec0 2024-03-14T09:58:29.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.349+0000 7f7054e00700 1 --2- 172.21.15.78:0/928791289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff6fec0 0x56155ff7f410 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f703c0960a0 tx=0x7f703c099810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9faabaa904e120d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:29.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.349+0000 7f7043400700 1 -- 172.21.15.78:0/928791289 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c09d9d0 con 0x56155ff6fec0 2024-03-14T09:58:29.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.349+0000 7f7043400700 1 -- 172.21.15.78:0/928791289 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f703c099970 con 0x56155ff6fec0 2024-03-14T09:58:29.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.349+0000 7f7043400700 1 -- 172.21.15.78:0/928791289 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c01fe90 con 0x56155ff6fec0 2024-03-14T09:58:29.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.350+0000 7f706bdb0b80 1 -- 172.21.15.78:0/928791289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff6fec0 msgr2=0x56155ff7f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.350+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/928791289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff6fec0 0x56155ff7f410 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f703c0960a0 tx=0x7f703c099810 comp rx=0 tx=0).stop 2024-03-14T09:58:29.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.350+0000 7f706bdb0b80 1 -- 172.21.15.78:0/928791289 shutdown_connections 2024-03-14T09:58:29.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.350+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/928791289 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff5c680 0x56155ff69f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.350+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/928791289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff65680 0x56155ff60c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.350+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/928791289 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff6fec0 0x56155ff7f410 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.350+0000 7f706bdb0b80 1 -- 172.21.15.78:0/928791289 >> 172.21.15.78:0/928791289 conn(0x56155fedb220 msgr2=0x56155ff6b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:29.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.350+0000 7f706bdb0b80 1 -- 172.21.15.78:0/928791289 shutdown_connections 2024-03-14T09:58:29.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.350+0000 7f706bdb0b80 1 -- 172.21.15.78:0/928791289 wait complete. 2024-03-14T09:58:29.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.351+0000 7f706bdb0b80 1 -- start start 2024-03-14T09:58:29.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.351+0000 7f706bdb0b80 1 Processor -- start 2024-03-14T09:58:29.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.351+0000 7f706bdb0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff5c680 0x56155ff7ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.351+0000 7f706bdb0b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff65680 0x56155ff7f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f706bdb0b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff6fec0 0x56155ff7d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f706bdb0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56155ff7d790 con 0x56155ff5c680 2024-03-14T09:58:29.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f706bdb0b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56155ff7d900 con 0x56155ff65680 2024-03-14T09:58:29.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f706bdb0b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56155ff7dac0 con 0x56155ff6fec0 2024-03-14T09:58:29.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f706bdb0b80 10 client.0 _ll_register_callbacks cb 0x56155fcbf3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:58:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f7054e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff65680 0x56155ff7f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f7055800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff5c680 0x56155ff7ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f7054e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff65680 0x56155ff7f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:38826/0 (socket says 172.21.15.78:38826) 2024-03-14T09:58:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f7054e00700 1 -- 172.21.15.78:0/1429310529 learned_addr learned my addr 172.21.15.78:0/1429310529 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f7055800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff5c680 0x56155ff7ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51192/0 (socket says 172.21.15.78:51192) 2024-03-14T09:58:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f7056200700 1 --2- 172.21.15.78:0/1429310529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff6fec0 0x56155ff7d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f7054e00700 1 -- 172.21.15.78:0/1429310529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff6fec0 msgr2=0x56155ff7d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f7054e00700 1 --2- 172.21.15.78:0/1429310529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff6fec0 0x56155ff7d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f7054e00700 1 -- 172.21.15.78:0/1429310529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff5c680 msgr2=0x56155ff7ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f7054e00700 1 --2- 172.21.15.78:0/1429310529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff5c680 0x56155ff7ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f7054e00700 1 -- 172.21.15.78:0/1429310529 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56155fce5b60 con 0x56155ff65680 2024-03-14T09:58:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f7054e00700 1 --2- 172.21.15.78:0/1429310529 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff65680 0x56155ff7f1c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f703c097ef0 tx=0x7f703c003790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f5d0ee28f869392 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.352+0000 7f7037400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:58:29.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.353+0000 7f7037400700 1 -- 172.21.15.78:0/1429310529 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703c09a9e0 con 0x56155ff65680 2024-03-14T09:58:29.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.353+0000 7f7037400700 1 -- 172.21.15.78:0/1429310529 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f703c0b9070 con 0x56155ff65680 2024-03-14T09:58:29.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.353+0000 7f706bdb0b80 1 -- 172.21.15.78:0/1429310529 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f703c099ba0 con 0x56155ff65680 2024-03-14T09:58:29.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.353+0000 7f7037400700 1 -- 172.21.15.78:0/1429310529 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f703c09a530 con 0x56155ff65680 2024-03-14T09:58:29.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.353+0000 7f706bdb0b80 10 client.26555 fetch_fsmap learned FSMap version 25 2024-03-14T09:58:29.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.353+0000 7f706bdb0b80 1 -- 172.21.15.78:0/1429310529 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56155fd3dcd0 con 0x56155ff65680 2024-03-14T09:58:29.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.353+0000 7f7037400700 1 -- 172.21.15.78:0/1429310529 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f703c099ba0 con 0x56155ff65680 2024-03-14T09:58:29.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.353+0000 7f7037400700 1 -- 172.21.15.78:0/1429310529 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f703c099f20 con 0x56155ff65680 2024-03-14T09:58:29.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.353+0000 7f706bdb0b80 10 client.26555 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:58:29.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.353+0000 7f706bdb0b80 10 client.26555 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:58:29.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.353+0000 7f706bdb0b80 20 client.26555 populate_metadata read hostname 'smithi078' 2024-03-14T09:58:29.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.353+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/1429310529 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56155ffda6e0 0x56155ffdcad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:29.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.353+0000 7f706bdb0b80 4 client.26555 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:58:29.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.353+0000 7f706bdb0b80 1 -- 172.21.15.78:0/1429310529 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56155fd06cf0 con 0x56155ffda6e0 2024-03-14T09:58:29.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.354+0000 7f7055800700 1 --2- 172.21.15.78:0/1429310529 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56155ffda6e0 0x56155ffdcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:29.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.354+0000 7f7055800700 1 --2- 172.21.15.78:0/1429310529 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56155ffda6e0 0x56155ffdcad0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cc8b843086ef4 server_cookie=782cfd0f4281d87f in_seq=0 out_seq=0 2024-03-14T09:58:29.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.354+0000 7f7037400700 10 client.26555 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:58:29.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.354+0000 7f7037400700 1 -- 172.21.15.78:0/1429310529 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56155fd06cf0 con 0x56155ffda6e0 2024-03-14T09:58:29.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.354+0000 7f7037400700 10 client.26555 handle_command_reply: tid=0 2024-03-14T09:58:29.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.355+0000 7f706bdb0b80 1 client.26555 shutdown 2024-03-14T09:58:29.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.355+0000 7f706bdb0b80 1 -- 172.21.15.78:0/1429310529 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff65680 msgr2=0x56155ff7f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.355+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/1429310529 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff65680 0x56155ff7f1c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f703c097ef0 tx=0x7f703c003790 comp rx=0 tx=0).stop 2024-03-14T09:58:29.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.355+0000 7f706bdb0b80 1 -- 172.21.15.78:0/1429310529 shutdown_connections 2024-03-14T09:58:29.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.355+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/1429310529 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56155ff6fec0 0x56155ff7d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.355+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/1429310529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56155ff5c680 0x56155ff7ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.355+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/1429310529 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56155ffda6e0 0x56155ffdcad0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.355+0000 7f706bdb0b80 1 --2- 172.21.15.78:0/1429310529 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56155ff65680 0x56155ff7f1c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.355+0000 7f7037400700 0 client.26555 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:58:29.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.355+0000 7f706bdb0b80 1 -- 172.21.15.78:0/1429310529 >> 172.21.15.78:0/1429310529 conn(0x56155fedb220 msgr2=0x56155ff64df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:29.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.355+0000 7f706bdb0b80 1 -- 172.21.15.78:0/1429310529 shutdown_connections 2024-03-14T09:58:29.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.355+0000 7f706bdb0b80 1 -- 172.21.15.78:0/1429310529 wait complete. 2024-03-14T09:58:29.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.355+0000 7f706bdb0b80 20 client.26555 trim_cache size 0 max 16384 2024-03-14T09:58:29.365 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:58:29.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.356+0000 7f7058200700 1 -- 172.21.15.78:0/1337808868 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f70380a0c00 msgr2=0x7f70380a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.356+0000 7f7058200700 1 --2- 172.21.15.78:0/1337808868 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f70380a0c00 0x7f70380a30c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7044000c00 tx=0x7f70440054d0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.356+0000 7f7058200700 1 -- 172.21.15.78:0/1337808868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7050106060 msgr2=0x7f705019fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:29.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.356+0000 7f7058200700 1 --2- 172.21.15.78:0/1337808868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7050106060 0x7f705019fa80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f703c00ea30 tx=0x7f703c00ed40 comp rx=0 tx=0).stop 2024-03-14T09:58:29.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.356+0000 7f7058200700 1 -- 172.21.15.78:0/1337808868 shutdown_connections 2024-03-14T09:58:29.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.356+0000 7f7058200700 1 --2- 172.21.15.78:0/1337808868 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f70380a0c00 0x7f70380a30c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.356+0000 7f7058200700 1 --2- 172.21.15.78:0/1337808868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70500680b0 0x7f705019f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.356+0000 7f7058200700 1 --2- 172.21.15.78:0/1337808868 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70501996c0 0x7f7050199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.356+0000 7f7058200700 1 --2- 172.21.15.78:0/1337808868 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7050106060 0x7f705019fa80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:29.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.356+0000 7f7058200700 1 -- 172.21.15.78:0/1337808868 >> 172.21.15.78:0/1337808868 conn(0x7f7050076d80 msgr2=0x7f705010c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:29.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.356+0000 7f7058200700 1 -- 172.21.15.78:0/1337808868 shutdown_connections 2024-03-14T09:58:29.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:29.356+0000 7f7058200700 1 -- 172.21.15.78:0/1337808868 wait complete. 2024-03-14T09:58:29.965 INFO:tasks.workunit.client.0.smithi078.stdout:50000 trunc from 0x3a0a6d to 0x51d279 2024-03-14T09:58:29.965 INFO:tasks.workunit.client.0.smithi078.stdout:All 50000 operations completed A-OK! 2024-03-14T09:58:29.970 INFO:tasks.workunit.client.0.smithi078.stderr:+ ./fsx -z 100MB -N 50000 -p 10000 -l 104857600 2024-03-14T09:58:29.972 INFO:tasks.workunit.client.0.smithi078.stdout:Seed set to 1 2024-03-14T09:58:30.377 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T09:58:30.378 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T09:58:30.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.579+0000 7f1633600700 1 -- 172.21.15.78:0/2825081090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f162c10d4f0 msgr2=0x7f162c10f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.579+0000 7f1633600700 1 --2- 172.21.15.78:0/2825081090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f162c10d4f0 0x7f162c10f960 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f162400b0a0 tx=0x7f162400b3b0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.579+0000 7f1633600700 1 -- 172.21.15.78:0/2825081090 shutdown_connections 2024-03-14T09:58:30.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.579+0000 7f1633600700 1 --2- 172.21.15.78:0/2825081090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f162c0731e0 0x7f162c0735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.579+0000 7f1633600700 1 --2- 172.21.15.78:0/2825081090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f162c10d4f0 0x7f162c10f960 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.579+0000 7f1633600700 1 --2- 172.21.15.78:0/2825081090 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f162c073b00 0x7f162c10cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.579+0000 7f1633600700 1 -- 172.21.15.78:0/2825081090 >> 172.21.15.78:0/2825081090 conn(0x7f162c0fc7a0 msgr2=0x7f162c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:30.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.579+0000 7f1633600700 1 -- 172.21.15.78:0/2825081090 shutdown_connections 2024-03-14T09:58:30.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.579+0000 7f1633600700 1 -- 172.21.15.78:0/2825081090 wait complete. 2024-03-14T09:58:30.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.580+0000 7f1633600700 1 Processor -- start 2024-03-14T09:58:30.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.581+0000 7f1633600700 1 -- start start 2024-03-14T09:58:30.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.581+0000 7f1633600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f162c0731e0 0x7f162c19f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.581+0000 7f1633600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f162c073b00 0x7f162c19faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.581+0000 7f1633600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f162c10d4f0 0x7f162c199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.581+0000 7f1633600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162c199d90 con 0x7f162c10d4f0 2024-03-14T09:58:30.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.581+0000 7f1633600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162c199f00 con 0x7f162c0731e0 2024-03-14T09:58:30.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.581+0000 7f1633600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162c19a0c0 con 0x7f162c073b00 2024-03-14T09:58:30.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.581+0000 7f1631600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f162c10d4f0 0x7f162c199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.582+0000 7f1630c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f162c0731e0 0x7f162c19f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.582+0000 7f162be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f162c073b00 0x7f162c19faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.582+0000 7f1630c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f162c0731e0 0x7f162c19f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:38852/0 (socket says 172.21.15.78:38852) 2024-03-14T09:58:30.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.582+0000 7f162be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f162c073b00 0x7f162c19faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59094/0 (socket says 172.21.15.78:59094) 2024-03-14T09:58:30.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.582+0000 7f1630c00700 1 -- 172.21.15.78:0/459828604 learned_addr learned my addr 172.21.15.78:0/459828604 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:30.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.582+0000 7f1631600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f162c10d4f0 0x7f162c199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51206/0 (socket says 172.21.15.78:51206) 2024-03-14T09:58:30.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.582+0000 7f1630c00700 1 -- 172.21.15.78:0/459828604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f162c073b00 msgr2=0x7f162c19faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.582+0000 7f1630c00700 1 --2- 172.21.15.78:0/459828604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f162c073b00 0x7f162c19faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.582+0000 7f1630c00700 1 -- 172.21.15.78:0/459828604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f162c10d4f0 msgr2=0x7f162c199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.582+0000 7f1630c00700 1 --2- 172.21.15.78:0/459828604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f162c10d4f0 0x7f162c199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.582+0000 7f1630c00700 1 -- 172.21.15.78:0/459828604 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1620009520 con 0x7f162c0731e0 2024-03-14T09:58:30.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.582+0000 7f1630c00700 1 --2- 172.21.15.78:0/459828604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f162c0731e0 0x7f162c19f5b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f162000eb30 tx=0x7f162000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:30.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.583+0000 7f1629600700 1 -- 172.21.15.78:0/459828604 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1620009880 con 0x7f162c0731e0 2024-03-14T09:58:30.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.583+0000 7f1629600700 1 -- 172.21.15.78:0/459828604 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16200044e0 con 0x7f162c0731e0 2024-03-14T09:58:30.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.583+0000 7f1633600700 1 -- 172.21.15.78:0/459828604 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1624009d00 con 0x7f162c0731e0 2024-03-14T09:58:30.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.583+0000 7f1629600700 1 -- 172.21.15.78:0/459828604 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1620003d80 con 0x7f162c0731e0 2024-03-14T09:58:30.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.583+0000 7f1633600700 1 -- 172.21.15.78:0/459828604 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f162c1a63c0 con 0x7f162c0731e0 2024-03-14T09:58:30.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.587+0000 7f1629600700 1 -- 172.21.15.78:0/459828604 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f1620003f20 con 0x7f162c0731e0 2024-03-14T09:58:30.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.588+0000 7f1647063b80 1 Processor -- start 2024-03-14T09:58:30.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.588+0000 7f1647063b80 1 -- start start 2024-03-14T09:58:30.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.588+0000 7f1647063b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c134700 0x55997c134ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.588+0000 7f1647063b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c135020 0x55997c13ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.588+0000 7f1647063b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c13ef40 0x55997c141330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.588+0000 7f1647063b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55997c08da10 con 0x55997c13ef40 2024-03-14T09:58:30.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.588+0000 7f1647063b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55997c08b3d0 con 0x55997c135020 2024-03-14T09:58:30.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.589+0000 7f1647063b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55997c086cc0 con 0x55997c134700 2024-03-14T09:58:30.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.589+0000 7f1630c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c135020 0x55997c13ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.589+0000 7f1630c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c135020 0x55997c13ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:38866/0 (socket says 172.21.15.78:38866) 2024-03-14T09:58:30.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.589+0000 7f1631600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c134700 0x55997c134ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.589+0000 7f1630c00700 1 -- 172.21.15.78:0/2011986278 learned_addr learned my addr 172.21.15.78:0/2011986278 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:30.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.589+0000 7f162be00700 1 --2- 172.21.15.78:0/2011986278 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c13ef40 0x55997c141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.589+0000 7f1631600700 1 -- 172.21.15.78:0/2011986278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c135020 msgr2=0x55997c13ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:30.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.589+0000 7f1631600700 1 --2- 172.21.15.78:0/2011986278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c135020 0x55997c13ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.589+0000 7f1631600700 1 -- 172.21.15.78:0/2011986278 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c13ef40 msgr2=0x55997c141330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:30.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.589+0000 7f1631600700 1 --2- 172.21.15.78:0/2011986278 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c13ef40 0x55997c141330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.589+0000 7f1629600700 1 --2- 172.21.15.78:0/459828604 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f16140a0c00 0x7f16140a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.589+0000 7f1631600700 1 -- 172.21.15.78:0/2011986278 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55997bf08430 con 0x55997c134700 2024-03-14T09:58:30.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.590+0000 7f1629600700 1 -- 172.21.15.78:0/459828604 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f1620014070 con 0x7f162c0731e0 2024-03-14T09:58:30.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.590+0000 7f1631600700 1 --2- 172.21.15.78:0/459828604 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f16140a0c00 0x7f16140a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.590+0000 7f1631600700 1 --2- 172.21.15.78:0/2011986278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c134700 0x55997c134ae0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1624009fd0 tx=0x7f1624009c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbb58b6a28d5a09d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:30.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.590+0000 7f161dc00700 1 -- 172.21.15.78:0/2011986278 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162400e040 con 0x55997c134700 2024-03-14T09:58:30.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.590+0000 7f161dc00700 1 -- 172.21.15.78:0/2011986278 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1624007560 con 0x55997c134700 2024-03-14T09:58:30.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.590+0000 7f161dc00700 1 -- 172.21.15.78:0/2011986278 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162401cc90 con 0x55997c134700 2024-03-14T09:58:30.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.591+0000 7f1647063b80 1 -- 172.21.15.78:0/2011986278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c134700 msgr2=0x55997c134ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.591+0000 7f1631600700 1 --2- 172.21.15.78:0/459828604 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f16140a0c00 0x7f16140a30c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f162400b780 tx=0x7f162401cf50 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:30.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.591+0000 7f1647063b80 1 --2- 172.21.15.78:0/2011986278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c134700 0x55997c134ae0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1624009fd0 tx=0x7f1624009c40 comp rx=0 tx=0).stop 2024-03-14T09:58:30.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.591+0000 7f1647063b80 1 -- 172.21.15.78:0/2011986278 shutdown_connections 2024-03-14T09:58:30.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.591+0000 7f1647063b80 1 --2- 172.21.15.78:0/2011986278 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c13ef40 0x55997c141330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.591+0000 7f1647063b80 1 --2- 172.21.15.78:0/2011986278 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c134700 0x55997c134ae0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.591+0000 7f1647063b80 1 --2- 172.21.15.78:0/2011986278 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c135020 0x55997c13ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.591+0000 7f1647063b80 1 -- 172.21.15.78:0/2011986278 >> 172.21.15.78:0/2011986278 conn(0x55997c0aa2a0 msgr2=0x55997c12fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:30.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.591+0000 7f1647063b80 1 -- 172.21.15.78:0/2011986278 shutdown_connections 2024-03-14T09:58:30.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.591+0000 7f1647063b80 1 -- 172.21.15.78:0/2011986278 wait complete. 2024-03-14T09:58:30.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.591+0000 7f1647063b80 1 -- start start 2024-03-14T09:58:30.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.592+0000 7f1647063b80 1 Processor -- start 2024-03-14T09:58:30.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.592+0000 7f1647063b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c134700 0x55997c13e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f1647063b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c135020 0x55997c13bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f1647063b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c13ef40 0x55997c13c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f1647063b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55997c022e00 con 0x55997c13ef40 2024-03-14T09:58:30.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f1647063b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55997c022aa0 con 0x55997c135020 2024-03-14T09:58:30.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f1647063b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55997bf0a720 con 0x55997c134700 2024-03-14T09:58:30.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f1647063b80 10 client.0 _ll_register_callbacks cb 0x55997be8e420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:58:30.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f1631600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c13ef40 0x55997c13c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f1630c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c134700 0x55997c13e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f1631600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c13ef40 0x55997c13c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51234/0 (socket says 172.21.15.78:51234) 2024-03-14T09:58:30.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f1631600700 1 -- 172.21.15.78:0/2535776806 learned_addr learned my addr 172.21.15.78:0/2535776806 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:30.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f162be00700 1 --2- 172.21.15.78:0/2535776806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c135020 0x55997c13bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f1631600700 1 -- 172.21.15.78:0/2535776806 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c134700 msgr2=0x55997c13e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:30.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f1631600700 1 --2- 172.21.15.78:0/2535776806 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c134700 0x55997c13e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f1631600700 1 -- 172.21.15.78:0/2535776806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c135020 msgr2=0x55997c13bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:30.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f1631600700 1 --2- 172.21.15.78:0/2535776806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c135020 0x55997c13bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f1631600700 1 -- 172.21.15.78:0/2535776806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f162c110620 con 0x55997c13ef40 2024-03-14T09:58:30.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f1630c00700 1 --2- 172.21.15.78:0/2535776806 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c134700 0x55997c13e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:58:30.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.593+0000 7f162be00700 1 --2- 172.21.15.78:0/2535776806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c135020 0x55997c13bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:58:30.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.594+0000 7f1631600700 1 --2- 172.21.15.78:0/2535776806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c13ef40 0x55997c13c010 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f1624009a20 tx=0x7f162401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4f80ba89dc6b1d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:30.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.594+0000 7f1612000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:58:30.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.594+0000 7f1612000700 1 -- 172.21.15.78:0/2535776806 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162400e040 con 0x55997c13ef40 2024-03-14T09:58:30.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.594+0000 7f1647063b80 1 -- 172.21.15.78:0/2535776806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55997c1277a0 con 0x55997c13ef40 2024-03-14T09:58:30.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.594+0000 7f1612000700 1 -- 172.21.15.78:0/2535776806 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f162401c560 con 0x55997c13ef40 2024-03-14T09:58:30.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.594+0000 7f1612000700 1 -- 172.21.15.78:0/2535776806 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16240078d0 con 0x55997c13ef40 2024-03-14T09:58:30.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.594+0000 7f1647063b80 10 client.16512 fetch_fsmap learned FSMap version 25 2024-03-14T09:58:30.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.594+0000 7f1647063b80 1 -- 172.21.15.78:0/2535776806 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55997beb4be0 con 0x55997c13ef40 2024-03-14T09:58:30.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.594+0000 7f1612000700 1 -- 172.21.15.78:0/2535776806 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55997c1277a0 con 0x55997c13ef40 2024-03-14T09:58:30.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.595+0000 7f1612000700 1 -- 172.21.15.78:0/2535776806 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f162402f070 con 0x55997c13ef40 2024-03-14T09:58:30.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.595+0000 7f1647063b80 10 client.16512 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:58:30.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.595+0000 7f1647063b80 10 client.16512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:58:30.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.595+0000 7f1647063b80 20 client.16512 populate_metadata read hostname 'smithi078' 2024-03-14T09:58:30.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.595+0000 7f1647063b80 1 --2- 172.21.15.78:0/2535776806 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55997c14e730 0x55997c150b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.595+0000 7f1647063b80 4 client.16512 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:58:30.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.595+0000 7f1647063b80 1 -- 172.21.15.78:0/2535776806 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55997c07f780 con 0x55997c14e730 2024-03-14T09:58:30.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.595+0000 7f1630c00700 1 --2- 172.21.15.78:0/2535776806 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55997c14e730 0x55997c150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.596+0000 7f1630c00700 1 --2- 172.21.15.78:0/2535776806 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55997c14e730 0x55997c150b50 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1cf038d54ac3993 server_cookie=5ad6cca72cd242b2 in_seq=0 out_seq=0 2024-03-14T09:58:30.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.596+0000 7f1612000700 10 client.16512 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:58:30.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.596+0000 7f1612000700 1 -- 172.21.15.78:0/2535776806 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55997c07f780 con 0x55997c14e730 2024-03-14T09:58:30.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.596+0000 7f1612000700 10 client.16512 handle_command_reply: tid=0 2024-03-14T09:58:30.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.596+0000 7f1647063b80 1 client.16512 shutdown 2024-03-14T09:58:30.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.596+0000 7f1647063b80 1 -- 172.21.15.78:0/2535776806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c13ef40 msgr2=0x55997c13c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.596+0000 7f1647063b80 1 --2- 172.21.15.78:0/2535776806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c13ef40 0x55997c13c010 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f1624009a20 tx=0x7f162401e000 comp rx=0 tx=0).stop 2024-03-14T09:58:30.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.597+0000 7f1647063b80 1 -- 172.21.15.78:0/2535776806 shutdown_connections 2024-03-14T09:58:30.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.597+0000 7f1647063b80 1 --2- 172.21.15.78:0/2535776806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c13ef40 0x55997c13c010 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.597+0000 7f1647063b80 1 --2- 172.21.15.78:0/2535776806 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c134700 0x55997c13e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.597+0000 7f1647063b80 1 --2- 172.21.15.78:0/2535776806 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55997c14e730 0x55997c150b50 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.597+0000 7f1647063b80 1 --2- 172.21.15.78:0/2535776806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c135020 0x55997c13bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.597+0000 7f1647063b80 1 -- 172.21.15.78:0/2535776806 >> 172.21.15.78:0/2535776806 conn(0x55997c0aa2a0 msgr2=0x55997c12f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:30.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.597+0000 7f1612000700 0 client.16512 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:58:30.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.597+0000 7f1647063b80 1 -- 172.21.15.78:0/2535776806 shutdown_connections 2024-03-14T09:58:30.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.597+0000 7f1647063b80 1 -- 172.21.15.78:0/2535776806 wait complete. 2024-03-14T09:58:30.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.597+0000 7f1647063b80 20 client.16512 trim_cache size 0 max 16384 2024-03-14T09:58:30.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.614+0000 7f1647063b80 1 Processor -- start 2024-03-14T09:58:30.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.614+0000 7f1647063b80 1 -- start start 2024-03-14T09:58:30.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.614+0000 7f1647063b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c134700 0x55997c14a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.614+0000 7f1647063b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c135020 0x55997c138db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.614+0000 7f1647063b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c13ef40 0x55997c1392f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.614+0000 7f1647063b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55997c130940 con 0x55997c135020 2024-03-14T09:58:30.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.614+0000 7f1647063b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55997c07f780 con 0x55997c134700 2024-03-14T09:58:30.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.614+0000 7f1647063b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55997c14b590 con 0x55997c13ef40 2024-03-14T09:58:30.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.614+0000 7f162be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c135020 0x55997c138db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.615+0000 7f1631600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c13ef40 0x55997c1392f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.615+0000 7f1630c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c134700 0x55997c14a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.615+0000 7f1631600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c13ef40 0x55997c1392f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59112/0 (socket says 172.21.15.78:59112) 2024-03-14T09:58:30.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.615+0000 7f1631600700 1 -- 172.21.15.78:0/1052054340 learned_addr learned my addr 172.21.15.78:0/1052054340 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:30.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.615+0000 7f162be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c135020 0x55997c138db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51236/0 (socket says 172.21.15.78:51236) 2024-03-14T09:58:30.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.615+0000 7f1631600700 1 -- 172.21.15.78:0/1052054340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c134700 msgr2=0x55997c14a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:30.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.615+0000 7f1631600700 1 --2- 172.21.15.78:0/1052054340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c134700 0x55997c14a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.615+0000 7f1631600700 1 -- 172.21.15.78:0/1052054340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c135020 msgr2=0x55997c138db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:30.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.615+0000 7f1631600700 1 --2- 172.21.15.78:0/1052054340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c135020 0x55997c138db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.615+0000 7f1631600700 1 -- 172.21.15.78:0/1052054340 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55997c04cf20 con 0x55997c13ef40 2024-03-14T09:58:30.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.615+0000 7f1631600700 1 --2- 172.21.15.78:0/1052054340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c13ef40 0x55997c1392f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1624002230 tx=0x7f1624009c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf37699541123cf server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:30.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.615+0000 7f161e600700 1 -- 172.21.15.78:0/1052054340 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162400e030 con 0x55997c13ef40 2024-03-14T09:58:30.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.615+0000 7f161e600700 1 -- 172.21.15.78:0/1052054340 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1624007e70 con 0x55997c13ef40 2024-03-14T09:58:30.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.615+0000 7f161e600700 1 -- 172.21.15.78:0/1052054340 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162403c670 con 0x55997c13ef40 2024-03-14T09:58:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.616+0000 7f1647063b80 1 -- 172.21.15.78:0/1052054340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c13ef40 msgr2=0x55997c1392f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.616+0000 7f1647063b80 1 --2- 172.21.15.78:0/1052054340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c13ef40 0x55997c1392f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1624002230 tx=0x7f1624009c40 comp rx=0 tx=0).stop 2024-03-14T09:58:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.616+0000 7f1647063b80 1 -- 172.21.15.78:0/1052054340 shutdown_connections 2024-03-14T09:58:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.616+0000 7f1647063b80 1 --2- 172.21.15.78:0/1052054340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c134700 0x55997c14a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.616+0000 7f1647063b80 1 --2- 172.21.15.78:0/1052054340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c13ef40 0x55997c1392f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.616+0000 7f1647063b80 1 --2- 172.21.15.78:0/1052054340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c135020 0x55997c138db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.616+0000 7f1647063b80 1 -- 172.21.15.78:0/1052054340 >> 172.21.15.78:0/1052054340 conn(0x55997c0aa2a0 msgr2=0x55997c14a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.616+0000 7f1647063b80 1 -- 172.21.15.78:0/1052054340 shutdown_connections 2024-03-14T09:58:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.616+0000 7f1647063b80 1 -- 172.21.15.78:0/1052054340 wait complete. 2024-03-14T09:58:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.617+0000 7f1647063b80 1 -- start start 2024-03-14T09:58:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.617+0000 7f1647063b80 1 Processor -- start 2024-03-14T09:58:30.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.617+0000 7f1647063b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c134700 0x55997c1448d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.617+0000 7f1647063b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c135020 0x55997c144e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.617+0000 7f1647063b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c13ef40 0x55997c1a0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.617+0000 7f1647063b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55997c145520 con 0x55997c134700 2024-03-14T09:58:30.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.617+0000 7f1647063b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55997c1a0760 con 0x55997c13ef40 2024-03-14T09:58:30.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.617+0000 7f1647063b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55997c1a08d0 con 0x55997c135020 2024-03-14T09:58:30.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.617+0000 7f1647063b80 10 client.0 _ll_register_callbacks cb 0x7f1618017bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:58:30.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.618+0000 7f1630c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c134700 0x55997c1448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.618+0000 7f1630c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c134700 0x55997c1448d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51248/0 (socket says 172.21.15.78:51248) 2024-03-14T09:58:30.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.618+0000 7f1630c00700 1 -- 172.21.15.78:0/2739259419 learned_addr learned my addr 172.21.15.78:0/2739259419 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:30.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.618+0000 7f162be00700 1 --2- 172.21.15.78:0/2739259419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c135020 0x55997c144e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.618+0000 7f1631600700 1 --2- 172.21.15.78:0/2739259419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c13ef40 0x55997c1a0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.618+0000 7f1630c00700 1 -- 172.21.15.78:0/2739259419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c135020 msgr2=0x55997c144e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:30.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.618+0000 7f1630c00700 1 --2- 172.21.15.78:0/2739259419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c135020 0x55997c144e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.618+0000 7f1630c00700 1 -- 172.21.15.78:0/2739259419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c13ef40 msgr2=0x55997c1a0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:30.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.618+0000 7f1630c00700 1 --2- 172.21.15.78:0/2739259419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c13ef40 0x55997c1a0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.618+0000 7f1630c00700 1 -- 172.21.15.78:0/2739259419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55997beb4be0 con 0x55997c134700 2024-03-14T09:58:30.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.618+0000 7f1630c00700 1 --2- 172.21.15.78:0/2739259419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c134700 0x55997c1448d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f16200036c0 tx=0x7f1620096170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a16b231a7faf35 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:30.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.618+0000 7f1612000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:58:30.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.619+0000 7f1612000700 1 -- 172.21.15.78:0/2739259419 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162000cbd0 con 0x55997c134700 2024-03-14T09:58:30.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.619+0000 7f1612000700 1 -- 172.21.15.78:0/2739259419 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f162009a410 con 0x55997c134700 2024-03-14T09:58:30.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.619+0000 7f1647063b80 1 -- 172.21.15.78:0/2739259419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1624007560 con 0x55997c134700 2024-03-14T09:58:30.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.619+0000 7f1612000700 1 -- 172.21.15.78:0/2739259419 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1620099d10 con 0x55997c134700 2024-03-14T09:58:30.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.619+0000 7f1647063b80 10 client.16518 fetch_fsmap learned FSMap version 25 2024-03-14T09:58:30.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.619+0000 7f1647063b80 1 -- 172.21.15.78:0/2739259419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55997bf0cd50 con 0x55997c134700 2024-03-14T09:58:30.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.619+0000 7f1612000700 1 -- 172.21.15.78:0/2739259419 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1624007560 con 0x55997c134700 2024-03-14T09:58:30.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.619+0000 7f1612000700 1 -- 172.21.15.78:0/2739259419 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f162009ad20 con 0x55997c134700 2024-03-14T09:58:30.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.619+0000 7f1647063b80 10 client.16518 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:58:30.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.619+0000 7f1647063b80 10 client.16518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:58:30.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.619+0000 7f1647063b80 20 client.16518 populate_metadata read hostname 'smithi078' 2024-03-14T09:58:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.619+0000 7f1647063b80 1 --2- 172.21.15.78:0/2739259419 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55997c1a2150 0x55997c1a4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.619+0000 7f1647063b80 4 client.16518 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T09:58:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.619+0000 7f1647063b80 1 -- 172.21.15.78:0/2739259419 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55997bed5d70 con 0x55997c1a2150 2024-03-14T09:58:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.620+0000 7f162be00700 1 --2- 172.21.15.78:0/2739259419 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55997c1a2150 0x55997c1a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.620+0000 7f162be00700 1 --2- 172.21.15.78:0/2739259419 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55997c1a2150 0x55997c1a4570 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=202af57f480b981 server_cookie=f36350e92d206a24 in_seq=0 out_seq=0 2024-03-14T09:58:30.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.620+0000 7f1612000700 10 client.16518 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:58:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.621+0000 7f1612000700 1 -- 172.21.15.78:0/2739259419 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55997bed5d70 con 0x55997c1a2150 2024-03-14T09:58:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.621+0000 7f1612000700 10 client.16518 handle_command_reply: tid=0 2024-03-14T09:58:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.621+0000 7f1647063b80 1 client.16518 shutdown 2024-03-14T09:58:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.621+0000 7f1647063b80 1 -- 172.21.15.78:0/2739259419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c134700 msgr2=0x55997c1448d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.621+0000 7f1647063b80 1 --2- 172.21.15.78:0/2739259419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c134700 0x55997c1448d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f16200036c0 tx=0x7f1620096170 comp rx=0 tx=0).stop 2024-03-14T09:58:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.622+0000 7f1647063b80 1 -- 172.21.15.78:0/2739259419 shutdown_connections 2024-03-14T09:58:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.622+0000 7f1647063b80 1 --2- 172.21.15.78:0/2739259419 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55997c1a2150 0x55997c1a4570 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.622+0000 7f1647063b80 1 --2- 172.21.15.78:0/2739259419 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55997c13ef40 0x55997c1a0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.622+0000 7f1647063b80 1 --2- 172.21.15.78:0/2739259419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55997c134700 0x55997c1448d0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.622+0000 7f1647063b80 1 --2- 172.21.15.78:0/2739259419 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55997c135020 0x55997c144e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.622+0000 7f1612000700 0 client.16518 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:58:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.622+0000 7f1647063b80 1 -- 172.21.15.78:0/2739259419 >> 172.21.15.78:0/2739259419 conn(0x55997c0aa2a0 msgr2=0x55997c14a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.622+0000 7f1647063b80 1 -- 172.21.15.78:0/2739259419 shutdown_connections 2024-03-14T09:58:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.622+0000 7f1647063b80 1 -- 172.21.15.78:0/2739259419 wait complete. 2024-03-14T09:58:30.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.622+0000 7f1647063b80 20 client.16518 trim_cache size 0 max 16384 2024-03-14T09:58:30.626 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:58:30.626 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T09:58:30.626 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "96b9e101-4be8-43c9-980a-d6f2c58464a5", 2024-03-14T09:58:30.627 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T09:58:30.627 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:58:30.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.623+0000 7f1633600700 1 -- 172.21.15.78:0/459828604 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f16140a0c00 msgr2=0x7f16140a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.623+0000 7f1633600700 1 --2- 172.21.15.78:0/459828604 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f16140a0c00 0x7f16140a30c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f162400b780 tx=0x7f162401cf50 comp rx=0 tx=0).stop 2024-03-14T09:58:30.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.623+0000 7f1633600700 1 -- 172.21.15.78:0/459828604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f162c0731e0 msgr2=0x7f162c19f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.623+0000 7f1633600700 1 --2- 172.21.15.78:0/459828604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f162c0731e0 0x7f162c19f5b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f162000eb30 tx=0x7f162000ee40 comp rx=0 tx=0).stop 2024-03-14T09:58:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.623+0000 7f1633600700 1 -- 172.21.15.78:0/459828604 shutdown_connections 2024-03-14T09:58:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.623+0000 7f1633600700 1 --2- 172.21.15.78:0/459828604 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f16140a0c00 0x7f16140a30c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.623+0000 7f1633600700 1 --2- 172.21.15.78:0/459828604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f162c0731e0 0x7f162c19f5b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.623+0000 7f1633600700 1 --2- 172.21.15.78:0/459828604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f162c10d4f0 0x7f162c199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.623+0000 7f1633600700 1 --2- 172.21.15.78:0/459828604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f162c073b00 0x7f162c19faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.623+0000 7f1633600700 1 -- 172.21.15.78:0/459828604 >> 172.21.15.78:0/459828604 conn(0x7f162c0fc7a0 msgr2=0x7f162c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.623+0000 7f1633600700 1 -- 172.21.15.78:0/459828604 shutdown_connections 2024-03-14T09:58:30.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.623+0000 7f1633600700 1 -- 172.21.15.78:0/459828604 wait complete. 2024-03-14T09:58:30.642 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:58:30.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.865+0000 7f7fbfe00700 1 -- 172.21.15.78:0/2267038271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fb8106060 msgr2=0x7f7fb810dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.865+0000 7f7fbfe00700 1 --2- 172.21.15.78:0/2267038271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fb8106060 0x7f7fb810dbe0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000b0a0 tx=0x7f7fb000b3b0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.865+0000 7f7fbfe00700 1 -- 172.21.15.78:0/2267038271 shutdown_connections 2024-03-14T09:58:30.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.865+0000 7f7fbfe00700 1 --2- 172.21.15.78:0/2267038271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fb8106060 0x7f7fb810dbe0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.865+0000 7f7fbfe00700 1 --2- 172.21.15.78:0/2267038271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fb80680b0 0x7f7fb8068490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.865+0000 7f7fbfe00700 1 --2- 172.21.15.78:0/2267038271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fb8068a60 0x7f7fb8105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.865+0000 7f7fbfe00700 1 -- 172.21.15.78:0/2267038271 >> 172.21.15.78:0/2267038271 conn(0x7f7fb8076d80 msgr2=0x7f7fb8077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:30.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.865+0000 7f7fbfe00700 1 -- 172.21.15.78:0/2267038271 shutdown_connections 2024-03-14T09:58:30.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.865+0000 7f7fbfe00700 1 -- 172.21.15.78:0/2267038271 wait complete. 2024-03-14T09:58:30.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.866+0000 7f7fbfe00700 1 Processor -- start 2024-03-14T09:58:30.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.866+0000 7f7fbfe00700 1 -- start start 2024-03-14T09:58:30.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.867+0000 7f7fbfe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fb80680b0 0x7f7fb819f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.867+0000 7f7fbfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fb8106060 0x7f7fb819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.867+0000 7f7fbfe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fb8199720 0x7f7fb8199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.867+0000 7f7fbfe00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb819a1a0 con 0x7f7fb8106060 2024-03-14T09:58:30.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.867+0000 7f7fbfe00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb819a310 con 0x7f7fb8199720 2024-03-14T09:58:30.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.867+0000 7f7fbfe00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb819a4d0 con 0x7f7fb80680b0 2024-03-14T09:58:30.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.868+0000 7f7fbca00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fb8106060 0x7f7fb819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.868+0000 7f7fbca00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fb8106060 0x7f7fb819fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51264/0 (socket says 172.21.15.78:51264) 2024-03-14T09:58:30.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.868+0000 7f7fbca00700 1 -- 172.21.15.78:0/3455197127 learned_addr learned my addr 172.21.15.78:0/3455197127 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:30.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.868+0000 7f7fbd400700 1 --2- 172.21.15.78:0/3455197127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fb80680b0 0x7f7fb819f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.868+0000 7f7fbde00700 1 --2- 172.21.15.78:0/3455197127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fb8199720 0x7f7fb8199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.868+0000 7f7fbca00700 1 -- 172.21.15.78:0/3455197127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fb80680b0 msgr2=0x7f7fb819f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.868+0000 7f7fbca00700 1 --2- 172.21.15.78:0/3455197127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fb80680b0 0x7f7fb819f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.868+0000 7f7fbca00700 1 -- 172.21.15.78:0/3455197127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fb8199720 msgr2=0x7f7fb8199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.868+0000 7f7fbca00700 1 --2- 172.21.15.78:0/3455197127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fb8199720 0x7f7fb8199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.868+0000 7f7fbca00700 1 -- 172.21.15.78:0/3455197127 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa4009520 con 0x7f7fb8106060 2024-03-14T09:58:30.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.868+0000 7f7fbde00700 1 --2- 172.21.15.78:0/3455197127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fb8199720 0x7f7fb8199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:58:30.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.868+0000 7f7fbca00700 1 --2- 172.21.15.78:0/3455197127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fb8106060 0x7f7fb819fae0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400ea30 tx=0x7f7fa400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:30.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.869+0000 7f7fb6000700 1 -- 172.21.15.78:0/3455197127 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4009820 con 0x7f7fb8106060 2024-03-14T09:58:30.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.869+0000 7f7fb6000700 1 -- 172.21.15.78:0/3455197127 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7fa40044e0 con 0x7f7fb8106060 2024-03-14T09:58:30.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.869+0000 7f7fbfe00700 1 -- 172.21.15.78:0/3455197127 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb0009d00 con 0x7f7fb8106060 2024-03-14T09:58:30.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.869+0000 7f7fb6000700 1 -- 172.21.15.78:0/3455197127 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4003dc0 con 0x7f7fb8106060 2024-03-14T09:58:30.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.869+0000 7f7fbfe00700 1 -- 172.21.15.78:0/3455197127 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb81a6810 con 0x7f7fb8106060 2024-03-14T09:58:30.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.873+0000 7f7fb6000700 1 -- 172.21.15.78:0/3455197127 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f7fa4004000 con 0x7f7fb8106060 2024-03-14T09:58:30.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.874+0000 7f7fd3853b80 1 Processor -- start 2024-03-14T09:58:30.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.875+0000 7f7fd3853b80 1 -- start start 2024-03-14T09:58:30.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.875+0000 7f7fd3853b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe346680 0x557efe346a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.875+0000 7f7fd3853b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 0x557efe350980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.875+0000 7f7fd3853b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe350ec0 0x557efe3532b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.875+0000 7f7fd3853b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557efe29f990 con 0x557efe346680 2024-03-14T09:58:30.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.875+0000 7f7fd3853b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557efe29d350 con 0x557efe350ec0 2024-03-14T09:58:30.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.875+0000 7f7fd3853b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557efe298c40 con 0x557efe346fa0 2024-03-14T09:58:30.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.875+0000 7f7fb6000700 1 --2- 172.21.15.78:0/3455197127 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7fa00a0c80 0x7f7fa00a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.875+0000 7f7fb6000700 1 -- 172.21.15.78:0/3455197127 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f7fa4014070 con 0x7f7fb8106060 2024-03-14T09:58:30.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.875+0000 7f7fbd400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 0x557efe350980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.876+0000 7f7fbd400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 0x557efe350980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59164/0 (socket says 172.21.15.78:59164) 2024-03-14T09:58:30.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.876+0000 7f7fbd400700 1 -- 172.21.15.78:0/385649886 learned_addr learned my addr 172.21.15.78:0/385649886 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:30.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.876+0000 7f7fbca00700 1 --2- 172.21.15.78:0/385649886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe350ec0 0x557efe3532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.876+0000 7f7fbd400700 1 -- 172.21.15.78:0/385649886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe350ec0 msgr2=0x557efe3532b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:30.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.876+0000 7f7fbd400700 1 --2- 172.21.15.78:0/385649886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe350ec0 0x557efe3532b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.876+0000 7f7fbd400700 1 -- 172.21.15.78:0/385649886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe346680 msgr2=0x557efe346a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:30.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.876+0000 7f7fbd400700 1 --2- 172.21.15.78:0/385649886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe346680 0x557efe346a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.876+0000 7f7fbd400700 1 -- 172.21.15.78:0/385649886 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557efe11a3b0 con 0x557efe346fa0 2024-03-14T09:58:30.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.876+0000 7f7fbca00700 1 --2- 172.21.15.78:0/385649886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe350ec0 0x557efe3532b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:58:30.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.876+0000 7f7fbde00700 1 --2- 172.21.15.78:0/3455197127 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7fa00a0c80 0x7f7fa00a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.876+0000 7f7fbd400700 1 --2- 172.21.15.78:0/385649886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 0x557efe350980 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009d80 tx=0x7f7fac011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc636763d2a7e3ee server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:30.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.877+0000 7f7faa600700 1 -- 172.21.15.78:0/385649886 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac004230 con 0x557efe346fa0 2024-03-14T09:58:30.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.877+0000 7f7faa600700 1 -- 172.21.15.78:0/385649886 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7fac004390 con 0x557efe346fa0 2024-03-14T09:58:30.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.877+0000 7f7faa600700 1 -- 172.21.15.78:0/385649886 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac019860 con 0x557efe346fa0 2024-03-14T09:58:30.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.877+0000 7f7fd3853b80 1 -- 172.21.15.78:0/385649886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 msgr2=0x557efe350980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.877+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/385649886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 0x557efe350980 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009d80 tx=0x7f7fac011720 comp rx=0 tx=0).stop 2024-03-14T09:58:30.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.877+0000 7f7fbde00700 1 --2- 172.21.15.78:0/3455197127 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7fa00a0c80 0x7f7fa00a3140 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000b780 tx=0x7f7fb0009780 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:30.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.878+0000 7f7fd3853b80 1 -- 172.21.15.78:0/385649886 shutdown_connections 2024-03-14T09:58:30.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.878+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/385649886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe350ec0 0x557efe3532b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.878+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/385649886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe346680 0x557efe346a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.878+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/385649886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 0x557efe350980 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.878+0000 7f7fd3853b80 1 -- 172.21.15.78:0/385649886 >> 172.21.15.78:0/385649886 conn(0x557efe2bc220 msgr2=0x557efe341aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:30.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.878+0000 7f7fd3853b80 1 -- 172.21.15.78:0/385649886 shutdown_connections 2024-03-14T09:58:30.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.878+0000 7f7fd3853b80 1 -- 172.21.15.78:0/385649886 wait complete. 2024-03-14T09:58:30.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.878+0000 7f7fd3853b80 1 -- start start 2024-03-14T09:58:30.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.879+0000 7f7fd3853b80 1 Processor -- start 2024-03-14T09:58:30.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.879+0000 7f7fd3853b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe346680 0x557efe3508a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.880+0000 7f7fd3853b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 0x557efe34db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.880+0000 7f7fd3853b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe350ec0 0x557efe34e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.880+0000 7f7fd3853b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557efe234d80 con 0x557efe350ec0 2024-03-14T09:58:30.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.880+0000 7f7fd3853b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557efe234a20 con 0x557efe346680 2024-03-14T09:58:30.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.880+0000 7f7fd3853b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557efe11c6a0 con 0x557efe346fa0 2024-03-14T09:58:30.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.880+0000 7f7fd3853b80 10 client.0 _ll_register_callbacks cb 0x557efe0a03a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:58:30.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.880+0000 7f7fbca00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 0x557efe34db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.880+0000 7f7fbde00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe350ec0 0x557efe34e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.880+0000 7f7fbde00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe350ec0 0x557efe34e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:51292/0 (socket says 172.21.15.78:51292) 2024-03-14T09:58:30.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.880+0000 7f7fbde00700 1 -- 172.21.15.78:0/1213456969 learned_addr learned my addr 172.21.15.78:0/1213456969 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:30.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.880+0000 7f7fbca00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 0x557efe34db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59166/0 (socket says 172.21.15.78:59166) 2024-03-14T09:58:30.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.880+0000 7f7fbd400700 1 --2- 172.21.15.78:0/1213456969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe346680 0x557efe3508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.881+0000 7f7fbca00700 1 -- 172.21.15.78:0/1213456969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe346680 msgr2=0x557efe3508a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:30.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.881+0000 7f7fbca00700 1 --2- 172.21.15.78:0/1213456969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe346680 0x557efe3508a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.881+0000 7f7fbca00700 1 -- 172.21.15.78:0/1213456969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe350ec0 msgr2=0x557efe34e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:30.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.881+0000 7f7fbca00700 1 --2- 172.21.15.78:0/1213456969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe350ec0 0x557efe34e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.881+0000 7f7fbca00700 1 -- 172.21.15.78:0/1213456969 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fac0099e0 con 0x557efe346fa0 2024-03-14T09:58:30.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.881+0000 7f7fbca00700 1 --2- 172.21.15.78:0/1213456969 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 0x557efe34db90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7fa40c9e40 tx=0x7f7fa4064970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eff6d4442bfbefef server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:30.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.881+0000 7f7f9ea00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:58:30.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.881+0000 7f7f9ea00700 1 -- 172.21.15.78:0/1213456969 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4099a40 con 0x557efe346fa0 2024-03-14T09:58:30.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.881+0000 7f7f9ea00700 1 -- 172.21.15.78:0/1213456969 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f7fa409d870 con 0x557efe346fa0 2024-03-14T09:58:30.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.882+0000 7f7fd3853b80 1 -- 172.21.15.78:0/1213456969 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557efe339720 con 0x557efe346fa0 2024-03-14T09:58:30.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.882+0000 7f7f9ea00700 1 -- 172.21.15.78:0/1213456969 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7fa40a1ec0 con 0x557efe346fa0 2024-03-14T09:58:30.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.882+0000 7f7fd3853b80 10 client.26572 fetch_fsmap learned FSMap version 25 2024-03-14T09:58:30.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.882+0000 7f7fd3853b80 1 -- 172.21.15.78:0/1213456969 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557efe0c6b60 con 0x557efe346fa0 2024-03-14T09:58:30.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.882+0000 7f7f9ea00700 1 -- 172.21.15.78:0/1213456969 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557efe339720 con 0x557efe346fa0 2024-03-14T09:58:30.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.882+0000 7f7f9ea00700 1 -- 172.21.15.78:0/1213456969 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7fa4063af0 con 0x557efe346fa0 2024-03-14T09:58:30.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.882+0000 7f7fd3853b80 10 client.26572 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:58:30.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.882+0000 7f7fd3853b80 10 client.26572 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:58:30.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.883+0000 7f7fd3853b80 20 client.26572 populate_metadata read hostname 'smithi078' 2024-03-14T09:58:30.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.883+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/1213456969 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557efe360a80 0x557efe362ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.883+0000 7f7fd3853b80 4 client.26572 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:58:30.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.883+0000 7f7fd3853b80 1 -- 172.21.15.78:0/1213456969 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557efe291700 con 0x557efe360a80 2024-03-14T09:58:30.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.883+0000 7f7fbd400700 1 --2- 172.21.15.78:0/1213456969 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557efe360a80 0x557efe362ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.883+0000 7f7fbd400700 1 --2- 172.21.15.78:0/1213456969 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557efe360a80 0x557efe362ea0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34dcc2ebbf8fd5be server_cookie=4bf51583dedc18ab in_seq=0 out_seq=0 2024-03-14T09:58:30.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.883+0000 7f7f9ea00700 10 client.26572 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:58:30.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.884+0000 7f7f9ea00700 1 -- 172.21.15.78:0/1213456969 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x557efe291700 con 0x557efe360a80 2024-03-14T09:58:30.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.884+0000 7f7f9ea00700 10 client.26572 handle_command_reply: tid=0 2024-03-14T09:58:30.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.884+0000 7f7fd3853b80 1 client.26572 shutdown 2024-03-14T09:58:30.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.884+0000 7f7fd3853b80 1 -- 172.21.15.78:0/1213456969 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 msgr2=0x557efe34db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.884+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/1213456969 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 0x557efe34db90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7fa40c9e40 tx=0x7f7fa4064970 comp rx=0 tx=0).stop 2024-03-14T09:58:30.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.885+0000 7f7fd3853b80 1 -- 172.21.15.78:0/1213456969 shutdown_connections 2024-03-14T09:58:30.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.885+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/1213456969 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557efe360a80 0x557efe362ea0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.885+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/1213456969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe346680 0x557efe3508a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.885+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/1213456969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe350ec0 0x557efe34e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.885+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/1213456969 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 0x557efe34db90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.885+0000 7f7fd3853b80 1 -- 172.21.15.78:0/1213456969 >> 172.21.15.78:0/1213456969 conn(0x557efe2bc220 msgr2=0x557efe341690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:30.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.885+0000 7f7f9ea00700 0 client.26572 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:58:30.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.885+0000 7f7fd3853b80 1 -- 172.21.15.78:0/1213456969 shutdown_connections 2024-03-14T09:58:30.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.885+0000 7f7fd3853b80 1 -- 172.21.15.78:0/1213456969 wait complete. 2024-03-14T09:58:30.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.885+0000 7f7fd3853b80 20 client.26572 trim_cache size 0 max 16384 2024-03-14T09:58:30.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.908+0000 7f7fd3853b80 1 Processor -- start 2024-03-14T09:58:30.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.908+0000 7f7fd3853b80 1 -- start start 2024-03-14T09:58:30.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.908+0000 7f7fd3853b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346680 0x557efe357b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.908+0000 7f7fd3853b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe346fa0 0x557efe3415b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.908+0000 7f7fd3853b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe350ec0 0x557efe348e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.908+0000 7f7fd3853b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557efe342e30 con 0x557efe346fa0 2024-03-14T09:58:30.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.908+0000 7f7fd3853b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557efe343060 con 0x557efe350ec0 2024-03-14T09:58:30.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.908+0000 7f7fd3853b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557efe360860 con 0x557efe346680 2024-03-14T09:58:30.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.909+0000 7f7fbd400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346680 0x557efe357b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.909+0000 7f7fbd400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346680 0x557efe357b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:59176/0 (socket says 172.21.15.78:59176) 2024-03-14T09:58:30.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.909+0000 7f7fbd400700 1 -- 172.21.15.78:0/2739004006 learned_addr learned my addr 172.21.15.78:0/2739004006 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:30.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.909+0000 7f7fbca00700 1 --2- 172.21.15.78:0/2739004006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe346fa0 0x557efe3415b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.909+0000 7f7fbde00700 1 --2- 172.21.15.78:0/2739004006 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe350ec0 0x557efe348e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.909+0000 7f7fbd400700 1 -- 172.21.15.78:0/2739004006 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe350ec0 msgr2=0x557efe348e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:30.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.909+0000 7f7fbd400700 1 --2- 172.21.15.78:0/2739004006 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe350ec0 0x557efe348e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.909+0000 7f7fbd400700 1 -- 172.21.15.78:0/2739004006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe346fa0 msgr2=0x557efe3415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:30.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.909+0000 7f7fbd400700 1 --2- 172.21.15.78:0/2739004006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe346fa0 0x557efe3415b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.909+0000 7f7fbd400700 1 -- 172.21.15.78:0/2739004006 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557efe25eea0 con 0x557efe346680 2024-03-14T09:58:30.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.909+0000 7f7fbca00700 1 --2- 172.21.15.78:0/2739004006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe346fa0 0x557efe3415b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:58:30.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.909+0000 7f7fbd400700 1 --2- 172.21.15.78:0/2739004006 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346680 0x557efe357b50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009d80 tx=0x7f7fac011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13973a7f0ca63b75 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:30.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.910+0000 7f7fab000700 1 -- 172.21.15.78:0/2739004006 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac005ce0 con 0x557efe346680 2024-03-14T09:58:30.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.910+0000 7f7fab000700 1 -- 172.21.15.78:0/2739004006 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7fac003b40 con 0x557efe346680 2024-03-14T09:58:30.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.910+0000 7f7fab000700 1 -- 172.21.15.78:0/2739004006 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac0052a0 con 0x557efe346680 2024-03-14T09:58:30.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.910+0000 7f7fd3853b80 1 -- 172.21.15.78:0/2739004006 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346680 msgr2=0x557efe357b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.910+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/2739004006 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346680 0x557efe357b50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009d80 tx=0x7f7fac011720 comp rx=0 tx=0).stop 2024-03-14T09:58:30.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.910+0000 7f7fd3853b80 1 -- 172.21.15.78:0/2739004006 shutdown_connections 2024-03-14T09:58:30.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.910+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/2739004006 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe350ec0 0x557efe348e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.910+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/2739004006 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346680 0x557efe357b50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.910+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/2739004006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe346fa0 0x557efe3415b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.910+0000 7f7fd3853b80 1 -- 172.21.15.78:0/2739004006 >> 172.21.15.78:0/2739004006 conn(0x557efe2bc220 msgr2=0x557efe340180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:30.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.911+0000 7f7fd3853b80 1 -- 172.21.15.78:0/2739004006 shutdown_connections 2024-03-14T09:58:30.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.911+0000 7f7fd3853b80 1 -- 172.21.15.78:0/2739004006 wait complete. 2024-03-14T09:58:30.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.911+0000 7f7fd3853b80 1 -- start start 2024-03-14T09:58:30.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.911+0000 7f7fd3853b80 1 Processor -- start 2024-03-14T09:58:30.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.912+0000 7f7fd3853b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe346680 0x557efe357690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.912+0000 7f7fd3853b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 0x557efe355b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.912+0000 7f7fd3853b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe350ec0 0x557efe356070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.912+0000 7f7fd3853b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557efe3565b0 con 0x557efe350ec0 2024-03-14T09:58:30.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.912+0000 7f7fd3853b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557efe356720 con 0x557efe346680 2024-03-14T09:58:30.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.912+0000 7f7fd3853b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557efe3568e0 con 0x557efe346fa0 2024-03-14T09:58:30.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.912+0000 7f7fd3853b80 10 client.0 _ll_register_callbacks cb 0x7f7fa40993a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:58:30.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.912+0000 7f7fbd400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe346680 0x557efe357690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.912+0000 7f7fbd400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe346680 0x557efe357690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:38950/0 (socket says 172.21.15.78:38950) 2024-03-14T09:58:30.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.912+0000 7f7fbd400700 1 -- 172.21.15.78:0/2175849271 learned_addr learned my addr 172.21.15.78:0/2175849271 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:58:30.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.912+0000 7f7fbca00700 1 --2- 172.21.15.78:0/2175849271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 0x557efe355b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.913+0000 7f7fbde00700 1 --2- 172.21.15.78:0/2175849271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe350ec0 0x557efe356070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.913+0000 7f7fbd400700 1 -- 172.21.15.78:0/2175849271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 msgr2=0x557efe355b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:30.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.913+0000 7f7fbd400700 1 --2- 172.21.15.78:0/2175849271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 0x557efe355b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.913+0000 7f7fbd400700 1 -- 172.21.15.78:0/2175849271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe350ec0 msgr2=0x557efe356070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:58:30.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.913+0000 7f7fbd400700 1 --2- 172.21.15.78:0/2175849271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe350ec0 0x557efe356070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.913+0000 7f7fbd400700 1 -- 172.21.15.78:0/2175849271 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557efe11a3b0 con 0x557efe346680 2024-03-14T09:58:30.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.913+0000 7f7fbde00700 1 --2- 172.21.15.78:0/2175849271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe350ec0 0x557efe356070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:58:30.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.913+0000 7f7fbd400700 1 --2- 172.21.15.78:0/2175849271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe346680 0x557efe357690 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7fac014040 tx=0x7f7fac003d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb3d67f36bf56a52 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:58:30.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.913+0000 7f7f9ea00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:58:30.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.913+0000 7f7f9ea00700 1 -- 172.21.15.78:0/2175849271 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac036040 con 0x557efe346680 2024-03-14T09:58:30.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.914+0000 7f7f9ea00700 1 -- 172.21.15.78:0/2175849271 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f7fac029070 con 0x557efe346680 2024-03-14T09:58:30.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.914+0000 7f7fd3853b80 1 -- 172.21.15.78:0/2175849271 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7fac004390 con 0x557efe346680 2024-03-14T09:58:30.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.914+0000 7f7f9ea00700 1 -- 172.21.15.78:0/2175849271 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7fac005d10 con 0x557efe346680 2024-03-14T09:58:30.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.914+0000 7f7fd3853b80 10 client.26591 fetch_fsmap learned FSMap version 25 2024-03-14T09:58:30.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.914+0000 7f7fd3853b80 1 -- 172.21.15.78:0/2175849271 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557efe11ecd0 con 0x557efe346680 2024-03-14T09:58:30.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.914+0000 7f7f9ea00700 1 -- 172.21.15.78:0/2175849271 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7fac004390 con 0x557efe346680 2024-03-14T09:58:30.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.915+0000 7f7f9ea00700 1 -- 172.21.15.78:0/2175849271 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7fac021040 con 0x557efe346680 2024-03-14T09:58:30.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.915+0000 7f7fd3853b80 10 client.26591 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:58:30.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.915+0000 7f7fd3853b80 10 client.26591 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:58:30.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.915+0000 7f7fd3853b80 20 client.26591 populate_metadata read hostname 'smithi078' 2024-03-14T09:58:30.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.915+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/2175849271 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557efe3bbcf0 0x557efe3be0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:58:30.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.915+0000 7f7fd3853b80 4 client.26591 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:58:30.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.915+0000 7f7fd3853b80 1 -- 172.21.15.78:0/2175849271 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557efe0e7cf0 con 0x557efe3bbcf0 2024-03-14T09:58:30.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.915+0000 7f7fbca00700 1 --2- 172.21.15.78:0/2175849271 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557efe3bbcf0 0x557efe3be0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:58:30.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.916+0000 7f7fbca00700 1 --2- 172.21.15.78:0/2175849271 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557efe3bbcf0 0x557efe3be0e0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e03ef6170a2d998 server_cookie=e96415aaae084452 in_seq=0 out_seq=0 2024-03-14T09:58:30.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.916+0000 7f7f9ea00700 10 client.26591 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:58:30.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.916+0000 7f7f9ea00700 1 -- 172.21.15.78:0/2175849271 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x557efe0e7cf0 con 0x557efe3bbcf0 2024-03-14T09:58:30.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.916+0000 7f7f9ea00700 10 client.26591 handle_command_reply: tid=0 2024-03-14T09:58:30.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.916+0000 7f7fd3853b80 1 client.26591 shutdown 2024-03-14T09:58:30.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.916+0000 7f7fd3853b80 1 -- 172.21.15.78:0/2175849271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe346680 msgr2=0x557efe357690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.916+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/2175849271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe346680 0x557efe357690 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7fac014040 tx=0x7f7fac003d20 comp rx=0 tx=0).stop 2024-03-14T09:58:30.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.916+0000 7f7fd3853b80 1 -- 172.21.15.78:0/2175849271 shutdown_connections 2024-03-14T09:58:30.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.916+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/2175849271 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557efe3bbcf0 0x557efe3be0e0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.916+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/2175849271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557efe346680 0x557efe357690 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.916+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/2175849271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557efe350ec0 0x557efe356070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.916+0000 7f7fd3853b80 1 --2- 172.21.15.78:0/2175849271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557efe346fa0 0x557efe355b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.916+0000 7f7f9ea00700 0 client.26591 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:58:30.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.916+0000 7f7fd3853b80 1 -- 172.21.15.78:0/2175849271 >> 172.21.15.78:0/2175849271 conn(0x557efe2bc220 msgr2=0x557efe3415b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:30.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.917+0000 7f7fd3853b80 1 -- 172.21.15.78:0/2175849271 shutdown_connections 2024-03-14T09:58:30.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.917+0000 7f7fd3853b80 1 -- 172.21.15.78:0/2175849271 wait complete. 2024-03-14T09:58:30.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.917+0000 7f7fd3853b80 20 client.26591 trim_cache size 0 max 16384 2024-03-14T09:58:30.920 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:58:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "scrub active (1151 inodes in the stack)", 2024-03-14T09:58:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T09:58:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: "96b9e101-4be8-43c9-980a-d6f2c58464a5": { 2024-03-14T09:58:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T09:58:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "96b9e101-4be8-43c9-980a-d6f2c58464a5", 2024-03-14T09:58:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T09:58:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:58:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:58:30.921 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:58:30.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.918+0000 7f7fbfe00700 1 -- 172.21.15.78:0/3455197127 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7fa00a0c80 msgr2=0x7f7fa00a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.918+0000 7f7fbfe00700 1 --2- 172.21.15.78:0/3455197127 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7fa00a0c80 0x7f7fa00a3140 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000b780 tx=0x7f7fb0009780 comp rx=0 tx=0).stop 2024-03-14T09:58:30.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.918+0000 7f7fbfe00700 1 -- 172.21.15.78:0/3455197127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fb8106060 msgr2=0x7f7fb819fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:58:30.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.918+0000 7f7fbfe00700 1 --2- 172.21.15.78:0/3455197127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fb8106060 0x7f7fb819fae0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400ea30 tx=0x7f7fa400ed40 comp rx=0 tx=0).stop 2024-03-14T09:58:30.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.918+0000 7f7fbfe00700 1 -- 172.21.15.78:0/3455197127 shutdown_connections 2024-03-14T09:58:30.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.918+0000 7f7fbfe00700 1 --2- 172.21.15.78:0/3455197127 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7fa00a0c80 0x7f7fa00a3140 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.918+0000 7f7fbfe00700 1 --2- 172.21.15.78:0/3455197127 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fb8199720 0x7f7fb8199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.918+0000 7f7fbfe00700 1 --2- 172.21.15.78:0/3455197127 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7fb80680b0 0x7f7fb819f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.918+0000 7f7fbfe00700 1 --2- 172.21.15.78:0/3455197127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fb8106060 0x7f7fb819fae0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:58:30.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.918+0000 7f7fbfe00700 1 -- 172.21.15.78:0/3455197127 >> 172.21.15.78:0/3455197127 conn(0x7f7fb8076d80 msgr2=0x7f7fb810c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:58:30.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.918+0000 7f7fbfe00700 1 -- 172.21.15.78:0/3455197127 shutdown_connections 2024-03-14T09:58:30.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:58:30.918+0000 7f7fbfe00700 1 -- 172.21.15.78:0/3455197127 wait complete. 2024-03-14T09:58:30.940 INFO:tasks.cephfs.filesystem:scrub status for tag:96b9e101-4be8-43c9-980a-d6f2c58464a5 - {'path': '/', 'tag': '96b9e101-4be8-43c9-980a-d6f2c58464a5', 'options': 'recursive,force'} 2024-03-14T09:58:31.337 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support fallocate mode 0, disabling! 2024-03-14T09:58:31.337 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support fallocate mode FALLOC_FL_KEEP_SIZE, disabling! 2024-03-14T09:58:31.339 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support fallocate mode FALLOC_FL_COLLAPSE_RANGE, disabling! 2024-03-14T09:58:31.339 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support fallocate mode FALLOC_FL_INSERT_RANGE, disabling! 2024-03-14T09:58:31.339 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support clone range, disabling! 2024-03-14T09:58:31.341 INFO:tasks.workunit.client.0.smithi078.stderr:main: filesystem does not support dedupe range, disabling! 2024-03-14T09:58:31.343 INFO:tasks.workunit.client.0.smithi078.stdout:skipping zero size read 2024-03-14T09:58:31.343 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0x7e0023 2024-03-14T09:58:31.350 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0xe91bf1 2024-03-14T09:58:31.359 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0x5a6de73 2024-03-14T09:58:31.409 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0x5bdf572 2024-03-14T09:58:31.413 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0x602fcad 2024-03-14T09:58:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:31 smithi152 ceph-mon[48279]: pgmap v133: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.9 MiB/s wr, 306 op/s 2024-03-14T09:58:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:31 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T09:58:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:31 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:58:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:31 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T09:58:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:31 smithi132 ceph-mon[47959]: pgmap v133: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.9 MiB/s wr, 306 op/s 2024-03-14T09:58:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:31 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T09:58:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:31 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:58:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:31 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T09:58:31.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:31 smithi078 ceph-mon[38143]: pgmap v133: 129 pgs: 129 active+clean; 161 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.9 MiB/s wr, 306 op/s 2024-03-14T09:58:31.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:31 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T09:58:31.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:31 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:58:31.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:31 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T09:58:31.649 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0x623914d 2024-03-14T09:58:31.792 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0x628f369 2024-03-14T09:58:31.894 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0x6328dee 2024-03-14T09:58:32.331 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0x635b95a 2024-03-14T09:58:32.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:32 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:58:32.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:32 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:58:32.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:32 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:58:32.870 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0x637b84e 2024-03-14T09:58:33.071 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0x63b535e 2024-03-14T09:58:33.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:33 smithi152 ceph-mon[48279]: pgmap v134: 129 pgs: 129 active+clean; 153 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 2.5 MiB/s wr, 424 op/s 2024-03-14T09:58:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:33 smithi132 ceph-mon[47959]: pgmap v134: 129 pgs: 129 active+clean; 153 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 2.5 MiB/s wr, 424 op/s 2024-03-14T09:58:33.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:33 smithi078 ceph-mon[38143]: pgmap v134: 129 pgs: 129 active+clean; 153 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 2.5 MiB/s wr, 424 op/s 2024-03-14T09:58:34.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:34 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T09:58:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:34 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T09:58:34.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:34 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T09:58:35.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:35 smithi152 ceph-mon[48279]: pgmap v135: 129 pgs: 129 active+clean; 153 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.6 MiB/s wr, 272 op/s 2024-03-14T09:58:35.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:35 smithi132 ceph-mon[47959]: pgmap v135: 129 pgs: 129 active+clean; 153 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.6 MiB/s wr, 272 op/s 2024-03-14T09:58:35.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:35 smithi078 ceph-mon[38143]: pgmap v135: 129 pgs: 129 active+clean; 153 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.6 MiB/s wr, 272 op/s 2024-03-14T09:58:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:37 smithi152 ceph-mon[48279]: pgmap v136: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.1 MiB/s wr, 435 op/s 2024-03-14T09:58:37.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:37 smithi132 ceph-mon[47959]: pgmap v136: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.1 MiB/s wr, 435 op/s 2024-03-14T09:58:37.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:37 smithi078 ceph-mon[38143]: pgmap v136: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.1 MiB/s wr, 435 op/s 2024-03-14T09:58:39.248 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:38 smithi152 ceph-mon[48279]: pgmap v137: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.1 MiB/s wr, 280 op/s 2024-03-14T09:58:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:38 smithi132 ceph-mon[47959]: pgmap v137: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.1 MiB/s wr, 280 op/s 2024-03-14T09:58:39.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:38 smithi078 ceph-mon[38143]: pgmap v137: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.1 MiB/s wr, 280 op/s 2024-03-14T09:58:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:41 smithi152 ceph-mon[48279]: pgmap v138: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.1 MiB/s wr, 280 op/s 2024-03-14T09:58:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:41 smithi132 ceph-mon[47959]: pgmap v138: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.1 MiB/s wr, 280 op/s 2024-03-14T09:58:41.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:41 smithi078 ceph-mon[38143]: pgmap v138: 129 pgs: 129 active+clean; 160 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.1 MiB/s wr, 280 op/s 2024-03-14T09:58:43.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:43 smithi152 ceph-mon[48279]: pgmap v139: 129 pgs: 129 active+clean; 179 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 488 op/s 2024-03-14T09:58:43.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:43 smithi132 ceph-mon[47959]: pgmap v139: 129 pgs: 129 active+clean; 179 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 488 op/s 2024-03-14T09:58:43.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:43 smithi078 ceph-mon[38143]: pgmap v139: 129 pgs: 129 active+clean; 179 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 488 op/s 2024-03-14T09:58:45.353 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0x63d8bf1 2024-03-14T09:58:45.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:45 smithi152 ceph-mon[48279]: pgmap v140: 129 pgs: 129 active+clean; 179 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1004 KiB/s wr, 370 op/s 2024-03-14T09:58:45.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:45 smithi132 ceph-mon[47959]: pgmap v140: 129 pgs: 129 active+clean; 179 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1004 KiB/s wr, 370 op/s 2024-03-14T09:58:45.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:45 smithi078 ceph-mon[38143]: pgmap v140: 129 pgs: 129 active+clean; 179 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1004 KiB/s wr, 370 op/s 2024-03-14T09:58:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:47 smithi152 ceph-mon[48279]: pgmap v141: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.5 MiB/s wr, 568 op/s 2024-03-14T09:58:47.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:47 smithi132 ceph-mon[47959]: pgmap v141: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.5 MiB/s wr, 568 op/s 2024-03-14T09:58:47.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:47 smithi078 ceph-mon[38143]: pgmap v141: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.5 MiB/s wr, 568 op/s 2024-03-14T09:58:49.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:48 smithi152 ceph-mon[48279]: pgmap v142: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 1.1 MiB/s wr, 405 op/s 2024-03-14T09:58:49.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:48 smithi132 ceph-mon[47959]: pgmap v142: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 1.1 MiB/s wr, 405 op/s 2024-03-14T09:58:49.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:48 smithi078 ceph-mon[38143]: pgmap v142: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 1.1 MiB/s wr, 405 op/s 2024-03-14T09:58:49.595 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0x63e65d6 2024-03-14T09:58:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:51 smithi152 ceph-mon[48279]: pgmap v143: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 1.1 MiB/s wr, 405 op/s 2024-03-14T09:58:51.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:51 smithi132 ceph-mon[47959]: pgmap v143: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 1.1 MiB/s wr, 405 op/s 2024-03-14T09:58:51.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:51 smithi078 ceph-mon[38143]: pgmap v143: 129 pgs: 129 active+clean; 174 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 1.1 MiB/s wr, 405 op/s 2024-03-14T09:58:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:53 smithi152 ceph-mon[48279]: pgmap v144: 129 pgs: 129 active+clean; 164 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 1.6 MiB/s wr, 603 op/s 2024-03-14T09:58:53.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:53 smithi132 ceph-mon[47959]: pgmap v144: 129 pgs: 129 active+clean; 164 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 1.6 MiB/s wr, 603 op/s 2024-03-14T09:58:53.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:53 smithi078 ceph-mon[38143]: pgmap v144: 129 pgs: 129 active+clean; 164 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 1.6 MiB/s wr, 603 op/s 2024-03-14T09:58:53.859 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0x63ea964 2024-03-14T09:58:55.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:55 smithi152 ceph-mon[48279]: pgmap v145: 129 pgs: 129 active+clean; 164 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 1.1 MiB/s wr, 395 op/s 2024-03-14T09:58:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:55 smithi132 ceph-mon[47959]: pgmap v145: 129 pgs: 129 active+clean; 164 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 1.1 MiB/s wr, 395 op/s 2024-03-14T09:58:55.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:55 smithi078 ceph-mon[38143]: pgmap v145: 129 pgs: 129 active+clean; 164 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 1.1 MiB/s wr, 395 op/s 2024-03-14T09:58:57.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:57 smithi152 ceph-mon[48279]: pgmap v146: 129 pgs: 129 active+clean; 174 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 1.6 MiB/s wr, 591 op/s 2024-03-14T09:58:57.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:57 smithi132 ceph-mon[47959]: pgmap v146: 129 pgs: 129 active+clean; 174 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 1.6 MiB/s wr, 591 op/s 2024-03-14T09:58:57.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:57 smithi078 ceph-mon[38143]: pgmap v146: 129 pgs: 129 active+clean; 174 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 1.6 MiB/s wr, 591 op/s 2024-03-14T09:58:59.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:58:58 smithi152 ceph-mon[48279]: pgmap v147: 129 pgs: 129 active+clean; 174 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 1.1 MiB/s wr, 393 op/s 2024-03-14T09:58:59.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:58:58 smithi132 ceph-mon[47959]: pgmap v147: 129 pgs: 129 active+clean; 174 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 1.1 MiB/s wr, 393 op/s 2024-03-14T09:58:59.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:58:58 smithi078 ceph-mon[38143]: pgmap v147: 129 pgs: 129 active+clean; 174 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 1.1 MiB/s wr, 393 op/s 2024-03-14T09:59:00.941 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:59:01.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.199+0000 7fceed400700 1 -- 172.21.15.78:0/2604633375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcee8075c00 msgr2=0x7fcee8111630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:01.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.199+0000 7fceed400700 1 --2- 172.21.15.78:0/2604633375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcee8075c00 0x7fcee8111630 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fcee000b0a0 tx=0x7fcee000b3b0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.200+0000 7fceed400700 1 -- 172.21.15.78:0/2604633375 shutdown_connections 2024-03-14T09:59:01.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.200+0000 7fceed400700 1 --2- 172.21.15.78:0/2604633375 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcee80752e0 0x7fcee80756c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.200+0000 7fceed400700 1 --2- 172.21.15.78:0/2604633375 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcee8111b70 0x7fcee8113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.200+0000 7fceed400700 1 --2- 172.21.15.78:0/2604633375 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcee8075c00 0x7fcee8111630 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.200+0000 7fceed400700 1 -- 172.21.15.78:0/2604633375 >> 172.21.15.78:0/2604633375 conn(0x7fcee80fe980 msgr2=0x7fcee8100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:01.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.200+0000 7fceed400700 1 -- 172.21.15.78:0/2604633375 shutdown_connections 2024-03-14T09:59:01.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.200+0000 7fceed400700 1 -- 172.21.15.78:0/2604633375 wait complete. 2024-03-14T09:59:01.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.202+0000 7fceed400700 1 Processor -- start 2024-03-14T09:59:01.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.202+0000 7fceed400700 1 -- start start 2024-03-14T09:59:01.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.202+0000 7fceed400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcee80752e0 0x7fcee81a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.202+0000 7fceed400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcee8075c00 0x7fcee81a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.202+0000 7fceed400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcee8111b70 0x7fcee819dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.202+0000 7fceed400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee819e120 con 0x7fcee8075c00 2024-03-14T09:59:01.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.202+0000 7fceed400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee819e290 con 0x7fcee8111b70 2024-03-14T09:59:01.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.202+0000 7fceed400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee819e450 con 0x7fcee80752e0 2024-03-14T09:59:01.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.202+0000 7fcee5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcee8075c00 0x7fcee81a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.203+0000 7fcee6800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcee80752e0 0x7fcee81a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.203+0000 7fcee5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcee8075c00 0x7fcee81a3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41258/0 (socket says 172.21.15.78:41258) 2024-03-14T09:59:01.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.203+0000 7fcee5e00700 1 -- 172.21.15.78:0/12028409 learned_addr learned my addr 172.21.15.78:0/12028409 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:01.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.203+0000 7fcee7200700 1 --2- 172.21.15.78:0/12028409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcee8111b70 0x7fcee819dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.203+0000 7fcee5e00700 1 -- 172.21.15.78:0/12028409 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcee80752e0 msgr2=0x7fcee81a3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:01.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.203+0000 7fcee5e00700 1 --2- 172.21.15.78:0/12028409 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcee80752e0 0x7fcee81a3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.203+0000 7fcee5e00700 1 -- 172.21.15.78:0/12028409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcee8111b70 msgr2=0x7fcee819dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:01.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.203+0000 7fcee5e00700 1 --2- 172.21.15.78:0/12028409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcee8111b70 0x7fcee819dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.203+0000 7fcee5e00700 1 -- 172.21.15.78:0/12028409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee0009d00 con 0x7fcee8075c00 2024-03-14T09:59:01.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.203+0000 7fcee7200700 1 --2- 172.21.15.78:0/12028409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcee8111b70 0x7fcee819dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:59:01.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.203+0000 7fcee6800700 1 --2- 172.21.15.78:0/12028409 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcee80752e0 0x7fcee81a3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:01.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.203+0000 7fcee5e00700 1 --2- 172.21.15.78:0/12028409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcee8075c00 0x7fcee81a3ed0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fcee0009bb0 tx=0x7fcee00077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:01.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.203+0000 7fcedf400700 1 -- 172.21.15.78:0/12028409 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee000e040 con 0x7fcee8075c00 2024-03-14T09:59:01.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.204+0000 7fcedf400700 1 -- 172.21.15.78:0/12028409 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcee0007a50 con 0x7fcee8075c00 2024-03-14T09:59:01.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.204+0000 7fcedf400700 1 -- 172.21.15.78:0/12028409 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee001d7f0 con 0x7fcee8075c00 2024-03-14T09:59:01.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.204+0000 7fceed400700 1 -- 172.21.15.78:0/12028409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee819e6d0 con 0x7fcee8075c00 2024-03-14T09:59:01.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.204+0000 7fceed400700 1 -- 172.21.15.78:0/12028409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee81aa980 con 0x7fcee8075c00 2024-03-14T09:59:01.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.207+0000 7fcedf400700 1 -- 172.21.15.78:0/12028409 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fcee0026400 con 0x7fcee8075c00 2024-03-14T09:59:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.208+0000 7fcf00e9db80 1 Processor -- start 2024-03-14T09:59:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.208+0000 7fcf00e9db80 1 -- start start 2024-03-14T09:59:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.209+0000 7fcedf400700 1 --2- 172.21.15.78:0/12028409 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcecc0a0cc0 0x7fcecc0a3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.209+0000 7fcf00e9db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194d2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.209+0000 7fcf00e9db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194d2fa0 0x556b194dc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.209+0000 7fcf00e9db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194dcec0 0x556b194df2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.209+0000 7fcf00e9db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b1942b990 con 0x556b194d2680 2024-03-14T09:59:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.209+0000 7fcf00e9db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b19429350 con 0x556b194dcec0 2024-03-14T09:59:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.209+0000 7fcf00e9db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b19424c40 con 0x556b194d2fa0 2024-03-14T09:59:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.209+0000 7fcedf400700 1 -- 172.21.15.78:0/12028409 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fcee00cd9c0 con 0x7fcee8075c00 2024-03-14T09:59:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.209+0000 7fcee5e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194d2fa0 0x556b194dc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.209+0000 7fcee5e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194d2fa0 0x556b194dc980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:42138/0 (socket says 172.21.15.78:42138) 2024-03-14T09:59:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.209+0000 7fcee5e00700 1 -- 172.21.15.78:0/59946717 learned_addr learned my addr 172.21.15.78:0/59946717 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.209+0000 7fcee7200700 1 --2- 172.21.15.78:0/59946717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194dcec0 0x556b194df2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.209+0000 7fcee6800700 1 --2- 172.21.15.78:0/59946717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194d2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.209+0000 7fcee7200700 1 --2- 172.21.15.78:0/12028409 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcecc0a0cc0 0x7fcecc0a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.210+0000 7fcee5e00700 1 -- 172.21.15.78:0/59946717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194dcec0 msgr2=0x556b194df2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.210+0000 7fcee5e00700 1 --2- 172.21.15.78:0/59946717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194dcec0 0x556b194df2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.210+0000 7fcee5e00700 1 -- 172.21.15.78:0/59946717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 msgr2=0x556b194d2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.210+0000 7fcee5e00700 1 --2- 172.21.15.78:0/59946717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194d2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.210+0000 7fcee5e00700 1 -- 172.21.15.78:0/59946717 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b192a63b0 con 0x556b194d2fa0 2024-03-14T09:59:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.210+0000 7fcee6800700 1 --2- 172.21.15.78:0/59946717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194d2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:59:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.210+0000 7fcee5e00700 1 --2- 172.21.15.78:0/59946717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194d2fa0 0x556b194dc980 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcee0096f80 tx=0x7fcee00057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf17af4dfb90fb76 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.210+0000 7fcee7200700 1 --2- 172.21.15.78:0/12028409 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcecc0a0cc0 0x7fcecc0a3180 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcee806a110 tx=0x7fced80047c0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.210+0000 7fced3400700 1 -- 172.21.15.78:0/59946717 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee009e8b0 con 0x556b194d2fa0 2024-03-14T09:59:01.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.210+0000 7fced3400700 1 -- 172.21.15.78:0/59946717 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcee00a2e70 con 0x556b194d2fa0 2024-03-14T09:59:01.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.210+0000 7fced3400700 1 -- 172.21.15.78:0/59946717 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee009aee0 con 0x556b194d2fa0 2024-03-14T09:59:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.211+0000 7fcf00e9db80 1 -- 172.21.15.78:0/59946717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194d2fa0 msgr2=0x556b194dc980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.211+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/59946717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194d2fa0 0x556b194dc980 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcee0096f80 tx=0x7fcee00057f0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.211+0000 7fcf00e9db80 1 -- 172.21.15.78:0/59946717 shutdown_connections 2024-03-14T09:59:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.211+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/59946717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194dcec0 0x556b194df2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.211+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/59946717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194d2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.211+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/59946717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194d2fa0 0x556b194dc980 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.211+0000 7fcf00e9db80 1 -- 172.21.15.78:0/59946717 >> 172.21.15.78:0/59946717 conn(0x556b19448220 msgr2=0x556b194cdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.211+0000 7fcf00e9db80 1 -- 172.21.15.78:0/59946717 shutdown_connections 2024-03-14T09:59:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.211+0000 7fcf00e9db80 1 -- 172.21.15.78:0/59946717 wait complete. 2024-03-14T09:59:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.211+0000 7fcf00e9db80 1 -- start start 2024-03-14T09:59:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.212+0000 7fcf00e9db80 1 Processor -- start 2024-03-14T09:59:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.212+0000 7fcf00e9db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194dc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.212+0000 7fcf00e9db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194d2fa0 0x556b194d9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.212+0000 7fcf00e9db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194dcec0 0x556b194da1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.212+0000 7fcf00e9db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b193c0d80 con 0x556b194d2680 2024-03-14T09:59:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.212+0000 7fcf00e9db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b193c0a20 con 0x556b194d2fa0 2024-03-14T09:59:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.212+0000 7fcf00e9db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b192a86a0 con 0x556b194dcec0 2024-03-14T09:59:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.212+0000 7fcf00e9db80 10 client.0 _ll_register_callbacks cb 0x556b1922c3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.212+0000 7fcee5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194d2fa0 0x556b194d9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.212+0000 7fcee5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194d2fa0 0x556b194d9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:51876/0 (socket says 172.21.15.78:51876) 2024-03-14T09:59:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.212+0000 7fcee5e00700 1 -- 172.21.15.78:0/4036598263 learned_addr learned my addr 172.21.15.78:0/4036598263 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.212+0000 7fcee6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194dc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.213+0000 7fcee7200700 1 --2- 172.21.15.78:0/4036598263 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194dcec0 0x556b194da1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.213+0000 7fcee5e00700 1 -- 172.21.15.78:0/4036598263 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194dcec0 msgr2=0x556b194da1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.213+0000 7fcee5e00700 1 --2- 172.21.15.78:0/4036598263 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194dcec0 0x556b194da1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.213+0000 7fcee5e00700 1 -- 172.21.15.78:0/4036598263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 msgr2=0x556b194dc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.213+0000 7fcee5e00700 1 --2- 172.21.15.78:0/4036598263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194dc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.213+0000 7fcee5e00700 1 -- 172.21.15.78:0/4036598263 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee81aa980 con 0x556b194d2fa0 2024-03-14T09:59:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.213+0000 7fcee7200700 1 --2- 172.21.15.78:0/4036598263 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194dcec0 0x556b194da1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:59:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.213+0000 7fcee6800700 1 --2- 172.21.15.78:0/4036598263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194dc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:59:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.213+0000 7fcee5e00700 1 --2- 172.21.15.78:0/4036598263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194d2fa0 0x556b194d9b90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcee00a2940 tx=0x7fcee009abf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9317ec41c2d27381 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.213+0000 7fcec7e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:59:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.213+0000 7fcec7e00700 1 -- 172.21.15.78:0/4036598263 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee009a180 con 0x556b194d2fa0 2024-03-14T09:59:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.213+0000 7fcec7e00700 1 -- 172.21.15.78:0/4036598263 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fcee00b3270 con 0x556b194d2fa0 2024-03-14T09:59:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.213+0000 7fcf00e9db80 1 -- 172.21.15.78:0/4036598263 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556b194c5720 con 0x556b194d2fa0 2024-03-14T09:59:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.213+0000 7fcec7e00700 1 -- 172.21.15.78:0/4036598263 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcee009ace0 con 0x556b194d2fa0 2024-03-14T09:59:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.213+0000 7fcf00e9db80 10 client.26600 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.213+0000 7fcf00e9db80 1 -- 172.21.15.78:0/4036598263 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556b19252b60 con 0x556b194d2fa0 2024-03-14T09:59:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.214+0000 7fcec7e00700 1 -- 172.21.15.78:0/4036598263 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcee007d020 con 0x556b194d2fa0 2024-03-14T09:59:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.214+0000 7fcec7e00700 1 -- 172.21.15.78:0/4036598263 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fcee00999e0 con 0x556b194d2fa0 2024-03-14T09:59:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.214+0000 7fcf00e9db80 10 client.26600 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.214+0000 7fcf00e9db80 10 client.26600 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:59:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.214+0000 7fcf00e9db80 20 client.26600 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.214+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/4036598263 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x556b194ec2d0 0x556b194ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.214+0000 7fcf00e9db80 4 client.26600 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:59:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.214+0000 7fcf00e9db80 1 -- 172.21.15.78:0/4036598263 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b1941d700 con 0x556b194ec2d0 2024-03-14T09:59:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.214+0000 7fcee6800700 1 --2- 172.21.15.78:0/4036598263 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x556b194ec2d0 0x556b194ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.215+0000 7fcec7e00700 10 client.26600 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.215+0000 7fcee6800700 1 --2- 172.21.15.78:0/4036598263 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x556b194ec2d0 0x556b194ee6c0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4a62d4b0139e34b server_cookie=85929ab2218c2202 in_seq=0 out_seq=0 2024-03-14T09:59:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.215+0000 7fcec7e00700 1 -- 172.21.15.78:0/4036598263 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x556b1941d700 con 0x556b194ec2d0 2024-03-14T09:59:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.215+0000 7fcec7e00700 10 client.26600 handle_command_reply: tid=0 2024-03-14T09:59:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.215+0000 7fcf00e9db80 1 client.26600 shutdown 2024-03-14T09:59:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.216+0000 7fcf00e9db80 1 -- 172.21.15.78:0/4036598263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194d2fa0 msgr2=0x556b194d9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.216+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/4036598263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194d2fa0 0x556b194d9b90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcee00a2940 tx=0x7fcee009abf0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.216+0000 7fcf00e9db80 1 -- 172.21.15.78:0/4036598263 shutdown_connections 2024-03-14T09:59:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.216+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/4036598263 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194dcec0 0x556b194da1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.216+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/4036598263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194dc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.216+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/4036598263 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x556b194ec2d0 0x556b194ee6c0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.216+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/4036598263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194d2fa0 0x556b194d9b90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.216+0000 7fcec7e00700 0 client.26600 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:01.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.216+0000 7fcf00e9db80 1 -- 172.21.15.78:0/4036598263 >> 172.21.15.78:0/4036598263 conn(0x556b19448220 msgr2=0x556b194cd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:01.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.216+0000 7fcf00e9db80 1 -- 172.21.15.78:0/4036598263 shutdown_connections 2024-03-14T09:59:01.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.216+0000 7fcf00e9db80 1 -- 172.21.15.78:0/4036598263 wait complete. 2024-03-14T09:59:01.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.216+0000 7fcf00e9db80 20 client.26600 trim_cache size 0 max 16384 2024-03-14T09:59:01.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.235+0000 7fcf00e9db80 1 Processor -- start 2024-03-14T09:59:01.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.235+0000 7fcf00e9db80 1 -- start start 2024-03-14T09:59:01.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.235+0000 7fcf00e9db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.235+0000 7fcf00e9db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194d2fa0 0x556b194d6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.235+0000 7fcf00e9db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194dcec0 0x556b194d7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.235+0000 7fcf00e9db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b194d3af0 con 0x556b194d2680 2024-03-14T09:59:01.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.235+0000 7fcf00e9db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b194cf060 con 0x556b194d2fa0 2024-03-14T09:59:01.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.235+0000 7fcf00e9db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b194f33a0 con 0x556b194dcec0 2024-03-14T09:59:01.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.236+0000 7fcee6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.236+0000 7fcee5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194d2fa0 0x556b194d6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.236+0000 7fcee5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194d2fa0 0x556b194d6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:51884/0 (socket says 172.21.15.78:51884) 2024-03-14T09:59:01.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.236+0000 7fcee5e00700 1 -- 172.21.15.78:0/136591101 learned_addr learned my addr 172.21.15.78:0/136591101 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:01.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.237+0000 7fcee7200700 1 --2- 172.21.15.78:0/136591101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194dcec0 0x556b194d7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.237+0000 7fcee6800700 1 -- 172.21.15.78:0/136591101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194dcec0 msgr2=0x556b194d7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:01.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.237+0000 7fcee6800700 1 --2- 172.21.15.78:0/136591101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194dcec0 0x556b194d7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.237+0000 7fcee6800700 1 -- 172.21.15.78:0/136591101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194d2fa0 msgr2=0x556b194d6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:01.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.237+0000 7fcee6800700 1 --2- 172.21.15.78:0/136591101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194d2fa0 0x556b194d6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.237+0000 7fcee6800700 1 -- 172.21.15.78:0/136591101 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b193eaea0 con 0x556b194d2680 2024-03-14T09:59:01.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.238+0000 7fcee7200700 1 --2- 172.21.15.78:0/136591101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194dcec0 0x556b194d7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:59:01.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.238+0000 7fcee5e00700 1 --2- 172.21.15.78:0/136591101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194d2fa0 0x556b194d6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:01.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.238+0000 7fcee6800700 1 --2- 172.21.15.78:0/136591101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194cdc30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x556b194eeca0 tx=0x7fced4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d943a6406e3437 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:01.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.239+0000 7fced3e00700 1 -- 172.21.15.78:0/136591101 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced401c070 con 0x556b194d2680 2024-03-14T09:59:01.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.239+0000 7fced3e00700 1 -- 172.21.15.78:0/136591101 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fced401ce90 con 0x556b194d2680 2024-03-14T09:59:01.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.239+0000 7fced3e00700 1 -- 172.21.15.78:0/136591101 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4024630 con 0x556b194d2680 2024-03-14T09:59:01.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.241+0000 7fcf00e9db80 1 -- 172.21.15.78:0/136591101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 msgr2=0x556b194cdc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:01.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.241+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/136591101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194cdc30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x556b194eeca0 tx=0x7fced4008040 comp rx=0 tx=0).stop 2024-03-14T09:59:01.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.241+0000 7fcf00e9db80 1 -- 172.21.15.78:0/136591101 shutdown_connections 2024-03-14T09:59:01.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.241+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/136591101 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194dcec0 0x556b194d7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.241+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/136591101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194cdc30 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.241+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/136591101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194d2fa0 0x556b194d6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.241+0000 7fcf00e9db80 1 -- 172.21.15.78:0/136591101 >> 172.21.15.78:0/136591101 conn(0x556b19448220 msgr2=0x556b194d8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:01.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.242+0000 7fcf00e9db80 1 -- 172.21.15.78:0/136591101 shutdown_connections 2024-03-14T09:59:01.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.242+0000 7fcf00e9db80 1 -- 172.21.15.78:0/136591101 wait complete. 2024-03-14T09:59:01.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.243+0000 7fcf00e9db80 1 -- start start 2024-03-14T09:59:01.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.243+0000 7fcf00e9db80 1 Processor -- start 2024-03-14T09:59:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.244+0000 7fcf00e9db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194e3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.244+0000 7fcf00e9db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194d2fa0 0x556b194e1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.244+0000 7fcf00e9db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194dcec0 0x556b194e1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.244+0000 7fcf00e9db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b194e24a0 con 0x556b194d2680 2024-03-14T09:59:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.244+0000 7fcf00e9db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b194e2610 con 0x556b194dcec0 2024-03-14T09:59:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.244+0000 7fcf00e9db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b194e27d0 con 0x556b194d2fa0 2024-03-14T09:59:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.244+0000 7fcf00e9db80 10 client.0 _ll_register_callbacks cb 0x7fced800cca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:01.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.244+0000 7fcee6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194e3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.244+0000 7fcee7200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194dcec0 0x556b194e1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.245+0000 7fcee5e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194d2fa0 0x556b194e1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.245+0000 7fcee7200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194dcec0 0x556b194e1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:51898/0 (socket says 172.21.15.78:51898) 2024-03-14T09:59:01.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.245+0000 7fcee7200700 1 -- 172.21.15.78:0/2843324415 learned_addr learned my addr 172.21.15.78:0/2843324415 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:01.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.244+0000 7fcee6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194e3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41290/0 (socket says 172.21.15.78:41290) 2024-03-14T09:59:01.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.245+0000 7fcee7200700 1 -- 172.21.15.78:0/2843324415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194d2fa0 msgr2=0x556b194e1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:01.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.245+0000 7fcee7200700 1 --2- 172.21.15.78:0/2843324415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194d2fa0 0x556b194e1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.245+0000 7fcee7200700 1 -- 172.21.15.78:0/2843324415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 msgr2=0x556b194e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:01.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.245+0000 7fcee7200700 1 --2- 172.21.15.78:0/2843324415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194e3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.245+0000 7fcee7200700 1 -- 172.21.15.78:0/2843324415 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b19252b60 con 0x556b194dcec0 2024-03-14T09:59:01.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.246+0000 7fcee5e00700 1 --2- 172.21.15.78:0/2843324415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194d2fa0 0x556b194e1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:01.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.246+0000 7fcee6800700 1 --2- 172.21.15.78:0/2843324415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194e3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:01.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.247+0000 7fcee7200700 1 --2- 172.21.15.78:0/2843324415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194dcec0 0x556b194e1f60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fced8011cd0 tx=0x7fced8004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1116a831089e0458 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:01.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.247+0000 7fcec7e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:59:01.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.247+0000 7fcec7e00700 1 -- 172.21.15.78:0/2843324415 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced800fd00 con 0x556b194dcec0 2024-03-14T09:59:01.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.247+0000 7fcec7e00700 1 -- 172.21.15.78:0/2843324415 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fced8026070 con 0x556b194dcec0 2024-03-14T09:59:01.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.248+0000 7fcf00e9db80 1 -- 172.21.15.78:0/2843324415 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcee00a2e70 con 0x556b194dcec0 2024-03-14T09:59:01.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.248+0000 7fcec7e00700 1 -- 172.21.15.78:0/2843324415 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fced800fe60 con 0x556b194dcec0 2024-03-14T09:59:01.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.248+0000 7fcf00e9db80 10 client.26606 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:01.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.248+0000 7fcf00e9db80 1 -- 172.21.15.78:0/2843324415 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7fced4009bb0 con 0x556b194dcec0 2024-03-14T09:59:01.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.248+0000 7fcec7e00700 1 -- 172.21.15.78:0/2843324415 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcee00a2e70 con 0x556b194dcec0 2024-03-14T09:59:01.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.249+0000 7fcec7e00700 1 -- 172.21.15.78:0/2843324415 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fced8035d00 con 0x556b194dcec0 2024-03-14T09:59:01.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.249+0000 7fcf00e9db80 10 client.26606 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:01.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.249+0000 7fcf00e9db80 10 client.26606 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:59:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.249+0000 7fcf00e9db80 20 client.26606 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.249+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/2843324415 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x556b195391f0 0x556b19539600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.249+0000 7fcf00e9db80 4 client.26606 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:59:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.249+0000 7fcf00e9db80 1 -- 172.21.15.78:0/2843324415 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556b19273cf0 con 0x556b195391f0 2024-03-14T09:59:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.249+0000 7fcee6800700 1 --2- 172.21.15.78:0/2843324415 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x556b195391f0 0x556b19539600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.250+0000 7fcec7e00700 10 client.26606 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:01.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.250+0000 7fcee6800700 1 --2- 172.21.15.78:0/2843324415 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x556b195391f0 0x556b19539600 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8917b8bec46a4e9 server_cookie=3af19278a7c901bd in_seq=0 out_seq=0 2024-03-14T09:59:01.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.250+0000 7fcec7e00700 1 -- 172.21.15.78:0/2843324415 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556b19273cf0 con 0x556b195391f0 2024-03-14T09:59:01.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.250+0000 7fcec7e00700 10 client.26606 handle_command_reply: tid=0 2024-03-14T09:59:01.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.250+0000 7fcf00e9db80 1 client.26606 shutdown 2024-03-14T09:59:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.250+0000 7fcf00e9db80 1 -- 172.21.15.78:0/2843324415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194dcec0 msgr2=0x556b194e1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.250+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/2843324415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194dcec0 0x556b194e1f60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fced8011cd0 tx=0x7fced8004750 comp rx=0 tx=0).stop 2024-03-14T09:59:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.251+0000 7fcf00e9db80 1 -- 172.21.15.78:0/2843324415 shutdown_connections 2024-03-14T09:59:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.251+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/2843324415 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x556b195391f0 0x556b19539600 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.251+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/2843324415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556b194dcec0 0x556b194e1f60 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.251+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/2843324415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b194d2680 0x556b194e3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.251+0000 7fcf00e9db80 1 --2- 172.21.15.78:0/2843324415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556b194d2fa0 0x556b194e1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.251+0000 7fcec7e00700 0 client.26606 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.251+0000 7fcf00e9db80 1 -- 172.21.15.78:0/2843324415 >> 172.21.15.78:0/2843324415 conn(0x556b19448220 msgr2=0x556b194d90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.251+0000 7fcf00e9db80 1 -- 172.21.15.78:0/2843324415 shutdown_connections 2024-03-14T09:59:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.251+0000 7fcf00e9db80 1 -- 172.21.15.78:0/2843324415 wait complete. 2024-03-14T09:59:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.251+0000 7fcf00e9db80 20 client.26606 trim_cache size 0 max 16384 2024-03-14T09:59:01.253 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:59:01.254 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T09:59:01.254 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T09:59:01.254 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:59:01.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.252+0000 7fceed400700 1 -- 172.21.15.78:0/12028409 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcecc0a0cc0 msgr2=0x7fcecc0a3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:01.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.252+0000 7fceed400700 1 --2- 172.21.15.78:0/12028409 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcecc0a0cc0 0x7fcecc0a3180 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcee806a110 tx=0x7fced80047c0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.252+0000 7fceed400700 1 -- 172.21.15.78:0/12028409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcee8075c00 msgr2=0x7fcee81a3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:01.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.252+0000 7fceed400700 1 --2- 172.21.15.78:0/12028409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcee8075c00 0x7fcee81a3ed0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fcee0009bb0 tx=0x7fcee00077f0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.252+0000 7fceed400700 1 -- 172.21.15.78:0/12028409 shutdown_connections 2024-03-14T09:59:01.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.252+0000 7fceed400700 1 --2- 172.21.15.78:0/12028409 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcecc0a0cc0 0x7fcecc0a3180 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.252+0000 7fceed400700 1 --2- 172.21.15.78:0/12028409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcee8111b70 0x7fcee819dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.252+0000 7fceed400700 1 --2- 172.21.15.78:0/12028409 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fcee80752e0 0x7fcee81a3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.252+0000 7fceed400700 1 --2- 172.21.15.78:0/12028409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcee8075c00 0x7fcee81a3ed0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.252+0000 7fceed400700 1 -- 172.21.15.78:0/12028409 >> 172.21.15.78:0/12028409 conn(0x7fcee80fe980 msgr2=0x7fcee81127e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:01.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.252+0000 7fceed400700 1 -- 172.21.15.78:0/12028409 shutdown_connections 2024-03-14T09:59:01.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.252+0000 7fceed400700 1 -- 172.21.15.78:0/12028409 wait complete. 2024-03-14T09:59:01.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:01 smithi078 ceph-mon[38143]: pgmap v148: 129 pgs: 129 active+clean; 174 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 1.1 MiB/s wr, 393 op/s 2024-03-14T09:59:01.274 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T09:59:01.274 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:59:01.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:01 smithi152 ceph-mon[48279]: pgmap v148: 129 pgs: 129 active+clean; 174 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 1.1 MiB/s wr, 393 op/s 2024-03-14T09:59:01.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.513+0000 7f9a2ee00700 1 -- 172.21.15.78:0/1927722588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a28073b00 msgr2=0x7f9a2810ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:01.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.513+0000 7f9a2ee00700 1 --2- 172.21.15.78:0/1927722588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a28073b00 0x7f9a2810ceb0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9a14009990 tx=0x7f9a14009ca0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.513+0000 7f9a2ee00700 1 -- 172.21.15.78:0/1927722588 shutdown_connections 2024-03-14T09:59:01.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.513+0000 7f9a2ee00700 1 --2- 172.21.15.78:0/1927722588 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a2810d3f0 0x7f9a2810f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.513+0000 7f9a2ee00700 1 --2- 172.21.15.78:0/1927722588 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a280731e0 0x7f9a280735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.513+0000 7f9a2ee00700 1 --2- 172.21.15.78:0/1927722588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a28073b00 0x7f9a2810ceb0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.513+0000 7f9a2ee00700 1 -- 172.21.15.78:0/1927722588 >> 172.21.15.78:0/1927722588 conn(0x7f9a280fc700 msgr2=0x7f9a280feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:01.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.513+0000 7f9a2ee00700 1 -- 172.21.15.78:0/1927722588 shutdown_connections 2024-03-14T09:59:01.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.513+0000 7f9a2ee00700 1 -- 172.21.15.78:0/1927722588 wait complete. 2024-03-14T09:59:01.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.514+0000 7f9a2ee00700 1 Processor -- start 2024-03-14T09:59:01.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.515+0000 7f9a2ee00700 1 -- start start 2024-03-14T09:59:01.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.515+0000 7f9a2ee00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a280731e0 0x7f9a28199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.515+0000 7f9a2ee00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a28073b00 0x7f9a28199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.515+0000 7f9a2ee00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a2810d3f0 0x7f9a2819e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.515+0000 7f9a2ee00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2819e650 con 0x7f9a2810d3f0 2024-03-14T09:59:01.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.515+0000 7f9a2ee00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2819e7c0 con 0x7f9a280731e0 2024-03-14T09:59:01.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.515+0000 7f9a2ee00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2819e980 con 0x7f9a28073b00 2024-03-14T09:59:01.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.515+0000 7f9a27e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a280731e0 0x7f9a28199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.515+0000 7f9a27e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a280731e0 0x7f9a28199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:51912/0 (socket says 172.21.15.78:51912) 2024-03-14T09:59:01.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.515+0000 7f9a27e00700 1 -- 172.21.15.78:0/1497166877 learned_addr learned my addr 172.21.15.78:0/1497166877 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:01.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.515+0000 7f9a27400700 1 --2- 172.21.15.78:0/1497166877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a28073b00 0x7f9a28199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.516+0000 7f9a2ce00700 1 --2- 172.21.15.78:0/1497166877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a2810d3f0 0x7f9a2819e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.516+0000 7f9a2ce00700 1 -- 172.21.15.78:0/1497166877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a28073b00 msgr2=0x7f9a28199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:01.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.516+0000 7f9a2ce00700 1 --2- 172.21.15.78:0/1497166877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a28073b00 0x7f9a28199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.516+0000 7f9a2ce00700 1 -- 172.21.15.78:0/1497166877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a280731e0 msgr2=0x7f9a28199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:01.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.517+0000 7f9a2ce00700 1 --2- 172.21.15.78:0/1497166877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a280731e0 0x7f9a28199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.517+0000 7f9a2ce00700 1 -- 172.21.15.78:0/1497166877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a140095f0 con 0x7f9a2810d3f0 2024-03-14T09:59:01.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.518+0000 7f9a2ce00700 1 --2- 172.21.15.78:0/1497166877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a2810d3f0 0x7f9a2819e010 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000efc0 tx=0x7f9a2000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:01.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.519+0000 7f9a24c00700 1 -- 172.21.15.78:0/1497166877 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a20004bb0 con 0x7f9a2810d3f0 2024-03-14T09:59:01.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.519+0000 7f9a24c00700 1 -- 172.21.15.78:0/1497166877 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a2000f040 con 0x7f9a2810d3f0 2024-03-14T09:59:01.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.519+0000 7f9a24c00700 1 -- 172.21.15.78:0/1497166877 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a2000cae0 con 0x7f9a2810d3f0 2024-03-14T09:59:01.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.519+0000 7f9a2ee00700 1 -- 172.21.15.78:0/1497166877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2819ec60 con 0x7f9a2810d3f0 2024-03-14T09:59:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:01 smithi132 ceph-mon[47959]: pgmap v148: 129 pgs: 129 active+clean; 174 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 1.1 MiB/s wr, 393 op/s 2024-03-14T09:59:01.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.519+0000 7f9a2ee00700 1 -- 172.21.15.78:0/1497166877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a28100c20 con 0x7f9a2810d3f0 2024-03-14T09:59:01.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.523+0000 7f9a2ee00700 1 -- 172.21.15.78:0/1497166877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a2819ee80 con 0x7f9a2810d3f0 2024-03-14T09:59:01.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.526+0000 7f9a24c00700 1 -- 172.21.15.78:0/1497166877 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f9a2000cc40 con 0x7f9a2810d3f0 2024-03-14T09:59:01.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.528+0000 7f9a24c00700 1 --2- 172.21.15.78:0/1497166877 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f9a0c0a0c80 0x7f9a0c0a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:01.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.529+0000 7f9a24c00700 1 -- 172.21.15.78:0/1497166877 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f9a200d41c0 con 0x7f9a2810d3f0 2024-03-14T09:59:01.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.531+0000 7f9a27e00700 1 --2- 172.21.15.78:0/1497166877 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f9a0c0a0c80 0x7f9a0c0a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:01.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.533+0000 7f9a27e00700 1 --2- 172.21.15.78:0/1497166877 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f9a0c0a0c80 0x7f9a0c0a3140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9a18009d20 tx=0x7f9a18009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:01.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.538+0000 7f9a24c00700 1 -- 172.21.15.78:0/1497166877 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9a2819ee80 con 0x7f9a2810d3f0 2024-03-14T09:59:01.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.860+0000 7f9a2ee00700 1 -- 172.21.15.78:0/1497166877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a2819edf0 con 0x7f9a2810d3f0 2024-03-14T09:59:01.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.860+0000 7f9a24c00700 1 -- 172.21.15.78:0/1497166877 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f9a2009d5f0 con 0x7f9a2810d3f0 2024-03-14T09:59:01.861 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:59:01.861 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:59:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.863+0000 7f9a2ee00700 1 -- 172.21.15.78:0/1497166877 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f9a0c0a0c80 msgr2=0x7f9a0c0a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.863+0000 7f9a2ee00700 1 --2- 172.21.15.78:0/1497166877 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f9a0c0a0c80 0x7f9a0c0a3140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9a18009d20 tx=0x7f9a18009260 comp rx=0 tx=0).stop 2024-03-14T09:59:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.863+0000 7f9a2ee00700 1 -- 172.21.15.78:0/1497166877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a2810d3f0 msgr2=0x7f9a2819e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.863+0000 7f9a2ee00700 1 --2- 172.21.15.78:0/1497166877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a2810d3f0 0x7f9a2819e010 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000efc0 tx=0x7f9a2000c3a0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.863+0000 7f9a2ee00700 1 -- 172.21.15.78:0/1497166877 shutdown_connections 2024-03-14T09:59:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.863+0000 7f9a2ee00700 1 --2- 172.21.15.78:0/1497166877 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f9a0c0a0c80 0x7f9a0c0a3140 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.863+0000 7f9a2ee00700 1 --2- 172.21.15.78:0/1497166877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9a280731e0 0x7f9a28199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.863+0000 7f9a2ee00700 1 --2- 172.21.15.78:0/1497166877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a2810d3f0 0x7f9a2819e010 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.863+0000 7f9a2ee00700 1 --2- 172.21.15.78:0/1497166877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9a28073b00 0x7f9a28199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:01.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.863+0000 7f9a2ee00700 1 -- 172.21.15.78:0/1497166877 >> 172.21.15.78:0/1497166877 conn(0x7f9a280fc700 msgr2=0x7f9a280fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:01.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.863+0000 7f9a2ee00700 1 -- 172.21.15.78:0/1497166877 shutdown_connections 2024-03-14T09:59:01.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:01.864+0000 7f9a2ee00700 1 -- 172.21.15.78:0/1497166877 wait complete. 2024-03-14T09:59:01.865 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T09:59:01.886 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T09:59:02.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.120+0000 7f7881a00700 1 -- 172.21.15.78:0/810562348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c106050 msgr2=0x7f787c10dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.120+0000 7f7881a00700 1 --2- 172.21.15.78:0/810562348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c106050 0x7f787c10dbd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f787400b0a0 tx=0x7f787400b3b0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.120+0000 7f7881a00700 1 -- 172.21.15.78:0/810562348 shutdown_connections 2024-03-14T09:59:02.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.120+0000 7f7881a00700 1 --2- 172.21.15.78:0/810562348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c106050 0x7f787c10dbd0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.120+0000 7f7881a00700 1 --2- 172.21.15.78:0/810562348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f787c100f30 0x7f787c101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.120+0000 7f7881a00700 1 --2- 172.21.15.78:0/810562348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f787c1018e0 0x7f787c105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.120+0000 7f7881a00700 1 -- 172.21.15.78:0/810562348 >> 172.21.15.78:0/810562348 conn(0x7f787c0fc7b0 msgr2=0x7f787c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.120+0000 7f7881a00700 1 -- 172.21.15.78:0/810562348 shutdown_connections 2024-03-14T09:59:02.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.120+0000 7f7881a00700 1 -- 172.21.15.78:0/810562348 wait complete. 2024-03-14T09:59:02.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.122+0000 7f7881a00700 1 Processor -- start 2024-03-14T09:59:02.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.122+0000 7f7881a00700 1 -- start start 2024-03-14T09:59:02.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.122+0000 7f7881a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c100f30 0x7f787c1996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.122+0000 7f7881a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f787c106050 0x7f787c199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.122+0000 7f7881a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f787c19dfa0 0x7f787c19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.122+0000 7f7881a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787c19ea20 con 0x7f787c19dfa0 2024-03-14T09:59:02.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.122+0000 7f7881a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787c19eb90 con 0x7f787c100f30 2024-03-14T09:59:02.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.122+0000 7f7881a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787c19ed50 con 0x7f787c106050 2024-03-14T09:59:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.122+0000 7f787b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f787c19dfa0 0x7f787c19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.123+0000 7f787a000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f787c106050 0x7f787c199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.123+0000 7f787b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f787c19dfa0 0x7f787c19e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41316/0 (socket says 172.21.15.78:41316) 2024-03-14T09:59:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.123+0000 7f787b400700 1 -- 172.21.15.78:0/2881227353 learned_addr learned my addr 172.21.15.78:0/2881227353 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.123+0000 7f787aa00700 1 --2- 172.21.15.78:0/2881227353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c100f30 0x7f787c1996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.123+0000 7f787a000700 1 -- 172.21.15.78:0/2881227353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c100f30 msgr2=0x7f787c1996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.123+0000 7f787a000700 1 --2- 172.21.15.78:0/2881227353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c100f30 0x7f787c1996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.123+0000 7f787a000700 1 -- 172.21.15.78:0/2881227353 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f787c19dfa0 msgr2=0x7f787c19e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.123+0000 7f787a000700 1 --2- 172.21.15.78:0/2881227353 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f787c19dfa0 0x7f787c19e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.123+0000 7f787a000700 1 -- 172.21.15.78:0/2881227353 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c009520 con 0x7f787c106050 2024-03-14T09:59:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.123+0000 7f787a000700 1 --2- 172.21.15.78:0/2881227353 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f787c106050 0x7f787c199c00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f786800e9c0 tx=0x7f786800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.123+0000 7f7873400700 1 -- 172.21.15.78:0/2881227353 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786800c9c0 con 0x7f787c106050 2024-03-14T09:59:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.123+0000 7f7873400700 1 -- 172.21.15.78:0/2881227353 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7868010430 con 0x7f787c106050 2024-03-14T09:59:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.123+0000 7f7881a00700 1 -- 172.21.15.78:0/2881227353 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7874009d00 con 0x7f787c106050 2024-03-14T09:59:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.123+0000 7f7873400700 1 -- 172.21.15.78:0/2881227353 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7868018680 con 0x7f787c106050 2024-03-14T09:59:02.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.124+0000 7f7881a00700 1 -- 172.21.15.78:0/2881227353 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f787c1a6650 con 0x7f787c106050 2024-03-14T09:59:02.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.126+0000 7f7873400700 1 -- 172.21.15.78:0/2881227353 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f786800cb20 con 0x7f787c106050 2024-03-14T09:59:02.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.127+0000 7f7873400700 1 --2- 172.21.15.78:0/2881227353 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f78600a0c80 0x7f78600a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.128+0000 7f7873400700 1 -- 172.21.15.78:0/2881227353 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f7868014070 con 0x7f787c106050 2024-03-14T09:59:02.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.128+0000 7f787aa00700 1 --2- 172.21.15.78:0/2881227353 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f78600a0c80 0x7f78600a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.128+0000 7f787aa00700 1 --2- 172.21.15.78:0/2881227353 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f78600a0c80 0x7f78600a3140 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f786c009df0 tx=0x7f786c009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.130+0000 7f7895471b80 1 Processor -- start 2024-03-14T09:59:02.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.130+0000 7f7895471b80 1 -- start start 2024-03-14T09:59:02.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.130+0000 7f7895471b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 0x561beda08a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.130+0000 7f7895471b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda08fa0 0x561beda12980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.131+0000 7f7895471b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda12ec0 0x561beda152b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.131+0000 7f7895471b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed961990 con 0x561beda08680 2024-03-14T09:59:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.131+0000 7f7895471b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed95f350 con 0x561beda12ec0 2024-03-14T09:59:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.131+0000 7f7895471b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed95ac40 con 0x561beda08fa0 2024-03-14T09:59:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.131+0000 7f787a000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda08fa0 0x561beda12980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.131+0000 7f787aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 0x561beda08a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.131+0000 7f787b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda12ec0 0x561beda152b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.131+0000 7f787a000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda08fa0 0x561beda12980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:42226/0 (socket says 172.21.15.78:42226) 2024-03-14T09:59:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.131+0000 7f787a000700 1 -- 172.21.15.78:0/4068910544 learned_addr learned my addr 172.21.15.78:0/4068910544 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.131+0000 7f787a000700 1 -- 172.21.15.78:0/4068910544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda12ec0 msgr2=0x561beda152b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.131+0000 7f787a000700 1 --2- 172.21.15.78:0/4068910544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda12ec0 0x561beda152b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.131+0000 7f787a000700 1 -- 172.21.15.78:0/4068910544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 msgr2=0x561beda08a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.131+0000 7f787a000700 1 --2- 172.21.15.78:0/4068910544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 0x561beda08a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.131+0000 7f787a000700 1 -- 172.21.15.78:0/4068910544 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561bed7dc3b0 con 0x561beda08fa0 2024-03-14T09:59:02.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.131+0000 7f787a000700 1 --2- 172.21.15.78:0/4068910544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda08fa0 0x561beda12980 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f78680a16e0 tx=0x7f78680993a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ebf3b07be7d9019 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.132+0000 7f7867e00700 1 -- 172.21.15.78:0/4068910544 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7868099790 con 0x561beda08fa0 2024-03-14T09:59:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.132+0000 7f7867e00700 1 -- 172.21.15.78:0/4068910544 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f786809d5c0 con 0x561beda08fa0 2024-03-14T09:59:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.132+0000 7f7867e00700 1 -- 172.21.15.78:0/4068910544 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7868010860 con 0x561beda08fa0 2024-03-14T09:59:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.133+0000 7f7895471b80 1 -- 172.21.15.78:0/4068910544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda08fa0 msgr2=0x561beda12980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.133+0000 7f7895471b80 1 --2- 172.21.15.78:0/4068910544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda08fa0 0x561beda12980 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f78680a16e0 tx=0x7f78680993a0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.133+0000 7f7895471b80 1 -- 172.21.15.78:0/4068910544 shutdown_connections 2024-03-14T09:59:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.133+0000 7f7895471b80 1 --2- 172.21.15.78:0/4068910544 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda12ec0 0x561beda152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.133+0000 7f7895471b80 1 --2- 172.21.15.78:0/4068910544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 0x561beda08a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.133+0000 7f7895471b80 1 --2- 172.21.15.78:0/4068910544 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda08fa0 0x561beda12980 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.133+0000 7f7895471b80 1 -- 172.21.15.78:0/4068910544 >> 172.21.15.78:0/4068910544 conn(0x561bed97e220 msgr2=0x561beda03aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.133+0000 7f7895471b80 1 -- 172.21.15.78:0/4068910544 shutdown_connections 2024-03-14T09:59:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.133+0000 7f7895471b80 1 -- 172.21.15.78:0/4068910544 wait complete. 2024-03-14T09:59:02.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.134+0000 7f7895471b80 1 -- start start 2024-03-14T09:59:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.134+0000 7f7895471b80 1 Processor -- start 2024-03-14T09:59:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.134+0000 7f7895471b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda08680 0x561beda128c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.135+0000 7f7895471b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08fa0 0x561beda0fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.135+0000 7f7895471b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda12ec0 0x561beda10230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.135+0000 7f7895471b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed8f6d80 con 0x561beda08fa0 2024-03-14T09:59:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.135+0000 7f7895471b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed8f6a20 con 0x561beda08680 2024-03-14T09:59:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.135+0000 7f7895471b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bed7de6a0 con 0x561beda12ec0 2024-03-14T09:59:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.135+0000 7f7895471b80 10 client.0 _ll_register_callbacks cb 0x561bed7623a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.135+0000 7f787a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08fa0 0x561beda0fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.135+0000 7f787b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda12ec0 0x561beda10230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.135+0000 7f787b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda12ec0 0x561beda10230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:42232/0 (socket says 172.21.15.78:42232) 2024-03-14T09:59:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.135+0000 7f787b400700 1 -- 172.21.15.78:0/882443485 learned_addr learned my addr 172.21.15.78:0/882443485 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.135+0000 7f787aa00700 1 --2- 172.21.15.78:0/882443485 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda08680 0x561beda128c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.135+0000 7f787b400700 1 -- 172.21.15.78:0/882443485 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda08680 msgr2=0x561beda128c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.135+0000 7f787b400700 1 --2- 172.21.15.78:0/882443485 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda08680 0x561beda128c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.135+0000 7f787b400700 1 -- 172.21.15.78:0/882443485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08fa0 msgr2=0x561beda0fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.135+0000 7f787b400700 1 --2- 172.21.15.78:0/882443485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08fa0 0x561beda0fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.135+0000 7f787b400700 1 -- 172.21.15.78:0/882443485 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f787c1a6650 con 0x561beda12ec0 2024-03-14T09:59:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.136+0000 7f787b400700 1 --2- 172.21.15.78:0/882443485 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda12ec0 0x561beda10230 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f787400bd50 tx=0x7f787400bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ce136e69834803d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.136+0000 7f7864c00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:59:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.136+0000 7f7864c00700 1 -- 172.21.15.78:0/882443485 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787400e040 con 0x561beda12ec0 2024-03-14T09:59:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.136+0000 7f7864c00700 1 -- 172.21.15.78:0/882443485 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f7874003e10 con 0x561beda12ec0 2024-03-14T09:59:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.136+0000 7f7895471b80 1 -- 172.21.15.78:0/882443485 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561bed9fb720 con 0x561beda12ec0 2024-03-14T09:59:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.136+0000 7f7864c00700 1 -- 172.21.15.78:0/882443485 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78740074e0 con 0x561beda12ec0 2024-03-14T09:59:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.136+0000 7f7895471b80 10 client.26608 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.136+0000 7f7895471b80 1 -- 172.21.15.78:0/882443485 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561bed788b60 con 0x561beda12ec0 2024-03-14T09:59:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.137+0000 7f7864c00700 1 -- 172.21.15.78:0/882443485 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f787402d020 con 0x561beda12ec0 2024-03-14T09:59:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.137+0000 7f7864c00700 1 -- 172.21.15.78:0/882443485 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7874023020 con 0x561beda12ec0 2024-03-14T09:59:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.137+0000 7f7895471b80 10 client.26608 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.137+0000 7f7895471b80 10 client.26608 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:59:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.137+0000 7f7895471b80 20 client.26608 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.137+0000 7f7895471b80 1 --2- 172.21.15.78:0/882443485 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x561beda22b40 0x561beda24f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.137+0000 7f7895471b80 4 client.26608 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:59:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.137+0000 7f7895471b80 1 -- 172.21.15.78:0/882443485 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561bed953700 con 0x561beda22b40 2024-03-14T09:59:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.137+0000 7f787aa00700 1 --2- 172.21.15.78:0/882443485 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x561beda22b40 0x561beda24f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.138+0000 7f7864c00700 10 client.26608 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:59:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.138+0000 7f787aa00700 1 --2- 172.21.15.78:0/882443485 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x561beda22b40 0x561beda24f60 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d93a7a83b79c6578 server_cookie=3d09ca682768bd9e in_seq=0 out_seq=0 2024-03-14T09:59:02.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.138+0000 7f7864c00700 1 -- 172.21.15.78:0/882443485 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x561bed953700 con 0x561beda22b40 2024-03-14T09:59:02.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.139+0000 7f7864c00700 10 client.26608 handle_command_reply: tid=0 2024-03-14T09:59:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.139+0000 7f7895471b80 1 client.26608 shutdown 2024-03-14T09:59:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.139+0000 7f7895471b80 1 -- 172.21.15.78:0/882443485 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda12ec0 msgr2=0x561beda10230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.139+0000 7f7895471b80 1 --2- 172.21.15.78:0/882443485 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda12ec0 0x561beda10230 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f787400bd50 tx=0x7f787400bd80 comp rx=0 tx=0).stop 2024-03-14T09:59:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.139+0000 7f7895471b80 1 -- 172.21.15.78:0/882443485 shutdown_connections 2024-03-14T09:59:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.139+0000 7f7895471b80 1 --2- 172.21.15.78:0/882443485 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x561beda22b40 0x561beda24f60 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.139+0000 7f7895471b80 1 --2- 172.21.15.78:0/882443485 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda08680 0x561beda128c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.139+0000 7f7895471b80 1 --2- 172.21.15.78:0/882443485 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda12ec0 0x561beda10230 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.139+0000 7f7895471b80 1 --2- 172.21.15.78:0/882443485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08fa0 0x561beda0fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.139+0000 7f7864c00700 0 client.26608 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:59:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.139+0000 7f7895471b80 1 -- 172.21.15.78:0/882443485 >> 172.21.15.78:0/882443485 conn(0x561bed97e220 msgr2=0x561beda03690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.139+0000 7f7895471b80 1 -- 172.21.15.78:0/882443485 shutdown_connections 2024-03-14T09:59:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.139+0000 7f7895471b80 1 -- 172.21.15.78:0/882443485 wait complete. 2024-03-14T09:59:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.139+0000 7f7895471b80 20 client.26608 trim_cache size 0 max 16384 2024-03-14T09:59:02.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.155+0000 7f7895471b80 1 Processor -- start 2024-03-14T09:59:02.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.155+0000 7f7895471b80 1 -- start start 2024-03-14T09:59:02.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.155+0000 7f7895471b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 0x561beda03c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.155+0000 7f7895471b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda08fa0 0x561beda19b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.155+0000 7f7895471b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda12ec0 0x561beda10190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.155+0000 7f7895471b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561beda0a390 con 0x561beda08680 2024-03-14T09:59:02.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.155+0000 7f7895471b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x561beda04770 con 0x561beda12ec0 2024-03-14T09:59:02.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.155+0000 7f7895471b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561beda02800 con 0x561beda08fa0 2024-03-14T09:59:02.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.156+0000 7f787aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 0x561beda03c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.156+0000 7f787aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 0x561beda03c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41346/0 (socket says 172.21.15.78:41346) 2024-03-14T09:59:02.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.156+0000 7f787a000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda08fa0 0x561beda19b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.156+0000 7f787aa00700 1 -- 172.21.15.78:0/1719127124 learned_addr learned my addr 172.21.15.78:0/1719127124 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:02.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.156+0000 7f787b400700 1 --2- 172.21.15.78:0/1719127124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda12ec0 0x561beda10190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.156+0000 7f787a000700 1 -- 172.21.15.78:0/1719127124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda12ec0 msgr2=0x561beda10190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.156+0000 7f787a000700 1 --2- 172.21.15.78:0/1719127124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda12ec0 0x561beda10190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.156+0000 7f787a000700 1 -- 172.21.15.78:0/1719127124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 msgr2=0x561beda03c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.156+0000 7f787a000700 1 --2- 172.21.15.78:0/1719127124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 0x561beda03c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.156+0000 7f787a000700 1 -- 172.21.15.78:0/1719127124 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561bed920ea0 con 0x561beda08fa0 2024-03-14T09:59:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.156+0000 7f787aa00700 1 --2- 172.21.15.78:0/1719127124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 0x561beda03c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.156+0000 7f787a000700 1 --2- 172.21.15.78:0/1719127124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda08fa0 0x561beda19b40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x561beda095b0 tx=0x7f7868010c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dcaec58939b53f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.157+0000 7f7870c00700 1 -- 172.21.15.78:0/1719127124 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78680978d0 con 0x561beda08fa0 2024-03-14T09:59:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.157+0000 7f7870c00700 1 -- 172.21.15.78:0/1719127124 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7868099790 con 0x561beda08fa0 2024-03-14T09:59:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.157+0000 7f7870c00700 1 -- 172.21.15.78:0/1719127124 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78680b9070 con 0x561beda08fa0 2024-03-14T09:59:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.158+0000 7f7895471b80 1 -- 172.21.15.78:0/1719127124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda08fa0 msgr2=0x561beda19b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.158+0000 7f7895471b80 1 --2- 172.21.15.78:0/1719127124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda08fa0 0x561beda19b40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x561beda095b0 tx=0x7f7868010c30 comp rx=0 tx=0).stop 2024-03-14T09:59:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.158+0000 7f7895471b80 1 -- 172.21.15.78:0/1719127124 shutdown_connections 2024-03-14T09:59:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.158+0000 7f7895471b80 1 --2- 172.21.15.78:0/1719127124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda12ec0 0x561beda10190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.158+0000 7f7895471b80 1 --2- 172.21.15.78:0/1719127124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 0x561beda03c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.158+0000 7f7895471b80 1 --2- 172.21.15.78:0/1719127124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda08fa0 0x561beda19b40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.158+0000 7f7895471b80 1 -- 172.21.15.78:0/1719127124 >> 172.21.15.78:0/1719127124 conn(0x561bed97e220 msgr2=0x561beda0e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.158+0000 7f7895471b80 1 -- 172.21.15.78:0/1719127124 shutdown_connections 2024-03-14T09:59:02.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.158+0000 7f7895471b80 1 -- 172.21.15.78:0/1719127124 wait complete. 2024-03-14T09:59:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.158+0000 7f7895471b80 1 -- start start 2024-03-14T09:59:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.159+0000 7f7895471b80 1 Processor -- start 2024-03-14T09:59:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.159+0000 7f7895471b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 0x561beda17d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.159+0000 7f7895471b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda08fa0 0x561beda18260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.159+0000 7f7895471b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda12ec0 0x561beda73660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.159+0000 7f7895471b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561beda73ba0 con 0x561beda08680 2024-03-14T09:59:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.159+0000 7f7895471b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x561beda73ce0 con 0x561beda08fa0 2024-03-14T09:59:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.159+0000 7f7895471b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561beda73e20 con 0x561beda12ec0 2024-03-14T09:59:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.159+0000 7f7895471b80 10 client.0 _ll_register_callbacks cb 0x7f7874007e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.160+0000 7f787aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 0x561beda17d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.160+0000 7f787b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda12ec0 0x561beda73660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.160+0000 7f787aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 0x561beda17d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41358/0 (socket says 172.21.15.78:41358) 2024-03-14T09:59:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.160+0000 7f787aa00700 1 -- 172.21.15.78:0/4060051719 learned_addr learned my addr 172.21.15.78:0/4060051719 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.160+0000 7f787a000700 1 --2- 172.21.15.78:0/4060051719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda08fa0 0x561beda18260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.160+0000 7f787b400700 1 -- 172.21.15.78:0/4060051719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda08fa0 msgr2=0x561beda18260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.160+0000 7f787b400700 1 --2- 172.21.15.78:0/4060051719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda08fa0 0x561beda18260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.160+0000 7f787b400700 1 -- 172.21.15.78:0/4060051719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 msgr2=0x561beda17d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.160+0000 7f787b400700 1 --2- 172.21.15.78:0/4060051719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 0x561beda17d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.160+0000 7f787b400700 1 -- 172.21.15.78:0/4060051719 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f786c017070 con 0x561beda12ec0 2024-03-14T09:59:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.160+0000 7f787aa00700 1 --2- 172.21.15.78:0/4060051719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 0x561beda17d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.160+0000 7f787b400700 1 --2- 172.21.15.78:0/4060051719 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda12ec0 0x561beda73660 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x561beda049e0 tx=0x7f7874005300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba2f4b3b852dfdde server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.160+0000 7f7864c00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:59:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.160+0000 7f7864c00700 1 -- 172.21.15.78:0/4060051719 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787400e040 con 0x561beda12ec0 2024-03-14T09:59:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.161+0000 7f7864c00700 1 -- 172.21.15.78:0/4060051719 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f787402e080 con 0x561beda12ec0 2024-03-14T09:59:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.161+0000 7f7895471b80 1 -- 172.21.15.78:0/4060051719 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f786809d5c0 con 0x561beda12ec0 2024-03-14T09:59:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.161+0000 7f7864c00700 1 -- 172.21.15.78:0/4060051719 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7874006160 con 0x561beda12ec0 2024-03-14T09:59:02.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.161+0000 7f7895471b80 10 client.26614 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:02.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.161+0000 7f7895471b80 1 -- 172.21.15.78:0/4060051719 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561bed7dc3b0 con 0x561beda12ec0 2024-03-14T09:59:02.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.161+0000 7f7864c00700 1 -- 172.21.15.78:0/4060051719 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f786809d5c0 con 0x561beda12ec0 2024-03-14T09:59:02.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.161+0000 7f7864c00700 1 -- 172.21.15.78:0/4060051719 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7874023070 con 0x561beda12ec0 2024-03-14T09:59:02.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.161+0000 7f7895471b80 10 client.26614 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:02.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.161+0000 7f7895471b80 10 client.26614 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:59:02.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.161+0000 7f7895471b80 20 client.26614 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:02.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.161+0000 7f7895471b80 1 --2- 172.21.15.78:0/4060051719 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x561beda75210 0x561beda77630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.161+0000 7f7895471b80 4 client.26614 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:59:02.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.161+0000 7f7895471b80 1 -- 172.21.15.78:0/4060051719 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561bed7a9cf0 con 0x561beda75210 2024-03-14T09:59:02.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.161+0000 7f787aa00700 1 --2- 172.21.15.78:0/4060051719 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x561beda75210 0x561beda77630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.162+0000 7f787aa00700 1 --2- 172.21.15.78:0/4060051719 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x561beda75210 0x561beda77630 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=683869f2fe8d10 server_cookie=d4e3a8ae094ab6 in_seq=0 out_seq=0 2024-03-14T09:59:02.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.162+0000 7f7864c00700 10 client.26614 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:59:02.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.162+0000 7f7864c00700 1 -- 172.21.15.78:0/4060051719 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561bed7a9cf0 con 0x561beda75210 2024-03-14T09:59:02.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.162+0000 7f7864c00700 10 client.26614 handle_command_reply: tid=0 2024-03-14T09:59:02.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.162+0000 7f7895471b80 1 client.26614 shutdown 2024-03-14T09:59:02.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.162+0000 7f7895471b80 1 -- 172.21.15.78:0/4060051719 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda12ec0 msgr2=0x561beda73660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.162+0000 7f7895471b80 1 --2- 172.21.15.78:0/4060051719 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda12ec0 0x561beda73660 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x561beda049e0 tx=0x7f7874005300 comp rx=0 tx=0).stop 2024-03-14T09:59:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.162+0000 7f7895471b80 1 -- 172.21.15.78:0/4060051719 shutdown_connections 2024-03-14T09:59:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.162+0000 7f7895471b80 1 --2- 172.21.15.78:0/4060051719 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x561beda75210 0x561beda77630 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.162+0000 7f7895471b80 1 --2- 172.21.15.78:0/4060051719 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561beda12ec0 0x561beda73660 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.162+0000 7f7864c00700 0 client.26614 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:59:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.162+0000 7f7895471b80 1 --2- 172.21.15.78:0/4060051719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561beda08680 0x561beda17d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.162+0000 7f7895471b80 1 --2- 172.21.15.78:0/4060051719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561beda08fa0 0x561beda18260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.162+0000 7f7895471b80 1 -- 172.21.15.78:0/4060051719 >> 172.21.15.78:0/4060051719 conn(0x561bed97e220 msgr2=0x561beda0e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.163+0000 7f7895471b80 1 -- 172.21.15.78:0/4060051719 shutdown_connections 2024-03-14T09:59:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.163+0000 7f7895471b80 1 -- 172.21.15.78:0/4060051719 wait complete. 2024-03-14T09:59:02.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.163+0000 7f7895471b80 20 client.26614 trim_cache size 0 max 16384 2024-03-14T09:59:02.166 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:59:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.163+0000 7f7881a00700 1 -- 172.21.15.78:0/2881227353 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f78600a0c80 msgr2=0x7f78600a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.163+0000 7f7881a00700 1 --2- 172.21.15.78:0/2881227353 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f78600a0c80 0x7f78600a3140 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f786c009df0 tx=0x7f786c009260 comp rx=0 tx=0).stop 2024-03-14T09:59:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.163+0000 7f7881a00700 1 -- 172.21.15.78:0/2881227353 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f787c106050 msgr2=0x7f787c199c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.163+0000 7f7881a00700 1 --2- 172.21.15.78:0/2881227353 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f787c106050 0x7f787c199c00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f786800e9c0 tx=0x7f786800ecd0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.163+0000 7f7881a00700 1 -- 172.21.15.78:0/2881227353 shutdown_connections 2024-03-14T09:59:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.163+0000 7f7881a00700 1 --2- 172.21.15.78:0/2881227353 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f78600a0c80 0x7f78600a3140 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.163+0000 7f7881a00700 1 --2- 172.21.15.78:0/2881227353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f787c100f30 0x7f787c1996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.163+0000 7f7881a00700 1 --2- 172.21.15.78:0/2881227353 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f787c19dfa0 0x7f787c19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.163+0000 7f7881a00700 1 --2- 172.21.15.78:0/2881227353 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f787c106050 0x7f787c199c00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.163+0000 7f7881a00700 1 -- 172.21.15.78:0/2881227353 >> 172.21.15.78:0/2881227353 conn(0x7f787c0fc7b0 msgr2=0x7f787c0fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.163+0000 7f7881a00700 1 -- 172.21.15.78:0/2881227353 shutdown_connections 2024-03-14T09:59:02.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.164+0000 7f7881a00700 1 -- 172.21.15.78:0/2881227353 wait complete. 2024-03-14T09:59:02.181 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:02 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1497166877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:59:02.185 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T09:59:02.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.440+0000 7f0387a00700 1 -- 172.21.15.78:0/4009600257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0380100f20 msgr2=0x7f0380101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.440+0000 7f0387a00700 1 --2- 172.21.15.78:0/4009600257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0380100f20 0x7f0380101300 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0374009990 tx=0x7f0374009ca0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.440+0000 7f0387a00700 1 -- 172.21.15.78:0/4009600257 shutdown_connections 2024-03-14T09:59:02.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.440+0000 7f0387a00700 1 --2- 172.21.15.78:0/4009600257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0380106040 0x7f038010dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.440+0000 7f0387a00700 1 --2- 172.21.15.78:0/4009600257 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0380100f20 0x7f0380101300 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.440+0000 7f0387a00700 1 --2- 172.21.15.78:0/4009600257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03801018d0 0x7f0380105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.440+0000 7f0387a00700 1 -- 172.21.15.78:0/4009600257 >> 172.21.15.78:0/4009600257 conn(0x7f03800fc780 msgr2=0x7f03800feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.440+0000 7f0387a00700 1 -- 172.21.15.78:0/4009600257 shutdown_connections 2024-03-14T09:59:02.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.440+0000 7f0387a00700 1 -- 172.21.15.78:0/4009600257 wait complete. 2024-03-14T09:59:02.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.441+0000 7f0387a00700 1 Processor -- start 2024-03-14T09:59:02.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.441+0000 7f0387a00700 1 -- start start 2024-03-14T09:59:02.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f0387a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0380100f20 0x7f03801975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f0387a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0380106040 0x7f0380197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f0387a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f038019beb0 0x7f038019c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f0387a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038019c930 con 0x7f0380106040 2024-03-14T09:59:02.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f0387a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038019caa0 con 0x7f038019beb0 2024-03-14T09:59:02.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f0387a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038019cc60 con 0x7f0380100f20 2024-03-14T09:59:02.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f0385000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0380100f20 0x7f03801975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f037fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0380106040 0x7f0380197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f0385a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f038019beb0 0x7f038019c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f0385000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0380100f20 0x7f03801975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:42282/0 (socket says 172.21.15.78:42282) 2024-03-14T09:59:02.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f0385000700 1 -- 172.21.15.78:0/4239321448 learned_addr learned my addr 172.21.15.78:0/4239321448 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:02.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f0385a00700 1 -- 172.21.15.78:0/4239321448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0380100f20 msgr2=0x7f03801975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f0385a00700 1 --2- 172.21.15.78:0/4239321448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0380100f20 0x7f03801975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f0385a00700 1 -- 172.21.15.78:0/4239321448 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0380106040 msgr2=0x7f0380197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f0385a00700 1 --2- 172.21.15.78:0/4239321448 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0380106040 0x7f0380197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f0385a00700 1 -- 172.21.15.78:0/4239321448 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03740095f0 con 0x7f038019beb0 2024-03-14T09:59:02.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.442+0000 7f0385000700 1 --2- 172.21.15.78:0/4239321448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0380100f20 0x7f03801975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:02.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.443+0000 7f0385a00700 1 --2- 172.21.15.78:0/4239321448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f038019beb0 0x7f038019c330 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f037800efc0 tx=0x7f037800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.443+0000 7f037d600700 1 -- 172.21.15.78:0/4239321448 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0378018070 con 0x7f038019beb0 2024-03-14T09:59:02.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.443+0000 7f0387a00700 1 -- 172.21.15.78:0/4239321448 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03801a4160 con 0x7f038019beb0 2024-03-14T09:59:02.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.443+0000 7f037d600700 1 -- 172.21.15.78:0/4239321448 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f037800f040 con 0x7f038019beb0 2024-03-14T09:59:02.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.443+0000 7f037d600700 1 -- 172.21.15.78:0/4239321448 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037800cd40 con 0x7f038019beb0 2024-03-14T09:59:02.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.443+0000 7f0387a00700 1 -- 172.21.15.78:0/4239321448 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03801a4690 con 0x7f038019beb0 2024-03-14T09:59:02.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.446+0000 7f037d600700 1 -- 172.21.15.78:0/4239321448 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f0378003a00 con 0x7f038019beb0 2024-03-14T09:59:02.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.447+0000 7f039b4c3b80 1 Processor -- start 2024-03-14T09:59:02.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.447+0000 7f039b4c3b80 1 -- start start 2024-03-14T09:59:02.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.448+0000 7f039b4c3b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56680 0x565321c56a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.448+0000 7f039b4c3b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c56fa0 0x565321c60980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.448+0000 7f039b4c3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c60ec0 0x565321c632b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.448+0000 7f039b4c3b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x565321baf990 con 0x565321c56680 2024-03-14T09:59:02.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.448+0000 7f039b4c3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x565321bad350 con 0x565321c60ec0 2024-03-14T09:59:02.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.448+0000 7f039b4c3b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x565321ba8c40 con 0x565321c56fa0 2024-03-14T09:59:02.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.448+0000 7f037fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56680 0x565321c56a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.448+0000 7f037fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56680 0x565321c56a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41394/0 (socket says 172.21.15.78:41394) 2024-03-14T09:59:02.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.448+0000 7f037fe00700 1 -- 172.21.15.78:0/872990540 learned_addr learned my addr 172.21.15.78:0/872990540 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:02.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.448+0000 7f0385000700 1 --2- 172.21.15.78:0/872990540 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c56fa0 0x565321c60980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.448+0000 7f037fe00700 1 --2- 172.21.15.78:0/872990540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c60ec0 0x565321c632b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.448+0000 7f037fe00700 1 -- 172.21.15.78:0/872990540 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c56fa0 msgr2=0x565321c60980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.448+0000 7f037fe00700 1 --2- 172.21.15.78:0/872990540 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c56fa0 0x565321c60980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.448+0000 7f037fe00700 1 -- 172.21.15.78:0/872990540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c60ec0 msgr2=0x565321c632b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.448+0000 7f037fe00700 1 --2- 172.21.15.78:0/872990540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c60ec0 0x565321c632b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.449+0000 7f037fe00700 1 -- 172.21.15.78:0/872990540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565321a2a3b0 con 0x565321c56680 2024-03-14T09:59:02.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.449+0000 7f037d600700 1 --2- 172.21.15.78:0/4239321448 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f03680a0c00 0x7f03680a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.449+0000 7f037fe00700 1 --2- 172.21.15.78:0/872990540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56680 0x565321c56a60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f036c013680 tx=0x7f036c009a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28300f8535d637f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.449+0000 7f037d600700 1 -- 172.21.15.78:0/4239321448 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f03780cc2f0 con 0x7f038019beb0 2024-03-14T09:59:02.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.449+0000 7f0385000700 1 --2- 172.21.15.78:0/4239321448 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f03680a0c00 0x7f03680a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.449+0000 7f0372000700 1 -- 172.21.15.78:0/872990540 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c004050 con 0x565321c56680 2024-03-14T09:59:02.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.449+0000 7f0372000700 1 -- 172.21.15.78:0/872990540 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f036c003680 con 0x565321c56680 2024-03-14T09:59:02.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.449+0000 7f0372000700 1 -- 172.21.15.78:0/872990540 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c010690 con 0x565321c56680 2024-03-14T09:59:02.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.450+0000 7f039b4c3b80 1 -- 172.21.15.78:0/872990540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56680 msgr2=0x565321c56a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.450+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/872990540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56680 0x565321c56a60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f036c013680 tx=0x7f036c009a60 comp rx=0 tx=0).stop 2024-03-14T09:59:02.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.450+0000 7f0385000700 1 --2- 172.21.15.78:0/4239321448 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f03680a0c00 0x7f03680a30c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0374006010 tx=0x7f037400b260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.450+0000 7f039b4c3b80 1 -- 172.21.15.78:0/872990540 shutdown_connections 2024-03-14T09:59:02.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.450+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/872990540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c60ec0 0x565321c632b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.450+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/872990540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56680 0x565321c56a60 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.450+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/872990540 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c56fa0 0x565321c60980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.450+0000 7f039b4c3b80 1 -- 172.21.15.78:0/872990540 >> 172.21.15.78:0/872990540 conn(0x565321bcc220 msgr2=0x565321c51aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.450+0000 7f039b4c3b80 1 -- 172.21.15.78:0/872990540 shutdown_connections 2024-03-14T09:59:02.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.450+0000 7f039b4c3b80 1 -- 172.21.15.78:0/872990540 wait complete. 2024-03-14T09:59:02.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.451+0000 7f039b4c3b80 1 -- start start 2024-03-14T09:59:02.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.451+0000 7f039b4c3b80 1 Processor -- start 2024-03-14T09:59:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.451+0000 7f039b4c3b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56680 0x565321c60870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.451+0000 7f039b4c3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56fa0 0x565321c59bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.451+0000 7f039b4c3b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c60ec0 0x565321c5a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.451+0000 7f039b4c3b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x565321b44d80 con 0x565321c56680 2024-03-14T09:59:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.451+0000 7f039b4c3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x565321b44a20 con 0x565321c56fa0 2024-03-14T09:59:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.451+0000 7f039b4c3b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x565321a2c6a0 con 0x565321c60ec0 2024-03-14T09:59:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.451+0000 7f039b4c3b80 10 client.0 _ll_register_callbacks cb 0x5653219b03a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.452+0000 7f037fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56fa0 0x565321c59bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.452+0000 7f037fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56fa0 0x565321c59bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:52022/0 (socket says 172.21.15.78:52022) 2024-03-14T09:59:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.452+0000 7f037fe00700 1 -- 172.21.15.78:0/3637542723 learned_addr learned my addr 172.21.15.78:0/3637542723 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.452+0000 7f0385000700 1 --2- 172.21.15.78:0/3637542723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56680 0x565321c60870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.452+0000 7f0385a00700 1 --2- 172.21.15.78:0/3637542723 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c60ec0 0x565321c5a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.452+0000 7f037fe00700 1 -- 172.21.15.78:0/3637542723 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c60ec0 msgr2=0x565321c5a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.452+0000 7f037fe00700 1 --2- 172.21.15.78:0/3637542723 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c60ec0 0x565321c5a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.452+0000 7f037fe00700 1 -- 172.21.15.78:0/3637542723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56680 msgr2=0x565321c60870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.452+0000 7f037fe00700 1 --2- 172.21.15.78:0/3637542723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56680 0x565321c60870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.452+0000 7f037fe00700 1 -- 172.21.15.78:0/3637542723 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f036c009e20 con 0x565321c56fa0 2024-03-14T09:59:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.452+0000 7f0385000700 1 --2- 172.21.15.78:0/3637542723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56680 0x565321c60870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:59:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.452+0000 7f0385a00700 1 --2- 172.21.15.78:0/3637542723 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c60ec0 0x565321c5a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.453+0000 7f037fe00700 1 --2- 172.21.15.78:0/3637542723 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56fa0 0x565321c59bf0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f036c0133e0 tx=0x7f036c003d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0b4e6ff86833cfc server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.453+0000 7f0366a00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:59:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.453+0000 7f0366a00700 1 -- 172.21.15.78:0/3637542723 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c004460 con 0x565321c56fa0 2024-03-14T09:59:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.453+0000 7f039b4c3b80 1 -- 172.21.15.78:0/3637542723 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565321c49720 con 0x565321c56fa0 2024-03-14T09:59:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.453+0000 7f0366a00700 1 -- 172.21.15.78:0/3637542723 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f036c005050 con 0x565321c56fa0 2024-03-14T09:59:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.453+0000 7f0366a00700 1 -- 172.21.15.78:0/3637542723 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f036c005210 con 0x565321c56fa0 2024-03-14T09:59:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.453+0000 7f039b4c3b80 10 client.26642 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.453+0000 7f039b4c3b80 1 -- 172.21.15.78:0/3637542723 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5653219d6b60 con 0x565321c56fa0 2024-03-14T09:59:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.453+0000 7f0366a00700 1 -- 172.21.15.78:0/3637542723 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x565321c49720 con 0x565321c56fa0 2024-03-14T09:59:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.454+0000 7f0366a00700 1 -- 172.21.15.78:0/3637542723 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f036c02b430 con 0x565321c56fa0 2024-03-14T09:59:02.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.454+0000 7f039b4c3b80 10 client.26642 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:02.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.454+0000 7f039b4c3b80 10 client.26642 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:59:02.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.454+0000 7f039b4c3b80 20 client.26642 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:02.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.454+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/3637542723 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x565321c791f0 0x565321c7b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.454+0000 7f039b4c3b80 4 client.26642 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:59:02.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.454+0000 7f039b4c3b80 1 -- 172.21.15.78:0/3637542723 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565321ba1700 con 0x565321c791f0 2024-03-14T09:59:02.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.454+0000 7f0385000700 1 --2- 172.21.15.78:0/3637542723 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x565321c791f0 0x565321c7b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.455+0000 7f0385000700 1 --2- 172.21.15.78:0/3637542723 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x565321c791f0 0x565321c7b610 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=274c258a5339237f server_cookie=45a5262b24465777 in_seq=0 out_seq=0 2024-03-14T09:59:02.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.455+0000 7f0366a00700 10 client.26642 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:59:02.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.455+0000 7f0366a00700 1 -- 172.21.15.78:0/3637542723 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x565321ba1700 con 0x565321c791f0 2024-03-14T09:59:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.455+0000 7f0366a00700 10 client.26642 handle_command_reply: tid=0 2024-03-14T09:59:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.455+0000 7f039b4c3b80 1 client.26642 shutdown 2024-03-14T09:59:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.456+0000 7f039b4c3b80 1 -- 172.21.15.78:0/3637542723 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56fa0 msgr2=0x565321c59bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.456+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/3637542723 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56fa0 0x565321c59bf0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f036c0133e0 tx=0x7f036c003d90 comp rx=0 tx=0).stop 2024-03-14T09:59:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.456+0000 7f039b4c3b80 1 -- 172.21.15.78:0/3637542723 shutdown_connections 2024-03-14T09:59:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.456+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/3637542723 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x565321c791f0 0x565321c7b610 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.456+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/3637542723 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c60ec0 0x565321c5a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.456+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/3637542723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56680 0x565321c60870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.456+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/3637542723 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56fa0 0x565321c59bf0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.456+0000 7f039b4c3b80 1 -- 172.21.15.78:0/3637542723 >> 172.21.15.78:0/3637542723 conn(0x565321bcc220 msgr2=0x565321c51690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.456+0000 7f0366a00700 0 client.26642 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:59:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.456+0000 7f039b4c3b80 1 -- 172.21.15.78:0/3637542723 shutdown_connections 2024-03-14T09:59:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.456+0000 7f039b4c3b80 1 -- 172.21.15.78:0/3637542723 wait complete. 2024-03-14T09:59:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.456+0000 7f039b4c3b80 20 client.26642 trim_cache size 0 max 16384 2024-03-14T09:59:02.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.472+0000 7f039b4c3b80 1 Processor -- start 2024-03-14T09:59:02.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.472+0000 7f039b4c3b80 1 -- start start 2024-03-14T09:59:02.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.473+0000 7f039b4c3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56680 0x565321c51c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.473+0000 7f039b4c3b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56fa0 0x565321c603f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.473+0000 7f039b4c3b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c60ec0 0x565321c66100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.473+0000 7f039b4c3b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x565321c58700 con 0x565321c56fa0 2024-03-14T09:59:02.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.473+0000 7f039b4c3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x565321c59a30 con 0x565321c56680 2024-03-14T09:59:02.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.473+0000 7f039b4c3b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x565321c57af0 con 0x565321c60ec0 2024-03-14T09:59:02.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.473+0000 7f037fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56fa0 0x565321c603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.473+0000 7f0385000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56680 0x565321c51c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.473+0000 7f037fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56fa0 0x565321c603f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41426/0 (socket says 172.21.15.78:41426) 2024-03-14T09:59:02.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.473+0000 7f037fe00700 1 -- 172.21.15.78:0/259691649 learned_addr learned my addr 172.21.15.78:0/259691649 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:02.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.473+0000 7f0385000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56680 0x565321c51c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:52028/0 (socket says 172.21.15.78:52028) 2024-03-14T09:59:02.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.473+0000 7f0385a00700 1 --2- 172.21.15.78:0/259691649 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c60ec0 0x565321c66100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.474+0000 7f037fe00700 1 -- 172.21.15.78:0/259691649 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c60ec0 msgr2=0x565321c66100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.474+0000 7f037fe00700 1 --2- 172.21.15.78:0/259691649 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c60ec0 0x565321c66100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.474+0000 7f037fe00700 1 -- 172.21.15.78:0/259691649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56680 msgr2=0x565321c51c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.474+0000 7f037fe00700 1 --2- 172.21.15.78:0/259691649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56680 0x565321c51c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.474+0000 7f037fe00700 1 -- 172.21.15.78:0/259691649 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565321b6eea0 con 0x565321c56fa0 2024-03-14T09:59:02.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.474+0000 7f0385000700 1 --2- 172.21.15.78:0/259691649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56680 0x565321c51c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:02.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.474+0000 7f037fe00700 1 --2- 172.21.15.78:0/259691649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56fa0 0x565321c603f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f036c009df0 tx=0x7f036c009a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16058f9336479d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.474+0000 7f0372a00700 1 -- 172.21.15.78:0/259691649 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c003eb0 con 0x565321c56fa0 2024-03-14T09:59:02.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.474+0000 7f0372a00700 1 -- 172.21.15.78:0/259691649 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f036c005ce0 con 0x565321c56fa0 2024-03-14T09:59:02.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.474+0000 7f0372a00700 1 -- 172.21.15.78:0/259691649 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c0108a0 con 0x565321c56fa0 2024-03-14T09:59:02.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.475+0000 7f039b4c3b80 1 -- 172.21.15.78:0/259691649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56fa0 msgr2=0x565321c603f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.475+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/259691649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56fa0 0x565321c603f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f036c009df0 tx=0x7f036c009a40 comp rx=0 tx=0).stop 2024-03-14T09:59:02.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.475+0000 7f039b4c3b80 1 -- 172.21.15.78:0/259691649 shutdown_connections 2024-03-14T09:59:02.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.475+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/259691649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56680 0x565321c51c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.475+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/259691649 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c60ec0 0x565321c66100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.475+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/259691649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c56fa0 0x565321c603f0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.475+0000 7f039b4c3b80 1 -- 172.21.15.78:0/259691649 >> 172.21.15.78:0/259691649 conn(0x565321bcc220 msgr2=0x565321c64180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.475+0000 7f039b4c3b80 1 -- 172.21.15.78:0/259691649 shutdown_connections 2024-03-14T09:59:02.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.475+0000 7f039b4c3b80 1 -- 172.21.15.78:0/259691649 wait complete. 2024-03-14T09:59:02.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.476+0000 7f039b4c3b80 1 -- start start 2024-03-14T09:59:02.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.476+0000 7f039b4c3b80 1 Processor -- start 2024-03-14T09:59:02.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.477+0000 7f039b4c3b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c56680 0x565321c71750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.477+0000 7f039b4c3b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56fa0 0x565321c71c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.477+0000 7f039b4c3b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c60ec0 0x565321c72210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.477+0000 7f039b4c3b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x565321cc9e90 con 0x565321c60ec0 2024-03-14T09:59:02.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.477+0000 7f039b4c3b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x565321cca000 con 0x565321c56fa0 2024-03-14T09:59:02.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.477+0000 7f039b4c3b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x565321cca1c0 con 0x565321c56680 2024-03-14T09:59:02.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.477+0000 7f039b4c3b80 10 client.0 _ll_register_callbacks cb 0x5653219b03a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:02.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.477+0000 7f0385000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c56680 0x565321c71750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.477+0000 7f0385000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c56680 0x565321c71750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:42322/0 (socket says 172.21.15.78:42322) 2024-03-14T09:59:02.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.477+0000 7f0385000700 1 -- 172.21.15.78:0/4045802244 learned_addr learned my addr 172.21.15.78:0/4045802244 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:02.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.477+0000 7f037fe00700 1 --2- 172.21.15.78:0/4045802244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56fa0 0x565321c71c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.477+0000 7f0385a00700 1 --2- 172.21.15.78:0/4045802244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c60ec0 0x565321c72210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.478+0000 7f037fe00700 1 -- 172.21.15.78:0/4045802244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c56680 msgr2=0x565321c71750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.478+0000 7f037fe00700 1 --2- 172.21.15.78:0/4045802244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c56680 0x565321c71750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.478+0000 7f037fe00700 1 -- 172.21.15.78:0/4045802244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c60ec0 msgr2=0x565321c72210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.478+0000 7f037fe00700 1 --2- 172.21.15.78:0/4045802244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c60ec0 0x565321c72210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.478+0000 7f037fe00700 1 -- 172.21.15.78:0/4045802244 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f038010e430 con 0x565321c56fa0 2024-03-14T09:59:02.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.478+0000 7f0385000700 1 --2- 172.21.15.78:0/4045802244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c56680 0x565321c71750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:59:02.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.478+0000 7f037fe00700 1 --2- 172.21.15.78:0/4045802244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56fa0 0x565321c71c90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x565321c56f70 tx=0x7f036c003d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7335fb43ed302c14 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.478+0000 7f0366a00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:59:02.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.478+0000 7f0366a00700 1 -- 172.21.15.78:0/4045802244 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f036c0054d0 con 0x565321c56fa0 2024-03-14T09:59:02.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.478+0000 7f0366a00700 1 -- 172.21.15.78:0/4045802244 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f036c000e40 con 0x565321c56fa0 2024-03-14T09:59:02.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.478+0000 7f039b4c3b80 1 -- 172.21.15.78:0/4045802244 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f036c003680 con 0x565321c56fa0 2024-03-14T09:59:02.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.478+0000 7f0366a00700 1 -- 172.21.15.78:0/4045802244 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f036c005630 con 0x565321c56fa0 2024-03-14T09:59:02.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.478+0000 7f039b4c3b80 10 client.26648 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:02.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.478+0000 7f039b4c3b80 1 -- 172.21.15.78:0/4045802244 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5653219d6b60 con 0x565321c56fa0 2024-03-14T09:59:02.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.479+0000 7f0366a00700 1 -- 172.21.15.78:0/4045802244 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f036c003680 con 0x565321c56fa0 2024-03-14T09:59:02.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.479+0000 7f0366a00700 1 -- 172.21.15.78:0/4045802244 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f036c013990 con 0x565321c56fa0 2024-03-14T09:59:02.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.479+0000 7f039b4c3b80 10 client.26648 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:02.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.479+0000 7f039b4c3b80 10 client.26648 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:59:02.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.479+0000 7f039b4c3b80 20 client.26648 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:02.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.479+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/4045802244 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x565321ccb500 0x565321ccd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.479+0000 7f039b4c3b80 4 client.26648 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:59:02.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.479+0000 7f039b4c3b80 1 -- 172.21.15.78:0/4045802244 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5653219f7cf0 con 0x565321ccb500 2024-03-14T09:59:02.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.479+0000 7f0385000700 1 --2- 172.21.15.78:0/4045802244 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x565321ccb500 0x565321ccd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.480+0000 7f0385000700 1 --2- 172.21.15.78:0/4045802244 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x565321ccb500 0x565321ccd8f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6952758823fc4597 server_cookie=eeacefe6fcc454bb in_seq=0 out_seq=0 2024-03-14T09:59:02.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.480+0000 7f0366a00700 10 client.26648 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:59:02.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.480+0000 7f0366a00700 1 -- 172.21.15.78:0/4045802244 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5653219f7cf0 con 0x565321ccb500 2024-03-14T09:59:02.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.480+0000 7f0366a00700 10 client.26648 handle_command_reply: tid=0 2024-03-14T09:59:02.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.480+0000 7f039b4c3b80 1 client.26648 shutdown 2024-03-14T09:59:02.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.480+0000 7f039b4c3b80 1 -- 172.21.15.78:0/4045802244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56fa0 msgr2=0x565321c71c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.481+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/4045802244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56fa0 0x565321c71c90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x565321c56f70 tx=0x7f036c003d90 comp rx=0 tx=0).stop 2024-03-14T09:59:02.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.481+0000 7f039b4c3b80 1 -- 172.21.15.78:0/4045802244 shutdown_connections 2024-03-14T09:59:02.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.481+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/4045802244 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x565321ccb500 0x565321ccd8f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.481+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/4045802244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565321c60ec0 0x565321c72210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.481+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/4045802244 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565321c56680 0x565321c71750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.481+0000 7f0366a00700 0 client.26648 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:59:02.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.481+0000 7f039b4c3b80 1 --2- 172.21.15.78:0/4045802244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565321c56fa0 0x565321c71c90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.481+0000 7f039b4c3b80 1 -- 172.21.15.78:0/4045802244 >> 172.21.15.78:0/4045802244 conn(0x565321bcc220 msgr2=0x565321c55560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.481+0000 7f039b4c3b80 1 -- 172.21.15.78:0/4045802244 shutdown_connections 2024-03-14T09:59:02.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.481+0000 7f039b4c3b80 1 -- 172.21.15.78:0/4045802244 wait complete. 2024-03-14T09:59:02.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.481+0000 7f039b4c3b80 20 client.26648 trim_cache size 0 max 16384 2024-03-14T09:59:02.485 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:59:02.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.482+0000 7f0387a00700 1 -- 172.21.15.78:0/4239321448 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f03680a0c00 msgr2=0x7f03680a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.482+0000 7f0387a00700 1 --2- 172.21.15.78:0/4239321448 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f03680a0c00 0x7f03680a30c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0374006010 tx=0x7f037400b260 comp rx=0 tx=0).stop 2024-03-14T09:59:02.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.482+0000 7f0387a00700 1 -- 172.21.15.78:0/4239321448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f038019beb0 msgr2=0x7f038019c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.482+0000 7f0387a00700 1 --2- 172.21.15.78:0/4239321448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f038019beb0 0x7f038019c330 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f037800efc0 tx=0x7f037800c3a0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.482+0000 7f0387a00700 1 -- 172.21.15.78:0/4239321448 shutdown_connections 2024-03-14T09:59:02.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.482+0000 7f0387a00700 1 --2- 172.21.15.78:0/4239321448 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f03680a0c00 0x7f03680a30c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.482+0000 7f0387a00700 1 --2- 172.21.15.78:0/4239321448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f038019beb0 0x7f038019c330 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.482+0000 7f0387a00700 1 --2- 172.21.15.78:0/4239321448 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0380100f20 0x7f03801975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.482+0000 7f0387a00700 1 --2- 172.21.15.78:0/4239321448 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0380106040 0x7f0380197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.482+0000 7f0387a00700 1 -- 172.21.15.78:0/4239321448 >> 172.21.15.78:0/4239321448 conn(0x7f03800fc780 msgr2=0x7f03800fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.482+0000 7f0387a00700 1 -- 172.21.15.78:0/4239321448 shutdown_connections 2024-03-14T09:59:02.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.482+0000 7f0387a00700 1 -- 172.21.15.78:0/4239321448 wait complete. 2024-03-14T09:59:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:02 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1497166877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:59:02.503 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T09:59:02.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:02 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1497166877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:59:02.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.765+0000 7ffa05800700 1 -- 172.21.15.78:0/1457706979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa001018d0 msgr2=0x7ffa00105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.765+0000 7ffa05800700 1 --2- 172.21.15.78:0/1457706979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa001018d0 0x7ffa00105910 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009990 tx=0x7ff9ec009ca0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.765+0000 7ffa05800700 1 -- 172.21.15.78:0/1457706979 shutdown_connections 2024-03-14T09:59:02.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.765+0000 7ffa05800700 1 --2- 172.21.15.78:0/1457706979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa00106040 0x7ffa0010dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.765+0000 7ffa05800700 1 --2- 172.21.15.78:0/1457706979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa00100f20 0x7ffa00101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.765+0000 7ffa05800700 1 --2- 172.21.15.78:0/1457706979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa001018d0 0x7ffa00105910 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.765+0000 7ffa05800700 1 -- 172.21.15.78:0/1457706979 >> 172.21.15.78:0/1457706979 conn(0x7ffa000fc7a0 msgr2=0x7ffa000febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.765+0000 7ffa05800700 1 -- 172.21.15.78:0/1457706979 shutdown_connections 2024-03-14T09:59:02.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.765+0000 7ffa05800700 1 -- 172.21.15.78:0/1457706979 wait complete. 2024-03-14T09:59:02.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.766+0000 7ffa05800700 1 Processor -- start 2024-03-14T09:59:02.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.766+0000 7ffa05800700 1 -- start start 2024-03-14T09:59:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.767+0000 7ffa05800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa00100f20 0x7ffa0019f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.767+0000 7ffa05800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa00106040 0x7ffa0019fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.767+0000 7ffa05800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa00199650 0x7ffa00199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.767+0000 7ffa05800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0019a0d0 con 0x7ffa00106040 2024-03-14T09:59:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.767+0000 7ffa05800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0019a240 con 0x7ffa00199650 2024-03-14T09:59:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.767+0000 7ffa05800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0019a400 con 0x7ffa00100f20 2024-03-14T09:59:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.767+0000 7ff9fea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa00100f20 0x7ffa0019f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.767+0000 7ff9fea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa00100f20 0x7ffa0019f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:42348/0 (socket says 172.21.15.78:42348) 2024-03-14T09:59:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.767+0000 7ff9fea00700 1 -- 172.21.15.78:0/4245983879 learned_addr learned my addr 172.21.15.78:0/4245983879 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:02.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.767+0000 7ff9fe000700 1 --2- 172.21.15.78:0/4245983879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa00106040 0x7ffa0019fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.767+0000 7ff9ff400700 1 --2- 172.21.15.78:0/4245983879 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa00199650 0x7ffa00199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.768+0000 7ff9fea00700 1 -- 172.21.15.78:0/4245983879 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa00199650 msgr2=0x7ffa00199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.768+0000 7ff9fea00700 1 --2- 172.21.15.78:0/4245983879 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa00199650 0x7ffa00199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.768+0000 7ff9fea00700 1 -- 172.21.15.78:0/4245983879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa00106040 msgr2=0x7ffa0019fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.768+0000 7ff9fea00700 1 --2- 172.21.15.78:0/4245983879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa00106040 0x7ffa0019fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.768+0000 7ff9fea00700 1 -- 172.21.15.78:0/4245983879 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f0009520 con 0x7ffa00100f20 2024-03-14T09:59:02.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.768+0000 7ff9fea00700 1 --2- 172.21.15.78:0/4245983879 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa00100f20 0x7ffa0019f520 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000eb30 tx=0x7ff9f000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.768+0000 7ff9f7400700 1 -- 172.21.15.78:0/4245983879 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f0009880 con 0x7ffa00100f20 2024-03-14T09:59:02.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.768+0000 7ff9f7400700 1 -- 172.21.15.78:0/4245983879 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9f00044e0 con 0x7ffa00100f20 2024-03-14T09:59:02.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.768+0000 7ffa05800700 1 -- 172.21.15.78:0/4245983879 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ec0095f0 con 0x7ffa00100f20 2024-03-14T09:59:02.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.769+0000 7ff9f7400700 1 -- 172.21.15.78:0/4245983879 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f0003e90 con 0x7ffa00100f20 2024-03-14T09:59:02.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.769+0000 7ffa05800700 1 -- 172.21.15.78:0/4245983879 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa001a6790 con 0x7ffa00100f20 2024-03-14T09:59:02.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.777+0000 7ffa19361b80 1 Processor -- start 2024-03-14T09:59:02.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.777+0000 7ffa19361b80 1 -- start start 2024-03-14T09:59:02.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.778+0000 7ff9f7400700 1 -- 172.21.15.78:0/4245983879 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7ff9f0010490 con 0x7ffa00100f20 2024-03-14T09:59:02.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.778+0000 7ffa19361b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 0x55d1f19d0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.778+0000 7ffa19361b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19d0fa0 0x55d1f19e3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.778+0000 7ffa19361b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19e36d0 0x55d1f19e5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.778+0000 7ffa19361b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f1929990 con 0x55d1f19d0fa0 2024-03-14T09:59:02.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.778+0000 7ffa19361b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f1927350 con 0x55d1f19d0680 2024-03-14T09:59:02.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.778+0000 7ffa19361b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f1922c40 con 0x55d1f19e36d0 2024-03-14T09:59:02.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.778+0000 7ff9ff400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 0x55d1f19d0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.778+0000 7ff9fe000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19e36d0 0x55d1f19e5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.778+0000 7ff9ff400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 0x55d1f19d0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:52056/0 (socket says 172.21.15.78:52056) 2024-03-14T09:59:02.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.778+0000 7ff9fe000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19e36d0 0x55d1f19e5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:42352/0 (socket says 172.21.15.78:42352) 2024-03-14T09:59:02.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.778+0000 7ff9ff400700 1 -- 172.21.15.78:0/3235176321 learned_addr learned my addr 172.21.15.78:0/3235176321 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:02.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.779+0000 7ff9fea00700 1 --2- 172.21.15.78:0/3235176321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19d0fa0 0x55d1f19e3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.779+0000 7ff9fe000700 1 -- 172.21.15.78:0/3235176321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 msgr2=0x55d1f19d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.779+0000 7ff9fe000700 1 --2- 172.21.15.78:0/3235176321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 0x55d1f19d0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.779+0000 7ff9fe000700 1 -- 172.21.15.78:0/3235176321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19d0fa0 msgr2=0x55d1f19e3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.779+0000 7ff9fe000700 1 --2- 172.21.15.78:0/3235176321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19d0fa0 0x55d1f19e3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.779+0000 7ff9fe000700 1 -- 172.21.15.78:0/3235176321 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1f17a43b0 con 0x55d1f19e36d0 2024-03-14T09:59:02.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.780+0000 7ff9fe000700 1 --2- 172.21.15.78:0/3235176321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19e36d0 0x55d1f19e5ac0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec0098e0 tx=0x7ff9ec005ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61dc643690fe87e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.782+0000 7ff9f7400700 1 --2- 172.21.15.78:0/4245983879 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff9e40a0c80 0x7ff9e40a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.783+0000 7ff9f7400700 1 -- 172.21.15.78:0/4245983879 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7ff9f0014070 con 0x7ffa00100f20 2024-03-14T09:59:02.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.783+0000 7ff9ebe00700 1 -- 172.21.15.78:0/3235176321 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec01d070 con 0x55d1f19e36d0 2024-03-14T09:59:02.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.783+0000 7ff9ebe00700 1 -- 172.21.15.78:0/3235176321 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9ec01e040 con 0x55d1f19e36d0 2024-03-14T09:59:02.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.783+0000 7ff9ff400700 1 --2- 172.21.15.78:0/4245983879 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff9e40a0c80 0x7ff9e40a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.783+0000 7ff9ebe00700 1 -- 172.21.15.78:0/3235176321 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec0047f0 con 0x55d1f19e36d0 2024-03-14T09:59:02.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.786+0000 7ffa19361b80 1 -- 172.21.15.78:0/3235176321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19e36d0 msgr2=0x55d1f19e5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.786+0000 7ff9ff400700 1 --2- 172.21.15.78:0/4245983879 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff9e40a0c80 0x7ff9e40a3140 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8007610 tx=0x7ff9f8009040 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.786+0000 7ffa19361b80 1 --2- 172.21.15.78:0/3235176321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19e36d0 0x55d1f19e5ac0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec0098e0 tx=0x7ff9ec005ec0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.786+0000 7ffa19361b80 1 -- 172.21.15.78:0/3235176321 shutdown_connections 2024-03-14T09:59:02.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.786+0000 7ffa19361b80 1 --2- 172.21.15.78:0/3235176321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 0x55d1f19d0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.786+0000 7ffa19361b80 1 --2- 172.21.15.78:0/3235176321 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19e36d0 0x55d1f19e5ac0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.786+0000 7ffa19361b80 1 --2- 172.21.15.78:0/3235176321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19d0fa0 0x55d1f19e3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.786+0000 7ffa19361b80 1 -- 172.21.15.78:0/3235176321 >> 172.21.15.78:0/3235176321 conn(0x55d1f1946220 msgr2=0x55d1f19cbaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.786+0000 7ffa19361b80 1 -- 172.21.15.78:0/3235176321 shutdown_connections 2024-03-14T09:59:02.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.786+0000 7ffa19361b80 1 -- 172.21.15.78:0/3235176321 wait complete. 2024-03-14T09:59:02.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.787+0000 7ffa19361b80 1 -- start start 2024-03-14T09:59:02.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.788+0000 7ffa19361b80 1 Processor -- start 2024-03-14T09:59:02.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.789+0000 7ffa19361b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 0x55d1f19e3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.789+0000 7ffa19361b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19d0fa0 0x55d1f19e0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.789+0000 7ffa19361b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19e36d0 0x55d1f19e0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.789+0000 7ffa19361b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f18bed80 con 0x55d1f19d0fa0 2024-03-14T09:59:02.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.789+0000 7ffa19361b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f18bea20 con 0x55d1f19d0680 2024-03-14T09:59:02.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.789+0000 7ffa19361b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f17a66a0 con 0x55d1f19e36d0 2024-03-14T09:59:02.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.789+0000 7ffa19361b80 10 client.0 _ll_register_callbacks cb 0x55d1f172a3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:02.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.789+0000 7ff9ff400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19e36d0 0x55d1f19e0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.789+0000 7ff9fea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 0x55d1f19e3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.789+0000 7ff9ff400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19e36d0 0x55d1f19e0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:42356/0 (socket says 172.21.15.78:42356) 2024-03-14T09:59:02.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.789+0000 7ff9ff400700 1 -- 172.21.15.78:0/638365472 learned_addr learned my addr 172.21.15.78:0/638365472 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:02.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.789+0000 7ff9fea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 0x55d1f19e3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:52064/0 (socket says 172.21.15.78:52064) 2024-03-14T09:59:02.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.790+0000 7ff9fe000700 1 --2- 172.21.15.78:0/638365472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19d0fa0 0x55d1f19e0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.790+0000 7ff9ff400700 1 -- 172.21.15.78:0/638365472 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 msgr2=0x55d1f19e3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.790+0000 7ff9ff400700 1 --2- 172.21.15.78:0/638365472 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 0x55d1f19e3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.790+0000 7ff9ff400700 1 -- 172.21.15.78:0/638365472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19d0fa0 msgr2=0x55d1f19e0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.790+0000 7ff9ff400700 1 --2- 172.21.15.78:0/638365472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19d0fa0 0x55d1f19e0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.790+0000 7ff9ff400700 1 -- 172.21.15.78:0/638365472 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa0010e430 con 0x55d1f19e36d0 2024-03-14T09:59:02.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.790+0000 7ff9fe000700 1 --2- 172.21.15.78:0/638365472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19d0fa0 0x55d1f19e0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:59:02.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.791+0000 7ff9ff400700 1 --2- 172.21.15.78:0/638365472 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19e36d0 0x55d1f19e0970 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff9f801ae60 tx=0x7ff9f800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37fadbd70d034dde server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.791+0000 7ff9e8c00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:59:02.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.791+0000 7ff9e8c00700 1 -- 172.21.15.78:0/638365472 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f801f070 con 0x55d1f19e36d0 2024-03-14T09:59:02.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.791+0000 7ffa19361b80 1 -- 172.21.15.78:0/638365472 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d1f19c3720 con 0x55d1f19e36d0 2024-03-14T09:59:02.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.791+0000 7ff9e8c00700 1 -- 172.21.15.78:0/638365472 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7ff9f8009270 con 0x55d1f19e36d0 2024-03-14T09:59:02.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.791+0000 7ff9e8c00700 1 -- 172.21.15.78:0/638365472 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9f801a2e0 con 0x55d1f19e36d0 2024-03-14T09:59:02.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.791+0000 7ffa19361b80 10 client.26644 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:02.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.791+0000 7ffa19361b80 1 -- 172.21.15.78:0/638365472 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d1f1750b60 con 0x55d1f19e36d0 2024-03-14T09:59:02.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.792+0000 7ff9e8c00700 1 -- 172.21.15.78:0/638365472 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d1f19c3720 con 0x55d1f19e36d0 2024-03-14T09:59:02.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.792+0000 7ff9e8c00700 1 -- 172.21.15.78:0/638365472 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff9f8018990 con 0x55d1f19e36d0 2024-03-14T09:59:02.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.792+0000 7ffa19361b80 10 client.26644 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:02.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.793+0000 7ffa19361b80 10 client.26644 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:59:02.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.793+0000 7ffa19361b80 20 client.26644 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:02.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.793+0000 7ffa19361b80 1 --2- 172.21.15.78:0/638365472 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d1f19f3360 0x55d1f19f5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.793+0000 7ffa19361b80 4 client.26644 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:59:02.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.793+0000 7ffa19361b80 1 -- 172.21.15.78:0/638365472 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d1f191b700 con 0x55d1f19f3360 2024-03-14T09:59:02.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.793+0000 7ff9fea00700 1 --2- 172.21.15.78:0/638365472 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d1f19f3360 0x55d1f19f5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.793+0000 7ff9fea00700 1 --2- 172.21.15.78:0/638365472 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d1f19f3360 0x55d1f19f5780 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8a64c2159099eb8 server_cookie=66eee43a7fd99a3d in_seq=0 out_seq=0 2024-03-14T09:59:02.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.793+0000 7ff9e8c00700 10 client.26644 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:02.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.794+0000 7ff9e8c00700 1 -- 172.21.15.78:0/638365472 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55d1f191b700 con 0x55d1f19f3360 2024-03-14T09:59:02.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.794+0000 7ff9e8c00700 10 client.26644 handle_command_reply: tid=0 2024-03-14T09:59:02.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.794+0000 7ffa19361b80 1 client.26644 shutdown 2024-03-14T09:59:02.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.794+0000 7ffa19361b80 1 -- 172.21.15.78:0/638365472 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19e36d0 msgr2=0x55d1f19e0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.794+0000 7ffa19361b80 1 --2- 172.21.15.78:0/638365472 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19e36d0 0x55d1f19e0970 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff9f801ae60 tx=0x7ff9f800b040 comp rx=0 tx=0).stop 2024-03-14T09:59:02.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.794+0000 7ffa19361b80 1 -- 172.21.15.78:0/638365472 shutdown_connections 2024-03-14T09:59:02.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.794+0000 7ffa19361b80 1 --2- 172.21.15.78:0/638365472 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d1f19f3360 0x55d1f19f5780 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.794+0000 7ffa19361b80 1 --2- 172.21.15.78:0/638365472 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 0x55d1f19e3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.794+0000 7ffa19361b80 1 --2- 172.21.15.78:0/638365472 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19e36d0 0x55d1f19e0970 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.794+0000 7ffa19361b80 1 --2- 172.21.15.78:0/638365472 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19d0fa0 0x55d1f19e0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.794+0000 7ffa19361b80 1 -- 172.21.15.78:0/638365472 >> 172.21.15.78:0/638365472 conn(0x55d1f1946220 msgr2=0x55d1f19cb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.794+0000 7ff9e8c00700 0 client.26644 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:02.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.795+0000 7ffa19361b80 1 -- 172.21.15.78:0/638365472 shutdown_connections 2024-03-14T09:59:02.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.795+0000 7ffa19361b80 1 -- 172.21.15.78:0/638365472 wait complete. 2024-03-14T09:59:02.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.795+0000 7ffa19361b80 20 client.26644 trim_cache size 0 max 16384 2024-03-14T09:59:02.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.811+0000 7ffa19361b80 1 Processor -- start 2024-03-14T09:59:02.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.811+0000 7ffa19361b80 1 -- start start 2024-03-14T09:59:02.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.811+0000 7ffa19361b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19d0680 0x55d1f19ea370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.811+0000 7ffa19361b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0fa0 0x55d1f19e08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.811+0000 7ffa19361b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19e36d0 0x55d1f19dd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.811+0000 7ffa19361b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f19d1850 con 0x55d1f19e36d0 2024-03-14T09:59:02.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.811+0000 7ffa19361b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f19d2760 con 0x55d1f19d0fa0 2024-03-14T09:59:02.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.811+0000 7ffa19361b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f19cd060 con 0x55d1f19d0680 2024-03-14T09:59:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.812+0000 7ff9fea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19d0680 0x55d1f19ea370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.812+0000 7ff9ff400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19e36d0 0x55d1f19dd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.812+0000 7ff9fea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19d0680 0x55d1f19ea370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:42362/0 (socket says 172.21.15.78:42362) 2024-03-14T09:59:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.812+0000 7ff9fea00700 1 -- 172.21.15.78:0/2819472681 learned_addr learned my addr 172.21.15.78:0/2819472681 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.812+0000 7ff9fe000700 1 --2- 172.21.15.78:0/2819472681 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0fa0 0x55d1f19e08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.812+0000 7ff9fea00700 1 -- 172.21.15.78:0/2819472681 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0fa0 msgr2=0x55d1f19e08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.812+0000 7ff9fea00700 1 --2- 172.21.15.78:0/2819472681 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0fa0 0x55d1f19e08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.812+0000 7ff9fea00700 1 -- 172.21.15.78:0/2819472681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19e36d0 msgr2=0x55d1f19dd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.812+0000 7ff9fea00700 1 --2- 172.21.15.78:0/2819472681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19e36d0 0x55d1f19dd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.812+0000 7ff9fea00700 1 -- 172.21.15.78:0/2819472681 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1f18e8ea0 con 0x55d1f19d0680 2024-03-14T09:59:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.812+0000 7ff9ff400700 1 --2- 172.21.15.78:0/2819472681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19e36d0 0x55d1f19dd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.812+0000 7ff9fea00700 1 --2- 172.21.15.78:0/2819472681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19d0680 0x55d1f19ea370 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000c5b0 tx=0x7ff9f00cecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b8c30a5536c76e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.813+0000 7ff9f4c00700 1 -- 172.21.15.78:0/2819472681 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f000cc90 con 0x55d1f19d0680 2024-03-14T09:59:02.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.813+0000 7ff9f4c00700 1 -- 172.21.15.78:0/2819472681 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9f0099a00 con 0x55d1f19d0680 2024-03-14T09:59:02.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.813+0000 7ff9f4c00700 1 -- 172.21.15.78:0/2819472681 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f00ca080 con 0x55d1f19d0680 2024-03-14T09:59:02.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.813+0000 7ffa19361b80 1 -- 172.21.15.78:0/2819472681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19d0680 msgr2=0x55d1f19ea370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.813+0000 7ffa19361b80 1 --2- 172.21.15.78:0/2819472681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19d0680 0x55d1f19ea370 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000c5b0 tx=0x7ff9f00cecb0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.813+0000 7ffa19361b80 1 -- 172.21.15.78:0/2819472681 shutdown_connections 2024-03-14T09:59:02.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.813+0000 7ffa19361b80 1 --2- 172.21.15.78:0/2819472681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19e36d0 0x55d1f19dd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.813+0000 7ffa19361b80 1 --2- 172.21.15.78:0/2819472681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19d0680 0x55d1f19ea370 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.813+0000 7ffa19361b80 1 --2- 172.21.15.78:0/2819472681 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0fa0 0x55d1f19e08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.813+0000 7ffa19361b80 1 -- 172.21.15.78:0/2819472681 >> 172.21.15.78:0/2819472681 conn(0x55d1f1946220 msgr2=0x55d1f19dec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.814+0000 7ffa19361b80 1 -- 172.21.15.78:0/2819472681 shutdown_connections 2024-03-14T09:59:02.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.814+0000 7ffa19361b80 1 -- 172.21.15.78:0/2819472681 wait complete. 2024-03-14T09:59:02.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.814+0000 7ffa19361b80 1 -- start start 2024-03-14T09:59:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.814+0000 7ffa19361b80 1 Processor -- start 2024-03-14T09:59:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ffa19361b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 0x55d1f19e9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ffa19361b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19d0fa0 0x55d1f19ea090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ffa19361b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19e36d0 0x55d1f19e8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ffa19361b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f19e8920 con 0x55d1f19e36d0 2024-03-14T09:59:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ffa19361b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f19e8a90 con 0x55d1f19d0680 2024-03-14T09:59:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ffa19361b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1f19e8c50 con 0x55d1f19d0fa0 2024-03-14T09:59:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ffa19361b80 10 client.0 _ll_register_callbacks cb 0x7ff9f801a830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ff9fe000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19d0fa0 0x55d1f19ea090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ff9fea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 0x55d1f19e9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ff9fe000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19d0fa0 0x55d1f19ea090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:42374/0 (socket says 172.21.15.78:42374) 2024-03-14T09:59:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ff9fe000700 1 -- 172.21.15.78:0/28920604 learned_addr learned my addr 172.21.15.78:0/28920604 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ff9ff400700 1 --2- 172.21.15.78:0/28920604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19e36d0 0x55d1f19e8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ff9fe000700 1 -- 172.21.15.78:0/28920604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 msgr2=0x55d1f19e9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ff9fe000700 1 --2- 172.21.15.78:0/28920604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 0x55d1f19e9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ff9fe000700 1 -- 172.21.15.78:0/28920604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19e36d0 msgr2=0x55d1f19e8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ff9fe000700 1 --2- 172.21.15.78:0/28920604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19e36d0 0x55d1f19e8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.815+0000 7ff9fe000700 1 -- 172.21.15.78:0/28920604 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d1f1750b60 con 0x55d1f19d0fa0 2024-03-14T09:59:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.816+0000 7ff9fe000700 1 --2- 172.21.15.78:0/28920604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19d0fa0 0x55d1f19ea090 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec0098e0 tx=0x7ff9ec00b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1cafeba250a8982 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.816+0000 7ff9e8c00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:59:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.816+0000 7ff9e8c00700 1 -- 172.21.15.78:0/28920604 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec01d070 con 0x55d1f19d0fa0 2024-03-14T09:59:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.816+0000 7ffa19361b80 1 -- 172.21.15.78:0/28920604 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff9ec01e040 con 0x55d1f19d0fa0 2024-03-14T09:59:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.816+0000 7ff9e8c00700 1 -- 172.21.15.78:0/28920604 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7ff9ec00bc10 con 0x55d1f19d0fa0 2024-03-14T09:59:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.816+0000 7ff9e8c00700 1 -- 172.21.15.78:0/28920604 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9ec00bdd0 con 0x55d1f19d0fa0 2024-03-14T09:59:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.816+0000 7ffa19361b80 10 client.26650 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.816+0000 7ffa19361b80 1 -- 172.21.15.78:0/28920604 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7ffa001a6790 con 0x55d1f19d0fa0 2024-03-14T09:59:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.817+0000 7ff9e8c00700 1 -- 172.21.15.78:0/28920604 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9ec03f080 con 0x55d1f19d0fa0 2024-03-14T09:59:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.817+0000 7ff9e8c00700 1 -- 172.21.15.78:0/28920604 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff9ec00f4f0 con 0x55d1f19d0fa0 2024-03-14T09:59:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.817+0000 7ffa19361b80 10 client.26650 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.817+0000 7ffa19361b80 10 client.26650 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:59:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.817+0000 7ffa19361b80 20 client.26650 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.817+0000 7ffa19361b80 1 --2- 172.21.15.78:0/28920604 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d1f1a45c80 0x55d1f1a480a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.817+0000 7ffa19361b80 4 client.26650 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:59:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.817+0000 7ffa19361b80 1 -- 172.21.15.78:0/28920604 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d1f1771cf0 con 0x55d1f1a45c80 2024-03-14T09:59:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.817+0000 7ff9fea00700 1 --2- 172.21.15.78:0/28920604 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d1f1a45c80 0x55d1f1a480a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:02.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.818+0000 7ff9fea00700 1 --2- 172.21.15.78:0/28920604 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d1f1a45c80 0x55d1f1a480a0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e218d08e8de8eebf server_cookie=d73a53668f8cfc63 in_seq=0 out_seq=0 2024-03-14T09:59:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.818+0000 7ff9e8c00700 10 client.26650 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.818+0000 7ff9e8c00700 1 -- 172.21.15.78:0/28920604 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d1f1771cf0 con 0x55d1f1a45c80 2024-03-14T09:59:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.818+0000 7ff9e8c00700 10 client.26650 handle_command_reply: tid=0 2024-03-14T09:59:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.818+0000 7ffa19361b80 1 client.26650 shutdown 2024-03-14T09:59:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.818+0000 7ffa19361b80 1 -- 172.21.15.78:0/28920604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19d0fa0 msgr2=0x55d1f19ea090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.818+0000 7ffa19361b80 1 --2- 172.21.15.78:0/28920604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19d0fa0 0x55d1f19ea090 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec0098e0 tx=0x7ff9ec00b920 comp rx=0 tx=0).stop 2024-03-14T09:59:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.818+0000 7ffa19361b80 1 -- 172.21.15.78:0/28920604 shutdown_connections 2024-03-14T09:59:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.818+0000 7ffa19361b80 1 --2- 172.21.15.78:0/28920604 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d1f1a45c80 0x55d1f1a480a0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.818+0000 7ffa19361b80 1 --2- 172.21.15.78:0/28920604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d1f19d0680 0x55d1f19e9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.818+0000 7ffa19361b80 1 --2- 172.21.15.78:0/28920604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d1f19e36d0 0x55d1f19e8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.818+0000 7ff9e8c00700 0 client.26650 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.818+0000 7ffa19361b80 1 --2- 172.21.15.78:0/28920604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d1f19d0fa0 0x55d1f19ea090 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.818+0000 7ffa19361b80 1 -- 172.21.15.78:0/28920604 >> 172.21.15.78:0/28920604 conn(0x55d1f1946220 msgr2=0x55d1f19cf560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.818+0000 7ffa19361b80 1 -- 172.21.15.78:0/28920604 shutdown_connections 2024-03-14T09:59:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.818+0000 7ffa19361b80 1 -- 172.21.15.78:0/28920604 wait complete. 2024-03-14T09:59:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.819+0000 7ffa19361b80 20 client.26650 trim_cache size 0 max 16384 2024-03-14T09:59:02.824 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:59:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.819+0000 7ffa05800700 1 -- 172.21.15.78:0/4245983879 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff9e40a0c80 msgr2=0x7ff9e40a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.819+0000 7ffa05800700 1 --2- 172.21.15.78:0/4245983879 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff9e40a0c80 0x7ff9e40a3140 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8007610 tx=0x7ff9f8009040 comp rx=0 tx=0).stop 2024-03-14T09:59:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.819+0000 7ffa05800700 1 -- 172.21.15.78:0/4245983879 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa00100f20 msgr2=0x7ffa0019f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.819+0000 7ffa05800700 1 --2- 172.21.15.78:0/4245983879 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa00100f20 0x7ffa0019f520 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff9f000eb30 tx=0x7ff9f000ee40 comp rx=0 tx=0).stop 2024-03-14T09:59:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.819+0000 7ffa05800700 1 -- 172.21.15.78:0/4245983879 shutdown_connections 2024-03-14T09:59:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.819+0000 7ffa05800700 1 --2- 172.21.15.78:0/4245983879 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff9e40a0c80 0x7ff9e40a3140 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.819+0000 7ffa05800700 1 --2- 172.21.15.78:0/4245983879 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa00199650 0x7ffa00199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.819+0000 7ffa05800700 1 --2- 172.21.15.78:0/4245983879 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa00100f20 0x7ffa0019f520 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.819+0000 7ffa05800700 1 --2- 172.21.15.78:0/4245983879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa00106040 0x7ffa0019fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.819+0000 7ffa05800700 1 -- 172.21.15.78:0/4245983879 >> 172.21.15.78:0/4245983879 conn(0x7ffa000fc7a0 msgr2=0x7ffa000fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.820+0000 7ffa05800700 1 -- 172.21.15.78:0/4245983879 shutdown_connections 2024-03-14T09:59:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:02.820+0000 7ffa05800700 1 -- 172.21.15.78:0/4245983879 wait complete. 2024-03-14T09:59:03.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:03 smithi078 ceph-mon[38143]: pgmap v149: 129 pgs: 129 active+clean; 182 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 577 op/s 2024-03-14T09:59:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:03 smithi152 ceph-mon[48279]: pgmap v149: 129 pgs: 129 active+clean; 182 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 577 op/s 2024-03-14T09:59:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:03 smithi132 ceph-mon[47959]: pgmap v149: 129 pgs: 129 active+clean; 182 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 577 op/s 2024-03-14T09:59:03.841 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T09:59:03.841 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T09:59:04.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.088+0000 7f5b87e00700 1 -- 172.21.15.78:0/2415091547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b881025e0 msgr2=0x7f5b8810aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.088+0000 7f5b87e00700 1 --2- 172.21.15.78:0/2415091547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b881025e0 0x7f5b8810aac0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009990 tx=0x7f5b74009ca0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.088+0000 7f5b87e00700 1 -- 172.21.15.78:0/2415091547 shutdown_connections 2024-03-14T09:59:04.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.088+0000 7f5b87e00700 1 --2- 172.21.15.78:0/2415091547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b8810b0e0 0x7f5b8810d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.088+0000 7f5b87e00700 1 --2- 172.21.15.78:0/2415091547 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b88101cc0 0x7f5b881020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.088+0000 7f5b87e00700 1 --2- 172.21.15.78:0/2415091547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b881025e0 0x7f5b8810aac0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.088+0000 7f5b87e00700 1 -- 172.21.15.78:0/2415091547 >> 172.21.15.78:0/2415091547 conn(0x7f5b880fb4e0 msgr2=0x7f5b880fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:04.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.088+0000 7f5b87e00700 1 -- 172.21.15.78:0/2415091547 shutdown_connections 2024-03-14T09:59:04.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.088+0000 7f5b87e00700 1 -- 172.21.15.78:0/2415091547 wait complete. 2024-03-14T09:59:04.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.089+0000 7f5b87e00700 1 Processor -- start 2024-03-14T09:59:04.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.089+0000 7f5b87e00700 1 -- start start 2024-03-14T09:59:04.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b87e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b88101cc0 0x7f5b880ff910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b87e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b881025e0 0x7f5b880ffe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b87e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b8810b0e0 0x7f5b88100390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b87e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b88101890 con 0x7f5b881025e0 2024-03-14T09:59:04.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b87e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881a9e20 con 0x7f5b88101cc0 2024-03-14T09:59:04.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b87e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b881a9f90 con 0x7f5b8810b0e0 2024-03-14T09:59:04.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b87400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b8810b0e0 0x7f5b88100390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b86000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b881025e0 0x7f5b880ffe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b86a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b88101cc0 0x7f5b880ff910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b87400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b8810b0e0 0x7f5b88100390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37022/0 (socket says 172.21.15.78:37022) 2024-03-14T09:59:04.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b87400700 1 -- 172.21.15.78:0/1032979203 learned_addr learned my addr 172.21.15.78:0/1032979203 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:04.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b86a00700 1 -- 172.21.15.78:0/1032979203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b8810b0e0 msgr2=0x7f5b88100390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b86a00700 1 --2- 172.21.15.78:0/1032979203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b8810b0e0 0x7f5b88100390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b86a00700 1 -- 172.21.15.78:0/1032979203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b881025e0 msgr2=0x7f5b880ffe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b86a00700 1 --2- 172.21.15.78:0/1032979203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b881025e0 0x7f5b880ffe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b86a00700 1 -- 172.21.15.78:0/1032979203 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b78009520 con 0x7f5b88101cc0 2024-03-14T09:59:04.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.090+0000 7f5b87400700 1 --2- 172.21.15.78:0/1032979203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b8810b0e0 0x7f5b88100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:04.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.091+0000 7f5b86a00700 1 --2- 172.21.15.78:0/1032979203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b88101cc0 0x7f5b880ff910 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800e9f0 tx=0x7f5b7800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:04.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.091+0000 7f5b7f400700 1 -- 172.21.15.78:0/1032979203 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7800cb70 con 0x7f5b88101cc0 2024-03-14T09:59:04.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.091+0000 7f5b7f400700 1 -- 172.21.15.78:0/1032979203 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b7800ee90 con 0x7f5b88101cc0 2024-03-14T09:59:04.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.091+0000 7f5b87e00700 1 -- 172.21.15.78:0/1032979203 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b740095f0 con 0x7f5b88101cc0 2024-03-14T09:59:04.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.091+0000 7f5b7f400700 1 -- 172.21.15.78:0/1032979203 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78003cf0 con 0x7f5b88101cc0 2024-03-14T09:59:04.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.091+0000 7f5b87e00700 1 -- 172.21.15.78:0/1032979203 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b881aa5f0 con 0x7f5b88101cc0 2024-03-14T09:59:04.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.094+0000 7f5b7f400700 1 -- 172.21.15.78:0/1032979203 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f5b78004020 con 0x7f5b88101cc0 2024-03-14T09:59:04.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.095+0000 7f5b7f400700 1 --2- 172.21.15.78:0/1032979203 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5b6c0a0c00 0x7f5b6c0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.095+0000 7f5b7f400700 1 -- 172.21.15.78:0/1032979203 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f5b78014070 con 0x7f5b88101cc0 2024-03-14T09:59:04.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.095+0000 7f5b86000700 1 --2- 172.21.15.78:0/1032979203 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5b6c0a0c00 0x7f5b6c0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.096+0000 7f5ba103db80 1 Processor -- start 2024-03-14T09:59:04.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.096+0000 7f5ba103db80 1 -- start start 2024-03-14T09:59:04.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.096+0000 7f5b86000700 1 --2- 172.21.15.78:0/1032979203 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5b6c0a0c00 0x7f5b6c0a30c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5b88100ff0 tx=0x7f5b7400b260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:04.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.096+0000 7f5ba103db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71e8e0 0x5649db71ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.096+0000 7f5ba103db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71f200 0x5649db729b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.096+0000 7f5ba103db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db72a0b0 0x5649db72c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.096+0000 7f5ba103db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649db678a10 con 0x5649db72a0b0 2024-03-14T09:59:04.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.096+0000 7f5ba103db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649db6763d0 con 0x5649db71e8e0 2024-03-14T09:59:04.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.096+0000 7f5ba103db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649db671cc0 con 0x5649db71f200 2024-03-14T09:59:04.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.096+0000 7f5b86000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71f200 0x5649db729b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.096+0000 7f5b86a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71e8e0 0x5649db71ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.096+0000 7f5b86000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71f200 0x5649db729b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37038/0 (socket says 172.21.15.78:37038) 2024-03-14T09:59:04.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.096+0000 7f5b86000700 1 -- 172.21.15.78:0/2816263641 learned_addr learned my addr 172.21.15.78:0/2816263641 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:04.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.097+0000 7f5b87400700 1 --2- 172.21.15.78:0/2816263641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db72a0b0 0x5649db72c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.097+0000 7f5b86000700 1 -- 172.21.15.78:0/2816263641 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71e8e0 msgr2=0x5649db71ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:04.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.097+0000 7f5b86000700 1 --2- 172.21.15.78:0/2816263641 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71e8e0 0x5649db71ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.097+0000 7f5b86000700 1 -- 172.21.15.78:0/2816263641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db72a0b0 msgr2=0x5649db72c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:04.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.097+0000 7f5b86000700 1 --2- 172.21.15.78:0/2816263641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db72a0b0 0x5649db72c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.097+0000 7f5b86000700 1 -- 172.21.15.78:0/2816263641 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649db4f3430 con 0x5649db71f200 2024-03-14T09:59:04.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.097+0000 7f5b86000700 1 --2- 172.21.15.78:0/2816263641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71f200 0x5649db729b70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5b74015000 tx=0x7f5b74020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22fa05fe8d579e3e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:04.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.097+0000 7f5b73400700 1 -- 172.21.15.78:0/2816263641 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7400bde0 con 0x5649db71f200 2024-03-14T09:59:04.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.097+0000 7f5b73400700 1 -- 172.21.15.78:0/2816263641 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b7401e040 con 0x5649db71f200 2024-03-14T09:59:04.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.097+0000 7f5b73400700 1 -- 172.21.15.78:0/2816263641 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7400f750 con 0x5649db71f200 2024-03-14T09:59:04.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.098+0000 7f5ba103db80 1 -- 172.21.15.78:0/2816263641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71f200 msgr2=0x5649db729b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.098+0000 7f5ba103db80 1 --2- 172.21.15.78:0/2816263641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71f200 0x5649db729b70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5b74015000 tx=0x7f5b74020000 comp rx=0 tx=0).stop 2024-03-14T09:59:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.098+0000 7f5ba103db80 1 -- 172.21.15.78:0/2816263641 shutdown_connections 2024-03-14T09:59:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.098+0000 7f5ba103db80 1 --2- 172.21.15.78:0/2816263641 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71e8e0 0x5649db71ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.098+0000 7f5ba103db80 1 --2- 172.21.15.78:0/2816263641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db72a0b0 0x5649db72c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.098+0000 7f5ba103db80 1 --2- 172.21.15.78:0/2816263641 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71f200 0x5649db729b70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.098+0000 7f5ba103db80 1 -- 172.21.15.78:0/2816263641 >> 172.21.15.78:0/2816263641 conn(0x5649db6952a0 msgr2=0x5649db71ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.099+0000 7f5ba103db80 1 -- 172.21.15.78:0/2816263641 shutdown_connections 2024-03-14T09:59:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.099+0000 7f5ba103db80 1 -- 172.21.15.78:0/2816263641 wait complete. 2024-03-14T09:59:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.099+0000 7f5ba103db80 1 -- start start 2024-03-14T09:59:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.100+0000 7f5ba103db80 1 Processor -- start 2024-03-14T09:59:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.100+0000 7f5ba103db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71e8e0 0x5649db729a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.100+0000 7f5ba103db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71f200 0x5649db726d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.100+0000 7f5ba103db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db72a0b0 0x5649db727370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.100+0000 7f5ba103db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649db60de00 con 0x5649db72a0b0 2024-03-14T09:59:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.100+0000 7f5ba103db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649db60daa0 con 0x5649db71e8e0 2024-03-14T09:59:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.100+0000 7f5ba103db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649db4f5720 con 0x5649db71f200 2024-03-14T09:59:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.100+0000 7f5ba103db80 10 client.0 _ll_register_callbacks cb 0x5649db479420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.101+0000 7f5b86a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71e8e0 0x5649db729a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.101+0000 7f5b86a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71e8e0 0x5649db729a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:59898/0 (socket says 172.21.15.78:59898) 2024-03-14T09:59:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.101+0000 7f5b86a00700 1 -- 172.21.15.78:0/618280233 learned_addr learned my addr 172.21.15.78:0/618280233 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.101+0000 7f5b87400700 1 --2- 172.21.15.78:0/618280233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db72a0b0 0x5649db727370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.101+0000 7f5b86000700 1 --2- 172.21.15.78:0/618280233 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71f200 0x5649db726d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.101+0000 7f5b86a00700 1 -- 172.21.15.78:0/618280233 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71f200 msgr2=0x5649db726d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.101+0000 7f5b86a00700 1 --2- 172.21.15.78:0/618280233 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71f200 0x5649db726d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.101+0000 7f5b86a00700 1 -- 172.21.15.78:0/618280233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db72a0b0 msgr2=0x5649db727370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.101+0000 7f5b86a00700 1 --2- 172.21.15.78:0/618280233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db72a0b0 0x5649db727370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.101+0000 7f5b86a00700 1 -- 172.21.15.78:0/618280233 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8810e260 con 0x5649db71e8e0 2024-03-14T09:59:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.102+0000 7f5b86a00700 1 --2- 172.21.15.78:0/618280233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71e8e0 0x5649db729a50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5b78005cb0 tx=0x7f5b780994e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a223da8f37370dd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.102+0000 7f5b67e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:59:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.102+0000 7f5b67e00700 1 -- 172.21.15.78:0/618280233 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7809d8c0 con 0x5649db71e8e0 2024-03-14T09:59:04.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.102+0000 7f5b67e00700 1 -- 172.21.15.78:0/618280233 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f5b780a1f10 con 0x5649db71e8e0 2024-03-14T09:59:04.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.102+0000 7f5ba103db80 1 -- 172.21.15.78:0/618280233 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5649db7127a0 con 0x5649db71e8e0 2024-03-14T09:59:04.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.102+0000 7f5b67e00700 1 -- 172.21.15.78:0/618280233 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b780b2270 con 0x5649db71e8e0 2024-03-14T09:59:04.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.102+0000 7f5ba103db80 10 client.26678 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:04.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.102+0000 7f5ba103db80 1 -- 172.21.15.78:0/618280233 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5649db49fbe0 con 0x5649db71e8e0 2024-03-14T09:59:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.102+0000 7f5b67e00700 1 -- 172.21.15.78:0/618280233 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5649db7127a0 con 0x5649db71e8e0 2024-03-14T09:59:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.103+0000 7f5b67e00700 1 -- 172.21.15.78:0/618280233 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5b78099cf0 con 0x5649db71e8e0 2024-03-14T09:59:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.103+0000 7f5ba103db80 10 client.26678 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.103+0000 7f5ba103db80 10 client.26678 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:59:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.103+0000 7f5ba103db80 20 client.26678 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.103+0000 7f5ba103db80 1 --2- 172.21.15.78:0/618280233 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5649db739bd0 0x5649db73bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.103+0000 7f5ba103db80 4 client.26678 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:59:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.103+0000 7f5ba103db80 1 -- 172.21.15.78:0/618280233 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5649db66a780 con 0x5649db739bd0 2024-03-14T09:59:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.103+0000 7f5b86000700 1 --2- 172.21.15.78:0/618280233 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5649db739bd0 0x5649db73bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.104+0000 7f5b86000700 1 --2- 172.21.15.78:0/618280233 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5649db739bd0 0x5649db73bff0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a15d369b57fbda5 server_cookie=b4f606b2348e0de9 in_seq=0 out_seq=0 2024-03-14T09:59:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.104+0000 7f5b67e00700 10 client.26678 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.105+0000 7f5b67e00700 1 -- 172.21.15.78:0/618280233 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5649db66a780 con 0x5649db739bd0 2024-03-14T09:59:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.105+0000 7f5b67e00700 10 client.26678 handle_command_reply: tid=0 2024-03-14T09:59:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.105+0000 7f5ba103db80 1 client.26678 shutdown 2024-03-14T09:59:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.105+0000 7f5ba103db80 1 -- 172.21.15.78:0/618280233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71e8e0 msgr2=0x5649db729a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.105+0000 7f5ba103db80 1 --2- 172.21.15.78:0/618280233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71e8e0 0x5649db729a50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5b78005cb0 tx=0x7f5b780994e0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.105+0000 7f5ba103db80 1 -- 172.21.15.78:0/618280233 shutdown_connections 2024-03-14T09:59:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.105+0000 7f5ba103db80 1 --2- 172.21.15.78:0/618280233 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5649db739bd0 0x5649db73bff0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.105+0000 7f5ba103db80 1 --2- 172.21.15.78:0/618280233 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71e8e0 0x5649db729a50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.105+0000 7f5ba103db80 1 --2- 172.21.15.78:0/618280233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db72a0b0 0x5649db727370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.105+0000 7f5b67e00700 0 client.26678 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.105+0000 7f5ba103db80 1 --2- 172.21.15.78:0/618280233 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71f200 0x5649db726d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.105+0000 7f5ba103db80 1 -- 172.21.15.78:0/618280233 >> 172.21.15.78:0/618280233 conn(0x5649db6952a0 msgr2=0x5649db71a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.106+0000 7f5ba103db80 1 -- 172.21.15.78:0/618280233 shutdown_connections 2024-03-14T09:59:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.106+0000 7f5ba103db80 1 -- 172.21.15.78:0/618280233 wait complete. 2024-03-14T09:59:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.106+0000 7f5ba103db80 20 client.26678 trim_cache size 0 max 16384 2024-03-14T09:59:04.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.127+0000 7f5ba103db80 1 Processor -- start 2024-03-14T09:59:04.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.127+0000 7f5ba103db80 1 -- start start 2024-03-14T09:59:04.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.128+0000 7f5ba103db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db71e8e0 0x5649db721780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.128+0000 7f5ba103db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71f200 0x5649db725630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.128+0000 7f5ba103db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db72a0b0 0x5649db725b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.128+0000 7f5ba103db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649db720430 con 0x5649db71e8e0 2024-03-14T09:59:04.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.128+0000 7f5ba103db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649db71c600 con 0x5649db71f200 2024-03-14T09:59:04.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.128+0000 7f5ba103db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649db726b30 con 0x5649db72a0b0 2024-03-14T09:59:04.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.128+0000 7f5b86000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71f200 0x5649db725630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.128+0000 7f5b86a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db71e8e0 0x5649db721780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.128+0000 7f5b86000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71f200 0x5649db725630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:59914/0 (socket says 172.21.15.78:59914) 2024-03-14T09:59:04.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.128+0000 7f5b87400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db72a0b0 0x5649db725b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.128+0000 7f5b86000700 1 -- 172.21.15.78:0/2424838219 learned_addr learned my addr 172.21.15.78:0/2424838219 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.128+0000 7f5b86000700 1 -- 172.21.15.78:0/2424838219 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db72a0b0 msgr2=0x5649db725b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.128+0000 7f5b86000700 1 --2- 172.21.15.78:0/2424838219 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db72a0b0 0x5649db725b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.128+0000 7f5b86000700 1 -- 172.21.15.78:0/2424838219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db71e8e0 msgr2=0x5649db721780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.128+0000 7f5b86000700 1 --2- 172.21.15.78:0/2424838219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db71e8e0 0x5649db721780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.128+0000 7f5b86000700 1 -- 172.21.15.78:0/2424838219 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649db716a70 con 0x5649db71f200 2024-03-14T09:59:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.129+0000 7f5b86000700 1 --2- 172.21.15.78:0/2424838219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71f200 0x5649db725630 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5b74032040 tx=0x7f5b7401a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf58bd98d8af7e67 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.129+0000 7f5b73e00700 1 -- 172.21.15.78:0/2424838219 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7403d070 con 0x5649db71f200 2024-03-14T09:59:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.129+0000 7f5b73e00700 1 -- 172.21.15.78:0/2424838219 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b7403de10 con 0x5649db71f200 2024-03-14T09:59:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.129+0000 7f5b73e00700 1 -- 172.21.15.78:0/2424838219 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b74004a90 con 0x5649db71f200 2024-03-14T09:59:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.130+0000 7f5ba103db80 1 -- 172.21.15.78:0/2424838219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71f200 msgr2=0x5649db725630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.130+0000 7f5ba103db80 1 --2- 172.21.15.78:0/2424838219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71f200 0x5649db725630 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5b74032040 tx=0x7f5b7401a040 comp rx=0 tx=0).stop 2024-03-14T09:59:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.130+0000 7f5ba103db80 1 -- 172.21.15.78:0/2424838219 shutdown_connections 2024-03-14T09:59:04.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.130+0000 7f5ba103db80 1 --2- 172.21.15.78:0/2424838219 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db72a0b0 0x5649db725b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.130+0000 7f5ba103db80 1 --2- 172.21.15.78:0/2424838219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db71e8e0 0x5649db721780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.130+0000 7f5ba103db80 1 --2- 172.21.15.78:0/2424838219 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71f200 0x5649db725630 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.130+0000 7f5ba103db80 1 -- 172.21.15.78:0/2424838219 >> 172.21.15.78:0/2424838219 conn(0x5649db6952a0 msgr2=0x5649db7220b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:04.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.130+0000 7f5ba103db80 1 -- 172.21.15.78:0/2424838219 shutdown_connections 2024-03-14T09:59:04.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.130+0000 7f5ba103db80 1 -- 172.21.15.78:0/2424838219 wait complete. 2024-03-14T09:59:04.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.131+0000 7f5ba103db80 1 -- start start 2024-03-14T09:59:04.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.131+0000 7f5ba103db80 1 Processor -- start 2024-03-14T09:59:04.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5ba103db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71e8e0 0x5649db730b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5ba103db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71f200 0x5649db72f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5ba103db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db72a0b0 0x5649db72f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5ba103db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649db72fc90 con 0x5649db72a0b0 2024-03-14T09:59:04.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5ba103db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649db72fe00 con 0x5649db71f200 2024-03-14T09:59:04.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5ba103db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649db78b700 con 0x5649db71e8e0 2024-03-14T09:59:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5ba103db80 10 client.0 _ll_register_callbacks cb 0x7f5b78003680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5b86000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71f200 0x5649db72f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5b86a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71e8e0 0x5649db730b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5b86000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71f200 0x5649db72f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:59930/0 (socket says 172.21.15.78:59930) 2024-03-14T09:59:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5b86a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71e8e0 0x5649db730b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37054/0 (socket says 172.21.15.78:37054) 2024-03-14T09:59:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5b86000700 1 -- 172.21.15.78:0/3063744057 learned_addr learned my addr 172.21.15.78:0/3063744057 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5b87400700 1 --2- 172.21.15.78:0/3063744057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db72a0b0 0x5649db72f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5b86000700 1 -- 172.21.15.78:0/3063744057 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71e8e0 msgr2=0x5649db730b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5b86000700 1 --2- 172.21.15.78:0/3063744057 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71e8e0 0x5649db730b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5b86000700 1 -- 172.21.15.78:0/3063744057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db72a0b0 msgr2=0x5649db72f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5b86000700 1 --2- 172.21.15.78:0/3063744057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db72a0b0 0x5649db72f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5b86000700 1 -- 172.21.15.78:0/3063744057 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649db4f3430 con 0x5649db71f200 2024-03-14T09:59:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5b87400700 1 --2- 172.21.15.78:0/3063744057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db72a0b0 0x5649db72f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:59:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.132+0000 7f5b86a00700 1 --2- 172.21.15.78:0/3063744057 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71e8e0 0x5649db730b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.133+0000 7f5b86000700 1 --2- 172.21.15.78:0/3063744057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71f200 0x5649db72f070 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5b740098e0 tx=0x7f5b74005410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5e73373642cc842 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.133+0000 7f5b67e00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:59:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.133+0000 7f5b67e00700 1 -- 172.21.15.78:0/3063744057 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7403d040 con 0x5649db71f200 2024-03-14T09:59:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.133+0000 7f5ba103db80 1 -- 172.21.15.78:0/3063744057 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5b7401e040 con 0x5649db71f200 2024-03-14T09:59:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.133+0000 7f5b67e00700 1 -- 172.21.15.78:0/3063744057 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f5b7403d1a0 con 0x5649db71f200 2024-03-14T09:59:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.133+0000 7f5b67e00700 1 -- 172.21.15.78:0/3063744057 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b7403d380 con 0x5649db71f200 2024-03-14T09:59:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.133+0000 7f5ba103db80 10 client.26684 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.133+0000 7f5ba103db80 1 -- 172.21.15.78:0/3063744057 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5649db637f20 con 0x5649db71f200 2024-03-14T09:59:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.134+0000 7f5b67e00700 1 -- 172.21.15.78:0/3063744057 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b7401e040 con 0x5649db71f200 2024-03-14T09:59:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.134+0000 7f5b67e00700 1 -- 172.21.15.78:0/3063744057 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5b7400f4f0 con 0x5649db71f200 2024-03-14T09:59:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.134+0000 7f5ba103db80 10 client.26684 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.134+0000 7f5ba103db80 10 client.26684 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:59:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.134+0000 7f5ba103db80 20 client.26684 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.134+0000 7f5ba103db80 1 --2- 172.21.15.78:0/3063744057 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5649db78caa0 0x5649db78eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.134+0000 7f5ba103db80 4 client.26684 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T09:59:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.134+0000 7f5ba103db80 1 -- 172.21.15.78:0/3063744057 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5649db4c0d70 con 0x5649db78caa0 2024-03-14T09:59:04.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.134+0000 7f5b86a00700 1 --2- 172.21.15.78:0/3063744057 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5649db78caa0 0x5649db78eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.134+0000 7f5b86a00700 1 --2- 172.21.15.78:0/3063744057 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5649db78caa0 0x5649db78eec0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38e16867c99623b5 server_cookie=a7135b57b358e65c in_seq=0 out_seq=0 2024-03-14T09:59:04.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.134+0000 7f5b67e00700 10 client.26684 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.136+0000 7f5b67e00700 1 -- 172.21.15.78:0/3063744057 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5649db4c0d70 con 0x5649db78caa0 2024-03-14T09:59:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.136+0000 7f5b67e00700 10 client.26684 handle_command_reply: tid=0 2024-03-14T09:59:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.136+0000 7f5ba103db80 1 client.26684 shutdown 2024-03-14T09:59:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.136+0000 7f5ba103db80 1 -- 172.21.15.78:0/3063744057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71f200 msgr2=0x5649db72f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.136+0000 7f5ba103db80 1 --2- 172.21.15.78:0/3063744057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71f200 0x5649db72f070 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5b740098e0 tx=0x7f5b74005410 comp rx=0 tx=0).stop 2024-03-14T09:59:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.136+0000 7f5ba103db80 1 -- 172.21.15.78:0/3063744057 shutdown_connections 2024-03-14T09:59:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.136+0000 7f5ba103db80 1 --2- 172.21.15.78:0/3063744057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5649db72a0b0 0x5649db72f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.136+0000 7f5ba103db80 1 --2- 172.21.15.78:0/3063744057 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5649db71e8e0 0x5649db730b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.136+0000 7f5ba103db80 1 --2- 172.21.15.78:0/3063744057 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5649db78caa0 0x5649db78eec0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.136+0000 7f5ba103db80 1 --2- 172.21.15.78:0/3063744057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5649db71f200 0x5649db72f070 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.136+0000 7f5b67e00700 0 client.26684 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.136+0000 7f5ba103db80 1 -- 172.21.15.78:0/3063744057 >> 172.21.15.78:0/3063744057 conn(0x5649db6952a0 msgr2=0x5649db725570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.136+0000 7f5ba103db80 1 -- 172.21.15.78:0/3063744057 shutdown_connections 2024-03-14T09:59:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.136+0000 7f5ba103db80 1 -- 172.21.15.78:0/3063744057 wait complete. 2024-03-14T09:59:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.137+0000 7f5ba103db80 20 client.26684 trim_cache size 0 max 16384 2024-03-14T09:59:04.140 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:59:04.140 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T09:59:04.140 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "6cbff20d-c082-43b4-ab87-1da5de616897", 2024-03-14T09:59:04.140 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T09:59:04.140 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:59:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.137+0000 7f5b87e00700 1 -- 172.21.15.78:0/1032979203 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5b6c0a0c00 msgr2=0x7f5b6c0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.137+0000 7f5b87e00700 1 --2- 172.21.15.78:0/1032979203 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5b6c0a0c00 0x7f5b6c0a30c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5b88100ff0 tx=0x7f5b7400b260 comp rx=0 tx=0).stop 2024-03-14T09:59:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.137+0000 7f5b87e00700 1 -- 172.21.15.78:0/1032979203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b88101cc0 msgr2=0x7f5b880ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.137+0000 7f5b87e00700 1 --2- 172.21.15.78:0/1032979203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b88101cc0 0x7f5b880ff910 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800e9f0 tx=0x7f5b7800ed00 comp rx=0 tx=0).stop 2024-03-14T09:59:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.137+0000 7f5b87e00700 1 -- 172.21.15.78:0/1032979203 shutdown_connections 2024-03-14T09:59:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.137+0000 7f5b87e00700 1 --2- 172.21.15.78:0/1032979203 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5b6c0a0c00 0x7f5b6c0a30c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.137+0000 7f5b87e00700 1 --2- 172.21.15.78:0/1032979203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b88101cc0 0x7f5b880ff910 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.137+0000 7f5b87e00700 1 --2- 172.21.15.78:0/1032979203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b8810b0e0 0x7f5b88100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.137+0000 7f5b87e00700 1 --2- 172.21.15.78:0/1032979203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b881025e0 0x7f5b880ffe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.137+0000 7f5b87e00700 1 -- 172.21.15.78:0/1032979203 >> 172.21.15.78:0/1032979203 conn(0x7f5b880fb4e0 msgr2=0x7f5b880fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.138+0000 7f5b87e00700 1 -- 172.21.15.78:0/1032979203 shutdown_connections 2024-03-14T09:59:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.138+0000 7f5b87e00700 1 -- 172.21.15.78:0/1032979203 wait complete. 2024-03-14T09:59:04.160 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:59:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.423+0000 7fe154600700 1 -- 172.21.15.78:0/2981862000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe14c068740 msgr2=0x7fe14c10d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.423+0000 7fe154600700 1 --2- 172.21.15.78:0/2981862000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe14c068740 0x7fe14c10d1c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe1380098f0 tx=0x7fe138009c00 comp rx=0 tx=0).stop 2024-03-14T09:59:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.423+0000 7fe154600700 1 -- 172.21.15.78:0/2981862000 shutdown_connections 2024-03-14T09:59:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.423+0000 7fe154600700 1 --2- 172.21.15.78:0/2981862000 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe14c067e20 0x7fe14c068200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.423+0000 7fe154600700 1 --2- 172.21.15.78:0/2981862000 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe14c10d700 0x7fe14c10faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.423+0000 7fe154600700 1 --2- 172.21.15.78:0/2981862000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe14c068740 0x7fe14c10d1c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.423+0000 7fe154600700 1 -- 172.21.15.78:0/2981862000 >> 172.21.15.78:0/2981862000 conn(0x7fe14c076d80 msgr2=0x7fe14c077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:04.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.425+0000 7fe154600700 1 -- 172.21.15.78:0/2981862000 shutdown_connections 2024-03-14T09:59:04.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.425+0000 7fe154600700 1 -- 172.21.15.78:0/2981862000 wait complete. 2024-03-14T09:59:04.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.426+0000 7fe154600700 1 Processor -- start 2024-03-14T09:59:04.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.426+0000 7fe154600700 1 -- start start 2024-03-14T09:59:04.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.427+0000 7fe154600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe14c067e20 0x7fe14c101e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.427+0000 7fe154600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe14c068740 0x7fe14c102370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.427+0000 7fe154600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe14c10d700 0x7fe14c1028b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.427+0000 7fe154600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c1a9eb0 con 0x7fe14c067e20 2024-03-14T09:59:04.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.427+0000 7fe154600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c1a9ff0 con 0x7fe14c10d700 2024-03-14T09:59:04.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.427+0000 7fe154600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c1aa160 con 0x7fe14c068740 2024-03-14T09:59:04.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.427+0000 7fe151200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe14c068740 0x7fe14c102370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.427+0000 7fe152600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe14c10d700 0x7fe14c1028b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.427+0000 7fe151200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe14c068740 0x7fe14c102370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37072/0 (socket says 172.21.15.78:37072) 2024-03-14T09:59:04.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.427+0000 7fe151200700 1 -- 172.21.15.78:0/3688290788 learned_addr learned my addr 172.21.15.78:0/3688290788 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:04.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.428+0000 7fe151c00700 1 --2- 172.21.15.78:0/3688290788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe14c067e20 0x7fe14c101e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.428+0000 7fe152600700 1 -- 172.21.15.78:0/3688290788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe14c068740 msgr2=0x7fe14c102370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.428+0000 7fe152600700 1 --2- 172.21.15.78:0/3688290788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe14c068740 0x7fe14c102370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.428+0000 7fe152600700 1 -- 172.21.15.78:0/3688290788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe14c067e20 msgr2=0x7fe14c101e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.429+0000 7fe152600700 1 --2- 172.21.15.78:0/3688290788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe14c067e20 0x7fe14c101e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.429+0000 7fe152600700 1 -- 172.21.15.78:0/3688290788 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe144009c40 con 0x7fe14c10d700 2024-03-14T09:59:04.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.429+0000 7fe151c00700 1 --2- 172.21.15.78:0/3688290788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe14c067e20 0x7fe14c101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:59:04.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.429+0000 7fe152600700 1 --2- 172.21.15.78:0/3688290788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe14c10d700 0x7fe14c1028b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe14c06a210 tx=0x7fe14400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:04.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.429+0000 7fe14a000700 1 -- 172.21.15.78:0/3688290788 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14400ca50 con 0x7fe14c10d700 2024-03-14T09:59:04.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.429+0000 7fe14a000700 1 -- 172.21.15.78:0/3688290788 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe14400f040 con 0x7fe14c10d700 2024-03-14T09:59:04.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.429+0000 7fe154600700 1 -- 172.21.15.78:0/3688290788 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe138009520 con 0x7fe14c10d700 2024-03-14T09:59:04.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.430+0000 7fe14a000700 1 -- 172.21.15.78:0/3688290788 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1440045b0 con 0x7fe14c10d700 2024-03-14T09:59:04.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.430+0000 7fe154600700 1 -- 172.21.15.78:0/3688290788 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe14c1aa7c0 con 0x7fe14c10d700 2024-03-14T09:59:04.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.434+0000 7fe14a000700 1 -- 172.21.15.78:0/3688290788 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fe144007500 con 0x7fe14c10d700 2024-03-14T09:59:04.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.437+0000 7fe168100b80 1 Processor -- start 2024-03-14T09:59:04.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.437+0000 7fe168100b80 1 -- start start 2024-03-14T09:59:04.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.438+0000 7fe14a000700 1 --2- 172.21.15.78:0/3688290788 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe1340a0d50 0x7fe1340a3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.438+0000 7fe14a000700 1 -- 172.21.15.78:0/3688290788 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe1440cc2a0 con 0x7fe14c10d700 2024-03-14T09:59:04.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.438+0000 7fe151200700 1 --2- 172.21.15.78:0/3688290788 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe1340a0d50 0x7fe1340a3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.439+0000 7fe168100b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 0x5651dbf1ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.439+0000 7fe168100b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf1cfa0 0x5651dbf26980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.439+0000 7fe168100b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf26ec0 0x5651dbf292b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.439+0000 7fe168100b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651dbe75990 con 0x5651dbf26ec0 2024-03-14T09:59:04.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.439+0000 7fe168100b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651dbe73350 con 0x5651dbf1cfa0 2024-03-14T09:59:04.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.439+0000 7fe168100b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651dbe6ec40 con 0x5651dbf1c680 2024-03-14T09:59:04.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.439+0000 7fe151c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 0x5651dbf1ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.439+0000 7fe151c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 0x5651dbf1ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37082/0 (socket says 172.21.15.78:37082) 2024-03-14T09:59:04.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.439+0000 7fe151c00700 1 -- 172.21.15.78:0/862398376 learned_addr learned my addr 172.21.15.78:0/862398376 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:04.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.439+0000 7fe151200700 1 --2- 172.21.15.78:0/862398376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf1cfa0 0x5651dbf26980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.440+0000 7fe151200700 1 --2- 172.21.15.78:0/3688290788 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe1340a0d50 0x7fe1340a3210 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe14c103560 tx=0x7fe138019040 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:04.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.440+0000 7fe151c00700 1 -- 172.21.15.78:0/862398376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf1cfa0 msgr2=0x5651dbf26980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:04.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.440+0000 7fe151c00700 1 --2- 172.21.15.78:0/862398376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf1cfa0 0x5651dbf26980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.440+0000 7fe151c00700 1 -- 172.21.15.78:0/862398376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf26ec0 msgr2=0x5651dbf292b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:04.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.440+0000 7fe151c00700 1 --2- 172.21.15.78:0/862398376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf26ec0 0x5651dbf292b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.440+0000 7fe151c00700 1 -- 172.21.15.78:0/862398376 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651dbcf03b0 con 0x5651dbf1c680 2024-03-14T09:59:04.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.440+0000 7fe151200700 1 --2- 172.21.15.78:0/862398376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf1cfa0 0x5651dbf26980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:59:04.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.441+0000 7fe151c00700 1 --2- 172.21.15.78:0/862398376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 0x5651dbf1ca60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe140009cc0 tx=0x7fe140010370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=327873e3ba1fe2fa server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:04.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.441+0000 7fe13ea00700 1 -- 172.21.15.78:0/862398376 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140010c70 con 0x5651dbf1c680 2024-03-14T09:59:04.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.441+0000 7fe13ea00700 1 -- 172.21.15.78:0/862398376 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe1400044f0 con 0x5651dbf1c680 2024-03-14T09:59:04.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.441+0000 7fe13ea00700 1 -- 172.21.15.78:0/862398376 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140012450 con 0x5651dbf1c680 2024-03-14T09:59:04.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.442+0000 7fe168100b80 1 -- 172.21.15.78:0/862398376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 msgr2=0x5651dbf1ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.442+0000 7fe168100b80 1 --2- 172.21.15.78:0/862398376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 0x5651dbf1ca60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe140009cc0 tx=0x7fe140010370 comp rx=0 tx=0).stop 2024-03-14T09:59:04.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.442+0000 7fe168100b80 1 -- 172.21.15.78:0/862398376 shutdown_connections 2024-03-14T09:59:04.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.442+0000 7fe168100b80 1 --2- 172.21.15.78:0/862398376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf26ec0 0x5651dbf292b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.442+0000 7fe168100b80 1 --2- 172.21.15.78:0/862398376 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 0x5651dbf1ca60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.442+0000 7fe168100b80 1 --2- 172.21.15.78:0/862398376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf1cfa0 0x5651dbf26980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.442+0000 7fe168100b80 1 -- 172.21.15.78:0/862398376 >> 172.21.15.78:0/862398376 conn(0x5651dbe92220 msgr2=0x5651dbf17aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:04.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.442+0000 7fe168100b80 1 -- 172.21.15.78:0/862398376 shutdown_connections 2024-03-14T09:59:04.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.442+0000 7fe168100b80 1 -- 172.21.15.78:0/862398376 wait complete. 2024-03-14T09:59:04.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.443+0000 7fe168100b80 1 -- start start 2024-03-14T09:59:04.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.444+0000 7fe168100b80 1 Processor -- start 2024-03-14T09:59:04.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.445+0000 7fe168100b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 0x5651dbf23d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.445+0000 7fe168100b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf1cfa0 0x5651dbf242c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.445+0000 7fe168100b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf26ec0 0x5651dbf24800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.445+0000 7fe168100b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651dbe0ad80 con 0x5651dbf1cfa0 2024-03-14T09:59:04.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.445+0000 7fe168100b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651dbe0aa20 con 0x5651dbf26ec0 2024-03-14T09:59:04.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.445+0000 7fe168100b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651dbcf26a0 con 0x5651dbf1c680 2024-03-14T09:59:04.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.445+0000 7fe168100b80 10 client.0 _ll_register_callbacks cb 0x5651dbc763a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:04.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.445+0000 7fe152600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf26ec0 0x5651dbf24800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.446+0000 7fe152600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf26ec0 0x5651dbf24800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:59964/0 (socket says 172.21.15.78:59964) 2024-03-14T09:59:04.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.446+0000 7fe152600700 1 -- 172.21.15.78:0/1054709642 learned_addr learned my addr 172.21.15.78:0/1054709642 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:04.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.446+0000 7fe151c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 0x5651dbf23d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.446+0000 7fe151200700 1 --2- 172.21.15.78:0/1054709642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf1cfa0 0x5651dbf242c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.446+0000 7fe152600700 1 -- 172.21.15.78:0/1054709642 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 msgr2=0x5651dbf23d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:04.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.446+0000 7fe152600700 1 --2- 172.21.15.78:0/1054709642 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 0x5651dbf23d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.446+0000 7fe152600700 1 -- 172.21.15.78:0/1054709642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf1cfa0 msgr2=0x5651dbf242c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:04.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.446+0000 7fe152600700 1 --2- 172.21.15.78:0/1054709642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf1cfa0 0x5651dbf242c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.446+0000 7fe152600700 1 -- 172.21.15.78:0/1054709642 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1400098f0 con 0x5651dbf26ec0 2024-03-14T09:59:04.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.446+0000 7fe152600700 1 --2- 172.21.15.78:0/1054709642 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf26ec0 0x5651dbf24800 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe1440a1190 tx=0x7fe144095d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=196ac6e9fca7995d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:04.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.446+0000 7fe133400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:59:04.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.447+0000 7fe133400700 1 -- 172.21.15.78:0/1054709642 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe144099770 con 0x5651dbf26ec0 2024-03-14T09:59:04.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.447+0000 7fe133400700 1 -- 172.21.15.78:0/1054709642 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe14401b520 con 0x5651dbf26ec0 2024-03-14T09:59:04.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.447+0000 7fe168100b80 1 -- 172.21.15.78:0/1054709642 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5651dbf0f720 con 0x5651dbf26ec0 2024-03-14T09:59:04.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.447+0000 7fe133400700 1 -- 172.21.15.78:0/1054709642 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe144095750 con 0x5651dbf26ec0 2024-03-14T09:59:04.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.447+0000 7fe168100b80 10 client.26696 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:04.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.447+0000 7fe168100b80 1 -- 172.21.15.78:0/1054709642 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5651dbc9cb60 con 0x5651dbf26ec0 2024-03-14T09:59:04.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.447+0000 7fe133400700 1 -- 172.21.15.78:0/1054709642 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5651dbf0f720 con 0x5651dbf26ec0 2024-03-14T09:59:04.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.448+0000 7fe133400700 1 -- 172.21.15.78:0/1054709642 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe144098860 con 0x5651dbf26ec0 2024-03-14T09:59:04.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.448+0000 7fe168100b80 10 client.26696 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:04.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.448+0000 7fe168100b80 10 client.26696 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:59:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.448+0000 7fe168100b80 20 client.26696 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.448+0000 7fe168100b80 1 --2- 172.21.15.78:0/1054709642 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5651dbf36ad0 0x5651dbf38ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.448+0000 7fe168100b80 4 client.26696 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:59:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.448+0000 7fe168100b80 1 -- 172.21.15.78:0/1054709642 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5651dbe67700 con 0x5651dbf36ad0 2024-03-14T09:59:04.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.448+0000 7fe151c00700 1 --2- 172.21.15.78:0/1054709642 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5651dbf36ad0 0x5651dbf38ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.449+0000 7fe151c00700 1 --2- 172.21.15.78:0/1054709642 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5651dbf36ad0 0x5651dbf38ef0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f369a26d6ace2b8 server_cookie=7d70ae5b6bcd01dd in_seq=0 out_seq=0 2024-03-14T09:59:04.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.449+0000 7fe133400700 10 client.26696 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:04.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.449+0000 7fe133400700 1 -- 172.21.15.78:0/1054709642 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5651dbe67700 con 0x5651dbf36ad0 2024-03-14T09:59:04.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.449+0000 7fe133400700 10 client.26696 handle_command_reply: tid=0 2024-03-14T09:59:04.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.450+0000 7fe168100b80 1 client.26696 shutdown 2024-03-14T09:59:04.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.450+0000 7fe168100b80 1 -- 172.21.15.78:0/1054709642 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf26ec0 msgr2=0x5651dbf24800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.450+0000 7fe168100b80 1 --2- 172.21.15.78:0/1054709642 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf26ec0 0x5651dbf24800 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe1440a1190 tx=0x7fe144095d90 comp rx=0 tx=0).stop 2024-03-14T09:59:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.450+0000 7fe168100b80 1 -- 172.21.15.78:0/1054709642 shutdown_connections 2024-03-14T09:59:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.450+0000 7fe168100b80 1 --2- 172.21.15.78:0/1054709642 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5651dbf36ad0 0x5651dbf38ef0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.450+0000 7fe168100b80 1 --2- 172.21.15.78:0/1054709642 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf26ec0 0x5651dbf24800 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.450+0000 7fe168100b80 1 --2- 172.21.15.78:0/1054709642 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 0x5651dbf23d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.450+0000 7fe168100b80 1 --2- 172.21.15.78:0/1054709642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf1cfa0 0x5651dbf242c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.450+0000 7fe168100b80 1 -- 172.21.15.78:0/1054709642 >> 172.21.15.78:0/1054709642 conn(0x5651dbe92220 msgr2=0x5651dbf17690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.450+0000 7fe133400700 0 client.26696 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.450+0000 7fe168100b80 1 -- 172.21.15.78:0/1054709642 shutdown_connections 2024-03-14T09:59:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.450+0000 7fe168100b80 1 -- 172.21.15.78:0/1054709642 wait complete. 2024-03-14T09:59:04.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.450+0000 7fe168100b80 20 client.26696 trim_cache size 0 max 16384 2024-03-14T09:59:04.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.478+0000 7fe168100b80 1 Processor -- start 2024-03-14T09:59:04.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.478+0000 7fe168100b80 1 -- start start 2024-03-14T09:59:04.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.479+0000 7fe168100b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 0x5651dbf23550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.479+0000 7fe168100b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf1cfa0 0x5651dbf32120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.479+0000 7fe168100b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf26ec0 0x5651dbf32660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.479+0000 7fe151c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 0x5651dbf23550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.479+0000 7fe151c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 0x5651dbf23550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37100/0 (socket says 172.21.15.78:37100) 2024-03-14T09:59:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.480+0000 7fe151200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf1cfa0 0x5651dbf32120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.480+0000 7fe151c00700 1 -- 172.21.15.78:0/2239286355 learned_addr learned my addr 172.21.15.78:0/2239286355 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.480+0000 7fe152600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf26ec0 0x5651dbf32660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.480+0000 7fe168100b80 1 -- 172.21.15.78:0/2239286355 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651dbf23be0 con 0x5651dbf1cfa0 2024-03-14T09:59:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.480+0000 7fe168100b80 1 -- 172.21.15.78:0/2239286355 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651dbf1e0c0 con 0x5651dbf26ec0 2024-03-14T09:59:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.480+0000 7fe168100b80 1 -- 172.21.15.78:0/2239286355 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651dbf19060 con 0x5651dbf1c680 2024-03-14T09:59:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.480+0000 7fe152600700 1 -- 172.21.15.78:0/2239286355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 msgr2=0x5651dbf23550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.480+0000 7fe152600700 1 --2- 172.21.15.78:0/2239286355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 0x5651dbf23550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.480+0000 7fe152600700 1 -- 172.21.15.78:0/2239286355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf1cfa0 msgr2=0x5651dbf32120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.481+0000 7fe152600700 1 --2- 172.21.15.78:0/2239286355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf1cfa0 0x5651dbf32120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.481+0000 7fe152600700 1 -- 172.21.15.78:0/2239286355 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651dbe34ea0 con 0x5651dbf26ec0 2024-03-14T09:59:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.481+0000 7fe151200700 1 --2- 172.21.15.78:0/2239286355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf1cfa0 0x5651dbf32120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:59:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.482+0000 7fe152600700 1 --2- 172.21.15.78:0/2239286355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf26ec0 0x5651dbf32660 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe14409ffe0 tx=0x7fe144004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddded02326edc8d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.482+0000 7fe13f400700 1 -- 172.21.15.78:0/2239286355 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe144095750 con 0x5651dbf26ec0 2024-03-14T09:59:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.482+0000 7fe13f400700 1 -- 172.21.15.78:0/2239286355 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe144007e50 con 0x5651dbf26ec0 2024-03-14T09:59:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.482+0000 7fe13f400700 1 -- 172.21.15.78:0/2239286355 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1440999d0 con 0x5651dbf26ec0 2024-03-14T09:59:04.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.485+0000 7fe168100b80 1 -- 172.21.15.78:0/2239286355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf26ec0 msgr2=0x5651dbf32660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.485+0000 7fe168100b80 1 --2- 172.21.15.78:0/2239286355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf26ec0 0x5651dbf32660 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe14409ffe0 tx=0x7fe144004b40 comp rx=0 tx=0).stop 2024-03-14T09:59:04.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.486+0000 7fe168100b80 1 -- 172.21.15.78:0/2239286355 shutdown_connections 2024-03-14T09:59:04.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.486+0000 7fe168100b80 1 --2- 172.21.15.78:0/2239286355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf26ec0 0x5651dbf32660 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.486+0000 7fe168100b80 1 --2- 172.21.15.78:0/2239286355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf1c680 0x5651dbf23550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.486+0000 7fe168100b80 1 --2- 172.21.15.78:0/2239286355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf1cfa0 0x5651dbf32120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.486+0000 7fe168100b80 1 -- 172.21.15.78:0/2239286355 >> 172.21.15.78:0/2239286355 conn(0x5651dbe92220 msgr2=0x5651dbf1f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:04.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.489+0000 7fe168100b80 1 -- 172.21.15.78:0/2239286355 shutdown_connections 2024-03-14T09:59:04.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.489+0000 7fe168100b80 1 -- 172.21.15.78:0/2239286355 wait complete. 2024-03-14T09:59:04.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.490+0000 7fe168100b80 1 -- start start 2024-03-14T09:59:04.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.491+0000 7fe168100b80 1 Processor -- start 2024-03-14T09:59:04.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.492+0000 7fe168100b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf1c680 0x5651dbf2bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.492+0000 7fe168100b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf1cfa0 0x5651dbf2c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.492+0000 7fe168100b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf26ec0 0x5651dbf2c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.492+0000 7fe168100b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651dbf87680 con 0x5651dbf1c680 2024-03-14T09:59:04.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.492+0000 7fe168100b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651dbf877f0 con 0x5651dbf1cfa0 2024-03-14T09:59:04.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.492+0000 7fe168100b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651dbf879b0 con 0x5651dbf26ec0 2024-03-14T09:59:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.492+0000 7fe168100b80 10 client.0 _ll_register_callbacks cb 0x5651dbc763a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.492+0000 7fe152600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf26ec0 0x5651dbf2c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.492+0000 7fe152600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf26ec0 0x5651dbf2c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:37110/0 (socket says 172.21.15.78:37110) 2024-03-14T09:59:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.492+0000 7fe152600700 1 -- 172.21.15.78:0/1437471277 learned_addr learned my addr 172.21.15.78:0/1437471277 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.492+0000 7fe151c00700 1 --2- 172.21.15.78:0/1437471277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf1c680 0x5651dbf2bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.492+0000 7fe151200700 1 --2- 172.21.15.78:0/1437471277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf1cfa0 0x5651dbf2c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.493+0000 7fe152600700 1 -- 172.21.15.78:0/1437471277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf1cfa0 msgr2=0x5651dbf2c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.493+0000 7fe152600700 1 --2- 172.21.15.78:0/1437471277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf1cfa0 0x5651dbf2c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.493+0000 7fe152600700 1 -- 172.21.15.78:0/1437471277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf1c680 msgr2=0x5651dbf2bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.493+0000 7fe152600700 1 --2- 172.21.15.78:0/1437471277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf1c680 0x5651dbf2bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.493+0000 7fe152600700 1 -- 172.21.15.78:0/1437471277 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5651dbc9cb60 con 0x5651dbf26ec0 2024-03-14T09:59:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.493+0000 7fe151200700 1 --2- 172.21.15.78:0/1437471277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf1cfa0 0x5651dbf2c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:59:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.493+0000 7fe152600700 1 --2- 172.21.15.78:0/1437471277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf26ec0 0x5651dbf2c5b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe144014520 tx=0x7fe14400cbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12dcbbd867ee6b15 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.493+0000 7fe133400700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:59:04.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.493+0000 7fe133400700 1 -- 172.21.15.78:0/1437471277 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14401b520 con 0x5651dbf26ec0 2024-03-14T09:59:04.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.493+0000 7fe133400700 1 -- 172.21.15.78:0/1437471277 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe144095750 con 0x5651dbf26ec0 2024-03-14T09:59:04.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.494+0000 7fe168100b80 1 -- 172.21.15.78:0/1437471277 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe1400044f0 con 0x5651dbf26ec0 2024-03-14T09:59:04.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.494+0000 7fe133400700 1 -- 172.21.15.78:0/1437471277 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe144098eb0 con 0x5651dbf26ec0 2024-03-14T09:59:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.494+0000 7fe168100b80 10 client.26686 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.494+0000 7fe168100b80 1 -- 172.21.15.78:0/1437471277 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5651dbcf4cd0 con 0x5651dbf26ec0 2024-03-14T09:59:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.494+0000 7fe133400700 1 -- 172.21.15.78:0/1437471277 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe1400044f0 con 0x5651dbf26ec0 2024-03-14T09:59:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.494+0000 7fe133400700 1 -- 172.21.15.78:0/1437471277 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe144081070 con 0x5651dbf26ec0 2024-03-14T09:59:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.494+0000 7fe168100b80 10 client.26686 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.494+0000 7fe168100b80 10 client.26686 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:59:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.494+0000 7fe168100b80 20 client.26686 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.494+0000 7fe168100b80 1 --2- 172.21.15.78:0/1437471277 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5651dbf88cb0 0x5651dbf8b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.494+0000 7fe168100b80 4 client.26686 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:59:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.494+0000 7fe168100b80 1 -- 172.21.15.78:0/1437471277 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5651dbcbdcf0 con 0x5651dbf88cb0 2024-03-14T09:59:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.495+0000 7fe151c00700 1 --2- 172.21.15.78:0/1437471277 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5651dbf88cb0 0x5651dbf8b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.495+0000 7fe151c00700 1 --2- 172.21.15.78:0/1437471277 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5651dbf88cb0 0x5651dbf8b0a0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2575311acc016ae1 server_cookie=4da578f01eb91b9e in_seq=0 out_seq=0 2024-03-14T09:59:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.495+0000 7fe133400700 10 client.26686 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.496+0000 7fe133400700 1 -- 172.21.15.78:0/1437471277 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x5651dbcbdcf0 con 0x5651dbf88cb0 2024-03-14T09:59:04.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.496+0000 7fe133400700 10 client.26686 handle_command_reply: tid=0 2024-03-14T09:59:04.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.496+0000 7fe168100b80 1 client.26686 shutdown 2024-03-14T09:59:04.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.496+0000 7fe168100b80 1 -- 172.21.15.78:0/1437471277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf26ec0 msgr2=0x5651dbf2c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.496+0000 7fe168100b80 1 --2- 172.21.15.78:0/1437471277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf26ec0 0x5651dbf2c5b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe144014520 tx=0x7fe14400cbb0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.496+0000 7fe168100b80 1 -- 172.21.15.78:0/1437471277 shutdown_connections 2024-03-14T09:59:04.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.496+0000 7fe168100b80 1 --2- 172.21.15.78:0/1437471277 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5651dbf26ec0 0x5651dbf2c5b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.496+0000 7fe168100b80 1 --2- 172.21.15.78:0/1437471277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651dbf1c680 0x5651dbf2bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.496+0000 7fe168100b80 1 --2- 172.21.15.78:0/1437471277 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5651dbf88cb0 0x5651dbf8b0a0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.496+0000 7fe168100b80 1 --2- 172.21.15.78:0/1437471277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5651dbf1cfa0 0x5651dbf2c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.496+0000 7fe133400700 0 client.26686 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:04.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.496+0000 7fe168100b80 1 -- 172.21.15.78:0/1437471277 >> 172.21.15.78:0/1437471277 conn(0x5651dbe92220 msgr2=0x5651dbf32120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:04.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.497+0000 7fe168100b80 1 -- 172.21.15.78:0/1437471277 shutdown_connections 2024-03-14T09:59:04.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.497+0000 7fe168100b80 1 -- 172.21.15.78:0/1437471277 wait complete. 2024-03-14T09:59:04.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.497+0000 7fe168100b80 20 client.26686 trim_cache size 0 max 16384 2024-03-14T09:59:04.499 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:59:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "scrub active (1374 inodes in the stack)", 2024-03-14T09:59:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T09:59:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "6cbff20d-c082-43b4-ab87-1da5de616897": { 2024-03-14T09:59:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T09:59:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "6cbff20d-c082-43b4-ab87-1da5de616897", 2024-03-14T09:59:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T09:59:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:59:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:59:04.500 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:59:04.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.497+0000 7fe154600700 1 -- 172.21.15.78:0/3688290788 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe1340a0d50 msgr2=0x7fe1340a3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.497+0000 7fe154600700 1 --2- 172.21.15.78:0/3688290788 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe1340a0d50 0x7fe1340a3210 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe14c103560 tx=0x7fe138019040 comp rx=0 tx=0).stop 2024-03-14T09:59:04.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.497+0000 7fe154600700 1 -- 172.21.15.78:0/3688290788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe14c10d700 msgr2=0x7fe14c1028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:04.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.497+0000 7fe154600700 1 --2- 172.21.15.78:0/3688290788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe14c10d700 0x7fe14c1028b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe14c06a210 tx=0x7fe14400c3a0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.498+0000 7fe154600700 1 -- 172.21.15.78:0/3688290788 shutdown_connections 2024-03-14T09:59:04.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.498+0000 7fe154600700 1 --2- 172.21.15.78:0/3688290788 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe1340a0d50 0x7fe1340a3210 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.498+0000 7fe154600700 1 --2- 172.21.15.78:0/3688290788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe14c10d700 0x7fe14c1028b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.498+0000 7fe154600700 1 --2- 172.21.15.78:0/3688290788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe14c067e20 0x7fe14c101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.498+0000 7fe154600700 1 --2- 172.21.15.78:0/3688290788 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe14c068740 0x7fe14c102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:04.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.498+0000 7fe154600700 1 -- 172.21.15.78:0/3688290788 >> 172.21.15.78:0/3688290788 conn(0x7fe14c076d80 msgr2=0x7fe14c10eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:04.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.498+0000 7fe154600700 1 -- 172.21.15.78:0/3688290788 shutdown_connections 2024-03-14T09:59:04.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:04.498+0000 7fe154600700 1 -- 172.21.15.78:0/3688290788 wait complete. 2024-03-14T09:59:04.534 INFO:tasks.cephfs.filesystem:scrub status for tag:6cbff20d-c082-43b4-ab87-1da5de616897 - {'path': '/', 'tag': '6cbff20d-c082-43b4-ab87-1da5de616897', 'options': 'recursive,force'} 2024-03-14T09:59:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:05 smithi152 ceph-mon[48279]: pgmap v150: 129 pgs: 129 active+clean; 182 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.1 MiB/s wr, 378 op/s 2024-03-14T09:59:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:05 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T09:59:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:05 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:59:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:05 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T09:59:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:05 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:59:05.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:05 smithi132 ceph-mon[47959]: pgmap v150: 129 pgs: 129 active+clean; 182 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.1 MiB/s wr, 378 op/s 2024-03-14T09:59:05.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:05 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T09:59:05.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:05 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:59:05.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:05 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T09:59:05.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:05 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:59:05.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:05 smithi078 ceph-mon[38143]: pgmap v150: 129 pgs: 129 active+clean; 182 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.1 MiB/s wr, 378 op/s 2024-03-14T09:59:05.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:05 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T09:59:05.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:05 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:59:05.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:05 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T09:59:05.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:05 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:59:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:07 smithi152 ceph-mon[48279]: pgmap v151: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 1.8 MiB/s wr, 582 op/s 2024-03-14T09:59:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:07 smithi132 ceph-mon[47959]: pgmap v151: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 1.8 MiB/s wr, 582 op/s 2024-03-14T09:59:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:07 smithi078 ceph-mon[38143]: pgmap v151: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 1.8 MiB/s wr, 582 op/s 2024-03-14T09:59:09.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:08 smithi152 ceph-mon[48279]: pgmap v152: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 1.2 MiB/s wr, 386 op/s 2024-03-14T09:59:09.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:08 smithi132 ceph-mon[47959]: pgmap v152: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 1.2 MiB/s wr, 386 op/s 2024-03-14T09:59:09.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:08 smithi078 ceph-mon[38143]: pgmap v152: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 1.2 MiB/s wr, 386 op/s 2024-03-14T09:59:10.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:09 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T09:59:10.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:09 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T09:59:10.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:09 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T09:59:11.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:10 smithi152 ceph-mon[48279]: pgmap v153: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 1.2 MiB/s wr, 386 op/s 2024-03-14T09:59:11.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:10 smithi132 ceph-mon[47959]: pgmap v153: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 1.2 MiB/s wr, 386 op/s 2024-03-14T09:59:11.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:10 smithi078 ceph-mon[38143]: pgmap v153: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 1.2 MiB/s wr, 386 op/s 2024-03-14T09:59:13.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:13 smithi152 ceph-mon[48279]: pgmap v154: 129 pgs: 129 active+clean; 233 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 570 op/s 2024-03-14T09:59:13.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:13 smithi132 ceph-mon[47959]: pgmap v154: 129 pgs: 129 active+clean; 233 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 570 op/s 2024-03-14T09:59:13.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:13 smithi078 ceph-mon[38143]: pgmap v154: 129 pgs: 129 active+clean; 233 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 570 op/s 2024-03-14T09:59:15.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:15 smithi152 ceph-mon[48279]: pgmap v155: 129 pgs: 129 active+clean; 233 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 1.2 MiB/s wr, 386 op/s 2024-03-14T09:59:15.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:15 smithi132 ceph-mon[47959]: pgmap v155: 129 pgs: 129 active+clean; 233 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 1.2 MiB/s wr, 386 op/s 2024-03-14T09:59:15.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:15 smithi078 ceph-mon[38143]: pgmap v155: 129 pgs: 129 active+clean; 233 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 1.2 MiB/s wr, 386 op/s 2024-03-14T09:59:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:17 smithi152 ceph-mon[48279]: pgmap v156: 129 pgs: 129 active+clean; 226 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 571 op/s 2024-03-14T09:59:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:17 smithi132 ceph-mon[47959]: pgmap v156: 129 pgs: 129 active+clean; 226 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 571 op/s 2024-03-14T09:59:17.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:17 smithi078 ceph-mon[38143]: pgmap v156: 129 pgs: 129 active+clean; 226 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 571 op/s 2024-03-14T09:59:19.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:18 smithi152 ceph-mon[48279]: pgmap v157: 129 pgs: 129 active+clean; 226 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 1.2 MiB/s wr, 368 op/s 2024-03-14T09:59:19.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:18 smithi132 ceph-mon[47959]: pgmap v157: 129 pgs: 129 active+clean; 226 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 1.2 MiB/s wr, 368 op/s 2024-03-14T09:59:19.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:18 smithi078 ceph-mon[38143]: pgmap v157: 129 pgs: 129 active+clean; 226 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 1.2 MiB/s wr, 368 op/s 2024-03-14T09:59:20.509 INFO:tasks.workunit.client.0.smithi078.stdout:copying to largest ever: 0x63fd4d5 2024-03-14T09:59:21.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:21 smithi152 ceph-mon[48279]: pgmap v158: 129 pgs: 129 active+clean; 226 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 1.2 MiB/s wr, 368 op/s 2024-03-14T09:59:21.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:21 smithi132 ceph-mon[47959]: pgmap v158: 129 pgs: 129 active+clean; 226 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 1.2 MiB/s wr, 368 op/s 2024-03-14T09:59:21.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:21 smithi078 ceph-mon[38143]: pgmap v158: 129 pgs: 129 active+clean; 226 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 1.2 MiB/s wr, 368 op/s 2024-03-14T09:59:22.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:22 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:59:22.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:22 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:59:22.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:22 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:59:22.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:22 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:59:22.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:22 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:59:22.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:22 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:59:22.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:22 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:59:22.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:22 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:59:22.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:22 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T09:59:22.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:22 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T09:59:22.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:22 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T09:59:22.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:22 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T09:59:23.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:23 smithi152 ceph-mon[48279]: pgmap v159: 129 pgs: 129 active+clean; 202 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 571 op/s 2024-03-14T09:59:23.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:23 smithi132 ceph-mon[47959]: pgmap v159: 129 pgs: 129 active+clean; 202 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 571 op/s 2024-03-14T09:59:23.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:23 smithi078 ceph-mon[38143]: pgmap v159: 129 pgs: 129 active+clean; 202 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 571 op/s 2024-03-14T09:59:25.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:25 smithi152 ceph-mon[48279]: pgmap v160: 129 pgs: 129 active+clean; 202 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 1.1 MiB/s wr, 388 op/s 2024-03-14T09:59:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:25 smithi132 ceph-mon[47959]: pgmap v160: 129 pgs: 129 active+clean; 202 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 1.1 MiB/s wr, 388 op/s 2024-03-14T09:59:25.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:25 smithi078 ceph-mon[38143]: pgmap v160: 129 pgs: 129 active+clean; 202 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 1.1 MiB/s wr, 388 op/s 2024-03-14T09:59:27.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:27 smithi152 ceph-mon[48279]: pgmap v161: 129 pgs: 129 active+clean; 204 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 573 op/s 2024-03-14T09:59:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:27 smithi132 ceph-mon[47959]: pgmap v161: 129 pgs: 129 active+clean; 204 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 573 op/s 2024-03-14T09:59:27.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:27 smithi078 ceph-mon[38143]: pgmap v161: 129 pgs: 129 active+clean; 204 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 573 op/s 2024-03-14T09:59:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:28 smithi152 ceph-mon[48279]: pgmap v162: 129 pgs: 129 active+clean; 204 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.0 MiB/s wr, 388 op/s 2024-03-14T09:59:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:28 smithi132 ceph-mon[47959]: pgmap v162: 129 pgs: 129 active+clean; 204 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.0 MiB/s wr, 388 op/s 2024-03-14T09:59:29.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:28 smithi078 ceph-mon[38143]: pgmap v162: 129 pgs: 129 active+clean; 204 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.0 MiB/s wr, 388 op/s 2024-03-14T09:59:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:31 smithi152 ceph-mon[48279]: pgmap v163: 129 pgs: 129 active+clean; 204 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.0 MiB/s wr, 388 op/s 2024-03-14T09:59:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:31 smithi132 ceph-mon[47959]: pgmap v163: 129 pgs: 129 active+clean; 204 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.0 MiB/s wr, 388 op/s 2024-03-14T09:59:31.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:31 smithi078 ceph-mon[38143]: pgmap v163: 129 pgs: 129 active+clean; 204 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.0 MiB/s wr, 388 op/s 2024-03-14T09:59:33.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:33 smithi152 ceph-mon[48279]: pgmap v164: 129 pgs: 129 active+clean; 217 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 1.6 MiB/s wr, 576 op/s 2024-03-14T09:59:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:33 smithi132 ceph-mon[47959]: pgmap v164: 129 pgs: 129 active+clean; 217 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 1.6 MiB/s wr, 576 op/s 2024-03-14T09:59:33.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:33 smithi078 ceph-mon[38143]: pgmap v164: 129 pgs: 129 active+clean; 217 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 1.6 MiB/s wr, 576 op/s 2024-03-14T09:59:34.535 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:59:34.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.793+0000 7fe5ace00700 1 -- 172.21.15.78:0/1863577564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5a8111cc0 msgr2=0x7fe5a81140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:34.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.793+0000 7fe5ace00700 1 --2- 172.21.15.78:0/1863577564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5a8111cc0 0x7fe5a81140b0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe598009f90 tx=0x7fe59800b330 comp rx=0 tx=0).stop 2024-03-14T09:59:34.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.793+0000 7fe5ace00700 1 -- 172.21.15.78:0/1863577564 shutdown_connections 2024-03-14T09:59:34.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.793+0000 7fe5ace00700 1 --2- 172.21.15.78:0/1863577564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5a8111cc0 0x7fe5a81140b0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.793+0000 7fe5ace00700 1 --2- 172.21.15.78:0/1863577564 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5a8073150 0x7fe5a8073530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.793+0000 7fe5ace00700 1 --2- 172.21.15.78:0/1863577564 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5a8073a70 0x7fe5a8111780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.793+0000 7fe5ace00700 1 -- 172.21.15.78:0/1863577564 >> 172.21.15.78:0/1863577564 conn(0x7fe5a80fc7a0 msgr2=0x7fe5a80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:34.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.794+0000 7fe5ace00700 1 -- 172.21.15.78:0/1863577564 shutdown_connections 2024-03-14T09:59:34.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.795+0000 7fe5ace00700 1 -- 172.21.15.78:0/1863577564 wait complete. 2024-03-14T09:59:34.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.797+0000 7fe5ace00700 1 Processor -- start 2024-03-14T09:59:34.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.797+0000 7fe5ace00700 1 -- start start 2024-03-14T09:59:34.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.798+0000 7fe5ace00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5a8073150 0x7fe5a81a3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.798+0000 7fe5ace00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5a8073a70 0x7fe5a81a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.798+0000 7fe5ace00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5a8111cc0 0x7fe5a819db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.798+0000 7fe5ace00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a819e1f0 con 0x7fe5a8073150 2024-03-14T09:59:34.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.798+0000 7fe5ace00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a819e360 con 0x7fe5a8073a70 2024-03-14T09:59:34.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.798+0000 7fe5ace00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a819e520 con 0x7fe5a8111cc0 2024-03-14T09:59:34.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.798+0000 7fe5a5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5a8073a70 0x7fe5a81a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.798+0000 7fe5a5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5a8073150 0x7fe5a81a3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.798+0000 7fe5a6800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5a8111cc0 0x7fe5a819db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.798+0000 7fe5a5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5a8073a70 0x7fe5a81a3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:43800/0 (socket says 172.21.15.78:43800) 2024-03-14T09:59:34.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.798+0000 7fe5a5400700 1 -- 172.21.15.78:0/746412332 learned_addr learned my addr 172.21.15.78:0/746412332 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:34.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.799+0000 7fe5a5400700 1 -- 172.21.15.78:0/746412332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5a8111cc0 msgr2=0x7fe5a819db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:34.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.799+0000 7fe5a5400700 1 --2- 172.21.15.78:0/746412332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5a8111cc0 0x7fe5a819db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.799+0000 7fe5a5400700 1 -- 172.21.15.78:0/746412332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5a8073150 msgr2=0x7fe5a81a3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:34.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.799+0000 7fe5a5400700 1 --2- 172.21.15.78:0/746412332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5a8073150 0x7fe5a81a3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.799+0000 7fe5a5400700 1 -- 172.21.15.78:0/746412332 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a00095f0 con 0x7fe5a8073a70 2024-03-14T09:59:34.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.799+0000 7fe5a6800700 1 --2- 172.21.15.78:0/746412332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5a8111cc0 0x7fe5a819db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:34.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.799+0000 7fe5a5e00700 1 --2- 172.21.15.78:0/746412332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5a8073150 0x7fe5a81a3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:34.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.799+0000 7fe5a5400700 1 --2- 172.21.15.78:0/746412332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5a8073a70 0x7fe5a81a3fa0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe5a000ec10 tx=0x7fe5a000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:34.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.799+0000 7fe59ea00700 1 -- 172.21.15.78:0/746412332 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a000cc90 con 0x7fe5a8073a70 2024-03-14T09:59:34.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.799+0000 7fe5ace00700 1 -- 172.21.15.78:0/746412332 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe598009c40 con 0x7fe5a8073a70 2024-03-14T09:59:34.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.799+0000 7fe5ace00700 1 -- 172.21.15.78:0/746412332 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a81aa870 con 0x7fe5a8073a70 2024-03-14T09:59:34.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.799+0000 7fe59ea00700 1 -- 172.21.15.78:0/746412332 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5a00044e0 con 0x7fe5a8073a70 2024-03-14T09:59:34.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.800+0000 7fe59ea00700 1 -- 172.21.15.78:0/746412332 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0010430 con 0x7fe5a8073a70 2024-03-14T09:59:34.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.802+0000 7fe59ea00700 1 -- 172.21.15.78:0/746412332 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fe5a0010590 con 0x7fe5a8073a70 2024-03-14T09:59:34.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.804+0000 7fe59ea00700 1 --2- 172.21.15.78:0/746412332 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe58c0a0c40 0x7fe58c0a3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.804+0000 7fe59ea00700 1 -- 172.21.15.78:0/746412332 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe5a0014070 con 0x7fe5a8073a70 2024-03-14T09:59:34.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.804+0000 7fe5c08ddb80 1 Processor -- start 2024-03-14T09:59:34.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.804+0000 7fe5a5e00700 1 --2- 172.21.15.78:0/746412332 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe58c0a0c40 0x7fe58c0a3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.804+0000 7fe5c08ddb80 1 -- start start 2024-03-14T09:59:34.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.805+0000 7fe5c08ddb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 0x55b5ae1bda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.805+0000 7fe5c08ddb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1c7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.805+0000 7fe5c08ddb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.805+0000 7fe5c08ddb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ae116990 con 0x55b5ae1c7ec0 2024-03-14T09:59:34.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.805+0000 7fe5c08ddb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ae114350 con 0x55b5ae1bdfa0 2024-03-14T09:59:34.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.805+0000 7fe5c08ddb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ae10fc40 con 0x55b5ae1bd680 2024-03-14T09:59:34.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.805+0000 7fe5a5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.805+0000 7fe5a5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1c7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:43812/0 (socket says 172.21.15.78:43812) 2024-03-14T09:59:34.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.805+0000 7fe5a5400700 1 -- 172.21.15.78:0/714499238 learned_addr learned my addr 172.21.15.78:0/714499238 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:34.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.805+0000 7fe5a6800700 1 --2- 172.21.15.78:0/714499238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.805+0000 7fe5a5e00700 1 --2- 172.21.15.78:0/746412332 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe58c0a0c40 0x7fe58c0a3100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe590005950 tx=0x7fe59000b410 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:34.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.805+0000 7fe5a5e00700 1 --2- 172.21.15.78:0/714499238 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 0x55b5ae1bda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.806+0000 7fe5a5400700 1 -- 172.21.15.78:0/714499238 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 msgr2=0x55b5ae1bda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.806+0000 7fe5a5400700 1 --2- 172.21.15.78:0/714499238 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 0x55b5ae1bda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.806+0000 7fe5a5400700 1 -- 172.21.15.78:0/714499238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 msgr2=0x55b5ae1ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.806+0000 7fe5a5400700 1 --2- 172.21.15.78:0/714499238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.806+0000 7fe5a5400700 1 -- 172.21.15.78:0/714499238 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5adf913b0 con 0x55b5ae1bdfa0 2024-03-14T09:59:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.806+0000 7fe5a6800700 1 --2- 172.21.15.78:0/714499238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:59:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.806+0000 7fe5a5400700 1 --2- 172.21.15.78:0/714499238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1c7980 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe5a00a1b50 tx=0x7fe5a009a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54c887e11f80c79a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.806+0000 7fe597400700 1 -- 172.21.15.78:0/714499238 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a00a2080 con 0x55b5ae1bdfa0 2024-03-14T09:59:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.806+0000 7fe597400700 1 -- 172.21.15.78:0/714499238 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5a001fde0 con 0x55b5ae1bdfa0 2024-03-14T09:59:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.806+0000 7fe597400700 1 -- 172.21.15.78:0/714499238 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a009b010 con 0x55b5ae1bdfa0 2024-03-14T09:59:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.807+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/714499238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 msgr2=0x55b5ae1c7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.807+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/714499238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1c7980 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe5a00a1b50 tx=0x7fe5a009a710 comp rx=0 tx=0).stop 2024-03-14T09:59:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.807+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/714499238 shutdown_connections 2024-03-14T09:59:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.807+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/714499238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.807+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/714499238 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 0x55b5ae1bda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.807+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/714499238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1c7980 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.807+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/714499238 >> 172.21.15.78:0/714499238 conn(0x55b5ae133220 msgr2=0x55b5ae1b8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.807+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/714499238 shutdown_connections 2024-03-14T09:59:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.807+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/714499238 wait complete. 2024-03-14T09:59:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.807+0000 7fe5c08ddb80 1 -- start start 2024-03-14T09:59:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.808+0000 7fe5c08ddb80 1 Processor -- start 2024-03-14T09:59:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.808+0000 7fe5c08ddb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 0x55b5ae1c7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.808+0000 7fe5c08ddb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.808+0000 7fe5c08ddb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.808+0000 7fe5c08ddb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ae0abd80 con 0x55b5ae1c7ec0 2024-03-14T09:59:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.808+0000 7fe5c08ddb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ae0aba20 con 0x55b5ae1bdfa0 2024-03-14T09:59:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.808+0000 7fe5c08ddb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5adf936a0 con 0x55b5ae1bd680 2024-03-14T09:59:34.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.808+0000 7fe5c08ddb80 10 client.0 _ll_register_callbacks cb 0x55b5adf173a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:34.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.808+0000 7fe5a6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.808+0000 7fe5a5e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 0x55b5ae1c7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.808+0000 7fe5a5e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 0x55b5ae1c7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60096/0 (socket says 172.21.15.78:60096) 2024-03-14T09:59:34.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.809+0000 7fe5a5e00700 1 -- 172.21.15.78:0/389747681 learned_addr learned my addr 172.21.15.78:0/389747681 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:34.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.809+0000 7fe5a5400700 1 --2- 172.21.15.78:0/389747681 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.809+0000 7fe5a6800700 1 -- 172.21.15.78:0/389747681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 msgr2=0x55b5ae1c7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:34.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.809+0000 7fe5a6800700 1 --2- 172.21.15.78:0/389747681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 0x55b5ae1c7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.809+0000 7fe5a6800700 1 -- 172.21.15.78:0/389747681 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 msgr2=0x55b5ae1c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:34.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.809+0000 7fe5a6800700 1 --2- 172.21.15.78:0/389747681 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.809+0000 7fe5a6800700 1 -- 172.21.15.78:0/389747681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a81aa870 con 0x55b5ae1c7ec0 2024-03-14T09:59:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.809+0000 7fe5a5400700 1 --2- 172.21.15.78:0/389747681 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.809+0000 7fe5a5e00700 1 --2- 172.21.15.78:0/389747681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 0x55b5ae1c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.809+0000 7fe5a6800700 1 --2- 172.21.15.78:0/389747681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1c53d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe5a806a350 tx=0x7fe59800bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=365db7f2ac39404f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.809+0000 7fe58be00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:59:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.809+0000 7fe58be00700 1 -- 172.21.15.78:0/389747681 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59800e040 con 0x55b5ae1c7ec0 2024-03-14T09:59:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.810+0000 7fe58be00700 1 -- 172.21.15.78:0/389747681 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe598007bd0 con 0x55b5ae1c7ec0 2024-03-14T09:59:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.810+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/389747681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b5ae1b0720 con 0x55b5ae1c7ec0 2024-03-14T09:59:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.810+0000 7fe58be00700 1 -- 172.21.15.78:0/389747681 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe598009ac0 con 0x55b5ae1c7ec0 2024-03-14T09:59:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.810+0000 7fe5c08ddb80 10 client.16644 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.810+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/389747681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b5adf3db60 con 0x55b5ae1c7ec0 2024-03-14T09:59:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.810+0000 7fe58be00700 1 -- 172.21.15.78:0/389747681 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b5ae1b0720 con 0x55b5ae1c7ec0 2024-03-14T09:59:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.810+0000 7fe58be00700 1 -- 172.21.15.78:0/389747681 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe59800b050 con 0x55b5ae1c7ec0 2024-03-14T09:59:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.811+0000 7fe5c08ddb80 10 client.16644 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.811+0000 7fe5c08ddb80 10 client.16644 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:59:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.811+0000 7fe5c08ddb80 20 client.16644 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.811+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/389747681 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b5ae1d7be0 0x55b5ae1da000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.811+0000 7fe5c08ddb80 4 client.16644 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:59:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.811+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/389747681 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b5ae108700 con 0x55b5ae1d7be0 2024-03-14T09:59:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.811+0000 7fe5a5e00700 1 --2- 172.21.15.78:0/389747681 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b5ae1d7be0 0x55b5ae1da000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.811+0000 7fe5a5e00700 1 --2- 172.21.15.78:0/389747681 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b5ae1d7be0 0x55b5ae1da000 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10da559918ea5c2a server_cookie=ae2fa33fba516eb7 in_seq=0 out_seq=0 2024-03-14T09:59:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.812+0000 7fe58be00700 10 client.16644 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.812+0000 7fe58be00700 1 -- 172.21.15.78:0/389747681 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55b5ae108700 con 0x55b5ae1d7be0 2024-03-14T09:59:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.812+0000 7fe58be00700 10 client.16644 handle_command_reply: tid=0 2024-03-14T09:59:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.812+0000 7fe5c08ddb80 1 client.16644 shutdown 2024-03-14T09:59:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.813+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/389747681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 msgr2=0x55b5ae1c53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.813+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/389747681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1c53d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe5a806a350 tx=0x7fe59800bc20 comp rx=0 tx=0).stop 2024-03-14T09:59:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.813+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/389747681 shutdown_connections 2024-03-14T09:59:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.813+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/389747681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1c53d0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.813+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/389747681 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 0x55b5ae1c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.813+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/389747681 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b5ae1d7be0 0x55b5ae1da000 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.813+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/389747681 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.813+0000 7fe58be00700 0 client.16644 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:34.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.813+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/389747681 >> 172.21.15.78:0/389747681 conn(0x55b5ae133220 msgr2=0x55b5ae1b8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:34.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.813+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/389747681 shutdown_connections 2024-03-14T09:59:34.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.813+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/389747681 wait complete. 2024-03-14T09:59:34.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.813+0000 7fe5c08ddb80 20 client.16644 trim_cache size 0 max 16384 2024-03-14T09:59:34.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.832+0000 7fe5c08ddb80 1 Processor -- start 2024-03-14T09:59:34.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.832+0000 7fe5c08ddb80 1 -- start start 2024-03-14T09:59:34.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.832+0000 7fe5c08ddb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bd680 0x55b5ae1b8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.832+0000 7fe5c08ddb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1c3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.832+0000 7fe5c08ddb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1c7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.833+0000 7fe5c08ddb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ae1bdcb0 con 0x55b5ae1bdfa0 2024-03-14T09:59:34.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.833+0000 7fe5c08ddb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ae1ba060 con 0x55b5ae1bd680 2024-03-14T09:59:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.833+0000 7fe5c08ddb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ae1b7800 con 0x55b5ae1c7ec0 2024-03-14T09:59:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.833+0000 7fe5a5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bd680 0x55b5ae1b8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.833+0000 7fe5a5400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1c3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.833+0000 7fe5a5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bd680 0x55b5ae1b8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:43824/0 (socket says 172.21.15.78:43824) 2024-03-14T09:59:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.833+0000 7fe5a5e00700 1 -- 172.21.15.78:0/490989754 learned_addr learned my addr 172.21.15.78:0/490989754 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.833+0000 7fe5a6800700 1 --2- 172.21.15.78:0/490989754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.833+0000 7fe5a5e00700 1 -- 172.21.15.78:0/490989754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1c7ec0 msgr2=0x55b5ae1c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.833+0000 7fe5a5e00700 1 --2- 172.21.15.78:0/490989754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1c7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.833+0000 7fe5a5e00700 1 -- 172.21.15.78:0/490989754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1bdfa0 msgr2=0x55b5ae1c3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.833+0000 7fe5a5e00700 1 --2- 172.21.15.78:0/490989754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1c3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.833+0000 7fe5a5e00700 1 -- 172.21.15.78:0/490989754 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5ae0d5ea0 con 0x55b5ae1bd680 2024-03-14T09:59:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.833+0000 7fe5a5e00700 1 --2- 172.21.15.78:0/490989754 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bd680 0x55b5ae1b8c30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x55b5ae1da8e0 tx=0x7fe590013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5463ad3b6676e2dc server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.834+0000 7fe597e00700 1 -- 172.21.15.78:0/490989754 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe590024040 con 0x55b5ae1bd680 2024-03-14T09:59:34.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.834+0000 7fe597e00700 1 -- 172.21.15.78:0/490989754 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe590015070 con 0x55b5ae1bd680 2024-03-14T09:59:34.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.834+0000 7fe597e00700 1 -- 172.21.15.78:0/490989754 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe590003680 con 0x55b5ae1bd680 2024-03-14T09:59:34.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.835+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/490989754 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bd680 msgr2=0x55b5ae1b8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:34.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.835+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/490989754 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bd680 0x55b5ae1b8c30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x55b5ae1da8e0 tx=0x7fe590013000 comp rx=0 tx=0).stop 2024-03-14T09:59:34.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.835+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/490989754 shutdown_connections 2024-03-14T09:59:34.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.835+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/490989754 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bd680 0x55b5ae1b8c30 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.835+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/490989754 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1c7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.835+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/490989754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1c3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.835+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/490989754 >> 172.21.15.78:0/490989754 conn(0x55b5ae133220 msgr2=0x55b5ae1c3560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:34.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.835+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/490989754 shutdown_connections 2024-03-14T09:59:34.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.835+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/490989754 wait complete. 2024-03-14T09:59:34.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.835+0000 7fe5c08ddb80 1 -- start start 2024-03-14T09:59:34.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.836+0000 7fe5c08ddb80 1 Processor -- start 2024-03-14T09:59:34.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.836+0000 7fe5c08ddb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 0x55b5ae1ce380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.836+0000 7fe5c08ddb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.836+0000 7fe5c08ddb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1cc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.836+0000 7fe5c08ddb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ae1cced0 con 0x55b5ae1c7ec0 2024-03-14T09:59:34.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.836+0000 7fe5c08ddb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ae1cd040 con 0x55b5ae1bdfa0 2024-03-14T09:59:34.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.836+0000 7fe5c08ddb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ae1cd200 con 0x55b5ae1bd680 2024-03-14T09:59:34.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.836+0000 7fe5c08ddb80 10 client.0 _ll_register_callbacks cb 0x7fe598003d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:34.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.836+0000 7fe5a5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.836+0000 7fe5a5e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 0x55b5ae1ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.836+0000 7fe5a5e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 0x55b5ae1ce380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60120/0 (socket says 172.21.15.78:60120) 2024-03-14T09:59:34.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.836+0000 7fe5a5e00700 1 -- 172.21.15.78:0/3767587115 learned_addr learned my addr 172.21.15.78:0/3767587115 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:34.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.836+0000 7fe5a5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1ce8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:43832/0 (socket says 172.21.15.78:43832) 2024-03-14T09:59:34.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.836+0000 7fe5a6800700 1 --2- 172.21.15.78:0/3767587115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.837+0000 7fe5a5e00700 1 -- 172.21.15.78:0/3767587115 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 msgr2=0x55b5ae1ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:34.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.837+0000 7fe5a5e00700 1 --2- 172.21.15.78:0/3767587115 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.837+0000 7fe5a5e00700 1 -- 172.21.15.78:0/3767587115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 msgr2=0x55b5ae1cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:34.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.837+0000 7fe5a5e00700 1 --2- 172.21.15.78:0/3767587115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1cc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.837+0000 7fe5a5e00700 1 -- 172.21.15.78:0/3767587115 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe590011440 con 0x55b5ae1bd680 2024-03-14T09:59:34.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.837+0000 7fe5a5400700 1 --2- 172.21.15.78:0/3767587115 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:34.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.837+0000 7fe5a5e00700 1 --2- 172.21.15.78:0/3767587115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 0x55b5ae1ce380 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe59002a550 tx=0x7fe590005760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=564a85d29406c575 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.837+0000 7fe58be00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:59:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.837+0000 7fe58be00700 1 -- 172.21.15.78:0/3767587115 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe590024040 con 0x55b5ae1bd680 2024-03-14T09:59:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.838+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/3767587115 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5a001fde0 con 0x55b5ae1bd680 2024-03-14T09:59:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.838+0000 7fe58be00700 1 -- 172.21.15.78:0/3767587115 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe590017070 con 0x55b5ae1bd680 2024-03-14T09:59:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.838+0000 7fe58be00700 1 -- 172.21.15.78:0/3767587115 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe590011790 con 0x55b5ae1bd680 2024-03-14T09:59:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.838+0000 7fe5c08ddb80 10 client.26704 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.838+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/3767587115 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b5adf95cd0 con 0x55b5ae1bd680 2024-03-14T09:59:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.838+0000 7fe58be00700 1 -- 172.21.15.78:0/3767587115 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5a001fde0 con 0x55b5ae1bd680 2024-03-14T09:59:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.838+0000 7fe58be00700 1 -- 172.21.15.78:0/3767587115 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe59001faa0 con 0x55b5ae1bd680 2024-03-14T09:59:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.838+0000 7fe5c08ddb80 10 client.26704 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.838+0000 7fe5c08ddb80 10 client.26704 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:59:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.838+0000 7fe5c08ddb80 20 client.26704 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.838+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/3767587115 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b5ae22a870 0x55b5ae22cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:34.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.838+0000 7fe5c08ddb80 4 client.26704 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:59:34.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.838+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/3767587115 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b5adf5ecf0 con 0x55b5ae22a870 2024-03-14T09:59:34.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.839+0000 7fe5a5400700 1 --2- 172.21.15.78:0/3767587115 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b5ae22a870 0x55b5ae22cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.839+0000 7fe5a5400700 1 --2- 172.21.15.78:0/3767587115 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b5ae22a870 0x55b5ae22cc90 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47f98679fdf29db1 server_cookie=70f0ebfa6e672201 in_seq=0 out_seq=0 2024-03-14T09:59:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.839+0000 7fe58be00700 10 client.26704 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.839+0000 7fe58be00700 1 -- 172.21.15.78:0/3767587115 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55b5adf5ecf0 con 0x55b5ae22a870 2024-03-14T09:59:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.839+0000 7fe58be00700 10 client.26704 handle_command_reply: tid=0 2024-03-14T09:59:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.839+0000 7fe5c08ddb80 1 client.26704 shutdown 2024-03-14T09:59:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.839+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/3767587115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 msgr2=0x55b5ae1ce380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.839+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/3767587115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 0x55b5ae1ce380 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe59002a550 tx=0x7fe590005760 comp rx=0 tx=0).stop 2024-03-14T09:59:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.839+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/3767587115 shutdown_connections 2024-03-14T09:59:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.839+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/3767587115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5ae1c7ec0 0x55b5ae1cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.839+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/3767587115 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b5ae1bd680 0x55b5ae1ce380 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.839+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/3767587115 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b5ae22a870 0x55b5ae22cc90 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.839+0000 7fe5c08ddb80 1 --2- 172.21.15.78:0/3767587115 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b5ae1bdfa0 0x55b5ae1ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.840+0000 7fe58be00700 0 client.26704 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:34.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.839+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/3767587115 >> 172.21.15.78:0/3767587115 conn(0x55b5ae133220 msgr2=0x55b5ae1bc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:34.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.840+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/3767587115 shutdown_connections 2024-03-14T09:59:34.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.840+0000 7fe5c08ddb80 1 -- 172.21.15.78:0/3767587115 wait complete. 2024-03-14T09:59:34.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.840+0000 7fe5c08ddb80 20 client.26704 trim_cache size 0 max 16384 2024-03-14T09:59:34.845 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:59:34.845 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T09:59:34.845 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T09:59:34.845 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:59:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.840+0000 7fe5ace00700 1 -- 172.21.15.78:0/746412332 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe58c0a0c40 msgr2=0x7fe58c0a3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.840+0000 7fe5ace00700 1 --2- 172.21.15.78:0/746412332 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe58c0a0c40 0x7fe58c0a3100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe590005950 tx=0x7fe59000b410 comp rx=0 tx=0).stop 2024-03-14T09:59:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.840+0000 7fe5ace00700 1 -- 172.21.15.78:0/746412332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5a8073a70 msgr2=0x7fe5a81a3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.841+0000 7fe5ace00700 1 --2- 172.21.15.78:0/746412332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5a8073a70 0x7fe5a81a3fa0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe5a000ec10 tx=0x7fe5a000ef20 comp rx=0 tx=0).stop 2024-03-14T09:59:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.841+0000 7fe5ace00700 1 -- 172.21.15.78:0/746412332 shutdown_connections 2024-03-14T09:59:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.841+0000 7fe5ace00700 1 --2- 172.21.15.78:0/746412332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5a8111cc0 0x7fe5a819db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.841+0000 7fe5ace00700 1 --2- 172.21.15.78:0/746412332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5a8073150 0x7fe5a81a3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.841+0000 7fe5ace00700 1 --2- 172.21.15.78:0/746412332 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe58c0a0c40 0x7fe58c0a3100 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.841+0000 7fe5ace00700 1 --2- 172.21.15.78:0/746412332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5a8073a70 0x7fe5a81a3fa0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:34.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.841+0000 7fe5ace00700 1 -- 172.21.15.78:0/746412332 >> 172.21.15.78:0/746412332 conn(0x7fe5a80fc7a0 msgr2=0x7fe5a80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:34.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.841+0000 7fe5ace00700 1 -- 172.21.15.78:0/746412332 shutdown_connections 2024-03-14T09:59:34.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:34.841+0000 7fe5ace00700 1 -- 172.21.15.78:0/746412332 wait complete. 2024-03-14T09:59:34.863 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T09:59:34.863 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T09:59:35.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.104+0000 7f7f45c00700 1 -- 172.21.15.78:0/2217586255 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f40105f40 msgr2=0x7f7f4010dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:35.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.104+0000 7f7f45c00700 1 --2- 172.21.15.78:0/2217586255 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f40105f40 0x7f7f4010dac0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7f3800b0a0 tx=0x7f7f3800b3b0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.105+0000 7f7f45c00700 1 -- 172.21.15.78:0/2217586255 shutdown_connections 2024-03-14T09:59:35.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.105+0000 7f7f45c00700 1 --2- 172.21.15.78:0/2217586255 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f40100e20 0x7f7f40101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.105+0000 7f7f45c00700 1 --2- 172.21.15.78:0/2217586255 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f40105f40 0x7f7f4010dac0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.105+0000 7f7f45c00700 1 --2- 172.21.15.78:0/2217586255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f401017d0 0x7f7f40105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.105+0000 7f7f45c00700 1 -- 172.21.15.78:0/2217586255 >> 172.21.15.78:0/2217586255 conn(0x7f7f400fc700 msgr2=0x7f7f400feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:35.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.105+0000 7f7f45c00700 1 -- 172.21.15.78:0/2217586255 shutdown_connections 2024-03-14T09:59:35.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.105+0000 7f7f45c00700 1 -- 172.21.15.78:0/2217586255 wait complete. 2024-03-14T09:59:35.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.106+0000 7f7f45c00700 1 Processor -- start 2024-03-14T09:59:35.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.106+0000 7f7f45c00700 1 -- start start 2024-03-14T09:59:35.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.107+0000 7f7f45c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f40100e20 0x7f7f4019f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.107+0000 7f7f45c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f40105f40 0x7f7f4019fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.107+0000 7f7f45c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f40199680 0x7f7f40199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.107+0000 7f7f45c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4019a100 con 0x7f7f40100e20 2024-03-14T09:59:35.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.107+0000 7f7f45c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4019a270 con 0x7f7f40105f40 2024-03-14T09:59:35.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.107+0000 7f7f45c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4019a430 con 0x7f7f40199680 2024-03-14T09:59:35.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.107+0000 7f7f3f400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f40199680 0x7f7f40199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.107+0000 7f7f3f400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f40199680 0x7f7f40199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60138/0 (socket says 172.21.15.78:60138) 2024-03-14T09:59:35.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.107+0000 7f7f3ea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f40100e20 0x7f7f4019f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.107+0000 7f7f3f400700 1 -- 172.21.15.78:0/940031223 learned_addr learned my addr 172.21.15.78:0/940031223 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:35.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.107+0000 7f7f3e000700 1 --2- 172.21.15.78:0/940031223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f40105f40 0x7f7f4019fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.108+0000 7f7f3f400700 1 -- 172.21.15.78:0/940031223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f40105f40 msgr2=0x7f7f4019fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:35.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.108+0000 7f7f3f400700 1 --2- 172.21.15.78:0/940031223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f40105f40 0x7f7f4019fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.108+0000 7f7f3f400700 1 -- 172.21.15.78:0/940031223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f40100e20 msgr2=0x7f7f4019f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:35.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.108+0000 7f7f3f400700 1 --2- 172.21.15.78:0/940031223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f40100e20 0x7f7f4019f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.108+0000 7f7f3f400700 1 -- 172.21.15.78:0/940031223 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f38009d00 con 0x7f7f40199680 2024-03-14T09:59:35.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.108+0000 7f7f3f400700 1 --2- 172.21.15.78:0/940031223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f40199680 0x7f7f40199b30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7f3800b070 tx=0x7f7f3800bda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:35.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.108+0000 7f7f37400700 1 -- 172.21.15.78:0/940031223 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3800e040 con 0x7f7f40199680 2024-03-14T09:59:35.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.108+0000 7f7f37400700 1 -- 172.21.15.78:0/940031223 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f3801d480 con 0x7f7f40199680 2024-03-14T09:59:35.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.108+0000 7f7f37400700 1 -- 172.21.15.78:0/940031223 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f380052a0 con 0x7f7f40199680 2024-03-14T09:59:35.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.108+0000 7f7f45c00700 1 -- 172.21.15.78:0/940031223 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f401a63c0 con 0x7f7f40199680 2024-03-14T09:59:35.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.109+0000 7f7f45c00700 1 -- 172.21.15.78:0/940031223 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f401a6790 con 0x7f7f40199680 2024-03-14T09:59:35.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.109+0000 7f7f45c00700 1 -- 172.21.15.78:0/940031223 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f401022f0 con 0x7f7f40199680 2024-03-14T09:59:35.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.112+0000 7f7f37400700 1 -- 172.21.15.78:0/940031223 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f7f3801d5f0 con 0x7f7f40199680 2024-03-14T09:59:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.113+0000 7f7f37400700 1 --2- 172.21.15.78:0/940031223 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7f240a0c80 0x7f7f240a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.113+0000 7f7f37400700 1 -- 172.21.15.78:0/940031223 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f7f380cd780 con 0x7f7f40199680 2024-03-14T09:59:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.113+0000 7f7f3ea00700 1 --2- 172.21.15.78:0/940031223 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7f240a0c80 0x7f7f240a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.114+0000 7f7f3ea00700 1 --2- 172.21.15.78:0/940031223 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7f240a0c80 0x7f7f240a3140 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009c60 tx=0x7f7f30009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:35.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.117+0000 7f7f37400700 1 -- 172.21.15.78:0/940031223 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7f38096b70 con 0x7f7f40199680 2024-03-14T09:59:35.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:35 smithi078 ceph-mon[38143]: pgmap v165: 129 pgs: 129 active+clean; 217 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 1.1 MiB/s wr, 373 op/s 2024-03-14T09:59:35.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.445+0000 7f7f45c00700 1 -- 172.21.15.78:0/940031223 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f400679e0 con 0x7f7f40199680 2024-03-14T09:59:35.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.446+0000 7f7f37400700 1 -- 172.21.15.78:0/940031223 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f7f3809a190 con 0x7f7f40199680 2024-03-14T09:59:35.447 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T09:59:35.447 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T09:59:35.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.449+0000 7f7f45c00700 1 -- 172.21.15.78:0/940031223 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7f240a0c80 msgr2=0x7f7f240a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:35.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.449+0000 7f7f45c00700 1 --2- 172.21.15.78:0/940031223 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7f240a0c80 0x7f7f240a3140 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009c60 tx=0x7f7f30009260 comp rx=0 tx=0).stop 2024-03-14T09:59:35.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.449+0000 7f7f45c00700 1 -- 172.21.15.78:0/940031223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f40199680 msgr2=0x7f7f40199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:35.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.449+0000 7f7f45c00700 1 --2- 172.21.15.78:0/940031223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f40199680 0x7f7f40199b30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7f3800b070 tx=0x7f7f3800bda0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.449+0000 7f7f45c00700 1 -- 172.21.15.78:0/940031223 shutdown_connections 2024-03-14T09:59:35.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.449+0000 7f7f45c00700 1 --2- 172.21.15.78:0/940031223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7f40199680 0x7f7f40199b30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.449+0000 7f7f45c00700 1 --2- 172.21.15.78:0/940031223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f40100e20 0x7f7f4019f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.449+0000 7f7f45c00700 1 --2- 172.21.15.78:0/940031223 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7f240a0c80 0x7f7f240a3140 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.449+0000 7f7f45c00700 1 --2- 172.21.15.78:0/940031223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f40105f40 0x7f7f4019fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.449+0000 7f7f45c00700 1 -- 172.21.15.78:0/940031223 >> 172.21.15.78:0/940031223 conn(0x7f7f400fc700 msgr2=0x7f7f400fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:35.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.449+0000 7f7f45c00700 1 -- 172.21.15.78:0/940031223 shutdown_connections 2024-03-14T09:59:35.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.449+0000 7f7f45c00700 1 -- 172.21.15.78:0/940031223 wait complete. 2024-03-14T09:59:35.451 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T09:59:35.471 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T09:59:35.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:35 smithi152 ceph-mon[48279]: pgmap v165: 129 pgs: 129 active+clean; 217 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 1.1 MiB/s wr, 373 op/s 2024-03-14T09:59:35.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:35 smithi132 ceph-mon[47959]: pgmap v165: 129 pgs: 129 active+clean; 217 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 1.1 MiB/s wr, 373 op/s 2024-03-14T09:59:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.725+0000 7fe6ab800700 1 -- 172.21.15.78:0/1575733559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6a41018d0 msgr2=0x7fe6a4105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:35.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.725+0000 7fe6ab800700 1 --2- 172.21.15.78:0/1575733559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6a41018d0 0x7fe6a4105910 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe690009990 tx=0x7fe690009ca0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.725+0000 7fe6ab800700 1 -- 172.21.15.78:0/1575733559 shutdown_connections 2024-03-14T09:59:35.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.725+0000 7fe6ab800700 1 --2- 172.21.15.78:0/1575733559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6a4106040 0x7fe6a410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.725+0000 7fe6ab800700 1 --2- 172.21.15.78:0/1575733559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a4100f20 0x7fe6a4101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.725+0000 7fe6ab800700 1 --2- 172.21.15.78:0/1575733559 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6a41018d0 0x7fe6a4105910 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.725+0000 7fe6ab800700 1 -- 172.21.15.78:0/1575733559 >> 172.21.15.78:0/1575733559 conn(0x7fe6a40fc7a0 msgr2=0x7fe6a40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:35.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.725+0000 7fe6ab800700 1 -- 172.21.15.78:0/1575733559 shutdown_connections 2024-03-14T09:59:35.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.725+0000 7fe6ab800700 1 -- 172.21.15.78:0/1575733559 wait complete. 2024-03-14T09:59:35.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.727+0000 7fe6ab800700 1 Processor -- start 2024-03-14T09:59:35.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.727+0000 7fe6ab800700 1 -- start start 2024-03-14T09:59:35.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.727+0000 7fe6ab800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6a4100f20 0x7fe6a4197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.727+0000 7fe6ab800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6a4106040 0x7fe6a4197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.727+0000 7fe6ab800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a419bf90 0x7fe6a419c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.727+0000 7fe6ab800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a419ca10 con 0x7fe6a419bf90 2024-03-14T09:59:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.727+0000 7fe6ab800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a419cb80 con 0x7fe6a4106040 2024-03-14T09:59:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.727+0000 7fe6ab800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a419cd40 con 0x7fe6a4100f20 2024-03-14T09:59:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.728+0000 7fe6a8e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6a4100f20 0x7fe6a4197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.728+0000 7fe6a3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6a4106040 0x7fe6a4197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.728+0000 7fe6a8e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6a4100f20 0x7fe6a4197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60154/0 (socket says 172.21.15.78:60154) 2024-03-14T09:59:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.728+0000 7fe6a8e00700 1 -- 172.21.15.78:0/2577197865 learned_addr learned my addr 172.21.15.78:0/2577197865 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.728+0000 7fe6a9800700 1 --2- 172.21.15.78:0/2577197865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a419bf90 0x7fe6a419c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.728+0000 7fe6a8e00700 1 -- 172.21.15.78:0/2577197865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6a4106040 msgr2=0x7fe6a4197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.728+0000 7fe6a8e00700 1 --2- 172.21.15.78:0/2577197865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6a4106040 0x7fe6a4197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.728+0000 7fe6a8e00700 1 -- 172.21.15.78:0/2577197865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a419bf90 msgr2=0x7fe6a419c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.728+0000 7fe6a8e00700 1 --2- 172.21.15.78:0/2577197865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a419bf90 0x7fe6a419c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.728+0000 7fe6a8e00700 1 -- 172.21.15.78:0/2577197865 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6900095f0 con 0x7fe6a4100f20 2024-03-14T09:59:35.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.729+0000 7fe6a8e00700 1 --2- 172.21.15.78:0/2577197865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6a4100f20 0x7fe6a4197700 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe69800e9a0 tx=0x7fe69800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.729+0000 7fe6a1600700 1 -- 172.21.15.78:0/2577197865 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe698009790 con 0x7fe6a4100f20 2024-03-14T09:59:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.729+0000 7fe6a1600700 1 -- 172.21.15.78:0/2577197865 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe69800cbd0 con 0x7fe6a4100f20 2024-03-14T09:59:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.729+0000 7fe6a1600700 1 -- 172.21.15.78:0/2577197865 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe698018680 con 0x7fe6a4100f20 2024-03-14T09:59:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.729+0000 7fe6ab800700 1 -- 172.21.15.78:0/2577197865 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a4073040 con 0x7fe6a4100f20 2024-03-14T09:59:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.729+0000 7fe6ab800700 1 -- 172.21.15.78:0/2577197865 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a4073500 con 0x7fe6a4100f20 2024-03-14T09:59:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.734+0000 7fe6a1600700 1 -- 172.21.15.78:0/2577197865 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fe69800cd40 con 0x7fe6a4100f20 2024-03-14T09:59:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.735+0000 7fe6bf2b8b80 1 Processor -- start 2024-03-14T09:59:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.735+0000 7fe6bf2b8b80 1 -- start start 2024-03-14T09:59:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.736+0000 7fe6bf2b8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6680 0x5614744d6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.736+0000 7fe6bf2b8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6fa0 0x5614744e0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.736+0000 7fe6bf2b8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744e0ec0 0x5614744e32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.736+0000 7fe6bf2b8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56147442f990 con 0x5614744d6fa0 2024-03-14T09:59:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.736+0000 7fe6bf2b8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56147442d350 con 0x5614744e0ec0 2024-03-14T09:59:35.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.736+0000 7fe6bf2b8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561474428c40 con 0x5614744d6680 2024-03-14T09:59:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.736+0000 7fe6a9800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6680 0x5614744d6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.736+0000 7fe6a9800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6680 0x5614744d6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60156/0 (socket says 172.21.15.78:60156) 2024-03-14T09:59:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.736+0000 7fe6a9800700 1 -- 172.21.15.78:0/3139999636 learned_addr learned my addr 172.21.15.78:0/3139999636 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.736+0000 7fe6a9800700 1 -- 172.21.15.78:0/3139999636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744e0ec0 msgr2=0x5614744e32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.736+0000 7fe6a9800700 1 --2- 172.21.15.78:0/3139999636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744e0ec0 0x5614744e32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.736+0000 7fe6a9800700 1 -- 172.21.15.78:0/3139999636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6fa0 msgr2=0x5614744e0980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-14T09:59:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.737+0000 7fe6a8e00700 1 --2- 172.21.15.78:0/3139999636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6fa0 0x5614744e0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.737+0000 7fe6a9800700 1 --2- 172.21.15.78:0/3139999636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6fa0 0x5614744e0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.737+0000 7fe6a9800700 1 -- 172.21.15.78:0/3139999636 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5614742aa3b0 con 0x5614744d6680 2024-03-14T09:59:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.737+0000 7fe6a8e00700 1 --2- 172.21.15.78:0/3139999636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6fa0 0x5614744e0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:59:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.737+0000 7fe6a1600700 1 --2- 172.21.15.78:0/2577197865 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe68c0a0d00 0x7fe68c0a31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.737+0000 7fe6a1600700 1 -- 172.21.15.78:0/2577197865 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe698014070 con 0x7fe6a4100f20 2024-03-14T09:59:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.737+0000 7fe6a9800700 1 --2- 172.21.15.78:0/3139999636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6680 0x5614744d6a60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009440 tx=0x7fe69c009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ec44e2a38bf6d88 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:35.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.737+0000 7fe6a8e00700 1 --2- 172.21.15.78:0/2577197865 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe68c0a0d00 0x7fe68c0a31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.737+0000 7fe696000700 1 -- 172.21.15.78:0/3139999636 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c00e070 con 0x5614744d6680 2024-03-14T09:59:35.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.738+0000 7fe696000700 1 -- 172.21.15.78:0/3139999636 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe69c00f040 con 0x5614744d6680 2024-03-14T09:59:35.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.738+0000 7fe696000700 1 -- 172.21.15.78:0/3139999636 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c015b60 con 0x5614744d6680 2024-03-14T09:59:35.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.738+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/3139999636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6680 msgr2=0x5614744d6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:35.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.738+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/3139999636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6680 0x5614744d6a60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009440 tx=0x7fe69c009920 comp rx=0 tx=0).stop 2024-03-14T09:59:35.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.738+0000 7fe6a8e00700 1 --2- 172.21.15.78:0/2577197865 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe68c0a0d00 0x7fe68c0a31c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe698095e80 tx=0x7fe69801f000 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:35.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.738+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/3139999636 shutdown_connections 2024-03-14T09:59:35.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.738+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/3139999636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744e0ec0 0x5614744e32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.738+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/3139999636 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6680 0x5614744d6a60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.738+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/3139999636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6fa0 0x5614744e0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.738+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/3139999636 >> 172.21.15.78:0/3139999636 conn(0x56147444c220 msgr2=0x5614744d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:35.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.739+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/3139999636 shutdown_connections 2024-03-14T09:59:35.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.739+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/3139999636 wait complete. 2024-03-14T09:59:35.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.739+0000 7fe6bf2b8b80 1 -- start start 2024-03-14T09:59:35.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.739+0000 7fe6bf2b8b80 1 Processor -- start 2024-03-14T09:59:35.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.740+0000 7fe6bf2b8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6680 0x5614744e0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.740+0000 7fe6bf2b8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744ddc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.740+0000 7fe6bf2b8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744e0ec0 0x5614744de380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.740+0000 7fe6bf2b8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614743c4d80 con 0x5614744d6680 2024-03-14T09:59:35.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.740+0000 7fe6bf2b8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614743c4a20 con 0x5614744e0ec0 2024-03-14T09:59:35.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.740+0000 7fe6bf2b8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614742ac6a0 con 0x5614744d6fa0 2024-03-14T09:59:35.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.740+0000 7fe6bf2b8b80 10 client.0 _ll_register_callbacks cb 0x5614742303a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:35.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.740+0000 7fe6a9800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6680 0x5614744e0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.740+0000 7fe6a3e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744ddc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.740+0000 7fe6a9800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6680 0x5614744e0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37358/0 (socket says 172.21.15.78:37358) 2024-03-14T09:59:35.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.740+0000 7fe6a9800700 1 -- 172.21.15.78:0/2305522441 learned_addr learned my addr 172.21.15.78:0/2305522441 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:35.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.740+0000 7fe6a3e00700 1 --2- 172.21.15.78:0/2305522441 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744ddc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60162/0 (socket says 172.21.15.78:60162) 2024-03-14T09:59:35.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.740+0000 7fe6a9800700 1 --2- 172.21.15.78:0/2305522441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744e0ec0 0x5614744de380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.741+0000 7fe6a3e00700 1 -- 172.21.15.78:0/2305522441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744e0ec0 msgr2=0x5614744de380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:35.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.741+0000 7fe6a3e00700 1 --2- 172.21.15.78:0/2305522441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744e0ec0 0x5614744de380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.741+0000 7fe6a3e00700 1 -- 172.21.15.78:0/2305522441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6680 msgr2=0x5614744e0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:35.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.741+0000 7fe6a3e00700 1 --2- 172.21.15.78:0/2305522441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6680 0x5614744e0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.741+0000 7fe6a3e00700 1 -- 172.21.15.78:0/2305522441 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe69c0090f0 con 0x5614744d6fa0 2024-03-14T09:59:35.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.741+0000 7fe6a3e00700 1 --2- 172.21.15.78:0/2305522441 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744ddc40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe6900098e0 tx=0x7fe69001cf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e3839ff01eea15c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:35.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.741+0000 7fe68aa00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:59:35.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.741+0000 7fe68aa00700 1 -- 172.21.15.78:0/2305522441 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe690021050 con 0x5614744d6fa0 2024-03-14T09:59:35.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.741+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/2305522441 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5614744c9720 con 0x5614744d6fa0 2024-03-14T09:59:35.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.741+0000 7fe68aa00700 1 -- 172.21.15.78:0/2305522441 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe69000b6f0 con 0x5614744d6fa0 2024-03-14T09:59:35.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.741+0000 7fe68aa00700 1 -- 172.21.15.78:0/2305522441 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe69000b8b0 con 0x5614744d6fa0 2024-03-14T09:59:35.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.741+0000 7fe6bf2b8b80 10 client.26722 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:35.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.741+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/2305522441 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561474256b60 con 0x5614744d6fa0 2024-03-14T09:59:35.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.742+0000 7fe68aa00700 1 -- 172.21.15.78:0/2305522441 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5614744c9720 con 0x5614744d6fa0 2024-03-14T09:59:35.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.742+0000 7fe68aa00700 1 -- 172.21.15.78:0/2305522441 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe6900048a0 con 0x5614744d6fa0 2024-03-14T09:59:35.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.742+0000 7fe6bf2b8b80 10 client.26722 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:35.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.742+0000 7fe6bf2b8b80 10 client.26722 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:59:35.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.742+0000 7fe6bf2b8b80 20 client.26722 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:35.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.742+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/2305522441 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5614744f0980 0x5614744f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.742+0000 7fe6bf2b8b80 4 client.26722 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:59:35.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.742+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/2305522441 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561474421700 con 0x5614744f0980 2024-03-14T09:59:35.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.742+0000 7fe6a9800700 1 --2- 172.21.15.78:0/2305522441 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5614744f0980 0x5614744f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.744+0000 7fe68aa00700 10 client.26722 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:59:35.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.744+0000 7fe6a9800700 1 --2- 172.21.15.78:0/2305522441 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5614744f0980 0x5614744f2da0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=54f597ae3b9d81b0 server_cookie=9bda6277cd864a19 in_seq=0 out_seq=0 2024-03-14T09:59:35.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.744+0000 7fe68aa00700 1 -- 172.21.15.78:0/2305522441 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x561474421700 con 0x5614744f0980 2024-03-14T09:59:35.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.744+0000 7fe68aa00700 10 client.26722 handle_command_reply: tid=0 2024-03-14T09:59:35.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.744+0000 7fe6bf2b8b80 1 client.26722 shutdown 2024-03-14T09:59:35.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.745+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/2305522441 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 msgr2=0x5614744ddc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:35.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.745+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/2305522441 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744ddc40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe6900098e0 tx=0x7fe69001cf40 comp rx=0 tx=0).stop 2024-03-14T09:59:35.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.745+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/2305522441 shutdown_connections 2024-03-14T09:59:35.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.745+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/2305522441 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5614744f0980 0x5614744f2da0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.745+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/2305522441 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744e0ec0 0x5614744de380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.745+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/2305522441 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6680 0x5614744e0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.745+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/2305522441 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744ddc40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.745+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/2305522441 >> 172.21.15.78:0/2305522441 conn(0x56147444c220 msgr2=0x5614744d1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:35.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.745+0000 7fe68aa00700 0 client.26722 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:59:35.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.745+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/2305522441 shutdown_connections 2024-03-14T09:59:35.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.745+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/2305522441 wait complete. 2024-03-14T09:59:35.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.745+0000 7fe6bf2b8b80 20 client.26722 trim_cache size 0 max 16384 2024-03-14T09:59:35.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.762+0000 7fe6bf2b8b80 1 Processor -- start 2024-03-14T09:59:35.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.762+0000 7fe6bf2b8b80 1 -- start start 2024-03-14T09:59:35.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.762+0000 7fe6bf2b8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6680 0x5614744e7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.762+0000 7fe6bf2b8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744ec120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.762+0000 7fe6bf2b8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744e0ec0 0x5614744ec660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.762+0000 7fe6bf2b8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614744d80c0 con 0x5614744d6680 2024-03-14T09:59:35.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.762+0000 7fe6bf2b8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614744f7ac0 con 0x5614744e0ec0 2024-03-14T09:59:35.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.762+0000 7fe6bf2b8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614744d7890 con 0x5614744d6fa0 2024-03-14T09:59:35.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.763+0000 7fe6a9800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6680 0x5614744e7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.763+0000 7fe6a3e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744ec120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.763+0000 7fe6a3e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744ec120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60170/0 (socket says 172.21.15.78:60170) 2024-03-14T09:59:35.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.763+0000 7fe6a9800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6680 0x5614744e7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37370/0 (socket says 172.21.15.78:37370) 2024-03-14T09:59:35.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.763+0000 7fe6a3e00700 1 -- 172.21.15.78:0/763770885 learned_addr learned my addr 172.21.15.78:0/763770885 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:35.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.763+0000 7fe6a9800700 1 --2- 172.21.15.78:0/763770885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744e0ec0 0x5614744ec660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.763+0000 7fe6a3e00700 1 -- 172.21.15.78:0/763770885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744e0ec0 msgr2=0x5614744ec660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:35.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.763+0000 7fe6a3e00700 1 --2- 172.21.15.78:0/763770885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744e0ec0 0x5614744ec660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.763+0000 7fe6a3e00700 1 -- 172.21.15.78:0/763770885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6680 msgr2=0x5614744e7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:35.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.763+0000 7fe6a3e00700 1 --2- 172.21.15.78:0/763770885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6680 0x5614744e7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.763+0000 7fe6a3e00700 1 -- 172.21.15.78:0/763770885 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5614743eeea0 con 0x5614744d6fa0 2024-03-14T09:59:35.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.764+0000 7fe6a3e00700 1 --2- 172.21.15.78:0/763770885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744ec120 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe6900095c0 tx=0x7fe69000b630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51ebcd92d85d001f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:35.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.764+0000 7fe696a00700 1 -- 172.21.15.78:0/763770885 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69003f070 con 0x5614744d6fa0 2024-03-14T09:59:35.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.764+0000 7fe696a00700 1 -- 172.21.15.78:0/763770885 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe690022040 con 0x5614744d6fa0 2024-03-14T09:59:35.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.764+0000 7fe696a00700 1 -- 172.21.15.78:0/763770885 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe690021040 con 0x5614744d6fa0 2024-03-14T09:59:35.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.765+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/763770885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 msgr2=0x5614744ec120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:35.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.765+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/763770885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744ec120 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe6900095c0 tx=0x7fe69000b630 comp rx=0 tx=0).stop 2024-03-14T09:59:35.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.765+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/763770885 shutdown_connections 2024-03-14T09:59:35.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.765+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/763770885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744e0ec0 0x5614744ec660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.765+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/763770885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744d6680 0x5614744e7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.765+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/763770885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744ec120 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.765+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/763770885 >> 172.21.15.78:0/763770885 conn(0x56147444c220 msgr2=0x5614744d8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.765+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/763770885 shutdown_connections 2024-03-14T09:59:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.765+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/763770885 wait complete. 2024-03-14T09:59:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.765+0000 7fe6bf2b8b80 1 -- start start 2024-03-14T09:59:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.765+0000 7fe6bf2b8b80 1 Processor -- start 2024-03-14T09:59:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.766+0000 7fe6bf2b8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744d6680 0x5614744e5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.766+0000 7fe6bf2b8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744e6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.766+0000 7fe6bf2b8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744e0ec0 0x5614744e65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.766+0000 7fe6bf2b8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614744e7f00 con 0x5614744e0ec0 2024-03-14T09:59:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.766+0000 7fe6bf2b8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614745414a0 con 0x5614744d6680 2024-03-14T09:59:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.766+0000 7fe6bf2b8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614745415e0 con 0x5614744d6fa0 2024-03-14T09:59:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.766+0000 7fe6bf2b8b80 10 client.0 _ll_register_callbacks cb 0x7fe690009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.766+0000 7fe6a3e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744e6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.766+0000 7fe6a3e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744e6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60174/0 (socket says 172.21.15.78:60174) 2024-03-14T09:59:35.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.766+0000 7fe6a3e00700 1 -- 172.21.15.78:0/3624795502 learned_addr learned my addr 172.21.15.78:0/3624795502 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:35.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.766+0000 7fe6a9800700 1 --2- 172.21.15.78:0/3624795502 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744d6680 0x5614744e5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.766+0000 7fe6a9800700 1 --2- 172.21.15.78:0/3624795502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744e0ec0 0x5614744e65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.767+0000 7fe6a3e00700 1 -- 172.21.15.78:0/3624795502 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744d6680 msgr2=0x5614744e5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:35.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.767+0000 7fe6a3e00700 1 --2- 172.21.15.78:0/3624795502 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744d6680 0x5614744e5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.767+0000 7fe6a3e00700 1 -- 172.21.15.78:0/3624795502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744e0ec0 msgr2=0x5614744e65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:35.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.767+0000 7fe6a3e00700 1 --2- 172.21.15.78:0/3624795502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744e0ec0 0x5614744e65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.767+0000 7fe6a3e00700 1 -- 172.21.15.78:0/3624795502 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561474256b60 con 0x5614744d6fa0 2024-03-14T09:59:35.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.767+0000 7fe6a3e00700 1 --2- 172.21.15.78:0/3624795502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744e6070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe69003d040 tx=0x7fe690005950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cb776726ea5962d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:35.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.767+0000 7fe68aa00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:59:35.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.767+0000 7fe68aa00700 1 -- 172.21.15.78:0/3624795502 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe690021040 con 0x5614744d6fa0 2024-03-14T09:59:35.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.767+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/3624795502 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe69c00f040 con 0x5614744d6fa0 2024-03-14T09:59:35.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.767+0000 7fe68aa00700 1 -- 172.21.15.78:0/3624795502 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fe690000e40 con 0x5614744d6fa0 2024-03-14T09:59:35.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.768+0000 7fe68aa00700 1 -- 172.21.15.78:0/3624795502 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe69003f040 con 0x5614744d6fa0 2024-03-14T09:59:35.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.768+0000 7fe6bf2b8b80 10 client.26728 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:35.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.768+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/3624795502 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5614742aecd0 con 0x5614744d6fa0 2024-03-14T09:59:35.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.768+0000 7fe68aa00700 1 -- 172.21.15.78:0/3624795502 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe69c00f040 con 0x5614744d6fa0 2024-03-14T09:59:35.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.768+0000 7fe68aa00700 1 -- 172.21.15.78:0/3624795502 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe69000f460 con 0x5614744d6fa0 2024-03-14T09:59:35.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.768+0000 7fe6bf2b8b80 10 client.26728 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:35.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.768+0000 7fe6bf2b8b80 10 client.26728 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T09:59:35.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.768+0000 7fe6bf2b8b80 20 client.26728 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:35.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.768+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/3624795502 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x561474542970 0x561474544d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:35.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.768+0000 7fe6bf2b8b80 4 client.26728 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:59:35.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.768+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/3624795502 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561474277cf0 con 0x561474542970 2024-03-14T09:59:35.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.768+0000 7fe6a9800700 1 --2- 172.21.15.78:0/3624795502 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x561474542970 0x561474544d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:35.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.769+0000 7fe68aa00700 10 client.26728 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T09:59:35.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.769+0000 7fe6a9800700 1 --2- 172.21.15.78:0/3624795502 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x561474542970 0x561474544d60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9460063486f10c6a server_cookie=425463395b2c241c in_seq=0 out_seq=0 2024-03-14T09:59:35.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.769+0000 7fe68aa00700 1 -- 172.21.15.78:0/3624795502 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561474277cf0 con 0x561474542970 2024-03-14T09:59:35.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.769+0000 7fe68aa00700 10 client.26728 handle_command_reply: tid=0 2024-03-14T09:59:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.769+0000 7fe6bf2b8b80 1 client.26728 shutdown 2024-03-14T09:59:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.769+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/3624795502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 msgr2=0x5614744e6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.769+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/3624795502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744e6070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe69003d040 tx=0x7fe690005950 comp rx=0 tx=0).stop 2024-03-14T09:59:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.769+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/3624795502 shutdown_connections 2024-03-14T09:59:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.769+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/3624795502 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x561474542970 0x561474544d60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/3624795502 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5614744d6680 0x5614744e5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe68aa00700 0 client.26728 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T09:59:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/3624795502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614744e0ec0 0x5614744e65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe6bf2b8b80 1 --2- 172.21.15.78:0/3624795502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5614744d6fa0 0x5614744e6070 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/3624795502 >> 172.21.15.78:0/3624795502 conn(0x56147444c220 msgr2=0x5614744ec120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/3624795502 shutdown_connections 2024-03-14T09:59:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe6bf2b8b80 1 -- 172.21.15.78:0/3624795502 wait complete. 2024-03-14T09:59:35.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe6bf2b8b80 20 client.26728 trim_cache size 0 max 16384 2024-03-14T09:59:35.811 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:59:35.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe6ab800700 1 -- 172.21.15.78:0/2577197865 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe68c0a0d00 msgr2=0x7fe68c0a31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:35.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe6ab800700 1 --2- 172.21.15.78:0/2577197865 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe68c0a0d00 0x7fe68c0a31c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe698095e80 tx=0x7fe69801f000 comp rx=0 tx=0).stop 2024-03-14T09:59:35.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe6ab800700 1 -- 172.21.15.78:0/2577197865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6a4100f20 msgr2=0x7fe6a4197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:35.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe6ab800700 1 --2- 172.21.15.78:0/2577197865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6a4100f20 0x7fe6a4197700 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe69800e9a0 tx=0x7fe69800ecb0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe6ab800700 1 -- 172.21.15.78:0/2577197865 shutdown_connections 2024-03-14T09:59:35.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe6ab800700 1 --2- 172.21.15.78:0/2577197865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6a419bf90 0x7fe6a419c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe6ab800700 1 --2- 172.21.15.78:0/2577197865 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe6a4100f20 0x7fe6a4197700 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe6ab800700 1 --2- 172.21.15.78:0/2577197865 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe68c0a0d00 0x7fe68c0a31c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.770+0000 7fe6ab800700 1 --2- 172.21.15.78:0/2577197865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe6a4106040 0x7fe6a4197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:35.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.771+0000 7fe6ab800700 1 -- 172.21.15.78:0/2577197865 >> 172.21.15.78:0/2577197865 conn(0x7fe6a40fc7a0 msgr2=0x7fe6a40fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:35.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.771+0000 7fe6ab800700 1 -- 172.21.15.78:0/2577197865 shutdown_connections 2024-03-14T09:59:35.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:35.771+0000 7fe6ab800700 1 -- 172.21.15.78:0/2577197865 wait complete. 2024-03-14T09:59:35.813 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T09:59:36.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.069+0000 7fd44de00700 1 -- 172.21.15.78:0/616386451 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4480680b0 msgr2=0x7fd448068490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.069+0000 7fd44de00700 1 --2- 172.21.15.78:0/616386451 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4480680b0 0x7fd448068490 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd438009990 tx=0x7fd438009ca0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.069+0000 7fd44de00700 1 -- 172.21.15.78:0/616386451 shutdown_connections 2024-03-14T09:59:36.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.069+0000 7fd44de00700 1 --2- 172.21.15.78:0/616386451 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4480680b0 0x7fd448068490 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.069+0000 7fd44de00700 1 --2- 172.21.15.78:0/616386451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd448106060 0x7fd44810dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.069+0000 7fd44de00700 1 --2- 172.21.15.78:0/616386451 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd448068a60 0x7fd448105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.069+0000 7fd44de00700 1 -- 172.21.15.78:0/616386451 >> 172.21.15.78:0/616386451 conn(0x7fd448076d80 msgr2=0x7fd448077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:36.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.069+0000 7fd44de00700 1 -- 172.21.15.78:0/616386451 shutdown_connections 2024-03-14T09:59:36.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.069+0000 7fd44de00700 1 -- 172.21.15.78:0/616386451 wait complete. 2024-03-14T09:59:36.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.070+0000 7fd44de00700 1 Processor -- start 2024-03-14T09:59:36.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.070+0000 7fd44de00700 1 -- start start 2024-03-14T09:59:36.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.071+0000 7fd44de00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4480680b0 0x7fd4480ffc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.071+0000 7fd44de00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd448106060 0x7fd448100160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.071+0000 7fd44de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4481006a0 0x7fd4481a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.071+0000 7fd44de00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4481a5cf0 con 0x7fd4481006a0 2024-03-14T09:59:36.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.071+0000 7fd44de00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4481a5e60 con 0x7fd448106060 2024-03-14T09:59:36.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.071+0000 7fd44de00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4481a6020 con 0x7fd4480680b0 2024-03-14T09:59:36.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.071+0000 7fd446a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4480680b0 0x7fd4480ffc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.071+0000 7fd446a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4480680b0 0x7fd4480ffc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60194/0 (socket says 172.21.15.78:60194) 2024-03-14T09:59:36.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.071+0000 7fd446a00700 1 -- 172.21.15.78:0/4042973563 learned_addr learned my addr 172.21.15.78:0/4042973563 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:36.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.071+0000 7fd447400700 1 --2- 172.21.15.78:0/4042973563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4481006a0 0x7fd4481a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.071+0000 7fd446000700 1 --2- 172.21.15.78:0/4042973563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd448106060 0x7fd448100160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.071+0000 7fd446a00700 1 -- 172.21.15.78:0/4042973563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd448106060 msgr2=0x7fd448100160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.071+0000 7fd446a00700 1 --2- 172.21.15.78:0/4042973563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd448106060 0x7fd448100160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.071+0000 7fd446a00700 1 -- 172.21.15.78:0/4042973563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4481006a0 msgr2=0x7fd4481a5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.071+0000 7fd446a00700 1 --2- 172.21.15.78:0/4042973563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4481006a0 0x7fd4481a5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.071+0000 7fd446a00700 1 -- 172.21.15.78:0/4042973563 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4380095f0 con 0x7fd4480680b0 2024-03-14T09:59:36.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.072+0000 7fd446a00700 1 --2- 172.21.15.78:0/4042973563 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4480680b0 0x7fd4480ffc20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd438000c00 tx=0x7fd438005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:36.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.072+0000 7fd44ca00700 1 -- 172.21.15.78:0/4042973563 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43801d070 con 0x7fd4480680b0 2024-03-14T09:59:36.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.072+0000 7fd44ca00700 1 -- 172.21.15.78:0/4042973563 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd438022470 con 0x7fd4480680b0 2024-03-14T09:59:36.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.072+0000 7fd44de00700 1 -- 172.21.15.78:0/4042973563 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4481a62a0 con 0x7fd4480680b0 2024-03-14T09:59:36.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.072+0000 7fd44de00700 1 -- 172.21.15.78:0/4042973563 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4481a66c0 con 0x7fd4480680b0 2024-03-14T09:59:36.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.072+0000 7fd44ca00700 1 -- 172.21.15.78:0/4042973563 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd438017650 con 0x7fd4480680b0 2024-03-14T09:59:36.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.075+0000 7fd44ca00700 1 -- 172.21.15.78:0/4042973563 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fd4380177b0 con 0x7fd4480680b0 2024-03-14T09:59:36.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.076+0000 7fd44ca00700 1 --2- 172.21.15.78:0/4042973563 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd42c0a0c00 0x7fd42c0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.076+0000 7fd44ca00700 1 -- 172.21.15.78:0/4042973563 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fd4380cd9e0 con 0x7fd4480680b0 2024-03-14T09:59:36.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.077+0000 7fd446000700 1 --2- 172.21.15.78:0/4042973563 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd42c0a0c00 0x7fd42c0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.077+0000 7fd4619a1b80 1 Processor -- start 2024-03-14T09:59:36.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.077+0000 7fd4619a1b80 1 -- start start 2024-03-14T09:59:36.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.077+0000 7fd4619a1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216680 0x558958216a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.077+0000 7fd4619a1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216fa0 0x558958220980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.077+0000 7fd4619a1b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x5589582232b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.077+0000 7fd4619a1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895816f990 con 0x558958216fa0 2024-03-14T09:59:36.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.077+0000 7fd4619a1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895816d350 con 0x558958216680 2024-03-14T09:59:36.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.077+0000 7fd4619a1b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558958168c40 con 0x558958220ec0 2024-03-14T09:59:36.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.077+0000 7fd446000700 1 --2- 172.21.15.78:0/4042973563 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd42c0a0c00 0x7fd42c0a30c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd448101350 tx=0x7fd43400a380 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:36.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.078+0000 7fd446a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216680 0x558958216a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.078+0000 7fd446a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216680 0x558958216a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:43906/0 (socket says 172.21.15.78:43906) 2024-03-14T09:59:36.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.078+0000 7fd446a00700 1 -- 172.21.15.78:0/4112331761 learned_addr learned my addr 172.21.15.78:0/4112331761 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:36.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.078+0000 7fd447400700 1 --2- 172.21.15.78:0/4112331761 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x5589582232b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.078+0000 7fd446000700 1 --2- 172.21.15.78:0/4112331761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216fa0 0x558958220980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.078+0000 7fd446a00700 1 -- 172.21.15.78:0/4112331761 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 msgr2=0x5589582232b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:36.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.078+0000 7fd446a00700 1 --2- 172.21.15.78:0/4112331761 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x5589582232b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.078+0000 7fd446a00700 1 -- 172.21.15.78:0/4112331761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216fa0 msgr2=0x558958220980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:36.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.078+0000 7fd446a00700 1 --2- 172.21.15.78:0/4112331761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216fa0 0x558958220980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.078+0000 7fd446a00700 1 -- 172.21.15.78:0/4112331761 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558957fea3b0 con 0x558958216680 2024-03-14T09:59:36.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.078+0000 7fd447400700 1 --2- 172.21.15.78:0/4112331761 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x5589582232b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:59:36.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.078+0000 7fd446a00700 1 --2- 172.21.15.78:0/4112331761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216680 0x558958216a60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd4380a2910 tx=0x7fd43809b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4f7e6976aab993c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:36.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.078+0000 7fd433e00700 1 -- 172.21.15.78:0/4112331761 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43809c190 con 0x558958216680 2024-03-14T09:59:36.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.079+0000 7fd433e00700 1 -- 172.21.15.78:0/4112331761 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd43800fb00 con 0x558958216680 2024-03-14T09:59:36.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.079+0000 7fd433e00700 1 -- 172.21.15.78:0/4112331761 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4380a29b0 con 0x558958216680 2024-03-14T09:59:36.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.079+0000 7fd4619a1b80 1 -- 172.21.15.78:0/4112331761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216680 msgr2=0x558958216a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.079+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/4112331761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216680 0x558958216a60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd4380a2910 tx=0x7fd43809b860 comp rx=0 tx=0).stop 2024-03-14T09:59:36.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.079+0000 7fd4619a1b80 1 -- 172.21.15.78:0/4112331761 shutdown_connections 2024-03-14T09:59:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.079+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/4112331761 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216680 0x558958216a60 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.079+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/4112331761 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x5589582232b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.079+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/4112331761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216fa0 0x558958220980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.079+0000 7fd4619a1b80 1 -- 172.21.15.78:0/4112331761 >> 172.21.15.78:0/4112331761 conn(0x55895818c220 msgr2=0x558958211aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.080+0000 7fd4619a1b80 1 -- 172.21.15.78:0/4112331761 shutdown_connections 2024-03-14T09:59:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.080+0000 7fd4619a1b80 1 -- 172.21.15.78:0/4112331761 wait complete. 2024-03-14T09:59:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.080+0000 7fd4619a1b80 1 -- start start 2024-03-14T09:59:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.080+0000 7fd4619a1b80 1 Processor -- start 2024-03-14T09:59:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.081+0000 7fd4619a1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216680 0x558958220880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.081+0000 7fd4619a1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216fa0 0x55895821dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.081+0000 7fd4619a1b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x55895821e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.081+0000 7fd4619a1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558958104d80 con 0x558958216fa0 2024-03-14T09:59:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.081+0000 7fd4619a1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558958104a20 con 0x558958216680 2024-03-14T09:59:36.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.081+0000 7fd4619a1b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558957fec6a0 con 0x558958220ec0 2024-03-14T09:59:36.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.081+0000 7fd4619a1b80 10 client.0 _ll_register_callbacks cb 0x558957f703a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:36.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.081+0000 7fd446000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216fa0 0x55895821dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.081+0000 7fd446000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216fa0 0x55895821dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37412/0 (socket says 172.21.15.78:37412) 2024-03-14T09:59:36.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.081+0000 7fd446000700 1 -- 172.21.15.78:0/4196260619 learned_addr learned my addr 172.21.15.78:0/4196260619 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:36.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.081+0000 7fd447400700 1 --2- 172.21.15.78:0/4196260619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x55895821e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.082+0000 7fd446000700 1 -- 172.21.15.78:0/4196260619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 msgr2=0x55895821e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:36.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.082+0000 7fd446000700 1 --2- 172.21.15.78:0/4196260619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x55895821e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.082+0000 7fd446000700 1 -- 172.21.15.78:0/4196260619 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216680 msgr2=0x558958220880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:36.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.082+0000 7fd446000700 1 --2- 172.21.15.78:0/4196260619 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216680 0x558958220880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.082+0000 7fd446000700 1 -- 172.21.15.78:0/4196260619 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4481a66c0 con 0x558958216fa0 2024-03-14T09:59:36.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.082+0000 7fd446000700 1 --2- 172.21.15.78:0/4196260619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216fa0 0x55895821dc00 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd434004690 tx=0x7fd4340047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f774f79c62cb26 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:36.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.082+0000 7fd430c00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:59:36.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.082+0000 7fd430c00700 1 -- 172.21.15.78:0/4196260619 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434010840 con 0x558958216fa0 2024-03-14T09:59:36.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.082+0000 7fd430c00700 1 -- 172.21.15.78:0/4196260619 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fd4340109a0 con 0x558958216fa0 2024-03-14T09:59:36.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.082+0000 7fd4619a1b80 1 -- 172.21.15.78:0/4196260619 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558958209720 con 0x558958216fa0 2024-03-14T09:59:36.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.083+0000 7fd430c00700 1 -- 172.21.15.78:0/4196260619 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd434010ba0 con 0x558958216fa0 2024-03-14T09:59:36.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.083+0000 7fd4619a1b80 10 client.16683 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:36.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.083+0000 7fd4619a1b80 1 -- 172.21.15.78:0/4196260619 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558957f96b60 con 0x558958216fa0 2024-03-14T09:59:36.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.083+0000 7fd430c00700 1 -- 172.21.15.78:0/4196260619 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558958209720 con 0x558958216fa0 2024-03-14T09:59:36.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.084+0000 7fd430c00700 1 -- 172.21.15.78:0/4196260619 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd43401e410 con 0x558958216fa0 2024-03-14T09:59:36.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.084+0000 7fd4619a1b80 10 client.16683 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:36.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.084+0000 7fd4619a1b80 10 client.16683 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:59:36.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.084+0000 7fd4619a1b80 20 client.16683 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:36.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.084+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/4196260619 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x558958239020 0x55895823b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.084+0000 7fd4619a1b80 4 client.16683 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:59:36.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.084+0000 7fd4619a1b80 1 -- 172.21.15.78:0/4196260619 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558958161700 con 0x558958239020 2024-03-14T09:59:36.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.084+0000 7fd446a00700 1 --2- 172.21.15.78:0/4196260619 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x558958239020 0x55895823b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.085+0000 7fd446a00700 1 --2- 172.21.15.78:0/4196260619 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x558958239020 0x55895823b440 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7fdea79d6623d76f server_cookie=6831f6f42fd1f1a7 in_seq=0 out_seq=0 2024-03-14T09:59:36.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.085+0000 7fd430c00700 10 client.16683 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:59:36.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.085+0000 7fd430c00700 1 -- 172.21.15.78:0/4196260619 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x558958161700 con 0x558958239020 2024-03-14T09:59:36.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.085+0000 7fd430c00700 10 client.16683 handle_command_reply: tid=0 2024-03-14T09:59:36.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.085+0000 7fd4619a1b80 1 client.16683 shutdown 2024-03-14T09:59:36.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.085+0000 7fd4619a1b80 1 -- 172.21.15.78:0/4196260619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216fa0 msgr2=0x55895821dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.085+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/4196260619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216fa0 0x55895821dc00 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd434004690 tx=0x7fd4340047c0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.085+0000 7fd4619a1b80 1 -- 172.21.15.78:0/4196260619 shutdown_connections 2024-03-14T09:59:36.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.085+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/4196260619 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x558958239020 0x55895823b440 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.085+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/4196260619 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216680 0x558958220880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.085+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/4196260619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x55895821e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.085+0000 7fd430c00700 0 client.16683 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:59:36.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.085+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/4196260619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216fa0 0x55895821dc00 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.085+0000 7fd4619a1b80 1 -- 172.21.15.78:0/4196260619 >> 172.21.15.78:0/4196260619 conn(0x55895818c220 msgr2=0x558958211690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:36.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.086+0000 7fd4619a1b80 1 -- 172.21.15.78:0/4196260619 shutdown_connections 2024-03-14T09:59:36.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.086+0000 7fd4619a1b80 1 -- 172.21.15.78:0/4196260619 wait complete. 2024-03-14T09:59:36.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.086+0000 7fd4619a1b80 20 client.16683 trim_cache size 0 max 16384 2024-03-14T09:59:36.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.100+0000 7fd4619a1b80 1 Processor -- start 2024-03-14T09:59:36.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.101+0000 7fd4619a1b80 1 -- start start 2024-03-14T09:59:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.101+0000 7fd4619a1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216680 0x5589582300d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.101+0000 7fd4619a1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216fa0 0x558958210180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.101+0000 7fd4619a1b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x558958218ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.101+0000 7fd4619a1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895820e230 con 0x558958216680 2024-03-14T09:59:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.101+0000 7fd4619a1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558958216de0 con 0x558958216fa0 2024-03-14T09:59:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.101+0000 7fd4619a1b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558958212e30 con 0x558958220ec0 2024-03-14T09:59:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.101+0000 7fd446a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216680 0x5589582300d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.101+0000 7fd446000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216fa0 0x558958210180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.101+0000 7fd446a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216680 0x5589582300d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37418/0 (socket says 172.21.15.78:37418) 2024-03-14T09:59:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.101+0000 7fd446a00700 1 -- 172.21.15.78:0/1647138432 learned_addr learned my addr 172.21.15.78:0/1647138432 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.102+0000 7fd447400700 1 --2- 172.21.15.78:0/1647138432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x558958218ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.102+0000 7fd446000700 1 -- 172.21.15.78:0/1647138432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 msgr2=0x558958218ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.102+0000 7fd446000700 1 --2- 172.21.15.78:0/1647138432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x558958218ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.102+0000 7fd446000700 1 -- 172.21.15.78:0/1647138432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216680 msgr2=0x5589582300d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:36.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.102+0000 7fd446000700 1 --2- 172.21.15.78:0/1647138432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216680 0x5589582300d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.102+0000 7fd446000700 1 -- 172.21.15.78:0/1647138432 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55895812eea0 con 0x558958216fa0 2024-03-14T09:59:36.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.102+0000 7fd446000700 1 --2- 172.21.15.78:0/1647138432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216fa0 0x558958210180 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd434004660 tx=0x7fd434005ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=835af8d485a2fd35 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:36.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.102+0000 7fd43d200700 1 -- 172.21.15.78:0/1647138432 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43400a8a0 con 0x558958216fa0 2024-03-14T09:59:36.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.102+0000 7fd43d200700 1 -- 172.21.15.78:0/1647138432 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd43401ea90 con 0x558958216fa0 2024-03-14T09:59:36.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.102+0000 7fd43d200700 1 -- 172.21.15.78:0/1647138432 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4340108a0 con 0x558958216fa0 2024-03-14T09:59:36.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.104+0000 7fd4619a1b80 1 -- 172.21.15.78:0/1647138432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216fa0 msgr2=0x558958210180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.104+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/1647138432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216fa0 0x558958210180 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd434004660 tx=0x7fd434005ee0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.104+0000 7fd4619a1b80 1 -- 172.21.15.78:0/1647138432 shutdown_connections 2024-03-14T09:59:36.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.104+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/1647138432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x558958218ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.104+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/1647138432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216680 0x5589582300d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.104+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/1647138432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216fa0 0x558958210180 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.104+0000 7fd4619a1b80 1 -- 172.21.15.78:0/1647138432 >> 172.21.15.78:0/1647138432 conn(0x55895818c220 msgr2=0x558958220670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:36.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.104+0000 7fd4619a1b80 1 -- 172.21.15.78:0/1647138432 shutdown_connections 2024-03-14T09:59:36.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.104+0000 7fd4619a1b80 1 -- 172.21.15.78:0/1647138432 wait complete. 2024-03-14T09:59:36.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.104+0000 7fd4619a1b80 1 -- start start 2024-03-14T09:59:36.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.105+0000 7fd4619a1b80 1 Processor -- start 2024-03-14T09:59:36.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.105+0000 7fd4619a1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216680 0x55895822dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.105+0000 7fd4619a1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216fa0 0x55895822e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.105+0000 7fd4619a1b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x558958289b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.105+0000 7fd4619a1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895828a080 con 0x558958216680 2024-03-14T09:59:36.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.105+0000 7fd4619a1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895828a1c0 con 0x558958216fa0 2024-03-14T09:59:36.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.105+0000 7fd4619a1b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895828a300 con 0x558958220ec0 2024-03-14T09:59:36.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.106+0000 7fd4619a1b80 10 client.0 _ll_register_callbacks cb 0x7fd434004030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:36.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.106+0000 7fd446000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216fa0 0x55895822e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.106+0000 7fd446000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216fa0 0x55895822e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:43932/0 (socket says 172.21.15.78:43932) 2024-03-14T09:59:36.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.106+0000 7fd447400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x558958289b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.106+0000 7fd446000700 1 -- 172.21.15.78:0/214686495 learned_addr learned my addr 172.21.15.78:0/214686495 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:36.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.106+0000 7fd446a00700 1 --2- 172.21.15.78:0/214686495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216680 0x55895822dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.106+0000 7fd446000700 1 -- 172.21.15.78:0/214686495 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 msgr2=0x558958289b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:36.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.106+0000 7fd446000700 1 --2- 172.21.15.78:0/214686495 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x558958289b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.106+0000 7fd446000700 1 -- 172.21.15.78:0/214686495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216680 msgr2=0x55895822dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:36.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.106+0000 7fd446000700 1 --2- 172.21.15.78:0/214686495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216680 0x55895822dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.106+0000 7fd446000700 1 -- 172.21.15.78:0/214686495 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd43402c040 con 0x558958216fa0 2024-03-14T09:59:36.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.106+0000 7fd446a00700 1 --2- 172.21.15.78:0/214686495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216680 0x55895822dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:59:36.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.107+0000 7fd446000700 1 --2- 172.21.15.78:0/214686495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216fa0 0x55895822e280 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd43400a630 tx=0x7fd434005ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28cd47f6572a5c42 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:36.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.107+0000 7fd430c00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:59:36.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.107+0000 7fd430c00700 1 -- 172.21.15.78:0/214686495 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434005250 con 0x558958216fa0 2024-03-14T09:59:36.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.107+0000 7fd430c00700 1 -- 172.21.15.78:0/214686495 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7fd4340053b0 con 0x558958216fa0 2024-03-14T09:59:36.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.107+0000 7fd4619a1b80 1 -- 172.21.15.78:0/214686495 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd43800fb00 con 0x558958216fa0 2024-03-14T09:59:36.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.107+0000 7fd430c00700 1 -- 172.21.15.78:0/214686495 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd434005570 con 0x558958216fa0 2024-03-14T09:59:36.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.107+0000 7fd4619a1b80 10 client.26756 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:36.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.107+0000 7fd4619a1b80 1 -- 172.21.15.78:0/214686495 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558957feecd0 con 0x558958216fa0 2024-03-14T09:59:36.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.107+0000 7fd430c00700 1 -- 172.21.15.78:0/214686495 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd43800fb00 con 0x558958216fa0 2024-03-14T09:59:36.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.108+0000 7fd430c00700 1 -- 172.21.15.78:0/214686495 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd434045020 con 0x558958216fa0 2024-03-14T09:59:36.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.108+0000 7fd4619a1b80 10 client.26756 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:36.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.108+0000 7fd4619a1b80 10 client.26756 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T09:59:36.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.108+0000 7fd4619a1b80 20 client.26756 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:36.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.108+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/214686495 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55895828b610 0x55895828da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.108+0000 7fd4619a1b80 4 client.26756 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:59:36.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.108+0000 7fd4619a1b80 1 -- 172.21.15.78:0/214686495 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558957fb7cf0 con 0x55895828b610 2024-03-14T09:59:36.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.109+0000 7fd446a00700 1 --2- 172.21.15.78:0/214686495 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55895828b610 0x55895828da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.109+0000 7fd446a00700 1 --2- 172.21.15.78:0/214686495 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55895828b610 0x55895828da00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ca70c2a3b8354de7 server_cookie=6d38724b74cf649a in_seq=0 out_seq=0 2024-03-14T09:59:36.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.109+0000 7fd430c00700 10 client.26756 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T09:59:36.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.109+0000 7fd430c00700 1 -- 172.21.15.78:0/214686495 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558957fb7cf0 con 0x55895828b610 2024-03-14T09:59:36.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.109+0000 7fd430c00700 10 client.26756 handle_command_reply: tid=0 2024-03-14T09:59:36.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.109+0000 7fd4619a1b80 1 client.26756 shutdown 2024-03-14T09:59:36.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.110+0000 7fd4619a1b80 1 -- 172.21.15.78:0/214686495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216fa0 msgr2=0x55895822e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.110+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/214686495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216fa0 0x55895822e280 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd43400a630 tx=0x7fd434005ba0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.110+0000 7fd4619a1b80 1 -- 172.21.15.78:0/214686495 shutdown_connections 2024-03-14T09:59:36.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.110+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/214686495 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55895828b610 0x55895828da00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.110+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/214686495 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558958220ec0 0x558958289b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.110+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/214686495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558958216680 0x55895822dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.110+0000 7fd430c00700 0 client.26756 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T09:59:36.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.110+0000 7fd4619a1b80 1 --2- 172.21.15.78:0/214686495 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558958216fa0 0x55895822e280 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.110+0000 7fd4619a1b80 1 -- 172.21.15.78:0/214686495 >> 172.21.15.78:0/214686495 conn(0x55895818c220 msgr2=0x558958218ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:36.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.110+0000 7fd4619a1b80 1 -- 172.21.15.78:0/214686495 shutdown_connections 2024-03-14T09:59:36.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.110+0000 7fd4619a1b80 1 -- 172.21.15.78:0/214686495 wait complete. 2024-03-14T09:59:36.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.110+0000 7fd4619a1b80 20 client.26756 trim_cache size 0 max 16384 2024-03-14T09:59:36.113 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:59:36.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.111+0000 7fd44de00700 1 -- 172.21.15.78:0/4042973563 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd42c0a0c00 msgr2=0x7fd42c0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.111+0000 7fd44de00700 1 --2- 172.21.15.78:0/4042973563 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd42c0a0c00 0x7fd42c0a30c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd448101350 tx=0x7fd43400a380 comp rx=0 tx=0).stop 2024-03-14T09:59:36.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.111+0000 7fd44de00700 1 -- 172.21.15.78:0/4042973563 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4480680b0 msgr2=0x7fd4480ffc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.111+0000 7fd44de00700 1 --2- 172.21.15.78:0/4042973563 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4480680b0 0x7fd4480ffc20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd438000c00 tx=0x7fd438005740 comp rx=0 tx=0).stop 2024-03-14T09:59:36.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.111+0000 7fd44de00700 1 -- 172.21.15.78:0/4042973563 shutdown_connections 2024-03-14T09:59:36.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.111+0000 7fd44de00700 1 --2- 172.21.15.78:0/4042973563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4481006a0 0x7fd4481a5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.111+0000 7fd44de00700 1 --2- 172.21.15.78:0/4042973563 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd4480680b0 0x7fd4480ffc20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.111+0000 7fd44de00700 1 --2- 172.21.15.78:0/4042973563 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fd42c0a0c00 0x7fd42c0a30c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.111+0000 7fd44de00700 1 --2- 172.21.15.78:0/4042973563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd448106060 0x7fd448100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.111+0000 7fd44de00700 1 -- 172.21.15.78:0/4042973563 >> 172.21.15.78:0/4042973563 conn(0x7fd448076d80 msgr2=0x7fd4480fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:36.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.111+0000 7fd44de00700 1 -- 172.21.15.78:0/4042973563 shutdown_connections 2024-03-14T09:59:36.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.111+0000 7fd44de00700 1 -- 172.21.15.78:0/4042973563 wait complete. 2024-03-14T09:59:36.132 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T09:59:36.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.354+0000 7f8a93a00700 1 -- 172.21.15.78:0/2690404648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a8c0731e0 msgr2=0x7f8a8c0735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.354+0000 7f8a93a00700 1 --2- 172.21.15.78:0/2690404648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a8c0731e0 0x7f8a8c0735c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8a80009990 tx=0x7f8a80009ca0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.354+0000 7f8a93a00700 1 -- 172.21.15.78:0/2690404648 shutdown_connections 2024-03-14T09:59:36.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.354+0000 7f8a93a00700 1 --2- 172.21.15.78:0/2690404648 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a8c0731e0 0x7f8a8c0735c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.354+0000 7f8a93a00700 1 --2- 172.21.15.78:0/2690404648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a8c10d500 0x7f8a8c10f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.354+0000 7f8a93a00700 1 --2- 172.21.15.78:0/2690404648 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a8c073b00 0x7f8a8c10cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.354+0000 7f8a93a00700 1 -- 172.21.15.78:0/2690404648 >> 172.21.15.78:0/2690404648 conn(0x7f8a8c0fc7b0 msgr2=0x7f8a8c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:36.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.354+0000 7f8a93a00700 1 -- 172.21.15.78:0/2690404648 shutdown_connections 2024-03-14T09:59:36.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.354+0000 7f8a93a00700 1 -- 172.21.15.78:0/2690404648 wait complete. 2024-03-14T09:59:36.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.355+0000 7f8a93a00700 1 Processor -- start 2024-03-14T09:59:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.355+0000 7f8a93a00700 1 -- start start 2024-03-14T09:59:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.355+0000 7f8a93a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a8c0731e0 0x7f8a8c19f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.355+0000 7f8a93a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a8c073b00 0x7f8a8c19fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.355+0000 7f8a93a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a8c10d500 0x7f8a8c199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.355+0000 7f8a93a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c199cc0 con 0x7f8a8c10d500 2024-03-14T09:59:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.355+0000 7f8a93a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c199e30 con 0x7f8a8c0731e0 2024-03-14T09:59:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.355+0000 7f8a93a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c199ff0 con 0x7f8a8c073b00 2024-03-14T09:59:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.356+0000 7f8a91a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a8c10d500 0x7f8a8c199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.356+0000 7f8a91a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a8c10d500 0x7f8a8c199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37446/0 (socket says 172.21.15.78:37446) 2024-03-14T09:59:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.356+0000 7f8a91a00700 1 -- 172.21.15.78:0/228597288 learned_addr learned my addr 172.21.15.78:0/228597288 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.356+0000 7f8a8be00700 1 --2- 172.21.15.78:0/228597288 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a8c073b00 0x7f8a8c19fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.356+0000 7f8a91000700 1 --2- 172.21.15.78:0/228597288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a8c0731e0 0x7f8a8c19f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.356+0000 7f8a91a00700 1 -- 172.21.15.78:0/228597288 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a8c073b00 msgr2=0x7f8a8c19fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.356+0000 7f8a91a00700 1 --2- 172.21.15.78:0/228597288 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a8c073b00 0x7f8a8c19fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.356+0000 7f8a91a00700 1 -- 172.21.15.78:0/228597288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a8c0731e0 msgr2=0x7f8a8c19f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.356+0000 7f8a91a00700 1 --2- 172.21.15.78:0/228597288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a8c0731e0 0x7f8a8c19f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.356+0000 7f8a91a00700 1 -- 172.21.15.78:0/228597288 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a800095f0 con 0x7f8a8c10d500 2024-03-14T09:59:36.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.356+0000 7f8a91a00700 1 --2- 172.21.15.78:0/228597288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a8c10d500 0x7f8a8c199660 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f8a8400ef30 tx=0x7f8a8400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:36.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.357+0000 7f8a89600700 1 -- 172.21.15.78:0/228597288 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84018070 con 0x7f8a8c10d500 2024-03-14T09:59:36.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.357+0000 7f8a93a00700 1 -- 172.21.15.78:0/228597288 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a8c19a270 con 0x7f8a8c10d500 2024-03-14T09:59:36.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.357+0000 7f8a89600700 1 -- 172.21.15.78:0/228597288 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a8400f040 con 0x7f8a8c10d500 2024-03-14T09:59:36.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.357+0000 7f8a89600700 1 -- 172.21.15.78:0/228597288 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8400cd00 con 0x7f8a8c10d500 2024-03-14T09:59:36.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.357+0000 7f8a93a00700 1 -- 172.21.15.78:0/228597288 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a8c1a6520 con 0x7f8a8c10d500 2024-03-14T09:59:36.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.360+0000 7f8a89600700 1 -- 172.21.15.78:0/228597288 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f8a84003a00 con 0x7f8a8c10d500 2024-03-14T09:59:36.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.361+0000 7f8aa75adb80 1 Processor -- start 2024-03-14T09:59:36.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.361+0000 7f8aa75adb80 1 -- start start 2024-03-14T09:59:36.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.361+0000 7f8a89600700 1 --2- 172.21.15.78:0/228597288 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8a740a0c00 0x7f8a740a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.361+0000 7f8a89600700 1 -- 172.21.15.78:0/228597288 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f8a840cc2a0 con 0x7f8a8c10d500 2024-03-14T09:59:36.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.361+0000 7f8aa75adb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a7831a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.361+0000 7f8aa75adb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831fa0 0x55a2a783b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.361+0000 7f8aa75adb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 0x55a2a783e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.361+0000 7f8aa75adb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2a778a990 con 0x55a2a7831fa0 2024-03-14T09:59:36.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.361+0000 7f8aa75adb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2a7788350 con 0x55a2a783bec0 2024-03-14T09:59:36.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.361+0000 7f8aa75adb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2a7783c40 con 0x55a2a7831680 2024-03-14T09:59:36.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.361+0000 7f8a8be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831fa0 0x55a2a783b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.361+0000 7f8a91000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a7831a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.361+0000 7f8a8be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831fa0 0x55a2a783b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37456/0 (socket says 172.21.15.78:37456) 2024-03-14T09:59:36.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.361+0000 7f8a91000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a7831a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60272/0 (socket says 172.21.15.78:60272) 2024-03-14T09:59:36.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.361+0000 7f8a8be00700 1 -- 172.21.15.78:0/3395994779 learned_addr learned my addr 172.21.15.78:0/3395994779 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:36.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.362+0000 7f8a8be00700 1 --2- 172.21.15.78:0/228597288 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8a740a0c00 0x7f8a740a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.362+0000 7f8a91a00700 1 --2- 172.21.15.78:0/3395994779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 0x55a2a783e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.362+0000 7f8a91000700 1 -- 172.21.15.78:0/3395994779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 msgr2=0x55a2a783e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:36.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.362+0000 7f8a91000700 1 --2- 172.21.15.78:0/3395994779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 0x55a2a783e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.362+0000 7f8a91000700 1 -- 172.21.15.78:0/3395994779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831fa0 msgr2=0x55a2a783b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:36.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.362+0000 7f8a91000700 1 --2- 172.21.15.78:0/3395994779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831fa0 0x55a2a783b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.362+0000 7f8a91000700 1 -- 172.21.15.78:0/3395994779 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2a76053b0 con 0x55a2a7831680 2024-03-14T09:59:36.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.362+0000 7f8a91000700 1 --2- 172.21.15.78:0/3395994779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a7831a60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8a80000c00 tx=0x7f8a80005ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9081302fe13c909 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:36.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.362+0000 7f8a8be00700 1 --2- 172.21.15.78:0/228597288 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8a740a0c00 0x7f8a740a30c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009fd0 tx=0x7f8a78009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:36.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.362+0000 7f8a7e000700 1 -- 172.21.15.78:0/3395994779 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8001d070 con 0x55a2a7831680 2024-03-14T09:59:36.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.362+0000 7f8a7e000700 1 -- 172.21.15.78:0/3395994779 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a8001e040 con 0x55a2a7831680 2024-03-14T09:59:36.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.362+0000 7f8a7e000700 1 -- 172.21.15.78:0/3395994779 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a800047f0 con 0x55a2a7831680 2024-03-14T09:59:36.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.363+0000 7f8aa75adb80 1 -- 172.21.15.78:0/3395994779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 msgr2=0x55a2a7831a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.363+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/3395994779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a7831a60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8a80000c00 tx=0x7f8a80005ec0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.363+0000 7f8aa75adb80 1 -- 172.21.15.78:0/3395994779 shutdown_connections 2024-03-14T09:59:36.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.363+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/3395994779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 0x55a2a783e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.363+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/3395994779 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a7831a60 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.363+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/3395994779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831fa0 0x55a2a783b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.363+0000 7f8aa75adb80 1 -- 172.21.15.78:0/3395994779 >> 172.21.15.78:0/3395994779 conn(0x55a2a77a7220 msgr2=0x55a2a782caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:36.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.363+0000 7f8aa75adb80 1 -- 172.21.15.78:0/3395994779 shutdown_connections 2024-03-14T09:59:36.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.363+0000 7f8aa75adb80 1 -- 172.21.15.78:0/3395994779 wait complete. 2024-03-14T09:59:36.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.364+0000 7f8aa75adb80 1 -- start start 2024-03-14T09:59:36.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.364+0000 7f8aa75adb80 1 Processor -- start 2024-03-14T09:59:36.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8aa75adb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a783b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8aa75adb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a7831fa0 0x55a2a7838ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8aa75adb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a783bec0 0x55a2a78391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8aa75adb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2a771fd80 con 0x55a2a783bec0 2024-03-14T09:59:36.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8aa75adb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2a771fa20 con 0x55a2a7831fa0 2024-03-14T09:59:36.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8aa75adb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2a76076a0 con 0x55a2a7831680 2024-03-14T09:59:36.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8aa75adb80 10 client.0 _ll_register_callbacks cb 0x55a2a758b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:36.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8a91000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a783b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8a91000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a783b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60288/0 (socket says 172.21.15.78:60288) 2024-03-14T09:59:36.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8a91000700 1 -- 172.21.15.78:0/1915833936 learned_addr learned my addr 172.21.15.78:0/1915833936 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:36.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8a8be00700 1 --2- 172.21.15.78:0/1915833936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a7831fa0 0x55a2a7838ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8a91a00700 1 --2- 172.21.15.78:0/1915833936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a783bec0 0x55a2a78391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8a91000700 1 -- 172.21.15.78:0/1915833936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a7831fa0 msgr2=0x55a2a7838ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:36.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8a91000700 1 --2- 172.21.15.78:0/1915833936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a7831fa0 0x55a2a7838ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8a91000700 1 -- 172.21.15.78:0/1915833936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a783bec0 msgr2=0x55a2a78391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:36.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8a91000700 1 --2- 172.21.15.78:0/1915833936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a783bec0 0x55a2a78391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.365+0000 7f8a91000700 1 -- 172.21.15.78:0/1915833936 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a8c110630 con 0x55a2a7831680 2024-03-14T09:59:36.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.366+0000 7f8a91000700 1 --2- 172.21.15.78:0/1915833936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a783b8b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8a80005f50 tx=0x7f8a8000b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=367bf76d2f734957 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:36.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.366+0000 7f8a72a00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:59:36.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.366+0000 7f8a72a00700 1 -- 172.21.15.78:0/1915833936 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8001d070 con 0x55a2a7831680 2024-03-14T09:59:36.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.366+0000 7f8a72a00700 1 -- 172.21.15.78:0/1915833936 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f8a80004700 con 0x55a2a7831680 2024-03-14T09:59:36.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.366+0000 7f8aa75adb80 1 -- 172.21.15.78:0/1915833936 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a2a7824720 con 0x55a2a7831680 2024-03-14T09:59:36.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.366+0000 7f8a72a00700 1 -- 172.21.15.78:0/1915833936 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a80005bc0 con 0x55a2a7831680 2024-03-14T09:59:36.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.366+0000 7f8aa75adb80 10 client.26755 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:36.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.366+0000 7f8aa75adb80 1 -- 172.21.15.78:0/1915833936 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a2a75b1b60 con 0x55a2a7831680 2024-03-14T09:59:36.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.366+0000 7f8a72a00700 1 -- 172.21.15.78:0/1915833936 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a2a7824720 con 0x55a2a7831680 2024-03-14T09:59:36.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.367+0000 7f8a72a00700 1 -- 172.21.15.78:0/1915833936 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8a8000f840 con 0x55a2a7831680 2024-03-14T09:59:36.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.367+0000 7f8aa75adb80 10 client.26755 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:36.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.367+0000 7f8aa75adb80 10 client.26755 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:59:36.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.367+0000 7f8aa75adb80 20 client.26755 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:36.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.367+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/1915833936 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a2a784b600 0x55a2a784da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.367+0000 7f8aa75adb80 4 client.26755 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:59:36.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.367+0000 7f8aa75adb80 1 -- 172.21.15.78:0/1915833936 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a2a777c700 con 0x55a2a784b600 2024-03-14T09:59:36.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.367+0000 7f8a8be00700 1 --2- 172.21.15.78:0/1915833936 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a2a784b600 0x55a2a784da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.367+0000 7f8a72a00700 10 client.26755 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:36.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.367+0000 7f8a8be00700 1 --2- 172.21.15.78:0/1915833936 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a2a784b600 0x55a2a784da20 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71ef0d66c2f751e3 server_cookie=1f7c767dca9de16 in_seq=0 out_seq=0 2024-03-14T09:59:36.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.368+0000 7f8a72a00700 1 -- 172.21.15.78:0/1915833936 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55a2a777c700 con 0x55a2a784b600 2024-03-14T09:59:36.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.368+0000 7f8a72a00700 10 client.26755 handle_command_reply: tid=0 2024-03-14T09:59:36.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.368+0000 7f8aa75adb80 1 client.26755 shutdown 2024-03-14T09:59:36.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.368+0000 7f8aa75adb80 1 -- 172.21.15.78:0/1915833936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 msgr2=0x55a2a783b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.368+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/1915833936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a783b8b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8a80005f50 tx=0x7f8a8000b980 comp rx=0 tx=0).stop 2024-03-14T09:59:36.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.368+0000 7f8aa75adb80 1 -- 172.21.15.78:0/1915833936 shutdown_connections 2024-03-14T09:59:36.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.368+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/1915833936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a783bec0 0x55a2a78391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.368+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/1915833936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a783b8b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.368+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/1915833936 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a2a784b600 0x55a2a784da20 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.368+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/1915833936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a7831fa0 0x55a2a7838ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.368+0000 7f8a72a00700 0 client.26755 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:36.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.368+0000 7f8aa75adb80 1 -- 172.21.15.78:0/1915833936 >> 172.21.15.78:0/1915833936 conn(0x55a2a77a7220 msgr2=0x55a2a782c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:36.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.369+0000 7f8aa75adb80 1 -- 172.21.15.78:0/1915833936 shutdown_connections 2024-03-14T09:59:36.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.369+0000 7f8aa75adb80 1 -- 172.21.15.78:0/1915833936 wait complete. 2024-03-14T09:59:36.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.369+0000 7f8aa75adb80 20 client.26755 trim_cache size 0 max 16384 2024-03-14T09:59:36.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.382+0000 7f8aa75adb80 1 Processor -- start 2024-03-14T09:59:36.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.382+0000 7f8aa75adb80 1 -- start start 2024-03-14T09:59:36.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.382+0000 7f8aa75adb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831680 0x55a2a7842af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.382+0000 7f8aa75adb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831fa0 0x55a2a78324b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.382+0000 7f8aa75adb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 0x55a2a783e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.382+0000 7f8aa75adb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2a78332e0 con 0x55a2a7831680 2024-03-14T09:59:36.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.382+0000 7f8aa75adb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2a782e060 con 0x55a2a783bec0 2024-03-14T09:59:36.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.382+0000 7f8aa75adb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2a7826d00 con 0x55a2a7831fa0 2024-03-14T09:59:36.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.382+0000 7f8a91000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831680 0x55a2a7842af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.382+0000 7f8a91000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831680 0x55a2a7842af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37484/0 (socket says 172.21.15.78:37484) 2024-03-14T09:59:36.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.382+0000 7f8a91a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 0x55a2a783e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.382+0000 7f8a91000700 1 -- 172.21.15.78:0/1991588038 learned_addr learned my addr 172.21.15.78:0/1991588038 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:36.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.382+0000 7f8a8be00700 1 --2- 172.21.15.78:0/1991588038 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831fa0 0x55a2a78324b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.383+0000 7f8a91a00700 1 -- 172.21.15.78:0/1991588038 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831fa0 msgr2=0x55a2a78324b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:36.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.383+0000 7f8a91a00700 1 --2- 172.21.15.78:0/1991588038 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831fa0 0x55a2a78324b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.383+0000 7f8a91a00700 1 -- 172.21.15.78:0/1991588038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831680 msgr2=0x55a2a7842af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:36.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.383+0000 7f8a91a00700 1 --2- 172.21.15.78:0/1991588038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831680 0x55a2a7842af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.383+0000 7f8a91a00700 1 -- 172.21.15.78:0/1991588038 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2a7749ea0 con 0x55a2a783bec0 2024-03-14T09:59:36.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.383+0000 7f8a8be00700 1 --2- 172.21.15.78:0/1991588038 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831fa0 0x55a2a78324b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:59:36.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.383+0000 7f8a91000700 1 --2- 172.21.15.78:0/1991588038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831680 0x55a2a7842af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:36.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.383+0000 7f8a91a00700 1 --2- 172.21.15.78:0/1991588038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 0x55a2a783e2b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8a840c9640 tx=0x7f8a840093f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc7275f5ab761baa server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:36.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.383+0000 7f8a7ea00700 1 -- 172.21.15.78:0/1991588038 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84004ad0 con 0x55a2a783bec0 2024-03-14T09:59:36.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.383+0000 7f8a7ea00700 1 -- 172.21.15.78:0/1991588038 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a840cc0c0 con 0x55a2a783bec0 2024-03-14T09:59:36.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.383+0000 7f8a7ea00700 1 -- 172.21.15.78:0/1991588038 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8409d100 con 0x55a2a783bec0 2024-03-14T09:59:36.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.384+0000 7f8aa75adb80 1 -- 172.21.15.78:0/1991588038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 msgr2=0x55a2a783e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.384+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/1991588038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 0x55a2a783e2b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8a840c9640 tx=0x7f8a840093f0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.384+0000 7f8aa75adb80 1 -- 172.21.15.78:0/1991588038 shutdown_connections 2024-03-14T09:59:36.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.384+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/1991588038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 0x55a2a783e2b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.384+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/1991588038 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831680 0x55a2a7842af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.384+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/1991588038 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831fa0 0x55a2a78324b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.384+0000 7f8aa75adb80 1 -- 172.21.15.78:0/1991588038 >> 172.21.15.78:0/1991588038 conn(0x55a2a77a7220 msgr2=0x55a2a782b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:36.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.384+0000 7f8aa75adb80 1 -- 172.21.15.78:0/1991588038 shutdown_connections 2024-03-14T09:59:36.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.384+0000 7f8aa75adb80 1 -- 172.21.15.78:0/1991588038 wait complete. 2024-03-14T09:59:36.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.384+0000 7f8aa75adb80 1 -- start start 2024-03-14T09:59:36.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.384+0000 7f8aa75adb80 1 Processor -- start 2024-03-14T09:59:36.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8aa75adb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a7840cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8aa75adb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831fa0 0x55a2a7841200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8aa75adb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 0x55a2a7896120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8aa75adb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2a78966a0 con 0x55a2a7831fa0 2024-03-14T09:59:36.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8aa75adb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2a7896810 con 0x55a2a783bec0 2024-03-14T09:59:36.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8aa75adb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2a78969d0 con 0x55a2a7831680 2024-03-14T09:59:36.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8aa75adb80 10 client.0 _ll_register_callbacks cb 0x7f8a78009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:36.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8a91000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a7840cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8a91a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 0x55a2a7896120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8a8be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831fa0 0x55a2a7841200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8a91000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a7840cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60314/0 (socket says 172.21.15.78:60314) 2024-03-14T09:59:36.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8a91000700 1 -- 172.21.15.78:0/3385960792 learned_addr learned my addr 172.21.15.78:0/3385960792 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:36.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8a91a00700 1 -- 172.21.15.78:0/3385960792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 msgr2=0x55a2a7840cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:36.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8a91a00700 1 --2- 172.21.15.78:0/3385960792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a7840cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8a91a00700 1 -- 172.21.15.78:0/3385960792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831fa0 msgr2=0x55a2a7841200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8a91a00700 1 --2- 172.21.15.78:0/3385960792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831fa0 0x55a2a7841200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.385+0000 7f8a91a00700 1 -- 172.21.15.78:0/3385960792 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a2a75b1b60 con 0x55a2a783bec0 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.386+0000 7f8a91a00700 1 --2- 172.21.15.78:0/3385960792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 0x55a2a7896120 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8a840957d0 tx=0x7f8a8401b320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c886794cc6f493ca server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.386+0000 7f8a72a00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.386+0000 7f8a72a00700 1 -- 172.21.15.78:0/3385960792 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8407f070 con 0x55a2a783bec0 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.386+0000 7f8a72a00700 1 -- 172.21.15.78:0/3385960792 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f8a840af070 con 0x55a2a783bec0 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.386+0000 7f8aa75adb80 1 -- 172.21.15.78:0/3385960792 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8a8001e040 con 0x55a2a783bec0 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.386+0000 7f8a72a00700 1 -- 172.21.15.78:0/3385960792 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a8400ce60 con 0x55a2a783bec0 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.386+0000 7f8aa75adb80 10 client.26774 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.386+0000 7f8aa75adb80 1 -- 172.21.15.78:0/3385960792 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f8a8c1a6520 con 0x55a2a783bec0 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.386+0000 7f8a72a00700 1 -- 172.21.15.78:0/3385960792 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a8001e040 con 0x55a2a783bec0 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.386+0000 7f8a72a00700 1 -- 172.21.15.78:0/3385960792 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8a84082020 con 0x55a2a783bec0 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.386+0000 7f8aa75adb80 10 client.26774 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.386+0000 7f8aa75adb80 10 client.26774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.386+0000 7f8aa75adb80 20 client.26774 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.386+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/3385960792 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a2a789dd80 0x55a2a78a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.386+0000 7f8aa75adb80 4 client.26774 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T09:59:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.386+0000 7f8aa75adb80 1 -- 172.21.15.78:0/3385960792 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a2a75d2cf0 con 0x55a2a789dd80 2024-03-14T09:59:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.387+0000 7f8a91000700 1 --2- 172.21.15.78:0/3385960792 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a2a789dd80 0x55a2a78a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.387+0000 7f8a91000700 1 --2- 172.21.15.78:0/3385960792 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a2a789dd80 0x55a2a78a01a0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26c5591eb1a70962 server_cookie=f1d68f9633830d8f in_seq=0 out_seq=0 2024-03-14T09:59:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.387+0000 7f8a72a00700 10 client.26774 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.387+0000 7f8a72a00700 1 -- 172.21.15.78:0/3385960792 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a2a75d2cf0 con 0x55a2a789dd80 2024-03-14T09:59:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.387+0000 7f8a72a00700 10 client.26774 handle_command_reply: tid=0 2024-03-14T09:59:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.387+0000 7f8aa75adb80 1 client.26774 shutdown 2024-03-14T09:59:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.388+0000 7f8aa75adb80 1 -- 172.21.15.78:0/3385960792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 msgr2=0x55a2a7896120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.388+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/3385960792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 0x55a2a7896120 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8a840957d0 tx=0x7f8a8401b320 comp rx=0 tx=0).stop 2024-03-14T09:59:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.388+0000 7f8aa75adb80 1 -- 172.21.15.78:0/3385960792 shutdown_connections 2024-03-14T09:59:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.388+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/3385960792 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55a2a789dd80 0x55a2a78a01a0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.388+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/3385960792 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a2a783bec0 0x55a2a7896120 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.388+0000 7f8a72a00700 0 client.26774 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.388+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/3385960792 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a2a7831680 0x55a2a7840cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.388+0000 7f8aa75adb80 1 --2- 172.21.15.78:0/3385960792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a2a7831fa0 0x55a2a7841200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.388+0000 7f8aa75adb80 1 -- 172.21.15.78:0/3385960792 >> 172.21.15.78:0/3385960792 conn(0x55a2a77a7220 msgr2=0x55a2a7830560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:36.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.388+0000 7f8aa75adb80 1 -- 172.21.15.78:0/3385960792 shutdown_connections 2024-03-14T09:59:36.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.388+0000 7f8aa75adb80 1 -- 172.21.15.78:0/3385960792 wait complete. 2024-03-14T09:59:36.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.388+0000 7f8aa75adb80 20 client.26774 trim_cache size 0 max 16384 2024-03-14T09:59:36.393 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T09:59:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.389+0000 7f8a93a00700 1 -- 172.21.15.78:0/228597288 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8a740a0c00 msgr2=0x7f8a740a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.389+0000 7f8a93a00700 1 --2- 172.21.15.78:0/228597288 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8a740a0c00 0x7f8a740a30c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009fd0 tx=0x7f8a78009260 comp rx=0 tx=0).stop 2024-03-14T09:59:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.389+0000 7f8a93a00700 1 -- 172.21.15.78:0/228597288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a8c10d500 msgr2=0x7f8a8c199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.389+0000 7f8a93a00700 1 --2- 172.21.15.78:0/228597288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a8c10d500 0x7f8a8c199660 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f8a8400ef30 tx=0x7f8a8400c3a0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.389+0000 7f8a93a00700 1 -- 172.21.15.78:0/228597288 shutdown_connections 2024-03-14T09:59:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.389+0000 7f8a93a00700 1 --2- 172.21.15.78:0/228597288 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f8a740a0c00 0x7f8a740a30c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.389+0000 7f8a93a00700 1 --2- 172.21.15.78:0/228597288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a8c0731e0 0x7f8a8c19f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.389+0000 7f8a93a00700 1 --2- 172.21.15.78:0/228597288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a8c10d500 0x7f8a8c199660 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.389+0000 7f8a93a00700 1 --2- 172.21.15.78:0/228597288 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f8a8c073b00 0x7f8a8c19fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.389+0000 7f8a93a00700 1 -- 172.21.15.78:0/228597288 >> 172.21.15.78:0/228597288 conn(0x7f8a8c0fc7b0 msgr2=0x7f8a8c0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.389+0000 7f8a93a00700 1 -- 172.21.15.78:0/228597288 shutdown_connections 2024-03-14T09:59:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:36.389+0000 7f8a93a00700 1 -- 172.21.15.78:0/228597288 wait complete. 2024-03-14T09:59:36.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:36 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/940031223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:59:36.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:36 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/940031223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:59:36.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:36 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/940031223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T09:59:37.418 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T09:59:37.418 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T09:59:37.490 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:37 smithi078 ceph-mon[38143]: pgmap v166: 129 pgs: 129 active+clean; 210 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 557 op/s 2024-03-14T09:59:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:37 smithi152 ceph-mon[48279]: pgmap v166: 129 pgs: 129 active+clean; 210 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 557 op/s 2024-03-14T09:59:37.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:37 smithi132 ceph-mon[47959]: pgmap v166: 129 pgs: 129 active+clean; 210 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 557 op/s 2024-03-14T09:59:37.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.671+0000 7f82f0200700 1 -- 172.21.15.78:0/752466577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82e811a590 msgr2=0x7f82e811c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:37.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.671+0000 7f82f0200700 1 --2- 172.21.15.78:0/752466577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82e811a590 0x7f82e811c980 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f82e000b0a0 tx=0x7f82e000b3b0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.671+0000 7f82f0200700 1 -- 172.21.15.78:0/752466577 shutdown_connections 2024-03-14T09:59:37.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.671+0000 7f82f0200700 1 --2- 172.21.15.78:0/752466577 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82e811a590 0x7f82e811c980 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.671+0000 7f82f0200700 1 --2- 172.21.15.78:0/752466577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82e810f4d0 0x7f82e8107db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.671+0000 7f82f0200700 1 --2- 172.21.15.78:0/752466577 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82e810f940 0x7f82e81082f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.671+0000 7f82f0200700 1 -- 172.21.15.78:0/752466577 >> 172.21.15.78:0/752466577 conn(0x7f82e806bdc0 msgr2=0x7f82e806c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.671+0000 7f82f0200700 1 -- 172.21.15.78:0/752466577 shutdown_connections 2024-03-14T09:59:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.672+0000 7f82f0200700 1 -- 172.21.15.78:0/752466577 wait complete. 2024-03-14T09:59:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.673+0000 7f82f0200700 1 Processor -- start 2024-03-14T09:59:37.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.674+0000 7f82f0200700 1 -- start start 2024-03-14T09:59:37.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.674+0000 7f82f0200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82e810f4d0 0x7f82e81a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.674+0000 7f82f0200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82e810f940 0x7f82e81a6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.674+0000 7f82f0200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82e811a590 0x7f82e81aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.674+0000 7f82f0200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e81ab530 con 0x7f82e810f940 2024-03-14T09:59:37.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.674+0000 7f82f0200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e81ab6a0 con 0x7f82e810f4d0 2024-03-14T09:59:37.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.674+0000 7f82f0200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e81ab860 con 0x7f82e811a590 2024-03-14T09:59:37.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.675+0000 7f82ed800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82e810f4d0 0x7f82e81a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.675+0000 7f82ed800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82e810f4d0 0x7f82e81a6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:43998/0 (socket says 172.21.15.78:43998) 2024-03-14T09:59:37.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.675+0000 7f82ed800700 1 -- 172.21.15.78:0/3863377971 learned_addr learned my addr 172.21.15.78:0/3863377971 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:37.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.675+0000 7f82ece00700 1 --2- 172.21.15.78:0/3863377971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82e810f940 0x7f82e81a6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.675+0000 7f82ee200700 1 --2- 172.21.15.78:0/3863377971 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82e811a590 0x7f82e81aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.675+0000 7f82ece00700 1 -- 172.21.15.78:0/3863377971 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82e811a590 msgr2=0x7f82e81aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:37.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.675+0000 7f82ece00700 1 --2- 172.21.15.78:0/3863377971 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82e811a590 0x7f82e81aaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.675+0000 7f82ece00700 1 -- 172.21.15.78:0/3863377971 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82e810f4d0 msgr2=0x7f82e81a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:37.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.675+0000 7f82ece00700 1 --2- 172.21.15.78:0/3863377971 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82e810f4d0 0x7f82e81a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.675+0000 7f82ece00700 1 -- 172.21.15.78:0/3863377971 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e0009d00 con 0x7f82e810f940 2024-03-14T09:59:37.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.675+0000 7f82ed800700 1 --2- 172.21.15.78:0/3863377971 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82e810f4d0 0x7f82e81a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.676+0000 7f82ece00700 1 --2- 172.21.15.78:0/3863377971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82e810f940 0x7f82e81a6b50 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f82e810cf70 tx=0x7f82d400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.676+0000 7f82e6000700 1 -- 172.21.15.78:0/3863377971 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d400cbd0 con 0x7f82e810f940 2024-03-14T09:59:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.676+0000 7f82e6000700 1 -- 172.21.15.78:0/3863377971 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82d400ee80 con 0x7f82e810f940 2024-03-14T09:59:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.676+0000 7f82e6000700 1 -- 172.21.15.78:0/3863377971 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4018680 con 0x7f82e810f940 2024-03-14T09:59:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.676+0000 7f82f0200700 1 -- 172.21.15.78:0/3863377971 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e81abb40 con 0x7f82e810f940 2024-03-14T09:59:37.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.676+0000 7f82f0200700 1 -- 172.21.15.78:0/3863377971 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e81b3380 con 0x7f82e810f940 2024-03-14T09:59:37.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.681+0000 7f82e6000700 1 -- 172.21.15.78:0/3863377971 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f82d4010430 con 0x7f82e810f940 2024-03-14T09:59:37.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.682+0000 7f8303d2cb80 1 Processor -- start 2024-03-14T09:59:37.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.682+0000 7f8303d2cb80 1 -- start start 2024-03-14T09:59:37.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.682+0000 7f82e6000700 1 --2- 172.21.15.78:0/3863377971 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f82d00a0c80 0x7f82d00a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.683+0000 7f82e6000700 1 -- 172.21.15.78:0/3863377971 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f82d4014070 con 0x7f82e810f940 2024-03-14T09:59:37.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.683+0000 7f8303d2cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.683+0000 7f8303d2cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 0x55e7221fda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.683+0000 7f8303d2cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 0x55e722200330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.683+0000 7f8303d2cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72214ca10 con 0x55e7221fdf40 2024-03-14T09:59:37.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.683+0000 7f8303d2cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72214a3d0 con 0x55e7221f3700 2024-03-14T09:59:37.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.683+0000 7f8303d2cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e722145cc0 con 0x55e7221f4020 2024-03-14T09:59:37.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.683+0000 7f82ed800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.683+0000 7f82ece00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 0x55e7221fda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.683+0000 7f82ed800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221f3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:44004/0 (socket says 172.21.15.78:44004) 2024-03-14T09:59:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.683+0000 7f82ed800700 1 -- 172.21.15.78:0/1577407092 learned_addr learned my addr 172.21.15.78:0/1577407092 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.683+0000 7f82ece00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 0x55e7221fda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60342/0 (socket says 172.21.15.78:60342) 2024-03-14T09:59:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.683+0000 7f82ee200700 1 --2- 172.21.15.78:0/3863377971 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f82d00a0c80 0x7f82d00a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.684+0000 7f82ee200700 1 --2- 172.21.15.78:0/1577407092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 0x55e722200330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.684+0000 7f82ed800700 1 -- 172.21.15.78:0/1577407092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 msgr2=0x55e7221fda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.684+0000 7f82ed800700 1 --2- 172.21.15.78:0/1577407092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 0x55e7221fda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.684+0000 7f82ed800700 1 -- 172.21.15.78:0/1577407092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 msgr2=0x55e722200330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.684+0000 7f82ed800700 1 --2- 172.21.15.78:0/1577407092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 0x55e722200330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.684+0000 7f82ed800700 1 -- 172.21.15.78:0/1577407092 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e721fc7430 con 0x55e7221f3700 2024-03-14T09:59:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.684+0000 7f82ee200700 1 --2- 172.21.15.78:0/1577407092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 0x55e722200330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:59:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.684+0000 7f82ed800700 1 --2- 172.21.15.78:0/1577407092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221f3ae0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f82dc0107a0 tx=0x7f82dc010ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3947b3683f8913c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:37.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.684+0000 7f82ee200700 1 --2- 172.21.15.78:0/3863377971 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f82d00a0c80 0x7f82d00a3140 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009850 tx=0x7f82e00097a0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:37.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.684+0000 7f82daa00700 1 -- 172.21.15.78:0/1577407092 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc011700 con 0x55e7221f3700 2024-03-14T09:59:37.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.685+0000 7f82daa00700 1 -- 172.21.15.78:0/1577407092 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82dc015070 con 0x55e7221f3700 2024-03-14T09:59:37.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.685+0000 7f82daa00700 1 -- 172.21.15.78:0/1577407092 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc019b40 con 0x55e7221f3700 2024-03-14T09:59:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.686+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1577407092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 msgr2=0x55e7221f3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.686+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/1577407092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221f3ae0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f82dc0107a0 tx=0x7f82dc010ab0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.686+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1577407092 shutdown_connections 2024-03-14T09:59:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.686+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/1577407092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221f3ae0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.686+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/1577407092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 0x55e722200330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.686+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/1577407092 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 0x55e7221fda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.686+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1577407092 >> 172.21.15.78:0/1577407092 conn(0x55e7221692a0 msgr2=0x55e7221eeb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.686+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1577407092 shutdown_connections 2024-03-14T09:59:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.686+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1577407092 wait complete. 2024-03-14T09:59:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.687+0000 7f8303d2cb80 1 -- start start 2024-03-14T09:59:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.687+0000 7f8303d2cb80 1 Processor -- start 2024-03-14T09:59:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.688+0000 7f8303d2cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221fd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.688+0000 7f8303d2cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 0x55e7221fac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.688+0000 7f8303d2cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 0x55e7221fb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.688+0000 7f8303d2cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7220e1e00 con 0x55e7221fdf40 2024-03-14T09:59:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.688+0000 7f8303d2cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7220e1aa0 con 0x55e7221f3700 2024-03-14T09:59:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.688+0000 7f8303d2cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e721fc9720 con 0x55e7221f4020 2024-03-14T09:59:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.688+0000 7f8303d2cb80 10 client.0 _ll_register_callbacks cb 0x55e721f4d420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.688+0000 7f82ed800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221fd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.688+0000 7f82ed800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221fd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:44010/0 (socket says 172.21.15.78:44010) 2024-03-14T09:59:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.688+0000 7f82ed800700 1 -- 172.21.15.78:0/1394383878 learned_addr learned my addr 172.21.15.78:0/1394383878 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.688+0000 7f82ee200700 1 --2- 172.21.15.78:0/1394383878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 0x55e7221fb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.688+0000 7f82ece00700 1 --2- 172.21.15.78:0/1394383878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 0x55e7221fac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.688+0000 7f82ed800700 1 -- 172.21.15.78:0/1394383878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 msgr2=0x55e7221fac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:37.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.689+0000 7f82ed800700 1 --2- 172.21.15.78:0/1394383878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 0x55e7221fac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.689+0000 7f82ed800700 1 -- 172.21.15.78:0/1394383878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 msgr2=0x55e7221fb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:37.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.689+0000 7f82ed800700 1 --2- 172.21.15.78:0/1394383878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 0x55e7221fb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.689+0000 7f82ed800700 1 -- 172.21.15.78:0/1394383878 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82dc0103d0 con 0x55e7221f3700 2024-03-14T09:59:37.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.689+0000 7f82ee200700 1 --2- 172.21.15.78:0/1394383878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 0x55e7221fb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:59:37.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.689+0000 7f82ed800700 1 --2- 172.21.15.78:0/1394383878 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221fd8f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00b7c0 tx=0x7f82dc0194e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59cfc1ec464b7696 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:37.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.689+0000 7f82cf400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T09:59:37.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.689+0000 7f82cf400700 1 -- 172.21.15.78:0/1394383878 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc0053c0 con 0x55e7221f3700 2024-03-14T09:59:37.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.689+0000 7f82cf400700 1 -- 172.21.15.78:0/1394383878 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f82dc005520 con 0x55e7221f3700 2024-03-14T09:59:37.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.689+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1394383878 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e7221e67a0 con 0x55e7221f3700 2024-03-14T09:59:37.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.690+0000 7f82cf400700 1 -- 172.21.15.78:0/1394383878 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82dc0056e0 con 0x55e7221f3700 2024-03-14T09:59:37.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.690+0000 7f8303d2cb80 10 client.26786 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:37.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.690+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1394383878 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e721f73be0 con 0x55e7221f3700 2024-03-14T09:59:37.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.690+0000 7f82cf400700 1 -- 172.21.15.78:0/1394383878 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e7221e67a0 con 0x55e7221f3700 2024-03-14T09:59:37.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.691+0000 7f82cf400700 1 -- 172.21.15.78:0/1394383878 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f82dc019c10 con 0x55e7221f3700 2024-03-14T09:59:37.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.691+0000 7f8303d2cb80 10 client.26786 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:37.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.691+0000 7f8303d2cb80 10 client.26786 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:59:37.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.691+0000 7f8303d2cb80 20 client.26786 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:37.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.691+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/1394383878 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e72220d9e0 0x55e72220fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.691+0000 7f8303d2cb80 4 client.26786 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:59:37.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.691+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1394383878 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e72213e780 con 0x55e72220d9e0 2024-03-14T09:59:37.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.691+0000 7f82ece00700 1 --2- 172.21.15.78:0/1394383878 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e72220d9e0 0x55e72220fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.693+0000 7f82ece00700 1 --2- 172.21.15.78:0/1394383878 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e72220d9e0 0x55e72220fe00 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f24e6127a6631699 server_cookie=d640a7d28494a69b in_seq=0 out_seq=0 2024-03-14T09:59:37.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.693+0000 7f82cf400700 10 client.26786 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:37.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.693+0000 7f82cf400700 1 -- 172.21.15.78:0/1394383878 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55e72213e780 con 0x55e72220d9e0 2024-03-14T09:59:37.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.694+0000 7f82cf400700 10 client.26786 handle_command_reply: tid=0 2024-03-14T09:59:37.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.694+0000 7f8303d2cb80 1 client.26786 shutdown 2024-03-14T09:59:37.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.695+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1394383878 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 msgr2=0x55e7221fd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:37.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.695+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/1394383878 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221fd8f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f82dc00b7c0 tx=0x7f82dc0194e0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.695+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1394383878 shutdown_connections 2024-03-14T09:59:37.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.695+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/1394383878 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e72220d9e0 0x55e72220fe00 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.695+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/1394383878 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221fd8f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.695+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/1394383878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 0x55e7221fb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.695+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/1394383878 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 0x55e7221fac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.695+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1394383878 >> 172.21.15.78:0/1394383878 conn(0x55e7221692a0 msgr2=0x55e7221ee710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:37.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.696+0000 7f82cf400700 0 client.26786 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:37.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.696+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1394383878 shutdown_connections 2024-03-14T09:59:37.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.696+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1394383878 wait complete. 2024-03-14T09:59:37.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.696+0000 7f8303d2cb80 20 client.26786 trim_cache size 0 max 16384 2024-03-14T09:59:37.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.722+0000 7f8303d2cb80 1 Processor -- start 2024-03-14T09:59:37.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.722+0000 7f8303d2cb80 1 -- start start 2024-03-14T09:59:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.723+0000 7f8303d2cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221eecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.723+0000 7f8303d2cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 0x55e7221f9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.723+0000 7f8303d2cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 0x55e7221f9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.723+0000 7f8303d2cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7221f9fd0 con 0x55e7221fdf40 2024-03-14T09:59:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.723+0000 7f8303d2cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7221f3e60 con 0x55e7221f3700 2024-03-14T09:59:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.723+0000 7f8303d2cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7221efeb0 con 0x55e7221f4020 2024-03-14T09:59:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.723+0000 7f82ece00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 0x55e7221f9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.723+0000 7f82ed800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221eecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.723+0000 7f82ee200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 0x55e7221f9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.723+0000 7f82ece00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 0x55e7221f9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60356/0 (socket says 172.21.15.78:60356) 2024-03-14T09:59:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.723+0000 7f82ece00700 1 -- 172.21.15.78:0/1532632432 learned_addr learned my addr 172.21.15.78:0/1532632432 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.723+0000 7f82ed800700 1 -- 172.21.15.78:0/1532632432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 msgr2=0x55e7221f9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.723+0000 7f82ed800700 1 --2- 172.21.15.78:0/1532632432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 0x55e7221f9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.723+0000 7f82ed800700 1 -- 172.21.15.78:0/1532632432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 msgr2=0x55e7221f9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.723+0000 7f82ed800700 1 --2- 172.21.15.78:0/1532632432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 0x55e7221f9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.724+0000 7f82ed800700 1 -- 172.21.15.78:0/1532632432 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e72210bf20 con 0x55e7221f3700 2024-03-14T09:59:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.724+0000 7f82ece00700 1 --2- 172.21.15.78:0/1532632432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 0x55e7221f9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.724+0000 7f82ed800700 1 --2- 172.21.15.78:0/1532632432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221eecb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f82dc010e00 tx=0x7f82dc010ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab5dcff28a8ab59 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.724+0000 7f82db400700 1 -- 172.21.15.78:0/1532632432 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc02cd70 con 0x55e7221f3700 2024-03-14T09:59:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.724+0000 7f82db400700 1 -- 172.21.15.78:0/1532632432 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82dc011e40 con 0x55e7221f3700 2024-03-14T09:59:37.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.724+0000 7f82db400700 1 -- 172.21.15.78:0/1532632432 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc005040 con 0x55e7221f3700 2024-03-14T09:59:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.725+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1532632432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 msgr2=0x55e7221eecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.725+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/1532632432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221eecb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f82dc010e00 tx=0x7f82dc010ab0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.725+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1532632432 shutdown_connections 2024-03-14T09:59:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.725+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/1532632432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221f3700 0x55e7221eecb0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.725+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/1532632432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221fdf40 0x55e7221f9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.725+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/1532632432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f4020 0x55e7221f9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.725+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1532632432 >> 172.21.15.78:0/1532632432 conn(0x55e7221692a0 msgr2=0x55e7221f5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.725+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1532632432 shutdown_connections 2024-03-14T09:59:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.725+0000 7f8303d2cb80 1 -- 172.21.15.78:0/1532632432 wait complete. 2024-03-14T09:59:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.725+0000 7f8303d2cb80 1 -- start start 2024-03-14T09:59:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.726+0000 7f8303d2cb80 1 Processor -- start 2024-03-14T09:59:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.726+0000 7f8303d2cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f3700 0x55e7222031b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.726+0000 7f8303d2cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221f4020 0x55e7222036f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.726+0000 7f8303d2cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221fdf40 0x55e722203c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.726+0000 7f8303d2cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72225f4f0 con 0x55e7221f4020 2024-03-14T09:59:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.726+0000 7f8303d2cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72225f630 con 0x55e7221fdf40 2024-03-14T09:59:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.726+0000 7f8303d2cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e72225f770 con 0x55e7221f3700 2024-03-14T09:59:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.726+0000 7f8303d2cb80 10 client.0 _ll_register_callbacks cb 0x7f82d4010f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.726+0000 7f82ed800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f3700 0x55e7222031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.726+0000 7f82ece00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221f4020 0x55e7222036f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.726+0000 7f82ed800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f3700 0x55e7222031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60372/0 (socket says 172.21.15.78:60372) 2024-03-14T09:59:37.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.726+0000 7f82ed800700 1 -- 172.21.15.78:0/3649433343 learned_addr learned my addr 172.21.15.78:0/3649433343 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:37.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.727+0000 7f82ee200700 1 --2- 172.21.15.78:0/3649433343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221fdf40 0x55e722203c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.727+0000 7f82ece00700 1 -- 172.21.15.78:0/3649433343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f3700 msgr2=0x55e7222031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:37.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.727+0000 7f82ece00700 1 --2- 172.21.15.78:0/3649433343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f3700 0x55e7222031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.727+0000 7f82ece00700 1 -- 172.21.15.78:0/3649433343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221fdf40 msgr2=0x55e722203c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:37.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.727+0000 7f82ece00700 1 --2- 172.21.15.78:0/3649433343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221fdf40 0x55e722203c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.727+0000 7f82ece00700 1 -- 172.21.15.78:0/3649433343 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d40b7e70 con 0x55e7221f4020 2024-03-14T09:59:37.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.727+0000 7f82ed800700 1 --2- 172.21.15.78:0/3649433343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f3700 0x55e7222031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:37.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.727+0000 7f82ece00700 1 --2- 172.21.15.78:0/3649433343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221f4020 0x55e7222036f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f82d4000c00 tx=0x7f82d40992e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eaab5521823bd96 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:37.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.727+0000 7f82cf400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T09:59:37.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.727+0000 7f82cf400700 1 -- 172.21.15.78:0/3649433343 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d4099880 con 0x55e7221f4020 2024-03-14T09:59:37.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.727+0000 7f8303d2cb80 1 -- 172.21.15.78:0/3649433343 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f82dc015070 con 0x55e7221f4020 2024-03-14T09:59:37.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.727+0000 7f82cf400700 1 -- 172.21.15.78:0/3649433343 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f82d409d6b0 con 0x55e7221f4020 2024-03-14T09:59:37.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.728+0000 7f82cf400700 1 -- 172.21.15.78:0/3649433343 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82d40a1d00 con 0x55e7221f4020 2024-03-14T09:59:37.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.728+0000 7f8303d2cb80 10 client.16722 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:37.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.728+0000 7f8303d2cb80 1 -- 172.21.15.78:0/3649433343 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e721f73be0 con 0x55e7221f4020 2024-03-14T09:59:37.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.728+0000 7f82cf400700 1 -- 172.21.15.78:0/3649433343 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82dc015070 con 0x55e7221f4020 2024-03-14T09:59:37.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.728+0000 7f82cf400700 1 -- 172.21.15.78:0/3649433343 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f82d4098d70 con 0x55e7221f4020 2024-03-14T09:59:37.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.728+0000 7f8303d2cb80 10 client.16722 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:37.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.728+0000 7f8303d2cb80 10 client.16722 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:59:37.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.728+0000 7f8303d2cb80 20 client.16722 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:37.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.728+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/3649433343 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e722261000 0x55e722263420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:37.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.728+0000 7f8303d2cb80 4 client.16722 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T09:59:37.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.728+0000 7f8303d2cb80 1 -- 172.21.15.78:0/3649433343 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e721f94d70 con 0x55e722261000 2024-03-14T09:59:37.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.729+0000 7f82ed800700 1 --2- 172.21.15.78:0/3649433343 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e722261000 0x55e722263420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:37.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.729+0000 7f82ed800700 1 --2- 172.21.15.78:0/3649433343 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e722261000 0x55e722263420 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e9b95c3f69f04e4 server_cookie=55355901edf91e94 in_seq=0 out_seq=0 2024-03-14T09:59:37.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.729+0000 7f82cf400700 10 client.16722 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:37.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.730+0000 7f82cf400700 1 -- 172.21.15.78:0/3649433343 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e721f94d70 con 0x55e722261000 2024-03-14T09:59:37.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.730+0000 7f82cf400700 10 client.16722 handle_command_reply: tid=0 2024-03-14T09:59:37.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.730+0000 7f8303d2cb80 1 client.16722 shutdown 2024-03-14T09:59:37.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.730+0000 7f8303d2cb80 1 -- 172.21.15.78:0/3649433343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221f4020 msgr2=0x55e7222036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:37.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.730+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/3649433343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221f4020 0x55e7222036f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f82d4000c00 tx=0x7f82d40992e0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.731+0000 7f8303d2cb80 1 -- 172.21.15.78:0/3649433343 shutdown_connections 2024-03-14T09:59:37.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.731+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/3649433343 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e722261000 0x55e722263420 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.731+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/3649433343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e7221fdf40 0x55e722203c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.731+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/3649433343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e7221f3700 0x55e7222031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.731+0000 7f8303d2cb80 1 --2- 172.21.15.78:0/3649433343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7221f4020 0x55e7222036f0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.731+0000 7f8303d2cb80 1 -- 172.21.15.78:0/3649433343 >> 172.21.15.78:0/3649433343 conn(0x55e7221692a0 msgr2=0x55e7221f5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:37.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.731+0000 7f82cf400700 0 client.16722 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:37.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.731+0000 7f8303d2cb80 1 -- 172.21.15.78:0/3649433343 shutdown_connections 2024-03-14T09:59:37.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.731+0000 7f8303d2cb80 1 -- 172.21.15.78:0/3649433343 wait complete. 2024-03-14T09:59:37.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.731+0000 7f8303d2cb80 20 client.16722 trim_cache size 0 max 16384 2024-03-14T09:59:37.735 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:59:37.735 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T09:59:37.735 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "7cf7baeb-1c3a-4f3a-ba57-1642f925ed73", 2024-03-14T09:59:37.735 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T09:59:37.735 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:59:37.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.732+0000 7f82f0200700 1 -- 172.21.15.78:0/3863377971 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f82d00a0c80 msgr2=0x7f82d00a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:37.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.732+0000 7f82f0200700 1 --2- 172.21.15.78:0/3863377971 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f82d00a0c80 0x7f82d00a3140 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009850 tx=0x7f82e00097a0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.732+0000 7f82f0200700 1 -- 172.21.15.78:0/3863377971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82e810f940 msgr2=0x7f82e81a6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:37.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.732+0000 7f82f0200700 1 --2- 172.21.15.78:0/3863377971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82e810f940 0x7f82e81a6b50 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f82e810cf70 tx=0x7f82d400eca0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.732+0000 7f82f0200700 1 -- 172.21.15.78:0/3863377971 shutdown_connections 2024-03-14T09:59:37.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.732+0000 7f82f0200700 1 --2- 172.21.15.78:0/3863377971 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f82d00a0c80 0x7f82d00a3140 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.732+0000 7f82f0200700 1 --2- 172.21.15.78:0/3863377971 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82e810f4d0 0x7f82e81a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.732+0000 7f82f0200700 1 --2- 172.21.15.78:0/3863377971 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f82e811a590 0x7f82e81aaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.732+0000 7f82f0200700 1 --2- 172.21.15.78:0/3863377971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82e810f940 0x7f82e81a6b50 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:37.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.732+0000 7f82f0200700 1 -- 172.21.15.78:0/3863377971 >> 172.21.15.78:0/3863377971 conn(0x7f82e806bdc0 msgr2=0x7f82e8118580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:37.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.732+0000 7f82f0200700 1 -- 172.21.15.78:0/3863377971 shutdown_connections 2024-03-14T09:59:37.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:37.732+0000 7f82f0200700 1 -- 172.21.15.78:0/3863377971 wait complete. 2024-03-14T09:59:37.755 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T09:59:38.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.024+0000 7f3ba7200700 1 -- 172.21.15.78:0/3686776606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ba8073aa0 msgr2=0x7f3ba810cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:38.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.024+0000 7f3ba7200700 1 --2- 172.21.15.78:0/3686776606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ba8073aa0 0x7f3ba810cea0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3ba00098f0 tx=0x7f3ba0009c00 comp rx=0 tx=0).stop 2024-03-14T09:59:38.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.024+0000 7f3ba7200700 1 -- 172.21.15.78:0/3686776606 shutdown_connections 2024-03-14T09:59:38.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.024+0000 7f3ba7200700 1 --2- 172.21.15.78:0/3686776606 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ba8073180 0x7f3ba8073560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.024+0000 7f3ba7200700 1 --2- 172.21.15.78:0/3686776606 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ba810d3e0 0x7f3ba810f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.024+0000 7f3ba7200700 1 --2- 172.21.15.78:0/3686776606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ba8073aa0 0x7f3ba810cea0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.024+0000 7f3ba7200700 1 -- 172.21.15.78:0/3686776606 >> 172.21.15.78:0/3686776606 conn(0x7f3ba80fc650 msgr2=0x7f3ba80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:38.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.024+0000 7f3ba7200700 1 -- 172.21.15.78:0/3686776606 shutdown_connections 2024-03-14T09:59:38.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.025+0000 7f3ba7200700 1 -- 172.21.15.78:0/3686776606 wait complete. 2024-03-14T09:59:38.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.026+0000 7f3ba7200700 1 Processor -- start 2024-03-14T09:59:38.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.026+0000 7f3ba7200700 1 -- start start 2024-03-14T09:59:38.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.026+0000 7f3ba7200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ba8073180 0x7f3ba8199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.026+0000 7f3ba7200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ba8073aa0 0x7f3ba8199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.026+0000 7f3ba7200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ba810d3e0 0x7f3ba819e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.027+0000 7f3ba7200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba819e650 con 0x7f3ba8073180 2024-03-14T09:59:38.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.027+0000 7f3ba7200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba819e7c0 con 0x7f3ba8073aa0 2024-03-14T09:59:38.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.027+0000 7f3ba7200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba819e980 con 0x7f3ba810d3e0 2024-03-14T09:59:38.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.027+0000 7f3ba6800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ba810d3e0 0x7f3ba819e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.027+0000 7f3ba6800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ba810d3e0 0x7f3ba819e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60402/0 (socket says 172.21.15.78:60402) 2024-03-14T09:59:38.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.027+0000 7f3ba6800700 1 -- 172.21.15.78:0/2465431308 learned_addr learned my addr 172.21.15.78:0/2465431308 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:38.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.027+0000 7f3ba5400700 1 --2- 172.21.15.78:0/2465431308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ba8073aa0 0x7f3ba8199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.027+0000 7f3ba6800700 1 -- 172.21.15.78:0/2465431308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ba8073aa0 msgr2=0x7f3ba8199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:38.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.027+0000 7f3ba6800700 1 --2- 172.21.15.78:0/2465431308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ba8073aa0 0x7f3ba8199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.027+0000 7f3ba6800700 1 -- 172.21.15.78:0/2465431308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ba8073180 msgr2=0x7f3ba8199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:38.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.027+0000 7f3ba6800700 1 --2- 172.21.15.78:0/2465431308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ba8073180 0x7f3ba8199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.028+0000 7f3ba6800700 1 -- 172.21.15.78:0/2465431308 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba0009520 con 0x7f3ba810d3e0 2024-03-14T09:59:38.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.028+0000 7f3ba5400700 1 --2- 172.21.15.78:0/2465431308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ba8073aa0 0x7f3ba8199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:38.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.028+0000 7f3ba6800700 1 --2- 172.21.15.78:0/2465431308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ba810d3e0 0x7f3ba819e010 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3b9000e940 tx=0x7f3b9000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:38.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.028+0000 7f3b9ea00700 1 -- 172.21.15.78:0/2465431308 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b90004a30 con 0x7f3ba810d3e0 2024-03-14T09:59:38.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.028+0000 7f3b9ea00700 1 -- 172.21.15.78:0/2465431308 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b9000cc80 con 0x7f3ba810d3e0 2024-03-14T09:59:38.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.028+0000 7f3b9ea00700 1 -- 172.21.15.78:0/2465431308 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b90005530 con 0x7f3ba810d3e0 2024-03-14T09:59:38.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.028+0000 7f3ba7200700 1 -- 172.21.15.78:0/2465431308 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba819ec60 con 0x7f3ba810d3e0 2024-03-14T09:59:38.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.029+0000 7f3ba7200700 1 -- 172.21.15.78:0/2465431308 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba81a64a0 con 0x7f3ba810d3e0 2024-03-14T09:59:38.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.032+0000 7f3b9ea00700 1 -- 172.21.15.78:0/2465431308 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f3b90004b90 con 0x7f3ba810d3e0 2024-03-14T09:59:38.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.034+0000 7f3b9ea00700 1 --2- 172.21.15.78:0/2465431308 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f3b8c0a0d00 0x7f3b8c0a31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.034+0000 7f3b9ea00700 1 -- 172.21.15.78:0/2465431308 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f3b90014070 con 0x7f3ba810d3e0 2024-03-14T09:59:38.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.034+0000 7f3bc0a21b80 1 Processor -- start 2024-03-14T09:59:38.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.035+0000 7f3ba5e00700 1 --2- 172.21.15.78:0/2465431308 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f3b8c0a0d00 0x7f3b8c0a31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.035+0000 7f3bc0a21b80 1 -- start start 2024-03-14T09:59:38.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.035+0000 7f3bc0a21b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7c860 0x5595aae7cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.035+0000 7f3bc0a21b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae7d180 0x5595aae90300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.035+0000 7f3bc0a21b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae90840 0x5595aae92c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.035+0000 7f3bc0a21b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aadd6990 con 0x5595aae7d180 2024-03-14T09:59:38.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.035+0000 7f3bc0a21b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aadd4350 con 0x5595aae7c860 2024-03-14T09:59:38.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.035+0000 7f3bc0a21b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aadcfc40 con 0x5595aae90840 2024-03-14T09:59:38.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.036+0000 7f3ba6800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae90840 0x5595aae92c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.036+0000 7f3ba6800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae90840 0x5595aae92c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60412/0 (socket says 172.21.15.78:60412) 2024-03-14T09:59:38.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.036+0000 7f3ba6800700 1 -- 172.21.15.78:0/1143619490 learned_addr learned my addr 172.21.15.78:0/1143619490 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:38.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.036+0000 7f3ba5400700 1 --2- 172.21.15.78:0/1143619490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae7d180 0x5595aae90300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.036+0000 7f3ba5e00700 1 --2- 172.21.15.78:0/1143619490 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7c860 0x5595aae7cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.036+0000 7f3ba6800700 1 -- 172.21.15.78:0/1143619490 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7c860 msgr2=0x5595aae7cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:38.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.036+0000 7f3ba6800700 1 --2- 172.21.15.78:0/1143619490 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7c860 0x5595aae7cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.036+0000 7f3ba6800700 1 -- 172.21.15.78:0/1143619490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae7d180 msgr2=0x5595aae90300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:38.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.037+0000 7f3ba6800700 1 --2- 172.21.15.78:0/1143619490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae7d180 0x5595aae90300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.037+0000 7f3ba6800700 1 -- 172.21.15.78:0/1143619490 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595aac513b0 con 0x5595aae90840 2024-03-14T09:59:38.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.037+0000 7f3ba6800700 1 --2- 172.21.15.78:0/1143619490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae90840 0x5595aae92c30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3b90095cb0 tx=0x7f3b900993f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5990f879130f2072 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:38.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.037+0000 7f3ba5e00700 1 --2- 172.21.15.78:0/2465431308 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f3b8c0a0d00 0x7f3b8c0a31c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3b98009690 tx=0x7f3b980079a0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:38.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.037+0000 7f3b97400700 1 -- 172.21.15.78:0/1143619490 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b900997b0 con 0x5595aae90840 2024-03-14T09:59:38.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.037+0000 7f3b97400700 1 -- 172.21.15.78:0/1143619490 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b9009d5e0 con 0x5595aae90840 2024-03-14T09:59:38.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.037+0000 7f3b97400700 1 -- 172.21.15.78:0/1143619490 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b900c9e30 con 0x5595aae90840 2024-03-14T09:59:38.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.038+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/1143619490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae90840 msgr2=0x5595aae92c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:38.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.038+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/1143619490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae90840 0x5595aae92c30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3b90095cb0 tx=0x7f3b900993f0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.038+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/1143619490 shutdown_connections 2024-03-14T09:59:38.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.038+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/1143619490 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7c860 0x5595aae7cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.038+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/1143619490 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae90840 0x5595aae92c30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.038+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/1143619490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae7d180 0x5595aae90300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.038+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/1143619490 >> 172.21.15.78:0/1143619490 conn(0x5595aadf3220 msgr2=0x5595aae78c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:38.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.039+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/1143619490 shutdown_connections 2024-03-14T09:59:38.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.039+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/1143619490 wait complete. 2024-03-14T09:59:38.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.039+0000 7f3bc0a21b80 1 -- start start 2024-03-14T09:59:38.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.040+0000 7f3bc0a21b80 1 Processor -- start 2024-03-14T09:59:38.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.040+0000 7f3bc0a21b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7c860 0x5595aae90260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.040+0000 7f3bc0a21b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7d180 0x5595aae8d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.040+0000 7f3bc0a21b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae90840 0x5595aae8dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.040+0000 7f3bc0a21b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aad6bd80 con 0x5595aae90840 2024-03-14T09:59:38.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.040+0000 7f3bc0a21b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aad6ba20 con 0x5595aae7c860 2024-03-14T09:59:38.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.040+0000 7f3bc0a21b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aac536a0 con 0x5595aae7d180 2024-03-14T09:59:38.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.041+0000 7f3bc0a21b80 10 client.0 _ll_register_callbacks cb 0x5595aabd73a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:38.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.041+0000 7f3ba5400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7d180 0x5595aae8d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.041+0000 7f3ba5400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7d180 0x5595aae8d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:60414/0 (socket says 172.21.15.78:60414) 2024-03-14T09:59:38.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.041+0000 7f3ba5400700 1 -- 172.21.15.78:0/4028442571 learned_addr learned my addr 172.21.15.78:0/4028442571 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:38.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.041+0000 7f3ba6800700 1 --2- 172.21.15.78:0/4028442571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae90840 0x5595aae8dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.041+0000 7f3ba5e00700 1 --2- 172.21.15.78:0/4028442571 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7c860 0x5595aae90260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.042+0000 7f3ba5400700 1 -- 172.21.15.78:0/4028442571 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7c860 msgr2=0x5595aae90260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:38.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.042+0000 7f3ba5400700 1 --2- 172.21.15.78:0/4028442571 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7c860 0x5595aae90260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.042+0000 7f3ba5400700 1 -- 172.21.15.78:0/4028442571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae90840 msgr2=0x5595aae8dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:38.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.042+0000 7f3ba5400700 1 --2- 172.21.15.78:0/4028442571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae90840 0x5595aae8dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.042+0000 7f3ba5400700 1 -- 172.21.15.78:0/4028442571 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba81a64a0 con 0x5595aae7d180 2024-03-14T09:59:38.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.042+0000 7f3ba5e00700 1 --2- 172.21.15.78:0/4028442571 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7c860 0x5595aae90260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T09:59:38.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.042+0000 7f3ba5400700 1 --2- 172.21.15.78:0/4028442571 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7d180 0x5595aae8d550 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0009230 tx=0x7f3ba000b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc5990ff671a0a75 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:38.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.042+0000 7f3b8be00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:59:38.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.042+0000 7f3b8be00700 1 -- 172.21.15.78:0/4028442571 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba001c040 con 0x5595aae7d180 2024-03-14T09:59:38.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.042+0000 7f3b8be00700 1 -- 172.21.15.78:0/4028442571 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f3ba0005240 con 0x5595aae7d180 2024-03-14T09:59:38.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.043+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/4028442571 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5595aae70720 con 0x5595aae7d180 2024-03-14T09:59:38.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.043+0000 7f3b8be00700 1 -- 172.21.15.78:0/4028442571 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ba0010ed0 con 0x5595aae7d180 2024-03-14T09:59:38.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.043+0000 7f3bc0a21b80 10 client.26791 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:38.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.043+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/4028442571 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5595aabfdb60 con 0x5595aae7d180 2024-03-14T09:59:38.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.044+0000 7f3b8be00700 1 -- 172.21.15.78:0/4028442571 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5595aae70720 con 0x5595aae7d180 2024-03-14T09:59:38.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.044+0000 7f3b8be00700 1 -- 172.21.15.78:0/4028442571 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3ba0015020 con 0x5595aae7d180 2024-03-14T09:59:38.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.044+0000 7f3bc0a21b80 10 client.26791 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:38.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.044+0000 7f3bc0a21b80 10 client.26791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:59:38.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.044+0000 7f3bc0a21b80 20 client.26791 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:38.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.044+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/4028442571 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5595aaea05a0 0x5595aaea29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.044+0000 7f3bc0a21b80 4 client.26791 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T09:59:38.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.044+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/4028442571 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5595aadc8700 con 0x5595aaea05a0 2024-03-14T09:59:38.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.045+0000 7f3ba5e00700 1 --2- 172.21.15.78:0/4028442571 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5595aaea05a0 0x5595aaea29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.045+0000 7f3ba5e00700 1 --2- 172.21.15.78:0/4028442571 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5595aaea05a0 0x5595aaea29c0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae4f92579ae5d3c8 server_cookie=34456c7ba295471d in_seq=0 out_seq=0 2024-03-14T09:59:38.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.045+0000 7f3b8be00700 10 client.26791 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:38.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.046+0000 7f3b8be00700 1 -- 172.21.15.78:0/4028442571 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5595aadc8700 con 0x5595aaea05a0 2024-03-14T09:59:38.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.046+0000 7f3b8be00700 10 client.26791 handle_command_reply: tid=0 2024-03-14T09:59:38.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.046+0000 7f3bc0a21b80 1 client.26791 shutdown 2024-03-14T09:59:38.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.046+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/4028442571 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7d180 msgr2=0x5595aae8d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:38.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.046+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/4028442571 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7d180 0x5595aae8d550 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0009230 tx=0x7f3ba000b840 comp rx=0 tx=0).stop 2024-03-14T09:59:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.047+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/4028442571 shutdown_connections 2024-03-14T09:59:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.047+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/4028442571 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5595aaea05a0 0x5595aaea29c0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.047+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/4028442571 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7c860 0x5595aae90260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.047+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/4028442571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae90840 0x5595aae8dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.047+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/4028442571 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7d180 0x5595aae8d550 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.047+0000 7f3b8be00700 0 client.26791 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.047+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/4028442571 >> 172.21.15.78:0/4028442571 conn(0x5595aadf3220 msgr2=0x5595aae78880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.047+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/4028442571 shutdown_connections 2024-03-14T09:59:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.047+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/4028442571 wait complete. 2024-03-14T09:59:38.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.047+0000 7f3bc0a21b80 20 client.26791 trim_cache size 0 max 16384 2024-03-14T09:59:38.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.074+0000 7f3bc0a21b80 1 Processor -- start 2024-03-14T09:59:38.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.074+0000 7f3bc0a21b80 1 -- start start 2024-03-14T09:59:38.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.074+0000 7f3bc0a21b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7c860 0x5595aae787c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.074+0000 7f3bc0a21b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7d180 0x5595aae88860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.075+0000 7f3bc0a21b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae90840 0x5595aae88da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.075+0000 7f3bc0a21b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aae7d870 con 0x5595aae90840 2024-03-14T09:59:38.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.075+0000 7f3bc0a21b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aae7a550 con 0x5595aae7d180 2024-03-14T09:59:38.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.075+0000 7f3bc0a21b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aae72d00 con 0x5595aae7c860 2024-03-14T09:59:38.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.075+0000 7f3ba6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae90840 0x5595aae88da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.076+0000 7f3ba5e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7c860 0x5595aae787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.076+0000 7f3ba6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae90840 0x5595aae88da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37566/0 (socket says 172.21.15.78:37566) 2024-03-14T09:59:38.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.076+0000 7f3ba6800700 1 -- 172.21.15.78:0/1535360322 learned_addr learned my addr 172.21.15.78:0/1535360322 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:38.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.076+0000 7f3ba5400700 1 --2- 172.21.15.78:0/1535360322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7d180 0x5595aae88860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.076+0000 7f3ba6800700 1 -- 172.21.15.78:0/1535360322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7c860 msgr2=0x5595aae787c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:38.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.076+0000 7f3ba6800700 1 --2- 172.21.15.78:0/1535360322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7c860 0x5595aae787c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.076+0000 7f3ba6800700 1 -- 172.21.15.78:0/1535360322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7d180 msgr2=0x5595aae88860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:38.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.076+0000 7f3ba6800700 1 --2- 172.21.15.78:0/1535360322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7d180 0x5595aae88860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.076+0000 7f3ba6800700 1 -- 172.21.15.78:0/1535360322 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595aad95ea0 con 0x5595aae90840 2024-03-14T09:59:38.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.076+0000 7f3ba5e00700 1 --2- 172.21.15.78:0/1535360322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7c860 0x5595aae787c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:38.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.076+0000 7f3ba5400700 1 --2- 172.21.15.78:0/1535360322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7d180 0x5595aae88860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T09:59:38.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.076+0000 7f3ba6800700 1 --2- 172.21.15.78:0/1535360322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae90840 0x5595aae88da0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3b90004ee0 tx=0x7f3b900996a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75960555c21a0c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:38.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.077+0000 7f3b97e00700 1 -- 172.21.15.78:0/1535360322 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b900997b0 con 0x5595aae90840 2024-03-14T09:59:38.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.077+0000 7f3b97e00700 1 -- 172.21.15.78:0/1535360322 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b900a1c30 con 0x5595aae90840 2024-03-14T09:59:38.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.077+0000 7f3b97e00700 1 -- 172.21.15.78:0/1535360322 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b900cc410 con 0x5595aae90840 2024-03-14T09:59:38.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.078+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/1535360322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae90840 msgr2=0x5595aae88da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:38.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.078+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/1535360322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae90840 0x5595aae88da0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3b90004ee0 tx=0x7f3b900996a0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.078+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/1535360322 shutdown_connections 2024-03-14T09:59:38.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.078+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/1535360322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae90840 0x5595aae88da0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.078+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/1535360322 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7c860 0x5595aae787c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.078+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/1535360322 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae7d180 0x5595aae88860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.078+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/1535360322 >> 172.21.15.78:0/1535360322 conn(0x5595aadf3220 msgr2=0x5595aae8fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:38.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.078+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/1535360322 shutdown_connections 2024-03-14T09:59:38.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.078+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/1535360322 wait complete. 2024-03-14T09:59:38.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.079+0000 7f3bc0a21b80 1 -- start start 2024-03-14T09:59:38.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.079+0000 7f3bc0a21b80 1 Processor -- start 2024-03-14T09:59:38.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.080+0000 7f3bc0a21b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7c860 0x5595aae96f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.080+0000 7f3bc0a21b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae7d180 0x5595aae974b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.080+0000 7f3bc0a21b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae90840 0x5595aae95590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.080+0000 7f3bc0a21b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aae95ca0 con 0x5595aae7d180 2024-03-14T09:59:38.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.080+0000 7f3bc0a21b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aae95e10 con 0x5595aae90840 2024-03-14T09:59:38.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.080+0000 7f3bc0a21b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595aae95fd0 con 0x5595aae7c860 2024-03-14T09:59:38.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.080+0000 7f3bc0a21b80 10 client.0 _ll_register_callbacks cb 0x7f3ba000f770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T09:59:38.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.081+0000 7f3ba5e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7c860 0x5595aae96f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.081+0000 7f3ba6800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae90840 0x5595aae95590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.081+0000 7f3ba6800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae90840 0x5595aae95590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:44096/0 (socket says 172.21.15.78:44096) 2024-03-14T09:59:38.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.081+0000 7f3ba6800700 1 -- 172.21.15.78:0/3264953394 learned_addr learned my addr 172.21.15.78:0/3264953394 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T09:59:38.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.081+0000 7f3ba5400700 1 --2- 172.21.15.78:0/3264953394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae7d180 0x5595aae974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.081+0000 7f3ba5e00700 1 -- 172.21.15.78:0/3264953394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae90840 msgr2=0x5595aae95590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:38.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.081+0000 7f3ba5e00700 1 --2- 172.21.15.78:0/3264953394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae90840 0x5595aae95590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.081+0000 7f3ba5e00700 1 -- 172.21.15.78:0/3264953394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae7d180 msgr2=0x5595aae974b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T09:59:38.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.081+0000 7f3ba5e00700 1 --2- 172.21.15.78:0/3264953394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae7d180 0x5595aae974b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.081+0000 7f3ba5e00700 1 -- 172.21.15.78:0/3264953394 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9801f040 con 0x5595aae7c860 2024-03-14T09:59:38.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.081+0000 7f3ba6800700 1 --2- 172.21.15.78:0/3264953394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae90840 0x5595aae95590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T09:59:38.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.082+0000 7f3ba5e00700 1 --2- 172.21.15.78:0/3264953394 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7c860 0x5595aae96f70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x5595aaea0980 tx=0x7f3b98017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9af53c3048b2a203 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T09:59:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.082+0000 7f3b8be00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T09:59:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.082+0000 7f3b8be00700 1 -- 172.21.15.78:0/3264953394 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b98025e10 con 0x5595aae7c860 2024-03-14T09:59:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.082+0000 7f3b8be00700 1 -- 172.21.15.78:0/3264953394 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f3b98004bb0 con 0x5595aae7c860 2024-03-14T09:59:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.082+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/3264953394 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3b9009d5e0 con 0x5595aae7c860 2024-03-14T09:59:38.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.083+0000 7f3b8be00700 1 -- 172.21.15.78:0/3264953394 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b98004d70 con 0x5595aae7c860 2024-03-14T09:59:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.083+0000 7f3bc0a21b80 10 client.26797 fetch_fsmap learned FSMap version 25 2024-03-14T09:59:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.083+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/3264953394 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5595aac513b0 con 0x5595aae7c860 2024-03-14T09:59:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.083+0000 7f3b8be00700 1 -- 172.21.15.78:0/3264953394 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b9009d5e0 con 0x5595aae7c860 2024-03-14T09:59:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.083+0000 7f3b8be00700 1 -- 172.21.15.78:0/3264953394 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3b9802e4f0 con 0x5595aae7c860 2024-03-14T09:59:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.084+0000 7f3bc0a21b80 10 client.26797 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T09:59:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.084+0000 7f3bc0a21b80 10 client.26797 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T09:59:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.084+0000 7f3bc0a21b80 20 client.26797 populate_metadata read hostname 'smithi078' 2024-03-14T09:59:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.084+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/3264953394 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5595aaeed320 0x5595aaeed730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T09:59:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.084+0000 7f3bc0a21b80 4 client.26797 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T09:59:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.084+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/3264953394 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5595aac1ecf0 con 0x5595aaeed320 2024-03-14T09:59:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.084+0000 7f3ba5400700 1 --2- 172.21.15.78:0/3264953394 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5595aaeed320 0x5595aaeed730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T09:59:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.084+0000 7f3b8be00700 10 client.26797 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:38.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.084+0000 7f3ba5400700 1 --2- 172.21.15.78:0/3264953394 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5595aaeed320 0x5595aaeed730 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc50a9b0362a3378 server_cookie=d83b2eb64a06528b in_seq=0 out_seq=0 2024-03-14T09:59:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.085+0000 7f3b8be00700 1 -- 172.21.15.78:0/3264953394 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x5595aac1ecf0 con 0x5595aaeed320 2024-03-14T09:59:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.085+0000 7f3b8be00700 10 client.26797 handle_command_reply: tid=0 2024-03-14T09:59:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.085+0000 7f3bc0a21b80 1 client.26797 shutdown 2024-03-14T09:59:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.085+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/3264953394 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7c860 msgr2=0x5595aae96f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.085+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/3264953394 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7c860 0x5595aae96f70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x5595aaea0980 tx=0x7f3b98017000 comp rx=0 tx=0).stop 2024-03-14T09:59:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.086+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/3264953394 shutdown_connections 2024-03-14T09:59:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.086+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/3264953394 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5595aaeed320 0x5595aaeed730 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.086+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/3264953394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5595aae90840 0x5595aae95590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.086+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/3264953394 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5595aae7c860 0x5595aae96f70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.086+0000 7f3bc0a21b80 1 --2- 172.21.15.78:0/3264953394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5595aae7d180 0x5595aae974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.086+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/3264953394 >> 172.21.15.78:0/3264953394 conn(0x5595aadf3220 msgr2=0x5595aae786f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.086+0000 7f3b8be00700 0 client.26797 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T09:59:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.086+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/3264953394 shutdown_connections 2024-03-14T09:59:38.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.086+0000 7f3bc0a21b80 1 -- 172.21.15.78:0/3264953394 wait complete. 2024-03-14T09:59:38.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.086+0000 7f3bc0a21b80 20 client.26797 trim_cache size 0 max 16384 2024-03-14T09:59:38.089 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T09:59:38.089 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "scrub active (1220 inodes in the stack)", 2024-03-14T09:59:38.089 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T09:59:38.089 INFO:teuthology.orchestra.run.smithi078.stdout: "7cf7baeb-1c3a-4f3a-ba57-1642f925ed73": { 2024-03-14T09:59:38.089 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T09:59:38.089 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "7cf7baeb-1c3a-4f3a-ba57-1642f925ed73", 2024-03-14T09:59:38.089 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T09:59:38.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:59:38.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T09:59:38.089 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T09:59:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.087+0000 7f3ba7200700 1 -- 172.21.15.78:0/2465431308 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f3b8c0a0d00 msgr2=0x7f3b8c0a31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.087+0000 7f3ba7200700 1 --2- 172.21.15.78:0/2465431308 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f3b8c0a0d00 0x7f3b8c0a31c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3b98009690 tx=0x7f3b980079a0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.087+0000 7f3ba7200700 1 -- 172.21.15.78:0/2465431308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ba810d3e0 msgr2=0x7f3ba819e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T09:59:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.087+0000 7f3ba7200700 1 --2- 172.21.15.78:0/2465431308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ba810d3e0 0x7f3ba819e010 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3b9000e940 tx=0x7f3b9000ec50 comp rx=0 tx=0).stop 2024-03-14T09:59:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.087+0000 7f3ba7200700 1 -- 172.21.15.78:0/2465431308 shutdown_connections 2024-03-14T09:59:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.087+0000 7f3ba7200700 1 --2- 172.21.15.78:0/2465431308 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3ba810d3e0 0x7f3ba819e010 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.087+0000 7f3ba7200700 1 --2- 172.21.15.78:0/2465431308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ba8073180 0x7f3ba8199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.087+0000 7f3ba7200700 1 --2- 172.21.15.78:0/2465431308 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f3b8c0a0d00 0x7f3b8c0a31c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.087+0000 7f3ba7200700 1 --2- 172.21.15.78:0/2465431308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ba8073aa0 0x7f3ba8199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T09:59:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.087+0000 7f3ba7200700 1 -- 172.21.15.78:0/2465431308 >> 172.21.15.78:0/2465431308 conn(0x7f3ba80fc650 msgr2=0x7f3ba80fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T09:59:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.087+0000 7f3ba7200700 1 -- 172.21.15.78:0/2465431308 shutdown_connections 2024-03-14T09:59:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T09:59:38.087+0000 7f3ba7200700 1 -- 172.21.15.78:0/2465431308 wait complete. 2024-03-14T09:59:38.108 INFO:tasks.cephfs.filesystem:scrub status for tag:7cf7baeb-1c3a-4f3a-ba57-1642f925ed73 - {'path': '/', 'tag': '7cf7baeb-1c3a-4f3a-ba57-1642f925ed73', 'options': 'recursive,force'} 2024-03-14T09:59:39.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:38 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T09:59:39.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:38 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:59:39.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:38 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T09:59:39.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:38 smithi152 ceph-mon[48279]: pgmap v167: 129 pgs: 129 active+clean; 210 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.1 MiB/s wr, 372 op/s 2024-03-14T09:59:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:38 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T09:59:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:38 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:59:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:38 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T09:59:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:38 smithi132 ceph-mon[47959]: pgmap v167: 129 pgs: 129 active+clean; 210 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.1 MiB/s wr, 372 op/s 2024-03-14T09:59:39.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:38 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T09:59:39.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:38 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:59:39.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:38 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T09:59:39.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:38 smithi078 ceph-mon[38143]: pgmap v167: 129 pgs: 129 active+clean; 210 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.1 MiB/s wr, 372 op/s 2024-03-14T09:59:40.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:39 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T09:59:40.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:39 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T09:59:40.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:39 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T09:59:40.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:39 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T09:59:40.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:39 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T09:59:40.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:39 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T09:59:41.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:40 smithi152 ceph-mon[48279]: pgmap v168: 129 pgs: 129 active+clean; 210 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.1 MiB/s wr, 372 op/s 2024-03-14T09:59:41.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:40 smithi132 ceph-mon[47959]: pgmap v168: 129 pgs: 129 active+clean; 210 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.1 MiB/s wr, 372 op/s 2024-03-14T09:59:41.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:40 smithi078 ceph-mon[38143]: pgmap v168: 129 pgs: 129 active+clean; 210 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 1.1 MiB/s wr, 372 op/s 2024-03-14T09:59:43.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:43 smithi152 ceph-mon[48279]: pgmap v169: 129 pgs: 129 active+clean; 235 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 541 op/s 2024-03-14T09:59:43.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:43 smithi132 ceph-mon[47959]: pgmap v169: 129 pgs: 129 active+clean; 235 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 541 op/s 2024-03-14T09:59:43.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:43 smithi078 ceph-mon[38143]: pgmap v169: 129 pgs: 129 active+clean; 235 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 541 op/s 2024-03-14T09:59:45.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:45 smithi152 ceph-mon[48279]: pgmap v170: 129 pgs: 129 active+clean; 235 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 1.1 MiB/s wr, 353 op/s 2024-03-14T09:59:45.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:45 smithi132 ceph-mon[47959]: pgmap v170: 129 pgs: 129 active+clean; 235 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 1.1 MiB/s wr, 353 op/s 2024-03-14T09:59:45.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:45 smithi078 ceph-mon[38143]: pgmap v170: 129 pgs: 129 active+clean; 235 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 1.1 MiB/s wr, 353 op/s 2024-03-14T09:59:47.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:47 smithi152 ceph-mon[48279]: pgmap v171: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 542 op/s 2024-03-14T09:59:47.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:47 smithi132 ceph-mon[47959]: pgmap v171: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 542 op/s 2024-03-14T09:59:47.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:47 smithi078 ceph-mon[38143]: pgmap v171: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 542 op/s 2024-03-14T09:59:49.248 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:48 smithi152 ceph-mon[48279]: pgmap v172: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.1 MiB/s wr, 358 op/s 2024-03-14T09:59:49.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:48 smithi132 ceph-mon[47959]: pgmap v172: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.1 MiB/s wr, 358 op/s 2024-03-14T09:59:49.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:48 smithi078 ceph-mon[38143]: pgmap v172: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.1 MiB/s wr, 358 op/s 2024-03-14T09:59:51.473 INFO:tasks.workunit.client.0.smithi078.stdout:30000 mapread 0x11b12cb thru 0x11b885f (0x7595 bytes) 2024-03-14T09:59:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:51 smithi152 ceph-mon[48279]: pgmap v173: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.1 MiB/s wr, 358 op/s 2024-03-14T09:59:51.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:51 smithi132 ceph-mon[47959]: pgmap v173: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.1 MiB/s wr, 358 op/s 2024-03-14T09:59:51.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:51 smithi078 ceph-mon[38143]: pgmap v173: 129 pgs: 129 active+clean; 219 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.1 MiB/s wr, 358 op/s 2024-03-14T09:59:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:53 smithi152 ceph-mon[48279]: pgmap v174: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 543 op/s 2024-03-14T09:59:53.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:53 smithi132 ceph-mon[47959]: pgmap v174: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 543 op/s 2024-03-14T09:59:53.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:53 smithi078 ceph-mon[38143]: pgmap v174: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 543 op/s 2024-03-14T09:59:55.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:55 smithi152 ceph-mon[48279]: pgmap v175: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 1.1 MiB/s wr, 373 op/s 2024-03-14T09:59:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:55 smithi132 ceph-mon[47959]: pgmap v175: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 1.1 MiB/s wr, 373 op/s 2024-03-14T09:59:55.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:55 smithi078 ceph-mon[38143]: pgmap v175: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 1.1 MiB/s wr, 373 op/s 2024-03-14T09:59:57.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:57 smithi152 ceph-mon[48279]: pgmap v176: 129 pgs: 129 active+clean; 219 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 1.7 MiB/s wr, 542 op/s 2024-03-14T09:59:57.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:57 smithi132 ceph-mon[47959]: pgmap v176: 129 pgs: 129 active+clean; 219 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 1.7 MiB/s wr, 542 op/s 2024-03-14T09:59:57.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:57 smithi078 ceph-mon[38143]: pgmap v176: 129 pgs: 129 active+clean; 219 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 1.7 MiB/s wr, 542 op/s 2024-03-14T09:59:59.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 09:59:58 smithi152 ceph-mon[48279]: pgmap v177: 129 pgs: 129 active+clean; 219 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.2 MiB/s wr, 353 op/s 2024-03-14T09:59:59.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 09:59:58 smithi132 ceph-mon[47959]: pgmap v177: 129 pgs: 129 active+clean; 219 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.2 MiB/s wr, 353 op/s 2024-03-14T09:59:59.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 09:59:58 smithi078 ceph-mon[38143]: pgmap v177: 129 pgs: 129 active+clean; 219 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.2 MiB/s wr, 353 op/s 2024-03-14T10:00:00.307 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:00 smithi078 ceph-mon[38143]: overall HEALTH_OK 2024-03-14T10:00:00.498 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:00 smithi152 ceph-mon[48279]: overall HEALTH_OK 2024-03-14T10:00:00.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:00 smithi132 ceph-mon[47959]: overall HEALTH_OK 2024-03-14T10:00:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:01 smithi152 ceph-mon[48279]: pgmap v178: 129 pgs: 129 active+clean; 219 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.2 MiB/s wr, 353 op/s 2024-03-14T10:00:01.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:01 smithi132 ceph-mon[47959]: pgmap v178: 129 pgs: 129 active+clean; 219 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.2 MiB/s wr, 353 op/s 2024-03-14T10:00:01.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:01 smithi078 ceph-mon[38143]: pgmap v178: 129 pgs: 129 active+clean; 219 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.2 MiB/s wr, 353 op/s 2024-03-14T10:00:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:03 smithi152 ceph-mon[48279]: pgmap v179: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 1.7 MiB/s wr, 550 op/s 2024-03-14T10:00:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:03 smithi132 ceph-mon[47959]: pgmap v179: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 1.7 MiB/s wr, 550 op/s 2024-03-14T10:00:03.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:03 smithi078 ceph-mon[38143]: pgmap v179: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 1.7 MiB/s wr, 550 op/s 2024-03-14T10:00:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:05 smithi152 ceph-mon[48279]: pgmap v180: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.1 MiB/s wr, 365 op/s 2024-03-14T10:00:05.523 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:05 smithi132 ceph-mon[47959]: pgmap v180: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.1 MiB/s wr, 365 op/s 2024-03-14T10:00:05.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:05 smithi078 ceph-mon[38143]: pgmap v180: 129 pgs: 129 active+clean; 230 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.1 MiB/s wr, 365 op/s 2024-03-14T10:00:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:07 smithi152 ceph-mon[48279]: pgmap v181: 129 pgs: 129 active+clean; 240 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 1.7 MiB/s wr, 549 op/s 2024-03-14T10:00:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:07 smithi132 ceph-mon[47959]: pgmap v181: 129 pgs: 129 active+clean; 240 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 1.7 MiB/s wr, 549 op/s 2024-03-14T10:00:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:07 smithi078 ceph-mon[38143]: pgmap v181: 129 pgs: 129 active+clean; 240 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 1.7 MiB/s wr, 549 op/s 2024-03-14T10:00:08.109 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:00:08.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.357+0000 7f6c4c000700 1 -- 172.21.15.78:0/1574317539 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c44100f30 msgr2=0x7f6c44101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:08.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.357+0000 7f6c4c000700 1 --2- 172.21.15.78:0/1574317539 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c44100f30 0x7f6c44101310 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6c38009990 tx=0x7f6c38009ca0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.357+0000 7f6c4c000700 1 -- 172.21.15.78:0/1574317539 shutdown_connections 2024-03-14T10:00:08.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.357+0000 7f6c4c000700 1 --2- 172.21.15.78:0/1574317539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c44106050 0x7f6c4410dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.357+0000 7f6c4c000700 1 --2- 172.21.15.78:0/1574317539 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c44100f30 0x7f6c44101310 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.357+0000 7f6c4c000700 1 --2- 172.21.15.78:0/1574317539 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c441018e0 0x7f6c44105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.357+0000 7f6c4c000700 1 -- 172.21.15.78:0/1574317539 >> 172.21.15.78:0/1574317539 conn(0x7f6c440fc7b0 msgr2=0x7f6c440febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:08.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.358+0000 7f6c4c000700 1 -- 172.21.15.78:0/1574317539 shutdown_connections 2024-03-14T10:00:08.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.358+0000 7f6c4c000700 1 -- 172.21.15.78:0/1574317539 wait complete. 2024-03-14T10:00:08.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.359+0000 7f6c4c000700 1 Processor -- start 2024-03-14T10:00:08.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.359+0000 7f6c4c000700 1 -- start start 2024-03-14T10:00:08.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.360+0000 7f6c4c000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c44106050 0x7f6c4410d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.360+0000 7f6c4c000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c4410dca0 0x7f6c4410a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.360+0000 7f6c4c000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c4410adc0 0x7f6c4410b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.360+0000 7f6c4c000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c441aa7e0 con 0x7f6c44106050 2024-03-14T10:00:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.360+0000 7f6c4c000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c441aa950 con 0x7f6c4410adc0 2024-03-14T10:00:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.360+0000 7f6c4c000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c441aab10 con 0x7f6c4410dca0 2024-03-14T10:00:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.360+0000 7f6c49600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c44106050 0x7f6c4410d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.360+0000 7f6c4a000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c4410adc0 0x7f6c4410b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.360+0000 7f6c4a000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c4410adc0 0x7f6c4410b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57300/0 (socket says 172.21.15.78:57300) 2024-03-14T10:00:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.360+0000 7f6c4a000700 1 -- 172.21.15.78:0/3682746083 learned_addr learned my addr 172.21.15.78:0/3682746083 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.360+0000 7f6c48c00700 1 --2- 172.21.15.78:0/3682746083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c4410dca0 0x7f6c4410a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.361+0000 7f6c4a000700 1 -- 172.21.15.78:0/3682746083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c4410dca0 msgr2=0x7f6c4410a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.361+0000 7f6c4a000700 1 --2- 172.21.15.78:0/3682746083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c4410dca0 0x7f6c4410a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.361+0000 7f6c4a000700 1 -- 172.21.15.78:0/3682746083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c44106050 msgr2=0x7f6c4410d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.361+0000 7f6c4a000700 1 --2- 172.21.15.78:0/3682746083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c44106050 0x7f6c4410d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.361+0000 7f6c4a000700 1 -- 172.21.15.78:0/3682746083 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c009c40 con 0x7f6c4410adc0 2024-03-14T10:00:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.361+0000 7f6c49600700 1 --2- 172.21.15.78:0/3682746083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c44106050 0x7f6c4410d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:08.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.361+0000 7f6c4a000700 1 --2- 172.21.15.78:0/3682746083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c4410adc0 0x7f6c4410b270 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00c370 tx=0x7f6c3c00c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:08.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.361+0000 7f6c42000700 1 -- 172.21.15.78:0/3682746083 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c017070 con 0x7f6c4410adc0 2024-03-14T10:00:08.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.361+0000 7f6c42000700 1 -- 172.21.15.78:0/3682746083 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c3c00f040 con 0x7f6c4410adc0 2024-03-14T10:00:08.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.361+0000 7f6c42000700 1 -- 172.21.15.78:0/3682746083 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c00e050 con 0x7f6c4410adc0 2024-03-14T10:00:08.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.361+0000 7f6c4c000700 1 -- 172.21.15.78:0/3682746083 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c380095f0 con 0x7f6c4410adc0 2024-03-14T10:00:08.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.362+0000 7f6c4c000700 1 -- 172.21.15.78:0/3682746083 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c441ab110 con 0x7f6c4410adc0 2024-03-14T10:00:08.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.365+0000 7f6c42000700 1 -- 172.21.15.78:0/3682746083 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f6c3c004520 con 0x7f6c4410adc0 2024-03-14T10:00:08.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.367+0000 7f6c5fab9b80 1 Processor -- start 2024-03-14T10:00:08.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.367+0000 7f6c5fab9b80 1 -- start start 2024-03-14T10:00:08.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.368+0000 7f6c5fab9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad08b680 0x55c6ad08ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.368+0000 7f6c5fab9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08bfa0 0x55c6ad095980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.368+0000 7f6c5fab9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad095ec0 0x55c6ad0982b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.368+0000 7f6c5fab9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6acfe4990 con 0x55c6ad08b680 2024-03-14T10:00:08.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.368+0000 7f6c5fab9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6acfe2350 con 0x55c6ad095ec0 2024-03-14T10:00:08.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.368+0000 7f6c5fab9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6acfddc40 con 0x55c6ad08bfa0 2024-03-14T10:00:08.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.368+0000 7f6c42000700 1 --2- 172.21.15.78:0/3682746083 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6c2c0a0c80 0x7f6c2c0a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.368+0000 7f6c49600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08bfa0 0x55c6ad095980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.368+0000 7f6c48c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad095ec0 0x55c6ad0982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.368+0000 7f6c4a000700 1 --2- 172.21.15.78:0/3682746083 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6c2c0a0c80 0x7f6c2c0a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.368+0000 7f6c42000700 1 -- 172.21.15.78:0/3682746083 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f6c3c0cc070 con 0x7f6c4410adc0 2024-03-14T10:00:08.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.369+0000 7f6c48c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad095ec0 0x55c6ad0982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57304/0 (socket says 172.21.15.78:57304) 2024-03-14T10:00:08.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.369+0000 7f6c48c00700 1 -- 172.21.15.78:0/1791479542 learned_addr learned my addr 172.21.15.78:0/1791479542 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:08.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.369+0000 7f6c48c00700 1 --2- 172.21.15.78:0/1791479542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad08b680 0x55c6ad08ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.369+0000 7f6c49600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08bfa0 0x55c6ad095980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35624/0 (socket says 172.21.15.78:35624) 2024-03-14T10:00:08.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.369+0000 7f6c48c00700 1 -- 172.21.15.78:0/1791479542 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08bfa0 msgr2=0x55c6ad095980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:08.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.369+0000 7f6c48c00700 1 --2- 172.21.15.78:0/1791479542 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08bfa0 0x55c6ad095980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.369+0000 7f6c48c00700 1 -- 172.21.15.78:0/1791479542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad08b680 msgr2=0x55c6ad08ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:08.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.369+0000 7f6c48c00700 1 --2- 172.21.15.78:0/1791479542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad08b680 0x55c6ad08ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.369+0000 7f6c48c00700 1 -- 172.21.15.78:0/1791479542 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6ace5f3b0 con 0x55c6ad095ec0 2024-03-14T10:00:08.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.369+0000 7f6c48c00700 1 --2- 172.21.15.78:0/1791479542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad095ec0 0x55c6ad0982b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6c300046e0 tx=0x7f6c30009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55e992155b469970 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:08.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.369+0000 7f6c4a000700 1 --2- 172.21.15.78:0/3682746083 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6c2c0a0c80 0x7f6c2c0a3140 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c017fd0 tx=0x7f6c3c00cf90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:08.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.369+0000 7f6c36600700 1 -- 172.21.15.78:0/1791479542 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c30012de0 con 0x55c6ad095ec0 2024-03-14T10:00:08.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.370+0000 7f6c36600700 1 -- 172.21.15.78:0/1791479542 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c3000f910 con 0x55c6ad095ec0 2024-03-14T10:00:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.370+0000 7f6c36600700 1 -- 172.21.15.78:0/1791479542 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c30012de0 con 0x55c6ad095ec0 2024-03-14T10:00:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.370+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/1791479542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad095ec0 msgr2=0x55c6ad0982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.370+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/1791479542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad095ec0 0x55c6ad0982b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6c300046e0 tx=0x7f6c30009750 comp rx=0 tx=0).stop 2024-03-14T10:00:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.370+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/1791479542 shutdown_connections 2024-03-14T10:00:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.370+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/1791479542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad095ec0 0x55c6ad0982b0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.370+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/1791479542 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad08b680 0x55c6ad08ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.370+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/1791479542 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08bfa0 0x55c6ad095980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.370+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/1791479542 >> 172.21.15.78:0/1791479542 conn(0x55c6ad001220 msgr2=0x55c6ad086aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.371+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/1791479542 shutdown_connections 2024-03-14T10:00:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.371+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/1791479542 wait complete. 2024-03-14T10:00:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.371+0000 7f6c5fab9b80 1 -- start start 2024-03-14T10:00:08.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.372+0000 7f6c5fab9b80 1 Processor -- start 2024-03-14T10:00:08.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.373+0000 7f6c5fab9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08b680 0x55c6ad0958f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.373+0000 7f6c5fab9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad08bfa0 0x55c6ad092c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.373+0000 7f6c5fab9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad095ec0 0x55c6ad0933c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.373+0000 7f6c5fab9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6acf79d80 con 0x55c6ad08bfa0 2024-03-14T10:00:08.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.373+0000 7f6c5fab9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6acf79a20 con 0x55c6ad095ec0 2024-03-14T10:00:08.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.373+0000 7f6c5fab9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6ace616a0 con 0x55c6ad08b680 2024-03-14T10:00:08.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.373+0000 7f6c5fab9b80 10 client.0 _ll_register_callbacks cb 0x55c6acde53a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:08.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.373+0000 7f6c48c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08b680 0x55c6ad0958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.373+0000 7f6c48c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08b680 0x55c6ad0958f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35626/0 (socket says 172.21.15.78:35626) 2024-03-14T10:00:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.373+0000 7f6c48c00700 1 -- 172.21.15.78:0/2731394323 learned_addr learned my addr 172.21.15.78:0/2731394323 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.373+0000 7f6c48c00700 1 --2- 172.21.15.78:0/2731394323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad095ec0 0x55c6ad0933c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.373+0000 7f6c49600700 1 --2- 172.21.15.78:0/2731394323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad08bfa0 0x55c6ad092c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.374+0000 7f6c48c00700 1 -- 172.21.15.78:0/2731394323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad095ec0 msgr2=0x55c6ad0933c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.374+0000 7f6c48c00700 1 --2- 172.21.15.78:0/2731394323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad095ec0 0x55c6ad0933c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.374+0000 7f6c48c00700 1 -- 172.21.15.78:0/2731394323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad08bfa0 msgr2=0x55c6ad092c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.374+0000 7f6c48c00700 1 --2- 172.21.15.78:0/2731394323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad08bfa0 0x55c6ad092c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.374+0000 7f6c48c00700 1 -- 172.21.15.78:0/2731394323 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3001a040 con 0x55c6ad08b680 2024-03-14T10:00:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.374+0000 7f6c48c00700 1 --2- 172.21.15.78:0/2731394323 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08b680 0x55c6ad0958f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6c300096f0 tx=0x7f6c3000ff10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ceef0fb62631d00 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.374+0000 7f6c2aa00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:00:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.374+0000 7f6c2aa00700 1 -- 172.21.15.78:0/2731394323 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c30005260 con 0x55c6ad08b680 2024-03-14T10:00:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.374+0000 7f6c2aa00700 1 -- 172.21.15.78:0/2731394323 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f6c300053c0 con 0x55c6ad08b680 2024-03-14T10:00:08.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.374+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/2731394323 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c6ad07e720 con 0x55c6ad08b680 2024-03-14T10:00:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.378+0000 7f6c2aa00700 1 -- 172.21.15.78:0/2731394323 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c30005580 con 0x55c6ad08b680 2024-03-14T10:00:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.378+0000 7f6c5fab9b80 10 client.26809 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.378+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/2731394323 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c6ace0bb60 con 0x55c6ad08b680 2024-03-14T10:00:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.379+0000 7f6c2aa00700 1 -- 172.21.15.78:0/2731394323 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c6ad07e720 con 0x55c6ad08b680 2024-03-14T10:00:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.379+0000 7f6c2aa00700 1 -- 172.21.15.78:0/2731394323 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6c3001f410 con 0x55c6ad08b680 2024-03-14T10:00:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.379+0000 7f6c5fab9b80 10 client.26809 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.379+0000 7f6c5fab9b80 10 client.26809 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:00:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.380+0000 7f6c5fab9b80 20 client.26809 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.380+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/2731394323 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c6ad0a5910 0x55c6ad0a7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.380+0000 7f6c5fab9b80 4 client.26809 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:00:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.380+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/2731394323 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c6acfd6700 con 0x55c6ad0a5910 2024-03-14T10:00:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.380+0000 7f6c49600700 1 --2- 172.21.15.78:0/2731394323 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c6ad0a5910 0x55c6ad0a7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.381+0000 7f6c49600700 1 --2- 172.21.15.78:0/2731394323 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c6ad0a5910 0x55c6ad0a7d00 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a18cd6d944a9d8a0 server_cookie=a8fbc2932d5a3204 in_seq=0 out_seq=0 2024-03-14T10:00:08.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.381+0000 7f6c2aa00700 10 client.26809 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:08.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.383+0000 7f6c2aa00700 1 -- 172.21.15.78:0/2731394323 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55c6acfd6700 con 0x55c6ad0a5910 2024-03-14T10:00:08.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.383+0000 7f6c2aa00700 10 client.26809 handle_command_reply: tid=0 2024-03-14T10:00:08.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.383+0000 7f6c5fab9b80 1 client.26809 shutdown 2024-03-14T10:00:08.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.384+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/2731394323 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08b680 msgr2=0x55c6ad0958f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:08.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.384+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/2731394323 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08b680 0x55c6ad0958f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6c300096f0 tx=0x7f6c3000ff10 comp rx=0 tx=0).stop 2024-03-14T10:00:08.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.384+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/2731394323 shutdown_connections 2024-03-14T10:00:08.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.384+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/2731394323 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c6ad0a5910 0x55c6ad0a7d00 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.384+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/2731394323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad095ec0 0x55c6ad0933c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.384+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/2731394323 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08b680 0x55c6ad0958f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.384+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/2731394323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad08bfa0 0x55c6ad092c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.384+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/2731394323 >> 172.21.15.78:0/2731394323 conn(0x55c6ad001220 msgr2=0x55c6ad086690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.384+0000 7f6c2aa00700 0 client.26809 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.384+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/2731394323 shutdown_connections 2024-03-14T10:00:08.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.385+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/2731394323 wait complete. 2024-03-14T10:00:08.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.385+0000 7f6c5fab9b80 20 client.26809 trim_cache size 0 max 16384 2024-03-14T10:00:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.411+0000 7f6c5fab9b80 1 Processor -- start 2024-03-14T10:00:08.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.411+0000 7f6c5fab9b80 1 -- start start 2024-03-14T10:00:08.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.412+0000 7f6c5fab9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad08b680 0x55c6ad092c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.412+0000 7f6c5fab9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08bfa0 0x55c6ad09cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.412+0000 7f6c5fab9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad095ec0 0x55c6ad095420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.412+0000 7f6c5fab9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6ad08d790 con 0x55c6ad095ec0 2024-03-14T10:00:08.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.412+0000 7f6c5fab9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6ad08caf0 con 0x55c6ad08b680 2024-03-14T10:00:08.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.412+0000 7f6c5fab9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6ad08dba0 con 0x55c6ad08bfa0 2024-03-14T10:00:08.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.412+0000 7f6c48c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad08b680 0x55c6ad092c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.412+0000 7f6c48c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad08b680 0x55c6ad092c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57318/0 (socket says 172.21.15.78:57318) 2024-03-14T10:00:08.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.412+0000 7f6c48c00700 1 -- 172.21.15.78:0/2540177675 learned_addr learned my addr 172.21.15.78:0/2540177675 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:08.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.412+0000 7f6c48c00700 1 --2- 172.21.15.78:0/2540177675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad095ec0 0x55c6ad095420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.412+0000 7f6c49600700 1 --2- 172.21.15.78:0/2540177675 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08bfa0 0x55c6ad09cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.413+0000 7f6c48c00700 1 -- 172.21.15.78:0/2540177675 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08bfa0 msgr2=0x55c6ad09cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:08.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.413+0000 7f6c48c00700 1 --2- 172.21.15.78:0/2540177675 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08bfa0 0x55c6ad09cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.413+0000 7f6c48c00700 1 -- 172.21.15.78:0/2540177675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad095ec0 msgr2=0x55c6ad095420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:08.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.413+0000 7f6c48c00700 1 --2- 172.21.15.78:0/2540177675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad095ec0 0x55c6ad095420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.413+0000 7f6c48c00700 1 -- 172.21.15.78:0/2540177675 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6acfa3ea0 con 0x55c6ad08b680 2024-03-14T10:00:08.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.413+0000 7f6c49600700 1 --2- 172.21.15.78:0/2540177675 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08bfa0 0x55c6ad09cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:00:08.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.413+0000 7f6c48c00700 1 --2- 172.21.15.78:0/2540177675 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad08b680 0x55c6ad092c20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6c3000fee0 tx=0x7f6c30000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccd93b95e04c5df9 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:08.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.413+0000 7f6c37000700 1 -- 172.21.15.78:0/2540177675 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3001fa60 con 0x55c6ad08b680 2024-03-14T10:00:08.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.413+0000 7f6c37000700 1 -- 172.21.15.78:0/2540177675 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c30005c40 con 0x55c6ad08b680 2024-03-14T10:00:08.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.413+0000 7f6c37000700 1 -- 172.21.15.78:0/2540177675 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3001f410 con 0x55c6ad08b680 2024-03-14T10:00:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.414+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/2540177675 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad08b680 msgr2=0x55c6ad092c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.414+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/2540177675 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad08b680 0x55c6ad092c20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6c3000fee0 tx=0x7f6c30000bc0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.414+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/2540177675 shutdown_connections 2024-03-14T10:00:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.414+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/2540177675 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad08b680 0x55c6ad092c20 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.414+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/2540177675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad095ec0 0x55c6ad095420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.414+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/2540177675 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad08bfa0 0x55c6ad09cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.414+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/2540177675 >> 172.21.15.78:0/2540177675 conn(0x55c6ad001220 msgr2=0x55c6ad092510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.414+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/2540177675 shutdown_connections 2024-03-14T10:00:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.414+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/2540177675 wait complete. 2024-03-14T10:00:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.415+0000 7f6c5fab9b80 1 -- start start 2024-03-14T10:00:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.415+0000 7f6c5fab9b80 1 Processor -- start 2024-03-14T10:00:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.415+0000 7f6c5fab9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad08b680 0x55c6ad09c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.415+0000 7f6c5fab9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad08bfa0 0x55c6ad09ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.415+0000 7f6c5fab9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad095ec0 0x55c6ad09b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.415+0000 7f6c5fab9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6ad09b700 con 0x55c6ad08bfa0 2024-03-14T10:00:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.415+0000 7f6c5fab9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6ad09b870 con 0x55c6ad08b680 2024-03-14T10:00:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.415+0000 7f6c5fab9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6ad09ba30 con 0x55c6ad095ec0 2024-03-14T10:00:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.415+0000 7f6c5fab9b80 10 client.0 _ll_register_callbacks cb 0x7f6c300126f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.416+0000 7f6c48c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad08b680 0x55c6ad09c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.416+0000 7f6c49600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad08bfa0 0x55c6ad09ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.416+0000 7f6c48c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad08b680 0x55c6ad09c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57320/0 (socket says 172.21.15.78:57320) 2024-03-14T10:00:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.416+0000 7f6c48c00700 1 -- 172.21.15.78:0/4235977835 learned_addr learned my addr 172.21.15.78:0/4235977835 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.416+0000 7f6c48c00700 1 --2- 172.21.15.78:0/4235977835 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad095ec0 0x55c6ad09b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.416+0000 7f6c48c00700 1 -- 172.21.15.78:0/4235977835 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad095ec0 msgr2=0x55c6ad09b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.416+0000 7f6c48c00700 1 --2- 172.21.15.78:0/4235977835 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad095ec0 0x55c6ad09b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.416+0000 7f6c48c00700 1 -- 172.21.15.78:0/4235977835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad08bfa0 msgr2=0x55c6ad09ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.416+0000 7f6c48c00700 1 --2- 172.21.15.78:0/4235977835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad08bfa0 0x55c6ad09ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.416+0000 7f6c48c00700 1 -- 172.21.15.78:0/4235977835 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c6ace0bb60 con 0x55c6ad08b680 2024-03-14T10:00:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.416+0000 7f6c49600700 1 --2- 172.21.15.78:0/4235977835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad08bfa0 0x55c6ad09ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.416+0000 7f6c48c00700 1 --2- 172.21.15.78:0/4235977835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad08b680 0x55c6ad09c7e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6c3002d5e0 tx=0x7f6c3000f820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=701432a53bc2cde7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.416+0000 7f6c2aa00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T10:00:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.416+0000 7f6c2aa00700 1 -- 172.21.15.78:0/4235977835 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3001f410 con 0x55c6ad08b680 2024-03-14T10:00:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.417+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/4235977835 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6c3000f910 con 0x55c6ad08b680 2024-03-14T10:00:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.417+0000 7f6c2aa00700 1 -- 172.21.15.78:0/4235977835 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f6c3001f570 con 0x55c6ad08b680 2024-03-14T10:00:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.417+0000 7f6c2aa00700 1 -- 172.21.15.78:0/4235977835 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c30005260 con 0x55c6ad08b680 2024-03-14T10:00:08.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.417+0000 7f6c5fab9b80 10 client.26825 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.417+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/4235977835 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c6ace63cd0 con 0x55c6ad08b680 2024-03-14T10:00:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.417+0000 7f6c2aa00700 1 -- 172.21.15.78:0/4235977835 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c3000f910 con 0x55c6ad08b680 2024-03-14T10:00:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.417+0000 7f6c2aa00700 1 -- 172.21.15.78:0/4235977835 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6c30003b40 con 0x55c6ad08b680 2024-03-14T10:00:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.417+0000 7f6c5fab9b80 10 client.26825 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.417+0000 7f6c5fab9b80 10 client.26825 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:00:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.417+0000 7f6c5fab9b80 20 client.26825 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.417+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/4235977835 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c6ad0f8410 0x55c6ad0fa800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.417+0000 7f6c5fab9b80 4 client.26825 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:00:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.417+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/4235977835 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c6ace2ccf0 con 0x55c6ad0f8410 2024-03-14T10:00:08.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.418+0000 7f6c49600700 1 --2- 172.21.15.78:0/4235977835 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c6ad0f8410 0x55c6ad0fa800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.418+0000 7f6c2aa00700 10 client.26825 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:08.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.418+0000 7f6c49600700 1 --2- 172.21.15.78:0/4235977835 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c6ad0f8410 0x55c6ad0fa800 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd8d845d35a6eb27 server_cookie=5dc5b40f464b61c4 in_seq=0 out_seq=0 2024-03-14T10:00:08.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.418+0000 7f6c2aa00700 1 -- 172.21.15.78:0/4235977835 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55c6ace2ccf0 con 0x55c6ad0f8410 2024-03-14T10:00:08.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.418+0000 7f6c2aa00700 10 client.26825 handle_command_reply: tid=0 2024-03-14T10:00:08.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.418+0000 7f6c5fab9b80 1 client.26825 shutdown 2024-03-14T10:00:08.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.418+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/4235977835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad08b680 msgr2=0x55c6ad09c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:08.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.419+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/4235977835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad08b680 0x55c6ad09c7e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6c3002d5e0 tx=0x7f6c3000f820 comp rx=0 tx=0).stop 2024-03-14T10:00:08.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.419+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/4235977835 shutdown_connections 2024-03-14T10:00:08.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.419+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/4235977835 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c6ad0f8410 0x55c6ad0fa800 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.419+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/4235977835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c6ad08b680 0x55c6ad09c7e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.419+0000 7f6c2aa00700 0 client.26825 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:08.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.419+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/4235977835 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c6ad095ec0 0x55c6ad09b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.419+0000 7f6c5fab9b80 1 --2- 172.21.15.78:0/4235977835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6ad08bfa0 0x55c6ad09ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.419+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/4235977835 >> 172.21.15.78:0/4235977835 conn(0x55c6ad001220 msgr2=0x55c6ad08adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:08.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.419+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/4235977835 shutdown_connections 2024-03-14T10:00:08.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.419+0000 7f6c5fab9b80 1 -- 172.21.15.78:0/4235977835 wait complete. 2024-03-14T10:00:08.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.419+0000 7f6c5fab9b80 20 client.26825 trim_cache size 0 max 16384 2024-03-14T10:00:08.424 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:00:08.424 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T10:00:08.424 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T10:00:08.424 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:00:08.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.419+0000 7f6c4c000700 1 -- 172.21.15.78:0/3682746083 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6c2c0a0c80 msgr2=0x7f6c2c0a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.419+0000 7f6c4c000700 1 --2- 172.21.15.78:0/3682746083 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6c2c0a0c80 0x7f6c2c0a3140 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c017fd0 tx=0x7f6c3c00cf90 comp rx=0 tx=0).stop 2024-03-14T10:00:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.419+0000 7f6c4c000700 1 -- 172.21.15.78:0/3682746083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c4410adc0 msgr2=0x7f6c4410b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.419+0000 7f6c4c000700 1 --2- 172.21.15.78:0/3682746083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c4410adc0 0x7f6c4410b270 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00c370 tx=0x7f6c3c00c680 comp rx=0 tx=0).stop 2024-03-14T10:00:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.420+0000 7f6c4c000700 1 -- 172.21.15.78:0/3682746083 shutdown_connections 2024-03-14T10:00:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.420+0000 7f6c4c000700 1 --2- 172.21.15.78:0/3682746083 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6c2c0a0c80 0x7f6c2c0a3140 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.420+0000 7f6c4c000700 1 --2- 172.21.15.78:0/3682746083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c4410adc0 0x7f6c4410b270 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.420+0000 7f6c4c000700 1 --2- 172.21.15.78:0/3682746083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c44106050 0x7f6c4410d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.420+0000 7f6c4c000700 1 --2- 172.21.15.78:0/3682746083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c4410dca0 0x7f6c4410a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.420+0000 7f6c4c000700 1 -- 172.21.15.78:0/3682746083 >> 172.21.15.78:0/3682746083 conn(0x7f6c440fc7b0 msgr2=0x7f6c440fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.420+0000 7f6c4c000700 1 -- 172.21.15.78:0/3682746083 shutdown_connections 2024-03-14T10:00:08.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.420+0000 7f6c4c000700 1 -- 172.21.15.78:0/3682746083 wait complete. 2024-03-14T10:00:08.440 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T10:00:08.441 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T10:00:08.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.696+0000 7f0b77800700 1 -- 172.21.15.78:0/3541703165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b70105ac0 msgr2=0x7f0b70105ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:08.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.696+0000 7f0b77800700 1 --2- 172.21.15.78:0/3541703165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b70105ac0 0x7f0b70105ea0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009990 tx=0x7f0b64009ca0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.697+0000 7f0b77800700 1 -- 172.21.15.78:0/3541703165 shutdown_connections 2024-03-14T10:00:08.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.697+0000 7f0b77800700 1 --2- 172.21.15.78:0/3541703165 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b70100470 0x7f0b7010c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.697+0000 7f0b77800700 1 --2- 172.21.15.78:0/3541703165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b70105ac0 0x7f0b70105ea0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.697+0000 7f0b77800700 1 --2- 172.21.15.78:0/3541703165 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b700ffab0 0x7f0b700fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.697+0000 7f0b77800700 1 -- 172.21.15.78:0/3541703165 >> 172.21.15.78:0/3541703165 conn(0x7f0b700fb5c0 msgr2=0x7f0b700fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:08.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.697+0000 7f0b77800700 1 -- 172.21.15.78:0/3541703165 shutdown_connections 2024-03-14T10:00:08.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.697+0000 7f0b77800700 1 -- 172.21.15.78:0/3541703165 wait complete. 2024-03-14T10:00:08.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.698+0000 7f0b77800700 1 Processor -- start 2024-03-14T10:00:08.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.698+0000 7f0b77800700 1 -- start start 2024-03-14T10:00:08.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.699+0000 7f0b77800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b700ffab0 0x7f0b70101680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.699+0000 7f0b77800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b70100470 0x7f0b70101bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.699+0000 7f0b77800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b70105ac0 0x7f0b70102100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.699+0000 7f0b77800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b701a57f0 con 0x7f0b700ffab0 2024-03-14T10:00:08.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.699+0000 7f0b77800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b701a5960 con 0x7f0b70100470 2024-03-14T10:00:08.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.699+0000 7f0b77800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b701a5b20 con 0x7f0b70105ac0 2024-03-14T10:00:08.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.699+0000 7f0b74e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b700ffab0 0x7f0b70101680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.699+0000 7f0b6fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b70100470 0x7f0b70101bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.699+0000 7f0b75800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b70105ac0 0x7f0b70102100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.699+0000 7f0b6fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b70100470 0x7f0b70101bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57344/0 (socket says 172.21.15.78:57344) 2024-03-14T10:00:08.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.699+0000 7f0b6fe00700 1 -- 172.21.15.78:0/3811599005 learned_addr learned my addr 172.21.15.78:0/3811599005 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:08.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.700+0000 7f0b6fe00700 1 -- 172.21.15.78:0/3811599005 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b70105ac0 msgr2=0x7f0b70102100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:08.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.700+0000 7f0b6fe00700 1 --2- 172.21.15.78:0/3811599005 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b70105ac0 0x7f0b70102100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.700+0000 7f0b6fe00700 1 -- 172.21.15.78:0/3811599005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b700ffab0 msgr2=0x7f0b70101680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:08.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.700+0000 7f0b6fe00700 1 --2- 172.21.15.78:0/3811599005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b700ffab0 0x7f0b70101680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:08.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.700+0000 7f0b6fe00700 1 -- 172.21.15.78:0/3811599005 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b640095f0 con 0x7f0b70100470 2024-03-14T10:00:08.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.700+0000 7f0b75800700 1 --2- 172.21.15.78:0/3811599005 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b70105ac0 0x7f0b70102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:08.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.701+0000 7f0b6fe00700 1 --2- 172.21.15.78:0/3811599005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b70100470 0x7f0b70101bc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c00e9a0 tx=0x7f0b5c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:08.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.701+0000 7f0b6d600700 1 -- 172.21.15.78:0/3811599005 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5c009790 con 0x7f0b70100470 2024-03-14T10:00:08.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.702+0000 7f0b6d600700 1 -- 172.21.15.78:0/3811599005 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b5c010430 con 0x7f0b70100470 2024-03-14T10:00:08.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.702+0000 7f0b6d600700 1 -- 172.21.15.78:0/3811599005 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5c018680 con 0x7f0b70100470 2024-03-14T10:00:08.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.702+0000 7f0b77800700 1 -- 172.21.15.78:0/3811599005 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b701a5e00 con 0x7f0b70100470 2024-03-14T10:00:08.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.702+0000 7f0b77800700 1 -- 172.21.15.78:0/3811599005 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b701a6230 con 0x7f0b70100470 2024-03-14T10:00:08.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.703+0000 7f0b77800700 1 -- 172.21.15.78:0/3811599005 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b701011f0 con 0x7f0b70100470 2024-03-14T10:00:08.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.705+0000 7f0b6d600700 1 -- 172.21.15.78:0/3811599005 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f0b5c0105a0 con 0x7f0b70100470 2024-03-14T10:00:08.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.706+0000 7f0b6d600700 1 --2- 172.21.15.78:0/3811599005 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0b580a0c80 0x7f0b580a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:08.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.706+0000 7f0b6d600700 1 -- 172.21.15.78:0/3811599005 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f0b5c014070 con 0x7f0b70100470 2024-03-14T10:00:08.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.706+0000 7f0b74e00700 1 --2- 172.21.15.78:0/3811599005 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0b580a0c80 0x7f0b580a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:08.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.707+0000 7f0b74e00700 1 --2- 172.21.15.78:0/3811599005 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0b580a0c80 0x7f0b580a3140 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0b64006010 tx=0x7f0b6400b260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:08.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:08.709+0000 7f0b6d600700 1 -- 172.21.15.78:0/3811599005 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0b5c095fc0 con 0x7f0b70100470 2024-03-14T10:00:09.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.024+0000 7f0b77800700 1 -- 172.21.15.78:0/3811599005 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b700679e0 con 0x7f0b70100470 2024-03-14T10:00:09.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.027+0000 7f0b6d600700 1 -- 172.21.15.78:0/3811599005 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f0b5c0995e0 con 0x7f0b70100470 2024-03-14T10:00:09.028 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T10:00:09.028 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T10:00:09.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.033+0000 7f0b77800700 1 -- 172.21.15.78:0/3811599005 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0b580a0c80 msgr2=0x7f0b580a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.033+0000 7f0b77800700 1 --2- 172.21.15.78:0/3811599005 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0b580a0c80 0x7f0b580a3140 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0b64006010 tx=0x7f0b6400b260 comp rx=0 tx=0).stop 2024-03-14T10:00:09.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.033+0000 7f0b77800700 1 -- 172.21.15.78:0/3811599005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b70100470 msgr2=0x7f0b70101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.033+0000 7f0b77800700 1 --2- 172.21.15.78:0/3811599005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b70100470 0x7f0b70101bc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c00e9a0 tx=0x7f0b5c00ecb0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.035+0000 7f0b77800700 1 -- 172.21.15.78:0/3811599005 shutdown_connections 2024-03-14T10:00:09.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.035+0000 7f0b77800700 1 --2- 172.21.15.78:0/3811599005 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0b70105ac0 0x7f0b70102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.035+0000 7f0b77800700 1 --2- 172.21.15.78:0/3811599005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b700ffab0 0x7f0b70101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.035+0000 7f0b77800700 1 --2- 172.21.15.78:0/3811599005 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f0b580a0c80 0x7f0b580a3140 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.035+0000 7f0b77800700 1 --2- 172.21.15.78:0/3811599005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b70100470 0x7f0b70101bc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.035+0000 7f0b77800700 1 -- 172.21.15.78:0/3811599005 >> 172.21.15.78:0/3811599005 conn(0x7f0b700fb5c0 msgr2=0x7f0b701039a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.036+0000 7f0b77800700 1 -- 172.21.15.78:0/3811599005 shutdown_connections 2024-03-14T10:00:09.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.037+0000 7f0b77800700 1 -- 172.21.15.78:0/3811599005 wait complete. 2024-03-14T10:00:09.040 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T10:00:09.062 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T10:00:09.124 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:08 smithi078 ceph-mon[38143]: pgmap v182: 129 pgs: 129 active+clean; 240 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.1 MiB/s wr, 380 op/s 2024-03-14T10:00:09.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:08 smithi152 ceph-mon[48279]: pgmap v182: 129 pgs: 129 active+clean; 240 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.1 MiB/s wr, 380 op/s 2024-03-14T10:00:09.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:08 smithi132 ceph-mon[47959]: pgmap v182: 129 pgs: 129 active+clean; 240 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.1 MiB/s wr, 380 op/s 2024-03-14T10:00:09.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.306+0000 7f26d4600700 1 -- 172.21.15.78:0/2121160870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26cc100f30 msgr2=0x7f26cc101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.306+0000 7f26d4600700 1 --2- 172.21.15.78:0/2121160870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26cc100f30 0x7f26cc101310 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f26c0009990 tx=0x7f26c0009ca0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.307+0000 7f26d4600700 1 -- 172.21.15.78:0/2121160870 shutdown_connections 2024-03-14T10:00:09.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.307+0000 7f26d4600700 1 --2- 172.21.15.78:0/2121160870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26cc100f30 0x7f26cc101310 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.307+0000 7f26d4600700 1 --2- 172.21.15.78:0/2121160870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f26cc106050 0x7f26cc10dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.307+0000 7f26d4600700 1 --2- 172.21.15.78:0/2121160870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26cc1018e0 0x7f26cc105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.307+0000 7f26d4600700 1 -- 172.21.15.78:0/2121160870 >> 172.21.15.78:0/2121160870 conn(0x7f26cc0fc7b0 msgr2=0x7f26cc0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.310+0000 7f26d4600700 1 -- 172.21.15.78:0/2121160870 shutdown_connections 2024-03-14T10:00:09.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.310+0000 7f26d4600700 1 -- 172.21.15.78:0/2121160870 wait complete. 2024-03-14T10:00:09.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.312+0000 7f26d4600700 1 Processor -- start 2024-03-14T10:00:09.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.313+0000 7f26d4600700 1 -- start start 2024-03-14T10:00:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.313+0000 7f26d4600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26cc100f30 0x7f26cc197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.313+0000 7f26d4600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26cc106050 0x7f26cc197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.313+0000 7f26d4600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f26cc19bee0 0x7f26cc19c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.313+0000 7f26d4600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc19c960 con 0x7f26cc100f30 2024-03-14T10:00:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.313+0000 7f26d4600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc19cad0 con 0x7f26cc106050 2024-03-14T10:00:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.313+0000 7f26d4600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26cc19cc90 con 0x7f26cc19bee0 2024-03-14T10:00:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.314+0000 7f26d1200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26cc106050 0x7f26cc197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.314+0000 7f26d1c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26cc100f30 0x7f26cc197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.314+0000 7f26d1200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26cc106050 0x7f26cc197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57356/0 (socket says 172.21.15.78:57356) 2024-03-14T10:00:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.314+0000 7f26d1200700 1 -- 172.21.15.78:0/150785695 learned_addr learned my addr 172.21.15.78:0/150785695 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.314+0000 7f26d2600700 1 --2- 172.21.15.78:0/150785695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f26cc19bee0 0x7f26cc19c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.315+0000 7f26d1200700 1 -- 172.21.15.78:0/150785695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f26cc19bee0 msgr2=0x7f26cc19c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.315+0000 7f26d1200700 1 --2- 172.21.15.78:0/150785695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f26cc19bee0 0x7f26cc19c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.315+0000 7f26d1200700 1 -- 172.21.15.78:0/150785695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26cc100f30 msgr2=0x7f26cc197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.315+0000 7f26d1200700 1 --2- 172.21.15.78:0/150785695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26cc100f30 0x7f26cc197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.315+0000 7f26d1200700 1 -- 172.21.15.78:0/150785695 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c4009c40 con 0x7f26cc106050 2024-03-14T10:00:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.315+0000 7f26d2600700 1 --2- 172.21.15.78:0/150785695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f26cc19bee0 0x7f26cc19c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T10:00:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.315+0000 7f26d1c00700 1 --2- 172.21.15.78:0/150785695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26cc100f30 0x7f26cc197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T10:00:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.315+0000 7f26d1200700 1 --2- 172.21.15.78:0/150785695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26cc106050 0x7f26cc197b40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f26b800e970 tx=0x7f26b800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.316+0000 7f26ca000700 1 -- 172.21.15.78:0/150785695 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b8009790 con 0x7f26cc106050 2024-03-14T10:00:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.316+0000 7f26d4600700 1 -- 172.21.15.78:0/150785695 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c00095f0 con 0x7f26cc106050 2024-03-14T10:00:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.316+0000 7f26ca000700 1 -- 172.21.15.78:0/150785695 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f26b8004cf0 con 0x7f26cc106050 2024-03-14T10:00:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.316+0000 7f26ca000700 1 -- 172.21.15.78:0/150785695 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b8005680 con 0x7f26cc106050 2024-03-14T10:00:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.316+0000 7f26d4600700 1 -- 172.21.15.78:0/150785695 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26cc1a4590 con 0x7f26cc106050 2024-03-14T10:00:09.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.320+0000 7f26ca000700 1 -- 172.21.15.78:0/150785695 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f26b8004750 con 0x7f26cc106050 2024-03-14T10:00:09.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.322+0000 7f26ca000700 1 --2- 172.21.15.78:0/150785695 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f26b40a0c80 0x7f26b40a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.322+0000 7f26ca000700 1 -- 172.21.15.78:0/150785695 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f26b8014070 con 0x7f26cc106050 2024-03-14T10:00:09.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.322+0000 7f26d1c00700 1 --2- 172.21.15.78:0/150785695 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f26b40a0c80 0x7f26b40a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.323+0000 7f26e805cb80 1 Processor -- start 2024-03-14T10:00:09.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.323+0000 7f26d1c00700 1 --2- 172.21.15.78:0/150785695 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f26b40a0c80 0x7f26b40a3140 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f26c0006010 tx=0x7f26c0005e80 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.323+0000 7f26e805cb80 1 -- start start 2024-03-14T10:00:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.324+0000 7f26e805cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23680 0x5580cfa23a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.324+0000 7f26e805cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23fa0 0x5580cfa31d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.324+0000 7f26e805cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa322d0 0x5580cfa346c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.324+0000 7f26e805cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580cf97c990 con 0x5580cfa322d0 2024-03-14T10:00:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.324+0000 7f26e805cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580cf97a350 con 0x5580cfa23fa0 2024-03-14T10:00:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.324+0000 7f26e805cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580cf975c40 con 0x5580cfa23680 2024-03-14T10:00:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.324+0000 7f26d1c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23680 0x5580cfa23a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.324+0000 7f26d1c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23680 0x5580cfa23a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35676/0 (socket says 172.21.15.78:35676) 2024-03-14T10:00:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.324+0000 7f26d1c00700 1 -- 172.21.15.78:0/523927444 learned_addr learned my addr 172.21.15.78:0/523927444 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.325+0000 7f26d1200700 1 --2- 172.21.15.78:0/523927444 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23fa0 0x5580cfa31d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.325+0000 7f26d2600700 1 --2- 172.21.15.78:0/523927444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa322d0 0x5580cfa346c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.325+0000 7f26d1c00700 1 -- 172.21.15.78:0/523927444 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23fa0 msgr2=0x5580cfa31d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.325+0000 7f26d1c00700 1 --2- 172.21.15.78:0/523927444 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23fa0 0x5580cfa31d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.325+0000 7f26d1c00700 1 -- 172.21.15.78:0/523927444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa322d0 msgr2=0x5580cfa346c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.325+0000 7f26d1c00700 1 --2- 172.21.15.78:0/523927444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa322d0 0x5580cfa346c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.325+0000 7f26d1c00700 1 -- 172.21.15.78:0/523927444 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580cf7f73b0 con 0x5580cfa23680 2024-03-14T10:00:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.325+0000 7f26d1200700 1 --2- 172.21.15.78:0/523927444 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23fa0 0x5580cfa31d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:00:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.325+0000 7f26d2600700 1 --2- 172.21.15.78:0/523927444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa322d0 0x5580cfa346c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T10:00:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.325+0000 7f26d1c00700 1 --2- 172.21.15.78:0/523927444 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23680 0x5580cfa23a60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f26c0015000 tx=0x7f26c001e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1be644af65b60780 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.325+0000 7f26bea00700 1 -- 172.21.15.78:0/523927444 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c001e070 con 0x5580cfa23680 2024-03-14T10:00:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.325+0000 7f26bea00700 1 -- 172.21.15.78:0/523927444 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f26c0005510 con 0x5580cfa23680 2024-03-14T10:00:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.325+0000 7f26bea00700 1 -- 172.21.15.78:0/523927444 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c0004880 con 0x5580cfa23680 2024-03-14T10:00:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.326+0000 7f26e805cb80 1 -- 172.21.15.78:0/523927444 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23680 msgr2=0x5580cfa23a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.326+0000 7f26e805cb80 1 --2- 172.21.15.78:0/523927444 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23680 0x5580cfa23a60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f26c0015000 tx=0x7f26c001e000 comp rx=0 tx=0).stop 2024-03-14T10:00:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.326+0000 7f26e805cb80 1 -- 172.21.15.78:0/523927444 shutdown_connections 2024-03-14T10:00:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.327+0000 7f26e805cb80 1 --2- 172.21.15.78:0/523927444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa322d0 0x5580cfa346c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.327+0000 7f26e805cb80 1 --2- 172.21.15.78:0/523927444 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23680 0x5580cfa23a60 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.327+0000 7f26e805cb80 1 --2- 172.21.15.78:0/523927444 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23fa0 0x5580cfa31d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.327+0000 7f26e805cb80 1 -- 172.21.15.78:0/523927444 >> 172.21.15.78:0/523927444 conn(0x5580cf999220 msgr2=0x5580cfa1eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.327+0000 7f26e805cb80 1 -- 172.21.15.78:0/523927444 shutdown_connections 2024-03-14T10:00:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.327+0000 7f26e805cb80 1 -- 172.21.15.78:0/523927444 wait complete. 2024-03-14T10:00:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.327+0000 7f26e805cb80 1 -- start start 2024-03-14T10:00:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.328+0000 7f26e805cb80 1 Processor -- start 2024-03-14T10:00:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.328+0000 7f26e805cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23680 0x5580cfa318a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.329+0000 7f26e805cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa23fa0 0x5580cfa2ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.329+0000 7f26e805cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa322d0 0x5580cfa2b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.329+0000 7f26e805cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580cf911d80 con 0x5580cfa23fa0 2024-03-14T10:00:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.329+0000 7f26e805cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580cf911a20 con 0x5580cfa322d0 2024-03-14T10:00:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.329+0000 7f26e805cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580cf7f96a0 con 0x5580cfa23680 2024-03-14T10:00:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.329+0000 7f26e805cb80 10 client.0 _ll_register_callbacks cb 0x5580cf77d3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:09.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.329+0000 7f26d1c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23680 0x5580cfa318a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.329+0000 7f26d2600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa322d0 0x5580cfa2b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.329+0000 7f26d1200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa23fa0 0x5580cfa2ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.329+0000 7f26d2600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa322d0 0x5580cfa2b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57374/0 (socket says 172.21.15.78:57374) 2024-03-14T10:00:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.329+0000 7f26d1c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23680 0x5580cfa318a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35688/0 (socket says 172.21.15.78:35688) 2024-03-14T10:00:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.329+0000 7f26d2600700 1 -- 172.21.15.78:0/2904286130 learned_addr learned my addr 172.21.15.78:0/2904286130 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.330+0000 7f26d1c00700 1 -- 172.21.15.78:0/2904286130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa322d0 msgr2=0x5580cfa2b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.330+0000 7f26d1c00700 1 --2- 172.21.15.78:0/2904286130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa322d0 0x5580cfa2b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.330+0000 7f26d1c00700 1 -- 172.21.15.78:0/2904286130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa23fa0 msgr2=0x5580cfa2ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.330+0000 7f26d1c00700 1 --2- 172.21.15.78:0/2904286130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa23fa0 0x5580cfa2ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.330+0000 7f26d1c00700 1 -- 172.21.15.78:0/2904286130 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26cc10e440 con 0x5580cfa23680 2024-03-14T10:00:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.330+0000 7f26d1200700 1 --2- 172.21.15.78:0/2904286130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa23fa0 0x5580cfa2ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.330+0000 7f26d2600700 1 --2- 172.21.15.78:0/2904286130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa322d0 0x5580cfa2b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.330+0000 7f26d1c00700 1 --2- 172.21.15.78:0/2904286130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23680 0x5580cfa318a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f26c000b6f0 tx=0x7f26c000be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef4bde0c7e925cb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.330+0000 7f26b3400700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:00:09.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.330+0000 7f26b3400700 1 -- 172.21.15.78:0/2904286130 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c0005050 con 0x5580cfa23680 2024-03-14T10:00:09.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.330+0000 7f26b3400700 1 -- 172.21.15.78:0/2904286130 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f26c00051b0 con 0x5580cfa23680 2024-03-14T10:00:09.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.330+0000 7f26e805cb80 1 -- 172.21.15.78:0/2904286130 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5580cfa16720 con 0x5580cfa23680 2024-03-14T10:00:09.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.331+0000 7f26b3400700 1 -- 172.21.15.78:0/2904286130 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26c0005370 con 0x5580cfa23680 2024-03-14T10:00:09.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.331+0000 7f26e805cb80 10 client.26830 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:09.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.331+0000 7f26e805cb80 1 -- 172.21.15.78:0/2904286130 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5580cf7a3b60 con 0x5580cfa23680 2024-03-14T10:00:09.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.331+0000 7f26b3400700 1 -- 172.21.15.78:0/2904286130 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f26c0030430 con 0x5580cfa23680 2024-03-14T10:00:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.332+0000 7f26b3400700 1 -- 172.21.15.78:0/2904286130 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f26c0026420 con 0x5580cfa23680 2024-03-14T10:00:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.332+0000 7f26e805cb80 10 client.26830 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.332+0000 7f26e805cb80 10 client.26830 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:00:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.332+0000 7f26e805cb80 20 client.26830 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.332+0000 7f26e805cb80 1 --2- 172.21.15.78:0/2904286130 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5580cfa45ff0 0x5580cfa483e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.332+0000 7f26e805cb80 4 client.26830 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:00:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.332+0000 7f26e805cb80 1 -- 172.21.15.78:0/2904286130 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5580cfa16720 con 0x5580cfa45ff0 2024-03-14T10:00:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.332+0000 7f26d1200700 1 --2- 172.21.15.78:0/2904286130 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5580cfa45ff0 0x5580cfa483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.333+0000 7f26d1200700 1 --2- 172.21.15.78:0/2904286130 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5580cfa45ff0 0x5580cfa483e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4bdd04bd8751abb1 server_cookie=1b8449bf25faadbc in_seq=0 out_seq=0 2024-03-14T10:00:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.333+0000 7f26b3400700 10 client.26830 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:00:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.333+0000 7f26b3400700 1 -- 172.21.15.78:0/2904286130 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5580cfa16720 con 0x5580cfa45ff0 2024-03-14T10:00:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.333+0000 7f26b3400700 10 client.26830 handle_command_reply: tid=0 2024-03-14T10:00:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.334+0000 7f26e805cb80 1 client.26830 shutdown 2024-03-14T10:00:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.334+0000 7f26e805cb80 1 -- 172.21.15.78:0/2904286130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23680 msgr2=0x5580cfa318a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.334+0000 7f26e805cb80 1 --2- 172.21.15.78:0/2904286130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23680 0x5580cfa318a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f26c000b6f0 tx=0x7f26c000be20 comp rx=0 tx=0).stop 2024-03-14T10:00:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.334+0000 7f26e805cb80 1 -- 172.21.15.78:0/2904286130 shutdown_connections 2024-03-14T10:00:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.334+0000 7f26e805cb80 1 --2- 172.21.15.78:0/2904286130 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5580cfa45ff0 0x5580cfa483e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.334+0000 7f26e805cb80 1 --2- 172.21.15.78:0/2904286130 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa322d0 0x5580cfa2b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.334+0000 7f26b3400700 0 client.26830 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:00:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.334+0000 7f26e805cb80 1 --2- 172.21.15.78:0/2904286130 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23680 0x5580cfa318a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.334+0000 7f26e805cb80 1 --2- 172.21.15.78:0/2904286130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa23fa0 0x5580cfa2ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.334+0000 7f26e805cb80 1 -- 172.21.15.78:0/2904286130 >> 172.21.15.78:0/2904286130 conn(0x5580cf999220 msgr2=0x5580cfa1e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.334+0000 7f26e805cb80 1 -- 172.21.15.78:0/2904286130 shutdown_connections 2024-03-14T10:00:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.334+0000 7f26e805cb80 1 -- 172.21.15.78:0/2904286130 wait complete. 2024-03-14T10:00:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.335+0000 7f26e805cb80 20 client.26830 trim_cache size 0 max 16384 2024-03-14T10:00:09.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.349+0000 7f26e805cb80 1 Processor -- start 2024-03-14T10:00:09.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.349+0000 7f26e805cb80 1 -- start start 2024-03-14T10:00:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.349+0000 7f26e805cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23680 0x5580cfa31560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.349+0000 7f26e805cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23fa0 0x5580cfa29420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.349+0000 7f26e805cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa322d0 0x5580cfa29960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.349+0000 7f26e805cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580cfa2c150 con 0x5580cfa322d0 2024-03-14T10:00:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.349+0000 7f26e805cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580cfa29ea0 con 0x5580cfa23680 2024-03-14T10:00:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.349+0000 7f26e805cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580cfa20060 con 0x5580cfa23fa0 2024-03-14T10:00:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.349+0000 7f26d1c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23680 0x5580cfa31560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.349+0000 7f26d1c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23680 0x5580cfa31560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57384/0 (socket says 172.21.15.78:57384) 2024-03-14T10:00:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.349+0000 7f26d1c00700 1 -- 172.21.15.78:0/1585702348 learned_addr learned my addr 172.21.15.78:0/1585702348 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.350+0000 7f26d2600700 1 --2- 172.21.15.78:0/1585702348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa322d0 0x5580cfa29960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.350+0000 7f26d1200700 1 --2- 172.21.15.78:0/1585702348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23fa0 0x5580cfa29420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.350+0000 7f26d1c00700 1 -- 172.21.15.78:0/1585702348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23fa0 msgr2=0x5580cfa29420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.350+0000 7f26d1c00700 1 --2- 172.21.15.78:0/1585702348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23fa0 0x5580cfa29420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.350+0000 7f26d1c00700 1 -- 172.21.15.78:0/1585702348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa322d0 msgr2=0x5580cfa29960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.350+0000 7f26d1c00700 1 --2- 172.21.15.78:0/1585702348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa322d0 0x5580cfa29960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.350+0000 7f26d1c00700 1 -- 172.21.15.78:0/1585702348 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580cf93bea0 con 0x5580cfa23680 2024-03-14T10:00:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.350+0000 7f26d2600700 1 --2- 172.21.15.78:0/1585702348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa322d0 0x5580cfa29960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:00:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.350+0000 7f26d1c00700 1 --2- 172.21.15.78:0/1585702348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23680 0x5580cfa31560 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f26c00057a0 tx=0x7f26c0009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f323b3458d1ae63 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.350+0000 7f26bf400700 1 -- 172.21.15.78:0/1585702348 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c001e040 con 0x5580cfa23680 2024-03-14T10:00:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.350+0000 7f26bf400700 1 -- 172.21.15.78:0/1585702348 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f26c000f870 con 0x5580cfa23680 2024-03-14T10:00:09.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.350+0000 7f26bf400700 1 -- 172.21.15.78:0/1585702348 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c0024b30 con 0x5580cfa23680 2024-03-14T10:00:09.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.351+0000 7f26e805cb80 1 -- 172.21.15.78:0/1585702348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23680 msgr2=0x5580cfa31560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.351+0000 7f26e805cb80 1 --2- 172.21.15.78:0/1585702348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23680 0x5580cfa31560 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f26c00057a0 tx=0x7f26c0009310 comp rx=0 tx=0).stop 2024-03-14T10:00:09.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.351+0000 7f26e805cb80 1 -- 172.21.15.78:0/1585702348 shutdown_connections 2024-03-14T10:00:09.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.351+0000 7f26e805cb80 1 --2- 172.21.15.78:0/1585702348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23680 0x5580cfa31560 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.351+0000 7f26e805cb80 1 --2- 172.21.15.78:0/1585702348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa322d0 0x5580cfa29960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.351+0000 7f26e805cb80 1 --2- 172.21.15.78:0/1585702348 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa23fa0 0x5580cfa29420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.351+0000 7f26e805cb80 1 -- 172.21.15.78:0/1585702348 >> 172.21.15.78:0/1585702348 conn(0x5580cf999220 msgr2=0x5580cfa25e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.352+0000 7f26e805cb80 1 -- 172.21.15.78:0/1585702348 shutdown_connections 2024-03-14T10:00:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.352+0000 7f26e805cb80 1 -- 172.21.15.78:0/1585702348 wait complete. 2024-03-14T10:00:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.352+0000 7f26e805cb80 1 -- start start 2024-03-14T10:00:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.352+0000 7f26e805cb80 1 Processor -- start 2024-03-14T10:00:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.353+0000 7f26e805cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa23680 0x5580cfa42810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.353+0000 7f26e805cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23fa0 0x5580cfa42d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.353+0000 7f26e805cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa322d0 0x5580cfa43290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.353+0000 7f26e805cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580cfa3f7e0 con 0x5580cfa23680 2024-03-14T10:00:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.353+0000 7f26e805cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580cfa3f950 con 0x5580cfa23fa0 2024-03-14T10:00:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.353+0000 7f26e805cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580cfa3fac0 con 0x5580cfa322d0 2024-03-14T10:00:09.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.353+0000 7f26e805cb80 10 client.0 _ll_register_callbacks cb 0x7f26c00056d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.353+0000 7f26d1200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23fa0 0x5580cfa42d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.353+0000 7f26d1c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa23680 0x5580cfa42810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.353+0000 7f26d1200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23fa0 0x5580cfa42d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57388/0 (socket says 172.21.15.78:57388) 2024-03-14T10:00:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.353+0000 7f26d1200700 1 -- 172.21.15.78:0/497960380 learned_addr learned my addr 172.21.15.78:0/497960380 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.353+0000 7f26d2600700 1 --2- 172.21.15.78:0/497960380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa322d0 0x5580cfa43290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.354+0000 7f26d1200700 1 -- 172.21.15.78:0/497960380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa322d0 msgr2=0x5580cfa43290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.354+0000 7f26d1200700 1 --2- 172.21.15.78:0/497960380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa322d0 0x5580cfa43290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.354+0000 7f26d1200700 1 -- 172.21.15.78:0/497960380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa23680 msgr2=0x5580cfa42810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.354+0000 7f26d1200700 1 --2- 172.21.15.78:0/497960380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa23680 0x5580cfa42810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.354+0000 7f26d1200700 1 -- 172.21.15.78:0/497960380 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5580cf7a3b60 con 0x5580cfa23fa0 2024-03-14T10:00:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.354+0000 7f26d1200700 1 --2- 172.21.15.78:0/497960380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23fa0 0x5580cfa42d50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f26b800c5b0 tx=0x7f26b8095750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0a90de052fb4aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.354+0000 7f26b3400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T10:00:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.354+0000 7f26b3400700 1 -- 172.21.15.78:0/497960380 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b80b1ea0 con 0x5580cfa23fa0 2024-03-14T10:00:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.354+0000 7f26e805cb80 1 -- 172.21.15.78:0/497960380 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5580cf96e700 con 0x5580cfa23fa0 2024-03-14T10:00:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.354+0000 7f26b3400700 1 -- 172.21.15.78:0/497960380 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f26b80992f0 con 0x5580cfa23fa0 2024-03-14T10:00:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.355+0000 7f26b3400700 1 -- 172.21.15.78:0/497960380 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26b809d120 con 0x5580cfa23fa0 2024-03-14T10:00:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.355+0000 7f26e805cb80 10 client.26849 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.355+0000 7f26e805cb80 1 -- 172.21.15.78:0/497960380 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5580cf7fbcd0 con 0x5580cfa23fa0 2024-03-14T10:00:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.355+0000 7f26b3400700 1 -- 172.21.15.78:0/497960380 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5580cf96e700 con 0x5580cfa23fa0 2024-03-14T10:00:09.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.355+0000 7f26b3400700 1 -- 172.21.15.78:0/497960380 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f26b8004a00 con 0x5580cfa23fa0 2024-03-14T10:00:09.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.355+0000 7f26e805cb80 10 client.26849 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:09.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.355+0000 7f26e805cb80 10 client.26849 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:00:09.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.355+0000 7f26e805cb80 20 client.26849 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:09.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.355+0000 7f26e805cb80 1 --2- 172.21.15.78:0/497960380 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5580cfa98130 0x5580cfa9a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.355+0000 7f26e805cb80 4 client.26849 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:00:09.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.355+0000 7f26e805cb80 1 -- 172.21.15.78:0/497960380 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5580cf7c4cf0 con 0x5580cfa98130 2024-03-14T10:00:09.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.355+0000 7f26d1c00700 1 --2- 172.21.15.78:0/497960380 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5580cfa98130 0x5580cfa9a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.356+0000 7f26d1c00700 1 --2- 172.21.15.78:0/497960380 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5580cfa98130 0x5580cfa9a550 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2d4054c47ff9c004 server_cookie=1ba3759ca270fcb in_seq=0 out_seq=0 2024-03-14T10:00:09.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.356+0000 7f26b3400700 10 client.26849 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:00:09.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.356+0000 7f26b3400700 1 -- 172.21.15.78:0/497960380 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5580cf7c4cf0 con 0x5580cfa98130 2024-03-14T10:00:09.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.356+0000 7f26b3400700 10 client.26849 handle_command_reply: tid=0 2024-03-14T10:00:09.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.356+0000 7f26e805cb80 1 client.26849 shutdown 2024-03-14T10:00:09.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.356+0000 7f26e805cb80 1 -- 172.21.15.78:0/497960380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23fa0 msgr2=0x5580cfa42d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.356+0000 7f26e805cb80 1 --2- 172.21.15.78:0/497960380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23fa0 0x5580cfa42d50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f26b800c5b0 tx=0x7f26b8095750 comp rx=0 tx=0).stop 2024-03-14T10:00:09.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.356+0000 7f26e805cb80 1 -- 172.21.15.78:0/497960380 shutdown_connections 2024-03-14T10:00:09.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.356+0000 7f26e805cb80 1 --2- 172.21.15.78:0/497960380 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x5580cfa98130 0x5580cfa9a550 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.356+0000 7f26e805cb80 1 --2- 172.21.15.78:0/497960380 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5580cfa322d0 0x5580cfa43290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.356+0000 7f26e805cb80 1 --2- 172.21.15.78:0/497960380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5580cfa23680 0x5580cfa42810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.356+0000 7f26b3400700 0 client.26849 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:00:09.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.356+0000 7f26e805cb80 1 --2- 172.21.15.78:0/497960380 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580cfa23fa0 0x5580cfa42d50 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.356+0000 7f26e805cb80 1 -- 172.21.15.78:0/497960380 >> 172.21.15.78:0/497960380 conn(0x5580cf999220 msgr2=0x5580cfa29360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.357+0000 7f26e805cb80 1 -- 172.21.15.78:0/497960380 shutdown_connections 2024-03-14T10:00:09.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.357+0000 7f26e805cb80 1 -- 172.21.15.78:0/497960380 wait complete. 2024-03-14T10:00:09.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.357+0000 7f26e805cb80 20 client.26849 trim_cache size 0 max 16384 2024-03-14T10:00:09.360 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:00:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.357+0000 7f26d4600700 1 -- 172.21.15.78:0/150785695 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f26b40a0c80 msgr2=0x7f26b40a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.357+0000 7f26d4600700 1 --2- 172.21.15.78:0/150785695 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f26b40a0c80 0x7f26b40a3140 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f26c0006010 tx=0x7f26c0005e80 comp rx=0 tx=0).stop 2024-03-14T10:00:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.357+0000 7f26d4600700 1 -- 172.21.15.78:0/150785695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26cc106050 msgr2=0x7f26cc197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.357+0000 7f26d4600700 1 --2- 172.21.15.78:0/150785695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26cc106050 0x7f26cc197b40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f26b800e970 tx=0x7f26b800ed30 comp rx=0 tx=0).stop 2024-03-14T10:00:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.358+0000 7f26d4600700 1 -- 172.21.15.78:0/150785695 shutdown_connections 2024-03-14T10:00:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.358+0000 7f26d4600700 1 --2- 172.21.15.78:0/150785695 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f26cc19bee0 0x7f26cc19c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.358+0000 7f26d4600700 1 --2- 172.21.15.78:0/150785695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26cc100f30 0x7f26cc197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.358+0000 7f26d4600700 1 --2- 172.21.15.78:0/150785695 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f26b40a0c80 0x7f26b40a3140 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.358+0000 7f26d4600700 1 --2- 172.21.15.78:0/150785695 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26cc106050 0x7f26cc197b40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.358+0000 7f26d4600700 1 -- 172.21.15.78:0/150785695 >> 172.21.15.78:0/150785695 conn(0x7f26cc0fc7b0 msgr2=0x7f26cc0fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.358+0000 7f26d4600700 1 -- 172.21.15.78:0/150785695 shutdown_connections 2024-03-14T10:00:09.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.358+0000 7f26d4600700 1 -- 172.21.15.78:0/150785695 wait complete. 2024-03-14T10:00:09.385 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T10:00:09.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.629+0000 7f2302a00700 1 -- 172.21.15.78:0/3220973918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22fc100f20 msgr2=0x7f22fc101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.629+0000 7f2302a00700 1 --2- 172.21.15.78:0/3220973918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22fc100f20 0x7f22fc101300 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009990 tx=0x7f22ec009ca0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.629+0000 7f2302a00700 1 -- 172.21.15.78:0/3220973918 shutdown_connections 2024-03-14T10:00:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.629+0000 7f2302a00700 1 --2- 172.21.15.78:0/3220973918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f22fc106040 0x7f22fc10dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.629+0000 7f2302a00700 1 --2- 172.21.15.78:0/3220973918 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22fc100f20 0x7f22fc101300 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.629+0000 7f2302a00700 1 --2- 172.21.15.78:0/3220973918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22fc1018d0 0x7f22fc105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.629+0000 7f2302a00700 1 -- 172.21.15.78:0/3220973918 >> 172.21.15.78:0/3220973918 conn(0x7f22fc0fc7a0 msgr2=0x7f22fc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.630+0000 7f2302a00700 1 -- 172.21.15.78:0/3220973918 shutdown_connections 2024-03-14T10:00:09.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.630+0000 7f2302a00700 1 -- 172.21.15.78:0/3220973918 wait complete. 2024-03-14T10:00:09.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.631+0000 7f2302a00700 1 Processor -- start 2024-03-14T10:00:09.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.631+0000 7f2302a00700 1 -- start start 2024-03-14T10:00:09.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.632+0000 7f2302a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22fc100f20 0x7f22fc1976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.632+0000 7f2302a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22fc106040 0x7f22fc197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.632+0000 7f2302a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f22fc19bf30 0x7f22fc19c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.632+0000 7f2302a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc19c9b0 con 0x7f22fc100f20 2024-03-14T10:00:09.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.632+0000 7f2302a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc19cb20 con 0x7f22fc19bf30 2024-03-14T10:00:09.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.632+0000 7f2302a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc19cce0 con 0x7f22fc106040 2024-03-14T10:00:09.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.632+0000 7f22fb400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22fc106040 0x7f22fc197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.632+0000 7f22fb400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22fc106040 0x7f22fc197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35722/0 (socket says 172.21.15.78:35722) 2024-03-14T10:00:09.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.632+0000 7f22fb400700 1 -- 172.21.15.78:0/2015288700 learned_addr learned my addr 172.21.15.78:0/2015288700 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:09.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.632+0000 7f22fbe00700 1 --2- 172.21.15.78:0/2015288700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22fc100f20 0x7f22fc1976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.632+0000 7f2300a00700 1 --2- 172.21.15.78:0/2015288700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f22fc19bf30 0x7f22fc19c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.633+0000 7f22fb400700 1 -- 172.21.15.78:0/2015288700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f22fc19bf30 msgr2=0x7f22fc19c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.633+0000 7f22fb400700 1 --2- 172.21.15.78:0/2015288700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f22fc19bf30 0x7f22fc19c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.633+0000 7f22fb400700 1 -- 172.21.15.78:0/2015288700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22fc100f20 msgr2=0x7f22fc1976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.633+0000 7f22fb400700 1 --2- 172.21.15.78:0/2015288700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22fc100f20 0x7f22fc1976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.633+0000 7f22fb400700 1 -- 172.21.15.78:0/2015288700 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec0095f0 con 0x7f22fc106040 2024-03-14T10:00:09.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.633+0000 7f22fb400700 1 --2- 172.21.15.78:0/2015288700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22fc106040 0x7f22fc197be0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f22e800d950 tx=0x7f22e800dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.633+0000 7f22f8c00700 1 -- 172.21.15.78:0/2015288700 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8004220 con 0x7f22fc106040 2024-03-14T10:00:09.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.633+0000 7f22f8c00700 1 -- 172.21.15.78:0/2015288700 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22e8007620 con 0x7f22fc106040 2024-03-14T10:00:09.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.633+0000 7f2302a00700 1 -- 172.21.15.78:0/2015288700 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22fc1a4240 con 0x7f22fc106040 2024-03-14T10:00:09.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.633+0000 7f22f8c00700 1 -- 172.21.15.78:0/2015288700 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e80056c0 con 0x7f22fc106040 2024-03-14T10:00:09.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.633+0000 7f2302a00700 1 -- 172.21.15.78:0/2015288700 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22fc1a4770 con 0x7f22fc106040 2024-03-14T10:00:09.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.637+0000 7f22f8c00700 1 -- 172.21.15.78:0/2015288700 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f22e8004380 con 0x7f22fc106040 2024-03-14T10:00:09.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.638+0000 7f23165c9b80 1 Processor -- start 2024-03-14T10:00:09.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.638+0000 7f23165c9b80 1 -- start start 2024-03-14T10:00:09.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.638+0000 7f23165c9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464680 0x565377464a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.638+0000 7f23165c9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464fa0 0x565377472d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.638+0000 7f23165c9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5653774732d0 0x5653774756c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.638+0000 7f23165c9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653773bd990 con 0x5653774732d0 2024-03-14T10:00:09.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.638+0000 7f23165c9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653773bb350 con 0x565377464fa0 2024-03-14T10:00:09.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.638+0000 7f23165c9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653773b6c40 con 0x565377464680 2024-03-14T10:00:09.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.638+0000 7f2300a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464680 0x565377464a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.638+0000 7f2300a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464680 0x565377464a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35732/0 (socket says 172.21.15.78:35732) 2024-03-14T10:00:09.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.638+0000 7f2300a00700 1 -- 172.21.15.78:0/2189414288 learned_addr learned my addr 172.21.15.78:0/2189414288 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:09.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.638+0000 7f22fbe00700 1 --2- 172.21.15.78:0/2189414288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464fa0 0x565377472d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.638+0000 7f22fb400700 1 --2- 172.21.15.78:0/2189414288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5653774732d0 0x5653774756c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.639+0000 7f2300a00700 1 -- 172.21.15.78:0/2189414288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464fa0 msgr2=0x565377472d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.639+0000 7f2300a00700 1 --2- 172.21.15.78:0/2189414288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464fa0 0x565377472d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.639+0000 7f2300a00700 1 -- 172.21.15.78:0/2189414288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5653774732d0 msgr2=0x5653774756c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.639+0000 7f2300a00700 1 --2- 172.21.15.78:0/2189414288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5653774732d0 0x5653774756c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.639+0000 7f2300a00700 1 -- 172.21.15.78:0/2189414288 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653772383b0 con 0x565377464680 2024-03-14T10:00:09.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.639+0000 7f22fb400700 1 --2- 172.21.15.78:0/2189414288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5653774732d0 0x5653774756c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:00:09.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.639+0000 7f22fbe00700 1 --2- 172.21.15.78:0/2189414288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464fa0 0x565377472d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:00:09.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.639+0000 7f22f8c00700 1 --2- 172.21.15.78:0/2015288700 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f22e00a0c80 0x7f22e00a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.639+0000 7f2300a00700 1 --2- 172.21.15.78:0/2189414288 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464680 0x565377464a60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f22f4003fa0 tx=0x7f22f4009ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16af49754142b563 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.639+0000 7f22f8c00700 1 -- 172.21.15.78:0/2015288700 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f22e80cc450 con 0x7f22fc106040 2024-03-14T10:00:09.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.639+0000 7f22f1200700 1 -- 172.21.15.78:0/2189414288 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4019070 con 0x565377464680 2024-03-14T10:00:09.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.639+0000 7f22fbe00700 1 --2- 172.21.15.78:0/2015288700 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f22e00a0c80 0x7f22e00a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.639+0000 7f22f1200700 1 -- 172.21.15.78:0/2189414288 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22f400f040 con 0x565377464680 2024-03-14T10:00:09.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.639+0000 7f22f1200700 1 -- 172.21.15.78:0/2189414288 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4007b60 con 0x565377464680 2024-03-14T10:00:09.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.640+0000 7f22fbe00700 1 --2- 172.21.15.78:0/2015288700 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f22e00a0c80 0x7f22e00a3140 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f22ec00b5c0 tx=0x7f22ec005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.640+0000 7f23165c9b80 1 -- 172.21.15.78:0/2189414288 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464680 msgr2=0x565377464a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.640+0000 7f23165c9b80 1 --2- 172.21.15.78:0/2189414288 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464680 0x565377464a60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f22f4003fa0 tx=0x7f22f4009ba0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.640+0000 7f23165c9b80 1 -- 172.21.15.78:0/2189414288 shutdown_connections 2024-03-14T10:00:09.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.640+0000 7f23165c9b80 1 --2- 172.21.15.78:0/2189414288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5653774732d0 0x5653774756c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.640+0000 7f23165c9b80 1 --2- 172.21.15.78:0/2189414288 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464680 0x565377464a60 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.640+0000 7f23165c9b80 1 --2- 172.21.15.78:0/2189414288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464fa0 0x565377472d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.640+0000 7f23165c9b80 1 -- 172.21.15.78:0/2189414288 >> 172.21.15.78:0/2189414288 conn(0x5653773da220 msgr2=0x56537745faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.641+0000 7f23165c9b80 1 -- 172.21.15.78:0/2189414288 shutdown_connections 2024-03-14T10:00:09.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.641+0000 7f23165c9b80 1 -- 172.21.15.78:0/2189414288 wait complete. 2024-03-14T10:00:09.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.641+0000 7f23165c9b80 1 -- start start 2024-03-14T10:00:09.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.641+0000 7f23165c9b80 1 Processor -- start 2024-03-14T10:00:09.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f23165c9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464680 0x565377472c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f23165c9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377464fa0 0x56537746ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f23165c9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653774732d0 0x5653774705e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f23165c9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x565377352d80 con 0x565377464fa0 2024-03-14T10:00:09.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f23165c9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x565377352a20 con 0x565377464680 2024-03-14T10:00:09.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f23165c9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56537723a6a0 con 0x5653774732d0 2024-03-14T10:00:09.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f23165c9b80 10 client.0 _ll_register_callbacks cb 0x5653771be3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:09.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f22fbe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464680 0x565377472c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f2300a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653774732d0 0x5653774705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f2300a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653774732d0 0x5653774705e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35744/0 (socket says 172.21.15.78:35744) 2024-03-14T10:00:09.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f22fbe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464680 0x565377472c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57432/0 (socket says 172.21.15.78:57432) 2024-03-14T10:00:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f22fb400700 1 --2- 172.21.15.78:0/665183315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377464fa0 0x56537746ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f2300a00700 1 -- 172.21.15.78:0/665183315 learned_addr learned my addr 172.21.15.78:0/665183315 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f2300a00700 1 -- 172.21.15.78:0/665183315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464680 msgr2=0x565377472c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f2300a00700 1 --2- 172.21.15.78:0/665183315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464680 0x565377472c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f2300a00700 1 -- 172.21.15.78:0/665183315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377464fa0 msgr2=0x56537746ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.642+0000 7f2300a00700 1 --2- 172.21.15.78:0/665183315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377464fa0 0x56537746ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.643+0000 7f2300a00700 1 -- 172.21.15.78:0/665183315 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f4003bd0 con 0x5653774732d0 2024-03-14T10:00:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.643+0000 7f2300a00700 1 --2- 172.21.15.78:0/665183315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653774732d0 0x5653774705e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f22f4007640 tx=0x7f22f40047a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=233b6f3e191b895c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.643+0000 7f22e5600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:00:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.643+0000 7f22e5600700 1 -- 172.21.15.78:0/665183315 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4019040 con 0x5653774732d0 2024-03-14T10:00:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.643+0000 7f23165c9b80 1 -- 172.21.15.78:0/665183315 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565377457720 con 0x5653774732d0 2024-03-14T10:00:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.643+0000 7f22e5600700 1 -- 172.21.15.78:0/665183315 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f22f4009460 con 0x5653774732d0 2024-03-14T10:00:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.643+0000 7f22e5600700 1 -- 172.21.15.78:0/665183315 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22f4007bf0 con 0x5653774732d0 2024-03-14T10:00:09.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.643+0000 7f23165c9b80 10 client.26848 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.643+0000 7f23165c9b80 1 -- 172.21.15.78:0/665183315 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5653771e4b60 con 0x5653774732d0 2024-03-14T10:00:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.644+0000 7f22e5600700 1 -- 172.21.15.78:0/665183315 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x565377457720 con 0x5653774732d0 2024-03-14T10:00:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.644+0000 7f22e5600700 1 -- 172.21.15.78:0/665183315 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f22f401f450 con 0x5653774732d0 2024-03-14T10:00:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.644+0000 7f23165c9b80 10 client.26848 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.644+0000 7f23165c9b80 10 client.26848 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:00:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.644+0000 7f23165c9b80 20 client.26848 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.644+0000 7f23165c9b80 1 --2- 172.21.15.78:0/665183315 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x565377482db0 0x5653774851a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.644+0000 7f23165c9b80 4 client.26848 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:00:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.644+0000 7f23165c9b80 1 -- 172.21.15.78:0/665183315 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5653773af700 con 0x565377482db0 2024-03-14T10:00:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.644+0000 7f22fbe00700 1 --2- 172.21.15.78:0/665183315 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x565377482db0 0x5653774851a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.645+0000 7f22e5600700 10 client.26848 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:00:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.645+0000 7f22fbe00700 1 --2- 172.21.15.78:0/665183315 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x565377482db0 0x5653774851a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d87ecc99775d1f6e server_cookie=49ee5c0757ff8174 in_seq=0 out_seq=0 2024-03-14T10:00:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.645+0000 7f22e5600700 1 -- 172.21.15.78:0/665183315 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5653773af700 con 0x565377482db0 2024-03-14T10:00:09.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.645+0000 7f22e5600700 10 client.26848 handle_command_reply: tid=0 2024-03-14T10:00:09.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.645+0000 7f23165c9b80 1 client.26848 shutdown 2024-03-14T10:00:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.646+0000 7f23165c9b80 1 -- 172.21.15.78:0/665183315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653774732d0 msgr2=0x5653774705e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.646+0000 7f23165c9b80 1 --2- 172.21.15.78:0/665183315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653774732d0 0x5653774705e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f22f4007640 tx=0x7f22f40047a0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.646+0000 7f23165c9b80 1 -- 172.21.15.78:0/665183315 shutdown_connections 2024-03-14T10:00:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.646+0000 7f23165c9b80 1 --2- 172.21.15.78:0/665183315 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x565377482db0 0x5653774851a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.646+0000 7f23165c9b80 1 --2- 172.21.15.78:0/665183315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464680 0x565377472c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.646+0000 7f23165c9b80 1 --2- 172.21.15.78:0/665183315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5653774732d0 0x5653774705e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.646+0000 7f22e5600700 0 client.26848 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:00:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.646+0000 7f23165c9b80 1 --2- 172.21.15.78:0/665183315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377464fa0 0x56537746ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.646+0000 7f23165c9b80 1 -- 172.21.15.78:0/665183315 >> 172.21.15.78:0/665183315 conn(0x5653773da220 msgr2=0x56537745f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.646+0000 7f23165c9b80 1 -- 172.21.15.78:0/665183315 shutdown_connections 2024-03-14T10:00:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.646+0000 7f23165c9b80 1 -- 172.21.15.78:0/665183315 wait complete. 2024-03-14T10:00:09.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.646+0000 7f23165c9b80 20 client.26848 trim_cache size 0 max 16384 2024-03-14T10:00:09.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.663+0000 7f23165c9b80 1 Processor -- start 2024-03-14T10:00:09.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.663+0000 7f23165c9b80 1 -- start start 2024-03-14T10:00:09.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.663+0000 7f23165c9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377464680 0x5653774727c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.663+0000 7f23165c9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464fa0 0x56537746d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.663+0000 7f23165c9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653774732d0 0x56537746d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.663+0000 7f23165c9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x565377459d30 con 0x565377464680 2024-03-14T10:00:09.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.663+0000 7f23165c9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x565377475a80 con 0x5653774732d0 2024-03-14T10:00:09.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.663+0000 7f23165c9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56537746be10 con 0x565377464fa0 2024-03-14T10:00:09.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.663+0000 7f22fbe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377464680 0x5653774727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.663+0000 7f22fbe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377464680 0x5653774727c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49510/0 (socket says 172.21.15.78:49510) 2024-03-14T10:00:09.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.664+0000 7f22fbe00700 1 -- 172.21.15.78:0/2633815887 learned_addr learned my addr 172.21.15.78:0/2633815887 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:09.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.664+0000 7f22fb400700 1 --2- 172.21.15.78:0/2633815887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464fa0 0x56537746d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.664+0000 7f22fbe00700 1 -- 172.21.15.78:0/2633815887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464fa0 msgr2=0x56537746d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.664+0000 7f22fbe00700 1 --2- 172.21.15.78:0/2633815887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464fa0 0x56537746d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.664+0000 7f22fbe00700 1 -- 172.21.15.78:0/2633815887 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653774732d0 msgr2=0x56537746d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.664+0000 7f22fbe00700 1 --2- 172.21.15.78:0/2633815887 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653774732d0 0x56537746d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.664+0000 7f22fbe00700 1 -- 172.21.15.78:0/2633815887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56537737cea0 con 0x565377464680 2024-03-14T10:00:09.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.664+0000 7f22fb400700 1 --2- 172.21.15.78:0/2633815887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464fa0 0x56537746d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:09.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.664+0000 7f22fbe00700 1 --2- 172.21.15.78:0/2633815887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377464680 0x5653774727c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0098e0 tx=0x7f22ec022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6d299147d42b7b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.665+0000 7f22f1c00700 1 -- 172.21.15.78:0/2633815887 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec01d050 con 0x565377464680 2024-03-14T10:00:09.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.665+0000 7f22f1c00700 1 -- 172.21.15.78:0/2633815887 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22ec01e040 con 0x565377464680 2024-03-14T10:00:09.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.665+0000 7f22f1c00700 1 -- 172.21.15.78:0/2633815887 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec004760 con 0x565377464680 2024-03-14T10:00:09.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.666+0000 7f23165c9b80 1 -- 172.21.15.78:0/2633815887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377464680 msgr2=0x5653774727c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.666+0000 7f23165c9b80 1 --2- 172.21.15.78:0/2633815887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377464680 0x5653774727c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0098e0 tx=0x7f22ec022000 comp rx=0 tx=0).stop 2024-03-14T10:00:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.668+0000 7f23165c9b80 1 -- 172.21.15.78:0/2633815887 shutdown_connections 2024-03-14T10:00:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.668+0000 7f23165c9b80 1 --2- 172.21.15.78:0/2633815887 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5653774732d0 0x56537746d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.668+0000 7f23165c9b80 1 --2- 172.21.15.78:0/2633815887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565377464680 0x5653774727c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.668+0000 7f23165c9b80 1 --2- 172.21.15.78:0/2633815887 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464fa0 0x56537746d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.668+0000 7f23165c9b80 1 -- 172.21.15.78:0/2633815887 >> 172.21.15.78:0/2633815887 conn(0x5653773da220 msgr2=0x56537746e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.668+0000 7f23165c9b80 1 -- 172.21.15.78:0/2633815887 shutdown_connections 2024-03-14T10:00:09.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.668+0000 7f23165c9b80 1 -- 172.21.15.78:0/2633815887 wait complete. 2024-03-14T10:00:09.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.669+0000 7f23165c9b80 1 -- start start 2024-03-14T10:00:09.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.669+0000 7f23165c9b80 1 Processor -- start 2024-03-14T10:00:09.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.669+0000 7f23165c9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464680 0x565377479aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.669+0000 7f23165c9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464fa0 0x565377479fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.670+0000 7f23165c9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5653774732d0 0x565377478120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.670+0000 7f23165c9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x565377478800 con 0x5653774732d0 2024-03-14T10:00:09.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.670+0000 7f23165c9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x565377478940 con 0x565377464fa0 2024-03-14T10:00:09.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.670+0000 7f23165c9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x565377478b00 con 0x565377464680 2024-03-14T10:00:09.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.670+0000 7f23165c9b80 10 client.0 _ll_register_callbacks cb 0x7f22f4003df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:09.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.670+0000 7f22fbe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464680 0x565377479aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.670+0000 7f22fb400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464fa0 0x565377479fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.670+0000 7f22fbe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464680 0x565377479aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35762/0 (socket says 172.21.15.78:35762) 2024-03-14T10:00:09.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.670+0000 7f22fbe00700 1 -- 172.21.15.78:0/1213226035 learned_addr learned my addr 172.21.15.78:0/1213226035 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:09.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.670+0000 7f2300a00700 1 --2- 172.21.15.78:0/1213226035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5653774732d0 0x565377478120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.670+0000 7f22fbe00700 1 -- 172.21.15.78:0/1213226035 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464fa0 msgr2=0x565377479fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.670+0000 7f22fbe00700 1 --2- 172.21.15.78:0/1213226035 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464fa0 0x565377479fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.670+0000 7f22fbe00700 1 -- 172.21.15.78:0/1213226035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5653774732d0 msgr2=0x565377478120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.670+0000 7f22fbe00700 1 --2- 172.21.15.78:0/1213226035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5653774732d0 0x565377478120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.670+0000 7f22fbe00700 1 -- 172.21.15.78:0/1213226035 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22fc1a4770 con 0x565377464680 2024-03-14T10:00:09.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.670+0000 7f22fb400700 1 --2- 172.21.15.78:0/1213226035 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464fa0 0x565377479fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:09.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.671+0000 7f22fbe00700 1 --2- 172.21.15.78:0/1213226035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464680 0x565377479aa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009ce0 tx=0x7f22ec005460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4448990263498d37 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.671+0000 7f22e5600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:00:09.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.671+0000 7f22e5600700 1 -- 172.21.15.78:0/1213226035 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec01d040 con 0x565377464680 2024-03-14T10:00:09.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.671+0000 7f23165c9b80 1 -- 172.21.15.78:0/1213226035 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22f400f040 con 0x565377464680 2024-03-14T10:00:09.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.671+0000 7f22e5600700 1 -- 172.21.15.78:0/1213226035 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f22ec00b6f0 con 0x565377464680 2024-03-14T10:00:09.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.671+0000 7f22e5600700 1 -- 172.21.15.78:0/1213226035 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22ec00b8b0 con 0x565377464680 2024-03-14T10:00:09.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.671+0000 7f23165c9b80 10 client.26854 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:09.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.671+0000 7f23165c9b80 1 -- 172.21.15.78:0/1213226035 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f22fc10e430 con 0x565377464680 2024-03-14T10:00:09.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.671+0000 7f22e5600700 1 -- 172.21.15.78:0/1213226035 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22f400f040 con 0x565377464680 2024-03-14T10:00:09.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.672+0000 7f22e5600700 1 -- 172.21.15.78:0/1213226035 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f22ec03f460 con 0x565377464680 2024-03-14T10:00:09.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.672+0000 7f23165c9b80 10 client.26854 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:09.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.672+0000 7f23165c9b80 10 client.26854 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:00:09.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.672+0000 7f23165c9b80 20 client.26854 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:09.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.672+0000 7f23165c9b80 1 --2- 172.21.15.78:0/1213226035 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5653774d5760 0x5653774d7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.672+0000 7f23165c9b80 4 client.26854 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:00:09.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.672+0000 7f23165c9b80 1 -- 172.21.15.78:0/1213226035 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x565377205cf0 con 0x5653774d5760 2024-03-14T10:00:09.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.672+0000 7f22fb400700 1 --2- 172.21.15.78:0/1213226035 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5653774d5760 0x5653774d7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.673+0000 7f22e5600700 10 client.26854 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:00:09.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.673+0000 7f22fb400700 1 --2- 172.21.15.78:0/1213226035 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5653774d5760 0x5653774d7b80 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f7d0eeef8cad7010 server_cookie=b32deb3fcaaa857c in_seq=0 out_seq=0 2024-03-14T10:00:09.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.673+0000 7f22e5600700 1 -- 172.21.15.78:0/1213226035 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x565377205cf0 con 0x5653774d5760 2024-03-14T10:00:09.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.673+0000 7f22e5600700 10 client.26854 handle_command_reply: tid=0 2024-03-14T10:00:09.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.673+0000 7f23165c9b80 1 client.26854 shutdown 2024-03-14T10:00:09.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.673+0000 7f23165c9b80 1 -- 172.21.15.78:0/1213226035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464680 msgr2=0x565377479aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.673+0000 7f23165c9b80 1 --2- 172.21.15.78:0/1213226035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464680 0x565377479aa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009ce0 tx=0x7f22ec005460 comp rx=0 tx=0).stop 2024-03-14T10:00:09.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.673+0000 7f23165c9b80 1 -- 172.21.15.78:0/1213226035 shutdown_connections 2024-03-14T10:00:09.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.673+0000 7f23165c9b80 1 --2- 172.21.15.78:0/1213226035 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5653774d5760 0x5653774d7b80 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.673+0000 7f23165c9b80 1 --2- 172.21.15.78:0/1213226035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5653774732d0 0x565377478120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.673+0000 7f23165c9b80 1 --2- 172.21.15.78:0/1213226035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x565377464680 0x565377479aa0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.673+0000 7f22e5600700 0 client.26854 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:00:09.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.673+0000 7f23165c9b80 1 --2- 172.21.15.78:0/1213226035 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x565377464fa0 0x565377479fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.673+0000 7f23165c9b80 1 -- 172.21.15.78:0/1213226035 >> 172.21.15.78:0/1213226035 conn(0x5653773da220 msgr2=0x565377463560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.673+0000 7f23165c9b80 1 -- 172.21.15.78:0/1213226035 shutdown_connections 2024-03-14T10:00:09.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.673+0000 7f23165c9b80 1 -- 172.21.15.78:0/1213226035 wait complete. 2024-03-14T10:00:09.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.674+0000 7f23165c9b80 20 client.26854 trim_cache size 0 max 16384 2024-03-14T10:00:09.677 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:00:09.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.674+0000 7f2302a00700 1 -- 172.21.15.78:0/2015288700 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f22e00a0c80 msgr2=0x7f22e00a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.674+0000 7f2302a00700 1 --2- 172.21.15.78:0/2015288700 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f22e00a0c80 0x7f22e00a3140 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f22ec00b5c0 tx=0x7f22ec005fd0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.674+0000 7f2302a00700 1 -- 172.21.15.78:0/2015288700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22fc106040 msgr2=0x7f22fc197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.674+0000 7f2302a00700 1 --2- 172.21.15.78:0/2015288700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22fc106040 0x7f22fc197be0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f22e800d950 tx=0x7f22e800dc60 comp rx=0 tx=0).stop 2024-03-14T10:00:09.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.675+0000 7f2302a00700 1 -- 172.21.15.78:0/2015288700 shutdown_connections 2024-03-14T10:00:09.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.675+0000 7f2302a00700 1 --2- 172.21.15.78:0/2015288700 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f22e00a0c80 0x7f22e00a3140 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.675+0000 7f2302a00700 1 --2- 172.21.15.78:0/2015288700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f22fc19bf30 0x7f22fc19c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.675+0000 7f2302a00700 1 --2- 172.21.15.78:0/2015288700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22fc100f20 0x7f22fc1976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.675+0000 7f2302a00700 1 --2- 172.21.15.78:0/2015288700 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f22fc106040 0x7f22fc197be0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.675+0000 7f2302a00700 1 -- 172.21.15.78:0/2015288700 >> 172.21.15.78:0/2015288700 conn(0x7f22fc0fc7a0 msgr2=0x7f22fc0fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.675+0000 7f2302a00700 1 -- 172.21.15.78:0/2015288700 shutdown_connections 2024-03-14T10:00:09.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.675+0000 7f2302a00700 1 -- 172.21.15.78:0/2015288700 wait complete. 2024-03-14T10:00:09.704 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T10:00:09.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.953+0000 7f4fd6c00700 1 -- 172.21.15.78:0/1917996734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4fd00731e0 msgr2=0x7f4fd00735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.953+0000 7f4fd6c00700 1 --2- 172.21.15.78:0/1917996734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4fd00731e0 0x7f4fd00735c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0009990 tx=0x7f4fc0009ca0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.953+0000 7f4fd6c00700 1 -- 172.21.15.78:0/1917996734 shutdown_connections 2024-03-14T10:00:09.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.953+0000 7f4fd6c00700 1 --2- 172.21.15.78:0/1917996734 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fd010d500 0x7f4fd010f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.953+0000 7f4fd6c00700 1 --2- 172.21.15.78:0/1917996734 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4fd00731e0 0x7f4fd00735c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.953+0000 7f4fd6c00700 1 --2- 172.21.15.78:0/1917996734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd0073b00 0x7f4fd010cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.953+0000 7f4fd6c00700 1 -- 172.21.15.78:0/1917996734 >> 172.21.15.78:0/1917996734 conn(0x7f4fd00fc7b0 msgr2=0x7f4fd00febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.953+0000 7f4fd6c00700 1 -- 172.21.15.78:0/1917996734 shutdown_connections 2024-03-14T10:00:09.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.953+0000 7f4fd6c00700 1 -- 172.21.15.78:0/1917996734 wait complete. 2024-03-14T10:00:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.954+0000 7f4fd6c00700 1 Processor -- start 2024-03-14T10:00:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.954+0000 7f4fd6c00700 1 -- start start 2024-03-14T10:00:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fd6c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4fd00731e0 0x7f4fd01997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fd6c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fd0073b00 0x7f4fd0199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fd6c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd010d500 0x7f4fd019d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fd6c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd019d610 con 0x7f4fd010d500 2024-03-14T10:00:09.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fd6c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd019d780 con 0x7f4fd0073b00 2024-03-14T10:00:09.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fd6c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd019d940 con 0x7f4fd00731e0 2024-03-14T10:00:09.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fcfe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4fd00731e0 0x7f4fd01997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fcf400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fd0073b00 0x7f4fd0199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fcfe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4fd00731e0 0x7f4fd01997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35776/0 (socket says 172.21.15.78:35776) 2024-03-14T10:00:09.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fcfe00700 1 -- 172.21.15.78:0/286182871 learned_addr learned my addr 172.21.15.78:0/286182871 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:09.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fd4c00700 1 --2- 172.21.15.78:0/286182871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd010d500 0x7f4fd019d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fcfe00700 1 -- 172.21.15.78:0/286182871 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fd0073b00 msgr2=0x7f4fd0199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fcfe00700 1 --2- 172.21.15.78:0/286182871 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fd0073b00 0x7f4fd0199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fcfe00700 1 -- 172.21.15.78:0/286182871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd010d500 msgr2=0x7f4fd019d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fcfe00700 1 --2- 172.21.15.78:0/286182871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd010d500 0x7f4fd019d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fcfe00700 1 -- 172.21.15.78:0/286182871 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc00095f0 con 0x7f4fd00731e0 2024-03-14T10:00:09.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.955+0000 7f4fd4c00700 1 --2- 172.21.15.78:0/286182871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd010d500 0x7f4fd019d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:00:09.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.956+0000 7f4fcfe00700 1 --2- 172.21.15.78:0/286182871 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4fd00731e0 0x7f4fd01997e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0005910 tx=0x7f4fc0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.956+0000 7f4fccc00700 1 -- 172.21.15.78:0/286182871 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc001d070 con 0x7f4fd00731e0 2024-03-14T10:00:09.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.956+0000 7f4fccc00700 1 -- 172.21.15.78:0/286182871 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4fc0022470 con 0x7f4fd00731e0 2024-03-14T10:00:09.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.956+0000 7f4fccc00700 1 -- 172.21.15.78:0/286182871 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc0017650 con 0x7f4fd00731e0 2024-03-14T10:00:09.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.956+0000 7f4fd6c00700 1 -- 172.21.15.78:0/286182871 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fd019dbc0 con 0x7f4fd00731e0 2024-03-14T10:00:09.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.956+0000 7f4fd6c00700 1 -- 172.21.15.78:0/286182871 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fd01a63a0 con 0x7f4fd00731e0 2024-03-14T10:00:09.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.960+0000 7f4fccc00700 1 -- 172.21.15.78:0/286182871 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f4fc00225e0 con 0x7f4fd00731e0 2024-03-14T10:00:09.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.960+0000 7f4fea739b80 1 Processor -- start 2024-03-14T10:00:09.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.961+0000 7f4fea739b80 1 -- start start 2024-03-14T10:00:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.961+0000 7f4fea739b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2ab680 0x56130c2aba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.961+0000 7f4fea739b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 0x56130c2b5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.961+0000 7f4fea739b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2b5ec0 0x56130c2b82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.961+0000 7f4fea739b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56130c204990 con 0x56130c2b5ec0 2024-03-14T10:00:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.961+0000 7f4fea739b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56130c202350 con 0x56130c2abfa0 2024-03-14T10:00:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.961+0000 7f4fea739b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56130c1fdc40 con 0x56130c2ab680 2024-03-14T10:00:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.961+0000 7f4fd4c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2ab680 0x56130c2aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.961+0000 7f4fcf400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2b5ec0 0x56130c2b82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.961+0000 7f4fcf400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2b5ec0 0x56130c2b82b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49546/0 (socket says 172.21.15.78:49546) 2024-03-14T10:00:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.961+0000 7f4fcf400700 1 -- 172.21.15.78:0/56072432 learned_addr learned my addr 172.21.15.78:0/56072432 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:09.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.962+0000 7f4fcfe00700 1 --2- 172.21.15.78:0/56072432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 0x56130c2b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.962+0000 7f4fd4c00700 1 -- 172.21.15.78:0/56072432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 msgr2=0x56130c2b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.962+0000 7f4fd4c00700 1 --2- 172.21.15.78:0/56072432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 0x56130c2b5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.962+0000 7f4fd4c00700 1 -- 172.21.15.78:0/56072432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2b5ec0 msgr2=0x56130c2b82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.962+0000 7f4fd4c00700 1 --2- 172.21.15.78:0/56072432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2b5ec0 0x56130c2b82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.962+0000 7f4fd4c00700 1 -- 172.21.15.78:0/56072432 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56130c07f3b0 con 0x56130c2ab680 2024-03-14T10:00:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.962+0000 7f4fcf400700 1 --2- 172.21.15.78:0/56072432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2b5ec0 0x56130c2b82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.962+0000 7f4fccc00700 1 --2- 172.21.15.78:0/286182871 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4fb40a0c00 0x7f4fb40a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.962+0000 7f4fd4c00700 1 --2- 172.21.15.78:0/56072432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2ab680 0x56130c2aba60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8009d10 tx=0x7f4fc80077e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d387057ebcdafd2d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.962+0000 7f4fccc00700 1 -- 172.21.15.78:0/286182871 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f4fc00cdcd0 con 0x7f4fd00731e0 2024-03-14T10:00:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.962+0000 7f4fcfe00700 1 --2- 172.21.15.78:0/286182871 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4fb40a0c00 0x7f4fb40a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.963+0000 7f4fc5200700 1 -- 172.21.15.78:0/56072432 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc8018070 con 0x56130c2ab680 2024-03-14T10:00:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.963+0000 7f4fc5200700 1 -- 172.21.15.78:0/56072432 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4fc800f050 con 0x56130c2ab680 2024-03-14T10:00:09.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.963+0000 7f4fc5200700 1 -- 172.21.15.78:0/56072432 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc8005900 con 0x56130c2ab680 2024-03-14T10:00:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.963+0000 7f4fea739b80 1 -- 172.21.15.78:0/56072432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2ab680 msgr2=0x56130c2aba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.963+0000 7f4fcfe00700 1 --2- 172.21.15.78:0/286182871 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4fb40a0c00 0x7f4fb40a30c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4fc00cb110 tx=0x7f4fc0022f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.963+0000 7f4fea739b80 1 --2- 172.21.15.78:0/56072432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2ab680 0x56130c2aba60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8009d10 tx=0x7f4fc80077e0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.964+0000 7f4fea739b80 1 -- 172.21.15.78:0/56072432 shutdown_connections 2024-03-14T10:00:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.964+0000 7f4fea739b80 1 --2- 172.21.15.78:0/56072432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2b5ec0 0x56130c2b82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.964+0000 7f4fea739b80 1 --2- 172.21.15.78:0/56072432 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2ab680 0x56130c2aba60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.964+0000 7f4fea739b80 1 --2- 172.21.15.78:0/56072432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 0x56130c2b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.964+0000 7f4fea739b80 1 -- 172.21.15.78:0/56072432 >> 172.21.15.78:0/56072432 conn(0x56130c221220 msgr2=0x56130c2a6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.964+0000 7f4fea739b80 1 -- 172.21.15.78:0/56072432 shutdown_connections 2024-03-14T10:00:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.964+0000 7f4fea739b80 1 -- 172.21.15.78:0/56072432 wait complete. 2024-03-14T10:00:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.964+0000 7f4fea739b80 1 -- start start 2024-03-14T10:00:09.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.964+0000 7f4fea739b80 1 Processor -- start 2024-03-14T10:00:09.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.965+0000 7f4fea739b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2ab680 0x56130c2b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.965+0000 7f4fea739b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2abfa0 0x56130c2b2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.965+0000 7f4fea739b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2b5ec0 0x56130c2b3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.965+0000 7f4fea739b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56130c199d80 con 0x56130c2abfa0 2024-03-14T10:00:09.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.965+0000 7f4fea739b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56130c199a20 con 0x56130c2ab680 2024-03-14T10:00:09.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.965+0000 7f4fea739b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56130c0816a0 con 0x56130c2b5ec0 2024-03-14T10:00:09.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.965+0000 7f4fea739b80 10 client.0 _ll_register_callbacks cb 0x56130c0053a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:09.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.965+0000 7f4fcf400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2abfa0 0x56130c2b2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.965+0000 7f4fcf400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2abfa0 0x56130c2b2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49550/0 (socket says 172.21.15.78:49550) 2024-03-14T10:00:09.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.965+0000 7f4fcf400700 1 -- 172.21.15.78:0/3299412018 learned_addr learned my addr 172.21.15.78:0/3299412018 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:09.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.965+0000 7f4fd4c00700 1 --2- 172.21.15.78:0/3299412018 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2ab680 0x56130c2b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.965+0000 7f4fd4c00700 1 --2- 172.21.15.78:0/3299412018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2b5ec0 0x56130c2b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.966+0000 7f4fcf400700 1 -- 172.21.15.78:0/3299412018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2b5ec0 msgr2=0x56130c2b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.966+0000 7f4fcf400700 1 --2- 172.21.15.78:0/3299412018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2b5ec0 0x56130c2b3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.966+0000 7f4fcf400700 1 -- 172.21.15.78:0/3299412018 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2ab680 msgr2=0x56130c2b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.966+0000 7f4fcf400700 1 --2- 172.21.15.78:0/3299412018 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2ab680 0x56130c2b58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.966+0000 7f4fcf400700 1 -- 172.21.15.78:0/3299412018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fbc009920 con 0x56130c2abfa0 2024-03-14T10:00:09.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.966+0000 7f4fd4c00700 1 --2- 172.21.15.78:0/3299412018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2b5ec0 0x56130c2b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T10:00:09.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.966+0000 7f4fcf400700 1 --2- 172.21.15.78:0/3299412018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2abfa0 0x56130c2b2c30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc0119d0 tx=0x7f4fbc011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd22b855f08a6a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.966+0000 7f4fb9600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:00:09.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.966+0000 7f4fb9600700 1 -- 172.21.15.78:0/3299412018 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc016070 con 0x56130c2abfa0 2024-03-14T10:00:09.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.966+0000 7f4fb9600700 1 -- 172.21.15.78:0/3299412018 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f4fbc00fb00 con 0x56130c2abfa0 2024-03-14T10:00:09.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.966+0000 7f4fea739b80 1 -- 172.21.15.78:0/3299412018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56130c29e720 con 0x56130c2abfa0 2024-03-14T10:00:09.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.967+0000 7f4fb9600700 1 -- 172.21.15.78:0/3299412018 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fbc00fcc0 con 0x56130c2abfa0 2024-03-14T10:00:09.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.967+0000 7f4fea739b80 10 client.16803 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:09.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.967+0000 7f4fea739b80 1 -- 172.21.15.78:0/3299412018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f4fc8009940 con 0x56130c2abfa0 2024-03-14T10:00:09.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.967+0000 7f4fb9600700 1 -- 172.21.15.78:0/3299412018 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56130c29e720 con 0x56130c2abfa0 2024-03-14T10:00:09.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.967+0000 7f4fb9600700 1 -- 172.21.15.78:0/3299412018 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4fbc019390 con 0x56130c2abfa0 2024-03-14T10:00:09.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.967+0000 7f4fea739b80 10 client.16803 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:09.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.967+0000 7f4fea739b80 10 client.16803 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:00:09.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.967+0000 7f4fea739b80 20 client.16803 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:09.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.967+0000 7f4fea739b80 1 --2- 172.21.15.78:0/3299412018 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56130c2c5b40 0x56130c2c7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.967+0000 7f4fea739b80 4 client.16803 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:00:09.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.967+0000 7f4fea739b80 1 -- 172.21.15.78:0/3299412018 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56130c1f6700 con 0x56130c2c5b40 2024-03-14T10:00:09.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.968+0000 7f4fd4c00700 1 --2- 172.21.15.78:0/3299412018 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56130c2c5b40 0x56130c2c7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.968+0000 7f4fd4c00700 1 --2- 172.21.15.78:0/3299412018 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56130c2c5b40 0x56130c2c7f60 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d65929af961e4796 server_cookie=416ce72923c6168f in_seq=0 out_seq=0 2024-03-14T10:00:09.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.968+0000 7f4fb9600700 10 client.16803 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:09.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.969+0000 7f4fb9600700 1 -- 172.21.15.78:0/3299412018 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56130c1f6700 con 0x56130c2c5b40 2024-03-14T10:00:09.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.969+0000 7f4fb9600700 10 client.16803 handle_command_reply: tid=0 2024-03-14T10:00:09.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.969+0000 7f4fea739b80 1 client.16803 shutdown 2024-03-14T10:00:09.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.969+0000 7f4fea739b80 1 -- 172.21.15.78:0/3299412018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2abfa0 msgr2=0x56130c2b2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.969+0000 7f4fea739b80 1 --2- 172.21.15.78:0/3299412018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2abfa0 0x56130c2b2c30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc0119d0 tx=0x7f4fbc011d90 comp rx=0 tx=0).stop 2024-03-14T10:00:09.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.969+0000 7f4fea739b80 1 -- 172.21.15.78:0/3299412018 shutdown_connections 2024-03-14T10:00:09.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.969+0000 7f4fea739b80 1 --2- 172.21.15.78:0/3299412018 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56130c2c5b40 0x56130c2c7f60 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.969+0000 7f4fea739b80 1 --2- 172.21.15.78:0/3299412018 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2ab680 0x56130c2b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.969+0000 7f4fea739b80 1 --2- 172.21.15.78:0/3299412018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2b5ec0 0x56130c2b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.969+0000 7f4fea739b80 1 --2- 172.21.15.78:0/3299412018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2abfa0 0x56130c2b2c30 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.969+0000 7f4fb9600700 0 client.16803 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:09.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.969+0000 7f4fea739b80 1 -- 172.21.15.78:0/3299412018 >> 172.21.15.78:0/3299412018 conn(0x56130c221220 msgr2=0x56130c2a6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.970+0000 7f4fea739b80 1 -- 172.21.15.78:0/3299412018 shutdown_connections 2024-03-14T10:00:09.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.970+0000 7f4fea739b80 1 -- 172.21.15.78:0/3299412018 wait complete. 2024-03-14T10:00:09.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.970+0000 7f4fea739b80 20 client.16803 trim_cache size 0 max 16384 2024-03-14T10:00:09.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.991+0000 7f4fea739b80 1 Processor -- start 2024-03-14T10:00:09.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.991+0000 7f4fea739b80 1 -- start start 2024-03-14T10:00:09.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.992+0000 7f4fea739b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2ab680 0x56130c2cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.992+0000 7f4fea739b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 0x56130c2bcb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.992+0000 7f4fea739b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2b5ec0 0x56130c2a5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.992+0000 7f4fea739b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56130c2b3b80 con 0x56130c2ab680 2024-03-14T10:00:09.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.992+0000 7f4fea739b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56130c2a7670 con 0x56130c2abfa0 2024-03-14T10:00:09.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.992+0000 7f4fea739b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56130c2acaf0 con 0x56130c2b5ec0 2024-03-14T10:00:09.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.992+0000 7f4fd4c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2ab680 0x56130c2cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.992+0000 7f4fcf400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 0x56130c2bcb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.992+0000 7f4fcf400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 0x56130c2bcb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57502/0 (socket says 172.21.15.78:57502) 2024-03-14T10:00:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.992+0000 7f4fcf400700 1 -- 172.21.15.78:0/1392082232 learned_addr learned my addr 172.21.15.78:0/1392082232 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.992+0000 7f4fcf400700 1 -- 172.21.15.78:0/1392082232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2b5ec0 msgr2=0x56130c2a5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.992+0000 7f4fcf400700 1 --2- 172.21.15.78:0/1392082232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2b5ec0 0x56130c2a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.993+0000 7f4fcf400700 1 -- 172.21.15.78:0/1392082232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2ab680 msgr2=0x56130c2cc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.993+0000 7f4fcf400700 1 --2- 172.21.15.78:0/1392082232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2ab680 0x56130c2cc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.993+0000 7f4fcf400700 1 -- 172.21.15.78:0/1392082232 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56130c02bb60 con 0x56130c2abfa0 2024-03-14T10:00:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.993+0000 7f4fcf400700 1 --2- 172.21.15.78:0/1392082232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 0x56130c2bcb50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc019770 tx=0x7f4fbc009ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=306a58e86a3d61ab server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.993+0000 7f4fc5c00700 1 -- 172.21.15.78:0/1392082232 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc016070 con 0x56130c2abfa0 2024-03-14T10:00:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.993+0000 7f4fc5c00700 1 -- 172.21.15.78:0/1392082232 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4fbc019410 con 0x56130c2abfa0 2024-03-14T10:00:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.993+0000 7f4fc5c00700 1 -- 172.21.15.78:0/1392082232 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc005350 con 0x56130c2abfa0 2024-03-14T10:00:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.994+0000 7f4fea739b80 1 -- 172.21.15.78:0/1392082232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 msgr2=0x56130c2bcb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.994+0000 7f4fea739b80 1 --2- 172.21.15.78:0/1392082232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 0x56130c2bcb50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc019770 tx=0x7f4fbc009ab0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.994+0000 7f4fea739b80 1 -- 172.21.15.78:0/1392082232 shutdown_connections 2024-03-14T10:00:09.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.994+0000 7f4fea739b80 1 --2- 172.21.15.78:0/1392082232 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2b5ec0 0x56130c2a5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.994+0000 7f4fea739b80 1 --2- 172.21.15.78:0/1392082232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2ab680 0x56130c2cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.994+0000 7f4fea739b80 1 --2- 172.21.15.78:0/1392082232 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 0x56130c2bcb50 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:09.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.994+0000 7f4fea739b80 1 -- 172.21.15.78:0/1392082232 >> 172.21.15.78:0/1392082232 conn(0x56130c221220 msgr2=0x56130c2a2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:09.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.994+0000 7f4fea739b80 1 -- 172.21.15.78:0/1392082232 shutdown_connections 2024-03-14T10:00:09.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.994+0000 7f4fea739b80 1 -- 172.21.15.78:0/1392082232 wait complete. 2024-03-14T10:00:09.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.994+0000 7f4fea739b80 1 -- start start 2024-03-14T10:00:09.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.995+0000 7f4fea739b80 1 Processor -- start 2024-03-14T10:00:09.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.995+0000 7f4fea739b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2ab680 0x56130c2bc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.995+0000 7f4fea739b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 0x56130c2bc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.995+0000 7f4fea739b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2b5ec0 0x56130c2bab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:09.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.995+0000 7f4fea739b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56130c2bb110 con 0x56130c2ab680 2024-03-14T10:00:09.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.995+0000 7f4fea739b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56130c2bb280 con 0x56130c2abfa0 2024-03-14T10:00:09.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.995+0000 7f4fea739b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56130c2bb440 con 0x56130c2b5ec0 2024-03-14T10:00:09.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.995+0000 7f4fea739b80 10 client.0 _ll_register_callbacks cb 0x7f4fbc0114d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:09.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.996+0000 7f4fd4c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2ab680 0x56130c2bc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.996+0000 7f4fcf400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 0x56130c2bc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:09.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.996+0000 7f4fd4c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2ab680 0x56130c2bc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49570/0 (socket says 172.21.15.78:49570) 2024-03-14T10:00:09.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.996+0000 7f4fd4c00700 1 -- 172.21.15.78:0/2502945952 learned_addr learned my addr 172.21.15.78:0/2502945952 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:09.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.996+0000 7f4fd4c00700 1 --2- 172.21.15.78:0/2502945952 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2b5ec0 0x56130c2bab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.996+0000 7f4fcf400700 1 -- 172.21.15.78:0/2502945952 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2b5ec0 msgr2=0x56130c2bab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.996+0000 7f4fcf400700 1 --2- 172.21.15.78:0/2502945952 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2b5ec0 0x56130c2bab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.996+0000 7f4fcf400700 1 -- 172.21.15.78:0/2502945952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2ab680 msgr2=0x56130c2bc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.996+0000 7f4fcf400700 1 --2- 172.21.15.78:0/2502945952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2ab680 0x56130c2bc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.996+0000 7f4fcf400700 1 -- 172.21.15.78:0/2502945952 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc8009940 con 0x56130c2abfa0 2024-03-14T10:00:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.996+0000 7f4fcf400700 1 --2- 172.21.15.78:0/2502945952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 0x56130c2bc940 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00f7a0 tx=0x7f4fbc019b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdf289b6a499e4e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.996+0000 7f4fb9600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T10:00:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.997+0000 7f4fb9600700 1 -- 172.21.15.78:0/2502945952 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc016070 con 0x56130c2abfa0 2024-03-14T10:00:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.997+0000 7f4fea739b80 1 -- 172.21.15.78:0/2502945952 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4fc800f050 con 0x56130c2abfa0 2024-03-14T10:00:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.997+0000 7f4fb9600700 1 -- 172.21.15.78:0/2502945952 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f4fbc000e40 con 0x56130c2abfa0 2024-03-14T10:00:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.997+0000 7f4fb9600700 1 -- 172.21.15.78:0/2502945952 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fbc00fd00 con 0x56130c2abfa0 2024-03-14T10:00:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.997+0000 7f4fea739b80 10 client.26882 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.997+0000 7f4fea739b80 1 -- 172.21.15.78:0/2502945952 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56130c1c3ea0 con 0x56130c2abfa0 2024-03-14T10:00:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.997+0000 7f4fb9600700 1 -- 172.21.15.78:0/2502945952 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4fc800f050 con 0x56130c2abfa0 2024-03-14T10:00:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.997+0000 7f4fb9600700 1 -- 172.21.15.78:0/2502945952 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4fbc003810 con 0x56130c2abfa0 2024-03-14T10:00:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.998+0000 7f4fea739b80 10 client.26882 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.998+0000 7f4fea739b80 10 client.26882 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:00:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.998+0000 7f4fea739b80 20 client.26882 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.998+0000 7f4fea739b80 1 --2- 172.21.15.78:0/2502945952 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56130c3185e0 0x56130c31a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.998+0000 7f4fea739b80 4 client.26882 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:00:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.998+0000 7f4fea739b80 1 -- 172.21.15.78:0/2502945952 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56130c04ccf0 con 0x56130c3185e0 2024-03-14T10:00:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.998+0000 7f4fd4c00700 1 --2- 172.21.15.78:0/2502945952 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56130c3185e0 0x56130c31a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.998+0000 7f4fb9600700 10 client.26882 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:10.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.998+0000 7f4fd4c00700 1 --2- 172.21.15.78:0/2502945952 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56130c3185e0 0x56130c31a9d0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29eae7fe529eb7b7 server_cookie=a0a3651c8bfb81b4 in_seq=0 out_seq=0 2024-03-14T10:00:10.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.998+0000 7f4fb9600700 1 -- 172.21.15.78:0/2502945952 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56130c04ccf0 con 0x56130c3185e0 2024-03-14T10:00:10.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.998+0000 7f4fb9600700 10 client.26882 handle_command_reply: tid=0 2024-03-14T10:00:10.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.999+0000 7f4fea739b80 1 client.26882 shutdown 2024-03-14T10:00:10.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.999+0000 7f4fea739b80 1 -- 172.21.15.78:0/2502945952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 msgr2=0x56130c2bc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:10.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.999+0000 7f4fea739b80 1 --2- 172.21.15.78:0/2502945952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 0x56130c2bc940 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00f7a0 tx=0x7f4fbc019b20 comp rx=0 tx=0).stop 2024-03-14T10:00:10.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.999+0000 7f4fea739b80 1 -- 172.21.15.78:0/2502945952 shutdown_connections 2024-03-14T10:00:10.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.999+0000 7f4fea739b80 1 --2- 172.21.15.78:0/2502945952 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56130c2b5ec0 0x56130c2bab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:10.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.999+0000 7f4fea739b80 1 --2- 172.21.15.78:0/2502945952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56130c2ab680 0x56130c2bc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:10.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.999+0000 7f4fea739b80 1 --2- 172.21.15.78:0/2502945952 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56130c3185e0 0x56130c31a9d0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:10.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.999+0000 7f4fea739b80 1 --2- 172.21.15.78:0/2502945952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130c2abfa0 0x56130c2bc940 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:10.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.999+0000 7f4fb9600700 0 client.26882 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:10.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.999+0000 7f4fea739b80 1 -- 172.21.15.78:0/2502945952 >> 172.21.15.78:0/2502945952 conn(0x56130c221220 msgr2=0x56130c2adf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:10.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.999+0000 7f4fea739b80 1 -- 172.21.15.78:0/2502945952 shutdown_connections 2024-03-14T10:00:10.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.999+0000 7f4fea739b80 1 -- 172.21.15.78:0/2502945952 wait complete. 2024-03-14T10:00:10.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:09.999+0000 7f4fea739b80 20 client.26882 trim_cache size 0 max 16384 2024-03-14T10:00:10.004 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:00:10.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:10.000+0000 7f4fd6c00700 1 -- 172.21.15.78:0/286182871 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4fb40a0c00 msgr2=0x7f4fb40a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:10.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:10.000+0000 7f4fd6c00700 1 --2- 172.21.15.78:0/286182871 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4fb40a0c00 0x7f4fb40a30c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4fc00cb110 tx=0x7f4fc0022f90 comp rx=0 tx=0).stop 2024-03-14T10:00:10.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:10.000+0000 7f4fd6c00700 1 -- 172.21.15.78:0/286182871 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4fd00731e0 msgr2=0x7f4fd01997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:10.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:10.000+0000 7f4fd6c00700 1 --2- 172.21.15.78:0/286182871 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4fd00731e0 0x7f4fd01997e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0005910 tx=0x7f4fc0004730 comp rx=0 tx=0).stop 2024-03-14T10:00:10.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:10.000+0000 7f4fd6c00700 1 -- 172.21.15.78:0/286182871 shutdown_connections 2024-03-14T10:00:10.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:10.000+0000 7f4fd6c00700 1 --2- 172.21.15.78:0/286182871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4fd010d500 0x7f4fd019d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:10.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:10.000+0000 7f4fd6c00700 1 --2- 172.21.15.78:0/286182871 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4fd00731e0 0x7f4fd01997e0 secure :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0005910 tx=0x7f4fc0004730 comp rx=0 tx=0).stop 2024-03-14T10:00:10.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:10.000+0000 7f4fd6c00700 1 --2- 172.21.15.78:0/286182871 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4fb40a0c00 0x7f4fb40a30c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:10.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:10.000+0000 7f4fd6c00700 1 --2- 172.21.15.78:0/286182871 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4fd0073b00 0x7f4fd0199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:10.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:10.000+0000 7f4fd6c00700 1 -- 172.21.15.78:0/286182871 >> 172.21.15.78:0/286182871 conn(0x7f4fd00fc7b0 msgr2=0x7f4fd00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:10.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:10.000+0000 7f4fd6c00700 1 -- 172.21.15.78:0/286182871 shutdown_connections 2024-03-14T10:00:10.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:10.000+0000 7f4fd6c00700 1 -- 172.21.15.78:0/286182871 wait complete. 2024-03-14T10:00:10.020 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:09 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3811599005' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:00:10.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:09 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3811599005' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:00:10.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:09 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3811599005' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:00:11.023 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T10:00:11.023 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T10:00:11.095 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:10 smithi078 ceph-mon[38143]: pgmap v183: 129 pgs: 129 active+clean; 240 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.1 MiB/s wr, 380 op/s 2024-03-14T10:00:11.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:10 smithi152 ceph-mon[48279]: pgmap v183: 129 pgs: 129 active+clean; 240 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.1 MiB/s wr, 380 op/s 2024-03-14T10:00:11.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.266+0000 7f3136600700 1 -- 172.21.15.78:0/2054358045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3130100f20 msgr2=0x7f3130101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.266+0000 7f3136600700 1 --2- 172.21.15.78:0/2054358045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3130100f20 0x7f3130101300 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3120009990 tx=0x7f3120009ca0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.267+0000 7f3136600700 1 -- 172.21.15.78:0/2054358045 shutdown_connections 2024-03-14T10:00:11.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.267+0000 7f3136600700 1 --2- 172.21.15.78:0/2054358045 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3130106040 0x7f313010dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.267+0000 7f3136600700 1 --2- 172.21.15.78:0/2054358045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3130100f20 0x7f3130101300 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.267+0000 7f3136600700 1 --2- 172.21.15.78:0/2054358045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31301018d0 0x7f3130105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.267+0000 7f3136600700 1 -- 172.21.15.78:0/2054358045 >> 172.21.15.78:0/2054358045 conn(0x7f31300fc780 msgr2=0x7f31300feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:11.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.267+0000 7f3136600700 1 -- 172.21.15.78:0/2054358045 shutdown_connections 2024-03-14T10:00:11.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.267+0000 7f3136600700 1 -- 172.21.15.78:0/2054358045 wait complete. 2024-03-14T10:00:11.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.268+0000 7f3136600700 1 Processor -- start 2024-03-14T10:00:11.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.268+0000 7f3136600700 1 -- start start 2024-03-14T10:00:11.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.268+0000 7f3136600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3130100f20 0x7f313019f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.268+0000 7f3136600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3130106040 0x7f313019fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.268+0000 7f3136600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3130199680 0x7f3130199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.268+0000 7f3136600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313019a100 con 0x7f3130106040 2024-03-14T10:00:11.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.268+0000 7f3136600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313019a270 con 0x7f3130100f20 2024-03-14T10:00:11.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.268+0000 7f3136600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313019a430 con 0x7f3130199680 2024-03-14T10:00:11.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.268+0000 7f312ea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3130106040 0x7f313019fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.268+0000 7f312ea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3130106040 0x7f313019fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49584/0 (socket says 172.21.15.78:49584) 2024-03-14T10:00:11.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.268+0000 7f312ea00700 1 -- 172.21.15.78:0/444413827 learned_addr learned my addr 172.21.15.78:0/444413827 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:11.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.269+0000 7f312f400700 1 --2- 172.21.15.78:0/444413827 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3130100f20 0x7f313019f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.269+0000 7f312fe00700 1 --2- 172.21.15.78:0/444413827 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3130199680 0x7f3130199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.269+0000 7f312ea00700 1 -- 172.21.15.78:0/444413827 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3130199680 msgr2=0x7f3130199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.269+0000 7f312ea00700 1 --2- 172.21.15.78:0/444413827 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3130199680 0x7f3130199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.269+0000 7f312ea00700 1 -- 172.21.15.78:0/444413827 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3130100f20 msgr2=0x7f313019f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.269+0000 7f312ea00700 1 --2- 172.21.15.78:0/444413827 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3130100f20 0x7f313019f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.269+0000 7f312ea00700 1 -- 172.21.15.78:0/444413827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31200095f0 con 0x7f3130106040 2024-03-14T10:00:11.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.269+0000 7f312f400700 1 --2- 172.21.15.78:0/444413827 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3130100f20 0x7f313019f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:11.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.269+0000 7f312ea00700 1 --2- 172.21.15.78:0/444413827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3130106040 0x7f313019fa90 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f311c00e970 tx=0x7f311c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:11.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.269+0000 7f3127e00700 1 -- 172.21.15.78:0/444413827 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c009790 con 0x7f3130106040 2024-03-14T10:00:11.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.269+0000 7f3127e00700 1 -- 172.21.15.78:0/444413827 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f311c00c7e0 con 0x7f3130106040 2024-03-14T10:00:11.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.269+0000 7f3136600700 1 -- 172.21.15.78:0/444413827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31301a6420 con 0x7f3130106040 2024-03-14T10:00:11.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.269+0000 7f3127e00700 1 -- 172.21.15.78:0/444413827 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c0054e0 con 0x7f3130106040 2024-03-14T10:00:11.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.270+0000 7f3136600700 1 -- 172.21.15.78:0/444413827 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31301a6900 con 0x7f3130106040 2024-03-14T10:00:11.272 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:10 smithi132 ceph-mon[47959]: pgmap v183: 129 pgs: 129 active+clean; 240 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 1.1 MiB/s wr, 380 op/s 2024-03-14T10:00:11.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.273+0000 7f3127e00700 1 -- 172.21.15.78:0/444413827 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f311c00c950 con 0x7f3130106040 2024-03-14T10:00:11.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.275+0000 7f314a0f8b80 1 Processor -- start 2024-03-14T10:00:11.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.275+0000 7f314a0f8b80 1 -- start start 2024-03-14T10:00:11.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.275+0000 7f314a0f8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc01700 0x56383cc01ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.275+0000 7f314a0f8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc02020 0x56383cc0ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.275+0000 7f314a0f8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc0bf40 0x56383cc0e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.275+0000 7f314a0f8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56383cb5aa10 con 0x56383cc0bf40 2024-03-14T10:00:11.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.275+0000 7f3127e00700 1 --2- 172.21.15.78:0/444413827 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f31140a0c00 0x7f31140a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.275+0000 7f314a0f8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56383cb583d0 con 0x56383cc02020 2024-03-14T10:00:11.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.275+0000 7f314a0f8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56383cb53cc0 con 0x56383cc01700 2024-03-14T10:00:11.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.275+0000 7f3127e00700 1 -- 172.21.15.78:0/444413827 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f311c014070 con 0x7f3130106040 2024-03-14T10:00:11.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.277+0000 7f312f400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc02020 0x56383cc0ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.277+0000 7f312f400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc02020 0x56383cc0ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57532/0 (socket says 172.21.15.78:57532) 2024-03-14T10:00:11.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.277+0000 7f312f400700 1 -- 172.21.15.78:0/2598850922 learned_addr learned my addr 172.21.15.78:0/2598850922 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:11.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.277+0000 7f312ea00700 1 --2- 172.21.15.78:0/444413827 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f31140a0c00 0x7f31140a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.277+0000 7f312f400700 1 -- 172.21.15.78:0/2598850922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc01700 msgr2=0x56383cc01ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-14T10:00:11.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.277+0000 7f312f400700 1 --2- 172.21.15.78:0/2598850922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc01700 0x56383cc01ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.277+0000 7f312f400700 1 -- 172.21.15.78:0/2598850922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc0bf40 msgr2=0x56383cc0e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:11.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.277+0000 7f312f400700 1 --2- 172.21.15.78:0/2598850922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc0bf40 0x56383cc0e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.277+0000 7f312f400700 1 -- 172.21.15.78:0/2598850922 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56383c9d5430 con 0x56383cc02020 2024-03-14T10:00:11.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.277+0000 7f312ea00700 1 --2- 172.21.15.78:0/444413827 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f31140a0c00 0x7f31140a30c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f311c096430 tx=0x7f311c00ce40 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:11.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.277+0000 7f312f400700 1 --2- 172.21.15.78:0/2598850922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc02020 0x56383cc0ba00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3120005f00 tx=0x7f312000b5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86e63d554ff28f7c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:11.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.277+0000 7f3124c00700 1 -- 172.21.15.78:0/2598850922 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312001d070 con 0x56383cc02020 2024-03-14T10:00:11.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.277+0000 7f3124c00700 1 -- 172.21.15.78:0/2598850922 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f312001e040 con 0x56383cc02020 2024-03-14T10:00:11.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.277+0000 7f3124c00700 1 -- 172.21.15.78:0/2598850922 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3120004bd0 con 0x56383cc02020 2024-03-14T10:00:11.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.278+0000 7f314a0f8b80 1 -- 172.21.15.78:0/2598850922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc02020 msgr2=0x56383cc0ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.278+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/2598850922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc02020 0x56383cc0ba00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3120005f00 tx=0x7f312000b5b0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.278+0000 7f314a0f8b80 1 -- 172.21.15.78:0/2598850922 shutdown_connections 2024-03-14T10:00:11.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.278+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/2598850922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc0bf40 0x56383cc0e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.278+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/2598850922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc01700 0x56383cc01ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.278+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/2598850922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc02020 0x56383cc0ba00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.278+0000 7f314a0f8b80 1 -- 172.21.15.78:0/2598850922 >> 172.21.15.78:0/2598850922 conn(0x56383cb772a0 msgr2=0x56383cbfcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:11.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.278+0000 7f314a0f8b80 1 -- 172.21.15.78:0/2598850922 shutdown_connections 2024-03-14T10:00:11.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.278+0000 7f314a0f8b80 1 -- 172.21.15.78:0/2598850922 wait complete. 2024-03-14T10:00:11.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.279+0000 7f314a0f8b80 1 -- start start 2024-03-14T10:00:11.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.279+0000 7f314a0f8b80 1 Processor -- start 2024-03-14T10:00:11.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.280+0000 7f314a0f8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc01700 0x56383cc0b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.280+0000 7f314a0f8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc02020 0x56383cc08d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.280+0000 7f314a0f8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc0bf40 0x56383cc09450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.280+0000 7f314a0f8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56383caefe00 con 0x56383cc02020 2024-03-14T10:00:11.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.280+0000 7f314a0f8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56383caefaa0 con 0x56383cc01700 2024-03-14T10:00:11.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.280+0000 7f314a0f8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56383c9d7720 con 0x56383cc0bf40 2024-03-14T10:00:11.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.280+0000 7f314a0f8b80 10 client.0 _ll_register_callbacks cb 0x56383c95b420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:11.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.280+0000 7f312fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc01700 0x56383cc0b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.280+0000 7f312fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc01700 0x56383cc0b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57536/0 (socket says 172.21.15.78:57536) 2024-03-14T10:00:11.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.280+0000 7f312fe00700 1 -- 172.21.15.78:0/4268882246 learned_addr learned my addr 172.21.15.78:0/4268882246 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:11.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.281+0000 7f312fe00700 1 --2- 172.21.15.78:0/4268882246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc0bf40 0x56383cc09450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.281+0000 7f312f400700 1 --2- 172.21.15.78:0/4268882246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc02020 0x56383cc08d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.281+0000 7f312fe00700 1 -- 172.21.15.78:0/4268882246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc0bf40 msgr2=0x56383cc09450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:11.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.281+0000 7f312fe00700 1 --2- 172.21.15.78:0/4268882246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc0bf40 0x56383cc09450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.281+0000 7f312fe00700 1 -- 172.21.15.78:0/4268882246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc02020 msgr2=0x56383cc08d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:11.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.281+0000 7f312fe00700 1 --2- 172.21.15.78:0/4268882246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc02020 0x56383cc08d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.281+0000 7f312fe00700 1 -- 172.21.15.78:0/4268882246 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313010e430 con 0x56383cc01700 2024-03-14T10:00:11.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.281+0000 7f312fe00700 1 --2- 172.21.15.78:0/4268882246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc01700 0x56383cc0b9e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3128009fd0 tx=0x7f3128009e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d302f41c01c84d07 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:11.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.282+0000 7f3119600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T10:00:11.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.282+0000 7f3119600700 1 -- 172.21.15.78:0/4268882246 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312800e070 con 0x56383cc01700 2024-03-14T10:00:11.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.282+0000 7f314a0f8b80 1 -- 172.21.15.78:0/4268882246 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56383cbf47a0 con 0x56383cc01700 2024-03-14T10:00:11.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.282+0000 7f3119600700 1 -- 172.21.15.78:0/4268882246 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f312801ac30 con 0x56383cc01700 2024-03-14T10:00:11.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.282+0000 7f3119600700 1 -- 172.21.15.78:0/4268882246 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f312801adf0 con 0x56383cc01700 2024-03-14T10:00:11.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.282+0000 7f314a0f8b80 10 client.26894 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:11.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.282+0000 7f314a0f8b80 1 -- 172.21.15.78:0/4268882246 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56383c981be0 con 0x56383cc01700 2024-03-14T10:00:11.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.282+0000 7f3119600700 1 -- 172.21.15.78:0/4268882246 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56383cbf47a0 con 0x56383cc01700 2024-03-14T10:00:11.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.282+0000 7f3119600700 1 -- 172.21.15.78:0/4268882246 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f31280046b0 con 0x56383cc01700 2024-03-14T10:00:11.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.283+0000 7f314a0f8b80 10 client.26894 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:11.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.283+0000 7f314a0f8b80 10 client.26894 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:00:11.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.283+0000 7f314a0f8b80 20 client.26894 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:11.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.283+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/4268882246 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56383cc1bb50 0x56383cc1df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.283+0000 7f314a0f8b80 4 client.26894 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:00:11.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.283+0000 7f314a0f8b80 1 -- 172.21.15.78:0/4268882246 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56383cb4c780 con 0x56383cc1bb50 2024-03-14T10:00:11.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.283+0000 7f312f400700 1 --2- 172.21.15.78:0/4268882246 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56383cc1bb50 0x56383cc1df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.284+0000 7f312f400700 1 --2- 172.21.15.78:0/4268882246 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56383cc1bb50 0x56383cc1df70 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afb9e0e7e9d47c81 server_cookie=a60f1ce73c2b474c in_seq=0 out_seq=0 2024-03-14T10:00:11.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.284+0000 7f3119600700 10 client.26894 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:11.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.284+0000 7f3119600700 1 -- 172.21.15.78:0/4268882246 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56383cb4c780 con 0x56383cc1bb50 2024-03-14T10:00:11.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.284+0000 7f3119600700 10 client.26894 handle_command_reply: tid=0 2024-03-14T10:00:11.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.284+0000 7f314a0f8b80 1 client.26894 shutdown 2024-03-14T10:00:11.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.284+0000 7f314a0f8b80 1 -- 172.21.15.78:0/4268882246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc01700 msgr2=0x56383cc0b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.284+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/4268882246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc01700 0x56383cc0b9e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3128009fd0 tx=0x7f3128009e30 comp rx=0 tx=0).stop 2024-03-14T10:00:11.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.285+0000 7f314a0f8b80 1 -- 172.21.15.78:0/4268882246 shutdown_connections 2024-03-14T10:00:11.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.285+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/4268882246 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56383cc1bb50 0x56383cc1df70 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.285+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/4268882246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc01700 0x56383cc0b9e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.285+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/4268882246 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc0bf40 0x56383cc09450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.285+0000 7f3119600700 0 client.26894 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:11.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.285+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/4268882246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc02020 0x56383cc08d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.285+0000 7f314a0f8b80 1 -- 172.21.15.78:0/4268882246 >> 172.21.15.78:0/4268882246 conn(0x56383cb772a0 msgr2=0x56383cbfc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:11.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.285+0000 7f314a0f8b80 1 -- 172.21.15.78:0/4268882246 shutdown_connections 2024-03-14T10:00:11.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.285+0000 7f314a0f8b80 1 -- 172.21.15.78:0/4268882246 wait complete. 2024-03-14T10:00:11.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.285+0000 7f314a0f8b80 20 client.26894 trim_cache size 0 max 16384 2024-03-14T10:00:11.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.307+0000 7f314a0f8b80 1 Processor -- start 2024-03-14T10:00:11.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.309+0000 7f314a0f8b80 1 -- start start 2024-03-14T10:00:11.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.309+0000 7f314a0f8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc01700 0x56383cbfccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.309+0000 7f314a0f8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc02020 0x56383cc08620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.309+0000 7f314a0f8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc0bf40 0x56383cc05fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.309+0000 7f314a0f8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56383cc192d0 con 0x56383cc02020 2024-03-14T10:00:11.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.309+0000 7f314a0f8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56383cbf92e0 con 0x56383cc0bf40 2024-03-14T10:00:11.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.309+0000 7f314a0f8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56383cbf6d80 con 0x56383cc01700 2024-03-14T10:00:11.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.310+0000 7f312fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc01700 0x56383cbfccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.310+0000 7f312fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc01700 0x56383cbfccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35846/0 (socket says 172.21.15.78:35846) 2024-03-14T10:00:11.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.310+0000 7f312fe00700 1 -- 172.21.15.78:0/3462280175 learned_addr learned my addr 172.21.15.78:0/3462280175 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:11.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.310+0000 7f312fe00700 1 --2- 172.21.15.78:0/3462280175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc0bf40 0x56383cc05fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.310+0000 7f312f400700 1 --2- 172.21.15.78:0/3462280175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc02020 0x56383cc08620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.310+0000 7f312fe00700 1 -- 172.21.15.78:0/3462280175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc0bf40 msgr2=0x56383cc05fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:11.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.310+0000 7f312fe00700 1 --2- 172.21.15.78:0/3462280175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc0bf40 0x56383cc05fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.310+0000 7f312fe00700 1 -- 172.21.15.78:0/3462280175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc02020 msgr2=0x56383cc08620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:11.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.310+0000 7f312fe00700 1 --2- 172.21.15.78:0/3462280175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc02020 0x56383cc08620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.310+0000 7f312fe00700 1 -- 172.21.15.78:0/3462280175 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56383cbf89d0 con 0x56383cc01700 2024-03-14T10:00:11.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.311+0000 7f312fe00700 1 --2- 172.21.15.78:0/3462280175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc01700 0x56383cbfccb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3128009190 tx=0x7f312800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2776ed1c43b689df server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:11.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.311+0000 7f3125600700 1 -- 172.21.15.78:0/3462280175 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312800e070 con 0x56383cc01700 2024-03-14T10:00:11.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.311+0000 7f3125600700 1 -- 172.21.15.78:0/3462280175 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f312800f040 con 0x56383cc01700 2024-03-14T10:00:11.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.311+0000 7f3125600700 1 -- 172.21.15.78:0/3462280175 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31280159c0 con 0x56383cc01700 2024-03-14T10:00:11.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.313+0000 7f314a0f8b80 1 -- 172.21.15.78:0/3462280175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc01700 msgr2=0x56383cbfccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.313+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/3462280175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc01700 0x56383cbfccb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3128009190 tx=0x7f312800b040 comp rx=0 tx=0).stop 2024-03-14T10:00:11.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.313+0000 7f314a0f8b80 1 -- 172.21.15.78:0/3462280175 shutdown_connections 2024-03-14T10:00:11.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.313+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/3462280175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc0bf40 0x56383cc05fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.313+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/3462280175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc01700 0x56383cbfccb0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.313+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/3462280175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc02020 0x56383cc08620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.313+0000 7f314a0f8b80 1 -- 172.21.15.78:0/3462280175 >> 172.21.15.78:0/3462280175 conn(0x56383cb772a0 msgr2=0x56383cc0ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.313+0000 7f314a0f8b80 1 -- 172.21.15.78:0/3462280175 shutdown_connections 2024-03-14T10:00:11.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.313+0000 7f314a0f8b80 1 -- 172.21.15.78:0/3462280175 wait complete. 2024-03-14T10:00:11.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.314+0000 7f314a0f8b80 1 -- start start 2024-03-14T10:00:11.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.315+0000 7f314a0f8b80 1 Processor -- start 2024-03-14T10:00:11.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.316+0000 7f314a0f8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc01700 0x56383cc110a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.316+0000 7f314a0f8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc02020 0x56383cc115e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.316+0000 7f314a0f8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc0bf40 0x56383cc11b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.316+0000 7f314a0f8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56383cc75e90 con 0x56383cc02020 2024-03-14T10:00:11.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.316+0000 7f314a0f8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56383cc75fd0 con 0x56383cc01700 2024-03-14T10:00:11.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.316+0000 7f314a0f8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56383cc76110 con 0x56383cc0bf40 2024-03-14T10:00:11.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.316+0000 7f314a0f8b80 10 client.0 _ll_register_callbacks cb 0x7f3128008920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:11.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.316+0000 7f312fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc01700 0x56383cc110a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.316+0000 7f312fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc01700 0x56383cc110a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57558/0 (socket says 172.21.15.78:57558) 2024-03-14T10:00:11.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.316+0000 7f312fe00700 1 -- 172.21.15.78:0/3397053964 learned_addr learned my addr 172.21.15.78:0/3397053964 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:11.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.316+0000 7f312fe00700 1 --2- 172.21.15.78:0/3397053964 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc0bf40 0x56383cc11b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.316+0000 7f312f400700 1 --2- 172.21.15.78:0/3397053964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc02020 0x56383cc115e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.317+0000 7f312fe00700 1 -- 172.21.15.78:0/3397053964 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc0bf40 msgr2=0x56383cc11b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.317+0000 7f312fe00700 1 --2- 172.21.15.78:0/3397053964 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc0bf40 0x56383cc11b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.317+0000 7f312fe00700 1 -- 172.21.15.78:0/3397053964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc02020 msgr2=0x56383cc115e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.317+0000 7f312fe00700 1 --2- 172.21.15.78:0/3397053964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc02020 0x56383cc115e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.317+0000 7f312fe00700 1 -- 172.21.15.78:0/3397053964 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56383c981be0 con 0x56383cc01700 2024-03-14T10:00:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.317+0000 7f312fe00700 1 --2- 172.21.15.78:0/3397053964 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc01700 0x56383cc110a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f31280086d0 tx=0x7f3128005b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8067f79a51c4f81 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.317+0000 7f3119600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T10:00:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.317+0000 7f3119600700 1 -- 172.21.15.78:0/3397053964 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312800e030 con 0x56383cc01700 2024-03-14T10:00:11.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.317+0000 7f3119600700 1 -- 172.21.15.78:0/3397053964 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f312801ac30 con 0x56383cc01700 2024-03-14T10:00:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.317+0000 7f314a0f8b80 1 -- 172.21.15.78:0/3397053964 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f312001e040 con 0x56383cc01700 2024-03-14T10:00:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.318+0000 7f3119600700 1 -- 172.21.15.78:0/3397053964 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f312801adf0 con 0x56383cc01700 2024-03-14T10:00:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.318+0000 7f314a0f8b80 10 client.26900 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.318+0000 7f314a0f8b80 1 -- 172.21.15.78:0/3397053964 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56383cb19f20 con 0x56383cc01700 2024-03-14T10:00:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.318+0000 7f3119600700 1 -- 172.21.15.78:0/3397053964 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f312001e040 con 0x56383cc01700 2024-03-14T10:00:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.318+0000 7f3119600700 1 -- 172.21.15.78:0/3397053964 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3128023070 con 0x56383cc01700 2024-03-14T10:00:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.318+0000 7f314a0f8b80 10 client.26900 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.318+0000 7f314a0f8b80 10 client.26900 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:00:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.318+0000 7f314a0f8b80 20 client.26900 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.318+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/3397053964 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56383cc775d0 0x56383cc799f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.318+0000 7f314a0f8b80 4 client.26900 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T10:00:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.318+0000 7f314a0f8b80 1 -- 172.21.15.78:0/3397053964 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56383c9a2d70 con 0x56383cc775d0 2024-03-14T10:00:11.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.319+0000 7f312f400700 1 --2- 172.21.15.78:0/3397053964 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56383cc775d0 0x56383cc799f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.319+0000 7f3119600700 10 client.26900 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:11.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.319+0000 7f312f400700 1 --2- 172.21.15.78:0/3397053964 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56383cc775d0 0x56383cc799f0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55bcc988e3b4bb25 server_cookie=f3b406b6117cdeb9 in_seq=0 out_seq=0 2024-03-14T10:00:11.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.320+0000 7f3119600700 1 -- 172.21.15.78:0/3397053964 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56383c9a2d70 con 0x56383cc775d0 2024-03-14T10:00:11.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.320+0000 7f3119600700 10 client.26900 handle_command_reply: tid=0 2024-03-14T10:00:11.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.320+0000 7f314a0f8b80 1 client.26900 shutdown 2024-03-14T10:00:11.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.320+0000 7f314a0f8b80 1 -- 172.21.15.78:0/3397053964 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc01700 msgr2=0x56383cc110a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.320+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/3397053964 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc01700 0x56383cc110a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f31280086d0 tx=0x7f3128005b60 comp rx=0 tx=0).stop 2024-03-14T10:00:11.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.320+0000 7f314a0f8b80 1 -- 172.21.15.78:0/3397053964 shutdown_connections 2024-03-14T10:00:11.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.320+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/3397053964 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x56383cc775d0 0x56383cc799f0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.320+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/3397053964 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56383cc01700 0x56383cc110a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.320+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/3397053964 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56383cc0bf40 0x56383cc11b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.320+0000 7f3119600700 0 client.26900 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:11.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.320+0000 7f314a0f8b80 1 --2- 172.21.15.78:0/3397053964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56383cc02020 0x56383cc115e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.320+0000 7f314a0f8b80 1 -- 172.21.15.78:0/3397053964 >> 172.21.15.78:0/3397053964 conn(0x56383cb772a0 msgr2=0x56383cc0a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:11.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.321+0000 7f314a0f8b80 1 -- 172.21.15.78:0/3397053964 shutdown_connections 2024-03-14T10:00:11.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.321+0000 7f314a0f8b80 1 -- 172.21.15.78:0/3397053964 wait complete. 2024-03-14T10:00:11.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.321+0000 7f314a0f8b80 20 client.26900 trim_cache size 0 max 16384 2024-03-14T10:00:11.323 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:00:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T10:00:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "aa3042d4-1866-44c9-9d66-a71d6cb44e84", 2024-03-14T10:00:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T10:00:11.323 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:00:11.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.322+0000 7f3136600700 1 -- 172.21.15.78:0/444413827 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f31140a0c00 msgr2=0x7f31140a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.322+0000 7f3136600700 1 --2- 172.21.15.78:0/444413827 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f31140a0c00 0x7f31140a30c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f311c096430 tx=0x7f311c00ce40 comp rx=0 tx=0).stop 2024-03-14T10:00:11.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.322+0000 7f3136600700 1 -- 172.21.15.78:0/444413827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3130106040 msgr2=0x7f313019fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.322+0000 7f3136600700 1 --2- 172.21.15.78:0/444413827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3130106040 0x7f313019fa90 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f311c00e970 tx=0x7f311c00ec80 comp rx=0 tx=0).stop 2024-03-14T10:00:11.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.322+0000 7f3136600700 1 -- 172.21.15.78:0/444413827 shutdown_connections 2024-03-14T10:00:11.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.322+0000 7f3136600700 1 --2- 172.21.15.78:0/444413827 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f31140a0c00 0x7f31140a30c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.322+0000 7f3136600700 1 --2- 172.21.15.78:0/444413827 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3130100f20 0x7f313019f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.322+0000 7f3136600700 1 --2- 172.21.15.78:0/444413827 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f3130199680 0x7f3130199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.322+0000 7f3136600700 1 --2- 172.21.15.78:0/444413827 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3130106040 0x7f313019fa90 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.322+0000 7f3136600700 1 -- 172.21.15.78:0/444413827 >> 172.21.15.78:0/444413827 conn(0x7f31300fc780 msgr2=0x7f31300fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:11.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.322+0000 7f3136600700 1 -- 172.21.15.78:0/444413827 shutdown_connections 2024-03-14T10:00:11.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.322+0000 7f3136600700 1 -- 172.21.15.78:0/444413827 wait complete. 2024-03-14T10:00:11.344 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:00:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.598+0000 7feec6a00700 1 -- 172.21.15.78:0/466759288 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feec00752e0 msgr2=0x7feec00756c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.598+0000 7feec6a00700 1 --2- 172.21.15.78:0/466759288 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feec00752e0 0x7feec00756c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7feeac009990 tx=0x7feeac009ca0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.598+0000 7feec6a00700 1 -- 172.21.15.78:0/466759288 shutdown_connections 2024-03-14T10:00:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.598+0000 7feec6a00700 1 --2- 172.21.15.78:0/466759288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feec0111b90 0x7feec0113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.598+0000 7feec6a00700 1 --2- 172.21.15.78:0/466759288 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feec00752e0 0x7feec00756c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.598+0000 7feec6a00700 1 --2- 172.21.15.78:0/466759288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feec0075c00 0x7feec0111650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.598+0000 7feec6a00700 1 -- 172.21.15.78:0/466759288 >> 172.21.15.78:0/466759288 conn(0x7feec00fe980 msgr2=0x7feec0100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:11.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.599+0000 7feec6a00700 1 -- 172.21.15.78:0/466759288 shutdown_connections 2024-03-14T10:00:11.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.599+0000 7feec6a00700 1 -- 172.21.15.78:0/466759288 wait complete. 2024-03-14T10:00:11.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.601+0000 7feec6a00700 1 Processor -- start 2024-03-14T10:00:11.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.601+0000 7feec6a00700 1 -- start start 2024-03-14T10:00:11.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.601+0000 7feec6a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feec00752e0 0x7feec01a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.601+0000 7feec6a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feec0075c00 0x7feec01a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.601+0000 7feec6a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feec0111b90 0x7feec019da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.601+0000 7feec6a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec019e200 con 0x7feec0075c00 2024-03-14T10:00:11.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.601+0000 7feec6a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec019e370 con 0x7feec0111b90 2024-03-14T10:00:11.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.601+0000 7feec6a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec019e530 con 0x7feec00752e0 2024-03-14T10:00:11.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.602+0000 7feebfe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feec00752e0 0x7feec01a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.602+0000 7feebfe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feec00752e0 0x7feec01a3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35878/0 (socket says 172.21.15.78:35878) 2024-03-14T10:00:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.602+0000 7feebfe00700 1 -- 172.21.15.78:0/2950417301 learned_addr learned my addr 172.21.15.78:0/2950417301 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.602+0000 7feebfe00700 1 -- 172.21.15.78:0/2950417301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feec0111b90 msgr2=0x7feec019da30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-14T10:00:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.602+0000 7feec4a00700 1 --2- 172.21.15.78:0/2950417301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feec0111b90 0x7feec019da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.602+0000 7feebfe00700 1 --2- 172.21.15.78:0/2950417301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feec0111b90 0x7feec019da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.602+0000 7feebfe00700 1 -- 172.21.15.78:0/2950417301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feec0075c00 msgr2=0x7feec01a3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.602+0000 7feebfe00700 1 --2- 172.21.15.78:0/2950417301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feec0075c00 0x7feec01a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.602+0000 7feebfe00700 1 -- 172.21.15.78:0/2950417301 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb0009520 con 0x7feec00752e0 2024-03-14T10:00:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.602+0000 7feebfe00700 1 --2- 172.21.15.78:0/2950417301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feec00752e0 0x7feec01a3990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7feeac000c00 tx=0x7feeac0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:11.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.603+0000 7feebcc00700 1 -- 172.21.15.78:0/2950417301 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac01d070 con 0x7feec00752e0 2024-03-14T10:00:11.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.603+0000 7feebcc00700 1 -- 172.21.15.78:0/2950417301 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feeac022470 con 0x7feec00752e0 2024-03-14T10:00:11.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.603+0000 7feebcc00700 1 -- 172.21.15.78:0/2950417301 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac017650 con 0x7feec00752e0 2024-03-14T10:00:11.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.603+0000 7feec6a00700 1 -- 172.21.15.78:0/2950417301 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeac0095f0 con 0x7feec00752e0 2024-03-14T10:00:11.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.603+0000 7feec6a00700 1 -- 172.21.15.78:0/2950417301 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec01aa730 con 0x7feec00752e0 2024-03-14T10:00:11.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.606+0000 7feebcc00700 1 -- 172.21.15.78:0/2950417301 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7feeac0225e0 con 0x7feec00752e0 2024-03-14T10:00:11.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.608+0000 7feebcc00700 1 --2- 172.21.15.78:0/2950417301 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7feea40a0c80 0x7feea40a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.608+0000 7feebcc00700 1 -- 172.21.15.78:0/2950417301 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7feeac0cdc50 con 0x7feec00752e0 2024-03-14T10:00:11.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.608+0000 7feebf400700 1 --2- 172.21.15.78:0/2950417301 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7feea40a0c80 0x7feea40a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.609+0000 7feeda5bdb80 1 Processor -- start 2024-03-14T10:00:11.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.609+0000 7feebf400700 1 --2- 172.21.15.78:0/2950417301 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7feea40a0c80 0x7feea40a3140 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7feeb0005d90 tx=0x7feeb0005ce0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:11.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.610+0000 7feeda5bdb80 1 -- start start 2024-03-14T10:00:11.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.610+0000 7feeda5bdb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65680 0x563357e65a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.610+0000 7feeda5bdb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65fa0 0x563357e6f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.610+0000 7feeda5bdb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 0x563357e722b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.610+0000 7feeda5bdb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563357dbe990 con 0x563357e6fec0 2024-03-14T10:00:11.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.610+0000 7feeda5bdb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563357dbc350 con 0x563357e65680 2024-03-14T10:00:11.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.610+0000 7feeda5bdb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563357db7c40 con 0x563357e65fa0 2024-03-14T10:00:11.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.610+0000 7feebfe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65680 0x563357e65a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.610+0000 7feebfe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65680 0x563357e65a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57576/0 (socket says 172.21.15.78:57576) 2024-03-14T10:00:11.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.610+0000 7feebfe00700 1 -- 172.21.15.78:0/1427741066 learned_addr learned my addr 172.21.15.78:0/1427741066 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:11.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.610+0000 7feebf400700 1 --2- 172.21.15.78:0/1427741066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65fa0 0x563357e6f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.610+0000 7feec4a00700 1 --2- 172.21.15.78:0/1427741066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 0x563357e722b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.611+0000 7feebfe00700 1 -- 172.21.15.78:0/1427741066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65fa0 msgr2=0x563357e6f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:11.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.611+0000 7feebfe00700 1 --2- 172.21.15.78:0/1427741066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65fa0 0x563357e6f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.611+0000 7feebfe00700 1 -- 172.21.15.78:0/1427741066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 msgr2=0x563357e722b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:11.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.611+0000 7feebfe00700 1 --2- 172.21.15.78:0/1427741066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 0x563357e722b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.611+0000 7feebfe00700 1 -- 172.21.15.78:0/1427741066 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563357c393b0 con 0x563357e65680 2024-03-14T10:00:11.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.611+0000 7feec4a00700 1 --2- 172.21.15.78:0/1427741066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 0x563357e722b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:00:11.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.611+0000 7feebfe00700 1 --2- 172.21.15.78:0/1427741066 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65680 0x563357e65a60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7feeac0a2c60 tx=0x7feeac022f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76ca6428e7c3b0f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:11.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.611+0000 7feeb5200700 1 -- 172.21.15.78:0/1427741066 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac09ada0 con 0x563357e65680 2024-03-14T10:00:11.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.611+0000 7feeb5200700 1 -- 172.21.15.78:0/1427741066 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feeac09ebd0 con 0x563357e65680 2024-03-14T10:00:11.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.611+0000 7feeb5200700 1 -- 172.21.15.78:0/1427741066 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac09a290 con 0x563357e65680 2024-03-14T10:00:11.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.612+0000 7feeda5bdb80 1 -- 172.21.15.78:0/1427741066 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65680 msgr2=0x563357e65a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.612+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/1427741066 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65680 0x563357e65a60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7feeac0a2c60 tx=0x7feeac022f90 comp rx=0 tx=0).stop 2024-03-14T10:00:11.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.612+0000 7feeda5bdb80 1 -- 172.21.15.78:0/1427741066 shutdown_connections 2024-03-14T10:00:11.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.612+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/1427741066 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65680 0x563357e65a60 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.612+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/1427741066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 0x563357e722b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.612+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/1427741066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65fa0 0x563357e6f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.612+0000 7feeda5bdb80 1 -- 172.21.15.78:0/1427741066 >> 172.21.15.78:0/1427741066 conn(0x563357ddb220 msgr2=0x563357e60aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:11.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.612+0000 7feeda5bdb80 1 -- 172.21.15.78:0/1427741066 shutdown_connections 2024-03-14T10:00:11.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.612+0000 7feeda5bdb80 1 -- 172.21.15.78:0/1427741066 wait complete. 2024-03-14T10:00:11.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.613+0000 7feeda5bdb80 1 -- start start 2024-03-14T10:00:11.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.613+0000 7feeda5bdb80 1 Processor -- start 2024-03-14T10:00:11.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feeda5bdb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65680 0x563357e6f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feeda5bdb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65fa0 0x563357e6cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feeda5bdb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 0x563357e6d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feeda5bdb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563357d53d80 con 0x563357e6fec0 2024-03-14T10:00:11.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feeda5bdb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563357d53a20 con 0x563357e65680 2024-03-14T10:00:11.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feeda5bdb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563357c3b6a0 con 0x563357e65fa0 2024-03-14T10:00:11.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feeda5bdb80 10 client.0 _ll_register_callbacks cb 0x563357bbf3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:11.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feebf400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65fa0 0x563357e6cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feebf400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65fa0 0x563357e6cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35900/0 (socket says 172.21.15.78:35900) 2024-03-14T10:00:11.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feebf400700 1 -- 172.21.15.78:0/1925805239 learned_addr learned my addr 172.21.15.78:0/1925805239 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:11.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feebfe00700 1 --2- 172.21.15.78:0/1925805239 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65680 0x563357e6f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feec4a00700 1 --2- 172.21.15.78:0/1925805239 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 0x563357e6d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feebf400700 1 -- 172.21.15.78:0/1925805239 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65680 msgr2=0x563357e6f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:11.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feebf400700 1 --2- 172.21.15.78:0/1925805239 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65680 0x563357e6f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feebf400700 1 -- 172.21.15.78:0/1925805239 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 msgr2=0x563357e6d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:11.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feebf400700 1 --2- 172.21.15.78:0/1925805239 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 0x563357e6d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.614+0000 7feebf400700 1 -- 172.21.15.78:0/1925805239 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec01aa730 con 0x563357e65fa0 2024-03-14T10:00:11.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.615+0000 7feebf400700 1 --2- 172.21.15.78:0/1925805239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65fa0 0x563357e6cc00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7feeb001d6b0 tx=0x7feeb0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d6f4299d1406b23 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:11.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.615+0000 7feea9600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:00:11.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.615+0000 7feea9600700 1 -- 172.21.15.78:0/1925805239 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb0004a30 con 0x563357e65fa0 2024-03-14T10:00:11.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.615+0000 7feea9600700 1 -- 172.21.15.78:0/1925805239 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7feeb0004b90 con 0x563357e65fa0 2024-03-14T10:00:11.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.615+0000 7feeda5bdb80 1 -- 172.21.15.78:0/1925805239 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563357e58720 con 0x563357e65fa0 2024-03-14T10:00:11.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.615+0000 7feea9600700 1 -- 172.21.15.78:0/1925805239 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7feeb0004d90 con 0x563357e65fa0 2024-03-14T10:00:11.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.616+0000 7feeda5bdb80 10 client.26893 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:11.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.616+0000 7feeda5bdb80 1 -- 172.21.15.78:0/1925805239 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563357be5b60 con 0x563357e65fa0 2024-03-14T10:00:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.616+0000 7feea9600700 1 -- 172.21.15.78:0/1925805239 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feeb00054e0 con 0x563357e65fa0 2024-03-14T10:00:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.616+0000 7feea9600700 1 -- 172.21.15.78:0/1925805239 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7feeb0018070 con 0x563357e65fa0 2024-03-14T10:00:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.616+0000 7feeda5bdb80 10 client.26893 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.616+0000 7feeda5bdb80 10 client.26893 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:00:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.616+0000 7feeda5bdb80 20 client.26893 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.616+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/1925805239 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563357e7f780 0x563357e81ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.616+0000 7feeda5bdb80 4 client.26893 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:00:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.616+0000 7feeda5bdb80 1 -- 172.21.15.78:0/1925805239 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563357e58720 con 0x563357e7f780 2024-03-14T10:00:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.617+0000 7feebfe00700 1 --2- 172.21.15.78:0/1925805239 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563357e7f780 0x563357e81ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.618+0000 7feea9600700 10 client.26893 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:11.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.618+0000 7feebfe00700 1 --2- 172.21.15.78:0/1925805239 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563357e7f780 0x563357e81ba0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37d72f834184c317 server_cookie=e04ce86f312fb54c in_seq=0 out_seq=0 2024-03-14T10:00:11.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.619+0000 7feea9600700 1 -- 172.21.15.78:0/1925805239 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x563357e58720 con 0x563357e7f780 2024-03-14T10:00:11.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.619+0000 7feea9600700 10 client.26893 handle_command_reply: tid=0 2024-03-14T10:00:11.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.619+0000 7feeda5bdb80 1 client.26893 shutdown 2024-03-14T10:00:11.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.620+0000 7feeda5bdb80 1 -- 172.21.15.78:0/1925805239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65fa0 msgr2=0x563357e6cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.620+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/1925805239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65fa0 0x563357e6cc00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7feeb001d6b0 tx=0x7feeb0004750 comp rx=0 tx=0).stop 2024-03-14T10:00:11.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.620+0000 7feeda5bdb80 1 -- 172.21.15.78:0/1925805239 shutdown_connections 2024-03-14T10:00:11.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.620+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/1925805239 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563357e7f780 0x563357e81ba0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.620+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/1925805239 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65680 0x563357e6f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.620+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/1925805239 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 0x563357e6d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.620+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/1925805239 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65fa0 0x563357e6cc00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.620+0000 7feeda5bdb80 1 -- 172.21.15.78:0/1925805239 >> 172.21.15.78:0/1925805239 conn(0x563357ddb220 msgr2=0x563357e60690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:11.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.620+0000 7feea9600700 0 client.26893 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:11.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.621+0000 7feeda5bdb80 1 -- 172.21.15.78:0/1925805239 shutdown_connections 2024-03-14T10:00:11.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.621+0000 7feeda5bdb80 1 -- 172.21.15.78:0/1925805239 wait complete. 2024-03-14T10:00:11.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.622+0000 7feeda5bdb80 20 client.26893 trim_cache size 0 max 16384 2024-03-14T10:00:11.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.647+0000 7feeda5bdb80 1 Processor -- start 2024-03-14T10:00:11.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.647+0000 7feeda5bdb80 1 -- start start 2024-03-14T10:00:11.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.650+0000 7feeda5bdb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65680 0x563357e67f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.650+0000 7feeda5bdb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e65fa0 0x563357e68460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.650+0000 7feeda5bdb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e6fec0 0x563357e605d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.650+0000 7feeda5bdb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563357e67760 con 0x563357e65fa0 2024-03-14T10:00:11.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.650+0000 7feeda5bdb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563357e5f800 con 0x563357e6fec0 2024-03-14T10:00:11.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.650+0000 7feeda5bdb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563357e5d230 con 0x563357e65680 2024-03-14T10:00:11.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.650+0000 7feebfe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65680 0x563357e67f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.650+0000 7feebfe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65680 0x563357e67f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35912/0 (socket says 172.21.15.78:35912) 2024-03-14T10:00:11.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.650+0000 7feec4a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e6fec0 0x563357e605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.651+0000 7feec4a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e6fec0 0x563357e605d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57596/0 (socket says 172.21.15.78:57596) 2024-03-14T10:00:11.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.651+0000 7feec4a00700 1 -- 172.21.15.78:0/3362628767 learned_addr learned my addr 172.21.15.78:0/3362628767 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.651+0000 7feebf400700 1 --2- 172.21.15.78:0/3362628767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e65fa0 0x563357e68460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.651+0000 7feebfe00700 1 -- 172.21.15.78:0/3362628767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e6fec0 msgr2=0x563357e605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.651+0000 7feebfe00700 1 --2- 172.21.15.78:0/3362628767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e6fec0 0x563357e605d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.651+0000 7feebfe00700 1 -- 172.21.15.78:0/3362628767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e65fa0 msgr2=0x563357e68460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.651+0000 7feebfe00700 1 --2- 172.21.15.78:0/3362628767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e65fa0 0x563357e68460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.651+0000 7feebfe00700 1 -- 172.21.15.78:0/3362628767 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563357d7dea0 con 0x563357e65680 2024-03-14T10:00:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.651+0000 7feebf400700 1 --2- 172.21.15.78:0/3362628767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e65fa0 0x563357e68460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:00:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.651+0000 7feebfe00700 1 --2- 172.21.15.78:0/3362628767 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65680 0x563357e67f20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7feeac097810 tx=0x7feeac0cd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45e9046e72a121c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.653+0000 7feeb5c00700 1 -- 172.21.15.78:0/3362628767 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac0b9040 con 0x563357e65680 2024-03-14T10:00:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.653+0000 7feeb5c00700 1 -- 172.21.15.78:0/3362628767 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feeac09ada0 con 0x563357e65680 2024-03-14T10:00:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.653+0000 7feeb5c00700 1 -- 172.21.15.78:0/3362628767 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac0aee20 con 0x563357e65680 2024-03-14T10:00:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.653+0000 7feeda5bdb80 1 -- 172.21.15.78:0/3362628767 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65680 msgr2=0x563357e67f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.653+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/3362628767 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65680 0x563357e67f20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7feeac097810 tx=0x7feeac0cd040 comp rx=0 tx=0).stop 2024-03-14T10:00:11.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.655+0000 7feeda5bdb80 1 -- 172.21.15.78:0/3362628767 shutdown_connections 2024-03-14T10:00:11.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.655+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/3362628767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e6fec0 0x563357e605d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.655+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/3362628767 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65680 0x563357e67f20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.655+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/3362628767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e65fa0 0x563357e68460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.655+0000 7feeda5bdb80 1 -- 172.21.15.78:0/3362628767 >> 172.21.15.78:0/3362628767 conn(0x563357ddb220 msgr2=0x563357e5fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:11.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.655+0000 7feeda5bdb80 1 -- 172.21.15.78:0/3362628767 shutdown_connections 2024-03-14T10:00:11.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.655+0000 7feeda5bdb80 1 -- 172.21.15.78:0/3362628767 wait complete. 2024-03-14T10:00:11.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.656+0000 7feeda5bdb80 1 -- start start 2024-03-14T10:00:11.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.657+0000 7feeda5bdb80 1 Processor -- start 2024-03-14T10:00:11.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.657+0000 7feeda5bdb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65680 0x563357e722b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.657+0000 7feeda5bdb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65fa0 0x563357e727f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.658+0000 7feeda5bdb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 0x563357ed0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.658+0000 7feeda5bdb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563357e73040 con 0x563357e6fec0 2024-03-14T10:00:11.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.658+0000 7feeda5bdb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563357ed07a0 con 0x563357e65fa0 2024-03-14T10:00:11.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.658+0000 7feeda5bdb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563357ed08e0 con 0x563357e65680 2024-03-14T10:00:11.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.658+0000 7feeda5bdb80 10 client.0 _ll_register_callbacks cb 0x7feeb000eec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:11.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.658+0000 7feebfe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65680 0x563357e722b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.658+0000 7feebfe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65680 0x563357e722b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35914/0 (socket says 172.21.15.78:35914) 2024-03-14T10:00:11.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.658+0000 7feebfe00700 1 -- 172.21.15.78:0/524140652 learned_addr learned my addr 172.21.15.78:0/524140652 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:11.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.658+0000 7feec4a00700 1 --2- 172.21.15.78:0/524140652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 0x563357ed0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.658+0000 7feebf400700 1 --2- 172.21.15.78:0/524140652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65fa0 0x563357e727f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.658+0000 7feebfe00700 1 -- 172.21.15.78:0/524140652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65fa0 msgr2=0x563357e727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:11.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.658+0000 7feebfe00700 1 --2- 172.21.15.78:0/524140652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65fa0 0x563357e727f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.658+0000 7feebfe00700 1 -- 172.21.15.78:0/524140652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 msgr2=0x563357ed0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:11.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.658+0000 7feebfe00700 1 --2- 172.21.15.78:0/524140652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 0x563357ed0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.658+0000 7feebfe00700 1 -- 172.21.15.78:0/524140652 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563357c393b0 con 0x563357e65680 2024-03-14T10:00:11.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.659+0000 7feec4a00700 1 --2- 172.21.15.78:0/524140652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 0x563357ed0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T10:00:11.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.659+0000 7feebfe00700 1 --2- 172.21.15.78:0/524140652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65680 0x563357e722b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7feeac097250 tx=0x7feeac09a700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10124ede03a6b84f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:11.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.659+0000 7feea9600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:00:11.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.659+0000 7feea9600700 1 -- 172.21.15.78:0/524140652 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac0b9040 con 0x563357e65680 2024-03-14T10:00:11.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.659+0000 7feeda5bdb80 1 -- 172.21.15.78:0/524140652 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feeac09ebd0 con 0x563357e65680 2024-03-14T10:00:11.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.661+0000 7feea9600700 1 -- 172.21.15.78:0/524140652 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7feeac0228c0 con 0x563357e65680 2024-03-14T10:00:11.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.661+0000 7feea9600700 1 -- 172.21.15.78:0/524140652 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7feeac0aebe0 con 0x563357e65680 2024-03-14T10:00:11.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.661+0000 7feeda5bdb80 10 client.26899 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:11.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.661+0000 7feeda5bdb80 1 -- 172.21.15.78:0/524140652 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563357c3dcd0 con 0x563357e65680 2024-03-14T10:00:11.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.661+0000 7feea9600700 1 -- 172.21.15.78:0/524140652 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feeac09ebd0 con 0x563357e65680 2024-03-14T10:00:11.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.662+0000 7feea9600700 1 -- 172.21.15.78:0/524140652 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7feeac09b090 con 0x563357e65680 2024-03-14T10:00:11.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.662+0000 7feeda5bdb80 10 client.26899 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:11.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.662+0000 7feeda5bdb80 10 client.26899 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:00:11.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.662+0000 7feeda5bdb80 20 client.26899 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:11.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.662+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/524140652 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563357ed1b60 0x563357ed3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:11.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.662+0000 7feeda5bdb80 4 client.26899 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:00:11.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.662+0000 7feeda5bdb80 1 -- 172.21.15.78:0/524140652 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563357db0700 con 0x563357ed1b60 2024-03-14T10:00:11.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.665+0000 7feebf400700 1 --2- 172.21.15.78:0/524140652 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563357ed1b60 0x563357ed3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:11.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.666+0000 7feebf400700 1 --2- 172.21.15.78:0/524140652 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563357ed1b60 0x563357ed3f50 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=918e6d2d750c3880 server_cookie=12225bf052faf946 in_seq=0 out_seq=0 2024-03-14T10:00:11.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.667+0000 7feea9600700 10 client.26899 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:11.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.667+0000 7feea9600700 1 -- 172.21.15.78:0/524140652 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x563357db0700 con 0x563357ed1b60 2024-03-14T10:00:11.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.667+0000 7feea9600700 10 client.26899 handle_command_reply: tid=0 2024-03-14T10:00:11.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.667+0000 7feeda5bdb80 1 client.26899 shutdown 2024-03-14T10:00:11.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.668+0000 7feeda5bdb80 1 -- 172.21.15.78:0/524140652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65680 msgr2=0x563357e722b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.668+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/524140652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65680 0x563357e722b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7feeac097250 tx=0x7feeac09a700 comp rx=0 tx=0).stop 2024-03-14T10:00:11.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.668+0000 7feeda5bdb80 1 -- 172.21.15.78:0/524140652 shutdown_connections 2024-03-14T10:00:11.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.668+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/524140652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563357e6fec0 0x563357ed0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.668+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/524140652 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563357e65680 0x563357e722b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.668+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/524140652 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563357ed1b60 0x563357ed3f50 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.669+0000 7feea9600700 0 client.26899 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:11.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.669+0000 7feeda5bdb80 1 --2- 172.21.15.78:0/524140652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563357e65fa0 0x563357e727f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.669+0000 7feeda5bdb80 1 -- 172.21.15.78:0/524140652 >> 172.21.15.78:0/524140652 conn(0x563357ddb220 msgr2=0x563357e6b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:11.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.669+0000 7feeda5bdb80 1 -- 172.21.15.78:0/524140652 shutdown_connections 2024-03-14T10:00:11.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.669+0000 7feeda5bdb80 1 -- 172.21.15.78:0/524140652 wait complete. 2024-03-14T10:00:11.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.669+0000 7feeda5bdb80 20 client.26899 trim_cache size 0 max 16384 2024-03-14T10:00:11.671 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:00:11.671 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "scrub active (1415 inodes in the stack)", 2024-03-14T10:00:11.671 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T10:00:11.671 INFO:teuthology.orchestra.run.smithi078.stdout: "aa3042d4-1866-44c9-9d66-a71d6cb44e84": { 2024-03-14T10:00:11.671 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T10:00:11.671 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "aa3042d4-1866-44c9-9d66-a71d6cb44e84", 2024-03-14T10:00:11.671 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T10:00:11.671 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:00:11.671 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:00:11.671 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:00:11.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.670+0000 7feec6a00700 1 -- 172.21.15.78:0/2950417301 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7feea40a0c80 msgr2=0x7feea40a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.670+0000 7feec6a00700 1 --2- 172.21.15.78:0/2950417301 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7feea40a0c80 0x7feea40a3140 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7feeb0005d90 tx=0x7feeb0005ce0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.670+0000 7feec6a00700 1 -- 172.21.15.78:0/2950417301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feec00752e0 msgr2=0x7feec01a3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:11.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.670+0000 7feec6a00700 1 --2- 172.21.15.78:0/2950417301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feec00752e0 0x7feec01a3990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7feeac000c00 tx=0x7feeac0057f0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.670+0000 7feec6a00700 1 -- 172.21.15.78:0/2950417301 shutdown_connections 2024-03-14T10:00:11.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.670+0000 7feec6a00700 1 --2- 172.21.15.78:0/2950417301 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7feea40a0c80 0x7feea40a3140 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.670+0000 7feec6a00700 1 --2- 172.21.15.78:0/2950417301 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feec0111b90 0x7feec019da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.670+0000 7feec6a00700 1 --2- 172.21.15.78:0/2950417301 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7feec00752e0 0x7feec01a3990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.670+0000 7feec6a00700 1 --2- 172.21.15.78:0/2950417301 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feec0075c00 0x7feec01a3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:11.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.671+0000 7feec6a00700 1 -- 172.21.15.78:0/2950417301 >> 172.21.15.78:0/2950417301 conn(0x7feec00fe980 msgr2=0x7feec0100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:11.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.671+0000 7feec6a00700 1 -- 172.21.15.78:0/2950417301 shutdown_connections 2024-03-14T10:00:11.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:11.671+0000 7feec6a00700 1 -- 172.21.15.78:0/2950417301 wait complete. 2024-03-14T10:00:11.699 INFO:tasks.cephfs.filesystem:scrub status for tag:aa3042d4-1866-44c9-9d66-a71d6cb44e84 - {'path': '/', 'tag': 'aa3042d4-1866-44c9-9d66-a71d6cb44e84', 'options': 'recursive,force'} 2024-03-14T10:00:12.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:11 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T10:00:12.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:11 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T10:00:12.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:11 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T10:00:12.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:11 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T10:00:12.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:11 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T10:00:12.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:11 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T10:00:12.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:11 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T10:00:12.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:11 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T10:00:12.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:11 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T10:00:12.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:11 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T10:00:12.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:11 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T10:00:12.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:11 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T10:00:13.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:12 smithi152 ceph-mon[48279]: pgmap v184: 129 pgs: 129 active+clean; 227 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.6 MiB/s wr, 563 op/s 2024-03-14T10:00:13.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:12 smithi132 ceph-mon[47959]: pgmap v184: 129 pgs: 129 active+clean; 227 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.6 MiB/s wr, 563 op/s 2024-03-14T10:00:13.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:12 smithi078 ceph-mon[38143]: pgmap v184: 129 pgs: 129 active+clean; 227 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 1.6 MiB/s wr, 563 op/s 2024-03-14T10:00:14.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:13 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T10:00:14.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:13 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T10:00:14.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:13 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T10:00:15.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:14 smithi152 ceph-mon[48279]: pgmap v185: 129 pgs: 129 active+clean; 227 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.1 MiB/s wr, 366 op/s 2024-03-14T10:00:15.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:14 smithi132 ceph-mon[47959]: pgmap v185: 129 pgs: 129 active+clean; 227 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.1 MiB/s wr, 366 op/s 2024-03-14T10:00:15.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:14 smithi078 ceph-mon[38143]: pgmap v185: 129 pgs: 129 active+clean; 227 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.1 MiB/s wr, 366 op/s 2024-03-14T10:00:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:17 smithi152 ceph-mon[48279]: pgmap v186: 129 pgs: 129 active+clean; 238 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 1.6 MiB/s wr, 541 op/s 2024-03-14T10:00:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:17 smithi132 ceph-mon[47959]: pgmap v186: 129 pgs: 129 active+clean; 238 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 1.6 MiB/s wr, 541 op/s 2024-03-14T10:00:17.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:17 smithi078 ceph-mon[38143]: pgmap v186: 129 pgs: 129 active+clean; 238 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 1.6 MiB/s wr, 541 op/s 2024-03-14T10:00:19.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:18 smithi152 ceph-mon[48279]: pgmap v187: 129 pgs: 129 active+clean; 238 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 1.0 MiB/s wr, 358 op/s 2024-03-14T10:00:19.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:18 smithi132 ceph-mon[47959]: pgmap v187: 129 pgs: 129 active+clean; 238 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 1.0 MiB/s wr, 358 op/s 2024-03-14T10:00:19.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:18 smithi078 ceph-mon[38143]: pgmap v187: 129 pgs: 129 active+clean; 238 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 1.0 MiB/s wr, 358 op/s 2024-03-14T10:00:20.311 INFO:tasks.workunit.client.0.smithi078.stdout:40000 copy from 0x98ca82 to 0x993e9d, (0x741b bytes) at 0xfd3925 2024-03-14T10:00:21.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:21 smithi152 ceph-mon[48279]: pgmap v188: 129 pgs: 129 active+clean; 238 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 1.0 MiB/s wr, 358 op/s 2024-03-14T10:00:21.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:21 smithi132 ceph-mon[47959]: pgmap v188: 129 pgs: 129 active+clean; 238 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 1.0 MiB/s wr, 358 op/s 2024-03-14T10:00:21.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:21 smithi078 ceph-mon[38143]: pgmap v188: 129 pgs: 129 active+clean; 238 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 1.0 MiB/s wr, 358 op/s 2024-03-14T10:00:23.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:23 smithi152 ceph-mon[48279]: pgmap v189: 129 pgs: 129 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 1.6 MiB/s wr, 525 op/s 2024-03-14T10:00:23.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:23 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T10:00:23.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:23 smithi132 ceph-mon[47959]: pgmap v189: 129 pgs: 129 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 1.6 MiB/s wr, 525 op/s 2024-03-14T10:00:23.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:23 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T10:00:23.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:23 smithi078 ceph-mon[38143]: pgmap v189: 129 pgs: 129 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 1.6 MiB/s wr, 525 op/s 2024-03-14T10:00:23.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:23 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T10:00:24.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:24 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T10:00:24.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:24 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T10:00:24.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:24 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:00:24.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:24 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T10:00:24.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:24 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T10:00:24.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:24 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:00:24.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:24 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T10:00:24.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:24 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T10:00:24.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:24 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:00:25.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:25 smithi152 ceph-mon[48279]: pgmap v190: 129 pgs: 129 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 1.1 MiB/s wr, 343 op/s 2024-03-14T10:00:25.523 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:25 smithi132 ceph-mon[47959]: pgmap v190: 129 pgs: 129 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 1.1 MiB/s wr, 343 op/s 2024-03-14T10:00:25.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:25 smithi078 ceph-mon[38143]: pgmap v190: 129 pgs: 129 active+clean; 252 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 1.1 MiB/s wr, 343 op/s 2024-03-14T10:00:27.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:27 smithi152 ceph-mon[48279]: pgmap v191: 129 pgs: 129 active+clean; 237 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 1.6 MiB/s wr, 514 op/s 2024-03-14T10:00:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:27 smithi132 ceph-mon[47959]: pgmap v191: 129 pgs: 129 active+clean; 237 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 1.6 MiB/s wr, 514 op/s 2024-03-14T10:00:27.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:27 smithi078 ceph-mon[38143]: pgmap v191: 129 pgs: 129 active+clean; 237 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 1.6 MiB/s wr, 514 op/s 2024-03-14T10:00:29.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:28 smithi152 ceph-mon[48279]: pgmap v192: 129 pgs: 129 active+clean; 237 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.1 MiB/s wr, 338 op/s 2024-03-14T10:00:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:28 smithi132 ceph-mon[47959]: pgmap v192: 129 pgs: 129 active+clean; 237 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.1 MiB/s wr, 338 op/s 2024-03-14T10:00:29.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:28 smithi078 ceph-mon[38143]: pgmap v192: 129 pgs: 129 active+clean; 237 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.1 MiB/s wr, 338 op/s 2024-03-14T10:00:30.878 INFO:tasks.workunit.client.0.smithi078.stdout:truncating to largest ever: 0x63ff6b1 2024-03-14T10:00:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:31 smithi152 ceph-mon[48279]: pgmap v193: 129 pgs: 129 active+clean; 237 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.1 MiB/s wr, 338 op/s 2024-03-14T10:00:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:31 smithi132 ceph-mon[47959]: pgmap v193: 129 pgs: 129 active+clean; 237 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.1 MiB/s wr, 338 op/s 2024-03-14T10:00:31.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:31 smithi078 ceph-mon[38143]: pgmap v193: 129 pgs: 129 active+clean; 237 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 1.1 MiB/s wr, 338 op/s 2024-03-14T10:00:33.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:33 smithi152 ceph-mon[48279]: pgmap v194: 129 pgs: 129 active+clean; 251 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 1.6 MiB/s wr, 510 op/s 2024-03-14T10:00:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:33 smithi132 ceph-mon[47959]: pgmap v194: 129 pgs: 129 active+clean; 251 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 1.6 MiB/s wr, 510 op/s 2024-03-14T10:00:33.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:33 smithi078 ceph-mon[38143]: pgmap v194: 129 pgs: 129 active+clean; 251 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 1.6 MiB/s wr, 510 op/s 2024-03-14T10:00:35.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:35 smithi152 ceph-mon[48279]: pgmap v195: 129 pgs: 129 active+clean; 251 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 1003 KiB/s wr, 343 op/s 2024-03-14T10:00:35.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:35 smithi132 ceph-mon[47959]: pgmap v195: 129 pgs: 129 active+clean; 251 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 1003 KiB/s wr, 343 op/s 2024-03-14T10:00:35.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:35 smithi078 ceph-mon[38143]: pgmap v195: 129 pgs: 129 active+clean; 251 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 1003 KiB/s wr, 343 op/s 2024-03-14T10:00:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:37 smithi152 ceph-mon[48279]: pgmap v196: 129 pgs: 129 active+clean; 237 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 1.4 MiB/s wr, 523 op/s 2024-03-14T10:00:37.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:37 smithi132 ceph-mon[47959]: pgmap v196: 129 pgs: 129 active+clean; 237 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 1.4 MiB/s wr, 523 op/s 2024-03-14T10:00:37.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:37 smithi078 ceph-mon[38143]: pgmap v196: 129 pgs: 129 active+clean; 237 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 1.4 MiB/s wr, 523 op/s 2024-03-14T10:00:39.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:38 smithi152 ceph-mon[48279]: pgmap v197: 129 pgs: 129 active+clean; 237 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 986 KiB/s wr, 351 op/s 2024-03-14T10:00:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:38 smithi132 ceph-mon[47959]: pgmap v197: 129 pgs: 129 active+clean; 237 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 986 KiB/s wr, 351 op/s 2024-03-14T10:00:39.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:38 smithi078 ceph-mon[38143]: pgmap v197: 129 pgs: 129 active+clean; 237 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 986 KiB/s wr, 351 op/s 2024-03-14T10:00:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:41 smithi152 ceph-mon[48279]: pgmap v198: 129 pgs: 129 active+clean; 237 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 986 KiB/s wr, 351 op/s 2024-03-14T10:00:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:41 smithi132 ceph-mon[47959]: pgmap v198: 129 pgs: 129 active+clean; 237 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 986 KiB/s wr, 351 op/s 2024-03-14T10:00:41.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:41 smithi078 ceph-mon[38143]: pgmap v198: 129 pgs: 129 active+clean; 237 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 986 KiB/s wr, 351 op/s 2024-03-14T10:00:41.700 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:00:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.972+0000 7f4c58800700 1 -- 172.21.15.78:0/3161469206 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c50068740 msgr2=0x7f4c5010d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.972+0000 7f4c58800700 1 --2- 172.21.15.78:0/3161469206 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c50068740 0x7f4c5010d1c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4c400098f0 tx=0x7f4c40009c00 comp rx=0 tx=0).stop 2024-03-14T10:00:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.973+0000 7f4c58800700 1 -- 172.21.15.78:0/3161469206 shutdown_connections 2024-03-14T10:00:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.974+0000 7f4c58800700 1 --2- 172.21.15.78:0/3161469206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c5010d700 0x7f4c5010faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.974+0000 7f4c58800700 1 --2- 172.21.15.78:0/3161469206 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c50067e20 0x7f4c50068200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.974+0000 7f4c58800700 1 --2- 172.21.15.78:0/3161469206 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c50068740 0x7f4c5010d1c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.974+0000 7f4c58800700 1 -- 172.21.15.78:0/3161469206 >> 172.21.15.78:0/3161469206 conn(0x7f4c50076d80 msgr2=0x7f4c50077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.975+0000 7f4c58800700 1 -- 172.21.15.78:0/3161469206 shutdown_connections 2024-03-14T10:00:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.975+0000 7f4c58800700 1 -- 172.21.15.78:0/3161469206 wait complete. 2024-03-14T10:00:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.976+0000 7f4c58800700 1 Processor -- start 2024-03-14T10:00:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.976+0000 7f4c58800700 1 -- start start 2024-03-14T10:00:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.977+0000 7f4c58800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c50067e20 0x7f4c5019f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.977+0000 7f4c58800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c50068740 0x7f4c5019fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:41.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.977+0000 7f4c58800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c5010d700 0x7f4c50199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:41.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.977+0000 7f4c58800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c50199cd0 con 0x7f4c5010d700 2024-03-14T10:00:41.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.977+0000 7f4c55e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c50067e20 0x7f4c5019f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:41.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.977+0000 7f4c58800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c50199e40 con 0x7f4c50068740 2024-03-14T10:00:41.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.977+0000 7f4c58800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5019a000 con 0x7f4c50067e20 2024-03-14T10:00:41.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.977+0000 7f4c55e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c50067e20 0x7f4c5019f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47776/0 (socket says 172.21.15.78:47776) 2024-03-14T10:00:41.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.977+0000 7f4c55e00700 1 -- 172.21.15.78:0/3072686958 learned_addr learned my addr 172.21.15.78:0/3072686958 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.978+0000 7f4c56800700 1 --2- 172.21.15.78:0/3072686958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c5010d700 0x7f4c50199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.978+0000 7f4c55e00700 1 -- 172.21.15.78:0/3072686958 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c50068740 msgr2=0x7f4c5019fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.978+0000 7f4c55e00700 1 --2- 172.21.15.78:0/3072686958 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c50068740 0x7f4c5019fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.978+0000 7f4c55e00700 1 -- 172.21.15.78:0/3072686958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c5010d700 msgr2=0x7f4c50199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.978+0000 7f4c55e00700 1 --2- 172.21.15.78:0/3072686958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c5010d700 0x7f4c50199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.978+0000 7f4c55e00700 1 -- 172.21.15.78:0/3072686958 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c40009520 con 0x7f4c50067e20 2024-03-14T10:00:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.978+0000 7f4c55e00700 1 --2- 172.21.15.78:0/3072686958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c50067e20 0x7f4c5019f540 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400e9c0 tx=0x7f4c4400ed80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.979+0000 7f4c4aa00700 1 -- 172.21.15.78:0/3072686958 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4400cbd0 con 0x7f4c50067e20 2024-03-14T10:00:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.979+0000 7f4c4aa00700 1 -- 172.21.15.78:0/3072686958 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c4400cd30 con 0x7f4c50067e20 2024-03-14T10:00:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.979+0000 7f4c4aa00700 1 -- 172.21.15.78:0/3072686958 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44005580 con 0x7f4c50067e20 2024-03-14T10:00:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.979+0000 7f4c58800700 1 -- 172.21.15.78:0/3072686958 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5019a2e0 con 0x7f4c50067e20 2024-03-14T10:00:41.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.979+0000 7f4c58800700 1 -- 172.21.15.78:0/3072686958 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c501a6530 con 0x7f4c50067e20 2024-03-14T10:00:41.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.983+0000 7f4c4aa00700 1 -- 172.21.15.78:0/3072686958 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f4c44004750 con 0x7f4c50067e20 2024-03-14T10:00:41.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.985+0000 7f4c4aa00700 1 --2- 172.21.15.78:0/3072686958 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4c380a0c80 0x7f4c380a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:41.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.986+0000 7f4c4aa00700 1 -- 172.21.15.78:0/3072686958 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f4c44014070 con 0x7f4c50067e20 2024-03-14T10:00:41.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.986+0000 7f4c55400700 1 --2- 172.21.15.78:0/3072686958 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4c380a0c80 0x7f4c380a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:41.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.987+0000 7f4c55400700 1 --2- 172.21.15.78:0/3072686958 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4c380a0c80 0x7f4c380a3140 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000c010 tx=0x7f4c40009f50 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:41.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.988+0000 7f4c6c34eb80 1 Processor -- start 2024-03-14T10:00:41.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.988+0000 7f4c6c34eb80 1 -- start start 2024-03-14T10:00:41.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.988+0000 7f4c6c34eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0827680 0x55fee0827a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:41.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.988+0000 7f4c6c34eb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee0831980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:41.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.988+0000 7f4c6c34eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0831ec0 0x55fee08342b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:41.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.988+0000 7f4c6c34eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee0780990 con 0x55fee0831ec0 2024-03-14T10:00:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.988+0000 7f4c6c34eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee077e350 con 0x55fee0827680 2024-03-14T10:00:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.988+0000 7f4c6c34eb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee0779c40 con 0x55fee0827fa0 2024-03-14T10:00:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.988+0000 7f4c55400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee0831980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.988+0000 7f4c55400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee0831980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47788/0 (socket says 172.21.15.78:47788) 2024-03-14T10:00:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.988+0000 7f4c55400700 1 -- 172.21.15.78:0/130860730 learned_addr learned my addr 172.21.15.78:0/130860730 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.988+0000 7f4c55e00700 1 --2- 172.21.15.78:0/130860730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0827680 0x55fee0827a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.989+0000 7f4c56800700 1 --2- 172.21.15.78:0/130860730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0831ec0 0x55fee08342b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:41.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.989+0000 7f4c55400700 1 -- 172.21.15.78:0/130860730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0827680 msgr2=0x55fee0827a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.989+0000 7f4c55400700 1 --2- 172.21.15.78:0/130860730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0827680 0x55fee0827a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.989+0000 7f4c55400700 1 -- 172.21.15.78:0/130860730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0831ec0 msgr2=0x55fee08342b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.989+0000 7f4c55400700 1 --2- 172.21.15.78:0/130860730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0831ec0 0x55fee08342b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.989+0000 7f4c55400700 1 -- 172.21.15.78:0/130860730 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fee05fb3b0 con 0x55fee0827fa0 2024-03-14T10:00:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.989+0000 7f4c55400700 1 --2- 172.21.15.78:0/130860730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee0831980 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4c40016000 tx=0x7f4c4001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60d06715a0ff7844 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.990+0000 7f4c3f000700 1 -- 172.21.15.78:0/130860730 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40028070 con 0x55fee0827fa0 2024-03-14T10:00:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.990+0000 7f4c3f000700 1 -- 172.21.15.78:0/130860730 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c4000f920 con 0x55fee0827fa0 2024-03-14T10:00:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.990+0000 7f4c3f000700 1 -- 172.21.15.78:0/130860730 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4001c070 con 0x55fee0827fa0 2024-03-14T10:00:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.991+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/130860730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 msgr2=0x55fee0831980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.991+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/130860730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee0831980 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4c40016000 tx=0x7f4c4001c000 comp rx=0 tx=0).stop 2024-03-14T10:00:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.991+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/130860730 shutdown_connections 2024-03-14T10:00:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.991+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/130860730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0827680 0x55fee0827a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.991+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/130860730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0831ec0 0x55fee08342b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.991+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/130860730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee0831980 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.991+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/130860730 >> 172.21.15.78:0/130860730 conn(0x55fee079d220 msgr2=0x55fee0822aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.991+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/130860730 shutdown_connections 2024-03-14T10:00:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.991+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/130860730 wait complete. 2024-03-14T10:00:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.992+0000 7f4c6c34eb80 1 -- start start 2024-03-14T10:00:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.992+0000 7f4c6c34eb80 1 Processor -- start 2024-03-14T10:00:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c6c34eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0827680 0x55fee082ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c6c34eb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee082f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c6c34eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0831ec0 0x55fee082f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c6c34eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee0715d80 con 0x55fee0827680 2024-03-14T10:00:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c6c34eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee0715a20 con 0x55fee0831ec0 2024-03-14T10:00:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c6c34eb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee05fd6a0 con 0x55fee0827fa0 2024-03-14T10:00:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c6c34eb80 10 client.0 _ll_register_callbacks cb 0x55fee05813a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c55400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee082f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c55400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee082f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47792/0 (socket says 172.21.15.78:47792) 2024-03-14T10:00:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c56800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0831ec0 0x55fee082f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c55400700 1 -- 172.21.15.78:0/2692685415 learned_addr learned my addr 172.21.15.78:0/2692685415 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c55e00700 1 --2- 172.21.15.78:0/2692685415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0827680 0x55fee082ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c56800700 1 -- 172.21.15.78:0/2692685415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 msgr2=0x55fee082f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c56800700 1 --2- 172.21.15.78:0/2692685415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee082f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c56800700 1 -- 172.21.15.78:0/2692685415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0827680 msgr2=0x55fee082ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c56800700 1 --2- 172.21.15.78:0/2692685415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0827680 0x55fee082ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c56800700 1 -- 172.21.15.78:0/2692685415 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c50110620 con 0x55fee0831ec0 2024-03-14T10:00:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.993+0000 7f4c55400700 1 --2- 172.21.15.78:0/2692685415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee082f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.994+0000 7f4c56800700 1 --2- 172.21.15.78:0/2692685415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0831ec0 0x55fee082f7e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c007a50 tx=0x7f4c4c007a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a2a015fceda4b4e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.994+0000 7f4c33400700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T10:00:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.994+0000 7f4c33400700 1 -- 172.21.15.78:0/2692685415 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00e030 con 0x55fee0831ec0 2024-03-14T10:00:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.994+0000 7f4c33400700 1 -- 172.21.15.78:0/2692685415 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f4c4c003c00 con 0x55fee0831ec0 2024-03-14T10:00:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.994+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/2692685415 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fee081a720 con 0x55fee0831ec0 2024-03-14T10:00:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.994+0000 7f4c33400700 1 -- 172.21.15.78:0/2692685415 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c4c003dc0 con 0x55fee0831ec0 2024-03-14T10:00:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.994+0000 7f4c6c34eb80 10 client.26924 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.994+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/2692685415 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55fee05a7b60 con 0x55fee0831ec0 2024-03-14T10:00:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.994+0000 7f4c33400700 1 -- 172.21.15.78:0/2692685415 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fee081a720 con 0x55fee0831ec0 2024-03-14T10:00:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.995+0000 7f4c33400700 1 -- 172.21.15.78:0/2692685415 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4c4c01c020 con 0x55fee0831ec0 2024-03-14T10:00:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.995+0000 7f4c6c34eb80 10 client.26924 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.995+0000 7f4c6c34eb80 10 client.26924 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:00:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.995+0000 7f4c6c34eb80 20 client.26924 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.995+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/2692685415 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55fee0841c30 0x55fee0844050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.995+0000 7f4c6c34eb80 4 client.26924 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:00:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.995+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/2692685415 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fee0772700 con 0x55fee0841c30 2024-03-14T10:00:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.995+0000 7f4c55e00700 1 --2- 172.21.15.78:0/2692685415 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55fee0841c30 0x55fee0844050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.995+0000 7f4c55e00700 1 --2- 172.21.15.78:0/2692685415 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55fee0841c30 0x55fee0844050 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89f8eb6ac26de55a server_cookie=1be0a5d983ce9b9e in_seq=0 out_seq=0 2024-03-14T10:00:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.996+0000 7f4c33400700 10 client.26924 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.996+0000 7f4c33400700 1 -- 172.21.15.78:0/2692685415 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55fee0772700 con 0x55fee0841c30 2024-03-14T10:00:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.996+0000 7f4c33400700 10 client.26924 handle_command_reply: tid=0 2024-03-14T10:00:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.996+0000 7f4c6c34eb80 1 client.26924 shutdown 2024-03-14T10:00:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.996+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/2692685415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0831ec0 msgr2=0x55fee082f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.996+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/2692685415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0831ec0 0x55fee082f7e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c007a50 tx=0x7f4c4c007a80 comp rx=0 tx=0).stop 2024-03-14T10:00:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.997+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/2692685415 shutdown_connections 2024-03-14T10:00:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.997+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/2692685415 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55fee0841c30 0x55fee0844050 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.997+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/2692685415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0831ec0 0x55fee082f7e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.997+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/2692685415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0827680 0x55fee082ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.997+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/2692685415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee082f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.997+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/2692685415 >> 172.21.15.78:0/2692685415 conn(0x55fee079d220 msgr2=0x55fee0822690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.997+0000 7f4c33400700 0 client.26924 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.997+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/2692685415 shutdown_connections 2024-03-14T10:00:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.997+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/2692685415 wait complete. 2024-03-14T10:00:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:41.998+0000 7f4c6c34eb80 20 client.26924 trim_cache size 0 max 16384 2024-03-14T10:00:42.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.024+0000 7f4c6c34eb80 1 Processor -- start 2024-03-14T10:00:42.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.024+0000 7f4c6c34eb80 1 -- start start 2024-03-14T10:00:42.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.024+0000 7f4c6c34eb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827680 0x55fee0822c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.024+0000 7f4c6c34eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0827fa0 0x55fee082dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.025+0000 7f4c6c34eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0831ec0 0x55fee082bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.025+0000 7f4c55e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827680 0x55fee0822c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.025+0000 7f4c6c34eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee0829390 con 0x55fee0827fa0 2024-03-14T10:00:42.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.025+0000 7f4c6c34eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee08302c0 con 0x55fee0831ec0 2024-03-14T10:00:42.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.025+0000 7f4c6c34eb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee0824060 con 0x55fee0827680 2024-03-14T10:00:42.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.025+0000 7f4c55e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827680 0x55fee0822c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47800/0 (socket says 172.21.15.78:47800) 2024-03-14T10:00:42.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.025+0000 7f4c55e00700 1 -- 172.21.15.78:0/2131307194 learned_addr learned my addr 172.21.15.78:0/2131307194 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:42.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.025+0000 7f4c55400700 1 --2- 172.21.15.78:0/2131307194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0827fa0 0x55fee082dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.025+0000 7f4c56800700 1 --2- 172.21.15.78:0/2131307194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0831ec0 0x55fee082bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.025+0000 7f4c55e00700 1 -- 172.21.15.78:0/2131307194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0831ec0 msgr2=0x55fee082bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:42.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.025+0000 7f4c55e00700 1 --2- 172.21.15.78:0/2131307194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0831ec0 0x55fee082bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.025+0000 7f4c55e00700 1 -- 172.21.15.78:0/2131307194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0827fa0 msgr2=0x55fee082dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:42.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.025+0000 7f4c55e00700 1 --2- 172.21.15.78:0/2131307194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0827fa0 0x55fee082dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.025+0000 7f4c55e00700 1 -- 172.21.15.78:0/2131307194 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fee073fea0 con 0x55fee0827680 2024-03-14T10:00:42.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.025+0000 7f4c55400700 1 --2- 172.21.15.78:0/2131307194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0827fa0 0x55fee082dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:42.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.026+0000 7f4c55e00700 1 --2- 172.21.15.78:0/2131307194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827680 0x55fee0822c30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4c44099980 tx=0x7f4c440b9040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dbd70eba98cdf62 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:42.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.026+0000 7f4c3fa00700 1 -- 172.21.15.78:0/2131307194 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c440a1e70 con 0x55fee0827680 2024-03-14T10:00:42.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.026+0000 7f4c3fa00700 1 -- 172.21.15.78:0/2131307194 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c44097d90 con 0x55fee0827680 2024-03-14T10:00:42.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.026+0000 7f4c3fa00700 1 -- 172.21.15.78:0/2131307194 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44016590 con 0x55fee0827680 2024-03-14T10:00:42.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.027+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/2131307194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827680 msgr2=0x55fee0822c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.027+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/2131307194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827680 0x55fee0822c30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4c44099980 tx=0x7f4c440b9040 comp rx=0 tx=0).stop 2024-03-14T10:00:42.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.027+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/2131307194 shutdown_connections 2024-03-14T10:00:42.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.027+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/2131307194 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0831ec0 0x55fee082bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.027+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/2131307194 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827680 0x55fee0822c30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.027+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/2131307194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0827fa0 0x55fee082dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.027+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/2131307194 >> 172.21.15.78:0/2131307194 conn(0x55fee079d220 msgr2=0x55fee082d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:42.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.027+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/2131307194 shutdown_connections 2024-03-14T10:00:42.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.027+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/2131307194 wait complete. 2024-03-14T10:00:42.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.028+0000 7f4c6c34eb80 1 -- start start 2024-03-14T10:00:42.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.028+0000 7f4c6c34eb80 1 Processor -- start 2024-03-14T10:00:42.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.029+0000 7f4c6c34eb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0827680 0x55fee0838820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.029+0000 7f4c6c34eb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee0836d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.029+0000 7f4c6c34eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0831ec0 0x55fee0837240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.029+0000 7f4c6c34eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee08377d0 con 0x55fee0831ec0 2024-03-14T10:00:42.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.029+0000 7f4c6c34eb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee0837940 con 0x55fee0827680 2024-03-14T10:00:42.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.029+0000 7f4c6c34eb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee0837b00 con 0x55fee0827fa0 2024-03-14T10:00:42.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.030+0000 7f4c6c34eb80 10 client.0 _ll_register_callbacks cb 0x7f4c4c007700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:42.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.030+0000 7f4c55400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee0836d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.030+0000 7f4c55400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee0836d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47810/0 (socket says 172.21.15.78:47810) 2024-03-14T10:00:42.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.030+0000 7f4c55400700 1 -- 172.21.15.78:0/784520628 learned_addr learned my addr 172.21.15.78:0/784520628 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:42.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.030+0000 7f4c55e00700 1 --2- 172.21.15.78:0/784520628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0827680 0x55fee0838820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.031+0000 7f4c56800700 1 --2- 172.21.15.78:0/784520628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0831ec0 0x55fee0837240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.031+0000 7f4c55400700 1 -- 172.21.15.78:0/784520628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0827680 msgr2=0x55fee0838820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:42.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.031+0000 7f4c55400700 1 --2- 172.21.15.78:0/784520628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0827680 0x55fee0838820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.031+0000 7f4c55400700 1 -- 172.21.15.78:0/784520628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0831ec0 msgr2=0x55fee0837240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:42.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.031+0000 7f4c55400700 1 --2- 172.21.15.78:0/784520628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0831ec0 0x55fee0837240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.031+0000 7f4c55400700 1 -- 172.21.15.78:0/784520628 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fee05fb3b0 con 0x55fee0827fa0 2024-03-14T10:00:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.031+0000 7f4c56800700 1 --2- 172.21.15.78:0/784520628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0831ec0 0x55fee0837240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.032+0000 7f4c55400700 1 --2- 172.21.15.78:0/784520628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee0836d00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000b9d0 tx=0x7f4c40010f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f4547a3ddb0f951 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.032+0000 7f4c33400700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:00:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.032+0000 7f4c33400700 1 -- 172.21.15.78:0/784520628 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40032070 con 0x55fee0827fa0 2024-03-14T10:00:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.032+0000 7f4c33400700 1 -- 172.21.15.78:0/784520628 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f4c40005250 con 0x55fee0827fa0 2024-03-14T10:00:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.032+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/784520628 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c4000f920 con 0x55fee0827fa0 2024-03-14T10:00:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.032+0000 7f4c33400700 1 -- 172.21.15.78:0/784520628 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c40028040 con 0x55fee0827fa0 2024-03-14T10:00:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.032+0000 7f4c6c34eb80 10 client.26917 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.032+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/784520628 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f4c501a6530 con 0x55fee0827fa0 2024-03-14T10:00:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.033+0000 7f4c33400700 1 -- 172.21.15.78:0/784520628 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c4000f920 con 0x55fee0827fa0 2024-03-14T10:00:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.033+0000 7f4c33400700 1 -- 172.21.15.78:0/784520628 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4c4002d460 con 0x55fee0827fa0 2024-03-14T10:00:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.033+0000 7f4c6c34eb80 10 client.26917 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:42.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.033+0000 7f4c6c34eb80 10 client.26917 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:00:42.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.033+0000 7f4c6c34eb80 20 client.26917 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:42.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.033+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/784520628 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55fee088a830 0x55fee088cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.034+0000 7f4c6c34eb80 4 client.26917 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:00:42.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.034+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/784520628 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fee05c8cf0 con 0x55fee088a830 2024-03-14T10:00:42.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.034+0000 7f4c55e00700 1 --2- 172.21.15.78:0/784520628 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55fee088a830 0x55fee088cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.034+0000 7f4c55e00700 1 --2- 172.21.15.78:0/784520628 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55fee088a830 0x55fee088cc20 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8c96cf5bcfecff8 server_cookie=f4c18af84a132287 in_seq=0 out_seq=0 2024-03-14T10:00:42.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.034+0000 7f4c33400700 10 client.26917 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:42.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.034+0000 7f4c33400700 1 -- 172.21.15.78:0/784520628 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55fee05c8cf0 con 0x55fee088a830 2024-03-14T10:00:42.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.035+0000 7f4c33400700 10 client.26917 handle_command_reply: tid=0 2024-03-14T10:00:42.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.035+0000 7f4c6c34eb80 1 client.26917 shutdown 2024-03-14T10:00:42.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.035+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/784520628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 msgr2=0x55fee0836d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.035+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/784520628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee0836d00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000b9d0 tx=0x7f4c40010f40 comp rx=0 tx=0).stop 2024-03-14T10:00:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.035+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/784520628 shutdown_connections 2024-03-14T10:00:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.035+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/784520628 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55fee088a830 0x55fee088cc20 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.035+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/784520628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fee0827680 0x55fee0838820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.035+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/784520628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fee0831ec0 0x55fee0837240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.035+0000 7f4c33400700 0 client.26917 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.035+0000 7f4c6c34eb80 1 --2- 172.21.15.78:0/784520628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55fee0827fa0 0x55fee0836d00 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.035+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/784520628 >> 172.21.15.78:0/784520628 conn(0x55fee079d220 msgr2=0x55fee082f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.035+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/784520628 shutdown_connections 2024-03-14T10:00:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.035+0000 7f4c6c34eb80 1 -- 172.21.15.78:0/784520628 wait complete. 2024-03-14T10:00:42.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.035+0000 7f4c6c34eb80 20 client.26917 trim_cache size 0 max 16384 2024-03-14T10:00:42.037 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:00:42.038 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T10:00:42.038 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T10:00:42.038 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:00:42.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.036+0000 7f4c58800700 1 -- 172.21.15.78:0/3072686958 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4c380a0c80 msgr2=0x7f4c380a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.036+0000 7f4c58800700 1 --2- 172.21.15.78:0/3072686958 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4c380a0c80 0x7f4c380a3140 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000c010 tx=0x7f4c40009f50 comp rx=0 tx=0).stop 2024-03-14T10:00:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.036+0000 7f4c58800700 1 -- 172.21.15.78:0/3072686958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c50067e20 msgr2=0x7f4c5019f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.036+0000 7f4c58800700 1 --2- 172.21.15.78:0/3072686958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c50067e20 0x7f4c5019f540 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400e9c0 tx=0x7f4c4400ed80 comp rx=0 tx=0).stop 2024-03-14T10:00:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.036+0000 7f4c58800700 1 -- 172.21.15.78:0/3072686958 shutdown_connections 2024-03-14T10:00:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.036+0000 7f4c58800700 1 --2- 172.21.15.78:0/3072686958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c5010d700 0x7f4c50199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.036+0000 7f4c58800700 1 --2- 172.21.15.78:0/3072686958 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4c50067e20 0x7f4c5019f540 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.036+0000 7f4c58800700 1 --2- 172.21.15.78:0/3072686958 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4c380a0c80 0x7f4c380a3140 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.036+0000 7f4c58800700 1 --2- 172.21.15.78:0/3072686958 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c50068740 0x7f4c5019fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.036+0000 7f4c58800700 1 -- 172.21.15.78:0/3072686958 >> 172.21.15.78:0/3072686958 conn(0x7f4c50076d80 msgr2=0x7f4c5010ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.036+0000 7f4c58800700 1 -- 172.21.15.78:0/3072686958 shutdown_connections 2024-03-14T10:00:42.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.036+0000 7f4c58800700 1 -- 172.21.15.78:0/3072686958 wait complete. 2024-03-14T10:00:42.068 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T10:00:42.068 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T10:00:42.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.294+0000 7f5373e00700 1 -- 172.21.15.78:0/943174045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f536c10d2d0 msgr2=0x7f536c10d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.294+0000 7f5373e00700 1 --2- 172.21.15.78:0/943174045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f536c10d2d0 0x7f536c10d6b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5360009990 tx=0x7f5360009ca0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.294+0000 7f5373e00700 1 -- 172.21.15.78:0/943174045 shutdown_connections 2024-03-14T10:00:42.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.294+0000 7f5373e00700 1 --2- 172.21.15.78:0/943174045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f536c10d2d0 0x7f536c10d6b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.294+0000 7f5373e00700 1 --2- 172.21.15.78:0/943174045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f536c107bd0 0x7f536c114070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.294+0000 7f5373e00700 1 --2- 172.21.15.78:0/943174045 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f536c107210 0x7f536c107690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.294+0000 7f5373e00700 1 -- 172.21.15.78:0/943174045 >> 172.21.15.78:0/943174045 conn(0x7f536c076960 msgr2=0x7f536c078d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:42.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.294+0000 7f5373e00700 1 -- 172.21.15.78:0/943174045 shutdown_connections 2024-03-14T10:00:42.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.294+0000 7f5373e00700 1 -- 172.21.15.78:0/943174045 wait complete. 2024-03-14T10:00:42.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.295+0000 7f5373e00700 1 Processor -- start 2024-03-14T10:00:42.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.296+0000 7f5373e00700 1 -- start start 2024-03-14T10:00:42.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.296+0000 7f5373e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f536c107210 0x7f536c19da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.296+0000 7f5373e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f536c107bd0 0x7f536c19df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.296+0000 7f5373e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f536c10d2d0 0x7f536c1a2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.296+0000 7f5373e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c1a2930 con 0x7f536c107bd0 2024-03-14T10:00:42.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.296+0000 7f5373e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c1a2aa0 con 0x7f536c107210 2024-03-14T10:00:42.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.296+0000 7f5373e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c1a2c60 con 0x7f536c10d2d0 2024-03-14T10:00:42.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.296+0000 7f5371400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f536c107210 0x7f536c19da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.296+0000 7f5371400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f536c107210 0x7f536c19da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:33110/0 (socket says 172.21.15.78:33110) 2024-03-14T10:00:42.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.296+0000 7f5371400700 1 -- 172.21.15.78:0/816454306 learned_addr learned my addr 172.21.15.78:0/816454306 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:42.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.296+0000 7f5370a00700 1 --2- 172.21.15.78:0/816454306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f536c107bd0 0x7f536c19df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.296+0000 7f5371e00700 1 --2- 172.21.15.78:0/816454306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f536c10d2d0 0x7f536c1a2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.297+0000 7f5371e00700 1 -- 172.21.15.78:0/816454306 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f536c107210 msgr2=0x7f536c19da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.297+0000 7f5371e00700 1 --2- 172.21.15.78:0/816454306 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f536c107210 0x7f536c19da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.297+0000 7f5371e00700 1 -- 172.21.15.78:0/816454306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f536c107bd0 msgr2=0x7f536c19df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.297+0000 7f5371e00700 1 --2- 172.21.15.78:0/816454306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f536c107bd0 0x7f536c19df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.297+0000 7f5371e00700 1 -- 172.21.15.78:0/816454306 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53600095f0 con 0x7f536c10d2d0 2024-03-14T10:00:42.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.297+0000 7f5371e00700 1 --2- 172.21.15.78:0/816454306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f536c10d2d0 0x7f536c1a2360 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f536400efc0 tx=0x7f536400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:42.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.297+0000 7f536a000700 1 -- 172.21.15.78:0/816454306 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5364018070 con 0x7f536c10d2d0 2024-03-14T10:00:42.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.298+0000 7f5371400700 1 --2- 172.21.15.78:0/816454306 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f536c107210 0x7f536c19da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:42.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.298+0000 7f5373e00700 1 -- 172.21.15.78:0/816454306 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536c1a2ee0 con 0x7f536c10d2d0 2024-03-14T10:00:42.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.298+0000 7f536a000700 1 -- 172.21.15.78:0/816454306 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f536400f040 con 0x7f536c10d2d0 2024-03-14T10:00:42.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.298+0000 7f536a000700 1 -- 172.21.15.78:0/816454306 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536400cde0 con 0x7f536c10d2d0 2024-03-14T10:00:42.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.298+0000 7f5373e00700 1 -- 172.21.15.78:0/816454306 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536c1aa7f0 con 0x7f536c10d2d0 2024-03-14T10:00:42.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.299+0000 7f5373e00700 1 -- 172.21.15.78:0/816454306 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f536c10f750 con 0x7f536c10d2d0 2024-03-14T10:00:42.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.301+0000 7f536a000700 1 -- 172.21.15.78:0/816454306 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f5364004ad0 con 0x7f536c10d2d0 2024-03-14T10:00:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.303+0000 7f536a000700 1 --2- 172.21.15.78:0/816454306 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f53540a0c80 0x7f53540a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.303+0000 7f536a000700 1 -- 172.21.15.78:0/816454306 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f53640cc2c0 con 0x7f536c10d2d0 2024-03-14T10:00:42.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.303+0000 7f5371400700 1 --2- 172.21.15.78:0/816454306 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f53540a0c80 0x7f53540a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.303+0000 7f5371400700 1 --2- 172.21.15.78:0/816454306 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f53540a0c80 0x7f53540a3140 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f536000b790 tx=0x7f5360005ab0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:42.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.305+0000 7f536a000700 1 -- 172.21.15.78:0/816454306 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5364095630 con 0x7f536c10d2d0 2024-03-14T10:00:42.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.621+0000 7f5373e00700 1 -- 172.21.15.78:0/816454306 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f536c108900 con 0x7f536c10d2d0 2024-03-14T10:00:42.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.622+0000 7f536a000700 1 -- 172.21.15.78:0/816454306 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f5364098c50 con 0x7f536c10d2d0 2024-03-14T10:00:42.622 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T10:00:42.623 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T10:00:42.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.624+0000 7f5373e00700 1 -- 172.21.15.78:0/816454306 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f53540a0c80 msgr2=0x7f53540a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.624+0000 7f5373e00700 1 --2- 172.21.15.78:0/816454306 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f53540a0c80 0x7f53540a3140 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f536000b790 tx=0x7f5360005ab0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.624+0000 7f5373e00700 1 -- 172.21.15.78:0/816454306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f536c10d2d0 msgr2=0x7f536c1a2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.624+0000 7f5373e00700 1 --2- 172.21.15.78:0/816454306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f536c10d2d0 0x7f536c1a2360 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f536400efc0 tx=0x7f536400c3a0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.624+0000 7f5373e00700 1 -- 172.21.15.78:0/816454306 shutdown_connections 2024-03-14T10:00:42.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.624+0000 7f5373e00700 1 --2- 172.21.15.78:0/816454306 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f53540a0c80 0x7f53540a3140 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.624+0000 7f5373e00700 1 --2- 172.21.15.78:0/816454306 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f536c107210 0x7f536c19da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.624+0000 7f5373e00700 1 --2- 172.21.15.78:0/816454306 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f536c10d2d0 0x7f536c1a2360 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.624+0000 7f5373e00700 1 --2- 172.21.15.78:0/816454306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f536c107bd0 0x7f536c19df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.624+0000 7f5373e00700 1 -- 172.21.15.78:0/816454306 >> 172.21.15.78:0/816454306 conn(0x7f536c076960 msgr2=0x7f536c0782f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:42.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.624+0000 7f5373e00700 1 -- 172.21.15.78:0/816454306 shutdown_connections 2024-03-14T10:00:42.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.624+0000 7f5373e00700 1 -- 172.21.15.78:0/816454306 wait complete. 2024-03-14T10:00:42.626 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T10:00:42.645 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T10:00:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.909+0000 7f973a600700 1 -- 172.21.15.78:0/4062794740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9734100f20 msgr2=0x7f9734101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.909+0000 7f973a600700 1 --2- 172.21.15.78:0/4062794740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9734100f20 0x7f9734101300 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f97240098f0 tx=0x7f9724009c00 comp rx=0 tx=0).stop 2024-03-14T10:00:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.909+0000 7f973a600700 1 -- 172.21.15.78:0/4062794740 shutdown_connections 2024-03-14T10:00:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.909+0000 7f973a600700 1 --2- 172.21.15.78:0/4062794740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9734106040 0x7f973410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.909+0000 7f973a600700 1 --2- 172.21.15.78:0/4062794740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9734100f20 0x7f9734101300 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.909+0000 7f973a600700 1 --2- 172.21.15.78:0/4062794740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f97341018d0 0x7f9734105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.909+0000 7f973a600700 1 -- 172.21.15.78:0/4062794740 >> 172.21.15.78:0/4062794740 conn(0x7f97340fc7a0 msgr2=0x7f97340febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.910+0000 7f973a600700 1 -- 172.21.15.78:0/4062794740 shutdown_connections 2024-03-14T10:00:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.910+0000 7f973a600700 1 -- 172.21.15.78:0/4062794740 wait complete. 2024-03-14T10:00:42.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.911+0000 7f973a600700 1 Processor -- start 2024-03-14T10:00:42.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.912+0000 7f973a600700 1 -- start start 2024-03-14T10:00:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.912+0000 7f973a600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9734100f20 0x7f973419f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.913+0000 7f973a600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9734106040 0x7f973419fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.913+0000 7f9733400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9734100f20 0x7f973419f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.913+0000 7f973a600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9734199770 0x7f9734199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.913+0000 7f9733400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9734100f20 0x7f973419f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47832/0 (socket says 172.21.15.78:47832) 2024-03-14T10:00:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.913+0000 7f9733400700 1 -- 172.21.15.78:0/3093023059 learned_addr learned my addr 172.21.15.78:0/3093023059 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.913+0000 7f973a600700 1 -- 172.21.15.78:0/3093023059 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973419a1f0 con 0x7f9734199770 2024-03-14T10:00:42.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.913+0000 7f973a600700 1 -- 172.21.15.78:0/3093023059 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973419a360 con 0x7f9734106040 2024-03-14T10:00:42.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.913+0000 7f973a600700 1 -- 172.21.15.78:0/3093023059 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973419a520 con 0x7f9734100f20 2024-03-14T10:00:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.913+0000 7f9732a00700 1 --2- 172.21.15.78:0/3093023059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9734106040 0x7f973419fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.913+0000 7f9733400700 1 -- 172.21.15.78:0/3093023059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9734106040 msgr2=0x7f973419fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.913+0000 7f9733e00700 1 --2- 172.21.15.78:0/3093023059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9734199770 0x7f9734199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.913+0000 7f9733400700 1 --2- 172.21.15.78:0/3093023059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9734106040 0x7f973419fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.913+0000 7f9733400700 1 -- 172.21.15.78:0/3093023059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9734199770 msgr2=0x7f9734199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.913+0000 7f9733400700 1 --2- 172.21.15.78:0/3093023059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9734199770 0x7f9734199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.913+0000 7f9733400700 1 -- 172.21.15.78:0/3093023059 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972c009c40 con 0x7f9734100f20 2024-03-14T10:00:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.913+0000 7f9732a00700 1 --2- 172.21.15.78:0/3093023059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9734106040 0x7f973419fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:00:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.914+0000 7f9733400700 1 --2- 172.21.15.78:0/3093023059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9734100f20 0x7f973419f640 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9724005950 tx=0x7f972400fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.914+0000 7f972be00700 1 -- 172.21.15.78:0/3093023059 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972401c070 con 0x7f9734100f20 2024-03-14T10:00:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.914+0000 7f972be00700 1 -- 172.21.15.78:0/3093023059 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f972400bac0 con 0x7f9734100f20 2024-03-14T10:00:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.914+0000 7f972be00700 1 -- 172.21.15.78:0/3093023059 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9724003970 con 0x7f9734100f20 2024-03-14T10:00:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.914+0000 7f973a600700 1 -- 172.21.15.78:0/3093023059 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9724009520 con 0x7f9734100f20 2024-03-14T10:00:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.914+0000 7f973a600700 1 -- 172.21.15.78:0/3093023059 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97341a6740 con 0x7f9734100f20 2024-03-14T10:00:42.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.917+0000 7f972be00700 1 -- 172.21.15.78:0/3093023059 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f9724020030 con 0x7f9734100f20 2024-03-14T10:00:42.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.919+0000 7f972be00700 1 --2- 172.21.15.78:0/3093023059 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f97180a0c80 0x7f97180a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.919+0000 7f974e1d8b80 1 Processor -- start 2024-03-14T10:00:42.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.919+0000 7f972be00700 1 -- 172.21.15.78:0/3093023059 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f97240cd6b0 con 0x7f9734100f20 2024-03-14T10:00:42.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.919+0000 7f974e1d8b80 1 -- start start 2024-03-14T10:00:42.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.919+0000 7f9732a00700 1 --2- 172.21.15.78:0/3093023059 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f97180a0c80 0x7f97180a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.919+0000 7f974e1d8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1dfb680 0x556fe1dfba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.919+0000 7f974e1d8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfbfa0 0x556fe1e05980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.919+0000 7f974e1d8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 0x556fe1e082b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.919+0000 7f974e1d8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fe1d54990 con 0x556fe1dfbfa0 2024-03-14T10:00:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.919+0000 7f974e1d8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fe1d52350 con 0x556fe1e05ec0 2024-03-14T10:00:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.919+0000 7f974e1d8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fe1d4dc40 con 0x556fe1dfb680 2024-03-14T10:00:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.920+0000 7f9733e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 0x556fe1e082b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.920+0000 7f9733e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 0x556fe1e082b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:33130/0 (socket says 172.21.15.78:33130) 2024-03-14T10:00:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.920+0000 7f9733e00700 1 -- 172.21.15.78:0/609763116 learned_addr learned my addr 172.21.15.78:0/609763116 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.920+0000 7f9732a00700 1 --2- 172.21.15.78:0/609763116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfbfa0 0x556fe1e05980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.920+0000 7f9733e00700 1 -- 172.21.15.78:0/609763116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1dfb680 msgr2=0x556fe1dfba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.920+0000 7f9733e00700 1 --2- 172.21.15.78:0/609763116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1dfb680 0x556fe1dfba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.920+0000 7f9733e00700 1 -- 172.21.15.78:0/609763116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfbfa0 msgr2=0x556fe1e05980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.920+0000 7f9733e00700 1 --2- 172.21.15.78:0/609763116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfbfa0 0x556fe1e05980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.920+0000 7f9733e00700 1 -- 172.21.15.78:0/609763116 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556fe1bcf3b0 con 0x556fe1e05ec0 2024-03-14T10:00:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.920+0000 7f9732a00700 1 --2- 172.21.15.78:0/3093023059 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f97180a0c80 0x7f97180a3140 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f973419ae80 tx=0x7f97200116c0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.920+0000 7f9733e00700 1 --2- 172.21.15.78:0/609763116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 0x556fe1e082b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f972c0094e0 tx=0x7f972c009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba6718dba3888ffd server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.921+0000 7f9728c00700 1 -- 172.21.15.78:0/609763116 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c0155f0 con 0x556fe1e05ec0 2024-03-14T10:00:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.921+0000 7f9728c00700 1 -- 172.21.15.78:0/609763116 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f972c011040 con 0x556fe1e05ec0 2024-03-14T10:00:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.921+0000 7f9728c00700 1 -- 172.21.15.78:0/609763116 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c012c70 con 0x556fe1e05ec0 2024-03-14T10:00:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.921+0000 7f974e1d8b80 1 -- 172.21.15.78:0/609763116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 msgr2=0x556fe1e082b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.921+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/609763116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 0x556fe1e082b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f972c0094e0 tx=0x7f972c009920 comp rx=0 tx=0).stop 2024-03-14T10:00:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.922+0000 7f974e1d8b80 1 -- 172.21.15.78:0/609763116 shutdown_connections 2024-03-14T10:00:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.922+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/609763116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 0x556fe1e082b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.922+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/609763116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1dfb680 0x556fe1dfba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.922+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/609763116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfbfa0 0x556fe1e05980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.922+0000 7f974e1d8b80 1 -- 172.21.15.78:0/609763116 >> 172.21.15.78:0/609763116 conn(0x556fe1d71220 msgr2=0x556fe1df6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.922+0000 7f974e1d8b80 1 -- 172.21.15.78:0/609763116 shutdown_connections 2024-03-14T10:00:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.922+0000 7f974e1d8b80 1 -- 172.21.15.78:0/609763116 wait complete. 2024-03-14T10:00:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.922+0000 7f974e1d8b80 1 -- start start 2024-03-14T10:00:42.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.922+0000 7f974e1d8b80 1 Processor -- start 2024-03-14T10:00:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.923+0000 7f974e1d8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfb680 0x556fe1e02df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.923+0000 7f974e1d8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1dfbfa0 0x556fe1e03330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.923+0000 7f974e1d8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1e05ec0 0x556fe1e03870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.923+0000 7f974e1d8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fe1ce9d80 con 0x556fe1dfb680 2024-03-14T10:00:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.923+0000 7f974e1d8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fe1ce9a20 con 0x556fe1dfbfa0 2024-03-14T10:00:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.923+0000 7f974e1d8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fe1bd16a0 con 0x556fe1e05ec0 2024-03-14T10:00:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.923+0000 7f974e1d8b80 10 client.0 _ll_register_callbacks cb 0x556fe1b553a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.923+0000 7f9732a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1dfbfa0 0x556fe1e03330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.923+0000 7f9732a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1dfbfa0 0x556fe1e03330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:33134/0 (socket says 172.21.15.78:33134) 2024-03-14T10:00:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.923+0000 7f9732a00700 1 -- 172.21.15.78:0/3089348870 learned_addr learned my addr 172.21.15.78:0/3089348870 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.923+0000 7f9733400700 1 --2- 172.21.15.78:0/3089348870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfb680 0x556fe1e02df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.924+0000 7f9733e00700 1 --2- 172.21.15.78:0/3089348870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1e05ec0 0x556fe1e03870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.924+0000 7f9732a00700 1 -- 172.21.15.78:0/3089348870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1e05ec0 msgr2=0x556fe1e03870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.924+0000 7f9732a00700 1 --2- 172.21.15.78:0/3089348870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1e05ec0 0x556fe1e03870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.924+0000 7f9732a00700 1 -- 172.21.15.78:0/3089348870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfb680 msgr2=0x556fe1e02df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.924+0000 7f9732a00700 1 --2- 172.21.15.78:0/3089348870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfb680 0x556fe1e02df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.924+0000 7f9732a00700 1 -- 172.21.15.78:0/3089348870 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972c009190 con 0x556fe1dfbfa0 2024-03-14T10:00:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.924+0000 7f9732a00700 1 --2- 172.21.15.78:0/3089348870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1dfbfa0 0x556fe1e03330 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9720014c30 tx=0x7f97200047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79d1c4018df33f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.924+0000 7f971d600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T10:00:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.925+0000 7f971d600700 1 -- 172.21.15.78:0/3089348870 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9720011e60 con 0x556fe1dfbfa0 2024-03-14T10:00:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.925+0000 7f971d600700 1 -- 172.21.15.78:0/3089348870 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f972001a040 con 0x556fe1dfbfa0 2024-03-14T10:00:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.925+0000 7f974e1d8b80 1 -- 172.21.15.78:0/3089348870 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556fe1dee720 con 0x556fe1dfbfa0 2024-03-14T10:00:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.925+0000 7f971d600700 1 -- 172.21.15.78:0/3089348870 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97200142e0 con 0x556fe1dfbfa0 2024-03-14T10:00:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.925+0000 7f974e1d8b80 10 client.26948 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.925+0000 7f974e1d8b80 1 -- 172.21.15.78:0/3089348870 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556fe1b7bb60 con 0x556fe1dfbfa0 2024-03-14T10:00:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.925+0000 7f971d600700 1 -- 172.21.15.78:0/3089348870 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556fe1dee720 con 0x556fe1dfbfa0 2024-03-14T10:00:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.925+0000 7f971d600700 1 -- 172.21.15.78:0/3089348870 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9720012a00 con 0x556fe1dfbfa0 2024-03-14T10:00:42.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.925+0000 7f974e1d8b80 10 client.26948 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.925+0000 7f974e1d8b80 10 client.26948 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:00:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.925+0000 7f974e1d8b80 20 client.26948 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.926+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/3089348870 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x556fe1e15c00 0x556fe1e18020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.926+0000 7f974e1d8b80 4 client.26948 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:00:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.926+0000 7f974e1d8b80 1 -- 172.21.15.78:0/3089348870 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556fe1d46700 con 0x556fe1e15c00 2024-03-14T10:00:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.926+0000 7f9733400700 1 --2- 172.21.15.78:0/3089348870 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x556fe1e15c00 0x556fe1e18020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.926+0000 7f9733400700 1 --2- 172.21.15.78:0/3089348870 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x556fe1e15c00 0x556fe1e18020 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=89d0824608f1d324 server_cookie=3f470add251940bc in_seq=0 out_seq=0 2024-03-14T10:00:42.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.926+0000 7f971d600700 10 client.26948 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:00:42.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.927+0000 7f971d600700 1 -- 172.21.15.78:0/3089348870 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x556fe1d46700 con 0x556fe1e15c00 2024-03-14T10:00:42.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.927+0000 7f971d600700 10 client.26948 handle_command_reply: tid=0 2024-03-14T10:00:42.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.927+0000 7f974e1d8b80 1 client.26948 shutdown 2024-03-14T10:00:42.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.927+0000 7f974e1d8b80 1 -- 172.21.15.78:0/3089348870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1dfbfa0 msgr2=0x556fe1e03330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.927+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/3089348870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1dfbfa0 0x556fe1e03330 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9720014c30 tx=0x7f97200047c0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.927+0000 7f974e1d8b80 1 -- 172.21.15.78:0/3089348870 shutdown_connections 2024-03-14T10:00:42.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.927+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/3089348870 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x556fe1e15c00 0x556fe1e18020 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.927+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/3089348870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1e05ec0 0x556fe1e03870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.927+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/3089348870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfb680 0x556fe1e02df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.927+0000 7f971d600700 0 client.26948 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:00:42.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.927+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/3089348870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1dfbfa0 0x556fe1e03330 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.927+0000 7f974e1d8b80 1 -- 172.21.15.78:0/3089348870 >> 172.21.15.78:0/3089348870 conn(0x556fe1d71220 msgr2=0x556fe1df6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:42.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.927+0000 7f974e1d8b80 1 -- 172.21.15.78:0/3089348870 shutdown_connections 2024-03-14T10:00:42.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.928+0000 7f974e1d8b80 1 -- 172.21.15.78:0/3089348870 wait complete. 2024-03-14T10:00:42.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.928+0000 7f974e1d8b80 20 client.26948 trim_cache size 0 max 16384 2024-03-14T10:00:42.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.946+0000 7f974e1d8b80 1 Processor -- start 2024-03-14T10:00:42.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.946+0000 7f974e1d8b80 1 -- start start 2024-03-14T10:00:42.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.946+0000 7f974e1d8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfb680 0x556fe1e026b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.947+0000 7f974e1d8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1dfbfa0 0x556fe1e1c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.947+0000 7f974e1d8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 0x556fe1dfe0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.947+0000 7f974e1d8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fe1df0d00 con 0x556fe1dfb680 2024-03-14T10:00:42.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.947+0000 7f974e1d8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fe1e13300 con 0x556fe1e05ec0 2024-03-14T10:00:42.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.947+0000 7f974e1d8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fe1dfec30 con 0x556fe1dfbfa0 2024-03-14T10:00:42.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.947+0000 7f9733400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfb680 0x556fe1e026b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.947+0000 7f9733400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfb680 0x556fe1e026b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55000/0 (socket says 172.21.15.78:55000) 2024-03-14T10:00:42.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.947+0000 7f9733400700 1 -- 172.21.15.78:0/428480264 learned_addr learned my addr 172.21.15.78:0/428480264 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:42.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.947+0000 7f9733e00700 1 --2- 172.21.15.78:0/428480264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 0x556fe1dfe0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.948+0000 7f9733400700 1 -- 172.21.15.78:0/428480264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1dfbfa0 msgr2=0x556fe1e1c980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-14T10:00:42.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.948+0000 7f9733400700 1 --2- 172.21.15.78:0/428480264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1dfbfa0 0x556fe1e1c980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.948+0000 7f9733400700 1 -- 172.21.15.78:0/428480264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 msgr2=0x556fe1dfe0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:42.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.948+0000 7f9733400700 1 --2- 172.21.15.78:0/428480264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 0x556fe1dfe0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.948+0000 7f9733400700 1 -- 172.21.15.78:0/428480264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556fe1d13ea0 con 0x556fe1dfb680 2024-03-14T10:00:42.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.948+0000 7f9733400700 1 --2- 172.21.15.78:0/428480264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfb680 0x556fe1e026b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f972409a620 tx=0x7f9724009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fbe5e3fb7427120 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:42.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.949+0000 7f9729600700 1 -- 172.21.15.78:0/428480264 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972409a6e0 con 0x556fe1dfb680 2024-03-14T10:00:42.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.949+0000 7f9729600700 1 -- 172.21.15.78:0/428480264 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f972409e510 con 0x556fe1dfb680 2024-03-14T10:00:42.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.949+0000 7f9729600700 1 -- 172.21.15.78:0/428480264 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9724096df0 con 0x556fe1dfb680 2024-03-14T10:00:42.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.950+0000 7f974e1d8b80 1 -- 172.21.15.78:0/428480264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfb680 msgr2=0x556fe1e026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.950+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/428480264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfb680 0x556fe1e026b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f972409a620 tx=0x7f9724009260 comp rx=0 tx=0).stop 2024-03-14T10:00:42.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.950+0000 7f974e1d8b80 1 -- 172.21.15.78:0/428480264 shutdown_connections 2024-03-14T10:00:42.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.950+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/428480264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 0x556fe1dfe0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.950+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/428480264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1dfb680 0x556fe1e026b0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.950+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/428480264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1dfbfa0 0x556fe1e1c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.950+0000 7f974e1d8b80 1 -- 172.21.15.78:0/428480264 >> 172.21.15.78:0/428480264 conn(0x556fe1d71220 msgr2=0x556fe1e11190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:42.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.950+0000 7f974e1d8b80 1 -- 172.21.15.78:0/428480264 shutdown_connections 2024-03-14T10:00:42.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.951+0000 7f974e1d8b80 1 -- 172.21.15.78:0/428480264 wait complete. 2024-03-14T10:00:42.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.951+0000 7f974e1d8b80 1 -- start start 2024-03-14T10:00:42.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.951+0000 7f974e1d8b80 1 Processor -- start 2024-03-14T10:00:42.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.952+0000 7f974e1d8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1dfbfa0 0x556fe1e0ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.952+0000 7f974e1d8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 0x556fe1e0b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.952+0000 7f974e1d8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1e64710 0x556fe1e66b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.952+0000 7f974e1d8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fe1e0b9b0 con 0x556fe1e64710 2024-03-14T10:00:42.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.952+0000 7f974e1d8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fe1e67040 con 0x556fe1e05ec0 2024-03-14T10:00:42.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.952+0000 7f974e1d8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fe1e67180 con 0x556fe1dfbfa0 2024-03-14T10:00:42.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.952+0000 7f974e1d8b80 10 client.0 _ll_register_callbacks cb 0x7f9720014810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:42.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.953+0000 7f9733400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1dfbfa0 0x556fe1e0ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.953+0000 7f9732a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 0x556fe1e0b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.953+0000 7f9733400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1dfbfa0 0x556fe1e0ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47862/0 (socket says 172.21.15.78:47862) 2024-03-14T10:00:42.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.953+0000 7f9733400700 1 -- 172.21.15.78:0/2910824733 learned_addr learned my addr 172.21.15.78:0/2910824733 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:42.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.953+0000 7f9733e00700 1 --2- 172.21.15.78:0/2910824733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1e64710 0x556fe1e66b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.953+0000 7f9732a00700 1 -- 172.21.15.78:0/2910824733 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1dfbfa0 msgr2=0x556fe1e0ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:42.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.953+0000 7f9732a00700 1 --2- 172.21.15.78:0/2910824733 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1dfbfa0 0x556fe1e0ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.953+0000 7f9732a00700 1 -- 172.21.15.78:0/2910824733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1e64710 msgr2=0x556fe1e66b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:42.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.953+0000 7f9732a00700 1 --2- 172.21.15.78:0/2910824733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1e64710 0x556fe1e66b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.953+0000 7f9732a00700 1 -- 172.21.15.78:0/2910824733 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97341a6740 con 0x556fe1e05ec0 2024-03-14T10:00:42.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.953+0000 7f9733e00700 1 --2- 172.21.15.78:0/2910824733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1e64710 0x556fe1e66b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:00:42.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.953+0000 7f9732a00700 1 --2- 172.21.15.78:0/2910824733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 0x556fe1e0b2d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9720012d90 tx=0x7f9720012780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89bdfb22886f984e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:42.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.953+0000 7f971d600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T10:00:42.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.954+0000 7f971d600700 1 -- 172.21.15.78:0/2910824733 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972000f8b0 con 0x556fe1e05ec0 2024-03-14T10:00:42.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.954+0000 7f971d600700 1 -- 172.21.15.78:0/2910824733 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f972001a070 con 0x556fe1e05ec0 2024-03-14T10:00:42.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.954+0000 7f974e1d8b80 1 -- 172.21.15.78:0/2910824733 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f972c011040 con 0x556fe1e05ec0 2024-03-14T10:00:42.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.954+0000 7f971d600700 1 -- 172.21.15.78:0/2910824733 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9720005a00 con 0x556fe1e05ec0 2024-03-14T10:00:42.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.954+0000 7f974e1d8b80 10 client.26954 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:42.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.954+0000 7f974e1d8b80 1 -- 172.21.15.78:0/2910824733 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556fe1bd3cd0 con 0x556fe1e05ec0 2024-03-14T10:00:42.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.954+0000 7f971d600700 1 -- 172.21.15.78:0/2910824733 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f972c011040 con 0x556fe1e05ec0 2024-03-14T10:00:42.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.954+0000 7f971d600700 1 -- 172.21.15.78:0/2910824733 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f972001f6e0 con 0x556fe1e05ec0 2024-03-14T10:00:42.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.954+0000 7f974e1d8b80 10 client.26954 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:42.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.954+0000 7f974e1d8b80 10 client.26954 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:00:42.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.954+0000 7f974e1d8b80 20 client.26954 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:42.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.954+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/2910824733 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x556fe1e68970 0x556fe1e6ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:42.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.954+0000 7f974e1d8b80 4 client.26954 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:00:42.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.954+0000 7f974e1d8b80 1 -- 172.21.15.78:0/2910824733 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556fe1b9ccf0 con 0x556fe1e68970 2024-03-14T10:00:42.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.955+0000 7f9733400700 1 --2- 172.21.15.78:0/2910824733 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x556fe1e68970 0x556fe1e6ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:42.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.955+0000 7f9733400700 1 --2- 172.21.15.78:0/2910824733 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x556fe1e68970 0x556fe1e6ad90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b6f3c2841f028f02 server_cookie=c36349bfc973a8c3 in_seq=0 out_seq=0 2024-03-14T10:00:42.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.955+0000 7f971d600700 10 client.26954 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:00:42.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.955+0000 7f971d600700 1 -- 172.21.15.78:0/2910824733 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556fe1b9ccf0 con 0x556fe1e68970 2024-03-14T10:00:42.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.955+0000 7f971d600700 10 client.26954 handle_command_reply: tid=0 2024-03-14T10:00:42.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.955+0000 7f974e1d8b80 1 client.26954 shutdown 2024-03-14T10:00:42.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.956+0000 7f974e1d8b80 1 -- 172.21.15.78:0/2910824733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 msgr2=0x556fe1e0b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.956+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/2910824733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 0x556fe1e0b2d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9720012d90 tx=0x7f9720012780 comp rx=0 tx=0).stop 2024-03-14T10:00:42.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.956+0000 7f974e1d8b80 1 -- 172.21.15.78:0/2910824733 shutdown_connections 2024-03-14T10:00:42.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.956+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/2910824733 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x556fe1e68970 0x556fe1e6ad90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.956+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/2910824733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556fe1e64710 0x556fe1e66b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.956+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/2910824733 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x556fe1dfbfa0 0x556fe1e0ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.956+0000 7f974e1d8b80 1 --2- 172.21.15.78:0/2910824733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556fe1e05ec0 0x556fe1e0b2d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.956+0000 7f974e1d8b80 1 -- 172.21.15.78:0/2910824733 >> 172.21.15.78:0/2910824733 conn(0x556fe1d71220 msgr2=0x556fe1dfe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:42.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.956+0000 7f971d600700 0 client.26954 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:00:42.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.956+0000 7f974e1d8b80 1 -- 172.21.15.78:0/2910824733 shutdown_connections 2024-03-14T10:00:42.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.956+0000 7f974e1d8b80 1 -- 172.21.15.78:0/2910824733 wait complete. 2024-03-14T10:00:42.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.956+0000 7f974e1d8b80 20 client.26954 trim_cache size 0 max 16384 2024-03-14T10:00:42.959 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:00:42.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.957+0000 7f973a600700 1 -- 172.21.15.78:0/3093023059 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f97180a0c80 msgr2=0x7f97180a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.957+0000 7f973a600700 1 --2- 172.21.15.78:0/3093023059 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f97180a0c80 0x7f97180a3140 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f973419ae80 tx=0x7f97200116c0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.957+0000 7f973a600700 1 -- 172.21.15.78:0/3093023059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9734100f20 msgr2=0x7f973419f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:42.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.957+0000 7f973a600700 1 --2- 172.21.15.78:0/3093023059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9734100f20 0x7f973419f640 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9724005950 tx=0x7f972400fa00 comp rx=0 tx=0).stop 2024-03-14T10:00:42.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.957+0000 7f973a600700 1 -- 172.21.15.78:0/3093023059 shutdown_connections 2024-03-14T10:00:42.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.957+0000 7f973a600700 1 --2- 172.21.15.78:0/3093023059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9734199770 0x7f9734199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.957+0000 7f973a600700 1 --2- 172.21.15.78:0/3093023059 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9734100f20 0x7f973419f640 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.957+0000 7f973a600700 1 --2- 172.21.15.78:0/3093023059 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f97180a0c80 0x7f97180a3140 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.957+0000 7f973a600700 1 --2- 172.21.15.78:0/3093023059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9734106040 0x7f973419fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:42.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.957+0000 7f973a600700 1 -- 172.21.15.78:0/3093023059 >> 172.21.15.78:0/3093023059 conn(0x7f97340fc7a0 msgr2=0x7f97340fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:42.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.958+0000 7f973a600700 1 -- 172.21.15.78:0/3093023059 shutdown_connections 2024-03-14T10:00:42.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:42.958+0000 7f973a600700 1 -- 172.21.15.78:0/3093023059 wait complete. 2024-03-14T10:00:42.983 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T10:00:43.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.211+0000 7ff792000700 1 -- 172.21.15.78:0/2146877597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff78c106050 msgr2=0x7ff78c10dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.211+0000 7ff792000700 1 --2- 172.21.15.78:0/2146877597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff78c106050 0x7ff78c10dbd0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff784009f90 tx=0x7ff78400b330 comp rx=0 tx=0).stop 2024-03-14T10:00:43.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.211+0000 7ff792000700 1 -- 172.21.15.78:0/2146877597 shutdown_connections 2024-03-14T10:00:43.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.211+0000 7ff792000700 1 --2- 172.21.15.78:0/2146877597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff78c100f30 0x7ff78c101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.211+0000 7ff792000700 1 --2- 172.21.15.78:0/2146877597 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff78c106050 0x7ff78c10dbd0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.211+0000 7ff792000700 1 --2- 172.21.15.78:0/2146877597 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff78c1018e0 0x7ff78c105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.211+0000 7ff792000700 1 -- 172.21.15.78:0/2146877597 >> 172.21.15.78:0/2146877597 conn(0x7ff78c0fc7b0 msgr2=0x7ff78c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:43.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.211+0000 7ff792000700 1 -- 172.21.15.78:0/2146877597 shutdown_connections 2024-03-14T10:00:43.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.211+0000 7ff792000700 1 -- 172.21.15.78:0/2146877597 wait complete. 2024-03-14T10:00:43.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.212+0000 7ff792000700 1 Processor -- start 2024-03-14T10:00:43.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.212+0000 7ff792000700 1 -- start start 2024-03-14T10:00:43.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.213+0000 7ff792000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff78c100f30 0x7ff78c19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.213+0000 7ff792000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff78c106050 0x7ff78c19fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.213+0000 7ff792000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff78c199760 0x7ff78c199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.213+0000 7ff792000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c19a1e0 con 0x7ff78c100f30 2024-03-14T10:00:43.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.213+0000 7ff792000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c19a350 con 0x7ff78c199760 2024-03-14T10:00:43.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.213+0000 7ff792000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c19a510 con 0x7ff78c106050 2024-03-14T10:00:43.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.213+0000 7ff78aa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff78c106050 0x7ff78c19fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.213+0000 7ff78b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff78c100f30 0x7ff78c19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.213+0000 7ff78aa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff78c106050 0x7ff78c19fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47878/0 (socket says 172.21.15.78:47878) 2024-03-14T10:00:43.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.213+0000 7ff78aa00700 1 -- 172.21.15.78:0/470648824 learned_addr learned my addr 172.21.15.78:0/470648824 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:43.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.213+0000 7ff78be00700 1 --2- 172.21.15.78:0/470648824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff78c199760 0x7ff78c199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.214+0000 7ff78aa00700 1 -- 172.21.15.78:0/470648824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff78c199760 msgr2=0x7ff78c199c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.214+0000 7ff78aa00700 1 --2- 172.21.15.78:0/470648824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff78c199760 0x7ff78c199c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.214+0000 7ff78aa00700 1 -- 172.21.15.78:0/470648824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff78c100f30 msgr2=0x7ff78c19f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.214+0000 7ff78aa00700 1 --2- 172.21.15.78:0/470648824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff78c100f30 0x7ff78c19f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.214+0000 7ff78aa00700 1 -- 172.21.15.78:0/470648824 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff784009c40 con 0x7ff78c106050 2024-03-14T10:00:43.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.214+0000 7ff78aa00700 1 --2- 172.21.15.78:0/470648824 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff78c106050 0x7ff78c19fb70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff77800e9f0 tx=0x7ff77800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:43.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.214+0000 7ff783e00700 1 -- 172.21.15.78:0/470648824 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff778009690 con 0x7ff78c106050 2024-03-14T10:00:43.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.214+0000 7ff792000700 1 -- 172.21.15.78:0/470648824 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78c1a6500 con 0x7ff78c106050 2024-03-14T10:00:43.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.214+0000 7ff783e00700 1 -- 172.21.15.78:0/470648824 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff77800ce80 con 0x7ff78c106050 2024-03-14T10:00:43.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.214+0000 7ff783e00700 1 -- 172.21.15.78:0/470648824 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff778018420 con 0x7ff78c106050 2024-03-14T10:00:43.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.214+0000 7ff792000700 1 -- 172.21.15.78:0/470648824 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78c1a6a30 con 0x7ff78c106050 2024-03-14T10:00:43.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.217+0000 7ff783e00700 1 -- 172.21.15.78:0/470648824 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7ff778018580 con 0x7ff78c106050 2024-03-14T10:00:43.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.219+0000 7ff783e00700 1 --2- 172.21.15.78:0/470648824 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff7700a0c00 0x7ff7700a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.219+0000 7ff783e00700 1 -- 172.21.15.78:0/470648824 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7ff778014070 con 0x7ff78c106050 2024-03-14T10:00:43.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.219+0000 7ff78b400700 1 --2- 172.21.15.78:0/470648824 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff7700a0c00 0x7ff7700a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.219+0000 7ff7a5a6bb80 1 Processor -- start 2024-03-14T10:00:43.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.219+0000 7ff7a5a6bb80 1 -- start start 2024-03-14T10:00:43.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff7a5a6bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a21680 0x555563a21a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff7a5a6bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 0x555563a2b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff7a5a6bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a2bec0 0x555563a2e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff7a5a6bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55556397a990 con 0x555563a21680 2024-03-14T10:00:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff7a5a6bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x555563978350 con 0x555563a21fa0 2024-03-14T10:00:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff7a5a6bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x555563973c40 con 0x555563a2bec0 2024-03-14T10:00:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff78aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 0x555563a2b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff78b400700 1 --2- 172.21.15.78:0/470648824 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff7700a0c00 0x7ff7700a30c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009c60 tx=0x7ff77c009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff78aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 0x555563a2b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:33182/0 (socket says 172.21.15.78:33182) 2024-03-14T10:00:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff78aa00700 1 -- 172.21.15.78:0/431388658 learned_addr learned my addr 172.21.15.78:0/431388658 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff78b400700 1 --2- 172.21.15.78:0/431388658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a21680 0x555563a21a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff78aa00700 1 -- 172.21.15.78:0/431388658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a2bec0 msgr2=0x555563a2e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff78be00700 1 --2- 172.21.15.78:0/431388658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a2bec0 0x555563a2e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff78aa00700 1 --2- 172.21.15.78:0/431388658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a2bec0 0x555563a2e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff78aa00700 1 -- 172.21.15.78:0/431388658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a21680 msgr2=0x555563a21a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:43.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff78aa00700 1 --2- 172.21.15.78:0/431388658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a21680 0x555563a21a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff78aa00700 1 -- 172.21.15.78:0/431388658 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555637f53b0 con 0x555563a21fa0 2024-03-14T10:00:43.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.220+0000 7ff78be00700 1 --2- 172.21.15.78:0/431388658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a2bec0 0x555563a2e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T10:00:43.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.221+0000 7ff78aa00700 1 --2- 172.21.15.78:0/431388658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 0x555563a2b980 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff7780a16e0 tx=0x7ff778099170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a588173e7d3e6a8d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:43.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.221+0000 7ff777e00700 1 -- 172.21.15.78:0/431388658 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff778099790 con 0x555563a21fa0 2024-03-14T10:00:43.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.221+0000 7ff777e00700 1 -- 172.21.15.78:0/431388658 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff77809d5c0 con 0x555563a21fa0 2024-03-14T10:00:43.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.221+0000 7ff777e00700 1 -- 172.21.15.78:0/431388658 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7780cc3f0 con 0x555563a21fa0 2024-03-14T10:00:43.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.221+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/431388658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 msgr2=0x555563a2b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.221+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/431388658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 0x555563a2b980 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff7780a16e0 tx=0x7ff778099170 comp rx=0 tx=0).stop 2024-03-14T10:00:43.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.222+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/431388658 shutdown_connections 2024-03-14T10:00:43.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.222+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/431388658 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a2bec0 0x555563a2e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.222+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/431388658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a21680 0x555563a21a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.222+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/431388658 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 0x555563a2b980 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.222+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/431388658 >> 172.21.15.78:0/431388658 conn(0x555563997220 msgr2=0x555563a1caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.222+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/431388658 shutdown_connections 2024-03-14T10:00:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.222+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/431388658 wait complete. 2024-03-14T10:00:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.222+0000 7ff7a5a6bb80 1 -- start start 2024-03-14T10:00:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.222+0000 7ff7a5a6bb80 1 Processor -- start 2024-03-14T10:00:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.223+0000 7ff7a5a6bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a21680 0x555563a2b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.223+0000 7ff7a5a6bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 0x555563a28c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.223+0000 7ff7a5a6bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a2bec0 0x555563a293d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.223+0000 7ff7a5a6bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55556390fd80 con 0x555563a2bec0 2024-03-14T10:00:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.223+0000 7ff7a5a6bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55556390fa20 con 0x555563a21fa0 2024-03-14T10:00:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.223+0000 7ff7a5a6bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555637f76a0 con 0x555563a21680 2024-03-14T10:00:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.223+0000 7ff7a5a6bb80 10 client.0 _ll_register_callbacks cb 0x55556377b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.224+0000 7ff78b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a21680 0x555563a2b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.224+0000 7ff78b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a21680 0x555563a2b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47894/0 (socket says 172.21.15.78:47894) 2024-03-14T10:00:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.224+0000 7ff78b400700 1 -- 172.21.15.78:0/3538256312 learned_addr learned my addr 172.21.15.78:0/3538256312 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.224+0000 7ff78aa00700 1 --2- 172.21.15.78:0/3538256312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 0x555563a28c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.224+0000 7ff78be00700 1 --2- 172.21.15.78:0/3538256312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a2bec0 0x555563a293d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.224+0000 7ff78b400700 1 -- 172.21.15.78:0/3538256312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 msgr2=0x555563a28c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.224+0000 7ff78b400700 1 --2- 172.21.15.78:0/3538256312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 0x555563a28c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.224+0000 7ff78b400700 1 -- 172.21.15.78:0/3538256312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a2bec0 msgr2=0x555563a293d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.224+0000 7ff78b400700 1 --2- 172.21.15.78:0/3538256312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a2bec0 0x555563a293d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.224+0000 7ff78b400700 1 -- 172.21.15.78:0/3538256312 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78c1a6a30 con 0x555563a21680 2024-03-14T10:00:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.224+0000 7ff78b400700 1 --2- 172.21.15.78:0/3538256312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a21680 0x555563a2b950 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff77c003900 tx=0x7ff77c0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eebc147140dd68a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.224+0000 7ff774c00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:00:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.225+0000 7ff774c00700 1 -- 172.21.15.78:0/3538256312 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c022070 con 0x555563a21680 2024-03-14T10:00:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.225+0000 7ff774c00700 1 -- 172.21.15.78:0/3538256312 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7ff77c01d820 con 0x555563a21680 2024-03-14T10:00:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.225+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/3538256312 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555563a14720 con 0x555563a21680 2024-03-14T10:00:43.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.225+0000 7ff774c00700 1 -- 172.21.15.78:0/3538256312 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff77c0045a0 con 0x555563a21680 2024-03-14T10:00:43.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.225+0000 7ff7a5a6bb80 10 client.26944 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:43.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.225+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/3538256312 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5555637a1b60 con 0x555563a21680 2024-03-14T10:00:43.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.225+0000 7ff774c00700 1 -- 172.21.15.78:0/3538256312 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555563a14720 con 0x555563a21680 2024-03-14T10:00:43.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.226+0000 7ff774c00700 1 -- 172.21.15.78:0/3538256312 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff77c027ac0 con 0x555563a21680 2024-03-14T10:00:43.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.226+0000 7ff7a5a6bb80 10 client.26944 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:43.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.226+0000 7ff7a5a6bb80 10 client.26944 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:00:43.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.226+0000 7ff7a5a6bb80 20 client.26944 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:43.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.226+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/3538256312 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x555563a3bc20 0x555563a3e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.226+0000 7ff7a5a6bb80 4 client.26944 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:00:43.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.226+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/3538256312 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55556396c700 con 0x555563a3bc20 2024-03-14T10:00:43.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.226+0000 7ff78aa00700 1 --2- 172.21.15.78:0/3538256312 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x555563a3bc20 0x555563a3e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.227+0000 7ff78aa00700 1 --2- 172.21.15.78:0/3538256312 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x555563a3bc20 0x555563a3e040 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8d997ee57a507ba1 server_cookie=70fdaab077ad64aa in_seq=0 out_seq=0 2024-03-14T10:00:43.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.227+0000 7ff774c00700 10 client.26944 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:00:43.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.227+0000 7ff774c00700 1 -- 172.21.15.78:0/3538256312 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55556396c700 con 0x555563a3bc20 2024-03-14T10:00:43.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.227+0000 7ff774c00700 10 client.26944 handle_command_reply: tid=0 2024-03-14T10:00:43.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.227+0000 7ff7a5a6bb80 1 client.26944 shutdown 2024-03-14T10:00:43.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.228+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/3538256312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a21680 msgr2=0x555563a2b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.228+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/3538256312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a21680 0x555563a2b950 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff77c003900 tx=0x7ff77c0047c0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.228+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/3538256312 shutdown_connections 2024-03-14T10:00:43.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.228+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/3538256312 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x555563a3bc20 0x555563a3e040 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.228+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/3538256312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a2bec0 0x555563a293d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.228+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/3538256312 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a21680 0x555563a2b950 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.228+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/3538256312 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 0x555563a28c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.228+0000 7ff774c00700 0 client.26944 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:00:43.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.228+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/3538256312 >> 172.21.15.78:0/3538256312 conn(0x555563997220 msgr2=0x555563a1c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:43.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.228+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/3538256312 shutdown_connections 2024-03-14T10:00:43.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.228+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/3538256312 wait complete. 2024-03-14T10:00:43.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.228+0000 7ff7a5a6bb80 20 client.26944 trim_cache size 0 max 16384 2024-03-14T10:00:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.251+0000 7ff7a5a6bb80 1 Processor -- start 2024-03-14T10:00:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.251+0000 7ff7a5a6bb80 1 -- start start 2024-03-14T10:00:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.251+0000 7ff7a5a6bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a21680 0x555563a285b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.251+0000 7ff7a5a6bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 0x555563a2b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.251+0000 7ff7a5a6bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a2bec0 0x555563a25f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.251+0000 7ff7a5a6bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555563a22af0 con 0x555563a2bec0 2024-03-14T10:00:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.251+0000 7ff7a5a6bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x555563a16d00 con 0x555563a21fa0 2024-03-14T10:00:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.251+0000 7ff7a5a6bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x555563a1e060 con 0x555563a21680 2024-03-14T10:00:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.252+0000 7ff78aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 0x555563a2b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.252+0000 7ff78be00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a2bec0 0x555563a25f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.252+0000 7ff78aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 0x555563a2b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:33204/0 (socket says 172.21.15.78:33204) 2024-03-14T10:00:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.252+0000 7ff78aa00700 1 -- 172.21.15.78:0/747609414 learned_addr learned my addr 172.21.15.78:0/747609414 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.252+0000 7ff78b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a21680 0x555563a285b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.252+0000 7ff78aa00700 1 -- 172.21.15.78:0/747609414 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a21680 msgr2=0x555563a285b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.252+0000 7ff78aa00700 1 --2- 172.21.15.78:0/747609414 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a21680 0x555563a285b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.252+0000 7ff78aa00700 1 -- 172.21.15.78:0/747609414 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a2bec0 msgr2=0x555563a25f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.252+0000 7ff78aa00700 1 --2- 172.21.15.78:0/747609414 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a2bec0 0x555563a25f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.252+0000 7ff78aa00700 1 -- 172.21.15.78:0/747609414 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555563939ea0 con 0x555563a21fa0 2024-03-14T10:00:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.252+0000 7ff78b400700 1 --2- 172.21.15.78:0/747609414 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a21680 0x555563a285b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.252+0000 7ff78aa00700 1 --2- 172.21.15.78:0/747609414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 0x555563a2b480 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff7780c9bf0 tx=0x7ff77800ef80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa74b8fe806a711a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.252+0000 7ff781200700 1 -- 172.21.15.78:0/747609414 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7780a1c90 con 0x555563a21fa0 2024-03-14T10:00:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.253+0000 7ff781200700 1 -- 172.21.15.78:0/747609414 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff7780b2130 con 0x555563a21fa0 2024-03-14T10:00:43.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.253+0000 7ff781200700 1 -- 172.21.15.78:0/747609414 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7780cc3f0 con 0x555563a21fa0 2024-03-14T10:00:43.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.253+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/747609414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 msgr2=0x555563a2b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.253+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/747609414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 0x555563a2b480 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff7780c9bf0 tx=0x7ff77800ef80 comp rx=0 tx=0).stop 2024-03-14T10:00:43.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.254+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/747609414 shutdown_connections 2024-03-14T10:00:43.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.254+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/747609414 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a2bec0 0x555563a25f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.254+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/747609414 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a21680 0x555563a285b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.254+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/747609414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21fa0 0x555563a2b480 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.254+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/747609414 >> 172.21.15.78:0/747609414 conn(0x555563997220 msgr2=0x555563a379f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:43.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.254+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/747609414 shutdown_connections 2024-03-14T10:00:43.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.254+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/747609414 wait complete. 2024-03-14T10:00:43.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.254+0000 7ff7a5a6bb80 1 -- start start 2024-03-14T10:00:43.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.254+0000 7ff7a5a6bb80 1 Processor -- start 2024-03-14T10:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.255+0000 7ff7a5a6bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21680 0x555563a32640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.255+0000 7ff7a5a6bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a21fa0 0x555563a32b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.255+0000 7ff7a5a6bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a2bec0 0x555563a30d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.255+0000 7ff7a5a6bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555563a31480 con 0x555563a21fa0 2024-03-14T10:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.255+0000 7ff7a5a6bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x555563a315f0 con 0x555563a21680 2024-03-14T10:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.255+0000 7ff7a5a6bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x555563a317b0 con 0x555563a2bec0 2024-03-14T10:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.255+0000 7ff7a5a6bb80 10 client.0 _ll_register_callbacks cb 0x7ff77c0043e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.255+0000 7ff78b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21680 0x555563a32640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.256+0000 7ff78aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a21fa0 0x555563a32b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.256+0000 7ff78b400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21680 0x555563a32640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:33220/0 (socket says 172.21.15.78:33220) 2024-03-14T10:00:43.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.256+0000 7ff78b400700 1 -- 172.21.15.78:0/3356593124 learned_addr learned my addr 172.21.15.78:0/3356593124 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.256+0000 7ff78be00700 1 --2- 172.21.15.78:0/3356593124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a2bec0 0x555563a30d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.256+0000 7ff78be00700 1 -- 172.21.15.78:0/3356593124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21680 msgr2=0x555563a32640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.256+0000 7ff78be00700 1 --2- 172.21.15.78:0/3356593124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21680 0x555563a32640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.256+0000 7ff78be00700 1 -- 172.21.15.78:0/3356593124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a21fa0 msgr2=0x555563a32b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.256+0000 7ff78be00700 1 --2- 172.21.15.78:0/3356593124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a21fa0 0x555563a32b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.256+0000 7ff78be00700 1 -- 172.21.15.78:0/3356593124 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5555637a1b60 con 0x555563a2bec0 2024-03-14T10:00:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.256+0000 7ff78aa00700 1 --2- 172.21.15.78:0/3356593124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a21fa0 0x555563a32b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T10:00:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.257+0000 7ff78be00700 1 --2- 172.21.15.78:0/3356593124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a2bec0 0x555563a30d70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff78400b6c0 tx=0x7ff784007cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=245e9ca76d8365e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.257+0000 7ff774c00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:00:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.257+0000 7ff774c00700 1 -- 172.21.15.78:0/3356593124 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78400e040 con 0x555563a2bec0 2024-03-14T10:00:43.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.257+0000 7ff774c00700 1 -- 172.21.15.78:0/3356593124 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7ff78400bb90 con 0x555563a2bec0 2024-03-14T10:00:43.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.257+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/3356593124 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff77809d5c0 con 0x555563a2bec0 2024-03-14T10:00:43.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.258+0000 7ff774c00700 1 -- 172.21.15.78:0/3356593124 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff784009ac0 con 0x555563a2bec0 2024-03-14T10:00:43.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.258+0000 7ff7a5a6bb80 10 client.26950 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:43.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.258+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/3356593124 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5555637f53b0 con 0x555563a2bec0 2024-03-14T10:00:43.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.259+0000 7ff774c00700 1 -- 172.21.15.78:0/3356593124 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff77809d5c0 con 0x555563a2bec0 2024-03-14T10:00:43.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.259+0000 7ff774c00700 1 -- 172.21.15.78:0/3356593124 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff78400b050 con 0x555563a2bec0 2024-03-14T10:00:43.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.259+0000 7ff7a5a6bb80 10 client.26950 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:43.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.259+0000 7ff7a5a6bb80 10 client.26950 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:00:43.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.259+0000 7ff7a5a6bb80 20 client.26950 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:43.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.259+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/3356593124 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x555563a972e0 0x555563a99700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.259+0000 7ff7a5a6bb80 4 client.26950 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:00:43.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.259+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/3356593124 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5555637c2cf0 con 0x555563a972e0 2024-03-14T10:00:43.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.260+0000 7ff78b400700 1 --2- 172.21.15.78:0/3356593124 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x555563a972e0 0x555563a99700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.260+0000 7ff774c00700 10 client.26950 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:00:43.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.260+0000 7ff78b400700 1 --2- 172.21.15.78:0/3356593124 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x555563a972e0 0x555563a99700 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d93b8a67c30ca961 server_cookie=310b58fefbd2ec51 in_seq=0 out_seq=0 2024-03-14T10:00:43.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.261+0000 7ff774c00700 1 -- 172.21.15.78:0/3356593124 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5555637c2cf0 con 0x555563a972e0 2024-03-14T10:00:43.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.261+0000 7ff774c00700 10 client.26950 handle_command_reply: tid=0 2024-03-14T10:00:43.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.261+0000 7ff7a5a6bb80 1 client.26950 shutdown 2024-03-14T10:00:43.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.261+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/3356593124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a2bec0 msgr2=0x555563a30d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.261+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/3356593124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a2bec0 0x555563a30d70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff78400b6c0 tx=0x7ff784007cc0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.261+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/3356593124 shutdown_connections 2024-03-14T10:00:43.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.261+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/3356593124 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x555563a972e0 0x555563a99700 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.261+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/3356593124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555563a21680 0x555563a32640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.261+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/3356593124 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x555563a2bec0 0x555563a30d70 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.261+0000 7ff774c00700 0 client.26950 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:00:43.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.261+0000 7ff7a5a6bb80 1 --2- 172.21.15.78:0/3356593124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555563a21fa0 0x555563a32b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.261+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/3356593124 >> 172.21.15.78:0/3356593124 conn(0x555563997220 msgr2=0x555563a20560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:43.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.262+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/3356593124 shutdown_connections 2024-03-14T10:00:43.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.262+0000 7ff7a5a6bb80 1 -- 172.21.15.78:0/3356593124 wait complete. 2024-03-14T10:00:43.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.262+0000 7ff7a5a6bb80 20 client.26950 trim_cache size 0 max 16384 2024-03-14T10:00:43.265 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:00:43.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.264+0000 7ff792000700 1 -- 172.21.15.78:0/470648824 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff7700a0c00 msgr2=0x7ff7700a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.264+0000 7ff792000700 1 --2- 172.21.15.78:0/470648824 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff7700a0c00 0x7ff7700a30c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009c60 tx=0x7ff77c009260 comp rx=0 tx=0).stop 2024-03-14T10:00:43.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.264+0000 7ff792000700 1 -- 172.21.15.78:0/470648824 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff78c106050 msgr2=0x7ff78c19fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.264+0000 7ff792000700 1 --2- 172.21.15.78:0/470648824 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff78c106050 0x7ff78c19fb70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff77800e9f0 tx=0x7ff77800ed00 comp rx=0 tx=0).stop 2024-03-14T10:00:43.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.264+0000 7ff792000700 1 -- 172.21.15.78:0/470648824 shutdown_connections 2024-03-14T10:00:43.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.264+0000 7ff792000700 1 --2- 172.21.15.78:0/470648824 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff7700a0c00 0x7ff7700a30c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.264+0000 7ff792000700 1 --2- 172.21.15.78:0/470648824 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff78c199760 0x7ff78c199c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.264+0000 7ff792000700 1 --2- 172.21.15.78:0/470648824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff78c100f30 0x7ff78c19f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.264+0000 7ff792000700 1 --2- 172.21.15.78:0/470648824 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff78c106050 0x7ff78c19fb70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.264+0000 7ff792000700 1 -- 172.21.15.78:0/470648824 >> 172.21.15.78:0/470648824 conn(0x7ff78c0fc7b0 msgr2=0x7ff78c0fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:43.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.264+0000 7ff792000700 1 -- 172.21.15.78:0/470648824 shutdown_connections 2024-03-14T10:00:43.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.264+0000 7ff792000700 1 -- 172.21.15.78:0/470648824 wait complete. 2024-03-14T10:00:43.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:43 smithi078 ceph-mon[38143]: pgmap v199: 129 pgs: 129 active+clean; 242 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 1.5 MiB/s wr, 529 op/s 2024-03-14T10:00:43.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:43 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/816454306' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:00:43.287 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T10:00:43.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:43 smithi152 ceph-mon[48279]: pgmap v199: 129 pgs: 129 active+clean; 242 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 1.5 MiB/s wr, 529 op/s 2024-03-14T10:00:43.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:43 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/816454306' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:00:43.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:43 smithi132 ceph-mon[47959]: pgmap v199: 129 pgs: 129 active+clean; 242 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 1.5 MiB/s wr, 529 op/s 2024-03-14T10:00:43.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:43 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/816454306' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:00:43.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.529+0000 7f45aa800700 1 -- 172.21.15.78:0/903109913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45a4073a80 msgr2=0x7f45a410d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.529+0000 7f45aa800700 1 --2- 172.21.15.78:0/903109913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45a4073a80 0x7f45a410d040 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4590009990 tx=0x7f4590009ca0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.529+0000 7f45aa800700 1 -- 172.21.15.78:0/903109913 shutdown_connections 2024-03-14T10:00:43.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.529+0000 7f45aa800700 1 --2- 172.21.15.78:0/903109913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a410d580 0x7f45a410f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.529+0000 7f45aa800700 1 --2- 172.21.15.78:0/903109913 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45a4073160 0x7f45a4073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.529+0000 7f45aa800700 1 --2- 172.21.15.78:0/903109913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45a4073a80 0x7f45a410d040 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.529+0000 7f45aa800700 1 -- 172.21.15.78:0/903109913 >> 172.21.15.78:0/903109913 conn(0x7f45a40fc7b0 msgr2=0x7f45a40febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:43.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.529+0000 7f45aa800700 1 -- 172.21.15.78:0/903109913 shutdown_connections 2024-03-14T10:00:43.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.529+0000 7f45aa800700 1 -- 172.21.15.78:0/903109913 wait complete. 2024-03-14T10:00:43.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.530+0000 7f45aa800700 1 Processor -- start 2024-03-14T10:00:43.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.530+0000 7f45aa800700 1 -- start start 2024-03-14T10:00:43.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.530+0000 7f45aa800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45a4073160 0x7f45a419f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.530+0000 7f45aa800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45a4073a80 0x7f45a419fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.530+0000 7f45aa800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a410d580 0x7f45a4199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.530+0000 7f45aa800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a4199cd0 con 0x7f45a410d580 2024-03-14T10:00:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.530+0000 7f45aa800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a4199e40 con 0x7f45a4073a80 2024-03-14T10:00:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.530+0000 7f45aa800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a419a000 con 0x7f45a4073160 2024-03-14T10:00:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.531+0000 7f45a2a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45a4073a80 0x7f45a419fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.531+0000 7f45a3400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45a4073160 0x7f45a419f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.531+0000 7f45a2a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45a4073a80 0x7f45a419fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:33240/0 (socket says 172.21.15.78:33240) 2024-03-14T10:00:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.531+0000 7f45a2a00700 1 -- 172.21.15.78:0/2601502315 learned_addr learned my addr 172.21.15.78:0/2601502315 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.531+0000 7f45a3e00700 1 --2- 172.21.15.78:0/2601502315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a410d580 0x7f45a4199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.531+0000 7f45a2a00700 1 -- 172.21.15.78:0/2601502315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45a4073160 msgr2=0x7f45a419f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.531+0000 7f45a2a00700 1 --2- 172.21.15.78:0/2601502315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45a4073160 0x7f45a419f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.531+0000 7f45a2a00700 1 -- 172.21.15.78:0/2601502315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a410d580 msgr2=0x7f45a4199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.531+0000 7f45a2a00700 1 --2- 172.21.15.78:0/2601502315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a410d580 0x7f45a4199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.531+0000 7f45a2a00700 1 -- 172.21.15.78:0/2601502315 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45900095f0 con 0x7f45a4073a80 2024-03-14T10:00:43.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.531+0000 7f45a2a00700 1 --2- 172.21.15.78:0/2601502315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45a4073a80 0x7f45a419fa80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4590005910 tx=0x7f4590004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:43.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.532+0000 7f459be00700 1 -- 172.21.15.78:0/2601502315 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459001d070 con 0x7f45a4073a80 2024-03-14T10:00:43.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.532+0000 7f459be00700 1 -- 172.21.15.78:0/2601502315 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4590022470 con 0x7f45a4073a80 2024-03-14T10:00:43.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.532+0000 7f45aa800700 1 -- 172.21.15.78:0/2601502315 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a419a280 con 0x7f45a4073a80 2024-03-14T10:00:43.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.532+0000 7f459be00700 1 -- 172.21.15.78:0/2601502315 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4590017650 con 0x7f45a4073a80 2024-03-14T10:00:43.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.532+0000 7f45aa800700 1 -- 172.21.15.78:0/2601502315 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a41a6530 con 0x7f45a4073a80 2024-03-14T10:00:43.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.535+0000 7f459be00700 1 -- 172.21.15.78:0/2601502315 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f45900225e0 con 0x7f45a4073a80 2024-03-14T10:00:43.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.536+0000 7f45be2d5b80 1 Processor -- start 2024-03-14T10:00:43.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.537+0000 7f45be2d5b80 1 -- start start 2024-03-14T10:00:43.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.537+0000 7f45be2d5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69ab680 0x558ab69aba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.537+0000 7f45be2d5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69abfa0 0x558ab69b5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.537+0000 7f45be2d5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69b5ec0 0x558ab69b82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.537+0000 7f45be2d5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab6904990 con 0x558ab69abfa0 2024-03-14T10:00:43.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.537+0000 7f45be2d5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab6902350 con 0x558ab69ab680 2024-03-14T10:00:43.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.537+0000 7f45be2d5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab68fdc40 con 0x558ab69b5ec0 2024-03-14T10:00:43.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.537+0000 7f459be00700 1 --2- 172.21.15.78:0/2601502315 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f45880a0c80 0x7f45880a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.537+0000 7f45a3400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69abfa0 0x558ab69b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.537+0000 7f45a3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69ab680 0x558ab69aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.537+0000 7f45a3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69ab680 0x558ab69aba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:33244/0 (socket says 172.21.15.78:33244) 2024-03-14T10:00:43.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.537+0000 7f45a3400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69abfa0 0x558ab69b5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55086/0 (socket says 172.21.15.78:55086) 2024-03-14T10:00:43.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.537+0000 7f45a3e00700 1 -- 172.21.15.78:0/270550058 learned_addr learned my addr 172.21.15.78:0/270550058 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:43.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.538+0000 7f459be00700 1 -- 172.21.15.78:0/2601502315 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f45900cd990 con 0x7f45a4073a80 2024-03-14T10:00:43.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.538+0000 7f45a2a00700 1 --2- 172.21.15.78:0/270550058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69b5ec0 0x558ab69b82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.538+0000 7f45a3e00700 1 --2- 172.21.15.78:0/2601502315 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f45880a0c80 0x7f45880a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.538+0000 7f45a3e00700 1 -- 172.21.15.78:0/270550058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69b5ec0 msgr2=0x558ab69b82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:43.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.538+0000 7f45a3e00700 1 --2- 172.21.15.78:0/270550058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69b5ec0 0x558ab69b82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.538+0000 7f45a3e00700 1 -- 172.21.15.78:0/270550058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69abfa0 msgr2=0x558ab69b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:43.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.538+0000 7f45a3e00700 1 --2- 172.21.15.78:0/270550058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69abfa0 0x558ab69b5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.538+0000 7f45a3e00700 1 -- 172.21.15.78:0/270550058 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ab677f3b0 con 0x558ab69ab680 2024-03-14T10:00:43.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.538+0000 7f45a3e00700 1 --2- 172.21.15.78:0/270550058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69ab680 0x558ab69aba60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f459c00fc70 tx=0x7f459c00ff80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=642112755e02e3c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:43.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.538+0000 7f45a3e00700 1 --2- 172.21.15.78:0/2601502315 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f45880a0c80 0x7f45880a3140 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f459c009000 tx=0x7f459c01e000 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:43.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.539+0000 7f4598c00700 1 -- 172.21.15.78:0/270550058 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c00e040 con 0x558ab69ab680 2024-03-14T10:00:43.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.539+0000 7f4598c00700 1 -- 172.21.15.78:0/270550058 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f459c00f040 con 0x558ab69ab680 2024-03-14T10:00:43.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.539+0000 7f4598c00700 1 -- 172.21.15.78:0/270550058 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c01a790 con 0x558ab69ab680 2024-03-14T10:00:43.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.539+0000 7f45be2d5b80 1 -- 172.21.15.78:0/270550058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69ab680 msgr2=0x558ab69aba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.539+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/270550058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69ab680 0x558ab69aba60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f459c00fc70 tx=0x7f459c00ff80 comp rx=0 tx=0).stop 2024-03-14T10:00:43.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.540+0000 7f45be2d5b80 1 -- 172.21.15.78:0/270550058 shutdown_connections 2024-03-14T10:00:43.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.540+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/270550058 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69ab680 0x558ab69aba60 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.540+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/270550058 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69b5ec0 0x558ab69b82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.540+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/270550058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69abfa0 0x558ab69b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.540+0000 7f45be2d5b80 1 -- 172.21.15.78:0/270550058 >> 172.21.15.78:0/270550058 conn(0x558ab6921220 msgr2=0x558ab69a6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:43.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.540+0000 7f45be2d5b80 1 -- 172.21.15.78:0/270550058 shutdown_connections 2024-03-14T10:00:43.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.540+0000 7f45be2d5b80 1 -- 172.21.15.78:0/270550058 wait complete. 2024-03-14T10:00:43.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.540+0000 7f45be2d5b80 1 -- start start 2024-03-14T10:00:43.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.540+0000 7f45be2d5b80 1 Processor -- start 2024-03-14T10:00:43.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.541+0000 7f45be2d5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69ab680 0x558ab69b5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.541+0000 7f45be2d5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69abfa0 0x558ab69b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.541+0000 7f45be2d5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69b5ec0 0x558ab69b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.541+0000 7f45be2d5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab6899d80 con 0x558ab69b5ec0 2024-03-14T10:00:43.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.541+0000 7f45be2d5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab6899a20 con 0x558ab69ab680 2024-03-14T10:00:43.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.541+0000 7f45be2d5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab67816a0 con 0x558ab69abfa0 2024-03-14T10:00:43.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.541+0000 7f45be2d5b80 10 client.0 _ll_register_callbacks cb 0x558ab67053a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:43.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.541+0000 7f45a2a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69abfa0 0x558ab69b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.541+0000 7f45a3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69b5ec0 0x558ab69b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.541+0000 7f45a2a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69abfa0 0x558ab69b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47936/0 (socket says 172.21.15.78:47936) 2024-03-14T10:00:43.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.541+0000 7f45a3400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69ab680 0x558ab69b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.541+0000 7f45a2a00700 1 -- 172.21.15.78:0/1980206997 learned_addr learned my addr 172.21.15.78:0/1980206997 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:43.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.542+0000 7f45a2a00700 1 -- 172.21.15.78:0/1980206997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69ab680 msgr2=0x558ab69b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:43.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.542+0000 7f45a2a00700 1 --2- 172.21.15.78:0/1980206997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69ab680 0x558ab69b5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.542+0000 7f45a2a00700 1 -- 172.21.15.78:0/1980206997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69b5ec0 msgr2=0x558ab69b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:43.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.542+0000 7f45a2a00700 1 --2- 172.21.15.78:0/1980206997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69b5ec0 0x558ab69b3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.542+0000 7f45a2a00700 1 -- 172.21.15.78:0/1980206997 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459c00f820 con 0x558ab69abfa0 2024-03-14T10:00:43.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.542+0000 7f45a3400700 1 --2- 172.21.15.78:0/1980206997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69ab680 0x558ab69b5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:43.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.542+0000 7f45a2a00700 1 --2- 172.21.15.78:0/1980206997 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69abfa0 0x558ab69b2d40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4590006010 tx=0x7f4590022f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6d854d88df71f6b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:43.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.542+0000 7f458d600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:00:43.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.542+0000 7f458d600700 1 -- 172.21.15.78:0/1980206997 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45900b3420 con 0x558ab69abfa0 2024-03-14T10:00:43.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.542+0000 7f458d600700 1 -- 172.21.15.78:0/1980206997 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f459009b1e0 con 0x558ab69abfa0 2024-03-14T10:00:43.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.542+0000 7f45be2d5b80 1 -- 172.21.15.78:0/1980206997 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558ab699e720 con 0x558ab69abfa0 2024-03-14T10:00:43.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.543+0000 7f458d600700 1 -- 172.21.15.78:0/1980206997 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f459009f830 con 0x558ab69abfa0 2024-03-14T10:00:43.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.543+0000 7f45be2d5b80 10 client.26962 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:43.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.543+0000 7f45be2d5b80 1 -- 172.21.15.78:0/1980206997 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558ab672bb60 con 0x558ab69abfa0 2024-03-14T10:00:43.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.543+0000 7f458d600700 1 -- 172.21.15.78:0/1980206997 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558ab699e720 con 0x558ab69abfa0 2024-03-14T10:00:43.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.543+0000 7f458d600700 1 -- 172.21.15.78:0/1980206997 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f459009a060 con 0x558ab69abfa0 2024-03-14T10:00:43.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.543+0000 7f45be2d5b80 10 client.26962 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:43.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.543+0000 7f45be2d5b80 10 client.26962 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:00:43.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.543+0000 7f45be2d5b80 20 client.26962 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:43.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.543+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/1980206997 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558ab69c55e0 0x558ab69c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.543+0000 7f45be2d5b80 4 client.26962 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:00:43.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.543+0000 7f45be2d5b80 1 -- 172.21.15.78:0/1980206997 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558ab68f6700 con 0x558ab69c55e0 2024-03-14T10:00:43.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.543+0000 7f45a3400700 1 --2- 172.21.15.78:0/1980206997 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558ab69c55e0 0x558ab69c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.544+0000 7f45a3400700 1 --2- 172.21.15.78:0/1980206997 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558ab69c55e0 0x558ab69c7a00 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e9d325a8a35e2d5 server_cookie=14dfdd8697123371 in_seq=0 out_seq=0 2024-03-14T10:00:43.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.544+0000 7f458d600700 10 client.26962 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:43.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.544+0000 7f458d600700 1 -- 172.21.15.78:0/1980206997 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x558ab68f6700 con 0x558ab69c55e0 2024-03-14T10:00:43.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.544+0000 7f458d600700 10 client.26962 handle_command_reply: tid=0 2024-03-14T10:00:43.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.544+0000 7f45be2d5b80 1 client.26962 shutdown 2024-03-14T10:00:43.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.545+0000 7f45be2d5b80 1 -- 172.21.15.78:0/1980206997 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69abfa0 msgr2=0x558ab69b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.545+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/1980206997 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69abfa0 0x558ab69b2d40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4590006010 tx=0x7f4590022f90 comp rx=0 tx=0).stop 2024-03-14T10:00:43.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.545+0000 7f45be2d5b80 1 -- 172.21.15.78:0/1980206997 shutdown_connections 2024-03-14T10:00:43.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.545+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/1980206997 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558ab69c55e0 0x558ab69c7a00 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.545+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/1980206997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69ab680 0x558ab69b5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.545+0000 7f458d600700 0 client.26962 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:43.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.545+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/1980206997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69b5ec0 0x558ab69b3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.545+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/1980206997 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69abfa0 0x558ab69b2d40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.545+0000 7f45be2d5b80 1 -- 172.21.15.78:0/1980206997 >> 172.21.15.78:0/1980206997 conn(0x558ab6921220 msgr2=0x558ab69a6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:43.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.545+0000 7f45be2d5b80 1 -- 172.21.15.78:0/1980206997 shutdown_connections 2024-03-14T10:00:43.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.545+0000 7f45be2d5b80 1 -- 172.21.15.78:0/1980206997 wait complete. 2024-03-14T10:00:43.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.545+0000 7f45be2d5b80 20 client.26962 trim_cache size 0 max 16384 2024-03-14T10:00:43.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.561+0000 7f45be2d5b80 1 Processor -- start 2024-03-14T10:00:43.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.561+0000 7f45be2d5b80 1 -- start start 2024-03-14T10:00:43.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.561+0000 7f45be2d5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69ab680 0x558ab69b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.561+0000 7f45be2d5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69abfa0 0x558ab69a65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.561+0000 7f45be2d5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69b5ec0 0x558ab69a6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.561+0000 7f45be2d5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab69b3a30 con 0x558ab69abfa0 2024-03-14T10:00:43.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.561+0000 7f45be2d5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab69a3230 con 0x558ab69b5ec0 2024-03-14T10:00:43.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.561+0000 7f45be2d5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab69a0d00 con 0x558ab69ab680 2024-03-14T10:00:43.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.561+0000 7f45a3400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69ab680 0x558ab69b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.561+0000 7f45a2a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69abfa0 0x558ab69a65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.561+0000 7f45a3400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69ab680 0x558ab69b2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47948/0 (socket says 172.21.15.78:47948) 2024-03-14T10:00:43.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.561+0000 7f45a3400700 1 -- 172.21.15.78:0/1498161811 learned_addr learned my addr 172.21.15.78:0/1498161811 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:43.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.561+0000 7f45a3e00700 1 --2- 172.21.15.78:0/1498161811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69b5ec0 0x558ab69a6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.562+0000 7f45a3400700 1 -- 172.21.15.78:0/1498161811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69b5ec0 msgr2=0x558ab69a6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:43.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.562+0000 7f45a3400700 1 --2- 172.21.15.78:0/1498161811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69b5ec0 0x558ab69a6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.562+0000 7f45a3400700 1 -- 172.21.15.78:0/1498161811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69abfa0 msgr2=0x558ab69a65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:43.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.562+0000 7f45a3400700 1 --2- 172.21.15.78:0/1498161811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69abfa0 0x558ab69a65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.562+0000 7f45a3400700 1 -- 172.21.15.78:0/1498161811 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ab68c3ea0 con 0x558ab69ab680 2024-03-14T10:00:43.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.562+0000 7f45a3400700 1 --2- 172.21.15.78:0/1498161811 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69ab680 0x558ab69b2600 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x558ab69c82e0 tx=0x7f4594004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=938f6d438c6182ea server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:43.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.562+0000 7f4599600700 1 -- 172.21.15.78:0/1498161811 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594016070 con 0x558ab69ab680 2024-03-14T10:00:43.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.562+0000 7f4599600700 1 -- 172.21.15.78:0/1498161811 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4594017040 con 0x558ab69ab680 2024-03-14T10:00:43.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.562+0000 7f4599600700 1 -- 172.21.15.78:0/1498161811 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594010df0 con 0x558ab69ab680 2024-03-14T10:00:43.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.563+0000 7f45be2d5b80 1 -- 172.21.15.78:0/1498161811 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69ab680 msgr2=0x558ab69b2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.563+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/1498161811 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69ab680 0x558ab69b2600 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x558ab69c82e0 tx=0x7f4594004750 comp rx=0 tx=0).stop 2024-03-14T10:00:43.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.563+0000 7f45be2d5b80 1 -- 172.21.15.78:0/1498161811 shutdown_connections 2024-03-14T10:00:43.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.563+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/1498161811 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69b5ec0 0x558ab69a6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.563+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/1498161811 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69ab680 0x558ab69b2600 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.563+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/1498161811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69abfa0 0x558ab69a65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.563+0000 7f45be2d5b80 1 -- 172.21.15.78:0/1498161811 >> 172.21.15.78:0/1498161811 conn(0x558ab6921220 msgr2=0x558ab69ae020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:43.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.563+0000 7f45be2d5b80 1 -- 172.21.15.78:0/1498161811 shutdown_connections 2024-03-14T10:00:43.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.563+0000 7f45be2d5b80 1 -- 172.21.15.78:0/1498161811 wait complete. 2024-03-14T10:00:43.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.564+0000 7f45be2d5b80 1 -- start start 2024-03-14T10:00:43.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.564+0000 7f45be2d5b80 1 Processor -- start 2024-03-14T10:00:43.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.564+0000 7f45be2d5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69ab680 0x558ab69bc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.564+0000 7f45be2d5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69abfa0 0x558ab69bc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.564+0000 7f45be2d5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69b5ec0 0x558ab69ba960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.564+0000 7f45be2d5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab69baef0 con 0x558ab69ab680 2024-03-14T10:00:43.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.564+0000 7f45be2d5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab69bb060 con 0x558ab69b5ec0 2024-03-14T10:00:43.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.564+0000 7f45be2d5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab69bb220 con 0x558ab69abfa0 2024-03-14T10:00:43.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.565+0000 7f45be2d5b80 10 client.0 _ll_register_callbacks cb 0x7f4590096e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:43.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.565+0000 7f45a3400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69ab680 0x558ab69bc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.565+0000 7f45a3400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69ab680 0x558ab69bc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:55116/0 (socket says 172.21.15.78:55116) 2024-03-14T10:00:43.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.565+0000 7f45a3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69b5ec0 0x558ab69ba960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.565+0000 7f45a3400700 1 -- 172.21.15.78:0/692509725 learned_addr learned my addr 172.21.15.78:0/692509725 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:43.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.565+0000 7f45a2a00700 1 --2- 172.21.15.78:0/692509725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69abfa0 0x558ab69bc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.565+0000 7f45a3400700 1 -- 172.21.15.78:0/692509725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69abfa0 msgr2=0x558ab69bc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:43.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.565+0000 7f45a3400700 1 --2- 172.21.15.78:0/692509725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69abfa0 0x558ab69bc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.565+0000 7f45a3400700 1 -- 172.21.15.78:0/692509725 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69b5ec0 msgr2=0x558ab69ba960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:43.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.565+0000 7f45a3400700 1 --2- 172.21.15.78:0/692509725 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69b5ec0 0x558ab69ba960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.565+0000 7f45a3400700 1 -- 172.21.15.78:0/692509725 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4594004040 con 0x558ab69ab680 2024-03-14T10:00:43.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.565+0000 7f45a3400700 1 --2- 172.21.15.78:0/692509725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69ab680 0x558ab69bc2b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4594005810 tx=0x7f4594009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f151c4ec996fd5f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:43.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.565+0000 7f458d600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:00:43.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.566+0000 7f458d600700 1 -- 172.21.15.78:0/692509725 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594016040 con 0x558ab69ab680 2024-03-14T10:00:43.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.566+0000 7f458d600700 1 -- 172.21.15.78:0/692509725 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7f459401d020 con 0x558ab69ab680 2024-03-14T10:00:43.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.566+0000 7f45be2d5b80 1 -- 172.21.15.78:0/692509725 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f459c00f040 con 0x558ab69ab680 2024-03-14T10:00:43.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.566+0000 7f458d600700 1 -- 172.21.15.78:0/692509725 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f459401d1e0 con 0x558ab69ab680 2024-03-14T10:00:43.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.566+0000 7f45be2d5b80 10 client.16908 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:43.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.566+0000 7f45be2d5b80 1 -- 172.21.15.78:0/692509725 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558ab6783cd0 con 0x558ab69ab680 2024-03-14T10:00:43.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.566+0000 7f458d600700 1 -- 172.21.15.78:0/692509725 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f459c00f040 con 0x558ab69ab680 2024-03-14T10:00:43.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.566+0000 7f458d600700 1 -- 172.21.15.78:0/692509725 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f45940047c0 con 0x558ab69ab680 2024-03-14T10:00:43.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.566+0000 7f45be2d5b80 10 client.16908 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:43.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.566+0000 7f45be2d5b80 10 client.16908 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:00:43.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.566+0000 7f45be2d5b80 20 client.16908 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:43.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.566+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/692509725 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558ab6a18270 0x558ab6a1a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:43.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.566+0000 7f45be2d5b80 4 client.16908 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:00:43.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.566+0000 7f45be2d5b80 1 -- 172.21.15.78:0/692509725 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558ab674ccf0 con 0x558ab6a18270 2024-03-14T10:00:43.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.567+0000 7f45a2a00700 1 --2- 172.21.15.78:0/692509725 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558ab6a18270 0x558ab6a1a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.567+0000 7f458d600700 10 client.16908 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.567+0000 7f45a2a00700 1 --2- 172.21.15.78:0/692509725 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558ab6a18270 0x558ab6a1a690 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a9497bed89c4fc8 server_cookie=25e055d23ec3eeda in_seq=0 out_seq=0 2024-03-14T10:00:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.567+0000 7f458d600700 1 -- 172.21.15.78:0/692509725 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558ab674ccf0 con 0x558ab6a18270 2024-03-14T10:00:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.567+0000 7f458d600700 10 client.16908 handle_command_reply: tid=0 2024-03-14T10:00:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.567+0000 7f45be2d5b80 1 client.16908 shutdown 2024-03-14T10:00:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.567+0000 7f45be2d5b80 1 -- 172.21.15.78:0/692509725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69ab680 msgr2=0x558ab69bc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.567+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/692509725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69ab680 0x558ab69bc2b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4594005810 tx=0x7f4594009760 comp rx=0 tx=0).stop 2024-03-14T10:00:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.567+0000 7f45be2d5b80 1 -- 172.21.15.78:0/692509725 shutdown_connections 2024-03-14T10:00:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.567+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/692509725 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x558ab6a18270 0x558ab6a1a690 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.567+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/692509725 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558ab69b5ec0 0x558ab69ba960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.567+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/692509725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558ab69ab680 0x558ab69bc2b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.567+0000 7f458d600700 0 client.16908 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.567+0000 7f45be2d5b80 1 --2- 172.21.15.78:0/692509725 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558ab69abfa0 0x558ab69bc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.567+0000 7f45be2d5b80 1 -- 172.21.15.78:0/692509725 >> 172.21.15.78:0/692509725 conn(0x558ab6921220 msgr2=0x558ab69c1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:43.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.568+0000 7f45be2d5b80 1 -- 172.21.15.78:0/692509725 shutdown_connections 2024-03-14T10:00:43.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.568+0000 7f45be2d5b80 1 -- 172.21.15.78:0/692509725 wait complete. 2024-03-14T10:00:43.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.568+0000 7f45be2d5b80 20 client.16908 trim_cache size 0 max 16384 2024-03-14T10:00:43.573 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:00:43.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.569+0000 7f45aa800700 1 -- 172.21.15.78:0/2601502315 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f45880a0c80 msgr2=0x7f45880a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.569+0000 7f45aa800700 1 --2- 172.21.15.78:0/2601502315 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f45880a0c80 0x7f45880a3140 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f459c009000 tx=0x7f459c01e000 comp rx=0 tx=0).stop 2024-03-14T10:00:43.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.569+0000 7f45aa800700 1 -- 172.21.15.78:0/2601502315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45a4073a80 msgr2=0x7f45a419fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:43.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.569+0000 7f45aa800700 1 --2- 172.21.15.78:0/2601502315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45a4073a80 0x7f45a419fa80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4590005910 tx=0x7f4590004730 comp rx=0 tx=0).stop 2024-03-14T10:00:43.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.569+0000 7f45aa800700 1 -- 172.21.15.78:0/2601502315 shutdown_connections 2024-03-14T10:00:43.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.569+0000 7f45aa800700 1 --2- 172.21.15.78:0/2601502315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a410d580 0x7f45a4199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.569+0000 7f45aa800700 1 --2- 172.21.15.78:0/2601502315 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45a4073160 0x7f45a419f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.569+0000 7f45aa800700 1 --2- 172.21.15.78:0/2601502315 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f45880a0c80 0x7f45880a3140 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.569+0000 7f45aa800700 1 --2- 172.21.15.78:0/2601502315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45a4073a80 0x7f45a419fa80 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:43.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.569+0000 7f45aa800700 1 -- 172.21.15.78:0/2601502315 >> 172.21.15.78:0/2601502315 conn(0x7f45a40fc7b0 msgr2=0x7f45a40fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:43.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.569+0000 7f45aa800700 1 -- 172.21.15.78:0/2601502315 shutdown_connections 2024-03-14T10:00:43.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:43.570+0000 7f45aa800700 1 -- 172.21.15.78:0/2601502315 wait complete. 2024-03-14T10:00:44.594 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T10:00:44.595 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T10:00:44.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.834+0000 7ff8d4400700 1 -- 172.21.15.78:0/1463835689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8cc0680b0 msgr2=0x7ff8cc068490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:44.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.834+0000 7ff8d4400700 1 --2- 172.21.15.78:0/1463835689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8cc0680b0 0x7ff8cc068490 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff8c00099c0 tx=0x7ff8c0009cd0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.834+0000 7ff8d4400700 1 -- 172.21.15.78:0/1463835689 shutdown_connections 2024-03-14T10:00:44.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.834+0000 7ff8d4400700 1 --2- 172.21.15.78:0/1463835689 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8cc106060 0x7ff8cc10dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.834+0000 7ff8d4400700 1 --2- 172.21.15.78:0/1463835689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8cc0680b0 0x7ff8cc068490 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.834+0000 7ff8d4400700 1 --2- 172.21.15.78:0/1463835689 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8cc068a60 0x7ff8cc105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.834+0000 7ff8d4400700 1 -- 172.21.15.78:0/1463835689 >> 172.21.15.78:0/1463835689 conn(0x7ff8cc076d80 msgr2=0x7ff8cc077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:44.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.834+0000 7ff8d4400700 1 -- 172.21.15.78:0/1463835689 shutdown_connections 2024-03-14T10:00:44.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.834+0000 7ff8d4400700 1 -- 172.21.15.78:0/1463835689 wait complete. 2024-03-14T10:00:44.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.835+0000 7ff8d4400700 1 Processor -- start 2024-03-14T10:00:44.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.836+0000 7ff8d4400700 1 -- start start 2024-03-14T10:00:44.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.836+0000 7ff8d4400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8cc0680b0 0x7ff8cc0ffc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.836+0000 7ff8d4400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8cc106060 0x7ff8cc100160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.836+0000 7ff8d4400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8cc1006a0 0x7ff8cc1a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.836+0000 7ff8d4400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc1a5e10 con 0x7ff8cc1006a0 2024-03-14T10:00:44.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.836+0000 7ff8d4400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc1a5f80 con 0x7ff8cc0680b0 2024-03-14T10:00:44.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.836+0000 7ff8d4400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc1a6140 con 0x7ff8cc106060 2024-03-14T10:00:44.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.836+0000 7ff8d2400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8cc1006a0 0x7ff8cc1a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.836+0000 7ff8d2400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8cc1006a0 0x7ff8cc1a5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47248/0 (socket says 172.21.15.78:47248) 2024-03-14T10:00:44.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.836+0000 7ff8d2400700 1 -- 172.21.15.78:0/1967573513 learned_addr learned my addr 172.21.15.78:0/1967573513 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:44.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.836+0000 7ff8d1000700 1 --2- 172.21.15.78:0/1967573513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8cc106060 0x7ff8cc100160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.836+0000 7ff8d1a00700 1 --2- 172.21.15.78:0/1967573513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8cc0680b0 0x7ff8cc0ffc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.837+0000 7ff8d2400700 1 -- 172.21.15.78:0/1967573513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8cc106060 msgr2=0x7ff8cc100160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:44.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.837+0000 7ff8d2400700 1 --2- 172.21.15.78:0/1967573513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8cc106060 0x7ff8cc100160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.837+0000 7ff8d2400700 1 -- 172.21.15.78:0/1967573513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8cc0680b0 msgr2=0x7ff8cc0ffc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:44.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.837+0000 7ff8d2400700 1 --2- 172.21.15.78:0/1967573513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8cc0680b0 0x7ff8cc0ffc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.837+0000 7ff8d2400700 1 -- 172.21.15.78:0/1967573513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c00095f0 con 0x7ff8cc1006a0 2024-03-14T10:00:44.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.837+0000 7ff8d1000700 1 --2- 172.21.15.78:0/1967573513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8cc106060 0x7ff8cc100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:44.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.837+0000 7ff8d2400700 1 --2- 172.21.15.78:0/1967573513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8cc1006a0 0x7ff8cc1a5840 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400ef30 tx=0x7ff8c400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:44.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.837+0000 7ff8ca000700 1 -- 172.21.15.78:0/1967573513 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4018070 con 0x7ff8cc1006a0 2024-03-14T10:00:44.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.837+0000 7ff8ca000700 1 -- 172.21.15.78:0/1967573513 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff8c400f040 con 0x7ff8cc1006a0 2024-03-14T10:00:44.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.837+0000 7ff8d4400700 1 -- 172.21.15.78:0/1967573513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8cc1a63c0 con 0x7ff8cc1006a0 2024-03-14T10:00:44.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.837+0000 7ff8ca000700 1 -- 172.21.15.78:0/1967573513 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c400cd00 con 0x7ff8cc1006a0 2024-03-14T10:00:44.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.838+0000 7ff8d4400700 1 -- 172.21.15.78:0/1967573513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8cc1a68d0 con 0x7ff8cc1006a0 2024-03-14T10:00:44.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.841+0000 7ff8ca000700 1 -- 172.21.15.78:0/1967573513 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7ff8c4003a00 con 0x7ff8cc1006a0 2024-03-14T10:00:44.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.842+0000 7ff8ca000700 1 --2- 172.21.15.78:0/1967573513 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff8b40a0c80 0x7ff8b40a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.842+0000 7ff8ca000700 1 -- 172.21.15.78:0/1967573513 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7ff8c40cc2a0 con 0x7ff8cc1006a0 2024-03-14T10:00:44.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.842+0000 7ff8d1a00700 1 --2- 172.21.15.78:0/1967573513 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff8b40a0c80 0x7ff8b40a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.843+0000 7ff8d1a00700 1 --2- 172.21.15.78:0/1967573513 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff8b40a0c80 0x7ff8b40a3140 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0009990 tx=0x7ff8c0005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:44.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.843+0000 7ff8e7fa4b80 1 Processor -- start 2024-03-14T10:00:44.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.844+0000 7ff8e7fa4b80 1 -- start start 2024-03-14T10:00:44.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.844+0000 7ff8e7fa4b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 0x55c265388ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.844+0000 7ff8e7fa4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 0x55c265392a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.844+0000 7ff8e7fa4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 0x55c265395330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.844+0000 7ff8e7fa4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2652e1a10 con 0x55c265389020 2024-03-14T10:00:44.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.844+0000 7ff8e7fa4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2652df3d0 con 0x55c265392f40 2024-03-14T10:00:44.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.844+0000 7ff8e7fa4b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2652dacc0 con 0x55c265388700 2024-03-14T10:00:44.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.844+0000 7ff8d1000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 0x55c265392a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.844+0000 7ff8d2400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 0x55c265395330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.845+0000 7ff8d1000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 0x55c265392a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47262/0 (socket says 172.21.15.78:47262) 2024-03-14T10:00:44.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.845+0000 7ff8d1000700 1 -- 172.21.15.78:0/3565957009 learned_addr learned my addr 172.21.15.78:0/3565957009 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:44.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.845+0000 7ff8d2400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 0x55c265395330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:49718/0 (socket says 172.21.15.78:49718) 2024-03-14T10:00:44.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.845+0000 7ff8d1a00700 1 --2- 172.21.15.78:0/3565957009 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 0x55c265388ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.845+0000 7ff8d1000700 1 -- 172.21.15.78:0/3565957009 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 msgr2=0x55c265388ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:44.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.845+0000 7ff8d1000700 1 --2- 172.21.15.78:0/3565957009 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 0x55c265388ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.845+0000 7ff8d1000700 1 -- 172.21.15.78:0/3565957009 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 msgr2=0x55c265395330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:44.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.845+0000 7ff8d1000700 1 --2- 172.21.15.78:0/3565957009 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 0x55c265395330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.845+0000 7ff8d1000700 1 -- 172.21.15.78:0/3565957009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c26515c430 con 0x55c265389020 2024-03-14T10:00:44.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.845+0000 7ff8d2400700 1 --2- 172.21.15.78:0/3565957009 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 0x55c265395330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:44.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.845+0000 7ff8d1000700 1 --2- 172.21.15.78:0/3565957009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 0x55c265392a00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8011540 tx=0x7ff8b8011850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85608011ef708f8f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:44.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.845+0000 7ff8bea00700 1 -- 172.21.15.78:0/3565957009 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b8004a30 con 0x55c265389020 2024-03-14T10:00:44.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.846+0000 7ff8bea00700 1 -- 172.21.15.78:0/3565957009 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff8b80054e0 con 0x55c265389020 2024-03-14T10:00:44.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.846+0000 7ff8bea00700 1 -- 172.21.15.78:0/3565957009 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b800e8b0 con 0x55c265389020 2024-03-14T10:00:44.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.846+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3565957009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 msgr2=0x55c265392a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:44.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.846+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3565957009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 0x55c265392a00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8011540 tx=0x7ff8b8011850 comp rx=0 tx=0).stop 2024-03-14T10:00:44.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.847+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3565957009 shutdown_connections 2024-03-14T10:00:44.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.847+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3565957009 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 0x55c265395330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.847+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3565957009 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 0x55c265388ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.847+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3565957009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 0x55c265392a00 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.847+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3565957009 >> 172.21.15.78:0/3565957009 conn(0x55c2652fe2a0 msgr2=0x55c265383b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:44.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.847+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3565957009 shutdown_connections 2024-03-14T10:00:44.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.847+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3565957009 wait complete. 2024-03-14T10:00:44.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.848+0000 7ff8e7fa4b80 1 -- start start 2024-03-14T10:00:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.848+0000 7ff8e7fa4b80 1 Processor -- start 2024-03-14T10:00:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.849+0000 7ff8e7fa4b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 0x55c265392920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.849+0000 7ff8e7fa4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 0x55c26538fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.849+0000 7ff8e7fa4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 0x55c2653902b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.849+0000 7ff8e7fa4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c265276e00 con 0x55c265389020 2024-03-14T10:00:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.849+0000 7ff8e7fa4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c265276aa0 con 0x55c265392f40 2024-03-14T10:00:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.849+0000 7ff8e7fa4b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c26515e720 con 0x55c265388700 2024-03-14T10:00:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.849+0000 7ff8e7fa4b80 10 client.0 _ll_register_callbacks cb 0x55c2650e2420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.849+0000 7ff8d1a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 0x55c265392920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.849+0000 7ff8d1a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 0x55c265392920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:51330/0 (socket says 172.21.15.78:51330) 2024-03-14T10:00:44.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.849+0000 7ff8d1000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 0x55c26538fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.849+0000 7ff8d1a00700 1 -- 172.21.15.78:0/3070568221 learned_addr learned my addr 172.21.15.78:0/3070568221 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:44.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.849+0000 7ff8d2400700 1 --2- 172.21.15.78:0/3070568221 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 0x55c2653902b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.850+0000 7ff8d1000700 1 -- 172.21.15.78:0/3070568221 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 msgr2=0x55c265392920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:44.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.850+0000 7ff8d1000700 1 --2- 172.21.15.78:0/3070568221 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 0x55c265392920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.850+0000 7ff8d1000700 1 -- 172.21.15.78:0/3070568221 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 msgr2=0x55c2653902b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:44.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.850+0000 7ff8d1000700 1 --2- 172.21.15.78:0/3070568221 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 0x55c2653902b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.850+0000 7ff8d1000700 1 -- 172.21.15.78:0/3070568221 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8cc1a68d0 con 0x55c265389020 2024-03-14T10:00:44.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.850+0000 7ff8d1000700 1 --2- 172.21.15.78:0/3070568221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 0x55c26538fca0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff8b801b040 tx=0x7ff8b80058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6106d5ef8b751180 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.850+0000 7ff8b3400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:00:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.850+0000 7ff8b3400700 1 -- 172.21.15.78:0/3070568221 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b8005ce0 con 0x55c265389020 2024-03-14T10:00:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.850+0000 7ff8b3400700 1 -- 172.21.15.78:0/3070568221 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7ff8b8004960 con 0x55c265389020 2024-03-14T10:00:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.850+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3070568221 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c26537b7a0 con 0x55c265389020 2024-03-14T10:00:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.851+0000 7ff8b3400700 1 -- 172.21.15.78:0/3070568221 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8b8004b20 con 0x55c265389020 2024-03-14T10:00:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.851+0000 7ff8e7fa4b80 10 client.16920 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.851+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3070568221 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7ff8b8009d10 con 0x55c265389020 2024-03-14T10:00:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.851+0000 7ff8b3400700 1 -- 172.21.15.78:0/3070568221 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c26537b7a0 con 0x55c265389020 2024-03-14T10:00:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.851+0000 7ff8b3400700 1 -- 172.21.15.78:0/3070568221 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff8b800e510 con 0x55c265389020 2024-03-14T10:00:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.851+0000 7ff8e7fa4b80 10 client.16920 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.851+0000 7ff8e7fa4b80 10 client.16920 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:00:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.851+0000 7ff8e7fa4b80 20 client.16920 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.851+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3070568221 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c2653a29d0 0x55c2653a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.851+0000 7ff8e7fa4b80 4 client.16920 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:00:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.851+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3070568221 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2652d3780 con 0x55c2653a29d0 2024-03-14T10:00:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.852+0000 7ff8d1a00700 1 --2- 172.21.15.78:0/3070568221 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c2653a29d0 0x55c2653a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.852+0000 7ff8b3400700 10 client.16920 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.852+0000 7ff8d1a00700 1 --2- 172.21.15.78:0/3070568221 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c2653a29d0 0x55c2653a4df0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e623d2bddb35a17a server_cookie=eb461e1263abec51 in_seq=0 out_seq=0 2024-03-14T10:00:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.853+0000 7ff8b3400700 1 -- 172.21.15.78:0/3070568221 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55c2652d3780 con 0x55c2653a29d0 2024-03-14T10:00:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.853+0000 7ff8b3400700 10 client.16920 handle_command_reply: tid=0 2024-03-14T10:00:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.853+0000 7ff8e7fa4b80 1 client.16920 shutdown 2024-03-14T10:00:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.853+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3070568221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 msgr2=0x55c26538fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.853+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3070568221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 0x55c26538fca0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff8b801b040 tx=0x7ff8b80058e0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.853+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3070568221 shutdown_connections 2024-03-14T10:00:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.853+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3070568221 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c2653a29d0 0x55c2653a4df0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.853+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3070568221 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 0x55c2653902b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.853+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3070568221 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 0x55c265392920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.853+0000 7ff8b3400700 0 client.16920 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.853+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3070568221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 0x55c26538fca0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.853+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3070568221 >> 172.21.15.78:0/3070568221 conn(0x55c2652fe2a0 msgr2=0x55c265383710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:44.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.853+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3070568221 shutdown_connections 2024-03-14T10:00:44.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.853+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3070568221 wait complete. 2024-03-14T10:00:44.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.854+0000 7ff8e7fa4b80 20 client.16920 trim_cache size 0 max 16384 2024-03-14T10:00:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.877+0000 7ff8e7fa4b80 1 Processor -- start 2024-03-14T10:00:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.877+0000 7ff8e7fa4b80 1 -- start start 2024-03-14T10:00:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.877+0000 7ff8e7fa4b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 0x55c265399bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.877+0000 7ff8e7fa4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 0x55c26538afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.877+0000 7ff8e7fa4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 0x55c26538b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.877+0000 7ff8e7fa4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2653a51c0 con 0x55c265389020 2024-03-14T10:00:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.877+0000 7ff8e7fa4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c265382880 con 0x55c265392f40 2024-03-14T10:00:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.877+0000 7ff8e7fa4b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c265388e90 con 0x55c265388700 2024-03-14T10:00:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.878+0000 7ff8d1a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 0x55c265399bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.878+0000 7ff8d1a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 0x55c265399bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:51346/0 (socket says 172.21.15.78:51346) 2024-03-14T10:00:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.878+0000 7ff8d1a00700 1 -- 172.21.15.78:0/3519460374 learned_addr learned my addr 172.21.15.78:0/3519460374 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.878+0000 7ff8d1000700 1 --2- 172.21.15.78:0/3519460374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 0x55c26538afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.878+0000 7ff8d2400700 1 --2- 172.21.15.78:0/3519460374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 0x55c26538b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.878+0000 7ff8d1a00700 1 -- 172.21.15.78:0/3519460374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 msgr2=0x55c26538b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.878+0000 7ff8d1a00700 1 --2- 172.21.15.78:0/3519460374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 0x55c26538b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.878+0000 7ff8d1a00700 1 -- 172.21.15.78:0/3519460374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 msgr2=0x55c26538afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.878+0000 7ff8d1a00700 1 --2- 172.21.15.78:0/3519460374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 0x55c26538afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.878+0000 7ff8d1a00700 1 -- 172.21.15.78:0/3519460374 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c26537f9d0 con 0x55c265388700 2024-03-14T10:00:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.879+0000 7ff8d1a00700 1 --2- 172.21.15.78:0/3519460374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 0x55c265399bd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0009fd0 tx=0x7ff8c0020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=446cdf7afdd5d757 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.880+0000 7ff8bf400700 1 -- 172.21.15.78:0/3519460374 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c000bed0 con 0x55c265388700 2024-03-14T10:00:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.880+0000 7ff8bf400700 1 -- 172.21.15.78:0/3519460374 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff8c001e040 con 0x55c265388700 2024-03-14T10:00:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.880+0000 7ff8bf400700 1 -- 172.21.15.78:0/3519460374 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0034ea0 con 0x55c265388700 2024-03-14T10:00:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.881+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3519460374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 msgr2=0x55c265399bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.881+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3519460374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 0x55c265399bd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0009fd0 tx=0x7ff8c0020000 comp rx=0 tx=0).stop 2024-03-14T10:00:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.881+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3519460374 shutdown_connections 2024-03-14T10:00:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.881+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3519460374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265392f40 0x55c26538b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.881+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3519460374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265388700 0x55c265399bd0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.881+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3519460374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265389020 0x55c26538afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.881+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3519460374 >> 172.21.15.78:0/3519460374 conn(0x55c2652fe2a0 msgr2=0x55c26538f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.881+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3519460374 shutdown_connections 2024-03-14T10:00:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.881+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3519460374 wait complete. 2024-03-14T10:00:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.882+0000 7ff8e7fa4b80 1 -- start start 2024-03-14T10:00:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.882+0000 7ff8e7fa4b80 1 Processor -- start 2024-03-14T10:00:44.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.883+0000 7ff8e7fa4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265388700 0x55c265398a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.883+0000 7ff8e7fa4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265389020 0x55c265398f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.883+0000 7ff8e7fa4b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265392f40 0x55c2653ee4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.883+0000 7ff8e7fa4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2653eea10 con 0x55c265388700 2024-03-14T10:00:44.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.883+0000 7ff8e7fa4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2653eeb50 con 0x55c265389020 2024-03-14T10:00:44.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.883+0000 7ff8e7fa4b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2653eed10 con 0x55c265392f40 2024-03-14T10:00:44.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.883+0000 7ff8e7fa4b80 10 client.0 _ll_register_callbacks cb 0x7ff8c4009370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:44.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.883+0000 7ff8d2400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265392f40 0x55c2653ee4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.883+0000 7ff8d1000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265389020 0x55c265398f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.883+0000 7ff8d2400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265392f40 0x55c2653ee4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:51348/0 (socket says 172.21.15.78:51348) 2024-03-14T10:00:44.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.883+0000 7ff8d1a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265388700 0x55c265398a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.883+0000 7ff8d2400700 1 -- 172.21.15.78:0/3922380719 learned_addr learned my addr 172.21.15.78:0/3922380719 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:44.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.884+0000 7ff8d2400700 1 -- 172.21.15.78:0/3922380719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265389020 msgr2=0x55c265398f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:44.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.884+0000 7ff8d2400700 1 --2- 172.21.15.78:0/3922380719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265389020 0x55c265398f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.884+0000 7ff8d2400700 1 -- 172.21.15.78:0/3922380719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265388700 msgr2=0x55c265398a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:44.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.884+0000 7ff8d2400700 1 --2- 172.21.15.78:0/3922380719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265388700 0x55c265398a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.884+0000 7ff8d2400700 1 -- 172.21.15.78:0/3922380719 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c002b040 con 0x55c265392f40 2024-03-14T10:00:44.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.884+0000 7ff8d1000700 1 --2- 172.21.15.78:0/3922380719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265389020 0x55c265398f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:44.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.884+0000 7ff8d2400700 1 --2- 172.21.15.78:0/3922380719 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265392f40 0x55c2653ee4d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4095d40 tx=0x7ff8c4004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f0d22643266c6a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:44.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.884+0000 7ff8b3400700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:00:44.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.884+0000 7ff8b3400700 1 -- 172.21.15.78:0/3922380719 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4099730 con 0x55c265392f40 2024-03-14T10:00:44.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.884+0000 7ff8b3400700 1 -- 172.21.15.78:0/3922380719 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7ff8c401b320 con 0x55c265392f40 2024-03-14T10:00:44.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.884+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3922380719 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff8b80054e0 con 0x55c265392f40 2024-03-14T10:00:44.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.884+0000 7ff8b3400700 1 -- 172.21.15.78:0/3922380719 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8c40999d0 con 0x55c265392f40 2024-03-14T10:00:44.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.885+0000 7ff8e7fa4b80 10 client.26986 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:44.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.885+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3922380719 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c265108be0 con 0x55c265392f40 2024-03-14T10:00:44.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.885+0000 7ff8b3400700 1 -- 172.21.15.78:0/3922380719 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff8b80054e0 con 0x55c265392f40 2024-03-14T10:00:44.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.885+0000 7ff8b3400700 1 -- 172.21.15.78:0/3922380719 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff8c40987c0 con 0x55c265392f40 2024-03-14T10:00:44.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.885+0000 7ff8e7fa4b80 10 client.26986 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:44.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.885+0000 7ff8e7fa4b80 10 client.26986 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:00:44.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.885+0000 7ff8e7fa4b80 20 client.26986 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:44.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.885+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3922380719 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c2653f6510 0x55c2653f8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:44.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.885+0000 7ff8e7fa4b80 4 client.26986 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T10:00:44.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.886+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3922380719 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c265129d70 con 0x55c2653f6510 2024-03-14T10:00:44.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.886+0000 7ff8d1a00700 1 --2- 172.21.15.78:0/3922380719 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c2653f6510 0x55c2653f8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:44.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.886+0000 7ff8d1a00700 1 --2- 172.21.15.78:0/3922380719 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c2653f6510 0x55c2653f8930 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e34886934fd7dac3 server_cookie=ff95d4d78d3de6be in_seq=0 out_seq=0 2024-03-14T10:00:44.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.886+0000 7ff8b3400700 10 client.26986 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:44.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.887+0000 7ff8b3400700 1 -- 172.21.15.78:0/3922380719 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c265129d70 con 0x55c2653f6510 2024-03-14T10:00:44.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.887+0000 7ff8b3400700 10 client.26986 handle_command_reply: tid=0 2024-03-14T10:00:44.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.887+0000 7ff8e7fa4b80 1 client.26986 shutdown 2024-03-14T10:00:44.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.888+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3922380719 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265392f40 msgr2=0x55c2653ee4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:44.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.888+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3922380719 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265392f40 0x55c2653ee4d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4095d40 tx=0x7ff8c4004ad0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.888+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3922380719 shutdown_connections 2024-03-14T10:00:44.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.888+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3922380719 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c265392f40 0x55c2653ee4d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.888+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3922380719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c265388700 0x55c265398a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.888+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3922380719 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c2653f6510 0x55c2653f8930 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.888+0000 7ff8e7fa4b80 1 --2- 172.21.15.78:0/3922380719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c265389020 0x55c265398f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.888+0000 7ff8b3400700 0 client.26986 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:44.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.888+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3922380719 >> 172.21.15.78:0/3922380719 conn(0x55c2652fe2a0 msgr2=0x55c265383630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:44.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.888+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3922380719 shutdown_connections 2024-03-14T10:00:44.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.888+0000 7ff8e7fa4b80 1 -- 172.21.15.78:0/3922380719 wait complete. 2024-03-14T10:00:44.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.888+0000 7ff8e7fa4b80 20 client.26986 trim_cache size 0 max 16384 2024-03-14T10:00:44.891 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:00:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T10:00:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "33afc302-09ee-47d0-8fa4-a3450f15fdbf", 2024-03-14T10:00:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T10:00:44.891 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:00:44.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.889+0000 7ff8d4400700 1 -- 172.21.15.78:0/1967573513 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff8b40a0c80 msgr2=0x7ff8b40a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:44.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.889+0000 7ff8d4400700 1 --2- 172.21.15.78:0/1967573513 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff8b40a0c80 0x7ff8b40a3140 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0009990 tx=0x7ff8c0005fd0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.889+0000 7ff8d4400700 1 -- 172.21.15.78:0/1967573513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8cc1006a0 msgr2=0x7ff8cc1a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:44.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.889+0000 7ff8d4400700 1 --2- 172.21.15.78:0/1967573513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8cc1006a0 0x7ff8cc1a5840 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400ef30 tx=0x7ff8c400c3a0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.889+0000 7ff8d4400700 1 -- 172.21.15.78:0/1967573513 shutdown_connections 2024-03-14T10:00:44.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.889+0000 7ff8d4400700 1 --2- 172.21.15.78:0/1967573513 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff8b40a0c80 0x7ff8b40a3140 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.889+0000 7ff8d4400700 1 --2- 172.21.15.78:0/1967573513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8cc0680b0 0x7ff8cc0ffc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.889+0000 7ff8d4400700 1 --2- 172.21.15.78:0/1967573513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8cc1006a0 0x7ff8cc1a5840 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.889+0000 7ff8d4400700 1 --2- 172.21.15.78:0/1967573513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff8cc106060 0x7ff8cc100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:44.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.889+0000 7ff8d4400700 1 -- 172.21.15.78:0/1967573513 >> 172.21.15.78:0/1967573513 conn(0x7ff8cc076d80 msgr2=0x7ff8cc0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:44.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.889+0000 7ff8d4400700 1 -- 172.21.15.78:0/1967573513 shutdown_connections 2024-03-14T10:00:44.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:44.890+0000 7ff8d4400700 1 -- 172.21.15.78:0/1967573513 wait complete. 2024-03-14T10:00:44.919 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:00:45.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.176+0000 7ffa9ae00700 1 -- 172.21.15.78:0/1340511629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa94068a60 msgr2=0x7ffa94105930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:45.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.176+0000 7ffa9ae00700 1 --2- 172.21.15.78:0/1340511629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa94068a60 0x7ffa94105930 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ffa84009920 tx=0x7ffa84009c30 comp rx=0 tx=0).stop 2024-03-14T10:00:45.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.176+0000 7ffa9ae00700 1 -- 172.21.15.78:0/1340511629 shutdown_connections 2024-03-14T10:00:45.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.176+0000 7ffa9ae00700 1 --2- 172.21.15.78:0/1340511629 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa940680b0 0x7ffa94068490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.176+0000 7ffa9ae00700 1 --2- 172.21.15.78:0/1340511629 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa94106060 0x7ffa9410dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.176+0000 7ffa9ae00700 1 --2- 172.21.15.78:0/1340511629 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa94068a60 0x7ffa94105930 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.176+0000 7ffa9ae00700 1 -- 172.21.15.78:0/1340511629 >> 172.21.15.78:0/1340511629 conn(0x7ffa94076d80 msgr2=0x7ffa94077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:45.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.176+0000 7ffa9ae00700 1 -- 172.21.15.78:0/1340511629 shutdown_connections 2024-03-14T10:00:45.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.176+0000 7ffa9ae00700 1 -- 172.21.15.78:0/1340511629 wait complete. 2024-03-14T10:00:45.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.177+0000 7ffa9ae00700 1 Processor -- start 2024-03-14T10:00:45.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.177+0000 7ffa9ae00700 1 -- start start 2024-03-14T10:00:45.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.178+0000 7ffa9ae00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa940680b0 0x7ffa941976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.178+0000 7ffa9ae00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa94106060 0x7ffa94197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.178+0000 7ffa9ae00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa9419bf90 0x7ffa9419c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.178+0000 7ffa9ae00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9419ca10 con 0x7ffa9419bf90 2024-03-14T10:00:45.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.178+0000 7ffa9ae00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9419cb80 con 0x7ffa94106060 2024-03-14T10:00:45.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.178+0000 7ffa9ae00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9419cd40 con 0x7ffa940680b0 2024-03-14T10:00:45.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.179+0000 7ffa93e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa940680b0 0x7ffa941976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.179+0000 7ffa93400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa94106060 0x7ffa94197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.179+0000 7ffa93e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa940680b0 0x7ffa941976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:51362/0 (socket says 172.21.15.78:51362) 2024-03-14T10:00:45.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.179+0000 7ffa93400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa94106060 0x7ffa94197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:49766/0 (socket says 172.21.15.78:49766) 2024-03-14T10:00:45.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.179+0000 7ffa93e00700 1 -- 172.21.15.78:0/2249481807 learned_addr learned my addr 172.21.15.78:0/2249481807 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:45.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.179+0000 7ffa98e00700 1 --2- 172.21.15.78:0/2249481807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa9419bf90 0x7ffa9419c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.179+0000 7ffa93e00700 1 -- 172.21.15.78:0/2249481807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa94106060 msgr2=0x7ffa94197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:45.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.179+0000 7ffa93e00700 1 --2- 172.21.15.78:0/2249481807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa94106060 0x7ffa94197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.179+0000 7ffa93e00700 1 -- 172.21.15.78:0/2249481807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa9419bf90 msgr2=0x7ffa9419c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:45.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.179+0000 7ffa93e00700 1 --2- 172.21.15.78:0/2249481807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa9419bf90 0x7ffa9419c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.179+0000 7ffa93e00700 1 -- 172.21.15.78:0/2249481807 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa84009520 con 0x7ffa940680b0 2024-03-14T10:00:45.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.180+0000 7ffa93e00700 1 --2- 172.21.15.78:0/2249481807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa940680b0 0x7ffa941976b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c007fa0 tx=0x7ffa8c00d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:45.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.180+0000 7ffa90c00700 1 -- 172.21.15.78:0/2249481807 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c00dcf0 con 0x7ffa940680b0 2024-03-14T10:00:45.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.180+0000 7ffa90c00700 1 -- 172.21.15.78:0/2249481807 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa8c00f040 con 0x7ffa940680b0 2024-03-14T10:00:45.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.180+0000 7ffa90c00700 1 -- 172.21.15.78:0/2249481807 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c014600 con 0x7ffa940680b0 2024-03-14T10:00:45.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.180+0000 7ffa9ae00700 1 -- 172.21.15.78:0/2249481807 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa940ffb50 con 0x7ffa940680b0 2024-03-14T10:00:45.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.180+0000 7ffa9ae00700 1 -- 172.21.15.78:0/2249481807 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa940fffe0 con 0x7ffa940680b0 2024-03-14T10:00:45.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.184+0000 7ffa90c00700 1 -- 172.21.15.78:0/2249481807 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7ffa8c015950 con 0x7ffa940680b0 2024-03-14T10:00:45.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.186+0000 7ffaae9bab80 1 Processor -- start 2024-03-14T10:00:45.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.186+0000 7ffaae9bab80 1 -- start start 2024-03-14T10:00:45.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.186+0000 7ffaae9bab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5d1680 0x562c0a5d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.186+0000 7ffaae9bab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1fa0 0x562c0a5db980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.186+0000 7ffa90c00700 1 --2- 172.21.15.78:0/2249481807 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ffa780a0c80 0x7ffa780a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.186+0000 7ffaae9bab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5dbec0 0x562c0a5de2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.186+0000 7ffaae9bab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c0a52a990 con 0x562c0a5d1680 2024-03-14T10:00:45.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.186+0000 7ffaae9bab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c0a528350 con 0x562c0a5d1fa0 2024-03-14T10:00:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.186+0000 7ffaae9bab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c0a523c40 con 0x562c0a5dbec0 2024-03-14T10:00:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.186+0000 7ffa90c00700 1 -- 172.21.15.78:0/2249481807 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7ffa8c0cbdd0 con 0x7ffa940680b0 2024-03-14T10:00:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.187+0000 7ffa98e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5d1680 0x562c0a5d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.187+0000 7ffa93400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1fa0 0x562c0a5db980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.187+0000 7ffa98e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5d1680 0x562c0a5d1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47310/0 (socket says 172.21.15.78:47310) 2024-03-14T10:00:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.187+0000 7ffa98e00700 1 -- 172.21.15.78:0/2347121300 learned_addr learned my addr 172.21.15.78:0/2347121300 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.187+0000 7ffa93e00700 1 --2- 172.21.15.78:0/2249481807 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ffa780a0c80 0x7ffa780a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.187+0000 7ffa98e00700 1 --2- 172.21.15.78:0/2347121300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5dbec0 0x562c0a5de2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.187+0000 7ffa93400700 1 -- 172.21.15.78:0/2347121300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5dbec0 msgr2=0x562c0a5de2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.187+0000 7ffa93400700 1 --2- 172.21.15.78:0/2347121300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5dbec0 0x562c0a5de2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.187+0000 7ffa93400700 1 -- 172.21.15.78:0/2347121300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5d1680 msgr2=0x562c0a5d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:45.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.187+0000 7ffa93400700 1 --2- 172.21.15.78:0/2347121300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5d1680 0x562c0a5d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.187+0000 7ffa93400700 1 -- 172.21.15.78:0/2347121300 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c0a3a53b0 con 0x562c0a5d1fa0 2024-03-14T10:00:45.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.188+0000 7ffa93e00700 1 --2- 172.21.15.78:0/2249481807 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ffa780a0c80 0x7ffa780a3140 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c015ca0 tx=0x7ffa8c00df70 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:45.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.188+0000 7ffa93400700 1 --2- 172.21.15.78:0/2347121300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1fa0 0x562c0a5db980 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ffa84000c00 tx=0x7ffa84005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de25875e6b21f12c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:45.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.188+0000 7ffa89600700 1 -- 172.21.15.78:0/2347121300 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8400f460 con 0x562c0a5d1fa0 2024-03-14T10:00:45.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.188+0000 7ffa89600700 1 -- 172.21.15.78:0/2347121300 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa8401d040 con 0x562c0a5d1fa0 2024-03-14T10:00:45.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.188+0000 7ffa89600700 1 -- 172.21.15.78:0/2347121300 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa840052a0 con 0x562c0a5d1fa0 2024-03-14T10:00:45.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.189+0000 7ffaae9bab80 1 -- 172.21.15.78:0/2347121300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1fa0 msgr2=0x562c0a5db980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:45.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.189+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/2347121300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1fa0 0x562c0a5db980 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ffa84000c00 tx=0x7ffa84005dc0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.189+0000 7ffaae9bab80 1 -- 172.21.15.78:0/2347121300 shutdown_connections 2024-03-14T10:00:45.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.189+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/2347121300 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5dbec0 0x562c0a5de2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.189+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/2347121300 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5d1680 0x562c0a5d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.189+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/2347121300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1fa0 0x562c0a5db980 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.189+0000 7ffaae9bab80 1 -- 172.21.15.78:0/2347121300 >> 172.21.15.78:0/2347121300 conn(0x562c0a547220 msgr2=0x562c0a5ccaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:45.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.189+0000 7ffaae9bab80 1 -- 172.21.15.78:0/2347121300 shutdown_connections 2024-03-14T10:00:45.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.189+0000 7ffaae9bab80 1 -- 172.21.15.78:0/2347121300 wait complete. 2024-03-14T10:00:45.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.190+0000 7ffaae9bab80 1 -- start start 2024-03-14T10:00:45.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.190+0000 7ffaae9bab80 1 Processor -- start 2024-03-14T10:00:45.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.191+0000 7ffaae9bab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1680 0x562c0a5db890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.191+0000 7ffaae9bab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1fa0 0x562c0a5d8c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.191+0000 7ffaae9bab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5dbec0 0x562c0a5d92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.191+0000 7ffaae9bab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c0a4bfd80 con 0x562c0a5dbec0 2024-03-14T10:00:45.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.191+0000 7ffaae9bab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c0a4bfa20 con 0x562c0a5d1680 2024-03-14T10:00:45.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.191+0000 7ffaae9bab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c0a3a76a0 con 0x562c0a5d1fa0 2024-03-14T10:00:45.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.191+0000 7ffaae9bab80 10 client.0 _ll_register_callbacks cb 0x562c0a32b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:45.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.191+0000 7ffa98e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1680 0x562c0a5db890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.191+0000 7ffa98e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1680 0x562c0a5db890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:49790/0 (socket says 172.21.15.78:49790) 2024-03-14T10:00:45.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.191+0000 7ffa98e00700 1 -- 172.21.15.78:0/1682006717 learned_addr learned my addr 172.21.15.78:0/1682006717 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:45.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.192+0000 7ffa98e00700 1 --2- 172.21.15.78:0/1682006717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5dbec0 0x562c0a5d92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.192+0000 7ffa98e00700 1 -- 172.21.15.78:0/1682006717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1fa0 msgr2=0x562c0a5d8c20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-14T10:00:45.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.192+0000 7ffa98e00700 1 --2- 172.21.15.78:0/1682006717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1fa0 0x562c0a5d8c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.192+0000 7ffa98e00700 1 -- 172.21.15.78:0/1682006717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5dbec0 msgr2=0x562c0a5d92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:45.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.192+0000 7ffa98e00700 1 --2- 172.21.15.78:0/1682006717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5dbec0 0x562c0a5d92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.192+0000 7ffa98e00700 1 -- 172.21.15.78:0/1682006717 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9410e420 con 0x562c0a5d1680 2024-03-14T10:00:45.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.193+0000 7ffa98e00700 1 --2- 172.21.15.78:0/1682006717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1680 0x562c0a5db890 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009230 tx=0x7ffa7c015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12b5596035284a97 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:45.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.193+0000 7ffa82000700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T10:00:45.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.193+0000 7ffa82000700 1 -- 172.21.15.78:0/1682006717 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c018070 con 0x562c0a5d1680 2024-03-14T10:00:45.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.193+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1682006717 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562c0a5c4720 con 0x562c0a5d1680 2024-03-14T10:00:45.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.193+0000 7ffa82000700 1 -- 172.21.15.78:0/1682006717 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7ffa7c00fc70 con 0x562c0a5d1680 2024-03-14T10:00:45.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.193+0000 7ffa82000700 1 -- 172.21.15.78:0/1682006717 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa7c00fe30 con 0x562c0a5d1680 2024-03-14T10:00:45.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.194+0000 7ffaae9bab80 10 client.27020 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:45.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.194+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1682006717 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562c0a351b60 con 0x562c0a5d1680 2024-03-14T10:00:45.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.194+0000 7ffa82000700 1 -- 172.21.15.78:0/1682006717 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562c0a5c4720 con 0x562c0a5d1680 2024-03-14T10:00:45.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.194+0000 7ffa82000700 1 -- 172.21.15.78:0/1682006717 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ffa7c0203c0 con 0x562c0a5d1680 2024-03-14T10:00:45.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.194+0000 7ffaae9bab80 10 client.27020 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:45.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.194+0000 7ffaae9bab80 10 client.27020 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:00:45.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.194+0000 7ffaae9bab80 20 client.27020 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:45.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.195+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/1682006717 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x562c0a5ebac0 0x562c0a5edee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.195+0000 7ffaae9bab80 4 client.27020 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:00:45.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.195+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1682006717 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562c0a51c700 con 0x562c0a5ebac0 2024-03-14T10:00:45.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.195+0000 7ffa93400700 1 --2- 172.21.15.78:0/1682006717 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x562c0a5ebac0 0x562c0a5edee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.195+0000 7ffa93400700 1 --2- 172.21.15.78:0/1682006717 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x562c0a5ebac0 0x562c0a5edee0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84c20881aa730da7 server_cookie=41d0cdd6dfb18d9c in_seq=0 out_seq=0 2024-03-14T10:00:45.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.195+0000 7ffa82000700 10 client.27020 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.197+0000 7ffa82000700 1 -- 172.21.15.78:0/1682006717 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x562c0a51c700 con 0x562c0a5ebac0 2024-03-14T10:00:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.197+0000 7ffa82000700 10 client.27020 handle_command_reply: tid=0 2024-03-14T10:00:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.197+0000 7ffaae9bab80 1 client.27020 shutdown 2024-03-14T10:00:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.197+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1682006717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1680 msgr2=0x562c0a5db890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.197+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/1682006717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1680 0x562c0a5db890 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009230 tx=0x7ffa7c015040 comp rx=0 tx=0).stop 2024-03-14T10:00:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.197+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1682006717 shutdown_connections 2024-03-14T10:00:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.197+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/1682006717 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x562c0a5ebac0 0x562c0a5edee0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.197+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/1682006717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1680 0x562c0a5db890 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.197+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/1682006717 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5dbec0 0x562c0a5d92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.197+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/1682006717 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1fa0 0x562c0a5d8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.197+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1682006717 >> 172.21.15.78:0/1682006717 conn(0x562c0a547220 msgr2=0x562c0a5cc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:45.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.197+0000 7ffa82000700 0 client.27020 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:45.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.197+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1682006717 shutdown_connections 2024-03-14T10:00:45.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.197+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1682006717 wait complete. 2024-03-14T10:00:45.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.197+0000 7ffaae9bab80 20 client.27020 trim_cache size 0 max 16384 2024-03-14T10:00:45.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.219+0000 7ffaae9bab80 1 Processor -- start 2024-03-14T10:00:45.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.219+0000 7ffaae9bab80 1 -- start start 2024-03-14T10:00:45.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.220+0000 7ffaae9bab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1680 0x562c0a5ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.220+0000 7ffaae9bab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1fa0 0x562c0a5d74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.220+0000 7ffaae9bab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5dbec0 0x562c0a5d7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.220+0000 7ffaae9bab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c0a5d3340 con 0x562c0a5dbec0 2024-03-14T10:00:45.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.220+0000 7ffaae9bab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c0a5d2b20 con 0x562c0a5d1fa0 2024-03-14T10:00:45.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.220+0000 7ffaae9bab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c0a5d1de0 con 0x562c0a5d1680 2024-03-14T10:00:45.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.221+0000 7ffa98e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1680 0x562c0a5ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.221+0000 7ffa98e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1680 0x562c0a5ccc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:51388/0 (socket says 172.21.15.78:51388) 2024-03-14T10:00:45.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.221+0000 7ffa98e00700 1 -- 172.21.15.78:0/1668469919 learned_addr learned my addr 172.21.15.78:0/1668469919 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:45.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.222+0000 7ffa98e00700 1 --2- 172.21.15.78:0/1668469919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5dbec0 0x562c0a5d7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.222+0000 7ffa98e00700 1 -- 172.21.15.78:0/1668469919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1680 msgr2=0x562c0a5ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.222+0000 7ffa98e00700 1 --2- 172.21.15.78:0/1668469919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1680 0x562c0a5ccc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.222+0000 7ffa93400700 1 --2- 172.21.15.78:0/1668469919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1fa0 0x562c0a5d74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.222+0000 7ffa98e00700 1 -- 172.21.15.78:0/1668469919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1fa0 msgr2=0x562c0a5d74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.222+0000 7ffa98e00700 1 --2- 172.21.15.78:0/1668469919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1fa0 0x562c0a5d74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.222+0000 7ffa98e00700 1 -- 172.21.15.78:0/1668469919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c0a4e9ea0 con 0x562c0a5dbec0 2024-03-14T10:00:45.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.223+0000 7ffa93400700 1 --2- 172.21.15.78:0/1668469919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1fa0 0x562c0a5d74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T10:00:45.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.223+0000 7ffa98e00700 1 --2- 172.21.15.78:0/1668469919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5dbec0 0x562c0a5d7a30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00fa30 tx=0x7ffa7c00dfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b616f67e658f5c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:45.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.224+0000 7ffa8a000700 1 -- 172.21.15.78:0/1668469919 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c018070 con 0x562c0a5dbec0 2024-03-14T10:00:45.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.224+0000 7ffa8a000700 1 -- 172.21.15.78:0/1668469919 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa7c009b50 con 0x562c0a5dbec0 2024-03-14T10:00:45.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.224+0000 7ffa8a000700 1 -- 172.21.15.78:0/1668469919 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c00fc70 con 0x562c0a5dbec0 2024-03-14T10:00:45.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.225+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1668469919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5dbec0 msgr2=0x562c0a5d7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:45.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.225+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/1668469919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5dbec0 0x562c0a5d7a30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00fa30 tx=0x7ffa7c00dfd0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.225+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1668469919 shutdown_connections 2024-03-14T10:00:45.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.225+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/1668469919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5dbec0 0x562c0a5d7a30 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.225+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/1668469919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1680 0x562c0a5ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.225+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/1668469919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5d1fa0 0x562c0a5d74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.225+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1668469919 >> 172.21.15.78:0/1668469919 conn(0x562c0a547220 msgr2=0x562c0a5d3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:45.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.226+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1668469919 shutdown_connections 2024-03-14T10:00:45.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.226+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1668469919 wait complete. 2024-03-14T10:00:45.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.226+0000 7ffaae9bab80 1 -- start start 2024-03-14T10:00:45.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.227+0000 7ffaae9bab80 1 Processor -- start 2024-03-14T10:00:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.227+0000 7ffaae9bab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1680 0x562c0a5e10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.227+0000 7ffaae9bab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5d1fa0 0x562c0a5e1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.227+0000 7ffaae9bab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5dbec0 0x562c0a5e1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.227+0000 7ffaae9bab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c0a5e2f40 con 0x562c0a5d1fa0 2024-03-14T10:00:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.227+0000 7ffaae9bab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c0a63d610 con 0x562c0a5dbec0 2024-03-14T10:00:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.227+0000 7ffaae9bab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c0a63d7d0 con 0x562c0a5d1680 2024-03-14T10:00:45.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.228+0000 7ffaae9bab80 10 client.0 _ll_register_callbacks cb 0x7ffa7c010980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:00:45.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.230+0000 7ffa98e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1680 0x562c0a5e10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.230+0000 7ffa98e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1680 0x562c0a5e10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:51400/0 (socket says 172.21.15.78:51400) 2024-03-14T10:00:45.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.231+0000 7ffa98e00700 1 -- 172.21.15.78:0/1971751866 learned_addr learned my addr 172.21.15.78:0/1971751866 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:00:45.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.231+0000 7ffa98e00700 1 --2- 172.21.15.78:0/1971751866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5dbec0 0x562c0a5e1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.231+0000 7ffa93400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5d1fa0 0x562c0a5e1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.231+0000 7ffa98e00700 1 -- 172.21.15.78:0/1971751866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5dbec0 msgr2=0x562c0a5e1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:45.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.231+0000 7ffa98e00700 1 --2- 172.21.15.78:0/1971751866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5dbec0 0x562c0a5e1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.231+0000 7ffa98e00700 1 -- 172.21.15.78:0/1971751866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5d1fa0 msgr2=0x562c0a5e1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:00:45.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.232+0000 7ffa98e00700 1 --2- 172.21.15.78:0/1971751866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5d1fa0 0x562c0a5e1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.232+0000 7ffa98e00700 1 -- 172.21.15.78:0/1971751866 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562c0a351b60 con 0x562c0a5d1680 2024-03-14T10:00:45.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.232+0000 7ffa98e00700 1 --2- 172.21.15.78:0/1971751866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1680 0x562c0a5e10e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c010bd0 tx=0x7ffa7c020430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bdb56a12abdc88a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:00:45.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.232+0000 7ffa82000700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:00:45.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.232+0000 7ffa82000700 1 -- 172.21.15.78:0/1971751866 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c018040 con 0x562c0a5d1680 2024-03-14T10:00:45.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.232+0000 7ffa82000700 1 -- 172.21.15.78:0/1971751866 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9408+0+0 (secure 0 0 0) 0x7ffa7c000e40 con 0x562c0a5d1680 2024-03-14T10:00:45.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.233+0000 7ffa93400700 1 --2- 172.21.15.78:0/1971751866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5d1fa0 0x562c0a5e1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:00:45.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.233+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1971751866 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffa8401d040 con 0x562c0a5d1680 2024-03-14T10:00:45.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.233+0000 7ffa82000700 1 -- 172.21.15.78:0/1971751866 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa7c00fe80 con 0x562c0a5d1680 2024-03-14T10:00:45.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.233+0000 7ffaae9bab80 10 client.26998 fetch_fsmap learned FSMap version 25 2024-03-14T10:00:45.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.233+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1971751866 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562c0a3a9cd0 con 0x562c0a5d1680 2024-03-14T10:00:45.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.233+0000 7ffa82000700 1 -- 172.21.15.78:0/1971751866 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa8401d040 con 0x562c0a5d1680 2024-03-14T10:00:45.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.234+0000 7ffa82000700 1 -- 172.21.15.78:0/1971751866 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ffa7c039040 con 0x562c0a5d1680 2024-03-14T10:00:45.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.234+0000 7ffaae9bab80 10 client.26998 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:00:45.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.234+0000 7ffaae9bab80 10 client.26998 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:00:45.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.234+0000 7ffaae9bab80 20 client.26998 populate_metadata read hostname 'smithi078' 2024-03-14T10:00:45.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.234+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/1971751866 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x562c0a63e9e0 0x562c0a640e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:00:45.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.234+0000 7ffaae9bab80 4 client.26998 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:00:45.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.234+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1971751866 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562c0a372cf0 con 0x562c0a63e9e0 2024-03-14T10:00:45.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.234+0000 7ffa93400700 1 --2- 172.21.15.78:0/1971751866 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x562c0a63e9e0 0x562c0a640e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:00:45.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.235+0000 7ffa82000700 10 client.26998 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:45.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.235+0000 7ffa93400700 1 --2- 172.21.15.78:0/1971751866 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x562c0a63e9e0 0x562c0a640e00 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f0777c32a73c59f server_cookie=dbca7e46a72a4777 in_seq=0 out_seq=0 2024-03-14T10:00:45.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.235+0000 7ffa82000700 1 -- 172.21.15.78:0/1971751866 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x562c0a372cf0 con 0x562c0a63e9e0 2024-03-14T10:00:45.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.235+0000 7ffa82000700 10 client.26998 handle_command_reply: tid=0 2024-03-14T10:00:45.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.235+0000 7ffaae9bab80 1 client.26998 shutdown 2024-03-14T10:00:45.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.236+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1971751866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1680 msgr2=0x562c0a5e10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:45.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.236+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/1971751866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1680 0x562c0a5e10e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c010bd0 tx=0x7ffa7c020430 comp rx=0 tx=0).stop 2024-03-14T10:00:45.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.236+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1971751866 shutdown_connections 2024-03-14T10:00:45.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.236+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/1971751866 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x562c0a63e9e0 0x562c0a640e00 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.236+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/1971751866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562c0a5dbec0 0x562c0a5e1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.236+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/1971751866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562c0a5d1680 0x562c0a5e10e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.236+0000 7ffa82000700 0 client.26998 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:00:45.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.236+0000 7ffaae9bab80 1 --2- 172.21.15.78:0/1971751866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c0a5d1fa0 0x562c0a5e1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.236+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1971751866 >> 172.21.15.78:0/1971751866 conn(0x562c0a547220 msgr2=0x562c0a5d8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:45.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.236+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1971751866 shutdown_connections 2024-03-14T10:00:45.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.236+0000 7ffaae9bab80 1 -- 172.21.15.78:0/1971751866 wait complete. 2024-03-14T10:00:45.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.237+0000 7ffaae9bab80 20 client.26998 trim_cache size 0 max 16384 2024-03-14T10:00:45.238 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:00:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "scrub active (1452 inodes in the stack)", 2024-03-14T10:00:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T10:00:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "33afc302-09ee-47d0-8fa4-a3450f15fdbf": { 2024-03-14T10:00:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T10:00:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "33afc302-09ee-47d0-8fa4-a3450f15fdbf", 2024-03-14T10:00:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T10:00:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:00:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:00:45.239 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:00:45.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.237+0000 7ffa9ae00700 1 -- 172.21.15.78:0/2249481807 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ffa780a0c80 msgr2=0x7ffa780a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:45.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.237+0000 7ffa9ae00700 1 --2- 172.21.15.78:0/2249481807 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ffa780a0c80 0x7ffa780a3140 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c015ca0 tx=0x7ffa8c00df70 comp rx=0 tx=0).stop 2024-03-14T10:00:45.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.237+0000 7ffa9ae00700 1 -- 172.21.15.78:0/2249481807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa940680b0 msgr2=0x7ffa941976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:00:45.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.237+0000 7ffa9ae00700 1 --2- 172.21.15.78:0/2249481807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa940680b0 0x7ffa941976b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c007fa0 tx=0x7ffa8c00d3b0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.238+0000 7ffa9ae00700 1 -- 172.21.15.78:0/2249481807 shutdown_connections 2024-03-14T10:00:45.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.238+0000 7ffa9ae00700 1 --2- 172.21.15.78:0/2249481807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffa9419bf90 0x7ffa9419c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.238+0000 7ffa9ae00700 1 --2- 172.21.15.78:0/2249481807 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ffa940680b0 0x7ffa941976b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.238+0000 7ffa9ae00700 1 --2- 172.21.15.78:0/2249481807 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ffa780a0c80 0x7ffa780a3140 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.238+0000 7ffa9ae00700 1 --2- 172.21.15.78:0/2249481807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffa94106060 0x7ffa94197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:00:45.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.238+0000 7ffa9ae00700 1 -- 172.21.15.78:0/2249481807 >> 172.21.15.78:0/2249481807 conn(0x7ffa94076d80 msgr2=0x7ffa940fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:00:45.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.239+0000 7ffa9ae00700 1 -- 172.21.15.78:0/2249481807 shutdown_connections 2024-03-14T10:00:45.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:00:45.239+0000 7ffa9ae00700 1 -- 172.21.15.78:0/2249481807 wait complete. 2024-03-14T10:00:45.267 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:45 smithi078 ceph-mon[38143]: pgmap v200: 129 pgs: 129 active+clean; 242 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 1013 KiB/s wr, 357 op/s 2024-03-14T10:00:45.269 INFO:tasks.cephfs.filesystem:scrub status for tag:33afc302-09ee-47d0-8fa4-a3450f15fdbf - {'path': '/', 'tag': '33afc302-09ee-47d0-8fa4-a3450f15fdbf', 'options': 'recursive,force'} 2024-03-14T10:00:45.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:45 smithi152 ceph-mon[48279]: pgmap v200: 129 pgs: 129 active+clean; 242 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 1013 KiB/s wr, 357 op/s 2024-03-14T10:00:45.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:45 smithi132 ceph-mon[47959]: pgmap v200: 129 pgs: 129 active+clean; 242 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 1013 KiB/s wr, 357 op/s 2024-03-14T10:00:46.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:46 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T10:00:46.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:46 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T10:00:46.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:46 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T10:00:46.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:46 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T10:00:46.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:46 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T10:00:46.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:46 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T10:00:46.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:46 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T10:00:46.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:46 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T10:00:46.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:46 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T10:00:46.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:46 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T10:00:46.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:46 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T10:00:46.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:46 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T10:00:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:47 smithi152 ceph-mon[48279]: pgmap v201: 129 pgs: 129 active+clean; 236 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 1.5 MiB/s wr, 518 op/s 2024-03-14T10:00:47.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:47 smithi132 ceph-mon[47959]: pgmap v201: 129 pgs: 129 active+clean; 236 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 1.5 MiB/s wr, 518 op/s 2024-03-14T10:00:47.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:47 smithi078 ceph-mon[38143]: pgmap v201: 129 pgs: 129 active+clean; 236 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 1.5 MiB/s wr, 518 op/s 2024-03-14T10:00:49.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:48 smithi152 ceph-mon[48279]: pgmap v202: 129 pgs: 129 active+clean; 236 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.0 MiB/s wr, 338 op/s 2024-03-14T10:00:49.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:48 smithi132 ceph-mon[47959]: pgmap v202: 129 pgs: 129 active+clean; 236 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.0 MiB/s wr, 338 op/s 2024-03-14T10:00:49.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:48 smithi078 ceph-mon[38143]: pgmap v202: 129 pgs: 129 active+clean; 236 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.0 MiB/s wr, 338 op/s 2024-03-14T10:00:49.429 INFO:tasks.workunit.client.0.smithi078.stdout:All 50000 operations completed A-OK! 2024-03-14T10:00:49.457 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-14T10:00:49.457 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-14T10:00:50.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:49 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T10:00:50.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:49 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T10:00:50.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:49 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T10:00:51.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:50 smithi152 ceph-mon[48279]: pgmap v203: 129 pgs: 129 active+clean; 236 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.0 MiB/s wr, 338 op/s 2024-03-14T10:00:51.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:50 smithi132 ceph-mon[47959]: pgmap v203: 129 pgs: 129 active+clean; 236 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.0 MiB/s wr, 338 op/s 2024-03-14T10:00:51.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:50 smithi078 ceph-mon[38143]: pgmap v203: 129 pgs: 129 active+clean; 236 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 1.0 MiB/s wr, 338 op/s 2024-03-14T10:00:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:53 smithi152 ceph-mon[48279]: pgmap v204: 129 pgs: 129 active+clean; 246 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 1.6 MiB/s wr, 473 op/s 2024-03-14T10:00:53.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:53 smithi132 ceph-mon[47959]: pgmap v204: 129 pgs: 129 active+clean; 246 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 1.6 MiB/s wr, 473 op/s 2024-03-14T10:00:53.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:53 smithi078 ceph-mon[38143]: pgmap v204: 129 pgs: 129 active+clean; 246 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 1.6 MiB/s wr, 473 op/s 2024-03-14T10:00:55.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:55 smithi152 ceph-mon[48279]: pgmap v205: 129 pgs: 129 active+clean; 246 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.1 MiB/s wr, 296 op/s 2024-03-14T10:00:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:55 smithi132 ceph-mon[47959]: pgmap v205: 129 pgs: 129 active+clean; 246 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.1 MiB/s wr, 296 op/s 2024-03-14T10:00:55.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:55 smithi078 ceph-mon[38143]: pgmap v205: 129 pgs: 129 active+clean; 246 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 1.1 MiB/s wr, 296 op/s 2024-03-14T10:00:57.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:57 smithi152 ceph-mon[48279]: pgmap v206: 129 pgs: 129 active+clean; 261 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.4 MiB/s wr, 441 op/s 2024-03-14T10:00:57.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:57 smithi132 ceph-mon[47959]: pgmap v206: 129 pgs: 129 active+clean; 261 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.4 MiB/s wr, 441 op/s 2024-03-14T10:00:57.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:57 smithi078 ceph-mon[38143]: pgmap v206: 129 pgs: 129 active+clean; 261 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.4 MiB/s wr, 441 op/s 2024-03-14T10:00:59.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:00:58 smithi152 ceph-mon[48279]: pgmap v207: 129 pgs: 129 active+clean; 261 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.9 MiB/s wr, 280 op/s 2024-03-14T10:00:59.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:00:58 smithi132 ceph-mon[47959]: pgmap v207: 129 pgs: 129 active+clean; 261 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.9 MiB/s wr, 280 op/s 2024-03-14T10:00:59.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:00:58 smithi078 ceph-mon[38143]: pgmap v207: 129 pgs: 129 active+clean; 261 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.9 MiB/s wr, 280 op/s 2024-03-14T10:00:59.738 INFO:tasks.workunit:Stopping ['suites/fsx.sh'] on client.0... 2024-03-14T10:00:59.738 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-14T10:01:00.790 DEBUG:teuthology.parallel:result is None 2024-03-14T10:01:00.791 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-14T10:01:00.865 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-14T10:01:00.865 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-14T10:01:00.876 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-14T10:01:00.876 INFO:teuthology.task.exec:Executing custom commands... 2024-03-14T10:01:00.876 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2024-03-14T10:01:00.876 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-14T10:01:00.944 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-14T10:01:01.373 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:01 smithi132 ceph-mon[47959]: pgmap v208: 129 pgs: 129 active+clean; 261 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.9 MiB/s wr, 280 op/s 2024-03-14T10:01:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:01 smithi152 ceph-mon[48279]: pgmap v208: 129 pgs: 129 active+clean; 261 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.9 MiB/s wr, 280 op/s 2024-03-14T10:01:01.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:01 smithi078 ceph-mon[38143]: pgmap v208: 129 pgs: 129 active+clean; 261 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.9 MiB/s wr, 280 op/s 2024-03-14T10:01:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:03 smithi152 ceph-mon[48279]: pgmap v209: 129 pgs: 129 active+clean; 270 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.7 MiB/s wr, 451 op/s 2024-03-14T10:01:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:03 smithi132 ceph-mon[47959]: pgmap v209: 129 pgs: 129 active+clean; 270 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.7 MiB/s wr, 451 op/s 2024-03-14T10:01:03.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:03 smithi078 ceph-mon[38143]: pgmap v209: 129 pgs: 129 active+clean; 270 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.7 MiB/s wr, 451 op/s 2024-03-14T10:01:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:05 smithi152 ceph-mon[48279]: pgmap v210: 129 pgs: 129 active+clean; 270 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 316 op/s 2024-03-14T10:01:05.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:05 smithi132 ceph-mon[47959]: pgmap v210: 129 pgs: 129 active+clean; 270 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 316 op/s 2024-03-14T10:01:05.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:05 smithi078 ceph-mon[38143]: pgmap v210: 129 pgs: 129 active+clean; 270 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 316 op/s 2024-03-14T10:01:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:07 smithi152 ceph-mon[48279]: pgmap v211: 129 pgs: 129 active+clean; 271 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 323 op/s 2024-03-14T10:01:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:07 smithi132 ceph-mon[47959]: pgmap v211: 129 pgs: 129 active+clean; 271 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 323 op/s 2024-03-14T10:01:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:07 smithi078 ceph-mon[38143]: pgmap v211: 129 pgs: 129 active+clean; 271 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 323 op/s 2024-03-14T10:01:09.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:08 smithi152 ceph-mon[48279]: pgmap v212: 129 pgs: 129 active+clean; 271 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 825 KiB/s wr, 178 op/s 2024-03-14T10:01:09.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:08 smithi132 ceph-mon[47959]: pgmap v212: 129 pgs: 129 active+clean; 271 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 825 KiB/s wr, 178 op/s 2024-03-14T10:01:09.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:08 smithi078 ceph-mon[38143]: pgmap v212: 129 pgs: 129 active+clean; 271 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 825 KiB/s wr, 178 op/s 2024-03-14T10:01:10.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:09 smithi152 ceph-mon[48279]: osdmap e89: 12 total, 12 up, 12 in 2024-03-14T10:01:10.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:09 smithi132 ceph-mon[47959]: osdmap e89: 12 total, 12 up, 12 in 2024-03-14T10:01:10.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:09 smithi078 ceph-mon[38143]: osdmap e89: 12 total, 12 up, 12 in 2024-03-14T10:01:11.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:10 smithi152 ceph-mon[48279]: pgmap v214: 129 pgs: 129 active+clean; 271 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 990 KiB/s wr, 214 op/s 2024-03-14T10:01:11.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:10 smithi132 ceph-mon[47959]: pgmap v214: 129 pgs: 129 active+clean; 271 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 990 KiB/s wr, 214 op/s 2024-03-14T10:01:11.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:10 smithi078 ceph-mon[38143]: pgmap v214: 129 pgs: 129 active+clean; 271 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 990 KiB/s wr, 214 op/s 2024-03-14T10:01:13.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:13 smithi152 ceph-mon[48279]: pgmap v215: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-03-14T10:01:13.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:13 smithi152 ceph-mon[48279]: osdmap e90: 12 total, 12 up, 12 in 2024-03-14T10:01:13.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:13 smithi132 ceph-mon[47959]: pgmap v215: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-03-14T10:01:13.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:13 smithi132 ceph-mon[47959]: osdmap e90: 12 total, 12 up, 12 in 2024-03-14T10:01:13.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:13 smithi078 ceph-mon[38143]: pgmap v215: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-03-14T10:01:13.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:13 smithi078 ceph-mon[38143]: osdmap e90: 12 total, 12 up, 12 in 2024-03-14T10:01:15.271 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:01:15.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.481+0000 7f474a000700 1 -- 172.21.15.78:0/3338664843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4744106040 msgr2=0x7f474410dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:15.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.481+0000 7f474a000700 1 --2- 172.21.15.78:0/3338664843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4744106040 0x7f474410dbc0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f473c00b0a0 tx=0x7f473c00b3b0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.481+0000 7f474a000700 1 -- 172.21.15.78:0/3338664843 shutdown_connections 2024-03-14T10:01:15.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.481+0000 7f474a000700 1 --2- 172.21.15.78:0/3338664843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4744106040 0x7f474410dbc0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.481+0000 7f474a000700 1 --2- 172.21.15.78:0/3338664843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4744100f20 0x7f4744101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.481+0000 7f474a000700 1 --2- 172.21.15.78:0/3338664843 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47441018d0 0x7f4744105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.481+0000 7f474a000700 1 -- 172.21.15.78:0/3338664843 >> 172.21.15.78:0/3338664843 conn(0x7f47440fc7a0 msgr2=0x7f47440febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:15.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.481+0000 7f474a000700 1 -- 172.21.15.78:0/3338664843 shutdown_connections 2024-03-14T10:01:15.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.481+0000 7f474a000700 1 -- 172.21.15.78:0/3338664843 wait complete. 2024-03-14T10:01:15.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.482+0000 7f474a000700 1 Processor -- start 2024-03-14T10:01:15.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.482+0000 7f474a000700 1 -- start start 2024-03-14T10:01:15.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f474a000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4744100f20 0x7f47440730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f474a000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4744106040 0x7f47440735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f474a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4744073b30 0x7f47441a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f474a000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47441a5ee0 con 0x7f4744073b30 2024-03-14T10:01:15.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f474a000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47441a6020 con 0x7f4744100f20 2024-03-14T10:01:15.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f474a000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47441a61e0 con 0x7f4744106040 2024-03-14T10:01:15.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f4743400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4744100f20 0x7f47440730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f4743e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4744073b30 0x7f47441a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f4742a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4744106040 0x7f47440735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f4743400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4744100f20 0x7f47440730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:58202/0 (socket says 172.21.15.78:58202) 2024-03-14T10:01:15.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f4743400700 1 -- 172.21.15.78:0/4121396346 learned_addr learned my addr 172.21.15.78:0/4121396346 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:15.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f4743400700 1 -- 172.21.15.78:0/4121396346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4744106040 msgr2=0x7f47440735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:15.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f4743400700 1 --2- 172.21.15.78:0/4121396346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4744106040 0x7f47440735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f4743400700 1 -- 172.21.15.78:0/4121396346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4744073b30 msgr2=0x7f47441a5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:15.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f4743400700 1 --2- 172.21.15.78:0/4121396346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4744073b30 0x7f47441a5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f4743400700 1 -- 172.21.15.78:0/4121396346 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473c009d00 con 0x7f4744100f20 2024-03-14T10:01:15.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.483+0000 7f4743e00700 1 --2- 172.21.15.78:0/4121396346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4744073b30 0x7f47441a5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:15.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.484+0000 7f4743400700 1 --2- 172.21.15.78:0/4121396346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4744100f20 0x7f47440730b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f473400d950 tx=0x7f473400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:15.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.484+0000 7f473be00700 1 -- 172.21.15.78:0/4121396346 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473400f890 con 0x7f4744100f20 2024-03-14T10:01:15.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.484+0000 7f474a000700 1 -- 172.21.15.78:0/4121396346 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47441a64c0 con 0x7f4744100f20 2024-03-14T10:01:15.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.484+0000 7f473be00700 1 -- 172.21.15.78:0/4121396346 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4734010460 con 0x7f4744100f20 2024-03-14T10:01:15.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.484+0000 7f473be00700 1 -- 172.21.15.78:0/4121396346 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473400e610 con 0x7f4744100f20 2024-03-14T10:01:15.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.484+0000 7f474a000700 1 -- 172.21.15.78:0/4121396346 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47441a69d0 con 0x7f4744100f20 2024-03-14T10:01:15.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.489+0000 7f473be00700 1 -- 172.21.15.78:0/4121396346 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f47340108d0 con 0x7f4744100f20 2024-03-14T10:01:15.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.489+0000 7f475daeab80 1 Processor -- start 2024-03-14T10:01:15.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.489+0000 7f475daeab80 1 -- start start 2024-03-14T10:01:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f475daeab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69680 0x563927a69a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f475daeab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a69fa0 0x563927a73980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f475daeab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a73ec0 0x563927a762b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f475daeab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639279c2990 con 0x563927a73ec0 2024-03-14T10:01:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f475daeab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639279c0350 con 0x563927a69680 2024-03-14T10:01:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f475daeab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639279bbc40 con 0x563927a69fa0 2024-03-14T10:01:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f4742a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a73ec0 0x563927a762b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f4743400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a69fa0 0x563927a73980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f4743e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69680 0x563927a69a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f4742a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a73ec0 0x563927a762b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41240/0 (socket says 172.21.15.78:41240) 2024-03-14T10:01:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f4742a00700 1 -- 172.21.15.78:0/108488678 learned_addr learned my addr 172.21.15.78:0/108488678 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f4743400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a69fa0 0x563927a73980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:46090/0 (socket says 172.21.15.78:46090) 2024-03-14T10:01:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f473be00700 1 --2- 172.21.15.78:0/4121396346 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f47280a0c00 0x7f47280a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f4742a00700 1 -- 172.21.15.78:0/108488678 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a69fa0 msgr2=0x563927a73980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f4742a00700 1 --2- 172.21.15.78:0/108488678 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a69fa0 0x563927a73980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f4742a00700 1 -- 172.21.15.78:0/108488678 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69680 msgr2=0x563927a69a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f4742a00700 1 --2- 172.21.15.78:0/108488678 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69680 0x563927a69a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f4742a00700 1 -- 172.21.15.78:0/108488678 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56392783d3b0 con 0x563927a73ec0 2024-03-14T10:01:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f473be00700 1 -- 172.21.15.78:0/4121396346 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f47340cc630 con 0x7f4744100f20 2024-03-14T10:01:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.490+0000 7f4743e00700 1 --2- 172.21.15.78:0/108488678 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69680 0x563927a69a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:01:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.491+0000 7f4743e00700 1 --2- 172.21.15.78:0/4121396346 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f47280a0c00 0x7f47280a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.491+0000 7f4742a00700 1 --2- 172.21.15.78:0/108488678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a73ec0 0x563927a762b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4730009cb0 tx=0x7f4730011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e835c626a4179b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.491+0000 7f472fe00700 1 -- 172.21.15.78:0/108488678 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730011d20 con 0x563927a73ec0 2024-03-14T10:01:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.491+0000 7f472fe00700 1 -- 172.21.15.78:0/108488678 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4730011e80 con 0x563927a73ec0 2024-03-14T10:01:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.491+0000 7f472fe00700 1 -- 172.21.15.78:0/108488678 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730005280 con 0x563927a73ec0 2024-03-14T10:01:15.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.492+0000 7f475daeab80 1 -- 172.21.15.78:0/108488678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a73ec0 msgr2=0x563927a762b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:15.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.492+0000 7f4743e00700 1 --2- 172.21.15.78:0/4121396346 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f47280a0c00 0x7f47280a30c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f473c00b780 tx=0x7f473c0099e0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:15.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.492+0000 7f475daeab80 1 --2- 172.21.15.78:0/108488678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a73ec0 0x563927a762b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4730009cb0 tx=0x7f4730011440 comp rx=0 tx=0).stop 2024-03-14T10:01:15.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.492+0000 7f475daeab80 1 -- 172.21.15.78:0/108488678 shutdown_connections 2024-03-14T10:01:15.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.492+0000 7f475daeab80 1 --2- 172.21.15.78:0/108488678 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69680 0x563927a69a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.492+0000 7f475daeab80 1 --2- 172.21.15.78:0/108488678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a73ec0 0x563927a762b0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.492+0000 7f475daeab80 1 --2- 172.21.15.78:0/108488678 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a69fa0 0x563927a73980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.492+0000 7f475daeab80 1 -- 172.21.15.78:0/108488678 >> 172.21.15.78:0/108488678 conn(0x5639279df220 msgr2=0x563927a64aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:15.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.492+0000 7f475daeab80 1 -- 172.21.15.78:0/108488678 shutdown_connections 2024-03-14T10:01:15.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.492+0000 7f475daeab80 1 -- 172.21.15.78:0/108488678 wait complete. 2024-03-14T10:01:15.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.492+0000 7f475daeab80 1 -- start start 2024-03-14T10:01:15.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.493+0000 7f475daeab80 1 Processor -- start 2024-03-14T10:01:15.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.493+0000 7f475daeab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a69680 0x563927a73820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.493+0000 7f475daeab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69fa0 0x563927a70b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.493+0000 7f475daeab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a73ec0 0x563927a710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.493+0000 7f475daeab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563927957d80 con 0x563927a73ec0 2024-03-14T10:01:15.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.493+0000 7f475daeab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563927957a20 con 0x563927a69fa0 2024-03-14T10:01:15.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.493+0000 7f475daeab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56392783f6a0 con 0x563927a69680 2024-03-14T10:01:15.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.493+0000 7f475daeab80 10 client.0 _ll_register_callbacks cb 0x5639277c33a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:15.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.493+0000 7f4743e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a73ec0 0x563927a710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.494+0000 7f4742a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69fa0 0x563927a70b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.494+0000 7f4743e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a73ec0 0x563927a710f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41242/0 (socket says 172.21.15.78:41242) 2024-03-14T10:01:15.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.494+0000 7f4743e00700 1 -- 172.21.15.78:0/4115069809 learned_addr learned my addr 172.21.15.78:0/4115069809 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:15.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.494+0000 7f4743400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a69680 0x563927a73820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.494+0000 7f4742a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69fa0 0x563927a70b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:58226/0 (socket says 172.21.15.78:58226) 2024-03-14T10:01:15.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.494+0000 7f4743e00700 1 -- 172.21.15.78:0/4115069809 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a69680 msgr2=0x563927a73820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:15.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.494+0000 7f4743e00700 1 --2- 172.21.15.78:0/4115069809 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a69680 0x563927a73820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.494+0000 7f4743e00700 1 -- 172.21.15.78:0/4115069809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69fa0 msgr2=0x563927a70b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:15.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.494+0000 7f4743e00700 1 --2- 172.21.15.78:0/4115069809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69fa0 0x563927a70b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.494+0000 7f4743e00700 1 -- 172.21.15.78:0/4115069809 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47300098e0 con 0x563927a73ec0 2024-03-14T10:01:15.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.494+0000 7f4743e00700 1 --2- 172.21.15.78:0/4115069809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a73ec0 0x563927a710f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f473c015000 tx=0x7f473c00b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3208183d1abeef40 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:15.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.494+0000 7f472cc00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:15.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.494+0000 7f472cc00700 1 -- 172.21.15.78:0/4115069809 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c00e040 con 0x563927a73ec0 2024-03-14T10:01:15.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.494+0000 7f475daeab80 1 -- 172.21.15.78:0/4115069809 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5639279b4700 con 0x563927a73ec0 2024-03-14T10:01:15.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.494+0000 7f472cc00700 1 -- 172.21.15.78:0/4115069809 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f473c022040 con 0x563927a73ec0 2024-03-14T10:01:15.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.495+0000 7f472cc00700 1 -- 172.21.15.78:0/4115069809 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f473c005240 con 0x563927a73ec0 2024-03-14T10:01:15.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.495+0000 7f475daeab80 10 client.16956 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:15.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.495+0000 7f475daeab80 1 -- 172.21.15.78:0/4115069809 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5639277e9b60 con 0x563927a73ec0 2024-03-14T10:01:15.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.495+0000 7f472cc00700 1 -- 172.21.15.78:0/4115069809 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5639279b4700 con 0x563927a73ec0 2024-03-14T10:01:15.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.495+0000 7f472cc00700 1 -- 172.21.15.78:0/4115069809 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f473c029720 con 0x563927a73ec0 2024-03-14T10:01:15.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.495+0000 7f475daeab80 10 client.16956 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:15.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.495+0000 7f475daeab80 10 client.16956 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:01:15.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.495+0000 7f475daeab80 20 client.16956 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:15.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.495+0000 7f475daeab80 1 --2- 172.21.15.78:0/4115069809 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563927a83af0 0x563927a85f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.495+0000 7f475daeab80 4 client.16956 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:01:15.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.495+0000 7f475daeab80 1 -- 172.21.15.78:0/4115069809 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56392780acf0 con 0x563927a83af0 2024-03-14T10:01:15.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.496+0000 7f4743400700 1 --2- 172.21.15.78:0/4115069809 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563927a83af0 0x563927a85f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.496+0000 7f4743400700 1 --2- 172.21.15.78:0/4115069809 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563927a83af0 0x563927a85f10 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2179b215734edeb4 server_cookie=8ebbc13756279e7f in_seq=0 out_seq=0 2024-03-14T10:01:15.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.496+0000 7f472cc00700 10 client.16956 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:15.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.497+0000 7f472cc00700 1 -- 172.21.15.78:0/4115069809 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56392780acf0 con 0x563927a83af0 2024-03-14T10:01:15.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.497+0000 7f472cc00700 10 client.16956 handle_command_reply: tid=0 2024-03-14T10:01:15.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.497+0000 7f475daeab80 1 client.16956 shutdown 2024-03-14T10:01:15.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.497+0000 7f475daeab80 1 -- 172.21.15.78:0/4115069809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a73ec0 msgr2=0x563927a710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:15.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.497+0000 7f475daeab80 1 --2- 172.21.15.78:0/4115069809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a73ec0 0x563927a710f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f473c015000 tx=0x7f473c00b3b0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.497+0000 7f475daeab80 1 -- 172.21.15.78:0/4115069809 shutdown_connections 2024-03-14T10:01:15.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.497+0000 7f475daeab80 1 --2- 172.21.15.78:0/4115069809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a73ec0 0x563927a710f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.497+0000 7f475daeab80 1 --2- 172.21.15.78:0/4115069809 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a69680 0x563927a73820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.497+0000 7f475daeab80 1 --2- 172.21.15.78:0/4115069809 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563927a83af0 0x563927a85f10 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.497+0000 7f475daeab80 1 --2- 172.21.15.78:0/4115069809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69fa0 0x563927a70b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.497+0000 7f472cc00700 0 client.16956 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:15.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.497+0000 7f475daeab80 1 -- 172.21.15.78:0/4115069809 >> 172.21.15.78:0/4115069809 conn(0x5639279df220 msgr2=0x563927a64690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:15.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.497+0000 7f475daeab80 1 -- 172.21.15.78:0/4115069809 shutdown_connections 2024-03-14T10:01:15.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.497+0000 7f475daeab80 1 -- 172.21.15.78:0/4115069809 wait complete. 2024-03-14T10:01:15.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.497+0000 7f475daeab80 20 client.16956 trim_cache size 0 max 16384 2024-03-14T10:01:15.501 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:15 smithi152 ceph-mon[48279]: pgmap v217: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 40 KiB/s wr, 12 op/s 2024-03-14T10:01:15.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.514+0000 7f475daeab80 1 Processor -- start 2024-03-14T10:01:15.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.514+0000 7f475daeab80 1 -- start start 2024-03-14T10:01:15.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.514+0000 7f475daeab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a69680 0x563927a71080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.514+0000 7f475daeab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a69fa0 0x563927a6ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.514+0000 7f475daeab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a73ec0 0x563927a6e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.514+0000 7f475daeab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563927a80610 con 0x563927a69680 2024-03-14T10:01:15.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.514+0000 7f475daeab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563927a63830 con 0x563927a73ec0 2024-03-14T10:01:15.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.514+0000 7f475daeab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563927a65e80 con 0x563927a69fa0 2024-03-14T10:01:15.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.515+0000 7f4743400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a69680 0x563927a71080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.515+0000 7f4743400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a69680 0x563927a71080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41250/0 (socket says 172.21.15.78:41250) 2024-03-14T10:01:15.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.515+0000 7f4743400700 1 -- 172.21.15.78:0/1140123578 learned_addr learned my addr 172.21.15.78:0/1140123578 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:15.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.515+0000 7f4742a00700 1 --2- 172.21.15.78:0/1140123578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a69fa0 0x563927a6ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.515+0000 7f4743e00700 1 --2- 172.21.15.78:0/1140123578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a73ec0 0x563927a6e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.515+0000 7f4743400700 1 -- 172.21.15.78:0/1140123578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a69fa0 msgr2=0x563927a6ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:15.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.515+0000 7f4743400700 1 --2- 172.21.15.78:0/1140123578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a69fa0 0x563927a6ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.515+0000 7f4743400700 1 -- 172.21.15.78:0/1140123578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a73ec0 msgr2=0x563927a6e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:15.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.515+0000 7f4743400700 1 --2- 172.21.15.78:0/1140123578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a73ec0 0x563927a6e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.515+0000 7f4743400700 1 -- 172.21.15.78:0/1140123578 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563927981ea0 con 0x563927a69680 2024-03-14T10:01:15.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.515+0000 7f4743400700 1 --2- 172.21.15.78:0/1140123578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a69680 0x563927a71080 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4734095a50 tx=0x7f4734084040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c4f135c65ed33e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:15.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.515+0000 7f4739200700 1 -- 172.21.15.78:0/1140123578 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734099090 con 0x563927a69680 2024-03-14T10:01:15.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.515+0000 7f4739200700 1 -- 172.21.15.78:0/1140123578 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4734098eb0 con 0x563927a69680 2024-03-14T10:01:15.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.515+0000 7f4739200700 1 -- 172.21.15.78:0/1140123578 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473409d020 con 0x563927a69680 2024-03-14T10:01:15.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.516+0000 7f475daeab80 1 -- 172.21.15.78:0/1140123578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a69680 msgr2=0x563927a71080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:15.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.516+0000 7f475daeab80 1 --2- 172.21.15.78:0/1140123578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a69680 0x563927a71080 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4734095a50 tx=0x7f4734084040 comp rx=0 tx=0).stop 2024-03-14T10:01:15.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.516+0000 7f475daeab80 1 -- 172.21.15.78:0/1140123578 shutdown_connections 2024-03-14T10:01:15.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.516+0000 7f475daeab80 1 --2- 172.21.15.78:0/1140123578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a73ec0 0x563927a6e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.516+0000 7f475daeab80 1 --2- 172.21.15.78:0/1140123578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a69680 0x563927a71080 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.516+0000 7f475daeab80 1 --2- 172.21.15.78:0/1140123578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a69fa0 0x563927a6ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.516+0000 7f475daeab80 1 -- 172.21.15.78:0/1140123578 >> 172.21.15.78:0/1140123578 conn(0x5639279df220 msgr2=0x563927a6f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:15.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.516+0000 7f475daeab80 1 -- 172.21.15.78:0/1140123578 shutdown_connections 2024-03-14T10:01:15.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.516+0000 7f475daeab80 1 -- 172.21.15.78:0/1140123578 wait complete. 2024-03-14T10:01:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.516+0000 7f475daeab80 1 -- start start 2024-03-14T10:01:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.517+0000 7f475daeab80 1 Processor -- start 2024-03-14T10:01:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.517+0000 7f475daeab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69680 0x563927a7a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.517+0000 7f475daeab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a69fa0 0x563927a78b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.517+0000 7f475daeab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a73ec0 0x563927a79070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.517+0000 7f475daeab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563927a7af70 con 0x563927a69fa0 2024-03-14T10:01:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.517+0000 7f475daeab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563927a795b0 con 0x563927a69680 2024-03-14T10:01:15.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.517+0000 7f475daeab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563927a79770 con 0x563927a73ec0 2024-03-14T10:01:15.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.517+0000 7f475daeab80 10 client.0 _ll_register_callbacks cb 0x7f473c007f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:15.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.517+0000 7f4742a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a69fa0 0x563927a78b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.517+0000 7f4743e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a73ec0 0x563927a79070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.517+0000 7f4743400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69680 0x563927a7a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.517+0000 7f4742a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a69fa0 0x563927a78b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41252/0 (socket says 172.21.15.78:41252) 2024-03-14T10:01:15.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.517+0000 7f4742a00700 1 -- 172.21.15.78:0/493981650 learned_addr learned my addr 172.21.15.78:0/493981650 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:15.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.518+0000 7f4743e00700 1 -- 172.21.15.78:0/493981650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69680 msgr2=0x563927a7a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:15.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.518+0000 7f4743e00700 1 --2- 172.21.15.78:0/493981650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69680 0x563927a7a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.518+0000 7f4743e00700 1 -- 172.21.15.78:0/493981650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a69fa0 msgr2=0x563927a78b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:15.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.518+0000 7f4743e00700 1 --2- 172.21.15.78:0/493981650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a69fa0 0x563927a78b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.518+0000 7f4743e00700 1 -- 172.21.15.78:0/493981650 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47441a69d0 con 0x563927a73ec0 2024-03-14T10:01:15.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.518+0000 7f4742a00700 1 --2- 172.21.15.78:0/493981650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a69fa0 0x563927a78b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:15.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.518+0000 7f4743e00700 1 --2- 172.21.15.78:0/493981650 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a73ec0 0x563927a79070 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f473c02c040 tx=0x7f473c01d6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df602f9bca2e0cb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.518+0000 7f472cc00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.518+0000 7f472cc00700 1 -- 172.21.15.78:0/493981650 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c00e040 con 0x563927a73ec0 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.519+0000 7f475daeab80 1 -- 172.21.15.78:0/493981650 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563927a5c720 con 0x563927a73ec0 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.519+0000 7f472cc00700 1 -- 172.21.15.78:0/493981650 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f473c029830 con 0x563927a73ec0 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.519+0000 7f472cc00700 1 -- 172.21.15.78:0/493981650 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f473c007ac0 con 0x563927a73ec0 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.519+0000 7f475daeab80 10 client.27016 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.519+0000 7f475daeab80 1 -- 172.21.15.78:0/493981650 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563927841cd0 con 0x563927a73ec0 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.519+0000 7f472cc00700 1 -- 172.21.15.78:0/493981650 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563927a5c720 con 0x563927a73ec0 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.519+0000 7f472cc00700 1 -- 172.21.15.78:0/493981650 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f473c019030 con 0x563927a73ec0 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.519+0000 7f475daeab80 10 client.27016 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.519+0000 7f475daeab80 10 client.27016 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.519+0000 7f475daeab80 20 client.27016 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.519+0000 7f475daeab80 1 --2- 172.21.15.78:0/493981650 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563927ad6670 0x563927ad8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.519+0000 7f475daeab80 4 client.27016 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.519+0000 7f475daeab80 1 -- 172.21.15.78:0/493981650 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56392783fce0 con 0x563927ad6670 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.520+0000 7f4743400700 1 --2- 172.21.15.78:0/493981650 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563927ad6670 0x563927ad8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.520+0000 7f4743400700 1 --2- 172.21.15.78:0/493981650 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563927ad6670 0x563927ad8a90 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d5c7cacd088f0c7 server_cookie=d00111162c109835 in_seq=0 out_seq=0 2024-03-14T10:01:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.520+0000 7f472cc00700 10 client.27016 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.520+0000 7f472cc00700 1 -- 172.21.15.78:0/493981650 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56392783fce0 con 0x563927ad6670 2024-03-14T10:01:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.520+0000 7f472cc00700 10 client.27016 handle_command_reply: tid=0 2024-03-14T10:01:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.520+0000 7f475daeab80 1 client.27016 shutdown 2024-03-14T10:01:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.521+0000 7f475daeab80 1 -- 172.21.15.78:0/493981650 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a73ec0 msgr2=0x563927a79070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.521+0000 7f475daeab80 1 --2- 172.21.15.78:0/493981650 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a73ec0 0x563927a79070 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f473c02c040 tx=0x7f473c01d6d0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.521+0000 7f475daeab80 1 -- 172.21.15.78:0/493981650 shutdown_connections 2024-03-14T10:01:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.521+0000 7f475daeab80 1 --2- 172.21.15.78:0/493981650 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563927ad6670 0x563927ad8a90 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.521+0000 7f475daeab80 1 --2- 172.21.15.78:0/493981650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563927a69680 0x563927a7a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.521+0000 7f475daeab80 1 --2- 172.21.15.78:0/493981650 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563927a73ec0 0x563927a79070 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.521+0000 7f472cc00700 0 client.27016 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.521+0000 7f475daeab80 1 --2- 172.21.15.78:0/493981650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563927a69fa0 0x563927a78b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.521+0000 7f475daeab80 1 -- 172.21.15.78:0/493981650 >> 172.21.15.78:0/493981650 conn(0x5639279df220 msgr2=0x563927a6f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.521+0000 7f475daeab80 1 -- 172.21.15.78:0/493981650 shutdown_connections 2024-03-14T10:01:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.521+0000 7f475daeab80 1 -- 172.21.15.78:0/493981650 wait complete. 2024-03-14T10:01:15.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.521+0000 7f475daeab80 20 client.27016 trim_cache size 0 max 16384 2024-03-14T10:01:15.525 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:01:15.525 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T10:01:15.525 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T10:01:15.525 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:01:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.522+0000 7f474a000700 1 -- 172.21.15.78:0/4121396346 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f47280a0c00 msgr2=0x7f47280a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.522+0000 7f474a000700 1 --2- 172.21.15.78:0/4121396346 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f47280a0c00 0x7f47280a30c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f473c00b780 tx=0x7f473c0099e0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.522+0000 7f474a000700 1 -- 172.21.15.78:0/4121396346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4744100f20 msgr2=0x7f47440730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.522+0000 7f474a000700 1 --2- 172.21.15.78:0/4121396346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4744100f20 0x7f47440730b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f473400d950 tx=0x7f473400dc60 comp rx=0 tx=0).stop 2024-03-14T10:01:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.522+0000 7f474a000700 1 -- 172.21.15.78:0/4121396346 shutdown_connections 2024-03-14T10:01:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.522+0000 7f474a000700 1 --2- 172.21.15.78:0/4121396346 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f47280a0c00 0x7f47280a30c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.522+0000 7f474a000700 1 --2- 172.21.15.78:0/4121396346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4744100f20 0x7f47440730b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.522+0000 7f474a000700 1 --2- 172.21.15.78:0/4121396346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4744073b30 0x7f47441a5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.522+0000 7f474a000700 1 --2- 172.21.15.78:0/4121396346 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4744106040 0x7f47440735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.522+0000 7f474a000700 1 -- 172.21.15.78:0/4121396346 >> 172.21.15.78:0/4121396346 conn(0x7f47440fc7a0 msgr2=0x7f47440fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.522+0000 7f474a000700 1 -- 172.21.15.78:0/4121396346 shutdown_connections 2024-03-14T10:01:15.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.522+0000 7f474a000700 1 -- 172.21.15.78:0/4121396346 wait complete. 2024-03-14T10:01:15.527 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:15 smithi132 ceph-mon[47959]: pgmap v217: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 40 KiB/s wr, 12 op/s 2024-03-14T10:01:15.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:15 smithi078 ceph-mon[38143]: pgmap v217: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 40 KiB/s wr, 12 op/s 2024-03-14T10:01:15.540 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T10:01:15.540 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T10:01:15.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.734+0000 7fd009e00700 1 -- 172.21.15.78:0/2200471415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd004100e30 msgr2=0x7fd004101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:15.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.734+0000 7fd009e00700 1 --2- 172.21.15.78:0/2200471415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd004100e30 0x7fd004101210 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fcff4009990 tx=0x7fcff4009ca0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.734+0000 7fd009e00700 1 -- 172.21.15.78:0/2200471415 shutdown_connections 2024-03-14T10:01:15.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.734+0000 7fd009e00700 1 --2- 172.21.15.78:0/2200471415 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd004105f50 0x7fd00410dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.734+0000 7fd009e00700 1 --2- 172.21.15.78:0/2200471415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd004100e30 0x7fd004101210 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.734+0000 7fd009e00700 1 --2- 172.21.15.78:0/2200471415 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0041017e0 0x7fd004105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.734+0000 7fd009e00700 1 -- 172.21.15.78:0/2200471415 >> 172.21.15.78:0/2200471415 conn(0x7fd0040fc730 msgr2=0x7fd0040feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:15.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.734+0000 7fd009e00700 1 -- 172.21.15.78:0/2200471415 shutdown_connections 2024-03-14T10:01:15.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.734+0000 7fd009e00700 1 -- 172.21.15.78:0/2200471415 wait complete. 2024-03-14T10:01:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.735+0000 7fd009e00700 1 Processor -- start 2024-03-14T10:01:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.735+0000 7fd009e00700 1 -- start start 2024-03-14T10:01:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.736+0000 7fd009e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd004100e30 0x7fd004197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.736+0000 7fd009e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd004105f50 0x7fd004197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.736+0000 7fd009e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd00419bea0 0x7fd00419c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.736+0000 7fd009e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00419c920 con 0x7fd004105f50 2024-03-14T10:01:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.736+0000 7fd009e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00419ca90 con 0x7fd004100e30 2024-03-14T10:01:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.736+0000 7fd009e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00419cc50 con 0x7fd00419bea0 2024-03-14T10:01:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.736+0000 7fd002000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd004105f50 0x7fd004197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.736+0000 7fd002000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd004105f50 0x7fd004197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41268/0 (socket says 172.21.15.78:41268) 2024-03-14T10:01:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.736+0000 7fd002000700 1 -- 172.21.15.78:0/2004807332 learned_addr learned my addr 172.21.15.78:0/2004807332 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.736+0000 7fd002a00700 1 --2- 172.21.15.78:0/2004807332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd004100e30 0x7fd004197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.736+0000 7fd003400700 1 --2- 172.21.15.78:0/2004807332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd00419bea0 0x7fd00419c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.737+0000 7fd002000700 1 -- 172.21.15.78:0/2004807332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd00419bea0 msgr2=0x7fd00419c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.737+0000 7fd002000700 1 --2- 172.21.15.78:0/2004807332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd00419bea0 0x7fd00419c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.737+0000 7fd002000700 1 -- 172.21.15.78:0/2004807332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd004100e30 msgr2=0x7fd004197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.737+0000 7fd002000700 1 --2- 172.21.15.78:0/2004807332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd004100e30 0x7fd004197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.737+0000 7fd002000700 1 -- 172.21.15.78:0/2004807332 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff40095f0 con 0x7fd004105f50 2024-03-14T10:01:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.737+0000 7fd002000700 1 --2- 172.21.15.78:0/2004807332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd004105f50 0x7fd004197b50 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcff000d950 tx=0x7fcff000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.737+0000 7fd008a00700 1 -- 172.21.15.78:0/2004807332 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0004220 con 0x7fd004105f50 2024-03-14T10:01:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.737+0000 7fd008a00700 1 -- 172.21.15.78:0/2004807332 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcff0007620 con 0x7fd004105f50 2024-03-14T10:01:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.737+0000 7fd009e00700 1 -- 172.21.15.78:0/2004807332 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd004073040 con 0x7fd004105f50 2024-03-14T10:01:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.737+0000 7fd008a00700 1 -- 172.21.15.78:0/2004807332 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff00056c0 con 0x7fd004105f50 2024-03-14T10:01:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.737+0000 7fd009e00700 1 -- 172.21.15.78:0/2004807332 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd004073550 con 0x7fd004105f50 2024-03-14T10:01:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.738+0000 7fd009e00700 1 -- 172.21.15.78:0/2004807332 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0041091c0 con 0x7fd004105f50 2024-03-14T10:01:15.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.740+0000 7fd008a00700 1 -- 172.21.15.78:0/2004807332 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fcff0004380 con 0x7fd004105f50 2024-03-14T10:01:15.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.741+0000 7fd008a00700 1 --2- 172.21.15.78:0/2004807332 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcfe80a0c00 0x7fcfe80a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:15.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.742+0000 7fd008a00700 1 -- 172.21.15.78:0/2004807332 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fcff00cc560 con 0x7fd004105f50 2024-03-14T10:01:15.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.742+0000 7fd002a00700 1 --2- 172.21.15.78:0/2004807332 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcfe80a0c00 0x7fcfe80a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:15.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.742+0000 7fd002a00700 1 --2- 172.21.15.78:0/2004807332 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcfe80a0c00 0x7fcfe80a30c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcff400b2e0 tx=0x7fcff4005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:15.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.746+0000 7fd008a00700 1 -- 172.21.15.78:0/2004807332 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcff00957c0 con 0x7fd004105f50 2024-03-14T10:01:15.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.998+0000 7fd009e00700 1 -- 172.21.15.78:0/2004807332 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0041988c0 con 0x7fd004105f50 2024-03-14T10:01:15.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:15.998+0000 7fd008a00700 1 -- 172.21.15.78:0/2004807332 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7fcff0098de0 con 0x7fd004105f50 2024-03-14T10:01:15.998 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T10:01:15.998 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T10:01:16.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.001+0000 7fd009e00700 1 -- 172.21.15.78:0/2004807332 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcfe80a0c00 msgr2=0x7fcfe80a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.001+0000 7fd009e00700 1 --2- 172.21.15.78:0/2004807332 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcfe80a0c00 0x7fcfe80a30c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcff400b2e0 tx=0x7fcff4005fd0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.001+0000 7fd009e00700 1 -- 172.21.15.78:0/2004807332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd004105f50 msgr2=0x7fd004197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.001+0000 7fd009e00700 1 --2- 172.21.15.78:0/2004807332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd004105f50 0x7fd004197b50 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcff000d950 tx=0x7fcff000dc60 comp rx=0 tx=0).stop 2024-03-14T10:01:16.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.001+0000 7fd009e00700 1 -- 172.21.15.78:0/2004807332 shutdown_connections 2024-03-14T10:01:16.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.001+0000 7fd009e00700 1 --2- 172.21.15.78:0/2004807332 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fcfe80a0c00 0x7fcfe80a30c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.001+0000 7fd009e00700 1 --2- 172.21.15.78:0/2004807332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd004100e30 0x7fd004197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.001+0000 7fd009e00700 1 --2- 172.21.15.78:0/2004807332 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fd00419bea0 0x7fd00419c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.001+0000 7fd009e00700 1 --2- 172.21.15.78:0/2004807332 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd004105f50 0x7fd004197b50 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.001+0000 7fd009e00700 1 -- 172.21.15.78:0/2004807332 >> 172.21.15.78:0/2004807332 conn(0x7fd0040fc730 msgr2=0x7fd0040fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.001+0000 7fd009e00700 1 -- 172.21.15.78:0/2004807332 shutdown_connections 2024-03-14T10:01:16.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.001+0000 7fd009e00700 1 -- 172.21.15.78:0/2004807332 wait complete. 2024-03-14T10:01:16.002 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T10:01:16.021 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T10:01:16.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.217+0000 7f6ca4200700 1 -- 172.21.15.78:0/2343123424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c9c1018e0 msgr2=0x7f6c9c105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.217+0000 7f6ca4200700 1 --2- 172.21.15.78:0/2343123424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c9c1018e0 0x7f6c9c105920 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009990 tx=0x7f6c88009ca0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.217+0000 7f6ca4200700 1 -- 172.21.15.78:0/2343123424 shutdown_connections 2024-03-14T10:01:16.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.217+0000 7f6ca4200700 1 --2- 172.21.15.78:0/2343123424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c9c106050 0x7f6c9c10dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.217+0000 7f6ca4200700 1 --2- 172.21.15.78:0/2343123424 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c9c100f30 0x7f6c9c101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.217+0000 7f6ca4200700 1 --2- 172.21.15.78:0/2343123424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c9c1018e0 0x7f6c9c105920 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.217+0000 7f6ca4200700 1 -- 172.21.15.78:0/2343123424 >> 172.21.15.78:0/2343123424 conn(0x7f6c9c0fc7b0 msgr2=0x7f6c9c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.217+0000 7f6ca4200700 1 -- 172.21.15.78:0/2343123424 shutdown_connections 2024-03-14T10:01:16.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.217+0000 7f6ca4200700 1 -- 172.21.15.78:0/2343123424 wait complete. 2024-03-14T10:01:16.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.218+0000 7f6ca4200700 1 Processor -- start 2024-03-14T10:01:16.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.218+0000 7f6ca4200700 1 -- start start 2024-03-14T10:01:16.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.218+0000 7f6ca4200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c9c100f30 0x7f6c9c19f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.218+0000 7f6ca4200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c9c106050 0x7f6c9c19fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.218+0000 7f6ca4200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c9c199660 0x7f6c9c199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.219+0000 7f6ca4200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c19a0e0 con 0x7f6c9c100f30 2024-03-14T10:01:16.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.219+0000 7f6ca4200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c19a250 con 0x7f6c9c106050 2024-03-14T10:01:16.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.219+0000 7f6ca4200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c19a410 con 0x7f6c9c199660 2024-03-14T10:01:16.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.219+0000 7f6ca1800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c9c100f30 0x7f6c9c19f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.219+0000 7f6ca1800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c9c100f30 0x7f6c9c19f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41288/0 (socket says 172.21.15.78:41288) 2024-03-14T10:01:16.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.219+0000 7f6ca1800700 1 -- 172.21.15.78:0/2530157145 learned_addr learned my addr 172.21.15.78:0/2530157145 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:16.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.219+0000 7f6ca2200700 1 --2- 172.21.15.78:0/2530157145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c9c199660 0x7f6c9c199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.219+0000 7f6ca0e00700 1 --2- 172.21.15.78:0/2530157145 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c9c106050 0x7f6c9c19fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.219+0000 7f6ca1800700 1 -- 172.21.15.78:0/2530157145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c9c199660 msgr2=0x7f6c9c199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.219+0000 7f6ca1800700 1 --2- 172.21.15.78:0/2530157145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c9c199660 0x7f6c9c199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.219+0000 7f6ca1800700 1 -- 172.21.15.78:0/2530157145 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c9c106050 msgr2=0x7f6c9c19fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.219+0000 7f6ca1800700 1 --2- 172.21.15.78:0/2530157145 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c9c106050 0x7f6c9c19fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.219+0000 7f6ca1800700 1 -- 172.21.15.78:0/2530157145 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c880095f0 con 0x7f6c9c100f30 2024-03-14T10:01:16.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.219+0000 7f6ca1800700 1 --2- 172.21.15.78:0/2530157145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c9c100f30 0x7f6c9c19f530 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000e9a0 tx=0x7f6c9000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.220+0000 7f6c9a000700 1 -- 172.21.15.78:0/2530157145 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90009790 con 0x7f6c9c100f30 2024-03-14T10:01:16.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.220+0000 7f6c9a000700 1 -- 172.21.15.78:0/2530157145 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c90010430 con 0x7f6c9c100f30 2024-03-14T10:01:16.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.220+0000 7f6ca4200700 1 -- 172.21.15.78:0/2530157145 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9c1a6400 con 0x7f6c9c100f30 2024-03-14T10:01:16.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.220+0000 7f6c9a000700 1 -- 172.21.15.78:0/2530157145 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90018680 con 0x7f6c9c100f30 2024-03-14T10:01:16.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.220+0000 7f6ca4200700 1 -- 172.21.15.78:0/2530157145 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9c1a68e0 con 0x7f6c9c100f30 2024-03-14T10:01:16.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.223+0000 7f6c9a000700 1 -- 172.21.15.78:0/2530157145 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f6c900105a0 con 0x7f6c9c100f30 2024-03-14T10:01:16.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.224+0000 7f6cb7d12b80 1 Processor -- start 2024-03-14T10:01:16.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.224+0000 7f6c9a000700 1 --2- 172.21.15.78:0/2530157145 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6c840a0c00 0x7f6c840a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.224+0000 7f6c9a000700 1 -- 172.21.15.78:0/2530157145 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f6c90014070 con 0x7f6c9c100f30 2024-03-14T10:01:16.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.224+0000 7f6cb7d12b80 1 -- start start 2024-03-14T10:01:16.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.224+0000 7f6ca2200700 1 --2- 172.21.15.78:0/2530157145 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6c840a0c00 0x7f6c840a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6cb7d12b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2b5680 0x56262c2b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6cb7d12b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2b5fa0 0x56262c2bf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6cb7d12b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2bfec0 0x56262c2c22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6cb7d12b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262c20e990 con 0x56262c2bfec0 2024-03-14T10:01:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6cb7d12b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262c20c350 con 0x56262c2b5680 2024-03-14T10:01:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6cb7d12b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262c207c40 con 0x56262c2b5fa0 2024-03-14T10:01:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6ca0e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2b5fa0 0x56262c2bf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6ca1800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2b5680 0x56262c2b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6ca0e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2b5fa0 0x56262c2bf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:46154/0 (socket says 172.21.15.78:46154) 2024-03-14T10:01:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6ca0e00700 1 -- 172.21.15.78:0/4221044374 learned_addr learned my addr 172.21.15.78:0/4221044374 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6ca2200700 1 --2- 172.21.15.78:0/4221044374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2bfec0 0x56262c2c22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6ca0e00700 1 -- 172.21.15.78:0/4221044374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2b5680 msgr2=0x56262c2b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6ca0e00700 1 --2- 172.21.15.78:0/4221044374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2b5680 0x56262c2b5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6ca0e00700 1 -- 172.21.15.78:0/4221044374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2bfec0 msgr2=0x56262c2c22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6ca0e00700 1 --2- 172.21.15.78:0/4221044374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2bfec0 0x56262c2c22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6ca0e00700 1 -- 172.21.15.78:0/4221044374 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56262c0893b0 con 0x56262c2b5fa0 2024-03-14T10:01:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.225+0000 7f6ca2200700 1 --2- 172.21.15.78:0/2530157145 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6c840a0c00 0x7f6c840a30c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400f590 tx=0x7f6c94009920 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.226+0000 7f6ca0e00700 1 --2- 172.21.15.78:0/4221044374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2b5fa0 0x56262c2bf980 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009990 tx=0x7f6c88005ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db27c90e3565fafb server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.226+0000 7f6c8ea00700 1 -- 172.21.15.78:0/4221044374 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8801d070 con 0x56262c2b5fa0 2024-03-14T10:01:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.226+0000 7f6c8ea00700 1 -- 172.21.15.78:0/4221044374 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c8801e040 con 0x56262c2b5fa0 2024-03-14T10:01:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.226+0000 7f6c8ea00700 1 -- 172.21.15.78:0/4221044374 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88004740 con 0x56262c2b5fa0 2024-03-14T10:01:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.227+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/4221044374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2b5fa0 msgr2=0x56262c2bf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.227+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/4221044374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2b5fa0 0x56262c2bf980 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009990 tx=0x7f6c88005ec0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.227+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/4221044374 shutdown_connections 2024-03-14T10:01:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.227+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/4221044374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2b5680 0x56262c2b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.227+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/4221044374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2bfec0 0x56262c2c22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.227+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/4221044374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2b5fa0 0x56262c2bf980 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.227+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/4221044374 >> 172.21.15.78:0/4221044374 conn(0x56262c22b220 msgr2=0x56262c2b0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.227+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/4221044374 shutdown_connections 2024-03-14T10:01:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.227+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/4221044374 wait complete. 2024-03-14T10:01:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.227+0000 7f6cb7d12b80 1 -- start start 2024-03-14T10:01:16.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.228+0000 7f6cb7d12b80 1 Processor -- start 2024-03-14T10:01:16.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.228+0000 7f6cb7d12b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2b5680 0x56262c2bf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.228+0000 7f6cb7d12b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5fa0 0x56262c2bcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.228+0000 7f6cb7d12b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2bfec0 0x56262c2bd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.228+0000 7f6cb7d12b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262c1a3d80 con 0x56262c2b5fa0 2024-03-14T10:01:16.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.228+0000 7f6cb7d12b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262c1a3a20 con 0x56262c2bfec0 2024-03-14T10:01:16.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.228+0000 7f6cb7d12b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262c08b6a0 con 0x56262c2b5680 2024-03-14T10:01:16.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.228+0000 7f6cb7d12b80 10 client.0 _ll_register_callbacks cb 0x56262c00f3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:16.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.228+0000 7f6ca0e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5fa0 0x56262c2bcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.228+0000 7f6ca0e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5fa0 0x56262c2bcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41302/0 (socket says 172.21.15.78:41302) 2024-03-14T10:01:16.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.228+0000 7f6ca0e00700 1 -- 172.21.15.78:0/415282635 learned_addr learned my addr 172.21.15.78:0/415282635 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:16.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.229+0000 7f6ca1800700 1 --2- 172.21.15.78:0/415282635 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2b5680 0x56262c2bf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.229+0000 7f6ca2200700 1 --2- 172.21.15.78:0/415282635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2bfec0 0x56262c2bd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.229+0000 7f6ca0e00700 1 -- 172.21.15.78:0/415282635 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2b5680 msgr2=0x56262c2bf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.229+0000 7f6ca0e00700 1 --2- 172.21.15.78:0/415282635 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2b5680 0x56262c2bf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.229+0000 7f6ca0e00700 1 -- 172.21.15.78:0/415282635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2bfec0 msgr2=0x56262c2bd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.229+0000 7f6ca0e00700 1 --2- 172.21.15.78:0/415282635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2bfec0 0x56262c2bd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.229+0000 7f6ca0e00700 1 -- 172.21.15.78:0/415282635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9c10e440 con 0x56262c2b5fa0 2024-03-14T10:01:16.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.229+0000 7f6ca0e00700 1 --2- 172.21.15.78:0/415282635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5fa0 0x56262c2bcbc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800bd70 tx=0x7f6c88005cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7c39235f25d536b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.229+0000 7f6c83400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:16.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.229+0000 7f6c83400700 1 -- 172.21.15.78:0/415282635 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8801d030 con 0x56262c2b5fa0 2024-03-14T10:01:16.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.229+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/415282635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56262c2a8720 con 0x56262c2b5fa0 2024-03-14T10:01:16.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.229+0000 7f6c83400700 1 -- 172.21.15.78:0/415282635 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f6c88004500 con 0x56262c2b5fa0 2024-03-14T10:01:16.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.230+0000 7f6c83400700 1 -- 172.21.15.78:0/415282635 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c880046e0 con 0x56262c2b5fa0 2024-03-14T10:01:16.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.230+0000 7f6cb7d12b80 10 client.16980 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:16.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.230+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/415282635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56262c035b60 con 0x56262c2b5fa0 2024-03-14T10:01:16.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.230+0000 7f6c83400700 1 -- 172.21.15.78:0/415282635 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56262c2a8720 con 0x56262c2b5fa0 2024-03-14T10:01:16.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.230+0000 7f6c83400700 1 -- 172.21.15.78:0/415282635 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6c8800f980 con 0x56262c2b5fa0 2024-03-14T10:01:16.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.230+0000 7f6cb7d12b80 10 client.16980 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:16.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.230+0000 7f6cb7d12b80 10 client.16980 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:01:16.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.230+0000 7f6cb7d12b80 20 client.16980 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:16.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.230+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/415282635 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56262c2cf550 0x56262c2d1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.230+0000 7f6cb7d12b80 4 client.16980 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:01:16.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.230+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/415282635 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56262c200700 con 0x56262c2cf550 2024-03-14T10:01:16.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.230+0000 7f6ca1800700 1 --2- 172.21.15.78:0/415282635 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56262c2cf550 0x56262c2d1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.231+0000 7f6c83400700 10 client.16980 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:01:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.231+0000 7f6ca1800700 1 --2- 172.21.15.78:0/415282635 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56262c2cf550 0x56262c2d1970 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f424716d38979eb8 server_cookie=43b8e02bae285b23 in_seq=0 out_seq=0 2024-03-14T10:01:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.232+0000 7f6c83400700 1 -- 172.21.15.78:0/415282635 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56262c200700 con 0x56262c2cf550 2024-03-14T10:01:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.232+0000 7f6c83400700 10 client.16980 handle_command_reply: tid=0 2024-03-14T10:01:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.232+0000 7f6cb7d12b80 1 client.16980 shutdown 2024-03-14T10:01:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.232+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/415282635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5fa0 msgr2=0x56262c2bcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.232+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/415282635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5fa0 0x56262c2bcbc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800bd70 tx=0x7f6c88005cb0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.232+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/415282635 shutdown_connections 2024-03-14T10:01:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.232+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/415282635 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56262c2cf550 0x56262c2d1970 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.232+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/415282635 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2bfec0 0x56262c2bd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.232+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/415282635 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2b5680 0x56262c2bf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.232+0000 7f6c83400700 0 client.16980 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:01:16.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.232+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/415282635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5fa0 0x56262c2bcbc0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.232+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/415282635 >> 172.21.15.78:0/415282635 conn(0x56262c22b220 msgr2=0x56262c2b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.232+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/415282635 shutdown_connections 2024-03-14T10:01:16.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.232+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/415282635 wait complete. 2024-03-14T10:01:16.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.232+0000 7f6cb7d12b80 20 client.16980 trim_cache size 0 max 16384 2024-03-14T10:01:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.245+0000 7f6cb7d12b80 1 Processor -- start 2024-03-14T10:01:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.245+0000 7f6cb7d12b80 1 -- start start 2024-03-14T10:01:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6cb7d12b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2b5680 0x56262c2c22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6cb7d12b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5fa0 0x56262c2c6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6cb7d12b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2bfec0 0x56262c2b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6cb7d12b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262c2af800 con 0x56262c2b5fa0 2024-03-14T10:01:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6cb7d12b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262c2ad230 con 0x56262c2b5680 2024-03-14T10:01:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6cb7d12b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262c2b6af0 con 0x56262c2bfec0 2024-03-14T10:01:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6ca0e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5fa0 0x56262c2c6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6ca0e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5fa0 0x56262c2c6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41318/0 (socket says 172.21.15.78:41318) 2024-03-14T10:01:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6ca0e00700 1 -- 172.21.15.78:0/1524641966 learned_addr learned my addr 172.21.15.78:0/1524641966 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6ca1800700 1 --2- 172.21.15.78:0/1524641966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2b5680 0x56262c2c22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6ca2200700 1 --2- 172.21.15.78:0/1524641966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2bfec0 0x56262c2b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6ca0e00700 1 -- 172.21.15.78:0/1524641966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2bfec0 msgr2=0x56262c2b05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6ca0e00700 1 --2- 172.21.15.78:0/1524641966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2bfec0 0x56262c2b05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6ca0e00700 1 -- 172.21.15.78:0/1524641966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2b5680 msgr2=0x56262c2c22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6ca0e00700 1 --2- 172.21.15.78:0/1524641966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2b5680 0x56262c2c22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6ca0e00700 1 -- 172.21.15.78:0/1524641966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56262c1cdea0 con 0x56262c2b5fa0 2024-03-14T10:01:16.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.246+0000 7f6ca2200700 1 --2- 172.21.15.78:0/1524641966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2bfec0 0x56262c2b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:01:16.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.247+0000 7f6ca0e00700 1 --2- 172.21.15.78:0/1524641966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5fa0 0x56262c2c6af0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800fde0 tx=0x7f6c8800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58943383f134d19d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.247+0000 7f6c8f400700 1 -- 172.21.15.78:0/1524641966 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8801d040 con 0x56262c2b5fa0 2024-03-14T10:01:16.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.247+0000 7f6c8f400700 1 -- 172.21.15.78:0/1524641966 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c88005320 con 0x56262c2b5fa0 2024-03-14T10:01:16.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.247+0000 7f6c8f400700 1 -- 172.21.15.78:0/1524641966 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8802ca60 con 0x56262c2b5fa0 2024-03-14T10:01:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.247+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/1524641966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5fa0 msgr2=0x56262c2c6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.247+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/1524641966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5fa0 0x56262c2c6af0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800fde0 tx=0x7f6c8800b260 comp rx=0 tx=0).stop 2024-03-14T10:01:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.247+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/1524641966 shutdown_connections 2024-03-14T10:01:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.247+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/1524641966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2b5680 0x56262c2c22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.247+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/1524641966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2bfec0 0x56262c2b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.247+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/1524641966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5fa0 0x56262c2c6af0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.247+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/1524641966 >> 172.21.15.78:0/1524641966 conn(0x56262c22b220 msgr2=0x56262c2b7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.248+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/1524641966 shutdown_connections 2024-03-14T10:01:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.248+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/1524641966 wait complete. 2024-03-14T10:01:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.248+0000 7f6cb7d12b80 1 -- start start 2024-03-14T10:01:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.248+0000 7f6cb7d12b80 1 Processor -- start 2024-03-14T10:01:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.248+0000 7f6cb7d12b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5680 0x56262c2c43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.248+0000 7f6cb7d12b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2b5fa0 0x56262c2c4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.248+0000 7f6cb7d12b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2bfec0 0x56262c2c4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.248+0000 7f6cb7d12b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262c3201e0 con 0x56262c2b5680 2024-03-14T10:01:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.249+0000 7f6cb7d12b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262c320350 con 0x56262c2b5fa0 2024-03-14T10:01:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.249+0000 7f6cb7d12b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262c320510 con 0x56262c2bfec0 2024-03-14T10:01:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.249+0000 7f6cb7d12b80 10 client.0 _ll_register_callbacks cb 0x7f6c9400f330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.249+0000 7f6ca1800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5680 0x56262c2c43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.249+0000 7f6ca1800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5680 0x56262c2c43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41334/0 (socket says 172.21.15.78:41334) 2024-03-14T10:01:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.249+0000 7f6ca1800700 1 -- 172.21.15.78:0/897362474 learned_addr learned my addr 172.21.15.78:0/897362474 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.249+0000 7f6ca2200700 1 --2- 172.21.15.78:0/897362474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2bfec0 0x56262c2c4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.249+0000 7f6ca0e00700 1 --2- 172.21.15.78:0/897362474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2b5fa0 0x56262c2c4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.249+0000 7f6ca1800700 1 -- 172.21.15.78:0/897362474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2bfec0 msgr2=0x56262c2c4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.249+0000 7f6ca1800700 1 --2- 172.21.15.78:0/897362474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2bfec0 0x56262c2c4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.249+0000 7f6ca1800700 1 -- 172.21.15.78:0/897362474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2b5fa0 msgr2=0x56262c2c4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.249+0000 7f6ca1800700 1 --2- 172.21.15.78:0/897362474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2b5fa0 0x56262c2c4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.249+0000 7f6ca1800700 1 -- 172.21.15.78:0/897362474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56262c035b60 con 0x56262c2b5680 2024-03-14T10:01:16.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.249+0000 7f6ca1800700 1 --2- 172.21.15.78:0/897362474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5680 0x56262c2c43e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6c900ca350 tx=0x7f6c900b9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84329b2c9c0af382 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.249+0000 7f6c83400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:16.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.250+0000 7f6c83400700 1 -- 172.21.15.78:0/897362474 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c900a1dd0 con 0x56262c2b5680 2024-03-14T10:01:16.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.250+0000 7f6c83400700 1 -- 172.21.15.78:0/897362474 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f6c900b2170 con 0x56262c2b5680 2024-03-14T10:01:16.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.250+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/897362474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56262c056cf0 con 0x56262c2b5680 2024-03-14T10:01:16.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.250+0000 7f6c83400700 1 -- 172.21.15.78:0/897362474 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c900b2330 con 0x56262c2b5680 2024-03-14T10:01:16.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.250+0000 7f6cb7d12b80 10 client.16986 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:16.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.250+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/897362474 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56262c08dcd0 con 0x56262c2b5680 2024-03-14T10:01:16.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.250+0000 7f6c83400700 1 -- 172.21.15.78:0/897362474 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56262c056cf0 con 0x56262c2b5680 2024-03-14T10:01:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.250+0000 7f6c83400700 1 -- 172.21.15.78:0/897362474 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6c900a1940 con 0x56262c2b5680 2024-03-14T10:01:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.250+0000 7f6cb7d12b80 10 client.16986 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.250+0000 7f6cb7d12b80 10 client.16986 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:01:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.250+0000 7f6cb7d12b80 20 client.16986 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.250+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/897362474 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56262c321d60 0x56262c324180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.250+0000 7f6cb7d12b80 4 client.16986 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:01:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.250+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/897362474 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56262c08bce0 con 0x56262c321d60 2024-03-14T10:01:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.251+0000 7f6ca0e00700 1 --2- 172.21.15.78:0/897362474 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56262c321d60 0x56262c324180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.251+0000 7f6ca0e00700 1 --2- 172.21.15.78:0/897362474 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56262c321d60 0x56262c324180 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=afdae79b59ce4706 server_cookie=e206ee70e2083800 in_seq=0 out_seq=0 2024-03-14T10:01:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.251+0000 7f6c83400700 10 client.16986 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:01:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.251+0000 7f6c83400700 1 -- 172.21.15.78:0/897362474 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56262c08bce0 con 0x56262c321d60 2024-03-14T10:01:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.251+0000 7f6c83400700 10 client.16986 handle_command_reply: tid=0 2024-03-14T10:01:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.251+0000 7f6cb7d12b80 1 client.16986 shutdown 2024-03-14T10:01:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.251+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/897362474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5680 msgr2=0x56262c2c43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.251+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/897362474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5680 0x56262c2c43e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6c900ca350 tx=0x7f6c900b9040 comp rx=0 tx=0).stop 2024-03-14T10:01:16.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.251+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/897362474 shutdown_connections 2024-03-14T10:01:16.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.251+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/897362474 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56262c321d60 0x56262c324180 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/897362474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56262c2bfec0 0x56262c2c4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/897362474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262c2b5680 0x56262c2c43e0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6c83400700 0 client.16986 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:01:16.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6cb7d12b80 1 --2- 172.21.15.78:0/897362474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56262c2b5fa0 0x56262c2c4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/897362474 >> 172.21.15.78:0/897362474 conn(0x56262c22b220 msgr2=0x56262c2b05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/897362474 shutdown_connections 2024-03-14T10:01:16.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6cb7d12b80 1 -- 172.21.15.78:0/897362474 wait complete. 2024-03-14T10:01:16.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6cb7d12b80 20 client.16986 trim_cache size 0 max 16384 2024-03-14T10:01:16.256 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:01:16.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6ca4200700 1 -- 172.21.15.78:0/2530157145 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6c840a0c00 msgr2=0x7f6c840a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6ca4200700 1 --2- 172.21.15.78:0/2530157145 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6c840a0c00 0x7f6c840a30c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400f590 tx=0x7f6c94009920 comp rx=0 tx=0).stop 2024-03-14T10:01:16.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6ca4200700 1 -- 172.21.15.78:0/2530157145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c9c100f30 msgr2=0x7f6c9c19f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6ca4200700 1 --2- 172.21.15.78:0/2530157145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c9c100f30 0x7f6c9c19f530 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000e9a0 tx=0x7f6c9000ecb0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6ca4200700 1 -- 172.21.15.78:0/2530157145 shutdown_connections 2024-03-14T10:01:16.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6ca4200700 1 --2- 172.21.15.78:0/2530157145 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6c9c199660 0x7f6c9c199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6ca4200700 1 --2- 172.21.15.78:0/2530157145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c9c100f30 0x7f6c9c19f530 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6ca4200700 1 --2- 172.21.15.78:0/2530157145 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6c840a0c00 0x7f6c840a30c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6ca4200700 1 --2- 172.21.15.78:0/2530157145 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c9c106050 0x7f6c9c19fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.252+0000 7f6ca4200700 1 -- 172.21.15.78:0/2530157145 >> 172.21.15.78:0/2530157145 conn(0x7f6c9c0fc7b0 msgr2=0x7f6c9c0feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.253+0000 7f6ca4200700 1 -- 172.21.15.78:0/2530157145 shutdown_connections 2024-03-14T10:01:16.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.253+0000 7f6ca4200700 1 -- 172.21.15.78:0/2530157145 wait complete. 2024-03-14T10:01:16.270 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T10:01:16.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:16 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2004807332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:01:16.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.465+0000 7fbecaa00700 1 -- 172.21.15.78:0/500411072 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbec41018e0 msgr2=0x7fbec4105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.465+0000 7fbecaa00700 1 --2- 172.21.15.78:0/500411072 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbec41018e0 0x7fbec4105920 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009990 tx=0x7fbeb0009ca0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.465+0000 7fbecaa00700 1 -- 172.21.15.78:0/500411072 shutdown_connections 2024-03-14T10:01:16.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.465+0000 7fbecaa00700 1 --2- 172.21.15.78:0/500411072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbec4100f30 0x7fbec4101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.465+0000 7fbecaa00700 1 --2- 172.21.15.78:0/500411072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbec4106050 0x7fbec410dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.465+0000 7fbecaa00700 1 --2- 172.21.15.78:0/500411072 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbec41018e0 0x7fbec4105920 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.465+0000 7fbecaa00700 1 -- 172.21.15.78:0/500411072 >> 172.21.15.78:0/500411072 conn(0x7fbec40fc7b0 msgr2=0x7fbec40febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.465+0000 7fbecaa00700 1 -- 172.21.15.78:0/500411072 shutdown_connections 2024-03-14T10:01:16.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.465+0000 7fbecaa00700 1 -- 172.21.15.78:0/500411072 wait complete. 2024-03-14T10:01:16.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.466+0000 7fbecaa00700 1 Processor -- start 2024-03-14T10:01:16.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.466+0000 7fbecaa00700 1 -- start start 2024-03-14T10:01:16.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.467+0000 7fbecaa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbec4100f30 0x7fbec41976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.467+0000 7fbecaa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbec4106050 0x7fbec4197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.467+0000 7fbecaa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbec419bf40 0x7fbec419c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.467+0000 7fbecaa00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec419c9c0 con 0x7fbec4100f30 2024-03-14T10:01:16.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.467+0000 7fbecaa00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec419cb30 con 0x7fbec4106050 2024-03-14T10:01:16.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.467+0000 7fbecaa00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec419ccf0 con 0x7fbec419bf40 2024-03-14T10:01:16.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.467+0000 7fbec3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbec4100f30 0x7fbec41976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.467+0000 7fbec3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbec4100f30 0x7fbec41976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41358/0 (socket says 172.21.15.78:41358) 2024-03-14T10:01:16.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.467+0000 7fbec3e00700 1 -- 172.21.15.78:0/1572964668 learned_addr learned my addr 172.21.15.78:0/1572964668 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:16.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.467+0000 7fbec3400700 1 --2- 172.21.15.78:0/1572964668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbec4106050 0x7fbec4197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.467+0000 7fbec8a00700 1 --2- 172.21.15.78:0/1572964668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbec419bf40 0x7fbec419c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.467+0000 7fbec3e00700 1 -- 172.21.15.78:0/1572964668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbec419bf40 msgr2=0x7fbec419c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.467+0000 7fbec3e00700 1 --2- 172.21.15.78:0/1572964668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbec419bf40 0x7fbec419c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.467+0000 7fbec3e00700 1 -- 172.21.15.78:0/1572964668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbec4106050 msgr2=0x7fbec4197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.467+0000 7fbec3e00700 1 --2- 172.21.15.78:0/1572964668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbec4106050 0x7fbec4197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.467+0000 7fbec3e00700 1 -- 172.21.15.78:0/1572964668 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb00095f0 con 0x7fbec4100f30 2024-03-14T10:01:16.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.468+0000 7fbec3e00700 1 --2- 172.21.15.78:0/1572964668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbec4100f30 0x7fbec41976b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400e9a0 tx=0x7fbeb400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.468+0000 7fbec0c00700 1 -- 172.21.15.78:0/1572964668 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb4009790 con 0x7fbec4100f30 2024-03-14T10:01:16.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.468+0000 7fbec0c00700 1 -- 172.21.15.78:0/1572964668 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbeb400cbd0 con 0x7fbec4100f30 2024-03-14T10:01:16.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.468+0000 7fbecaa00700 1 -- 172.21.15.78:0/1572964668 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec41a4250 con 0x7fbec4100f30 2024-03-14T10:01:16.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.468+0000 7fbec0c00700 1 -- 172.21.15.78:0/1572964668 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb4018680 con 0x7fbec4100f30 2024-03-14T10:01:16.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.468+0000 7fbecaa00700 1 -- 172.21.15.78:0/1572964668 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec41a4730 con 0x7fbec4100f30 2024-03-14T10:01:16.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.471+0000 7fbec0c00700 1 -- 172.21.15.78:0/1572964668 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fbeb400cd40 con 0x7fbec4100f30 2024-03-14T10:01:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.472+0000 7fbede61bb80 1 Processor -- start 2024-03-14T10:01:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.472+0000 7fbede61bb80 1 -- start start 2024-03-14T10:01:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.472+0000 7fbec0c00700 1 --2- 172.21.15.78:0/1572964668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbea80a0c00 0x7fbea80a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.472+0000 7fbec0c00700 1 -- 172.21.15.78:0/1572964668 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fbeb4014070 con 0x7fbec4100f30 2024-03-14T10:01:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.472+0000 7fbede61bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9680 0x557055fa9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.472+0000 7fbede61bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9fa0 0x557055fb3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.472+0000 7fbede61bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fb3ec0 0x557055fb62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.472+0000 7fbede61bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557055f02990 con 0x557055fa9fa0 2024-03-14T10:01:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.472+0000 7fbede61bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557055f00350 con 0x557055fb3ec0 2024-03-14T10:01:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.472+0000 7fbede61bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557055efbc40 con 0x557055fa9680 2024-03-14T10:01:16.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.473+0000 7fbec8a00700 1 --2- 172.21.15.78:0/1572964668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbea80a0c00 0x7fbea80a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.473+0000 7fbec3400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9fa0 0x557055fb3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.473+0000 7fbec3400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9fa0 0x557055fb3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41366/0 (socket says 172.21.15.78:41366) 2024-03-14T10:01:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.473+0000 7fbec3400700 1 -- 172.21.15.78:0/2068163900 learned_addr learned my addr 172.21.15.78:0/2068163900 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.473+0000 7fbec3e00700 1 --2- 172.21.15.78:0/2068163900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9680 0x557055fa9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.473+0000 7fbec8a00700 1 --2- 172.21.15.78:0/2068163900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fb3ec0 0x557055fb62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.473+0000 7fbec3400700 1 -- 172.21.15.78:0/2068163900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9680 msgr2=0x557055fa9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.473+0000 7fbec3400700 1 --2- 172.21.15.78:0/2068163900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9680 0x557055fa9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.473+0000 7fbec3400700 1 -- 172.21.15.78:0/2068163900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fb3ec0 msgr2=0x557055fb62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.473+0000 7fbec3400700 1 --2- 172.21.15.78:0/2068163900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fb3ec0 0x557055fb62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.473+0000 7fbec3400700 1 -- 172.21.15.78:0/2068163900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557055d7d3b0 con 0x557055fa9fa0 2024-03-14T10:01:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.473+0000 7fbec3e00700 1 --2- 172.21.15.78:0/2068163900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9680 0x557055fa9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.473+0000 7fbec3400700 1 --2- 172.21.15.78:0/2068163900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9fa0 0x557055fb3980 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009990 tx=0x7fbeb0005ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0cfb30244a676b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.473+0000 7fbec8a00700 1 --2- 172.21.15.78:0/1572964668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbea80a0c00 0x7fbea80a30c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009510 tx=0x7fbebc009920 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.474+0000 7fbeb9200700 1 -- 172.21.15.78:0/2068163900 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb001d070 con 0x557055fa9fa0 2024-03-14T10:01:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.474+0000 7fbeb9200700 1 -- 172.21.15.78:0/2068163900 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbeb001e040 con 0x557055fa9fa0 2024-03-14T10:01:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.474+0000 7fbeb9200700 1 -- 172.21.15.78:0/2068163900 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb00047f0 con 0x557055fa9fa0 2024-03-14T10:01:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.474+0000 7fbede61bb80 1 -- 172.21.15.78:0/2068163900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9fa0 msgr2=0x557055fb3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.474+0000 7fbede61bb80 1 --2- 172.21.15.78:0/2068163900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9fa0 0x557055fb3980 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009990 tx=0x7fbeb0005ec0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.474+0000 7fbede61bb80 1 -- 172.21.15.78:0/2068163900 shutdown_connections 2024-03-14T10:01:16.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.474+0000 7fbede61bb80 1 --2- 172.21.15.78:0/2068163900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fb3ec0 0x557055fb62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.475+0000 7fbede61bb80 1 --2- 172.21.15.78:0/2068163900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9680 0x557055fa9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.475+0000 7fbede61bb80 1 --2- 172.21.15.78:0/2068163900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9fa0 0x557055fb3980 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.475+0000 7fbede61bb80 1 -- 172.21.15.78:0/2068163900 >> 172.21.15.78:0/2068163900 conn(0x557055f1f220 msgr2=0x557055fa4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.475+0000 7fbede61bb80 1 -- 172.21.15.78:0/2068163900 shutdown_connections 2024-03-14T10:01:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.475+0000 7fbede61bb80 1 -- 172.21.15.78:0/2068163900 wait complete. 2024-03-14T10:01:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.475+0000 7fbede61bb80 1 -- start start 2024-03-14T10:01:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.475+0000 7fbede61bb80 1 Processor -- start 2024-03-14T10:01:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbede61bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9680 0x557055fb3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbede61bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9fa0 0x557055fb0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbede61bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fb3ec0 0x557055fb1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbede61bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557055e97d80 con 0x557055fa9680 2024-03-14T10:01:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbede61bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557055e97a20 con 0x557055fb3ec0 2024-03-14T10:01:16.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbede61bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557055d7f6a0 con 0x557055fa9fa0 2024-03-14T10:01:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbede61bb80 10 client.0 _ll_register_callbacks cb 0x557055d033a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbec3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9680 0x557055fb3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbec3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9680 0x557055fb3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41378/0 (socket says 172.21.15.78:41378) 2024-03-14T10:01:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbec3e00700 1 -- 172.21.15.78:0/972903140 learned_addr learned my addr 172.21.15.78:0/972903140 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbec3400700 1 --2- 172.21.15.78:0/972903140 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9fa0 0x557055fb0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbec8a00700 1 --2- 172.21.15.78:0/972903140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fb3ec0 0x557055fb1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbec3e00700 1 -- 172.21.15.78:0/972903140 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9fa0 msgr2=0x557055fb0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbec3e00700 1 --2- 172.21.15.78:0/972903140 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9fa0 0x557055fb0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbec3e00700 1 -- 172.21.15.78:0/972903140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fb3ec0 msgr2=0x557055fb1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbec3e00700 1 --2- 172.21.15.78:0/972903140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fb3ec0 0x557055fb1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbec3e00700 1 -- 172.21.15.78:0/972903140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec41a4730 con 0x557055fa9680 2024-03-14T10:01:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.476+0000 7fbec8a00700 1 --2- 172.21.15.78:0/972903140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fb3ec0 0x557055fb1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:01:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.477+0000 7fbec3e00700 1 --2- 172.21.15.78:0/972903140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9680 0x557055fb3850 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4095ef0 tx=0x7fbeb40993d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21259d36d452fa8d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.477+0000 7fbead600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:16.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.477+0000 7fbead600700 1 -- 172.21.15.78:0/972903140 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb40b1070 con 0x557055fa9680 2024-03-14T10:01:16.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.477+0000 7fbead600700 1 -- 172.21.15.78:0/972903140 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fbeb40999f0 con 0x557055fa9680 2024-03-14T10:01:16.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.477+0000 7fbede61bb80 1 -- 172.21.15.78:0/972903140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557055f9c720 con 0x557055fa9680 2024-03-14T10:01:16.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.477+0000 7fbead600700 1 -- 172.21.15.78:0/972903140 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbeb409d820 con 0x557055fa9680 2024-03-14T10:01:16.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.477+0000 7fbede61bb80 10 client.16998 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:16.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.477+0000 7fbede61bb80 1 -- 172.21.15.78:0/972903140 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7fbec410e440 con 0x557055fa9680 2024-03-14T10:01:16.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.477+0000 7fbead600700 1 -- 172.21.15.78:0/972903140 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557055f9c720 con 0x557055fa9680 2024-03-14T10:01:16.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.478+0000 7fbead600700 1 -- 172.21.15.78:0/972903140 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbeb40a1a70 con 0x557055fa9680 2024-03-14T10:01:16.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.478+0000 7fbede61bb80 10 client.16998 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:16.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.478+0000 7fbede61bb80 10 client.16998 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:01:16.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.478+0000 7fbede61bb80 20 client.16998 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:16.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.478+0000 7fbede61bb80 1 --2- 172.21.15.78:0/972903140 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x557055fc3a30 0x557055fc5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.478+0000 7fbede61bb80 4 client.16998 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:01:16.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.478+0000 7fbede61bb80 1 -- 172.21.15.78:0/972903140 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557055ef4700 con 0x557055fc3a30 2024-03-14T10:01:16.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.478+0000 7fbec3400700 1 --2- 172.21.15.78:0/972903140 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x557055fc3a30 0x557055fc5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.478+0000 7fbec3400700 1 --2- 172.21.15.78:0/972903140 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x557055fc3a30 0x557055fc5e20 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7e11046f2632204a server_cookie=f679542c5528e952 in_seq=0 out_seq=0 2024-03-14T10:01:16.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.478+0000 7fbead600700 10 client.16998 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:01:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.479+0000 7fbead600700 1 -- 172.21.15.78:0/972903140 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x557055ef4700 con 0x557055fc3a30 2024-03-14T10:01:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.479+0000 7fbead600700 10 client.16998 handle_command_reply: tid=0 2024-03-14T10:01:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.479+0000 7fbede61bb80 1 client.16998 shutdown 2024-03-14T10:01:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.479+0000 7fbede61bb80 1 -- 172.21.15.78:0/972903140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9680 msgr2=0x557055fb3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.479+0000 7fbede61bb80 1 --2- 172.21.15.78:0/972903140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9680 0x557055fb3850 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4095ef0 tx=0x7fbeb40993d0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.479+0000 7fbede61bb80 1 -- 172.21.15.78:0/972903140 shutdown_connections 2024-03-14T10:01:16.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.479+0000 7fbede61bb80 1 --2- 172.21.15.78:0/972903140 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x557055fc3a30 0x557055fc5e20 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.479+0000 7fbede61bb80 1 --2- 172.21.15.78:0/972903140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fb3ec0 0x557055fb1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.479+0000 7fbead600700 0 client.16998 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:01:16.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.479+0000 7fbede61bb80 1 --2- 172.21.15.78:0/972903140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9680 0x557055fb3850 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.479+0000 7fbede61bb80 1 --2- 172.21.15.78:0/972903140 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9fa0 0x557055fb0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.479+0000 7fbede61bb80 1 -- 172.21.15.78:0/972903140 >> 172.21.15.78:0/972903140 conn(0x557055f1f220 msgr2=0x557055fa4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.479+0000 7fbede61bb80 1 -- 172.21.15.78:0/972903140 shutdown_connections 2024-03-14T10:01:16.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.480+0000 7fbede61bb80 1 -- 172.21.15.78:0/972903140 wait complete. 2024-03-14T10:01:16.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.480+0000 7fbede61bb80 20 client.16998 trim_cache size 0 max 16384 2024-03-14T10:01:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.493+0000 7fbede61bb80 1 Processor -- start 2024-03-14T10:01:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.493+0000 7fbede61bb80 1 -- start start 2024-03-14T10:01:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.493+0000 7fbede61bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9680 0x557055fb62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.493+0000 7fbede61bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9fa0 0x557055fbaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.493+0000 7fbede61bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fb3ec0 0x557055fb3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.493+0000 7fbede61bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557055fa5e30 con 0x557055fa9680 2024-03-14T10:01:16.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.493+0000 7fbede61bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557055fa6060 con 0x557055fb3ec0 2024-03-14T10:01:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.493+0000 7fbede61bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557055fc3810 con 0x557055fa9fa0 2024-03-14T10:01:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.493+0000 7fbec3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9680 0x557055fb62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.493+0000 7fbec3400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9fa0 0x557055fbaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.493+0000 7fbec8a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fb3ec0 0x557055fb3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.493+0000 7fbec3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9680 0x557055fb62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41392/0 (socket says 172.21.15.78:41392) 2024-03-14T10:01:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.493+0000 7fbec3e00700 1 -- 172.21.15.78:0/113921592 learned_addr learned my addr 172.21.15.78:0/113921592 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.494+0000 7fbec3e00700 1 -- 172.21.15.78:0/113921592 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9fa0 msgr2=0x557055fbaaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.494+0000 7fbec3e00700 1 --2- 172.21.15.78:0/113921592 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9fa0 0x557055fbaaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.494+0000 7fbec3e00700 1 -- 172.21.15.78:0/113921592 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fb3ec0 msgr2=0x557055fb3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.494+0000 7fbec3e00700 1 --2- 172.21.15.78:0/113921592 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fb3ec0 0x557055fb3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.494+0000 7fbec3e00700 1 -- 172.21.15.78:0/113921592 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557055d29b60 con 0x557055fa9680 2024-03-14T10:01:16.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.494+0000 7fbec3e00700 1 --2- 172.21.15.78:0/113921592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9680 0x557055fb62b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fbeb40996f0 tx=0x7fbeb4099f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d3172613dfeaae server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.494+0000 7fbeb9c00700 1 -- 172.21.15.78:0/113921592 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb40b1030 con 0x557055fa9680 2024-03-14T10:01:16.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.494+0000 7fbeb9c00700 1 -- 172.21.15.78:0/113921592 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbeb4098af0 con 0x557055fa9680 2024-03-14T10:01:16.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.494+0000 7fbeb9c00700 1 -- 172.21.15.78:0/113921592 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb40a19e0 con 0x557055fa9680 2024-03-14T10:01:16.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.495+0000 7fbede61bb80 1 -- 172.21.15.78:0/113921592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9680 msgr2=0x557055fb62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.495+0000 7fbede61bb80 1 --2- 172.21.15.78:0/113921592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9680 0x557055fb62b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fbeb40996f0 tx=0x7fbeb4099f30 comp rx=0 tx=0).stop 2024-03-14T10:01:16.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.495+0000 7fbede61bb80 1 -- 172.21.15.78:0/113921592 shutdown_connections 2024-03-14T10:01:16.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.495+0000 7fbede61bb80 1 --2- 172.21.15.78:0/113921592 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fb3ec0 0x557055fb3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.495+0000 7fbede61bb80 1 --2- 172.21.15.78:0/113921592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fa9680 0x557055fb62b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.495+0000 7fbede61bb80 1 --2- 172.21.15.78:0/113921592 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9fa0 0x557055fbaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.495+0000 7fbede61bb80 1 -- 172.21.15.78:0/113921592 >> 172.21.15.78:0/113921592 conn(0x557055f1f220 msgr2=0x557055faf4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.495+0000 7fbede61bb80 1 -- 172.21.15.78:0/113921592 shutdown_connections 2024-03-14T10:01:16.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.495+0000 7fbede61bb80 1 -- 172.21.15.78:0/113921592 wait complete. 2024-03-14T10:01:16.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.495+0000 7fbede61bb80 1 -- start start 2024-03-14T10:01:16.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.495+0000 7fbede61bb80 1 Processor -- start 2024-03-14T10:01:16.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.496+0000 7fbede61bb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9680 0x557055fb8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.496+0000 7fbede61bb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fa9fa0 0x557055fb8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.496+0000 7fbede61bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fb3ec0 0x5570560144f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.496+0000 7fbede61bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557056014a30 con 0x557055fb3ec0 2024-03-14T10:01:16.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.496+0000 7fbede61bb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557056014b70 con 0x557055fa9fa0 2024-03-14T10:01:16.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.496+0000 7fbede61bb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557056014cb0 con 0x557055fa9680 2024-03-14T10:01:16.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.496+0000 7fbede61bb80 10 client.0 _ll_register_callbacks cb 0x7fbeb4099530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:16.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.496+0000 7fbec8a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fb3ec0 0x5570560144f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.496+0000 7fbec8a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fb3ec0 0x5570560144f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41402/0 (socket says 172.21.15.78:41402) 2024-03-14T10:01:16.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.496+0000 7fbec8a00700 1 -- 172.21.15.78:0/1014741437 learned_addr learned my addr 172.21.15.78:0/1014741437 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:16.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.496+0000 7fbec3e00700 1 --2- 172.21.15.78:0/1014741437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9680 0x557055fb8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.496+0000 7fbec3400700 1 --2- 172.21.15.78:0/1014741437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fa9fa0 0x557055fb8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.497+0000 7fbec8a00700 1 -- 172.21.15.78:0/1014741437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9680 msgr2=0x557055fb8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.497+0000 7fbec8a00700 1 --2- 172.21.15.78:0/1014741437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9680 0x557055fb8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.497+0000 7fbec8a00700 1 -- 172.21.15.78:0/1014741437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fa9fa0 msgr2=0x557055fb8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.497+0000 7fbec8a00700 1 --2- 172.21.15.78:0/1014741437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fa9fa0 0x557055fb8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.497+0000 7fbec8a00700 1 -- 172.21.15.78:0/1014741437 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec410e440 con 0x557055fb3ec0 2024-03-14T10:01:16.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.497+0000 7fbec3e00700 1 --2- 172.21.15.78:0/1014741437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9680 0x557055fb8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:16.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.497+0000 7fbec3400700 1 --2- 172.21.15.78:0/1014741437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fa9fa0 0x557055fb8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:16.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.497+0000 7fbec8a00700 1 --2- 172.21.15.78:0/1014741437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fb3ec0 0x5570560144f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbebc008010 tx=0x7fbebc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9b1ad50d90ffc34 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.497+0000 7fbead600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:16.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.497+0000 7fbead600700 1 -- 172.21.15.78:0/1014741437 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc00e050 con 0x557055fb3ec0 2024-03-14T10:01:16.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.497+0000 7fbead600700 1 -- 172.21.15.78:0/1014741437 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fbebc009e20 con 0x557055fb3ec0 2024-03-14T10:01:16.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.497+0000 7fbede61bb80 1 -- 172.21.15.78:0/1014741437 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbeb001e040 con 0x557055fb3ec0 2024-03-14T10:01:16.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.497+0000 7fbead600700 1 -- 172.21.15.78:0/1014741437 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbebc01ba30 con 0x557055fb3ec0 2024-03-14T10:01:16.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.497+0000 7fbede61bb80 10 client.17004 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:16.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.497+0000 7fbede61bb80 1 -- 172.21.15.78:0/1014741437 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557055ec1ea0 con 0x557055fb3ec0 2024-03-14T10:01:16.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.498+0000 7fbead600700 1 -- 172.21.15.78:0/1014741437 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbeb001e040 con 0x557055fb3ec0 2024-03-14T10:01:16.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.498+0000 7fbead600700 1 -- 172.21.15.78:0/1014741437 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbebc028d10 con 0x557055fb3ec0 2024-03-14T10:01:16.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.498+0000 7fbede61bb80 10 client.17004 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:16.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.498+0000 7fbede61bb80 10 client.17004 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:01:16.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.498+0000 7fbede61bb80 20 client.17004 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:16.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.498+0000 7fbede61bb80 1 --2- 172.21.15.78:0/1014741437 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5570560164a0 0x5570560188c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.498+0000 7fbede61bb80 4 client.17004 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:01:16.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.498+0000 7fbede61bb80 1 -- 172.21.15.78:0/1014741437 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557055d4acf0 con 0x5570560164a0 2024-03-14T10:01:16.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.498+0000 7fbec3e00700 1 --2- 172.21.15.78:0/1014741437 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5570560164a0 0x5570560188c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.499+0000 7fbead600700 10 client.17004 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:01:16.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.499+0000 7fbec3e00700 1 --2- 172.21.15.78:0/1014741437 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5570560164a0 0x5570560188c0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d96fc6878cea2931 server_cookie=db64a3cd6aecf36f in_seq=0 out_seq=0 2024-03-14T10:01:16.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.499+0000 7fbead600700 1 -- 172.21.15.78:0/1014741437 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557055d4acf0 con 0x5570560164a0 2024-03-14T10:01:16.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.499+0000 7fbead600700 10 client.17004 handle_command_reply: tid=0 2024-03-14T10:01:16.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.499+0000 7fbede61bb80 1 client.17004 shutdown 2024-03-14T10:01:16.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.499+0000 7fbede61bb80 1 -- 172.21.15.78:0/1014741437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fb3ec0 msgr2=0x5570560144f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.499+0000 7fbede61bb80 1 --2- 172.21.15.78:0/1014741437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fb3ec0 0x5570560144f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbebc008010 tx=0x7fbebc00b040 comp rx=0 tx=0).stop 2024-03-14T10:01:16.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.499+0000 7fbede61bb80 1 -- 172.21.15.78:0/1014741437 shutdown_connections 2024-03-14T10:01:16.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.499+0000 7fbede61bb80 1 --2- 172.21.15.78:0/1014741437 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5570560164a0 0x5570560188c0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.499+0000 7fbede61bb80 1 --2- 172.21.15.78:0/1014741437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557055fb3ec0 0x5570560144f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.499+0000 7fbede61bb80 1 --2- 172.21.15.78:0/1014741437 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557055fa9680 0x557055fb8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.499+0000 7fbead600700 0 client.17004 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:01:16.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.499+0000 7fbede61bb80 1 --2- 172.21.15.78:0/1014741437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557055fa9fa0 0x557055fb8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.499+0000 7fbede61bb80 1 -- 172.21.15.78:0/1014741437 >> 172.21.15.78:0/1014741437 conn(0x557055f1f220 msgr2=0x557055faf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.499+0000 7fbede61bb80 1 -- 172.21.15.78:0/1014741437 shutdown_connections 2024-03-14T10:01:16.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.499+0000 7fbede61bb80 1 -- 172.21.15.78:0/1014741437 wait complete. 2024-03-14T10:01:16.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.500+0000 7fbede61bb80 20 client.17004 trim_cache size 0 max 16384 2024-03-14T10:01:16.504 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:01:16.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.500+0000 7fbecaa00700 1 -- 172.21.15.78:0/1572964668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbea80a0c00 msgr2=0x7fbea80a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.500+0000 7fbecaa00700 1 --2- 172.21.15.78:0/1572964668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbea80a0c00 0x7fbea80a30c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009510 tx=0x7fbebc009920 comp rx=0 tx=0).stop 2024-03-14T10:01:16.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.500+0000 7fbecaa00700 1 -- 172.21.15.78:0/1572964668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbec4100f30 msgr2=0x7fbec41976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.500+0000 7fbecaa00700 1 --2- 172.21.15.78:0/1572964668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbec4100f30 0x7fbec41976b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400e9a0 tx=0x7fbeb400ecb0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.500+0000 7fbecaa00700 1 -- 172.21.15.78:0/1572964668 shutdown_connections 2024-03-14T10:01:16.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.500+0000 7fbecaa00700 1 --2- 172.21.15.78:0/1572964668 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbec419bf40 0x7fbec419c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.500+0000 7fbecaa00700 1 --2- 172.21.15.78:0/1572964668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbec4100f30 0x7fbec41976b0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.500+0000 7fbecaa00700 1 --2- 172.21.15.78:0/1572964668 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbea80a0c00 0x7fbea80a30c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.500+0000 7fbecaa00700 1 --2- 172.21.15.78:0/1572964668 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbec4106050 0x7fbec4197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.500+0000 7fbecaa00700 1 -- 172.21.15.78:0/1572964668 >> 172.21.15.78:0/1572964668 conn(0x7fbec40fc7b0 msgr2=0x7fbec40fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.500+0000 7fbecaa00700 1 -- 172.21.15.78:0/1572964668 shutdown_connections 2024-03-14T10:01:16.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.500+0000 7fbecaa00700 1 -- 172.21.15.78:0/1572964668 wait complete. 2024-03-14T10:01:16.506 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:16 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2004807332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:01:16.518 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T10:01:16.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:16 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2004807332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:01:16.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.754+0000 7f5591e00700 1 -- 172.21.15.78:0/485359029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f558c106050 msgr2=0x7f558c10dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.754+0000 7f5591e00700 1 --2- 172.21.15.78:0/485359029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f558c106050 0x7f558c10dbd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f558400b0a0 tx=0x7f558400b3b0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.754+0000 7f5591e00700 1 -- 172.21.15.78:0/485359029 shutdown_connections 2024-03-14T10:01:16.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.754+0000 7f5591e00700 1 --2- 172.21.15.78:0/485359029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f558c106050 0x7f558c10dbd0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.754+0000 7f5591e00700 1 --2- 172.21.15.78:0/485359029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f558c100f30 0x7f558c101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.754+0000 7f5591e00700 1 --2- 172.21.15.78:0/485359029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f558c1018e0 0x7f558c105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.754+0000 7f5591e00700 1 -- 172.21.15.78:0/485359029 >> 172.21.15.78:0/485359029 conn(0x7f558c0fc7b0 msgr2=0x7f558c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.754+0000 7f5591e00700 1 -- 172.21.15.78:0/485359029 shutdown_connections 2024-03-14T10:01:16.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.754+0000 7f5591e00700 1 -- 172.21.15.78:0/485359029 wait complete. 2024-03-14T10:01:16.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.755+0000 7f5591e00700 1 Processor -- start 2024-03-14T10:01:16.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.755+0000 7f5591e00700 1 -- start start 2024-03-14T10:01:16.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.756+0000 7f5591e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f558c100f30 0x7f558c19f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.756+0000 7f5591e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f558c106050 0x7f558c19fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.756+0000 7f5591e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f558c199720 0x7f558c199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.756+0000 7f5591e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c19a1a0 con 0x7f558c100f30 2024-03-14T10:01:16.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.756+0000 7f5591e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c19a310 con 0x7f558c199720 2024-03-14T10:01:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.756+0000 7f5591e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c19a4d0 con 0x7f558c106050 2024-03-14T10:01:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.756+0000 7f558aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f558c100f30 0x7f558c19f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.756+0000 7f558aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f558c100f30 0x7f558c19f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41422/0 (socket says 172.21.15.78:41422) 2024-03-14T10:01:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.756+0000 7f558aa00700 1 -- 172.21.15.78:0/4196572922 learned_addr learned my addr 172.21.15.78:0/4196572922 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.756+0000 7f558a000700 1 --2- 172.21.15.78:0/4196572922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f558c106050 0x7f558c19fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.757+0000 7f558b400700 1 --2- 172.21.15.78:0/4196572922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f558c199720 0x7f558c199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.757+0000 7f558aa00700 1 -- 172.21.15.78:0/4196572922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f558c106050 msgr2=0x7f558c19fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.757+0000 7f558aa00700 1 --2- 172.21.15.78:0/4196572922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f558c106050 0x7f558c19fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.757+0000 7f558aa00700 1 -- 172.21.15.78:0/4196572922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f558c199720 msgr2=0x7f558c199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.757+0000 7f558aa00700 1 --2- 172.21.15.78:0/4196572922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f558c199720 0x7f558c199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.757+0000 7f558aa00700 1 -- 172.21.15.78:0/4196572922 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5584009d00 con 0x7f558c100f30 2024-03-14T10:01:16.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.757+0000 7f558aa00700 1 --2- 172.21.15.78:0/4196572922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f558c100f30 0x7f558c19f5f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f557c00e9a0 tx=0x7f557c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.757+0000 7f5590a00700 1 -- 172.21.15.78:0/4196572922 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c009790 con 0x7f558c100f30 2024-03-14T10:01:16.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.757+0000 7f5590a00700 1 -- 172.21.15.78:0/4196572922 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f557c010430 con 0x7f558c100f30 2024-03-14T10:01:16.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.757+0000 7f5591e00700 1 -- 172.21.15.78:0/4196572922 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558c1a64c0 con 0x7f558c100f30 2024-03-14T10:01:16.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.757+0000 7f5590a00700 1 -- 172.21.15.78:0/4196572922 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c018680 con 0x7f558c100f30 2024-03-14T10:01:16.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.757+0000 7f5591e00700 1 -- 172.21.15.78:0/4196572922 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558c1a69f0 con 0x7f558c100f30 2024-03-14T10:01:16.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.760+0000 7f5590a00700 1 -- 172.21.15.78:0/4196572922 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f557c0105a0 con 0x7f558c100f30 2024-03-14T10:01:16.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.761+0000 7f5590a00700 1 --2- 172.21.15.78:0/4196572922 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f55700a0c80 0x7f55700a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.761+0000 7f55a588db80 1 Processor -- start 2024-03-14T10:01:16.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.762+0000 7f55a588db80 1 -- start start 2024-03-14T10:01:16.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.762+0000 7f5590a00700 1 -- 172.21.15.78:0/4196572922 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f557c014070 con 0x7f558c100f30 2024-03-14T10:01:16.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.762+0000 7f558a000700 1 --2- 172.21.15.78:0/4196572922 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f55700a0c80 0x7f55700a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.762+0000 7f55a588db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107b680 0x55ec9107ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.762+0000 7f55a588db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec9107bfa0 0x55ec91085980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.762+0000 7f55a588db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec91085ec0 0x55ec910882b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.762+0000 7f55a588db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec90fd4990 con 0x55ec91085ec0 2024-03-14T10:01:16.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.762+0000 7f55a588db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec90fd2350 con 0x55ec9107b680 2024-03-14T10:01:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.762+0000 7f55a588db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec90fcdc40 con 0x55ec9107bfa0 2024-03-14T10:01:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.762+0000 7f558a000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec9107bfa0 0x55ec91085980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.762+0000 7f558b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec91085ec0 0x55ec910882b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.762+0000 7f558aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107b680 0x55ec9107ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.762+0000 7f558b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec91085ec0 0x55ec910882b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41438/0 (socket says 172.21.15.78:41438) 2024-03-14T10:01:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.762+0000 7f558b400700 1 -- 172.21.15.78:0/1674067152 learned_addr learned my addr 172.21.15.78:0/1674067152 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.762+0000 7f558aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107b680 0x55ec9107ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:58400/0 (socket says 172.21.15.78:58400) 2024-03-14T10:01:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.763+0000 7f558a000700 1 --2- 172.21.15.78:0/4196572922 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f55700a0c80 0x7f55700a3140 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f558c19ae30 tx=0x7f557800b410 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.763+0000 7f558a000700 1 -- 172.21.15.78:0/1674067152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107b680 msgr2=0x55ec9107ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.763+0000 7f558a000700 1 --2- 172.21.15.78:0/1674067152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107b680 0x55ec9107ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.763+0000 7f558a000700 1 -- 172.21.15.78:0/1674067152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec91085ec0 msgr2=0x55ec910882b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.763+0000 7f558a000700 1 --2- 172.21.15.78:0/1674067152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec91085ec0 0x55ec910882b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.763+0000 7f558a000700 1 -- 172.21.15.78:0/1674067152 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec90e4f3b0 con 0x55ec9107bfa0 2024-03-14T10:01:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.763+0000 7f558aa00700 1 --2- 172.21.15.78:0/1674067152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107b680 0x55ec9107ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.763+0000 7f558b400700 1 --2- 172.21.15.78:0/1674067152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec91085ec0 0x55ec910882b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:16.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.763+0000 7f558a000700 1 --2- 172.21.15.78:0/1674067152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec9107bfa0 0x55ec91085980 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f558c06a3a0 tx=0x7f5578018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f68e4234111853a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.763+0000 7f5577e00700 1 -- 172.21.15.78:0/1674067152 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5578011ad0 con 0x55ec9107bfa0 2024-03-14T10:01:16.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.763+0000 7f5577e00700 1 -- 172.21.15.78:0/1674067152 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5578011c30 con 0x55ec9107bfa0 2024-03-14T10:01:16.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.763+0000 7f5577e00700 1 -- 172.21.15.78:0/1674067152 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557801d690 con 0x55ec9107bfa0 2024-03-14T10:01:16.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.764+0000 7f55a588db80 1 -- 172.21.15.78:0/1674067152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec9107bfa0 msgr2=0x55ec91085980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.764+0000 7f55a588db80 1 --2- 172.21.15.78:0/1674067152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec9107bfa0 0x55ec91085980 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f558c06a3a0 tx=0x7f5578018000 comp rx=0 tx=0).stop 2024-03-14T10:01:16.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.764+0000 7f55a588db80 1 -- 172.21.15.78:0/1674067152 shutdown_connections 2024-03-14T10:01:16.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.764+0000 7f55a588db80 1 --2- 172.21.15.78:0/1674067152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107b680 0x55ec9107ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.764+0000 7f55a588db80 1 --2- 172.21.15.78:0/1674067152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec91085ec0 0x55ec910882b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.764+0000 7f55a588db80 1 --2- 172.21.15.78:0/1674067152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec9107bfa0 0x55ec91085980 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.764+0000 7f55a588db80 1 -- 172.21.15.78:0/1674067152 >> 172.21.15.78:0/1674067152 conn(0x55ec90ff1220 msgr2=0x55ec91076aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.764+0000 7f55a588db80 1 -- 172.21.15.78:0/1674067152 shutdown_connections 2024-03-14T10:01:16.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.764+0000 7f55a588db80 1 -- 172.21.15.78:0/1674067152 wait complete. 2024-03-14T10:01:16.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.765+0000 7f55a588db80 1 -- start start 2024-03-14T10:01:16.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.765+0000 7f55a588db80 1 Processor -- start 2024-03-14T10:01:16.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.765+0000 7f55a588db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec9107b680 0x55ec91082ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.765+0000 7f55a588db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107bfa0 0x55ec910831e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.765+0000 7f55a588db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec91085ec0 0x55ec91083720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.765+0000 7f55a588db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec90f69d80 con 0x55ec9107b680 2024-03-14T10:01:16.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.765+0000 7f55a588db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec90f69a20 con 0x55ec9107bfa0 2024-03-14T10:01:16.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.765+0000 7f55a588db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec90e516a0 con 0x55ec91085ec0 2024-03-14T10:01:16.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.766+0000 7f55a588db80 10 client.0 _ll_register_callbacks cb 0x55ec90dd53a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:16.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.766+0000 7f558aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec9107b680 0x55ec91082ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.766+0000 7f558b400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec91085ec0 0x55ec91083720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.766+0000 7f558aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec9107b680 0x55ec91082ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41450/0 (socket says 172.21.15.78:41450) 2024-03-14T10:01:16.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.766+0000 7f558aa00700 1 -- 172.21.15.78:0/2115389630 learned_addr learned my addr 172.21.15.78:0/2115389630 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:16.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.766+0000 7f558a000700 1 --2- 172.21.15.78:0/2115389630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107bfa0 0x55ec910831e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.766+0000 7f558aa00700 1 -- 172.21.15.78:0/2115389630 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec91085ec0 msgr2=0x55ec91083720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.766+0000 7f558aa00700 1 --2- 172.21.15.78:0/2115389630 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec91085ec0 0x55ec91083720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.766+0000 7f558aa00700 1 -- 172.21.15.78:0/2115389630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107bfa0 msgr2=0x55ec910831e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.766+0000 7f558aa00700 1 --2- 172.21.15.78:0/2115389630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107bfa0 0x55ec910831e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.766+0000 7f558aa00700 1 -- 172.21.15.78:0/2115389630 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558c10e440 con 0x55ec9107b680 2024-03-14T10:01:16.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.766+0000 7f558b400700 1 --2- 172.21.15.78:0/2115389630 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec91085ec0 0x55ec91083720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:16.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.766+0000 7f558aa00700 1 --2- 172.21.15.78:0/2115389630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec9107b680 0x55ec91082ca0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f557c096340 tx=0x7f557c09a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c6b3c8036f5addc server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.766+0000 7f5574c00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:16.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.767+0000 7f5574c00700 1 -- 172.21.15.78:0/2115389630 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c098fc0 con 0x55ec9107b680 2024-03-14T10:01:16.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.767+0000 7f5574c00700 1 -- 172.21.15.78:0/2115389630 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f557c09d700 con 0x55ec9107b680 2024-03-14T10:01:16.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.767+0000 7f55a588db80 1 -- 172.21.15.78:0/2115389630 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ec9106e720 con 0x55ec9107b680 2024-03-14T10:01:16.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.767+0000 7f5574c00700 1 -- 172.21.15.78:0/2115389630 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f557c0b2250 con 0x55ec9107b680 2024-03-14T10:01:16.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.767+0000 7f55a588db80 10 client.17016 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:16.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.767+0000 7f55a588db80 1 -- 172.21.15.78:0/2115389630 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f5578010a00 con 0x55ec9107b680 2024-03-14T10:01:16.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.767+0000 7f5574c00700 1 -- 172.21.15.78:0/2115389630 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ec9106e720 con 0x55ec9107b680 2024-03-14T10:01:16.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.767+0000 7f5574c00700 1 -- 172.21.15.78:0/2115389630 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f557c0b9140 con 0x55ec9107b680 2024-03-14T10:01:16.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.767+0000 7f55a588db80 10 client.17016 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:16.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.767+0000 7f55a588db80 10 client.17016 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:01:16.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.767+0000 7f55a588db80 20 client.17016 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:16.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.767+0000 7f55a588db80 1 --2- 172.21.15.78:0/2115389630 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55ec91095ce0 0x55ec91098100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.767+0000 7f55a588db80 4 client.17016 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:01:16.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.767+0000 7f55a588db80 1 -- 172.21.15.78:0/2115389630 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec90fc6700 con 0x55ec91095ce0 2024-03-14T10:01:16.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.768+0000 7f558a000700 1 --2- 172.21.15.78:0/2115389630 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55ec91095ce0 0x55ec91098100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.768+0000 7f5574c00700 10 client.17016 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:16.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.768+0000 7f558a000700 1 --2- 172.21.15.78:0/2115389630 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55ec91095ce0 0x55ec91098100 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23ff3a31828ecf08 server_cookie=ffddeb190577da77 in_seq=0 out_seq=0 2024-03-14T10:01:16.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.769+0000 7f5574c00700 1 -- 172.21.15.78:0/2115389630 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55ec90fc6700 con 0x55ec91095ce0 2024-03-14T10:01:16.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.769+0000 7f5574c00700 10 client.17016 handle_command_reply: tid=0 2024-03-14T10:01:16.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.769+0000 7f55a588db80 1 client.17016 shutdown 2024-03-14T10:01:16.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.769+0000 7f55a588db80 1 -- 172.21.15.78:0/2115389630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec9107b680 msgr2=0x55ec91082ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.769+0000 7f55a588db80 1 --2- 172.21.15.78:0/2115389630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec9107b680 0x55ec91082ca0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f557c096340 tx=0x7f557c09a050 comp rx=0 tx=0).stop 2024-03-14T10:01:16.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.769+0000 7f55a588db80 1 -- 172.21.15.78:0/2115389630 shutdown_connections 2024-03-14T10:01:16.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.769+0000 7f55a588db80 1 --2- 172.21.15.78:0/2115389630 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec91085ec0 0x55ec91083720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.769+0000 7f55a588db80 1 --2- 172.21.15.78:0/2115389630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec9107b680 0x55ec91082ca0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.769+0000 7f55a588db80 1 --2- 172.21.15.78:0/2115389630 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55ec91095ce0 0x55ec91098100 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.769+0000 7f55a588db80 1 --2- 172.21.15.78:0/2115389630 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107bfa0 0x55ec910831e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.769+0000 7f55a588db80 1 -- 172.21.15.78:0/2115389630 >> 172.21.15.78:0/2115389630 conn(0x55ec90ff1220 msgr2=0x55ec91076690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.769+0000 7f5574c00700 0 client.17016 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:16.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.769+0000 7f55a588db80 1 -- 172.21.15.78:0/2115389630 shutdown_connections 2024-03-14T10:01:16.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.769+0000 7f55a588db80 1 -- 172.21.15.78:0/2115389630 wait complete. 2024-03-14T10:01:16.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.770+0000 7f55a588db80 20 client.17016 trim_cache size 0 max 16384 2024-03-14T10:01:16.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.782+0000 7f55a588db80 1 Processor -- start 2024-03-14T10:01:16.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.783+0000 7f55a588db80 1 -- start start 2024-03-14T10:01:16.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.783+0000 7f55a588db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107b680 0x55ec91076c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.783+0000 7f55a588db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec9107bfa0 0x55ec9109ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.783+0000 7f55a588db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec91085ec0 0x55ec91091120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.783+0000 7f55a588db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec91078060 con 0x55ec91085ec0 2024-03-14T10:01:16.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.783+0000 7f55a588db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec91070d00 con 0x55ec9107b680 2024-03-14T10:01:16.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.783+0000 7f55a588db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec9107d310 con 0x55ec9107bfa0 2024-03-14T10:01:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.783+0000 7f558b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec91085ec0 0x55ec91091120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.783+0000 7f558aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107b680 0x55ec91076c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.783+0000 7f558a000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec9107bfa0 0x55ec9109ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.783+0000 7f558b400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec91085ec0 0x55ec91091120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41454/0 (socket says 172.21.15.78:41454) 2024-03-14T10:01:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.783+0000 7f558b400700 1 -- 172.21.15.78:0/3793979727 learned_addr learned my addr 172.21.15.78:0/3793979727 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.784+0000 7f558a000700 1 -- 172.21.15.78:0/3793979727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107b680 msgr2=0x55ec91076c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.784+0000 7f558a000700 1 --2- 172.21.15.78:0/3793979727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107b680 0x55ec91076c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.784+0000 7f558a000700 1 -- 172.21.15.78:0/3793979727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec91085ec0 msgr2=0x55ec91091120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.784+0000 7f558a000700 1 --2- 172.21.15.78:0/3793979727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec91085ec0 0x55ec91091120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.784+0000 7f558a000700 1 -- 172.21.15.78:0/3793979727 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec90dfbb60 con 0x55ec9107bfa0 2024-03-14T10:01:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.784+0000 7f558a000700 1 --2- 172.21.15.78:0/3793979727 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec9107bfa0 0x55ec9109ca60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x55ec910841e0 tx=0x7f5578019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2053aa782895e4d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.784+0000 7f5581200700 1 -- 172.21.15.78:0/3793979727 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557802b040 con 0x55ec9107bfa0 2024-03-14T10:01:16.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.784+0000 7f5581200700 1 -- 172.21.15.78:0/3793979727 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5578036070 con 0x55ec9107bfa0 2024-03-14T10:01:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.784+0000 7f5581200700 1 -- 172.21.15.78:0/3793979727 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5578032be0 con 0x55ec9107bfa0 2024-03-14T10:01:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.785+0000 7f55a588db80 1 -- 172.21.15.78:0/3793979727 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec9107bfa0 msgr2=0x55ec9109ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.785+0000 7f55a588db80 1 --2- 172.21.15.78:0/3793979727 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec9107bfa0 0x55ec9109ca60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x55ec910841e0 tx=0x7f5578019000 comp rx=0 tx=0).stop 2024-03-14T10:01:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.785+0000 7f55a588db80 1 -- 172.21.15.78:0/3793979727 shutdown_connections 2024-03-14T10:01:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.785+0000 7f55a588db80 1 --2- 172.21.15.78:0/3793979727 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107b680 0x55ec91076c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.785+0000 7f55a588db80 1 --2- 172.21.15.78:0/3793979727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec91085ec0 0x55ec91091120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.785+0000 7f55a588db80 1 --2- 172.21.15.78:0/3793979727 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec9107bfa0 0x55ec9109ca60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.785+0000 7f55a588db80 1 -- 172.21.15.78:0/3793979727 >> 172.21.15.78:0/3793979727 conn(0x55ec90ff1220 msgr2=0x55ec9107df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.785+0000 7f55a588db80 1 -- 172.21.15.78:0/3793979727 shutdown_connections 2024-03-14T10:01:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.785+0000 7f55a588db80 1 -- 172.21.15.78:0/3793979727 wait complete. 2024-03-14T10:01:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.785+0000 7f55a588db80 1 -- start start 2024-03-14T10:01:16.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.785+0000 7f55a588db80 1 Processor -- start 2024-03-14T10:01:16.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f55a588db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec9107b680 0x55ec9108ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f55a588db80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107bfa0 0x55ec9108b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f55a588db80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec91085ec0 0x55ec9108b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f55a588db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec910e6980 con 0x55ec9107b680 2024-03-14T10:01:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f55a588db80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec910e6af0 con 0x55ec9107bfa0 2024-03-14T10:01:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f55a588db80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec910e6cb0 con 0x55ec91085ec0 2024-03-14T10:01:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f55a588db80 10 client.0 _ll_register_callbacks cb 0x7f557c0994e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f558aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec9107b680 0x55ec9108ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f558aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec9107b680 0x55ec9108ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41470/0 (socket says 172.21.15.78:41470) 2024-03-14T10:01:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f558aa00700 1 -- 172.21.15.78:0/3719488815 learned_addr learned my addr 172.21.15.78:0/3719488815 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f558a000700 1 --2- 172.21.15.78:0/3719488815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107bfa0 0x55ec9108b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f558b400700 1 --2- 172.21.15.78:0/3719488815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec91085ec0 0x55ec9108b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f558aa00700 1 -- 172.21.15.78:0/3719488815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec91085ec0 msgr2=0x55ec9108b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f558aa00700 1 --2- 172.21.15.78:0/3719488815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec91085ec0 0x55ec9108b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f558aa00700 1 -- 172.21.15.78:0/3719488815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107bfa0 msgr2=0x55ec9108b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f558aa00700 1 --2- 172.21.15.78:0/3719488815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107bfa0 0x55ec9108b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.786+0000 7f558aa00700 1 -- 172.21.15.78:0/3719488815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec90e4f3b0 con 0x55ec9107b680 2024-03-14T10:01:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.787+0000 7f558aa00700 1 --2- 172.21.15.78:0/3719488815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec9107b680 0x55ec9108ab30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f557c095dd0 tx=0x7f557c0b92e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2cbf55e0d87d25b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.787+0000 7f5574c00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.787+0000 7f5574c00700 1 -- 172.21.15.78:0/3719488815 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c098c10 con 0x55ec9107b680 2024-03-14T10:01:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.787+0000 7f5574c00700 1 -- 172.21.15.78:0/3719488815 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f557c0b2250 con 0x55ec9107b680 2024-03-14T10:01:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.787+0000 7f55a588db80 1 -- 172.21.15.78:0/3719488815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5578011c30 con 0x55ec9107b680 2024-03-14T10:01:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.787+0000 7f5574c00700 1 -- 172.21.15.78:0/3719488815 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f557c09d700 con 0x55ec9107b680 2024-03-14T10:01:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.787+0000 7f55a588db80 10 client.17022 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.787+0000 7f55a588db80 1 -- 172.21.15.78:0/3719488815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ec90f93ea0 con 0x55ec9107b680 2024-03-14T10:01:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.787+0000 7f5574c00700 1 -- 172.21.15.78:0/3719488815 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f557c0a5080 con 0x55ec9107b680 2024-03-14T10:01:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.787+0000 7f5574c00700 1 -- 172.21.15.78:0/3719488815 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f557c07f030 con 0x55ec9107b680 2024-03-14T10:01:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.787+0000 7f55a588db80 10 client.17022 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.787+0000 7f55a588db80 10 client.17022 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:01:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.787+0000 7f55a588db80 20 client.17022 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.787+0000 7f55a588db80 1 --2- 172.21.15.78:0/3719488815 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55ec910e80c0 0x55ec910ea4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:16.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.788+0000 7f55a588db80 4 client.17022 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:01:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.788+0000 7f55a588db80 1 -- 172.21.15.78:0/3719488815 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ec90e1ccf0 con 0x55ec910e80c0 2024-03-14T10:01:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.788+0000 7f558a000700 1 --2- 172.21.15.78:0/3719488815 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55ec910e80c0 0x55ec910ea4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.788+0000 7f558a000700 1 --2- 172.21.15.78:0/3719488815 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55ec910e80c0 0x55ec910ea4e0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d8ab2f9e3b006ff server_cookie=3193ba686d1811dc in_seq=0 out_seq=0 2024-03-14T10:01:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.788+0000 7f5574c00700 10 client.17022 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.789+0000 7f5574c00700 1 -- 172.21.15.78:0/3719488815 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ec90e1ccf0 con 0x55ec910e80c0 2024-03-14T10:01:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.789+0000 7f5574c00700 10 client.17022 handle_command_reply: tid=0 2024-03-14T10:01:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.789+0000 7f55a588db80 1 client.17022 shutdown 2024-03-14T10:01:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.789+0000 7f55a588db80 1 -- 172.21.15.78:0/3719488815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec9107b680 msgr2=0x55ec9108ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.789+0000 7f55a588db80 1 --2- 172.21.15.78:0/3719488815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec9107b680 0x55ec9108ab30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f557c095dd0 tx=0x7f557c0b92e0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.789+0000 7f55a588db80 1 -- 172.21.15.78:0/3719488815 shutdown_connections 2024-03-14T10:01:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.789+0000 7f55a588db80 1 --2- 172.21.15.78:0/3719488815 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ec91085ec0 0x55ec9108b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.789+0000 7f55a588db80 1 --2- 172.21.15.78:0/3719488815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec9107b680 0x55ec9108ab30 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.789+0000 7f55a588db80 1 --2- 172.21.15.78:0/3719488815 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55ec910e80c0 0x55ec910ea4e0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.789+0000 7f55a588db80 1 --2- 172.21.15.78:0/3719488815 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ec9107bfa0 0x55ec9108b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.789+0000 7f5574c00700 0 client.17022 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.789+0000 7f55a588db80 1 -- 172.21.15.78:0/3719488815 >> 172.21.15.78:0/3719488815 conn(0x55ec90ff1220 msgr2=0x55ec91091120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.789+0000 7f55a588db80 1 -- 172.21.15.78:0/3719488815 shutdown_connections 2024-03-14T10:01:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.789+0000 7f55a588db80 1 -- 172.21.15.78:0/3719488815 wait complete. 2024-03-14T10:01:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.789+0000 7f55a588db80 20 client.17022 trim_cache size 0 max 16384 2024-03-14T10:01:16.793 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:01:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.790+0000 7f5591e00700 1 -- 172.21.15.78:0/4196572922 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f55700a0c80 msgr2=0x7f55700a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.790+0000 7f5591e00700 1 --2- 172.21.15.78:0/4196572922 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f55700a0c80 0x7f55700a3140 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f558c19ae30 tx=0x7f557800b410 comp rx=0 tx=0).stop 2024-03-14T10:01:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.790+0000 7f5591e00700 1 -- 172.21.15.78:0/4196572922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f558c100f30 msgr2=0x7f558c19f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.790+0000 7f5591e00700 1 --2- 172.21.15.78:0/4196572922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f558c100f30 0x7f558c19f5f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f557c00e9a0 tx=0x7f557c00ecb0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.790+0000 7f5591e00700 1 -- 172.21.15.78:0/4196572922 shutdown_connections 2024-03-14T10:01:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.790+0000 7f5591e00700 1 --2- 172.21.15.78:0/4196572922 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f55700a0c80 0x7f55700a3140 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.790+0000 7f5591e00700 1 --2- 172.21.15.78:0/4196572922 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f558c199720 0x7f558c199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.790+0000 7f5591e00700 1 --2- 172.21.15.78:0/4196572922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f558c100f30 0x7f558c19f5f0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.790+0000 7f5591e00700 1 --2- 172.21.15.78:0/4196572922 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f558c106050 0x7f558c19fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.790+0000 7f5591e00700 1 -- 172.21.15.78:0/4196572922 >> 172.21.15.78:0/4196572922 conn(0x7f558c0fc7b0 msgr2=0x7f558c10d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.790+0000 7f5591e00700 1 -- 172.21.15.78:0/4196572922 shutdown_connections 2024-03-14T10:01:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:16.790+0000 7f5591e00700 1 -- 172.21.15.78:0/4196572922 wait complete. 2024-03-14T10:01:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:17 smithi152 ceph-mon[48279]: pgmap v218: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-03-14T10:01:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:17 smithi132 ceph-mon[47959]: pgmap v218: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-03-14T10:01:17.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:17 smithi078 ceph-mon[38143]: pgmap v218: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-03-14T10:01:17.809 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T10:01:17.809 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T10:01:18.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.020+0000 7f2370a00700 1 -- 172.21.15.78:0/3153291619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c106040 msgr2=0x7f236c10dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.020+0000 7f2370a00700 1 --2- 172.21.15.78:0/3153291619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c106040 0x7f236c10dbc0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f236400b0a0 tx=0x7f236400b3b0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.020+0000 7f2370a00700 1 -- 172.21.15.78:0/3153291619 shutdown_connections 2024-03-14T10:01:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.020+0000 7f2370a00700 1 --2- 172.21.15.78:0/3153291619 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c106040 0x7f236c10dbc0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.020+0000 7f2370a00700 1 --2- 172.21.15.78:0/3153291619 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f236c100f20 0x7f236c101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.020+0000 7f2370a00700 1 --2- 172.21.15.78:0/3153291619 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f236c1018d0 0x7f236c105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.020+0000 7f2370a00700 1 -- 172.21.15.78:0/3153291619 >> 172.21.15.78:0/3153291619 conn(0x7f236c0fc7a0 msgr2=0x7f236c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.020+0000 7f2370a00700 1 -- 172.21.15.78:0/3153291619 shutdown_connections 2024-03-14T10:01:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.020+0000 7f2370a00700 1 -- 172.21.15.78:0/3153291619 wait complete. 2024-03-14T10:01:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.021+0000 7f2370a00700 1 Processor -- start 2024-03-14T10:01:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.021+0000 7f2370a00700 1 -- start start 2024-03-14T10:01:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.022+0000 7f2370a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f236c100f20 0x7f236c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.022+0000 7f2370a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c106040 0x7f236c0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.022+0000 7f2370a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f236c073b30 0x7f236c1a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.022+0000 7f2370a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c1a5f50 con 0x7f236c106040 2024-03-14T10:01:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.022+0000 7f2370a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c1a60c0 con 0x7f236c073b30 2024-03-14T10:01:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.022+0000 7f2370a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c1a6280 con 0x7f236c100f20 2024-03-14T10:01:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.022+0000 7f2363e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c106040 0x7f236c0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.022+0000 7f2363e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c106040 0x7f236c0735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41496/0 (socket says 172.21.15.78:41496) 2024-03-14T10:01:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.022+0000 7f2369e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f236c100f20 0x7f236c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.022+0000 7f2363e00700 1 -- 172.21.15.78:0/367044578 learned_addr learned my addr 172.21.15.78:0/367044578 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.022+0000 7f236a800700 1 --2- 172.21.15.78:0/367044578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f236c073b30 0x7f236c1a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.022+0000 7f2363e00700 1 -- 172.21.15.78:0/367044578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f236c100f20 msgr2=0x7f236c0730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.022+0000 7f2363e00700 1 --2- 172.21.15.78:0/367044578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f236c100f20 0x7f236c0730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.022+0000 7f2363e00700 1 -- 172.21.15.78:0/367044578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f236c073b30 msgr2=0x7f236c1a5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.022+0000 7f2363e00700 1 --2- 172.21.15.78:0/367044578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f236c073b30 0x7f236c1a5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.022+0000 7f2363e00700 1 -- 172.21.15.78:0/367044578 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2364009d00 con 0x7f236c106040 2024-03-14T10:01:18.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.023+0000 7f2363e00700 1 --2- 172.21.15.78:0/367044578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c106040 0x7f236c0735f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f235400e970 tx=0x7f235400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:18.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.023+0000 7f2362a00700 1 -- 172.21.15.78:0/367044578 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2354009790 con 0x7f236c106040 2024-03-14T10:01:18.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.023+0000 7f2362a00700 1 -- 172.21.15.78:0/367044578 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f235400c7e0 con 0x7f236c106040 2024-03-14T10:01:18.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.023+0000 7f2370a00700 1 -- 172.21.15.78:0/367044578 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236c1a6560 con 0x7f236c106040 2024-03-14T10:01:18.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.023+0000 7f2362a00700 1 -- 172.21.15.78:0/367044578 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23540054e0 con 0x7f236c106040 2024-03-14T10:01:18.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.023+0000 7f2370a00700 1 -- 172.21.15.78:0/367044578 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236c1a6a70 con 0x7f236c106040 2024-03-14T10:01:18.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.026+0000 7f2362a00700 1 -- 172.21.15.78:0/367044578 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f235400c950 con 0x7f236c106040 2024-03-14T10:01:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.027+0000 7f2362a00700 1 --2- 172.21.15.78:0/367044578 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f23500a0c00 0x7f23500a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.027+0000 7f23844cbb80 1 Processor -- start 2024-03-14T10:01:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.027+0000 7f2362a00700 1 -- 172.21.15.78:0/367044578 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f2354014070 con 0x7f236c106040 2024-03-14T10:01:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.027+0000 7f2369e00700 1 --2- 172.21.15.78:0/367044578 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f23500a0c00 0x7f23500a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.028+0000 7f23844cbb80 1 -- start start 2024-03-14T10:01:18.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.028+0000 7f23844cbb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135baf700 0x55f135bafae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.028+0000 7f23844cbb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135bb0020 0x55f135bb9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.028+0000 7f23844cbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 0x55f135bbc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.028+0000 7f23844cbb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f135b08a10 con 0x55f135bb0020 2024-03-14T10:01:18.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.028+0000 7f23844cbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f135b063d0 con 0x55f135bb9f40 2024-03-14T10:01:18.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.028+0000 7f23844cbb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f135b01cc0 con 0x55f135baf700 2024-03-14T10:01:18.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.028+0000 7f2369e00700 1 --2- 172.21.15.78:0/367044578 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f23500a0c00 0x7f23500a30c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f235c009c60 tx=0x7f235c009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:18.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.028+0000 7f2363e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135bb0020 0x55f135bb9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.028+0000 7f2369e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135baf700 0x55f135bafae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.028+0000 7f2363e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135bb0020 0x55f135bb9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41512/0 (socket says 172.21.15.78:41512) 2024-03-14T10:01:18.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.028+0000 7f2363e00700 1 -- 172.21.15.78:0/2657220899 learned_addr learned my addr 172.21.15.78:0/2657220899 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:18.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.028+0000 7f236a800700 1 --2- 172.21.15.78:0/2657220899 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 0x55f135bbc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.029+0000 7f2363e00700 1 -- 172.21.15.78:0/2657220899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135baf700 msgr2=0x55f135bafae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:18.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.029+0000 7f2363e00700 1 --2- 172.21.15.78:0/2657220899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135baf700 0x55f135bafae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.029+0000 7f2363e00700 1 -- 172.21.15.78:0/2657220899 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 msgr2=0x55f135bbc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:18.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.029+0000 7f2363e00700 1 --2- 172.21.15.78:0/2657220899 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 0x55f135bbc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.029+0000 7f2363e00700 1 -- 172.21.15.78:0/2657220899 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f135983430 con 0x55f135bb0020 2024-03-14T10:01:18.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.029+0000 7f2363e00700 1 --2- 172.21.15.78:0/2657220899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135bb0020 0x55f135bb9a00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f23540963c0 tx=0x7f235400ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef76f3ef174bccf server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:18.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.029+0000 7f235b000700 1 -- 172.21.15.78:0/2657220899 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2354099950 con 0x55f135bb0020 2024-03-14T10:01:18.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.029+0000 7f235b000700 1 -- 172.21.15.78:0/2657220899 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f235409d780 con 0x55f135bb0020 2024-03-14T10:01:18.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.029+0000 7f235b000700 1 -- 172.21.15.78:0/2657220899 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2354016330 con 0x55f135bb0020 2024-03-14T10:01:18.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.030+0000 7f23844cbb80 1 -- 172.21.15.78:0/2657220899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135bb0020 msgr2=0x55f135bb9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.030+0000 7f23844cbb80 1 --2- 172.21.15.78:0/2657220899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135bb0020 0x55f135bb9a00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f23540963c0 tx=0x7f235400ccd0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.030+0000 7f23844cbb80 1 -- 172.21.15.78:0/2657220899 shutdown_connections 2024-03-14T10:01:18.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.030+0000 7f23844cbb80 1 --2- 172.21.15.78:0/2657220899 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 0x55f135bbc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.030+0000 7f23844cbb80 1 --2- 172.21.15.78:0/2657220899 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135baf700 0x55f135bafae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.030+0000 7f23844cbb80 1 --2- 172.21.15.78:0/2657220899 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135bb0020 0x55f135bb9a00 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.030+0000 7f23844cbb80 1 -- 172.21.15.78:0/2657220899 >> 172.21.15.78:0/2657220899 conn(0x55f135b252a0 msgr2=0x55f135baab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:18.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.030+0000 7f23844cbb80 1 -- 172.21.15.78:0/2657220899 shutdown_connections 2024-03-14T10:01:18.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.030+0000 7f23844cbb80 1 -- 172.21.15.78:0/2657220899 wait complete. 2024-03-14T10:01:18.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.031+0000 7f23844cbb80 1 -- start start 2024-03-14T10:01:18.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.031+0000 7f23844cbb80 1 Processor -- start 2024-03-14T10:01:18.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.031+0000 7f23844cbb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135baf700 0x55f135bb9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.031+0000 7f23844cbb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135bb0020 0x55f135bb6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.031+0000 7f23844cbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 0x55f135bb7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.031+0000 7f23844cbb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f135a9de00 con 0x55f135baf700 2024-03-14T10:01:18.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.031+0000 7f23844cbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f135a9daa0 con 0x55f135bb9f40 2024-03-14T10:01:18.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.031+0000 7f23844cbb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f135985720 con 0x55f135bb0020 2024-03-14T10:01:18.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.031+0000 7f23844cbb80 10 client.0 _ll_register_callbacks cb 0x55f135909420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:18.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.031+0000 7f2369e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135baf700 0x55f135bb9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.031+0000 7f2369e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135baf700 0x55f135bb9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41520/0 (socket says 172.21.15.78:41520) 2024-03-14T10:01:18.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.032+0000 7f2369e00700 1 -- 172.21.15.78:0/570060012 learned_addr learned my addr 172.21.15.78:0/570060012 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:18.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.032+0000 7f2363e00700 1 --2- 172.21.15.78:0/570060012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135bb0020 0x55f135bb6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.032+0000 7f236a800700 1 --2- 172.21.15.78:0/570060012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 0x55f135bb7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.032+0000 7f2369e00700 1 -- 172.21.15.78:0/570060012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135bb0020 msgr2=0x55f135bb6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:18.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.032+0000 7f2369e00700 1 --2- 172.21.15.78:0/570060012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135bb0020 0x55f135bb6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.032+0000 7f2369e00700 1 -- 172.21.15.78:0/570060012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 msgr2=0x55f135bb7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:18.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.032+0000 7f2369e00700 1 --2- 172.21.15.78:0/570060012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 0x55f135bb7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.032+0000 7f2369e00700 1 -- 172.21.15.78:0/570060012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236c1a6a70 con 0x55f135baf700 2024-03-14T10:01:18.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.032+0000 7f236a800700 1 --2- 172.21.15.78:0/570060012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 0x55f135bb7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T10:01:18.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.032+0000 7f2369e00700 1 --2- 172.21.15.78:0/570060012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135baf700 0x55f135bb9a00 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f235c003900 tx=0x7f235c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed6ab17c39c24d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:18.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.032+0000 7f234f400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:18.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.032+0000 7f234f400700 1 -- 172.21.15.78:0/570060012 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c022070 con 0x55f135baf700 2024-03-14T10:01:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.032+0000 7f23844cbb80 1 -- 172.21.15.78:0/570060012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f135ba27a0 con 0x55f135baf700 2024-03-14T10:01:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.032+0000 7f234f400700 1 -- 172.21.15.78:0/570060012 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f235c01c430 con 0x55f135baf700 2024-03-14T10:01:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.033+0000 7f234f400700 1 -- 172.21.15.78:0/570060012 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f235c01c5f0 con 0x55f135baf700 2024-03-14T10:01:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.033+0000 7f23844cbb80 10 client.17034 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.033+0000 7f23844cbb80 1 -- 172.21.15.78:0/570060012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f13592fbe0 con 0x55f135baf700 2024-03-14T10:01:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.033+0000 7f234f400700 1 -- 172.21.15.78:0/570060012 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f135ba27a0 con 0x55f135baf700 2024-03-14T10:01:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.033+0000 7f234f400700 1 -- 172.21.15.78:0/570060012 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f235c027980 con 0x55f135baf700 2024-03-14T10:01:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.033+0000 7f23844cbb80 10 client.17034 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.033+0000 7f23844cbb80 10 client.17034 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:01:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.033+0000 7f23844cbb80 20 client.17034 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.033+0000 7f23844cbb80 1 --2- 172.21.15.78:0/570060012 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55f135bc9c70 0x55f135bcc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.033+0000 7f23844cbb80 4 client.17034 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:01:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.033+0000 7f23844cbb80 1 -- 172.21.15.78:0/570060012 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f135afa780 con 0x55f135bc9c70 2024-03-14T10:01:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.034+0000 7f2363e00700 1 --2- 172.21.15.78:0/570060012 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55f135bc9c70 0x55f135bcc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.034+0000 7f2363e00700 1 --2- 172.21.15.78:0/570060012 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55f135bc9c70 0x55f135bcc090 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d2a37128863c902 server_cookie=7e585560393ad93d in_seq=0 out_seq=0 2024-03-14T10:01:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.034+0000 7f234f400700 10 client.17034 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.035+0000 7f234f400700 1 -- 172.21.15.78:0/570060012 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55f135afa780 con 0x55f135bc9c70 2024-03-14T10:01:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.035+0000 7f234f400700 10 client.17034 handle_command_reply: tid=0 2024-03-14T10:01:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.035+0000 7f23844cbb80 1 client.17034 shutdown 2024-03-14T10:01:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.035+0000 7f23844cbb80 1 -- 172.21.15.78:0/570060012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135baf700 msgr2=0x55f135bb9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.035+0000 7f23844cbb80 1 --2- 172.21.15.78:0/570060012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135baf700 0x55f135bb9a00 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f235c003900 tx=0x7f235c0047c0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.035+0000 7f23844cbb80 1 -- 172.21.15.78:0/570060012 shutdown_connections 2024-03-14T10:01:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.035+0000 7f23844cbb80 1 --2- 172.21.15.78:0/570060012 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55f135bc9c70 0x55f135bcc090 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.035+0000 7f23844cbb80 1 --2- 172.21.15.78:0/570060012 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 0x55f135bb7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.035+0000 7f23844cbb80 1 --2- 172.21.15.78:0/570060012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135baf700 0x55f135bb9a00 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.035+0000 7f23844cbb80 1 --2- 172.21.15.78:0/570060012 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135bb0020 0x55f135bb6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.035+0000 7f23844cbb80 1 -- 172.21.15.78:0/570060012 >> 172.21.15.78:0/570060012 conn(0x55f135b252a0 msgr2=0x55f135baa710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:18.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.035+0000 7f234f400700 0 client.17034 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:18.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.035+0000 7f23844cbb80 1 -- 172.21.15.78:0/570060012 shutdown_connections 2024-03-14T10:01:18.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.035+0000 7f23844cbb80 1 -- 172.21.15.78:0/570060012 wait complete. 2024-03-14T10:01:18.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.036+0000 7f23844cbb80 20 client.17034 trim_cache size 0 max 16384 2024-03-14T10:01:18.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.052+0000 7f23844cbb80 1 Processor -- start 2024-03-14T10:01:18.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.052+0000 7f23844cbb80 1 -- start start 2024-03-14T10:01:18.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.053+0000 7f23844cbb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135baf700 0x55f135bc0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.053+0000 7f23844cbb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135bb0020 0x55f135bb5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.053+0000 7f23844cbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 0x55f135bb9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.053+0000 7f23844cbb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f135babeb0 con 0x55f135baf700 2024-03-14T10:01:18.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.053+0000 7f23844cbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f135bac0e0 con 0x55f135bb9f40 2024-03-14T10:01:18.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.053+0000 7f23844cbb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f135bafd30 con 0x55f135bb0020 2024-03-14T10:01:18.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.053+0000 7f2369e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135baf700 0x55f135bc0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.053+0000 7f2369e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135baf700 0x55f135bc0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41532/0 (socket says 172.21.15.78:41532) 2024-03-14T10:01:18.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.053+0000 7f2369e00700 1 -- 172.21.15.78:0/605932243 learned_addr learned my addr 172.21.15.78:0/605932243 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:18.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.053+0000 7f2363e00700 1 --2- 172.21.15.78:0/605932243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135bb0020 0x55f135bb5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.053+0000 7f236a800700 1 --2- 172.21.15.78:0/605932243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 0x55f135bb9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.053+0000 7f2369e00700 1 -- 172.21.15.78:0/605932243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135bb0020 msgr2=0x55f135bb5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:18.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.053+0000 7f2369e00700 1 --2- 172.21.15.78:0/605932243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135bb0020 0x55f135bb5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.054+0000 7f2369e00700 1 -- 172.21.15.78:0/605932243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 msgr2=0x55f135bb9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:18.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.054+0000 7f2369e00700 1 --2- 172.21.15.78:0/605932243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 0x55f135bb9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.054+0000 7f2369e00700 1 -- 172.21.15.78:0/605932243 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f135ac7f20 con 0x55f135baf700 2024-03-14T10:01:18.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.054+0000 7f236a800700 1 --2- 172.21.15.78:0/605932243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 0x55f135bb9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T10:01:18.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.054+0000 7f2369e00700 1 --2- 172.21.15.78:0/605932243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135baf700 0x55f135bc0bd0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f235c0097c0 tx=0x7f235c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fa59c3404cf7dcc server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:18.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.054+0000 7f235ba00700 1 -- 172.21.15.78:0/605932243 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c038070 con 0x55f135baf700 2024-03-14T10:01:18.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.054+0000 7f235ba00700 1 -- 172.21.15.78:0/605932243 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f235c0278e0 con 0x55f135baf700 2024-03-14T10:01:18.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.054+0000 7f235ba00700 1 -- 172.21.15.78:0/605932243 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c022040 con 0x55f135baf700 2024-03-14T10:01:18.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.055+0000 7f23844cbb80 1 -- 172.21.15.78:0/605932243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135baf700 msgr2=0x55f135bc0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.055+0000 7f23844cbb80 1 --2- 172.21.15.78:0/605932243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135baf700 0x55f135bc0bd0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f235c0097c0 tx=0x7f235c005880 comp rx=0 tx=0).stop 2024-03-14T10:01:18.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.055+0000 7f23844cbb80 1 -- 172.21.15.78:0/605932243 shutdown_connections 2024-03-14T10:01:18.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.055+0000 7f23844cbb80 1 --2- 172.21.15.78:0/605932243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb9f40 0x55f135bb9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.055+0000 7f23844cbb80 1 --2- 172.21.15.78:0/605932243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135baf700 0x55f135bc0bd0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.055+0000 7f23844cbb80 1 --2- 172.21.15.78:0/605932243 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135bb0020 0x55f135bb5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.055+0000 7f23844cbb80 1 -- 172.21.15.78:0/605932243 >> 172.21.15.78:0/605932243 conn(0x55f135b252a0 msgr2=0x55f135bb55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:18.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.055+0000 7f23844cbb80 1 -- 172.21.15.78:0/605932243 shutdown_connections 2024-03-14T10:01:18.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.055+0000 7f23844cbb80 1 -- 172.21.15.78:0/605932243 wait complete. 2024-03-14T10:01:18.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.055+0000 7f23844cbb80 1 -- start start 2024-03-14T10:01:18.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.055+0000 7f23844cbb80 1 Processor -- start 2024-03-14T10:01:18.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f23844cbb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135baf700 0x55f135bbfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f23844cbb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb0020 0x55f135bc0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f23844cbb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135bb9f40 0x55f135c157e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f23844cbb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f135c15d20 con 0x55f135bb9f40 2024-03-14T10:01:18.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f23844cbb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f135c15e90 con 0x55f135bb0020 2024-03-14T10:01:18.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f23844cbb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f135c16050 con 0x55f135baf700 2024-03-14T10:01:18.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f23844cbb80 10 client.0 _ll_register_callbacks cb 0x55f135909420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:18.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f236a800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135bb9f40 0x55f135c157e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f2369e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135baf700 0x55f135bbfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f236a800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135bb9f40 0x55f135c157e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41540/0 (socket says 172.21.15.78:41540) 2024-03-14T10:01:18.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f236a800700 1 -- 172.21.15.78:0/2505559877 learned_addr learned my addr 172.21.15.78:0/2505559877 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:18.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f2363e00700 1 --2- 172.21.15.78:0/2505559877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb0020 0x55f135bc0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f236a800700 1 -- 172.21.15.78:0/2505559877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135baf700 msgr2=0x55f135bbfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:18.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f236a800700 1 --2- 172.21.15.78:0/2505559877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135baf700 0x55f135bbfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f236a800700 1 -- 172.21.15.78:0/2505559877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb0020 msgr2=0x55f135bc0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:18.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f236a800700 1 --2- 172.21.15.78:0/2505559877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb0020 0x55f135bc0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f236a800700 1 -- 172.21.15.78:0/2505559877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f235c02c040 con 0x55f135bb9f40 2024-03-14T10:01:18.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.056+0000 7f2369e00700 1 --2- 172.21.15.78:0/2505559877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135baf700 0x55f135bbfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:18.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.057+0000 7f236a800700 1 --2- 172.21.15.78:0/2505559877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135bb9f40 0x55f135c157e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f236400b0a0 tx=0x7f23640095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef0895987d22a00 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:18.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.057+0000 7f234f400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:18.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.057+0000 7f234f400700 1 -- 172.21.15.78:0/2505559877 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236400e040 con 0x55f135bb9f40 2024-03-14T10:01:18.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.057+0000 7f234f400700 1 -- 172.21.15.78:0/2505559877 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f23640076e0 con 0x55f135bb9f40 2024-03-14T10:01:18.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.057+0000 7f23844cbb80 1 -- 172.21.15.78:0/2505559877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f235409d780 con 0x55f135bb9f40 2024-03-14T10:01:18.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.057+0000 7f234f400700 1 -- 172.21.15.78:0/2505559877 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23640078a0 con 0x55f135bb9f40 2024-03-14T10:01:18.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.057+0000 7f23844cbb80 10 client.17040 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:18.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.057+0000 7f23844cbb80 1 -- 172.21.15.78:0/2505559877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f135987d50 con 0x55f135bb9f40 2024-03-14T10:01:18.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.057+0000 7f234f400700 1 -- 172.21.15.78:0/2505559877 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f235409d780 con 0x55f135bb9f40 2024-03-14T10:01:18.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.058+0000 7f234f400700 1 -- 172.21.15.78:0/2505559877 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2364023020 con 0x55f135bb9f40 2024-03-14T10:01:18.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.058+0000 7f23844cbb80 10 client.17040 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:18.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.058+0000 7f23844cbb80 10 client.17040 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:01:18.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.058+0000 7f23844cbb80 20 client.17040 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:18.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.058+0000 7f23844cbb80 1 --2- 172.21.15.78:0/2505559877 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55f135c1d870 0x55f135c1fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.058+0000 7f23844cbb80 4 client.17040 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T10:01:18.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.058+0000 7f23844cbb80 1 -- 172.21.15.78:0/2505559877 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f135950d70 con 0x55f135c1d870 2024-03-14T10:01:18.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.058+0000 7f2369e00700 1 --2- 172.21.15.78:0/2505559877 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55f135c1d870 0x55f135c1fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.058+0000 7f234f400700 10 client.17040 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:18.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.058+0000 7f2369e00700 1 --2- 172.21.15.78:0/2505559877 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55f135c1d870 0x55f135c1fc90 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f821a85072dfa96d server_cookie=7b7b71d95cf2fc47 in_seq=0 out_seq=0 2024-03-14T10:01:18.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.059+0000 7f234f400700 1 -- 172.21.15.78:0/2505559877 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f135950d70 con 0x55f135c1d870 2024-03-14T10:01:18.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.060+0000 7f234f400700 10 client.17040 handle_command_reply: tid=0 2024-03-14T10:01:18.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.060+0000 7f23844cbb80 1 client.17040 shutdown 2024-03-14T10:01:18.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.060+0000 7f23844cbb80 1 -- 172.21.15.78:0/2505559877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135bb9f40 msgr2=0x55f135c157e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.060+0000 7f23844cbb80 1 --2- 172.21.15.78:0/2505559877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135bb9f40 0x55f135c157e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f236400b0a0 tx=0x7f23640095c0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.060+0000 7f23844cbb80 1 -- 172.21.15.78:0/2505559877 shutdown_connections 2024-03-14T10:01:18.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.060+0000 7f23844cbb80 1 --2- 172.21.15.78:0/2505559877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f135bb9f40 0x55f135c157e0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.060+0000 7f23844cbb80 1 --2- 172.21.15.78:0/2505559877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f135baf700 0x55f135bbfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.060+0000 7f23844cbb80 1 --2- 172.21.15.78:0/2505559877 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55f135c1d870 0x55f135c1fc90 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.060+0000 7f23844cbb80 1 --2- 172.21.15.78:0/2505559877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f135bb0020 0x55f135bc0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.060+0000 7f23844cbb80 1 -- 172.21.15.78:0/2505559877 >> 172.21.15.78:0/2505559877 conn(0x55f135b252a0 msgr2=0x55f135ba9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:18.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.060+0000 7f234f400700 0 client.17040 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:18.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.060+0000 7f23844cbb80 1 -- 172.21.15.78:0/2505559877 shutdown_connections 2024-03-14T10:01:18.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.060+0000 7f23844cbb80 1 -- 172.21.15.78:0/2505559877 wait complete. 2024-03-14T10:01:18.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.060+0000 7f23844cbb80 20 client.17040 trim_cache size 0 max 16384 2024-03-14T10:01:18.064 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:01:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T10:01:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "c5a28d5f-0723-423b-9254-81cc176b727c", 2024-03-14T10:01:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T10:01:18.065 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:01:18.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.061+0000 7f2370a00700 1 -- 172.21.15.78:0/367044578 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f23500a0c00 msgr2=0x7f23500a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.061+0000 7f2370a00700 1 --2- 172.21.15.78:0/367044578 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f23500a0c00 0x7f23500a30c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f235c009c60 tx=0x7f235c009260 comp rx=0 tx=0).stop 2024-03-14T10:01:18.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.061+0000 7f2370a00700 1 -- 172.21.15.78:0/367044578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c106040 msgr2=0x7f236c0735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.061+0000 7f2370a00700 1 --2- 172.21.15.78:0/367044578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c106040 0x7f236c0735f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f235400e970 tx=0x7f235400ec80 comp rx=0 tx=0).stop 2024-03-14T10:01:18.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.061+0000 7f2370a00700 1 -- 172.21.15.78:0/367044578 shutdown_connections 2024-03-14T10:01:18.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.061+0000 7f2370a00700 1 --2- 172.21.15.78:0/367044578 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f23500a0c00 0x7f23500a30c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.061+0000 7f2370a00700 1 --2- 172.21.15.78:0/367044578 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f236c073b30 0x7f236c1a5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.061+0000 7f2370a00700 1 --2- 172.21.15.78:0/367044578 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f236c100f20 0x7f236c0730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.061+0000 7f2370a00700 1 --2- 172.21.15.78:0/367044578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c106040 0x7f236c0735f0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.061+0000 7f2370a00700 1 -- 172.21.15.78:0/367044578 >> 172.21.15.78:0/367044578 conn(0x7f236c0fc7a0 msgr2=0x7f236c0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:18.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.061+0000 7f2370a00700 1 -- 172.21.15.78:0/367044578 shutdown_connections 2024-03-14T10:01:18.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.061+0000 7f2370a00700 1 -- 172.21.15.78:0/367044578 wait complete. 2024-03-14T10:01:18.082 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:01:18.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.278+0000 7f2bad400700 1 -- 172.21.15.78:0/1748587645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba8100f30 msgr2=0x7f2ba8101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.278+0000 7f2bad400700 1 --2- 172.21.15.78:0/1748587645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba8100f30 0x7f2ba8101310 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009990 tx=0x7f2b98009ca0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.278+0000 7f2bad400700 1 -- 172.21.15.78:0/1748587645 shutdown_connections 2024-03-14T10:01:18.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.278+0000 7f2bad400700 1 --2- 172.21.15.78:0/1748587645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ba8106050 0x7f2ba810dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.278+0000 7f2bad400700 1 --2- 172.21.15.78:0/1748587645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba8100f30 0x7f2ba8101310 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.278+0000 7f2bad400700 1 --2- 172.21.15.78:0/1748587645 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ba81018e0 0x7f2ba8105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.278+0000 7f2bad400700 1 -- 172.21.15.78:0/1748587645 >> 172.21.15.78:0/1748587645 conn(0x7f2ba80fc7b0 msgr2=0x7f2ba80febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:18.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.278+0000 7f2bad400700 1 -- 172.21.15.78:0/1748587645 shutdown_connections 2024-03-14T10:01:18.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.278+0000 7f2bad400700 1 -- 172.21.15.78:0/1748587645 wait complete. 2024-03-14T10:01:18.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.279+0000 7f2bad400700 1 Processor -- start 2024-03-14T10:01:18.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.279+0000 7f2bad400700 1 -- start start 2024-03-14T10:01:18.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2bad400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ba8100f30 0x7f2ba80730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2bad400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba8106050 0x7f2ba80735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2bad400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ba8073b30 0x7f2ba81a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2bad400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba81a5e20 con 0x7f2ba8106050 2024-03-14T10:01:18.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2bad400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba81a5f90 con 0x7f2ba8100f30 2024-03-14T10:01:18.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2bad400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba81a6150 con 0x7f2ba8073b30 2024-03-14T10:01:18.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2ba6800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ba8100f30 0x7f2ba80730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2ba5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba8106050 0x7f2ba80735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2ba6800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ba8100f30 0x7f2ba80730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:58516/0 (socket says 172.21.15.78:58516) 2024-03-14T10:01:18.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2ba6800700 1 -- 172.21.15.78:0/757734139 learned_addr learned my addr 172.21.15.78:0/757734139 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:18.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2ba7200700 1 --2- 172.21.15.78:0/757734139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ba8073b30 0x7f2ba81a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2ba6800700 1 -- 172.21.15.78:0/757734139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ba8073b30 msgr2=0x7f2ba81a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2ba6800700 1 --2- 172.21.15.78:0/757734139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ba8073b30 0x7f2ba81a57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2ba6800700 1 -- 172.21.15.78:0/757734139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba8106050 msgr2=0x7f2ba80735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2ba6800700 1 --2- 172.21.15.78:0/757734139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba8106050 0x7f2ba80735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2ba6800700 1 -- 172.21.15.78:0/757734139 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b980095f0 con 0x7f2ba8100f30 2024-03-14T10:01:18.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2ba5e00700 1 --2- 172.21.15.78:0/757734139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba8106050 0x7f2ba80735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:18.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.280+0000 7f2ba6800700 1 --2- 172.21.15.78:0/757734139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ba8100f30 0x7f2ba80730b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2b98000c00 tx=0x7f2b98005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:18.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.281+0000 7f2b9f400700 1 -- 172.21.15.78:0/757734139 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9801d070 con 0x7f2ba8100f30 2024-03-14T10:01:18.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.281+0000 7f2bad400700 1 -- 172.21.15.78:0/757734139 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba81a63d0 con 0x7f2ba8100f30 2024-03-14T10:01:18.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.281+0000 7f2b9f400700 1 -- 172.21.15.78:0/757734139 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b98022470 con 0x7f2ba8100f30 2024-03-14T10:01:18.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.281+0000 7f2b9f400700 1 -- 172.21.15.78:0/757734139 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b98017650 con 0x7f2ba8100f30 2024-03-14T10:01:18.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.281+0000 7f2bad400700 1 -- 172.21.15.78:0/757734139 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba81a6850 con 0x7f2ba8100f30 2024-03-14T10:01:18.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.284+0000 7f2b9f400700 1 -- 172.21.15.78:0/757734139 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f2b980225e0 con 0x7f2ba8100f30 2024-03-14T10:01:18.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.285+0000 7f2bc101ab80 1 Processor -- start 2024-03-14T10:01:18.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.285+0000 7f2bc101ab80 1 -- start start 2024-03-14T10:01:18.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.285+0000 7f2bc101ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807ef0680 0x557807ef0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.285+0000 7f2b9f400700 1 --2- 172.21.15.78:0/757734139 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2b8c0a0c00 0x7f2b8c0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.285+0000 7f2bc101ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0fa0 0x557807efa980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.285+0000 7f2bc101ab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807efaec0 0x557807efd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.285+0000 7f2bc101ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557807e49990 con 0x557807ef0fa0 2024-03-14T10:01:18.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.285+0000 7f2bc101ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557807e47350 con 0x557807ef0680 2024-03-14T10:01:18.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.285+0000 7f2bc101ab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557807e42c40 con 0x557807efaec0 2024-03-14T10:01:18.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.286+0000 7f2b9f400700 1 -- 172.21.15.78:0/757734139 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f2b980cd9e0 con 0x7f2ba8100f30 2024-03-14T10:01:18.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.286+0000 7f2ba7200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807ef0680 0x557807ef0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.286+0000 7f2ba6800700 1 --2- 172.21.15.78:0/757734139 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2b8c0a0c00 0x7f2b8c0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.286+0000 7f2ba7200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807ef0680 0x557807ef0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:58524/0 (socket says 172.21.15.78:58524) 2024-03-14T10:01:18.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.286+0000 7f2ba7200700 1 -- 172.21.15.78:0/2607413355 learned_addr learned my addr 172.21.15.78:0/2607413355 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:18.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.286+0000 7f2ba7200700 1 --2- 172.21.15.78:0/2607413355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807efaec0 0x557807efd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.286+0000 7f2ba5e00700 1 --2- 172.21.15.78:0/2607413355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0fa0 0x557807efa980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.286+0000 7f2ba7200700 1 -- 172.21.15.78:0/2607413355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807efaec0 msgr2=0x557807efd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:18.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.286+0000 7f2ba7200700 1 --2- 172.21.15.78:0/2607413355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807efaec0 0x557807efd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.286+0000 7f2ba7200700 1 -- 172.21.15.78:0/2607413355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0fa0 msgr2=0x557807efa980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:18.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.286+0000 7f2ba7200700 1 --2- 172.21.15.78:0/2607413355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0fa0 0x557807efa980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.286+0000 7f2ba7200700 1 -- 172.21.15.78:0/2607413355 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557807cc43b0 con 0x557807ef0680 2024-03-14T10:01:18.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.286+0000 7f2ba6800700 1 --2- 172.21.15.78:0/757734139 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2b8c0a0c00 0x7f2b8c0a30c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2b980cad60 tx=0x7f2b98009ca0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:18.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.286+0000 7f2ba7200700 1 --2- 172.21.15.78:0/2607413355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807ef0680 0x557807ef0a60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0012e40 tx=0x7f2ba0009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=213dd82cc17fe7c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:18.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.287+0000 7f2b93400700 1 -- 172.21.15.78:0/2607413355 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba000e040 con 0x557807ef0680 2024-03-14T10:01:18.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.287+0000 7f2b93400700 1 -- 172.21.15.78:0/2607413355 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ba00092d0 con 0x557807ef0680 2024-03-14T10:01:18.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.287+0000 7f2b93400700 1 -- 172.21.15.78:0/2607413355 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0007b60 con 0x557807ef0680 2024-03-14T10:01:18.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.287+0000 7f2bc101ab80 1 -- 172.21.15.78:0/2607413355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807ef0680 msgr2=0x557807ef0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.287+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/2607413355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807ef0680 0x557807ef0a60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0012e40 tx=0x7f2ba0009920 comp rx=0 tx=0).stop 2024-03-14T10:01:18.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.287+0000 7f2bc101ab80 1 -- 172.21.15.78:0/2607413355 shutdown_connections 2024-03-14T10:01:18.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.287+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/2607413355 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807ef0680 0x557807ef0a60 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.287+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/2607413355 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807efaec0 0x557807efd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.287+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/2607413355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0fa0 0x557807efa980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.287+0000 7f2bc101ab80 1 -- 172.21.15.78:0/2607413355 >> 172.21.15.78:0/2607413355 conn(0x557807e66220 msgr2=0x557807eebaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:18.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.287+0000 7f2bc101ab80 1 -- 172.21.15.78:0/2607413355 shutdown_connections 2024-03-14T10:01:18.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.287+0000 7f2bc101ab80 1 -- 172.21.15.78:0/2607413355 wait complete. 2024-03-14T10:01:18.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.288+0000 7f2bc101ab80 1 -- start start 2024-03-14T10:01:18.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.288+0000 7f2bc101ab80 1 Processor -- start 2024-03-14T10:01:18.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2bc101ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0680 0x557807efa8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2bc101ab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807ef0fa0 0x557807ef7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2bc101ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807efaec0 0x557807ef82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2bc101ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557807dded80 con 0x557807ef0680 2024-03-14T10:01:18.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2bc101ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557807ddea20 con 0x557807efaec0 2024-03-14T10:01:18.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2bc101ab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557807cc66a0 con 0x557807ef0fa0 2024-03-14T10:01:18.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2bc101ab80 10 client.0 _ll_register_callbacks cb 0x557807c4a3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:18.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2ba7200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0680 0x557807efa8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2ba7200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0680 0x557807efa8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41564/0 (socket says 172.21.15.78:41564) 2024-03-14T10:01:18.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2ba7200700 1 -- 172.21.15.78:0/1860424611 learned_addr learned my addr 172.21.15.78:0/1860424611 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:18.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2ba5e00700 1 --2- 172.21.15.78:0/1860424611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807ef0fa0 0x557807ef7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2ba7200700 1 --2- 172.21.15.78:0/1860424611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807efaec0 0x557807ef82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2ba7200700 1 -- 172.21.15.78:0/1860424611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807ef0fa0 msgr2=0x557807ef7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:18.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2ba7200700 1 --2- 172.21.15.78:0/1860424611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807ef0fa0 0x557807ef7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2ba7200700 1 -- 172.21.15.78:0/1860424611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807efaec0 msgr2=0x557807ef82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:18.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2ba7200700 1 --2- 172.21.15.78:0/1860424611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807efaec0 0x557807ef82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2ba7200700 1 -- 172.21.15.78:0/1860424611 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba0012af0 con 0x557807ef0680 2024-03-14T10:01:18.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2ba7200700 1 --2- 172.21.15.78:0/1860424611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0680 0x557807efa8a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2ba00125f0 tx=0x7f2ba000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4bff187f8172fe8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:18.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.289+0000 7f2b87e00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:18.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.290+0000 7f2b87e00700 1 -- 172.21.15.78:0/1860424611 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba000e050 con 0x557807ef0680 2024-03-14T10:01:18.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.290+0000 7f2b87e00700 1 -- 172.21.15.78:0/1860424611 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f2ba0004cd0 con 0x557807ef0680 2024-03-14T10:01:18.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.290+0000 7f2bc101ab80 1 -- 172.21.15.78:0/1860424611 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557807ee3720 con 0x557807ef0680 2024-03-14T10:01:18.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.290+0000 7f2b87e00700 1 -- 172.21.15.78:0/1860424611 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ba0004e90 con 0x557807ef0680 2024-03-14T10:01:18.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.290+0000 7f2bc101ab80 10 client.17052 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:18.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.290+0000 7f2bc101ab80 1 -- 172.21.15.78:0/1860424611 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557807c70b60 con 0x557807ef0680 2024-03-14T10:01:18.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.290+0000 7f2b87e00700 1 -- 172.21.15.78:0/1860424611 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557807ee3720 con 0x557807ef0680 2024-03-14T10:01:18.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.290+0000 7f2b87e00700 1 -- 172.21.15.78:0/1860424611 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2ba0017350 con 0x557807ef0680 2024-03-14T10:01:18.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.290+0000 7f2bc101ab80 10 client.17052 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:18.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.290+0000 7f2bc101ab80 10 client.17052 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:01:18.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.290+0000 7f2bc101ab80 20 client.17052 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:18.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.291+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/1860424611 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557807f0a8e0 0x557807f0cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.291+0000 7f2bc101ab80 4 client.17052 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:01:18.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.291+0000 7f2bc101ab80 1 -- 172.21.15.78:0/1860424611 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557807e3b700 con 0x557807f0a8e0 2024-03-14T10:01:18.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.291+0000 7f2ba5e00700 1 --2- 172.21.15.78:0/1860424611 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557807f0a8e0 0x557807f0cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.291+0000 7f2b87e00700 10 client.17052 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:18.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.291+0000 7f2ba5e00700 1 --2- 172.21.15.78:0/1860424611 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557807f0a8e0 0x557807f0cd00 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=154f4cbfbd32c703 server_cookie=aff9a0799b495151 in_seq=0 out_seq=0 2024-03-14T10:01:18.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.292+0000 7f2b87e00700 1 -- 172.21.15.78:0/1860424611 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x557807e3b700 con 0x557807f0a8e0 2024-03-14T10:01:18.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.292+0000 7f2b87e00700 10 client.17052 handle_command_reply: tid=0 2024-03-14T10:01:18.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.292+0000 7f2bc101ab80 1 client.17052 shutdown 2024-03-14T10:01:18.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.292+0000 7f2bc101ab80 1 -- 172.21.15.78:0/1860424611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0680 msgr2=0x557807efa8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.292+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/1860424611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0680 0x557807efa8a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2ba00125f0 tx=0x7f2ba000b040 comp rx=0 tx=0).stop 2024-03-14T10:01:18.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.292+0000 7f2bc101ab80 1 -- 172.21.15.78:0/1860424611 shutdown_connections 2024-03-14T10:01:18.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.292+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/1860424611 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557807f0a8e0 0x557807f0cd00 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.292+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/1860424611 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807efaec0 0x557807ef82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.292+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/1860424611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0680 0x557807efa8a0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.292+0000 7f2b87e00700 0 client.17052 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:18.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.292+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/1860424611 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807ef0fa0 0x557807ef7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.292+0000 7f2bc101ab80 1 -- 172.21.15.78:0/1860424611 >> 172.21.15.78:0/1860424611 conn(0x557807e66220 msgr2=0x557807eeb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:18.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.292+0000 7f2bc101ab80 1 -- 172.21.15.78:0/1860424611 shutdown_connections 2024-03-14T10:01:18.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.292+0000 7f2bc101ab80 1 -- 172.21.15.78:0/1860424611 wait complete. 2024-03-14T10:01:18.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.293+0000 7f2bc101ab80 20 client.17052 trim_cache size 0 max 16384 2024-03-14T10:01:18.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.309+0000 7f2bc101ab80 1 Processor -- start 2024-03-14T10:01:18.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.309+0000 7f2bc101ab80 1 -- start start 2024-03-14T10:01:18.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.309+0000 7f2bc101ab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807ef0680 0x557807eebc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.309+0000 7f2bc101ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807ef0fa0 0x557807f01bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.309+0000 7f2bc101ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807efaec0 0x557807ef2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.309+0000 7f2bc101ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557807eece30 con 0x557807efaec0 2024-03-14T10:01:18.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.309+0000 7f2bc101ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557807eed060 con 0x557807ef0fa0 2024-03-14T10:01:18.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.309+0000 7f2bc101ab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557807ee5d00 con 0x557807ef0680 2024-03-14T10:01:18.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.310+0000 7f2ba7200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807efaec0 0x557807ef2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.310+0000 7f2ba5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807ef0fa0 0x557807f01bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.310+0000 7f2ba7200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807efaec0 0x557807ef2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41576/0 (socket says 172.21.15.78:41576) 2024-03-14T10:01:18.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.310+0000 7f2ba7200700 1 -- 172.21.15.78:0/2893537498 learned_addr learned my addr 172.21.15.78:0/2893537498 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:18.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.310+0000 7f2ba7200700 1 --2- 172.21.15.78:0/2893537498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807ef0680 0x557807eebc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.310+0000 7f2ba5e00700 1 -- 172.21.15.78:0/2893537498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807ef0680 msgr2=0x557807eebc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:18.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.310+0000 7f2ba5e00700 1 --2- 172.21.15.78:0/2893537498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807ef0680 0x557807eebc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.310+0000 7f2ba5e00700 1 -- 172.21.15.78:0/2893537498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807efaec0 msgr2=0x557807ef2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:18.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.310+0000 7f2ba5e00700 1 --2- 172.21.15.78:0/2893537498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807efaec0 0x557807ef2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.310+0000 7f2ba5e00700 1 -- 172.21.15.78:0/2893537498 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557807e08ea0 con 0x557807ef0fa0 2024-03-14T10:01:18.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.310+0000 7f2ba7200700 1 --2- 172.21.15.78:0/2893537498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807efaec0 0x557807ef2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:18.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.310+0000 7f2ba5e00700 1 --2- 172.21.15.78:0/2893537498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807ef0fa0 0x557807f01bc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x557807f0d5e0 tx=0x7f2b94009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a9d09566d2e503b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:18.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.311+0000 7f2b93e00700 1 -- 172.21.15.78:0/2893537498 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b94016030 con 0x557807ef0fa0 2024-03-14T10:01:18.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.311+0000 7f2b93e00700 1 -- 172.21.15.78:0/2893537498 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b94017040 con 0x557807ef0fa0 2024-03-14T10:01:18.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.311+0000 7f2b93e00700 1 -- 172.21.15.78:0/2893537498 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b94010430 con 0x557807ef0fa0 2024-03-14T10:01:18.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.311+0000 7f2bc101ab80 1 -- 172.21.15.78:0/2893537498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807ef0fa0 msgr2=0x557807f01bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.311+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/2893537498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807ef0fa0 0x557807f01bc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x557807f0d5e0 tx=0x7f2b94009260 comp rx=0 tx=0).stop 2024-03-14T10:01:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.311+0000 7f2bc101ab80 1 -- 172.21.15.78:0/2893537498 shutdown_connections 2024-03-14T10:01:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.311+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/2893537498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807efaec0 0x557807ef2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.311+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/2893537498 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807ef0680 0x557807eebc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.311+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/2893537498 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807ef0fa0 0x557807f01bc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.311+0000 7f2bc101ab80 1 -- 172.21.15.78:0/2893537498 >> 172.21.15.78:0/2893537498 conn(0x557807e66220 msgr2=0x557807efa430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.311+0000 7f2bc101ab80 1 -- 172.21.15.78:0/2893537498 shutdown_connections 2024-03-14T10:01:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.311+0000 7f2bc101ab80 1 -- 172.21.15.78:0/2893537498 wait complete. 2024-03-14T10:01:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.312+0000 7f2bc101ab80 1 -- start start 2024-03-14T10:01:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.312+0000 7f2bc101ab80 1 Processor -- start 2024-03-14T10:01:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.312+0000 7f2bc101ab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807ef0680 0x557807f01a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.312+0000 7f2bc101ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0fa0 0x557807efffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.312+0000 7f2bc101ab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807efaec0 0x557807f004f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.312+0000 7f2bc101ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557807f00bd0 con 0x557807ef0fa0 2024-03-14T10:01:18.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.312+0000 7f2bc101ab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x557807f00d40 con 0x557807efaec0 2024-03-14T10:01:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.312+0000 7f2bc101ab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x557807f5c410 con 0x557807ef0680 2024-03-14T10:01:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.312+0000 7f2bc101ab80 10 client.0 _ll_register_callbacks cb 0x7f2b94009be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.313+0000 7f2ba5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0fa0 0x557807efffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.313+0000 7f2ba7200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807ef0680 0x557807f01a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.313+0000 7f2ba5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0fa0 0x557807efffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41580/0 (socket says 172.21.15.78:41580) 2024-03-14T10:01:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.313+0000 7f2ba5e00700 1 -- 172.21.15.78:0/1642338099 learned_addr learned my addr 172.21.15.78:0/1642338099 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.313+0000 7f2ba7200700 1 --2- 172.21.15.78:0/1642338099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807efaec0 0x557807f004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.313+0000 7f2ba5e00700 1 -- 172.21.15.78:0/1642338099 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807ef0680 msgr2=0x557807f01a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.313+0000 7f2ba5e00700 1 --2- 172.21.15.78:0/1642338099 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807ef0680 0x557807f01a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.313+0000 7f2ba5e00700 1 -- 172.21.15.78:0/1642338099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807efaec0 msgr2=0x557807f004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.313+0000 7f2ba5e00700 1 --2- 172.21.15.78:0/1642338099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807efaec0 0x557807f004f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.313+0000 7f2ba5e00700 1 -- 172.21.15.78:0/1642338099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557807c70b60 con 0x557807ef0fa0 2024-03-14T10:01:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.313+0000 7f2ba5e00700 1 --2- 172.21.15.78:0/1642338099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0fa0 0x557807efffb0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2b94012000 tx=0x7f2b9401f170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31bb89ef57f7fdf server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.313+0000 7f2b87e00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:18.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.313+0000 7f2b87e00700 1 -- 172.21.15.78:0/1642338099 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b94016040 con 0x557807ef0fa0 2024-03-14T10:01:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.313+0000 7f2b87e00700 1 -- 172.21.15.78:0/1642338099 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f2b94005250 con 0x557807ef0fa0 2024-03-14T10:01:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.313+0000 7f2bc101ab80 1 -- 172.21.15.78:0/1642338099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2ba00092d0 con 0x557807ef0fa0 2024-03-14T10:01:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.314+0000 7f2b87e00700 1 -- 172.21.15.78:0/1642338099 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b94005410 con 0x557807ef0fa0 2024-03-14T10:01:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.314+0000 7f2bc101ab80 10 client.17058 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.314+0000 7f2bc101ab80 1 -- 172.21.15.78:0/1642338099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f2b940043e0 con 0x557807ef0fa0 2024-03-14T10:01:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.314+0000 7f2b87e00700 1 -- 172.21.15.78:0/1642338099 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ba00092d0 con 0x557807ef0fa0 2024-03-14T10:01:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.314+0000 7f2b87e00700 1 -- 172.21.15.78:0/1642338099 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2b940038b0 con 0x557807ef0fa0 2024-03-14T10:01:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.314+0000 7f2bc101ab80 10 client.17058 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.314+0000 7f2bc101ab80 10 client.17058 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:01:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.314+0000 7f2bc101ab80 20 client.17058 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.314+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/1642338099 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557807f5da80 0x557807f5fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.314+0000 7f2bc101ab80 4 client.17058 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:01:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.314+0000 7f2bc101ab80 1 -- 172.21.15.78:0/1642338099 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557807c91cf0 con 0x557807f5da80 2024-03-14T10:01:18.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.314+0000 7f2ba7200700 1 --2- 172.21.15.78:0/1642338099 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557807f5da80 0x557807f5fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.315+0000 7f2b87e00700 10 client.17058 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.315+0000 7f2ba7200700 1 --2- 172.21.15.78:0/1642338099 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557807f5da80 0x557807f5fea0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b60a6b2773daf7d6 server_cookie=711b9942516910fd in_seq=0 out_seq=0 2024-03-14T10:01:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.315+0000 7f2b87e00700 1 -- 172.21.15.78:0/1642338099 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x557807c91cf0 con 0x557807f5da80 2024-03-14T10:01:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.315+0000 7f2b87e00700 10 client.17058 handle_command_reply: tid=0 2024-03-14T10:01:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.315+0000 7f2bc101ab80 1 client.17058 shutdown 2024-03-14T10:01:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.315+0000 7f2bc101ab80 1 -- 172.21.15.78:0/1642338099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0fa0 msgr2=0x557807efffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.315+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/1642338099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0fa0 0x557807efffb0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2b94012000 tx=0x7f2b9401f170 comp rx=0 tx=0).stop 2024-03-14T10:01:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.315+0000 7f2bc101ab80 1 -- 172.21.15.78:0/1642338099 shutdown_connections 2024-03-14T10:01:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.315+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/1642338099 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x557807f5da80 0x557807f5fea0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.315+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/1642338099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557807efaec0 0x557807f004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.315+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/1642338099 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x557807ef0680 0x557807f01a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.315+0000 7f2bc101ab80 1 --2- 172.21.15.78:0/1642338099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557807ef0fa0 0x557807efffb0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.315+0000 7f2b87e00700 0 client.17058 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:18.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.315+0000 7f2bc101ab80 1 -- 172.21.15.78:0/1642338099 >> 172.21.15.78:0/1642338099 conn(0x557807e66220 msgr2=0x557807ef6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:18.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.316+0000 7f2bc101ab80 1 -- 172.21.15.78:0/1642338099 shutdown_connections 2024-03-14T10:01:18.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.316+0000 7f2bc101ab80 1 -- 172.21.15.78:0/1642338099 wait complete. 2024-03-14T10:01:18.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.316+0000 7f2bc101ab80 20 client.17058 trim_cache size 0 max 16384 2024-03-14T10:01:18.320 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:01:18.320 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-14T10:01:18.320 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T10:01:18.320 INFO:teuthology.orchestra.run.smithi078.stdout: "c5a28d5f-0723-423b-9254-81cc176b727c": { 2024-03-14T10:01:18.320 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T10:01:18.320 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "c5a28d5f-0723-423b-9254-81cc176b727c", 2024-03-14T10:01:18.320 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T10:01:18.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:01:18.321 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:01:18.321 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:01:18.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.316+0000 7f2bad400700 1 -- 172.21.15.78:0/757734139 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2b8c0a0c00 msgr2=0x7f2b8c0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.316+0000 7f2bad400700 1 --2- 172.21.15.78:0/757734139 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2b8c0a0c00 0x7f2b8c0a30c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2b980cad60 tx=0x7f2b98009ca0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.316+0000 7f2bad400700 1 -- 172.21.15.78:0/757734139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ba8100f30 msgr2=0x7f2ba80730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:18.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.316+0000 7f2bad400700 1 --2- 172.21.15.78:0/757734139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ba8100f30 0x7f2ba80730b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2b98000c00 tx=0x7f2b98005740 comp rx=0 tx=0).stop 2024-03-14T10:01:18.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.316+0000 7f2bad400700 1 -- 172.21.15.78:0/757734139 shutdown_connections 2024-03-14T10:01:18.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.316+0000 7f2bad400700 1 --2- 172.21.15.78:0/757734139 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2b8c0a0c00 0x7f2b8c0a30c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.316+0000 7f2bad400700 1 --2- 172.21.15.78:0/757734139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ba8100f30 0x7f2ba80730b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.316+0000 7f2bad400700 1 --2- 172.21.15.78:0/757734139 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ba8073b30 0x7f2ba81a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.316+0000 7f2bad400700 1 --2- 172.21.15.78:0/757734139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ba8106050 0x7f2ba80735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:18.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.316+0000 7f2bad400700 1 -- 172.21.15.78:0/757734139 >> 172.21.15.78:0/757734139 conn(0x7f2ba80fc7b0 msgr2=0x7f2ba80fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:18.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.316+0000 7f2bad400700 1 -- 172.21.15.78:0/757734139 shutdown_connections 2024-03-14T10:01:18.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:18.316+0000 7f2bad400700 1 -- 172.21.15.78:0/757734139 wait complete. 2024-03-14T10:01:18.333 INFO:tasks.cephfs.filesystem:scrub status for tag:c5a28d5f-0723-423b-9254-81cc176b727c - {'path': '/', 'tag': 'c5a28d5f-0723-423b-9254-81cc176b727c', 'options': 'recursive,force'} 2024-03-14T10:01:19.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:18 smithi152 ceph-mon[48279]: pgmap v219: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 38 KiB/s wr, 15 op/s 2024-03-14T10:01:19.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:18 smithi132 ceph-mon[47959]: pgmap v219: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 38 KiB/s wr, 15 op/s 2024-03-14T10:01:19.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:18 smithi078 ceph-mon[38143]: pgmap v219: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 38 KiB/s wr, 15 op/s 2024-03-14T10:01:20.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:19 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T10:01:20.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:19 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T10:01:20.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:19 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T10:01:20.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:19 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T10:01:20.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:19 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T10:01:20.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:19 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T10:01:20.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:19 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T10:01:20.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:19 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T10:01:20.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:19 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T10:01:20.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:19 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T10:01:20.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:19 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T10:01:20.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:19 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T10:01:20.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:19 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T10:01:20.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:19 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T10:01:20.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:19 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T10:01:21.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:20 smithi152 ceph-mon[48279]: pgmap v220: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 35 KiB/s wr, 14 op/s 2024-03-14T10:01:21.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:20 smithi132 ceph-mon[47959]: pgmap v220: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 35 KiB/s wr, 14 op/s 2024-03-14T10:01:21.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:20 smithi078 ceph-mon[38143]: pgmap v220: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 35 KiB/s wr, 14 op/s 2024-03-14T10:01:23.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:23 smithi152 ceph-mon[48279]: pgmap v221: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-03-14T10:01:23.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:23 smithi132 ceph-mon[47959]: pgmap v221: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-03-14T10:01:23.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:23 smithi078 ceph-mon[38143]: pgmap v221: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-03-14T10:01:24.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:24 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T10:01:24.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:24 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T10:01:24.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:24 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T10:01:25.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:25 smithi152 ceph-mon[48279]: pgmap v222: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-14T10:01:25.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:25 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T10:01:25.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:25 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T10:01:25.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:25 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:01:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:25 smithi132 ceph-mon[47959]: pgmap v222: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-14T10:01:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:25 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T10:01:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:25 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T10:01:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:25 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:01:25.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:25 smithi078 ceph-mon[38143]: pgmap v222: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-14T10:01:25.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:25 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T10:01:25.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:25 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T10:01:25.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:25 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:01:27.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:27 smithi152 ceph-mon[48279]: pgmap v223: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-14T10:01:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:27 smithi132 ceph-mon[47959]: pgmap v223: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-14T10:01:27.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:27 smithi078 ceph-mon[38143]: pgmap v223: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-03-14T10:01:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:28 smithi152 ceph-mon[48279]: pgmap v224: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:01:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:28 smithi132 ceph-mon[47959]: pgmap v224: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:01:29.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:28 smithi078 ceph-mon[38143]: pgmap v224: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:01:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:31 smithi152 ceph-mon[48279]: pgmap v225: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:01:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:31 smithi132 ceph-mon[47959]: pgmap v225: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:01:31.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:31 smithi078 ceph-mon[38143]: pgmap v225: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:01:33.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:33 smithi152 ceph-mon[48279]: pgmap v226: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:01:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:33 smithi132 ceph-mon[47959]: pgmap v226: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:01:33.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:33 smithi078 ceph-mon[38143]: pgmap v226: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:01:35.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:35 smithi152 ceph-mon[48279]: pgmap v227: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:35.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:35 smithi132 ceph-mon[47959]: pgmap v227: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:35.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:35 smithi078 ceph-mon[38143]: pgmap v227: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:37 smithi152 ceph-mon[48279]: pgmap v228: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:01:37.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:37 smithi132 ceph-mon[47959]: pgmap v228: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:01:37.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:37 smithi078 ceph-mon[38143]: pgmap v228: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:01:39.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:38 smithi152 ceph-mon[48279]: pgmap v229: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:38 smithi132 ceph-mon[47959]: pgmap v229: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:39.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:38 smithi078 ceph-mon[38143]: pgmap v229: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:41 smithi152 ceph-mon[48279]: pgmap v230: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:41 smithi132 ceph-mon[47959]: pgmap v230: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:41.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:41 smithi078 ceph-mon[38143]: pgmap v230: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:43.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:43 smithi152 ceph-mon[48279]: pgmap v231: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:01:43.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:43 smithi132 ceph-mon[47959]: pgmap v231: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:01:43.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:43 smithi078 ceph-mon[38143]: pgmap v231: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:01:45.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:45 smithi152 ceph-mon[48279]: pgmap v232: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:45.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:45 smithi132 ceph-mon[47959]: pgmap v232: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:45.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:45 smithi078 ceph-mon[38143]: pgmap v232: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:47 smithi152 ceph-mon[48279]: pgmap v233: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:01:47.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:47 smithi132 ceph-mon[47959]: pgmap v233: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:01:47.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:47 smithi078 ceph-mon[38143]: pgmap v233: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:01:48.335 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:01:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.545+0000 7f69f1400700 1 -- 172.21.15.78:0/2523040798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec1018d0 msgr2=0x7f69ec105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.545+0000 7f69f1400700 1 --2- 172.21.15.78:0/2523040798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec1018d0 0x7f69ec105910 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009990 tx=0x7f69d4009ca0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.545+0000 7f69f1400700 1 -- 172.21.15.78:0/2523040798 shutdown_connections 2024-03-14T10:01:48.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.545+0000 7f69f1400700 1 --2- 172.21.15.78:0/2523040798 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69ec106040 0x7f69ec10dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.545+0000 7f69f1400700 1 --2- 172.21.15.78:0/2523040798 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69ec100f20 0x7f69ec101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.545+0000 7f69f1400700 1 --2- 172.21.15.78:0/2523040798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec1018d0 0x7f69ec105910 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.545+0000 7f69f1400700 1 -- 172.21.15.78:0/2523040798 >> 172.21.15.78:0/2523040798 conn(0x7f69ec0fc7a0 msgr2=0x7f69ec0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.545+0000 7f69f1400700 1 -- 172.21.15.78:0/2523040798 shutdown_connections 2024-03-14T10:01:48.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.545+0000 7f69f1400700 1 -- 172.21.15.78:0/2523040798 wait complete. 2024-03-14T10:01:48.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.546+0000 7f69f1400700 1 Processor -- start 2024-03-14T10:01:48.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.546+0000 7f69f1400700 1 -- start start 2024-03-14T10:01:48.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69f1400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec100f20 0x7f69ec19f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69f1400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69ec106040 0x7f69ec19f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69f1400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69ec1996b0 0x7f69ec199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69f1400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec19a100 con 0x7f69ec100f20 2024-03-14T10:01:48.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69f1400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec19a240 con 0x7f69ec106040 2024-03-14T10:01:48.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69f1400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec19a400 con 0x7f69ec1996b0 2024-03-14T10:01:48.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69ea800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec100f20 0x7f69ec19f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69ea800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec100f20 0x7f69ec19f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42136/0 (socket says 172.21.15.78:42136) 2024-03-14T10:01:48.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69e3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69ec106040 0x7f69ec19f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69ea800700 1 -- 172.21.15.78:0/1754897406 learned_addr learned my addr 172.21.15.78:0/1754897406 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:48.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69eb200700 1 --2- 172.21.15.78:0/1754897406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69ec1996b0 0x7f69ec199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69e3e00700 1 -- 172.21.15.78:0/1754897406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69ec1996b0 msgr2=0x7f69ec199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:48.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69e3e00700 1 --2- 172.21.15.78:0/1754897406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69ec1996b0 0x7f69ec199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69e3e00700 1 -- 172.21.15.78:0/1754897406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec100f20 msgr2=0x7f69ec19f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:48.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69e3e00700 1 --2- 172.21.15.78:0/1754897406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec100f20 0x7f69ec19f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69e3e00700 1 -- 172.21.15.78:0/1754897406 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d40095f0 con 0x7f69ec106040 2024-03-14T10:01:48.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.547+0000 7f69ea800700 1 --2- 172.21.15.78:0/1754897406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec100f20 0x7f69ec19f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:48.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.548+0000 7f69e3e00700 1 --2- 172.21.15.78:0/1754897406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69ec106040 0x7f69ec19f9a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f69d4000c00 tx=0x7f69d4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:48.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.548+0000 7f69e3400700 1 -- 172.21.15.78:0/1754897406 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d401d070 con 0x7f69ec106040 2024-03-14T10:01:48.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.548+0000 7f69f1400700 1 -- 172.21.15.78:0/1754897406 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ec1a6320 con 0x7f69ec106040 2024-03-14T10:01:48.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.548+0000 7f69e3400700 1 -- 172.21.15.78:0/1754897406 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f69d4021410 con 0x7f69ec106040 2024-03-14T10:01:48.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.548+0000 7f69e3400700 1 -- 172.21.15.78:0/1754897406 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4017650 con 0x7f69ec106040 2024-03-14T10:01:48.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.548+0000 7f69f1400700 1 -- 172.21.15.78:0/1754897406 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ec1a67a0 con 0x7f69ec106040 2024-03-14T10:01:48.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.553+0000 7f69e3400700 1 -- 172.21.15.78:0/1754897406 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f69d40217f0 con 0x7f69ec106040 2024-03-14T10:01:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.553+0000 7f6a04f26b80 1 Processor -- start 2024-03-14T10:01:48.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.553+0000 7f6a04f26b80 1 -- start start 2024-03-14T10:01:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.554+0000 7f6a04f26b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307680 0x55c45e307a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.554+0000 7f6a04f26b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307fa0 0x55c45e311980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.554+0000 7f6a04f26b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e311ec0 0x55c45e3142b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.554+0000 7f6a04f26b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c45e260990 con 0x55c45e307680 2024-03-14T10:01:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.554+0000 7f6a04f26b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c45e25e350 con 0x55c45e307fa0 2024-03-14T10:01:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.554+0000 7f6a04f26b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c45e259c40 con 0x55c45e311ec0 2024-03-14T10:01:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.554+0000 7f69eb200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307680 0x55c45e307a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.556 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.554+0000 7f69eb200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307680 0x55c45e307a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42144/0 (socket says 172.21.15.78:42144) 2024-03-14T10:01:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.554+0000 7f69eb200700 1 -- 172.21.15.78:0/3735509983 learned_addr learned my addr 172.21.15.78:0/3735509983 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.554+0000 7f69e3400700 1 --2- 172.21.15.78:0/1754897406 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f69d00a0c00 0x7f69d00a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.554+0000 7f69e3e00700 1 --2- 172.21.15.78:0/3735509983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e311ec0 0x55c45e3142b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.554+0000 7f69ea800700 1 --2- 172.21.15.78:0/3735509983 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307fa0 0x55c45e311980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.554+0000 7f69e3400700 1 -- 172.21.15.78:0/1754897406 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f69d40cdaa0 con 0x7f69ec106040 2024-03-14T10:01:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.554+0000 7f69eb200700 1 --2- 172.21.15.78:0/1754897406 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f69d00a0c00 0x7f69d00a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.555+0000 7f69ea800700 1 -- 172.21.15.78:0/3735509983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e311ec0 msgr2=0x55c45e3142b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.555+0000 7f69ea800700 1 --2- 172.21.15.78:0/3735509983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e311ec0 0x55c45e3142b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.555+0000 7f69ea800700 1 -- 172.21.15.78:0/3735509983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307680 msgr2=0x55c45e307a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.555+0000 7f69ea800700 1 --2- 172.21.15.78:0/3735509983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307680 0x55c45e307a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.555+0000 7f69ea800700 1 -- 172.21.15.78:0/3735509983 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c45e0db3b0 con 0x55c45e307fa0 2024-03-14T10:01:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.555+0000 7f69eb200700 1 --2- 172.21.15.78:0/3735509983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307680 0x55c45e307a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.555+0000 7f69ea800700 1 --2- 172.21.15.78:0/3735509983 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307fa0 0x55c45e311980 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f69dc011570 tx=0x7f69dc011880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc9dfc529dccdbc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.555+0000 7f69eb200700 1 --2- 172.21.15.78:0/1754897406 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f69d00a0c00 0x7f69d00a30c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f69e4003fd0 tx=0x7f69e4004b40 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:48.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.555+0000 7f69db400700 1 -- 172.21.15.78:0/3735509983 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69dc004230 con 0x55c45e307fa0 2024-03-14T10:01:48.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.555+0000 7f69db400700 1 -- 172.21.15.78:0/3735509983 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f69dc004390 con 0x55c45e307fa0 2024-03-14T10:01:48.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.555+0000 7f69db400700 1 -- 172.21.15.78:0/3735509983 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69dc0198c0 con 0x55c45e307fa0 2024-03-14T10:01:48.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.556+0000 7f6a04f26b80 1 -- 172.21.15.78:0/3735509983 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307fa0 msgr2=0x55c45e311980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:48.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.556+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/3735509983 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307fa0 0x55c45e311980 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f69dc011570 tx=0x7f69dc011880 comp rx=0 tx=0).stop 2024-03-14T10:01:48.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.556+0000 7f6a04f26b80 1 -- 172.21.15.78:0/3735509983 shutdown_connections 2024-03-14T10:01:48.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.556+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/3735509983 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e311ec0 0x55c45e3142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.556+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/3735509983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307680 0x55c45e307a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.556+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/3735509983 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307fa0 0x55c45e311980 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.556+0000 7f6a04f26b80 1 -- 172.21.15.78:0/3735509983 >> 172.21.15.78:0/3735509983 conn(0x55c45e27d220 msgr2=0x55c45e302aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:48.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.556+0000 7f6a04f26b80 1 -- 172.21.15.78:0/3735509983 shutdown_connections 2024-03-14T10:01:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.556+0000 7f6a04f26b80 1 -- 172.21.15.78:0/3735509983 wait complete. 2024-03-14T10:01:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.556+0000 7f6a04f26b80 1 -- start start 2024-03-14T10:01:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.557+0000 7f6a04f26b80 1 Processor -- start 2024-03-14T10:01:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.557+0000 7f6a04f26b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e307680 0x55c45e311870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.557+0000 7f6a04f26b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307fa0 0x55c45e30ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.557+0000 7f6a04f26b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e311ec0 0x55c45e30f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.557+0000 7f6a04f26b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c45e1f5d80 con 0x55c45e311ec0 2024-03-14T10:01:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.557+0000 7f6a04f26b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c45e1f5a20 con 0x55c45e307fa0 2024-03-14T10:01:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.557+0000 7f6a04f26b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c45e0dd6a0 con 0x55c45e307680 2024-03-14T10:01:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.557+0000 7f6a04f26b80 10 client.0 _ll_register_callbacks cb 0x55c45e0613a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.557+0000 7f69eb200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e311ec0 0x55c45e30f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.557+0000 7f69ea800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e307680 0x55c45e311870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.557+0000 7f69e3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307fa0 0x55c45e30ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.557+0000 7f69e3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307fa0 0x55c45e30ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:49578/0 (socket says 172.21.15.78:49578) 2024-03-14T10:01:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.557+0000 7f69e3e00700 1 -- 172.21.15.78:0/381807440 learned_addr learned my addr 172.21.15.78:0/381807440 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.557+0000 7f69ea800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e307680 0x55c45e311870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:52026/0 (socket says 172.21.15.78:52026) 2024-03-14T10:01:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.558+0000 7f69e3e00700 1 -- 172.21.15.78:0/381807440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e307680 msgr2=0x55c45e311870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.558+0000 7f69e3e00700 1 --2- 172.21.15.78:0/381807440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e307680 0x55c45e311870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.558+0000 7f69e3e00700 1 -- 172.21.15.78:0/381807440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e311ec0 msgr2=0x55c45e30f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.558+0000 7f69e3e00700 1 --2- 172.21.15.78:0/381807440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e311ec0 0x55c45e30f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.558+0000 7f69e3e00700 1 -- 172.21.15.78:0/381807440 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e4009ba0 con 0x55c45e307fa0 2024-03-14T10:01:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.558+0000 7f69eb200700 1 --2- 172.21.15.78:0/381807440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e311ec0 0x55c45e30f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.558+0000 7f69ea800700 1 --2- 172.21.15.78:0/381807440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e307680 0x55c45e311870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.558+0000 7f69e3e00700 1 --2- 172.21.15.78:0/381807440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307fa0 0x55c45e30ec60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009990 tx=0x7f69d409a980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6265038c220557a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.558+0000 7f69cfe00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T10:01:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.558+0000 7f69cfe00700 1 -- 172.21.15.78:0/381807440 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d409af10 con 0x55c45e307fa0 2024-03-14T10:01:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.558+0000 7f6a04f26b80 1 -- 172.21.15.78:0/381807440 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c45e2fa720 con 0x55c45e307fa0 2024-03-14T10:01:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.558+0000 7f69cfe00700 1 -- 172.21.15.78:0/381807440 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f69d409b1b0 con 0x55c45e307fa0 2024-03-14T10:01:48.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.558+0000 7f69cfe00700 1 -- 172.21.15.78:0/381807440 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69d409a410 con 0x55c45e307fa0 2024-03-14T10:01:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.559+0000 7f6a04f26b80 10 client.27143 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.559+0000 7f6a04f26b80 1 -- 172.21.15.78:0/381807440 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f69dc009d60 con 0x55c45e307fa0 2024-03-14T10:01:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.559+0000 7f69cfe00700 1 -- 172.21.15.78:0/381807440 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c45e2fa720 con 0x55c45e307fa0 2024-03-14T10:01:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.559+0000 7f69cfe00700 1 -- 172.21.15.78:0/381807440 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f69d40a2a10 con 0x55c45e307fa0 2024-03-14T10:01:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.559+0000 7f6a04f26b80 10 client.27143 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.559+0000 7f6a04f26b80 10 client.27143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:01:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.559+0000 7f6a04f26b80 20 client.27143 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.559+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/381807440 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c45e321690 0x55c45e323a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.559+0000 7f6a04f26b80 4 client.27143 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:01:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.559+0000 7f6a04f26b80 1 -- 172.21.15.78:0/381807440 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c45e252700 con 0x55c45e321690 2024-03-14T10:01:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.559+0000 7f69ea800700 1 --2- 172.21.15.78:0/381807440 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c45e321690 0x55c45e323a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.560+0000 7f69ea800700 1 --2- 172.21.15.78:0/381807440 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c45e321690 0x55c45e323a80 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9eae746c24039b2d server_cookie=510bc02d70be2f62 in_seq=0 out_seq=0 2024-03-14T10:01:48.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.560+0000 7f69cfe00700 10 client.27143 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:48.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.561+0000 7f69cfe00700 1 -- 172.21.15.78:0/381807440 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55c45e252700 con 0x55c45e321690 2024-03-14T10:01:48.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.561+0000 7f69cfe00700 10 client.27143 handle_command_reply: tid=0 2024-03-14T10:01:48.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.561+0000 7f6a04f26b80 1 client.27143 shutdown 2024-03-14T10:01:48.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.561+0000 7f6a04f26b80 1 -- 172.21.15.78:0/381807440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307fa0 msgr2=0x55c45e30ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:48.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.561+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/381807440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307fa0 0x55c45e30ec60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009990 tx=0x7f69d409a980 comp rx=0 tx=0).stop 2024-03-14T10:01:48.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.561+0000 7f6a04f26b80 1 -- 172.21.15.78:0/381807440 shutdown_connections 2024-03-14T10:01:48.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.561+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/381807440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e311ec0 0x55c45e30f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.561+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/381807440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e307680 0x55c45e311870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.561+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/381807440 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c45e321690 0x55c45e323a80 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.561+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/381807440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307fa0 0x55c45e30ec60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.561+0000 7f6a04f26b80 1 -- 172.21.15.78:0/381807440 >> 172.21.15.78:0/381807440 conn(0x55c45e27d220 msgr2=0x55c45e302690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:48.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.561+0000 7f69cfe00700 0 client.27143 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:48.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.561+0000 7f6a04f26b80 1 -- 172.21.15.78:0/381807440 shutdown_connections 2024-03-14T10:01:48.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.561+0000 7f6a04f26b80 1 -- 172.21.15.78:0/381807440 wait complete. 2024-03-14T10:01:48.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.562+0000 7f6a04f26b80 20 client.27143 trim_cache size 0 max 16384 2024-03-14T10:01:48.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.578+0000 7f6a04f26b80 1 Processor -- start 2024-03-14T10:01:48.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.578+0000 7f6a04f26b80 1 -- start start 2024-03-14T10:01:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.578+0000 7f6a04f26b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307680 0x55c45e318b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.578+0000 7f6a04f26b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307fa0 0x55c45e30ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.578+0000 7f6a04f26b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e311ec0 0x55c45e30f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.578+0000 7f6a04f26b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c45e303e30 con 0x55c45e307fa0 2024-03-14T10:01:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.578+0000 7f6a04f26b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c45e304060 con 0x55c45e307680 2024-03-14T10:01:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.578+0000 7f6a04f26b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c45e303770 con 0x55c45e311ec0 2024-03-14T10:01:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.579+0000 7f69e3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307fa0 0x55c45e30ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.579+0000 7f69e3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307fa0 0x55c45e30ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42172/0 (socket says 172.21.15.78:42172) 2024-03-14T10:01:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.579+0000 7f69e3e00700 1 -- 172.21.15.78:0/4069034053 learned_addr learned my addr 172.21.15.78:0/4069034053 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:48.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.579+0000 7f69ea800700 1 --2- 172.21.15.78:0/4069034053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307680 0x55c45e318b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.579+0000 7f69eb200700 1 --2- 172.21.15.78:0/4069034053 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e311ec0 0x55c45e30f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.579+0000 7f69e3e00700 1 -- 172.21.15.78:0/4069034053 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e311ec0 msgr2=0x55c45e30f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:48.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.579+0000 7f69e3e00700 1 --2- 172.21.15.78:0/4069034053 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e311ec0 0x55c45e30f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.579+0000 7f69e3e00700 1 -- 172.21.15.78:0/4069034053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307680 msgr2=0x55c45e318b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:48.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.579+0000 7f69e3e00700 1 --2- 172.21.15.78:0/4069034053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307680 0x55c45e318b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.579+0000 7f69e3e00700 1 -- 172.21.15.78:0/4069034053 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c45e087b60 con 0x55c45e307fa0 2024-03-14T10:01:48.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.579+0000 7f69e3e00700 1 --2- 172.21.15.78:0/4069034053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307fa0 0x55c45e30ec60 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f69d4096fb0 tx=0x7f69d40cddf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d47dfe61e92247 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:48.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.579+0000 7f69dbe00700 1 -- 172.21.15.78:0/4069034053 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d40b3140 con 0x55c45e307fa0 2024-03-14T10:01:48.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.580+0000 7f69dbe00700 1 -- 172.21.15.78:0/4069034053 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f69d4021e90 con 0x55c45e307fa0 2024-03-14T10:01:48.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.580+0000 7f69dbe00700 1 -- 172.21.15.78:0/4069034053 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d409aab0 con 0x55c45e307fa0 2024-03-14T10:01:48.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.580+0000 7f6a04f26b80 1 -- 172.21.15.78:0/4069034053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307fa0 msgr2=0x55c45e30ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:48.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.580+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/4069034053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307fa0 0x55c45e30ec60 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f69d4096fb0 tx=0x7f69d40cddf0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.580+0000 7f6a04f26b80 1 -- 172.21.15.78:0/4069034053 shutdown_connections 2024-03-14T10:01:48.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.580+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/4069034053 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e307680 0x55c45e318b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.580+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/4069034053 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e311ec0 0x55c45e30f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.580+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/4069034053 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307fa0 0x55c45e30ec60 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.580+0000 7f6a04f26b80 1 -- 172.21.15.78:0/4069034053 >> 172.21.15.78:0/4069034053 conn(0x55c45e27d220 msgr2=0x55c45e301180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:48.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.580+0000 7f6a04f26b80 1 -- 172.21.15.78:0/4069034053 shutdown_connections 2024-03-14T10:01:48.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.580+0000 7f6a04f26b80 1 -- 172.21.15.78:0/4069034053 wait complete. 2024-03-14T10:01:48.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.581+0000 7f6a04f26b80 1 -- start start 2024-03-14T10:01:48.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.581+0000 7f6a04f26b80 1 Processor -- start 2024-03-14T10:01:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.581+0000 7f6a04f26b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e307680 0x55c45e316b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.581+0000 7f6a04f26b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307fa0 0x55c45e317070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.581+0000 7f6a04f26b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e311ec0 0x55c45e372180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.581+0000 7f6a04f26b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c45e317960 con 0x55c45e307fa0 2024-03-14T10:01:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.581+0000 7f6a04f26b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c45e3726c0 con 0x55c45e311ec0 2024-03-14T10:01:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.581+0000 7f6a04f26b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c45e372800 con 0x55c45e307680 2024-03-14T10:01:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.581+0000 7f6a04f26b80 10 client.0 _ll_register_callbacks cb 0x7f69d4005c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.581+0000 7f69e3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307fa0 0x55c45e317070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f69e3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307fa0 0x55c45e317070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42174/0 (socket says 172.21.15.78:42174) 2024-03-14T10:01:48.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f69e3e00700 1 -- 172.21.15.78:0/1180990929 learned_addr learned my addr 172.21.15.78:0/1180990929 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:48.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f69eb200700 1 --2- 172.21.15.78:0/1180990929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e311ec0 0x55c45e372180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f69ea800700 1 --2- 172.21.15.78:0/1180990929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e307680 0x55c45e316b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f69e3e00700 1 -- 172.21.15.78:0/1180990929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e307680 msgr2=0x55c45e316b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:48.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f69e3e00700 1 --2- 172.21.15.78:0/1180990929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e307680 0x55c45e316b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f69e3e00700 1 -- 172.21.15.78:0/1180990929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e311ec0 msgr2=0x55c45e372180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:48.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f69e3e00700 1 --2- 172.21.15.78:0/1180990929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e311ec0 0x55c45e372180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f69e3e00700 1 -- 172.21.15.78:0/1180990929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69dc009d60 con 0x55c45e307fa0 2024-03-14T10:01:48.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f69e3e00700 1 --2- 172.21.15.78:0/1180990929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307fa0 0x55c45e317070 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f69d40a2970 tx=0x7f69d40047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bce1aa8613c14c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:48.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f69cfe00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:48.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f69cfe00700 1 -- 172.21.15.78:0/1180990929 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d40b3140 con 0x55c45e307fa0 2024-03-14T10:01:48.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f69cfe00700 1 -- 172.21.15.78:0/1180990929 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f69d409e8e0 con 0x55c45e307fa0 2024-03-14T10:01:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f6a04f26b80 1 -- 172.21.15.78:0/1180990929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f69dc004390 con 0x55c45e307fa0 2024-03-14T10:01:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f69cfe00700 1 -- 172.21.15.78:0/1180990929 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69d409a410 con 0x55c45e307fa0 2024-03-14T10:01:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f6a04f26b80 10 client.17076 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.582+0000 7f6a04f26b80 1 -- 172.21.15.78:0/1180990929 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c45e21fea0 con 0x55c45e307fa0 2024-03-14T10:01:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.583+0000 7f69cfe00700 1 -- 172.21.15.78:0/1180990929 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f69dc004390 con 0x55c45e307fa0 2024-03-14T10:01:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.583+0000 7f69cfe00700 1 -- 172.21.15.78:0/1180990929 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f69d400fc20 con 0x55c45e307fa0 2024-03-14T10:01:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.583+0000 7f6a04f26b80 10 client.17076 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.583+0000 7f6a04f26b80 10 client.17076 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:01:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.583+0000 7f6a04f26b80 20 client.17076 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.583+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/1180990929 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c45e373a00 0x55c45e375df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.583+0000 7f6a04f26b80 4 client.17076 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:01:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.583+0000 7f6a04f26b80 1 -- 172.21.15.78:0/1180990929 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c45e0a8cf0 con 0x55c45e373a00 2024-03-14T10:01:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.583+0000 7f69ea800700 1 --2- 172.21.15.78:0/1180990929 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c45e373a00 0x55c45e375df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f69cfe00700 10 client.17076 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:48.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f69ea800700 1 --2- 172.21.15.78:0/1180990929 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c45e373a00 0x55c45e375df0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4d10058179ade48 server_cookie=87476ea12844b3d8 in_seq=0 out_seq=0 2024-03-14T10:01:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f69cfe00700 1 -- 172.21.15.78:0/1180990929 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55c45e0a8cf0 con 0x55c45e373a00 2024-03-14T10:01:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f69cfe00700 10 client.17076 handle_command_reply: tid=0 2024-03-14T10:01:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f6a04f26b80 1 client.17076 shutdown 2024-03-14T10:01:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f6a04f26b80 1 -- 172.21.15.78:0/1180990929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307fa0 msgr2=0x55c45e317070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/1180990929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307fa0 0x55c45e317070 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f69d40a2970 tx=0x7f69d40047a0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f6a04f26b80 1 -- 172.21.15.78:0/1180990929 shutdown_connections 2024-03-14T10:01:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/1180990929 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c45e373a00 0x55c45e375df0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/1180990929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c45e311ec0 0x55c45e372180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/1180990929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c45e307680 0x55c45e316b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f69cfe00700 0 client.17076 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f6a04f26b80 1 --2- 172.21.15.78:0/1180990929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c45e307fa0 0x55c45e317070 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f6a04f26b80 1 -- 172.21.15.78:0/1180990929 >> 172.21.15.78:0/1180990929 conn(0x55c45e27d220 msgr2=0x55c45e31d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f6a04f26b80 1 -- 172.21.15.78:0/1180990929 shutdown_connections 2024-03-14T10:01:48.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f6a04f26b80 1 -- 172.21.15.78:0/1180990929 wait complete. 2024-03-14T10:01:48.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.584+0000 7f6a04f26b80 20 client.17076 trim_cache size 0 max 16384 2024-03-14T10:01:48.591 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:01:48.591 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T10:01:48.591 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T10:01:48.591 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:01:48.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.585+0000 7f69f1400700 1 -- 172.21.15.78:0/1754897406 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f69d00a0c00 msgr2=0x7f69d00a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:48.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.585+0000 7f69f1400700 1 --2- 172.21.15.78:0/1754897406 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f69d00a0c00 0x7f69d00a30c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f69e4003fd0 tx=0x7f69e4004b40 comp rx=0 tx=0).stop 2024-03-14T10:01:48.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.585+0000 7f69f1400700 1 -- 172.21.15.78:0/1754897406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69ec106040 msgr2=0x7f69ec19f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:48.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.585+0000 7f69f1400700 1 --2- 172.21.15.78:0/1754897406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69ec106040 0x7f69ec19f9a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f69d4000c00 tx=0x7f69d4004730 comp rx=0 tx=0).stop 2024-03-14T10:01:48.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.585+0000 7f69f1400700 1 -- 172.21.15.78:0/1754897406 shutdown_connections 2024-03-14T10:01:48.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.585+0000 7f69f1400700 1 --2- 172.21.15.78:0/1754897406 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f69ec1996b0 0x7f69ec199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.585+0000 7f69f1400700 1 --2- 172.21.15.78:0/1754897406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69ec100f20 0x7f69ec19f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.585+0000 7f69f1400700 1 --2- 172.21.15.78:0/1754897406 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f69d00a0c00 0x7f69d00a30c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.585+0000 7f69f1400700 1 --2- 172.21.15.78:0/1754897406 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f69ec106040 0x7f69ec19f9a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.585+0000 7f69f1400700 1 -- 172.21.15.78:0/1754897406 >> 172.21.15.78:0/1754897406 conn(0x7f69ec0fc7a0 msgr2=0x7f69ec0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:48.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.585+0000 7f69f1400700 1 -- 172.21.15.78:0/1754897406 shutdown_connections 2024-03-14T10:01:48.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.585+0000 7f69f1400700 1 -- 172.21.15.78:0/1754897406 wait complete. 2024-03-14T10:01:48.604 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T10:01:48.604 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T10:01:48.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.798+0000 7ff3b2800700 1 -- 172.21.15.78:0/3520849886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ac100e30 msgr2=0x7ff3ac101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:48.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.798+0000 7ff3b2800700 1 --2- 172.21.15.78:0/3520849886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ac100e30 0x7ff3ac101210 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff39c009990 tx=0x7ff39c009ca0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.798+0000 7ff3b2800700 1 -- 172.21.15.78:0/3520849886 shutdown_connections 2024-03-14T10:01:48.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.798+0000 7ff3b2800700 1 --2- 172.21.15.78:0/3520849886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3ac105f50 0x7ff3ac10dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.798+0000 7ff3b2800700 1 --2- 172.21.15.78:0/3520849886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ac100e30 0x7ff3ac101210 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.798+0000 7ff3b2800700 1 --2- 172.21.15.78:0/3520849886 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3ac1017e0 0x7ff3ac105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.798+0000 7ff3b2800700 1 -- 172.21.15.78:0/3520849886 >> 172.21.15.78:0/3520849886 conn(0x7ff3ac0fc730 msgr2=0x7ff3ac0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:48.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.798+0000 7ff3b2800700 1 -- 172.21.15.78:0/3520849886 shutdown_connections 2024-03-14T10:01:48.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.798+0000 7ff3b2800700 1 -- 172.21.15.78:0/3520849886 wait complete. 2024-03-14T10:01:48.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.799+0000 7ff3b2800700 1 Processor -- start 2024-03-14T10:01:48.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.799+0000 7ff3b2800700 1 -- start start 2024-03-14T10:01:48.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.799+0000 7ff3b2800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ac100e30 0x7ff3ac19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.799+0000 7ff3b2800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3ac105f50 0x7ff3ac19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.799+0000 7ff3b2800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3ac199650 0x7ff3ac199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.799+0000 7ff3b2800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac19a0d0 con 0x7ff3ac100e30 2024-03-14T10:01:48.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.799+0000 7ff3b2800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac19a240 con 0x7ff3ac105f50 2024-03-14T10:01:48.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.799+0000 7ff3b2800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac19a400 con 0x7ff3ac199650 2024-03-14T10:01:48.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.799+0000 7ff3ab400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ac100e30 0x7ff3ac19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.800+0000 7ff3ab400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ac100e30 0x7ff3ac19f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42194/0 (socket says 172.21.15.78:42194) 2024-03-14T10:01:48.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.800+0000 7ff3ab400700 1 -- 172.21.15.78:0/355121 learned_addr learned my addr 172.21.15.78:0/355121 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:48.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.800+0000 7ff3a3e00700 1 --2- 172.21.15.78:0/355121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3ac105f50 0x7ff3ac19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.800+0000 7ff3abe00700 1 --2- 172.21.15.78:0/355121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3ac199650 0x7ff3ac199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.800+0000 7ff3ab400700 1 -- 172.21.15.78:0/355121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3ac199650 msgr2=0x7ff3ac199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:48.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.800+0000 7ff3ab400700 1 --2- 172.21.15.78:0/355121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3ac199650 0x7ff3ac199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.800+0000 7ff3ab400700 1 -- 172.21.15.78:0/355121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3ac105f50 msgr2=0x7ff3ac19fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:48.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.800+0000 7ff3ab400700 1 --2- 172.21.15.78:0/355121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3ac105f50 0x7ff3ac19fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:48.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.800+0000 7ff3ab400700 1 -- 172.21.15.78:0/355121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff39c0095f0 con 0x7ff3ac100e30 2024-03-14T10:01:48.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.800+0000 7ff3abe00700 1 --2- 172.21.15.78:0/355121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3ac199650 0x7ff3ac199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:01:48.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.800+0000 7ff3ab400700 1 --2- 172.21.15.78:0/355121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ac100e30 0x7ff3ac19f520 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff39c005910 tx=0x7ff39c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:48.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.800+0000 7ff3a8c00700 1 -- 172.21.15.78:0/355121 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c01d070 con 0x7ff3ac100e30 2024-03-14T10:01:48.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.800+0000 7ff3a8c00700 1 -- 172.21.15.78:0/355121 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff39c022470 con 0x7ff3ac100e30 2024-03-14T10:01:48.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.800+0000 7ff3b2800700 1 -- 172.21.15.78:0/355121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ac1a6390 con 0x7ff3ac100e30 2024-03-14T10:01:48.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.800+0000 7ff3a8c00700 1 -- 172.21.15.78:0/355121 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c017650 con 0x7ff3ac100e30 2024-03-14T10:01:48.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.801+0000 7ff3b2800700 1 -- 172.21.15.78:0/355121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ac1a6810 con 0x7ff3ac100e30 2024-03-14T10:01:48.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.802+0000 7ff3b2800700 1 -- 172.21.15.78:0/355121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ac102300 con 0x7ff3ac100e30 2024-03-14T10:01:48.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.803+0000 7ff3a8c00700 1 -- 172.21.15.78:0/355121 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7ff39c0225e0 con 0x7ff3ac100e30 2024-03-14T10:01:48.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.805+0000 7ff3a8c00700 1 --2- 172.21.15.78:0/355121 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff3900a0c80 0x7ff3900a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:48.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.806+0000 7ff3a8c00700 1 -- 172.21.15.78:0/355121 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7ff39c0cdbf0 con 0x7ff3ac100e30 2024-03-14T10:01:48.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.806+0000 7ff3a3e00700 1 --2- 172.21.15.78:0/355121 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff3900a0c80 0x7ff3900a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:48.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.806+0000 7ff3a3e00700 1 --2- 172.21.15.78:0/355121 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff3900a0c80 0x7ff3900a3140 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac19ad60 tx=0x7ff394009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:48.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:48.810+0000 7ff3a8c00700 1 -- 172.21.15.78:0/355121 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff39c097000 con 0x7ff3ac100e30 2024-03-14T10:01:49.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:48 smithi078 ceph-mon[38143]: pgmap v234: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:49.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.062+0000 7ff3b2800700 1 -- 172.21.15.78:0/355121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3ac0679e0 con 0x7ff3ac100e30 2024-03-14T10:01:49.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.062+0000 7ff3a8c00700 1 -- 172.21.15.78:0/355121 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7ff39c027020 con 0x7ff3ac100e30 2024-03-14T10:01:49.063 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T10:01:49.063 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T10:01:49.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.065+0000 7ff3b2800700 1 -- 172.21.15.78:0/355121 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff3900a0c80 msgr2=0x7ff3900a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.065+0000 7ff3b2800700 1 --2- 172.21.15.78:0/355121 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff3900a0c80 0x7ff3900a3140 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac19ad60 tx=0x7ff394009260 comp rx=0 tx=0).stop 2024-03-14T10:01:49.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.065+0000 7ff3b2800700 1 -- 172.21.15.78:0/355121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ac100e30 msgr2=0x7ff3ac19f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.065+0000 7ff3b2800700 1 --2- 172.21.15.78:0/355121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ac100e30 0x7ff3ac19f520 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff39c005910 tx=0x7ff39c004730 comp rx=0 tx=0).stop 2024-03-14T10:01:49.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.065+0000 7ff3b2800700 1 -- 172.21.15.78:0/355121 shutdown_connections 2024-03-14T10:01:49.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.065+0000 7ff3b2800700 1 --2- 172.21.15.78:0/355121 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff3ac199650 0x7ff3ac199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.065+0000 7ff3b2800700 1 --2- 172.21.15.78:0/355121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3ac100e30 0x7ff3ac19f520 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.065+0000 7ff3b2800700 1 --2- 172.21.15.78:0/355121 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff3900a0c80 0x7ff3900a3140 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.065+0000 7ff3b2800700 1 --2- 172.21.15.78:0/355121 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3ac105f50 0x7ff3ac19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.065+0000 7ff3b2800700 1 -- 172.21.15.78:0/355121 >> 172.21.15.78:0/355121 conn(0x7ff3ac0fc730 msgr2=0x7ff3ac0fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.065+0000 7ff3b2800700 1 -- 172.21.15.78:0/355121 shutdown_connections 2024-03-14T10:01:49.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.065+0000 7ff3b2800700 1 -- 172.21.15.78:0/355121 wait complete. 2024-03-14T10:01:49.067 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T10:01:49.085 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T10:01:49.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:48 smithi152 ceph-mon[48279]: pgmap v234: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:49.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:48 smithi132 ceph-mon[47959]: pgmap v234: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.282+0000 7fc7b3a00700 1 -- 172.21.15.78:0/1929284692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7ac100f20 msgr2=0x7fc7ac101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.282+0000 7fc7b3a00700 1 --2- 172.21.15.78:0/1929284692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7ac100f20 0x7fc7ac101300 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0009990 tx=0x7fc7a0009ca0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.282+0000 7fc7b3a00700 1 -- 172.21.15.78:0/1929284692 shutdown_connections 2024-03-14T10:01:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.282+0000 7fc7b3a00700 1 --2- 172.21.15.78:0/1929284692 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7ac106040 0x7fc7ac10dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.282+0000 7fc7b3a00700 1 --2- 172.21.15.78:0/1929284692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7ac100f20 0x7fc7ac101300 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.282+0000 7fc7b3a00700 1 --2- 172.21.15.78:0/1929284692 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7ac1018d0 0x7fc7ac105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.282+0000 7fc7b3a00700 1 -- 172.21.15.78:0/1929284692 >> 172.21.15.78:0/1929284692 conn(0x7fc7ac0fc7a0 msgr2=0x7fc7ac0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.282+0000 7fc7b3a00700 1 -- 172.21.15.78:0/1929284692 shutdown_connections 2024-03-14T10:01:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.282+0000 7fc7b3a00700 1 -- 172.21.15.78:0/1929284692 wait complete. 2024-03-14T10:01:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.283+0000 7fc7b3a00700 1 Processor -- start 2024-03-14T10:01:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.283+0000 7fc7b3a00700 1 -- start start 2024-03-14T10:01:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.283+0000 7fc7b3a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7ac100f20 0x7fc7ac19f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7b3a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7ac106040 0x7fc7ac19fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7b3a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7ac1996c0 0x7fc7ac199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7b3a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac19a140 con 0x7fc7ac1996c0 2024-03-14T10:01:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7b3a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac19a2b0 con 0x7fc7ac100f20 2024-03-14T10:01:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7b3a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac19a470 con 0x7fc7ac106040 2024-03-14T10:01:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7b1a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7ac1996c0 0x7fc7ac199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7b1000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7ac100f20 0x7fc7ac19f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7b1000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7ac100f20 0x7fc7ac19f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:49636/0 (socket says 172.21.15.78:49636) 2024-03-14T10:01:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7b1a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7ac1996c0 0x7fc7ac199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42206/0 (socket says 172.21.15.78:42206) 2024-03-14T10:01:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7b1000700 1 -- 172.21.15.78:0/2228635934 learned_addr learned my addr 172.21.15.78:0/2228635934 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7abe00700 1 --2- 172.21.15.78:0/2228635934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7ac106040 0x7fc7ac19fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7b1000700 1 -- 172.21.15.78:0/2228635934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7ac106040 msgr2=0x7fc7ac19fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7b1000700 1 --2- 172.21.15.78:0/2228635934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7ac106040 0x7fc7ac19fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7b1000700 1 -- 172.21.15.78:0/2228635934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7ac1996c0 msgr2=0x7fc7ac199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7b1000700 1 --2- 172.21.15.78:0/2228635934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7ac1996c0 0x7fc7ac199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7b1000700 1 -- 172.21.15.78:0/2228635934 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a00095f0 con 0x7fc7ac100f20 2024-03-14T10:01:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.284+0000 7fc7b1a00700 1 --2- 172.21.15.78:0/2228635934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7ac1996c0 0x7fc7ac199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.285+0000 7fc7b1000700 1 --2- 172.21.15.78:0/2228635934 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7ac100f20 0x7fc7ac19f540 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0005910 tx=0x7fc7a0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.285+0000 7fc7a9600700 1 -- 172.21.15.78:0/2228635934 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a001d070 con 0x7fc7ac100f20 2024-03-14T10:01:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.285+0000 7fc7a9600700 1 -- 172.21.15.78:0/2228635934 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc7a0022470 con 0x7fc7ac100f20 2024-03-14T10:01:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.285+0000 7fc7b3a00700 1 -- 172.21.15.78:0/2228635934 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ac1a63b0 con 0x7fc7ac100f20 2024-03-14T10:01:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.285+0000 7fc7a9600700 1 -- 172.21.15.78:0/2228635934 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0017650 con 0x7fc7ac100f20 2024-03-14T10:01:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.285+0000 7fc7b3a00700 1 -- 172.21.15.78:0/2228635934 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ac1a67d0 con 0x7fc7ac100f20 2024-03-14T10:01:49.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.288+0000 7fc7a9600700 1 -- 172.21.15.78:0/2228635934 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fc7a00225e0 con 0x7fc7ac100f20 2024-03-14T10:01:49.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.289+0000 7fc7c75e9b80 1 Processor -- start 2024-03-14T10:01:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.289+0000 7fc7c75e9b80 1 -- start start 2024-03-14T10:01:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.289+0000 7fc7c75e9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870dd6680 0x55a870dd6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.289+0000 7fc7c75e9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870de0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.289+0000 7fc7c75e9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870de0ec0 0x55a870de32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.289+0000 7fc7c75e9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a870d2f990 con 0x55a870dd6fa0 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.290+0000 7fc7c75e9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a870d2d350 con 0x55a870dd6680 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.290+0000 7fc7c75e9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a870d28c40 con 0x55a870de0ec0 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.290+0000 7fc7b1000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870de0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.290+0000 7fc7b1000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870de0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42216/0 (socket says 172.21.15.78:42216) 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.290+0000 7fc7b1000700 1 -- 172.21.15.78:0/4262400847 learned_addr learned my addr 172.21.15.78:0/4262400847 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.290+0000 7fc7b1a00700 1 --2- 172.21.15.78:0/4262400847 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870dd6680 0x55a870dd6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.290+0000 7fc7abe00700 1 --2- 172.21.15.78:0/4262400847 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870de0ec0 0x55a870de32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.290+0000 7fc7a9600700 1 --2- 172.21.15.78:0/2228635934 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fc7940a0c00 0x7fc7940a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.290+0000 7fc7a9600700 1 -- 172.21.15.78:0/2228635934 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fc7a00cdac0 con 0x7fc7ac100f20 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.290+0000 7fc7b1000700 1 -- 172.21.15.78:0/4262400847 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870de0ec0 msgr2=0x55a870de32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.290+0000 7fc7b1000700 1 --2- 172.21.15.78:0/4262400847 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870de0ec0 0x55a870de32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.290+0000 7fc7b1a00700 1 --2- 172.21.15.78:0/2228635934 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fc7940a0c00 0x7fc7940a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.290+0000 7fc7b1000700 1 -- 172.21.15.78:0/4262400847 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870dd6680 msgr2=0x55a870dd6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.290+0000 7fc7b1000700 1 --2- 172.21.15.78:0/4262400847 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870dd6680 0x55a870dd6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.290+0000 7fc7b1000700 1 -- 172.21.15.78:0/4262400847 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a870baa3b0 con 0x55a870dd6fa0 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.291+0000 7fc7b1000700 1 --2- 172.21.15.78:0/4262400847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870de0980 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fc7a009ab20 tx=0x7fc7a0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d67d26380ae17da server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.291+0000 7fc79e000700 1 -- 172.21.15.78:0/4262400847 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a009a6d0 con 0x55a870dd6fa0 2024-03-14T10:01:49.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.291+0000 7fc79e000700 1 -- 172.21.15.78:0/4262400847 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc7a009a4f0 con 0x55a870dd6fa0 2024-03-14T10:01:49.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.291+0000 7fc79e000700 1 -- 172.21.15.78:0/4262400847 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a00a2f80 con 0x55a870dd6fa0 2024-03-14T10:01:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.291+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/4262400847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 msgr2=0x55a870de0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.291+0000 7fc7b1a00700 1 --2- 172.21.15.78:0/2228635934 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fc7940a0c00 0x7fc7940a30c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc7a40060b0 tx=0x7fc7a4004b40 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.291+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/4262400847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870de0980 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fc7a009ab20 tx=0x7fc7a0009310 comp rx=0 tx=0).stop 2024-03-14T10:01:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.292+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/4262400847 shutdown_connections 2024-03-14T10:01:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.292+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/4262400847 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870dd6680 0x55a870dd6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.292+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/4262400847 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870de0ec0 0x55a870de32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.292+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/4262400847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870de0980 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.292+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/4262400847 >> 172.21.15.78:0/4262400847 conn(0x55a870d4c220 msgr2=0x55a870dd1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.292+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/4262400847 shutdown_connections 2024-03-14T10:01:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.292+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/4262400847 wait complete. 2024-03-14T10:01:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.292+0000 7fc7c75e9b80 1 -- start start 2024-03-14T10:01:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.292+0000 7fc7c75e9b80 1 Processor -- start 2024-03-14T10:01:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7c75e9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870dd6680 0x55a870de0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7c75e9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870dd6fa0 0x55a870dddaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7c75e9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870de0ec0 0x55a870dde030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7c75e9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a870cc4d80 con 0x55a870de0ec0 2024-03-14T10:01:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7c75e9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a870cc4a20 con 0x55a870dd6680 2024-03-14T10:01:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7c75e9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a870bac6a0 con 0x55a870dd6fa0 2024-03-14T10:01:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7c75e9b80 10 client.0 _ll_register_callbacks cb 0x55a870b303a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7b1a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870de0ec0 0x55a870dde030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7abe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870dd6fa0 0x55a870dddaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7b1a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870de0ec0 0x55a870dde030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42222/0 (socket says 172.21.15.78:42222) 2024-03-14T10:01:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7b1a00700 1 -- 172.21.15.78:0/2121150548 learned_addr learned my addr 172.21.15.78:0/2121150548 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7b1000700 1 --2- 172.21.15.78:0/2121150548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870dd6680 0x55a870de0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7b1a00700 1 -- 172.21.15.78:0/2121150548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870dd6fa0 msgr2=0x55a870dddaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7b1a00700 1 --2- 172.21.15.78:0/2121150548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870dd6fa0 0x55a870dddaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7b1a00700 1 -- 172.21.15.78:0/2121150548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870dd6680 msgr2=0x55a870de0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7b1a00700 1 --2- 172.21.15.78:0/2121150548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870dd6680 0x55a870de0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.293+0000 7fc7b1a00700 1 -- 172.21.15.78:0/2121150548 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ac1a67d0 con 0x55a870de0ec0 2024-03-14T10:01:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.294+0000 7fc7b1a00700 1 --2- 172.21.15.78:0/2121150548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870de0ec0 0x55a870dde030 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400eb10 tx=0x7fc7a4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5946640fd38e7ecc server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.294+0000 7fc792a00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.294+0000 7fc792a00700 1 -- 172.21.15.78:0/2121150548 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a4021070 con 0x55a870de0ec0 2024-03-14T10:01:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.294+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/2121150548 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a870dc9720 con 0x55a870de0ec0 2024-03-14T10:01:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.294+0000 7fc792a00700 1 -- 172.21.15.78:0/2121150548 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fc7a4007910 con 0x55a870de0ec0 2024-03-14T10:01:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.294+0000 7fc792a00700 1 -- 172.21.15.78:0/2121150548 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7a4007ad0 con 0x55a870de0ec0 2024-03-14T10:01:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.294+0000 7fc7c75e9b80 10 client.17094 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.294+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/2121150548 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a870b56b60 con 0x55a870de0ec0 2024-03-14T10:01:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.294+0000 7fc792a00700 1 -- 172.21.15.78:0/2121150548 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a870dc9720 con 0x55a870de0ec0 2024-03-14T10:01:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.295+0000 7fc792a00700 1 -- 172.21.15.78:0/2121150548 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc7a4026420 con 0x55a870de0ec0 2024-03-14T10:01:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.295+0000 7fc7c75e9b80 10 client.17094 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.295+0000 7fc7c75e9b80 10 client.17094 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:01:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.295+0000 7fc7c75e9b80 20 client.17094 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.295+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/2121150548 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55a870df07f0 0x55a870df2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.295+0000 7fc7c75e9b80 4 client.17094 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:01:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.295+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/2121150548 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a870d21700 con 0x55a870df07f0 2024-03-14T10:01:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.295+0000 7fc7b1000700 1 --2- 172.21.15.78:0/2121150548 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55a870df07f0 0x55a870df2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.296+0000 7fc792a00700 10 client.17094 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:01:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.296+0000 7fc7b1000700 1 --2- 172.21.15.78:0/2121150548 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55a870df07f0 0x55a870df2c10 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f3a85f07174fd917 server_cookie=e13af3fa45e72d6e in_seq=0 out_seq=0 2024-03-14T10:01:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.296+0000 7fc792a00700 1 -- 172.21.15.78:0/2121150548 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55a870d21700 con 0x55a870df07f0 2024-03-14T10:01:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.296+0000 7fc792a00700 10 client.17094 handle_command_reply: tid=0 2024-03-14T10:01:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.296+0000 7fc7c75e9b80 1 client.17094 shutdown 2024-03-14T10:01:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.296+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/2121150548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870de0ec0 msgr2=0x55a870dde030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.296+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/2121150548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870de0ec0 0x55a870dde030 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400eb10 tx=0x7fc7a4004ad0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.296+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/2121150548 shutdown_connections 2024-03-14T10:01:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.296+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/2121150548 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55a870df07f0 0x55a870df2c10 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.296+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/2121150548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870dd6680 0x55a870de0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.296+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/2121150548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870de0ec0 0x55a870dde030 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.296+0000 7fc792a00700 0 client.17094 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:01:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.296+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/2121150548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870dd6fa0 0x55a870dddaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.296+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/2121150548 >> 172.21.15.78:0/2121150548 conn(0x55a870d4c220 msgr2=0x55a870dd1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.297+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/2121150548 shutdown_connections 2024-03-14T10:01:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.297+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/2121150548 wait complete. 2024-03-14T10:01:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.297+0000 7fc7c75e9b80 20 client.17094 trim_cache size 0 max 16384 2024-03-14T10:01:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.310+0000 7fc7c75e9b80 1 Processor -- start 2024-03-14T10:01:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.310+0000 7fc7c75e9b80 1 -- start start 2024-03-14T10:01:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.310+0000 7fc7c75e9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870dd6680 0x55a870dd8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.310+0000 7fc7c75e9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870dd9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.310+0000 7fc7c75e9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870de0ec0 0x55a870ddc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.310+0000 7fc7c75e9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a870dd7af0 con 0x55a870dd6fa0 2024-03-14T10:01:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.310+0000 7fc7c75e9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a870dce230 con 0x55a870de0ec0 2024-03-14T10:01:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.310+0000 7fc7c75e9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a870dd3060 con 0x55a870dd6680 2024-03-14T10:01:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.310+0000 7fc7abe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870dd9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.310+0000 7fc7abe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870dd9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42236/0 (socket says 172.21.15.78:42236) 2024-03-14T10:01:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.310+0000 7fc7abe00700 1 -- 172.21.15.78:0/2102161707 learned_addr learned my addr 172.21.15.78:0/2102161707 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.311+0000 7fc7b1a00700 1 --2- 172.21.15.78:0/2102161707 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870de0ec0 0x55a870ddc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.311+0000 7fc7b1000700 1 --2- 172.21.15.78:0/2102161707 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870dd6680 0x55a870dd8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.311+0000 7fc7abe00700 1 -- 172.21.15.78:0/2102161707 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870dd6680 msgr2=0x55a870dd8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.311+0000 7fc7abe00700 1 --2- 172.21.15.78:0/2102161707 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870dd6680 0x55a870dd8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.311+0000 7fc7abe00700 1 -- 172.21.15.78:0/2102161707 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870de0ec0 msgr2=0x55a870ddc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.311+0000 7fc7abe00700 1 --2- 172.21.15.78:0/2102161707 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870de0ec0 0x55a870ddc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.311+0000 7fc7abe00700 1 -- 172.21.15.78:0/2102161707 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a870ceeea0 con 0x55a870dd6fa0 2024-03-14T10:01:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.311+0000 7fc7b1000700 1 --2- 172.21.15.78:0/2102161707 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870dd6680 0x55a870dd8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:01:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.311+0000 7fc7abe00700 1 --2- 172.21.15.78:0/2102161707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870dd9360 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc798009f80 tx=0x7fc798004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6981376d9eb8c29e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.311+0000 7fc79ea00700 1 -- 172.21.15.78:0/2102161707 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79801f070 con 0x55a870dd6fa0 2024-03-14T10:01:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.311+0000 7fc79ea00700 1 -- 172.21.15.78:0/2102161707 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc798017040 con 0x55a870dd6fa0 2024-03-14T10:01:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.311+0000 7fc79ea00700 1 -- 172.21.15.78:0/2102161707 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc798016070 con 0x55a870dd6fa0 2024-03-14T10:01:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.312+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/2102161707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 msgr2=0x55a870dd9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.312+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/2102161707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870dd9360 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc798009f80 tx=0x7fc798004320 comp rx=0 tx=0).stop 2024-03-14T10:01:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.312+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/2102161707 shutdown_connections 2024-03-14T10:01:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.312+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/2102161707 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870de0ec0 0x55a870ddc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.312+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/2102161707 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870dd6680 0x55a870dd8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.312+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/2102161707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870dd9360 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.312+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/2102161707 >> 172.21.15.78:0/2102161707 conn(0x55a870d4c220 msgr2=0x55a870dd15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.312+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/2102161707 shutdown_connections 2024-03-14T10:01:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.312+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/2102161707 wait complete. 2024-03-14T10:01:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.312+0000 7fc7c75e9b80 1 -- start start 2024-03-14T10:01:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.313+0000 7fc7c75e9b80 1 Processor -- start 2024-03-14T10:01:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.313+0000 7fc7c75e9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870dd6680 0x55a870de46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.313+0000 7fc7c75e9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870de4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.313+0000 7fc7c75e9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870de0ec0 0x55a870e41310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.313+0000 7fc7c75e9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a870e41850 con 0x55a870dd6fa0 2024-03-14T10:01:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.313+0000 7fc7c75e9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a870e41990 con 0x55a870de0ec0 2024-03-14T10:01:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.313+0000 7fc7c75e9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a870e41ad0 con 0x55a870dd6680 2024-03-14T10:01:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.313+0000 7fc7c75e9b80 10 client.0 _ll_register_callbacks cb 0x7fc7a40094d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.313+0000 7fc7abe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870de4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.313+0000 7fc7abe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870de4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42250/0 (socket says 172.21.15.78:42250) 2024-03-14T10:01:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.313+0000 7fc7abe00700 1 -- 172.21.15.78:0/357567709 learned_addr learned my addr 172.21.15.78:0/357567709 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.313+0000 7fc7b1a00700 1 --2- 172.21.15.78:0/357567709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870de0ec0 0x55a870e41310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.313+0000 7fc7b1000700 1 --2- 172.21.15.78:0/357567709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870dd6680 0x55a870de46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.314+0000 7fc7abe00700 1 -- 172.21.15.78:0/357567709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870dd6680 msgr2=0x55a870de46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.314+0000 7fc7abe00700 1 --2- 172.21.15.78:0/357567709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870dd6680 0x55a870de46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.314+0000 7fc7abe00700 1 -- 172.21.15.78:0/357567709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870de0ec0 msgr2=0x55a870e41310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.314+0000 7fc7abe00700 1 --2- 172.21.15.78:0/357567709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870de0ec0 0x55a870e41310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.314+0000 7fc7abe00700 1 -- 172.21.15.78:0/357567709 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc798009c00 con 0x55a870dd6fa0 2024-03-14T10:01:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.314+0000 7fc7abe00700 1 --2- 172.21.15.78:0/357567709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870de4c20 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc798004b80 tx=0x7fc7980038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f78bc1da167c91 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.314+0000 7fc792a00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.314+0000 7fc792a00700 1 -- 172.21.15.78:0/357567709 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79801f030 con 0x55a870dd6fa0 2024-03-14T10:01:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.314+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/357567709 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc7a009a4f0 con 0x55a870dd6fa0 2024-03-14T10:01:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.314+0000 7fc792a00700 1 -- 172.21.15.78:0/357567709 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fc798005020 con 0x55a870dd6fa0 2024-03-14T10:01:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.314+0000 7fc792a00700 1 -- 172.21.15.78:0/357567709 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc798016040 con 0x55a870dd6fa0 2024-03-14T10:01:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.314+0000 7fc7c75e9b80 10 client.17100 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.314+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/357567709 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a870baecd0 con 0x55a870dd6fa0 2024-03-14T10:01:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.315+0000 7fc792a00700 1 -- 172.21.15.78:0/357567709 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc7a009a4f0 con 0x55a870dd6fa0 2024-03-14T10:01:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.315+0000 7fc792a00700 1 -- 172.21.15.78:0/357567709 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc79802c060 con 0x55a870dd6fa0 2024-03-14T10:01:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.315+0000 7fc7c75e9b80 10 client.17100 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.315+0000 7fc7c75e9b80 10 client.17100 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:01:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.315+0000 7fc7c75e9b80 20 client.17100 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.315+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/357567709 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55a870e43180 0x55a870e455a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.315+0000 7fc7c75e9b80 4 client.17100 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:01:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.315+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/357567709 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a870b77cf0 con 0x55a870e43180 2024-03-14T10:01:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.315+0000 7fc7b1000700 1 --2- 172.21.15.78:0/357567709 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55a870e43180 0x55a870e455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.316+0000 7fc792a00700 10 client.17100 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:01:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.316+0000 7fc7b1000700 1 --2- 172.21.15.78:0/357567709 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55a870e43180 0x55a870e455a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8a1fc7e6b07d155a server_cookie=ecbc66fad72bfa86 in_seq=0 out_seq=0 2024-03-14T10:01:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.316+0000 7fc792a00700 1 -- 172.21.15.78:0/357567709 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a870b77cf0 con 0x55a870e43180 2024-03-14T10:01:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.316+0000 7fc792a00700 10 client.17100 handle_command_reply: tid=0 2024-03-14T10:01:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.316+0000 7fc7c75e9b80 1 client.17100 shutdown 2024-03-14T10:01:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.316+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/357567709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 msgr2=0x55a870de4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.316+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/357567709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870de4c20 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc798004b80 tx=0x7fc7980038a0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.316+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/357567709 shutdown_connections 2024-03-14T10:01:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.316+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/357567709 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55a870e43180 0x55a870e455a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.316+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/357567709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a870de0ec0 0x55a870e41310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.316+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/357567709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55a870dd6680 0x55a870de46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.316+0000 7fc7c75e9b80 1 --2- 172.21.15.78:0/357567709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a870dd6fa0 0x55a870de4c20 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.316+0000 7fc792a00700 0 client.17100 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:01:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.316+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/357567709 >> 172.21.15.78:0/357567709 conn(0x55a870d4c220 msgr2=0x55a870dd8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.317+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/357567709 shutdown_connections 2024-03-14T10:01:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.317+0000 7fc7c75e9b80 1 -- 172.21.15.78:0/357567709 wait complete. 2024-03-14T10:01:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.317+0000 7fc7c75e9b80 20 client.17100 trim_cache size 0 max 16384 2024-03-14T10:01:49.321 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:01:49.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.317+0000 7fc7b3a00700 1 -- 172.21.15.78:0/2228635934 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fc7940a0c00 msgr2=0x7fc7940a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.317+0000 7fc7b3a00700 1 --2- 172.21.15.78:0/2228635934 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fc7940a0c00 0x7fc7940a30c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc7a40060b0 tx=0x7fc7a4004b40 comp rx=0 tx=0).stop 2024-03-14T10:01:49.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.317+0000 7fc7b3a00700 1 -- 172.21.15.78:0/2228635934 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7ac100f20 msgr2=0x7fc7ac19f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.317+0000 7fc7b3a00700 1 --2- 172.21.15.78:0/2228635934 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7ac100f20 0x7fc7ac19f540 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0005910 tx=0x7fc7a0004730 comp rx=0 tx=0).stop 2024-03-14T10:01:49.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.317+0000 7fc7b3a00700 1 -- 172.21.15.78:0/2228635934 shutdown_connections 2024-03-14T10:01:49.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.317+0000 7fc7b3a00700 1 --2- 172.21.15.78:0/2228635934 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fc7940a0c00 0x7fc7940a30c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.317+0000 7fc7b3a00700 1 --2- 172.21.15.78:0/2228635934 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7ac100f20 0x7fc7ac19f540 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.317+0000 7fc7b3a00700 1 --2- 172.21.15.78:0/2228635934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7ac1996c0 0x7fc7ac199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.317+0000 7fc7b3a00700 1 --2- 172.21.15.78:0/2228635934 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fc7ac106040 0x7fc7ac19fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.317+0000 7fc7b3a00700 1 -- 172.21.15.78:0/2228635934 >> 172.21.15.78:0/2228635934 conn(0x7fc7ac0fc7a0 msgr2=0x7fc7ac0fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.317+0000 7fc7b3a00700 1 -- 172.21.15.78:0/2228635934 shutdown_connections 2024-03-14T10:01:49.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.317+0000 7fc7b3a00700 1 -- 172.21.15.78:0/2228635934 wait complete. 2024-03-14T10:01:49.336 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T10:01:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.530+0000 7fb612a00700 1 -- 172.21.15.78:0/2455243707 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb60c100f30 msgr2=0x7fb60c101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.530+0000 7fb612a00700 1 --2- 172.21.15.78:0/2455243707 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb60c100f30 0x7fb60c101310 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc009990 tx=0x7fb5fc009ca0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.531+0000 7fb612a00700 1 -- 172.21.15.78:0/2455243707 shutdown_connections 2024-03-14T10:01:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.531+0000 7fb612a00700 1 --2- 172.21.15.78:0/2455243707 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb60c106050 0x7fb60c10dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.531+0000 7fb612a00700 1 --2- 172.21.15.78:0/2455243707 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb60c100f30 0x7fb60c101310 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.531+0000 7fb612a00700 1 --2- 172.21.15.78:0/2455243707 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb60c1018e0 0x7fb60c105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.531+0000 7fb612a00700 1 -- 172.21.15.78:0/2455243707 >> 172.21.15.78:0/2455243707 conn(0x7fb60c0fc7b0 msgr2=0x7fb60c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.531+0000 7fb612a00700 1 -- 172.21.15.78:0/2455243707 shutdown_connections 2024-03-14T10:01:49.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.531+0000 7fb612a00700 1 -- 172.21.15.78:0/2455243707 wait complete. 2024-03-14T10:01:49.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.532+0000 7fb612a00700 1 Processor -- start 2024-03-14T10:01:49.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.532+0000 7fb612a00700 1 -- start start 2024-03-14T10:01:49.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.532+0000 7fb612a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb60c100f30 0x7fb60c19f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.532+0000 7fb612a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb60c106050 0x7fb60c19fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.532+0000 7fb612a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb60c199640 0x7fb60c199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.532+0000 7fb612a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c19a0c0 con 0x7fb60c199640 2024-03-14T10:01:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.532+0000 7fb612a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c19a230 con 0x7fb60c106050 2024-03-14T10:01:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.532+0000 7fb612a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c19a3f0 con 0x7fb60c100f30 2024-03-14T10:01:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.532+0000 7fb60be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb60c100f30 0x7fb60c19f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.532+0000 7fb60be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb60c100f30 0x7fb60c19f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:52158/0 (socket says 172.21.15.78:52158) 2024-03-14T10:01:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.532+0000 7fb60be00700 1 -- 172.21.15.78:0/1538861548 learned_addr learned my addr 172.21.15.78:0/1538861548 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.532+0000 7fb610a00700 1 --2- 172.21.15.78:0/1538861548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb60c199640 0x7fb60c199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.533+0000 7fb603e00700 1 --2- 172.21.15.78:0/1538861548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb60c106050 0x7fb60c19fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.533+0000 7fb60be00700 1 -- 172.21.15.78:0/1538861548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb60c106050 msgr2=0x7fb60c19fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.533+0000 7fb60be00700 1 --2- 172.21.15.78:0/1538861548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb60c106050 0x7fb60c19fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.533+0000 7fb60be00700 1 -- 172.21.15.78:0/1538861548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb60c199640 msgr2=0x7fb60c199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.533+0000 7fb60be00700 1 --2- 172.21.15.78:0/1538861548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb60c199640 0x7fb60c199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.533+0000 7fb60be00700 1 -- 172.21.15.78:0/1538861548 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc0095f0 con 0x7fb60c100f30 2024-03-14T10:01:49.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.533+0000 7fb60be00700 1 --2- 172.21.15.78:0/1538861548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb60c100f30 0x7fb60c19f510 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc005770 tx=0x7fb5fc0057a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.533+0000 7fb609600700 1 -- 172.21.15.78:0/1538861548 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc01d070 con 0x7fb60c100f30 2024-03-14T10:01:49.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.533+0000 7fb612a00700 1 -- 172.21.15.78:0/1538861548 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60c1a6380 con 0x7fb60c100f30 2024-03-14T10:01:49.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.533+0000 7fb609600700 1 -- 172.21.15.78:0/1538861548 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5fc022470 con 0x7fb60c100f30 2024-03-14T10:01:49.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.533+0000 7fb609600700 1 -- 172.21.15.78:0/1538861548 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc017650 con 0x7fb60c100f30 2024-03-14T10:01:49.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.534+0000 7fb612a00700 1 -- 172.21.15.78:0/1538861548 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60c1a6800 con 0x7fb60c100f30 2024-03-14T10:01:49.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.536+0000 7fb609600700 1 -- 172.21.15.78:0/1538861548 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fb5fc00fb00 con 0x7fb60c100f30 2024-03-14T10:01:49.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.538+0000 7fb6264e8b80 1 Processor -- start 2024-03-14T10:01:49.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.538+0000 7fb6264e8b80 1 -- start start 2024-03-14T10:01:49.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb6264e8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69e4680 0x5572e69e4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb6264e8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4fa0 0x5572e69ee980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb6264e8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69eeec0 0x5572e69f12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb6264e8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e693d990 con 0x5572e69e4680 2024-03-14T10:01:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb6264e8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e693b350 con 0x5572e69e4fa0 2024-03-14T10:01:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb6264e8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e6936c40 con 0x5572e69eeec0 2024-03-14T10:01:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb609600700 1 --2- 172.21.15.78:0/1538861548 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb5f00a0c00 0x7fb5f00a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb60be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4fa0 0x5572e69ee980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb609600700 1 -- 172.21.15.78:0/1538861548 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fb5fc0cdb50 con 0x7fb60c100f30 2024-03-14T10:01:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb60be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4fa0 0x5572e69ee980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:49696/0 (socket says 172.21.15.78:49696) 2024-03-14T10:01:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb610a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69e4680 0x5572e69e4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb60be00700 1 -- 172.21.15.78:0/338278201 learned_addr learned my addr 172.21.15.78:0/338278201 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb603e00700 1 --2- 172.21.15.78:0/338278201 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69eeec0 0x5572e69f12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb610a00700 1 --2- 172.21.15.78:0/1538861548 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb5f00a0c00 0x7fb5f00a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb60be00700 1 -- 172.21.15.78:0/338278201 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69eeec0 msgr2=0x5572e69f12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb60be00700 1 --2- 172.21.15.78:0/338278201 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69eeec0 0x5572e69f12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb60be00700 1 -- 172.21.15.78:0/338278201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69e4680 msgr2=0x5572e69e4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb60be00700 1 --2- 172.21.15.78:0/338278201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69e4680 0x5572e69e4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.539+0000 7fb60be00700 1 -- 172.21.15.78:0/338278201 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572e67b83b0 con 0x5572e69e4fa0 2024-03-14T10:01:49.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.540+0000 7fb60be00700 1 --2- 172.21.15.78:0/338278201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4fa0 0x5572e69ee980 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc022bf0 tx=0x7fb5fc00ff50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d09ba2ddf16c154 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.540+0000 7fb610a00700 1 --2- 172.21.15.78:0/1538861548 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb5f00a0c00 0x7fb5f00a30c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb604009790 tx=0x7fb604009130 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.540+0000 7fb601200700 1 -- 172.21.15.78:0/338278201 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc09a580 con 0x5572e69e4fa0 2024-03-14T10:01:49.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.540+0000 7fb601200700 1 -- 172.21.15.78:0/338278201 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5fc00fdb0 con 0x5572e69e4fa0 2024-03-14T10:01:49.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.540+0000 7fb601200700 1 -- 172.21.15.78:0/338278201 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc022870 con 0x5572e69e4fa0 2024-03-14T10:01:49.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.540+0000 7fb6264e8b80 1 -- 172.21.15.78:0/338278201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4fa0 msgr2=0x5572e69ee980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.540+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/338278201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4fa0 0x5572e69ee980 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc022bf0 tx=0x7fb5fc00ff50 comp rx=0 tx=0).stop 2024-03-14T10:01:49.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.541+0000 7fb6264e8b80 1 -- 172.21.15.78:0/338278201 shutdown_connections 2024-03-14T10:01:49.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.541+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/338278201 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69eeec0 0x5572e69f12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.541+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/338278201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69e4680 0x5572e69e4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.541+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/338278201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4fa0 0x5572e69ee980 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.541+0000 7fb6264e8b80 1 -- 172.21.15.78:0/338278201 >> 172.21.15.78:0/338278201 conn(0x5572e695a220 msgr2=0x5572e69dfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.541+0000 7fb6264e8b80 1 -- 172.21.15.78:0/338278201 shutdown_connections 2024-03-14T10:01:49.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.541+0000 7fb6264e8b80 1 -- 172.21.15.78:0/338278201 wait complete. 2024-03-14T10:01:49.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.541+0000 7fb6264e8b80 1 -- start start 2024-03-14T10:01:49.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.542+0000 7fb6264e8b80 1 Processor -- start 2024-03-14T10:01:49.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.542+0000 7fb6264e8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 0x5572e69ee800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.542+0000 7fb6264e8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69e4fa0 0x5572e69ebaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.542+0000 7fb6264e8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69eeec0 0x5572e69ec0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.542+0000 7fb6264e8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e68d2d80 con 0x5572e69eeec0 2024-03-14T10:01:49.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.542+0000 7fb6264e8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e68d2a20 con 0x5572e69e4680 2024-03-14T10:01:49.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.542+0000 7fb6264e8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e67ba6a0 con 0x5572e69e4fa0 2024-03-14T10:01:49.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.542+0000 7fb6264e8b80 10 client.0 _ll_register_callbacks cb 0x5572e673e3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:49.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.543+0000 7fb60be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 0x5572e69ee800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.543+0000 7fb60be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 0x5572e69ee800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:49710/0 (socket says 172.21.15.78:49710) 2024-03-14T10:01:49.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.543+0000 7fb60be00700 1 -- 172.21.15.78:0/1779455131 learned_addr learned my addr 172.21.15.78:0/1779455131 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:49.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.543+0000 7fb610a00700 1 --2- 172.21.15.78:0/1779455131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69eeec0 0x5572e69ec0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.543+0000 7fb603e00700 1 --2- 172.21.15.78:0/1779455131 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69e4fa0 0x5572e69ebaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.543+0000 7fb60be00700 1 -- 172.21.15.78:0/1779455131 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69e4fa0 msgr2=0x5572e69ebaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.543+0000 7fb60be00700 1 --2- 172.21.15.78:0/1779455131 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69e4fa0 0x5572e69ebaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.543+0000 7fb60be00700 1 -- 172.21.15.78:0/1779455131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69eeec0 msgr2=0x5572e69ec0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.543+0000 7fb60be00700 1 --2- 172.21.15.78:0/1779455131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69eeec0 0x5572e69ec0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.543+0000 7fb60be00700 1 -- 172.21.15.78:0/1779455131 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60c1a6800 con 0x5572e69e4680 2024-03-14T10:01:49.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.543+0000 7fb60be00700 1 --2- 172.21.15.78:0/1779455131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 0x5572e69ee800 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc09abb0 tx=0x7fb5fc09a580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d34b6f5cadcdef76 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.543+0000 7fb5f9600700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T10:01:49.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.543+0000 7fb5f9600700 1 -- 172.21.15.78:0/1779455131 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc09ea50 con 0x5572e69e4680 2024-03-14T10:01:49.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.543+0000 7fb6264e8b80 1 -- 172.21.15.78:0/1779455131 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5572e69d7720 con 0x5572e69e4680 2024-03-14T10:01:49.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.543+0000 7fb5f9600700 1 -- 172.21.15.78:0/1779455131 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fb5fc09b320 con 0x5572e69e4680 2024-03-14T10:01:49.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.544+0000 7fb5f9600700 1 -- 172.21.15.78:0/1779455131 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5fc0a3010 con 0x5572e69e4680 2024-03-14T10:01:49.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.544+0000 7fb6264e8b80 10 client.27185 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:49.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.544+0000 7fb6264e8b80 1 -- 172.21.15.78:0/1779455131 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5572e6764b60 con 0x5572e69e4680 2024-03-14T10:01:49.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.544+0000 7fb5f9600700 1 -- 172.21.15.78:0/1779455131 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5572e69d7720 con 0x5572e69e4680 2024-03-14T10:01:49.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.544+0000 7fb5f9600700 1 -- 172.21.15.78:0/1779455131 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb5fc0a5050 con 0x5572e69e4680 2024-03-14T10:01:49.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.544+0000 7fb6264e8b80 10 client.27185 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:49.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.544+0000 7fb6264e8b80 10 client.27185 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:01:49.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.544+0000 7fb6264e8b80 20 client.27185 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:49.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.544+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/1779455131 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5572e69fe280 0x5572e6a006a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.544+0000 7fb6264e8b80 4 client.27185 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:01:49.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.544+0000 7fb6264e8b80 1 -- 172.21.15.78:0/1779455131 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5572e692f700 con 0x5572e69fe280 2024-03-14T10:01:49.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.544+0000 7fb603e00700 1 --2- 172.21.15.78:0/1779455131 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5572e69fe280 0x5572e6a006a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.545+0000 7fb603e00700 1 --2- 172.21.15.78:0/1779455131 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5572e69fe280 0x5572e6a006a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a1b0ad50a5c4fdd0 server_cookie=47a01870a0c9f216 in_seq=0 out_seq=0 2024-03-14T10:01:49.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.545+0000 7fb5f9600700 10 client.27185 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:01:49.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.545+0000 7fb5f9600700 1 -- 172.21.15.78:0/1779455131 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5572e692f700 con 0x5572e69fe280 2024-03-14T10:01:49.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.545+0000 7fb5f9600700 10 client.27185 handle_command_reply: tid=0 2024-03-14T10:01:49.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.545+0000 7fb6264e8b80 1 client.27185 shutdown 2024-03-14T10:01:49.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.546+0000 7fb6264e8b80 1 -- 172.21.15.78:0/1779455131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 msgr2=0x5572e69ee800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.546+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/1779455131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 0x5572e69ee800 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc09abb0 tx=0x7fb5fc09a580 comp rx=0 tx=0).stop 2024-03-14T10:01:49.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.546+0000 7fb6264e8b80 1 -- 172.21.15.78:0/1779455131 shutdown_connections 2024-03-14T10:01:49.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.546+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/1779455131 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5572e69fe280 0x5572e6a006a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.546+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/1779455131 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 0x5572e69ee800 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.546+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/1779455131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69eeec0 0x5572e69ec0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.546+0000 7fb5f9600700 0 client.27185 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:01:49.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.546+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/1779455131 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69e4fa0 0x5572e69ebaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.546+0000 7fb6264e8b80 1 -- 172.21.15.78:0/1779455131 >> 172.21.15.78:0/1779455131 conn(0x5572e695a220 msgr2=0x5572e69df690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.546+0000 7fb6264e8b80 1 -- 172.21.15.78:0/1779455131 shutdown_connections 2024-03-14T10:01:49.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.546+0000 7fb6264e8b80 1 -- 172.21.15.78:0/1779455131 wait complete. 2024-03-14T10:01:49.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.546+0000 7fb6264e8b80 20 client.27185 trim_cache size 0 max 16384 2024-03-14T10:01:49.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.559+0000 7fb6264e8b80 1 Processor -- start 2024-03-14T10:01:49.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.559+0000 7fb6264e8b80 1 -- start start 2024-03-14T10:01:49.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.559+0000 7fb6264e8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 0x5572e69ec050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.559+0000 7fb6264e8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69e4fa0 0x5572e69eb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.559+0000 7fb6264e8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69eeec0 0x5572e69f9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.559+0000 7fb6264e8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e69e1060 con 0x5572e69eeec0 2024-03-14T10:01:49.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.559+0000 7fb6264e8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e6a05380 con 0x5572e69e4680 2024-03-14T10:01:49.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.559+0000 7fb6264e8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e69e6700 con 0x5572e69e4fa0 2024-03-14T10:01:49.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.560+0000 7fb60be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 0x5572e69ec050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.560+0000 7fb60be00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 0x5572e69ec050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:49716/0 (socket says 172.21.15.78:49716) 2024-03-14T10:01:49.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.560+0000 7fb60be00700 1 -- 172.21.15.78:0/2659329876 learned_addr learned my addr 172.21.15.78:0/2659329876 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:49.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.560+0000 7fb610a00700 1 --2- 172.21.15.78:0/2659329876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69eeec0 0x5572e69f9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.560+0000 7fb603e00700 1 --2- 172.21.15.78:0/2659329876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69e4fa0 0x5572e69eb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.560+0000 7fb60be00700 1 -- 172.21.15.78:0/2659329876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69e4fa0 msgr2=0x5572e69eb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.560+0000 7fb60be00700 1 --2- 172.21.15.78:0/2659329876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69e4fa0 0x5572e69eb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.560+0000 7fb60be00700 1 -- 172.21.15.78:0/2659329876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69eeec0 msgr2=0x5572e69f9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.560+0000 7fb60be00700 1 --2- 172.21.15.78:0/2659329876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69eeec0 0x5572e69f9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.560+0000 7fb60be00700 1 -- 172.21.15.78:0/2659329876 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572e68fcea0 con 0x5572e69e4680 2024-03-14T10:01:49.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.560+0000 7fb603e00700 1 --2- 172.21.15.78:0/2659329876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69e4fa0 0x5572e69eb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:01:49.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.560+0000 7fb60be00700 1 --2- 172.21.15.78:0/2659329876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 0x5572e69ec050 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc005740 tx=0x7fb5fc00b650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8454dc352d39199a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.560+0000 7fb601c00700 1 -- 172.21.15.78:0/2659329876 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0a2b80 con 0x5572e69e4680 2024-03-14T10:01:49.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.560+0000 7fb601c00700 1 -- 172.21.15.78:0/2659329876 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5fc09a110 con 0x5572e69e4680 2024-03-14T10:01:49.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.560+0000 7fb601c00700 1 -- 172.21.15.78:0/2659329876 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc09ea50 con 0x5572e69e4680 2024-03-14T10:01:49.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.561+0000 7fb6264e8b80 1 -- 172.21.15.78:0/2659329876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 msgr2=0x5572e69ec050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.561+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/2659329876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 0x5572e69ec050 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc005740 tx=0x7fb5fc00b650 comp rx=0 tx=0).stop 2024-03-14T10:01:49.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.561+0000 7fb6264e8b80 1 -- 172.21.15.78:0/2659329876 shutdown_connections 2024-03-14T10:01:49.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.561+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/2659329876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 0x5572e69ec050 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.561+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/2659329876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69eeec0 0x5572e69f9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.561+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/2659329876 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69e4fa0 0x5572e69eb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.561+0000 7fb6264e8b80 1 -- 172.21.15.78:0/2659329876 >> 172.21.15.78:0/2659329876 conn(0x5572e695a220 msgr2=0x5572e69e6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.561+0000 7fb6264e8b80 1 -- 172.21.15.78:0/2659329876 shutdown_connections 2024-03-14T10:01:49.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.561+0000 7fb6264e8b80 1 -- 172.21.15.78:0/2659329876 wait complete. 2024-03-14T10:01:49.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.561+0000 7fb6264e8b80 1 -- start start 2024-03-14T10:01:49.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.562+0000 7fb6264e8b80 1 Processor -- start 2024-03-14T10:01:49.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.562+0000 7fb6264e8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 0x5572e69f5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.562+0000 7fb6264e8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69e4fa0 0x5572e69f55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.562+0000 7fb6264e8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69eeec0 0x5572e69f3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.562+0000 7fb6264e8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e69f3df0 con 0x5572e69e4fa0 2024-03-14T10:01:49.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.562+0000 7fb6264e8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e69f3f60 con 0x5572e69e4680 2024-03-14T10:01:49.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.562+0000 7fb6264e8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572e69f4120 con 0x5572e69eeec0 2024-03-14T10:01:49.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.562+0000 7fb6264e8b80 10 client.0 _ll_register_callbacks cb 0x7fb604009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:49.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.562+0000 7fb603e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69e4fa0 0x5572e69f55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.562+0000 7fb603e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69e4fa0 0x5572e69f55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42298/0 (socket says 172.21.15.78:42298) 2024-03-14T10:01:49.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.562+0000 7fb603e00700 1 -- 172.21.15.78:0/2671954626 learned_addr learned my addr 172.21.15.78:0/2671954626 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:49.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.563+0000 7fb610a00700 1 --2- 172.21.15.78:0/2671954626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69eeec0 0x5572e69f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.563+0000 7fb60be00700 1 --2- 172.21.15.78:0/2671954626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 0x5572e69f5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.563+0000 7fb603e00700 1 -- 172.21.15.78:0/2671954626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69eeec0 msgr2=0x5572e69f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.563+0000 7fb603e00700 1 --2- 172.21.15.78:0/2671954626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69eeec0 0x5572e69f3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.563+0000 7fb603e00700 1 -- 172.21.15.78:0/2671954626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 msgr2=0x5572e69f5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.563+0000 7fb603e00700 1 --2- 172.21.15.78:0/2671954626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 0x5572e69f5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.563+0000 7fb603e00700 1 -- 172.21.15.78:0/2671954626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572e6764b60 con 0x5572e69e4fa0 2024-03-14T10:01:49.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.563+0000 7fb603e00700 1 --2- 172.21.15.78:0/2671954626 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69e4fa0 0x5572e69f55c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fb60c19ad50 tx=0x7fb5f4004480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b703ad8ec8e32141 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.563+0000 7fb5f9600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:49.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.563+0000 7fb5f9600700 1 -- 172.21.15.78:0/2671954626 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4016040 con 0x5572e69e4fa0 2024-03-14T10:01:49.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.563+0000 7fb5f9600700 1 -- 172.21.15.78:0/2671954626 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fb5f4003c60 con 0x5572e69e4fa0 2024-03-14T10:01:49.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.563+0000 7fb6264e8b80 1 -- 172.21.15.78:0/2671954626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb5fc00fdb0 con 0x5572e69e4fa0 2024-03-14T10:01:49.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.564+0000 7fb5f9600700 1 -- 172.21.15.78:0/2671954626 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5f4003e20 con 0x5572e69e4fa0 2024-03-14T10:01:49.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.564+0000 7fb6264e8b80 10 client.17118 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:49.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.564+0000 7fb6264e8b80 1 -- 172.21.15.78:0/2671954626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5572e67bccd0 con 0x5572e69e4fa0 2024-03-14T10:01:49.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.564+0000 7fb5f9600700 1 -- 172.21.15.78:0/2671954626 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5fc00fdb0 con 0x5572e69e4fa0 2024-03-14T10:01:49.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.564+0000 7fb5f9600700 1 -- 172.21.15.78:0/2671954626 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb5f4010bb0 con 0x5572e69e4fa0 2024-03-14T10:01:49.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.564+0000 7fb6264e8b80 10 client.17118 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:49.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.564+0000 7fb6264e8b80 10 client.17118 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:01:49.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.564+0000 7fb6264e8b80 20 client.17118 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:49.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.564+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/2671954626 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5572e6a510f0 0x5572e6a53510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.564+0000 7fb6264e8b80 4 client.17118 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:01:49.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.564+0000 7fb6264e8b80 1 -- 172.21.15.78:0/2671954626 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5572e6785cf0 con 0x5572e6a510f0 2024-03-14T10:01:49.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.564+0000 7fb60be00700 1 --2- 172.21.15.78:0/2671954626 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5572e6a510f0 0x5572e6a53510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.565+0000 7fb60be00700 1 --2- 172.21.15.78:0/2671954626 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5572e6a510f0 0x5572e6a53510 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9b50caa73d10b5c7 server_cookie=671a00fd1c077222 in_seq=0 out_seq=0 2024-03-14T10:01:49.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.565+0000 7fb5f9600700 10 client.17118 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:01:49.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.565+0000 7fb5f9600700 1 -- 172.21.15.78:0/2671954626 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5572e6785cf0 con 0x5572e6a510f0 2024-03-14T10:01:49.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.565+0000 7fb5f9600700 10 client.17118 handle_command_reply: tid=0 2024-03-14T10:01:49.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.565+0000 7fb6264e8b80 1 client.17118 shutdown 2024-03-14T10:01:49.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.565+0000 7fb6264e8b80 1 -- 172.21.15.78:0/2671954626 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69e4fa0 msgr2=0x5572e69f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.565+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/2671954626 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69e4fa0 0x5572e69f55c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fb60c19ad50 tx=0x7fb5f4004480 comp rx=0 tx=0).stop 2024-03-14T10:01:49.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.565+0000 7fb6264e8b80 1 -- 172.21.15.78:0/2671954626 shutdown_connections 2024-03-14T10:01:49.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.565+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/2671954626 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5572e6a510f0 0x5572e6a53510 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.565+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/2671954626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5572e69e4680 0x5572e69f5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.565+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/2671954626 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5572e69eeec0 0x5572e69f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.565+0000 7fb5f9600700 0 client.17118 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:01:49.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.565+0000 7fb6264e8b80 1 --2- 172.21.15.78:0/2671954626 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5572e69e4fa0 0x5572e69f55c0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.565+0000 7fb6264e8b80 1 -- 172.21.15.78:0/2671954626 >> 172.21.15.78:0/2671954626 conn(0x5572e695a220 msgr2=0x5572e69e3560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.565+0000 7fb6264e8b80 1 -- 172.21.15.78:0/2671954626 shutdown_connections 2024-03-14T10:01:49.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.565+0000 7fb6264e8b80 1 -- 172.21.15.78:0/2671954626 wait complete. 2024-03-14T10:01:49.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.566+0000 7fb6264e8b80 20 client.17118 trim_cache size 0 max 16384 2024-03-14T10:01:49.571 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:01:49.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.566+0000 7fb612a00700 1 -- 172.21.15.78:0/1538861548 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb5f00a0c00 msgr2=0x7fb5f00a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.566+0000 7fb612a00700 1 --2- 172.21.15.78:0/1538861548 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb5f00a0c00 0x7fb5f00a30c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb604009790 tx=0x7fb604009130 comp rx=0 tx=0).stop 2024-03-14T10:01:49.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.566+0000 7fb612a00700 1 -- 172.21.15.78:0/1538861548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb60c100f30 msgr2=0x7fb60c19f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.566+0000 7fb612a00700 1 --2- 172.21.15.78:0/1538861548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb60c100f30 0x7fb60c19f510 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc005770 tx=0x7fb5fc0057a0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.566+0000 7fb612a00700 1 -- 172.21.15.78:0/1538861548 shutdown_connections 2024-03-14T10:01:49.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.566+0000 7fb612a00700 1 --2- 172.21.15.78:0/1538861548 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb60c199640 0x7fb60c199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.566+0000 7fb612a00700 1 --2- 172.21.15.78:0/1538861548 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb60c100f30 0x7fb60c19f510 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.566+0000 7fb612a00700 1 --2- 172.21.15.78:0/1538861548 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb5f00a0c00 0x7fb5f00a30c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.566+0000 7fb612a00700 1 --2- 172.21.15.78:0/1538861548 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb60c106050 0x7fb60c19fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.566+0000 7fb612a00700 1 -- 172.21.15.78:0/1538861548 >> 172.21.15.78:0/1538861548 conn(0x7fb60c0fc7b0 msgr2=0x7fb60c0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.566+0000 7fb612a00700 1 -- 172.21.15.78:0/1538861548 shutdown_connections 2024-03-14T10:01:49.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.566+0000 7fb612a00700 1 -- 172.21.15.78:0/1538861548 wait complete. 2024-03-14T10:01:49.587 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T10:01:49.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.784+0000 7f7cb7800700 1 -- 172.21.15.78:0/1918602152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cb0106040 msgr2=0x7f7cb010dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.784+0000 7f7cb7800700 1 --2- 172.21.15.78:0/1918602152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cb0106040 0x7f7cb010dbc0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800b0a0 tx=0x7f7ca800b3b0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.785+0000 7f7cb7800700 1 -- 172.21.15.78:0/1918602152 shutdown_connections 2024-03-14T10:01:49.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.785+0000 7f7cb7800700 1 --2- 172.21.15.78:0/1918602152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7cb0100f20 0x7f7cb0101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.785+0000 7f7cb7800700 1 --2- 172.21.15.78:0/1918602152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cb0106040 0x7f7cb010dbc0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.785+0000 7f7cb7800700 1 --2- 172.21.15.78:0/1918602152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cb01018d0 0x7f7cb0105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.785+0000 7f7cb7800700 1 -- 172.21.15.78:0/1918602152 >> 172.21.15.78:0/1918602152 conn(0x7f7cb00fc7a0 msgr2=0x7f7cb00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.785+0000 7f7cb7800700 1 -- 172.21.15.78:0/1918602152 shutdown_connections 2024-03-14T10:01:49.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.785+0000 7f7cb7800700 1 -- 172.21.15.78:0/1918602152 wait complete. 2024-03-14T10:01:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.786+0000 7f7cb7800700 1 Processor -- start 2024-03-14T10:01:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.786+0000 7f7cb7800700 1 -- start start 2024-03-14T10:01:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.786+0000 7f7cb7800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cb0100f20 0x7f7cb019f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.786+0000 7f7cb7800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7cb0106040 0x7f7cb019fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.786+0000 7f7cb7800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cb0199760 0x7f7cb0199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.786+0000 7f7cb7800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb019a1e0 con 0x7f7cb0199760 2024-03-14T10:01:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.786+0000 7f7cb7800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb019a350 con 0x7f7cb0106040 2024-03-14T10:01:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.786+0000 7f7cb7800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb019a510 con 0x7f7cb0100f20 2024-03-14T10:01:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.787+0000 7f7cb5800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cb0199760 0x7f7cb0199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.787+0000 7f7cb5800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cb0199760 0x7f7cb0199c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42316/0 (socket says 172.21.15.78:42316) 2024-03-14T10:01:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.787+0000 7f7cb5800700 1 -- 172.21.15.78:0/2186157440 learned_addr learned my addr 172.21.15.78:0/2186157440 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.787+0000 7f7cb4e00700 1 --2- 172.21.15.78:0/2186157440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cb0100f20 0x7f7cb019f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.787+0000 7f7cafe00700 1 --2- 172.21.15.78:0/2186157440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7cb0106040 0x7f7cb019fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.787+0000 7f7cb5800700 1 -- 172.21.15.78:0/2186157440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cb0100f20 msgr2=0x7f7cb019f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.787+0000 7f7cb5800700 1 --2- 172.21.15.78:0/2186157440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cb0100f20 0x7f7cb019f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.787+0000 7f7cb5800700 1 -- 172.21.15.78:0/2186157440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7cb0106040 msgr2=0x7f7cb019fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.787+0000 7f7cb5800700 1 --2- 172.21.15.78:0/2186157440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7cb0106040 0x7f7cb019fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.787+0000 7f7cb5800700 1 -- 172.21.15.78:0/2186157440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca4009520 con 0x7f7cb0199760 2024-03-14T10:01:49.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.787+0000 7f7cb5800700 1 --2- 172.21.15.78:0/2186157440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cb0199760 0x7f7cb0199c10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800b070 tx=0x7f7ca800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.787+0000 7f7cad600700 1 -- 172.21.15.78:0/2186157440 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca800e040 con 0x7f7cb0199760 2024-03-14T10:01:49.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.787+0000 7f7cad600700 1 -- 172.21.15.78:0/2186157440 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ca801d480 con 0x7f7cb0199760 2024-03-14T10:01:49.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.787+0000 7f7cb7800700 1 -- 172.21.15.78:0/2186157440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca8009d00 con 0x7f7cb0199760 2024-03-14T10:01:49.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.787+0000 7f7cad600700 1 -- 172.21.15.78:0/2186157440 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca80052a0 con 0x7f7cb0199760 2024-03-14T10:01:49.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.788+0000 7f7cb7800700 1 -- 172.21.15.78:0/2186157440 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb01a66e0 con 0x7f7cb0199760 2024-03-14T10:01:49.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.790+0000 7f7cad600700 1 -- 172.21.15.78:0/2186157440 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f7ca801d5f0 con 0x7f7cb0199760 2024-03-14T10:01:49.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.792+0000 7f7ccb2cfb80 1 Processor -- start 2024-03-14T10:01:49.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.792+0000 7f7cad600700 1 --2- 172.21.15.78:0/2186157440 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7c980a0c00 0x7f7c980a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.792+0000 7f7ccb2cfb80 1 -- start start 2024-03-14T10:01:49.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.792+0000 7f7cad600700 1 -- 172.21.15.78:0/2186157440 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f7ca80cd860 con 0x7f7cb0199760 2024-03-14T10:01:49.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.792+0000 7f7ccb2cfb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a78a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.792+0000 7f7ccb2cfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 0x55e644a82980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.792+0000 7f7ccb2cfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a852b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.792+0000 7f7ccb2cfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6449d1990 con 0x55e644a78fa0 2024-03-14T10:01:49.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.792+0000 7f7ccb2cfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6449cf350 con 0x55e644a82ec0 2024-03-14T10:01:49.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.792+0000 7f7ccb2cfb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6449cac40 con 0x55e644a78680 2024-03-14T10:01:49.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.792+0000 7f7cafe00700 1 --2- 172.21.15.78:0/2186157440 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7c980a0c00 0x7f7c980a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.792+0000 7f7cb5800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a852b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.792+0000 7f7cb4e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a78a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.793+0000 7f7cb5800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a852b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:49742/0 (socket says 172.21.15.78:49742) 2024-03-14T10:01:49.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.793+0000 7f7cb5800700 1 -- 172.21.15.78:0/3528482159 learned_addr learned my addr 172.21.15.78:0/3528482159 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:49.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.793+0000 7f7cb4e00700 1 --2- 172.21.15.78:0/3528482159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a78a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:52206/0 (socket says 172.21.15.78:52206) 2024-03-14T10:01:49.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.793+0000 7f7cafe00700 1 --2- 172.21.15.78:0/3528482159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 0x55e644a82980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.793+0000 7f7cb4e00700 1 -- 172.21.15.78:0/3528482159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 msgr2=0x55e644a852b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.793+0000 7f7cb4e00700 1 --2- 172.21.15.78:0/3528482159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a852b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.793+0000 7f7cb4e00700 1 -- 172.21.15.78:0/3528482159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 msgr2=0x55e644a82980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.793+0000 7f7cb4e00700 1 --2- 172.21.15.78:0/3528482159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 0x55e644a82980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.793+0000 7f7cb4e00700 1 -- 172.21.15.78:0/3528482159 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e64484c3b0 con 0x55e644a78680 2024-03-14T10:01:49.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.793+0000 7f7cb5800700 1 --2- 172.21.15.78:0/3528482159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a852b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:49.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.793+0000 7f7cafe00700 1 --2- 172.21.15.78:0/3528482159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 0x55e644a82980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:01:49.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.793+0000 7f7cafe00700 1 --2- 172.21.15.78:0/2186157440 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7c980a0c00 0x7f7c980a30c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7cb019ae70 tx=0x7f7c9c013840 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.793+0000 7f7cb4e00700 1 --2- 172.21.15.78:0/3528482159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a78a60 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009e40 tx=0x7f7ca4011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb670557c3871564 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.794+0000 7f7ca2000700 1 -- 172.21.15.78:0/3528482159 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4019630 con 0x55e644a78680 2024-03-14T10:01:49.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.794+0000 7f7ca2000700 1 -- 172.21.15.78:0/3528482159 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ca4019c50 con 0x55e644a78680 2024-03-14T10:01:49.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.794+0000 7f7ca2000700 1 -- 172.21.15.78:0/3528482159 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca400ec70 con 0x55e644a78680 2024-03-14T10:01:49.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.794+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/3528482159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 msgr2=0x55e644a78a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.794+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/3528482159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a78a60 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009e40 tx=0x7f7ca4011720 comp rx=0 tx=0).stop 2024-03-14T10:01:49.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.794+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/3528482159 shutdown_connections 2024-03-14T10:01:49.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.794+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/3528482159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a852b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.794+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/3528482159 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a78a60 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.794+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/3528482159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 0x55e644a82980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.794+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/3528482159 >> 172.21.15.78:0/3528482159 conn(0x55e6449ee220 msgr2=0x55e644a73aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.794+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/3528482159 shutdown_connections 2024-03-14T10:01:49.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.795+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/3528482159 wait complete. 2024-03-14T10:01:49.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.795+0000 7f7ccb2cfb80 1 -- start start 2024-03-14T10:01:49.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.795+0000 7f7ccb2cfb80 1 Processor -- start 2024-03-14T10:01:49.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7ccb2cfb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a82840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7ccb2cfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 0x55e644a7fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7ccb2cfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a80160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7ccb2cfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e644966d80 con 0x55e644a78fa0 2024-03-14T10:01:49.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7ccb2cfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e644966a20 con 0x55e644a82ec0 2024-03-14T10:01:49.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7ccb2cfb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e64484e6a0 con 0x55e644a78680 2024-03-14T10:01:49.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7ccb2cfb80 10 client.0 _ll_register_callbacks cb 0x55e6447d23a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:49.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7cb4e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a82840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7cb5800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a80160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7cb4e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a82840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:52212/0 (socket says 172.21.15.78:52212) 2024-03-14T10:01:49.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7cb4e00700 1 -- 172.21.15.78:0/2264668062 learned_addr learned my addr 172.21.15.78:0/2264668062 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:49.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7cafe00700 1 --2- 172.21.15.78:0/2264668062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 0x55e644a7fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7cb4e00700 1 -- 172.21.15.78:0/2264668062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 msgr2=0x55e644a80160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7cb4e00700 1 --2- 172.21.15.78:0/2264668062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a80160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7cb4e00700 1 -- 172.21.15.78:0/2264668062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 msgr2=0x55e644a7fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7cb4e00700 1 --2- 172.21.15.78:0/2264668062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 0x55e644a7fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7cb4e00700 1 -- 172.21.15.78:0/2264668062 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb01a66e0 con 0x55e644a78680 2024-03-14T10:01:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.796+0000 7f7cb5800700 1 --2- 172.21.15.78:0/2264668062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a80160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.797+0000 7f7cb4e00700 1 --2- 172.21.15.78:0/2264668062 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a82840 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4011a70 tx=0x7f7ca4005760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2c686fd247a36e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.797+0000 7f7c96a00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:01:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.797+0000 7f7c96a00700 1 -- 172.21.15.78:0/2264668062 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca400eaf0 con 0x55e644a78680 2024-03-14T10:01:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.797+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/2264668062 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e644a6b720 con 0x55e644a78680 2024-03-14T10:01:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.797+0000 7f7c96a00700 1 -- 172.21.15.78:0/2264668062 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f7ca400ec50 con 0x55e644a78680 2024-03-14T10:01:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.797+0000 7f7c96a00700 1 -- 172.21.15.78:0/2264668062 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ca400ee10 con 0x55e644a78680 2024-03-14T10:01:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.797+0000 7f7ccb2cfb80 10 client.27184 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.797+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/2264668062 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f7ca4009a70 con 0x55e644a78680 2024-03-14T10:01:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.797+0000 7f7c96a00700 1 -- 172.21.15.78:0/2264668062 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e644a6b720 con 0x55e644a78680 2024-03-14T10:01:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.797+0000 7f7c96a00700 1 -- 172.21.15.78:0/2264668062 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7ca4029020 con 0x55e644a78680 2024-03-14T10:01:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.798+0000 7f7ccb2cfb80 10 client.27184 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:49.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.798+0000 7f7ccb2cfb80 10 client.27184 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:01:49.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.798+0000 7f7ccb2cfb80 20 client.27184 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:49.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.798+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/2264668062 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e644a92a40 0x55e644a94e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.798+0000 7f7ccb2cfb80 4 client.27184 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:01:49.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.798+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/2264668062 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e6449c3700 con 0x55e644a92a40 2024-03-14T10:01:49.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.798+0000 7f7cafe00700 1 --2- 172.21.15.78:0/2264668062 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e644a92a40 0x55e644a94e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.798+0000 7f7cafe00700 1 --2- 172.21.15.78:0/2264668062 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e644a92a40 0x55e644a94e60 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20defd341c07720c server_cookie=97b2813531ba211d in_seq=0 out_seq=0 2024-03-14T10:01:49.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.798+0000 7f7c96a00700 10 client.27184 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:49.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.799+0000 7f7c96a00700 1 -- 172.21.15.78:0/2264668062 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55e6449c3700 con 0x55e644a92a40 2024-03-14T10:01:49.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.799+0000 7f7c96a00700 10 client.27184 handle_command_reply: tid=0 2024-03-14T10:01:49.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.799+0000 7f7ccb2cfb80 1 client.27184 shutdown 2024-03-14T10:01:49.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.799+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/2264668062 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 msgr2=0x55e644a82840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.799+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/2264668062 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a82840 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4011a70 tx=0x7f7ca4005760 comp rx=0 tx=0).stop 2024-03-14T10:01:49.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.799+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/2264668062 shutdown_connections 2024-03-14T10:01:49.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.799+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/2264668062 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e644a92a40 0x55e644a94e60 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.799+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/2264668062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a80160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.799+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/2264668062 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a82840 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.799+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/2264668062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 0x55e644a7fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.799+0000 7f7c96a00700 0 client.27184 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:49.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.799+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/2264668062 >> 172.21.15.78:0/2264668062 conn(0x55e6449ee220 msgr2=0x55e644a73690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.799+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/2264668062 shutdown_connections 2024-03-14T10:01:49.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.799+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/2264668062 wait complete. 2024-03-14T10:01:49.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.800+0000 7f7ccb2cfb80 20 client.27184 trim_cache size 0 max 16384 2024-03-14T10:01:49.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.813+0000 7f7ccb2cfb80 1 Processor -- start 2024-03-14T10:01:49.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.813+0000 7f7ccb2cfb80 1 -- start start 2024-03-14T10:01:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.814+0000 7f7ccb2cfb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a94e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.814+0000 7f7ccb2cfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 0x55e644a99780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.814+0000 7f7ccb2cfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a7e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.814+0000 7f7ccb2cfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e644a7a950 con 0x55e644a78fa0 2024-03-14T10:01:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.814+0000 7f7ccb2cfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e644a7e080 con 0x55e644a82ec0 2024-03-14T10:01:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.814+0000 7f7ccb2cfb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e644a7b990 con 0x55e644a78680 2024-03-14T10:01:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.814+0000 7f7cafe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 0x55e644a99780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.814+0000 7f7cafe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 0x55e644a99780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42346/0 (socket says 172.21.15.78:42346) 2024-03-14T10:01:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.814+0000 7f7cafe00700 1 -- 172.21.15.78:0/2830880929 learned_addr learned my addr 172.21.15.78:0/2830880929 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.814+0000 7f7cb4e00700 1 --2- 172.21.15.78:0/2830880929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a94e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.814+0000 7f7cb5800700 1 --2- 172.21.15.78:0/2830880929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a7e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.815+0000 7f7cb4e00700 1 -- 172.21.15.78:0/2830880929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 msgr2=0x55e644a7e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.815+0000 7f7cb4e00700 1 --2- 172.21.15.78:0/2830880929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a7e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.815+0000 7f7cb4e00700 1 -- 172.21.15.78:0/2830880929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 msgr2=0x55e644a99780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.815+0000 7f7cb4e00700 1 --2- 172.21.15.78:0/2830880929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 0x55e644a99780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.815+0000 7f7cb4e00700 1 -- 172.21.15.78:0/2830880929 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6447f8b60 con 0x55e644a78680 2024-03-14T10:01:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.815+0000 7f7cafe00700 1 --2- 172.21.15.78:0/2830880929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 0x55e644a99780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.815+0000 7f7cb4e00700 1 --2- 172.21.15.78:0/2830880929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a94e30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40092a0 tx=0x7f7ca4011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bc82d64f71932f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.815+0000 7f7ca2a00700 1 -- 172.21.15.78:0/2830880929 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4003be0 con 0x55e644a78680 2024-03-14T10:01:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.815+0000 7f7ca2a00700 1 -- 172.21.15.78:0/2830880929 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ca4021cf0 con 0x55e644a78680 2024-03-14T10:01:49.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.815+0000 7f7ca2a00700 1 -- 172.21.15.78:0/2830880929 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4019ae0 con 0x55e644a78680 2024-03-14T10:01:49.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.816+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/2830880929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 msgr2=0x55e644a94e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.816+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/2830880929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a94e30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40092a0 tx=0x7f7ca4011720 comp rx=0 tx=0).stop 2024-03-14T10:01:49.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.816+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/2830880929 shutdown_connections 2024-03-14T10:01:49.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.816+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/2830880929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a7e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.816+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/2830880929 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78680 0x55e644a94e30 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.816+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/2830880929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78fa0 0x55e644a99780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.816+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/2830880929 >> 172.21.15.78:0/2830880929 conn(0x55e6449ee220 msgr2=0x55e644a72180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.816+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/2830880929 shutdown_connections 2024-03-14T10:01:49.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.816+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/2830880929 wait complete. 2024-03-14T10:01:49.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.816+0000 7f7ccb2cfb80 1 -- start start 2024-03-14T10:01:49.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.816+0000 7f7ccb2cfb80 1 Processor -- start 2024-03-14T10:01:49.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.817+0000 7f7ccb2cfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78680 0x55e644a87b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.817+0000 7f7ccb2cfb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78fa0 0x55e644a880b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.817+0000 7f7ccb2cfb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a885f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.817+0000 7f7ccb2cfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e644ae3660 con 0x55e644a78680 2024-03-14T10:01:49.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.817+0000 7f7ccb2cfb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e644ae37d0 con 0x55e644a82ec0 2024-03-14T10:01:49.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.817+0000 7f7ccb2cfb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e644ae3990 con 0x55e644a78fa0 2024-03-14T10:01:49.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.817+0000 7f7ccb2cfb80 10 client.0 _ll_register_callbacks cb 0x7f7ca8009b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:49.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.817+0000 7f7cb4e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78680 0x55e644a87b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.817+0000 7f7cb4e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78680 0x55e644a87b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42354/0 (socket says 172.21.15.78:42354) 2024-03-14T10:01:49.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.817+0000 7f7cb4e00700 1 -- 172.21.15.78:0/4025126250 learned_addr learned my addr 172.21.15.78:0/4025126250 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:49.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.817+0000 7f7cafe00700 1 --2- 172.21.15.78:0/4025126250 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78fa0 0x55e644a880b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.817+0000 7f7cb5800700 1 --2- 172.21.15.78:0/4025126250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a885f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.818+0000 7f7cb4e00700 1 -- 172.21.15.78:0/4025126250 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78fa0 msgr2=0x55e644a880b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.818+0000 7f7cb4e00700 1 --2- 172.21.15.78:0/4025126250 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78fa0 0x55e644a880b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.818+0000 7f7cb4e00700 1 -- 172.21.15.78:0/4025126250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 msgr2=0x55e644a885f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:49.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.818+0000 7f7cb4e00700 1 --2- 172.21.15.78:0/4025126250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a885f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.818+0000 7f7cb4e00700 1 -- 172.21.15.78:0/4025126250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c9c004030 con 0x55e644a78680 2024-03-14T10:01:49.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.818+0000 7f7cb4e00700 1 --2- 172.21.15.78:0/4025126250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78680 0x55e644a87b70 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4021cc0 tx=0x7f7ca4019ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38fe2a2489413f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:49.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.818+0000 7f7c96a00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:49.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.818+0000 7f7c96a00700 1 -- 172.21.15.78:0/4025126250 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4021770 con 0x55e644a78680 2024-03-14T10:01:49.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.818+0000 7f7c96a00700 1 -- 172.21.15.78:0/4025126250 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f7ca4000e40 con 0x55e644a78680 2024-03-14T10:01:49.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.818+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/4025126250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ca4019c50 con 0x55e644a78680 2024-03-14T10:01:49.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.818+0000 7f7c96a00700 1 -- 172.21.15.78:0/4025126250 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ca40218d0 con 0x55e644a78680 2024-03-14T10:01:49.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.818+0000 7f7ccb2cfb80 10 client.17136 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:49.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.818+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/4025126250 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f7ca4009a70 con 0x55e644a78680 2024-03-14T10:01:49.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.819+0000 7f7c96a00700 1 -- 172.21.15.78:0/4025126250 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ca4019c50 con 0x55e644a78680 2024-03-14T10:01:49.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.819+0000 7f7c96a00700 1 -- 172.21.15.78:0/4025126250 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7ca4011790 con 0x55e644a78680 2024-03-14T10:01:49.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.819+0000 7f7ccb2cfb80 10 client.17136 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:49.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.819+0000 7f7ccb2cfb80 10 client.17136 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:01:49.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.819+0000 7f7ccb2cfb80 20 client.17136 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:49.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.819+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/4025126250 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e644ae4ce0 0x55e644ae70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:49.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.819+0000 7f7ccb2cfb80 4 client.17136 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:01:49.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.819+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/4025126250 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e644819cf0 con 0x55e644ae4ce0 2024-03-14T10:01:49.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.819+0000 7f7cafe00700 1 --2- 172.21.15.78:0/4025126250 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e644ae4ce0 0x55e644ae70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:49.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7cafe00700 1 --2- 172.21.15.78:0/4025126250 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e644ae4ce0 0x55e644ae70d0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ee6f9e6a0a6d9b8 server_cookie=f7806442be6f1461 in_seq=0 out_seq=0 2024-03-14T10:01:49.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7c96a00700 10 client.17136 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:49.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7c96a00700 1 -- 172.21.15.78:0/4025126250 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e644819cf0 con 0x55e644ae4ce0 2024-03-14T10:01:49.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7c96a00700 10 client.17136 handle_command_reply: tid=0 2024-03-14T10:01:49.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7ccb2cfb80 1 client.17136 shutdown 2024-03-14T10:01:49.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/4025126250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78680 msgr2=0x55e644a87b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/4025126250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78680 0x55e644a87b70 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4021cc0 tx=0x7f7ca4019ae0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/4025126250 shutdown_connections 2024-03-14T10:01:49.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/4025126250 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e644ae4ce0 0x55e644ae70d0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/4025126250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e644a82ec0 0x55e644a885f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7c96a00700 0 client.17136 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:49.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/4025126250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e644a78680 0x55e644a87b70 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7ccb2cfb80 1 --2- 172.21.15.78:0/4025126250 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e644a78fa0 0x55e644a880b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/4025126250 >> 172.21.15.78:0/4025126250 conn(0x55e6449ee220 msgr2=0x55e644a7e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/4025126250 shutdown_connections 2024-03-14T10:01:49.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7ccb2cfb80 1 -- 172.21.15.78:0/4025126250 wait complete. 2024-03-14T10:01:49.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.820+0000 7f7ccb2cfb80 20 client.17136 trim_cache size 0 max 16384 2024-03-14T10:01:49.828 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:01:49.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.821+0000 7f7cb7800700 1 -- 172.21.15.78:0/2186157440 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7c980a0c00 msgr2=0x7f7c980a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.821+0000 7f7cb7800700 1 --2- 172.21.15.78:0/2186157440 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7c980a0c00 0x7f7c980a30c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7cb019ae70 tx=0x7f7c9c013840 comp rx=0 tx=0).stop 2024-03-14T10:01:49.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.821+0000 7f7cb7800700 1 -- 172.21.15.78:0/2186157440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cb0199760 msgr2=0x7f7cb0199c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:49.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.821+0000 7f7cb7800700 1 --2- 172.21.15.78:0/2186157440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cb0199760 0x7f7cb0199c10 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800b070 tx=0x7f7ca800bda0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.821+0000 7f7cb7800700 1 -- 172.21.15.78:0/2186157440 shutdown_connections 2024-03-14T10:01:49.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.821+0000 7f7cb7800700 1 --2- 172.21.15.78:0/2186157440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cb0199760 0x7f7cb0199c10 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.821+0000 7f7cb7800700 1 --2- 172.21.15.78:0/2186157440 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7cb0100f20 0x7f7cb019f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.821+0000 7f7cb7800700 1 --2- 172.21.15.78:0/2186157440 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7c980a0c00 0x7f7c980a30c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.821+0000 7f7cb7800700 1 --2- 172.21.15.78:0/2186157440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7cb0106040 0x7f7cb019fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:49.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.821+0000 7f7cb7800700 1 -- 172.21.15.78:0/2186157440 >> 172.21.15.78:0/2186157440 conn(0x7f7cb00fc7a0 msgr2=0x7f7cb00fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:49.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.821+0000 7f7cb7800700 1 -- 172.21.15.78:0/2186157440 shutdown_connections 2024-03-14T10:01:49.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:49.821+0000 7f7cb7800700 1 -- 172.21.15.78:0/2186157440 wait complete. 2024-03-14T10:01:50.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:49 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/355121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:01:50.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:49 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/355121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:01:50.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:49 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/355121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:01:50.841 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T10:01:50.842 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T10:01:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.045+0000 7fe5d0400700 1 -- 172.21.15.78:0/3666754409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5c8100f30 msgr2=0x7fe5c8101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.045+0000 7fe5d0400700 1 --2- 172.21.15.78:0/3666754409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5c8100f30 0x7fe5c8101310 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009990 tx=0x7fe5bc009ca0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.045+0000 7fe5d0400700 1 -- 172.21.15.78:0/3666754409 shutdown_connections 2024-03-14T10:01:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.045+0000 7fe5d0400700 1 --2- 172.21.15.78:0/3666754409 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5c8100f30 0x7fe5c8101310 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.045+0000 7fe5d0400700 1 --2- 172.21.15.78:0/3666754409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5c8106050 0x7fe5c810dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.045+0000 7fe5d0400700 1 --2- 172.21.15.78:0/3666754409 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c81018e0 0x7fe5c8105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.045+0000 7fe5d0400700 1 -- 172.21.15.78:0/3666754409 >> 172.21.15.78:0/3666754409 conn(0x7fe5c80fc7b0 msgr2=0x7fe5c80febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.045+0000 7fe5d0400700 1 -- 172.21.15.78:0/3666754409 shutdown_connections 2024-03-14T10:01:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.045+0000 7fe5d0400700 1 -- 172.21.15.78:0/3666754409 wait complete. 2024-03-14T10:01:51.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.046+0000 7fe5d0400700 1 Processor -- start 2024-03-14T10:01:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.047+0000 7fe5d0400700 1 -- start start 2024-03-14T10:01:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.047+0000 7fe5d0400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c8100f30 0x7fe5c819f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.047+0000 7fe5d0400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5c8106050 0x7fe5c819fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.047+0000 7fe5d0400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5c81996a0 0x7fe5c8199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.047+0000 7fe5d0400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c819a120 con 0x7fe5c8106050 2024-03-14T10:01:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.047+0000 7fe5d0400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c819a290 con 0x7fe5c81996a0 2024-03-14T10:01:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.047+0000 7fe5d0400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c819a450 con 0x7fe5c8100f30 2024-03-14T10:01:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.047+0000 7fe5cd000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5c8106050 0x7fe5c819fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.047+0000 7fe5cd000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5c8106050 0x7fe5c819fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42372/0 (socket says 172.21.15.78:42372) 2024-03-14T10:01:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.047+0000 7fe5cd000700 1 -- 172.21.15.78:0/2242035493 learned_addr learned my addr 172.21.15.78:0/2242035493 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.047+0000 7fe5cda00700 1 --2- 172.21.15.78:0/2242035493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c8100f30 0x7fe5c819f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.047+0000 7fe5ce400700 1 --2- 172.21.15.78:0/2242035493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5c81996a0 0x7fe5c8199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.048+0000 7fe5cd000700 1 -- 172.21.15.78:0/2242035493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c8100f30 msgr2=0x7fe5c819f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.048+0000 7fe5cd000700 1 --2- 172.21.15.78:0/2242035493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c8100f30 0x7fe5c819f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.048+0000 7fe5cd000700 1 -- 172.21.15.78:0/2242035493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5c81996a0 msgr2=0x7fe5c8199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.048+0000 7fe5cd000700 1 --2- 172.21.15.78:0/2242035493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5c81996a0 0x7fe5c8199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.048+0000 7fe5cd000700 1 -- 172.21.15.78:0/2242035493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c0009c40 con 0x7fe5c8106050 2024-03-14T10:01:51.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.048+0000 7fe5cda00700 1 --2- 172.21.15.78:0/2242035493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c8100f30 0x7fe5c819f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:51.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.048+0000 7fe5cd000700 1 --2- 172.21.15.78:0/2242035493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5c8106050 0x7fe5c819fa60 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400e970 tx=0x7fe5b400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:51.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.048+0000 7fe5c6000700 1 -- 172.21.15.78:0/2242035493 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b4009790 con 0x7fe5c8106050 2024-03-14T10:01:51.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.048+0000 7fe5d0400700 1 -- 172.21.15.78:0/2242035493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5bc0095f0 con 0x7fe5c8106050 2024-03-14T10:01:51.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.048+0000 7fe5c6000700 1 -- 172.21.15.78:0/2242035493 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5b400c7c0 con 0x7fe5c8106050 2024-03-14T10:01:51.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.048+0000 7fe5c6000700 1 -- 172.21.15.78:0/2242035493 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b4005460 con 0x7fe5c8106050 2024-03-14T10:01:51.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.048+0000 7fe5d0400700 1 -- 172.21.15.78:0/2242035493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c81a6790 con 0x7fe5c8106050 2024-03-14T10:01:51.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.051+0000 7fe5c6000700 1 -- 172.21.15.78:0/2242035493 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fe5b4005600 con 0x7fe5c8106050 2024-03-14T10:01:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.052+0000 7fe5c6000700 1 --2- 172.21.15.78:0/2242035493 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe5b00a0c80 0x7fe5b00a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.052+0000 7fe5e3faeb80 1 Processor -- start 2024-03-14T10:01:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.053+0000 7fe5c6000700 1 -- 172.21.15.78:0/2242035493 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fe5b4014070 con 0x7fe5c8106050 2024-03-14T10:01:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.053+0000 7fe5e3faeb80 1 -- start start 2024-03-14T10:01:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.053+0000 7fe5cda00700 1 --2- 172.21.15.78:0/2242035493 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe5b00a0c80 0x7fe5b00a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.053+0000 7fe5e3faeb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df52700 0x561a9df52ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.053+0000 7fe5e3faeb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df53020 0x561a9df5ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.053+0000 7fe5e3faeb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df5cf40 0x561a9df5f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.053+0000 7fe5e3faeb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a9deaba10 con 0x561a9df52700 2024-03-14T10:01:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.053+0000 7fe5e3faeb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a9dea93d0 con 0x561a9df53020 2024-03-14T10:01:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.053+0000 7fe5e3faeb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a9dea4cc0 con 0x561a9df5cf40 2024-03-14T10:01:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.053+0000 7fe5cda00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df52700 0x561a9df52ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.053+0000 7fe5ce400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df5cf40 0x561a9df5f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.053+0000 7fe5ce400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df5cf40 0x561a9df5f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:52276/0 (socket says 172.21.15.78:52276) 2024-03-14T10:01:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.053+0000 7fe5ce400700 1 -- 172.21.15.78:0/1824277772 learned_addr learned my addr 172.21.15.78:0/1824277772 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.053+0000 7fe5cd000700 1 --2- 172.21.15.78:0/1824277772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df53020 0x561a9df5ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.054+0000 7fe5ce400700 1 -- 172.21.15.78:0/1824277772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df53020 msgr2=0x561a9df5ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.054+0000 7fe5ce400700 1 --2- 172.21.15.78:0/1824277772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df53020 0x561a9df5ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.054+0000 7fe5ce400700 1 -- 172.21.15.78:0/1824277772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df52700 msgr2=0x561a9df52ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.054+0000 7fe5ce400700 1 --2- 172.21.15.78:0/1824277772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df52700 0x561a9df52ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.054+0000 7fe5ce400700 1 -- 172.21.15.78:0/1824277772 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a9dd26430 con 0x561a9df5cf40 2024-03-14T10:01:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.054+0000 7fe5cda00700 1 --2- 172.21.15.78:0/1824277772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df52700 0x561a9df52ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:01:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.054+0000 7fe5cda00700 1 --2- 172.21.15.78:0/2242035493 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe5b00a0c80 0x7fe5b00a3140 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc006010 tx=0x7fe5bc00be70 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.054+0000 7fe5ce400700 1 --2- 172.21.15.78:0/1824277772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df5cf40 0x561a9df5f330 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe5c000e730 tx=0x7fe5c0009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffe55b1fd5f18fdf server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.054+0000 7fe5baa00700 1 -- 172.21.15.78:0/1824277772 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c000e040 con 0x561a9df5cf40 2024-03-14T10:01:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.054+0000 7fe5baa00700 1 -- 172.21.15.78:0/1824277772 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5c000f040 con 0x561a9df5cf40 2024-03-14T10:01:51.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.054+0000 7fe5baa00700 1 -- 172.21.15.78:0/1824277772 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0007500 con 0x561a9df5cf40 2024-03-14T10:01:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.055+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/1824277772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df5cf40 msgr2=0x561a9df5f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.055+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/1824277772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df5cf40 0x561a9df5f330 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe5c000e730 tx=0x7fe5c0009920 comp rx=0 tx=0).stop 2024-03-14T10:01:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.055+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/1824277772 shutdown_connections 2024-03-14T10:01:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.055+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/1824277772 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df5cf40 0x561a9df5f330 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.055+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/1824277772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df52700 0x561a9df52ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.055+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/1824277772 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df53020 0x561a9df5ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.055+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/1824277772 >> 172.21.15.78:0/1824277772 conn(0x561a9dec82a0 msgr2=0x561a9df4db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.055+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/1824277772 shutdown_connections 2024-03-14T10:01:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.055+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/1824277772 wait complete. 2024-03-14T10:01:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.056+0000 7fe5e3faeb80 1 -- start start 2024-03-14T10:01:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.056+0000 7fe5e3faeb80 1 Processor -- start 2024-03-14T10:01:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.056+0000 7fe5e3faeb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df52700 0x561a9df5c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.056+0000 7fe5e3faeb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df53020 0x561a9df59d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.056+0000 7fe5e3faeb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df5cf40 0x561a9df5a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.056+0000 7fe5e3faeb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a9de40e00 con 0x561a9df53020 2024-03-14T10:01:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.056+0000 7fe5e3faeb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a9de40aa0 con 0x561a9df5cf40 2024-03-14T10:01:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.056+0000 7fe5e3faeb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a9dd28720 con 0x561a9df52700 2024-03-14T10:01:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.056+0000 7fe5e3faeb80 10 client.0 _ll_register_callbacks cb 0x561a9dcac420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.057+0000 7fe5cd000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df53020 0x561a9df59d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.057+0000 7fe5cd000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df53020 0x561a9df59d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42390/0 (socket says 172.21.15.78:42390) 2024-03-14T10:01:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.057+0000 7fe5cd000700 1 -- 172.21.15.78:0/325699040 learned_addr learned my addr 172.21.15.78:0/325699040 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.057+0000 7fe5cda00700 1 --2- 172.21.15.78:0/325699040 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df52700 0x561a9df5c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.057+0000 7fe5ce400700 1 --2- 172.21.15.78:0/325699040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df5cf40 0x561a9df5a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.057+0000 7fe5cd000700 1 -- 172.21.15.78:0/325699040 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df52700 msgr2=0x561a9df5c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.057+0000 7fe5cd000700 1 --2- 172.21.15.78:0/325699040 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df52700 0x561a9df5c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.057+0000 7fe5cd000700 1 -- 172.21.15.78:0/325699040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df5cf40 msgr2=0x561a9df5a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.057+0000 7fe5cd000700 1 --2- 172.21.15.78:0/325699040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df5cf40 0x561a9df5a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.057+0000 7fe5cd000700 1 -- 172.21.15.78:0/325699040 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c000e410 con 0x561a9df53020 2024-03-14T10:01:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.057+0000 7fe5cd000700 1 --2- 172.21.15.78:0/325699040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df53020 0x561a9df59d10 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40058b0 tx=0x7fe5b40999e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c1767e438ef6b06 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.057+0000 7fe5af400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.058+0000 7fe5af400700 1 -- 172.21.15.78:0/325699040 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b40b1070 con 0x561a9df53020 2024-03-14T10:01:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.058+0000 7fe5af400700 1 -- 172.21.15.78:0/325699040 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fe5b40c9f20 con 0x561a9df53020 2024-03-14T10:01:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.058+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/325699040 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561a9df457a0 con 0x561a9df53020 2024-03-14T10:01:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.058+0000 7fe5af400700 1 -- 172.21.15.78:0/325699040 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5b40ca0e0 con 0x561a9df53020 2024-03-14T10:01:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.058+0000 7fe5e3faeb80 10 client.17148 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.058+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/325699040 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561a9dcd2be0 con 0x561a9df53020 2024-03-14T10:01:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.058+0000 7fe5af400700 1 -- 172.21.15.78:0/325699040 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561a9df457a0 con 0x561a9df53020 2024-03-14T10:01:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.058+0000 7fe5af400700 1 -- 172.21.15.78:0/325699040 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe5b4098d90 con 0x561a9df53020 2024-03-14T10:01:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.058+0000 7fe5e3faeb80 10 client.17148 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.058+0000 7fe5e3faeb80 10 client.17148 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.058+0000 7fe5e3faeb80 20 client.17148 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.058+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/325699040 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561a9df6cb00 0x561a9df6ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.058+0000 7fe5e3faeb80 4 client.17148 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.058+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/325699040 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561a9de9d780 con 0x561a9df6cb00 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.059+0000 7fe5cda00700 1 --2- 172.21.15.78:0/325699040 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561a9df6cb00 0x561a9df6ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.059+0000 7fe5af400700 10 client.17148 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.059+0000 7fe5cda00700 1 --2- 172.21.15.78:0/325699040 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561a9df6cb00 0x561a9df6ef20 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aefbd317f1d30d9e server_cookie=6a0caa70a2e28b0b in_seq=0 out_seq=0 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.060+0000 7fe5af400700 1 -- 172.21.15.78:0/325699040 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x561a9de9d780 con 0x561a9df6cb00 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.060+0000 7fe5af400700 10 client.17148 handle_command_reply: tid=0 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.060+0000 7fe5e3faeb80 1 client.17148 shutdown 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.060+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/325699040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df53020 msgr2=0x561a9df59d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.060+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/325699040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df53020 0x561a9df59d10 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fe5b40058b0 tx=0x7fe5b40999e0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.060+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/325699040 shutdown_connections 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.060+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/325699040 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561a9df6cb00 0x561a9df6ef20 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.060+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/325699040 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df5cf40 0x561a9df5a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.060+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/325699040 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df52700 0x561a9df5c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.060+0000 7fe5af400700 0 client.17148 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.060+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/325699040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df53020 0x561a9df59d10 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.060+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/325699040 >> 172.21.15.78:0/325699040 conn(0x561a9dec82a0 msgr2=0x561a9df4d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.060+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/325699040 shutdown_connections 2024-03-14T10:01:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.060+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/325699040 wait complete. 2024-03-14T10:01:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.060+0000 7fe5e3faeb80 20 client.17148 trim_cache size 0 max 16384 2024-03-14T10:01:51.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.077+0000 7fe5e3faeb80 1 Processor -- start 2024-03-14T10:01:51.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.077+0000 7fe5e3faeb80 1 -- start start 2024-03-14T10:01:51.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.077+0000 7fe5e3faeb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df52700 0x561a9df63be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.077+0000 7fe5e3faeb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df53020 0x561a9df58ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.077+0000 7fe5e3faeb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df5cf40 0x561a9df688c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.077+0000 7fe5e3faeb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a9df47dd0 con 0x561a9df52700 2024-03-14T10:01:51.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.077+0000 7fe5e3faeb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a9df4a2e0 con 0x561a9df5cf40 2024-03-14T10:01:51.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.077+0000 7fe5e3faeb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a9df4f160 con 0x561a9df53020 2024-03-14T10:01:51.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.078+0000 7fe5cda00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df52700 0x561a9df63be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.078+0000 7fe5cda00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df52700 0x561a9df63be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42392/0 (socket says 172.21.15.78:42392) 2024-03-14T10:01:51.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.078+0000 7fe5cda00700 1 -- 172.21.15.78:0/600836740 learned_addr learned my addr 172.21.15.78:0/600836740 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:51.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.078+0000 7fe5cd000700 1 --2- 172.21.15.78:0/600836740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df53020 0x561a9df58ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.078+0000 7fe5ce400700 1 --2- 172.21.15.78:0/600836740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df5cf40 0x561a9df688c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.078+0000 7fe5cda00700 1 -- 172.21.15.78:0/600836740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df53020 msgr2=0x561a9df58ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:51.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.078+0000 7fe5cda00700 1 --2- 172.21.15.78:0/600836740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df53020 0x561a9df58ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.078+0000 7fe5cda00700 1 -- 172.21.15.78:0/600836740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df5cf40 msgr2=0x561a9df688c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:51.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.078+0000 7fe5cda00700 1 --2- 172.21.15.78:0/600836740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df5cf40 0x561a9df688c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.078+0000 7fe5cda00700 1 -- 172.21.15.78:0/600836740 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a9de6af20 con 0x561a9df52700 2024-03-14T10:01:51.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.078+0000 7fe5cd000700 1 --2- 172.21.15.78:0/600836740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df53020 0x561a9df58ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:01:51.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.078+0000 7fe5cda00700 1 --2- 172.21.15.78:0/600836740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df52700 0x561a9df63be0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00b650 tx=0x7fe5bc027000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7b3af008c93180 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:51.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.078+0000 7fe5bb400700 1 -- 172.21.15.78:0/600836740 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc01d070 con 0x561a9df52700 2024-03-14T10:01:51.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.079+0000 7fe5bb400700 1 -- 172.21.15.78:0/600836740 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5bc005640 con 0x561a9df52700 2024-03-14T10:01:51.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.079+0000 7fe5bb400700 1 -- 172.21.15.78:0/600836740 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc004d00 con 0x561a9df52700 2024-03-14T10:01:51.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.079+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/600836740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df52700 msgr2=0x561a9df63be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.079+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/600836740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df52700 0x561a9df63be0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00b650 tx=0x7fe5bc027000 comp rx=0 tx=0).stop 2024-03-14T10:01:51.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.079+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/600836740 shutdown_connections 2024-03-14T10:01:51.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.079+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/600836740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df5cf40 0x561a9df688c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.079+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/600836740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df52700 0x561a9df63be0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.079+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/600836740 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df53020 0x561a9df58ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.079+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/600836740 >> 172.21.15.78:0/600836740 conn(0x561a9dec82a0 msgr2=0x561a9df681b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:51.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.079+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/600836740 shutdown_connections 2024-03-14T10:01:51.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.079+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/600836740 wait complete. 2024-03-14T10:01:51.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.080+0000 7fe5e3faeb80 1 -- start start 2024-03-14T10:01:51.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.080+0000 7fe5e3faeb80 1 Processor -- start 2024-03-14T10:01:51.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.080+0000 7fe5e3faeb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df52700 0x561a9dfb45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.080+0000 7fe5e3faeb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df53020 0x561a9dfb4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.080+0000 7fe5e3faeb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df5cf40 0x561a9dfb5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.080+0000 7fe5e3faeb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a9df63f80 con 0x561a9df53020 2024-03-14T10:01:51.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.080+0000 7fe5e3faeb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a9dfbf640 con 0x561a9df5cf40 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.080+0000 7fe5e3faeb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a9dfbf780 con 0x561a9df52700 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.080+0000 7fe5e3faeb80 10 client.0 _ll_register_callbacks cb 0x7fe5b4099fa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.081+0000 7fe5cda00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df52700 0x561a9dfb45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.081+0000 7fe5cda00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df52700 0x561a9dfb45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:52300/0 (socket says 172.21.15.78:52300) 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.081+0000 7fe5cda00700 1 -- 172.21.15.78:0/2533085191 learned_addr learned my addr 172.21.15.78:0/2533085191 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.081+0000 7fe5cd000700 1 --2- 172.21.15.78:0/2533085191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df53020 0x561a9dfb4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.081+0000 7fe5ce400700 1 --2- 172.21.15.78:0/2533085191 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df5cf40 0x561a9dfb5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.081+0000 7fe5cd000700 1 -- 172.21.15.78:0/2533085191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df52700 msgr2=0x561a9dfb45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.081+0000 7fe5cd000700 1 --2- 172.21.15.78:0/2533085191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df52700 0x561a9dfb45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.081+0000 7fe5cd000700 1 -- 172.21.15.78:0/2533085191 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df5cf40 msgr2=0x561a9dfb5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.081+0000 7fe5cd000700 1 --2- 172.21.15.78:0/2533085191 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df5cf40 0x561a9dfb5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.081+0000 7fe5cd000700 1 -- 172.21.15.78:0/2533085191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5bc02e040 con 0x561a9df53020 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.081+0000 7fe5cd000700 1 --2- 172.21.15.78:0/2533085191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df53020 0x561a9dfb4b00 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4099830 tx=0x7fe5b40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb71d54cef10dcb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.081+0000 7fe5af400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.081+0000 7fe5af400700 1 -- 172.21.15.78:0/2533085191 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b40b1040 con 0x561a9df53020 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.081+0000 7fe5af400700 1 -- 172.21.15.78:0/2533085191 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fe5b40a1ad0 con 0x561a9df53020 2024-03-14T10:01:51.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.081+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/2533085191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5c000f040 con 0x561a9df53020 2024-03-14T10:01:51.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.082+0000 7fe5af400700 1 -- 172.21.15.78:0/2533085191 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5b40a1890 con 0x561a9df53020 2024-03-14T10:01:51.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.082+0000 7fe5e3faeb80 10 client.17154 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:51.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.082+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/2533085191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561a9dd2ad50 con 0x561a9df53020 2024-03-14T10:01:51.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.082+0000 7fe5af400700 1 -- 172.21.15.78:0/2533085191 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5c000f040 con 0x561a9df53020 2024-03-14T10:01:51.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.082+0000 7fe5af400700 1 -- 172.21.15.78:0/2533085191 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe5b4078bc0 con 0x561a9df53020 2024-03-14T10:01:51.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.082+0000 7fe5e3faeb80 10 client.17154 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:51.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.082+0000 7fe5e3faeb80 10 client.17154 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:01:51.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.082+0000 7fe5e3faeb80 20 client.17154 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:51.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.082+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/2533085191 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561a9dfc0ae0 0x561a9dfc2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.082+0000 7fe5e3faeb80 4 client.17154 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T10:01:51.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.082+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/2533085191 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x561a9dcf3d70 con 0x561a9dfc0ae0 2024-03-14T10:01:51.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.082+0000 7fe5cda00700 1 --2- 172.21.15.78:0/2533085191 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561a9dfc0ae0 0x561a9dfc2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.083+0000 7fe5cda00700 1 --2- 172.21.15.78:0/2533085191 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561a9dfc0ae0 0x561a9dfc2ed0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f38b578e341906a5 server_cookie=ef986c9ac1a67db5 in_seq=0 out_seq=0 2024-03-14T10:01:51.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.083+0000 7fe5af400700 10 client.17154 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:51.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.084+0000 7fe5af400700 1 -- 172.21.15.78:0/2533085191 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561a9dcf3d70 con 0x561a9dfc0ae0 2024-03-14T10:01:51.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.084+0000 7fe5af400700 10 client.17154 handle_command_reply: tid=0 2024-03-14T10:01:51.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.084+0000 7fe5e3faeb80 1 client.17154 shutdown 2024-03-14T10:01:51.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.084+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/2533085191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df53020 msgr2=0x561a9dfb4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.084+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/2533085191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df53020 0x561a9dfb4b00 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4099830 tx=0x7fe5b40047c0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.084+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/2533085191 shutdown_connections 2024-03-14T10:01:51.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.084+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/2533085191 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561a9dfc0ae0 0x561a9dfc2ed0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.084+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/2533085191 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561a9df5cf40 0x561a9dfb5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.084+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/2533085191 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561a9df52700 0x561a9dfb45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.084+0000 7fe5e3faeb80 1 --2- 172.21.15.78:0/2533085191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a9df53020 0x561a9dfb4b00 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.084+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/2533085191 >> 172.21.15.78:0/2533085191 conn(0x561a9dec82a0 msgr2=0x561a9df68a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:51.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.084+0000 7fe5af400700 0 client.17154 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:51.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.084+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/2533085191 shutdown_connections 2024-03-14T10:01:51.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.085+0000 7fe5e3faeb80 1 -- 172.21.15.78:0/2533085191 wait complete. 2024-03-14T10:01:51.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.085+0000 7fe5e3faeb80 20 client.17154 trim_cache size 0 max 16384 2024-03-14T10:01:51.089 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:01:51.089 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T10:01:51.089 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "2c943d8d-2758-4cf7-931e-3f36d28ca8e0", 2024-03-14T10:01:51.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T10:01:51.089 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:01:51.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.085+0000 7fe5d0400700 1 -- 172.21.15.78:0/2242035493 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe5b00a0c80 msgr2=0x7fe5b00a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.085+0000 7fe5d0400700 1 --2- 172.21.15.78:0/2242035493 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe5b00a0c80 0x7fe5b00a3140 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc006010 tx=0x7fe5bc00be70 comp rx=0 tx=0).stop 2024-03-14T10:01:51.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.085+0000 7fe5d0400700 1 -- 172.21.15.78:0/2242035493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5c8106050 msgr2=0x7fe5c819fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.085+0000 7fe5d0400700 1 --2- 172.21.15.78:0/2242035493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5c8106050 0x7fe5c819fa60 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400e970 tx=0x7fe5b400ec80 comp rx=0 tx=0).stop 2024-03-14T10:01:51.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.085+0000 7fe5d0400700 1 -- 172.21.15.78:0/2242035493 shutdown_connections 2024-03-14T10:01:51.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.085+0000 7fe5d0400700 1 --2- 172.21.15.78:0/2242035493 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe5b00a0c80 0x7fe5b00a3140 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.085+0000 7fe5d0400700 1 --2- 172.21.15.78:0/2242035493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5c81996a0 0x7fe5c8199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.085+0000 7fe5d0400700 1 --2- 172.21.15.78:0/2242035493 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe5c8100f30 0x7fe5c819f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.085+0000 7fe5d0400700 1 --2- 172.21.15.78:0/2242035493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5c8106050 0x7fe5c819fa60 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.085+0000 7fe5d0400700 1 -- 172.21.15.78:0/2242035493 >> 172.21.15.78:0/2242035493 conn(0x7fe5c80fc7b0 msgr2=0x7fe5c80fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:51.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.085+0000 7fe5d0400700 1 -- 172.21.15.78:0/2242035493 shutdown_connections 2024-03-14T10:01:51.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.086+0000 7fe5d0400700 1 -- 172.21.15.78:0/2242035493 wait complete. 2024-03-14T10:01:51.104 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:01:51.161 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:50 smithi078 ceph-mon[38143]: pgmap v235: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:51.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:50 smithi152 ceph-mon[48279]: pgmap v235: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:51.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:50 smithi132 ceph-mon[47959]: pgmap v235: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:51.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.300+0000 7f7ceb400700 1 -- 172.21.15.78:0/21173556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ce41018d0 msgr2=0x7f7ce4105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.300+0000 7f7ceb400700 1 --2- 172.21.15.78:0/21173556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ce41018d0 0x7f7ce4105910 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009990 tx=0x7f7cd0009ca0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.300+0000 7f7ceb400700 1 -- 172.21.15.78:0/21173556 shutdown_connections 2024-03-14T10:01:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.300+0000 7f7ceb400700 1 --2- 172.21.15.78:0/21173556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ce4100f20 0x7f7ce4101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.300+0000 7f7ceb400700 1 --2- 172.21.15.78:0/21173556 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ce4106040 0x7f7ce410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.300+0000 7f7ceb400700 1 --2- 172.21.15.78:0/21173556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ce41018d0 0x7f7ce4105910 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.300+0000 7f7ceb400700 1 -- 172.21.15.78:0/21173556 >> 172.21.15.78:0/21173556 conn(0x7f7ce40fc7a0 msgr2=0x7f7ce40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.300+0000 7f7ceb400700 1 -- 172.21.15.78:0/21173556 shutdown_connections 2024-03-14T10:01:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.300+0000 7f7ceb400700 1 -- 172.21.15.78:0/21173556 wait complete. 2024-03-14T10:01:51.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.301+0000 7f7ceb400700 1 Processor -- start 2024-03-14T10:01:51.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.302+0000 7f7ceb400700 1 -- start start 2024-03-14T10:01:51.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.302+0000 7f7ceb400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ce4100f20 0x7f7ce419f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.302+0000 7f7ceb400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ce4106040 0x7f7ce419fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.302+0000 7f7ceb400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ce4199690 0x7f7ce4199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.302+0000 7f7ceb400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce419a190 con 0x7f7ce4199690 2024-03-14T10:01:51.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.302+0000 7f7ceb400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce419a300 con 0x7f7ce4100f20 2024-03-14T10:01:51.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.302+0000 7f7ceb400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce419a4c0 con 0x7f7ce4106040 2024-03-14T10:01:51.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.302+0000 7f7ce9400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ce4199690 0x7f7ce4199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.302+0000 7f7ce9400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ce4199690 0x7f7ce4199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42418/0 (socket says 172.21.15.78:42418) 2024-03-14T10:01:51.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.302+0000 7f7ce9400700 1 -- 172.21.15.78:0/1025574923 learned_addr learned my addr 172.21.15.78:0/1025574923 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:51.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.302+0000 7f7ce3e00700 1 --2- 172.21.15.78:0/1025574923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ce4106040 0x7f7ce419fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.302+0000 7f7ce8a00700 1 --2- 172.21.15.78:0/1025574923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ce4100f20 0x7f7ce419f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.303+0000 7f7ce9400700 1 -- 172.21.15.78:0/1025574923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ce4106040 msgr2=0x7f7ce419fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.303+0000 7f7ce9400700 1 --2- 172.21.15.78:0/1025574923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ce4106040 0x7f7ce419fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.303+0000 7f7ce9400700 1 -- 172.21.15.78:0/1025574923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ce4100f20 msgr2=0x7f7ce419f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.303+0000 7f7ce9400700 1 --2- 172.21.15.78:0/1025574923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ce4100f20 0x7f7ce419f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.303+0000 7f7ce9400700 1 -- 172.21.15.78:0/1025574923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd00095f0 con 0x7f7ce4199690 2024-03-14T10:01:51.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.303+0000 7f7ce9400700 1 --2- 172.21.15.78:0/1025574923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ce4199690 0x7f7ce4199b40 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc00ef30 tx=0x7f7cdc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:51.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.303+0000 7f7ce1600700 1 -- 172.21.15.78:0/1025574923 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc018070 con 0x7f7ce4199690 2024-03-14T10:01:51.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.303+0000 7f7ceb400700 1 -- 172.21.15.78:0/1025574923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce41a6450 con 0x7f7ce4199690 2024-03-14T10:01:51.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.303+0000 7f7ce1600700 1 -- 172.21.15.78:0/1025574923 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7cdc00f040 con 0x7f7ce4199690 2024-03-14T10:01:51.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.303+0000 7f7ce1600700 1 -- 172.21.15.78:0/1025574923 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc00cd00 con 0x7f7ce4199690 2024-03-14T10:01:51.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.303+0000 7f7ceb400700 1 -- 172.21.15.78:0/1025574923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce41a6930 con 0x7f7ce4199690 2024-03-14T10:01:51.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.307+0000 7f7ce1600700 1 -- 172.21.15.78:0/1025574923 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f7cdc003a00 con 0x7f7ce4199690 2024-03-14T10:01:51.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.308+0000 7f7cfee1fb80 1 Processor -- start 2024-03-14T10:01:51.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.309+0000 7f7cfee1fb80 1 -- start start 2024-03-14T10:01:51.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.309+0000 7f7ce1600700 1 --2- 172.21.15.78:0/1025574923 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7ccc0a0c00 0x7f7ccc0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.309+0000 7f7cfee1fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221680 0x5612a8221a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.309+0000 7f7ce1600700 1 -- 172.21.15.78:0/1025574923 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f7cdc0cc2c0 con 0x7f7ce4199690 2024-03-14T10:01:51.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.309+0000 7f7cfee1fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a8221fa0 0x5612a822b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.309+0000 7f7cfee1fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a822bec0 0x5612a822e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.309+0000 7f7cfee1fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612a817a990 con 0x5612a822bec0 2024-03-14T10:01:51.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.309+0000 7f7cfee1fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612a8178350 con 0x5612a8221680 2024-03-14T10:01:51.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.309+0000 7f7cfee1fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612a8173c40 con 0x5612a8221fa0 2024-03-14T10:01:51.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.309+0000 7f7ce3e00700 1 --2- 172.21.15.78:0/1025574923 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7ccc0a0c00 0x7f7ccc0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.309+0000 7f7ce8a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221680 0x5612a8221a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.309+0000 7f7ce9400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a822bec0 0x5612a822e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.309+0000 7f7ce8a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221680 0x5612a8221a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:49838/0 (socket says 172.21.15.78:49838) 2024-03-14T10:01:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.309+0000 7f7ce8a00700 1 -- 172.21.15.78:0/492470915 learned_addr learned my addr 172.21.15.78:0/492470915 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.309+0000 7f7ce3e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a8221fa0 0x5612a822b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.310+0000 7f7ce8a00700 1 -- 172.21.15.78:0/492470915 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a8221fa0 msgr2=0x5612a822b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.310+0000 7f7ce8a00700 1 --2- 172.21.15.78:0/492470915 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a8221fa0 0x5612a822b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.310+0000 7f7ce8a00700 1 -- 172.21.15.78:0/492470915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a822bec0 msgr2=0x5612a822e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.310+0000 7f7ce8a00700 1 --2- 172.21.15.78:0/492470915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a822bec0 0x5612a822e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.310+0000 7f7ce8a00700 1 -- 172.21.15.78:0/492470915 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612a7ff53b0 con 0x5612a8221680 2024-03-14T10:01:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.310+0000 7f7ce9400700 1 --2- 172.21.15.78:0/492470915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a822bec0 0x5612a822e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:01:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.310+0000 7f7ce3e00700 1 --2- 172.21.15.78:0/1025574923 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7ccc0a0c00 0x7f7ccc0a30c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7ce419af00 tx=0x7f7cd0005c20 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:51.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.310+0000 7f7ce8a00700 1 --2- 172.21.15.78:0/492470915 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221680 0x5612a8221a60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8009d80 tx=0x7f7cd8011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab2d856d46721548 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:51.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.310+0000 7f7cd5c00700 1 -- 172.21.15.78:0/492470915 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8004230 con 0x5612a8221680 2024-03-14T10:01:51.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.310+0000 7f7cd5c00700 1 -- 172.21.15.78:0/492470915 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7cd8004390 con 0x5612a8221680 2024-03-14T10:01:51.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.310+0000 7f7cd5c00700 1 -- 172.21.15.78:0/492470915 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd80197c0 con 0x5612a8221680 2024-03-14T10:01:51.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.311+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/492470915 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221680 msgr2=0x5612a8221a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.311+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/492470915 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221680 0x5612a8221a60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8009d80 tx=0x7f7cd8011720 comp rx=0 tx=0).stop 2024-03-14T10:01:51.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.311+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/492470915 shutdown_connections 2024-03-14T10:01:51.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.311+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/492470915 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221680 0x5612a8221a60 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.311+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/492470915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a822bec0 0x5612a822e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.311+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/492470915 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a8221fa0 0x5612a822b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.311+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/492470915 >> 172.21.15.78:0/492470915 conn(0x5612a8197220 msgr2=0x5612a821caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:51.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.311+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/492470915 shutdown_connections 2024-03-14T10:01:51.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.311+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/492470915 wait complete. 2024-03-14T10:01:51.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.312+0000 7f7cfee1fb80 1 -- start start 2024-03-14T10:01:51.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.312+0000 7f7cfee1fb80 1 Processor -- start 2024-03-14T10:01:51.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.312+0000 7f7cfee1fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a822b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.312+0000 7f7cfee1fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221fa0 0x5612a8228b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.312+0000 7f7cfee1fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a822bec0 0x5612a8229110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.312+0000 7f7cfee1fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612a810fd80 con 0x5612a8221680 2024-03-14T10:01:51.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.312+0000 7f7cfee1fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612a810fa20 con 0x5612a8221fa0 2024-03-14T10:01:51.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.312+0000 7f7cfee1fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612a7ff76a0 con 0x5612a822bec0 2024-03-14T10:01:51.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.312+0000 7f7cfee1fb80 10 client.0 _ll_register_callbacks cb 0x5612a7f7b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:51.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.313+0000 7f7ce8a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a822b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.313+0000 7f7ce8a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a822b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42438/0 (socket says 172.21.15.78:42438) 2024-03-14T10:01:51.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.313+0000 7f7ce8a00700 1 -- 172.21.15.78:0/1909214281 learned_addr learned my addr 172.21.15.78:0/1909214281 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:51.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.313+0000 7f7ce9400700 1 --2- 172.21.15.78:0/1909214281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a822bec0 0x5612a8229110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.313+0000 7f7ce3e00700 1 --2- 172.21.15.78:0/1909214281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221fa0 0x5612a8228b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.313+0000 7f7ce8a00700 1 -- 172.21.15.78:0/1909214281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a822bec0 msgr2=0x5612a8229110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:51.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.313+0000 7f7ce8a00700 1 --2- 172.21.15.78:0/1909214281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a822bec0 0x5612a8229110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.313+0000 7f7ce8a00700 1 -- 172.21.15.78:0/1909214281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221fa0 msgr2=0x5612a8228b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:51.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.313+0000 7f7ce8a00700 1 --2- 172.21.15.78:0/1909214281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221fa0 0x5612a8228b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.313+0000 7f7ce8a00700 1 -- 172.21.15.78:0/1909214281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce41a6930 con 0x5612a8221680 2024-03-14T10:01:51.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.313+0000 7f7ce8a00700 1 --2- 172.21.15.78:0/1909214281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a822b7e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7cd801b040 tx=0x7f7cd8003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1539a2b6a114419 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:51.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.313+0000 7f7cca000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:51.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.313+0000 7f7cca000700 1 -- 172.21.15.78:0/1909214281 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8005ce0 con 0x5612a8221680 2024-03-14T10:01:51.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.314+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/1909214281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5612a8214720 con 0x5612a8221680 2024-03-14T10:01:51.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.314+0000 7f7cca000700 1 -- 172.21.15.78:0/1909214281 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f7cd8005040 con 0x5612a8221680 2024-03-14T10:01:51.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.314+0000 7f7cca000700 1 -- 172.21.15.78:0/1909214281 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cd8005200 con 0x5612a8221680 2024-03-14T10:01:51.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.314+0000 7f7cfee1fb80 10 client.17166 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:51.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.314+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/1909214281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f7cd80099e0 con 0x5612a8221680 2024-03-14T10:01:51.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.314+0000 7f7cca000700 1 -- 172.21.15.78:0/1909214281 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5612a8214720 con 0x5612a8221680 2024-03-14T10:01:51.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.314+0000 7f7cca000700 1 -- 172.21.15.78:0/1909214281 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7cd8021040 con 0x5612a8221680 2024-03-14T10:01:51.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.314+0000 7f7cfee1fb80 10 client.17166 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:51.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.314+0000 7f7cfee1fb80 10 client.17166 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:01:51.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.314+0000 7f7cfee1fb80 20 client.17166 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:51.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.314+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/1909214281 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5612a823b950 0x5612a823dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.314+0000 7f7cfee1fb80 4 client.17166 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:01:51.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.314+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/1909214281 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5612a816c700 con 0x5612a823b950 2024-03-14T10:01:51.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.315+0000 7f7ce3e00700 1 --2- 172.21.15.78:0/1909214281 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5612a823b950 0x5612a823dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.315+0000 7f7ce3e00700 1 --2- 172.21.15.78:0/1909214281 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5612a823b950 0x5612a823dd70 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42a0425cc449cf2d server_cookie=cf58afbe5196fca in_seq=0 out_seq=0 2024-03-14T10:01:51.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.315+0000 7f7cca000700 10 client.17166 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:51.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.316+0000 7f7cca000700 1 -- 172.21.15.78:0/1909214281 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5612a816c700 con 0x5612a823b950 2024-03-14T10:01:51.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.316+0000 7f7cca000700 10 client.17166 handle_command_reply: tid=0 2024-03-14T10:01:51.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.316+0000 7f7cfee1fb80 1 client.17166 shutdown 2024-03-14T10:01:51.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.316+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/1909214281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 msgr2=0x5612a822b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.316+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/1909214281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a822b7e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7cd801b040 tx=0x7f7cd8003910 comp rx=0 tx=0).stop 2024-03-14T10:01:51.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.316+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/1909214281 shutdown_connections 2024-03-14T10:01:51.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.316+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/1909214281 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a822bec0 0x5612a8229110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.316+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/1909214281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a822b7e0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.316+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/1909214281 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5612a823b950 0x5612a823dd70 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.316+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/1909214281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221fa0 0x5612a8228b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.316+0000 7f7cca000700 0 client.17166 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:51.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.316+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/1909214281 >> 172.21.15.78:0/1909214281 conn(0x5612a8197220 msgr2=0x5612a821c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:51.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.316+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/1909214281 shutdown_connections 2024-03-14T10:01:51.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.316+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/1909214281 wait complete. 2024-03-14T10:01:51.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.317+0000 7f7cfee1fb80 20 client.17166 trim_cache size 0 max 16384 2024-03-14T10:01:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.333+0000 7f7cfee1fb80 1 Processor -- start 2024-03-14T10:01:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.333+0000 7f7cfee1fb80 1 -- start start 2024-03-14T10:01:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.333+0000 7f7cfee1fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a821cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.333+0000 7f7cfee1fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221fa0 0x5612a8225dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.333+0000 7f7cfee1fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a822bec0 0x5612a8226310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.333+0000 7f7cfee1fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612a821de30 con 0x5612a8221680 2024-03-14T10:01:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.333+0000 7f7cfee1fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612a821e060 con 0x5612a8221fa0 2024-03-14T10:01:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.333+0000 7f7cfee1fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612a8242a50 con 0x5612a822bec0 2024-03-14T10:01:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.334+0000 7f7ce8a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a821cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.334+0000 7f7ce8a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a821cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42448/0 (socket says 172.21.15.78:42448) 2024-03-14T10:01:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.334+0000 7f7ce8a00700 1 -- 172.21.15.78:0/3008048575 learned_addr learned my addr 172.21.15.78:0/3008048575 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.334+0000 7f7ce3e00700 1 --2- 172.21.15.78:0/3008048575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221fa0 0x5612a8225dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.334+0000 7f7ce9400700 1 --2- 172.21.15.78:0/3008048575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a822bec0 0x5612a8226310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.334+0000 7f7ce8a00700 1 -- 172.21.15.78:0/3008048575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a822bec0 msgr2=0x5612a8226310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:51.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.334+0000 7f7ce8a00700 1 --2- 172.21.15.78:0/3008048575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a822bec0 0x5612a8226310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.334+0000 7f7ce8a00700 1 -- 172.21.15.78:0/3008048575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221fa0 msgr2=0x5612a8225dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:51.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.334+0000 7f7ce8a00700 1 --2- 172.21.15.78:0/3008048575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221fa0 0x5612a8225dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.334+0000 7f7ce8a00700 1 -- 172.21.15.78:0/3008048575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612a7fa1b60 con 0x5612a8221680 2024-03-14T10:01:51.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.334+0000 7f7ce8a00700 1 --2- 172.21.15.78:0/3008048575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a821cc30 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8023040 tx=0x7f7cd8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ee795f350d68d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.334+0000 7f7cd6600700 1 -- 172.21.15.78:0/3008048575 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8003980 con 0x5612a8221680 2024-03-14T10:01:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.335+0000 7f7cd6600700 1 -- 172.21.15.78:0/3008048575 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7cd8023600 con 0x5612a8221680 2024-03-14T10:01:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.335+0000 7f7cd6600700 1 -- 172.21.15.78:0/3008048575 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8005240 con 0x5612a8221680 2024-03-14T10:01:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.335+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/3008048575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 msgr2=0x5612a821cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.335+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/3008048575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a821cc30 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8023040 tx=0x7f7cd8009f90 comp rx=0 tx=0).stop 2024-03-14T10:01:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.335+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/3008048575 shutdown_connections 2024-03-14T10:01:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.335+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/3008048575 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a822bec0 0x5612a8226310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.335+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/3008048575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a821cc30 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.335+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/3008048575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a8221fa0 0x5612a8225dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.335+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/3008048575 >> 172.21.15.78:0/3008048575 conn(0x5612a8197220 msgr2=0x5612a8227440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.335+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/3008048575 shutdown_connections 2024-03-14T10:01:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.335+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/3008048575 wait complete. 2024-03-14T10:01:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.336+0000 7f7cfee1fb80 1 -- start start 2024-03-14T10:01:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.336+0000 7f7cfee1fb80 1 Processor -- start 2024-03-14T10:01:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.336+0000 7f7cfee1fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a8232490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.336+0000 7f7cfee1fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a8221fa0 0x5612a82329d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.336+0000 7f7cfee1fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a822bec0 0x5612a8230910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.336+0000 7f7cfee1fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612a8230ff0 con 0x5612a8221680 2024-03-14T10:01:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.336+0000 7f7cfee1fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612a8231160 con 0x5612a822bec0 2024-03-14T10:01:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.336+0000 7f7cfee1fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612a8231320 con 0x5612a8221fa0 2024-03-14T10:01:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.336+0000 7f7cfee1fb80 10 client.0 _ll_register_callbacks cb 0x7f7cdc009370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:01:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.336+0000 7f7ce8a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a8232490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.336+0000 7f7ce8a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a8232490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42454/0 (socket says 172.21.15.78:42454) 2024-03-14T10:01:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.336+0000 7f7ce8a00700 1 -- 172.21.15.78:0/2533976866 learned_addr learned my addr 172.21.15.78:0/2533976866 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:01:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.336+0000 7f7ce3e00700 1 --2- 172.21.15.78:0/2533976866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a8221fa0 0x5612a82329d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.337+0000 7f7ce9400700 1 --2- 172.21.15.78:0/2533976866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a822bec0 0x5612a8230910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.337+0000 7f7ce8a00700 1 -- 172.21.15.78:0/2533976866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a8221fa0 msgr2=0x5612a82329d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.337+0000 7f7ce8a00700 1 --2- 172.21.15.78:0/2533976866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a8221fa0 0x5612a82329d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.337+0000 7f7ce8a00700 1 -- 172.21.15.78:0/2533976866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a822bec0 msgr2=0x5612a8230910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:01:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.337+0000 7f7ce8a00700 1 --2- 172.21.15.78:0/2533976866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a822bec0 0x5612a8230910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.337+0000 7f7ce8a00700 1 -- 172.21.15.78:0/2533976866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd80099e0 con 0x5612a8221680 2024-03-14T10:01:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.337+0000 7f7ce8a00700 1 --2- 172.21.15.78:0/2533976866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a8232490 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8013040 tx=0x7f7cd8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1c22f5da08cd4e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:01:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.337+0000 7f7cca000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:01:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.337+0000 7f7cca000700 1 -- 172.21.15.78:0/2533976866 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8023a40 con 0x5612a8221680 2024-03-14T10:01:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.337+0000 7f7cca000700 1 -- 172.21.15.78:0/2533976866 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f7cd8000e40 con 0x5612a8221680 2024-03-14T10:01:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.337+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/2533976866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7cd8004390 con 0x5612a8221680 2024-03-14T10:01:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.338+0000 7f7cca000700 1 -- 172.21.15.78:0/2533976866 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cd8003dd0 con 0x5612a8221680 2024-03-14T10:01:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.338+0000 7f7cfee1fb80 10 client.17172 fetch_fsmap learned FSMap version 25 2024-03-14T10:01:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.338+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/2533976866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5612a8139ea0 con 0x5612a8221680 2024-03-14T10:01:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.338+0000 7f7cca000700 1 -- 172.21.15.78:0/2533976866 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7cd8004390 con 0x5612a8221680 2024-03-14T10:01:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.338+0000 7f7cca000700 1 -- 172.21.15.78:0/2533976866 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7cd8021070 con 0x5612a8221680 2024-03-14T10:01:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.338+0000 7f7cfee1fb80 10 client.17172 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:01:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.338+0000 7f7cfee1fb80 10 client.17172 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:01:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.338+0000 7f7cfee1fb80 20 client.17172 populate_metadata read hostname 'smithi078' 2024-03-14T10:01:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.338+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/2533976866 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5612a828e1d0 0x5612a82905c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:01:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.338+0000 7f7cfee1fb80 4 client.17172 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:01:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.338+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/2533976866 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5612a7fc2cf0 con 0x5612a828e1d0 2024-03-14T10:01:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.338+0000 7f7ce3e00700 1 --2- 172.21.15.78:0/2533976866 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5612a828e1d0 0x5612a82905c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:01:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.339+0000 7f7ce3e00700 1 --2- 172.21.15.78:0/2533976866 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5612a828e1d0 0x5612a82905c0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=690099b928d7ce6d server_cookie=49ceb6d01d4e9a6e in_seq=0 out_seq=0 2024-03-14T10:01:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.339+0000 7f7cca000700 10 client.17172 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.339+0000 7f7cca000700 1 -- 172.21.15.78:0/2533976866 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5612a7fc2cf0 con 0x5612a828e1d0 2024-03-14T10:01:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.339+0000 7f7cca000700 10 client.17172 handle_command_reply: tid=0 2024-03-14T10:01:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.339+0000 7f7cfee1fb80 1 client.17172 shutdown 2024-03-14T10:01:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.339+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/2533976866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 msgr2=0x5612a8232490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.339+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/2533976866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a8232490 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8013040 tx=0x7f7cd8009da0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.339+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/2533976866 shutdown_connections 2024-03-14T10:01:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.339+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/2533976866 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5612a828e1d0 0x5612a82905c0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.339+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/2533976866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5612a822bec0 0x5612a8230910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.339+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/2533976866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5612a8221680 0x5612a8232490 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.339+0000 7f7cca000700 0 client.17172 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:01:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.339+0000 7f7cfee1fb80 1 --2- 172.21.15.78:0/2533976866 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5612a8221fa0 0x5612a82329d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.339+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/2533976866 >> 172.21.15.78:0/2533976866 conn(0x5612a8197220 msgr2=0x5612a82280b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.339+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/2533976866 shutdown_connections 2024-03-14T10:01:51.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.340+0000 7f7cfee1fb80 1 -- 172.21.15.78:0/2533976866 wait complete. 2024-03-14T10:01:51.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.340+0000 7f7cfee1fb80 20 client.17172 trim_cache size 0 max 16384 2024-03-14T10:01:51.345 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:01:51.345 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-14T10:01:51.345 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T10:01:51.345 INFO:teuthology.orchestra.run.smithi078.stdout: "2c943d8d-2758-4cf7-931e-3f36d28ca8e0": { 2024-03-14T10:01:51.345 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T10:01:51.345 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "2c943d8d-2758-4cf7-931e-3f36d28ca8e0", 2024-03-14T10:01:51.345 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T10:01:51.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:01:51.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:01:51.345 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:01:51.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.340+0000 7f7ceb400700 1 -- 172.21.15.78:0/1025574923 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7ccc0a0c00 msgr2=0x7f7ccc0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.340+0000 7f7ceb400700 1 --2- 172.21.15.78:0/1025574923 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7ccc0a0c00 0x7f7ccc0a30c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7ce419af00 tx=0x7f7cd0005c20 comp rx=0 tx=0).stop 2024-03-14T10:01:51.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.340+0000 7f7ceb400700 1 -- 172.21.15.78:0/1025574923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ce4199690 msgr2=0x7f7ce4199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:01:51.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.340+0000 7f7ceb400700 1 --2- 172.21.15.78:0/1025574923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ce4199690 0x7f7ce4199b40 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc00ef30 tx=0x7f7cdc00c3a0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.340+0000 7f7ceb400700 1 -- 172.21.15.78:0/1025574923 shutdown_connections 2024-03-14T10:01:51.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.340+0000 7f7ceb400700 1 --2- 172.21.15.78:0/1025574923 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7ccc0a0c00 0x7f7ccc0a30c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.340+0000 7f7ceb400700 1 --2- 172.21.15.78:0/1025574923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7ce4100f20 0x7f7ce419f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.340+0000 7f7ceb400700 1 --2- 172.21.15.78:0/1025574923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ce4199690 0x7f7ce4199b40 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.340+0000 7f7ceb400700 1 --2- 172.21.15.78:0/1025574923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7ce4106040 0x7f7ce419fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:01:51.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.340+0000 7f7ceb400700 1 -- 172.21.15.78:0/1025574923 >> 172.21.15.78:0/1025574923 conn(0x7f7ce40fc7a0 msgr2=0x7f7ce40feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:01:51.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.340+0000 7f7ceb400700 1 -- 172.21.15.78:0/1025574923 shutdown_connections 2024-03-14T10:01:51.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:01:51.340+0000 7f7ceb400700 1 -- 172.21.15.78:0/1025574923 wait complete. 2024-03-14T10:01:51.360 INFO:tasks.cephfs.filesystem:scrub status for tag:2c943d8d-2758-4cf7-931e-3f36d28ca8e0 - {'path': '/', 'tag': '2c943d8d-2758-4cf7-931e-3f36d28ca8e0', 'options': 'recursive,force'} 2024-03-14T10:01:52.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:51 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T10:01:52.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:51 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T10:01:52.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:51 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T10:01:52.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:51 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T10:01:52.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:51 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T10:01:52.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:51 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T10:01:52.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:51 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T10:01:52.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:51 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T10:01:52.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:51 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T10:01:52.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:51 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T10:01:52.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:51 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T10:01:52.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:51 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T10:01:53.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:52 smithi152 ceph-mon[48279]: pgmap v236: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:01:53.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:52 smithi132 ceph-mon[47959]: pgmap v236: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:01:53.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:52 smithi078 ceph-mon[38143]: pgmap v236: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:01:54.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:53 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T10:01:54.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:53 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T10:01:54.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:53 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T10:01:55.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:54 smithi152 ceph-mon[48279]: pgmap v237: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:55.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:54 smithi132 ceph-mon[47959]: pgmap v237: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:55.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:54 smithi078 ceph-mon[38143]: pgmap v237: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:01:57.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:57 smithi152 ceph-mon[48279]: pgmap v238: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:01:57.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:57 smithi132 ceph-mon[47959]: pgmap v238: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:01:57.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:57 smithi078 ceph-mon[38143]: pgmap v238: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:01:59.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:01:58 smithi152 ceph-mon[48279]: pgmap v239: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:01:59.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:01:58 smithi132 ceph-mon[47959]: pgmap v239: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:01:59.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:01:58 smithi078 ceph-mon[38143]: pgmap v239: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:02:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:01 smithi152 ceph-mon[48279]: pgmap v240: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:02:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:01 smithi132 ceph-mon[47959]: pgmap v240: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:02:01.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:01 smithi078 ceph-mon[38143]: pgmap v240: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:02:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:03 smithi152 ceph-mon[48279]: pgmap v241: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:02:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:03 smithi132 ceph-mon[47959]: pgmap v241: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:02:03.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:03 smithi078 ceph-mon[38143]: pgmap v241: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:02:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:05 smithi152 ceph-mon[48279]: pgmap v242: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:02:05.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:05 smithi132 ceph-mon[47959]: pgmap v242: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:02:05.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:05 smithi078 ceph-mon[38143]: pgmap v242: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:02:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:07 smithi152 ceph-mon[48279]: pgmap v243: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:02:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:07 smithi132 ceph-mon[47959]: pgmap v243: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:02:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:07 smithi078 ceph-mon[38143]: pgmap v243: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:02:09.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:08 smithi152 ceph-mon[48279]: pgmap v244: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:02:09.275 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:08 smithi132 ceph-mon[47959]: pgmap v244: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:02:09.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:08 smithi078 ceph-mon[38143]: pgmap v244: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:02:10.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:09 smithi152 ceph-mon[48279]: osdmap e91: 12 total, 12 up, 12 in 2024-03-14T10:02:10.273 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:09 smithi132 ceph-mon[47959]: osdmap e91: 12 total, 12 up, 12 in 2024-03-14T10:02:10.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:09 smithi078 ceph-mon[38143]: osdmap e91: 12 total, 12 up, 12 in 2024-03-14T10:02:11.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:10 smithi152 ceph-mon[48279]: pgmap v246: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-14T10:02:11.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:10 smithi132 ceph-mon[47959]: pgmap v246: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-14T10:02:11.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:10 smithi078 ceph-mon[38143]: pgmap v246: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-14T10:02:13.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:13 smithi152 ceph-mon[48279]: pgmap v247: 129 pgs: 6 active+clean+snaptrim, 117 active+clean, 6 active+clean+snaptrim_wait; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:02:13.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:13 smithi132 ceph-mon[47959]: pgmap v247: 129 pgs: 6 active+clean+snaptrim, 117 active+clean, 6 active+clean+snaptrim_wait; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:02:13.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:13 smithi078 ceph-mon[38143]: pgmap v247: 129 pgs: 6 active+clean+snaptrim, 117 active+clean, 6 active+clean+snaptrim_wait; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:02:15.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:15 smithi152 ceph-mon[48279]: pgmap v248: 129 pgs: 6 active+clean+snaptrim, 117 active+clean, 6 active+clean+snaptrim_wait; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:02:15.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:15 smithi132 ceph-mon[47959]: pgmap v248: 129 pgs: 6 active+clean+snaptrim, 117 active+clean, 6 active+clean+snaptrim_wait; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:02:15.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:15 smithi078 ceph-mon[38143]: pgmap v248: 129 pgs: 6 active+clean+snaptrim, 117 active+clean, 6 active+clean+snaptrim_wait; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:02:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:17 smithi152 ceph-mon[48279]: pgmap v249: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:02:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:17 smithi132 ceph-mon[47959]: pgmap v249: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:02:17.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:17 smithi078 ceph-mon[38143]: pgmap v249: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:02:19.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:18 smithi152 ceph-mon[48279]: osdmap e92: 12 total, 12 up, 12 in 2024-03-14T10:02:19.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:18 smithi152 ceph-mon[48279]: pgmap v251: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2024-03-14T10:02:19.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:18 smithi132 ceph-mon[47959]: osdmap e92: 12 total, 12 up, 12 in 2024-03-14T10:02:19.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:18 smithi132 ceph-mon[47959]: pgmap v251: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2024-03-14T10:02:19.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:18 smithi078 ceph-mon[38143]: osdmap e92: 12 total, 12 up, 12 in 2024-03-14T10:02:19.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:18 smithi078 ceph-mon[38143]: pgmap v251: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2024-03-14T10:02:21.361 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:02:21.428 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:21 smithi078 ceph-mon[38143]: pgmap v252: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:02:21.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:21 smithi152 ceph-mon[48279]: pgmap v252: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:02:21.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:21 smithi132 ceph-mon[47959]: pgmap v252: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:02:21.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.572+0000 7f5f21e00700 1 -- 172.21.15.78:0/2868183676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f1c1018e0 msgr2=0x7f5f1c105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:21.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.572+0000 7f5f21e00700 1 --2- 172.21.15.78:0/2868183676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f1c1018e0 0x7f5f1c105920 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5f08009990 tx=0x7f5f08009ca0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.572+0000 7f5f21e00700 1 -- 172.21.15.78:0/2868183676 shutdown_connections 2024-03-14T10:02:21.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.572+0000 7f5f21e00700 1 --2- 172.21.15.78:0/2868183676 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f1c106050 0x7f5f1c10dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.572+0000 7f5f21e00700 1 --2- 172.21.15.78:0/2868183676 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f1c100f30 0x7f5f1c101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.572+0000 7f5f21e00700 1 --2- 172.21.15.78:0/2868183676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f1c1018e0 0x7f5f1c105920 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.572+0000 7f5f21e00700 1 -- 172.21.15.78:0/2868183676 >> 172.21.15.78:0/2868183676 conn(0x7f5f1c0fc7b0 msgr2=0x7f5f1c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:21.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.572+0000 7f5f21e00700 1 -- 172.21.15.78:0/2868183676 shutdown_connections 2024-03-14T10:02:21.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.572+0000 7f5f21e00700 1 -- 172.21.15.78:0/2868183676 wait complete. 2024-03-14T10:02:21.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.573+0000 7f5f21e00700 1 Processor -- start 2024-03-14T10:02:21.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.573+0000 7f5f21e00700 1 -- start start 2024-03-14T10:02:21.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f21e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f1c100f30 0x7f5f1c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f21e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f1c106050 0x7f5f1c0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f21e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f1c073b30 0x7f5f1c1a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f21e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c1a5db0 con 0x7f5f1c106050 2024-03-14T10:02:21.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f21e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c1a5f20 con 0x7f5f1c100f30 2024-03-14T10:02:21.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f21e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c1a60e0 con 0x7f5f1c073b30 2024-03-14T10:02:21.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f1a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f1c106050 0x7f5f1c0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f1a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f1c106050 0x7f5f1c0735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38508/0 (socket says 172.21.15.78:38508) 2024-03-14T10:02:21.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f1a000700 1 -- 172.21.15.78:0/610819426 learned_addr learned my addr 172.21.15.78:0/610819426 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:21.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f1aa00700 1 --2- 172.21.15.78:0/610819426 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f1c100f30 0x7f5f1c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f1b400700 1 --2- 172.21.15.78:0/610819426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f1c073b30 0x7f5f1c1a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f1a000700 1 -- 172.21.15.78:0/610819426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f1c073b30 msgr2=0x7f5f1c1a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:21.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f1a000700 1 --2- 172.21.15.78:0/610819426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f1c073b30 0x7f5f1c1a57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f1a000700 1 -- 172.21.15.78:0/610819426 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f1c100f30 msgr2=0x7f5f1c0730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:21.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f1a000700 1 --2- 172.21.15.78:0/610819426 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f1c100f30 0x7f5f1c0730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f1a000700 1 -- 172.21.15.78:0/610819426 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f080095f0 con 0x7f5f1c106050 2024-03-14T10:02:21.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f1aa00700 1 --2- 172.21.15.78:0/610819426 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f1c100f30 0x7f5f1c0730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:02:21.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.574+0000 7f5f1b400700 1 --2- 172.21.15.78:0/610819426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f1c073b30 0x7f5f1c1a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:02:21.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.575+0000 7f5f1a000700 1 --2- 172.21.15.78:0/610819426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f1c106050 0x7f5f1c0735f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5f08000c00 tx=0x7f5f08009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:21.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.575+0000 7f5f20a00700 1 -- 172.21.15.78:0/610819426 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0801d070 con 0x7f5f1c106050 2024-03-14T10:02:21.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.575+0000 7f5f21e00700 1 -- 172.21.15.78:0/610819426 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f1c1a6360 con 0x7f5f1c106050 2024-03-14T10:02:21.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.575+0000 7f5f20a00700 1 -- 172.21.15.78:0/610819426 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f08021410 con 0x7f5f1c106050 2024-03-14T10:02:21.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.575+0000 7f5f20a00700 1 -- 172.21.15.78:0/610819426 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f08017650 con 0x7f5f1c106050 2024-03-14T10:02:21.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.575+0000 7f5f21e00700 1 -- 172.21.15.78:0/610819426 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f1c1a6780 con 0x7f5f1c106050 2024-03-14T10:02:21.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.578+0000 7f5f20a00700 1 -- 172.21.15.78:0/610819426 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f5f080217f0 con 0x7f5f1c106050 2024-03-14T10:02:21.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.579+0000 7f5f358c8b80 1 Processor -- start 2024-03-14T10:02:21.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.579+0000 7f5f20a00700 1 --2- 172.21.15.78:0/610819426 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5f000a0c00 0x7f5f000a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.579+0000 7f5f358c8b80 1 -- start start 2024-03-14T10:02:21.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.579+0000 7f5f20a00700 1 -- 172.21.15.78:0/610819426 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f5f080cdae0 con 0x7f5f1c106050 2024-03-14T10:02:21.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.579+0000 7f5f1b400700 1 --2- 172.21.15.78:0/610819426 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5f000a0c00 0x7f5f000a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f358c8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb92c680 0x5597cb92ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f358c8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 0x5597cb936980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f358c8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb936ec0 0x5597cb9392b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f358c8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597cb885990 con 0x5597cb936ec0 2024-03-14T10:02:21.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f358c8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597cb883350 con 0x5597cb92c680 2024-03-14T10:02:21.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f358c8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597cb87ec40 con 0x5597cb92cfa0 2024-03-14T10:02:21.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f1aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb92c680 0x5597cb92ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f1aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb92c680 0x5597cb92ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:49484/0 (socket says 172.21.15.78:49484) 2024-03-14T10:02:21.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f1aa00700 1 -- 172.21.15.78:0/348125828 learned_addr learned my addr 172.21.15.78:0/348125828 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:21.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f1a000700 1 --2- 172.21.15.78:0/348125828 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 0x5597cb936980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f1b400700 1 --2- 172.21.15.78:0/348125828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb936ec0 0x5597cb9392b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f1b400700 1 --2- 172.21.15.78:0/610819426 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5f000a0c00 0x7f5f000a30c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5f14009410 tx=0x7f5f14009920 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:21.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f1aa00700 1 -- 172.21.15.78:0/348125828 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 msgr2=0x5597cb936980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:21.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f1aa00700 1 --2- 172.21.15.78:0/348125828 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 0x5597cb936980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f1aa00700 1 -- 172.21.15.78:0/348125828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb936ec0 msgr2=0x5597cb9392b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:21.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f1aa00700 1 --2- 172.21.15.78:0/348125828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb936ec0 0x5597cb9392b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f1aa00700 1 -- 172.21.15.78:0/348125828 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597cb7003b0 con 0x5597cb92c680 2024-03-14T10:02:21.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.580+0000 7f5f1a000700 1 --2- 172.21.15.78:0/348125828 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 0x5597cb936980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:21.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.581+0000 7f5f1aa00700 1 --2- 172.21.15.78:0/348125828 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb92c680 0x5597cb92ca60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c009de0 tx=0x7f5f0c0116c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fef2501bb10d53be server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:21.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.581+0000 7f5f07e00700 1 -- 172.21.15.78:0/348125828 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c011e20 con 0x5597cb92c680 2024-03-14T10:02:21.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.581+0000 7f5f07e00700 1 -- 172.21.15.78:0/348125828 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f0c012910 con 0x5597cb92c680 2024-03-14T10:02:21.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.581+0000 7f5f07e00700 1 -- 172.21.15.78:0/348125828 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c00f8a0 con 0x5597cb92c680 2024-03-14T10:02:21.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.582+0000 7f5f358c8b80 1 -- 172.21.15.78:0/348125828 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb92c680 msgr2=0x5597cb92ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:21.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.582+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/348125828 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb92c680 0x5597cb92ca60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c009de0 tx=0x7f5f0c0116c0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.582+0000 7f5f358c8b80 1 -- 172.21.15.78:0/348125828 shutdown_connections 2024-03-14T10:02:21.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.582+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/348125828 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb92c680 0x5597cb92ca60 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.582+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/348125828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb936ec0 0x5597cb9392b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.582+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/348125828 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 0x5597cb936980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.582+0000 7f5f358c8b80 1 -- 172.21.15.78:0/348125828 >> 172.21.15.78:0/348125828 conn(0x5597cb8a2220 msgr2=0x5597cb927aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:21.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.582+0000 7f5f358c8b80 1 -- 172.21.15.78:0/348125828 shutdown_connections 2024-03-14T10:02:21.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.582+0000 7f5f358c8b80 1 -- 172.21.15.78:0/348125828 wait complete. 2024-03-14T10:02:21.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.582+0000 7f5f358c8b80 1 -- start start 2024-03-14T10:02:21.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.582+0000 7f5f358c8b80 1 Processor -- start 2024-03-14T10:02:21.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.583+0000 7f5f358c8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb92c680 0x5597cb936800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.583+0000 7f5f358c8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92cfa0 0x5597cb933b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.583+0000 7f5f358c8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb936ec0 0x5597cb934150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.583+0000 7f5f358c8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597cb81ad80 con 0x5597cb92cfa0 2024-03-14T10:02:21.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.583+0000 7f5f358c8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597cb81aa20 con 0x5597cb92c680 2024-03-14T10:02:21.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.583+0000 7f5f358c8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597cb7026a0 con 0x5597cb936ec0 2024-03-14T10:02:21.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.583+0000 7f5f358c8b80 10 client.0 _ll_register_callbacks cb 0x5597cb6863a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:21.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.583+0000 7f5f1a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92cfa0 0x5597cb933b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.583+0000 7f5f1a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92cfa0 0x5597cb933b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38526/0 (socket says 172.21.15.78:38526) 2024-03-14T10:02:21.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.583+0000 7f5f1aa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb92c680 0x5597cb936800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.583+0000 7f5f1a000700 1 -- 172.21.15.78:0/2922790099 learned_addr learned my addr 172.21.15.78:0/2922790099 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:21.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.583+0000 7f5f1b400700 1 --2- 172.21.15.78:0/2922790099 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb936ec0 0x5597cb934150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.583+0000 7f5f1a000700 1 -- 172.21.15.78:0/2922790099 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb936ec0 msgr2=0x5597cb934150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:21.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.584+0000 7f5f1a000700 1 --2- 172.21.15.78:0/2922790099 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb936ec0 0x5597cb934150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.584+0000 7f5f1a000700 1 -- 172.21.15.78:0/2922790099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb92c680 msgr2=0x5597cb936800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.584+0000 7f5f1a000700 1 --2- 172.21.15.78:0/2922790099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb92c680 0x5597cb936800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.584+0000 7f5f1a000700 1 -- 172.21.15.78:0/2922790099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f1c1a6780 con 0x5597cb92cfa0 2024-03-14T10:02:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.584+0000 7f5f1a000700 1 --2- 172.21.15.78:0/2922790099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92cfa0 0x5597cb933b90 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5f0809ab30 tx=0x7f5f0809a6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc664c76cee99ae server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.584+0000 7f5f04c00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.584+0000 7f5f04c00700 1 -- 172.21.15.78:0/2922790099 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0809a290 con 0x5597cb92cfa0 2024-03-14T10:02:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.584+0000 7f5f358c8b80 1 -- 172.21.15.78:0/2922790099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5597cb91f720 con 0x5597cb92cfa0 2024-03-14T10:02:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.584+0000 7f5f04c00700 1 -- 172.21.15.78:0/2922790099 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f5f0809b2a0 con 0x5597cb92cfa0 2024-03-14T10:02:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.584+0000 7f5f04c00700 1 -- 172.21.15.78:0/2922790099 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f0809aba0 con 0x5597cb92cfa0 2024-03-14T10:02:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.584+0000 7f5f358c8b80 10 client.17184 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.584+0000 7f5f358c8b80 1 -- 172.21.15.78:0/2922790099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f5f0c009a10 con 0x5597cb92cfa0 2024-03-14T10:02:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.584+0000 7f5f04c00700 1 -- 172.21.15.78:0/2922790099 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5597cb91f720 con 0x5597cb92cfa0 2024-03-14T10:02:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.585+0000 7f5f04c00700 1 -- 172.21.15.78:0/2922790099 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5f0809bbb0 con 0x5597cb92cfa0 2024-03-14T10:02:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.585+0000 7f5f358c8b80 10 client.17184 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.585+0000 7f5f358c8b80 10 client.17184 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:02:21.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.585+0000 7f5f358c8b80 20 client.17184 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.585+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/2922790099 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5597cb946620 0x5597cb948a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.585+0000 7f5f358c8b80 4 client.17184 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:02:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.585+0000 7f5f358c8b80 1 -- 172.21.15.78:0/2922790099 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5597cb877700 con 0x5597cb946620 2024-03-14T10:02:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.585+0000 7f5f1aa00700 1 --2- 172.21.15.78:0/2922790099 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5597cb946620 0x5597cb948a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.586+0000 7f5f04c00700 10 client.17184 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:21.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.586+0000 7f5f1aa00700 1 --2- 172.21.15.78:0/2922790099 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5597cb946620 0x5597cb948a40 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c826c9cc347d16c3 server_cookie=5af102511ddbc3f1 in_seq=0 out_seq=0 2024-03-14T10:02:21.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.586+0000 7f5f04c00700 1 -- 172.21.15.78:0/2922790099 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5597cb877700 con 0x5597cb946620 2024-03-14T10:02:21.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.587+0000 7f5f04c00700 10 client.17184 handle_command_reply: tid=0 2024-03-14T10:02:21.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.587+0000 7f5f358c8b80 1 client.17184 shutdown 2024-03-14T10:02:21.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.587+0000 7f5f358c8b80 1 -- 172.21.15.78:0/2922790099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92cfa0 msgr2=0x5597cb933b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:21.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.587+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/2922790099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92cfa0 0x5597cb933b90 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5f0809ab30 tx=0x7f5f0809a6e0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.587+0000 7f5f358c8b80 1 -- 172.21.15.78:0/2922790099 shutdown_connections 2024-03-14T10:02:21.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.587+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/2922790099 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5597cb946620 0x5597cb948a40 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.587+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/2922790099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb92c680 0x5597cb936800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.587+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/2922790099 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb936ec0 0x5597cb934150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.587+0000 7f5f04c00700 0 client.17184 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:21.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.587+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/2922790099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92cfa0 0x5597cb933b90 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.587+0000 7f5f358c8b80 1 -- 172.21.15.78:0/2922790099 >> 172.21.15.78:0/2922790099 conn(0x5597cb8a2220 msgr2=0x5597cb927690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:21.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.587+0000 7f5f358c8b80 1 -- 172.21.15.78:0/2922790099 shutdown_connections 2024-03-14T10:02:21.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.587+0000 7f5f358c8b80 1 -- 172.21.15.78:0/2922790099 wait complete. 2024-03-14T10:02:21.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.587+0000 7f5f358c8b80 20 client.17184 trim_cache size 0 max 16384 2024-03-14T10:02:21.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.604+0000 7f5f358c8b80 1 Processor -- start 2024-03-14T10:02:21.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.604+0000 7f5f358c8b80 1 -- start start 2024-03-14T10:02:21.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.604+0000 7f5f358c8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92c680 0x5597cb93db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.604+0000 7f5f358c8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 0x5597cb9275d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.604+0000 7f5f358c8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb936ec0 0x5597cb927b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.604+0000 7f5f358c8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597cb92e310 con 0x5597cb92c680 2024-03-14T10:02:21.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.604+0000 7f5f358c8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597cb92cde0 con 0x5597cb936ec0 2024-03-14T10:02:21.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.604+0000 7f5f358c8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597cb928e30 con 0x5597cb92cfa0 2024-03-14T10:02:21.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.605+0000 7f5f1aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92c680 0x5597cb93db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.605+0000 7f5f1aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92c680 0x5597cb93db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38532/0 (socket says 172.21.15.78:38532) 2024-03-14T10:02:21.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.605+0000 7f5f1aa00700 1 -- 172.21.15.78:0/373900119 learned_addr learned my addr 172.21.15.78:0/373900119 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:21.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.605+0000 7f5f1a000700 1 --2- 172.21.15.78:0/373900119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 0x5597cb9275d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.605+0000 7f5f1b400700 1 --2- 172.21.15.78:0/373900119 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb936ec0 0x5597cb927b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.605+0000 7f5f1aa00700 1 -- 172.21.15.78:0/373900119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 msgr2=0x5597cb9275d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:21.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.605+0000 7f5f1aa00700 1 --2- 172.21.15.78:0/373900119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 0x5597cb9275d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.605+0000 7f5f1aa00700 1 -- 172.21.15.78:0/373900119 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb936ec0 msgr2=0x5597cb927b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:21.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.605+0000 7f5f1aa00700 1 --2- 172.21.15.78:0/373900119 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb936ec0 0x5597cb927b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.605+0000 7f5f1aa00700 1 -- 172.21.15.78:0/373900119 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597cb6acb60 con 0x5597cb92c680 2024-03-14T10:02:21.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.605+0000 7f5f1a000700 1 --2- 172.21.15.78:0/373900119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 0x5597cb9275d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T10:02:21.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.605+0000 7f5f1b400700 1 --2- 172.21.15.78:0/373900119 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb936ec0 0x5597cb927b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T10:02:21.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.605+0000 7f5f1aa00700 1 --2- 172.21.15.78:0/373900119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92c680 0x5597cb93db50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c009db0 tx=0x7f5f0c0098d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e2d952e6cc0c03 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:21.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.605+0000 7f5f11200700 1 -- 172.21.15.78:0/373900119 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c011d40 con 0x5597cb92c680 2024-03-14T10:02:21.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.605+0000 7f5f11200700 1 -- 172.21.15.78:0/373900119 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f0c0290b0 con 0x5597cb92c680 2024-03-14T10:02:21.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.606+0000 7f5f11200700 1 -- 172.21.15.78:0/373900119 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c029980 con 0x5597cb92c680 2024-03-14T10:02:21.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.606+0000 7f5f358c8b80 1 -- 172.21.15.78:0/373900119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92c680 msgr2=0x5597cb93db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:21.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.606+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/373900119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92c680 0x5597cb93db50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c009db0 tx=0x7f5f0c0098d0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.606+0000 7f5f358c8b80 1 -- 172.21.15.78:0/373900119 shutdown_connections 2024-03-14T10:02:21.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.606+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/373900119 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb936ec0 0x5597cb927b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.606+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/373900119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92c680 0x5597cb93db50 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.606+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/373900119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 0x5597cb9275d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.606+0000 7f5f358c8b80 1 -- 172.21.15.78:0/373900119 >> 172.21.15.78:0/373900119 conn(0x5597cb8a2220 msgr2=0x5597cb92eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:21.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.606+0000 7f5f358c8b80 1 -- 172.21.15.78:0/373900119 shutdown_connections 2024-03-14T10:02:21.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.606+0000 7f5f358c8b80 1 -- 172.21.15.78:0/373900119 wait complete. 2024-03-14T10:02:21.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.607+0000 7f5f358c8b80 1 -- start start 2024-03-14T10:02:21.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.607+0000 7f5f358c8b80 1 Processor -- start 2024-03-14T10:02:21.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.607+0000 7f5f358c8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92c680 0x5597cb93c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.607+0000 7f5f358c8b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 0x5597cb93c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.607+0000 7f5f358c8b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb936ec0 0x5597cb93cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.608+0000 7f5f358c8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597cb998110 con 0x5597cb92c680 2024-03-14T10:02:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.608+0000 7f5f358c8b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597cb998250 con 0x5597cb936ec0 2024-03-14T10:02:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.608+0000 7f5f358c8b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597cb998390 con 0x5597cb92cfa0 2024-03-14T10:02:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.608+0000 7f5f358c8b80 10 client.0 _ll_register_callbacks cb 0x7f5f08096ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.608+0000 7f5f1aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92c680 0x5597cb93c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.608+0000 7f5f1aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92c680 0x5597cb93c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38540/0 (socket says 172.21.15.78:38540) 2024-03-14T10:02:21.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.608+0000 7f5f1aa00700 1 -- 172.21.15.78:0/582199786 learned_addr learned my addr 172.21.15.78:0/582199786 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:21.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.608+0000 7f5f1a000700 1 --2- 172.21.15.78:0/582199786 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 0x5597cb93c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.608+0000 7f5f1b400700 1 --2- 172.21.15.78:0/582199786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb936ec0 0x5597cb93cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.608+0000 7f5f1aa00700 1 -- 172.21.15.78:0/582199786 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 msgr2=0x5597cb93c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:21.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.608+0000 7f5f1aa00700 1 --2- 172.21.15.78:0/582199786 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 0x5597cb93c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.608+0000 7f5f1aa00700 1 -- 172.21.15.78:0/582199786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb936ec0 msgr2=0x5597cb93cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:21.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.608+0000 7f5f1aa00700 1 --2- 172.21.15.78:0/582199786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb936ec0 0x5597cb93cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.608+0000 7f5f1aa00700 1 -- 172.21.15.78:0/582199786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5597cb7003b0 con 0x5597cb92c680 2024-03-14T10:02:21.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.608+0000 7f5f1aa00700 1 --2- 172.21.15.78:0/582199786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92c680 0x5597cb93c120 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c011e80 tx=0x7f5f0c011a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118012299850d019 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:21.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.608+0000 7f5f04c00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:21.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.609+0000 7f5f04c00700 1 -- 172.21.15.78:0/582199786 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c012b00 con 0x5597cb92c680 2024-03-14T10:02:21.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.609+0000 7f5f04c00700 1 -- 172.21.15.78:0/582199786 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f5f0c037070 con 0x5597cb92c680 2024-03-14T10:02:21.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.609+0000 7f5f358c8b80 1 -- 172.21.15.78:0/582199786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5f0c012910 con 0x5597cb92c680 2024-03-14T10:02:21.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.609+0000 7f5f04c00700 1 -- 172.21.15.78:0/582199786 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f0c012c60 con 0x5597cb92c680 2024-03-14T10:02:21.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.609+0000 7f5f358c8b80 10 client.17190 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:21.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.609+0000 7f5f358c8b80 1 -- 172.21.15.78:0/582199786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5597cb844ea0 con 0x5597cb92c680 2024-03-14T10:02:21.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.609+0000 7f5f04c00700 1 -- 172.21.15.78:0/582199786 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5f0c012910 con 0x5597cb92c680 2024-03-14T10:02:21.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.609+0000 7f5f04c00700 1 -- 172.21.15.78:0/582199786 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5f0c02f020 con 0x5597cb92c680 2024-03-14T10:02:21.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.609+0000 7f5f358c8b80 10 client.17190 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:21.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.609+0000 7f5f358c8b80 10 client.17190 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:02:21.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.609+0000 7f5f358c8b80 20 client.17190 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:21.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.609+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/582199786 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5597cb9995b0 0x5597cb99b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.609+0000 7f5f358c8b80 4 client.17190 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:02:21.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.609+0000 7f5f358c8b80 1 -- 172.21.15.78:0/582199786 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5597cb6cdcf0 con 0x5597cb9995b0 2024-03-14T10:02:21.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.610+0000 7f5f1a000700 1 --2- 172.21.15.78:0/582199786 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5597cb9995b0 0x5597cb99b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.610+0000 7f5f04c00700 10 client.17190 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:21.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.610+0000 7f5f1a000700 1 --2- 172.21.15.78:0/582199786 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5597cb9995b0 0x5597cb99b9d0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c3e34675ecd7139 server_cookie=c242d3dfd9bff4db in_seq=0 out_seq=0 2024-03-14T10:02:21.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.610+0000 7f5f04c00700 1 -- 172.21.15.78:0/582199786 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5597cb6cdcf0 con 0x5597cb9995b0 2024-03-14T10:02:21.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.610+0000 7f5f04c00700 10 client.17190 handle_command_reply: tid=0 2024-03-14T10:02:21.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.610+0000 7f5f358c8b80 1 client.17190 shutdown 2024-03-14T10:02:21.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.610+0000 7f5f358c8b80 1 -- 172.21.15.78:0/582199786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92c680 msgr2=0x5597cb93c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:21.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.610+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/582199786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92c680 0x5597cb93c120 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c011e80 tx=0x7f5f0c011a40 comp rx=0 tx=0).stop 2024-03-14T10:02:21.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.610+0000 7f5f358c8b80 1 -- 172.21.15.78:0/582199786 shutdown_connections 2024-03-14T10:02:21.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.610+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/582199786 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5597cb9995b0 0x5597cb99b9d0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/582199786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5597cb936ec0 0x5597cb93cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/582199786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5597cb92c680 0x5597cb93c120 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f358c8b80 1 --2- 172.21.15.78:0/582199786 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5597cb92cfa0 0x5597cb93c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f04c00700 0 client.17190 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:21.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f358c8b80 1 -- 172.21.15.78:0/582199786 >> 172.21.15.78:0/582199786 conn(0x5597cb8a2220 msgr2=0x5597cb927510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:21.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f358c8b80 1 -- 172.21.15.78:0/582199786 shutdown_connections 2024-03-14T10:02:21.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f358c8b80 1 -- 172.21.15.78:0/582199786 wait complete. 2024-03-14T10:02:21.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f358c8b80 20 client.17190 trim_cache size 0 max 16384 2024-03-14T10:02:21.616 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:02:21.616 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T10:02:21.616 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T10:02:21.616 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:02:21.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f21e00700 1 -- 172.21.15.78:0/610819426 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5f000a0c00 msgr2=0x7f5f000a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:21.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f21e00700 1 --2- 172.21.15.78:0/610819426 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5f000a0c00 0x7f5f000a30c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5f14009410 tx=0x7f5f14009920 comp rx=0 tx=0).stop 2024-03-14T10:02:21.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f21e00700 1 -- 172.21.15.78:0/610819426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f1c106050 msgr2=0x7f5f1c0735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:21.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f21e00700 1 --2- 172.21.15.78:0/610819426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f1c106050 0x7f5f1c0735f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5f08000c00 tx=0x7f5f08009510 comp rx=0 tx=0).stop 2024-03-14T10:02:21.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f21e00700 1 -- 172.21.15.78:0/610819426 shutdown_connections 2024-03-14T10:02:21.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f21e00700 1 --2- 172.21.15.78:0/610819426 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5f000a0c00 0x7f5f000a30c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f21e00700 1 --2- 172.21.15.78:0/610819426 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5f1c100f30 0x7f5f1c0730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f21e00700 1 --2- 172.21.15.78:0/610819426 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5f1c073b30 0x7f5f1c1a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f21e00700 1 --2- 172.21.15.78:0/610819426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5f1c106050 0x7f5f1c0735f0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.611+0000 7f5f21e00700 1 -- 172.21.15.78:0/610819426 >> 172.21.15.78:0/610819426 conn(0x7f5f1c0fc7b0 msgr2=0x7f5f1c0fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:21.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.612+0000 7f5f21e00700 1 -- 172.21.15.78:0/610819426 shutdown_connections 2024-03-14T10:02:21.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.612+0000 7f5f21e00700 1 -- 172.21.15.78:0/610819426 wait complete. 2024-03-14T10:02:21.629 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T10:02:21.629 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T10:02:21.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.829+0000 7ff6b6000700 1 -- 172.21.15.78:0/4116933465 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6b0100e20 msgr2=0x7ff6b0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:21.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.830+0000 7ff6b6000700 1 --2- 172.21.15.78:0/4116933465 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6b0100e20 0x7ff6b0101200 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00098f0 tx=0x7ff6a0009c00 comp rx=0 tx=0).stop 2024-03-14T10:02:21.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.830+0000 7ff6b6000700 1 -- 172.21.15.78:0/4116933465 shutdown_connections 2024-03-14T10:02:21.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.830+0000 7ff6b6000700 1 --2- 172.21.15.78:0/4116933465 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6b0100e20 0x7ff6b0101200 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.830+0000 7ff6b6000700 1 --2- 172.21.15.78:0/4116933465 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6b0105f40 0x7ff6b010dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.830+0000 7ff6b6000700 1 --2- 172.21.15.78:0/4116933465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6b01017d0 0x7ff6b0105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.830+0000 7ff6b6000700 1 -- 172.21.15.78:0/4116933465 >> 172.21.15.78:0/4116933465 conn(0x7ff6b00fc700 msgr2=0x7ff6b00feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:21.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.830+0000 7ff6b6000700 1 -- 172.21.15.78:0/4116933465 shutdown_connections 2024-03-14T10:02:21.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.830+0000 7ff6b6000700 1 -- 172.21.15.78:0/4116933465 wait complete. 2024-03-14T10:02:21.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.831+0000 7ff6b6000700 1 Processor -- start 2024-03-14T10:02:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.831+0000 7ff6b6000700 1 -- start start 2024-03-14T10:02:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.831+0000 7ff6b6000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6b0100e20 0x7ff6b00730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.831+0000 7ff6b6000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6b0105f40 0x7ff6b00735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.831+0000 7ff6b6000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6b0073b30 0x7ff6b01a5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.831+0000 7ff6b6000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01a5d40 con 0x7ff6b0105f40 2024-03-14T10:02:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.831+0000 7ff6b6000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01a5eb0 con 0x7ff6b0073b30 2024-03-14T10:02:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.831+0000 7ff6b6000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01a6070 con 0x7ff6b0100e20 2024-03-14T10:02:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.832+0000 7ff6aea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6b0105f40 0x7ff6b00735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.832+0000 7ff6af400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6b0100e20 0x7ff6b00730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.832+0000 7ff6aea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6b0105f40 0x7ff6b00735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38552/0 (socket says 172.21.15.78:38552) 2024-03-14T10:02:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.832+0000 7ff6afe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6b0073b30 0x7ff6b01a5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.832+0000 7ff6aea00700 1 -- 172.21.15.78:0/4220194081 learned_addr learned my addr 172.21.15.78:0/4220194081 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.832+0000 7ff6af400700 1 -- 172.21.15.78:0/4220194081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6b0073b30 msgr2=0x7ff6b01a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:21.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.832+0000 7ff6af400700 1 --2- 172.21.15.78:0/4220194081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6b0073b30 0x7ff6b01a5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.832+0000 7ff6af400700 1 -- 172.21.15.78:0/4220194081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6b0105f40 msgr2=0x7ff6b00735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:21.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.832+0000 7ff6af400700 1 --2- 172.21.15.78:0/4220194081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6b0105f40 0x7ff6b00735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:21.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.832+0000 7ff6af400700 1 -- 172.21.15.78:0/4220194081 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a8009c40 con 0x7ff6b0100e20 2024-03-14T10:02:21.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.832+0000 7ff6afe00700 1 --2- 172.21.15.78:0/4220194081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6b0073b30 0x7ff6b01a5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:21.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.832+0000 7ff6af400700 1 --2- 172.21.15.78:0/4220194081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6b0100e20 0x7ff6b00730b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0009fd0 tx=0x7ff6a000bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:21.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.833+0000 7ff6a7e00700 1 -- 172.21.15.78:0/4220194081 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a001c070 con 0x7ff6b0100e20 2024-03-14T10:02:21.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.833+0000 7ff6b6000700 1 -- 172.21.15.78:0/4220194081 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a0009520 con 0x7ff6b0100e20 2024-03-14T10:02:21.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.833+0000 7ff6a7e00700 1 -- 172.21.15.78:0/4220194081 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6a000bba0 con 0x7ff6b0100e20 2024-03-14T10:02:21.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.833+0000 7ff6a7e00700 1 -- 172.21.15.78:0/4220194081 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0003900 con 0x7ff6b0100e20 2024-03-14T10:02:21.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.833+0000 7ff6b6000700 1 -- 172.21.15.78:0/4220194081 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b01a65e0 con 0x7ff6b0100e20 2024-03-14T10:02:21.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.834+0000 7ff6b6000700 1 -- 172.21.15.78:0/4220194081 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b0109160 con 0x7ff6b0100e20 2024-03-14T10:02:21.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.836+0000 7ff6a7e00700 1 -- 172.21.15.78:0/4220194081 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7ff6a0020030 con 0x7ff6b0100e20 2024-03-14T10:02:21.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.838+0000 7ff6a7e00700 1 --2- 172.21.15.78:0/4220194081 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff6940a0c00 0x7ff6940a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:21.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.838+0000 7ff6a7e00700 1 -- 172.21.15.78:0/4220194081 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7ff6a00cd9c0 con 0x7ff6b0100e20 2024-03-14T10:02:21.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.838+0000 7ff6aea00700 1 --2- 172.21.15.78:0/4220194081 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff6940a0c00 0x7ff6940a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:21.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.838+0000 7ff6aea00700 1 --2- 172.21.15.78:0/4220194081 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff6940a0c00 0x7ff6940a30c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff6b00747e0 tx=0x7ff69c009310 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:21.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:21.840+0000 7ff6a7e00700 1 -- 172.21.15.78:0/4220194081 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff6a0096d10 con 0x7ff6b0100e20 2024-03-14T10:02:22.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.093+0000 7ff6b6000700 1 -- 172.21.15.78:0/4220194081 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6b01022f0 con 0x7ff6b0100e20 2024-03-14T10:02:22.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.094+0000 7ff6a7e00700 1 -- 172.21.15.78:0/4220194081 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7ff6a009a330 con 0x7ff6b0100e20 2024-03-14T10:02:22.094 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T10:02:22.094 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T10:02:22.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.096+0000 7ff6b6000700 1 -- 172.21.15.78:0/4220194081 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff6940a0c00 msgr2=0x7ff6940a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.096+0000 7ff6b6000700 1 --2- 172.21.15.78:0/4220194081 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff6940a0c00 0x7ff6940a30c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff6b00747e0 tx=0x7ff69c009310 comp rx=0 tx=0).stop 2024-03-14T10:02:22.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.096+0000 7ff6b6000700 1 -- 172.21.15.78:0/4220194081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6b0100e20 msgr2=0x7ff6b00730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.096+0000 7ff6b6000700 1 --2- 172.21.15.78:0/4220194081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6b0100e20 0x7ff6b00730b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0009fd0 tx=0x7ff6a000bef0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.096+0000 7ff6b6000700 1 -- 172.21.15.78:0/4220194081 shutdown_connections 2024-03-14T10:02:22.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.096+0000 7ff6b6000700 1 --2- 172.21.15.78:0/4220194081 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff6940a0c00 0x7ff6940a30c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.096+0000 7ff6b6000700 1 --2- 172.21.15.78:0/4220194081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff6b0073b30 0x7ff6b01a5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.096+0000 7ff6b6000700 1 --2- 172.21.15.78:0/4220194081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff6b0100e20 0x7ff6b00730b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.096+0000 7ff6b6000700 1 --2- 172.21.15.78:0/4220194081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6b0105f40 0x7ff6b00735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.096+0000 7ff6b6000700 1 -- 172.21.15.78:0/4220194081 >> 172.21.15.78:0/4220194081 conn(0x7ff6b00fc700 msgr2=0x7ff6b00fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.096+0000 7ff6b6000700 1 -- 172.21.15.78:0/4220194081 shutdown_connections 2024-03-14T10:02:22.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.096+0000 7ff6b6000700 1 -- 172.21.15.78:0/4220194081 wait complete. 2024-03-14T10:02:22.097 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T10:02:22.115 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T10:02:22.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.311+0000 7f279b000700 1 -- 172.21.15.78:0/3546920318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2794100f20 msgr2=0x7f2794101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.311+0000 7f279b000700 1 --2- 172.21.15.78:0/3546920318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2794100f20 0x7f2794101300 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f27840098f0 tx=0x7f2784009c00 comp rx=0 tx=0).stop 2024-03-14T10:02:22.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.311+0000 7f279b000700 1 -- 172.21.15.78:0/3546920318 shutdown_connections 2024-03-14T10:02:22.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.311+0000 7f279b000700 1 --2- 172.21.15.78:0/3546920318 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2794106040 0x7f279410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.311+0000 7f279b000700 1 --2- 172.21.15.78:0/3546920318 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2794100f20 0x7f2794101300 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.311+0000 7f279b000700 1 --2- 172.21.15.78:0/3546920318 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27941018d0 0x7f2794105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.311+0000 7f279b000700 1 -- 172.21.15.78:0/3546920318 >> 172.21.15.78:0/3546920318 conn(0x7f27940fc7a0 msgr2=0x7f27940febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.311+0000 7f279b000700 1 -- 172.21.15.78:0/3546920318 shutdown_connections 2024-03-14T10:02:22.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.311+0000 7f279b000700 1 -- 172.21.15.78:0/3546920318 wait complete. 2024-03-14T10:02:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.312+0000 7f279b000700 1 Processor -- start 2024-03-14T10:02:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.312+0000 7f279b000700 1 -- start start 2024-03-14T10:02:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.313+0000 7f279b000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2794100f20 0x7f27940730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.313+0000 7f279b000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2794106040 0x7f27940735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.313+0000 7f279b000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2794073b30 0x7f27941a5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.313+0000 7f279b000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27941a5ed0 con 0x7f2794073b30 2024-03-14T10:02:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.313+0000 7f279b000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27941a6040 con 0x7f2794100f20 2024-03-14T10:02:22.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.313+0000 7f279b000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27941a6200 con 0x7f2794106040 2024-03-14T10:02:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.313+0000 7f2799000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2794073b30 0x7f27941a5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.313+0000 7f2793400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2794106040 0x7f27940735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.313+0000 7f2799000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2794073b30 0x7f27941a5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38562/0 (socket says 172.21.15.78:38562) 2024-03-14T10:02:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.313+0000 7f2799000700 1 -- 172.21.15.78:0/3564991271 learned_addr learned my addr 172.21.15.78:0/3564991271 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.313+0000 7f2793e00700 1 --2- 172.21.15.78:0/3564991271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2794100f20 0x7f27940730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.313+0000 7f2799000700 1 -- 172.21.15.78:0/3564991271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2794106040 msgr2=0x7f27940735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.313+0000 7f2799000700 1 --2- 172.21.15.78:0/3564991271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2794106040 0x7f27940735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.313+0000 7f2799000700 1 -- 172.21.15.78:0/3564991271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2794100f20 msgr2=0x7f27940730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.313+0000 7f2799000700 1 --2- 172.21.15.78:0/3564991271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2794100f20 0x7f27940730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.313+0000 7f2799000700 1 -- 172.21.15.78:0/3564991271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27800095f0 con 0x7f2794073b30 2024-03-14T10:02:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.314+0000 7f2799000700 1 --2- 172.21.15.78:0/3564991271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2794073b30 0x7f27941a5890 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f278c00ef30 tx=0x7f278c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.314+0000 7f2790c00700 1 -- 172.21.15.78:0/3564991271 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278c018070 con 0x7f2794073b30 2024-03-14T10:02:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.314+0000 7f2790c00700 1 -- 172.21.15.78:0/3564991271 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f278c00f040 con 0x7f2794073b30 2024-03-14T10:02:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.314+0000 7f279b000700 1 -- 172.21.15.78:0/3564991271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2784009520 con 0x7f2794073b30 2024-03-14T10:02:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.314+0000 7f2790c00700 1 -- 172.21.15.78:0/3564991271 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278c00cd00 con 0x7f2794073b30 2024-03-14T10:02:22.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.314+0000 7f279b000700 1 -- 172.21.15.78:0/3564991271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27941a67b0 con 0x7f2794073b30 2024-03-14T10:02:22.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.317+0000 7f2790c00700 1 -- 172.21.15.78:0/3564991271 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f278c003a00 con 0x7f2794073b30 2024-03-14T10:02:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.318+0000 7f27aeaa2b80 1 Processor -- start 2024-03-14T10:02:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.318+0000 7f27aeaa2b80 1 -- start start 2024-03-14T10:02:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.318+0000 7f2790c00700 1 --2- 172.21.15.78:0/3564991271 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f27780a0c00 0x7f27780a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.318+0000 7f27aeaa2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316680 0x55d0f2316a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.318+0000 7f27aeaa2b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2316fa0 0x55d0f2320980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f2790c00700 1 -- 172.21.15.78:0/3564991271 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f278c0cc320 con 0x7f2794073b30 2024-03-14T10:02:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f27aeaa2b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2320ec0 0x55d0f23232b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f27aeaa2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0f226f990 con 0x55d0f2316680 2024-03-14T10:02:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f27aeaa2b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0f226d350 con 0x55d0f2316fa0 2024-03-14T10:02:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f27aeaa2b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0f2268c40 con 0x55d0f2320ec0 2024-03-14T10:02:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f2793400700 1 --2- 172.21.15.78:0/3564991271 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f27780a0c00 0x7f27780a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f2793e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316680 0x55d0f2316a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f2793e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316680 0x55d0f2316a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38568/0 (socket says 172.21.15.78:38568) 2024-03-14T10:02:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f2793e00700 1 -- 172.21.15.78:0/2949121549 learned_addr learned my addr 172.21.15.78:0/2949121549 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f2793400700 1 --2- 172.21.15.78:0/2949121549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2316fa0 0x55d0f2320980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f2799000700 1 --2- 172.21.15.78:0/2949121549 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2320ec0 0x55d0f23232b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f2793e00700 1 -- 172.21.15.78:0/2949121549 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2320ec0 msgr2=0x55d0f23232b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f2793e00700 1 --2- 172.21.15.78:0/2949121549 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2320ec0 0x55d0f23232b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f2793e00700 1 -- 172.21.15.78:0/2949121549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2316fa0 msgr2=0x55d0f2320980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f2793e00700 1 --2- 172.21.15.78:0/2949121549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2316fa0 0x55d0f2320980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f2793e00700 1 -- 172.21.15.78:0/2949121549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0f20ea3b0 con 0x55d0f2316680 2024-03-14T10:02:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.319+0000 7f2793400700 1 --2- 172.21.15.78:0/2949121549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2316fa0 0x55d0f2320980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:02:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.320+0000 7f2793400700 1 --2- 172.21.15.78:0/3564991271 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f27780a0c00 0x7f27780a30c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2780009fd0 tx=0x7f2780013a60 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.320+0000 7f2793e00700 1 --2- 172.21.15.78:0/2949121549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316680 0x55d0f2316a60 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2784009230 tx=0x7f2784005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f9e09a662c6cd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.320+0000 7f2789600700 1 -- 172.21.15.78:0/2949121549 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278400fd70 con 0x55d0f2316680 2024-03-14T10:02:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.320+0000 7f2789600700 1 -- 172.21.15.78:0/2949121549 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f278401d040 con 0x55d0f2316680 2024-03-14T10:02:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.320+0000 7f2789600700 1 -- 172.21.15.78:0/2949121549 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784005040 con 0x55d0f2316680 2024-03-14T10:02:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.321+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/2949121549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316680 msgr2=0x55d0f2316a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.321+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/2949121549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316680 0x55d0f2316a60 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2784009230 tx=0x7f2784005dc0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.321+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/2949121549 shutdown_connections 2024-03-14T10:02:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.321+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/2949121549 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2320ec0 0x55d0f23232b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.321+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/2949121549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316680 0x55d0f2316a60 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.321+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/2949121549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2316fa0 0x55d0f2320980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.321+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/2949121549 >> 172.21.15.78:0/2949121549 conn(0x55d0f228c220 msgr2=0x55d0f2311aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.321+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/2949121549 shutdown_connections 2024-03-14T10:02:22.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.321+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/2949121549 wait complete. 2024-03-14T10:02:22.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.321+0000 7f27aeaa2b80 1 -- start start 2024-03-14T10:02:22.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.321+0000 7f27aeaa2b80 1 Processor -- start 2024-03-14T10:02:22.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.322+0000 7f27aeaa2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316680 0x55d0f23208e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.322+0000 7f27aeaa2b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2316fa0 0x55d0f231db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.322+0000 7f27aeaa2b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2320ec0 0x55d0f231e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.322+0000 7f27aeaa2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0f2204d80 con 0x55d0f2316680 2024-03-14T10:02:22.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.322+0000 7f27aeaa2b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0f2204a20 con 0x55d0f2320ec0 2024-03-14T10:02:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.322+0000 7f27aeaa2b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0f20ec6a0 con 0x55d0f2316fa0 2024-03-14T10:02:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.322+0000 7f27aeaa2b80 10 client.0 _ll_register_callbacks cb 0x55d0f20703a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.322+0000 7f2793e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316680 0x55d0f23208e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.322+0000 7f2793e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316680 0x55d0f23208e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38570/0 (socket says 172.21.15.78:38570) 2024-03-14T10:02:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.322+0000 7f2793e00700 1 -- 172.21.15.78:0/2056525920 learned_addr learned my addr 172.21.15.78:0/2056525920 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.322+0000 7f2793400700 1 --2- 172.21.15.78:0/2056525920 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2316fa0 0x55d0f231db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.322+0000 7f2799000700 1 --2- 172.21.15.78:0/2056525920 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2320ec0 0x55d0f231e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.322+0000 7f2793e00700 1 -- 172.21.15.78:0/2056525920 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2316fa0 msgr2=0x55d0f231db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.323+0000 7f2793e00700 1 --2- 172.21.15.78:0/2056525920 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2316fa0 0x55d0f231db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.323+0000 7f2793e00700 1 -- 172.21.15.78:0/2056525920 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2320ec0 msgr2=0x55d0f231e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.323+0000 7f2793e00700 1 --2- 172.21.15.78:0/2056525920 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2320ec0 0x55d0f231e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.323+0000 7f2793e00700 1 -- 172.21.15.78:0/2056525920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279410e430 con 0x55d0f2316680 2024-03-14T10:02:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.323+0000 7f2793e00700 1 --2- 172.21.15.78:0/2056525920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316680 0x55d0f23208e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2784009810 tx=0x7f278400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb714f66aa667cd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.323+0000 7f277e000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.323+0000 7f277e000700 1 -- 172.21.15.78:0/2056525920 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278400f460 con 0x55d0f2316680 2024-03-14T10:02:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.323+0000 7f277e000700 1 -- 172.21.15.78:0/2056525920 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f278400f5c0 con 0x55d0f2316680 2024-03-14T10:02:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.323+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/2056525920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d0f2309720 con 0x55d0f2316680 2024-03-14T10:02:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.323+0000 7f277e000700 1 -- 172.21.15.78:0/2056525920 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f278400f7a0 con 0x55d0f2316680 2024-03-14T10:02:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.323+0000 7f27aeaa2b80 10 client.17208 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.323+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/2056525920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d0f2096b60 con 0x55d0f2316680 2024-03-14T10:02:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.324+0000 7f277e000700 1 -- 172.21.15.78:0/2056525920 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d0f2309720 con 0x55d0f2316680 2024-03-14T10:02:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.324+0000 7f277e000700 1 -- 172.21.15.78:0/2056525920 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2784015020 con 0x55d0f2316680 2024-03-14T10:02:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.324+0000 7f27aeaa2b80 10 client.17208 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.324+0000 7f27aeaa2b80 10 client.17208 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:02:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.324+0000 7f27aeaa2b80 20 client.17208 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.324+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/2056525920 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d0f2330620 0x55d0f2332a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.324+0000 7f27aeaa2b80 4 client.17208 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:02:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.324+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/2056525920 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d0f2261700 con 0x55d0f2330620 2024-03-14T10:02:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.324+0000 7f2793400700 1 --2- 172.21.15.78:0/2056525920 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d0f2330620 0x55d0f2332a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.325+0000 7f277e000700 10 client.17208 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:02:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.325+0000 7f2793400700 1 --2- 172.21.15.78:0/2056525920 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d0f2330620 0x55d0f2332a40 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=50d39c602631ad41 server_cookie=452427b57cc6a87c in_seq=0 out_seq=0 2024-03-14T10:02:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.326+0000 7f277e000700 1 -- 172.21.15.78:0/2056525920 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55d0f2261700 con 0x55d0f2330620 2024-03-14T10:02:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.326+0000 7f277e000700 10 client.17208 handle_command_reply: tid=0 2024-03-14T10:02:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.326+0000 7f27aeaa2b80 1 client.17208 shutdown 2024-03-14T10:02:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.326+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/2056525920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316680 msgr2=0x55d0f23208e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.326+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/2056525920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316680 0x55d0f23208e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2784009810 tx=0x7f278400bf90 comp rx=0 tx=0).stop 2024-03-14T10:02:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.326+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/2056525920 shutdown_connections 2024-03-14T10:02:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.326+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/2056525920 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d0f2330620 0x55d0f2332a40 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.326+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/2056525920 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2320ec0 0x55d0f231e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.326+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/2056525920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316680 0x55d0f23208e0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.326+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/2056525920 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2316fa0 0x55d0f231db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.326+0000 7f277e000700 0 client.17208 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:02:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.326+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/2056525920 >> 172.21.15.78:0/2056525920 conn(0x55d0f228c220 msgr2=0x55d0f2311690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.326+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/2056525920 shutdown_connections 2024-03-14T10:02:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.326+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/2056525920 wait complete. 2024-03-14T10:02:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.326+0000 7f27aeaa2b80 20 client.17208 trim_cache size 0 max 16384 2024-03-14T10:02:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.339+0000 7f27aeaa2b80 1 Processor -- start 2024-03-14T10:02:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.339+0000 7f27aeaa2b80 1 -- start start 2024-03-14T10:02:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f27aeaa2b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2316680 0x55d0f2337360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f27aeaa2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316fa0 0x55d0f23232b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f27aeaa2b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2320ec0 0x55d0f2320410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f27aeaa2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0f230bd00 con 0x55d0f2316fa0 2024-03-14T10:02:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f27aeaa2b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0f2318970 con 0x55d0f2316680 2024-03-14T10:02:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f27aeaa2b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0f231c0b0 con 0x55d0f2320ec0 2024-03-14T10:02:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f2793400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316fa0 0x55d0f23232b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f2793400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316fa0 0x55d0f23232b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38586/0 (socket says 172.21.15.78:38586) 2024-03-14T10:02:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f2793400700 1 -- 172.21.15.78:0/203581179 learned_addr learned my addr 172.21.15.78:0/203581179 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f2793e00700 1 --2- 172.21.15.78:0/203581179 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2316680 0x55d0f2337360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f2799000700 1 --2- 172.21.15.78:0/203581179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2320ec0 0x55d0f2320410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f2793400700 1 -- 172.21.15.78:0/203581179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2320ec0 msgr2=0x55d0f2320410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f2793400700 1 --2- 172.21.15.78:0/203581179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2320ec0 0x55d0f2320410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f2793400700 1 -- 172.21.15.78:0/203581179 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2316680 msgr2=0x55d0f2337360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f2793400700 1 --2- 172.21.15.78:0/203581179 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2316680 0x55d0f2337360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f2793400700 1 -- 172.21.15.78:0/203581179 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0f222eea0 con 0x55d0f2316fa0 2024-03-14T10:02:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.340+0000 7f2799000700 1 --2- 172.21.15.78:0/203581179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2320ec0 0x55d0f2320410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:02:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.341+0000 7f2793400700 1 --2- 172.21.15.78:0/203581179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316fa0 0x55d0f23232b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2794074b60 tx=0x7f2780009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc7c3ed130127da7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.341+0000 7f278a000700 1 -- 172.21.15.78:0/203581179 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2780018050 con 0x55d0f2316fa0 2024-03-14T10:02:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.341+0000 7f278a000700 1 -- 172.21.15.78:0/203581179 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2780026240 con 0x55d0f2316fa0 2024-03-14T10:02:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.341+0000 7f278a000700 1 -- 172.21.15.78:0/203581179 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2780025040 con 0x55d0f2316fa0 2024-03-14T10:02:22.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.341+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/203581179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316fa0 msgr2=0x55d0f23232b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.342+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/203581179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316fa0 0x55d0f23232b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2794074b60 tx=0x7f2780009310 comp rx=0 tx=0).stop 2024-03-14T10:02:22.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.342+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/203581179 shutdown_connections 2024-03-14T10:02:22.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.342+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/203581179 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2316680 0x55d0f2337360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.342+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/203581179 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2320ec0 0x55d0f2320410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.342+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/203581179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316fa0 0x55d0f23232b0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.342+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/203581179 >> 172.21.15.78:0/203581179 conn(0x55d0f228c220 msgr2=0x55d0f2310180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.342+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/203581179 shutdown_connections 2024-03-14T10:02:22.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.342+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/203581179 wait complete. 2024-03-14T10:02:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.342+0000 7f27aeaa2b80 1 -- start start 2024-03-14T10:02:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.342+0000 7f27aeaa2b80 1 Processor -- start 2024-03-14T10:02:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f27aeaa2b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2316680 0x55d0f2325980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f27aeaa2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316fa0 0x55d0f2325ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f27aeaa2b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2320ec0 0x55d0f2381100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f27aeaa2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0f2381640 con 0x55d0f2316fa0 2024-03-14T10:02:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f27aeaa2b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0f23817b0 con 0x55d0f2320ec0 2024-03-14T10:02:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f27aeaa2b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0f2381970 con 0x55d0f2316680 2024-03-14T10:02:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f27aeaa2b80 10 client.0 _ll_register_callbacks cb 0x7f278c009370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f2793400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316fa0 0x55d0f2325ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f2793400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316fa0 0x55d0f2325ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38594/0 (socket says 172.21.15.78:38594) 2024-03-14T10:02:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f2793e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2316680 0x55d0f2325980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f2799000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2320ec0 0x55d0f2381100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f2793400700 1 -- 172.21.15.78:0/1230751063 learned_addr learned my addr 172.21.15.78:0/1230751063 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f2793400700 1 -- 172.21.15.78:0/1230751063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2316680 msgr2=0x55d0f2325980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f2793400700 1 --2- 172.21.15.78:0/1230751063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2316680 0x55d0f2325980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f2793400700 1 -- 172.21.15.78:0/1230751063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2320ec0 msgr2=0x55d0f2381100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f2793400700 1 --2- 172.21.15.78:0/1230751063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2320ec0 0x55d0f2381100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f2793400700 1 -- 172.21.15.78:0/1230751063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2780004030 con 0x55d0f2316fa0 2024-03-14T10:02:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f2799000700 1 --2- 172.21.15.78:0/1230751063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2320ec0 0x55d0f2381100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.343+0000 7f2793e00700 1 --2- 172.21.15.78:0/1230751063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2316680 0x55d0f2325980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.344+0000 7f2793400700 1 --2- 172.21.15.78:0/1230751063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316fa0 0x55d0f2325ec0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2780005b40 tx=0x7f2780005560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec93c6d28178f527 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.344+0000 7f277e000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.344+0000 7f277e000700 1 -- 172.21.15.78:0/1230751063 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2780025040 con 0x55d0f2316fa0 2024-03-14T10:02:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.344+0000 7f277e000700 1 -- 172.21.15.78:0/1230751063 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f2780003a60 con 0x55d0f2316fa0 2024-03-14T10:02:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.344+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/1230751063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f278401d040 con 0x55d0f2316fa0 2024-03-14T10:02:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.344+0000 7f277e000700 1 -- 172.21.15.78:0/1230751063 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2780018030 con 0x55d0f2316fa0 2024-03-14T10:02:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.344+0000 7f27aeaa2b80 10 client.17214 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.344+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/1230751063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d0f20eecd0 con 0x55d0f2316fa0 2024-03-14T10:02:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.344+0000 7f277e000700 1 -- 172.21.15.78:0/1230751063 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2780040020 con 0x55d0f2316fa0 2024-03-14T10:02:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.345+0000 7f277e000700 1 -- 172.21.15.78:0/1230751063 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f278000f720 con 0x55d0f2316fa0 2024-03-14T10:02:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.345+0000 7f27aeaa2b80 10 client.17214 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.345+0000 7f27aeaa2b80 10 client.17214 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:02:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.345+0000 7f27aeaa2b80 20 client.17214 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.345+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/1230751063 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d0f23831b0 0x55d0f23855d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.345+0000 7f27aeaa2b80 4 client.17214 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:02:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.345+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/1230751063 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d0f20b7cf0 con 0x55d0f23831b0 2024-03-14T10:02:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.345+0000 7f2793e00700 1 --2- 172.21.15.78:0/1230751063 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d0f23831b0 0x55d0f23855d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f2793e00700 1 --2- 172.21.15.78:0/1230751063 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d0f23831b0 0x55d0f23855d0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8ab0580ad4092c56 server_cookie=71b5e7bc742c2edf in_seq=0 out_seq=0 2024-03-14T10:02:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f277e000700 10 client.17214 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:02:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f277e000700 1 -- 172.21.15.78:0/1230751063 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d0f20b7cf0 con 0x55d0f23831b0 2024-03-14T10:02:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f277e000700 10 client.17214 handle_command_reply: tid=0 2024-03-14T10:02:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f27aeaa2b80 1 client.17214 shutdown 2024-03-14T10:02:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/1230751063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316fa0 msgr2=0x55d0f2325ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/1230751063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316fa0 0x55d0f2325ec0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2780005b40 tx=0x7f2780005560 comp rx=0 tx=0).stop 2024-03-14T10:02:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/1230751063 shutdown_connections 2024-03-14T10:02:22.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/1230751063 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x55d0f23831b0 0x55d0f23855d0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/1230751063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0f2320ec0 0x55d0f2381100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/1230751063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0f2316680 0x55d0f2325980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f277e000700 0 client.17214 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:02:22.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f27aeaa2b80 1 --2- 172.21.15.78:0/1230751063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0f2316fa0 0x55d0f2325ec0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/1230751063 >> 172.21.15.78:0/1230751063 conn(0x55d0f228c220 msgr2=0x55d0f231d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/1230751063 shutdown_connections 2024-03-14T10:02:22.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f27aeaa2b80 1 -- 172.21.15.78:0/1230751063 wait complete. 2024-03-14T10:02:22.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.346+0000 7f27aeaa2b80 20 client.17214 trim_cache size 0 max 16384 2024-03-14T10:02:22.350 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:02:22.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.347+0000 7f279b000700 1 -- 172.21.15.78:0/3564991271 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f27780a0c00 msgr2=0x7f27780a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.347+0000 7f279b000700 1 --2- 172.21.15.78:0/3564991271 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f27780a0c00 0x7f27780a30c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2780009fd0 tx=0x7f2780013a60 comp rx=0 tx=0).stop 2024-03-14T10:02:22.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.347+0000 7f279b000700 1 -- 172.21.15.78:0/3564991271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2794073b30 msgr2=0x7f27941a5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.347+0000 7f279b000700 1 --2- 172.21.15.78:0/3564991271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2794073b30 0x7f27941a5890 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f278c00ef30 tx=0x7f278c00c3a0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.347+0000 7f279b000700 1 -- 172.21.15.78:0/3564991271 shutdown_connections 2024-03-14T10:02:22.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.347+0000 7f279b000700 1 --2- 172.21.15.78:0/3564991271 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f27780a0c00 0x7f27780a30c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.347+0000 7f279b000700 1 --2- 172.21.15.78:0/3564991271 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2794100f20 0x7f27940730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.347+0000 7f279b000700 1 --2- 172.21.15.78:0/3564991271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2794073b30 0x7f27941a5890 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.347+0000 7f279b000700 1 --2- 172.21.15.78:0/3564991271 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2794106040 0x7f27940735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.347+0000 7f279b000700 1 -- 172.21.15.78:0/3564991271 >> 172.21.15.78:0/3564991271 conn(0x7f27940fc7a0 msgr2=0x7f27940fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.347+0000 7f279b000700 1 -- 172.21.15.78:0/3564991271 shutdown_connections 2024-03-14T10:02:22.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.347+0000 7f279b000700 1 -- 172.21.15.78:0/3564991271 wait complete. 2024-03-14T10:02:22.362 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:22 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/4220194081' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:02:22.365 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T10:02:22.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:22 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/4220194081' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:02:22.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:22 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/4220194081' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:02:22.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.559+0000 7f1b7fa00700 1 -- 172.21.15.78:0/4055675723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b78100f20 msgr2=0x7f1b78101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.559+0000 7f1b7fa00700 1 --2- 172.21.15.78:0/4055675723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b78100f20 0x7f1b78101300 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c009990 tx=0x7f1b6c009ca0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.559+0000 7f1b7fa00700 1 -- 172.21.15.78:0/4055675723 shutdown_connections 2024-03-14T10:02:22.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.559+0000 7f1b7fa00700 1 --2- 172.21.15.78:0/4055675723 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b78106040 0x7f1b7810dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.559+0000 7f1b7fa00700 1 --2- 172.21.15.78:0/4055675723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b78100f20 0x7f1b78101300 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.559+0000 7f1b7fa00700 1 --2- 172.21.15.78:0/4055675723 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b781018d0 0x7f1b78105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.559+0000 7f1b7fa00700 1 -- 172.21.15.78:0/4055675723 >> 172.21.15.78:0/4055675723 conn(0x7f1b780fc7a0 msgr2=0x7f1b780febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.559+0000 7f1b7fa00700 1 -- 172.21.15.78:0/4055675723 shutdown_connections 2024-03-14T10:02:22.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.559+0000 7f1b7fa00700 1 -- 172.21.15.78:0/4055675723 wait complete. 2024-03-14T10:02:22.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.560+0000 7f1b7fa00700 1 Processor -- start 2024-03-14T10:02:22.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.560+0000 7f1b7fa00700 1 -- start start 2024-03-14T10:02:22.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.560+0000 7f1b7fa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b78100f20 0x7f1b78197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.560+0000 7f1b7fa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b78106040 0x7f1b78197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.560+0000 7f1b7fa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b7819bf60 0x7f1b7819c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.560+0000 7f1b7fa00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7819c9e0 con 0x7f1b78106040 2024-03-14T10:02:22.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.560+0000 7f1b7fa00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7819cb50 con 0x7f1b7819bf60 2024-03-14T10:02:22.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.560+0000 7f1b7fa00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7819cd10 con 0x7f1b78100f20 2024-03-14T10:02:22.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.561+0000 7f1b7d000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b78100f20 0x7f1b78197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.561+0000 7f1b7d000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b78100f20 0x7f1b78197680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:42604/0 (socket says 172.21.15.78:42604) 2024-03-14T10:02:22.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.561+0000 7f1b7d000700 1 -- 172.21.15.78:0/835597075 learned_addr learned my addr 172.21.15.78:0/835597075 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:22.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.561+0000 7f1b77e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b78106040 0x7f1b78197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.561+0000 7f1b7da00700 1 --2- 172.21.15.78:0/835597075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b7819bf60 0x7f1b7819c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.561+0000 7f1b7d000700 1 -- 172.21.15.78:0/835597075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b7819bf60 msgr2=0x7f1b7819c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.561+0000 7f1b7d000700 1 --2- 172.21.15.78:0/835597075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b7819bf60 0x7f1b7819c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.561+0000 7f1b7d000700 1 -- 172.21.15.78:0/835597075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b78106040 msgr2=0x7f1b78197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.561+0000 7f1b7d000700 1 --2- 172.21.15.78:0/835597075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b78106040 0x7f1b78197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.561+0000 7f1b7d000700 1 -- 172.21.15.78:0/835597075 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6c0095f0 con 0x7f1b78100f20 2024-03-14T10:02:22.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.561+0000 7f1b7da00700 1 --2- 172.21.15.78:0/835597075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b7819bf60 0x7f1b7819c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:02:22.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.561+0000 7f1b7d000700 1 --2- 172.21.15.78:0/835597075 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b78100f20 0x7f1b78197680 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c005910 tx=0x7f1b6c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.562+0000 7f1b75600700 1 -- 172.21.15.78:0/835597075 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c01d070 con 0x7f1b78100f20 2024-03-14T10:02:22.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.562+0000 7f1b7fa00700 1 -- 172.21.15.78:0/835597075 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b781a4210 con 0x7f1b78100f20 2024-03-14T10:02:22.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.562+0000 7f1b75600700 1 -- 172.21.15.78:0/835597075 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b6c022470 con 0x7f1b78100f20 2024-03-14T10:02:22.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.562+0000 7f1b75600700 1 -- 172.21.15.78:0/835597075 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c017650 con 0x7f1b78100f20 2024-03-14T10:02:22.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.562+0000 7f1b7fa00700 1 -- 172.21.15.78:0/835597075 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b781a4690 con 0x7f1b78100f20 2024-03-14T10:02:22.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.565+0000 7f1b75600700 1 -- 172.21.15.78:0/835597075 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f1b6c0225e0 con 0x7f1b78100f20 2024-03-14T10:02:22.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.566+0000 7f1b93594b80 1 Processor -- start 2024-03-14T10:02:22.565 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.566+0000 7f1b93594b80 1 -- start start 2024-03-14T10:02:22.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.566+0000 7f1b93594b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff70a90 0x55e10ff70e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.566+0000 7f1b93594b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff713b0 0x55e10ff7ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.566+0000 7f1b93594b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff7b2d0 0x55e10ff7d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.566+0000 7f1b93594b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e10fec5990 con 0x55e10ff713b0 2024-03-14T10:02:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.566+0000 7f1b93594b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e10fec3350 con 0x55e10ff7b2d0 2024-03-14T10:02:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.566+0000 7f1b93594b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e10febec40 con 0x55e10ff70a90 2024-03-14T10:02:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.567+0000 7f1b7d000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff713b0 0x55e10ff7ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.567+0000 7f1b7da00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff70a90 0x55e10ff70e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.567+0000 7f1b7da00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff70a90 0x55e10ff70e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:42612/0 (socket says 172.21.15.78:42612) 2024-03-14T10:02:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.567+0000 7f1b7da00700 1 -- 172.21.15.78:0/3761015088 learned_addr learned my addr 172.21.15.78:0/3761015088 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.567+0000 7f1b77e00700 1 --2- 172.21.15.78:0/3761015088 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff7b2d0 0x55e10ff7d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.567+0000 7f1b75600700 1 --2- 172.21.15.78:0/835597075 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1b600a0c00 0x7f1b600a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.567+0000 7f1b75600700 1 -- 172.21.15.78:0/835597075 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f1b6c0cdb60 con 0x7f1b78100f20 2024-03-14T10:02:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.567+0000 7f1b7d000700 1 -- 172.21.15.78:0/3761015088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff70a90 msgr2=0x55e10ff70e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.567+0000 7f1b7d000700 1 --2- 172.21.15.78:0/3761015088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff70a90 0x55e10ff70e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.567+0000 7f1b7d000700 1 -- 172.21.15.78:0/3761015088 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff7b2d0 msgr2=0x55e10ff7d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.567+0000 7f1b7d000700 1 --2- 172.21.15.78:0/3761015088 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff7b2d0 0x55e10ff7d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.567+0000 7f1b7d000700 1 -- 172.21.15.78:0/3761015088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e10fd403b0 con 0x55e10ff713b0 2024-03-14T10:02:22.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.567+0000 7f1b7da00700 1 --2- 172.21.15.78:0/3761015088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff70a90 0x55e10ff70e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:22.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.567+0000 7f1b77e00700 1 --2- 172.21.15.78:0/3761015088 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff7b2d0 0x55e10ff7d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:22.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.567+0000 7f1b7da00700 1 --2- 172.21.15.78:0/835597075 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1b600a0c00 0x7f1b600a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.567+0000 7f1b7d000700 1 --2- 172.21.15.78:0/3761015088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff713b0 0x55e10ff7ad90 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0cae80 tx=0x7f1b6c009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfcaa3b6c2876be server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.568+0000 7f1b6a000700 1 -- 172.21.15.78:0/3761015088 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c09a650 con 0x55e10ff713b0 2024-03-14T10:02:22.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.568+0000 7f1b6a000700 1 -- 172.21.15.78:0/3761015088 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b6c09a470 con 0x55e10ff713b0 2024-03-14T10:02:22.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.568+0000 7f1b6a000700 1 -- 172.21.15.78:0/3761015088 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c09ad70 con 0x55e10ff713b0 2024-03-14T10:02:22.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.568+0000 7f1b93594b80 1 -- 172.21.15.78:0/3761015088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff713b0 msgr2=0x55e10ff7ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.568+0000 7f1b7da00700 1 --2- 172.21.15.78:0/835597075 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1b600a0c00 0x7f1b600a30c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1b70007840 tx=0x7f1b70009040 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.568+0000 7f1b93594b80 1 --2- 172.21.15.78:0/3761015088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff713b0 0x55e10ff7ad90 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0cae80 tx=0x7f1b6c009ca0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.568+0000 7f1b93594b80 1 -- 172.21.15.78:0/3761015088 shutdown_connections 2024-03-14T10:02:22.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.568+0000 7f1b93594b80 1 --2- 172.21.15.78:0/3761015088 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff7b2d0 0x55e10ff7d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.568+0000 7f1b93594b80 1 --2- 172.21.15.78:0/3761015088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff70a90 0x55e10ff70e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.568+0000 7f1b93594b80 1 --2- 172.21.15.78:0/3761015088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff713b0 0x55e10ff7ad90 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.568+0000 7f1b93594b80 1 -- 172.21.15.78:0/3761015088 >> 172.21.15.78:0/3761015088 conn(0x55e10fee2220 msgr2=0x55e10ff67aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.568+0000 7f1b93594b80 1 -- 172.21.15.78:0/3761015088 shutdown_connections 2024-03-14T10:02:22.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.569+0000 7f1b93594b80 1 -- 172.21.15.78:0/3761015088 wait complete. 2024-03-14T10:02:22.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.569+0000 7f1b93594b80 1 -- start start 2024-03-14T10:02:22.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.569+0000 7f1b93594b80 1 Processor -- start 2024-03-14T10:02:22.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.570+0000 7f1b93594b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff70a90 0x55e10ff7ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.570+0000 7f1b93594b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff713b0 0x55e10ff78090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.570+0000 7f1b93594b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff7b2d0 0x55e10ff78630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.570+0000 7f1b93594b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e10fe5ad80 con 0x55e10ff7b2d0 2024-03-14T10:02:22.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.570+0000 7f1b93594b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e10fe5aa20 con 0x55e10ff70a90 2024-03-14T10:02:22.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.570+0000 7f1b93594b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e10fd426a0 con 0x55e10ff713b0 2024-03-14T10:02:22.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.570+0000 7f1b93594b80 10 client.0 _ll_register_callbacks cb 0x55e10fcc63a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.570+0000 7f1b7da00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff7b2d0 0x55e10ff78630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.570+0000 7f1b7d000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff70a90 0x55e10ff7ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.570+0000 7f1b7da00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff7b2d0 0x55e10ff78630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38648/0 (socket says 172.21.15.78:38648) 2024-03-14T10:02:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.570+0000 7f1b77e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff713b0 0x55e10ff78090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.570+0000 7f1b7da00700 1 -- 172.21.15.78:0/2799843374 learned_addr learned my addr 172.21.15.78:0/2799843374 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.570+0000 7f1b7da00700 1 -- 172.21.15.78:0/2799843374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff713b0 msgr2=0x55e10ff78090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.570+0000 7f1b7da00700 1 --2- 172.21.15.78:0/2799843374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff713b0 0x55e10ff78090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.570+0000 7f1b7da00700 1 -- 172.21.15.78:0/2799843374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff70a90 msgr2=0x55e10ff7ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.570+0000 7f1b7da00700 1 --2- 172.21.15.78:0/2799843374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff70a90 0x55e10ff7ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.571+0000 7f1b7da00700 1 -- 172.21.15.78:0/2799843374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b640098e0 con 0x55e10ff7b2d0 2024-03-14T10:02:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.571+0000 7f1b77e00700 1 --2- 172.21.15.78:0/2799843374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff713b0 0x55e10ff78090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.571+0000 7f1b7da00700 1 --2- 172.21.15.78:0/2799843374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff7b2d0 0x55e10ff78630 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f1b70017e50 tx=0x7f1b70004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f191d7497556a1ac server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.571+0000 7f1b5ea00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.571+0000 7f1b5ea00700 1 -- 172.21.15.78:0/2799843374 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b70017e80 con 0x55e10ff7b2d0 2024-03-14T10:02:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.571+0000 7f1b5ea00700 1 -- 172.21.15.78:0/2799843374 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f1b70023450 con 0x55e10ff7b2d0 2024-03-14T10:02:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.571+0000 7f1b93594b80 1 -- 172.21.15.78:0/2799843374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e10ff5f720 con 0x55e10ff7b2d0 2024-03-14T10:02:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.571+0000 7f1b5ea00700 1 -- 172.21.15.78:0/2799843374 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b70009140 con 0x55e10ff7b2d0 2024-03-14T10:02:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.571+0000 7f1b93594b80 10 client.17226 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.571+0000 7f1b93594b80 1 -- 172.21.15.78:0/2799843374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f1b781a4690 con 0x55e10ff7b2d0 2024-03-14T10:02:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.572+0000 7f1b5ea00700 1 -- 172.21.15.78:0/2799843374 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e10ff5f720 con 0x55e10ff7b2d0 2024-03-14T10:02:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.572+0000 7f1b5ea00700 1 -- 172.21.15.78:0/2799843374 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1b7002a060 con 0x55e10ff7b2d0 2024-03-14T10:02:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.572+0000 7f1b93594b80 10 client.17226 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.572+0000 7f1b93594b80 10 client.17226 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:02:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.572+0000 7f1b93594b80 20 client.17226 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.572+0000 7f1b93594b80 1 --2- 172.21.15.78:0/2799843374 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55e10ff8ac00 0x55e10ff8d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.572+0000 7f1b93594b80 4 client.17226 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:02:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.572+0000 7f1b93594b80 1 -- 172.21.15.78:0/2799843374 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e10feb7700 con 0x55e10ff8ac00 2024-03-14T10:02:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.572+0000 7f1b7d000700 1 --2- 172.21.15.78:0/2799843374 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55e10ff8ac00 0x55e10ff8d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.573+0000 7f1b7d000700 1 --2- 172.21.15.78:0/2799843374 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55e10ff8ac00 0x55e10ff8d020 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e73d78b0cc8af3d9 server_cookie=724275b78974aafd in_seq=0 out_seq=0 2024-03-14T10:02:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.573+0000 7f1b5ea00700 10 client.17226 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:02:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.573+0000 7f1b5ea00700 1 -- 172.21.15.78:0/2799843374 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55e10feb7700 con 0x55e10ff8ac00 2024-03-14T10:02:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.573+0000 7f1b5ea00700 10 client.17226 handle_command_reply: tid=0 2024-03-14T10:02:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.573+0000 7f1b93594b80 1 client.17226 shutdown 2024-03-14T10:02:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.573+0000 7f1b93594b80 1 -- 172.21.15.78:0/2799843374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff7b2d0 msgr2=0x55e10ff78630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.573+0000 7f1b93594b80 1 --2- 172.21.15.78:0/2799843374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff7b2d0 0x55e10ff78630 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f1b70017e50 tx=0x7f1b70004ad0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.573+0000 7f1b93594b80 1 -- 172.21.15.78:0/2799843374 shutdown_connections 2024-03-14T10:02:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.573+0000 7f1b93594b80 1 --2- 172.21.15.78:0/2799843374 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55e10ff8ac00 0x55e10ff8d020 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.573+0000 7f1b93594b80 1 --2- 172.21.15.78:0/2799843374 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff70a90 0x55e10ff7ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.573+0000 7f1b93594b80 1 --2- 172.21.15.78:0/2799843374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff7b2d0 0x55e10ff78630 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.573+0000 7f1b93594b80 1 --2- 172.21.15.78:0/2799843374 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff713b0 0x55e10ff78090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.573+0000 7f1b5ea00700 0 client.17226 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:02:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.573+0000 7f1b93594b80 1 -- 172.21.15.78:0/2799843374 >> 172.21.15.78:0/2799843374 conn(0x55e10fee2220 msgr2=0x55e10ff67690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.574+0000 7f1b93594b80 1 -- 172.21.15.78:0/2799843374 shutdown_connections 2024-03-14T10:02:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.574+0000 7f1b93594b80 1 -- 172.21.15.78:0/2799843374 wait complete. 2024-03-14T10:02:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.574+0000 7f1b93594b80 20 client.17226 trim_cache size 0 max 16384 2024-03-14T10:02:22.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.587+0000 7f1b93594b80 1 Processor -- start 2024-03-14T10:02:22.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.587+0000 7f1b93594b80 1 -- start start 2024-03-14T10:02:22.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.587+0000 7f1b93594b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff70a90 0x55e10ff72b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.587+0000 7f1b93594b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff713b0 0x55e10ff7a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.587+0000 7f1b93594b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff7b2d0 0x55e10ff76980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.587+0000 7f1b93594b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e10ff79550 con 0x55e10ff7b2d0 2024-03-14T10:02:22.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.587+0000 7f1b93594b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e10ff69060 con 0x55e10ff713b0 2024-03-14T10:02:22.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.587+0000 7f1b93594b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e10ff61d00 con 0x55e10ff70a90 2024-03-14T10:02:22.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.587+0000 7f1b7da00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff7b2d0 0x55e10ff76980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.587+0000 7f1b7da00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff7b2d0 0x55e10ff76980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38660/0 (socket says 172.21.15.78:38660) 2024-03-14T10:02:22.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.587+0000 7f1b7da00700 1 -- 172.21.15.78:0/3617388175 learned_addr learned my addr 172.21.15.78:0/3617388175 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:22.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.588+0000 7f1b7d000700 1 --2- 172.21.15.78:0/3617388175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff70a90 0x55e10ff72b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.588+0000 7f1b77e00700 1 --2- 172.21.15.78:0/3617388175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff713b0 0x55e10ff7a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.588+0000 7f1b7da00700 1 -- 172.21.15.78:0/3617388175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff70a90 msgr2=0x55e10ff72b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.588+0000 7f1b7da00700 1 --2- 172.21.15.78:0/3617388175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff70a90 0x55e10ff72b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.588+0000 7f1b7da00700 1 -- 172.21.15.78:0/3617388175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff713b0 msgr2=0x55e10ff7a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.588+0000 7f1b7da00700 1 --2- 172.21.15.78:0/3617388175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff713b0 0x55e10ff7a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.588+0000 7f1b7da00700 1 -- 172.21.15.78:0/3617388175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e10fcecb60 con 0x55e10ff7b2d0 2024-03-14T10:02:22.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.588+0000 7f1b77e00700 1 --2- 172.21.15.78:0/3617388175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff713b0 0x55e10ff7a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:02:22.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.588+0000 7f1b7da00700 1 --2- 172.21.15.78:0/3617388175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff7b2d0 0x55e10ff76980 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f1b7002c7f0 tx=0x7f1b700091c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90085288e4a655e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.588+0000 7f1b6aa00700 1 -- 172.21.15.78:0/3617388175 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b700099a0 con 0x55e10ff7b2d0 2024-03-14T10:02:22.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.588+0000 7f1b6aa00700 1 -- 172.21.15.78:0/3617388175 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b7000f040 con 0x55e10ff7b2d0 2024-03-14T10:02:22.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.588+0000 7f1b6aa00700 1 -- 172.21.15.78:0/3617388175 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7003eb70 con 0x55e10ff7b2d0 2024-03-14T10:02:22.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.589+0000 7f1b93594b80 1 -- 172.21.15.78:0/3617388175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff7b2d0 msgr2=0x55e10ff76980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.589+0000 7f1b93594b80 1 --2- 172.21.15.78:0/3617388175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff7b2d0 0x55e10ff76980 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f1b7002c7f0 tx=0x7f1b700091c0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.589+0000 7f1b93594b80 1 -- 172.21.15.78:0/3617388175 shutdown_connections 2024-03-14T10:02:22.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.589+0000 7f1b93594b80 1 --2- 172.21.15.78:0/3617388175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff7b2d0 0x55e10ff76980 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.589+0000 7f1b93594b80 1 --2- 172.21.15.78:0/3617388175 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff70a90 0x55e10ff72b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.589+0000 7f1b93594b80 1 --2- 172.21.15.78:0/3617388175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff713b0 0x55e10ff7a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.589+0000 7f1b93594b80 1 -- 172.21.15.78:0/3617388175 >> 172.21.15.78:0/3617388175 conn(0x55e10fee2220 msgr2=0x55e10ff675d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.589+0000 7f1b93594b80 1 -- 172.21.15.78:0/3617388175 shutdown_connections 2024-03-14T10:02:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.589+0000 7f1b93594b80 1 -- 172.21.15.78:0/3617388175 wait complete. 2024-03-14T10:02:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.589+0000 7f1b93594b80 1 -- start start 2024-03-14T10:02:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.590+0000 7f1b93594b80 1 Processor -- start 2024-03-14T10:02:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.590+0000 7f1b93594b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff70a90 0x55e10ff7d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.590+0000 7f1b93594b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff713b0 0x55e10ff7dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.590+0000 7f1b93594b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff7b2d0 0x55e10ff7e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.590+0000 7f1b93594b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e10ffdb720 con 0x55e10ff70a90 2024-03-14T10:02:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.590+0000 7f1b93594b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e10ffdb860 con 0x55e10ff7b2d0 2024-03-14T10:02:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.590+0000 7f1b93594b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e10ffdb9d0 con 0x55e10ff713b0 2024-03-14T10:02:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.590+0000 7f1b93594b80 10 client.0 _ll_register_callbacks cb 0x55e10fcc63a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.590+0000 7f1b7d000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff70a90 0x55e10ff7d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.590+0000 7f1b7d000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff70a90 0x55e10ff7d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38666/0 (socket says 172.21.15.78:38666) 2024-03-14T10:02:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.590+0000 7f1b7d000700 1 -- 172.21.15.78:0/2450919671 learned_addr learned my addr 172.21.15.78:0/2450919671 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.590+0000 7f1b77e00700 1 --2- 172.21.15.78:0/2450919671 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff713b0 0x55e10ff7dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.590+0000 7f1b7da00700 1 --2- 172.21.15.78:0/2450919671 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff7b2d0 0x55e10ff7e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.591+0000 7f1b7d000700 1 -- 172.21.15.78:0/2450919671 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff713b0 msgr2=0x55e10ff7dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.591+0000 7f1b7d000700 1 --2- 172.21.15.78:0/2450919671 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff713b0 0x55e10ff7dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.591+0000 7f1b7d000700 1 -- 172.21.15.78:0/2450919671 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff7b2d0 msgr2=0x55e10ff7e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.591+0000 7f1b7d000700 1 --2- 172.21.15.78:0/2450919671 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff7b2d0 0x55e10ff7e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.591+0000 7f1b7d000700 1 -- 172.21.15.78:0/2450919671 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b781a4690 con 0x55e10ff70a90 2024-03-14T10:02:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.591+0000 7f1b7d000700 1 --2- 172.21.15.78:0/2450919671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff70a90 0x55e10ff7d6c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c09a810 tx=0x7f1b6c096f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6416a2cf03dcabbf server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.591+0000 7f1b5ea00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.591+0000 7f1b5ea00700 1 -- 172.21.15.78:0/2450919671 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c09b210 con 0x55e10ff70a90 2024-03-14T10:02:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.591+0000 7f1b5ea00700 1 -- 172.21.15.78:0/2450919671 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f1b6c0b2070 con 0x55e10ff70a90 2024-03-14T10:02:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.591+0000 7f1b93594b80 1 -- 172.21.15.78:0/2450919671 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1b6c09a470 con 0x55e10ff70a90 2024-03-14T10:02:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.591+0000 7f1b5ea00700 1 -- 172.21.15.78:0/2450919671 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b6c0a2f00 con 0x55e10ff70a90 2024-03-14T10:02:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.591+0000 7f1b93594b80 10 client.17232 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.591+0000 7f1b93594b80 1 -- 172.21.15.78:0/2450919671 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e10fe84ea0 con 0x55e10ff70a90 2024-03-14T10:02:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.592+0000 7f1b5ea00700 1 -- 172.21.15.78:0/2450919671 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b6c09a470 con 0x55e10ff70a90 2024-03-14T10:02:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.592+0000 7f1b5ea00700 1 -- 172.21.15.78:0/2450919671 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1b6c060d20 con 0x55e10ff70a90 2024-03-14T10:02:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.592+0000 7f1b93594b80 10 client.17232 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.592+0000 7f1b93594b80 10 client.17232 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:02:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.592+0000 7f1b93594b80 20 client.17232 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.592+0000 7f1b93594b80 1 --2- 172.21.15.78:0/2450919671 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55e10ffdce20 0x55e10ffdf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.592+0000 7f1b93594b80 4 client.17232 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:02:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.592+0000 7f1b93594b80 1 -- 172.21.15.78:0/2450919671 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e10fd0dcf0 con 0x55e10ffdce20 2024-03-14T10:02:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.592+0000 7f1b77e00700 1 --2- 172.21.15.78:0/2450919671 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55e10ffdce20 0x55e10ffdf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.592+0000 7f1b5ea00700 10 client.17232 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:02:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.592+0000 7f1b77e00700 1 --2- 172.21.15.78:0/2450919671 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55e10ffdce20 0x55e10ffdf240 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5b60c86cf08b9e3 server_cookie=f5616e37c9945206 in_seq=0 out_seq=0 2024-03-14T10:02:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.593+0000 7f1b5ea00700 1 -- 172.21.15.78:0/2450919671 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e10fd0dcf0 con 0x55e10ffdce20 2024-03-14T10:02:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.593+0000 7f1b5ea00700 10 client.17232 handle_command_reply: tid=0 2024-03-14T10:02:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.593+0000 7f1b93594b80 1 client.17232 shutdown 2024-03-14T10:02:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.593+0000 7f1b93594b80 1 -- 172.21.15.78:0/2450919671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff70a90 msgr2=0x55e10ff7d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.593+0000 7f1b93594b80 1 --2- 172.21.15.78:0/2450919671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff70a90 0x55e10ff7d6c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c09a810 tx=0x7f1b6c096f80 comp rx=0 tx=0).stop 2024-03-14T10:02:22.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.593+0000 7f1b93594b80 1 -- 172.21.15.78:0/2450919671 shutdown_connections 2024-03-14T10:02:22.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.593+0000 7f1b93594b80 1 --2- 172.21.15.78:0/2450919671 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55e10ffdce20 0x55e10ffdf240 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.593+0000 7f1b93594b80 1 --2- 172.21.15.78:0/2450919671 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e10ff7b2d0 0x55e10ff7e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.593+0000 7f1b93594b80 1 --2- 172.21.15.78:0/2450919671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e10ff70a90 0x55e10ff7d6c0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.593+0000 7f1b93594b80 1 --2- 172.21.15.78:0/2450919671 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e10ff713b0 0x55e10ff7dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.593+0000 7f1b5ea00700 0 client.17232 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:02:22.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.593+0000 7f1b93594b80 1 -- 172.21.15.78:0/2450919671 >> 172.21.15.78:0/2450919671 conn(0x55e10fee2220 msgr2=0x55e10ff768c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.593+0000 7f1b93594b80 1 -- 172.21.15.78:0/2450919671 shutdown_connections 2024-03-14T10:02:22.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.593+0000 7f1b93594b80 1 -- 172.21.15.78:0/2450919671 wait complete. 2024-03-14T10:02:22.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.593+0000 7f1b93594b80 20 client.17232 trim_cache size 0 max 16384 2024-03-14T10:02:22.597 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:02:22.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.594+0000 7f1b7fa00700 1 -- 172.21.15.78:0/835597075 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1b600a0c00 msgr2=0x7f1b600a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.594+0000 7f1b7fa00700 1 --2- 172.21.15.78:0/835597075 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1b600a0c00 0x7f1b600a30c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1b70007840 tx=0x7f1b70009040 comp rx=0 tx=0).stop 2024-03-14T10:02:22.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.594+0000 7f1b7fa00700 1 -- 172.21.15.78:0/835597075 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b78100f20 msgr2=0x7f1b78197680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.594+0000 7f1b7fa00700 1 --2- 172.21.15.78:0/835597075 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b78100f20 0x7f1b78197680 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c005910 tx=0x7f1b6c004730 comp rx=0 tx=0).stop 2024-03-14T10:02:22.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.594+0000 7f1b7fa00700 1 -- 172.21.15.78:0/835597075 shutdown_connections 2024-03-14T10:02:22.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.594+0000 7f1b7fa00700 1 --2- 172.21.15.78:0/835597075 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1b600a0c00 0x7f1b600a30c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.594+0000 7f1b7fa00700 1 --2- 172.21.15.78:0/835597075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b7819bf60 0x7f1b7819c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.594+0000 7f1b7fa00700 1 --2- 172.21.15.78:0/835597075 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1b78100f20 0x7f1b78197680 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.594+0000 7f1b7fa00700 1 --2- 172.21.15.78:0/835597075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b78106040 0x7f1b78197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.594+0000 7f1b7fa00700 1 -- 172.21.15.78:0/835597075 >> 172.21.15.78:0/835597075 conn(0x7f1b780fc7a0 msgr2=0x7f1b780fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.594+0000 7f1b7fa00700 1 -- 172.21.15.78:0/835597075 shutdown_connections 2024-03-14T10:02:22.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.594+0000 7f1b7fa00700 1 -- 172.21.15.78:0/835597075 wait complete. 2024-03-14T10:02:22.612 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T10:02:22.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.808+0000 7f452d200700 1 -- 172.21.15.78:0/3225252870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4528100f20 msgr2=0x7f4528101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.808+0000 7f452d200700 1 --2- 172.21.15.78:0/3225252870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4528100f20 0x7f4528101300 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4518009990 tx=0x7f4518009ca0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.808+0000 7f452d200700 1 -- 172.21.15.78:0/3225252870 shutdown_connections 2024-03-14T10:02:22.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.808+0000 7f452d200700 1 --2- 172.21.15.78:0/3225252870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4528106040 0x7f452810dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.808+0000 7f452d200700 1 --2- 172.21.15.78:0/3225252870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4528100f20 0x7f4528101300 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.808+0000 7f452d200700 1 --2- 172.21.15.78:0/3225252870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f45281018d0 0x7f4528105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.808+0000 7f452d200700 1 -- 172.21.15.78:0/3225252870 >> 172.21.15.78:0/3225252870 conn(0x7f45280fc7a0 msgr2=0x7f45280febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.808+0000 7f452d200700 1 -- 172.21.15.78:0/3225252870 shutdown_connections 2024-03-14T10:02:22.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.808+0000 7f452d200700 1 -- 172.21.15.78:0/3225252870 wait complete. 2024-03-14T10:02:22.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.809+0000 7f452d200700 1 Processor -- start 2024-03-14T10:02:22.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.809+0000 7f452d200700 1 -- start start 2024-03-14T10:02:22.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.809+0000 7f452d200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4528100f20 0x7f45281a39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.809+0000 7f452d200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4528106040 0x7f45281a3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.809+0000 7f452d200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f452810cd20 0x7f452810d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.809+0000 7f452d200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452810d7a0 con 0x7f4528106040 2024-03-14T10:02:22.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.809+0000 7f452d200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452810d8e0 con 0x7f4528100f20 2024-03-14T10:02:22.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.809+0000 7f452d200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452810daa0 con 0x7f452810cd20 2024-03-14T10:02:22.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.810+0000 7f451fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4528106040 0x7f45281a3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.810+0000 7f4526800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f452810cd20 0x7f452810d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.810+0000 7f451fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4528106040 0x7f45281a3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38688/0 (socket says 172.21.15.78:38688) 2024-03-14T10:02:22.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.810+0000 7f451fe00700 1 -- 172.21.15.78:0/1042807262 learned_addr learned my addr 172.21.15.78:0/1042807262 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:22.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.810+0000 7f4525e00700 1 --2- 172.21.15.78:0/1042807262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4528100f20 0x7f45281a39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.810+0000 7f451fe00700 1 -- 172.21.15.78:0/1042807262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f452810cd20 msgr2=0x7f452810d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.810+0000 7f451fe00700 1 --2- 172.21.15.78:0/1042807262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f452810cd20 0x7f452810d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.810+0000 7f451fe00700 1 -- 172.21.15.78:0/1042807262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4528100f20 msgr2=0x7f45281a39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.810+0000 7f451fe00700 1 --2- 172.21.15.78:0/1042807262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4528100f20 0x7f45281a39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.810+0000 7f451fe00700 1 -- 172.21.15.78:0/1042807262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45180095f0 con 0x7f4528106040 2024-03-14T10:02:22.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.810+0000 7f4525e00700 1 --2- 172.21.15.78:0/1042807262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4528100f20 0x7f45281a39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:02:22.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.810+0000 7f451fe00700 1 --2- 172.21.15.78:0/1042807262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4528106040 0x7f45281a3f00 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f451000e970 tx=0x7f451000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.810+0000 7f451ea00700 1 -- 172.21.15.78:0/1042807262 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4510009790 con 0x7f4528106040 2024-03-14T10:02:22.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.810+0000 7f451ea00700 1 -- 172.21.15.78:0/1042807262 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f451000eec0 con 0x7f4528106040 2024-03-14T10:02:22.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.810+0000 7f452d200700 1 -- 172.21.15.78:0/1042807262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45281aa830 con 0x7f4528106040 2024-03-14T10:02:22.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.810+0000 7f451ea00700 1 -- 172.21.15.78:0/1042807262 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4510005590 con 0x7f4528106040 2024-03-14T10:02:22.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.811+0000 7f452d200700 1 -- 172.21.15.78:0/1042807262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45281aad40 con 0x7f4528106040 2024-03-14T10:02:22.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.813+0000 7f451ea00700 1 -- 172.21.15.78:0/1042807262 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f451000cbd0 con 0x7f4528106040 2024-03-14T10:02:22.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.815+0000 7f4540db6b80 1 Processor -- start 2024-03-14T10:02:22.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.815+0000 7f4540db6b80 1 -- start start 2024-03-14T10:02:22.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.815+0000 7f451ea00700 1 --2- 172.21.15.78:0/1042807262 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f450c0a0c00 0x7f450c0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.815+0000 7f451ea00700 1 -- 172.21.15.78:0/1042807262 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f4510014070 con 0x7f4528106040 2024-03-14T10:02:22.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.815+0000 7f4540db6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6680 0x564a35ca6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.815+0000 7f4526800700 1 --2- 172.21.15.78:0/1042807262 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f450c0a0c00 0x7f450c0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.815+0000 7f4540db6b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35ca6fa0 0x564a35cb0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.815+0000 7f4540db6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35cb0ec0 0x564a35cb32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.815+0000 7f4540db6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a35bff990 con 0x564a35ca6680 2024-03-14T10:02:22.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.815+0000 7f4540db6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a35bfd350 con 0x564a35cb0ec0 2024-03-14T10:02:22.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.815+0000 7f4540db6b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a35bf8c40 con 0x564a35ca6fa0 2024-03-14T10:02:22.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.816+0000 7f4525e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6680 0x564a35ca6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.816+0000 7f4525e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6680 0x564a35ca6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38694/0 (socket says 172.21.15.78:38694) 2024-03-14T10:02:22.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.816+0000 7f4525e00700 1 -- 172.21.15.78:0/3913161693 learned_addr learned my addr 172.21.15.78:0/3913161693 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:22.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.816+0000 7f451fe00700 1 --2- 172.21.15.78:0/3913161693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35ca6fa0 0x564a35cb0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.816+0000 7f4526800700 1 --2- 172.21.15.78:0/3913161693 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35cb0ec0 0x564a35cb32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.816+0000 7f4525e00700 1 -- 172.21.15.78:0/3913161693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35ca6fa0 msgr2=0x564a35cb0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.816+0000 7f4525e00700 1 --2- 172.21.15.78:0/3913161693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35ca6fa0 0x564a35cb0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.816+0000 7f4525e00700 1 -- 172.21.15.78:0/3913161693 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35cb0ec0 msgr2=0x564a35cb32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.816+0000 7f4525e00700 1 --2- 172.21.15.78:0/3913161693 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35cb0ec0 0x564a35cb32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.816+0000 7f4525e00700 1 -- 172.21.15.78:0/3913161693 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a35a7a3b0 con 0x564a35ca6680 2024-03-14T10:02:22.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.816+0000 7f4526800700 1 --2- 172.21.15.78:0/1042807262 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f450c0a0c00 0x7f450c0a30c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f452000f540 tx=0x7f4520009920 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.816+0000 7f4525e00700 1 --2- 172.21.15.78:0/3913161693 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6680 0x564a35ca6a60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f451800b2e0 tx=0x7f4518005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac2a003b4255ecb server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.817+0000 7f4517400700 1 -- 172.21.15.78:0/3913161693 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4518004a10 con 0x564a35ca6680 2024-03-14T10:02:22.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.817+0000 7f4517400700 1 -- 172.21.15.78:0/3913161693 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f451801e040 con 0x564a35ca6680 2024-03-14T10:02:22.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.817+0000 7f4517400700 1 -- 172.21.15.78:0/3913161693 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451800b9e0 con 0x564a35ca6680 2024-03-14T10:02:22.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.817+0000 7f4540db6b80 1 -- 172.21.15.78:0/3913161693 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6680 msgr2=0x564a35ca6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.817+0000 7f4540db6b80 1 --2- 172.21.15.78:0/3913161693 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6680 0x564a35ca6a60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f451800b2e0 tx=0x7f4518005c00 comp rx=0 tx=0).stop 2024-03-14T10:02:22.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.817+0000 7f4540db6b80 1 -- 172.21.15.78:0/3913161693 shutdown_connections 2024-03-14T10:02:22.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.817+0000 7f4540db6b80 1 --2- 172.21.15.78:0/3913161693 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35cb0ec0 0x564a35cb32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.817+0000 7f4540db6b80 1 --2- 172.21.15.78:0/3913161693 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6680 0x564a35ca6a60 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.817+0000 7f4540db6b80 1 --2- 172.21.15.78:0/3913161693 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35ca6fa0 0x564a35cb0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.817+0000 7f4540db6b80 1 -- 172.21.15.78:0/3913161693 >> 172.21.15.78:0/3913161693 conn(0x564a35c1c220 msgr2=0x564a35ca1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.817+0000 7f4540db6b80 1 -- 172.21.15.78:0/3913161693 shutdown_connections 2024-03-14T10:02:22.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.818+0000 7f4540db6b80 1 -- 172.21.15.78:0/3913161693 wait complete. 2024-03-14T10:02:22.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.818+0000 7f4540db6b80 1 -- start start 2024-03-14T10:02:22.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.818+0000 7f4540db6b80 1 Processor -- start 2024-03-14T10:02:22.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f4540db6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6680 0x564a35cb0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f4540db6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6fa0 0x564a35cadcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f4540db6b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35cb0ec0 0x564a35cae400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f4540db6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a35b94d80 con 0x564a35ca6fa0 2024-03-14T10:02:22.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f4540db6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a35b94a20 con 0x564a35ca6680 2024-03-14T10:02:22.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f4540db6b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a35a7c6a0 con 0x564a35cb0ec0 2024-03-14T10:02:22.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f4540db6b80 10 client.0 _ll_register_callbacks cb 0x564a35a003a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f451fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6fa0 0x564a35cadcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f4525e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6680 0x564a35cb0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f451fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6fa0 0x564a35cadcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38700/0 (socket says 172.21.15.78:38700) 2024-03-14T10:02:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f4525e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6680 0x564a35cb0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:49660/0 (socket says 172.21.15.78:49660) 2024-03-14T10:02:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f451fe00700 1 -- 172.21.15.78:0/3856202302 learned_addr learned my addr 172.21.15.78:0/3856202302 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f4526800700 1 --2- 172.21.15.78:0/3856202302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35cb0ec0 0x564a35cae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f451fe00700 1 -- 172.21.15.78:0/3856202302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35cb0ec0 msgr2=0x564a35cae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f451fe00700 1 --2- 172.21.15.78:0/3856202302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35cb0ec0 0x564a35cae400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f451fe00700 1 -- 172.21.15.78:0/3856202302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6680 msgr2=0x564a35cb0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f451fe00700 1 --2- 172.21.15.78:0/3856202302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6680 0x564a35cb0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f451fe00700 1 -- 172.21.15.78:0/3856202302 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452810e430 con 0x564a35ca6fa0 2024-03-14T10:02:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.819+0000 7f4525e00700 1 --2- 172.21.15.78:0/3856202302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6680 0x564a35cb0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.820+0000 7f451fe00700 1 --2- 172.21.15.78:0/3856202302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6fa0 0x564a35cadcc0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f45100964c0 tx=0x7f451009a240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f52ddedc58510ede server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.820+0000 7f450be00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.820+0000 7f450be00700 1 -- 172.21.15.78:0/3856202302 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451009d880 con 0x564a35ca6fa0 2024-03-14T10:02:22.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.820+0000 7f450be00700 1 -- 172.21.15.78:0/3856202302 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f45100a1ed0 con 0x564a35ca6fa0 2024-03-14T10:02:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.820+0000 7f4540db6b80 1 -- 172.21.15.78:0/3856202302 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564a35c99720 con 0x564a35ca6fa0 2024-03-14T10:02:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.820+0000 7f450be00700 1 -- 172.21.15.78:0/3856202302 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45100b2260 con 0x564a35ca6fa0 2024-03-14T10:02:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.820+0000 7f4540db6b80 10 client.17244 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.820+0000 7f4540db6b80 1 -- 172.21.15.78:0/3856202302 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564a35a26b60 con 0x564a35ca6fa0 2024-03-14T10:02:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.820+0000 7f450be00700 1 -- 172.21.15.78:0/3856202302 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564a35c99720 con 0x564a35ca6fa0 2024-03-14T10:02:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.820+0000 7f450be00700 1 -- 172.21.15.78:0/3856202302 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f45100989b0 con 0x564a35ca6fa0 2024-03-14T10:02:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.820+0000 7f4540db6b80 10 client.17244 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.820+0000 7f4540db6b80 10 client.17244 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:02:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.820+0000 7f4540db6b80 20 client.17244 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.820+0000 7f4540db6b80 1 --2- 172.21.15.78:0/3856202302 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x564a35cc0b60 0x564a35cc2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.820+0000 7f4540db6b80 4 client.17244 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:02:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.821+0000 7f4540db6b80 1 -- 172.21.15.78:0/3856202302 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564a35bf1700 con 0x564a35cc0b60 2024-03-14T10:02:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.821+0000 7f4525e00700 1 --2- 172.21.15.78:0/3856202302 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x564a35cc0b60 0x564a35cc2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.821+0000 7f4525e00700 1 --2- 172.21.15.78:0/3856202302 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x564a35cc0b60 0x564a35cc2f50 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ed147325feec38b server_cookie=c23837aae8bff03 in_seq=0 out_seq=0 2024-03-14T10:02:22.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.821+0000 7f450be00700 10 client.17244 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:22.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.822+0000 7f450be00700 1 -- 172.21.15.78:0/3856202302 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x564a35bf1700 con 0x564a35cc0b60 2024-03-14T10:02:22.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.822+0000 7f450be00700 10 client.17244 handle_command_reply: tid=0 2024-03-14T10:02:22.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.822+0000 7f4540db6b80 1 client.17244 shutdown 2024-03-14T10:02:22.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.822+0000 7f4540db6b80 1 -- 172.21.15.78:0/3856202302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6fa0 msgr2=0x564a35cadcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.822+0000 7f4540db6b80 1 --2- 172.21.15.78:0/3856202302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6fa0 0x564a35cadcc0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f45100964c0 tx=0x7f451009a240 comp rx=0 tx=0).stop 2024-03-14T10:02:22.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.822+0000 7f4540db6b80 1 -- 172.21.15.78:0/3856202302 shutdown_connections 2024-03-14T10:02:22.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.822+0000 7f4540db6b80 1 --2- 172.21.15.78:0/3856202302 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x564a35cc0b60 0x564a35cc2f50 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.822+0000 7f4540db6b80 1 --2- 172.21.15.78:0/3856202302 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6680 0x564a35cb0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.822+0000 7f4540db6b80 1 --2- 172.21.15.78:0/3856202302 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35cb0ec0 0x564a35cae400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.822+0000 7f450be00700 0 client.17244 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:22.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.822+0000 7f4540db6b80 1 --2- 172.21.15.78:0/3856202302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6fa0 0x564a35cadcc0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.822+0000 7f4540db6b80 1 -- 172.21.15.78:0/3856202302 >> 172.21.15.78:0/3856202302 conn(0x564a35c1c220 msgr2=0x564a35ca1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.822+0000 7f4540db6b80 1 -- 172.21.15.78:0/3856202302 shutdown_connections 2024-03-14T10:02:22.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.822+0000 7f4540db6b80 1 -- 172.21.15.78:0/3856202302 wait complete. 2024-03-14T10:02:22.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.823+0000 7f4540db6b80 20 client.17244 trim_cache size 0 max 16384 2024-03-14T10:02:22.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.836+0000 7f4540db6b80 1 Processor -- start 2024-03-14T10:02:22.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.836+0000 7f4540db6b80 1 -- start start 2024-03-14T10:02:22.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.836+0000 7f4540db6b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35ca6680 0x564a35ca1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.836+0000 7f4540db6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6fa0 0x564a35cc7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.836+0000 7f4540db6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35cb0ec0 0x564a35cb32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.836+0000 7f4540db6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a35ca30b0 con 0x564a35cb0ec0 2024-03-14T10:02:22.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.836+0000 7f4540db6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a35cc0940 con 0x564a35ca6fa0 2024-03-14T10:02:22.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.836+0000 7f4540db6b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a35ca7890 con 0x564a35ca6680 2024-03-14T10:02:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.836+0000 7f4526800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35cb0ec0 0x564a35cb32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.836+0000 7f4525e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35ca6680 0x564a35ca1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.836+0000 7f4526800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35cb0ec0 0x564a35cb32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38706/0 (socket says 172.21.15.78:38706) 2024-03-14T10:02:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.836+0000 7f4526800700 1 -- 172.21.15.78:0/3259577365 learned_addr learned my addr 172.21.15.78:0/3259577365 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.836+0000 7f451fe00700 1 --2- 172.21.15.78:0/3259577365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6fa0 0x564a35cc7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.837+0000 7f4525e00700 1 -- 172.21.15.78:0/3259577365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6fa0 msgr2=0x564a35cc7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.837+0000 7f4525e00700 1 --2- 172.21.15.78:0/3259577365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6fa0 0x564a35cc7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.837+0000 7f4525e00700 1 -- 172.21.15.78:0/3259577365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35cb0ec0 msgr2=0x564a35cb32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.837+0000 7f4525e00700 1 --2- 172.21.15.78:0/3259577365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35cb0ec0 0x564a35cb32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.837+0000 7f4525e00700 1 -- 172.21.15.78:0/3259577365 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a35bbeea0 con 0x564a35ca6680 2024-03-14T10:02:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.837+0000 7f451fe00700 1 --2- 172.21.15.78:0/3259577365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6fa0 0x564a35cc7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:22.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.837+0000 7f4525e00700 1 --2- 172.21.15.78:0/3259577365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35ca6680 0x564a35ca1c30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4518000c00 tx=0x7f4518005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=678b5b063dde1430 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.837+0000 7f4517e00700 1 -- 172.21.15.78:0/3259577365 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451800b6f0 con 0x564a35ca6680 2024-03-14T10:02:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.837+0000 7f4517e00700 1 -- 172.21.15.78:0/3259577365 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45180313d0 con 0x564a35ca6680 2024-03-14T10:02:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.837+0000 7f4517e00700 1 -- 172.21.15.78:0/3259577365 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4518005160 con 0x564a35ca6680 2024-03-14T10:02:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.838+0000 7f4540db6b80 1 -- 172.21.15.78:0/3259577365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35ca6680 msgr2=0x564a35ca1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.838+0000 7f4540db6b80 1 --2- 172.21.15.78:0/3259577365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35ca6680 0x564a35ca1c30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4518000c00 tx=0x7f4518005fd0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.838+0000 7f4540db6b80 1 -- 172.21.15.78:0/3259577365 shutdown_connections 2024-03-14T10:02:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.838+0000 7f4540db6b80 1 --2- 172.21.15.78:0/3259577365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35cb0ec0 0x564a35cb32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.838+0000 7f4540db6b80 1 --2- 172.21.15.78:0/3259577365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35ca6680 0x564a35ca1c30 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.838+0000 7f4540db6b80 1 --2- 172.21.15.78:0/3259577365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6fa0 0x564a35cc7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.838+0000 7f4540db6b80 1 -- 172.21.15.78:0/3259577365 >> 172.21.15.78:0/3259577365 conn(0x564a35c1c220 msgr2=0x564a35ca8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.838+0000 7f4540db6b80 1 -- 172.21.15.78:0/3259577365 shutdown_connections 2024-03-14T10:02:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.838+0000 7f4540db6b80 1 -- 172.21.15.78:0/3259577365 wait complete. 2024-03-14T10:02:22.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.838+0000 7f4540db6b80 1 -- start start 2024-03-14T10:02:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.839+0000 7f4540db6b80 1 Processor -- start 2024-03-14T10:02:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.839+0000 7f4540db6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6680 0x564a35cb5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.839+0000 7f4540db6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6fa0 0x564a35cb5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.839+0000 7f4540db6b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35cb0ec0 0x564a35cb6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.839+0000 7f4540db6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a35d11620 con 0x564a35ca6680 2024-03-14T10:02:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.839+0000 7f4540db6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a35d11760 con 0x564a35ca6fa0 2024-03-14T10:02:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.839+0000 7f4540db6b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a35d118a0 con 0x564a35cb0ec0 2024-03-14T10:02:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.839+0000 7f4540db6b80 10 client.0 _ll_register_callbacks cb 0x7f4510004ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.839+0000 7f4525e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6680 0x564a35cb5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.839+0000 7f4525e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6680 0x564a35cb5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:38716/0 (socket says 172.21.15.78:38716) 2024-03-14T10:02:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.839+0000 7f4525e00700 1 -- 172.21.15.78:0/2315075709 learned_addr learned my addr 172.21.15.78:0/2315075709 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.839+0000 7f451fe00700 1 --2- 172.21.15.78:0/2315075709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6fa0 0x564a35cb5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.839+0000 7f4526800700 1 --2- 172.21.15.78:0/2315075709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35cb0ec0 0x564a35cb6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.840+0000 7f4525e00700 1 -- 172.21.15.78:0/2315075709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35cb0ec0 msgr2=0x564a35cb6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.840+0000 7f4525e00700 1 --2- 172.21.15.78:0/2315075709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35cb0ec0 0x564a35cb6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.840+0000 7f4525e00700 1 -- 172.21.15.78:0/2315075709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6fa0 msgr2=0x564a35cb5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.840+0000 7f4525e00700 1 --2- 172.21.15.78:0/2315075709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6fa0 0x564a35cb5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.840+0000 7f4525e00700 1 -- 172.21.15.78:0/2315075709 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564a35a26b60 con 0x564a35ca6680 2024-03-14T10:02:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.840+0000 7f451fe00700 1 --2- 172.21.15.78:0/2315075709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6fa0 0x564a35cb5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.840+0000 7f4525e00700 1 --2- 172.21.15.78:0/2315075709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6680 0x564a35cb5aa0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4518005c40 tx=0x7f4518009950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4981928f0d74dc55 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.840+0000 7f450be00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.840+0000 7f450be00700 1 -- 172.21.15.78:0/2315075709 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4518031e60 con 0x564a35ca6680 2024-03-14T10:02:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.840+0000 7f450be00700 1 -- 172.21.15.78:0/2315075709 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f451803e070 con 0x564a35ca6680 2024-03-14T10:02:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.840+0000 7f4540db6b80 1 -- 172.21.15.78:0/2315075709 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f451801e040 con 0x564a35ca6680 2024-03-14T10:02:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.840+0000 7f450be00700 1 -- 172.21.15.78:0/2315075709 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f451800ba70 con 0x564a35ca6680 2024-03-14T10:02:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.840+0000 7f4540db6b80 10 client.17250 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.840+0000 7f4540db6b80 1 -- 172.21.15.78:0/2315075709 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564a35a7a3b0 con 0x564a35ca6680 2024-03-14T10:02:22.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.841+0000 7f450be00700 1 -- 172.21.15.78:0/2315075709 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f451801e040 con 0x564a35ca6680 2024-03-14T10:02:22.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.841+0000 7f450be00700 1 -- 172.21.15.78:0/2315075709 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f451802d480 con 0x564a35ca6680 2024-03-14T10:02:22.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.841+0000 7f4540db6b80 10 client.17250 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:22.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.841+0000 7f4540db6b80 10 client.17250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:02:22.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.841+0000 7f4540db6b80 20 client.17250 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:22.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.841+0000 7f4540db6b80 1 --2- 172.21.15.78:0/2315075709 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x564a35d12b60 0x564a35d14f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:22.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.841+0000 7f4540db6b80 4 client.17250 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:02:22.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.841+0000 7f4540db6b80 1 -- 172.21.15.78:0/2315075709 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564a35a47cf0 con 0x564a35d12b60 2024-03-14T10:02:22.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.841+0000 7f451fe00700 1 --2- 172.21.15.78:0/2315075709 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x564a35d12b60 0x564a35d14f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:22.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.842+0000 7f451fe00700 1 --2- 172.21.15.78:0/2315075709 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x564a35d12b60 0x564a35d14f50 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7055da8667db941 server_cookie=4c8a2506d0e29e22 in_seq=0 out_seq=0 2024-03-14T10:02:22.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.842+0000 7f450be00700 10 client.17250 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:22.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.842+0000 7f450be00700 1 -- 172.21.15.78:0/2315075709 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564a35a47cf0 con 0x564a35d12b60 2024-03-14T10:02:22.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.842+0000 7f450be00700 10 client.17250 handle_command_reply: tid=0 2024-03-14T10:02:22.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.842+0000 7f4540db6b80 1 client.17250 shutdown 2024-03-14T10:02:22.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.842+0000 7f4540db6b80 1 -- 172.21.15.78:0/2315075709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6680 msgr2=0x564a35cb5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.842+0000 7f4540db6b80 1 --2- 172.21.15.78:0/2315075709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6680 0x564a35cb5aa0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4518005c40 tx=0x7f4518009950 comp rx=0 tx=0).stop 2024-03-14T10:02:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.842+0000 7f4540db6b80 1 -- 172.21.15.78:0/2315075709 shutdown_connections 2024-03-14T10:02:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.842+0000 7f4540db6b80 1 --2- 172.21.15.78:0/2315075709 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x564a35cb0ec0 0x564a35cb6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.842+0000 7f4540db6b80 1 --2- 172.21.15.78:0/2315075709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564a35ca6680 0x564a35cb5aa0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.842+0000 7f4540db6b80 1 --2- 172.21.15.78:0/2315075709 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x564a35d12b60 0x564a35d14f50 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.842+0000 7f4540db6b80 1 --2- 172.21.15.78:0/2315075709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564a35ca6fa0 0x564a35cb5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.842+0000 7f450be00700 0 client.17250 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.842+0000 7f4540db6b80 1 -- 172.21.15.78:0/2315075709 >> 172.21.15.78:0/2315075709 conn(0x564a35c1c220 msgr2=0x564a35cbc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.842+0000 7f4540db6b80 1 -- 172.21.15.78:0/2315075709 shutdown_connections 2024-03-14T10:02:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.842+0000 7f4540db6b80 1 -- 172.21.15.78:0/2315075709 wait complete. 2024-03-14T10:02:22.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.843+0000 7f4540db6b80 20 client.17250 trim_cache size 0 max 16384 2024-03-14T10:02:22.847 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:02:22.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.843+0000 7f452d200700 1 -- 172.21.15.78:0/1042807262 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f450c0a0c00 msgr2=0x7f450c0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.843+0000 7f452d200700 1 --2- 172.21.15.78:0/1042807262 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f450c0a0c00 0x7f450c0a30c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f452000f540 tx=0x7f4520009920 comp rx=0 tx=0).stop 2024-03-14T10:02:22.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.843+0000 7f452d200700 1 -- 172.21.15.78:0/1042807262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4528106040 msgr2=0x7f45281a3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:22.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.843+0000 7f452d200700 1 --2- 172.21.15.78:0/1042807262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4528106040 0x7f45281a3f00 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f451000e970 tx=0x7f451000ec80 comp rx=0 tx=0).stop 2024-03-14T10:02:22.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.843+0000 7f452d200700 1 -- 172.21.15.78:0/1042807262 shutdown_connections 2024-03-14T10:02:22.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.843+0000 7f452d200700 1 --2- 172.21.15.78:0/1042807262 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f450c0a0c00 0x7f450c0a30c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.843+0000 7f452d200700 1 --2- 172.21.15.78:0/1042807262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4528100f20 0x7f45281a39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.843+0000 7f452d200700 1 --2- 172.21.15.78:0/1042807262 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f452810cd20 0x7f452810d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.843+0000 7f452d200700 1 --2- 172.21.15.78:0/1042807262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4528106040 0x7f45281a3f00 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:22.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.843+0000 7f452d200700 1 -- 172.21.15.78:0/1042807262 >> 172.21.15.78:0/1042807262 conn(0x7f45280fc7a0 msgr2=0x7f45280fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:22.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.843+0000 7f452d200700 1 -- 172.21.15.78:0/1042807262 shutdown_connections 2024-03-14T10:02:22.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:22.843+0000 7f452d200700 1 -- 172.21.15.78:0/1042807262 wait complete. 2024-03-14T10:02:23.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:23 smithi152 ceph-mon[48279]: pgmap v253: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-14T10:02:23.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:23 smithi132 ceph-mon[47959]: pgmap v253: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-14T10:02:23.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:23 smithi078 ceph-mon[38143]: pgmap v253: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-14T10:02:23.862 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T10:02:23.862 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T10:02:24.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.068+0000 7f048cc00700 1 -- 172.21.15.78:0/1305348959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f048810d4f0 msgr2=0x7f048810f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.068+0000 7f048cc00700 1 --2- 172.21.15.78:0/1305348959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f048810d4f0 0x7f048810f960 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f048000b0a0 tx=0x7f048000b3b0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.068+0000 7f048cc00700 1 -- 172.21.15.78:0/1305348959 shutdown_connections 2024-03-14T10:02:24.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.068+0000 7f048cc00700 1 --2- 172.21.15.78:0/1305348959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04880731e0 0x7f04880735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.068+0000 7f048cc00700 1 --2- 172.21.15.78:0/1305348959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f048810d4f0 0x7f048810f960 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.068+0000 7f048cc00700 1 --2- 172.21.15.78:0/1305348959 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0488073b00 0x7f048810cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.068+0000 7f048cc00700 1 -- 172.21.15.78:0/1305348959 >> 172.21.15.78:0/1305348959 conn(0x7f04880fc7a0 msgr2=0x7f04880febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:24.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.068+0000 7f048cc00700 1 -- 172.21.15.78:0/1305348959 shutdown_connections 2024-03-14T10:02:24.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.068+0000 7f048cc00700 1 -- 172.21.15.78:0/1305348959 wait complete. 2024-03-14T10:02:24.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.069+0000 7f048cc00700 1 Processor -- start 2024-03-14T10:02:24.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.070+0000 7f048cc00700 1 -- start start 2024-03-14T10:02:24.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.070+0000 7f048cc00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04880731e0 0x7f048819f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.070+0000 7f048cc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0488073b00 0x7f048819faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.070+0000 7f048cc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f048810d4f0 0x7f0488199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.070+0000 7f048cc00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0488199d90 con 0x7f048810d4f0 2024-03-14T10:02:24.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.070+0000 7f048cc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0488199f00 con 0x7f0488073b00 2024-03-14T10:02:24.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.070+0000 7f048cc00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048819a0c0 con 0x7f04880731e0 2024-03-14T10:02:24.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.071+0000 7f0485e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04880731e0 0x7f048819f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.071+0000 7f0485400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0488073b00 0x7f048819faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.071+0000 7f0486800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f048810d4f0 0x7f0488199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.071+0000 7f0485e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04880731e0 0x7f048819f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:57314/0 (socket says 172.21.15.78:57314) 2024-03-14T10:02:24.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.071+0000 7f0485e00700 1 -- 172.21.15.78:0/924465247 learned_addr learned my addr 172.21.15.78:0/924465247 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:24.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.071+0000 7f0485400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0488073b00 0x7f048819faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:54234/0 (socket says 172.21.15.78:54234) 2024-03-14T10:02:24.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.071+0000 7f0485e00700 1 -- 172.21.15.78:0/924465247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0488073b00 msgr2=0x7f048819faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.071+0000 7f0485e00700 1 --2- 172.21.15.78:0/924465247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0488073b00 0x7f048819faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.071+0000 7f0485e00700 1 -- 172.21.15.78:0/924465247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f048810d4f0 msgr2=0x7f0488199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.071+0000 7f0485e00700 1 --2- 172.21.15.78:0/924465247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f048810d4f0 0x7f0488199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.071+0000 7f0485e00700 1 -- 172.21.15.78:0/924465247 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0480009d00 con 0x7f04880731e0 2024-03-14T10:02:24.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.071+0000 7f0485400700 1 --2- 172.21.15.78:0/924465247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0488073b00 0x7f048819faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:24.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.071+0000 7f0486800700 1 --2- 172.21.15.78:0/924465247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f048810d4f0 0x7f0488199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:24.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.071+0000 7f0485e00700 1 --2- 172.21.15.78:0/924465247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04880731e0 0x7f048819f5b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f047800e9a0 tx=0x7f047800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:24.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.071+0000 7f047ea00700 1 -- 172.21.15.78:0/924465247 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0478009790 con 0x7f04880731e0 2024-03-14T10:02:24.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.071+0000 7f048cc00700 1 -- 172.21.15.78:0/924465247 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048819a3a0 con 0x7f04880731e0 2024-03-14T10:02:24.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.072+0000 7f047ea00700 1 -- 172.21.15.78:0/924465247 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0478010430 con 0x7f04880731e0 2024-03-14T10:02:24.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.072+0000 7f047ea00700 1 -- 172.21.15.78:0/924465247 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0478018680 con 0x7f04880731e0 2024-03-14T10:02:24.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.072+0000 7f048cc00700 1 -- 172.21.15.78:0/924465247 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04881a65a0 con 0x7f04880731e0 2024-03-14T10:02:24.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.076+0000 7f047ea00700 1 -- 172.21.15.78:0/924465247 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f04780105a0 con 0x7f04880731e0 2024-03-14T10:02:24.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.077+0000 7f04a0735b80 1 Processor -- start 2024-03-14T10:02:24.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.077+0000 7f04a0735b80 1 -- start start 2024-03-14T10:02:24.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.077+0000 7f04a0735b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827d7700 0x55d0827d7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.077+0000 7f04a0735b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827d8020 0x55d0827e1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.077+0000 7f04a0735b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827e1f40 0x55d0827e4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.077+0000 7f04a0735b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d082730a10 con 0x55d0827d8020 2024-03-14T10:02:24.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.077+0000 7f04a0735b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d08272e3d0 con 0x55d0827d7700 2024-03-14T10:02:24.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.077+0000 7f04a0735b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d082729cc0 con 0x55d0827e1f40 2024-03-14T10:02:24.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.077+0000 7f047ea00700 1 --2- 172.21.15.78:0/924465247 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f046c0a0c00 0x7f046c0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.077+0000 7f0485e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827d8020 0x55d0827e1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.078+0000 7f0485e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827d8020 0x55d0827e1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57704/0 (socket says 172.21.15.78:57704) 2024-03-14T10:02:24.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.078+0000 7f0485e00700 1 -- 172.21.15.78:0/2897399553 learned_addr learned my addr 172.21.15.78:0/2897399553 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:24.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.078+0000 7f0486800700 1 --2- 172.21.15.78:0/2897399553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827d7700 0x55d0827d7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.078+0000 7f0485400700 1 --2- 172.21.15.78:0/2897399553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827e1f40 0x55d0827e4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.078+0000 7f0486800700 1 --2- 172.21.15.78:0/924465247 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f046c0a0c00 0x7f046c0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.078+0000 7f047ea00700 1 -- 172.21.15.78:0/924465247 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f0478014070 con 0x7f04880731e0 2024-03-14T10:02:24.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.078+0000 7f0485e00700 1 -- 172.21.15.78:0/2897399553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827e1f40 msgr2=0x55d0827e4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:24.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.078+0000 7f0485e00700 1 --2- 172.21.15.78:0/2897399553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827e1f40 0x55d0827e4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.078+0000 7f0485e00700 1 -- 172.21.15.78:0/2897399553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827d7700 msgr2=0x55d0827d7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:24.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.078+0000 7f0485e00700 1 --2- 172.21.15.78:0/2897399553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827d7700 0x55d0827d7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.078+0000 7f0485e00700 1 -- 172.21.15.78:0/2897399553 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0825ab430 con 0x55d0827d8020 2024-03-14T10:02:24.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.078+0000 7f0485400700 1 --2- 172.21.15.78:0/2897399553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827e1f40 0x55d0827e4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:24.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.078+0000 7f0485e00700 1 --2- 172.21.15.78:0/2897399553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827d8020 0x55d0827e1a00 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f04780c9cf0 tx=0x7f0478099510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55a2c8b90dc60eea server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:24.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.078+0000 7f0486800700 1 --2- 172.21.15.78:0/924465247 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f046c0a0c00 0x7f046c0a30c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0480009040 tx=0x7f0480006230 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:24.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.078+0000 7f0473000700 1 -- 172.21.15.78:0/2897399553 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0478099bb0 con 0x55d0827d8020 2024-03-14T10:02:24.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.079+0000 7f0473000700 1 -- 172.21.15.78:0/2897399553 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f047809a050 con 0x55d0827d8020 2024-03-14T10:02:24.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.079+0000 7f0473000700 1 -- 172.21.15.78:0/2897399553 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0478098e40 con 0x55d0827d8020 2024-03-14T10:02:24.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.079+0000 7f04a0735b80 1 -- 172.21.15.78:0/2897399553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827d8020 msgr2=0x55d0827e1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.079+0000 7f04a0735b80 1 --2- 172.21.15.78:0/2897399553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827d8020 0x55d0827e1a00 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f04780c9cf0 tx=0x7f0478099510 comp rx=0 tx=0).stop 2024-03-14T10:02:24.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.079+0000 7f04a0735b80 1 -- 172.21.15.78:0/2897399553 shutdown_connections 2024-03-14T10:02:24.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.079+0000 7f04a0735b80 1 --2- 172.21.15.78:0/2897399553 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827d7700 0x55d0827d7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.079+0000 7f04a0735b80 1 --2- 172.21.15.78:0/2897399553 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827e1f40 0x55d0827e4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.079+0000 7f04a0735b80 1 --2- 172.21.15.78:0/2897399553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827d8020 0x55d0827e1a00 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.079+0000 7f04a0735b80 1 -- 172.21.15.78:0/2897399553 >> 172.21.15.78:0/2897399553 conn(0x55d08274d2a0 msgr2=0x55d0827d2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:24.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.079+0000 7f04a0735b80 1 -- 172.21.15.78:0/2897399553 shutdown_connections 2024-03-14T10:02:24.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.079+0000 7f04a0735b80 1 -- 172.21.15.78:0/2897399553 wait complete. 2024-03-14T10:02:24.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.080+0000 7f04a0735b80 1 -- start start 2024-03-14T10:02:24.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.080+0000 7f04a0735b80 1 Processor -- start 2024-03-14T10:02:24.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.080+0000 7f04a0735b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827d7700 0x55d0827e1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.080+0000 7f04a0735b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827d8020 0x55d0827dec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.080+0000 7f04a0735b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827e1f40 0x55d0827df3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.080+0000 7f04a0735b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0826c5e00 con 0x55d0827e1f40 2024-03-14T10:02:24.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.080+0000 7f04a0735b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0826c5aa0 con 0x55d0827d8020 2024-03-14T10:02:24.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.080+0000 7f04a0735b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0825ad720 con 0x55d0827d7700 2024-03-14T10:02:24.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.081+0000 7f04a0735b80 10 client.0 _ll_register_callbacks cb 0x55d082531420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:24.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.081+0000 7f0486800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827e1f40 0x55d0827df3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.081+0000 7f0486800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827e1f40 0x55d0827df3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57720/0 (socket says 172.21.15.78:57720) 2024-03-14T10:02:24.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.081+0000 7f0486800700 1 -- 172.21.15.78:0/3709708591 learned_addr learned my addr 172.21.15.78:0/3709708591 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:24.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.081+0000 7f0485e00700 1 --2- 172.21.15.78:0/3709708591 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827d7700 0x55d0827e1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.081+0000 7f0485400700 1 --2- 172.21.15.78:0/3709708591 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827d8020 0x55d0827dec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.081+0000 7f0485e00700 1 -- 172.21.15.78:0/3709708591 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827d8020 msgr2=0x55d0827dec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:24.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.081+0000 7f0485e00700 1 --2- 172.21.15.78:0/3709708591 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827d8020 0x55d0827dec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.081+0000 7f0485e00700 1 -- 172.21.15.78:0/3709708591 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827e1f40 msgr2=0x55d0827df3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:24.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.081+0000 7f0485e00700 1 --2- 172.21.15.78:0/3709708591 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827e1f40 0x55d0827df3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.081+0000 7f0485e00700 1 -- 172.21.15.78:0/3709708591 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0474009c90 con 0x55d0827d7700 2024-03-14T10:02:24.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.081+0000 7f0486800700 1 --2- 172.21.15.78:0/3709708591 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827e1f40 0x55d0827df3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:24.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.081+0000 7f0485e00700 1 --2- 172.21.15.78:0/3709708591 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827d7700 0x55d0827e1980 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0478099b10 tx=0x7f0478005bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=813e41fd37c5da3f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:24.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.081+0000 7f0467400700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:02:24.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.082+0000 7f0467400700 1 -- 172.21.15.78:0/3709708591 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047809d780 con 0x55d0827d7700 2024-03-14T10:02:24.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.082+0000 7f0467400700 1 -- 172.21.15.78:0/3709708591 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f0478099950 con 0x55d0827d7700 2024-03-14T10:02:24.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.082+0000 7f04a0735b80 1 -- 172.21.15.78:0/3709708591 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d0827ca7a0 con 0x55d0827d7700 2024-03-14T10:02:24.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.082+0000 7f0467400700 1 -- 172.21.15.78:0/3709708591 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04780a1dd0 con 0x55d0827d7700 2024-03-14T10:02:24.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.082+0000 7f04a0735b80 10 client.27313 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:24.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.082+0000 7f04a0735b80 1 -- 172.21.15.78:0/3709708591 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f04881a65a0 con 0x55d0827d7700 2024-03-14T10:02:24.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.082+0000 7f0467400700 1 -- 172.21.15.78:0/3709708591 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d0827ca7a0 con 0x55d0827d7700 2024-03-14T10:02:24.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.082+0000 7f0467400700 1 -- 172.21.15.78:0/3709708591 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f04780b9100 con 0x55d0827d7700 2024-03-14T10:02:24.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.082+0000 7f04a0735b80 10 client.27313 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:24.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.082+0000 7f04a0735b80 10 client.27313 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:02:24.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.082+0000 7f04a0735b80 20 client.27313 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:24.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.082+0000 7f04a0735b80 1 --2- 172.21.15.78:0/3709708591 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d0827f17e0 0x55d0827f3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.082+0000 7f04a0735b80 4 client.27313 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:02:24.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.082+0000 7f04a0735b80 1 -- 172.21.15.78:0/3709708591 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d082722780 con 0x55d0827f17e0 2024-03-14T10:02:24.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.083+0000 7f0485400700 1 --2- 172.21.15.78:0/3709708591 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d0827f17e0 0x55d0827f3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.083+0000 7f0467400700 10 client.27313 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:24.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.083+0000 7f0485400700 1 --2- 172.21.15.78:0/3709708591 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d0827f17e0 0x55d0827f3c00 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46a2b77faef09868 server_cookie=c53fe13e34bd30bd in_seq=0 out_seq=0 2024-03-14T10:02:24.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.084+0000 7f0467400700 1 -- 172.21.15.78:0/3709708591 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55d082722780 con 0x55d0827f17e0 2024-03-14T10:02:24.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.084+0000 7f0467400700 10 client.27313 handle_command_reply: tid=0 2024-03-14T10:02:24.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.084+0000 7f04a0735b80 1 client.27313 shutdown 2024-03-14T10:02:24.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.084+0000 7f04a0735b80 1 -- 172.21.15.78:0/3709708591 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827d7700 msgr2=0x55d0827e1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.084+0000 7f04a0735b80 1 --2- 172.21.15.78:0/3709708591 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827d7700 0x55d0827e1980 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0478099b10 tx=0x7f0478005bb0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.084+0000 7f04a0735b80 1 -- 172.21.15.78:0/3709708591 shutdown_connections 2024-03-14T10:02:24.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.084+0000 7f04a0735b80 1 --2- 172.21.15.78:0/3709708591 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827e1f40 0x55d0827df3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.084+0000 7f04a0735b80 1 --2- 172.21.15.78:0/3709708591 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827d7700 0x55d0827e1980 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.084+0000 7f04a0735b80 1 --2- 172.21.15.78:0/3709708591 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d0827f17e0 0x55d0827f3c00 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.084+0000 7f04a0735b80 1 --2- 172.21.15.78:0/3709708591 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827d8020 0x55d0827dec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.084+0000 7f0467400700 0 client.27313 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:24.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.084+0000 7f04a0735b80 1 -- 172.21.15.78:0/3709708591 >> 172.21.15.78:0/3709708591 conn(0x55d08274d2a0 msgr2=0x55d0827d2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:24.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.084+0000 7f04a0735b80 1 -- 172.21.15.78:0/3709708591 shutdown_connections 2024-03-14T10:02:24.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.084+0000 7f04a0735b80 1 -- 172.21.15.78:0/3709708591 wait complete. 2024-03-14T10:02:24.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.085+0000 7f04a0735b80 20 client.27313 trim_cache size 0 max 16384 2024-03-14T10:02:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.101+0000 7f04a0735b80 1 Processor -- start 2024-03-14T10:02:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.101+0000 7f04a0735b80 1 -- start start 2024-03-14T10:02:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.102+0000 7f04a0735b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827d7700 0x55d0827d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.102+0000 7f04a0735b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827d8020 0x55d0827e8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.102+0000 7f04a0735b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827e1f40 0x55d0827ed180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.102+0000 7f04a0735b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0827dfde0 con 0x55d0827d8020 2024-03-14T10:02:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.102+0000 7f04a0735b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d082722780 con 0x55d0827e1f40 2024-03-14T10:02:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.102+0000 7f04a0735b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0827dfc70 con 0x55d0827d7700 2024-03-14T10:02:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.102+0000 7f0485400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827d8020 0x55d0827e8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.102+0000 7f0485400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827d8020 0x55d0827e8bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57734/0 (socket says 172.21.15.78:57734) 2024-03-14T10:02:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.102+0000 7f0485400700 1 -- 172.21.15.78:0/1655325836 learned_addr learned my addr 172.21.15.78:0/1655325836 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.102+0000 7f0485e00700 1 --2- 172.21.15.78:0/1655325836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827d7700 0x55d0827d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.102+0000 7f0486800700 1 --2- 172.21.15.78:0/1655325836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827e1f40 0x55d0827ed180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.102+0000 7f0485400700 1 -- 172.21.15.78:0/1655325836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827d7700 msgr2=0x55d0827d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.102+0000 7f0485400700 1 --2- 172.21.15.78:0/1655325836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827d7700 0x55d0827d2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.102+0000 7f0485400700 1 -- 172.21.15.78:0/1655325836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827e1f40 msgr2=0x55d0827ed180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.102+0000 7f0485400700 1 --2- 172.21.15.78:0/1655325836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827e1f40 0x55d0827ed180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.102+0000 7f0485400700 1 -- 172.21.15.78:0/1655325836 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0827ce9d0 con 0x55d0827d8020 2024-03-14T10:02:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.103+0000 7f0485400700 1 --2- 172.21.15.78:0/1655325836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827d8020 0x55d0827e8bb0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x55d0827e01b0 tx=0x7f0474009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e404a8624da571 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:24.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.103+0000 7f0473a00700 1 -- 172.21.15.78:0/1655325836 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0474004830 con 0x55d0827d8020 2024-03-14T10:02:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.103+0000 7f0473a00700 1 -- 172.21.15.78:0/1655325836 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0474018040 con 0x55d0827d8020 2024-03-14T10:02:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.103+0000 7f0473a00700 1 -- 172.21.15.78:0/1655325836 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0474005280 con 0x55d0827d8020 2024-03-14T10:02:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.104+0000 7f04a0735b80 1 -- 172.21.15.78:0/1655325836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827d8020 msgr2=0x55d0827e8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.104+0000 7f04a0735b80 1 --2- 172.21.15.78:0/1655325836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827d8020 0x55d0827e8bb0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x55d0827e01b0 tx=0x7f0474009260 comp rx=0 tx=0).stop 2024-03-14T10:02:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.104+0000 7f04a0735b80 1 -- 172.21.15.78:0/1655325836 shutdown_connections 2024-03-14T10:02:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.104+0000 7f04a0735b80 1 --2- 172.21.15.78:0/1655325836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827e1f40 0x55d0827ed180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.104+0000 7f04a0735b80 1 --2- 172.21.15.78:0/1655325836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827d7700 0x55d0827d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.104+0000 7f04a0735b80 1 --2- 172.21.15.78:0/1655325836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827d8020 0x55d0827e8bb0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.104+0000 7f04a0735b80 1 -- 172.21.15.78:0/1655325836 >> 172.21.15.78:0/1655325836 conn(0x55d08274d2a0 msgr2=0x55d0827d9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.104+0000 7f04a0735b80 1 -- 172.21.15.78:0/1655325836 shutdown_connections 2024-03-14T10:02:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.104+0000 7f04a0735b80 1 -- 172.21.15.78:0/1655325836 wait complete. 2024-03-14T10:02:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.104+0000 7f04a0735b80 1 -- start start 2024-03-14T10:02:24.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.105+0000 7f04a0735b80 1 Processor -- start 2024-03-14T10:02:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.105+0000 7f04a0735b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827d7700 0x55d0827e7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.105+0000 7f04a0735b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827d8020 0x55d0827e7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.105+0000 7f04a0735b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827e1f40 0x55d08283d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.105+0000 7f04a0735b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d08283d850 con 0x55d0827e1f40 2024-03-14T10:02:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.105+0000 7f04a0735b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d08283d9c0 con 0x55d0827d8020 2024-03-14T10:02:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.105+0000 7f04a0735b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d08283db80 con 0x55d0827d7700 2024-03-14T10:02:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.105+0000 7f04a0735b80 10 client.0 _ll_register_callbacks cb 0x7f0478005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:24.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.106+0000 7f0486800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827e1f40 0x55d08283d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.106+0000 7f0486800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827e1f40 0x55d08283d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57746/0 (socket says 172.21.15.78:57746) 2024-03-14T10:02:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.106+0000 7f0485e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827d7700 0x55d0827e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.106+0000 7f0486800700 1 -- 172.21.15.78:0/1773120447 learned_addr learned my addr 172.21.15.78:0/1773120447 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.106+0000 7f0485400700 1 --2- 172.21.15.78:0/1773120447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827d8020 0x55d0827e7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.106+0000 7f0486800700 1 -- 172.21.15.78:0/1773120447 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827d7700 msgr2=0x55d0827e7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.106+0000 7f0486800700 1 --2- 172.21.15.78:0/1773120447 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827d7700 0x55d0827e7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.106+0000 7f0486800700 1 -- 172.21.15.78:0/1773120447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827d8020 msgr2=0x55d0827e7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.106+0000 7f0486800700 1 --2- 172.21.15.78:0/1773120447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827d8020 0x55d0827e7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.106+0000 7f0486800700 1 -- 172.21.15.78:0/1773120447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0474004260 con 0x55d0827e1f40 2024-03-14T10:02:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.106+0000 7f0486800700 1 --2- 172.21.15.78:0/1773120447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827e1f40 0x55d08283d310 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f048000b0a0 tx=0x7f04800099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f469ca9cc960187 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.106+0000 7f0467400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.106+0000 7f0467400700 1 -- 172.21.15.78:0/1773120447 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048000e040 con 0x55d0827e1f40 2024-03-14T10:02:24.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.107+0000 7f04a0735b80 1 -- 172.21.15.78:0/1773120447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f047809a050 con 0x55d0827e1f40 2024-03-14T10:02:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.107+0000 7f0467400700 1 -- 172.21.15.78:0/1773120447 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f0480005040 con 0x55d0827e1f40 2024-03-14T10:02:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.107+0000 7f0467400700 1 -- 172.21.15.78:0/1773120447 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0480007950 con 0x55d0827e1f40 2024-03-14T10:02:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.107+0000 7f04a0735b80 10 client.17268 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.107+0000 7f04a0735b80 1 -- 172.21.15.78:0/1773120447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d082557be0 con 0x55d0827e1f40 2024-03-14T10:02:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.107+0000 7f0467400700 1 -- 172.21.15.78:0/1773120447 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f047809a050 con 0x55d0827e1f40 2024-03-14T10:02:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.107+0000 7f0467400700 1 -- 172.21.15.78:0/1773120447 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f048002c040 con 0x55d0827e1f40 2024-03-14T10:02:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.107+0000 7f04a0735b80 10 client.17268 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.107+0000 7f04a0735b80 10 client.17268 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:02:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.107+0000 7f04a0735b80 20 client.17268 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.107+0000 7f04a0735b80 1 --2- 172.21.15.78:0/1773120447 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d082845520 0x55d082847940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.107+0000 7f04a0735b80 4 client.17268 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T10:02:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.107+0000 7f04a0735b80 1 -- 172.21.15.78:0/1773120447 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d082578d70 con 0x55d082845520 2024-03-14T10:02:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.108+0000 7f0485e00700 1 --2- 172.21.15.78:0/1773120447 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d082845520 0x55d082847940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.108+0000 7f0467400700 10 client.17268 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:24.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.108+0000 7f0485e00700 1 --2- 172.21.15.78:0/1773120447 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d082845520 0x55d082847940 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a43225411fa9d86 server_cookie=1a6eea6de6066a90 in_seq=0 out_seq=0 2024-03-14T10:02:24.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.109+0000 7f0467400700 1 -- 172.21.15.78:0/1773120447 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d082578d70 con 0x55d082845520 2024-03-14T10:02:24.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.109+0000 7f0467400700 10 client.17268 handle_command_reply: tid=0 2024-03-14T10:02:24.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.109+0000 7f04a0735b80 1 client.17268 shutdown 2024-03-14T10:02:24.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.109+0000 7f04a0735b80 1 -- 172.21.15.78:0/1773120447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827e1f40 msgr2=0x55d08283d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.109+0000 7f04a0735b80 1 --2- 172.21.15.78:0/1773120447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827e1f40 0x55d08283d310 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f048000b0a0 tx=0x7f04800099e0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.109+0000 7f04a0735b80 1 -- 172.21.15.78:0/1773120447 shutdown_connections 2024-03-14T10:02:24.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.109+0000 7f04a0735b80 1 --2- 172.21.15.78:0/1773120447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0827e1f40 0x55d08283d310 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.110+0000 7f04a0735b80 1 --2- 172.21.15.78:0/1773120447 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55d0827d7700 0x55d0827e7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.110+0000 7f04a0735b80 1 --2- 172.21.15.78:0/1773120447 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55d082845520 0x55d082847940 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.110+0000 7f04a0735b80 1 --2- 172.21.15.78:0/1773120447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d0827d8020 0x55d0827e7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.110+0000 7f04a0735b80 1 -- 172.21.15.78:0/1773120447 >> 172.21.15.78:0/1773120447 conn(0x55d08274d2a0 msgr2=0x55d0827ed180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:24.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.110+0000 7f0467400700 0 client.17268 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:24.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.110+0000 7f04a0735b80 1 -- 172.21.15.78:0/1773120447 shutdown_connections 2024-03-14T10:02:24.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.110+0000 7f04a0735b80 1 -- 172.21.15.78:0/1773120447 wait complete. 2024-03-14T10:02:24.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.110+0000 7f04a0735b80 20 client.17268 trim_cache size 0 max 16384 2024-03-14T10:02:24.114 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:02:24.114 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T10:02:24.114 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "4ef8a118-9588-45bd-a570-c66453eee760", 2024-03-14T10:02:24.114 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T10:02:24.114 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:02:24.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.110+0000 7f048cc00700 1 -- 172.21.15.78:0/924465247 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f046c0a0c00 msgr2=0x7f046c0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.110+0000 7f048cc00700 1 --2- 172.21.15.78:0/924465247 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f046c0a0c00 0x7f046c0a30c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0480009040 tx=0x7f0480006230 comp rx=0 tx=0).stop 2024-03-14T10:02:24.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.110+0000 7f048cc00700 1 -- 172.21.15.78:0/924465247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04880731e0 msgr2=0x7f048819f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.110+0000 7f048cc00700 1 --2- 172.21.15.78:0/924465247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04880731e0 0x7f048819f5b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f047800e9a0 tx=0x7f047800ecb0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.111+0000 7f048cc00700 1 -- 172.21.15.78:0/924465247 shutdown_connections 2024-03-14T10:02:24.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.111+0000 7f048cc00700 1 --2- 172.21.15.78:0/924465247 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f048810d4f0 0x7f0488199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.111+0000 7f048cc00700 1 --2- 172.21.15.78:0/924465247 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f04880731e0 0x7f048819f5b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.111+0000 7f048cc00700 1 --2- 172.21.15.78:0/924465247 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f046c0a0c00 0x7f046c0a30c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.111+0000 7f048cc00700 1 --2- 172.21.15.78:0/924465247 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0488073b00 0x7f048819faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.111+0000 7f048cc00700 1 -- 172.21.15.78:0/924465247 >> 172.21.15.78:0/924465247 conn(0x7f04880fc7a0 msgr2=0x7f04880fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:24.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.111+0000 7f048cc00700 1 -- 172.21.15.78:0/924465247 shutdown_connections 2024-03-14T10:02:24.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.111+0000 7f048cc00700 1 -- 172.21.15.78:0/924465247 wait complete. 2024-03-14T10:02:24.130 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:02:24.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.323+0000 7f5bed200700 1 -- 172.21.15.78:0/4125796965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5be8106050 msgr2=0x7f5be810dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.323+0000 7f5bed200700 1 --2- 172.21.15.78:0/4125796965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5be8106050 0x7f5be810dbd0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5be000b0a0 tx=0x7f5be000b3b0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.323+0000 7f5bed200700 1 -- 172.21.15.78:0/4125796965 shutdown_connections 2024-03-14T10:02:24.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.323+0000 7f5bed200700 1 --2- 172.21.15.78:0/4125796965 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5be8106050 0x7f5be810dbd0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.323+0000 7f5bed200700 1 --2- 172.21.15.78:0/4125796965 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5be8100f30 0x7f5be8101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.323+0000 7f5bed200700 1 --2- 172.21.15.78:0/4125796965 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5be81018e0 0x7f5be8105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.323+0000 7f5bed200700 1 -- 172.21.15.78:0/4125796965 >> 172.21.15.78:0/4125796965 conn(0x7f5be80fc7b0 msgr2=0x7f5be80febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:24.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.323+0000 7f5bed200700 1 -- 172.21.15.78:0/4125796965 shutdown_connections 2024-03-14T10:02:24.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.323+0000 7f5bed200700 1 -- 172.21.15.78:0/4125796965 wait complete. 2024-03-14T10:02:24.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.324+0000 7f5bed200700 1 Processor -- start 2024-03-14T10:02:24.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.324+0000 7f5bed200700 1 -- start start 2024-03-14T10:02:24.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5bed200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5be8100f30 0x7f5be819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5bed200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5be8106050 0x7f5be819fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5bed200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5be81997f0 0x7f5be8199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5bed200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be819a270 con 0x7f5be8106050 2024-03-14T10:02:24.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5bed200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be819a3b0 con 0x7f5be8100f30 2024-03-14T10:02:24.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5bed200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be819a570 con 0x7f5be81997f0 2024-03-14T10:02:24.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5be5400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5be8106050 0x7f5be819fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5be5400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5be8106050 0x7f5be819fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57766/0 (socket says 172.21.15.78:57766) 2024-03-14T10:02:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5be5e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5be8100f30 0x7f5be819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5be5400700 1 -- 172.21.15.78:0/3685157953 learned_addr learned my addr 172.21.15.78:0/3685157953 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5be6800700 1 --2- 172.21.15.78:0/3685157953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5be81997f0 0x7f5be8199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5be5400700 1 -- 172.21.15.78:0/3685157953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5be81997f0 msgr2=0x7f5be8199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5be5400700 1 --2- 172.21.15.78:0/3685157953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5be81997f0 0x7f5be8199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5be5400700 1 -- 172.21.15.78:0/3685157953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5be8100f30 msgr2=0x7f5be819f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5be5400700 1 --2- 172.21.15.78:0/3685157953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5be8100f30 0x7f5be819f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5be5400700 1 -- 172.21.15.78:0/3685157953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be0009d00 con 0x7f5be8106050 2024-03-14T10:02:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.325+0000 7f5be5e00700 1 --2- 172.21.15.78:0/3685157953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5be8100f30 0x7f5be819f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.326+0000 7f5be5400700 1 --2- 172.21.15.78:0/3685157953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5be8106050 0x7f5be819fc00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f5bd400e970 tx=0x7f5bd400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.326+0000 7f5bdea00700 1 -- 172.21.15.78:0/3685157953 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4009790 con 0x7f5be8106050 2024-03-14T10:02:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.326+0000 7f5bdea00700 1 -- 172.21.15.78:0/3685157953 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5bd400c7e0 con 0x7f5be8106050 2024-03-14T10:02:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.326+0000 7f5bed200700 1 -- 172.21.15.78:0/3685157953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be81a6530 con 0x7f5be8106050 2024-03-14T10:02:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.326+0000 7f5bdea00700 1 -- 172.21.15.78:0/3685157953 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd40054e0 con 0x7f5be8106050 2024-03-14T10:02:24.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.326+0000 7f5bed200700 1 -- 172.21.15.78:0/3685157953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be81a6a40 con 0x7f5be8106050 2024-03-14T10:02:24.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.330+0000 7f5bdea00700 1 -- 172.21.15.78:0/3685157953 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f5bd400c950 con 0x7f5be8106050 2024-03-14T10:02:24.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.332+0000 7f5c00cafb80 1 Processor -- start 2024-03-14T10:02:24.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.332+0000 7f5c00cafb80 1 -- start start 2024-03-14T10:02:24.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.332+0000 7f5bdea00700 1 --2- 172.21.15.78:0/3685157953 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5bcc0a0c00 0x7f5bcc0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.332+0000 7f5bdea00700 1 -- 172.21.15.78:0/3685157953 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f5bd4014070 con 0x7f5be8106050 2024-03-14T10:02:24.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.332+0000 7f5c00cafb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a70680 0x55f7c5a70a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.332+0000 7f5c00cafb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a7a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.332+0000 7f5c00cafb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a7d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.332+0000 7f5c00cafb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7c59c9990 con 0x55f7c5a7aec0 2024-03-14T10:02:24.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.332+0000 7f5c00cafb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7c59c7350 con 0x55f7c5a70fa0 2024-03-14T10:02:24.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.332+0000 7f5c00cafb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7c59c2c40 con 0x55f7c5a70680 2024-03-14T10:02:24.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.332+0000 7f5be6800700 1 --2- 172.21.15.78:0/3685157953 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5bcc0a0c00 0x7f5bcc0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.332+0000 7f5be6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a7d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.332+0000 7f5be6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a7d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57778/0 (socket says 172.21.15.78:57778) 2024-03-14T10:02:24.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.332+0000 7f5be5e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a70680 0x55f7c5a70a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.332+0000 7f5be6800700 1 -- 172.21.15.78:0/3062593520 learned_addr learned my addr 172.21.15.78:0/3062593520 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:24.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.332+0000 7f5be5400700 1 --2- 172.21.15.78:0/3062593520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a7a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.333+0000 7f5be5e00700 1 -- 172.21.15.78:0/3062593520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a70fa0 msgr2=0x55f7c5a7a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:24.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.333+0000 7f5be5e00700 1 --2- 172.21.15.78:0/3062593520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a7a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.333+0000 7f5be5e00700 1 -- 172.21.15.78:0/3062593520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a7aec0 msgr2=0x55f7c5a7d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:24.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.333+0000 7f5be5e00700 1 --2- 172.21.15.78:0/3062593520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a7d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.333+0000 7f5be5e00700 1 -- 172.21.15.78:0/3062593520 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7c58443b0 con 0x55f7c5a70680 2024-03-14T10:02:24.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.333+0000 7f5be5400700 1 --2- 172.21.15.78:0/3062593520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a7a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:24.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.333+0000 7f5be6800700 1 --2- 172.21.15.78:0/3685157953 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5bcc0a0c00 0x7f5bcc0a30c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5be0009610 tx=0x7f5be00094e0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:24.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.333+0000 7f5be5e00700 1 --2- 172.21.15.78:0/3062593520 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a70680 0x55f7c5a70a60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80114f0 tx=0x7f5bd8011800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d145717407248eae server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:24.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.333+0000 7f5bd3400700 1 -- 172.21.15.78:0/3062593520 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8004230 con 0x55f7c5a70680 2024-03-14T10:02:24.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.333+0000 7f5bd3400700 1 -- 172.21.15.78:0/3062593520 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5bd8004cf0 con 0x55f7c5a70680 2024-03-14T10:02:24.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.333+0000 7f5bd3400700 1 -- 172.21.15.78:0/3062593520 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8005630 con 0x55f7c5a70680 2024-03-14T10:02:24.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.334+0000 7f5c00cafb80 1 -- 172.21.15.78:0/3062593520 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a70680 msgr2=0x55f7c5a70a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.334+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/3062593520 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a70680 0x55f7c5a70a60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80114f0 tx=0x7f5bd8011800 comp rx=0 tx=0).stop 2024-03-14T10:02:24.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.334+0000 7f5c00cafb80 1 -- 172.21.15.78:0/3062593520 shutdown_connections 2024-03-14T10:02:24.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.334+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/3062593520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a7d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.334+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/3062593520 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a70680 0x55f7c5a70a60 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.334+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/3062593520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a7a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.334+0000 7f5c00cafb80 1 -- 172.21.15.78:0/3062593520 >> 172.21.15.78:0/3062593520 conn(0x55f7c59e6220 msgr2=0x55f7c5a6baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:24.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.334+0000 7f5c00cafb80 1 -- 172.21.15.78:0/3062593520 shutdown_connections 2024-03-14T10:02:24.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.334+0000 7f5c00cafb80 1 -- 172.21.15.78:0/3062593520 wait complete. 2024-03-14T10:02:24.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.335+0000 7f5c00cafb80 1 -- start start 2024-03-14T10:02:24.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.335+0000 7f5c00cafb80 1 Processor -- start 2024-03-14T10:02:24.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.335+0000 7f5c00cafb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a70680 0x55f7c5a7a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.335+0000 7f5c00cafb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a77b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.335+0000 7f5c00cafb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a78180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.335+0000 7f5c00cafb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7c595ed80 con 0x55f7c5a7aec0 2024-03-14T10:02:24.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.335+0000 7f5c00cafb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7c595ea20 con 0x55f7c5a70fa0 2024-03-14T10:02:24.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.335+0000 7f5c00cafb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7c58466a0 con 0x55f7c5a70680 2024-03-14T10:02:24.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.336+0000 7f5c00cafb80 10 client.0 _ll_register_callbacks cb 0x55f7c57ca3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:24.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.336+0000 7f5be6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a78180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.336+0000 7f5be6800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a78180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57788/0 (socket says 172.21.15.78:57788) 2024-03-14T10:02:24.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.336+0000 7f5be5400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a77b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.336+0000 7f5be6800700 1 -- 172.21.15.78:0/210326923 learned_addr learned my addr 172.21.15.78:0/210326923 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:24.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.336+0000 7f5be5e00700 1 --2- 172.21.15.78:0/210326923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a70680 0x55f7c5a7a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.336+0000 7f5be6800700 1 -- 172.21.15.78:0/210326923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a70680 msgr2=0x55f7c5a7a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:24.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.336+0000 7f5be6800700 1 --2- 172.21.15.78:0/210326923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a70680 0x55f7c5a7a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.336+0000 7f5be6800700 1 -- 172.21.15.78:0/210326923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a70fa0 msgr2=0x55f7c5a77b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:24.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.336+0000 7f5be6800700 1 --2- 172.21.15.78:0/210326923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a77b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.336+0000 7f5be6800700 1 -- 172.21.15.78:0/210326923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be81a6a40 con 0x55f7c5a7aec0 2024-03-14T10:02:24.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.336+0000 7f5be6800700 1 --2- 172.21.15.78:0/210326923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a78180 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f5be0009480 tx=0x7f5be0006210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19686f1149066b46 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:24.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.336+0000 7f5bc7e00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:24.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.337+0000 7f5bc7e00700 1 -- 172.21.15.78:0/210326923 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be0025070 con 0x55f7c5a7aec0 2024-03-14T10:02:24.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.337+0000 7f5bc7e00700 1 -- 172.21.15.78:0/210326923 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f5be0003c10 con 0x55f7c5a7aec0 2024-03-14T10:02:24.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.337+0000 7f5c00cafb80 1 -- 172.21.15.78:0/210326923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f7c5a63720 con 0x55f7c5a7aec0 2024-03-14T10:02:24.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.337+0000 7f5bc7e00700 1 -- 172.21.15.78:0/210326923 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5be0003dd0 con 0x55f7c5a7aec0 2024-03-14T10:02:24.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.337+0000 7f5c00cafb80 10 client.17280 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:24.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.337+0000 7f5c00cafb80 1 -- 172.21.15.78:0/210326923 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f5bd8009c90 con 0x55f7c5a7aec0 2024-03-14T10:02:24.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.337+0000 7f5bc7e00700 1 -- 172.21.15.78:0/210326923 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f7c5a63720 con 0x55f7c5a7aec0 2024-03-14T10:02:24.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.337+0000 7f5bc7e00700 1 -- 172.21.15.78:0/210326923 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5be002d430 con 0x55f7c5a7aec0 2024-03-14T10:02:24.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.337+0000 7f5c00cafb80 10 client.17280 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:24.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.337+0000 7f5c00cafb80 10 client.17280 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:02:24.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.337+0000 7f5c00cafb80 20 client.17280 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:24.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.337+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/210326923 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55f7c5a8aab0 0x55f7c5a8ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.337+0000 7f5c00cafb80 4 client.17280 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:02:24.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.337+0000 7f5c00cafb80 1 -- 172.21.15.78:0/210326923 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f7c59bb700 con 0x55f7c5a8aab0 2024-03-14T10:02:24.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.338+0000 7f5be5e00700 1 --2- 172.21.15.78:0/210326923 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55f7c5a8aab0 0x55f7c5a8ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.338+0000 7f5be5e00700 1 --2- 172.21.15.78:0/210326923 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55f7c5a8aab0 0x55f7c5a8ced0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c931f5d939427a97 server_cookie=c67d93bc22e9446f in_seq=0 out_seq=0 2024-03-14T10:02:24.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.338+0000 7f5bc7e00700 10 client.17280 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:24.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.339+0000 7f5bc7e00700 1 -- 172.21.15.78:0/210326923 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55f7c59bb700 con 0x55f7c5a8aab0 2024-03-14T10:02:24.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.339+0000 7f5bc7e00700 10 client.17280 handle_command_reply: tid=0 2024-03-14T10:02:24.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.339+0000 7f5c00cafb80 1 client.17280 shutdown 2024-03-14T10:02:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.339+0000 7f5c00cafb80 1 -- 172.21.15.78:0/210326923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a7aec0 msgr2=0x55f7c5a78180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.339+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/210326923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a78180 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f5be0009480 tx=0x7f5be0006210 comp rx=0 tx=0).stop 2024-03-14T10:02:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.339+0000 7f5c00cafb80 1 -- 172.21.15.78:0/210326923 shutdown_connections 2024-03-14T10:02:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.339+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/210326923 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a78180 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.339+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/210326923 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a70680 0x55f7c5a7a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.339+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/210326923 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55f7c5a8aab0 0x55f7c5a8ced0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.339+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/210326923 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a77b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.339+0000 7f5bc7e00700 0 client.17280 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.339+0000 7f5c00cafb80 1 -- 172.21.15.78:0/210326923 >> 172.21.15.78:0/210326923 conn(0x55f7c59e6220 msgr2=0x55f7c5a6b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.339+0000 7f5c00cafb80 1 -- 172.21.15.78:0/210326923 shutdown_connections 2024-03-14T10:02:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.340+0000 7f5c00cafb80 1 -- 172.21.15.78:0/210326923 wait complete. 2024-03-14T10:02:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.340+0000 7f5c00cafb80 20 client.17280 trim_cache size 0 max 16384 2024-03-14T10:02:24.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.356+0000 7f5c00cafb80 1 Processor -- start 2024-03-14T10:02:24.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.357+0000 7f5c00cafb80 1 -- start start 2024-03-14T10:02:24.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.357+0000 7f5c00cafb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a70680 0x55f7c5a86830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.357+0000 7f5c00cafb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a7a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.357+0000 7f5c00cafb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a74e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.357+0000 7f5c00cafb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7c5a79170 con 0x55f7c5a70680 2024-03-14T10:02:24.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.357+0000 7f5c00cafb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7c5a6c670 con 0x55f7c5a7aec0 2024-03-14T10:02:24.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.357+0000 7f5c00cafb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7c5a6ce30 con 0x55f7c5a70fa0 2024-03-14T10:02:24.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.357+0000 7f5be5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a70680 0x55f7c5a86830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.357+0000 7f5be5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a70680 0x55f7c5a86830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57790/0 (socket says 172.21.15.78:57790) 2024-03-14T10:02:24.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.357+0000 7f5be5e00700 1 -- 172.21.15.78:0/3553178268 learned_addr learned my addr 172.21.15.78:0/3553178268 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.357+0000 7f5be5400700 1 --2- 172.21.15.78:0/3553178268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a7a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.357+0000 7f5be6800700 1 --2- 172.21.15.78:0/3553178268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a74e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.358+0000 7f5be5e00700 1 -- 172.21.15.78:0/3553178268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a70fa0 msgr2=0x55f7c5a7a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.358+0000 7f5be5e00700 1 --2- 172.21.15.78:0/3553178268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a7a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.358+0000 7f5be5e00700 1 -- 172.21.15.78:0/3553178268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a7aec0 msgr2=0x55f7c5a74e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.358+0000 7f5be5e00700 1 --2- 172.21.15.78:0/3553178268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a74e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.358+0000 7f5be5e00700 1 -- 172.21.15.78:0/3553178268 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7c57f0b60 con 0x55f7c5a70680 2024-03-14T10:02:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.358+0000 7f5be6800700 1 --2- 172.21.15.78:0/3553178268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a74e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:02:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.358+0000 7f5be5e00700 1 --2- 172.21.15.78:0/3553178268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a70680 0x55f7c5a86830 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80114c0 tx=0x7f5bd8009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d62548b6fb084ad server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.358+0000 7f5bd3e00700 1 -- 172.21.15.78:0/3553178268 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd802a3d0 con 0x55f7c5a70680 2024-03-14T10:02:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.358+0000 7f5bd3e00700 1 -- 172.21.15.78:0/3553178268 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5bd8005040 con 0x55f7c5a70680 2024-03-14T10:02:24.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.358+0000 7f5bd3e00700 1 -- 172.21.15.78:0/3553178268 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd802acf0 con 0x55f7c5a70680 2024-03-14T10:02:24.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.359+0000 7f5c00cafb80 1 -- 172.21.15.78:0/3553178268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a70680 msgr2=0x55f7c5a86830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.359+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/3553178268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a70680 0x55f7c5a86830 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5bd80114c0 tx=0x7f5bd8009f60 comp rx=0 tx=0).stop 2024-03-14T10:02:24.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.359+0000 7f5c00cafb80 1 -- 172.21.15.78:0/3553178268 shutdown_connections 2024-03-14T10:02:24.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.359+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/3553178268 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a74e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.359+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/3553178268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a70680 0x55f7c5a86830 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.359+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/3553178268 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a7a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.359+0000 7f5c00cafb80 1 -- 172.21.15.78:0/3553178268 >> 172.21.15.78:0/3553178268 conn(0x55f7c59e6220 msgr2=0x55f7c5a86120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:24.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.359+0000 7f5c00cafb80 1 -- 172.21.15.78:0/3553178268 shutdown_connections 2024-03-14T10:02:24.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.359+0000 7f5c00cafb80 1 -- 172.21.15.78:0/3553178268 wait complete. 2024-03-14T10:02:24.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.359+0000 7f5c00cafb80 1 -- start start 2024-03-14T10:02:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.359+0000 7f5c00cafb80 1 Processor -- start 2024-03-14T10:02:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.360+0000 7f5c00cafb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a70680 0x55f7c5a81ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.360+0000 7f5c00cafb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a80150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.360+0000 7f5c00cafb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a80840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.360+0000 7f5c00cafb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7c5a80f80 con 0x55f7c5a70680 2024-03-14T10:02:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.360+0000 7f5c00cafb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7c5adc5e0 con 0x55f7c5a70fa0 2024-03-14T10:02:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.360+0000 7f5c00cafb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7c5adc720 con 0x55f7c5a7aec0 2024-03-14T10:02:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.360+0000 7f5c00cafb80 10 client.0 _ll_register_callbacks cb 0x55f7c57ca3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.360+0000 7f5be5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a70680 0x55f7c5a81ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.360+0000 7f5be5e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a70680 0x55f7c5a81ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:57806/0 (socket says 172.21.15.78:57806) 2024-03-14T10:02:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.360+0000 7f5be5e00700 1 -- 172.21.15.78:0/1897962525 learned_addr learned my addr 172.21.15.78:0/1897962525 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.360+0000 7f5be5400700 1 --2- 172.21.15.78:0/1897962525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a80150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.360+0000 7f5be6800700 1 --2- 172.21.15.78:0/1897962525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a80840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.360+0000 7f5be5e00700 1 -- 172.21.15.78:0/1897962525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a7aec0 msgr2=0x55f7c5a80840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:24.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.360+0000 7f5be5e00700 1 --2- 172.21.15.78:0/1897962525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a80840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.361+0000 7f5be5e00700 1 -- 172.21.15.78:0/1897962525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a70fa0 msgr2=0x55f7c5a80150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:24.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.361+0000 7f5be5e00700 1 --2- 172.21.15.78:0/1897962525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a80150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.361+0000 7f5be5e00700 1 -- 172.21.15.78:0/1897962525 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f7c58443b0 con 0x55f7c5a70680 2024-03-14T10:02:24.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.361+0000 7f5be5e00700 1 --2- 172.21.15.78:0/1897962525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a70680 0x55f7c5a81ba0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5bd802a340 tx=0x7f5bd802a370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d3325f4dd3c418 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:24.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.361+0000 7f5bc7e00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:24.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.361+0000 7f5bc7e00700 1 -- 172.21.15.78:0/1897962525 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8005440 con 0x55f7c5a70680 2024-03-14T10:02:24.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.361+0000 7f5bc7e00700 1 -- 172.21.15.78:0/1897962525 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f5bd80055a0 con 0x55f7c5a70680 2024-03-14T10:02:24.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.361+0000 7f5c00cafb80 1 -- 172.21.15.78:0/1897962525 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5bd8004cf0 con 0x55f7c5a70680 2024-03-14T10:02:24.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.361+0000 7f5bc7e00700 1 -- 172.21.15.78:0/1897962525 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5bd8004b60 con 0x55f7c5a70680 2024-03-14T10:02:24.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.361+0000 7f5c00cafb80 10 client.17286 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:24.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.361+0000 7f5c00cafb80 1 -- 172.21.15.78:0/1897962525 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f7c5988ea0 con 0x55f7c5a70680 2024-03-14T10:02:24.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.361+0000 7f5bc7e00700 1 -- 172.21.15.78:0/1897962525 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5bd8004cf0 con 0x55f7c5a70680 2024-03-14T10:02:24.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.362+0000 7f5bc7e00700 1 -- 172.21.15.78:0/1897962525 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5bd803b070 con 0x55f7c5a70680 2024-03-14T10:02:24.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.362+0000 7f5c00cafb80 10 client.17286 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:24.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.362+0000 7f5c00cafb80 10 client.17286 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:02:24.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.362+0000 7f5c00cafb80 20 client.17286 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:24.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.362+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/1897962525 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55f7c5addaf0 0x55f7c5adfee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:24.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.362+0000 7f5c00cafb80 4 client.17286 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:02:24.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.362+0000 7f5c00cafb80 1 -- 172.21.15.78:0/1897962525 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f7c5811cf0 con 0x55f7c5addaf0 2024-03-14T10:02:24.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.362+0000 7f5be5400700 1 --2- 172.21.15.78:0/1897962525 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55f7c5addaf0 0x55f7c5adfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:24.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.362+0000 7f5bc7e00700 10 client.17286 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:24.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.362+0000 7f5be5400700 1 --2- 172.21.15.78:0/1897962525 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55f7c5addaf0 0x55f7c5adfee0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c36c271355455d6f server_cookie=614ce24e7808bf0 in_seq=0 out_seq=0 2024-03-14T10:02:24.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.363+0000 7f5bc7e00700 1 -- 172.21.15.78:0/1897962525 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55f7c5811cf0 con 0x55f7c5addaf0 2024-03-14T10:02:24.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.363+0000 7f5bc7e00700 10 client.17286 handle_command_reply: tid=0 2024-03-14T10:02:24.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.363+0000 7f5c00cafb80 1 client.17286 shutdown 2024-03-14T10:02:24.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.363+0000 7f5c00cafb80 1 -- 172.21.15.78:0/1897962525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a70680 msgr2=0x55f7c5a81ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.363+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/1897962525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a70680 0x55f7c5a81ba0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5bd802a340 tx=0x7f5bd802a370 comp rx=0 tx=0).stop 2024-03-14T10:02:24.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.363+0000 7f5c00cafb80 1 -- 172.21.15.78:0/1897962525 shutdown_connections 2024-03-14T10:02:24.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.363+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/1897962525 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55f7c5a7aec0 0x55f7c5a80840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.363+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/1897962525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f7c5a70680 0x55f7c5a81ba0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.363+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/1897962525 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55f7c5addaf0 0x55f7c5adfee0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.363+0000 7f5c00cafb80 1 --2- 172.21.15.78:0/1897962525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f7c5a70fa0 0x55f7c5a80150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.363+0000 7f5bc7e00700 0 client.17286 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:24.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.363+0000 7f5c00cafb80 1 -- 172.21.15.78:0/1897962525 >> 172.21.15.78:0/1897962525 conn(0x55f7c59e6220 msgr2=0x55f7c5a6fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:24.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.363+0000 7f5c00cafb80 1 -- 172.21.15.78:0/1897962525 shutdown_connections 2024-03-14T10:02:24.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.363+0000 7f5c00cafb80 1 -- 172.21.15.78:0/1897962525 wait complete. 2024-03-14T10:02:24.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.363+0000 7f5c00cafb80 20 client.17286 trim_cache size 0 max 16384 2024-03-14T10:02:24.369 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:02:24.369 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-14T10:02:24.369 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T10:02:24.370 INFO:teuthology.orchestra.run.smithi078.stdout: "4ef8a118-9588-45bd-a570-c66453eee760": { 2024-03-14T10:02:24.370 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T10:02:24.370 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "4ef8a118-9588-45bd-a570-c66453eee760", 2024-03-14T10:02:24.370 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T10:02:24.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:02:24.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:02:24.370 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:02:24.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.364+0000 7f5bed200700 1 -- 172.21.15.78:0/3685157953 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5bcc0a0c00 msgr2=0x7f5bcc0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.364+0000 7f5bed200700 1 --2- 172.21.15.78:0/3685157953 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5bcc0a0c00 0x7f5bcc0a30c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5be0009610 tx=0x7f5be00094e0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.364+0000 7f5bed200700 1 -- 172.21.15.78:0/3685157953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5be8106050 msgr2=0x7f5be819fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:24.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.364+0000 7f5bed200700 1 --2- 172.21.15.78:0/3685157953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5be8106050 0x7f5be819fc00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f5bd400e970 tx=0x7f5bd400ec80 comp rx=0 tx=0).stop 2024-03-14T10:02:24.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.364+0000 7f5bed200700 1 -- 172.21.15.78:0/3685157953 shutdown_connections 2024-03-14T10:02:24.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.364+0000 7f5bed200700 1 --2- 172.21.15.78:0/3685157953 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5bcc0a0c00 0x7f5bcc0a30c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.364+0000 7f5bed200700 1 --2- 172.21.15.78:0/3685157953 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5be8100f30 0x7f5be819f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.364+0000 7f5bed200700 1 --2- 172.21.15.78:0/3685157953 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5be81997f0 0x7f5be8199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.364+0000 7f5bed200700 1 --2- 172.21.15.78:0/3685157953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5be8106050 0x7f5be819fc00 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:24.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.364+0000 7f5bed200700 1 -- 172.21.15.78:0/3685157953 >> 172.21.15.78:0/3685157953 conn(0x7f5be80fc7b0 msgr2=0x7f5be80fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:24.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.364+0000 7f5bed200700 1 -- 172.21.15.78:0/3685157953 shutdown_connections 2024-03-14T10:02:24.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:24.364+0000 7f5bed200700 1 -- 172.21.15.78:0/3685157953 wait complete. 2024-03-14T10:02:24.382 INFO:tasks.cephfs.filesystem:scrub status for tag:4ef8a118-9588-45bd-a570-c66453eee760 - {'path': '/', 'tag': '4ef8a118-9588-45bd-a570-c66453eee760', 'options': 'recursive,force'} 2024-03-14T10:02:25.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:25 smithi152 ceph-mon[48279]: pgmap v254: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-14T10:02:25.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:25 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T10:02:25.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:25 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T10:02:25.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:25 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T10:02:25.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:25 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T10:02:25.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:25 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T10:02:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:25 smithi132 ceph-mon[47959]: pgmap v254: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-14T10:02:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:25 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T10:02:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:25 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T10:02:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:25 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T10:02:25.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:25 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T10:02:25.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:25 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T10:02:25.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:25 smithi078 ceph-mon[38143]: pgmap v254: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-14T10:02:25.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:25 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T10:02:25.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:25 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T10:02:25.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:25 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T10:02:25.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:25 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T10:02:25.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:25 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T10:02:26.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:26 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T10:02:26.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:26 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T10:02:26.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:26 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:02:26.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:26 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T10:02:26.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:26 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T10:02:26.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:26 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:02:26.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:26 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T10:02:26.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:26 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T10:02:26.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:26 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:02:27.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:27 smithi152 ceph-mon[48279]: pgmap v255: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-14T10:02:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:27 smithi132 ceph-mon[47959]: pgmap v255: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-14T10:02:27.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:27 smithi078 ceph-mon[38143]: pgmap v255: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-14T10:02:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:28 smithi152 ceph-mon[48279]: pgmap v256: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-14T10:02:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:28 smithi132 ceph-mon[47959]: pgmap v256: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-14T10:02:29.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:28 smithi078 ceph-mon[38143]: pgmap v256: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-14T10:02:30.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:29 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T10:02:30.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:29 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T10:02:30.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:29 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T10:02:31.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:30 smithi152 ceph-mon[48279]: pgmap v257: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:02:31.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:30 smithi132 ceph-mon[47959]: pgmap v257: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:02:31.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:30 smithi078 ceph-mon[38143]: pgmap v257: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:02:33.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:33 smithi152 ceph-mon[48279]: pgmap v258: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:02:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:33 smithi132 ceph-mon[47959]: pgmap v258: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:02:33.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:33 smithi078 ceph-mon[38143]: pgmap v258: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:02:35.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:35 smithi152 ceph-mon[48279]: pgmap v259: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:02:35.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:35 smithi132 ceph-mon[47959]: pgmap v259: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:02:35.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:35 smithi078 ceph-mon[38143]: pgmap v259: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:02:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:37 smithi152 ceph-mon[48279]: pgmap v260: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:02:37.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:37 smithi132 ceph-mon[47959]: pgmap v260: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:02:37.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:37 smithi078 ceph-mon[38143]: pgmap v260: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:02:39.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:38 smithi152 ceph-mon[48279]: pgmap v261: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:38 smithi132 ceph-mon[47959]: pgmap v261: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:39.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:38 smithi078 ceph-mon[38143]: pgmap v261: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:41 smithi152 ceph-mon[48279]: pgmap v262: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:41 smithi132 ceph-mon[47959]: pgmap v262: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:41.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:41 smithi078 ceph-mon[38143]: pgmap v262: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:43.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:43 smithi152 ceph-mon[48279]: pgmap v263: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:02:43.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:43 smithi132 ceph-mon[47959]: pgmap v263: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:02:43.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:43 smithi078 ceph-mon[38143]: pgmap v263: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:02:45.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:45 smithi152 ceph-mon[48279]: pgmap v264: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:45.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:45 smithi132 ceph-mon[47959]: pgmap v264: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:45.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:45 smithi078 ceph-mon[38143]: pgmap v264: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:47 smithi152 ceph-mon[48279]: pgmap v265: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:02:47.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:47 smithi132 ceph-mon[47959]: pgmap v265: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:02:47.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:47 smithi078 ceph-mon[38143]: pgmap v265: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:02:49.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:48 smithi152 ceph-mon[48279]: pgmap v266: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:49.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:48 smithi132 ceph-mon[47959]: pgmap v266: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:49.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:48 smithi078 ceph-mon[38143]: pgmap v266: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:51 smithi152 ceph-mon[48279]: pgmap v267: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:51.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:51 smithi132 ceph-mon[47959]: pgmap v267: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:51.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:51 smithi078 ceph-mon[38143]: pgmap v267: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:53 smithi152 ceph-mon[48279]: pgmap v268: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:02:53.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:53 smithi132 ceph-mon[47959]: pgmap v268: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:02:53.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:53 smithi078 ceph-mon[38143]: pgmap v268: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:02:54.383 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:02:54.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.599+0000 7f6f67000700 1 -- 172.21.15.78:0/895048936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f601018d0 msgr2=0x7f6f60105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:54.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.599+0000 7f6f67000700 1 --2- 172.21.15.78:0/895048936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f601018d0 0x7f6f60105910 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009990 tx=0x7f6f4c009ca0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.599+0000 7f6f67000700 1 -- 172.21.15.78:0/895048936 shutdown_connections 2024-03-14T10:02:54.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.599+0000 7f6f67000700 1 --2- 172.21.15.78:0/895048936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f60106040 0x7f6f6010dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.599+0000 7f6f67000700 1 --2- 172.21.15.78:0/895048936 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f60100f20 0x7f6f60101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.599+0000 7f6f67000700 1 --2- 172.21.15.78:0/895048936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f601018d0 0x7f6f60105910 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.599+0000 7f6f67000700 1 -- 172.21.15.78:0/895048936 >> 172.21.15.78:0/895048936 conn(0x7f6f600fc7a0 msgr2=0x7f6f600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:54.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.599+0000 7f6f67000700 1 -- 172.21.15.78:0/895048936 shutdown_connections 2024-03-14T10:02:54.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.599+0000 7f6f67000700 1 -- 172.21.15.78:0/895048936 wait complete. 2024-03-14T10:02:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.600+0000 7f6f67000700 1 Processor -- start 2024-03-14T10:02:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.600+0000 7f6f67000700 1 -- start start 2024-03-14T10:02:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.601+0000 7f6f67000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f60100f20 0x7f6f60199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.601+0000 7f6f67000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f60106040 0x7f6f60199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.601+0000 7f6f67000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f6019cfb0 0x7f6f6019d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.601+0000 7f6f67000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6019d9a0 con 0x7f6f6019cfb0 2024-03-14T10:02:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.601+0000 7f6f67000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6019db10 con 0x7f6f60100f20 2024-03-14T10:02:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.601+0000 7f6f67000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6019dcd0 con 0x7f6f60106040 2024-03-14T10:02:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.601+0000 7f6f5fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f60100f20 0x7f6f60199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.601+0000 7f6f5fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f60100f20 0x7f6f60199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:36136/0 (socket says 172.21.15.78:36136) 2024-03-14T10:02:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.601+0000 7f6f5fe00700 1 -- 172.21.15.78:0/1860051081 learned_addr learned my addr 172.21.15.78:0/1860051081 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.601+0000 7f6f5f400700 1 --2- 172.21.15.78:0/1860051081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f60106040 0x7f6f60199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.601+0000 7f6f65000700 1 --2- 172.21.15.78:0/1860051081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f6019cfb0 0x7f6f6019d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.601+0000 7f6f5fe00700 1 -- 172.21.15.78:0/1860051081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f60106040 msgr2=0x7f6f60199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.601+0000 7f6f5fe00700 1 --2- 172.21.15.78:0/1860051081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f60106040 0x7f6f60199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.601+0000 7f6f5fe00700 1 -- 172.21.15.78:0/1860051081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f6019cfb0 msgr2=0x7f6f6019d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.601+0000 7f6f5fe00700 1 --2- 172.21.15.78:0/1860051081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f6019cfb0 0x7f6f6019d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.601+0000 7f6f5fe00700 1 -- 172.21.15.78:0/1860051081 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c0095f0 con 0x7f6f60100f20 2024-03-14T10:02:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.602+0000 7f6f5fe00700 1 --2- 172.21.15.78:0/1860051081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f60100f20 0x7f6f60199780 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6f5000e9a0 tx=0x7f6f5000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.602+0000 7f6f5cc00700 1 -- 172.21.15.78:0/1860051081 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f50009790 con 0x7f6f60100f20 2024-03-14T10:02:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.602+0000 7f6f5cc00700 1 -- 172.21.15.78:0/1860051081 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f50010430 con 0x7f6f60100f20 2024-03-14T10:02:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.602+0000 7f6f67000700 1 -- 172.21.15.78:0/1860051081 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f601a6340 con 0x7f6f60100f20 2024-03-14T10:02:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.602+0000 7f6f5cc00700 1 -- 172.21.15.78:0/1860051081 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f50018680 con 0x7f6f60100f20 2024-03-14T10:02:54.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.602+0000 7f6f67000700 1 -- 172.21.15.78:0/1860051081 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f601a6820 con 0x7f6f60100f20 2024-03-14T10:02:54.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.605+0000 7f6f5cc00700 1 -- 172.21.15.78:0/1860051081 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f6f500105a0 con 0x7f6f60100f20 2024-03-14T10:02:54.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.606+0000 7f6f7ab00b80 1 Processor -- start 2024-03-14T10:02:54.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.606+0000 7f6f5cc00700 1 --2- 172.21.15.78:0/1860051081 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6f440a0c30 0x7f6f440a30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.607+0000 7f6f7ab00b80 1 -- start start 2024-03-14T10:02:54.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.607+0000 7f6f5cc00700 1 -- 172.21.15.78:0/1860051081 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f6f50014070 con 0x7f6f60100f20 2024-03-14T10:02:54.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.607+0000 7f6f65000700 1 --2- 172.21.15.78:0/1860051081 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6f440a0c30 0x7f6f440a30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.607+0000 7f6f7ab00b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26a6680 0x55cbd26a6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.607+0000 7f6f7ab00b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 0x55cbd26b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.607+0000 7f6f7ab00b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26b0ec0 0x55cbd26b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.607+0000 7f6f7ab00b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbd25ff990 con 0x55cbd26b0ec0 2024-03-14T10:02:54.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.607+0000 7f6f7ab00b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbd25fd350 con 0x55cbd26a6fa0 2024-03-14T10:02:54.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.607+0000 7f6f7ab00b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbd25f8c40 con 0x55cbd26a6680 2024-03-14T10:02:54.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.607+0000 7f6f5f400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 0x55cbd26b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.607+0000 7f6f5f400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 0x55cbd26b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:36140/0 (socket says 172.21.15.78:36140) 2024-03-14T10:02:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.607+0000 7f6f5f400700 1 -- 172.21.15.78:0/962331774 learned_addr learned my addr 172.21.15.78:0/962331774 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.607+0000 7f6f5fe00700 1 --2- 172.21.15.78:0/962331774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26a6680 0x55cbd26a6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.608+0000 7f6f65000700 1 --2- 172.21.15.78:0/962331774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26b0ec0 0x55cbd26b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.608+0000 7f6f65000700 1 --2- 172.21.15.78:0/1860051081 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6f440a0c30 0x7f6f440a30f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6f5800f590 tx=0x7f6f58009920 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.608+0000 7f6f5f400700 1 -- 172.21.15.78:0/962331774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26a6680 msgr2=0x55cbd26a6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.608+0000 7f6f5f400700 1 --2- 172.21.15.78:0/962331774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26a6680 0x55cbd26a6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.608+0000 7f6f5f400700 1 -- 172.21.15.78:0/962331774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26b0ec0 msgr2=0x55cbd26b32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.608+0000 7f6f5f400700 1 --2- 172.21.15.78:0/962331774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26b0ec0 0x55cbd26b32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.608+0000 7f6f5f400700 1 -- 172.21.15.78:0/962331774 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cbd247a3b0 con 0x55cbd26a6fa0 2024-03-14T10:02:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.608+0000 7f6f65000700 1 --2- 172.21.15.78:0/962331774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26b0ec0 0x55cbd26b32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.608+0000 7f6f5f400700 1 --2- 172.21.15.78:0/962331774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 0x55cbd26b0980 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009990 tx=0x7f6f4c00b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=475d7649748d1d8c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.608+0000 7f6f55600700 1 -- 172.21.15.78:0/962331774 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c01d070 con 0x55cbd26a6fa0 2024-03-14T10:02:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.608+0000 7f6f55600700 1 -- 172.21.15.78:0/962331774 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f4c01e040 con 0x55cbd26a6fa0 2024-03-14T10:02:54.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.608+0000 7f6f55600700 1 -- 172.21.15.78:0/962331774 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c004760 con 0x55cbd26a6fa0 2024-03-14T10:02:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.609+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/962331774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 msgr2=0x55cbd26b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.609+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/962331774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 0x55cbd26b0980 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009990 tx=0x7f6f4c00b610 comp rx=0 tx=0).stop 2024-03-14T10:02:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.609+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/962331774 shutdown_connections 2024-03-14T10:02:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.609+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/962331774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26b0ec0 0x55cbd26b32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.609+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/962331774 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26a6680 0x55cbd26a6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.609+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/962331774 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 0x55cbd26b0980 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.609+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/962331774 >> 172.21.15.78:0/962331774 conn(0x55cbd261c220 msgr2=0x55cbd26a1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.609+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/962331774 shutdown_connections 2024-03-14T10:02:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.609+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/962331774 wait complete. 2024-03-14T10:02:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.610+0000 7f6f7ab00b80 1 -- start start 2024-03-14T10:02:54.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.610+0000 7f6f7ab00b80 1 Processor -- start 2024-03-14T10:02:54.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.611+0000 7f6f7ab00b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26a6680 0x55cbd26b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.611+0000 7f6f7ab00b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6fa0 0x55cbd26adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.611+0000 7f6f7ab00b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26b0ec0 0x55cbd26ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.611+0000 7f6f7ab00b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbd2594d80 con 0x55cbd26a6fa0 2024-03-14T10:02:54.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.611+0000 7f6f7ab00b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbd2594a20 con 0x55cbd26b0ec0 2024-03-14T10:02:54.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.611+0000 7f6f7ab00b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbd247c6a0 con 0x55cbd26a6680 2024-03-14T10:02:54.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.611+0000 7f6f7ab00b80 10 client.0 _ll_register_callbacks cb 0x55cbd24003a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:54.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.611+0000 7f6f5f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6fa0 0x55cbd26adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.611+0000 7f6f5f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6fa0 0x55cbd26adbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34824/0 (socket says 172.21.15.78:34824) 2024-03-14T10:02:54.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.611+0000 7f6f5f400700 1 -- 172.21.15.78:0/3951647706 learned_addr learned my addr 172.21.15.78:0/3951647706 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:54.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.611+0000 7f6f5fe00700 1 --2- 172.21.15.78:0/3951647706 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26a6680 0x55cbd26b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.611+0000 7f6f65000700 1 --2- 172.21.15.78:0/3951647706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26b0ec0 0x55cbd26ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.612+0000 7f6f5f400700 1 -- 172.21.15.78:0/3951647706 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26a6680 msgr2=0x55cbd26b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:54.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.612+0000 7f6f5f400700 1 --2- 172.21.15.78:0/3951647706 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26a6680 0x55cbd26b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.612+0000 7f6f5f400700 1 -- 172.21.15.78:0/3951647706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26b0ec0 msgr2=0x55cbd26ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:54.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.612+0000 7f6f5f400700 1 --2- 172.21.15.78:0/3951647706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26b0ec0 0x55cbd26ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.612+0000 7f6f5f400700 1 -- 172.21.15.78:0/3951647706 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f58019460 con 0x55cbd26a6fa0 2024-03-14T10:02:54.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.612+0000 7f6f65000700 1 --2- 172.21.15.78:0/3951647706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26b0ec0 0x55cbd26ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:02:54.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.612+0000 7f6f5f400700 1 --2- 172.21.15.78:0/3951647706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6fa0 0x55cbd26adbb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c00b6c0 tx=0x7f6f4c00ba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af25fa3390056e3c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:54.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.612+0000 7f6f4a000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:54.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.612+0000 7f6f4a000700 1 -- 172.21.15.78:0/3951647706 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c01d030 con 0x55cbd26a6fa0 2024-03-14T10:02:54.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.612+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/3951647706 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cbd2699720 con 0x55cbd26a6fa0 2024-03-14T10:02:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.612+0000 7f6f4a000700 1 -- 172.21.15.78:0/3951647706 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f6f4c004910 con 0x55cbd26a6fa0 2024-03-14T10:02:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.612+0000 7f6f4a000700 1 -- 172.21.15.78:0/3951647706 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f4c005bc0 con 0x55cbd26a6fa0 2024-03-14T10:02:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.612+0000 7f6f7ab00b80 10 client.17298 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.612+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/3951647706 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f6f6010e430 con 0x55cbd26a6fa0 2024-03-14T10:02:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.613+0000 7f6f4a000700 1 -- 172.21.15.78:0/3951647706 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cbd2699720 con 0x55cbd26a6fa0 2024-03-14T10:02:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.613+0000 7f6f4a000700 1 -- 172.21.15.78:0/3951647706 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6f4c00f9f0 con 0x55cbd26a6fa0 2024-03-14T10:02:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.613+0000 7f6f7ab00b80 10 client.17298 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.613+0000 7f6f7ab00b80 10 client.17298 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:02:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.613+0000 7f6f7ab00b80 20 client.17298 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.613+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/3951647706 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55cbd26c0560 0x55cbd26c2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.613+0000 7f6f7ab00b80 4 client.17298 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:02:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.613+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/3951647706 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cbd25f1700 con 0x55cbd26c0560 2024-03-14T10:02:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.613+0000 7f6f5fe00700 1 --2- 172.21.15.78:0/3951647706 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55cbd26c0560 0x55cbd26c2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.614+0000 7f6f5fe00700 1 --2- 172.21.15.78:0/3951647706 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55cbd26c0560 0x55cbd26c2980 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4267aade3e6848e2 server_cookie=632eedeaaf517344 in_seq=0 out_seq=0 2024-03-14T10:02:54.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.614+0000 7f6f4a000700 10 client.17298 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:54.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.614+0000 7f6f4a000700 1 -- 172.21.15.78:0/3951647706 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55cbd25f1700 con 0x55cbd26c0560 2024-03-14T10:02:54.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.614+0000 7f6f4a000700 10 client.17298 handle_command_reply: tid=0 2024-03-14T10:02:54.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.615+0000 7f6f7ab00b80 1 client.17298 shutdown 2024-03-14T10:02:54.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.615+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/3951647706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6fa0 msgr2=0x55cbd26adbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:54.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.615+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/3951647706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6fa0 0x55cbd26adbb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c00b6c0 tx=0x7f6f4c00ba00 comp rx=0 tx=0).stop 2024-03-14T10:02:54.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.615+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/3951647706 shutdown_connections 2024-03-14T10:02:54.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.615+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/3951647706 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55cbd26c0560 0x55cbd26c2980 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.615+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/3951647706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26b0ec0 0x55cbd26ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.615+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/3951647706 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26a6680 0x55cbd26b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.615+0000 7f6f4a000700 0 client.17298 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:54.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.615+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/3951647706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6fa0 0x55cbd26adbb0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.615+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/3951647706 >> 172.21.15.78:0/3951647706 conn(0x55cbd261c220 msgr2=0x55cbd26a1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:54.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.615+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/3951647706 shutdown_connections 2024-03-14T10:02:54.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.615+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/3951647706 wait complete. 2024-03-14T10:02:54.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.615+0000 7f6f7ab00b80 20 client.17298 trim_cache size 0 max 16384 2024-03-14T10:02:54.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.632+0000 7f6f7ab00b80 1 Processor -- start 2024-03-14T10:02:54.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.632+0000 7f6f7ab00b80 1 -- start start 2024-03-14T10:02:54.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.632+0000 7f6f7ab00b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6680 0x55cbd26b7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.632+0000 7f6f7ab00b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 0x55cbd26ac4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.632+0000 7f6f7ab00b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26b0ec0 0x55cbd26ac9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.632+0000 7f6f7ab00b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbd26a2670 con 0x55cbd26a6680 2024-03-14T10:02:54.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.632+0000 7f6f7ab00b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbd269e230 con 0x55cbd26a6fa0 2024-03-14T10:02:54.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.632+0000 7f6f7ab00b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbd26a7af0 con 0x55cbd26b0ec0 2024-03-14T10:02:54.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.632+0000 7f6f5fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6680 0x55cbd26b7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.632+0000 7f6f5fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6680 0x55cbd26b7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34830/0 (socket says 172.21.15.78:34830) 2024-03-14T10:02:54.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.632+0000 7f6f5fe00700 1 -- 172.21.15.78:0/3874396977 learned_addr learned my addr 172.21.15.78:0/3874396977 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:54.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.632+0000 7f6f65000700 1 --2- 172.21.15.78:0/3874396977 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26b0ec0 0x55cbd26ac9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.633+0000 7f6f5f400700 1 --2- 172.21.15.78:0/3874396977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 0x55cbd26ac4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.633+0000 7f6f5fe00700 1 -- 172.21.15.78:0/3874396977 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26b0ec0 msgr2=0x55cbd26ac9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:54.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.633+0000 7f6f5fe00700 1 --2- 172.21.15.78:0/3874396977 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26b0ec0 0x55cbd26ac9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.633+0000 7f6f5fe00700 1 -- 172.21.15.78:0/3874396977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 msgr2=0x55cbd26ac4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:54.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.633+0000 7f6f5fe00700 1 --2- 172.21.15.78:0/3874396977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 0x55cbd26ac4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.633+0000 7f6f5fe00700 1 -- 172.21.15.78:0/3874396977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cbd2426b60 con 0x55cbd26a6680 2024-03-14T10:02:54.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.633+0000 7f6f5f400700 1 --2- 172.21.15.78:0/3874396977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 0x55cbd26ac4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:02:54.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.633+0000 7f6f5fe00700 1 --2- 172.21.15.78:0/3874396977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6680 0x55cbd26b7af0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f6f500963c0 tx=0x7f6f5000c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84a7ab5e2566b6e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:54.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.633+0000 7f6f56000700 1 -- 172.21.15.78:0/3874396977 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f500a1dd0 con 0x55cbd26a6680 2024-03-14T10:02:54.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.633+0000 7f6f56000700 1 -- 172.21.15.78:0/3874396977 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f500b2260 con 0x55cbd26a6680 2024-03-14T10:02:54.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.633+0000 7f6f56000700 1 -- 172.21.15.78:0/3874396977 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f50099dd0 con 0x55cbd26a6680 2024-03-14T10:02:54.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.634+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/3874396977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6680 msgr2=0x55cbd26b7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:54.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.634+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/3874396977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6680 0x55cbd26b7af0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f6f500963c0 tx=0x7f6f5000c480 comp rx=0 tx=0).stop 2024-03-14T10:02:54.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.634+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/3874396977 shutdown_connections 2024-03-14T10:02:54.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.634+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/3874396977 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26b0ec0 0x55cbd26ac9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.634+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/3874396977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6680 0x55cbd26b7af0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.634+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/3874396977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 0x55cbd26ac4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.634+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/3874396977 >> 172.21.15.78:0/3874396977 conn(0x55cbd261c220 msgr2=0x55cbd26a15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:54.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.634+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/3874396977 shutdown_connections 2024-03-14T10:02:54.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.634+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/3874396977 wait complete. 2024-03-14T10:02:54.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.634+0000 7f6f7ab00b80 1 -- start start 2024-03-14T10:02:54.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.635+0000 7f6f7ab00b80 1 Processor -- start 2024-03-14T10:02:54.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.635+0000 7f6f7ab00b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6680 0x55cbd26b7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.635+0000 7f6f7ab00b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 0x55cbd26b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.635+0000 7f6f7ab00b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26b0ec0 0x55cbd26b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.635+0000 7f6f7ab00b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbd26b68c0 con 0x55cbd26a6680 2024-03-14T10:02:54.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.635+0000 7f6f7ab00b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbd26b6a30 con 0x55cbd26a6fa0 2024-03-14T10:02:54.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.635+0000 7f6f7ab00b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbd2712090 con 0x55cbd26b0ec0 2024-03-14T10:02:54.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.635+0000 7f6f7ab00b80 10 client.0 _ll_register_callbacks cb 0x7f6f5800f330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:54.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.636+0000 7f6f5fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6680 0x55cbd26b7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.636+0000 7f6f5fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6680 0x55cbd26b7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34838/0 (socket says 172.21.15.78:34838) 2024-03-14T10:02:54.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.636+0000 7f6f5fe00700 1 -- 172.21.15.78:0/1778783222 learned_addr learned my addr 172.21.15.78:0/1778783222 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:54.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.636+0000 7f6f65000700 1 --2- 172.21.15.78:0/1778783222 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26b0ec0 0x55cbd26b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.636+0000 7f6f5f400700 1 --2- 172.21.15.78:0/1778783222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 0x55cbd26b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.636+0000 7f6f5fe00700 1 -- 172.21.15.78:0/1778783222 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26b0ec0 msgr2=0x55cbd26b61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:54.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.636+0000 7f6f5fe00700 1 --2- 172.21.15.78:0/1778783222 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26b0ec0 0x55cbd26b61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.636+0000 7f6f5fe00700 1 -- 172.21.15.78:0/1778783222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 msgr2=0x55cbd26b5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:54.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.636+0000 7f6f5fe00700 1 --2- 172.21.15.78:0/1778783222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 0x55cbd26b5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.636+0000 7f6f5fe00700 1 -- 172.21.15.78:0/1778783222 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f601a6820 con 0x55cbd26a6680 2024-03-14T10:02:54.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.636+0000 7f6f5fe00700 1 --2- 172.21.15.78:0/1778783222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6680 0x55cbd26b7980 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6f500a18a0 tx=0x7f6f50099850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7624304cae191a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:54.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.636+0000 7f6f4a000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:54.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.636+0000 7f6f4a000700 1 -- 172.21.15.78:0/1778783222 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f50010880 con 0x55cbd26a6680 2024-03-14T10:02:54.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.636+0000 7f6f4a000700 1 -- 172.21.15.78:0/1778783222 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f6f50098e40 con 0x55cbd26a6680 2024-03-14T10:02:54.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.636+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/1778783222 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6f4c01e040 con 0x55cbd26a6680 2024-03-14T10:02:54.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.637+0000 7f6f4a000700 1 -- 172.21.15.78:0/1778783222 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f500109e0 con 0x55cbd26a6680 2024-03-14T10:02:54.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.637+0000 7f6f7ab00b80 10 client.17304 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:54.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.637+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/1778783222 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55cbd25beea0 con 0x55cbd26a6680 2024-03-14T10:02:54.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.637+0000 7f6f4a000700 1 -- 172.21.15.78:0/1778783222 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f4c01e040 con 0x55cbd26a6680 2024-03-14T10:02:54.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.637+0000 7f6f4a000700 1 -- 172.21.15.78:0/1778783222 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6f500a19d0 con 0x55cbd26a6680 2024-03-14T10:02:54.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.637+0000 7f6f7ab00b80 10 client.17304 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:54.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.637+0000 7f6f7ab00b80 10 client.17304 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:02:54.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.637+0000 7f6f7ab00b80 20 client.17304 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:54.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.637+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/1778783222 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55cbd2713540 0x55cbd2715930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.637+0000 7f6f7ab00b80 4 client.17304 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:02:54.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.637+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/1778783222 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cbd2447cf0 con 0x55cbd2713540 2024-03-14T10:02:54.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.637+0000 7f6f5f400700 1 --2- 172.21.15.78:0/1778783222 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55cbd2713540 0x55cbd2715930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.638+0000 7f6f4a000700 10 client.17304 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:54.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.638+0000 7f6f5f400700 1 --2- 172.21.15.78:0/1778783222 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55cbd2713540 0x55cbd2715930 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d06a24aa8f85ea4 server_cookie=19ff6f5db9d3a77e in_seq=0 out_seq=0 2024-03-14T10:02:54.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.638+0000 7f6f4a000700 1 -- 172.21.15.78:0/1778783222 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55cbd2447cf0 con 0x55cbd2713540 2024-03-14T10:02:54.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.638+0000 7f6f4a000700 10 client.17304 handle_command_reply: tid=0 2024-03-14T10:02:54.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.638+0000 7f6f7ab00b80 1 client.17304 shutdown 2024-03-14T10:02:54.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.638+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/1778783222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6680 msgr2=0x55cbd26b7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:54.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.638+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/1778783222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6680 0x55cbd26b7980 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6f500a18a0 tx=0x7f6f50099850 comp rx=0 tx=0).stop 2024-03-14T10:02:54.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.638+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/1778783222 shutdown_connections 2024-03-14T10:02:54.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.638+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/1778783222 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55cbd26b0ec0 0x55cbd26b61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.638+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/1778783222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cbd26a6680 0x55cbd26b7980 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.638+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/1778783222 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55cbd2713540 0x55cbd2715930 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.638+0000 7f6f7ab00b80 1 --2- 172.21.15.78:0/1778783222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cbd26a6fa0 0x55cbd26b5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.638+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/1778783222 >> 172.21.15.78:0/1778783222 conn(0x55cbd261c220 msgr2=0x55cbd26a15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:54.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.638+0000 7f6f4a000700 0 client.17304 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:54.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.638+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/1778783222 shutdown_connections 2024-03-14T10:02:54.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.638+0000 7f6f7ab00b80 1 -- 172.21.15.78:0/1778783222 wait complete. 2024-03-14T10:02:54.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.639+0000 7f6f7ab00b80 20 client.17304 trim_cache size 0 max 16384 2024-03-14T10:02:54.644 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:02:54.645 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T10:02:54.645 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T10:02:54.645 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:02:54.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.639+0000 7f6f67000700 1 -- 172.21.15.78:0/1860051081 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6f440a0c30 msgr2=0x7f6f440a30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:54.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.639+0000 7f6f67000700 1 --2- 172.21.15.78:0/1860051081 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6f440a0c30 0x7f6f440a30f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6f5800f590 tx=0x7f6f58009920 comp rx=0 tx=0).stop 2024-03-14T10:02:54.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.639+0000 7f6f67000700 1 -- 172.21.15.78:0/1860051081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f60100f20 msgr2=0x7f6f60199780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:54.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.639+0000 7f6f67000700 1 --2- 172.21.15.78:0/1860051081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f60100f20 0x7f6f60199780 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6f5000e9a0 tx=0x7f6f5000ecb0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.639+0000 7f6f67000700 1 -- 172.21.15.78:0/1860051081 shutdown_connections 2024-03-14T10:02:54.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.639+0000 7f6f67000700 1 --2- 172.21.15.78:0/1860051081 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6f440a0c30 0x7f6f440a30f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.639+0000 7f6f67000700 1 --2- 172.21.15.78:0/1860051081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6f60100f20 0x7f6f60199780 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.639+0000 7f6f67000700 1 --2- 172.21.15.78:0/1860051081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f6019cfb0 0x7f6f6019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.639+0000 7f6f67000700 1 --2- 172.21.15.78:0/1860051081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6f60106040 0x7f6f60199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.639+0000 7f6f67000700 1 -- 172.21.15.78:0/1860051081 >> 172.21.15.78:0/1860051081 conn(0x7f6f600fc7a0 msgr2=0x7f6f600fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:54.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.639+0000 7f6f67000700 1 -- 172.21.15.78:0/1860051081 shutdown_connections 2024-03-14T10:02:54.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.639+0000 7f6f67000700 1 -- 172.21.15.78:0/1860051081 wait complete. 2024-03-14T10:02:54.658 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T10:02:54.658 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T10:02:54.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.850+0000 7f1827600700 1 -- 172.21.15.78:0/1201687890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1820100e20 msgr2=0x7f1820101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:54.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.850+0000 7f1827600700 1 --2- 172.21.15.78:0/1201687890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1820100e20 0x7f1820101200 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f1814009990 tx=0x7f1814009ca0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.850+0000 7f1827600700 1 -- 172.21.15.78:0/1201687890 shutdown_connections 2024-03-14T10:02:54.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.850+0000 7f1827600700 1 --2- 172.21.15.78:0/1201687890 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820105f40 0x7f182010dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.850+0000 7f1827600700 1 --2- 172.21.15.78:0/1201687890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1820100e20 0x7f1820101200 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.850+0000 7f1827600700 1 --2- 172.21.15.78:0/1201687890 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f18201017d0 0x7f1820105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.850+0000 7f1827600700 1 -- 172.21.15.78:0/1201687890 >> 172.21.15.78:0/1201687890 conn(0x7f18200fc700 msgr2=0x7f18200feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:54.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.850+0000 7f1827600700 1 -- 172.21.15.78:0/1201687890 shutdown_connections 2024-03-14T10:02:54.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.850+0000 7f1827600700 1 -- 172.21.15.78:0/1201687890 wait complete. 2024-03-14T10:02:54.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.851+0000 7f1827600700 1 Processor -- start 2024-03-14T10:02:54.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.851+0000 7f1827600700 1 -- start start 2024-03-14T10:02:54.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.852+0000 7f1827600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100e20 0x7f18200730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.852+0000 7f1827600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1820105f40 0x7f18200735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.852+0000 7f1827600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1820074d20 0x7f1820073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.852+0000 7f1827600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18201aa1a0 con 0x7f1820105f40 2024-03-14T10:02:54.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.852+0000 7f1827600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18201aa310 con 0x7f1820074d20 2024-03-14T10:02:54.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.852+0000 7f1827600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18201aa4d0 con 0x7f1820100e20 2024-03-14T10:02:54.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.852+0000 7f1824c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100e20 0x7f18200730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.852+0000 7f181fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1820105f40 0x7f18200735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.852+0000 7f1824c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100e20 0x7f18200730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:58612/0 (socket says 172.21.15.78:58612) 2024-03-14T10:02:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.852+0000 7f1824c00700 1 -- 172.21.15.78:0/1075613888 learned_addr learned my addr 172.21.15.78:0/1075613888 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.852+0000 7f1825600700 1 --2- 172.21.15.78:0/1075613888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1820074d20 0x7f1820073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.852+0000 7f1824c00700 1 -- 172.21.15.78:0/1075613888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1820074d20 msgr2=0x7f1820073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.852+0000 7f1824c00700 1 --2- 172.21.15.78:0/1075613888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1820074d20 0x7f1820073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.852+0000 7f1824c00700 1 -- 172.21.15.78:0/1075613888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1820105f40 msgr2=0x7f18200735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.852+0000 7f1824c00700 1 --2- 172.21.15.78:0/1075613888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1820105f40 0x7f18200735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.852+0000 7f1824c00700 1 -- 172.21.15.78:0/1075613888 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18140095f0 con 0x7f1820100e20 2024-03-14T10:02:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.853+0000 7f1824c00700 1 --2- 172.21.15.78:0/1075613888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100e20 0x7f18200730b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1814005910 tx=0x7f1814004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.853+0000 7f181d600700 1 -- 172.21.15.78:0/1075613888 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181401d070 con 0x7f1820100e20 2024-03-14T10:02:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.853+0000 7f1827600700 1 -- 172.21.15.78:0/1075613888 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18201aa750 con 0x7f1820100e20 2024-03-14T10:02:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.853+0000 7f181d600700 1 -- 172.21.15.78:0/1075613888 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1814022470 con 0x7f1820100e20 2024-03-14T10:02:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.853+0000 7f181d600700 1 -- 172.21.15.78:0/1075613888 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1814017650 con 0x7f1820100e20 2024-03-14T10:02:54.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.853+0000 7f1827600700 1 -- 172.21.15.78:0/1075613888 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18201aabd0 con 0x7f1820100e20 2024-03-14T10:02:54.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.854+0000 7f1827600700 1 -- 172.21.15.78:0/1075613888 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18201091b0 con 0x7f1820100e20 2024-03-14T10:02:54.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.856+0000 7f181d600700 1 -- 172.21.15.78:0/1075613888 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f18140225e0 con 0x7f1820100e20 2024-03-14T10:02:54.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.857+0000 7f181d600700 1 --2- 172.21.15.78:0/1075613888 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f18080a0c00 0x7f18080a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:54.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.857+0000 7f181d600700 1 -- 172.21.15.78:0/1075613888 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f18140cdfe0 con 0x7f1820100e20 2024-03-14T10:02:54.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.858+0000 7f181fe00700 1 --2- 172.21.15.78:0/1075613888 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f18080a0c00 0x7f18080a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:54.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.858+0000 7f181fe00700 1 --2- 172.21.15.78:0/1075613888 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f18080a0c00 0x7f18080a30c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1820074790 tx=0x7f180c00b410 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:54.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:54.860+0000 7f181d600700 1 -- 172.21.15.78:0/1075613888 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f18140972f0 con 0x7f1820100e20 2024-03-14T10:02:55.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.111+0000 7f1827600700 1 -- 172.21.15.78:0/1075613888 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18201022f0 con 0x7f1820100e20 2024-03-14T10:02:55.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.112+0000 7f181d600700 1 -- 172.21.15.78:0/1075613888 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f181409a910 con 0x7f1820100e20 2024-03-14T10:02:55.112 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T10:02:55.113 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T10:02:55.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.114+0000 7f1827600700 1 -- 172.21.15.78:0/1075613888 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f18080a0c00 msgr2=0x7f18080a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.114+0000 7f1827600700 1 --2- 172.21.15.78:0/1075613888 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f18080a0c00 0x7f18080a30c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1820074790 tx=0x7f180c00b410 comp rx=0 tx=0).stop 2024-03-14T10:02:55.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.114+0000 7f1827600700 1 -- 172.21.15.78:0/1075613888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100e20 msgr2=0x7f18200730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.114+0000 7f1827600700 1 --2- 172.21.15.78:0/1075613888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100e20 0x7f18200730b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1814005910 tx=0x7f1814004730 comp rx=0 tx=0).stop 2024-03-14T10:02:55.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.114+0000 7f1827600700 1 -- 172.21.15.78:0/1075613888 shutdown_connections 2024-03-14T10:02:55.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.114+0000 7f1827600700 1 --2- 172.21.15.78:0/1075613888 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f18080a0c00 0x7f18080a30c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.114+0000 7f1827600700 1 --2- 172.21.15.78:0/1075613888 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1820074d20 0x7f1820073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.114+0000 7f1827600700 1 --2- 172.21.15.78:0/1075613888 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1820100e20 0x7f18200730b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.114+0000 7f1827600700 1 --2- 172.21.15.78:0/1075613888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1820105f40 0x7f18200735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.114+0000 7f1827600700 1 -- 172.21.15.78:0/1075613888 >> 172.21.15.78:0/1075613888 conn(0x7f18200fc700 msgr2=0x7f18200fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.114+0000 7f1827600700 1 -- 172.21.15.78:0/1075613888 shutdown_connections 2024-03-14T10:02:55.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.114+0000 7f1827600700 1 -- 172.21.15.78:0/1075613888 wait complete. 2024-03-14T10:02:55.116 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T10:02:55.134 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T10:02:55.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.328+0000 7f60bac00700 1 -- 172.21.15.78:0/3325942841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b410d4f0 msgr2=0x7f60b410f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.328+0000 7f60bac00700 1 --2- 172.21.15.78:0/3325942841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b410d4f0 0x7f60b410f960 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f60ac00b0a0 tx=0x7f60ac00b3b0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.329+0000 7f60bac00700 1 -- 172.21.15.78:0/3325942841 shutdown_connections 2024-03-14T10:02:55.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.329+0000 7f60bac00700 1 --2- 172.21.15.78:0/3325942841 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60b40731e0 0x7f60b40735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.329+0000 7f60bac00700 1 --2- 172.21.15.78:0/3325942841 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b410d4f0 0x7f60b410f960 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.329+0000 7f60bac00700 1 --2- 172.21.15.78:0/3325942841 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f60b4073b00 0x7f60b410cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.329+0000 7f60bac00700 1 -- 172.21.15.78:0/3325942841 >> 172.21.15.78:0/3325942841 conn(0x7f60b40fc7a0 msgr2=0x7f60b40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.329+0000 7f60bac00700 1 -- 172.21.15.78:0/3325942841 shutdown_connections 2024-03-14T10:02:55.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.329+0000 7f60bac00700 1 -- 172.21.15.78:0/3325942841 wait complete. 2024-03-14T10:02:55.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.330+0000 7f60bac00700 1 Processor -- start 2024-03-14T10:02:55.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.330+0000 7f60bac00700 1 -- start start 2024-03-14T10:02:55.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.330+0000 7f60bac00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f60b40731e0 0x7f60b419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.330+0000 7f60bac00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b4073b00 0x7f60b419faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.330+0000 7f60bac00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60b410d4f0 0x7f60b4199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.330+0000 7f60bac00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b4199d90 con 0x7f60b4073b00 2024-03-14T10:02:55.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.330+0000 7f60bac00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b4199f00 con 0x7f60b410d4f0 2024-03-14T10:02:55.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.330+0000 7f60bac00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b419a0c0 con 0x7f60b40731e0 2024-03-14T10:02:55.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.330+0000 7f60b3400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b4073b00 0x7f60b419faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.330+0000 7f60b3400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b4073b00 0x7f60b419faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34876/0 (socket says 172.21.15.78:34876) 2024-03-14T10:02:55.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.330+0000 7f60b3400700 1 -- 172.21.15.78:0/853828030 learned_addr learned my addr 172.21.15.78:0/853828030 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:55.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.330+0000 7f60b3e00700 1 --2- 172.21.15.78:0/853828030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f60b40731e0 0x7f60b419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.331+0000 7f60b8c00700 1 --2- 172.21.15.78:0/853828030 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60b410d4f0 0x7f60b4199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.331+0000 7f60b3400700 1 -- 172.21.15.78:0/853828030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f60b40731e0 msgr2=0x7f60b419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.331+0000 7f60b3400700 1 --2- 172.21.15.78:0/853828030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f60b40731e0 0x7f60b419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.331+0000 7f60b3400700 1 -- 172.21.15.78:0/853828030 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60b410d4f0 msgr2=0x7f60b4199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.331+0000 7f60b3400700 1 --2- 172.21.15.78:0/853828030 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60b410d4f0 0x7f60b4199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.331+0000 7f60b3400700 1 -- 172.21.15.78:0/853828030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac009d00 con 0x7f60b4073b00 2024-03-14T10:02:55.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.331+0000 7f60b3400700 1 --2- 172.21.15.78:0/853828030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b4073b00 0x7f60b419faf0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f60a000e9a0 tx=0x7f60a000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.331+0000 7f60b0c00700 1 -- 172.21.15.78:0/853828030 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0009790 con 0x7f60b4073b00 2024-03-14T10:02:55.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.331+0000 7f60bac00700 1 -- 172.21.15.78:0/853828030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b419a3a0 con 0x7f60b4073b00 2024-03-14T10:02:55.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.331+0000 7f60b0c00700 1 -- 172.21.15.78:0/853828030 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60a0010430 con 0x7f60b4073b00 2024-03-14T10:02:55.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.331+0000 7f60b0c00700 1 -- 172.21.15.78:0/853828030 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0018680 con 0x7f60b4073b00 2024-03-14T10:02:55.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.331+0000 7f60bac00700 1 -- 172.21.15.78:0/853828030 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b41a65a0 con 0x7f60b4073b00 2024-03-14T10:02:55.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.334+0000 7f60b0c00700 1 -- 172.21.15.78:0/853828030 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f60a00105a0 con 0x7f60b4073b00 2024-03-14T10:02:55.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.335+0000 7f60b0c00700 1 --2- 172.21.15.78:0/853828030 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f60980a0c00 0x7f60980a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.336+0000 7f60ce7deb80 1 Processor -- start 2024-03-14T10:02:55.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.336+0000 7f60b0c00700 1 -- 172.21.15.78:0/853828030 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f60a0014070 con 0x7f60b4073b00 2024-03-14T10:02:55.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.336+0000 7f60ce7deb80 1 -- start start 2024-03-14T10:02:55.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.336+0000 7f60b3e00700 1 --2- 172.21.15.78:0/853828030 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f60980a0c00 0x7f60980a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.336+0000 7f60ce7deb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c8fc680 0x559e8c8fca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.336+0000 7f60ce7deb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fcfa0 0x559e8c906980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.336+0000 7f60ce7deb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c906ec0 0x559e8c9092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.336+0000 7f60ce7deb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8c855990 con 0x559e8c8fc680 2024-03-14T10:02:55.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.336+0000 7f60ce7deb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8c853350 con 0x559e8c8fcfa0 2024-03-14T10:02:55.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.336+0000 7f60ce7deb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8c84ec40 con 0x559e8c906ec0 2024-03-14T10:02:55.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.337+0000 7f60b3e00700 1 --2- 172.21.15.78:0/853828030 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f60980a0c00 0x7f60980a30c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009c60 tx=0x7f60a4009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.337+0000 7f60b3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c8fc680 0x559e8c8fca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.337+0000 7f60b3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c8fc680 0x559e8c8fca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34884/0 (socket says 172.21.15.78:34884) 2024-03-14T10:02:55.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.337+0000 7f60b3e00700 1 -- 172.21.15.78:0/2552328874 learned_addr learned my addr 172.21.15.78:0/2552328874 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:55.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.337+0000 7f60b8c00700 1 --2- 172.21.15.78:0/2552328874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c906ec0 0x559e8c9092b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.337+0000 7f60b3400700 1 --2- 172.21.15.78:0/2552328874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fcfa0 0x559e8c906980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.337+0000 7f60b3e00700 1 -- 172.21.15.78:0/2552328874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c906ec0 msgr2=0x559e8c9092b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.337+0000 7f60b3e00700 1 --2- 172.21.15.78:0/2552328874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c906ec0 0x559e8c9092b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.337+0000 7f60b3e00700 1 -- 172.21.15.78:0/2552328874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fcfa0 msgr2=0x559e8c906980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.337+0000 7f60b3e00700 1 --2- 172.21.15.78:0/2552328874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fcfa0 0x559e8c906980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.337+0000 7f60b3e00700 1 -- 172.21.15.78:0/2552328874 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e8c6d03b0 con 0x559e8c8fc680 2024-03-14T10:02:55.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.337+0000 7f60b3e00700 1 --2- 172.21.15.78:0/2552328874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c8fc680 0x559e8c8fca60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f60a400e430 tx=0x7f60a4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb63d472209ae53 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.338+0000 7f60a9200700 1 -- 172.21.15.78:0/2552328874 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a400edd0 con 0x559e8c8fc680 2024-03-14T10:02:55.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.338+0000 7f60a9200700 1 -- 172.21.15.78:0/2552328874 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60a4021070 con 0x559e8c8fc680 2024-03-14T10:02:55.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.338+0000 7f60a9200700 1 -- 172.21.15.78:0/2552328874 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4026480 con 0x559e8c8fc680 2024-03-14T10:02:55.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.338+0000 7f60ce7deb80 1 -- 172.21.15.78:0/2552328874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c8fc680 msgr2=0x559e8c8fca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.338+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/2552328874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c8fc680 0x559e8c8fca60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f60a400e430 tx=0x7f60a4014000 comp rx=0 tx=0).stop 2024-03-14T10:02:55.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.338+0000 7f60ce7deb80 1 -- 172.21.15.78:0/2552328874 shutdown_connections 2024-03-14T10:02:55.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.338+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/2552328874 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c906ec0 0x559e8c9092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.338+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/2552328874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c8fc680 0x559e8c8fca60 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.338+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/2552328874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fcfa0 0x559e8c906980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.338+0000 7f60ce7deb80 1 -- 172.21.15.78:0/2552328874 >> 172.21.15.78:0/2552328874 conn(0x559e8c872220 msgr2=0x559e8c8f7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.338+0000 7f60ce7deb80 1 -- 172.21.15.78:0/2552328874 shutdown_connections 2024-03-14T10:02:55.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.338+0000 7f60ce7deb80 1 -- 172.21.15.78:0/2552328874 wait complete. 2024-03-14T10:02:55.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.339+0000 7f60ce7deb80 1 -- start start 2024-03-14T10:02:55.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.339+0000 7f60ce7deb80 1 Processor -- start 2024-03-14T10:02:55.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60ce7deb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fc680 0x559e8c9068f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60ce7deb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c8fcfa0 0x559e8c903be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60ce7deb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c906ec0 0x559e8c904320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60ce7deb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8c7ead80 con 0x559e8c906ec0 2024-03-14T10:02:55.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60ce7deb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8c7eaa20 con 0x559e8c8fc680 2024-03-14T10:02:55.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60ce7deb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8c6d26a0 con 0x559e8c8fcfa0 2024-03-14T10:02:55.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60ce7deb80 10 client.0 _ll_register_callbacks cb 0x559e8c6563a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60b8c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c906ec0 0x559e8c904320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60b3400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c8fcfa0 0x559e8c903be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60b3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fc680 0x559e8c9068f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60b8c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c906ec0 0x559e8c904320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34894/0 (socket says 172.21.15.78:34894) 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60b3400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c8fcfa0 0x559e8c903be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:58658/0 (socket says 172.21.15.78:58658) 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60b3400700 1 -- 172.21.15.78:0/3090061207 learned_addr learned my addr 172.21.15.78:0/3090061207 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60b3400700 1 -- 172.21.15.78:0/3090061207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fc680 msgr2=0x559e8c9068f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60b3400700 1 --2- 172.21.15.78:0/3090061207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fc680 0x559e8c9068f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60b3400700 1 -- 172.21.15.78:0/3090061207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c906ec0 msgr2=0x559e8c904320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60b3400700 1 --2- 172.21.15.78:0/3090061207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c906ec0 0x559e8c904320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60b3400700 1 -- 172.21.15.78:0/3090061207 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a4017070 con 0x559e8c8fcfa0 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60b3e00700 1 --2- 172.21.15.78:0/3090061207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fc680 0x559e8c9068f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f60b3400700 1 --2- 172.21.15.78:0/3090061207 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c8fcfa0 0x559e8c903be0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f60a0095da0 tx=0x7f60a009a070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6fb3408503fdac8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.340+0000 7f609d600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.341+0000 7f609d600700 1 -- 172.21.15.78:0/3090061207 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a0098fe0 con 0x559e8c8fcfa0 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.341+0000 7f60ce7deb80 1 -- 172.21.15.78:0/3090061207 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559e8c8ef720 con 0x559e8c8fcfa0 2024-03-14T10:02:55.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.341+0000 7f609d600700 1 -- 172.21.15.78:0/3090061207 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f60a009d720 con 0x559e8c8fcfa0 2024-03-14T10:02:55.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.341+0000 7f609d600700 1 -- 172.21.15.78:0/3090061207 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60a00a1d70 con 0x559e8c8fcfa0 2024-03-14T10:02:55.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.341+0000 7f60ce7deb80 10 client.27373 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:55.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.341+0000 7f60ce7deb80 1 -- 172.21.15.78:0/3090061207 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559e8c67cb60 con 0x559e8c8fcfa0 2024-03-14T10:02:55.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.341+0000 7f609d600700 1 -- 172.21.15.78:0/3090061207 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559e8c8ef720 con 0x559e8c8fcfa0 2024-03-14T10:02:55.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.341+0000 7f609d600700 1 -- 172.21.15.78:0/3090061207 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f60a00b90b0 con 0x559e8c8fcfa0 2024-03-14T10:02:55.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.341+0000 7f60ce7deb80 10 client.27373 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:55.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.341+0000 7f60ce7deb80 10 client.27373 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:02:55.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.341+0000 7f60ce7deb80 20 client.27373 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:55.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.341+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/3090061207 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559e8c916be0 0x559e8c919000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.341+0000 7f60ce7deb80 4 client.27373 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:02:55.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.341+0000 7f60ce7deb80 1 -- 172.21.15.78:0/3090061207 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559e8c847700 con 0x559e8c916be0 2024-03-14T10:02:55.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.342+0000 7f60b3e00700 1 --2- 172.21.15.78:0/3090061207 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559e8c916be0 0x559e8c919000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.342+0000 7f609d600700 10 client.27373 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:02:55.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.342+0000 7f60b3e00700 1 --2- 172.21.15.78:0/3090061207 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559e8c916be0 0x559e8c919000 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4f8b2b0b3543f45 server_cookie=1b2d05c364f7128a in_seq=0 out_seq=0 2024-03-14T10:02:55.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.343+0000 7f609d600700 1 -- 172.21.15.78:0/3090061207 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x559e8c847700 con 0x559e8c916be0 2024-03-14T10:02:55.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.343+0000 7f609d600700 10 client.27373 handle_command_reply: tid=0 2024-03-14T10:02:55.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.343+0000 7f60ce7deb80 1 client.27373 shutdown 2024-03-14T10:02:55.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.343+0000 7f60ce7deb80 1 -- 172.21.15.78:0/3090061207 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c8fcfa0 msgr2=0x559e8c903be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.343+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/3090061207 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c8fcfa0 0x559e8c903be0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f60a0095da0 tx=0x7f60a009a070 comp rx=0 tx=0).stop 2024-03-14T10:02:55.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.343+0000 7f60ce7deb80 1 -- 172.21.15.78:0/3090061207 shutdown_connections 2024-03-14T10:02:55.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.343+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/3090061207 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559e8c916be0 0x559e8c919000 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.343+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/3090061207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fc680 0x559e8c9068f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.343+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/3090061207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c906ec0 0x559e8c904320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.343+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/3090061207 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c8fcfa0 0x559e8c903be0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.343+0000 7f609d600700 0 client.27373 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:02:55.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.343+0000 7f60ce7deb80 1 -- 172.21.15.78:0/3090061207 >> 172.21.15.78:0/3090061207 conn(0x559e8c872220 msgr2=0x559e8c8f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.344+0000 7f60ce7deb80 1 -- 172.21.15.78:0/3090061207 shutdown_connections 2024-03-14T10:02:55.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.344+0000 7f60ce7deb80 1 -- 172.21.15.78:0/3090061207 wait complete. 2024-03-14T10:02:55.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.344+0000 7f60ce7deb80 20 client.27373 trim_cache size 0 max 16384 2024-03-14T10:02:55.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.357+0000 7f60ce7deb80 1 Processor -- start 2024-03-14T10:02:55.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.357+0000 7f60ce7deb80 1 -- start start 2024-03-14T10:02:55.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.357+0000 7f60ce7deb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c8fc680 0x559e8c90db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.357+0000 7f60ce7deb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fcfa0 0x559e8c8f3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.357+0000 7f60ce7deb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c906ec0 0x559e8c906420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.357+0000 7f60ce7deb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8c8fe310 con 0x559e8c906ec0 2024-03-14T10:02:55.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.357+0000 7f60ce7deb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8c8f4230 con 0x559e8c8fcfa0 2024-03-14T10:02:55.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.357+0000 7f60ce7deb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8c8fcde0 con 0x559e8c8fc680 2024-03-14T10:02:55.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.357+0000 7f60b3e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c8fc680 0x559e8c90db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.357+0000 7f60b3e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c8fc680 0x559e8c90db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:58662/0 (socket says 172.21.15.78:58662) 2024-03-14T10:02:55.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.357+0000 7f60b3e00700 1 -- 172.21.15.78:0/1397565817 learned_addr learned my addr 172.21.15.78:0/1397565817 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:55.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.357+0000 7f60b8c00700 1 --2- 172.21.15.78:0/1397565817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c906ec0 0x559e8c906420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.358+0000 7f60b3400700 1 --2- 172.21.15.78:0/1397565817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fcfa0 0x559e8c8f3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.358+0000 7f60b3e00700 1 -- 172.21.15.78:0/1397565817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fcfa0 msgr2=0x559e8c8f3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.358+0000 7f60b3e00700 1 --2- 172.21.15.78:0/1397565817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fcfa0 0x559e8c8f3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.358+0000 7f60b3e00700 1 -- 172.21.15.78:0/1397565817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c906ec0 msgr2=0x559e8c906420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.358+0000 7f60b3e00700 1 --2- 172.21.15.78:0/1397565817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c906ec0 0x559e8c906420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.358+0000 7f60b3e00700 1 -- 172.21.15.78:0/1397565817 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e8c814ea0 con 0x559e8c8fc680 2024-03-14T10:02:55.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.358+0000 7f60b8c00700 1 --2- 172.21.15.78:0/1397565817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c906ec0 0x559e8c906420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:02:55.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.358+0000 7f60b3e00700 1 --2- 172.21.15.78:0/1397565817 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c8fc680 0x559e8c90db50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f60a400e3d0 tx=0x7f60a4029040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ebe81fe1f9792ba server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.358+0000 7f60a9c00700 1 -- 172.21.15.78:0/1397565817 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a402c040 con 0x559e8c8fc680 2024-03-14T10:02:55.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.358+0000 7f60a9c00700 1 -- 172.21.15.78:0/1397565817 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60a40313d0 con 0x559e8c8fc680 2024-03-14T10:02:55.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.358+0000 7f60a9c00700 1 -- 172.21.15.78:0/1397565817 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4004fb0 con 0x559e8c8fc680 2024-03-14T10:02:55.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.359+0000 7f60ce7deb80 1 -- 172.21.15.78:0/1397565817 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c8fc680 msgr2=0x559e8c90db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.359+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/1397565817 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c8fc680 0x559e8c90db50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f60a400e3d0 tx=0x7f60a4029040 comp rx=0 tx=0).stop 2024-03-14T10:02:55.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.359+0000 7f60ce7deb80 1 -- 172.21.15.78:0/1397565817 shutdown_connections 2024-03-14T10:02:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.359+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/1397565817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c906ec0 0x559e8c906420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.359+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/1397565817 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c8fc680 0x559e8c90db50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.359+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/1397565817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fcfa0 0x559e8c8f3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.359+0000 7f60ce7deb80 1 -- 172.21.15.78:0/1397565817 >> 172.21.15.78:0/1397565817 conn(0x559e8c872220 msgr2=0x559e8c903470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.359+0000 7f60ce7deb80 1 -- 172.21.15.78:0/1397565817 shutdown_connections 2024-03-14T10:02:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.359+0000 7f60ce7deb80 1 -- 172.21.15.78:0/1397565817 wait complete. 2024-03-14T10:02:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.359+0000 7f60ce7deb80 1 -- start start 2024-03-14T10:02:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.360+0000 7f60ce7deb80 1 Processor -- start 2024-03-14T10:02:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.360+0000 7f60ce7deb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c8fc680 0x559e8c90b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.360+0000 7f60ce7deb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fcfa0 0x559e8c90be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.360+0000 7f60ce7deb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c906ec0 0x559e8c967700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.360+0000 7f60ce7deb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8c967c40 con 0x559e8c8fc680 2024-03-14T10:02:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.360+0000 7f60ce7deb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8c967d80 con 0x559e8c8fcfa0 2024-03-14T10:02:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.360+0000 7f60ce7deb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8c967f10 con 0x559e8c906ec0 2024-03-14T10:02:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.360+0000 7f60ce7deb80 10 client.0 _ll_register_callbacks cb 0x7f60a0099250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:55.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.360+0000 7f60b3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c8fc680 0x559e8c90b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.360+0000 7f60b3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c8fc680 0x559e8c90b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34904/0 (socket says 172.21.15.78:34904) 2024-03-14T10:02:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.360+0000 7f60b3e00700 1 -- 172.21.15.78:0/2918938747 learned_addr learned my addr 172.21.15.78:0/2918938747 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.360+0000 7f60b3400700 1 --2- 172.21.15.78:0/2918938747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fcfa0 0x559e8c90be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.360+0000 7f60b8c00700 1 --2- 172.21.15.78:0/2918938747 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c906ec0 0x559e8c967700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.361+0000 7f60b3e00700 1 -- 172.21.15.78:0/2918938747 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c906ec0 msgr2=0x559e8c967700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.361+0000 7f60b3e00700 1 --2- 172.21.15.78:0/2918938747 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c906ec0 0x559e8c967700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.361+0000 7f60b3e00700 1 -- 172.21.15.78:0/2918938747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fcfa0 msgr2=0x559e8c90be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.361+0000 7f60b3e00700 1 --2- 172.21.15.78:0/2918938747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fcfa0 0x559e8c90be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.361+0000 7f60b3e00700 1 -- 172.21.15.78:0/2918938747 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559e8c6d03b0 con 0x559e8c8fc680 2024-03-14T10:02:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.361+0000 7f60b3e00700 1 --2- 172.21.15.78:0/2918938747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c8fc680 0x559e8c90b940 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f60a40055b0 tx=0x7f60a4031540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb432b3220887370 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.361+0000 7f609d600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.361+0000 7f609d600700 1 -- 172.21.15.78:0/2918938747 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a402c040 con 0x559e8c8fc680 2024-03-14T10:02:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.361+0000 7f60ce7deb80 1 -- 172.21.15.78:0/2918938747 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60a4021070 con 0x559e8c8fc680 2024-03-14T10:02:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.361+0000 7f609d600700 1 -- 172.21.15.78:0/2918938747 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f60a4031780 con 0x559e8c8fc680 2024-03-14T10:02:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.361+0000 7f609d600700 1 -- 172.21.15.78:0/2918938747 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60a400eda0 con 0x559e8c8fc680 2024-03-14T10:02:55.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.362+0000 7f60ce7deb80 10 client.17328 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.362+0000 7f60ce7deb80 1 -- 172.21.15.78:0/2918938747 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559e8c6d4cd0 con 0x559e8c8fc680 2024-03-14T10:02:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.362+0000 7f609d600700 1 -- 172.21.15.78:0/2918938747 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60a4021070 con 0x559e8c8fc680 2024-03-14T10:02:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.362+0000 7f609d600700 1 -- 172.21.15.78:0/2918938747 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f60a4004c90 con 0x559e8c8fc680 2024-03-14T10:02:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.362+0000 7f60ce7deb80 10 client.17328 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.362+0000 7f60ce7deb80 10 client.17328 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:02:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.362+0000 7f60ce7deb80 20 client.17328 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.362+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/2918938747 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559e8c969440 0x559e8c96b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.362+0000 7f60ce7deb80 4 client.17328 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:02:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.362+0000 7f60ce7deb80 1 -- 172.21.15.78:0/2918938747 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559e8c69dcf0 con 0x559e8c969440 2024-03-14T10:02:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.362+0000 7f60b3400700 1 --2- 172.21.15.78:0/2918938747 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559e8c969440 0x559e8c96b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.363+0000 7f609d600700 10 client.17328 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:02:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.363+0000 7f60b3400700 1 --2- 172.21.15.78:0/2918938747 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559e8c969440 0x559e8c96b860 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ee22596fe9b5f0c7 server_cookie=bcae568240ac97f4 in_seq=0 out_seq=0 2024-03-14T10:02:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.363+0000 7f609d600700 1 -- 172.21.15.78:0/2918938747 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559e8c69dcf0 con 0x559e8c969440 2024-03-14T10:02:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.363+0000 7f609d600700 10 client.17328 handle_command_reply: tid=0 2024-03-14T10:02:55.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.363+0000 7f60ce7deb80 1 client.17328 shutdown 2024-03-14T10:02:55.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.363+0000 7f60ce7deb80 1 -- 172.21.15.78:0/2918938747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c8fc680 msgr2=0x559e8c90b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.363+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/2918938747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c8fc680 0x559e8c90b940 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f60a40055b0 tx=0x7f60a4031540 comp rx=0 tx=0).stop 2024-03-14T10:02:55.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.363+0000 7f60ce7deb80 1 -- 172.21.15.78:0/2918938747 shutdown_connections 2024-03-14T10:02:55.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.363+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/2918938747 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559e8c969440 0x559e8c96b860 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.363+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/2918938747 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559e8c906ec0 0x559e8c967700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.363+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/2918938747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e8c8fc680 0x559e8c90b940 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.363+0000 7f609d600700 0 client.17328 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:02:55.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.363+0000 7f60ce7deb80 1 --2- 172.21.15.78:0/2918938747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559e8c8fcfa0 0x559e8c90be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.363+0000 7f60ce7deb80 1 -- 172.21.15.78:0/2918938747 >> 172.21.15.78:0/2918938747 conn(0x559e8c872220 msgr2=0x559e8c8feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.363+0000 7f60ce7deb80 1 -- 172.21.15.78:0/2918938747 shutdown_connections 2024-03-14T10:02:55.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.363+0000 7f60ce7deb80 1 -- 172.21.15.78:0/2918938747 wait complete. 2024-03-14T10:02:55.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.364+0000 7f60ce7deb80 20 client.17328 trim_cache size 0 max 16384 2024-03-14T10:02:55.369 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:02:55.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.364+0000 7f60bac00700 1 -- 172.21.15.78:0/853828030 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f60980a0c00 msgr2=0x7f60980a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.364+0000 7f60bac00700 1 --2- 172.21.15.78:0/853828030 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f60980a0c00 0x7f60980a30c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009c60 tx=0x7f60a4009260 comp rx=0 tx=0).stop 2024-03-14T10:02:55.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.364+0000 7f60bac00700 1 -- 172.21.15.78:0/853828030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b4073b00 msgr2=0x7f60b419faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.364+0000 7f60bac00700 1 --2- 172.21.15.78:0/853828030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b4073b00 0x7f60b419faf0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f60a000e9a0 tx=0x7f60a000ecb0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.364+0000 7f60bac00700 1 -- 172.21.15.78:0/853828030 shutdown_connections 2024-03-14T10:02:55.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.364+0000 7f60bac00700 1 --2- 172.21.15.78:0/853828030 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f60980a0c00 0x7f60980a30c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.364+0000 7f60bac00700 1 --2- 172.21.15.78:0/853828030 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60b410d4f0 0x7f60b4199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.364+0000 7f60bac00700 1 --2- 172.21.15.78:0/853828030 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f60b40731e0 0x7f60b419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.364+0000 7f60bac00700 1 --2- 172.21.15.78:0/853828030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b4073b00 0x7f60b419faf0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.364+0000 7f60bac00700 1 -- 172.21.15.78:0/853828030 >> 172.21.15.78:0/853828030 conn(0x7f60b40fc7a0 msgr2=0x7f60b40fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.364+0000 7f60bac00700 1 -- 172.21.15.78:0/853828030 shutdown_connections 2024-03-14T10:02:55.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.364+0000 7f60bac00700 1 -- 172.21.15.78:0/853828030 wait complete. 2024-03-14T10:02:55.383 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T10:02:55.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:55 smithi078 ceph-mon[38143]: pgmap v269: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:55.440 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:55 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1075613888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:02:55.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:55 smithi152 ceph-mon[48279]: pgmap v269: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:55.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:55 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1075613888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:02:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:55 smithi132 ceph-mon[47959]: pgmap v269: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:55 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1075613888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:02:55.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.577+0000 7f4a26a00700 1 -- 172.21.15.78:0/941889632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a20100f30 msgr2=0x7f4a20101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.577+0000 7f4a26a00700 1 --2- 172.21.15.78:0/941889632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a20100f30 0x7f4a20101310 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009990 tx=0x7f4a10009ca0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.577+0000 7f4a26a00700 1 -- 172.21.15.78:0/941889632 shutdown_connections 2024-03-14T10:02:55.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.577+0000 7f4a26a00700 1 --2- 172.21.15.78:0/941889632 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a20106050 0x7f4a2010dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.577+0000 7f4a26a00700 1 --2- 172.21.15.78:0/941889632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a20100f30 0x7f4a20101310 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.577+0000 7f4a26a00700 1 --2- 172.21.15.78:0/941889632 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a201018e0 0x7f4a20105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.577+0000 7f4a26a00700 1 -- 172.21.15.78:0/941889632 >> 172.21.15.78:0/941889632 conn(0x7f4a200fc7b0 msgr2=0x7f4a200febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.577+0000 7f4a26a00700 1 -- 172.21.15.78:0/941889632 shutdown_connections 2024-03-14T10:02:55.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.577+0000 7f4a26a00700 1 -- 172.21.15.78:0/941889632 wait complete. 2024-03-14T10:02:55.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.578+0000 7f4a26a00700 1 Processor -- start 2024-03-14T10:02:55.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.578+0000 7f4a26a00700 1 -- start start 2024-03-14T10:02:55.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.578+0000 7f4a26a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a20100f30 0x7f4a2019f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.578+0000 7f4a26a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a20106050 0x7f4a2019fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.578+0000 7f4a26a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a20199720 0x7f4a20199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.578+0000 7f4a26a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2019a1a0 con 0x7f4a20199720 2024-03-14T10:02:55.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.578+0000 7f4a26a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2019a310 con 0x7f4a20106050 2024-03-14T10:02:55.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.578+0000 7f4a26a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2019a4d0 con 0x7f4a20100f30 2024-03-14T10:02:55.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.579+0000 7f4a24a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a20199720 0x7f4a20199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.579+0000 7f4a24a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a20199720 0x7f4a20199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34934/0 (socket says 172.21.15.78:34934) 2024-03-14T10:02:55.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.579+0000 7f4a24a00700 1 -- 172.21.15.78:0/3526734142 learned_addr learned my addr 172.21.15.78:0/3526734142 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:55.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.579+0000 7f4a1fe00700 1 --2- 172.21.15.78:0/3526734142 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a20100f30 0x7f4a2019f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.579+0000 7f4a17e00700 1 --2- 172.21.15.78:0/3526734142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a20106050 0x7f4a2019fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.579+0000 7f4a24a00700 1 -- 172.21.15.78:0/3526734142 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a20100f30 msgr2=0x7f4a2019f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.579+0000 7f4a24a00700 1 --2- 172.21.15.78:0/3526734142 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a20100f30 0x7f4a2019f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.579+0000 7f4a24a00700 1 -- 172.21.15.78:0/3526734142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a20106050 msgr2=0x7f4a2019fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.579+0000 7f4a24a00700 1 --2- 172.21.15.78:0/3526734142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a20106050 0x7f4a2019fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.579+0000 7f4a24a00700 1 -- 172.21.15.78:0/3526734142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a100095f0 con 0x7f4a20199720 2024-03-14T10:02:55.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.579+0000 7f4a24a00700 1 --2- 172.21.15.78:0/3526734142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a20199720 0x7f4a20199bd0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f4a18007ee0 tx=0x7f4a1800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.580+0000 7f4a1d600700 1 -- 172.21.15.78:0/3526734142 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a18017070 con 0x7f4a20199720 2024-03-14T10:02:55.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.580+0000 7f4a1d600700 1 -- 172.21.15.78:0/3526734142 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a1800f040 con 0x7f4a20199720 2024-03-14T10:02:55.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.580+0000 7f4a1d600700 1 -- 172.21.15.78:0/3526734142 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a18005580 con 0x7f4a20199720 2024-03-14T10:02:55.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.580+0000 7f4a26a00700 1 -- 172.21.15.78:0/3526734142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a201a64c0 con 0x7f4a20199720 2024-03-14T10:02:55.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.580+0000 7f4a26a00700 1 -- 172.21.15.78:0/3526734142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a201a6950 con 0x7f4a20199720 2024-03-14T10:02:55.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.584+0000 7f4a1d600700 1 -- 172.21.15.78:0/3526734142 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f4a1800db20 con 0x7f4a20199720 2024-03-14T10:02:55.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.585+0000 7f4a3a442b80 1 Processor -- start 2024-03-14T10:02:55.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.585+0000 7f4a3a442b80 1 -- start start 2024-03-14T10:02:55.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.585+0000 7f4a3a442b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838680 0x558350838a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.585+0000 7f4a3a442b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350838fa0 0x558350842980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.585+0000 7f4a1d600700 1 --2- 172.21.15.78:0/3526734142 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4a040a0c00 0x7f4a040a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.585+0000 7f4a3a442b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350842ec0 0x5583508452b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.585+0000 7f4a3a442b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558350791990 con 0x558350838680 2024-03-14T10:02:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.586+0000 7f4a3a442b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55835078f350 con 0x558350838fa0 2024-03-14T10:02:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.586+0000 7f4a3a442b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55835078ac40 con 0x558350842ec0 2024-03-14T10:02:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.586+0000 7f4a1d600700 1 -- 172.21.15.78:0/3526734142 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f4a180cbe80 con 0x7f4a20199720 2024-03-14T10:02:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.586+0000 7f4a17e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838680 0x558350838a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.586+0000 7f4a24a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350842ec0 0x5583508452b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.586+0000 7f4a17e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838680 0x558350838a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34944/0 (socket says 172.21.15.78:34944) 2024-03-14T10:02:55.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.586+0000 7f4a17e00700 1 -- 172.21.15.78:0/3742186628 learned_addr learned my addr 172.21.15.78:0/3742186628 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:55.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.586+0000 7f4a1fe00700 1 --2- 172.21.15.78:0/3742186628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350838fa0 0x558350842980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.586+0000 7f4a17e00700 1 --2- 172.21.15.78:0/3526734142 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4a040a0c00 0x7f4a040a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.586+0000 7f4a24a00700 1 -- 172.21.15.78:0/3742186628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350838fa0 msgr2=0x558350842980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.586+0000 7f4a24a00700 1 --2- 172.21.15.78:0/3742186628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350838fa0 0x558350842980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.586+0000 7f4a24a00700 1 -- 172.21.15.78:0/3742186628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838680 msgr2=0x558350838a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.586+0000 7f4a24a00700 1 --2- 172.21.15.78:0/3742186628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838680 0x558350838a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.586+0000 7f4a24a00700 1 -- 172.21.15.78:0/3742186628 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55835060c3b0 con 0x558350842ec0 2024-03-14T10:02:55.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.587+0000 7f4a24a00700 1 --2- 172.21.15.78:0/3742186628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350842ec0 0x5583508452b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4a180c9200 tx=0x7f4a18004f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57446ffc3f17f49a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.587+0000 7f4a17e00700 1 --2- 172.21.15.78:0/3526734142 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4a040a0c00 0x7f4a040a30c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4a08009fd0 tx=0x7f4a08009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.587+0000 7f4a15200700 1 -- 172.21.15.78:0/3742186628 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a18099500 con 0x558350842ec0 2024-03-14T10:02:55.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.587+0000 7f4a15200700 1 -- 172.21.15.78:0/3742186628 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a18098e00 con 0x558350842ec0 2024-03-14T10:02:55.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.587+0000 7f4a15200700 1 -- 172.21.15.78:0/3742186628 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a18095510 con 0x558350842ec0 2024-03-14T10:02:55.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.588+0000 7f4a3a442b80 1 -- 172.21.15.78:0/3742186628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350842ec0 msgr2=0x5583508452b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.588+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/3742186628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350842ec0 0x5583508452b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4a180c9200 tx=0x7f4a18004f10 comp rx=0 tx=0).stop 2024-03-14T10:02:55.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.588+0000 7f4a3a442b80 1 -- 172.21.15.78:0/3742186628 shutdown_connections 2024-03-14T10:02:55.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.588+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/3742186628 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350842ec0 0x5583508452b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.588+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/3742186628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838680 0x558350838a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.588+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/3742186628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350838fa0 0x558350842980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.588+0000 7f4a3a442b80 1 -- 172.21.15.78:0/3742186628 >> 172.21.15.78:0/3742186628 conn(0x5583507ae220 msgr2=0x558350833aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.588+0000 7f4a3a442b80 1 -- 172.21.15.78:0/3742186628 shutdown_connections 2024-03-14T10:02:55.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.588+0000 7f4a3a442b80 1 -- 172.21.15.78:0/3742186628 wait complete. 2024-03-14T10:02:55.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.588+0000 7f4a3a442b80 1 -- start start 2024-03-14T10:02:55.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.588+0000 7f4a3a442b80 1 Processor -- start 2024-03-14T10:02:55.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.589+0000 7f4a3a442b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838680 0x5583508428f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.589+0000 7f4a3a442b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838fa0 0x55835083fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.589+0000 7f4a3a442b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350842ec0 0x5583508403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.589+0000 7f4a3a442b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558350726d80 con 0x558350838680 2024-03-14T10:02:55.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.589+0000 7f4a3a442b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558350726a20 con 0x558350842ec0 2024-03-14T10:02:55.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.589+0000 7f4a3a442b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55835060e6a0 con 0x558350838fa0 2024-03-14T10:02:55.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.589+0000 7f4a3a442b80 10 client.0 _ll_register_callbacks cb 0x5583505923a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:55.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.589+0000 7f4a1fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838680 0x5583508428f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.589+0000 7f4a17e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838fa0 0x55835083fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.589+0000 7f4a17e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838fa0 0x55835083fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:58698/0 (socket says 172.21.15.78:58698) 2024-03-14T10:02:55.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.589+0000 7f4a17e00700 1 -- 172.21.15.78:0/1067447517 learned_addr learned my addr 172.21.15.78:0/1067447517 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:55.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.589+0000 7f4a1fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838680 0x5583508428f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34960/0 (socket says 172.21.15.78:34960) 2024-03-14T10:02:55.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.589+0000 7f4a24a00700 1 --2- 172.21.15.78:0/1067447517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350842ec0 0x5583508403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.590+0000 7f4a17e00700 1 -- 172.21.15.78:0/1067447517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350842ec0 msgr2=0x5583508403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.590+0000 7f4a17e00700 1 --2- 172.21.15.78:0/1067447517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350842ec0 0x5583508403c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.590+0000 7f4a17e00700 1 -- 172.21.15.78:0/1067447517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838680 msgr2=0x5583508428f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.590+0000 7f4a17e00700 1 --2- 172.21.15.78:0/1067447517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838680 0x5583508428f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.590+0000 7f4a17e00700 1 -- 172.21.15.78:0/1067447517 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a201a6950 con 0x558350838fa0 2024-03-14T10:02:55.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.590+0000 7f4a17e00700 1 --2- 172.21.15.78:0/1067447517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838fa0 0x55835083fc80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4a08004550 tx=0x7f4a08004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5be7ff48c8b38af9 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.590+0000 7f4a0d600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:02:55.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.590+0000 7f4a0d600700 1 -- 172.21.15.78:0/1067447517 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a08004580 con 0x558350838fa0 2024-03-14T10:02:55.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.590+0000 7f4a3a442b80 1 -- 172.21.15.78:0/1067447517 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55835082b720 con 0x558350838fa0 2024-03-14T10:02:55.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.590+0000 7f4a0d600700 1 -- 172.21.15.78:0/1067447517 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f4a08003890 con 0x558350838fa0 2024-03-14T10:02:55.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.590+0000 7f4a0d600700 1 -- 172.21.15.78:0/1067447517 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a08003a50 con 0x558350838fa0 2024-03-14T10:02:55.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.590+0000 7f4a3a442b80 10 client.27391 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:55.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.591+0000 7f4a3a442b80 1 -- 172.21.15.78:0/1067447517 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5583505b8b60 con 0x558350838fa0 2024-03-14T10:02:55.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.591+0000 7f4a0d600700 1 -- 172.21.15.78:0/1067447517 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55835082b720 con 0x558350838fa0 2024-03-14T10:02:55.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.591+0000 7f4a0d600700 1 -- 172.21.15.78:0/1067447517 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4a08005090 con 0x558350838fa0 2024-03-14T10:02:55.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.591+0000 7f4a3a442b80 10 client.27391 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:55.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.591+0000 7f4a3a442b80 10 client.27391 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:02:55.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.591+0000 7f4a3a442b80 20 client.27391 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:55.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.591+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/1067447517 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x558350852a80 0x558350854ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.591+0000 7f4a3a442b80 4 client.27391 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:02:55.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.591+0000 7f4a3a442b80 1 -- 172.21.15.78:0/1067447517 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558350783700 con 0x558350852a80 2024-03-14T10:02:55.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.591+0000 7f4a1fe00700 1 --2- 172.21.15.78:0/1067447517 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x558350852a80 0x558350854ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.592+0000 7f4a1fe00700 1 --2- 172.21.15.78:0/1067447517 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x558350852a80 0x558350854ea0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1aefb5ba734954b1 server_cookie=b3c8835e1331d8c6 in_seq=0 out_seq=0 2024-03-14T10:02:55.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.592+0000 7f4a0d600700 10 client.27391 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:02:55.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.592+0000 7f4a0d600700 1 -- 172.21.15.78:0/1067447517 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x558350783700 con 0x558350852a80 2024-03-14T10:02:55.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.592+0000 7f4a0d600700 10 client.27391 handle_command_reply: tid=0 2024-03-14T10:02:55.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.592+0000 7f4a3a442b80 1 client.27391 shutdown 2024-03-14T10:02:55.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.593+0000 7f4a3a442b80 1 -- 172.21.15.78:0/1067447517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838fa0 msgr2=0x55835083fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.593+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/1067447517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838fa0 0x55835083fc80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4a08004550 tx=0x7f4a08004750 comp rx=0 tx=0).stop 2024-03-14T10:02:55.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.593+0000 7f4a3a442b80 1 -- 172.21.15.78:0/1067447517 shutdown_connections 2024-03-14T10:02:55.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.593+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/1067447517 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x558350852a80 0x558350854ea0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.593+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/1067447517 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350842ec0 0x5583508403c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.593+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/1067447517 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838680 0x5583508428f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.593+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/1067447517 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838fa0 0x55835083fc80 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.593+0000 7f4a0d600700 0 client.27391 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:02:55.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.593+0000 7f4a3a442b80 1 -- 172.21.15.78:0/1067447517 >> 172.21.15.78:0/1067447517 conn(0x5583507ae220 msgr2=0x558350833690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.593+0000 7f4a3a442b80 1 -- 172.21.15.78:0/1067447517 shutdown_connections 2024-03-14T10:02:55.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.593+0000 7f4a3a442b80 1 -- 172.21.15.78:0/1067447517 wait complete. 2024-03-14T10:02:55.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.593+0000 7f4a3a442b80 20 client.27391 trim_cache size 0 max 16384 2024-03-14T10:02:55.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.606+0000 7f4a3a442b80 1 Processor -- start 2024-03-14T10:02:55.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.606+0000 7f4a3a442b80 1 -- start start 2024-03-14T10:02:55.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.607+0000 7f4a3a442b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838680 0x558350859800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.607+0000 7f4a3a442b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838fa0 0x558350849b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.607+0000 7f4a3a442b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350842ec0 0x55835084e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.607+0000 7f4a3a442b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55835083a310 con 0x558350838fa0 2024-03-14T10:02:55.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.607+0000 7f4a3a442b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558350855230 con 0x558350842ec0 2024-03-14T10:02:55.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.607+0000 7f4a3a442b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558350835060 con 0x558350838680 2024-03-14T10:02:55.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.607+0000 7f4a17e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838fa0 0x558350849b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.607+0000 7f4a1fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838680 0x558350859800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.607+0000 7f4a24a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350842ec0 0x55835084e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.607+0000 7f4a1fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838680 0x558350859800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:58710/0 (socket says 172.21.15.78:58710) 2024-03-14T10:02:55.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.607+0000 7f4a1fe00700 1 -- 172.21.15.78:0/3216635369 learned_addr learned my addr 172.21.15.78:0/3216635369 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:55.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.607+0000 7f4a17e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838fa0 0x558350849b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34966/0 (socket says 172.21.15.78:34966) 2024-03-14T10:02:55.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.607+0000 7f4a17e00700 1 -- 172.21.15.78:0/3216635369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838680 msgr2=0x558350859800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.608+0000 7f4a17e00700 1 --2- 172.21.15.78:0/3216635369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838680 0x558350859800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.608+0000 7f4a17e00700 1 -- 172.21.15.78:0/3216635369 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350842ec0 msgr2=0x55835084e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.608+0000 7f4a17e00700 1 --2- 172.21.15.78:0/3216635369 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350842ec0 0x55835084e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.608+0000 7f4a17e00700 1 -- 172.21.15.78:0/3216635369 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558350750ea0 con 0x558350838fa0 2024-03-14T10:02:55.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.608+0000 7f4a1fe00700 1 --2- 172.21.15.78:0/3216635369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838680 0x558350859800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:55.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.608+0000 7f4a24a00700 1 --2- 172.21.15.78:0/3216635369 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350842ec0 0x55835084e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:55.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.608+0000 7f4a17e00700 1 --2- 172.21.15.78:0/3216635369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838fa0 0x558350849b50 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f4a08005470 tx=0x7f4a08009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b869367128a681 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.608+0000 7f4a15c00700 1 -- 172.21.15.78:0/3216635369 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a08011870 con 0x558350838fa0 2024-03-14T10:02:55.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.608+0000 7f4a15c00700 1 -- 172.21.15.78:0/3216635369 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a08004580 con 0x558350838fa0 2024-03-14T10:02:55.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.608+0000 7f4a15c00700 1 -- 172.21.15.78:0/3216635369 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a080204b0 con 0x558350838fa0 2024-03-14T10:02:55.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.608+0000 7f4a3a442b80 1 -- 172.21.15.78:0/3216635369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838fa0 msgr2=0x558350849b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.609+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/3216635369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838fa0 0x558350849b50 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f4a08005470 tx=0x7f4a08009f30 comp rx=0 tx=0).stop 2024-03-14T10:02:55.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.609+0000 7f4a3a442b80 1 -- 172.21.15.78:0/3216635369 shutdown_connections 2024-03-14T10:02:55.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.609+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/3216635369 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350842ec0 0x55835084e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.609+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/3216635369 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838680 0x558350859800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.609+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/3216635369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350838fa0 0x558350849b50 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.609+0000 7f4a3a442b80 1 -- 172.21.15.78:0/3216635369 >> 172.21.15.78:0/3216635369 conn(0x5583507ae220 msgr2=0x558350832180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.609+0000 7f4a3a442b80 1 -- 172.21.15.78:0/3216635369 shutdown_connections 2024-03-14T10:02:55.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.609+0000 7f4a3a442b80 1 -- 172.21.15.78:0/3216635369 wait complete. 2024-03-14T10:02:55.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.609+0000 7f4a3a442b80 1 -- start start 2024-03-14T10:02:55.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.609+0000 7f4a3a442b80 1 Processor -- start 2024-03-14T10:02:55.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a3a442b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350838680 0x558350849780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a3a442b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838fa0 0x558350847c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a3a442b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350842ec0 0x5583508481b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a3a442b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583508486f0 con 0x558350842ec0 2024-03-14T10:02:55.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a3a442b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558350848860 con 0x558350838680 2024-03-14T10:02:55.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a3a442b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583508489d0 con 0x558350838fa0 2024-03-14T10:02:55.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a3a442b80 10 client.0 _ll_register_callbacks cb 0x7f4a08004340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:55.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a24a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350842ec0 0x5583508481b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a17e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838fa0 0x558350847c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a24a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350842ec0 0x5583508481b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34982/0 (socket says 172.21.15.78:34982) 2024-03-14T10:02:55.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a24a00700 1 -- 172.21.15.78:0/90764787 learned_addr learned my addr 172.21.15.78:0/90764787 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:55.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a1fe00700 1 --2- 172.21.15.78:0/90764787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350838680 0x558350849780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a24a00700 1 -- 172.21.15.78:0/90764787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838fa0 msgr2=0x558350847c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a24a00700 1 --2- 172.21.15.78:0/90764787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838fa0 0x558350847c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a24a00700 1 -- 172.21.15.78:0/90764787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350838680 msgr2=0x558350849780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a24a00700 1 --2- 172.21.15.78:0/90764787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350838680 0x558350849780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a24a00700 1 -- 172.21.15.78:0/90764787 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2010e440 con 0x558350842ec0 2024-03-14T10:02:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.610+0000 7f4a17e00700 1 --2- 172.21.15.78:0/90764787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838fa0 0x558350847c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.611+0000 7f4a24a00700 1 --2- 172.21.15.78:0/90764787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350842ec0 0x5583508481b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f4a180c92b0 tx=0x7f4a18005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929bf1502d92130d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.611+0000 7f4a0d600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.611+0000 7f4a0d600700 1 -- 172.21.15.78:0/90764787 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a18099500 con 0x558350842ec0 2024-03-14T10:02:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.611+0000 7f4a0d600700 1 -- 172.21.15.78:0/90764787 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f4a1809cc30 con 0x558350842ec0 2024-03-14T10:02:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.611+0000 7f4a3a442b80 1 -- 172.21.15.78:0/90764787 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a18098e00 con 0x558350842ec0 2024-03-14T10:02:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.611+0000 7f4a0d600700 1 -- 172.21.15.78:0/90764787 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a180a1280 con 0x558350842ec0 2024-03-14T10:02:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.611+0000 7f4a3a442b80 10 client.17346 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.611+0000 7f4a3a442b80 1 -- 172.21.15.78:0/90764787 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55835060c3b0 con 0x558350842ec0 2024-03-14T10:02:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.611+0000 7f4a0d600700 1 -- 172.21.15.78:0/90764787 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a18098e00 con 0x558350842ec0 2024-03-14T10:02:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.611+0000 7f4a0d600700 1 -- 172.21.15.78:0/90764787 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4a18099060 con 0x558350842ec0 2024-03-14T10:02:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.611+0000 7f4a3a442b80 10 client.17346 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:55.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.611+0000 7f4a3a442b80 10 client.17346 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:02:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.611+0000 7f4a3a442b80 20 client.17346 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.611+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/90764787 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5583508a56c0 0x5583508a7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.611+0000 7f4a3a442b80 4 client.17346 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:02:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.611+0000 7f4a3a442b80 1 -- 172.21.15.78:0/90764787 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5583505d9cf0 con 0x5583508a56c0 2024-03-14T10:02:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.612+0000 7f4a1fe00700 1 --2- 172.21.15.78:0/90764787 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5583508a56c0 0x5583508a7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.612+0000 7f4a1fe00700 1 --2- 172.21.15.78:0/90764787 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5583508a56c0 0x5583508a7ae0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b3e7200a46376843 server_cookie=6f48200f1342abba in_seq=0 out_seq=0 2024-03-14T10:02:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.612+0000 7f4a0d600700 10 client.17346 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:02:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.612+0000 7f4a0d600700 1 -- 172.21.15.78:0/90764787 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5583505d9cf0 con 0x5583508a56c0 2024-03-14T10:02:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.612+0000 7f4a0d600700 10 client.17346 handle_command_reply: tid=0 2024-03-14T10:02:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.612+0000 7f4a3a442b80 1 client.17346 shutdown 2024-03-14T10:02:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.612+0000 7f4a3a442b80 1 -- 172.21.15.78:0/90764787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350842ec0 msgr2=0x5583508481b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.612+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/90764787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350842ec0 0x5583508481b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f4a180c92b0 tx=0x7f4a18005000 comp rx=0 tx=0).stop 2024-03-14T10:02:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.612+0000 7f4a3a442b80 1 -- 172.21.15.78:0/90764787 shutdown_connections 2024-03-14T10:02:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.612+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/90764787 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x5583508a56c0 0x5583508a7ae0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/90764787 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558350838680 0x558350849780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a0d600700 0 client.17346 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:02:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/90764787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558350842ec0 0x5583508481b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a3a442b80 1 --2- 172.21.15.78:0/90764787 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558350838fa0 0x558350847c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a3a442b80 1 -- 172.21.15.78:0/90764787 >> 172.21.15.78:0/90764787 conn(0x5583507ae220 msgr2=0x55835084e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a3a442b80 1 -- 172.21.15.78:0/90764787 shutdown_connections 2024-03-14T10:02:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a3a442b80 1 -- 172.21.15.78:0/90764787 wait complete. 2024-03-14T10:02:55.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a3a442b80 20 client.17346 trim_cache size 0 max 16384 2024-03-14T10:02:55.622 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:02:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a26a00700 1 -- 172.21.15.78:0/3526734142 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4a040a0c00 msgr2=0x7f4a040a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a26a00700 1 --2- 172.21.15.78:0/3526734142 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4a040a0c00 0x7f4a040a30c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4a08009fd0 tx=0x7f4a08009260 comp rx=0 tx=0).stop 2024-03-14T10:02:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a26a00700 1 -- 172.21.15.78:0/3526734142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a20199720 msgr2=0x7f4a20199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a26a00700 1 --2- 172.21.15.78:0/3526734142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a20199720 0x7f4a20199bd0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f4a18007ee0 tx=0x7f4a1800d3b0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a26a00700 1 -- 172.21.15.78:0/3526734142 shutdown_connections 2024-03-14T10:02:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a26a00700 1 --2- 172.21.15.78:0/3526734142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a20199720 0x7f4a20199bd0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a26a00700 1 --2- 172.21.15.78:0/3526734142 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4a20100f30 0x7f4a2019f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a26a00700 1 --2- 172.21.15.78:0/3526734142 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4a040a0c00 0x7f4a040a30c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a26a00700 1 --2- 172.21.15.78:0/3526734142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a20106050 0x7f4a2019fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.613+0000 7f4a26a00700 1 -- 172.21.15.78:0/3526734142 >> 172.21.15.78:0/3526734142 conn(0x7f4a200fc7b0 msgr2=0x7f4a200fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.614+0000 7f4a26a00700 1 -- 172.21.15.78:0/3526734142 shutdown_connections 2024-03-14T10:02:55.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.614+0000 7f4a26a00700 1 -- 172.21.15.78:0/3526734142 wait complete. 2024-03-14T10:02:55.632 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T10:02:55.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.826+0000 7f2acec00700 1 -- 172.21.15.78:0/776728119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8100f20 msgr2=0x7f2ac8101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.826+0000 7f2acec00700 1 --2- 172.21.15.78:0/776728119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8100f20 0x7f2ac8101300 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8009990 tx=0x7f2ab8009ca0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.826+0000 7f2acec00700 1 -- 172.21.15.78:0/776728119 shutdown_connections 2024-03-14T10:02:55.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.826+0000 7f2acec00700 1 --2- 172.21.15.78:0/776728119 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ac8106040 0x7f2ac810dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.826+0000 7f2acec00700 1 --2- 172.21.15.78:0/776728119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac8100f20 0x7f2ac8101300 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.826+0000 7f2acec00700 1 --2- 172.21.15.78:0/776728119 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ac81018d0 0x7f2ac8105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.826+0000 7f2acec00700 1 -- 172.21.15.78:0/776728119 >> 172.21.15.78:0/776728119 conn(0x7f2ac80fc7a0 msgr2=0x7f2ac80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.826+0000 7f2acec00700 1 -- 172.21.15.78:0/776728119 shutdown_connections 2024-03-14T10:02:55.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.826+0000 7f2acec00700 1 -- 172.21.15.78:0/776728119 wait complete. 2024-03-14T10:02:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.827+0000 7f2acec00700 1 Processor -- start 2024-03-14T10:02:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.827+0000 7f2acec00700 1 -- start start 2024-03-14T10:02:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.828+0000 7f2acec00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ac8100f20 0x7f2ac819f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.828+0000 7f2acec00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ac8106040 0x7f2ac819faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.828+0000 7f2acec00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac81996e0 0x7f2ac8199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.828+0000 7f2acec00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac819a160 con 0x7f2ac81996e0 2024-03-14T10:02:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.828+0000 7f2acec00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac819a2a0 con 0x7f2ac8100f20 2024-03-14T10:02:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.828+0000 7f2acec00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac819a460 con 0x7f2ac8106040 2024-03-14T10:02:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.828+0000 7f2ac7e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ac8100f20 0x7f2ac819f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.828+0000 7f2ac7e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ac8100f20 0x7f2ac819f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:36328/0 (socket says 172.21.15.78:36328) 2024-03-14T10:02:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.828+0000 7f2ac7e00700 1 -- 172.21.15.78:0/2607711078 learned_addr learned my addr 172.21.15.78:0/2607711078 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.828+0000 7f2ac7400700 1 --2- 172.21.15.78:0/2607711078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ac8106040 0x7f2ac819faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.828+0000 7f2accc00700 1 --2- 172.21.15.78:0/2607711078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac81996e0 0x7f2ac8199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.828+0000 7f2ac7e00700 1 -- 172.21.15.78:0/2607711078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ac8106040 msgr2=0x7f2ac819faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.828+0000 7f2ac7e00700 1 --2- 172.21.15.78:0/2607711078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ac8106040 0x7f2ac819faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.828+0000 7f2ac7e00700 1 -- 172.21.15.78:0/2607711078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac81996e0 msgr2=0x7f2ac8199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.828+0000 7f2ac7e00700 1 --2- 172.21.15.78:0/2607711078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac81996e0 0x7f2ac8199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.828+0000 7f2ac7e00700 1 -- 172.21.15.78:0/2607711078 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab80095f0 con 0x7f2ac8100f20 2024-03-14T10:02:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.829+0000 7f2ac7e00700 1 --2- 172.21.15.78:0/2607711078 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ac8100f20 0x7f2ac819f5b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8000c00 tx=0x7f2ab8005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.829+0000 7f2ac4c00700 1 -- 172.21.15.78:0/2607711078 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab801d070 con 0x7f2ac8100f20 2024-03-14T10:02:55.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.829+0000 7f2acec00700 1 -- 172.21.15.78:0/2607711078 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac81a63c0 con 0x7f2ac8100f20 2024-03-14T10:02:55.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.829+0000 7f2ac4c00700 1 -- 172.21.15.78:0/2607711078 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ab8022470 con 0x7f2ac8100f20 2024-03-14T10:02:55.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.829+0000 7f2ac4c00700 1 -- 172.21.15.78:0/2607711078 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8017650 con 0x7f2ac8100f20 2024-03-14T10:02:55.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.829+0000 7f2acec00700 1 -- 172.21.15.78:0/2607711078 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac81a6840 con 0x7f2ac8100f20 2024-03-14T10:02:55.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.832+0000 7f2ac4c00700 1 -- 172.21.15.78:0/2607711078 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f2ab80225e0 con 0x7f2ac8100f20 2024-03-14T10:02:55.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.833+0000 7f2ae26dab80 1 Processor -- start 2024-03-14T10:02:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.833+0000 7f2ae26dab80 1 -- start start 2024-03-14T10:02:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.833+0000 7f2ac4c00700 1 --2- 172.21.15.78:0/2607711078 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2aac0a0c00 0x7f2aac0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.833+0000 7f2ae26dab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fbda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.833+0000 7f2ae26dab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fc7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2ae26dab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2ae26dab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c3f16990 con 0x55b7c3fbd680 2024-03-14T10:02:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2ae26dab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c3f14350 con 0x55b7c3fbdfa0 2024-03-14T10:02:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2ae26dab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c3f0fc40 con 0x55b7c3fc7ec0 2024-03-14T10:02:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2ac4c00700 1 -- 172.21.15.78:0/2607711078 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f2ab80cdad0 con 0x7f2ac8100f20 2024-03-14T10:02:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2ac7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fbda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2ac7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fbda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35016/0 (socket says 172.21.15.78:35016) 2024-03-14T10:02:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2ac7e00700 1 -- 172.21.15.78:0/151631296 learned_addr learned my addr 172.21.15.78:0/151631296 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2ac7400700 1 --2- 172.21.15.78:0/151631296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fc7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2accc00700 1 --2- 172.21.15.78:0/2607711078 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2aac0a0c00 0x7f2aac0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2accc00700 1 --2- 172.21.15.78:0/151631296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2ac7e00700 1 -- 172.21.15.78:0/151631296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 msgr2=0x55b7c3fca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2ac7e00700 1 --2- 172.21.15.78:0/151631296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2ac7e00700 1 -- 172.21.15.78:0/151631296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbdfa0 msgr2=0x55b7c3fc7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2ac7e00700 1 --2- 172.21.15.78:0/151631296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fc7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2ac7e00700 1 -- 172.21.15.78:0/151631296 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b7c3d913b0 con 0x55b7c3fbd680 2024-03-14T10:02:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.834+0000 7f2accc00700 1 --2- 172.21.15.78:0/151631296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T10:02:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.835+0000 7f2ac7e00700 1 --2- 172.21.15.78:0/151631296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fbda60 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8097610 tx=0x7f2ab8096fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2ac50633300658 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.835+0000 7f2accc00700 1 --2- 172.21.15.78:0/2607711078 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2aac0a0c00 0x7f2aac0a30c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0009ec0 tx=0x7f2ac0009920 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.835+0000 7f2abd200700 1 -- 172.21.15.78:0/151631296 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab809a500 con 0x55b7c3fbd680 2024-03-14T10:02:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.835+0000 7f2abd200700 1 -- 172.21.15.78:0/151631296 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ab809b2a0 con 0x55b7c3fbd680 2024-03-14T10:02:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.835+0000 7f2abd200700 1 -- 172.21.15.78:0/151631296 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab809e9d0 con 0x55b7c3fbd680 2024-03-14T10:02:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.835+0000 7f2ae26dab80 1 -- 172.21.15.78:0/151631296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 msgr2=0x55b7c3fbda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.835+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/151631296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fbda60 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8097610 tx=0x7f2ab8096fb0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.835+0000 7f2ae26dab80 1 -- 172.21.15.78:0/151631296 shutdown_connections 2024-03-14T10:02:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.835+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/151631296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.835+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/151631296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fbda60 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.835+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/151631296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fc7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.835+0000 7f2ae26dab80 1 -- 172.21.15.78:0/151631296 >> 172.21.15.78:0/151631296 conn(0x55b7c3f33220 msgr2=0x55b7c3fb8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.836+0000 7f2ae26dab80 1 -- 172.21.15.78:0/151631296 shutdown_connections 2024-03-14T10:02:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.836+0000 7f2ae26dab80 1 -- 172.21.15.78:0/151631296 wait complete. 2024-03-14T10:02:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.836+0000 7f2ae26dab80 1 -- start start 2024-03-14T10:02:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.836+0000 7f2ae26dab80 1 Processor -- start 2024-03-14T10:02:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ae26dab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fc7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ae26dab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fc4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ae26dab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fc5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ae26dab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c3eabd80 con 0x55b7c3fbd680 2024-03-14T10:02:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ae26dab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c3eaba20 con 0x55b7c3fbdfa0 2024-03-14T10:02:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ae26dab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c3d936a0 con 0x55b7c3fc7ec0 2024-03-14T10:02:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ae26dab80 10 client.0 _ll_register_callbacks cb 0x55b7c3d173a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ac7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fc7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ac7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fc7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35032/0 (socket says 172.21.15.78:35032) 2024-03-14T10:02:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ac7e00700 1 -- 172.21.15.78:0/2055725784 learned_addr learned my addr 172.21.15.78:0/2055725784 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2accc00700 1 --2- 172.21.15.78:0/2055725784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fc5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ac7400700 1 --2- 172.21.15.78:0/2055725784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fc4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ac7e00700 1 -- 172.21.15.78:0/2055725784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 msgr2=0x55b7c3fc5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ac7e00700 1 --2- 172.21.15.78:0/2055725784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fc5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ac7e00700 1 -- 172.21.15.78:0/2055725784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbdfa0 msgr2=0x55b7c3fc4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ac7e00700 1 --2- 172.21.15.78:0/2055725784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fc4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ac7e00700 1 -- 172.21.15.78:0/2055725784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac81a6840 con 0x55b7c3fbd680 2024-03-14T10:02:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2accc00700 1 --2- 172.21.15.78:0/2055725784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fc5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:02:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ac7e00700 1 --2- 172.21.15.78:0/2055725784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fc7850 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2ab809ab30 tx=0x7f2ab8005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da1bc95bc415b8cf server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.837+0000 7f2ab1600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.838+0000 7f2ab1600700 1 -- 172.21.15.78:0/2055725784 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8066b30 con 0x55b7c3fbd680 2024-03-14T10:02:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.838+0000 7f2ab1600700 1 -- 172.21.15.78:0/2055725784 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f2ab80a2f90 con 0x55b7c3fbd680 2024-03-14T10:02:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.838+0000 7f2ae26dab80 1 -- 172.21.15.78:0/2055725784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b7c3fb0720 con 0x55b7c3fbd680 2024-03-14T10:02:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.838+0000 7f2ab1600700 1 -- 172.21.15.78:0/2055725784 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ab809e9d0 con 0x55b7c3fbd680 2024-03-14T10:02:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.838+0000 7f2ae26dab80 10 client.17358 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:55.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.838+0000 7f2ae26dab80 1 -- 172.21.15.78:0/2055725784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b7c3d3db60 con 0x55b7c3fbd680 2024-03-14T10:02:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.838+0000 7f2ab1600700 1 -- 172.21.15.78:0/2055725784 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b7c3fb0720 con 0x55b7c3fbd680 2024-03-14T10:02:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.838+0000 7f2ab1600700 1 -- 172.21.15.78:0/2055725784 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2ab809ae00 con 0x55b7c3fbd680 2024-03-14T10:02:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.838+0000 7f2ae26dab80 10 client.17358 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.838+0000 7f2ae26dab80 10 client.17358 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:02:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.838+0000 7f2ae26dab80 20 client.17358 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.838+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/2055725784 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b7c3fd71e0 0x55b7c3fd95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.838+0000 7f2ae26dab80 4 client.17358 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:02:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.839+0000 7f2ae26dab80 1 -- 172.21.15.78:0/2055725784 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b7c3f08700 con 0x55b7c3fd71e0 2024-03-14T10:02:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.839+0000 7f2ac7400700 1 --2- 172.21.15.78:0/2055725784 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b7c3fd71e0 0x55b7c3fd95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.839+0000 7f2ac7400700 1 --2- 172.21.15.78:0/2055725784 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b7c3fd71e0 0x55b7c3fd95d0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ec7e440cf96f7ab server_cookie=888b46e9bd8e5399 in_seq=0 out_seq=0 2024-03-14T10:02:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.839+0000 7f2ab1600700 10 client.17358 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.840+0000 7f2ab1600700 1 -- 172.21.15.78:0/2055725784 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55b7c3f08700 con 0x55b7c3fd71e0 2024-03-14T10:02:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.840+0000 7f2ab1600700 10 client.17358 handle_command_reply: tid=0 2024-03-14T10:02:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.840+0000 7f2ae26dab80 1 client.17358 shutdown 2024-03-14T10:02:55.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.840+0000 7f2ae26dab80 1 -- 172.21.15.78:0/2055725784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 msgr2=0x55b7c3fc7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.840+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/2055725784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fc7850 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2ab809ab30 tx=0x7f2ab8005ef0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.840+0000 7f2ae26dab80 1 -- 172.21.15.78:0/2055725784 shutdown_connections 2024-03-14T10:02:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.840+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/2055725784 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fc5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.840+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/2055725784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fc7850 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.840+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/2055725784 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b7c3fd71e0 0x55b7c3fd95d0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.840+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/2055725784 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fc4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.840+0000 7f2ae26dab80 1 -- 172.21.15.78:0/2055725784 >> 172.21.15.78:0/2055725784 conn(0x55b7c3f33220 msgr2=0x55b7c3fb8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.841+0000 7f2ab1600700 0 client.17358 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.841+0000 7f2ae26dab80 1 -- 172.21.15.78:0/2055725784 shutdown_connections 2024-03-14T10:02:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.841+0000 7f2ae26dab80 1 -- 172.21.15.78:0/2055725784 wait complete. 2024-03-14T10:02:55.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.841+0000 7f2ae26dab80 20 client.17358 trim_cache size 0 max 16384 2024-03-14T10:02:55.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.854+0000 7f2ae26dab80 1 Processor -- start 2024-03-14T10:02:55.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.854+0000 7f2ae26dab80 1 -- start start 2024-03-14T10:02:55.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.854+0000 7f2ae26dab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbd680 0x55b7c3fddf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.854+0000 7f2ae26dab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fc7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.854+0000 7f2ae26dab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fc4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.854+0000 7f2ae26dab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c3fc6140 con 0x55b7c3fc7ec0 2024-03-14T10:02:55.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.854+0000 7f2ae26dab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c3fd4200 con 0x55b7c3fbd680 2024-03-14T10:02:55.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.854+0000 7f2ae26dab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c3fb7800 con 0x55b7c3fbdfa0 2024-03-14T10:02:55.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.855+0000 7f2accc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fc4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.855+0000 7f2accc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fc4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35048/0 (socket says 172.21.15.78:35048) 2024-03-14T10:02:55.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.855+0000 7f2accc00700 1 -- 172.21.15.78:0/3834295060 learned_addr learned my addr 172.21.15.78:0/3834295060 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:55.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.855+0000 7f2ac7e00700 1 --2- 172.21.15.78:0/3834295060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbd680 0x55b7c3fddf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.855+0000 7f2ac7400700 1 --2- 172.21.15.78:0/3834295060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fc7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.855+0000 7f2accc00700 1 -- 172.21.15.78:0/3834295060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fbdfa0 msgr2=0x55b7c3fc7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.855+0000 7f2accc00700 1 --2- 172.21.15.78:0/3834295060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fc7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.855+0000 7f2accc00700 1 -- 172.21.15.78:0/3834295060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbd680 msgr2=0x55b7c3fddf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.855+0000 7f2accc00700 1 --2- 172.21.15.78:0/3834295060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbd680 0x55b7c3fddf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.855+0000 7f2accc00700 1 -- 172.21.15.78:0/3834295060 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b7c3ed5ea0 con 0x55b7c3fc7ec0 2024-03-14T10:02:55.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.855+0000 7f2accc00700 1 --2- 172.21.15.78:0/3834295060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fc4470 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2ac001a840 tx=0x7f2ac0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc4e6c6510261430 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.855+0000 7f2abdc00700 1 -- 172.21.15.78:0/3834295060 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac002a070 con 0x55b7c3fc7ec0 2024-03-14T10:02:55.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.856+0000 7f2abdc00700 1 -- 172.21.15.78:0/3834295060 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ac000f040 con 0x55b7c3fc7ec0 2024-03-14T10:02:55.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.856+0000 7f2abdc00700 1 -- 172.21.15.78:0/3834295060 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac000e040 con 0x55b7c3fc7ec0 2024-03-14T10:02:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.856+0000 7f2ae26dab80 1 -- 172.21.15.78:0/3834295060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fc7ec0 msgr2=0x55b7c3fc4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.856+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/3834295060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fc4470 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2ac001a840 tx=0x7f2ac0011000 comp rx=0 tx=0).stop 2024-03-14T10:02:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.856+0000 7f2ae26dab80 1 -- 172.21.15.78:0/3834295060 shutdown_connections 2024-03-14T10:02:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.856+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/3834295060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbd680 0x55b7c3fddf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.856+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/3834295060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fc4470 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.856+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/3834295060 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fc7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.856+0000 7f2ae26dab80 1 -- 172.21.15.78:0/3834295060 >> 172.21.15.78:0/3834295060 conn(0x55b7c3f33220 msgr2=0x55b7c3fc34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.856+0000 7f2ae26dab80 1 -- 172.21.15.78:0/3834295060 shutdown_connections 2024-03-14T10:02:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.856+0000 7f2ae26dab80 1 -- 172.21.15.78:0/3834295060 wait complete. 2024-03-14T10:02:55.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.857+0000 7f2ae26dab80 1 -- start start 2024-03-14T10:02:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.857+0000 7f2ae26dab80 1 Processor -- start 2024-03-14T10:02:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.857+0000 7f2ae26dab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.857+0000 7f2ae26dab80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.857+0000 7f2ae26dab80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fcad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.857+0000 7f2ae26dab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c4027de0 con 0x55b7c3fbd680 2024-03-14T10:02:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.857+0000 7f2ae26dab80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c4027f50 con 0x55b7c3fbdfa0 2024-03-14T10:02:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.857+0000 7f2ae26dab80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7c4028110 con 0x55b7c3fc7ec0 2024-03-14T10:02:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.857+0000 7f2ae26dab80 10 client.0 _ll_register_callbacks cb 0x7f2ab4009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.857+0000 7f2ac7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.857+0000 7f2ac7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fca2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35054/0 (socket says 172.21.15.78:35054) 2024-03-14T10:02:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.857+0000 7f2ac7e00700 1 -- 172.21.15.78:0/1960409919 learned_addr learned my addr 172.21.15.78:0/1960409919 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.858+0000 7f2accc00700 1 --2- 172.21.15.78:0/1960409919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fcad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.858+0000 7f2ac7400700 1 --2- 172.21.15.78:0/1960409919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.858+0000 7f2ac7e00700 1 -- 172.21.15.78:0/1960409919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 msgr2=0x55b7c3fcad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.858+0000 7f2ac7e00700 1 --2- 172.21.15.78:0/1960409919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fcad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.858+0000 7f2ac7e00700 1 -- 172.21.15.78:0/1960409919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbdfa0 msgr2=0x55b7c3fca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.858+0000 7f2ac7e00700 1 --2- 172.21.15.78:0/1960409919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fca7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.858+0000 7f2ac7e00700 1 -- 172.21.15.78:0/1960409919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac001a470 con 0x55b7c3fbd680 2024-03-14T10:02:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.858+0000 7f2ac7400700 1 --2- 172.21.15.78:0/1960409919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:02:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.858+0000 7f2ac7e00700 1 --2- 172.21.15.78:0/1960409919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fca2b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8097610 tx=0x7f2ab8022af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07a8ff3db1a8ccf server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.858+0000 7f2ab1600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.858+0000 7f2ab1600700 1 -- 172.21.15.78:0/1960409919 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab809b010 con 0x55b7c3fbd680 2024-03-14T10:02:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.858+0000 7f2ab1600700 1 -- 172.21.15.78:0/1960409919 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f2ab80a2f90 con 0x55b7c3fbd680 2024-03-14T10:02:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.858+0000 7f2ae26dab80 1 -- 172.21.15.78:0/1960409919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2ab809b2a0 con 0x55b7c3fbd680 2024-03-14T10:02:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.858+0000 7f2ab1600700 1 -- 172.21.15.78:0/1960409919 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ab809a970 con 0x55b7c3fbd680 2024-03-14T10:02:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.858+0000 7f2ae26dab80 10 client.17364 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.858+0000 7f2ae26dab80 1 -- 172.21.15.78:0/1960409919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b7c3d95cd0 con 0x55b7c3fbd680 2024-03-14T10:02:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.859+0000 7f2ab1600700 1 -- 172.21.15.78:0/1960409919 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ab809b2a0 con 0x55b7c3fbd680 2024-03-14T10:02:55.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.859+0000 7f2ab1600700 1 -- 172.21.15.78:0/1960409919 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2ab80a2b00 con 0x55b7c3fbd680 2024-03-14T10:02:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.859+0000 7f2ae26dab80 10 client.17364 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.859+0000 7f2ae26dab80 10 client.17364 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:02:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.859+0000 7f2ae26dab80 20 client.17364 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.859+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/1960409919 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b7c4029400 0x55b7c402b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.859+0000 7f2ae26dab80 4 client.17364 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:02:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.859+0000 7f2ae26dab80 1 -- 172.21.15.78:0/1960409919 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b7c3d5ecf0 con 0x55b7c4029400 2024-03-14T10:02:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.859+0000 7f2ac7400700 1 --2- 172.21.15.78:0/1960409919 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b7c4029400 0x55b7c402b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.860+0000 7f2ab1600700 10 client.17364 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.860+0000 7f2ac7400700 1 --2- 172.21.15.78:0/1960409919 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b7c4029400 0x55b7c402b7f0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9f0e21443554778 server_cookie=c4dbf98d784fb80 in_seq=0 out_seq=0 2024-03-14T10:02:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.860+0000 7f2ab1600700 1 -- 172.21.15.78:0/1960409919 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b7c3d5ecf0 con 0x55b7c4029400 2024-03-14T10:02:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.860+0000 7f2ab1600700 10 client.17364 handle_command_reply: tid=0 2024-03-14T10:02:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.860+0000 7f2ae26dab80 1 client.17364 shutdown 2024-03-14T10:02:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.860+0000 7f2ae26dab80 1 -- 172.21.15.78:0/1960409919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 msgr2=0x55b7c3fca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.860+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/1960409919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fca2b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8097610 tx=0x7f2ab8022af0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.860+0000 7f2ae26dab80 1 -- 172.21.15.78:0/1960409919 shutdown_connections 2024-03-14T10:02:55.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.860+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/1960409919 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b7c3fc7ec0 0x55b7c3fcad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.860+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/1960409919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b7c3fbd680 0x55b7c3fca2b0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.860+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/1960409919 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b7c4029400 0x55b7c402b7f0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.860+0000 7f2ae26dab80 1 --2- 172.21.15.78:0/1960409919 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b7c3fbdfa0 0x55b7c3fca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.860+0000 7f2ae26dab80 1 -- 172.21.15.78:0/1960409919 >> 172.21.15.78:0/1960409919 conn(0x55b7c3f33220 msgr2=0x55b7c3fc3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.860+0000 7f2ab1600700 0 client.17364 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:55.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.860+0000 7f2ae26dab80 1 -- 172.21.15.78:0/1960409919 shutdown_connections 2024-03-14T10:02:55.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.860+0000 7f2ae26dab80 1 -- 172.21.15.78:0/1960409919 wait complete. 2024-03-14T10:02:55.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.861+0000 7f2ae26dab80 20 client.17364 trim_cache size 0 max 16384 2024-03-14T10:02:55.866 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:02:55.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.861+0000 7f2acec00700 1 -- 172.21.15.78:0/2607711078 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2aac0a0c00 msgr2=0x7f2aac0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.861+0000 7f2acec00700 1 --2- 172.21.15.78:0/2607711078 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2aac0a0c00 0x7f2aac0a30c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0009ec0 tx=0x7f2ac0009920 comp rx=0 tx=0).stop 2024-03-14T10:02:55.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.861+0000 7f2acec00700 1 -- 172.21.15.78:0/2607711078 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ac8100f20 msgr2=0x7f2ac819f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:55.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.861+0000 7f2acec00700 1 --2- 172.21.15.78:0/2607711078 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ac8100f20 0x7f2ac819f5b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8000c00 tx=0x7f2ab8005740 comp rx=0 tx=0).stop 2024-03-14T10:02:55.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.861+0000 7f2acec00700 1 -- 172.21.15.78:0/2607711078 shutdown_connections 2024-03-14T10:02:55.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.861+0000 7f2acec00700 1 --2- 172.21.15.78:0/2607711078 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f2aac0a0c00 0x7f2aac0a30c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.861+0000 7f2acec00700 1 --2- 172.21.15.78:0/2607711078 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ac8100f20 0x7f2ac819f5b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.861+0000 7f2acec00700 1 --2- 172.21.15.78:0/2607711078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ac81996e0 0x7f2ac8199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.861+0000 7f2acec00700 1 --2- 172.21.15.78:0/2607711078 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f2ac8106040 0x7f2ac819faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:55.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.861+0000 7f2acec00700 1 -- 172.21.15.78:0/2607711078 >> 172.21.15.78:0/2607711078 conn(0x7f2ac80fc7a0 msgr2=0x7f2ac80feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:55.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.861+0000 7f2acec00700 1 -- 172.21.15.78:0/2607711078 shutdown_connections 2024-03-14T10:02:55.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:55.861+0000 7f2acec00700 1 -- 172.21.15.78:0/2607711078 wait complete. 2024-03-14T10:02:56.880 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T10:02:56.881 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T10:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.085+0000 7f4b56c00700 1 -- 172.21.15.78:0/2435811587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b500731e0 msgr2=0x7f4b500735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.085+0000 7f4b56c00700 1 --2- 172.21.15.78:0/2435811587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b500731e0 0x7f4b500735c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4b40009990 tx=0x7f4b40009ca0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.085+0000 7f4b56c00700 1 -- 172.21.15.78:0/2435811587 shutdown_connections 2024-03-14T10:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.085+0000 7f4b56c00700 1 --2- 172.21.15.78:0/2435811587 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b5010d4f0 0x7f4b5010f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.085+0000 7f4b56c00700 1 --2- 172.21.15.78:0/2435811587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b500731e0 0x7f4b500735c0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.085+0000 7f4b56c00700 1 --2- 172.21.15.78:0/2435811587 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b50073b00 0x7f4b5010cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.085+0000 7f4b56c00700 1 -- 172.21.15.78:0/2435811587 >> 172.21.15.78:0/2435811587 conn(0x7f4b500fc7a0 msgr2=0x7f4b500febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:57.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.085+0000 7f4b56c00700 1 -- 172.21.15.78:0/2435811587 shutdown_connections 2024-03-14T10:02:57.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.085+0000 7f4b56c00700 1 -- 172.21.15.78:0/2435811587 wait complete. 2024-03-14T10:02:57.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.086+0000 7f4b56c00700 1 Processor -- start 2024-03-14T10:02:57.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.086+0000 7f4b56c00700 1 -- start start 2024-03-14T10:02:57.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b56c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b500731e0 0x7f4b5019f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b56c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b50073b00 0x7f4b5019f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b56c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b5010d4f0 0x7f4b50199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b56c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b50199cb0 con 0x7f4b50073b00 2024-03-14T10:02:57.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b56c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b50199e20 con 0x7f4b500731e0 2024-03-14T10:02:57.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b56c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b50199fe0 con 0x7f4b5010d4f0 2024-03-14T10:02:57.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b4f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b50073b00 0x7f4b5019f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b4fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b500731e0 0x7f4b5019f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b4f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b50073b00 0x7f4b5019f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35062/0 (socket says 172.21.15.78:35062) 2024-03-14T10:02:57.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b4f400700 1 -- 172.21.15.78:0/3055988842 learned_addr learned my addr 172.21.15.78:0/3055988842 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:57.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b54c00700 1 --2- 172.21.15.78:0/3055988842 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b5010d4f0 0x7f4b50199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b4f400700 1 -- 172.21.15.78:0/3055988842 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b5010d4f0 msgr2=0x7f4b50199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b4f400700 1 --2- 172.21.15.78:0/3055988842 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b5010d4f0 0x7f4b50199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b4f400700 1 -- 172.21.15.78:0/3055988842 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b500731e0 msgr2=0x7f4b5019f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b4f400700 1 --2- 172.21.15.78:0/3055988842 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b500731e0 0x7f4b5019f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b4f400700 1 -- 172.21.15.78:0/3055988842 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b400095f0 con 0x7f4b50073b00 2024-03-14T10:02:57.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.087+0000 7f4b4fe00700 1 --2- 172.21.15.78:0/3055988842 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b500731e0 0x7f4b5019f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:57.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.088+0000 7f4b4f400700 1 --2- 172.21.15.78:0/3055988842 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b50073b00 0x7f4b5019f9b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00e9a0 tx=0x7f4b3c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:57.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.088+0000 7f4b4cc00700 1 -- 172.21.15.78:0/3055988842 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c009790 con 0x7f4b50073b00 2024-03-14T10:02:57.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.088+0000 7f4b4cc00700 1 -- 172.21.15.78:0/3055988842 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b3c010430 con 0x7f4b50073b00 2024-03-14T10:02:57.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.088+0000 7f4b4cc00700 1 -- 172.21.15.78:0/3055988842 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c018680 con 0x7f4b50073b00 2024-03-14T10:02:57.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.088+0000 7f4b56c00700 1 -- 172.21.15.78:0/3055988842 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5019a2c0 con 0x7f4b50073b00 2024-03-14T10:02:57.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.088+0000 7f4b56c00700 1 -- 172.21.15.78:0/3055988842 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b501a6510 con 0x7f4b50073b00 2024-03-14T10:02:57.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.091+0000 7f4b4cc00700 1 -- 172.21.15.78:0/3055988842 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f4b3c0105a0 con 0x7f4b50073b00 2024-03-14T10:02:57.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.093+0000 7f4b6a61fb80 1 Processor -- start 2024-03-14T10:02:57.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.093+0000 7f4b4cc00700 1 --2- 172.21.15.78:0/3055988842 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4b340a0c00 0x7f4b340a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.093+0000 7f4b6a61fb80 1 -- start start 2024-03-14T10:02:57.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.093+0000 7f4b4cc00700 1 -- 172.21.15.78:0/3055988842 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f4b3c014070 con 0x7f4b50073b00 2024-03-14T10:02:57.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.093+0000 7f4b54c00700 1 --2- 172.21.15.78:0/3055988842 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4b340a0c00 0x7f4b340a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b6a61fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 0x5560f82feae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b6a61fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 0x5560f8308a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b6a61fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f830b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b6a61fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f8257a10 con 0x5560f82ff020 2024-03-14T10:02:57.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b6a61fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f82553d0 con 0x5560f8308f40 2024-03-14T10:02:57.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b6a61fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f8250cc0 con 0x5560f82fe700 2024-03-14T10:02:57.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b4fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 0x5560f82feae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b4fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 0x5560f82feae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:58784/0 (socket says 172.21.15.78:58784) 2024-03-14T10:02:57.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b4fe00700 1 -- 172.21.15.78:0/3685537912 learned_addr learned my addr 172.21.15.78:0/3685537912 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:57.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b4f400700 1 --2- 172.21.15.78:0/3685537912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 0x5560f8308a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b54c00700 1 --2- 172.21.15.78:0/3685537912 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f830b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b4fe00700 1 -- 172.21.15.78:0/3685537912 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 msgr2=0x5560f830b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:57.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b4fe00700 1 --2- 172.21.15.78:0/3685537912 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f830b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b4fe00700 1 -- 172.21.15.78:0/3685537912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 msgr2=0x5560f8308a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:57.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b4fe00700 1 --2- 172.21.15.78:0/3685537912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 0x5560f8308a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b4fe00700 1 -- 172.21.15.78:0/3685537912 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560f80d2430 con 0x5560f82fe700 2024-03-14T10:02:57.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.094+0000 7f4b54c00700 1 --2- 172.21.15.78:0/3055988842 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4b340a0c00 0x7f4b340a30c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4b4800f540 tx=0x7f4b48009920 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:57.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.095+0000 7f4b4fe00700 1 --2- 172.21.15.78:0/3685537912 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 0x5560f82feae0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4b40005ec0 tx=0x7f4b40005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=487bc26948433e4c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:57.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.095+0000 7f4b45200700 1 -- 172.21.15.78:0/3685537912 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b40027070 con 0x5560f82fe700 2024-03-14T10:02:57.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.095+0000 7f4b45200700 1 -- 172.21.15.78:0/3685537912 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b4001e040 con 0x5560f82fe700 2024-03-14T10:02:57.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.095+0000 7f4b45200700 1 -- 172.21.15.78:0/3685537912 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4001d050 con 0x5560f82fe700 2024-03-14T10:02:57.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.096+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/3685537912 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 msgr2=0x5560f82feae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.096+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/3685537912 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 0x5560f82feae0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4b40005ec0 tx=0x7f4b40005ef0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.096+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/3685537912 shutdown_connections 2024-03-14T10:02:57.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.096+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/3685537912 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f830b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.096+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/3685537912 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 0x5560f82feae0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.096+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/3685537912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 0x5560f8308a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.096+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/3685537912 >> 172.21.15.78:0/3685537912 conn(0x5560f82742a0 msgr2=0x5560f82f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:57.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.096+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/3685537912 shutdown_connections 2024-03-14T10:02:57.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.096+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/3685537912 wait complete. 2024-03-14T10:02:57.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.096+0000 7f4b6a61fb80 1 -- start start 2024-03-14T10:02:57.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.096+0000 7f4b6a61fb80 1 Processor -- start 2024-03-14T10:02:57.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.097+0000 7f4b6a61fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 0x5560f83088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.097+0000 7f4b6a61fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 0x5560f8305ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.097+0000 7f4b6a61fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f83061d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.097+0000 7f4b6a61fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f81ece00 con 0x5560f82ff020 2024-03-14T10:02:57.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.097+0000 7f4b6a61fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f81ecaa0 con 0x5560f8308f40 2024-03-14T10:02:57.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.097+0000 7f4b6a61fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f80d4720 con 0x5560f82fe700 2024-03-14T10:02:57.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.097+0000 7f4b6a61fb80 10 client.0 _ll_register_callbacks cb 0x5560f8058420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:57.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.097+0000 7f4b4fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 0x5560f83088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.097+0000 7f4b4fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 0x5560f83088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:58798/0 (socket says 172.21.15.78:58798) 2024-03-14T10:02:57.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.097+0000 7f4b4fe00700 1 -- 172.21.15.78:0/412218198 learned_addr learned my addr 172.21.15.78:0/412218198 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:57.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.097+0000 7f4b4f400700 1 --2- 172.21.15.78:0/412218198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 0x5560f8305ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.097+0000 7f4b54c00700 1 --2- 172.21.15.78:0/412218198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f83061d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.097+0000 7f4b4fe00700 1 -- 172.21.15.78:0/412218198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 msgr2=0x5560f83061d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:57.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.097+0000 7f4b4fe00700 1 --2- 172.21.15.78:0/412218198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f83061d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.098+0000 7f4b4fe00700 1 -- 172.21.15.78:0/412218198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 msgr2=0x5560f8305ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:57.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.098+0000 7f4b4fe00700 1 --2- 172.21.15.78:0/412218198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 0x5560f8305ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.098+0000 7f4b4fe00700 1 -- 172.21.15.78:0/412218198 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b50110620 con 0x5560f82fe700 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.098+0000 7f4b4fe00700 1 --2- 172.21.15.78:0/412218198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 0x5560f83088b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4b400098e0 tx=0x7f4b40005990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=569ff5cde506d700 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.098+0000 7f4b39600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.098+0000 7f4b39600700 1 -- 172.21.15.78:0/412218198 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b40027030 con 0x5560f82fe700 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.098+0000 7f4b39600700 1 -- 172.21.15.78:0/412218198 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f4b400053c0 con 0x5560f82fe700 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.098+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/412218198 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5560f82f17a0 con 0x5560f82fe700 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.098+0000 7f4b39600700 1 -- 172.21.15.78:0/412218198 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b40005580 con 0x5560f82fe700 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.098+0000 7f4b6a61fb80 10 client.27424 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.098+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/412218198 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5560f807ebe0 con 0x5560f82fe700 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.099+0000 7f4b39600700 1 -- 172.21.15.78:0/412218198 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5560f82f17a0 con 0x5560f82fe700 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.099+0000 7f4b39600700 1 -- 172.21.15.78:0/412218198 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4b40023460 con 0x5560f82fe700 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.099+0000 7f4b6a61fb80 10 client.27424 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.099+0000 7f4b6a61fb80 10 client.27424 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.099+0000 7f4b6a61fb80 20 client.27424 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.099+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/412218198 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5560f8318760 0x5560f831ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.099+0000 7f4b6a61fb80 4 client.27424 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.099+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/412218198 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5560f8249780 con 0x5560f8318760 2024-03-14T10:02:57.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.100+0000 7f4b4f400700 1 --2- 172.21.15.78:0/412218198 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5560f8318760 0x5560f831ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.100+0000 7f4b4f400700 1 --2- 172.21.15.78:0/412218198 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5560f8318760 0x5560f831ab80 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d9cdf09d57bfd83 server_cookie=cc619b9d06dbb484 in_seq=0 out_seq=0 2024-03-14T10:02:57.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.100+0000 7f4b39600700 10 client.27424 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.101+0000 7f4b39600700 1 -- 172.21.15.78:0/412218198 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5560f8249780 con 0x5560f8318760 2024-03-14T10:02:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.101+0000 7f4b39600700 10 client.27424 handle_command_reply: tid=0 2024-03-14T10:02:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.101+0000 7f4b6a61fb80 1 client.27424 shutdown 2024-03-14T10:02:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.101+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/412218198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 msgr2=0x5560f83088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.101+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/412218198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 0x5560f83088b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4b400098e0 tx=0x7f4b40005990 comp rx=0 tx=0).stop 2024-03-14T10:02:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.101+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/412218198 shutdown_connections 2024-03-14T10:02:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.101+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/412218198 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5560f8318760 0x5560f831ab80 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.101+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/412218198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f83061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.101+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/412218198 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 0x5560f83088b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.101+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/412218198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 0x5560f8305ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.101+0000 7f4b39600700 0 client.27424 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.101+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/412218198 >> 172.21.15.78:0/412218198 conn(0x5560f82742a0 msgr2=0x5560f82f9710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.101+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/412218198 shutdown_connections 2024-03-14T10:02:57.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.101+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/412218198 wait complete. 2024-03-14T10:02:57.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.101+0000 7f4b6a61fb80 20 client.27424 trim_cache size 0 max 16384 2024-03-14T10:02:57.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.118+0000 7f4b6a61fb80 1 Processor -- start 2024-03-14T10:02:57.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.118+0000 7f4b6a61fb80 1 -- start start 2024-03-14T10:02:57.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.118+0000 7f4b6a61fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 0x5560f830fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.118+0000 7f4b6a61fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 0x5560f8304470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.119+0000 7f4b6a61fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f83049b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.119+0000 7f4b6a61fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f82f3d80 con 0x5560f82ff020 2024-03-14T10:02:57.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.119+0000 7f4b6a61fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f83009d0 con 0x5560f8308f40 2024-03-14T10:02:57.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.119+0000 7f4b6a61fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f831f860 con 0x5560f82fe700 2024-03-14T10:02:57.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.119+0000 7f4b4f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 0x5560f8304470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.119+0000 7f4b4f400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 0x5560f8304470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35088/0 (socket says 172.21.15.78:35088) 2024-03-14T10:02:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.119+0000 7f4b4f400700 1 -- 172.21.15.78:0/1580486743 learned_addr learned my addr 172.21.15.78:0/1580486743 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.119+0000 7f4b4fe00700 1 --2- 172.21.15.78:0/1580486743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 0x5560f830fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.119+0000 7f4b54c00700 1 --2- 172.21.15.78:0/1580486743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f83049b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.119+0000 7f4b4f400700 1 -- 172.21.15.78:0/1580486743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 msgr2=0x5560f830fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.119+0000 7f4b4f400700 1 --2- 172.21.15.78:0/1580486743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 0x5560f830fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.119+0000 7f4b4f400700 1 -- 172.21.15.78:0/1580486743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 msgr2=0x5560f83049b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.119+0000 7f4b4f400700 1 --2- 172.21.15.78:0/1580486743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f83049b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.119+0000 7f4b4f400700 1 -- 172.21.15.78:0/1580486743 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560f8216f20 con 0x5560f82ff020 2024-03-14T10:02:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.119+0000 7f4b4f400700 1 --2- 172.21.15.78:0/1580486743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 0x5560f8304470 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00c2d0 tx=0x7f4b3c0b9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1745cd85f2595442 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.119+0000 7f4b45c00700 1 -- 172.21.15.78:0/1580486743 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c09d780 con 0x5560f82ff020 2024-03-14T10:02:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.120+0000 7f4b45c00700 1 -- 172.21.15.78:0/1580486743 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b3c0a1dd0 con 0x5560f82ff020 2024-03-14T10:02:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.120+0000 7f4b45c00700 1 -- 172.21.15.78:0/1580486743 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c098e40 con 0x5560f82ff020 2024-03-14T10:02:57.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.120+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/1580486743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 msgr2=0x5560f8304470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.120+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/1580486743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 0x5560f8304470 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00c2d0 tx=0x7f4b3c0b9040 comp rx=0 tx=0).stop 2024-03-14T10:02:57.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.120+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/1580486743 shutdown_connections 2024-03-14T10:02:57.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.120+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/1580486743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f83049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.120+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/1580486743 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82fe700 0x5560f830fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.120+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/1580486743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82ff020 0x5560f8304470 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.120+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/1580486743 >> 172.21.15.78:0/1580486743 conn(0x5560f82742a0 msgr2=0x5560f8300ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:57.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.120+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/1580486743 shutdown_connections 2024-03-14T10:02:57.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.120+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/1580486743 wait complete. 2024-03-14T10:02:57.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.121+0000 7f4b6a61fb80 1 -- start start 2024-03-14T10:02:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.121+0000 7f4b6a61fb80 1 Processor -- start 2024-03-14T10:02:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.121+0000 7f4b6a61fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82fe700 0x5560f830ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.121+0000 7f4b6a61fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82ff020 0x5560f830ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.121+0000 7f4b6a61fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f8364220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.121+0000 7f4b6a61fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f8364760 con 0x5560f82fe700 2024-03-14T10:02:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.121+0000 7f4b6a61fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f83648d0 con 0x5560f8308f40 2024-03-14T10:02:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.121+0000 7f4b6a61fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560f8364a90 con 0x5560f82ff020 2024-03-14T10:02:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.121+0000 7f4b6a61fb80 10 client.0 _ll_register_callbacks cb 0x7f4b40009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.121+0000 7f4b4fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82fe700 0x5560f830ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.121+0000 7f4b4fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82fe700 0x5560f830ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35100/0 (socket says 172.21.15.78:35100) 2024-03-14T10:02:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.121+0000 7f4b4fe00700 1 -- 172.21.15.78:0/339910223 learned_addr learned my addr 172.21.15.78:0/339910223 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.122+0000 7f4b4f400700 1 --2- 172.21.15.78:0/339910223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82ff020 0x5560f830ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.122+0000 7f4b54c00700 1 --2- 172.21.15.78:0/339910223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f8364220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.122+0000 7f4b4fe00700 1 -- 172.21.15.78:0/339910223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82ff020 msgr2=0x5560f830ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.122+0000 7f4b4fe00700 1 --2- 172.21.15.78:0/339910223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82ff020 0x5560f830ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.122+0000 7f4b4fe00700 1 -- 172.21.15.78:0/339910223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 msgr2=0x5560f8364220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.122+0000 7f4b4fe00700 1 --2- 172.21.15.78:0/339910223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f8364220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.122+0000 7f4b4fe00700 1 -- 172.21.15.78:0/339910223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b501a6510 con 0x5560f82fe700 2024-03-14T10:02:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.122+0000 7f4b54c00700 1 --2- 172.21.15.78:0/339910223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f8364220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:02:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.122+0000 7f4b4fe00700 1 --2- 172.21.15.78:0/339910223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82fe700 0x5560f830ea10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4b4000b2e0 tx=0x7f4b40023690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7946923d2b3d82ec server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.122+0000 7f4b39600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.122+0000 7f4b39600700 1 -- 172.21.15.78:0/339910223 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b40027040 con 0x5560f82fe700 2024-03-14T10:02:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.122+0000 7f4b39600700 1 -- 172.21.15.78:0/339910223 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f4b40000e40 con 0x5560f82fe700 2024-03-14T10:02:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.122+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/339910223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4b4001e040 con 0x5560f82fe700 2024-03-14T10:02:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.123+0000 7f4b39600700 1 -- 172.21.15.78:0/339910223 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b4001d040 con 0x5560f82fe700 2024-03-14T10:02:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.123+0000 7f4b6a61fb80 10 client.17382 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.123+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/339910223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5560f80d6d50 con 0x5560f82fe700 2024-03-14T10:02:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.123+0000 7f4b39600700 1 -- 172.21.15.78:0/339910223 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b4001e040 con 0x5560f82fe700 2024-03-14T10:02:57.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.123+0000 7f4b39600700 1 -- 172.21.15.78:0/339910223 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4b4002b7e0 con 0x5560f82fe700 2024-03-14T10:02:57.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.123+0000 7f4b6a61fb80 10 client.17382 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:57.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.123+0000 7f4b6a61fb80 10 client.17382 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:02:57.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.123+0000 7f4b6a61fb80 20 client.17382 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:57.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.123+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/339910223 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5560f836bf10 0x5560f836e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.123+0000 7f4b6a61fb80 4 client.17382 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T10:02:57.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.123+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/339910223 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5560f809fd70 con 0x5560f836bf10 2024-03-14T10:02:57.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.123+0000 7f4b4f400700 1 --2- 172.21.15.78:0/339910223 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5560f836bf10 0x5560f836e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.124+0000 7f4b39600700 10 client.17382 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:57.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.124+0000 7f4b4f400700 1 --2- 172.21.15.78:0/339910223 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5560f836bf10 0x5560f836e330 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bcc9e12de0b2150 server_cookie=6e8a4093b303dec4 in_seq=0 out_seq=0 2024-03-14T10:02:57.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.125+0000 7f4b39600700 1 -- 172.21.15.78:0/339910223 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5560f809fd70 con 0x5560f836bf10 2024-03-14T10:02:57.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.125+0000 7f4b39600700 10 client.17382 handle_command_reply: tid=0 2024-03-14T10:02:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.125+0000 7f4b6a61fb80 1 client.17382 shutdown 2024-03-14T10:02:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.125+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/339910223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82fe700 msgr2=0x5560f830ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.125+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/339910223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82fe700 0x5560f830ea10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4b4000b2e0 tx=0x7f4b40023690 comp rx=0 tx=0).stop 2024-03-14T10:02:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.125+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/339910223 shutdown_connections 2024-03-14T10:02:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.125+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/339910223 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5560f836bf10 0x5560f836e330 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.125+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/339910223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5560f8308f40 0x5560f8364220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.125+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/339910223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5560f82fe700 0x5560f830ea10 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.125+0000 7f4b39600700 0 client.17382 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.125+0000 7f4b6a61fb80 1 --2- 172.21.15.78:0/339910223 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5560f82ff020 0x5560f830ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.125+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/339910223 >> 172.21.15.78:0/339910223 conn(0x5560f82742a0 msgr2=0x5560f8304060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.125+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/339910223 shutdown_connections 2024-03-14T10:02:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.126+0000 7f4b6a61fb80 1 -- 172.21.15.78:0/339910223 wait complete. 2024-03-14T10:02:57.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.126+0000 7f4b6a61fb80 20 client.17382 trim_cache size 0 max 16384 2024-03-14T10:02:57.131 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:02:57.131 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T10:02:57.131 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "1cd6d5bb-34eb-4b12-9407-8eea1021cf68", 2024-03-14T10:02:57.131 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T10:02:57.131 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:02:57.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.126+0000 7f4b56c00700 1 -- 172.21.15.78:0/3055988842 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4b340a0c00 msgr2=0x7f4b340a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.126+0000 7f4b56c00700 1 --2- 172.21.15.78:0/3055988842 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4b340a0c00 0x7f4b340a30c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4b4800f540 tx=0x7f4b48009920 comp rx=0 tx=0).stop 2024-03-14T10:02:57.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.126+0000 7f4b56c00700 1 -- 172.21.15.78:0/3055988842 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b50073b00 msgr2=0x7f4b5019f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.126+0000 7f4b56c00700 1 --2- 172.21.15.78:0/3055988842 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b50073b00 0x7f4b5019f9b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00e9a0 tx=0x7f4b3c00ecb0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.126+0000 7f4b56c00700 1 -- 172.21.15.78:0/3055988842 shutdown_connections 2024-03-14T10:02:57.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.126+0000 7f4b56c00700 1 --2- 172.21.15.78:0/3055988842 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4b340a0c00 0x7f4b340a30c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.126+0000 7f4b56c00700 1 --2- 172.21.15.78:0/3055988842 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b500731e0 0x7f4b5019f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.126+0000 7f4b56c00700 1 --2- 172.21.15.78:0/3055988842 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4b5010d4f0 0x7f4b50199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.126+0000 7f4b56c00700 1 --2- 172.21.15.78:0/3055988842 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b50073b00 0x7f4b5019f9b0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.126+0000 7f4b56c00700 1 -- 172.21.15.78:0/3055988842 >> 172.21.15.78:0/3055988842 conn(0x7f4b500fc7a0 msgr2=0x7f4b500fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:57.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.127+0000 7f4b56c00700 1 -- 172.21.15.78:0/3055988842 shutdown_connections 2024-03-14T10:02:57.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.127+0000 7f4b56c00700 1 -- 172.21.15.78:0/3055988842 wait complete. 2024-03-14T10:02:57.145 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:02:57.202 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:57 smithi078 ceph-mon[38143]: pgmap v270: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:02:57.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.341+0000 7efdebc00700 1 -- 172.21.15.78:0/647411678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efde4068a60 msgr2=0x7efde4105930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.341+0000 7efdebc00700 1 --2- 172.21.15.78:0/647411678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efde4068a60 0x7efde4105930 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7efdd00099c0 tx=0x7efdd0009cd0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.341+0000 7efdebc00700 1 -- 172.21.15.78:0/647411678 shutdown_connections 2024-03-14T10:02:57.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.341+0000 7efdebc00700 1 --2- 172.21.15.78:0/647411678 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efde4106060 0x7efde410dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.341+0000 7efdebc00700 1 --2- 172.21.15.78:0/647411678 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efde40680b0 0x7efde4068490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.341+0000 7efdebc00700 1 --2- 172.21.15.78:0/647411678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efde4068a60 0x7efde4105930 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.341+0000 7efdebc00700 1 -- 172.21.15.78:0/647411678 >> 172.21.15.78:0/647411678 conn(0x7efde4076d80 msgr2=0x7efde4077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:57.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.341+0000 7efdebc00700 1 -- 172.21.15.78:0/647411678 shutdown_connections 2024-03-14T10:02:57.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.341+0000 7efdebc00700 1 -- 172.21.15.78:0/647411678 wait complete. 2024-03-14T10:02:57.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.342+0000 7efdebc00700 1 Processor -- start 2024-03-14T10:02:57.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.342+0000 7efdebc00700 1 -- start start 2024-03-14T10:02:57.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.343+0000 7efdebc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efde40680b0 0x7efde40ffc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.343+0000 7efdebc00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efde4106060 0x7efde4100160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.343+0000 7efdebc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efde41006a0 0x7efde41a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.343+0000 7efdebc00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde41a5d90 con 0x7efde41006a0 2024-03-14T10:02:57.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.343+0000 7efdebc00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde41a5f00 con 0x7efde40680b0 2024-03-14T10:02:57.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.343+0000 7efdebc00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde41a60c0 con 0x7efde4106060 2024-03-14T10:02:57.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.343+0000 7efde9200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efde40680b0 0x7efde40ffc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.343+0000 7efde9200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efde40680b0 0x7efde40ffc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:36436/0 (socket says 172.21.15.78:36436) 2024-03-14T10:02:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.343+0000 7efde9c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efde41006a0 0x7efde41a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.343+0000 7efde9200700 1 -- 172.21.15.78:0/2226524921 learned_addr learned my addr 172.21.15.78:0/2226524921 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.343+0000 7efde3e00700 1 --2- 172.21.15.78:0/2226524921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efde4106060 0x7efde4100160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.343+0000 7efde9200700 1 -- 172.21.15.78:0/2226524921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efde4106060 msgr2=0x7efde4100160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.343+0000 7efde9200700 1 --2- 172.21.15.78:0/2226524921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efde4106060 0x7efde4100160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.343+0000 7efde9200700 1 -- 172.21.15.78:0/2226524921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efde41006a0 msgr2=0x7efde41a57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.343+0000 7efde9200700 1 --2- 172.21.15.78:0/2226524921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efde41006a0 0x7efde41a57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.343+0000 7efde9200700 1 -- 172.21.15.78:0/2226524921 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd00095f0 con 0x7efde40680b0 2024-03-14T10:02:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.344+0000 7efde9200700 1 --2- 172.21.15.78:0/2226524921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efde40680b0 0x7efde40ffc20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7efdd800e9a0 tx=0x7efdd800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.344+0000 7efde1600700 1 -- 172.21.15.78:0/2226524921 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd8009790 con 0x7efde40680b0 2024-03-14T10:02:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.344+0000 7efdebc00700 1 -- 172.21.15.78:0/2226524921 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde41a63a0 con 0x7efde40680b0 2024-03-14T10:02:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.344+0000 7efde1600700 1 -- 172.21.15.78:0/2226524921 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdd8010430 con 0x7efde40680b0 2024-03-14T10:02:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.344+0000 7efde1600700 1 -- 172.21.15.78:0/2226524921 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd8018680 con 0x7efde40680b0 2024-03-14T10:02:57.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.344+0000 7efdebc00700 1 -- 172.21.15.78:0/2226524921 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde41a68d0 con 0x7efde40680b0 2024-03-14T10:02:57.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.347+0000 7efde1600700 1 -- 172.21.15.78:0/2226524921 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7efdd80105a0 con 0x7efde40680b0 2024-03-14T10:02:57.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.348+0000 7efdff7adb80 1 Processor -- start 2024-03-14T10:02:57.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.348+0000 7efdff7adb80 1 -- start start 2024-03-14T10:02:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efde1600700 1 --2- 172.21.15.78:0/2226524921 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efdcc0a0c00 0x7efdcc0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efdff7adb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58a680 0x55c2ce58aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efdff7adb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce58afa0 0x55c2ce594980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efdff7adb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce594ec0 0x55c2ce5972b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efdff7adb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ce4e3990 con 0x55c2ce58a680 2024-03-14T10:02:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efdff7adb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ce4e1350 con 0x55c2ce58afa0 2024-03-14T10:02:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efdff7adb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ce4dcc40 con 0x55c2ce594ec0 2024-03-14T10:02:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efde1600700 1 -- 172.21.15.78:0/2226524921 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7efdd8014070 con 0x7efde40680b0 2024-03-14T10:02:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efde9200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58a680 0x55c2ce58aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efde9c00700 1 --2- 172.21.15.78:0/2226524921 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efdcc0a0c00 0x7efdcc0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efde9200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58a680 0x55c2ce58aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35126/0 (socket says 172.21.15.78:35126) 2024-03-14T10:02:57.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efde9200700 1 -- 172.21.15.78:0/4219982605 learned_addr learned my addr 172.21.15.78:0/4219982605 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:57.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efde3e00700 1 --2- 172.21.15.78:0/4219982605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce58afa0 0x55c2ce594980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efde9c00700 1 --2- 172.21.15.78:0/4219982605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce594ec0 0x55c2ce5972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efde9200700 1 -- 172.21.15.78:0/4219982605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce594ec0 msgr2=0x55c2ce5972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:57.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efde9200700 1 --2- 172.21.15.78:0/4219982605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce594ec0 0x55c2ce5972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efde9200700 1 -- 172.21.15.78:0/4219982605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce58afa0 msgr2=0x55c2ce594980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:57.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efde9200700 1 --2- 172.21.15.78:0/4219982605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce58afa0 0x55c2ce594980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efde9200700 1 -- 172.21.15.78:0/4219982605 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2ce35e3b0 con 0x55c2ce58a680 2024-03-14T10:02:57.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.349+0000 7efde9c00700 1 --2- 172.21.15.78:0/4219982605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce594ec0 0x55c2ce5972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T10:02:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.350+0000 7efde9200700 1 --2- 172.21.15.78:0/4219982605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58a680 0x55c2ce58aa60 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7efdd80a18a0 tx=0x7efdd8099330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741c5cb193a35f27 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:57.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.350+0000 7efde9c00700 1 --2- 172.21.15.78:0/2226524921 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efdcc0a0c00 0x7efdcc0a30c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efddc009ec0 tx=0x7efddc009920 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.350+0000 7efdd6000700 1 -- 172.21.15.78:0/4219982605 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd809a050 con 0x55c2ce58a680 2024-03-14T10:02:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.350+0000 7efdd6000700 1 -- 172.21.15.78:0/4219982605 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdd8099950 con 0x55c2ce58a680 2024-03-14T10:02:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.350+0000 7efdd6000700 1 -- 172.21.15.78:0/4219982605 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd80b8c30 con 0x55c2ce58a680 2024-03-14T10:02:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.350+0000 7efdff7adb80 1 -- 172.21.15.78:0/4219982605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58a680 msgr2=0x55c2ce58aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.350+0000 7efdff7adb80 1 --2- 172.21.15.78:0/4219982605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58a680 0x55c2ce58aa60 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7efdd80a18a0 tx=0x7efdd8099330 comp rx=0 tx=0).stop 2024-03-14T10:02:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.351+0000 7efdff7adb80 1 -- 172.21.15.78:0/4219982605 shutdown_connections 2024-03-14T10:02:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.351+0000 7efdff7adb80 1 --2- 172.21.15.78:0/4219982605 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce594ec0 0x55c2ce5972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.351+0000 7efdff7adb80 1 --2- 172.21.15.78:0/4219982605 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58a680 0x55c2ce58aa60 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.351+0000 7efdff7adb80 1 --2- 172.21.15.78:0/4219982605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce58afa0 0x55c2ce594980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.351+0000 7efdff7adb80 1 -- 172.21.15.78:0/4219982605 >> 172.21.15.78:0/4219982605 conn(0x55c2ce500220 msgr2=0x55c2ce585aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.351+0000 7efdff7adb80 1 -- 172.21.15.78:0/4219982605 shutdown_connections 2024-03-14T10:02:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.351+0000 7efdff7adb80 1 -- 172.21.15.78:0/4219982605 wait complete. 2024-03-14T10:02:57.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.351+0000 7efdff7adb80 1 -- start start 2024-03-14T10:02:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.351+0000 7efdff7adb80 1 Processor -- start 2024-03-14T10:02:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.352+0000 7efdff7adb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce58a680 0x55c2ce5948f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.352+0000 7efdff7adb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce591c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.352+0000 7efdff7adb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce594ec0 0x55c2ce5923c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.352+0000 7efdff7adb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ce478d80 con 0x55c2ce58afa0 2024-03-14T10:02:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.352+0000 7efdff7adb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ce478a20 con 0x55c2ce58a680 2024-03-14T10:02:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.352+0000 7efdff7adb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ce3606a0 con 0x55c2ce594ec0 2024-03-14T10:02:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.352+0000 7efdff7adb80 10 client.0 _ll_register_callbacks cb 0x55c2ce2e43a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.352+0000 7efde3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce591c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.352+0000 7efde3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce591c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35138/0 (socket says 172.21.15.78:35138) 2024-03-14T10:02:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.352+0000 7efde3e00700 1 -- 172.21.15.78:0/4050238660 learned_addr learned my addr 172.21.15.78:0/4050238660 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.352+0000 7efde9200700 1 --2- 172.21.15.78:0/4050238660 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce58a680 0x55c2ce5948f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.352+0000 7efde9c00700 1 --2- 172.21.15.78:0/4050238660 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce594ec0 0x55c2ce5923c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.353+0000 7efde3e00700 1 -- 172.21.15.78:0/4050238660 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce594ec0 msgr2=0x55c2ce5923c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.353+0000 7efde3e00700 1 --2- 172.21.15.78:0/4050238660 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce594ec0 0x55c2ce5923c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.353+0000 7efde3e00700 1 -- 172.21.15.78:0/4050238660 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce58a680 msgr2=0x55c2ce5948f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.353+0000 7efde3e00700 1 --2- 172.21.15.78:0/4050238660 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce58a680 0x55c2ce5948f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.353+0000 7efde3e00700 1 -- 172.21.15.78:0/4050238660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde41a68d0 con 0x55c2ce58afa0 2024-03-14T10:02:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.353+0000 7efde3e00700 1 --2- 172.21.15.78:0/4050238660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce591c80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7efde406a4a0 tx=0x7efdd0005790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f286444bcfc66ce0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.353+0000 7efdcaa00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.353+0000 7efdcaa00700 1 -- 172.21.15.78:0/4050238660 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd001d030 con 0x55c2ce58afa0 2024-03-14T10:02:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.353+0000 7efdff7adb80 1 -- 172.21.15.78:0/4050238660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c2ce57d720 con 0x55c2ce58afa0 2024-03-14T10:02:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.353+0000 7efdcaa00700 1 -- 172.21.15.78:0/4050238660 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7efdd000bc20 con 0x55c2ce58afa0 2024-03-14T10:02:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.353+0000 7efdcaa00700 1 -- 172.21.15.78:0/4050238660 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdd000bde0 con 0x55c2ce58afa0 2024-03-14T10:02:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.353+0000 7efdff7adb80 10 client.17394 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.353+0000 7efdff7adb80 1 -- 172.21.15.78:0/4050238660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c2ce30ab60 con 0x55c2ce58afa0 2024-03-14T10:02:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.354+0000 7efdcaa00700 1 -- 172.21.15.78:0/4050238660 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c2ce57d720 con 0x55c2ce58afa0 2024-03-14T10:02:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.354+0000 7efdcaa00700 1 -- 172.21.15.78:0/4050238660 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7efdd0024880 con 0x55c2ce58afa0 2024-03-14T10:02:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.354+0000 7efdff7adb80 10 client.17394 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.354+0000 7efdff7adb80 10 client.17394 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:02:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.354+0000 7efdff7adb80 20 client.17394 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:57.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.354+0000 7efdff7adb80 1 --2- 172.21.15.78:0/4050238660 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c2ce5a4aa0 0x55c2ce5a6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.354+0000 7efdff7adb80 4 client.17394 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:02:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.354+0000 7efdff7adb80 1 -- 172.21.15.78:0/4050238660 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2ce4d5700 con 0x55c2ce5a4aa0 2024-03-14T10:02:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.354+0000 7efde9200700 1 --2- 172.21.15.78:0/4050238660 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c2ce5a4aa0 0x55c2ce5a6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.355+0000 7efdcaa00700 10 client.17394 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:57.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.355+0000 7efde9200700 1 --2- 172.21.15.78:0/4050238660 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c2ce5a4aa0 0x55c2ce5a6ec0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12be20b0b5b05b0b server_cookie=5555d6821795e5b5 in_seq=0 out_seq=0 2024-03-14T10:02:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.356+0000 7efdcaa00700 1 -- 172.21.15.78:0/4050238660 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55c2ce4d5700 con 0x55c2ce5a4aa0 2024-03-14T10:02:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.356+0000 7efdcaa00700 10 client.17394 handle_command_reply: tid=0 2024-03-14T10:02:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.356+0000 7efdff7adb80 1 client.17394 shutdown 2024-03-14T10:02:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.356+0000 7efdff7adb80 1 -- 172.21.15.78:0/4050238660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 msgr2=0x55c2ce591c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.356+0000 7efdff7adb80 1 --2- 172.21.15.78:0/4050238660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce591c80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7efde406a4a0 tx=0x7efdd0005790 comp rx=0 tx=0).stop 2024-03-14T10:02:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.356+0000 7efdff7adb80 1 -- 172.21.15.78:0/4050238660 shutdown_connections 2024-03-14T10:02:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.356+0000 7efdff7adb80 1 --2- 172.21.15.78:0/4050238660 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c2ce5a4aa0 0x55c2ce5a6ec0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.356+0000 7efdff7adb80 1 --2- 172.21.15.78:0/4050238660 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce58a680 0x55c2ce5948f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.356+0000 7efdff7adb80 1 --2- 172.21.15.78:0/4050238660 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce594ec0 0x55c2ce5923c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.356+0000 7efdcaa00700 0 client.17394 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.356+0000 7efdff7adb80 1 --2- 172.21.15.78:0/4050238660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce591c80 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.356+0000 7efdff7adb80 1 -- 172.21.15.78:0/4050238660 >> 172.21.15.78:0/4050238660 conn(0x55c2ce500220 msgr2=0x55c2ce585690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:57.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.356+0000 7efdff7adb80 1 -- 172.21.15.78:0/4050238660 shutdown_connections 2024-03-14T10:02:57.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.356+0000 7efdff7adb80 1 -- 172.21.15.78:0/4050238660 wait complete. 2024-03-14T10:02:57.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.356+0000 7efdff7adb80 20 client.17394 trim_cache size 0 max 16384 2024-03-14T10:02:57.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.373+0000 7efdff7adb80 1 Processor -- start 2024-03-14T10:02:57.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.373+0000 7efdff7adb80 1 -- start start 2024-03-14T10:02:57.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.373+0000 7efdff7adb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce58a680 0x55c2ce59bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.373+0000 7efdff7adb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce5855d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.373+0000 7efdff7adb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce594ec0 0x55c2ce585b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.373+0000 7efdff7adb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ce58b660 con 0x55c2ce58afa0 2024-03-14T10:02:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.373+0000 7efdff7adb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ce5931e0 con 0x55c2ce594ec0 2024-03-14T10:02:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.373+0000 7efdff7adb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ce592bd0 con 0x55c2ce58a680 2024-03-14T10:02:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.374+0000 7efde3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce5855d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.374+0000 7efde3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce5855d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35140/0 (socket says 172.21.15.78:35140) 2024-03-14T10:02:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.374+0000 7efde9200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce58a680 0x55c2ce59bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.374+0000 7efde3e00700 1 -- 172.21.15.78:0/1325206116 learned_addr learned my addr 172.21.15.78:0/1325206116 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.374+0000 7efde9c00700 1 --2- 172.21.15.78:0/1325206116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce594ec0 0x55c2ce585b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.374+0000 7efde3e00700 1 -- 172.21.15.78:0/1325206116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce58a680 msgr2=0x55c2ce59bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.374+0000 7efde3e00700 1 --2- 172.21.15.78:0/1325206116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce58a680 0x55c2ce59bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.374+0000 7efde3e00700 1 -- 172.21.15.78:0/1325206116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce594ec0 msgr2=0x55c2ce585b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.374+0000 7efde3e00700 1 --2- 172.21.15.78:0/1325206116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce594ec0 0x55c2ce585b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.374+0000 7efde3e00700 1 -- 172.21.15.78:0/1325206116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2ce4a2ea0 con 0x55c2ce58afa0 2024-03-14T10:02:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.374+0000 7efde9200700 1 --2- 172.21.15.78:0/1325206116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce58a680 0x55c2ce59bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.374+0000 7efde9c00700 1 --2- 172.21.15.78:0/1325206116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce594ec0 0x55c2ce585b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:02:57.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.374+0000 7efde3e00700 1 --2- 172.21.15.78:0/1325206116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce5855d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7efdd00098e0 tx=0x7efdd0005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8816226caa05cfff server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:57.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.375+0000 7efdd6a00700 1 -- 172.21.15.78:0/1325206116 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd001d040 con 0x55c2ce58afa0 2024-03-14T10:02:57.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.375+0000 7efdd6a00700 1 -- 172.21.15.78:0/1325206116 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdd001e040 con 0x55c2ce58afa0 2024-03-14T10:02:57.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.375+0000 7efdd6a00700 1 -- 172.21.15.78:0/1325206116 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd0024a20 con 0x55c2ce58afa0 2024-03-14T10:02:57.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.375+0000 7efdff7adb80 1 -- 172.21.15.78:0/1325206116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 msgr2=0x55c2ce5855d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.375+0000 7efdff7adb80 1 --2- 172.21.15.78:0/1325206116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce5855d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7efdd00098e0 tx=0x7efdd0005e80 comp rx=0 tx=0).stop 2024-03-14T10:02:57.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.375+0000 7efdff7adb80 1 -- 172.21.15.78:0/1325206116 shutdown_connections 2024-03-14T10:02:57.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.375+0000 7efdff7adb80 1 --2- 172.21.15.78:0/1325206116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce594ec0 0x55c2ce585b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.375+0000 7efdff7adb80 1 --2- 172.21.15.78:0/1325206116 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce58a680 0x55c2ce59bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.375+0000 7efdff7adb80 1 --2- 172.21.15.78:0/1325206116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce5855d0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.375+0000 7efdff7adb80 1 -- 172.21.15.78:0/1325206116 >> 172.21.15.78:0/1325206116 conn(0x55c2ce500220 msgr2=0x55c2ce5a0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:57.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.375+0000 7efdff7adb80 1 -- 172.21.15.78:0/1325206116 shutdown_connections 2024-03-14T10:02:57.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.375+0000 7efdff7adb80 1 -- 172.21.15.78:0/1325206116 wait complete. 2024-03-14T10:02:57.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.376+0000 7efdff7adb80 1 -- start start 2024-03-14T10:02:57.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.376+0000 7efdff7adb80 1 Processor -- start 2024-03-14T10:02:57.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.377+0000 7efdff7adb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce58a680 0x55c2ce599d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.377+0000 7efdff7adb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce59a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.377+0000 7efdff7adb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce594ec0 0x55c2ce5ef5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.377+0000 7efdff7adb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ce5efaf0 con 0x55c2ce58afa0 2024-03-14T10:02:57.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.377+0000 7efdff7adb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ce5efc60 con 0x55c2ce58a680 2024-03-14T10:02:57.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.377+0000 7efdff7adb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ce5efe20 con 0x55c2ce594ec0 2024-03-14T10:02:57.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.377+0000 7efdff7adb80 10 client.0 _ll_register_callbacks cb 0x7efdd0005350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:02:57.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.377+0000 7efde3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce59a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.377+0000 7efde9200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce58a680 0x55c2ce599d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.377+0000 7efde3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce59a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35150/0 (socket says 172.21.15.78:35150) 2024-03-14T10:02:57.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.377+0000 7efde3e00700 1 -- 172.21.15.78:0/1348889304 learned_addr learned my addr 172.21.15.78:0/1348889304 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:02:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.377+0000 7efde9c00700 1 --2- 172.21.15.78:0/1348889304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce594ec0 0x55c2ce5ef5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.377+0000 7efde3e00700 1 -- 172.21.15.78:0/1348889304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce594ec0 msgr2=0x55c2ce5ef5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.377+0000 7efde3e00700 1 --2- 172.21.15.78:0/1348889304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce594ec0 0x55c2ce5ef5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.378+0000 7efde3e00700 1 -- 172.21.15.78:0/1348889304 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce58a680 msgr2=0x55c2ce599d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:02:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.378+0000 7efde3e00700 1 --2- 172.21.15.78:0/1348889304 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce58a680 0x55c2ce599d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.378+0000 7efde3e00700 1 -- 172.21.15.78:0/1348889304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2ce35e3b0 con 0x55c2ce58afa0 2024-03-14T10:02:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.378+0000 7efde9200700 1 --2- 172.21.15.78:0/1348889304 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce58a680 0x55c2ce599d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:02:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.378+0000 7efde3e00700 1 --2- 172.21.15.78:0/1348889304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce59a260 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009350 tx=0x7efdd000b750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7be7c7656c40b8b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:02:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.378+0000 7efdcaa00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:02:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.378+0000 7efdcaa00700 1 -- 172.21.15.78:0/1348889304 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd001d070 con 0x55c2ce58afa0 2024-03-14T10:02:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.378+0000 7efdcaa00700 1 -- 172.21.15.78:0/1348889304 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9496+0+0 (secure 0 0 0) 0x7efdd0000e40 con 0x55c2ce58afa0 2024-03-14T10:02:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.378+0000 7efdff7adb80 1 -- 172.21.15.78:0/1348889304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efdd8099950 con 0x55c2ce58afa0 2024-03-14T10:02:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.378+0000 7efdcaa00700 1 -- 172.21.15.78:0/1348889304 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdd000bcb0 con 0x55c2ce58afa0 2024-03-14T10:02:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.378+0000 7efdff7adb80 10 client.17400 fetch_fsmap learned FSMap version 25 2024-03-14T10:02:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.378+0000 7efdff7adb80 1 -- 172.21.15.78:0/1348889304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c2ce30ab60 con 0x55c2ce58afa0 2024-03-14T10:02:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.379+0000 7efdcaa00700 1 -- 172.21.15.78:0/1348889304 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdd8099950 con 0x55c2ce58afa0 2024-03-14T10:02:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.379+0000 7efdcaa00700 1 -- 172.21.15.78:0/1348889304 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7efdd0003c50 con 0x55c2ce58afa0 2024-03-14T10:02:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.379+0000 7efdff7adb80 10 client.17400 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:02:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.379+0000 7efdff7adb80 10 client.17400 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:02:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.379+0000 7efdff7adb80 20 client.17400 populate_metadata read hostname 'smithi078' 2024-03-14T10:02:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.379+0000 7efdff7adb80 1 --2- 172.21.15.78:0/1348889304 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c2ce5f7110 0x55c2ce5f9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:02:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.379+0000 7efdff7adb80 4 client.17400 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:02:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.379+0000 7efdff7adb80 1 -- 172.21.15.78:0/1348889304 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c2ce32bcf0 con 0x55c2ce5f7110 2024-03-14T10:02:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.379+0000 7efde9200700 1 --2- 172.21.15.78:0/1348889304 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c2ce5f7110 0x55c2ce5f9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:02:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efdcaa00700 10 client.17400 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efde9200700 1 --2- 172.21.15.78:0/1348889304 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c2ce5f7110 0x55c2ce5f9500 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=943de4e90609c9f4 server_cookie=9aba5ab7164c89cd in_seq=0 out_seq=0 2024-03-14T10:02:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efdcaa00700 1 -- 172.21.15.78:0/1348889304 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c2ce32bcf0 con 0x55c2ce5f7110 2024-03-14T10:02:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efdcaa00700 10 client.17400 handle_command_reply: tid=0 2024-03-14T10:02:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efdff7adb80 1 client.17400 shutdown 2024-03-14T10:02:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efdff7adb80 1 -- 172.21.15.78:0/1348889304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 msgr2=0x55c2ce59a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efdff7adb80 1 --2- 172.21.15.78:0/1348889304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce59a260 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009350 tx=0x7efdd000b750 comp rx=0 tx=0).stop 2024-03-14T10:02:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efdff7adb80 1 -- 172.21.15.78:0/1348889304 shutdown_connections 2024-03-14T10:02:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efdff7adb80 1 --2- 172.21.15.78:0/1348889304 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55c2ce5f7110 0x55c2ce5f9500 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efdff7adb80 1 --2- 172.21.15.78:0/1348889304 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c2ce58a680 0x55c2ce599d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efdcaa00700 0 client.17400 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:02:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efdff7adb80 1 --2- 172.21.15.78:0/1348889304 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55c2ce594ec0 0x55c2ce5ef5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efdff7adb80 1 --2- 172.21.15.78:0/1348889304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c2ce58afa0 0x55c2ce59a260 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efdff7adb80 1 -- 172.21.15.78:0/1348889304 >> 172.21.15.78:0/1348889304 conn(0x55c2ce500220 msgr2=0x55c2ce5a0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efdff7adb80 1 -- 172.21.15.78:0/1348889304 shutdown_connections 2024-03-14T10:02:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efdff7adb80 1 -- 172.21.15.78:0/1348889304 wait complete. 2024-03-14T10:02:57.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.380+0000 7efdff7adb80 20 client.17400 trim_cache size 0 max 16384 2024-03-14T10:02:57.386 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:02:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-14T10:02:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T10:02:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: "1cd6d5bb-34eb-4b12-9407-8eea1021cf68": { 2024-03-14T10:02:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T10:02:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "1cd6d5bb-34eb-4b12-9407-8eea1021cf68", 2024-03-14T10:02:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T10:02:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:02:57.386 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:02:57.386 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:02:57.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.381+0000 7efdebc00700 1 -- 172.21.15.78:0/2226524921 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efdcc0a0c00 msgr2=0x7efdcc0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.381+0000 7efdebc00700 1 --2- 172.21.15.78:0/2226524921 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efdcc0a0c00 0x7efdcc0a30c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efddc009ec0 tx=0x7efddc009920 comp rx=0 tx=0).stop 2024-03-14T10:02:57.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.381+0000 7efdebc00700 1 -- 172.21.15.78:0/2226524921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efde40680b0 msgr2=0x7efde40ffc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:02:57.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.381+0000 7efdebc00700 1 --2- 172.21.15.78:0/2226524921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efde40680b0 0x7efde40ffc20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7efdd800e9a0 tx=0x7efdd800ecb0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.381+0000 7efdebc00700 1 -- 172.21.15.78:0/2226524921 shutdown_connections 2024-03-14T10:02:57.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.381+0000 7efdebc00700 1 --2- 172.21.15.78:0/2226524921 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efdcc0a0c00 0x7efdcc0a30c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.381+0000 7efdebc00700 1 --2- 172.21.15.78:0/2226524921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efde40680b0 0x7efde40ffc20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.381+0000 7efdebc00700 1 --2- 172.21.15.78:0/2226524921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efde41006a0 0x7efde41a57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.381+0000 7efdebc00700 1 --2- 172.21.15.78:0/2226524921 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efde4106060 0x7efde4100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:02:57.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.381+0000 7efdebc00700 1 -- 172.21.15.78:0/2226524921 >> 172.21.15.78:0/2226524921 conn(0x7efde4076d80 msgr2=0x7efde40fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:02:57.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.381+0000 7efdebc00700 1 -- 172.21.15.78:0/2226524921 shutdown_connections 2024-03-14T10:02:57.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:02:57.381+0000 7efdebc00700 1 -- 172.21.15.78:0/2226524921 wait complete. 2024-03-14T10:02:57.399 INFO:tasks.cephfs.filesystem:scrub status for tag:1cd6d5bb-34eb-4b12-9407-8eea1021cf68 - {'path': '/', 'tag': '1cd6d5bb-34eb-4b12-9407-8eea1021cf68', 'options': 'recursive,force'} 2024-03-14T10:02:57.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:57 smithi152 ceph-mon[48279]: pgmap v270: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:02:57.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:57 smithi132 ceph-mon[47959]: pgmap v270: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:02:58.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:58 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T10:02:58.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:58 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T10:02:58.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:58 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T10:02:58.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:58 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T10:02:58.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:58 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T10:02:58.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:58 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T10:02:58.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:58 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T10:02:58.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:58 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T10:02:58.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:58 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T10:02:58.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:58 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T10:02:58.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:58 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T10:02:58.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:58 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T10:02:59.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:59 smithi152 ceph-mon[48279]: pgmap v271: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:59.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:02:59 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T10:02:59.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:59 smithi132 ceph-mon[47959]: pgmap v271: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:59.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:02:59 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T10:02:59.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:59 smithi078 ceph-mon[38143]: pgmap v271: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:02:59.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:02:59 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T10:03:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:01 smithi152 ceph-mon[48279]: pgmap v272: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:01 smithi132 ceph-mon[47959]: pgmap v272: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:01.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:01 smithi078 ceph-mon[38143]: pgmap v272: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:03 smithi152 ceph-mon[48279]: pgmap v273: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:03:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:03 smithi132 ceph-mon[47959]: pgmap v273: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:03:03.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:03 smithi078 ceph-mon[38143]: pgmap v273: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:03:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:05 smithi152 ceph-mon[48279]: pgmap v274: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:03:05.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:05 smithi132 ceph-mon[47959]: pgmap v274: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:03:05.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:05 smithi078 ceph-mon[38143]: pgmap v274: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:03:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:07 smithi152 ceph-mon[48279]: pgmap v275: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:03:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:07 smithi132 ceph-mon[47959]: pgmap v275: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:03:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:07 smithi078 ceph-mon[38143]: pgmap v275: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:03:09.252 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:08 smithi152 ceph-mon[48279]: pgmap v276: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:03:09.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:08 smithi132 ceph-mon[47959]: pgmap v276: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:03:09.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:08 smithi078 ceph-mon[38143]: pgmap v276: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:03:10.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:09 smithi152 ceph-mon[48279]: osdmap e93: 12 total, 12 up, 12 in 2024-03-14T10:03:10.273 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:09 smithi132 ceph-mon[47959]: osdmap e93: 12 total, 12 up, 12 in 2024-03-14T10:03:10.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:09 smithi078 ceph-mon[38143]: osdmap e93: 12 total, 12 up, 12 in 2024-03-14T10:03:11.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:10 smithi152 ceph-mon[48279]: pgmap v278: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-14T10:03:11.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:10 smithi132 ceph-mon[47959]: pgmap v278: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-14T10:03:11.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:10 smithi078 ceph-mon[38143]: pgmap v278: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-14T10:03:13.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:13 smithi152 ceph-mon[48279]: pgmap v279: 129 pgs: 9 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 107 active+clean; 261 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:03:13.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:13 smithi132 ceph-mon[47959]: pgmap v279: 129 pgs: 9 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 107 active+clean; 261 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:03:13.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:13 smithi078 ceph-mon[38143]: pgmap v279: 129 pgs: 9 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 107 active+clean; 261 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:03:15.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:15 smithi152 ceph-mon[48279]: pgmap v280: 129 pgs: 9 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 107 active+clean; 261 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:03:15.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:15 smithi132 ceph-mon[47959]: pgmap v280: 129 pgs: 9 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 107 active+clean; 261 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:03:15.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:15 smithi078 ceph-mon[38143]: pgmap v280: 129 pgs: 9 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 107 active+clean; 261 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:03:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:17 smithi152 ceph-mon[48279]: pgmap v281: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:03:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:17 smithi132 ceph-mon[47959]: pgmap v281: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:03:17.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:17 smithi078 ceph-mon[38143]: pgmap v281: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:03:19.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:18 smithi152 ceph-mon[48279]: osdmap e94: 12 total, 12 up, 12 in 2024-03-14T10:03:19.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:18 smithi152 ceph-mon[48279]: pgmap v283: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2024-03-14T10:03:19.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:18 smithi132 ceph-mon[47959]: osdmap e94: 12 total, 12 up, 12 in 2024-03-14T10:03:19.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:18 smithi132 ceph-mon[47959]: pgmap v283: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2024-03-14T10:03:19.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:18 smithi078 ceph-mon[38143]: osdmap e94: 12 total, 12 up, 12 in 2024-03-14T10:03:19.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:18 smithi078 ceph-mon[38143]: pgmap v283: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 222 B/s wr, 6 op/s 2024-03-14T10:03:21.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:21 smithi152 ceph-mon[48279]: pgmap v284: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:03:21.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:21 smithi132 ceph-mon[47959]: pgmap v284: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:03:21.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:21 smithi078 ceph-mon[38143]: pgmap v284: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:03:23.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:23 smithi152 ceph-mon[48279]: pgmap v285: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-14T10:03:23.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:23 smithi132 ceph-mon[47959]: pgmap v285: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-14T10:03:23.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:23 smithi078 ceph-mon[38143]: pgmap v285: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-14T10:03:25.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:25 smithi152 ceph-mon[48279]: pgmap v286: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-14T10:03:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:25 smithi132 ceph-mon[47959]: pgmap v286: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-14T10:03:25.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:25 smithi078 ceph-mon[38143]: pgmap v286: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-14T10:03:26.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:26 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T10:03:26.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:26 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T10:03:26.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:26 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T10:03:27.400 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:03:27.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:27 smithi078 ceph-mon[38143]: pgmap v287: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-14T10:03:27.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:27 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T10:03:27.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:27 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T10:03:27.462 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:27 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:03:27.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:27 smithi152 ceph-mon[48279]: pgmap v287: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-14T10:03:27.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:27 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T10:03:27.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:27 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T10:03:27.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:27 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:03:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:27 smithi132 ceph-mon[47959]: pgmap v287: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-14T10:03:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:27 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T10:03:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:27 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T10:03:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:27 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:03:27.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.602+0000 7faabb800700 1 -- 172.21.15.78:0/945658140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faab40731e0 msgr2=0x7faab40735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:27.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.602+0000 7faabb800700 1 --2- 172.21.15.78:0/945658140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faab40731e0 0x7faab40735c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7faaa8009990 tx=0x7faaa8009ca0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.602+0000 7faabb800700 1 -- 172.21.15.78:0/945658140 shutdown_connections 2024-03-14T10:03:27.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.602+0000 7faabb800700 1 --2- 172.21.15.78:0/945658140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faab410d500 0x7faab410f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.602+0000 7faabb800700 1 --2- 172.21.15.78:0/945658140 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faab40731e0 0x7faab40735c0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.602+0000 7faabb800700 1 --2- 172.21.15.78:0/945658140 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faab4073b00 0x7faab410cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.602+0000 7faabb800700 1 -- 172.21.15.78:0/945658140 >> 172.21.15.78:0/945658140 conn(0x7faab40fc7b0 msgr2=0x7faab40febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:27.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.602+0000 7faabb800700 1 -- 172.21.15.78:0/945658140 shutdown_connections 2024-03-14T10:03:27.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.602+0000 7faabb800700 1 -- 172.21.15.78:0/945658140 wait complete. 2024-03-14T10:03:27.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.603+0000 7faabb800700 1 Processor -- start 2024-03-14T10:03:27.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.603+0000 7faabb800700 1 -- start start 2024-03-14T10:03:27.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.603+0000 7faabb800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faab40731e0 0x7faab41997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.603+0000 7faabb800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faab4073b00 0x7faab4199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.603+0000 7faabb800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faab410d500 0x7faab419d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.603+0000 7faabb800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab419d610 con 0x7faab40731e0 2024-03-14T10:03:27.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.603+0000 7faabb800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab419d780 con 0x7faab410d500 2024-03-14T10:03:27.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.603+0000 7faabb800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab419d940 con 0x7faab4073b00 2024-03-14T10:03:27.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.604+0000 7faab8e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faab40731e0 0x7faab41997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.604+0000 7faab3e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faab4073b00 0x7faab4199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.604+0000 7faab8e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faab40731e0 0x7faab41997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37288/0 (socket says 172.21.15.78:37288) 2024-03-14T10:03:27.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.604+0000 7faab8e00700 1 -- 172.21.15.78:0/107041746 learned_addr learned my addr 172.21.15.78:0/107041746 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:27.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.604+0000 7faab9800700 1 --2- 172.21.15.78:0/107041746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faab410d500 0x7faab419d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.604+0000 7faab3e00700 1 -- 172.21.15.78:0/107041746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faab410d500 msgr2=0x7faab419d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:27.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.604+0000 7faab3e00700 1 --2- 172.21.15.78:0/107041746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faab410d500 0x7faab419d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.604+0000 7faab3e00700 1 -- 172.21.15.78:0/107041746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faab40731e0 msgr2=0x7faab41997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:27.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.604+0000 7faab3e00700 1 --2- 172.21.15.78:0/107041746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faab40731e0 0x7faab41997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.604+0000 7faab3e00700 1 -- 172.21.15.78:0/107041746 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa80095f0 con 0x7faab4073b00 2024-03-14T10:03:27.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.604+0000 7faab8e00700 1 --2- 172.21.15.78:0/107041746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faab40731e0 0x7faab41997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:03:27.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.604+0000 7faab3e00700 1 --2- 172.21.15.78:0/107041746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faab4073b00 0x7faab4199d20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7faaa000e9f0 tx=0x7faaa000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:27.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.605+0000 7faab1600700 1 -- 172.21.15.78:0/107041746 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa000c9c0 con 0x7faab4073b00 2024-03-14T10:03:27.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.605+0000 7faab1600700 1 -- 172.21.15.78:0/107041746 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faaa0010430 con 0x7faab4073b00 2024-03-14T10:03:27.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.605+0000 7faabb800700 1 -- 172.21.15.78:0/107041746 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab419dc20 con 0x7faab4073b00 2024-03-14T10:03:27.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.605+0000 7faab1600700 1 -- 172.21.15.78:0/107041746 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa0018680 con 0x7faab4073b00 2024-03-14T10:03:27.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.605+0000 7faabb800700 1 -- 172.21.15.78:0/107041746 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab41a6580 con 0x7faab4073b00 2024-03-14T10:03:27.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.608+0000 7faab1600700 1 -- 172.21.15.78:0/107041746 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7faaa0010860 con 0x7faab4073b00 2024-03-14T10:03:27.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.609+0000 7faacf2b6b80 1 Processor -- start 2024-03-14T10:03:27.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.609+0000 7faacf2b6b80 1 -- start start 2024-03-14T10:03:27.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.609+0000 7faab1600700 1 --2- 172.21.15.78:0/107041746 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7faa9c0a0c00 0x7faa9c0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.609+0000 7faacf2b6b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e35680 0x561282e35a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.609+0000 7faacf2b6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35fa0 0x561282e3f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.609+0000 7faacf2b6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e3fec0 0x561282e422b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.609+0000 7faacf2b6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561282d8e990 con 0x561282e35fa0 2024-03-14T10:03:27.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.609+0000 7faacf2b6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x561282d8c350 con 0x561282e3fec0 2024-03-14T10:03:27.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.609+0000 7faacf2b6b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561282d87c40 con 0x561282e35680 2024-03-14T10:03:27.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.609+0000 7faab1600700 1 -- 172.21.15.78:0/107041746 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7faaa0014070 con 0x7faab4073b00 2024-03-14T10:03:27.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.610+0000 7faab9800700 1 --2- 172.21.15.78:0/107041746 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7faa9c0a0c00 0x7faa9c0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.610+0000 7faab3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35fa0 0x561282e3f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.610+0000 7faab3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35fa0 0x561282e3f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37302/0 (socket says 172.21.15.78:37302) 2024-03-14T10:03:27.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.610+0000 7faab3e00700 1 -- 172.21.15.78:0/1175690995 learned_addr learned my addr 172.21.15.78:0/1175690995 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:27.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.610+0000 7faab8e00700 1 --2- 172.21.15.78:0/1175690995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e35680 0x561282e35a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.610+0000 7faab9800700 1 --2- 172.21.15.78:0/1175690995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e3fec0 0x561282e422b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.610+0000 7faab3e00700 1 -- 172.21.15.78:0/1175690995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e35680 msgr2=0x561282e35a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:27.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.610+0000 7faab3e00700 1 --2- 172.21.15.78:0/1175690995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e35680 0x561282e35a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.610+0000 7faab3e00700 1 -- 172.21.15.78:0/1175690995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e3fec0 msgr2=0x561282e422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:27.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.610+0000 7faab3e00700 1 --2- 172.21.15.78:0/1175690995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e3fec0 0x561282e422b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.610+0000 7faab3e00700 1 -- 172.21.15.78:0/1175690995 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561282c093b0 con 0x561282e35fa0 2024-03-14T10:03:27.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.610+0000 7faab8e00700 1 --2- 172.21.15.78:0/1175690995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e35680 0x561282e35a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:03:27.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.610+0000 7faab3e00700 1 --2- 172.21.15.78:0/1175690995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35fa0 0x561282e3f980 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7faaa0096330 tx=0x7faaa0099320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1889426354e0edf server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:27.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.610+0000 7faab9800700 1 --2- 172.21.15.78:0/107041746 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7faa9c0a0c00 0x7faa9c0a30c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7faaac009ec0 tx=0x7faaac0134a0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:27.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.611+0000 7faaa6000700 1 -- 172.21.15.78:0/1175690995 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa009d6f0 con 0x561282e35fa0 2024-03-14T10:03:27.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.611+0000 7faaa6000700 1 -- 172.21.15.78:0/1175690995 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faaa00a1d40 con 0x561282e35fa0 2024-03-14T10:03:27.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.611+0000 7faaa6000700 1 -- 172.21.15.78:0/1175690995 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa0098fb0 con 0x561282e35fa0 2024-03-14T10:03:27.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.611+0000 7faacf2b6b80 1 -- 172.21.15.78:0/1175690995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35fa0 msgr2=0x561282e3f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:27.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.611+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/1175690995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35fa0 0x561282e3f980 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7faaa0096330 tx=0x7faaa0099320 comp rx=0 tx=0).stop 2024-03-14T10:03:27.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.611+0000 7faacf2b6b80 1 -- 172.21.15.78:0/1175690995 shutdown_connections 2024-03-14T10:03:27.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.611+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/1175690995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e3fec0 0x561282e422b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.611+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/1175690995 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e35680 0x561282e35a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.611+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/1175690995 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35fa0 0x561282e3f980 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.611+0000 7faacf2b6b80 1 -- 172.21.15.78:0/1175690995 >> 172.21.15.78:0/1175690995 conn(0x561282dab220 msgr2=0x561282e30aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:27.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.611+0000 7faacf2b6b80 1 -- 172.21.15.78:0/1175690995 shutdown_connections 2024-03-14T10:03:27.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.611+0000 7faacf2b6b80 1 -- 172.21.15.78:0/1175690995 wait complete. 2024-03-14T10:03:27.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.612+0000 7faacf2b6b80 1 -- start start 2024-03-14T10:03:27.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.612+0000 7faacf2b6b80 1 Processor -- start 2024-03-14T10:03:27.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faacf2b6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e3f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faacf2b6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e35fa0 0x561282e3cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faacf2b6b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e3fec0 0x561282e3d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faacf2b6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561282d23d80 con 0x561282e35680 2024-03-14T10:03:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faacf2b6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x561282d23a20 con 0x561282e35fa0 2024-03-14T10:03:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faacf2b6b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561282c0b6a0 con 0x561282e3fec0 2024-03-14T10:03:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faacf2b6b80 10 client.0 _ll_register_callbacks cb 0x561282b8f3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:03:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faab8e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e3f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faab8e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e3f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37314/0 (socket says 172.21.15.78:37314) 2024-03-14T10:03:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faab3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e35fa0 0x561282e3cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faab8e00700 1 -- 172.21.15.78:0/3030913861 learned_addr learned my addr 172.21.15.78:0/3030913861 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faab9800700 1 --2- 172.21.15.78:0/3030913861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e3fec0 0x561282e3d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faab8e00700 1 -- 172.21.15.78:0/3030913861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e3fec0 msgr2=0x561282e3d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faab8e00700 1 --2- 172.21.15.78:0/3030913861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e3fec0 0x561282e3d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faab8e00700 1 -- 172.21.15.78:0/3030913861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e35fa0 msgr2=0x561282e3cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faab8e00700 1 --2- 172.21.15.78:0/3030913861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e35fa0 0x561282e3cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faab8e00700 1 -- 172.21.15.78:0/3030913861 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab4110630 con 0x561282e35680 2024-03-14T10:03:27.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.613+0000 7faab3e00700 1 --2- 172.21.15.78:0/3030913861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e35fa0 0x561282e3cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:03:27.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.614+0000 7faab8e00700 1 --2- 172.21.15.78:0/3030913861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e3f940 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7faaa8005280 tx=0x7faaa80052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=208d5491bb245d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:27.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.614+0000 7faa9aa00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:03:27.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.614+0000 7faa9aa00700 1 -- 172.21.15.78:0/3030913861 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa801d070 con 0x561282e35680 2024-03-14T10:03:27.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.614+0000 7faa9aa00700 1 -- 172.21.15.78:0/3030913861 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7faaa8004910 con 0x561282e35680 2024-03-14T10:03:27.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.614+0000 7faacf2b6b80 1 -- 172.21.15.78:0/3030913861 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561282e28720 con 0x561282e35680 2024-03-14T10:03:27.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.614+0000 7faa9aa00700 1 -- 172.21.15.78:0/3030913861 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7faaa8004ad0 con 0x561282e35680 2024-03-14T10:03:27.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.614+0000 7faacf2b6b80 10 client.17412 fetch_fsmap learned FSMap version 25 2024-03-14T10:03:27.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.614+0000 7faacf2b6b80 1 -- 172.21.15.78:0/3030913861 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7faab41a6580 con 0x561282e35680 2024-03-14T10:03:27.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.614+0000 7faa9aa00700 1 -- 172.21.15.78:0/3030913861 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561282e28720 con 0x561282e35680 2024-03-14T10:03:27.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.614+0000 7faa9aa00700 1 -- 172.21.15.78:0/3030913861 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7faaa800fc20 con 0x561282e35680 2024-03-14T10:03:27.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.615+0000 7faacf2b6b80 10 client.17412 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:03:27.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.615+0000 7faacf2b6b80 10 client.17412 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:03:27.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.615+0000 7faacf2b6b80 20 client.17412 populate_metadata read hostname 'smithi078' 2024-03-14T10:03:27.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.615+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/3030913861 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561282e4fae0 0x561282e51f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.615+0000 7faacf2b6b80 4 client.17412 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:03:27.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.615+0000 7faacf2b6b80 1 -- 172.21.15.78:0/3030913861 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561282d80700 con 0x561282e4fae0 2024-03-14T10:03:27.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.615+0000 7faab3e00700 1 --2- 172.21.15.78:0/3030913861 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561282e4fae0 0x561282e51f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.615+0000 7faab3e00700 1 --2- 172.21.15.78:0/3030913861 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561282e4fae0 0x561282e51f00 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e99ff69e637d627 server_cookie=e5b7d94cfb5af507 in_seq=0 out_seq=0 2024-03-14T10:03:27.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.615+0000 7faa9aa00700 10 client.17412 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:03:27.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.616+0000 7faa9aa00700 1 -- 172.21.15.78:0/3030913861 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x561282d80700 con 0x561282e4fae0 2024-03-14T10:03:27.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.616+0000 7faa9aa00700 10 client.17412 handle_command_reply: tid=0 2024-03-14T10:03:27.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.616+0000 7faacf2b6b80 1 client.17412 shutdown 2024-03-14T10:03:27.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.616+0000 7faacf2b6b80 1 -- 172.21.15.78:0/3030913861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 msgr2=0x561282e3f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:27.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.616+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/3030913861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e3f940 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7faaa8005280 tx=0x7faaa80052b0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.616+0000 7faacf2b6b80 1 -- 172.21.15.78:0/3030913861 shutdown_connections 2024-03-14T10:03:27.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.616+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/3030913861 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e3fec0 0x561282e3d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.616+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/3030913861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e3f940 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.616+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/3030913861 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561282e4fae0 0x561282e51f00 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.616+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/3030913861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e35fa0 0x561282e3cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.616+0000 7faacf2b6b80 1 -- 172.21.15.78:0/3030913861 >> 172.21.15.78:0/3030913861 conn(0x561282dab220 msgr2=0x561282e30690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:27.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.616+0000 7faa9aa00700 0 client.17412 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:03:27.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.616+0000 7faacf2b6b80 1 -- 172.21.15.78:0/3030913861 shutdown_connections 2024-03-14T10:03:27.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.617+0000 7faacf2b6b80 1 -- 172.21.15.78:0/3030913861 wait complete. 2024-03-14T10:03:27.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.617+0000 7faacf2b6b80 20 client.17412 trim_cache size 0 max 16384 2024-03-14T10:03:27.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.633+0000 7faacf2b6b80 1 Processor -- start 2024-03-14T10:03:27.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.633+0000 7faacf2b6b80 1 -- start start 2024-03-14T10:03:27.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.634+0000 7faacf2b6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e3cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.634+0000 7faacf2b6b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e35fa0 0x561282e46af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.634+0000 7faacf2b6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e3fec0 0x561282e3f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.634+0000 7faacf2b6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561282e370c0 con 0x561282e35680 2024-03-14T10:03:27.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.634+0000 7faacf2b6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x561282e31e30 con 0x561282e3fec0 2024-03-14T10:03:27.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.634+0000 7faacf2b6b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561282e32060 con 0x561282e35fa0 2024-03-14T10:03:27.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.634+0000 7faab8e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e3cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.634+0000 7faab8e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e3cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37328/0 (socket says 172.21.15.78:37328) 2024-03-14T10:03:27.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.634+0000 7faab8e00700 1 -- 172.21.15.78:0/2522517211 learned_addr learned my addr 172.21.15.78:0/2522517211 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:27.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.634+0000 7faab3e00700 1 --2- 172.21.15.78:0/2522517211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e35fa0 0x561282e46af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.634+0000 7faab9800700 1 --2- 172.21.15.78:0/2522517211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e3fec0 0x561282e3f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.634+0000 7faab8e00700 1 -- 172.21.15.78:0/2522517211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e35fa0 msgr2=0x561282e46af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:27.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.634+0000 7faab8e00700 1 --2- 172.21.15.78:0/2522517211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e35fa0 0x561282e46af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.634+0000 7faab8e00700 1 -- 172.21.15.78:0/2522517211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e3fec0 msgr2=0x561282e3f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:27.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.634+0000 7faab8e00700 1 --2- 172.21.15.78:0/2522517211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e3fec0 0x561282e3f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.634+0000 7faab8e00700 1 -- 172.21.15.78:0/2522517211 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561282bb5b60 con 0x561282e35680 2024-03-14T10:03:27.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.635+0000 7faab8e00700 1 --2- 172.21.15.78:0/2522517211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e3cbd0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7faaa8000c00 tx=0x7faaa80244f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b242f0e7cc6b70 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:27.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.635+0000 7faaa6a00700 1 -- 172.21.15.78:0/2522517211 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa801d070 con 0x561282e35680 2024-03-14T10:03:27.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.635+0000 7faaa6a00700 1 -- 172.21.15.78:0/2522517211 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faaa801e040 con 0x561282e35680 2024-03-14T10:03:27.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.635+0000 7faaa6a00700 1 -- 172.21.15.78:0/2522517211 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa8024850 con 0x561282e35680 2024-03-14T10:03:27.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.635+0000 7faacf2b6b80 1 -- 172.21.15.78:0/2522517211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 msgr2=0x561282e3cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:27.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.635+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/2522517211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e3cbd0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7faaa8000c00 tx=0x7faaa80244f0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.635+0000 7faacf2b6b80 1 -- 172.21.15.78:0/2522517211 shutdown_connections 2024-03-14T10:03:27.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.635+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/2522517211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e3fec0 0x561282e3f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.635+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/2522517211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e3cbd0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.635+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/2522517211 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e35fa0 0x561282e46af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.635+0000 7faacf2b6b80 1 -- 172.21.15.78:0/2522517211 >> 172.21.15.78:0/2522517211 conn(0x561282dab220 msgr2=0x561282e3c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.636+0000 7faacf2b6b80 1 -- 172.21.15.78:0/2522517211 shutdown_connections 2024-03-14T10:03:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.636+0000 7faacf2b6b80 1 -- 172.21.15.78:0/2522517211 wait complete. 2024-03-14T10:03:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.636+0000 7faacf2b6b80 1 -- start start 2024-03-14T10:03:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.636+0000 7faacf2b6b80 1 Processor -- start 2024-03-14T10:03:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.637+0000 7faacf2b6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.637+0000 7faacf2b6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e35fa0 0x561282e44730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.637+0000 7faacf2b6b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e3fec0 0x561282e44c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.637+0000 7faacf2b6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561282ea06e0 con 0x561282e35680 2024-03-14T10:03:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.637+0000 7faacf2b6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x561282ea0850 con 0x561282e35fa0 2024-03-14T10:03:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.637+0000 7faacf2b6b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x561282ea0a10 con 0x561282e3fec0 2024-03-14T10:03:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.637+0000 7faacf2b6b80 10 client.0 _ll_register_callbacks cb 0x7faaa800be10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:03:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.637+0000 7faab8e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.637+0000 7faab3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e35fa0 0x561282e44730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.637+0000 7faab8e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e441f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37334/0 (socket says 172.21.15.78:37334) 2024-03-14T10:03:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.637+0000 7faab3e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e35fa0 0x561282e44730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:60840/0 (socket says 172.21.15.78:60840) 2024-03-14T10:03:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.637+0000 7faab8e00700 1 -- 172.21.15.78:0/915986992 learned_addr learned my addr 172.21.15.78:0/915986992 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.637+0000 7faab9800700 1 --2- 172.21.15.78:0/915986992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e3fec0 0x561282e44c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.638+0000 7faab3e00700 1 -- 172.21.15.78:0/915986992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e3fec0 msgr2=0x561282e44c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.638+0000 7faab3e00700 1 --2- 172.21.15.78:0/915986992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e3fec0 0x561282e44c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.638+0000 7faab3e00700 1 -- 172.21.15.78:0/915986992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 msgr2=0x561282e441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.638+0000 7faab3e00700 1 --2- 172.21.15.78:0/915986992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e441f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.638+0000 7faab3e00700 1 -- 172.21.15.78:0/915986992 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab41a6580 con 0x561282e35fa0 2024-03-14T10:03:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.638+0000 7faab3e00700 1 --2- 172.21.15.78:0/915986992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e35fa0 0x561282e44730 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x561282e364b0 tx=0x7faaa0009790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b749e50fcb962fd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.638+0000 7faa9aa00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T10:03:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.638+0000 7faa9aa00700 1 -- 172.21.15.78:0/915986992 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa00b2280 con 0x561282e35fa0 2024-03-14T10:03:27.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.638+0000 7faacf2b6b80 1 -- 172.21.15.78:0/915986992 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faaa00a1d40 con 0x561282e35fa0 2024-03-14T10:03:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.638+0000 7faa9aa00700 1 -- 172.21.15.78:0/915986992 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7faaa0099b20 con 0x561282e35fa0 2024-03-14T10:03:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.638+0000 7faa9aa00700 1 -- 172.21.15.78:0/915986992 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7faaa0099ce0 con 0x561282e35fa0 2024-03-14T10:03:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.638+0000 7faacf2b6b80 10 client.27488 fetch_fsmap learned FSMap version 25 2024-03-14T10:03:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.638+0000 7faacf2b6b80 1 -- 172.21.15.78:0/915986992 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561282d4dea0 con 0x561282e35fa0 2024-03-14T10:03:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.639+0000 7faa9aa00700 1 -- 172.21.15.78:0/915986992 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faaa00a1d40 con 0x561282e35fa0 2024-03-14T10:03:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.639+0000 7faa9aa00700 1 -- 172.21.15.78:0/915986992 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7faaa00b91b0 con 0x561282e35fa0 2024-03-14T10:03:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.639+0000 7faacf2b6b80 10 client.27488 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:03:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.639+0000 7faacf2b6b80 10 client.27488 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:03:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.639+0000 7faacf2b6b80 20 client.27488 populate_metadata read hostname 'smithi078' 2024-03-14T10:03:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.639+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/915986992 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561282ea1f10 0x561282ea4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.639+0000 7faacf2b6b80 4 client.27488 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:03:27.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.639+0000 7faacf2b6b80 1 -- 172.21.15.78:0/915986992 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561282bd6cf0 con 0x561282ea1f10 2024-03-14T10:03:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.639+0000 7faab8e00700 1 --2- 172.21.15.78:0/915986992 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561282ea1f10 0x561282ea4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.640+0000 7faa9aa00700 10 client.27488 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:03:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.640+0000 7faab8e00700 1 --2- 172.21.15.78:0/915986992 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561282ea1f10 0x561282ea4330 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=829492e1461de986 server_cookie=b667aaf54675d3c2 in_seq=0 out_seq=0 2024-03-14T10:03:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.640+0000 7faa9aa00700 1 -- 172.21.15.78:0/915986992 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561282bd6cf0 con 0x561282ea1f10 2024-03-14T10:03:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.640+0000 7faa9aa00700 10 client.27488 handle_command_reply: tid=0 2024-03-14T10:03:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.640+0000 7faacf2b6b80 1 client.27488 shutdown 2024-03-14T10:03:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.640+0000 7faacf2b6b80 1 -- 172.21.15.78:0/915986992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e35fa0 msgr2=0x561282e44730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.640+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/915986992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e35fa0 0x561282e44730 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x561282e364b0 tx=0x7faaa0009790 comp rx=0 tx=0).stop 2024-03-14T10:03:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.640+0000 7faacf2b6b80 1 -- 172.21.15.78:0/915986992 shutdown_connections 2024-03-14T10:03:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.640+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/915986992 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x561282e3fec0 0x561282e44c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.640+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/915986992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561282e35680 0x561282e441f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.640+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/915986992 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561282ea1f10 0x561282ea4330 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.640+0000 7faacf2b6b80 1 --2- 172.21.15.78:0/915986992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x561282e35fa0 0x561282e44730 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.640+0000 7faacf2b6b80 1 -- 172.21.15.78:0/915986992 >> 172.21.15.78:0/915986992 conn(0x561282dab220 msgr2=0x561282e30500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:27.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.640+0000 7faa9aa00700 0 client.27488 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:03:27.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.640+0000 7faacf2b6b80 1 -- 172.21.15.78:0/915986992 shutdown_connections 2024-03-14T10:03:27.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.640+0000 7faacf2b6b80 1 -- 172.21.15.78:0/915986992 wait complete. 2024-03-14T10:03:27.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.641+0000 7faacf2b6b80 20 client.27488 trim_cache size 0 max 16384 2024-03-14T10:03:27.647 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:03:27.647 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T10:03:27.648 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T10:03:27.648 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:03:27.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.641+0000 7faabb800700 1 -- 172.21.15.78:0/107041746 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7faa9c0a0c00 msgr2=0x7faa9c0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.641+0000 7faabb800700 1 --2- 172.21.15.78:0/107041746 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7faa9c0a0c00 0x7faa9c0a30c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7faaac009ec0 tx=0x7faaac0134a0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.641+0000 7faabb800700 1 -- 172.21.15.78:0/107041746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faab4073b00 msgr2=0x7faab4199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.641+0000 7faabb800700 1 --2- 172.21.15.78:0/107041746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faab4073b00 0x7faab4199d20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7faaa000e9f0 tx=0x7faaa000ed00 comp rx=0 tx=0).stop 2024-03-14T10:03:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.641+0000 7faabb800700 1 -- 172.21.15.78:0/107041746 shutdown_connections 2024-03-14T10:03:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.641+0000 7faabb800700 1 --2- 172.21.15.78:0/107041746 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7faa9c0a0c00 0x7faa9c0a30c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.641+0000 7faabb800700 1 --2- 172.21.15.78:0/107041746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faab410d500 0x7faab419d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.641+0000 7faabb800700 1 --2- 172.21.15.78:0/107041746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faab40731e0 0x7faab41997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.641+0000 7faabb800700 1 --2- 172.21.15.78:0/107041746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7faab4073b00 0x7faab4199d20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.641+0000 7faabb800700 1 -- 172.21.15.78:0/107041746 >> 172.21.15.78:0/107041746 conn(0x7faab40fc7b0 msgr2=0x7faab40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.641+0000 7faabb800700 1 -- 172.21.15.78:0/107041746 shutdown_connections 2024-03-14T10:03:27.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.641+0000 7faabb800700 1 -- 172.21.15.78:0/107041746 wait complete. 2024-03-14T10:03:27.660 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T10:03:27.660 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T10:03:27.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.853+0000 7f461fa00700 1 -- 172.21.15.78:0/1106719764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4618105f40 msgr2=0x7f461810dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:27.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.853+0000 7f461fa00700 1 --2- 172.21.15.78:0/1106719764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4618105f40 0x7f461810dac0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f461000b0a0 tx=0x7f461000b3b0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.853+0000 7f461fa00700 1 -- 172.21.15.78:0/1106719764 shutdown_connections 2024-03-14T10:03:27.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.853+0000 7f461fa00700 1 --2- 172.21.15.78:0/1106719764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4618105f40 0x7f461810dac0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.853+0000 7f461fa00700 1 --2- 172.21.15.78:0/1106719764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4618100e20 0x7f4618101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.853+0000 7f461fa00700 1 --2- 172.21.15.78:0/1106719764 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46181017d0 0x7f4618105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.853+0000 7f461fa00700 1 -- 172.21.15.78:0/1106719764 >> 172.21.15.78:0/1106719764 conn(0x7f46180fc700 msgr2=0x7f46180feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:27.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.854+0000 7f461fa00700 1 -- 172.21.15.78:0/1106719764 shutdown_connections 2024-03-14T10:03:27.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.854+0000 7f461fa00700 1 -- 172.21.15.78:0/1106719764 wait complete. 2024-03-14T10:03:27.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.854+0000 7f461fa00700 1 Processor -- start 2024-03-14T10:03:27.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.855+0000 7f461fa00700 1 -- start start 2024-03-14T10:03:27.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.855+0000 7f461fa00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4618100e20 0x7f46180730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.855+0000 7f461fa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4618105f40 0x7f46180735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.855+0000 7f461fa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4618073b30 0x7f46181a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.855+0000 7f461fa00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46181a5e70 con 0x7f4618105f40 2024-03-14T10:03:27.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.855+0000 7f461fa00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46181a5fe0 con 0x7f4618100e20 2024-03-14T10:03:27.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.855+0000 7f461fa00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46181a61a0 con 0x7f4618073b30 2024-03-14T10:03:27.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.855+0000 7f4617e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4618105f40 0x7f46180735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.855+0000 7f4617e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4618105f40 0x7f46180735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37364/0 (socket says 172.21.15.78:37364) 2024-03-14T10:03:27.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.855+0000 7f4617e00700 1 -- 172.21.15.78:0/1122599884 learned_addr learned my addr 172.21.15.78:0/1122599884 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:27.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.855+0000 7f461d000700 1 --2- 172.21.15.78:0/1122599884 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4618100e20 0x7f46180730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.855+0000 7f461da00700 1 --2- 172.21.15.78:0/1122599884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4618073b30 0x7f46181a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.856+0000 7f4617e00700 1 -- 172.21.15.78:0/1122599884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4618073b30 msgr2=0x7f46181a5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:27.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.856+0000 7f4617e00700 1 --2- 172.21.15.78:0/1122599884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4618073b30 0x7f46181a5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.856+0000 7f4617e00700 1 -- 172.21.15.78:0/1122599884 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4618100e20 msgr2=0x7f46180730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:27.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.856+0000 7f4617e00700 1 --2- 172.21.15.78:0/1122599884 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4618100e20 0x7f46180730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:27.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.856+0000 7f4617e00700 1 -- 172.21.15.78:0/1122599884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4610009d00 con 0x7f4618105f40 2024-03-14T10:03:27.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.856+0000 7f4617e00700 1 --2- 172.21.15.78:0/1122599884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4618105f40 0x7f46180735f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f460400e970 tx=0x7f460400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:27.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.856+0000 7f4615600700 1 -- 172.21.15.78:0/1122599884 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4604009790 con 0x7f4618105f40 2024-03-14T10:03:27.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.856+0000 7f4615600700 1 -- 172.21.15.78:0/1122599884 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f460400c7e0 con 0x7f4618105f40 2024-03-14T10:03:27.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.856+0000 7f461fa00700 1 -- 172.21.15.78:0/1122599884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46181a6480 con 0x7f4618105f40 2024-03-14T10:03:27.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.856+0000 7f4615600700 1 -- 172.21.15.78:0/1122599884 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46040054e0 con 0x7f4618105f40 2024-03-14T10:03:27.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.856+0000 7f461fa00700 1 -- 172.21.15.78:0/1122599884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46181a6990 con 0x7f4618105f40 2024-03-14T10:03:27.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.857+0000 7f461fa00700 1 -- 172.21.15.78:0/1122599884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4618109210 con 0x7f4618105f40 2024-03-14T10:03:27.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.859+0000 7f4615600700 1 -- 172.21.15.78:0/1122599884 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f460400c950 con 0x7f4618105f40 2024-03-14T10:03:27.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.860+0000 7f4615600700 1 --2- 172.21.15.78:0/1122599884 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f46000a0c00 0x7f46000a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:27.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.860+0000 7f4615600700 1 -- 172.21.15.78:0/1122599884 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f4604014070 con 0x7f4618105f40 2024-03-14T10:03:27.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.861+0000 7f461d000700 1 --2- 172.21.15.78:0/1122599884 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f46000a0c00 0x7f46000a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:27.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.861+0000 7f461d000700 1 --2- 172.21.15.78:0/1122599884 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f46000a0c00 0x7f46000a30c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f460c009c60 tx=0x7f460c009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:27.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:27.865+0000 7f4615600700 1 -- 172.21.15.78:0/1122599884 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4604095c90 con 0x7f4618105f40 2024-03-14T10:03:28.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.118+0000 7f461fa00700 1 -- 172.21.15.78:0/1122599884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4618074540 con 0x7f4618105f40 2024-03-14T10:03:28.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.118+0000 7f4615600700 1 -- 172.21.15.78:0/1122599884 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f46040992b0 con 0x7f4618105f40 2024-03-14T10:03:28.119 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T10:03:28.119 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T10:03:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.121+0000 7f461fa00700 1 -- 172.21.15.78:0/1122599884 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f46000a0c00 msgr2=0x7f46000a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.121+0000 7f461fa00700 1 --2- 172.21.15.78:0/1122599884 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f46000a0c00 0x7f46000a30c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f460c009c60 tx=0x7f460c009260 comp rx=0 tx=0).stop 2024-03-14T10:03:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.121+0000 7f461fa00700 1 -- 172.21.15.78:0/1122599884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4618105f40 msgr2=0x7f46180735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.121+0000 7f461fa00700 1 --2- 172.21.15.78:0/1122599884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4618105f40 0x7f46180735f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f460400e970 tx=0x7f460400ec80 comp rx=0 tx=0).stop 2024-03-14T10:03:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.121+0000 7f461fa00700 1 -- 172.21.15.78:0/1122599884 shutdown_connections 2024-03-14T10:03:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.121+0000 7f461fa00700 1 --2- 172.21.15.78:0/1122599884 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f46000a0c00 0x7f46000a30c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.121+0000 7f461fa00700 1 --2- 172.21.15.78:0/1122599884 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4618100e20 0x7f46180730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.121+0000 7f461fa00700 1 --2- 172.21.15.78:0/1122599884 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4618073b30 0x7f46181a5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.121+0000 7f461fa00700 1 --2- 172.21.15.78:0/1122599884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4618105f40 0x7f46180735f0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.121+0000 7f461fa00700 1 -- 172.21.15.78:0/1122599884 >> 172.21.15.78:0/1122599884 conn(0x7f46180fc700 msgr2=0x7f46180fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.121+0000 7f461fa00700 1 -- 172.21.15.78:0/1122599884 shutdown_connections 2024-03-14T10:03:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.121+0000 7f461fa00700 1 -- 172.21.15.78:0/1122599884 wait complete. 2024-03-14T10:03:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T10:03:28.141 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T10:03:28.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.336+0000 7fa74f600700 1 -- 172.21.15.78:0/1960842429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa748106050 msgr2=0x7fa74810dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.336+0000 7fa74f600700 1 --2- 172.21.15.78:0/1960842429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa748106050 0x7fa74810dbd0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa74000b0a0 tx=0x7fa74000b3b0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.336+0000 7fa74f600700 1 -- 172.21.15.78:0/1960842429 shutdown_connections 2024-03-14T10:03:28.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.336+0000 7fa74f600700 1 --2- 172.21.15.78:0/1960842429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa748100f30 0x7fa748101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.336+0000 7fa74f600700 1 --2- 172.21.15.78:0/1960842429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa748106050 0x7fa74810dbd0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.336+0000 7fa74f600700 1 --2- 172.21.15.78:0/1960842429 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa7481018e0 0x7fa748105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.336+0000 7fa74f600700 1 -- 172.21.15.78:0/1960842429 >> 172.21.15.78:0/1960842429 conn(0x7fa7480fc7b0 msgr2=0x7fa7480febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.336+0000 7fa74f600700 1 -- 172.21.15.78:0/1960842429 shutdown_connections 2024-03-14T10:03:28.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.336+0000 7fa74f600700 1 -- 172.21.15.78:0/1960842429 wait complete. 2024-03-14T10:03:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.337+0000 7fa74f600700 1 Processor -- start 2024-03-14T10:03:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.337+0000 7fa74f600700 1 -- start start 2024-03-14T10:03:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.337+0000 7fa74f600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa748100f30 0x7fa748197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.337+0000 7fa74f600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa748106050 0x7fa748197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.337+0000 7fa74f600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa74819bff0 0x7fa74819c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.337+0000 7fa74f600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74819ca70 con 0x7fa748106050 2024-03-14T10:03:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.337+0000 7fa74f600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74819cbe0 con 0x7fa748100f30 2024-03-14T10:03:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.337+0000 7fa74f600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74819cda0 con 0x7fa74819bff0 2024-03-14T10:03:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.338+0000 7fa747e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa748106050 0x7fa748197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.338+0000 7fa74cc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa748100f30 0x7fa748197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.338+0000 7fa747e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa748106050 0x7fa748197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37374/0 (socket says 172.21.15.78:37374) 2024-03-14T10:03:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.338+0000 7fa747e00700 1 -- 172.21.15.78:0/1631576601 learned_addr learned my addr 172.21.15.78:0/1631576601 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.338+0000 7fa74d600700 1 --2- 172.21.15.78:0/1631576601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa74819bff0 0x7fa74819c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.338+0000 7fa747e00700 1 -- 172.21.15.78:0/1631576601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa74819bff0 msgr2=0x7fa74819c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.338+0000 7fa747e00700 1 --2- 172.21.15.78:0/1631576601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa74819bff0 0x7fa74819c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.338+0000 7fa747e00700 1 -- 172.21.15.78:0/1631576601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa748100f30 msgr2=0x7fa748197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.338+0000 7fa747e00700 1 --2- 172.21.15.78:0/1631576601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa748100f30 0x7fa748197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.338+0000 7fa747e00700 1 -- 172.21.15.78:0/1631576601 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa740009d00 con 0x7fa748106050 2024-03-14T10:03:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.338+0000 7fa74cc00700 1 --2- 172.21.15.78:0/1631576601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa748100f30 0x7fa748197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:03:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.338+0000 7fa747e00700 1 --2- 172.21.15.78:0/1631576601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa748106050 0x7fa748197c50 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa73400e970 tx=0x7fa73400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.339+0000 7fa745600700 1 -- 172.21.15.78:0/1631576601 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa734009790 con 0x7fa748106050 2024-03-14T10:03:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.339+0000 7fa745600700 1 -- 172.21.15.78:0/1631576601 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa73400c7e0 con 0x7fa748106050 2024-03-14T10:03:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.339+0000 7fa74f600700 1 -- 172.21.15.78:0/1631576601 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7481a4300 con 0x7fa748106050 2024-03-14T10:03:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.339+0000 7fa745600700 1 -- 172.21.15.78:0/1631576601 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7340054e0 con 0x7fa748106050 2024-03-14T10:03:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.339+0000 7fa74f600700 1 -- 172.21.15.78:0/1631576601 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7481a47e0 con 0x7fa748106050 2024-03-14T10:03:28.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.342+0000 7fa745600700 1 -- 172.21.15.78:0/1631576601 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fa73400c950 con 0x7fa748106050 2024-03-14T10:03:28.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.343+0000 7fa745600700 1 --2- 172.21.15.78:0/1631576601 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa7300a0c00 0x7fa7300a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.343+0000 7fa76320cb80 1 Processor -- start 2024-03-14T10:03:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.343+0000 7fa745600700 1 -- 172.21.15.78:0/1631576601 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fa734014070 con 0x7fa748106050 2024-03-14T10:03:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.343+0000 7fa76320cb80 1 -- start start 2024-03-14T10:03:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.344+0000 7fa74cc00700 1 --2- 172.21.15.78:0/1631576601 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa7300a0c00 0x7fa7300a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.344+0000 7fa76320cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59680 0x562cc8a59a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.344+0000 7fa76320cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a59fa0 0x562cc8a67d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.344+0000 7fa76320cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a682d0 0x562cc8a6a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.344+0000 7fa76320cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cc89b2990 con 0x562cc8a59680 2024-03-14T10:03:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.344+0000 7fa76320cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cc89b0350 con 0x562cc8a59fa0 2024-03-14T10:03:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.344+0000 7fa76320cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cc89abc40 con 0x562cc8a682d0 2024-03-14T10:03:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.344+0000 7fa74cc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59680 0x562cc8a59a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.344+0000 7fa74cc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59680 0x562cc8a59a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37382/0 (socket says 172.21.15.78:37382) 2024-03-14T10:03:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.344+0000 7fa74cc00700 1 -- 172.21.15.78:0/165641264 learned_addr learned my addr 172.21.15.78:0/165641264 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.344+0000 7fa747e00700 1 --2- 172.21.15.78:0/165641264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a59fa0 0x562cc8a67d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.344+0000 7fa74d600700 1 --2- 172.21.15.78:0/165641264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a682d0 0x562cc8a6a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.345+0000 7fa74cc00700 1 -- 172.21.15.78:0/165641264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a682d0 msgr2=0x562cc8a6a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.345+0000 7fa74cc00700 1 --2- 172.21.15.78:0/165641264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a682d0 0x562cc8a6a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.345+0000 7fa74cc00700 1 -- 172.21.15.78:0/165641264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a59fa0 msgr2=0x562cc8a67d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.345+0000 7fa74cc00700 1 --2- 172.21.15.78:0/165641264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a59fa0 0x562cc8a67d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.345+0000 7fa74cc00700 1 -- 172.21.15.78:0/165641264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562cc882d3b0 con 0x562cc8a59680 2024-03-14T10:03:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.345+0000 7fa747e00700 1 --2- 172.21.15.78:0/165641264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a59fa0 0x562cc8a67d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:03:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.345+0000 7fa74cc00700 1 --2- 172.21.15.78:0/1631576601 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa7300a0c00 0x7fa7300a30c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0139a0 tx=0x7fa73c013d60 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.345+0000 7fa74cc00700 1 --2- 172.21.15.78:0/165641264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59680 0x562cc8a59a60 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009fd0 tx=0x7fa73c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336c86f4666ed839 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.345+0000 7fa739c00700 1 -- 172.21.15.78:0/165641264 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c00fe10 con 0x562cc8a59680 2024-03-14T10:03:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.345+0000 7fa739c00700 1 -- 172.21.15.78:0/165641264 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa73c0044f0 con 0x562cc8a59680 2024-03-14T10:03:28.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.345+0000 7fa739c00700 1 -- 172.21.15.78:0/165641264 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c004d10 con 0x562cc8a59680 2024-03-14T10:03:28.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.346+0000 7fa76320cb80 1 -- 172.21.15.78:0/165641264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59680 msgr2=0x562cc8a59a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.346+0000 7fa76320cb80 1 --2- 172.21.15.78:0/165641264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59680 0x562cc8a59a60 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009fd0 tx=0x7fa73c020000 comp rx=0 tx=0).stop 2024-03-14T10:03:28.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.346+0000 7fa76320cb80 1 -- 172.21.15.78:0/165641264 shutdown_connections 2024-03-14T10:03:28.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.346+0000 7fa76320cb80 1 --2- 172.21.15.78:0/165641264 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a682d0 0x562cc8a6a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.346+0000 7fa76320cb80 1 --2- 172.21.15.78:0/165641264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59680 0x562cc8a59a60 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.346+0000 7fa76320cb80 1 --2- 172.21.15.78:0/165641264 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a59fa0 0x562cc8a67d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.346+0000 7fa76320cb80 1 -- 172.21.15.78:0/165641264 >> 172.21.15.78:0/165641264 conn(0x562cc89cf220 msgr2=0x562cc8a54aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.346+0000 7fa76320cb80 1 -- 172.21.15.78:0/165641264 shutdown_connections 2024-03-14T10:03:28.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.346+0000 7fa76320cb80 1 -- 172.21.15.78:0/165641264 wait complete. 2024-03-14T10:03:28.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.346+0000 7fa76320cb80 1 -- start start 2024-03-14T10:03:28.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.347+0000 7fa76320cb80 1 Processor -- start 2024-03-14T10:03:28.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.347+0000 7fa76320cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a59680 0x562cc8a67c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.347+0000 7fa76320cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59fa0 0x562cc8a64f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.347+0000 7fa76320cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a682d0 0x562cc8a65600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.347+0000 7fa76320cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cc8947d80 con 0x562cc8a59fa0 2024-03-14T10:03:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.347+0000 7fa76320cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cc8947a20 con 0x562cc8a682d0 2024-03-14T10:03:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.347+0000 7fa76320cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cc882f6a0 con 0x562cc8a59680 2024-03-14T10:03:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.347+0000 7fa76320cb80 10 client.0 _ll_register_callbacks cb 0x562cc87b33a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:03:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.347+0000 7fa747e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59fa0 0x562cc8a64f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.347+0000 7fa747e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59fa0 0x562cc8a64f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37396/0 (socket says 172.21.15.78:37396) 2024-03-14T10:03:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.347+0000 7fa747e00700 1 -- 172.21.15.78:0/1880091764 learned_addr learned my addr 172.21.15.78:0/1880091764 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.348+0000 7fa74cc00700 1 --2- 172.21.15.78:0/1880091764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a59680 0x562cc8a67c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.348+0000 7fa74d600700 1 --2- 172.21.15.78:0/1880091764 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a682d0 0x562cc8a65600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.348+0000 7fa747e00700 1 -- 172.21.15.78:0/1880091764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a59680 msgr2=0x562cc8a67c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.348+0000 7fa747e00700 1 --2- 172.21.15.78:0/1880091764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a59680 0x562cc8a67c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.348+0000 7fa747e00700 1 -- 172.21.15.78:0/1880091764 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a682d0 msgr2=0x562cc8a65600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.348+0000 7fa747e00700 1 --2- 172.21.15.78:0/1880091764 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a682d0 0x562cc8a65600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.348+0000 7fa747e00700 1 -- 172.21.15.78:0/1880091764 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7481a47e0 con 0x562cc8a59fa0 2024-03-14T10:03:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.348+0000 7fa74cc00700 1 --2- 172.21.15.78:0/1880091764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a59680 0x562cc8a67c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:03:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.348+0000 7fa747e00700 1 --2- 172.21.15.78:0/1880091764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59fa0 0x562cc8a64f80 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa7340c9da0 tx=0x7fa734099940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdaf6514156f8530 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.348+0000 7fa72e000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:03:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.348+0000 7fa72e000700 1 -- 172.21.15.78:0/1880091764 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa734099ba0 con 0x562cc8a59fa0 2024-03-14T10:03:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.348+0000 7fa72e000700 1 -- 172.21.15.78:0/1880091764 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fa734099d00 con 0x562cc8a59fa0 2024-03-14T10:03:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.348+0000 7fa76320cb80 1 -- 172.21.15.78:0/1880091764 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562cc8a4c720 con 0x562cc8a59fa0 2024-03-14T10:03:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.348+0000 7fa72e000700 1 -- 172.21.15.78:0/1880091764 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa734098e30 con 0x562cc8a59fa0 2024-03-14T10:03:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.348+0000 7fa76320cb80 10 client.17436 fetch_fsmap learned FSMap version 25 2024-03-14T10:03:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.348+0000 7fa76320cb80 1 -- 172.21.15.78:0/1880091764 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7fa73c00f8b0 con 0x562cc8a59fa0 2024-03-14T10:03:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.349+0000 7fa72e000700 1 -- 172.21.15.78:0/1880091764 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562cc8a4c720 con 0x562cc8a59fa0 2024-03-14T10:03:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.349+0000 7fa72e000700 1 -- 172.21.15.78:0/1880091764 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa7340652a0 con 0x562cc8a59fa0 2024-03-14T10:03:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.349+0000 7fa76320cb80 10 client.17436 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:03:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.349+0000 7fa76320cb80 10 client.17436 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:03:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.349+0000 7fa76320cb80 20 client.17436 populate_metadata read hostname 'smithi078' 2024-03-14T10:03:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.349+0000 7fa76320cb80 1 --2- 172.21.15.78:0/1880091764 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x562cc8a77df0 0x562cc8a7a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.349+0000 7fa76320cb80 4 client.17436 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:03:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.349+0000 7fa76320cb80 1 -- 172.21.15.78:0/1880091764 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562cc89a4700 con 0x562cc8a77df0 2024-03-14T10:03:28.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.349+0000 7fa74cc00700 1 --2- 172.21.15.78:0/1880091764 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x562cc8a77df0 0x562cc8a7a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.350+0000 7fa74cc00700 1 --2- 172.21.15.78:0/1880091764 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x562cc8a77df0 0x562cc8a7a210 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f20507e48881fff0 server_cookie=9ea67cfe0e18cb59 in_seq=0 out_seq=0 2024-03-14T10:03:28.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.350+0000 7fa72e000700 10 client.17436 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:03:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.350+0000 7fa72e000700 1 -- 172.21.15.78:0/1880091764 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x562cc89a4700 con 0x562cc8a77df0 2024-03-14T10:03:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.350+0000 7fa72e000700 10 client.17436 handle_command_reply: tid=0 2024-03-14T10:03:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.351+0000 7fa76320cb80 1 client.17436 shutdown 2024-03-14T10:03:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.351+0000 7fa76320cb80 1 -- 172.21.15.78:0/1880091764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59fa0 msgr2=0x562cc8a64f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.351+0000 7fa76320cb80 1 --2- 172.21.15.78:0/1880091764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59fa0 0x562cc8a64f80 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa7340c9da0 tx=0x7fa734099940 comp rx=0 tx=0).stop 2024-03-14T10:03:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.351+0000 7fa76320cb80 1 -- 172.21.15.78:0/1880091764 shutdown_connections 2024-03-14T10:03:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.351+0000 7fa76320cb80 1 --2- 172.21.15.78:0/1880091764 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x562cc8a77df0 0x562cc8a7a210 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.351+0000 7fa76320cb80 1 --2- 172.21.15.78:0/1880091764 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a682d0 0x562cc8a65600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.351+0000 7fa72e000700 0 client.17436 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:03:28.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.351+0000 7fa76320cb80 1 --2- 172.21.15.78:0/1880091764 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a59680 0x562cc8a67c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.351+0000 7fa76320cb80 1 --2- 172.21.15.78:0/1880091764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59fa0 0x562cc8a64f80 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.351+0000 7fa76320cb80 1 -- 172.21.15.78:0/1880091764 >> 172.21.15.78:0/1880091764 conn(0x562cc89cf220 msgr2=0x562cc8a54690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.351+0000 7fa76320cb80 1 -- 172.21.15.78:0/1880091764 shutdown_connections 2024-03-14T10:03:28.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.351+0000 7fa76320cb80 1 -- 172.21.15.78:0/1880091764 wait complete. 2024-03-14T10:03:28.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.351+0000 7fa76320cb80 20 client.17436 trim_cache size 0 max 16384 2024-03-14T10:03:28.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.364+0000 7fa76320cb80 1 Processor -- start 2024-03-14T10:03:28.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.364+0000 7fa76320cb80 1 -- start start 2024-03-14T10:03:28.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.365+0000 7fa76320cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59680 0x562cc8a6ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.365+0000 7fa76320cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a59fa0 0x562cc8a5a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.365+0000 7fa76320cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a682d0 0x562cc8a63870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.365+0000 7fa76320cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cc8a56060 con 0x562cc8a59680 2024-03-14T10:03:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.365+0000 7fa76320cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cc8a4ed00 con 0x562cc8a682d0 2024-03-14T10:03:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.365+0000 7fa76320cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cc8a6aa80 con 0x562cc8a59fa0 2024-03-14T10:03:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.365+0000 7fa74cc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59680 0x562cc8a6ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.365+0000 7fa74cc00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59680 0x562cc8a6ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37410/0 (socket says 172.21.15.78:37410) 2024-03-14T10:03:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.365+0000 7fa74cc00700 1 -- 172.21.15.78:0/2776697372 learned_addr learned my addr 172.21.15.78:0/2776697372 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.365+0000 7fa747e00700 1 --2- 172.21.15.78:0/2776697372 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a59fa0 0x562cc8a5a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.365+0000 7fa74d600700 1 --2- 172.21.15.78:0/2776697372 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a682d0 0x562cc8a63870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.365+0000 7fa74cc00700 1 -- 172.21.15.78:0/2776697372 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a59fa0 msgr2=0x562cc8a5a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.365+0000 7fa74cc00700 1 --2- 172.21.15.78:0/2776697372 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a59fa0 0x562cc8a5a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.365+0000 7fa74cc00700 1 -- 172.21.15.78:0/2776697372 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a682d0 msgr2=0x562cc8a63870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.365+0000 7fa74cc00700 1 --2- 172.21.15.78:0/2776697372 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a682d0 0x562cc8a63870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.365+0000 7fa74cc00700 1 -- 172.21.15.78:0/2776697372 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562cc87d9b60 con 0x562cc8a59680 2024-03-14T10:03:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.366+0000 7fa74cc00700 1 --2- 172.21.15.78:0/2776697372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59680 0x562cc8a6ee90 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00ffd0 tx=0x7fa73c009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d925826bf6986672 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.366+0000 7fa73a600700 1 -- 172.21.15.78:0/2776697372 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c012140 con 0x562cc8a59680 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.366+0000 7fa73a600700 1 -- 172.21.15.78:0/2776697372 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa73c0134e0 con 0x562cc8a59680 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.366+0000 7fa73a600700 1 -- 172.21.15.78:0/2776697372 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c004910 con 0x562cc8a59680 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.366+0000 7fa76320cb80 1 -- 172.21.15.78:0/2776697372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59680 msgr2=0x562cc8a6ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.366+0000 7fa76320cb80 1 --2- 172.21.15.78:0/2776697372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59680 0x562cc8a6ee90 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00ffd0 tx=0x7fa73c009e40 comp rx=0 tx=0).stop 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.366+0000 7fa76320cb80 1 -- 172.21.15.78:0/2776697372 shutdown_connections 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.366+0000 7fa76320cb80 1 --2- 172.21.15.78:0/2776697372 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a682d0 0x562cc8a63870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.366+0000 7fa76320cb80 1 --2- 172.21.15.78:0/2776697372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59680 0x562cc8a6ee90 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.366+0000 7fa76320cb80 1 --2- 172.21.15.78:0/2776697372 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a59fa0 0x562cc8a5a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.366+0000 7fa76320cb80 1 -- 172.21.15.78:0/2776697372 >> 172.21.15.78:0/2776697372 conn(0x562cc89cf220 msgr2=0x562cc8a53180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.367+0000 7fa76320cb80 1 -- 172.21.15.78:0/2776697372 shutdown_connections 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.367+0000 7fa76320cb80 1 -- 172.21.15.78:0/2776697372 wait complete. 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.367+0000 7fa76320cb80 1 -- start start 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.367+0000 7fa76320cb80 1 Processor -- start 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.367+0000 7fa76320cb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a59680 0x562cc8a6ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa76320cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59fa0 0x562cc8a6d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa76320cb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a682d0 0x562cc8a6d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa76320cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cc8a6db80 con 0x562cc8a59fa0 2024-03-14T10:03:28.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa76320cb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cc8a6dcf0 con 0x562cc8a59680 2024-03-14T10:03:28.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa76320cb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x562cc8a6deb0 con 0x562cc8a682d0 2024-03-14T10:03:28.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa76320cb80 10 client.0 _ll_register_callbacks cb 0x7fa73400cc50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:03:28.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa747e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59fa0 0x562cc8a6d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa747e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59fa0 0x562cc8a6d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37416/0 (socket says 172.21.15.78:37416) 2024-03-14T10:03:28.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa74d600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a682d0 0x562cc8a6d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa747e00700 1 -- 172.21.15.78:0/1926188455 learned_addr learned my addr 172.21.15.78:0/1926188455 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:28.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa74cc00700 1 --2- 172.21.15.78:0/1926188455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a59680 0x562cc8a6ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa747e00700 1 -- 172.21.15.78:0/1926188455 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a682d0 msgr2=0x562cc8a6d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa747e00700 1 --2- 172.21.15.78:0/1926188455 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a682d0 0x562cc8a6d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa747e00700 1 -- 172.21.15.78:0/1926188455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a59680 msgr2=0x562cc8a6ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa747e00700 1 --2- 172.21.15.78:0/1926188455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a59680 0x562cc8a6ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa747e00700 1 -- 172.21.15.78:0/1926188455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562cc882d3b0 con 0x562cc8a59fa0 2024-03-14T10:03:28.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa747e00700 1 --2- 172.21.15.78:0/1926188455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59fa0 0x562cc8a6d0c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fa73400cc00 tx=0x7fa734099840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8433814e94e71c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.368+0000 7fa72e000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:03:28.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.369+0000 7fa72e000700 1 -- 172.21.15.78:0/1926188455 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa734099030 con 0x562cc8a59fa0 2024-03-14T10:03:28.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.369+0000 7fa72e000700 1 -- 172.21.15.78:0/1926188455 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fa7340a1930 con 0x562cc8a59fa0 2024-03-14T10:03:28.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.369+0000 7fa76320cb80 1 -- 172.21.15.78:0/1926188455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa73c0044f0 con 0x562cc8a59fa0 2024-03-14T10:03:28.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.369+0000 7fa72e000700 1 -- 172.21.15.78:0/1926188455 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7340a1af0 con 0x562cc8a59fa0 2024-03-14T10:03:28.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.369+0000 7fa76320cb80 10 client.17442 fetch_fsmap learned FSMap version 25 2024-03-14T10:03:28.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.369+0000 7fa76320cb80 1 -- 172.21.15.78:0/1926188455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x562cc8971ea0 con 0x562cc8a59fa0 2024-03-14T10:03:28.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.369+0000 7fa72e000700 1 -- 172.21.15.78:0/1926188455 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa73c0044f0 con 0x562cc8a59fa0 2024-03-14T10:03:28.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.369+0000 7fa72e000700 1 -- 172.21.15.78:0/1926188455 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa734004a30 con 0x562cc8a59fa0 2024-03-14T10:03:28.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.369+0000 7fa76320cb80 10 client.17442 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:03:28.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.369+0000 7fa76320cb80 10 client.17442 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:03:28.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.369+0000 7fa76320cb80 20 client.17442 populate_metadata read hostname 'smithi078' 2024-03-14T10:03:28.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.369+0000 7fa76320cb80 1 --2- 172.21.15.78:0/1926188455 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x562cc8acac10 0x562cc8acd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.369+0000 7fa76320cb80 4 client.17442 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:03:28.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.369+0000 7fa76320cb80 1 -- 172.21.15.78:0/1926188455 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562cc87facf0 con 0x562cc8acac10 2024-03-14T10:03:28.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.370+0000 7fa74cc00700 1 --2- 172.21.15.78:0/1926188455 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x562cc8acac10 0x562cc8acd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.370+0000 7fa74cc00700 1 --2- 172.21.15.78:0/1926188455 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x562cc8acac10 0x562cc8acd030 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3093f7b8c289fb9b server_cookie=2b6bd4b6bb059017 in_seq=0 out_seq=0 2024-03-14T10:03:28.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.370+0000 7fa72e000700 10 client.17442 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:03:28.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.370+0000 7fa72e000700 1 -- 172.21.15.78:0/1926188455 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562cc87facf0 con 0x562cc8acac10 2024-03-14T10:03:28.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.370+0000 7fa72e000700 10 client.17442 handle_command_reply: tid=0 2024-03-14T10:03:28.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.370+0000 7fa76320cb80 1 client.17442 shutdown 2024-03-14T10:03:28.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.371+0000 7fa76320cb80 1 -- 172.21.15.78:0/1926188455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59fa0 msgr2=0x562cc8a6d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.371+0000 7fa76320cb80 1 --2- 172.21.15.78:0/1926188455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59fa0 0x562cc8a6d0c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fa73400cc00 tx=0x7fa734099840 comp rx=0 tx=0).stop 2024-03-14T10:03:28.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.371+0000 7fa76320cb80 1 -- 172.21.15.78:0/1926188455 shutdown_connections 2024-03-14T10:03:28.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.371+0000 7fa76320cb80 1 --2- 172.21.15.78:0/1926188455 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x562cc8acac10 0x562cc8acd030 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.371+0000 7fa76320cb80 1 --2- 172.21.15.78:0/1926188455 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x562cc8a59680 0x562cc8a6ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.371+0000 7fa76320cb80 1 --2- 172.21.15.78:0/1926188455 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x562cc8a682d0 0x562cc8a6d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.371+0000 7fa76320cb80 1 --2- 172.21.15.78:0/1926188455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562cc8a59fa0 0x562cc8a6d0c0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.371+0000 7fa76320cb80 1 -- 172.21.15.78:0/1926188455 >> 172.21.15.78:0/1926188455 conn(0x562cc89cf220 msgr2=0x562cc8a636c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.371+0000 7fa72e000700 0 client.17442 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:03:28.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.371+0000 7fa76320cb80 1 -- 172.21.15.78:0/1926188455 shutdown_connections 2024-03-14T10:03:28.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.371+0000 7fa76320cb80 1 -- 172.21.15.78:0/1926188455 wait complete. 2024-03-14T10:03:28.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.371+0000 7fa76320cb80 20 client.17442 trim_cache size 0 max 16384 2024-03-14T10:03:28.377 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:03:28.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.371+0000 7fa74f600700 1 -- 172.21.15.78:0/1631576601 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa7300a0c00 msgr2=0x7fa7300a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.371+0000 7fa74f600700 1 --2- 172.21.15.78:0/1631576601 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa7300a0c00 0x7fa7300a30c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0139a0 tx=0x7fa73c013d60 comp rx=0 tx=0).stop 2024-03-14T10:03:28.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.371+0000 7fa74f600700 1 -- 172.21.15.78:0/1631576601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa748106050 msgr2=0x7fa748197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.371+0000 7fa74f600700 1 --2- 172.21.15.78:0/1631576601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa748106050 0x7fa748197c50 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa73400e970 tx=0x7fa73400ec80 comp rx=0 tx=0).stop 2024-03-14T10:03:28.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.372+0000 7fa74f600700 1 -- 172.21.15.78:0/1631576601 shutdown_connections 2024-03-14T10:03:28.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.372+0000 7fa74f600700 1 --2- 172.21.15.78:0/1631576601 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fa7300a0c00 0x7fa7300a30c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.372+0000 7fa74f600700 1 --2- 172.21.15.78:0/1631576601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa748100f30 0x7fa748197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.372+0000 7fa74f600700 1 --2- 172.21.15.78:0/1631576601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fa74819bff0 0x7fa74819c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.372+0000 7fa74f600700 1 --2- 172.21.15.78:0/1631576601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa748106050 0x7fa748197c50 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.372+0000 7fa74f600700 1 -- 172.21.15.78:0/1631576601 >> 172.21.15.78:0/1631576601 conn(0x7fa7480fc7b0 msgr2=0x7fa7480feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.372+0000 7fa74f600700 1 -- 172.21.15.78:0/1631576601 shutdown_connections 2024-03-14T10:03:28.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.372+0000 7fa74f600700 1 -- 172.21.15.78:0/1631576601 wait complete. 2024-03-14T10:03:28.390 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T10:03:28.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.584+0000 7f5105e00700 1 -- 172.21.15.78:0/2692761365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5100100f20 msgr2=0x7f5100101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.584+0000 7f5105e00700 1 --2- 172.21.15.78:0/2692761365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5100100f20 0x7f5100101300 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009990 tx=0x7f50f0009ca0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.585+0000 7f5105e00700 1 -- 172.21.15.78:0/2692761365 shutdown_connections 2024-03-14T10:03:28.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.585+0000 7f5105e00700 1 --2- 172.21.15.78:0/2692761365 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5100106040 0x7f510010dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.585+0000 7f5105e00700 1 --2- 172.21.15.78:0/2692761365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5100100f20 0x7f5100101300 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.585+0000 7f5105e00700 1 --2- 172.21.15.78:0/2692761365 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f51001018d0 0x7f5100105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.585+0000 7f5105e00700 1 -- 172.21.15.78:0/2692761365 >> 172.21.15.78:0/2692761365 conn(0x7f51000fc7a0 msgr2=0x7f51000febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.585+0000 7f5105e00700 1 -- 172.21.15.78:0/2692761365 shutdown_connections 2024-03-14T10:03:28.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.585+0000 7f5105e00700 1 -- 172.21.15.78:0/2692761365 wait complete. 2024-03-14T10:03:28.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.586+0000 7f5105e00700 1 Processor -- start 2024-03-14T10:03:28.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.586+0000 7f5105e00700 1 -- start start 2024-03-14T10:03:28.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.586+0000 7f5105e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5100100f20 0x7f510019f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.586+0000 7f5105e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5100106040 0x7f510019faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.586+0000 7f5105e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51001996e0 0x7f5100199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.586+0000 7f5105e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510019a160 con 0x7f51001996e0 2024-03-14T10:03:28.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.586+0000 7f5105e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510019a2a0 con 0x7f5100100f20 2024-03-14T10:03:28.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.586+0000 7f5105e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510019a460 con 0x7f5100106040 2024-03-14T10:03:28.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.586+0000 7f50f7e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5100106040 0x7f510019faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.586+0000 7f50ff400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51001996e0 0x7f5100199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.586+0000 7f50f7e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5100106040 0x7f510019faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:52056/0 (socket says 172.21.15.78:52056) 2024-03-14T10:03:28.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.586+0000 7f50f7e00700 1 -- 172.21.15.78:0/1659449063 learned_addr learned my addr 172.21.15.78:0/1659449063 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:28.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.587+0000 7f50fea00700 1 --2- 172.21.15.78:0/1659449063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5100100f20 0x7f510019f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.587+0000 7f50ff400700 1 -- 172.21.15.78:0/1659449063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5100106040 msgr2=0x7f510019faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.587+0000 7f50ff400700 1 --2- 172.21.15.78:0/1659449063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5100106040 0x7f510019faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.587+0000 7f50ff400700 1 -- 172.21.15.78:0/1659449063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5100100f20 msgr2=0x7f510019f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.587+0000 7f50ff400700 1 --2- 172.21.15.78:0/1659449063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5100100f20 0x7f510019f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.587+0000 7f50ff400700 1 -- 172.21.15.78:0/1659449063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f00095f0 con 0x7f51001996e0 2024-03-14T10:03:28.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.587+0000 7f50ff400700 1 --2- 172.21.15.78:0/1659449063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51001996e0 0x7f5100199b90 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f50f800efc0 tx=0x7f50f800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.587+0000 7f5104a00700 1 -- 172.21.15.78:0/1659449063 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f8018070 con 0x7f51001996e0 2024-03-14T10:03:28.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.587+0000 7f5104a00700 1 -- 172.21.15.78:0/1659449063 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50f800f040 con 0x7f51001996e0 2024-03-14T10:03:28.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.587+0000 7f5105e00700 1 -- 172.21.15.78:0/1659449063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51001a6420 con 0x7f51001996e0 2024-03-14T10:03:28.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.587+0000 7f5104a00700 1 -- 172.21.15.78:0/1659449063 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f800cd40 con 0x7f51001996e0 2024-03-14T10:03:28.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.588+0000 7f5105e00700 1 -- 172.21.15.78:0/1659449063 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51001a6930 con 0x7f51001996e0 2024-03-14T10:03:28.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.590+0000 7f5104a00700 1 -- 172.21.15.78:0/1659449063 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f50f8003a00 con 0x7f51001996e0 2024-03-14T10:03:28.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.591+0000 7f51199e9b80 1 Processor -- start 2024-03-14T10:03:28.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.592+0000 7f5104a00700 1 --2- 172.21.15.78:0/1659449063 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f50e40a0c00 0x7f50e40a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.592+0000 7f51199e9b80 1 -- start start 2024-03-14T10:03:28.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.592+0000 7f5104a00700 1 -- 172.21.15.78:0/1659449063 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f50f80cc350 con 0x7f51001996e0 2024-03-14T10:03:28.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.592+0000 7f51199e9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5680 0x560f678e5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.592+0000 7f50f7e00700 1 --2- 172.21.15.78:0/1659449063 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f50e40a0c00 0x7f50e40a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.592+0000 7f51199e9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5fa0 0x560f678ef980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.592+0000 7f51199e9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 0x560f678f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.592+0000 7f51199e9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f6783e990 con 0x560f678e5680 2024-03-14T10:03:28.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.592+0000 7f51199e9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f6783c350 con 0x560f678e5fa0 2024-03-14T10:03:28.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.592+0000 7f51199e9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f67837c40 con 0x560f678efec0 2024-03-14T10:03:28.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.592+0000 7f50fea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5680 0x560f678e5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.592+0000 7f50fea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5680 0x560f678e5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37454/0 (socket says 172.21.15.78:37454) 2024-03-14T10:03:28.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.592+0000 7f50ff400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 0x560f678f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.592+0000 7f50fea00700 1 -- 172.21.15.78:0/417954279 learned_addr learned my addr 172.21.15.78:0/417954279 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:28.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.592+0000 7f50f7e00700 1 --2- 172.21.15.78:0/417954279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5fa0 0x560f678ef980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.593+0000 7f50fea00700 1 -- 172.21.15.78:0/417954279 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 msgr2=0x560f678f22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.593+0000 7f50fea00700 1 --2- 172.21.15.78:0/417954279 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 0x560f678f22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.593+0000 7f50fea00700 1 -- 172.21.15.78:0/417954279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5fa0 msgr2=0x560f678ef980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.593+0000 7f50fea00700 1 --2- 172.21.15.78:0/417954279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5fa0 0x560f678ef980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.593+0000 7f50fea00700 1 -- 172.21.15.78:0/417954279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f676b93b0 con 0x560f678e5680 2024-03-14T10:03:28.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.593+0000 7f50f7e00700 1 --2- 172.21.15.78:0/417954279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5fa0 0x560f678ef980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:03:28.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.593+0000 7f50f7e00700 1 --2- 172.21.15.78:0/1659449063 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f50e40a0c00 0x7f50e40a30c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f51000fcea0 tx=0x7f50e8012970 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.593+0000 7f50fea00700 1 --2- 172.21.15.78:0/417954279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5680 0x560f678e5a60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f50f000b5b0 tx=0x7f50f000b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689db806f2c47a94 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.593+0000 7f50efe00700 1 -- 172.21.15.78:0/417954279 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f001d070 con 0x560f678e5680 2024-03-14T10:03:28.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.593+0000 7f50efe00700 1 -- 172.21.15.78:0/417954279 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50f001e040 con 0x560f678e5680 2024-03-14T10:03:28.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.593+0000 7f50efe00700 1 -- 172.21.15.78:0/417954279 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f00047d0 con 0x560f678e5680 2024-03-14T10:03:28.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.594+0000 7f51199e9b80 1 -- 172.21.15.78:0/417954279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5680 msgr2=0x560f678e5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.594+0000 7f51199e9b80 1 --2- 172.21.15.78:0/417954279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5680 0x560f678e5a60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f50f000b5b0 tx=0x7f50f000b5e0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.594+0000 7f51199e9b80 1 -- 172.21.15.78:0/417954279 shutdown_connections 2024-03-14T10:03:28.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.594+0000 7f51199e9b80 1 --2- 172.21.15.78:0/417954279 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 0x560f678f22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.594+0000 7f51199e9b80 1 --2- 172.21.15.78:0/417954279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5680 0x560f678e5a60 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.594+0000 7f51199e9b80 1 --2- 172.21.15.78:0/417954279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5fa0 0x560f678ef980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.594+0000 7f51199e9b80 1 -- 172.21.15.78:0/417954279 >> 172.21.15.78:0/417954279 conn(0x560f6785b220 msgr2=0x560f678e0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.594+0000 7f51199e9b80 1 -- 172.21.15.78:0/417954279 shutdown_connections 2024-03-14T10:03:28.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.594+0000 7f51199e9b80 1 -- 172.21.15.78:0/417954279 wait complete. 2024-03-14T10:03:28.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.595+0000 7f51199e9b80 1 -- start start 2024-03-14T10:03:28.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.595+0000 7f51199e9b80 1 Processor -- start 2024-03-14T10:03:28.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f51199e9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5680 0x560f678ef7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f51199e9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678e5fa0 0x560f678ecad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f51199e9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678efec0 0x560f678ed070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f51199e9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f677d3d80 con 0x560f678e5680 2024-03-14T10:03:28.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f51199e9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f677d3a20 con 0x560f678efec0 2024-03-14T10:03:28.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f51199e9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f676bb6a0 con 0x560f678e5fa0 2024-03-14T10:03:28.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f51199e9b80 10 client.0 _ll_register_callbacks cb 0x560f6763f3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:03:28.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f50fea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5680 0x560f678ef7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f50fea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5680 0x560f678ef7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37466/0 (socket says 172.21.15.78:37466) 2024-03-14T10:03:28.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f50fea00700 1 -- 172.21.15.78:0/2963037477 learned_addr learned my addr 172.21.15.78:0/2963037477 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:28.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f50ff400700 1 --2- 172.21.15.78:0/2963037477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678efec0 0x560f678ed070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f50f7e00700 1 --2- 172.21.15.78:0/2963037477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678e5fa0 0x560f678ecad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f50fea00700 1 -- 172.21.15.78:0/2963037477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678e5fa0 msgr2=0x560f678ecad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f50fea00700 1 --2- 172.21.15.78:0/2963037477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678e5fa0 0x560f678ecad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f50fea00700 1 -- 172.21.15.78:0/2963037477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678efec0 msgr2=0x560f678ed070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f50fea00700 1 --2- 172.21.15.78:0/2963037477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678efec0 0x560f678ed070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f50fea00700 1 -- 172.21.15.78:0/2963037477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f510010e430 con 0x560f678e5680 2024-03-14T10:03:28.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f50f7e00700 1 --2- 172.21.15.78:0/2963037477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678e5fa0 0x560f678ecad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:03:28.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.596+0000 7f50ff400700 1 --2- 172.21.15.78:0/2963037477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678efec0 0x560f678ed070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:03:28.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.597+0000 7f50fea00700 1 --2- 172.21.15.78:0/2963037477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5680 0x560f678ef7e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f50f000be80 tx=0x7f50f000b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=707998e6bc65e1be server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.597+0000 7f50ecc00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:03:28.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.597+0000 7f50ecc00700 1 -- 172.21.15.78:0/2963037477 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f001d030 con 0x560f678e5680 2024-03-14T10:03:28.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.597+0000 7f51199e9b80 1 -- 172.21.15.78:0/2963037477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560f678d8720 con 0x560f678e5680 2024-03-14T10:03:28.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.597+0000 7f50ecc00700 1 -- 172.21.15.78:0/2963037477 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f50f0004910 con 0x560f678e5680 2024-03-14T10:03:28.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.597+0000 7f50ecc00700 1 -- 172.21.15.78:0/2963037477 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50f0004ad0 con 0x560f678e5680 2024-03-14T10:03:28.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.597+0000 7f51199e9b80 10 client.17454 fetch_fsmap learned FSMap version 25 2024-03-14T10:03:28.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.597+0000 7f51199e9b80 1 -- 172.21.15.78:0/2963037477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560f67665b60 con 0x560f678e5680 2024-03-14T10:03:28.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.597+0000 7f50ecc00700 1 -- 172.21.15.78:0/2963037477 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560f678d8720 con 0x560f678e5680 2024-03-14T10:03:28.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.598+0000 7f50ecc00700 1 -- 172.21.15.78:0/2963037477 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f50f000fac0 con 0x560f678e5680 2024-03-14T10:03:28.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.598+0000 7f51199e9b80 10 client.17454 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:03:28.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.598+0000 7f51199e9b80 10 client.17454 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:03:28.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.598+0000 7f51199e9b80 20 client.17454 populate_metadata read hostname 'smithi078' 2024-03-14T10:03:28.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.598+0000 7f51199e9b80 1 --2- 172.21.15.78:0/2963037477 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x560f678ff5a0 0x560f679019c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.598+0000 7f51199e9b80 4 client.17454 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:03:28.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.598+0000 7f51199e9b80 1 -- 172.21.15.78:0/2963037477 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560f67830700 con 0x560f678ff5a0 2024-03-14T10:03:28.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.598+0000 7f50f7e00700 1 --2- 172.21.15.78:0/2963037477 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x560f678ff5a0 0x560f679019c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.599+0000 7f50ecc00700 10 client.17454 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:03:28.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.599+0000 7f50f7e00700 1 --2- 172.21.15.78:0/2963037477 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x560f678ff5a0 0x560f679019c0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ee4360abd9f5bc41 server_cookie=5a4738440a6272db in_seq=0 out_seq=0 2024-03-14T10:03:28.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.599+0000 7f50ecc00700 1 -- 172.21.15.78:0/2963037477 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x560f67830700 con 0x560f678ff5a0 2024-03-14T10:03:28.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.599+0000 7f50ecc00700 10 client.17454 handle_command_reply: tid=0 2024-03-14T10:03:28.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.599+0000 7f51199e9b80 1 client.17454 shutdown 2024-03-14T10:03:28.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.599+0000 7f51199e9b80 1 -- 172.21.15.78:0/2963037477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5680 msgr2=0x560f678ef7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.599+0000 7f51199e9b80 1 --2- 172.21.15.78:0/2963037477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5680 0x560f678ef7e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f50f000be80 tx=0x7f50f000b980 comp rx=0 tx=0).stop 2024-03-14T10:03:28.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.599+0000 7f51199e9b80 1 -- 172.21.15.78:0/2963037477 shutdown_connections 2024-03-14T10:03:28.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.599+0000 7f51199e9b80 1 --2- 172.21.15.78:0/2963037477 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x560f678ff5a0 0x560f679019c0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.599+0000 7f51199e9b80 1 --2- 172.21.15.78:0/2963037477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678efec0 0x560f678ed070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.599+0000 7f50ecc00700 0 client.17454 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:03:28.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.599+0000 7f51199e9b80 1 --2- 172.21.15.78:0/2963037477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5680 0x560f678ef7e0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.599+0000 7f51199e9b80 1 --2- 172.21.15.78:0/2963037477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678e5fa0 0x560f678ecad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.599+0000 7f51199e9b80 1 -- 172.21.15.78:0/2963037477 >> 172.21.15.78:0/2963037477 conn(0x560f6785b220 msgr2=0x560f678e0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.600+0000 7f51199e9b80 1 -- 172.21.15.78:0/2963037477 shutdown_connections 2024-03-14T10:03:28.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.600+0000 7f51199e9b80 1 -- 172.21.15.78:0/2963037477 wait complete. 2024-03-14T10:03:28.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.600+0000 7f51199e9b80 20 client.17454 trim_cache size 0 max 16384 2024-03-14T10:03:28.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.613+0000 7f51199e9b80 1 Processor -- start 2024-03-14T10:03:28.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.613+0000 7f51199e9b80 1 -- start start 2024-03-14T10:03:28.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.613+0000 7f51199e9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5680 0x560f678f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.613+0000 7f51199e9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5fa0 0x560f678ec3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.613+0000 7f51199e9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 0x560f678ec910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.613+0000 7f51199e9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f678dd230 con 0x560f678e5fa0 2024-03-14T10:03:28.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.613+0000 7f51199e9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f678ece50 con 0x560f678e5680 2024-03-14T10:03:28.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.613+0000 7f51199e9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f679066a0 con 0x560f678efec0 2024-03-14T10:03:28.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.613+0000 7f50f7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5fa0 0x560f678ec3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.613+0000 7f50ff400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 0x560f678ec910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.613+0000 7f50f7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5fa0 0x560f678ec3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37470/0 (socket says 172.21.15.78:37470) 2024-03-14T10:03:28.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.613+0000 7f50ff400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 0x560f678ec910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:52084/0 (socket says 172.21.15.78:52084) 2024-03-14T10:03:28.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.613+0000 7f50f7e00700 1 -- 172.21.15.78:0/419416670 learned_addr learned my addr 172.21.15.78:0/419416670 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:28.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.613+0000 7f50fea00700 1 --2- 172.21.15.78:0/419416670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5680 0x560f678f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.614+0000 7f50ff400700 1 -- 172.21.15.78:0/419416670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5680 msgr2=0x560f678f22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.614+0000 7f50ff400700 1 --2- 172.21.15.78:0/419416670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5680 0x560f678f22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.614+0000 7f50ff400700 1 -- 172.21.15.78:0/419416670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5fa0 msgr2=0x560f678ec3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.614+0000 7f50ff400700 1 --2- 172.21.15.78:0/419416670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5fa0 0x560f678ec3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.614+0000 7f50ff400700 1 -- 172.21.15.78:0/419416670 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f677fdea0 con 0x560f678efec0 2024-03-14T10:03:28.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.614+0000 7f50f7e00700 1 --2- 172.21.15.78:0/419416670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5fa0 0x560f678ec3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:03:28.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.614+0000 7f50ff400700 1 --2- 172.21.15.78:0/419416670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 0x560f678ec910 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f50f80c9670 tx=0x7f50f80092e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=111b789a00a7fdb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.614+0000 7f50f5200700 1 -- 172.21.15.78:0/419416670 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f8098c60 con 0x560f678efec0 2024-03-14T10:03:28.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.614+0000 7f50f5200700 1 -- 172.21.15.78:0/419416670 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50f80ae070 con 0x560f678efec0 2024-03-14T10:03:28.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.614+0000 7f50f5200700 1 -- 172.21.15.78:0/419416670 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f80a1780 con 0x560f678efec0 2024-03-14T10:03:28.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.615+0000 7f51199e9b80 1 -- 172.21.15.78:0/419416670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 msgr2=0x560f678ec910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.615+0000 7f51199e9b80 1 --2- 172.21.15.78:0/419416670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 0x560f678ec910 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f50f80c9670 tx=0x7f50f80092e0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.615+0000 7f51199e9b80 1 -- 172.21.15.78:0/419416670 shutdown_connections 2024-03-14T10:03:28.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.615+0000 7f51199e9b80 1 --2- 172.21.15.78:0/419416670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5680 0x560f678f22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.615+0000 7f51199e9b80 1 --2- 172.21.15.78:0/419416670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 0x560f678ec910 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.615+0000 7f51199e9b80 1 --2- 172.21.15.78:0/419416670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5fa0 0x560f678ec3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.615+0000 7f51199e9b80 1 -- 172.21.15.78:0/419416670 >> 172.21.15.78:0/419416670 conn(0x560f6785b220 msgr2=0x560f678e7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.615+0000 7f51199e9b80 1 -- 172.21.15.78:0/419416670 shutdown_connections 2024-03-14T10:03:28.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.615+0000 7f51199e9b80 1 -- 172.21.15.78:0/419416670 wait complete. 2024-03-14T10:03:28.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.616+0000 7f51199e9b80 1 -- start start 2024-03-14T10:03:28.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.616+0000 7f51199e9b80 1 Processor -- start 2024-03-14T10:03:28.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.616+0000 7f51199e9b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5680 0x560f678f6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f51199e9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5fa0 0x560f678f6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f51199e9b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 0x560f678f4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f51199e9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f678f51e0 con 0x560f678e5fa0 2024-03-14T10:03:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f51199e9b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f678f5350 con 0x560f678e5680 2024-03-14T10:03:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f51199e9b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f678f5510 con 0x560f678efec0 2024-03-14T10:03:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f51199e9b80 10 client.0 _ll_register_callbacks cb 0x560f6763f3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:03:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f50f7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5fa0 0x560f678f6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f50f7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5fa0 0x560f678f6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37476/0 (socket says 172.21.15.78:37476) 2024-03-14T10:03:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f50f7e00700 1 -- 172.21.15.78:0/3469223661 learned_addr learned my addr 172.21.15.78:0/3469223661 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f50ff400700 1 --2- 172.21.15.78:0/3469223661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 0x560f678f4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f50fea00700 1 --2- 172.21.15.78:0/3469223661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5680 0x560f678f6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f50f7e00700 1 -- 172.21.15.78:0/3469223661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 msgr2=0x560f678f4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f50f7e00700 1 --2- 172.21.15.78:0/3469223661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 0x560f678f4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f50f7e00700 1 -- 172.21.15.78:0/3469223661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5680 msgr2=0x560f678f6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f50f7e00700 1 --2- 172.21.15.78:0/3469223661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5680 0x560f678f6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f50f7e00700 1 -- 172.21.15.78:0/3469223661 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e800be40 con 0x560f678e5fa0 2024-03-14T10:03:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f50fea00700 1 --2- 172.21.15.78:0/3469223661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5680 0x560f678f6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:03:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f50f7e00700 1 --2- 172.21.15.78:0/3469223661 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5fa0 0x560f678f6880 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f50e800b750 tx=0x7f50e80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b955cf7e7c4401 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.617+0000 7f50ecc00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:03:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.618+0000 7f50ecc00700 1 -- 172.21.15.78:0/3469223661 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e8026050 con 0x560f678e5fa0 2024-03-14T10:03:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.618+0000 7f50ecc00700 1 -- 172.21.15.78:0/3469223661 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f50e8004560 con 0x560f678e5fa0 2024-03-14T10:03:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.618+0000 7f51199e9b80 1 -- 172.21.15.78:0/3469223661 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f50f001e040 con 0x560f678e5fa0 2024-03-14T10:03:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.618+0000 7f50ecc00700 1 -- 172.21.15.78:0/3469223661 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50e8017040 con 0x560f678e5fa0 2024-03-14T10:03:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.618+0000 7f51199e9b80 10 client.17460 fetch_fsmap learned FSMap version 25 2024-03-14T10:03:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.618+0000 7f51199e9b80 1 -- 172.21.15.78:0/3469223661 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f51001a6930 con 0x560f678e5fa0 2024-03-14T10:03:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.618+0000 7f50ecc00700 1 -- 172.21.15.78:0/3469223661 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50f001e040 con 0x560f678e5fa0 2024-03-14T10:03:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.618+0000 7f50ecc00700 1 -- 172.21.15.78:0/3469223661 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f50e8028620 con 0x560f678e5fa0 2024-03-14T10:03:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.619+0000 7f51199e9b80 10 client.17460 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:03:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.619+0000 7f51199e9b80 10 client.17460 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:03:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.619+0000 7f51199e9b80 20 client.17460 populate_metadata read hostname 'smithi078' 2024-03-14T10:03:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.619+0000 7f51199e9b80 1 --2- 172.21.15.78:0/3469223661 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x560f679522c0 0x560f679546e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.619+0000 7f51199e9b80 4 client.17460 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:03:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.619+0000 7f51199e9b80 1 -- 172.21.15.78:0/3469223661 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560f67686cf0 con 0x560f679522c0 2024-03-14T10:03:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.619+0000 7f50fea00700 1 --2- 172.21.15.78:0/3469223661 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x560f679522c0 0x560f679546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.619+0000 7f50ecc00700 10 client.17460 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:03:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.619+0000 7f50fea00700 1 --2- 172.21.15.78:0/3469223661 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x560f679522c0 0x560f679546e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d54c7f664d51b269 server_cookie=5e6eb14370b62f79 in_seq=0 out_seq=0 2024-03-14T10:03:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.619+0000 7f50ecc00700 1 -- 172.21.15.78:0/3469223661 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560f67686cf0 con 0x560f679522c0 2024-03-14T10:03:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.619+0000 7f50ecc00700 10 client.17460 handle_command_reply: tid=0 2024-03-14T10:03:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.619+0000 7f51199e9b80 1 client.17460 shutdown 2024-03-14T10:03:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.620+0000 7f51199e9b80 1 -- 172.21.15.78:0/3469223661 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5fa0 msgr2=0x560f678f6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.620+0000 7f51199e9b80 1 --2- 172.21.15.78:0/3469223661 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5fa0 0x560f678f6880 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f50e800b750 tx=0x7f50e80047c0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.620+0000 7f51199e9b80 1 -- 172.21.15.78:0/3469223661 shutdown_connections 2024-03-14T10:03:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.620+0000 7f51199e9b80 1 --2- 172.21.15.78:0/3469223661 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x560f679522c0 0x560f679546e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.620+0000 7f51199e9b80 1 --2- 172.21.15.78:0/3469223661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f678e5680 0x560f678f6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.620+0000 7f51199e9b80 1 --2- 172.21.15.78:0/3469223661 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f678efec0 0x560f678f4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.620+0000 7f50ecc00700 0 client.17460 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:03:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.620+0000 7f51199e9b80 1 --2- 172.21.15.78:0/3469223661 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f678e5fa0 0x560f678f6880 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.620+0000 7f51199e9b80 1 -- 172.21.15.78:0/3469223661 >> 172.21.15.78:0/3469223661 conn(0x560f6785b220 msgr2=0x560f678e4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.620+0000 7f51199e9b80 1 -- 172.21.15.78:0/3469223661 shutdown_connections 2024-03-14T10:03:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.620+0000 7f51199e9b80 1 -- 172.21.15.78:0/3469223661 wait complete. 2024-03-14T10:03:28.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.620+0000 7f51199e9b80 20 client.17460 trim_cache size 0 max 16384 2024-03-14T10:03:28.625 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:03:28.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.621+0000 7f5105e00700 1 -- 172.21.15.78:0/1659449063 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f50e40a0c00 msgr2=0x7f50e40a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.621+0000 7f5105e00700 1 --2- 172.21.15.78:0/1659449063 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f50e40a0c00 0x7f50e40a30c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f51000fcea0 tx=0x7f50e8012970 comp rx=0 tx=0).stop 2024-03-14T10:03:28.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.621+0000 7f5105e00700 1 -- 172.21.15.78:0/1659449063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51001996e0 msgr2=0x7f5100199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.621+0000 7f5105e00700 1 --2- 172.21.15.78:0/1659449063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51001996e0 0x7f5100199b90 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f50f800efc0 tx=0x7f50f800c3a0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.621+0000 7f5105e00700 1 -- 172.21.15.78:0/1659449063 shutdown_connections 2024-03-14T10:03:28.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.621+0000 7f5105e00700 1 --2- 172.21.15.78:0/1659449063 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f50e40a0c00 0x7f50e40a30c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.621+0000 7f5105e00700 1 --2- 172.21.15.78:0/1659449063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5100100f20 0x7f510019f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.621+0000 7f5105e00700 1 --2- 172.21.15.78:0/1659449063 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51001996e0 0x7f5100199b90 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.621+0000 7f5105e00700 1 --2- 172.21.15.78:0/1659449063 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5100106040 0x7f510019faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.621+0000 7f5105e00700 1 -- 172.21.15.78:0/1659449063 >> 172.21.15.78:0/1659449063 conn(0x7f51000fc7a0 msgr2=0x7f51000feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.621+0000 7f5105e00700 1 -- 172.21.15.78:0/1659449063 shutdown_connections 2024-03-14T10:03:28.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.621+0000 7f5105e00700 1 -- 172.21.15.78:0/1659449063 wait complete. 2024-03-14T10:03:28.639 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T10:03:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.834+0000 7f9305800700 1 -- 172.21.15.78:0/2745630894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9300106060 msgr2=0x7f930010dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.834+0000 7f9305800700 1 --2- 172.21.15.78:0/2745630894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9300106060 0x7f930010dbe0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f92f800b0d0 tx=0x7f92f800b3e0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.834+0000 7f9305800700 1 -- 172.21.15.78:0/2745630894 shutdown_connections 2024-03-14T10:03:28.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.834+0000 7f9305800700 1 --2- 172.21.15.78:0/2745630894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9300106060 0x7f930010dbe0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.834+0000 7f9305800700 1 --2- 172.21.15.78:0/2745630894 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f93000680b0 0x7f9300068490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.834+0000 7f9305800700 1 --2- 172.21.15.78:0/2745630894 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9300068a60 0x7f9300105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.834+0000 7f9305800700 1 -- 172.21.15.78:0/2745630894 >> 172.21.15.78:0/2745630894 conn(0x7f9300076d80 msgr2=0x7f9300077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.834+0000 7f9305800700 1 -- 172.21.15.78:0/2745630894 shutdown_connections 2024-03-14T10:03:28.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.834+0000 7f9305800700 1 -- 172.21.15.78:0/2745630894 wait complete. 2024-03-14T10:03:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.835+0000 7f9305800700 1 Processor -- start 2024-03-14T10:03:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.836+0000 7f9305800700 1 -- start start 2024-03-14T10:03:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.836+0000 7f9305800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93000680b0 0x7f930019f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.836+0000 7f9305800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9300106060 0x7f930019fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.836+0000 7f9305800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9300199770 0x7f9300199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.836+0000 7f9305800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930019a1f0 con 0x7f9300106060 2024-03-14T10:03:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.836+0000 7f9305800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930019a360 con 0x7f93000680b0 2024-03-14T10:03:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.836+0000 7f9305800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930019a520 con 0x7f9300199770 2024-03-14T10:03:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.836+0000 7f92f7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9300106060 0x7f930019fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.836+0000 7f92f7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9300106060 0x7f930019fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37496/0 (socket says 172.21.15.78:37496) 2024-03-14T10:03:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.836+0000 7f92f7e00700 1 -- 172.21.15.78:0/4247397488 learned_addr learned my addr 172.21.15.78:0/4247397488 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:28.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.836+0000 7f92fea00700 1 --2- 172.21.15.78:0/4247397488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93000680b0 0x7f930019f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.836+0000 7f92ff400700 1 --2- 172.21.15.78:0/4247397488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9300199770 0x7f9300199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.837+0000 7f92f7e00700 1 -- 172.21.15.78:0/4247397488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9300199770 msgr2=0x7f9300199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.837+0000 7f92f7e00700 1 --2- 172.21.15.78:0/4247397488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9300199770 0x7f9300199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.837+0000 7f92f7e00700 1 -- 172.21.15.78:0/4247397488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93000680b0 msgr2=0x7f930019f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.837+0000 7f92f7e00700 1 --2- 172.21.15.78:0/4247397488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93000680b0 0x7f930019f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.837+0000 7f92f7e00700 1 -- 172.21.15.78:0/4247397488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e8009520 con 0x7f9300106060 2024-03-14T10:03:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.837+0000 7f92fea00700 1 --2- 172.21.15.78:0/4247397488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93000680b0 0x7f930019f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:03:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.837+0000 7f92f7e00700 1 --2- 172.21.15.78:0/4247397488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9300106060 0x7f930019fb30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f92e800ea30 tx=0x7f92e800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.837+0000 7f92f7400700 1 -- 172.21.15.78:0/4247397488 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e800cbd0 con 0x7f9300106060 2024-03-14T10:03:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.837+0000 7f92f7400700 1 -- 172.21.15.78:0/4247397488 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f92e800eed0 con 0x7f9300106060 2024-03-14T10:03:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.837+0000 7f9305800700 1 -- 172.21.15.78:0/4247397488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f8009d00 con 0x7f9300106060 2024-03-14T10:03:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.837+0000 7f92f7400700 1 -- 172.21.15.78:0/4247397488 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e8003d20 con 0x7f9300106060 2024-03-14T10:03:28.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.837+0000 7f9305800700 1 -- 172.21.15.78:0/4247397488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93001a6860 con 0x7f9300106060 2024-03-14T10:03:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.840+0000 7f92f7400700 1 -- 172.21.15.78:0/4247397488 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f92e8003ee0 con 0x7f9300106060 2024-03-14T10:03:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.842+0000 7f9319404b80 1 Processor -- start 2024-03-14T10:03:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.842+0000 7f9319404b80 1 -- start start 2024-03-14T10:03:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.843+0000 7f92f7400700 1 --2- 172.21.15.78:0/4247397488 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f92e40a0c80 0x7f92e40a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.843+0000 7f9319404b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4680 0x563f1fec4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.843+0000 7f92f7400700 1 -- 172.21.15.78:0/4247397488 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f92e8014070 con 0x7f9300106060 2024-03-14T10:03:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.843+0000 7f9319404b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4fa0 0x563f1fece980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.843+0000 7f9319404b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 0x563f1fed12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.843+0000 7f9319404b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1fe1d990 con 0x563f1fec4fa0 2024-03-14T10:03:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.843+0000 7f9319404b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1fe1b350 con 0x563f1feceec0 2024-03-14T10:03:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.843+0000 7f9319404b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1fe16c40 con 0x563f1fec4680 2024-03-14T10:03:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.843+0000 7f92ff400700 1 --2- 172.21.15.78:0/4247397488 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f92e40a0c80 0x7f92e40a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.843+0000 7f92fea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4680 0x563f1fec4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.843+0000 7f92fea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4680 0x563f1fec4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:52134/0 (socket says 172.21.15.78:52134) 2024-03-14T10:03:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.843+0000 7f92f7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4fa0 0x563f1fece980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.843+0000 7f92fea00700 1 -- 172.21.15.78:0/4172431077 learned_addr learned my addr 172.21.15.78:0/4172431077 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.843+0000 7f92ff400700 1 --2- 172.21.15.78:0/4172431077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 0x563f1fed12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.844+0000 7f92fea00700 1 -- 172.21.15.78:0/4172431077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 msgr2=0x563f1fed12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.844+0000 7f92fea00700 1 --2- 172.21.15.78:0/4172431077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 0x563f1fed12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.844+0000 7f92fea00700 1 -- 172.21.15.78:0/4172431077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4fa0 msgr2=0x563f1fece980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.844+0000 7f92fea00700 1 --2- 172.21.15.78:0/4172431077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4fa0 0x563f1fece980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.844+0000 7f92fea00700 1 -- 172.21.15.78:0/4172431077 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f1fc983b0 con 0x563f1fec4680 2024-03-14T10:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.844+0000 7f92ff400700 1 --2- 172.21.15.78:0/4172431077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 0x563f1fed12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.844+0000 7f92f7e00700 1 --2- 172.21.15.78:0/4172431077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4fa0 0x563f1fece980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.844+0000 7f92ff400700 1 --2- 172.21.15.78:0/4247397488 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f92e40a0c80 0x7f92e40a3140 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f92f8009fd0 tx=0x7f92f8008bf0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.844+0000 7f92fea00700 1 --2- 172.21.15.78:0/4172431077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4680 0x563f1fec4a60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f92f00115f0 tx=0x7f92f0011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5a2c99f2e3e62a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.844+0000 7f92efe00700 1 -- 172.21.15.78:0/4172431077 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f0019630 con 0x563f1fec4680 2024-03-14T10:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.844+0000 7f92efe00700 1 -- 172.21.15.78:0/4172431077 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f92f0019c50 con 0x563f1fec4680 2024-03-14T10:03:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.844+0000 7f92efe00700 1 -- 172.21.15.78:0/4172431077 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f000eb60 con 0x563f1fec4680 2024-03-14T10:03:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.845+0000 7f9319404b80 1 -- 172.21.15.78:0/4172431077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4680 msgr2=0x563f1fec4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.845+0000 7f9319404b80 1 --2- 172.21.15.78:0/4172431077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4680 0x563f1fec4a60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f92f00115f0 tx=0x7f92f0011900 comp rx=0 tx=0).stop 2024-03-14T10:03:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.845+0000 7f9319404b80 1 -- 172.21.15.78:0/4172431077 shutdown_connections 2024-03-14T10:03:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.845+0000 7f9319404b80 1 --2- 172.21.15.78:0/4172431077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 0x563f1fed12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.845+0000 7f9319404b80 1 --2- 172.21.15.78:0/4172431077 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4680 0x563f1fec4a60 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.845+0000 7f9319404b80 1 --2- 172.21.15.78:0/4172431077 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4fa0 0x563f1fece980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.845+0000 7f9319404b80 1 -- 172.21.15.78:0/4172431077 >> 172.21.15.78:0/4172431077 conn(0x563f1fe3a220 msgr2=0x563f1febfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.845+0000 7f9319404b80 1 -- 172.21.15.78:0/4172431077 shutdown_connections 2024-03-14T10:03:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.845+0000 7f9319404b80 1 -- 172.21.15.78:0/4172431077 wait complete. 2024-03-14T10:03:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.846+0000 7f9319404b80 1 -- start start 2024-03-14T10:03:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.846+0000 7f9319404b80 1 Processor -- start 2024-03-14T10:03:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.846+0000 7f9319404b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fece8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.846+0000 7f9319404b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4fa0 0x563f1fecbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.846+0000 7f9319404b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 0x563f1fecc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.846+0000 7f9319404b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1fdb2d80 con 0x563f1fec4680 2024-03-14T10:03:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.846+0000 7f9319404b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1fdb2a20 con 0x563f1feceec0 2024-03-14T10:03:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.846+0000 7f9319404b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1fc9a6a0 con 0x563f1fec4fa0 2024-03-14T10:03:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.846+0000 7f9319404b80 10 client.0 _ll_register_callbacks cb 0x563f1fc1e3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:03:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.847+0000 7f92fea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fece8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.847+0000 7f92fea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fece8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37514/0 (socket says 172.21.15.78:37514) 2024-03-14T10:03:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.847+0000 7f92fea00700 1 -- 172.21.15.78:0/2589630413 learned_addr learned my addr 172.21.15.78:0/2589630413 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.847+0000 7f92ff400700 1 --2- 172.21.15.78:0/2589630413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 0x563f1fecc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.847+0000 7f92f7e00700 1 --2- 172.21.15.78:0/2589630413 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4fa0 0x563f1fecbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.847+0000 7f92fea00700 1 -- 172.21.15.78:0/2589630413 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4fa0 msgr2=0x563f1fecbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.847+0000 7f92fea00700 1 --2- 172.21.15.78:0/2589630413 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4fa0 0x563f1fecbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.847+0000 7f92fea00700 1 -- 172.21.15.78:0/2589630413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 msgr2=0x563f1fecc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.847+0000 7f92fea00700 1 --2- 172.21.15.78:0/2589630413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 0x563f1fecc280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.847+0000 7f92fea00700 1 -- 172.21.15.78:0/2589630413 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93001a6860 con 0x563f1fec4680 2024-03-14T10:03:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.847+0000 7f92ff400700 1 --2- 172.21.15.78:0/2589630413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 0x563f1fecc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:03:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.847+0000 7f92fea00700 1 --2- 172.21.15.78:0/2589630413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fece8c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f92f0005920 tx=0x7f92f00046a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa5a057c6efc054 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.847+0000 7f92ecc00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:03:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.848+0000 7f92ecc00700 1 -- 172.21.15.78:0/2589630413 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f0019630 con 0x563f1fec4680 2024-03-14T10:03:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.848+0000 7f92ecc00700 1 -- 172.21.15.78:0/2589630413 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f92f0019790 con 0x563f1fec4680 2024-03-14T10:03:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.848+0000 7f9319404b80 1 -- 172.21.15.78:0/2589630413 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563f1feb7720 con 0x563f1fec4680 2024-03-14T10:03:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.848+0000 7f92ecc00700 1 -- 172.21.15.78:0/2589630413 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92f0019950 con 0x563f1fec4680 2024-03-14T10:03:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.848+0000 7f9319404b80 10 client.17472 fetch_fsmap learned FSMap version 25 2024-03-14T10:03:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.848+0000 7f9319404b80 1 -- 172.21.15.78:0/2589630413 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f92f0009dd0 con 0x563f1fec4680 2024-03-14T10:03:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.848+0000 7f92ecc00700 1 -- 172.21.15.78:0/2589630413 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563f1feb7720 con 0x563f1fec4680 2024-03-14T10:03:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.848+0000 7f92ecc00700 1 -- 172.21.15.78:0/2589630413 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f92f000eab0 con 0x563f1fec4680 2024-03-14T10:03:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.849+0000 7f9319404b80 10 client.17472 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:03:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.849+0000 7f9319404b80 10 client.17472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:03:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.849+0000 7f9319404b80 20 client.17472 populate_metadata read hostname 'smithi078' 2024-03-14T10:03:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.849+0000 7f9319404b80 1 --2- 172.21.15.78:0/2589630413 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563f1fede9b0 0x563f1fee0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.849+0000 7f9319404b80 4 client.17472 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:03:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.849+0000 7f9319404b80 1 -- 172.21.15.78:0/2589630413 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f1fe0f700 con 0x563f1fede9b0 2024-03-14T10:03:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.849+0000 7f92f7e00700 1 --2- 172.21.15.78:0/2589630413 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563f1fede9b0 0x563f1fee0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.849+0000 7f92ecc00700 10 client.17472 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:03:28.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.849+0000 7f92f7e00700 1 --2- 172.21.15.78:0/2589630413 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563f1fede9b0 0x563f1fee0dd0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e44378aa8269d11 server_cookie=56ed8e3db07637b0 in_seq=0 out_seq=0 2024-03-14T10:03:28.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.850+0000 7f92ecc00700 1 -- 172.21.15.78:0/2589630413 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x563f1fe0f700 con 0x563f1fede9b0 2024-03-14T10:03:28.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.850+0000 7f92ecc00700 10 client.17472 handle_command_reply: tid=0 2024-03-14T10:03:28.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.850+0000 7f9319404b80 1 client.17472 shutdown 2024-03-14T10:03:28.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.850+0000 7f9319404b80 1 -- 172.21.15.78:0/2589630413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 msgr2=0x563f1fece8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.850+0000 7f9319404b80 1 --2- 172.21.15.78:0/2589630413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fece8c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f92f0005920 tx=0x7f92f00046a0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.851+0000 7f9319404b80 1 -- 172.21.15.78:0/2589630413 shutdown_connections 2024-03-14T10:03:28.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.851+0000 7f9319404b80 1 --2- 172.21.15.78:0/2589630413 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563f1fede9b0 0x563f1fee0dd0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.851+0000 7f9319404b80 1 --2- 172.21.15.78:0/2589630413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 0x563f1fecc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.851+0000 7f9319404b80 1 --2- 172.21.15.78:0/2589630413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fece8c0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.851+0000 7f92ecc00700 0 client.17472 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:03:28.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.851+0000 7f9319404b80 1 --2- 172.21.15.78:0/2589630413 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4fa0 0x563f1fecbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.851+0000 7f9319404b80 1 -- 172.21.15.78:0/2589630413 >> 172.21.15.78:0/2589630413 conn(0x563f1fe3a220 msgr2=0x563f1febf690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.851+0000 7f9319404b80 1 -- 172.21.15.78:0/2589630413 shutdown_connections 2024-03-14T10:03:28.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.851+0000 7f9319404b80 1 -- 172.21.15.78:0/2589630413 wait complete. 2024-03-14T10:03:28.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.851+0000 7f9319404b80 20 client.17472 trim_cache size 0 max 16384 2024-03-14T10:03:28.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.864+0000 7f9319404b80 1 Processor -- start 2024-03-14T10:03:28.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.864+0000 7f9319404b80 1 -- start start 2024-03-14T10:03:28.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.864+0000 7f9319404b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fee5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.864+0000 7f9319404b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4fa0 0x563f1fec6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.864+0000 7f9319404b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 0x563f1fec7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.864+0000 7f9319404b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1fec1080 con 0x563f1fec4680 2024-03-14T10:03:28.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.864+0000 7f9319404b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1fec5af0 con 0x563f1feceec0 2024-03-14T10:03:28.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.864+0000 7f9319404b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1fec60c0 con 0x563f1fec4fa0 2024-03-14T10:03:28.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.864+0000 7f92fea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fee5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.864+0000 7f92fea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fee5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37518/0 (socket says 172.21.15.78:37518) 2024-03-14T10:03:28.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.865+0000 7f92fea00700 1 -- 172.21.15.78:0/2152739513 learned_addr learned my addr 172.21.15.78:0/2152739513 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:28.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.865+0000 7f92f7e00700 1 --2- 172.21.15.78:0/2152739513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4fa0 0x563f1fec6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.865+0000 7f92ff400700 1 --2- 172.21.15.78:0/2152739513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 0x563f1fec7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.865+0000 7f92fea00700 1 -- 172.21.15.78:0/2152739513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4fa0 msgr2=0x563f1fec6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.865+0000 7f92fea00700 1 --2- 172.21.15.78:0/2152739513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4fa0 0x563f1fec6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.865+0000 7f92fea00700 1 -- 172.21.15.78:0/2152739513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 msgr2=0x563f1fec7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.865+0000 7f92fea00700 1 --2- 172.21.15.78:0/2152739513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 0x563f1fec7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.865+0000 7f92fea00700 1 -- 172.21.15.78:0/2152739513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f1fc44b60 con 0x563f1fec4680 2024-03-14T10:03:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.865+0000 7f92fea00700 1 --2- 172.21.15.78:0/2152739513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fee5730 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f92f0005950 tx=0x7f92f000e5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78c1ed5224aa19f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.865+0000 7f92f4c00700 1 -- 172.21.15.78:0/2152739513 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f0011d00 con 0x563f1fec4680 2024-03-14T10:03:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.865+0000 7f92f4c00700 1 -- 172.21.15.78:0/2152739513 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f92f0019dc0 con 0x563f1fec4680 2024-03-14T10:03:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.865+0000 7f92f4c00700 1 -- 172.21.15.78:0/2152739513 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f000f6f0 con 0x563f1fec4680 2024-03-14T10:03:28.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.866+0000 7f9319404b80 1 -- 172.21.15.78:0/2152739513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 msgr2=0x563f1fee5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.866+0000 7f9319404b80 1 --2- 172.21.15.78:0/2152739513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fee5730 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f92f0005950 tx=0x7f92f000e5c0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.866+0000 7f9319404b80 1 -- 172.21.15.78:0/2152739513 shutdown_connections 2024-03-14T10:03:28.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.866+0000 7f9319404b80 1 --2- 172.21.15.78:0/2152739513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1feceec0 0x563f1fec7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.866+0000 7f9319404b80 1 --2- 172.21.15.78:0/2152739513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fee5730 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.866+0000 7f9319404b80 1 --2- 172.21.15.78:0/2152739513 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1fec4fa0 0x563f1fec6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.866+0000 7f9319404b80 1 -- 172.21.15.78:0/2152739513 >> 172.21.15.78:0/2152739513 conn(0x563f1fe3a220 msgr2=0x563f1febe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.866+0000 7f9319404b80 1 -- 172.21.15.78:0/2152739513 shutdown_connections 2024-03-14T10:03:28.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.866+0000 7f9319404b80 1 -- 172.21.15.78:0/2152739513 wait complete. 2024-03-14T10:03:28.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.866+0000 7f9319404b80 1 -- start start 2024-03-14T10:03:28.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.867+0000 7f9319404b80 1 Processor -- start 2024-03-14T10:03:28.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.867+0000 7f9319404b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fed3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.867+0000 7f9319404b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1fec4fa0 0x563f1fed4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.867+0000 7f9319404b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1feceec0 0x563f1fed45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.867+0000 7f9319404b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1ff2f4d0 con 0x563f1fec4680 2024-03-14T10:03:28.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.867+0000 7f9319404b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1ff2f610 con 0x563f1fec4fa0 2024-03-14T10:03:28.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.867+0000 7f9319404b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f1ff2f750 con 0x563f1feceec0 2024-03-14T10:03:28.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.867+0000 7f9319404b80 10 client.0 _ll_register_callbacks cb 0x7f92e8095cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:03:28.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.867+0000 7f92fea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fed3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.867+0000 7f92fea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fed3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37534/0 (socket says 172.21.15.78:37534) 2024-03-14T10:03:28.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.867+0000 7f92fea00700 1 -- 172.21.15.78:0/3459009555 learned_addr learned my addr 172.21.15.78:0/3459009555 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:28.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.867+0000 7f92f7e00700 1 --2- 172.21.15.78:0/3459009555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1fec4fa0 0x563f1fed4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.867+0000 7f92ff400700 1 --2- 172.21.15.78:0/3459009555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1feceec0 0x563f1fed45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.868+0000 7f92fea00700 1 -- 172.21.15.78:0/3459009555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1feceec0 msgr2=0x563f1fed45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.868+0000 7f92fea00700 1 --2- 172.21.15.78:0/3459009555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1feceec0 0x563f1fed45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.868+0000 7f92fea00700 1 -- 172.21.15.78:0/3459009555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1fec4fa0 msgr2=0x563f1fed4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.868+0000 7f92fea00700 1 --2- 172.21.15.78:0/3459009555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1fec4fa0 0x563f1fed4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.868+0000 7f92fea00700 1 -- 172.21.15.78:0/3459009555 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f0009dd0 con 0x563f1fec4680 2024-03-14T10:03:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.868+0000 7f92ff400700 1 --2- 172.21.15.78:0/3459009555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1feceec0 0x563f1fed45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:03:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.868+0000 7f92fea00700 1 --2- 172.21.15.78:0/3459009555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fed3b30 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f92f000ff00 tx=0x7f92f000ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91828334d8e06d1f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.868+0000 7f92ecc00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:03:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.868+0000 7f92ecc00700 1 -- 172.21.15.78:0/3459009555 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f000ea40 con 0x563f1fec4680 2024-03-14T10:03:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.868+0000 7f92ecc00700 1 -- 172.21.15.78:0/3459009555 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f92f0000e40 con 0x563f1fec4680 2024-03-14T10:03:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.868+0000 7f9319404b80 1 -- 172.21.15.78:0/3459009555 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f92f0019c50 con 0x563f1fec4680 2024-03-14T10:03:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.869+0000 7f92ecc00700 1 -- 172.21.15.78:0/3459009555 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92f000f3c0 con 0x563f1fec4680 2024-03-14T10:03:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.869+0000 7f9319404b80 10 client.17478 fetch_fsmap learned FSMap version 25 2024-03-14T10:03:28.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.869+0000 7f9319404b80 1 -- 172.21.15.78:0/3459009555 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x563f1fddcea0 con 0x563f1fec4680 2024-03-14T10:03:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.869+0000 7f92ecc00700 1 -- 172.21.15.78:0/3459009555 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f92f0019c50 con 0x563f1fec4680 2024-03-14T10:03:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.869+0000 7f92ecc00700 1 -- 172.21.15.78:0/3459009555 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f92f0019960 con 0x563f1fec4680 2024-03-14T10:03:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.869+0000 7f9319404b80 10 client.17478 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:03:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.869+0000 7f9319404b80 10 client.17478 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:03:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.869+0000 7f9319404b80 20 client.17478 populate_metadata read hostname 'smithi078' 2024-03-14T10:03:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.869+0000 7f9319404b80 1 --2- 172.21.15.78:0/3459009555 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563f1ff309c0 0x563f1ff32db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.869+0000 7f9319404b80 4 client.17478 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:03:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.869+0000 7f9319404b80 1 -- 172.21.15.78:0/3459009555 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563f1fc65cf0 con 0x563f1ff309c0 2024-03-14T10:03:28.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.869+0000 7f92f7e00700 1 --2- 172.21.15.78:0/3459009555 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563f1ff309c0 0x563f1ff32db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.870+0000 7f92ecc00700 10 client.17478 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:03:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.870+0000 7f92f7e00700 1 --2- 172.21.15.78:0/3459009555 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563f1ff309c0 0x563f1ff32db0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8d3b909c4e76c43 server_cookie=bc5f7beed007ee80 in_seq=0 out_seq=0 2024-03-14T10:03:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.870+0000 7f92ecc00700 1 -- 172.21.15.78:0/3459009555 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563f1fc65cf0 con 0x563f1ff309c0 2024-03-14T10:03:28.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.870+0000 7f92ecc00700 10 client.17478 handle_command_reply: tid=0 2024-03-14T10:03:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.870+0000 7f9319404b80 1 client.17478 shutdown 2024-03-14T10:03:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.870+0000 7f9319404b80 1 -- 172.21.15.78:0/3459009555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 msgr2=0x563f1fed3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.870+0000 7f9319404b80 1 --2- 172.21.15.78:0/3459009555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fed3b30 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f92f000ff00 tx=0x7f92f000ed10 comp rx=0 tx=0).stop 2024-03-14T10:03:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.870+0000 7f9319404b80 1 -- 172.21.15.78:0/3459009555 shutdown_connections 2024-03-14T10:03:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.870+0000 7f9319404b80 1 --2- 172.21.15.78:0/3459009555 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x563f1feceec0 0x563f1fed45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.870+0000 7f9319404b80 1 --2- 172.21.15.78:0/3459009555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563f1fec4680 0x563f1fed3b30 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.870+0000 7f9319404b80 1 --2- 172.21.15.78:0/3459009555 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x563f1ff309c0 0x563f1ff32db0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.870+0000 7f9319404b80 1 --2- 172.21.15.78:0/3459009555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x563f1fec4fa0 0x563f1fed4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.870+0000 7f9319404b80 1 -- 172.21.15.78:0/3459009555 >> 172.21.15.78:0/3459009555 conn(0x563f1fe3a220 msgr2=0x563f1fec6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.870+0000 7f92ecc00700 0 client.17478 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:03:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.870+0000 7f9319404b80 1 -- 172.21.15.78:0/3459009555 shutdown_connections 2024-03-14T10:03:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.870+0000 7f9319404b80 1 -- 172.21.15.78:0/3459009555 wait complete. 2024-03-14T10:03:28.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.871+0000 7f9319404b80 20 client.17478 trim_cache size 0 max 16384 2024-03-14T10:03:28.876 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:03:28.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.871+0000 7f9305800700 1 -- 172.21.15.78:0/4247397488 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f92e40a0c80 msgr2=0x7f92e40a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.871+0000 7f9305800700 1 --2- 172.21.15.78:0/4247397488 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f92e40a0c80 0x7f92e40a3140 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f92f8009fd0 tx=0x7f92f8008bf0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.871+0000 7f9305800700 1 -- 172.21.15.78:0/4247397488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9300106060 msgr2=0x7f930019fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:28.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.871+0000 7f9305800700 1 --2- 172.21.15.78:0/4247397488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9300106060 0x7f930019fb30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f92e800ea30 tx=0x7f92e800ed40 comp rx=0 tx=0).stop 2024-03-14T10:03:28.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.871+0000 7f9305800700 1 -- 172.21.15.78:0/4247397488 shutdown_connections 2024-03-14T10:03:28.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.871+0000 7f9305800700 1 --2- 172.21.15.78:0/4247397488 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f92e40a0c80 0x7f92e40a3140 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.871+0000 7f9305800700 1 --2- 172.21.15.78:0/4247397488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93000680b0 0x7f930019f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.871+0000 7f9305800700 1 --2- 172.21.15.78:0/4247397488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9300199770 0x7f9300199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.871+0000 7f9305800700 1 --2- 172.21.15.78:0/4247397488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9300106060 0x7f930019fb30 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:28.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.871+0000 7f9305800700 1 -- 172.21.15.78:0/4247397488 >> 172.21.15.78:0/4247397488 conn(0x7f9300076d80 msgr2=0x7f930010c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:28.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.871+0000 7f9305800700 1 -- 172.21.15.78:0/4247397488 shutdown_connections 2024-03-14T10:03:28.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:28.871+0000 7f9305800700 1 -- 172.21.15.78:0/4247397488 wait complete. 2024-03-14T10:03:29.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:28 smithi078 ceph-mon[38143]: pgmap v288: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T10:03:29.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:28 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1122599884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:03:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:28 smithi152 ceph-mon[48279]: pgmap v288: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T10:03:29.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:28 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1122599884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:03:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:28 smithi132 ceph-mon[47959]: pgmap v288: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T10:03:29.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:28 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1122599884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:03:29.891 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T10:03:29.891 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T10:03:30.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.092+0000 7f81e1000700 1 -- 172.21.15.78:0/2234873108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81dc1018e0 msgr2=0x7f81dc105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.092+0000 7f81e1000700 1 --2- 172.21.15.78:0/2234873108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81dc1018e0 0x7f81dc105920 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009990 tx=0x7f81c4009ca0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.092+0000 7f81e1000700 1 -- 172.21.15.78:0/2234873108 shutdown_connections 2024-03-14T10:03:30.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.092+0000 7f81e1000700 1 --2- 172.21.15.78:0/2234873108 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f81dc106050 0x7f81dc10dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.092+0000 7f81e1000700 1 --2- 172.21.15.78:0/2234873108 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f81dc100f30 0x7f81dc101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.092+0000 7f81e1000700 1 --2- 172.21.15.78:0/2234873108 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81dc1018e0 0x7f81dc105920 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.092+0000 7f81e1000700 1 -- 172.21.15.78:0/2234873108 >> 172.21.15.78:0/2234873108 conn(0x7f81dc0fc7b0 msgr2=0x7f81dc0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:30.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.092+0000 7f81e1000700 1 -- 172.21.15.78:0/2234873108 shutdown_connections 2024-03-14T10:03:30.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.092+0000 7f81e1000700 1 -- 172.21.15.78:0/2234873108 wait complete. 2024-03-14T10:03:30.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.093+0000 7f81e1000700 1 Processor -- start 2024-03-14T10:03:30.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.093+0000 7f81e1000700 1 -- start start 2024-03-14T10:03:30.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.093+0000 7f81e1000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f81dc100f30 0x7f81dc19f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.093+0000 7f81e1000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81dc106050 0x7f81dc19fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.093+0000 7f81e1000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f81dc199660 0x7f81dc199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.093+0000 7f81e1000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc19a0e0 con 0x7f81dc106050 2024-03-14T10:03:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.093+0000 7f81e1000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc19a250 con 0x7f81dc199660 2024-03-14T10:03:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.093+0000 7f81e1000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81dc19a410 con 0x7f81dc100f30 2024-03-14T10:03:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.094+0000 7f81d3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81dc106050 0x7f81dc19fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.094+0000 7f81d3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81dc106050 0x7f81dc19fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37544/0 (socket says 172.21.15.78:37544) 2024-03-14T10:03:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.094+0000 7f81d3e00700 1 -- 172.21.15.78:0/2795429634 learned_addr learned my addr 172.21.15.78:0/2795429634 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.094+0000 7f81d9e00700 1 --2- 172.21.15.78:0/2795429634 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f81dc100f30 0x7f81dc19f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.094+0000 7f81da800700 1 --2- 172.21.15.78:0/2795429634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f81dc199660 0x7f81dc199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.094+0000 7f81d3e00700 1 -- 172.21.15.78:0/2795429634 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f81dc100f30 msgr2=0x7f81dc19f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.094+0000 7f81d3e00700 1 --2- 172.21.15.78:0/2795429634 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f81dc100f30 0x7f81dc19f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.094+0000 7f81d3e00700 1 -- 172.21.15.78:0/2795429634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f81dc199660 msgr2=0x7f81dc199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.094+0000 7f81d3e00700 1 --2- 172.21.15.78:0/2795429634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f81dc199660 0x7f81dc199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.094+0000 7f81d3e00700 1 -- 172.21.15.78:0/2795429634 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c40095f0 con 0x7f81dc106050 2024-03-14T10:03:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.094+0000 7f81d3e00700 1 --2- 172.21.15.78:0/2795429634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81dc106050 0x7f81dc19fa70 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f81c4000c00 tx=0x7f81c4005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.095+0000 7f81d2a00700 1 -- 172.21.15.78:0/2795429634 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c401d070 con 0x7f81dc106050 2024-03-14T10:03:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.095+0000 7f81d2a00700 1 -- 172.21.15.78:0/2795429634 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f81c4022470 con 0x7f81dc106050 2024-03-14T10:03:30.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.095+0000 7f81e1000700 1 -- 172.21.15.78:0/2795429634 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81dc1a63a0 con 0x7f81dc106050 2024-03-14T10:03:30.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.095+0000 7f81d2a00700 1 -- 172.21.15.78:0/2795429634 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c4017650 con 0x7f81dc106050 2024-03-14T10:03:30.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.095+0000 7f81e1000700 1 -- 172.21.15.78:0/2795429634 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81dc1a67c0 con 0x7f81dc106050 2024-03-14T10:03:30.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.098+0000 7f81d2a00700 1 -- 172.21.15.78:0/2795429634 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f81c400fad0 con 0x7f81dc106050 2024-03-14T10:03:30.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.100+0000 7f81f4bd6b80 1 Processor -- start 2024-03-14T10:03:30.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.100+0000 7f81f4bd6b80 1 -- start start 2024-03-14T10:03:30.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.100+0000 7f81d2a00700 1 --2- 172.21.15.78:0/2795429634 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f81c00a0c00 0x7f81c00a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.100+0000 7f81f4bd6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db55700 0x560f2db55ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.100+0000 7f81f4bd6b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db56020 0x560f2db5fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.100+0000 7f81f4bd6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db5ff40 0x560f2db62330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.100+0000 7f81f4bd6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2daaea10 con 0x560f2db55700 2024-03-14T10:03:30.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.100+0000 7f81f4bd6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2daac3d0 con 0x560f2db5ff40 2024-03-14T10:03:30.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.100+0000 7f81f4bd6b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2daa7cc0 con 0x560f2db56020 2024-03-14T10:03:30.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.100+0000 7f81d2a00700 1 -- 172.21.15.78:0/2795429634 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f81c40cdbe0 con 0x7f81dc106050 2024-03-14T10:03:30.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.101+0000 7f81d9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db55700 0x560f2db55ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.101+0000 7f81da800700 1 --2- 172.21.15.78:0/2795429634 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f81c00a0c00 0x7f81c00a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.101+0000 7f81d9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db55700 0x560f2db55ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37550/0 (socket says 172.21.15.78:37550) 2024-03-14T10:03:30.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.101+0000 7f81d9e00700 1 -- 172.21.15.78:0/2486995655 learned_addr learned my addr 172.21.15.78:0/2486995655 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:30.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.101+0000 7f81d3e00700 1 --2- 172.21.15.78:0/2486995655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db56020 0x560f2db5fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.101+0000 7f81da800700 1 --2- 172.21.15.78:0/2486995655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db5ff40 0x560f2db62330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.101+0000 7f81d3e00700 1 -- 172.21.15.78:0/2486995655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db5ff40 msgr2=0x560f2db62330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:30.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.101+0000 7f81d3e00700 1 --2- 172.21.15.78:0/2486995655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db5ff40 0x560f2db62330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.101+0000 7f81d3e00700 1 -- 172.21.15.78:0/2486995655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db55700 msgr2=0x560f2db55ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:30.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.101+0000 7f81d3e00700 1 --2- 172.21.15.78:0/2486995655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db55700 0x560f2db55ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.101+0000 7f81d3e00700 1 -- 172.21.15.78:0/2486995655 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f2d929430 con 0x560f2db56020 2024-03-14T10:03:30.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.101+0000 7f81d9e00700 1 --2- 172.21.15.78:0/2486995655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db55700 0x560f2db55ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:03:30.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.101+0000 7f81da800700 1 --2- 172.21.15.78:0/2795429634 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f81c00a0c00 0x7f81c00a30c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f81d4003fd0 tx=0x7f81d4003a00 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:30.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.101+0000 7f81d3e00700 1 --2- 172.21.15.78:0/2486995655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db56020 0x560f2db5fa00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f81c40caf00 tx=0x7f81c4096fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=266046b2dc0271c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:30.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.102+0000 7f81cb400700 1 -- 172.21.15.78:0/2486995655 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c409a4f0 con 0x560f2db56020 2024-03-14T10:03:30.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.102+0000 7f81cb400700 1 -- 172.21.15.78:0/2486995655 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f81c4072f90 con 0x560f2db56020 2024-03-14T10:03:30.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.102+0000 7f81cb400700 1 -- 172.21.15.78:0/2486995655 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c40a2f80 con 0x560f2db56020 2024-03-14T10:03:30.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.102+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/2486995655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db56020 msgr2=0x560f2db5fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.102+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/2486995655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db56020 0x560f2db5fa00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f81c40caf00 tx=0x7f81c4096fc0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.102+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/2486995655 shutdown_connections 2024-03-14T10:03:30.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.102+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/2486995655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db5ff40 0x560f2db62330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.102+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/2486995655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db55700 0x560f2db55ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.102+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/2486995655 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db56020 0x560f2db5fa00 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.102+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/2486995655 >> 172.21.15.78:0/2486995655 conn(0x560f2dacb2a0 msgr2=0x560f2db50b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:30.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.102+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/2486995655 shutdown_connections 2024-03-14T10:03:30.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.102+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/2486995655 wait complete. 2024-03-14T10:03:30.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.103+0000 7f81f4bd6b80 1 -- start start 2024-03-14T10:03:30.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.103+0000 7f81f4bd6b80 1 Processor -- start 2024-03-14T10:03:30.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.104+0000 7f81f4bd6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db55700 0x560f2db5f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.104+0000 7f81f4bd6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db56020 0x560f2db5cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.104+0000 7f81f4bd6b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db5ff40 0x560f2db5d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.104+0000 7f81f4bd6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2da43e00 con 0x560f2db56020 2024-03-14T10:03:30.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.104+0000 7f81f4bd6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2da43aa0 con 0x560f2db55700 2024-03-14T10:03:30.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.104+0000 7f81f4bd6b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2d92b720 con 0x560f2db5ff40 2024-03-14T10:03:30.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.104+0000 7f81f4bd6b80 10 client.0 _ll_register_callbacks cb 0x560f2d8af420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:03:30.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.104+0000 7f81d3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db56020 0x560f2db5cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.104+0000 7f81d3e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db56020 0x560f2db5cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37554/0 (socket says 172.21.15.78:37554) 2024-03-14T10:03:30.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.104+0000 7f81d3e00700 1 -- 172.21.15.78:0/4042131499 learned_addr learned my addr 172.21.15.78:0/4042131499 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:30.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.104+0000 7f81d9e00700 1 --2- 172.21.15.78:0/4042131499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db55700 0x560f2db5f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.104+0000 7f81da800700 1 --2- 172.21.15.78:0/4042131499 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db5ff40 0x560f2db5d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.105+0000 7f81d3e00700 1 -- 172.21.15.78:0/4042131499 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db5ff40 msgr2=0x560f2db5d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:30.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.105+0000 7f81d3e00700 1 --2- 172.21.15.78:0/4042131499 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db5ff40 0x560f2db5d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.105+0000 7f81d3e00700 1 -- 172.21.15.78:0/4042131499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db55700 msgr2=0x560f2db5f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:30.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.105+0000 7f81d3e00700 1 --2- 172.21.15.78:0/4042131499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db55700 0x560f2db5f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.105+0000 7f81d3e00700 1 -- 172.21.15.78:0/4042131499 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81cc0099e0 con 0x560f2db56020 2024-03-14T10:03:30.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.105+0000 7f81d9e00700 1 --2- 172.21.15.78:0/4042131499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db55700 0x560f2db5f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.105+0000 7f81da800700 1 --2- 172.21.15.78:0/4042131499 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db5ff40 0x560f2db5d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.105+0000 7f81d3e00700 1 --2- 172.21.15.78:0/4042131499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db56020 0x560f2db5cbb0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f81c4005910 tx=0x7f81c409ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0834ca1964f985 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.105+0000 7f81bfe00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.105+0000 7f81bfe00700 1 -- 172.21.15.78:0/4042131499 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c4074040 con 0x560f2db56020 2024-03-14T10:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.105+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/4042131499 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560f2db487a0 con 0x560f2db56020 2024-03-14T10:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.105+0000 7f81bfe00700 1 -- 172.21.15.78:0/4042131499 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f81c406f440 con 0x560f2db56020 2024-03-14T10:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.105+0000 7f81bfe00700 1 -- 172.21.15.78:0/4042131499 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81c409b290 con 0x560f2db56020 2024-03-14T10:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.105+0000 7f81f4bd6b80 10 client.17490 fetch_fsmap learned FSMap version 25 2024-03-14T10:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.105+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/4042131499 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f81dc1a67c0 con 0x560f2db56020 2024-03-14T10:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.106+0000 7f81bfe00700 1 -- 172.21.15.78:0/4042131499 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560f2db487a0 con 0x560f2db56020 2024-03-14T10:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.106+0000 7f81bfe00700 1 -- 172.21.15.78:0/4042131499 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f81c40b32e0 con 0x560f2db56020 2024-03-14T10:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.106+0000 7f81f4bd6b80 10 client.17490 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.106+0000 7f81f4bd6b80 10 client.17490 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.106+0000 7f81f4bd6b80 20 client.17490 populate_metadata read hostname 'smithi078' 2024-03-14T10:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.106+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/4042131499 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x560f2db6f3e0 0x560f2db717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.106+0000 7f81f4bd6b80 4 client.17490 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.106+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/4042131499 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560f2daa0780 con 0x560f2db6f3e0 2024-03-14T10:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.106+0000 7f81d9e00700 1 --2- 172.21.15.78:0/4042131499 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x560f2db6f3e0 0x560f2db717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.107+0000 7f81d9e00700 1 --2- 172.21.15.78:0/4042131499 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x560f2db6f3e0 0x560f2db717d0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41c208794c6da806 server_cookie=9d4240d05c53da9f in_seq=0 out_seq=0 2024-03-14T10:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.107+0000 7f81bfe00700 10 client.17490 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.108+0000 7f81bfe00700 1 -- 172.21.15.78:0/4042131499 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x560f2daa0780 con 0x560f2db6f3e0 2024-03-14T10:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.108+0000 7f81bfe00700 10 client.17490 handle_command_reply: tid=0 2024-03-14T10:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.108+0000 7f81f4bd6b80 1 client.17490 shutdown 2024-03-14T10:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.108+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/4042131499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db56020 msgr2=0x560f2db5cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.108+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/4042131499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db56020 0x560f2db5cbb0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f81c4005910 tx=0x7f81c409ac40 comp rx=0 tx=0).stop 2024-03-14T10:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.108+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/4042131499 shutdown_connections 2024-03-14T10:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.108+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/4042131499 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x560f2db6f3e0 0x560f2db717d0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.108+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/4042131499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db55700 0x560f2db5f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.108+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/4042131499 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db5ff40 0x560f2db5d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.108+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/4042131499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db56020 0x560f2db5cbb0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.108+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/4042131499 >> 172.21.15.78:0/4042131499 conn(0x560f2dacb2a0 msgr2=0x560f2db50710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.108+0000 7f81bfe00700 0 client.17490 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:03:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.108+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/4042131499 shutdown_connections 2024-03-14T10:03:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.108+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/4042131499 wait complete. 2024-03-14T10:03:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.108+0000 7f81f4bd6b80 20 client.17490 trim_cache size 0 max 16384 2024-03-14T10:03:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.125+0000 7f81f4bd6b80 1 Processor -- start 2024-03-14T10:03:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.125+0000 7f81f4bd6b80 1 -- start start 2024-03-14T10:03:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.126+0000 7f81f4bd6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db55700 0x560f2db6b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.126+0000 7f81f4bd6b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db56020 0x560f2db59e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.126+0000 7f81f4bd6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db5ff40 0x560f2db5a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.126+0000 7f81f4bd6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2db5e260 con 0x560f2db55700 2024-03-14T10:03:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.126+0000 7f81f4bd6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2db51940 con 0x560f2db5ff40 2024-03-14T10:03:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.126+0000 7f81f4bd6b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2db56b70 con 0x560f2db56020 2024-03-14T10:03:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.126+0000 7f81d9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db55700 0x560f2db6b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.126+0000 7f81d9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db55700 0x560f2db6b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37564/0 (socket says 172.21.15.78:37564) 2024-03-14T10:03:30.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.126+0000 7f81d9e00700 1 -- 172.21.15.78:0/1313712509 learned_addr learned my addr 172.21.15.78:0/1313712509 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:30.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.126+0000 7f81d3e00700 1 --2- 172.21.15.78:0/1313712509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db56020 0x560f2db59e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.126+0000 7f81da800700 1 --2- 172.21.15.78:0/1313712509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db5ff40 0x560f2db5a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.126+0000 7f81d9e00700 1 -- 172.21.15.78:0/1313712509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db56020 msgr2=0x560f2db59e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:30.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.126+0000 7f81d9e00700 1 --2- 172.21.15.78:0/1313712509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db56020 0x560f2db59e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.126+0000 7f81d9e00700 1 -- 172.21.15.78:0/1313712509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db5ff40 msgr2=0x560f2db5a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:30.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.126+0000 7f81d9e00700 1 --2- 172.21.15.78:0/1313712509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db5ff40 0x560f2db5a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.126+0000 7f81d9e00700 1 -- 172.21.15.78:0/1313712509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f2d8d5be0 con 0x560f2db55700 2024-03-14T10:03:30.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.127+0000 7f81d9e00700 1 --2- 172.21.15.78:0/1313712509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db55700 0x560f2db6b5e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f81cc005c20 tx=0x7f81cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1017557017775488 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:30.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.127+0000 7f81cbe00700 1 -- 172.21.15.78:0/1313712509 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc0039e0 con 0x560f2db55700 2024-03-14T10:03:30.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.127+0000 7f81cbe00700 1 -- 172.21.15.78:0/1313712509 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f81cc018040 con 0x560f2db55700 2024-03-14T10:03:30.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.127+0000 7f81cbe00700 1 -- 172.21.15.78:0/1313712509 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc010820 con 0x560f2db55700 2024-03-14T10:03:30.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.127+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/1313712509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db55700 msgr2=0x560f2db6b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.127+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/1313712509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db55700 0x560f2db6b5e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f81cc005c20 tx=0x7f81cc004750 comp rx=0 tx=0).stop 2024-03-14T10:03:30.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.128+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/1313712509 shutdown_connections 2024-03-14T10:03:30.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.128+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/1313712509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db5ff40 0x560f2db5a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.128+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/1313712509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db55700 0x560f2db6b5e0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.128+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/1313712509 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db56020 0x560f2db59e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.128+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/1313712509 >> 172.21.15.78:0/1313712509 conn(0x560f2dacb2a0 msgr2=0x560f2db6aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:30.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.128+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/1313712509 shutdown_connections 2024-03-14T10:03:30.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.128+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/1313712509 wait complete. 2024-03-14T10:03:30.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.128+0000 7f81f4bd6b80 1 -- start start 2024-03-14T10:03:30.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.128+0000 7f81f4bd6b80 1 Processor -- start 2024-03-14T10:03:30.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81f4bd6b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db55700 0x560f2dbbb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81f4bd6b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db56020 0x560f2dbbbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81f4bd6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db5ff40 0x560f2db65c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81f4bd6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2db66310 con 0x560f2db5ff40 2024-03-14T10:03:30.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81f4bd6b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2db66480 con 0x560f2db56020 2024-03-14T10:03:30.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81f4bd6b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2db66640 con 0x560f2db55700 2024-03-14T10:03:30.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81f4bd6b80 10 client.0 _ll_register_callbacks cb 0x7f81cc009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:03:30.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81d9e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db55700 0x560f2dbbb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81d9e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db55700 0x560f2dbbb6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:52218/0 (socket says 172.21.15.78:52218) 2024-03-14T10:03:30.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81d9e00700 1 -- 172.21.15.78:0/645341728 learned_addr learned my addr 172.21.15.78:0/645341728 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:30.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81da800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db5ff40 0x560f2db65c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81d3e00700 1 --2- 172.21.15.78:0/645341728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db56020 0x560f2dbbbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81d9e00700 1 -- 172.21.15.78:0/645341728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db56020 msgr2=0x560f2dbbbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:30.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81d9e00700 1 --2- 172.21.15.78:0/645341728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db56020 0x560f2dbbbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81d9e00700 1 -- 172.21.15.78:0/645341728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db5ff40 msgr2=0x560f2db65c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:30.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81d9e00700 1 --2- 172.21.15.78:0/645341728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db5ff40 0x560f2db65c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81d9e00700 1 -- 172.21.15.78:0/645341728 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81cc009d90 con 0x560f2db55700 2024-03-14T10:03:30.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.129+0000 7f81d3e00700 1 --2- 172.21.15.78:0/645341728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db56020 0x560f2dbbbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:03:30.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.130+0000 7f81d9e00700 1 --2- 172.21.15.78:0/645341728 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db55700 0x560f2dbbb6e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f81cc005800 tx=0x7f81cc005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=832584291b7b80eb server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:30.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.130+0000 7f81bfe00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:03:30.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.130+0000 7f81bfe00700 1 -- 172.21.15.78:0/645341728 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc01da10 con 0x560f2db55700 2024-03-14T10:03:30.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.130+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/645341728 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f81c4072f90 con 0x560f2db55700 2024-03-14T10:03:30.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.130+0000 7f81bfe00700 1 -- 172.21.15.78:0/645341728 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f81cc01db70 con 0x560f2db55700 2024-03-14T10:03:30.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.130+0000 7f81bfe00700 1 -- 172.21.15.78:0/645341728 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81cc003780 con 0x560f2db55700 2024-03-14T10:03:30.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.130+0000 7f81f4bd6b80 10 client.27541 fetch_fsmap learned FSMap version 25 2024-03-14T10:03:30.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.130+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/645341728 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x560f2da6df20 con 0x560f2db55700 2024-03-14T10:03:30.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.130+0000 7f81bfe00700 1 -- 172.21.15.78:0/645341728 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f81c4072f90 con 0x560f2db55700 2024-03-14T10:03:30.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.131+0000 7f81bfe00700 1 -- 172.21.15.78:0/645341728 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f81cc024070 con 0x560f2db55700 2024-03-14T10:03:30.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.131+0000 7f81f4bd6b80 10 client.27541 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:03:30.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.131+0000 7f81f4bd6b80 10 client.27541 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:03:30.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.131+0000 7f81f4bd6b80 20 client.27541 populate_metadata read hostname 'smithi078' 2024-03-14T10:03:30.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.131+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/645341728 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x560f2dbc3560 0x560f2dbc5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.131+0000 7f81f4bd6b80 4 client.27541 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T10:03:30.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.131+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/645341728 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x560f2d8f6d70 con 0x560f2dbc3560 2024-03-14T10:03:30.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.131+0000 7f81d3e00700 1 --2- 172.21.15.78:0/645341728 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x560f2dbc3560 0x560f2dbc5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.132+0000 7f81bfe00700 10 client.27541 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:03:30.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.132+0000 7f81d3e00700 1 --2- 172.21.15.78:0/645341728 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x560f2dbc3560 0x560f2dbc5980 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7abce49623bf6ed server_cookie=5f16bd651f7f17fa in_seq=0 out_seq=0 2024-03-14T10:03:30.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.133+0000 7f81bfe00700 1 -- 172.21.15.78:0/645341728 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x560f2d8f6d70 con 0x560f2dbc3560 2024-03-14T10:03:30.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.133+0000 7f81bfe00700 10 client.27541 handle_command_reply: tid=0 2024-03-14T10:03:30.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.133+0000 7f81f4bd6b80 1 client.27541 shutdown 2024-03-14T10:03:30.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.133+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/645341728 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db55700 msgr2=0x560f2dbbb6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.133+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/645341728 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db55700 0x560f2dbbb6e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f81cc005800 tx=0x7f81cc005830 comp rx=0 tx=0).stop 2024-03-14T10:03:30.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.133+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/645341728 shutdown_connections 2024-03-14T10:03:30.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.133+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/645341728 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f2db5ff40 0x560f2db65c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.133+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/645341728 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x560f2db55700 0x560f2dbbb6e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.133+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/645341728 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x560f2dbc3560 0x560f2dbc5980 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.133+0000 7f81f4bd6b80 1 --2- 172.21.15.78:0/645341728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560f2db56020 0x560f2dbbbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.133+0000 7f81bfe00700 0 client.27541 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:03:30.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.133+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/645341728 >> 172.21.15.78:0/645341728 conn(0x560f2dacb2a0 msgr2=0x560f2db54e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:30.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.133+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/645341728 shutdown_connections 2024-03-14T10:03:30.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.133+0000 7f81f4bd6b80 1 -- 172.21.15.78:0/645341728 wait complete. 2024-03-14T10:03:30.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.134+0000 7f81f4bd6b80 20 client.27541 trim_cache size 0 max 16384 2024-03-14T10:03:30.137 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:03:30.138 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T10:03:30.138 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "faa6dad0-72a5-4719-99af-5f24269a3a29", 2024-03-14T10:03:30.138 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T10:03:30.138 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:03:30.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.134+0000 7f81e1000700 1 -- 172.21.15.78:0/2795429634 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f81c00a0c00 msgr2=0x7f81c00a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.134+0000 7f81e1000700 1 --2- 172.21.15.78:0/2795429634 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f81c00a0c00 0x7f81c00a30c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f81d4003fd0 tx=0x7f81d4003a00 comp rx=0 tx=0).stop 2024-03-14T10:03:30.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.134+0000 7f81e1000700 1 -- 172.21.15.78:0/2795429634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81dc106050 msgr2=0x7f81dc19fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.134+0000 7f81e1000700 1 --2- 172.21.15.78:0/2795429634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81dc106050 0x7f81dc19fa70 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f81c4000c00 tx=0x7f81c4005740 comp rx=0 tx=0).stop 2024-03-14T10:03:30.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.134+0000 7f81e1000700 1 -- 172.21.15.78:0/2795429634 shutdown_connections 2024-03-14T10:03:30.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.134+0000 7f81e1000700 1 --2- 172.21.15.78:0/2795429634 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f81c00a0c00 0x7f81c00a30c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.134+0000 7f81e1000700 1 --2- 172.21.15.78:0/2795429634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f81dc199660 0x7f81dc199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.134+0000 7f81e1000700 1 --2- 172.21.15.78:0/2795429634 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f81dc100f30 0x7f81dc19f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.134+0000 7f81e1000700 1 --2- 172.21.15.78:0/2795429634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81dc106050 0x7f81dc19fa70 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.134+0000 7f81e1000700 1 -- 172.21.15.78:0/2795429634 >> 172.21.15.78:0/2795429634 conn(0x7f81dc0fc7b0 msgr2=0x7f81dc0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:30.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.134+0000 7f81e1000700 1 -- 172.21.15.78:0/2795429634 shutdown_connections 2024-03-14T10:03:30.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.134+0000 7f81e1000700 1 -- 172.21.15.78:0/2795429634 wait complete. 2024-03-14T10:03:30.153 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:03:30.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.348+0000 7f5003200700 1 -- 172.21.15.78:0/1431075474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ffc1018d0 msgr2=0x7f4ffc105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.348+0000 7f5003200700 1 --2- 172.21.15.78:0/1431075474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ffc1018d0 0x7f4ffc105910 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8009990 tx=0x7f4fe8009ca0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.348+0000 7f5003200700 1 -- 172.21.15.78:0/1431075474 shutdown_connections 2024-03-14T10:03:30.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.348+0000 7f5003200700 1 --2- 172.21.15.78:0/1431075474 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ffc106040 0x7f4ffc10dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.348+0000 7f5003200700 1 --2- 172.21.15.78:0/1431075474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ffc100f20 0x7f4ffc101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.348+0000 7f5003200700 1 --2- 172.21.15.78:0/1431075474 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ffc1018d0 0x7f4ffc105910 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.348+0000 7f5003200700 1 -- 172.21.15.78:0/1431075474 >> 172.21.15.78:0/1431075474 conn(0x7f4ffc0fc7a0 msgr2=0x7f4ffc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:30.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.349+0000 7f5003200700 1 -- 172.21.15.78:0/1431075474 shutdown_connections 2024-03-14T10:03:30.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.349+0000 7f5003200700 1 -- 172.21.15.78:0/1431075474 wait complete. 2024-03-14T10:03:30.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.349+0000 7f5003200700 1 Processor -- start 2024-03-14T10:03:30.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.350+0000 7f5003200700 1 -- start start 2024-03-14T10:03:30.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.350+0000 7f5003200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ffc100f20 0x7f4ffc197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.350+0000 7f5003200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ffc106040 0x7f4ffc197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.350+0000 7f5003200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ffc19bf90 0x7f4ffc19c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.350+0000 7f5003200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc19ca10 con 0x7f4ffc19bf90 2024-03-14T10:03:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.350+0000 7f5003200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc19cb80 con 0x7f4ffc106040 2024-03-14T10:03:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.350+0000 7f5003200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc19cd40 con 0x7f4ffc100f20 2024-03-14T10:03:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.350+0000 7f5001200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ffc19bf90 0x7f4ffc19c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.350+0000 7f5001200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ffc19bf90 0x7f4ffc19c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37596/0 (socket says 172.21.15.78:37596) 2024-03-14T10:03:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.350+0000 7f5001200700 1 -- 172.21.15.78:0/83119184 learned_addr learned my addr 172.21.15.78:0/83119184 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.350+0000 7f4ffbe00700 1 --2- 172.21.15.78:0/83119184 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ffc100f20 0x7f4ffc197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.350+0000 7f4ffb400700 1 --2- 172.21.15.78:0/83119184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ffc106040 0x7f4ffc197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.351+0000 7f5001200700 1 -- 172.21.15.78:0/83119184 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ffc100f20 msgr2=0x7f4ffc197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.351+0000 7f5001200700 1 --2- 172.21.15.78:0/83119184 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ffc100f20 0x7f4ffc197700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.351+0000 7f5001200700 1 -- 172.21.15.78:0/83119184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ffc106040 msgr2=0x7f4ffc197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.351+0000 7f5001200700 1 --2- 172.21.15.78:0/83119184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ffc106040 0x7f4ffc197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.351+0000 7f5001200700 1 -- 172.21.15.78:0/83119184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe80095f0 con 0x7f4ffc19bf90 2024-03-14T10:03:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.351+0000 7f5001200700 1 --2- 172.21.15.78:0/83119184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ffc19bf90 0x7f4ffc19c410 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400ef30 tx=0x7f4ff400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.351+0000 7f4ff8c00700 1 -- 172.21.15.78:0/83119184 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4018070 con 0x7f4ffc19bf90 2024-03-14T10:03:30.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.351+0000 7f5003200700 1 -- 172.21.15.78:0/83119184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ffc1a4240 con 0x7f4ffc19bf90 2024-03-14T10:03:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.351+0000 7f4ff8c00700 1 -- 172.21.15.78:0/83119184 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ff400f040 con 0x7f4ffc19bf90 2024-03-14T10:03:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.351+0000 7f4ff8c00700 1 -- 172.21.15.78:0/83119184 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff400cd00 con 0x7f4ffc19bf90 2024-03-14T10:03:30.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.351+0000 7f5003200700 1 -- 172.21.15.78:0/83119184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ffc1a4770 con 0x7f4ffc19bf90 2024-03-14T10:03:30.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.356+0000 7f4ff8c00700 1 -- 172.21.15.78:0/83119184 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f4ff4003a00 con 0x7f4ffc19bf90 2024-03-14T10:03:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.357+0000 7f5016e09b80 1 Processor -- start 2024-03-14T10:03:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.357+0000 7f5016e09b80 1 -- start start 2024-03-14T10:03:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.357+0000 7f4ff8c00700 1 --2- 172.21.15.78:0/83119184 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4fe00a0c80 0x7f4fe00a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.357+0000 7f5016e09b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473a680 0x55e87473aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.357+0000 7f5016e09b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473afa0 0x55e874748d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.357+0000 7f5016e09b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e87474b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.357+0000 7f5016e09b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e874693990 con 0x55e8747492d0 2024-03-14T10:03:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.357+0000 7f5016e09b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e874691350 con 0x55e87473afa0 2024-03-14T10:03:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.357+0000 7f5016e09b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e87468cc40 con 0x55e87473a680 2024-03-14T10:03:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.357+0000 7f4ff8c00700 1 -- 172.21.15.78:0/83119184 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f4ff40cc240 con 0x7f4ffc19bf90 2024-03-14T10:03:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.357+0000 7f4ffbe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473a680 0x55e87473aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.357+0000 7f5001200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e87474b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.357+0000 7f4ffbe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473a680 0x55e87473aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:52240/0 (socket says 172.21.15.78:52240) 2024-03-14T10:03:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.357+0000 7f4ffbe00700 1 -- 172.21.15.78:0/822093342 learned_addr learned my addr 172.21.15.78:0/822093342 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.357+0000 7f5001200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e87474b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37602/0 (socket says 172.21.15.78:37602) 2024-03-14T10:03:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.357+0000 7f4ffb400700 1 --2- 172.21.15.78:0/83119184 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4fe00a0c80 0x7f4fe00a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.358+0000 7f4ffb400700 1 --2- 172.21.15.78:0/822093342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473afa0 0x55e874748d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.358+0000 7f5001200700 1 -- 172.21.15.78:0/822093342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473a680 msgr2=0x55e87473aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:30.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.358+0000 7f5001200700 1 --2- 172.21.15.78:0/822093342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473a680 0x55e87473aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.358+0000 7f5001200700 1 -- 172.21.15.78:0/822093342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473afa0 msgr2=0x55e874748d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:30.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.358+0000 7f5001200700 1 --2- 172.21.15.78:0/822093342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473afa0 0x55e874748d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.358+0000 7f5001200700 1 -- 172.21.15.78:0/822093342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e87450e3b0 con 0x55e8747492d0 2024-03-14T10:03:30.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.358+0000 7f4ffbe00700 1 --2- 172.21.15.78:0/822093342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473a680 0x55e87473aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:03:30.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.358+0000 7f5001200700 1 --2- 172.21.15.78:0/822093342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e87474b6c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40c95c0 tx=0x7f4ff40990a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc08809ab46ec9c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:30.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.358+0000 7f4ffb400700 1 --2- 172.21.15.78:0/83119184 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4fe00a0c80 0x7f4fe00a3140 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc198c00 tx=0x7f4fe800b5b0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:30.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.358+0000 7f4ff1600700 1 -- 172.21.15.78:0/822093342 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4004ad0 con 0x55e8747492d0 2024-03-14T10:03:30.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.358+0000 7f4ff1600700 1 -- 172.21.15.78:0/822093342 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ff401b320 con 0x55e8747492d0 2024-03-14T10:03:30.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.358+0000 7f4ff1600700 1 -- 172.21.15.78:0/822093342 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff40a1640 con 0x55e8747492d0 2024-03-14T10:03:30.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.359+0000 7f5016e09b80 1 -- 172.21.15.78:0/822093342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 msgr2=0x55e87474b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.359+0000 7f5016e09b80 1 --2- 172.21.15.78:0/822093342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e87474b6c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40c95c0 tx=0x7f4ff40990a0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.359+0000 7f5016e09b80 1 -- 172.21.15.78:0/822093342 shutdown_connections 2024-03-14T10:03:30.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.359+0000 7f5016e09b80 1 --2- 172.21.15.78:0/822093342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e87474b6c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.359+0000 7f5016e09b80 1 --2- 172.21.15.78:0/822093342 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473a680 0x55e87473aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.359+0000 7f5016e09b80 1 --2- 172.21.15.78:0/822093342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473afa0 0x55e874748d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.359+0000 7f5016e09b80 1 -- 172.21.15.78:0/822093342 >> 172.21.15.78:0/822093342 conn(0x55e8746b0220 msgr2=0x55e874735aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:30.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.359+0000 7f5016e09b80 1 -- 172.21.15.78:0/822093342 shutdown_connections 2024-03-14T10:03:30.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.359+0000 7f5016e09b80 1 -- 172.21.15.78:0/822093342 wait complete. 2024-03-14T10:03:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.360+0000 7f5016e09b80 1 -- start start 2024-03-14T10:03:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.360+0000 7f5016e09b80 1 Processor -- start 2024-03-14T10:03:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f5016e09b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e87473a680 0x55e874748cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f5016e09b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473afa0 0x55e874745fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f5016e09b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e8747492d0 0x55e874746630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f5016e09b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e874628d80 con 0x55e87473a680 2024-03-14T10:03:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f5016e09b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e874628a20 con 0x55e87473afa0 2024-03-14T10:03:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f5016e09b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8745106a0 con 0x55e8747492d0 2024-03-14T10:03:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f5016e09b80 10 client.0 _ll_register_callbacks cb 0x55e8744943a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:03:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f4ffbe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e87473a680 0x55e874748cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f4ffbe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e87473a680 0x55e874748cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37614/0 (socket says 172.21.15.78:37614) 2024-03-14T10:03:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f4ffbe00700 1 -- 172.21.15.78:0/1575034213 learned_addr learned my addr 172.21.15.78:0/1575034213 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f4ffb400700 1 --2- 172.21.15.78:0/1575034213 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473afa0 0x55e874745fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f5001200700 1 --2- 172.21.15.78:0/1575034213 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e8747492d0 0x55e874746630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f4ffbe00700 1 -- 172.21.15.78:0/1575034213 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e8747492d0 msgr2=0x55e874746630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f4ffbe00700 1 --2- 172.21.15.78:0/1575034213 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e8747492d0 0x55e874746630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f4ffbe00700 1 -- 172.21.15.78:0/1575034213 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473afa0 msgr2=0x55e874745fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f4ffbe00700 1 --2- 172.21.15.78:0/1575034213 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473afa0 0x55e874745fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.361+0000 7f4ffbe00700 1 -- 172.21.15.78:0/1575034213 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec0099e0 con 0x55e87473a680 2024-03-14T10:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.362+0000 7f4ffbe00700 1 --2- 172.21.15.78:0/1575034213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e87473a680 0x55e874748cc0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4fec012a10 tx=0x7f4fec012dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ee2852577d4833 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.362+0000 7f4fe6000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.362+0000 7f4fe6000700 1 -- 172.21.15.78:0/1575034213 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec017070 con 0x55e87473a680 2024-03-14T10:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.362+0000 7f4fe6000700 1 -- 172.21.15.78:0/1575034213 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f4fec004050 con 0x55e87473a680 2024-03-14T10:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.362+0000 7f5016e09b80 1 -- 172.21.15.78:0/1575034213 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e87472d720 con 0x55e87473a680 2024-03-14T10:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.362+0000 7f4fe6000700 1 -- 172.21.15.78:0/1575034213 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fec004210 con 0x55e87473a680 2024-03-14T10:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.362+0000 7f5016e09b80 10 client.17508 fetch_fsmap learned FSMap version 25 2024-03-14T10:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.362+0000 7f5016e09b80 1 -- 172.21.15.78:0/1575034213 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f4ffc1a4770 con 0x55e87473a680 2024-03-14T10:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.362+0000 7f4fe6000700 1 -- 172.21.15.78:0/1575034213 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e87472d720 con 0x55e87473a680 2024-03-14T10:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.363+0000 7f4fe6000700 1 -- 172.21.15.78:0/1575034213 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4fec003880 con 0x55e87473a680 2024-03-14T10:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.363+0000 7f5016e09b80 10 client.17508 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:03:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.363+0000 7f5016e09b80 10 client.17508 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.363+0000 7f5016e09b80 20 client.17508 populate_metadata read hostname 'smithi078' 2024-03-14T10:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.363+0000 7f5016e09b80 1 --2- 172.21.15.78:0/1575034213 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e874758e10 0x55e87475b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.363+0000 7f5016e09b80 4 client.17508 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.363+0000 7f5016e09b80 1 -- 172.21.15.78:0/1575034213 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e874685700 con 0x55e874758e10 2024-03-14T10:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.363+0000 7f4ffb400700 1 --2- 172.21.15.78:0/1575034213 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e874758e10 0x55e87475b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.364+0000 7f4fe6000700 10 client.17508 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.364+0000 7f4ffb400700 1 --2- 172.21.15.78:0/1575034213 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e874758e10 0x55e87475b200 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=370eb4bbf0bd2e1e server_cookie=1e5e17b36d541530 in_seq=0 out_seq=0 2024-03-14T10:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.364+0000 7f4fe6000700 1 -- 172.21.15.78:0/1575034213 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55e874685700 con 0x55e874758e10 2024-03-14T10:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.364+0000 7f4fe6000700 10 client.17508 handle_command_reply: tid=0 2024-03-14T10:03:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.365+0000 7f5016e09b80 1 client.17508 shutdown 2024-03-14T10:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.365+0000 7f5016e09b80 1 -- 172.21.15.78:0/1575034213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e87473a680 msgr2=0x55e874748cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.365+0000 7f5016e09b80 1 --2- 172.21.15.78:0/1575034213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e87473a680 0x55e874748cc0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4fec012a10 tx=0x7f4fec012dd0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.365+0000 7f5016e09b80 1 -- 172.21.15.78:0/1575034213 shutdown_connections 2024-03-14T10:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.365+0000 7f5016e09b80 1 --2- 172.21.15.78:0/1575034213 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e8747492d0 0x55e874746630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.365+0000 7f5016e09b80 1 --2- 172.21.15.78:0/1575034213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e87473a680 0x55e874748cc0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.365+0000 7f5016e09b80 1 --2- 172.21.15.78:0/1575034213 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e874758e10 0x55e87475b200 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.365+0000 7f5016e09b80 1 --2- 172.21.15.78:0/1575034213 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473afa0 0x55e874745fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.365+0000 7f4fe6000700 0 client.17508 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:03:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.365+0000 7f5016e09b80 1 -- 172.21.15.78:0/1575034213 >> 172.21.15.78:0/1575034213 conn(0x55e8746b0220 msgr2=0x55e874735690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:30.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.365+0000 7f5016e09b80 1 -- 172.21.15.78:0/1575034213 shutdown_connections 2024-03-14T10:03:30.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.365+0000 7f5016e09b80 1 -- 172.21.15.78:0/1575034213 wait complete. 2024-03-14T10:03:30.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.365+0000 7f5016e09b80 20 client.17508 trim_cache size 0 max 16384 2024-03-14T10:03:30.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.382+0000 7f5016e09b80 1 Processor -- start 2024-03-14T10:03:30.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.382+0000 7f5016e09b80 1 -- start start 2024-03-14T10:03:30.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.382+0000 7f5016e09b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473a680 0x55e874735c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.382+0000 7f5016e09b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473afa0 0x55e87474fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.382+0000 7f5016e09b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e874744880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.382+0000 7f5016e09b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e87473ade0 con 0x55e8747492d0 2024-03-14T10:03:30.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.382+0000 7f5016e09b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e87475ff20 con 0x55e87473afa0 2024-03-14T10:03:30.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.382+0000 7f5016e09b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e874758bf0 con 0x55e87473a680 2024-03-14T10:03:30.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.383+0000 7f5001200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e874744880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.383+0000 7f4ffbe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473a680 0x55e874735c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.383+0000 7f5001200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e874744880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37628/0 (socket says 172.21.15.78:37628) 2024-03-14T10:03:30.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.383+0000 7f5001200700 1 -- 172.21.15.78:0/866705252 learned_addr learned my addr 172.21.15.78:0/866705252 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:30.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.383+0000 7f4ffb400700 1 --2- 172.21.15.78:0/866705252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473afa0 0x55e87474fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.383+0000 7f5001200700 1 -- 172.21.15.78:0/866705252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473a680 msgr2=0x55e874735c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:30.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.383+0000 7f5001200700 1 --2- 172.21.15.78:0/866705252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473a680 0x55e874735c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.383+0000 7f5001200700 1 -- 172.21.15.78:0/866705252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473afa0 msgr2=0x55e87474fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:30.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.383+0000 7f5001200700 1 --2- 172.21.15.78:0/866705252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473afa0 0x55e87474fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.383+0000 7f5001200700 1 -- 172.21.15.78:0/866705252 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e8744bab60 con 0x55e8747492d0 2024-03-14T10:03:30.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.383+0000 7f4ffb400700 1 --2- 172.21.15.78:0/866705252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473afa0 0x55e87474fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:03:30.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.383+0000 7f5001200700 1 --2- 172.21.15.78:0/866705252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e874744880 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40c9b40 tx=0x7f4ff401b490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab834f93c5ecfef server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:30.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.383+0000 7f4ff2000700 1 -- 172.21.15.78:0/866705252 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff40af020 con 0x55e8747492d0 2024-03-14T10:03:30.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.384+0000 7f4ff2000700 1 -- 172.21.15.78:0/866705252 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ff4004ad0 con 0x55e8747492d0 2024-03-14T10:03:30.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.384+0000 7f4ff2000700 1 -- 172.21.15.78:0/866705252 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff409d080 con 0x55e8747492d0 2024-03-14T10:03:30.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.384+0000 7f5016e09b80 1 -- 172.21.15.78:0/866705252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 msgr2=0x55e874744880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.384+0000 7f5016e09b80 1 --2- 172.21.15.78:0/866705252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e874744880 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40c9b40 tx=0x7f4ff401b490 comp rx=0 tx=0).stop 2024-03-14T10:03:30.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.384+0000 7f5016e09b80 1 -- 172.21.15.78:0/866705252 shutdown_connections 2024-03-14T10:03:30.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.384+0000 7f5016e09b80 1 --2- 172.21.15.78:0/866705252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e874744880 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.384+0000 7f5016e09b80 1 --2- 172.21.15.78:0/866705252 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473a680 0x55e874735c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.384+0000 7f5016e09b80 1 --2- 172.21.15.78:0/866705252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473afa0 0x55e87474fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.384+0000 7f5016e09b80 1 -- 172.21.15.78:0/866705252 >> 172.21.15.78:0/866705252 conn(0x55e8746b0220 msgr2=0x55e874734180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:30.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.384+0000 7f5016e09b80 1 -- 172.21.15.78:0/866705252 shutdown_connections 2024-03-14T10:03:30.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.384+0000 7f5016e09b80 1 -- 172.21.15.78:0/866705252 wait complete. 2024-03-14T10:03:30.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.385+0000 7f5016e09b80 1 -- start start 2024-03-14T10:03:30.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.385+0000 7f5016e09b80 1 Processor -- start 2024-03-14T10:03:30.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.385+0000 7f5016e09b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473a680 0x55e87474f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.385+0000 7f5016e09b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473afa0 0x55e87474fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.385+0000 7f5016e09b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e87474dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.385+0000 7f5016e09b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e87474e300 con 0x55e8747492d0 2024-03-14T10:03:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.385+0000 7f5016e09b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e87474e470 con 0x55e87473a680 2024-03-14T10:03:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.385+0000 7f5016e09b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e87474e630 con 0x55e87473afa0 2024-03-14T10:03:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.385+0000 7f5016e09b80 10 client.0 _ll_register_callbacks cb 0x7f4fec0124e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:03:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.385+0000 7f5001200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e87474dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.386+0000 7f5001200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e87474dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:37636/0 (socket says 172.21.15.78:37636) 2024-03-14T10:03:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.386+0000 7f5001200700 1 -- 172.21.15.78:0/3547547500 learned_addr learned my addr 172.21.15.78:0/3547547500 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:03:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.386+0000 7f4ffb400700 1 --2- 172.21.15.78:0/3547547500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473afa0 0x55e87474fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.386+0000 7f4ffbe00700 1 --2- 172.21.15.78:0/3547547500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473a680 0x55e87474f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.386+0000 7f5001200700 1 -- 172.21.15.78:0/3547547500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473afa0 msgr2=0x55e87474fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.386+0000 7f5001200700 1 --2- 172.21.15.78:0/3547547500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473afa0 0x55e87474fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.386+0000 7f5001200700 1 -- 172.21.15.78:0/3547547500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473a680 msgr2=0x55e87474f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:03:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.386+0000 7f5001200700 1 --2- 172.21.15.78:0/3547547500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473a680 0x55e87474f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.386+0000 7f5001200700 1 -- 172.21.15.78:0/3547547500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e87450e3b0 con 0x55e8747492d0 2024-03-14T10:03:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.386+0000 7f4ffb400700 1 --2- 172.21.15.78:0/3547547500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473afa0 0x55e87474fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:03:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.386+0000 7f4ffbe00700 1 --2- 172.21.15.78:0/3547547500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473a680 0x55e87474f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:03:30.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.386+0000 7f5001200700 1 --2- 172.21.15.78:0/3547547500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e87474dd20 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4095750 tx=0x7f4ff4099300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ea45f73788455d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:03:30.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.386+0000 7f4fe6000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:03:30.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.386+0000 7f4fe6000700 1 -- 172.21.15.78:0/3547547500 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff40af030 con 0x55e8747492d0 2024-03-14T10:03:30.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.386+0000 7f4fe6000700 1 -- 172.21.15.78:0/3547547500 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f4ff409d080 con 0x55e8747492d0 2024-03-14T10:03:30.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.386+0000 7f5016e09b80 1 -- 172.21.15.78:0/3547547500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ff401b320 con 0x55e8747492d0 2024-03-14T10:03:30.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.387+0000 7f4fe6000700 1 -- 172.21.15.78:0/3547547500 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ff400cee0 con 0x55e8747492d0 2024-03-14T10:03:30.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.387+0000 7f5016e09b80 10 client.17514 fetch_fsmap learned FSMap version 25 2024-03-14T10:03:30.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.387+0000 7f5016e09b80 1 -- 172.21.15.78:0/3547547500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e874652ea0 con 0x55e8747492d0 2024-03-14T10:03:30.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.387+0000 7f4fe6000700 1 -- 172.21.15.78:0/3547547500 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ff40a5080 con 0x55e8747492d0 2024-03-14T10:03:30.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.387+0000 7f4fe6000700 1 -- 172.21.15.78:0/3547547500 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4ff409c070 con 0x55e8747492d0 2024-03-14T10:03:30.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.387+0000 7f5016e09b80 10 client.17514 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:03:30.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.387+0000 7f5016e09b80 10 client.17514 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:03:30.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.387+0000 7f5016e09b80 20 client.17514 populate_metadata read hostname 'smithi078' 2024-03-14T10:03:30.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.387+0000 7f5016e09b80 1 --2- 172.21.15.78:0/3547547500 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e8747ab700 0x55e8747adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:03:30.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.387+0000 7f5016e09b80 4 client.17514 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:03:30.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.387+0000 7f5016e09b80 1 -- 172.21.15.78:0/3547547500 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e8744dbcf0 con 0x55e8747ab700 2024-03-14T10:03:30.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.387+0000 7f4ffbe00700 1 --2- 172.21.15.78:0/3547547500 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e8747ab700 0x55e8747adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:03:30.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.388+0000 7f4fe6000700 10 client.17514 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:03:30.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.388+0000 7f4ffbe00700 1 --2- 172.21.15.78:0/3547547500 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e8747ab700 0x55e8747adaf0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa66be03d4f1b53 server_cookie=807b9288dfb6844a in_seq=0 out_seq=0 2024-03-14T10:03:30.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.388+0000 7f4fe6000700 1 -- 172.21.15.78:0/3547547500 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55e8744dbcf0 con 0x55e8747ab700 2024-03-14T10:03:30.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.388+0000 7f4fe6000700 10 client.17514 handle_command_reply: tid=0 2024-03-14T10:03:30.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.388+0000 7f5016e09b80 1 client.17514 shutdown 2024-03-14T10:03:30.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.388+0000 7f5016e09b80 1 -- 172.21.15.78:0/3547547500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 msgr2=0x55e87474dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.388+0000 7f5016e09b80 1 --2- 172.21.15.78:0/3547547500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e87474dd20 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4095750 tx=0x7f4ff4099300 comp rx=0 tx=0).stop 2024-03-14T10:03:30.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.388+0000 7f5016e09b80 1 -- 172.21.15.78:0/3547547500 shutdown_connections 2024-03-14T10:03:30.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.388+0000 7f5016e09b80 1 --2- 172.21.15.78:0/3547547500 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55e8747ab700 0x55e8747adaf0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.388+0000 7f5016e09b80 1 --2- 172.21.15.78:0/3547547500 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55e87473a680 0x55e87474f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.388+0000 7f5016e09b80 1 --2- 172.21.15.78:0/3547547500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e8747492d0 0x55e87474dd20 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.388+0000 7f4fe6000700 0 client.17514 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:03:30.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.388+0000 7f5016e09b80 1 --2- 172.21.15.78:0/3547547500 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55e87473afa0 0x55e87474fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.388+0000 7f5016e09b80 1 -- 172.21.15.78:0/3547547500 >> 172.21.15.78:0/3547547500 conn(0x55e8746b0220 msgr2=0x55e8747446f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:30.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.389+0000 7f5016e09b80 1 -- 172.21.15.78:0/3547547500 shutdown_connections 2024-03-14T10:03:30.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.389+0000 7f5016e09b80 1 -- 172.21.15.78:0/3547547500 wait complete. 2024-03-14T10:03:30.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.389+0000 7f5016e09b80 20 client.17514 trim_cache size 0 max 16384 2024-03-14T10:03:30.394 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:03:30.394 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-14T10:03:30.394 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T10:03:30.394 INFO:teuthology.orchestra.run.smithi078.stdout: "faa6dad0-72a5-4719-99af-5f24269a3a29": { 2024-03-14T10:03:30.394 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T10:03:30.395 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "faa6dad0-72a5-4719-99af-5f24269a3a29", 2024-03-14T10:03:30.395 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T10:03:30.395 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:03:30.395 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:03:30.395 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:03:30.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.389+0000 7f5003200700 1 -- 172.21.15.78:0/83119184 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4fe00a0c80 msgr2=0x7f4fe00a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.389+0000 7f5003200700 1 --2- 172.21.15.78:0/83119184 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4fe00a0c80 0x7f4fe00a3140 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc198c00 tx=0x7f4fe800b5b0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.389+0000 7f5003200700 1 -- 172.21.15.78:0/83119184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ffc19bf90 msgr2=0x7f4ffc19c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:03:30.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.389+0000 7f5003200700 1 --2- 172.21.15.78:0/83119184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ffc19bf90 0x7f4ffc19c410 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400ef30 tx=0x7f4ff400c3a0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.389+0000 7f5003200700 1 -- 172.21.15.78:0/83119184 shutdown_connections 2024-03-14T10:03:30.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.389+0000 7f5003200700 1 --2- 172.21.15.78:0/83119184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ffc19bf90 0x7f4ffc19c410 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.389+0000 7f5003200700 1 --2- 172.21.15.78:0/83119184 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f4ffc100f20 0x7f4ffc197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.389+0000 7f5003200700 1 --2- 172.21.15.78:0/83119184 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f4fe00a0c80 0x7f4fe00a3140 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.389+0000 7f5003200700 1 --2- 172.21.15.78:0/83119184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4ffc106040 0x7f4ffc197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:03:30.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.389+0000 7f5003200700 1 -- 172.21.15.78:0/83119184 >> 172.21.15.78:0/83119184 conn(0x7f4ffc0fc7a0 msgr2=0x7f4ffc0fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:03:30.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.389+0000 7f5003200700 1 -- 172.21.15.78:0/83119184 shutdown_connections 2024-03-14T10:03:30.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:03:30.390+0000 7f5003200700 1 -- 172.21.15.78:0/83119184 wait complete. 2024-03-14T10:03:30.408 INFO:tasks.cephfs.filesystem:scrub status for tag:faa6dad0-72a5-4719-99af-5f24269a3a29 - {'path': '/', 'tag': 'faa6dad0-72a5-4719-99af-5f24269a3a29', 'options': 'recursive,force'} 2024-03-14T10:03:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:31 smithi152 ceph-mon[48279]: pgmap v289: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:31 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T10:03:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:31 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T10:03:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:31 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T10:03:31.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:31 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T10:03:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:31 smithi132 ceph-mon[47959]: pgmap v289: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:31 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T10:03:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:31 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T10:03:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:31 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T10:03:31.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:31 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T10:03:31.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:31 smithi078 ceph-mon[38143]: pgmap v289: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:31.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:31 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T10:03:31.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:31 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T10:03:31.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:31 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T10:03:31.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:31 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T10:03:33.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:33 smithi152 ceph-mon[48279]: pgmap v290: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:03:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:33 smithi132 ceph-mon[47959]: pgmap v290: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:03:33.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:33 smithi078 ceph-mon[38143]: pgmap v290: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:03:34.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:34 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T10:03:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:34 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T10:03:34.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:34 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T10:03:35.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:35 smithi152 ceph-mon[48279]: pgmap v291: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:03:35.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:35 smithi132 ceph-mon[47959]: pgmap v291: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:03:35.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:35 smithi078 ceph-mon[38143]: pgmap v291: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:03:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:37 smithi152 ceph-mon[48279]: pgmap v292: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:03:37.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:37 smithi132 ceph-mon[47959]: pgmap v292: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:03:37.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:37 smithi078 ceph-mon[38143]: pgmap v292: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:03:39.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:38 smithi152 ceph-mon[48279]: pgmap v293: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:03:39.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:38 smithi132 ceph-mon[47959]: pgmap v293: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:03:39.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:38 smithi078 ceph-mon[38143]: pgmap v293: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:03:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:41 smithi152 ceph-mon[48279]: pgmap v294: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:03:41.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:41 smithi132 ceph-mon[47959]: pgmap v294: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:03:41.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:41 smithi078 ceph-mon[38143]: pgmap v294: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-14T10:03:43.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:43 smithi152 ceph-mon[48279]: pgmap v295: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:03:43.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:43 smithi132 ceph-mon[47959]: pgmap v295: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:03:43.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:43 smithi078 ceph-mon[38143]: pgmap v295: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-14T10:03:45.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:45 smithi152 ceph-mon[48279]: pgmap v296: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:45.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:45 smithi132 ceph-mon[47959]: pgmap v296: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:45.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:45 smithi078 ceph-mon[38143]: pgmap v296: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:47 smithi152 ceph-mon[48279]: pgmap v297: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:03:47.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:47 smithi132 ceph-mon[47959]: pgmap v297: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:03:47.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:47 smithi078 ceph-mon[38143]: pgmap v297: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:03:49.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:48 smithi152 ceph-mon[48279]: pgmap v298: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:49.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:48 smithi132 ceph-mon[47959]: pgmap v298: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:49.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:48 smithi078 ceph-mon[38143]: pgmap v298: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:51 smithi152 ceph-mon[48279]: pgmap v299: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:51.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:51 smithi132 ceph-mon[47959]: pgmap v299: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:51.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:51 smithi078 ceph-mon[38143]: pgmap v299: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:53 smithi152 ceph-mon[48279]: pgmap v300: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:03:53.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:53 smithi132 ceph-mon[47959]: pgmap v300: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:03:53.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:53 smithi078 ceph-mon[38143]: pgmap v300: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:03:55.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:55 smithi152 ceph-mon[48279]: pgmap v301: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:55 smithi132 ceph-mon[47959]: pgmap v301: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:55.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:55 smithi078 ceph-mon[38143]: pgmap v301: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:57.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:57 smithi152 ceph-mon[48279]: pgmap v302: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:03:57.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:57 smithi132 ceph-mon[47959]: pgmap v302: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:03:57.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:57 smithi078 ceph-mon[38143]: pgmap v302: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:03:59.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:03:58 smithi152 ceph-mon[48279]: pgmap v303: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:59.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:03:58 smithi132 ceph-mon[47959]: pgmap v303: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:03:59.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:03:58 smithi078 ceph-mon[38143]: pgmap v303: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:00.409 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:04:00.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.640+0000 7f08f7200700 1 -- 172.21.15.78:0/763076088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08f0106050 msgr2=0x7f08f010dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:00.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.640+0000 7f08f7200700 1 --2- 172.21.15.78:0/763076088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08f0106050 0x7f08f010dbd0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f08e800b0a0 tx=0x7f08e800b3b0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.640+0000 7f08f7200700 1 -- 172.21.15.78:0/763076088 shutdown_connections 2024-03-14T10:04:00.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.640+0000 7f08f7200700 1 --2- 172.21.15.78:0/763076088 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08f0100f30 0x7f08f0101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.640+0000 7f08f7200700 1 --2- 172.21.15.78:0/763076088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08f0106050 0x7f08f010dbd0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.640+0000 7f08f7200700 1 --2- 172.21.15.78:0/763076088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f08f01018e0 0x7f08f0105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.640+0000 7f08f7200700 1 -- 172.21.15.78:0/763076088 >> 172.21.15.78:0/763076088 conn(0x7f08f00fc7b0 msgr2=0x7f08f00febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:00.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.640+0000 7f08f7200700 1 -- 172.21.15.78:0/763076088 shutdown_connections 2024-03-14T10:04:00.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.640+0000 7f08f7200700 1 -- 172.21.15.78:0/763076088 wait complete. 2024-03-14T10:04:00.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.641+0000 7f08f7200700 1 Processor -- start 2024-03-14T10:04:00.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.641+0000 7f08f7200700 1 -- start start 2024-03-14T10:04:00.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08f7200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f08f0100f30 0x7f08f00730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08f7200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08f0106050 0x7f08f00735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08f7200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08f0073b30 0x7f08f01a58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08f7200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f01a5eb0 con 0x7f08f0106050 2024-03-14T10:04:00.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08f7200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f01a6020 con 0x7f08f0073b30 2024-03-14T10:04:00.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08f7200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f01a61e0 con 0x7f08f0100f30 2024-03-14T10:04:00.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08ef400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08f0106050 0x7f08f00735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08ef400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08f0106050 0x7f08f00735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40786/0 (socket says 172.21.15.78:40786) 2024-03-14T10:04:00.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08ef400700 1 -- 172.21.15.78:0/1715696836 learned_addr learned my addr 172.21.15.78:0/1715696836 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:00.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08efe00700 1 --2- 172.21.15.78:0/1715696836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f08f0100f30 0x7f08f00730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08f5200700 1 --2- 172.21.15.78:0/1715696836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08f0073b30 0x7f08f01a58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08ef400700 1 -- 172.21.15.78:0/1715696836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f08f0100f30 msgr2=0x7f08f00730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:00.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08ef400700 1 --2- 172.21.15.78:0/1715696836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f08f0100f30 0x7f08f00730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08ef400700 1 -- 172.21.15.78:0/1715696836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08f0073b30 msgr2=0x7f08f01a58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:00.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08ef400700 1 --2- 172.21.15.78:0/1715696836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08f0073b30 0x7f08f01a58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08ef400700 1 -- 172.21.15.78:0/1715696836 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e8009d00 con 0x7f08f0106050 2024-03-14T10:04:00.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.642+0000 7f08ef400700 1 --2- 172.21.15.78:0/1715696836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08f0106050 0x7f08f00735f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00e970 tx=0x7f08dc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:00.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.643+0000 7f08ecc00700 1 -- 172.21.15.78:0/1715696836 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc009790 con 0x7f08f0106050 2024-03-14T10:04:00.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.643+0000 7f08ecc00700 1 -- 172.21.15.78:0/1715696836 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08dc00eec0 con 0x7f08f0106050 2024-03-14T10:04:00.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.643+0000 7f08f7200700 1 -- 172.21.15.78:0/1715696836 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f01a64c0 con 0x7f08f0106050 2024-03-14T10:04:00.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.643+0000 7f08ecc00700 1 -- 172.21.15.78:0/1715696836 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc005590 con 0x7f08f0106050 2024-03-14T10:04:00.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.643+0000 7f08f7200700 1 -- 172.21.15.78:0/1715696836 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f01a69d0 con 0x7f08f0106050 2024-03-14T10:04:00.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.647+0000 7f08ecc00700 1 -- 172.21.15.78:0/1715696836 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f08dc00cbd0 con 0x7f08f0106050 2024-03-14T10:04:00.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.648+0000 7f090ae16b80 1 Processor -- start 2024-03-14T10:04:00.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.648+0000 7f090ae16b80 1 -- start start 2024-03-14T10:04:00.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.648+0000 7f08ecc00700 1 --2- 172.21.15.78:0/1715696836 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f08d40a0c00 0x7f08d40a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f090ae16b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8e680 0x55b68ca8ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f090ae16b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8efa0 0x55b68ca98980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f08ecc00700 1 -- 172.21.15.78:0/1715696836 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f08dc014070 con 0x7f08f0106050 2024-03-14T10:04:00.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f090ae16b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68ca9b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f090ae16b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b68c9e7990 con 0x55b68ca98ec0 2024-03-14T10:04:00.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f090ae16b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b68c9e5350 con 0x55b68ca8e680 2024-03-14T10:04:00.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f090ae16b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b68c9e0c40 con 0x55b68ca8efa0 2024-03-14T10:04:00.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f08efe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8e680 0x55b68ca8ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f08efe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8e680 0x55b68ca8ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:47342/0 (socket says 172.21.15.78:47342) 2024-03-14T10:04:00.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f08efe00700 1 -- 172.21.15.78:0/1399506081 learned_addr learned my addr 172.21.15.78:0/1399506081 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:00.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f08f5200700 1 --2- 172.21.15.78:0/1715696836 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f08d40a0c00 0x7f08d40a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f08f5200700 1 --2- 172.21.15.78:0/1399506081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68ca9b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f08ef400700 1 --2- 172.21.15.78:0/1399506081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8efa0 0x55b68ca98980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f08efe00700 1 -- 172.21.15.78:0/1399506081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8efa0 msgr2=0x55b68ca98980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:00.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f08efe00700 1 --2- 172.21.15.78:0/1399506081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8efa0 0x55b68ca98980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f08efe00700 1 -- 172.21.15.78:0/1399506081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 msgr2=0x55b68ca9b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:00.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.649+0000 7f08efe00700 1 --2- 172.21.15.78:0/1399506081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68ca9b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.650+0000 7f08efe00700 1 -- 172.21.15.78:0/1399506081 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b68c8623b0 con 0x55b68ca8e680 2024-03-14T10:04:00.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.650+0000 7f08f5200700 1 --2- 172.21.15.78:0/1715696836 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f08d40a0c00 0x7f08d40a30c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f08e8009850 tx=0x7f08e80094a0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:00.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.650+0000 7f08efe00700 1 --2- 172.21.15.78:0/1399506081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8e680 0x55b68ca8ea60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f08e000bd70 tx=0x7f08e00106b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b16d5c563ff2f36 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:00.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.650+0000 7f08e5600700 1 -- 172.21.15.78:0/1399506081 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e0010e10 con 0x55b68ca8e680 2024-03-14T10:04:00.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.650+0000 7f08e5600700 1 -- 172.21.15.78:0/1399506081 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08e0015070 con 0x55b68ca8e680 2024-03-14T10:04:00.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.650+0000 7f08e5600700 1 -- 172.21.15.78:0/1399506081 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e00116a0 con 0x55b68ca8e680 2024-03-14T10:04:00.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.651+0000 7f090ae16b80 1 -- 172.21.15.78:0/1399506081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8e680 msgr2=0x55b68ca8ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:00.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.651+0000 7f090ae16b80 1 --2- 172.21.15.78:0/1399506081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8e680 0x55b68ca8ea60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f08e000bd70 tx=0x7f08e00106b0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.651+0000 7f090ae16b80 1 -- 172.21.15.78:0/1399506081 shutdown_connections 2024-03-14T10:04:00.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.651+0000 7f090ae16b80 1 --2- 172.21.15.78:0/1399506081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8e680 0x55b68ca8ea60 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.651+0000 7f090ae16b80 1 --2- 172.21.15.78:0/1399506081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68ca9b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.651+0000 7f090ae16b80 1 --2- 172.21.15.78:0/1399506081 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8efa0 0x55b68ca98980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.651+0000 7f090ae16b80 1 -- 172.21.15.78:0/1399506081 >> 172.21.15.78:0/1399506081 conn(0x55b68ca04220 msgr2=0x55b68ca89aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:00.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.651+0000 7f090ae16b80 1 -- 172.21.15.78:0/1399506081 shutdown_connections 2024-03-14T10:04:00.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.651+0000 7f090ae16b80 1 -- 172.21.15.78:0/1399506081 wait complete. 2024-03-14T10:04:00.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.652+0000 7f090ae16b80 1 -- start start 2024-03-14T10:04:00.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.652+0000 7f090ae16b80 1 Processor -- start 2024-03-14T10:04:00.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.652+0000 7f090ae16b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68ca988d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.652+0000 7f090ae16b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8efa0 0x55b68ca95c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.652+0000 7f090ae16b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68ca96290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.652+0000 7f090ae16b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b68c97cd80 con 0x55b68ca98ec0 2024-03-14T10:04:00.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.652+0000 7f090ae16b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b68c97ca20 con 0x55b68ca8efa0 2024-03-14T10:04:00.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.652+0000 7f090ae16b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b68c8646a0 con 0x55b68ca8e680 2024-03-14T10:04:00.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.652+0000 7f090ae16b80 10 client.0 _ll_register_callbacks cb 0x55b68c7e83a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:00.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.653+0000 7f08efe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68ca988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.653+0000 7f08f5200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68ca96290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.653+0000 7f08efe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68ca988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:40122/0 (socket says 172.21.15.78:40122) 2024-03-14T10:04:00.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.653+0000 7f08efe00700 1 -- 172.21.15.78:0/4118014251 learned_addr learned my addr 172.21.15.78:0/4118014251 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:00.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.653+0000 7f08ef400700 1 --2- 172.21.15.78:0/4118014251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8efa0 0x55b68ca95c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.653+0000 7f08efe00700 1 -- 172.21.15.78:0/4118014251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8efa0 msgr2=0x55b68ca95c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:00.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.653+0000 7f08efe00700 1 --2- 172.21.15.78:0/4118014251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8efa0 0x55b68ca95c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.653+0000 7f08efe00700 1 -- 172.21.15.78:0/4118014251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 msgr2=0x55b68ca96290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:00.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.653+0000 7f08efe00700 1 --2- 172.21.15.78:0/4118014251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68ca96290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.653+0000 7f08efe00700 1 -- 172.21.15.78:0/4118014251 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e000b9a0 con 0x55b68ca8e680 2024-03-14T10:04:00.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.653+0000 7f08ef400700 1 --2- 172.21.15.78:0/4118014251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8efa0 0x55b68ca95c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T10:04:00.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.653+0000 7f08efe00700 1 --2- 172.21.15.78:0/4118014251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68ca988d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f08e001cd40 tx=0x7f08e0010ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=100eb09c28a0fed server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:00.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.653+0000 7f08da000700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:04:00.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.653+0000 7f08da000700 1 -- 172.21.15.78:0/4118014251 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e001c890 con 0x55b68ca8e680 2024-03-14T10:04:00.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.654+0000 7f090ae16b80 1 -- 172.21.15.78:0/4118014251 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b68ca81720 con 0x55b68ca8e680 2024-03-14T10:04:00.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.654+0000 7f08da000700 1 -- 172.21.15.78:0/4118014251 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f08e0017070 con 0x55b68ca8e680 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.654+0000 7f08da000700 1 -- 172.21.15.78:0/4118014251 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08e000bec0 con 0x55b68ca8e680 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.654+0000 7f090ae16b80 10 client.27571 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.654+0000 7f090ae16b80 1 -- 172.21.15.78:0/4118014251 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b68c80eb60 con 0x55b68ca8e680 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.654+0000 7f08da000700 1 -- 172.21.15.78:0/4118014251 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b68ca81720 con 0x55b68ca8e680 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.654+0000 7f08da000700 1 -- 172.21.15.78:0/4118014251 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f08e0003c10 con 0x55b68ca8e680 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.654+0000 7f090ae16b80 10 client.27571 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.654+0000 7f090ae16b80 10 client.27571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.654+0000 7f090ae16b80 20 client.27571 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.654+0000 7f090ae16b80 1 --2- 172.21.15.78:0/4118014251 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b68caa8970 0x55b68caaad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.654+0000 7f090ae16b80 4 client.27571 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.654+0000 7f090ae16b80 1 -- 172.21.15.78:0/4118014251 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b68c9d9700 con 0x55b68caa8970 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.655+0000 7f08ef400700 1 --2- 172.21.15.78:0/4118014251 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b68caa8970 0x55b68caaad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.655+0000 7f08ef400700 1 --2- 172.21.15.78:0/4118014251 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b68caa8970 0x55b68caaad90 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4253fe11a74bb031 server_cookie=125e54c094a355a3 in_seq=0 out_seq=0 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.655+0000 7f08da000700 10 client.27571 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.656+0000 7f08da000700 1 -- 172.21.15.78:0/4118014251 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55b68c9d9700 con 0x55b68caa8970 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.656+0000 7f08da000700 10 client.27571 handle_command_reply: tid=0 2024-03-14T10:04:00.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.656+0000 7f090ae16b80 1 client.27571 shutdown 2024-03-14T10:04:00.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.656+0000 7f090ae16b80 1 -- 172.21.15.78:0/4118014251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 msgr2=0x55b68ca988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:00.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.656+0000 7f090ae16b80 1 --2- 172.21.15.78:0/4118014251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68ca988d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f08e001cd40 tx=0x7f08e0010ec0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.656+0000 7f090ae16b80 1 -- 172.21.15.78:0/4118014251 shutdown_connections 2024-03-14T10:04:00.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.656+0000 7f090ae16b80 1 --2- 172.21.15.78:0/4118014251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68ca96290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.656+0000 7f090ae16b80 1 --2- 172.21.15.78:0/4118014251 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68ca988d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.656+0000 7f090ae16b80 1 --2- 172.21.15.78:0/4118014251 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b68caa8970 0x55b68caaad90 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.656+0000 7f090ae16b80 1 --2- 172.21.15.78:0/4118014251 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8efa0 0x55b68ca95c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.656+0000 7f090ae16b80 1 -- 172.21.15.78:0/4118014251 >> 172.21.15.78:0/4118014251 conn(0x55b68ca04220 msgr2=0x55b68ca89690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:00.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.656+0000 7f08da000700 0 client.27571 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:00.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.656+0000 7f090ae16b80 1 -- 172.21.15.78:0/4118014251 shutdown_connections 2024-03-14T10:04:00.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.656+0000 7f090ae16b80 1 -- 172.21.15.78:0/4118014251 wait complete. 2024-03-14T10:04:00.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.656+0000 7f090ae16b80 20 client.27571 trim_cache size 0 max 16384 2024-03-14T10:04:00.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.673+0000 7f090ae16b80 1 Processor -- start 2024-03-14T10:04:00.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.673+0000 7f090ae16b80 1 -- start start 2024-03-14T10:04:00.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.674+0000 7f090ae16b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68caaf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.674+0000 7f090ae16b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8efa0 0x55b68ca9fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.674+0000 7f090ae16b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68ca98400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.674+0000 7f090ae16b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b68ca90360 con 0x55b68ca98ec0 2024-03-14T10:04:00.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.674+0000 7f090ae16b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b68caab150 con 0x55b68ca8efa0 2024-03-14T10:04:00.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.674+0000 7f090ae16b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b68ca8b0e0 con 0x55b68ca8e680 2024-03-14T10:04:00.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.674+0000 7f08efe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68caaf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.674+0000 7f08efe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68caaf710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:40126/0 (socket says 172.21.15.78:40126) 2024-03-14T10:04:00.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.674+0000 7f08efe00700 1 -- 172.21.15.78:0/852468327 learned_addr learned my addr 172.21.15.78:0/852468327 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:00.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.674+0000 7f08f5200700 1 --2- 172.21.15.78:0/852468327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68ca98400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.674+0000 7f08ef400700 1 --2- 172.21.15.78:0/852468327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8efa0 0x55b68ca9fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.674+0000 7f08efe00700 1 -- 172.21.15.78:0/852468327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8efa0 msgr2=0x55b68ca9fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:00.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.674+0000 7f08efe00700 1 --2- 172.21.15.78:0/852468327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8efa0 0x55b68ca9fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.674+0000 7f08efe00700 1 -- 172.21.15.78:0/852468327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 msgr2=0x55b68ca98400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:00.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.674+0000 7f08efe00700 1 --2- 172.21.15.78:0/852468327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68ca98400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.675+0000 7f08efe00700 1 -- 172.21.15.78:0/852468327 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b68c9a6ea0 con 0x55b68ca8e680 2024-03-14T10:04:00.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.675+0000 7f08efe00700 1 --2- 172.21.15.78:0/852468327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68caaf710 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f08e000bd70 tx=0x7f08e0005ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b5264f92f7e514e server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:00.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.675+0000 7f08e6000700 1 -- 172.21.15.78:0/852468327 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e0011630 con 0x55b68ca8e680 2024-03-14T10:04:00.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.675+0000 7f08e6000700 1 -- 172.21.15.78:0/852468327 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08e0010a00 con 0x55b68ca8e680 2024-03-14T10:04:00.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.675+0000 7f08e6000700 1 -- 172.21.15.78:0/852468327 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e0011be0 con 0x55b68ca8e680 2024-03-14T10:04:00.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.675+0000 7f090ae16b80 1 -- 172.21.15.78:0/852468327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 msgr2=0x55b68caaf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:00.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.676+0000 7f090ae16b80 1 --2- 172.21.15.78:0/852468327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68caaf710 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f08e000bd70 tx=0x7f08e0005ce0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.676+0000 7f090ae16b80 1 -- 172.21.15.78:0/852468327 shutdown_connections 2024-03-14T10:04:00.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.676+0000 7f090ae16b80 1 --2- 172.21.15.78:0/852468327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68ca98400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.676+0000 7f090ae16b80 1 --2- 172.21.15.78:0/852468327 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68caaf710 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.676+0000 7f090ae16b80 1 --2- 172.21.15.78:0/852468327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8efa0 0x55b68ca9fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.676+0000 7f090ae16b80 1 -- 172.21.15.78:0/852468327 >> 172.21.15.78:0/852468327 conn(0x55b68ca04220 msgr2=0x55b68ca88180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:00.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.676+0000 7f090ae16b80 1 -- 172.21.15.78:0/852468327 shutdown_connections 2024-03-14T10:04:00.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.676+0000 7f090ae16b80 1 -- 172.21.15.78:0/852468327 wait complete. 2024-03-14T10:04:00.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.676+0000 7f090ae16b80 1 -- start start 2024-03-14T10:04:00.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.676+0000 7f090ae16b80 1 Processor -- start 2024-03-14T10:04:00.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.677+0000 7f090ae16b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68ca9d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.677+0000 7f090ae16b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8efa0 0x55b68ca9dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.677+0000 7f090ae16b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68caf94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.677+0000 7f090ae16b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b68ca9e460 con 0x55b68ca98ec0 2024-03-14T10:04:00.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.677+0000 7f090ae16b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b68caf99f0 con 0x55b68ca8efa0 2024-03-14T10:04:00.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.677+0000 7f090ae16b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b68caf9bb0 con 0x55b68ca8e680 2024-03-14T10:04:00.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.677+0000 7f090ae16b80 10 client.0 _ll_register_callbacks cb 0x7f08e8009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:00.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.677+0000 7f08efe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68ca9d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.677+0000 7f08efe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68ca9d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:40132/0 (socket says 172.21.15.78:40132) 2024-03-14T10:04:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.677+0000 7f08efe00700 1 -- 172.21.15.78:0/2947819238 learned_addr learned my addr 172.21.15.78:0/2947819238 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.677+0000 7f08ef400700 1 --2- 172.21.15.78:0/2947819238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8efa0 0x55b68ca9dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.677+0000 7f08f5200700 1 --2- 172.21.15.78:0/2947819238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68caf94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.678+0000 7f08efe00700 1 -- 172.21.15.78:0/2947819238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8efa0 msgr2=0x55b68ca9dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.678+0000 7f08efe00700 1 --2- 172.21.15.78:0/2947819238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8efa0 0x55b68ca9dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.678+0000 7f08efe00700 1 -- 172.21.15.78:0/2947819238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 msgr2=0x55b68caf94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.678+0000 7f08efe00700 1 --2- 172.21.15.78:0/2947819238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68caf94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.678+0000 7f08efe00700 1 -- 172.21.15.78:0/2947819238 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b68c80eb60 con 0x55b68ca8e680 2024-03-14T10:04:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.678+0000 7f08f5200700 1 --2- 172.21.15.78:0/2947819238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68caf94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:04:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.678+0000 7f08efe00700 1 --2- 172.21.15.78:0/2947819238 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68ca9d7d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f08e0010f00 tx=0x7f08e000b410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39c1d4de61f6225c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.678+0000 7f08da000700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:04:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.678+0000 7f08da000700 1 -- 172.21.15.78:0/2947819238 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e0011630 con 0x55b68ca8e680 2024-03-14T10:04:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.678+0000 7f08da000700 1 -- 172.21.15.78:0/2947819238 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f08e0017070 con 0x55b68ca8e680 2024-03-14T10:04:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.678+0000 7f090ae16b80 1 -- 172.21.15.78:0/2947819238 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f08e0015070 con 0x55b68ca8e680 2024-03-14T10:04:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.679+0000 7f08da000700 1 -- 172.21.15.78:0/2947819238 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08e0005070 con 0x55b68ca8e680 2024-03-14T10:04:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.679+0000 7f090ae16b80 10 client.27577 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:00.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.679+0000 7f090ae16b80 1 -- 172.21.15.78:0/2947819238 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55b68c866cd0 con 0x55b68ca8e680 2024-03-14T10:04:00.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.679+0000 7f08da000700 1 -- 172.21.15.78:0/2947819238 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08e0015070 con 0x55b68ca8e680 2024-03-14T10:04:00.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.679+0000 7f08da000700 1 -- 172.21.15.78:0/2947819238 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f08e003c030 con 0x55b68ca8e680 2024-03-14T10:04:00.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.679+0000 7f090ae16b80 10 client.27577 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:00.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.679+0000 7f090ae16b80 10 client.27577 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:04:00.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.679+0000 7f090ae16b80 20 client.27577 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:00.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.679+0000 7f090ae16b80 1 --2- 172.21.15.78:0/2947819238 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b68cafae90 0x55b68cafd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.679+0000 7f090ae16b80 4 client.27577 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:04:00.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.679+0000 7f090ae16b80 1 -- 172.21.15.78:0/2947819238 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b68c82fcf0 con 0x55b68cafae90 2024-03-14T10:04:00.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.679+0000 7f08ef400700 1 --2- 172.21.15.78:0/2947819238 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b68cafae90 0x55b68cafd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f08da000700 10 client.27577 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:00.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f08ef400700 1 --2- 172.21.15.78:0/2947819238 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b68cafae90 0x55b68cafd2b0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcf6071784c2e8e7 server_cookie=76bf9ed161aadf1c in_seq=0 out_seq=0 2024-03-14T10:04:00.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f08da000700 1 -- 172.21.15.78:0/2947819238 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55b68c82fcf0 con 0x55b68cafae90 2024-03-14T10:04:00.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f08da000700 10 client.27577 handle_command_reply: tid=0 2024-03-14T10:04:00.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f090ae16b80 1 client.27577 shutdown 2024-03-14T10:04:00.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f090ae16b80 1 -- 172.21.15.78:0/2947819238 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 msgr2=0x55b68ca9d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:00.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f090ae16b80 1 --2- 172.21.15.78:0/2947819238 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68ca9d7d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f08e0010f00 tx=0x7f08e000b410 comp rx=0 tx=0).stop 2024-03-14T10:04:00.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f090ae16b80 1 -- 172.21.15.78:0/2947819238 shutdown_connections 2024-03-14T10:04:00.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f090ae16b80 1 --2- 172.21.15.78:0/2947819238 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b68ca98ec0 0x55b68caf94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f090ae16b80 1 --2- 172.21.15.78:0/2947819238 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55b68ca8e680 0x55b68ca9d7d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f090ae16b80 1 --2- 172.21.15.78:0/2947819238 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55b68cafae90 0x55b68cafd2b0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f090ae16b80 1 --2- 172.21.15.78:0/2947819238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b68ca8efa0 0x55b68ca9dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f08da000700 0 client.27577 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:00.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f090ae16b80 1 -- 172.21.15.78:0/2947819238 >> 172.21.15.78:0/2947819238 conn(0x55b68ca04220 msgr2=0x55b68ca895b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:00.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f090ae16b80 1 -- 172.21.15.78:0/2947819238 shutdown_connections 2024-03-14T10:04:00.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f090ae16b80 1 -- 172.21.15.78:0/2947819238 wait complete. 2024-03-14T10:04:00.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.680+0000 7f090ae16b80 20 client.27577 trim_cache size 0 max 16384 2024-03-14T10:04:00.688 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:04:00.688 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T10:04:00.688 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T10:04:00.688 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:04:00.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.681+0000 7f08f7200700 1 -- 172.21.15.78:0/1715696836 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f08d40a0c00 msgr2=0x7f08d40a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:00.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.681+0000 7f08f7200700 1 --2- 172.21.15.78:0/1715696836 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f08d40a0c00 0x7f08d40a30c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f08e8009850 tx=0x7f08e80094a0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.681+0000 7f08f7200700 1 -- 172.21.15.78:0/1715696836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08f0106050 msgr2=0x7f08f00735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:00.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.681+0000 7f08f7200700 1 --2- 172.21.15.78:0/1715696836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08f0106050 0x7f08f00735f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00e970 tx=0x7f08dc00ec80 comp rx=0 tx=0).stop 2024-03-14T10:04:00.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.681+0000 7f08f7200700 1 -- 172.21.15.78:0/1715696836 shutdown_connections 2024-03-14T10:04:00.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.681+0000 7f08f7200700 1 --2- 172.21.15.78:0/1715696836 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f08d40a0c00 0x7f08d40a30c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.681+0000 7f08f7200700 1 --2- 172.21.15.78:0/1715696836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08f0073b30 0x7f08f01a58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.681+0000 7f08f7200700 1 --2- 172.21.15.78:0/1715696836 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f08f0100f30 0x7f08f00730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.681+0000 7f08f7200700 1 --2- 172.21.15.78:0/1715696836 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08f0106050 0x7f08f00735f0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.681+0000 7f08f7200700 1 -- 172.21.15.78:0/1715696836 >> 172.21.15.78:0/1715696836 conn(0x7f08f00fc7b0 msgr2=0x7f08f00feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:00.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.681+0000 7f08f7200700 1 -- 172.21.15.78:0/1715696836 shutdown_connections 2024-03-14T10:04:00.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.681+0000 7f08f7200700 1 -- 172.21.15.78:0/1715696836 wait complete. 2024-03-14T10:04:00.700 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T10:04:00.700 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T10:04:00.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.892+0000 7f1d7c600700 1 -- 172.21.15.78:0/1048547870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d741017d0 msgr2=0x7f1d74105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:00.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.892+0000 7f1d7c600700 1 --2- 172.21.15.78:0/1048547870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d741017d0 0x7f1d74105810 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f1d60009990 tx=0x7f1d60009ca0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.892+0000 7f1d7c600700 1 -- 172.21.15.78:0/1048547870 shutdown_connections 2024-03-14T10:04:00.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.892+0000 7f1d7c600700 1 --2- 172.21.15.78:0/1048547870 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d74100e20 0x7f1d74101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.892+0000 7f1d7c600700 1 --2- 172.21.15.78:0/1048547870 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d74105f40 0x7f1d7410dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.892+0000 7f1d7c600700 1 --2- 172.21.15.78:0/1048547870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d741017d0 0x7f1d74105810 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.892+0000 7f1d7c600700 1 -- 172.21.15.78:0/1048547870 >> 172.21.15.78:0/1048547870 conn(0x7f1d740fc700 msgr2=0x7f1d740feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:00.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.892+0000 7f1d7c600700 1 -- 172.21.15.78:0/1048547870 shutdown_connections 2024-03-14T10:04:00.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.892+0000 7f1d7c600700 1 -- 172.21.15.78:0/1048547870 wait complete. 2024-03-14T10:04:00.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.893+0000 7f1d7c600700 1 Processor -- start 2024-03-14T10:04:00.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.893+0000 7f1d7c600700 1 -- start start 2024-03-14T10:04:00.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.894+0000 7f1d7c600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d74100e20 0x7f1d74197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.894+0000 7f1d7c600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d74105f40 0x7f1d74197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.894+0000 7f1d7c600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d7419be00 0x7f1d7419c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.894+0000 7f1d7c600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7419c880 con 0x7f1d7419be00 2024-03-14T10:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.894+0000 7f1d7c600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7419c9f0 con 0x7f1d74105f40 2024-03-14T10:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.894+0000 7f1d7c600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7419cbb0 con 0x7f1d74100e20 2024-03-14T10:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.894+0000 7f1d7a600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d7419be00 0x7f1d7419c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.894+0000 7f1d7a600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d7419be00 0x7f1d7419c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40832/0 (socket says 172.21.15.78:40832) 2024-03-14T10:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.894+0000 7f1d7a600700 1 -- 172.21.15.78:0/3419250514 learned_addr learned my addr 172.21.15.78:0/3419250514 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.894+0000 7f1d79c00700 1 --2- 172.21.15.78:0/3419250514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d74100e20 0x7f1d74197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.894+0000 7f1d79200700 1 --2- 172.21.15.78:0/3419250514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d74105f40 0x7f1d74197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.894+0000 7f1d7a600700 1 -- 172.21.15.78:0/3419250514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d74100e20 msgr2=0x7f1d74197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.894+0000 7f1d7a600700 1 --2- 172.21.15.78:0/3419250514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d74100e20 0x7f1d74197520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.894+0000 7f1d7a600700 1 -- 172.21.15.78:0/3419250514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d74105f40 msgr2=0x7f1d74197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.894+0000 7f1d7a600700 1 --2- 172.21.15.78:0/3419250514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d74105f40 0x7f1d74197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.894+0000 7f1d7a600700 1 -- 172.21.15.78:0/3419250514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d600095f0 con 0x7f1d7419be00 2024-03-14T10:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.895+0000 7f1d7a600700 1 --2- 172.21.15.78:0/3419250514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d7419be00 0x7f1d7419c280 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00ef30 tx=0x7f1d6c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.895+0000 7f1d72000700 1 -- 172.21.15.78:0/3419250514 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c018070 con 0x7f1d7419be00 2024-03-14T10:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.895+0000 7f1d72000700 1 -- 172.21.15.78:0/3419250514 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d6c00f040 con 0x7f1d7419be00 2024-03-14T10:04:00.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.895+0000 7f1d7c600700 1 -- 172.21.15.78:0/3419250514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d741a40b0 con 0x7f1d7419be00 2024-03-14T10:04:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.895+0000 7f1d72000700 1 -- 172.21.15.78:0/3419250514 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c00cd00 con 0x7f1d7419be00 2024-03-14T10:04:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.895+0000 7f1d7c600700 1 -- 172.21.15.78:0/3419250514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d741a45e0 con 0x7f1d7419be00 2024-03-14T10:04:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.896+0000 7f1d7c600700 1 -- 172.21.15.78:0/3419250514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d74109210 con 0x7f1d7419be00 2024-03-14T10:04:00.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.898+0000 7f1d72000700 1 -- 172.21.15.78:0/3419250514 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f1d6c003a00 con 0x7f1d7419be00 2024-03-14T10:04:00.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.899+0000 7f1d72000700 1 --2- 172.21.15.78:0/3419250514 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1d5c0a0c80 0x7f1d5c0a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:00.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.899+0000 7f1d72000700 1 -- 172.21.15.78:0/3419250514 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f1d6c0cc2a0 con 0x7f1d7419be00 2024-03-14T10:04:00.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.900+0000 7f1d79c00700 1 --2- 172.21.15.78:0/3419250514 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1d5c0a0c80 0x7f1d5c0a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:00.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.900+0000 7f1d79c00700 1 --2- 172.21.15.78:0/3419250514 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1d5c0a0c80 0x7f1d5c0a3140 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009c60 tx=0x7f1d68009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:00.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:00.904+0000 7f1d72000700 1 -- 172.21.15.78:0/3419250514 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1d6c095590 con 0x7f1d7419be00 2024-03-14T10:04:01.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.161+0000 7f1d7c600700 1 -- 172.21.15.78:0/3419250514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d741022f0 con 0x7f1d7419be00 2024-03-14T10:04:01.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.162+0000 7f1d72000700 1 -- 172.21.15.78:0/3419250514 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f1d6c093e60 con 0x7f1d7419be00 2024-03-14T10:04:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T10:04:01.163 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T10:04:01.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.164+0000 7f1d7c600700 1 -- 172.21.15.78:0/3419250514 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1d5c0a0c80 msgr2=0x7f1d5c0a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.164+0000 7f1d7c600700 1 --2- 172.21.15.78:0/3419250514 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1d5c0a0c80 0x7f1d5c0a3140 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009c60 tx=0x7f1d68009260 comp rx=0 tx=0).stop 2024-03-14T10:04:01.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.164+0000 7f1d7c600700 1 -- 172.21.15.78:0/3419250514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d7419be00 msgr2=0x7f1d7419c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.164+0000 7f1d7c600700 1 --2- 172.21.15.78:0/3419250514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d7419be00 0x7f1d7419c280 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00ef30 tx=0x7f1d6c00c3a0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.164+0000 7f1d7c600700 1 -- 172.21.15.78:0/3419250514 shutdown_connections 2024-03-14T10:04:01.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.164+0000 7f1d7c600700 1 --2- 172.21.15.78:0/3419250514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d7419be00 0x7f1d7419c280 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.164+0000 7f1d7c600700 1 --2- 172.21.15.78:0/3419250514 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d74100e20 0x7f1d74197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.164+0000 7f1d7c600700 1 --2- 172.21.15.78:0/3419250514 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1d5c0a0c80 0x7f1d5c0a3140 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.164+0000 7f1d7c600700 1 --2- 172.21.15.78:0/3419250514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d74105f40 0x7f1d74197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.164+0000 7f1d7c600700 1 -- 172.21.15.78:0/3419250514 >> 172.21.15.78:0/3419250514 conn(0x7f1d740fc700 msgr2=0x7f1d740fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.165+0000 7f1d7c600700 1 -- 172.21.15.78:0/3419250514 shutdown_connections 2024-03-14T10:04:01.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.165+0000 7f1d7c600700 1 -- 172.21.15.78:0/3419250514 wait complete. 2024-03-14T10:04:01.166 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T10:04:01.184 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T10:04:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.377+0000 7f6520000700 1 -- 172.21.15.78:0/3610810390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65181018d0 msgr2=0x7f6518105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.377+0000 7f6520000700 1 --2- 172.21.15.78:0/3610810390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65181018d0 0x7f6518105910 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f6504009990 tx=0x7f6504009ca0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.377+0000 7f6520000700 1 -- 172.21.15.78:0/3610810390 shutdown_connections 2024-03-14T10:04:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.377+0000 7f6520000700 1 --2- 172.21.15.78:0/3610810390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6518100f20 0x7f6518101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.377+0000 7f6520000700 1 --2- 172.21.15.78:0/3610810390 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6518106040 0x7f651810dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.377+0000 7f6520000700 1 --2- 172.21.15.78:0/3610810390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f65181018d0 0x7f6518105910 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.377+0000 7f6520000700 1 -- 172.21.15.78:0/3610810390 >> 172.21.15.78:0/3610810390 conn(0x7f65180fc7a0 msgr2=0x7f65180febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.377+0000 7f6520000700 1 -- 172.21.15.78:0/3610810390 shutdown_connections 2024-03-14T10:04:01.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.377+0000 7f6520000700 1 -- 172.21.15.78:0/3610810390 wait complete. 2024-03-14T10:04:01.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.378+0000 7f6520000700 1 Processor -- start 2024-03-14T10:04:01.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.378+0000 7f6520000700 1 -- start start 2024-03-14T10:04:01.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.378+0000 7f6520000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6518100f20 0x7f65180730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.378+0000 7f6520000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6518106040 0x7f65180735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.378+0000 7f6520000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6518073b30 0x7f65181a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.378+0000 7f6520000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65181a5e20 con 0x7f6518073b30 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.378+0000 7f6520000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65181a5f60 con 0x7f6518100f20 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.378+0000 7f6520000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65181a6120 con 0x7f6518106040 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.379+0000 7f651d600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6518100f20 0x7f65180730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.379+0000 7f651e000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6518073b30 0x7f65181a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.379+0000 7f651d600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6518100f20 0x7f65180730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:47410/0 (socket says 172.21.15.78:47410) 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.379+0000 7f651d600700 1 -- 172.21.15.78:0/3933945943 learned_addr learned my addr 172.21.15.78:0/3933945943 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.379+0000 7f651cc00700 1 --2- 172.21.15.78:0/3933945943 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6518106040 0x7f65180735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.379+0000 7f651e000700 1 -- 172.21.15.78:0/3933945943 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6518106040 msgr2=0x7f65180735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.379+0000 7f651e000700 1 --2- 172.21.15.78:0/3933945943 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6518106040 0x7f65180735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.379+0000 7f651e000700 1 -- 172.21.15.78:0/3933945943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6518100f20 msgr2=0x7f65180730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.379+0000 7f651e000700 1 --2- 172.21.15.78:0/3933945943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6518100f20 0x7f65180730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.379+0000 7f651e000700 1 -- 172.21.15.78:0/3933945943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65040095f0 con 0x7f6518073b30 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.379+0000 7f651e000700 1 --2- 172.21.15.78:0/3933945943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6518073b30 0x7f65181a5850 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f6510007f10 tx=0x7f651000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.379+0000 7f6516000700 1 -- 172.21.15.78:0/3933945943 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6510009ea0 con 0x7f6518073b30 2024-03-14T10:04:01.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.379+0000 7f6520000700 1 -- 172.21.15.78:0/3933945943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65181a6400 con 0x7f6518073b30 2024-03-14T10:04:01.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.379+0000 7f6516000700 1 -- 172.21.15.78:0/3933945943 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f651000f040 con 0x7f6518073b30 2024-03-14T10:04:01.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.379+0000 7f6516000700 1 -- 172.21.15.78:0/3933945943 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6510004350 con 0x7f6518073b30 2024-03-14T10:04:01.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.380+0000 7f6520000700 1 -- 172.21.15.78:0/3933945943 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65181a6910 con 0x7f6518073b30 2024-03-14T10:04:01.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.382+0000 7f6516000700 1 -- 172.21.15.78:0/3933945943 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f651000db20 con 0x7f6518073b30 2024-03-14T10:04:01.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.384+0000 7f6533be4b80 1 Processor -- start 2024-03-14T10:04:01.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.384+0000 7f6516000700 1 --2- 172.21.15.78:0/3933945943 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f65000a0c00 0x7f65000a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.384+0000 7f6533be4b80 1 -- start start 2024-03-14T10:04:01.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.384+0000 7f6516000700 1 -- 172.21.15.78:0/3933945943 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f65100cbff0 con 0x7f6518073b30 2024-03-14T10:04:01.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.384+0000 7f651cc00700 1 --2- 172.21.15.78:0/3933945943 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f65000a0c00 0x7f65000a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.384+0000 7f6533be4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db88680 0x56243db88a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.384+0000 7f6533be4b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 0x56243db92980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.384+0000 7f6533be4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db92ec0 0x56243db952b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.384+0000 7f6533be4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56243dae1990 con 0x56243db92ec0 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.384+0000 7f6533be4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56243dadf350 con 0x56243db88680 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.384+0000 7f6533be4b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56243dadac40 con 0x56243db88fa0 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.385+0000 7f651d600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db88680 0x56243db88a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.385+0000 7f651e000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db92ec0 0x56243db952b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.385+0000 7f651d600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db88680 0x56243db88a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:47416/0 (socket says 172.21.15.78:47416) 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.385+0000 7f651e000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db92ec0 0x56243db952b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40876/0 (socket says 172.21.15.78:40876) 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.385+0000 7f651d600700 1 -- 172.21.15.78:0/2063519267 learned_addr learned my addr 172.21.15.78:0/2063519267 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.385+0000 7f651cc00700 1 --2- 172.21.15.78:0/2063519267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 0x56243db92980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.385+0000 7f651d600700 1 -- 172.21.15.78:0/2063519267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 msgr2=0x56243db92980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.385+0000 7f651d600700 1 --2- 172.21.15.78:0/2063519267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 0x56243db92980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.385+0000 7f651d600700 1 -- 172.21.15.78:0/2063519267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db92ec0 msgr2=0x56243db952b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.385+0000 7f651d600700 1 --2- 172.21.15.78:0/2063519267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db92ec0 0x56243db952b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.385+0000 7f651d600700 1 -- 172.21.15.78:0/2063519267 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56243d95c3b0 con 0x56243db88680 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.385+0000 7f651e000700 1 --2- 172.21.15.78:0/2063519267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db92ec0 0x56243db952b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.385+0000 7f651cc00700 1 --2- 172.21.15.78:0/3933945943 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f65000a0c00 0x7f65000a30c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6518074790 tx=0x7f6504005060 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.385+0000 7f651d600700 1 --2- 172.21.15.78:0/2063519267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db88680 0x56243db88a60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f650c009e50 tx=0x7f650c011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0f7772389830199 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.385+0000 7f650a600700 1 -- 172.21.15.78:0/2063519267 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c004a30 con 0x56243db88680 2024-03-14T10:04:01.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.386+0000 7f650a600700 1 -- 172.21.15.78:0/2063519267 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f650c0054e0 con 0x56243db88680 2024-03-14T10:04:01.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.386+0000 7f650a600700 1 -- 172.21.15.78:0/2063519267 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c0198f0 con 0x56243db88680 2024-03-14T10:04:01.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.386+0000 7f6533be4b80 1 -- 172.21.15.78:0/2063519267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db88680 msgr2=0x56243db88a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.386+0000 7f6533be4b80 1 --2- 172.21.15.78:0/2063519267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db88680 0x56243db88a60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f650c009e50 tx=0x7f650c011720 comp rx=0 tx=0).stop 2024-03-14T10:04:01.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.386+0000 7f6533be4b80 1 -- 172.21.15.78:0/2063519267 shutdown_connections 2024-03-14T10:04:01.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.386+0000 7f6533be4b80 1 --2- 172.21.15.78:0/2063519267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db88680 0x56243db88a60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.386+0000 7f6533be4b80 1 --2- 172.21.15.78:0/2063519267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db92ec0 0x56243db952b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.386+0000 7f6533be4b80 1 --2- 172.21.15.78:0/2063519267 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 0x56243db92980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.386+0000 7f6533be4b80 1 -- 172.21.15.78:0/2063519267 >> 172.21.15.78:0/2063519267 conn(0x56243dafe220 msgr2=0x56243db83aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.386+0000 7f6533be4b80 1 -- 172.21.15.78:0/2063519267 shutdown_connections 2024-03-14T10:04:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.386+0000 7f6533be4b80 1 -- 172.21.15.78:0/2063519267 wait complete. 2024-03-14T10:04:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.387+0000 7f6533be4b80 1 -- start start 2024-03-14T10:04:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.387+0000 7f6533be4b80 1 Processor -- start 2024-03-14T10:04:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f6533be4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db92870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f6533be4b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 0x56243db8fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f6533be4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db92ec0 0x56243db90230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f6533be4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56243da76d80 con 0x56243db88680 2024-03-14T10:04:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f6533be4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56243da76a20 con 0x56243db92ec0 2024-03-14T10:04:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f6533be4b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56243d95e6a0 con 0x56243db88fa0 2024-03-14T10:04:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f6533be4b80 10 client.0 _ll_register_callbacks cb 0x56243d8e23a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:01.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f651d600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db92870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f651e000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db92ec0 0x56243db90230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f651d600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db92870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40890/0 (socket says 172.21.15.78:40890) 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f651cc00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 0x56243db8fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f651d600700 1 -- 172.21.15.78:0/1929947925 learned_addr learned my addr 172.21.15.78:0/1929947925 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f651e000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db92ec0 0x56243db90230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:47432/0 (socket says 172.21.15.78:47432) 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f651e000700 1 -- 172.21.15.78:0/1929947925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 msgr2=0x56243db8fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f651e000700 1 --2- 172.21.15.78:0/1929947925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 0x56243db8fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f651e000700 1 -- 172.21.15.78:0/1929947925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 msgr2=0x56243db92870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f651e000700 1 --2- 172.21.15.78:0/1929947925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db92870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f651e000700 1 -- 172.21.15.78:0/1929947925 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65181a6910 con 0x56243db92ec0 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.388+0000 7f651d600700 1 --2- 172.21.15.78:0/1929947925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db92870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.389+0000 7f651e000700 1 --2- 172.21.15.78:0/1929947925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db92ec0 0x56243db90230 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f65100c9370 tx=0x7f65100988d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f70e65d7c85237d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.389+0000 7f64fea00700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.389+0000 7f64fea00700 1 -- 172.21.15.78:0/1929947925 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651009cda0 con 0x56243db92ec0 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.389+0000 7f6533be4b80 1 -- 172.21.15.78:0/1929947925 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56243db7b720 con 0x56243db92ec0 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.389+0000 7f64fea00700 1 -- 172.21.15.78:0/1929947925 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f65100ace40 con 0x56243db92ec0 2024-03-14T10:04:01.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.389+0000 7f64fea00700 1 -- 172.21.15.78:0/1929947925 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65100a13f0 con 0x56243db92ec0 2024-03-14T10:04:01.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.389+0000 7f6533be4b80 10 client.27611 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:01.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.389+0000 7f6533be4b80 1 -- 172.21.15.78:0/1929947925 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f650c009a80 con 0x56243db92ec0 2024-03-14T10:04:01.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.389+0000 7f64fea00700 1 -- 172.21.15.78:0/1929947925 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56243db7b720 con 0x56243db92ec0 2024-03-14T10:04:01.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.389+0000 7f64fea00700 1 -- 172.21.15.78:0/1929947925 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f65100991d0 con 0x56243db92ec0 2024-03-14T10:04:01.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.390+0000 7f6533be4b80 10 client.27611 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:01.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.390+0000 7f6533be4b80 10 client.27611 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:04:01.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.390+0000 7f6533be4b80 20 client.27611 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:01.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.390+0000 7f6533be4b80 1 --2- 172.21.15.78:0/1929947925 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56243dba2b70 0x56243dba4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.390+0000 7f6533be4b80 4 client.27611 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:04:01.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.390+0000 7f6533be4b80 1 -- 172.21.15.78:0/1929947925 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56243dad3700 con 0x56243dba2b70 2024-03-14T10:04:01.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.390+0000 7f651d600700 1 --2- 172.21.15.78:0/1929947925 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56243dba2b70 0x56243dba4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.390+0000 7f651d600700 1 --2- 172.21.15.78:0/1929947925 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56243dba2b70 0x56243dba4f90 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2381e779bd00bd57 server_cookie=814e47ddef696eb4 in_seq=0 out_seq=0 2024-03-14T10:04:01.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.390+0000 7f64fea00700 10 client.27611 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:04:01.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.391+0000 7f64fea00700 1 -- 172.21.15.78:0/1929947925 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56243dad3700 con 0x56243dba2b70 2024-03-14T10:04:01.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.391+0000 7f64fea00700 10 client.27611 handle_command_reply: tid=0 2024-03-14T10:04:01.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.391+0000 7f6533be4b80 1 client.27611 shutdown 2024-03-14T10:04:01.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.392+0000 7f6533be4b80 1 -- 172.21.15.78:0/1929947925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db92ec0 msgr2=0x56243db90230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.392+0000 7f6533be4b80 1 --2- 172.21.15.78:0/1929947925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db92ec0 0x56243db90230 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f65100c9370 tx=0x7f65100988d0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.392+0000 7f6533be4b80 1 -- 172.21.15.78:0/1929947925 shutdown_connections 2024-03-14T10:04:01.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.392+0000 7f6533be4b80 1 --2- 172.21.15.78:0/1929947925 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56243dba2b70 0x56243dba4f90 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.392+0000 7f6533be4b80 1 --2- 172.21.15.78:0/1929947925 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db92ec0 0x56243db90230 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.392+0000 7f6533be4b80 1 --2- 172.21.15.78:0/1929947925 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db92870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.392+0000 7f64fea00700 0 client.27611 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:04:01.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.392+0000 7f6533be4b80 1 --2- 172.21.15.78:0/1929947925 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 0x56243db8fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.392+0000 7f6533be4b80 1 -- 172.21.15.78:0/1929947925 >> 172.21.15.78:0/1929947925 conn(0x56243dafe220 msgr2=0x56243db83690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.392+0000 7f6533be4b80 1 -- 172.21.15.78:0/1929947925 shutdown_connections 2024-03-14T10:04:01.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.392+0000 7f6533be4b80 1 -- 172.21.15.78:0/1929947925 wait complete. 2024-03-14T10:04:01.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.392+0000 7f6533be4b80 20 client.27611 trim_cache size 0 max 16384 2024-03-14T10:04:01.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.405+0000 7f6533be4b80 1 Processor -- start 2024-03-14T10:04:01.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.405+0000 7f6533be4b80 1 -- start start 2024-03-14T10:04:01.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.405+0000 7f6533be4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db83c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.405+0000 7f6533be4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db88fa0 0x56243dba98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.405+0000 7f6533be4b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db92ec0 0x56243db8fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.405+0000 7f6533be4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56243db85060 con 0x56243db88680 2024-03-14T10:04:01.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.405+0000 7f6533be4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56243db7dd00 con 0x56243db88fa0 2024-03-14T10:04:01.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.405+0000 7f6533be4b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56243db8f9d0 con 0x56243db92ec0 2024-03-14T10:04:01.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.406+0000 7f651d600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db83c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.406+0000 7f651cc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db88fa0 0x56243dba98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.406+0000 7f651d600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db83c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40892/0 (socket says 172.21.15.78:40892) 2024-03-14T10:04:01.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.406+0000 7f651d600700 1 -- 172.21.15.78:0/332849093 learned_addr learned my addr 172.21.15.78:0/332849093 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:01.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.406+0000 7f651e000700 1 --2- 172.21.15.78:0/332849093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db92ec0 0x56243db8fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.406+0000 7f651d600700 1 -- 172.21.15.78:0/332849093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db92ec0 msgr2=0x56243db8fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.406+0000 7f651d600700 1 --2- 172.21.15.78:0/332849093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db92ec0 0x56243db8fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.406+0000 7f651d600700 1 -- 172.21.15.78:0/332849093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db88fa0 msgr2=0x56243dba98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.406+0000 7f651d600700 1 --2- 172.21.15.78:0/332849093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db88fa0 0x56243dba98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.406+0000 7f651d600700 1 -- 172.21.15.78:0/332849093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56243d908b60 con 0x56243db88680 2024-03-14T10:04:01.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.406+0000 7f651d600700 1 --2- 172.21.15.78:0/332849093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db83c30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f650c009840 tx=0x7f650c011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77407a1af720df5f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.406+0000 7f650b000700 1 -- 172.21.15.78:0/332849093 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c029040 con 0x56243db88680 2024-03-14T10:04:01.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.406+0000 7f650b000700 1 -- 172.21.15.78:0/332849093 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f650c004500 con 0x56243db88680 2024-03-14T10:04:01.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.406+0000 7f650b000700 1 -- 172.21.15.78:0/332849093 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c004f50 con 0x56243db88680 2024-03-14T10:04:01.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.407+0000 7f6533be4b80 1 -- 172.21.15.78:0/332849093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 msgr2=0x56243db83c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.407+0000 7f6533be4b80 1 --2- 172.21.15.78:0/332849093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db83c30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f650c009840 tx=0x7f650c011720 comp rx=0 tx=0).stop 2024-03-14T10:04:01.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.407+0000 7f6533be4b80 1 -- 172.21.15.78:0/332849093 shutdown_connections 2024-03-14T10:04:01.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.407+0000 7f6533be4b80 1 --2- 172.21.15.78:0/332849093 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db92ec0 0x56243db8fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.407+0000 7f6533be4b80 1 --2- 172.21.15.78:0/332849093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db83c30 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.407+0000 7f6533be4b80 1 --2- 172.21.15.78:0/332849093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db88fa0 0x56243dba98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.407+0000 7f6533be4b80 1 -- 172.21.15.78:0/332849093 >> 172.21.15.78:0/332849093 conn(0x56243dafe220 msgr2=0x56243db8af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.407+0000 7f6533be4b80 1 -- 172.21.15.78:0/332849093 shutdown_connections 2024-03-14T10:04:01.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.407+0000 7f6533be4b80 1 -- 172.21.15.78:0/332849093 wait complete. 2024-03-14T10:04:01.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.408+0000 7f6533be4b80 1 -- start start 2024-03-14T10:04:01.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.408+0000 7f6533be4b80 1 Processor -- start 2024-03-14T10:04:01.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.408+0000 7f6533be4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db97b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.408+0000 7f6533be4b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 0x56243db98070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.408+0000 7f6533be4b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db92ec0 0x56243db985b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.408+0000 7f6533be4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56243dbf3770 con 0x56243db88680 2024-03-14T10:04:01.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.408+0000 7f6533be4b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x56243dbf38e0 con 0x56243db92ec0 2024-03-14T10:04:01.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.408+0000 7f6533be4b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56243dbf3aa0 con 0x56243db88fa0 2024-03-14T10:04:01.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.408+0000 7f6533be4b80 10 client.0 _ll_register_callbacks cb 0x7f6510098d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:01.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.409+0000 7f651d600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db97b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.409+0000 7f651cc00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 0x56243db98070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.409+0000 7f651cc00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 0x56243db98070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:40208/0 (socket says 172.21.15.78:40208) 2024-03-14T10:04:01.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.409+0000 7f651d600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db97b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40902/0 (socket says 172.21.15.78:40902) 2024-03-14T10:04:01.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.409+0000 7f651cc00700 1 -- 172.21.15.78:0/1293936987 learned_addr learned my addr 172.21.15.78:0/1293936987 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:01.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.409+0000 7f651e000700 1 --2- 172.21.15.78:0/1293936987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db92ec0 0x56243db985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.409+0000 7f651cc00700 1 -- 172.21.15.78:0/1293936987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db92ec0 msgr2=0x56243db985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.409+0000 7f651cc00700 1 --2- 172.21.15.78:0/1293936987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db92ec0 0x56243db985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.409+0000 7f651cc00700 1 -- 172.21.15.78:0/1293936987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 msgr2=0x56243db97b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.409+0000 7f651cc00700 1 --2- 172.21.15.78:0/1293936987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db97b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.409+0000 7f651cc00700 1 -- 172.21.15.78:0/1293936987 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56243d95c3b0 con 0x56243db88fa0 2024-03-14T10:04:01.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.409+0000 7f651cc00700 1 --2- 172.21.15.78:0/1293936987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 0x56243db98070 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f651806a350 tx=0x7f6504009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=430eaa44e35f90ae server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.409+0000 7f64fea00700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:04:01.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.410+0000 7f64fea00700 1 -- 172.21.15.78:0/1293936987 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6504004b70 con 0x56243db88fa0 2024-03-14T10:04:01.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.410+0000 7f64fea00700 1 -- 172.21.15.78:0/1293936987 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f650402c040 con 0x56243db88fa0 2024-03-14T10:04:01.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.410+0000 7f6533be4b80 1 -- 172.21.15.78:0/1293936987 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f650c0054e0 con 0x56243db88fa0 2024-03-14T10:04:01.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.410+0000 7f64fea00700 1 -- 172.21.15.78:0/1293936987 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6504004cd0 con 0x56243db88fa0 2024-03-14T10:04:01.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.410+0000 7f6533be4b80 10 client.27601 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:01.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.410+0000 7f6533be4b80 1 -- 172.21.15.78:0/1293936987 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56243daa0ea0 con 0x56243db88fa0 2024-03-14T10:04:01.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.410+0000 7f64fea00700 1 -- 172.21.15.78:0/1293936987 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f650c0054e0 con 0x56243db88fa0 2024-03-14T10:04:01.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.410+0000 7f64fea00700 1 -- 172.21.15.78:0/1293936987 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f650400f460 con 0x56243db88fa0 2024-03-14T10:04:01.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.410+0000 7f6533be4b80 10 client.27601 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:01.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.410+0000 7f6533be4b80 10 client.27601 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:04:01.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.410+0000 7f6533be4b80 20 client.27601 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:01.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.410+0000 7f6533be4b80 1 --2- 172.21.15.78:0/1293936987 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56243dbf5180 0x56243dbf75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.410+0000 7f6533be4b80 4 client.27601 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:04:01.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.410+0000 7f6533be4b80 1 -- 172.21.15.78:0/1293936987 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56243d929cf0 con 0x56243dbf5180 2024-03-14T10:04:01.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.411+0000 7f651d600700 1 --2- 172.21.15.78:0/1293936987 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56243dbf5180 0x56243dbf75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.411+0000 7f64fea00700 10 client.27601 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:04:01.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.411+0000 7f651d600700 1 --2- 172.21.15.78:0/1293936987 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56243dbf5180 0x56243dbf75a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=832a039869fe2405 server_cookie=d7315c22f66df857 in_seq=0 out_seq=0 2024-03-14T10:04:01.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.411+0000 7f64fea00700 1 -- 172.21.15.78:0/1293936987 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56243d929cf0 con 0x56243dbf5180 2024-03-14T10:04:01.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.411+0000 7f64fea00700 10 client.27601 handle_command_reply: tid=0 2024-03-14T10:04:01.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.411+0000 7f6533be4b80 1 client.27601 shutdown 2024-03-14T10:04:01.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.411+0000 7f6533be4b80 1 -- 172.21.15.78:0/1293936987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 msgr2=0x56243db98070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.411+0000 7f6533be4b80 1 --2- 172.21.15.78:0/1293936987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 0x56243db98070 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f651806a350 tx=0x7f6504009ca0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.411+0000 7f6533be4b80 1 -- 172.21.15.78:0/1293936987 shutdown_connections 2024-03-14T10:04:01.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.411+0000 7f6533be4b80 1 --2- 172.21.15.78:0/1293936987 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x56243dbf5180 0x56243dbf75a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.411+0000 7f6533be4b80 1 --2- 172.21.15.78:0/1293936987 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56243db92ec0 0x56243db985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.411+0000 7f64fea00700 0 client.27601 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:04:01.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.411+0000 7f6533be4b80 1 --2- 172.21.15.78:0/1293936987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56243db88680 0x56243db97b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.411+0000 7f6533be4b80 1 --2- 172.21.15.78:0/1293936987 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x56243db88fa0 0x56243db98070 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.411+0000 7f6533be4b80 1 -- 172.21.15.78:0/1293936987 >> 172.21.15.78:0/1293936987 conn(0x56243dafe220 msgr2=0x56243db8fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.412+0000 7f6533be4b80 1 -- 172.21.15.78:0/1293936987 shutdown_connections 2024-03-14T10:04:01.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.412+0000 7f6533be4b80 1 -- 172.21.15.78:0/1293936987 wait complete. 2024-03-14T10:04:01.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.412+0000 7f6533be4b80 20 client.27601 trim_cache size 0 max 16384 2024-03-14T10:04:01.418 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:04:01.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.412+0000 7f6520000700 1 -- 172.21.15.78:0/3933945943 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f65000a0c00 msgr2=0x7f65000a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.412+0000 7f6520000700 1 --2- 172.21.15.78:0/3933945943 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f65000a0c00 0x7f65000a30c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6518074790 tx=0x7f6504005060 comp rx=0 tx=0).stop 2024-03-14T10:04:01.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.412+0000 7f6520000700 1 -- 172.21.15.78:0/3933945943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6518073b30 msgr2=0x7f65181a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.412+0000 7f6520000700 1 --2- 172.21.15.78:0/3933945943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6518073b30 0x7f65181a5850 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f6510007f10 tx=0x7f651000d3b0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.412+0000 7f6520000700 1 -- 172.21.15.78:0/3933945943 shutdown_connections 2024-03-14T10:04:01.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.412+0000 7f6520000700 1 --2- 172.21.15.78:0/3933945943 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f65000a0c00 0x7f65000a30c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.412+0000 7f6520000700 1 --2- 172.21.15.78:0/3933945943 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6518100f20 0x7f65180730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.412+0000 7f6520000700 1 --2- 172.21.15.78:0/3933945943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6518073b30 0x7f65181a5850 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.412+0000 7f6520000700 1 --2- 172.21.15.78:0/3933945943 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6518106040 0x7f65180735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.412+0000 7f6520000700 1 -- 172.21.15.78:0/3933945943 >> 172.21.15.78:0/3933945943 conn(0x7f65180fc7a0 msgr2=0x7f65180fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.413+0000 7f6520000700 1 -- 172.21.15.78:0/3933945943 shutdown_connections 2024-03-14T10:04:01.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.413+0000 7f6520000700 1 -- 172.21.15.78:0/3933945943 wait complete. 2024-03-14T10:04:01.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:01 smithi078 ceph-mon[38143]: pgmap v304: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:01.429 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:01 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3419250514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:04:01.432 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T10:04:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:01 smithi152 ceph-mon[48279]: pgmap v304: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:01 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3419250514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:04:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:01 smithi132 ceph-mon[47959]: pgmap v304: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:01 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3419250514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:04:01.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.627+0000 7fe51d200700 1 -- 172.21.15.78:0/3506600073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5180731e0 msgr2=0x7fe5180735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.627+0000 7fe51d200700 1 --2- 172.21.15.78:0/3506600073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5180731e0 0x7fe5180735c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe508009990 tx=0x7fe508009ca0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.627+0000 7fe51d200700 1 -- 172.21.15.78:0/3506600073 shutdown_connections 2024-03-14T10:04:01.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.627+0000 7fe51d200700 1 --2- 172.21.15.78:0/3506600073 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe51810d4f0 0x7fe51810f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.627+0000 7fe51d200700 1 --2- 172.21.15.78:0/3506600073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5180731e0 0x7fe5180735c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.627+0000 7fe51d200700 1 --2- 172.21.15.78:0/3506600073 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe518073b00 0x7fe51810cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.627+0000 7fe51d200700 1 -- 172.21.15.78:0/3506600073 >> 172.21.15.78:0/3506600073 conn(0x7fe5180fc7a0 msgr2=0x7fe5180febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.627+0000 7fe51d200700 1 -- 172.21.15.78:0/3506600073 shutdown_connections 2024-03-14T10:04:01.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.627+0000 7fe51d200700 1 -- 172.21.15.78:0/3506600073 wait complete. 2024-03-14T10:04:01.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.628+0000 7fe51d200700 1 Processor -- start 2024-03-14T10:04:01.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.628+0000 7fe51d200700 1 -- start start 2024-03-14T10:04:01.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.628+0000 7fe51d200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5180731e0 0x7fe51819f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.628+0000 7fe51d200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe518073b00 0x7fe51819f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.629+0000 7fe51d200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe51810d4f0 0x7fe5181995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.629+0000 7fe51d200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe518199c50 con 0x7fe518073b00 2024-03-14T10:04:01.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.629+0000 7fe51d200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe518199dc0 con 0x7fe5180731e0 2024-03-14T10:04:01.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.629+0000 7fe51d200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe518199f80 con 0x7fe51810d4f0 2024-03-14T10:04:01.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.629+0000 7fe50fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe518073b00 0x7fe51819f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.629+0000 7fe50fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe518073b00 0x7fe51819f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40912/0 (socket says 172.21.15.78:40912) 2024-03-14T10:04:01.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.629+0000 7fe50fe00700 1 -- 172.21.15.78:0/728819502 learned_addr learned my addr 172.21.15.78:0/728819502 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:01.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.629+0000 7fe515e00700 1 --2- 172.21.15.78:0/728819502 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5180731e0 0x7fe51819f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.629+0000 7fe516800700 1 --2- 172.21.15.78:0/728819502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe51810d4f0 0x7fe5181995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.629+0000 7fe50fe00700 1 -- 172.21.15.78:0/728819502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe51810d4f0 msgr2=0x7fe5181995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.629+0000 7fe50fe00700 1 --2- 172.21.15.78:0/728819502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe51810d4f0 0x7fe5181995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.629+0000 7fe50fe00700 1 -- 172.21.15.78:0/728819502 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5180731e0 msgr2=0x7fe51819f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.629+0000 7fe50fe00700 1 --2- 172.21.15.78:0/728819502 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5180731e0 0x7fe51819f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.629+0000 7fe50fe00700 1 -- 172.21.15.78:0/728819502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5080095f0 con 0x7fe518073b00 2024-03-14T10:04:01.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.629+0000 7fe50fe00700 1 --2- 172.21.15.78:0/728819502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe518073b00 0x7fe51819f9a0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe50000e9a0 tx=0x7fe50000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.630+0000 7fe50ea00700 1 -- 172.21.15.78:0/728819502 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe500009790 con 0x7fe518073b00 2024-03-14T10:04:01.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.630+0000 7fe50ea00700 1 -- 172.21.15.78:0/728819502 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe500010430 con 0x7fe518073b00 2024-03-14T10:04:01.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.630+0000 7fe51d200700 1 -- 172.21.15.78:0/728819502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe51819a260 con 0x7fe518073b00 2024-03-14T10:04:01.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.630+0000 7fe50ea00700 1 -- 172.21.15.78:0/728819502 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe500018680 con 0x7fe518073b00 2024-03-14T10:04:01.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.630+0000 7fe51d200700 1 -- 172.21.15.78:0/728819502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5181a6500 con 0x7fe518073b00 2024-03-14T10:04:01.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.633+0000 7fe50ea00700 1 -- 172.21.15.78:0/728819502 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fe5000105a0 con 0x7fe518073b00 2024-03-14T10:04:01.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.634+0000 7fe50ea00700 1 --2- 172.21.15.78:0/728819502 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe4fc0a0c80 0x7fe4fc0a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.634+0000 7fe530c23b80 1 Processor -- start 2024-03-14T10:04:01.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.634+0000 7fe530c23b80 1 -- start start 2024-03-14T10:04:01.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.634+0000 7fe50ea00700 1 -- 172.21.15.78:0/728819502 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fe500014070 con 0x7fe518073b00 2024-03-14T10:04:01.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.634+0000 7fe515e00700 1 --2- 172.21.15.78:0/728819502 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe4fc0a0c80 0x7fe4fc0a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.634+0000 7fe530c23b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0680 0x558826dc0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.634+0000 7fe530c23b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 0x558826dca980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.634+0000 7fe530c23b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dcaec0 0x558826dcd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.635+0000 7fe530c23b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558826d19990 con 0x558826dcaec0 2024-03-14T10:04:01.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.635+0000 7fe530c23b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558826d17350 con 0x558826dc0fa0 2024-03-14T10:04:01.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.635+0000 7fe530c23b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558826d12c40 con 0x558826dc0680 2024-03-14T10:04:01.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.635+0000 7fe516800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dcaec0 0x558826dcd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.635+0000 7fe50fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 0x558826dca980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.635+0000 7fe516800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dcaec0 0x558826dcd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40916/0 (socket says 172.21.15.78:40916) 2024-03-14T10:04:01.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.635+0000 7fe50fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 0x558826dca980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:47460/0 (socket says 172.21.15.78:47460) 2024-03-14T10:04:01.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.635+0000 7fe516800700 1 -- 172.21.15.78:0/4270337977 learned_addr learned my addr 172.21.15.78:0/4270337977 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:01.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.635+0000 7fe515e00700 1 --2- 172.21.15.78:0/4270337977 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0680 0x558826dc0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.635+0000 7fe50fe00700 1 -- 172.21.15.78:0/4270337977 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0680 msgr2=0x558826dc0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.635+0000 7fe50fe00700 1 --2- 172.21.15.78:0/4270337977 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0680 0x558826dc0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.635+0000 7fe515e00700 1 --2- 172.21.15.78:0/728819502 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe4fc0a0c80 0x7fe4fc0a3140 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe508006010 tx=0x7fe508005060 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.635+0000 7fe50fe00700 1 -- 172.21.15.78:0/4270337977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dcaec0 msgr2=0x558826dcd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.635+0000 7fe50fe00700 1 --2- 172.21.15.78:0/4270337977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dcaec0 0x558826dcd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.635+0000 7fe50fe00700 1 -- 172.21.15.78:0/4270337977 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558826b943b0 con 0x558826dc0fa0 2024-03-14T10:04:01.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.636+0000 7fe50fe00700 1 --2- 172.21.15.78:0/4270337977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 0x558826dca980 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe5000c9cd0 tx=0x7fe500099520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83d0de517c264ad0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.636+0000 7fe507400700 1 -- 172.21.15.78:0/4270337977 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50009d700 con 0x558826dc0fa0 2024-03-14T10:04:01.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.636+0000 7fe507400700 1 -- 172.21.15.78:0/4270337977 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5000a1d50 con 0x558826dc0fa0 2024-03-14T10:04:01.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.636+0000 7fe507400700 1 -- 172.21.15.78:0/4270337977 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe500098fc0 con 0x558826dc0fa0 2024-03-14T10:04:01.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.636+0000 7fe530c23b80 1 -- 172.21.15.78:0/4270337977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 msgr2=0x558826dca980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.636+0000 7fe530c23b80 1 --2- 172.21.15.78:0/4270337977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 0x558826dca980 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe5000c9cd0 tx=0x7fe500099520 comp rx=0 tx=0).stop 2024-03-14T10:04:01.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.637+0000 7fe530c23b80 1 -- 172.21.15.78:0/4270337977 shutdown_connections 2024-03-14T10:04:01.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.637+0000 7fe530c23b80 1 --2- 172.21.15.78:0/4270337977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dcaec0 0x558826dcd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.637+0000 7fe530c23b80 1 --2- 172.21.15.78:0/4270337977 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0680 0x558826dc0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.637+0000 7fe530c23b80 1 --2- 172.21.15.78:0/4270337977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 0x558826dca980 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.637+0000 7fe530c23b80 1 -- 172.21.15.78:0/4270337977 >> 172.21.15.78:0/4270337977 conn(0x558826d36220 msgr2=0x558826dbbaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.637+0000 7fe530c23b80 1 -- 172.21.15.78:0/4270337977 shutdown_connections 2024-03-14T10:04:01.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.637+0000 7fe530c23b80 1 -- 172.21.15.78:0/4270337977 wait complete. 2024-03-14T10:04:01.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.637+0000 7fe530c23b80 1 -- start start 2024-03-14T10:04:01.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.637+0000 7fe530c23b80 1 Processor -- start 2024-03-14T10:04:01.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe530c23b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dc0680 0x558826dc7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe530c23b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 0x558826dc81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe530c23b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dcaec0 0x558826dc8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe530c23b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558826caed80 con 0x558826dc0680 2024-03-14T10:04:01.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe530c23b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558826caea20 con 0x558826dc0fa0 2024-03-14T10:04:01.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe530c23b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558826b966a0 con 0x558826dcaec0 2024-03-14T10:04:01.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe530c23b80 10 client.0 _ll_register_callbacks cb 0x558826b1a3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:01.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe515e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dc0680 0x558826dc7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe515e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dc0680 0x558826dc7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40930/0 (socket says 172.21.15.78:40930) 2024-03-14T10:04:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe515e00700 1 -- 172.21.15.78:0/1939362937 learned_addr learned my addr 172.21.15.78:0/1939362937 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe50fe00700 1 --2- 172.21.15.78:0/1939362937 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 0x558826dc81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe516800700 1 --2- 172.21.15.78:0/1939362937 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dcaec0 0x558826dc8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe515e00700 1 -- 172.21.15.78:0/1939362937 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dcaec0 msgr2=0x558826dc8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe515e00700 1 --2- 172.21.15.78:0/1939362937 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dcaec0 0x558826dc8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe515e00700 1 -- 172.21.15.78:0/1939362937 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 msgr2=0x558826dc81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe515e00700 1 --2- 172.21.15.78:0/1939362937 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 0x558826dc81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.638+0000 7fe515e00700 1 -- 172.21.15.78:0/1939362937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5181a6500 con 0x558826dc0680 2024-03-14T10:04:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.639+0000 7fe515e00700 1 --2- 172.21.15.78:0/1939362937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dc0680 0x558826dc7c80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe508015000 tx=0x7fe508009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ffb83e8f349846 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.639+0000 7fe4fbe00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:04:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.639+0000 7fe4fbe00700 1 -- 172.21.15.78:0/1939362937 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe508004950 con 0x558826dc0680 2024-03-14T10:04:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.639+0000 7fe4fbe00700 1 -- 172.21.15.78:0/1939362937 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fe50802c040 con 0x558826dc0680 2024-03-14T10:04:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.639+0000 7fe530c23b80 1 -- 172.21.15.78:0/1939362937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558826db3720 con 0x558826dc0680 2024-03-14T10:04:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.639+0000 7fe4fbe00700 1 -- 172.21.15.78:0/1939362937 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe508004ab0 con 0x558826dc0680 2024-03-14T10:04:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.639+0000 7fe530c23b80 10 client.17568 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.639+0000 7fe530c23b80 1 -- 172.21.15.78:0/1939362937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558826b40b60 con 0x558826dc0680 2024-03-14T10:04:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.639+0000 7fe4fbe00700 1 -- 172.21.15.78:0/1939362937 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558826db3720 con 0x558826dc0680 2024-03-14T10:04:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.640+0000 7fe4fbe00700 1 -- 172.21.15.78:0/1939362937 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe508030450 con 0x558826dc0680 2024-03-14T10:04:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.640+0000 7fe530c23b80 10 client.17568 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.640+0000 7fe530c23b80 10 client.17568 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:04:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.640+0000 7fe530c23b80 20 client.17568 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.640+0000 7fe530c23b80 1 --2- 172.21.15.78:0/1939362937 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x558826ddab40 0x558826ddcf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.640+0000 7fe530c23b80 4 client.17568 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:04:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.640+0000 7fe530c23b80 1 -- 172.21.15.78:0/1939362937 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558826d0b700 con 0x558826ddab40 2024-03-14T10:04:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.640+0000 7fe50fe00700 1 --2- 172.21.15.78:0/1939362937 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x558826ddab40 0x558826ddcf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.640+0000 7fe50fe00700 1 --2- 172.21.15.78:0/1939362937 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x558826ddab40 0x558826ddcf60 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f75cd41b81633251 server_cookie=7af7fca0f1f72feb in_seq=0 out_seq=0 2024-03-14T10:04:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.640+0000 7fe4fbe00700 10 client.17568 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:04:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.641+0000 7fe4fbe00700 1 -- 172.21.15.78:0/1939362937 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x558826d0b700 con 0x558826ddab40 2024-03-14T10:04:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.641+0000 7fe4fbe00700 10 client.17568 handle_command_reply: tid=0 2024-03-14T10:04:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.641+0000 7fe530c23b80 1 client.17568 shutdown 2024-03-14T10:04:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.641+0000 7fe530c23b80 1 -- 172.21.15.78:0/1939362937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dc0680 msgr2=0x558826dc7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.641+0000 7fe530c23b80 1 --2- 172.21.15.78:0/1939362937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dc0680 0x558826dc7c80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe508015000 tx=0x7fe508009ca0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.641+0000 7fe530c23b80 1 -- 172.21.15.78:0/1939362937 shutdown_connections 2024-03-14T10:04:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.641+0000 7fe530c23b80 1 --2- 172.21.15.78:0/1939362937 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x558826ddab40 0x558826ddcf60 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.641+0000 7fe530c23b80 1 --2- 172.21.15.78:0/1939362937 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dcaec0 0x558826dc8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.641+0000 7fe530c23b80 1 --2- 172.21.15.78:0/1939362937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dc0680 0x558826dc7c80 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.641+0000 7fe4fbe00700 0 client.17568 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:04:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.641+0000 7fe530c23b80 1 --2- 172.21.15.78:0/1939362937 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 0x558826dc81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.641+0000 7fe530c23b80 1 -- 172.21.15.78:0/1939362937 >> 172.21.15.78:0/1939362937 conn(0x558826d36220 msgr2=0x558826dbb690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.641+0000 7fe530c23b80 1 -- 172.21.15.78:0/1939362937 shutdown_connections 2024-03-14T10:04:01.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.641+0000 7fe530c23b80 1 -- 172.21.15.78:0/1939362937 wait complete. 2024-03-14T10:04:01.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.642+0000 7fe530c23b80 20 client.17568 trim_cache size 0 max 16384 2024-03-14T10:04:01.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.654+0000 7fe530c23b80 1 Processor -- start 2024-03-14T10:04:01.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.654+0000 7fe530c23b80 1 -- start start 2024-03-14T10:04:01.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.655+0000 7fe530c23b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0680 0x558826dbbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.655+0000 7fe530c23b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 0x558826dc6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.655+0000 7fe530c23b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dcaec0 0x558826dc6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.655+0000 7fe530c23b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558826dca7f0 con 0x558826dcaec0 2024-03-14T10:04:01.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.655+0000 7fe530c23b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558826dba800 con 0x558826dc0fa0 2024-03-14T10:04:01.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.655+0000 7fe530c23b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558826dc7a70 con 0x558826dc0680 2024-03-14T10:04:01.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.655+0000 7fe516800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dcaec0 0x558826dc6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.655+0000 7fe50fe00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 0x558826dc6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.655+0000 7fe515e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0680 0x558826dbbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.655+0000 7fe516800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dcaec0 0x558826dc6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40946/0 (socket says 172.21.15.78:40946) 2024-03-14T10:04:01.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.655+0000 7fe516800700 1 -- 172.21.15.78:0/2762772979 learned_addr learned my addr 172.21.15.78:0/2762772979 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:01.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.655+0000 7fe515e00700 1 -- 172.21.15.78:0/2762772979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 msgr2=0x558826dc6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.655+0000 7fe515e00700 1 --2- 172.21.15.78:0/2762772979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 0x558826dc6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.655+0000 7fe515e00700 1 -- 172.21.15.78:0/2762772979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dcaec0 msgr2=0x558826dc6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.655+0000 7fe515e00700 1 --2- 172.21.15.78:0/2762772979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dcaec0 0x558826dc6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.655+0000 7fe515e00700 1 -- 172.21.15.78:0/2762772979 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558826cd8ea0 con 0x558826dc0680 2024-03-14T10:04:01.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.656+0000 7fe515e00700 1 --2- 172.21.15.78:0/2762772979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0680 0x558826dbbc30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe508030830 tx=0x7fe50800b2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa423d10bc692cc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.656+0000 7fe507e00700 1 -- 172.21.15.78:0/2762772979 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe508030450 con 0x558826dc0680 2024-03-14T10:04:01.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.656+0000 7fe507e00700 1 -- 172.21.15.78:0/2762772979 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe50801e040 con 0x558826dc0680 2024-03-14T10:04:01.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.656+0000 7fe507e00700 1 -- 172.21.15.78:0/2762772979 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50800f8b0 con 0x558826dc0680 2024-03-14T10:04:01.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.657+0000 7fe530c23b80 1 -- 172.21.15.78:0/2762772979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0680 msgr2=0x558826dbbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.657+0000 7fe530c23b80 1 --2- 172.21.15.78:0/2762772979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0680 0x558826dbbc30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe508030830 tx=0x7fe50800b2f0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.657+0000 7fe530c23b80 1 -- 172.21.15.78:0/2762772979 shutdown_connections 2024-03-14T10:04:01.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.657+0000 7fe530c23b80 1 --2- 172.21.15.78:0/2762772979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dcaec0 0x558826dc6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.657+0000 7fe530c23b80 1 --2- 172.21.15.78:0/2762772979 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0680 0x558826dbbc30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.657+0000 7fe530c23b80 1 --2- 172.21.15.78:0/2762772979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dc0fa0 0x558826dc6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.657+0000 7fe530c23b80 1 -- 172.21.15.78:0/2762772979 >> 172.21.15.78:0/2762772979 conn(0x558826d36220 msgr2=0x558826dd6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.657+0000 7fe530c23b80 1 -- 172.21.15.78:0/2762772979 shutdown_connections 2024-03-14T10:04:01.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.657+0000 7fe530c23b80 1 -- 172.21.15.78:0/2762772979 wait complete. 2024-03-14T10:04:01.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.657+0000 7fe530c23b80 1 -- start start 2024-03-14T10:04:01.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.657+0000 7fe530c23b80 1 Processor -- start 2024-03-14T10:04:01.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe530c23b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dc0680 0x558826dcf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe530c23b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0fa0 0x558826dcfd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe530c23b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dcaec0 0x558826e2b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe530c23b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558826e2bba0 con 0x558826dc0680 2024-03-14T10:04:01.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe530c23b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x558826e2bce0 con 0x558826dcaec0 2024-03-14T10:04:01.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe530c23b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x558826e2be20 con 0x558826dc0fa0 2024-03-14T10:04:01.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe530c23b80 10 client.0 _ll_register_callbacks cb 0x558826b1a3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:01.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe515e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dc0680 0x558826dcf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe50fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0fa0 0x558826dcfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe50fe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0fa0 0x558826dcfd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:40240/0 (socket says 172.21.15.78:40240) 2024-03-14T10:04:01.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe515e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dc0680 0x558826dcf840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40950/0 (socket says 172.21.15.78:40950) 2024-03-14T10:04:01.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe50fe00700 1 -- 172.21.15.78:0/3494622585 learned_addr learned my addr 172.21.15.78:0/3494622585 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:01.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe516800700 1 --2- 172.21.15.78:0/3494622585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dcaec0 0x558826e2b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe515e00700 1 -- 172.21.15.78:0/3494622585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0fa0 msgr2=0x558826dcfd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe515e00700 1 --2- 172.21.15.78:0/3494622585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0fa0 0x558826dcfd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe515e00700 1 -- 172.21.15.78:0/3494622585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dcaec0 msgr2=0x558826e2b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe515e00700 1 --2- 172.21.15.78:0/3494622585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dcaec0 0x558826e2b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe515e00700 1 -- 172.21.15.78:0/3494622585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558826b40b60 con 0x558826dc0680 2024-03-14T10:04:01.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.658+0000 7fe516800700 1 --2- 172.21.15.78:0/3494622585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dcaec0 0x558826e2b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:04:01.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.659+0000 7fe515e00700 1 --2- 172.21.15.78:0/3494622585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dc0680 0x558826dcf840 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe5080098e0 tx=0x7fe508022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe335e23c445396 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.659+0000 7fe4fbe00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:04:01.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.659+0000 7fe4fbe00700 1 -- 172.21.15.78:0/3494622585 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50804f070 con 0x558826dc0680 2024-03-14T10:04:01.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.659+0000 7fe4fbe00700 1 -- 172.21.15.78:0/3494622585 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fe508004950 con 0x558826dc0680 2024-03-14T10:04:01.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.659+0000 7fe530c23b80 1 -- 172.21.15.78:0/3494622585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5000a1d50 con 0x558826dc0680 2024-03-14T10:04:01.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.659+0000 7fe4fbe00700 1 -- 172.21.15.78:0/3494622585 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe508004b10 con 0x558826dc0680 2024-03-14T10:04:01.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.659+0000 7fe530c23b80 10 client.17574 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:01.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.659+0000 7fe530c23b80 1 -- 172.21.15.78:0/3494622585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558826b98cd0 con 0x558826dc0680 2024-03-14T10:04:01.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.659+0000 7fe4fbe00700 1 -- 172.21.15.78:0/3494622585 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5000a1d50 con 0x558826dc0680 2024-03-14T10:04:01.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.659+0000 7fe4fbe00700 1 -- 172.21.15.78:0/3494622585 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe508052030 con 0x558826dc0680 2024-03-14T10:04:01.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.659+0000 7fe530c23b80 10 client.17574 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:01.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.659+0000 7fe530c23b80 10 client.17574 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:04:01.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.659+0000 7fe530c23b80 20 client.17574 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:01.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.659+0000 7fe530c23b80 1 --2- 172.21.15.78:0/3494622585 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x558826e2d110 0x558826e2f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.660+0000 7fe530c23b80 4 client.17574 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:04:01.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.660+0000 7fe530c23b80 1 -- 172.21.15.78:0/3494622585 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558826b61cf0 con 0x558826e2d110 2024-03-14T10:04:01.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.660+0000 7fe50fe00700 1 --2- 172.21.15.78:0/3494622585 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x558826e2d110 0x558826e2f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.660+0000 7fe4fbe00700 10 client.17574 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:04:01.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.660+0000 7fe50fe00700 1 --2- 172.21.15.78:0/3494622585 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x558826e2d110 0x558826e2f500 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e198e7b3f4c78ded server_cookie=dfbdf2a4e01608b0 in_seq=0 out_seq=0 2024-03-14T10:04:01.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.660+0000 7fe4fbe00700 1 -- 172.21.15.78:0/3494622585 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558826b61cf0 con 0x558826e2d110 2024-03-14T10:04:01.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.660+0000 7fe4fbe00700 10 client.17574 handle_command_reply: tid=0 2024-03-14T10:04:01.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.660+0000 7fe530c23b80 1 client.17574 shutdown 2024-03-14T10:04:01.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.661+0000 7fe530c23b80 1 -- 172.21.15.78:0/3494622585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dc0680 msgr2=0x558826dcf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.661+0000 7fe530c23b80 1 --2- 172.21.15.78:0/3494622585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dc0680 0x558826dcf840 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe5080098e0 tx=0x7fe508022000 comp rx=0 tx=0).stop 2024-03-14T10:04:01.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.661+0000 7fe530c23b80 1 -- 172.21.15.78:0/3494622585 shutdown_connections 2024-03-14T10:04:01.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.661+0000 7fe530c23b80 1 --2- 172.21.15.78:0/3494622585 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x558826e2d110 0x558826e2f500 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.661+0000 7fe530c23b80 1 --2- 172.21.15.78:0/3494622585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558826dcaec0 0x558826e2b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.661+0000 7fe530c23b80 1 --2- 172.21.15.78:0/3494622585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558826dc0680 0x558826dcf840 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.661+0000 7fe4fbe00700 0 client.17574 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:04:01.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.661+0000 7fe530c23b80 1 --2- 172.21.15.78:0/3494622585 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x558826dc0fa0 0x558826dcfd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.661+0000 7fe530c23b80 1 -- 172.21.15.78:0/3494622585 >> 172.21.15.78:0/3494622585 conn(0x558826d36220 msgr2=0x558826dc6510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.661+0000 7fe530c23b80 1 -- 172.21.15.78:0/3494622585 shutdown_connections 2024-03-14T10:04:01.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.661+0000 7fe530c23b80 1 -- 172.21.15.78:0/3494622585 wait complete. 2024-03-14T10:04:01.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.661+0000 7fe530c23b80 20 client.17574 trim_cache size 0 max 16384 2024-03-14T10:04:01.667 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:04:01.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.661+0000 7fe51d200700 1 -- 172.21.15.78:0/728819502 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe4fc0a0c80 msgr2=0x7fe4fc0a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.661+0000 7fe51d200700 1 --2- 172.21.15.78:0/728819502 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe4fc0a0c80 0x7fe4fc0a3140 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe508006010 tx=0x7fe508005060 comp rx=0 tx=0).stop 2024-03-14T10:04:01.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.661+0000 7fe51d200700 1 -- 172.21.15.78:0/728819502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe518073b00 msgr2=0x7fe51819f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.661+0000 7fe51d200700 1 --2- 172.21.15.78:0/728819502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe518073b00 0x7fe51819f9a0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe50000e9a0 tx=0x7fe50000ecb0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.662+0000 7fe51d200700 1 -- 172.21.15.78:0/728819502 shutdown_connections 2024-03-14T10:04:01.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.662+0000 7fe51d200700 1 --2- 172.21.15.78:0/728819502 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe4fc0a0c80 0x7fe4fc0a3140 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.662+0000 7fe51d200700 1 --2- 172.21.15.78:0/728819502 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5180731e0 0x7fe51819f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.662+0000 7fe51d200700 1 --2- 172.21.15.78:0/728819502 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe51810d4f0 0x7fe5181995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.662+0000 7fe51d200700 1 --2- 172.21.15.78:0/728819502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe518073b00 0x7fe51819f9a0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.662+0000 7fe51d200700 1 -- 172.21.15.78:0/728819502 >> 172.21.15.78:0/728819502 conn(0x7fe5180fc7a0 msgr2=0x7fe5180fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.662+0000 7fe51d200700 1 -- 172.21.15.78:0/728819502 shutdown_connections 2024-03-14T10:04:01.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.662+0000 7fe51d200700 1 -- 172.21.15.78:0/728819502 wait complete. 2024-03-14T10:04:01.680 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T10:04:01.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.878+0000 7f1ec7200700 1 -- 172.21.15.78:0/131354066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec01018d0 msgr2=0x7f1ec0105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.878+0000 7f1ec7200700 1 --2- 172.21.15.78:0/131354066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec01018d0 0x7f1ec0105910 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8009990 tx=0x7f1ea8009ca0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.878+0000 7f1ec7200700 1 -- 172.21.15.78:0/131354066 shutdown_connections 2024-03-14T10:04:01.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.878+0000 7f1ec7200700 1 --2- 172.21.15.78:0/131354066 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ec0100f20 0x7f1ec0101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.878+0000 7f1ec7200700 1 --2- 172.21.15.78:0/131354066 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ec0106040 0x7f1ec010dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.878+0000 7f1ec7200700 1 --2- 172.21.15.78:0/131354066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec01018d0 0x7f1ec0105910 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.878+0000 7f1ec7200700 1 -- 172.21.15.78:0/131354066 >> 172.21.15.78:0/131354066 conn(0x7f1ec00fc7a0 msgr2=0x7f1ec00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.878+0000 7f1ec7200700 1 -- 172.21.15.78:0/131354066 shutdown_connections 2024-03-14T10:04:01.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.878+0000 7f1ec7200700 1 -- 172.21.15.78:0/131354066 wait complete. 2024-03-14T10:04:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.879+0000 7f1ec7200700 1 Processor -- start 2024-03-14T10:04:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.879+0000 7f1ec7200700 1 -- start start 2024-03-14T10:04:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.880+0000 7f1ec7200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec0100f20 0x7f1ec019f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.880+0000 7f1ec7200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ec0106040 0x7f1ec019fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.880+0000 7f1ec7200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ec0199660 0x7f1ec0199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.880+0000 7f1ec7200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec019a160 con 0x7f1ec0100f20 2024-03-14T10:04:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.880+0000 7f1ec7200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec019a2d0 con 0x7f1ec0106040 2024-03-14T10:04:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.880+0000 7f1ec7200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec019a490 con 0x7f1ec0199660 2024-03-14T10:04:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.880+0000 7f1ebfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec0100f20 0x7f1ec019f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.880+0000 7f1ebfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec0100f20 0x7f1ec019f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40972/0 (socket says 172.21.15.78:40972) 2024-03-14T10:04:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.880+0000 7f1ebfe00700 1 -- 172.21.15.78:0/1729807613 learned_addr learned my addr 172.21.15.78:0/1729807613 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.880+0000 7f1eb7e00700 1 --2- 172.21.15.78:0/1729807613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ec0106040 0x7f1ec019fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.880+0000 7f1ec5200700 1 --2- 172.21.15.78:0/1729807613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ec0199660 0x7f1ec0199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.880+0000 7f1ebfe00700 1 -- 172.21.15.78:0/1729807613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ec0199660 msgr2=0x7f1ec0199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.880+0000 7f1ebfe00700 1 --2- 172.21.15.78:0/1729807613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ec0199660 0x7f1ec0199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.880+0000 7f1ebfe00700 1 -- 172.21.15.78:0/1729807613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ec0106040 msgr2=0x7f1ec019fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.880+0000 7f1ebfe00700 1 --2- 172.21.15.78:0/1729807613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ec0106040 0x7f1ec019fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.880+0000 7f1ebfe00700 1 -- 172.21.15.78:0/1729807613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea80095f0 con 0x7f1ec0100f20 2024-03-14T10:04:01.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.881+0000 7f1ebfe00700 1 --2- 172.21.15.78:0/1729807613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec0100f20 0x7f1ec019f570 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000e9a0 tx=0x7f1eb000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.881+0000 7f1ebd600700 1 -- 172.21.15.78:0/1729807613 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0009790 con 0x7f1ec0100f20 2024-03-14T10:04:01.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.881+0000 7f1ebd600700 1 -- 172.21.15.78:0/1729807613 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1eb0010430 con 0x7f1ec0100f20 2024-03-14T10:04:01.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.881+0000 7f1ebd600700 1 -- 172.21.15.78:0/1729807613 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0018680 con 0x7f1ec0100f20 2024-03-14T10:04:01.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.881+0000 7f1ec7200700 1 -- 172.21.15.78:0/1729807613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec01a63d0 con 0x7f1ec0100f20 2024-03-14T10:04:01.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.881+0000 7f1ec7200700 1 -- 172.21.15.78:0/1729807613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec01a6860 con 0x7f1ec0100f20 2024-03-14T10:04:01.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.884+0000 7f1ebd600700 1 -- 172.21.15.78:0/1729807613 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f1eb00105a0 con 0x7f1ec0100f20 2024-03-14T10:04:01.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.885+0000 7f1edacd1b80 1 Processor -- start 2024-03-14T10:04:01.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.885+0000 7f1edacd1b80 1 -- start start 2024-03-14T10:04:01.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.885+0000 7f1ebd600700 1 --2- 172.21.15.78:0/1729807613 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1ea40a0c00 0x7f1ea40a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.886+0000 7f1ebd600700 1 -- 172.21.15.78:0/1729807613 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f1eb0014070 con 0x7f1ec0100f20 2024-03-14T10:04:01.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.886+0000 7f1edacd1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8680 0x5610249a8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.886+0000 7f1edacd1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8fa0 0x5610249b2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.886+0000 7f1edacd1b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249b2ec0 0x5610249b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.886+0000 7f1edacd1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561024901990 con 0x5610249a8680 2024-03-14T10:04:01.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.886+0000 7f1edacd1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610248ff350 con 0x5610249a8fa0 2024-03-14T10:04:01.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.886+0000 7f1edacd1b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610248fac40 con 0x5610249b2ec0 2024-03-14T10:04:01.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.886+0000 7f1ec5200700 1 --2- 172.21.15.78:0/1729807613 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1ea40a0c00 0x7f1ea40a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.886+0000 7f1ebfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8680 0x5610249a8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.886+0000 7f1eb7e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8fa0 0x5610249b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.886+0000 7f1ebfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8680 0x5610249a8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40988/0 (socket says 172.21.15.78:40988) 2024-03-14T10:04:01.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.886+0000 7f1ebfe00700 1 -- 172.21.15.78:0/342035387 learned_addr learned my addr 172.21.15.78:0/342035387 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:01.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.886+0000 7f1ebfe00700 1 -- 172.21.15.78:0/342035387 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249b2ec0 msgr2=0x5610249b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.886+0000 7f1ec5200700 1 --2- 172.21.15.78:0/342035387 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249b2ec0 0x5610249b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.886+0000 7f1ebfe00700 1 --2- 172.21.15.78:0/342035387 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249b2ec0 0x5610249b52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.887+0000 7f1ebfe00700 1 -- 172.21.15.78:0/342035387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8fa0 msgr2=0x5610249b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.887+0000 7f1ebfe00700 1 --2- 172.21.15.78:0/342035387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8fa0 0x5610249b2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.887+0000 7f1ebfe00700 1 -- 172.21.15.78:0/342035387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56102477c3b0 con 0x5610249a8680 2024-03-14T10:04:01.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.887+0000 7f1ec5200700 1 --2- 172.21.15.78:0/342035387 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249b2ec0 0x5610249b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T10:04:01.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.887+0000 7f1ec5200700 1 --2- 172.21.15.78:0/1729807613 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1ea40a0c00 0x7f1ea40a30c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800f540 tx=0x7f1eb8009920 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.887+0000 7f1ebfe00700 1 --2- 172.21.15.78:0/342035387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8680 0x5610249a8a60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00a18e0 tx=0x7f1eb0099890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6249b83022917af0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.887+0000 7f1eb5600700 1 -- 172.21.15.78:0/342035387 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb009d7c0 con 0x5610249a8680 2024-03-14T10:04:01.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.887+0000 7f1eb5600700 1 -- 172.21.15.78:0/342035387 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1eb00a1e10 con 0x5610249a8680 2024-03-14T10:04:01.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.887+0000 7f1eb5600700 1 -- 172.21.15.78:0/342035387 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb00990e0 con 0x5610249a8680 2024-03-14T10:04:01.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.888+0000 7f1edacd1b80 1 -- 172.21.15.78:0/342035387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8680 msgr2=0x5610249a8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.888+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/342035387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8680 0x5610249a8a60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00a18e0 tx=0x7f1eb0099890 comp rx=0 tx=0).stop 2024-03-14T10:04:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.888+0000 7f1edacd1b80 1 -- 172.21.15.78:0/342035387 shutdown_connections 2024-03-14T10:04:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.888+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/342035387 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249b2ec0 0x5610249b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.888+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/342035387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8680 0x5610249a8a60 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.888+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/342035387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8fa0 0x5610249b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.888+0000 7f1edacd1b80 1 -- 172.21.15.78:0/342035387 >> 172.21.15.78:0/342035387 conn(0x56102491e220 msgr2=0x5610249a3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.888+0000 7f1edacd1b80 1 -- 172.21.15.78:0/342035387 shutdown_connections 2024-03-14T10:04:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.888+0000 7f1edacd1b80 1 -- 172.21.15.78:0/342035387 wait complete. 2024-03-14T10:04:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.888+0000 7f1edacd1b80 1 -- start start 2024-03-14T10:04:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.889+0000 7f1edacd1b80 1 Processor -- start 2024-03-14T10:04:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.889+0000 7f1edacd1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8680 0x5610249b2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.889+0000 7f1edacd1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8fa0 0x5610249afca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.889+0000 7f1edacd1b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249b2ec0 0x5610249b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.889+0000 7f1edacd1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561024896d80 con 0x5610249a8680 2024-03-14T10:04:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.889+0000 7f1edacd1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x561024896a20 con 0x5610249a8fa0 2024-03-14T10:04:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.889+0000 7f1edacd1b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x56102477e6a0 con 0x5610249b2ec0 2024-03-14T10:04:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.889+0000 7f1edacd1b80 10 client.0 _ll_register_callbacks cb 0x5610247023a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.889+0000 7f1ebfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8680 0x5610249b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.889+0000 7f1ebfe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8680 0x5610249b2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41004/0 (socket says 172.21.15.78:41004) 2024-03-14T10:04:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.889+0000 7f1ebfe00700 1 -- 172.21.15.78:0/2090293360 learned_addr learned my addr 172.21.15.78:0/2090293360 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.889+0000 7f1ec5200700 1 --2- 172.21.15.78:0/2090293360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249b2ec0 0x5610249b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.889+0000 7f1eb7e00700 1 --2- 172.21.15.78:0/2090293360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8fa0 0x5610249afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.890+0000 7f1ebfe00700 1 -- 172.21.15.78:0/2090293360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249b2ec0 msgr2=0x5610249b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.890+0000 7f1ebfe00700 1 --2- 172.21.15.78:0/2090293360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249b2ec0 0x5610249b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.890+0000 7f1ebfe00700 1 -- 172.21.15.78:0/2090293360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8fa0 msgr2=0x5610249afca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.890+0000 7f1ebfe00700 1 --2- 172.21.15.78:0/2090293360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8fa0 0x5610249afca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.890+0000 7f1ebfe00700 1 -- 172.21.15.78:0/2090293360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec01a6860 con 0x5610249a8680 2024-03-14T10:04:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.890+0000 7f1ebfe00700 1 --2- 172.21.15.78:0/2090293360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8680 0x5610249b2920 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0099920 tx=0x7f1eb0010b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b846fe8398ebdc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.890+0000 7f1eae000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:04:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.890+0000 7f1eae000700 1 -- 172.21.15.78:0/2090293360 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0099df0 con 0x5610249a8680 2024-03-14T10:04:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.890+0000 7f1eae000700 1 -- 172.21.15.78:0/2090293360 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f1eb00a1980 con 0x5610249a8680 2024-03-14T10:04:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.890+0000 7f1edacd1b80 1 -- 172.21.15.78:0/2090293360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56102499b720 con 0x5610249a8680 2024-03-14T10:04:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.890+0000 7f1eae000700 1 -- 172.21.15.78:0/2090293360 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1eb00a1b40 con 0x5610249a8680 2024-03-14T10:04:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.890+0000 7f1edacd1b80 10 client.17586 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.890+0000 7f1edacd1b80 1 -- 172.21.15.78:0/2090293360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561024728b60 con 0x5610249a8680 2024-03-14T10:04:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.891+0000 7f1eae000700 1 -- 172.21.15.78:0/2090293360 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56102499b720 con 0x5610249a8680 2024-03-14T10:04:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.891+0000 7f1eae000700 1 -- 172.21.15.78:0/2090293360 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1eb00652a0 con 0x5610249a8680 2024-03-14T10:04:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.891+0000 7f1edacd1b80 10 client.17586 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.891+0000 7f1edacd1b80 10 client.17586 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:04:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.891+0000 7f1edacd1b80 20 client.17586 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.891+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/2090293360 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5610249c2630 0x5610249c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.891+0000 7f1edacd1b80 4 client.17586 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:04:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.891+0000 7f1edacd1b80 1 -- 172.21.15.78:0/2090293360 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5610248f3700 con 0x5610249c2630 2024-03-14T10:04:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.891+0000 7f1eb7e00700 1 --2- 172.21.15.78:0/2090293360 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5610249c2630 0x5610249c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.892+0000 7f1eb7e00700 1 --2- 172.21.15.78:0/2090293360 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5610249c2630 0x5610249c4a50 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7daccb29c025378 server_cookie=a6a2f22d8d0d574 in_seq=0 out_seq=0 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.892+0000 7f1eae000700 10 client.17586 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.892+0000 7f1eae000700 1 -- 172.21.15.78:0/2090293360 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5610248f3700 con 0x5610249c2630 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.892+0000 7f1eae000700 10 client.17586 handle_command_reply: tid=0 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.892+0000 7f1edacd1b80 1 client.17586 shutdown 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.893+0000 7f1edacd1b80 1 -- 172.21.15.78:0/2090293360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8680 msgr2=0x5610249b2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.893+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/2090293360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8680 0x5610249b2920 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0099920 tx=0x7f1eb0010b80 comp rx=0 tx=0).stop 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.893+0000 7f1edacd1b80 1 -- 172.21.15.78:0/2090293360 shutdown_connections 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.893+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/2090293360 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249b2ec0 0x5610249b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.893+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/2090293360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8680 0x5610249b2920 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.893+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/2090293360 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5610249c2630 0x5610249c4a50 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.893+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/2090293360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8fa0 0x5610249afca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.893+0000 7f1eae000700 0 client.17586 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.893+0000 7f1edacd1b80 1 -- 172.21.15.78:0/2090293360 >> 172.21.15.78:0/2090293360 conn(0x56102491e220 msgr2=0x5610249a3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.893+0000 7f1edacd1b80 1 -- 172.21.15.78:0/2090293360 shutdown_connections 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.893+0000 7f1edacd1b80 1 -- 172.21.15.78:0/2090293360 wait complete. 2024-03-14T10:04:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.893+0000 7f1edacd1b80 20 client.17586 trim_cache size 0 max 16384 2024-03-14T10:04:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.906+0000 7f1edacd1b80 1 Processor -- start 2024-03-14T10:04:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.906+0000 7f1edacd1b80 1 -- start start 2024-03-14T10:04:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.906+0000 7f1edacd1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8680 0x5610249a3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.906+0000 7f1edacd1b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249a8fa0 0x5610249c9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.906+0000 7f1edacd1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249b2ec0 0x5610249be830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.906+0000 7f1edacd1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610249b09c0 con 0x5610249b2ec0 2024-03-14T10:04:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.906+0000 7f1edacd1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610249a5060 con 0x5610249a8680 2024-03-14T10:04:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.906+0000 7f1edacd1b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610249a9800 con 0x5610249a8fa0 2024-03-14T10:04:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.907+0000 7f1ec5200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249b2ec0 0x5610249be830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.907+0000 7f1ec5200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249b2ec0 0x5610249be830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41012/0 (socket says 172.21.15.78:41012) 2024-03-14T10:04:01.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.907+0000 7f1ec5200700 1 -- 172.21.15.78:0/400428549 learned_addr learned my addr 172.21.15.78:0/400428549 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:01.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.907+0000 7f1eb7e00700 1 --2- 172.21.15.78:0/400428549 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249a8fa0 0x5610249c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.907+0000 7f1ebfe00700 1 --2- 172.21.15.78:0/400428549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8680 0x5610249a3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.907+0000 7f1ec5200700 1 -- 172.21.15.78:0/400428549 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249a8fa0 msgr2=0x5610249c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.907+0000 7f1ec5200700 1 --2- 172.21.15.78:0/400428549 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249a8fa0 0x5610249c9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.907+0000 7f1ec5200700 1 -- 172.21.15.78:0/400428549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8680 msgr2=0x5610249a3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.907+0000 7f1ec5200700 1 --2- 172.21.15.78:0/400428549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8680 0x5610249a3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.907+0000 7f1ec5200700 1 -- 172.21.15.78:0/400428549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610248c0ea0 con 0x5610249b2ec0 2024-03-14T10:04:01.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.907+0000 7f1ec5200700 1 --2- 172.21.15.78:0/400428549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249b2ec0 0x5610249be830 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8019780 tx=0x7f1eb800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ade83d1a1fc033 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.908+0000 7f1eb6000700 1 -- 172.21.15.78:0/400428549 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8029070 con 0x5610249b2ec0 2024-03-14T10:04:01.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.908+0000 7f1eb6000700 1 -- 172.21.15.78:0/400428549 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1eb800f040 con 0x5610249b2ec0 2024-03-14T10:04:01.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.908+0000 7f1eb6000700 1 -- 172.21.15.78:0/400428549 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb800e040 con 0x5610249b2ec0 2024-03-14T10:04:01.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.908+0000 7f1edacd1b80 1 -- 172.21.15.78:0/400428549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249b2ec0 msgr2=0x5610249be830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.908+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/400428549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249b2ec0 0x5610249be830 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8019780 tx=0x7f1eb800b040 comp rx=0 tx=0).stop 2024-03-14T10:04:01.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.908+0000 7f1edacd1b80 1 -- 172.21.15.78:0/400428549 shutdown_connections 2024-03-14T10:04:01.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.908+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/400428549 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8680 0x5610249a3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.908+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/400428549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249b2ec0 0x5610249be830 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.908+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/400428549 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249a8fa0 0x5610249c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.908+0000 7f1edacd1b80 1 -- 172.21.15.78:0/400428549 >> 172.21.15.78:0/400428549 conn(0x56102491e220 msgr2=0x5610249be120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.908+0000 7f1edacd1b80 1 -- 172.21.15.78:0/400428549 shutdown_connections 2024-03-14T10:04:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.908+0000 7f1edacd1b80 1 -- 172.21.15.78:0/400428549 wait complete. 2024-03-14T10:04:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.909+0000 7f1edacd1b80 1 -- start start 2024-03-14T10:04:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.909+0000 7f1edacd1b80 1 Processor -- start 2024-03-14T10:04:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.909+0000 7f1edacd1b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8680 0x5610249b9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.909+0000 7f1edacd1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8fa0 0x5610249b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.909+0000 7f1edacd1b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249b2ec0 0x5610249b8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.909+0000 7f1edacd1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610249b8900 con 0x5610249a8fa0 2024-03-14T10:04:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.909+0000 7f1edacd1b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610249b8a70 con 0x5610249a8680 2024-03-14T10:04:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.909+0000 7f1edacd1b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610249b8c30 con 0x5610249b2ec0 2024-03-14T10:04:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.909+0000 7f1edacd1b80 10 client.0 _ll_register_callbacks cb 0x7f1eb009d070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.910+0000 7f1eb7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8fa0 0x5610249b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.910+0000 7f1eb7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8fa0 0x5610249b7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41028/0 (socket says 172.21.15.78:41028) 2024-03-14T10:04:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.910+0000 7f1eb7e00700 1 -- 172.21.15.78:0/2635352052 learned_addr learned my addr 172.21.15.78:0/2635352052 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.910+0000 7f1ebfe00700 1 --2- 172.21.15.78:0/2635352052 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8680 0x5610249b9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.910+0000 7f1ec5200700 1 --2- 172.21.15.78:0/2635352052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249b2ec0 0x5610249b8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.910+0000 7f1eb7e00700 1 -- 172.21.15.78:0/2635352052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249b2ec0 msgr2=0x5610249b8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.910+0000 7f1eb7e00700 1 --2- 172.21.15.78:0/2635352052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249b2ec0 0x5610249b8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.910+0000 7f1eb7e00700 1 -- 172.21.15.78:0/2635352052 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8680 msgr2=0x5610249b9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.910+0000 7f1eb7e00700 1 --2- 172.21.15.78:0/2635352052 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8680 0x5610249b9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.910+0000 7f1eb7e00700 1 -- 172.21.15.78:0/2635352052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb8019460 con 0x5610249a8fa0 2024-03-14T10:04:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.910+0000 7f1eb7e00700 1 --2- 172.21.15.78:0/2635352052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8fa0 0x5610249b7df0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f1ec006a350 tx=0x7f1ea8005500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425450b251abb416 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.910+0000 7f1eae000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:04:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.911+0000 7f1eae000700 1 -- 172.21.15.78:0/2635352052 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea801d040 con 0x5610249a8fa0 2024-03-14T10:04:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.911+0000 7f1eae000700 1 -- 172.21.15.78:0/2635352052 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f1ea800b8f0 con 0x5610249a8fa0 2024-03-14T10:04:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.911+0000 7f1edacd1b80 1 -- 172.21.15.78:0/2635352052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1eb00a1e10 con 0x5610249a8fa0 2024-03-14T10:04:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.911+0000 7f1eae000700 1 -- 172.21.15.78:0/2635352052 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ea8005980 con 0x5610249a8fa0 2024-03-14T10:04:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.911+0000 7f1edacd1b80 10 client.17592 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.911+0000 7f1edacd1b80 1 -- 172.21.15.78:0/2635352052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561024780cd0 con 0x5610249a8fa0 2024-03-14T10:04:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.911+0000 7f1eae000700 1 -- 172.21.15.78:0/2635352052 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1eb00a1e10 con 0x5610249a8fa0 2024-03-14T10:04:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.911+0000 7f1eae000700 1 -- 172.21.15.78:0/2635352052 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1ea800f460 con 0x5610249a8fa0 2024-03-14T10:04:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.911+0000 7f1edacd1b80 10 client.17592 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.911+0000 7f1edacd1b80 10 client.17592 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:04:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.911+0000 7f1edacd1b80 20 client.17592 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.911+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/2635352052 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561024a15770 0x561024a17b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.911+0000 7f1edacd1b80 4 client.17592 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:04:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.911+0000 7f1edacd1b80 1 -- 172.21.15.78:0/2635352052 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561024749cf0 con 0x561024a15770 2024-03-14T10:04:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.912+0000 7f1ebfe00700 1 --2- 172.21.15.78:0/2635352052 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561024a15770 0x561024a17b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.912+0000 7f1eae000700 10 client.17592 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.912+0000 7f1ebfe00700 1 --2- 172.21.15.78:0/2635352052 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561024a15770 0x561024a17b90 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32de10ce56cb3eab server_cookie=6af64522f0929b14 in_seq=0 out_seq=0 2024-03-14T10:04:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.912+0000 7f1eae000700 1 -- 172.21.15.78:0/2635352052 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561024749cf0 con 0x561024a15770 2024-03-14T10:04:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.912+0000 7f1eae000700 10 client.17592 handle_command_reply: tid=0 2024-03-14T10:04:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.912+0000 7f1edacd1b80 1 client.17592 shutdown 2024-03-14T10:04:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.913+0000 7f1edacd1b80 1 -- 172.21.15.78:0/2635352052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8fa0 msgr2=0x5610249b7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.913+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/2635352052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8fa0 0x5610249b7df0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f1ec006a350 tx=0x7f1ea8005500 comp rx=0 tx=0).stop 2024-03-14T10:04:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.913+0000 7f1edacd1b80 1 -- 172.21.15.78:0/2635352052 shutdown_connections 2024-03-14T10:04:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.913+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/2635352052 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x561024a15770 0x561024a17b90 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.913+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/2635352052 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5610249a8680 0x5610249b9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.913+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/2635352052 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5610249b2ec0 0x5610249b8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.913+0000 7f1eae000700 0 client.17592 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.913+0000 7f1edacd1b80 1 --2- 172.21.15.78:0/2635352052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610249a8fa0 0x5610249b7df0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.913+0000 7f1edacd1b80 1 -- 172.21.15.78:0/2635352052 >> 172.21.15.78:0/2635352052 conn(0x56102491e220 msgr2=0x5610249be9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.913+0000 7f1edacd1b80 1 -- 172.21.15.78:0/2635352052 shutdown_connections 2024-03-14T10:04:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.913+0000 7f1edacd1b80 1 -- 172.21.15.78:0/2635352052 wait complete. 2024-03-14T10:04:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.913+0000 7f1edacd1b80 20 client.17592 trim_cache size 0 max 16384 2024-03-14T10:04:01.919 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:04:01.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.913+0000 7f1ec7200700 1 -- 172.21.15.78:0/1729807613 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1ea40a0c00 msgr2=0x7f1ea40a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.913+0000 7f1ec7200700 1 --2- 172.21.15.78:0/1729807613 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1ea40a0c00 0x7f1ea40a30c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800f540 tx=0x7f1eb8009920 comp rx=0 tx=0).stop 2024-03-14T10:04:01.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.913+0000 7f1ec7200700 1 -- 172.21.15.78:0/1729807613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec0100f20 msgr2=0x7f1ec019f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:01.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.913+0000 7f1ec7200700 1 --2- 172.21.15.78:0/1729807613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec0100f20 0x7f1ec019f570 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000e9a0 tx=0x7f1eb000ecb0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.914+0000 7f1ec7200700 1 -- 172.21.15.78:0/1729807613 shutdown_connections 2024-03-14T10:04:01.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.914+0000 7f1ec7200700 1 --2- 172.21.15.78:0/1729807613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1ec0199660 0x7f1ec0199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.914+0000 7f1ec7200700 1 --2- 172.21.15.78:0/1729807613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ec0100f20 0x7f1ec019f570 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.914+0000 7f1ec7200700 1 --2- 172.21.15.78:0/1729807613 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1ea40a0c00 0x7f1ea40a30c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.914+0000 7f1ec7200700 1 --2- 172.21.15.78:0/1729807613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1ec0106040 0x7f1ec019fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:01.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.914+0000 7f1ec7200700 1 -- 172.21.15.78:0/1729807613 >> 172.21.15.78:0/1729807613 conn(0x7f1ec00fc7a0 msgr2=0x7f1ec00feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:01.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.914+0000 7f1ec7200700 1 -- 172.21.15.78:0/1729807613 shutdown_connections 2024-03-14T10:04:01.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:01.914+0000 7f1ec7200700 1 -- 172.21.15.78:0/1729807613 wait complete. 2024-03-14T10:04:02.932 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-14T10:04:02.932 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-14T10:04:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.135+0000 7fafbb200700 1 -- 172.21.15.78:0/650247928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafb4100b30 msgr2=0x7fafb4100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.135+0000 7fafbb200700 1 --2- 172.21.15.78:0/650247928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafb4100b30 0x7fafb4100fb0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fafa40099c0 tx=0x7fafa4009cd0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.135+0000 7fafbb200700 1 -- 172.21.15.78:0/650247928 shutdown_connections 2024-03-14T10:04:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.135+0000 7fafbb200700 1 --2- 172.21.15.78:0/650247928 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafb4106bf0 0x7fafb4106fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.135+0000 7fafbb200700 1 --2- 172.21.15.78:0/650247928 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fafb41014f0 0x7fafb410d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.135+0000 7fafbb200700 1 --2- 172.21.15.78:0/650247928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafb4100b30 0x7fafb4100fb0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.135+0000 7fafbb200700 1 -- 172.21.15.78:0/650247928 >> 172.21.15.78:0/650247928 conn(0x7fafb40fc620 msgr2=0x7fafb40fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.135+0000 7fafbb200700 1 -- 172.21.15.78:0/650247928 shutdown_connections 2024-03-14T10:04:03.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.135+0000 7fafbb200700 1 -- 172.21.15.78:0/650247928 wait complete. 2024-03-14T10:04:03.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.136+0000 7fafbb200700 1 Processor -- start 2024-03-14T10:04:03.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.136+0000 7fafbb200700 1 -- start start 2024-03-14T10:04:03.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.136+0000 7fafbb200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fafb4100b30 0x7fafb4199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.136+0000 7fafbb200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafb41014f0 0x7fafb4199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.136+0000 7fafbb200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafb4106bf0 0x7fafb419def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.137+0000 7fafbb200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb419e530 con 0x7fafb4106bf0 2024-03-14T10:04:03.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.137+0000 7fafbb200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb419e6a0 con 0x7fafb41014f0 2024-03-14T10:04:03.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.137+0000 7fafbb200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb419e860 con 0x7fafb4100b30 2024-03-14T10:04:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.137+0000 7fafb9400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafb41014f0 0x7fafb4199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.137+0000 7fafba800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafb4106bf0 0x7fafb419def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.137+0000 7fafba800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafb4106bf0 0x7fafb419def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41052/0 (socket says 172.21.15.78:41052) 2024-03-14T10:04:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.137+0000 7fafb9400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafb41014f0 0x7fafb4199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:47582/0 (socket says 172.21.15.78:47582) 2024-03-14T10:04:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.137+0000 7fafba800700 1 -- 172.21.15.78:0/783973604 learned_addr learned my addr 172.21.15.78:0/783973604 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.137+0000 7fafb9e00700 1 --2- 172.21.15.78:0/783973604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fafb4100b30 0x7fafb4199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.137+0000 7fafb9400700 1 -- 172.21.15.78:0/783973604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fafb4100b30 msgr2=0x7fafb4199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.137+0000 7fafb9400700 1 --2- 172.21.15.78:0/783973604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fafb4100b30 0x7fafb4199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.137+0000 7fafb9400700 1 -- 172.21.15.78:0/783973604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafb4106bf0 msgr2=0x7fafb419def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.137+0000 7fafb9400700 1 --2- 172.21.15.78:0/783973604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafb4106bf0 0x7fafb419def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.137+0000 7fafb9400700 1 -- 172.21.15.78:0/783973604 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa40095f0 con 0x7fafb41014f0 2024-03-14T10:04:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.137+0000 7fafba800700 1 --2- 172.21.15.78:0/783973604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafb4106bf0 0x7fafb419def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:04:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.138+0000 7fafb9400700 1 --2- 172.21.15.78:0/783973604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafb41014f0 0x7fafb4199b50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fafa40098e0 tx=0x7fafa400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.138+0000 7fafaea00700 1 -- 172.21.15.78:0/783973604 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa401c070 con 0x7fafb41014f0 2024-03-14T10:04:03.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.138+0000 7fafbb200700 1 -- 172.21.15.78:0/783973604 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb419eae0 con 0x7fafb41014f0 2024-03-14T10:04:03.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.138+0000 7fafaea00700 1 -- 172.21.15.78:0/783973604 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fafa400fd00 con 0x7fafb41014f0 2024-03-14T10:04:03.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.138+0000 7fafaea00700 1 -- 172.21.15.78:0/783973604 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4022c60 con 0x7fafb41014f0 2024-03-14T10:04:03.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.138+0000 7fafbb200700 1 -- 172.21.15.78:0/783973604 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb41a6380 con 0x7fafb41014f0 2024-03-14T10:04:03.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.141+0000 7fafaea00700 1 -- 172.21.15.78:0/783973604 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fafa4020070 con 0x7fafb41014f0 2024-03-14T10:04:03.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.142+0000 7fafd0358b80 1 Processor -- start 2024-03-14T10:04:03.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.142+0000 7fafd0358b80 1 -- start start 2024-03-14T10:04:03.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.142+0000 7fafd0358b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42ca88e0 0x55db42ca8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.142+0000 7fafd0358b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca9200 0x55db42cb3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.142+0000 7fafd0358b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42cb40b0 0x55db42cb64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.142+0000 7fafd0358b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db42c02a10 con 0x55db42cb40b0 2024-03-14T10:04:03.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.142+0000 7fafd0358b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db42c003d0 con 0x55db42ca88e0 2024-03-14T10:04:03.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.142+0000 7fafd0358b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db42bfbcc0 con 0x55db42ca9200 2024-03-14T10:04:03.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.143+0000 7fafaea00700 1 --2- 172.21.15.78:0/783973604 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7faf9c0a0c00 0x7faf9c0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.143+0000 7fafb9400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42cb40b0 0x55db42cb64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.143+0000 7fafb9e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca9200 0x55db42cb3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.143+0000 7fafba800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42ca88e0 0x55db42ca8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.143+0000 7fafaea00700 1 -- 172.21.15.78:0/783973604 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fafa40cd430 con 0x7fafb41014f0 2024-03-14T10:04:03.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.143+0000 7fafb9400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42cb40b0 0x55db42cb64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41054/0 (socket says 172.21.15.78:41054) 2024-03-14T10:04:03.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.143+0000 7fafb9400700 1 -- 172.21.15.78:0/3309758394 learned_addr learned my addr 172.21.15.78:0/3309758394 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:03.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.143+0000 7fafba800700 1 --2- 172.21.15.78:0/783973604 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7faf9c0a0c00 0x7faf9c0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.143+0000 7fafb9400700 1 -- 172.21.15.78:0/3309758394 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca9200 msgr2=0x55db42cb3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:03.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.143+0000 7fafb9400700 1 --2- 172.21.15.78:0/3309758394 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca9200 0x55db42cb3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.143+0000 7fafb9400700 1 -- 172.21.15.78:0/3309758394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42ca88e0 msgr2=0x55db42ca8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:03.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.143+0000 7fafb9400700 1 --2- 172.21.15.78:0/3309758394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42ca88e0 0x55db42ca8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.143+0000 7fafb9400700 1 -- 172.21.15.78:0/3309758394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db42a7d430 con 0x55db42cb40b0 2024-03-14T10:04:03.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.143+0000 7fafb9400700 1 --2- 172.21.15.78:0/3309758394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42cb40b0 0x55db42cb64a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fafa401e040 tx=0x7fafa40047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67425e954f97483b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:03.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.144+0000 7fafba800700 1 --2- 172.21.15.78:0/783973604 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7faf9c0a0c00 0x7faf9c0a30c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009f00 tx=0x7fafb000b040 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:03.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.144+0000 7fafa3000700 1 -- 172.21.15.78:0/3309758394 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa40a0050 con 0x55db42cb40b0 2024-03-14T10:04:03.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.144+0000 7fafa3000700 1 -- 172.21.15.78:0/3309758394 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fafa4099f20 con 0x55db42cb40b0 2024-03-14T10:04:03.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.144+0000 7fafa3000700 1 -- 172.21.15.78:0/3309758394 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa40caea0 con 0x55db42cb40b0 2024-03-14T10:04:03.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.144+0000 7fafd0358b80 1 -- 172.21.15.78:0/3309758394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42cb40b0 msgr2=0x55db42cb64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.144+0000 7fafd0358b80 1 --2- 172.21.15.78:0/3309758394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42cb40b0 0x55db42cb64a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fafa401e040 tx=0x7fafa40047a0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.144+0000 7fafd0358b80 1 -- 172.21.15.78:0/3309758394 shutdown_connections 2024-03-14T10:04:03.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.144+0000 7fafd0358b80 1 --2- 172.21.15.78:0/3309758394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42ca88e0 0x55db42ca8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.144+0000 7fafd0358b80 1 --2- 172.21.15.78:0/3309758394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42cb40b0 0x55db42cb64a0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.144+0000 7fafd0358b80 1 --2- 172.21.15.78:0/3309758394 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca9200 0x55db42cb3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.144+0000 7fafd0358b80 1 -- 172.21.15.78:0/3309758394 >> 172.21.15.78:0/3309758394 conn(0x55db42c1f2a0 msgr2=0x55db42ca4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:03.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.145+0000 7fafd0358b80 1 -- 172.21.15.78:0/3309758394 shutdown_connections 2024-03-14T10:04:03.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.145+0000 7fafd0358b80 1 -- 172.21.15.78:0/3309758394 wait complete. 2024-03-14T10:04:03.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.145+0000 7fafd0358b80 1 -- start start 2024-03-14T10:04:03.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.145+0000 7fafd0358b80 1 Processor -- start 2024-03-14T10:04:03.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.146+0000 7fafd0358b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42ca88e0 0x55db42cb4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.146+0000 7fafd0358b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca9200 0x55db42cb49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.146+0000 7fafd0358b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42cb40b0 0x55db42cbf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.146+0000 7fafd0358b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db42b97e00 con 0x55db42cb40b0 2024-03-14T10:04:03.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.146+0000 7fafd0358b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db42b97aa0 con 0x55db42ca88e0 2024-03-14T10:04:03.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.146+0000 7fafd0358b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db42a7f720 con 0x55db42ca9200 2024-03-14T10:04:03.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.146+0000 7fafd0358b80 10 client.0 _ll_register_callbacks cb 0x55db42a03420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:03.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.146+0000 7fafba800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42cb40b0 0x55db42cbf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.146+0000 7fafba800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42cb40b0 0x55db42cbf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41066/0 (socket says 172.21.15.78:41066) 2024-03-14T10:04:03.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.146+0000 7fafba800700 1 -- 172.21.15.78:0/2604206834 learned_addr learned my addr 172.21.15.78:0/2604206834 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.146+0000 7fafb9e00700 1 --2- 172.21.15.78:0/2604206834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42ca88e0 0x55db42cb4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.147+0000 7fafb9400700 1 --2- 172.21.15.78:0/2604206834 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca9200 0x55db42cb49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.147+0000 7fafba800700 1 -- 172.21.15.78:0/2604206834 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca9200 msgr2=0x55db42cb49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.147+0000 7fafba800700 1 --2- 172.21.15.78:0/2604206834 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca9200 0x55db42cb49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.147+0000 7fafba800700 1 -- 172.21.15.78:0/2604206834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42ca88e0 msgr2=0x55db42cb4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.147+0000 7fafba800700 1 --2- 172.21.15.78:0/2604206834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42ca88e0 0x55db42cb4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.147+0000 7fafba800700 1 -- 172.21.15.78:0/2604206834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb41a6380 con 0x55db42cb40b0 2024-03-14T10:04:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.147+0000 7fafba800700 1 --2- 172.21.15.78:0/2604206834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42cb40b0 0x55db42cbf510 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009690 tx=0x7fafb0004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11a05a95a6b50372 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.147+0000 7faf97400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:04:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.147+0000 7faf97400700 1 -- 172.21.15.78:0/2604206834 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb001ada0 con 0x55db42cb40b0 2024-03-14T10:04:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.147+0000 7faf97400700 1 -- 172.21.15.78:0/2604206834 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fafb0024480 con 0x55db42cb40b0 2024-03-14T10:04:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.147+0000 7fafd0358b80 1 -- 172.21.15.78:0/2604206834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55db42c9c7a0 con 0x55db42cb40b0 2024-03-14T10:04:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.148+0000 7faf97400700 1 -- 172.21.15.78:0/2604206834 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafb0024640 con 0x55db42cb40b0 2024-03-14T10:04:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.148+0000 7fafd0358b80 10 client.17604 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:03.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.148+0000 7fafd0358b80 1 -- 172.21.15.78:0/2604206834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55db42a29be0 con 0x55db42cb40b0 2024-03-14T10:04:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.148+0000 7faf97400700 1 -- 172.21.15.78:0/2604206834 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55db42c9c7a0 con 0x55db42cb40b0 2024-03-14T10:04:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.148+0000 7faf97400700 1 -- 172.21.15.78:0/2604206834 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fafb0025760 con 0x55db42cb40b0 2024-03-14T10:04:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.148+0000 7fafd0358b80 10 client.17604 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.148+0000 7fafd0358b80 10 client.17604 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:04:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.148+0000 7fafd0358b80 20 client.17604 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.148+0000 7fafd0358b80 1 --2- 172.21.15.78:0/2604206834 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55db42cc38b0 0x55db42cc5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.148+0000 7fafd0358b80 4 client.17604 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:04:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.148+0000 7fafd0358b80 1 -- 172.21.15.78:0/2604206834 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55db42bf4780 con 0x55db42cc38b0 2024-03-14T10:04:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.149+0000 7fafb9e00700 1 --2- 172.21.15.78:0/2604206834 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55db42cc38b0 0x55db42cc5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.149+0000 7fafb9e00700 1 --2- 172.21.15.78:0/2604206834 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55db42cc38b0 0x55db42cc5cd0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2584a4a438982d30 server_cookie=efd67f261714fb22 in_seq=0 out_seq=0 2024-03-14T10:04:03.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.149+0000 7faf97400700 10 client.17604 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.150+0000 7faf97400700 1 -- 172.21.15.78:0/2604206834 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55db42bf4780 con 0x55db42cc38b0 2024-03-14T10:04:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.150+0000 7faf97400700 10 client.17604 handle_command_reply: tid=0 2024-03-14T10:04:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.150+0000 7fafd0358b80 1 client.17604 shutdown 2024-03-14T10:04:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.150+0000 7fafd0358b80 1 -- 172.21.15.78:0/2604206834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42cb40b0 msgr2=0x55db42cbf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.150+0000 7fafd0358b80 1 --2- 172.21.15.78:0/2604206834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42cb40b0 0x55db42cbf510 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009690 tx=0x7fafb0004b40 comp rx=0 tx=0).stop 2024-03-14T10:04:03.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.150+0000 7fafd0358b80 1 -- 172.21.15.78:0/2604206834 shutdown_connections 2024-03-14T10:04:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.150+0000 7fafd0358b80 1 --2- 172.21.15.78:0/2604206834 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55db42cc38b0 0x55db42cc5cd0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.150+0000 7fafd0358b80 1 --2- 172.21.15.78:0/2604206834 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42ca88e0 0x55db42cb4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.150+0000 7fafd0358b80 1 --2- 172.21.15.78:0/2604206834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42cb40b0 0x55db42cbf510 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.150+0000 7fafd0358b80 1 --2- 172.21.15.78:0/2604206834 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca9200 0x55db42cb49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.150+0000 7faf97400700 0 client.17604 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.150+0000 7fafd0358b80 1 -- 172.21.15.78:0/2604206834 >> 172.21.15.78:0/2604206834 conn(0x55db42c1f2a0 msgr2=0x55db42ca4900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.150+0000 7fafd0358b80 1 -- 172.21.15.78:0/2604206834 shutdown_connections 2024-03-14T10:04:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.150+0000 7fafd0358b80 1 -- 172.21.15.78:0/2604206834 wait complete. 2024-03-14T10:04:03.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.150+0000 7fafd0358b80 20 client.17604 trim_cache size 0 max 16384 2024-03-14T10:04:03.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.167+0000 7fafd0358b80 1 Processor -- start 2024-03-14T10:04:03.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.167+0000 7fafd0358b80 1 -- start start 2024-03-14T10:04:03.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.167+0000 7fafd0358b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42ca88e0 0x55db42cb9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.167+0000 7fafd0358b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca9200 0x55db42cb0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.167+0000 7fafd0358b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42cb40b0 0x55db42cb0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.167+0000 7fafd0358b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db42cca9f0 con 0x55db42ca88e0 2024-03-14T10:04:03.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.167+0000 7fafd0358b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db42caf100 con 0x55db42cb40b0 2024-03-14T10:04:03.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.167+0000 7fafd0358b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db42caee80 con 0x55db42ca9200 2024-03-14T10:04:03.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.167+0000 7fafb9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42ca88e0 0x55db42cb9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.167+0000 7fafb9e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42ca88e0 0x55db42cb9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41078/0 (socket says 172.21.15.78:41078) 2024-03-14T10:04:03.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.167+0000 7fafb9e00700 1 -- 172.21.15.78:0/1240902296 learned_addr learned my addr 172.21.15.78:0/1240902296 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:03.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.168+0000 7fafb9400700 1 --2- 172.21.15.78:0/1240902296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca9200 0x55db42cb0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.168+0000 7fafba800700 1 --2- 172.21.15.78:0/1240902296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42cb40b0 0x55db42cb0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.168+0000 7fafb9e00700 1 -- 172.21.15.78:0/1240902296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca9200 msgr2=0x55db42cb0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:03.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.168+0000 7fafb9e00700 1 --2- 172.21.15.78:0/1240902296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca9200 0x55db42cb0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.168+0000 7fafb9e00700 1 -- 172.21.15.78:0/1240902296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42cb40b0 msgr2=0x55db42cb0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:03.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.168+0000 7fafb9e00700 1 --2- 172.21.15.78:0/1240902296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42cb40b0 0x55db42cb0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.168+0000 7fafb9e00700 1 -- 172.21.15.78:0/1240902296 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db42cb3210 con 0x55db42ca88e0 2024-03-14T10:04:03.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.168+0000 7fafba800700 1 --2- 172.21.15.78:0/1240902296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42cb40b0 0x55db42cb0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:04:03.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.168+0000 7fafb9e00700 1 --2- 172.21.15.78:0/1240902296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42ca88e0 0x55db42cb9360 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x55db42cc65e0 tx=0x7fafa8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ea2942d4cdeeeb server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:03.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.168+0000 7fafa3a00700 1 -- 172.21.15.78:0/1240902296 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8017030 con 0x55db42ca88e0 2024-03-14T10:04:03.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.168+0000 7fafa3a00700 1 -- 172.21.15.78:0/1240902296 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fafa8018040 con 0x55db42ca88e0 2024-03-14T10:04:03.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.168+0000 7fafa3a00700 1 -- 172.21.15.78:0/1240902296 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa801e660 con 0x55db42ca88e0 2024-03-14T10:04:03.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.169+0000 7fafd0358b80 1 -- 172.21.15.78:0/1240902296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42ca88e0 msgr2=0x55db42cb9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.169+0000 7fafd0358b80 1 --2- 172.21.15.78:0/1240902296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42ca88e0 0x55db42cb9360 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x55db42cc65e0 tx=0x7fafa8009260 comp rx=0 tx=0).stop 2024-03-14T10:04:03.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.169+0000 7fafd0358b80 1 -- 172.21.15.78:0/1240902296 shutdown_connections 2024-03-14T10:04:03.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.169+0000 7fafd0358b80 1 --2- 172.21.15.78:0/1240902296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42cb40b0 0x55db42cb0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.169+0000 7fafd0358b80 1 --2- 172.21.15.78:0/1240902296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42ca88e0 0x55db42cb9360 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.169+0000 7fafd0358b80 1 --2- 172.21.15.78:0/1240902296 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca9200 0x55db42cb0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.169+0000 7fafd0358b80 1 -- 172.21.15.78:0/1240902296 >> 172.21.15.78:0/1240902296 conn(0x55db42c1f2a0 msgr2=0x55db42cb1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:03.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.169+0000 7fafd0358b80 1 -- 172.21.15.78:0/1240902296 shutdown_connections 2024-03-14T10:04:03.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.169+0000 7fafd0358b80 1 -- 172.21.15.78:0/1240902296 wait complete. 2024-03-14T10:04:03.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.169+0000 7fafd0358b80 1 -- start start 2024-03-14T10:04:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.170+0000 7fafd0358b80 1 Processor -- start 2024-03-14T10:04:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.170+0000 7fafd0358b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca88e0 0x55db42d0f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.170+0000 7fafd0358b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42ca9200 0x55db42d0f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.170+0000 7fafd0358b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42cb40b0 0x55db42d0a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.170+0000 7fafd0358b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db42d0ad30 con 0x55db42ca9200 2024-03-14T10:04:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.170+0000 7fafd0358b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db42d0ae70 con 0x55db42cb40b0 2024-03-14T10:04:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.170+0000 7fafd0358b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db42d0b030 con 0x55db42ca88e0 2024-03-14T10:04:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.170+0000 7fafd0358b80 10 client.0 _ll_register_callbacks cb 0x7fafb0008f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.170+0000 7fafb9400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42ca9200 0x55db42d0f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.170+0000 7fafb9400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42ca9200 0x55db42d0f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41092/0 (socket says 172.21.15.78:41092) 2024-03-14T10:04:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.170+0000 7fafb9400700 1 -- 172.21.15.78:0/1868263429 learned_addr learned my addr 172.21.15.78:0/1868263429 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.170+0000 7fafb9e00700 1 --2- 172.21.15.78:0/1868263429 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca88e0 0x55db42d0f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.170+0000 7fafba800700 1 --2- 172.21.15.78:0/1868263429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42cb40b0 0x55db42d0a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.171+0000 7fafb9400700 1 -- 172.21.15.78:0/1868263429 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca88e0 msgr2=0x55db42d0f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:03.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.171+0000 7fafb9400700 1 --2- 172.21.15.78:0/1868263429 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca88e0 0x55db42d0f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.171+0000 7fafb9400700 1 -- 172.21.15.78:0/1868263429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42cb40b0 msgr2=0x55db42d0a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:03.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.171+0000 7fafb9400700 1 --2- 172.21.15.78:0/1868263429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42cb40b0 0x55db42d0a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.171+0000 7fafb9400700 1 -- 172.21.15.78:0/1868263429 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa8009b90 con 0x55db42ca9200 2024-03-14T10:04:03.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.171+0000 7fafb9400700 1 --2- 172.21.15.78:0/1868263429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42ca9200 0x55db42d0f630 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fafa409a0e0 tx=0x7fafa40852e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ff0091f702a48f server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:03.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.171+0000 7faf97400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:04:03.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.171+0000 7faf97400700 1 -- 172.21.15.78:0/1868263429 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa40a0040 con 0x55db42ca9200 2024-03-14T10:04:03.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.171+0000 7faf97400700 1 -- 172.21.15.78:0/1868263429 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fafa40a27d0 con 0x55db42ca9200 2024-03-14T10:04:03.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.171+0000 7fafd0358b80 1 -- 172.21.15.78:0/1868263429 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fafa4099f20 con 0x55db42ca9200 2024-03-14T10:04:03.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.171+0000 7faf97400700 1 -- 172.21.15.78:0/1868263429 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafa40caea0 con 0x55db42ca9200 2024-03-14T10:04:03.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.171+0000 7fafd0358b80 10 client.17610 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:03.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.171+0000 7fafd0358b80 1 -- 172.21.15.78:0/1868263429 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55db42bc1f20 con 0x55db42ca9200 2024-03-14T10:04:03.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.172+0000 7faf97400700 1 -- 172.21.15.78:0/1868263429 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fafa4064020 con 0x55db42ca9200 2024-03-14T10:04:03.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.172+0000 7faf97400700 1 -- 172.21.15.78:0/1868263429 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fafa40998d0 con 0x55db42ca9200 2024-03-14T10:04:03.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.172+0000 7fafd0358b80 10 client.17610 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:03.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.172+0000 7fafd0358b80 10 client.17610 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:04:03.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.172+0000 7fafd0358b80 20 client.17610 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:03.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.172+0000 7fafd0358b80 1 --2- 172.21.15.78:0/1868263429 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55db42d17d50 0x55db42d1a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.172+0000 7fafd0358b80 4 client.17610 mds_command: new command op to 24572 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-14T10:04:03.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.172+0000 7fafd0358b80 1 -- 172.21.15.78:0/1868263429 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55db42a4ad70 con 0x55db42d17d50 2024-03-14T10:04:03.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.172+0000 7fafb9e00700 1 --2- 172.21.15.78:0/1868263429 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55db42d17d50 0x55db42d1a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.173+0000 7faf97400700 10 client.17610 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:03.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.173+0000 7fafb9e00700 1 --2- 172.21.15.78:0/1868263429 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55db42d17d50 0x55db42d1a170 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=690a25d98e4ef15c server_cookie=17948dc9557f7296 in_seq=0 out_seq=0 2024-03-14T10:04:03.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.174+0000 7faf97400700 1 -- 172.21.15.78:0/1868263429 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55db42a4ad70 con 0x55db42d17d50 2024-03-14T10:04:03.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.174+0000 7faf97400700 10 client.17610 handle_command_reply: tid=0 2024-03-14T10:04:03.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.174+0000 7fafd0358b80 1 client.17610 shutdown 2024-03-14T10:04:03.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.174+0000 7fafd0358b80 1 -- 172.21.15.78:0/1868263429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42ca9200 msgr2=0x55db42d0f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.174+0000 7fafd0358b80 1 --2- 172.21.15.78:0/1868263429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42ca9200 0x55db42d0f630 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fafa409a0e0 tx=0x7fafa40852e0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.174+0000 7fafd0358b80 1 -- 172.21.15.78:0/1868263429 shutdown_connections 2024-03-14T10:04:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.174+0000 7fafd0358b80 1 --2- 172.21.15.78:0/1868263429 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x55db42d17d50 0x55db42d1a170 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.174+0000 7fafd0358b80 1 --2- 172.21.15.78:0/1868263429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55db42cb40b0 0x55db42d0a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.174+0000 7fafd0358b80 1 --2- 172.21.15.78:0/1868263429 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55db42ca88e0 0x55db42d0f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.174+0000 7faf97400700 0 client.17610 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.174+0000 7fafd0358b80 1 --2- 172.21.15.78:0/1868263429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55db42ca9200 0x55db42d0f630 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.174+0000 7fafd0358b80 1 -- 172.21.15.78:0/1868263429 >> 172.21.15.78:0/1868263429 conn(0x55db42c1f2a0 msgr2=0x55db42cad220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.174+0000 7fafd0358b80 1 -- 172.21.15.78:0/1868263429 shutdown_connections 2024-03-14T10:04:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.174+0000 7fafd0358b80 1 -- 172.21.15.78:0/1868263429 wait complete. 2024-03-14T10:04:03.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.175+0000 7fafd0358b80 20 client.17610 trim_cache size 0 max 16384 2024-03-14T10:04:03.179 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:04:03.179 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2024-03-14T10:04:03.179 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "5e88b8ae-24f4-4ced-b3b6-1a8570513173", 2024-03-14T10:04:03.179 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2024-03-14T10:04:03.180 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:04:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.175+0000 7fafbb200700 1 -- 172.21.15.78:0/783973604 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7faf9c0a0c00 msgr2=0x7faf9c0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.175+0000 7fafbb200700 1 --2- 172.21.15.78:0/783973604 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7faf9c0a0c00 0x7faf9c0a30c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009f00 tx=0x7fafb000b040 comp rx=0 tx=0).stop 2024-03-14T10:04:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.175+0000 7fafbb200700 1 -- 172.21.15.78:0/783973604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafb41014f0 msgr2=0x7fafb4199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.175+0000 7fafbb200700 1 --2- 172.21.15.78:0/783973604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafb41014f0 0x7fafb4199b50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fafa40098e0 tx=0x7fafa400f690 comp rx=0 tx=0).stop 2024-03-14T10:04:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.175+0000 7fafbb200700 1 -- 172.21.15.78:0/783973604 shutdown_connections 2024-03-14T10:04:03.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.175+0000 7fafbb200700 1 --2- 172.21.15.78:0/783973604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafb4106bf0 0x7fafb419def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.175+0000 7fafbb200700 1 --2- 172.21.15.78:0/783973604 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fafb4100b30 0x7fafb4199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.175+0000 7fafbb200700 1 --2- 172.21.15.78:0/783973604 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7faf9c0a0c00 0x7faf9c0a30c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.175+0000 7fafbb200700 1 --2- 172.21.15.78:0/783973604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafb41014f0 0x7fafb4199b50 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.175+0000 7fafbb200700 1 -- 172.21.15.78:0/783973604 >> 172.21.15.78:0/783973604 conn(0x7fafb40fc620 msgr2=0x7fafb40fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.175+0000 7fafbb200700 1 -- 172.21.15.78:0/783973604 shutdown_connections 2024-03-14T10:04:03.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.175+0000 7fafbb200700 1 -- 172.21.15.78:0/783973604 wait complete. 2024-03-14T10:04:03.195 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:04:03.252 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:03 smithi078 ceph-mon[38143]: pgmap v305: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:04:03.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.388+0000 7fb4b6400700 1 -- 172.21.15.78:0/2229970777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4b0100f20 msgr2=0x7fb4b0101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.388+0000 7fb4b6400700 1 --2- 172.21.15.78:0/2229970777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4b0100f20 0x7fb4b0101300 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009990 tx=0x7fb4a0009ca0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.388+0000 7fb4b6400700 1 -- 172.21.15.78:0/2229970777 shutdown_connections 2024-03-14T10:04:03.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.388+0000 7fb4b6400700 1 --2- 172.21.15.78:0/2229970777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4b0106040 0x7fb4b010dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.388+0000 7fb4b6400700 1 --2- 172.21.15.78:0/2229970777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4b0100f20 0x7fb4b0101300 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.388+0000 7fb4b6400700 1 --2- 172.21.15.78:0/2229970777 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4b01018d0 0x7fb4b0105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.388+0000 7fb4b6400700 1 -- 172.21.15.78:0/2229970777 >> 172.21.15.78:0/2229970777 conn(0x7fb4b00fc7a0 msgr2=0x7fb4b00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:03.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.388+0000 7fb4b6400700 1 -- 172.21.15.78:0/2229970777 shutdown_connections 2024-03-14T10:04:03.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.388+0000 7fb4b6400700 1 -- 172.21.15.78:0/2229970777 wait complete. 2024-03-14T10:04:03.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.389+0000 7fb4b6400700 1 Processor -- start 2024-03-14T10:04:03.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.390+0000 7fb4b6400700 1 -- start start 2024-03-14T10:04:03.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.390+0000 7fb4b6400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4b0100f20 0x7fb4b00730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.390+0000 7fb4b6400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4b0106040 0x7fb4b00735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.390+0000 7fb4b6400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4b0073b30 0x7fb4b01a57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.390+0000 7fb4b6400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b01a5da0 con 0x7fb4b0073b30 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.390+0000 7fb4b6400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b01a5f10 con 0x7fb4b0100f20 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.390+0000 7fb4b6400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b01a60d0 con 0x7fb4b0106040 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.390+0000 7fb4afe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4b0073b30 0x7fb4b01a57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.390+0000 7fb4afe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4b0073b30 0x7fb4b01a57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41122/0 (socket says 172.21.15.78:41122) 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.390+0000 7fb4afe00700 1 -- 172.21.15.78:0/2376220203 learned_addr learned my addr 172.21.15.78:0/2376220203 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.390+0000 7fb4aea00700 1 --2- 172.21.15.78:0/2376220203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4b0106040 0x7fb4b00735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.390+0000 7fb4af400700 1 --2- 172.21.15.78:0/2376220203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4b0100f20 0x7fb4b00730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.391+0000 7fb4afe00700 1 -- 172.21.15.78:0/2376220203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4b0106040 msgr2=0x7fb4b00735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.391+0000 7fb4afe00700 1 --2- 172.21.15.78:0/2376220203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4b0106040 0x7fb4b00735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.391+0000 7fb4afe00700 1 -- 172.21.15.78:0/2376220203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4b0100f20 msgr2=0x7fb4b00730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.391+0000 7fb4afe00700 1 --2- 172.21.15.78:0/2376220203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4b0100f20 0x7fb4b00730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.391+0000 7fb4afe00700 1 -- 172.21.15.78:0/2376220203 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a00095f0 con 0x7fb4b0073b30 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.391+0000 7fb4afe00700 1 --2- 172.21.15.78:0/2376220203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4b0073b30 0x7fb4b01a57d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fb4a800ef30 tx=0x7fb4a800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.391+0000 7fb4a7e00700 1 -- 172.21.15.78:0/2376220203 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a8018070 con 0x7fb4b0073b30 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.391+0000 7fb4a7e00700 1 -- 172.21.15.78:0/2376220203 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb4a800f040 con 0x7fb4b0073b30 2024-03-14T10:04:03.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.391+0000 7fb4b6400700 1 -- 172.21.15.78:0/2376220203 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b01a6350 con 0x7fb4b0073b30 2024-03-14T10:04:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.391+0000 7fb4a7e00700 1 -- 172.21.15.78:0/2376220203 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a800cd00 con 0x7fb4b0073b30 2024-03-14T10:04:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.391+0000 7fb4b6400700 1 -- 172.21.15.78:0/2376220203 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b01a68b0 con 0x7fb4b0073b30 2024-03-14T10:04:03.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.394+0000 7fb4a7e00700 1 -- 172.21.15.78:0/2376220203 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fb4a8003a00 con 0x7fb4b0073b30 2024-03-14T10:04:03.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.395+0000 7fb4a7e00700 1 --2- 172.21.15.78:0/2376220203 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb4940a0c80 0x7fb4940a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.395+0000 7fb4c9fc5b80 1 Processor -- start 2024-03-14T10:04:03.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.395+0000 7fb4a7e00700 1 -- 172.21.15.78:0/2376220203 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fb4a80cc2a0 con 0x7fb4b0073b30 2024-03-14T10:04:03.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.396+0000 7fb4c9fc5b80 1 -- start start 2024-03-14T10:04:03.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.396+0000 7fb4af400700 1 --2- 172.21.15.78:0/2376220203 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb4940a0c80 0x7fb4940a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.396+0000 7fb4c9fc5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21e1680 0x5605a21e1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.396+0000 7fb4c9fc5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21e1fa0 0x5605a21eb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.396+0000 7fb4c9fc5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21ebec0 0x5605a21ee2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.396+0000 7fb4c9fc5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a213a990 con 0x5605a21e1680 2024-03-14T10:04:03.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.396+0000 7fb4c9fc5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a2138350 con 0x5605a21ebec0 2024-03-14T10:04:03.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.396+0000 7fb4c9fc5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a2133c40 con 0x5605a21e1fa0 2024-03-14T10:04:03.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.396+0000 7fb4af400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21e1680 0x5605a21e1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.396+0000 7fb4af400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21e1680 0x5605a21e1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41130/0 (socket says 172.21.15.78:41130) 2024-03-14T10:04:03.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.396+0000 7fb4af400700 1 -- 172.21.15.78:0/1927024340 learned_addr learned my addr 172.21.15.78:0/1927024340 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:03.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.396+0000 7fb4aea00700 1 --2- 172.21.15.78:0/1927024340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21e1fa0 0x5605a21eb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.396+0000 7fb4afe00700 1 --2- 172.21.15.78:0/1927024340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21ebec0 0x5605a21ee2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.396+0000 7fb4af400700 1 --2- 172.21.15.78:0/2376220203 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb4940a0c80 0x7fb4940a3140 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0006010 tx=0x7fb4a0005060 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:03.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.397+0000 7fb4af400700 1 -- 172.21.15.78:0/1927024340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21e1fa0 msgr2=0x5605a21eb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:03.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.397+0000 7fb4af400700 1 --2- 172.21.15.78:0/1927024340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21e1fa0 0x5605a21eb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.397+0000 7fb4af400700 1 -- 172.21.15.78:0/1927024340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21ebec0 msgr2=0x5605a21ee2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:03.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.397+0000 7fb4af400700 1 --2- 172.21.15.78:0/1927024340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21ebec0 0x5605a21ee2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.397+0000 7fb4af400700 1 -- 172.21.15.78:0/1927024340 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605a1fb53b0 con 0x5605a21e1680 2024-03-14T10:04:03.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.397+0000 7fb4af400700 1 --2- 172.21.15.78:0/1927024340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21e1680 0x5605a21e1a60 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0015000 tx=0x7fb4a0028000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95f0a1927a3f1131 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:03.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.397+0000 7fb4a4c00700 1 -- 172.21.15.78:0/1927024340 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0005700 con 0x5605a21e1680 2024-03-14T10:04:03.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.397+0000 7fb4a4c00700 1 -- 172.21.15.78:0/1927024340 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb4a001e040 con 0x5605a21e1680 2024-03-14T10:04:03.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.397+0000 7fb4a4c00700 1 -- 172.21.15.78:0/1927024340 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a002f7b0 con 0x5605a21e1680 2024-03-14T10:04:03.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.398+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/1927024340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21e1680 msgr2=0x5605a21e1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.398+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/1927024340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21e1680 0x5605a21e1a60 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0015000 tx=0x7fb4a0028000 comp rx=0 tx=0).stop 2024-03-14T10:04:03.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.398+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/1927024340 shutdown_connections 2024-03-14T10:04:03.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.398+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/1927024340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21ebec0 0x5605a21ee2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.398+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/1927024340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21e1680 0x5605a21e1a60 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.398+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/1927024340 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21e1fa0 0x5605a21eb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.398+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/1927024340 >> 172.21.15.78:0/1927024340 conn(0x5605a2157220 msgr2=0x5605a21dcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:03.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.398+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/1927024340 shutdown_connections 2024-03-14T10:04:03.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.398+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/1927024340 wait complete. 2024-03-14T10:04:03.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.399+0000 7fb4c9fc5b80 1 -- start start 2024-03-14T10:04:03.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.399+0000 7fb4c9fc5b80 1 Processor -- start 2024-03-14T10:04:03.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.399+0000 7fb4c9fc5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21e1680 0x5605a21eb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.399+0000 7fb4c9fc5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21e1fa0 0x5605a21e8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.399+0000 7fb4c9fc5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21ebec0 0x5605a21e91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.399+0000 7fb4c9fc5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a20cfd80 con 0x5605a21e1fa0 2024-03-14T10:04:03.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.399+0000 7fb4c9fc5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a20cfa20 con 0x5605a21e1680 2024-03-14T10:04:03.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.399+0000 7fb4c9fc5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a1fb76a0 con 0x5605a21ebec0 2024-03-14T10:04:03.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.399+0000 7fb4c9fc5b80 10 client.0 _ll_register_callbacks cb 0x5605a1f3b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:03.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.400+0000 7fb4aea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21e1fa0 0x5605a21e8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.400+0000 7fb4aea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21e1fa0 0x5605a21e8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41146/0 (socket says 172.21.15.78:41146) 2024-03-14T10:04:03.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.400+0000 7fb4aea00700 1 -- 172.21.15.78:0/1176318394 learned_addr learned my addr 172.21.15.78:0/1176318394 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:03.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.400+0000 7fb4af400700 1 --2- 172.21.15.78:0/1176318394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21e1680 0x5605a21eb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.400+0000 7fb4afe00700 1 --2- 172.21.15.78:0/1176318394 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21ebec0 0x5605a21e91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.400+0000 7fb4aea00700 1 -- 172.21.15.78:0/1176318394 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21ebec0 msgr2=0x5605a21e91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:03.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.400+0000 7fb4aea00700 1 --2- 172.21.15.78:0/1176318394 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21ebec0 0x5605a21e91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.400+0000 7fb4aea00700 1 -- 172.21.15.78:0/1176318394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21e1680 msgr2=0x5605a21eb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:03.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.400+0000 7fb4aea00700 1 --2- 172.21.15.78:0/1176318394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21e1680 0x5605a21eb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.400+0000 7fb4aea00700 1 -- 172.21.15.78:0/1176318394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b010e430 con 0x5605a21e1fa0 2024-03-14T10:04:03.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.400+0000 7fb4aea00700 1 --2- 172.21.15.78:0/1176318394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21e1fa0 0x5605a21e8ba0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fb49c0126d0 tx=0x7fb49c0129e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfbb3dad7cb5e4d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:03.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.400+0000 7fb499600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:04:03.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.400+0000 7fb499600700 1 -- 172.21.15.78:0/1176318394 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c017070 con 0x5605a21e1fa0 2024-03-14T10:04:03.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.400+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/1176318394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5605a21d4720 con 0x5605a21e1fa0 2024-03-14T10:04:03.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.400+0000 7fb499600700 1 -- 172.21.15.78:0/1176318394 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fb49c010430 con 0x5605a21e1fa0 2024-03-14T10:04:03.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.401+0000 7fb499600700 1 -- 172.21.15.78:0/1176318394 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb49c0105f0 con 0x5605a21e1fa0 2024-03-14T10:04:03.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.401+0000 7fb4c9fc5b80 10 client.17622 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:03.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.401+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/1176318394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5605a1f61b60 con 0x5605a21e1fa0 2024-03-14T10:04:03.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.401+0000 7fb499600700 1 -- 172.21.15.78:0/1176318394 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5605a21d4720 con 0x5605a21e1fa0 2024-03-14T10:04:03.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.401+0000 7fb499600700 1 -- 172.21.15.78:0/1176318394 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb49c01c750 con 0x5605a21e1fa0 2024-03-14T10:04:03.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.401+0000 7fb4c9fc5b80 10 client.17622 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:03.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.401+0000 7fb4c9fc5b80 10 client.17622 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:04:03.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.401+0000 7fb4c9fc5b80 20 client.17622 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:03.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.401+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/1176318394 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5605a21fba40 0x5605a21fde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.401+0000 7fb4c9fc5b80 4 client.17622 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:04:03.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.401+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/1176318394 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5605a212c700 con 0x5605a21fba40 2024-03-14T10:04:03.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.402+0000 7fb4af400700 1 --2- 172.21.15.78:0/1176318394 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5605a21fba40 0x5605a21fde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.402+0000 7fb4af400700 1 --2- 172.21.15.78:0/1176318394 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5605a21fba40 0x5605a21fde60 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf669194702877a8 server_cookie=396728ca61b22837 in_seq=0 out_seq=0 2024-03-14T10:04:03.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.402+0000 7fb499600700 10 client.17622 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:03.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.403+0000 7fb499600700 1 -- 172.21.15.78:0/1176318394 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5605a212c700 con 0x5605a21fba40 2024-03-14T10:04:03.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.403+0000 7fb499600700 10 client.17622 handle_command_reply: tid=0 2024-03-14T10:04:03.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.403+0000 7fb4c9fc5b80 1 client.17622 shutdown 2024-03-14T10:04:03.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.403+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/1176318394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21e1fa0 msgr2=0x5605a21e8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.403+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/1176318394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21e1fa0 0x5605a21e8ba0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fb49c0126d0 tx=0x7fb49c0129e0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.403+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/1176318394 shutdown_connections 2024-03-14T10:04:03.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.403+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/1176318394 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5605a21fba40 0x5605a21fde60 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.403+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/1176318394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21e1680 0x5605a21eb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.403+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/1176318394 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21ebec0 0x5605a21e91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.403+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/1176318394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21e1fa0 0x5605a21e8ba0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.403+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/1176318394 >> 172.21.15.78:0/1176318394 conn(0x5605a2157220 msgr2=0x5605a21dc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:03.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.403+0000 7fb499600700 0 client.17622 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:03.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.404+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/1176318394 shutdown_connections 2024-03-14T10:04:03.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.404+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/1176318394 wait complete. 2024-03-14T10:04:03.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.404+0000 7fb4c9fc5b80 20 client.17622 trim_cache size 0 max 16384 2024-03-14T10:04:03.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.420+0000 7fb4c9fc5b80 1 Processor -- start 2024-03-14T10:04:03.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.421+0000 7fb4c9fc5b80 1 -- start start 2024-03-14T10:04:03.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.421+0000 7fb4c9fc5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21e1680 0x5605a21e7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.421+0000 7fb4c9fc5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21e1fa0 0x5605a21e79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.421+0000 7fb4c9fc5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21ebec0 0x5605a21dc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.421+0000 7fb4c9fc5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a21e7ef0 con 0x5605a21ebec0 2024-03-14T10:04:03.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.421+0000 7fb4c9fc5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a21d6d00 con 0x5605a21e1fa0 2024-03-14T10:04:03.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.421+0000 7fb4c9fc5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a21e2890 con 0x5605a21e1680 2024-03-14T10:04:03.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.421+0000 7fb4afe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21ebec0 0x5605a21dc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.421+0000 7fb4afe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21ebec0 0x5605a21dc520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41160/0 (socket says 172.21.15.78:41160) 2024-03-14T10:04:03.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.421+0000 7fb4afe00700 1 -- 172.21.15.78:0/2079556892 learned_addr learned my addr 172.21.15.78:0/2079556892 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:03.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.422+0000 7fb4aea00700 1 --2- 172.21.15.78:0/2079556892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21e1fa0 0x5605a21e79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.422+0000 7fb4af400700 1 --2- 172.21.15.78:0/2079556892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21e1680 0x5605a21e7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.422+0000 7fb4afe00700 1 -- 172.21.15.78:0/2079556892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21e1680 msgr2=0x5605a21e7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:03.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.422+0000 7fb4afe00700 1 --2- 172.21.15.78:0/2079556892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21e1680 0x5605a21e7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.422+0000 7fb4afe00700 1 -- 172.21.15.78:0/2079556892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21e1fa0 msgr2=0x5605a21e79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:03.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.422+0000 7fb4afe00700 1 --2- 172.21.15.78:0/2079556892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21e1fa0 0x5605a21e79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.422+0000 7fb4afe00700 1 -- 172.21.15.78:0/2079556892 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605a20f9ea0 con 0x5605a21ebec0 2024-03-14T10:04:03.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.422+0000 7fb4aea00700 1 --2- 172.21.15.78:0/2079556892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21e1fa0 0x5605a21e79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:04:03.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.422+0000 7fb4afe00700 1 --2- 172.21.15.78:0/2079556892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21ebec0 0x5605a21dc520 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fb4a800c6f0 tx=0x7fb4a8084040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f56b398ed0155b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:03.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.422+0000 7fb4a5600700 1 -- 172.21.15.78:0/2079556892 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a8004ad0 con 0x5605a21ebec0 2024-03-14T10:04:03.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.422+0000 7fb4a5600700 1 -- 172.21.15.78:0/2079556892 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb4a80b0070 con 0x5605a21ebec0 2024-03-14T10:04:03.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.422+0000 7fb4a5600700 1 -- 172.21.15.78:0/2079556892 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a809d110 con 0x5605a21ebec0 2024-03-14T10:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.423+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/2079556892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21ebec0 msgr2=0x5605a21dc520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.423+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/2079556892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21ebec0 0x5605a21dc520 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fb4a800c6f0 tx=0x7fb4a8084040 comp rx=0 tx=0).stop 2024-03-14T10:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.423+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/2079556892 shutdown_connections 2024-03-14T10:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.423+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/2079556892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21ebec0 0x5605a21dc520 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.423+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/2079556892 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21e1680 0x5605a21e7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.423+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/2079556892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21e1fa0 0x5605a21e79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.423+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/2079556892 >> 172.21.15.78:0/2079556892 conn(0x5605a2157220 msgr2=0x5605a21e3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.423+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/2079556892 shutdown_connections 2024-03-14T10:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.423+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/2079556892 wait complete. 2024-03-14T10:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.423+0000 7fb4c9fc5b80 1 -- start start 2024-03-14T10:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.423+0000 7fb4c9fc5b80 1 Processor -- start 2024-03-14T10:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.424+0000 7fb4c9fc5b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21e1680 0x5605a21f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.424+0000 7fb4c9fc5b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21e1fa0 0x5605a21f0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.424+0000 7fb4c9fc5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21ebec0 0x5605a21f10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.424+0000 7fb4c9fc5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a21f17e0 con 0x5605a21ebec0 2024-03-14T10:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.424+0000 7fb4c9fc5b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a21f1950 con 0x5605a21e1fa0 2024-03-14T10:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.424+0000 7fb4c9fc5b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605a224d570 con 0x5605a21e1680 2024-03-14T10:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.424+0000 7fb4c9fc5b80 10 client.0 _ll_register_callbacks cb 0x7fb49c009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.424+0000 7fb4afe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21ebec0 0x5605a21f10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.424+0000 7fb4af400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21e1680 0x5605a21f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.424+0000 7fb4afe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21ebec0 0x5605a21f10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41164/0 (socket says 172.21.15.78:41164) 2024-03-14T10:04:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.424+0000 7fb4afe00700 1 -- 172.21.15.78:0/3917197885 learned_addr learned my addr 172.21.15.78:0/3917197885 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.424+0000 7fb4aea00700 1 --2- 172.21.15.78:0/3917197885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21e1fa0 0x5605a21f0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb4afe00700 1 -- 172.21.15.78:0/3917197885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21e1680 msgr2=0x5605a21f27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb4afe00700 1 --2- 172.21.15.78:0/3917197885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21e1680 0x5605a21f27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb4afe00700 1 -- 172.21.15.78:0/3917197885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21e1fa0 msgr2=0x5605a21f0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb4afe00700 1 --2- 172.21.15.78:0/3917197885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21e1fa0 0x5605a21f0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb4afe00700 1 -- 172.21.15.78:0/3917197885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b01a68b0 con 0x5605a21ebec0 2024-03-14T10:04:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb4af400700 1 --2- 172.21.15.78:0/3917197885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21e1680 0x5605a21f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:04:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb4afe00700 1 --2- 172.21.15.78:0/3917197885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21ebec0 0x5605a21f10d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8095750 tx=0x7fb4a80092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96ba4e86680bec88 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb499600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:04:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb499600700 1 -- 172.21.15.78:0/3917197885 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a8099250 con 0x5605a21ebec0 2024-03-14T10:04:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb499600700 1 -- 172.21.15.78:0/3917197885 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fb4a801b320 con 0x5605a21ebec0 2024-03-14T10:04:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/3917197885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb4a001e040 con 0x5605a21ebec0 2024-03-14T10:04:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb499600700 1 -- 172.21.15.78:0/3917197885 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4a8099950 con 0x5605a21ebec0 2024-03-14T10:04:03.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb4c9fc5b80 10 client.17628 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/3917197885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5605a1fb9cd0 con 0x5605a21ebec0 2024-03-14T10:04:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb499600700 1 -- 172.21.15.78:0/3917197885 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb4a8082020 con 0x5605a21ebec0 2024-03-14T10:04:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb499600700 1 -- 172.21.15.78:0/3917197885 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb4a80994b0 con 0x5605a21ebec0 2024-03-14T10:04:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.425+0000 7fb4c9fc5b80 10 client.17628 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.426+0000 7fb4c9fc5b80 10 client.17628 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:04:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.426+0000 7fb4c9fc5b80 20 client.17628 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.426+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/3917197885 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5605a224e800 0x5605a2250bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.426+0000 7fb4c9fc5b80 4 client.17628 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:04:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.426+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/3917197885 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5605a1f82cf0 con 0x5605a224e800 2024-03-14T10:04:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.426+0000 7fb4af400700 1 --2- 172.21.15.78:0/3917197885 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5605a224e800 0x5605a2250bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.426+0000 7fb499600700 10 client.17628 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.426+0000 7fb4af400700 1 --2- 172.21.15.78:0/3917197885 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5605a224e800 0x5605a2250bf0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf9477bcda332556 server_cookie=1cb28c17a94d2607 in_seq=0 out_seq=0 2024-03-14T10:04:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.427+0000 7fb499600700 1 -- 172.21.15.78:0/3917197885 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5605a1f82cf0 con 0x5605a224e800 2024-03-14T10:04:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.427+0000 7fb499600700 10 client.17628 handle_command_reply: tid=0 2024-03-14T10:04:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.427+0000 7fb4c9fc5b80 1 client.17628 shutdown 2024-03-14T10:04:03.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.427+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/3917197885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21ebec0 msgr2=0x5605a21f10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.427+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/3917197885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21ebec0 0x5605a21f10d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8095750 tx=0x7fb4a80092e0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.427+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/3917197885 shutdown_connections 2024-03-14T10:04:03.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.427+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/3917197885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5605a21ebec0 0x5605a21f10d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.427+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/3917197885 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5605a21e1680 0x5605a21f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.427+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/3917197885 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5605a224e800 0x5605a2250bf0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.427+0000 7fb4c9fc5b80 1 --2- 172.21.15.78:0/3917197885 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5605a21e1fa0 0x5605a21f0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.427+0000 7fb499600700 0 client.17628 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:03.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.427+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/3917197885 >> 172.21.15.78:0/3917197885 conn(0x5605a2157220 msgr2=0x5605a21dc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:03.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.427+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/3917197885 shutdown_connections 2024-03-14T10:04:03.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.427+0000 7fb4c9fc5b80 1 -- 172.21.15.78:0/3917197885 wait complete. 2024-03-14T10:04:03.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.427+0000 7fb4c9fc5b80 20 client.17628 trim_cache size 0 max 16384 2024-03-14T10:04:03.432 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:04:03.433 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-14T10:04:03.433 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2024-03-14T10:04:03.433 INFO:teuthology.orchestra.run.smithi078.stdout: "5e88b8ae-24f4-4ced-b3b6-1a8570513173": { 2024-03-14T10:04:03.433 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2024-03-14T10:04:03.433 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "5e88b8ae-24f4-4ced-b3b6-1a8570513173", 2024-03-14T10:04:03.433 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2024-03-14T10:04:03.433 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:03.433 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:03.433 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:04:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.428+0000 7fb4b6400700 1 -- 172.21.15.78:0/2376220203 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb4940a0c80 msgr2=0x7fb4940a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.428+0000 7fb4b6400700 1 --2- 172.21.15.78:0/2376220203 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb4940a0c80 0x7fb4940a3140 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0006010 tx=0x7fb4a0005060 comp rx=0 tx=0).stop 2024-03-14T10:04:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.428+0000 7fb4b6400700 1 -- 172.21.15.78:0/2376220203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4b0073b30 msgr2=0x7fb4b01a57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.428+0000 7fb4b6400700 1 --2- 172.21.15.78:0/2376220203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4b0073b30 0x7fb4b01a57d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fb4a800ef30 tx=0x7fb4a800c3a0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.428+0000 7fb4b6400700 1 -- 172.21.15.78:0/2376220203 shutdown_connections 2024-03-14T10:04:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.428+0000 7fb4b6400700 1 --2- 172.21.15.78:0/2376220203 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb4940a0c80 0x7fb4940a3140 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.428+0000 7fb4b6400700 1 --2- 172.21.15.78:0/2376220203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb4b0100f20 0x7fb4b00730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.428+0000 7fb4b6400700 1 --2- 172.21.15.78:0/2376220203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4b0073b30 0x7fb4b01a57d0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.428+0000 7fb4b6400700 1 --2- 172.21.15.78:0/2376220203 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb4b0106040 0x7fb4b00735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:03.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.428+0000 7fb4b6400700 1 -- 172.21.15.78:0/2376220203 >> 172.21.15.78:0/2376220203 conn(0x7fb4b00fc7a0 msgr2=0x7fb4b00fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:03.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.428+0000 7fb4b6400700 1 -- 172.21.15.78:0/2376220203 shutdown_connections 2024-03-14T10:04:03.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:03.428+0000 7fb4b6400700 1 -- 172.21.15.78:0/2376220203 wait complete. 2024-03-14T10:04:03.446 INFO:tasks.cephfs.filesystem:scrub status for tag:5e88b8ae-24f4-4ced-b3b6-1a8570513173 - {'path': '/', 'tag': '5e88b8ae-24f4-4ced-b3b6-1a8570513173', 'options': 'recursive,force'} 2024-03-14T10:04:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:03 smithi152 ceph-mon[48279]: pgmap v305: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:04:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:03 smithi132 ceph-mon[47959]: pgmap v305: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:04:04.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:04 smithi152 ceph-mon[48279]: scrub queued for path: / 2024-03-14T10:04:04.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:04 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T10:04:04.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:04 smithi152 ceph-mon[48279]: scrub summary: active paths [/] 2024-03-14T10:04:04.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:04 smithi152 ceph-mon[48279]: scrub summary: idle+waiting paths [/] 2024-03-14T10:04:04.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:04 smithi152 ceph-mon[48279]: scrub summary: idle 2024-03-14T10:04:04.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:04 smithi132 ceph-mon[47959]: scrub queued for path: / 2024-03-14T10:04:04.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:04 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T10:04:04.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:04 smithi132 ceph-mon[47959]: scrub summary: active paths [/] 2024-03-14T10:04:04.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:04 smithi132 ceph-mon[47959]: scrub summary: idle+waiting paths [/] 2024-03-14T10:04:04.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:04 smithi132 ceph-mon[47959]: scrub summary: idle 2024-03-14T10:04:04.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:04 smithi078 ceph-mon[38143]: scrub queued for path: / 2024-03-14T10:04:04.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:04 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T10:04:04.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:04 smithi078 ceph-mon[38143]: scrub summary: active paths [/] 2024-03-14T10:04:04.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:04 smithi078 ceph-mon[38143]: scrub summary: idle+waiting paths [/] 2024-03-14T10:04:04.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:04 smithi078 ceph-mon[38143]: scrub summary: idle 2024-03-14T10:04:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:05 smithi152 ceph-mon[48279]: pgmap v306: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:04:05.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:05 smithi132 ceph-mon[47959]: pgmap v306: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:04:05.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:05 smithi078 ceph-mon[38143]: pgmap v306: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:04:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:07 smithi152 ceph-mon[48279]: pgmap v307: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:04:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:07 smithi132 ceph-mon[47959]: pgmap v307: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:04:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:07 smithi078 ceph-mon[38143]: pgmap v307: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-14T10:04:09.252 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:08 smithi152 ceph-mon[48279]: pgmap v308: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:04:09.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:08 smithi132 ceph-mon[47959]: pgmap v308: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:04:09.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:08 smithi078 ceph-mon[38143]: pgmap v308: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-14T10:04:10.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:09 smithi152 ceph-mon[48279]: osdmap e95: 12 total, 12 up, 12 in 2024-03-14T10:04:10.272 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:09 smithi132 ceph-mon[47959]: osdmap e95: 12 total, 12 up, 12 in 2024-03-14T10:04:10.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:09 smithi078 ceph-mon[38143]: osdmap e95: 12 total, 12 up, 12 in 2024-03-14T10:04:11.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:10 smithi152 ceph-mon[48279]: pgmap v310: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-14T10:04:11.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:10 smithi132 ceph-mon[47959]: pgmap v310: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-14T10:04:11.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:10 smithi078 ceph-mon[38143]: pgmap v310: 129 pgs: 129 active+clean; 261 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-14T10:04:13.019 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-14T10:04:13.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.224+0000 7f157ea00700 1 -- 172.21.15.78:0/500358730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1578105fa0 msgr2=0x7f157810db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:13.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.224+0000 7f157ea00700 1 --2- 172.21.15.78:0/500358730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1578105fa0 0x7f157810db90 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1570009f90 tx=0x7f157000b330 comp rx=0 tx=0).stop 2024-03-14T10:04:13.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.224+0000 7f157ea00700 1 -- 172.21.15.78:0/500358730 shutdown_connections 2024-03-14T10:04:13.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.224+0000 7f157ea00700 1 --2- 172.21.15.78:0/500358730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15780682c0 0x7f15780686a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.224+0000 7f157ea00700 1 --2- 172.21.15.78:0/500358730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1578105fa0 0x7f157810db90 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.224+0000 7f157ea00700 1 --2- 172.21.15.78:0/500358730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1578068c70 0x7f15781058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.224+0000 7f157ea00700 1 -- 172.21.15.78:0/500358730 >> 172.21.15.78:0/500358730 conn(0x7f15780769c0 msgr2=0x7f1578076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:13.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.224+0000 7f157ea00700 1 -- 172.21.15.78:0/500358730 shutdown_connections 2024-03-14T10:04:13.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.224+0000 7f157ea00700 1 -- 172.21.15.78:0/500358730 wait complete. 2024-03-14T10:04:13.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.225+0000 7f157ea00700 1 Processor -- start 2024-03-14T10:04:13.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.226+0000 7f157ea00700 1 -- start start 2024-03-14T10:04:13.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.226+0000 7f157ea00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15780682c0 0x7f1578197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:13.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.226+0000 7f157ea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1578105fa0 0x7f1578197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:13.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.226+0000 7f157ea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f157819bf70 0x7f157819c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:13.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.226+0000 7f157ea00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157819c9f0 con 0x7f1578105fa0 2024-03-14T10:04:13.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.226+0000 7f157ea00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157819cb60 con 0x7f15780682c0 2024-03-14T10:04:13.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.226+0000 7f157ea00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157819cd20 con 0x7f157819bf70 2024-03-14T10:04:13.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.226+0000 7f156fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1578105fa0 0x7f1578197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:13.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.226+0000 7f156fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1578105fa0 0x7f1578197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39224/0 (socket says 172.21.15.78:39224) 2024-03-14T10:04:13.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.226+0000 7f156fe00700 1 -- 172.21.15.78:0/1849231897 learned_addr learned my addr 172.21.15.78:0/1849231897 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:13.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.226+0000 7f1577e00700 1 --2- 172.21.15.78:0/1849231897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15780682c0 0x7f1578197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:13.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.226+0000 7f157ca00700 1 --2- 172.21.15.78:0/1849231897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f157819bf70 0x7f157819c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:13.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.226+0000 7f156fe00700 1 -- 172.21.15.78:0/1849231897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f157819bf70 msgr2=0x7f157819c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:13.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.226+0000 7f156fe00700 1 --2- 172.21.15.78:0/1849231897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f157819bf70 0x7f157819c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.227+0000 7f156fe00700 1 -- 172.21.15.78:0/1849231897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15780682c0 msgr2=0x7f1578197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:13.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.227+0000 7f156fe00700 1 --2- 172.21.15.78:0/1849231897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15780682c0 0x7f1578197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.227+0000 7f156fe00700 1 -- 172.21.15.78:0/1849231897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1570009c40 con 0x7f1578105fa0 2024-03-14T10:04:13.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.227+0000 7f1577e00700 1 --2- 172.21.15.78:0/1849231897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15780682c0 0x7f1578197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:04:13.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.227+0000 7f156fe00700 1 --2- 172.21.15.78:0/1849231897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1578105fa0 0x7f1578197bd0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f156000e940 tx=0x7f156000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:13.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.227+0000 7f1575600700 1 -- 172.21.15.78:0/1849231897 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1560009730 con 0x7f1578105fa0 2024-03-14T10:04:13.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.227+0000 7f1575600700 1 -- 172.21.15.78:0/1849231897 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f156000cbd0 con 0x7f1578105fa0 2024-03-14T10:04:13.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.227+0000 7f157ea00700 1 -- 172.21.15.78:0/1849231897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15781a4280 con 0x7f1578105fa0 2024-03-14T10:04:13.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.227+0000 7f1575600700 1 -- 172.21.15.78:0/1849231897 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1560018680 con 0x7f1578105fa0 2024-03-14T10:04:13.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.227+0000 7f157ea00700 1 -- 172.21.15.78:0/1849231897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15781a47b0 con 0x7f1578105fa0 2024-03-14T10:04:13.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.228+0000 7f157ea00700 1 -- 172.21.15.78:0/1849231897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1578109270 con 0x7f1578105fa0 2024-03-14T10:04:13.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.230+0000 7f1575600700 1 -- 172.21.15.78:0/1849231897 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f156000cd40 con 0x7f1578105fa0 2024-03-14T10:04:13.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.231+0000 7f1575600700 1 --2- 172.21.15.78:0/1849231897 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f155c0a0c80 0x7f155c0a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:13.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.231+0000 7f1575600700 1 -- 172.21.15.78:0/1849231897 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9584+0+0 (secure 0 0 0) 0x7f1560014070 con 0x7f1578105fa0 2024-03-14T10:04:13.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.231+0000 7f1577e00700 1 --2- 172.21.15.78:0/1849231897 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f155c0a0c80 0x7f155c0a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:13.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.232+0000 7f1577e00700 1 --2- 172.21.15.78:0/1849231897 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f155c0a0c80 0x7f155c0a3140 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1568005fd0 tx=0x7f1568009310 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:13.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.236+0000 7f1575600700 1 -- 172.21.15.78:0/1849231897 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1560052960 con 0x7f1578105fa0 2024-03-14T10:04:13.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:13 smithi078 ceph-mon[38143]: pgmap v311: 129 pgs: 10 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 100 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:04:13.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.473+0000 7f157ea00700 1 -- 172.21.15.78:0/1849231897 --> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f1578068c70 con 0x7f155c0a0c80 2024-03-14T10:04:13.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.481+0000 7f1575600700 1 -- 172.21.15.78:0/1849231897 <== mgr.15546 v2:172.21.15.78:6800/2192201293 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+350 (secure 0 0 0) 0x7f1578068c70 con 0x7f155c0a0c80 2024-03-14T10:04:13.482 INFO:teuthology.orchestra.run.smithi078.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-14T00:00:00", "created": "2024-03-14T09:54:10", "first": "2024-03-14T09:55:00", "last": "2024-03-14T10:04:00", "last_pruned": "2024-03-14T10:04:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-14T10:04:13.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.483+0000 7f157ea00700 1 -- 172.21.15.78:0/1849231897 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f155c0a0c80 msgr2=0x7f155c0a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:13.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.483+0000 7f157ea00700 1 --2- 172.21.15.78:0/1849231897 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f155c0a0c80 0x7f155c0a3140 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1568005fd0 tx=0x7f1568009310 comp rx=0 tx=0).stop 2024-03-14T10:04:13.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.483+0000 7f157ea00700 1 -- 172.21.15.78:0/1849231897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1578105fa0 msgr2=0x7f1578197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:13.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.483+0000 7f157ea00700 1 --2- 172.21.15.78:0/1849231897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1578105fa0 0x7f1578197bd0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f156000e940 tx=0x7f156000ed00 comp rx=0 tx=0).stop 2024-03-14T10:04:13.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.484+0000 7f157ea00700 1 -- 172.21.15.78:0/1849231897 shutdown_connections 2024-03-14T10:04:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.484+0000 7f157ea00700 1 --2- 172.21.15.78:0/1849231897 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f155c0a0c80 0x7f155c0a3140 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.484+0000 7f157ea00700 1 --2- 172.21.15.78:0/1849231897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15780682c0 0x7f1578197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.484+0000 7f157ea00700 1 --2- 172.21.15.78:0/1849231897 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f157819bf70 0x7f157819c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.484+0000 7f157ea00700 1 --2- 172.21.15.78:0/1849231897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1578105fa0 0x7f1578197bd0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.484+0000 7f157ea00700 1 -- 172.21.15.78:0/1849231897 >> 172.21.15.78:0/1849231897 conn(0x7f15780769c0 msgr2=0x7f15780fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.484+0000 7f157ea00700 1 -- 172.21.15.78:0/1849231897 shutdown_connections 2024-03-14T10:04:13.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.484+0000 7f157ea00700 1 -- 172.21.15.78:0/1849231897 wait complete. 2024-03-14T10:04:13.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:13 smithi152 ceph-mon[48279]: pgmap v311: 129 pgs: 10 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 100 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:04:13.502 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-14T10:04:13.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:13 smithi132 ceph-mon[47959]: pgmap v311: 129 pgs: 10 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 100 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:04:13.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.693+0000 7f9628200700 1 -- 172.21.15.78:0/847584336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9620105ff0 msgr2=0x7f962010db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:13.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.693+0000 7f9628200700 1 --2- 172.21.15.78:0/847584336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9620105ff0 0x7f962010db70 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f961800b0a0 tx=0x7f961800b3b0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.693+0000 7f9628200700 1 -- 172.21.15.78:0/847584336 shutdown_connections 2024-03-14T10:04:13.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.693+0000 7f9628200700 1 --2- 172.21.15.78:0/847584336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9620105ff0 0x7f962010db70 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.693+0000 7f9628200700 1 --2- 172.21.15.78:0/847584336 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9620100ed0 0x7f96201012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.693+0000 7f9628200700 1 --2- 172.21.15.78:0/847584336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9620101880 0x7f96201058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.693+0000 7f9628200700 1 -- 172.21.15.78:0/847584336 >> 172.21.15.78:0/847584336 conn(0x7f96200fc790 msgr2=0x7f96200febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:13.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.693+0000 7f9628200700 1 -- 172.21.15.78:0/847584336 shutdown_connections 2024-03-14T10:04:13.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.693+0000 7f9628200700 1 -- 172.21.15.78:0/847584336 wait complete. 2024-03-14T10:04:13.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.694+0000 7f9628200700 1 Processor -- start 2024-03-14T10:04:13.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.694+0000 7f9628200700 1 -- start start 2024-03-14T10:04:13.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.694+0000 7f9628200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9620100ed0 0x7f9620197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:13.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.694+0000 7f9628200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9620105ff0 0x7f9620197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:13.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.694+0000 7f9628200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f962019af80 0x7f962019b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:13.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.694+0000 7f9628200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962019b970 con 0x7f9620105ff0 2024-03-14T10:04:13.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.694+0000 7f9628200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962019bae0 con 0x7f962019af80 2024-03-14T10:04:13.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.694+0000 7f9628200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962019bca0 con 0x7f9620100ed0 2024-03-14T10:04:13.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.695+0000 7f9624e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9620105ff0 0x7f9620197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:13.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.695+0000 7f9625800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9620100ed0 0x7f9620197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:13.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.695+0000 7f9625800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9620100ed0 0x7f9620197750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:47186/0 (socket says 172.21.15.78:47186) 2024-03-14T10:04:13.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.695+0000 7f9624e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9620105ff0 0x7f9620197c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:39246/0 (socket says 172.21.15.78:39246) 2024-03-14T10:04:13.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.695+0000 7f9625800700 1 -- 172.21.15.78:0/1529585592 learned_addr learned my addr 172.21.15.78:0/1529585592 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:13.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.695+0000 7f9626200700 1 --2- 172.21.15.78:0/1529585592 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f962019af80 0x7f962019b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:13.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.695+0000 7f9624e00700 1 -- 172.21.15.78:0/1529585592 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9620100ed0 msgr2=0x7f9620197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:13.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.695+0000 7f9624e00700 1 --2- 172.21.15.78:0/1529585592 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9620100ed0 0x7f9620197750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.695+0000 7f9624e00700 1 -- 172.21.15.78:0/1529585592 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f962019af80 msgr2=0x7f962019b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:13.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.695+0000 7f9624e00700 1 --2- 172.21.15.78:0/1529585592 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f962019af80 0x7f962019b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.695+0000 7f9624e00700 1 -- 172.21.15.78:0/1529585592 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9618009d00 con 0x7f9620105ff0 2024-03-14T10:04:13.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.695+0000 7f9625800700 1 --2- 172.21.15.78:0/1529585592 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9620100ed0 0x7f9620197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:04:13.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.695+0000 7f9624e00700 1 --2- 172.21.15.78:0/1529585592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9620105ff0 0x7f9620197c90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f960c00e9c0 tx=0x7f960c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:13.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.696+0000 7f961e000700 1 -- 172.21.15.78:0/1529585592 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c00c9c0 con 0x7f9620105ff0 2024-03-14T10:04:13.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.696+0000 7f961e000700 1 -- 172.21.15.78:0/1529585592 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f960c010430 con 0x7f9620105ff0 2024-03-14T10:04:13.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.696+0000 7f961e000700 1 -- 172.21.15.78:0/1529585592 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c018680 con 0x7f9620105ff0 2024-03-14T10:04:13.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.696+0000 7f9628200700 1 -- 172.21.15.78:0/1529585592 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96201a4310 con 0x7f9620105ff0 2024-03-14T10:04:13.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.696+0000 7f9628200700 1 -- 172.21.15.78:0/1529585592 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96201a47d0 con 0x7f9620105ff0 2024-03-14T10:04:13.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.697+0000 7f9628200700 1 -- 172.21.15.78:0/1529585592 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96201092c0 con 0x7f9620105ff0 2024-03-14T10:04:13.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.699+0000 7f961e000700 1 -- 172.21.15.78:0/1529585592 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f960c00cb20 con 0x7f9620105ff0 2024-03-14T10:04:13.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.700+0000 7f961e000700 1 --2- 172.21.15.78:0/1529585592 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f96080a0c30 0x7f96080a30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:13.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.700+0000 7f961e000700 1 -- 172.21.15.78:0/1529585592 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9584+0+0 (secure 0 0 0) 0x7f960c014070 con 0x7f9620105ff0 2024-03-14T10:04:13.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.700+0000 7f9625800700 1 --2- 172.21.15.78:0/1529585592 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f96080a0c30 0x7f96080a30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:13.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.701+0000 7f9625800700 1 --2- 172.21.15.78:0/1529585592 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f96080a0c30 0x7f96080a30f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9614011550 tx=0x7f9614009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:13.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.704+0000 7f961e000700 1 -- 172.21.15.78:0/1529585592 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f960c052960 con 0x7f9620105ff0 2024-03-14T10:04:13.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.943+0000 7f9628200700 1 -- 172.21.15.78:0/1529585592 --> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9620198a50 con 0x7f96080a0c30 2024-03-14T10:04:13.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.950+0000 7f961e000700 1 -- 172.21.15.78:0/1529585592 <== mgr.15546 v2:172.21.15.78:6800/2192201293 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f9620198a50 con 0x7f96080a0c30 2024-03-14T10:04:13.951 INFO:teuthology.orchestra.run.smithi078.stdout:/ 1m 6m3h 2024-03-14T10:04:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.953+0000 7f9628200700 1 -- 172.21.15.78:0/1529585592 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f96080a0c30 msgr2=0x7f96080a30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.953+0000 7f9628200700 1 --2- 172.21.15.78:0/1529585592 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f96080a0c30 0x7f96080a30f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9614011550 tx=0x7f9614009260 comp rx=0 tx=0).stop 2024-03-14T10:04:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.953+0000 7f9628200700 1 -- 172.21.15.78:0/1529585592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9620105ff0 msgr2=0x7f9620197c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.953+0000 7f9628200700 1 --2- 172.21.15.78:0/1529585592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9620105ff0 0x7f9620197c90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f960c00e9c0 tx=0x7f960c00ecd0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.953+0000 7f9628200700 1 -- 172.21.15.78:0/1529585592 shutdown_connections 2024-03-14T10:04:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.953+0000 7f9628200700 1 --2- 172.21.15.78:0/1529585592 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f96080a0c30 0x7f96080a30f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.953+0000 7f9628200700 1 --2- 172.21.15.78:0/1529585592 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f962019af80 0x7f962019b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.953+0000 7f9628200700 1 --2- 172.21.15.78:0/1529585592 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f9620100ed0 0x7f9620197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.953+0000 7f9628200700 1 --2- 172.21.15.78:0/1529585592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9620105ff0 0x7f9620197c90 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.953+0000 7f9628200700 1 -- 172.21.15.78:0/1529585592 >> 172.21.15.78:0/1529585592 conn(0x7f96200fc790 msgr2=0x7f96200feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:13.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.953+0000 7f9628200700 1 -- 172.21.15.78:0/1529585592 shutdown_connections 2024-03-14T10:04:13.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:13.953+0000 7f9628200700 1 -- 172.21.15.78:0/1529585592 wait complete. 2024-03-14T10:04:13.972 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-14T10:04:13.983 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-14T10:04:14.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:14 smithi078 ceph-mon[38143]: from='client.17634 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T10:04:14.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:14 smithi152 ceph-mon[48279]: from='client.17634 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T10:04:14.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:14 smithi132 ceph-mon[47959]: from='client.17634 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T10:04:15.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:15 smithi152 ceph-mon[48279]: from='client.17640 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T10:04:15.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:15 smithi152 ceph-mon[48279]: pgmap v312: 129 pgs: 10 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 100 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:04:15.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:15 smithi132 ceph-mon[47959]: from='client.17640 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T10:04:15.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:15 smithi132 ceph-mon[47959]: pgmap v312: 129 pgs: 10 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 100 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:04:15.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:15 smithi078 ceph-mon[38143]: from='client.17640 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T10:04:15.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:15 smithi078 ceph-mon[38143]: pgmap v312: 129 pgs: 10 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 100 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-14T10:04:17.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:17 smithi152 ceph-mon[48279]: pgmap v313: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 9 op/s 2024-03-14T10:04:17.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:17 smithi132 ceph-mon[47959]: pgmap v313: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 9 op/s 2024-03-14T10:04:17.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:17 smithi078 ceph-mon[38143]: pgmap v313: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 9 op/s 2024-03-14T10:04:19.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:18 smithi152 ceph-mon[48279]: osdmap e96: 12 total, 12 up, 12 in 2024-03-14T10:04:19.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:18 smithi152 ceph-mon[48279]: pgmap v315: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 222 B/s wr, 10 op/s 2024-03-14T10:04:19.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:18 smithi132 ceph-mon[47959]: osdmap e96: 12 total, 12 up, 12 in 2024-03-14T10:04:19.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:18 smithi132 ceph-mon[47959]: pgmap v315: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 222 B/s wr, 10 op/s 2024-03-14T10:04:19.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:18 smithi078 ceph-mon[38143]: osdmap e96: 12 total, 12 up, 12 in 2024-03-14T10:04:19.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:18 smithi078 ceph-mon[38143]: pgmap v315: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 222 B/s wr, 10 op/s 2024-03-14T10:04:21.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:21 smithi152 ceph-mon[48279]: pgmap v316: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 9 op/s 2024-03-14T10:04:21.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:21 smithi132 ceph-mon[47959]: pgmap v316: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 9 op/s 2024-03-14T10:04:21.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:21 smithi078 ceph-mon[38143]: pgmap v316: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 9 op/s 2024-03-14T10:04:23.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:23 smithi152 ceph-mon[48279]: pgmap v317: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-14T10:04:23.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:23 smithi132 ceph-mon[47959]: pgmap v317: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-14T10:04:23.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:23 smithi078 ceph-mon[38143]: pgmap v317: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-14T10:04:25.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:25 smithi152 ceph-mon[48279]: pgmap v318: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-14T10:04:25.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:25 smithi132 ceph-mon[47959]: pgmap v318: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-14T10:04:25.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:25 smithi078 ceph-mon[38143]: pgmap v318: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 9 op/s 2024-03-14T10:04:27.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:27 smithi152 ceph-mon[48279]: pgmap v319: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-14T10:04:27.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:27 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T10:04:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:27 smithi132 ceph-mon[47959]: pgmap v319: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-14T10:04:27.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:27 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T10:04:27.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:27 smithi078 ceph-mon[38143]: pgmap v319: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-14T10:04:27.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:27 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T10:04:28.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:28.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:28.999 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:29.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:29.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi132 to 182.5M 2024-03-14T10:04:29.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: Unable to set osd_memory_target on smithi132 to 191379148: error parsing value: Value '191379148' is below minimum 939524096 2024-03-14T10:04:29.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi152 to 182.5M 2024-03-14T10:04:29.000 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: Unable to set osd_memory_target on smithi152 to 191374848: error parsing value: Value '191374848' is below minimum 939524096 2024-03-14T10:04:29.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: pgmap v320: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T10:04:29.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:29.001 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:28 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:29.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:29.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:29.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:29.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:29.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi132 to 182.5M 2024-03-14T10:04:29.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: Unable to set osd_memory_target on smithi132 to 191379148: error parsing value: Value '191379148' is below minimum 939524096 2024-03-14T10:04:29.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi152 to 182.5M 2024-03-14T10:04:29.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: Unable to set osd_memory_target on smithi152 to 191374848: error parsing value: Value '191374848' is below minimum 939524096 2024-03-14T10:04:29.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: pgmap v320: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T10:04:29.021 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:29.022 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:28 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:29.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:29.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:29.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:29.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:29.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi132 to 182.5M 2024-03-14T10:04:29.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: Unable to set osd_memory_target on smithi132 to 191379148: error parsing value: Value '191379148' is below minimum 939524096 2024-03-14T10:04:29.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:29.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi152 to 182.5M 2024-03-14T10:04:29.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: Unable to set osd_memory_target on smithi152 to 191374848: error parsing value: Value '191374848' is below minimum 939524096 2024-03-14T10:04:29.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: pgmap v320: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-14T10:04:29.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:29.055 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:28 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:30.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:29 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:30.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:29 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:30.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:29 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:30.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:29 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:30.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:29 smithi152 ceph-mon[48279]: Adjusting osd_memory_target on smithi078 to 221.5M 2024-03-14T10:04:30.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:29 smithi152 ceph-mon[48279]: Unable to set osd_memory_target on smithi078 to 232331366: error parsing value: Value '232331366' is below minimum 939524096 2024-03-14T10:04:30.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:29 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T10:04:30.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:29 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T10:04:30.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:29 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:30.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:29 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:30.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:29 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:30.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:29 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:30.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:29 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:30.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:29 smithi132 ceph-mon[47959]: Adjusting osd_memory_target on smithi078 to 221.5M 2024-03-14T10:04:30.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:29 smithi132 ceph-mon[47959]: Unable to set osd_memory_target on smithi078 to 232331366: error parsing value: Value '232331366' is below minimum 939524096 2024-03-14T10:04:30.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:29 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T10:04:30.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:29 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T10:04:30.271 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:29 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:30.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:29 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:30.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:29 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:30.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:29 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:30.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:29 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-14T10:04:30.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:29 smithi078 ceph-mon[38143]: Adjusting osd_memory_target on smithi078 to 221.5M 2024-03-14T10:04:30.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:29 smithi078 ceph-mon[38143]: Unable to set osd_memory_target on smithi078 to 232331366: error parsing value: Value '232331366' is below minimum 939524096 2024-03-14T10:04:30.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:29 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T10:04:30.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:29 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T10:04:30.304 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:29 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' 2024-03-14T10:04:31.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:30 smithi152 ceph-mon[48279]: pgmap v321: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:31.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:30 smithi132 ceph-mon[47959]: pgmap v321: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:31.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:30 smithi078 ceph-mon[38143]: pgmap v321: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:33.448 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-14T10:04:33.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:33 smithi152 ceph-mon[48279]: pgmap v322: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:33.514 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:33 smithi078 ceph-mon[38143]: pgmap v322: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:33.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:33 smithi132 ceph-mon[47959]: pgmap v322: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:33.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.655+0000 7f7dae000700 1 -- 172.21.15.78:0/2625122968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8106040 msgr2=0x7f7da810dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.655+0000 7f7dae000700 1 --2- 172.21.15.78:0/2625122968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8106040 0x7f7da810dbc0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f7da000b0a0 tx=0x7f7da000b3b0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.655+0000 7f7dae000700 1 -- 172.21.15.78:0/2625122968 shutdown_connections 2024-03-14T10:04:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.655+0000 7f7dae000700 1 --2- 172.21.15.78:0/2625122968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8106040 0x7f7da810dbc0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.655+0000 7f7dae000700 1 --2- 172.21.15.78:0/2625122968 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7da8100f20 0x7f7da8101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.655+0000 7f7dae000700 1 --2- 172.21.15.78:0/2625122968 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7da81018d0 0x7f7da8105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.655+0000 7f7dae000700 1 -- 172.21.15.78:0/2625122968 >> 172.21.15.78:0/2625122968 conn(0x7f7da80fc7a0 msgr2=0x7f7da80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.655+0000 7f7dae000700 1 -- 172.21.15.78:0/2625122968 shutdown_connections 2024-03-14T10:04:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.655+0000 7f7dae000700 1 -- 172.21.15.78:0/2625122968 wait complete. 2024-03-14T10:04:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.657+0000 7f7dae000700 1 Processor -- start 2024-03-14T10:04:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.657+0000 7f7dae000700 1 -- start start 2024-03-14T10:04:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.657+0000 7f7dae000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7da8100f20 0x7f7da80730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.657+0000 7f7dae000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7da8106040 0x7f7da80735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.657+0000 7f7dae000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8073b30 0x7f7da81a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.657+0000 7f7dae000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da81a5ee0 con 0x7f7da8073b30 2024-03-14T10:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.657+0000 7f7dae000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da81a6020 con 0x7f7da8100f20 2024-03-14T10:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.657+0000 7f7dae000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da81a61e0 con 0x7f7da8106040 2024-03-14T10:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.658+0000 7f7da7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8073b30 0x7f7da81a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.658+0000 7f7da7e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8073b30 0x7f7da81a5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36552/0 (socket says 172.21.15.78:36552) 2024-03-14T10:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.658+0000 7f7da7400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7da8100f20 0x7f7da80730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.658+0000 7f7da7e00700 1 -- 172.21.15.78:0/1997788129 learned_addr learned my addr 172.21.15.78:0/1997788129 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.658+0000 7f7d9fe00700 1 --2- 172.21.15.78:0/1997788129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7da8106040 0x7f7da80735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.658+0000 7f7da7e00700 1 -- 172.21.15.78:0/1997788129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7da8106040 msgr2=0x7f7da80735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.658+0000 7f7da7e00700 1 --2- 172.21.15.78:0/1997788129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7da8106040 0x7f7da80735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.658+0000 7f7da7e00700 1 -- 172.21.15.78:0/1997788129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7da8100f20 msgr2=0x7f7da80730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.658+0000 7f7da7e00700 1 --2- 172.21.15.78:0/1997788129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7da8100f20 0x7f7da80730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.658+0000 7f7da7e00700 1 -- 172.21.15.78:0/1997788129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da0009d00 con 0x7f7da8073b30 2024-03-14T10:04:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.658+0000 7f7da7e00700 1 --2- 172.21.15.78:0/1997788129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8073b30 0x7f7da81a5910 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f7da000b070 tx=0x7f7da0007cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.658+0000 7f7da4c00700 1 -- 172.21.15.78:0/1997788129 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da000e040 con 0x7f7da8073b30 2024-03-14T10:04:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.658+0000 7f7da4c00700 1 -- 172.21.15.78:0/1997788129 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7da000bdc0 con 0x7f7da8073b30 2024-03-14T10:04:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.658+0000 7f7da4c00700 1 -- 172.21.15.78:0/1997788129 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da00052a0 con 0x7f7da8073b30 2024-03-14T10:04:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.658+0000 7f7dae000700 1 -- 172.21.15.78:0/1997788129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da81a6460 con 0x7f7da8073b30 2024-03-14T10:04:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.659+0000 7f7dae000700 1 -- 172.21.15.78:0/1997788129 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da81a6830 con 0x7f7da8073b30 2024-03-14T10:04:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.662+0000 7f7da4c00700 1 -- 172.21.15.78:0/1997788129 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f7da001d480 con 0x7f7da8073b30 2024-03-14T10:04:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.663+0000 7f7dc1ae7b80 1 Processor -- start 2024-03-14T10:04:33.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.663+0000 7f7dc1ae7b80 1 -- start start 2024-03-14T10:04:33.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.664+0000 7f7da4c00700 1 --2- 172.21.15.78:0/1997788129 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7d8c0a0c00 0x7f7d8c0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.664+0000 7f7dc1ae7b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90a680 0x5620ca90aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.664+0000 7f7dc1ae7b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca90afa0 0x5620ca914980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.664+0000 7f7dc1ae7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 0x5620ca9172b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.664+0000 7f7dc1ae7b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620ca863990 con 0x5620ca90a680 2024-03-14T10:04:33.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.664+0000 7f7dc1ae7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620ca861350 con 0x5620ca914ec0 2024-03-14T10:04:33.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.664+0000 7f7dc1ae7b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620ca85cc40 con 0x5620ca90afa0 2024-03-14T10:04:33.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.664+0000 7f7da4c00700 1 -- 172.21.15.78:0/1997788129 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f7da00cdac0 con 0x7f7da8073b30 2024-03-14T10:04:33.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.664+0000 7f7d9fe00700 1 --2- 172.21.15.78:0/1997788129 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7d8c0a0c00 0x7f7d8c0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.664+0000 7f7da7e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 0x5620ca9172b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.664+0000 7f7da7400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90a680 0x5620ca90aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.664+0000 7f7da7e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 0x5620ca9172b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:56002/0 (socket says 172.21.15.78:56002) 2024-03-14T10:04:33.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.664+0000 7f7da7e00700 1 -- 172.21.15.78:0/3599566757 learned_addr learned my addr 172.21.15.78:0/3599566757 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:33.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.664+0000 7f7da7400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90a680 0x5620ca90aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36566/0 (socket says 172.21.15.78:36566) 2024-03-14T10:04:33.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.664+0000 7f7d9fe00700 1 --2- 172.21.15.78:0/3599566757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca90afa0 0x5620ca914980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.665+0000 7f7da7e00700 1 -- 172.21.15.78:0/3599566757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca90afa0 msgr2=0x5620ca914980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:33.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.665+0000 7f7da7e00700 1 --2- 172.21.15.78:0/3599566757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca90afa0 0x5620ca914980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.665+0000 7f7da7e00700 1 -- 172.21.15.78:0/3599566757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90a680 msgr2=0x5620ca90aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:33.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.665+0000 7f7da7e00700 1 --2- 172.21.15.78:0/3599566757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90a680 0x5620ca90aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.665+0000 7f7da7e00700 1 -- 172.21.15.78:0/3599566757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5620ca6de3b0 con 0x5620ca914ec0 2024-03-14T10:04:33.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.665+0000 7f7da7400700 1 --2- 172.21.15.78:0/3599566757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90a680 0x5620ca90aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:04:33.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.665+0000 7f7da7e00700 1 --2- 172.21.15.78:0/3599566757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 0x5620ca9172b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7da00a2930 tx=0x7f7da00967b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f5b115f2a3745ec server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:33.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.665+0000 7f7d9fe00700 1 --2- 172.21.15.78:0/1997788129 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7d8c0a0c00 0x7f7d8c0a30c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7d90005e90 tx=0x7f7d9000ba90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:33.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.665+0000 7f7d97e00700 1 -- 172.21.15.78:0/3599566757 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da009a3d0 con 0x5620ca914ec0 2024-03-14T10:04:33.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.665+0000 7f7d97e00700 1 -- 172.21.15.78:0/3599566757 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7da007b070 con 0x5620ca914ec0 2024-03-14T10:04:33.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.665+0000 7f7d97e00700 1 -- 172.21.15.78:0/3599566757 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da00a2e60 con 0x5620ca914ec0 2024-03-14T10:04:33.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.666+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/3599566757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 msgr2=0x5620ca9172b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:33.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.666+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/3599566757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 0x5620ca9172b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7da00a2930 tx=0x7f7da00967b0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.666+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/3599566757 shutdown_connections 2024-03-14T10:04:33.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.666+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/3599566757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 0x5620ca9172b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.666+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/3599566757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90a680 0x5620ca90aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.666+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/3599566757 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca90afa0 0x5620ca914980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.666+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/3599566757 >> 172.21.15.78:0/3599566757 conn(0x5620ca880220 msgr2=0x5620ca905aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:33.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.666+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/3599566757 shutdown_connections 2024-03-14T10:04:33.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.666+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/3599566757 wait complete. 2024-03-14T10:04:33.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.667+0000 7f7dc1ae7b80 1 -- start start 2024-03-14T10:04:33.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.667+0000 7f7dc1ae7b80 1 Processor -- start 2024-03-14T10:04:33.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.667+0000 7f7dc1ae7b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca90a680 0x5620ca914900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.667+0000 7f7dc1ae7b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90afa0 0x5620ca911d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.667+0000 7f7dc1ae7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 0x5620ca912470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.667+0000 7f7dc1ae7b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620ca7f8d80 con 0x5620ca90afa0 2024-03-14T10:04:33.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.667+0000 7f7dc1ae7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620ca7f8a20 con 0x5620ca914ec0 2024-03-14T10:04:33.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.667+0000 7f7dc1ae7b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620ca6e06a0 con 0x5620ca90a680 2024-03-14T10:04:33.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.667+0000 7f7dc1ae7b80 10 client.0 _ll_register_callbacks cb 0x5620ca6643a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:33.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.668+0000 7f7d9fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90afa0 0x5620ca911d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.668+0000 7f7d9fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90afa0 0x5620ca911d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36572/0 (socket says 172.21.15.78:36572) 2024-03-14T10:04:33.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.668+0000 7f7d9fe00700 1 -- 172.21.15.78:0/1828444980 learned_addr learned my addr 172.21.15.78:0/1828444980 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:33.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.668+0000 7f7da7e00700 1 --2- 172.21.15.78:0/1828444980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 0x5620ca912470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.668+0000 7f7da7400700 1 --2- 172.21.15.78:0/1828444980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca90a680 0x5620ca914900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.668+0000 7f7d9fe00700 1 -- 172.21.15.78:0/1828444980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca90a680 msgr2=0x5620ca914900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:33.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.668+0000 7f7d9fe00700 1 --2- 172.21.15.78:0/1828444980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca90a680 0x5620ca914900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.668+0000 7f7d9fe00700 1 -- 172.21.15.78:0/1828444980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 msgr2=0x5620ca912470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:33.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.668+0000 7f7d9fe00700 1 --2- 172.21.15.78:0/1828444980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 0x5620ca912470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.668+0000 7f7d9fe00700 1 -- 172.21.15.78:0/1828444980 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d980098e0 con 0x5620ca90afa0 2024-03-14T10:04:33.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.668+0000 7f7d9fe00700 1 --2- 172.21.15.78:0/1828444980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90afa0 0x5620ca911d30 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000bfc0 tx=0x7f7d900047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eccaf9d850aba0c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:33.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.668+0000 7f7d94c00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:04:33.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.669+0000 7f7d94c00700 1 -- 172.21.15.78:0/1828444980 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d90029600 con 0x5620ca90afa0 2024-03-14T10:04:33.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.669+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/1828444980 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5620ca8fd720 con 0x5620ca90afa0 2024-03-14T10:04:33.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.669+0000 7f7d94c00700 1 -- 172.21.15.78:0/1828444980 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f7d900213f0 con 0x5620ca90afa0 2024-03-14T10:04:33.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.669+0000 7f7d94c00700 1 -- 172.21.15.78:0/1828444980 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d90029760 con 0x5620ca90afa0 2024-03-14T10:04:33.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.669+0000 7f7dc1ae7b80 10 client.17652 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:33.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.669+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/1828444980 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f7da81a6830 con 0x5620ca90afa0 2024-03-14T10:04:33.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.669+0000 7f7d94c00700 1 -- 172.21.15.78:0/1828444980 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5620ca8fd720 con 0x5620ca90afa0 2024-03-14T10:04:33.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.669+0000 7f7d94c00700 1 -- 172.21.15.78:0/1828444980 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7d90027020 con 0x5620ca90afa0 2024-03-14T10:04:33.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.669+0000 7f7dc1ae7b80 10 client.17652 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:33.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.669+0000 7f7dc1ae7b80 10 client.17652 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:04:33.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.669+0000 7f7dc1ae7b80 20 client.17652 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:33.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.669+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/1828444980 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5620ca924980 0x5620ca926da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.669+0000 7f7dc1ae7b80 4 client.17652 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:04:33.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.669+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/1828444980 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5620ca855700 con 0x5620ca924980 2024-03-14T10:04:33.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.670+0000 7f7da7400700 1 --2- 172.21.15.78:0/1828444980 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5620ca924980 0x5620ca926da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.670+0000 7f7da7400700 1 --2- 172.21.15.78:0/1828444980 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5620ca924980 0x5620ca926da0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd44e210959b2d50 server_cookie=b4abcdd6a4010a04 in_seq=0 out_seq=0 2024-03-14T10:04:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.670+0000 7f7d94c00700 10 client.17652 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.671+0000 7f7d94c00700 1 -- 172.21.15.78:0/1828444980 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5620ca855700 con 0x5620ca924980 2024-03-14T10:04:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.671+0000 7f7d94c00700 10 client.17652 handle_command_reply: tid=0 2024-03-14T10:04:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.671+0000 7f7dc1ae7b80 1 client.17652 shutdown 2024-03-14T10:04:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.671+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/1828444980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90afa0 msgr2=0x5620ca911d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.671+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/1828444980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90afa0 0x5620ca911d30 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000bfc0 tx=0x7f7d900047c0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.671+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/1828444980 shutdown_connections 2024-03-14T10:04:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.671+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/1828444980 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5620ca924980 0x5620ca926da0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.671+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/1828444980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 0x5620ca912470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.671+0000 7f7d94c00700 0 client.17652 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.671+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/1828444980 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca90a680 0x5620ca914900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.671+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/1828444980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90afa0 0x5620ca911d30 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.671+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/1828444980 >> 172.21.15.78:0/1828444980 conn(0x5620ca880220 msgr2=0x5620ca905690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:33.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.672+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/1828444980 shutdown_connections 2024-03-14T10:04:33.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.672+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/1828444980 wait complete. 2024-03-14T10:04:33.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.672+0000 7f7dc1ae7b80 20 client.17652 trim_cache size 0 max 16384 2024-03-14T10:04:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.688+0000 7f7dc1ae7b80 1 Processor -- start 2024-03-14T10:04:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.688+0000 7f7dc1ae7b80 1 -- start start 2024-03-14T10:04:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.688+0000 7f7dc1ae7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca90a680 0x5620ca91b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.688+0000 7f7dc1ae7b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90afa0 0x5620ca905590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.688+0000 7f7dc1ae7b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca914ec0 0x5620ca905ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.688+0000 7f7dc1ae7b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620ca8ffd00 con 0x5620ca90afa0 2024-03-14T10:04:33.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.688+0000 7f7dc1ae7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620ca906770 con 0x5620ca90a680 2024-03-14T10:04:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.688+0000 7f7dc1ae7b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620ca90b890 con 0x5620ca914ec0 2024-03-14T10:04:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.689+0000 7f7d9fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90afa0 0x5620ca905590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.689+0000 7f7d9fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90afa0 0x5620ca905590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36584/0 (socket says 172.21.15.78:36584) 2024-03-14T10:04:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.689+0000 7f7d9fe00700 1 -- 172.21.15.78:0/2065825843 learned_addr learned my addr 172.21.15.78:0/2065825843 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.689+0000 7f7da7e00700 1 --2- 172.21.15.78:0/2065825843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca914ec0 0x5620ca905ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.689+0000 7f7da7400700 1 --2- 172.21.15.78:0/2065825843 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca90a680 0x5620ca91b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.689+0000 7f7d9fe00700 1 -- 172.21.15.78:0/2065825843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca914ec0 msgr2=0x5620ca905ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.689+0000 7f7d9fe00700 1 --2- 172.21.15.78:0/2065825843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca914ec0 0x5620ca905ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.689+0000 7f7d9fe00700 1 -- 172.21.15.78:0/2065825843 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca90a680 msgr2=0x5620ca91b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.689+0000 7f7d9fe00700 1 --2- 172.21.15.78:0/2065825843 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca90a680 0x5620ca91b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.689+0000 7f7d9fe00700 1 -- 172.21.15.78:0/2065825843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5620ca68ab60 con 0x5620ca90afa0 2024-03-14T10:04:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.689+0000 7f7da7e00700 1 --2- 172.21.15.78:0/2065825843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca914ec0 0x5620ca905ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:04:33.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.689+0000 7f7da7400700 1 --2- 172.21.15.78:0/2065825843 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca90a680 0x5620ca91b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:04:33.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.689+0000 7f7d9fe00700 1 --2- 172.21.15.78:0/2065825843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90afa0 0x5620ca905590 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f7d90021b60 tx=0x7f7d90011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3682bbdda7eda363 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:33.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.690+0000 7f7d9d200700 1 -- 172.21.15.78:0/2065825843 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d90011a80 con 0x5620ca90afa0 2024-03-14T10:04:33.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.690+0000 7f7d9d200700 1 -- 172.21.15.78:0/2065825843 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d9001a040 con 0x5620ca90afa0 2024-03-14T10:04:33.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.690+0000 7f7d9d200700 1 -- 172.21.15.78:0/2065825843 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d90036990 con 0x5620ca90afa0 2024-03-14T10:04:33.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.690+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/2065825843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90afa0 msgr2=0x5620ca905590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:33.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.690+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/2065825843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90afa0 0x5620ca905590 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f7d90021b60 tx=0x7f7d90011f00 comp rx=0 tx=0).stop 2024-03-14T10:04:33.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.690+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/2065825843 shutdown_connections 2024-03-14T10:04:33.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.690+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/2065825843 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca90a680 0x5620ca91b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.690+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/2065825843 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca914ec0 0x5620ca905ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.690+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/2065825843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90afa0 0x5620ca905590 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.690+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/2065825843 >> 172.21.15.78:0/2065825843 conn(0x5620ca880220 msgr2=0x5620ca90d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:33.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.690+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/2065825843 shutdown_connections 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.690+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/2065825843 wait complete. 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.691+0000 7f7dc1ae7b80 1 -- start start 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.691+0000 7f7dc1ae7b80 1 Processor -- start 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.691+0000 7f7dc1ae7b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90a680 0x5620ca919ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.691+0000 7f7dc1ae7b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca90afa0 0x5620ca91a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.691+0000 7f7dc1ae7b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 0x5620ca91a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.691+0000 7f7dc1ae7b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620ca9764b0 con 0x5620ca90a680 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.691+0000 7f7dc1ae7b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620ca9765f0 con 0x5620ca914ec0 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.692+0000 7f7dc1ae7b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620ca976730 con 0x5620ca90afa0 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.692+0000 7f7dc1ae7b80 10 client.0 _ll_register_callbacks cb 0x7f7d9000b820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.692+0000 7f7da7400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90a680 0x5620ca919ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.692+0000 7f7da7400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90a680 0x5620ca919ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36590/0 (socket says 172.21.15.78:36590) 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.692+0000 7f7da7400700 1 -- 172.21.15.78:0/879617617 learned_addr learned my addr 172.21.15.78:0/879617617 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.692+0000 7f7d9fe00700 1 --2- 172.21.15.78:0/879617617 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca90afa0 0x5620ca91a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.692+0000 7f7da7e00700 1 --2- 172.21.15.78:0/879617617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 0x5620ca91a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.692+0000 7f7da7400700 1 -- 172.21.15.78:0/879617617 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca90afa0 msgr2=0x5620ca91a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:33.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.692+0000 7f7da7400700 1 --2- 172.21.15.78:0/879617617 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca90afa0 0x5620ca91a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.692+0000 7f7da7400700 1 -- 172.21.15.78:0/879617617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 msgr2=0x5620ca91a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:33.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.692+0000 7f7da7400700 1 --2- 172.21.15.78:0/879617617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 0x5620ca91a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.692+0000 7f7da7400700 1 -- 172.21.15.78:0/879617617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da81a6830 con 0x5620ca90a680 2024-03-14T10:04:33.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.692+0000 7f7d9fe00700 1 --2- 172.21.15.78:0/879617617 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca90afa0 0x5620ca91a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:04:33.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.692+0000 7f7da7400700 1 --2- 172.21.15.78:0/879617617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90a680 0x5620ca919ef0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x5620ca927680 tx=0x7f7d980045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99c9ff50d20ca9b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:33.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.692+0000 7f7d94c00700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:04:33.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.692+0000 7f7d94c00700 1 -- 172.21.15.78:0/879617617 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98017040 con 0x5620ca90a680 2024-03-14T10:04:33.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.693+0000 7f7d94c00700 1 -- 172.21.15.78:0/879617617 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f7d98021300 con 0x5620ca90a680 2024-03-14T10:04:33.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.693+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/879617617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7da007b070 con 0x5620ca90a680 2024-03-14T10:04:33.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.693+0000 7f7d94c00700 1 -- 172.21.15.78:0/879617617 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d98003a40 con 0x5620ca90a680 2024-03-14T10:04:33.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.693+0000 7f7dc1ae7b80 10 client.17658 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:33.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.693+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/879617617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5620ca822ea0 con 0x5620ca90a680 2024-03-14T10:04:33.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.693+0000 7f7d94c00700 1 -- 172.21.15.78:0/879617617 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7da007b070 con 0x5620ca90a680 2024-03-14T10:04:33.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.693+0000 7f7d94c00700 1 -- 172.21.15.78:0/879617617 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7d98012620 con 0x5620ca90a680 2024-03-14T10:04:33.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.693+0000 7f7dc1ae7b80 10 client.17658 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:33.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.693+0000 7f7dc1ae7b80 10 client.17658 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:04:33.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.693+0000 7f7dc1ae7b80 20 client.17658 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:33.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.693+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/879617617 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5620ca977f60 0x5620ca97a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.693+0000 7f7dc1ae7b80 4 client.17658 mds_command: new command op to 24572 tid=0[{"prefix": "scrub status"}] 2024-03-14T10:04:33.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.693+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/879617617 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5620ca6abcf0 con 0x5620ca977f60 2024-03-14T10:04:33.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.694+0000 7f7d9fe00700 1 --2- 172.21.15.78:0/879617617 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5620ca977f60 0x5620ca97a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.694+0000 7f7d9fe00700 1 --2- 172.21.15.78:0/879617617 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5620ca977f60 0x5620ca97a380 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3169703d3ceaa043 server_cookie=73a5e83153c831ae in_seq=0 out_seq=0 2024-03-14T10:04:33.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.694+0000 7f7d94c00700 10 client.17658 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:33.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.694+0000 7f7d94c00700 1 -- 172.21.15.78:0/879617617 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5620ca6abcf0 con 0x5620ca977f60 2024-03-14T10:04:33.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.694+0000 7f7d94c00700 10 client.17658 handle_command_reply: tid=0 2024-03-14T10:04:33.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.694+0000 7f7dc1ae7b80 1 client.17658 shutdown 2024-03-14T10:04:33.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.694+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/879617617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90a680 msgr2=0x5620ca919ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:33.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.694+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/879617617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90a680 0x5620ca919ef0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x5620ca927680 tx=0x7f7d980045e0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.694+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/879617617 shutdown_connections 2024-03-14T10:04:33.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.694+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/879617617 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5620ca977f60 0x5620ca97a380 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.694+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/879617617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5620ca914ec0 0x5620ca91a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.694+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/879617617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5620ca90a680 0x5620ca919ef0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.694+0000 7f7dc1ae7b80 1 --2- 172.21.15.78:0/879617617 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5620ca90afa0 0x5620ca91a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.694+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/879617617 >> 172.21.15.78:0/879617617 conn(0x5620ca880220 msgr2=0x5620ca905590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:33.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.695+0000 7f7d94c00700 0 client.17658 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:33.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.695+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/879617617 shutdown_connections 2024-03-14T10:04:33.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.695+0000 7f7dc1ae7b80 1 -- 172.21.15.78:0/879617617 wait complete. 2024-03-14T10:04:33.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.695+0000 7f7dc1ae7b80 20 client.17658 trim_cache size 0 max 16384 2024-03-14T10:04:33.700 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:04:33.700 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2024-03-14T10:04:33.700 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2024-03-14T10:04:33.701 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:04:33.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.695+0000 7f7dae000700 1 -- 172.21.15.78:0/1997788129 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7d8c0a0c00 msgr2=0x7f7d8c0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:33.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.695+0000 7f7dae000700 1 --2- 172.21.15.78:0/1997788129 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7d8c0a0c00 0x7f7d8c0a30c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7d90005e90 tx=0x7f7d9000ba90 comp rx=0 tx=0).stop 2024-03-14T10:04:33.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.695+0000 7f7dae000700 1 -- 172.21.15.78:0/1997788129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8073b30 msgr2=0x7f7da81a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:33.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.695+0000 7f7dae000700 1 --2- 172.21.15.78:0/1997788129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8073b30 0x7f7da81a5910 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f7da000b070 tx=0x7f7da0007cf0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.695+0000 7f7dae000700 1 -- 172.21.15.78:0/1997788129 shutdown_connections 2024-03-14T10:04:33.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.695+0000 7f7dae000700 1 --2- 172.21.15.78:0/1997788129 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f7d8c0a0c00 0x7f7d8c0a30c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.695+0000 7f7dae000700 1 --2- 172.21.15.78:0/1997788129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7da8100f20 0x7f7da80730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.695+0000 7f7dae000700 1 --2- 172.21.15.78:0/1997788129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7da8073b30 0x7f7da81a5910 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.695+0000 7f7dae000700 1 --2- 172.21.15.78:0/1997788129 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f7da8106040 0x7f7da80735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.695+0000 7f7dae000700 1 -- 172.21.15.78:0/1997788129 >> 172.21.15.78:0/1997788129 conn(0x7f7da80fc7a0 msgr2=0x7f7da80fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:33.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.696+0000 7f7dae000700 1 -- 172.21.15.78:0/1997788129 shutdown_connections 2024-03-14T10:04:33.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.696+0000 7f7dae000700 1 -- 172.21.15.78:0/1997788129 wait complete. 2024-03-14T10:04:33.714 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-14T10:04:33.715 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T10:04:33.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.909+0000 7f5aeb800700 1 -- 172.21.15.78:0/2230745018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae41017d0 msgr2=0x7f5ae4105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:33.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.909+0000 7f5aeb800700 1 --2- 172.21.15.78:0/2230745018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae41017d0 0x7f5ae4105810 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0009990 tx=0x7f5ad0009ca0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.910+0000 7f5aeb800700 1 -- 172.21.15.78:0/2230745018 shutdown_connections 2024-03-14T10:04:33.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.910+0000 7f5aeb800700 1 --2- 172.21.15.78:0/2230745018 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ae4100e20 0x7f5ae4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.910+0000 7f5aeb800700 1 --2- 172.21.15.78:0/2230745018 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ae4105f40 0x7f5ae410dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.910+0000 7f5aeb800700 1 --2- 172.21.15.78:0/2230745018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae41017d0 0x7f5ae4105810 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.910+0000 7f5aeb800700 1 -- 172.21.15.78:0/2230745018 >> 172.21.15.78:0/2230745018 conn(0x7f5ae40fc700 msgr2=0x7f5ae40feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:33.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.910+0000 7f5aeb800700 1 -- 172.21.15.78:0/2230745018 shutdown_connections 2024-03-14T10:04:33.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.910+0000 7f5aeb800700 1 -- 172.21.15.78:0/2230745018 wait complete. 2024-03-14T10:04:33.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.911+0000 7f5aeb800700 1 Processor -- start 2024-03-14T10:04:33.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.911+0000 7f5aeb800700 1 -- start start 2024-03-14T10:04:33.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.911+0000 7f5aeb800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae4100e20 0x7f5ae4199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.911+0000 7f5aeb800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ae4105f40 0x7f5ae4199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.911+0000 7f5aeb800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ae419e020 0x7f5ae419e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.911+0000 7f5aeb800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae419eaa0 con 0x7f5ae4100e20 2024-03-14T10:04:33.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.911+0000 7f5aeb800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae419ec10 con 0x7f5ae4105f40 2024-03-14T10:04:33.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.911+0000 7f5aeb800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae419edd0 con 0x7f5ae419e020 2024-03-14T10:04:33.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.911+0000 7f5ae8e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae4100e20 0x7f5ae4199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.911+0000 7f5ae8e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae4100e20 0x7f5ae4199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45698/0 (socket says 172.21.15.78:45698) 2024-03-14T10:04:33.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.912+0000 7f5ae8e00700 1 -- 172.21.15.78:0/1132203349 learned_addr learned my addr 172.21.15.78:0/1132203349 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:33.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.912+0000 7f5ae9800700 1 --2- 172.21.15.78:0/1132203349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ae419e020 0x7f5ae419e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.912+0000 7f5ae3e00700 1 --2- 172.21.15.78:0/1132203349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ae4105f40 0x7f5ae4199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.912+0000 7f5ae8e00700 1 -- 172.21.15.78:0/1132203349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ae419e020 msgr2=0x7f5ae419e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:33.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.912+0000 7f5ae8e00700 1 --2- 172.21.15.78:0/1132203349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ae419e020 0x7f5ae419e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.912+0000 7f5ae8e00700 1 -- 172.21.15.78:0/1132203349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ae4105f40 msgr2=0x7f5ae4199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:33.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.912+0000 7f5ae8e00700 1 --2- 172.21.15.78:0/1132203349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ae4105f40 0x7f5ae4199cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:33.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.912+0000 7f5ae8e00700 1 -- 172.21.15.78:0/1132203349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc009c40 con 0x7f5ae4100e20 2024-03-14T10:04:33.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.912+0000 7f5ae3e00700 1 --2- 172.21.15.78:0/1132203349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ae4105f40 0x7f5ae4199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:04:33.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.912+0000 7f5ae8e00700 1 --2- 172.21.15.78:0/1132203349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae4100e20 0x7f5ae4199790 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5ad800e9a0 tx=0x7f5ad800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:33.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.912+0000 7f5ae1600700 1 -- 172.21.15.78:0/1132203349 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad8009790 con 0x7f5ae4100e20 2024-03-14T10:04:33.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.912+0000 7f5ae1600700 1 -- 172.21.15.78:0/1132203349 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ad800cbd0 con 0x7f5ae4100e20 2024-03-14T10:04:33.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.912+0000 7f5ae1600700 1 -- 172.21.15.78:0/1132203349 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad800eee0 con 0x7f5ae4100e20 2024-03-14T10:04:33.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.912+0000 7f5aeb800700 1 -- 172.21.15.78:0/1132203349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad00095f0 con 0x7f5ae4100e20 2024-03-14T10:04:33.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.912+0000 7f5aeb800700 1 -- 172.21.15.78:0/1132203349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae41a6630 con 0x7f5ae4100e20 2024-03-14T10:04:33.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.913+0000 7f5aeb800700 1 -- 172.21.15.78:0/1132203349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae41022f0 con 0x7f5ae4100e20 2024-03-14T10:04:33.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.915+0000 7f5ae1600700 1 -- 172.21.15.78:0/1132203349 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f5ad800cd40 con 0x7f5ae4100e20 2024-03-14T10:04:33.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.917+0000 7f5ae1600700 1 --2- 172.21.15.78:0/1132203349 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5acc0a0c00 0x7f5acc0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:33.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.917+0000 7f5ae1600700 1 -- 172.21.15.78:0/1132203349 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f5ad8014070 con 0x7f5ae4100e20 2024-03-14T10:04:33.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.917+0000 7f5ae3e00700 1 --2- 172.21.15.78:0/1132203349 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5acc0a0c00 0x7f5acc0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:33.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.917+0000 7f5ae3e00700 1 --2- 172.21.15.78:0/1132203349 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5acc0a0c00 0x7f5acc0a30c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0006010 tx=0x7f5ad000b260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:33.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:33.919+0000 7f5ae1600700 1 -- 172.21.15.78:0/1132203349 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5ad80961a0 con 0x7f5ae4100e20 2024-03-14T10:04:34.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.171+0000 7f5aeb800700 1 -- 172.21.15.78:0/1132203349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ae41017d0 con 0x7f5ae4100e20 2024-03-14T10:04:34.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.172+0000 7f5ae1600700 1 -- 172.21.15.78:0/1132203349 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f5ad80997c0 con 0x7f5ae4100e20 2024-03-14T10:04:34.173 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T10:04:34.173 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T10:04:34.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.175+0000 7f5aeb800700 1 -- 172.21.15.78:0/1132203349 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5acc0a0c00 msgr2=0x7f5acc0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.175+0000 7f5aeb800700 1 --2- 172.21.15.78:0/1132203349 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5acc0a0c00 0x7f5acc0a30c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0006010 tx=0x7f5ad000b260 comp rx=0 tx=0).stop 2024-03-14T10:04:34.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.175+0000 7f5aeb800700 1 -- 172.21.15.78:0/1132203349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae4100e20 msgr2=0x7f5ae4199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.175+0000 7f5aeb800700 1 --2- 172.21.15.78:0/1132203349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae4100e20 0x7f5ae4199790 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5ad800e9a0 tx=0x7f5ad800ed60 comp rx=0 tx=0).stop 2024-03-14T10:04:34.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.175+0000 7f5aeb800700 1 -- 172.21.15.78:0/1132203349 shutdown_connections 2024-03-14T10:04:34.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.175+0000 7f5aeb800700 1 --2- 172.21.15.78:0/1132203349 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5ae419e020 0x7f5ae419e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.175+0000 7f5aeb800700 1 --2- 172.21.15.78:0/1132203349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ae4100e20 0x7f5ae4199790 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.175+0000 7f5aeb800700 1 --2- 172.21.15.78:0/1132203349 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5acc0a0c00 0x7f5acc0a30c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.175+0000 7f5aeb800700 1 --2- 172.21.15.78:0/1132203349 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ae4105f40 0x7f5ae4199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.175+0000 7f5aeb800700 1 -- 172.21.15.78:0/1132203349 >> 172.21.15.78:0/1132203349 conn(0x7f5ae40fc700 msgr2=0x7f5ae40fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.175+0000 7f5aeb800700 1 -- 172.21.15.78:0/1132203349 shutdown_connections 2024-03-14T10:04:34.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.175+0000 7f5aeb800700 1 -- 172.21.15.78:0/1132203349 wait complete. 2024-03-14T10:04:34.177 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T10:04:34.195 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-14T10:04:34.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.389+0000 7f25d6400700 1 -- 172.21.15.78:0/1604722606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25d01018e0 msgr2=0x7f25d0105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.389+0000 7f25d6400700 1 --2- 172.21.15.78:0/1604722606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25d01018e0 0x7f25d0105920 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f25bc009990 tx=0x7f25bc009ca0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.389+0000 7f25d6400700 1 -- 172.21.15.78:0/1604722606 shutdown_connections 2024-03-14T10:04:34.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.389+0000 7f25d6400700 1 --2- 172.21.15.78:0/1604722606 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25d0106050 0x7f25d010dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.389+0000 7f25d6400700 1 --2- 172.21.15.78:0/1604722606 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25d0100f30 0x7f25d0101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.389+0000 7f25d6400700 1 --2- 172.21.15.78:0/1604722606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25d01018e0 0x7f25d0105920 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.389+0000 7f25d6400700 1 -- 172.21.15.78:0/1604722606 >> 172.21.15.78:0/1604722606 conn(0x7f25d00fc7b0 msgr2=0x7f25d00febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.389+0000 7f25d6400700 1 -- 172.21.15.78:0/1604722606 shutdown_connections 2024-03-14T10:04:34.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.389+0000 7f25d6400700 1 -- 172.21.15.78:0/1604722606 wait complete. 2024-03-14T10:04:34.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.390+0000 7f25d6400700 1 Processor -- start 2024-03-14T10:04:34.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.390+0000 7f25d6400700 1 -- start start 2024-03-14T10:04:34.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.390+0000 7f25d6400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25d0100f30 0x7f25d00730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.390+0000 7f25d6400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25d0106050 0x7f25d00735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.390+0000 7f25d6400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25d0073b30 0x7f25d01a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.390+0000 7f25d6400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d01a5db0 con 0x7f25d0100f30 2024-03-14T10:04:34.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.390+0000 7f25d6400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d01a5f20 con 0x7f25d0073b30 2024-03-14T10:04:34.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.390+0000 7f25d6400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d01a60e0 con 0x7f25d0106050 2024-03-14T10:04:34.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.390+0000 7f25cf400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25d0100f30 0x7f25d00730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.391+0000 7f25cf400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25d0100f30 0x7f25d00730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45706/0 (socket says 172.21.15.78:45706) 2024-03-14T10:04:34.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.391+0000 7f25cf400700 1 -- 172.21.15.78:0/3019799083 learned_addr learned my addr 172.21.15.78:0/3019799083 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:34.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.391+0000 7f25cfe00700 1 --2- 172.21.15.78:0/3019799083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25d0073b30 0x7f25d01a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.391+0000 7f25cea00700 1 --2- 172.21.15.78:0/3019799083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25d0106050 0x7f25d00735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.391+0000 7f25cf400700 1 -- 172.21.15.78:0/3019799083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25d0106050 msgr2=0x7f25d00735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.391+0000 7f25cf400700 1 --2- 172.21.15.78:0/3019799083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25d0106050 0x7f25d00735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.391+0000 7f25cf400700 1 -- 172.21.15.78:0/3019799083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25d0073b30 msgr2=0x7f25d01a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.391+0000 7f25cf400700 1 --2- 172.21.15.78:0/3019799083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25d0073b30 0x7f25d01a57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.391+0000 7f25cf400700 1 -- 172.21.15.78:0/3019799083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc0095f0 con 0x7f25d0100f30 2024-03-14T10:04:34.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.391+0000 7f25cfe00700 1 --2- 172.21.15.78:0/3019799083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25d0073b30 0x7f25d01a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:04:34.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.391+0000 7f25cf400700 1 --2- 172.21.15.78:0/3019799083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25d0100f30 0x7f25d00730b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f25c000e9a0 tx=0x7f25c000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.391+0000 7f25c7e00700 1 -- 172.21.15.78:0/3019799083 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c0009790 con 0x7f25d0100f30 2024-03-14T10:04:34.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.391+0000 7f25c7e00700 1 -- 172.21.15.78:0/3019799083 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f25c0010430 con 0x7f25d0100f30 2024-03-14T10:04:34.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.391+0000 7f25d6400700 1 -- 172.21.15.78:0/3019799083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d01a63c0 con 0x7f25d0100f30 2024-03-14T10:04:34.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.391+0000 7f25c7e00700 1 -- 172.21.15.78:0/3019799083 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c0018680 con 0x7f25d0100f30 2024-03-14T10:04:34.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.392+0000 7f25d6400700 1 -- 172.21.15.78:0/3019799083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d01a68d0 con 0x7f25d0100f30 2024-03-14T10:04:34.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.395+0000 7f25c7e00700 1 -- 172.21.15.78:0/3019799083 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f25c00105a0 con 0x7f25d0100f30 2024-03-14T10:04:34.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.396+0000 7f25c7e00700 1 --2- 172.21.15.78:0/3019799083 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f25b40a0c00 0x7f25b40a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.396+0000 7f25e9e44b80 1 Processor -- start 2024-03-14T10:04:34.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.396+0000 7f25c7e00700 1 -- 172.21.15.78:0/3019799083 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f25c0014070 con 0x7f25d0100f30 2024-03-14T10:04:34.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.396+0000 7f25cea00700 1 --2- 172.21.15.78:0/3019799083 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f25b40a0c00 0x7f25b40a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.396+0000 7f25e9e44b80 1 -- start start 2024-03-14T10:04:34.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.396+0000 7f25e9e44b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa05a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.396+0000 7f25e9e44b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa05fa0 0x559b0aa0f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.396+0000 7f25e9e44b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa0fec0 0x559b0aa122b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.397+0000 7f25e9e44b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b0a95e990 con 0x559b0aa05680 2024-03-14T10:04:34.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.397+0000 7f25e9e44b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b0a95c350 con 0x559b0aa0fec0 2024-03-14T10:04:34.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.397+0000 7f25e9e44b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b0a957c40 con 0x559b0aa05fa0 2024-03-14T10:04:34.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.397+0000 7f25cf400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa05a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.397+0000 7f25cf400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa05a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45710/0 (socket says 172.21.15.78:45710) 2024-03-14T10:04:34.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.397+0000 7f25cf400700 1 -- 172.21.15.78:0/774028488 learned_addr learned my addr 172.21.15.78:0/774028488 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:34.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.397+0000 7f25cfe00700 1 --2- 172.21.15.78:0/774028488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa0fec0 0x559b0aa122b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.397+0000 7f25cea00700 1 --2- 172.21.15.78:0/774028488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa05fa0 0x559b0aa0f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.397+0000 7f25cea00700 1 --2- 172.21.15.78:0/3019799083 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f25b40a0c00 0x7f25b40a30c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f25d00747e0 tx=0x7f25bc005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.397+0000 7f25cf400700 1 -- 172.21.15.78:0/774028488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa05fa0 msgr2=0x559b0aa0f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.397+0000 7f25cf400700 1 --2- 172.21.15.78:0/774028488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa05fa0 0x559b0aa0f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.397+0000 7f25cf400700 1 -- 172.21.15.78:0/774028488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa0fec0 msgr2=0x559b0aa122b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.397+0000 7f25cf400700 1 --2- 172.21.15.78:0/774028488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa0fec0 0x559b0aa122b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.397+0000 7f25cf400700 1 -- 172.21.15.78:0/774028488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b0a7d93b0 con 0x559b0aa05680 2024-03-14T10:04:34.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.397+0000 7f25cea00700 1 --2- 172.21.15.78:0/774028488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa05fa0 0x559b0aa0f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:04:34.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.397+0000 7f25cf400700 1 --2- 172.21.15.78:0/774028488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa05a60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f25c00a18a0 tx=0x7f25c009a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9864ecebb26ba815 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.398+0000 7f25c4c00700 1 -- 172.21.15.78:0/774028488 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c009d780 con 0x559b0aa05680 2024-03-14T10:04:34.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.398+0000 7f25c4c00700 1 -- 172.21.15.78:0/774028488 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f25c00a1dd0 con 0x559b0aa05680 2024-03-14T10:04:34.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.398+0000 7f25c4c00700 1 -- 172.21.15.78:0/774028488 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c0099040 con 0x559b0aa05680 2024-03-14T10:04:34.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.398+0000 7f25e9e44b80 1 -- 172.21.15.78:0/774028488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 msgr2=0x559b0aa05a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.398+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/774028488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa05a60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f25c00a18a0 tx=0x7f25c009a050 comp rx=0 tx=0).stop 2024-03-14T10:04:34.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.398+0000 7f25e9e44b80 1 -- 172.21.15.78:0/774028488 shutdown_connections 2024-03-14T10:04:34.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.398+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/774028488 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa0fec0 0x559b0aa122b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.398+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/774028488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa05a60 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.398+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/774028488 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa05fa0 0x559b0aa0f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.398+0000 7f25e9e44b80 1 -- 172.21.15.78:0/774028488 >> 172.21.15.78:0/774028488 conn(0x559b0a97b220 msgr2=0x559b0aa00aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.399+0000 7f25e9e44b80 1 -- 172.21.15.78:0/774028488 shutdown_connections 2024-03-14T10:04:34.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.399+0000 7f25e9e44b80 1 -- 172.21.15.78:0/774028488 wait complete. 2024-03-14T10:04:34.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.399+0000 7f25e9e44b80 1 -- start start 2024-03-14T10:04:34.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.399+0000 7f25e9e44b80 1 Processor -- start 2024-03-14T10:04:34.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.400+0000 7f25e9e44b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa0f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.400+0000 7f25e9e44b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa05fa0 0x559b0aa0cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.400+0000 7f25e9e44b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa0fec0 0x559b0aa0d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.400+0000 7f25e9e44b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b0a8f3d80 con 0x559b0aa05680 2024-03-14T10:04:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.400+0000 7f25e9e44b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b0a8f3a20 con 0x559b0aa0fec0 2024-03-14T10:04:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.400+0000 7f25e9e44b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b0a7db6a0 con 0x559b0aa05fa0 2024-03-14T10:04:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.400+0000 7f25e9e44b80 10 client.0 _ll_register_callbacks cb 0x559b0a75f3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.400+0000 7f25cf400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa0f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.400+0000 7f25cf400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa0f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45724/0 (socket says 172.21.15.78:45724) 2024-03-14T10:04:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.400+0000 7f25cea00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa05fa0 0x559b0aa0cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.400+0000 7f25cf400700 1 -- 172.21.15.78:0/612779746 learned_addr learned my addr 172.21.15.78:0/612779746 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.400+0000 7f25cfe00700 1 --2- 172.21.15.78:0/612779746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa0fec0 0x559b0aa0d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.401+0000 7f25cf400700 1 -- 172.21.15.78:0/612779746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa05fa0 msgr2=0x559b0aa0cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.401+0000 7f25cf400700 1 --2- 172.21.15.78:0/612779746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa05fa0 0x559b0aa0cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.401+0000 7f25cf400700 1 -- 172.21.15.78:0/612779746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa0fec0 msgr2=0x559b0aa0d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.401+0000 7f25cf400700 1 --2- 172.21.15.78:0/612779746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa0fec0 0x559b0aa0d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.401+0000 7f25cf400700 1 -- 172.21.15.78:0/612779746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d01a68d0 con 0x559b0aa05680 2024-03-14T10:04:34.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.401+0000 7f25cea00700 1 --2- 172.21.15.78:0/612779746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa05fa0 0x559b0aa0cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:04:34.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.401+0000 7f25cfe00700 1 --2- 172.21.15.78:0/612779746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa0fec0 0x559b0aa0d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:04:34.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.401+0000 7f25cf400700 1 --2- 172.21.15.78:0/612779746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa0f930 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f25c0095e50 tx=0x7f25c00c9cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61984b22a47a0bd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.401+0000 7f25b9600700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:04:34.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.401+0000 7f25b9600700 1 -- 172.21.15.78:0/612779746 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c0099db0 con 0x559b0aa05680 2024-03-14T10:04:34.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.401+0000 7f25b9600700 1 -- 172.21.15.78:0/612779746 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f25c00a1940 con 0x559b0aa05680 2024-03-14T10:04:34.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.401+0000 7f25e9e44b80 1 -- 172.21.15.78:0/612779746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559b0a9f8720 con 0x559b0aa05680 2024-03-14T10:04:34.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.401+0000 7f25b9600700 1 -- 172.21.15.78:0/612779746 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25c00a1b00 con 0x559b0aa05680 2024-03-14T10:04:34.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.401+0000 7f25e9e44b80 10 client.17676 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:34.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.401+0000 7f25e9e44b80 1 -- 172.21.15.78:0/612779746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559b0a785b60 con 0x559b0aa05680 2024-03-14T10:04:34.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.401+0000 7f25b9600700 1 -- 172.21.15.78:0/612779746 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559b0a9f8720 con 0x559b0aa05680 2024-03-14T10:04:34.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.402+0000 7f25b9600700 1 -- 172.21.15.78:0/612779746 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f25c00988b0 con 0x559b0aa05680 2024-03-14T10:04:34.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.402+0000 7f25e9e44b80 10 client.17676 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:34.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.402+0000 7f25e9e44b80 10 client.17676 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:04:34.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.402+0000 7f25e9e44b80 20 client.17676 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:34.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.402+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/612779746 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559b0aa1f800 0x559b0aa21bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.402+0000 7f25e9e44b80 4 client.17676 mds_command: new command op to 24527 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:04:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.402+0000 7f25e9e44b80 1 -- 172.21.15.78:0/612779746 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559b0a950700 con 0x559b0aa1f800 2024-03-14T10:04:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.402+0000 7f25cea00700 1 --2- 172.21.15.78:0/612779746 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559b0aa1f800 0x559b0aa21bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.403+0000 7f25b9600700 10 client.17676 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:04:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.403+0000 7f25cea00700 1 --2- 172.21.15.78:0/612779746 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559b0aa1f800 0x559b0aa21bf0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=68b570a01d1621e6 server_cookie=d5f3ce299afc4072 in_seq=0 out_seq=0 2024-03-14T10:04:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.403+0000 7f25b9600700 1 -- 172.21.15.78:0/612779746 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x559b0a950700 con 0x559b0aa1f800 2024-03-14T10:04:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.403+0000 7f25b9600700 10 client.17676 handle_command_reply: tid=0 2024-03-14T10:04:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.403+0000 7f25e9e44b80 1 client.17676 shutdown 2024-03-14T10:04:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.404+0000 7f25e9e44b80 1 -- 172.21.15.78:0/612779746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 msgr2=0x559b0aa0f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.404+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/612779746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa0f930 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f25c0095e50 tx=0x7f25c00c9cc0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.404+0000 7f25e9e44b80 1 -- 172.21.15.78:0/612779746 shutdown_connections 2024-03-14T10:04:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.404+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/612779746 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559b0aa1f800 0x559b0aa21bf0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.404+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/612779746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa0fec0 0x559b0aa0d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.404+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/612779746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa0f930 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.404+0000 7f25b9600700 0 client.17676 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:04:34.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.404+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/612779746 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa05fa0 0x559b0aa0cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.404+0000 7f25e9e44b80 1 -- 172.21.15.78:0/612779746 >> 172.21.15.78:0/612779746 conn(0x559b0a97b220 msgr2=0x559b0aa00690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.404+0000 7f25e9e44b80 1 -- 172.21.15.78:0/612779746 shutdown_connections 2024-03-14T10:04:34.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.404+0000 7f25e9e44b80 1 -- 172.21.15.78:0/612779746 wait complete. 2024-03-14T10:04:34.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.404+0000 7f25e9e44b80 20 client.17676 trim_cache size 0 max 16384 2024-03-14T10:04:34.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.417+0000 7f25e9e44b80 1 Processor -- start 2024-03-14T10:04:34.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.417+0000 7f25e9e44b80 1 -- start start 2024-03-14T10:04:34.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.417+0000 7f25e9e44b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa00c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.417+0000 7f25e9e44b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa05fa0 0x559b0aa09e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.417+0000 7f25e9e44b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa0fec0 0x559b0aa0a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.417+0000 7f25e9e44b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b0a9fd230 con 0x559b0aa05680 2024-03-14T10:04:34.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.417+0000 7f25e9e44b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b0aa05de0 con 0x559b0aa05fa0 2024-03-14T10:04:34.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.417+0000 7f25e9e44b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b0aa02060 con 0x559b0aa0fec0 2024-03-14T10:04:34.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.418+0000 7f25cf400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa00c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.418+0000 7f25cf400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa00c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45736/0 (socket says 172.21.15.78:45736) 2024-03-14T10:04:34.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.418+0000 7f25cf400700 1 -- 172.21.15.78:0/4208658706 learned_addr learned my addr 172.21.15.78:0/4208658706 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:34.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.418+0000 7f25cfe00700 1 --2- 172.21.15.78:0/4208658706 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa0fec0 0x559b0aa0a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.418+0000 7f25cea00700 1 --2- 172.21.15.78:0/4208658706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa05fa0 0x559b0aa09e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.418+0000 7f25cf400700 1 -- 172.21.15.78:0/4208658706 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa0fec0 msgr2=0x559b0aa0a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.418+0000 7f25cf400700 1 --2- 172.21.15.78:0/4208658706 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa0fec0 0x559b0aa0a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.418+0000 7f25cf400700 1 -- 172.21.15.78:0/4208658706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa05fa0 msgr2=0x559b0aa09e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.418+0000 7f25cf400700 1 --2- 172.21.15.78:0/4208658706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa05fa0 0x559b0aa09e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.418+0000 7f25cf400700 1 -- 172.21.15.78:0/4208658706 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b0a91dea0 con 0x559b0aa05680 2024-03-14T10:04:34.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.418+0000 7f25cf400700 1 --2- 172.21.15.78:0/4208658706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa00c30 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f25c0099650 tx=0x7f25c0099850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5334c1490c29960c server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.419+0000 7f25c5600700 1 -- 172.21.15.78:0/4208658706 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c009a050 con 0x559b0aa05680 2024-03-14T10:04:34.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.419+0000 7f25c5600700 1 -- 172.21.15.78:0/4208658706 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f25c00187e0 con 0x559b0aa05680 2024-03-14T10:04:34.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.419+0000 7f25c5600700 1 -- 172.21.15.78:0/4208658706 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c00a1940 con 0x559b0aa05680 2024-03-14T10:04:34.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.419+0000 7f25e9e44b80 1 -- 172.21.15.78:0/4208658706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 msgr2=0x559b0aa00c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.419+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/4208658706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa00c30 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f25c0099650 tx=0x7f25c0099850 comp rx=0 tx=0).stop 2024-03-14T10:04:34.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.419+0000 7f25e9e44b80 1 -- 172.21.15.78:0/4208658706 shutdown_connections 2024-03-14T10:04:34.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.419+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/4208658706 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa0fec0 0x559b0aa0a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.419+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/4208658706 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05680 0x559b0aa00c30 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.419+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/4208658706 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa05fa0 0x559b0aa09e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.419+0000 7f25e9e44b80 1 -- 172.21.15.78:0/4208658706 >> 172.21.15.78:0/4208658706 conn(0x559b0a97b220 msgr2=0x559b0aa0b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.419+0000 7f25e9e44b80 1 -- 172.21.15.78:0/4208658706 shutdown_connections 2024-03-14T10:04:34.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.419+0000 7f25e9e44b80 1 -- 172.21.15.78:0/4208658706 wait complete. 2024-03-14T10:04:34.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.420+0000 7f25e9e44b80 1 -- start start 2024-03-14T10:04:34.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.420+0000 7f25e9e44b80 1 Processor -- start 2024-03-14T10:04:34.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.420+0000 7f25e9e44b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa05680 0x559b0aa14260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.420+0000 7f25e9e44b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05fa0 0x559b0aa147a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.420+0000 7f25e9e44b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa0fec0 0x559b0aa14ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.420+0000 7f25e9e44b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b0aa703c0 con 0x559b0aa05fa0 2024-03-14T10:04:34.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.420+0000 7f25e9e44b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b0aa70530 con 0x559b0aa05680 2024-03-14T10:04:34.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.420+0000 7f25e9e44b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b0aa706f0 con 0x559b0aa0fec0 2024-03-14T10:04:34.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.421+0000 7f25e9e44b80 10 client.0 _ll_register_callbacks cb 0x7f25c8009f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:34.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.421+0000 7f25cea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05fa0 0x559b0aa147a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.421+0000 7f25cea00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05fa0 0x559b0aa147a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45750/0 (socket says 172.21.15.78:45750) 2024-03-14T10:04:34.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.421+0000 7f25cea00700 1 -- 172.21.15.78:0/2720484613 learned_addr learned my addr 172.21.15.78:0/2720484613 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:34.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.421+0000 7f25cfe00700 1 --2- 172.21.15.78:0/2720484613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa0fec0 0x559b0aa14ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.421+0000 7f25cf400700 1 --2- 172.21.15.78:0/2720484613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa05680 0x559b0aa14260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.421+0000 7f25cfe00700 1 -- 172.21.15.78:0/2720484613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa05680 msgr2=0x559b0aa14260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.421+0000 7f25cfe00700 1 --2- 172.21.15.78:0/2720484613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa05680 0x559b0aa14260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.421+0000 7f25cfe00700 1 -- 172.21.15.78:0/2720484613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05fa0 msgr2=0x559b0aa147a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.421+0000 7f25cfe00700 1 --2- 172.21.15.78:0/2720484613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05fa0 0x559b0aa147a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.421+0000 7f25cfe00700 1 -- 172.21.15.78:0/2720484613 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b0a785b60 con 0x559b0aa0fec0 2024-03-14T10:04:34.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.421+0000 7f25cea00700 1 --2- 172.21.15.78:0/2720484613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05fa0 0x559b0aa147a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:04:34.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.421+0000 7f25cfe00700 1 --2- 172.21.15.78:0/2720484613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa0fec0 0x559b0aa14ce0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f25c800ec70 tx=0x7f25c800ef80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fd6b0d6508eace1 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.421+0000 7f25b9600700 10 client.0 ms_handle_connect on v2:172.21.15.152:3300/0 2024-03-14T10:04:34.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.422+0000 7f25b9600700 1 -- 172.21.15.78:0/2720484613 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c801c070 con 0x559b0aa0fec0 2024-03-14T10:04:34.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.422+0000 7f25e9e44b80 1 -- 172.21.15.78:0/2720484613 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f25c00a1dd0 con 0x559b0aa0fec0 2024-03-14T10:04:34.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.422+0000 7f25b9600700 1 -- 172.21.15.78:0/2720484613 <== mon.1 v2:172.21.15.152:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f25c8007c60 con 0x559b0aa0fec0 2024-03-14T10:04:34.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.422+0000 7f25b9600700 1 -- 172.21.15.78:0/2720484613 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25c8007e20 con 0x559b0aa0fec0 2024-03-14T10:04:34.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.422+0000 7f25e9e44b80 10 client.27724 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:34.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.422+0000 7f25e9e44b80 1 -- 172.21.15.78:0/2720484613 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x559b0a7ddcd0 con 0x559b0aa0fec0 2024-03-14T10:04:34.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.422+0000 7f25b9600700 1 -- 172.21.15.78:0/2720484613 <== mon.1 v2:172.21.15.152:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f25c00a1dd0 con 0x559b0aa0fec0 2024-03-14T10:04:34.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.422+0000 7f25b9600700 1 -- 172.21.15.78:0/2720484613 <== mon.1 v2:172.21.15.152:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f25c8016540 con 0x559b0aa0fec0 2024-03-14T10:04:34.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.422+0000 7f25e9e44b80 10 client.27724 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:34.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.422+0000 7f25e9e44b80 10 client.27724 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-14T10:04:34.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.423+0000 7f25e9e44b80 20 client.27724 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:34.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.423+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/2720484613 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559b0aa71f80 0x559b0aa743a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.423+0000 7f25e9e44b80 4 client.27724 mds_command: new command op to 24527 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:04:34.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.423+0000 7f25e9e44b80 1 -- 172.21.15.78:0/2720484613 --> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559b0a7a6cf0 con 0x559b0aa71f80 2024-03-14T10:04:34.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.423+0000 7f25cf400700 1 --2- 172.21.15.78:0/2720484613 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559b0aa71f80 0x559b0aa743a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.423+0000 7f25b9600700 10 client.27724 ms_handle_connect on v2:172.21.15.132:6836/142126094 2024-03-14T10:04:34.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.423+0000 7f25cf400700 1 --2- 172.21.15.78:0/2720484613 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559b0aa71f80 0x559b0aa743a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7bcadf8d990c1046 server_cookie=1f186188f6f5ae26 in_seq=0 out_seq=0 2024-03-14T10:04:34.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.423+0000 7f25b9600700 1 -- 172.21.15.78:0/2720484613 <== mds.2 v2:172.21.15.132:6836/142126094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559b0a7a6cf0 con 0x559b0aa71f80 2024-03-14T10:04:34.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.423+0000 7f25b9600700 10 client.27724 handle_command_reply: tid=0 2024-03-14T10:04:34.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.423+0000 7f25e9e44b80 1 client.27724 shutdown 2024-03-14T10:04:34.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.424+0000 7f25e9e44b80 1 -- 172.21.15.78:0/2720484613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa0fec0 msgr2=0x559b0aa14ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.424+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/2720484613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa0fec0 0x559b0aa14ce0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f25c800ec70 tx=0x7f25c800ef80 comp rx=0 tx=0).stop 2024-03-14T10:04:34.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.424+0000 7f25e9e44b80 1 -- 172.21.15.78:0/2720484613 shutdown_connections 2024-03-14T10:04:34.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.424+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/2720484613 >> [v2:172.21.15.132:6836/142126094,v1:172.21.15.132:6837/142126094] conn(0x559b0aa71f80 0x559b0aa743a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.424+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/2720484613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559b0aa05680 0x559b0aa14260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.424+0000 7f25b9600700 0 client.27724 ms_handle_reset on v2:172.21.15.132:6836/142126094 2024-03-14T10:04:34.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.424+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/2720484613 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x559b0aa0fec0 0x559b0aa14ce0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.424+0000 7f25e9e44b80 1 --2- 172.21.15.78:0/2720484613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559b0aa05fa0 0x559b0aa147a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.424+0000 7f25e9e44b80 1 -- 172.21.15.78:0/2720484613 >> 172.21.15.78:0/2720484613 conn(0x559b0a97b220 msgr2=0x559b0aa0c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.424+0000 7f25e9e44b80 1 -- 172.21.15.78:0/2720484613 shutdown_connections 2024-03-14T10:04:34.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.424+0000 7f25e9e44b80 1 -- 172.21.15.78:0/2720484613 wait complete. 2024-03-14T10:04:34.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.424+0000 7f25e9e44b80 20 client.27724 trim_cache size 0 max 16384 2024-03-14T10:04:34.429 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:04:34.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.424+0000 7f25d6400700 1 -- 172.21.15.78:0/3019799083 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f25b40a0c00 msgr2=0x7f25b40a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.424+0000 7f25d6400700 1 --2- 172.21.15.78:0/3019799083 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f25b40a0c00 0x7f25b40a30c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f25d00747e0 tx=0x7f25bc005fd0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.425+0000 7f25d6400700 1 -- 172.21.15.78:0/3019799083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25d0100f30 msgr2=0x7f25d00730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.425+0000 7f25d6400700 1 --2- 172.21.15.78:0/3019799083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25d0100f30 0x7f25d00730b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f25c000e9a0 tx=0x7f25c000ecb0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.425+0000 7f25d6400700 1 -- 172.21.15.78:0/3019799083 shutdown_connections 2024-03-14T10:04:34.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.425+0000 7f25d6400700 1 --2- 172.21.15.78:0/3019799083 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f25b40a0c00 0x7f25b40a30c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.425+0000 7f25d6400700 1 --2- 172.21.15.78:0/3019799083 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f25d0073b30 0x7f25d01a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.425+0000 7f25d6400700 1 --2- 172.21.15.78:0/3019799083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25d0100f30 0x7f25d00730b0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.425+0000 7f25d6400700 1 --2- 172.21.15.78:0/3019799083 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f25d0106050 0x7f25d00735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.425+0000 7f25d6400700 1 -- 172.21.15.78:0/3019799083 >> 172.21.15.78:0/3019799083 conn(0x7f25d00fc7b0 msgr2=0x7f25d00fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.425+0000 7f25d6400700 1 -- 172.21.15.78:0/3019799083 shutdown_connections 2024-03-14T10:04:34.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.425+0000 7f25d6400700 1 -- 172.21.15.78:0/3019799083 wait complete. 2024-03-14T10:04:34.441 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:34 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1132203349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:04:34.443 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-14T10:04:34.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:34 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1132203349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:04:34.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:34 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1132203349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:04:34.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.636+0000 7f1094600700 1 -- 172.21.15.78:0/75705507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108c106040 msgr2=0x7f108c10dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.636+0000 7f1094600700 1 --2- 172.21.15.78:0/75705507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108c106040 0x7f108c10dbc0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f108400b0a0 tx=0x7f108400b3b0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.636+0000 7f1094600700 1 -- 172.21.15.78:0/75705507 shutdown_connections 2024-03-14T10:04:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.636+0000 7f1094600700 1 --2- 172.21.15.78:0/75705507 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108c106040 0x7f108c10dbc0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.636+0000 7f1094600700 1 --2- 172.21.15.78:0/75705507 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f108c100f20 0x7f108c101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.636+0000 7f1094600700 1 --2- 172.21.15.78:0/75705507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f108c1018d0 0x7f108c105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.636+0000 7f1094600700 1 -- 172.21.15.78:0/75705507 >> 172.21.15.78:0/75705507 conn(0x7f108c0fc7a0 msgr2=0x7f108c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.636+0000 7f1094600700 1 -- 172.21.15.78:0/75705507 shutdown_connections 2024-03-14T10:04:34.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.636+0000 7f1094600700 1 -- 172.21.15.78:0/75705507 wait complete. 2024-03-14T10:04:34.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.637+0000 7f1094600700 1 Processor -- start 2024-03-14T10:04:34.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.638+0000 7f1094600700 1 -- start start 2024-03-14T10:04:34.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.638+0000 7f1094600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f108c100f20 0x7f108c19f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.638+0000 7f1094600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108c106040 0x7f108c19fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.638+0000 7f1094600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f108c199750 0x7f108c199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.638+0000 7f1094600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108c19a1d0 con 0x7f108c106040 2024-03-14T10:04:34.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.638+0000 7f1094600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108c19a310 con 0x7f108c100f20 2024-03-14T10:04:34.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.638+0000 7f1094600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108c19a4d0 con 0x7f108c199750 2024-03-14T10:04:34.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.638+0000 7f1091200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108c106040 0x7f108c19fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.638+0000 7f1091200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108c106040 0x7f108c19fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45770/0 (socket says 172.21.15.78:45770) 2024-03-14T10:04:34.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.638+0000 7f1091200700 1 -- 172.21.15.78:0/495310088 learned_addr learned my addr 172.21.15.78:0/495310088 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:34.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.638+0000 7f1092600700 1 --2- 172.21.15.78:0/495310088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f108c199750 0x7f108c199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.638+0000 7f1091c00700 1 --2- 172.21.15.78:0/495310088 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f108c100f20 0x7f108c19f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.639+0000 7f1091200700 1 -- 172.21.15.78:0/495310088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f108c199750 msgr2=0x7f108c199c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.639+0000 7f1091200700 1 --2- 172.21.15.78:0/495310088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f108c199750 0x7f108c199c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.639+0000 7f1091200700 1 -- 172.21.15.78:0/495310088 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f108c100f20 msgr2=0x7f108c19f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.639+0000 7f1091200700 1 --2- 172.21.15.78:0/495310088 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f108c100f20 0x7f108c19f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.639+0000 7f1091200700 1 -- 172.21.15.78:0/495310088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1084009d00 con 0x7f108c106040 2024-03-14T10:04:34.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.639+0000 7f1091200700 1 --2- 172.21.15.78:0/495310088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108c106040 0x7f108c19fb60 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f107800e970 tx=0x7f107800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.639+0000 7f108a000700 1 -- 172.21.15.78:0/495310088 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1078009790 con 0x7f108c106040 2024-03-14T10:04:34.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.639+0000 7f1094600700 1 -- 172.21.15.78:0/495310088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108c1a6490 con 0x7f108c106040 2024-03-14T10:04:34.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.639+0000 7f108a000700 1 -- 172.21.15.78:0/495310088 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f107800c7e0 con 0x7f108c106040 2024-03-14T10:04:34.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.639+0000 7f108a000700 1 -- 172.21.15.78:0/495310088 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10780054e0 con 0x7f108c106040 2024-03-14T10:04:34.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.639+0000 7f1094600700 1 -- 172.21.15.78:0/495310088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f108c1a69a0 con 0x7f108c106040 2024-03-14T10:04:34.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.642+0000 7f108a000700 1 -- 172.21.15.78:0/495310088 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f107800c950 con 0x7f108c106040 2024-03-14T10:04:34.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.643+0000 7f10a8179b80 1 Processor -- start 2024-03-14T10:04:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.644+0000 7f10a8179b80 1 -- start start 2024-03-14T10:04:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.644+0000 7f108a000700 1 --2- 172.21.15.78:0/495310088 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f10740a0c00 0x7f10740a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.644+0000 7f108a000700 1 -- 172.21.15.78:0/495310088 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f1078014070 con 0x7f108c106040 2024-03-14T10:04:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.644+0000 7f1092600700 1 --2- 172.21.15.78:0/495310088 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f10740a0c00 0x7f10740a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.644+0000 7f10a8179b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63e680 0x55ed1d63ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.644+0000 7f10a8179b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63efa0 0x55ed1d648980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.644+0000 7f10a8179b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d648ec0 0x55ed1d64b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.644+0000 7f10a8179b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed1d597990 con 0x55ed1d63e680 2024-03-14T10:04:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.644+0000 7f10a8179b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed1d595350 con 0x55ed1d648ec0 2024-03-14T10:04:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.644+0000 7f10a8179b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed1d590c40 con 0x55ed1d63efa0 2024-03-14T10:04:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.644+0000 7f1091c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63e680 0x55ed1d63ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.644+0000 7f1091c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63e680 0x55ed1d63ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45778/0 (socket says 172.21.15.78:45778) 2024-03-14T10:04:34.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.644+0000 7f1091c00700 1 -- 172.21.15.78:0/826613877 learned_addr learned my addr 172.21.15.78:0/826613877 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:34.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.644+0000 7f1092600700 1 --2- 172.21.15.78:0/826613877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d648ec0 0x55ed1d64b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.645+0000 7f1091200700 1 --2- 172.21.15.78:0/826613877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63efa0 0x55ed1d648980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.645+0000 7f1092600700 1 --2- 172.21.15.78:0/495310088 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f10740a0c00 0x7f10740a30c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f108400b780 tx=0x7f10840095c0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.645+0000 7f1091c00700 1 -- 172.21.15.78:0/826613877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63efa0 msgr2=0x55ed1d648980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.645+0000 7f1091c00700 1 --2- 172.21.15.78:0/826613877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63efa0 0x55ed1d648980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.645+0000 7f1091c00700 1 -- 172.21.15.78:0/826613877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d648ec0 msgr2=0x55ed1d64b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.645+0000 7f1091c00700 1 --2- 172.21.15.78:0/826613877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d648ec0 0x55ed1d64b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.645+0000 7f1091c00700 1 -- 172.21.15.78:0/826613877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed1d4123b0 con 0x55ed1d63e680 2024-03-14T10:04:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.645+0000 7f1091c00700 1 --2- 172.21.15.78:0/826613877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63e680 0x55ed1d63ea60 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f1080009d80 tx=0x7f1080011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21bcda7ce4658364 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.645+0000 7f107ea00700 1 -- 172.21.15.78:0/826613877 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1080004230 con 0x55ed1d63e680 2024-03-14T10:04:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.645+0000 7f107ea00700 1 -- 172.21.15.78:0/826613877 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1080004390 con 0x55ed1d63e680 2024-03-14T10:04:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.645+0000 7f107ea00700 1 -- 172.21.15.78:0/826613877 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1080019880 con 0x55ed1d63e680 2024-03-14T10:04:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.646+0000 7f10a8179b80 1 -- 172.21.15.78:0/826613877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63e680 msgr2=0x55ed1d63ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.646+0000 7f10a8179b80 1 --2- 172.21.15.78:0/826613877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63e680 0x55ed1d63ea60 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f1080009d80 tx=0x7f1080011720 comp rx=0 tx=0).stop 2024-03-14T10:04:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.646+0000 7f10a8179b80 1 -- 172.21.15.78:0/826613877 shutdown_connections 2024-03-14T10:04:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.646+0000 7f10a8179b80 1 --2- 172.21.15.78:0/826613877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d648ec0 0x55ed1d64b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.646+0000 7f10a8179b80 1 --2- 172.21.15.78:0/826613877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63e680 0x55ed1d63ea60 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.646+0000 7f10a8179b80 1 --2- 172.21.15.78:0/826613877 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63efa0 0x55ed1d648980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.646+0000 7f10a8179b80 1 -- 172.21.15.78:0/826613877 >> 172.21.15.78:0/826613877 conn(0x55ed1d5b4220 msgr2=0x55ed1d639aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.646+0000 7f10a8179b80 1 -- 172.21.15.78:0/826613877 shutdown_connections 2024-03-14T10:04:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.646+0000 7f10a8179b80 1 -- 172.21.15.78:0/826613877 wait complete. 2024-03-14T10:04:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.647+0000 7f10a8179b80 1 -- start start 2024-03-14T10:04:34.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.647+0000 7f10a8179b80 1 Processor -- start 2024-03-14T10:04:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.647+0000 7f10a8179b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63e680 0x55ed1d6488c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.647+0000 7f10a8179b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63efa0 0x55ed1d645bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.647+0000 7f10a8179b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d648ec0 0x55ed1d6461c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.647+0000 7f10a8179b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed1d52cd80 con 0x55ed1d63efa0 2024-03-14T10:04:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.647+0000 7f10a8179b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed1d52ca20 con 0x55ed1d648ec0 2024-03-14T10:04:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.647+0000 7f10a8179b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed1d4146a0 con 0x55ed1d63e680 2024-03-14T10:04:34.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.648+0000 7f10a8179b80 10 client.0 _ll_register_callbacks cb 0x55ed1d3983a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.648+0000 7f1091200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63efa0 0x55ed1d645bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.648+0000 7f1091c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63e680 0x55ed1d6488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.648+0000 7f1091200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63efa0 0x55ed1d645bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45780/0 (socket says 172.21.15.78:45780) 2024-03-14T10:04:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.648+0000 7f1091200700 1 -- 172.21.15.78:0/752177460 learned_addr learned my addr 172.21.15.78:0/752177460 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.648+0000 7f1091c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63e680 0x55ed1d6488c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:48874/0 (socket says 172.21.15.78:48874) 2024-03-14T10:04:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.648+0000 7f1092600700 1 --2- 172.21.15.78:0/752177460 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d648ec0 0x55ed1d6461c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.648+0000 7f1091200700 1 -- 172.21.15.78:0/752177460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63e680 msgr2=0x55ed1d6488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.648+0000 7f1091200700 1 --2- 172.21.15.78:0/752177460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63e680 0x55ed1d6488c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.648+0000 7f1091200700 1 -- 172.21.15.78:0/752177460 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d648ec0 msgr2=0x55ed1d6461c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.648+0000 7f1091200700 1 --2- 172.21.15.78:0/752177460 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d648ec0 0x55ed1d6461c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.648+0000 7f1091200700 1 -- 172.21.15.78:0/752177460 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10800099e0 con 0x55ed1d63efa0 2024-03-14T10:04:34.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.648+0000 7f1092600700 1 --2- 172.21.15.78:0/752177460 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d648ec0 0x55ed1d6461c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:04:34.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.648+0000 7f1091200700 1 --2- 172.21.15.78:0/752177460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63efa0 0x55ed1d645bb0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f1078099960 tx=0x7f107800cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47081c53aed83a18 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.648+0000 7f1073400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:04:34.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.649+0000 7f1073400700 1 -- 172.21.15.78:0/752177460 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1078016530 con 0x55ed1d63efa0 2024-03-14T10:04:34.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.649+0000 7f1073400700 1 -- 172.21.15.78:0/752177460 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f10780a1e50 con 0x55ed1d63efa0 2024-03-14T10:04:34.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.649+0000 7f10a8179b80 1 -- 172.21.15.78:0/752177460 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ed1d631720 con 0x55ed1d63efa0 2024-03-14T10:04:34.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.649+0000 7f1073400700 1 -- 172.21.15.78:0/752177460 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10780b2200 con 0x55ed1d63efa0 2024-03-14T10:04:34.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.649+0000 7f10a8179b80 10 client.17694 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:34.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.649+0000 7f10a8179b80 1 -- 172.21.15.78:0/752177460 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ed1d3beb60 con 0x55ed1d63efa0 2024-03-14T10:04:34.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.649+0000 7f1073400700 1 -- 172.21.15.78:0/752177460 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ed1d631720 con 0x55ed1d63efa0 2024-03-14T10:04:34.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.649+0000 7f1073400700 1 -- 172.21.15.78:0/752177460 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1078098f50 con 0x55ed1d63efa0 2024-03-14T10:04:34.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.649+0000 7f10a8179b80 10 client.17694 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:34.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.649+0000 7f10a8179b80 10 client.17694 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:04:34.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.649+0000 7f10a8179b80 20 client.17694 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:34.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.649+0000 7f10a8179b80 1 --2- 172.21.15.78:0/752177460 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55ed1d658aa0 0x55ed1d65aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.649+0000 7f10a8179b80 4 client.17694 mds_command: new command op to 24538 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:04:34.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.649+0000 7f10a8179b80 1 -- 172.21.15.78:0/752177460 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed1d589700 con 0x55ed1d658aa0 2024-03-14T10:04:34.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.650+0000 7f1091c00700 1 --2- 172.21.15.78:0/752177460 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55ed1d658aa0 0x55ed1d65aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.650+0000 7f1091c00700 1 --2- 172.21.15.78:0/752177460 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55ed1d658aa0 0x55ed1d65aec0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8dedaf57a770d2eb server_cookie=27a26fcd142ae436 in_seq=0 out_seq=0 2024-03-14T10:04:34.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.650+0000 7f1073400700 10 client.17694 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:04:34.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.651+0000 7f1073400700 1 -- 172.21.15.78:0/752177460 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55ed1d589700 con 0x55ed1d658aa0 2024-03-14T10:04:34.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.651+0000 7f1073400700 10 client.17694 handle_command_reply: tid=0 2024-03-14T10:04:34.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.651+0000 7f10a8179b80 1 client.17694 shutdown 2024-03-14T10:04:34.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.651+0000 7f10a8179b80 1 -- 172.21.15.78:0/752177460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63efa0 msgr2=0x55ed1d645bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.651+0000 7f10a8179b80 1 --2- 172.21.15.78:0/752177460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63efa0 0x55ed1d645bb0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f1078099960 tx=0x7f107800cd60 comp rx=0 tx=0).stop 2024-03-14T10:04:34.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.651+0000 7f10a8179b80 1 -- 172.21.15.78:0/752177460 shutdown_connections 2024-03-14T10:04:34.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.651+0000 7f10a8179b80 1 --2- 172.21.15.78:0/752177460 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55ed1d658aa0 0x55ed1d65aec0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.651+0000 7f10a8179b80 1 --2- 172.21.15.78:0/752177460 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d648ec0 0x55ed1d6461c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.651+0000 7f1073400700 0 client.17694 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:04:34.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.651+0000 7f10a8179b80 1 --2- 172.21.15.78:0/752177460 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63e680 0x55ed1d6488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.651+0000 7f10a8179b80 1 --2- 172.21.15.78:0/752177460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63efa0 0x55ed1d645bb0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.651+0000 7f10a8179b80 1 -- 172.21.15.78:0/752177460 >> 172.21.15.78:0/752177460 conn(0x55ed1d5b4220 msgr2=0x55ed1d639690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.651+0000 7f10a8179b80 1 -- 172.21.15.78:0/752177460 shutdown_connections 2024-03-14T10:04:34.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.651+0000 7f10a8179b80 1 -- 172.21.15.78:0/752177460 wait complete. 2024-03-14T10:04:34.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.651+0000 7f10a8179b80 20 client.17694 trim_cache size 0 max 16384 2024-03-14T10:04:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.664+0000 7f10a8179b80 1 Processor -- start 2024-03-14T10:04:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.664+0000 7f10a8179b80 1 -- start start 2024-03-14T10:04:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.664+0000 7f10a8179b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d63e680 0x55ed1d654830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.664+0000 7f10a8179b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63efa0 0x55ed1d6483f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.664+0000 7f10a8179b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d648ec0 0x55ed1d642e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.664+0000 7f10a8179b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed1d63b060 con 0x55ed1d648ec0 2024-03-14T10:04:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.664+0000 7f10a8179b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed1d633d00 con 0x55ed1d63e680 2024-03-14T10:04:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.664+0000 7f10a8179b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed1d65fbe0 con 0x55ed1d63efa0 2024-03-14T10:04:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.665+0000 7f1092600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d648ec0 0x55ed1d642e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.665+0000 7f1091200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63efa0 0x55ed1d6483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.665+0000 7f1092600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d648ec0 0x55ed1d642e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45790/0 (socket says 172.21.15.78:45790) 2024-03-14T10:04:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.665+0000 7f1092600700 1 -- 172.21.15.78:0/3172928104 learned_addr learned my addr 172.21.15.78:0/3172928104 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.665+0000 7f1091c00700 1 --2- 172.21.15.78:0/3172928104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d63e680 0x55ed1d654830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.665+0000 7f1091200700 1 -- 172.21.15.78:0/3172928104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d63e680 msgr2=0x55ed1d654830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.665+0000 7f1091200700 1 --2- 172.21.15.78:0/3172928104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d63e680 0x55ed1d654830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.665+0000 7f1091200700 1 -- 172.21.15.78:0/3172928104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d648ec0 msgr2=0x55ed1d642e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.665+0000 7f1091200700 1 --2- 172.21.15.78:0/3172928104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d648ec0 0x55ed1d642e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.665+0000 7f1091200700 1 -- 172.21.15.78:0/3172928104 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed1d556ea0 con 0x55ed1d63efa0 2024-03-14T10:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.665+0000 7f1091200700 1 --2- 172.21.15.78:0/3172928104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63efa0 0x55ed1d6483f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f10780996d0 tx=0x7f1078097f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa0585c1190e8108 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.666+0000 7f107f400700 1 -- 172.21.15.78:0/3172928104 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10780a1e50 con 0x55ed1d63efa0 2024-03-14T10:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.666+0000 7f107f400700 1 -- 172.21.15.78:0/3172928104 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f10780b2200 con 0x55ed1d63efa0 2024-03-14T10:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.666+0000 7f107f400700 1 -- 172.21.15.78:0/3172928104 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1078016330 con 0x55ed1d63efa0 2024-03-14T10:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.666+0000 7f10a8179b80 1 -- 172.21.15.78:0/3172928104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63efa0 msgr2=0x55ed1d6483f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.666+0000 7f10a8179b80 1 --2- 172.21.15.78:0/3172928104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63efa0 0x55ed1d6483f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f10780996d0 tx=0x7f1078097f80 comp rx=0 tx=0).stop 2024-03-14T10:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.666+0000 7f10a8179b80 1 -- 172.21.15.78:0/3172928104 shutdown_connections 2024-03-14T10:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.666+0000 7f10a8179b80 1 --2- 172.21.15.78:0/3172928104 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d63e680 0x55ed1d654830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.666+0000 7f10a8179b80 1 --2- 172.21.15.78:0/3172928104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d648ec0 0x55ed1d642e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.666+0000 7f10a8179b80 1 --2- 172.21.15.78:0/3172928104 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d63efa0 0x55ed1d6483f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.666+0000 7f10a8179b80 1 -- 172.21.15.78:0/3172928104 >> 172.21.15.78:0/3172928104 conn(0x55ed1d5b4220 msgr2=0x55ed1d654120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.666+0000 7f10a8179b80 1 -- 172.21.15.78:0/3172928104 shutdown_connections 2024-03-14T10:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.666+0000 7f10a8179b80 1 -- 172.21.15.78:0/3172928104 wait complete. 2024-03-14T10:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.667+0000 7f10a8179b80 1 -- start start 2024-03-14T10:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.667+0000 7f10a8179b80 1 Processor -- start 2024-03-14T10:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.667+0000 7f10a8179b80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d63e680 0x55ed1d64dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.667+0000 7f10a8179b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63efa0 0x55ed1d64e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.667+0000 7f10a8179b80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d648ec0 0x55ed1d6a95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.667+0000 7f10a8179b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed1d64e940 con 0x55ed1d63efa0 2024-03-14T10:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.667+0000 7f10a8179b80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed1d6a9b00 con 0x55ed1d63e680 2024-03-14T10:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.667+0000 7f10a8179b80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed1d6a9c40 con 0x55ed1d648ec0 2024-03-14T10:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.667+0000 7f10a8179b80 10 client.0 _ll_register_callbacks cb 0x7f1078099350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1091200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63efa0 0x55ed1d64e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1091c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d63e680 0x55ed1d64dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1092600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d648ec0 0x55ed1d6a95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1091200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63efa0 0x55ed1d64e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45802/0 (socket says 172.21.15.78:45802) 2024-03-14T10:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1091c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d63e680 0x55ed1d64dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:57360/0 (socket says 172.21.15.78:57360) 2024-03-14T10:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1091200700 1 -- 172.21.15.78:0/648915029 learned_addr learned my addr 172.21.15.78:0/648915029 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1091200700 1 -- 172.21.15.78:0/648915029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d648ec0 msgr2=0x55ed1d6a95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1091200700 1 --2- 172.21.15.78:0/648915029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d648ec0 0x55ed1d6a95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1091200700 1 -- 172.21.15.78:0/648915029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d63e680 msgr2=0x55ed1d64dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1091200700 1 --2- 172.21.15.78:0/648915029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d63e680 0x55ed1d64dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1091200700 1 -- 172.21.15.78:0/648915029 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed1d3beb60 con 0x55ed1d63efa0 2024-03-14T10:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1092600700 1 --2- 172.21.15.78:0/648915029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d648ec0 0x55ed1d6a95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1091c00700 1 --2- 172.21.15.78:0/648915029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d63e680 0x55ed1d64dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1091200700 1 --2- 172.21.15.78:0/648915029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63efa0 0x55ed1d64e260 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f10780058b0 tx=0x7f1078099210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a4925118747583 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1073400700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1073400700 1 -- 172.21.15.78:0/648915029 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10780b2370 con 0x55ed1d63efa0 2024-03-14T10:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f1073400700 1 -- 172.21.15.78:0/648915029 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f10780a1e50 con 0x55ed1d63efa0 2024-03-14T10:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.668+0000 7f10a8179b80 1 -- 172.21.15.78:0/648915029 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1080004390 con 0x55ed1d63efa0 2024-03-14T10:04:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.669+0000 7f1073400700 1 -- 172.21.15.78:0/648915029 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10780957c0 con 0x55ed1d63efa0 2024-03-14T10:04:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.669+0000 7f10a8179b80 10 client.17700 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.669+0000 7f10a8179b80 1 -- 172.21.15.78:0/648915029 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ed1d416cd0 con 0x55ed1d63efa0 2024-03-14T10:04:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.669+0000 7f1073400700 1 -- 172.21.15.78:0/648915029 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1080004390 con 0x55ed1d63efa0 2024-03-14T10:04:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.669+0000 7f1073400700 1 -- 172.21.15.78:0/648915029 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1078082e60 con 0x55ed1d63efa0 2024-03-14T10:04:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.669+0000 7f10a8179b80 10 client.17700 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.669+0000 7f10a8179b80 10 client.17700 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-14T10:04:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.669+0000 7f10a8179b80 20 client.17700 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.669+0000 7f10a8179b80 1 --2- 172.21.15.78:0/648915029 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55ed1d6aaec0 0x55ed1d6ad2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.669+0000 7f10a8179b80 4 client.17700 mds_command: new command op to 24538 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:04:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.669+0000 7f10a8179b80 1 -- 172.21.15.78:0/648915029 --> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ed1d3dfcf0 con 0x55ed1d6aaec0 2024-03-14T10:04:34.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.669+0000 7f1091c00700 1 --2- 172.21.15.78:0/648915029 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55ed1d6aaec0 0x55ed1d6ad2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.670+0000 7f1091c00700 1 --2- 172.21.15.78:0/648915029 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55ed1d6aaec0 0x55ed1d6ad2b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=587e313327a8f4c2 server_cookie=f1f4b8fbab69f585 in_seq=0 out_seq=0 2024-03-14T10:04:34.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.670+0000 7f1073400700 10 client.17700 ms_handle_connect on v2:172.21.15.78:6838/4227580185 2024-03-14T10:04:34.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.670+0000 7f1073400700 1 -- 172.21.15.78:0/648915029 <== mds.1 v2:172.21.15.78:6838/4227580185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ed1d3dfcf0 con 0x55ed1d6aaec0 2024-03-14T10:04:34.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.670+0000 7f1073400700 10 client.17700 handle_command_reply: tid=0 2024-03-14T10:04:34.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.670+0000 7f10a8179b80 1 client.17700 shutdown 2024-03-14T10:04:34.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.670+0000 7f10a8179b80 1 -- 172.21.15.78:0/648915029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63efa0 msgr2=0x55ed1d64e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.670+0000 7f10a8179b80 1 --2- 172.21.15.78:0/648915029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63efa0 0x55ed1d64e260 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f10780058b0 tx=0x7f1078099210 comp rx=0 tx=0).stop 2024-03-14T10:04:34.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.670+0000 7f10a8179b80 1 -- 172.21.15.78:0/648915029 shutdown_connections 2024-03-14T10:04:34.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.670+0000 7f10a8179b80 1 --2- 172.21.15.78:0/648915029 >> [v2:172.21.15.78:6838/4227580185,v1:172.21.15.78:6839/4227580185] conn(0x55ed1d6aaec0 0x55ed1d6ad2b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.670+0000 7f10a8179b80 1 --2- 172.21.15.78:0/648915029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ed1d63e680 0x55ed1d64dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.670+0000 7f10a8179b80 1 --2- 172.21.15.78:0/648915029 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x55ed1d648ec0 0x55ed1d6a95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.670+0000 7f1073400700 0 client.17700 ms_handle_reset on v2:172.21.15.78:6838/4227580185 2024-03-14T10:04:34.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.670+0000 7f10a8179b80 1 --2- 172.21.15.78:0/648915029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ed1d63efa0 0x55ed1d64e260 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.670+0000 7f10a8179b80 1 -- 172.21.15.78:0/648915029 >> 172.21.15.78:0/648915029 conn(0x55ed1d5b4220 msgr2=0x55ed1d654120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.670+0000 7f10a8179b80 1 -- 172.21.15.78:0/648915029 shutdown_connections 2024-03-14T10:04:34.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.671+0000 7f10a8179b80 1 -- 172.21.15.78:0/648915029 wait complete. 2024-03-14T10:04:34.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.671+0000 7f10a8179b80 20 client.17700 trim_cache size 0 max 16384 2024-03-14T10:04:34.677 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:04:34.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.671+0000 7f1094600700 1 -- 172.21.15.78:0/495310088 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f10740a0c00 msgr2=0x7f10740a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.671+0000 7f1094600700 1 --2- 172.21.15.78:0/495310088 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f10740a0c00 0x7f10740a30c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f108400b780 tx=0x7f10840095c0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.671+0000 7f1094600700 1 -- 172.21.15.78:0/495310088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108c106040 msgr2=0x7f108c19fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.671+0000 7f1094600700 1 --2- 172.21.15.78:0/495310088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108c106040 0x7f108c19fb60 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f107800e970 tx=0x7f107800ec80 comp rx=0 tx=0).stop 2024-03-14T10:04:34.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.671+0000 7f1094600700 1 -- 172.21.15.78:0/495310088 shutdown_connections 2024-03-14T10:04:34.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.671+0000 7f1094600700 1 --2- 172.21.15.78:0/495310088 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f10740a0c00 0x7f10740a30c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.671+0000 7f1094600700 1 --2- 172.21.15.78:0/495310088 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f108c100f20 0x7f108c19f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.671+0000 7f1094600700 1 --2- 172.21.15.78:0/495310088 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f108c199750 0x7f108c199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.671+0000 7f1094600700 1 --2- 172.21.15.78:0/495310088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f108c106040 0x7f108c19fb60 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.671+0000 7f1094600700 1 -- 172.21.15.78:0/495310088 >> 172.21.15.78:0/495310088 conn(0x7f108c0fc7a0 msgr2=0x7f108c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.671+0000 7f1094600700 1 -- 172.21.15.78:0/495310088 shutdown_connections 2024-03-14T10:04:34.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.671+0000 7f1094600700 1 -- 172.21.15.78:0/495310088 wait complete. 2024-03-14T10:04:34.690 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-14T10:04:34.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.883+0000 7fbb5f600700 1 -- 172.21.15.78:0/245350615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb58100f20 msgr2=0x7fbb58101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.883+0000 7fbb5f600700 1 --2- 172.21.15.78:0/245350615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb58100f20 0x7fbb58101300 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c009990 tx=0x7fbb4c009ca0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.883+0000 7fbb5f600700 1 -- 172.21.15.78:0/245350615 shutdown_connections 2024-03-14T10:04:34.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.883+0000 7fbb5f600700 1 --2- 172.21.15.78:0/245350615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb58106040 0x7fbb5810dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.883+0000 7fbb5f600700 1 --2- 172.21.15.78:0/245350615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb58100f20 0x7fbb58101300 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.883+0000 7fbb5f600700 1 --2- 172.21.15.78:0/245350615 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb581018d0 0x7fbb58105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.883+0000 7fbb5f600700 1 -- 172.21.15.78:0/245350615 >> 172.21.15.78:0/245350615 conn(0x7fbb580fc7a0 msgr2=0x7fbb580febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.883+0000 7fbb5f600700 1 -- 172.21.15.78:0/245350615 shutdown_connections 2024-03-14T10:04:34.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.884+0000 7fbb5f600700 1 -- 172.21.15.78:0/245350615 wait complete. 2024-03-14T10:04:34.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.884+0000 7fbb5f600700 1 Processor -- start 2024-03-14T10:04:34.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.885+0000 7fbb5f600700 1 -- start start 2024-03-14T10:04:34.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.885+0000 7fbb5f600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb58100f20 0x7fbb5819f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.885+0000 7fbb5f600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb58106040 0x7fbb5819faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.885+0000 7fbb5f600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb581996e0 0x7fbb58199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.885+0000 7fbb5f600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5819a160 con 0x7fbb581996e0 2024-03-14T10:04:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.885+0000 7fbb5f600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5819a2a0 con 0x7fbb58106040 2024-03-14T10:04:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.885+0000 7fbb5f600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5819a460 con 0x7fbb58100f20 2024-03-14T10:04:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.885+0000 7fbb5d600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb581996e0 0x7fbb58199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.885+0000 7fbb5d600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb581996e0 0x7fbb58199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45826/0 (socket says 172.21.15.78:45826) 2024-03-14T10:04:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.885+0000 7fbb5d600700 1 -- 172.21.15.78:0/2022438538 learned_addr learned my addr 172.21.15.78:0/2022438538 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.885+0000 7fbb5cc00700 1 --2- 172.21.15.78:0/2022438538 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb58100f20 0x7fbb5819f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.885+0000 7fbb57e00700 1 --2- 172.21.15.78:0/2022438538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb58106040 0x7fbb5819faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.886+0000 7fbb5d600700 1 -- 172.21.15.78:0/2022438538 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb58100f20 msgr2=0x7fbb5819f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.886+0000 7fbb5d600700 1 --2- 172.21.15.78:0/2022438538 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb58100f20 0x7fbb5819f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.886+0000 7fbb5d600700 1 -- 172.21.15.78:0/2022438538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb58106040 msgr2=0x7fbb5819faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.886+0000 7fbb5d600700 1 --2- 172.21.15.78:0/2022438538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb58106040 0x7fbb5819faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.886+0000 7fbb5d600700 1 -- 172.21.15.78:0/2022438538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4c0095f0 con 0x7fbb581996e0 2024-03-14T10:04:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.886+0000 7fbb57e00700 1 --2- 172.21.15.78:0/2022438538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb58106040 0x7fbb5819faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:04:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.886+0000 7fbb5d600700 1 --2- 172.21.15.78:0/2022438538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb581996e0 0x7fbb58199b90 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fbb5000ef30 tx=0x7fbb5000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.886+0000 7fbb55600700 1 -- 172.21.15.78:0/2022438538 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb50018070 con 0x7fbb581996e0 2024-03-14T10:04:34.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.886+0000 7fbb55600700 1 -- 172.21.15.78:0/2022438538 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb5000f040 con 0x7fbb581996e0 2024-03-14T10:04:34.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.886+0000 7fbb55600700 1 -- 172.21.15.78:0/2022438538 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5000cdb0 con 0x7fbb581996e0 2024-03-14T10:04:34.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.886+0000 7fbb5f600700 1 -- 172.21.15.78:0/2022438538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb581a6420 con 0x7fbb581996e0 2024-03-14T10:04:34.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.886+0000 7fbb5f600700 1 -- 172.21.15.78:0/2022438538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb581a6930 con 0x7fbb581996e0 2024-03-14T10:04:34.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.889+0000 7fbb55600700 1 -- 172.21.15.78:0/2022438538 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fbb50003a00 con 0x7fbb581996e0 2024-03-14T10:04:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.890+0000 7fbb55600700 1 --2- 172.21.15.78:0/2022438538 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbb400a0c80 0x7fbb400a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.890+0000 7fbb7307fb80 1 Processor -- start 2024-03-14T10:04:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.890+0000 7fbb55600700 1 -- 172.21.15.78:0/2022438538 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fbb500cc360 con 0x7fbb581996e0 2024-03-14T10:04:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.891+0000 7fbb5cc00700 1 --2- 172.21.15.78:0/2022438538 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbb400a0c80 0x7fbb400a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.891+0000 7fbb7307fb80 1 -- start start 2024-03-14T10:04:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.891+0000 7fbb7307fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611eddd9680 0x5611eddd9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.891+0000 7fbb7307fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9fa0 0x5611edde3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.891+0000 7fbb7307fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611edde3ec0 0x5611edde62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.891+0000 7fbb7307fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611edd32990 con 0x5611edde3ec0 2024-03-14T10:04:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.891+0000 7fbb7307fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611edd30350 con 0x5611eddd9fa0 2024-03-14T10:04:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.891+0000 7fbb7307fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611edd2bc40 con 0x5611eddd9680 2024-03-14T10:04:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.891+0000 7fbb5cc00700 1 --2- 172.21.15.78:0/2022438538 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbb400a0c80 0x7fbb400a3140 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c006010 tx=0x7fbb4c00b260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.891+0000 7fbb5d600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611edde3ec0 0x5611edde62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.891+0000 7fbb5d600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611edde3ec0 0x5611edde62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45842/0 (socket says 172.21.15.78:45842) 2024-03-14T10:04:34.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.892+0000 7fbb5cc00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611eddd9680 0x5611eddd9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.892+0000 7fbb5d600700 1 -- 172.21.15.78:0/2890269730 learned_addr learned my addr 172.21.15.78:0/2890269730 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:34.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.892+0000 7fbb57e00700 1 --2- 172.21.15.78:0/2890269730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9fa0 0x5611edde3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.892+0000 7fbb5d600700 1 -- 172.21.15.78:0/2890269730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611eddd9680 msgr2=0x5611eddd9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.892+0000 7fbb5d600700 1 --2- 172.21.15.78:0/2890269730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611eddd9680 0x5611eddd9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.892+0000 7fbb5d600700 1 -- 172.21.15.78:0/2890269730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9fa0 msgr2=0x5611edde3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.892+0000 7fbb5d600700 1 --2- 172.21.15.78:0/2890269730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9fa0 0x5611edde3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.892+0000 7fbb5d600700 1 -- 172.21.15.78:0/2890269730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611edbad3b0 con 0x5611edde3ec0 2024-03-14T10:04:34.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.892+0000 7fbb5d600700 1 --2- 172.21.15.78:0/2890269730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611edde3ec0 0x5611edde62b0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbb50095d80 tx=0x7fbb50095050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d9f9f25de952515 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.892+0000 7fbb49c00700 1 -- 172.21.15.78:0/2890269730 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb50098c70 con 0x5611edde3ec0 2024-03-14T10:04:34.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.892+0000 7fbb49c00700 1 -- 172.21.15.78:0/2890269730 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb5000f3f0 con 0x5611edde3ec0 2024-03-14T10:04:34.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.892+0000 7fbb49c00700 1 -- 172.21.15.78:0/2890269730 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5000f5e0 con 0x5611edde3ec0 2024-03-14T10:04:34.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.893+0000 7fbb7307fb80 1 -- 172.21.15.78:0/2890269730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611edde3ec0 msgr2=0x5611edde62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.893+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/2890269730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611edde3ec0 0x5611edde62b0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbb50095d80 tx=0x7fbb50095050 comp rx=0 tx=0).stop 2024-03-14T10:04:34.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.893+0000 7fbb7307fb80 1 -- 172.21.15.78:0/2890269730 shutdown_connections 2024-03-14T10:04:34.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.893+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/2890269730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611edde3ec0 0x5611edde62b0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.893+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/2890269730 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611eddd9680 0x5611eddd9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.893+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/2890269730 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9fa0 0x5611edde3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.893+0000 7fbb7307fb80 1 -- 172.21.15.78:0/2890269730 >> 172.21.15.78:0/2890269730 conn(0x5611edd4f220 msgr2=0x5611eddd4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.893+0000 7fbb7307fb80 1 -- 172.21.15.78:0/2890269730 shutdown_connections 2024-03-14T10:04:34.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.893+0000 7fbb7307fb80 1 -- 172.21.15.78:0/2890269730 wait complete. 2024-03-14T10:04:34.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.894+0000 7fbb7307fb80 1 -- start start 2024-03-14T10:04:34.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.894+0000 7fbb7307fb80 1 Processor -- start 2024-03-14T10:04:34.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.894+0000 7fbb7307fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9680 0x5611edde3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.894+0000 7fbb7307fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611eddd9fa0 0x5611edde0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.894+0000 7fbb7307fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611edde3ec0 0x5611edde1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.894+0000 7fbb7307fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611edcc7d80 con 0x5611eddd9fa0 2024-03-14T10:04:34.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.894+0000 7fbb7307fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611edcc7a20 con 0x5611eddd9680 2024-03-14T10:04:34.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.894+0000 7fbb7307fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611edbaf6a0 con 0x5611edde3ec0 2024-03-14T10:04:34.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.894+0000 7fbb7307fb80 10 client.0 _ll_register_callbacks cb 0x5611edb333a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:34.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.895+0000 7fbb57e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611eddd9fa0 0x5611edde0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.895+0000 7fbb57e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611eddd9fa0 0x5611edde0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45858/0 (socket says 172.21.15.78:45858) 2024-03-14T10:04:34.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.895+0000 7fbb57e00700 1 -- 172.21.15.78:0/2792557097 learned_addr learned my addr 172.21.15.78:0/2792557097 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:34.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.895+0000 7fbb5cc00700 1 --2- 172.21.15.78:0/2792557097 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9680 0x5611edde3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.895+0000 7fbb5d600700 1 --2- 172.21.15.78:0/2792557097 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611edde3ec0 0x5611edde1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.895+0000 7fbb57e00700 1 -- 172.21.15.78:0/2792557097 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611edde3ec0 msgr2=0x5611edde1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.895+0000 7fbb57e00700 1 --2- 172.21.15.78:0/2792557097 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611edde3ec0 0x5611edde1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.895+0000 7fbb57e00700 1 -- 172.21.15.78:0/2792557097 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9680 msgr2=0x5611edde3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.895+0000 7fbb57e00700 1 --2- 172.21.15.78:0/2792557097 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9680 0x5611edde3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.895+0000 7fbb57e00700 1 -- 172.21.15.78:0/2792557097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb581a6930 con 0x5611eddd9fa0 2024-03-14T10:04:34.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.895+0000 7fbb57e00700 1 --2- 172.21.15.78:0/2792557097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611eddd9fa0 0x5611edde0b20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbb44012840 tx=0x7fbb44012b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9209b2722d204c4b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.895+0000 7fbb3e000700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-03-14T10:04:34.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.895+0000 7fbb3e000700 1 -- 172.21.15.78:0/2792557097 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44017070 con 0x5611eddd9fa0 2024-03-14T10:04:34.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.895+0000 7fbb7307fb80 1 -- 172.21.15.78:0/2792557097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5611eddcc720 con 0x5611eddd9fa0 2024-03-14T10:04:34.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.895+0000 7fbb3e000700 1 -- 172.21.15.78:0/2792557097 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fbb44004260 con 0x5611eddd9fa0 2024-03-14T10:04:34.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.896+0000 7fbb3e000700 1 -- 172.21.15.78:0/2792557097 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb44004420 con 0x5611eddd9fa0 2024-03-14T10:04:34.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.896+0000 7fbb7307fb80 10 client.17712 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:34.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.896+0000 7fbb7307fb80 1 -- 172.21.15.78:0/2792557097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5611edb59b60 con 0x5611eddd9fa0 2024-03-14T10:04:34.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.896+0000 7fbb3e000700 1 -- 172.21.15.78:0/2792557097 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5611eddcc720 con 0x5611eddd9fa0 2024-03-14T10:04:34.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.896+0000 7fbb3e000700 1 -- 172.21.15.78:0/2792557097 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbb440118b0 con 0x5611eddd9fa0 2024-03-14T10:04:34.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.896+0000 7fbb7307fb80 10 client.17712 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:34.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.896+0000 7fbb7307fb80 10 client.17712 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:04:34.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.896+0000 7fbb7307fb80 20 client.17712 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:34.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.896+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/2792557097 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5611eddf3900 0x5611eddf5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.896+0000 7fbb7307fb80 4 client.17712 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2024-03-14T10:04:34.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.896+0000 7fbb7307fb80 1 -- 172.21.15.78:0/2792557097 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5611edd24700 con 0x5611eddf3900 2024-03-14T10:04:34.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.897+0000 7fbb5cc00700 1 --2- 172.21.15.78:0/2792557097 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5611eddf3900 0x5611eddf5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.897+0000 7fbb5cc00700 1 --2- 172.21.15.78:0/2792557097 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5611eddf3900 0x5611eddf5d20 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5f41a6622152f06 server_cookie=943325940305fc57 in_seq=0 out_seq=0 2024-03-14T10:04:34.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.897+0000 7fbb3e000700 10 client.17712 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:34.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.898+0000 7fbb3e000700 1 -- 172.21.15.78:0/2792557097 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5611edd24700 con 0x5611eddf3900 2024-03-14T10:04:34.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.898+0000 7fbb3e000700 10 client.17712 handle_command_reply: tid=0 2024-03-14T10:04:34.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.898+0000 7fbb7307fb80 1 client.17712 shutdown 2024-03-14T10:04:34.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.898+0000 7fbb7307fb80 1 -- 172.21.15.78:0/2792557097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611eddd9fa0 msgr2=0x5611edde0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.898+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/2792557097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611eddd9fa0 0x5611edde0b20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbb44012840 tx=0x7fbb44012b50 comp rx=0 tx=0).stop 2024-03-14T10:04:34.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.898+0000 7fbb7307fb80 1 -- 172.21.15.78:0/2792557097 shutdown_connections 2024-03-14T10:04:34.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.898+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/2792557097 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5611eddf3900 0x5611eddf5d20 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.898+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/2792557097 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9680 0x5611edde3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.898+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/2792557097 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611edde3ec0 0x5611edde1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.898+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/2792557097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611eddd9fa0 0x5611edde0b20 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.898+0000 7fbb3e000700 0 client.17712 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:34.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.898+0000 7fbb7307fb80 1 -- 172.21.15.78:0/2792557097 >> 172.21.15.78:0/2792557097 conn(0x5611edd4f220 msgr2=0x5611eddd4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.899+0000 7fbb7307fb80 1 -- 172.21.15.78:0/2792557097 shutdown_connections 2024-03-14T10:04:34.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.899+0000 7fbb7307fb80 1 -- 172.21.15.78:0/2792557097 wait complete. 2024-03-14T10:04:34.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.899+0000 7fbb7307fb80 20 client.17712 trim_cache size 0 max 16384 2024-03-14T10:04:34.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.912+0000 7fbb7307fb80 1 Processor -- start 2024-03-14T10:04:34.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.912+0000 7fbb7307fb80 1 -- start start 2024-03-14T10:04:34.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.912+0000 7fbb7307fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9680 0x5611eddd4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.912+0000 7fbb7307fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611eddd9fa0 0x5611eddeaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.912+0000 7fbb7307fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611edde3ec0 0x5611eddddd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.912+0000 7fbb7307fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611edddb9b0 con 0x5611eddd9fa0 2024-03-14T10:04:34.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.912+0000 7fbb7307fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611eddfaa40 con 0x5611eddd9680 2024-03-14T10:04:34.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.912+0000 7fbb7307fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611edddf070 con 0x5611edde3ec0 2024-03-14T10:04:34.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.912+0000 7fbb57e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611eddd9fa0 0x5611eddeaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.912+0000 7fbb5d600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611edde3ec0 0x5611eddddd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.912+0000 7fbb57e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611eddd9fa0 0x5611eddeaa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45870/0 (socket says 172.21.15.78:45870) 2024-03-14T10:04:34.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.912+0000 7fbb57e00700 1 -- 172.21.15.78:0/3634899177 learned_addr learned my addr 172.21.15.78:0/3634899177 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:34.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.912+0000 7fbb5cc00700 1 --2- 172.21.15.78:0/3634899177 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9680 0x5611eddd4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.913+0000 7fbb5d600700 1 -- 172.21.15.78:0/3634899177 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9680 msgr2=0x5611eddd4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.913+0000 7fbb5d600700 1 --2- 172.21.15.78:0/3634899177 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9680 0x5611eddd4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.913+0000 7fbb5d600700 1 -- 172.21.15.78:0/3634899177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611eddd9fa0 msgr2=0x5611eddeaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.913+0000 7fbb5d600700 1 --2- 172.21.15.78:0/3634899177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611eddd9fa0 0x5611eddeaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.913+0000 7fbb5d600700 1 -- 172.21.15.78:0/3634899177 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611edcf1ea0 con 0x5611edde3ec0 2024-03-14T10:04:34.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.913+0000 7fbb5d600700 1 --2- 172.21.15.78:0/3634899177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611edde3ec0 0x5611eddddd80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbb50095020 tx=0x7fbb50004dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99023b605f0c012 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.913+0000 7fbb4a600700 1 -- 172.21.15.78:0/3634899177 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb500b5070 con 0x5611edde3ec0 2024-03-14T10:04:34.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.913+0000 7fbb4a600700 1 -- 172.21.15.78:0/3634899177 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb50098c70 con 0x5611edde3ec0 2024-03-14T10:04:34.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.913+0000 7fbb4a600700 1 -- 172.21.15.78:0/3634899177 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb500a1700 con 0x5611edde3ec0 2024-03-14T10:04:34.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.914+0000 7fbb7307fb80 1 -- 172.21.15.78:0/3634899177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611edde3ec0 msgr2=0x5611eddddd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.914+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/3634899177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611edde3ec0 0x5611eddddd80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbb50095020 tx=0x7fbb50004dc0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.914+0000 7fbb7307fb80 1 -- 172.21.15.78:0/3634899177 shutdown_connections 2024-03-14T10:04:34.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.914+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/3634899177 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9680 0x5611eddd4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.914+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/3634899177 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611edde3ec0 0x5611eddddd80 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.914+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/3634899177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611eddd9fa0 0x5611eddeaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.914+0000 7fbb7307fb80 1 -- 172.21.15.78:0/3634899177 >> 172.21.15.78:0/3634899177 conn(0x5611edd4f220 msgr2=0x5611edddf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.914+0000 7fbb7307fb80 1 -- 172.21.15.78:0/3634899177 shutdown_connections 2024-03-14T10:04:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.914+0000 7fbb7307fb80 1 -- 172.21.15.78:0/3634899177 wait complete. 2024-03-14T10:04:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.914+0000 7fbb7307fb80 1 -- start start 2024-03-14T10:04:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.915+0000 7fbb7307fb80 1 Processor -- start 2024-03-14T10:04:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.915+0000 7fbb7307fb80 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9680 0x5611eddea4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.915+0000 7fbb7307fb80 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611eddd9fa0 0x5611eddeaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.915+0000 7fbb7307fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611edde3ec0 0x5611edde8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.915+0000 7fbb7307fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611edde9260 con 0x5611edde3ec0 2024-03-14T10:04:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.915+0000 7fbb7307fb80 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611edde93d0 con 0x5611eddd9680 2024-03-14T10:04:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.915+0000 7fbb7307fb80 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611edde9590 con 0x5611eddd9fa0 2024-03-14T10:04:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.915+0000 7fbb7307fb80 10 client.0 _ll_register_callbacks cb 0x7fbb44009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-14T10:04:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.915+0000 7fbb5d600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611edde3ec0 0x5611edde8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.915+0000 7fbb57e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611eddd9fa0 0x5611eddeaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.915+0000 7fbb5cc00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9680 0x5611eddea4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.915+0000 7fbb57e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611eddd9fa0 0x5611eddeaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:48954/0 (socket says 172.21.15.78:48954) 2024-03-14T10:04:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.915+0000 7fbb57e00700 1 -- 172.21.15.78:0/965221484 learned_addr learned my addr 172.21.15.78:0/965221484 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.916+0000 7fbb5cc00700 1 -- 172.21.15.78:0/965221484 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611eddd9fa0 msgr2=0x5611eddeaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.916+0000 7fbb5cc00700 1 --2- 172.21.15.78:0/965221484 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611eddd9fa0 0x5611eddeaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.916+0000 7fbb5cc00700 1 -- 172.21.15.78:0/965221484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611edde3ec0 msgr2=0x5611edde8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-14T10:04:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.916+0000 7fbb5cc00700 1 --2- 172.21.15.78:0/965221484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611edde3ec0 0x5611edde8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.916+0000 7fbb5cc00700 1 -- 172.21.15.78:0/965221484 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5611edbad3b0 con 0x5611eddd9680 2024-03-14T10:04:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.916+0000 7fbb5cc00700 1 --2- 172.21.15.78:0/965221484 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9680 0x5611eddea4e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00b690 tx=0x7fbb4c009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bc5a7404bb9cf4a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.916+0000 7fbb3e000700 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-03-14T10:04:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.916+0000 7fbb3e000700 1 -- 172.21.15.78:0/965221484 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c034900 con 0x5611eddd9680 2024-03-14T10:04:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.916+0000 7fbb3e000700 1 -- 172.21.15.78:0/965221484 <== mon.2 v2:172.21.15.132:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fbb4c034a60 con 0x5611eddd9680 2024-03-14T10:04:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.916+0000 7fbb7307fb80 1 -- 172.21.15.78:0/965221484 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb5000f3f0 con 0x5611eddd9680 2024-03-14T10:04:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.916+0000 7fbb3e000700 1 -- 172.21.15.78:0/965221484 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb4c034c20 con 0x5611eddd9680 2024-03-14T10:04:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.916+0000 7fbb7307fb80 10 client.27779 fetch_fsmap learned FSMap version 25 2024-03-14T10:04:34.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.916+0000 7fbb7307fb80 1 -- 172.21.15.78:0/965221484 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5611edbb1cd0 con 0x5611eddd9680 2024-03-14T10:04:34.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.917+0000 7fbb3e000700 1 -- 172.21.15.78:0/965221484 <== mon.2 v2:172.21.15.132:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb5000f3f0 con 0x5611eddd9680 2024-03-14T10:04:34.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.917+0000 7fbb3e000700 1 -- 172.21.15.78:0/965221484 <== mon.2 v2:172.21.15.132:3300/0 5 ==== fsmap(e 25) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbb4c00f460 con 0x5611eddd9680 2024-03-14T10:04:34.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.917+0000 7fbb7307fb80 10 client.27779 fetch_fsmap finished waiting for FSMap version 25 2024-03-14T10:04:34.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.917+0000 7fbb7307fb80 10 client.27779 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-14T10:04:34.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.917+0000 7fbb7307fb80 20 client.27779 populate_metadata read hostname 'smithi078' 2024-03-14T10:04:34.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.917+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/965221484 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5611ede467a0 0x5611ede48bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:34.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.917+0000 7fbb7307fb80 4 client.27779 mds_command: new command op to 24572 tid=0[{"prefix": "damage ls"}] 2024-03-14T10:04:34.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.917+0000 7fbb7307fb80 1 -- 172.21.15.78:0/965221484 --> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5611edb7acf0 con 0x5611ede467a0 2024-03-14T10:04:34.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.917+0000 7fbb57e00700 1 --2- 172.21.15.78:0/965221484 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5611ede467a0 0x5611ede48bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:34.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.918+0000 7fbb57e00700 1 --2- 172.21.15.78:0/965221484 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5611ede467a0 0x5611ede48bc0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=435237d688583328 server_cookie=5be42cdad2ea6d29 in_seq=0 out_seq=0 2024-03-14T10:04:34.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.918+0000 7fbb3e000700 10 client.27779 ms_handle_connect on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:34.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.918+0000 7fbb3e000700 1 -- 172.21.15.78:0/965221484 <== mds.0 v2:172.21.15.132:6838/2192113559 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5611edb7acf0 con 0x5611ede467a0 2024-03-14T10:04:34.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.918+0000 7fbb3e000700 10 client.27779 handle_command_reply: tid=0 2024-03-14T10:04:34.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.918+0000 7fbb7307fb80 1 client.27779 shutdown 2024-03-14T10:04:34.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.918+0000 7fbb7307fb80 1 -- 172.21.15.78:0/965221484 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9680 msgr2=0x5611eddea4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.918+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/965221484 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9680 0x5611eddea4e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00b690 tx=0x7fbb4c009ca0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.918+0000 7fbb7307fb80 1 -- 172.21.15.78:0/965221484 shutdown_connections 2024-03-14T10:04:34.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.918+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/965221484 >> [v2:172.21.15.132:6838/2192113559,v1:172.21.15.132:6839/2192113559] conn(0x5611ede467a0 0x5611ede48bc0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.918+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/965221484 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5611eddd9680 0x5611eddea4e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.918+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/965221484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5611edde3ec0 0x5611edde8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.918+0000 7fbb7307fb80 1 --2- 172.21.15.78:0/965221484 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x5611eddd9fa0 0x5611eddeaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.918+0000 7fbb3e000700 0 client.27779 ms_handle_reset on v2:172.21.15.132:6838/2192113559 2024-03-14T10:04:34.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.918+0000 7fbb7307fb80 1 -- 172.21.15.78:0/965221484 >> 172.21.15.78:0/965221484 conn(0x5611edd4f220 msgr2=0x5611eddd8560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.918+0000 7fbb7307fb80 1 -- 172.21.15.78:0/965221484 shutdown_connections 2024-03-14T10:04:34.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.918+0000 7fbb7307fb80 1 -- 172.21.15.78:0/965221484 wait complete. 2024-03-14T10:04:34.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.919+0000 7fbb7307fb80 20 client.27779 trim_cache size 0 max 16384 2024-03-14T10:04:34.924 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2024-03-14T10:04:34.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.919+0000 7fbb5f600700 1 -- 172.21.15.78:0/2022438538 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbb400a0c80 msgr2=0x7fbb400a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.919+0000 7fbb5f600700 1 --2- 172.21.15.78:0/2022438538 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbb400a0c80 0x7fbb400a3140 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c006010 tx=0x7fbb4c00b260 comp rx=0 tx=0).stop 2024-03-14T10:04:34.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.919+0000 7fbb5f600700 1 -- 172.21.15.78:0/2022438538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb581996e0 msgr2=0x7fbb58199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:34.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.919+0000 7fbb5f600700 1 --2- 172.21.15.78:0/2022438538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb581996e0 0x7fbb58199b90 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fbb5000ef30 tx=0x7fbb5000c3a0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.919+0000 7fbb5f600700 1 -- 172.21.15.78:0/2022438538 shutdown_connections 2024-03-14T10:04:34.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.919+0000 7fbb5f600700 1 --2- 172.21.15.78:0/2022438538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb581996e0 0x7fbb58199b90 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.919+0000 7fbb5f600700 1 --2- 172.21.15.78:0/2022438538 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fbb58100f20 0x7fbb5819f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.919+0000 7fbb5f600700 1 --2- 172.21.15.78:0/2022438538 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fbb400a0c80 0x7fbb400a3140 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.919+0000 7fbb5f600700 1 --2- 172.21.15.78:0/2022438538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb58106040 0x7fbb5819faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:34.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.919+0000 7fbb5f600700 1 -- 172.21.15.78:0/2022438538 >> 172.21.15.78:0/2022438538 conn(0x7fbb580fc7a0 msgr2=0x7fbb580feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:34.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.919+0000 7fbb5f600700 1 -- 172.21.15.78:0/2022438538 shutdown_connections 2024-03-14T10:04:34.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:34.919+0000 7fbb5f600700 1 -- 172.21.15.78:0/2022438538 wait complete. 2024-03-14T10:04:35.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:35 smithi152 ceph-mon[48279]: pgmap v323: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:35.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:35 smithi132 ceph-mon[47959]: pgmap v323: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:35.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:35 smithi078 ceph-mon[38143]: pgmap v323: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:35.939 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-14T10:04:35.940 INFO:tasks.fwd_scrub:done joining 2024-03-14T10:04:35.940 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-14T10:04:35.952 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T10:04:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.163+0000 7f1f68a00700 1 -- 172.21.15.78:0/3672294418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f64100e30 msgr2=0x7f1f64101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.163+0000 7f1f68a00700 1 --2- 172.21.15.78:0/3672294418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f64100e30 0x7f1f64101210 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f1f54009990 tx=0x7f1f54009ca0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.163+0000 7f1f68a00700 1 -- 172.21.15.78:0/3672294418 shutdown_connections 2024-03-14T10:04:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.163+0000 7f1f68a00700 1 --2- 172.21.15.78:0/3672294418 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f64105f50 0x7f1f6410dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.163+0000 7f1f68a00700 1 --2- 172.21.15.78:0/3672294418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f64100e30 0x7f1f64101210 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.163+0000 7f1f68a00700 1 --2- 172.21.15.78:0/3672294418 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f641017e0 0x7f1f64105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.163+0000 7f1f68a00700 1 -- 172.21.15.78:0/3672294418 >> 172.21.15.78:0/3672294418 conn(0x7f1f640fc730 msgr2=0x7f1f640feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.163+0000 7f1f68a00700 1 -- 172.21.15.78:0/3672294418 shutdown_connections 2024-03-14T10:04:36.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.163+0000 7f1f68a00700 1 -- 172.21.15.78:0/3672294418 wait complete. 2024-03-14T10:04:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.164+0000 7f1f68a00700 1 Processor -- start 2024-03-14T10:04:36.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.164+0000 7f1f68a00700 1 -- start start 2024-03-14T10:04:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.164+0000 7f1f68a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f64100e30 0x7f1f640730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.165+0000 7f1f68a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f64105f50 0x7f1f640735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.165+0000 7f1f68a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f64073b30 0x7f1f641a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.165+0000 7f1f68a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f641a5db0 con 0x7f1f64105f50 2024-03-14T10:04:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.165+0000 7f1f68a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f641a5ef0 con 0x7f1f64073b30 2024-03-14T10:04:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.165+0000 7f1f68a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f641a60b0 con 0x7f1f64100e30 2024-03-14T10:04:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.165+0000 7f1f61400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f64105f50 0x7f1f640735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.165+0000 7f1f61400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f64105f50 0x7f1f640735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45904/0 (socket says 172.21.15.78:45904) 2024-03-14T10:04:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.165+0000 7f1f61400700 1 -- 172.21.15.78:0/1328593966 learned_addr learned my addr 172.21.15.78:0/1328593966 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.165+0000 7f1f62800700 1 --2- 172.21.15.78:0/1328593966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f64073b30 0x7f1f641a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.165+0000 7f1f61e00700 1 --2- 172.21.15.78:0/1328593966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f64100e30 0x7f1f640730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.165+0000 7f1f61400700 1 -- 172.21.15.78:0/1328593966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f64100e30 msgr2=0x7f1f640730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.165+0000 7f1f61400700 1 --2- 172.21.15.78:0/1328593966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f64100e30 0x7f1f640730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.165+0000 7f1f61400700 1 -- 172.21.15.78:0/1328593966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f64073b30 msgr2=0x7f1f641a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:36.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.165+0000 7f1f61400700 1 --2- 172.21.15.78:0/1328593966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f64073b30 0x7f1f641a57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.165+0000 7f1f61400700 1 -- 172.21.15.78:0/1328593966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f540095f0 con 0x7f1f64105f50 2024-03-14T10:04:36.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.165+0000 7f1f61400700 1 --2- 172.21.15.78:0/1328593966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f64105f50 0x7f1f640735f0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000e970 tx=0x7f1f5000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:36.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.166+0000 7f1f5aa00700 1 -- 172.21.15.78:0/1328593966 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f50009790 con 0x7f1f64105f50 2024-03-14T10:04:36.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.166+0000 7f1f5aa00700 1 -- 172.21.15.78:0/1328593966 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f5000c7e0 con 0x7f1f64105f50 2024-03-14T10:04:36.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.166+0000 7f1f5aa00700 1 -- 172.21.15.78:0/1328593966 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f500054e0 con 0x7f1f64105f50 2024-03-14T10:04:36.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.166+0000 7f1f68a00700 1 -- 172.21.15.78:0/1328593966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f641a6390 con 0x7f1f64105f50 2024-03-14T10:04:36.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.166+0000 7f1f68a00700 1 -- 172.21.15.78:0/1328593966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f641a6850 con 0x7f1f64105f50 2024-03-14T10:04:36.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.167+0000 7f1f68a00700 1 -- 172.21.15.78:0/1328593966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f641091c0 con 0x7f1f64105f50 2024-03-14T10:04:36.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.169+0000 7f1f5aa00700 1 -- 172.21.15.78:0/1328593966 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f1f5000c950 con 0x7f1f64105f50 2024-03-14T10:04:36.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.170+0000 7f1f5aa00700 1 --2- 172.21.15.78:0/1328593966 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1f480a0c80 0x7f1f480a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:36.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.170+0000 7f1f5aa00700 1 -- 172.21.15.78:0/1328593966 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f1f50014070 con 0x7f1f64105f50 2024-03-14T10:04:36.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.170+0000 7f1f61e00700 1 --2- 172.21.15.78:0/1328593966 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1f480a0c80 0x7f1f480a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:36.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.171+0000 7f1f61e00700 1 --2- 172.21.15.78:0/1328593966 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1f480a0c80 0x7f1f480a3140 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400b2e0 tx=0x7f1f54005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:36.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.174+0000 7f1f5aa00700 1 -- 172.21.15.78:0/1328593966 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1f50095ce0 con 0x7f1f64105f50 2024-03-14T10:04:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.396+0000 7f1f68a00700 1 -- 172.21.15.78:0/1328593966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f64102300 con 0x7f1f64105f50 2024-03-14T10:04:36.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.398+0000 7f1f5aa00700 1 -- 172.21.15.78:0/1328593966 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+25214 (secure 0 0 0) 0x7f1f50099300 con 0x7f1f64105f50 2024-03-14T10:04:36.399 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T10:04:36.399 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":96,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T10:04:17.907246+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1935098096":"2024-03-15T09:54:09.924780+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6801/2580616009":"2024-03-15T09:54:09.924780+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/3308792589":"2024-03-15T09:54:09.924780+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/555376800":"2024-03-15T09:54:09.924780+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/3220727810":"2024-03-15T09:54:09.924780+0000","172.21.15.78:0/2934089773":"2024-03-15T09:54:09.924780+0000","172.21.15.78:6800/2580616009":"2024-03-15T09:54:09.924780+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T10:04:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.400+0000 7f1f68a00700 1 -- 172.21.15.78:0/1328593966 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1f480a0c80 msgr2=0x7f1f480a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:36.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.400+0000 7f1f68a00700 1 --2- 172.21.15.78:0/1328593966 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1f480a0c80 0x7f1f480a3140 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400b2e0 tx=0x7f1f54005fd0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.400+0000 7f1f68a00700 1 -- 172.21.15.78:0/1328593966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f64105f50 msgr2=0x7f1f640735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.400+0000 7f1f68a00700 1 --2- 172.21.15.78:0/1328593966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f64105f50 0x7f1f640735f0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000e970 tx=0x7f1f5000ec80 comp rx=0 tx=0).stop 2024-03-14T10:04:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.400+0000 7f1f68a00700 1 -- 172.21.15.78:0/1328593966 shutdown_connections 2024-03-14T10:04:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.400+0000 7f1f68a00700 1 --2- 172.21.15.78:0/1328593966 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1f480a0c80 0x7f1f480a3140 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.400+0000 7f1f68a00700 1 --2- 172.21.15.78:0/1328593966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f64073b30 0x7f1f641a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.400+0000 7f1f68a00700 1 --2- 172.21.15.78:0/1328593966 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1f64100e30 0x7f1f640730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.400+0000 7f1f68a00700 1 --2- 172.21.15.78:0/1328593966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f64105f50 0x7f1f640735f0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.400+0000 7f1f68a00700 1 -- 172.21.15.78:0/1328593966 >> 172.21.15.78:0/1328593966 conn(0x7f1f640fc730 msgr2=0x7f1f640fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.401+0000 7f1f68a00700 1 -- 172.21.15.78:0/1328593966 shutdown_connections 2024-03-14T10:04:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.401+0000 7f1f68a00700 1 -- 172.21.15.78:0/1328593966 wait complete. 2024-03-14T10:04:36.421 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T09:48:37.618450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-14T09:52:03.938666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-14T09:52:05.149985+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 95, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-14T09:52:08.227698+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 95, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-14T10:04:36.422 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T10:04:36.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.616+0000 7efe3f800700 1 -- 172.21.15.78:0/995035184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe38073150 msgr2=0x7efe38073530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:36.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.616+0000 7efe3f800700 1 --2- 172.21.15.78:0/995035184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe38073150 0x7efe38073530 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7efe2c009990 tx=0x7efe2c009ca0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.616+0000 7efe3f800700 1 -- 172.21.15.78:0/995035184 shutdown_connections 2024-03-14T10:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.616+0000 7efe3f800700 1 --2- 172.21.15.78:0/995035184 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe38111cc0 0x7efe381140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.616+0000 7efe3f800700 1 --2- 172.21.15.78:0/995035184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe38073150 0x7efe38073530 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.616+0000 7efe3f800700 1 --2- 172.21.15.78:0/995035184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe38073a70 0x7efe38111780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.616+0000 7efe3f800700 1 -- 172.21.15.78:0/995035184 >> 172.21.15.78:0/995035184 conn(0x7efe380fc780 msgr2=0x7efe380feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.616+0000 7efe3f800700 1 -- 172.21.15.78:0/995035184 shutdown_connections 2024-03-14T10:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.616+0000 7efe3f800700 1 -- 172.21.15.78:0/995035184 wait complete. 2024-03-14T10:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.618+0000 7efe3f800700 1 Processor -- start 2024-03-14T10:04:36.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.618+0000 7efe3f800700 1 -- start start 2024-03-14T10:04:36.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.618+0000 7efe3f800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe38073150 0x7efe381a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:36.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.618+0000 7efe3f800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe38073a70 0x7efe381a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:36.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.619+0000 7efe3f800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe38111cc0 0x7efe3819da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.619+0000 7efe3f800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3819e0c0 con 0x7efe38073a70 2024-03-14T10:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.619+0000 7efe3f800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3819e230 con 0x7efe38073150 2024-03-14T10:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.619+0000 7efe3f800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3819e3f0 con 0x7efe38111cc0 2024-03-14T10:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.619+0000 7efe37e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe38073a70 0x7efe381a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.619+0000 7efe37e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe38073a70 0x7efe381a3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45908/0 (socket says 172.21.15.78:45908) 2024-03-14T10:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.619+0000 7efe37e00700 1 -- 172.21.15.78:0/623851574 learned_addr learned my addr 172.21.15.78:0/623851574 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.619+0000 7efe3ce00700 1 --2- 172.21.15.78:0/623851574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe38073150 0x7efe381a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.619+0000 7efe3d800700 1 --2- 172.21.15.78:0/623851574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe38111cc0 0x7efe3819da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.619+0000 7efe37e00700 1 -- 172.21.15.78:0/623851574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe38111cc0 msgr2=0x7efe3819da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.619+0000 7efe37e00700 1 --2- 172.21.15.78:0/623851574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe38111cc0 0x7efe3819da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.619+0000 7efe37e00700 1 -- 172.21.15.78:0/623851574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe38073150 msgr2=0x7efe381a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.619+0000 7efe37e00700 1 --2- 172.21.15.78:0/623851574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe38073150 0x7efe381a3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.619+0000 7efe37e00700 1 -- 172.21.15.78:0/623851574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2c0095f0 con 0x7efe38073a70 2024-03-14T10:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.619+0000 7efe37e00700 1 --2- 172.21.15.78:0/623851574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe38073a70 0x7efe381a3e70 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7efe2400d950 tx=0x7efe2400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:36.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.620+0000 7efe35600700 1 -- 172.21.15.78:0/623851574 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2400f890 con 0x7efe38073a70 2024-03-14T10:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.620+0000 7efe35600700 1 -- 172.21.15.78:0/623851574 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe24010460 con 0x7efe38073a70 2024-03-14T10:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.620+0000 7efe35600700 1 -- 172.21.15.78:0/623851574 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2400e610 con 0x7efe38073a70 2024-03-14T10:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.620+0000 7efe3f800700 1 -- 172.21.15.78:0/623851574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe3819e6d0 con 0x7efe38073a70 2024-03-14T10:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.620+0000 7efe3f800700 1 -- 172.21.15.78:0/623851574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe381aa920 con 0x7efe38073a70 2024-03-14T10:04:36.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.621+0000 7efe3f800700 1 -- 172.21.15.78:0/623851574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe381041a0 con 0x7efe38073a70 2024-03-14T10:04:36.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.623+0000 7efe35600700 1 -- 172.21.15.78:0/623851574 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7efe240108d0 con 0x7efe38073a70 2024-03-14T10:04:36.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.624+0000 7efe35600700 1 --2- 172.21.15.78:0/623851574 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efe200a0c80 0x7efe200a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:36.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.624+0000 7efe35600700 1 -- 172.21.15.78:0/623851574 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7efe240cc590 con 0x7efe38073a70 2024-03-14T10:04:36.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.624+0000 7efe3ce00700 1 --2- 172.21.15.78:0/623851574 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efe200a0c80 0x7efe200a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:36.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.625+0000 7efe3ce00700 1 --2- 172.21.15.78:0/623851574 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efe200a0c80 0x7efe200a3140 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efe2c006010 tx=0x7efe2c00b540 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:36.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.629+0000 7efe35600700 1 -- 172.21.15.78:0/623851574 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7efe24095850 con 0x7efe38073a70 2024-03-14T10:04:36.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.834+0000 7efe3f800700 1 -- 172.21.15.78:0/623851574 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efe3819ef20 con 0x7efe38073a70 2024-03-14T10:04:36.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.835+0000 7efe35600700 1 -- 172.21.15.78:0/623851574 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7efe24098e70 con 0x7efe38073a70 2024-03-14T10:04:36.836 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-14T10:04:36.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.837+0000 7efe3f800700 1 -- 172.21.15.78:0/623851574 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efe200a0c80 msgr2=0x7efe200a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:36.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.837+0000 7efe3f800700 1 --2- 172.21.15.78:0/623851574 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efe200a0c80 0x7efe200a3140 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efe2c006010 tx=0x7efe2c00b540 comp rx=0 tx=0).stop 2024-03-14T10:04:36.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.837+0000 7efe3f800700 1 -- 172.21.15.78:0/623851574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe38073a70 msgr2=0x7efe381a3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:36.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.837+0000 7efe3f800700 1 --2- 172.21.15.78:0/623851574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe38073a70 0x7efe381a3e70 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7efe2400d950 tx=0x7efe2400dc60 comp rx=0 tx=0).stop 2024-03-14T10:04:36.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.837+0000 7efe3f800700 1 -- 172.21.15.78:0/623851574 shutdown_connections 2024-03-14T10:04:36.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.837+0000 7efe3f800700 1 --2- 172.21.15.78:0/623851574 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efe200a0c80 0x7efe200a3140 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.837+0000 7efe3f800700 1 --2- 172.21.15.78:0/623851574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe38073150 0x7efe381a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.837+0000 7efe3f800700 1 --2- 172.21.15.78:0/623851574 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efe38111cc0 0x7efe3819da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.837+0000 7efe3f800700 1 --2- 172.21.15.78:0/623851574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe38073a70 0x7efe381a3e70 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:36.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.837+0000 7efe3f800700 1 -- 172.21.15.78:0/623851574 >> 172.21.15.78:0/623851574 conn(0x7efe380fc780 msgr2=0x7efe380fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:36.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.837+0000 7efe3f800700 1 -- 172.21.15.78:0/623851574 shutdown_connections 2024-03-14T10:04:36.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:36.837+0000 7efe3f800700 1 -- 172.21.15.78:0/623851574 wait complete. 2024-03-14T10:04:36.856 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-14T10:04:37.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.052+0000 7ff270800700 1 -- 172.21.15.78:0/3654439464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff268073180 msgr2=0x7ff268073560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:37.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.052+0000 7ff270800700 1 --2- 172.21.15.78:0/3654439464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff268073180 0x7ff268073560 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff25c009990 tx=0x7ff25c009ca0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.052+0000 7ff270800700 1 -- 172.21.15.78:0/3654439464 shutdown_connections 2024-03-14T10:04:37.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.052+0000 7ff270800700 1 --2- 172.21.15.78:0/3654439464 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff26810d560 0x7ff26810f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.052+0000 7ff270800700 1 --2- 172.21.15.78:0/3654439464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff268073180 0x7ff268073560 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.052+0000 7ff270800700 1 --2- 172.21.15.78:0/3654439464 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff268073aa0 0x7ff26810d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.052+0000 7ff270800700 1 -- 172.21.15.78:0/3654439464 >> 172.21.15.78:0/3654439464 conn(0x7ff2680fc7b0 msgr2=0x7ff2680febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:37.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.052+0000 7ff270800700 1 -- 172.21.15.78:0/3654439464 shutdown_connections 2024-03-14T10:04:37.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.052+0000 7ff270800700 1 -- 172.21.15.78:0/3654439464 wait complete. 2024-03-14T10:04:37.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.053+0000 7ff270800700 1 Processor -- start 2024-03-14T10:04:37.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.053+0000 7ff270800700 1 -- start start 2024-03-14T10:04:37.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff270800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff268073180 0x7ff26819f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:37.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff270800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff268073aa0 0x7ff26819fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:37.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff270800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff26810d560 0x7ff268199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:37.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff270800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff268199cc0 con 0x7ff268073180 2024-03-14T10:04:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff270800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff268199e30 con 0x7ff26810d560 2024-03-14T10:04:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff270800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff268199ff0 con 0x7ff268073aa0 2024-03-14T10:04:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff26de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff268073180 0x7ff26819f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff26de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff268073180 0x7ff26819f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45924/0 (socket says 172.21.15.78:45924) 2024-03-14T10:04:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff26de00700 1 -- 172.21.15.78:0/846579900 learned_addr learned my addr 172.21.15.78:0/846579900 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff26e800700 1 --2- 172.21.15.78:0/846579900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff26810d560 0x7ff268199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff26d400700 1 --2- 172.21.15.78:0/846579900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff268073aa0 0x7ff26819fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff26de00700 1 -- 172.21.15.78:0/846579900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff268073aa0 msgr2=0x7ff26819fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff26de00700 1 --2- 172.21.15.78:0/846579900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff268073aa0 0x7ff26819fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff26de00700 1 -- 172.21.15.78:0/846579900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff26810d560 msgr2=0x7ff268199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff26de00700 1 --2- 172.21.15.78:0/846579900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff26810d560 0x7ff268199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff26de00700 1 -- 172.21.15.78:0/846579900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c0095f0 con 0x7ff268073180 2024-03-14T10:04:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.054+0000 7ff26de00700 1 --2- 172.21.15.78:0/846579900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff268073180 0x7ff26819f530 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff25c000c00 tx=0x7ff25c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:37.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.055+0000 7ff262a00700 1 -- 172.21.15.78:0/846579900 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c01d070 con 0x7ff268073180 2024-03-14T10:04:37.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.055+0000 7ff262a00700 1 -- 172.21.15.78:0/846579900 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff25c022470 con 0x7ff268073180 2024-03-14T10:04:37.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.055+0000 7ff270800700 1 -- 172.21.15.78:0/846579900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff26819a270 con 0x7ff268073180 2024-03-14T10:04:37.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.055+0000 7ff262a00700 1 -- 172.21.15.78:0/846579900 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c017650 con 0x7ff268073180 2024-03-14T10:04:37.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.055+0000 7ff270800700 1 -- 172.21.15.78:0/846579900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2681a6520 con 0x7ff268073180 2024-03-14T10:04:37.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.056+0000 7ff270800700 1 -- 172.21.15.78:0/846579900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff268108770 con 0x7ff268073180 2024-03-14T10:04:37.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.058+0000 7ff262a00700 1 -- 172.21.15.78:0/846579900 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7ff25c0225e0 con 0x7ff268073180 2024-03-14T10:04:37.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.059+0000 7ff262a00700 1 --2- 172.21.15.78:0/846579900 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff2500a0c00 0x7ff2500a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:37.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.059+0000 7ff262a00700 1 -- 172.21.15.78:0/846579900 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7ff25c0cdc60 con 0x7ff268073180 2024-03-14T10:04:37.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.059+0000 7ff26d400700 1 --2- 172.21.15.78:0/846579900 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff2500a0c00 0x7ff2500a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:37.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.060+0000 7ff26d400700 1 --2- 172.21.15.78:0/846579900 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff2500a0c00 0x7ff2500a30c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff26819ad70 tx=0x7ff258009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:37.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.063+0000 7ff262a00700 1 -- 172.21.15.78:0/846579900 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff25c097070 con 0x7ff268073180 2024-03-14T10:04:37.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.272+0000 7ff270800700 1 -- 172.21.15.78:0/846579900 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff26819ab20 con 0x7ff268073180 2024-03-14T10:04:37.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.272+0000 7ff262a00700 1 -- 172.21.15.78:0/846579900 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff25c027020 con 0x7ff268073180 2024-03-14T10:04:37.273 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T10:04:37.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.275+0000 7ff270800700 1 -- 172.21.15.78:0/846579900 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff2500a0c00 msgr2=0x7ff2500a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.275+0000 7ff270800700 1 --2- 172.21.15.78:0/846579900 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff2500a0c00 0x7ff2500a30c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff26819ad70 tx=0x7ff258009260 comp rx=0 tx=0).stop 2024-03-14T10:04:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.275+0000 7ff270800700 1 -- 172.21.15.78:0/846579900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff268073180 msgr2=0x7ff26819f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.275+0000 7ff270800700 1 --2- 172.21.15.78:0/846579900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff268073180 0x7ff26819f530 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff25c000c00 tx=0x7ff25c005740 comp rx=0 tx=0).stop 2024-03-14T10:04:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.275+0000 7ff270800700 1 -- 172.21.15.78:0/846579900 shutdown_connections 2024-03-14T10:04:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.275+0000 7ff270800700 1 --2- 172.21.15.78:0/846579900 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7ff2500a0c00 0x7ff2500a30c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.275+0000 7ff270800700 1 --2- 172.21.15.78:0/846579900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff26810d560 0x7ff268199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.275+0000 7ff270800700 1 --2- 172.21.15.78:0/846579900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff268073180 0x7ff26819f530 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.275+0000 7ff270800700 1 --2- 172.21.15.78:0/846579900 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7ff268073aa0 0x7ff26819fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.275+0000 7ff270800700 1 -- 172.21.15.78:0/846579900 >> 172.21.15.78:0/846579900 conn(0x7ff2680fc7b0 msgr2=0x7ff2680fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.275+0000 7ff270800700 1 -- 172.21.15.78:0/846579900 shutdown_connections 2024-03-14T10:04:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.275+0000 7ff270800700 1 -- 172.21.15.78:0/846579900 wait complete. 2024-03-14T10:04:37.294 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-14T10:04:37.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.491+0000 7f5b87800700 1 -- 172.21.15.78:0/2009683045 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b801018d0 msgr2=0x7f5b80105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:37.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.491+0000 7f5b87800700 1 --2- 172.21.15.78:0/2009683045 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b801018d0 0x7f5b80105910 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c009990 tx=0x7f5b6c009ca0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.491+0000 7f5b87800700 1 -- 172.21.15.78:0/2009683045 shutdown_connections 2024-03-14T10:04:37.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.491+0000 7f5b87800700 1 --2- 172.21.15.78:0/2009683045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b80100f20 0x7f5b80101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.491+0000 7f5b87800700 1 --2- 172.21.15.78:0/2009683045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b80106040 0x7f5b8010dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.491+0000 7f5b87800700 1 --2- 172.21.15.78:0/2009683045 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b801018d0 0x7f5b80105910 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.491+0000 7f5b87800700 1 -- 172.21.15.78:0/2009683045 >> 172.21.15.78:0/2009683045 conn(0x7f5b800fc7a0 msgr2=0x7f5b800febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:37.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.491+0000 7f5b87800700 1 -- 172.21.15.78:0/2009683045 shutdown_connections 2024-03-14T10:04:37.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.491+0000 7f5b87800700 1 -- 172.21.15.78:0/2009683045 wait complete. 2024-03-14T10:04:37.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.492+0000 7f5b87800700 1 Processor -- start 2024-03-14T10:04:37.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.492+0000 7f5b87800700 1 -- start start 2024-03-14T10:04:37.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.492+0000 7f5b87800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b80100f20 0x7f5b800730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:37.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.492+0000 7f5b87800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b80106040 0x7f5b800735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:37.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.492+0000 7f5b87800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b80073b30 0x7f5b801a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:37.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.493+0000 7f5b87800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b801a5ea0 con 0x7f5b80100f20 2024-03-14T10:04:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.493+0000 7f5b87800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b801a5fe0 con 0x7f5b80073b30 2024-03-14T10:04:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.493+0000 7f5b87800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b801a61a0 con 0x7f5b80106040 2024-03-14T10:04:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.493+0000 7f5b84e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b80100f20 0x7f5b800730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.493+0000 7f5b84e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b80100f20 0x7f5b800730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45952/0 (socket says 172.21.15.78:45952) 2024-03-14T10:04:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.493+0000 7f5b84e00700 1 -- 172.21.15.78:0/1918114550 learned_addr learned my addr 172.21.15.78:0/1918114550 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.493+0000 7f5b7fe00700 1 --2- 172.21.15.78:0/1918114550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b80106040 0x7f5b800735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.493+0000 7f5b85800700 1 --2- 172.21.15.78:0/1918114550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b80073b30 0x7f5b801a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.493+0000 7f5b84e00700 1 -- 172.21.15.78:0/1918114550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b80106040 msgr2=0x7f5b800735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.493+0000 7f5b84e00700 1 --2- 172.21.15.78:0/1918114550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b80106040 0x7f5b800735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.493+0000 7f5b84e00700 1 -- 172.21.15.78:0/1918114550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b80073b30 msgr2=0x7f5b801a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.493+0000 7f5b84e00700 1 --2- 172.21.15.78:0/1918114550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b80073b30 0x7f5b801a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.493+0000 7f5b84e00700 1 -- 172.21.15.78:0/1918114550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c0095f0 con 0x7f5b80100f20 2024-03-14T10:04:37.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.494+0000 7f5b84e00700 1 --2- 172.21.15.78:0/1918114550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b80100f20 0x7f5b800730b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400d980 tx=0x7f5b7400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:37.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.494+0000 7f5b7d600700 1 -- 172.21.15.78:0/1918114550 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b74004220 con 0x7f5b80100f20 2024-03-14T10:04:37.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.494+0000 7f5b7d600700 1 -- 172.21.15.78:0/1918114550 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b74007640 con 0x7f5b80100f20 2024-03-14T10:04:37.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.494+0000 7f5b87800700 1 -- 172.21.15.78:0/1918114550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b801a6480 con 0x7f5b80100f20 2024-03-14T10:04:37.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.494+0000 7f5b7d600700 1 -- 172.21.15.78:0/1918114550 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b74005690 con 0x7f5b80100f20 2024-03-14T10:04:37.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.494+0000 7f5b87800700 1 -- 172.21.15.78:0/1918114550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b801a6990 con 0x7f5b80100f20 2024-03-14T10:04:37.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.495+0000 7f5b87800700 1 -- 172.21.15.78:0/1918114550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b80109310 con 0x7f5b80100f20 2024-03-14T10:04:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:37 smithi152 ceph-mon[48279]: pgmap v324: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:37 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1328593966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T10:04:37.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:37 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/623851574' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T10:04:37.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.498+0000 7f5b7d600700 1 -- 172.21.15.78:0/1918114550 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f5b74004750 con 0x7f5b80100f20 2024-03-14T10:04:37.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.499+0000 7f5b7d600700 1 --2- 172.21.15.78:0/1918114550 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5b680a0c00 0x7f5b680a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:37.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.500+0000 7f5b7d600700 1 -- 172.21.15.78:0/1918114550 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f5b740ccb30 con 0x7f5b80100f20 2024-03-14T10:04:37.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.500+0000 7f5b7fe00700 1 --2- 172.21.15.78:0/1918114550 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5b680a0c00 0x7f5b680a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:37.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.500+0000 7f5b7fe00700 1 --2- 172.21.15.78:0/1918114550 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5b680a0c00 0x7f5b680a30c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5b80074790 tx=0x7f5b6c00b260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:37.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.502+0000 7f5b7d600700 1 -- 172.21.15.78:0/1918114550 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5b74095d90 con 0x7f5b80100f20 2024-03-14T10:04:37.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:37 smithi132 ceph-mon[47959]: pgmap v324: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:37.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:37 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1328593966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T10:04:37.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:37 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/623851574' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T10:04:37.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:37 smithi078 ceph-mon[38143]: pgmap v324: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:37.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:37 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1328593966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T10:04:37.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:37 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/623851574' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T10:04:37.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.706+0000 7f5b87800700 1 -- 172.21.15.78:0/1918114550 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5b80067af0 con 0x7f5b80100f20 2024-03-14T10:04:37.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.707+0000 7f5b7d600700 1 -- 172.21.15.78:0/1918114550 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5b74016020 con 0x7f5b80100f20 2024-03-14T10:04:37.707 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-14T10:04:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.709+0000 7f5b87800700 1 -- 172.21.15.78:0/1918114550 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5b680a0c00 msgr2=0x7f5b680a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.709+0000 7f5b87800700 1 --2- 172.21.15.78:0/1918114550 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5b680a0c00 0x7f5b680a30c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5b80074790 tx=0x7f5b6c00b260 comp rx=0 tx=0).stop 2024-03-14T10:04:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.709+0000 7f5b87800700 1 -- 172.21.15.78:0/1918114550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b80100f20 msgr2=0x7f5b800730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.709+0000 7f5b87800700 1 --2- 172.21.15.78:0/1918114550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b80100f20 0x7f5b800730b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5b7400d980 tx=0x7f5b7400dc90 comp rx=0 tx=0).stop 2024-03-14T10:04:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.709+0000 7f5b87800700 1 -- 172.21.15.78:0/1918114550 shutdown_connections 2024-03-14T10:04:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.709+0000 7f5b87800700 1 --2- 172.21.15.78:0/1918114550 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f5b680a0c00 0x7f5b680a30c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.709+0000 7f5b87800700 1 --2- 172.21.15.78:0/1918114550 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5b80073b30 0x7f5b801a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.709+0000 7f5b87800700 1 --2- 172.21.15.78:0/1918114550 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5b80100f20 0x7f5b800730b0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.709+0000 7f5b87800700 1 --2- 172.21.15.78:0/1918114550 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5b80106040 0x7f5b800735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.709+0000 7f5b87800700 1 -- 172.21.15.78:0/1918114550 >> 172.21.15.78:0/1918114550 conn(0x7f5b800fc7a0 msgr2=0x7f5b800fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.709+0000 7f5b87800700 1 -- 172.21.15.78:0/1918114550 shutdown_connections 2024-03-14T10:04:37.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.709+0000 7f5b87800700 1 -- 172.21.15.78:0/1918114550 wait complete. 2024-03-14T10:04:37.728 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-14T10:04:37.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.928+0000 7f5945200700 1 -- 172.21.15.78:0/309708363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5940100f30 msgr2=0x7f5940101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:37.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.928+0000 7f5945200700 1 --2- 172.21.15.78:0/309708363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5940100f30 0x7f5940101310 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f59300098f0 tx=0x7f5930009c00 comp rx=0 tx=0).stop 2024-03-14T10:04:37.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.929+0000 7f5945200700 1 -- 172.21.15.78:0/309708363 shutdown_connections 2024-03-14T10:04:37.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.929+0000 7f5945200700 1 --2- 172.21.15.78:0/309708363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5940100f30 0x7f5940101310 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.929+0000 7f5945200700 1 --2- 172.21.15.78:0/309708363 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5940106050 0x7f594010dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.929+0000 7f5945200700 1 --2- 172.21.15.78:0/309708363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59401018e0 0x7f5940105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.929+0000 7f5945200700 1 -- 172.21.15.78:0/309708363 >> 172.21.15.78:0/309708363 conn(0x7f59400fc7b0 msgr2=0x7f59400febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:37.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.929+0000 7f5945200700 1 -- 172.21.15.78:0/309708363 shutdown_connections 2024-03-14T10:04:37.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.929+0000 7f5945200700 1 -- 172.21.15.78:0/309708363 wait complete. 2024-03-14T10:04:37.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.930+0000 7f5945200700 1 Processor -- start 2024-03-14T10:04:37.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.930+0000 7f5945200700 1 -- start start 2024-03-14T10:04:37.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.930+0000 7f5945200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5940100f30 0x7f594019f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:37.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.930+0000 7f5945200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5940106050 0x7f594019fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:37.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.930+0000 7f5945200700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59401996f0 0x7f5940199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:37.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.930+0000 7f5945200700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594019a170 con 0x7f5940100f30 2024-03-14T10:04:37.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.930+0000 7f5945200700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594019a2e0 con 0x7f5940106050 2024-03-14T10:04:37.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.930+0000 7f5945200700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594019a4a0 con 0x7f59401996f0 2024-03-14T10:04:37.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.930+0000 7f593de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5940100f30 0x7f594019f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:37.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.930+0000 7f593d400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5940106050 0x7f594019fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:37.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.930+0000 7f593de00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5940100f30 0x7f594019f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45976/0 (socket says 172.21.15.78:45976) 2024-03-14T10:04:37.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.930+0000 7f593de00700 1 -- 172.21.15.78:0/3555172035 learned_addr learned my addr 172.21.15.78:0/3555172035 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:37.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.931+0000 7f593e800700 1 --2- 172.21.15.78:0/3555172035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59401996f0 0x7f5940199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:37.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.931+0000 7f593de00700 1 -- 172.21.15.78:0/3555172035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59401996f0 msgr2=0x7f5940199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:37.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.931+0000 7f593de00700 1 --2- 172.21.15.78:0/3555172035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59401996f0 0x7f5940199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.931+0000 7f593de00700 1 -- 172.21.15.78:0/3555172035 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5940106050 msgr2=0x7f594019fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:37.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.931+0000 7f593de00700 1 --2- 172.21.15.78:0/3555172035 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5940106050 0x7f594019fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:37.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.931+0000 7f593de00700 1 -- 172.21.15.78:0/3555172035 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592c0095f0 con 0x7f5940100f30 2024-03-14T10:04:37.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.931+0000 7f593de00700 1 --2- 172.21.15.78:0/3555172035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5940100f30 0x7f594019f570 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f5930005950 tx=0x7f593000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:37.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.931+0000 7f5936a00700 1 -- 172.21.15.78:0/3555172035 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593001c070 con 0x7f5940100f30 2024-03-14T10:04:37.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.931+0000 7f5936a00700 1 -- 172.21.15.78:0/3555172035 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f593000baa0 con 0x7f5940100f30 2024-03-14T10:04:37.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.931+0000 7f5945200700 1 -- 172.21.15.78:0/3555172035 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5930009520 con 0x7f5940100f30 2024-03-14T10:04:37.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.931+0000 7f5936a00700 1 -- 172.21.15.78:0/3555172035 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59300038e0 con 0x7f5940100f30 2024-03-14T10:04:37.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.932+0000 7f5945200700 1 -- 172.21.15.78:0/3555172035 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59401a66d0 con 0x7f5940100f30 2024-03-14T10:04:37.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.933+0000 7f5945200700 1 -- 172.21.15.78:0/3555172035 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5940102400 con 0x7f5940100f30 2024-03-14T10:04:37.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.935+0000 7f5936a00700 1 -- 172.21.15.78:0/3555172035 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f5930020030 con 0x7f5940100f30 2024-03-14T10:04:37.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.937+0000 7f5936a00700 1 --2- 172.21.15.78:0/3555172035 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f59240a0c00 0x7f59240a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:37.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.937+0000 7f5936a00700 1 -- 172.21.15.78:0/3555172035 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f59300cd590 con 0x7f5940100f30 2024-03-14T10:04:37.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.937+0000 7f593d400700 1 --2- 172.21.15.78:0/3555172035 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f59240a0c00 0x7f59240a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:37.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.938+0000 7f593d400700 1 --2- 172.21.15.78:0/3555172035 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f59240a0c00 0x7f59240a30c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f594019adb0 tx=0x7f592c009310 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:37.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:37.941+0000 7f5936a00700 1 -- 172.21.15.78:0/3555172035 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f59300968e0 con 0x7f5940100f30 2024-03-14T10:04:38.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.147+0000 7f5945200700 1 -- 172.21.15.78:0/3555172035 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f594019ac00 con 0x7f5940100f30 2024-03-14T10:04:38.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.147+0000 7f5936a00700 1 -- 172.21.15.78:0/3555172035 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5930099f00 con 0x7f5940100f30 2024-03-14T10:04:38.148 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T10:04:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.150+0000 7f5945200700 1 -- 172.21.15.78:0/3555172035 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f59240a0c00 msgr2=0x7f59240a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.150+0000 7f5945200700 1 --2- 172.21.15.78:0/3555172035 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f59240a0c00 0x7f59240a30c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f594019adb0 tx=0x7f592c009310 comp rx=0 tx=0).stop 2024-03-14T10:04:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.150+0000 7f5945200700 1 -- 172.21.15.78:0/3555172035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5940100f30 msgr2=0x7f594019f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.150+0000 7f5945200700 1 --2- 172.21.15.78:0/3555172035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5940100f30 0x7f594019f570 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f5930005950 tx=0x7f593000fa00 comp rx=0 tx=0).stop 2024-03-14T10:04:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.150+0000 7f5945200700 1 -- 172.21.15.78:0/3555172035 shutdown_connections 2024-03-14T10:04:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.150+0000 7f5945200700 1 --2- 172.21.15.78:0/3555172035 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f59401996f0 0x7f5940199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.150+0000 7f5945200700 1 --2- 172.21.15.78:0/3555172035 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5940100f30 0x7f594019f570 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.150+0000 7f5945200700 1 --2- 172.21.15.78:0/3555172035 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f59240a0c00 0x7f59240a30c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.150+0000 7f5945200700 1 --2- 172.21.15.78:0/3555172035 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5940106050 0x7f594019fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.150+0000 7f5945200700 1 -- 172.21.15.78:0/3555172035 >> 172.21.15.78:0/3555172035 conn(0x7f59400fc7b0 msgr2=0x7f59400fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.150+0000 7f5945200700 1 -- 172.21.15.78:0/3555172035 shutdown_connections 2024-03-14T10:04:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.150+0000 7f5945200700 1 -- 172.21.15.78:0/3555172035 wait complete. 2024-03-14T10:04:38.169 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-14T10:04:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.362+0000 7f68c1c00700 1 -- 172.21.15.78:0/1488716801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68bc100e20 msgr2=0x7f68bc101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.362+0000 7f68c1c00700 1 --2- 172.21.15.78:0/1488716801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68bc100e20 0x7f68bc101200 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009990 tx=0x7f68ac009ca0 comp rx=0 tx=0).stop 2024-03-14T10:04:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.362+0000 7f68c1c00700 1 -- 172.21.15.78:0/1488716801 shutdown_connections 2024-03-14T10:04:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.362+0000 7f68c1c00700 1 --2- 172.21.15.78:0/1488716801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68bc105f40 0x7f68bc10dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.362+0000 7f68c1c00700 1 --2- 172.21.15.78:0/1488716801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68bc100e20 0x7f68bc101200 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.362+0000 7f68c1c00700 1 --2- 172.21.15.78:0/1488716801 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68bc1017d0 0x7f68bc105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.362+0000 7f68c1c00700 1 -- 172.21.15.78:0/1488716801 >> 172.21.15.78:0/1488716801 conn(0x7f68bc0fc700 msgr2=0x7f68bc0feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.362+0000 7f68c1c00700 1 -- 172.21.15.78:0/1488716801 shutdown_connections 2024-03-14T10:04:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.362+0000 7f68c1c00700 1 -- 172.21.15.78:0/1488716801 wait complete. 2024-03-14T10:04:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.363+0000 7f68c1c00700 1 Processor -- start 2024-03-14T10:04:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.363+0000 7f68c1c00700 1 -- start start 2024-03-14T10:04:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68c1c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68bc100e20 0x7f68bc199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68c1c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68bc105f40 0x7f68bc199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68c1c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68bc19e050 0x7f68bc19e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68c1c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc19ea70 con 0x7f68bc105f40 2024-03-14T10:04:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68c1c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc19ebe0 con 0x7f68bc19e050 2024-03-14T10:04:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68c1c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68bc19eda0 con 0x7f68bc100e20 2024-03-14T10:04:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68ba000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68bc105f40 0x7f68bc199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68baa00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68bc100e20 0x7f68bc199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68ba000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68bc105f40 0x7f68bc199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45988/0 (socket says 172.21.15.78:45988) 2024-03-14T10:04:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68ba000700 1 -- 172.21.15.78:0/3313851237 learned_addr learned my addr 172.21.15.78:0/3313851237 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:04:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68bb400700 1 --2- 172.21.15.78:0/3313851237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68bc19e050 0x7f68bc19e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68ba000700 1 -- 172.21.15.78:0/3313851237 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68bc100e20 msgr2=0x7f68bc199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68ba000700 1 --2- 172.21.15.78:0/3313851237 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68bc100e20 0x7f68bc199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68ba000700 1 -- 172.21.15.78:0/3313851237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68bc19e050 msgr2=0x7f68bc19e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68ba000700 1 --2- 172.21.15.78:0/3313851237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68bc19e050 0x7f68bc19e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68ba000700 1 -- 172.21.15.78:0/3313851237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac0095f0 con 0x7f68bc105f40 2024-03-14T10:04:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.364+0000 7f68bb400700 1 --2- 172.21.15.78:0/3313851237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68bc19e050 0x7f68bc19e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:04:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.365+0000 7f68ba000700 1 --2- 172.21.15.78:0/3313851237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68bc105f40 0x7f68bc199d70 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f68a800e970 tx=0x7f68a800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.365+0000 7f68b3400700 1 -- 172.21.15.78:0/3313851237 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a8009790 con 0x7f68bc105f40 2024-03-14T10:04:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.365+0000 7f68b3400700 1 -- 172.21.15.78:0/3313851237 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68a800eec0 con 0x7f68bc105f40 2024-03-14T10:04:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.365+0000 7f68b3400700 1 -- 172.21.15.78:0/3313851237 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a8005590 con 0x7f68bc105f40 2024-03-14T10:04:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.365+0000 7f68c1c00700 1 -- 172.21.15.78:0/3313851237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68bc1a6380 con 0x7f68bc105f40 2024-03-14T10:04:38.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.365+0000 7f68c1c00700 1 -- 172.21.15.78:0/3313851237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68bc1a6860 con 0x7f68bc105f40 2024-03-14T10:04:38.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.366+0000 7f68c1c00700 1 -- 172.21.15.78:0/3313851237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68bc1022f0 con 0x7f68bc105f40 2024-03-14T10:04:38.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.368+0000 7f68b3400700 1 -- 172.21.15.78:0/3313851237 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f68a800cbd0 con 0x7f68bc105f40 2024-03-14T10:04:38.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.370+0000 7f68b3400700 1 --2- 172.21.15.78:0/3313851237 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f68a00a0c80 0x7f68a00a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:04:38.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.370+0000 7f68b3400700 1 -- 172.21.15.78:0/3313851237 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f68a8014070 con 0x7f68bc105f40 2024-03-14T10:04:38.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.370+0000 7f68baa00700 1 --2- 172.21.15.78:0/3313851237 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f68a00a0c80 0x7f68a00a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:04:38.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.371+0000 7f68baa00700 1 --2- 172.21.15.78:0/3313851237 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f68a00a0c80 0x7f68a00a3140 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f68bc0fff60 tx=0x7f68ac00b260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:04:38.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.374+0000 7f68b3400700 1 -- 172.21.15.78:0/3313851237 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f68a8095c90 con 0x7f68bc105f40 2024-03-14T10:04:38.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:38 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/846579900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T10:04:38.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:38 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1918114550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T10:04:38.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:38 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3555172035' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T10:04:38.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:38 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/846579900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T10:04:38.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:38 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1918114550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T10:04:38.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:38 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3555172035' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T10:04:38.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:38 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/846579900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T10:04:38.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:38 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1918114550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T10:04:38.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:38 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3555172035' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T10:04:38.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.638+0000 7f68c1c00700 1 -- 172.21.15.78:0/3313851237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f68bc0679e0 con 0x7f68bc105f40 2024-03-14T10:04:38.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.647+0000 7f68b3400700 1 -- 172.21.15.78:0/3313851237 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) v1 ==== 81+0+732000 (secure 0 0 0) 0x7f68a8004ef0 con 0x7f68bc105f40 2024-03-14T10:04:38.660 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T10:04:38.660 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:04:38.660 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 27, 2024-03-14T10:04:38.660 INFO:teuthology.orchestra.run.smithi078.stdout: "active_gid": 15546, 2024-03-14T10:04:38.660 INFO:teuthology.orchestra.run.smithi078.stdout: "active_name": "x", 2024-03-14T10:04:38.661 INFO:teuthology.orchestra.run.smithi078.stdout: "active_addrs": { 2024-03-14T10:04:38.661 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-14T10:04:38.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.661 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-14T10:04:38.661 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6800", 2024-03-14T10:04:38.661 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2192201293 2024-03-14T10:04:38.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.661 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-03-14T10:04:38.661 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6801", 2024-03-14T10:04:38.661 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2192201293 2024-03-14T10:04:38.661 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.661 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-14T10:04:38.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.661 INFO:teuthology.orchestra.run.smithi078.stdout: "active_addr": "172.21.15.78:6801/2192201293", 2024-03-14T10:04:38.662 INFO:teuthology.orchestra.run.smithi078.stdout: "active_change": "2024-03-14T09:54:09.925087+0000", 2024-03-14T10:04:38.662 INFO:teuthology.orchestra.run.smithi078.stdout: "active_mgr_features": 4540138322906710015, 2024-03-14T10:04:38.662 INFO:teuthology.orchestra.run.smithi078.stdout: "available": true, 2024-03-14T10:04:38.662 INFO:teuthology.orchestra.run.smithi078.stdout: "standbys": [ 2024-03-14T10:04:38.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.662 INFO:teuthology.orchestra.run.smithi078.stdout: "gid": 25546, 2024-03-14T10:04:38.662 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "z", 2024-03-14T10:04:38.662 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_features": 4540138322906710015, 2024-03-14T10:04:38.662 INFO:teuthology.orchestra.run.smithi078.stdout: "available_modules": [ 2024-03-14T10:04:38.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.662 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alerts", 2024-03-14T10:04:38.662 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.662 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.663 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.663 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-14T10:04:38.663 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-14T10:04:38.663 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.663 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.663 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.663 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-14T10:04:38.663 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.663 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.663 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.663 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How frequently to reexamine health status", 2024-03-14T10:04:38.663 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.663 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.663 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.664 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.664 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.664 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.664 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.664 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.664 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.664 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.664 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.664 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.664 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.664 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.664 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.664 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.664 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.664 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.665 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.665 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.665 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.665 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.665 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.665 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.665 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.665 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.665 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.665 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.665 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.666 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.666 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.666 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.666 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.666 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.666 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.666 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.666 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.666 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.666 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.666 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.667 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.668 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.669 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_destination": { 2024-03-14T10:04:38.669 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_destination", 2024-03-14T10:04:38.669 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.669 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.669 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.669 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.669 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.669 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.669 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.669 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email address to send alerts to", 2024-03-14T10:04:38.669 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.669 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.669 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.670 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_from_name": { 2024-03-14T10:04:38.670 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_from_name", 2024-03-14T10:04:38.670 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.670 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.670 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.670 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "Ceph", 2024-03-14T10:04:38.670 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.670 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.670 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.670 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email From: name", 2024-03-14T10:04:38.670 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.670 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.670 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.670 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_host": { 2024-03-14T10:04:38.671 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_host", 2024-03-14T10:04:38.671 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.671 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.671 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.671 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.671 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.671 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.671 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.671 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP server", 2024-03-14T10:04:38.671 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.671 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.671 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.671 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_password": { 2024-03-14T10:04:38.671 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_password", 2024-03-14T10:04:38.671 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Password to authenticate with", 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_port": { 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_port", 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.672 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "465", 2024-03-14T10:04:38.673 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.673 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.673 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.673 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP port", 2024-03-14T10:04:38.673 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.673 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.673 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.673 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_sender": { 2024-03-14T10:04:38.673 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_sender", 2024-03-14T10:04:38.673 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.673 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.673 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.673 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.674 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.674 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.674 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.674 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP envelope sender", 2024-03-14T10:04:38.674 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.674 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.674 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.674 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_ssl": { 2024-03-14T10:04:38.674 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_ssl", 2024-03-14T10:04:38.674 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.674 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.674 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.675 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.675 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.675 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.675 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.675 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-14T10:04:38.675 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.675 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.675 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.675 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_user": { 2024-03-14T10:04:38.675 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_user", 2024-03-14T10:04:38.675 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.675 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.675 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.675 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.675 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.676 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.676 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.676 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "User to authenticate as", 2024-03-14T10:04:38.676 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.676 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.676 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.676 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.676 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.676 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "balancer", 2024-03-14T10:04:38.676 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.676 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.676 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.676 INFO:teuthology.orchestra.run.smithi078.stdout: "active": { 2024-03-14T10:04:38.676 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "active", 2024-03-14T10:04:38.677 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.677 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.677 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.677 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.677 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.677 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.677 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.677 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically balance PGs across cluster", 2024-03-14T10:04:38.677 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.677 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.677 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.677 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_time": { 2024-03-14T10:04:38.677 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_time", 2024-03-14T10:04:38.677 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.678 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.678 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.678 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0000", 2024-03-14T10:04:38.678 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.678 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.678 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.678 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "beginning time of day to automatically balance", 2024-03-14T10:04:38.678 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-14T10:04:38.678 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.678 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.678 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_weekday": { 2024-03-14T10:04:38.678 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_weekday", 2024-03-14T10:04:38.679 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-14T10:04:38.679 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.679 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.679 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-14T10:04:38.679 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-14T10:04:38.679 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-14T10:04:38.679 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.679 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-14T10:04:38.679 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-14T10:04:38.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.679 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.679 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_max_iterations": { 2024-03-14T10:04:38.679 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_max_iterations", 2024-03-14T10:04:38.680 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-14T10:04:38.680 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.680 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.680 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "25", 2024-03-14T10:04:38.680 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-14T10:04:38.680 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "250", 2024-03-14T10:04:38.680 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.680 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-14T10:04:38.680 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.680 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.680 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.680 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_metrics": { 2024-03-14T10:04:38.680 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_metrics", 2024-03-14T10:04:38.680 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.681 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.681 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.681 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "pgs,objects,bytes", 2024-03-14T10:04:38.681 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.681 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.681 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.681 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-14T10:04:38.681 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-14T10:04:38.681 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.681 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.681 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_step": { 2024-03-14T10:04:38.681 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_step", 2024-03-14T10:04:38.681 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:38.681 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.682 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.682 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.5", 2024-03-14T10:04:38.682 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0.001", 2024-03-14T10:04:38.682 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "0.999", 2024-03-14T10:04:38.682 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.682 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "aggressiveness of optimization", 2024-03-14T10:04:38.682 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-14T10:04:38.682 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.682 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.682 INFO:teuthology.orchestra.run.smithi078.stdout: "end_time": { 2024-03-14T10:04:38.682 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_time", 2024-03-14T10:04:38.682 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.682 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.683 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.683 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2359", 2024-03-14T10:04:38.683 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.683 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.683 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.683 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ending time of day to automatically balance", 2024-03-14T10:04:38.683 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-14T10:04:38.683 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.683 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.683 INFO:teuthology.orchestra.run.smithi078.stdout: "end_weekday": { 2024-03-14T10:04:38.683 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_weekday", 2024-03-14T10:04:38.683 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-14T10:04:38.683 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.683 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.684 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.685 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.685 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.685 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.685 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.685 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.685 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.685 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.685 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.685 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.685 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.685 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.685 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.685 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.685 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.685 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.686 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.686 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.686 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.686 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.686 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.686 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.686 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.686 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.686 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.686 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.686 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.687 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.687 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.687 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.687 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.687 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.687 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.687 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.687 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.687 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.687 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.687 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.687 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.687 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.688 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.688 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.688 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.688 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.688 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.688 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.688 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.688 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.688 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.688 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.688 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.688 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.689 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.689 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.689 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.689 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.689 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.689 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.689 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.689 INFO:teuthology.orchestra.run.smithi078.stdout: "min_score": { 2024-03-14T10:04:38.689 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_score", 2024-03-14T10:04:38.689 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:38.689 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.689 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.690 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-14T10:04:38.690 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.690 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.690 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.690 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-14T10:04:38.690 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.690 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.690 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.690 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-14T10:04:38.690 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-14T10:04:38.690 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.691 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.691 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.691 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "upmap", 2024-03-14T10:04:38.691 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.691 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.691 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.691 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-compat", 2024-03-14T10:04:38.691 INFO:teuthology.orchestra.run.smithi078.stdout: "none", 2024-03-14T10:04:38.691 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap" 2024-03-14T10:04:38.691 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.691 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Balancer mode", 2024-03-14T10:04:38.691 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.691 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.692 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.692 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_ids": { 2024-03-14T10:04:38.692 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_ids", 2024-03-14T10:04:38.692 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.692 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.692 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.692 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.692 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.692 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.692 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.692 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-14T10:04:38.693 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.693 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.693 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.693 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-14T10:04:38.693 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-14T10:04:38.693 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.693 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.693 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.693 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-14T10:04:38.693 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.693 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.694 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.694 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-14T10:04:38.694 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.694 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.694 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.694 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_deviation": { 2024-03-14T10:04:38.694 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_deviation", 2024-03-14T10:04:38.694 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.694 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.694 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.694 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-14T10:04:38.694 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-14T10:04:38.695 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.695 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.695 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-14T10:04:38.695 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-14T10:04:38.695 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.695 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.695 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_optimizations": { 2024-03-14T10:04:38.695 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_optimizations", 2024-03-14T10:04:38.695 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-14T10:04:38.695 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.695 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.695 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:38.696 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.696 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.696 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.696 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-14T10:04:38.696 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.696 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.696 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.696 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm", 2024-03-14T10:04:38.696 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.697 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.697 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.697 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_down_multiplier": { 2024-03-14T10:04:38.697 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_down_multiplier", 2024-03-14T10:04:38.697 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:38.697 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.697 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.697 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-14T10:04:38.697 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.697 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.697 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.697 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-14T10:04:38.697 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.697 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_refresh_rate": { 2024-03-14T10:04:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_refresh_rate", 2024-03-14T10:04:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "20", 2024-03-14T10:04:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.698 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-14T10:04:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_starting_port": { 2024-03-14T10:04:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_starting_port", 2024-03-14T10:04:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4721", 2024-03-14T10:04:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.699 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-14T10:04:38.700 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.700 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.700 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.700 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_ptrace": { 2024-03-14T10:04:38.700 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_ptrace", 2024-03-14T10:04:38.700 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.700 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.700 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.700 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.700 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.700 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.701 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.701 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-14T10:04:38.701 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-14T10:04:38.701 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.701 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.701 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_interval": { 2024-03-14T10:04:38.701 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_interval", 2024-03-14T10:04:38.701 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.701 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.701 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.701 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-14T10:04:38.701 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.702 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.702 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.702 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-14T10:04:38.702 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.702 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.702 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.702 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_memory_target_ratio": { 2024-03-14T10:04:38.702 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_memory_target_ratio", 2024-03-14T10:04:38.702 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:38.702 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.702 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.702 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.7", 2024-03-14T10:04:38.702 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.703 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.703 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.703 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-14T10:04:38.703 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.703 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.703 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.703 INFO:teuthology.orchestra.run.smithi078.stdout: "cgroups_split": { 2024-03-14T10:04:38.703 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cgroups_split", 2024-03-14T10:04:38.703 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.703 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.703 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.703 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.703 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.704 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.704 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.704 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-14T10:04:38.704 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.704 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.704 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.704 INFO:teuthology.orchestra.run.smithi078.stdout: "config_checks_enabled": { 2024-03-14T10:04:38.704 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_checks_enabled", 2024-03-14T10:04:38.704 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.704 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.704 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.704 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.705 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.705 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.705 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.705 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-14T10:04:38.705 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.705 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.705 INFO:teuthology.orchestra.run.smithi078.stdout: "config_dashboard": { 2024-03-14T10:04:38.705 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_dashboard", 2024-03-14T10:04:38.705 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.705 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.705 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.705 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.705 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.706 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.706 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.706 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-14T10:04:38.706 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.706 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.706 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.706 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_alertmanager": { 2024-03-14T10:04:38.706 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_alertmanager", 2024-03-14T10:04:38.706 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.706 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.706 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.706 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-14T10:04:38.706 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.707 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.707 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.707 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-14T10:04:38.707 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.707 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.707 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.707 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_base": { 2024-03-14T10:04:38.707 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_base", 2024-03-14T10:04:38.707 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.707 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.707 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.707 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-14T10:04:38.707 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.708 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.708 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.708 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Container image name, without the tag", 2024-03-14T10:04:38.708 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.708 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.708 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.708 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_elasticsearch": { 2024-03-14T10:04:38.708 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_elasticsearch", 2024-03-14T10:04:38.708 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.708 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.708 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.708 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-14T10:04:38.708 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.708 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "elasticsearch container image", 2024-03-14T10:04:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_grafana": { 2024-03-14T10:04:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_grafana", 2024-03-14T10:04:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2024-03-14T10:04:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.709 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.710 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-14T10:04:38.710 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.710 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.710 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.710 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_haproxy": { 2024-03-14T10:04:38.710 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_haproxy", 2024-03-14T10:04:38.710 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.710 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.711 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.711 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-14T10:04:38.711 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.711 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.711 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.711 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "HAproxy container image", 2024-03-14T10:04:38.711 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.711 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.711 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.711 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_agent": { 2024-03-14T10:04:38.711 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_agent", 2024-03-14T10:04:38.711 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.712 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.712 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.712 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-14T10:04:38.712 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.712 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.712 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.712 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger agent container image", 2024-03-14T10:04:38.712 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.712 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.712 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.712 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_collector": { 2024-03-14T10:04:38.712 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_collector", 2024-03-14T10:04:38.712 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.712 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.712 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.713 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-14T10:04:38.713 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.713 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.713 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.713 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger collector container image", 2024-03-14T10:04:38.713 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.713 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.713 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.713 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_query": { 2024-03-14T10:04:38.713 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_query", 2024-03-14T10:04:38.713 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.713 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.713 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-14T10:04:38.714 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.714 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.714 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.714 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger query container image", 2024-03-14T10:04:38.714 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.714 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.714 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.714 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_keepalived": { 2024-03-14T10:04:38.714 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_keepalived", 2024-03-14T10:04:38.714 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.714 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.714 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.714 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-14T10:04:38.715 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.715 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.715 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.715 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Keepalived container image", 2024-03-14T10:04:38.715 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.715 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.715 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.715 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_loki": { 2024-03-14T10:04:38.715 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_loki", 2024-03-14T10:04:38.715 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.715 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.715 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.715 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-14T10:04:38.716 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.716 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.716 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.716 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Loki container image", 2024-03-14T10:04:38.716 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.716 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.716 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.716 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_node_exporter": { 2024-03-14T10:04:38.716 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_node_exporter", 2024-03-14T10:04:38.716 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.716 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.716 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.716 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-14T10:04:38.716 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.716 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.717 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.717 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-14T10:04:38.717 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.717 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.717 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.717 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_nvmeof": { 2024-03-14T10:04:38.717 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_nvmeof", 2024-03-14T10:04:38.717 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.717 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.717 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.717 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/nvmeof:0.0.2", 2024-03-14T10:04:38.717 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.717 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.717 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.718 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Nvme-of container image", 2024-03-14T10:04:38.718 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.718 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.718 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.718 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_prometheus": { 2024-03-14T10:04:38.718 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_prometheus", 2024-03-14T10:04:38.718 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.718 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.718 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.718 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-14T10:04:38.718 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.718 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.718 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.719 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-14T10:04:38.719 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.719 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.719 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.719 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_promtail": { 2024-03-14T10:04:38.719 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_promtail", 2024-03-14T10:04:38.719 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.719 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.719 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.719 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-14T10:04:38.719 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.719 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.719 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.719 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Promtail container image", 2024-03-14T10:04:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_snmp_gateway": { 2024-03-14T10:04:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_snmp_gateway", 2024-03-14T10:04:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-14T10:04:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SNMP Gateway container image", 2024-03-14T10:04:38.720 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "container_init": { 2024-03-14T10:04:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_init", 2024-03-14T10:04:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Run podman/docker with `--init`", 2024-03-14T10:04:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.721 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "daemon_cache_timeout": { 2024-03-14T10:04:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "daemon_cache_timeout", 2024-03-14T10:04:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-14T10:04:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-14T10:04:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.722 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "default_cephadm_command_timeout": { 2024-03-14T10:04:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_cephadm_command_timeout", 2024-03-14T10:04:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "900", 2024-03-14T10:04:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-14T10:04:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "default_registry": { 2024-03-14T10:04:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_registry", 2024-03-14T10:04:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io", 2024-03-14T10:04:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-14T10:04:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.724 INFO:teuthology.orchestra.run.smithi078.stdout: "device_cache_timeout": { 2024-03-14T10:04:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_cache_timeout", 2024-03-14T10:04:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1800", 2024-03-14T10:04:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache device inventory", 2024-03-14T10:04:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "device_enhanced_scan": { 2024-03-14T10:04:38.725 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_enhanced_scan", 2024-03-14T10:04:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-14T10:04:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "facts_cache_timeout": { 2024-03-14T10:04:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "facts_cache_timeout", 2024-03-14T10:04:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.726 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-14T10:04:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache host facts data", 2024-03-14T10:04:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "host_check_interval": { 2024-03-14T10:04:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "host_check_interval", 2024-03-14T10:04:38.727 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-14T10:04:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to perform a host check", 2024-03-14T10:04:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: "hw_monitoring": { 2024-03-14T10:04:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hw_monitoring", 2024-03-14T10:04:38.728 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.729 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.729 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.729 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.729 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.729 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.729 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.729 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-14T10:04:38.729 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.729 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.729 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.729 INFO:teuthology.orchestra.run.smithi078.stdout: "inventory_list_all": { 2024-03-14T10:04:38.729 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "inventory_list_all", 2024-03-14T10:04:38.729 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.729 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.730 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.730 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.730 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.730 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.730 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.730 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-14T10:04:38.730 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.730 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.730 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.730 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.730 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.730 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.730 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.730 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.731 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.731 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.731 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.731 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.731 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.731 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.731 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.731 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.731 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.731 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.731 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.731 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.731 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.731 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.731 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.732 INFO:teuthology.orchestra.run.smithi078.stdout: "log_refresh_metadata": { 2024-03-14T10:04:38.732 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_refresh_metadata", 2024-03-14T10:04:38.732 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.732 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.732 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.732 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.732 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.732 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.732 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.732 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-14T10:04:38.732 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.732 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.732 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.732 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.733 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.733 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.733 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.733 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.733 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.733 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.733 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.733 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.733 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-14T10:04:38.733 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.733 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.733 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.733 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.734 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.734 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.734 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.734 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.734 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.734 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.734 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.734 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.734 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.734 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.734 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.734 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.734 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.735 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.735 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.735 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.735 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.735 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.735 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.735 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.735 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.735 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.735 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.735 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.735 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.735 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.736 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.736 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.736 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.736 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.736 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.736 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.736 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-14T10:04:38.736 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-14T10:04:38.736 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.736 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.736 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.736 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.736 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.737 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.737 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.737 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-14T10:04:38.737 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.737 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.737 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.737 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-14T10:04:38.737 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-14T10:04:38.737 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.737 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.737 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.737 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "*", 2024-03-14T10:04:38.738 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.738 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.738 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.738 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-14T10:04:38.738 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.738 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.738 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.738 INFO:teuthology.orchestra.run.smithi078.stdout: "max_count_per_host": { 2024-03-14T10:04:38.738 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_count_per_host", 2024-03-14T10:04:38.738 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.738 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.738 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.738 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:38.739 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.739 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.739 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.739 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of daemons per service per host", 2024-03-14T10:04:38.739 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.739 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.739 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.739 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd_draining_count": { 2024-03-14T10:04:38.739 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_osd_draining_count", 2024-03-14T10:04:38.739 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.739 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.739 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.739 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:38.740 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.740 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.740 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.740 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-14T10:04:38.740 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.740 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.740 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.740 INFO:teuthology.orchestra.run.smithi078.stdout: "migration_current": { 2024-03-14T10:04:38.740 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "migration_current", 2024-03-14T10:04:38.740 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.740 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.740 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.740 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.741 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.741 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.741 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.741 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "internal - do not modify", 2024-03-14T10:04:38.741 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.741 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.741 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.741 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-14T10:04:38.741 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-14T10:04:38.741 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.741 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.741 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.741 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "root", 2024-03-14T10:04:38.741 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.742 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.742 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.742 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm-package", 2024-03-14T10:04:38.742 INFO:teuthology.orchestra.run.smithi078.stdout: "root" 2024-03-14T10:04:38.742 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.742 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "mode for remote execution of cephadm", 2024-03-14T10:04:38.742 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.742 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.742 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.742 INFO:teuthology.orchestra.run.smithi078.stdout: "oob_default_addr": { 2024-03-14T10:04:38.742 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "oob_default_addr", 2024-03-14T10:04:38.742 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.743 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.743 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.743 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "169.254.1.1", 2024-03-14T10:04:38.743 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.743 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.743 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.743 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-14T10:04:38.743 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.743 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.743 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.743 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_alerts_path": { 2024-03-14T10:04:38.743 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_alerts_path", 2024-03-14T10:04:38.743 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.744 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.744 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.744 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-14T10:04:38.744 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.744 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.744 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.744 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-14T10:04:38.744 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.744 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.744 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.744 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_insecure": { 2024-03-14T10:04:38.744 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_insecure", 2024-03-14T10:04:38.744 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.745 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.745 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.745 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.745 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.745 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.745 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.745 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-14T10:04:38.745 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.745 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.745 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.745 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_password": { 2024-03-14T10:04:38.745 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_password", 2024-03-14T10:04:38.745 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.745 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.745 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.746 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.746 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.746 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.746 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.746 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-14T10:04:38.746 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.746 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.746 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.746 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_url": { 2024-03-14T10:04:38.746 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_url", 2024-03-14T10:04:38.746 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.746 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.746 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.747 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.747 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.747 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.747 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.747 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-14T10:04:38.747 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.747 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.747 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.747 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_username": { 2024-03-14T10:04:38.747 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_username", 2024-03-14T10:04:38.747 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.747 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.747 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.748 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.748 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.748 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.748 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.748 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-14T10:04:38.748 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.748 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.748 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.748 INFO:teuthology.orchestra.run.smithi078.stdout: "secure_monitoring_stack": { 2024-03-14T10:04:38.748 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secure_monitoring_stack", 2024-03-14T10:04:38.748 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.749 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.749 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.749 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.749 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.749 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.749 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.749 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-14T10:04:38.749 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.749 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.749 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.749 INFO:teuthology.orchestra.run.smithi078.stdout: "service_discovery_port": { 2024-03-14T10:04:38.749 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "service_discovery_port", 2024-03-14T10:04:38.749 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.750 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.750 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.750 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8765", 2024-03-14T10:04:38.750 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.750 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.750 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.750 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "cephadm service discovery port", 2024-03-14T10:04:38.750 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.750 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.750 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.750 INFO:teuthology.orchestra.run.smithi078.stdout: "ssh_config_file": { 2024-03-14T10:04:38.750 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssh_config_file", 2024-03-14T10:04:38.751 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.751 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.751 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.751 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.751 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.751 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.751 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.751 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-14T10:04:38.751 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.751 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.751 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.751 INFO:teuthology.orchestra.run.smithi078.stdout: "use_agent": { 2024-03-14T10:04:38.751 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_agent", 2024-03-14T10:04:38.751 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.751 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.752 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.752 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.752 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.752 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.752 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.752 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-14T10:04:38.752 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.752 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.752 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.752 INFO:teuthology.orchestra.run.smithi078.stdout: "use_repo_digest": { 2024-03-14T10:04:38.752 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_repo_digest", 2024-03-14T10:04:38.752 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.752 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.752 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.752 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.753 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.753 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.753 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.753 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-14T10:04:38.753 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.753 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.753 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.753 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_failed_host_check": { 2024-03-14T10:04:38.753 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_failed_host_check", 2024-03-14T10:04:38.753 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.753 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.753 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.753 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.753 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.753 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.754 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.754 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if the host check fails", 2024-03-14T10:04:38.754 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.754 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.754 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.754 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_daemons": { 2024-03-14T10:04:38.754 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_daemons", 2024-03-14T10:04:38.754 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.754 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.754 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.754 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.754 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.754 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-14T10:04:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_hosts": { 2024-03-14T10:04:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_hosts", 2024-03-14T10:04:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.755 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.756 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-14T10:04:38.756 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.756 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.756 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.756 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.756 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.756 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crash", 2024-03-14T10:04:38.757 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.757 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.757 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.757 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.757 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.757 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.757 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.757 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.757 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.757 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.757 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.757 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.757 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.757 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.757 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.757 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.758 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.758 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.758 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.758 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.758 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.758 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.758 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.758 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.758 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.758 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.758 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.758 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.759 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.759 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.759 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.759 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.759 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.759 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.759 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.759 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.759 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.759 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.759 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.759 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.759 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.760 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.760 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.760 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.760 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.760 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.760 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.760 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.760 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.760 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.760 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.760 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.760 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.760 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.760 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.761 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.761 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.761 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.761 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.761 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.761 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.761 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.761 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.761 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.761 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.761 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.761 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.761 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.761 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.762 INFO:teuthology.orchestra.run.smithi078.stdout: "retain_interval": { 2024-03-14T10:04:38.762 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retain_interval", 2024-03-14T10:04:38.762 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.762 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.762 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.762 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "31536000", 2024-03-14T10:04:38.762 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.762 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.762 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.762 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-14T10:04:38.762 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.763 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.763 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.763 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_recent_interval": { 2024-03-14T10:04:38.763 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_recent_interval", 2024-03-14T10:04:38.763 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.763 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.763 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.763 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1209600", 2024-03-14T10:04:38.763 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.763 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.763 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.763 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-14T10:04:38.763 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.764 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.764 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.764 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dashboard", 2024-03-14T10:04:38.764 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.764 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.764 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.764 INFO:teuthology.orchestra.run.smithi078.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-14T10:04:38.764 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-14T10:04:38.764 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.764 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.765 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.765 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:38.765 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.765 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.765 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.765 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.765 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.765 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.765 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.765 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_HOST": { 2024-03-14T10:04:38.765 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-14T10:04:38.765 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.765 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.766 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.766 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.766 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.766 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.766 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.766 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.766 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.766 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.766 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.766 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-14T10:04:38.766 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-14T10:04:38.766 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.766 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_ENABLED": { 2024-03-14T10:04:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_ENABLED", 2024-03-14T10:04:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-14T10:04:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-14T10:04:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "ENABLE_BROWSABLE_API": { 2024-03-14T10:04:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-14T10:04:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-14T10:04:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-14T10:04:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-14T10:04:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-14T10:04:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-14T10:04:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-14T10:04:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-14T10:04:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-14T10:04:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-14T10:04:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-14T10:04:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-14T10:04:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-14T10:04:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-14T10:04:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-14T10:04:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-14T10:04:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-14T10:04:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_PASSWORD": { 2024-03-14T10:04:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-14T10:04:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-14T10:04:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-14T10:04:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-14T10:04:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.780 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.780 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.780 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.780 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.780 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.780 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.780 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.780 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.780 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_URL": { 2024-03-14T10:04:38.780 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_URL", 2024-03-14T10:04:38.780 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.780 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.781 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.781 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.781 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.781 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.781 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.781 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.781 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.781 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.781 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.781 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_USERNAME": { 2024-03-14T10:04:38.781 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-14T10:04:38.781 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.781 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.781 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.782 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-14T10:04:38.782 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.782 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.782 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.782 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.782 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.782 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.782 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.782 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-14T10:04:38.782 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-14T10:04:38.782 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.782 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.783 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.783 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.783 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.783 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.783 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.783 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.783 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.783 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.783 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.783 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-14T10:04:38.783 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-14T10:04:38.783 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.783 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.783 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.784 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.784 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.784 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.784 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.784 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.784 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.784 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.784 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.784 INFO:teuthology.orchestra.run.smithi078.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-14T10:04:38.784 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-14T10:04:38.784 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.784 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.784 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.784 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.785 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.785 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.785 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.785 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.785 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.785 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.785 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.785 INFO:teuthology.orchestra.run.smithi078.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-14T10:04:38.785 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-14T10:04:38.785 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.785 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.785 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.785 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.786 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.786 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.786 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.786 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.786 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.786 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.786 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.786 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_HOST": { 2024-03-14T10:04:38.786 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-14T10:04:38.786 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.786 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.786 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.786 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.786 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.787 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.787 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.787 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.787 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.787 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.787 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.787 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-14T10:04:38.787 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-14T10:04:38.787 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.787 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.787 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.787 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.788 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.788 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.788 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.788 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.788 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.788 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.788 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.788 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-14T10:04:38.788 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-14T10:04:38.788 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.788 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.788 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.789 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.789 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.789 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.789 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.789 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.789 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.789 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.789 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.789 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-14T10:04:38.789 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-14T10:04:38.789 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.789 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.789 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.790 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.790 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.790 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.790 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.790 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.790 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.790 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.790 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.790 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-14T10:04:38.790 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-14T10:04:38.790 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.790 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.791 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.791 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.791 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.791 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.791 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.791 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.791 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.791 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.791 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.791 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-14T10:04:38.791 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-14T10:04:38.792 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.792 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.792 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.792 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.792 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.792 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.792 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.792 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.792 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.792 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.792 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.792 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-14T10:04:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-14T10:04:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.793 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-14T10:04:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-14T10:04:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.794 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-14T10:04:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-14T10:04:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.795 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_ENABLED": { 2024-03-14T10:04:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-14T10:04:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.796 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-14T10:04:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-14T10:04:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-14T10:04:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.797 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-14T10:04:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-14T10:04:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-14T10:04:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-14T10:04:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8", 2024-03-14T10:04:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.799 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-14T10:04:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-14T10:04:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "45", 2024-03-14T10:04:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.800 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ACCESS_KEY": { 2024-03-14T10:04:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-14T10:04:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.801 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-14T10:04:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-14T10:04:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-14T10:04:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.802 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SECRET_KEY": { 2024-03-14T10:04:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-14T10:04:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.803 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SSL_VERIFY": { 2024-03-14T10:04:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-14T10:04:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.804 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "UNSAFE_TLS_v1_2": { 2024-03-14T10:04:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-14T10:04:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.805 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-14T10:04:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-14T10:04:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-14T10:04:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.806 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-14T10:04:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-14T10:04:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.807 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-14T10:04:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-14T10:04:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-14T10:04:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.808 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "cross_origin_url": { 2024-03-14T10:04:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cross_origin_url", 2024-03-14T10:04:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "crt_file": { 2024-03-14T10:04:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crt_file", 2024-03-14T10:04:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.810 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "debug": { 2024-03-14T10:04:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "debug", 2024-03-14T10:04:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable/disable debug options", 2024-03-14T10:04:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "jwt_token_ttl": { 2024-03-14T10:04:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "jwt_token_ttl", 2024-03-14T10:04:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "28800", 2024-03-14T10:04:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.812 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-14T10:04:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-14T10:04:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.813 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.814 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.815 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.816 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.817 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.818 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "motd": { 2024-03-14T10:04:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "motd", 2024-03-14T10:04:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.819 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "The message of the day", 2024-03-14T10:04:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_resolve_ip_addr": { 2024-03-14T10:04:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "redirect_resolve_ip_addr", 2024-03-14T10:04:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.820 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-14T10:04:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-14T10:04:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-14T10:04:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.821 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-14T10:04:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-14T10:04:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8080", 2024-03-14T10:04:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.822 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-14T10:04:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-14T10:04:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.823 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl_server_port": { 2024-03-14T10:04:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl_server_port", 2024-03-14T10:04:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8443", 2024-03-14T10:04:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.824 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-14T10:04:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-14T10:04:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "redirect", 2024-03-14T10:04:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.825 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect" 2024-03-14T10:04:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-14T10:04:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-14T10:04:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.826 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-14T10:04:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-14T10:04:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-14T10:04:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "url_prefix": { 2024-03-14T10:04:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url_prefix", 2024-03-14T10:04:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.827 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2024-03-14T10:04:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_monitoring": { 2024-03-14T10:04:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_monitoring", 2024-03-14T10:04:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.829 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "monitor device health metrics", 2024-03-14T10:04:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.830 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.831 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.832 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.833 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.834 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.835 INFO:teuthology.orchestra.run.smithi078.stdout: "mark_out_threshold": { 2024-03-14T10:04:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mark_out_threshold", 2024-03-14T10:04:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2419200", 2024-03-14T10:04:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-14T10:04:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": { 2024-03-14T10:04:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_name", 2024-03-14T10:04:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "device_health_metrics", 2024-03-14T10:04:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-14T10:04:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.837 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "retention_period": { 2024-03-14T10:04:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retention_period", 2024-03-14T10:04:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15552000", 2024-03-14T10:04:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain device health metrics", 2024-03-14T10:04:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.838 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_frequency": { 2024-03-14T10:04:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_frequency", 2024-03-14T10:04:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-14T10:04:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-14T10:04:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.839 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "self_heal": { 2024-03-14T10:04:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "self_heal", 2024-03-14T10:04:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-14T10:04:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-14T10:04:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-14T10:04:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-14T10:04:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and check device health", 2024-03-14T10:04:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.841 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_threshold": { 2024-03-14T10:04:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_threshold", 2024-03-14T10:04:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7257600", 2024-03-14T10:04:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-14T10:04:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "diskprediction_local", 2024-03-14T10:04:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.843 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.846 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.847 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "predict_interval": { 2024-03-14T10:04:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predict_interval", 2024-03-14T10:04:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-14T10:04:38.848 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "predictor_model": { 2024-03-14T10:04:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predictor_model", 2024-03-14T10:04:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "prophetstor", 2024-03-14T10:04:38.849 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-14T10:04:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-14T10:04:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-14T10:04:38.850 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "influx", 2024-03-14T10:04:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": false, 2024-03-14T10:04:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "influxdb python module not found", 2024-03-14T10:04:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.851 INFO:teuthology.orchestra.run.smithi078.stdout: "batch_size": { 2024-03-14T10:04:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "batch_size", 2024-03-14T10:04:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5000", 2024-03-14T10:04:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-14T10:04:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.852 INFO:teuthology.orchestra.run.smithi078.stdout: "database": { 2024-03-14T10:04:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "database", 2024-03-14T10:04:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "ceph", 2024-03-14T10:04:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-14T10:04:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": { 2024-03-14T10:04:38.853 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hostname", 2024-03-14T10:04:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server hostname", 2024-03-14T10:04:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-14T10:04:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-14T10:04:38.854 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "30", 2024-03-14T10:04:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "5", 2024-03-14T10:04:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-14T10:04:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.855 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.856 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.857 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.858 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.859 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.860 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "password": { 2024-03-14T10:04:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "password", 2024-03-14T10:04:38.861 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "password of InfluxDB server user", 2024-03-14T10:04:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "port": { 2024-03-14T10:04:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "port", 2024-03-14T10:04:38.862 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8086", 2024-03-14T10:04:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server port", 2024-03-14T10:04:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-14T10:04:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-14T10:04:38.863 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "false", 2024-03-14T10:04:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-14T10:04:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "threads": { 2024-03-14T10:04:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threads", 2024-03-14T10:04:38.864 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-14T10:04:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-14T10:04:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "32", 2024-03-14T10:04:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-14T10:04:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "username": { 2024-03-14T10:04:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "username", 2024-03-14T10:04:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.865 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "username of InfluxDB server user", 2024-03-14T10:04:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "verify_ssl": { 2024-03-14T10:04:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "verify_ssl", 2024-03-14T10:04:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.866 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "true", 2024-03-14T10:04:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-14T10:04:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "insights", 2024-03-14T10:04:38.867 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.868 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.869 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.870 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.871 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.872 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "iostat", 2024-03-14T10:04:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.873 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.874 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.875 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.876 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.878 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "k8sevents", 2024-03-14T10:04:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_event_retention_days": { 2024-03-14T10:04:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ceph_event_retention_days", 2024-03-14T10:04:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.879 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7", 2024-03-14T10:04:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-14T10:04:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "config_check_secs": { 2024-03-14T10:04:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_check_secs", 2024-03-14T10:04:38.880 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "10", 2024-03-14T10:04:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-14T10:04:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.881 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.882 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.883 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.884 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.885 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.886 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.887 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.887 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.887 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.887 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.887 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "localpool", 2024-03-14T10:04:38.887 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.887 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.887 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.887 INFO:teuthology.orchestra.run.smithi078.stdout: "failure_domain": { 2024-03-14T10:04:38.887 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "failure_domain", 2024-03-14T10:04:38.888 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.888 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.888 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.888 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "host", 2024-03-14T10:04:38.888 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.888 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.888 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.888 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "failure domain for any created local pool", 2024-03-14T10:04:38.888 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-14T10:04:38.888 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.888 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.888 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.888 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.889 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.889 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.889 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.889 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.889 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.889 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.889 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.889 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.889 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.889 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.889 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.889 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.889 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.890 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.890 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.890 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.890 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.890 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.890 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.890 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.890 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.890 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.890 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.890 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.890 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.890 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.891 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.891 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.891 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.891 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.891 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.891 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.891 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.891 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.891 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.891 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.891 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.891 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.891 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.891 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.892 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.892 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.892 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.892 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.892 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.892 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.892 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.892 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.892 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.892 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.892 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.892 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.892 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.893 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.893 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.893 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.893 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.893 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.893 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.893 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.893 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.893 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.893 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.893 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.893 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": { 2024-03-14T10:04:38.893 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_size", 2024-03-14T10:04:38.894 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.894 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.894 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.894 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.894 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.894 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.894 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.894 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default min_size for any created local pool", 2024-03-14T10:04:38.894 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-14T10:04:38.894 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.894 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.894 INFO:teuthology.orchestra.run.smithi078.stdout: "num_rep": { 2024-03-14T10:04:38.895 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "num_rep", 2024-03-14T10:04:38.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.895 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.895 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.895 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3", 2024-03-14T10:04:38.895 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.895 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.895 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.895 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default replica count for any created local pool", 2024-03-14T10:04:38.895 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.895 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.895 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.895 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": { 2024-03-14T10:04:38.896 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_num", 2024-03-14T10:04:38.896 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.896 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.896 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.896 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "128", 2024-03-14T10:04:38.896 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.896 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.896 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.896 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default pg_num for any created local pool", 2024-03-14T10:04:38.896 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.896 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.896 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.896 INFO:teuthology.orchestra.run.smithi078.stdout: "prefix": { 2024-03-14T10:04:38.897 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prefix", 2024-03-14T10:04:38.897 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.897 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.897 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.897 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.897 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.897 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.897 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.897 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name prefix for any created local pool", 2024-03-14T10:04:38.897 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.897 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.897 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.897 INFO:teuthology.orchestra.run.smithi078.stdout: "subtree": { 2024-03-14T10:04:38.898 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "subtree", 2024-03-14T10:04:38.898 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.898 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.898 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.898 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "rack", 2024-03-14T10:04:38.898 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.898 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.898 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.898 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-14T10:04:38.898 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-14T10:04:38.898 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.898 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.899 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mds_autoscaler", 2024-03-14T10:04:38.899 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.899 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.899 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.899 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.899 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.899 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.899 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.899 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.899 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.900 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.900 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.900 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.900 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.900 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.900 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.900 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.900 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.900 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.900 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.900 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.900 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.900 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.901 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.901 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.901 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.901 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.901 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.901 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.901 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.901 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.901 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.901 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.901 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.901 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.901 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.902 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.902 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.902 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.902 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.902 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.902 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.902 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.902 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.902 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.902 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.902 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.903 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.905 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirroring", 2024-03-14T10:04:38.905 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.905 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.905 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.905 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.905 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.905 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.905 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.905 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.905 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.905 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.905 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.905 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.905 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.906 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.906 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.906 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.906 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.906 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.906 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.906 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.906 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.906 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.906 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.906 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.906 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.906 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.907 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.907 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.907 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.907 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.907 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.907 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.907 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.907 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.907 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.907 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.907 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.907 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.908 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.908 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.908 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.908 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.908 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.908 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.908 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.908 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.908 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.908 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.908 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.908 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.908 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.908 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.909 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.910 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.910 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.910 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.910 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "nfs", 2024-03-14T10:04:38.910 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.910 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.910 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.910 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.910 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.910 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.911 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.911 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.911 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.911 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.911 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.911 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.911 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.911 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.911 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.911 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.911 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.911 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.911 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.911 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.912 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.912 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.912 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.912 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.912 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.912 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.912 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.912 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.912 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.912 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.912 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.912 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.913 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.913 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.913 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.913 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.913 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.913 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.913 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.913 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.913 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.913 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.913 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.913 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.913 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.913 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.914 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.914 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.914 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.914 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.914 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.914 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.914 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.914 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.914 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.914 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.914 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.914 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.915 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.915 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.915 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.915 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.915 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.915 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.915 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.915 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.915 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.915 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.916 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-14T10:04:38.916 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.916 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.916 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.916 INFO:teuthology.orchestra.run.smithi078.stdout: "fail_fs": { 2024-03-14T10:04:38.916 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "fail_fs", 2024-03-14T10:04:38.916 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.916 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.916 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.916 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.916 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.916 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.916 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-14T10:04:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.919 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.919 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.919 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.919 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.919 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.919 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.919 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.919 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.919 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.919 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.919 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.919 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.919 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.920 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.920 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.920 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.920 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.920 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.920 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.920 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.920 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.920 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.920 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.920 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.920 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.920 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.921 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.921 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.921 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.921 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.921 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.921 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.921 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.921 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.921 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.921 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.921 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.921 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.921 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.922 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.922 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.922 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.922 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator": { 2024-03-14T10:04:38.922 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-14T10:04:38.922 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.922 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.922 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.922 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.922 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.922 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.922 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.923 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2024-03-14T10:04:38.923 INFO:teuthology.orchestra.run.smithi078.stdout: "rook", 2024-03-14T10:04:38.923 INFO:teuthology.orchestra.run.smithi078.stdout: "test_orchestrator" 2024-03-14T10:04:38.923 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.923 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Orchestrator backend", 2024-03-14T10:04:38.923 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.923 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.923 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.923 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.923 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.923 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_perf_query", 2024-03-14T10:04:38.924 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.924 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.924 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.924 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.924 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.924 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.924 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.924 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.924 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.924 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.924 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.924 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.924 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.924 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.925 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.925 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.925 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.925 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.925 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.925 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.925 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.925 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.925 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.925 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.925 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.925 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.925 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.925 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.926 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.926 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.926 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.926 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.926 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.926 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.926 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.926 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.926 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.926 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.926 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.926 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.927 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.927 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.927 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.927 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.927 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.927 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.927 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.927 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.927 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.927 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.927 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.927 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.927 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.928 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.928 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.928 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.928 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.928 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.928 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.928 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.928 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.928 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.928 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.928 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.928 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.928 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.929 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.929 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.929 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.929 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_support", 2024-03-14T10:04:38.929 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.929 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.929 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.929 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.929 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.929 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.930 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.930 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.930 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.930 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.930 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.930 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.930 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.930 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.930 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.930 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.930 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.930 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.931 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.931 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.931 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.931 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.931 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.931 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.931 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.931 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.931 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.931 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.931 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.931 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.932 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.932 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.932 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.932 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.932 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.932 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.932 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.932 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.932 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.932 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.932 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_autoscaler", 2024-03-14T10:04:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-14T10:04:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-14T10:04:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-14T10:04:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "threshold": { 2024-03-14T10:04:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threshold", 2024-03-14T10:04:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-14T10:04:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1.0", 2024-03-14T10:04:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "scaling threshold", 2024-03-14T10:04:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-14T10:04:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "progress", 2024-03-14T10:04:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_pg_recovery_event": { 2024-03-14T10:04:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_pg_recovery_event", 2024-03-14T10:04:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-14T10:04:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-14T10:04:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-14T10:04:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.946 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.946 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.946 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.946 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.946 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.946 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.946 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.946 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.946 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.946 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.946 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.946 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.946 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.947 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.947 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.947 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.947 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.947 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.947 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.947 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.947 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.947 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.947 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.947 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.947 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.947 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.947 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.948 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.948 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.948 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.948 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.948 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.948 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.948 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.948 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.948 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.948 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.948 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.948 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.948 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.949 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.949 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.949 INFO:teuthology.orchestra.run.smithi078.stdout: "max_completed_events": { 2024-03-14T10:04:38.949 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_completed_events", 2024-03-14T10:04:38.949 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.949 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.949 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.949 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "50", 2024-03-14T10:04:38.949 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.949 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.949 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.949 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "number of past completed events to remember", 2024-03-14T10:04:38.949 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.950 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.950 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.950 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-14T10:04:38.950 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-14T10:04:38.950 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:38.950 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.950 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.950 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-14T10:04:38.950 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.950 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.950 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.950 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long the module is going to sleep", 2024-03-14T10:04:38.950 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus", 2024-03-14T10:04:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "cache": { 2024-03-14T10:04:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cache", 2024-03-14T10:04:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.951 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "exclude_perf_counters": { 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "exclude_perf_counters", 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.952 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-14T10:04:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-14T10:04:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.953 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.954 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.956 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.957 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools": { 2024-03-14T10:04:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools", 2024-03-14T10:04:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.958 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-14T10:04:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-14T10:04:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300", 2024-03-14T10:04:38.959 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_interval": { 2024-03-14T10:04:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_interval", 2024-03-14T10:04:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15.0", 2024-03-14T10:04:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.960 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-14T10:04:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-14T10:04:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-14T10:04:38.961 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.962 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.962 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.962 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-14T10:04:38.962 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.962 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.962 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.962 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-14T10:04:38.962 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-14T10:04:38.962 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.962 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.962 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.962 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "9283", 2024-03-14T10:04:38.962 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.963 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.963 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.963 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-14T10:04:38.963 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.963 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.963 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.963 INFO:teuthology.orchestra.run.smithi078.stdout: "stale_cache_strategy": { 2024-03-14T10:04:38.963 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stale_cache_strategy", 2024-03-14T10:04:38.963 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.963 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.963 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.963 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "log", 2024-03-14T10:04:38.963 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.963 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.964 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.964 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.964 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.964 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.964 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.964 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-14T10:04:38.964 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-14T10:04:38.964 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.964 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.964 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.964 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "default", 2024-03-14T10:04:38.964 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.964 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.964 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.965 INFO:teuthology.orchestra.run.smithi078.stdout: "default", 2024-03-14T10:04:38.965 INFO:teuthology.orchestra.run.smithi078.stdout: "error" 2024-03-14T10:04:38.965 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.965 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.965 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.965 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.965 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.965 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-14T10:04:38.965 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-14T10:04:38.965 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.965 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.965 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.965 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-14T10:04:38.966 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-14T10:04:38.966 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-14T10:04:38.966 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.966 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.966 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.966 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.966 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.966 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.966 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.966 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2024-03-14T10:04:38.966 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.966 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.967 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.967 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.967 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.967 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.967 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.967 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.967 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.967 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.967 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.967 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.967 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.967 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.967 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.967 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.968 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.968 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.968 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.968 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.968 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.968 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.968 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.968 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.968 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.968 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.968 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.968 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.968 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.969 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.969 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.969 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.969 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.969 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.969 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.969 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.969 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.969 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.969 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.969 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.969 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.969 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.970 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.970 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.970 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.970 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.970 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.970 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.970 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.970 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.970 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.970 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.970 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.970 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.970 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.970 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.971 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.971 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.971 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.971 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.971 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.971 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.971 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.971 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.971 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.971 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.971 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.971 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.971 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.972 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_snap_create": { 2024-03-14T10:04:38.972 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_snap_create", 2024-03-14T10:04:38.972 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:38.972 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.972 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.972 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:38.972 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.972 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.972 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.972 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.972 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.972 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.972 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.973 INFO:teuthology.orchestra.run.smithi078.stdout: "mirror_snapshot_schedule": { 2024-03-14T10:04:38.973 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirror_snapshot_schedule", 2024-03-14T10:04:38.973 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.973 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.973 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.973 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.973 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.973 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.973 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.973 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.973 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.973 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.973 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.974 INFO:teuthology.orchestra.run.smithi078.stdout: "trash_purge_schedule": { 2024-03-14T10:04:38.974 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "trash_purge_schedule", 2024-03-14T10:04:38.974 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.974 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.974 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.974 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.974 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.974 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.974 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.974 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.974 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.974 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.975 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.975 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.975 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "restful", 2024-03-14T10:04:38.975 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.975 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.975 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.975 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_auth": { 2024-03-14T10:04:38.975 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_auth", 2024-03-14T10:04:38.975 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.975 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.975 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.975 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:38.975 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.976 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.976 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.976 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.976 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.976 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.976 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.976 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-14T10:04:38.976 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-14T10:04:38.976 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.976 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.976 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.976 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.977 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.978 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.978 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.978 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.978 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.978 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.978 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.978 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.978 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.978 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.978 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.978 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.978 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.978 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.978 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.979 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.980 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.980 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.980 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.980 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.980 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.980 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.980 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.980 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.980 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.980 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.981 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.981 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.981 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.981 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.981 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.981 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.981 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.981 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.981 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.981 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.981 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.981 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.981 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.981 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.982 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.982 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.982 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.982 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.982 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.982 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.982 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.982 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.982 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-14T10:04:38.982 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-14T10:04:38.982 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.982 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.983 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.983 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.983 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.983 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.983 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.983 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.983 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.983 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.983 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.983 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-14T10:04:38.983 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-14T10:04:38.983 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.983 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.983 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.984 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.984 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.984 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.984 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.984 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.984 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.984 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.984 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.984 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw", 2024-03-14T10:04:38.984 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.984 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.985 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.985 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.985 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.985 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.985 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.985 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.985 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.985 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.985 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.985 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.985 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.985 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.985 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.985 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.985 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.985 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.986 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.986 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.986 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.986 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.986 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.986 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.986 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.986 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.986 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.986 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.986 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.986 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.986 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.987 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.987 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.987 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.987 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.987 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.987 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.987 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.987 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.987 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.987 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.987 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.987 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.987 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.987 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.988 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.988 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.988 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.988 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.988 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.988 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.988 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.988 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.988 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.988 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.988 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.988 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.988 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.989 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.989 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.989 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.989 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.989 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.989 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.989 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.989 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.989 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.989 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.989 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.989 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rook", 2024-03-14T10:04:38.990 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.990 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.990 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.990 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.990 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.990 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.990 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.990 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.990 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.990 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.990 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.990 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.990 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.990 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.990 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.991 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.991 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.991 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.991 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.991 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.991 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.991 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.991 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.991 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.991 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.991 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.991 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.991 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.991 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.992 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.993 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.993 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.993 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.993 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.993 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.993 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.993 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.993 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.993 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.993 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.993 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.993 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.993 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.994 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:38.994 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:38.994 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.994 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.994 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.994 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.994 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.994 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.994 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.994 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.994 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.994 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.994 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.995 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_class": { 2024-03-14T10:04:38.995 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "storage_class", 2024-03-14T10:04:38.995 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.995 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.995 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:38.995 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "local", 2024-03-14T10:04:38.995 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.995 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.995 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.995 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-14T10:04:38.995 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.995 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.995 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:38.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:38.996 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "selftest", 2024-03-14T10:04:38.996 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:38.996 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:38.996 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:38.996 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:38.996 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:38.996 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.996 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.996 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.996 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:38.997 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.997 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.997 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.997 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:38.997 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:38.997 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:38.997 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:38.997 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:38.997 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:38.997 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:38.997 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.997 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.997 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.997 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.998 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:38.998 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:38.998 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:38.998 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.998 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.998 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:38.998 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.998 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.998 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:38.998 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:38.998 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:38.999 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:38.999 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:38.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:38.999 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:38.999 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:38.999 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:38.999 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:38.999 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:38.999 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:38.999 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:38.999 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:38.999 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:38.999 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.000 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.000 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.000 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.000 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.000 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.000 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.000 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.000 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.000 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.000 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.000 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.000 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.000 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.001 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.001 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.001 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.001 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.001 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.001 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.001 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.001 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.001 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.001 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.001 INFO:teuthology.orchestra.run.smithi078.stdout: "roption1": { 2024-03-14T10:04:39.001 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption1", 2024-03-14T10:04:39.001 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.001 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.002 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.002 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.002 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.002 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.002 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.002 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.002 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.002 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.002 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.002 INFO:teuthology.orchestra.run.smithi078.stdout: "roption2": { 2024-03-14T10:04:39.002 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption2", 2024-03-14T10:04:39.002 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.002 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.003 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.003 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "xyz", 2024-03-14T10:04:39.003 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.003 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.003 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.003 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.003 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.003 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.003 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.003 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption1": { 2024-03-14T10:04:39.003 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption1", 2024-03-14T10:04:39.003 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.003 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.004 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.004 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.004 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.004 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.004 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.004 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.004 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.004 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.004 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.004 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption2": { 2024-03-14T10:04:39.004 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption2", 2024-03-14T10:04:39.004 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.004 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.004 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.005 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.005 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.005 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.005 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.005 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.005 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.005 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.005 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.005 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption3": { 2024-03-14T10:04:39.005 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption3", 2024-03-14T10:04:39.005 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:39.005 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.005 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.005 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption4": { 2024-03-14T10:04:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption4", 2024-03-14T10:04:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.006 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption5": { 2024-03-14T10:04:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption5", 2024-03-14T10:04:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.007 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption6": { 2024-03-14T10:04:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption6", 2024-03-14T10:04:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.008 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption7": { 2024-03-14T10:04:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption7", 2024-03-14T10:04:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.009 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-14T10:04:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "42", 2024-03-14T10:04:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: "testkey": { 2024-03-14T10:04:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testkey", 2024-03-14T10:04:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.010 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "testlkey": { 2024-03-14T10:04:39.011 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testlkey", 2024-03-14T10:04:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "testnewline": { 2024-03-14T10:04:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testnewline", 2024-03-14T10:04:39.012 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.013 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-03-14T10:04:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_m_granularity": { 2024-03-14T10:04:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_m_granularity", 2024-03-14T10:04:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.014 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow minute scheduled snapshots", 2024-03-14T10:04:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "dump_on_update": { 2024-03-14T10:04:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dump_on_update", 2024-03-14T10:04:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.015 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "dump database to debug log on update", 2024-03-14T10:04:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.016 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.017 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.017 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.017 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.017 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.017 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.017 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.017 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.017 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.017 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.017 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.017 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.017 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.018 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.018 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.018 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.018 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.018 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.018 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.018 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.018 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.018 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.018 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.018 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.018 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.018 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.019 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.019 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.019 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.019 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.019 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.019 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.019 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.019 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.019 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.019 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.019 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.019 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.019 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.019 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.020 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.020 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.020 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.020 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.020 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.020 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.020 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.020 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.020 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.020 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.020 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.020 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.020 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.021 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.021 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.021 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.021 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.021 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.021 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.021 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stats", 2024-03-14T10:04:39.021 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.021 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.021 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.021 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.022 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.022 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.022 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.022 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.022 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.022 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.022 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.022 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.022 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.022 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.022 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.022 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.023 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.023 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.023 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.023 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.023 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.023 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.023 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.023 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.023 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.023 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.023 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.023 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.023 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.023 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.023 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.024 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.024 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.024 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.024 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.024 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.024 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.024 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.024 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.024 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.024 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.024 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.024 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.024 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.024 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.024 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.025 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.025 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.025 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.025 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.025 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.025 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.025 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.025 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.025 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.025 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.025 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.025 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.025 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.026 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.026 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.026 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.026 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.026 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.026 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.026 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.026 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.026 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.026 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.027 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "status", 2024-03-14T10:04:39.027 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.027 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.027 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.027 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.027 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.027 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.027 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.027 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.027 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.027 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.027 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.027 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.027 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.027 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.027 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.028 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.028 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.028 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.028 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.028 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.028 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.028 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.028 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.028 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.028 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.028 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.028 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.028 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.029 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.029 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.029 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.029 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.029 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.029 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.029 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.029 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.029 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.029 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.029 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.029 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.029 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.030 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.030 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.030 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.030 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.030 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.030 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.030 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.030 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.030 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.030 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.030 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.030 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.030 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.030 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.031 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.031 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.031 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.031 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.031 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.031 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.031 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.031 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.031 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.031 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.031 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.031 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.031 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.031 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telegraf", 2024-03-14T10:04:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "address": { 2024-03-14T10:04:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "address", 2024-03-14T10:04:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-14T10:04:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.032 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.033 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.033 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.033 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.033 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.033 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.033 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-14T10:04:39.033 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-14T10:04:39.033 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.033 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.033 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.033 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15", 2024-03-14T10:04:39.033 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.033 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.033 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.033 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.034 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.034 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.034 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.034 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.034 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.034 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.034 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.034 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.034 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.034 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.034 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.034 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.034 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.034 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.035 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.035 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.035 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.035 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.035 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.035 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.035 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.035 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.035 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.035 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.035 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.035 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.035 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.035 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.035 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.036 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.036 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.036 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.036 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.036 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.036 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.036 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.036 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.036 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.036 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.036 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.036 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.036 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.036 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.037 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.037 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.037 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.037 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.037 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.037 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.037 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.037 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.037 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.037 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.037 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.037 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.037 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.038 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.038 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.038 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.038 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.038 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.038 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.038 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.038 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.038 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.038 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.038 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.038 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.038 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.039 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telemetry", 2024-03-14T10:04:39.039 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.039 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.039 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.039 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_basic": { 2024-03-14T10:04:39.039 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_basic", 2024-03-14T10:04:39.039 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.039 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.039 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.040 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.040 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.040 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.040 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.040 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-14T10:04:39.040 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.040 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.040 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.040 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_crash": { 2024-03-14T10:04:39.040 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_crash", 2024-03-14T10:04:39.040 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.040 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.040 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.041 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.041 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.041 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.041 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.041 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-14T10:04:39.041 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.041 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.041 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.041 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_device": { 2024-03-14T10:04:39.041 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_device", 2024-03-14T10:04:39.041 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.041 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.041 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.041 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.042 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.042 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.042 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.042 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-14T10:04:39.042 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.042 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.042 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.042 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_ident": { 2024-03-14T10:04:39.042 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_ident", 2024-03-14T10:04:39.042 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.042 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.042 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.042 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.043 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.043 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.043 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.043 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-14T10:04:39.043 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.043 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.043 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.043 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_perf": { 2024-03-14T10:04:39.043 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_perf", 2024-03-14T10:04:39.043 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.043 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.043 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.044 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.044 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.044 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.044 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.044 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-14T10:04:39.044 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.044 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.044 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.044 INFO:teuthology.orchestra.run.smithi078.stdout: "contact": { 2024-03-14T10:04:39.044 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "contact", 2024-03-14T10:04:39.044 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.044 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.044 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.045 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.045 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.045 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.045 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.045 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.045 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.045 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.045 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.045 INFO:teuthology.orchestra.run.smithi078.stdout: "description": { 2024-03-14T10:04:39.045 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "description", 2024-03-14T10:04:39.045 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.046 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.046 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.046 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.046 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.046 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.046 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.046 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.046 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.046 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.046 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.046 INFO:teuthology.orchestra.run.smithi078.stdout: "device_url": { 2024-03-14T10:04:39.047 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_url", 2024-03-14T10:04:39.047 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.047 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.047 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.047 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-14T10:04:39.047 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.047 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.047 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.047 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.047 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.047 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.047 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.047 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-14T10:04:39.047 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-14T10:04:39.048 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.048 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.048 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.048 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.048 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.048 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.048 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.048 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.048 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.048 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.048 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.048 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-14T10:04:39.049 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-14T10:04:39.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.049 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.049 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.049 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "24", 2024-03-14T10:04:39.049 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "8", 2024-03-14T10:04:39.049 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.049 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.049 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.049 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.049 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.049 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.050 INFO:teuthology.orchestra.run.smithi078.stdout: "last_opt_revision": { 2024-03-14T10:04:39.050 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "last_opt_revision", 2024-03-14T10:04:39.050 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.050 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.050 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.050 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1", 2024-03-14T10:04:39.050 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.050 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.050 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.050 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.050 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.050 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.050 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.051 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard": { 2024-03-14T10:04:39.051 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard", 2024-03-14T10:04:39.051 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.051 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.051 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.051 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.051 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.051 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.051 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.051 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.051 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.051 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.051 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.052 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard_description": { 2024-03-14T10:04:39.052 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard_description", 2024-03-14T10:04:39.052 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.052 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.052 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.052 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.052 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.052 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.052 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.052 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.052 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.052 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.052 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.053 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.053 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.053 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.053 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.053 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.053 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.053 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.053 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.053 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.053 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.053 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.053 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.053 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.053 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.053 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.054 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.054 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.054 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.054 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.054 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.054 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.054 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.054 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.054 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.054 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.054 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.054 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.055 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.055 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.055 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.055 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.055 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.055 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.055 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.055 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.055 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.055 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.055 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.055 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.055 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.056 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.056 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.056 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.056 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.056 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.056 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.056 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.056 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.056 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.056 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.056 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.056 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.056 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.056 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.057 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.057 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.057 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.057 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.057 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.057 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.057 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.057 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.057 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.057 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.057 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.057 INFO:teuthology.orchestra.run.smithi078.stdout: "organization": { 2024-03-14T10:04:39.057 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "organization", 2024-03-14T10:04:39.058 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.058 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.058 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.058 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.058 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.058 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.058 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.058 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.058 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.058 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.058 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.058 INFO:teuthology.orchestra.run.smithi078.stdout: "proxy": { 2024-03-14T10:04:39.058 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "proxy", 2024-03-14T10:04:39.059 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.059 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.059 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.059 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.059 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.059 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.059 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.059 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.059 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.059 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.059 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.059 INFO:teuthology.orchestra.run.smithi078.stdout: "url": { 2024-03-14T10:04:39.059 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url", 2024-03-14T10:04:39.059 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.060 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.060 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.060 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-14T10:04:39.060 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.060 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.060 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.060 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.060 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.060 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.060 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test_orchestrator", 2024-03-14T10:04:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.061 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.062 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.063 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.063 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.063 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.063 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.063 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.063 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.063 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.063 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.063 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.063 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.063 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.063 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.063 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.064 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.064 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.064 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.064 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.064 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.064 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.064 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.064 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.064 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.064 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.064 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.064 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.064 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.064 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.065 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.065 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.065 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.065 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.065 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.065 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.065 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.065 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.065 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.065 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.065 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.065 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.065 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2024-03-14T10:04:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.066 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.067 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.068 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.068 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.068 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.068 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.068 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.068 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.068 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.068 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.068 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.068 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.068 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.069 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.069 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.069 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.069 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.069 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.069 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.069 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.069 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.069 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.069 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.069 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.069 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.070 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.070 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.070 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.070 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.070 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.070 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.070 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.070 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.070 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.070 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.070 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.071 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.071 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.071 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.071 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.071 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.071 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.071 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.071 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.071 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.071 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.071 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.071 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.071 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.071 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.072 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.072 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_clones": { 2024-03-14T10:04:39.072 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_clones", 2024-03-14T10:04:39.072 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.072 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.072 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.072 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4", 2024-03-14T10:04:39.072 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.072 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.072 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.072 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-14T10:04:39.072 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.073 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.073 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.073 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_delay": { 2024-03-14T10:04:39.073 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_delay", 2024-03-14T10:04:39.073 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.073 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.073 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.073 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-14T10:04:39.073 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.073 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.073 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.073 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-14T10:04:39.073 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.074 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.074 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.074 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix", 2024-03-14T10:04:39.074 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.074 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.074 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.074 INFO:teuthology.orchestra.run.smithi078.stdout: "discovery_interval": { 2024-03-14T10:04:39.074 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "discovery_interval", 2024-03-14T10:04:39.074 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-14T10:04:39.075 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.075 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.075 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "100", 2024-03-14T10:04:39.075 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.075 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.075 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.075 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.075 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.075 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.075 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.075 INFO:teuthology.orchestra.run.smithi078.stdout: "identifier": { 2024-03-14T10:04:39.075 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "identifier", 2024-03-14T10:04:39.075 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.076 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.076 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.076 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.076 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.076 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.076 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.076 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.076 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.076 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.076 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.076 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-14T10:04:39.076 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-14T10:04:39.076 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.076 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.077 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.077 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-14T10:04:39.077 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.077 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.077 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.077 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.077 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.077 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.077 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.077 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.077 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.078 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.078 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.078 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.078 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.078 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.078 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.078 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.078 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.078 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.078 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.078 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.078 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.078 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.079 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.079 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.079 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.079 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.079 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.079 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.079 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.079 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.079 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.079 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.079 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.079 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.079 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.079 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.080 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.080 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.080 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.080 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.080 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.080 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.080 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.080 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.080 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.080 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.080 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.080 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.081 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.081 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.081 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.081 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.081 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.081 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.081 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.081 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.081 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.081 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.081 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.081 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.081 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.081 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.082 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.082 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.082 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.082 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.082 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.082 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.082 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.082 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.082 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.082 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.082 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.083 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_host": { 2024-03-14T10:04:39.083 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_host", 2024-03-14T10:04:39.083 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.083 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.083 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.083 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.083 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.083 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.083 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.083 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.083 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.083 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.083 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.083 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_port": { 2024-03-14T10:04:39.084 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_port", 2024-03-14T10:04:39.084 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.084 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.084 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.084 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10051", 2024-03-14T10:04:39.084 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.084 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.084 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.084 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.084 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.084 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.084 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.084 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_sender": { 2024-03-14T10:04:39.084 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_sender", 2024-03-14T10:04:39.085 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.085 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.085 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.085 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-14T10:04:39.085 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.085 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.085 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.085 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.085 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.085 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.085 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.085 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-14T10:04:39.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.086 INFO:teuthology.orchestra.run.smithi078.stdout: "gid": 25549, 2024-03-14T10:04:39.086 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "y", 2024-03-14T10:04:39.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_features": 4540138322906710015, 2024-03-14T10:04:39.086 INFO:teuthology.orchestra.run.smithi078.stdout: "available_modules": [ 2024-03-14T10:04:39.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.086 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alerts", 2024-03-14T10:04:39.086 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.086 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.086 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.086 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-14T10:04:39.086 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-14T10:04:39.087 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.087 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.087 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.087 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-14T10:04:39.087 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.087 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.087 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.087 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How frequently to reexamine health status", 2024-03-14T10:04:39.087 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.087 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.087 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.087 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.088 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.088 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.088 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.088 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.088 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.088 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.088 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.088 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.088 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.088 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.088 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.088 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.088 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.088 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.089 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.089 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.089 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.089 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.089 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.089 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.089 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.089 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.089 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.089 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.089 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.089 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.089 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.089 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.090 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.090 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.090 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.090 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.090 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.090 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.090 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.090 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.090 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.090 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.090 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.090 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.090 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.091 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.091 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.091 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.091 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.091 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.091 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.091 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.091 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.091 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.091 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.091 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.091 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.091 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.092 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.092 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.092 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.092 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.092 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.092 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.092 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.092 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.092 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.092 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.092 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.092 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_destination": { 2024-03-14T10:04:39.092 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_destination", 2024-03-14T10:04:39.093 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.093 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.093 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.093 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.093 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.093 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.093 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.093 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email address to send alerts to", 2024-03-14T10:04:39.093 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.093 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.093 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.093 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_from_name": { 2024-03-14T10:04:39.093 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_from_name", 2024-03-14T10:04:39.094 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.094 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.094 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.094 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "Ceph", 2024-03-14T10:04:39.094 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.094 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.094 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.094 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email From: name", 2024-03-14T10:04:39.094 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.094 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.094 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.094 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_host": { 2024-03-14T10:04:39.095 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_host", 2024-03-14T10:04:39.095 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.095 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.095 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.095 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.095 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.095 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.095 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.095 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP server", 2024-03-14T10:04:39.095 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.095 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.095 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.096 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_password": { 2024-03-14T10:04:39.096 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_password", 2024-03-14T10:04:39.096 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.096 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.096 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.096 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.096 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.096 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.096 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.096 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Password to authenticate with", 2024-03-14T10:04:39.096 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.096 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.096 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.097 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_port": { 2024-03-14T10:04:39.097 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_port", 2024-03-14T10:04:39.097 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.097 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.097 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.097 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "465", 2024-03-14T10:04:39.097 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.097 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.097 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.097 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP port", 2024-03-14T10:04:39.097 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.097 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.097 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.098 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_sender": { 2024-03-14T10:04:39.098 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_sender", 2024-03-14T10:04:39.098 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.098 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.098 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.098 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.098 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.098 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.098 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.098 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP envelope sender", 2024-03-14T10:04:39.098 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.098 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.098 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.099 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_ssl": { 2024-03-14T10:04:39.099 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_ssl", 2024-03-14T10:04:39.099 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.099 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.099 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.099 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.099 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.099 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.099 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.099 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-14T10:04:39.099 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.099 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.099 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.100 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_user": { 2024-03-14T10:04:39.100 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_user", 2024-03-14T10:04:39.100 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.100 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.100 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.100 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.100 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.100 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.100 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.100 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "User to authenticate as", 2024-03-14T10:04:39.100 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.100 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.100 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "balancer", 2024-03-14T10:04:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "active": { 2024-03-14T10:04:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "active", 2024-03-14T10:04:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.101 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.102 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.102 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically balance PGs across cluster", 2024-03-14T10:04:39.102 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.102 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.102 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.102 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_time": { 2024-03-14T10:04:39.102 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_time", 2024-03-14T10:04:39.102 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.102 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.102 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.102 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0000", 2024-03-14T10:04:39.102 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.102 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.103 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.103 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "beginning time of day to automatically balance", 2024-03-14T10:04:39.103 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-14T10:04:39.103 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.103 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.103 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_weekday": { 2024-03-14T10:04:39.103 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_weekday", 2024-03-14T10:04:39.103 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-14T10:04:39.103 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.103 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.103 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-14T10:04:39.103 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-14T10:04:39.103 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-14T10:04:39.104 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.104 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-14T10:04:39.104 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-14T10:04:39.104 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.104 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.104 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_max_iterations": { 2024-03-14T10:04:39.104 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_max_iterations", 2024-03-14T10:04:39.104 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-14T10:04:39.104 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.104 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.104 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "25", 2024-03-14T10:04:39.104 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-14T10:04:39.105 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "250", 2024-03-14T10:04:39.105 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.105 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-14T10:04:39.105 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.105 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.105 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.105 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_metrics": { 2024-03-14T10:04:39.105 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_metrics", 2024-03-14T10:04:39.105 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.105 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.105 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.105 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "pgs,objects,bytes", 2024-03-14T10:04:39.105 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.106 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.106 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.106 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-14T10:04:39.106 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-14T10:04:39.106 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.106 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.106 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_step": { 2024-03-14T10:04:39.106 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_step", 2024-03-14T10:04:39.106 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:39.106 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.106 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.106 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.5", 2024-03-14T10:04:39.106 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0.001", 2024-03-14T10:04:39.107 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "0.999", 2024-03-14T10:04:39.107 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.107 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "aggressiveness of optimization", 2024-03-14T10:04:39.107 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-14T10:04:39.107 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.107 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.107 INFO:teuthology.orchestra.run.smithi078.stdout: "end_time": { 2024-03-14T10:04:39.107 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_time", 2024-03-14T10:04:39.107 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.107 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.107 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.107 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2359", 2024-03-14T10:04:39.107 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.108 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.108 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.108 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ending time of day to automatically balance", 2024-03-14T10:04:39.108 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-14T10:04:39.108 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.108 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.108 INFO:teuthology.orchestra.run.smithi078.stdout: "end_weekday": { 2024-03-14T10:04:39.108 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_weekday", 2024-03-14T10:04:39.108 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-14T10:04:39.108 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.108 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.108 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-14T10:04:39.109 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-14T10:04:39.109 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-14T10:04:39.109 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.109 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-14T10:04:39.109 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-14T10:04:39.109 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.109 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.109 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.109 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.109 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.109 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.109 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.110 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.110 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.110 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.110 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.110 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.110 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.110 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.110 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.110 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.110 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.110 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.110 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.111 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.111 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.111 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.111 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.111 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.111 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.111 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.111 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.111 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.111 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.111 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.111 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.111 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.112 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.112 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.112 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.112 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.112 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.112 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.112 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.112 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.112 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.112 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.112 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.112 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.112 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.113 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.113 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.113 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.113 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.113 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.113 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.113 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.113 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.113 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.113 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.113 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.113 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.113 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.113 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.114 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.114 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.114 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.114 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.114 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.114 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.114 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.114 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.114 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.114 INFO:teuthology.orchestra.run.smithi078.stdout: "min_score": { 2024-03-14T10:04:39.114 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_score", 2024-03-14T10:04:39.114 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:39.114 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.115 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.115 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-14T10:04:39.115 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.115 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.115 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.115 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-14T10:04:39.115 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.115 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.115 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.115 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-14T10:04:39.115 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-14T10:04:39.115 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.115 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.116 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.116 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "upmap", 2024-03-14T10:04:39.116 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.116 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.116 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.116 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-compat", 2024-03-14T10:04:39.116 INFO:teuthology.orchestra.run.smithi078.stdout: "none", 2024-03-14T10:04:39.116 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap" 2024-03-14T10:04:39.116 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.116 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Balancer mode", 2024-03-14T10:04:39.116 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.116 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.116 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.116 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_ids": { 2024-03-14T10:04:39.117 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_ids", 2024-03-14T10:04:39.117 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.117 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.117 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.117 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.117 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.117 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.117 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.117 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-14T10:04:39.117 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.117 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.117 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.118 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-14T10:04:39.118 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-14T10:04:39.118 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.118 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.118 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.118 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-14T10:04:39.118 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.118 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.118 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.118 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-14T10:04:39.118 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.118 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.119 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.119 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_deviation": { 2024-03-14T10:04:39.119 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_deviation", 2024-03-14T10:04:39.119 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.119 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.119 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.119 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-14T10:04:39.119 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-14T10:04:39.119 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.119 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.119 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-14T10:04:39.119 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-14T10:04:39.119 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.119 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.120 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_optimizations": { 2024-03-14T10:04:39.120 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_optimizations", 2024-03-14T10:04:39.120 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-14T10:04:39.120 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.120 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.120 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:39.120 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.120 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.120 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.120 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-14T10:04:39.120 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.120 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.120 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.121 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm", 2024-03-14T10:04:39.121 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.121 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.121 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.121 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_down_multiplier": { 2024-03-14T10:04:39.121 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_down_multiplier", 2024-03-14T10:04:39.121 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:39.121 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.121 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.121 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-14T10:04:39.121 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.121 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.121 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.122 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-14T10:04:39.122 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.122 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.122 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.122 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_refresh_rate": { 2024-03-14T10:04:39.122 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_refresh_rate", 2024-03-14T10:04:39.122 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.122 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.122 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.122 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "20", 2024-03-14T10:04:39.122 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.122 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_starting_port": { 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_starting_port", 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4721", 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-14T10:04:39.123 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.124 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.124 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.124 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_ptrace": { 2024-03-14T10:04:39.124 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_ptrace", 2024-03-14T10:04:39.124 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.124 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.124 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.124 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.124 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.124 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.124 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.124 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-14T10:04:39.124 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-14T10:04:39.124 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.124 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.125 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_interval": { 2024-03-14T10:04:39.125 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_interval", 2024-03-14T10:04:39.125 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.125 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.125 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.125 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-14T10:04:39.125 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.125 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.125 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.125 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-14T10:04:39.125 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.125 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.125 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.125 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_memory_target_ratio": { 2024-03-14T10:04:39.126 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_memory_target_ratio", 2024-03-14T10:04:39.126 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:39.126 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.126 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.126 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.7", 2024-03-14T10:04:39.126 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.126 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.126 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.126 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-14T10:04:39.126 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.126 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.126 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.126 INFO:teuthology.orchestra.run.smithi078.stdout: "cgroups_split": { 2024-03-14T10:04:39.127 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cgroups_split", 2024-03-14T10:04:39.127 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.127 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.127 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.127 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.127 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.127 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.127 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.127 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-14T10:04:39.127 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.127 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.127 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.127 INFO:teuthology.orchestra.run.smithi078.stdout: "config_checks_enabled": { 2024-03-14T10:04:39.128 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_checks_enabled", 2024-03-14T10:04:39.128 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.128 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.128 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.128 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.128 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.128 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.128 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.128 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-14T10:04:39.128 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.128 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.128 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.128 INFO:teuthology.orchestra.run.smithi078.stdout: "config_dashboard": { 2024-03-14T10:04:39.129 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_dashboard", 2024-03-14T10:04:39.129 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.129 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.129 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.129 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.129 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.129 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.129 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.129 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-14T10:04:39.129 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.129 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.129 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.130 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_alertmanager": { 2024-03-14T10:04:39.130 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_alertmanager", 2024-03-14T10:04:39.130 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.130 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.130 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.130 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-14T10:04:39.130 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.130 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.130 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.130 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-14T10:04:39.130 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.130 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.130 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.131 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_base": { 2024-03-14T10:04:39.131 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_base", 2024-03-14T10:04:39.131 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.131 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.131 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.131 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-14T10:04:39.131 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.131 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.131 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.131 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Container image name, without the tag", 2024-03-14T10:04:39.131 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.131 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.131 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.131 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_elasticsearch": { 2024-03-14T10:04:39.132 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_elasticsearch", 2024-03-14T10:04:39.132 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.132 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.132 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.132 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-14T10:04:39.132 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.132 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.132 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.132 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "elasticsearch container image", 2024-03-14T10:04:39.132 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.132 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.132 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.132 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_grafana": { 2024-03-14T10:04:39.132 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_grafana", 2024-03-14T10:04:39.133 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.133 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.133 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.133 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2024-03-14T10:04:39.133 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.133 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.133 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.133 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-14T10:04:39.133 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.133 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.133 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.133 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_haproxy": { 2024-03-14T10:04:39.134 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_haproxy", 2024-03-14T10:04:39.134 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.134 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.134 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.134 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-14T10:04:39.134 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.134 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.134 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.134 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "HAproxy container image", 2024-03-14T10:04:39.134 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.134 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.134 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.135 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_agent": { 2024-03-14T10:04:39.135 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_agent", 2024-03-14T10:04:39.135 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.135 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.135 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.135 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-14T10:04:39.135 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.135 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.135 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.135 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger agent container image", 2024-03-14T10:04:39.135 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.135 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.135 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.136 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_collector": { 2024-03-14T10:04:39.136 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_collector", 2024-03-14T10:04:39.136 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.136 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.136 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.136 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-14T10:04:39.136 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.136 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.136 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.136 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger collector container image", 2024-03-14T10:04:39.136 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.136 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.136 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.137 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_query": { 2024-03-14T10:04:39.137 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_query", 2024-03-14T10:04:39.137 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.137 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.137 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.137 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-14T10:04:39.137 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.137 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.137 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.137 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger query container image", 2024-03-14T10:04:39.137 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.137 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.137 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_keepalived": { 2024-03-14T10:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_keepalived", 2024-03-14T10:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-14T10:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Keepalived container image", 2024-03-14T10:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.138 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_loki": { 2024-03-14T10:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_loki", 2024-03-14T10:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-14T10:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Loki container image", 2024-03-14T10:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.139 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_node_exporter": { 2024-03-14T10:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_node_exporter", 2024-03-14T10:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-14T10:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-14T10:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.141 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_nvmeof": { 2024-03-14T10:04:39.141 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_nvmeof", 2024-03-14T10:04:39.141 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.141 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.141 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.141 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/nvmeof:0.0.2", 2024-03-14T10:04:39.141 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.141 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.141 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.141 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Nvme-of container image", 2024-03-14T10:04:39.141 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.141 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.141 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.142 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_prometheus": { 2024-03-14T10:04:39.142 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_prometheus", 2024-03-14T10:04:39.142 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.142 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.142 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.142 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-14T10:04:39.142 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.142 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.142 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.142 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-14T10:04:39.142 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.142 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.142 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.143 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_promtail": { 2024-03-14T10:04:39.143 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_promtail", 2024-03-14T10:04:39.143 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.143 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.143 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.143 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-14T10:04:39.143 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.143 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.143 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.143 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Promtail container image", 2024-03-14T10:04:39.143 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.143 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.143 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.144 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_snmp_gateway": { 2024-03-14T10:04:39.144 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_snmp_gateway", 2024-03-14T10:04:39.144 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.144 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.144 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.144 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-14T10:04:39.144 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.144 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.144 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.144 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SNMP Gateway container image", 2024-03-14T10:04:39.144 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.144 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.144 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.145 INFO:teuthology.orchestra.run.smithi078.stdout: "container_init": { 2024-03-14T10:04:39.145 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_init", 2024-03-14T10:04:39.145 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.145 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.145 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.145 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.145 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.145 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.145 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.145 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Run podman/docker with `--init`", 2024-03-14T10:04:39.145 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.145 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.145 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.146 INFO:teuthology.orchestra.run.smithi078.stdout: "daemon_cache_timeout": { 2024-03-14T10:04:39.146 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "daemon_cache_timeout", 2024-03-14T10:04:39.146 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.146 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.146 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.146 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-14T10:04:39.146 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.146 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.146 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.146 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-14T10:04:39.146 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.146 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.146 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.147 INFO:teuthology.orchestra.run.smithi078.stdout: "default_cephadm_command_timeout": { 2024-03-14T10:04:39.147 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_cephadm_command_timeout", 2024-03-14T10:04:39.147 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.147 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.147 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.147 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "900", 2024-03-14T10:04:39.147 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.147 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.147 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.147 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-14T10:04:39.147 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.147 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.147 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.148 INFO:teuthology.orchestra.run.smithi078.stdout: "default_registry": { 2024-03-14T10:04:39.148 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_registry", 2024-03-14T10:04:39.148 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.148 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.148 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.148 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io", 2024-03-14T10:04:39.148 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.148 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.148 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.148 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-14T10:04:39.148 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.148 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.148 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.149 INFO:teuthology.orchestra.run.smithi078.stdout: "device_cache_timeout": { 2024-03-14T10:04:39.149 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_cache_timeout", 2024-03-14T10:04:39.149 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.149 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.149 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.149 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1800", 2024-03-14T10:04:39.149 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.149 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.149 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.149 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache device inventory", 2024-03-14T10:04:39.149 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.149 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.149 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.150 INFO:teuthology.orchestra.run.smithi078.stdout: "device_enhanced_scan": { 2024-03-14T10:04:39.150 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_enhanced_scan", 2024-03-14T10:04:39.150 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.150 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.150 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.150 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.150 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.150 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.150 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.150 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-14T10:04:39.150 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.150 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.150 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.151 INFO:teuthology.orchestra.run.smithi078.stdout: "facts_cache_timeout": { 2024-03-14T10:04:39.151 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "facts_cache_timeout", 2024-03-14T10:04:39.151 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.151 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.151 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.151 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-14T10:04:39.151 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.151 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.151 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.151 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache host facts data", 2024-03-14T10:04:39.151 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.151 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.152 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.152 INFO:teuthology.orchestra.run.smithi078.stdout: "host_check_interval": { 2024-03-14T10:04:39.152 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "host_check_interval", 2024-03-14T10:04:39.152 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.152 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.152 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.152 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-14T10:04:39.152 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.152 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.152 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.152 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to perform a host check", 2024-03-14T10:04:39.152 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.152 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.153 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.153 INFO:teuthology.orchestra.run.smithi078.stdout: "hw_monitoring": { 2024-03-14T10:04:39.153 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hw_monitoring", 2024-03-14T10:04:39.153 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.153 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.153 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.153 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.153 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.153 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.153 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.153 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-14T10:04:39.153 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.154 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.154 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.154 INFO:teuthology.orchestra.run.smithi078.stdout: "inventory_list_all": { 2024-03-14T10:04:39.154 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "inventory_list_all", 2024-03-14T10:04:39.154 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.154 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.154 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.154 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.154 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.154 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.154 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.154 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-14T10:04:39.154 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.155 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.155 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.155 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.155 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.155 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.155 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.155 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.155 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.155 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.155 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.155 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.155 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.155 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.155 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.156 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.156 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.156 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.156 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.156 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.156 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.156 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.156 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.156 INFO:teuthology.orchestra.run.smithi078.stdout: "log_refresh_metadata": { 2024-03-14T10:04:39.156 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_refresh_metadata", 2024-03-14T10:04:39.156 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.156 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.156 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.157 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.157 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.157 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.157 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.157 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-14T10:04:39.157 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.157 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.157 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.157 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.157 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.157 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.157 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.157 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.157 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.158 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.158 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.158 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.158 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-14T10:04:39.158 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.158 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.158 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.158 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.158 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.158 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.161 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.161 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.161 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.161 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.161 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.161 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.161 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.161 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.162 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.162 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.162 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.162 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.162 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.162 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.162 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.162 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.162 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.162 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.162 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.162 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.162 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.163 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.163 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.163 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.163 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.163 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.163 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.163 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.163 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.163 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.163 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-14T10:04:39.163 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-14T10:04:39.163 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.163 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.164 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.164 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.164 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.164 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.164 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.164 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-14T10:04:39.164 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.164 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.164 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.164 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-14T10:04:39.164 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-14T10:04:39.164 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.164 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.165 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.165 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "*", 2024-03-14T10:04:39.165 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.165 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.165 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.165 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-14T10:04:39.165 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.165 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.165 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.165 INFO:teuthology.orchestra.run.smithi078.stdout: "max_count_per_host": { 2024-03-14T10:04:39.165 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_count_per_host", 2024-03-14T10:04:39.165 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.165 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.166 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.166 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:39.166 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.166 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.166 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.166 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of daemons per service per host", 2024-03-14T10:04:39.166 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.166 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.166 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.166 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd_draining_count": { 2024-03-14T10:04:39.166 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_osd_draining_count", 2024-03-14T10:04:39.166 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.166 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.167 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.167 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:39.167 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.167 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.167 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.167 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-14T10:04:39.167 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.167 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.167 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.167 INFO:teuthology.orchestra.run.smithi078.stdout: "migration_current": { 2024-03-14T10:04:39.167 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "migration_current", 2024-03-14T10:04:39.167 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.167 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.167 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.168 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.168 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.168 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.168 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.168 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "internal - do not modify", 2024-03-14T10:04:39.168 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.168 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.168 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.168 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-14T10:04:39.168 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-14T10:04:39.168 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.168 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.169 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.169 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "root", 2024-03-14T10:04:39.169 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.169 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.169 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.169 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm-package", 2024-03-14T10:04:39.169 INFO:teuthology.orchestra.run.smithi078.stdout: "root" 2024-03-14T10:04:39.169 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.169 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "mode for remote execution of cephadm", 2024-03-14T10:04:39.169 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.169 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.169 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.169 INFO:teuthology.orchestra.run.smithi078.stdout: "oob_default_addr": { 2024-03-14T10:04:39.170 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "oob_default_addr", 2024-03-14T10:04:39.170 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.170 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.170 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.170 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "169.254.1.1", 2024-03-14T10:04:39.170 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.170 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.170 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.170 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-14T10:04:39.170 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.170 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.170 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.170 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_alerts_path": { 2024-03-14T10:04:39.171 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_alerts_path", 2024-03-14T10:04:39.171 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.171 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.171 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.171 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-14T10:04:39.171 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.171 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.171 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.171 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-14T10:04:39.171 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.171 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.171 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.171 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_insecure": { 2024-03-14T10:04:39.172 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_insecure", 2024-03-14T10:04:39.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.172 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.172 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.172 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.172 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.172 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.172 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.172 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-14T10:04:39.172 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.172 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.172 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.172 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_password": { 2024-03-14T10:04:39.173 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_password", 2024-03-14T10:04:39.173 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.173 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.173 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.173 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.173 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.173 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.173 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.173 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-14T10:04:39.173 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.173 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.173 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.174 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_url": { 2024-03-14T10:04:39.174 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_url", 2024-03-14T10:04:39.174 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.174 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.174 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.174 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.174 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.174 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.174 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.174 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-14T10:04:39.174 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.174 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.174 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.175 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_username": { 2024-03-14T10:04:39.175 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_username", 2024-03-14T10:04:39.175 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.175 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.175 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.175 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.175 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.175 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.175 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.175 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-14T10:04:39.175 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.175 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.175 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.176 INFO:teuthology.orchestra.run.smithi078.stdout: "secure_monitoring_stack": { 2024-03-14T10:04:39.176 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secure_monitoring_stack", 2024-03-14T10:04:39.176 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.176 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.176 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.176 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.176 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.176 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.176 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.176 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-14T10:04:39.176 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.176 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.177 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.177 INFO:teuthology.orchestra.run.smithi078.stdout: "service_discovery_port": { 2024-03-14T10:04:39.177 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "service_discovery_port", 2024-03-14T10:04:39.177 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.177 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.177 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.177 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8765", 2024-03-14T10:04:39.177 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.177 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.177 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.177 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "cephadm service discovery port", 2024-03-14T10:04:39.177 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.177 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.177 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.178 INFO:teuthology.orchestra.run.smithi078.stdout: "ssh_config_file": { 2024-03-14T10:04:39.178 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssh_config_file", 2024-03-14T10:04:39.178 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.178 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.178 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.178 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.178 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.178 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.178 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.178 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-14T10:04:39.178 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.178 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.178 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.179 INFO:teuthology.orchestra.run.smithi078.stdout: "use_agent": { 2024-03-14T10:04:39.179 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_agent", 2024-03-14T10:04:39.179 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.179 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.179 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.179 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.179 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.179 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.179 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.179 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-14T10:04:39.179 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.179 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.179 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.180 INFO:teuthology.orchestra.run.smithi078.stdout: "use_repo_digest": { 2024-03-14T10:04:39.180 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_repo_digest", 2024-03-14T10:04:39.180 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.180 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.180 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.180 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.180 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.180 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.180 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.180 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-14T10:04:39.180 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.180 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.180 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.181 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_failed_host_check": { 2024-03-14T10:04:39.181 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_failed_host_check", 2024-03-14T10:04:39.181 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.181 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.181 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.181 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.181 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.181 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.181 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.181 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if the host check fails", 2024-03-14T10:04:39.181 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.181 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.181 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.182 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_daemons": { 2024-03-14T10:04:39.182 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_daemons", 2024-03-14T10:04:39.182 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.182 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.182 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.182 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.182 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.182 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.182 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.182 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-14T10:04:39.182 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.182 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.182 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.183 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_hosts": { 2024-03-14T10:04:39.183 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_hosts", 2024-03-14T10:04:39.183 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.183 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.183 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.183 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.183 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.183 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.183 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.183 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-14T10:04:39.183 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.183 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.183 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.184 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.184 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.184 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crash", 2024-03-14T10:04:39.184 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.184 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.184 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.184 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.184 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.184 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.184 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.184 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.185 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.186 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.186 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.186 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.186 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.186 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.186 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.186 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.186 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.186 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.186 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.186 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.186 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.187 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.187 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.187 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.187 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.187 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.187 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.187 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.187 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.187 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.187 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.187 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.187 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.187 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.188 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.188 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.188 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.188 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.188 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.188 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.188 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.188 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.188 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.188 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.188 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.188 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.189 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.189 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.189 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.189 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.189 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.189 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.189 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.189 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.189 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.189 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.189 INFO:teuthology.orchestra.run.smithi078.stdout: "retain_interval": { 2024-03-14T10:04:39.189 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retain_interval", 2024-03-14T10:04:39.189 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.189 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.190 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.190 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "31536000", 2024-03-14T10:04:39.190 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.190 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.190 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.190 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-14T10:04:39.190 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.190 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.190 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.190 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_recent_interval": { 2024-03-14T10:04:39.190 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_recent_interval", 2024-03-14T10:04:39.190 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.191 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.191 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.191 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1209600", 2024-03-14T10:04:39.191 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.191 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.191 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.191 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-14T10:04:39.191 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.191 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.191 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.192 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dashboard", 2024-03-14T10:04:39.192 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.192 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.192 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.192 INFO:teuthology.orchestra.run.smithi078.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-14T10:04:39.192 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-14T10:04:39.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.192 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.192 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.192 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:39.192 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.192 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.192 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_HOST": { 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.193 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.194 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.194 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-14T10:04:39.194 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-14T10:04:39.194 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.194 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.194 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.194 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.194 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.194 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.194 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.194 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.194 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.195 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.195 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.195 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_ENABLED": { 2024-03-14T10:04:39.195 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_ENABLED", 2024-03-14T10:04:39.195 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.195 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.195 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.195 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.195 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.195 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.195 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.195 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.195 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.195 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.195 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.196 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-14T10:04:39.196 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-14T10:04:39.196 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.196 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.196 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.196 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.196 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.196 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.196 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.196 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.196 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.196 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.196 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.197 INFO:teuthology.orchestra.run.smithi078.stdout: "ENABLE_BROWSABLE_API": { 2024-03-14T10:04:39.197 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-14T10:04:39.197 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.197 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.197 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.197 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.197 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.197 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.197 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.197 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.197 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.197 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.197 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.198 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-14T10:04:39.198 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-14T10:04:39.198 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.198 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.198 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.198 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.198 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.198 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.198 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.198 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.198 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.198 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.198 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.199 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-14T10:04:39.199 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-14T10:04:39.199 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.199 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.199 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.199 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.199 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.199 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.199 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.199 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.199 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.199 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.199 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.200 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-14T10:04:39.200 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-14T10:04:39.200 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.200 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.200 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.200 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.200 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.200 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.200 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.200 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.200 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.200 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.200 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.200 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-14T10:04:39.201 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-14T10:04:39.201 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.201 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.201 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.201 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.201 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.201 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.201 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.201 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.201 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.201 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.201 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.201 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-14T10:04:39.201 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-14T10:04:39.202 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.202 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.202 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.202 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.202 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.202 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.202 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.202 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.202 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.202 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.202 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.202 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-14T10:04:39.202 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-14T10:04:39.203 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.203 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.203 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.203 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.203 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.203 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.203 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.203 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.203 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.203 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.203 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.203 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-14T10:04:39.203 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-14T10:04:39.204 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.204 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.204 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.204 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.204 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.204 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.204 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.204 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.204 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.204 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.204 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.204 INFO:teuthology.orchestra.run.smithi078.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-14T10:04:39.204 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-14T10:04:39.205 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.205 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.205 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.205 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.205 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.205 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.205 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.205 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.205 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.205 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.205 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.205 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_PASSWORD": { 2024-03-14T10:04:39.205 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-14T10:04:39.206 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.206 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.206 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.206 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-14T10:04:39.206 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.206 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.206 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.206 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.206 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.206 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.206 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.207 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-14T10:04:39.207 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-14T10:04:39.207 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.207 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.207 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.207 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.207 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.207 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.207 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.207 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.207 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.207 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.207 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.208 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_URL": { 2024-03-14T10:04:39.208 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_URL", 2024-03-14T10:04:39.208 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.208 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.208 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.208 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.208 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.208 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.208 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.208 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.208 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.209 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.209 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.209 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_USERNAME": { 2024-03-14T10:04:39.209 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-14T10:04:39.209 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.209 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.209 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.209 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-14T10:04:39.209 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.209 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.209 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.210 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.210 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.210 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.210 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.210 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-14T10:04:39.210 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-14T10:04:39.210 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.210 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.210 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.210 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.210 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.211 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.211 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.211 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.211 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.211 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.211 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.211 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-14T10:04:39.211 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-14T10:04:39.211 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.211 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.211 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.211 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.211 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.211 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.212 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.212 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.212 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.212 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.212 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.212 INFO:teuthology.orchestra.run.smithi078.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-14T10:04:39.212 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-14T10:04:39.212 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.212 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.212 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.212 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.213 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.213 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.213 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.213 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.213 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.213 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.213 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.213 INFO:teuthology.orchestra.run.smithi078.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-14T10:04:39.213 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-14T10:04:39.213 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.213 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.213 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.213 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.214 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.214 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.214 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.214 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.214 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.214 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.214 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.214 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_HOST": { 2024-03-14T10:04:39.214 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-14T10:04:39.214 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.214 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.214 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.214 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.215 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.215 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.215 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.215 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.215 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.215 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.215 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.215 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-14T10:04:39.215 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-14T10:04:39.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.215 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.215 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.216 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.216 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.216 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.216 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.216 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.216 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.216 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.216 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-14T10:04:39.216 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-14T10:04:39.216 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.216 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-14T10:04:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-14T10:04:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.217 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.218 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.218 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.218 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.218 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.218 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.218 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.218 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.218 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.218 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.218 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-14T10:04:39.218 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-14T10:04:39.219 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.219 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.219 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.219 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.219 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.219 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.219 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.219 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.219 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.219 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.219 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.219 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-14T10:04:39.219 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-14T10:04:39.220 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.220 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.220 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.220 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.220 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.220 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.220 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.220 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.220 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.220 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.220 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.220 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-14T10:04:39.220 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-14T10:04:39.221 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.221 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.221 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.221 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.221 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.221 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.221 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.221 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.221 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.221 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.221 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.221 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-14T10:04:39.221 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-14T10:04:39.222 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.222 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.222 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.222 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.222 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.222 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.222 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.222 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.222 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.222 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.222 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.222 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-14T10:04:39.223 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-14T10:04:39.223 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.223 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.223 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.223 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.223 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.223 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.223 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.223 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.223 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.223 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.223 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.223 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_ENABLED": { 2024-03-14T10:04:39.224 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-14T10:04:39.224 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.224 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.224 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.224 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.224 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.224 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.224 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.224 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.224 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.224 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.224 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.224 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-14T10:04:39.224 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-14T10:04:39.225 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.225 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.225 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.225 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-14T10:04:39.225 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.225 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.225 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.225 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.225 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.225 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.225 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.225 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-14T10:04:39.225 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-14T10:04:39.225 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.226 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.226 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.226 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:39.226 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.226 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.226 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.226 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.226 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.226 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.226 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.226 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-14T10:04:39.226 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-14T10:04:39.227 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.227 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.227 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.227 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8", 2024-03-14T10:04:39.227 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.227 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.227 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.227 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.227 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.227 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.227 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.227 INFO:teuthology.orchestra.run.smithi078.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-14T10:04:39.227 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-14T10:04:39.227 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.227 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.228 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.228 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "45", 2024-03-14T10:04:39.228 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.228 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.228 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.228 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.228 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.228 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.228 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.228 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ACCESS_KEY": { 2024-03-14T10:04:39.228 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-14T10:04:39.228 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.228 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.228 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.228 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.229 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.229 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.229 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.229 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.229 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.229 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.229 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.229 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-14T10:04:39.229 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-14T10:04:39.229 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.229 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.229 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.229 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-14T10:04:39.229 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.230 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.230 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.230 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.230 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.230 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.230 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.230 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SECRET_KEY": { 2024-03-14T10:04:39.230 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-14T10:04:39.230 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.230 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.230 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.230 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.230 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.231 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.231 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.231 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.231 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.231 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.231 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.231 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SSL_VERIFY": { 2024-03-14T10:04:39.231 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-14T10:04:39.231 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.231 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.231 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.231 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.231 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.231 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.231 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: "UNSAFE_TLS_v1_2": { 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.232 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.233 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.233 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-14T10:04:39.233 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-14T10:04:39.233 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.233 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.233 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.233 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-14T10:04:39.233 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.233 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.233 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.233 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.233 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.233 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.234 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.234 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-14T10:04:39.234 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-14T10:04:39.234 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.234 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.234 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.234 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:39.234 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.234 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.234 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.234 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.234 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.234 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.235 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.235 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-14T10:04:39.235 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-14T10:04:39.235 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.235 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.235 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.235 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-14T10:04:39.235 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.235 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.235 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.235 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.235 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.235 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.235 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.236 INFO:teuthology.orchestra.run.smithi078.stdout: "cross_origin_url": { 2024-03-14T10:04:39.236 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cross_origin_url", 2024-03-14T10:04:39.236 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.236 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.236 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.236 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.236 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.236 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.236 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.236 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.236 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.236 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.236 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.237 INFO:teuthology.orchestra.run.smithi078.stdout: "crt_file": { 2024-03-14T10:04:39.237 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crt_file", 2024-03-14T10:04:39.237 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.237 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.237 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.237 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.237 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.237 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.237 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.237 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.237 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.237 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.237 INFO:teuthology.orchestra.run.smithi078.stdout: "debug": { 2024-03-14T10:04:39.238 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "debug", 2024-03-14T10:04:39.238 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.238 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.238 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.238 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.238 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.238 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.238 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.238 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable/disable debug options", 2024-03-14T10:04:39.238 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.238 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.238 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.238 INFO:teuthology.orchestra.run.smithi078.stdout: "jwt_token_ttl": { 2024-03-14T10:04:39.239 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "jwt_token_ttl", 2024-03-14T10:04:39.239 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.239 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.239 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.239 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "28800", 2024-03-14T10:04:39.239 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.239 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.239 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.239 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.239 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.239 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.239 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.239 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-14T10:04:39.239 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-14T10:04:39.240 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.240 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.240 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.240 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.240 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.240 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.240 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.240 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.240 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.240 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.240 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.240 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.240 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.240 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.241 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.241 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.241 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.241 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.241 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.241 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.241 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.241 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.241 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.241 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.241 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.241 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.241 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.241 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.242 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.242 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.242 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.242 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.242 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.242 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.242 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.242 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.242 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.242 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.242 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.242 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.242 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.243 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.243 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.243 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.243 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.243 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.243 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.243 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.243 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.243 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.243 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.243 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.243 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.244 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.244 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.244 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.244 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.244 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.244 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.244 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.244 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.244 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.244 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.244 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.244 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.244 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.244 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.245 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.245 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.245 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.245 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.245 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.245 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.245 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.245 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.245 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.245 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.245 INFO:teuthology.orchestra.run.smithi078.stdout: "motd": { 2024-03-14T10:04:39.245 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "motd", 2024-03-14T10:04:39.245 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.245 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.246 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.246 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.246 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.246 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.246 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.246 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "The message of the day", 2024-03-14T10:04:39.246 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.246 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.246 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.246 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_resolve_ip_addr": { 2024-03-14T10:04:39.246 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "redirect_resolve_ip_addr", 2024-03-14T10:04:39.247 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.247 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.247 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.247 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.247 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.247 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.247 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.247 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.247 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.247 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.247 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.247 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-14T10:04:39.247 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-14T10:04:39.247 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.248 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.248 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.248 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-14T10:04:39.248 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.248 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.248 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.248 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.248 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.248 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.248 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.248 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-14T10:04:39.248 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-14T10:04:39.249 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.249 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.249 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.249 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8080", 2024-03-14T10:04:39.249 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.249 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.249 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.249 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.249 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.249 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.249 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.249 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-14T10:04:39.249 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-14T10:04:39.250 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.250 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.250 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.250 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.250 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.250 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.250 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.250 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.250 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.250 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.250 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.251 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl_server_port": { 2024-03-14T10:04:39.251 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl_server_port", 2024-03-14T10:04:39.251 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.251 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.251 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.251 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8443", 2024-03-14T10:04:39.251 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.251 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.251 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.251 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.251 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.251 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.251 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.252 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-14T10:04:39.252 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-14T10:04:39.252 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.252 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.252 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.252 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "redirect", 2024-03-14T10:04:39.252 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.252 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.252 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.252 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.252 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect" 2024-03-14T10:04:39.252 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.252 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.252 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.252 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.252 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.253 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-14T10:04:39.253 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-14T10:04:39.253 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.253 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.253 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.253 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-14T10:04:39.253 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-14T10:04:39.253 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-14T10:04:39.253 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.253 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.253 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.253 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.253 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.253 INFO:teuthology.orchestra.run.smithi078.stdout: "url_prefix": { 2024-03-14T10:04:39.254 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url_prefix", 2024-03-14T10:04:39.254 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.254 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.254 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.254 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.254 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.254 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.254 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.254 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.254 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.254 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.254 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.254 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.255 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.255 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2024-03-14T10:04:39.255 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.255 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.255 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.255 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_monitoring": { 2024-03-14T10:04:39.255 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_monitoring", 2024-03-14T10:04:39.255 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.255 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.255 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.255 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.255 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.255 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.256 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.256 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "monitor device health metrics", 2024-03-14T10:04:39.256 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.256 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.256 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.256 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.256 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.256 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.256 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.256 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.256 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.256 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.256 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.256 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.256 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.257 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.257 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.257 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.257 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.257 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.257 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.257 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.257 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.257 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.257 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.257 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.257 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.257 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.257 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.258 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.258 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.258 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.258 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.258 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.258 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.258 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.258 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.258 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.258 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.258 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.258 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.258 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.259 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "mark_out_threshold": { 2024-03-14T10:04:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mark_out_threshold", 2024-03-14T10:04:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2419200", 2024-03-14T10:04:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-14T10:04:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": { 2024-03-14T10:04:39.261 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_name", 2024-03-14T10:04:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "device_health_metrics", 2024-03-14T10:04:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-14T10:04:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "retention_period": { 2024-03-14T10:04:39.262 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retention_period", 2024-03-14T10:04:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15552000", 2024-03-14T10:04:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain device health metrics", 2024-03-14T10:04:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_frequency": { 2024-03-14T10:04:39.263 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_frequency", 2024-03-14T10:04:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-14T10:04:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-14T10:04:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "self_heal": { 2024-03-14T10:04:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "self_heal", 2024-03-14T10:04:39.264 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-14T10:04:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-14T10:04:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-14T10:04:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.265 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-14T10:04:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and check device health", 2024-03-14T10:04:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_threshold": { 2024-03-14T10:04:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_threshold", 2024-03-14T10:04:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.266 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7257600", 2024-03-14T10:04:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-14T10:04:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "diskprediction_local", 2024-03-14T10:04:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.267 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.268 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.269 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.269 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.269 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.269 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.269 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.269 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.269 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.269 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.269 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.269 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.269 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.269 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.269 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.270 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.270 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.270 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.270 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.270 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.270 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.270 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.270 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.270 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.270 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.270 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.270 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.270 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.271 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.272 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.272 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.272 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.272 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.272 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.272 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.272 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.272 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.272 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.272 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.272 INFO:teuthology.orchestra.run.smithi078.stdout: "predict_interval": { 2024-03-14T10:04:39.272 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predict_interval", 2024-03-14T10:04:39.272 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.272 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.272 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.273 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-14T10:04:39.273 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.273 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.273 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.273 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.273 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.273 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.273 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.273 INFO:teuthology.orchestra.run.smithi078.stdout: "predictor_model": { 2024-03-14T10:04:39.273 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predictor_model", 2024-03-14T10:04:39.273 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.273 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.273 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.273 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "prophetstor", 2024-03-14T10:04:39.274 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.274 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.274 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.274 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.274 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.274 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.274 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.274 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-14T10:04:39.274 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-14T10:04:39.274 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.274 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.274 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.274 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-14T10:04:39.275 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.275 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.275 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.275 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.275 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.275 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.275 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.275 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "influx", 2024-03-14T10:04:39.275 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": false, 2024-03-14T10:04:39.275 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "influxdb python module not found", 2024-03-14T10:04:39.275 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.276 INFO:teuthology.orchestra.run.smithi078.stdout: "batch_size": { 2024-03-14T10:04:39.276 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "batch_size", 2024-03-14T10:04:39.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.276 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.276 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.276 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5000", 2024-03-14T10:04:39.276 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.276 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.276 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.276 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-14T10:04:39.276 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.276 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.276 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.276 INFO:teuthology.orchestra.run.smithi078.stdout: "database": { 2024-03-14T10:04:39.276 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "database", 2024-03-14T10:04:39.277 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.277 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.277 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.277 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "ceph", 2024-03-14T10:04:39.277 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.277 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.277 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.277 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-14T10:04:39.277 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.277 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.277 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.277 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": { 2024-03-14T10:04:39.277 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hostname", 2024-03-14T10:04:39.277 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.278 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.278 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.278 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.278 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.278 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.278 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.278 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server hostname", 2024-03-14T10:04:39.278 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.278 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.278 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.278 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-14T10:04:39.278 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-14T10:04:39.278 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.278 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.279 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.279 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "30", 2024-03-14T10:04:39.279 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "5", 2024-03-14T10:04:39.279 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.279 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.279 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-14T10:04:39.279 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.279 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.279 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.279 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.279 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.279 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.280 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.280 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.280 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.280 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.280 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.280 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.280 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.280 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.280 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.280 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.280 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.280 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.280 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.281 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.281 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.281 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.281 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.281 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.281 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.281 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.281 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.281 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.281 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.281 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.281 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.281 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.282 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.282 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.282 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.282 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.282 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.282 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.282 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.282 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.282 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.282 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.282 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.283 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.283 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.283 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.283 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.283 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.283 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.283 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.283 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.283 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.283 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.283 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.283 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.283 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.283 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.284 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.284 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.284 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.284 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.284 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.284 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.284 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.284 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.284 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.284 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.284 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.284 INFO:teuthology.orchestra.run.smithi078.stdout: "password": { 2024-03-14T10:04:39.284 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "password", 2024-03-14T10:04:39.284 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.285 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.285 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.285 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.285 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.285 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.285 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.285 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "password of InfluxDB server user", 2024-03-14T10:04:39.285 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.285 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.285 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.285 INFO:teuthology.orchestra.run.smithi078.stdout: "port": { 2024-03-14T10:04:39.285 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "port", 2024-03-14T10:04:39.286 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.286 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.286 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.286 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8086", 2024-03-14T10:04:39.286 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.286 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.286 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.286 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server port", 2024-03-14T10:04:39.286 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.286 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.286 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.286 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-14T10:04:39.287 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-14T10:04:39.287 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.287 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.287 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.287 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "false", 2024-03-14T10:04:39.287 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.287 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.287 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.287 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-14T10:04:39.287 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.287 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.287 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.287 INFO:teuthology.orchestra.run.smithi078.stdout: "threads": { 2024-03-14T10:04:39.288 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threads", 2024-03-14T10:04:39.288 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.288 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.288 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.288 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-14T10:04:39.288 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-14T10:04:39.288 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "32", 2024-03-14T10:04:39.288 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.288 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-14T10:04:39.288 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.288 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.288 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.288 INFO:teuthology.orchestra.run.smithi078.stdout: "username": { 2024-03-14T10:04:39.288 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "username", 2024-03-14T10:04:39.288 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.289 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.289 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.289 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.289 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.289 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.289 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.289 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "username of InfluxDB server user", 2024-03-14T10:04:39.289 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.289 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.289 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.289 INFO:teuthology.orchestra.run.smithi078.stdout: "verify_ssl": { 2024-03-14T10:04:39.289 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "verify_ssl", 2024-03-14T10:04:39.289 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.289 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.290 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.290 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "true", 2024-03-14T10:04:39.290 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.290 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.290 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.290 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-14T10:04:39.290 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.290 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.290 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.290 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "insights", 2024-03-14T10:04:39.290 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.291 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.291 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.291 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.291 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.291 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.291 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.291 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.291 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.291 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.291 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.291 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.291 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.291 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.291 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.291 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.291 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.292 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.292 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.292 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.292 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.292 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.292 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.292 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.292 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.292 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.292 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.292 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.292 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.293 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.293 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.293 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.293 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.293 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.293 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.293 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.293 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.293 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.293 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.293 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.293 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.293 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.294 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.294 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.294 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.294 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.294 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.294 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.294 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.294 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.294 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.294 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.294 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.294 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.294 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.294 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.294 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.295 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.295 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.295 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.295 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.295 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.295 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.295 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.295 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.295 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.295 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.295 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.295 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.296 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "iostat", 2024-03-14T10:04:39.296 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.296 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.296 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.296 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.296 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.296 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.296 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.296 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.296 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.296 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.296 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.297 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.298 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.298 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.298 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.298 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.298 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.298 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.298 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.298 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.298 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.298 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.298 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.298 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.298 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.299 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.299 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.299 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.299 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.299 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.299 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.299 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.299 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.299 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.299 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.299 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.299 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.299 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.299 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.300 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.300 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.300 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.300 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.300 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.300 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.300 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.300 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.300 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.300 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.300 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.300 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.300 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.300 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "k8sevents", 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_event_retention_days": { 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ceph_event_retention_days", 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7", 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.301 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.302 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.302 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-14T10:04:39.302 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.302 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.302 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.302 INFO:teuthology.orchestra.run.smithi078.stdout: "config_check_secs": { 2024-03-14T10:04:39.302 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_check_secs", 2024-03-14T10:04:39.302 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.302 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.302 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.302 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:39.302 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "10", 2024-03-14T10:04:39.302 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.303 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.303 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-14T10:04:39.303 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.303 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.303 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.303 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.303 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.303 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.303 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.303 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.303 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.303 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.303 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.303 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.303 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.304 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.304 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.304 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.304 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.304 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.304 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.304 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.304 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.304 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.304 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.304 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.304 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.304 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.304 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.304 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.305 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.305 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.305 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.305 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.305 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.305 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.305 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.305 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.305 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.305 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.305 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.305 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.305 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.305 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.305 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.306 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.306 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.306 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.306 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.306 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.306 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.306 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.306 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.306 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.306 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.306 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.306 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.306 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.306 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.307 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "localpool", 2024-03-14T10:04:39.308 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.308 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.308 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.308 INFO:teuthology.orchestra.run.smithi078.stdout: "failure_domain": { 2024-03-14T10:04:39.308 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "failure_domain", 2024-03-14T10:04:39.308 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.308 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.308 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.308 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "host", 2024-03-14T10:04:39.308 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.308 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.308 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.308 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "failure domain for any created local pool", 2024-03-14T10:04:39.308 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-14T10:04:39.309 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.309 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.309 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.309 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.309 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.309 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.309 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.309 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.309 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.309 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.309 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.309 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.310 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.310 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.310 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.310 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.310 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.310 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.310 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.310 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.310 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.310 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.310 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.310 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.310 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.311 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.311 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.311 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.311 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.311 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.311 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.311 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.311 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.311 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.311 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.311 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.311 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.312 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.312 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.312 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.312 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.312 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.312 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.312 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.312 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.312 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.312 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.312 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.312 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.312 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.312 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.313 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.313 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.313 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.313 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.313 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.313 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.313 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.313 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.313 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.313 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.313 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.313 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.313 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.314 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.314 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.314 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.314 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.314 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": { 2024-03-14T10:04:39.314 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_size", 2024-03-14T10:04:39.314 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.314 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.314 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.314 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.314 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.314 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.314 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.315 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default min_size for any created local pool", 2024-03-14T10:04:39.315 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-14T10:04:39.315 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.315 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.315 INFO:teuthology.orchestra.run.smithi078.stdout: "num_rep": { 2024-03-14T10:04:39.315 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "num_rep", 2024-03-14T10:04:39.315 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.315 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.315 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.315 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3", 2024-03-14T10:04:39.315 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.315 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.315 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default replica count for any created local pool", 2024-03-14T10:04:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": { 2024-03-14T10:04:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_num", 2024-03-14T10:04:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "128", 2024-03-14T10:04:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.316 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default pg_num for any created local pool", 2024-03-14T10:04:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "prefix": { 2024-03-14T10:04:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prefix", 2024-03-14T10:04:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.317 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name prefix for any created local pool", 2024-03-14T10:04:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "subtree": { 2024-03-14T10:04:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "subtree", 2024-03-14T10:04:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "rack", 2024-03-14T10:04:39.318 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-14T10:04:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-14T10:04:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mds_autoscaler", 2024-03-14T10:04:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.319 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.320 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.321 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.322 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.323 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirroring", 2024-03-14T10:04:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.325 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.327 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.327 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.327 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.327 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.327 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.327 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.327 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.327 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.327 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.327 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.327 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.327 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.327 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.327 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.328 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.329 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "nfs", 2024-03-14T10:04:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.330 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.331 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.332 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.333 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.334 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-14T10:04:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "fail_fs": { 2024-03-14T10:04:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "fail_fs", 2024-03-14T10:04:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.336 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-14T10:04:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.337 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.338 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.339 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.340 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.341 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator": { 2024-03-14T10:04:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-14T10:04:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.342 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2024-03-14T10:04:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: "rook", 2024-03-14T10:04:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: "test_orchestrator" 2024-03-14T10:04:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Orchestrator backend", 2024-03-14T10:04:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_perf_query", 2024-03-14T10:04:39.343 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.344 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.345 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.346 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.347 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.348 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_support", 2024-03-14T10:04:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.349 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.350 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.351 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.352 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.353 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_autoscaler", 2024-03-14T10:04:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.354 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.355 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.356 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.357 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.359 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-14T10:04:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-14T10:04:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-14T10:04:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.360 INFO:teuthology.orchestra.run.smithi078.stdout: "threshold": { 2024-03-14T10:04:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threshold", 2024-03-14T10:04:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-14T10:04:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1.0", 2024-03-14T10:04:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "scaling threshold", 2024-03-14T10:04:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-14T10:04:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "progress", 2024-03-14T10:04:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_pg_recovery_event": { 2024-03-14T10:04:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_pg_recovery_event", 2024-03-14T10:04:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-14T10:04:39.362 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-14T10:04:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-14T10:04:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.363 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.364 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.365 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.366 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "max_completed_events": { 2024-03-14T10:04:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_completed_events", 2024-03-14T10:04:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "50", 2024-03-14T10:04:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "number of past completed events to remember", 2024-03-14T10:04:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.369 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-14T10:04:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-14T10:04:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-14T10:04:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long the module is going to sleep", 2024-03-14T10:04:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.370 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus", 2024-03-14T10:04:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "cache": { 2024-03-14T10:04:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cache", 2024-03-14T10:04:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.371 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "exclude_perf_counters": { 2024-03-14T10:04:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "exclude_perf_counters", 2024-03-14T10:04:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.372 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-14T10:04:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-14T10:04:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.373 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.374 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.375 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.376 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.377 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools": { 2024-03-14T10:04:39.378 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools", 2024-03-14T10:04:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-14T10:04:39.379 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-14T10:04:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300", 2024-03-14T10:04:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_interval": { 2024-03-14T10:04:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_interval", 2024-03-14T10:04:39.380 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15.0", 2024-03-14T10:04:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-14T10:04:39.381 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-14T10:04:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-14T10:04:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-14T10:04:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-14T10:04:39.382 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-14T10:04:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "9283", 2024-03-14T10:04:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-14T10:04:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "stale_cache_strategy": { 2024-03-14T10:04:39.383 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stale_cache_strategy", 2024-03-14T10:04:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "log", 2024-03-14T10:04:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-14T10:04:39.384 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-14T10:04:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "default", 2024-03-14T10:04:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "default", 2024-03-14T10:04:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "error" 2024-03-14T10:04:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.385 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-14T10:04:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-14T10:04:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-14T10:04:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-14T10:04:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-14T10:04:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.386 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2024-03-14T10:04:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.387 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.390 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.391 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_snap_create": { 2024-03-14T10:04:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_snap_create", 2024-03-14T10:04:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.392 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "mirror_snapshot_schedule": { 2024-03-14T10:04:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirror_snapshot_schedule", 2024-03-14T10:04:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.393 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "trash_purge_schedule": { 2024-03-14T10:04:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "trash_purge_schedule", 2024-03-14T10:04:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.394 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "restful", 2024-03-14T10:04:39.395 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_auth": { 2024-03-14T10:04:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_auth", 2024-03-14T10:04:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.396 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-14T10:04:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-14T10:04:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.397 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.398 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.399 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.400 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.401 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.402 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-14T10:04:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-14T10:04:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.403 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-14T10:04:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-14T10:04:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.404 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw", 2024-03-14T10:04:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.405 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.406 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.407 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.408 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.409 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rook", 2024-03-14T10:04:39.410 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.411 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.412 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.413 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.414 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.415 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_class": { 2024-03-14T10:04:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "storage_class", 2024-03-14T10:04:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "local", 2024-03-14T10:04:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-14T10:04:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.416 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "selftest", 2024-03-14T10:04:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.417 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.418 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.419 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.420 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.421 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "roption1": { 2024-03-14T10:04:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption1", 2024-03-14T10:04:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.422 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "roption2": { 2024-03-14T10:04:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption2", 2024-03-14T10:04:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.423 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "xyz", 2024-03-14T10:04:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption1": { 2024-03-14T10:04:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption1", 2024-03-14T10:04:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.424 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption2": { 2024-03-14T10:04:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption2", 2024-03-14T10:04:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.425 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption3": { 2024-03-14T10:04:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption3", 2024-03-14T10:04:39.426 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption4": { 2024-03-14T10:04:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption4", 2024-03-14T10:04:39.427 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption5": { 2024-03-14T10:04:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption5", 2024-03-14T10:04:39.428 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption6": { 2024-03-14T10:04:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption6", 2024-03-14T10:04:39.429 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption7": { 2024-03-14T10:04:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption7", 2024-03-14T10:04:39.430 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-14T10:04:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "42", 2024-03-14T10:04:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "testkey": { 2024-03-14T10:04:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testkey", 2024-03-14T10:04:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.431 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "testlkey": { 2024-03-14T10:04:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testlkey", 2024-03-14T10:04:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.432 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "testnewline": { 2024-03-14T10:04:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testnewline", 2024-03-14T10:04:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.433 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-03-14T10:04:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_m_granularity": { 2024-03-14T10:04:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_m_granularity", 2024-03-14T10:04:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.435 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow minute scheduled snapshots", 2024-03-14T10:04:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "dump_on_update": { 2024-03-14T10:04:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dump_on_update", 2024-03-14T10:04:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.436 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "dump database to debug log on update", 2024-03-14T10:04:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.437 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.438 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.439 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.440 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.440 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.440 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.440 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.440 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.440 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.440 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.440 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.440 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.440 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.440 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.440 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.440 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.441 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.441 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.441 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.441 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.441 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.441 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.441 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.441 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.441 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.441 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.441 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.441 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.441 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.441 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.442 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.442 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.442 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.442 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.442 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.442 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.442 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.442 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stats", 2024-03-14T10:04:39.442 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.442 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.442 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.443 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.443 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.443 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.443 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.443 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.443 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.443 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.443 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.443 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.443 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.443 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.443 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.443 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.443 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.444 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.444 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.444 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.444 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.444 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.444 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.444 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.444 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.444 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.444 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.444 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.444 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.444 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.445 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.445 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.445 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.445 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.445 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.445 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.445 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.445 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.445 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.445 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.445 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.446 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.446 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.446 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.446 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.446 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.446 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.446 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.446 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.446 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.446 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.446 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.446 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.446 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.446 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.447 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.447 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.447 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.447 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.447 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.447 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.447 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.447 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.447 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.447 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.447 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.447 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.447 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.447 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "status", 2024-03-14T10:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.448 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.449 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.450 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.451 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.451 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.451 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.451 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.451 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.451 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.451 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.451 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.451 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.451 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.451 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.451 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.451 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.452 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.452 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.452 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.452 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.452 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.452 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.452 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.452 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.452 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.452 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.452 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.452 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.453 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.453 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.453 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.453 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.453 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.453 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.453 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.453 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.453 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.453 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.453 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.454 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telegraf", 2024-03-14T10:04:39.454 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.454 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.454 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.454 INFO:teuthology.orchestra.run.smithi078.stdout: "address": { 2024-03-14T10:04:39.454 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "address", 2024-03-14T10:04:39.454 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.454 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.454 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.454 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-14T10:04:39.454 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.454 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.454 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.455 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.455 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.455 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.455 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.455 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-14T10:04:39.455 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-14T10:04:39.455 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.455 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.455 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.455 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15", 2024-03-14T10:04:39.455 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.455 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.455 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.455 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.456 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.456 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.456 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.456 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.456 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.456 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.456 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.456 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.456 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.456 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.456 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.456 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.456 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.456 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.457 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.457 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.457 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.457 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.457 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.457 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.457 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.457 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.457 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.457 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.457 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.457 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.457 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.457 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.458 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.458 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.458 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.458 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.458 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.458 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.458 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.458 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.458 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.458 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.458 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.458 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.459 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.459 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.459 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.459 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.459 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.459 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.459 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.459 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.459 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.459 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.459 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.459 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.459 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.459 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.459 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.459 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.460 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.460 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.460 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.460 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.460 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.460 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.460 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.460 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.460 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.460 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.460 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.460 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.460 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.461 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telemetry", 2024-03-14T10:04:39.461 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.461 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.461 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.461 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_basic": { 2024-03-14T10:04:39.461 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_basic", 2024-03-14T10:04:39.461 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.461 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.461 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.461 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.461 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.461 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.462 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.462 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-14T10:04:39.462 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.462 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.462 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.462 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_crash": { 2024-03-14T10:04:39.462 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_crash", 2024-03-14T10:04:39.462 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.462 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.462 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.462 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.462 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.462 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.463 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.463 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-14T10:04:39.463 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.463 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.463 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.463 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_device": { 2024-03-14T10:04:39.463 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_device", 2024-03-14T10:04:39.463 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.463 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.463 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.463 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.463 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.463 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.464 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.464 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-14T10:04:39.464 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.464 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.464 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.464 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_ident": { 2024-03-14T10:04:39.464 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_ident", 2024-03-14T10:04:39.464 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.464 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.464 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.464 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.464 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.464 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.464 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-14T10:04:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_perf": { 2024-03-14T10:04:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_perf", 2024-03-14T10:04:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.465 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.466 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-14T10:04:39.466 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.466 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.466 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.466 INFO:teuthology.orchestra.run.smithi078.stdout: "contact": { 2024-03-14T10:04:39.466 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "contact", 2024-03-14T10:04:39.466 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.466 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.466 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.466 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.466 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.467 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.467 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.467 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.467 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.467 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.467 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.467 INFO:teuthology.orchestra.run.smithi078.stdout: "description": { 2024-03-14T10:04:39.467 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "description", 2024-03-14T10:04:39.467 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.467 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.467 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.467 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.467 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.468 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.468 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.468 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.468 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.468 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.468 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.468 INFO:teuthology.orchestra.run.smithi078.stdout: "device_url": { 2024-03-14T10:04:39.468 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_url", 2024-03-14T10:04:39.468 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.468 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.468 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.468 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-14T10:04:39.468 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.469 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.469 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.469 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.469 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.469 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.469 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.469 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-14T10:04:39.469 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-14T10:04:39.469 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.469 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.469 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.469 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.469 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-14T10:04:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-14T10:04:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.470 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.471 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "24", 2024-03-14T10:04:39.471 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "8", 2024-03-14T10:04:39.471 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.471 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.471 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.471 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.471 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.471 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.471 INFO:teuthology.orchestra.run.smithi078.stdout: "last_opt_revision": { 2024-03-14T10:04:39.471 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "last_opt_revision", 2024-03-14T10:04:39.471 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.471 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.471 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.471 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1", 2024-03-14T10:04:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard": { 2024-03-14T10:04:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard", 2024-03-14T10:04:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.472 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.473 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.473 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.473 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.473 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.473 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.473 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.473 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.473 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard_description": { 2024-03-14T10:04:39.473 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard_description", 2024-03-14T10:04:39.473 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.473 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.473 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.473 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.474 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.474 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.474 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.474 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.474 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.474 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.474 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.474 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.474 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.474 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.474 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.474 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.475 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.475 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.475 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.475 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.475 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.475 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.475 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.475 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.475 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.475 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.475 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.475 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.475 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.475 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.475 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.476 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.476 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.476 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.476 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.476 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.476 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.476 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.476 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.476 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.476 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.476 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.476 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.476 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.477 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.477 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.477 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.477 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.477 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.477 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.477 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.477 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.477 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.477 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.477 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.477 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.477 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.478 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.478 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.478 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.478 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.478 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.478 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.478 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.478 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.478 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.478 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.478 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.478 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.479 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.479 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.479 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.479 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.479 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.479 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.479 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.479 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.479 INFO:teuthology.orchestra.run.smithi078.stdout: "organization": { 2024-03-14T10:04:39.479 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "organization", 2024-03-14T10:04:39.479 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.479 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.479 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.480 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.480 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.480 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.480 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.480 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.480 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.480 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.480 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.480 INFO:teuthology.orchestra.run.smithi078.stdout: "proxy": { 2024-03-14T10:04:39.480 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "proxy", 2024-03-14T10:04:39.480 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.480 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.480 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.481 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.481 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.481 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.481 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.481 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.481 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.481 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.481 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.481 INFO:teuthology.orchestra.run.smithi078.stdout: "url": { 2024-03-14T10:04:39.481 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url", 2024-03-14T10:04:39.481 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.481 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.481 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.481 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-14T10:04:39.482 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.482 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.482 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.482 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.482 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.482 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.482 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.482 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.482 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.482 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test_orchestrator", 2024-03-14T10:04:39.483 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.483 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.483 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.483 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.483 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.483 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.483 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.483 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.483 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.483 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.483 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.483 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.483 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.483 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.484 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.484 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.484 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.484 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.484 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.484 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.484 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.484 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.484 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.484 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.484 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.484 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.484 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.484 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.485 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.485 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.485 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.485 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.485 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.485 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.485 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.485 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.485 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.485 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.485 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.485 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.485 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.485 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.486 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.486 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.486 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.486 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.486 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.486 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.486 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.486 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.486 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.486 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.486 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.486 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.487 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.487 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.487 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.487 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.487 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.487 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.487 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.487 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.487 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.487 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.487 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.487 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.487 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.487 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.487 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.488 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2024-03-14T10:04:39.488 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.488 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.488 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.488 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.488 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.488 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.488 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.488 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.489 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.489 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.489 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.489 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.489 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.489 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.489 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.489 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.489 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.489 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.489 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.489 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.489 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.489 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.489 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.490 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.490 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.490 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.490 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.490 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.490 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.490 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.490 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.490 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.490 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.490 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.491 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.491 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.491 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.491 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.491 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.491 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.491 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.491 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.491 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.491 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.491 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.491 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.492 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.492 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.492 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.492 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.492 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.492 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.492 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.492 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.492 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.492 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.492 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.492 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.493 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.493 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.493 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.493 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.493 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.493 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.493 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.493 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.493 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.493 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.493 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.493 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_clones": { 2024-03-14T10:04:39.493 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_clones", 2024-03-14T10:04:39.494 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.494 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.494 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.494 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4", 2024-03-14T10:04:39.494 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.494 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.494 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.494 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-14T10:04:39.494 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.494 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.494 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.495 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_delay": { 2024-03-14T10:04:39.495 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_delay", 2024-03-14T10:04:39.495 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.495 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.495 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.495 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-14T10:04:39.495 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.495 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.495 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.495 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-14T10:04:39.495 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.495 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.496 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.496 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.496 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.496 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix", 2024-03-14T10:04:39.496 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.496 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.496 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.496 INFO:teuthology.orchestra.run.smithi078.stdout: "discovery_interval": { 2024-03-14T10:04:39.496 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "discovery_interval", 2024-03-14T10:04:39.496 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-14T10:04:39.496 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.497 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.497 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "100", 2024-03-14T10:04:39.497 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.497 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.497 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.497 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.497 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.497 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.497 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.497 INFO:teuthology.orchestra.run.smithi078.stdout: "identifier": { 2024-03-14T10:04:39.497 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "identifier", 2024-03-14T10:04:39.497 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.498 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.498 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.498 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.498 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.498 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.498 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.498 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.498 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.498 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.498 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.498 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-14T10:04:39.498 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-14T10:04:39.499 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.499 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.499 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.499 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-14T10:04:39.499 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.499 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.499 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.499 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.499 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.499 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.499 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.499 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.499 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.499 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.500 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.500 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.500 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.500 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.500 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.500 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.500 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.500 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.500 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.500 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.500 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.500 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.500 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.500 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.500 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.500 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.501 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.501 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.501 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.501 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.501 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.501 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.501 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.501 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.501 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.501 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.501 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.501 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.501 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.501 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.502 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.502 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.502 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.502 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.502 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.502 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.502 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.502 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.502 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.502 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.502 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.502 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.502 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.502 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.503 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.503 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.503 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.503 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.503 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.503 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.503 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.503 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.503 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.503 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.503 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.503 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.503 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.503 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.504 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.504 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.504 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.504 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.504 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.504 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_host": { 2024-03-14T10:04:39.504 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_host", 2024-03-14T10:04:39.504 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.504 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.504 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.504 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.504 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.505 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.505 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.505 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.505 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.505 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.505 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.505 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_port": { 2024-03-14T10:04:39.505 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_port", 2024-03-14T10:04:39.505 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.505 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.505 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.505 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10051", 2024-03-14T10:04:39.505 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.506 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.506 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.506 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.506 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.506 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.506 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.506 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_sender": { 2024-03-14T10:04:39.506 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_sender", 2024-03-14T10:04:39.506 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.506 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.506 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.506 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-14T10:04:39.507 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.507 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.507 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.507 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.507 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.507 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.507 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.507 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.507 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.507 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.507 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-14T10:04:39.507 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.508 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.508 INFO:teuthology.orchestra.run.smithi078.stdout: "modules": [ 2024-03-14T10:04:39.508 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2024-03-14T10:04:39.508 INFO:teuthology.orchestra.run.smithi078.stdout: "dashboard", 2024-03-14T10:04:39.508 INFO:teuthology.orchestra.run.smithi078.stdout: "iostat", 2024-03-14T10:04:39.508 INFO:teuthology.orchestra.run.smithi078.stdout: "nfs", 2024-03-14T10:04:39.508 INFO:teuthology.orchestra.run.smithi078.stdout: "restful", 2024-03-14T10:04:39.508 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_schedule" 2024-03-14T10:04:39.508 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.508 INFO:teuthology.orchestra.run.smithi078.stdout: "available_modules": [ 2024-03-14T10:04:39.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.508 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alerts", 2024-03-14T10:04:39.508 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.508 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.508 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.509 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-14T10:04:39.509 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-14T10:04:39.509 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.509 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.509 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.509 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-14T10:04:39.509 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.509 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.509 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.509 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How frequently to reexamine health status", 2024-03-14T10:04:39.509 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.509 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.509 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.510 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.510 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.510 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.510 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.510 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.510 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.510 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.510 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.510 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.510 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.510 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.510 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.510 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.511 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.511 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.511 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.511 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.511 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.511 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.511 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.511 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.511 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.511 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.511 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.512 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.512 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.512 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.512 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.512 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.512 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.512 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.512 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.512 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.512 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.512 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.512 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.512 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.513 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.513 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.513 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.513 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.513 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.513 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.513 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.513 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.513 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.513 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.513 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.513 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.513 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.513 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.514 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.514 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.514 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.514 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.514 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.514 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.514 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.514 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.514 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.514 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.514 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.515 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.515 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.515 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.515 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.515 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_destination": { 2024-03-14T10:04:39.515 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_destination", 2024-03-14T10:04:39.515 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.515 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.515 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.515 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.515 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.515 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.515 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.515 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email address to send alerts to", 2024-03-14T10:04:39.515 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.516 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.516 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.516 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_from_name": { 2024-03-14T10:04:39.516 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_from_name", 2024-03-14T10:04:39.516 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.516 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.516 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.516 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "Ceph", 2024-03-14T10:04:39.516 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.516 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.516 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.516 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email From: name", 2024-03-14T10:04:39.516 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.516 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.516 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.517 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_host": { 2024-03-14T10:04:39.517 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_host", 2024-03-14T10:04:39.517 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.517 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.517 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.517 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.517 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.517 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.517 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.517 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP server", 2024-03-14T10:04:39.517 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.517 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.517 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.518 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_password": { 2024-03-14T10:04:39.518 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_password", 2024-03-14T10:04:39.518 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.518 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.518 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.518 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.518 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.518 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.518 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.518 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Password to authenticate with", 2024-03-14T10:04:39.518 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.518 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.518 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.519 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_port": { 2024-03-14T10:04:39.519 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_port", 2024-03-14T10:04:39.519 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.519 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.519 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.519 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "465", 2024-03-14T10:04:39.519 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.519 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.519 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.519 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP port", 2024-03-14T10:04:39.519 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.519 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.519 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.520 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_sender": { 2024-03-14T10:04:39.520 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_sender", 2024-03-14T10:04:39.520 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.520 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.520 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.520 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.520 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.520 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.520 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.520 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP envelope sender", 2024-03-14T10:04:39.520 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.520 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.520 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.521 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_ssl": { 2024-03-14T10:04:39.521 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_ssl", 2024-03-14T10:04:39.521 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.521 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.521 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.521 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.521 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.521 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.521 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.521 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-14T10:04:39.521 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.521 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.521 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.521 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_user": { 2024-03-14T10:04:39.522 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_user", 2024-03-14T10:04:39.522 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.522 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.522 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.522 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.522 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.522 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.522 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.522 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "User to authenticate as", 2024-03-14T10:04:39.522 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.522 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.522 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.523 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "balancer", 2024-03-14T10:04:39.523 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.523 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.523 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.523 INFO:teuthology.orchestra.run.smithi078.stdout: "active": { 2024-03-14T10:04:39.523 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "active", 2024-03-14T10:04:39.523 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.523 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.523 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.523 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.523 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.523 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.523 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.524 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically balance PGs across cluster", 2024-03-14T10:04:39.524 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.524 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.524 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.524 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_time": { 2024-03-14T10:04:39.524 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_time", 2024-03-14T10:04:39.524 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.524 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.524 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.524 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0000", 2024-03-14T10:04:39.524 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.524 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.524 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.524 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "beginning time of day to automatically balance", 2024-03-14T10:04:39.525 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-14T10:04:39.525 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.525 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.525 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_weekday": { 2024-03-14T10:04:39.525 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_weekday", 2024-03-14T10:04:39.525 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-14T10:04:39.525 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.525 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.525 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-14T10:04:39.525 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-14T10:04:39.525 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-14T10:04:39.525 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.525 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-14T10:04:39.526 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-14T10:04:39.526 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.526 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.526 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_max_iterations": { 2024-03-14T10:04:39.526 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_max_iterations", 2024-03-14T10:04:39.526 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-14T10:04:39.526 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.526 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.526 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "25", 2024-03-14T10:04:39.526 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-14T10:04:39.526 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "250", 2024-03-14T10:04:39.526 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.526 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-14T10:04:39.527 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.527 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.527 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.527 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_metrics": { 2024-03-14T10:04:39.527 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_metrics", 2024-03-14T10:04:39.527 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.527 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.527 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.527 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "pgs,objects,bytes", 2024-03-14T10:04:39.527 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.527 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.527 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.527 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-14T10:04:39.528 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-14T10:04:39.528 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.528 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.528 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_step": { 2024-03-14T10:04:39.528 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_step", 2024-03-14T10:04:39.528 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:39.528 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.528 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.528 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.5", 2024-03-14T10:04:39.528 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0.001", 2024-03-14T10:04:39.528 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "0.999", 2024-03-14T10:04:39.528 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.529 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "aggressiveness of optimization", 2024-03-14T10:04:39.529 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-14T10:04:39.529 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.529 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.529 INFO:teuthology.orchestra.run.smithi078.stdout: "end_time": { 2024-03-14T10:04:39.529 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_time", 2024-03-14T10:04:39.529 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.529 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.529 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.529 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2359", 2024-03-14T10:04:39.529 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.529 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.529 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.529 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ending time of day to automatically balance", 2024-03-14T10:04:39.530 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-14T10:04:39.530 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.530 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.530 INFO:teuthology.orchestra.run.smithi078.stdout: "end_weekday": { 2024-03-14T10:04:39.530 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_weekday", 2024-03-14T10:04:39.530 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-14T10:04:39.530 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.530 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.530 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-14T10:04:39.530 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2024-03-14T10:04:39.530 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2024-03-14T10:04:39.530 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.531 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-14T10:04:39.531 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-14T10:04:39.531 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.531 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.531 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.531 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.531 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.531 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.531 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.531 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.531 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.532 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.532 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.532 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.532 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.532 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.532 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.532 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.532 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.532 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.532 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.532 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.532 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.532 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.533 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.533 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.533 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.533 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.533 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.533 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.533 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.533 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.533 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.533 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.533 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.533 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.533 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.534 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.534 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.534 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.534 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.534 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.534 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.534 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.534 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.534 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.534 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.534 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.534 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.534 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.534 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.534 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.535 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.535 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.535 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.535 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.535 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.535 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.535 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.535 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.535 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.535 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.535 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.535 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.535 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.536 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.536 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.536 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.536 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.536 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.536 INFO:teuthology.orchestra.run.smithi078.stdout: "min_score": { 2024-03-14T10:04:39.536 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_score", 2024-03-14T10:04:39.536 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:39.536 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.536 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.536 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-14T10:04:39.536 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.536 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.537 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.537 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-14T10:04:39.537 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.537 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.537 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.537 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-14T10:04:39.537 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-14T10:04:39.537 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.537 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.537 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.537 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "upmap", 2024-03-14T10:04:39.537 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.537 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.538 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.538 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-compat", 2024-03-14T10:04:39.538 INFO:teuthology.orchestra.run.smithi078.stdout: "none", 2024-03-14T10:04:39.538 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap" 2024-03-14T10:04:39.538 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.538 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Balancer mode", 2024-03-14T10:04:39.538 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.538 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.538 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.538 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_ids": { 2024-03-14T10:04:39.538 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_ids", 2024-03-14T10:04:39.538 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.538 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-14T10:04:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-14T10:04:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-14T10:04:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.539 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-14T10:04:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-14T10:04:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_deviation": { 2024-03-14T10:04:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_deviation", 2024-03-14T10:04:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.540 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-14T10:04:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-14T10:04:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-14T10:04:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-14T10:04:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_optimizations": { 2024-03-14T10:04:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_optimizations", 2024-03-14T10:04:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-14T10:04:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.541 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.542 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:39.542 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.542 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.542 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.542 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-14T10:04:39.542 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.542 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.542 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.542 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm", 2024-03-14T10:04:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_down_multiplier": { 2024-03-14T10:04:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_down_multiplier", 2024-03-14T10:04:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-14T10:04:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-14T10:04:39.543 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_refresh_rate": { 2024-03-14T10:04:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_refresh_rate", 2024-03-14T10:04:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "20", 2024-03-14T10:04:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-14T10:04:39.544 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_starting_port": { 2024-03-14T10:04:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_starting_port", 2024-03-14T10:04:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4721", 2024-03-14T10:04:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-14T10:04:39.545 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_ptrace": { 2024-03-14T10:04:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_ptrace", 2024-03-14T10:04:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-14T10:04:39.546 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-14T10:04:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_interval": { 2024-03-14T10:04:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_interval", 2024-03-14T10:04:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-14T10:04:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-14T10:04:39.547 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_memory_target_ratio": { 2024-03-14T10:04:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_memory_target_ratio", 2024-03-14T10:04:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.7", 2024-03-14T10:04:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-14T10:04:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.548 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: "cgroups_split": { 2024-03-14T10:04:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cgroups_split", 2024-03-14T10:04:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-14T10:04:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.549 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.550 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.550 INFO:teuthology.orchestra.run.smithi078.stdout: "config_checks_enabled": { 2024-03-14T10:04:39.550 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_checks_enabled", 2024-03-14T10:04:39.550 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.550 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.550 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.550 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.550 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.550 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.550 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.550 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-14T10:04:39.550 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.550 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.550 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.551 INFO:teuthology.orchestra.run.smithi078.stdout: "config_dashboard": { 2024-03-14T10:04:39.551 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_dashboard", 2024-03-14T10:04:39.551 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.551 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.551 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.551 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.551 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.551 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.551 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.551 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-14T10:04:39.551 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.551 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.551 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.552 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_alertmanager": { 2024-03-14T10:04:39.552 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_alertmanager", 2024-03-14T10:04:39.552 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.552 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.552 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.552 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-14T10:04:39.552 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.552 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.552 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.552 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-14T10:04:39.552 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.552 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.552 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.553 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_base": { 2024-03-14T10:04:39.553 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_base", 2024-03-14T10:04:39.553 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.553 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.553 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.553 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-14T10:04:39.553 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.553 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.553 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.553 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Container image name, without the tag", 2024-03-14T10:04:39.553 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.553 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.554 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.554 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_elasticsearch": { 2024-03-14T10:04:39.554 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_elasticsearch", 2024-03-14T10:04:39.554 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.554 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.554 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.554 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-14T10:04:39.554 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.554 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.554 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.554 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "elasticsearch container image", 2024-03-14T10:04:39.554 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.555 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.555 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.555 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_grafana": { 2024-03-14T10:04:39.555 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_grafana", 2024-03-14T10:04:39.555 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.555 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.555 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.555 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2024-03-14T10:04:39.555 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.555 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.555 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.555 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-14T10:04:39.555 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.555 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.556 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.556 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_haproxy": { 2024-03-14T10:04:39.556 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_haproxy", 2024-03-14T10:04:39.556 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.556 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.556 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.556 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-14T10:04:39.556 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.556 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.556 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.556 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "HAproxy container image", 2024-03-14T10:04:39.556 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.556 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.556 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.557 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_agent": { 2024-03-14T10:04:39.557 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_agent", 2024-03-14T10:04:39.557 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.557 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.557 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.557 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-14T10:04:39.557 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.557 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.557 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.557 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger agent container image", 2024-03-14T10:04:39.557 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.557 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.557 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.558 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_collector": { 2024-03-14T10:04:39.558 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_collector", 2024-03-14T10:04:39.558 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.558 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.558 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.558 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-14T10:04:39.558 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.558 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.558 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.558 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger collector container image", 2024-03-14T10:04:39.558 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.558 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.558 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.559 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_query": { 2024-03-14T10:04:39.559 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_query", 2024-03-14T10:04:39.559 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.559 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.559 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.559 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-14T10:04:39.559 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.559 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.559 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.559 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger query container image", 2024-03-14T10:04:39.559 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.559 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.559 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_keepalived": { 2024-03-14T10:04:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_keepalived", 2024-03-14T10:04:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-14T10:04:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Keepalived container image", 2024-03-14T10:04:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.560 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_loki": { 2024-03-14T10:04:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_loki", 2024-03-14T10:04:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-14T10:04:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Loki container image", 2024-03-14T10:04:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.561 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_node_exporter": { 2024-03-14T10:04:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_node_exporter", 2024-03-14T10:04:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-14T10:04:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-14T10:04:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_nvmeof": { 2024-03-14T10:04:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_nvmeof", 2024-03-14T10:04:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/nvmeof:0.0.2", 2024-03-14T10:04:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Nvme-of container image", 2024-03-14T10:04:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.563 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_prometheus": { 2024-03-14T10:04:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_prometheus", 2024-03-14T10:04:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-14T10:04:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2024-03-14T10:04:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_promtail": { 2024-03-14T10:04:39.564 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_promtail", 2024-03-14T10:04:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-14T10:04:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Promtail container image", 2024-03-14T10:04:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.565 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_snmp_gateway": { 2024-03-14T10:04:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_snmp_gateway", 2024-03-14T10:04:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-14T10:04:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SNMP Gateway container image", 2024-03-14T10:04:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "container_init": { 2024-03-14T10:04:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_init", 2024-03-14T10:04:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.567 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.569 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.569 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.569 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.569 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Run podman/docker with `--init`", 2024-03-14T10:04:39.569 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.570 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.570 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.570 INFO:teuthology.orchestra.run.smithi078.stdout: "daemon_cache_timeout": { 2024-03-14T10:04:39.570 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "daemon_cache_timeout", 2024-03-14T10:04:39.570 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.570 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.570 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.570 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-14T10:04:39.570 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.570 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.570 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.570 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-14T10:04:39.571 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.571 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.571 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.571 INFO:teuthology.orchestra.run.smithi078.stdout: "default_cephadm_command_timeout": { 2024-03-14T10:04:39.571 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_cephadm_command_timeout", 2024-03-14T10:04:39.571 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.571 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.571 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.571 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "900", 2024-03-14T10:04:39.571 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.571 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.571 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.571 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-14T10:04:39.572 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.572 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.572 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.572 INFO:teuthology.orchestra.run.smithi078.stdout: "default_registry": { 2024-03-14T10:04:39.572 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_registry", 2024-03-14T10:04:39.572 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.572 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.572 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.572 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io", 2024-03-14T10:04:39.572 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.572 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.572 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.573 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-14T10:04:39.573 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.573 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.573 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.573 INFO:teuthology.orchestra.run.smithi078.stdout: "device_cache_timeout": { 2024-03-14T10:04:39.573 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_cache_timeout", 2024-03-14T10:04:39.573 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.573 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.573 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.573 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1800", 2024-03-14T10:04:39.573 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.573 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.574 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.574 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache device inventory", 2024-03-14T10:04:39.574 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.574 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.574 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.574 INFO:teuthology.orchestra.run.smithi078.stdout: "device_enhanced_scan": { 2024-03-14T10:04:39.574 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_enhanced_scan", 2024-03-14T10:04:39.574 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.574 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.574 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.574 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.574 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.575 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.575 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.575 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-14T10:04:39.575 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.575 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.575 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.575 INFO:teuthology.orchestra.run.smithi078.stdout: "facts_cache_timeout": { 2024-03-14T10:04:39.575 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "facts_cache_timeout", 2024-03-14T10:04:39.575 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.575 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.575 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.575 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-14T10:04:39.575 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.575 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.576 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.576 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache host facts data", 2024-03-14T10:04:39.576 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.576 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.576 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.576 INFO:teuthology.orchestra.run.smithi078.stdout: "host_check_interval": { 2024-03-14T10:04:39.576 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "host_check_interval", 2024-03-14T10:04:39.576 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.576 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.576 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.576 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-14T10:04:39.576 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.576 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.577 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.577 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to perform a host check", 2024-03-14T10:04:39.577 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.577 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.577 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.577 INFO:teuthology.orchestra.run.smithi078.stdout: "hw_monitoring": { 2024-03-14T10:04:39.577 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hw_monitoring", 2024-03-14T10:04:39.577 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.577 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.577 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.577 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.577 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.577 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.577 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.578 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-14T10:04:39.578 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.578 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.578 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.578 INFO:teuthology.orchestra.run.smithi078.stdout: "inventory_list_all": { 2024-03-14T10:04:39.578 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "inventory_list_all", 2024-03-14T10:04:39.578 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.578 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.578 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.578 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.578 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.578 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.578 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.579 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-14T10:04:39.579 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.579 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.579 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.579 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.579 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.579 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.579 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.579 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.579 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.579 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.579 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.579 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.580 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.580 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.580 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.580 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.580 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.580 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.580 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.580 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.580 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.580 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.580 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.580 INFO:teuthology.orchestra.run.smithi078.stdout: "log_refresh_metadata": { 2024-03-14T10:04:39.580 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_refresh_metadata", 2024-03-14T10:04:39.581 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.581 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.581 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.581 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.581 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.581 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.581 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.581 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-14T10:04:39.581 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.581 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.581 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.581 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.581 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.581 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.582 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.582 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.582 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.582 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.582 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.582 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.582 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-14T10:04:39.582 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.582 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.582 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.582 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.582 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.583 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.583 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.583 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.583 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.583 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.583 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.583 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.583 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.583 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.583 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.583 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.583 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.583 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.583 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.584 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.584 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.584 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.584 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.584 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.584 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.584 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.584 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.584 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.584 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.584 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.584 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.584 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.585 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.585 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.585 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.585 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.585 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-14T10:04:39.585 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-14T10:04:39.585 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.585 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.585 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.585 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.585 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.585 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.585 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.586 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-14T10:04:39.586 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.586 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.586 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.586 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-14T10:04:39.586 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-14T10:04:39.586 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.586 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.586 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.586 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "*", 2024-03-14T10:04:39.586 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.586 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.587 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.587 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-14T10:04:39.587 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.587 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.587 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.587 INFO:teuthology.orchestra.run.smithi078.stdout: "max_count_per_host": { 2024-03-14T10:04:39.587 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_count_per_host", 2024-03-14T10:04:39.587 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.587 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.587 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.587 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:39.587 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.587 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.588 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.588 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of daemons per service per host", 2024-03-14T10:04:39.588 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.588 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.588 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.588 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd_draining_count": { 2024-03-14T10:04:39.588 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_osd_draining_count", 2024-03-14T10:04:39.588 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.588 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.588 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.588 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:39.588 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.589 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.589 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.589 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-14T10:04:39.589 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.589 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.589 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.589 INFO:teuthology.orchestra.run.smithi078.stdout: "migration_current": { 2024-03-14T10:04:39.589 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "migration_current", 2024-03-14T10:04:39.589 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.589 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.589 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.589 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.589 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.590 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.590 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.590 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "internal - do not modify", 2024-03-14T10:04:39.590 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.590 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.590 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.590 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2024-03-14T10:04:39.590 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2024-03-14T10:04:39.590 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.590 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.590 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.590 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "root", 2024-03-14T10:04:39.591 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.591 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.591 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.591 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm-package", 2024-03-14T10:04:39.591 INFO:teuthology.orchestra.run.smithi078.stdout: "root" 2024-03-14T10:04:39.591 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.591 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "mode for remote execution of cephadm", 2024-03-14T10:04:39.591 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.591 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.591 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.591 INFO:teuthology.orchestra.run.smithi078.stdout: "oob_default_addr": { 2024-03-14T10:04:39.591 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "oob_default_addr", 2024-03-14T10:04:39.591 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.592 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.592 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.592 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "169.254.1.1", 2024-03-14T10:04:39.592 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.592 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.592 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.592 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-14T10:04:39.592 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.592 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.592 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.592 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_alerts_path": { 2024-03-14T10:04:39.592 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_alerts_path", 2024-03-14T10:04:39.592 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.592 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.593 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.593 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-14T10:04:39.593 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.593 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.593 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.593 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-14T10:04:39.593 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.593 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.593 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.593 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_insecure": { 2024-03-14T10:04:39.593 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_insecure", 2024-03-14T10:04:39.593 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.593 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.594 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.594 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.594 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.594 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.594 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.594 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-14T10:04:39.594 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.594 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.594 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.594 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_password": { 2024-03-14T10:04:39.594 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_password", 2024-03-14T10:04:39.594 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.594 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.595 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.595 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.595 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.595 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.595 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.595 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-14T10:04:39.595 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.595 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.595 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.595 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_url": { 2024-03-14T10:04:39.595 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_url", 2024-03-14T10:04:39.595 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.595 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.595 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.596 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.596 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.596 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.596 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.596 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-14T10:04:39.596 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.596 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.596 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.596 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_username": { 2024-03-14T10:04:39.596 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_username", 2024-03-14T10:04:39.596 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.596 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.596 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.596 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.597 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.597 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.597 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.597 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-14T10:04:39.597 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.597 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.597 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.597 INFO:teuthology.orchestra.run.smithi078.stdout: "secure_monitoring_stack": { 2024-03-14T10:04:39.597 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secure_monitoring_stack", 2024-03-14T10:04:39.597 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.597 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.597 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.597 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.598 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.598 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.598 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.598 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-14T10:04:39.598 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.598 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.598 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.598 INFO:teuthology.orchestra.run.smithi078.stdout: "service_discovery_port": { 2024-03-14T10:04:39.598 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "service_discovery_port", 2024-03-14T10:04:39.598 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.598 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.598 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.598 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8765", 2024-03-14T10:04:39.599 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.599 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.599 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.599 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "cephadm service discovery port", 2024-03-14T10:04:39.599 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.599 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.599 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.599 INFO:teuthology.orchestra.run.smithi078.stdout: "ssh_config_file": { 2024-03-14T10:04:39.599 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssh_config_file", 2024-03-14T10:04:39.599 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.599 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.599 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.599 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.599 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.600 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.600 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.600 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-14T10:04:39.600 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.600 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.600 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.600 INFO:teuthology.orchestra.run.smithi078.stdout: "use_agent": { 2024-03-14T10:04:39.600 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_agent", 2024-03-14T10:04:39.600 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.600 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.600 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.600 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.601 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.601 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.601 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.601 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-14T10:04:39.601 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.601 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.601 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.601 INFO:teuthology.orchestra.run.smithi078.stdout: "use_repo_digest": { 2024-03-14T10:04:39.601 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_repo_digest", 2024-03-14T10:04:39.601 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.601 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.601 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.602 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.602 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.602 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.602 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.602 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-14T10:04:39.602 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.602 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.602 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.602 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_failed_host_check": { 2024-03-14T10:04:39.602 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_failed_host_check", 2024-03-14T10:04:39.602 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.602 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.602 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.603 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.603 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.603 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.603 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.603 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if the host check fails", 2024-03-14T10:04:39.603 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.603 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.603 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.603 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_daemons": { 2024-03-14T10:04:39.603 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_daemons", 2024-03-14T10:04:39.603 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.603 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.603 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.603 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.604 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.604 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.604 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.604 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-14T10:04:39.604 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.604 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.604 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.604 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_hosts": { 2024-03-14T10:04:39.604 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_hosts", 2024-03-14T10:04:39.604 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.604 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.604 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.604 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.604 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.604 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.605 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.605 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-14T10:04:39.605 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.605 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.605 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.605 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.605 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.605 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crash", 2024-03-14T10:04:39.605 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.605 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.605 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.605 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.606 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.606 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.606 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.606 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.606 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.606 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.606 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.606 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.606 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.606 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.606 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.606 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.606 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.607 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.608 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.609 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.609 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.609 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.609 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.609 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.609 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.609 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.609 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.609 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.609 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.609 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.609 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.609 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.609 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.610 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.610 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.610 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.610 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.610 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.610 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.610 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.610 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.610 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.610 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.610 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.610 INFO:teuthology.orchestra.run.smithi078.stdout: "retain_interval": { 2024-03-14T10:04:39.611 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retain_interval", 2024-03-14T10:04:39.611 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.611 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.611 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.611 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "31536000", 2024-03-14T10:04:39.611 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.611 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.611 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.611 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-14T10:04:39.611 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.611 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.611 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.611 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_recent_interval": { 2024-03-14T10:04:39.611 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_recent_interval", 2024-03-14T10:04:39.612 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.612 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.612 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.612 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1209600", 2024-03-14T10:04:39.612 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.612 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.612 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.612 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-14T10:04:39.612 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.612 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.612 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.613 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dashboard", 2024-03-14T10:04:39.613 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.613 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.613 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.613 INFO:teuthology.orchestra.run.smithi078.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-14T10:04:39.613 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-14T10:04:39.613 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.613 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.613 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.613 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:39.613 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.613 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.613 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.614 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.614 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.614 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.614 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.614 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_HOST": { 2024-03-14T10:04:39.614 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-14T10:04:39.614 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.614 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.614 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.614 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.614 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.614 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.615 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.615 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.615 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.615 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.615 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.615 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-14T10:04:39.615 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-14T10:04:39.615 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.615 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.615 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.615 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.615 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.616 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.616 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.616 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.616 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.616 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.616 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.616 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_ENABLED": { 2024-03-14T10:04:39.616 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_ENABLED", 2024-03-14T10:04:39.616 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.616 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.616 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.616 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.616 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.617 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.617 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.617 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.617 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.617 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.617 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.617 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-14T10:04:39.617 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-14T10:04:39.617 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.617 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.617 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.617 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.618 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.618 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.618 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.618 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.618 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.618 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.618 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.618 INFO:teuthology.orchestra.run.smithi078.stdout: "ENABLE_BROWSABLE_API": { 2024-03-14T10:04:39.618 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-14T10:04:39.618 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.618 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.619 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.619 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.619 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.619 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.619 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.619 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.619 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.619 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.619 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.619 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-14T10:04:39.619 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-14T10:04:39.619 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.619 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.620 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.620 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.620 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.620 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.620 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.620 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.620 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.620 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-14T10:04:39.620 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-14T10:04:39.620 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.620 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.621 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.621 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.621 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.621 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.621 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.621 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.621 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.621 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.621 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-14T10:04:39.621 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-14T10:04:39.621 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.621 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.621 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.621 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.621 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.622 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.622 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.622 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.622 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.622 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.622 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.622 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-14T10:04:39.622 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-14T10:04:39.622 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.622 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.622 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.622 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.623 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.623 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.623 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.623 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.623 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.623 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.623 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.623 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-14T10:04:39.623 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-14T10:04:39.623 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.623 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.623 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.623 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.623 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.623 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.624 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.625 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.625 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.625 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-14T10:04:39.625 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-14T10:04:39.625 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.625 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.625 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.625 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.625 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.625 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.625 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.625 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.625 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.625 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.626 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.626 INFO:teuthology.orchestra.run.smithi078.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-14T10:04:39.626 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-14T10:04:39.626 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.626 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.626 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.626 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.626 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.626 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.626 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.626 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.626 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.626 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.627 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.627 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_PASSWORD": { 2024-03-14T10:04:39.627 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-14T10:04:39.627 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.627 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.627 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.627 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-14T10:04:39.627 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.627 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.627 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.627 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.627 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.627 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.627 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.628 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-14T10:04:39.628 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-14T10:04:39.628 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.628 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.628 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.628 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.628 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.628 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.628 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.628 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.628 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.628 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.628 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.628 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_URL": { 2024-03-14T10:04:39.629 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_URL", 2024-03-14T10:04:39.629 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.629 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.629 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.629 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.629 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.629 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.629 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.629 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.629 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.629 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.629 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.629 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_USERNAME": { 2024-03-14T10:04:39.630 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-14T10:04:39.630 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.630 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.630 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.630 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-14T10:04:39.630 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.630 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.630 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.630 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.630 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.630 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.630 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.631 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-14T10:04:39.631 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-14T10:04:39.631 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.631 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.631 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.631 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.631 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.631 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.631 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.631 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.631 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.631 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.632 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.632 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-14T10:04:39.632 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-14T10:04:39.632 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.632 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.632 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.632 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.632 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.632 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.632 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.632 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.632 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.633 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.633 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.633 INFO:teuthology.orchestra.run.smithi078.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-14T10:04:39.633 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-14T10:04:39.633 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.633 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.633 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.633 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.633 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.633 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.633 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.633 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.633 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.634 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.634 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.634 INFO:teuthology.orchestra.run.smithi078.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-14T10:04:39.634 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-14T10:04:39.634 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.634 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.634 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.634 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.634 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.634 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.634 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.635 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.635 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.635 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.635 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.635 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_HOST": { 2024-03-14T10:04:39.635 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-14T10:04:39.635 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.635 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.635 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.635 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.635 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.636 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.636 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.636 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.636 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.636 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.636 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.636 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-14T10:04:39.636 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-14T10:04:39.636 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.637 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.637 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.637 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.637 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.637 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.637 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.637 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.637 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.637 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-14T10:04:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-14T10:04:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.638 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.639 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.639 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.639 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.639 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.639 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.639 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.639 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-14T10:04:39.639 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-14T10:04:39.639 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.640 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.640 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.640 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.640 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.640 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.640 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.640 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.640 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.641 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.641 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.641 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-14T10:04:39.641 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-14T10:04:39.641 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.641 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.641 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.641 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.641 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.641 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.641 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.642 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.642 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.642 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.642 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.642 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-14T10:04:39.642 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-14T10:04:39.642 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.642 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.642 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.642 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.642 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.642 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.642 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.643 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.643 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.643 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.643 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.643 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-14T10:04:39.643 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-14T10:04:39.643 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.643 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.643 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.643 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.643 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.643 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.643 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.644 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.644 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.644 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.644 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.644 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-14T10:04:39.644 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-14T10:04:39.644 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.644 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.644 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.644 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.644 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.644 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.644 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.645 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.645 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.645 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.645 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.645 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-14T10:04:39.645 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-14T10:04:39.645 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.645 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.645 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.645 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.645 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.645 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.645 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_ENABLED": { 2024-03-14T10:04:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-14T10:04:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.646 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-14T10:04:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-14T10:04:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-14T10:04:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.647 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.648 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.648 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.648 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.648 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.648 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.648 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-14T10:04:39.648 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-14T10:04:39.648 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.648 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.648 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.648 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:39.648 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.648 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.649 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.649 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.649 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.649 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.649 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.649 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-14T10:04:39.649 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-14T10:04:39.649 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.649 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.649 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.649 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8", 2024-03-14T10:04:39.649 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.649 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.650 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.650 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.650 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.650 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.650 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.650 INFO:teuthology.orchestra.run.smithi078.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-14T10:04:39.650 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-14T10:04:39.650 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.650 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.650 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.650 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "45", 2024-03-14T10:04:39.650 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.650 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.651 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.651 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.651 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.651 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.651 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.651 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ACCESS_KEY": { 2024-03-14T10:04:39.651 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-14T10:04:39.651 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.651 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.651 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.651 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.651 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.651 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.652 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.652 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.652 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.652 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.652 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.652 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-14T10:04:39.652 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-14T10:04:39.652 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.652 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.652 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.652 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2024-03-14T10:04:39.652 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.652 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.652 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.653 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.653 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.653 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.653 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.653 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SECRET_KEY": { 2024-03-14T10:04:39.653 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-14T10:04:39.653 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.653 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.653 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.653 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.653 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.653 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.653 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.653 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.654 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.654 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.654 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.654 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SSL_VERIFY": { 2024-03-14T10:04:39.654 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-14T10:04:39.654 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.654 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.654 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.654 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.654 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.654 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.654 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.655 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.655 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.655 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.655 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.655 INFO:teuthology.orchestra.run.smithi078.stdout: "UNSAFE_TLS_v1_2": { 2024-03-14T10:04:39.655 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-14T10:04:39.655 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.655 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.655 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.655 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.655 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.655 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.655 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.656 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.656 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.656 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.656 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.656 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-14T10:04:39.656 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-14T10:04:39.656 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.656 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.656 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.656 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-14T10:04:39.656 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.656 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.656 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.656 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.657 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.657 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.657 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.657 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-14T10:04:39.657 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-14T10:04:39.657 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.657 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.657 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.657 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:39.657 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.657 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.657 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.657 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.657 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.658 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.658 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.658 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-14T10:04:39.658 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-14T10:04:39.658 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.658 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.658 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.658 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-14T10:04:39.658 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.658 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.658 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.658 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.659 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.659 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.659 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.659 INFO:teuthology.orchestra.run.smithi078.stdout: "cross_origin_url": { 2024-03-14T10:04:39.659 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cross_origin_url", 2024-03-14T10:04:39.659 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.659 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.659 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.659 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.659 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.659 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.659 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.659 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.660 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.660 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.660 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.660 INFO:teuthology.orchestra.run.smithi078.stdout: "crt_file": { 2024-03-14T10:04:39.660 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crt_file", 2024-03-14T10:04:39.660 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.660 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.660 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.660 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.660 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.660 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.660 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.661 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.661 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.661 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.661 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.661 INFO:teuthology.orchestra.run.smithi078.stdout: "debug": { 2024-03-14T10:04:39.661 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "debug", 2024-03-14T10:04:39.661 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.661 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.661 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.661 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.661 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.661 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.661 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.662 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable/disable debug options", 2024-03-14T10:04:39.662 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.662 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.662 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.662 INFO:teuthology.orchestra.run.smithi078.stdout: "jwt_token_ttl": { 2024-03-14T10:04:39.662 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "jwt_token_ttl", 2024-03-14T10:04:39.662 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.662 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.662 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.662 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "28800", 2024-03-14T10:04:39.662 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.662 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.662 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.662 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.662 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.663 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.663 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.663 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-14T10:04:39.663 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-14T10:04:39.663 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.663 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.663 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.663 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.663 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.663 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.663 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.663 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.663 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.664 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.665 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.666 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.667 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "motd": { 2024-03-14T10:04:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "motd", 2024-03-14T10:04:39.668 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "The message of the day", 2024-03-14T10:04:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_resolve_ip_addr": { 2024-03-14T10:04:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "redirect_resolve_ip_addr", 2024-03-14T10:04:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.669 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-14T10:04:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-14T10:04:39.670 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-14T10:04:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.671 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-14T10:04:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-14T10:04:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8080", 2024-03-14T10:04:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-14T10:04:39.672 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-14T10:04:39.673 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.673 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.673 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.673 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.673 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.673 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.673 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.673 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.673 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.673 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.673 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.673 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl_server_port": { 2024-03-14T10:04:39.674 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl_server_port", 2024-03-14T10:04:39.674 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.674 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.674 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.674 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8443", 2024-03-14T10:04:39.674 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.674 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.674 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.674 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.674 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.674 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.674 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.674 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-14T10:04:39.675 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-14T10:04:39.675 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.675 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.675 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.675 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "redirect", 2024-03-14T10:04:39.675 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.675 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.675 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.675 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.675 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect" 2024-03-14T10:04:39.675 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.675 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.675 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.675 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.675 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.676 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-14T10:04:39.676 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-14T10:04:39.676 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.676 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.676 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.676 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-14T10:04:39.676 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-14T10:04:39.676 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-14T10:04:39.676 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.676 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.676 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.676 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.676 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.676 INFO:teuthology.orchestra.run.smithi078.stdout: "url_prefix": { 2024-03-14T10:04:39.676 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url_prefix", 2024-03-14T10:04:39.677 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.677 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.677 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.677 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.677 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.677 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.677 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.677 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.677 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.677 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.677 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.677 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.677 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.678 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2024-03-14T10:04:39.678 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.678 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.678 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.678 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_monitoring": { 2024-03-14T10:04:39.678 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_monitoring", 2024-03-14T10:04:39.678 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.678 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.678 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.678 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.678 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.678 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.678 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.678 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "monitor device health metrics", 2024-03-14T10:04:39.679 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.679 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.679 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.679 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.679 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.679 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.679 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.679 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.679 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.679 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.679 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.679 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.679 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.680 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.680 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.680 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.680 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.680 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.680 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.680 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.680 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.680 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.680 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.680 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.680 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.680 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.680 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.681 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.682 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.682 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.682 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.682 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.682 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.682 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.682 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.682 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.682 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.682 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.682 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.682 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.682 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.682 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.682 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.683 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.683 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.683 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.683 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.683 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.683 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.683 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.683 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.683 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.683 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.683 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.683 INFO:teuthology.orchestra.run.smithi078.stdout: "mark_out_threshold": { 2024-03-14T10:04:39.683 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mark_out_threshold", 2024-03-14T10:04:39.684 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.684 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.684 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.684 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2419200", 2024-03-14T10:04:39.684 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.684 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.684 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.684 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-14T10:04:39.684 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.684 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.684 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.684 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": { 2024-03-14T10:04:39.684 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_name", 2024-03-14T10:04:39.685 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.685 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.685 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.685 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "device_health_metrics", 2024-03-14T10:04:39.685 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.685 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.685 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.685 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-14T10:04:39.685 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.685 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.685 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.685 INFO:teuthology.orchestra.run.smithi078.stdout: "retention_period": { 2024-03-14T10:04:39.685 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retention_period", 2024-03-14T10:04:39.685 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.685 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.686 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.686 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15552000", 2024-03-14T10:04:39.686 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.686 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.686 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.686 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain device health metrics", 2024-03-14T10:04:39.686 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.686 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.686 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.686 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_frequency": { 2024-03-14T10:04:39.686 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_frequency", 2024-03-14T10:04:39.686 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.686 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.686 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.687 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-14T10:04:39.687 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.687 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.687 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.687 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-14T10:04:39.687 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.687 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.687 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.687 INFO:teuthology.orchestra.run.smithi078.stdout: "self_heal": { 2024-03-14T10:04:39.687 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "self_heal", 2024-03-14T10:04:39.687 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.687 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.687 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.687 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.687 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.688 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.688 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.688 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-14T10:04:39.688 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.688 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.688 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.688 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-14T10:04:39.688 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-14T10:04:39.688 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.688 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.688 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.688 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-14T10:04:39.688 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.688 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.689 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.689 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and check device health", 2024-03-14T10:04:39.689 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.689 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.689 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.689 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_threshold": { 2024-03-14T10:04:39.689 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_threshold", 2024-03-14T10:04:39.689 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.689 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.689 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.689 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7257600", 2024-03-14T10:04:39.689 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.689 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.689 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.689 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-14T10:04:39.690 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.690 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.690 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.690 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "diskprediction_local", 2024-03-14T10:04:39.690 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.690 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.690 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.690 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.690 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.690 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.690 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.691 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.692 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.692 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.692 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.692 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.692 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.692 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.692 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.692 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.692 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.692 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.692 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.692 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.692 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.693 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.694 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.694 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.694 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.694 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.694 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.694 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.694 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.694 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.694 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.694 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.694 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.694 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.694 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.694 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.695 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.695 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.695 INFO:teuthology.orchestra.run.smithi078.stdout: "predict_interval": { 2024-03-14T10:04:39.695 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predict_interval", 2024-03-14T10:04:39.695 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.695 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.695 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.695 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2024-03-14T10:04:39.695 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.695 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.695 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.695 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.695 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.695 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.695 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.696 INFO:teuthology.orchestra.run.smithi078.stdout: "predictor_model": { 2024-03-14T10:04:39.696 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predictor_model", 2024-03-14T10:04:39.696 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.696 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.696 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.696 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "prophetstor", 2024-03-14T10:04:39.696 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.696 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.696 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.696 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.696 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.696 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.696 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.696 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-14T10:04:39.697 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-14T10:04:39.697 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.697 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.697 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.697 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2024-03-14T10:04:39.697 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.697 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.697 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.697 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.697 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.697 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.697 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.697 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.697 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.698 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "influx", 2024-03-14T10:04:39.698 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": false, 2024-03-14T10:04:39.698 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "influxdb python module not found", 2024-03-14T10:04:39.698 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.698 INFO:teuthology.orchestra.run.smithi078.stdout: "batch_size": { 2024-03-14T10:04:39.698 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "batch_size", 2024-03-14T10:04:39.698 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.698 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.698 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.698 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5000", 2024-03-14T10:04:39.698 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.698 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.698 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.698 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-14T10:04:39.699 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.699 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.699 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.699 INFO:teuthology.orchestra.run.smithi078.stdout: "database": { 2024-03-14T10:04:39.699 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "database", 2024-03-14T10:04:39.699 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.699 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.699 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.699 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "ceph", 2024-03-14T10:04:39.699 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.699 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.699 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.699 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-14T10:04:39.700 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.700 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.700 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.700 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": { 2024-03-14T10:04:39.700 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hostname", 2024-03-14T10:04:39.700 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.700 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.700 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.700 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.700 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.700 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.700 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.700 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server hostname", 2024-03-14T10:04:39.701 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.701 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.701 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.701 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-14T10:04:39.701 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-14T10:04:39.701 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.701 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.701 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.701 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "30", 2024-03-14T10:04:39.701 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "5", 2024-03-14T10:04:39.701 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.701 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.702 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-14T10:04:39.702 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.702 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.702 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.702 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.702 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.702 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.702 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.702 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.702 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.702 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.702 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.702 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.703 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.703 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.703 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.703 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.703 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.703 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.703 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.703 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.703 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.703 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.703 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.703 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.704 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.704 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.704 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.704 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.704 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.704 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.704 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.704 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.704 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.704 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.704 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.704 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.705 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.706 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.707 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.707 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.707 INFO:teuthology.orchestra.run.smithi078.stdout: "password": { 2024-03-14T10:04:39.707 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "password", 2024-03-14T10:04:39.707 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.707 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.707 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.707 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.707 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.707 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.707 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.707 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "password of InfluxDB server user", 2024-03-14T10:04:39.707 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.707 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.707 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.708 INFO:teuthology.orchestra.run.smithi078.stdout: "port": { 2024-03-14T10:04:39.708 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "port", 2024-03-14T10:04:39.708 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.708 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.708 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.708 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8086", 2024-03-14T10:04:39.708 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.708 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.708 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.708 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server port", 2024-03-14T10:04:39.708 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.708 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.708 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.709 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2024-03-14T10:04:39.709 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2024-03-14T10:04:39.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.709 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.709 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "false", 2024-03-14T10:04:39.709 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.709 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.709 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.709 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-14T10:04:39.709 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.710 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.710 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.710 INFO:teuthology.orchestra.run.smithi078.stdout: "threads": { 2024-03-14T10:04:39.710 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threads", 2024-03-14T10:04:39.710 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.710 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.710 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.710 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-14T10:04:39.710 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-14T10:04:39.710 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "32", 2024-03-14T10:04:39.710 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.711 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-14T10:04:39.711 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.711 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.711 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.711 INFO:teuthology.orchestra.run.smithi078.stdout: "username": { 2024-03-14T10:04:39.711 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "username", 2024-03-14T10:04:39.711 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.711 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.711 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.711 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.711 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.711 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.711 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.711 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "username of InfluxDB server user", 2024-03-14T10:04:39.712 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.712 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.712 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.712 INFO:teuthology.orchestra.run.smithi078.stdout: "verify_ssl": { 2024-03-14T10:04:39.712 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "verify_ssl", 2024-03-14T10:04:39.712 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.712 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.712 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.712 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "true", 2024-03-14T10:04:39.712 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.712 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.712 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.712 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-14T10:04:39.712 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.713 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.713 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.713 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "insights", 2024-03-14T10:04:39.713 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.713 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.713 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.713 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.713 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.714 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.714 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.714 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.714 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.714 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.714 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.714 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.714 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.714 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.714 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.714 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.714 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.714 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.714 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.715 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.716 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.716 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.716 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.716 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.716 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.716 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.716 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.716 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.716 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.716 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.716 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.716 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.717 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.718 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.719 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.719 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "iostat", 2024-03-14T10:04:39.719 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.719 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.719 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.719 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.719 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.719 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.719 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.719 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.719 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.719 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.719 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.721 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.721 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.721 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.721 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.721 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.721 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.721 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.721 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.721 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.721 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.721 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.721 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.721 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.721 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.721 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "k8sevents", 2024-03-14T10:04:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_event_retention_days": { 2024-03-14T10:04:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ceph_event_retention_days", 2024-03-14T10:04:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7", 2024-03-14T10:04:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-14T10:04:39.724 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "config_check_secs": { 2024-03-14T10:04:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_check_secs", 2024-03-14T10:04:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "10", 2024-03-14T10:04:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-14T10:04:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.725 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.726 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.727 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.728 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "localpool", 2024-03-14T10:04:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "failure_domain": { 2024-03-14T10:04:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "failure_domain", 2024-03-14T10:04:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "host", 2024-03-14T10:04:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.731 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.732 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "failure domain for any created local pool", 2024-03-14T10:04:39.732 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-14T10:04:39.732 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.732 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.732 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.732 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.732 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.732 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.732 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.732 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.732 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.732 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.732 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.733 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.733 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.733 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.733 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.733 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.733 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.733 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.733 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.733 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.733 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.733 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.733 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.733 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.733 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.733 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.734 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.734 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.734 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.734 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.734 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.734 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.734 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.734 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.734 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.734 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.734 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.734 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.735 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.735 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.735 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.735 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.735 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.735 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.735 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.735 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.735 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.735 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.735 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.735 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.735 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.735 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.735 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.736 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.736 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.736 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.736 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.736 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.736 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.736 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.736 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.736 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.736 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.736 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.736 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.737 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.737 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.737 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.737 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": { 2024-03-14T10:04:39.737 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_size", 2024-03-14T10:04:39.737 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.737 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.737 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.737 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.737 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.737 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.737 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.737 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default min_size for any created local pool", 2024-03-14T10:04:39.737 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-14T10:04:39.738 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.738 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.738 INFO:teuthology.orchestra.run.smithi078.stdout: "num_rep": { 2024-03-14T10:04:39.738 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "num_rep", 2024-03-14T10:04:39.738 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.738 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.738 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.738 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3", 2024-03-14T10:04:39.738 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.738 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.738 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.738 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default replica count for any created local pool", 2024-03-14T10:04:39.738 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.739 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.739 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.739 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": { 2024-03-14T10:04:39.739 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_num", 2024-03-14T10:04:39.739 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.739 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.739 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.739 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "128", 2024-03-14T10:04:39.739 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.739 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.739 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.739 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default pg_num for any created local pool", 2024-03-14T10:04:39.739 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.740 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.740 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.740 INFO:teuthology.orchestra.run.smithi078.stdout: "prefix": { 2024-03-14T10:04:39.740 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prefix", 2024-03-14T10:04:39.740 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.740 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.740 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.740 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.740 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.740 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.740 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.740 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name prefix for any created local pool", 2024-03-14T10:04:39.741 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.741 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.741 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.741 INFO:teuthology.orchestra.run.smithi078.stdout: "subtree": { 2024-03-14T10:04:39.741 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "subtree", 2024-03-14T10:04:39.741 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.741 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.741 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.741 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "rack", 2024-03-14T10:04:39.741 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.741 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.741 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.741 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-14T10:04:39.741 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-14T10:04:39.742 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.742 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.742 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.742 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.742 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mds_autoscaler", 2024-03-14T10:04:39.742 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.742 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.742 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.742 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.742 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.742 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.743 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.743 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.743 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.743 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.743 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.743 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.743 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.743 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.743 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.743 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.743 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.743 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.743 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.743 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.744 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.744 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.744 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.744 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.744 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.744 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.744 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.744 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.744 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.744 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.744 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.744 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.744 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.744 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.745 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.745 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.745 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.745 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.745 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.745 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.745 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.745 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.745 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.745 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.745 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.745 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.745 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.745 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.746 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.746 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.746 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.746 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.746 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.746 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.746 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.746 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.746 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.746 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.746 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.746 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.746 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.747 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.747 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.747 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.747 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.747 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.747 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.747 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.747 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.747 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.747 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.747 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirroring", 2024-03-14T10:04:39.747 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.748 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.748 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.748 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.748 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.748 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.748 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.748 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.748 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.748 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.748 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.748 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.748 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.748 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.748 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.749 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.749 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.749 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.749 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.749 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.749 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.749 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.749 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.749 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.749 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.749 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.749 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.749 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.750 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.750 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.750 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.750 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.750 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.750 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.750 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.750 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.750 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.750 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.750 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.750 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.751 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.751 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.751 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.751 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.751 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.751 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.751 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.751 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.751 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.751 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.751 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.751 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.751 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.751 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.751 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.751 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.752 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.752 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.752 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.752 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.752 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.752 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.752 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.752 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.752 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.752 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.752 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.752 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.752 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.753 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "nfs", 2024-03-14T10:04:39.753 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.753 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.753 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.753 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.753 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.753 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.753 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.753 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.754 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.754 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.754 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.754 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.754 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.754 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.754 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.754 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.754 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.754 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.754 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.754 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.754 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.754 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.755 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.755 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.755 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.755 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.755 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.755 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.755 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.755 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.755 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.755 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.755 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.755 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.755 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.756 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.757 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.758 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-14T10:04:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "fail_fs": { 2024-03-14T10:04:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "fail_fs", 2024-03-14T10:04:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-14T10:04:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.759 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.760 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.761 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.762 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.763 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.764 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator": { 2024-03-14T10:04:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2024-03-14T10:04:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2024-03-14T10:04:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "rook", 2024-03-14T10:04:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "test_orchestrator" 2024-03-14T10:04:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.765 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Orchestrator backend", 2024-03-14T10:04:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_perf_query", 2024-03-14T10:04:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.766 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.767 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.768 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.769 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.770 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.771 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.772 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.772 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.773 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_support", 2024-03-14T10:04:39.773 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.773 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.774 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.775 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.776 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.777 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_autoscaler", 2024-03-14T10:04:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.779 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.780 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.781 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.782 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.783 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-14T10:04:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-14T10:04:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-14T10:04:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.784 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "threshold": { 2024-03-14T10:04:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threshold", 2024-03-14T10:04:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2024-03-14T10:04:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1.0", 2024-03-14T10:04:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.785 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "scaling threshold", 2024-03-14T10:04:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-14T10:04:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "progress", 2024-03-14T10:04:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.786 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_pg_recovery_event": { 2024-03-14T10:04:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_pg_recovery_event", 2024-03-14T10:04:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-14T10:04:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.787 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-14T10:04:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-14T10:04:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.788 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.789 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.790 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.791 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.792 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.793 INFO:teuthology.orchestra.run.smithi078.stdout: "max_completed_events": { 2024-03-14T10:04:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_completed_events", 2024-03-14T10:04:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "50", 2024-03-14T10:04:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "number of past completed events to remember", 2024-03-14T10:04:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2024-03-14T10:04:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2024-03-14T10:04:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2024-03-14T10:04:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long the module is going to sleep", 2024-03-14T10:04:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.795 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus", 2024-03-14T10:04:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "cache": { 2024-03-14T10:04:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cache", 2024-03-14T10:04:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.796 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "exclude_perf_counters": { 2024-03-14T10:04:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "exclude_perf_counters", 2024-03-14T10:04:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.797 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-14T10:04:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-14T10:04:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.798 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.799 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.801 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.802 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools": { 2024-03-14T10:04:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools", 2024-03-14T10:04:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.803 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-14T10:04:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-14T10:04:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300", 2024-03-14T10:04:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.804 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_interval": { 2024-03-14T10:04:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_interval", 2024-03-14T10:04:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15.0", 2024-03-14T10:04:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.805 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-14T10:04:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-14T10:04:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2024-03-14T10:04:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.806 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-14T10:04:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-14T10:04:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-14T10:04:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "9283", 2024-03-14T10:04:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.807 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-14T10:04:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "stale_cache_strategy": { 2024-03-14T10:04:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stale_cache_strategy", 2024-03-14T10:04:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "log", 2024-03-14T10:04:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.808 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2024-03-14T10:04:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2024-03-14T10:04:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "default", 2024-03-14T10:04:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.809 INFO:teuthology.orchestra.run.smithi078.stdout: "default", 2024-03-14T10:04:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "error" 2024-03-14T10:04:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2024-03-14T10:04:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2024-03-14T10:04:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2024-03-14T10:04:39.810 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2024-03-14T10:04:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2024-03-14T10:04:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2024-03-14T10:04:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.811 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.812 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.813 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.814 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.815 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.816 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_snap_create": { 2024-03-14T10:04:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_snap_create", 2024-03-14T10:04:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2024-03-14T10:04:39.817 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "mirror_snapshot_schedule": { 2024-03-14T10:04:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirror_snapshot_schedule", 2024-03-14T10:04:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.818 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "trash_purge_schedule": { 2024-03-14T10:04:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "trash_purge_schedule", 2024-03-14T10:04:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.819 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.820 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "restful", 2024-03-14T10:04:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_auth": { 2024-03-14T10:04:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_auth", 2024-03-14T10:04:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2024-03-14T10:04:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2024-03-14T10:04:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.822 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.823 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.826 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.827 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2024-03-14T10:04:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2024-03-14T10:04:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.828 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2024-03-14T10:04:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2024-03-14T10:04:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.829 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw", 2024-03-14T10:04:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.830 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.831 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.832 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.833 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.834 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rook", 2024-03-14T10:04:39.835 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.836 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.837 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.838 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.839 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.840 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_class": { 2024-03-14T10:04:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "storage_class", 2024-03-14T10:04:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "local", 2024-03-14T10:04:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.841 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-14T10:04:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "selftest", 2024-03-14T10:04:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.842 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.843 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.844 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.845 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.846 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "roption1": { 2024-03-14T10:04:39.847 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption1", 2024-03-14T10:04:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "roption2": { 2024-03-14T10:04:39.848 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption2", 2024-03-14T10:04:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "xyz", 2024-03-14T10:04:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption1": { 2024-03-14T10:04:39.849 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption1", 2024-03-14T10:04:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption2": { 2024-03-14T10:04:39.850 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption2", 2024-03-14T10:04:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption3": { 2024-03-14T10:04:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption3", 2024-03-14T10:04:39.851 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2024-03-14T10:04:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption4": { 2024-03-14T10:04:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption4", 2024-03-14T10:04:39.852 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption5": { 2024-03-14T10:04:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption5", 2024-03-14T10:04:39.853 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption6": { 2024-03-14T10:04:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption6", 2024-03-14T10:04:39.854 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption7": { 2024-03-14T10:04:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption7", 2024-03-14T10:04:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.855 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2024-03-14T10:04:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "42", 2024-03-14T10:04:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: "testkey": { 2024-03-14T10:04:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testkey", 2024-03-14T10:04:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.856 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: "testlkey": { 2024-03-14T10:04:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testlkey", 2024-03-14T10:04:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.857 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: "testnewline": { 2024-03-14T10:04:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testnewline", 2024-03-14T10:04:39.858 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-03-14T10:04:39.860 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.860 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.860 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.860 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_m_granularity": { 2024-03-14T10:04:39.860 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_m_granularity", 2024-03-14T10:04:39.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.860 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.860 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.860 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.860 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.860 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.860 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.860 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow minute scheduled snapshots", 2024-03-14T10:04:39.860 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.861 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.861 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.861 INFO:teuthology.orchestra.run.smithi078.stdout: "dump_on_update": { 2024-03-14T10:04:39.861 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dump_on_update", 2024-03-14T10:04:39.861 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.861 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.861 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.861 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.861 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.861 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.861 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.861 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "dump database to debug log on update", 2024-03-14T10:04:39.862 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.862 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.862 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.862 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.862 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.862 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.862 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.862 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.862 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.862 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.862 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.862 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.862 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.863 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.863 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.863 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.863 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.863 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.863 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.863 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.863 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.863 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.863 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.863 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.863 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.863 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.864 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.864 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.864 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.864 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.864 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.864 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.864 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.864 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.864 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.864 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.864 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.864 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.864 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.865 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.865 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.865 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.865 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.865 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.865 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.865 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.865 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.865 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.865 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.865 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.865 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.865 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.865 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.865 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.866 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.866 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.866 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.866 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.866 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.866 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.866 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.866 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.866 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.866 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.866 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.866 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.866 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.867 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.867 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.867 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stats", 2024-03-14T10:04:39.867 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.867 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.867 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.867 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.867 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.867 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.868 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.868 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.868 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.868 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.868 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.868 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.868 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.868 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.868 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.868 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.868 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.868 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.868 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.869 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.869 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.869 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.869 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.869 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.869 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.869 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.869 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.869 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.869 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.869 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.869 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.869 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.869 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.869 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.870 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.870 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.870 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.870 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.870 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.870 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.870 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.870 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.870 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.870 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.870 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.870 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.870 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.871 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.871 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.871 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.871 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.871 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.871 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.871 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.871 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.871 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.871 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.871 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.871 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.871 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.871 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.872 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.872 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.872 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.872 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.872 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.872 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.872 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.872 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.872 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "status", 2024-03-14T10:04:39.872 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.873 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.873 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.873 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.873 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.873 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.873 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.873 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.873 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.873 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.873 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.873 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.873 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.873 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.873 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.873 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.873 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.874 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.874 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.874 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.874 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.874 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.874 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.874 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.874 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.874 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.874 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.874 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.874 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.874 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.875 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.876 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.876 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.876 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.876 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.876 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.876 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.876 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.876 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.876 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.876 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.876 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.876 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.876 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.876 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.876 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.877 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.877 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.877 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.877 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.877 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.877 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.877 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.877 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.877 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telegraf", 2024-03-14T10:04:39.877 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.877 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.877 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.878 INFO:teuthology.orchestra.run.smithi078.stdout: "address": { 2024-03-14T10:04:39.878 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "address", 2024-03-14T10:04:39.878 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.878 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.878 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.878 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-14T10:04:39.878 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.878 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.878 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.878 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.878 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.878 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.878 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.878 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15", 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.879 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.880 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.880 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.880 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.880 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.880 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.880 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.880 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.880 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.880 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.880 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.880 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.880 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.880 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.880 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.880 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.881 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.881 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.881 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.881 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.881 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.881 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.881 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.881 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.881 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.881 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.881 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.881 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.881 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.881 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.882 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.882 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.882 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.882 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.882 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.882 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.882 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.882 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.882 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.882 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.882 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.882 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.882 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.882 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.883 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.883 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.883 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.883 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.883 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.883 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.883 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.883 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.883 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.883 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.883 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.883 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.883 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.884 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.884 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.884 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.884 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.884 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.884 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.884 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telemetry", 2024-03-14T10:04:39.884 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.884 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.884 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.885 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_basic": { 2024-03-14T10:04:39.885 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_basic", 2024-03-14T10:04:39.885 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.885 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.885 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.885 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.885 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.885 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.885 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.885 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-14T10:04:39.885 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.885 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.885 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.886 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_crash": { 2024-03-14T10:04:39.886 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_crash", 2024-03-14T10:04:39.886 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.886 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.886 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.886 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.886 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.886 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.886 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-14T10:04:39.886 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.886 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.887 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.887 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_device": { 2024-03-14T10:04:39.887 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_device", 2024-03-14T10:04:39.887 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.887 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.887 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.887 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2024-03-14T10:04:39.887 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.887 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.887 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.888 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-14T10:04:39.888 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.888 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.888 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.888 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_ident": { 2024-03-14T10:04:39.888 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_ident", 2024-03-14T10:04:39.888 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.888 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.888 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.888 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.888 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.888 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.889 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.889 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-14T10:04:39.889 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.889 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.889 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.889 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_perf": { 2024-03-14T10:04:39.889 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_perf", 2024-03-14T10:04:39.889 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.889 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.889 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.889 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.889 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.889 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.890 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.890 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-14T10:04:39.890 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.890 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.890 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.890 INFO:teuthology.orchestra.run.smithi078.stdout: "contact": { 2024-03-14T10:04:39.890 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "contact", 2024-03-14T10:04:39.890 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.890 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.890 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.890 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.890 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: "description": { 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "description", 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.891 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.892 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.892 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.892 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.892 INFO:teuthology.orchestra.run.smithi078.stdout: "device_url": { 2024-03-14T10:04:39.892 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_url", 2024-03-14T10:04:39.892 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.892 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.892 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.892 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-14T10:04:39.892 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.892 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.892 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.892 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.892 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.892 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.893 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.893 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2024-03-14T10:04:39.893 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2024-03-14T10:04:39.893 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.893 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.893 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.893 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.893 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.893 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.893 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.893 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.893 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.893 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.893 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.894 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-14T10:04:39.894 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-14T10:04:39.894 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.894 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.894 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.894 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "24", 2024-03-14T10:04:39.894 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "8", 2024-03-14T10:04:39.894 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.894 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.894 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.894 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.894 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.894 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.894 INFO:teuthology.orchestra.run.smithi078.stdout: "last_opt_revision": { 2024-03-14T10:04:39.895 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "last_opt_revision", 2024-03-14T10:04:39.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.895 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.895 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.895 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1", 2024-03-14T10:04:39.895 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.895 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.895 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.895 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.895 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.895 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.895 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.895 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard": { 2024-03-14T10:04:39.895 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard", 2024-03-14T10:04:39.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.896 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.896 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.896 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.896 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.896 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.896 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.896 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.896 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.896 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.896 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.896 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard_description": { 2024-03-14T10:04:39.896 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard_description", 2024-03-14T10:04:39.896 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.896 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.897 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.897 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.897 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.897 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.897 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.897 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.897 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.897 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.897 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.897 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.897 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.897 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.897 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.897 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.898 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.898 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.898 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.898 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.898 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.898 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.898 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.898 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.898 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.898 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.898 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.898 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.898 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.898 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.898 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.899 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.899 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.899 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.899 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.899 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.899 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.899 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.899 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.899 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.899 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.899 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.899 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.899 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.899 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.900 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.900 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.900 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.900 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.900 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.900 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.900 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.900 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.900 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.900 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.900 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.900 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.900 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.900 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.901 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.901 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.901 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.901 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.901 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.901 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.901 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.901 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.901 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.901 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.901 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.901 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.901 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.901 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.902 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.902 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.902 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.902 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.902 INFO:teuthology.orchestra.run.smithi078.stdout: "organization": { 2024-03-14T10:04:39.902 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "organization", 2024-03-14T10:04:39.902 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.902 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.902 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.902 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.902 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.902 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.902 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.903 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.903 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.903 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.903 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.903 INFO:teuthology.orchestra.run.smithi078.stdout: "proxy": { 2024-03-14T10:04:39.903 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "proxy", 2024-03-14T10:04:39.903 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.903 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.903 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.903 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.903 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.903 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.903 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.904 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.904 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.904 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.904 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.904 INFO:teuthology.orchestra.run.smithi078.stdout: "url": { 2024-03-14T10:04:39.904 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url", 2024-03-14T10:04:39.904 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.904 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.904 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.904 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-14T10:04:39.904 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.904 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.904 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.904 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.905 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.905 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.905 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.905 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.905 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.905 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test_orchestrator", 2024-03-14T10:04:39.905 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.905 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.905 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.905 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.905 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.905 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.905 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.906 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.906 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.906 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.906 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.906 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.906 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.906 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.906 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.906 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.906 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.906 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.906 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.906 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.906 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.907 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.908 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.908 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.908 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.908 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.908 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.908 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.908 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.908 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.908 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.908 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.908 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.908 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.908 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.908 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.909 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.909 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.909 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.909 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.909 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.909 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.909 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.909 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.909 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.909 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.909 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.909 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.909 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.909 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.910 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.910 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.910 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.910 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.910 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.910 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.910 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2024-03-14T10:04:39.910 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.910 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.910 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.911 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.911 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.911 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.911 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.911 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.911 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.911 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.911 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.911 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.911 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.911 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.911 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.911 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.912 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.912 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.912 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.912 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.912 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.912 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.912 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.912 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.912 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.912 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.912 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.912 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.912 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.912 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.913 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.913 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.913 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.913 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.913 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.913 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.913 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.913 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.913 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.913 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.913 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.913 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.913 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.914 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.914 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.914 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.914 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.914 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.914 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.914 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.914 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.914 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.914 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.914 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.914 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.914 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.915 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.915 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.915 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.915 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.915 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.915 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.915 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.915 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.915 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.915 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.915 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.916 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_clones": { 2024-03-14T10:04:39.916 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_clones", 2024-03-14T10:04:39.916 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.916 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.916 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.916 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4", 2024-03-14T10:04:39.916 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.916 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.916 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-14T10:04:39.916 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.917 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.917 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.917 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_delay": { 2024-03-14T10:04:39.917 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_delay", 2024-03-14T10:04:39.917 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.917 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.917 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.917 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2024-03-14T10:04:39.917 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.917 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.917 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.917 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-14T10:04:39.917 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.918 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.918 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.918 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.918 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.918 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix", 2024-03-14T10:04:39.918 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2024-03-14T10:04:39.918 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2024-03-14T10:04:39.918 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2024-03-14T10:04:39.918 INFO:teuthology.orchestra.run.smithi078.stdout: "discovery_interval": { 2024-03-14T10:04:39.918 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "discovery_interval", 2024-03-14T10:04:39.918 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2024-03-14T10:04:39.919 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.919 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.919 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "100", 2024-03-14T10:04:39.919 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.919 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.919 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.919 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.919 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.919 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.919 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.919 INFO:teuthology.orchestra.run.smithi078.stdout: "identifier": { 2024-03-14T10:04:39.919 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "identifier", 2024-03-14T10:04:39.920 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.920 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.920 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.920 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.920 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.920 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.920 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.920 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.920 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.920 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.920 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.920 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2024-03-14T10:04:39.920 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2024-03-14T10:04:39.920 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2024-03-14T10:04:39.921 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.921 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.921 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2024-03-14T10:04:39.921 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.921 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.921 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.921 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.921 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.921 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.921 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.921 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2024-03-14T10:04:39.921 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2024-03-14T10:04:39.921 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.922 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.922 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.922 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.922 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.922 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.922 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.922 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.922 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.922 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.922 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.922 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.922 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.923 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.923 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.923 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.923 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.923 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.923 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2024-03-14T10:04:39.923 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2024-03-14T10:04:39.923 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.923 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.923 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.923 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.923 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.923 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.923 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.924 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.924 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.924 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.924 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.924 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2024-03-14T10:04:39.924 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2024-03-14T10:04:39.924 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.924 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.924 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.924 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2024-03-14T10:04:39.924 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.924 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.924 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2024-03-14T10:04:39.924 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2024-03-14T10:04:39.924 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2024-03-14T10:04:39.925 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2024-03-14T10:04:39.925 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2024-03-14T10:04:39.925 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2024-03-14T10:04:39.925 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2024-03-14T10:04:39.925 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.925 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.925 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.925 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.925 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.925 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2024-03-14T10:04:39.925 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2024-03-14T10:04:39.925 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2024-03-14T10:04:39.925 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.925 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-03-14T10:04:39.925 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2024-03-14T10:04:39.926 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.926 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.926 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.926 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.926 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.926 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.926 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.926 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_host": { 2024-03-14T10:04:39.926 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_host", 2024-03-14T10:04:39.926 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.926 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.926 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.926 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2024-03-14T10:04:39.926 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.927 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.927 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.927 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.927 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.927 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.927 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.927 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_port": { 2024-03-14T10:04:39.927 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_port", 2024-03-14T10:04:39.927 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2024-03-14T10:04:39.927 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.927 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.927 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10051", 2024-03-14T10:04:39.927 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.927 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.928 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.928 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.928 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.928 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.928 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.928 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_sender": { 2024-03-14T10:04:39.928 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_sender", 2024-03-14T10:04:39.928 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2024-03-14T10:04:39.928 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2024-03-14T10:04:39.928 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2024-03-14T10:04:39.928 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-14T10:04:39.928 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2024-03-14T10:04:39.928 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2024-03-14T10:04:39.929 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2024-03-14T10:04:39.929 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2024-03-14T10:04:39.929 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2024-03-14T10:04:39.929 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2024-03-14T10:04:39.929 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2024-03-14T10:04:39.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.929 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.929 INFO:teuthology.orchestra.run.smithi078.stdout: "services": { 2024-03-14T10:04:39.929 INFO:teuthology.orchestra.run.smithi078.stdout: "dashboard": "https://172.21.15.78:8443/" 2024-03-14T10:04:39.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.929 INFO:teuthology.orchestra.run.smithi078.stdout: "always_on_modules": { 2024-03-14T10:04:39.930 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus": [ 2024-03-14T10:04:39.930 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-03-14T10:04:39.930 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-03-14T10:04:39.930 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-03-14T10:04:39.930 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-03-14T10:04:39.930 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-03-14T10:04:39.930 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-03-14T10:04:39.930 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-03-14T10:04:39.930 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-03-14T10:04:39.930 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-03-14T10:04:39.930 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-03-14T10:04:39.930 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.930 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific": [ 2024-03-14T10:04:39.931 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-03-14T10:04:39.931 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-03-14T10:04:39.931 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-03-14T10:04:39.931 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-03-14T10:04:39.931 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-03-14T10:04:39.931 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-03-14T10:04:39.931 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-03-14T10:04:39.931 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-03-14T10:04:39.931 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-03-14T10:04:39.931 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-03-14T10:04:39.931 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.931 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy": [ 2024-03-14T10:04:39.931 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-03-14T10:04:39.931 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-03-14T10:04:39.932 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-03-14T10:04:39.932 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-03-14T10:04:39.932 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-03-14T10:04:39.932 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-03-14T10:04:39.932 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-03-14T10:04:39.932 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-03-14T10:04:39.932 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-03-14T10:04:39.932 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-03-14T10:04:39.932 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-03-14T10:04:39.932 INFO:teuthology.orchestra.run.smithi078.stdout: "reef": [ 2024-03-14T10:04:39.932 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2024-03-14T10:04:39.932 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2024-03-14T10:04:39.932 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2024-03-14T10:04:39.932 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2024-03-14T10:04:39.932 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2024-03-14T10:04:39.932 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2024-03-14T10:04:39.933 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2024-03-14T10:04:39.933 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2024-03-14T10:04:39.933 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2024-03-14T10:04:39.933 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2024-03-14T10:04:39.933 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-14T10:04:39.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.933 INFO:teuthology.orchestra.run.smithi078.stdout: "last_failure_osd_epoch": 88, 2024-03-14T10:04:39.933 INFO:teuthology.orchestra.run.smithi078.stdout: "active_clients": [ 2024-03-14T10:04:39.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.933 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2024-03-14T10:04:39.933 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-14T10:04:39.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.933 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-14T10:04:39.933 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-14T10:04:39.933 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3539763221 2024-03-14T10:04:39.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.934 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-14T10:04:39.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.934 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "libcephsqlite", 2024-03-14T10:04:39.934 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-14T10:04:39.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.934 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-14T10:04:39.934 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-14T10:04:39.934 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3706105864 2024-03-14T10:04:39.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.934 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-14T10:04:39.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.934 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2024-03-14T10:04:39.935 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-14T10:04:39.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.935 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-14T10:04:39.935 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-14T10:04:39.935 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 4068386221 2024-03-14T10:04:39.935 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.935 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-14T10:04:39.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.935 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-03-14T10:04:39.935 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-14T10:04:39.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.935 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-14T10:04:39.935 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-14T10:04:39.935 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1595016889 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 646949211 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-03-14T10:04:39.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-03-14T10:04:39.937 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-03-14T10:04:39.937 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2024-03-14T10:04:39.937 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2159101844 2024-03-14T10:04:39.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.937 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-14T10:04:39.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:39.937 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-03-14T10:04:39.937 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:04:39.938 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:39 smithi132 ceph-mon[47959]: pgmap v325: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:39.938 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:39 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3313851237' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-14T10:04:39.939 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:39 smithi152 ceph-mon[48279]: pgmap v325: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:39.939 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:39 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3313851237' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-14T10:04:39.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.654+0000 7f68c1c00700 1 -- 172.21.15.78:0/3313851237 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f68a00a0c80 msgr2=0x7f68a00a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:39.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.654+0000 7f68c1c00700 1 --2- 172.21.15.78:0/3313851237 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f68a00a0c80 0x7f68a00a3140 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f68bc0fff60 tx=0x7f68ac00b260 comp rx=0 tx=0).stop 2024-03-14T10:04:39.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.654+0000 7f68c1c00700 1 -- 172.21.15.78:0/3313851237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68bc105f40 msgr2=0x7f68bc199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:04:39.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.654+0000 7f68c1c00700 1 --2- 172.21.15.78:0/3313851237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68bc105f40 0x7f68bc199d70 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f68a800e970 tx=0x7f68a800ec80 comp rx=0 tx=0).stop 2024-03-14T10:04:39.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.654+0000 7f68c1c00700 1 -- 172.21.15.78:0/3313851237 shutdown_connections 2024-03-14T10:04:39.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.654+0000 7f68c1c00700 1 --2- 172.21.15.78:0/3313851237 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f68a00a0c80 0x7f68a00a3140 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:39.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.654+0000 7f68c1c00700 1 --2- 172.21.15.78:0/3313851237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68bc19e050 0x7f68bc19e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:39.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.654+0000 7f68c1c00700 1 --2- 172.21.15.78:0/3313851237 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f68bc100e20 0x7f68bc199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:39.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.654+0000 7f68c1c00700 1 --2- 172.21.15.78:0/3313851237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f68bc105f40 0x7f68bc199d70 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:04:39.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.654+0000 7f68c1c00700 1 -- 172.21.15.78:0/3313851237 >> 172.21.15.78:0/3313851237 conn(0x7f68bc0fc700 msgr2=0x7f68bc0fcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:04:39.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.655+0000 7f68c1c00700 1 -- 172.21.15.78:0/3313851237 shutdown_connections 2024-03-14T10:04:39.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:04:38.655+0000 7f68c1c00700 1 -- 172.21.15.78:0/3313851237 wait complete. 2024-03-14T10:04:39.974 DEBUG:tasks.check_counter:Getting stats from a 2024-03-14T10:04:39.974 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-14T10:04:39.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:39 smithi078 ceph-mon[38143]: pgmap v325: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:39.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:39 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3313851237' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-14T10:04:40.356 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T10:04:41.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:41 smithi152 ceph-mon[48279]: pgmap v326: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:41.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:41 smithi132 ceph-mon[47959]: pgmap v326: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:41.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:41 smithi078 ceph-mon[38143]: pgmap v326: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:41.558 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:04:41.559 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-0": { 2024-03-14T10:04:41.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 218, 2024-03-14T10:04:41.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 206, 2024-03-14T10:04:41.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 1688064, 2024-03-14T10:04:41.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 117392, 2024-03-14T10:04:41.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 4, 2024-03-14T10:04:41.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 0, 2024-03-14T10:04:41.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.053605699, 2024-03-14T10:04:41.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.018128822, 2024-03-14T10:04:41.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.040418637, 2024-03-14T10:04:41.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.004321480, 2024-03-14T10:04:41.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:41.560 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 205, 2024-03-14T10:04:41.560 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.012648425, 2024-03-14T10:04:41.560 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000061699 2024-03-14T10:04:41.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:41.560 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:41.560 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:41.560 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:41.560 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:41.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:41.560 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 1688064, 2024-03-14T10:04:41.560 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 117392 2024-03-14T10:04:41.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:41.560 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-1": { 2024-03-14T10:04:41.560 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 6, 2024-03-14T10:04:41.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 159, 2024-03-14T10:04:41.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 3312, 2024-03-14T10:04:41.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 180288, 2024-03-14T10:04:41.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 5, 2024-03-14T10:04:41.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 3, 2024-03-14T10:04:41.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.018128435, 2024-03-14T10:04:41.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.011837060, 2024-03-14T10:04:41.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.003117627, 2024-03-14T10:04:41.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-14T10:04:41.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:41.561 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 157, 2024-03-14T10:04:41.561 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.024779932, 2024-03-14T10:04:41.561 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000157833 2024-03-14T10:04:41.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:41.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:41.562 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:41.562 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:41.562 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:41.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:41.562 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-14T10:04:41.562 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 180288 2024-03-14T10:04:41.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:41.562 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-2": { 2024-03-14T10:04:41.562 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 0, 2024-03-14T10:04:41.562 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 2, 2024-03-14T10:04:41.562 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 0, 2024-03-14T10:04:41.562 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 0, 2024-03-14T10:04:41.562 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 2, 2024-03-14T10:04:41.562 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 1, 2024-03-14T10:04:41.563 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.000895606, 2024-03-14T10:04:41.563 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.000000000, 2024-03-14T10:04:41.563 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.000766770, 2024-03-14T10:04:41.563 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-14T10:04:41.563 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:41.563 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:41.563 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:41.563 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:41.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:41.563 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:41.563 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:41.563 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:41.563 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:41.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:41.563 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-14T10:04:41.564 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 0 2024-03-14T10:04:41.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:41.564 INFO:teuthology.orchestra.run.smithi078.stdout: "cct": { 2024-03-14T10:04:41.564 INFO:teuthology.orchestra.run.smithi078.stdout: "total_workers": 0, 2024-03-14T10:04:41.564 INFO:teuthology.orchestra.run.smithi078.stdout: "unhealthy_workers": 0 2024-03-14T10:04:41.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:41.564 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool": { 2024-03-14T10:04:41.564 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_bytes": 0, 2024-03-14T10:04:41.564 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_items": 0, 2024-03-14T10:04:41.564 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_bytes": 0, 2024-03-14T10:04:41.564 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_items": 0, 2024-03-14T10:04:41.564 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_bytes": 0, 2024-03-14T10:04:41.564 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_items": 0, 2024-03-14T10:04:41.564 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-14T10:04:41.564 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_items": 0, 2024-03-14T10:04:41.565 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-14T10:04:41.565 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_items": 0, 2024-03-14T10:04:41.565 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_bytes": 0, 2024-03-14T10:04:41.565 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_items": 0, 2024-03-14T10:04:41.565 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-14T10:04:41.565 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_items": 0, 2024-03-14T10:04:41.565 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_bytes": 0, 2024-03-14T10:04:41.565 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_items": 0, 2024-03-14T10:04:41.565 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_bytes": 0, 2024-03-14T10:04:41.565 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_items": 0, 2024-03-14T10:04:41.565 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-14T10:04:41.565 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_items": 0, 2024-03-14T10:04:41.565 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-14T10:04:41.565 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_items": 0, 2024-03-14T10:04:41.565 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_bytes": 0, 2024-03-14T10:04:41.565 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_items": 0, 2024-03-14T10:04:41.566 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_bytes": 0, 2024-03-14T10:04:41.566 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_items": 0, 2024-03-14T10:04:41.566 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-14T10:04:41.566 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_items": 0, 2024-03-14T10:04:41.566 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_bytes": 0, 2024-03-14T10:04:41.566 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_items": 0, 2024-03-14T10:04:41.566 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_bytes": 0, 2024-03-14T10:04:41.566 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_items": 0, 2024-03-14T10:04:41.566 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_bytes": 0, 2024-03-14T10:04:41.566 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_items": 0, 2024-03-14T10:04:41.566 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_bytes": 0, 2024-03-14T10:04:41.566 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_items": 0, 2024-03-14T10:04:41.566 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_bytes": 148994, 2024-03-14T10:04:41.566 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_items": 31, 2024-03-14T10:04:41.567 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_bytes": 0, 2024-03-14T10:04:41.567 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_items": 0, 2024-03-14T10:04:41.567 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bytes": 0, 2024-03-14T10:04:41.567 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_items": 0, 2024-03-14T10:04:41.567 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_bytes": 0, 2024-03-14T10:04:41.567 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_items": 0, 2024-03-14T10:04:41.567 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_bytes": 0, 2024-03-14T10:04:41.567 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_items": 0, 2024-03-14T10:04:41.567 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_bytes": 0, 2024-03-14T10:04:41.567 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_items": 0, 2024-03-14T10:04:41.567 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_bytes": 0, 2024-03-14T10:04:41.567 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_items": 0, 2024-03-14T10:04:41.567 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_bytes": 0, 2024-03-14T10:04:41.568 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_items": 0, 2024-03-14T10:04:41.568 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_bytes": 928, 2024-03-14T10:04:41.568 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_items": 2, 2024-03-14T10:04:41.568 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_bytes": 0, 2024-03-14T10:04:41.568 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_items": 0, 2024-03-14T10:04:41.568 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_bytes": 0, 2024-03-14T10:04:41.568 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_items": 0 2024-03-14T10:04:41.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:41.568 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-14T10:04:41.568 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2024-03-14T10:04:41.568 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2024-03-14T10:04:41.568 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2024-03-14T10:04:41.568 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 220, 2024-03-14T10:04:41.568 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 1657380, 2024-03-14T10:04:41.569 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:41.569 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 220, 2024-03-14T10:04:41.569 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2024-03-14T10:04:41.569 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2024-03-14T10:04:41.569 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 220, 2024-03-14T10:04:41.569 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 1657380, 2024-03-14T10:04:41.569 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2024-03-14T10:04:41.569 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:41.569 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:41.569 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:41.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:41.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:41.569 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:04:41.958 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-14T10:04:41.959 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-14T10:04:41.959 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-03-14T10:04:41.959 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-03-14T10:04:41.959 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-03-14T10:04:41.959 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-03-14T10:04:41.959 DEBUG:tasks.check_counter:Getting stats from d 2024-03-14T10:04:41.959 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-14T10:04:42.321 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T10:04:43.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:43 smithi152 ceph-mon[48279]: pgmap v327: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:43.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:43 smithi132 ceph-mon[47959]: pgmap v327: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:43.532 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:04:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-0": { 2024-03-14T10:04:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 4, 2024-03-14T10:04:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 4, 2024-03-14T10:04:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 2992, 2024-03-14T10:04:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 448, 2024-03-14T10:04:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 4, 2024-03-14T10:04:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 0, 2024-03-14T10:04:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.004444211, 2024-03-14T10:04:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.000096953, 2024-03-14T10:04:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.001321690, 2024-03-14T10:04:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-14T10:04:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 3, 2024-03-14T10:04:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.001253812, 2024-03-14T10:04:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000417937 2024-03-14T10:04:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-14T10:04:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 448 2024-03-14T10:04:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-1": { 2024-03-14T10:04:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 222, 2024-03-14T10:04:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 210, 2024-03-14T10:04:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 1689152, 2024-03-14T10:04:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 119760, 2024-03-14T10:04:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 2, 2024-03-14T10:04:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 1, 2024-03-14T10:04:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.052825840, 2024-03-14T10:04:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.017563300, 2024-03-14T10:04:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.043473687, 2024-03-14T10:04:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.004602860, 2024-03-14T10:04:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 209, 2024-03-14T10:04:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.013638279, 2024-03-14T10:04:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000065254 2024-03-14T10:04:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 1689152, 2024-03-14T10:04:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 119760 2024-03-14T10:04:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-2": { 2024-03-14T10:04:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 2, 2024-03-14T10:04:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 160, 2024-03-14T10:04:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 320, 2024-03-14T10:04:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 180368, 2024-03-14T10:04:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 5, 2024-03-14T10:04:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 3, 2024-03-14T10:04:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.017251038, 2024-03-14T10:04:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.011690311, 2024-03-14T10:04:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.002581647, 2024-03-14T10:04:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-14T10:04:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 157, 2024-03-14T10:04:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.014537754, 2024-03-14T10:04:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000092597 2024-03-14T10:04:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-14T10:04:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 180368 2024-03-14T10:04:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "cct": { 2024-03-14T10:04:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "total_workers": 0, 2024-03-14T10:04:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "unhealthy_workers": 0 2024-03-14T10:04:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool": { 2024-03-14T10:04:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_bytes": 0, 2024-03-14T10:04:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_items": 0, 2024-03-14T10:04:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_bytes": 0, 2024-03-14T10:04:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_items": 0, 2024-03-14T10:04:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_bytes": 0, 2024-03-14T10:04:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_items": 0, 2024-03-14T10:04:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-14T10:04:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_items": 0, 2024-03-14T10:04:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-14T10:04:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_items": 0, 2024-03-14T10:04:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_bytes": 0, 2024-03-14T10:04:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_items": 0, 2024-03-14T10:04:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-14T10:04:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_items": 0, 2024-03-14T10:04:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_bytes": 0, 2024-03-14T10:04:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_items": 0, 2024-03-14T10:04:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_bytes": 0, 2024-03-14T10:04:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_items": 0, 2024-03-14T10:04:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-14T10:04:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_items": 0, 2024-03-14T10:04:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-14T10:04:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_items": 0, 2024-03-14T10:04:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_bytes": 0, 2024-03-14T10:04:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_items": 0, 2024-03-14T10:04:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_bytes": 0, 2024-03-14T10:04:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_items": 0, 2024-03-14T10:04:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-14T10:04:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_items": 0, 2024-03-14T10:04:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_bytes": 0, 2024-03-14T10:04:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_items": 0, 2024-03-14T10:04:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_bytes": 0, 2024-03-14T10:04:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_items": 0, 2024-03-14T10:04:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_bytes": 0, 2024-03-14T10:04:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_items": 0, 2024-03-14T10:04:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_bytes": 0, 2024-03-14T10:04:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_items": 0, 2024-03-14T10:04:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_bytes": 150048, 2024-03-14T10:04:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_items": 32, 2024-03-14T10:04:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_bytes": 0, 2024-03-14T10:04:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_items": 0, 2024-03-14T10:04:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bytes": 0, 2024-03-14T10:04:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_items": 0, 2024-03-14T10:04:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_bytes": 0, 2024-03-14T10:04:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_items": 0, 2024-03-14T10:04:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_bytes": 0, 2024-03-14T10:04:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_items": 0, 2024-03-14T10:04:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_bytes": 0, 2024-03-14T10:04:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_items": 0, 2024-03-14T10:04:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_bytes": 0, 2024-03-14T10:04:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_items": 0, 2024-03-14T10:04:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_bytes": 0, 2024-03-14T10:04:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_items": 0, 2024-03-14T10:04:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_bytes": 928, 2024-03-14T10:04:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_items": 2, 2024-03-14T10:04:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_bytes": 0, 2024-03-14T10:04:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_items": 0, 2024-03-14T10:04:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_bytes": 0, 2024-03-14T10:04:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_items": 0 2024-03-14T10:04:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-14T10:04:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2024-03-14T10:04:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2024-03-14T10:04:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2024-03-14T10:04:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 224, 2024-03-14T10:04:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 1657900, 2024-03-14T10:04:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 224, 2024-03-14T10:04:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2024-03-14T10:04:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2024-03-14T10:04:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 224, 2024-03-14T10:04:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 1657900, 2024-03-14T10:04:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2024-03-14T10:04:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:43.543 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:04:43.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:43 smithi078 ceph-mon[38143]: pgmap v327: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:43.899 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-14T10:04:43.900 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-14T10:04:43.900 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-14T10:04:43.900 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-14T10:04:43.900 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-14T10:04:43.900 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-14T10:04:43.900 DEBUG:tasks.check_counter:Getting stats from g 2024-03-14T10:04:43.900 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-14T10:04:44.261 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T10:04:45.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:45 smithi152 ceph-mon[48279]: pgmap v328: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:45.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:45 smithi132 ceph-mon[47959]: pgmap v328: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:45.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:45 smithi078 ceph-mon[38143]: pgmap v328: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:45.556 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:04:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-0": { 2024-03-14T10:04:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 935, 2024-03-14T10:04:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 701, 2024-03-14T10:04:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 264173, 2024-03-14T10:04:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 141036, 2024-03-14T10:04:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 8, 2024-03-14T10:04:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 3, 2024-03-14T10:04:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.096266557, 2024-03-14T10:04:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.037713428, 2024-03-14T10:04:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.055915498, 2024-03-14T10:04:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.004571609, 2024-03-14T10:04:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 699, 2024-03-14T10:04:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.063433218, 2024-03-14T10:04:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000090748 2024-03-14T10:04:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 1539, 2024-03-14T10:04:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000401095, 2024-03-14T10:04:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000260 2024-03-14T10:04:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-14T10:04:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 0 2024-03-14T10:04:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-1": { 2024-03-14T10:04:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 286, 2024-03-14T10:04:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 394, 2024-03-14T10:04:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 1734536, 2024-03-14T10:04:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 474597, 2024-03-14T10:04:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 11, 2024-03-14T10:04:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 4, 2024-03-14T10:04:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.066313325, 2024-03-14T10:04:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.022619320, 2024-03-14T10:04:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.050010453, 2024-03-14T10:04:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.005942233, 2024-03-14T10:04:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 394, 2024-03-14T10:04:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.039539939, 2024-03-14T10:04:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000100355 2024-03-14T10:04:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 4, 2024-03-14T10:04:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000548, 2024-03-14T10:04:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000137 2024-03-14T10:04:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 1731168, 2024-03-14T10:04:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 432960 2024-03-14T10:04:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-2": { 2024-03-14T10:04:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 53, 2024-03-14T10:04:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 83, 2024-03-14T10:04:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 9095, 2024-03-14T10:04:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 490195, 2024-03-14T10:04:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 48, 2024-03-14T10:04:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 7, 2024-03-14T10:04:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.032624847, 2024-03-14T10:04:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.005047048, 2024-03-14T10:04:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.022188416, 2024-03-14T10:04:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000331743, 2024-03-14T10:04:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 80, 2024-03-14T10:04:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.014765439, 2024-03-14T10:04:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000184567 2024-03-14T10:04:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 48, 2024-03-14T10:04:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000006403, 2024-03-14T10:04:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000133 2024-03-14T10:04:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-03-14T10:04:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 120832 2024-03-14T10:04:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "cct": { 2024-03-14T10:04:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "total_workers": 1, 2024-03-14T10:04:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "unhealthy_workers": 0 2024-03-14T10:04:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "finisher-MDSRank": { 2024-03-14T10:04:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "queue_len": 0, 2024-03-14T10:04:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "complete_latency": { 2024-03-14T10:04:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 34, 2024-03-14T10:04:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.002271249, 2024-03-14T10:04:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000066801 2024-03-14T10:04:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "finisher-PurgeQueue": { 2024-03-14T10:04:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "queue_len": 0, 2024-03-14T10:04:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "complete_latency": { 2024-03-14T10:04:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 2, 2024-03-14T10:04:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000023914, 2024-03-14T10:04:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000011957 2024-03-14T10:04:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "mds": { 2024-03-14T10:04:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "request": 0, 2024-03-14T10:04:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "reply": 0, 2024-03-14T10:04:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "reply_latency": { 2024-03-14T10:04:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "slow_reply": 0, 2024-03-14T10:04:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "forward": 0, 2024-03-14T10:04:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "dir_fetch_complete": 0, 2024-03-14T10:04:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "dir_fetch_keys": 0, 2024-03-14T10:04:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "dir_commit": 11, 2024-03-14T10:04:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "dir_split": 0, 2024-03-14T10:04:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "dir_merge": 0, 2024-03-14T10:04:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "inodes": 15, 2024-03-14T10:04:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "inodes_top": 0, 2024-03-14T10:04:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "inodes_bottom": 15, 2024-03-14T10:04:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "inodes_pin_tail": 0, 2024-03-14T10:04:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "inodes_pinned": 13, 2024-03-14T10:04:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "inodes_expired": 6, 2024-03-14T10:04:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "inodes_with_caps": 0, 2024-03-14T10:04:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "caps": 0, 2024-03-14T10:04:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "subtrees": 3, 2024-03-14T10:04:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "traverse": 2, 2024-03-14T10:04:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "traverse_hit": 1, 2024-03-14T10:04:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "traverse_forward": 0, 2024-03-14T10:04:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "traverse_discover": 1, 2024-03-14T10:04:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "traverse_dir_fetch": 0, 2024-03-14T10:04:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "traverse_remote_ino": 0, 2024-03-14T10:04:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "traverse_lock": 0, 2024-03-14T10:04:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "load_cent": 0, 2024-03-14T10:04:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "q": 0, 2024-03-14T10:04:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "exported": 0, 2024-03-14T10:04:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "exported_inodes": 0, 2024-03-14T10:04:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "imported": 0, 2024-03-14T10:04:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "imported_inodes": 0, 2024-03-14T10:04:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "openino_dir_fetch": 0, 2024-03-14T10:04:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "openino_backtrace_fetch": 0, 2024-03-14T10:04:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "openino_peer_discover": 0, 2024-03-14T10:04:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "root_rfiles": 4, 2024-03-14T10:04:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "root_rbytes": 234, 2024-03-14T10:04:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "root_rsnaps": 6, 2024-03-14T10:04:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_backtrace_fetch": 0, 2024-03-14T10:04:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_set_tag": 0, 2024-03-14T10:04:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_backtrace_repaired": 0, 2024-03-14T10:04:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_inotable_repaired": 0, 2024-03-14T10:04:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_dir_inodes": 0, 2024-03-14T10:04:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_dir_base_inodes": 0, 2024-03-14T10:04:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_dirfrag_rstats": 0, 2024-03-14T10:04:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_file_inodes": 0, 2024-03-14T10:04:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "handle_inode_file_caps": 0, 2024-03-14T10:04:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_cap_op_revoke": 0, 2024-03-14T10:04:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_cap_op_grant": 0, 2024-03-14T10:04:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_cap_op_trunc": 0, 2024-03-14T10:04:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-14T10:04:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-14T10:04:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "handle_client_caps": 0, 2024-03-14T10:04:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "handle_client_caps_dirty": 0, 2024-03-14T10:04:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "handle_client_cap_release": 0, 2024-03-14T10:04:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "process_request_cap_release": 0 2024-03-14T10:04:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache": { 2024-03-14T10:04:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "dir_update": 0, 2024-03-14T10:04:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "dir_update_receipt": 3, 2024-03-14T10:04:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "dir_try_discover": 2, 2024-03-14T10:04:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "dir_send_discover": 3, 2024-03-14T10:04:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "dir_handle_discover": 0, 2024-03-14T10:04:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "num_strays": 0, 2024-03-14T10:04:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "num_strays_delayed": 0, 2024-03-14T10:04:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "num_strays_enqueuing": 0, 2024-03-14T10:04:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "strays_created": 0, 2024-03-14T10:04:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "strays_enqueued": 0, 2024-03-14T10:04:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "strays_reintegrated": 0, 2024-03-14T10:04:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "strays_migrated": 0, 2024-03-14T10:04:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "num_recovering_processing": 0, 2024-03-14T10:04:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "num_recovering_enqueued": 0, 2024-03-14T10:04:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "num_recovering_prioritized": 0, 2024-03-14T10:04:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "recovery_started": 0, 2024-03-14T10:04:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "recovery_completed": 0, 2024-03-14T10:04:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "ireq_enqueue_scrub": 0, 2024-03-14T10:04:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "ireq_exportdir": 0, 2024-03-14T10:04:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "ireq_flush": 0, 2024-03-14T10:04:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "ireq_fragmentdir": 0, 2024-03-14T10:04:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "ireq_fragstats": 0, 2024-03-14T10:04:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "ireq_inodestats": 0 2024-03-14T10:04:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log": { 2024-03-14T10:04:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "evadd": 3, 2024-03-14T10:04:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "evex": 0, 2024-03-14T10:04:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "evtrm": 0, 2024-03-14T10:04:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "ev": 3, 2024-03-14T10:04:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "evexg": 0, 2024-03-14T10:04:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "evexd": 0, 2024-03-14T10:04:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "segadd": 1, 2024-03-14T10:04:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "segex": 0, 2024-03-14T10:04:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "segtrm": 0, 2024-03-14T10:04:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "seg": 1, 2024-03-14T10:04:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "segexg": 0, 2024-03-14T10:04:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "segexd": 0, 2024-03-14T10:04:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "expos": 4194304, 2024-03-14T10:04:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "wrpos": 4195377, 2024-03-14T10:04:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "rdpos": 0, 2024-03-14T10:04:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "jlat": { 2024-03-14T10:04:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 3, 2024-03-14T10:04:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.005363208, 2024-03-14T10:04:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.001787736 2024-03-14T10:04:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "replayed": 0 2024-03-14T10:04:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_mem": { 2024-03-14T10:04:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "ino": 19, 2024-03-14T10:04:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "ino+": 22, 2024-03-14T10:04:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "ino-": 3, 2024-03-14T10:04:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "dir": 18, 2024-03-14T10:04:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "dir+": 22, 2024-03-14T10:04:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "dir-": 4, 2024-03-14T10:04:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "dn": 15, 2024-03-14T10:04:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "dn+": 21, 2024-03-14T10:04:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "dn-": 6, 2024-03-14T10:04:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "cap": 0, 2024-03-14T10:04:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "cap+": 0, 2024-03-14T10:04:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "cap-": 0, 2024-03-14T10:04:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "rss": 49276, 2024-03-14T10:04:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "heap": 198964 2024-03-14T10:04:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_server": { 2024-03-14T10:04:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "dispatch_client_request": 0, 2024-03-14T10:04:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "dispatch_server_request": 0, 2024-03-14T10:04:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "handle_client_request": 0, 2024-03-14T10:04:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "handle_client_session": 4, 2024-03-14T10:04:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "handle_peer_request": 0, 2024-03-14T10:04:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "req_create_latency": { 2024-03-14T10:04:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "req_getattr_latency": { 2024-03-14T10:04:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "req_getfilelock_latency": { 2024-03-14T10:04:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "req_link_latency": { 2024-03-14T10:04:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "req_lookup_latency": { 2024-03-14T10:04:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "req_lookuphash_latency": { 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "req_lookupino_latency": { 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "req_lookupname_latency": { 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "req_lookupparent_latency": { 2024-03-14T10:04:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "req_lookupsnap_latency": { 2024-03-14T10:04:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "req_lssnap_latency": { 2024-03-14T10:04:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "req_mkdir_latency": { 2024-03-14T10:04:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "req_mknod_latency": { 2024-03-14T10:04:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "req_mksnap_latency": { 2024-03-14T10:04:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "req_open_latency": { 2024-03-14T10:04:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "req_readdir_latency": { 2024-03-14T10:04:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "req_rename_latency": { 2024-03-14T10:04:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "req_renamesnap_latency": { 2024-03-14T10:04:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "req_snapdiff_latency": { 2024-03-14T10:04:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "req_rmdir_latency": { 2024-03-14T10:04:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "req_rmsnap_latency": { 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "req_rmxattr_latency": { 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "req_setattr_latency": { 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "req_setdirlayout_latency": { 2024-03-14T10:04:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "req_setfilelock_latency": { 2024-03-14T10:04:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "req_setlayout_latency": { 2024-03-14T10:04:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "req_setxattr_latency": { 2024-03-14T10:04:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "req_symlink_latency": { 2024-03-14T10:04:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "req_unlink_latency": { 2024-03-14T10:04:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "cap_revoke_eviction": 0, 2024-03-14T10:04:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "cap_acquisition_throttle": 0, 2024-03-14T10:04:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "req_getvxattr_latency": { 2024-03-14T10:04:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_sessions": { 2024-03-14T10:04:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "session_count": 0, 2024-03-14T10:04:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "session_add": 1, 2024-03-14T10:04:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "session_remove": 1, 2024-03-14T10:04:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "sessions_open": 0, 2024-03-14T10:04:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "sessions_stale": 0, 2024-03-14T10:04:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "total_load": 0, 2024-03-14T10:04:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "average_load": 0, 2024-03-14T10:04:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "avg_session_uptime": 89, 2024-03-14T10:04:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "mdthresh_evicted": 0 2024-03-14T10:04:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool": { 2024-03-14T10:04:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_bytes": 0, 2024-03-14T10:04:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_items": 0, 2024-03-14T10:04:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_bytes": 0, 2024-03-14T10:04:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_items": 0, 2024-03-14T10:04:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_bytes": 0, 2024-03-14T10:04:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_items": 0, 2024-03-14T10:04:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-14T10:04:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_items": 0, 2024-03-14T10:04:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-14T10:04:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_items": 0, 2024-03-14T10:04:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_bytes": 0, 2024-03-14T10:04:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_items": 0, 2024-03-14T10:04:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-14T10:04:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_items": 0, 2024-03-14T10:04:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_bytes": 0, 2024-03-14T10:04:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_items": 0, 2024-03-14T10:04:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_bytes": 0, 2024-03-14T10:04:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_items": 0, 2024-03-14T10:04:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-14T10:04:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_items": 0, 2024-03-14T10:04:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-14T10:04:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_items": 0, 2024-03-14T10:04:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_bytes": 0, 2024-03-14T10:04:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_items": 0, 2024-03-14T10:04:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_bytes": 0, 2024-03-14T10:04:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_items": 0, 2024-03-14T10:04:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-14T10:04:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_items": 0, 2024-03-14T10:04:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_bytes": 0, 2024-03-14T10:04:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_items": 0, 2024-03-14T10:04:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_bytes": 0, 2024-03-14T10:04:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_items": 0, 2024-03-14T10:04:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_bytes": 0, 2024-03-14T10:04:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_items": 0, 2024-03-14T10:04:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_bytes": 0, 2024-03-14T10:04:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_items": 0, 2024-03-14T10:04:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_bytes": 246343, 2024-03-14T10:04:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_items": 81, 2024-03-14T10:04:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_bytes": 0, 2024-03-14T10:04:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_items": 0, 2024-03-14T10:04:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bytes": 0, 2024-03-14T10:04:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_items": 0, 2024-03-14T10:04:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_bytes": 0, 2024-03-14T10:04:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_items": 0, 2024-03-14T10:04:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_bytes": 0, 2024-03-14T10:04:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_items": 0, 2024-03-14T10:04:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_bytes": 9832, 2024-03-14T10:04:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_items": 205, 2024-03-14T10:04:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_bytes": 0, 2024-03-14T10:04:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_items": 0, 2024-03-14T10:04:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_bytes": 0, 2024-03-14T10:04:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_items": 0, 2024-03-14T10:04:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_bytes": 84965, 2024-03-14T10:04:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_items": 320, 2024-03-14T10:04:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_bytes": 0, 2024-03-14T10:04:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_items": 0, 2024-03-14T10:04:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_bytes": 0, 2024-03-14T10:04:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_items": 0 2024-03-14T10:04:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter": { 2024-03-14T10:04:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "op_active": 0, 2024-03-14T10:04:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "op_laggy": 0, 2024-03-14T10:04:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "op_send": 39, 2024-03-14T10:04:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "op_send_bytes": 11375, 2024-03-14T10:04:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "op_resend": 27, 2024-03-14T10:04:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "op_reply": 39, 2024-03-14T10:04:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "op_latency": { 2024-03-14T10:04:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 39, 2024-03-14T10:04:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.251995733, 2024-03-14T10:04:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.006461429 2024-03-14T10:04:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "op_inflight": 0, 2024-03-14T10:04:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "oplen_avg": { 2024-03-14T10:04:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 39, 2024-03-14T10:04:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 61 2024-03-14T10:04:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "op": 39, 2024-03-14T10:04:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "op_r": 0, 2024-03-14T10:04:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "op_w": 39, 2024-03-14T10:04:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "op_rmw": 0, 2024-03-14T10:04:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "op_pg": 0, 2024-03-14T10:04:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_stat": 0, 2024-03-14T10:04:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_create": 10, 2024-03-14T10:04:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_read": 0, 2024-03-14T10:04:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_write": 3, 2024-03-14T10:04:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_writefull": 7, 2024-03-14T10:04:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_writesame": 0, 2024-03-14T10:04:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_append": 0, 2024-03-14T10:04:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_zero": 0, 2024-03-14T10:04:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_truncate": 0, 2024-03-14T10:04:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_delete": 6, 2024-03-14T10:04:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_mapext": 0, 2024-03-14T10:04:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_sparse_read": 0, 2024-03-14T10:04:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_clonerange": 0, 2024-03-14T10:04:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_getxattr": 0, 2024-03-14T10:04:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_setxattr": 20, 2024-03-14T10:04:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_cmpxattr": 0, 2024-03-14T10:04:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_rmxattr": 0, 2024-03-14T10:04:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_resetxattrs": 0, 2024-03-14T10:04:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_call": 0, 2024-03-14T10:04:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_watch": 0, 2024-03-14T10:04:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_notify": 0, 2024-03-14T10:04:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_src_cmpxattr": 0, 2024-03-14T10:04:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_pgls": 0, 2024-03-14T10:04:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_pgls_filter": 0, 2024-03-14T10:04:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "osdop_other": 0, 2024-03-14T10:04:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "linger_active": 0, 2024-03-14T10:04:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "linger_send": 0, 2024-03-14T10:04:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "linger_resend": 0, 2024-03-14T10:04:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "linger_ping": 0, 2024-03-14T10:04:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "poolop_active": 0, 2024-03-14T10:04:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "poolop_send": 0, 2024-03-14T10:04:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "poolop_resend": 0, 2024-03-14T10:04:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "poolstat_active": 0, 2024-03-14T10:04:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "poolstat_send": 0, 2024-03-14T10:04:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "poolstat_resend": 0, 2024-03-14T10:04:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "statfs_active": 0, 2024-03-14T10:04:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "statfs_send": 0, 2024-03-14T10:04:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "statfs_resend": 0, 2024-03-14T10:04:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "command_active": 0, 2024-03-14T10:04:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "command_send": 0, 2024-03-14T10:04:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "command_resend": 0, 2024-03-14T10:04:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "map_epoch": 96, 2024-03-14T10:04:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "map_full": 0, 2024-03-14T10:04:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "map_inc": 13, 2024-03-14T10:04:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_sessions": 9, 2024-03-14T10:04:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_session_open": 9, 2024-03-14T10:04:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_session_close": 0, 2024-03-14T10:04:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_laggy": 0, 2024-03-14T10:04:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "omap_wr": 14, 2024-03-14T10:04:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "omap_rd": 0, 2024-03-14T10:04:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "omap_del": 1 2024-03-14T10:04:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "oft": { 2024-03-14T10:04:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "omap_total_objs": 1, 2024-03-14T10:04:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "omap_total_kv_pairs": 0, 2024-03-14T10:04:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "omap_total_updates": 0, 2024-03-14T10:04:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "omap_total_removes": 0 2024-03-14T10:04:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "osd-slow-ops": { 2024-03-14T10:04:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "slow_ops_count": 0 2024-03-14T10:04:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "purge_queue": { 2024-03-14T10:04:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "pq_executing_ops": 0, 2024-03-14T10:04:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "pq_executing_ops_high_water": 0, 2024-03-14T10:04:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "pq_executing": 0, 2024-03-14T10:04:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "pq_executing_high_water": 0, 2024-03-14T10:04:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "pq_executed": 0, 2024-03-14T10:04:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "pq_item_in_journal": 0 2024-03-14T10:04:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-14T10:04:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2024-03-14T10:04:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2024-03-14T10:04:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2024-03-14T10:04:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 1270, 2024-03-14T10:04:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 1876831, 2024-03-14T10:04:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 1270, 2024-03-14T10:04:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2024-03-14T10:04:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2024-03-14T10:04:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 1270, 2024-03-14T10:04:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 1876831, 2024-03-14T10:04:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2024-03-14T10:04:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-objecter_bytes": { 2024-03-14T10:04:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2024-03-14T10:04:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2024-03-14T10:04:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2024-03-14T10:04:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 0, 2024-03-14T10:04:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 0, 2024-03-14T10:04:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 39, 2024-03-14T10:04:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 11375, 2024-03-14T10:04:45.596 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 39, 2024-03-14T10:04:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 11375, 2024-03-14T10:04:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2024-03-14T10:04:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-objecter_ops": { 2024-03-14T10:04:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2024-03-14T10:04:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 1024, 2024-03-14T10:04:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2024-03-14T10:04:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 0, 2024-03-14T10:04:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 0, 2024-03-14T10:04:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 39, 2024-03-14T10:04:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 39, 2024-03-14T10:04:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 39, 2024-03-14T10:04:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 39, 2024-03-14T10:04:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2024-03-14T10:04:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-write_buf_throttle": { 2024-03-14T10:04:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2024-03-14T10:04:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 3758096384, 2024-03-14T10:04:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2024-03-14T10:04:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 0, 2024-03-14T10:04:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 0, 2024-03-14T10:04:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2024-03-14T10:04:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2024-03-14T10:04:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 0, 2024-03-14T10:04:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 0, 2024-03-14T10:04:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2024-03-14T10:04:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-write_buf_throttle-0x557815fec0a0": { 2024-03-14T10:04:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2024-03-14T10:04:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 3758096384, 2024-03-14T10:04:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2024-03-14T10:04:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 3, 2024-03-14T10:04:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 1073, 2024-03-14T10:04:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 3, 2024-03-14T10:04:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2024-03-14T10:04:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2024-03-14T10:04:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 3, 2024-03-14T10:04:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 1073, 2024-03-14T10:04:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2024-03-14T10:04:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:45.601 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:04:45.934 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-14T10:04:45.934 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-14T10:04:45.935 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=6 2024-03-14T10:04:45.935 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-03-14T10:04:45.935 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-03-14T10:04:45.935 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-03-14T10:04:45.935 DEBUG:tasks.check_counter:Getting stats from j 2024-03-14T10:04:45.935 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-14T10:04:46.299 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.a/config 2024-03-14T10:04:47.471 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-03-14T10:04:47.471 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-0": { 2024-03-14T10:04:47.471 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 223, 2024-03-14T10:04:47.471 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 211, 2024-03-14T10:04:47.471 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 1689424, 2024-03-14T10:04:47.471 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 120352, 2024-03-14T10:04:47.471 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 4, 2024-03-14T10:04:47.472 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 0, 2024-03-14T10:04:47.472 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.055922998, 2024-03-14T10:04:47.472 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.017251391, 2024-03-14T10:04:47.472 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.046299603, 2024-03-14T10:04:47.472 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.004639460, 2024-03-14T10:04:47.472 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:47.472 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 210, 2024-03-14T10:04:47.472 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.013503532, 2024-03-14T10:04:47.472 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000064302 2024-03-14T10:04:47.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:47.472 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:47.472 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 1689424, 2024-03-14T10:04:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 120352 2024-03-14T10:04:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-1": { 2024-03-14T10:04:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 6, 2024-03-14T10:04:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 164, 2024-03-14T10:04:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 3312, 2024-03-14T10:04:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 181168, 2024-03-14T10:04:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 5, 2024-03-14T10:04:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 3, 2024-03-14T10:04:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.019301848, 2024-03-14T10:04:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.012489954, 2024-03-14T10:04:47.474 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.003566718, 2024-03-14T10:04:47.474 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-14T10:04:47.474 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:47.474 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 162, 2024-03-14T10:04:47.474 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.022847125, 2024-03-14T10:04:47.474 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000141031 2024-03-14T10:04:47.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:47.474 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:47.474 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:47.474 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:47.474 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:47.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:47.474 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-14T10:04:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 181168 2024-03-14T10:04:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-2": { 2024-03-14T10:04:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 0, 2024-03-14T10:04:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 2, 2024-03-14T10:04:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 0, 2024-03-14T10:04:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 0, 2024-03-14T10:04:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 2, 2024-03-14T10:04:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 1, 2024-03-14T10:04:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.000804483, 2024-03-14T10:04:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.000000000, 2024-03-14T10:04:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.000703449, 2024-03-14T10:04:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-14T10:04:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:47.476 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:47.476 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:47.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:47.476 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:47.476 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:47.476 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:47.476 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:47.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:47.476 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-14T10:04:47.476 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 0 2024-03-14T10:04:47.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:47.476 INFO:teuthology.orchestra.run.smithi078.stdout: "cct": { 2024-03-14T10:04:47.476 INFO:teuthology.orchestra.run.smithi078.stdout: "total_workers": 0, 2024-03-14T10:04:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: "unhealthy_workers": 0 2024-03-14T10:04:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool": { 2024-03-14T10:04:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_bytes": 0, 2024-03-14T10:04:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_items": 0, 2024-03-14T10:04:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_bytes": 0, 2024-03-14T10:04:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_items": 0, 2024-03-14T10:04:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_bytes": 0, 2024-03-14T10:04:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_items": 0, 2024-03-14T10:04:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-14T10:04:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_items": 0, 2024-03-14T10:04:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-14T10:04:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_items": 0, 2024-03-14T10:04:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_bytes": 0, 2024-03-14T10:04:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_items": 0, 2024-03-14T10:04:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-14T10:04:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_items": 0, 2024-03-14T10:04:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_bytes": 0, 2024-03-14T10:04:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_items": 0, 2024-03-14T10:04:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_bytes": 0, 2024-03-14T10:04:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_items": 0, 2024-03-14T10:04:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-14T10:04:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_items": 0, 2024-03-14T10:04:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-14T10:04:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_items": 0, 2024-03-14T10:04:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_bytes": 0, 2024-03-14T10:04:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_items": 0, 2024-03-14T10:04:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_bytes": 0, 2024-03-14T10:04:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_items": 0, 2024-03-14T10:04:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-14T10:04:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_items": 0, 2024-03-14T10:04:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_bytes": 0, 2024-03-14T10:04:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_items": 0, 2024-03-14T10:04:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_bytes": 0, 2024-03-14T10:04:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_items": 0, 2024-03-14T10:04:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_bytes": 0, 2024-03-14T10:04:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_items": 0, 2024-03-14T10:04:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_bytes": 0, 2024-03-14T10:04:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_items": 0, 2024-03-14T10:04:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_bytes": 149017, 2024-03-14T10:04:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_items": 31, 2024-03-14T10:04:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_bytes": 0, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_items": 0, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bytes": 0, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_items": 0, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_bytes": 0, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_items": 0, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_bytes": 0, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_items": 0, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_bytes": 0, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_items": 0, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_bytes": 0, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_items": 0, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_bytes": 0, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_items": 0, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_bytes": 928, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_items": 2, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_bytes": 0, 2024-03-14T10:04:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_items": 0, 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_bytes": 0, 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_items": 0 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 225, 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 1658030, 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 225, 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 225, 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 1658030, 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2024-03-14T10:04:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2024-03-14T10:04:47.482 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2024-03-14T10:04:47.482 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2024-03-14T10:04:47.482 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:47.482 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-03-14T10:04:47.482 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-03-14T10:04:47.490 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:47 smithi078 ceph-mon[38143]: pgmap v329: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:47.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:47 smithi152 ceph-mon[48279]: pgmap v329: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:47.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:47 smithi132 ceph-mon[47959]: pgmap v329: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:47.868 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-14T10:04:47.868 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-14T10:04:47.868 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-14T10:04:47.868 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-14T10:04:47.868 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-14T10:04:47.868 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-14T10:04:47.868 DEBUG:tasks.check_counter:Getting stats from b 2024-03-14T10:04:47.868 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-14T10:04:48.114 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.b/config 2024-03-14T10:04:49.115 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-14T10:04:49.115 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-0": { 2024-03-14T10:04:49.115 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 91594, 2024-03-14T10:04:49.115 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 146702, 2024-03-14T10:04:49.115 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 26926045, 2024-03-14T10:04:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 142688844, 2024-03-14T10:04:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 156, 2024-03-14T10:04:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 10, 2024-03-14T10:04:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 5.452547785, 2024-03-14T10:04:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 2.195527911, 2024-03-14T10:04:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 2.636003549, 2024-03-14T10:04:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 1.011295958, 2024-03-14T10:04:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 146696, 2024-03-14T10:04:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 6.742460889, 2024-03-14T10:04:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000045962 2024-03-14T10:04:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:49.116 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 82164, 2024-03-14T10:04:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.025639869, 2024-03-14T10:04:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000312 2024-03-14T10:04:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 96384, 2024-03-14T10:04:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 32016 2024-03-14T10:04:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-1": { 2024-03-14T10:04:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 32974, 2024-03-14T10:04:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 33334, 2024-03-14T10:04:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 11037198, 2024-03-14T10:04:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 59303675, 2024-03-14T10:04:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 12, 2024-03-14T10:04:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 6, 2024-03-14T10:04:49.117 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 1.999647953, 2024-03-14T10:04:49.118 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.619255644, 2024-03-14T10:04:49.118 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.522841310, 2024-03-14T10:04:49.118 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.726052300, 2024-03-14T10:04:49.118 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:49.118 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 33335, 2024-03-14T10:04:49.118 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 2.292723161, 2024-03-14T10:04:49.118 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000068778 2024-03-14T10:04:49.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.118 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:49.118 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 1558, 2024-03-14T10:04:49.118 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000313501, 2024-03-14T10:04:49.118 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000201 2024-03-14T10:04:49.118 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.119 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 1747840, 2024-03-14T10:04:49.119 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 465440 2024-03-14T10:04:49.119 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.119 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-2": { 2024-03-14T10:04:49.119 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 42923, 2024-03-14T10:04:49.119 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 42527, 2024-03-14T10:04:49.119 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 10988834, 2024-03-14T10:04:49.119 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 44499973, 2024-03-14T10:04:49.119 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 30, 2024-03-14T10:04:49.119 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 5, 2024-03-14T10:04:49.119 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 2.564728769, 2024-03-14T10:04:49.119 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.746647816, 2024-03-14T10:04:49.120 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.687529162, 2024-03-14T10:04:49.120 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.969015691, 2024-03-14T10:04:49.120 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:49.120 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 42527, 2024-03-14T10:04:49.120 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 2.605432211, 2024-03-14T10:04:49.120 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000061265 2024-03-14T10:04:49.120 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.120 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:49.120 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 2493, 2024-03-14T10:04:49.120 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000512354, 2024-03-14T10:04:49.120 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000205 2024-03-14T10:04:49.120 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.120 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 353872, 2024-03-14T10:04:49.121 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 213856 2024-03-14T10:04:49.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.121 INFO:teuthology.orchestra.run.smithi132.stdout: "cct": { 2024-03-14T10:04:49.121 INFO:teuthology.orchestra.run.smithi132.stdout: "total_workers": 1, 2024-03-14T10:04:49.121 INFO:teuthology.orchestra.run.smithi132.stdout: "unhealthy_workers": 0 2024-03-14T10:04:49.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.121 INFO:teuthology.orchestra.run.smithi132.stdout: "finisher-MDSRank": { 2024-03-14T10:04:49.121 INFO:teuthology.orchestra.run.smithi132.stdout: "queue_len": 0, 2024-03-14T10:04:49.121 INFO:teuthology.orchestra.run.smithi132.stdout: "complete_latency": { 2024-03-14T10:04:49.121 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 114746, 2024-03-14T10:04:49.121 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 38.776519732, 2024-03-14T10:04:49.121 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000337933 2024-03-14T10:04:49.121 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:49.121 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.121 INFO:teuthology.orchestra.run.smithi132.stdout: "finisher-PurgeQueue": { 2024-03-14T10:04:49.122 INFO:teuthology.orchestra.run.smithi132.stdout: "queue_len": 0, 2024-03-14T10:04:49.122 INFO:teuthology.orchestra.run.smithi132.stdout: "complete_latency": { 2024-03-14T10:04:49.122 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 11050, 2024-03-14T10:04:49.122 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 2.026837984, 2024-03-14T10:04:49.122 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000183424 2024-03-14T10:04:49.122 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:49.122 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.122 INFO:teuthology.orchestra.run.smithi132.stdout: "mds": { 2024-03-14T10:04:49.122 INFO:teuthology.orchestra.run.smithi132.stdout: "request": 24165, 2024-03-14T10:04:49.122 INFO:teuthology.orchestra.run.smithi132.stdout: "reply": 24165, 2024-03-14T10:04:49.122 INFO:teuthology.orchestra.run.smithi132.stdout: "reply_latency": { 2024-03-14T10:04:49.122 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 24165, 2024-03-14T10:04:49.122 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 570.986924550, 2024-03-14T10:04:49.122 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.023628674 2024-03-14T10:04:49.123 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.123 INFO:teuthology.orchestra.run.smithi132.stdout: "slow_reply": 0, 2024-03-14T10:04:49.123 INFO:teuthology.orchestra.run.smithi132.stdout: "forward": 0, 2024-03-14T10:04:49.123 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_fetch_complete": 0, 2024-03-14T10:04:49.123 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_fetch_keys": 0, 2024-03-14T10:04:49.123 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_commit": 257, 2024-03-14T10:04:49.123 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_split": 29, 2024-03-14T10:04:49.123 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_merge": 7, 2024-03-14T10:04:49.123 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes": 10726, 2024-03-14T10:04:49.123 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_top": 1535, 2024-03-14T10:04:49.123 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_bottom": 9191, 2024-03-14T10:04:49.123 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_pin_tail": 0, 2024-03-14T10:04:49.123 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_pinned": 8533, 2024-03-14T10:04:49.123 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_expired": 0, 2024-03-14T10:04:49.123 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_with_caps": 2, 2024-03-14T10:04:49.124 INFO:teuthology.orchestra.run.smithi132.stdout: "caps": 3, 2024-03-14T10:04:49.124 INFO:teuthology.orchestra.run.smithi132.stdout: "subtrees": 4, 2024-03-14T10:04:49.124 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse": 25537, 2024-03-14T10:04:49.124 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_hit": 24987, 2024-03-14T10:04:49.124 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_forward": 0, 2024-03-14T10:04:49.124 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_discover": 0, 2024-03-14T10:04:49.124 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_dir_fetch": 0, 2024-03-14T10:04:49.124 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_remote_ino": 0, 2024-03-14T10:04:49.124 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_lock": 12, 2024-03-14T10:04:49.124 INFO:teuthology.orchestra.run.smithi132.stdout: "load_cent": 0, 2024-03-14T10:04:49.124 INFO:teuthology.orchestra.run.smithi132.stdout: "q": 0, 2024-03-14T10:04:49.124 INFO:teuthology.orchestra.run.smithi132.stdout: "exported": 0, 2024-03-14T10:04:49.124 INFO:teuthology.orchestra.run.smithi132.stdout: "exported_inodes": 0, 2024-03-14T10:04:49.124 INFO:teuthology.orchestra.run.smithi132.stdout: "imported": 0, 2024-03-14T10:04:49.124 INFO:teuthology.orchestra.run.smithi132.stdout: "imported_inodes": 0, 2024-03-14T10:04:49.125 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_dir_fetch": 0, 2024-03-14T10:04:49.125 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_backtrace_fetch": 36404, 2024-03-14T10:04:49.125 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_peer_discover": 0, 2024-03-14T10:04:49.125 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rfiles": 4, 2024-03-14T10:04:49.125 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rbytes": 234, 2024-03-14T10:04:49.125 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rsnaps": 6, 2024-03-14T10:04:49.125 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_backtrace_fetch": 36404, 2024-03-14T10:04:49.125 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_set_tag": 0, 2024-03-14T10:04:49.125 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_backtrace_repaired": 0, 2024-03-14T10:04:49.125 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_inotable_repaired": 0, 2024-03-14T10:04:49.125 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dir_inodes": 867, 2024-03-14T10:04:49.125 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dir_base_inodes": 20, 2024-03-14T10:04:49.125 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dirfrag_rstats": 847, 2024-03-14T10:04:49.125 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_file_inodes": 35537, 2024-03-14T10:04:49.126 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_inode_file_caps": 1, 2024-03-14T10:04:49.126 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_revoke": 9542, 2024-03-14T10:04:49.126 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_grant": 24307, 2024-03-14T10:04:49.126 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_trunc": 8294, 2024-03-14T10:04:49.126 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-14T10:04:49.126 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-14T10:04:49.126 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_caps": 23180, 2024-03-14T10:04:49.126 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_caps_dirty": 14586, 2024-03-14T10:04:49.126 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_cap_release": 1148, 2024-03-14T10:04:49.126 INFO:teuthology.orchestra.run.smithi132.stdout: "process_request_cap_release": 16680 2024-03-14T10:04:49.126 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.126 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_cache": { 2024-03-14T10:04:49.126 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_update": 2, 2024-03-14T10:04:49.127 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_update_receipt": 0, 2024-03-14T10:04:49.127 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_try_discover": 0, 2024-03-14T10:04:49.127 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_send_discover": 0, 2024-03-14T10:04:49.127 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_handle_discover": 8, 2024-03-14T10:04:49.127 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays": 71, 2024-03-14T10:04:49.127 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays_delayed": 0, 2024-03-14T10:04:49.127 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays_enqueuing": 0, 2024-03-14T10:04:49.127 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_created": 6852, 2024-03-14T10:04:49.127 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_enqueued": 6779, 2024-03-14T10:04:49.127 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_reintegrated": 2, 2024-03-14T10:04:49.127 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_migrated": 0, 2024-03-14T10:04:49.127 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_processing": 0, 2024-03-14T10:04:49.127 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_enqueued": 0, 2024-03-14T10:04:49.127 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_prioritized": 0, 2024-03-14T10:04:49.127 INFO:teuthology.orchestra.run.smithi132.stdout: "recovery_started": 0, 2024-03-14T10:04:49.128 INFO:teuthology.orchestra.run.smithi132.stdout: "recovery_completed": 0, 2024-03-14T10:04:49.128 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_enqueue_scrub": 20, 2024-03-14T10:04:49.128 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_exportdir": 0, 2024-03-14T10:04:49.128 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_flush": 0, 2024-03-14T10:04:49.128 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_fragmentdir": 36, 2024-03-14T10:04:49.128 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_fragstats": 0, 2024-03-14T10:04:49.128 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_inodestats": 0 2024-03-14T10:04:49.128 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.128 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_log": { 2024-03-14T10:04:49.128 INFO:teuthology.orchestra.run.smithi132.stdout: "evadd": 63282, 2024-03-14T10:04:49.128 INFO:teuthology.orchestra.run.smithi132.stdout: "evex": 0, 2024-03-14T10:04:49.128 INFO:teuthology.orchestra.run.smithi132.stdout: "evtrm": 0, 2024-03-14T10:04:49.128 INFO:teuthology.orchestra.run.smithi132.stdout: "ev": 63282, 2024-03-14T10:04:49.128 INFO:teuthology.orchestra.run.smithi132.stdout: "evexg": 0, 2024-03-14T10:04:49.128 INFO:teuthology.orchestra.run.smithi132.stdout: "evexd": 0, 2024-03-14T10:04:49.129 INFO:teuthology.orchestra.run.smithi132.stdout: "segadd": 82, 2024-03-14T10:04:49.129 INFO:teuthology.orchestra.run.smithi132.stdout: "segex": 0, 2024-03-14T10:04:49.129 INFO:teuthology.orchestra.run.smithi132.stdout: "segtrm": 0, 2024-03-14T10:04:49.129 INFO:teuthology.orchestra.run.smithi132.stdout: "seg": 82, 2024-03-14T10:04:49.129 INFO:teuthology.orchestra.run.smithi132.stdout: "segexg": 0, 2024-03-14T10:04:49.129 INFO:teuthology.orchestra.run.smithi132.stdout: "segexd": 0, 2024-03-14T10:04:49.129 INFO:teuthology.orchestra.run.smithi132.stdout: "expos": 4194304, 2024-03-14T10:04:49.129 INFO:teuthology.orchestra.run.smithi132.stdout: "wrpos": 185843980, 2024-03-14T10:04:49.129 INFO:teuthology.orchestra.run.smithi132.stdout: "rdpos": 0, 2024-03-14T10:04:49.129 INFO:teuthology.orchestra.run.smithi132.stdout: "jlat": { 2024-03-14T10:04:49.129 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 26156, 2024-03-14T10:04:49.129 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 54.272087028, 2024-03-14T10:04:49.129 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.002074938 2024-03-14T10:04:49.129 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.129 INFO:teuthology.orchestra.run.smithi132.stdout: "replayed": 0 2024-03-14T10:04:49.130 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.130 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_mem": { 2024-03-14T10:04:49.130 INFO:teuthology.orchestra.run.smithi132.stdout: "ino": 3707, 2024-03-14T10:04:49.130 INFO:teuthology.orchestra.run.smithi132.stdout: "ino+": 10506, 2024-03-14T10:04:49.130 INFO:teuthology.orchestra.run.smithi132.stdout: "ino-": 6799, 2024-03-14T10:04:49.130 INFO:teuthology.orchestra.run.smithi132.stdout: "dir": 243, 2024-03-14T10:04:49.130 INFO:teuthology.orchestra.run.smithi132.stdout: "dir+": 338, 2024-03-14T10:04:49.130 INFO:teuthology.orchestra.run.smithi132.stdout: "dir-": 95, 2024-03-14T10:04:49.130 INFO:teuthology.orchestra.run.smithi132.stdout: "dn": 10726, 2024-03-14T10:04:49.130 INFO:teuthology.orchestra.run.smithi132.stdout: "dn+": 15818, 2024-03-14T10:04:49.130 INFO:teuthology.orchestra.run.smithi132.stdout: "dn-": 5092, 2024-03-14T10:04:49.130 INFO:teuthology.orchestra.run.smithi132.stdout: "cap": 3, 2024-03-14T10:04:49.130 INFO:teuthology.orchestra.run.smithi132.stdout: "cap+": 6866, 2024-03-14T10:04:49.130 INFO:teuthology.orchestra.run.smithi132.stdout: "cap-": 6863, 2024-03-14T10:04:49.131 INFO:teuthology.orchestra.run.smithi132.stdout: "rss": 108420, 2024-03-14T10:04:49.131 INFO:teuthology.orchestra.run.smithi132.stdout: "heap": 207156 2024-03-14T10:04:49.131 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.131 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_server": { 2024-03-14T10:04:49.131 INFO:teuthology.orchestra.run.smithi132.stdout: "dispatch_client_request": 43764, 2024-03-14T10:04:49.131 INFO:teuthology.orchestra.run.smithi132.stdout: "dispatch_server_request": 1, 2024-03-14T10:04:49.131 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_request": 24165, 2024-03-14T10:04:49.131 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_session": 39, 2024-03-14T10:04:49.131 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_peer_request": 2, 2024-03-14T10:04:49.131 INFO:teuthology.orchestra.run.smithi132.stdout: "req_create_latency": { 2024-03-14T10:04:49.131 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 6766, 2024-03-14T10:04:49.131 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 508.303647995, 2024-03-14T10:04:49.131 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.075126167 2024-03-14T10:04:49.131 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.132 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getattr_latency": { 2024-03-14T10:04:49.132 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 122, 2024-03-14T10:04:49.132 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.032899983, 2024-03-14T10:04:49.132 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000269671 2024-03-14T10:04:49.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.132 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getfilelock_latency": { 2024-03-14T10:04:49.132 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.132 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.132 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.132 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.132 INFO:teuthology.orchestra.run.smithi132.stdout: "req_link_latency": { 2024-03-14T10:04:49.132 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 2, 2024-03-14T10:04:49.133 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.002229225, 2024-03-14T10:04:49.133 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.001114612 2024-03-14T10:04:49.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.133 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookup_latency": { 2024-03-14T10:04:49.133 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 167, 2024-03-14T10:04:49.133 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.089620078, 2024-03-14T10:04:49.133 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000536647 2024-03-14T10:04:49.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.133 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookuphash_latency": { 2024-03-14T10:04:49.133 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.133 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.133 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.133 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.133 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupino_latency": { 2024-03-14T10:04:49.133 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 1, 2024-03-14T10:04:49.134 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000179978, 2024-03-14T10:04:49.134 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000179978 2024-03-14T10:04:49.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.134 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupname_latency": { 2024-03-14T10:04:49.134 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.134 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.134 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.134 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupparent_latency": { 2024-03-14T10:04:49.134 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.134 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.134 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.134 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.134 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupsnap_latency": { 2024-03-14T10:04:49.134 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.135 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.135 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.135 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lssnap_latency": { 2024-03-14T10:04:49.135 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 10, 2024-03-14T10:04:49.135 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.002084848, 2024-03-14T10:04:49.135 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000208484 2024-03-14T10:04:49.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.135 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mkdir_latency": { 2024-03-14T10:04:49.135 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 87, 2024-03-14T10:04:49.135 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 4.737271633, 2024-03-14T10:04:49.135 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.054451398 2024-03-14T10:04:49.135 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.135 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mknod_latency": { 2024-03-14T10:04:49.135 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.136 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.136 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.136 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mksnap_latency": { 2024-03-14T10:04:49.136 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 10, 2024-03-14T10:04:49.136 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.088743006, 2024-03-14T10:04:49.136 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.008874300 2024-03-14T10:04:49.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.136 INFO:teuthology.orchestra.run.smithi132.stdout: "req_open_latency": { 2024-03-14T10:04:49.136 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 445, 2024-03-14T10:04:49.136 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.596864023, 2024-03-14T10:04:49.136 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.001341267 2024-03-14T10:04:49.136 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.136 INFO:teuthology.orchestra.run.smithi132.stdout: "req_readdir_latency": { 2024-03-14T10:04:49.137 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 317, 2024-03-14T10:04:49.137 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 1.470838468, 2024-03-14T10:04:49.137 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.004639868 2024-03-14T10:04:49.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.137 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rename_latency": { 2024-03-14T10:04:49.137 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 81, 2024-03-14T10:04:49.137 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.209012780, 2024-03-14T10:04:49.137 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.002580404 2024-03-14T10:04:49.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.137 INFO:teuthology.orchestra.run.smithi132.stdout: "req_renamesnap_latency": { 2024-03-14T10:04:49.137 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.137 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.137 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.137 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.137 INFO:teuthology.orchestra.run.smithi132.stdout: "req_snapdiff_latency": { 2024-03-14T10:04:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmdir_latency": { 2024-03-14T10:04:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 93, 2024-03-14T10:04:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 3.333513442, 2024-03-14T10:04:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.035844230 2024-03-14T10:04:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmsnap_latency": { 2024-03-14T10:04:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 4, 2024-03-14T10:04:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.042017176, 2024-03-14T10:04:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.010504294 2024-03-14T10:04:49.138 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmxattr_latency": { 2024-03-14T10:04:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setattr_latency": { 2024-03-14T10:04:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 9314, 2024-03-14T10:04:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 37.800048689, 2024-03-14T10:04:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.004058411 2024-03-14T10:04:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setdirlayout_latency": { 2024-03-14T10:04:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.139 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setfilelock_latency": { 2024-03-14T10:04:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 8, 2024-03-14T10:04:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.003049300, 2024-03-14T10:04:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000381162 2024-03-14T10:04:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setlayout_latency": { 2024-03-14T10:04:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setxattr_latency": { 2024-03-14T10:04:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 9, 2024-03-14T10:04:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.022423158, 2024-03-14T10:04:49.140 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.002491462 2024-03-14T10:04:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "req_symlink_latency": { 2024-03-14T10:04:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 7, 2024-03-14T10:04:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.006149124, 2024-03-14T10:04:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000878446 2024-03-14T10:04:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "req_unlink_latency": { 2024-03-14T10:04:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 6722, 2024-03-14T10:04:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 14.246331644, 2024-03-14T10:04:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.002119359 2024-03-14T10:04:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "cap_revoke_eviction": 0, 2024-03-14T10:04:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "cap_acquisition_throttle": 0, 2024-03-14T10:04:49.141 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getvxattr_latency": { 2024-03-14T10:04:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_sessions": { 2024-03-14T10:04:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "session_count": 2, 2024-03-14T10:04:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "session_add": 7, 2024-03-14T10:04:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "session_remove": 5, 2024-03-14T10:04:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "sessions_open": 2, 2024-03-14T10:04:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "sessions_stale": 0, 2024-03-14T10:04:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "total_load": 740, 2024-03-14T10:04:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "average_load": 370, 2024-03-14T10:04:49.142 INFO:teuthology.orchestra.run.smithi132.stdout: "avg_session_uptime": 624, 2024-03-14T10:04:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "mdthresh_evicted": 0 2024-03-14T10:04:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "mempool": { 2024-03-14T10:04:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "bloom_filter_bytes": 0, 2024-03-14T10:04:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "bloom_filter_items": 0, 2024-03-14T10:04:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_alloc_bytes": 0, 2024-03-14T10:04:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_alloc_items": 0, 2024-03-14T10:04:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_data_bytes": 0, 2024-03-14T10:04:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_data_items": 0, 2024-03-14T10:04:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-14T10:04:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_onode_items": 0, 2024-03-14T10:04:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-14T10:04:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_meta_items": 0, 2024-03-14T10:04:49.143 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_other_bytes": 0, 2024-03-14T10:04:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_other_items": 0, 2024-03-14T10:04:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-14T10:04:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_buffer_items": 0, 2024-03-14T10:04:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_extent_bytes": 0, 2024-03-14T10:04:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_extent_items": 0, 2024-03-14T10:04:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_blob_bytes": 0, 2024-03-14T10:04:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_blob_items": 0, 2024-03-14T10:04:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-14T10:04:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_shared_blob_items": 0, 2024-03-14T10:04:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-14T10:04:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_inline_bl_items": 0, 2024-03-14T10:04:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_fsck_bytes": 0, 2024-03-14T10:04:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_fsck_items": 0, 2024-03-14T10:04:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_txc_bytes": 0, 2024-03-14T10:04:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_txc_items": 0, 2024-03-14T10:04:49.144 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-14T10:04:49.145 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_deferred_items": 0, 2024-03-14T10:04:49.145 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_bytes": 0, 2024-03-14T10:04:49.145 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_items": 0, 2024-03-14T10:04:49.145 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_bytes": 0, 2024-03-14T10:04:49.145 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_items": 0, 2024-03-14T10:04:49.145 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_reader_bytes": 0, 2024-03-14T10:04:49.145 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_reader_items": 0, 2024-03-14T10:04:49.145 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_writer_bytes": 0, 2024-03-14T10:04:49.145 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_writer_items": 0, 2024-03-14T10:04:49.145 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_anon_bytes": 992414, 2024-03-14T10:04:49.145 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_anon_items": 3916, 2024-03-14T10:04:49.145 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_meta_bytes": 1496, 2024-03-14T10:04:49.145 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_meta_items": 17, 2024-03-14T10:04:49.145 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_bytes": 0, 2024-03-14T10:04:49.146 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_items": 0, 2024-03-14T10:04:49.146 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_mapbl_bytes": 0, 2024-03-14T10:04:49.146 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_mapbl_items": 0, 2024-03-14T10:04:49.146 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_pglog_bytes": 0, 2024-03-14T10:04:49.146 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_pglog_items": 0, 2024-03-14T10:04:49.146 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_bytes": 9832, 2024-03-14T10:04:49.146 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_items": 205, 2024-03-14T10:04:49.146 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_mapping_bytes": 0, 2024-03-14T10:04:49.146 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_mapping_items": 0, 2024-03-14T10:04:49.146 INFO:teuthology.orchestra.run.smithi132.stdout: "pgmap_bytes": 0, 2024-03-14T10:04:49.146 INFO:teuthology.orchestra.run.smithi132.stdout: "pgmap_items": 0, 2024-03-14T10:04:49.146 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_co_bytes": 16733941, 2024-03-14T10:04:49.146 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_co_items": 190055, 2024-03-14T10:04:49.147 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_1_bytes": 0, 2024-03-14T10:04:49.147 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_1_items": 0, 2024-03-14T10:04:49.147 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_2_bytes": 0, 2024-03-14T10:04:49.147 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_2_items": 0 2024-03-14T10:04:49.147 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.147 INFO:teuthology.orchestra.run.smithi132.stdout: "objecter": { 2024-03-14T10:04:49.147 INFO:teuthology.orchestra.run.smithi132.stdout: "op_active": 0, 2024-03-14T10:04:49.147 INFO:teuthology.orchestra.run.smithi132.stdout: "op_laggy": 0, 2024-03-14T10:04:49.147 INFO:teuthology.orchestra.run.smithi132.stdout: "op_send": 116012, 2024-03-14T10:04:49.147 INFO:teuthology.orchestra.run.smithi132.stdout: "op_send_bytes": 188012136, 2024-03-14T10:04:49.147 INFO:teuthology.orchestra.run.smithi132.stdout: "op_resend": 37, 2024-03-14T10:04:49.147 INFO:teuthology.orchestra.run.smithi132.stdout: "op_reply": 116012, 2024-03-14T10:04:49.147 INFO:teuthology.orchestra.run.smithi132.stdout: "op_latency": { 2024-03-14T10:04:49.147 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 116012, 2024-03-14T10:04:49.147 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 484.095149705, 2024-03-14T10:04:49.147 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.004172802 2024-03-14T10:04:49.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.148 INFO:teuthology.orchestra.run.smithi132.stdout: "op_inflight": 0, 2024-03-14T10:04:49.148 INFO:teuthology.orchestra.run.smithi132.stdout: "oplen_avg": { 2024-03-14T10:04:49.148 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 116012, 2024-03-14T10:04:49.148 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 116337 2024-03-14T10:04:49.148 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.148 INFO:teuthology.orchestra.run.smithi132.stdout: "op": 116012, 2024-03-14T10:04:49.148 INFO:teuthology.orchestra.run.smithi132.stdout: "op_r": 36513, 2024-03-14T10:04:49.148 INFO:teuthology.orchestra.run.smithi132.stdout: "op_w": 79499, 2024-03-14T10:04:49.148 INFO:teuthology.orchestra.run.smithi132.stdout: "op_rmw": 0, 2024-03-14T10:04:49.148 INFO:teuthology.orchestra.run.smithi132.stdout: "op_pg": 0, 2024-03-14T10:04:49.148 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_stat": 0, 2024-03-14T10:04:49.148 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_create": 11, 2024-03-14T10:04:49.148 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_read": 89, 2024-03-14T10:04:49.149 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_write": 26225, 2024-03-14T10:04:49.149 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_writefull": 71, 2024-03-14T10:04:49.149 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_writesame": 0, 2024-03-14T10:04:49.149 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_append": 0, 2024-03-14T10:04:49.149 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_zero": 0, 2024-03-14T10:04:49.149 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_truncate": 13, 2024-03-14T10:04:49.149 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_delete": 14197, 2024-03-14T10:04:49.149 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_mapext": 0, 2024-03-14T10:04:49.149 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_sparse_read": 0, 2024-03-14T10:04:49.149 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_clonerange": 0, 2024-03-14T10:04:49.149 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_getxattr": 36424, 2024-03-14T10:04:49.149 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_setxattr": 22, 2024-03-14T10:04:49.149 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_cmpxattr": 0, 2024-03-14T10:04:49.149 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_rmxattr": 0, 2024-03-14T10:04:49.150 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_resetxattrs": 0, 2024-03-14T10:04:49.150 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_call": 0, 2024-03-14T10:04:49.150 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_watch": 0, 2024-03-14T10:04:49.150 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_notify": 0, 2024-03-14T10:04:49.150 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_src_cmpxattr": 0, 2024-03-14T10:04:49.150 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_pgls": 0, 2024-03-14T10:04:49.150 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_pgls_filter": 0, 2024-03-14T10:04:49.150 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_other": 38665, 2024-03-14T10:04:49.150 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_active": 0, 2024-03-14T10:04:49.150 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_send": 0, 2024-03-14T10:04:49.150 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_resend": 0, 2024-03-14T10:04:49.150 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_ping": 0, 2024-03-14T10:04:49.150 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_active": 0, 2024-03-14T10:04:49.150 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_send": 0, 2024-03-14T10:04:49.151 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_resend": 0, 2024-03-14T10:04:49.151 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_active": 0, 2024-03-14T10:04:49.151 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_send": 0, 2024-03-14T10:04:49.151 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_resend": 0, 2024-03-14T10:04:49.151 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_active": 0, 2024-03-14T10:04:49.151 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_send": 0, 2024-03-14T10:04:49.151 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_resend": 0, 2024-03-14T10:04:49.151 INFO:teuthology.orchestra.run.smithi132.stdout: "command_active": 0, 2024-03-14T10:04:49.151 INFO:teuthology.orchestra.run.smithi132.stdout: "command_send": 0, 2024-03-14T10:04:49.151 INFO:teuthology.orchestra.run.smithi132.stdout: "command_resend": 0, 2024-03-14T10:04:49.151 INFO:teuthology.orchestra.run.smithi132.stdout: "map_epoch": 96, 2024-03-14T10:04:49.151 INFO:teuthology.orchestra.run.smithi132.stdout: "map_full": 0, 2024-03-14T10:04:49.151 INFO:teuthology.orchestra.run.smithi132.stdout: "map_inc": 18, 2024-03-14T10:04:49.151 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_sessions": 12, 2024-03-14T10:04:49.151 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_session_open": 12, 2024-03-14T10:04:49.151 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_session_close": 0, 2024-03-14T10:04:49.152 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_laggy": 0, 2024-03-14T10:04:49.152 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_wr": 578, 2024-03-14T10:04:49.152 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_rd": 0, 2024-03-14T10:04:49.152 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_del": 42 2024-03-14T10:04:49.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.152 INFO:teuthology.orchestra.run.smithi132.stdout: "oft": { 2024-03-14T10:04:49.152 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_objs": 1, 2024-03-14T10:04:49.152 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_kv_pairs": 0, 2024-03-14T10:04:49.152 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_updates": 0, 2024-03-14T10:04:49.152 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_removes": 0 2024-03-14T10:04:49.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.152 INFO:teuthology.orchestra.run.smithi132.stdout: "osd-slow-ops": { 2024-03-14T10:04:49.152 INFO:teuthology.orchestra.run.smithi132.stdout: "slow_ops_count": 24263 2024-03-14T10:04:49.152 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.153 INFO:teuthology.orchestra.run.smithi132.stdout: "purge_queue": { 2024-03-14T10:04:49.153 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_ops": 0, 2024-03-14T10:04:49.153 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_ops_high_water": 28, 2024-03-14T10:04:49.153 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing": 0, 2024-03-14T10:04:49.153 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_high_water": 10, 2024-03-14T10:04:49.153 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executed": 6779, 2024-03-14T10:04:49.153 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_item_in_journal": 0 2024-03-14T10:04:49.153 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.153 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-14T10:04:49.153 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-03-14T10:04:49.153 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 104857600, 2024-03-14T10:04:49.153 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-03-14T10:04:49.153 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 167487, 2024-03-14T10:04:49.153 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 34588757, 2024-03-14T10:04:49.154 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:49.154 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 167487, 2024-03-14T10:04:49.154 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-03-14T10:04:49.154 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-03-14T10:04:49.154 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 167487, 2024-03-14T10:04:49.154 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 34588757, 2024-03-14T10:04:49.154 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-03-14T10:04:49.154 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.154 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.154 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.154 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:49.154 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.154 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-objecter_bytes": { 2024-03-14T10:04:49.154 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-03-14T10:04:49.155 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 104857600, 2024-03-14T10:04:49.155 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-03-14T10:04:49.155 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-03-14T10:04:49.155 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-03-14T10:04:49.155 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:49.155 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:49.155 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 116012, 2024-03-14T10:04:49.155 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 188892303, 2024-03-14T10:04:49.155 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 116012, 2024-03-14T10:04:49.155 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 188892303, 2024-03-14T10:04:49.155 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-03-14T10:04:49.155 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.155 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.155 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.155 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:49.155 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.156 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-objecter_ops": { 2024-03-14T10:04:49.156 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-03-14T10:04:49.156 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 1024, 2024-03-14T10:04:49.156 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-03-14T10:04:49.156 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-03-14T10:04:49.156 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-03-14T10:04:49.156 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:49.156 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:49.156 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 116012, 2024-03-14T10:04:49.156 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 116012, 2024-03-14T10:04:49.156 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 116012, 2024-03-14T10:04:49.156 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 116012, 2024-03-14T10:04:49.156 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-03-14T10:04:49.156 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.156 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.156 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.157 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:49.157 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.157 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-write_buf_throttle": { 2024-03-14T10:04:49.157 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-03-14T10:04:49.157 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 3758096384, 2024-03-14T10:04:49.157 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-03-14T10:04:49.157 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 6779, 2024-03-14T10:04:49.157 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 880167, 2024-03-14T10:04:49.157 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:49.157 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 6779, 2024-03-14T10:04:49.157 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-03-14T10:04:49.157 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-03-14T10:04:49.157 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 69, 2024-03-14T10:04:49.157 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 880167, 2024-03-14T10:04:49.157 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-03-14T10:04:49.157 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.158 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.158 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.158 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:49.158 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:49.158 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-write_buf_throttle-0x55bd1098c0a0": { 2024-03-14T10:04:49.158 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-03-14T10:04:49.158 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 3758096384, 2024-03-14T10:04:49.158 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-03-14T10:04:49.158 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 63282, 2024-03-14T10:04:49.158 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 181649676, 2024-03-14T10:04:49.158 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:49.158 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 63282, 2024-03-14T10:04:49.158 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-03-14T10:04:49.158 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-03-14T10:04:49.158 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 26156, 2024-03-14T10:04:49.159 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 181649676, 2024-03-14T10:04:49.159 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-03-14T10:04:49.159 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:49.159 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:49.159 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:49.159 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:49.159 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:49.159 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-14T10:04:49.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:48 smithi152 ceph-mon[48279]: pgmap v330: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:49.266 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:48 smithi132 ceph-mon[47959]: pgmap v330: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:49.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:48 smithi078 ceph-mon[38143]: pgmap v330: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:49.462 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-03-14T10:04:49.462 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-14T10:04:49.462 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-14T10:04:49.462 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-03-14T10:04:49.462 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-03-14T10:04:49.463 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=29 2024-03-14T10:04:49.463 DEBUG:tasks.check_counter:Getting stats from e 2024-03-14T10:04:49.463 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-14T10:04:49.684 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.b/config 2024-03-14T10:04:50.589 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-14T10:04:50.589 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-0": { 2024-03-14T10:04:50.589 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 2126, 2024-03-14T10:04:50.589 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 2126, 2024-03-14T10:04:50.589 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 605034, 2024-03-14T10:04:50.589 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 660397, 2024-03-14T10:04:50.589 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 6, 2024-03-14T10:04:50.589 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 2, 2024-03-14T10:04:50.590 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.149024672, 2024-03-14T10:04:50.590 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.045962647, 2024-03-14T10:04:50.590 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.036876650, 2024-03-14T10:04:50.590 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.053690573, 2024-03-14T10:04:50.590 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:50.590 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 2125, 2024-03-14T10:04:50.590 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.076594118, 2024-03-14T10:04:50.590 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000036044 2024-03-14T10:04:50.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.590 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:50.590 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.590 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.590 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.590 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.590 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-14T10:04:50.591 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 448 2024-03-14T10:04:50.591 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.591 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-1": { 2024-03-14T10:04:50.591 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 244, 2024-03-14T10:04:50.591 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 231, 2024-03-14T10:04:50.591 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 1717292, 2024-03-14T10:04:50.591 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 124140, 2024-03-14T10:04:50.591 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 3, 2024-03-14T10:04:50.591 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 2, 2024-03-14T10:04:50.591 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.044301546, 2024-03-14T10:04:50.591 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.012055549, 2024-03-14T10:04:50.591 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.037997098, 2024-03-14T10:04:50.591 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.005289489, 2024-03-14T10:04:50.591 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:50.592 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 230, 2024-03-14T10:04:50.592 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.010920142, 2024-03-14T10:04:50.592 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000047478 2024-03-14T10:04:50.592 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.592 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:50.592 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.592 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.592 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.592 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.592 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 1716736, 2024-03-14T10:04:50.592 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 123456 2024-03-14T10:04:50.592 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.592 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-2": { 2024-03-14T10:04:50.592 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 3, 2024-03-14T10:04:50.593 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 162, 2024-03-14T10:04:50.593 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 588, 2024-03-14T10:04:50.593 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 427671, 2024-03-14T10:04:50.593 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 6, 2024-03-14T10:04:50.593 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 4, 2024-03-14T10:04:50.593 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.009310613, 2024-03-14T10:04:50.593 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.005637966, 2024-03-14T10:04:50.593 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.001817983, 2024-03-14T10:04:50.593 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.000022760, 2024-03-14T10:04:50.593 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:50.593 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 159, 2024-03-14T10:04:50.593 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.024141302, 2024-03-14T10:04:50.593 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000151832 2024-03-14T10:04:50.593 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.593 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:50.594 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.594 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.594 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.594 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.594 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-14T10:04:50.594 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 427360 2024-03-14T10:04:50.594 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.594 INFO:teuthology.orchestra.run.smithi132.stdout: "cct": { 2024-03-14T10:04:50.594 INFO:teuthology.orchestra.run.smithi132.stdout: "total_workers": 1, 2024-03-14T10:04:50.594 INFO:teuthology.orchestra.run.smithi132.stdout: "unhealthy_workers": 0 2024-03-14T10:04:50.594 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.594 INFO:teuthology.orchestra.run.smithi132.stdout: "finisher-MDSRank": { 2024-03-14T10:04:50.594 INFO:teuthology.orchestra.run.smithi132.stdout: "queue_len": 0, 2024-03-14T10:04:50.594 INFO:teuthology.orchestra.run.smithi132.stdout: "complete_latency": { 2024-03-14T10:04:50.595 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 3591, 2024-03-14T10:04:50.595 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.082887170, 2024-03-14T10:04:50.595 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000023081 2024-03-14T10:04:50.595 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:50.595 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.595 INFO:teuthology.orchestra.run.smithi132.stdout: "finisher-PurgeQueue": { 2024-03-14T10:04:50.595 INFO:teuthology.orchestra.run.smithi132.stdout: "queue_len": 0, 2024-03-14T10:04:50.595 INFO:teuthology.orchestra.run.smithi132.stdout: "complete_latency": { 2024-03-14T10:04:50.595 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.595 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.595 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.595 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:50.595 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.595 INFO:teuthology.orchestra.run.smithi132.stdout: "mds": { 2024-03-14T10:04:50.595 INFO:teuthology.orchestra.run.smithi132.stdout: "request": 0, 2024-03-14T10:04:50.596 INFO:teuthology.orchestra.run.smithi132.stdout: "reply": 0, 2024-03-14T10:04:50.596 INFO:teuthology.orchestra.run.smithi132.stdout: "reply_latency": { 2024-03-14T10:04:50.596 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.596 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.596 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.596 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.596 INFO:teuthology.orchestra.run.smithi132.stdout: "slow_reply": 0, 2024-03-14T10:04:50.596 INFO:teuthology.orchestra.run.smithi132.stdout: "forward": 0, 2024-03-14T10:04:50.596 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_fetch_complete": 0, 2024-03-14T10:04:50.596 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_fetch_keys": 0, 2024-03-14T10:04:50.596 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_commit": 0, 2024-03-14T10:04:50.596 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_split": 0, 2024-03-14T10:04:50.596 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_merge": 0, 2024-03-14T10:04:50.597 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes": 0, 2024-03-14T10:04:50.597 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_top": 0, 2024-03-14T10:04:50.597 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_bottom": 0, 2024-03-14T10:04:50.597 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_pin_tail": 0, 2024-03-14T10:04:50.597 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_pinned": 0, 2024-03-14T10:04:50.597 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_expired": 0, 2024-03-14T10:04:50.597 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_with_caps": 0, 2024-03-14T10:04:50.597 INFO:teuthology.orchestra.run.smithi132.stdout: "caps": 0, 2024-03-14T10:04:50.597 INFO:teuthology.orchestra.run.smithi132.stdout: "subtrees": 1, 2024-03-14T10:04:50.597 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse": 0, 2024-03-14T10:04:50.597 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_hit": 0, 2024-03-14T10:04:50.597 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_forward": 0, 2024-03-14T10:04:50.597 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_discover": 0, 2024-03-14T10:04:50.597 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_dir_fetch": 0, 2024-03-14T10:04:50.598 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_remote_ino": 0, 2024-03-14T10:04:50.598 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_lock": 0, 2024-03-14T10:04:50.598 INFO:teuthology.orchestra.run.smithi132.stdout: "load_cent": 0, 2024-03-14T10:04:50.598 INFO:teuthology.orchestra.run.smithi132.stdout: "q": 0, 2024-03-14T10:04:50.598 INFO:teuthology.orchestra.run.smithi132.stdout: "exported": 0, 2024-03-14T10:04:50.598 INFO:teuthology.orchestra.run.smithi132.stdout: "exported_inodes": 0, 2024-03-14T10:04:50.598 INFO:teuthology.orchestra.run.smithi132.stdout: "imported": 0, 2024-03-14T10:04:50.598 INFO:teuthology.orchestra.run.smithi132.stdout: "imported_inodes": 0, 2024-03-14T10:04:50.598 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_dir_fetch": 0, 2024-03-14T10:04:50.598 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_backtrace_fetch": 0, 2024-03-14T10:04:50.598 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_peer_discover": 0, 2024-03-14T10:04:50.598 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rfiles": 0, 2024-03-14T10:04:50.598 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rbytes": 0, 2024-03-14T10:04:50.598 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rsnaps": 0, 2024-03-14T10:04:50.599 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_backtrace_fetch": 0, 2024-03-14T10:04:50.599 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_set_tag": 0, 2024-03-14T10:04:50.599 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_backtrace_repaired": 0, 2024-03-14T10:04:50.599 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_inotable_repaired": 0, 2024-03-14T10:04:50.599 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dir_inodes": 0, 2024-03-14T10:04:50.599 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dir_base_inodes": 0, 2024-03-14T10:04:50.599 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dirfrag_rstats": 0, 2024-03-14T10:04:50.599 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_file_inodes": 0, 2024-03-14T10:04:50.599 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_inode_file_caps": 0, 2024-03-14T10:04:50.599 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_revoke": 0, 2024-03-14T10:04:50.599 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_grant": 0, 2024-03-14T10:04:50.599 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_trunc": 0, 2024-03-14T10:04:50.599 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-14T10:04:50.599 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-14T10:04:50.600 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_caps": 0, 2024-03-14T10:04:50.600 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_caps_dirty": 0, 2024-03-14T10:04:50.600 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_cap_release": 0, 2024-03-14T10:04:50.600 INFO:teuthology.orchestra.run.smithi132.stdout: "process_request_cap_release": 0 2024-03-14T10:04:50.600 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.600 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_cache": { 2024-03-14T10:04:50.600 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_update": 0, 2024-03-14T10:04:50.600 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_update_receipt": 0, 2024-03-14T10:04:50.600 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_try_discover": 0, 2024-03-14T10:04:50.600 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_send_discover": 0, 2024-03-14T10:04:50.600 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_handle_discover": 0, 2024-03-14T10:04:50.600 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays": 0, 2024-03-14T10:04:50.600 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays_delayed": 0, 2024-03-14T10:04:50.600 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays_enqueuing": 0, 2024-03-14T10:04:50.601 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_created": 0, 2024-03-14T10:04:50.601 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_enqueued": 0, 2024-03-14T10:04:50.601 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_reintegrated": 0, 2024-03-14T10:04:50.601 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_migrated": 0, 2024-03-14T10:04:50.601 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_processing": 0, 2024-03-14T10:04:50.601 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_enqueued": 0, 2024-03-14T10:04:50.601 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_prioritized": 0, 2024-03-14T10:04:50.601 INFO:teuthology.orchestra.run.smithi132.stdout: "recovery_started": 0, 2024-03-14T10:04:50.601 INFO:teuthology.orchestra.run.smithi132.stdout: "recovery_completed": 0, 2024-03-14T10:04:50.601 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_enqueue_scrub": 0, 2024-03-14T10:04:50.601 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_exportdir": 0, 2024-03-14T10:04:50.601 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_flush": 0, 2024-03-14T10:04:50.601 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_fragmentdir": 0, 2024-03-14T10:04:50.601 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_fragstats": 0, 2024-03-14T10:04:50.601 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_inodestats": 0 2024-03-14T10:04:50.602 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.602 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_log": { 2024-03-14T10:04:50.602 INFO:teuthology.orchestra.run.smithi132.stdout: "evadd": 0, 2024-03-14T10:04:50.602 INFO:teuthology.orchestra.run.smithi132.stdout: "evex": 0, 2024-03-14T10:04:50.602 INFO:teuthology.orchestra.run.smithi132.stdout: "evtrm": 0, 2024-03-14T10:04:50.602 INFO:teuthology.orchestra.run.smithi132.stdout: "ev": 3, 2024-03-14T10:04:50.602 INFO:teuthology.orchestra.run.smithi132.stdout: "evexg": 0, 2024-03-14T10:04:50.602 INFO:teuthology.orchestra.run.smithi132.stdout: "evexd": 0, 2024-03-14T10:04:50.602 INFO:teuthology.orchestra.run.smithi132.stdout: "segadd": 0, 2024-03-14T10:04:50.602 INFO:teuthology.orchestra.run.smithi132.stdout: "segex": 0, 2024-03-14T10:04:50.602 INFO:teuthology.orchestra.run.smithi132.stdout: "segtrm": 0, 2024-03-14T10:04:50.602 INFO:teuthology.orchestra.run.smithi132.stdout: "seg": 1, 2024-03-14T10:04:50.602 INFO:teuthology.orchestra.run.smithi132.stdout: "segexg": 0, 2024-03-14T10:04:50.602 INFO:teuthology.orchestra.run.smithi132.stdout: "segexd": 0, 2024-03-14T10:04:50.603 INFO:teuthology.orchestra.run.smithi132.stdout: "expos": 4194304, 2024-03-14T10:04:50.603 INFO:teuthology.orchestra.run.smithi132.stdout: "wrpos": 4195377, 2024-03-14T10:04:50.603 INFO:teuthology.orchestra.run.smithi132.stdout: "rdpos": 4195236, 2024-03-14T10:04:50.603 INFO:teuthology.orchestra.run.smithi132.stdout: "jlat": { 2024-03-14T10:04:50.603 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.603 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.603 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.603 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.603 INFO:teuthology.orchestra.run.smithi132.stdout: "replayed": 3 2024-03-14T10:04:50.603 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.603 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_mem": { 2024-03-14T10:04:50.604 INFO:teuthology.orchestra.run.smithi132.stdout: "ino": 3, 2024-03-14T10:04:50.604 INFO:teuthology.orchestra.run.smithi132.stdout: "ino+": 3, 2024-03-14T10:04:50.604 INFO:teuthology.orchestra.run.smithi132.stdout: "ino-": 0, 2024-03-14T10:04:50.604 INFO:teuthology.orchestra.run.smithi132.stdout: "dir": 1, 2024-03-14T10:04:50.604 INFO:teuthology.orchestra.run.smithi132.stdout: "dir+": 1, 2024-03-14T10:04:50.604 INFO:teuthology.orchestra.run.smithi132.stdout: "dir-": 0, 2024-03-14T10:04:50.604 INFO:teuthology.orchestra.run.smithi132.stdout: "dn": 0, 2024-03-14T10:04:50.604 INFO:teuthology.orchestra.run.smithi132.stdout: "dn+": 0, 2024-03-14T10:04:50.604 INFO:teuthology.orchestra.run.smithi132.stdout: "dn-": 0, 2024-03-14T10:04:50.604 INFO:teuthology.orchestra.run.smithi132.stdout: "cap": 0, 2024-03-14T10:04:50.604 INFO:teuthology.orchestra.run.smithi132.stdout: "cap+": 0, 2024-03-14T10:04:50.604 INFO:teuthology.orchestra.run.smithi132.stdout: "cap-": 0, 2024-03-14T10:04:50.604 INFO:teuthology.orchestra.run.smithi132.stdout: "rss": 46716, 2024-03-14T10:04:50.604 INFO:teuthology.orchestra.run.smithi132.stdout: "heap": 207156 2024-03-14T10:04:50.605 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.605 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_server": { 2024-03-14T10:04:50.605 INFO:teuthology.orchestra.run.smithi132.stdout: "dispatch_client_request": 0, 2024-03-14T10:04:50.605 INFO:teuthology.orchestra.run.smithi132.stdout: "dispatch_server_request": 0, 2024-03-14T10:04:50.605 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_request": 0, 2024-03-14T10:04:50.605 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_session": 0, 2024-03-14T10:04:50.605 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_peer_request": 0, 2024-03-14T10:04:50.605 INFO:teuthology.orchestra.run.smithi132.stdout: "req_create_latency": { 2024-03-14T10:04:50.605 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.605 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.605 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.605 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.605 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getattr_latency": { 2024-03-14T10:04:50.605 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.605 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.606 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.606 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.606 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getfilelock_latency": { 2024-03-14T10:04:50.606 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.606 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.606 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.606 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.606 INFO:teuthology.orchestra.run.smithi132.stdout: "req_link_latency": { 2024-03-14T10:04:50.606 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.606 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.606 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.606 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.606 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookup_latency": { 2024-03-14T10:04:50.606 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.606 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.607 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.607 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookuphash_latency": { 2024-03-14T10:04:50.607 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.607 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.607 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.607 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupino_latency": { 2024-03-14T10:04:50.607 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.607 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.607 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.607 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.607 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupname_latency": { 2024-03-14T10:04:50.608 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.608 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.608 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.608 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupparent_latency": { 2024-03-14T10:04:50.608 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.608 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.608 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.608 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupsnap_latency": { 2024-03-14T10:04:50.608 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.608 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.608 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.608 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.608 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lssnap_latency": { 2024-03-14T10:04:50.609 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.609 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.609 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.609 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mkdir_latency": { 2024-03-14T10:04:50.609 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.609 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.609 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.609 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.609 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mknod_latency": { 2024-03-14T10:04:50.609 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.609 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.609 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.610 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mksnap_latency": { 2024-03-14T10:04:50.610 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.610 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.610 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.610 INFO:teuthology.orchestra.run.smithi132.stdout: "req_open_latency": { 2024-03-14T10:04:50.610 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.610 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.610 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.610 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.610 INFO:teuthology.orchestra.run.smithi132.stdout: "req_readdir_latency": { 2024-03-14T10:04:50.610 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.611 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.611 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.611 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rename_latency": { 2024-03-14T10:04:50.611 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.611 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.611 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.611 INFO:teuthology.orchestra.run.smithi132.stdout: "req_renamesnap_latency": { 2024-03-14T10:04:50.611 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.611 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.611 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.611 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.611 INFO:teuthology.orchestra.run.smithi132.stdout: "req_snapdiff_latency": { 2024-03-14T10:04:50.612 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.612 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.612 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.612 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmdir_latency": { 2024-03-14T10:04:50.612 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.612 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.612 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.612 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmsnap_latency": { 2024-03-14T10:04:50.612 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.612 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.612 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.612 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.613 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmxattr_latency": { 2024-03-14T10:04:50.613 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.613 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.613 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.613 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setattr_latency": { 2024-03-14T10:04:50.613 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.613 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.613 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.613 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setdirlayout_latency": { 2024-03-14T10:04:50.613 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.613 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.613 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.613 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setfilelock_latency": { 2024-03-14T10:04:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setlayout_latency": { 2024-03-14T10:04:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setxattr_latency": { 2024-03-14T10:04:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.614 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: "req_symlink_latency": { 2024-03-14T10:04:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: "req_unlink_latency": { 2024-03-14T10:04:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.615 INFO:teuthology.orchestra.run.smithi132.stdout: "cap_revoke_eviction": 0, 2024-03-14T10:04:50.616 INFO:teuthology.orchestra.run.smithi132.stdout: "cap_acquisition_throttle": 0, 2024-03-14T10:04:50.616 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getvxattr_latency": { 2024-03-14T10:04:50.616 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.616 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.616 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.616 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:50.616 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.616 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_sessions": { 2024-03-14T10:04:50.616 INFO:teuthology.orchestra.run.smithi132.stdout: "session_count": 0, 2024-03-14T10:04:50.616 INFO:teuthology.orchestra.run.smithi132.stdout: "session_add": 1, 2024-03-14T10:04:50.616 INFO:teuthology.orchestra.run.smithi132.stdout: "session_remove": 1, 2024-03-14T10:04:50.616 INFO:teuthology.orchestra.run.smithi132.stdout: "sessions_open": 1, 2024-03-14T10:04:50.617 INFO:teuthology.orchestra.run.smithi132.stdout: "sessions_stale": 0, 2024-03-14T10:04:50.617 INFO:teuthology.orchestra.run.smithi132.stdout: "total_load": 0, 2024-03-14T10:04:50.617 INFO:teuthology.orchestra.run.smithi132.stdout: "average_load": 0, 2024-03-14T10:04:50.617 INFO:teuthology.orchestra.run.smithi132.stdout: "avg_session_uptime": 1894, 2024-03-14T10:04:50.617 INFO:teuthology.orchestra.run.smithi132.stdout: "mdthresh_evicted": 0 2024-03-14T10:04:50.617 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.617 INFO:teuthology.orchestra.run.smithi132.stdout: "mempool": { 2024-03-14T10:04:50.617 INFO:teuthology.orchestra.run.smithi132.stdout: "bloom_filter_bytes": 0, 2024-03-14T10:04:50.617 INFO:teuthology.orchestra.run.smithi132.stdout: "bloom_filter_items": 0, 2024-03-14T10:04:50.617 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_alloc_bytes": 0, 2024-03-14T10:04:50.617 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_alloc_items": 0, 2024-03-14T10:04:50.617 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_data_bytes": 0, 2024-03-14T10:04:50.617 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_data_items": 0, 2024-03-14T10:04:50.617 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-14T10:04:50.618 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_onode_items": 0, 2024-03-14T10:04:50.618 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-14T10:04:50.618 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_meta_items": 0, 2024-03-14T10:04:50.618 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_other_bytes": 0, 2024-03-14T10:04:50.618 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_other_items": 0, 2024-03-14T10:04:50.618 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-14T10:04:50.618 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_buffer_items": 0, 2024-03-14T10:04:50.618 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_extent_bytes": 0, 2024-03-14T10:04:50.619 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_extent_items": 0, 2024-03-14T10:04:50.619 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_blob_bytes": 0, 2024-03-14T10:04:50.619 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_blob_items": 0, 2024-03-14T10:04:50.619 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-14T10:04:50.619 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_shared_blob_items": 0, 2024-03-14T10:04:50.619 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-14T10:04:50.619 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_inline_bl_items": 0, 2024-03-14T10:04:50.619 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_fsck_bytes": 0, 2024-03-14T10:04:50.619 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_fsck_items": 0, 2024-03-14T10:04:50.619 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_txc_bytes": 0, 2024-03-14T10:04:50.619 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_txc_items": 0, 2024-03-14T10:04:50.619 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-14T10:04:50.619 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_deferred_items": 0, 2024-03-14T10:04:50.619 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_bytes": 0, 2024-03-14T10:04:50.620 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_items": 0, 2024-03-14T10:04:50.620 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_bytes": 0, 2024-03-14T10:04:50.620 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_items": 0, 2024-03-14T10:04:50.620 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_reader_bytes": 0, 2024-03-14T10:04:50.620 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_reader_items": 0, 2024-03-14T10:04:50.620 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_writer_bytes": 0, 2024-03-14T10:04:50.620 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_writer_items": 0, 2024-03-14T10:04:50.620 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_anon_bytes": 181330, 2024-03-14T10:04:50.620 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_anon_items": 48, 2024-03-14T10:04:50.620 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_meta_bytes": 0, 2024-03-14T10:04:50.620 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_meta_items": 0, 2024-03-14T10:04:50.620 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_bytes": 0, 2024-03-14T10:04:50.620 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_items": 0, 2024-03-14T10:04:50.620 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_mapbl_bytes": 0, 2024-03-14T10:04:50.621 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_mapbl_items": 0, 2024-03-14T10:04:50.621 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_pglog_bytes": 0, 2024-03-14T10:04:50.621 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_pglog_items": 0, 2024-03-14T10:04:50.621 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_bytes": 9832, 2024-03-14T10:04:50.621 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_items": 205, 2024-03-14T10:04:50.621 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_mapping_bytes": 0, 2024-03-14T10:04:50.621 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_mapping_items": 0, 2024-03-14T10:04:50.621 INFO:teuthology.orchestra.run.smithi132.stdout: "pgmap_bytes": 0, 2024-03-14T10:04:50.621 INFO:teuthology.orchestra.run.smithi132.stdout: "pgmap_items": 0, 2024-03-14T10:04:50.621 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_co_bytes": 8320, 2024-03-14T10:04:50.621 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_co_items": 13, 2024-03-14T10:04:50.621 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_1_bytes": 0, 2024-03-14T10:04:50.621 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_1_items": 0, 2024-03-14T10:04:50.622 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_2_bytes": 0, 2024-03-14T10:04:50.622 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_2_items": 0 2024-03-14T10:04:50.622 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.622 INFO:teuthology.orchestra.run.smithi132.stdout: "objecter": { 2024-03-14T10:04:50.622 INFO:teuthology.orchestra.run.smithi132.stdout: "op_active": 0, 2024-03-14T10:04:50.622 INFO:teuthology.orchestra.run.smithi132.stdout: "op_laggy": 0, 2024-03-14T10:04:50.622 INFO:teuthology.orchestra.run.smithi132.stdout: "op_send": 2125, 2024-03-14T10:04:50.622 INFO:teuthology.orchestra.run.smithi132.stdout: "op_send_bytes": 21, 2024-03-14T10:04:50.622 INFO:teuthology.orchestra.run.smithi132.stdout: "op_resend": 3, 2024-03-14T10:04:50.622 INFO:teuthology.orchestra.run.smithi132.stdout: "op_reply": 2125, 2024-03-14T10:04:50.622 INFO:teuthology.orchestra.run.smithi132.stdout: "op_latency": { 2024-03-14T10:04:50.622 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 2125, 2024-03-14T10:04:50.623 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 1.119970414, 2024-03-14T10:04:50.623 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000527044 2024-03-14T10:04:50.623 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.623 INFO:teuthology.orchestra.run.smithi132.stdout: "op_inflight": 0, 2024-03-14T10:04:50.623 INFO:teuthology.orchestra.run.smithi132.stdout: "oplen_avg": { 2024-03-14T10:04:50.623 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 2125, 2024-03-14T10:04:50.623 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 2126 2024-03-14T10:04:50.623 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.623 INFO:teuthology.orchestra.run.smithi132.stdout: "op": 2125, 2024-03-14T10:04:50.623 INFO:teuthology.orchestra.run.smithi132.stdout: "op_r": 2125, 2024-03-14T10:04:50.623 INFO:teuthology.orchestra.run.smithi132.stdout: "op_w": 0, 2024-03-14T10:04:50.623 INFO:teuthology.orchestra.run.smithi132.stdout: "op_rmw": 0, 2024-03-14T10:04:50.623 INFO:teuthology.orchestra.run.smithi132.stdout: "op_pg": 0, 2024-03-14T10:04:50.623 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_stat": 1378, 2024-03-14T10:04:50.623 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_create": 0, 2024-03-14T10:04:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_read": 745, 2024-03-14T10:04:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_write": 0, 2024-03-14T10:04:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_writefull": 0, 2024-03-14T10:04:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_writesame": 0, 2024-03-14T10:04:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_append": 0, 2024-03-14T10:04:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_zero": 0, 2024-03-14T10:04:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_truncate": 0, 2024-03-14T10:04:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_delete": 0, 2024-03-14T10:04:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_mapext": 0, 2024-03-14T10:04:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_sparse_read": 0, 2024-03-14T10:04:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_clonerange": 0, 2024-03-14T10:04:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_getxattr": 1, 2024-03-14T10:04:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_setxattr": 0, 2024-03-14T10:04:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_cmpxattr": 0, 2024-03-14T10:04:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_rmxattr": 0, 2024-03-14T10:04:50.624 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_resetxattrs": 0, 2024-03-14T10:04:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_call": 0, 2024-03-14T10:04:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_watch": 0, 2024-03-14T10:04:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_notify": 0, 2024-03-14T10:04:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_src_cmpxattr": 0, 2024-03-14T10:04:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_pgls": 0, 2024-03-14T10:04:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_pgls_filter": 0, 2024-03-14T10:04:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_other": 0, 2024-03-14T10:04:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_active": 0, 2024-03-14T10:04:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_send": 0, 2024-03-14T10:04:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_resend": 0, 2024-03-14T10:04:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_ping": 0, 2024-03-14T10:04:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_active": 0, 2024-03-14T10:04:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_send": 0, 2024-03-14T10:04:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_resend": 0, 2024-03-14T10:04:50.625 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_active": 0, 2024-03-14T10:04:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_send": 0, 2024-03-14T10:04:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_resend": 0, 2024-03-14T10:04:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_active": 0, 2024-03-14T10:04:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_send": 0, 2024-03-14T10:04:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_resend": 0, 2024-03-14T10:04:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "command_active": 0, 2024-03-14T10:04:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "command_send": 0, 2024-03-14T10:04:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "command_resend": 0, 2024-03-14T10:04:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "map_epoch": 96, 2024-03-14T10:04:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "map_full": 0, 2024-03-14T10:04:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "map_inc": 13, 2024-03-14T10:04:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_sessions": 4, 2024-03-14T10:04:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_session_open": 4, 2024-03-14T10:04:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_session_close": 0, 2024-03-14T10:04:50.626 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_laggy": 0, 2024-03-14T10:04:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_wr": 0, 2024-03-14T10:04:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_rd": 2, 2024-03-14T10:04:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_del": 0 2024-03-14T10:04:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "oft": { 2024-03-14T10:04:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_objs": 0, 2024-03-14T10:04:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_kv_pairs": 0, 2024-03-14T10:04:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_updates": 0, 2024-03-14T10:04:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_removes": 0 2024-03-14T10:04:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "osd-slow-ops": { 2024-03-14T10:04:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "slow_ops_count": 0 2024-03-14T10:04:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "purge_queue": { 2024-03-14T10:04:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_ops": 0, 2024-03-14T10:04:50.627 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_ops_high_water": 0, 2024-03-14T10:04:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing": 0, 2024-03-14T10:04:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_high_water": 0, 2024-03-14T10:04:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executed": 0, 2024-03-14T10:04:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_item_in_journal": 0 2024-03-14T10:04:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-14T10:04:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-03-14T10:04:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 104857600, 2024-03-14T10:04:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-03-14T10:04:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 2369, 2024-03-14T10:04:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 2094344, 2024-03-14T10:04:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 2369, 2024-03-14T10:04:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-03-14T10:04:50.628 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 2369, 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 2094344, 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-objecter_bytes": { 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 104857600, 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:50.629 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 2125, 2024-03-14T10:04:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 1078, 2024-03-14T10:04:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 2125, 2024-03-14T10:04:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 1078, 2024-03-14T10:04:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-03-14T10:04:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-objecter_ops": { 2024-03-14T10:04:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-03-14T10:04:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 1024, 2024-03-14T10:04:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-03-14T10:04:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-03-14T10:04:50.630 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-03-14T10:04:50.631 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:50.631 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:50.631 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 2125, 2024-03-14T10:04:50.631 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 2125, 2024-03-14T10:04:50.631 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 2125, 2024-03-14T10:04:50.631 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 2125, 2024-03-14T10:04:50.631 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-03-14T10:04:50.631 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.631 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.631 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.631 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:50.631 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.631 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-write_buf_throttle": { 2024-03-14T10:04:50.631 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-03-14T10:04:50.631 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 3758096384, 2024-03-14T10:04:50.632 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-03-14T10:04:50.632 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-03-14T10:04:50.632 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-03-14T10:04:50.632 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:50.632 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:50.632 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-03-14T10:04:50.632 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-03-14T10:04:50.632 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 0, 2024-03-14T10:04:50.632 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 0, 2024-03-14T10:04:50.632 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-03-14T10:04:50.632 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.632 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.632 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.632 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:50.632 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:50.633 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-write_buf_throttle-0x559084412500": { 2024-03-14T10:04:50.633 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-03-14T10:04:50.633 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 3758096384, 2024-03-14T10:04:50.633 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-03-14T10:04:50.633 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-03-14T10:04:50.633 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-03-14T10:04:50.633 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:50.633 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:50.633 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-03-14T10:04:50.633 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-03-14T10:04:50.633 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 0, 2024-03-14T10:04:50.633 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 0, 2024-03-14T10:04:50.633 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-03-14T10:04:50.633 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:50.633 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:50.633 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:50.634 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:50.634 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:50.634 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-14T10:04:50.987 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-14T10:04:50.987 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-14T10:04:50.987 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-03-14T10:04:50.987 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-14T10:04:50.987 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-14T10:04:50.987 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-03-14T10:04:50.987 DEBUG:tasks.check_counter:Getting stats from h 2024-03-14T10:04:50.987 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-14T10:04:51.207 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.b/config 2024-03-14T10:04:51.463 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:51 smithi132 ceph-mon[47959]: pgmap v331: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:51.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:51 smithi152 ceph-mon[48279]: pgmap v331: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:51.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:51 smithi078 ceph-mon[38143]: pgmap v331: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:52.313 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-14T10:04:52.313 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-0": { 2024-03-14T10:04:52.313 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 71, 2024-03-14T10:04:52.313 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 132, 2024-03-14T10:04:52.314 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 18367, 2024-03-14T10:04:52.314 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 91613, 2024-03-14T10:04:52.314 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 17, 2024-03-14T10:04:52.314 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 3, 2024-03-14T10:04:52.314 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.019221712, 2024-03-14T10:04:52.314 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.004425414, 2024-03-14T10:04:52.314 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.009166172, 2024-03-14T10:04:52.314 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.000774494, 2024-03-14T10:04:52.314 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:52.314 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 130, 2024-03-14T10:04:52.314 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.017178715, 2024-03-14T10:04:52.315 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000132143 2024-03-14T10:04:52.315 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.315 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:52.315 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 103, 2024-03-14T10:04:52.315 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000042205, 2024-03-14T10:04:52.315 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000409 2024-03-14T10:04:52.315 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.315 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 5424, 2024-03-14T10:04:52.315 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 12640 2024-03-14T10:04:52.315 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.315 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-1": { 2024-03-14T10:04:52.315 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 349, 2024-03-14T10:04:52.315 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 334, 2024-03-14T10:04:52.315 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 1743826, 2024-03-14T10:04:52.316 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 491659, 2024-03-14T10:04:52.316 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 45, 2024-03-14T10:04:52.316 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 6, 2024-03-14T10:04:52.316 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.060432543, 2024-03-14T10:04:52.316 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.014010032, 2024-03-14T10:04:52.316 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.051447366, 2024-03-14T10:04:52.316 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.005371638, 2024-03-14T10:04:52.316 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:52.316 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 334, 2024-03-14T10:04:52.316 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.025772613, 2024-03-14T10:04:52.316 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000077163 2024-03-14T10:04:52.316 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.316 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:52.317 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 68, 2024-03-14T10:04:52.317 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000007689, 2024-03-14T10:04:52.317 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000113 2024-03-14T10:04:52.317 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.317 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 1733088, 2024-03-14T10:04:52.317 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 134304 2024-03-14T10:04:52.317 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.317 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-2": { 2024-03-14T10:04:52.317 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 964, 2024-03-14T10:04:52.317 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 873, 2024-03-14T10:04:52.317 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 269644, 2024-03-14T10:04:52.317 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 809415, 2024-03-14T10:04:52.317 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 9, 2024-03-14T10:04:52.317 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 4, 2024-03-14T10:04:52.318 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.078499386, 2024-03-14T10:04:52.318 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.037852536, 2024-03-14T10:04:52.318 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.040587265, 2024-03-14T10:04:52.318 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.003456547, 2024-03-14T10:04:52.318 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:52.318 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 870, 2024-03-14T10:04:52.318 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.051887623, 2024-03-14T10:04:52.318 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000059640 2024-03-14T10:04:52.318 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.318 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:52.318 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 1619, 2024-03-14T10:04:52.318 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000364086, 2024-03-14T10:04:52.318 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000224 2024-03-14T10:04:52.319 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.319 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 2176, 2024-03-14T10:04:52.319 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 433360 2024-03-14T10:04:52.319 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.319 INFO:teuthology.orchestra.run.smithi132.stdout: "cct": { 2024-03-14T10:04:52.319 INFO:teuthology.orchestra.run.smithi132.stdout: "total_workers": 1, 2024-03-14T10:04:52.319 INFO:teuthology.orchestra.run.smithi132.stdout: "unhealthy_workers": 0 2024-03-14T10:04:52.319 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.319 INFO:teuthology.orchestra.run.smithi132.stdout: "finisher-MDSRank": { 2024-03-14T10:04:52.319 INFO:teuthology.orchestra.run.smithi132.stdout: "queue_len": 0, 2024-03-14T10:04:52.319 INFO:teuthology.orchestra.run.smithi132.stdout: "complete_latency": { 2024-03-14T10:04:52.319 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 76, 2024-03-14T10:04:52.319 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.002034665, 2024-03-14T10:04:52.320 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000026771 2024-03-14T10:04:52.320 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:52.320 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.320 INFO:teuthology.orchestra.run.smithi132.stdout: "finisher-PurgeQueue": { 2024-03-14T10:04:52.320 INFO:teuthology.orchestra.run.smithi132.stdout: "queue_len": 0, 2024-03-14T10:04:52.320 INFO:teuthology.orchestra.run.smithi132.stdout: "complete_latency": { 2024-03-14T10:04:52.320 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 2, 2024-03-14T10:04:52.320 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000010114, 2024-03-14T10:04:52.320 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000005057 2024-03-14T10:04:52.320 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:52.320 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.320 INFO:teuthology.orchestra.run.smithi132.stdout: "mds": { 2024-03-14T10:04:52.320 INFO:teuthology.orchestra.run.smithi132.stdout: "request": 23, 2024-03-14T10:04:52.321 INFO:teuthology.orchestra.run.smithi132.stdout: "reply": 14, 2024-03-14T10:04:52.321 INFO:teuthology.orchestra.run.smithi132.stdout: "reply_latency": { 2024-03-14T10:04:52.321 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 14, 2024-03-14T10:04:52.321 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.005571288, 2024-03-14T10:04:52.321 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000397949 2024-03-14T10:04:52.321 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.321 INFO:teuthology.orchestra.run.smithi132.stdout: "slow_reply": 0, 2024-03-14T10:04:52.321 INFO:teuthology.orchestra.run.smithi132.stdout: "forward": 9, 2024-03-14T10:04:52.321 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_fetch_complete": 0, 2024-03-14T10:04:52.321 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_fetch_keys": 0, 2024-03-14T10:04:52.321 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_commit": 11, 2024-03-14T10:04:52.321 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_split": 0, 2024-03-14T10:04:52.322 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_merge": 0, 2024-03-14T10:04:52.322 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes": 15, 2024-03-14T10:04:52.322 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_top": 0, 2024-03-14T10:04:52.322 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_bottom": 15, 2024-03-14T10:04:52.322 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_pin_tail": 0, 2024-03-14T10:04:52.322 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_pinned": 14, 2024-03-14T10:04:52.322 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_expired": 7, 2024-03-14T10:04:52.322 INFO:teuthology.orchestra.run.smithi132.stdout: "inodes_with_caps": 3, 2024-03-14T10:04:52.322 INFO:teuthology.orchestra.run.smithi132.stdout: "caps": 3, 2024-03-14T10:04:52.322 INFO:teuthology.orchestra.run.smithi132.stdout: "subtrees": 3, 2024-03-14T10:04:52.322 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse": 35, 2024-03-14T10:04:52.322 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_hit": 24, 2024-03-14T10:04:52.322 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_forward": 2, 2024-03-14T10:04:52.323 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_discover": 3, 2024-03-14T10:04:52.323 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_dir_fetch": 0, 2024-03-14T10:04:52.323 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_remote_ino": 0, 2024-03-14T10:04:52.323 INFO:teuthology.orchestra.run.smithi132.stdout: "traverse_lock": 0, 2024-03-14T10:04:52.323 INFO:teuthology.orchestra.run.smithi132.stdout: "load_cent": 0, 2024-03-14T10:04:52.323 INFO:teuthology.orchestra.run.smithi132.stdout: "q": 0, 2024-03-14T10:04:52.323 INFO:teuthology.orchestra.run.smithi132.stdout: "exported": 0, 2024-03-14T10:04:52.323 INFO:teuthology.orchestra.run.smithi132.stdout: "exported_inodes": 0, 2024-03-14T10:04:52.323 INFO:teuthology.orchestra.run.smithi132.stdout: "imported": 0, 2024-03-14T10:04:52.323 INFO:teuthology.orchestra.run.smithi132.stdout: "imported_inodes": 0, 2024-03-14T10:04:52.323 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_dir_fetch": 0, 2024-03-14T10:04:52.323 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_backtrace_fetch": 0, 2024-03-14T10:04:52.323 INFO:teuthology.orchestra.run.smithi132.stdout: "openino_peer_discover": 0, 2024-03-14T10:04:52.324 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rfiles": 4, 2024-03-14T10:04:52.324 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rbytes": 234, 2024-03-14T10:04:52.324 INFO:teuthology.orchestra.run.smithi132.stdout: "root_rsnaps": 6, 2024-03-14T10:04:52.324 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_backtrace_fetch": 0, 2024-03-14T10:04:52.324 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_set_tag": 0, 2024-03-14T10:04:52.324 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_backtrace_repaired": 0, 2024-03-14T10:04:52.324 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_inotable_repaired": 0, 2024-03-14T10:04:52.324 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dir_inodes": 0, 2024-03-14T10:04:52.324 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dir_base_inodes": 0, 2024-03-14T10:04:52.324 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_dirfrag_rstats": 0, 2024-03-14T10:04:52.324 INFO:teuthology.orchestra.run.smithi132.stdout: "scrub_file_inodes": 0, 2024-03-14T10:04:52.324 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_inode_file_caps": 0, 2024-03-14T10:04:52.324 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_revoke": 2, 2024-03-14T10:04:52.324 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_grant": 3, 2024-03-14T10:04:52.325 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_trunc": 0, 2024-03-14T10:04:52.325 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-14T10:04:52.325 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-14T10:04:52.325 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_caps": 2, 2024-03-14T10:04:52.325 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_caps_dirty": 0, 2024-03-14T10:04:52.325 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_cap_release": 0, 2024-03-14T10:04:52.325 INFO:teuthology.orchestra.run.smithi132.stdout: "process_request_cap_release": 3 2024-03-14T10:04:52.325 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.325 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_cache": { 2024-03-14T10:04:52.325 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_update": 0, 2024-03-14T10:04:52.325 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_update_receipt": 3, 2024-03-14T10:04:52.325 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_try_discover": 2, 2024-03-14T10:04:52.325 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_send_discover": 5, 2024-03-14T10:04:52.326 INFO:teuthology.orchestra.run.smithi132.stdout: "dir_handle_discover": 0, 2024-03-14T10:04:52.326 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays": 0, 2024-03-14T10:04:52.326 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays_delayed": 0, 2024-03-14T10:04:52.326 INFO:teuthology.orchestra.run.smithi132.stdout: "num_strays_enqueuing": 0, 2024-03-14T10:04:52.326 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_created": 0, 2024-03-14T10:04:52.326 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_enqueued": 0, 2024-03-14T10:04:52.326 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_reintegrated": 0, 2024-03-14T10:04:52.326 INFO:teuthology.orchestra.run.smithi132.stdout: "strays_migrated": 0, 2024-03-14T10:04:52.326 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_processing": 0, 2024-03-14T10:04:52.326 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_enqueued": 0, 2024-03-14T10:04:52.326 INFO:teuthology.orchestra.run.smithi132.stdout: "num_recovering_prioritized": 0, 2024-03-14T10:04:52.326 INFO:teuthology.orchestra.run.smithi132.stdout: "recovery_started": 0, 2024-03-14T10:04:52.326 INFO:teuthology.orchestra.run.smithi132.stdout: "recovery_completed": 0, 2024-03-14T10:04:52.326 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_enqueue_scrub": 0, 2024-03-14T10:04:52.327 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_exportdir": 0, 2024-03-14T10:04:52.327 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_flush": 0, 2024-03-14T10:04:52.327 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_fragmentdir": 0, 2024-03-14T10:04:52.327 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_fragstats": 0, 2024-03-14T10:04:52.327 INFO:teuthology.orchestra.run.smithi132.stdout: "ireq_inodestats": 0 2024-03-14T10:04:52.327 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.327 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_log": { 2024-03-14T10:04:52.327 INFO:teuthology.orchestra.run.smithi132.stdout: "evadd": 11, 2024-03-14T10:04:52.327 INFO:teuthology.orchestra.run.smithi132.stdout: "evex": 0, 2024-03-14T10:04:52.327 INFO:teuthology.orchestra.run.smithi132.stdout: "evtrm": 0, 2024-03-14T10:04:52.327 INFO:teuthology.orchestra.run.smithi132.stdout: "ev": 11, 2024-03-14T10:04:52.327 INFO:teuthology.orchestra.run.smithi132.stdout: "evexg": 0, 2024-03-14T10:04:52.327 INFO:teuthology.orchestra.run.smithi132.stdout: "evexd": 0, 2024-03-14T10:04:52.328 INFO:teuthology.orchestra.run.smithi132.stdout: "segadd": 1, 2024-03-14T10:04:52.328 INFO:teuthology.orchestra.run.smithi132.stdout: "segex": 0, 2024-03-14T10:04:52.328 INFO:teuthology.orchestra.run.smithi132.stdout: "segtrm": 0, 2024-03-14T10:04:52.328 INFO:teuthology.orchestra.run.smithi132.stdout: "seg": 1, 2024-03-14T10:04:52.328 INFO:teuthology.orchestra.run.smithi132.stdout: "segexg": 0, 2024-03-14T10:04:52.328 INFO:teuthology.orchestra.run.smithi132.stdout: "segexd": 0, 2024-03-14T10:04:52.328 INFO:teuthology.orchestra.run.smithi132.stdout: "expos": 4194304, 2024-03-14T10:04:52.328 INFO:teuthology.orchestra.run.smithi132.stdout: "wrpos": 4197678, 2024-03-14T10:04:52.328 INFO:teuthology.orchestra.run.smithi132.stdout: "rdpos": 0, 2024-03-14T10:04:52.328 INFO:teuthology.orchestra.run.smithi132.stdout: "jlat": { 2024-03-14T10:04:52.328 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 11, 2024-03-14T10:04:52.328 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.026205263, 2024-03-14T10:04:52.328 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.002382296 2024-03-14T10:04:52.328 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.329 INFO:teuthology.orchestra.run.smithi132.stdout: "replayed": 0 2024-03-14T10:04:52.329 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.329 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_mem": { 2024-03-14T10:04:52.329 INFO:teuthology.orchestra.run.smithi132.stdout: "ino": 19, 2024-03-14T10:04:52.329 INFO:teuthology.orchestra.run.smithi132.stdout: "ino+": 22, 2024-03-14T10:04:52.329 INFO:teuthology.orchestra.run.smithi132.stdout: "ino-": 3, 2024-03-14T10:04:52.329 INFO:teuthology.orchestra.run.smithi132.stdout: "dir": 18, 2024-03-14T10:04:52.329 INFO:teuthology.orchestra.run.smithi132.stdout: "dir+": 22, 2024-03-14T10:04:52.329 INFO:teuthology.orchestra.run.smithi132.stdout: "dir-": 4, 2024-03-14T10:04:52.329 INFO:teuthology.orchestra.run.smithi132.stdout: "dn": 15, 2024-03-14T10:04:52.329 INFO:teuthology.orchestra.run.smithi132.stdout: "dn+": 22, 2024-03-14T10:04:52.329 INFO:teuthology.orchestra.run.smithi132.stdout: "dn-": 7, 2024-03-14T10:04:52.329 INFO:teuthology.orchestra.run.smithi132.stdout: "cap": 3, 2024-03-14T10:04:52.330 INFO:teuthology.orchestra.run.smithi132.stdout: "cap+": 11, 2024-03-14T10:04:52.330 INFO:teuthology.orchestra.run.smithi132.stdout: "cap-": 8, 2024-03-14T10:04:52.330 INFO:teuthology.orchestra.run.smithi132.stdout: "rss": 51968, 2024-03-14T10:04:52.330 INFO:teuthology.orchestra.run.smithi132.stdout: "heap": 198964 2024-03-14T10:04:52.330 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.330 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_server": { 2024-03-14T10:04:52.330 INFO:teuthology.orchestra.run.smithi132.stdout: "dispatch_client_request": 28, 2024-03-14T10:04:52.330 INFO:teuthology.orchestra.run.smithi132.stdout: "dispatch_server_request": 0, 2024-03-14T10:04:52.330 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_request": 23, 2024-03-14T10:04:52.330 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_client_session": 32, 2024-03-14T10:04:52.330 INFO:teuthology.orchestra.run.smithi132.stdout: "handle_peer_request": 1, 2024-03-14T10:04:52.330 INFO:teuthology.orchestra.run.smithi132.stdout: "req_create_latency": { 2024-03-14T10:04:52.331 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.331 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.331 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.331 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getattr_latency": { 2024-03-14T10:04:52.331 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 8, 2024-03-14T10:04:52.331 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.004138614, 2024-03-14T10:04:52.331 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000517326 2024-03-14T10:04:52.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.331 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getfilelock_latency": { 2024-03-14T10:04:52.331 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.331 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.331 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.331 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.332 INFO:teuthology.orchestra.run.smithi132.stdout: "req_link_latency": { 2024-03-14T10:04:52.332 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.332 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.332 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.332 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.332 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookup_latency": { 2024-03-14T10:04:52.332 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 5, 2024-03-14T10:04:52.332 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.001290900, 2024-03-14T10:04:52.332 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000258180 2024-03-14T10:04:52.332 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.332 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookuphash_latency": { 2024-03-14T10:04:52.332 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.332 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.332 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.333 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupino_latency": { 2024-03-14T10:04:52.333 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 1, 2024-03-14T10:04:52.333 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000141774, 2024-03-14T10:04:52.333 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000141774 2024-03-14T10:04:52.333 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.333 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupname_latency": { 2024-03-14T10:04:52.333 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.333 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.333 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.334 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.334 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupparent_latency": { 2024-03-14T10:04:52.334 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.334 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.334 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.334 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.334 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lookupsnap_latency": { 2024-03-14T10:04:52.334 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.334 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.334 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.334 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.334 INFO:teuthology.orchestra.run.smithi132.stdout: "req_lssnap_latency": { 2024-03-14T10:04:52.334 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.335 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.335 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.335 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.335 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mkdir_latency": { 2024-03-14T10:04:52.335 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.335 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.335 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.335 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.335 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mknod_latency": { 2024-03-14T10:04:52.335 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.335 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.335 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.335 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.335 INFO:teuthology.orchestra.run.smithi132.stdout: "req_mksnap_latency": { 2024-03-14T10:04:52.336 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.336 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.336 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.336 INFO:teuthology.orchestra.run.smithi132.stdout: "req_open_latency": { 2024-03-14T10:04:52.336 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.336 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.336 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.336 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.336 INFO:teuthology.orchestra.run.smithi132.stdout: "req_readdir_latency": { 2024-03-14T10:04:52.336 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.336 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.337 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.337 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rename_latency": { 2024-03-14T10:04:52.337 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.337 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.337 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.337 INFO:teuthology.orchestra.run.smithi132.stdout: "req_renamesnap_latency": { 2024-03-14T10:04:52.337 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.337 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.337 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.337 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.337 INFO:teuthology.orchestra.run.smithi132.stdout: "req_snapdiff_latency": { 2024-03-14T10:04:52.338 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.338 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.338 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.338 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.338 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmdir_latency": { 2024-03-14T10:04:52.338 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.338 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.338 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.338 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.338 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmsnap_latency": { 2024-03-14T10:04:52.338 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.338 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.339 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.339 INFO:teuthology.orchestra.run.smithi132.stdout: "req_rmxattr_latency": { 2024-03-14T10:04:52.339 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.339 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.339 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.339 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setattr_latency": { 2024-03-14T10:04:52.339 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.339 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.339 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.339 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.340 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setdirlayout_latency": { 2024-03-14T10:04:52.340 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.340 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.340 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.340 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setfilelock_latency": { 2024-03-14T10:04:52.340 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.340 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.340 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.340 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.340 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setlayout_latency": { 2024-03-14T10:04:52.340 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.340 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.341 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.341 INFO:teuthology.orchestra.run.smithi132.stdout: "req_setxattr_latency": { 2024-03-14T10:04:52.341 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.341 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.341 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.341 INFO:teuthology.orchestra.run.smithi132.stdout: "req_symlink_latency": { 2024-03-14T10:04:52.341 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.341 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.341 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.341 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.342 INFO:teuthology.orchestra.run.smithi132.stdout: "req_unlink_latency": { 2024-03-14T10:04:52.342 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.342 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.342 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.342 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.342 INFO:teuthology.orchestra.run.smithi132.stdout: "cap_revoke_eviction": 0, 2024-03-14T10:04:52.342 INFO:teuthology.orchestra.run.smithi132.stdout: "cap_acquisition_throttle": 0, 2024-03-14T10:04:52.342 INFO:teuthology.orchestra.run.smithi132.stdout: "req_getvxattr_latency": { 2024-03-14T10:04:52.342 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.342 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.342 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.343 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:52.343 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.343 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_sessions": { 2024-03-14T10:04:52.343 INFO:teuthology.orchestra.run.smithi132.stdout: "session_count": 2, 2024-03-14T10:04:52.343 INFO:teuthology.orchestra.run.smithi132.stdout: "session_add": 6, 2024-03-14T10:04:52.343 INFO:teuthology.orchestra.run.smithi132.stdout: "session_remove": 4, 2024-03-14T10:04:52.343 INFO:teuthology.orchestra.run.smithi132.stdout: "sessions_open": 2, 2024-03-14T10:04:52.343 INFO:teuthology.orchestra.run.smithi132.stdout: "sessions_stale": 0, 2024-03-14T10:04:52.343 INFO:teuthology.orchestra.run.smithi132.stdout: "total_load": 8, 2024-03-14T10:04:52.343 INFO:teuthology.orchestra.run.smithi132.stdout: "average_load": 2, 2024-03-14T10:04:52.343 INFO:teuthology.orchestra.run.smithi132.stdout: "avg_session_uptime": 625, 2024-03-14T10:04:52.343 INFO:teuthology.orchestra.run.smithi132.stdout: "mdthresh_evicted": 0 2024-03-14T10:04:52.344 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.344 INFO:teuthology.orchestra.run.smithi132.stdout: "mempool": { 2024-03-14T10:04:52.344 INFO:teuthology.orchestra.run.smithi132.stdout: "bloom_filter_bytes": 0, 2024-03-14T10:04:52.344 INFO:teuthology.orchestra.run.smithi132.stdout: "bloom_filter_items": 0, 2024-03-14T10:04:52.344 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_alloc_bytes": 0, 2024-03-14T10:04:52.344 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_alloc_items": 0, 2024-03-14T10:04:52.344 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_data_bytes": 0, 2024-03-14T10:04:52.344 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_data_items": 0, 2024-03-14T10:04:52.344 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-14T10:04:52.344 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_onode_items": 0, 2024-03-14T10:04:52.344 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-14T10:04:52.344 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_meta_items": 0, 2024-03-14T10:04:52.344 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_other_bytes": 0, 2024-03-14T10:04:52.345 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_other_items": 0, 2024-03-14T10:04:52.345 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-14T10:04:52.345 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_buffer_items": 0, 2024-03-14T10:04:52.345 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_extent_bytes": 0, 2024-03-14T10:04:52.345 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_extent_items": 0, 2024-03-14T10:04:52.345 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_blob_bytes": 0, 2024-03-14T10:04:52.345 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_blob_items": 0, 2024-03-14T10:04:52.345 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-14T10:04:52.345 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_shared_blob_items": 0, 2024-03-14T10:04:52.345 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-14T10:04:52.345 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_inline_bl_items": 0, 2024-03-14T10:04:52.345 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_fsck_bytes": 0, 2024-03-14T10:04:52.345 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_fsck_items": 0, 2024-03-14T10:04:52.346 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_txc_bytes": 0, 2024-03-14T10:04:52.346 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_txc_items": 0, 2024-03-14T10:04:52.346 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-14T10:04:52.346 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_deferred_items": 0, 2024-03-14T10:04:52.346 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_bytes": 0, 2024-03-14T10:04:52.346 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_items": 0, 2024-03-14T10:04:52.346 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_bytes": 0, 2024-03-14T10:04:52.346 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_items": 0, 2024-03-14T10:04:52.346 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_reader_bytes": 0, 2024-03-14T10:04:52.346 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_reader_items": 0, 2024-03-14T10:04:52.346 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_writer_bytes": 0, 2024-03-14T10:04:52.347 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_writer_items": 0, 2024-03-14T10:04:52.347 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_anon_bytes": 267157, 2024-03-14T10:04:52.347 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_anon_items": 114, 2024-03-14T10:04:52.347 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_meta_bytes": 88, 2024-03-14T10:04:52.347 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_meta_items": 1, 2024-03-14T10:04:52.347 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_bytes": 0, 2024-03-14T10:04:52.347 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_items": 0, 2024-03-14T10:04:52.347 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_mapbl_bytes": 0, 2024-03-14T10:04:52.347 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_mapbl_items": 0, 2024-03-14T10:04:52.347 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_pglog_bytes": 0, 2024-03-14T10:04:52.347 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_pglog_items": 0, 2024-03-14T10:04:52.347 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_bytes": 9832, 2024-03-14T10:04:52.347 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_items": 205, 2024-03-14T10:04:52.348 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_mapping_bytes": 0, 2024-03-14T10:04:52.348 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_mapping_items": 0, 2024-03-14T10:04:52.348 INFO:teuthology.orchestra.run.smithi132.stdout: "pgmap_bytes": 0, 2024-03-14T10:04:52.348 INFO:teuthology.orchestra.run.smithi132.stdout: "pgmap_items": 0, 2024-03-14T10:04:52.348 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_co_bytes": 82893, 2024-03-14T10:04:52.348 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_co_items": 341, 2024-03-14T10:04:52.348 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_1_bytes": 0, 2024-03-14T10:04:52.348 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_1_items": 0, 2024-03-14T10:04:52.348 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_2_bytes": 0, 2024-03-14T10:04:52.348 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_2_items": 0 2024-03-14T10:04:52.348 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.348 INFO:teuthology.orchestra.run.smithi132.stdout: "objecter": { 2024-03-14T10:04:52.349 INFO:teuthology.orchestra.run.smithi132.stdout: "op_active": 0, 2024-03-14T10:04:52.349 INFO:teuthology.orchestra.run.smithi132.stdout: "op_laggy": 0, 2024-03-14T10:04:52.349 INFO:teuthology.orchestra.run.smithi132.stdout: "op_send": 54, 2024-03-14T10:04:52.349 INFO:teuthology.orchestra.run.smithi132.stdout: "op_send_bytes": 14769, 2024-03-14T10:04:52.349 INFO:teuthology.orchestra.run.smithi132.stdout: "op_resend": 33, 2024-03-14T10:04:52.349 INFO:teuthology.orchestra.run.smithi132.stdout: "op_reply": 54, 2024-03-14T10:04:52.349 INFO:teuthology.orchestra.run.smithi132.stdout: "op_latency": { 2024-03-14T10:04:52.349 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 54, 2024-03-14T10:04:52.349 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.269992949, 2024-03-14T10:04:52.349 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.004999869 2024-03-14T10:04:52.349 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.349 INFO:teuthology.orchestra.run.smithi132.stdout: "op_inflight": 0, 2024-03-14T10:04:52.349 INFO:teuthology.orchestra.run.smithi132.stdout: "oplen_avg": { 2024-03-14T10:04:52.350 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 54, 2024-03-14T10:04:52.350 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 81 2024-03-14T10:04:52.350 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.350 INFO:teuthology.orchestra.run.smithi132.stdout: "op": 54, 2024-03-14T10:04:52.350 INFO:teuthology.orchestra.run.smithi132.stdout: "op_r": 0, 2024-03-14T10:04:52.350 INFO:teuthology.orchestra.run.smithi132.stdout: "op_w": 54, 2024-03-14T10:04:52.350 INFO:teuthology.orchestra.run.smithi132.stdout: "op_rmw": 0, 2024-03-14T10:04:52.350 INFO:teuthology.orchestra.run.smithi132.stdout: "op_pg": 0, 2024-03-14T10:04:52.350 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_stat": 0, 2024-03-14T10:04:52.350 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_create": 10, 2024-03-14T10:04:52.350 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_read": 0, 2024-03-14T10:04:52.350 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_write": 11, 2024-03-14T10:04:52.351 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_writefull": 9, 2024-03-14T10:04:52.351 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_writesame": 0, 2024-03-14T10:04:52.351 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_append": 0, 2024-03-14T10:04:52.351 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_zero": 0, 2024-03-14T10:04:52.351 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_truncate": 0, 2024-03-14T10:04:52.351 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_delete": 6, 2024-03-14T10:04:52.351 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_mapext": 0, 2024-03-14T10:04:52.351 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_sparse_read": 0, 2024-03-14T10:04:52.351 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_clonerange": 0, 2024-03-14T10:04:52.351 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_getxattr": 0, 2024-03-14T10:04:52.351 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_setxattr": 20, 2024-03-14T10:04:52.351 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_cmpxattr": 0, 2024-03-14T10:04:52.351 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_rmxattr": 0, 2024-03-14T10:04:52.352 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_resetxattrs": 0, 2024-03-14T10:04:52.352 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_call": 0, 2024-03-14T10:04:52.352 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_watch": 0, 2024-03-14T10:04:52.352 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_notify": 0, 2024-03-14T10:04:52.352 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_src_cmpxattr": 0, 2024-03-14T10:04:52.352 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_pgls": 0, 2024-03-14T10:04:52.352 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_pgls_filter": 0, 2024-03-14T10:04:52.352 INFO:teuthology.orchestra.run.smithi132.stdout: "osdop_other": 0, 2024-03-14T10:04:52.352 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_active": 0, 2024-03-14T10:04:52.352 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_send": 0, 2024-03-14T10:04:52.352 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_resend": 0, 2024-03-14T10:04:52.352 INFO:teuthology.orchestra.run.smithi132.stdout: "linger_ping": 0, 2024-03-14T10:04:52.352 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_active": 0, 2024-03-14T10:04:52.353 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_send": 0, 2024-03-14T10:04:52.353 INFO:teuthology.orchestra.run.smithi132.stdout: "poolop_resend": 0, 2024-03-14T10:04:52.353 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_active": 0, 2024-03-14T10:04:52.353 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_send": 0, 2024-03-14T10:04:52.353 INFO:teuthology.orchestra.run.smithi132.stdout: "poolstat_resend": 0, 2024-03-14T10:04:52.353 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_active": 0, 2024-03-14T10:04:52.353 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_send": 0, 2024-03-14T10:04:52.353 INFO:teuthology.orchestra.run.smithi132.stdout: "statfs_resend": 0, 2024-03-14T10:04:52.353 INFO:teuthology.orchestra.run.smithi132.stdout: "command_active": 0, 2024-03-14T10:04:52.353 INFO:teuthology.orchestra.run.smithi132.stdout: "command_send": 0, 2024-03-14T10:04:52.353 INFO:teuthology.orchestra.run.smithi132.stdout: "command_resend": 0, 2024-03-14T10:04:52.353 INFO:teuthology.orchestra.run.smithi132.stdout: "map_epoch": 96, 2024-03-14T10:04:52.353 INFO:teuthology.orchestra.run.smithi132.stdout: "map_full": 0, 2024-03-14T10:04:52.353 INFO:teuthology.orchestra.run.smithi132.stdout: "map_inc": 13, 2024-03-14T10:04:52.354 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_sessions": 8, 2024-03-14T10:04:52.354 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_session_open": 8, 2024-03-14T10:04:52.354 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_session_close": 0, 2024-03-14T10:04:52.354 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_laggy": 0, 2024-03-14T10:04:52.354 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_wr": 24, 2024-03-14T10:04:52.354 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_rd": 0, 2024-03-14T10:04:52.354 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_del": 1 2024-03-14T10:04:52.354 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.354 INFO:teuthology.orchestra.run.smithi132.stdout: "oft": { 2024-03-14T10:04:52.354 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_objs": 1, 2024-03-14T10:04:52.354 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_kv_pairs": 0, 2024-03-14T10:04:52.354 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_updates": 0, 2024-03-14T10:04:52.355 INFO:teuthology.orchestra.run.smithi132.stdout: "omap_total_removes": 0 2024-03-14T10:04:52.355 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.355 INFO:teuthology.orchestra.run.smithi132.stdout: "osd-slow-ops": { 2024-03-14T10:04:52.355 INFO:teuthology.orchestra.run.smithi132.stdout: "slow_ops_count": 23 2024-03-14T10:04:52.355 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.355 INFO:teuthology.orchestra.run.smithi132.stdout: "purge_queue": { 2024-03-14T10:04:52.355 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_ops": 0, 2024-03-14T10:04:52.355 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_ops_high_water": 0, 2024-03-14T10:04:52.355 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing": 0, 2024-03-14T10:04:52.355 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executing_high_water": 0, 2024-03-14T10:04:52.355 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_executed": 0, 2024-03-14T10:04:52.355 INFO:teuthology.orchestra.run.smithi132.stdout: "pq_item_in_journal": 0 2024-03-14T10:04:52.355 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.355 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-14T10:04:52.356 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-03-14T10:04:52.356 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 104857600, 2024-03-14T10:04:52.356 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-03-14T10:04:52.356 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 1380, 2024-03-14T10:04:52.356 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 1889192, 2024-03-14T10:04:52.356 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:52.356 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 1380, 2024-03-14T10:04:52.356 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-03-14T10:04:52.356 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-03-14T10:04:52.356 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 1380, 2024-03-14T10:04:52.356 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 1889192, 2024-03-14T10:04:52.357 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-03-14T10:04:52.357 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.357 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.357 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.357 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:52.357 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.357 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-objecter_bytes": { 2024-03-14T10:04:52.357 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-03-14T10:04:52.357 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 104857600, 2024-03-14T10:04:52.357 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-03-14T10:04:52.357 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-03-14T10:04:52.357 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-03-14T10:04:52.357 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:52.358 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:52.358 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 54, 2024-03-14T10:04:52.358 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 14769, 2024-03-14T10:04:52.358 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 54, 2024-03-14T10:04:52.358 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 14769, 2024-03-14T10:04:52.358 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-03-14T10:04:52.358 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.358 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.358 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.358 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:52.358 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.359 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-objecter_ops": { 2024-03-14T10:04:52.359 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-03-14T10:04:52.359 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 1024, 2024-03-14T10:04:52.359 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-03-14T10:04:52.359 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-03-14T10:04:52.359 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-03-14T10:04:52.359 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:52.359 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:52.359 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 54, 2024-03-14T10:04:52.359 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 54, 2024-03-14T10:04:52.359 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 54, 2024-03-14T10:04:52.359 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 54, 2024-03-14T10:04:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-03-14T10:04:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-write_buf_throttle": { 2024-03-14T10:04:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-03-14T10:04:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 3758096384, 2024-03-14T10:04:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-03-14T10:04:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 0, 2024-03-14T10:04:52.360 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 0, 2024-03-14T10:04:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-03-14T10:04:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-03-14T10:04:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 0, 2024-03-14T10:04:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 0, 2024-03-14T10:04:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-03-14T10:04:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:52.361 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-write_buf_throttle-0x561d39b420a0": { 2024-03-14T10:04:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-03-14T10:04:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 3758096384, 2024-03-14T10:04:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-03-14T10:04:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 11, 2024-03-14T10:04:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 3374, 2024-03-14T10:04:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 11, 2024-03-14T10:04:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-03-14T10:04:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-03-14T10:04:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 11, 2024-03-14T10:04:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 3374, 2024-03-14T10:04:52.362 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-03-14T10:04:52.363 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:52.363 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:52.363 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:52.363 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:52.363 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:52.363 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-14T10:04:52.679 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-14T10:04:52.680 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-14T10:04:52.680 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-03-14T10:04:52.680 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-14T10:04:52.680 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-14T10:04:52.680 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-03-14T10:04:52.680 DEBUG:tasks.check_counter:Getting stats from k 2024-03-14T10:04:52.680 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-14T10:04:52.900 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.b/config 2024-03-14T10:04:53.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:53 smithi132 ceph-mon[47959]: pgmap v332: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:53.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:53 smithi152 ceph-mon[48279]: pgmap v332: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:53.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:53 smithi078 ceph-mon[38143]: pgmap v332: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:53.899 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-03-14T10:04:53.899 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-0": { 2024-03-14T10:04:53.899 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 0, 2024-03-14T10:04:53.899 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 2, 2024-03-14T10:04:53.899 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 0, 2024-03-14T10:04:53.899 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 0, 2024-03-14T10:04:53.899 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 4, 2024-03-14T10:04:53.899 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 0, 2024-03-14T10:04:53.899 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.001089844, 2024-03-14T10:04:53.899 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.000000000, 2024-03-14T10:04:53.899 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.000235254, 2024-03-14T10:04:53.899 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-14T10:04:53.899 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:53.900 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:53.900 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:53.900 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:53.900 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:53.900 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:53.900 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:53.900 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:53.900 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:53.900 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:53.900 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-14T10:04:53.900 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 0 2024-03-14T10:04:53.900 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:53.900 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-1": { 2024-03-14T10:04:53.900 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 225, 2024-03-14T10:04:53.900 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 213, 2024-03-14T10:04:53.901 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 1689968, 2024-03-14T10:04:53.901 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 121520, 2024-03-14T10:04:53.901 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 2, 2024-03-14T10:04:53.901 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 1, 2024-03-14T10:04:53.901 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.043223063, 2024-03-14T10:04:53.901 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.011990086, 2024-03-14T10:04:53.901 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.037092060, 2024-03-14T10:04:53.901 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.004996819, 2024-03-14T10:04:53.901 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:53.901 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 212, 2024-03-14T10:04:53.901 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.012480184, 2024-03-14T10:04:53.901 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000058868 2024-03-14T10:04:53.901 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:53.901 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:53.902 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:53.902 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:53.902 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:53.902 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:53.902 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 1689968, 2024-03-14T10:04:53.902 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 121520 2024-03-14T10:04:53.902 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:53.902 INFO:teuthology.orchestra.run.smithi132.stdout: "AsyncMessenger::Worker-2": { 2024-03-14T10:04:53.902 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_messages": 6, 2024-03-14T10:04:53.902 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages": 165, 2024-03-14T10:04:53.902 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_bytes": 3312, 2024-03-14T10:04:53.902 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_bytes": 181344, 2024-03-14T10:04:53.902 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_created_connections": 5, 2024-03-14T10:04:53.902 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_active_connections": 3, 2024-03-14T10:04:53.903 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_total_time": 0.010362446, 2024-03-14T10:04:53.903 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_send_time": 0.006319907, 2024-03-14T10:04:53.903 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_recv_time": 0.002004358, 2024-03-14T10:04:53.903 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-14T10:04:53.903 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:53.903 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 163, 2024-03-14T10:04:53.903 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.024520528, 2024-03-14T10:04:53.903 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000150432 2024-03-14T10:04:53.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:53.903 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:53.903 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:53.903 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:53.903 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:53.903 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-14T10:04:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "msgr_send_encrypted_bytes": 181344 2024-03-14T10:04:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "cct": { 2024-03-14T10:04:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "total_workers": 0, 2024-03-14T10:04:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "unhealthy_workers": 0 2024-03-14T10:04:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "mempool": { 2024-03-14T10:04:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "bloom_filter_bytes": 0, 2024-03-14T10:04:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "bloom_filter_items": 0, 2024-03-14T10:04:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_alloc_bytes": 0, 2024-03-14T10:04:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_alloc_items": 0, 2024-03-14T10:04:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_data_bytes": 0, 2024-03-14T10:04:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_data_items": 0, 2024-03-14T10:04:53.904 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-14T10:04:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_onode_items": 0, 2024-03-14T10:04:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-14T10:04:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_meta_items": 0, 2024-03-14T10:04:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_other_bytes": 0, 2024-03-14T10:04:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_other_items": 0, 2024-03-14T10:04:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-14T10:04:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_cache_buffer_items": 0, 2024-03-14T10:04:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_extent_bytes": 0, 2024-03-14T10:04:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_extent_items": 0, 2024-03-14T10:04:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_blob_bytes": 0, 2024-03-14T10:04:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_blob_items": 0, 2024-03-14T10:04:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-14T10:04:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_shared_blob_items": 0, 2024-03-14T10:04:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-14T10:04:53.905 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_inline_bl_items": 0, 2024-03-14T10:04:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_fsck_bytes": 0, 2024-03-14T10:04:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_fsck_items": 0, 2024-03-14T10:04:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_txc_bytes": 0, 2024-03-14T10:04:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_txc_items": 0, 2024-03-14T10:04:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-14T10:04:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_deferred_items": 0, 2024-03-14T10:04:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_bytes": 0, 2024-03-14T10:04:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "bluestore_writing_items": 0, 2024-03-14T10:04:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_bytes": 0, 2024-03-14T10:04:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_items": 0, 2024-03-14T10:04:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_reader_bytes": 0, 2024-03-14T10:04:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_reader_items": 0, 2024-03-14T10:04:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_writer_bytes": 0, 2024-03-14T10:04:53.906 INFO:teuthology.orchestra.run.smithi132.stdout: "bluefs_file_writer_items": 0, 2024-03-14T10:04:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_anon_bytes": 149057, 2024-03-14T10:04:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_anon_items": 32, 2024-03-14T10:04:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_meta_bytes": 0, 2024-03-14T10:04:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "buffer_meta_items": 0, 2024-03-14T10:04:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_bytes": 0, 2024-03-14T10:04:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_items": 0, 2024-03-14T10:04:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_mapbl_bytes": 0, 2024-03-14T10:04:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_mapbl_items": 0, 2024-03-14T10:04:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_pglog_bytes": 0, 2024-03-14T10:04:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "osd_pglog_items": 0, 2024-03-14T10:04:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_bytes": 0, 2024-03-14T10:04:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_items": 0, 2024-03-14T10:04:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_mapping_bytes": 0, 2024-03-14T10:04:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "osdmap_mapping_items": 0, 2024-03-14T10:04:53.907 INFO:teuthology.orchestra.run.smithi132.stdout: "pgmap_bytes": 0, 2024-03-14T10:04:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "pgmap_items": 0, 2024-03-14T10:04:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_co_bytes": 928, 2024-03-14T10:04:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "mds_co_items": 2, 2024-03-14T10:04:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_1_bytes": 0, 2024-03-14T10:04:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_1_items": 0, 2024-03-14T10:04:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_2_bytes": 0, 2024-03-14T10:04:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "unittest_2_items": 0 2024-03-14T10:04:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: }, 2024-03-14T10:04:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-14T10:04:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "val": 0, 2024-03-14T10:04:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "max": 104857600, 2024-03-14T10:04:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "get_started": 0, 2024-03-14T10:04:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "get": 227, 2024-03-14T10:04:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "get_sum": 1658290, 2024-03-14T10:04:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:53.908 INFO:teuthology.orchestra.run.smithi132.stdout: "get_or_fail_success": 227, 2024-03-14T10:04:53.909 INFO:teuthology.orchestra.run.smithi132.stdout: "take": 0, 2024-03-14T10:04:53.909 INFO:teuthology.orchestra.run.smithi132.stdout: "take_sum": 0, 2024-03-14T10:04:53.909 INFO:teuthology.orchestra.run.smithi132.stdout: "put": 227, 2024-03-14T10:04:53.909 INFO:teuthology.orchestra.run.smithi132.stdout: "put_sum": 1658290, 2024-03-14T10:04:53.909 INFO:teuthology.orchestra.run.smithi132.stdout: "wait": { 2024-03-14T10:04:53.909 INFO:teuthology.orchestra.run.smithi132.stdout: "avgcount": 0, 2024-03-14T10:04:53.909 INFO:teuthology.orchestra.run.smithi132.stdout: "sum": 0.000000000, 2024-03-14T10:04:53.909 INFO:teuthology.orchestra.run.smithi132.stdout: "avgtime": 0.000000000 2024-03-14T10:04:53.909 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:53.909 INFO:teuthology.orchestra.run.smithi132.stdout: } 2024-03-14T10:04:53.909 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-03-14T10:04:54.262 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-14T10:04:54.263 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-14T10:04:54.263 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-03-14T10:04:54.263 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-03-14T10:04:54.263 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-03-14T10:04:54.263 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-03-14T10:04:54.263 DEBUG:tasks.check_counter:Getting stats from c 2024-03-14T10:04:54.263 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-14T10:04:54.508 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.c/config 2024-03-14T10:04:55.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:55 smithi152 ceph-mon[48279]: pgmap v333: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:55.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:55 smithi132 ceph-mon[47959]: pgmap v333: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:55.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:55 smithi078 ceph-mon[38143]: pgmap v333: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:55.557 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-03-14T10:04:55.557 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-0": { 2024-03-14T10:04:55.557 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 755, 2024-03-14T10:04:55.558 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 757, 2024-03-14T10:04:55.558 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 201695, 2024-03-14T10:04:55.558 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 234869, 2024-03-14T10:04:55.558 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 6, 2024-03-14T10:04:55.558 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 2, 2024-03-14T10:04:55.558 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.066178619, 2024-03-14T10:04:55.558 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.019067324, 2024-03-14T10:04:55.558 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.018231916, 2024-03-14T10:04:55.558 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.021342249, 2024-03-14T10:04:55.558 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:55.558 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 755, 2024-03-14T10:04:55.558 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.029515040, 2024-03-14T10:04:55.558 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000039092 2024-03-14T10:04:55.559 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.559 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:55.559 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.559 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.559 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.559 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.559 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-14T10:04:55.559 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 0 2024-03-14T10:04:55.559 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.559 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-1": { 2024-03-14T10:04:55.559 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 917, 2024-03-14T10:04:55.559 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 904, 2024-03-14T10:04:55.560 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 1882808, 2024-03-14T10:04:55.560 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 333386, 2024-03-14T10:04:55.560 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 4, 2024-03-14T10:04:55.560 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 3, 2024-03-14T10:04:55.560 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.105420141, 2024-03-14T10:04:55.560 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.027706651, 2024-03-14T10:04:55.560 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.051835828, 2024-03-14T10:04:55.560 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.030774232, 2024-03-14T10:04:55.560 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:55.560 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 903, 2024-03-14T10:04:55.560 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.034241241, 2024-03-14T10:04:55.560 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000037919 2024-03-14T10:04:55.560 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.560 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:55.560 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.561 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.561 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.561 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.561 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 1716736, 2024-03-14T10:04:55.561 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 123456 2024-03-14T10:04:55.561 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.561 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-2": { 2024-03-14T10:04:55.561 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 750, 2024-03-14T10:04:55.561 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 907, 2024-03-14T10:04:55.561 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 253296, 2024-03-14T10:04:55.561 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 660760, 2024-03-14T10:04:55.561 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 6, 2024-03-14T10:04:55.561 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 4, 2024-03-14T10:04:55.561 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.089882870, 2024-03-14T10:04:55.562 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.028169806, 2024-03-14T10:04:55.562 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.023558860, 2024-03-14T10:04:55.562 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.027847003, 2024-03-14T10:04:55.562 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:55.562 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 905, 2024-03-14T10:04:55.562 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.059887210, 2024-03-14T10:04:55.562 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000066173 2024-03-14T10:04:55.562 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.562 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:55.562 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.562 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.562 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.562 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.563 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-14T10:04:55.563 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 429376 2024-03-14T10:04:55.563 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.563 INFO:teuthology.orchestra.run.smithi152.stdout: "cct": { 2024-03-14T10:04:55.563 INFO:teuthology.orchestra.run.smithi152.stdout: "total_workers": 1, 2024-03-14T10:04:55.563 INFO:teuthology.orchestra.run.smithi152.stdout: "unhealthy_workers": 0 2024-03-14T10:04:55.563 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.563 INFO:teuthology.orchestra.run.smithi152.stdout: "finisher-MDSRank": { 2024-03-14T10:04:55.563 INFO:teuthology.orchestra.run.smithi152.stdout: "queue_len": 0, 2024-03-14T10:04:55.563 INFO:teuthology.orchestra.run.smithi152.stdout: "complete_latency": { 2024-03-14T10:04:55.563 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 3550, 2024-03-14T10:04:55.563 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.101751471, 2024-03-14T10:04:55.563 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000028662 2024-03-14T10:04:55.563 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:55.564 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.564 INFO:teuthology.orchestra.run.smithi152.stdout: "finisher-PurgeQueue": { 2024-03-14T10:04:55.564 INFO:teuthology.orchestra.run.smithi152.stdout: "queue_len": 0, 2024-03-14T10:04:55.564 INFO:teuthology.orchestra.run.smithi152.stdout: "complete_latency": { 2024-03-14T10:04:55.564 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.564 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.564 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.564 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:55.564 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.564 INFO:teuthology.orchestra.run.smithi152.stdout: "mds": { 2024-03-14T10:04:55.564 INFO:teuthology.orchestra.run.smithi152.stdout: "request": 0, 2024-03-14T10:04:55.564 INFO:teuthology.orchestra.run.smithi152.stdout: "reply": 0, 2024-03-14T10:04:55.564 INFO:teuthology.orchestra.run.smithi152.stdout: "reply_latency": { 2024-03-14T10:04:55.564 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.565 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.565 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.565 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.565 INFO:teuthology.orchestra.run.smithi152.stdout: "slow_reply": 0, 2024-03-14T10:04:55.565 INFO:teuthology.orchestra.run.smithi152.stdout: "forward": 0, 2024-03-14T10:04:55.565 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_fetch_complete": 0, 2024-03-14T10:04:55.565 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_fetch_keys": 0, 2024-03-14T10:04:55.565 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_commit": 0, 2024-03-14T10:04:55.565 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_split": 0, 2024-03-14T10:04:55.565 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_merge": 0, 2024-03-14T10:04:55.565 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes": 0, 2024-03-14T10:04:55.565 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_top": 0, 2024-03-14T10:04:55.565 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_bottom": 0, 2024-03-14T10:04:55.565 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_pin_tail": 0, 2024-03-14T10:04:55.566 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_pinned": 0, 2024-03-14T10:04:55.566 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_expired": 0, 2024-03-14T10:04:55.566 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_with_caps": 0, 2024-03-14T10:04:55.566 INFO:teuthology.orchestra.run.smithi152.stdout: "caps": 0, 2024-03-14T10:04:55.566 INFO:teuthology.orchestra.run.smithi152.stdout: "subtrees": 1, 2024-03-14T10:04:55.566 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse": 0, 2024-03-14T10:04:55.566 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_hit": 0, 2024-03-14T10:04:55.566 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_forward": 0, 2024-03-14T10:04:55.566 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_discover": 0, 2024-03-14T10:04:55.566 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_dir_fetch": 0, 2024-03-14T10:04:55.566 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_remote_ino": 0, 2024-03-14T10:04:55.566 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_lock": 0, 2024-03-14T10:04:55.567 INFO:teuthology.orchestra.run.smithi152.stdout: "load_cent": 0, 2024-03-14T10:04:55.567 INFO:teuthology.orchestra.run.smithi152.stdout: "q": 0, 2024-03-14T10:04:55.567 INFO:teuthology.orchestra.run.smithi152.stdout: "exported": 0, 2024-03-14T10:04:55.567 INFO:teuthology.orchestra.run.smithi152.stdout: "exported_inodes": 0, 2024-03-14T10:04:55.567 INFO:teuthology.orchestra.run.smithi152.stdout: "imported": 0, 2024-03-14T10:04:55.567 INFO:teuthology.orchestra.run.smithi152.stdout: "imported_inodes": 0, 2024-03-14T10:04:55.567 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_dir_fetch": 0, 2024-03-14T10:04:55.567 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_backtrace_fetch": 0, 2024-03-14T10:04:55.567 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_peer_discover": 0, 2024-03-14T10:04:55.567 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rfiles": 0, 2024-03-14T10:04:55.567 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rbytes": 0, 2024-03-14T10:04:55.567 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rsnaps": 0, 2024-03-14T10:04:55.567 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_backtrace_fetch": 0, 2024-03-14T10:04:55.567 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_set_tag": 0, 2024-03-14T10:04:55.567 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_backtrace_repaired": 0, 2024-03-14T10:04:55.568 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_inotable_repaired": 0, 2024-03-14T10:04:55.568 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dir_inodes": 0, 2024-03-14T10:04:55.568 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dir_base_inodes": 0, 2024-03-14T10:04:55.568 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dirfrag_rstats": 0, 2024-03-14T10:04:55.568 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_file_inodes": 0, 2024-03-14T10:04:55.568 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_inode_file_caps": 0, 2024-03-14T10:04:55.568 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_revoke": 0, 2024-03-14T10:04:55.568 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_grant": 0, 2024-03-14T10:04:55.568 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_trunc": 0, 2024-03-14T10:04:55.568 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-14T10:04:55.568 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-14T10:04:55.568 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_caps": 0, 2024-03-14T10:04:55.568 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_caps_dirty": 0, 2024-03-14T10:04:55.568 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_cap_release": 0, 2024-03-14T10:04:55.569 INFO:teuthology.orchestra.run.smithi152.stdout: "process_request_cap_release": 0 2024-03-14T10:04:55.569 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.569 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_cache": { 2024-03-14T10:04:55.569 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_update": 0, 2024-03-14T10:04:55.569 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_update_receipt": 0, 2024-03-14T10:04:55.569 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_try_discover": 0, 2024-03-14T10:04:55.569 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_send_discover": 0, 2024-03-14T10:04:55.569 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_handle_discover": 0, 2024-03-14T10:04:55.569 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays": 0, 2024-03-14T10:04:55.569 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays_delayed": 0, 2024-03-14T10:04:55.569 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays_enqueuing": 0, 2024-03-14T10:04:55.569 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_created": 0, 2024-03-14T10:04:55.569 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_enqueued": 0, 2024-03-14T10:04:55.569 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_reintegrated": 0, 2024-03-14T10:04:55.570 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_migrated": 0, 2024-03-14T10:04:55.570 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_processing": 0, 2024-03-14T10:04:55.570 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_enqueued": 0, 2024-03-14T10:04:55.570 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_prioritized": 0, 2024-03-14T10:04:55.570 INFO:teuthology.orchestra.run.smithi152.stdout: "recovery_started": 0, 2024-03-14T10:04:55.570 INFO:teuthology.orchestra.run.smithi152.stdout: "recovery_completed": 0, 2024-03-14T10:04:55.570 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_enqueue_scrub": 0, 2024-03-14T10:04:55.570 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_exportdir": 0, 2024-03-14T10:04:55.570 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_flush": 0, 2024-03-14T10:04:55.570 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_fragmentdir": 0, 2024-03-14T10:04:55.570 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_fragstats": 0, 2024-03-14T10:04:55.570 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_inodestats": 0 2024-03-14T10:04:55.570 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.570 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_log": { 2024-03-14T10:04:55.571 INFO:teuthology.orchestra.run.smithi152.stdout: "evadd": 0, 2024-03-14T10:04:55.571 INFO:teuthology.orchestra.run.smithi152.stdout: "evex": 0, 2024-03-14T10:04:55.571 INFO:teuthology.orchestra.run.smithi152.stdout: "evtrm": 0, 2024-03-14T10:04:55.571 INFO:teuthology.orchestra.run.smithi152.stdout: "ev": 11, 2024-03-14T10:04:55.571 INFO:teuthology.orchestra.run.smithi152.stdout: "evexg": 0, 2024-03-14T10:04:55.571 INFO:teuthology.orchestra.run.smithi152.stdout: "evexd": 0, 2024-03-14T10:04:55.571 INFO:teuthology.orchestra.run.smithi152.stdout: "segadd": 0, 2024-03-14T10:04:55.571 INFO:teuthology.orchestra.run.smithi152.stdout: "segex": 0, 2024-03-14T10:04:55.571 INFO:teuthology.orchestra.run.smithi152.stdout: "segtrm": 0, 2024-03-14T10:04:55.571 INFO:teuthology.orchestra.run.smithi152.stdout: "seg": 1, 2024-03-14T10:04:55.571 INFO:teuthology.orchestra.run.smithi152.stdout: "segexg": 0, 2024-03-14T10:04:55.571 INFO:teuthology.orchestra.run.smithi152.stdout: "segexd": 0, 2024-03-14T10:04:55.571 INFO:teuthology.orchestra.run.smithi152.stdout: "expos": 4194304, 2024-03-14T10:04:55.571 INFO:teuthology.orchestra.run.smithi152.stdout: "wrpos": 4197678, 2024-03-14T10:04:55.571 INFO:teuthology.orchestra.run.smithi152.stdout: "rdpos": 4197537, 2024-03-14T10:04:55.572 INFO:teuthology.orchestra.run.smithi152.stdout: "jlat": { 2024-03-14T10:04:55.572 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.572 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.572 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.572 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.572 INFO:teuthology.orchestra.run.smithi152.stdout: "replayed": 11 2024-03-14T10:04:55.572 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.572 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_mem": { 2024-03-14T10:04:55.572 INFO:teuthology.orchestra.run.smithi152.stdout: "ino": 3, 2024-03-14T10:04:55.572 INFO:teuthology.orchestra.run.smithi152.stdout: "ino+": 3, 2024-03-14T10:04:55.572 INFO:teuthology.orchestra.run.smithi152.stdout: "ino-": 0, 2024-03-14T10:04:55.572 INFO:teuthology.orchestra.run.smithi152.stdout: "dir": 1, 2024-03-14T10:04:55.572 INFO:teuthology.orchestra.run.smithi152.stdout: "dir+": 1, 2024-03-14T10:04:55.572 INFO:teuthology.orchestra.run.smithi152.stdout: "dir-": 0, 2024-03-14T10:04:55.573 INFO:teuthology.orchestra.run.smithi152.stdout: "dn": 0, 2024-03-14T10:04:55.573 INFO:teuthology.orchestra.run.smithi152.stdout: "dn+": 0, 2024-03-14T10:04:55.573 INFO:teuthology.orchestra.run.smithi152.stdout: "dn-": 0, 2024-03-14T10:04:55.573 INFO:teuthology.orchestra.run.smithi152.stdout: "cap": 0, 2024-03-14T10:04:55.573 INFO:teuthology.orchestra.run.smithi152.stdout: "cap+": 0, 2024-03-14T10:04:55.573 INFO:teuthology.orchestra.run.smithi152.stdout: "cap-": 0, 2024-03-14T10:04:55.573 INFO:teuthology.orchestra.run.smithi152.stdout: "rss": 50600, 2024-03-14T10:04:55.573 INFO:teuthology.orchestra.run.smithi152.stdout: "heap": 207156 2024-03-14T10:04:55.573 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.573 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_server": { 2024-03-14T10:04:55.573 INFO:teuthology.orchestra.run.smithi152.stdout: "dispatch_client_request": 0, 2024-03-14T10:04:55.573 INFO:teuthology.orchestra.run.smithi152.stdout: "dispatch_server_request": 0, 2024-03-14T10:04:55.573 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_request": 0, 2024-03-14T10:04:55.573 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_session": 0, 2024-03-14T10:04:55.573 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_peer_request": 0, 2024-03-14T10:04:55.574 INFO:teuthology.orchestra.run.smithi152.stdout: "req_create_latency": { 2024-03-14T10:04:55.574 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.574 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.574 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.574 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.574 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getattr_latency": { 2024-03-14T10:04:55.574 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.574 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.574 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.574 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.574 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getfilelock_latency": { 2024-03-14T10:04:55.574 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.574 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.575 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.575 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.575 INFO:teuthology.orchestra.run.smithi152.stdout: "req_link_latency": { 2024-03-14T10:04:55.575 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.575 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.575 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.575 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.575 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookup_latency": { 2024-03-14T10:04:55.575 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.575 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.575 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.575 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.575 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookuphash_latency": { 2024-03-14T10:04:55.575 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.576 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.576 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.576 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.576 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupino_latency": { 2024-03-14T10:04:55.576 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.576 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.576 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.576 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.576 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupname_latency": { 2024-03-14T10:04:55.576 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.576 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.576 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.576 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.577 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupparent_latency": { 2024-03-14T10:04:55.577 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.577 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.577 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.577 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.577 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupsnap_latency": { 2024-03-14T10:04:55.577 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.577 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.577 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.577 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.577 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lssnap_latency": { 2024-03-14T10:04:55.577 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.577 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.577 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.577 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.578 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mkdir_latency": { 2024-03-14T10:04:55.578 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.578 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.578 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.578 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.578 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mknod_latency": { 2024-03-14T10:04:55.578 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.578 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.578 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.578 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.578 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mksnap_latency": { 2024-03-14T10:04:55.578 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.578 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.579 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.579 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.579 INFO:teuthology.orchestra.run.smithi152.stdout: "req_open_latency": { 2024-03-14T10:04:55.579 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.579 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.579 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.579 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.579 INFO:teuthology.orchestra.run.smithi152.stdout: "req_readdir_latency": { 2024-03-14T10:04:55.579 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.579 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.579 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.579 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.579 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rename_latency": { 2024-03-14T10:04:55.579 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.580 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.580 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.580 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.580 INFO:teuthology.orchestra.run.smithi152.stdout: "req_renamesnap_latency": { 2024-03-14T10:04:55.580 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.580 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.580 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.580 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.580 INFO:teuthology.orchestra.run.smithi152.stdout: "req_snapdiff_latency": { 2024-03-14T10:04:55.580 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.580 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.580 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.580 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.580 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmdir_latency": { 2024-03-14T10:04:55.580 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.580 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.581 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.581 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.581 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmsnap_latency": { 2024-03-14T10:04:55.581 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.581 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.581 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.581 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.581 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmxattr_latency": { 2024-03-14T10:04:55.581 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.581 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.581 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.581 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.581 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setattr_latency": { 2024-03-14T10:04:55.581 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.581 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.581 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.582 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.582 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setdirlayout_latency": { 2024-03-14T10:04:55.582 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.582 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.582 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.582 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.582 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setfilelock_latency": { 2024-03-14T10:04:55.582 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.582 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.582 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.582 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.582 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setlayout_latency": { 2024-03-14T10:04:55.582 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.583 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.583 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.583 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.583 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setxattr_latency": { 2024-03-14T10:04:55.583 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.583 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.583 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.583 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.583 INFO:teuthology.orchestra.run.smithi152.stdout: "req_symlink_latency": { 2024-03-14T10:04:55.583 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.583 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.583 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.583 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.583 INFO:teuthology.orchestra.run.smithi152.stdout: "req_unlink_latency": { 2024-03-14T10:04:55.583 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.583 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.584 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.584 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.584 INFO:teuthology.orchestra.run.smithi152.stdout: "cap_revoke_eviction": 0, 2024-03-14T10:04:55.584 INFO:teuthology.orchestra.run.smithi152.stdout: "cap_acquisition_throttle": 0, 2024-03-14T10:04:55.584 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getvxattr_latency": { 2024-03-14T10:04:55.584 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.584 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.584 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.584 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:55.584 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.584 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_sessions": { 2024-03-14T10:04:55.584 INFO:teuthology.orchestra.run.smithi152.stdout: "session_count": 2, 2024-03-14T10:04:55.584 INFO:teuthology.orchestra.run.smithi152.stdout: "session_add": 6, 2024-03-14T10:04:55.584 INFO:teuthology.orchestra.run.smithi152.stdout: "session_remove": 4, 2024-03-14T10:04:55.584 INFO:teuthology.orchestra.run.smithi152.stdout: "sessions_open": 4, 2024-03-14T10:04:55.585 INFO:teuthology.orchestra.run.smithi152.stdout: "sessions_stale": 0, 2024-03-14T10:04:55.585 INFO:teuthology.orchestra.run.smithi152.stdout: "total_load": 0, 2024-03-14T10:04:55.585 INFO:teuthology.orchestra.run.smithi152.stdout: "average_load": 0, 2024-03-14T10:04:55.585 INFO:teuthology.orchestra.run.smithi152.stdout: "avg_session_uptime": 7679, 2024-03-14T10:04:55.585 INFO:teuthology.orchestra.run.smithi152.stdout: "mdthresh_evicted": 0 2024-03-14T10:04:55.585 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.585 INFO:teuthology.orchestra.run.smithi152.stdout: "mempool": { 2024-03-14T10:04:55.585 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_bytes": 0, 2024-03-14T10:04:55.585 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_items": 0, 2024-03-14T10:04:55.585 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_bytes": 0, 2024-03-14T10:04:55.585 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_items": 0, 2024-03-14T10:04:55.585 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_bytes": 0, 2024-03-14T10:04:55.585 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_items": 0, 2024-03-14T10:04:55.585 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-14T10:04:55.586 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_items": 0, 2024-03-14T10:04:55.586 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-14T10:04:55.586 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_items": 0, 2024-03-14T10:04:55.586 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_bytes": 0, 2024-03-14T10:04:55.586 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_items": 0, 2024-03-14T10:04:55.586 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-14T10:04:55.586 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_items": 0, 2024-03-14T10:04:55.586 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_bytes": 0, 2024-03-14T10:04:55.586 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_items": 0, 2024-03-14T10:04:55.586 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_bytes": 0, 2024-03-14T10:04:55.586 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_items": 0, 2024-03-14T10:04:55.586 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-14T10:04:55.586 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_items": 0, 2024-03-14T10:04:55.587 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-14T10:04:55.587 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_items": 0, 2024-03-14T10:04:55.587 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_bytes": 0, 2024-03-14T10:04:55.587 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_items": 0, 2024-03-14T10:04:55.587 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_bytes": 0, 2024-03-14T10:04:55.587 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_items": 0, 2024-03-14T10:04:55.587 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-14T10:04:55.587 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_items": 0, 2024-03-14T10:04:55.587 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_bytes": 0, 2024-03-14T10:04:55.587 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_items": 0, 2024-03-14T10:04:55.587 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_bytes": 0, 2024-03-14T10:04:55.587 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_items": 0, 2024-03-14T10:04:55.587 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_bytes": 0, 2024-03-14T10:04:55.587 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_items": 0, 2024-03-14T10:04:55.587 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_bytes": 0, 2024-03-14T10:04:55.588 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_items": 0, 2024-03-14T10:04:55.588 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_bytes": 189433, 2024-03-14T10:04:55.588 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_items": 52, 2024-03-14T10:04:55.588 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_bytes": 0, 2024-03-14T10:04:55.588 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_items": 0, 2024-03-14T10:04:55.588 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_bytes": 0, 2024-03-14T10:04:55.588 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_items": 0, 2024-03-14T10:04:55.588 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_bytes": 0, 2024-03-14T10:04:55.588 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_items": 0, 2024-03-14T10:04:55.588 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_bytes": 0, 2024-03-14T10:04:55.588 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_items": 0, 2024-03-14T10:04:55.588 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_bytes": 9832, 2024-03-14T10:04:55.588 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_items": 205, 2024-03-14T10:04:55.588 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_bytes": 0, 2024-03-14T10:04:55.588 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_items": 0, 2024-03-14T10:04:55.589 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_bytes": 0, 2024-03-14T10:04:55.589 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_items": 0, 2024-03-14T10:04:55.589 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_bytes": 8320, 2024-03-14T10:04:55.589 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_items": 13, 2024-03-14T10:04:55.589 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_bytes": 0, 2024-03-14T10:04:55.589 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_items": 0, 2024-03-14T10:04:55.589 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_bytes": 0, 2024-03-14T10:04:55.589 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_items": 0 2024-03-14T10:04:55.589 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.589 INFO:teuthology.orchestra.run.smithi152.stdout: "objecter": { 2024-03-14T10:04:55.589 INFO:teuthology.orchestra.run.smithi152.stdout: "op_active": 0, 2024-03-14T10:04:55.589 INFO:teuthology.orchestra.run.smithi152.stdout: "op_laggy": 0, 2024-03-14T10:04:55.589 INFO:teuthology.orchestra.run.smithi152.stdout: "op_send": 2174, 2024-03-14T10:04:55.589 INFO:teuthology.orchestra.run.smithi152.stdout: "op_send_bytes": 21, 2024-03-14T10:04:55.590 INFO:teuthology.orchestra.run.smithi152.stdout: "op_resend": 3, 2024-03-14T10:04:55.590 INFO:teuthology.orchestra.run.smithi152.stdout: "op_reply": 2174, 2024-03-14T10:04:55.590 INFO:teuthology.orchestra.run.smithi152.stdout: "op_latency": { 2024-03-14T10:04:55.590 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 2174, 2024-03-14T10:04:55.590 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 1.336983072, 2024-03-14T10:04:55.590 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000614987 2024-03-14T10:04:55.590 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.590 INFO:teuthology.orchestra.run.smithi152.stdout: "op_inflight": 0, 2024-03-14T10:04:55.590 INFO:teuthology.orchestra.run.smithi152.stdout: "oplen_avg": { 2024-03-14T10:04:55.590 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 2174, 2024-03-14T10:04:55.590 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 2175 2024-03-14T10:04:55.590 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.590 INFO:teuthology.orchestra.run.smithi152.stdout: "op": 2174, 2024-03-14T10:04:55.590 INFO:teuthology.orchestra.run.smithi152.stdout: "op_r": 2174, 2024-03-14T10:04:55.590 INFO:teuthology.orchestra.run.smithi152.stdout: "op_w": 0, 2024-03-14T10:04:55.591 INFO:teuthology.orchestra.run.smithi152.stdout: "op_rmw": 0, 2024-03-14T10:04:55.591 INFO:teuthology.orchestra.run.smithi152.stdout: "op_pg": 0, 2024-03-14T10:04:55.591 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_stat": 1417, 2024-03-14T10:04:55.591 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_create": 0, 2024-03-14T10:04:55.591 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_read": 755, 2024-03-14T10:04:55.591 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_write": 0, 2024-03-14T10:04:55.591 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_writefull": 0, 2024-03-14T10:04:55.591 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_writesame": 0, 2024-03-14T10:04:55.591 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_append": 0, 2024-03-14T10:04:55.591 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_zero": 0, 2024-03-14T10:04:55.591 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_truncate": 0, 2024-03-14T10:04:55.591 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_delete": 0, 2024-03-14T10:04:55.591 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_mapext": 0, 2024-03-14T10:04:55.591 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_sparse_read": 0, 2024-03-14T10:04:55.591 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_clonerange": 0, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_getxattr": 1, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_setxattr": 0, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_cmpxattr": 0, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_rmxattr": 0, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_resetxattrs": 0, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_call": 0, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_watch": 0, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_notify": 0, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_src_cmpxattr": 0, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_pgls": 0, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_pgls_filter": 0, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_other": 0, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_active": 0, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_send": 0, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_resend": 0, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_ping": 0, 2024-03-14T10:04:55.592 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_active": 0, 2024-03-14T10:04:55.593 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_send": 0, 2024-03-14T10:04:55.593 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_resend": 0, 2024-03-14T10:04:55.593 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_active": 0, 2024-03-14T10:04:55.593 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_send": 0, 2024-03-14T10:04:55.593 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_resend": 0, 2024-03-14T10:04:55.593 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_active": 0, 2024-03-14T10:04:55.593 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_send": 0, 2024-03-14T10:04:55.593 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_resend": 0, 2024-03-14T10:04:55.593 INFO:teuthology.orchestra.run.smithi152.stdout: "command_active": 0, 2024-03-14T10:04:55.593 INFO:teuthology.orchestra.run.smithi152.stdout: "command_send": 0, 2024-03-14T10:04:55.593 INFO:teuthology.orchestra.run.smithi152.stdout: "command_resend": 0, 2024-03-14T10:04:55.593 INFO:teuthology.orchestra.run.smithi152.stdout: "map_epoch": 96, 2024-03-14T10:04:55.593 INFO:teuthology.orchestra.run.smithi152.stdout: "map_full": 0, 2024-03-14T10:04:55.593 INFO:teuthology.orchestra.run.smithi152.stdout: "map_inc": 13, 2024-03-14T10:04:55.593 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_sessions": 5, 2024-03-14T10:04:55.593 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_session_open": 5, 2024-03-14T10:04:55.594 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_session_close": 0, 2024-03-14T10:04:55.594 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_laggy": 0, 2024-03-14T10:04:55.594 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_wr": 0, 2024-03-14T10:04:55.594 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_rd": 2, 2024-03-14T10:04:55.594 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_del": 0 2024-03-14T10:04:55.594 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.594 INFO:teuthology.orchestra.run.smithi152.stdout: "oft": { 2024-03-14T10:04:55.594 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_objs": 0, 2024-03-14T10:04:55.594 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_kv_pairs": 0, 2024-03-14T10:04:55.594 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_updates": 0, 2024-03-14T10:04:55.594 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_removes": 0 2024-03-14T10:04:55.594 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.594 INFO:teuthology.orchestra.run.smithi152.stdout: "osd-slow-ops": { 2024-03-14T10:04:55.594 INFO:teuthology.orchestra.run.smithi152.stdout: "slow_ops_count": 0 2024-03-14T10:04:55.595 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.595 INFO:teuthology.orchestra.run.smithi152.stdout: "purge_queue": { 2024-03-14T10:04:55.595 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_ops": 0, 2024-03-14T10:04:55.595 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_ops_high_water": 0, 2024-03-14T10:04:55.595 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing": 0, 2024-03-14T10:04:55.595 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_high_water": 0, 2024-03-14T10:04:55.595 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executed": 0, 2024-03-14T10:04:55.595 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_item_in_journal": 0 2024-03-14T10:04:55.595 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.595 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-14T10:04:55.595 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-14T10:04:55.595 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-03-14T10:04:55.595 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-14T10:04:55.595 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 2418, 2024-03-14T10:04:55.595 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 2104819, 2024-03-14T10:04:55.595 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:55.596 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 2418, 2024-03-14T10:04:55.596 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-14T10:04:55.596 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-14T10:04:55.596 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 2418, 2024-03-14T10:04:55.596 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 2104819, 2024-03-14T10:04:55.596 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-14T10:04:55.596 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.596 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.596 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.596 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:55.596 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.596 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-objecter_bytes": { 2024-03-14T10:04:55.596 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-14T10:04:55.596 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-03-14T10:04:55.596 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-14T10:04:55.596 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-03-14T10:04:55.597 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-03-14T10:04:55.597 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:55.597 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:55.597 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 2174, 2024-03-14T10:04:55.597 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 3379, 2024-03-14T10:04:55.597 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 2174, 2024-03-14T10:04:55.597 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 3379, 2024-03-14T10:04:55.597 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-14T10:04:55.597 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.597 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.597 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.597 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:55.597 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.597 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-objecter_ops": { 2024-03-14T10:04:55.597 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-14T10:04:55.597 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 1024, 2024-03-14T10:04:55.598 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-14T10:04:55.598 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-03-14T10:04:55.598 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-03-14T10:04:55.598 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:55.598 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:55.598 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 2174, 2024-03-14T10:04:55.598 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 2174, 2024-03-14T10:04:55.598 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 2174, 2024-03-14T10:04:55.598 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 2174, 2024-03-14T10:04:55.598 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-14T10:04:55.598 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.598 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.598 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.599 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:55.599 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.599 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-write_buf_throttle": { 2024-03-14T10:04:55.599 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-14T10:04:55.599 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 3758096384, 2024-03-14T10:04:55.599 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-14T10:04:55.599 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-03-14T10:04:55.599 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-03-14T10:04:55.599 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:55.599 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:55.599 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-14T10:04:55.599 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-14T10:04:55.599 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 0, 2024-03-14T10:04:55.599 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 0, 2024-03-14T10:04:55.599 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-14T10:04:55.600 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.600 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.600 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.600 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:55.600 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:55.600 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-write_buf_throttle-0x556c491505a0": { 2024-03-14T10:04:55.600 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-14T10:04:55.600 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 3758096384, 2024-03-14T10:04:55.600 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-14T10:04:55.600 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-03-14T10:04:55.600 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-03-14T10:04:55.600 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:55.600 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:55.600 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-14T10:04:55.600 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-14T10:04:55.600 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 0, 2024-03-14T10:04:55.601 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 0, 2024-03-14T10:04:55.601 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-14T10:04:55.601 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:55.601 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:55.601 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:55.601 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:55.601 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:55.601 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-03-14T10:04:55.928 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-14T10:04:55.928 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-14T10:04:55.928 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-03-14T10:04:55.928 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-14T10:04:55.928 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-14T10:04:55.928 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-03-14T10:04:55.928 DEBUG:tasks.check_counter:Getting stats from f 2024-03-14T10:04:55.928 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-14T10:04:56.151 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.c/config 2024-03-14T10:04:57.206 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-03-14T10:04:57.206 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-0": { 2024-03-14T10:04:57.206 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 4, 2024-03-14T10:04:57.206 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 4, 2024-03-14T10:04:57.207 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 2992, 2024-03-14T10:04:57.207 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 448, 2024-03-14T10:04:57.207 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 4, 2024-03-14T10:04:57.207 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 0, 2024-03-14T10:04:57.207 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.001467124, 2024-03-14T10:04:57.207 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.000037911, 2024-03-14T10:04:57.207 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.000581708, 2024-03-14T10:04:57.207 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-14T10:04:57.207 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:57.207 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 3, 2024-03-14T10:04:57.207 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000681690, 2024-03-14T10:04:57.207 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000227230 2024-03-14T10:04:57.207 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:57.207 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:57.208 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:57.208 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:57.208 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:57.208 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:57.208 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-14T10:04:57.208 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 448 2024-03-14T10:04:57.208 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:57.208 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-1": { 2024-03-14T10:04:57.208 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 2, 2024-03-14T10:04:57.208 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 162, 2024-03-14T10:04:57.208 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 320, 2024-03-14T10:04:57.208 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 180720, 2024-03-14T10:04:57.208 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 5, 2024-03-14T10:04:57.208 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 3, 2024-03-14T10:04:57.209 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.010115688, 2024-03-14T10:04:57.209 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.006526367, 2024-03-14T10:04:57.209 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.001280906, 2024-03-14T10:04:57.209 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-14T10:04:57.209 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:57.209 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 159, 2024-03-14T10:04:57.209 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.020743955, 2024-03-14T10:04:57.209 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000130465 2024-03-14T10:04:57.209 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:57.209 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:57.209 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:57.209 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:57.209 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:57.209 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:57.209 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-14T10:04:57.209 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 180720 2024-03-14T10:04:57.210 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:57.210 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-2": { 2024-03-14T10:04:57.210 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 224, 2024-03-14T10:04:57.210 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 212, 2024-03-14T10:04:57.210 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 1689696, 2024-03-14T10:04:57.210 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 120928, 2024-03-14T10:04:57.210 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 2, 2024-03-14T10:04:57.210 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 1, 2024-03-14T10:04:57.210 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.042378197, 2024-03-14T10:04:57.210 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.011230632, 2024-03-14T10:04:57.210 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.036391706, 2024-03-14T10:04:57.210 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.004968696, 2024-03-14T10:04:57.210 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:57.210 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 211, 2024-03-14T10:04:57.211 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.010494909, 2024-03-14T10:04:57.211 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000049738 2024-03-14T10:04:57.211 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:57.211 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:57.211 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:57.211 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:57.211 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:57.211 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:57.211 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 1689696, 2024-03-14T10:04:57.211 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 120928 2024-03-14T10:04:57.211 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:57.211 INFO:teuthology.orchestra.run.smithi152.stdout: "cct": { 2024-03-14T10:04:57.211 INFO:teuthology.orchestra.run.smithi152.stdout: "total_workers": 0, 2024-03-14T10:04:57.211 INFO:teuthology.orchestra.run.smithi152.stdout: "unhealthy_workers": 0 2024-03-14T10:04:57.212 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:57.212 INFO:teuthology.orchestra.run.smithi152.stdout: "mempool": { 2024-03-14T10:04:57.212 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_bytes": 0, 2024-03-14T10:04:57.212 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_items": 0, 2024-03-14T10:04:57.212 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_bytes": 0, 2024-03-14T10:04:57.212 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_items": 0, 2024-03-14T10:04:57.212 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_bytes": 0, 2024-03-14T10:04:57.212 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_items": 0, 2024-03-14T10:04:57.212 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-14T10:04:57.212 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_items": 0, 2024-03-14T10:04:57.212 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-14T10:04:57.212 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_items": 0, 2024-03-14T10:04:57.212 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_bytes": 0, 2024-03-14T10:04:57.213 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_items": 0, 2024-03-14T10:04:57.213 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-14T10:04:57.213 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_items": 0, 2024-03-14T10:04:57.213 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_bytes": 0, 2024-03-14T10:04:57.213 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_items": 0, 2024-03-14T10:04:57.213 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_bytes": 0, 2024-03-14T10:04:57.213 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_items": 0, 2024-03-14T10:04:57.213 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-14T10:04:57.213 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_items": 0, 2024-03-14T10:04:57.213 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-14T10:04:57.213 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_items": 0, 2024-03-14T10:04:57.213 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_bytes": 0, 2024-03-14T10:04:57.213 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_items": 0, 2024-03-14T10:04:57.213 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_bytes": 0, 2024-03-14T10:04:57.213 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_items": 0, 2024-03-14T10:04:57.213 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-14T10:04:57.214 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_items": 0, 2024-03-14T10:04:57.214 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_bytes": 0, 2024-03-14T10:04:57.214 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_items": 0, 2024-03-14T10:04:57.214 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_bytes": 0, 2024-03-14T10:04:57.214 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_items": 0, 2024-03-14T10:04:57.214 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_bytes": 0, 2024-03-14T10:04:57.214 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_items": 0, 2024-03-14T10:04:57.214 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_bytes": 0, 2024-03-14T10:04:57.214 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_items": 0, 2024-03-14T10:04:57.214 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_bytes": 148977, 2024-03-14T10:04:57.214 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_items": 30, 2024-03-14T10:04:57.214 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_bytes": 0, 2024-03-14T10:04:57.214 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_items": 0, 2024-03-14T10:04:57.215 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_bytes": 0, 2024-03-14T10:04:57.215 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_items": 0, 2024-03-14T10:04:57.215 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_bytes": 0, 2024-03-14T10:04:57.215 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_items": 0, 2024-03-14T10:04:57.215 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_bytes": 0, 2024-03-14T10:04:57.215 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_items": 0, 2024-03-14T10:04:57.215 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_bytes": 0, 2024-03-14T10:04:57.215 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_items": 0, 2024-03-14T10:04:57.215 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_bytes": 0, 2024-03-14T10:04:57.215 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_items": 0, 2024-03-14T10:04:57.215 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_bytes": 0, 2024-03-14T10:04:57.215 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_items": 0, 2024-03-14T10:04:57.215 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_bytes": 928, 2024-03-14T10:04:57.215 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_items": 2, 2024-03-14T10:04:57.215 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_bytes": 0, 2024-03-14T10:04:57.215 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_items": 0, 2024-03-14T10:04:57.216 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_bytes": 0, 2024-03-14T10:04:57.216 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_items": 0 2024-03-14T10:04:57.216 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:57.216 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-14T10:04:57.216 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-14T10:04:57.216 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-03-14T10:04:57.216 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-14T10:04:57.216 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 226, 2024-03-14T10:04:57.216 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 1658160, 2024-03-14T10:04:57.216 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:57.216 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 226, 2024-03-14T10:04:57.216 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-14T10:04:57.216 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-14T10:04:57.216 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 226, 2024-03-14T10:04:57.216 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 1658160, 2024-03-14T10:04:57.217 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-14T10:04:57.217 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:57.217 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:57.217 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:57.217 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:57.217 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:57.217 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-03-14T10:04:57.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:57 smithi152 ceph-mon[48279]: pgmap v334: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:57.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:57 smithi132 ceph-mon[47959]: pgmap v334: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:57.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:57 smithi078 ceph-mon[38143]: pgmap v334: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-14T10:04:57.577 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.f 2024-03-14T10:04:57.577 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.f 2024-03-14T10:04:57.577 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.f 2024-03-14T10:04:57.577 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.f 2024-03-14T10:04:57.577 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.f 2024-03-14T10:04:57.577 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.f 2024-03-14T10:04:57.577 DEBUG:tasks.check_counter:Getting stats from i 2024-03-14T10:04:57.578 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-14T10:04:57.799 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.c/config 2024-03-14T10:04:58.814 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-03-14T10:04:58.814 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-0": { 2024-03-14T10:04:58.815 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 1484, 2024-03-14T10:04:58.815 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 1471, 2024-03-14T10:04:58.815 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 48240905, 2024-03-14T10:04:58.815 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 509571, 2024-03-14T10:04:58.815 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 7, 2024-03-14T10:04:58.815 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 3, 2024-03-14T10:04:58.815 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.182046530, 2024-03-14T10:04:58.815 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.045259792, 2024-03-14T10:04:58.815 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.091110690, 2024-03-14T10:04:58.815 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.043380240, 2024-03-14T10:04:58.815 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:58.815 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 1470, 2024-03-14T10:04:58.815 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.064891263, 2024-03-14T10:04:58.815 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000044143 2024-03-14T10:04:58.815 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:58.816 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.816 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.816 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.816 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 1716896, 2024-03-14T10:04:58.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 123616 2024-03-14T10:04:58.816 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.816 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-1": { 2024-03-14T10:04:58.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 758, 2024-03-14T10:04:58.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 917, 2024-03-14T10:04:58.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 81190077, 2024-03-14T10:04:58.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 665678, 2024-03-14T10:04:58.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 9, 2024-03-14T10:04:58.816 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 7, 2024-03-14T10:04:58.817 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.106818816, 2024-03-14T10:04:58.817 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.025457604, 2024-03-14T10:04:58.817 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.056964754, 2024-03-14T10:04:58.817 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.016812351, 2024-03-14T10:04:58.817 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:58.817 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 914, 2024-03-14T10:04:58.817 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.049959010, 2024-03-14T10:04:58.817 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000054659 2024-03-14T10:04:58.817 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.817 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:58.817 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.817 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.817 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.817 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.817 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-14T10:04:58.818 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 430496 2024-03-14T10:04:58.818 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.818 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-2": { 2024-03-14T10:04:58.818 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 743, 2024-03-14T10:04:58.818 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 743, 2024-03-14T10:04:58.818 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 54715587, 2024-03-14T10:04:58.818 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 230284, 2024-03-14T10:04:58.818 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 6, 2024-03-14T10:04:58.818 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 5, 2024-03-14T10:04:58.818 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.086309273, 2024-03-14T10:04:58.818 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.021147670, 2024-03-14T10:04:58.818 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.037708557, 2024-03-14T10:04:58.818 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.021226642, 2024-03-14T10:04:58.818 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:04:58.819 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 742, 2024-03-14T10:04:58.819 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.028722230, 2024-03-14T10:04:58.819 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000038709 2024-03-14T10:04:58.819 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.819 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:04:58.819 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.819 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.819 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.819 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.819 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-14T10:04:58.819 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 448 2024-03-14T10:04:58.819 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.819 INFO:teuthology.orchestra.run.smithi152.stdout: "cct": { 2024-03-14T10:04:58.819 INFO:teuthology.orchestra.run.smithi152.stdout: "total_workers": 1, 2024-03-14T10:04:58.820 INFO:teuthology.orchestra.run.smithi152.stdout: "unhealthy_workers": 0 2024-03-14T10:04:58.820 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.820 INFO:teuthology.orchestra.run.smithi152.stdout: "finisher-MDSRank": { 2024-03-14T10:04:58.820 INFO:teuthology.orchestra.run.smithi152.stdout: "queue_len": 0, 2024-03-14T10:04:58.820 INFO:teuthology.orchestra.run.smithi152.stdout: "complete_latency": { 2024-03-14T10:04:58.820 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 4499, 2024-03-14T10:04:58.820 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.111932276, 2024-03-14T10:04:58.820 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000024879 2024-03-14T10:04:58.820 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:58.820 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.820 INFO:teuthology.orchestra.run.smithi152.stdout: "finisher-PurgeQueue": { 2024-03-14T10:04:58.820 INFO:teuthology.orchestra.run.smithi152.stdout: "queue_len": 0, 2024-03-14T10:04:58.820 INFO:teuthology.orchestra.run.smithi152.stdout: "complete_latency": { 2024-03-14T10:04:58.820 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.820 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.820 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: "mds": { 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: "request": 0, 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: "reply": 0, 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: "reply_latency": { 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: "slow_reply": 0, 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: "forward": 0, 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_fetch_complete": 0, 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_fetch_keys": 0, 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_commit": 0, 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_split": 0, 2024-03-14T10:04:58.821 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_merge": 0, 2024-03-14T10:04:58.822 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes": 15314, 2024-03-14T10:04:58.822 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_top": 0, 2024-03-14T10:04:58.822 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_bottom": 15314, 2024-03-14T10:04:58.822 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_pin_tail": 0, 2024-03-14T10:04:58.822 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_pinned": 15314, 2024-03-14T10:04:58.822 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_expired": 0, 2024-03-14T10:04:58.822 INFO:teuthology.orchestra.run.smithi152.stdout: "inodes_with_caps": 0, 2024-03-14T10:04:58.822 INFO:teuthology.orchestra.run.smithi152.stdout: "caps": 0, 2024-03-14T10:04:58.822 INFO:teuthology.orchestra.run.smithi152.stdout: "subtrees": 2, 2024-03-14T10:04:58.822 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse": 0, 2024-03-14T10:04:58.822 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_hit": 0, 2024-03-14T10:04:58.822 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_forward": 0, 2024-03-14T10:04:58.822 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_discover": 0, 2024-03-14T10:04:58.822 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_dir_fetch": 0, 2024-03-14T10:04:58.823 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_remote_ino": 0, 2024-03-14T10:04:58.823 INFO:teuthology.orchestra.run.smithi152.stdout: "traverse_lock": 0, 2024-03-14T10:04:58.823 INFO:teuthology.orchestra.run.smithi152.stdout: "load_cent": 0, 2024-03-14T10:04:58.823 INFO:teuthology.orchestra.run.smithi152.stdout: "q": 0, 2024-03-14T10:04:58.823 INFO:teuthology.orchestra.run.smithi152.stdout: "exported": 0, 2024-03-14T10:04:58.823 INFO:teuthology.orchestra.run.smithi152.stdout: "exported_inodes": 0, 2024-03-14T10:04:58.823 INFO:teuthology.orchestra.run.smithi152.stdout: "imported": 0, 2024-03-14T10:04:58.823 INFO:teuthology.orchestra.run.smithi152.stdout: "imported_inodes": 0, 2024-03-14T10:04:58.823 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_dir_fetch": 0, 2024-03-14T10:04:58.823 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_backtrace_fetch": 0, 2024-03-14T10:04:58.823 INFO:teuthology.orchestra.run.smithi152.stdout: "openino_peer_discover": 0, 2024-03-14T10:04:58.823 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rfiles": 4, 2024-03-14T10:04:58.823 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rbytes": 234, 2024-03-14T10:04:58.823 INFO:teuthology.orchestra.run.smithi152.stdout: "root_rsnaps": 6, 2024-03-14T10:04:58.823 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_backtrace_fetch": 0, 2024-03-14T10:04:58.824 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_set_tag": 0, 2024-03-14T10:04:58.824 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_backtrace_repaired": 0, 2024-03-14T10:04:58.824 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_inotable_repaired": 0, 2024-03-14T10:04:58.824 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dir_inodes": 0, 2024-03-14T10:04:58.824 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dir_base_inodes": 0, 2024-03-14T10:04:58.824 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_dirfrag_rstats": 0, 2024-03-14T10:04:58.824 INFO:teuthology.orchestra.run.smithi152.stdout: "scrub_file_inodes": 0, 2024-03-14T10:04:58.824 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_inode_file_caps": 0, 2024-03-14T10:04:58.824 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_revoke": 0, 2024-03-14T10:04:58.824 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_grant": 0, 2024-03-14T10:04:58.824 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_trunc": 0, 2024-03-14T10:04:58.824 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-14T10:04:58.824 INFO:teuthology.orchestra.run.smithi152.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-14T10:04:58.824 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_caps": 0, 2024-03-14T10:04:58.824 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_caps_dirty": 0, 2024-03-14T10:04:58.824 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_cap_release": 0, 2024-03-14T10:04:58.825 INFO:teuthology.orchestra.run.smithi152.stdout: "process_request_cap_release": 0 2024-03-14T10:04:58.825 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.825 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_cache": { 2024-03-14T10:04:58.825 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_update": 0, 2024-03-14T10:04:58.825 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_update_receipt": 0, 2024-03-14T10:04:58.825 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_try_discover": 0, 2024-03-14T10:04:58.825 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_send_discover": 0, 2024-03-14T10:04:58.825 INFO:teuthology.orchestra.run.smithi152.stdout: "dir_handle_discover": 0, 2024-03-14T10:04:58.825 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays": 0, 2024-03-14T10:04:58.825 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays_delayed": 0, 2024-03-14T10:04:58.825 INFO:teuthology.orchestra.run.smithi152.stdout: "num_strays_enqueuing": 0, 2024-03-14T10:04:58.825 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_created": 0, 2024-03-14T10:04:58.825 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_enqueued": 0, 2024-03-14T10:04:58.825 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_reintegrated": 0, 2024-03-14T10:04:58.825 INFO:teuthology.orchestra.run.smithi152.stdout: "strays_migrated": 0, 2024-03-14T10:04:58.826 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_processing": 0, 2024-03-14T10:04:58.826 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_enqueued": 0, 2024-03-14T10:04:58.826 INFO:teuthology.orchestra.run.smithi152.stdout: "num_recovering_prioritized": 0, 2024-03-14T10:04:58.826 INFO:teuthology.orchestra.run.smithi152.stdout: "recovery_started": 0, 2024-03-14T10:04:58.826 INFO:teuthology.orchestra.run.smithi152.stdout: "recovery_completed": 0, 2024-03-14T10:04:58.826 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_enqueue_scrub": 0, 2024-03-14T10:04:58.826 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_exportdir": 0, 2024-03-14T10:04:58.826 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_flush": 0, 2024-03-14T10:04:58.826 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_fragmentdir": 0, 2024-03-14T10:04:58.826 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_fragstats": 0, 2024-03-14T10:04:58.826 INFO:teuthology.orchestra.run.smithi152.stdout: "ireq_inodestats": 0 2024-03-14T10:04:58.826 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.826 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_log": { 2024-03-14T10:04:58.826 INFO:teuthology.orchestra.run.smithi152.stdout: "evadd": 0, 2024-03-14T10:04:58.826 INFO:teuthology.orchestra.run.smithi152.stdout: "evex": 0, 2024-03-14T10:04:58.827 INFO:teuthology.orchestra.run.smithi152.stdout: "evtrm": 0, 2024-03-14T10:04:58.827 INFO:teuthology.orchestra.run.smithi152.stdout: "ev": 63282, 2024-03-14T10:04:58.827 INFO:teuthology.orchestra.run.smithi152.stdout: "evexg": 0, 2024-03-14T10:04:58.827 INFO:teuthology.orchestra.run.smithi152.stdout: "evexd": 0, 2024-03-14T10:04:58.827 INFO:teuthology.orchestra.run.smithi152.stdout: "segadd": 0, 2024-03-14T10:04:58.827 INFO:teuthology.orchestra.run.smithi152.stdout: "segex": 0, 2024-03-14T10:04:58.827 INFO:teuthology.orchestra.run.smithi152.stdout: "segtrm": 0, 2024-03-14T10:04:58.827 INFO:teuthology.orchestra.run.smithi152.stdout: "seg": 82, 2024-03-14T10:04:58.827 INFO:teuthology.orchestra.run.smithi152.stdout: "segexg": 0, 2024-03-14T10:04:58.827 INFO:teuthology.orchestra.run.smithi152.stdout: "segexd": 0, 2024-03-14T10:04:58.827 INFO:teuthology.orchestra.run.smithi152.stdout: "expos": 4194304, 2024-03-14T10:04:58.827 INFO:teuthology.orchestra.run.smithi152.stdout: "wrpos": 185843980, 2024-03-14T10:04:58.827 INFO:teuthology.orchestra.run.smithi152.stdout: "rdpos": 185843844, 2024-03-14T10:04:58.827 INFO:teuthology.orchestra.run.smithi152.stdout: "jlat": { 2024-03-14T10:04:58.827 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.827 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.828 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.828 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.828 INFO:teuthology.orchestra.run.smithi152.stdout: "replayed": 63282 2024-03-14T10:04:58.828 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.828 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_mem": { 2024-03-14T10:04:58.828 INFO:teuthology.orchestra.run.smithi152.stdout: "ino": 3707, 2024-03-14T10:04:58.828 INFO:teuthology.orchestra.run.smithi152.stdout: "ino+": 10486, 2024-03-14T10:04:58.828 INFO:teuthology.orchestra.run.smithi152.stdout: "ino-": 6779, 2024-03-14T10:04:58.828 INFO:teuthology.orchestra.run.smithi152.stdout: "dir": 243, 2024-03-14T10:04:58.828 INFO:teuthology.orchestra.run.smithi152.stdout: "dir+": 338, 2024-03-14T10:04:58.828 INFO:teuthology.orchestra.run.smithi152.stdout: "dir-": 95, 2024-03-14T10:04:58.828 INFO:teuthology.orchestra.run.smithi152.stdout: "dn": 15314, 2024-03-14T10:04:58.828 INFO:teuthology.orchestra.run.smithi152.stdout: "dn+": 15367, 2024-03-14T10:04:58.828 INFO:teuthology.orchestra.run.smithi152.stdout: "dn-": 53, 2024-03-14T10:04:58.828 INFO:teuthology.orchestra.run.smithi152.stdout: "cap": 0, 2024-03-14T10:04:58.828 INFO:teuthology.orchestra.run.smithi152.stdout: "cap+": 0, 2024-03-14T10:04:58.829 INFO:teuthology.orchestra.run.smithi152.stdout: "cap-": 0, 2024-03-14T10:04:58.829 INFO:teuthology.orchestra.run.smithi152.stdout: "rss": 82872, 2024-03-14T10:04:58.829 INFO:teuthology.orchestra.run.smithi152.stdout: "heap": 207156 2024-03-14T10:04:58.829 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.829 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_server": { 2024-03-14T10:04:58.829 INFO:teuthology.orchestra.run.smithi152.stdout: "dispatch_client_request": 0, 2024-03-14T10:04:58.829 INFO:teuthology.orchestra.run.smithi152.stdout: "dispatch_server_request": 0, 2024-03-14T10:04:58.829 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_request": 0, 2024-03-14T10:04:58.829 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_client_session": 0, 2024-03-14T10:04:58.829 INFO:teuthology.orchestra.run.smithi152.stdout: "handle_peer_request": 0, 2024-03-14T10:04:58.829 INFO:teuthology.orchestra.run.smithi152.stdout: "req_create_latency": { 2024-03-14T10:04:58.829 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.830 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.830 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.830 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.830 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getattr_latency": { 2024-03-14T10:04:58.830 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.830 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.830 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.830 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.830 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getfilelock_latency": { 2024-03-14T10:04:58.830 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.830 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.830 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.830 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.830 INFO:teuthology.orchestra.run.smithi152.stdout: "req_link_latency": { 2024-03-14T10:04:58.831 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.831 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.831 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.831 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.831 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookup_latency": { 2024-03-14T10:04:58.831 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.831 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.831 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.831 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.831 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookuphash_latency": { 2024-03-14T10:04:58.831 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.831 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.831 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.831 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.831 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupino_latency": { 2024-03-14T10:04:58.831 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.832 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.832 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.832 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.832 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupname_latency": { 2024-03-14T10:04:58.832 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.832 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.832 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.832 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.832 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupparent_latency": { 2024-03-14T10:04:58.832 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.832 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.832 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.832 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.832 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lookupsnap_latency": { 2024-03-14T10:04:58.832 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.832 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.833 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.833 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.833 INFO:teuthology.orchestra.run.smithi152.stdout: "req_lssnap_latency": { 2024-03-14T10:04:58.833 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.833 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.833 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.833 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.833 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mkdir_latency": { 2024-03-14T10:04:58.833 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.833 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.833 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.833 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.833 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mknod_latency": { 2024-03-14T10:04:58.833 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.833 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.834 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.834 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.834 INFO:teuthology.orchestra.run.smithi152.stdout: "req_mksnap_latency": { 2024-03-14T10:04:58.834 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.834 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.834 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.834 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.834 INFO:teuthology.orchestra.run.smithi152.stdout: "req_open_latency": { 2024-03-14T10:04:58.834 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.834 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.834 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.834 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.834 INFO:teuthology.orchestra.run.smithi152.stdout: "req_readdir_latency": { 2024-03-14T10:04:58.834 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.835 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.835 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.835 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.835 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rename_latency": { 2024-03-14T10:04:58.835 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.835 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.835 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.835 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.835 INFO:teuthology.orchestra.run.smithi152.stdout: "req_renamesnap_latency": { 2024-03-14T10:04:58.835 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.835 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.835 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.835 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.835 INFO:teuthology.orchestra.run.smithi152.stdout: "req_snapdiff_latency": { 2024-03-14T10:04:58.835 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.836 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.836 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.836 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.836 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmdir_latency": { 2024-03-14T10:04:58.836 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.836 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.836 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.836 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.836 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmsnap_latency": { 2024-03-14T10:04:58.836 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.836 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.836 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.836 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.836 INFO:teuthology.orchestra.run.smithi152.stdout: "req_rmxattr_latency": { 2024-03-14T10:04:58.836 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.836 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.837 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.837 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.837 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setattr_latency": { 2024-03-14T10:04:58.837 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.837 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.837 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.837 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.837 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setdirlayout_latency": { 2024-03-14T10:04:58.837 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.837 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.837 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.837 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.837 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setfilelock_latency": { 2024-03-14T10:04:58.837 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.837 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setlayout_latency": { 2024-03-14T10:04:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: "req_setxattr_latency": { 2024-03-14T10:04:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: "req_symlink_latency": { 2024-03-14T10:04:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.838 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.839 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.839 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.839 INFO:teuthology.orchestra.run.smithi152.stdout: "req_unlink_latency": { 2024-03-14T10:04:58.839 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.839 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.839 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.839 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.839 INFO:teuthology.orchestra.run.smithi152.stdout: "cap_revoke_eviction": 0, 2024-03-14T10:04:58.839 INFO:teuthology.orchestra.run.smithi152.stdout: "cap_acquisition_throttle": 0, 2024-03-14T10:04:58.839 INFO:teuthology.orchestra.run.smithi152.stdout: "req_getvxattr_latency": { 2024-03-14T10:04:58.839 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.839 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.839 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.839 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:58.840 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.840 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_sessions": { 2024-03-14T10:04:58.840 INFO:teuthology.orchestra.run.smithi152.stdout: "session_count": 2, 2024-03-14T10:04:58.840 INFO:teuthology.orchestra.run.smithi152.stdout: "session_add": 7, 2024-03-14T10:04:58.840 INFO:teuthology.orchestra.run.smithi152.stdout: "session_remove": 5, 2024-03-14T10:04:58.840 INFO:teuthology.orchestra.run.smithi152.stdout: "sessions_open": 4, 2024-03-14T10:04:58.840 INFO:teuthology.orchestra.run.smithi152.stdout: "sessions_stale": 0, 2024-03-14T10:04:58.840 INFO:teuthology.orchestra.run.smithi152.stdout: "total_load": 0, 2024-03-14T10:04:58.840 INFO:teuthology.orchestra.run.smithi152.stdout: "average_load": 0, 2024-03-14T10:04:58.840 INFO:teuthology.orchestra.run.smithi152.stdout: "avg_session_uptime": 7683, 2024-03-14T10:04:58.840 INFO:teuthology.orchestra.run.smithi152.stdout: "mdthresh_evicted": 0 2024-03-14T10:04:58.840 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.841 INFO:teuthology.orchestra.run.smithi152.stdout: "mempool": { 2024-03-14T10:04:58.841 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_bytes": 0, 2024-03-14T10:04:58.841 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_items": 0, 2024-03-14T10:04:58.841 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_bytes": 0, 2024-03-14T10:04:58.841 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_items": 0, 2024-03-14T10:04:58.841 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_bytes": 0, 2024-03-14T10:04:58.841 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_items": 0, 2024-03-14T10:04:58.841 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-14T10:04:58.841 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_items": 0, 2024-03-14T10:04:58.841 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-14T10:04:58.841 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_items": 0, 2024-03-14T10:04:58.841 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_bytes": 0, 2024-03-14T10:04:58.841 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_items": 0, 2024-03-14T10:04:58.841 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-14T10:04:58.841 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_items": 0, 2024-03-14T10:04:58.841 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_bytes": 0, 2024-03-14T10:04:58.842 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_items": 0, 2024-03-14T10:04:58.842 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_bytes": 0, 2024-03-14T10:04:58.842 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_items": 0, 2024-03-14T10:04:58.842 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-14T10:04:58.842 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_items": 0, 2024-03-14T10:04:58.842 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-14T10:04:58.842 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_items": 0, 2024-03-14T10:04:58.842 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_bytes": 0, 2024-03-14T10:04:58.842 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_items": 0, 2024-03-14T10:04:58.842 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_bytes": 0, 2024-03-14T10:04:58.842 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_items": 0, 2024-03-14T10:04:58.842 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-14T10:04:58.842 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_items": 0, 2024-03-14T10:04:58.842 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_bytes": 0, 2024-03-14T10:04:58.843 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_items": 0, 2024-03-14T10:04:58.843 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_bytes": 0, 2024-03-14T10:04:58.843 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_items": 0, 2024-03-14T10:04:58.843 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_bytes": 0, 2024-03-14T10:04:58.843 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_items": 0, 2024-03-14T10:04:58.843 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_bytes": 0, 2024-03-14T10:04:58.843 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_items": 0, 2024-03-14T10:04:58.843 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_bytes": 368335, 2024-03-14T10:04:58.843 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_items": 3870, 2024-03-14T10:04:58.843 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_bytes": 0, 2024-03-14T10:04:58.843 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_items": 0, 2024-03-14T10:04:58.843 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_bytes": 0, 2024-03-14T10:04:58.843 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_items": 0, 2024-03-14T10:04:58.843 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_bytes": 0, 2024-03-14T10:04:58.843 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_items": 0, 2024-03-14T10:04:58.843 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_bytes": 0, 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_items": 0, 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_bytes": 9832, 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_items": 205, 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_bytes": 0, 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_items": 0, 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_bytes": 0, 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_items": 0, 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_bytes": 19618503, 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_items": 165906, 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_bytes": 0, 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_items": 0, 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_bytes": 0, 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_items": 0 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: "objecter": { 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: "op_active": 0, 2024-03-14T10:04:58.844 INFO:teuthology.orchestra.run.smithi152.stdout: "op_laggy": 0, 2024-03-14T10:04:58.845 INFO:teuthology.orchestra.run.smithi152.stdout: "op_send": 2736, 2024-03-14T10:04:58.845 INFO:teuthology.orchestra.run.smithi152.stdout: "op_send_bytes": 26, 2024-03-14T10:04:58.845 INFO:teuthology.orchestra.run.smithi152.stdout: "op_resend": 4, 2024-03-14T10:04:58.845 INFO:teuthology.orchestra.run.smithi152.stdout: "op_reply": 2736, 2024-03-14T10:04:58.845 INFO:teuthology.orchestra.run.smithi152.stdout: "op_latency": { 2024-03-14T10:04:58.845 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 2736, 2024-03-14T10:04:58.845 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 1.881976143, 2024-03-14T10:04:58.845 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000687856 2024-03-14T10:04:58.845 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.845 INFO:teuthology.orchestra.run.smithi152.stdout: "op_inflight": 0, 2024-03-14T10:04:58.845 INFO:teuthology.orchestra.run.smithi152.stdout: "oplen_avg": { 2024-03-14T10:04:58.845 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 2736, 2024-03-14T10:04:58.845 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 2737 2024-03-14T10:04:58.845 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.845 INFO:teuthology.orchestra.run.smithi152.stdout: "op": 2736, 2024-03-14T10:04:58.845 INFO:teuthology.orchestra.run.smithi152.stdout: "op_r": 2736, 2024-03-14T10:04:58.846 INFO:teuthology.orchestra.run.smithi152.stdout: "op_w": 0, 2024-03-14T10:04:58.846 INFO:teuthology.orchestra.run.smithi152.stdout: "op_rmw": 0, 2024-03-14T10:04:58.846 INFO:teuthology.orchestra.run.smithi152.stdout: "op_pg": 0, 2024-03-14T10:04:58.846 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_stat": 1586, 2024-03-14T10:04:58.846 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_create": 0, 2024-03-14T10:04:58.846 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_read": 1147, 2024-03-14T10:04:58.846 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_write": 0, 2024-03-14T10:04:58.846 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_writefull": 0, 2024-03-14T10:04:58.846 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_writesame": 0, 2024-03-14T10:04:58.846 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_append": 0, 2024-03-14T10:04:58.846 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_zero": 0, 2024-03-14T10:04:58.846 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_truncate": 0, 2024-03-14T10:04:58.846 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_delete": 0, 2024-03-14T10:04:58.846 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_mapext": 0, 2024-03-14T10:04:58.847 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_sparse_read": 0, 2024-03-14T10:04:58.847 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_clonerange": 0, 2024-03-14T10:04:58.847 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_getxattr": 2, 2024-03-14T10:04:58.847 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_setxattr": 0, 2024-03-14T10:04:58.847 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_cmpxattr": 0, 2024-03-14T10:04:58.847 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_rmxattr": 0, 2024-03-14T10:04:58.847 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_resetxattrs": 0, 2024-03-14T10:04:58.847 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_call": 0, 2024-03-14T10:04:58.847 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_watch": 0, 2024-03-14T10:04:58.847 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_notify": 0, 2024-03-14T10:04:58.847 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_src_cmpxattr": 0, 2024-03-14T10:04:58.847 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_pgls": 0, 2024-03-14T10:04:58.847 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_pgls_filter": 0, 2024-03-14T10:04:58.847 INFO:teuthology.orchestra.run.smithi152.stdout: "osdop_other": 0, 2024-03-14T10:04:58.847 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_active": 0, 2024-03-14T10:04:58.847 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_send": 0, 2024-03-14T10:04:58.848 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_resend": 0, 2024-03-14T10:04:58.848 INFO:teuthology.orchestra.run.smithi152.stdout: "linger_ping": 0, 2024-03-14T10:04:58.848 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_active": 0, 2024-03-14T10:04:58.848 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_send": 0, 2024-03-14T10:04:58.848 INFO:teuthology.orchestra.run.smithi152.stdout: "poolop_resend": 0, 2024-03-14T10:04:58.848 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_active": 0, 2024-03-14T10:04:58.848 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_send": 0, 2024-03-14T10:04:58.848 INFO:teuthology.orchestra.run.smithi152.stdout: "poolstat_resend": 0, 2024-03-14T10:04:58.848 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_active": 0, 2024-03-14T10:04:58.848 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_send": 0, 2024-03-14T10:04:58.848 INFO:teuthology.orchestra.run.smithi152.stdout: "statfs_resend": 0, 2024-03-14T10:04:58.848 INFO:teuthology.orchestra.run.smithi152.stdout: "command_active": 0, 2024-03-14T10:04:58.848 INFO:teuthology.orchestra.run.smithi152.stdout: "command_send": 0, 2024-03-14T10:04:58.848 INFO:teuthology.orchestra.run.smithi152.stdout: "command_resend": 0, 2024-03-14T10:04:58.848 INFO:teuthology.orchestra.run.smithi152.stdout: "map_epoch": 96, 2024-03-14T10:04:58.849 INFO:teuthology.orchestra.run.smithi152.stdout: "map_full": 0, 2024-03-14T10:04:58.849 INFO:teuthology.orchestra.run.smithi152.stdout: "map_inc": 13, 2024-03-14T10:04:58.849 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_sessions": 11, 2024-03-14T10:04:58.849 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_session_open": 11, 2024-03-14T10:04:58.849 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_session_close": 0, 2024-03-14T10:04:58.849 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_laggy": 0, 2024-03-14T10:04:58.849 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_wr": 0, 2024-03-14T10:04:58.849 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_rd": 2, 2024-03-14T10:04:58.849 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_del": 0 2024-03-14T10:04:58.849 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.849 INFO:teuthology.orchestra.run.smithi152.stdout: "oft": { 2024-03-14T10:04:58.849 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_objs": 0, 2024-03-14T10:04:58.849 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_kv_pairs": 0, 2024-03-14T10:04:58.849 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_updates": 0, 2024-03-14T10:04:58.849 INFO:teuthology.orchestra.run.smithi152.stdout: "omap_total_removes": 0 2024-03-14T10:04:58.850 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.850 INFO:teuthology.orchestra.run.smithi152.stdout: "osd-slow-ops": { 2024-03-14T10:04:58.850 INFO:teuthology.orchestra.run.smithi152.stdout: "slow_ops_count": 0 2024-03-14T10:04:58.850 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.850 INFO:teuthology.orchestra.run.smithi152.stdout: "purge_queue": { 2024-03-14T10:04:58.850 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_ops": 0, 2024-03-14T10:04:58.850 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_ops_high_water": 0, 2024-03-14T10:04:58.850 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing": 0, 2024-03-14T10:04:58.850 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executing_high_water": 0, 2024-03-14T10:04:58.850 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_executed": 0, 2024-03-14T10:04:58.850 INFO:teuthology.orchestra.run.smithi152.stdout: "pq_item_in_journal": 0 2024-03-14T10:04:58.850 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.850 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-14T10:04:58.850 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-14T10:04:58.850 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-03-14T10:04:58.851 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-14T10:04:58.851 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 2981, 2024-03-14T10:04:58.851 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 183862873, 2024-03-14T10:04:58.851 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:58.851 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 2981, 2024-03-14T10:04:58.851 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-14T10:04:58.851 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-14T10:04:58.851 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 2981, 2024-03-14T10:04:58.851 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 183862873, 2024-03-14T10:04:58.851 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-14T10:04:58.851 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.851 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.851 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.851 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:58.851 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.852 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-objecter_bytes": { 2024-03-14T10:04:58.852 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-14T10:04:58.852 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-03-14T10:04:58.852 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-14T10:04:58.852 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-03-14T10:04:58.852 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-03-14T10:04:58.852 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:58.852 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:58.852 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 2736, 2024-03-14T10:04:58.852 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 181667981, 2024-03-14T10:04:58.852 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 2736, 2024-03-14T10:04:58.852 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 181667981, 2024-03-14T10:04:58.852 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-14T10:04:58.852 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.852 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.852 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.853 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:58.853 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.853 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-objecter_ops": { 2024-03-14T10:04:58.853 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-14T10:04:58.853 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 1024, 2024-03-14T10:04:58.853 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-14T10:04:58.853 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-03-14T10:04:58.853 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-03-14T10:04:58.853 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:58.853 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:58.853 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 2736, 2024-03-14T10:04:58.853 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 2736, 2024-03-14T10:04:58.853 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 2736, 2024-03-14T10:04:58.853 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 2736, 2024-03-14T10:04:58.853 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-14T10:04:58.854 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.854 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.854 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.854 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:58.854 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.854 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-write_buf_throttle": { 2024-03-14T10:04:58.854 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-14T10:04:58.854 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 3758096384, 2024-03-14T10:04:58.854 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-14T10:04:58.854 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-03-14T10:04:58.854 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-03-14T10:04:58.854 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:58.854 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:58.854 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 0, 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 0, 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-write_buf_throttle-0x558ed33465a0": { 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 3758096384, 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 0, 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 0, 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-14T10:04:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 0, 2024-03-14T10:04:58.856 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-14T10:04:58.856 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-14T10:04:58.856 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 0, 2024-03-14T10:04:58.856 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 0, 2024-03-14T10:04:58.856 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-14T10:04:58.856 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:04:58.856 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:04:58.856 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:04:58.856 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:58.856 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:04:58.856 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-03-14T10:04:59.192 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:04:58 smithi152 ceph-mon[48279]: pgmap v335: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:59.194 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-14T10:04:59.194 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-14T10:04:59.194 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-14T10:04:59.194 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-14T10:04:59.194 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-14T10:04:59.194 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-03-14T10:04:59.194 DEBUG:tasks.check_counter:Getting stats from l 2024-03-14T10:04:59.195 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 shell --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-14T10:04:59.270 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:04:58 smithi132 ceph-mon[47959]: pgmap v335: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:59.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:04:58 smithi078 ceph-mon[38143]: pgmap v335: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:04:59.418 INFO:teuthology.orchestra.run.smithi152.stderr:Inferring config /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/mon.c/config 2024-03-14T10:05:00.419 INFO:teuthology.orchestra.run.smithi152.stdout:{ 2024-03-14T10:05:00.419 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-0": { 2024-03-14T10:05:00.419 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 231, 2024-03-14T10:05:00.419 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 217, 2024-03-14T10:05:00.420 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 1693504, 2024-03-14T10:05:00.420 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 123152, 2024-03-14T10:05:00.420 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 4, 2024-03-14T10:05:00.420 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 0, 2024-03-14T10:05:00.420 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.044563833, 2024-03-14T10:05:00.420 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.011715555, 2024-03-14T10:05:00.420 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.037331900, 2024-03-14T10:05:00.420 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.005102944, 2024-03-14T10:05:00.420 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:05:00.420 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 217, 2024-03-14T10:05:00.420 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.011740384, 2024-03-14T10:05:00.420 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000054103 2024-03-14T10:05:00.420 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:05:00.421 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:05:00.421 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:05:00.421 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:05:00.421 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:05:00.421 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:05:00.421 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 1693504, 2024-03-14T10:05:00.421 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 123152 2024-03-14T10:05:00.421 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:05:00.421 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-1": { 2024-03-14T10:05:00.421 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 2, 2024-03-14T10:05:00.421 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 164, 2024-03-14T10:05:00.421 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 320, 2024-03-14T10:05:00.421 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 181072, 2024-03-14T10:05:00.421 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 5, 2024-03-14T10:05:00.421 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 3, 2024-03-14T10:05:00.422 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.009641383, 2024-03-14T10:05:00.422 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.006432450, 2024-03-14T10:05:00.422 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.001207426, 2024-03-14T10:05:00.422 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-14T10:05:00.422 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:05:00.422 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 161, 2024-03-14T10:05:00.422 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.010572272, 2024-03-14T10:05:00.422 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000065666 2024-03-14T10:05:00.422 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:05:00.422 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:05:00.422 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:05:00.422 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:05:00.422 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:05:00.423 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:05:00.423 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-14T10:05:00.423 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 181072 2024-03-14T10:05:00.423 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:05:00.423 INFO:teuthology.orchestra.run.smithi152.stdout: "AsyncMessenger::Worker-2": { 2024-03-14T10:05:00.423 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_messages": 0, 2024-03-14T10:05:00.423 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages": 2, 2024-03-14T10:05:00.423 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_bytes": 0, 2024-03-14T10:05:00.423 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_bytes": 0, 2024-03-14T10:05:00.423 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_created_connections": 2, 2024-03-14T10:05:00.423 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_active_connections": 1, 2024-03-14T10:05:00.423 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_total_time": 0.000370533, 2024-03-14T10:05:00.423 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_send_time": 0.000000000, 2024-03-14T10:05:00.423 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_recv_time": 0.000303746, 2024-03-14T10:05:00.424 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-14T10:05:00.424 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_messages_queue_lat": { 2024-03-14T10:05:00.424 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:05:00.424 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:05:00.424 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:05:00.424 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:05:00.424 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_handle_ack_lat": { 2024-03-14T10:05:00.424 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:05:00.424 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:05:00.424 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:05:00.424 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:05:00.424 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-14T10:05:00.424 INFO:teuthology.orchestra.run.smithi152.stdout: "msgr_send_encrypted_bytes": 0 2024-03-14T10:05:00.424 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:05:00.424 INFO:teuthology.orchestra.run.smithi152.stdout: "cct": { 2024-03-14T10:05:00.425 INFO:teuthology.orchestra.run.smithi152.stdout: "total_workers": 0, 2024-03-14T10:05:00.425 INFO:teuthology.orchestra.run.smithi152.stdout: "unhealthy_workers": 0 2024-03-14T10:05:00.425 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:05:00.425 INFO:teuthology.orchestra.run.smithi152.stdout: "mempool": { 2024-03-14T10:05:00.425 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_bytes": 0, 2024-03-14T10:05:00.425 INFO:teuthology.orchestra.run.smithi152.stdout: "bloom_filter_items": 0, 2024-03-14T10:05:00.425 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_bytes": 0, 2024-03-14T10:05:00.425 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_alloc_items": 0, 2024-03-14T10:05:00.425 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_bytes": 0, 2024-03-14T10:05:00.425 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_data_items": 0, 2024-03-14T10:05:00.425 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-14T10:05:00.425 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_onode_items": 0, 2024-03-14T10:05:00.425 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-14T10:05:00.425 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_meta_items": 0, 2024-03-14T10:05:00.426 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_bytes": 0, 2024-03-14T10:05:00.426 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_other_items": 0, 2024-03-14T10:05:00.426 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-14T10:05:00.426 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_cache_buffer_items": 0, 2024-03-14T10:05:00.426 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_bytes": 0, 2024-03-14T10:05:00.426 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_extent_items": 0, 2024-03-14T10:05:00.426 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_bytes": 0, 2024-03-14T10:05:00.426 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_blob_items": 0, 2024-03-14T10:05:00.426 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-14T10:05:00.426 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_shared_blob_items": 0, 2024-03-14T10:05:00.426 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-14T10:05:00.426 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_inline_bl_items": 0, 2024-03-14T10:05:00.426 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_bytes": 0, 2024-03-14T10:05:00.426 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_fsck_items": 0, 2024-03-14T10:05:00.426 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_bytes": 0, 2024-03-14T10:05:00.427 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_txc_items": 0, 2024-03-14T10:05:00.427 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-14T10:05:00.427 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_deferred_items": 0, 2024-03-14T10:05:00.427 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_bytes": 0, 2024-03-14T10:05:00.427 INFO:teuthology.orchestra.run.smithi152.stdout: "bluestore_writing_items": 0, 2024-03-14T10:05:00.427 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_bytes": 0, 2024-03-14T10:05:00.427 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_items": 0, 2024-03-14T10:05:00.427 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_bytes": 0, 2024-03-14T10:05:00.427 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_reader_items": 0, 2024-03-14T10:05:00.427 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_bytes": 0, 2024-03-14T10:05:00.427 INFO:teuthology.orchestra.run.smithi152.stdout: "bluefs_file_writer_items": 0, 2024-03-14T10:05:00.427 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_bytes": 149017, 2024-03-14T10:05:00.427 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_anon_items": 31, 2024-03-14T10:05:00.427 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_bytes": 0, 2024-03-14T10:05:00.427 INFO:teuthology.orchestra.run.smithi152.stdout: "buffer_meta_items": 0, 2024-03-14T10:05:00.428 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_bytes": 0, 2024-03-14T10:05:00.428 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_items": 0, 2024-03-14T10:05:00.428 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_bytes": 0, 2024-03-14T10:05:00.428 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_mapbl_items": 0, 2024-03-14T10:05:00.428 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_bytes": 0, 2024-03-14T10:05:00.428 INFO:teuthology.orchestra.run.smithi152.stdout: "osd_pglog_items": 0, 2024-03-14T10:05:00.428 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_bytes": 0, 2024-03-14T10:05:00.428 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_items": 0, 2024-03-14T10:05:00.428 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_bytes": 0, 2024-03-14T10:05:00.428 INFO:teuthology.orchestra.run.smithi152.stdout: "osdmap_mapping_items": 0, 2024-03-14T10:05:00.428 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_bytes": 0, 2024-03-14T10:05:00.428 INFO:teuthology.orchestra.run.smithi152.stdout: "pgmap_items": 0, 2024-03-14T10:05:00.428 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_bytes": 928, 2024-03-14T10:05:00.428 INFO:teuthology.orchestra.run.smithi152.stdout: "mds_co_items": 2, 2024-03-14T10:05:00.428 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_bytes": 0, 2024-03-14T10:05:00.429 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_1_items": 0, 2024-03-14T10:05:00.429 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_bytes": 0, 2024-03-14T10:05:00.429 INFO:teuthology.orchestra.run.smithi152.stdout: "unittest_2_items": 0 2024-03-14T10:05:00.429 INFO:teuthology.orchestra.run.smithi152.stdout: }, 2024-03-14T10:05:00.429 INFO:teuthology.orchestra.run.smithi152.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-14T10:05:00.429 INFO:teuthology.orchestra.run.smithi152.stdout: "val": 0, 2024-03-14T10:05:00.429 INFO:teuthology.orchestra.run.smithi152.stdout: "max": 104857600, 2024-03-14T10:05:00.429 INFO:teuthology.orchestra.run.smithi152.stdout: "get_started": 0, 2024-03-14T10:05:00.429 INFO:teuthology.orchestra.run.smithi152.stdout: "get": 229, 2024-03-14T10:05:00.429 INFO:teuthology.orchestra.run.smithi152.stdout: "get_sum": 1658550, 2024-03-14T10:05:00.429 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_fail": 0, 2024-03-14T10:05:00.429 INFO:teuthology.orchestra.run.smithi152.stdout: "get_or_fail_success": 229, 2024-03-14T10:05:00.429 INFO:teuthology.orchestra.run.smithi152.stdout: "take": 0, 2024-03-14T10:05:00.429 INFO:teuthology.orchestra.run.smithi152.stdout: "take_sum": 0, 2024-03-14T10:05:00.429 INFO:teuthology.orchestra.run.smithi152.stdout: "put": 229, 2024-03-14T10:05:00.429 INFO:teuthology.orchestra.run.smithi152.stdout: "put_sum": 1658550, 2024-03-14T10:05:00.430 INFO:teuthology.orchestra.run.smithi152.stdout: "wait": { 2024-03-14T10:05:00.430 INFO:teuthology.orchestra.run.smithi152.stdout: "avgcount": 0, 2024-03-14T10:05:00.430 INFO:teuthology.orchestra.run.smithi152.stdout: "sum": 0.000000000, 2024-03-14T10:05:00.430 INFO:teuthology.orchestra.run.smithi152.stdout: "avgtime": 0.000000000 2024-03-14T10:05:00.430 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:05:00.430 INFO:teuthology.orchestra.run.smithi152.stdout: } 2024-03-14T10:05:00.430 INFO:teuthology.orchestra.run.smithi152.stdout:} 2024-03-14T10:05:00.801 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-14T10:05:00.801 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-14T10:05:00.802 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-03-14T10:05:00.802 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-03-14T10:05:00.802 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-03-14T10:05:00.802 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-03-14T10:05:00.802 ERROR:teuthology.run_tasks:Manager failed: check-counter Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_2f465319e3c83d48f999325459bcdfd0346449c0/qa/tasks/check_counter.py", line 125, in end raise RuntimeError("The following counters failed to be set " RuntimeError: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} 2024-03-14T10:05:00.803 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-14T10:05:00.813 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-14T10:05:00.813 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T10:05:00.813 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-14T10:05:00.862 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T10:05:00.862 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-14T10:05:00.947 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T10:05:01.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.187+0000 7fb107400700 1 -- 172.21.15.78:0/340704688 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb100100dd0 msgr2=0x7fb1001011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:01.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.187+0000 7fb107400700 1 --2- 172.21.15.78:0/340704688 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb100100dd0 0x7fb1001011b0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009990 tx=0x7fb0f4009ca0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.187+0000 7fb107400700 1 -- 172.21.15.78:0/340704688 shutdown_connections 2024-03-14T10:05:01.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.187+0000 7fb107400700 1 --2- 172.21.15.78:0/340704688 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb100105ef0 0x7fb10010da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.187+0000 7fb107400700 1 --2- 172.21.15.78:0/340704688 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb100100dd0 0x7fb1001011b0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.187+0000 7fb107400700 1 --2- 172.21.15.78:0/340704688 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb100101780 0x7fb1001057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.187+0000 7fb107400700 1 -- 172.21.15.78:0/340704688 >> 172.21.15.78:0/340704688 conn(0x7fb1000fc630 msgr2=0x7fb1000fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:01.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.187+0000 7fb107400700 1 -- 172.21.15.78:0/340704688 shutdown_connections 2024-03-14T10:05:01.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.187+0000 7fb107400700 1 -- 172.21.15.78:0/340704688 wait complete. 2024-03-14T10:05:01.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.188+0000 7fb107400700 1 Processor -- start 2024-03-14T10:05:01.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.188+0000 7fb107400700 1 -- start start 2024-03-14T10:05:01.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb107400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb100100dd0 0x7fb100199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:01.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb107400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb100105ef0 0x7fb100199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:01.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb107400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb10019e000 0x7fb10019e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb107400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10019ea80 con 0x7fb100100dd0 2024-03-14T10:05:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb107400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10019ebf0 con 0x7fb10019e000 2024-03-14T10:05:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb107400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10019edb0 con 0x7fb100105ef0 2024-03-14T10:05:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb104a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb100100dd0 0x7fb100199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb105400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb10019e000 0x7fb10019e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb0ffe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb100105ef0 0x7fb100199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb105400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb10019e000 0x7fb10019e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:51726/0 (socket says 172.21.15.78:51726) 2024-03-14T10:05:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb105400700 1 -- 172.21.15.78:0/2133621747 learned_addr learned my addr 172.21.15.78:0/2133621747 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:05:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb0ffe00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb100105ef0 0x7fb100199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:35920/0 (socket says 172.21.15.78:35920) 2024-03-14T10:05:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb0ffe00700 1 -- 172.21.15.78:0/2133621747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb10019e000 msgr2=0x7fb10019e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb0ffe00700 1 --2- 172.21.15.78:0/2133621747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb10019e000 0x7fb10019e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb0ffe00700 1 -- 172.21.15.78:0/2133621747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb100100dd0 msgr2=0x7fb100199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb0ffe00700 1 --2- 172.21.15.78:0/2133621747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb100100dd0 0x7fb100199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb0ffe00700 1 -- 172.21.15.78:0/2133621747 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f40095f0 con 0x7fb100105ef0 2024-03-14T10:05:01.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb105400700 1 --2- 172.21.15.78:0/2133621747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb10019e000 0x7fb10019e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:05:01.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.189+0000 7fb104a00700 1 --2- 172.21.15.78:0/2133621747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb100100dd0 0x7fb100199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:05:01.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.190+0000 7fb0ffe00700 1 --2- 172.21.15.78:0/2133621747 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb100105ef0 0x7fb100199c60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb10006a1e0 tx=0x7fb0ec00dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:01.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.191+0000 7fb0fd600700 1 -- 172.21.15.78:0/2133621747 <== mon.1 v2:172.21.15.152:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec00f890 con 0x7fb100105ef0 2024-03-14T10:05:01.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.191+0000 7fb0fd600700 1 -- 172.21.15.78:0/2133621747 <== mon.1 v2:172.21.15.152:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0ec010460 con 0x7fb100105ef0 2024-03-14T10:05:01.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.191+0000 7fb107400700 1 -- 172.21.15.78:0/2133621747 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1001a6310 con 0x7fb100105ef0 2024-03-14T10:05:01.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.191+0000 7fb0fd600700 1 -- 172.21.15.78:0/2133621747 <== mon.1 v2:172.21.15.152:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec00e610 con 0x7fb100105ef0 2024-03-14T10:05:01.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.191+0000 7fb107400700 1 -- 172.21.15.78:0/2133621747 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1001a6840 con 0x7fb100105ef0 2024-03-14T10:05:01.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.192+0000 7fb107400700 1 -- 172.21.15.78:0/2133621747 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1001022a0 con 0x7fb100105ef0 2024-03-14T10:05:01.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.194+0000 7fb0fd600700 1 -- 172.21.15.78:0/2133621747 <== mon.1 v2:172.21.15.152:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fb0ec00f9f0 con 0x7fb100105ef0 2024-03-14T10:05:01.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.195+0000 7fb0fd600700 1 --2- 172.21.15.78:0/2133621747 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0e80a0c80 0x7fb0e80a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:01.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.195+0000 7fb0fd600700 1 -- 172.21.15.78:0/2133621747 <== mon.1 v2:172.21.15.152:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fb0ec0cc440 con 0x7fb100105ef0 2024-03-14T10:05:01.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.195+0000 7fb104a00700 1 --2- 172.21.15.78:0/2133621747 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0e80a0c80 0x7fb0e80a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:01.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.196+0000 7fb104a00700 1 --2- 172.21.15.78:0/2133621747 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0e80a0c80 0x7fb0e80a3140 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4006010 tx=0x7fb0f400b260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:01.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.199+0000 7fb0fd600700 1 -- 172.21.15.78:0/2133621747 <== mon.1 v2:172.21.15.152:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb0ec0956a0 con 0x7fb100105ef0 2024-03-14T10:05:01.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:01 smithi078 ceph-mon[38143]: pgmap v336: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:05:01.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.423+0000 7fb107400700 1 -- 172.21.15.78:0/2133621747 --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb10019aa20 con 0x7fb100105ef0 2024-03-14T10:05:01.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.425+0000 7fb0fd600700 1 -- 172.21.15.78:0/2133621747 <== mon.1 v2:172.21.15.152:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+25214 (secure 0 0 0) 0x7fb0ec098cc0 con 0x7fb100105ef0 2024-03-14T10:05:01.426 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T10:05:01.426 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":96,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T10:04:17.907246+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1935098096":"2024-03-15T09:54:09.924780+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6801/2580616009":"2024-03-15T09:54:09.924780+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/3308792589":"2024-03-15T09:54:09.924780+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/555376800":"2024-03-15T09:54:09.924780+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/3220727810":"2024-03-15T09:54:09.924780+0000","172.21.15.78:0/2934089773":"2024-03-15T09:54:09.924780+0000","172.21.15.78:6800/2580616009":"2024-03-15T09:54:09.924780+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T10:05:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.427+0000 7fb107400700 1 -- 172.21.15.78:0/2133621747 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0e80a0c80 msgr2=0x7fb0e80a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.427+0000 7fb107400700 1 --2- 172.21.15.78:0/2133621747 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0e80a0c80 0x7fb0e80a3140 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4006010 tx=0x7fb0f400b260 comp rx=0 tx=0).stop 2024-03-14T10:05:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.427+0000 7fb107400700 1 -- 172.21.15.78:0/2133621747 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb100105ef0 msgr2=0x7fb100199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.427+0000 7fb107400700 1 --2- 172.21.15.78:0/2133621747 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb100105ef0 0x7fb100199c60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb10006a1e0 tx=0x7fb0ec00dc30 comp rx=0 tx=0).stop 2024-03-14T10:05:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.427+0000 7fb107400700 1 -- 172.21.15.78:0/2133621747 shutdown_connections 2024-03-14T10:05:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.427+0000 7fb107400700 1 --2- 172.21.15.78:0/2133621747 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fb0e80a0c80 0x7fb0e80a3140 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.427+0000 7fb107400700 1 --2- 172.21.15.78:0/2133621747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb10019e000 0x7fb10019e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.427+0000 7fb107400700 1 --2- 172.21.15.78:0/2133621747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb100100dd0 0x7fb100199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.427+0000 7fb107400700 1 --2- 172.21.15.78:0/2133621747 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fb100105ef0 0x7fb100199c60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.427+0000 7fb107400700 1 -- 172.21.15.78:0/2133621747 >> 172.21.15.78:0/2133621747 conn(0x7fb1000fc630 msgr2=0x7fb1000fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:01.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.427+0000 7fb107400700 1 -- 172.21.15.78:0/2133621747 shutdown_connections 2024-03-14T10:05:01.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.427+0000 7fb107400700 1 -- 172.21.15.78:0/2133621747 wait complete. 2024-03-14T10:05:01.447 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T09:48:37.618450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-14T09:52:03.938666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-14T09:52:05.149985+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 95, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-14T09:52:08.227698+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 95, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-14T10:05:01.448 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T10:05:01.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:01 smithi152 ceph-mon[48279]: pgmap v336: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:05:01.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:01 smithi132 ceph-mon[47959]: pgmap v336: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T10:05:01.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.643+0000 7f579b600700 1 -- 172.21.15.78:0/1321984261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57941018d0 msgr2=0x7f5794105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:01.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.643+0000 7f579b600700 1 --2- 172.21.15.78:0/1321984261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57941018d0 0x7f5794105910 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f5780009990 tx=0x7f5780009ca0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.643+0000 7f579b600700 1 -- 172.21.15.78:0/1321984261 shutdown_connections 2024-03-14T10:05:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.643+0000 7f579b600700 1 --2- 172.21.15.78:0/1321984261 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5794106040 0x7f579410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.643+0000 7f579b600700 1 --2- 172.21.15.78:0/1321984261 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f5794100f20 0x7f5794101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.643+0000 7f579b600700 1 --2- 172.21.15.78:0/1321984261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57941018d0 0x7f5794105910 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.643+0000 7f579b600700 1 -- 172.21.15.78:0/1321984261 >> 172.21.15.78:0/1321984261 conn(0x7f57940fc780 msgr2=0x7f57940feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.643+0000 7f579b600700 1 -- 172.21.15.78:0/1321984261 shutdown_connections 2024-03-14T10:05:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.643+0000 7f579b600700 1 -- 172.21.15.78:0/1321984261 wait complete. 2024-03-14T10:05:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.645+0000 7f579b600700 1 Processor -- start 2024-03-14T10:05:01.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.645+0000 7f579b600700 1 -- start start 2024-03-14T10:05:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.645+0000 7f579b600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794100f20 0x7f57941997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f579b600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5794106040 0x7f5794199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f579b600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f579419cff0 0x7f579419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:01.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f579b600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579419d9e0 con 0x7f5794100f20 2024-03-14T10:05:01.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f579b600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579419db50 con 0x7f5794106040 2024-03-14T10:05:01.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f579b600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579419dd10 con 0x7f579419cff0 2024-03-14T10:05:01.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f5798c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794100f20 0x7f57941997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:01.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f5798c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794100f20 0x7f57941997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58430/0 (socket says 172.21.15.78:58430) 2024-03-14T10:05:01.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f5798c00700 1 -- 172.21.15.78:0/2583104864 learned_addr learned my addr 172.21.15.78:0/2583104864 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:05:01.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f5793e00700 1 --2- 172.21.15.78:0/2583104864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5794106040 0x7f5794199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:01.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f5799600700 1 --2- 172.21.15.78:0/2583104864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f579419cff0 0x7f579419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:01.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f5798c00700 1 -- 172.21.15.78:0/2583104864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f579419cff0 msgr2=0x7f579419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:01.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f5798c00700 1 --2- 172.21.15.78:0/2583104864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f579419cff0 0x7f579419d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f5798c00700 1 -- 172.21.15.78:0/2583104864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5794106040 msgr2=0x7f5794199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:01.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f5798c00700 1 --2- 172.21.15.78:0/2583104864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5794106040 0x7f5794199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f5798c00700 1 -- 172.21.15.78:0/2583104864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57800095f0 con 0x7f5794100f20 2024-03-14T10:05:01.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f5799600700 1 --2- 172.21.15.78:0/2583104864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f579419cff0 0x7f579419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:05:01.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f5793e00700 1 --2- 172.21.15.78:0/2583104864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5794106040 0x7f5794199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:05:01.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.646+0000 7f5798c00700 1 --2- 172.21.15.78:0/2583104864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794100f20 0x7f57941997c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f578800e9a0 tx=0x7f578800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:01.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.647+0000 7f5791600700 1 -- 172.21.15.78:0/2583104864 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5788009790 con 0x7f5794100f20 2024-03-14T10:05:01.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.647+0000 7f579b600700 1 -- 172.21.15.78:0/2583104864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57941a6380 con 0x7f5794100f20 2024-03-14T10:05:01.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.647+0000 7f5791600700 1 -- 172.21.15.78:0/2583104864 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f578800cbd0 con 0x7f5794100f20 2024-03-14T10:05:01.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.647+0000 7f5791600700 1 -- 172.21.15.78:0/2583104864 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5788018620 con 0x7f5794100f20 2024-03-14T10:05:01.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.647+0000 7f579b600700 1 -- 172.21.15.78:0/2583104864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57941a6890 con 0x7f5794100f20 2024-03-14T10:05:01.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.648+0000 7f579b600700 1 -- 172.21.15.78:0/2583104864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57941023f0 con 0x7f5794100f20 2024-03-14T10:05:01.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.650+0000 7f5791600700 1 -- 172.21.15.78:0/2583104864 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f578800cd40 con 0x7f5794100f20 2024-03-14T10:05:01.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.651+0000 7f5791600700 1 --2- 172.21.15.78:0/2583104864 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f577c0a0c30 0x7f577c0a30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:01.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.651+0000 7f5791600700 1 -- 172.21.15.78:0/2583104864 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f5788014070 con 0x7f5794100f20 2024-03-14T10:05:01.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.651+0000 7f5793e00700 1 --2- 172.21.15.78:0/2583104864 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f577c0a0c30 0x7f577c0a30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:01.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.652+0000 7f5793e00700 1 --2- 172.21.15.78:0/2583104864 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f577c0a0c30 0x7f577c0a30f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f579419ad10 tx=0x7f5780005fb0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:01.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.653+0000 7f5791600700 1 -- 172.21.15.78:0/2583104864 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5788095ff0 con 0x7f5794100f20 2024-03-14T10:05:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.863+0000 7f579b600700 1 -- 172.21.15.78:0/2583104864 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f579419ab60 con 0x7f5794100f20 2024-03-14T10:05:01.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.864+0000 7f5791600700 1 -- 172.21.15.78:0/2583104864 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7f57880260a0 con 0x7f5794100f20 2024-03-14T10:05:01.865 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-03-14T10:05:01.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.866+0000 7f579b600700 1 -- 172.21.15.78:0/2583104864 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f577c0a0c30 msgr2=0x7f577c0a30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:01.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.866+0000 7f579b600700 1 --2- 172.21.15.78:0/2583104864 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f577c0a0c30 0x7f577c0a30f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f579419ad10 tx=0x7f5780005fb0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.866+0000 7f579b600700 1 -- 172.21.15.78:0/2583104864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794100f20 msgr2=0x7f57941997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:01.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.866+0000 7f579b600700 1 --2- 172.21.15.78:0/2583104864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794100f20 0x7f57941997c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f578800e9a0 tx=0x7f578800ed60 comp rx=0 tx=0).stop 2024-03-14T10:05:01.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.866+0000 7f579b600700 1 -- 172.21.15.78:0/2583104864 shutdown_connections 2024-03-14T10:05:01.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.866+0000 7f579b600700 1 --2- 172.21.15.78:0/2583104864 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f579419cff0 0x7f579419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.866+0000 7f579b600700 1 --2- 172.21.15.78:0/2583104864 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5794100f20 0x7f57941997c0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.866+0000 7f579b600700 1 --2- 172.21.15.78:0/2583104864 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f577c0a0c30 0x7f577c0a30f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.866+0000 7f579b600700 1 --2- 172.21.15.78:0/2583104864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5794106040 0x7f5794199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:01.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.866+0000 7f579b600700 1 -- 172.21.15.78:0/2583104864 >> 172.21.15.78:0/2583104864 conn(0x7f57940fc780 msgr2=0x7f57940feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:01.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.866+0000 7f579b600700 1 -- 172.21.15.78:0/2583104864 shutdown_connections 2024-03-14T10:05:01.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:01.867+0000 7f579b600700 1 -- 172.21.15.78:0/2583104864 wait complete. 2024-03-14T10:05:01.886 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-14T10:05:02.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.083+0000 7f2534000700 1 -- 172.21.15.78:0/1924124005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f252c100f20 msgr2=0x7f252c101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:02.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.083+0000 7f2534000700 1 --2- 172.21.15.78:0/1924124005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f252c100f20 0x7f252c101300 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f2520009990 tx=0x7f2520009ca0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.083+0000 7f2534000700 1 -- 172.21.15.78:0/1924124005 shutdown_connections 2024-03-14T10:05:02.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.083+0000 7f2534000700 1 --2- 172.21.15.78:0/1924124005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f252c106040 0x7f252c10dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.083+0000 7f2534000700 1 --2- 172.21.15.78:0/1924124005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f252c100f20 0x7f252c101300 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.083+0000 7f2534000700 1 --2- 172.21.15.78:0/1924124005 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f252c1018d0 0x7f252c105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.083+0000 7f2534000700 1 -- 172.21.15.78:0/1924124005 >> 172.21.15.78:0/1924124005 conn(0x7f252c0fc7a0 msgr2=0x7f252c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:02.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.083+0000 7f2534000700 1 -- 172.21.15.78:0/1924124005 shutdown_connections 2024-03-14T10:05:02.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.083+0000 7f2534000700 1 -- 172.21.15.78:0/1924124005 wait complete. 2024-03-14T10:05:02.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.084+0000 7f2534000700 1 Processor -- start 2024-03-14T10:05:02.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.084+0000 7f2534000700 1 -- start start 2024-03-14T10:05:02.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.085+0000 7f2534000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f252c100f20 0x7f252c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:02.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.085+0000 7f2534000700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f252c106040 0x7f252c0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:02.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.085+0000 7f2534000700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f252c073b30 0x7f252c1a57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:02.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.085+0000 7f2534000700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c1a5dc0 con 0x7f252c100f20 2024-03-14T10:05:02.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.085+0000 7f2534000700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c1a5f30 con 0x7f252c106040 2024-03-14T10:05:02.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.085+0000 7f2534000700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c1a60f0 con 0x7f252c073b30 2024-03-14T10:05:02.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.085+0000 7f2531600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f252c100f20 0x7f252c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:02.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.085+0000 7f2531600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f252c100f20 0x7f252c0730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58450/0 (socket says 172.21.15.78:58450) 2024-03-14T10:05:02.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.085+0000 7f2531600700 1 -- 172.21.15.78:0/2496982785 learned_addr learned my addr 172.21.15.78:0/2496982785 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:05:02.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.085+0000 7f2532000700 1 --2- 172.21.15.78:0/2496982785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f252c073b30 0x7f252c1a57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:02.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.085+0000 7f2530c00700 1 --2- 172.21.15.78:0/2496982785 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f252c106040 0x7f252c0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:02.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.085+0000 7f2531600700 1 -- 172.21.15.78:0/2496982785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f252c073b30 msgr2=0x7f252c1a57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:02.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.085+0000 7f2531600700 1 --2- 172.21.15.78:0/2496982785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f252c073b30 0x7f252c1a57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.085+0000 7f2531600700 1 -- 172.21.15.78:0/2496982785 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f252c106040 msgr2=0x7f252c0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:02.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.085+0000 7f2531600700 1 --2- 172.21.15.78:0/2496982785 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f252c106040 0x7f252c0735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.086+0000 7f2531600700 1 -- 172.21.15.78:0/2496982785 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25200095f0 con 0x7f252c100f20 2024-03-14T10:05:02.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.086+0000 7f2531600700 1 --2- 172.21.15.78:0/2496982785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f252c100f20 0x7f252c0730b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f2520005910 tx=0x7f2520004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:02.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.086+0000 7f252a000700 1 -- 172.21.15.78:0/2496982785 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252001d070 con 0x7f252c100f20 2024-03-14T10:05:02.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.086+0000 7f252a000700 1 -- 172.21.15.78:0/2496982785 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2520022470 con 0x7f252c100f20 2024-03-14T10:05:02.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.086+0000 7f2534000700 1 -- 172.21.15.78:0/2496982785 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252c1a6370 con 0x7f252c100f20 2024-03-14T10:05:02.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.086+0000 7f252a000700 1 -- 172.21.15.78:0/2496982785 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2520017650 con 0x7f252c100f20 2024-03-14T10:05:02.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.086+0000 7f2534000700 1 -- 172.21.15.78:0/2496982785 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f252c1a6790 con 0x7f252c100f20 2024-03-14T10:05:02.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.087+0000 7f2534000700 1 -- 172.21.15.78:0/2496982785 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f252c109260 con 0x7f252c100f20 2024-03-14T10:05:02.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.089+0000 7f252a000700 1 -- 172.21.15.78:0/2496982785 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f25200225e0 con 0x7f252c100f20 2024-03-14T10:05:02.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.090+0000 7f252a000700 1 --2- 172.21.15.78:0/2496982785 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f25140a0c00 0x7f25140a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:02.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.090+0000 7f252a000700 1 -- 172.21.15.78:0/2496982785 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f25200cda80 con 0x7f252c100f20 2024-03-14T10:05:02.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.091+0000 7f2530c00700 1 --2- 172.21.15.78:0/2496982785 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f25140a0c00 0x7f25140a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:02.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.091+0000 7f2530c00700 1 --2- 172.21.15.78:0/2496982785 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f25140a0c00 0x7f25140a30c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f252c0747e0 tx=0x7f2518009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:02.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.094+0000 7f252a000700 1 -- 172.21.15.78:0/2496982785 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2520096d70 con 0x7f252c100f20 2024-03-14T10:05:02.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.302+0000 7f2534000700 1 -- 172.21.15.78:0/2496982785 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f252c074590 con 0x7f252c100f20 2024-03-14T10:05:02.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.302+0000 7f252a000700 1 -- 172.21.15.78:0/2496982785 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) v1 ==== 104+0+11 (secure 0 0 0) 0x7f252009a390 con 0x7f252c100f20 2024-03-14T10:05:02.303 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T10:05:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.305+0000 7f2534000700 1 -- 172.21.15.78:0/2496982785 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f25140a0c00 msgr2=0x7f25140a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.305+0000 7f2534000700 1 --2- 172.21.15.78:0/2496982785 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f25140a0c00 0x7f25140a30c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f252c0747e0 tx=0x7f2518009260 comp rx=0 tx=0).stop 2024-03-14T10:05:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.305+0000 7f2534000700 1 -- 172.21.15.78:0/2496982785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f252c100f20 msgr2=0x7f252c0730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.305+0000 7f2534000700 1 --2- 172.21.15.78:0/2496982785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f252c100f20 0x7f252c0730b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f2520005910 tx=0x7f2520004730 comp rx=0 tx=0).stop 2024-03-14T10:05:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.305+0000 7f2534000700 1 -- 172.21.15.78:0/2496982785 shutdown_connections 2024-03-14T10:05:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.305+0000 7f2534000700 1 --2- 172.21.15.78:0/2496982785 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f252c073b30 0x7f252c1a57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.305+0000 7f2534000700 1 --2- 172.21.15.78:0/2496982785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f252c100f20 0x7f252c0730b0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.305+0000 7f2534000700 1 --2- 172.21.15.78:0/2496982785 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f25140a0c00 0x7f25140a30c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.305+0000 7f2534000700 1 --2- 172.21.15.78:0/2496982785 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f252c106040 0x7f252c0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.305+0000 7f2534000700 1 -- 172.21.15.78:0/2496982785 >> 172.21.15.78:0/2496982785 conn(0x7f252c0fc7a0 msgr2=0x7f252c0fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:02.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.305+0000 7f2534000700 1 -- 172.21.15.78:0/2496982785 shutdown_connections 2024-03-14T10:05:02.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.305+0000 7f2534000700 1 -- 172.21.15.78:0/2496982785 wait complete. 2024-03-14T10:05:02.324 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-14T10:05:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:02 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2133621747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T10:05:02.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:02 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2583104864' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T10:05:02.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:02 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2133621747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T10:05:02.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:02 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2583104864' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T10:05:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.521+0000 7f47d8400700 1 -- 172.21.15.78:0/2292593643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47d0106050 msgr2=0x7f47d010dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.521+0000 7f47d8400700 1 --2- 172.21.15.78:0/2292593643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47d0106050 0x7f47d010dbd0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f47c800b0a0 tx=0x7f47c800b3b0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.521+0000 7f47d8400700 1 -- 172.21.15.78:0/2292593643 shutdown_connections 2024-03-14T10:05:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.521+0000 7f47d8400700 1 --2- 172.21.15.78:0/2292593643 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47d0100f30 0x7f47d0101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.521+0000 7f47d8400700 1 --2- 172.21.15.78:0/2292593643 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47d0106050 0x7f47d010dbd0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.521+0000 7f47d8400700 1 --2- 172.21.15.78:0/2292593643 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47d01018e0 0x7f47d0105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.521+0000 7f47d8400700 1 -- 172.21.15.78:0/2292593643 >> 172.21.15.78:0/2292593643 conn(0x7f47d00fc7b0 msgr2=0x7f47d00febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.521+0000 7f47d8400700 1 -- 172.21.15.78:0/2292593643 shutdown_connections 2024-03-14T10:05:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.521+0000 7f47d8400700 1 -- 172.21.15.78:0/2292593643 wait complete. 2024-03-14T10:05:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.522+0000 7f47d8400700 1 Processor -- start 2024-03-14T10:05:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.523+0000 7f47d8400700 1 -- start start 2024-03-14T10:05:02.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.523+0000 7f47d8400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47d0100f30 0x7f47d019f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:02.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.523+0000 7f47d8400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47d0106050 0x7f47d019fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:02.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.523+0000 7f47d8400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47d0199760 0x7f47d0199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:02.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.523+0000 7f47d8400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d019a1e0 con 0x7f47d0106050 2024-03-14T10:05:02.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.523+0000 7f47d8400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d019a350 con 0x7f47d0199760 2024-03-14T10:05:02.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.523+0000 7f47d8400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d019a510 con 0x7f47d0100f30 2024-03-14T10:05:02.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.524+0000 7f47d5000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47d0106050 0x7f47d019fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:02.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.524+0000 7f47d5000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47d0106050 0x7f47d019fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58456/0 (socket says 172.21.15.78:58456) 2024-03-14T10:05:02.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.524+0000 7f47d5a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47d0100f30 0x7f47d019f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:02.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.524+0000 7f47d5000700 1 -- 172.21.15.78:0/2178587846 learned_addr learned my addr 172.21.15.78:0/2178587846 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:05:02.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.524+0000 7f47d6400700 1 --2- 172.21.15.78:0/2178587846 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47d0199760 0x7f47d0199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:02.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.524+0000 7f47d5000700 1 -- 172.21.15.78:0/2178587846 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47d0100f30 msgr2=0x7f47d019f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:02.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.524+0000 7f47d5000700 1 --2- 172.21.15.78:0/2178587846 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47d0100f30 0x7f47d019f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.524+0000 7f47d5000700 1 -- 172.21.15.78:0/2178587846 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47d0199760 msgr2=0x7f47d0199c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:02.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.524+0000 7f47d5000700 1 --2- 172.21.15.78:0/2178587846 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47d0199760 0x7f47d0199c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.524+0000 7f47d5000700 1 -- 172.21.15.78:0/2178587846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c8009d00 con 0x7f47d0106050 2024-03-14T10:05:02.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.524+0000 7f47d5000700 1 --2- 172.21.15.78:0/2178587846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47d0106050 0x7f47d019fb70 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f47bc00d950 tx=0x7f47bc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:02.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.525+0000 7f47ce000700 1 -- 172.21.15.78:0/2178587846 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc004220 con 0x7f47d0106050 2024-03-14T10:05:02.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.525+0000 7f47ce000700 1 -- 172.21.15.78:0/2178587846 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f47bc007620 con 0x7f47d0106050 2024-03-14T10:05:02.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.525+0000 7f47d8400700 1 -- 172.21.15.78:0/2178587846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d01a6500 con 0x7f47d0106050 2024-03-14T10:05:02.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.525+0000 7f47ce000700 1 -- 172.21.15.78:0/2178587846 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc005670 con 0x7f47d0106050 2024-03-14T10:05:02.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.525+0000 7f47d8400700 1 -- 172.21.15.78:0/2178587846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d01a69e0 con 0x7f47d0106050 2024-03-14T10:05:02.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.526+0000 7f47d8400700 1 -- 172.21.15.78:0/2178587846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d0102400 con 0x7f47d0106050 2024-03-14T10:05:02.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.527+0000 7f47ce000700 1 -- 172.21.15.78:0/2178587846 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f47bc003680 con 0x7f47d0106050 2024-03-14T10:05:02.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.529+0000 7f47ce000700 1 --2- 172.21.15.78:0/2178587846 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f47b80a0c00 0x7f47b80a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:02.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.530+0000 7f47ce000700 1 -- 172.21.15.78:0/2178587846 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f47bc0cc4f0 con 0x7f47d0106050 2024-03-14T10:05:02.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.530+0000 7f47d5a00700 1 --2- 172.21.15.78:0/2178587846 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f47b80a0c00 0x7f47b80a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:02.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.530+0000 7f47d5a00700 1 --2- 172.21.15.78:0/2178587846 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f47b80a0c00 0x7f47b80a30c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f47c4005950 tx=0x7f47c400b410 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:02.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.534+0000 7f47ce000700 1 -- 172.21.15.78:0/2178587846 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f47bc095750 con 0x7f47d0106050 2024-03-14T10:05:02.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:02 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2133621747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T10:05:02.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:02 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2583104864' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T10:05:02.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.738+0000 7f47d8400700 1 -- 172.21.15.78:0/2178587846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f47d01018e0 con 0x7f47d0106050 2024-03-14T10:05:02.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.739+0000 7f47ce000700 1 -- 172.21.15.78:0/2178587846 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) v1 ==== 100+0+11 (secure 0 0 0) 0x7f47bc098d70 con 0x7f47d0106050 2024-03-14T10:05:02.740 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2024-03-14T10:05:02.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.741+0000 7f47d8400700 1 -- 172.21.15.78:0/2178587846 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f47b80a0c00 msgr2=0x7f47b80a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:02.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.741+0000 7f47d8400700 1 --2- 172.21.15.78:0/2178587846 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f47b80a0c00 0x7f47b80a30c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f47c4005950 tx=0x7f47c400b410 comp rx=0 tx=0).stop 2024-03-14T10:05:02.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.741+0000 7f47d8400700 1 -- 172.21.15.78:0/2178587846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47d0106050 msgr2=0x7f47d019fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:02.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.741+0000 7f47d8400700 1 --2- 172.21.15.78:0/2178587846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47d0106050 0x7f47d019fb70 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f47bc00d950 tx=0x7f47bc00dc60 comp rx=0 tx=0).stop 2024-03-14T10:05:02.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.741+0000 7f47d8400700 1 -- 172.21.15.78:0/2178587846 shutdown_connections 2024-03-14T10:05:02.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.741+0000 7f47d8400700 1 --2- 172.21.15.78:0/2178587846 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f47b80a0c00 0x7f47b80a30c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.741+0000 7f47d8400700 1 --2- 172.21.15.78:0/2178587846 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47d0199760 0x7f47d0199c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.741+0000 7f47d8400700 1 --2- 172.21.15.78:0/2178587846 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f47d0100f30 0x7f47d019f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.741+0000 7f47d8400700 1 --2- 172.21.15.78:0/2178587846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47d0106050 0x7f47d019fb70 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.741+0000 7f47d8400700 1 -- 172.21.15.78:0/2178587846 >> 172.21.15.78:0/2178587846 conn(0x7f47d00fc7b0 msgr2=0x7f47d00fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:02.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.741+0000 7f47d8400700 1 -- 172.21.15.78:0/2178587846 shutdown_connections 2024-03-14T10:05:02.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.741+0000 7f47d8400700 1 -- 172.21.15.78:0/2178587846 wait complete. 2024-03-14T10:05:02.760 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-14T10:05:02.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.956+0000 7fec11600700 1 -- 172.21.15.78:0/3232995114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec0c100f20 msgr2=0x7fec0c101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:02.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.956+0000 7fec11600700 1 --2- 172.21.15.78:0/3232995114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec0c100f20 0x7fec0c101300 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7febfc009990 tx=0x7febfc009ca0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.956+0000 7fec11600700 1 -- 172.21.15.78:0/3232995114 shutdown_connections 2024-03-14T10:05:02.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.956+0000 7fec11600700 1 --2- 172.21.15.78:0/3232995114 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec0c106040 0x7fec0c10dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.956+0000 7fec11600700 1 --2- 172.21.15.78:0/3232995114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec0c100f20 0x7fec0c101300 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.956+0000 7fec11600700 1 --2- 172.21.15.78:0/3232995114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec0c1018d0 0x7fec0c105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.956+0000 7fec11600700 1 -- 172.21.15.78:0/3232995114 >> 172.21.15.78:0/3232995114 conn(0x7fec0c0fc7a0 msgr2=0x7fec0c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:02.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.957+0000 7fec11600700 1 -- 172.21.15.78:0/3232995114 shutdown_connections 2024-03-14T10:05:02.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.957+0000 7fec11600700 1 -- 172.21.15.78:0/3232995114 wait complete. 2024-03-14T10:05:02.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.958+0000 7fec11600700 1 Processor -- start 2024-03-14T10:05:02.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.958+0000 7fec11600700 1 -- start start 2024-03-14T10:05:02.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.958+0000 7fec11600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec0c100f20 0x7fec0c19f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:02.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.958+0000 7fec11600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec0c106040 0x7fec0c19faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:02.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.958+0000 7fec11600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec0c1996e0 0x7fec0c199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:02.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.958+0000 7fec11600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c19a160 con 0x7fec0c106040 2024-03-14T10:05:02.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.958+0000 7fec11600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c19a2d0 con 0x7fec0c1996e0 2024-03-14T10:05:02.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.958+0000 7fec11600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c19a490 con 0x7fec0c100f20 2024-03-14T10:05:02.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.958+0000 7fec0a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec0c106040 0x7fec0c19faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:02.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.958+0000 7fec0a000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec0c106040 0x7fec0c19faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:58474/0 (socket says 172.21.15.78:58474) 2024-03-14T10:05:02.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.958+0000 7fec0a000700 1 -- 172.21.15.78:0/1078299240 learned_addr learned my addr 172.21.15.78:0/1078299240 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:05:02.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.958+0000 7fec0aa00700 1 --2- 172.21.15.78:0/1078299240 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec0c100f20 0x7fec0c19f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:02.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.959+0000 7fec0b400700 1 --2- 172.21.15.78:0/1078299240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec0c1996e0 0x7fec0c199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:02.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.959+0000 7fec0a000700 1 -- 172.21.15.78:0/1078299240 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec0c100f20 msgr2=0x7fec0c19f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:02.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.959+0000 7fec0a000700 1 --2- 172.21.15.78:0/1078299240 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec0c100f20 0x7fec0c19f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.959+0000 7fec0a000700 1 -- 172.21.15.78:0/1078299240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec0c1996e0 msgr2=0x7fec0c199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:02.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.959+0000 7fec0a000700 1 --2- 172.21.15.78:0/1078299240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec0c1996e0 0x7fec0c199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:02.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.959+0000 7fec0a000700 1 -- 172.21.15.78:0/1078299240 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc0095f0 con 0x7fec0c106040 2024-03-14T10:05:02.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.959+0000 7fec0a000700 1 --2- 172.21.15.78:0/1078299240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec0c106040 0x7fec0c19faa0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7febf800d980 tx=0x7febf800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:02.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.959+0000 7fec03400700 1 -- 172.21.15.78:0/1078299240 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf800f890 con 0x7fec0c106040 2024-03-14T10:05:02.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.959+0000 7fec11600700 1 -- 172.21.15.78:0/1078299240 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec0c1a6430 con 0x7fec0c106040 2024-03-14T10:05:02.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.959+0000 7fec03400700 1 -- 172.21.15.78:0/1078299240 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7febf8010460 con 0x7fec0c106040 2024-03-14T10:05:02.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.959+0000 7fec03400700 1 -- 172.21.15.78:0/1078299240 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf800e610 con 0x7fec0c106040 2024-03-14T10:05:02.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.959+0000 7fec11600700 1 -- 172.21.15.78:0/1078299240 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec0c1a6960 con 0x7fec0c106040 2024-03-14T10:05:02.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.960+0000 7fec11600700 1 -- 172.21.15.78:0/1078299240 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec0c1023f0 con 0x7fec0c106040 2024-03-14T10:05:02.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.962+0000 7fec03400700 1 -- 172.21.15.78:0/1078299240 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7febf80105d0 con 0x7fec0c106040 2024-03-14T10:05:02.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.963+0000 7fec03400700 1 --2- 172.21.15.78:0/1078299240 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7febf00a0c00 0x7febf00a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:02.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.964+0000 7fec03400700 1 -- 172.21.15.78:0/1078299240 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7febf80cc260 con 0x7fec0c106040 2024-03-14T10:05:02.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.964+0000 7fec0aa00700 1 --2- 172.21.15.78:0/1078299240 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7febf00a0c00 0x7febf00a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:02.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.964+0000 7fec0aa00700 1 --2- 172.21.15.78:0/1078299240 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7febf00a0c00 0x7febf00a30c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7febfc00b2e0 tx=0x7febfc005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:02.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:02.968+0000 7fec03400700 1 -- 172.21.15.78:0/1078299240 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7febf80954c0 con 0x7fec0c106040 2024-03-14T10:05:03.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.189+0000 7fec11600700 1 -- 172.21.15.78:0/1078299240 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fec0c19abb0 con 0x7fec0c106040 2024-03-14T10:05:03.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.190+0000 7fec03400700 1 -- 172.21.15.78:0/1078299240 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v96) v1 ==== 103+0+11 (secure 0 0 0) 0x7febf8098ae0 con 0x7fec0c106040 2024-03-14T10:05:03.191 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2024-03-14T10:05:03.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.193+0000 7fec11600700 1 -- 172.21.15.78:0/1078299240 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7febf00a0c00 msgr2=0x7febf00a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:03.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.193+0000 7fec11600700 1 --2- 172.21.15.78:0/1078299240 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7febf00a0c00 0x7febf00a30c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7febfc00b2e0 tx=0x7febfc005fd0 comp rx=0 tx=0).stop 2024-03-14T10:05:03.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.193+0000 7fec11600700 1 -- 172.21.15.78:0/1078299240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec0c106040 msgr2=0x7fec0c19faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:03.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.193+0000 7fec11600700 1 --2- 172.21.15.78:0/1078299240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec0c106040 0x7fec0c19faa0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7febf800d980 tx=0x7febf800dc90 comp rx=0 tx=0).stop 2024-03-14T10:05:03.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.193+0000 7fec11600700 1 -- 172.21.15.78:0/1078299240 shutdown_connections 2024-03-14T10:05:03.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.193+0000 7fec11600700 1 --2- 172.21.15.78:0/1078299240 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7febf00a0c00 0x7febf00a30c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:03.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.193+0000 7fec11600700 1 --2- 172.21.15.78:0/1078299240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec0c1996e0 0x7fec0c199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:03.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.193+0000 7fec11600700 1 --2- 172.21.15.78:0/1078299240 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fec0c100f20 0x7fec0c19f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:03.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.193+0000 7fec11600700 1 --2- 172.21.15.78:0/1078299240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec0c106040 0x7fec0c19faa0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:03.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.193+0000 7fec11600700 1 -- 172.21.15.78:0/1078299240 >> 172.21.15.78:0/1078299240 conn(0x7fec0c0fc7a0 msgr2=0x7fec0c0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:03.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.193+0000 7fec11600700 1 -- 172.21.15.78:0/1078299240 shutdown_connections 2024-03-14T10:05:03.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.193+0000 7fec11600700 1 -- 172.21.15.78:0/1078299240 wait complete. 2024-03-14T10:05:03.213 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-14T10:05:03.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.407+0000 7fe872c00700 1 -- 172.21.15.78:0/3713516806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe86c106050 msgr2=0x7fe86c10dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:03.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.407+0000 7fe872c00700 1 --2- 172.21.15.78:0/3713516806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe86c106050 0x7fe86c10dbd0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe86400b0a0 tx=0x7fe86400b3b0 comp rx=0 tx=0).stop 2024-03-14T10:05:03.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.407+0000 7fe872c00700 1 -- 172.21.15.78:0/3713516806 shutdown_connections 2024-03-14T10:05:03.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.407+0000 7fe872c00700 1 --2- 172.21.15.78:0/3713516806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe86c106050 0x7fe86c10dbd0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:03.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.407+0000 7fe872c00700 1 --2- 172.21.15.78:0/3713516806 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe86c100f30 0x7fe86c101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:03.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.407+0000 7fe872c00700 1 --2- 172.21.15.78:0/3713516806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe86c1018e0 0x7fe86c105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:03.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.407+0000 7fe872c00700 1 -- 172.21.15.78:0/3713516806 >> 172.21.15.78:0/3713516806 conn(0x7fe86c0fc7b0 msgr2=0x7fe86c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:03.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.407+0000 7fe872c00700 1 -- 172.21.15.78:0/3713516806 shutdown_connections 2024-03-14T10:05:03.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.407+0000 7fe872c00700 1 -- 172.21.15.78:0/3713516806 wait complete. 2024-03-14T10:05:03.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.408+0000 7fe872c00700 1 Processor -- start 2024-03-14T10:05:03.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.408+0000 7fe872c00700 1 -- start start 2024-03-14T10:05:03.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.409+0000 7fe872c00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe86c100f30 0x7fe86c19f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:03.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.409+0000 7fe872c00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe86c106050 0x7fe86c19fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:03.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.409+0000 7fe872c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe86c199770 0x7fe86c199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:03.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.409+0000 7fe872c00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c19a1f0 con 0x7fe86c199770 2024-03-14T10:05:03.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.409+0000 7fe872c00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c19a360 con 0x7fe86c106050 2024-03-14T10:05:03.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.409+0000 7fe872c00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86c19a520 con 0x7fe86c100f30 2024-03-14T10:05:03.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.409+0000 7fe86be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe86c100f30 0x7fe86c19f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:03.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.409+0000 7fe86be00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe86c100f30 0x7fe86c19f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.152:3300/0 says I am v2:172.21.15.78:36034/0 (socket says 172.21.15.78:36034) 2024-03-14T10:05:03.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.409+0000 7fe870c00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe86c199770 0x7fe86c199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:03.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.409+0000 7fe86be00700 1 -- 172.21.15.78:0/875066813 learned_addr learned my addr 172.21.15.78:0/875066813 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:05:03.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.409+0000 7fe863e00700 1 --2- 172.21.15.78:0/875066813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe86c106050 0x7fe86c19fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:03.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.409+0000 7fe870c00700 1 -- 172.21.15.78:0/875066813 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe86c100f30 msgr2=0x7fe86c19f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:03.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.409+0000 7fe870c00700 1 --2- 172.21.15.78:0/875066813 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe86c100f30 0x7fe86c19f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:03.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.409+0000 7fe870c00700 1 -- 172.21.15.78:0/875066813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe86c106050 msgr2=0x7fe86c19fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:03.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.409+0000 7fe870c00700 1 --2- 172.21.15.78:0/875066813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe86c106050 0x7fe86c19fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:03.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.409+0000 7fe870c00700 1 -- 172.21.15.78:0/875066813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe864009d00 con 0x7fe86c199770 2024-03-14T10:05:03.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.410+0000 7fe870c00700 1 --2- 172.21.15.78:0/875066813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe86c199770 0x7fe86c199c20 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe86400b070 tx=0x7fe86400bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:03.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.410+0000 7fe869600700 1 -- 172.21.15.78:0/875066813 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86400e040 con 0x7fe86c199770 2024-03-14T10:05:03.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.410+0000 7fe869600700 1 -- 172.21.15.78:0/875066813 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe86401d480 con 0x7fe86c199770 2024-03-14T10:05:03.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.410+0000 7fe872c00700 1 -- 172.21.15.78:0/875066813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe86c1a64b0 con 0x7fe86c199770 2024-03-14T10:05:03.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.410+0000 7fe869600700 1 -- 172.21.15.78:0/875066813 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8640052a0 con 0x7fe86c199770 2024-03-14T10:05:03.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.410+0000 7fe872c00700 1 -- 172.21.15.78:0/875066813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe86c1a68d0 con 0x7fe86c199770 2024-03-14T10:05:03.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.411+0000 7fe872c00700 1 -- 172.21.15.78:0/875066813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe86c102400 con 0x7fe86c199770 2024-03-14T10:05:03.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.413+0000 7fe869600700 1 -- 172.21.15.78:0/875066813 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fe86401d5f0 con 0x7fe86c199770 2024-03-14T10:05:03.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.414+0000 7fe869600700 1 --2- 172.21.15.78:0/875066813 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe8500a0c80 0x7fe8500a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:03.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.414+0000 7fe869600700 1 -- 172.21.15.78:0/875066813 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fe8640cd710 con 0x7fe86c199770 2024-03-14T10:05:03.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.414+0000 7fe86be00700 1 --2- 172.21.15.78:0/875066813 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe8500a0c80 0x7fe8500a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:03.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.415+0000 7fe86be00700 1 --2- 172.21.15.78:0/875066813 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe8500a0c80 0x7fe8500a3140 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009d20 tx=0x7fe85c009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:03.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.418+0000 7fe869600700 1 -- 172.21.15.78:0/875066813 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe864096a00 con 0x7fe86c199770 2024-03-14T10:05:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:03 smithi152 ceph-mon[48279]: pgmap v337: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-14T10:05:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:03 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2496982785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T10:05:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:03 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2178587846' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T10:05:03.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:03 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1078299240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T10:05:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:03 smithi132 ceph-mon[47959]: pgmap v337: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-14T10:05:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:03 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2496982785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T10:05:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:03 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2178587846' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T10:05:03.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:03 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1078299240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T10:05:03.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:03 smithi078 ceph-mon[38143]: pgmap v337: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-14T10:05:03.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:03 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2496982785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-14T10:05:03.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:03 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2178587846' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-14T10:05:03.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:03 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1078299240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-14T10:05:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.658+0000 7fe872c00700 1 -- 172.21.15.78:0/875066813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe86c067ac0 con 0x7fe86c199770 2024-03-14T10:05:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.659+0000 7fe869600700 1 -- 172.21.15.78:0/875066813 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v96) v1 ==== 81+0+1002 (secure 0 0 0) 0x7fe86409a020 con 0x7fe86c199770 2024-03-14T10:05:03.659 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1935098096 2024-03-15T09:54:09.924780+0000 2024-03-14T10:05:03.660 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/4237457492 2024-03-15T09:46:40.988022+0000 2024-03-14T10:05:03.660 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2472038902 2024-03-15T09:46:17.100896+0000 2024-03-14T10:05:03.660 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2380275837 2024-03-15T09:46:40.988022+0000 2024-03-14T10:05:03.660 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/2580616009 2024-03-15T09:54:09.924780+0000 2024-03-14T10:05:03.660 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1127810454 2024-03-15T09:46:17.100896+0000 2024-03-14T10:05:03.660 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/1325295736 2024-03-15T09:46:17.100896+0000 2024-03-14T10:05:03.660 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/119757382 2024-03-15T09:46:17.100896+0000 2024-03-14T10:05:03.660 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/3308792589 2024-03-15T09:54:09.924780+0000 2024-03-14T10:05:03.660 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/1325295736 2024-03-15T09:46:17.100896+0000 2024-03-14T10:05:03.660 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2260754576 2024-03-15T09:46:40.988022+0000 2024-03-14T10:05:03.660 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/555376800 2024-03-15T09:54:09.924780+0000 2024-03-14T10:05:03.660 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/3890232539 2024-03-15T09:46:40.988022+0000 2024-03-14T10:05:03.660 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/3890232539 2024-03-15T09:46:40.988022+0000 2024-03-14T10:05:03.660 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/3220727810 2024-03-15T09:54:09.924780+0000 2024-03-14T10:05:03.660 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2934089773 2024-03-15T09:54:09.924780+0000 2024-03-14T10:05:03.661 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/2580616009 2024-03-15T09:54:09.924780+0000 2024-03-14T10:05:03.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.661+0000 7fe872c00700 1 -- 172.21.15.78:0/875066813 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe8500a0c80 msgr2=0x7fe8500a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:03.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.661+0000 7fe872c00700 1 --2- 172.21.15.78:0/875066813 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe8500a0c80 0x7fe8500a3140 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009d20 tx=0x7fe85c009260 comp rx=0 tx=0).stop 2024-03-14T10:05:03.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.661+0000 7fe872c00700 1 -- 172.21.15.78:0/875066813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe86c199770 msgr2=0x7fe86c199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:03.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.661+0000 7fe872c00700 1 --2- 172.21.15.78:0/875066813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe86c199770 0x7fe86c199c20 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe86400b070 tx=0x7fe86400bcf0 comp rx=0 tx=0).stop 2024-03-14T10:05:03.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.661+0000 7fe872c00700 1 -- 172.21.15.78:0/875066813 shutdown_connections 2024-03-14T10:05:03.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.661+0000 7fe872c00700 1 --2- 172.21.15.78:0/875066813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe86c199770 0x7fe86c199c20 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.661+0000 7fe872c00700 1 --2- 172.21.15.78:0/875066813 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fe86c100f30 0x7fe86c19f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.661+0000 7fe872c00700 1 --2- 172.21.15.78:0/875066813 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fe8500a0c80 0x7fe8500a3140 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.661+0000 7fe872c00700 1 --2- 172.21.15.78:0/875066813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe86c106050 0x7fe86c19fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.661+0000 7fe872c00700 1 -- 172.21.15.78:0/875066813 >> 172.21.15.78:0/875066813 conn(0x7fe86c0fc7b0 msgr2=0x7fe86c10d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.661+0000 7fe872c00700 1 -- 172.21.15.78:0/875066813 shutdown_connections 2024-03-14T10:05:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:03.661+0000 7fe872c00700 1 -- 172.21.15.78:0/875066813 wait complete. 2024-03-14T10:05:03.663 INFO:teuthology.orchestra.run.smithi078.stderr:listed 17 entries 2024-03-14T10:05:03.681 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-14T10:05:03.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T10:05:03.681 DEBUG:teuthology.orchestra.run.smithi078:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-14T10:05:03.771 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2024-03-14T10:05:03.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T10:05:03.771 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-14T10:05:03.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T10:05:03.827 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-03-14T10:05:03.868 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-14T10:05:03.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T10:05:03.869 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-14T10:05:03.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T10:05:03.978 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2024-03-14T10:05:04.078 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-14T10:05:04.079 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-03-14T10:05:04.079 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-14T10:05:04.125 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16345604k,nr_inodes=4086401,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275376k,mode=700,uid=1000,gid=1304 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/c6568732149f5e3d8d8e3def7a7d63e170b9f9f85754d0f769486d3c248b1190/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/DNKOGY5XFVJLKSN5VXO7DYJWMY:/var/lib/containers/storage/overlay/l/CBDD535RENOYXP6T7IHCOXXE65,upperdir=/var/lib/containers/storage/overlay/c6568732149f5e3d8d8e3def7a7d63e170b9f9f85754d0f769486d3c248b1190/diff,workdir=/var/lib/containers/storage/overlay/c6568732149f5e3d8d8e3def7a7d63e170b9f9f85754d0f769486d3c248b1190/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/0fe4ec1ef2653d1120b4930fb4d381cf9189b7052d6e3101a408f413c75e4908/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/DNKOGY5XFVJLKSN5VXO7DYJWMY:/var/lib/containers/storage/overlay/l/CBDD535RENOYXP6T7IHCOXXE65,upperdir=/var/lib/containers/storage/overlay/0fe4ec1ef2653d1120b4930fb4d381cf9189b7052d6e3101a408f413c75e4908/diff,workdir=/var/lib/containers/storage/overlay/0fe4ec1ef2653d1120b4930fb4d381cf9189b7052d6e3101a408f413c75e4908/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275376k,mode=700 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0 overlay /var/lib/containers/storage/overlay/61a6e0d94e55a7c8eee4434c6b3cf789a51334336ab197c46e9d5b526ae7835f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/DNKOGY5XFVJLKSN5VXO7DYJWMY:/var/lib/containers/storage/overlay/l/CBDD535RENOYXP6T7IHCOXXE65,upperdir=/var/lib/containers/storage/overlay/61a6e0d94e55a7c8eee4434c6b3cf789a51334336ab197c46e9d5b526ae7835f/diff,workdir=/var/lib/containers/storage/overlay/61a6e0d94e55a7c8eee4434c6b3cf789a51334336ab197c46e9d5b526ae7835f/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/a6057b1179f745256c12e8ef0f761bd4c6e542c9db421bec42b416bf33e0d60a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/DNKOGY5XFVJLKSN5VXO7DYJWMY:/var/lib/containers/storage/overlay/l/CBDD535RENOYXP6T7IHCOXXE65,upperdir=/var/lib/containers/storage/overlay/a6057b1179f745256c12e8ef0f761bd4c6e542c9db421bec42b416bf33e0d60a/diff,workdir=/var/lib/containers/storage/overlay/a6057b1179f745256c12e8ef0f761bd4c6e542c9db421bec42b416bf33e0d60a/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/409e0703a808127b9f3cab604d608c3edad7054d453f0b2a30bc8e7c8abc1084/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/DNKOGY5XFVJLKSN5VXO7DYJWMY:/var/lib/containers/storage/overlay/l/CBDD535RENOYXP6T7IHCOXXE65,upperdir=/var/lib/containers/storage/overlay/409e0703a808127b9f3cab604d608c3edad7054d453f0b2a30bc8e7c8abc1084/diff,workdir=/var/lib/containers/storage/overlay/409e0703a808127b9f3cab604d608c3edad7054d453f0b2a30bc8e7c8abc1084/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/261e74edc50edf4dfbdf46be660a65d705f844761a59f63af64fa2c44d855ba6/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/DNKOGY5XFVJLKSN5VXO7DYJWMY:/var/lib/containers/storage/overlay/l/CBDD535RENOYXP6T7IHCOXXE65,upperdir=/var/lib/containers/storage/overlay/261e74edc50edf4dfbdf46be660a65d705f844761a59f63af64fa2c44d855ba6/diff,workdir=/var/lib/containers/storage/overlay/261e74edc50edf4dfbdf46be660a65d705f844761a59f63af64fa2c44d855ba6/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b9ea6932640d67b5364d0a4b3de8f9a6918f9ad55fd42a83caf6f9866e64897c/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/DNKOGY5XFVJLKSN5VXO7DYJWMY:/var/lib/containers/storage/overlay/l/CBDD535RENOYXP6T7IHCOXXE65,upperdir=/var/lib/containers/storage/overlay/b9ea6932640d67b5364d0a4b3de8f9a6918f9ad55fd42a83caf6f9866e64897c/diff,workdir=/var/lib/containers/storage/overlay/b9ea6932640d67b5364d0a4b3de8f9a6918f9ad55fd42a83caf6f9866e64897c/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/6d7cdf1b201493965a51043ce3c052f586152e758987dd480c7de576bb907474/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/DNKOGY5XFVJLKSN5VXO7DYJWMY:/var/lib/containers/storage/overlay/l/CBDD535RENOYXP6T7IHCOXXE65,upperdir=/var/lib/containers/storage/overlay/6d7cdf1b201493965a51043ce3c052f586152e758987dd480c7de576bb907474/diff,workdir=/var/lib/containers/storage/overlay/6d7cdf1b201493965a51043ce3c052f586152e758987dd480c7de576bb907474/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/7cb54caaddff035c8381db6ef5cdb3f6b39d03ec3adb4906151524d798539a37/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/DNKOGY5XFVJLKSN5VXO7DYJWMY:/var/lib/containers/storage/overlay/l/CBDD535RENOYXP6T7IHCOXXE65,upperdir=/var/lib/containers/storage/overlay/7cb54caaddff035c8381db6ef5cdb3f6b39d03ec3adb4906151524d798539a37/diff,workdir=/var/lib/containers/storage/overlay/7cb54caaddff035c8381db6ef5cdb3f6b39d03ec3adb4906151524d798539a37/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/1a4e18d0c11d197abd1ea55c2363b2ad58cf2fee9fc5aa192f22967d2d8b79ac/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/DNKOGY5XFVJLKSN5VXO7DYJWMY:/var/lib/containers/storage/overlay/l/CBDD535RENOYXP6T7IHCOXXE65,upperdir=/var/lib/containers/storage/overlay/1a4e18d0c11d197abd1ea55c2363b2ad58cf2fee9fc5aa192f22967d2d8b79ac/diff,workdir=/var/lib/containers/storage/overlay/1a4e18d0c11d197abd1ea55c2363b2ad58cf2fee9fc5aa192f22967d2d8b79ac/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 2024-03-14T10:05:04.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T10:05:04.126 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-03-14T10:05:04.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T10:05:04.210 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2024-03-14T10:05:04.303 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:04 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/875066813' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-14T10:05:04.320 INFO:teuthology.orchestra.run.smithi078.stderr:Cannot find device "ceph-brx" 2024-03-14T10:05:04.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T10:05:04.324 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-14T10:05:04.336 INFO:tasks.cephadm:Teardown begin 2024-03-14T10:05:04.336 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-14T10:05:04.336 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T10:05:04.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:04 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/875066813' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-14T10:05:04.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:04 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/875066813' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-14T10:05:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.530+0000 7f16f6400700 1 -- 172.21.15.78:0/1961342828 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16f0100e20 msgr2=0x7f16f0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.530+0000 7f16f6400700 1 --2- 172.21.15.78:0/1961342828 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16f0100e20 0x7f16f0101200 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009990 tx=0x7f16e0009ca0 comp rx=0 tx=0).stop 2024-03-14T10:05:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.530+0000 7f16f6400700 1 -- 172.21.15.78:0/1961342828 shutdown_connections 2024-03-14T10:05:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.530+0000 7f16f6400700 1 --2- 172.21.15.78:0/1961342828 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16f0100e20 0x7f16f0101200 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.530+0000 7f16f6400700 1 --2- 172.21.15.78:0/1961342828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f0105f40 0x7f16f010dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.530+0000 7f16f6400700 1 --2- 172.21.15.78:0/1961342828 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16f01017d0 0x7f16f0105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.530+0000 7f16f6400700 1 -- 172.21.15.78:0/1961342828 >> 172.21.15.78:0/1961342828 conn(0x7f16f00fc700 msgr2=0x7f16f00feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.530+0000 7f16f6400700 1 -- 172.21.15.78:0/1961342828 shutdown_connections 2024-03-14T10:05:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.530+0000 7f16f6400700 1 -- 172.21.15.78:0/1961342828 wait complete. 2024-03-14T10:05:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.531+0000 7f16f6400700 1 Processor -- start 2024-03-14T10:05:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.531+0000 7f16f6400700 1 -- start start 2024-03-14T10:05:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16f6400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16f0100e20 0x7f16f0199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16f6400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16f0105f40 0x7f16f0199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16f6400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f019e010 0x7f16f019e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16f6400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f019ea90 con 0x7f16f019e010 2024-03-14T10:05:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16f6400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f019ec00 con 0x7f16f0100e20 2024-03-14T10:05:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16f6400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f019edc0 con 0x7f16f0105f40 2024-03-14T10:05:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16ef400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16f0100e20 0x7f16f0199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16e7e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16f0105f40 0x7f16f0199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16efe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f019e010 0x7f16f019e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16ef400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16f0100e20 0x7f16f0199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.78:40084/0 (socket says 172.21.15.78:40084) 2024-03-14T10:05:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16ef400700 1 -- 172.21.15.78:0/1520055557 learned_addr learned my addr 172.21.15.78:0/1520055557 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:05:04.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16ef400700 1 -- 172.21.15.78:0/1520055557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16f0105f40 msgr2=0x7f16f0199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16ef400700 1 --2- 172.21.15.78:0/1520055557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16f0105f40 0x7f16f0199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16ef400700 1 -- 172.21.15.78:0/1520055557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f019e010 msgr2=0x7f16f019e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16ef400700 1 --2- 172.21.15.78:0/1520055557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f019e010 0x7f16f019e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16ef400700 1 -- 172.21.15.78:0/1520055557 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e00095f0 con 0x7f16f0100e20 2024-03-14T10:05:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16e7e00700 1 --2- 172.21.15.78:0/1520055557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16f0105f40 0x7f16f0199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:05:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.532+0000 7f16efe00700 1 --2- 172.21.15.78:0/1520055557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f019e010 0x7f16f019e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:05:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.533+0000 7f16ef400700 1 --2- 172.21.15.78:0/1520055557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16f0100e20 0x7f16f0199730 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f16e0000c00 tx=0x7f16e0005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.533+0000 7f16ecc00700 1 -- 172.21.15.78:0/1520055557 <== mon.2 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e001d070 con 0x7f16f0100e20 2024-03-14T10:05:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.533+0000 7f16ecc00700 1 -- 172.21.15.78:0/1520055557 <== mon.2 v2:172.21.15.132:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16e0022470 con 0x7f16f0100e20 2024-03-14T10:05:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.533+0000 7f16f6400700 1 -- 172.21.15.78:0/1520055557 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f01a62c0 con 0x7f16f0100e20 2024-03-14T10:05:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.533+0000 7f16ecc00700 1 -- 172.21.15.78:0/1520055557 <== mon.2 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e0017650 con 0x7f16f0100e20 2024-03-14T10:05:04.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.533+0000 7f16f6400700 1 -- 172.21.15.78:0/1520055557 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f01a6740 con 0x7f16f0100e20 2024-03-14T10:05:04.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.534+0000 7f16f6400700 1 -- 172.21.15.78:0/1520055557 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f01022f0 con 0x7f16f0100e20 2024-03-14T10:05:04.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.536+0000 7f16ecc00700 1 -- 172.21.15.78:0/1520055557 <== mon.2 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f16e00225e0 con 0x7f16f0100e20 2024-03-14T10:05:04.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.537+0000 7f16ecc00700 1 --2- 172.21.15.78:0/1520055557 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f16d40a0c00 0x7f16d40a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:04.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.537+0000 7f16ecc00700 1 -- 172.21.15.78:0/1520055557 <== mon.2 v2:172.21.15.132:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f16e00ce180 con 0x7f16f0100e20 2024-03-14T10:05:04.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.538+0000 7f16e7e00700 1 --2- 172.21.15.78:0/1520055557 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f16d40a0c00 0x7f16d40a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:04.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.538+0000 7f16e7e00700 1 --2- 172.21.15.78:0/1520055557 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f16d40a0c00 0x7f16d40a30c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f16f019ac80 tx=0x7f16d8009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:04.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.540+0000 7f16ecc00700 1 -- 172.21.15.78:0/1520055557 <== mon.2 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f16e0097470 con 0x7f16f0100e20 2024-03-14T10:05:04.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.792+0000 7f16f6400700 1 -- 172.21.15.78:0/1520055557 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16f00679e0 con 0x7f16f0100e20 2024-03-14T10:05:04.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.793+0000 7f16ecc00700 1 -- 172.21.15.78:0/1520055557 <== mon.2 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7f16e009aa90 con 0x7f16f0100e20 2024-03-14T10:05:04.794 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T10:05:04.794 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T10:05:04.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.796+0000 7f16f6400700 1 -- 172.21.15.78:0/1520055557 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f16d40a0c00 msgr2=0x7f16d40a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:04.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.796+0000 7f16f6400700 1 --2- 172.21.15.78:0/1520055557 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f16d40a0c00 0x7f16d40a30c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f16f019ac80 tx=0x7f16d8009260 comp rx=0 tx=0).stop 2024-03-14T10:05:04.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.796+0000 7f16f6400700 1 -- 172.21.15.78:0/1520055557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16f0100e20 msgr2=0x7f16f0199730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:04.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.796+0000 7f16f6400700 1 --2- 172.21.15.78:0/1520055557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16f0100e20 0x7f16f0199730 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f16e0000c00 tx=0x7f16e0005740 comp rx=0 tx=0).stop 2024-03-14T10:05:04.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.796+0000 7f16f6400700 1 -- 172.21.15.78:0/1520055557 shutdown_connections 2024-03-14T10:05:04.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.796+0000 7f16f6400700 1 --2- 172.21.15.78:0/1520055557 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f16d40a0c00 0x7f16d40a30c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:04.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.796+0000 7f16f6400700 1 --2- 172.21.15.78:0/1520055557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16f0100e20 0x7f16f0199730 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:04.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.796+0000 7f16f6400700 1 --2- 172.21.15.78:0/1520055557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16f019e010 0x7f16f019e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:04.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.796+0000 7f16f6400700 1 --2- 172.21.15.78:0/1520055557 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f16f0105f40 0x7f16f0199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:04.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.796+0000 7f16f6400700 1 -- 172.21.15.78:0/1520055557 >> 172.21.15.78:0/1520055557 conn(0x7f16f00fc700 msgr2=0x7f16f00fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:04.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.796+0000 7f16f6400700 1 -- 172.21.15.78:0/1520055557 shutdown_connections 2024-03-14T10:05:04.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:04.796+0000 7f16f6400700 1 -- 172.21.15.78:0/1520055557 wait complete. 2024-03-14T10:05:04.798 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T10:05:04.816 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T10:05:05.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.009+0000 7efeb3800700 1 -- 172.21.15.78:0/507808685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeac100e20 msgr2=0x7efeac101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:05.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.009+0000 7efeb3800700 1 --2- 172.21.15.78:0/507808685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeac100e20 0x7efeac101200 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7efea0009990 tx=0x7efea0009ca0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.009+0000 7efeb3800700 1 -- 172.21.15.78:0/507808685 shutdown_connections 2024-03-14T10:05:05.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.009+0000 7efeb3800700 1 --2- 172.21.15.78:0/507808685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efeac105f40 0x7efeac10dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.009+0000 7efeb3800700 1 --2- 172.21.15.78:0/507808685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeac100e20 0x7efeac101200 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.009+0000 7efeb3800700 1 --2- 172.21.15.78:0/507808685 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efeac1017d0 0x7efeac105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.009+0000 7efeb3800700 1 -- 172.21.15.78:0/507808685 >> 172.21.15.78:0/507808685 conn(0x7efeac0fc700 msgr2=0x7efeac0feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:05.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.009+0000 7efeb3800700 1 -- 172.21.15.78:0/507808685 shutdown_connections 2024-03-14T10:05:05.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.009+0000 7efeb3800700 1 -- 172.21.15.78:0/507808685 wait complete. 2024-03-14T10:05:05.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.010+0000 7efeb3800700 1 Processor -- start 2024-03-14T10:05:05.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.010+0000 7efeb3800700 1 -- start start 2024-03-14T10:05:05.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.010+0000 7efeb3800700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeac100e20 0x7efeac0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:05.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.010+0000 7efeb3800700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efeac105f40 0x7efeac0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:05.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.010+0000 7efeb3800700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efeac074d20 0x7efeac073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:05.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.010+0000 7efeb3800700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac1aa180 con 0x7efeac100e20 2024-03-14T10:05:05.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.010+0000 7efeb3800700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac1aa2f0 con 0x7efeac105f40 2024-03-14T10:05:05.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.010+0000 7efeb3800700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac1aa4b0 con 0x7efeac074d20 2024-03-14T10:05:05.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.011+0000 7efeb0e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeac100e20 0x7efeac0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:05.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.011+0000 7efeb0e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeac100e20 0x7efeac0730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41116/0 (socket says 172.21.15.78:41116) 2024-03-14T10:05:05.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.011+0000 7efeb0e00700 1 -- 172.21.15.78:0/677278494 learned_addr learned my addr 172.21.15.78:0/677278494 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:05:05.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.011+0000 7efeabe00700 1 --2- 172.21.15.78:0/677278494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efeac105f40 0x7efeac0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:05.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.011+0000 7efeb1800700 1 --2- 172.21.15.78:0/677278494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efeac074d20 0x7efeac073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:05.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.011+0000 7efeb0e00700 1 -- 172.21.15.78:0/677278494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efeac074d20 msgr2=0x7efeac073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:05.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.011+0000 7efeb0e00700 1 --2- 172.21.15.78:0/677278494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efeac074d20 0x7efeac073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.011+0000 7efeb0e00700 1 -- 172.21.15.78:0/677278494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efeac105f40 msgr2=0x7efeac0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:05.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.011+0000 7efeb0e00700 1 --2- 172.21.15.78:0/677278494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efeac105f40 0x7efeac0735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.011+0000 7efeb0e00700 1 -- 172.21.15.78:0/677278494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea00095f0 con 0x7efeac100e20 2024-03-14T10:05:05.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.011+0000 7efeb1800700 1 --2- 172.21.15.78:0/677278494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efeac074d20 0x7efeac073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:05:05.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.011+0000 7efeabe00700 1 --2- 172.21.15.78:0/677278494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efeac105f40 0x7efeac0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:05:05.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.011+0000 7efeb0e00700 1 --2- 172.21.15.78:0/677278494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeac100e20 0x7efeac0730b0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7efea0005910 tx=0x7efea0009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:05.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.011+0000 7efea9600700 1 -- 172.21.15.78:0/677278494 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea001d070 con 0x7efeac100e20 2024-03-14T10:05:05.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.011+0000 7efeb3800700 1 -- 172.21.15.78:0/677278494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeac1aa730 con 0x7efeac100e20 2024-03-14T10:05:05.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.011+0000 7efea9600700 1 -- 172.21.15.78:0/677278494 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efea0022470 con 0x7efeac100e20 2024-03-14T10:05:05.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.012+0000 7efeb3800700 1 -- 172.21.15.78:0/677278494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeac1aab60 con 0x7efeac100e20 2024-03-14T10:05:05.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.012+0000 7efea9600700 1 -- 172.21.15.78:0/677278494 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea00175f0 con 0x7efeac100e20 2024-03-14T10:05:05.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.013+0000 7efeb3800700 1 -- 172.21.15.78:0/677278494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeac109160 con 0x7efeac100e20 2024-03-14T10:05:05.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.014+0000 7efea9600700 1 -- 172.21.15.78:0/677278494 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7efea00225e0 con 0x7efeac100e20 2024-03-14T10:05:05.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.016+0000 7efea9600700 1 --2- 172.21.15.78:0/677278494 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efe940a0c00 0x7efe940a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:05.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.016+0000 7efea9600700 1 -- 172.21.15.78:0/677278494 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7efea00cda80 con 0x7efeac100e20 2024-03-14T10:05:05.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.016+0000 7efeabe00700 1 --2- 172.21.15.78:0/677278494 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efe940a0c00 0x7efe940a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:05.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.016+0000 7efeabe00700 1 --2- 172.21.15.78:0/677278494 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efe940a0c00 0x7efe940a30c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efeac074790 tx=0x7efe98009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:05.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.019+0000 7efea9600700 1 -- 172.21.15.78:0/677278494 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7efea005d960 con 0x7efeac100e20 2024-03-14T10:05:05.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.273+0000 7efeb3800700 1 -- 172.21.15.78:0/677278494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efeac1022f0 con 0x7efeac100e20 2024-03-14T10:05:05.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.274+0000 7efea9600700 1 -- 172.21.15.78:0/677278494 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10870 (secure 0 0 0) 0x7efea0096d70 con 0x7efeac100e20 2024-03-14T10:05:05.274 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T10:05:05.275 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14454,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/996011838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":996011838},{"type":"v1","addr":"172.21.15.78:6835","nonce":996011838}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14466,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/176895986","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":176895986},{"type":"v1","addr":"172.21.15.78:6837","nonce":176895986}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14532,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1109060843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1109060843},{"type":"v1","addr":"172.21.15.78:6841","nonce":1109060843}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6833/4187874184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6832","nonce":4187874184},{"type":"v1","addr":"172.21.15.152:6833","nonce":4187874184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24479,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/726198552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":726198552},{"type":"v1","addr":"172.21.15.132:6833","nonce":726198552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.152:6835/2986677339","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6834","nonce":2986677339},{"type":"v1","addr":"172.21.15.152:6835","nonce":2986677339}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T09:52:05.708711+0000","modified":"2024-03-14T09:53:50.201933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24572,"mds_1":24538,"mds_2":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6837/2123600730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6836","nonce":2123600730},{"type":"v1","addr":"172.21.15.152:6837","nonce":2123600730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6835/1709886688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1709886688},{"type":"v1","addr":"172.21.15.132:6835","nonce":1709886688}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24526":{"gid":24526,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.152:6839/1965809497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6838","nonce":1965809497},{"type":"v1","addr":"172.21.15.152:6839","nonce":1965809497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24527":{"gid":24527,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.132:6837/142126094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":142126094},{"type":"v1","addr":"172.21.15.132:6837","nonce":142126094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24538":{"gid":24538,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.78:6839/4227580185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":4227580185},{"type":"v1","addr":"172.21.15.78:6839","nonce":4227580185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.132:6839/2192113559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":2192113559},{"type":"v1","addr":"172.21.15.132:6839","nonce":2192113559}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-14T10:05:05.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.276+0000 7efeb3800700 1 -- 172.21.15.78:0/677278494 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efe940a0c00 msgr2=0x7efe940a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:05.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.276+0000 7efeb3800700 1 --2- 172.21.15.78:0/677278494 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efe940a0c00 0x7efe940a30c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efeac074790 tx=0x7efe98009260 comp rx=0 tx=0).stop 2024-03-14T10:05:05.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.276+0000 7efeb3800700 1 -- 172.21.15.78:0/677278494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeac100e20 msgr2=0x7efeac0730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:05.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.276+0000 7efeb3800700 1 --2- 172.21.15.78:0/677278494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeac100e20 0x7efeac0730b0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7efea0005910 tx=0x7efea0009570 comp rx=0 tx=0).stop 2024-03-14T10:05:05.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.276+0000 7efeb3800700 1 -- 172.21.15.78:0/677278494 shutdown_connections 2024-03-14T10:05:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.276+0000 7efeb3800700 1 --2- 172.21.15.78:0/677278494 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7efeac074d20 0x7efeac073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.276+0000 7efeb3800700 1 --2- 172.21.15.78:0/677278494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efeac100e20 0x7efeac0730b0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.276+0000 7efeb3800700 1 --2- 172.21.15.78:0/677278494 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7efe940a0c00 0x7efe940a30c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.276+0000 7efeb3800700 1 --2- 172.21.15.78:0/677278494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efeac105f40 0x7efeac0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.276+0000 7efeb3800700 1 -- 172.21.15.78:0/677278494 >> 172.21.15.78:0/677278494 conn(0x7efeac0fc700 msgr2=0x7efeac0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.277+0000 7efeb3800700 1 -- 172.21.15.78:0/677278494 shutdown_connections 2024-03-14T10:05:05.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.277+0000 7efeb3800700 1 -- 172.21.15.78:0/677278494 wait complete. 2024-03-14T10:05:05.279 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 25 2024-03-14T10:05:05.296 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T10:05:05.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.487+0000 7fdf5ae00700 1 -- 172.21.15.78:0/4207571210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf54100e20 msgr2=0x7fdf54101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:05.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.487+0000 7fdf5ae00700 1 --2- 172.21.15.78:0/4207571210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf54100e20 0x7fdf54101200 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fdf44009990 tx=0x7fdf44009ca0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.487+0000 7fdf5ae00700 1 -- 172.21.15.78:0/4207571210 shutdown_connections 2024-03-14T10:05:05.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.487+0000 7fdf5ae00700 1 --2- 172.21.15.78:0/4207571210 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf54105f40 0x7fdf5410dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.487+0000 7fdf5ae00700 1 --2- 172.21.15.78:0/4207571210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf54100e20 0x7fdf54101200 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.487+0000 7fdf5ae00700 1 --2- 172.21.15.78:0/4207571210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf541017d0 0x7fdf54105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.487+0000 7fdf5ae00700 1 -- 172.21.15.78:0/4207571210 >> 172.21.15.78:0/4207571210 conn(0x7fdf540fc700 msgr2=0x7fdf540feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:05.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.488+0000 7fdf5ae00700 1 -- 172.21.15.78:0/4207571210 shutdown_connections 2024-03-14T10:05:05.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.488+0000 7fdf5ae00700 1 -- 172.21.15.78:0/4207571210 wait complete. 2024-03-14T10:05:05.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.489+0000 7fdf5ae00700 1 Processor -- start 2024-03-14T10:05:05.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.489+0000 7fdf5ae00700 1 -- start start 2024-03-14T10:05:05.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.489+0000 7fdf5ae00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf54100e20 0x7fdf5419f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:05.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.489+0000 7fdf5ae00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf54105f40 0x7fdf5419faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:05.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.489+0000 7fdf5ae00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf54199690 0x7fdf54199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:05.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.489+0000 7fdf5ae00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5419a110 con 0x7fdf54105f40 2024-03-14T10:05:05.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.489+0000 7fdf5ae00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5419a280 con 0x7fdf54100e20 2024-03-14T10:05:05.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.489+0000 7fdf5ae00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5419a440 con 0x7fdf54199690 2024-03-14T10:05:05.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.489+0000 7fdf53400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf54105f40 0x7fdf5419faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:05.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.489+0000 7fdf53e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf54100e20 0x7fdf5419f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:05.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.489+0000 7fdf53400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf54105f40 0x7fdf5419faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41136/0 (socket says 172.21.15.78:41136) 2024-03-14T10:05:05.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.489+0000 7fdf53400700 1 -- 172.21.15.78:0/1028509952 learned_addr learned my addr 172.21.15.78:0/1028509952 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:05:05.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.490+0000 7fdf58e00700 1 --2- 172.21.15.78:0/1028509952 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf54199690 0x7fdf54199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:05.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.490+0000 7fdf53400700 1 -- 172.21.15.78:0/1028509952 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf54199690 msgr2=0x7fdf54199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:05.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.490+0000 7fdf53400700 1 --2- 172.21.15.78:0/1028509952 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf54199690 0x7fdf54199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.490+0000 7fdf53400700 1 -- 172.21.15.78:0/1028509952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf54100e20 msgr2=0x7fdf5419f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:05.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.490+0000 7fdf53400700 1 --2- 172.21.15.78:0/1028509952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf54100e20 0x7fdf5419f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.490+0000 7fdf53400700 1 -- 172.21.15.78:0/1028509952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf440095f0 con 0x7fdf54105f40 2024-03-14T10:05:05.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.490+0000 7fdf58e00700 1 --2- 172.21.15.78:0/1028509952 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf54199690 0x7fdf54199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:05:05.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.490+0000 7fdf53e00700 1 --2- 172.21.15.78:0/1028509952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf54100e20 0x7fdf5419f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:05:05.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.490+0000 7fdf53400700 1 --2- 172.21.15.78:0/1028509952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf54105f40 0x7fdf5419faa0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdf4000e970 tx=0x7fdf4000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:05.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.490+0000 7fdf50c00700 1 -- 172.21.15.78:0/1028509952 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf40009790 con 0x7fdf54105f40 2024-03-14T10:05:05.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.490+0000 7fdf50c00700 1 -- 172.21.15.78:0/1028509952 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf4000eec0 con 0x7fdf54105f40 2024-03-14T10:05:05.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.490+0000 7fdf5ae00700 1 -- 172.21.15.78:0/1028509952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf541a6430 con 0x7fdf54105f40 2024-03-14T10:05:05.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.490+0000 7fdf50c00700 1 -- 172.21.15.78:0/1028509952 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf40005590 con 0x7fdf54105f40 2024-03-14T10:05:05.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.490+0000 7fdf5ae00700 1 -- 172.21.15.78:0/1028509952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf541a6960 con 0x7fdf54105f40 2024-03-14T10:05:05.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.491+0000 7fdf5ae00700 1 -- 172.21.15.78:0/1028509952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf541022f0 con 0x7fdf54105f40 2024-03-14T10:05:05.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.493+0000 7fdf50c00700 1 -- 172.21.15.78:0/1028509952 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fdf4000cbd0 con 0x7fdf54105f40 2024-03-14T10:05:05.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.494+0000 7fdf50c00700 1 --2- 172.21.15.78:0/1028509952 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fdf380a0c80 0x7fdf380a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:05.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.494+0000 7fdf50c00700 1 -- 172.21.15.78:0/1028509952 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7fdf40014070 con 0x7fdf54105f40 2024-03-14T10:05:05.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.495+0000 7fdf53e00700 1 --2- 172.21.15.78:0/1028509952 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fdf380a0c80 0x7fdf380a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:05.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.495+0000 7fdf53e00700 1 --2- 172.21.15.78:0/1028509952 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fdf380a0c80 0x7fdf380a3140 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdf4400b2e0 tx=0x7fdf44005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:05.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.498+0000 7fdf50c00700 1 -- 172.21.15.78:0/1028509952 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fdf40095c90 con 0x7fdf54105f40 2024-03-14T10:05:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:05 smithi152 ceph-mon[48279]: pgmap v338: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-14T10:05:05.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:05 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1520055557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:05:05.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:05 smithi132 ceph-mon[47959]: pgmap v338: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-14T10:05:05.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:05 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1520055557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:05:05.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:05 smithi078 ceph-mon[38143]: pgmap v338: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-03-14T10:05:05.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:05 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1520055557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:05:05.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.720+0000 7fdf5ae00700 1 -- 172.21.15.78:0/1028509952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf541017d0 con 0x7fdf54105f40 2024-03-14T10:05:05.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.722+0000 7fdf50c00700 1 -- 172.21.15.78:0/1028509952 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+25214 (secure 0 0 0) 0x7fdf40004ef0 con 0x7fdf54105f40 2024-03-14T10:05:05.723 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-03-14T10:05:05.723 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":96,"fsid":"83662c4e-e1e7-11ee-95c9-87774f69a715","created":"2024-03-14T09:45:43.515085+0000","modified":"2024-03-14T10:04:17.907246+0000","last_up_change":"2024-03-14T09:51:26.740805+0000","last_in_change":"2024-03-14T09:51:10.574553+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T09:48:37.618450+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-14T09:52:03.938666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-14T09:52:05.149985+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-14T09:52:08.227698+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":95,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a9918c01-e344-4789-83f9-6f9425f98542","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6803","nonce":1911630001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6805","nonce":1911630001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6809","nonce":1911630001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1911630001},{"type":"v1","addr":"172.21.15.78:6807","nonce":1911630001}]},"public_addr":"172.21.15.78:6803/1911630001","cluster_addr":"172.21.15.78:6805/1911630001","heartbeat_back_addr":"172.21.15.78:6809/1911630001","heartbeat_front_addr":"172.21.15.78:6807/1911630001","state":["exists","up"]},{"osd":1,"uuid":"1a9ceeed-ab6b-43c6-b822-fcf07b8cbc71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6801","nonce":162295960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6803","nonce":162295960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6807","nonce":162295960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":162295960},{"type":"v1","addr":"172.21.15.132:6805","nonce":162295960}]},"public_addr":"172.21.15.132:6801/162295960","cluster_addr":"172.21.15.132:6803/162295960","heartbeat_back_addr":"172.21.15.132:6807/162295960","heartbeat_front_addr":"172.21.15.132:6805/162295960","state":["exists","up"]},{"osd":2,"uuid":"410edac5-366f-4f37-aaf3-75383d995b28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6801","nonce":3822041535}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6803","nonce":3822041535}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6807","nonce":3822041535}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":3822041535},{"type":"v1","addr":"172.21.15.152:6805","nonce":3822041535}]},"public_addr":"172.21.15.152:6801/3822041535","cluster_addr":"172.21.15.152:6803/3822041535","heartbeat_back_addr":"172.21.15.152:6807/3822041535","heartbeat_front_addr":"172.21.15.152:6805/3822041535","state":["exists","up"]},{"osd":3,"uuid":"3f480434-66c7-4aba-bea5-e74d94e480d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6811","nonce":3422121058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6813","nonce":3422121058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6817","nonce":3422121058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3422121058},{"type":"v1","addr":"172.21.15.78:6815","nonce":3422121058}]},"public_addr":"172.21.15.78:6811/3422121058","cluster_addr":"172.21.15.78:6813/3422121058","heartbeat_back_addr":"172.21.15.78:6817/3422121058","heartbeat_front_addr":"172.21.15.78:6815/3422121058","state":["exists","up"]},{"osd":4,"uuid":"fc8670cb-b4c1-42fc-9134-4fce4b31e03c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6809","nonce":830577797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6811","nonce":830577797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6815","nonce":830577797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":830577797},{"type":"v1","addr":"172.21.15.132:6813","nonce":830577797}]},"public_addr":"172.21.15.132:6809/830577797","cluster_addr":"172.21.15.132:6811/830577797","heartbeat_back_addr":"172.21.15.132:6815/830577797","heartbeat_front_addr":"172.21.15.132:6813/830577797","state":["exists","up"]},{"osd":5,"uuid":"a315a4be-a807-4aef-b476-21ca966b20fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6809","nonce":1175069776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6811","nonce":1175069776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6814","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6815","nonce":1175069776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1175069776},{"type":"v1","addr":"172.21.15.152:6813","nonce":1175069776}]},"public_addr":"172.21.15.152:6809/1175069776","cluster_addr":"172.21.15.152:6811/1175069776","heartbeat_back_addr":"172.21.15.152:6815/1175069776","heartbeat_front_addr":"172.21.15.152:6813/1175069776","state":["exists","up"]},{"osd":6,"uuid":"8b9f16ed-3677-433d-976b-99b3b21f3479","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6819","nonce":996423467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6821","nonce":996423467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6825","nonce":996423467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":996423467},{"type":"v1","addr":"172.21.15.78:6823","nonce":996423467}]},"public_addr":"172.21.15.78:6819/996423467","cluster_addr":"172.21.15.78:6821/996423467","heartbeat_back_addr":"172.21.15.78:6825/996423467","heartbeat_front_addr":"172.21.15.78:6823/996423467","state":["exists","up"]},{"osd":7,"uuid":"f097c83a-d182-409b-bfe8-5bf6a6627fef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6817","nonce":1879810712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6819","nonce":1879810712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6823","nonce":1879810712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":1879810712},{"type":"v1","addr":"172.21.15.132:6821","nonce":1879810712}]},"public_addr":"172.21.15.132:6817/1879810712","cluster_addr":"172.21.15.132:6819/1879810712","heartbeat_back_addr":"172.21.15.132:6823/1879810712","heartbeat_front_addr":"172.21.15.132:6821/1879810712","state":["exists","up"]},{"osd":8,"uuid":"238d2afc-7d81-4089-98be-2852f0504b6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6816","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6817","nonce":2604540397}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6818","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6819","nonce":2604540397}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6822","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6823","nonce":2604540397}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":2604540397},{"type":"v1","addr":"172.21.15.152:6821","nonce":2604540397}]},"public_addr":"172.21.15.152:6817/2604540397","cluster_addr":"172.21.15.152:6819/2604540397","heartbeat_back_addr":"172.21.15.152:6823/2604540397","heartbeat_front_addr":"172.21.15.152:6821/2604540397","state":["exists","up"]},{"osd":9,"uuid":"15f5ddad-e37a-4291-8d62-05fa69894552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6827","nonce":803422096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6829","nonce":803422096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6833","nonce":803422096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":803422096},{"type":"v1","addr":"172.21.15.78:6831","nonce":803422096}]},"public_addr":"172.21.15.78:6827/803422096","cluster_addr":"172.21.15.78:6829/803422096","heartbeat_back_addr":"172.21.15.78:6833/803422096","heartbeat_front_addr":"172.21.15.78:6831/803422096","state":["exists","up"]},{"osd":10,"uuid":"7a325f26-e31f-4f0e-b09f-8bda5b27fbcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6825","nonce":81920089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6827","nonce":81920089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6831","nonce":81920089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":81920089},{"type":"v1","addr":"172.21.15.132:6829","nonce":81920089}]},"public_addr":"172.21.15.132:6825/81920089","cluster_addr":"172.21.15.132:6827/81920089","heartbeat_back_addr":"172.21.15.132:6831/81920089","heartbeat_front_addr":"172.21.15.132:6829/81920089","state":["exists","up"]},{"osd":11,"uuid":"37c97c4c-a16b-4b02-9bd0-585aea0baafc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6824","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6825","nonce":2318800042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6826","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6827","nonce":2318800042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6830","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6831","nonce":2318800042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6828","nonce":2318800042},{"type":"v1","addr":"172.21.15.152:6829","nonce":2318800042}]},"public_addr":"172.21.15.152:6825/2318800042","cluster_addr":"172.21.15.152:6827/2318800042","heartbeat_back_addr":"172.21.15.152:6831/2318800042","heartbeat_front_addr":"172.21.15.152:6829/2318800042","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:47:59.077198+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:16.800001+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:34.671929+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:48:55.433074+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:13.156245+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:31.098744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:49:51.916387+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:10.087534+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:28.114592+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:50:49.040146+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:07.073244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T09:51:25.370522+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.1d","mappings":[{"from":4,"to":6}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1935098096":"2024-03-15T09:54:09.924780+0000","172.21.15.78:0/4237457492":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/2472038902":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2380275837":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6801/2580616009":"2024-03-15T09:54:09.924780+0000","172.21.15.78:0/1127810454":"2024-03-15T09:46:17.100896+0000","172.21.15.78:6801/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/119757382":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/3308792589":"2024-03-15T09:54:09.924780+0000","172.21.15.78:6800/1325295736":"2024-03-15T09:46:17.100896+0000","172.21.15.78:0/2260754576":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/555376800":"2024-03-15T09:54:09.924780+0000","172.21.15.78:6800/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:6801/3890232539":"2024-03-15T09:46:40.988022+0000","172.21.15.78:0/3220727810":"2024-03-15T09:54:09.924780+0000","172.21.15.78:0/2934089773":"2024-03-15T09:54:09.924780+0000","172.21.15.78:6800/2580616009":"2024-03-15T09:54:09.924780+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T10:05:05.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.724+0000 7fdf5ae00700 1 -- 172.21.15.78:0/1028509952 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fdf380a0c80 msgr2=0x7fdf380a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:05.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.724+0000 7fdf5ae00700 1 --2- 172.21.15.78:0/1028509952 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fdf380a0c80 0x7fdf380a3140 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdf4400b2e0 tx=0x7fdf44005fd0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.724+0000 7fdf5ae00700 1 -- 172.21.15.78:0/1028509952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf54105f40 msgr2=0x7fdf5419faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:05.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.724+0000 7fdf5ae00700 1 --2- 172.21.15.78:0/1028509952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf54105f40 0x7fdf5419faa0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdf4000e970 tx=0x7fdf4000ec80 comp rx=0 tx=0).stop 2024-03-14T10:05:05.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.724+0000 7fdf5ae00700 1 -- 172.21.15.78:0/1028509952 shutdown_connections 2024-03-14T10:05:05.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.724+0000 7fdf5ae00700 1 --2- 172.21.15.78:0/1028509952 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fdf380a0c80 0x7fdf380a3140 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.724+0000 7fdf5ae00700 1 --2- 172.21.15.78:0/1028509952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf54100e20 0x7fdf5419f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.724+0000 7fdf5ae00700 1 --2- 172.21.15.78:0/1028509952 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fdf54199690 0x7fdf54199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.724+0000 7fdf5ae00700 1 --2- 172.21.15.78:0/1028509952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdf54105f40 0x7fdf5419faa0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.724+0000 7fdf5ae00700 1 -- 172.21.15.78:0/1028509952 >> 172.21.15.78:0/1028509952 conn(0x7fdf540fc700 msgr2=0x7fdf540fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.725+0000 7fdf5ae00700 1 -- 172.21.15.78:0/1028509952 shutdown_connections 2024-03-14T10:05:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.725+0000 7fdf5ae00700 1 -- 172.21.15.78:0/1028509952 wait complete. 2024-03-14T10:05:05.744 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-14T10:05:05.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.940+0000 7f1d56e00700 1 -- 172.21.15.78:0/612119343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d50100f20 msgr2=0x7f1d50101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:05.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.940+0000 7f1d56e00700 1 --2- 172.21.15.78:0/612119343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d50100f20 0x7f1d50101300 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1d400098f0 tx=0x7f1d40009c00 comp rx=0 tx=0).stop 2024-03-14T10:05:05.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.940+0000 7f1d56e00700 1 -- 172.21.15.78:0/612119343 shutdown_connections 2024-03-14T10:05:05.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.940+0000 7f1d56e00700 1 --2- 172.21.15.78:0/612119343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d50100f20 0x7f1d50101300 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.940+0000 7f1d56e00700 1 --2- 172.21.15.78:0/612119343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d50106040 0x7f1d5010dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.940+0000 7f1d56e00700 1 --2- 172.21.15.78:0/612119343 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d501018d0 0x7f1d50105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.940+0000 7f1d56e00700 1 -- 172.21.15.78:0/612119343 >> 172.21.15.78:0/612119343 conn(0x7f1d500fc7a0 msgr2=0x7f1d500febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:05.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.940+0000 7f1d56e00700 1 -- 172.21.15.78:0/612119343 shutdown_connections 2024-03-14T10:05:05.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.940+0000 7f1d56e00700 1 -- 172.21.15.78:0/612119343 wait complete. 2024-03-14T10:05:05.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.942+0000 7f1d56e00700 1 Processor -- start 2024-03-14T10:05:05.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.942+0000 7f1d56e00700 1 -- start start 2024-03-14T10:05:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.942+0000 7f1d56e00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d50100f20 0x7f1d5019f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.942+0000 7f1d56e00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d50106040 0x7f1d5019fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.942+0000 7f1d56e00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d50199710 0x7f1d50199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.942+0000 7f1d56e00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5019a190 con 0x7f1d50100f20 2024-03-14T10:05:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.943+0000 7f1d56e00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5019a300 con 0x7f1d50106040 2024-03-14T10:05:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.943+0000 7f1d56e00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5019a4c0 con 0x7f1d50199710 2024-03-14T10:05:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.943+0000 7f1d4fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d50100f20 0x7f1d5019f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.943+0000 7f1d4fe00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d50100f20 0x7f1d5019f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41150/0 (socket says 172.21.15.78:41150) 2024-03-14T10:05:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.943+0000 7f1d4fe00700 1 -- 172.21.15.78:0/3363942670 learned_addr learned my addr 172.21.15.78:0/3363942670 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:05:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.943+0000 7f1d54e00700 1 --2- 172.21.15.78:0/3363942670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d50199710 0x7f1d50199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.943+0000 7f1d4f400700 1 --2- 172.21.15.78:0/3363942670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d50106040 0x7f1d5019fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.943+0000 7f1d4fe00700 1 -- 172.21.15.78:0/3363942670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d50199710 msgr2=0x7f1d50199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:05.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.943+0000 7f1d4fe00700 1 --2- 172.21.15.78:0/3363942670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d50199710 0x7f1d50199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.943+0000 7f1d4fe00700 1 -- 172.21.15.78:0/3363942670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d50106040 msgr2=0x7f1d5019fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:05.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.943+0000 7f1d4fe00700 1 --2- 172.21.15.78:0/3363942670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d50106040 0x7f1d5019fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:05.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.943+0000 7f1d4fe00700 1 -- 172.21.15.78:0/3363942670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d48009d00 con 0x7f1d50100f20 2024-03-14T10:05:05.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.943+0000 7f1d54e00700 1 --2- 172.21.15.78:0/3363942670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d50199710 0x7f1d50199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T10:05:05.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.943+0000 7f1d4fe00700 1 --2- 172.21.15.78:0/3363942670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d50100f20 0x7f1d5019f540 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f1d4000fa00 tx=0x7f1d4000fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:05.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.944+0000 7f1d4cc00700 1 -- 172.21.15.78:0/3363942670 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4001c070 con 0x7f1d50100f20 2024-03-14T10:05:05.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.944+0000 7f1d56e00700 1 -- 172.21.15.78:0/3363942670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d40009520 con 0x7f1d50100f20 2024-03-14T10:05:05.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.944+0000 7f1d4cc00700 1 -- 172.21.15.78:0/3363942670 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d40005040 con 0x7f1d50100f20 2024-03-14T10:05:05.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.944+0000 7f1d4cc00700 1 -- 172.21.15.78:0/3363942670 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d40004a90 con 0x7f1d50100f20 2024-03-14T10:05:05.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.944+0000 7f1d56e00700 1 -- 172.21.15.78:0/3363942670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d501a66a0 con 0x7f1d50100f20 2024-03-14T10:05:05.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.945+0000 7f1d56e00700 1 -- 172.21.15.78:0/3363942670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d501023f0 con 0x7f1d50100f20 2024-03-14T10:05:05.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.947+0000 7f1d4cc00700 1 -- 172.21.15.78:0/3363942670 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f1d400051b0 con 0x7f1d50100f20 2024-03-14T10:05:05.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.948+0000 7f1d4cc00700 1 --2- 172.21.15.78:0/3363942670 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1d340a0c00 0x7f1d340a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:05.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.948+0000 7f1d4cc00700 1 -- 172.21.15.78:0/3363942670 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9496+0+0 (secure 0 0 0) 0x7f1d400cdff0 con 0x7f1d50100f20 2024-03-14T10:05:05.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.948+0000 7f1d4f400700 1 --2- 172.21.15.78:0/3363942670 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1d340a0c00 0x7f1d340a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:05.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.949+0000 7f1d4f400700 1 --2- 172.21.15.78:0/3363942670 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1d340a0c00 0x7f1d340a30c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1d5019ae20 tx=0x7f1d3c009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:05.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:05.950+0000 7f1d4cc00700 1 -- 172.21.15.78:0/3363942670 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1d40097300 con 0x7f1d50100f20 2024-03-14T10:05:06.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.208+0000 7f1d56e00700 1 -- 172.21.15.78:0/3363942670 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f1d50067ac0 con 0x7f1d50100f20 2024-03-14T10:05:06.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.228+0000 7f1d4cc00700 1 -- 172.21.15.78:0/3363942670 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1d4009a920 con 0x7f1d50100f20 2024-03-14T10:05:06.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.230+0000 7f1d56e00700 1 -- 172.21.15.78:0/3363942670 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1d340a0c00 msgr2=0x7f1d340a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:06.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.230+0000 7f1d56e00700 1 --2- 172.21.15.78:0/3363942670 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1d340a0c00 0x7f1d340a30c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1d5019ae20 tx=0x7f1d3c009260 comp rx=0 tx=0).stop 2024-03-14T10:05:06.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.230+0000 7f1d56e00700 1 -- 172.21.15.78:0/3363942670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d50100f20 msgr2=0x7f1d5019f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:06.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.230+0000 7f1d56e00700 1 --2- 172.21.15.78:0/3363942670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d50100f20 0x7f1d5019f540 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f1d4000fa00 tx=0x7f1d4000fa30 comp rx=0 tx=0).stop 2024-03-14T10:05:06.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.231+0000 7f1d56e00700 1 -- 172.21.15.78:0/3363942670 shutdown_connections 2024-03-14T10:05:06.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.231+0000 7f1d56e00700 1 --2- 172.21.15.78:0/3363942670 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f1d50199710 0x7f1d50199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:06.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.231+0000 7f1d56e00700 1 --2- 172.21.15.78:0/3363942670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d50100f20 0x7f1d5019f540 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:06.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.231+0000 7f1d56e00700 1 --2- 172.21.15.78:0/3363942670 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f1d340a0c00 0x7f1d340a30c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:06.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.231+0000 7f1d56e00700 1 --2- 172.21.15.78:0/3363942670 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1d50106040 0x7f1d5019fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:06.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.231+0000 7f1d56e00700 1 -- 172.21.15.78:0/3363942670 >> 172.21.15.78:0/3363942670 conn(0x7f1d500fc7a0 msgr2=0x7f1d500fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:06.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.231+0000 7f1d56e00700 1 -- 172.21.15.78:0/3363942670 shutdown_connections 2024-03-14T10:05:06.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.231+0000 7f1d56e00700 1 -- 172.21.15.78:0/3363942670 wait complete. 2024-03-14T10:05:06.232 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-14T10:05:06.251 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-14T10:05:06.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.447+0000 7f62f1a00700 1 -- 172.21.15.78:0/2468546522 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62ec1018e0 msgr2=0x7f62ec105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:06.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.447+0000 7f62f1a00700 1 --2- 172.21.15.78:0/2468546522 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62ec1018e0 0x7f62ec105920 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009990 tx=0x7f62d8009ca0 comp rx=0 tx=0).stop 2024-03-14T10:05:06.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.447+0000 7f62f1a00700 1 -- 172.21.15.78:0/2468546522 shutdown_connections 2024-03-14T10:05:06.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.447+0000 7f62f1a00700 1 --2- 172.21.15.78:0/2468546522 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f62ec100f30 0x7f62ec101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:06.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.447+0000 7f62f1a00700 1 --2- 172.21.15.78:0/2468546522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62ec106050 0x7f62ec10dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:06.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.447+0000 7f62f1a00700 1 --2- 172.21.15.78:0/2468546522 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62ec1018e0 0x7f62ec105920 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:06.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.447+0000 7f62f1a00700 1 -- 172.21.15.78:0/2468546522 >> 172.21.15.78:0/2468546522 conn(0x7f62ec0fc7b0 msgr2=0x7f62ec0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:06.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.448+0000 7f62f1a00700 1 -- 172.21.15.78:0/2468546522 shutdown_connections 2024-03-14T10:05:06.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.448+0000 7f62f1a00700 1 -- 172.21.15.78:0/2468546522 wait complete. 2024-03-14T10:05:06.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.448+0000 7f62f1a00700 1 Processor -- start 2024-03-14T10:05:06.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.449+0000 7f62f1a00700 1 -- start start 2024-03-14T10:05:06.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.449+0000 7f62f1a00700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62ec100f30 0x7f62ec19f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:06.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.449+0000 7f62f1a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62ec106050 0x7f62ec19fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:06.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.449+0000 7f62f1a00700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f62ec199680 0x7f62ec199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.449+0000 7f62f1a00700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec19a100 con 0x7f62ec106050 2024-03-14T10:05:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.449+0000 7f62f1a00700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec19a270 con 0x7f62ec199680 2024-03-14T10:05:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.449+0000 7f62f1a00700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec19a430 con 0x7f62ec100f30 2024-03-14T10:05:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.449+0000 7f62ea000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62ec106050 0x7f62ec19fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.449+0000 7f62ea000700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62ec106050 0x7f62ec19fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41176/0 (socket says 172.21.15.78:41176) 2024-03-14T10:05:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.449+0000 7f62ea000700 1 -- 172.21.15.78:0/1664640950 learned_addr learned my addr 172.21.15.78:0/1664640950 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:05:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.449+0000 7f62eaa00700 1 --2- 172.21.15.78:0/1664640950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62ec100f30 0x7f62ec19f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.449+0000 7f62eb400700 1 --2- 172.21.15.78:0/1664640950 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f62ec199680 0x7f62ec199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.450+0000 7f62ea000700 1 -- 172.21.15.78:0/1664640950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62ec100f30 msgr2=0x7f62ec19f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.450+0000 7f62ea000700 1 --2- 172.21.15.78:0/1664640950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62ec100f30 0x7f62ec19f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.450+0000 7f62ea000700 1 -- 172.21.15.78:0/1664640950 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f62ec199680 msgr2=0x7f62ec199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.450+0000 7f62ea000700 1 --2- 172.21.15.78:0/1664640950 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f62ec199680 0x7f62ec199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.450+0000 7f62ea000700 1 -- 172.21.15.78:0/1664640950 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d80095f0 con 0x7f62ec106050 2024-03-14T10:05:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.450+0000 7f62ea000700 1 --2- 172.21.15.78:0/1664640950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62ec106050 0x7f62ec19fa90 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f62d8000c00 tx=0x7f62d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:06.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.450+0000 7f62e3400700 1 -- 172.21.15.78:0/1664640950 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d801d070 con 0x7f62ec106050 2024-03-14T10:05:06.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.450+0000 7f62e3400700 1 -- 172.21.15.78:0/1664640950 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62d8021410 con 0x7f62ec106050 2024-03-14T10:05:06.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.450+0000 7f62f1a00700 1 -- 172.21.15.78:0/1664640950 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ec1a63c0 con 0x7f62ec106050 2024-03-14T10:05:06.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.450+0000 7f62e3400700 1 -- 172.21.15.78:0/1664640950 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d8017650 con 0x7f62ec106050 2024-03-14T10:05:06.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.450+0000 7f62f1a00700 1 -- 172.21.15.78:0/1664640950 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ec1a6790 con 0x7f62ec106050 2024-03-14T10:05:06.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.451+0000 7f62f1a00700 1 -- 172.21.15.78:0/1664640950 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ec102400 con 0x7f62ec106050 2024-03-14T10:05:06.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.453+0000 7f62e3400700 1 -- 172.21.15.78:0/1664640950 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f62d8021580 con 0x7f62ec106050 2024-03-14T10:05:06.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.454+0000 7f62e3400700 1 --2- 172.21.15.78:0/1664640950 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f62d00a0c00 0x7f62d00a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:06.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.455+0000 7f62e3400700 1 -- 172.21.15.78:0/1664640950 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9666+0+0 (secure 0 0 0) 0x7f62d80cda30 con 0x7f62ec106050 2024-03-14T10:05:06.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.455+0000 7f62eaa00700 1 --2- 172.21.15.78:0/1664640950 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f62d00a0c00 0x7f62d00a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:06.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.455+0000 7f62eaa00700 1 --2- 172.21.15.78:0/1664640950 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f62d00a0c00 0x7f62d00a30c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f62dc009c60 tx=0x7f62dc009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:06.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.458+0000 7f62e3400700 1 -- 172.21.15.78:0/1664640950 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f62d8096cd0 con 0x7f62ec106050 2024-03-14T10:05:06.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:06 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/677278494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:05:06.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:06 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1028509952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T10:05:06.499 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 14 10:05:06 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-c[85458]: ignoring --setuser ceph since I am not root 2024-03-14T10:05:06.499 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 14 10:05:06 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-c[85458]: ignoring --setgroup ceph since I am not root 2024-03-14T10:05:06.499 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 14 10:05:06 smithi152 ceph-mds[85481]: -- 172.21.15.152:0/111894254 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56212eee61a0 con 0x56212eebc800 2024-03-14T10:05:06.499 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 14 10:05:06 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-c[85458]: starting mds.c at 2024-03-14T10:05:06.500 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 14 10:05:06 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-i[85801]: ignoring --setuser ceph since I am not root 2024-03-14T10:05:06.500 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 14 10:05:06 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-i[85801]: ignoring --setgroup ceph since I am not root 2024-03-14T10:05:06.500 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 14 10:05:06 smithi152 ceph-mds[85824]: -- 172.21.15.152:0/3759860822 <== mon.2 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561e49eb21a0 con 0x561e49e88c00 2024-03-14T10:05:06.500 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 14 10:05:06 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-i[85801]: starting mds.i at 2024-03-14T10:05:06.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:06 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/677278494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:05:06.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:06 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1028509952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T10:05:06.520 INFO:journalctl@ceph.mds.e.smithi132.stdout:Mar 14 10:05:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-e[85339]: ignoring --setuser ceph since I am not root 2024-03-14T10:05:06.520 INFO:journalctl@ceph.mds.e.smithi132.stdout:Mar 14 10:05:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-e[85339]: ignoring --setgroup ceph since I am not root 2024-03-14T10:05:06.520 INFO:journalctl@ceph.mds.e.smithi132.stdout:Mar 14 10:05:06 smithi132 ceph-mds[85362]: -- 172.21.15.132:0/2486406469 <== mon.2 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c0b53821a0 con 0x55c0b5358800 2024-03-14T10:05:06.521 INFO:journalctl@ceph.mds.e.smithi132.stdout:Mar 14 10:05:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-e[85339]: starting mds.e at 2024-03-14T10:05:06.521 INFO:journalctl@ceph.mds.h.smithi132.stdout:Mar 14 10:05:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-h[85684]: ignoring --setuser ceph since I am not root 2024-03-14T10:05:06.521 INFO:journalctl@ceph.mds.h.smithi132.stdout:Mar 14 10:05:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-h[85684]: ignoring --setgroup ceph since I am not root 2024-03-14T10:05:06.521 INFO:journalctl@ceph.mds.h.smithi132.stdout:Mar 14 10:05:06 smithi132 ceph-mds[85707]: -- 172.21.15.132:0/3421267424 <== mon.2 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d7cb5e61a0 con 0x55d7cb5bc800 2024-03-14T10:05:06.521 INFO:journalctl@ceph.mds.h.smithi132.stdout:Mar 14 10:05:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-h[85684]: starting mds.h at 2024-03-14T10:05:06.521 INFO:journalctl@ceph.mds.b.smithi132.stdout:Mar 14 10:05:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-b[86027]: ignoring --setuser ceph since I am not root 2024-03-14T10:05:06.521 INFO:journalctl@ceph.mds.b.smithi132.stdout:Mar 14 10:05:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-b[86027]: ignoring --setgroup ceph since I am not root 2024-03-14T10:05:06.521 INFO:journalctl@ceph.mds.b.smithi132.stdout:Mar 14 10:05:06 smithi132 ceph-mds[86050]: -- 172.21.15.132:0/187702498 <== mon.1 v2:172.21.15.152:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a55a3421a0 con 0x55a55a318c00 2024-03-14T10:05:06.521 INFO:journalctl@ceph.mds.b.smithi132.stdout:Mar 14 10:05:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-b[86027]: starting mds.b at 2024-03-14T10:05:06.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:06 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mon-a[38120]: 2024-03-14T10:05:06.210+0000 7f52d4a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-14T10:05:06.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:06 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/677278494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T10:05:06.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:06 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1028509952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T10:05:06.553 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 14 10:05:06 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-g[87656]: ignoring --setuser ceph since I am not root 2024-03-14T10:05:06.553 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 14 10:05:06 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-g[87656]: ignoring --setgroup ceph since I am not root 2024-03-14T10:05:06.554 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 14 10:05:06 smithi078 ceph-mds[87679]: -- 172.21.15.78:0/2099033376 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5578fcb081a0 con 0x5578fcae0800 2024-03-14T10:05:06.554 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 14 10:05:06 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-g[87656]: starting mds.g at 2024-03-14T10:05:06.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:06.713+0000 7f62f1a00700 1 -- 172.21.15.78:0/1664640950 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f62ec1018e0 con 0x7f62ec106050 2024-03-14T10:05:07.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.225+0000 7f62e3400700 1 -- 172.21.15.78:0/1664640950 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f62d809a2f0 con 0x7f62ec106050 2024-03-14T10:05:07.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.228+0000 7f62f1a00700 1 -- 172.21.15.78:0/1664640950 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f62d00a0c00 msgr2=0x7f62d00a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:07.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.228+0000 7f62f1a00700 1 --2- 172.21.15.78:0/1664640950 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f62d00a0c00 0x7f62d00a30c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f62dc009c60 tx=0x7f62dc009260 comp rx=0 tx=0).stop 2024-03-14T10:05:07.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.228+0000 7f62f1a00700 1 -- 172.21.15.78:0/1664640950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62ec106050 msgr2=0x7f62ec19fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:07.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.228+0000 7f62f1a00700 1 --2- 172.21.15.78:0/1664640950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62ec106050 0x7f62ec19fa90 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f62d8000c00 tx=0x7f62d8004730 comp rx=0 tx=0).stop 2024-03-14T10:05:07.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.228+0000 7f62f1a00700 1 -- 172.21.15.78:0/1664640950 shutdown_connections 2024-03-14T10:05:07.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.228+0000 7f62f1a00700 1 --2- 172.21.15.78:0/1664640950 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f62d00a0c00 0x7f62d00a30c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:07.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.228+0000 7f62f1a00700 1 --2- 172.21.15.78:0/1664640950 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f62ec199680 0x7f62ec199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:07.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.228+0000 7f62f1a00700 1 --2- 172.21.15.78:0/1664640950 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f62ec100f30 0x7f62ec19f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:07.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.228+0000 7f62f1a00700 1 --2- 172.21.15.78:0/1664640950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62ec106050 0x7f62ec19fa90 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:07.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.228+0000 7f62f1a00700 1 -- 172.21.15.78:0/1664640950 >> 172.21.15.78:0/1664640950 conn(0x7f62ec0fc7b0 msgr2=0x7f62ec0fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:07.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.228+0000 7f62f1a00700 1 -- 172.21.15.78:0/1664640950 shutdown_connections 2024-03-14T10:05:07.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.228+0000 7f62f1a00700 1 -- 172.21.15.78:0/1664640950 wait complete. 2024-03-14T10:05:07.251 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-14T10:05:07.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.451+0000 7f6d0c400700 1 -- 172.21.15.78:0/1753429446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d0410d560 msgr2=0x7f6d0410f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:07.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.451+0000 7f6d0c400700 1 --2- 172.21.15.78:0/1753429446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d0410d560 0x7f6d0410f990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc00b0a0 tx=0x7f6cfc00b3b0 comp rx=0 tx=0).stop 2024-03-14T10:05:07.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.451+0000 7f6d0c400700 1 -- 172.21.15.78:0/1753429446 shutdown_connections 2024-03-14T10:05:07.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.451+0000 7f6d0c400700 1 --2- 172.21.15.78:0/1753429446 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d04073180 0x7f6d04073560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:07.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.451+0000 7f6d0c400700 1 --2- 172.21.15.78:0/1753429446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d0410d560 0x7f6d0410f990 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:07.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.451+0000 7f6d0c400700 1 --2- 172.21.15.78:0/1753429446 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d04073aa0 0x7f6d0410d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:07.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.451+0000 7f6d0c400700 1 -- 172.21.15.78:0/1753429446 >> 172.21.15.78:0/1753429446 conn(0x7f6d040fc7b0 msgr2=0x7f6d040febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:07.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.451+0000 7f6d0c400700 1 -- 172.21.15.78:0/1753429446 shutdown_connections 2024-03-14T10:05:07.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.451+0000 7f6d0c400700 1 -- 172.21.15.78:0/1753429446 wait complete. 2024-03-14T10:05:07.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.452+0000 7f6d0c400700 1 Processor -- start 2024-03-14T10:05:07.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.452+0000 7f6d0c400700 1 -- start start 2024-03-14T10:05:07.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.452+0000 7f6d0c400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d04073180 0x7f6d0419f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:07.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.452+0000 7f6d0c400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d04073aa0 0x7f6d0419fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:07.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.452+0000 7f6d0c400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d0410d560 0x7f6d04199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:07.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.452+0000 7f6d0c400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d04199dd0 con 0x7f6d04073180 2024-03-14T10:05:07.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.452+0000 7f6d0c400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d04199f40 con 0x7f6d04073aa0 2024-03-14T10:05:07.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.452+0000 7f6d0c400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0419a100 con 0x7f6d0410d560 2024-03-14T10:05:07.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.453+0000 7f6d09a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d04073180 0x7f6d0419f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:07.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.453+0000 7f6d09a00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d04073180 0x7f6d0419f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41198/0 (socket says 172.21.15.78:41198) 2024-03-14T10:05:07.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.453+0000 7f6d09a00700 1 -- 172.21.15.78:0/2580956152 learned_addr learned my addr 172.21.15.78:0/2580956152 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:05:07.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.453+0000 7f6d09000700 1 --2- 172.21.15.78:0/2580956152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d04073aa0 0x7f6d0419fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:07.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.453+0000 7f6d0a400700 1 --2- 172.21.15.78:0/2580956152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d0410d560 0x7f6d04199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:07.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.453+0000 7f6d09a00700 1 -- 172.21.15.78:0/2580956152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d0410d560 msgr2=0x7f6d04199770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:07.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.453+0000 7f6d09a00700 1 --2- 172.21.15.78:0/2580956152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d0410d560 0x7f6d04199770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:07.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.453+0000 7f6d09a00700 1 -- 172.21.15.78:0/2580956152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d04073aa0 msgr2=0x7f6d0419fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:07.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.453+0000 7f6d09a00700 1 --2- 172.21.15.78:0/2580956152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d04073aa0 0x7f6d0419fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:07.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.453+0000 7f6d09a00700 1 -- 172.21.15.78:0/2580956152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cfc009d00 con 0x7f6d04073180 2024-03-14T10:05:07.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.453+0000 7f6d09a00700 1 --2- 172.21.15.78:0/2580956152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d04073180 0x7f6d0419f640 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800e9a0 tx=0x7f6cf800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:07.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.453+0000 7f6d02000700 1 -- 172.21.15.78:0/2580956152 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8009790 con 0x7f6d04073180 2024-03-14T10:05:07.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.453+0000 7f6d02000700 1 -- 172.21.15.78:0/2580956152 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6cf800cbd0 con 0x7f6d04073180 2024-03-14T10:05:07.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.454+0000 7f6d0c400700 1 -- 172.21.15.78:0/2580956152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d0419a3e0 con 0x7f6d04073180 2024-03-14T10:05:07.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.454+0000 7f6d02000700 1 -- 172.21.15.78:0/2580956152 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8018680 con 0x7f6d04073180 2024-03-14T10:05:07.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.454+0000 7f6d0c400700 1 -- 172.21.15.78:0/2580956152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d041a6630 con 0x7f6d04073180 2024-03-14T10:05:07.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.455+0000 7f6d0c400700 1 -- 172.21.15.78:0/2580956152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d04108770 con 0x7f6d04073180 2024-03-14T10:05:07.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.458+0000 7f6d02000700 1 -- 172.21.15.78:0/2580956152 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f6cf800cd40 con 0x7f6d04073180 2024-03-14T10:05:07.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.459+0000 7f6d02000700 1 --2- 172.21.15.78:0/2580956152 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6cec0a0c00 0x7f6cec0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:07.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.459+0000 7f6d02000700 1 -- 172.21.15.78:0/2580956152 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9666+0+0 (secure 0 0 0) 0x7f6cf8014070 con 0x7f6d04073180 2024-03-14T10:05:07.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.460+0000 7f6d09000700 1 --2- 172.21.15.78:0/2580956152 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6cec0a0c00 0x7f6cec0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:07.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.460+0000 7f6d09000700 1 --2- 172.21.15.78:0/2580956152 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6cec0a0c00 0x7f6cec0a30c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6d0419ae80 tx=0x7f6cf00073d0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:07.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.462+0000 7f6d02000700 1 -- 172.21.15.78:0/2580956152 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6cf8096050 con 0x7f6d04073180 2024-03-14T10:05:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:07 smithi152 ceph-mon[48279]: pgmap v339: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-14T10:05:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:07 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3363942670' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-14T10:05:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:07 smithi152 ceph-mon[48279]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-14T10:05:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:07 smithi152 ceph-mon[48279]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-14T10:05:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:07 smithi152 ceph-mon[48279]: osdmap e97: 12 total, 12 up, 12 in 2024-03-14T10:05:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:07 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/3363942670' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-14T10:05:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:07 smithi152 ceph-mon[48279]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-14T10:05:07.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:07 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1664640950' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-14T10:05:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:07 smithi132 ceph-mon[47959]: pgmap v339: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-14T10:05:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:07 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3363942670' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-14T10:05:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:07 smithi132 ceph-mon[47959]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-14T10:05:07.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:07 smithi132 ceph-mon[47959]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-14T10:05:07.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:07 smithi132 ceph-mon[47959]: osdmap e97: 12 total, 12 up, 12 in 2024-03-14T10:05:07.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:07 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/3363942670' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-14T10:05:07.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:07 smithi132 ceph-mon[47959]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-14T10:05:07.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:07 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1664640950' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-14T10:05:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:07 smithi078 ceph-mon[38143]: pgmap v339: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-14T10:05:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:07 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3363942670' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-14T10:05:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:07 smithi078 ceph-mon[38143]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-14T10:05:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:07 smithi078 ceph-mon[38143]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-14T10:05:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:07 smithi078 ceph-mon[38143]: osdmap e97: 12 total, 12 up, 12 in 2024-03-14T10:05:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:07 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/3363942670' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-14T10:05:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:07 smithi078 ceph-mon[38143]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-14T10:05:07.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:07 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1664640950' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-14T10:05:07.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:07.706+0000 7f6d0c400700 1 -- 172.21.15.78:0/2580956152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6d04074830 con 0x7f6d04073180 2024-03-14T10:05:08.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.228+0000 7f6d02000700 1 -- 172.21.15.78:0/2580956152 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v98) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6cf8099670 con 0x7f6d04073180 2024-03-14T10:05:08.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.232+0000 7f6d0c400700 1 -- 172.21.15.78:0/2580956152 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6cec0a0c00 msgr2=0x7f6cec0a30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:08.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.232+0000 7f6d0c400700 1 --2- 172.21.15.78:0/2580956152 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6cec0a0c00 0x7f6cec0a30c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6d0419ae80 tx=0x7f6cf00073d0 comp rx=0 tx=0).stop 2024-03-14T10:05:08.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.232+0000 7f6d0c400700 1 -- 172.21.15.78:0/2580956152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d04073180 msgr2=0x7f6d0419f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:08.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.232+0000 7f6d0c400700 1 --2- 172.21.15.78:0/2580956152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d04073180 0x7f6d0419f640 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6cf800e9a0 tx=0x7f6cf800ecb0 comp rx=0 tx=0).stop 2024-03-14T10:05:08.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.232+0000 7f6d0c400700 1 -- 172.21.15.78:0/2580956152 shutdown_connections 2024-03-14T10:05:08.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.232+0000 7f6d0c400700 1 --2- 172.21.15.78:0/2580956152 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f6d0410d560 0x7f6d04199770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:08.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.232+0000 7f6d0c400700 1 --2- 172.21.15.78:0/2580956152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d04073180 0x7f6d0419f640 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:08.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.232+0000 7f6d0c400700 1 --2- 172.21.15.78:0/2580956152 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f6cec0a0c00 0x7f6cec0a30c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:08.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.232+0000 7f6d0c400700 1 --2- 172.21.15.78:0/2580956152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d04073aa0 0x7f6d0419fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:08.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.232+0000 7f6d0c400700 1 -- 172.21.15.78:0/2580956152 >> 172.21.15.78:0/2580956152 conn(0x7f6d040fc7b0 msgr2=0x7f6d040fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:08.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.233+0000 7f6d0c400700 1 -- 172.21.15.78:0/2580956152 shutdown_connections 2024-03-14T10:05:08.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.233+0000 7f6d0c400700 1 -- 172.21.15.78:0/2580956152 wait complete. 2024-03-14T10:05:08.235 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2024-03-14T10:05:08.264 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-14T10:05:08.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.460+0000 7fde91600700 1 -- 172.21.15.78:0/2231352633 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde8c1018e0 msgr2=0x7fde8c105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:08.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.460+0000 7fde91600700 1 --2- 172.21.15.78:0/2231352633 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde8c1018e0 0x7fde8c105920 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fde78009990 tx=0x7fde78009ca0 comp rx=0 tx=0).stop 2024-03-14T10:05:08.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.460+0000 7fde91600700 1 -- 172.21.15.78:0/2231352633 shutdown_connections 2024-03-14T10:05:08.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.460+0000 7fde91600700 1 --2- 172.21.15.78:0/2231352633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fde8c106050 0x7fde8c10dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:08.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.460+0000 7fde91600700 1 --2- 172.21.15.78:0/2231352633 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde8c100f30 0x7fde8c101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:08.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.460+0000 7fde91600700 1 --2- 172.21.15.78:0/2231352633 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde8c1018e0 0x7fde8c105920 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:08.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.460+0000 7fde91600700 1 -- 172.21.15.78:0/2231352633 >> 172.21.15.78:0/2231352633 conn(0x7fde8c0fc7b0 msgr2=0x7fde8c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:08.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.460+0000 7fde91600700 1 -- 172.21.15.78:0/2231352633 shutdown_connections 2024-03-14T10:05:08.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.460+0000 7fde91600700 1 -- 172.21.15.78:0/2231352633 wait complete. 2024-03-14T10:05:08.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.461+0000 7fde91600700 1 Processor -- start 2024-03-14T10:05:08.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.461+0000 7fde91600700 1 -- start start 2024-03-14T10:05:08.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.461+0000 7fde91600700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde8c100f30 0x7fde8c19f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:08.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.461+0000 7fde91600700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fde8c106050 0x7fde8c19fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:08.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.461+0000 7fde91600700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde8c199670 0x7fde8c199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:08.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.461+0000 7fde91600700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c19a0f0 con 0x7fde8c100f30 2024-03-14T10:05:08.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.461+0000 7fde91600700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c19a230 con 0x7fde8c106050 2024-03-14T10:05:08.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.461+0000 7fde91600700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c19a3f0 con 0x7fde8c199670 2024-03-14T10:05:08.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.462+0000 7fde8aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde8c100f30 0x7fde8c19f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:08.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.462+0000 7fde8aa00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde8c100f30 0x7fde8c19f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41218/0 (socket says 172.21.15.78:41218) 2024-03-14T10:05:08.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.462+0000 7fde8aa00700 1 -- 172.21.15.78:0/1358600183 learned_addr learned my addr 172.21.15.78:0/1358600183 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:05:08.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.462+0000 7fde8a000700 1 --2- 172.21.15.78:0/1358600183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fde8c106050 0x7fde8c19fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:08.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.462+0000 7fde8b400700 1 --2- 172.21.15.78:0/1358600183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde8c199670 0x7fde8c199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:08.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.462+0000 7fde8aa00700 1 -- 172.21.15.78:0/1358600183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde8c199670 msgr2=0x7fde8c199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:08.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.462+0000 7fde8aa00700 1 --2- 172.21.15.78:0/1358600183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde8c199670 0x7fde8c199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:08.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.462+0000 7fde8aa00700 1 -- 172.21.15.78:0/1358600183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fde8c106050 msgr2=0x7fde8c19fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:08.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.462+0000 7fde8aa00700 1 --2- 172.21.15.78:0/1358600183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fde8c106050 0x7fde8c19fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:08.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.462+0000 7fde8aa00700 1 -- 172.21.15.78:0/1358600183 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde780095f0 con 0x7fde8c100f30 2024-03-14T10:05:08.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.462+0000 7fde8b400700 1 --2- 172.21.15.78:0/1358600183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde8c199670 0x7fde8c199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T10:05:08.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.462+0000 7fde8aa00700 1 --2- 172.21.15.78:0/1358600183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde8c100f30 0x7fde8c19f540 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fde7c00e9a0 tx=0x7fde7c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:08.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.463+0000 7fde83400700 1 -- 172.21.15.78:0/1358600183 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c009790 con 0x7fde8c100f30 2024-03-14T10:05:08.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.463+0000 7fde83400700 1 -- 172.21.15.78:0/1358600183 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde7c00cbd0 con 0x7fde8c100f30 2024-03-14T10:05:08.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.463+0000 7fde91600700 1 -- 172.21.15.78:0/1358600183 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8c1a63b0 con 0x7fde8c100f30 2024-03-14T10:05:08.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.463+0000 7fde83400700 1 -- 172.21.15.78:0/1358600183 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c018680 con 0x7fde8c100f30 2024-03-14T10:05:08.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.463+0000 7fde91600700 1 -- 172.21.15.78:0/1358600183 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8c1a6910 con 0x7fde8c100f30 2024-03-14T10:05:08.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.464+0000 7fde91600700 1 -- 172.21.15.78:0/1358600183 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8c102400 con 0x7fde8c100f30 2024-03-14T10:05:08.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.466+0000 7fde83400700 1 -- 172.21.15.78:0/1358600183 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7fde7c00cd40 con 0x7fde8c100f30 2024-03-14T10:05:08.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.468+0000 7fde83400700 1 --2- 172.21.15.78:0/1358600183 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fde700a0c80 0x7fde700a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:08.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.468+0000 7fde83400700 1 -- 172.21.15.78:0/1358600183 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9112+0+0 (secure 0 0 0) 0x7fde7c014070 con 0x7fde8c100f30 2024-03-14T10:05:08.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.468+0000 7fde8a000700 1 --2- 172.21.15.78:0/1358600183 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fde700a0c80 0x7fde700a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:08.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.469+0000 7fde8a000700 1 --2- 172.21.15.78:0/1358600183 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fde700a0c80 0x7fde700a3140 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0ffdc0 tx=0x7fde7800b260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:08.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.471+0000 7fde83400700 1 -- 172.21.15.78:0/1358600183 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fde7c095ee0 con 0x7fde8c100f30 2024-03-14T10:05:08.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-14T10:05:08.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-14T10:05:08.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: Cluster is now healthy 2024-03-14T10:05:08.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.152:6837/669608924,v1:172.21.15.152:6839/669608924] up:boot 2024-03-14T10:05:08.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.152:6836/1854385091,v1:172.21.15.152:6838/1854385091] up:boot 2024-03-14T10:05:08.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.132:6834/3972294842,v1:172.21.15.132:6835/3972294842] up:boot 2024-03-14T10:05:08.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.132:6838/4223868270,v1:172.21.15.132:6839/4223868270] up:boot 2024-03-14T10:05:08.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.132:6836/2013596851,v1:172.21.15.132:6837/2013596851] up:boot 2024-03-14T10:05:08.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: mds.? [v2:172.21.15.78:6838/78444342,v1:172.21.15.78:6839/78444342] up:boot 2024-03-14T10:05:08.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1664640950' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-14T10:05:08.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: fsmap 12 up:standby 2024-03-14T10:05:08.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-14T10:05:08.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-14T10:05:08.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-14T10:05:08.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-14T10:05:08.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-14T10:05:08.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-14T10:05:08.500 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:08 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2580956152' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-14T10:05:08.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-14T10:05:08.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-14T10:05:08.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: Cluster is now healthy 2024-03-14T10:05:08.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.152:6837/669608924,v1:172.21.15.152:6839/669608924] up:boot 2024-03-14T10:05:08.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.152:6836/1854385091,v1:172.21.15.152:6838/1854385091] up:boot 2024-03-14T10:05:08.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.132:6834/3972294842,v1:172.21.15.132:6835/3972294842] up:boot 2024-03-14T10:05:08.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.132:6838/4223868270,v1:172.21.15.132:6839/4223868270] up:boot 2024-03-14T10:05:08.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.132:6836/2013596851,v1:172.21.15.132:6837/2013596851] up:boot 2024-03-14T10:05:08.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: mds.? [v2:172.21.15.78:6838/78444342,v1:172.21.15.78:6839/78444342] up:boot 2024-03-14T10:05:08.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1664640950' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-14T10:05:08.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: fsmap 12 up:standby 2024-03-14T10:05:08.521 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-14T10:05:08.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-14T10:05:08.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-14T10:05:08.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-14T10:05:08.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-14T10:05:08.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-14T10:05:08.522 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:08 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2580956152' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-14T10:05:08.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-14T10:05:08.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-14T10:05:08.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: Cluster is now healthy 2024-03-14T10:05:08.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.152:6837/669608924,v1:172.21.15.152:6839/669608924] up:boot 2024-03-14T10:05:08.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.152:6836/1854385091,v1:172.21.15.152:6838/1854385091] up:boot 2024-03-14T10:05:08.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.132:6834/3972294842,v1:172.21.15.132:6835/3972294842] up:boot 2024-03-14T10:05:08.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.132:6838/4223868270,v1:172.21.15.132:6839/4223868270] up:boot 2024-03-14T10:05:08.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.132:6836/2013596851,v1:172.21.15.132:6837/2013596851] up:boot 2024-03-14T10:05:08.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: mds.? [v2:172.21.15.78:6838/78444342,v1:172.21.15.78:6839/78444342] up:boot 2024-03-14T10:05:08.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1664640950' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-14T10:05:08.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: fsmap 12 up:standby 2024-03-14T10:05:08.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-14T10:05:08.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-14T10:05:08.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-14T10:05:08.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-14T10:05:08.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-14T10:05:08.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: from='mgr.15546 172.21.15.78:0/1954605105' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-14T10:05:08.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:08 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2580956152' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-14T10:05:08.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:08.719+0000 7fde91600700 1 -- 172.21.15.78:0/1358600183 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fde8c067ac0 con 0x7fde8c100f30 2024-03-14T10:05:09.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.229+0000 7fde83400700 1 -- 172.21.15.78:0/1358600183 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v99) v1 ==== 169+0+0 (secure 0 0 0) 0x7fde7c0260d0 con 0x7fde8c100f30 2024-03-14T10:05:09.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.236+0000 7fde91600700 1 -- 172.21.15.78:0/1358600183 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fde700a0c80 msgr2=0x7fde700a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:09.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.236+0000 7fde91600700 1 --2- 172.21.15.78:0/1358600183 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fde700a0c80 0x7fde700a3140 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0ffdc0 tx=0x7fde7800b260 comp rx=0 tx=0).stop 2024-03-14T10:05:09.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.236+0000 7fde91600700 1 -- 172.21.15.78:0/1358600183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde8c100f30 msgr2=0x7fde8c19f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:09.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.236+0000 7fde91600700 1 --2- 172.21.15.78:0/1358600183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde8c100f30 0x7fde8c19f540 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fde7c00e9a0 tx=0x7fde7c00ecb0 comp rx=0 tx=0).stop 2024-03-14T10:05:09.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.236+0000 7fde91600700 1 -- 172.21.15.78:0/1358600183 shutdown_connections 2024-03-14T10:05:09.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.236+0000 7fde91600700 1 --2- 172.21.15.78:0/1358600183 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7fde8c199670 0x7fde8c199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:09.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.236+0000 7fde91600700 1 --2- 172.21.15.78:0/1358600183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde8c100f30 0x7fde8c19f540 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:09.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.236+0000 7fde91600700 1 --2- 172.21.15.78:0/1358600183 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7fde700a0c80 0x7fde700a3140 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:09.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.236+0000 7fde91600700 1 --2- 172.21.15.78:0/1358600183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fde8c106050 0x7fde8c19fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:09.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.236+0000 7fde91600700 1 -- 172.21.15.78:0/1358600183 >> 172.21.15.78:0/1358600183 conn(0x7fde8c0fc7b0 msgr2=0x7fde8c0feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:09.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.236+0000 7fde91600700 1 -- 172.21.15.78:0/1358600183 shutdown_connections 2024-03-14T10:05:09.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.237+0000 7fde91600700 1 -- 172.21.15.78:0/1358600183 wait complete. 2024-03-14T10:05:09.241 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2024-03-14T10:05:09.263 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-14T10:05:09.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.462+0000 7f0957400700 1 -- 172.21.15.78:0/2565352601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09500680b0 msgr2=0x7f0950068490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:09.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.462+0000 7f0957400700 1 --2- 172.21.15.78:0/2565352601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09500680b0 0x7f0950068490 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f09440099c0 tx=0x7f0944009cd0 comp rx=0 tx=0).stop 2024-03-14T10:05:09.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.462+0000 7f0957400700 1 -- 172.21.15.78:0/2565352601 shutdown_connections 2024-03-14T10:05:09.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.462+0000 7f0957400700 1 --2- 172.21.15.78:0/2565352601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0950106060 0x7f095010dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:09.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.462+0000 7f0957400700 1 --2- 172.21.15.78:0/2565352601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09500680b0 0x7f0950068490 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:09.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.462+0000 7f0957400700 1 --2- 172.21.15.78:0/2565352601 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f0950068a60 0x7f0950105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:09.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.462+0000 7f0957400700 1 -- 172.21.15.78:0/2565352601 >> 172.21.15.78:0/2565352601 conn(0x7f0950076d80 msgr2=0x7f0950077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:09.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.462+0000 7f0957400700 1 -- 172.21.15.78:0/2565352601 shutdown_connections 2024-03-14T10:05:09.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.462+0000 7f0957400700 1 -- 172.21.15.78:0/2565352601 wait complete. 2024-03-14T10:05:09.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.463+0000 7f0957400700 1 Processor -- start 2024-03-14T10:05:09.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.463+0000 7f0957400700 1 -- start start 2024-03-14T10:05:09.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.464+0000 7f0957400700 1 --2- >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f09500680b0 0x7f095019f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:09.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.464+0000 7f0957400700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0950106060 0x7f095019fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:09.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.464+0000 7f0957400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0950199690 0x7f0950199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:09.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.464+0000 7f0957400700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095019a110 con 0x7f0950199690 2024-03-14T10:05:09.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.464+0000 7f0957400700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095019a280 con 0x7f0950106060 2024-03-14T10:05:09.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.464+0000 7f0957400700 1 -- --> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095019a440 con 0x7f09500680b0 2024-03-14T10:05:09.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.464+0000 7f0955400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0950199690 0x7f0950199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:09.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.464+0000 7f0955400700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0950199690 0x7f0950199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41250/0 (socket says 172.21.15.78:41250) 2024-03-14T10:05:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.464+0000 7f0955400700 1 -- 172.21.15.78:0/337924477 learned_addr learned my addr 172.21.15.78:0/337924477 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-03-14T10:05:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.464+0000 7f0954a00700 1 --2- 172.21.15.78:0/337924477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f09500680b0 0x7f095019f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.464+0000 7f094fe00700 1 --2- 172.21.15.78:0/337924477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0950106060 0x7f095019fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.464+0000 7f0955400700 1 -- 172.21.15.78:0/337924477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f09500680b0 msgr2=0x7f095019f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.464+0000 7f0955400700 1 --2- 172.21.15.78:0/337924477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f09500680b0 0x7f095019f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.464+0000 7f0955400700 1 -- 172.21.15.78:0/337924477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0950106060 msgr2=0x7f095019fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.464+0000 7f0955400700 1 --2- 172.21.15.78:0/337924477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0950106060 0x7f095019fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.464+0000 7f0955400700 1 -- 172.21.15.78:0/337924477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09440095f0 con 0x7f0950199690 2024-03-14T10:05:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.465+0000 7f0955400700 1 --2- 172.21.15.78:0/337924477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0950199690 0x7f0950199b40 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f0948007ee0 tx=0x7f094800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:09.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.465+0000 7f094d600700 1 -- 172.21.15.78:0/337924477 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948017070 con 0x7f0950199690 2024-03-14T10:05:09.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.465+0000 7f094d600700 1 -- 172.21.15.78:0/337924477 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f094800f040 con 0x7f0950199690 2024-03-14T10:05:09.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.465+0000 7f0957400700 1 -- 172.21.15.78:0/337924477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09501a63e0 con 0x7f0950199690 2024-03-14T10:05:09.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.465+0000 7f094d600700 1 -- 172.21.15.78:0/337924477 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948005580 con 0x7f0950199690 2024-03-14T10:05:09.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.465+0000 7f0957400700 1 -- 172.21.15.78:0/337924477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09501a6910 con 0x7f0950199690 2024-03-14T10:05:09.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.466+0000 7f0957400700 1 -- 172.21.15.78:0/337924477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0950102410 con 0x7f0950199690 2024-03-14T10:05:09.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.468+0000 7f094d600700 1 -- 172.21.15.78:0/337924477 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 27) v1 ==== 135519+0+0 (secure 0 0 0) 0x7f094800db20 con 0x7f0950199690 2024-03-14T10:05:09.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.469+0000 7f094d600700 1 --2- 172.21.15.78:0/337924477 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f09380a0c80 0x7f09380a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T10:05:09.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.469+0000 7f094d600700 1 -- 172.21.15.78:0/337924477 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8553+0+0 (secure 0 0 0) 0x7f09480cc0c0 con 0x7f0950199690 2024-03-14T10:05:09.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.469+0000 7f0954a00700 1 --2- 172.21.15.78:0/337924477 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f09380a0c80 0x7f09380a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T10:05:09.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.470+0000 7f0954a00700 1 --2- 172.21.15.78:0/337924477 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f09380a0c80 0x7f09380a3140 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0944009990 tx=0x7f0944003680 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T10:05:09.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.471+0000 7f094d600700 1 -- 172.21.15.78:0/337924477 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f09480956d0 con 0x7f0950199690 2024-03-14T10:05:09.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:09 smithi152 ceph-mon[48279]: pgmap v341: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 KiB/s wr, 25 op/s 2024-03-14T10:05:09.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:09 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/2580956152' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-14T10:05:09.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:09 smithi152 ceph-mon[48279]: osdmap e98: 12 total, 12 up, 12 in 2024-03-14T10:05:09.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:09 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1358600183' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-14T10:05:09.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:09 smithi132 ceph-mon[47959]: pgmap v341: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 KiB/s wr, 25 op/s 2024-03-14T10:05:09.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:09 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/2580956152' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-14T10:05:09.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:09 smithi132 ceph-mon[47959]: osdmap e98: 12 total, 12 up, 12 in 2024-03-14T10:05:09.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:09 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1358600183' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-14T10:05:09.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:09 smithi078 ceph-mon[38143]: pgmap v341: 129 pgs: 129 active+clean; 251 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 KiB/s wr, 25 op/s 2024-03-14T10:05:09.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:09 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/2580956152' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-14T10:05:09.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:09 smithi078 ceph-mon[38143]: osdmap e98: 12 total, 12 up, 12 in 2024-03-14T10:05:09.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:09 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1358600183' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-14T10:05:09.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:09.716+0000 7f0957400700 1 -- 172.21.15.78:0/337924477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0950067ac0 con 0x7f0950199690 2024-03-14T10:05:10.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:10.242+0000 7f094d600700 1 -- 172.21.15.78:0/337924477 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v100) v1 ==== 178+0+0 (secure 0 0 0) 0x7f0948098cf0 con 0x7f0950199690 2024-03-14T10:05:10.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:10.248+0000 7f0957400700 1 -- 172.21.15.78:0/337924477 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f09380a0c80 msgr2=0x7f09380a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:10.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:10.248+0000 7f0957400700 1 --2- 172.21.15.78:0/337924477 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f09380a0c80 0x7f09380a3140 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0944009990 tx=0x7f0944003680 comp rx=0 tx=0).stop 2024-03-14T10:05:10.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:10.249+0000 7f0957400700 1 -- 172.21.15.78:0/337924477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0950199690 msgr2=0x7f0950199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T10:05:10.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:10.249+0000 7f0957400700 1 --2- 172.21.15.78:0/337924477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0950199690 0x7f0950199b40 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f0948007ee0 tx=0x7f094800d3b0 comp rx=0 tx=0).stop 2024-03-14T10:05:10.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:10.250+0000 7f0957400700 1 -- 172.21.15.78:0/337924477 shutdown_connections 2024-03-14T10:05:10.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:10.250+0000 7f0957400700 1 --2- 172.21.15.78:0/337924477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0950199690 0x7f0950199b40 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:10.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:10.250+0000 7f0957400700 1 --2- 172.21.15.78:0/337924477 >> [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] conn(0x7f09500680b0 0x7f095019f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:10.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:10.250+0000 7f0957400700 1 --2- 172.21.15.78:0/337924477 >> [v2:172.21.15.78:6800/2192201293,v1:172.21.15.78:6801/2192201293] conn(0x7f09380a0c80 0x7f09380a3140 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:10.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:10.250+0000 7f0957400700 1 --2- 172.21.15.78:0/337924477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0950106060 0x7f095019fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T10:05:10.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:10.250+0000 7f0957400700 1 -- 172.21.15.78:0/337924477 >> 172.21.15.78:0/337924477 conn(0x7f0950076d80 msgr2=0x7f095010bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T10:05:10.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:10.250+0000 7f0957400700 1 -- 172.21.15.78:0/337924477 shutdown_connections 2024-03-14T10:05:10.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-03-14T10:05:10.251+0000 7f0957400700 1 -- 172.21.15.78:0/337924477 wait complete. 2024-03-14T10:05:10.253 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data_ec' removed 2024-03-14T10:05:10.281 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-14T10:05:10.346 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-14T10:05:10.381 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-14T10:05:10.421 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-14T10:05:10.421 DEBUG:teuthology.orchestra.run.smithi078:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-14T10:05:10.462 DEBUG:teuthology.orchestra.run.smithi132:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-14T10:05:10.480 DEBUG:teuthology.orchestra.run.smithi152:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-14T10:05:10.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:10 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/1358600183' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-14T10:05:10.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:10 smithi152 ceph-mon[48279]: osdmap e99: 12 total, 12 up, 12 in 2024-03-14T10:05:10.499 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:10 smithi152 ceph-mon[48279]: from='client.? 172.21.15.78:0/337924477' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-14T10:05:10.499 INFO:tasks.cephadm:Stopping all daemons... 2024-03-14T10:05:10.500 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-14T10:05:10.500 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mon.a 2024-03-14T10:05:10.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:10 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/1358600183' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-14T10:05:10.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:10 smithi132 ceph-mon[47959]: osdmap e99: 12 total, 12 up, 12 in 2024-03-14T10:05:10.520 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:10 smithi132 ceph-mon[47959]: from='client.? 172.21.15.78:0/337924477' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-14T10:05:10.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:10 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/1358600183' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-14T10:05:10.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:10 smithi078 ceph-mon[38143]: osdmap e99: 12 total, 12 up, 12 in 2024-03-14T10:05:10.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:10 smithi078 ceph-mon[38143]: from='client.? 172.21.15.78:0/337924477' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-14T10:05:10.864 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:10 smithi078 systemd[1]: Stopping Ceph mon.a for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:05:11.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mon-a[38120]: 2024-03-14T10:05:10.863+0000 7f52db800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:05:11.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mon-a[38120]: 2024-03-14T10:05:10.863+0000 7f52db800700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-14T10:05:11.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Mar 14 10:05:11 smithi078 bash[135386]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mon-a 2024-03-14T10:05:11.634 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mon.a.service' 2024-03-14T10:05:11.735 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:05:11.735 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-14T10:05:11.736 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-14T10:05:11.736 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mon.b 2024-03-14T10:05:12.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:11 smithi132 systemd[1]: Stopping Ceph mon.b for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:05:12.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:11 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mon-b[47936]: 2024-03-14T10:05:11.982+0000 7fae06dd6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:05:12.020 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:11 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mon-b[47936]: 2024-03-14T10:05:11.982+0000 7fae06dd6700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-14T10:05:12.717 INFO:journalctl@ceph.mon.b.smithi132.stdout:Mar 14 10:05:12 smithi132 bash[91663]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mon-b 2024-03-14T10:05:12.724 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mon.b.service' 2024-03-14T10:05:12.778 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:05:12.778 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-14T10:05:12.779 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-14T10:05:12.779 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mon.c 2024-03-14T10:05:13.249 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:12 smithi152 systemd[1]: Stopping Ceph mon.c for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:05:13.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:13 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mon-c[48256]: 2024-03-14T10:05:13.024+0000 7f8001de5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:05:13.250 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:13 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mon-c[48256]: 2024-03-14T10:05:13.024+0000 7f8001de5700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-14T10:05:13.656 INFO:journalctl@ceph.mon.c.smithi152.stdout:Mar 14 10:05:13 smithi152 bash[91779]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mon-c 2024-03-14T10:05:13.663 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mon.c.service' 2024-03-14T10:05:13.741 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:05:13.742 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-14T10:05:13.742 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-14T10:05:13.742 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mgr.x 2024-03-14T10:05:14.131 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 10:05:13 smithi078 systemd[1]: Stopping Ceph mgr.x for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:05:14.395 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 10:05:14 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T10:05:14.129+0000 7fd972c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:05:14.396 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 10:05:14 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x[38394]: 2024-03-14T10:05:14.129+0000 7fd972c00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-14T10:05:14.780 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Mar 14 10:05:14 smithi078 bash[135528]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-x 2024-03-14T10:05:14.806 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-14T10:05:14.889 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:05:14.889 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-14T10:05:14.889 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-14T10:05:14.890 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mgr.y 2024-03-14T10:05:15.246 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 10:05:14 smithi132 systemd[1]: Stopping Ceph mgr.y for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:05:15.520 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 10:05:15 smithi132 bash[91802]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-y 2024-03-14T10:05:15.520 INFO:journalctl@ceph.mgr.y.smithi132.stdout:Mar 14 10:05:15 smithi132 systemd[1]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-14T10:05:15.609 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-14T10:05:15.646 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:05:15.646 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-14T10:05:15.646 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-14T10:05:15.646 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mgr.z 2024-03-14T10:05:15.999 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 10:05:15 smithi152 systemd[1]: Stopping Ceph mgr.z for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:05:16.428 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 10:05:16 smithi152 bash[91924]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mgr-z 2024-03-14T10:05:16.428 INFO:journalctl@ceph.mgr.z.smithi152.stdout:Mar 14 10:05:16 smithi152 systemd[1]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-14T10:05:16.434 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-14T10:05:16.471 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:05:16.471 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-14T10:05:16.471 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-14T10:05:16.471 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.0 2024-03-14T10:05:16.803 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 14 10:05:16 smithi078 systemd[1]: Stopping Ceph osd.0 for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:05:17.303 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 14 10:05:16 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-0[56324]: 2024-03-14T10:05:16.845+0000 7f18e7400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:05:17.303 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 14 10:05:16 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-0[56324]: 2024-03-14T10:05:16.845+0000 7f18e7400700 -1 osd.0 100 *** Got signal Terminated *** 2024-03-14T10:05:17.303 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 14 10:05:16 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-0[56324]: 2024-03-14T10:05:16.845+0000 7f18e7400700 -1 osd.0 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T10:05:23.053 INFO:journalctl@ceph.osd.0.smithi078.stdout:Mar 14 10:05:22 smithi078 bash[135670]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-0 2024-03-14T10:05:24.191 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.0.service' 2024-03-14T10:05:24.278 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:05:24.278 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-14T10:05:24.279 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-14T10:05:24.279 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.1 2024-03-14T10:05:24.532 INFO:journalctl@ceph.osd.1.smithi132.stdout:Mar 14 10:05:24 smithi132 systemd[1]: Stopping Ceph osd.1 for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:05:25.020 INFO:journalctl@ceph.osd.1.smithi132.stdout:Mar 14 10:05:24 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-1[52934]: 2024-03-14T10:05:24.530+0000 7fed57357700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:05:25.020 INFO:journalctl@ceph.osd.1.smithi132.stdout:Mar 14 10:05:24 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-1[52934]: 2024-03-14T10:05:24.530+0000 7fed57357700 -1 osd.1 100 *** Got signal Terminated *** 2024-03-14T10:05:25.020 INFO:journalctl@ceph.osd.1.smithi132.stdout:Mar 14 10:05:24 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-1[52934]: 2024-03-14T10:05:24.530+0000 7fed57357700 -1 osd.1 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T10:05:30.520 INFO:journalctl@ceph.osd.1.smithi132.stdout:Mar 14 10:05:30 smithi132 bash[91947]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-1 2024-03-14T10:05:31.335 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.1.service' 2024-03-14T10:05:31.376 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:05:31.376 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-14T10:05:31.377 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-14T10:05:31.377 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.2 2024-03-14T10:05:31.749 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 14 10:05:31 smithi152 systemd[1]: Stopping Ceph osd.2 for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:05:31.749 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 14 10:05:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-2[54057]: 2024-03-14T10:05:31.613+0000 7f749edb0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:05:31.749 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 14 10:05:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-2[54057]: 2024-03-14T10:05:31.613+0000 7f749edb0700 -1 osd.2 100 *** Got signal Terminated *** 2024-03-14T10:05:31.749 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 14 10:05:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-2[54057]: 2024-03-14T10:05:31.613+0000 7f749edb0700 -1 osd.2 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T10:05:37.499 INFO:journalctl@ceph.osd.2.smithi152.stdout:Mar 14 10:05:37 smithi152 bash[92070]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-2 2024-03-14T10:05:38.275 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.2.service' 2024-03-14T10:05:38.316 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:05:38.317 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-14T10:05:38.317 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-14T10:05:38.317 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.3 2024-03-14T10:05:38.697 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 14 10:05:38 smithi078 systemd[1]: Stopping Ceph osd.3 for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:05:39.053 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 14 10:05:38 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-3[64184]: 2024-03-14T10:05:38.692+0000 7fc405a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:05:39.053 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 14 10:05:38 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-3[64184]: 2024-03-14T10:05:38.692+0000 7fc405a00700 -1 osd.3 100 *** Got signal Terminated *** 2024-03-14T10:05:39.053 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 14 10:05:38 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-3[64184]: 2024-03-14T10:05:38.692+0000 7fc405a00700 -1 osd.3 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T10:05:43.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:43 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:43.436+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:44.304 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:43 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:43.960+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:05:44.749 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:44 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:44.302+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:19.529872+0000 front 2024-03-14T10:05:19.529680+0000 (oldest deadline 2024-03-14T10:05:43.629702+0000) 2024-03-14T10:05:44.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:44 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:44.423+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:44.803 INFO:journalctl@ceph.osd.3.smithi078.stdout:Mar 14 10:05:44 smithi078 bash[135958]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-3 2024-03-14T10:05:45.238 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:44 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:44.954+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:05:45.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:45 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:45.078+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:05:45.553 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:45 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:45.448+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:45.749 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:45 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:45.282+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:19.529872+0000 front 2024-03-14T10:05:19.529680+0000 (oldest deadline 2024-03-14T10:05:43.629702+0000) 2024-03-14T10:05:46.053 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:45 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:45.943+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:05:46.088 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.3.service' 2024-03-14T10:05:46.191 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:05:46.192 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-14T10:05:46.192 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-14T10:05:46.192 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.4 2024-03-14T10:05:46.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:45 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:45.828+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:05:46.437 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 14 10:05:46 smithi132 systemd[1]: Stopping Ceph osd.4 for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:05:46.437 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:46 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:46.111+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:05:46.749 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:46 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:46.315+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:19.529872+0000 front 2024-03-14T10:05:19.529680+0000 (oldest deadline 2024-03-14T10:05:43.629702+0000) 2024-03-14T10:05:46.770 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 14 10:05:46 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-4[61611]: 2024-03-14T10:05:46.435+0000 7f7a58933700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:05:46.770 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 14 10:05:46 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-4[61611]: 2024-03-14T10:05:46.435+0000 7f7a58933700 -1 osd.4 100 *** Got signal Terminated *** 2024-03-14T10:05:46.770 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 14 10:05:46 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-4[61611]: 2024-03-14T10:05:46.435+0000 7f7a58933700 -1 osd.4 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T10:05:46.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:46 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:46.482+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:47.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:46 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:46.873+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:05:47.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:46 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:46.943+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:05:47.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:47 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:47.080+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:05:47.749 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:47 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:47.266+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:19.529872+0000 front 2024-03-14T10:05:19.529680+0000 (oldest deadline 2024-03-14T10:05:43.629702+0000) 2024-03-14T10:05:47.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:47 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:47.439+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:48.246 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:47 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:47.918+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:05:48.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:47 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:47.971+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:05:48.499 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:48 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:48.245+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:19.529872+0000 front 2024-03-14T10:05:19.529680+0000 (oldest deadline 2024-03-14T10:05:43.629702+0000) 2024-03-14T10:05:48.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:48 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:48.103+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:05:48.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:48 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:48.396+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:49.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:48 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:48.911+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:05:49.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:48 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:48.990+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:05:49.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:49 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:49.120+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:05:49.749 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:49 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:49.265+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:19.529872+0000 front 2024-03-14T10:05:19.529680+0000 (oldest deadline 2024-03-14T10:05:43.629702+0000) 2024-03-14T10:05:49.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:49 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:49.368+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:50.227 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:49 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:49.877+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:05:50.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:50 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:50.002+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:05:50.499 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:50 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:50.227+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:19.529872+0000 front 2024-03-14T10:05:19.529680+0000 (oldest deadline 2024-03-14T10:05:43.629702+0000) 2024-03-14T10:05:50.499 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:50 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:50.227+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:26.530724+0000 front 2024-03-14T10:05:26.530750+0000 (oldest deadline 2024-03-14T10:05:50.030781+0000) 2024-03-14T10:05:50.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:50 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:50.145+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:05:50.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:50 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:50.325+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:51.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:50 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:50.900+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:05:51.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:50 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:50.900+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:05:51.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:50 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:50.959+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:05:51.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:50 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:50.959+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.729014+0000 front 2024-03-14T10:05:28.729078+0000 (oldest deadline 2024-03-14T10:05:50.429055+0000) 2024-03-14T10:05:51.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:51 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:51.175+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:05:51.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:05:51 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:05:51.341+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:05:51.749 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:51 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:51.248+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:19.529872+0000 front 2024-03-14T10:05:19.529680+0000 (oldest deadline 2024-03-14T10:05:43.629702+0000) 2024-03-14T10:05:51.749 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:51 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:51.248+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:26.530724+0000 front 2024-03-14T10:05:26.530750+0000 (oldest deadline 2024-03-14T10:05:50.030781+0000) 2024-03-14T10:05:51.749 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:05:51 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:05:51.275+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:05:51.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:51 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:51.355+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:52.208 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:51 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:51.909+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:05:52.208 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:51 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:51.910+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:05:52.270 INFO:journalctl@ceph.osd.4.smithi132.stdout:Mar 14 10:05:52 smithi132 bash[92212]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-4 2024-03-14T10:05:52.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:52 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:52.152+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:05:52.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:52 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:52.152+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:05:52.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:51 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:51.941+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:05:52.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:51 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:51.941+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.729014+0000 front 2024-03-14T10:05:28.729078+0000 (oldest deadline 2024-03-14T10:05:50.429055+0000) 2024-03-14T10:05:52.499 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:52 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:52.208+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:19.529872+0000 front 2024-03-14T10:05:19.529680+0000 (oldest deadline 2024-03-14T10:05:43.629702+0000) 2024-03-14T10:05:52.499 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:52 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:52.208+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:26.530724+0000 front 2024-03-14T10:05:26.530750+0000 (oldest deadline 2024-03-14T10:05:50.030781+0000) 2024-03-14T10:05:52.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:05:52 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:05:52.261+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:05:52.616 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:05:52 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:05:52.360+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:05:52.805 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:52 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:52.366+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:53.231 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:52 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:52.953+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:05:53.231 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:52 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:52.953+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:05:53.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:53 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:53.149+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:05:53.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:53 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:53.149+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:05:53.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:52 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:52.942+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:05:53.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:52 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:52.942+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.729014+0000 front 2024-03-14T10:05:28.729078+0000 (oldest deadline 2024-03-14T10:05:50.429055+0000) 2024-03-14T10:05:53.349 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.4.service' 2024-03-14T10:05:53.390 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:05:53.390 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-14T10:05:53.390 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-14T10:05:53.390 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.5 2024-03-14T10:05:53.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:05:53 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:05:53.304+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:05:53.499 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:53 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:53.231+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:19.529872+0000 front 2024-03-14T10:05:19.529680+0000 (oldest deadline 2024-03-14T10:05:43.629702+0000) 2024-03-14T10:05:53.499 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:53 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:53.231+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:26.530724+0000 front 2024-03-14T10:05:26.530750+0000 (oldest deadline 2024-03-14T10:05:50.030781+0000) 2024-03-14T10:05:53.499 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:53 smithi152 systemd[1]: Stopping Ceph osd.5 for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:05:53.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:05:53 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:05:53.317+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:05:53.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:53 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:53.403+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:53.980 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:53 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:53.633+0000 7f5fa1111700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:05:53.980 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:53 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:53.633+0000 7f5fa1111700 -1 osd.5 100 *** Got signal Terminated *** 2024-03-14T10:05:53.980 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:53 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:53.633+0000 7f5fa1111700 -1 osd.5 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T10:05:54.241 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:53 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:53.979+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:05:54.242 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:53 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:53.979+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:05:54.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:53 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:53.933+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:05:54.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:53 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:53.933+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.729014+0000 front 2024-03-14T10:05:28.729078+0000 (oldest deadline 2024-03-14T10:05:50.429055+0000) 2024-03-14T10:05:54.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:05:54 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:05:54.301+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:05:54.499 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:54 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:54.240+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:19.529872+0000 front 2024-03-14T10:05:19.529680+0000 (oldest deadline 2024-03-14T10:05:43.629702+0000) 2024-03-14T10:05:54.499 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:54 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:54.240+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:26.530724+0000 front 2024-03-14T10:05:26.530750+0000 (oldest deadline 2024-03-14T10:05:50.030781+0000) 2024-03-14T10:05:54.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:05:54 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:05:54.276+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:05:54.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:54 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:54.147+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:05:54.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:54 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:54.147+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:05:54.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:54 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:54.370+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:54.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:54 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:54.370+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:05:55.294 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:55 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:55.218+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:19.529872+0000 front 2024-03-14T10:05:19.529680+0000 (oldest deadline 2024-03-14T10:05:43.629702+0000) 2024-03-14T10:05:55.295 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:55 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:55.218+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:26.530724+0000 front 2024-03-14T10:05:26.530750+0000 (oldest deadline 2024-03-14T10:05:50.030781+0000) 2024-03-14T10:05:55.295 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:55 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:55.029+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:05:55.295 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:55 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:55.029+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:05:55.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:54 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:54.901+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:05:55.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:54 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:54.901+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.729014+0000 front 2024-03-14T10:05:28.729078+0000 (oldest deadline 2024-03-14T10:05:50.429055+0000) 2024-03-14T10:05:55.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:05:55 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:05:55.245+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:05:55.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:55 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:55.153+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:05:55.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:55 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:55.153+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:05:55.749 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:05:55 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:05:55.293+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:05:55.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:55 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:55.369+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:55.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:55 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:55.369+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:05:56.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:55 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:55.861+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:05:56.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:55 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:55.861+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.729014+0000 front 2024-03-14T10:05:28.729078+0000 (oldest deadline 2024-03-14T10:05:50.429055+0000) 2024-03-14T10:05:56.329 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:56 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:56.257+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:19.529872+0000 front 2024-03-14T10:05:19.529680+0000 (oldest deadline 2024-03-14T10:05:43.629702+0000) 2024-03-14T10:05:56.330 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:56 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:56.257+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:26.530724+0000 front 2024-03-14T10:05:26.530750+0000 (oldest deadline 2024-03-14T10:05:50.030781+0000) 2024-03-14T10:05:56.330 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:56 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:56.008+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:05:56.330 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:56 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:56.008+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:05:56.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:05:56 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:05:56.266+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:05:56.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:56 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:56.111+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:05:56.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:56 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:56.111+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:05:56.749 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:05:56 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:05:56.329+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:05:56.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:56 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:56.399+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:56.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:56 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:56.399+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:05:57.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:56 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:56.870+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:05:57.304 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:56 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:56.870+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.729014+0000 front 2024-03-14T10:05:28.729078+0000 (oldest deadline 2024-03-14T10:05:50.429055+0000) 2024-03-14T10:05:57.343 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:57 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:57.249+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:19.529872+0000 front 2024-03-14T10:05:19.529680+0000 (oldest deadline 2024-03-14T10:05:43.629702+0000) 2024-03-14T10:05:57.343 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:57 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:57.249+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:26.530724+0000 front 2024-03-14T10:05:26.530750+0000 (oldest deadline 2024-03-14T10:05:50.030781+0000) 2024-03-14T10:05:57.344 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:57 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:57.005+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:05:57.344 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:57 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:57.005+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:05:57.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:05:57 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:05:57.287+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:05:57.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:57 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:57.134+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:05:57.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:57 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:57.134+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:05:57.749 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:05:57 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:05:57.343+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:05:57.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:57 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:57.408+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:57.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:57 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:57.408+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:05:58.264 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:58 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:58.001+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:05:58.265 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:58 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:58.001+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:05:58.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:57 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:57.882+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:05:58.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:57 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:57.882+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.729014+0000 front 2024-03-14T10:05:28.729078+0000 (oldest deadline 2024-03-14T10:05:50.429055+0000) 2024-03-14T10:05:58.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:05:58 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:05:58.290+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:05:58.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:58 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:58.176+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:05:58.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:58 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:58.176+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:05:58.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:58 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:58.176+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:05:58.749 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:58 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:58.264+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:19.529872+0000 front 2024-03-14T10:05:19.529680+0000 (oldest deadline 2024-03-14T10:05:43.629702+0000) 2024-03-14T10:05:58.749 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:58 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5[61902]: 2024-03-14T10:05:58.264+0000 7f5f9b703700 -1 osd.5 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:26.530724+0000 front 2024-03-14T10:05:26.530750+0000 (oldest deadline 2024-03-14T10:05:50.030781+0000) 2024-03-14T10:05:58.749 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:05:58 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:05:58.381+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:05:58.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:58 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:58.407+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:58.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:58 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:58.407+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:05:59.221 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:58 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:58.953+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:05:59.221 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:58 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:58.953+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:05:59.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:58 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:58.861+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:05:59.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:58 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:58.861+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.729014+0000 front 2024-03-14T10:05:28.729078+0000 (oldest deadline 2024-03-14T10:05:50.429055+0000) 2024-03-14T10:05:59.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:58 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:58.861+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.230869+0000 front 2024-03-14T10:05:36.230882+0000 (oldest deadline 2024-03-14T10:05:58.531028+0000) 2024-03-14T10:05:59.499 INFO:journalctl@ceph.osd.5.smithi152.stdout:Mar 14 10:05:59 smithi152 bash[92334]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-5 2024-03-14T10:05:59.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:05:59 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:05:59.333+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:05:59.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:05:59 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:05:59.311+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:05:59.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:59 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:59.133+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:05:59.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:59 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:59.133+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:05:59.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:05:59 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:05:59.133+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:05:59.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:59 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:59.366+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:05:59.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:59 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:59.366+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:05:59.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:05:59 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:05:59.366+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:00.131 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:59 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:59.958+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:00.131 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:59 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:59.958+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:00.131 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:05:59 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:05:59.958+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:00.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:59 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:59.872+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:06:00.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:59 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:59.872+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.729014+0000 front 2024-03-14T10:05:28.729078+0000 (oldest deadline 2024-03-14T10:05:50.429055+0000) 2024-03-14T10:06:00.304 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:05:59 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:05:59.872+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.230869+0000 front 2024-03-14T10:05:36.230882+0000 (oldest deadline 2024-03-14T10:05:58.531028+0000) 2024-03-14T10:06:00.463 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:00 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:00.290+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:00.471 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.5.service' 2024-03-14T10:06:00.512 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:06:00.512 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-14T10:06:00.512 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-14T10:06:00.513 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.6 2024-03-14T10:06:00.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:00 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:00.336+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:06:00.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:00 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:00.336+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:35.436126+0000 front 2024-03-14T10:05:35.436063+0000 (oldest deadline 2024-03-14T10:06:00.136130+0000) 2024-03-14T10:06:00.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:00 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:00.177+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:00.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:00 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:00.177+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:00.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:00 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:00.177+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:00.592 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:00 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:00.340+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:00.592 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:00 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:00.340+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:00.593 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:00 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:00.340+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:00.873 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:00 smithi078 systemd[1]: Stopping Ceph osd.6 for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:06:01.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:00 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:00.933+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:01.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:00 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:00.933+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:01.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:00 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:00.933+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:01.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:00 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:00.872+0000 7fc385000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:06:01.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:00 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:00.872+0000 7fc385000700 -1 osd.6 100 *** Got signal Terminated *** 2024-03-14T10:06:01.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:00 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:00.872+0000 7fc385000700 -1 osd.6 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T10:06:01.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:00 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:00.913+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:06:01.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:00 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:00.913+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.729014+0000 front 2024-03-14T10:05:28.729078+0000 (oldest deadline 2024-03-14T10:05:50.429055+0000) 2024-03-14T10:06:01.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:00 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:00.913+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.230869+0000 front 2024-03-14T10:05:36.230882+0000 (oldest deadline 2024-03-14T10:05:58.531028+0000) 2024-03-14T10:06:01.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:01 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:01.375+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:06:01.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:01 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:01.375+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:35.436126+0000 front 2024-03-14T10:05:35.436063+0000 (oldest deadline 2024-03-14T10:06:00.136130+0000) 2024-03-14T10:06:01.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:01 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:01.218+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:01.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:01 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:01.218+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:01.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:01 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:01.218+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:01.749 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:01 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:01.261+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:01.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:01 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:01.384+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:01.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:01 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:01.384+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:01.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:01 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:01.385+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:02.215 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:01 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:01.957+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:02.215 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:01 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:01.957+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:02.215 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:01 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:01.957+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:02.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:01 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:01.950+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:06:02.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:01 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:01.950+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.729014+0000 front 2024-03-14T10:05:28.729078+0000 (oldest deadline 2024-03-14T10:05:50.429055+0000) 2024-03-14T10:06:02.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:01 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:01.950+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.230869+0000 front 2024-03-14T10:05:36.230882+0000 (oldest deadline 2024-03-14T10:05:58.531028+0000) 2024-03-14T10:06:02.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:02 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:02.215+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:02.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:02 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:02.215+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:02.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:02 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:02.402+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:06:02.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:02 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:02.402+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:35.436126+0000 front 2024-03-14T10:05:35.436063+0000 (oldest deadline 2024-03-14T10:06:00.136130+0000) 2024-03-14T10:06:02.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:02 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:02.192+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:02.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:02 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:02.193+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:02.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:02 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:02.193+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:02.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:02 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:02.389+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:02.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:02 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:02.389+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:02.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:02 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:02.389+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:03.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:03 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:03.223+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:03.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:03 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:03.223+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:03.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:02 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:02.975+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:03.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:02 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:02.975+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:03.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:02 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:02.975+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:03.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:02 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:02.997+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:06:03.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:02 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:02.997+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.729014+0000 front 2024-03-14T10:05:28.729078+0000 (oldest deadline 2024-03-14T10:05:50.429055+0000) 2024-03-14T10:06:03.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:02 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:02.997+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.230869+0000 front 2024-03-14T10:05:36.230882+0000 (oldest deadline 2024-03-14T10:05:58.531028+0000) 2024-03-14T10:06:03.433 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:03 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:03.152+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:03.433 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:03 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:03.152+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:03.434 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:03 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:03.152+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:03.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:03 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:03.433+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:06:03.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:03 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:03.433+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:35.436126+0000 front 2024-03-14T10:05:35.436063+0000 (oldest deadline 2024-03-14T10:06:00.136130+0000) 2024-03-14T10:06:03.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:03 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:03.432+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:03.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:03 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:03.432+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:03.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:03 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:03.432+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:04.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:04 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:04.174+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:04.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:04 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:04.174+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:04.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:03 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:03.990+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:04.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:03 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:03.990+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:04.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:03 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:03.990+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:04.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:03 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:03.954+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:06:04.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:03 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:03.954+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.729014+0000 front 2024-03-14T10:05:28.729078+0000 (oldest deadline 2024-03-14T10:05:50.429055+0000) 2024-03-14T10:06:04.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:03 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:03.954+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.230869+0000 front 2024-03-14T10:05:36.230882+0000 (oldest deadline 2024-03-14T10:05:58.531028+0000) 2024-03-14T10:06:04.436 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:04 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:04.106+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:04.437 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:04 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:04.106+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:04.437 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:04 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:04.106+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:04.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:04 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:04.436+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:06:04.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:04 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:04.436+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:35.436126+0000 front 2024-03-14T10:05:35.436063+0000 (oldest deadline 2024-03-14T10:06:00.136130+0000) 2024-03-14T10:06:04.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:04 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:04.439+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:04.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:04 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:04.439+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:04.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:04 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:04.439+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:05.212 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:04 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:04.953+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:05.212 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:04 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:04.953+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:05.212 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:04 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:04.953+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:05.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:04 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:04.974+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:17.527632+0000 front 2024-03-14T10:05:17.527774+0000 (oldest deadline 2024-03-14T10:05:43.427765+0000) 2024-03-14T10:06:05.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:04 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:04.974+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.729014+0000 front 2024-03-14T10:05:28.729078+0000 (oldest deadline 2024-03-14T10:05:50.429055+0000) 2024-03-14T10:06:05.303 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:04 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6[72193]: 2024-03-14T10:06:04.974+0000 7fc37c200700 -1 osd.6 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.230869+0000 front 2024-03-14T10:05:36.230882+0000 (oldest deadline 2024-03-14T10:05:58.531028+0000) 2024-03-14T10:06:05.398 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:05 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:05.093+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:05.398 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:05 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:05.093+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:05.398 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:05 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:05.093+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:05.398 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:05 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:05.093+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:05.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:05 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:05.212+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:05.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:05 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:05.212+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:05.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:05 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:05.397+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:06:05.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:05 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:05.397+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:35.436126+0000 front 2024-03-14T10:05:35.436063+0000 (oldest deadline 2024-03-14T10:06:00.136130+0000) 2024-03-14T10:06:05.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:05 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:05.462+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:05.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:05 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:05.462+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:05.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:05 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:05.462+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:05.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:05 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:05.462+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:06.188 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:05 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:05.908+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:06.188 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:05 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:05.908+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:06.188 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:05 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:05.908+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:06.352 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:06.073+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:06.353 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:06.073+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:06.353 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:06.073+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:06.353 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:06.073+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:06.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:06 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:06.188+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:06.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:06 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:06.188+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:06.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:06.352+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:06:06.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:06 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:06.352+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:35.436126+0000 front 2024-03-14T10:05:35.436063+0000 (oldest deadline 2024-03-14T10:06:00.136130+0000) 2024-03-14T10:06:06.803 INFO:journalctl@ceph.osd.6.smithi078.stdout:Mar 14 10:06:06 smithi078 bash[136216]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-6 2024-03-14T10:06:06.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:06 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:06.460+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:06.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:06 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:06.460+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:06.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:06 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:06.460+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:06.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:06 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:06.460+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:07.193 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:06 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:06.885+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:07.193 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:06 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:06.885+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:07.193 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:06 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:06.885+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:07.398 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:07 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:07.051+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:07.398 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:07 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:07.051+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:07.398 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:07 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:07.051+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:07.399 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:07 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:07.051+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:07.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:07 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:07.193+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:07.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:07 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:07.193+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:07.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:07 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:07.193+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:42.029021+0000 front 2024-03-14T10:05:42.029128+0000 (oldest deadline 2024-03-14T10:06:06.728977+0000) 2024-03-14T10:06:07.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:07 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:07.398+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:06:07.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:07 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:07.398+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:35.436126+0000 front 2024-03-14T10:05:35.436063+0000 (oldest deadline 2024-03-14T10:06:00.136130+0000) 2024-03-14T10:06:07.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:07 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:07.398+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:41.237464+0000 front 2024-03-14T10:05:41.237517+0000 (oldest deadline 2024-03-14T10:06:07.137584+0000) 2024-03-14T10:06:07.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:07 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:07.505+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:07.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:07 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:07.505+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:07.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:07 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:07.505+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:07.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:07 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:07.505+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:08.164 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:07 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:07.890+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:08.165 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:07 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:07.890+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:08.165 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:07 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:07.890+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:08.196 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.6.service' 2024-03-14T10:06:08.281 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:06:08.282 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-14T10:06:08.282 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-14T10:06:08.282 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.7 2024-03-14T10:06:08.312 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:08 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:08.059+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:08.312 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:08 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:08.059+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:08.312 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:08 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:08.059+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:08.312 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:08 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:08.059+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:08.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:08 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:08.164+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:08.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:08 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:08.164+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:08.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:08 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:08.164+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:42.029021+0000 front 2024-03-14T10:05:42.029128+0000 (oldest deadline 2024-03-14T10:06:06.728977+0000) 2024-03-14T10:06:08.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:08 smithi132 systemd[1]: Stopping Ceph osd.7 for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:06:08.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:08 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:08.370+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:06:08.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:08 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:08.370+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:35.436126+0000 front 2024-03-14T10:05:35.436063+0000 (oldest deadline 2024-03-14T10:06:00.136130+0000) 2024-03-14T10:06:08.771 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:08 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:08.370+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:41.237464+0000 front 2024-03-14T10:05:41.237517+0000 (oldest deadline 2024-03-14T10:06:07.137584+0000) 2024-03-14T10:06:08.771 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:08 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:08.523+0000 7fc271492700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:06:08.771 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:08 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:08.523+0000 7fc271492700 -1 osd.7 100 *** Got signal Terminated *** 2024-03-14T10:06:08.771 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:08 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:08.523+0000 7fc271492700 -1 osd.7 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T10:06:08.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:08 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:08.499+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:08.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:08 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:08.499+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:08.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:08 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:08.499+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:08.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:08 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:08.499+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:09.203 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:08 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:08.876+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:09.203 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:08 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:08.876+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:09.203 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:08 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:08.876+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:09.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:09 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:09.099+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:09.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:09 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:09.099+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:09.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:09 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:09.099+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:09.350 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:09 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:09.099+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:09.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:09 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:09.203+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:09.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:09 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:09.203+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:09.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:09 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:09.203+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:42.029021+0000 front 2024-03-14T10:05:42.029128+0000 (oldest deadline 2024-03-14T10:06:06.728977+0000) 2024-03-14T10:06:09.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:09 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:09.350+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:06:09.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:09 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:09.350+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:35.436126+0000 front 2024-03-14T10:05:35.436063+0000 (oldest deadline 2024-03-14T10:06:00.136130+0000) 2024-03-14T10:06:09.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:09 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:09.350+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:41.237464+0000 front 2024-03-14T10:05:41.237517+0000 (oldest deadline 2024-03-14T10:06:07.137584+0000) 2024-03-14T10:06:09.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:09 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:09.517+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:09.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:09 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:09.517+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:09.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:09 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:09.517+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:09.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:09 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:09.517+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:10.227 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:09 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:09.917+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:10.227 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:09 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:09.918+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:10.228 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:09 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:09.918+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:10.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:10 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:10.226+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:10.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:10 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:10.226+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:10.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:10 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:10.226+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:42.029021+0000 front 2024-03-14T10:05:42.029128+0000 (oldest deadline 2024-03-14T10:06:06.728977+0000) 2024-03-14T10:06:10.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:10 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:10.112+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:10.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:10 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:10.112+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:10.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:10 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:10.112+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:10.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:10 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:10.112+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:10.521 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:10 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:10.351+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:06:10.521 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:10 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:10.352+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:35.436126+0000 front 2024-03-14T10:05:35.436063+0000 (oldest deadline 2024-03-14T10:06:00.136130+0000) 2024-03-14T10:06:10.521 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:10 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:10.352+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:41.237464+0000 front 2024-03-14T10:05:41.237517+0000 (oldest deadline 2024-03-14T10:06:07.137584+0000) 2024-03-14T10:06:10.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:10.524+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:10.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:10.524+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:10.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:10.524+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:10.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:10 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:10.524+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:11.216 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:10 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:10.894+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:11.216 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:10 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:10.894+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:11.216 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:10 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:10.894+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:11.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:11 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:11.215+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:11.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:11 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:11.215+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:11.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:11 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:11.215+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:42.029021+0000 front 2024-03-14T10:05:42.029128+0000 (oldest deadline 2024-03-14T10:06:06.728977+0000) 2024-03-14T10:06:11.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:11 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:11.344+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:06:11.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:11 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:11.344+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:35.436126+0000 front 2024-03-14T10:05:35.436063+0000 (oldest deadline 2024-03-14T10:06:00.136130+0000) 2024-03-14T10:06:11.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:11 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:11.344+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:41.237464+0000 front 2024-03-14T10:05:41.237517+0000 (oldest deadline 2024-03-14T10:06:07.137584+0000) 2024-03-14T10:06:11.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:11 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:11.140+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:11.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:11 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:11.140+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:11.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:11 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:11.140+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:11.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:11 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:11.140+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:11.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:11 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:11.538+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:11.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:11 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:11.538+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:11.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:11 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:11.538+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:11.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:11 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:11.538+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:12.194 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:11 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:11.848+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:12.194 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:11 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:11.848+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:12.194 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:11 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:11.848+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:12.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:12 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:12.193+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:12.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:12 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:12.193+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:12.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:12 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:12.193+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:42.029021+0000 front 2024-03-14T10:05:42.029128+0000 (oldest deadline 2024-03-14T10:06:06.728977+0000) 2024-03-14T10:06:12.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:12 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:12.386+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:06:12.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:12 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:12.386+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:35.436126+0000 front 2024-03-14T10:05:35.436063+0000 (oldest deadline 2024-03-14T10:06:00.136130+0000) 2024-03-14T10:06:12.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:12 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:12.386+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:41.237464+0000 front 2024-03-14T10:05:41.237517+0000 (oldest deadline 2024-03-14T10:06:07.137584+0000) 2024-03-14T10:06:12.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:12 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:12.144+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:12.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:12 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:12.144+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:12.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:12 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:12.144+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:12.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:12 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:12.144+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:12.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:12 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:12.526+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:12.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:12 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:12.526+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:12.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:12 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:12.526+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:12.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:12 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:12.526+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:13.218 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:12 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:12.864+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:13.218 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:12 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:12.864+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:13.218 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:12 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:12.864+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:13.400 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:13 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:13.102+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:13.400 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:13 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:13.102+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:13.400 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:13 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:13.102+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:13.400 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:13 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:13.102+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:13.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:13 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:13.218+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:13.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:13 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:13.218+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:13.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:13 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:13.218+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:42.029021+0000 front 2024-03-14T10:05:42.029128+0000 (oldest deadline 2024-03-14T10:06:06.728977+0000) 2024-03-14T10:06:13.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:13 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:13.218+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.631741+0000 front 2024-03-14T10:05:50.631671+0000 (oldest deadline 2024-03-14T10:06:12.331483+0000) 2024-03-14T10:06:13.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:13 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:13.400+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.633338+0000 front 2024-03-14T10:05:28.633298+0000 (oldest deadline 2024-03-14T10:05:50.333465+0000) 2024-03-14T10:06:13.770 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:13 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:13.400+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:35.436126+0000 front 2024-03-14T10:05:35.436063+0000 (oldest deadline 2024-03-14T10:06:00.136130+0000) 2024-03-14T10:06:13.771 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:13 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:13.400+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:41.237464+0000 front 2024-03-14T10:05:41.237517+0000 (oldest deadline 2024-03-14T10:06:07.137584+0000) 2024-03-14T10:06:13.771 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:13 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7[70321]: 2024-03-14T10:06:13.400+0000 7fc26da88700 -1 osd.7 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:47.138044+0000 front 2024-03-14T10:05:47.137991+0000 (oldest deadline 2024-03-14T10:06:13.038030+0000) 2024-03-14T10:06:13.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:13 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:13.544+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:13.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:13 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:13.544+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:13.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:13 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:13.544+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:13.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:13 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:13.544+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:14.182 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:13 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:13.880+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:14.182 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:13 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:13.880+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:14.182 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:13 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:13.880+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:14.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:14 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:14.182+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:14.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:14 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:14.182+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:14.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:14 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:14.182+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:42.029021+0000 front 2024-03-14T10:05:42.029128+0000 (oldest deadline 2024-03-14T10:06:06.728977+0000) 2024-03-14T10:06:14.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:14 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:14.182+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.631741+0000 front 2024-03-14T10:05:50.631671+0000 (oldest deadline 2024-03-14T10:06:12.331483+0000) 2024-03-14T10:06:14.520 INFO:journalctl@ceph.osd.7.smithi132.stdout:Mar 14 10:06:14 smithi132 bash[92466]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-7 2024-03-14T10:06:14.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:14 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:14.056+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:14.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:14 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:14.056+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:14.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:14 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:14.056+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:14.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:14 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:14.056+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:14.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:14 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:14.542+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:14.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:14 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:14.542+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:14.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:14 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:14.542+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:14.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:14 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:14.542+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:15.138 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:14 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:14.845+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:15.139 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:14 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:14.845+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:15.139 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:14 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:14.845+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:15.139 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:14 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:14.845+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:15.446 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:15 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:15.052+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:15.446 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:15 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:15.052+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:15.446 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:15 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:15.052+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:15.446 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:15 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:15.052+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:15.446 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:15 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:15.052+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:15.453 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.7.service' 2024-03-14T10:06:15.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:15 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:15.137+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:15.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:15 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:15.137+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:15.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:15 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:15.137+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:42.029021+0000 front 2024-03-14T10:05:42.029128+0000 (oldest deadline 2024-03-14T10:06:06.728977+0000) 2024-03-14T10:06:15.499 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:15 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:15.137+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.631741+0000 front 2024-03-14T10:05:50.631671+0000 (oldest deadline 2024-03-14T10:06:12.331483+0000) 2024-03-14T10:06:15.534 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:06:15.535 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-14T10:06:15.535 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-14T10:06:15.535 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.8 2024-03-14T10:06:15.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:15 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:15.510+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:15.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:15 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:15.510+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:15.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:15 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:15.510+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:15.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:15 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:15.510+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:15.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:15 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:15.510+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:51.086918+0000 front 2024-03-14T10:05:51.087163+0000 (oldest deadline 2024-03-14T10:06:15.186904+0000) 2024-03-14T10:06:15.848 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:15 smithi152 systemd[1]: Stopping Ceph osd.8 for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:06:15.848 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:15 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:15.771+0000 7f5c2b6de700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:06:15.848 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:15 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:15.771+0000 7f5c2b6de700 -1 osd.8 100 *** Got signal Terminated *** 2024-03-14T10:06:15.848 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:15 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:15.771+0000 7f5c2b6de700 -1 osd.8 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T10:06:16.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:16 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:16.091+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:16.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:16 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:16.091+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:16.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:16 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:16.091+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:42.029021+0000 front 2024-03-14T10:05:42.029128+0000 (oldest deadline 2024-03-14T10:06:06.728977+0000) 2024-03-14T10:06:16.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:16 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:16.091+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.631741+0000 front 2024-03-14T10:05:50.631671+0000 (oldest deadline 2024-03-14T10:06:12.331483+0000) 2024-03-14T10:06:16.250 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:15 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:15.847+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:16.250 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:15 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:15.847+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:16.250 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:15 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:15.847+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:16.250 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:15 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:15.847+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:16.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:16 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:16.057+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:16.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:16 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:16.057+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:16.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:16 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:16.057+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:16.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:16 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:16.057+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:16.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:16 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:16.057+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:16.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:16 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:16.484+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:16.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:16 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:16.484+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:16.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:16 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:16.484+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:16.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:16 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:16.484+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:16.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:16 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:16.484+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:51.086918+0000 front 2024-03-14T10:05:51.087163+0000 (oldest deadline 2024-03-14T10:06:15.186904+0000) 2024-03-14T10:06:17.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:17 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:17.059+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:17.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:17 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:17.059+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:17.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:17 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:17.059+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:42.029021+0000 front 2024-03-14T10:05:42.029128+0000 (oldest deadline 2024-03-14T10:06:06.728977+0000) 2024-03-14T10:06:17.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:17 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:17.059+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.631741+0000 front 2024-03-14T10:05:50.631671+0000 (oldest deadline 2024-03-14T10:06:12.331483+0000) 2024-03-14T10:06:17.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:16 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:16.897+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:17.250 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:16 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:16.897+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:17.250 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:16 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:16.897+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:17.250 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:16 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:16.897+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:17.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:17 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:17.009+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:17.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:17 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:17.009+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:17.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:17 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:17.009+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:17.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:17 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:17.009+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:17.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:17 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:17.009+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:17.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:17 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:17.465+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:17.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:17 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:17.465+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:17.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:17 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:17.465+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:17.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:17 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:17.465+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:17.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:17 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:17.465+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:51.086918+0000 front 2024-03-14T10:05:51.087163+0000 (oldest deadline 2024-03-14T10:06:15.186904+0000) 2024-03-14T10:06:18.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:18 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:18.010+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:18.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:18 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:18.010+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:18.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:18 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:18.010+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:42.029021+0000 front 2024-03-14T10:05:42.029128+0000 (oldest deadline 2024-03-14T10:06:06.728977+0000) 2024-03-14T10:06:18.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:18 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:18.010+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.631741+0000 front 2024-03-14T10:05:50.631671+0000 (oldest deadline 2024-03-14T10:06:12.331483+0000) 2024-03-14T10:06:18.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:17 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:17.945+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:18.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:17 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:17.945+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:18.250 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:17 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:17.945+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:18.250 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:17 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:17.945+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:18.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:17 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:17.991+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:18.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:17 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:17.991+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:18.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:17 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:17.991+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:18.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:17 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:17.991+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:18.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:17 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:17.991+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:18.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:18 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:18.452+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:18.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:18 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:18.452+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:18.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:18 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:18.452+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:18.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:18 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:18.452+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:18.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:18 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:18.453+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:51.086918+0000 front 2024-03-14T10:05:51.087163+0000 (oldest deadline 2024-03-14T10:06:15.186904+0000) 2024-03-14T10:06:19.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:18 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:18.919+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:19.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:18 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:18.919+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:19.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:18 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:18.919+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:19.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:18 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:18.919+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:19.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:19 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:19.044+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:19.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:19 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:19.044+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:19.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:19 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:19.044+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:42.029021+0000 front 2024-03-14T10:05:42.029128+0000 (oldest deadline 2024-03-14T10:06:06.728977+0000) 2024-03-14T10:06:19.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:19 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:19.044+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.631741+0000 front 2024-03-14T10:05:50.631671+0000 (oldest deadline 2024-03-14T10:06:12.331483+0000) 2024-03-14T10:06:19.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:18 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:18.966+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:19.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:18 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:18.966+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:19.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:18 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:18.966+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:19.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:18 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:18.966+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:19.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:18 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:18.966+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:19.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:19 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:19.491+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:19.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:19 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:19.491+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:19.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:19 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:19.491+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:19.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:19 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:19.491+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:19.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:19 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:19.491+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:51.086918+0000 front 2024-03-14T10:05:51.087163+0000 (oldest deadline 2024-03-14T10:06:15.186904+0000) 2024-03-14T10:06:20.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:20 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:20.089+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:25.725924+0000 front 2024-03-14T10:05:25.725953+0000 (oldest deadline 2024-03-14T10:05:50.425923+0000) 2024-03-14T10:06:20.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:20 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:20.089+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.128154+0000 front 2024-03-14T10:05:36.128234+0000 (oldest deadline 2024-03-14T10:06:02.028391+0000) 2024-03-14T10:06:20.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:20 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:20.089+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:42.029021+0000 front 2024-03-14T10:05:42.029128+0000 (oldest deadline 2024-03-14T10:06:06.728977+0000) 2024-03-14T10:06:20.249 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:20 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8[70596]: 2024-03-14T10:06:20.089+0000 7f5c274d3700 -1 osd.8 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.631741+0000 front 2024-03-14T10:05:50.631671+0000 (oldest deadline 2024-03-14T10:06:12.331483+0000) 2024-03-14T10:06:20.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:19 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:19.918+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:20.250 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:19 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:19.918+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:20.250 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:19 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:19.918+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:20.250 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:19 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:19.918+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:20.250 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:19 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:19.918+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:20.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:20 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:19.999+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:20.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:20 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:19.999+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:20.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:20 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:19.999+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:20.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:20 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:19.999+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:20.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:20 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:19.999+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:20.806 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:20 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:20.524+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:20.806 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:20 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:20.524+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:20.806 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:20 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:20.524+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:20.806 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:20 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:20.524+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:20.806 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:20 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:20.524+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:51.086918+0000 front 2024-03-14T10:05:51.087163+0000 (oldest deadline 2024-03-14T10:06:15.186904+0000) 2024-03-14T10:06:20.806 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:20 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:20.524+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:55.187724+0000 front 2024-03-14T10:05:55.187633+0000 (oldest deadline 2024-03-14T10:06:19.887398+0000) 2024-03-14T10:06:21.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:20 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:20.946+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:21.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:20 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:20.946+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:21.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:20 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:20.946+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:21.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:20 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:20.946+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:21.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:20 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:20.946+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:21.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:20 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:20.986+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:21.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:20 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:20.986+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:21.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:20 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:20.986+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:21.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:20 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:20.986+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:21.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:20 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:20.986+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:21.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:20 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:20.986+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:57.738286+0000 front 2024-03-14T10:05:57.738199+0000 (oldest deadline 2024-03-14T10:06:20.038027+0000) 2024-03-14T10:06:21.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:21.530+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:21.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:21.530+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:21.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:21.530+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:21.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:21.530+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:21.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:21.530+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:51.086918+0000 front 2024-03-14T10:05:51.087163+0000 (oldest deadline 2024-03-14T10:06:15.186904+0000) 2024-03-14T10:06:21.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:21 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:21.530+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:55.187724+0000 front 2024-03-14T10:05:55.187633+0000 (oldest deadline 2024-03-14T10:06:19.887398+0000) 2024-03-14T10:06:21.996 INFO:journalctl@ceph.osd.8.smithi152.stdout:Mar 14 10:06:21 smithi152 bash[92588]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-8 2024-03-14T10:06:22.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:21 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:21.994+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:22.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:21 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:21.994+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:22.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:21 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:21.994+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:22.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:21 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:21.996+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:22.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:21 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:21.996+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:22.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:22 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:22.032+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:22.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:22 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:22.032+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:22.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:22 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:22.032+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:22.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:22 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:22.032+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:22.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:22 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:22.032+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:22.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:22 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:22.032+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:57.738286+0000 front 2024-03-14T10:05:57.738199+0000 (oldest deadline 2024-03-14T10:06:20.038027+0000) 2024-03-14T10:06:22.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:22 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:22.523+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:22.803 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:22 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:22.523+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:22.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:22 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:22.523+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:22.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:22 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:22.523+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:22.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:22 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:22.523+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:51.086918+0000 front 2024-03-14T10:05:51.087163+0000 (oldest deadline 2024-03-14T10:06:15.186904+0000) 2024-03-14T10:06:22.804 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:22 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:22.523+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:55.187724+0000 front 2024-03-14T10:05:55.187633+0000 (oldest deadline 2024-03-14T10:06:19.887398+0000) 2024-03-14T10:06:22.901 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.8.service' 2024-03-14T10:06:22.942 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:06:22.942 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-14T10:06:22.943 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-14T10:06:22.943 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.9 2024-03-14T10:06:23.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:22 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:22.946+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:23.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:22 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:22.946+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:23.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:22 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:22.946+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:23.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:22 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:22.946+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:23.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:22 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:22.946+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:23.303 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:23 smithi078 systemd[1]: Stopping Ceph osd.9 for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:06:23.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:23 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:23.044+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:23.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:23 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:23.044+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:23.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:23 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:23.044+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:23.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:23 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:23.044+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:23.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:23 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:23.044+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:23.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:23 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:23.044+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:57.738286+0000 front 2024-03-14T10:05:57.738199+0000 (oldest deadline 2024-03-14T10:06:20.038027+0000) 2024-03-14T10:06:23.572 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:23 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:23.304+0000 7fa7ad600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:06:23.572 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:23 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:23.304+0000 7fa7ad600700 -1 osd.9 100 *** Got signal Terminated *** 2024-03-14T10:06:23.572 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:23 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:23.304+0000 7fa7ad600700 -1 osd.9 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T10:06:24.053 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:23 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:23.572+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:24.053 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:23 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:23.572+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:24.053 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:23 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:23.572+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:24.053 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:23 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:23.572+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:24.053 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:23 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:23.572+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:51.086918+0000 front 2024-03-14T10:05:51.087163+0000 (oldest deadline 2024-03-14T10:06:15.186904+0000) 2024-03-14T10:06:24.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:23 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:23.572+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:55.187724+0000 front 2024-03-14T10:05:55.187633+0000 (oldest deadline 2024-03-14T10:06:19.887398+0000) 2024-03-14T10:06:24.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:23 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:23.981+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:24.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:23 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:23.981+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:24.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:23 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:23.981+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:24.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:23 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:23.981+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:24.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:23 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:23.981+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:24.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:24 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:24.024+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:24.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:24 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:24.024+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:24.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:24 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:24.024+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:24.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:24 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:24.024+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:24.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:24 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:24.024+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:24.521 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:24 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:24.024+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:57.738286+0000 front 2024-03-14T10:05:57.738199+0000 (oldest deadline 2024-03-14T10:06:20.038027+0000) 2024-03-14T10:06:25.053 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:24 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:24.586+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:25.053 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:24 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:24.586+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:25.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:24 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:24.586+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:25.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:24 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:24.586+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:25.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:24 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:24.586+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:51.086918+0000 front 2024-03-14T10:05:51.087163+0000 (oldest deadline 2024-03-14T10:06:15.186904+0000) 2024-03-14T10:06:25.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:24 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:24.586+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:55.187724+0000 front 2024-03-14T10:05:55.187633+0000 (oldest deadline 2024-03-14T10:06:19.887398+0000) 2024-03-14T10:06:25.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:24 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:24.989+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:25.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:24 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:24.989+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:25.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:24 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:24.989+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:25.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:24 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:24.989+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:25.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:24 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:24.989+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:25.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:24 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:24.995+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:25.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:24 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:24.995+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:25.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:24 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:24.995+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:25.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:24 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:24.995+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:25.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:24 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:24.995+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:25.271 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:24 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:24.995+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:57.738286+0000 front 2024-03-14T10:05:57.738199+0000 (oldest deadline 2024-03-14T10:06:20.038027+0000) 2024-03-14T10:06:26.053 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:25 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:25.611+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:26.053 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:25 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:25.611+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:26.053 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:25 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:25.611+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:26.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:25 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:25.611+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:26.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:25 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:25.611+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:51.086918+0000 front 2024-03-14T10:05:51.087163+0000 (oldest deadline 2024-03-14T10:06:15.186904+0000) 2024-03-14T10:06:26.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:25 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:25.611+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:55.187724+0000 front 2024-03-14T10:05:55.187633+0000 (oldest deadline 2024-03-14T10:06:19.887398+0000) 2024-03-14T10:06:26.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:25 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:25.989+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:26.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:25 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:25.989+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:26.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:25 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:25.989+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:26.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:25 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:25.989+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:26.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:25 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:25.989+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:26.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:26 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:26.045+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:26.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:26 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:26.045+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:26.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:26 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:26.045+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:26.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:26 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:26.045+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:26.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:26 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:26.045+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:26.520 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:26 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:26.045+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:57.738286+0000 front 2024-03-14T10:05:57.738199+0000 (oldest deadline 2024-03-14T10:06:20.038027+0000) 2024-03-14T10:06:27.053 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:26 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:26.601+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:27.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:26 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:26.601+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:27.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:26 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:26.601+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:27.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:26 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:26.601+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:27.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:26 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:26.601+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:51.086918+0000 front 2024-03-14T10:05:51.087163+0000 (oldest deadline 2024-03-14T10:06:15.186904+0000) 2024-03-14T10:06:27.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:26 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:26.601+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:55.187724+0000 front 2024-03-14T10:05:55.187633+0000 (oldest deadline 2024-03-14T10:06:19.887398+0000) 2024-03-14T10:06:27.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:27 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:27.000+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:27.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:27 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:27.000+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:27.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:27 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:27.000+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:27.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:27 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:27.000+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:27.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:27 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:27.000+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:27.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:27 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:27.000+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:57.738286+0000 front 2024-03-14T10:05:57.738199+0000 (oldest deadline 2024-03-14T10:06:20.038027+0000) 2024-03-14T10:06:27.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:27 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:27.000+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:02.939315+0000 front 2024-03-14T10:06:02.939331+0000 (oldest deadline 2024-03-14T10:06:26.439017+0000) 2024-03-14T10:06:27.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:27 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:27.015+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:27.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:27 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:27.015+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:27.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:27 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:27.015+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:27.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:27 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:27.015+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:27.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:27 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:27.015+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:28.053 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:27 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:27.636+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:21.280822+0000 front 2024-03-14T10:05:21.280846+0000 (oldest deadline 2024-03-14T10:05:42.981001+0000) 2024-03-14T10:06:28.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:27 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:27.636+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:28.283338+0000 front 2024-03-14T10:05:28.283019+0000 (oldest deadline 2024-03-14T10:05:53.583268+0000) 2024-03-14T10:06:28.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:27 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:27.636+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:33.583869+0000 front 2024-03-14T10:05:33.584023+0000 (oldest deadline 2024-03-14T10:05:58.883868+0000) 2024-03-14T10:06:28.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:27 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:27.636+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:43.584946+0000 front 2024-03-14T10:05:43.585040+0000 (oldest deadline 2024-03-14T10:06:04.685247+0000) 2024-03-14T10:06:28.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:27 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:27.636+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:51.086918+0000 front 2024-03-14T10:05:51.087163+0000 (oldest deadline 2024-03-14T10:06:15.186904+0000) 2024-03-14T10:06:28.054 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:27 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9[80322]: 2024-03-14T10:06:27.636+0000 7fa7a3e00700 -1 osd.9 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:55.187724+0000 front 2024-03-14T10:05:55.187633+0000 (oldest deadline 2024-03-14T10:06:19.887398+0000) 2024-03-14T10:06:28.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:27 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:27.994+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:28.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:27 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:27.994+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:28.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:27 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:27.994+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:28.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:27 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:27.994+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:28.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:27 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:27.994+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:28.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:28 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:28.010+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:28.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:28 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:28.010+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:28.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:28 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:28.010+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:28.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:28 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:28.010+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:28.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:28 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:28.010+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:28.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:28 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:28.010+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:57.738286+0000 front 2024-03-14T10:05:57.738199+0000 (oldest deadline 2024-03-14T10:06:20.038027+0000) 2024-03-14T10:06:28.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:28 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:28.010+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:02.939315+0000 front 2024-03-14T10:06:02.939331+0000 (oldest deadline 2024-03-14T10:06:26.439017+0000) 2024-03-14T10:06:29.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:28 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:28.964+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:29.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:28 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:28.964+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:29.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:28 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:28.964+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:29.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:28 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:28.964+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:29.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:28 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:28.964+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:29.271 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:28 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:28.964+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:57.738286+0000 front 2024-03-14T10:05:57.738199+0000 (oldest deadline 2024-03-14T10:06:20.038027+0000) 2024-03-14T10:06:29.271 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:28 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:28.964+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:02.939315+0000 front 2024-03-14T10:06:02.939331+0000 (oldest deadline 2024-03-14T10:06:26.439017+0000) 2024-03-14T10:06:29.303 INFO:journalctl@ceph.osd.9.smithi078.stdout:Mar 14 10:06:28 smithi078 bash[136473]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-9 2024-03-14T10:06:29.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:29 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:29.035+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:29.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:29 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:29.035+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:29.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:29 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:29.035+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:29.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:29 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:29.035+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:29.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:29 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:29.035+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:29.500 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:29 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:29.035+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:04.450528+0000 front 2024-03-14T10:06:04.450474+0000 (oldest deadline 2024-03-14T10:06:28.550365+0000) 2024-03-14T10:06:30.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:29 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:29.919+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:30.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:29 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:29.919+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:30.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:29 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:29.919+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:30.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:29 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:29.919+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:30.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:29 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:29.919+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:30.271 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:29 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:29.919+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:57.738286+0000 front 2024-03-14T10:05:57.738199+0000 (oldest deadline 2024-03-14T10:06:20.038027+0000) 2024-03-14T10:06:30.271 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:29 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:29.919+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:02.939315+0000 front 2024-03-14T10:06:02.939331+0000 (oldest deadline 2024-03-14T10:06:26.439017+0000) 2024-03-14T10:06:30.418 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.9.service' 2024-03-14T10:06:30.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:30 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:30.041+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:30.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:30 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:30.041+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:30.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:30 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:30.041+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:30.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:30 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:30.041+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:30.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:30 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:30.041+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:30.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:30 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:30.041+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:04.450528+0000 front 2024-03-14T10:06:04.450474+0000 (oldest deadline 2024-03-14T10:06:28.550365+0000) 2024-03-14T10:06:30.503 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:06:30.503 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-14T10:06:30.503 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-14T10:06:30.503 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.10 2024-03-14T10:06:30.941 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:30 smithi132 systemd[1]: Stopping Ceph osd.10 for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:06:30.941 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:30 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:30.736+0000 7f5cc9f3d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:06:30.941 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:30 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:30.736+0000 7f5cc9f3d700 -1 osd.10 100 *** Got signal Terminated *** 2024-03-14T10:06:30.941 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:30 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:30.736+0000 7f5cc9f3d700 -1 osd.10 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T10:06:31.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:30 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:30.940+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:31.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:30 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:30.940+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:31.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:30 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:30.940+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:31.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:30 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:30.940+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:31.271 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:30 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:30.940+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:31.271 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:30 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:30.940+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:57.738286+0000 front 2024-03-14T10:05:57.738199+0000 (oldest deadline 2024-03-14T10:06:20.038027+0000) 2024-03-14T10:06:31.271 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:30 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:30.940+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:02.939315+0000 front 2024-03-14T10:06:02.939331+0000 (oldest deadline 2024-03-14T10:06:26.439017+0000) 2024-03-14T10:06:31.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:31.001+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:31.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:31.001+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:31.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:31.001+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:31.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:31.001+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:31.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:31.001+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:31.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:31.001+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:04.450528+0000 front 2024-03-14T10:06:04.450474+0000 (oldest deadline 2024-03-14T10:06:28.550365+0000) 2024-03-14T10:06:32.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:31.973+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:32.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:31.973+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:32.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:31.973+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:32.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:31.973+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:32.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:31.973+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:32.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:31 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:31.973+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:04.450528+0000 front 2024-03-14T10:06:04.450474+0000 (oldest deadline 2024-03-14T10:06:28.550365+0000) 2024-03-14T10:06:32.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:31 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:31.913+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:32.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:31 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:31.913+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:32.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:31 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:31.913+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:32.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:31 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:31.913+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:32.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:31 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:31.913+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:32.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:31 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:31.913+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:57.738286+0000 front 2024-03-14T10:05:57.738199+0000 (oldest deadline 2024-03-14T10:06:20.038027+0000) 2024-03-14T10:06:32.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:31 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:31.913+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:02.939315+0000 front 2024-03-14T10:06:02.939331+0000 (oldest deadline 2024-03-14T10:06:26.439017+0000) 2024-03-14T10:06:33.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:32 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:32.969+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:33.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:32 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:32.969+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:33.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:32 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:32.969+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:33.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:32 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:32.969+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:33.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:32 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:32.969+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:33.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:32 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:32.969+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:04.450528+0000 front 2024-03-14T10:06:04.450474+0000 (oldest deadline 2024-03-14T10:06:28.550365+0000) 2024-03-14T10:06:33.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:32 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:32.941+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:33.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:32 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:32.941+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:33.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:32 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:32.941+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:33.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:32 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:32.941+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:33.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:32 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:32.941+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:33.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:32 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:32.941+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:57.738286+0000 front 2024-03-14T10:05:57.738199+0000 (oldest deadline 2024-03-14T10:06:20.038027+0000) 2024-03-14T10:06:33.271 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:32 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:32.941+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:02.939315+0000 front 2024-03-14T10:06:02.939331+0000 (oldest deadline 2024-03-14T10:06:26.439017+0000) 2024-03-14T10:06:34.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:33 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:33.978+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:34.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:33 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:33.978+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:34.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:33 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:33.978+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:34.271 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:33 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:33.978+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:34.271 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:33 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:33.978+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:34.271 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:33 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:33.978+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:57.738286+0000 front 2024-03-14T10:05:57.738199+0000 (oldest deadline 2024-03-14T10:06:20.038027+0000) 2024-03-14T10:06:34.271 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:33 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:33.978+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:02.939315+0000 front 2024-03-14T10:06:02.939331+0000 (oldest deadline 2024-03-14T10:06:26.439017+0000) 2024-03-14T10:06:34.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:34 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:34.000+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:34.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:34 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:34.000+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:34.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:34 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:34.000+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:34.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:34 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:34.000+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:34.500 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:34 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:34.000+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:34.500 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:34 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:34.000+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:04.450528+0000 front 2024-03-14T10:06:04.450474+0000 (oldest deadline 2024-03-14T10:06:28.550365+0000) 2024-03-14T10:06:35.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:34 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:34.968+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:20.333318+0000 front 2024-03-14T10:05:20.333269+0000 (oldest deadline 2024-03-14T10:05:44.432950+0000) 2024-03-14T10:06:35.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:34 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:34.968+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:29.133778+0000 front 2024-03-14T10:05:29.133798+0000 (oldest deadline 2024-03-14T10:05:51.434117+0000) 2024-03-14T10:06:35.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:34 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:34.968+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:31.434589+0000 front 2024-03-14T10:05:31.434688+0000 (oldest deadline 2024-03-14T10:05:57.334603+0000) 2024-03-14T10:06:35.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:34 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:34.968+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2024-03-14T10:05:38.435797+0000 front 2024-03-14T10:05:38.435948+0000 (oldest deadline 2024-03-14T10:06:04.836163+0000) 2024-03-14T10:06:35.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:34 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:34.968+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:50.136965+0000 front 2024-03-14T10:05:50.136920+0000 (oldest deadline 2024-03-14T10:06:14.237090+0000) 2024-03-14T10:06:35.270 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:34 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:34.968+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:57.738286+0000 front 2024-03-14T10:05:57.738199+0000 (oldest deadline 2024-03-14T10:06:20.038027+0000) 2024-03-14T10:06:35.271 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:34 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10[79121]: 2024-03-14T10:06:34.968+0000 7f5cc452f700 -1 osd.10 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:02.939315+0000 front 2024-03-14T10:06:02.939331+0000 (oldest deadline 2024-03-14T10:06:26.439017+0000) 2024-03-14T10:06:35.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:35.010+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:35.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:35.010+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:35.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:35.010+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:35.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:35.010+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:35.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:35.010+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:35.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:35.010+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:04.450528+0000 front 2024-03-14T10:06:04.450474+0000 (oldest deadline 2024-03-14T10:06:28.550365+0000) 2024-03-14T10:06:35.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:35.010+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-03-14T10:06:08.551011+0000 front 2024-03-14T10:06:08.550943+0000 (oldest deadline 2024-03-14T10:06:34.450887+0000) 2024-03-14T10:06:36.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:35.962+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:36.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:35.962+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:36.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:35.962+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:36.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:35.962+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:36.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:35.962+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:36.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:35.962+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:04.450528+0000 front 2024-03-14T10:06:04.450474+0000 (oldest deadline 2024-03-14T10:06:28.550365+0000) 2024-03-14T10:06:36.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:35 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:35.962+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-03-14T10:06:08.551011+0000 front 2024-03-14T10:06:08.550943+0000 (oldest deadline 2024-03-14T10:06:34.450887+0000) 2024-03-14T10:06:36.773 INFO:journalctl@ceph.osd.10.smithi132.stdout:Mar 14 10:06:36 smithi132 bash[92728]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-10 2024-03-14T10:06:37.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:36 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:36.963+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:37.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:36 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:36.963+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:37.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:36 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:36.963+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:37.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:36 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:36.963+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:37.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:36 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:36.963+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:37.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:36 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:36.963+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:04.450528+0000 front 2024-03-14T10:06:04.450474+0000 (oldest deadline 2024-03-14T10:06:28.550365+0000) 2024-03-14T10:06:37.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:36 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:36.963+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-03-14T10:06:08.551011+0000 front 2024-03-14T10:06:08.550943+0000 (oldest deadline 2024-03-14T10:06:34.450887+0000) 2024-03-14T10:06:37.599 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.10.service' 2024-03-14T10:06:37.639 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:06:37.639 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-14T10:06:37.639 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-14T10:06:37.639 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.11 2024-03-14T10:06:37.999 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:37 smithi152 systemd[1]: Stopping Ceph osd.11 for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:06:37.999 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:37 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:37.869+0000 7f7ce09e4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:06:37.999 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:37 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:37.869+0000 7f7ce09e4700 -1 osd.11 100 *** Got signal Terminated *** 2024-03-14T10:06:37.999 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:37 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:37.869+0000 7f7ce09e4700 -1 osd.11 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T10:06:38.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:38.003+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:38.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:38.003+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:38.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:38.003+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:38.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:38.003+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:38.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:38.003+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:38.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:38.003+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:04.450528+0000 front 2024-03-14T10:06:04.450474+0000 (oldest deadline 2024-03-14T10:06:28.550365+0000) 2024-03-14T10:06:38.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:38.003+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-03-14T10:06:08.551011+0000 front 2024-03-14T10:06:08.550943+0000 (oldest deadline 2024-03-14T10:06:34.450887+0000) 2024-03-14T10:06:39.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:38.991+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:39.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:38.991+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:39.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:38.991+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:39.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:38.991+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:39.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:38.991+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:39.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:38.991+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:04.450528+0000 front 2024-03-14T10:06:04.450474+0000 (oldest deadline 2024-03-14T10:06:28.550365+0000) 2024-03-14T10:06:39.249 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:38 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:38.991+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-03-14T10:06:08.551011+0000 front 2024-03-14T10:06:08.550943+0000 (oldest deadline 2024-03-14T10:06:34.450887+0000) 2024-03-14T10:06:40.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:40 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:40.039+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:40.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:40 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:40.039+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:40.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:40 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:40.039+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:40.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:40 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:40.039+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:40.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:40 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:40.039+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:40.500 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:40 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:40.039+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:04.450528+0000 front 2024-03-14T10:06:04.450474+0000 (oldest deadline 2024-03-14T10:06:28.550365+0000) 2024-03-14T10:06:40.500 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:40 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:40.039+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-03-14T10:06:08.551011+0000 front 2024-03-14T10:06:08.550943+0000 (oldest deadline 2024-03-14T10:06:34.450887+0000) 2024-03-14T10:06:41.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:41 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:41.012+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:41.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:41 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:41.012+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:41.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:41 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:41.012+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:41.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:41 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:41.012+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:41.500 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:41 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:41.012+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:41.500 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:41 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:41.012+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:04.450528+0000 front 2024-03-14T10:06:04.450474+0000 (oldest deadline 2024-03-14T10:06:28.550365+0000) 2024-03-14T10:06:41.500 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:41 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:41.012+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-03-14T10:06:08.551011+0000 front 2024-03-14T10:06:08.550943+0000 (oldest deadline 2024-03-14T10:06:34.450887+0000) 2024-03-14T10:06:42.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:42 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:42.009+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2024-03-14T10:05:16.643170+0000 front 2024-03-14T10:05:16.643115+0000 (oldest deadline 2024-03-14T10:05:44.843777+0000) 2024-03-14T10:06:42.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:42 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:42.009+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6804 osd.1 since back 2024-03-14T10:05:24.844420+0000 front 2024-03-14T10:05:24.844435+0000 (oldest deadline 2024-03-14T10:05:50.144409+0000) 2024-03-14T10:06:42.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:42 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:42.009+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6804 osd.2 since back 2024-03-14T10:05:36.545911+0000 front 2024-03-14T10:05:36.545990+0000 (oldest deadline 2024-03-14T10:05:59.445961+0000) 2024-03-14T10:06:42.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:42 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:42.009+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6812 osd.4 since back 2024-03-14T10:05:49.848705+0000 front 2024-03-14T10:05:49.848757+0000 (oldest deadline 2024-03-14T10:06:13.948704+0000) 2024-03-14T10:06:42.499 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:42 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:42.009+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6812 osd.5 since back 2024-03-14T10:05:53.949036+0000 front 2024-03-14T10:05:53.949060+0000 (oldest deadline 2024-03-14T10:06:19.249139+0000) 2024-03-14T10:06:42.500 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:42 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:42.009+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2024-03-14T10:06:04.450528+0000 front 2024-03-14T10:06:04.450474+0000 (oldest deadline 2024-03-14T10:06:28.550365+0000) 2024-03-14T10:06:42.500 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:42 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:42.009+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.132:6820 osd.7 since back 2024-03-14T10:06:08.551011+0000 front 2024-03-14T10:06:08.550943+0000 (oldest deadline 2024-03-14T10:06:34.450887+0000) 2024-03-14T10:06:42.500 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:42 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11[79383]: 2024-03-14T10:06:42.009+0000 7f7cdb7d7700 -1 osd.11 100 heartbeat_check: no reply from 172.21.15.152:6820 osd.8 since back 2024-03-14T10:06:16.652282+0000 front 2024-03-14T10:06:16.652320+0000 (oldest deadline 2024-03-14T10:06:41.952180+0000) 2024-03-14T10:06:43.749 INFO:journalctl@ceph.osd.11.smithi152.stdout:Mar 14 10:06:43 smithi152 bash[92853]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-osd-11 2024-03-14T10:06:44.639 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@osd.11.service' 2024-03-14T10:06:44.681 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:06:44.681 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-14T10:06:44.681 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-14T10:06:44.681 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.a 2024-03-14T10:06:45.053 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 14 10:06:44 smithi078 systemd[1]: Stopping Ceph mds.a for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:06:45.553 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 14 10:06:45 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-a[88044]: 2024-03-14T10:06:45.059+0000 7f93a7600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:06:45.553 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 14 10:06:45 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-a[88044]: 2024-03-14T10:06:45.059+0000 7f93a7600700 -1 mds.a *** got signal Terminated *** 2024-03-14T10:06:49.303 INFO:journalctl@ceph.mds.a.smithi078.stdout:Mar 14 10:06:49 smithi078 bash[136736]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-a 2024-03-14T10:06:49.453 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.a.service' 2024-03-14T10:06:49.533 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:06:49.534 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-14T10:06:49.534 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-14T10:06:49.534 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.d 2024-03-14T10:06:49.803 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 14 10:06:49 smithi078 systemd[1]: Stopping Ceph mds.d for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:06:50.303 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 14 10:06:49 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-d[87089]: 2024-03-14T10:06:49.874+0000 7f839bc00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:06:50.303 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 14 10:06:49 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-d[87089]: 2024-03-14T10:06:49.874+0000 7f839bc00700 -1 mds.d *** got signal Terminated *** 2024-03-14T10:06:51.303 INFO:journalctl@ceph.mds.d.smithi078.stdout:Mar 14 10:06:51 smithi078 bash[136875]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-d 2024-03-14T10:06:51.495 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.d.service' 2024-03-14T10:06:51.572 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:06:51.573 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-14T10:06:51.573 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-14T10:06:51.573 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.g 2024-03-14T10:06:51.803 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 14 10:06:51 smithi078 systemd[1]: Stopping Ceph mds.g for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:06:52.303 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 14 10:06:51 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-g[87656]: 2024-03-14T10:06:51.916+0000 7fbc80e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:06:52.304 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 14 10:06:51 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-g[87656]: 2024-03-14T10:06:51.916+0000 7fbc80e00700 -1 mds.g *** got signal Terminated *** 2024-03-14T10:06:54.803 INFO:journalctl@ceph.mds.g.smithi078.stdout:Mar 14 10:06:54 smithi078 bash[137015]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-g 2024-03-14T10:06:54.984 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.g.service' 2024-03-14T10:06:55.064 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:06:55.065 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-14T10:06:55.065 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-14T10:06:55.065 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.j 2024-03-14T10:06:55.303 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 14 10:06:55 smithi078 systemd[1]: Stopping Ceph mds.j for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:06:55.803 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 14 10:06:55 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-j[86701]: 2024-03-14T10:06:55.423+0000 7f68f3a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:06:55.803 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 14 10:06:55 smithi078 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-j[86701]: 2024-03-14T10:06:55.423+0000 7f68f3a00700 -1 mds.j *** got signal Terminated *** 2024-03-14T10:06:57.054 INFO:journalctl@ceph.mds.j.smithi078.stdout:Mar 14 10:06:56 smithi078 bash[137155]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-j 2024-03-14T10:06:57.219 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.j.service' 2024-03-14T10:06:57.297 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:06:57.297 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-14T10:06:57.297 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-14T10:06:57.298 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.b 2024-03-14T10:06:57.770 INFO:journalctl@ceph.mds.b.smithi132.stdout:Mar 14 10:06:57 smithi132 systemd[1]: Stopping Ceph mds.b for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:06:57.771 INFO:journalctl@ceph.mds.b.smithi132.stdout:Mar 14 10:06:57 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-b[86027]: 2024-03-14T10:06:57.542+0000 7f17f22b4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:06:57.771 INFO:journalctl@ceph.mds.b.smithi132.stdout:Mar 14 10:06:57 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-b[86027]: 2024-03-14T10:06:57.542+0000 7f17f22b4700 -1 mds.b *** got signal Terminated *** 2024-03-14T10:07:02.020 INFO:journalctl@ceph.mds.b.smithi132.stdout:Mar 14 10:07:01 smithi132 bash[92983]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-b 2024-03-14T10:07:02.072 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.b.service' 2024-03-14T10:07:02.109 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:07:02.109 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-14T10:07:02.109 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-14T10:07:02.109 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.e 2024-03-14T10:07:02.382 INFO:journalctl@ceph.mds.e.smithi132.stdout:Mar 14 10:07:02 smithi132 systemd[1]: Stopping Ceph mds.e for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:07:02.770 INFO:journalctl@ceph.mds.e.smithi132.stdout:Mar 14 10:07:02 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-e[85339]: 2024-03-14T10:07:02.381+0000 7f89e0dad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:07:02.770 INFO:journalctl@ceph.mds.e.smithi132.stdout:Mar 14 10:07:02 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-e[85339]: 2024-03-14T10:07:02.381+0000 7f89e0dad700 -1 mds.e *** got signal Terminated *** 2024-03-14T10:07:06.733 INFO:journalctl@ceph.mds.e.smithi132.stdout:Mar 14 10:07:06 smithi132 bash[93123]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-e 2024-03-14T10:07:06.738 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.e.service' 2024-03-14T10:07:06.771 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:07:06.772 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-14T10:07:06.772 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-14T10:07:06.772 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.h 2024-03-14T10:07:07.020 INFO:journalctl@ceph.mds.h.smithi132.stdout:Mar 14 10:07:06 smithi132 systemd[1]: Stopping Ceph mds.h for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:07:07.520 INFO:journalctl@ceph.mds.h.smithi132.stdout:Mar 14 10:07:07 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-h[85684]: 2024-03-14T10:07:07.037+0000 7ff4b4f72700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:07:07.520 INFO:journalctl@ceph.mds.h.smithi132.stdout:Mar 14 10:07:07 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-h[85684]: 2024-03-14T10:07:07.037+0000 7ff4b4f72700 -1 mds.h *** got signal Terminated *** 2024-03-14T10:07:10.770 INFO:journalctl@ceph.mds.h.smithi132.stdout:Mar 14 10:07:10 smithi132 bash[93261]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-h 2024-03-14T10:07:10.812 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.h.service' 2024-03-14T10:07:10.845 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:07:10.846 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-14T10:07:10.846 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-14T10:07:10.846 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.k 2024-03-14T10:07:11.105 INFO:journalctl@ceph.mds.k.smithi132.stdout:Mar 14 10:07:10 smithi132 systemd[1]: Stopping Ceph mds.k for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:07:11.520 INFO:journalctl@ceph.mds.k.smithi132.stdout:Mar 14 10:07:11 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-k[84999]: 2024-03-14T10:07:11.104+0000 7fbc915bc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:07:11.520 INFO:journalctl@ceph.mds.k.smithi132.stdout:Mar 14 10:07:11 smithi132 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-k[84999]: 2024-03-14T10:07:11.104+0000 7fbc915bc700 -1 mds.k *** got signal Terminated *** 2024-03-14T10:07:14.770 INFO:journalctl@ceph.mds.k.smithi132.stdout:Mar 14 10:07:14 smithi132 bash[93400]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-k 2024-03-14T10:07:14.783 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.k.service' 2024-03-14T10:07:14.817 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:07:14.817 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-14T10:07:14.817 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-14T10:07:14.818 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.c 2024-03-14T10:07:15.249 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 14 10:07:14 smithi152 systemd[1]: Stopping Ceph mds.c for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:07:15.249 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 14 10:07:15 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-c[85458]: 2024-03-14T10:07:15.062+0000 7fe5e32cc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:07:15.249 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 14 10:07:15 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-c[85458]: 2024-03-14T10:07:15.062+0000 7fe5e32cc700 -1 mds.c *** got signal Terminated *** 2024-03-14T10:07:18.749 INFO:journalctl@ceph.mds.c.smithi152.stdout:Mar 14 10:07:18 smithi152 bash[93112]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-c 2024-03-14T10:07:18.757 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.c.service' 2024-03-14T10:07:18.794 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:07:18.794 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-14T10:07:18.794 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-14T10:07:18.795 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.f 2024-03-14T10:07:19.045 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 14 10:07:18 smithi152 systemd[1]: Stopping Ceph mds.f for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:07:19.499 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 14 10:07:19 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-f[85116]: 2024-03-14T10:07:19.044+0000 7f7da0ef7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:07:19.499 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 14 10:07:19 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-f[85116]: 2024-03-14T10:07:19.044+0000 7f7da0ef7700 -1 mds.f *** got signal Terminated *** 2024-03-14T10:07:23.052 INFO:journalctl@ceph.mds.f.smithi152.stdout:Mar 14 10:07:22 smithi152 bash[93251]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-f 2024-03-14T10:07:23.057 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.f.service' 2024-03-14T10:07:23.091 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:07:23.091 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-14T10:07:23.091 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-14T10:07:23.092 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.i 2024-03-14T10:07:23.353 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 14 10:07:23 smithi152 systemd[1]: Stopping Ceph mds.i for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:07:23.749 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 14 10:07:23 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-i[85801]: 2024-03-14T10:07:23.352+0000 7fced5106700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:07:23.749 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 14 10:07:23 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-i[85801]: 2024-03-14T10:07:23.352+0000 7fced5106700 -1 mds.i *** got signal Terminated *** 2024-03-14T10:07:26.749 INFO:journalctl@ceph.mds.i.smithi152.stdout:Mar 14 10:07:26 smithi152 bash[93392]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-i 2024-03-14T10:07:26.777 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.i.service' 2024-03-14T10:07:26.811 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:07:26.811 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-14T10:07:26.811 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-14T10:07:26.811 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.l 2024-03-14T10:07:27.070 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 14 10:07:26 smithi152 systemd[1]: Stopping Ceph mds.l for 83662c4e-e1e7-11ee-95c9-87774f69a715... 2024-03-14T10:07:27.499 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 14 10:07:27 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-l[84773]: 2024-03-14T10:07:27.068+0000 7f3a537ec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T10:07:27.499 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 14 10:07:27 smithi152 ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-l[84773]: 2024-03-14T10:07:27.068+0000 7f3a537ec700 -1 mds.l *** got signal Terminated *** 2024-03-14T10:07:28.999 INFO:journalctl@ceph.mds.l.smithi152.stdout:Mar 14 10:07:28 smithi152 bash[93530]: ceph-83662c4e-e1e7-11ee-95c9-87774f69a715-mds-l 2024-03-14T10:07:29.022 DEBUG:teuthology.orchestra.run.smithi152:> sudo pkill -f 'journalctl -f -n 0 -u ceph-83662c4e-e1e7-11ee-95c9-87774f69a715@mds.l.service' 2024-03-14T10:07:29.055 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T10:07:29.055 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-14T10:07:29.055 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-14T10:07:29.315 INFO:teuthology.orchestra.run.smithi078.stdout:Deleting cluster with fsid: 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T10:07:32.177 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-14T10:07:32.363 INFO:teuthology.orchestra.run.smithi132.stdout:Deleting cluster with fsid: 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T10:07:33.731 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-14T10:07:33.920 INFO:teuthology.orchestra.run.smithi152.stdout:Deleting cluster with fsid: 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T10:07:35.303 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-14T10:07:35.379 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-14T10:07:35.414 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-14T10:07:35.444 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-14T10:07:35.446 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679/remote/smithi078/crash 2024-03-14T10:07:35.447 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/crash -- . 2024-03-14T10:07:35.549 INFO:teuthology.orchestra.run.smithi078.stderr:tar: /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-14T10:07:35.549 INFO:teuthology.orchestra.run.smithi078.stderr:tar: Error is not recoverable: exiting now 2024-03-14T10:07:35.552 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679/remote/smithi132/crash 2024-03-14T10:07:35.553 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/crash -- . 2024-03-14T10:07:35.600 INFO:teuthology.orchestra.run.smithi132.stderr:tar: /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-14T10:07:35.600 INFO:teuthology.orchestra.run.smithi132.stderr:tar: Error is not recoverable: exiting now 2024-03-14T10:07:35.602 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679/remote/smithi152/crash 2024-03-14T10:07:35.603 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/crash -- . 2024-03-14T10:07:35.637 INFO:teuthology.orchestra.run.smithi152.stderr:tar: /var/lib/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-14T10:07:35.637 INFO:teuthology.orchestra.run.smithi152.stderr:tar: Error is not recoverable: exiting now 2024-03-14T10:07:35.639 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-14T10:07:35.639 DEBUG:teuthology.orchestra.run.smithi078:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-14T10:07:35.715 INFO:teuthology.orchestra.run.smithi078.stderr:grep: /var/log/ceph/83662c4e-e1e7-11ee-95c9-87774f69a715/ceph.log: No such file or directory 2024-03-14T10:07:35.721 INFO:tasks.cephadm:Compressing logs... 2024-03-14T10:07:35.721 DEBUG:teuthology.orchestra.run.smithi078:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-14T10:07:35.725 DEBUG:teuthology.orchestra.run.smithi132:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-14T10:07:35.727 DEBUG:teuthology.orchestra.run.smithi152:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-14T10:07:35.754 INFO:teuthology.orchestra.run.smithi152.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-14T10:07:35.757 INFO:teuthology.orchestra.run.smithi132.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-14T10:07:35.784 INFO:teuthology.orchestra.run.smithi078.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-14T10:09:47.484 INFO:tasks.cephadm:Archiving logs... 2024-03-14T10:09:47.485 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679/remote/smithi078/log 2024-03-14T10:09:47.487 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-14T10:09:59.515 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679/remote/smithi132/log 2024-03-14T10:09:59.516 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-14T10:10:15.498 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679/remote/smithi152/log 2024-03-14T10:10:15.498 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-14T10:10:26.173 INFO:tasks.cephadm:Removing cluster... 2024-03-14T10:10:26.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 --force 2024-03-14T10:10:26.447 INFO:teuthology.orchestra.run.smithi078.stdout:Deleting cluster with fsid: 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T10:10:27.142 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 --force 2024-03-14T10:10:27.332 INFO:teuthology.orchestra.run.smithi132.stdout:Deleting cluster with fsid: 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T10:10:27.697 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 83662c4e-e1e7-11ee-95c9-87774f69a715 --force 2024-03-14T10:10:27.883 INFO:teuthology.orchestra.run.smithi152.stdout:Deleting cluster with fsid: 83662c4e-e1e7-11ee-95c9-87774f69a715 2024-03-14T10:10:28.205 INFO:tasks.cephadm:Removing cephadm ... 2024-03-14T10:10:28.205 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-14T10:10:28.254 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-14T10:10:28.274 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-14T10:10:28.291 INFO:tasks.cephadm:Teardown complete 2024-03-14T10:10:28.292 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-14T10:10:28.306 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-14T10:10:28.306 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-14T10:10:28.311 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-14T10:10:28.317 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-14T10:10:28.448 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-14T10:10:28.459 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-14T10:10:28.459 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-14T10:10:28.464 DEBUG:teuthology.orchestra.run.smithi132:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-14T10:10:28.467 DEBUG:teuthology.orchestra.run.smithi152:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-14T10:10:28.482 INFO:teuthology.orchestra.run.smithi132.stderr:bash: ntpq: command not found 2024-03-14T10:10:28.483 INFO:teuthology.orchestra.run.smithi152.stderr:bash: ntpq: command not found 2024-03-14T10:10:28.487 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-14T10:10:28.487 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-03-14T10:10:28.487 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 20 +3609us[+3610us] +/- 105ms 2024-03-14T10:10:28.487 INFO:teuthology.orchestra.run.smithi132.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -1894us[-1894us] +/- 69ms 2024-03-14T10:10:28.487 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 -383us[ -383us] +/- 68ms 2024-03-14T10:10:28.487 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T10:10:28.488 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-14T10:10:28.488 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-03-14T10:10:28.488 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 +3599us[+3633us] +/- 105ms 2024-03-14T10:10:28.488 INFO:teuthology.orchestra.run.smithi152.stdout:^* hv02.front.sepia.ceph.com 3 6 377 17 -1893us[-1859us] +/- 69ms 2024-03-14T10:10:28.488 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 -389us[ -355us] +/- 68ms 2024-03-14T10:10:28.488 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T10:10:28.500 INFO:teuthology.orchestra.run.smithi078.stderr:bash: ntpq: command not found 2024-03-14T10:10:28.509 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-14T10:10:28.509 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-03-14T10:10:28.509 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 +3666us[+3680us] +/- 105ms 2024-03-14T10:10:28.509 INFO:teuthology.orchestra.run.smithi078.stdout:^* hv02.front.sepia.ceph.com 3 6 377 19 -1874us[-1861us] +/- 69ms 2024-03-14T10:10:28.509 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 -381us[ -381us] +/- 68ms 2024-03-14T10:10:28.509 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T10:10:28.511 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-14T10:10:28.522 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-14T10:10:28.522 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-14T10:10:28.532 DEBUG:teuthology.orchestra.run.smithi078:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-14T10:10:28.689 DEBUG:teuthology.orchestra.run.smithi132:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-14T10:10:28.764 DEBUG:teuthology.orchestra.run.smithi152:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-14T10:10:28.832 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-14T10:10:28.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T10:10:28.903 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-14T10:10:28.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T10:10:28.937 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-14T10:10:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T10:10:28.968 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-14T10:10:28.980 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-14T10:10:29.027 INFO:teuthology.task.internal:Duration was 2350.736798 seconds 2024-03-14T10:10:29.027 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-14T10:10:29.052 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-14T10:10:29.052 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-14T10:10:29.057 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-14T10:10:29.059 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-14T10:10:29.106 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-14T10:10:29.106 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-14T10:10:29.165 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-14T10:10:29.533 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-14T10:10:29.533 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2024-03-14T10:10:29.534 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-14T10:10:29.590 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi132.front.sepia.ceph.com 2024-03-14T10:10:29.591 DEBUG:teuthology.orchestra.run.smithi132:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-14T10:10:29.614 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi152.front.sepia.ceph.com 2024-03-14T10:10:29.614 DEBUG:teuthology.orchestra.run.smithi152:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-14T10:10:29.637 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-14T10:10:29.637 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-14T10:10:29.641 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-14T10:10:29.657 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-14T10:10:29.738 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-14T10:10:29.738 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-14T10:10:29.742 DEBUG:teuthology.orchestra.run.smithi132:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-14T10:10:29.744 DEBUG:teuthology.orchestra.run.smithi152:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-14T10:10:29.902 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-14T10:10:29.915 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-14T10:10:29.915 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-14T10:10:29.982 DEBUG:teuthology.orchestra.run.smithi132:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-14T10:10:30.009 DEBUG:teuthology.orchestra.run.smithi152:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-14T10:10:30.037 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-14T10:10:30.051 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-14T10:10:30.055 DEBUG:teuthology.orchestra.run.smithi132:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-14T10:10:30.057 DEBUG:teuthology.orchestra.run.smithi152:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-14T10:10:30.080 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = core 2024-03-14T10:10:30.104 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = core 2024-03-14T10:10:30.115 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2024-03-14T10:10:30.149 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-14T10:10:30.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T10:10:30.189 DEBUG:teuthology.orchestra.run.smithi132:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-14T10:10:30.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T10:10:30.206 DEBUG:teuthology.orchestra.run.smithi152:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-14T10:10:30.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T10:10:30.222 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-14T10:10:30.236 INFO:teuthology.task.internal:Transferring archived files... 2024-03-14T10:10:30.237 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679/remote/smithi078 2024-03-14T10:10:30.237 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-14T10:10:30.318 DEBUG:teuthology.misc:Transferring archived files from smithi132:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679/remote/smithi132 2024-03-14T10:10:30.318 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-14T10:10:30.358 DEBUG:teuthology.misc:Transferring archived files from smithi152:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679/remote/smithi152 2024-03-14T10:10:30.358 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-14T10:10:30.399 INFO:teuthology.task.internal:Removing archive directory... 2024-03-14T10:10:30.399 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-14T10:10:30.404 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-14T10:10:30.406 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-14T10:10:30.449 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-14T10:10:30.463 INFO:teuthology.task.internal:Not uploading archives. 2024-03-14T10:10:30.463 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-14T10:10:30.500 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-14T10:10:30.500 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-14T10:10:30.505 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-14T10:10:30.507 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-14T10:10:30.521 INFO:teuthology.orchestra.run.smithi132.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 14 10:10 /home/ubuntu/cephtest 2024-03-14T10:10:30.523 INFO:teuthology.orchestra.run.smithi152.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 14 10:10 /home/ubuntu/cephtest 2024-03-14T10:10:30.542 INFO:teuthology.orchestra.run.smithi078.stdout: 524301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 14 10:10 /home/ubuntu/cephtest 2024-03-14T10:10:30.545 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-14T10:10:30.557 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-14T10:10:30.580 INFO:teuthology.nuke:Checking targets against current locks 2024-03-14T10:10:30.607 DEBUG:teuthology.nuke:shortname: smithi078 2024-03-14T10:10:30.607 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-14T10:10:30.632 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-14 09:20:12.353943', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIFWTHVy4dzhYc0F7qesokjZimANuuyTtAS+YDFQ+0JH/BDVHyhOXOzrddNrj9tQP+mCglI5bJ9ZMbpDbyUAyNE='} 2024-03-14T10:10:30.636 DEBUG:teuthology.nuke:shortname: smithi132 2024-03-14T10:10:30.637 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-14T10:10:30.660 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-14 09:20:12.352774', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB0jb3R9uXHmMo7Jr4xV1KZDSVG6sUsXjd19ChB3zt+yG9QrqEJ9vm8+gZ0Zh50EwHZWpspwDnDDlKLChabP6Ak='} 2024-03-14T10:10:30.665 DEBUG:teuthology.nuke:shortname: smithi152 2024-03-14T10:10:30.665 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-14T10:10:30.693 INFO:teuthology.orchestra.console.smithi078:Power off 2024-03-14T10:10:30.694 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-14T10:10:30.706 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598679', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-14 09:20:12.351403', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBECpe1ikaXqUwa5K2sajdqkpk32DfnqCajHjGJiA4oxb1AuwXb0TZ/p0fE/wUCumnkGQw5mXKypRyCfem6JEYQc='} 2024-03-14T10:10:30.717 DEBUG:teuthology.orchestra.console.smithi078:power off output: Chassis Power Control: Down/Off 2024-03-14T10:10:30.718 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-14T10:10:30.741 INFO:teuthology.orchestra.console.smithi132:Power off 2024-03-14T10:10:30.741 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-14T10:10:30.751 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-03-14T10:10:30.763 DEBUG:teuthology.orchestra.console.smithi132:power off output: Chassis Power Control: Down/Off 2024-03-14T10:10:30.763 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-14T10:10:30.777 INFO:teuthology.orchestra.console.smithi152:Power off 2024-03-14T10:10:30.777 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-14T10:10:30.788 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-03-14T10:10:30.799 DEBUG:teuthology.orchestra.console.smithi152:power off output: Chassis Power Control: Down/Off 2024-03-14T10:10:30.799 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-14T10:10:30.827 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-14T10:10:34.753 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-14T10:10:34.789 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-14T10:10:34.828 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-14T10:10:34.864 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-03-14T10:10:34.899 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-03-14T10:10:34.937 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-03-14T10:10:38.868 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-14T10:10:38.900 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-14T10:10:38.939 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-14T10:10:38.979 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off 2024-03-14T10:10:39.010 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is off 2024-03-14T10:10:39.048 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is off 2024-03-14T10:10:39.080 INFO:teuthology.orchestra.console.smithi078:Power off completed 2024-03-14T10:10:39.111 INFO:teuthology.orchestra.console.smithi132:Power off completed 2024-03-14T10:10:39.148 INFO:teuthology.orchestra.console.smithi152:Power off completed 2024-03-14T10:10:39.254 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 5f16de4b3661fb3d4a041ecb2925baeffc01da4a description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} duration: 2350.736797809601 failure_reason: 'The following counters failed to be set on mds daemons: {''mds.exported'', ''mds.imported''}' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2024-03-14T10:10:39.254 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-14T10:10:39.347 INFO:teuthology.run:FAIL